Nov 03 00:40:09.944185 ip-10-0-140-107 systemd[1]: Starting Kubernetes Kubelet... Nov 03 00:40:10.332011 ip-10-0-140-107 kubenswrapper[2578]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Nov 03 00:40:10.332011 ip-10-0-140-107 kubenswrapper[2578]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version. Nov 03 00:40:10.332011 ip-10-0-140-107 kubenswrapper[2578]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Nov 03 00:40:10.332011 ip-10-0-140-107 kubenswrapper[2578]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Nov 03 00:40:10.335436 ip-10-0-140-107 kubenswrapper[2578]: W1103 00:40:10.335358 2578 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 03 00:40:10.335436 ip-10-0-140-107 kubenswrapper[2578]: W1103 00:40:10.335436 2578 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 03 00:40:10.335504 ip-10-0-140-107 kubenswrapper[2578]: W1103 00:40:10.335440 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 03 00:40:10.335504 ip-10-0-140-107 kubenswrapper[2578]: W1103 00:40:10.335445 2578 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 03 00:40:10.335504 ip-10-0-140-107 kubenswrapper[2578]: W1103 00:40:10.335449 2578 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 03 00:40:10.335504 ip-10-0-140-107 kubenswrapper[2578]: W1103 00:40:10.335452 2578 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 03 00:40:10.335504 ip-10-0-140-107 kubenswrapper[2578]: W1103 00:40:10.335454 2578 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 03 00:40:10.335504 ip-10-0-140-107 kubenswrapper[2578]: W1103 00:40:10.335458 2578 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 03 00:40:10.335504 ip-10-0-140-107 kubenswrapper[2578]: W1103 00:40:10.335460 2578 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 03 00:40:10.335504 ip-10-0-140-107 kubenswrapper[2578]: W1103 00:40:10.335463 2578 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 03 00:40:10.335504 ip-10-0-140-107 kubenswrapper[2578]: W1103 00:40:10.335466 2578 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 03 00:40:10.335504 ip-10-0-140-107 kubenswrapper[2578]: W1103 00:40:10.335469 2578 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 03 00:40:10.335504 ip-10-0-140-107 kubenswrapper[2578]: W1103 00:40:10.335471 2578 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 03 00:40:10.335504 ip-10-0-140-107 kubenswrapper[2578]: W1103 00:40:10.335474 2578 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 03 00:40:10.335504 ip-10-0-140-107 kubenswrapper[2578]: W1103 00:40:10.335478 2578 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 03 00:40:10.335504 ip-10-0-140-107 kubenswrapper[2578]: W1103 00:40:10.335484 2578 feature_gate.go:386] unrecognized feature gate: Example Nov 03 00:40:10.335504 ip-10-0-140-107 kubenswrapper[2578]: W1103 00:40:10.335488 2578 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 03 00:40:10.335504 ip-10-0-140-107 kubenswrapper[2578]: W1103 00:40:10.335491 2578 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 03 00:40:10.335504 ip-10-0-140-107 kubenswrapper[2578]: W1103 00:40:10.335493 2578 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 03 00:40:10.335504 ip-10-0-140-107 kubenswrapper[2578]: W1103 00:40:10.335496 2578 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 03 00:40:10.335504 ip-10-0-140-107 kubenswrapper[2578]: W1103 00:40:10.335499 2578 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 03 00:40:10.335951 ip-10-0-140-107 kubenswrapper[2578]: W1103 00:40:10.335502 2578 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 03 00:40:10.335951 ip-10-0-140-107 kubenswrapper[2578]: W1103 00:40:10.335505 2578 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 03 00:40:10.335951 ip-10-0-140-107 kubenswrapper[2578]: W1103 00:40:10.335508 2578 feature_gate.go:386] unrecognized feature gate: EVPN Nov 03 00:40:10.335951 ip-10-0-140-107 kubenswrapper[2578]: W1103 00:40:10.335511 2578 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 03 00:40:10.335951 ip-10-0-140-107 kubenswrapper[2578]: W1103 00:40:10.335514 2578 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 03 00:40:10.335951 ip-10-0-140-107 kubenswrapper[2578]: W1103 00:40:10.335517 2578 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 03 00:40:10.335951 ip-10-0-140-107 kubenswrapper[2578]: W1103 00:40:10.335520 2578 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 03 00:40:10.335951 ip-10-0-140-107 kubenswrapper[2578]: W1103 00:40:10.335523 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 03 00:40:10.335951 ip-10-0-140-107 kubenswrapper[2578]: W1103 00:40:10.335525 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 03 00:40:10.335951 ip-10-0-140-107 kubenswrapper[2578]: W1103 00:40:10.335528 2578 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 03 00:40:10.335951 ip-10-0-140-107 kubenswrapper[2578]: W1103 00:40:10.335531 2578 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 03 00:40:10.335951 ip-10-0-140-107 kubenswrapper[2578]: W1103 00:40:10.335533 2578 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 03 00:40:10.335951 ip-10-0-140-107 kubenswrapper[2578]: W1103 00:40:10.335536 2578 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 03 00:40:10.335951 ip-10-0-140-107 kubenswrapper[2578]: W1103 00:40:10.335538 2578 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 03 00:40:10.335951 ip-10-0-140-107 kubenswrapper[2578]: W1103 00:40:10.335541 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 03 00:40:10.335951 ip-10-0-140-107 kubenswrapper[2578]: W1103 00:40:10.335544 2578 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 03 00:40:10.335951 ip-10-0-140-107 kubenswrapper[2578]: W1103 00:40:10.335546 2578 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 03 00:40:10.335951 ip-10-0-140-107 kubenswrapper[2578]: W1103 00:40:10.335549 2578 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 03 00:40:10.335951 ip-10-0-140-107 kubenswrapper[2578]: W1103 00:40:10.335551 2578 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 03 00:40:10.335951 ip-10-0-140-107 kubenswrapper[2578]: W1103 00:40:10.335554 2578 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 03 00:40:10.336417 ip-10-0-140-107 kubenswrapper[2578]: W1103 00:40:10.335557 2578 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 03 00:40:10.336417 ip-10-0-140-107 kubenswrapper[2578]: W1103 00:40:10.335559 2578 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 03 00:40:10.336417 ip-10-0-140-107 kubenswrapper[2578]: W1103 00:40:10.335562 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 03 00:40:10.336417 ip-10-0-140-107 kubenswrapper[2578]: W1103 00:40:10.335564 2578 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 03 00:40:10.336417 ip-10-0-140-107 kubenswrapper[2578]: W1103 00:40:10.335567 2578 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 03 00:40:10.336417 ip-10-0-140-107 kubenswrapper[2578]: W1103 00:40:10.335570 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 03 00:40:10.336417 ip-10-0-140-107 kubenswrapper[2578]: W1103 00:40:10.335572 2578 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 03 00:40:10.336417 ip-10-0-140-107 kubenswrapper[2578]: W1103 00:40:10.335575 2578 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 03 00:40:10.336417 ip-10-0-140-107 kubenswrapper[2578]: W1103 00:40:10.335578 2578 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 03 00:40:10.336417 ip-10-0-140-107 kubenswrapper[2578]: W1103 00:40:10.335582 2578 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 03 00:40:10.336417 ip-10-0-140-107 kubenswrapper[2578]: W1103 00:40:10.335586 2578 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 03 00:40:10.336417 ip-10-0-140-107 kubenswrapper[2578]: W1103 00:40:10.335590 2578 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 03 00:40:10.336417 ip-10-0-140-107 kubenswrapper[2578]: W1103 00:40:10.335594 2578 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 03 00:40:10.336417 ip-10-0-140-107 kubenswrapper[2578]: W1103 00:40:10.335597 2578 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 03 00:40:10.336417 ip-10-0-140-107 kubenswrapper[2578]: W1103 00:40:10.335600 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 03 00:40:10.336417 ip-10-0-140-107 kubenswrapper[2578]: W1103 00:40:10.335603 2578 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 03 00:40:10.336417 ip-10-0-140-107 kubenswrapper[2578]: W1103 00:40:10.335606 2578 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 03 00:40:10.336417 ip-10-0-140-107 kubenswrapper[2578]: W1103 00:40:10.335609 2578 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 03 00:40:10.336417 ip-10-0-140-107 kubenswrapper[2578]: W1103 00:40:10.335611 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 03 00:40:10.336860 ip-10-0-140-107 kubenswrapper[2578]: W1103 00:40:10.335614 2578 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 03 00:40:10.336860 ip-10-0-140-107 kubenswrapper[2578]: W1103 00:40:10.335617 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 03 00:40:10.336860 ip-10-0-140-107 kubenswrapper[2578]: W1103 00:40:10.335619 2578 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 03 00:40:10.336860 ip-10-0-140-107 kubenswrapper[2578]: W1103 00:40:10.335622 2578 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 03 00:40:10.336860 ip-10-0-140-107 kubenswrapper[2578]: W1103 00:40:10.335624 2578 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 03 00:40:10.336860 ip-10-0-140-107 kubenswrapper[2578]: W1103 00:40:10.335627 2578 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 03 00:40:10.336860 ip-10-0-140-107 kubenswrapper[2578]: W1103 00:40:10.335629 2578 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 03 00:40:10.336860 ip-10-0-140-107 kubenswrapper[2578]: W1103 00:40:10.335632 2578 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 03 00:40:10.336860 ip-10-0-140-107 kubenswrapper[2578]: W1103 00:40:10.335635 2578 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 03 00:40:10.336860 ip-10-0-140-107 kubenswrapper[2578]: W1103 00:40:10.335637 2578 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 03 00:40:10.336860 ip-10-0-140-107 kubenswrapper[2578]: W1103 00:40:10.335640 2578 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 03 00:40:10.336860 ip-10-0-140-107 kubenswrapper[2578]: W1103 00:40:10.335643 2578 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 03 00:40:10.336860 ip-10-0-140-107 kubenswrapper[2578]: W1103 00:40:10.335645 2578 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 03 00:40:10.336860 ip-10-0-140-107 kubenswrapper[2578]: W1103 00:40:10.335648 2578 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 03 00:40:10.336860 ip-10-0-140-107 kubenswrapper[2578]: W1103 00:40:10.335651 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 03 00:40:10.336860 ip-10-0-140-107 kubenswrapper[2578]: W1103 00:40:10.335653 2578 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 03 00:40:10.336860 ip-10-0-140-107 kubenswrapper[2578]: W1103 00:40:10.335657 2578 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 03 00:40:10.336860 ip-10-0-140-107 kubenswrapper[2578]: W1103 00:40:10.335660 2578 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 03 00:40:10.336860 ip-10-0-140-107 kubenswrapper[2578]: W1103 00:40:10.335663 2578 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 03 00:40:10.336860 ip-10-0-140-107 kubenswrapper[2578]: W1103 00:40:10.335665 2578 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 03 00:40:10.337367 ip-10-0-140-107 kubenswrapper[2578]: W1103 00:40:10.335668 2578 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 03 00:40:10.337367 ip-10-0-140-107 kubenswrapper[2578]: W1103 00:40:10.335670 2578 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 03 00:40:10.337367 ip-10-0-140-107 kubenswrapper[2578]: W1103 00:40:10.335673 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 03 00:40:10.337367 ip-10-0-140-107 kubenswrapper[2578]: W1103 00:40:10.335675 2578 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 03 00:40:10.337367 ip-10-0-140-107 kubenswrapper[2578]: W1103 00:40:10.335679 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 03 00:40:10.337367 ip-10-0-140-107 kubenswrapper[2578]: W1103 00:40:10.335682 2578 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 03 00:40:10.337367 ip-10-0-140-107 kubenswrapper[2578]: W1103 00:40:10.335685 2578 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 03 00:40:10.337367 ip-10-0-140-107 kubenswrapper[2578]: W1103 00:40:10.335687 2578 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 03 00:40:10.337367 ip-10-0-140-107 kubenswrapper[2578]: W1103 00:40:10.335690 2578 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 03 00:40:10.337367 ip-10-0-140-107 kubenswrapper[2578]: W1103 00:40:10.335692 2578 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 03 00:40:10.337367 ip-10-0-140-107 kubenswrapper[2578]: W1103 00:40:10.335695 2578 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 03 00:40:10.337367 ip-10-0-140-107 kubenswrapper[2578]: W1103 00:40:10.335698 2578 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 03 00:40:10.337367 ip-10-0-140-107 kubenswrapper[2578]: W1103 00:40:10.335701 2578 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 03 00:40:10.337367 ip-10-0-140-107 kubenswrapper[2578]: W1103 00:40:10.335704 2578 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 03 00:40:10.337367 ip-10-0-140-107 kubenswrapper[2578]: W1103 00:40:10.335706 2578 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 03 00:40:10.337367 ip-10-0-140-107 kubenswrapper[2578]: W1103 00:40:10.335709 2578 feature_gate.go:386] unrecognized feature gate: Example2 Nov 03 00:40:10.337367 ip-10-0-140-107 kubenswrapper[2578]: W1103 00:40:10.335712 2578 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 03 00:40:10.337367 ip-10-0-140-107 kubenswrapper[2578]: W1103 00:40:10.335715 2578 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 03 00:40:10.337367 ip-10-0-140-107 kubenswrapper[2578]: W1103 00:40:10.335718 2578 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 03 00:40:10.337367 ip-10-0-140-107 kubenswrapper[2578]: W1103 00:40:10.335720 2578 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 03 00:40:10.337893 ip-10-0-140-107 kubenswrapper[2578]: W1103 00:40:10.335723 2578 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 03 00:40:10.337893 ip-10-0-140-107 kubenswrapper[2578]: W1103 00:40:10.335726 2578 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 03 00:40:10.337893 ip-10-0-140-107 kubenswrapper[2578]: W1103 00:40:10.335731 2578 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 03 00:40:10.337893 ip-10-0-140-107 kubenswrapper[2578]: W1103 00:40:10.335735 2578 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 03 00:40:10.337893 ip-10-0-140-107 kubenswrapper[2578]: W1103 00:40:10.335740 2578 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 03 00:40:10.337893 ip-10-0-140-107 kubenswrapper[2578]: W1103 00:40:10.335743 2578 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 03 00:40:10.337893 ip-10-0-140-107 kubenswrapper[2578]: W1103 00:40:10.336548 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 03 00:40:10.337893 ip-10-0-140-107 kubenswrapper[2578]: W1103 00:40:10.336556 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 03 00:40:10.337893 ip-10-0-140-107 kubenswrapper[2578]: W1103 00:40:10.336559 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 03 00:40:10.337893 ip-10-0-140-107 kubenswrapper[2578]: W1103 00:40:10.336563 2578 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 03 00:40:10.337893 ip-10-0-140-107 kubenswrapper[2578]: W1103 00:40:10.336566 2578 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 03 00:40:10.337893 ip-10-0-140-107 kubenswrapper[2578]: W1103 00:40:10.336569 2578 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 03 00:40:10.337893 ip-10-0-140-107 kubenswrapper[2578]: W1103 00:40:10.336572 2578 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 03 00:40:10.337893 ip-10-0-140-107 kubenswrapper[2578]: W1103 00:40:10.336574 2578 feature_gate.go:386] unrecognized feature gate: Example2 Nov 03 00:40:10.337893 ip-10-0-140-107 kubenswrapper[2578]: W1103 00:40:10.336577 2578 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 03 00:40:10.337893 ip-10-0-140-107 kubenswrapper[2578]: W1103 00:40:10.336580 2578 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 03 00:40:10.337893 ip-10-0-140-107 kubenswrapper[2578]: W1103 00:40:10.336583 2578 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 03 00:40:10.337893 ip-10-0-140-107 kubenswrapper[2578]: W1103 00:40:10.336587 2578 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 03 00:40:10.337893 ip-10-0-140-107 kubenswrapper[2578]: W1103 00:40:10.336590 2578 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 03 00:40:10.337893 ip-10-0-140-107 kubenswrapper[2578]: W1103 00:40:10.336593 2578 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 03 00:40:10.338483 ip-10-0-140-107 kubenswrapper[2578]: W1103 00:40:10.336596 2578 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 03 00:40:10.338483 ip-10-0-140-107 kubenswrapper[2578]: W1103 00:40:10.336599 2578 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 03 00:40:10.338483 ip-10-0-140-107 kubenswrapper[2578]: W1103 00:40:10.336602 2578 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 03 00:40:10.338483 ip-10-0-140-107 kubenswrapper[2578]: W1103 00:40:10.336604 2578 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 03 00:40:10.338483 ip-10-0-140-107 kubenswrapper[2578]: W1103 00:40:10.336607 2578 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 03 00:40:10.338483 ip-10-0-140-107 kubenswrapper[2578]: W1103 00:40:10.336610 2578 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 03 00:40:10.338483 ip-10-0-140-107 kubenswrapper[2578]: W1103 00:40:10.336612 2578 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 03 00:40:10.338483 ip-10-0-140-107 kubenswrapper[2578]: W1103 00:40:10.336615 2578 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 03 00:40:10.338483 ip-10-0-140-107 kubenswrapper[2578]: W1103 00:40:10.336617 2578 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 03 00:40:10.338483 ip-10-0-140-107 kubenswrapper[2578]: W1103 00:40:10.336620 2578 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 03 00:40:10.338483 ip-10-0-140-107 kubenswrapper[2578]: W1103 00:40:10.336624 2578 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 03 00:40:10.338483 ip-10-0-140-107 kubenswrapper[2578]: W1103 00:40:10.336628 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 03 00:40:10.338483 ip-10-0-140-107 kubenswrapper[2578]: W1103 00:40:10.336631 2578 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 03 00:40:10.338483 ip-10-0-140-107 kubenswrapper[2578]: W1103 00:40:10.336634 2578 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 03 00:40:10.338483 ip-10-0-140-107 kubenswrapper[2578]: W1103 00:40:10.336638 2578 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 03 00:40:10.338483 ip-10-0-140-107 kubenswrapper[2578]: W1103 00:40:10.336641 2578 feature_gate.go:386] unrecognized feature gate: Example Nov 03 00:40:10.338483 ip-10-0-140-107 kubenswrapper[2578]: W1103 00:40:10.336643 2578 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 03 00:40:10.338483 ip-10-0-140-107 kubenswrapper[2578]: W1103 00:40:10.336646 2578 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 03 00:40:10.338483 ip-10-0-140-107 kubenswrapper[2578]: W1103 00:40:10.336649 2578 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 03 00:40:10.338960 ip-10-0-140-107 kubenswrapper[2578]: W1103 00:40:10.336651 2578 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 03 00:40:10.338960 ip-10-0-140-107 kubenswrapper[2578]: W1103 00:40:10.336654 2578 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 03 00:40:10.338960 ip-10-0-140-107 kubenswrapper[2578]: W1103 00:40:10.336658 2578 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 03 00:40:10.338960 ip-10-0-140-107 kubenswrapper[2578]: W1103 00:40:10.336661 2578 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 03 00:40:10.338960 ip-10-0-140-107 kubenswrapper[2578]: W1103 00:40:10.336664 2578 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 03 00:40:10.338960 ip-10-0-140-107 kubenswrapper[2578]: W1103 00:40:10.336666 2578 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 03 00:40:10.338960 ip-10-0-140-107 kubenswrapper[2578]: W1103 00:40:10.336669 2578 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 03 00:40:10.338960 ip-10-0-140-107 kubenswrapper[2578]: W1103 00:40:10.336673 2578 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 03 00:40:10.338960 ip-10-0-140-107 kubenswrapper[2578]: W1103 00:40:10.336676 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 03 00:40:10.338960 ip-10-0-140-107 kubenswrapper[2578]: W1103 00:40:10.336679 2578 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 03 00:40:10.338960 ip-10-0-140-107 kubenswrapper[2578]: W1103 00:40:10.336682 2578 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 03 00:40:10.338960 ip-10-0-140-107 kubenswrapper[2578]: W1103 00:40:10.336685 2578 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 03 00:40:10.338960 ip-10-0-140-107 kubenswrapper[2578]: W1103 00:40:10.336688 2578 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 03 00:40:10.338960 ip-10-0-140-107 kubenswrapper[2578]: W1103 00:40:10.336691 2578 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 03 00:40:10.338960 ip-10-0-140-107 kubenswrapper[2578]: W1103 00:40:10.336693 2578 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 03 00:40:10.338960 ip-10-0-140-107 kubenswrapper[2578]: W1103 00:40:10.336697 2578 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 03 00:40:10.338960 ip-10-0-140-107 kubenswrapper[2578]: W1103 00:40:10.336700 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 03 00:40:10.338960 ip-10-0-140-107 kubenswrapper[2578]: W1103 00:40:10.336702 2578 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 03 00:40:10.338960 ip-10-0-140-107 kubenswrapper[2578]: W1103 00:40:10.336705 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 03 00:40:10.339417 ip-10-0-140-107 kubenswrapper[2578]: W1103 00:40:10.336707 2578 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 03 00:40:10.339417 ip-10-0-140-107 kubenswrapper[2578]: W1103 00:40:10.336710 2578 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 03 00:40:10.339417 ip-10-0-140-107 kubenswrapper[2578]: W1103 00:40:10.336712 2578 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 03 00:40:10.339417 ip-10-0-140-107 kubenswrapper[2578]: W1103 00:40:10.336715 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 03 00:40:10.339417 ip-10-0-140-107 kubenswrapper[2578]: W1103 00:40:10.336717 2578 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 03 00:40:10.339417 ip-10-0-140-107 kubenswrapper[2578]: W1103 00:40:10.336720 2578 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 03 00:40:10.339417 ip-10-0-140-107 kubenswrapper[2578]: W1103 00:40:10.336723 2578 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 03 00:40:10.339417 ip-10-0-140-107 kubenswrapper[2578]: W1103 00:40:10.336725 2578 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 03 00:40:10.339417 ip-10-0-140-107 kubenswrapper[2578]: W1103 00:40:10.336729 2578 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 03 00:40:10.339417 ip-10-0-140-107 kubenswrapper[2578]: W1103 00:40:10.336732 2578 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 03 00:40:10.339417 ip-10-0-140-107 kubenswrapper[2578]: W1103 00:40:10.336734 2578 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 03 00:40:10.339417 ip-10-0-140-107 kubenswrapper[2578]: W1103 00:40:10.336737 2578 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 03 00:40:10.339417 ip-10-0-140-107 kubenswrapper[2578]: W1103 00:40:10.336739 2578 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 03 00:40:10.339417 ip-10-0-140-107 kubenswrapper[2578]: W1103 00:40:10.336742 2578 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 03 00:40:10.339417 ip-10-0-140-107 kubenswrapper[2578]: W1103 00:40:10.336745 2578 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 03 00:40:10.339417 ip-10-0-140-107 kubenswrapper[2578]: W1103 00:40:10.336748 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 03 00:40:10.339417 ip-10-0-140-107 kubenswrapper[2578]: W1103 00:40:10.336751 2578 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 03 00:40:10.339417 ip-10-0-140-107 kubenswrapper[2578]: W1103 00:40:10.336753 2578 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 03 00:40:10.339417 ip-10-0-140-107 kubenswrapper[2578]: W1103 00:40:10.336756 2578 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 03 00:40:10.339417 ip-10-0-140-107 kubenswrapper[2578]: W1103 00:40:10.336758 2578 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 03 00:40:10.339921 ip-10-0-140-107 kubenswrapper[2578]: W1103 00:40:10.336761 2578 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 03 00:40:10.339921 ip-10-0-140-107 kubenswrapper[2578]: W1103 00:40:10.336764 2578 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 03 00:40:10.339921 ip-10-0-140-107 kubenswrapper[2578]: W1103 00:40:10.336766 2578 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 03 00:40:10.339921 ip-10-0-140-107 kubenswrapper[2578]: W1103 00:40:10.336769 2578 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 03 00:40:10.339921 ip-10-0-140-107 kubenswrapper[2578]: W1103 00:40:10.336772 2578 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 03 00:40:10.339921 ip-10-0-140-107 kubenswrapper[2578]: W1103 00:40:10.336775 2578 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 03 00:40:10.339921 ip-10-0-140-107 kubenswrapper[2578]: W1103 00:40:10.336777 2578 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 03 00:40:10.339921 ip-10-0-140-107 kubenswrapper[2578]: W1103 00:40:10.336780 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 03 00:40:10.339921 ip-10-0-140-107 kubenswrapper[2578]: W1103 00:40:10.336783 2578 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 03 00:40:10.339921 ip-10-0-140-107 kubenswrapper[2578]: W1103 00:40:10.336785 2578 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 03 00:40:10.339921 ip-10-0-140-107 kubenswrapper[2578]: W1103 00:40:10.336788 2578 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 03 00:40:10.339921 ip-10-0-140-107 kubenswrapper[2578]: W1103 00:40:10.336791 2578 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 03 00:40:10.339921 ip-10-0-140-107 kubenswrapper[2578]: W1103 00:40:10.336793 2578 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 03 00:40:10.339921 ip-10-0-140-107 kubenswrapper[2578]: W1103 00:40:10.336796 2578 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 03 00:40:10.339921 ip-10-0-140-107 kubenswrapper[2578]: W1103 00:40:10.336799 2578 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 03 00:40:10.339921 ip-10-0-140-107 kubenswrapper[2578]: W1103 00:40:10.336801 2578 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 03 00:40:10.339921 ip-10-0-140-107 kubenswrapper[2578]: W1103 00:40:10.336804 2578 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 03 00:40:10.339921 ip-10-0-140-107 kubenswrapper[2578]: W1103 00:40:10.336807 2578 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 03 00:40:10.339921 ip-10-0-140-107 kubenswrapper[2578]: W1103 00:40:10.336809 2578 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 03 00:40:10.340375 ip-10-0-140-107 kubenswrapper[2578]: W1103 00:40:10.336812 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 03 00:40:10.340375 ip-10-0-140-107 kubenswrapper[2578]: W1103 00:40:10.336815 2578 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 03 00:40:10.340375 ip-10-0-140-107 kubenswrapper[2578]: W1103 00:40:10.336817 2578 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 03 00:40:10.340375 ip-10-0-140-107 kubenswrapper[2578]: W1103 00:40:10.336820 2578 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 03 00:40:10.340375 ip-10-0-140-107 kubenswrapper[2578]: W1103 00:40:10.336823 2578 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 03 00:40:10.340375 ip-10-0-140-107 kubenswrapper[2578]: W1103 00:40:10.336825 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 03 00:40:10.340375 ip-10-0-140-107 kubenswrapper[2578]: W1103 00:40:10.336828 2578 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 03 00:40:10.340375 ip-10-0-140-107 kubenswrapper[2578]: W1103 00:40:10.336831 2578 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 03 00:40:10.340375 ip-10-0-140-107 kubenswrapper[2578]: W1103 00:40:10.336834 2578 feature_gate.go:386] unrecognized feature gate: EVPN Nov 03 00:40:10.340375 ip-10-0-140-107 kubenswrapper[2578]: W1103 00:40:10.336837 2578 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 03 00:40:10.340375 ip-10-0-140-107 kubenswrapper[2578]: W1103 00:40:10.336840 2578 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 03 00:40:10.340375 ip-10-0-140-107 kubenswrapper[2578]: W1103 00:40:10.336842 2578 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 03 00:40:10.340375 ip-10-0-140-107 kubenswrapper[2578]: W1103 00:40:10.336845 2578 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 03 00:40:10.340375 ip-10-0-140-107 kubenswrapper[2578]: W1103 00:40:10.336848 2578 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 03 00:40:10.340375 ip-10-0-140-107 kubenswrapper[2578]: W1103 00:40:10.336850 2578 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 03 00:40:10.340375 ip-10-0-140-107 kubenswrapper[2578]: W1103 00:40:10.337398 2578 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 03 00:40:10.340375 ip-10-0-140-107 kubenswrapper[2578]: W1103 00:40:10.337405 2578 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 03 00:40:10.340375 ip-10-0-140-107 kubenswrapper[2578]: W1103 00:40:10.337409 2578 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 03 00:40:10.340375 ip-10-0-140-107 kubenswrapper[2578]: W1103 00:40:10.337413 2578 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 03 00:40:10.340375 ip-10-0-140-107 kubenswrapper[2578]: W1103 00:40:10.337416 2578 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 03 00:40:10.340842 ip-10-0-140-107 kubenswrapper[2578]: W1103 00:40:10.337418 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 03 00:40:10.340842 ip-10-0-140-107 kubenswrapper[2578]: W1103 00:40:10.337421 2578 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 03 00:40:10.340842 ip-10-0-140-107 kubenswrapper[2578]: W1103 00:40:10.337424 2578 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 03 00:40:10.340842 ip-10-0-140-107 kubenswrapper[2578]: W1103 00:40:10.337427 2578 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 03 00:40:10.340842 ip-10-0-140-107 kubenswrapper[2578]: W1103 00:40:10.337429 2578 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 03 00:40:10.340842 ip-10-0-140-107 kubenswrapper[2578]: W1103 00:40:10.337432 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 03 00:40:10.340842 ip-10-0-140-107 kubenswrapper[2578]: W1103 00:40:10.337435 2578 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 03 00:40:10.340842 ip-10-0-140-107 kubenswrapper[2578]: W1103 00:40:10.337438 2578 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 03 00:40:10.340842 ip-10-0-140-107 kubenswrapper[2578]: W1103 00:40:10.337440 2578 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 03 00:40:10.340842 ip-10-0-140-107 kubenswrapper[2578]: W1103 00:40:10.337443 2578 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 03 00:40:10.340842 ip-10-0-140-107 kubenswrapper[2578]: W1103 00:40:10.337445 2578 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 03 00:40:10.340842 ip-10-0-140-107 kubenswrapper[2578]: W1103 00:40:10.337448 2578 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 03 00:40:10.340842 ip-10-0-140-107 kubenswrapper[2578]: W1103 00:40:10.337451 2578 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 03 00:40:10.340842 ip-10-0-140-107 kubenswrapper[2578]: W1103 00:40:10.337454 2578 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 03 00:40:10.340842 ip-10-0-140-107 kubenswrapper[2578]: W1103 00:40:10.337457 2578 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 03 00:40:10.340842 ip-10-0-140-107 kubenswrapper[2578]: W1103 00:40:10.337460 2578 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 03 00:40:10.340842 ip-10-0-140-107 kubenswrapper[2578]: W1103 00:40:10.337463 2578 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 03 00:40:10.340842 ip-10-0-140-107 kubenswrapper[2578]: W1103 00:40:10.337465 2578 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 03 00:40:10.340842 ip-10-0-140-107 kubenswrapper[2578]: W1103 00:40:10.337468 2578 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 03 00:40:10.340842 ip-10-0-140-107 kubenswrapper[2578]: W1103 00:40:10.337471 2578 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 03 00:40:10.341306 ip-10-0-140-107 kubenswrapper[2578]: W1103 00:40:10.337474 2578 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 03 00:40:10.341306 ip-10-0-140-107 kubenswrapper[2578]: W1103 00:40:10.337476 2578 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 03 00:40:10.341306 ip-10-0-140-107 kubenswrapper[2578]: W1103 00:40:10.337479 2578 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 03 00:40:10.341306 ip-10-0-140-107 kubenswrapper[2578]: W1103 00:40:10.337482 2578 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 03 00:40:10.341306 ip-10-0-140-107 kubenswrapper[2578]: W1103 00:40:10.337484 2578 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 03 00:40:10.341306 ip-10-0-140-107 kubenswrapper[2578]: W1103 00:40:10.337487 2578 feature_gate.go:386] unrecognized feature gate: Example2 Nov 03 00:40:10.341306 ip-10-0-140-107 kubenswrapper[2578]: W1103 00:40:10.337490 2578 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 03 00:40:10.341306 ip-10-0-140-107 kubenswrapper[2578]: W1103 00:40:10.337493 2578 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 03 00:40:10.341306 ip-10-0-140-107 kubenswrapper[2578]: W1103 00:40:10.337495 2578 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 03 00:40:10.341306 ip-10-0-140-107 kubenswrapper[2578]: W1103 00:40:10.337498 2578 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 03 00:40:10.341306 ip-10-0-140-107 kubenswrapper[2578]: W1103 00:40:10.337502 2578 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 03 00:40:10.341306 ip-10-0-140-107 kubenswrapper[2578]: W1103 00:40:10.337506 2578 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 03 00:40:10.341306 ip-10-0-140-107 kubenswrapper[2578]: W1103 00:40:10.337509 2578 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 03 00:40:10.341306 ip-10-0-140-107 kubenswrapper[2578]: W1103 00:40:10.337512 2578 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 03 00:40:10.341306 ip-10-0-140-107 kubenswrapper[2578]: W1103 00:40:10.337515 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 03 00:40:10.341306 ip-10-0-140-107 kubenswrapper[2578]: W1103 00:40:10.337517 2578 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 03 00:40:10.341306 ip-10-0-140-107 kubenswrapper[2578]: W1103 00:40:10.337520 2578 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 03 00:40:10.341306 ip-10-0-140-107 kubenswrapper[2578]: W1103 00:40:10.337522 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 03 00:40:10.341306 ip-10-0-140-107 kubenswrapper[2578]: W1103 00:40:10.337525 2578 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 03 00:40:10.341761 ip-10-0-140-107 kubenswrapper[2578]: W1103 00:40:10.337527 2578 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 03 00:40:10.341761 ip-10-0-140-107 kubenswrapper[2578]: W1103 00:40:10.337530 2578 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 03 00:40:10.341761 ip-10-0-140-107 kubenswrapper[2578]: W1103 00:40:10.337532 2578 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 03 00:40:10.341761 ip-10-0-140-107 kubenswrapper[2578]: W1103 00:40:10.337535 2578 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 03 00:40:10.341761 ip-10-0-140-107 kubenswrapper[2578]: W1103 00:40:10.337538 2578 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 03 00:40:10.341761 ip-10-0-140-107 kubenswrapper[2578]: W1103 00:40:10.337541 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 03 00:40:10.341761 ip-10-0-140-107 kubenswrapper[2578]: W1103 00:40:10.337545 2578 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 03 00:40:10.341761 ip-10-0-140-107 kubenswrapper[2578]: W1103 00:40:10.337547 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 03 00:40:10.341761 ip-10-0-140-107 kubenswrapper[2578]: W1103 00:40:10.337550 2578 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 03 00:40:10.341761 ip-10-0-140-107 kubenswrapper[2578]: W1103 00:40:10.337552 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 03 00:40:10.341761 ip-10-0-140-107 kubenswrapper[2578]: W1103 00:40:10.337555 2578 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 03 00:40:10.341761 ip-10-0-140-107 kubenswrapper[2578]: W1103 00:40:10.337561 2578 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 03 00:40:10.341761 ip-10-0-140-107 kubenswrapper[2578]: W1103 00:40:10.337564 2578 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 03 00:40:10.341761 ip-10-0-140-107 kubenswrapper[2578]: W1103 00:40:10.337567 2578 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 03 00:40:10.341761 ip-10-0-140-107 kubenswrapper[2578]: W1103 00:40:10.337569 2578 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 03 00:40:10.341761 ip-10-0-140-107 kubenswrapper[2578]: W1103 00:40:10.337572 2578 feature_gate.go:386] unrecognized feature gate: EVPN Nov 03 00:40:10.341761 ip-10-0-140-107 kubenswrapper[2578]: W1103 00:40:10.337575 2578 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 03 00:40:10.341761 ip-10-0-140-107 kubenswrapper[2578]: W1103 00:40:10.337577 2578 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 03 00:40:10.341761 ip-10-0-140-107 kubenswrapper[2578]: W1103 00:40:10.337580 2578 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 03 00:40:10.342222 ip-10-0-140-107 kubenswrapper[2578]: W1103 00:40:10.337583 2578 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 03 00:40:10.342222 ip-10-0-140-107 kubenswrapper[2578]: W1103 00:40:10.337585 2578 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 03 00:40:10.342222 ip-10-0-140-107 kubenswrapper[2578]: W1103 00:40:10.337588 2578 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 03 00:40:10.342222 ip-10-0-140-107 kubenswrapper[2578]: W1103 00:40:10.337591 2578 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 03 00:40:10.342222 ip-10-0-140-107 kubenswrapper[2578]: W1103 00:40:10.337593 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 03 00:40:10.342222 ip-10-0-140-107 kubenswrapper[2578]: W1103 00:40:10.337596 2578 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 03 00:40:10.342222 ip-10-0-140-107 kubenswrapper[2578]: W1103 00:40:10.337602 2578 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 03 00:40:10.342222 ip-10-0-140-107 kubenswrapper[2578]: W1103 00:40:10.337605 2578 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 03 00:40:10.342222 ip-10-0-140-107 kubenswrapper[2578]: W1103 00:40:10.337607 2578 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 03 00:40:10.342222 ip-10-0-140-107 kubenswrapper[2578]: W1103 00:40:10.337610 2578 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 03 00:40:10.342222 ip-10-0-140-107 kubenswrapper[2578]: W1103 00:40:10.337612 2578 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 03 00:40:10.342222 ip-10-0-140-107 kubenswrapper[2578]: W1103 00:40:10.337615 2578 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 03 00:40:10.342222 ip-10-0-140-107 kubenswrapper[2578]: W1103 00:40:10.337618 2578 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 03 00:40:10.342222 ip-10-0-140-107 kubenswrapper[2578]: W1103 00:40:10.337620 2578 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 03 00:40:10.342222 ip-10-0-140-107 kubenswrapper[2578]: W1103 00:40:10.337623 2578 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 03 00:40:10.342222 ip-10-0-140-107 kubenswrapper[2578]: W1103 00:40:10.337626 2578 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 03 00:40:10.342222 ip-10-0-140-107 kubenswrapper[2578]: W1103 00:40:10.337628 2578 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 03 00:40:10.342222 ip-10-0-140-107 kubenswrapper[2578]: W1103 00:40:10.337631 2578 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 03 00:40:10.342222 ip-10-0-140-107 kubenswrapper[2578]: W1103 00:40:10.337633 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 03 00:40:10.342711 ip-10-0-140-107 kubenswrapper[2578]: W1103 00:40:10.337637 2578 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 03 00:40:10.342711 ip-10-0-140-107 kubenswrapper[2578]: W1103 00:40:10.337639 2578 feature_gate.go:386] unrecognized feature gate: Example Nov 03 00:40:10.342711 ip-10-0-140-107 kubenswrapper[2578]: W1103 00:40:10.337642 2578 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 03 00:40:10.342711 ip-10-0-140-107 kubenswrapper[2578]: W1103 00:40:10.337644 2578 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 03 00:40:10.342711 ip-10-0-140-107 kubenswrapper[2578]: W1103 00:40:10.337647 2578 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 03 00:40:10.342711 ip-10-0-140-107 kubenswrapper[2578]: W1103 00:40:10.337651 2578 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 03 00:40:10.342711 ip-10-0-140-107 kubenswrapper[2578]: W1103 00:40:10.337654 2578 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 03 00:40:10.342711 ip-10-0-140-107 kubenswrapper[2578]: W1103 00:40:10.337657 2578 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 03 00:40:10.342711 ip-10-0-140-107 kubenswrapper[2578]: W1103 00:40:10.337659 2578 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 03 00:40:10.342711 ip-10-0-140-107 kubenswrapper[2578]: W1103 00:40:10.337662 2578 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 03 00:40:10.342711 ip-10-0-140-107 kubenswrapper[2578]: W1103 00:40:10.337664 2578 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 03 00:40:10.342711 ip-10-0-140-107 kubenswrapper[2578]: W1103 00:40:10.337667 2578 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 03 00:40:10.342711 ip-10-0-140-107 kubenswrapper[2578]: W1103 00:40:10.337669 2578 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 03 00:40:10.342711 ip-10-0-140-107 kubenswrapper[2578]: W1103 00:40:10.337672 2578 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 03 00:40:10.342711 ip-10-0-140-107 kubenswrapper[2578]: W1103 00:40:10.337675 2578 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 03 00:40:10.342711 ip-10-0-140-107 kubenswrapper[2578]: W1103 00:40:10.337677 2578 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 03 00:40:10.342711 ip-10-0-140-107 kubenswrapper[2578]: W1103 00:40:10.337680 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 03 00:40:10.342711 ip-10-0-140-107 kubenswrapper[2578]: W1103 00:40:10.337683 2578 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 03 00:40:10.342711 ip-10-0-140-107 kubenswrapper[2578]: W1103 00:40:10.337686 2578 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 03 00:40:10.342711 ip-10-0-140-107 kubenswrapper[2578]: W1103 00:40:10.337688 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 03 00:40:10.343201 ip-10-0-140-107 kubenswrapper[2578]: W1103 00:40:10.337694 2578 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 03 00:40:10.343201 ip-10-0-140-107 kubenswrapper[2578]: W1103 00:40:10.337697 2578 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 03 00:40:10.343201 ip-10-0-140-107 kubenswrapper[2578]: W1103 00:40:10.337700 2578 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 03 00:40:10.343201 ip-10-0-140-107 kubenswrapper[2578]: W1103 00:40:10.337702 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 03 00:40:10.343201 ip-10-0-140-107 kubenswrapper[2578]: I1103 00:40:10.338555 2578 flags.go:64] FLAG: --address="0.0.0.0" Nov 03 00:40:10.343201 ip-10-0-140-107 kubenswrapper[2578]: I1103 00:40:10.338565 2578 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]" Nov 03 00:40:10.343201 ip-10-0-140-107 kubenswrapper[2578]: I1103 00:40:10.338572 2578 flags.go:64] FLAG: --anonymous-auth="true" Nov 03 00:40:10.343201 ip-10-0-140-107 kubenswrapper[2578]: I1103 00:40:10.338577 2578 flags.go:64] FLAG: --application-metrics-count-limit="100" Nov 03 00:40:10.343201 ip-10-0-140-107 kubenswrapper[2578]: I1103 00:40:10.338582 2578 flags.go:64] FLAG: --authentication-token-webhook="false" Nov 03 00:40:10.343201 ip-10-0-140-107 kubenswrapper[2578]: I1103 00:40:10.338585 2578 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s" Nov 03 00:40:10.343201 ip-10-0-140-107 kubenswrapper[2578]: I1103 00:40:10.338597 2578 flags.go:64] FLAG: --authorization-mode="AlwaysAllow" Nov 03 00:40:10.343201 ip-10-0-140-107 kubenswrapper[2578]: I1103 00:40:10.338602 2578 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s" Nov 03 00:40:10.343201 ip-10-0-140-107 kubenswrapper[2578]: I1103 00:40:10.338606 2578 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s" Nov 03 00:40:10.343201 ip-10-0-140-107 kubenswrapper[2578]: I1103 00:40:10.338610 2578 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id" Nov 03 00:40:10.343201 ip-10-0-140-107 kubenswrapper[2578]: I1103 00:40:10.338613 2578 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig" Nov 03 00:40:10.343201 ip-10-0-140-107 kubenswrapper[2578]: I1103 00:40:10.338617 2578 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki" Nov 03 00:40:10.343201 ip-10-0-140-107 kubenswrapper[2578]: I1103 00:40:10.338620 2578 flags.go:64] FLAG: --cgroup-driver="cgroupfs" Nov 03 00:40:10.343201 ip-10-0-140-107 kubenswrapper[2578]: I1103 00:40:10.338624 2578 flags.go:64] FLAG: --cgroup-root="" Nov 03 00:40:10.343201 ip-10-0-140-107 kubenswrapper[2578]: I1103 00:40:10.338628 2578 flags.go:64] FLAG: --cgroups-per-qos="true" Nov 03 00:40:10.343201 ip-10-0-140-107 kubenswrapper[2578]: I1103 00:40:10.338631 2578 flags.go:64] FLAG: --client-ca-file="" Nov 03 00:40:10.343201 ip-10-0-140-107 kubenswrapper[2578]: I1103 00:40:10.338634 2578 flags.go:64] FLAG: --cloud-provider="external" Nov 03 00:40:10.343201 ip-10-0-140-107 kubenswrapper[2578]: I1103 00:40:10.338637 2578 flags.go:64] FLAG: --cluster-dns="[]" Nov 03 00:40:10.343201 ip-10-0-140-107 kubenswrapper[2578]: I1103 00:40:10.338641 2578 flags.go:64] FLAG: --cluster-domain="" Nov 03 00:40:10.343741 ip-10-0-140-107 kubenswrapper[2578]: I1103 00:40:10.338644 2578 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf" Nov 03 00:40:10.343741 ip-10-0-140-107 kubenswrapper[2578]: I1103 00:40:10.338647 2578 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d" Nov 03 00:40:10.343741 ip-10-0-140-107 kubenswrapper[2578]: I1103 00:40:10.338650 2578 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json" Nov 03 00:40:10.343741 ip-10-0-140-107 kubenswrapper[2578]: I1103 00:40:10.338653 2578 flags.go:64] FLAG: --container-log-max-files="5" Nov 03 00:40:10.343741 ip-10-0-140-107 kubenswrapper[2578]: I1103 00:40:10.338660 2578 flags.go:64] FLAG: --container-log-max-size="10Mi" Nov 03 00:40:10.343741 ip-10-0-140-107 kubenswrapper[2578]: I1103 00:40:10.338669 2578 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock" Nov 03 00:40:10.343741 ip-10-0-140-107 kubenswrapper[2578]: I1103 00:40:10.338674 2578 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock" Nov 03 00:40:10.343741 ip-10-0-140-107 kubenswrapper[2578]: I1103 00:40:10.338680 2578 flags.go:64] FLAG: --containerd-namespace="k8s.io" Nov 03 00:40:10.343741 ip-10-0-140-107 kubenswrapper[2578]: I1103 00:40:10.338684 2578 flags.go:64] FLAG: --contention-profiling="false" Nov 03 00:40:10.343741 ip-10-0-140-107 kubenswrapper[2578]: I1103 00:40:10.338689 2578 flags.go:64] FLAG: --cpu-cfs-quota="true" Nov 03 00:40:10.343741 ip-10-0-140-107 kubenswrapper[2578]: I1103 00:40:10.338693 2578 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms" Nov 03 00:40:10.343741 ip-10-0-140-107 kubenswrapper[2578]: I1103 00:40:10.338696 2578 flags.go:64] FLAG: --cpu-manager-policy="none" Nov 03 00:40:10.343741 ip-10-0-140-107 kubenswrapper[2578]: I1103 00:40:10.338699 2578 flags.go:64] FLAG: --cpu-manager-policy-options="" Nov 03 00:40:10.343741 ip-10-0-140-107 kubenswrapper[2578]: I1103 00:40:10.338704 2578 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s" Nov 03 00:40:10.343741 ip-10-0-140-107 kubenswrapper[2578]: I1103 00:40:10.338707 2578 flags.go:64] FLAG: --enable-controller-attach-detach="true" Nov 03 00:40:10.343741 ip-10-0-140-107 kubenswrapper[2578]: I1103 00:40:10.338710 2578 flags.go:64] FLAG: --enable-debugging-handlers="true" Nov 03 00:40:10.343741 ip-10-0-140-107 kubenswrapper[2578]: I1103 00:40:10.338712 2578 flags.go:64] FLAG: --enable-load-reader="false" Nov 03 00:40:10.343741 ip-10-0-140-107 kubenswrapper[2578]: I1103 00:40:10.338716 2578 flags.go:64] FLAG: --enable-server="true" Nov 03 00:40:10.343741 ip-10-0-140-107 kubenswrapper[2578]: I1103 00:40:10.338719 2578 flags.go:64] FLAG: --enforce-node-allocatable="[pods]" Nov 03 00:40:10.343741 ip-10-0-140-107 kubenswrapper[2578]: I1103 00:40:10.338723 2578 flags.go:64] FLAG: --event-burst="100" Nov 03 00:40:10.343741 ip-10-0-140-107 kubenswrapper[2578]: I1103 00:40:10.338726 2578 flags.go:64] FLAG: --event-qps="50" Nov 03 00:40:10.343741 ip-10-0-140-107 kubenswrapper[2578]: I1103 00:40:10.338730 2578 flags.go:64] FLAG: --event-storage-age-limit="default=0" Nov 03 00:40:10.343741 ip-10-0-140-107 kubenswrapper[2578]: I1103 00:40:10.338733 2578 flags.go:64] FLAG: --event-storage-event-limit="default=0" Nov 03 00:40:10.343741 ip-10-0-140-107 kubenswrapper[2578]: I1103 00:40:10.338736 2578 flags.go:64] FLAG: --eviction-hard="" Nov 03 00:40:10.344392 ip-10-0-140-107 kubenswrapper[2578]: I1103 00:40:10.338741 2578 flags.go:64] FLAG: --eviction-max-pod-grace-period="0" Nov 03 00:40:10.344392 ip-10-0-140-107 kubenswrapper[2578]: I1103 00:40:10.338744 2578 flags.go:64] FLAG: --eviction-minimum-reclaim="" Nov 03 00:40:10.344392 ip-10-0-140-107 kubenswrapper[2578]: I1103 00:40:10.338748 2578 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s" Nov 03 00:40:10.344392 ip-10-0-140-107 kubenswrapper[2578]: I1103 00:40:10.338752 2578 flags.go:64] FLAG: --eviction-soft="" Nov 03 00:40:10.344392 ip-10-0-140-107 kubenswrapper[2578]: I1103 00:40:10.338755 2578 flags.go:64] FLAG: --eviction-soft-grace-period="" Nov 03 00:40:10.344392 ip-10-0-140-107 kubenswrapper[2578]: I1103 00:40:10.338758 2578 flags.go:64] FLAG: --exit-on-lock-contention="false" Nov 03 00:40:10.344392 ip-10-0-140-107 kubenswrapper[2578]: I1103 00:40:10.338761 2578 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false" Nov 03 00:40:10.344392 ip-10-0-140-107 kubenswrapper[2578]: I1103 00:40:10.338764 2578 flags.go:64] FLAG: --experimental-mounter-path="" Nov 03 00:40:10.344392 ip-10-0-140-107 kubenswrapper[2578]: I1103 00:40:10.338768 2578 flags.go:64] FLAG: --fail-cgroupv1="true" Nov 03 00:40:10.344392 ip-10-0-140-107 kubenswrapper[2578]: I1103 00:40:10.338770 2578 flags.go:64] FLAG: --fail-swap-on="true" Nov 03 00:40:10.344392 ip-10-0-140-107 kubenswrapper[2578]: I1103 00:40:10.338773 2578 flags.go:64] FLAG: --feature-gates="" Nov 03 00:40:10.344392 ip-10-0-140-107 kubenswrapper[2578]: I1103 00:40:10.338777 2578 flags.go:64] FLAG: --file-check-frequency="20s" Nov 03 00:40:10.344392 ip-10-0-140-107 kubenswrapper[2578]: I1103 00:40:10.338781 2578 flags.go:64] FLAG: --global-housekeeping-interval="1m0s" Nov 03 00:40:10.344392 ip-10-0-140-107 kubenswrapper[2578]: I1103 00:40:10.338784 2578 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge" Nov 03 00:40:10.344392 ip-10-0-140-107 kubenswrapper[2578]: I1103 00:40:10.338787 2578 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1" Nov 03 00:40:10.344392 ip-10-0-140-107 kubenswrapper[2578]: I1103 00:40:10.338790 2578 flags.go:64] FLAG: --healthz-port="10248" Nov 03 00:40:10.344392 ip-10-0-140-107 kubenswrapper[2578]: I1103 00:40:10.338794 2578 flags.go:64] FLAG: --help="false" Nov 03 00:40:10.344392 ip-10-0-140-107 kubenswrapper[2578]: I1103 00:40:10.338797 2578 flags.go:64] FLAG: --hostname-override="ip-10-0-140-107.us-east-2.compute.internal" Nov 03 00:40:10.344392 ip-10-0-140-107 kubenswrapper[2578]: I1103 00:40:10.338800 2578 flags.go:64] FLAG: --housekeeping-interval="10s" Nov 03 00:40:10.344392 ip-10-0-140-107 kubenswrapper[2578]: I1103 00:40:10.338804 2578 flags.go:64] FLAG: --http-check-frequency="20s" Nov 03 00:40:10.344392 ip-10-0-140-107 kubenswrapper[2578]: I1103 00:40:10.338807 2578 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins" Nov 03 00:40:10.344392 ip-10-0-140-107 kubenswrapper[2578]: I1103 00:40:10.338811 2578 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml" Nov 03 00:40:10.344392 ip-10-0-140-107 kubenswrapper[2578]: I1103 00:40:10.338814 2578 flags.go:64] FLAG: --image-gc-high-threshold="85" Nov 03 00:40:10.344948 ip-10-0-140-107 kubenswrapper[2578]: I1103 00:40:10.338817 2578 flags.go:64] FLAG: --image-gc-low-threshold="80" Nov 03 00:40:10.344948 ip-10-0-140-107 kubenswrapper[2578]: I1103 00:40:10.338820 2578 flags.go:64] FLAG: --image-service-endpoint="" Nov 03 00:40:10.344948 ip-10-0-140-107 kubenswrapper[2578]: I1103 00:40:10.338823 2578 flags.go:64] FLAG: --kernel-memcg-notification="false" Nov 03 00:40:10.344948 ip-10-0-140-107 kubenswrapper[2578]: I1103 00:40:10.338826 2578 flags.go:64] FLAG: --kube-api-burst="100" Nov 03 00:40:10.344948 ip-10-0-140-107 kubenswrapper[2578]: I1103 00:40:10.338829 2578 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf" Nov 03 00:40:10.344948 ip-10-0-140-107 kubenswrapper[2578]: I1103 00:40:10.338832 2578 flags.go:64] FLAG: --kube-api-qps="50" Nov 03 00:40:10.344948 ip-10-0-140-107 kubenswrapper[2578]: I1103 00:40:10.338835 2578 flags.go:64] FLAG: --kube-reserved="" Nov 03 00:40:10.344948 ip-10-0-140-107 kubenswrapper[2578]: I1103 00:40:10.338838 2578 flags.go:64] FLAG: --kube-reserved-cgroup="" Nov 03 00:40:10.344948 ip-10-0-140-107 kubenswrapper[2578]: I1103 00:40:10.338841 2578 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig" Nov 03 00:40:10.344948 ip-10-0-140-107 kubenswrapper[2578]: I1103 00:40:10.338844 2578 flags.go:64] FLAG: --kubelet-cgroups="" Nov 03 00:40:10.344948 ip-10-0-140-107 kubenswrapper[2578]: I1103 00:40:10.338847 2578 flags.go:64] FLAG: --local-storage-capacity-isolation="true" Nov 03 00:40:10.344948 ip-10-0-140-107 kubenswrapper[2578]: I1103 00:40:10.338851 2578 flags.go:64] FLAG: --lock-file="" Nov 03 00:40:10.344948 ip-10-0-140-107 kubenswrapper[2578]: I1103 00:40:10.338854 2578 flags.go:64] FLAG: --log-cadvisor-usage="false" Nov 03 00:40:10.344948 ip-10-0-140-107 kubenswrapper[2578]: I1103 00:40:10.338857 2578 flags.go:64] FLAG: --log-flush-frequency="5s" Nov 03 00:40:10.344948 ip-10-0-140-107 kubenswrapper[2578]: I1103 00:40:10.338860 2578 flags.go:64] FLAG: --log-json-info-buffer-size="0" Nov 03 00:40:10.344948 ip-10-0-140-107 kubenswrapper[2578]: I1103 00:40:10.338866 2578 flags.go:64] FLAG: --log-json-split-stream="false" Nov 03 00:40:10.344948 ip-10-0-140-107 kubenswrapper[2578]: I1103 00:40:10.338869 2578 flags.go:64] FLAG: --log-text-info-buffer-size="0" Nov 03 00:40:10.344948 ip-10-0-140-107 kubenswrapper[2578]: I1103 00:40:10.338873 2578 flags.go:64] FLAG: --log-text-split-stream="false" Nov 03 00:40:10.344948 ip-10-0-140-107 kubenswrapper[2578]: I1103 00:40:10.338876 2578 flags.go:64] FLAG: --logging-format="text" Nov 03 00:40:10.344948 ip-10-0-140-107 kubenswrapper[2578]: I1103 00:40:10.338879 2578 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id" Nov 03 00:40:10.344948 ip-10-0-140-107 kubenswrapper[2578]: I1103 00:40:10.338882 2578 flags.go:64] FLAG: --make-iptables-util-chains="true" Nov 03 00:40:10.344948 ip-10-0-140-107 kubenswrapper[2578]: I1103 00:40:10.338885 2578 flags.go:64] FLAG: --manifest-url="" Nov 03 00:40:10.344948 ip-10-0-140-107 kubenswrapper[2578]: I1103 00:40:10.338888 2578 flags.go:64] FLAG: --manifest-url-header="" Nov 03 00:40:10.344948 ip-10-0-140-107 kubenswrapper[2578]: I1103 00:40:10.338893 2578 flags.go:64] FLAG: --max-housekeeping-interval="15s" Nov 03 00:40:10.344948 ip-10-0-140-107 kubenswrapper[2578]: I1103 00:40:10.338897 2578 flags.go:64] FLAG: --max-open-files="1000000" Nov 03 00:40:10.345567 ip-10-0-140-107 kubenswrapper[2578]: I1103 00:40:10.338901 2578 flags.go:64] FLAG: --max-pods="110" Nov 03 00:40:10.345567 ip-10-0-140-107 kubenswrapper[2578]: I1103 00:40:10.338915 2578 flags.go:64] FLAG: --maximum-dead-containers="-1" Nov 03 00:40:10.345567 ip-10-0-140-107 kubenswrapper[2578]: I1103 00:40:10.338918 2578 flags.go:64] FLAG: --maximum-dead-containers-per-container="1" Nov 03 00:40:10.345567 ip-10-0-140-107 kubenswrapper[2578]: I1103 00:40:10.338921 2578 flags.go:64] FLAG: --memory-manager-policy="None" Nov 03 00:40:10.345567 ip-10-0-140-107 kubenswrapper[2578]: I1103 00:40:10.338925 2578 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s" Nov 03 00:40:10.345567 ip-10-0-140-107 kubenswrapper[2578]: I1103 00:40:10.338929 2578 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s" Nov 03 00:40:10.345567 ip-10-0-140-107 kubenswrapper[2578]: I1103 00:40:10.338932 2578 flags.go:64] FLAG: --node-ip="0.0.0.0" Nov 03 00:40:10.345567 ip-10-0-140-107 kubenswrapper[2578]: I1103 00:40:10.338935 2578 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel" Nov 03 00:40:10.345567 ip-10-0-140-107 kubenswrapper[2578]: I1103 00:40:10.338943 2578 flags.go:64] FLAG: --node-status-max-images="50" Nov 03 00:40:10.345567 ip-10-0-140-107 kubenswrapper[2578]: I1103 00:40:10.338946 2578 flags.go:64] FLAG: --node-status-update-frequency="10s" Nov 03 00:40:10.345567 ip-10-0-140-107 kubenswrapper[2578]: I1103 00:40:10.338949 2578 flags.go:64] FLAG: --oom-score-adj="-999" Nov 03 00:40:10.345567 ip-10-0-140-107 kubenswrapper[2578]: I1103 00:40:10.338952 2578 flags.go:64] FLAG: --pod-cidr="" Nov 03 00:40:10.345567 ip-10-0-140-107 kubenswrapper[2578]: I1103 00:40:10.338957 2578 flags.go:64] FLAG: --pod-manifest-path="" Nov 03 00:40:10.345567 ip-10-0-140-107 kubenswrapper[2578]: I1103 00:40:10.338960 2578 flags.go:64] FLAG: --pod-max-pids="-1" Nov 03 00:40:10.345567 ip-10-0-140-107 kubenswrapper[2578]: I1103 00:40:10.338963 2578 flags.go:64] FLAG: --pods-per-core="0" Nov 03 00:40:10.345567 ip-10-0-140-107 kubenswrapper[2578]: I1103 00:40:10.338967 2578 flags.go:64] FLAG: --port="10250" Nov 03 00:40:10.345567 ip-10-0-140-107 kubenswrapper[2578]: I1103 00:40:10.338970 2578 flags.go:64] FLAG: --protect-kernel-defaults="false" Nov 03 00:40:10.345567 ip-10-0-140-107 kubenswrapper[2578]: I1103 00:40:10.338973 2578 flags.go:64] FLAG: --provider-id="aws:///us-east-2a/i-02caf7f9962a007b4" Nov 03 00:40:10.345567 ip-10-0-140-107 kubenswrapper[2578]: I1103 00:40:10.338977 2578 flags.go:64] FLAG: --qos-reserved="" Nov 03 00:40:10.345567 ip-10-0-140-107 kubenswrapper[2578]: I1103 00:40:10.338980 2578 flags.go:64] FLAG: --read-only-port="10255" Nov 03 00:40:10.345567 ip-10-0-140-107 kubenswrapper[2578]: I1103 00:40:10.338983 2578 flags.go:64] FLAG: --register-node="true" Nov 03 00:40:10.345567 ip-10-0-140-107 kubenswrapper[2578]: I1103 00:40:10.338986 2578 flags.go:64] FLAG: --register-with-taints="" Nov 03 00:40:10.345567 ip-10-0-140-107 kubenswrapper[2578]: I1103 00:40:10.338990 2578 flags.go:64] FLAG: --registry-burst="10" Nov 03 00:40:10.345567 ip-10-0-140-107 kubenswrapper[2578]: I1103 00:40:10.338993 2578 flags.go:64] FLAG: --registry-qps="5" Nov 03 00:40:10.345567 ip-10-0-140-107 kubenswrapper[2578]: I1103 00:40:10.338996 2578 flags.go:64] FLAG: --reserved-cpus="" Nov 03 00:40:10.345567 ip-10-0-140-107 kubenswrapper[2578]: I1103 00:40:10.338999 2578 flags.go:64] FLAG: --reserved-memory="" Nov 03 00:40:10.346418 ip-10-0-140-107 kubenswrapper[2578]: I1103 00:40:10.339003 2578 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf" Nov 03 00:40:10.346418 ip-10-0-140-107 kubenswrapper[2578]: I1103 00:40:10.339006 2578 flags.go:64] FLAG: --root-dir="/var/lib/kubelet" Nov 03 00:40:10.346418 ip-10-0-140-107 kubenswrapper[2578]: I1103 00:40:10.339010 2578 flags.go:64] FLAG: --rotate-certificates="false" Nov 03 00:40:10.346418 ip-10-0-140-107 kubenswrapper[2578]: I1103 00:40:10.339013 2578 flags.go:64] FLAG: --rotate-server-certificates="false" Nov 03 00:40:10.346418 ip-10-0-140-107 kubenswrapper[2578]: I1103 00:40:10.339016 2578 flags.go:64] FLAG: --runonce="false" Nov 03 00:40:10.346418 ip-10-0-140-107 kubenswrapper[2578]: I1103 00:40:10.339019 2578 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service" Nov 03 00:40:10.346418 ip-10-0-140-107 kubenswrapper[2578]: I1103 00:40:10.339022 2578 flags.go:64] FLAG: --runtime-request-timeout="2m0s" Nov 03 00:40:10.346418 ip-10-0-140-107 kubenswrapper[2578]: I1103 00:40:10.339025 2578 flags.go:64] FLAG: --seccomp-default="false" Nov 03 00:40:10.346418 ip-10-0-140-107 kubenswrapper[2578]: I1103 00:40:10.339028 2578 flags.go:64] FLAG: --serialize-image-pulls="true" Nov 03 00:40:10.346418 ip-10-0-140-107 kubenswrapper[2578]: I1103 00:40:10.339031 2578 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s" Nov 03 00:40:10.346418 ip-10-0-140-107 kubenswrapper[2578]: I1103 00:40:10.339034 2578 flags.go:64] FLAG: --storage-driver-db="cadvisor" Nov 03 00:40:10.346418 ip-10-0-140-107 kubenswrapper[2578]: I1103 00:40:10.339038 2578 flags.go:64] FLAG: --storage-driver-host="localhost:8086" Nov 03 00:40:10.346418 ip-10-0-140-107 kubenswrapper[2578]: I1103 00:40:10.339041 2578 flags.go:64] FLAG: --storage-driver-password="root" Nov 03 00:40:10.346418 ip-10-0-140-107 kubenswrapper[2578]: I1103 00:40:10.339045 2578 flags.go:64] FLAG: --storage-driver-secure="false" Nov 03 00:40:10.346418 ip-10-0-140-107 kubenswrapper[2578]: I1103 00:40:10.339048 2578 flags.go:64] FLAG: --storage-driver-table="stats" Nov 03 00:40:10.346418 ip-10-0-140-107 kubenswrapper[2578]: I1103 00:40:10.339051 2578 flags.go:64] FLAG: --storage-driver-user="root" Nov 03 00:40:10.346418 ip-10-0-140-107 kubenswrapper[2578]: I1103 00:40:10.339054 2578 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s" Nov 03 00:40:10.346418 ip-10-0-140-107 kubenswrapper[2578]: I1103 00:40:10.339057 2578 flags.go:64] FLAG: --sync-frequency="1m0s" Nov 03 00:40:10.346418 ip-10-0-140-107 kubenswrapper[2578]: I1103 00:40:10.339062 2578 flags.go:64] FLAG: --system-cgroups="" Nov 03 00:40:10.346418 ip-10-0-140-107 kubenswrapper[2578]: I1103 00:40:10.339064 2578 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi" Nov 03 00:40:10.346418 ip-10-0-140-107 kubenswrapper[2578]: I1103 00:40:10.339070 2578 flags.go:64] FLAG: --system-reserved-cgroup="" Nov 03 00:40:10.346418 ip-10-0-140-107 kubenswrapper[2578]: I1103 00:40:10.339073 2578 flags.go:64] FLAG: --tls-cert-file="" Nov 03 00:40:10.346418 ip-10-0-140-107 kubenswrapper[2578]: I1103 00:40:10.339076 2578 flags.go:64] FLAG: --tls-cipher-suites="[]" Nov 03 00:40:10.346418 ip-10-0-140-107 kubenswrapper[2578]: I1103 00:40:10.339079 2578 flags.go:64] FLAG: --tls-min-version="" Nov 03 00:40:10.346418 ip-10-0-140-107 kubenswrapper[2578]: I1103 00:40:10.339082 2578 flags.go:64] FLAG: --tls-private-key-file="" Nov 03 00:40:10.347257 ip-10-0-140-107 kubenswrapper[2578]: I1103 00:40:10.339085 2578 flags.go:64] FLAG: --topology-manager-policy="none" Nov 03 00:40:10.347257 ip-10-0-140-107 kubenswrapper[2578]: I1103 00:40:10.339088 2578 flags.go:64] FLAG: --topology-manager-policy-options="" Nov 03 00:40:10.347257 ip-10-0-140-107 kubenswrapper[2578]: I1103 00:40:10.339091 2578 flags.go:64] FLAG: --topology-manager-scope="container" Nov 03 00:40:10.347257 ip-10-0-140-107 kubenswrapper[2578]: I1103 00:40:10.339094 2578 flags.go:64] FLAG: --v="2" Nov 03 00:40:10.347257 ip-10-0-140-107 kubenswrapper[2578]: I1103 00:40:10.339099 2578 flags.go:64] FLAG: --version="false" Nov 03 00:40:10.347257 ip-10-0-140-107 kubenswrapper[2578]: I1103 00:40:10.339103 2578 flags.go:64] FLAG: --vmodule="" Nov 03 00:40:10.347257 ip-10-0-140-107 kubenswrapper[2578]: I1103 00:40:10.339107 2578 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec" Nov 03 00:40:10.347257 ip-10-0-140-107 kubenswrapper[2578]: I1103 00:40:10.339110 2578 flags.go:64] FLAG: --volume-stats-agg-period="1m0s" Nov 03 00:40:10.347257 ip-10-0-140-107 kubenswrapper[2578]: W1103 00:40:10.339250 2578 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 03 00:40:10.347257 ip-10-0-140-107 kubenswrapper[2578]: W1103 00:40:10.339254 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 03 00:40:10.347257 ip-10-0-140-107 kubenswrapper[2578]: W1103 00:40:10.339257 2578 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 03 00:40:10.347257 ip-10-0-140-107 kubenswrapper[2578]: W1103 00:40:10.339260 2578 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 03 00:40:10.347257 ip-10-0-140-107 kubenswrapper[2578]: W1103 00:40:10.339263 2578 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 03 00:40:10.347257 ip-10-0-140-107 kubenswrapper[2578]: W1103 00:40:10.339265 2578 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 03 00:40:10.347257 ip-10-0-140-107 kubenswrapper[2578]: W1103 00:40:10.339268 2578 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 03 00:40:10.347257 ip-10-0-140-107 kubenswrapper[2578]: W1103 00:40:10.339270 2578 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 03 00:40:10.347257 ip-10-0-140-107 kubenswrapper[2578]: W1103 00:40:10.339273 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 03 00:40:10.347257 ip-10-0-140-107 kubenswrapper[2578]: W1103 00:40:10.339276 2578 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 03 00:40:10.347257 ip-10-0-140-107 kubenswrapper[2578]: W1103 00:40:10.339278 2578 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 03 00:40:10.347257 ip-10-0-140-107 kubenswrapper[2578]: W1103 00:40:10.339281 2578 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 03 00:40:10.347257 ip-10-0-140-107 kubenswrapper[2578]: W1103 00:40:10.339284 2578 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 03 00:40:10.347750 ip-10-0-140-107 kubenswrapper[2578]: W1103 00:40:10.339287 2578 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 03 00:40:10.347750 ip-10-0-140-107 kubenswrapper[2578]: W1103 00:40:10.339290 2578 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 03 00:40:10.347750 ip-10-0-140-107 kubenswrapper[2578]: W1103 00:40:10.339292 2578 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 03 00:40:10.347750 ip-10-0-140-107 kubenswrapper[2578]: W1103 00:40:10.339295 2578 feature_gate.go:386] unrecognized feature gate: EVPN Nov 03 00:40:10.347750 ip-10-0-140-107 kubenswrapper[2578]: W1103 00:40:10.339300 2578 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 03 00:40:10.347750 ip-10-0-140-107 kubenswrapper[2578]: W1103 00:40:10.339302 2578 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 03 00:40:10.347750 ip-10-0-140-107 kubenswrapper[2578]: W1103 00:40:10.339305 2578 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 03 00:40:10.347750 ip-10-0-140-107 kubenswrapper[2578]: W1103 00:40:10.339307 2578 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 03 00:40:10.347750 ip-10-0-140-107 kubenswrapper[2578]: W1103 00:40:10.339310 2578 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 03 00:40:10.347750 ip-10-0-140-107 kubenswrapper[2578]: W1103 00:40:10.339312 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 03 00:40:10.347750 ip-10-0-140-107 kubenswrapper[2578]: W1103 00:40:10.339315 2578 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 03 00:40:10.347750 ip-10-0-140-107 kubenswrapper[2578]: W1103 00:40:10.339318 2578 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 03 00:40:10.347750 ip-10-0-140-107 kubenswrapper[2578]: W1103 00:40:10.339320 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 03 00:40:10.347750 ip-10-0-140-107 kubenswrapper[2578]: W1103 00:40:10.339323 2578 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 03 00:40:10.347750 ip-10-0-140-107 kubenswrapper[2578]: W1103 00:40:10.339325 2578 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 03 00:40:10.347750 ip-10-0-140-107 kubenswrapper[2578]: W1103 00:40:10.339328 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 03 00:40:10.347750 ip-10-0-140-107 kubenswrapper[2578]: W1103 00:40:10.339331 2578 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 03 00:40:10.347750 ip-10-0-140-107 kubenswrapper[2578]: W1103 00:40:10.339333 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 03 00:40:10.347750 ip-10-0-140-107 kubenswrapper[2578]: W1103 00:40:10.339336 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 03 00:40:10.347750 ip-10-0-140-107 kubenswrapper[2578]: W1103 00:40:10.339338 2578 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 03 00:40:10.348241 ip-10-0-140-107 kubenswrapper[2578]: W1103 00:40:10.339341 2578 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 03 00:40:10.348241 ip-10-0-140-107 kubenswrapper[2578]: W1103 00:40:10.339344 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 03 00:40:10.348241 ip-10-0-140-107 kubenswrapper[2578]: W1103 00:40:10.339346 2578 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 03 00:40:10.348241 ip-10-0-140-107 kubenswrapper[2578]: W1103 00:40:10.339349 2578 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 03 00:40:10.348241 ip-10-0-140-107 kubenswrapper[2578]: W1103 00:40:10.339351 2578 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 03 00:40:10.348241 ip-10-0-140-107 kubenswrapper[2578]: W1103 00:40:10.339354 2578 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 03 00:40:10.348241 ip-10-0-140-107 kubenswrapper[2578]: W1103 00:40:10.339357 2578 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 03 00:40:10.348241 ip-10-0-140-107 kubenswrapper[2578]: W1103 00:40:10.339360 2578 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 03 00:40:10.348241 ip-10-0-140-107 kubenswrapper[2578]: W1103 00:40:10.339362 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 03 00:40:10.348241 ip-10-0-140-107 kubenswrapper[2578]: W1103 00:40:10.339365 2578 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 03 00:40:10.348241 ip-10-0-140-107 kubenswrapper[2578]: W1103 00:40:10.339367 2578 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 03 00:40:10.348241 ip-10-0-140-107 kubenswrapper[2578]: W1103 00:40:10.339370 2578 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 03 00:40:10.348241 ip-10-0-140-107 kubenswrapper[2578]: W1103 00:40:10.339373 2578 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 03 00:40:10.348241 ip-10-0-140-107 kubenswrapper[2578]: W1103 00:40:10.339375 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 03 00:40:10.348241 ip-10-0-140-107 kubenswrapper[2578]: W1103 00:40:10.339378 2578 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 03 00:40:10.348241 ip-10-0-140-107 kubenswrapper[2578]: W1103 00:40:10.339381 2578 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 03 00:40:10.348241 ip-10-0-140-107 kubenswrapper[2578]: W1103 00:40:10.339385 2578 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 03 00:40:10.348241 ip-10-0-140-107 kubenswrapper[2578]: W1103 00:40:10.339388 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 03 00:40:10.348241 ip-10-0-140-107 kubenswrapper[2578]: W1103 00:40:10.339391 2578 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 03 00:40:10.348241 ip-10-0-140-107 kubenswrapper[2578]: W1103 00:40:10.339393 2578 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 03 00:40:10.348714 ip-10-0-140-107 kubenswrapper[2578]: W1103 00:40:10.339396 2578 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 03 00:40:10.348714 ip-10-0-140-107 kubenswrapper[2578]: W1103 00:40:10.339399 2578 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 03 00:40:10.348714 ip-10-0-140-107 kubenswrapper[2578]: W1103 00:40:10.339401 2578 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 03 00:40:10.348714 ip-10-0-140-107 kubenswrapper[2578]: W1103 00:40:10.339404 2578 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 03 00:40:10.348714 ip-10-0-140-107 kubenswrapper[2578]: W1103 00:40:10.339407 2578 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 03 00:40:10.348714 ip-10-0-140-107 kubenswrapper[2578]: W1103 00:40:10.339409 2578 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 03 00:40:10.348714 ip-10-0-140-107 kubenswrapper[2578]: W1103 00:40:10.339412 2578 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 03 00:40:10.348714 ip-10-0-140-107 kubenswrapper[2578]: W1103 00:40:10.339414 2578 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 03 00:40:10.348714 ip-10-0-140-107 kubenswrapper[2578]: W1103 00:40:10.339417 2578 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 03 00:40:10.348714 ip-10-0-140-107 kubenswrapper[2578]: W1103 00:40:10.339420 2578 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 03 00:40:10.348714 ip-10-0-140-107 kubenswrapper[2578]: W1103 00:40:10.339423 2578 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 03 00:40:10.348714 ip-10-0-140-107 kubenswrapper[2578]: W1103 00:40:10.339425 2578 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 03 00:40:10.348714 ip-10-0-140-107 kubenswrapper[2578]: W1103 00:40:10.339428 2578 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 03 00:40:10.348714 ip-10-0-140-107 kubenswrapper[2578]: W1103 00:40:10.339431 2578 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 03 00:40:10.348714 ip-10-0-140-107 kubenswrapper[2578]: W1103 00:40:10.339434 2578 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 03 00:40:10.348714 ip-10-0-140-107 kubenswrapper[2578]: W1103 00:40:10.339436 2578 feature_gate.go:386] unrecognized feature gate: Example2 Nov 03 00:40:10.348714 ip-10-0-140-107 kubenswrapper[2578]: W1103 00:40:10.339439 2578 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 03 00:40:10.348714 ip-10-0-140-107 kubenswrapper[2578]: W1103 00:40:10.339443 2578 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 03 00:40:10.348714 ip-10-0-140-107 kubenswrapper[2578]: W1103 00:40:10.339447 2578 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 03 00:40:10.349164 ip-10-0-140-107 kubenswrapper[2578]: W1103 00:40:10.339451 2578 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 03 00:40:10.349164 ip-10-0-140-107 kubenswrapper[2578]: W1103 00:40:10.339453 2578 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 03 00:40:10.349164 ip-10-0-140-107 kubenswrapper[2578]: W1103 00:40:10.339456 2578 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 03 00:40:10.349164 ip-10-0-140-107 kubenswrapper[2578]: W1103 00:40:10.339459 2578 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 03 00:40:10.349164 ip-10-0-140-107 kubenswrapper[2578]: W1103 00:40:10.339462 2578 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 03 00:40:10.349164 ip-10-0-140-107 kubenswrapper[2578]: W1103 00:40:10.339465 2578 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 03 00:40:10.349164 ip-10-0-140-107 kubenswrapper[2578]: W1103 00:40:10.339470 2578 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 03 00:40:10.349164 ip-10-0-140-107 kubenswrapper[2578]: W1103 00:40:10.339473 2578 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 03 00:40:10.349164 ip-10-0-140-107 kubenswrapper[2578]: W1103 00:40:10.339475 2578 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 03 00:40:10.349164 ip-10-0-140-107 kubenswrapper[2578]: W1103 00:40:10.339479 2578 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 03 00:40:10.349164 ip-10-0-140-107 kubenswrapper[2578]: W1103 00:40:10.339482 2578 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 03 00:40:10.349164 ip-10-0-140-107 kubenswrapper[2578]: W1103 00:40:10.339485 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 03 00:40:10.349164 ip-10-0-140-107 kubenswrapper[2578]: W1103 00:40:10.339488 2578 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 03 00:40:10.349164 ip-10-0-140-107 kubenswrapper[2578]: W1103 00:40:10.339490 2578 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 03 00:40:10.349164 ip-10-0-140-107 kubenswrapper[2578]: W1103 00:40:10.339493 2578 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 03 00:40:10.349164 ip-10-0-140-107 kubenswrapper[2578]: W1103 00:40:10.339496 2578 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 03 00:40:10.349164 ip-10-0-140-107 kubenswrapper[2578]: W1103 00:40:10.339498 2578 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 03 00:40:10.349164 ip-10-0-140-107 kubenswrapper[2578]: W1103 00:40:10.339501 2578 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 03 00:40:10.349164 ip-10-0-140-107 kubenswrapper[2578]: W1103 00:40:10.339504 2578 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 03 00:40:10.349605 ip-10-0-140-107 kubenswrapper[2578]: W1103 00:40:10.339507 2578 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 03 00:40:10.349605 ip-10-0-140-107 kubenswrapper[2578]: W1103 00:40:10.339510 2578 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 03 00:40:10.349605 ip-10-0-140-107 kubenswrapper[2578]: W1103 00:40:10.339513 2578 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 03 00:40:10.349605 ip-10-0-140-107 kubenswrapper[2578]: W1103 00:40:10.339515 2578 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 03 00:40:10.349605 ip-10-0-140-107 kubenswrapper[2578]: W1103 00:40:10.339518 2578 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 03 00:40:10.349605 ip-10-0-140-107 kubenswrapper[2578]: W1103 00:40:10.339522 2578 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 03 00:40:10.349605 ip-10-0-140-107 kubenswrapper[2578]: W1103 00:40:10.339526 2578 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 03 00:40:10.349605 ip-10-0-140-107 kubenswrapper[2578]: W1103 00:40:10.339529 2578 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 03 00:40:10.349605 ip-10-0-140-107 kubenswrapper[2578]: W1103 00:40:10.339532 2578 feature_gate.go:386] unrecognized feature gate: Example Nov 03 00:40:10.349605 ip-10-0-140-107 kubenswrapper[2578]: W1103 00:40:10.339534 2578 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 03 00:40:10.349605 ip-10-0-140-107 kubenswrapper[2578]: W1103 00:40:10.339537 2578 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 03 00:40:10.349605 ip-10-0-140-107 kubenswrapper[2578]: W1103 00:40:10.339540 2578 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 03 00:40:10.349605 ip-10-0-140-107 kubenswrapper[2578]: W1103 00:40:10.339542 2578 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 03 00:40:10.349605 ip-10-0-140-107 kubenswrapper[2578]: W1103 00:40:10.339545 2578 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 03 00:40:10.349605 ip-10-0-140-107 kubenswrapper[2578]: W1103 00:40:10.339548 2578 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 03 00:40:10.349605 ip-10-0-140-107 kubenswrapper[2578]: I1103 00:40:10.339574 2578 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Nov 03 00:40:10.350006 ip-10-0-140-107 kubenswrapper[2578]: W1103 00:40:10.339609 2578 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 03 00:40:10.350006 ip-10-0-140-107 kubenswrapper[2578]: W1103 00:40:10.339613 2578 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 03 00:40:10.350006 ip-10-0-140-107 kubenswrapper[2578]: W1103 00:40:10.339617 2578 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 03 00:40:10.350006 ip-10-0-140-107 kubenswrapper[2578]: W1103 00:40:10.339620 2578 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 03 00:40:10.350006 ip-10-0-140-107 kubenswrapper[2578]: W1103 00:40:10.339625 2578 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 03 00:40:10.350006 ip-10-0-140-107 kubenswrapper[2578]: W1103 00:40:10.339628 2578 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 03 00:40:10.350006 ip-10-0-140-107 kubenswrapper[2578]: W1103 00:40:10.339633 2578 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 03 00:40:10.350006 ip-10-0-140-107 kubenswrapper[2578]: W1103 00:40:10.339635 2578 feature_gate.go:386] unrecognized feature gate: EVPN Nov 03 00:40:10.350006 ip-10-0-140-107 kubenswrapper[2578]: W1103 00:40:10.339639 2578 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 03 00:40:10.350006 ip-10-0-140-107 kubenswrapper[2578]: W1103 00:40:10.339641 2578 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 03 00:40:10.350006 ip-10-0-140-107 kubenswrapper[2578]: W1103 00:40:10.339644 2578 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 03 00:40:10.350006 ip-10-0-140-107 kubenswrapper[2578]: W1103 00:40:10.339647 2578 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 03 00:40:10.350006 ip-10-0-140-107 kubenswrapper[2578]: W1103 00:40:10.339649 2578 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 03 00:40:10.350006 ip-10-0-140-107 kubenswrapper[2578]: W1103 00:40:10.339652 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 03 00:40:10.350006 ip-10-0-140-107 kubenswrapper[2578]: W1103 00:40:10.339655 2578 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 03 00:40:10.350006 ip-10-0-140-107 kubenswrapper[2578]: W1103 00:40:10.339657 2578 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 03 00:40:10.350006 ip-10-0-140-107 kubenswrapper[2578]: W1103 00:40:10.339660 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 03 00:40:10.350006 ip-10-0-140-107 kubenswrapper[2578]: W1103 00:40:10.339663 2578 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 03 00:40:10.350006 ip-10-0-140-107 kubenswrapper[2578]: W1103 00:40:10.339665 2578 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 03 00:40:10.350006 ip-10-0-140-107 kubenswrapper[2578]: W1103 00:40:10.339668 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 03 00:40:10.350479 ip-10-0-140-107 kubenswrapper[2578]: W1103 00:40:10.339670 2578 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 03 00:40:10.350479 ip-10-0-140-107 kubenswrapper[2578]: W1103 00:40:10.339673 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 03 00:40:10.350479 ip-10-0-140-107 kubenswrapper[2578]: W1103 00:40:10.339676 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 03 00:40:10.350479 ip-10-0-140-107 kubenswrapper[2578]: W1103 00:40:10.339678 2578 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 03 00:40:10.350479 ip-10-0-140-107 kubenswrapper[2578]: W1103 00:40:10.339681 2578 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 03 00:40:10.350479 ip-10-0-140-107 kubenswrapper[2578]: W1103 00:40:10.339684 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 03 00:40:10.350479 ip-10-0-140-107 kubenswrapper[2578]: W1103 00:40:10.339687 2578 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 03 00:40:10.350479 ip-10-0-140-107 kubenswrapper[2578]: W1103 00:40:10.339689 2578 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 03 00:40:10.350479 ip-10-0-140-107 kubenswrapper[2578]: W1103 00:40:10.339692 2578 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 03 00:40:10.350479 ip-10-0-140-107 kubenswrapper[2578]: W1103 00:40:10.339695 2578 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 03 00:40:10.350479 ip-10-0-140-107 kubenswrapper[2578]: W1103 00:40:10.339698 2578 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 03 00:40:10.350479 ip-10-0-140-107 kubenswrapper[2578]: W1103 00:40:10.339700 2578 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 03 00:40:10.350479 ip-10-0-140-107 kubenswrapper[2578]: W1103 00:40:10.339703 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 03 00:40:10.350479 ip-10-0-140-107 kubenswrapper[2578]: W1103 00:40:10.339705 2578 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 03 00:40:10.350479 ip-10-0-140-107 kubenswrapper[2578]: W1103 00:40:10.339708 2578 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 03 00:40:10.350479 ip-10-0-140-107 kubenswrapper[2578]: W1103 00:40:10.339711 2578 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 03 00:40:10.350479 ip-10-0-140-107 kubenswrapper[2578]: W1103 00:40:10.339715 2578 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 03 00:40:10.350479 ip-10-0-140-107 kubenswrapper[2578]: W1103 00:40:10.339718 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 03 00:40:10.350479 ip-10-0-140-107 kubenswrapper[2578]: W1103 00:40:10.339722 2578 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 03 00:40:10.350479 ip-10-0-140-107 kubenswrapper[2578]: W1103 00:40:10.339725 2578 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 03 00:40:10.350981 ip-10-0-140-107 kubenswrapper[2578]: W1103 00:40:10.339727 2578 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 03 00:40:10.350981 ip-10-0-140-107 kubenswrapper[2578]: W1103 00:40:10.339730 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 03 00:40:10.350981 ip-10-0-140-107 kubenswrapper[2578]: W1103 00:40:10.339733 2578 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 03 00:40:10.350981 ip-10-0-140-107 kubenswrapper[2578]: W1103 00:40:10.339736 2578 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 03 00:40:10.350981 ip-10-0-140-107 kubenswrapper[2578]: W1103 00:40:10.339739 2578 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 03 00:40:10.350981 ip-10-0-140-107 kubenswrapper[2578]: W1103 00:40:10.339741 2578 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 03 00:40:10.350981 ip-10-0-140-107 kubenswrapper[2578]: W1103 00:40:10.339744 2578 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 03 00:40:10.350981 ip-10-0-140-107 kubenswrapper[2578]: W1103 00:40:10.339747 2578 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 03 00:40:10.350981 ip-10-0-140-107 kubenswrapper[2578]: W1103 00:40:10.339749 2578 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 03 00:40:10.350981 ip-10-0-140-107 kubenswrapper[2578]: W1103 00:40:10.339752 2578 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 03 00:40:10.350981 ip-10-0-140-107 kubenswrapper[2578]: W1103 00:40:10.339755 2578 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 03 00:40:10.350981 ip-10-0-140-107 kubenswrapper[2578]: W1103 00:40:10.339757 2578 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 03 00:40:10.350981 ip-10-0-140-107 kubenswrapper[2578]: W1103 00:40:10.339760 2578 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 03 00:40:10.350981 ip-10-0-140-107 kubenswrapper[2578]: W1103 00:40:10.339763 2578 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 03 00:40:10.350981 ip-10-0-140-107 kubenswrapper[2578]: W1103 00:40:10.339765 2578 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 03 00:40:10.350981 ip-10-0-140-107 kubenswrapper[2578]: W1103 00:40:10.339768 2578 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 03 00:40:10.350981 ip-10-0-140-107 kubenswrapper[2578]: W1103 00:40:10.339771 2578 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 03 00:40:10.350981 ip-10-0-140-107 kubenswrapper[2578]: W1103 00:40:10.339774 2578 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 03 00:40:10.350981 ip-10-0-140-107 kubenswrapper[2578]: W1103 00:40:10.339776 2578 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 03 00:40:10.350981 ip-10-0-140-107 kubenswrapper[2578]: W1103 00:40:10.339779 2578 feature_gate.go:386] unrecognized feature gate: Example2 Nov 03 00:40:10.351476 ip-10-0-140-107 kubenswrapper[2578]: W1103 00:40:10.339782 2578 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 03 00:40:10.351476 ip-10-0-140-107 kubenswrapper[2578]: W1103 00:40:10.339786 2578 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 03 00:40:10.351476 ip-10-0-140-107 kubenswrapper[2578]: W1103 00:40:10.339790 2578 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 03 00:40:10.351476 ip-10-0-140-107 kubenswrapper[2578]: W1103 00:40:10.339793 2578 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 03 00:40:10.351476 ip-10-0-140-107 kubenswrapper[2578]: W1103 00:40:10.339796 2578 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 03 00:40:10.351476 ip-10-0-140-107 kubenswrapper[2578]: W1103 00:40:10.339799 2578 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 03 00:40:10.351476 ip-10-0-140-107 kubenswrapper[2578]: W1103 00:40:10.339802 2578 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 03 00:40:10.351476 ip-10-0-140-107 kubenswrapper[2578]: W1103 00:40:10.339805 2578 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 03 00:40:10.351476 ip-10-0-140-107 kubenswrapper[2578]: W1103 00:40:10.339808 2578 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 03 00:40:10.351476 ip-10-0-140-107 kubenswrapper[2578]: W1103 00:40:10.339812 2578 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 03 00:40:10.351476 ip-10-0-140-107 kubenswrapper[2578]: W1103 00:40:10.339815 2578 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 03 00:40:10.351476 ip-10-0-140-107 kubenswrapper[2578]: W1103 00:40:10.339819 2578 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 03 00:40:10.351476 ip-10-0-140-107 kubenswrapper[2578]: W1103 00:40:10.339821 2578 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 03 00:40:10.351476 ip-10-0-140-107 kubenswrapper[2578]: W1103 00:40:10.339824 2578 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 03 00:40:10.351476 ip-10-0-140-107 kubenswrapper[2578]: W1103 00:40:10.339827 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 03 00:40:10.351476 ip-10-0-140-107 kubenswrapper[2578]: W1103 00:40:10.339829 2578 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 03 00:40:10.351476 ip-10-0-140-107 kubenswrapper[2578]: W1103 00:40:10.339832 2578 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 03 00:40:10.351476 ip-10-0-140-107 kubenswrapper[2578]: W1103 00:40:10.339835 2578 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 03 00:40:10.351476 ip-10-0-140-107 kubenswrapper[2578]: W1103 00:40:10.339837 2578 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 03 00:40:10.352017 ip-10-0-140-107 kubenswrapper[2578]: W1103 00:40:10.339840 2578 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 03 00:40:10.352017 ip-10-0-140-107 kubenswrapper[2578]: W1103 00:40:10.339843 2578 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 03 00:40:10.352017 ip-10-0-140-107 kubenswrapper[2578]: W1103 00:40:10.339845 2578 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 03 00:40:10.352017 ip-10-0-140-107 kubenswrapper[2578]: W1103 00:40:10.339848 2578 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 03 00:40:10.352017 ip-10-0-140-107 kubenswrapper[2578]: W1103 00:40:10.339851 2578 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 03 00:40:10.352017 ip-10-0-140-107 kubenswrapper[2578]: W1103 00:40:10.339853 2578 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 03 00:40:10.352017 ip-10-0-140-107 kubenswrapper[2578]: W1103 00:40:10.339856 2578 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 03 00:40:10.352017 ip-10-0-140-107 kubenswrapper[2578]: W1103 00:40:10.339859 2578 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 03 00:40:10.352017 ip-10-0-140-107 kubenswrapper[2578]: W1103 00:40:10.339863 2578 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 03 00:40:10.352017 ip-10-0-140-107 kubenswrapper[2578]: W1103 00:40:10.339867 2578 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 03 00:40:10.352017 ip-10-0-140-107 kubenswrapper[2578]: W1103 00:40:10.339870 2578 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 03 00:40:10.352017 ip-10-0-140-107 kubenswrapper[2578]: W1103 00:40:10.339873 2578 feature_gate.go:386] unrecognized feature gate: Example Nov 03 00:40:10.352017 ip-10-0-140-107 kubenswrapper[2578]: W1103 00:40:10.339876 2578 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 03 00:40:10.352017 ip-10-0-140-107 kubenswrapper[2578]: W1103 00:40:10.339879 2578 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 03 00:40:10.352017 ip-10-0-140-107 kubenswrapper[2578]: W1103 00:40:10.339881 2578 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 03 00:40:10.352017 ip-10-0-140-107 kubenswrapper[2578]: W1103 00:40:10.339884 2578 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 03 00:40:10.352017 ip-10-0-140-107 kubenswrapper[2578]: W1103 00:40:10.339887 2578 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 03 00:40:10.352017 ip-10-0-140-107 kubenswrapper[2578]: W1103 00:40:10.339889 2578 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 03 00:40:10.352017 ip-10-0-140-107 kubenswrapper[2578]: W1103 00:40:10.339892 2578 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 03 00:40:10.352479 ip-10-0-140-107 kubenswrapper[2578]: W1103 00:40:10.339895 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 03 00:40:10.352479 ip-10-0-140-107 kubenswrapper[2578]: W1103 00:40:10.339897 2578 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 03 00:40:10.352479 ip-10-0-140-107 kubenswrapper[2578]: W1103 00:40:10.339901 2578 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 03 00:40:10.352479 ip-10-0-140-107 kubenswrapper[2578]: W1103 00:40:10.339915 2578 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 03 00:40:10.352479 ip-10-0-140-107 kubenswrapper[2578]: W1103 00:40:10.339920 2578 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 03 00:40:10.352479 ip-10-0-140-107 kubenswrapper[2578]: W1103 00:40:10.339923 2578 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 03 00:40:10.352479 ip-10-0-140-107 kubenswrapper[2578]: W1103 00:40:10.339926 2578 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 03 00:40:10.352479 ip-10-0-140-107 kubenswrapper[2578]: W1103 00:40:10.339929 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 03 00:40:10.352479 ip-10-0-140-107 kubenswrapper[2578]: I1103 00:40:10.347613 2578 server.go:525] "Kubelet version" kubeletVersion="v1.35.9" Nov 03 00:40:10.352479 ip-10-0-140-107 kubenswrapper[2578]: I1103 00:40:10.347639 2578 server.go:527] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK="" Nov 03 00:40:10.352479 ip-10-0-140-107 kubenswrapper[2578]: I1103 00:40:10.347655 2578 watchdog_linux.go:95] "Systemd watchdog is not enabled" Nov 03 00:40:10.352479 ip-10-0-140-107 kubenswrapper[2578]: I1103 00:40:10.347660 2578 watchdog_linux.go:138] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started." Nov 03 00:40:10.352479 ip-10-0-140-107 kubenswrapper[2578]: W1103 00:40:10.347719 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 03 00:40:10.352479 ip-10-0-140-107 kubenswrapper[2578]: W1103 00:40:10.347724 2578 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 03 00:40:10.352479 ip-10-0-140-107 kubenswrapper[2578]: W1103 00:40:10.347728 2578 feature_gate.go:386] unrecognized feature gate: EVPN Nov 03 00:40:10.352479 ip-10-0-140-107 kubenswrapper[2578]: W1103 00:40:10.347731 2578 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 03 00:40:10.352479 ip-10-0-140-107 kubenswrapper[2578]: W1103 00:40:10.347735 2578 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 03 00:40:10.352479 ip-10-0-140-107 kubenswrapper[2578]: W1103 00:40:10.347737 2578 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 03 00:40:10.352479 ip-10-0-140-107 kubenswrapper[2578]: W1103 00:40:10.347740 2578 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 03 00:40:10.352479 ip-10-0-140-107 kubenswrapper[2578]: W1103 00:40:10.347743 2578 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 03 00:40:10.353014 ip-10-0-140-107 kubenswrapper[2578]: W1103 00:40:10.347746 2578 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 03 00:40:10.353014 ip-10-0-140-107 kubenswrapper[2578]: W1103 00:40:10.347749 2578 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 03 00:40:10.353014 ip-10-0-140-107 kubenswrapper[2578]: W1103 00:40:10.347753 2578 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 03 00:40:10.353014 ip-10-0-140-107 kubenswrapper[2578]: W1103 00:40:10.347758 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 03 00:40:10.353014 ip-10-0-140-107 kubenswrapper[2578]: W1103 00:40:10.347761 2578 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 03 00:40:10.353014 ip-10-0-140-107 kubenswrapper[2578]: W1103 00:40:10.347764 2578 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 03 00:40:10.353014 ip-10-0-140-107 kubenswrapper[2578]: W1103 00:40:10.347767 2578 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 03 00:40:10.353014 ip-10-0-140-107 kubenswrapper[2578]: W1103 00:40:10.347770 2578 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 03 00:40:10.353014 ip-10-0-140-107 kubenswrapper[2578]: W1103 00:40:10.347772 2578 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 03 00:40:10.353014 ip-10-0-140-107 kubenswrapper[2578]: W1103 00:40:10.347776 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 03 00:40:10.353014 ip-10-0-140-107 kubenswrapper[2578]: W1103 00:40:10.347778 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 03 00:40:10.353014 ip-10-0-140-107 kubenswrapper[2578]: W1103 00:40:10.347781 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 03 00:40:10.353014 ip-10-0-140-107 kubenswrapper[2578]: W1103 00:40:10.347784 2578 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 03 00:40:10.353014 ip-10-0-140-107 kubenswrapper[2578]: W1103 00:40:10.347787 2578 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 03 00:40:10.353014 ip-10-0-140-107 kubenswrapper[2578]: W1103 00:40:10.347790 2578 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 03 00:40:10.353014 ip-10-0-140-107 kubenswrapper[2578]: W1103 00:40:10.347792 2578 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 03 00:40:10.353014 ip-10-0-140-107 kubenswrapper[2578]: W1103 00:40:10.347795 2578 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 03 00:40:10.353014 ip-10-0-140-107 kubenswrapper[2578]: W1103 00:40:10.347798 2578 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 03 00:40:10.353014 ip-10-0-140-107 kubenswrapper[2578]: W1103 00:40:10.347801 2578 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 03 00:40:10.353454 ip-10-0-140-107 kubenswrapper[2578]: W1103 00:40:10.347804 2578 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 03 00:40:10.353454 ip-10-0-140-107 kubenswrapper[2578]: W1103 00:40:10.347807 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 03 00:40:10.353454 ip-10-0-140-107 kubenswrapper[2578]: W1103 00:40:10.347810 2578 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 03 00:40:10.353454 ip-10-0-140-107 kubenswrapper[2578]: W1103 00:40:10.347813 2578 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 03 00:40:10.353454 ip-10-0-140-107 kubenswrapper[2578]: W1103 00:40:10.347816 2578 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 03 00:40:10.353454 ip-10-0-140-107 kubenswrapper[2578]: W1103 00:40:10.347819 2578 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 03 00:40:10.353454 ip-10-0-140-107 kubenswrapper[2578]: W1103 00:40:10.347822 2578 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 03 00:40:10.353454 ip-10-0-140-107 kubenswrapper[2578]: W1103 00:40:10.347824 2578 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 03 00:40:10.353454 ip-10-0-140-107 kubenswrapper[2578]: W1103 00:40:10.347827 2578 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 03 00:40:10.353454 ip-10-0-140-107 kubenswrapper[2578]: W1103 00:40:10.347830 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 03 00:40:10.353454 ip-10-0-140-107 kubenswrapper[2578]: W1103 00:40:10.347833 2578 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 03 00:40:10.353454 ip-10-0-140-107 kubenswrapper[2578]: W1103 00:40:10.347837 2578 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 03 00:40:10.353454 ip-10-0-140-107 kubenswrapper[2578]: W1103 00:40:10.347840 2578 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 03 00:40:10.353454 ip-10-0-140-107 kubenswrapper[2578]: W1103 00:40:10.347843 2578 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 03 00:40:10.353454 ip-10-0-140-107 kubenswrapper[2578]: W1103 00:40:10.347846 2578 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 03 00:40:10.353454 ip-10-0-140-107 kubenswrapper[2578]: W1103 00:40:10.347848 2578 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 03 00:40:10.353454 ip-10-0-140-107 kubenswrapper[2578]: W1103 00:40:10.347851 2578 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 03 00:40:10.353454 ip-10-0-140-107 kubenswrapper[2578]: W1103 00:40:10.347854 2578 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 03 00:40:10.353454 ip-10-0-140-107 kubenswrapper[2578]: W1103 00:40:10.347856 2578 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 03 00:40:10.353942 ip-10-0-140-107 kubenswrapper[2578]: W1103 00:40:10.347859 2578 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 03 00:40:10.353942 ip-10-0-140-107 kubenswrapper[2578]: W1103 00:40:10.347862 2578 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 03 00:40:10.353942 ip-10-0-140-107 kubenswrapper[2578]: W1103 00:40:10.347864 2578 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 03 00:40:10.353942 ip-10-0-140-107 kubenswrapper[2578]: W1103 00:40:10.347867 2578 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 03 00:40:10.353942 ip-10-0-140-107 kubenswrapper[2578]: W1103 00:40:10.347870 2578 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 03 00:40:10.353942 ip-10-0-140-107 kubenswrapper[2578]: W1103 00:40:10.347873 2578 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 03 00:40:10.353942 ip-10-0-140-107 kubenswrapper[2578]: W1103 00:40:10.347876 2578 feature_gate.go:386] unrecognized feature gate: Example Nov 03 00:40:10.353942 ip-10-0-140-107 kubenswrapper[2578]: W1103 00:40:10.347878 2578 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 03 00:40:10.353942 ip-10-0-140-107 kubenswrapper[2578]: W1103 00:40:10.347881 2578 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 03 00:40:10.353942 ip-10-0-140-107 kubenswrapper[2578]: W1103 00:40:10.347883 2578 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 03 00:40:10.353942 ip-10-0-140-107 kubenswrapper[2578]: W1103 00:40:10.347886 2578 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 03 00:40:10.353942 ip-10-0-140-107 kubenswrapper[2578]: W1103 00:40:10.347889 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 03 00:40:10.353942 ip-10-0-140-107 kubenswrapper[2578]: W1103 00:40:10.347891 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 03 00:40:10.353942 ip-10-0-140-107 kubenswrapper[2578]: W1103 00:40:10.347894 2578 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 03 00:40:10.353942 ip-10-0-140-107 kubenswrapper[2578]: W1103 00:40:10.347897 2578 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 03 00:40:10.353942 ip-10-0-140-107 kubenswrapper[2578]: W1103 00:40:10.347900 2578 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 03 00:40:10.353942 ip-10-0-140-107 kubenswrapper[2578]: W1103 00:40:10.347903 2578 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 03 00:40:10.353942 ip-10-0-140-107 kubenswrapper[2578]: W1103 00:40:10.347923 2578 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 03 00:40:10.353942 ip-10-0-140-107 kubenswrapper[2578]: W1103 00:40:10.347926 2578 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 03 00:40:10.353942 ip-10-0-140-107 kubenswrapper[2578]: W1103 00:40:10.347929 2578 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 03 00:40:10.354422 ip-10-0-140-107 kubenswrapper[2578]: W1103 00:40:10.347931 2578 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 03 00:40:10.354422 ip-10-0-140-107 kubenswrapper[2578]: W1103 00:40:10.347934 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 03 00:40:10.354422 ip-10-0-140-107 kubenswrapper[2578]: W1103 00:40:10.347937 2578 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 03 00:40:10.354422 ip-10-0-140-107 kubenswrapper[2578]: W1103 00:40:10.347945 2578 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 03 00:40:10.354422 ip-10-0-140-107 kubenswrapper[2578]: W1103 00:40:10.347947 2578 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 03 00:40:10.354422 ip-10-0-140-107 kubenswrapper[2578]: W1103 00:40:10.347950 2578 feature_gate.go:386] unrecognized feature gate: Example2 Nov 03 00:40:10.354422 ip-10-0-140-107 kubenswrapper[2578]: W1103 00:40:10.347953 2578 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 03 00:40:10.354422 ip-10-0-140-107 kubenswrapper[2578]: W1103 00:40:10.347956 2578 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 03 00:40:10.354422 ip-10-0-140-107 kubenswrapper[2578]: W1103 00:40:10.347958 2578 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 03 00:40:10.354422 ip-10-0-140-107 kubenswrapper[2578]: W1103 00:40:10.347961 2578 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 03 00:40:10.354422 ip-10-0-140-107 kubenswrapper[2578]: W1103 00:40:10.347963 2578 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 03 00:40:10.354422 ip-10-0-140-107 kubenswrapper[2578]: W1103 00:40:10.347966 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 03 00:40:10.354422 ip-10-0-140-107 kubenswrapper[2578]: W1103 00:40:10.347968 2578 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 03 00:40:10.354422 ip-10-0-140-107 kubenswrapper[2578]: W1103 00:40:10.347972 2578 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 03 00:40:10.354422 ip-10-0-140-107 kubenswrapper[2578]: W1103 00:40:10.347975 2578 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 03 00:40:10.354422 ip-10-0-140-107 kubenswrapper[2578]: W1103 00:40:10.347977 2578 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 03 00:40:10.354422 ip-10-0-140-107 kubenswrapper[2578]: W1103 00:40:10.347980 2578 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 03 00:40:10.354422 ip-10-0-140-107 kubenswrapper[2578]: W1103 00:40:10.347983 2578 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 03 00:40:10.354422 ip-10-0-140-107 kubenswrapper[2578]: W1103 00:40:10.347986 2578 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 03 00:40:10.354422 ip-10-0-140-107 kubenswrapper[2578]: W1103 00:40:10.347988 2578 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 03 00:40:10.354888 ip-10-0-140-107 kubenswrapper[2578]: W1103 00:40:10.347991 2578 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 03 00:40:10.354888 ip-10-0-140-107 kubenswrapper[2578]: W1103 00:40:10.347993 2578 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 03 00:40:10.354888 ip-10-0-140-107 kubenswrapper[2578]: W1103 00:40:10.347996 2578 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 03 00:40:10.354888 ip-10-0-140-107 kubenswrapper[2578]: W1103 00:40:10.347999 2578 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 03 00:40:10.354888 ip-10-0-140-107 kubenswrapper[2578]: W1103 00:40:10.348001 2578 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 03 00:40:10.354888 ip-10-0-140-107 kubenswrapper[2578]: W1103 00:40:10.348004 2578 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 03 00:40:10.354888 ip-10-0-140-107 kubenswrapper[2578]: W1103 00:40:10.348007 2578 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 03 00:40:10.354888 ip-10-0-140-107 kubenswrapper[2578]: W1103 00:40:10.348011 2578 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 03 00:40:10.354888 ip-10-0-140-107 kubenswrapper[2578]: W1103 00:40:10.348014 2578 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 03 00:40:10.354888 ip-10-0-140-107 kubenswrapper[2578]: W1103 00:40:10.348017 2578 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 03 00:40:10.354888 ip-10-0-140-107 kubenswrapper[2578]: W1103 00:40:10.348020 2578 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 03 00:40:10.354888 ip-10-0-140-107 kubenswrapper[2578]: W1103 00:40:10.348022 2578 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 03 00:40:10.354888 ip-10-0-140-107 kubenswrapper[2578]: W1103 00:40:10.348025 2578 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 03 00:40:10.354888 ip-10-0-140-107 kubenswrapper[2578]: W1103 00:40:10.348028 2578 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 03 00:40:10.354888 ip-10-0-140-107 kubenswrapper[2578]: W1103 00:40:10.348030 2578 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 03 00:40:10.354888 ip-10-0-140-107 kubenswrapper[2578]: W1103 00:40:10.348033 2578 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 03 00:40:10.354888 ip-10-0-140-107 kubenswrapper[2578]: W1103 00:40:10.348035 2578 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 03 00:40:10.354888 ip-10-0-140-107 kubenswrapper[2578]: W1103 00:40:10.348038 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 03 00:40:10.354888 ip-10-0-140-107 kubenswrapper[2578]: W1103 00:40:10.348041 2578 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 03 00:40:10.354888 ip-10-0-140-107 kubenswrapper[2578]: W1103 00:40:10.348043 2578 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 03 00:40:10.355363 ip-10-0-140-107 kubenswrapper[2578]: I1103 00:40:10.348075 2578 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Nov 03 00:40:10.355363 ip-10-0-140-107 kubenswrapper[2578]: W1103 00:40:10.348239 2578 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 03 00:40:10.355363 ip-10-0-140-107 kubenswrapper[2578]: W1103 00:40:10.348243 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 03 00:40:10.355363 ip-10-0-140-107 kubenswrapper[2578]: W1103 00:40:10.348248 2578 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 03 00:40:10.355363 ip-10-0-140-107 kubenswrapper[2578]: W1103 00:40:10.348251 2578 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 03 00:40:10.355363 ip-10-0-140-107 kubenswrapper[2578]: W1103 00:40:10.348254 2578 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 03 00:40:10.355363 ip-10-0-140-107 kubenswrapper[2578]: W1103 00:40:10.348259 2578 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 03 00:40:10.355363 ip-10-0-140-107 kubenswrapper[2578]: W1103 00:40:10.348262 2578 feature_gate.go:386] unrecognized feature gate: Example Nov 03 00:40:10.355363 ip-10-0-140-107 kubenswrapper[2578]: W1103 00:40:10.348265 2578 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 03 00:40:10.355363 ip-10-0-140-107 kubenswrapper[2578]: W1103 00:40:10.348268 2578 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 03 00:40:10.355363 ip-10-0-140-107 kubenswrapper[2578]: W1103 00:40:10.348271 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 03 00:40:10.355363 ip-10-0-140-107 kubenswrapper[2578]: W1103 00:40:10.348274 2578 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 03 00:40:10.355363 ip-10-0-140-107 kubenswrapper[2578]: W1103 00:40:10.348277 2578 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 03 00:40:10.355363 ip-10-0-140-107 kubenswrapper[2578]: W1103 00:40:10.348280 2578 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 03 00:40:10.355363 ip-10-0-140-107 kubenswrapper[2578]: W1103 00:40:10.348282 2578 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 03 00:40:10.355363 ip-10-0-140-107 kubenswrapper[2578]: W1103 00:40:10.348285 2578 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 03 00:40:10.355786 ip-10-0-140-107 kubenswrapper[2578]: W1103 00:40:10.348288 2578 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 03 00:40:10.355786 ip-10-0-140-107 kubenswrapper[2578]: W1103 00:40:10.348290 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 03 00:40:10.355786 ip-10-0-140-107 kubenswrapper[2578]: W1103 00:40:10.348293 2578 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 03 00:40:10.355786 ip-10-0-140-107 kubenswrapper[2578]: W1103 00:40:10.348296 2578 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 03 00:40:10.355786 ip-10-0-140-107 kubenswrapper[2578]: W1103 00:40:10.348299 2578 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 03 00:40:10.355786 ip-10-0-140-107 kubenswrapper[2578]: W1103 00:40:10.348302 2578 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 03 00:40:10.355786 ip-10-0-140-107 kubenswrapper[2578]: W1103 00:40:10.348305 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 03 00:40:10.355786 ip-10-0-140-107 kubenswrapper[2578]: W1103 00:40:10.348308 2578 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 03 00:40:10.355786 ip-10-0-140-107 kubenswrapper[2578]: W1103 00:40:10.348311 2578 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 03 00:40:10.355786 ip-10-0-140-107 kubenswrapper[2578]: W1103 00:40:10.348313 2578 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 03 00:40:10.355786 ip-10-0-140-107 kubenswrapper[2578]: W1103 00:40:10.348316 2578 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 03 00:40:10.355786 ip-10-0-140-107 kubenswrapper[2578]: W1103 00:40:10.348319 2578 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 03 00:40:10.355786 ip-10-0-140-107 kubenswrapper[2578]: W1103 00:40:10.348322 2578 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 03 00:40:10.355786 ip-10-0-140-107 kubenswrapper[2578]: W1103 00:40:10.348324 2578 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 03 00:40:10.355786 ip-10-0-140-107 kubenswrapper[2578]: W1103 00:40:10.348327 2578 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 03 00:40:10.355786 ip-10-0-140-107 kubenswrapper[2578]: W1103 00:40:10.348329 2578 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 03 00:40:10.355786 ip-10-0-140-107 kubenswrapper[2578]: W1103 00:40:10.348332 2578 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 03 00:40:10.355786 ip-10-0-140-107 kubenswrapper[2578]: W1103 00:40:10.348335 2578 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 03 00:40:10.355786 ip-10-0-140-107 kubenswrapper[2578]: W1103 00:40:10.348337 2578 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 03 00:40:10.355786 ip-10-0-140-107 kubenswrapper[2578]: W1103 00:40:10.348340 2578 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 03 00:40:10.356285 ip-10-0-140-107 kubenswrapper[2578]: W1103 00:40:10.348343 2578 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 03 00:40:10.356285 ip-10-0-140-107 kubenswrapper[2578]: W1103 00:40:10.348345 2578 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 03 00:40:10.356285 ip-10-0-140-107 kubenswrapper[2578]: W1103 00:40:10.348348 2578 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 03 00:40:10.356285 ip-10-0-140-107 kubenswrapper[2578]: W1103 00:40:10.348351 2578 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 03 00:40:10.356285 ip-10-0-140-107 kubenswrapper[2578]: W1103 00:40:10.348355 2578 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 03 00:40:10.356285 ip-10-0-140-107 kubenswrapper[2578]: W1103 00:40:10.348360 2578 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 03 00:40:10.356285 ip-10-0-140-107 kubenswrapper[2578]: W1103 00:40:10.348363 2578 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 03 00:40:10.356285 ip-10-0-140-107 kubenswrapper[2578]: W1103 00:40:10.348366 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 03 00:40:10.356285 ip-10-0-140-107 kubenswrapper[2578]: W1103 00:40:10.348369 2578 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 03 00:40:10.356285 ip-10-0-140-107 kubenswrapper[2578]: W1103 00:40:10.348372 2578 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 03 00:40:10.356285 ip-10-0-140-107 kubenswrapper[2578]: W1103 00:40:10.348374 2578 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 03 00:40:10.356285 ip-10-0-140-107 kubenswrapper[2578]: W1103 00:40:10.348377 2578 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 03 00:40:10.356285 ip-10-0-140-107 kubenswrapper[2578]: W1103 00:40:10.348380 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 03 00:40:10.356285 ip-10-0-140-107 kubenswrapper[2578]: W1103 00:40:10.348383 2578 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 03 00:40:10.356285 ip-10-0-140-107 kubenswrapper[2578]: W1103 00:40:10.348385 2578 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 03 00:40:10.356285 ip-10-0-140-107 kubenswrapper[2578]: W1103 00:40:10.348389 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 03 00:40:10.356285 ip-10-0-140-107 kubenswrapper[2578]: W1103 00:40:10.348391 2578 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 03 00:40:10.356285 ip-10-0-140-107 kubenswrapper[2578]: W1103 00:40:10.348394 2578 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 03 00:40:10.356719 ip-10-0-140-107 kubenswrapper[2578]: W1103 00:40:10.348397 2578 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 03 00:40:10.356719 ip-10-0-140-107 kubenswrapper[2578]: W1103 00:40:10.348399 2578 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 03 00:40:10.356719 ip-10-0-140-107 kubenswrapper[2578]: W1103 00:40:10.348402 2578 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 03 00:40:10.356719 ip-10-0-140-107 kubenswrapper[2578]: W1103 00:40:10.348405 2578 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 03 00:40:10.356719 ip-10-0-140-107 kubenswrapper[2578]: W1103 00:40:10.348407 2578 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 03 00:40:10.356719 ip-10-0-140-107 kubenswrapper[2578]: W1103 00:40:10.348410 2578 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 03 00:40:10.356719 ip-10-0-140-107 kubenswrapper[2578]: W1103 00:40:10.348413 2578 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 03 00:40:10.356719 ip-10-0-140-107 kubenswrapper[2578]: W1103 00:40:10.348415 2578 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 03 00:40:10.356719 ip-10-0-140-107 kubenswrapper[2578]: W1103 00:40:10.348418 2578 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 03 00:40:10.356719 ip-10-0-140-107 kubenswrapper[2578]: W1103 00:40:10.348420 2578 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 03 00:40:10.356719 ip-10-0-140-107 kubenswrapper[2578]: W1103 00:40:10.348423 2578 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 03 00:40:10.356719 ip-10-0-140-107 kubenswrapper[2578]: W1103 00:40:10.348425 2578 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 03 00:40:10.356719 ip-10-0-140-107 kubenswrapper[2578]: W1103 00:40:10.348428 2578 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 03 00:40:10.356719 ip-10-0-140-107 kubenswrapper[2578]: W1103 00:40:10.348431 2578 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 03 00:40:10.356719 ip-10-0-140-107 kubenswrapper[2578]: W1103 00:40:10.348434 2578 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 03 00:40:10.356719 ip-10-0-140-107 kubenswrapper[2578]: W1103 00:40:10.348437 2578 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 03 00:40:10.356719 ip-10-0-140-107 kubenswrapper[2578]: W1103 00:40:10.348440 2578 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 03 00:40:10.356719 ip-10-0-140-107 kubenswrapper[2578]: W1103 00:40:10.348443 2578 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 03 00:40:10.356719 ip-10-0-140-107 kubenswrapper[2578]: W1103 00:40:10.348449 2578 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 03 00:40:10.356719 ip-10-0-140-107 kubenswrapper[2578]: W1103 00:40:10.348452 2578 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 03 00:40:10.357322 ip-10-0-140-107 kubenswrapper[2578]: W1103 00:40:10.348454 2578 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 03 00:40:10.357322 ip-10-0-140-107 kubenswrapper[2578]: W1103 00:40:10.348457 2578 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 03 00:40:10.357322 ip-10-0-140-107 kubenswrapper[2578]: W1103 00:40:10.348460 2578 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 03 00:40:10.357322 ip-10-0-140-107 kubenswrapper[2578]: W1103 00:40:10.348463 2578 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 03 00:40:10.357322 ip-10-0-140-107 kubenswrapper[2578]: W1103 00:40:10.348466 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 03 00:40:10.357322 ip-10-0-140-107 kubenswrapper[2578]: W1103 00:40:10.348468 2578 feature_gate.go:386] unrecognized feature gate: Example2 Nov 03 00:40:10.357322 ip-10-0-140-107 kubenswrapper[2578]: W1103 00:40:10.348471 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 03 00:40:10.357322 ip-10-0-140-107 kubenswrapper[2578]: W1103 00:40:10.348474 2578 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 03 00:40:10.357322 ip-10-0-140-107 kubenswrapper[2578]: W1103 00:40:10.348476 2578 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 03 00:40:10.357322 ip-10-0-140-107 kubenswrapper[2578]: W1103 00:40:10.348479 2578 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 03 00:40:10.357322 ip-10-0-140-107 kubenswrapper[2578]: W1103 00:40:10.348482 2578 feature_gate.go:386] unrecognized feature gate: EVPN Nov 03 00:40:10.357322 ip-10-0-140-107 kubenswrapper[2578]: W1103 00:40:10.348484 2578 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 03 00:40:10.357322 ip-10-0-140-107 kubenswrapper[2578]: W1103 00:40:10.348487 2578 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 03 00:40:10.357322 ip-10-0-140-107 kubenswrapper[2578]: W1103 00:40:10.348490 2578 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 03 00:40:10.357322 ip-10-0-140-107 kubenswrapper[2578]: W1103 00:40:10.348492 2578 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 03 00:40:10.357322 ip-10-0-140-107 kubenswrapper[2578]: W1103 00:40:10.348495 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 03 00:40:10.357322 ip-10-0-140-107 kubenswrapper[2578]: W1103 00:40:10.348498 2578 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 03 00:40:10.357322 ip-10-0-140-107 kubenswrapper[2578]: W1103 00:40:10.348500 2578 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 03 00:40:10.357322 ip-10-0-140-107 kubenswrapper[2578]: W1103 00:40:10.348503 2578 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 03 00:40:10.357322 ip-10-0-140-107 kubenswrapper[2578]: W1103 00:40:10.348506 2578 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 03 00:40:10.357781 ip-10-0-140-107 kubenswrapper[2578]: W1103 00:40:10.348508 2578 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 03 00:40:10.357781 ip-10-0-140-107 kubenswrapper[2578]: W1103 00:40:10.348511 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 03 00:40:10.357781 ip-10-0-140-107 kubenswrapper[2578]: W1103 00:40:10.348514 2578 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 03 00:40:10.357781 ip-10-0-140-107 kubenswrapper[2578]: W1103 00:40:10.348517 2578 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 03 00:40:10.357781 ip-10-0-140-107 kubenswrapper[2578]: W1103 00:40:10.348519 2578 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 03 00:40:10.357781 ip-10-0-140-107 kubenswrapper[2578]: W1103 00:40:10.348522 2578 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 03 00:40:10.357781 ip-10-0-140-107 kubenswrapper[2578]: W1103 00:40:10.348525 2578 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 03 00:40:10.357781 ip-10-0-140-107 kubenswrapper[2578]: W1103 00:40:10.348528 2578 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 03 00:40:10.357781 ip-10-0-140-107 kubenswrapper[2578]: W1103 00:40:10.348530 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 03 00:40:10.357781 ip-10-0-140-107 kubenswrapper[2578]: W1103 00:40:10.348533 2578 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 03 00:40:10.357781 ip-10-0-140-107 kubenswrapper[2578]: W1103 00:40:10.348537 2578 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 03 00:40:10.357781 ip-10-0-140-107 kubenswrapper[2578]: W1103 00:40:10.348540 2578 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 03 00:40:10.357781 ip-10-0-140-107 kubenswrapper[2578]: W1103 00:40:10.348543 2578 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 03 00:40:10.357781 ip-10-0-140-107 kubenswrapper[2578]: I1103 00:40:10.348565 2578 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Nov 03 00:40:10.357781 ip-10-0-140-107 kubenswrapper[2578]: W1103 00:40:10.348598 2578 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 03 00:40:10.357781 ip-10-0-140-107 kubenswrapper[2578]: W1103 00:40:10.348602 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 03 00:40:10.358304 ip-10-0-140-107 kubenswrapper[2578]: W1103 00:40:10.348605 2578 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 03 00:40:10.358304 ip-10-0-140-107 kubenswrapper[2578]: W1103 00:40:10.348609 2578 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 03 00:40:10.358304 ip-10-0-140-107 kubenswrapper[2578]: W1103 00:40:10.348612 2578 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 03 00:40:10.358304 ip-10-0-140-107 kubenswrapper[2578]: W1103 00:40:10.348614 2578 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 03 00:40:10.358304 ip-10-0-140-107 kubenswrapper[2578]: W1103 00:40:10.348617 2578 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 03 00:40:10.358304 ip-10-0-140-107 kubenswrapper[2578]: W1103 00:40:10.348620 2578 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 03 00:40:10.358304 ip-10-0-140-107 kubenswrapper[2578]: W1103 00:40:10.348623 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 03 00:40:10.358304 ip-10-0-140-107 kubenswrapper[2578]: W1103 00:40:10.348625 2578 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 03 00:40:10.358304 ip-10-0-140-107 kubenswrapper[2578]: W1103 00:40:10.348628 2578 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 03 00:40:10.358304 ip-10-0-140-107 kubenswrapper[2578]: W1103 00:40:10.348631 2578 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 03 00:40:10.358304 ip-10-0-140-107 kubenswrapper[2578]: W1103 00:40:10.348633 2578 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 03 00:40:10.358304 ip-10-0-140-107 kubenswrapper[2578]: W1103 00:40:10.348636 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 03 00:40:10.358304 ip-10-0-140-107 kubenswrapper[2578]: W1103 00:40:10.348639 2578 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 03 00:40:10.358304 ip-10-0-140-107 kubenswrapper[2578]: W1103 00:40:10.348642 2578 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 03 00:40:10.358304 ip-10-0-140-107 kubenswrapper[2578]: W1103 00:40:10.348645 2578 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 03 00:40:10.358304 ip-10-0-140-107 kubenswrapper[2578]: W1103 00:40:10.348647 2578 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 03 00:40:10.358304 ip-10-0-140-107 kubenswrapper[2578]: W1103 00:40:10.348650 2578 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 03 00:40:10.358304 ip-10-0-140-107 kubenswrapper[2578]: W1103 00:40:10.348653 2578 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 03 00:40:10.358304 ip-10-0-140-107 kubenswrapper[2578]: W1103 00:40:10.348656 2578 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 03 00:40:10.358840 ip-10-0-140-107 kubenswrapper[2578]: W1103 00:40:10.348658 2578 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 03 00:40:10.358840 ip-10-0-140-107 kubenswrapper[2578]: W1103 00:40:10.348661 2578 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 03 00:40:10.358840 ip-10-0-140-107 kubenswrapper[2578]: W1103 00:40:10.348663 2578 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 03 00:40:10.358840 ip-10-0-140-107 kubenswrapper[2578]: W1103 00:40:10.348666 2578 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 03 00:40:10.358840 ip-10-0-140-107 kubenswrapper[2578]: W1103 00:40:10.348669 2578 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 03 00:40:10.358840 ip-10-0-140-107 kubenswrapper[2578]: W1103 00:40:10.348672 2578 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 03 00:40:10.358840 ip-10-0-140-107 kubenswrapper[2578]: W1103 00:40:10.348674 2578 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 03 00:40:10.358840 ip-10-0-140-107 kubenswrapper[2578]: W1103 00:40:10.348677 2578 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 03 00:40:10.358840 ip-10-0-140-107 kubenswrapper[2578]: W1103 00:40:10.348681 2578 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 03 00:40:10.358840 ip-10-0-140-107 kubenswrapper[2578]: W1103 00:40:10.348684 2578 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 03 00:40:10.358840 ip-10-0-140-107 kubenswrapper[2578]: W1103 00:40:10.348688 2578 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 03 00:40:10.358840 ip-10-0-140-107 kubenswrapper[2578]: W1103 00:40:10.348693 2578 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 03 00:40:10.358840 ip-10-0-140-107 kubenswrapper[2578]: W1103 00:40:10.348697 2578 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 03 00:40:10.358840 ip-10-0-140-107 kubenswrapper[2578]: W1103 00:40:10.348701 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 03 00:40:10.358840 ip-10-0-140-107 kubenswrapper[2578]: W1103 00:40:10.348705 2578 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 03 00:40:10.358840 ip-10-0-140-107 kubenswrapper[2578]: W1103 00:40:10.348708 2578 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 03 00:40:10.358840 ip-10-0-140-107 kubenswrapper[2578]: W1103 00:40:10.348711 2578 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 03 00:40:10.358840 ip-10-0-140-107 kubenswrapper[2578]: W1103 00:40:10.348713 2578 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 03 00:40:10.359581 ip-10-0-140-107 kubenswrapper[2578]: W1103 00:40:10.348716 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 03 00:40:10.359581 ip-10-0-140-107 kubenswrapper[2578]: W1103 00:40:10.348720 2578 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 03 00:40:10.359581 ip-10-0-140-107 kubenswrapper[2578]: W1103 00:40:10.348722 2578 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 03 00:40:10.359581 ip-10-0-140-107 kubenswrapper[2578]: W1103 00:40:10.348725 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 03 00:40:10.359581 ip-10-0-140-107 kubenswrapper[2578]: W1103 00:40:10.348728 2578 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 03 00:40:10.359581 ip-10-0-140-107 kubenswrapper[2578]: W1103 00:40:10.348730 2578 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 03 00:40:10.359581 ip-10-0-140-107 kubenswrapper[2578]: W1103 00:40:10.348734 2578 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 03 00:40:10.359581 ip-10-0-140-107 kubenswrapper[2578]: W1103 00:40:10.348736 2578 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 03 00:40:10.359581 ip-10-0-140-107 kubenswrapper[2578]: W1103 00:40:10.348739 2578 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 03 00:40:10.359581 ip-10-0-140-107 kubenswrapper[2578]: W1103 00:40:10.348742 2578 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 03 00:40:10.359581 ip-10-0-140-107 kubenswrapper[2578]: W1103 00:40:10.348744 2578 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 03 00:40:10.359581 ip-10-0-140-107 kubenswrapper[2578]: W1103 00:40:10.348747 2578 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 03 00:40:10.359581 ip-10-0-140-107 kubenswrapper[2578]: W1103 00:40:10.348750 2578 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 03 00:40:10.359581 ip-10-0-140-107 kubenswrapper[2578]: W1103 00:40:10.348752 2578 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 03 00:40:10.359581 ip-10-0-140-107 kubenswrapper[2578]: W1103 00:40:10.348755 2578 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 03 00:40:10.359581 ip-10-0-140-107 kubenswrapper[2578]: W1103 00:40:10.348758 2578 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 03 00:40:10.359581 ip-10-0-140-107 kubenswrapper[2578]: W1103 00:40:10.348760 2578 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 03 00:40:10.359581 ip-10-0-140-107 kubenswrapper[2578]: W1103 00:40:10.348763 2578 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 03 00:40:10.359581 ip-10-0-140-107 kubenswrapper[2578]: W1103 00:40:10.348766 2578 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 03 00:40:10.359581 ip-10-0-140-107 kubenswrapper[2578]: W1103 00:40:10.348768 2578 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 03 00:40:10.360080 ip-10-0-140-107 kubenswrapper[2578]: W1103 00:40:10.348772 2578 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 03 00:40:10.360080 ip-10-0-140-107 kubenswrapper[2578]: W1103 00:40:10.348775 2578 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 03 00:40:10.360080 ip-10-0-140-107 kubenswrapper[2578]: W1103 00:40:10.348779 2578 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 03 00:40:10.360080 ip-10-0-140-107 kubenswrapper[2578]: W1103 00:40:10.348781 2578 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 03 00:40:10.360080 ip-10-0-140-107 kubenswrapper[2578]: W1103 00:40:10.348785 2578 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 03 00:40:10.360080 ip-10-0-140-107 kubenswrapper[2578]: W1103 00:40:10.348787 2578 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 03 00:40:10.360080 ip-10-0-140-107 kubenswrapper[2578]: W1103 00:40:10.348790 2578 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 03 00:40:10.360080 ip-10-0-140-107 kubenswrapper[2578]: W1103 00:40:10.348793 2578 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 03 00:40:10.360080 ip-10-0-140-107 kubenswrapper[2578]: W1103 00:40:10.348795 2578 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 03 00:40:10.360080 ip-10-0-140-107 kubenswrapper[2578]: W1103 00:40:10.348798 2578 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 03 00:40:10.360080 ip-10-0-140-107 kubenswrapper[2578]: W1103 00:40:10.348801 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 03 00:40:10.360080 ip-10-0-140-107 kubenswrapper[2578]: W1103 00:40:10.348804 2578 feature_gate.go:386] unrecognized feature gate: Example2 Nov 03 00:40:10.360080 ip-10-0-140-107 kubenswrapper[2578]: W1103 00:40:10.348806 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 03 00:40:10.360080 ip-10-0-140-107 kubenswrapper[2578]: W1103 00:40:10.348809 2578 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 03 00:40:10.360080 ip-10-0-140-107 kubenswrapper[2578]: W1103 00:40:10.348812 2578 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 03 00:40:10.360080 ip-10-0-140-107 kubenswrapper[2578]: W1103 00:40:10.348815 2578 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 03 00:40:10.360080 ip-10-0-140-107 kubenswrapper[2578]: W1103 00:40:10.348817 2578 feature_gate.go:386] unrecognized feature gate: EVPN Nov 03 00:40:10.360080 ip-10-0-140-107 kubenswrapper[2578]: W1103 00:40:10.348820 2578 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 03 00:40:10.360080 ip-10-0-140-107 kubenswrapper[2578]: W1103 00:40:10.348823 2578 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 03 00:40:10.360080 ip-10-0-140-107 kubenswrapper[2578]: W1103 00:40:10.348825 2578 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 03 00:40:10.360736 ip-10-0-140-107 kubenswrapper[2578]: W1103 00:40:10.348828 2578 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 03 00:40:10.360736 ip-10-0-140-107 kubenswrapper[2578]: W1103 00:40:10.348831 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 03 00:40:10.360736 ip-10-0-140-107 kubenswrapper[2578]: W1103 00:40:10.348834 2578 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 03 00:40:10.360736 ip-10-0-140-107 kubenswrapper[2578]: W1103 00:40:10.348836 2578 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 03 00:40:10.360736 ip-10-0-140-107 kubenswrapper[2578]: W1103 00:40:10.348839 2578 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 03 00:40:10.360736 ip-10-0-140-107 kubenswrapper[2578]: W1103 00:40:10.348841 2578 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 03 00:40:10.360736 ip-10-0-140-107 kubenswrapper[2578]: W1103 00:40:10.348844 2578 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 03 00:40:10.360736 ip-10-0-140-107 kubenswrapper[2578]: W1103 00:40:10.348847 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 03 00:40:10.360736 ip-10-0-140-107 kubenswrapper[2578]: W1103 00:40:10.348850 2578 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 03 00:40:10.360736 ip-10-0-140-107 kubenswrapper[2578]: W1103 00:40:10.348853 2578 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 03 00:40:10.360736 ip-10-0-140-107 kubenswrapper[2578]: W1103 00:40:10.348855 2578 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 03 00:40:10.360736 ip-10-0-140-107 kubenswrapper[2578]: W1103 00:40:10.348858 2578 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 03 00:40:10.360736 ip-10-0-140-107 kubenswrapper[2578]: W1103 00:40:10.348861 2578 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 03 00:40:10.360736 ip-10-0-140-107 kubenswrapper[2578]: W1103 00:40:10.348864 2578 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 03 00:40:10.360736 ip-10-0-140-107 kubenswrapper[2578]: W1103 00:40:10.348868 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 03 00:40:10.360736 ip-10-0-140-107 kubenswrapper[2578]: W1103 00:40:10.348870 2578 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 03 00:40:10.360736 ip-10-0-140-107 kubenswrapper[2578]: W1103 00:40:10.348873 2578 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 03 00:40:10.360736 ip-10-0-140-107 kubenswrapper[2578]: W1103 00:40:10.348876 2578 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 03 00:40:10.360736 ip-10-0-140-107 kubenswrapper[2578]: W1103 00:40:10.348879 2578 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 03 00:40:10.360736 ip-10-0-140-107 kubenswrapper[2578]: W1103 00:40:10.348881 2578 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 03 00:40:10.361221 ip-10-0-140-107 kubenswrapper[2578]: W1103 00:40:10.348884 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 03 00:40:10.361221 ip-10-0-140-107 kubenswrapper[2578]: W1103 00:40:10.348888 2578 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 03 00:40:10.361221 ip-10-0-140-107 kubenswrapper[2578]: W1103 00:40:10.348890 2578 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 03 00:40:10.361221 ip-10-0-140-107 kubenswrapper[2578]: W1103 00:40:10.348893 2578 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 03 00:40:10.361221 ip-10-0-140-107 kubenswrapper[2578]: W1103 00:40:10.348895 2578 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 03 00:40:10.361221 ip-10-0-140-107 kubenswrapper[2578]: W1103 00:40:10.348898 2578 feature_gate.go:386] unrecognized feature gate: Example Nov 03 00:40:10.361221 ip-10-0-140-107 kubenswrapper[2578]: W1103 00:40:10.348901 2578 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 03 00:40:10.361221 ip-10-0-140-107 kubenswrapper[2578]: I1103 00:40:10.348974 2578 server.go:957] "Client rotation is on, will bootstrap in background" Nov 03 00:40:10.361221 ip-10-0-140-107 kubenswrapper[2578]: I1103 00:40:10.350964 2578 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir" Nov 03 00:40:10.361221 ip-10-0-140-107 kubenswrapper[2578]: I1103 00:40:10.351923 2578 server.go:1014] "Starting client certificate rotation" Nov 03 00:40:10.361221 ip-10-0-140-107 kubenswrapper[2578]: I1103 00:40:10.352015 2578 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet" Nov 03 00:40:10.361221 ip-10-0-140-107 kubenswrapper[2578]: I1103 00:40:10.352045 2578 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet" Nov 03 00:40:10.361221 ip-10-0-140-107 kubenswrapper[2578]: I1103 00:40:10.354362 2578 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Nov 03 00:40:10.361221 ip-10-0-140-107 kubenswrapper[2578]: I1103 00:40:10.355485 2578 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Nov 03 00:40:10.367265 ip-10-0-140-107 kubenswrapper[2578]: I1103 00:40:10.367247 2578 log.go:25] "Validated CRI v1 runtime API" Nov 03 00:40:10.370799 ip-10-0-140-107 kubenswrapper[2578]: I1103 00:40:10.370783 2578 log.go:25] "Validated CRI v1 image API" Nov 03 00:40:10.371599 ip-10-0-140-107 kubenswrapper[2578]: I1103 00:40:10.371569 2578 server.go:1424] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd" Nov 03 00:40:10.377110 ip-10-0-140-107 kubenswrapper[2578]: I1103 00:40:10.377087 2578 fs.go:135] Filesystem UUIDs: map[2ebfe389-d801-4aca-8bd7-d371bb69eba5:/dev/nvme0n1p3 7B77-95E7:/dev/nvme0n1p2 b51bc7d0-7d6e-464a-bc11-e10a6cd8889d:/dev/nvme0n1p4] Nov 03 00:40:10.377178 ip-10-0-140-107 kubenswrapper[2578]: I1103 00:40:10.377106 2578 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:24 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:26 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:32 fsType:tmpfs blockSize:0} composefs_0-34:{mountpoint:/ major:0 minor:34 fsType:overlay blockSize:0}] Nov 03 00:40:10.381341 ip-10-0-140-107 kubenswrapper[2578]: I1103 00:40:10.381233 2578 manager.go:217] Machine: {Timestamp:2026-11-03 00:40:10.380741591 +0000 UTC m=+0.326321082 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3097021 MemoryCapacity:33164337152 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec24d6a3924334a334030f111abe3d55 SystemUUID:ec24d6a3-9243-34a3-3403-0f111abe3d55 BootID:c386f19e-6a05-4b96-a313-40a9fad63fe5 Filesystems:[{Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:24 Capacity:16582168576 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:26 Capacity:6632869888 Type:vfs Inodes:819200 HasInodes:true} {Device:composefs_0-34 DeviceMajor:0 DeviceMinor:34 Capacity:6422528 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:32 Capacity:16582168576 Type:vfs Inodes:1048576 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:ff:f4:66:da:9b Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:ff:f4:66:da:9b Speed:0 Mtu:9001} {Name:ovs-system MacAddress:42:b8:eb:2d:a2:38 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164337152 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None} Nov 03 00:40:10.381936 ip-10-0-140-107 kubenswrapper[2578]: I1103 00:40:10.381925 2578 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available. Nov 03 00:40:10.382048 ip-10-0-140-107 kubenswrapper[2578]: I1103 00:40:10.382037 2578 manager.go:233] Version: {KernelVersion:5.14.0-687.60.1.el9_8.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.8.20261023-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:} Nov 03 00:40:10.383894 ip-10-0-140-107 kubenswrapper[2578]: I1103 00:40:10.383872 2578 container_manager_linux.go:275] "Container manager verified user specified cgroup-root exists" cgroupRoot=[] Nov 03 00:40:10.384048 ip-10-0-140-107 kubenswrapper[2578]: I1103 00:40:10.383896 2578 container_manager_linux.go:280] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-140-107.us-east-2.compute.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"/system.slice","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{},"system-reserved-compressible":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2} Nov 03 00:40:10.384134 ip-10-0-140-107 kubenswrapper[2578]: I1103 00:40:10.384051 2578 topology_manager.go:143] "Creating topology manager with none policy" Nov 03 00:40:10.384134 ip-10-0-140-107 kubenswrapper[2578]: I1103 00:40:10.384059 2578 container_manager_linux.go:311] "Creating device plugin manager" Nov 03 00:40:10.384134 ip-10-0-140-107 kubenswrapper[2578]: I1103 00:40:10.384086 2578 manager.go:144] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock" Nov 03 00:40:10.384134 ip-10-0-140-107 kubenswrapper[2578]: I1103 00:40:10.384101 2578 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock" Nov 03 00:40:10.384946 ip-10-0-140-107 kubenswrapper[2578]: I1103 00:40:10.384937 2578 container_manager_linux.go:320] "Creating Dynamic Resource Allocation (DRA) manager" Nov 03 00:40:10.384995 ip-10-0-140-107 kubenswrapper[2578]: I1103 00:40:10.384960 2578 reflector.go:446] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Nov 03 00:40:10.388540 ip-10-0-140-107 kubenswrapper[2578]: I1103 00:40:10.388527 2578 state_mem.go:41] "Initialized" logger="CPUManager state memory" Nov 03 00:40:10.388664 ip-10-0-140-107 kubenswrapper[2578]: I1103 00:40:10.388654 2578 server.go:1235] "Using root directory" path="/var/lib/kubelet" Nov 03 00:40:10.388693 ip-10-0-140-107 kubenswrapper[2578]: I1103 00:40:10.388689 2578 kubelet.go:493] "Attempting to sync node with API server" Nov 03 00:40:10.388719 ip-10-0-140-107 kubenswrapper[2578]: I1103 00:40:10.388700 2578 kubelet.go:394] "Adding static pod path" path="/etc/kubernetes/manifests" Nov 03 00:40:10.388719 ip-10-0-140-107 kubenswrapper[2578]: I1103 00:40:10.388710 2578 file.go:69] "Watching path" path="/etc/kubernetes/manifests" Nov 03 00:40:10.388719 ip-10-0-140-107 kubenswrapper[2578]: I1103 00:40:10.388719 2578 kubelet.go:405] "Adding apiserver pod source" Nov 03 00:40:10.388792 ip-10-0-140-107 kubenswrapper[2578]: I1103 00:40:10.388728 2578 apiserver.go:42] "Waiting for node sync before watching apiserver pods" Nov 03 00:40:10.390278 ip-10-0-140-107 kubenswrapper[2578]: I1103 00:40:10.390253 2578 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Nov 03 00:40:10.393496 ip-10-0-140-107 kubenswrapper[2578]: I1103 00:40:10.393475 2578 kuberuntime_manager.go:304] "Container runtime initialized" containerRuntime="cri-o" version="1.35.10-3.rhaos4.22.git9383502.el9" apiVersion="v1" Nov 03 00:40:10.397717 ip-10-0-140-107 kubenswrapper[2578]: I1103 00:40:10.397701 2578 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Nov 03 00:40:10.398507 ip-10-0-140-107 kubenswrapper[2578]: I1103 00:40:10.398493 2578 kubelet.go:963] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled" Nov 03 00:40:10.398565 ip-10-0-140-107 kubenswrapper[2578]: I1103 00:40:10.398522 2578 kubelet.go:990] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled" Nov 03 00:40:10.398683 ip-10-0-140-107 kubenswrapper[2578]: I1103 00:40:10.398673 2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume" Nov 03 00:40:10.398713 ip-10-0-140-107 kubenswrapper[2578]: I1103 00:40:10.398689 2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir" Nov 03 00:40:10.398713 ip-10-0-140-107 kubenswrapper[2578]: I1103 00:40:10.398696 2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo" Nov 03 00:40:10.398713 ip-10-0-140-107 kubenswrapper[2578]: I1103 00:40:10.398702 2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path" Nov 03 00:40:10.398713 ip-10-0-140-107 kubenswrapper[2578]: I1103 00:40:10.398707 2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs" Nov 03 00:40:10.398713 ip-10-0-140-107 kubenswrapper[2578]: I1103 00:40:10.398713 2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret" Nov 03 00:40:10.399444 ip-10-0-140-107 kubenswrapper[2578]: I1103 00:40:10.399431 2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi" Nov 03 00:40:10.399444 ip-10-0-140-107 kubenswrapper[2578]: I1103 00:40:10.399444 2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api" Nov 03 00:40:10.399574 ip-10-0-140-107 kubenswrapper[2578]: I1103 00:40:10.399454 2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc" Nov 03 00:40:10.399574 ip-10-0-140-107 kubenswrapper[2578]: I1103 00:40:10.399463 2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap" Nov 03 00:40:10.399574 ip-10-0-140-107 kubenswrapper[2578]: I1103 00:40:10.399474 2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected" Nov 03 00:40:10.399574 ip-10-0-140-107 kubenswrapper[2578]: I1103 00:40:10.399488 2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume" Nov 03 00:40:10.401108 ip-10-0-140-107 kubenswrapper[2578]: I1103 00:40:10.401096 2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi" Nov 03 00:40:10.401164 ip-10-0-140-107 kubenswrapper[2578]: I1103 00:40:10.401112 2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image" Nov 03 00:40:10.405014 ip-10-0-140-107 kubenswrapper[2578]: I1103 00:40:10.404999 2578 server.go:1263] "Started kubelet" Nov 03 00:40:10.405137 ip-10-0-140-107 kubenswrapper[2578]: I1103 00:40:10.405095 2578 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10 Nov 03 00:40:10.405188 ip-10-0-140-107 kubenswrapper[2578]: I1103 00:40:10.405154 2578 server_v1.go:49] "podresources" method="list" useActivePods=true Nov 03 00:40:10.405604 ip-10-0-140-107 kubenswrapper[2578]: I1103 00:40:10.405580 2578 server.go:182] "Starting to listen" address="0.0.0.0" port=10250 Nov 03 00:40:10.405691 ip-10-0-140-107 systemd[1]: Started Kubernetes Kubelet. Nov 03 00:40:10.406227 ip-10-0-140-107 kubenswrapper[2578]: I1103 00:40:10.406210 2578 server.go:254] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock" Nov 03 00:40:10.407568 ip-10-0-140-107 kubenswrapper[2578]: I1103 00:40:10.407550 2578 server.go:317] "Adding debug handlers to kubelet server" Nov 03 00:40:10.412677 ip-10-0-140-107 kubenswrapper[2578]: I1103 00:40:10.412657 2578 fs_resource_analyzer.go:69] "Starting FS ResourceAnalyzer" Nov 03 00:40:10.412763 ip-10-0-140-107 kubenswrapper[2578]: I1103 00:40:10.412681 2578 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving" Nov 03 00:40:10.412818 ip-10-0-140-107 kubenswrapper[2578]: E1103 00:40:10.412780 2578 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-140-107.us-east-2.compute.internal\" not found" Nov 03 00:40:10.413215 ip-10-0-140-107 kubenswrapper[2578]: E1103 00:40:10.413179 2578 kubelet.go:1679] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache" Nov 03 00:40:10.413520 ip-10-0-140-107 kubenswrapper[2578]: I1103 00:40:10.413505 2578 volume_manager.go:309] "The desired_state_of_world populator starts" Nov 03 00:40:10.413604 ip-10-0-140-107 kubenswrapper[2578]: I1103 00:40:10.413525 2578 volume_manager.go:311] "Starting Kubelet Volume Manager" Nov 03 00:40:10.413668 ip-10-0-140-107 kubenswrapper[2578]: I1103 00:40:10.413608 2578 desired_state_of_world_populator.go:146] "Desired state populator starts to run" Nov 03 00:40:10.413718 ip-10-0-140-107 kubenswrapper[2578]: I1103 00:40:10.413707 2578 reconstruct.go:97] "Volume reconstruction finished" Nov 03 00:40:10.413718 ip-10-0-140-107 kubenswrapper[2578]: I1103 00:40:10.413715 2578 reconciler.go:29] "Reconciler: start to sync state" Nov 03 00:40:10.413815 ip-10-0-140-107 kubenswrapper[2578]: E1103 00:40:10.413737 2578 reflector.go:204] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-140-107.us-east-2.compute.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.Node" Nov 03 00:40:10.413815 ip-10-0-140-107 kubenswrapper[2578]: E1103 00:40:10.413764 2578 reflector.go:204] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.Service" Nov 03 00:40:10.413940 ip-10-0-140-107 kubenswrapper[2578]: I1103 00:40:10.413827 2578 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-140-107.us-east-2.compute.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope Nov 03 00:40:10.414506 ip-10-0-140-107 kubenswrapper[2578]: E1103 00:40:10.413496 2578 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-140-107.us-east-2.compute.internal.18e46218d5bd077e default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-140-107.us-east-2.compute.internal,UID:,APIVersion:v1,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-140-107.us-east-2.compute.internal,},FirstTimestamp:2026-11-03 00:40:10.404972414 +0000 UTC m=+0.350551903,LastTimestamp:2026-11-03 00:40:10.404972414 +0000 UTC m=+0.350551903,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-140-107.us-east-2.compute.internal,}" Nov 03 00:40:10.414669 ip-10-0-140-107 kubenswrapper[2578]: I1103 00:40:10.414517 2578 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory Nov 03 00:40:10.414669 ip-10-0-140-107 kubenswrapper[2578]: I1103 00:40:10.414529 2578 factory.go:55] Registering systemd factory Nov 03 00:40:10.414669 ip-10-0-140-107 kubenswrapper[2578]: I1103 00:40:10.414537 2578 factory.go:223] Registration of the systemd container factory successfully Nov 03 00:40:10.414811 ip-10-0-140-107 kubenswrapper[2578]: I1103 00:40:10.414763 2578 factory.go:153] Registering CRI-O factory Nov 03 00:40:10.414811 ip-10-0-140-107 kubenswrapper[2578]: I1103 00:40:10.414777 2578 factory.go:223] Registration of the crio container factory successfully Nov 03 00:40:10.414811 ip-10-0-140-107 kubenswrapper[2578]: I1103 00:40:10.414795 2578 factory.go:103] Registering Raw factory Nov 03 00:40:10.414953 ip-10-0-140-107 kubenswrapper[2578]: I1103 00:40:10.414819 2578 manager.go:1199] Started watching for new ooms in manager Nov 03 00:40:10.415597 ip-10-0-140-107 kubenswrapper[2578]: I1103 00:40:10.415575 2578 manager.go:319] Starting recovery of all containers Nov 03 00:40:10.420380 ip-10-0-140-107 kubenswrapper[2578]: E1103 00:40:10.420347 2578 controller.go:201] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-140-107.us-east-2.compute.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms" Nov 03 00:40:10.422243 ip-10-0-140-107 kubenswrapper[2578]: E1103 00:40:10.422216 2578 reflector.go:204] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.CSIDriver" Nov 03 00:40:10.427452 ip-10-0-140-107 kubenswrapper[2578]: I1103 00:40:10.427437 2578 manager.go:324] Recovery completed Nov 03 00:40:10.431292 ip-10-0-140-107 kubenswrapper[2578]: I1103 00:40:10.431279 2578 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Nov 03 00:40:10.433055 ip-10-0-140-107 kubenswrapper[2578]: I1103 00:40:10.433001 2578 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-140-107.us-east-2.compute.internal" event="NodeHasSufficientMemory" Nov 03 00:40:10.433055 ip-10-0-140-107 kubenswrapper[2578]: I1103 00:40:10.433028 2578 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-140-107.us-east-2.compute.internal" event="NodeHasNoDiskPressure" Nov 03 00:40:10.433055 ip-10-0-140-107 kubenswrapper[2578]: I1103 00:40:10.433042 2578 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-140-107.us-east-2.compute.internal" event="NodeHasSufficientPID" Nov 03 00:40:10.434244 ip-10-0-140-107 kubenswrapper[2578]: I1103 00:40:10.434220 2578 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-jjvx9" Nov 03 00:40:10.434347 ip-10-0-140-107 kubenswrapper[2578]: I1103 00:40:10.434288 2578 cpu_manager.go:226] "Starting" policy="none" Nov 03 00:40:10.434347 ip-10-0-140-107 kubenswrapper[2578]: I1103 00:40:10.434301 2578 cpu_manager.go:227] "Reconciling" reconcilePeriod="10s" Nov 03 00:40:10.434347 ip-10-0-140-107 kubenswrapper[2578]: I1103 00:40:10.434307 2578 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Nov 03 00:40:10.434347 ip-10-0-140-107 kubenswrapper[2578]: I1103 00:40:10.434319 2578 state_mem.go:41] "Initialized" logger="CPUManager state checkpoint.CPUManager state memory" Nov 03 00:40:10.435017 ip-10-0-140-107 kubenswrapper[2578]: I1103 00:40:10.435001 2578 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-140-107.us-east-2.compute.internal" event="NodeHasSufficientMemory" Nov 03 00:40:10.435089 ip-10-0-140-107 kubenswrapper[2578]: I1103 00:40:10.435028 2578 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-140-107.us-east-2.compute.internal" event="NodeHasNoDiskPressure" Nov 03 00:40:10.435089 ip-10-0-140-107 kubenswrapper[2578]: I1103 00:40:10.435043 2578 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-140-107.us-east-2.compute.internal" event="NodeHasSufficientPID" Nov 03 00:40:10.436539 ip-10-0-140-107 kubenswrapper[2578]: I1103 00:40:10.436524 2578 policy_none.go:50] "Start" Nov 03 00:40:10.436539 ip-10-0-140-107 kubenswrapper[2578]: I1103 00:40:10.436539 2578 memory_manager.go:187] "Starting memorymanager" policy="None" Nov 03 00:40:10.436619 ip-10-0-140-107 kubenswrapper[2578]: I1103 00:40:10.436557 2578 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint" Nov 03 00:40:10.438586 ip-10-0-140-107 kubenswrapper[2578]: I1103 00:40:10.438574 2578 policy_none.go:44] "Start" Nov 03 00:40:10.442336 ip-10-0-140-107 kubenswrapper[2578]: I1103 00:40:10.442316 2578 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-jjvx9" Nov 03 00:40:10.445351 ip-10-0-140-107 kubenswrapper[2578]: E1103 00:40:10.445259 2578 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-140-107.us-east-2.compute.internal.18e46218d768ecae default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-140-107.us-east-2.compute.internal,UID:,APIVersion:v1,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-140-107.us-east-2.compute.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-140-107.us-east-2.compute.internal,},FirstTimestamp:2026-11-03 00:40:10.433014958 +0000 UTC m=+0.378594449,LastTimestamp:2026-11-03 00:40:10.433014958 +0000 UTC m=+0.378594449,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-140-107.us-east-2.compute.internal,}" Nov 03 00:40:10.463485 ip-10-0-140-107 kubenswrapper[2578]: I1103 00:40:10.463455 2578 node_container_manager_linux.go:130] "Enforcing system reserved compressible on cgroup" cgroupName="/system.slice" limits={"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"} Nov 03 00:40:10.467499 ip-10-0-140-107 kubenswrapper[2578]: I1103 00:40:10.467485 2578 manager.go:341] "Starting Device Plugin manager" Nov 03 00:40:10.467571 ip-10-0-140-107 kubenswrapper[2578]: E1103 00:40:10.467523 2578 manager.go:525] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint" Nov 03 00:40:10.467571 ip-10-0-140-107 kubenswrapper[2578]: I1103 00:40:10.467538 2578 server.go:87] "Starting device plugin registration server" Nov 03 00:40:10.467820 ip-10-0-140-107 kubenswrapper[2578]: I1103 00:40:10.467804 2578 eviction_manager.go:194] "Eviction manager: starting control loop" Nov 03 00:40:10.467896 ip-10-0-140-107 kubenswrapper[2578]: I1103 00:40:10.467820 2578 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s" Nov 03 00:40:10.468708 ip-10-0-140-107 kubenswrapper[2578]: I1103 00:40:10.468013 2578 plugin_watcher.go:53] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry" Nov 03 00:40:10.468708 ip-10-0-140-107 kubenswrapper[2578]: I1103 00:40:10.468103 2578 plugin_manager.go:119] "The desired_state_of_world populator (plugin watcher) starts" Nov 03 00:40:10.468708 ip-10-0-140-107 kubenswrapper[2578]: I1103 00:40:10.468110 2578 plugin_manager.go:121] "Starting Kubelet Plugin Manager" Nov 03 00:40:10.468851 ip-10-0-140-107 kubenswrapper[2578]: E1103 00:40:10.468725 2578 eviction_manager.go:272] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\"" Nov 03 00:40:10.468851 ip-10-0-140-107 kubenswrapper[2578]: E1103 00:40:10.468765 2578 eviction_manager.go:297] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-140-107.us-east-2.compute.internal\" not found" Nov 03 00:40:10.568451 ip-10-0-140-107 kubenswrapper[2578]: I1103 00:40:10.568413 2578 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Nov 03 00:40:10.569563 ip-10-0-140-107 kubenswrapper[2578]: I1103 00:40:10.569544 2578 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-140-107.us-east-2.compute.internal" event="NodeHasSufficientMemory" Nov 03 00:40:10.569669 ip-10-0-140-107 kubenswrapper[2578]: I1103 00:40:10.569578 2578 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-140-107.us-east-2.compute.internal" event="NodeHasNoDiskPressure" Nov 03 00:40:10.569669 ip-10-0-140-107 kubenswrapper[2578]: I1103 00:40:10.569592 2578 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-140-107.us-east-2.compute.internal" event="NodeHasSufficientPID" Nov 03 00:40:10.569669 ip-10-0-140-107 kubenswrapper[2578]: I1103 00:40:10.569625 2578 kubelet_node_status.go:77] "Attempting to register node" node="ip-10-0-140-107.us-east-2.compute.internal" Nov 03 00:40:10.580400 ip-10-0-140-107 kubenswrapper[2578]: I1103 00:40:10.580385 2578 kubelet_node_status.go:80] "Successfully registered node" node="ip-10-0-140-107.us-east-2.compute.internal" Nov 03 00:40:10.580445 ip-10-0-140-107 kubenswrapper[2578]: E1103 00:40:10.580408 2578 kubelet_node_status.go:524] "Error updating node status, will retry" err="error getting node \"ip-10-0-140-107.us-east-2.compute.internal\": node \"ip-10-0-140-107.us-east-2.compute.internal\" not found" Nov 03 00:40:10.599760 ip-10-0-140-107 kubenswrapper[2578]: E1103 00:40:10.599715 2578 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-140-107.us-east-2.compute.internal\" not found" Nov 03 00:40:10.603144 ip-10-0-140-107 kubenswrapper[2578]: I1103 00:40:10.603115 2578 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4" Nov 03 00:40:10.604275 ip-10-0-140-107 kubenswrapper[2578]: I1103 00:40:10.604250 2578 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6" Nov 03 00:40:10.604275 ip-10-0-140-107 kubenswrapper[2578]: I1103 00:40:10.604272 2578 status_manager.go:249] "Starting to sync pod status with apiserver" Nov 03 00:40:10.604380 ip-10-0-140-107 kubenswrapper[2578]: I1103 00:40:10.604293 2578 kubelet.go:2521] "Starting kubelet main sync loop" Nov 03 00:40:10.604380 ip-10-0-140-107 kubenswrapper[2578]: E1103 00:40:10.604333 2578 kubelet.go:2545] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful" Nov 03 00:40:10.612553 ip-10-0-140-107 kubenswrapper[2578]: I1103 00:40:10.612537 2578 reflector.go:446] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:161" Nov 03 00:40:10.700335 ip-10-0-140-107 kubenswrapper[2578]: E1103 00:40:10.700287 2578 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-140-107.us-east-2.compute.internal\" not found" Nov 03 00:40:10.705467 ip-10-0-140-107 kubenswrapper[2578]: I1103 00:40:10.705447 2578 kubelet.go:2608] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-107.us-east-2.compute.internal","kube-system/kube-apiserver-proxy-ip-10-0-140-107.us-east-2.compute.internal"] Nov 03 00:40:10.705543 ip-10-0-140-107 kubenswrapper[2578]: I1103 00:40:10.705532 2578 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Nov 03 00:40:10.706556 ip-10-0-140-107 kubenswrapper[2578]: I1103 00:40:10.706540 2578 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-140-107.us-east-2.compute.internal" event="NodeHasSufficientMemory" Nov 03 00:40:10.706659 ip-10-0-140-107 kubenswrapper[2578]: I1103 00:40:10.706570 2578 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-140-107.us-east-2.compute.internal" event="NodeHasNoDiskPressure" Nov 03 00:40:10.706659 ip-10-0-140-107 kubenswrapper[2578]: I1103 00:40:10.706582 2578 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-140-107.us-east-2.compute.internal" event="NodeHasSufficientPID" Nov 03 00:40:10.708859 ip-10-0-140-107 kubenswrapper[2578]: I1103 00:40:10.708848 2578 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Nov 03 00:40:10.709019 ip-10-0-140-107 kubenswrapper[2578]: I1103 00:40:10.709004 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-107.us-east-2.compute.internal" Nov 03 00:40:10.709101 ip-10-0-140-107 kubenswrapper[2578]: I1103 00:40:10.709034 2578 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Nov 03 00:40:10.709613 ip-10-0-140-107 kubenswrapper[2578]: I1103 00:40:10.709597 2578 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-140-107.us-east-2.compute.internal" event="NodeHasSufficientMemory" Nov 03 00:40:10.709658 ip-10-0-140-107 kubenswrapper[2578]: I1103 00:40:10.709618 2578 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-140-107.us-east-2.compute.internal" event="NodeHasSufficientMemory" Nov 03 00:40:10.709658 ip-10-0-140-107 kubenswrapper[2578]: I1103 00:40:10.709640 2578 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-140-107.us-east-2.compute.internal" event="NodeHasNoDiskPressure" Nov 03 00:40:10.709658 ip-10-0-140-107 kubenswrapper[2578]: I1103 00:40:10.709649 2578 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-140-107.us-east-2.compute.internal" event="NodeHasSufficientPID" Nov 03 00:40:10.709739 ip-10-0-140-107 kubenswrapper[2578]: I1103 00:40:10.709623 2578 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-140-107.us-east-2.compute.internal" event="NodeHasNoDiskPressure" Nov 03 00:40:10.709739 ip-10-0-140-107 kubenswrapper[2578]: I1103 00:40:10.709718 2578 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-140-107.us-east-2.compute.internal" event="NodeHasSufficientPID" Nov 03 00:40:10.713303 ip-10-0-140-107 kubenswrapper[2578]: I1103 00:40:10.712014 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-140-107.us-east-2.compute.internal" Nov 03 00:40:10.713303 ip-10-0-140-107 kubenswrapper[2578]: I1103 00:40:10.712050 2578 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Nov 03 00:40:10.713703 ip-10-0-140-107 kubenswrapper[2578]: I1103 00:40:10.713683 2578 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-140-107.us-east-2.compute.internal" event="NodeHasSufficientMemory" Nov 03 00:40:10.713764 ip-10-0-140-107 kubenswrapper[2578]: I1103 00:40:10.713710 2578 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-140-107.us-east-2.compute.internal" event="NodeHasNoDiskPressure" Nov 03 00:40:10.713764 ip-10-0-140-107 kubenswrapper[2578]: I1103 00:40:10.713754 2578 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-140-107.us-east-2.compute.internal" event="NodeHasSufficientPID" Nov 03 00:40:10.714724 ip-10-0-140-107 kubenswrapper[2578]: I1103 00:40:10.714709 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/0fb35ee5f976a9435ef306be01bc3e71-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-140-107.us-east-2.compute.internal\" (UID: \"0fb35ee5f976a9435ef306be01bc3e71\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-107.us-east-2.compute.internal" Nov 03 00:40:10.714791 ip-10-0-140-107 kubenswrapper[2578]: I1103 00:40:10.714732 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/0fb35ee5f976a9435ef306be01bc3e71-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-140-107.us-east-2.compute.internal\" (UID: \"0fb35ee5f976a9435ef306be01bc3e71\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-107.us-east-2.compute.internal" Nov 03 00:40:10.714791 ip-10-0-140-107 kubenswrapper[2578]: I1103 00:40:10.714749 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/a730b11d5a720feea75c07e6e10d3b92-config\") pod \"kube-apiserver-proxy-ip-10-0-140-107.us-east-2.compute.internal\" (UID: \"a730b11d5a720feea75c07e6e10d3b92\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-140-107.us-east-2.compute.internal" Nov 03 00:40:10.748478 ip-10-0-140-107 kubenswrapper[2578]: E1103 00:40:10.748461 2578 kubelet.go:3364] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-140-107.us-east-2.compute.internal\" not found" node="ip-10-0-140-107.us-east-2.compute.internal" Nov 03 00:40:10.750476 ip-10-0-140-107 kubenswrapper[2578]: E1103 00:40:10.750461 2578 kubelet.go:3364] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-140-107.us-east-2.compute.internal\" not found" node="ip-10-0-140-107.us-east-2.compute.internal" Nov 03 00:40:10.800551 ip-10-0-140-107 kubenswrapper[2578]: E1103 00:40:10.800402 2578 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-140-107.us-east-2.compute.internal\" not found" Nov 03 00:40:10.815702 ip-10-0-140-107 kubenswrapper[2578]: I1103 00:40:10.815682 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/0fb35ee5f976a9435ef306be01bc3e71-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-140-107.us-east-2.compute.internal\" (UID: \"0fb35ee5f976a9435ef306be01bc3e71\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-107.us-east-2.compute.internal" Nov 03 00:40:10.815775 ip-10-0-140-107 kubenswrapper[2578]: I1103 00:40:10.815708 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/0fb35ee5f976a9435ef306be01bc3e71-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-140-107.us-east-2.compute.internal\" (UID: \"0fb35ee5f976a9435ef306be01bc3e71\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-107.us-east-2.compute.internal" Nov 03 00:40:10.815775 ip-10-0-140-107 kubenswrapper[2578]: I1103 00:40:10.815725 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/a730b11d5a720feea75c07e6e10d3b92-config\") pod \"kube-apiserver-proxy-ip-10-0-140-107.us-east-2.compute.internal\" (UID: \"a730b11d5a720feea75c07e6e10d3b92\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-140-107.us-east-2.compute.internal" Nov 03 00:40:10.815843 ip-10-0-140-107 kubenswrapper[2578]: I1103 00:40:10.815782 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/0fb35ee5f976a9435ef306be01bc3e71-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-140-107.us-east-2.compute.internal\" (UID: \"0fb35ee5f976a9435ef306be01bc3e71\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-107.us-east-2.compute.internal" Nov 03 00:40:10.815843 ip-10-0-140-107 kubenswrapper[2578]: I1103 00:40:10.815793 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/0fb35ee5f976a9435ef306be01bc3e71-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-140-107.us-east-2.compute.internal\" (UID: \"0fb35ee5f976a9435ef306be01bc3e71\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-107.us-east-2.compute.internal" Nov 03 00:40:10.815843 ip-10-0-140-107 kubenswrapper[2578]: I1103 00:40:10.815783 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/a730b11d5a720feea75c07e6e10d3b92-config\") pod \"kube-apiserver-proxy-ip-10-0-140-107.us-east-2.compute.internal\" (UID: \"a730b11d5a720feea75c07e6e10d3b92\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-140-107.us-east-2.compute.internal" Nov 03 00:40:10.901126 ip-10-0-140-107 kubenswrapper[2578]: E1103 00:40:10.901059 2578 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-140-107.us-east-2.compute.internal\" not found" Nov 03 00:40:11.001647 ip-10-0-140-107 kubenswrapper[2578]: E1103 00:40:11.001609 2578 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-140-107.us-east-2.compute.internal\" not found" Nov 03 00:40:11.051033 ip-10-0-140-107 kubenswrapper[2578]: I1103 00:40:11.051007 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-107.us-east-2.compute.internal" Nov 03 00:40:11.053127 ip-10-0-140-107 kubenswrapper[2578]: I1103 00:40:11.053111 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-140-107.us-east-2.compute.internal" Nov 03 00:40:11.102628 ip-10-0-140-107 kubenswrapper[2578]: E1103 00:40:11.102601 2578 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-140-107.us-east-2.compute.internal\" not found" Nov 03 00:40:11.203165 ip-10-0-140-107 kubenswrapper[2578]: E1103 00:40:11.203078 2578 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-140-107.us-east-2.compute.internal\" not found" Nov 03 00:40:11.303673 ip-10-0-140-107 kubenswrapper[2578]: E1103 00:40:11.303645 2578 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-140-107.us-east-2.compute.internal\" not found" Nov 03 00:40:11.352115 ip-10-0-140-107 kubenswrapper[2578]: I1103 00:40:11.352093 2578 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials" Nov 03 00:40:11.352514 ip-10-0-140-107 kubenswrapper[2578]: I1103 00:40:11.352221 2578 reflector.go:578] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:161: Unexpected watch close - watch lasted less than a second and no items received" Nov 03 00:40:11.403735 ip-10-0-140-107 kubenswrapper[2578]: E1103 00:40:11.403707 2578 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-140-107.us-east-2.compute.internal\" not found" Nov 03 00:40:11.427739 ip-10-0-140-107 kubenswrapper[2578]: I1103 00:40:11.427715 2578 reflector.go:446] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:161" Nov 03 00:40:11.448264 ip-10-0-140-107 kubenswrapper[2578]: I1103 00:40:11.448211 2578 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-11-02 00:35:10 +0000 UTC" deadline="2028-08-08 00:47:12.574900704 +0000 UTC" Nov 03 00:40:11.448264 ip-10-0-140-107 kubenswrapper[2578]: I1103 00:40:11.448258 2578 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="15456h7m1.126646619s" Nov 03 00:40:11.460609 ip-10-0-140-107 kubenswrapper[2578]: I1103 00:40:11.460553 2578 reflector.go:446] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:161" Nov 03 00:40:11.498349 ip-10-0-140-107 kubenswrapper[2578]: I1103 00:40:11.498321 2578 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Nov 03 00:40:11.499431 ip-10-0-140-107 kubenswrapper[2578]: I1103 00:40:11.499413 2578 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-140-107.us-east-2.compute.internal" event="NodeHasSufficientMemory" Nov 03 00:40:11.499492 ip-10-0-140-107 kubenswrapper[2578]: I1103 00:40:11.499447 2578 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-140-107.us-east-2.compute.internal" event="NodeHasNoDiskPressure" Nov 03 00:40:11.499492 ip-10-0-140-107 kubenswrapper[2578]: I1103 00:40:11.499458 2578 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-140-107.us-east-2.compute.internal" event="NodeHasSufficientPID" Nov 03 00:40:11.504655 ip-10-0-140-107 kubenswrapper[2578]: E1103 00:40:11.504638 2578 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-140-107.us-east-2.compute.internal\" not found" Nov 03 00:40:11.605167 ip-10-0-140-107 kubenswrapper[2578]: E1103 00:40:11.605129 2578 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-140-107.us-east-2.compute.internal\" not found" Nov 03 00:40:11.706011 ip-10-0-140-107 kubenswrapper[2578]: E1103 00:40:11.705972 2578 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-140-107.us-east-2.compute.internal\" not found" Nov 03 00:40:11.732391 ip-10-0-140-107 kubenswrapper[2578]: W1103 00:40:11.732337 2578 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0fb35ee5f976a9435ef306be01bc3e71.slice/crio-20d31a990fda9ed8069aa304f22413ca1c459b7d9b9b0e268ac30fa12af2d633 WatchSource:0}: Error finding container 20d31a990fda9ed8069aa304f22413ca1c459b7d9b9b0e268ac30fa12af2d633: Status 404 returned error can't find the container with id 20d31a990fda9ed8069aa304f22413ca1c459b7d9b9b0e268ac30fa12af2d633 Nov 03 00:40:11.737851 ip-10-0-140-107 kubenswrapper[2578]: W1103 00:40:11.737829 2578 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda730b11d5a720feea75c07e6e10d3b92.slice/crio-a012e88fc324e4ecd1bc5f9557af5fe91bc73d451cee98e718e6ecf7cf8da48f WatchSource:0}: Error finding container a012e88fc324e4ecd1bc5f9557af5fe91bc73d451cee98e718e6ecf7cf8da48f: Status 404 returned error can't find the container with id a012e88fc324e4ecd1bc5f9557af5fe91bc73d451cee98e718e6ecf7cf8da48f Nov 03 00:40:11.739107 ip-10-0-140-107 kubenswrapper[2578]: I1103 00:40:11.739087 2578 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Nov 03 00:40:11.806014 ip-10-0-140-107 kubenswrapper[2578]: I1103 00:40:11.805990 2578 reflector.go:446] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:161" Nov 03 00:40:11.814224 ip-10-0-140-107 kubenswrapper[2578]: I1103 00:40:11.814203 2578 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-107.us-east-2.compute.internal" Nov 03 00:40:11.836645 ip-10-0-140-107 kubenswrapper[2578]: I1103 00:40:11.836629 2578 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Nov 03 00:40:11.837724 ip-10-0-140-107 kubenswrapper[2578]: I1103 00:40:11.837711 2578 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-140-107.us-east-2.compute.internal" Nov 03 00:40:11.857469 ip-10-0-140-107 kubenswrapper[2578]: I1103 00:40:11.857451 2578 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Nov 03 00:40:12.172810 ip-10-0-140-107 kubenswrapper[2578]: I1103 00:40:12.172782 2578 reflector.go:446] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:161" Nov 03 00:40:12.389512 ip-10-0-140-107 kubenswrapper[2578]: I1103 00:40:12.389476 2578 apiserver.go:52] "Watching apiserver" Nov 03 00:40:12.401270 ip-10-0-140-107 kubenswrapper[2578]: I1103 00:40:12.400550 2578 reflector.go:446] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66" Nov 03 00:40:12.401270 ip-10-0-140-107 kubenswrapper[2578]: I1103 00:40:12.400899 2578 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kube-system/kube-apiserver-proxy-ip-10-0-140-107.us-east-2.compute.internal","openshift-cluster-node-tuning-operator/tuned-gtwxp","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-107.us-east-2.compute.internal","openshift-multus/multus-additional-cni-plugins-gqppj","openshift-multus/multus-tb9hd","openshift-ovn-kubernetes/ovnkube-node-psqpn","openshift-multus/network-metrics-daemon-rt2hx","openshift-network-operator/iptables-alerter-5spnk","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-br8vt","kube-system/global-pull-secret-syncer-jv7cw","kube-system/konnectivity-agent-82x8c","openshift-dns/node-resolver-cjj2h","openshift-image-registry/node-ca-bbk2h","openshift-network-diagnostics/network-check-target-w46kd"] Nov 03 00:40:12.403631 ip-10-0-140-107 kubenswrapper[2578]: I1103 00:40:12.403603 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-5spnk" Nov 03 00:40:12.406068 ip-10-0-140-107 kubenswrapper[2578]: I1103 00:40:12.406042 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-w46kd" Nov 03 00:40:12.406190 ip-10-0-140-107 kubenswrapper[2578]: E1103 00:40:12.406119 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-w46kd" podUID="8cd3ca22-e4ce-4896-ae31-97605af4e4c6" Nov 03 00:40:12.408779 ip-10-0-140-107 kubenswrapper[2578]: I1103 00:40:12.408745 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-rt2hx" Nov 03 00:40:12.408905 ip-10-0-140-107 kubenswrapper[2578]: E1103 00:40:12.408859 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-rt2hx" podUID="136086c0-4846-437a-ad98-3cfef50610a0" Nov 03 00:40:12.408905 ip-10-0-140-107 kubenswrapper[2578]: I1103 00:40:12.408897 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\"" Nov 03 00:40:12.409093 ip-10-0-140-107 kubenswrapper[2578]: I1103 00:40:12.409016 2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-w9l2l\"" Nov 03 00:40:12.409093 ip-10-0-140-107 kubenswrapper[2578]: I1103 00:40:12.409034 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\"" Nov 03 00:40:12.409185 ip-10-0-140-107 kubenswrapper[2578]: I1103 00:40:12.409138 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\"" Nov 03 00:40:12.411196 ip-10-0-140-107 kubenswrapper[2578]: I1103 00:40:12.411175 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-gqppj" Nov 03 00:40:12.414190 ip-10-0-140-107 kubenswrapper[2578]: I1103 00:40:12.414160 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-tb9hd" Nov 03 00:40:12.414318 ip-10-0-140-107 kubenswrapper[2578]: I1103 00:40:12.414157 2578 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving" Nov 03 00:40:12.416906 ip-10-0-140-107 kubenswrapper[2578]: I1103 00:40:12.416890 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-psqpn" Nov 03 00:40:12.420103 ip-10-0-140-107 kubenswrapper[2578]: I1103 00:40:12.420082 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-br8vt" Nov 03 00:40:12.423271 ip-10-0-140-107 kubenswrapper[2578]: I1103 00:40:12.423007 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-jv7cw" Nov 03 00:40:12.423271 ip-10-0-140-107 kubenswrapper[2578]: E1103 00:40:12.423070 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-jv7cw" podUID="cabd402c-79c5-4699-9c85-d415d362b4d0" Nov 03 00:40:12.428336 ip-10-0-140-107 kubenswrapper[2578]: I1103 00:40:12.426085 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/98ca3972-fd0c-47f5-9d26-569a58e18bd1-env-overrides\") pod \"ovnkube-node-psqpn\" (UID: \"98ca3972-fd0c-47f5-9d26-569a58e18bd1\") " pod="openshift-ovn-kubernetes/ovnkube-node-psqpn" Nov 03 00:40:12.428336 ip-10-0-140-107 kubenswrapper[2578]: I1103 00:40:12.426117 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/00dc2920-34f2-4e0a-892e-7759a16162d5-host-var-lib-cni-bin\") pod \"multus-tb9hd\" (UID: \"00dc2920-34f2-4e0a-892e-7759a16162d5\") " pod="openshift-multus/multus-tb9hd" Nov 03 00:40:12.428336 ip-10-0-140-107 kubenswrapper[2578]: I1103 00:40:12.426139 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/98ca3972-fd0c-47f5-9d26-569a58e18bd1-host-slash\") pod \"ovnkube-node-psqpn\" (UID: \"98ca3972-fd0c-47f5-9d26-569a58e18bd1\") " pod="openshift-ovn-kubernetes/ovnkube-node-psqpn" Nov 03 00:40:12.428336 ip-10-0-140-107 kubenswrapper[2578]: I1103 00:40:12.426156 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/98ca3972-fd0c-47f5-9d26-569a58e18bd1-run-openvswitch\") pod \"ovnkube-node-psqpn\" (UID: \"98ca3972-fd0c-47f5-9d26-569a58e18bd1\") " pod="openshift-ovn-kubernetes/ovnkube-node-psqpn" Nov 03 00:40:12.428336 ip-10-0-140-107 kubenswrapper[2578]: I1103 00:40:12.426173 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/98ca3972-fd0c-47f5-9d26-569a58e18bd1-ovnkube-script-lib\") pod \"ovnkube-node-psqpn\" (UID: \"98ca3972-fd0c-47f5-9d26-569a58e18bd1\") " pod="openshift-ovn-kubernetes/ovnkube-node-psqpn" Nov 03 00:40:12.428336 ip-10-0-140-107 kubenswrapper[2578]: I1103 00:40:12.426196 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-llh2g\" (UniqueName: \"kubernetes.io/projected/136086c0-4846-437a-ad98-3cfef50610a0-kube-api-access-llh2g\") pod \"network-metrics-daemon-rt2hx\" (UID: \"136086c0-4846-437a-ad98-3cfef50610a0\") " pod="openshift-multus/network-metrics-daemon-rt2hx" Nov 03 00:40:12.428336 ip-10-0-140-107 kubenswrapper[2578]: I1103 00:40:12.426215 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/6650035b-2220-491c-9683-8a03ccb7af6a-cni-binary-copy\") pod \"multus-additional-cni-plugins-gqppj\" (UID: \"6650035b-2220-491c-9683-8a03ccb7af6a\") " pod="openshift-multus/multus-additional-cni-plugins-gqppj" Nov 03 00:40:12.428336 ip-10-0-140-107 kubenswrapper[2578]: I1103 00:40:12.426231 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/00dc2920-34f2-4e0a-892e-7759a16162d5-multus-daemon-config\") pod \"multus-tb9hd\" (UID: \"00dc2920-34f2-4e0a-892e-7759a16162d5\") " pod="openshift-multus/multus-tb9hd" Nov 03 00:40:12.428336 ip-10-0-140-107 kubenswrapper[2578]: I1103 00:40:12.426246 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/98ca3972-fd0c-47f5-9d26-569a58e18bd1-host-kubelet\") pod \"ovnkube-node-psqpn\" (UID: \"98ca3972-fd0c-47f5-9d26-569a58e18bd1\") " pod="openshift-ovn-kubernetes/ovnkube-node-psqpn" Nov 03 00:40:12.428336 ip-10-0-140-107 kubenswrapper[2578]: I1103 00:40:12.426268 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/98ca3972-fd0c-47f5-9d26-569a58e18bd1-host-cni-netd\") pod \"ovnkube-node-psqpn\" (UID: \"98ca3972-fd0c-47f5-9d26-569a58e18bd1\") " pod="openshift-ovn-kubernetes/ovnkube-node-psqpn" Nov 03 00:40:12.428336 ip-10-0-140-107 kubenswrapper[2578]: I1103 00:40:12.426292 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4bhqh\" (UniqueName: \"kubernetes.io/projected/98ca3972-fd0c-47f5-9d26-569a58e18bd1-kube-api-access-4bhqh\") pod \"ovnkube-node-psqpn\" (UID: \"98ca3972-fd0c-47f5-9d26-569a58e18bd1\") " pod="openshift-ovn-kubernetes/ovnkube-node-psqpn" Nov 03 00:40:12.428336 ip-10-0-140-107 kubenswrapper[2578]: I1103 00:40:12.426314 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/c736fd3f-973f-4999-8d4e-c405079b2888-host-slash\") pod \"iptables-alerter-5spnk\" (UID: \"c736fd3f-973f-4999-8d4e-c405079b2888\") " pod="openshift-network-operator/iptables-alerter-5spnk" Nov 03 00:40:12.428336 ip-10-0-140-107 kubenswrapper[2578]: I1103 00:40:12.426329 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/136086c0-4846-437a-ad98-3cfef50610a0-metrics-certs\") pod \"network-metrics-daemon-rt2hx\" (UID: \"136086c0-4846-437a-ad98-3cfef50610a0\") " pod="openshift-multus/network-metrics-daemon-rt2hx" Nov 03 00:40:12.428336 ip-10-0-140-107 kubenswrapper[2578]: I1103 00:40:12.426350 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/00dc2920-34f2-4e0a-892e-7759a16162d5-multus-socket-dir-parent\") pod \"multus-tb9hd\" (UID: \"00dc2920-34f2-4e0a-892e-7759a16162d5\") " pod="openshift-multus/multus-tb9hd" Nov 03 00:40:12.428336 ip-10-0-140-107 kubenswrapper[2578]: I1103 00:40:12.426374 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/00dc2920-34f2-4e0a-892e-7759a16162d5-host-var-lib-cni-multus\") pod \"multus-tb9hd\" (UID: \"00dc2920-34f2-4e0a-892e-7759a16162d5\") " pod="openshift-multus/multus-tb9hd" Nov 03 00:40:12.428336 ip-10-0-140-107 kubenswrapper[2578]: I1103 00:40:12.426398 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/00dc2920-34f2-4e0a-892e-7759a16162d5-hostroot\") pod \"multus-tb9hd\" (UID: \"00dc2920-34f2-4e0a-892e-7759a16162d5\") " pod="openshift-multus/multus-tb9hd" Nov 03 00:40:12.429170 ip-10-0-140-107 kubenswrapper[2578]: I1103 00:40:12.426419 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/6650035b-2220-491c-9683-8a03ccb7af6a-cnibin\") pod \"multus-additional-cni-plugins-gqppj\" (UID: \"6650035b-2220-491c-9683-8a03ccb7af6a\") " pod="openshift-multus/multus-additional-cni-plugins-gqppj" Nov 03 00:40:12.429170 ip-10-0-140-107 kubenswrapper[2578]: I1103 00:40:12.426445 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/00dc2920-34f2-4e0a-892e-7759a16162d5-cni-binary-copy\") pod \"multus-tb9hd\" (UID: \"00dc2920-34f2-4e0a-892e-7759a16162d5\") " pod="openshift-multus/multus-tb9hd" Nov 03 00:40:12.429170 ip-10-0-140-107 kubenswrapper[2578]: I1103 00:40:12.426471 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/00dc2920-34f2-4e0a-892e-7759a16162d5-host-run-multus-certs\") pod \"multus-tb9hd\" (UID: \"00dc2920-34f2-4e0a-892e-7759a16162d5\") " pod="openshift-multus/multus-tb9hd" Nov 03 00:40:12.429170 ip-10-0-140-107 kubenswrapper[2578]: I1103 00:40:12.426496 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j62gg\" (UniqueName: \"kubernetes.io/projected/00dc2920-34f2-4e0a-892e-7759a16162d5-kube-api-access-j62gg\") pod \"multus-tb9hd\" (UID: \"00dc2920-34f2-4e0a-892e-7759a16162d5\") " pod="openshift-multus/multus-tb9hd" Nov 03 00:40:12.429170 ip-10-0-140-107 kubenswrapper[2578]: I1103 00:40:12.426513 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/98ca3972-fd0c-47f5-9d26-569a58e18bd1-host-run-netns\") pod \"ovnkube-node-psqpn\" (UID: \"98ca3972-fd0c-47f5-9d26-569a58e18bd1\") " pod="openshift-ovn-kubernetes/ovnkube-node-psqpn" Nov 03 00:40:12.429170 ip-10-0-140-107 kubenswrapper[2578]: I1103 00:40:12.426531 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/98ca3972-fd0c-47f5-9d26-569a58e18bd1-run-systemd\") pod \"ovnkube-node-psqpn\" (UID: \"98ca3972-fd0c-47f5-9d26-569a58e18bd1\") " pod="openshift-ovn-kubernetes/ovnkube-node-psqpn" Nov 03 00:40:12.429170 ip-10-0-140-107 kubenswrapper[2578]: I1103 00:40:12.426558 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/98ca3972-fd0c-47f5-9d26-569a58e18bd1-host-cni-bin\") pod \"ovnkube-node-psqpn\" (UID: \"98ca3972-fd0c-47f5-9d26-569a58e18bd1\") " pod="openshift-ovn-kubernetes/ovnkube-node-psqpn" Nov 03 00:40:12.429170 ip-10-0-140-107 kubenswrapper[2578]: I1103 00:40:12.426584 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2tdmf\" (UniqueName: \"kubernetes.io/projected/c736fd3f-973f-4999-8d4e-c405079b2888-kube-api-access-2tdmf\") pod \"iptables-alerter-5spnk\" (UID: \"c736fd3f-973f-4999-8d4e-c405079b2888\") " pod="openshift-network-operator/iptables-alerter-5spnk" Nov 03 00:40:12.429170 ip-10-0-140-107 kubenswrapper[2578]: I1103 00:40:12.426606 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d8l25\" (UniqueName: \"kubernetes.io/projected/8cd3ca22-e4ce-4896-ae31-97605af4e4c6-kube-api-access-d8l25\") pod \"network-check-target-w46kd\" (UID: \"8cd3ca22-e4ce-4896-ae31-97605af4e4c6\") " pod="openshift-network-diagnostics/network-check-target-w46kd" Nov 03 00:40:12.429170 ip-10-0-140-107 kubenswrapper[2578]: I1103 00:40:12.426631 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rkf7s\" (UniqueName: \"kubernetes.io/projected/6650035b-2220-491c-9683-8a03ccb7af6a-kube-api-access-rkf7s\") pod \"multus-additional-cni-plugins-gqppj\" (UID: \"6650035b-2220-491c-9683-8a03ccb7af6a\") " pod="openshift-multus/multus-additional-cni-plugins-gqppj" Nov 03 00:40:12.429170 ip-10-0-140-107 kubenswrapper[2578]: I1103 00:40:12.426667 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/00dc2920-34f2-4e0a-892e-7759a16162d5-multus-cni-dir\") pod \"multus-tb9hd\" (UID: \"00dc2920-34f2-4e0a-892e-7759a16162d5\") " pod="openshift-multus/multus-tb9hd" Nov 03 00:40:12.429170 ip-10-0-140-107 kubenswrapper[2578]: I1103 00:40:12.426691 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/98ca3972-fd0c-47f5-9d26-569a58e18bd1-etc-openvswitch\") pod \"ovnkube-node-psqpn\" (UID: \"98ca3972-fd0c-47f5-9d26-569a58e18bd1\") " pod="openshift-ovn-kubernetes/ovnkube-node-psqpn" Nov 03 00:40:12.429170 ip-10-0-140-107 kubenswrapper[2578]: I1103 00:40:12.426715 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/98ca3972-fd0c-47f5-9d26-569a58e18bd1-run-ovn\") pod \"ovnkube-node-psqpn\" (UID: \"98ca3972-fd0c-47f5-9d26-569a58e18bd1\") " pod="openshift-ovn-kubernetes/ovnkube-node-psqpn" Nov 03 00:40:12.429170 ip-10-0-140-107 kubenswrapper[2578]: I1103 00:40:12.426740 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/98ca3972-fd0c-47f5-9d26-569a58e18bd1-node-log\") pod \"ovnkube-node-psqpn\" (UID: \"98ca3972-fd0c-47f5-9d26-569a58e18bd1\") " pod="openshift-ovn-kubernetes/ovnkube-node-psqpn" Nov 03 00:40:12.429170 ip-10-0-140-107 kubenswrapper[2578]: I1103 00:40:12.426764 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/98ca3972-fd0c-47f5-9d26-569a58e18bd1-ovnkube-config\") pod \"ovnkube-node-psqpn\" (UID: \"98ca3972-fd0c-47f5-9d26-569a58e18bd1\") " pod="openshift-ovn-kubernetes/ovnkube-node-psqpn" Nov 03 00:40:12.429170 ip-10-0-140-107 kubenswrapper[2578]: I1103 00:40:12.426790 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/6650035b-2220-491c-9683-8a03ccb7af6a-system-cni-dir\") pod \"multus-additional-cni-plugins-gqppj\" (UID: \"6650035b-2220-491c-9683-8a03ccb7af6a\") " pod="openshift-multus/multus-additional-cni-plugins-gqppj" Nov 03 00:40:12.429952 ip-10-0-140-107 kubenswrapper[2578]: I1103 00:40:12.426816 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/6650035b-2220-491c-9683-8a03ccb7af6a-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-gqppj\" (UID: \"6650035b-2220-491c-9683-8a03ccb7af6a\") " pod="openshift-multus/multus-additional-cni-plugins-gqppj" Nov 03 00:40:12.429952 ip-10-0-140-107 kubenswrapper[2578]: I1103 00:40:12.426845 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/00dc2920-34f2-4e0a-892e-7759a16162d5-system-cni-dir\") pod \"multus-tb9hd\" (UID: \"00dc2920-34f2-4e0a-892e-7759a16162d5\") " pod="openshift-multus/multus-tb9hd" Nov 03 00:40:12.429952 ip-10-0-140-107 kubenswrapper[2578]: I1103 00:40:12.426881 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/00dc2920-34f2-4e0a-892e-7759a16162d5-cnibin\") pod \"multus-tb9hd\" (UID: \"00dc2920-34f2-4e0a-892e-7759a16162d5\") " pod="openshift-multus/multus-tb9hd" Nov 03 00:40:12.429952 ip-10-0-140-107 kubenswrapper[2578]: I1103 00:40:12.426900 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/00dc2920-34f2-4e0a-892e-7759a16162d5-host-var-lib-kubelet\") pod \"multus-tb9hd\" (UID: \"00dc2920-34f2-4e0a-892e-7759a16162d5\") " pod="openshift-multus/multus-tb9hd" Nov 03 00:40:12.429952 ip-10-0-140-107 kubenswrapper[2578]: I1103 00:40:12.426922 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/98ca3972-fd0c-47f5-9d26-569a58e18bd1-var-lib-openvswitch\") pod \"ovnkube-node-psqpn\" (UID: \"98ca3972-fd0c-47f5-9d26-569a58e18bd1\") " pod="openshift-ovn-kubernetes/ovnkube-node-psqpn" Nov 03 00:40:12.429952 ip-10-0-140-107 kubenswrapper[2578]: I1103 00:40:12.426948 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/98ca3972-fd0c-47f5-9d26-569a58e18bd1-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-psqpn\" (UID: \"98ca3972-fd0c-47f5-9d26-569a58e18bd1\") " pod="openshift-ovn-kubernetes/ovnkube-node-psqpn" Nov 03 00:40:12.429952 ip-10-0-140-107 kubenswrapper[2578]: I1103 00:40:12.426980 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/c736fd3f-973f-4999-8d4e-c405079b2888-iptables-alerter-script\") pod \"iptables-alerter-5spnk\" (UID: \"c736fd3f-973f-4999-8d4e-c405079b2888\") " pod="openshift-network-operator/iptables-alerter-5spnk" Nov 03 00:40:12.429952 ip-10-0-140-107 kubenswrapper[2578]: I1103 00:40:12.427027 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/6650035b-2220-491c-9683-8a03ccb7af6a-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-gqppj\" (UID: \"6650035b-2220-491c-9683-8a03ccb7af6a\") " pod="openshift-multus/multus-additional-cni-plugins-gqppj" Nov 03 00:40:12.429952 ip-10-0-140-107 kubenswrapper[2578]: I1103 00:40:12.427053 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/00dc2920-34f2-4e0a-892e-7759a16162d5-host-run-k8s-cni-cncf-io\") pod \"multus-tb9hd\" (UID: \"00dc2920-34f2-4e0a-892e-7759a16162d5\") " pod="openshift-multus/multus-tb9hd" Nov 03 00:40:12.429952 ip-10-0-140-107 kubenswrapper[2578]: I1103 00:40:12.427079 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/00dc2920-34f2-4e0a-892e-7759a16162d5-multus-conf-dir\") pod \"multus-tb9hd\" (UID: \"00dc2920-34f2-4e0a-892e-7759a16162d5\") " pod="openshift-multus/multus-tb9hd" Nov 03 00:40:12.429952 ip-10-0-140-107 kubenswrapper[2578]: I1103 00:40:12.427103 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/98ca3972-fd0c-47f5-9d26-569a58e18bd1-log-socket\") pod \"ovnkube-node-psqpn\" (UID: \"98ca3972-fd0c-47f5-9d26-569a58e18bd1\") " pod="openshift-ovn-kubernetes/ovnkube-node-psqpn" Nov 03 00:40:12.429952 ip-10-0-140-107 kubenswrapper[2578]: I1103 00:40:12.427127 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/98ca3972-fd0c-47f5-9d26-569a58e18bd1-ovn-node-metrics-cert\") pod \"ovnkube-node-psqpn\" (UID: \"98ca3972-fd0c-47f5-9d26-569a58e18bd1\") " pod="openshift-ovn-kubernetes/ovnkube-node-psqpn" Nov 03 00:40:12.429952 ip-10-0-140-107 kubenswrapper[2578]: I1103 00:40:12.427157 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/6650035b-2220-491c-9683-8a03ccb7af6a-tuning-conf-dir\") pod \"multus-additional-cni-plugins-gqppj\" (UID: \"6650035b-2220-491c-9683-8a03ccb7af6a\") " pod="openshift-multus/multus-additional-cni-plugins-gqppj" Nov 03 00:40:12.429952 ip-10-0-140-107 kubenswrapper[2578]: I1103 00:40:12.427182 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/00dc2920-34f2-4e0a-892e-7759a16162d5-host-run-netns\") pod \"multus-tb9hd\" (UID: \"00dc2920-34f2-4e0a-892e-7759a16162d5\") " pod="openshift-multus/multus-tb9hd" Nov 03 00:40:12.429952 ip-10-0-140-107 kubenswrapper[2578]: I1103 00:40:12.427207 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/00dc2920-34f2-4e0a-892e-7759a16162d5-etc-kubernetes\") pod \"multus-tb9hd\" (UID: \"00dc2920-34f2-4e0a-892e-7759a16162d5\") " pod="openshift-multus/multus-tb9hd" Nov 03 00:40:12.429952 ip-10-0-140-107 kubenswrapper[2578]: I1103 00:40:12.427232 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/98ca3972-fd0c-47f5-9d26-569a58e18bd1-systemd-units\") pod \"ovnkube-node-psqpn\" (UID: \"98ca3972-fd0c-47f5-9d26-569a58e18bd1\") " pod="openshift-ovn-kubernetes/ovnkube-node-psqpn" Nov 03 00:40:12.430803 ip-10-0-140-107 kubenswrapper[2578]: I1103 00:40:12.427257 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/98ca3972-fd0c-47f5-9d26-569a58e18bd1-host-run-ovn-kubernetes\") pod \"ovnkube-node-psqpn\" (UID: \"98ca3972-fd0c-47f5-9d26-569a58e18bd1\") " pod="openshift-ovn-kubernetes/ovnkube-node-psqpn" Nov 03 00:40:12.431516 ip-10-0-140-107 kubenswrapper[2578]: I1103 00:40:12.431161 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-82x8c" Nov 03 00:40:12.434435 ip-10-0-140-107 kubenswrapper[2578]: I1103 00:40:12.434402 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-cjj2h" Nov 03 00:40:12.437161 ip-10-0-140-107 kubenswrapper[2578]: I1103 00:40:12.436469 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-bbk2h" Nov 03 00:40:12.440460 ip-10-0-140-107 kubenswrapper[2578]: I1103 00:40:12.440439 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-gtwxp" Nov 03 00:40:12.453453 ip-10-0-140-107 kubenswrapper[2578]: I1103 00:40:12.453415 2578 reflector.go:446] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Nov 03 00:40:12.453548 ip-10-0-140-107 kubenswrapper[2578]: I1103 00:40:12.453491 2578 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-7z7l7" Nov 03 00:40:12.462722 ip-10-0-140-107 kubenswrapper[2578]: I1103 00:40:12.462703 2578 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-7z7l7" Nov 03 00:40:12.515144 ip-10-0-140-107 kubenswrapper[2578]: I1103 00:40:12.515121 2578 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world" Nov 03 00:40:12.518409 ip-10-0-140-107 kubenswrapper[2578]: I1103 00:40:12.518234 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\"" Nov 03 00:40:12.518409 ip-10-0-140-107 kubenswrapper[2578]: I1103 00:40:12.518270 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\"" Nov 03 00:40:12.518409 ip-10-0-140-107 kubenswrapper[2578]: I1103 00:40:12.518379 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\"" Nov 03 00:40:12.518409 ip-10-0-140-107 kubenswrapper[2578]: I1103 00:40:12.518404 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\"" Nov 03 00:40:12.518748 ip-10-0-140-107 kubenswrapper[2578]: I1103 00:40:12.518691 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\"" Nov 03 00:40:12.519036 ip-10-0-140-107 kubenswrapper[2578]: I1103 00:40:12.519016 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\"" Nov 03 00:40:12.519203 ip-10-0-140-107 kubenswrapper[2578]: I1103 00:40:12.519186 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\"" Nov 03 00:40:12.519292 ip-10-0-140-107 kubenswrapper[2578]: I1103 00:40:12.519222 2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\"" Nov 03 00:40:12.519347 ip-10-0-140-107 kubenswrapper[2578]: I1103 00:40:12.519329 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\"" Nov 03 00:40:12.519397 ip-10-0-140-107 kubenswrapper[2578]: I1103 00:40:12.519356 2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\"" Nov 03 00:40:12.519450 ip-10-0-140-107 kubenswrapper[2578]: I1103 00:40:12.519426 2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-node-identity-dockercfg-229lw\"" Nov 03 00:40:12.519510 ip-10-0-140-107 kubenswrapper[2578]: I1103 00:40:12.519495 2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent-dockercfg-tq982\"" Nov 03 00:40:12.519618 ip-10-0-140-107 kubenswrapper[2578]: I1103 00:40:12.519604 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\"" Nov 03 00:40:12.520256 ip-10-0-140-107 kubenswrapper[2578]: I1103 00:40:12.520230 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\"" Nov 03 00:40:12.520340 ip-10-0-140-107 kubenswrapper[2578]: I1103 00:40:12.520308 2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-kp9k4\"" Nov 03 00:40:12.520483 ip-10-0-140-107 kubenswrapper[2578]: I1103 00:40:12.520468 2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-d54vs\"" Nov 03 00:40:12.520483 ip-10-0-140-107 kubenswrapper[2578]: I1103 00:40:12.520480 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\"" Nov 03 00:40:12.520572 ip-10-0-140-107 kubenswrapper[2578]: I1103 00:40:12.520467 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\"" Nov 03 00:40:12.520990 ip-10-0-140-107 kubenswrapper[2578]: I1103 00:40:12.520855 2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-9v27r\"" Nov 03 00:40:12.520990 ip-10-0-140-107 kubenswrapper[2578]: I1103 00:40:12.520887 2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\"" Nov 03 00:40:12.520990 ip-10-0-140-107 kubenswrapper[2578]: I1103 00:40:12.520896 2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-4zb6t\"" Nov 03 00:40:12.520990 ip-10-0-140-107 kubenswrapper[2578]: I1103 00:40:12.520924 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\"" Nov 03 00:40:12.520990 ip-10-0-140-107 kubenswrapper[2578]: I1103 00:40:12.520964 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\"" Nov 03 00:40:12.520990 ip-10-0-140-107 kubenswrapper[2578]: I1103 00:40:12.520859 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\"" Nov 03 00:40:12.522120 ip-10-0-140-107 kubenswrapper[2578]: I1103 00:40:12.522078 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\"" Nov 03 00:40:12.522189 ip-10-0-140-107 kubenswrapper[2578]: I1103 00:40:12.522143 2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-97hfk\"" Nov 03 00:40:12.522265 ip-10-0-140-107 kubenswrapper[2578]: I1103 00:40:12.522239 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\"" Nov 03 00:40:12.522356 ip-10-0-140-107 kubenswrapper[2578]: I1103 00:40:12.522309 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\"" Nov 03 00:40:12.522422 ip-10-0-140-107 kubenswrapper[2578]: I1103 00:40:12.522407 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\"" Nov 03 00:40:12.522496 ip-10-0-140-107 kubenswrapper[2578]: I1103 00:40:12.522470 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\"" Nov 03 00:40:12.522716 ip-10-0-140-107 kubenswrapper[2578]: I1103 00:40:12.522689 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\"" Nov 03 00:40:12.522787 ip-10-0-140-107 kubenswrapper[2578]: I1103 00:40:12.522774 2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-7mmgk\"" Nov 03 00:40:12.527986 ip-10-0-140-107 kubenswrapper[2578]: I1103 00:40:12.527949 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/98ca3972-fd0c-47f5-9d26-569a58e18bd1-log-socket\") pod \"ovnkube-node-psqpn\" (UID: \"98ca3972-fd0c-47f5-9d26-569a58e18bd1\") " pod="openshift-ovn-kubernetes/ovnkube-node-psqpn" Nov 03 00:40:12.528093 ip-10-0-140-107 kubenswrapper[2578]: I1103 00:40:12.527994 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/98ca3972-fd0c-47f5-9d26-569a58e18bd1-ovn-node-metrics-cert\") pod \"ovnkube-node-psqpn\" (UID: \"98ca3972-fd0c-47f5-9d26-569a58e18bd1\") " pod="openshift-ovn-kubernetes/ovnkube-node-psqpn" Nov 03 00:40:12.528093 ip-10-0-140-107 kubenswrapper[2578]: I1103 00:40:12.528017 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/cabd402c-79c5-4699-9c85-d415d362b4d0-kubelet-config\") pod \"global-pull-secret-syncer-jv7cw\" (UID: \"cabd402c-79c5-4699-9c85-d415d362b4d0\") " pod="kube-system/global-pull-secret-syncer-jv7cw" Nov 03 00:40:12.528093 ip-10-0-140-107 kubenswrapper[2578]: I1103 00:40:12.528035 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/b748c4f8-d021-4d37-b390-d395482fcddd-etc-sysconfig\") pod \"tuned-gtwxp\" (UID: \"b748c4f8-d021-4d37-b390-d395482fcddd\") " pod="openshift-cluster-node-tuning-operator/tuned-gtwxp" Nov 03 00:40:12.528093 ip-10-0-140-107 kubenswrapper[2578]: I1103 00:40:12.528051 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/b748c4f8-d021-4d37-b390-d395482fcddd-etc-sysctl-conf\") pod \"tuned-gtwxp\" (UID: \"b748c4f8-d021-4d37-b390-d395482fcddd\") " pod="openshift-cluster-node-tuning-operator/tuned-gtwxp" Nov 03 00:40:12.528093 ip-10-0-140-107 kubenswrapper[2578]: I1103 00:40:12.528068 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/00dc2920-34f2-4e0a-892e-7759a16162d5-host-run-netns\") pod \"multus-tb9hd\" (UID: \"00dc2920-34f2-4e0a-892e-7759a16162d5\") " pod="openshift-multus/multus-tb9hd" Nov 03 00:40:12.528093 ip-10-0-140-107 kubenswrapper[2578]: I1103 00:40:12.528083 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/00dc2920-34f2-4e0a-892e-7759a16162d5-etc-kubernetes\") pod \"multus-tb9hd\" (UID: \"00dc2920-34f2-4e0a-892e-7759a16162d5\") " pod="openshift-multus/multus-tb9hd" Nov 03 00:40:12.528093 ip-10-0-140-107 kubenswrapper[2578]: I1103 00:40:12.528097 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/98ca3972-fd0c-47f5-9d26-569a58e18bd1-systemd-units\") pod \"ovnkube-node-psqpn\" (UID: \"98ca3972-fd0c-47f5-9d26-569a58e18bd1\") " pod="openshift-ovn-kubernetes/ovnkube-node-psqpn" Nov 03 00:40:12.528456 ip-10-0-140-107 kubenswrapper[2578]: I1103 00:40:12.528118 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/b748c4f8-d021-4d37-b390-d395482fcddd-var-lib-kubelet\") pod \"tuned-gtwxp\" (UID: \"b748c4f8-d021-4d37-b390-d395482fcddd\") " pod="openshift-cluster-node-tuning-operator/tuned-gtwxp" Nov 03 00:40:12.528456 ip-10-0-140-107 kubenswrapper[2578]: I1103 00:40:12.528143 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/98ca3972-fd0c-47f5-9d26-569a58e18bd1-host-slash\") pod \"ovnkube-node-psqpn\" (UID: \"98ca3972-fd0c-47f5-9d26-569a58e18bd1\") " pod="openshift-ovn-kubernetes/ovnkube-node-psqpn" Nov 03 00:40:12.528456 ip-10-0-140-107 kubenswrapper[2578]: I1103 00:40:12.528169 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/98ca3972-fd0c-47f5-9d26-569a58e18bd1-ovnkube-script-lib\") pod \"ovnkube-node-psqpn\" (UID: \"98ca3972-fd0c-47f5-9d26-569a58e18bd1\") " pod="openshift-ovn-kubernetes/ovnkube-node-psqpn" Nov 03 00:40:12.528456 ip-10-0-140-107 kubenswrapper[2578]: I1103 00:40:12.528193 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sfdp9\" (UniqueName: \"kubernetes.io/projected/ad5bf442-729d-411c-96e4-2e5105ebadcf-kube-api-access-sfdp9\") pod \"node-resolver-cjj2h\" (UID: \"ad5bf442-729d-411c-96e4-2e5105ebadcf\") " pod="openshift-dns/node-resolver-cjj2h" Nov 03 00:40:12.528456 ip-10-0-140-107 kubenswrapper[2578]: I1103 00:40:12.528210 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/b748c4f8-d021-4d37-b390-d395482fcddd-lib-modules\") pod \"tuned-gtwxp\" (UID: \"b748c4f8-d021-4d37-b390-d395482fcddd\") " pod="openshift-cluster-node-tuning-operator/tuned-gtwxp" Nov 03 00:40:12.528456 ip-10-0-140-107 kubenswrapper[2578]: I1103 00:40:12.528223 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/b748c4f8-d021-4d37-b390-d395482fcddd-host\") pod \"tuned-gtwxp\" (UID: \"b748c4f8-d021-4d37-b390-d395482fcddd\") " pod="openshift-cluster-node-tuning-operator/tuned-gtwxp" Nov 03 00:40:12.528456 ip-10-0-140-107 kubenswrapper[2578]: I1103 00:40:12.528245 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/98ca3972-fd0c-47f5-9d26-569a58e18bd1-host-cni-netd\") pod \"ovnkube-node-psqpn\" (UID: \"98ca3972-fd0c-47f5-9d26-569a58e18bd1\") " pod="openshift-ovn-kubernetes/ovnkube-node-psqpn" Nov 03 00:40:12.528456 ip-10-0-140-107 kubenswrapper[2578]: I1103 00:40:12.528266 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/cabd402c-79c5-4699-9c85-d415d362b4d0-original-pull-secret\") pod \"global-pull-secret-syncer-jv7cw\" (UID: \"cabd402c-79c5-4699-9c85-d415d362b4d0\") " pod="kube-system/global-pull-secret-syncer-jv7cw" Nov 03 00:40:12.528456 ip-10-0-140-107 kubenswrapper[2578]: I1103 00:40:12.528283 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/c736fd3f-973f-4999-8d4e-c405079b2888-host-slash\") pod \"iptables-alerter-5spnk\" (UID: \"c736fd3f-973f-4999-8d4e-c405079b2888\") " pod="openshift-network-operator/iptables-alerter-5spnk" Nov 03 00:40:12.528456 ip-10-0-140-107 kubenswrapper[2578]: I1103 00:40:12.528299 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/136086c0-4846-437a-ad98-3cfef50610a0-metrics-certs\") pod \"network-metrics-daemon-rt2hx\" (UID: \"136086c0-4846-437a-ad98-3cfef50610a0\") " pod="openshift-multus/network-metrics-daemon-rt2hx" Nov 03 00:40:12.528456 ip-10-0-140-107 kubenswrapper[2578]: I1103 00:40:12.528320 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/00dc2920-34f2-4e0a-892e-7759a16162d5-multus-socket-dir-parent\") pod \"multus-tb9hd\" (UID: \"00dc2920-34f2-4e0a-892e-7759a16162d5\") " pod="openshift-multus/multus-tb9hd" Nov 03 00:40:12.528456 ip-10-0-140-107 kubenswrapper[2578]: I1103 00:40:12.528356 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/00dc2920-34f2-4e0a-892e-7759a16162d5-hostroot\") pod \"multus-tb9hd\" (UID: \"00dc2920-34f2-4e0a-892e-7759a16162d5\") " pod="openshift-multus/multus-tb9hd" Nov 03 00:40:12.528456 ip-10-0-140-107 kubenswrapper[2578]: I1103 00:40:12.528379 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/cabd402c-79c5-4699-9c85-d415d362b4d0-dbus\") pod \"global-pull-secret-syncer-jv7cw\" (UID: \"cabd402c-79c5-4699-9c85-d415d362b4d0\") " pod="kube-system/global-pull-secret-syncer-jv7cw" Nov 03 00:40:12.528456 ip-10-0-140-107 kubenswrapper[2578]: I1103 00:40:12.528404 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/b748c4f8-d021-4d37-b390-d395482fcddd-etc-sysctl-d\") pod \"tuned-gtwxp\" (UID: \"b748c4f8-d021-4d37-b390-d395482fcddd\") " pod="openshift-cluster-node-tuning-operator/tuned-gtwxp" Nov 03 00:40:12.528456 ip-10-0-140-107 kubenswrapper[2578]: I1103 00:40:12.528436 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/00dc2920-34f2-4e0a-892e-7759a16162d5-host-run-multus-certs\") pod \"multus-tb9hd\" (UID: \"00dc2920-34f2-4e0a-892e-7759a16162d5\") " pod="openshift-multus/multus-tb9hd" Nov 03 00:40:12.529146 ip-10-0-140-107 kubenswrapper[2578]: I1103 00:40:12.528468 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/98ca3972-fd0c-47f5-9d26-569a58e18bd1-host-run-netns\") pod \"ovnkube-node-psqpn\" (UID: \"98ca3972-fd0c-47f5-9d26-569a58e18bd1\") " pod="openshift-ovn-kubernetes/ovnkube-node-psqpn" Nov 03 00:40:12.529146 ip-10-0-140-107 kubenswrapper[2578]: I1103 00:40:12.528494 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/98ca3972-fd0c-47f5-9d26-569a58e18bd1-host-cni-bin\") pod \"ovnkube-node-psqpn\" (UID: \"98ca3972-fd0c-47f5-9d26-569a58e18bd1\") " pod="openshift-ovn-kubernetes/ovnkube-node-psqpn" Nov 03 00:40:12.529146 ip-10-0-140-107 kubenswrapper[2578]: I1103 00:40:12.528519 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/460779b2-3903-4b23-9b4d-e25142782a21-serviceca\") pod \"node-ca-bbk2h\" (UID: \"460779b2-3903-4b23-9b4d-e25142782a21\") " pod="openshift-image-registry/node-ca-bbk2h" Nov 03 00:40:12.529146 ip-10-0-140-107 kubenswrapper[2578]: I1103 00:40:12.528546 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2tdmf\" (UniqueName: \"kubernetes.io/projected/c736fd3f-973f-4999-8d4e-c405079b2888-kube-api-access-2tdmf\") pod \"iptables-alerter-5spnk\" (UID: \"c736fd3f-973f-4999-8d4e-c405079b2888\") " pod="openshift-network-operator/iptables-alerter-5spnk" Nov 03 00:40:12.529146 ip-10-0-140-107 kubenswrapper[2578]: I1103 00:40:12.528575 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d8l25\" (UniqueName: \"kubernetes.io/projected/8cd3ca22-e4ce-4896-ae31-97605af4e4c6-kube-api-access-d8l25\") pod \"network-check-target-w46kd\" (UID: \"8cd3ca22-e4ce-4896-ae31-97605af4e4c6\") " pod="openshift-network-diagnostics/network-check-target-w46kd" Nov 03 00:40:12.529146 ip-10-0-140-107 kubenswrapper[2578]: I1103 00:40:12.528601 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/98ca3972-fd0c-47f5-9d26-569a58e18bd1-run-ovn\") pod \"ovnkube-node-psqpn\" (UID: \"98ca3972-fd0c-47f5-9d26-569a58e18bd1\") " pod="openshift-ovn-kubernetes/ovnkube-node-psqpn" Nov 03 00:40:12.529146 ip-10-0-140-107 kubenswrapper[2578]: I1103 00:40:12.528625 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/37bf96f5-045f-49e8-83e8-410df66bfb1f-sys-fs\") pod \"aws-ebs-csi-driver-node-br8vt\" (UID: \"37bf96f5-045f-49e8-83e8-410df66bfb1f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-br8vt" Nov 03 00:40:12.529146 ip-10-0-140-107 kubenswrapper[2578]: I1103 00:40:12.528656 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/c3ffbe53-7f23-4750-b95d-476e6cfc58a6-konnectivity-ca\") pod \"konnectivity-agent-82x8c\" (UID: \"c3ffbe53-7f23-4750-b95d-476e6cfc58a6\") " pod="kube-system/konnectivity-agent-82x8c" Nov 03 00:40:12.529146 ip-10-0-140-107 kubenswrapper[2578]: I1103 00:40:12.528672 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/b748c4f8-d021-4d37-b390-d395482fcddd-sys\") pod \"tuned-gtwxp\" (UID: \"b748c4f8-d021-4d37-b390-d395482fcddd\") " pod="openshift-cluster-node-tuning-operator/tuned-gtwxp" Nov 03 00:40:12.529146 ip-10-0-140-107 kubenswrapper[2578]: I1103 00:40:12.528690 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/00dc2920-34f2-4e0a-892e-7759a16162d5-cnibin\") pod \"multus-tb9hd\" (UID: \"00dc2920-34f2-4e0a-892e-7759a16162d5\") " pod="openshift-multus/multus-tb9hd" Nov 03 00:40:12.529146 ip-10-0-140-107 kubenswrapper[2578]: I1103 00:40:12.528712 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/98ca3972-fd0c-47f5-9d26-569a58e18bd1-var-lib-openvswitch\") pod \"ovnkube-node-psqpn\" (UID: \"98ca3972-fd0c-47f5-9d26-569a58e18bd1\") " pod="openshift-ovn-kubernetes/ovnkube-node-psqpn" Nov 03 00:40:12.529146 ip-10-0-140-107 kubenswrapper[2578]: I1103 00:40:12.528744 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/37bf96f5-045f-49e8-83e8-410df66bfb1f-etc-selinux\") pod \"aws-ebs-csi-driver-node-br8vt\" (UID: \"37bf96f5-045f-49e8-83e8-410df66bfb1f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-br8vt" Nov 03 00:40:12.529146 ip-10-0-140-107 kubenswrapper[2578]: I1103 00:40:12.528769 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/b748c4f8-d021-4d37-b390-d395482fcddd-etc-tuned\") pod \"tuned-gtwxp\" (UID: \"b748c4f8-d021-4d37-b390-d395482fcddd\") " pod="openshift-cluster-node-tuning-operator/tuned-gtwxp" Nov 03 00:40:12.529146 ip-10-0-140-107 kubenswrapper[2578]: I1103 00:40:12.528796 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/c736fd3f-973f-4999-8d4e-c405079b2888-iptables-alerter-script\") pod \"iptables-alerter-5spnk\" (UID: \"c736fd3f-973f-4999-8d4e-c405079b2888\") " pod="openshift-network-operator/iptables-alerter-5spnk" Nov 03 00:40:12.529146 ip-10-0-140-107 kubenswrapper[2578]: I1103 00:40:12.528823 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/6650035b-2220-491c-9683-8a03ccb7af6a-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-gqppj\" (UID: \"6650035b-2220-491c-9683-8a03ccb7af6a\") " pod="openshift-multus/multus-additional-cni-plugins-gqppj" Nov 03 00:40:12.529146 ip-10-0-140-107 kubenswrapper[2578]: I1103 00:40:12.528847 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/00dc2920-34f2-4e0a-892e-7759a16162d5-host-run-k8s-cni-cncf-io\") pod \"multus-tb9hd\" (UID: \"00dc2920-34f2-4e0a-892e-7759a16162d5\") " pod="openshift-multus/multus-tb9hd" Nov 03 00:40:12.529146 ip-10-0-140-107 kubenswrapper[2578]: I1103 00:40:12.528886 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/00dc2920-34f2-4e0a-892e-7759a16162d5-multus-conf-dir\") pod \"multus-tb9hd\" (UID: \"00dc2920-34f2-4e0a-892e-7759a16162d5\") " pod="openshift-multus/multus-tb9hd" Nov 03 00:40:12.529714 ip-10-0-140-107 kubenswrapper[2578]: I1103 00:40:12.528933 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/c3ffbe53-7f23-4750-b95d-476e6cfc58a6-agent-certs\") pod \"konnectivity-agent-82x8c\" (UID: \"c3ffbe53-7f23-4750-b95d-476e6cfc58a6\") " pod="kube-system/konnectivity-agent-82x8c" Nov 03 00:40:12.529714 ip-10-0-140-107 kubenswrapper[2578]: I1103 00:40:12.528963 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hsvqw\" (UniqueName: \"kubernetes.io/projected/b748c4f8-d021-4d37-b390-d395482fcddd-kube-api-access-hsvqw\") pod \"tuned-gtwxp\" (UID: \"b748c4f8-d021-4d37-b390-d395482fcddd\") " pod="openshift-cluster-node-tuning-operator/tuned-gtwxp" Nov 03 00:40:12.529714 ip-10-0-140-107 kubenswrapper[2578]: I1103 00:40:12.529008 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/6650035b-2220-491c-9683-8a03ccb7af6a-tuning-conf-dir\") pod \"multus-additional-cni-plugins-gqppj\" (UID: \"6650035b-2220-491c-9683-8a03ccb7af6a\") " pod="openshift-multus/multus-additional-cni-plugins-gqppj" Nov 03 00:40:12.529714 ip-10-0-140-107 kubenswrapper[2578]: I1103 00:40:12.529049 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/98ca3972-fd0c-47f5-9d26-569a58e18bd1-host-run-ovn-kubernetes\") pod \"ovnkube-node-psqpn\" (UID: \"98ca3972-fd0c-47f5-9d26-569a58e18bd1\") " pod="openshift-ovn-kubernetes/ovnkube-node-psqpn" Nov 03 00:40:12.529714 ip-10-0-140-107 kubenswrapper[2578]: I1103 00:40:12.529086 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/98ca3972-fd0c-47f5-9d26-569a58e18bd1-env-overrides\") pod \"ovnkube-node-psqpn\" (UID: \"98ca3972-fd0c-47f5-9d26-569a58e18bd1\") " pod="openshift-ovn-kubernetes/ovnkube-node-psqpn" Nov 03 00:40:12.529714 ip-10-0-140-107 kubenswrapper[2578]: I1103 00:40:12.529111 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/37bf96f5-045f-49e8-83e8-410df66bfb1f-kubelet-dir\") pod \"aws-ebs-csi-driver-node-br8vt\" (UID: \"37bf96f5-045f-49e8-83e8-410df66bfb1f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-br8vt" Nov 03 00:40:12.529714 ip-10-0-140-107 kubenswrapper[2578]: I1103 00:40:12.529140 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/37bf96f5-045f-49e8-83e8-410df66bfb1f-registration-dir\") pod \"aws-ebs-csi-driver-node-br8vt\" (UID: \"37bf96f5-045f-49e8-83e8-410df66bfb1f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-br8vt" Nov 03 00:40:12.529714 ip-10-0-140-107 kubenswrapper[2578]: I1103 00:40:12.529170 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/ad5bf442-729d-411c-96e4-2e5105ebadcf-hosts-file\") pod \"node-resolver-cjj2h\" (UID: \"ad5bf442-729d-411c-96e4-2e5105ebadcf\") " pod="openshift-dns/node-resolver-cjj2h" Nov 03 00:40:12.529714 ip-10-0-140-107 kubenswrapper[2578]: I1103 00:40:12.529197 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/00dc2920-34f2-4e0a-892e-7759a16162d5-host-var-lib-cni-bin\") pod \"multus-tb9hd\" (UID: \"00dc2920-34f2-4e0a-892e-7759a16162d5\") " pod="openshift-multus/multus-tb9hd" Nov 03 00:40:12.529714 ip-10-0-140-107 kubenswrapper[2578]: I1103 00:40:12.529228 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/98ca3972-fd0c-47f5-9d26-569a58e18bd1-run-openvswitch\") pod \"ovnkube-node-psqpn\" (UID: \"98ca3972-fd0c-47f5-9d26-569a58e18bd1\") " pod="openshift-ovn-kubernetes/ovnkube-node-psqpn" Nov 03 00:40:12.529714 ip-10-0-140-107 kubenswrapper[2578]: I1103 00:40:12.529256 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/37bf96f5-045f-49e8-83e8-410df66bfb1f-socket-dir\") pod \"aws-ebs-csi-driver-node-br8vt\" (UID: \"37bf96f5-045f-49e8-83e8-410df66bfb1f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-br8vt" Nov 03 00:40:12.529714 ip-10-0-140-107 kubenswrapper[2578]: I1103 00:40:12.529282 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lp26w\" (UniqueName: \"kubernetes.io/projected/37bf96f5-045f-49e8-83e8-410df66bfb1f-kube-api-access-lp26w\") pod \"aws-ebs-csi-driver-node-br8vt\" (UID: \"37bf96f5-045f-49e8-83e8-410df66bfb1f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-br8vt" Nov 03 00:40:12.529714 ip-10-0-140-107 kubenswrapper[2578]: I1103 00:40:12.529311 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/460779b2-3903-4b23-9b4d-e25142782a21-host\") pod \"node-ca-bbk2h\" (UID: \"460779b2-3903-4b23-9b4d-e25142782a21\") " pod="openshift-image-registry/node-ca-bbk2h" Nov 03 00:40:12.529714 ip-10-0-140-107 kubenswrapper[2578]: I1103 00:40:12.529336 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q4b9t\" (UniqueName: \"kubernetes.io/projected/460779b2-3903-4b23-9b4d-e25142782a21-kube-api-access-q4b9t\") pod \"node-ca-bbk2h\" (UID: \"460779b2-3903-4b23-9b4d-e25142782a21\") " pod="openshift-image-registry/node-ca-bbk2h" Nov 03 00:40:12.529714 ip-10-0-140-107 kubenswrapper[2578]: I1103 00:40:12.529365 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ad5bf442-729d-411c-96e4-2e5105ebadcf-tmp-dir\") pod \"node-resolver-cjj2h\" (UID: \"ad5bf442-729d-411c-96e4-2e5105ebadcf\") " pod="openshift-dns/node-resolver-cjj2h" Nov 03 00:40:12.529714 ip-10-0-140-107 kubenswrapper[2578]: I1103 00:40:12.529395 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-llh2g\" (UniqueName: \"kubernetes.io/projected/136086c0-4846-437a-ad98-3cfef50610a0-kube-api-access-llh2g\") pod \"network-metrics-daemon-rt2hx\" (UID: \"136086c0-4846-437a-ad98-3cfef50610a0\") " pod="openshift-multus/network-metrics-daemon-rt2hx" Nov 03 00:40:12.530369 ip-10-0-140-107 kubenswrapper[2578]: I1103 00:40:12.529440 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/6650035b-2220-491c-9683-8a03ccb7af6a-cni-binary-copy\") pod \"multus-additional-cni-plugins-gqppj\" (UID: \"6650035b-2220-491c-9683-8a03ccb7af6a\") " pod="openshift-multus/multus-additional-cni-plugins-gqppj" Nov 03 00:40:12.530369 ip-10-0-140-107 kubenswrapper[2578]: I1103 00:40:12.529466 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/00dc2920-34f2-4e0a-892e-7759a16162d5-multus-daemon-config\") pod \"multus-tb9hd\" (UID: \"00dc2920-34f2-4e0a-892e-7759a16162d5\") " pod="openshift-multus/multus-tb9hd" Nov 03 00:40:12.530369 ip-10-0-140-107 kubenswrapper[2578]: I1103 00:40:12.529492 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/98ca3972-fd0c-47f5-9d26-569a58e18bd1-host-kubelet\") pod \"ovnkube-node-psqpn\" (UID: \"98ca3972-fd0c-47f5-9d26-569a58e18bd1\") " pod="openshift-ovn-kubernetes/ovnkube-node-psqpn" Nov 03 00:40:12.530369 ip-10-0-140-107 kubenswrapper[2578]: I1103 00:40:12.529522 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4bhqh\" (UniqueName: \"kubernetes.io/projected/98ca3972-fd0c-47f5-9d26-569a58e18bd1-kube-api-access-4bhqh\") pod \"ovnkube-node-psqpn\" (UID: \"98ca3972-fd0c-47f5-9d26-569a58e18bd1\") " pod="openshift-ovn-kubernetes/ovnkube-node-psqpn" Nov 03 00:40:12.530369 ip-10-0-140-107 kubenswrapper[2578]: I1103 00:40:12.529549 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/00dc2920-34f2-4e0a-892e-7759a16162d5-host-var-lib-cni-multus\") pod \"multus-tb9hd\" (UID: \"00dc2920-34f2-4e0a-892e-7759a16162d5\") " pod="openshift-multus/multus-tb9hd" Nov 03 00:40:12.530369 ip-10-0-140-107 kubenswrapper[2578]: I1103 00:40:12.529572 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/b748c4f8-d021-4d37-b390-d395482fcddd-etc-systemd\") pod \"tuned-gtwxp\" (UID: \"b748c4f8-d021-4d37-b390-d395482fcddd\") " pod="openshift-cluster-node-tuning-operator/tuned-gtwxp" Nov 03 00:40:12.530369 ip-10-0-140-107 kubenswrapper[2578]: I1103 00:40:12.529597 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/6650035b-2220-491c-9683-8a03ccb7af6a-cnibin\") pod \"multus-additional-cni-plugins-gqppj\" (UID: \"6650035b-2220-491c-9683-8a03ccb7af6a\") " pod="openshift-multus/multus-additional-cni-plugins-gqppj" Nov 03 00:40:12.530369 ip-10-0-140-107 kubenswrapper[2578]: I1103 00:40:12.529626 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/00dc2920-34f2-4e0a-892e-7759a16162d5-cni-binary-copy\") pod \"multus-tb9hd\" (UID: \"00dc2920-34f2-4e0a-892e-7759a16162d5\") " pod="openshift-multus/multus-tb9hd" Nov 03 00:40:12.530369 ip-10-0-140-107 kubenswrapper[2578]: I1103 00:40:12.529656 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j62gg\" (UniqueName: \"kubernetes.io/projected/00dc2920-34f2-4e0a-892e-7759a16162d5-kube-api-access-j62gg\") pod \"multus-tb9hd\" (UID: \"00dc2920-34f2-4e0a-892e-7759a16162d5\") " pod="openshift-multus/multus-tb9hd" Nov 03 00:40:12.530369 ip-10-0-140-107 kubenswrapper[2578]: I1103 00:40:12.529682 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/98ca3972-fd0c-47f5-9d26-569a58e18bd1-run-systemd\") pod \"ovnkube-node-psqpn\" (UID: \"98ca3972-fd0c-47f5-9d26-569a58e18bd1\") " pod="openshift-ovn-kubernetes/ovnkube-node-psqpn" Nov 03 00:40:12.530369 ip-10-0-140-107 kubenswrapper[2578]: I1103 00:40:12.529711 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/b748c4f8-d021-4d37-b390-d395482fcddd-etc-modprobe-d\") pod \"tuned-gtwxp\" (UID: \"b748c4f8-d021-4d37-b390-d395482fcddd\") " pod="openshift-cluster-node-tuning-operator/tuned-gtwxp" Nov 03 00:40:12.530369 ip-10-0-140-107 kubenswrapper[2578]: I1103 00:40:12.529737 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/b748c4f8-d021-4d37-b390-d395482fcddd-tmp\") pod \"tuned-gtwxp\" (UID: \"b748c4f8-d021-4d37-b390-d395482fcddd\") " pod="openshift-cluster-node-tuning-operator/tuned-gtwxp" Nov 03 00:40:12.530369 ip-10-0-140-107 kubenswrapper[2578]: I1103 00:40:12.529772 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rkf7s\" (UniqueName: \"kubernetes.io/projected/6650035b-2220-491c-9683-8a03ccb7af6a-kube-api-access-rkf7s\") pod \"multus-additional-cni-plugins-gqppj\" (UID: \"6650035b-2220-491c-9683-8a03ccb7af6a\") " pod="openshift-multus/multus-additional-cni-plugins-gqppj" Nov 03 00:40:12.530369 ip-10-0-140-107 kubenswrapper[2578]: I1103 00:40:12.529808 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/00dc2920-34f2-4e0a-892e-7759a16162d5-multus-cni-dir\") pod \"multus-tb9hd\" (UID: \"00dc2920-34f2-4e0a-892e-7759a16162d5\") " pod="openshift-multus/multus-tb9hd" Nov 03 00:40:12.530369 ip-10-0-140-107 kubenswrapper[2578]: I1103 00:40:12.529837 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/98ca3972-fd0c-47f5-9d26-569a58e18bd1-etc-openvswitch\") pod \"ovnkube-node-psqpn\" (UID: \"98ca3972-fd0c-47f5-9d26-569a58e18bd1\") " pod="openshift-ovn-kubernetes/ovnkube-node-psqpn" Nov 03 00:40:12.530369 ip-10-0-140-107 kubenswrapper[2578]: I1103 00:40:12.529865 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/98ca3972-fd0c-47f5-9d26-569a58e18bd1-node-log\") pod \"ovnkube-node-psqpn\" (UID: \"98ca3972-fd0c-47f5-9d26-569a58e18bd1\") " pod="openshift-ovn-kubernetes/ovnkube-node-psqpn" Nov 03 00:40:12.530369 ip-10-0-140-107 kubenswrapper[2578]: I1103 00:40:12.529893 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/98ca3972-fd0c-47f5-9d26-569a58e18bd1-ovnkube-config\") pod \"ovnkube-node-psqpn\" (UID: \"98ca3972-fd0c-47f5-9d26-569a58e18bd1\") " pod="openshift-ovn-kubernetes/ovnkube-node-psqpn" Nov 03 00:40:12.531205 ip-10-0-140-107 kubenswrapper[2578]: I1103 00:40:12.529918 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/b748c4f8-d021-4d37-b390-d395482fcddd-run\") pod \"tuned-gtwxp\" (UID: \"b748c4f8-d021-4d37-b390-d395482fcddd\") " pod="openshift-cluster-node-tuning-operator/tuned-gtwxp" Nov 03 00:40:12.531205 ip-10-0-140-107 kubenswrapper[2578]: I1103 00:40:12.529944 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/6650035b-2220-491c-9683-8a03ccb7af6a-system-cni-dir\") pod \"multus-additional-cni-plugins-gqppj\" (UID: \"6650035b-2220-491c-9683-8a03ccb7af6a\") " pod="openshift-multus/multus-additional-cni-plugins-gqppj" Nov 03 00:40:12.531205 ip-10-0-140-107 kubenswrapper[2578]: I1103 00:40:12.529999 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/6650035b-2220-491c-9683-8a03ccb7af6a-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-gqppj\" (UID: \"6650035b-2220-491c-9683-8a03ccb7af6a\") " pod="openshift-multus/multus-additional-cni-plugins-gqppj" Nov 03 00:40:12.531205 ip-10-0-140-107 kubenswrapper[2578]: I1103 00:40:12.530043 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/00dc2920-34f2-4e0a-892e-7759a16162d5-system-cni-dir\") pod \"multus-tb9hd\" (UID: \"00dc2920-34f2-4e0a-892e-7759a16162d5\") " pod="openshift-multus/multus-tb9hd" Nov 03 00:40:12.531205 ip-10-0-140-107 kubenswrapper[2578]: I1103 00:40:12.530072 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/00dc2920-34f2-4e0a-892e-7759a16162d5-host-var-lib-kubelet\") pod \"multus-tb9hd\" (UID: \"00dc2920-34f2-4e0a-892e-7759a16162d5\") " pod="openshift-multus/multus-tb9hd" Nov 03 00:40:12.531205 ip-10-0-140-107 kubenswrapper[2578]: I1103 00:40:12.530105 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/98ca3972-fd0c-47f5-9d26-569a58e18bd1-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-psqpn\" (UID: \"98ca3972-fd0c-47f5-9d26-569a58e18bd1\") " pod="openshift-ovn-kubernetes/ovnkube-node-psqpn" Nov 03 00:40:12.531205 ip-10-0-140-107 kubenswrapper[2578]: I1103 00:40:12.530134 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/37bf96f5-045f-49e8-83e8-410df66bfb1f-device-dir\") pod \"aws-ebs-csi-driver-node-br8vt\" (UID: \"37bf96f5-045f-49e8-83e8-410df66bfb1f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-br8vt" Nov 03 00:40:12.531205 ip-10-0-140-107 kubenswrapper[2578]: I1103 00:40:12.530167 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/b748c4f8-d021-4d37-b390-d395482fcddd-etc-kubernetes\") pod \"tuned-gtwxp\" (UID: \"b748c4f8-d021-4d37-b390-d395482fcddd\") " pod="openshift-cluster-node-tuning-operator/tuned-gtwxp" Nov 03 00:40:12.531205 ip-10-0-140-107 kubenswrapper[2578]: I1103 00:40:12.530350 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/98ca3972-fd0c-47f5-9d26-569a58e18bd1-log-socket\") pod \"ovnkube-node-psqpn\" (UID: \"98ca3972-fd0c-47f5-9d26-569a58e18bd1\") " pod="openshift-ovn-kubernetes/ovnkube-node-psqpn" Nov 03 00:40:12.531205 ip-10-0-140-107 kubenswrapper[2578]: I1103 00:40:12.530703 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/00dc2920-34f2-4e0a-892e-7759a16162d5-host-run-netns\") pod \"multus-tb9hd\" (UID: \"00dc2920-34f2-4e0a-892e-7759a16162d5\") " pod="openshift-multus/multus-tb9hd" Nov 03 00:40:12.531205 ip-10-0-140-107 kubenswrapper[2578]: I1103 00:40:12.530735 2578 swap_util.go:78] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory" Nov 03 00:40:12.531205 ip-10-0-140-107 kubenswrapper[2578]: I1103 00:40:12.530770 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/00dc2920-34f2-4e0a-892e-7759a16162d5-etc-kubernetes\") pod \"multus-tb9hd\" (UID: \"00dc2920-34f2-4e0a-892e-7759a16162d5\") " pod="openshift-multus/multus-tb9hd" Nov 03 00:40:12.531205 ip-10-0-140-107 kubenswrapper[2578]: I1103 00:40:12.530819 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/98ca3972-fd0c-47f5-9d26-569a58e18bd1-systemd-units\") pod \"ovnkube-node-psqpn\" (UID: \"98ca3972-fd0c-47f5-9d26-569a58e18bd1\") " pod="openshift-ovn-kubernetes/ovnkube-node-psqpn" Nov 03 00:40:12.531205 ip-10-0-140-107 kubenswrapper[2578]: I1103 00:40:12.530882 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/98ca3972-fd0c-47f5-9d26-569a58e18bd1-host-slash\") pod \"ovnkube-node-psqpn\" (UID: \"98ca3972-fd0c-47f5-9d26-569a58e18bd1\") " pod="openshift-ovn-kubernetes/ovnkube-node-psqpn" Nov 03 00:40:12.531205 ip-10-0-140-107 kubenswrapper[2578]: E1103 00:40:12.531056 2578 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Nov 03 00:40:12.531205 ip-10-0-140-107 kubenswrapper[2578]: I1103 00:40:12.531117 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/6650035b-2220-491c-9683-8a03ccb7af6a-tuning-conf-dir\") pod \"multus-additional-cni-plugins-gqppj\" (UID: \"6650035b-2220-491c-9683-8a03ccb7af6a\") " pod="openshift-multus/multus-additional-cni-plugins-gqppj" Nov 03 00:40:12.531205 ip-10-0-140-107 kubenswrapper[2578]: E1103 00:40:12.531140 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/136086c0-4846-437a-ad98-3cfef50610a0-metrics-certs podName:136086c0-4846-437a-ad98-3cfef50610a0 nodeName:}" failed. No retries permitted until 2026-11-03 00:40:13.031102918 +0000 UTC m=+2.976682429 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/136086c0-4846-437a-ad98-3cfef50610a0-metrics-certs") pod "network-metrics-daemon-rt2hx" (UID: "136086c0-4846-437a-ad98-3cfef50610a0") : object "openshift-multus"/"metrics-daemon-secret" not registered Nov 03 00:40:12.532073 ip-10-0-140-107 kubenswrapper[2578]: I1103 00:40:12.531737 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/00dc2920-34f2-4e0a-892e-7759a16162d5-host-run-k8s-cni-cncf-io\") pod \"multus-tb9hd\" (UID: \"00dc2920-34f2-4e0a-892e-7759a16162d5\") " pod="openshift-multus/multus-tb9hd" Nov 03 00:40:12.532073 ip-10-0-140-107 kubenswrapper[2578]: I1103 00:40:12.531890 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/6650035b-2220-491c-9683-8a03ccb7af6a-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-gqppj\" (UID: \"6650035b-2220-491c-9683-8a03ccb7af6a\") " pod="openshift-multus/multus-additional-cni-plugins-gqppj" Nov 03 00:40:12.532073 ip-10-0-140-107 kubenswrapper[2578]: I1103 00:40:12.531899 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/98ca3972-fd0c-47f5-9d26-569a58e18bd1-ovnkube-script-lib\") pod \"ovnkube-node-psqpn\" (UID: \"98ca3972-fd0c-47f5-9d26-569a58e18bd1\") " pod="openshift-ovn-kubernetes/ovnkube-node-psqpn" Nov 03 00:40:12.532073 ip-10-0-140-107 kubenswrapper[2578]: I1103 00:40:12.531960 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/98ca3972-fd0c-47f5-9d26-569a58e18bd1-host-run-ovn-kubernetes\") pod \"ovnkube-node-psqpn\" (UID: \"98ca3972-fd0c-47f5-9d26-569a58e18bd1\") " pod="openshift-ovn-kubernetes/ovnkube-node-psqpn" Nov 03 00:40:12.532073 ip-10-0-140-107 kubenswrapper[2578]: I1103 00:40:12.532021 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/6650035b-2220-491c-9683-8a03ccb7af6a-system-cni-dir\") pod \"multus-additional-cni-plugins-gqppj\" (UID: \"6650035b-2220-491c-9683-8a03ccb7af6a\") " pod="openshift-multus/multus-additional-cni-plugins-gqppj" Nov 03 00:40:12.532331 ip-10-0-140-107 kubenswrapper[2578]: I1103 00:40:12.532089 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/00dc2920-34f2-4e0a-892e-7759a16162d5-cni-binary-copy\") pod \"multus-tb9hd\" (UID: \"00dc2920-34f2-4e0a-892e-7759a16162d5\") " pod="openshift-multus/multus-tb9hd" Nov 03 00:40:12.532331 ip-10-0-140-107 kubenswrapper[2578]: I1103 00:40:12.532148 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/98ca3972-fd0c-47f5-9d26-569a58e18bd1-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-psqpn\" (UID: \"98ca3972-fd0c-47f5-9d26-569a58e18bd1\") " pod="openshift-ovn-kubernetes/ovnkube-node-psqpn" Nov 03 00:40:12.532331 ip-10-0-140-107 kubenswrapper[2578]: I1103 00:40:12.532189 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/00dc2920-34f2-4e0a-892e-7759a16162d5-host-var-lib-cni-multus\") pod \"multus-tb9hd\" (UID: \"00dc2920-34f2-4e0a-892e-7759a16162d5\") " pod="openshift-multus/multus-tb9hd" Nov 03 00:40:12.532331 ip-10-0-140-107 kubenswrapper[2578]: I1103 00:40:12.532227 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/98ca3972-fd0c-47f5-9d26-569a58e18bd1-ovnkube-config\") pod \"ovnkube-node-psqpn\" (UID: \"98ca3972-fd0c-47f5-9d26-569a58e18bd1\") " pod="openshift-ovn-kubernetes/ovnkube-node-psqpn" Nov 03 00:40:12.532331 ip-10-0-140-107 kubenswrapper[2578]: I1103 00:40:12.532236 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/00dc2920-34f2-4e0a-892e-7759a16162d5-cnibin\") pod \"multus-tb9hd\" (UID: \"00dc2920-34f2-4e0a-892e-7759a16162d5\") " pod="openshift-multus/multus-tb9hd" Nov 03 00:40:12.532331 ip-10-0-140-107 kubenswrapper[2578]: I1103 00:40:12.532322 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/98ca3972-fd0c-47f5-9d26-569a58e18bd1-etc-openvswitch\") pod \"ovnkube-node-psqpn\" (UID: \"98ca3972-fd0c-47f5-9d26-569a58e18bd1\") " pod="openshift-ovn-kubernetes/ovnkube-node-psqpn" Nov 03 00:40:12.532632 ip-10-0-140-107 kubenswrapper[2578]: I1103 00:40:12.532354 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/00dc2920-34f2-4e0a-892e-7759a16162d5-host-run-multus-certs\") pod \"multus-tb9hd\" (UID: \"00dc2920-34f2-4e0a-892e-7759a16162d5\") " pod="openshift-multus/multus-tb9hd" Nov 03 00:40:12.532632 ip-10-0-140-107 kubenswrapper[2578]: I1103 00:40:12.532409 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/00dc2920-34f2-4e0a-892e-7759a16162d5-hostroot\") pod \"multus-tb9hd\" (UID: \"00dc2920-34f2-4e0a-892e-7759a16162d5\") " pod="openshift-multus/multus-tb9hd" Nov 03 00:40:12.532632 ip-10-0-140-107 kubenswrapper[2578]: I1103 00:40:12.532419 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/00dc2920-34f2-4e0a-892e-7759a16162d5-multus-cni-dir\") pod \"multus-tb9hd\" (UID: \"00dc2920-34f2-4e0a-892e-7759a16162d5\") " pod="openshift-multus/multus-tb9hd" Nov 03 00:40:12.532632 ip-10-0-140-107 kubenswrapper[2578]: I1103 00:40:12.532431 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/6650035b-2220-491c-9683-8a03ccb7af6a-cni-binary-copy\") pod \"multus-additional-cni-plugins-gqppj\" (UID: \"6650035b-2220-491c-9683-8a03ccb7af6a\") " pod="openshift-multus/multus-additional-cni-plugins-gqppj" Nov 03 00:40:12.532632 ip-10-0-140-107 kubenswrapper[2578]: I1103 00:40:12.532465 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/98ca3972-fd0c-47f5-9d26-569a58e18bd1-node-log\") pod \"ovnkube-node-psqpn\" (UID: \"98ca3972-fd0c-47f5-9d26-569a58e18bd1\") " pod="openshift-ovn-kubernetes/ovnkube-node-psqpn" Nov 03 00:40:12.532632 ip-10-0-140-107 kubenswrapper[2578]: I1103 00:40:12.532472 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/98ca3972-fd0c-47f5-9d26-569a58e18bd1-host-cni-bin\") pod \"ovnkube-node-psqpn\" (UID: \"98ca3972-fd0c-47f5-9d26-569a58e18bd1\") " pod="openshift-ovn-kubernetes/ovnkube-node-psqpn" Nov 03 00:40:12.532632 ip-10-0-140-107 kubenswrapper[2578]: I1103 00:40:12.532477 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/98ca3972-fd0c-47f5-9d26-569a58e18bd1-host-run-netns\") pod \"ovnkube-node-psqpn\" (UID: \"98ca3972-fd0c-47f5-9d26-569a58e18bd1\") " pod="openshift-ovn-kubernetes/ovnkube-node-psqpn" Nov 03 00:40:12.532632 ip-10-0-140-107 kubenswrapper[2578]: I1103 00:40:12.532503 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/98ca3972-fd0c-47f5-9d26-569a58e18bd1-run-systemd\") pod \"ovnkube-node-psqpn\" (UID: \"98ca3972-fd0c-47f5-9d26-569a58e18bd1\") " pod="openshift-ovn-kubernetes/ovnkube-node-psqpn" Nov 03 00:40:12.532632 ip-10-0-140-107 kubenswrapper[2578]: I1103 00:40:12.532515 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/98ca3972-fd0c-47f5-9d26-569a58e18bd1-run-ovn\") pod \"ovnkube-node-psqpn\" (UID: \"98ca3972-fd0c-47f5-9d26-569a58e18bd1\") " pod="openshift-ovn-kubernetes/ovnkube-node-psqpn" Nov 03 00:40:12.532632 ip-10-0-140-107 kubenswrapper[2578]: I1103 00:40:12.532557 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/00dc2920-34f2-4e0a-892e-7759a16162d5-multus-socket-dir-parent\") pod \"multus-tb9hd\" (UID: \"00dc2920-34f2-4e0a-892e-7759a16162d5\") " pod="openshift-multus/multus-tb9hd" Nov 03 00:40:12.532632 ip-10-0-140-107 kubenswrapper[2578]: I1103 00:40:12.532597 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/00dc2920-34f2-4e0a-892e-7759a16162d5-host-var-lib-cni-bin\") pod \"multus-tb9hd\" (UID: \"00dc2920-34f2-4e0a-892e-7759a16162d5\") " pod="openshift-multus/multus-tb9hd" Nov 03 00:40:12.532632 ip-10-0-140-107 kubenswrapper[2578]: I1103 00:40:12.532600 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/98ca3972-fd0c-47f5-9d26-569a58e18bd1-env-overrides\") pod \"ovnkube-node-psqpn\" (UID: \"98ca3972-fd0c-47f5-9d26-569a58e18bd1\") " pod="openshift-ovn-kubernetes/ovnkube-node-psqpn" Nov 03 00:40:12.532632 ip-10-0-140-107 kubenswrapper[2578]: I1103 00:40:12.532638 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/98ca3972-fd0c-47f5-9d26-569a58e18bd1-run-openvswitch\") pod \"ovnkube-node-psqpn\" (UID: \"98ca3972-fd0c-47f5-9d26-569a58e18bd1\") " pod="openshift-ovn-kubernetes/ovnkube-node-psqpn" Nov 03 00:40:12.533403 ip-10-0-140-107 kubenswrapper[2578]: I1103 00:40:12.532652 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/98ca3972-fd0c-47f5-9d26-569a58e18bd1-var-lib-openvswitch\") pod \"ovnkube-node-psqpn\" (UID: \"98ca3972-fd0c-47f5-9d26-569a58e18bd1\") " pod="openshift-ovn-kubernetes/ovnkube-node-psqpn" Nov 03 00:40:12.533403 ip-10-0-140-107 kubenswrapper[2578]: I1103 00:40:12.532694 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/6650035b-2220-491c-9683-8a03ccb7af6a-cnibin\") pod \"multus-additional-cni-plugins-gqppj\" (UID: \"6650035b-2220-491c-9683-8a03ccb7af6a\") " pod="openshift-multus/multus-additional-cni-plugins-gqppj" Nov 03 00:40:12.533403 ip-10-0-140-107 kubenswrapper[2578]: I1103 00:40:12.532736 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/98ca3972-fd0c-47f5-9d26-569a58e18bd1-host-cni-netd\") pod \"ovnkube-node-psqpn\" (UID: \"98ca3972-fd0c-47f5-9d26-569a58e18bd1\") " pod="openshift-ovn-kubernetes/ovnkube-node-psqpn" Nov 03 00:40:12.533403 ip-10-0-140-107 kubenswrapper[2578]: I1103 00:40:12.532761 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/98ca3972-fd0c-47f5-9d26-569a58e18bd1-host-kubelet\") pod \"ovnkube-node-psqpn\" (UID: \"98ca3972-fd0c-47f5-9d26-569a58e18bd1\") " pod="openshift-ovn-kubernetes/ovnkube-node-psqpn" Nov 03 00:40:12.533403 ip-10-0-140-107 kubenswrapper[2578]: I1103 00:40:12.532779 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/00dc2920-34f2-4e0a-892e-7759a16162d5-multus-conf-dir\") pod \"multus-tb9hd\" (UID: \"00dc2920-34f2-4e0a-892e-7759a16162d5\") " pod="openshift-multus/multus-tb9hd" Nov 03 00:40:12.533403 ip-10-0-140-107 kubenswrapper[2578]: I1103 00:40:12.532831 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/00dc2920-34f2-4e0a-892e-7759a16162d5-system-cni-dir\") pod \"multus-tb9hd\" (UID: \"00dc2920-34f2-4e0a-892e-7759a16162d5\") " pod="openshift-multus/multus-tb9hd" Nov 03 00:40:12.533403 ip-10-0-140-107 kubenswrapper[2578]: I1103 00:40:12.532858 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/00dc2920-34f2-4e0a-892e-7759a16162d5-host-var-lib-kubelet\") pod \"multus-tb9hd\" (UID: \"00dc2920-34f2-4e0a-892e-7759a16162d5\") " pod="openshift-multus/multus-tb9hd" Nov 03 00:40:12.533403 ip-10-0-140-107 kubenswrapper[2578]: I1103 00:40:12.532903 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/c736fd3f-973f-4999-8d4e-c405079b2888-host-slash\") pod \"iptables-alerter-5spnk\" (UID: \"c736fd3f-973f-4999-8d4e-c405079b2888\") " pod="openshift-network-operator/iptables-alerter-5spnk" Nov 03 00:40:12.533403 ip-10-0-140-107 kubenswrapper[2578]: I1103 00:40:12.532959 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/6650035b-2220-491c-9683-8a03ccb7af6a-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-gqppj\" (UID: \"6650035b-2220-491c-9683-8a03ccb7af6a\") " pod="openshift-multus/multus-additional-cni-plugins-gqppj" Nov 03 00:40:12.533403 ip-10-0-140-107 kubenswrapper[2578]: I1103 00:40:12.532959 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/00dc2920-34f2-4e0a-892e-7759a16162d5-multus-daemon-config\") pod \"multus-tb9hd\" (UID: \"00dc2920-34f2-4e0a-892e-7759a16162d5\") " pod="openshift-multus/multus-tb9hd" Nov 03 00:40:12.533403 ip-10-0-140-107 kubenswrapper[2578]: I1103 00:40:12.533366 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/c736fd3f-973f-4999-8d4e-c405079b2888-iptables-alerter-script\") pod \"iptables-alerter-5spnk\" (UID: \"c736fd3f-973f-4999-8d4e-c405079b2888\") " pod="openshift-network-operator/iptables-alerter-5spnk" Nov 03 00:40:12.535191 ip-10-0-140-107 kubenswrapper[2578]: I1103 00:40:12.535161 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/98ca3972-fd0c-47f5-9d26-569a58e18bd1-ovn-node-metrics-cert\") pod \"ovnkube-node-psqpn\" (UID: \"98ca3972-fd0c-47f5-9d26-569a58e18bd1\") " pod="openshift-ovn-kubernetes/ovnkube-node-psqpn" Nov 03 00:40:12.543472 ip-10-0-140-107 kubenswrapper[2578]: E1103 00:40:12.543438 2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Nov 03 00:40:12.543592 ip-10-0-140-107 kubenswrapper[2578]: E1103 00:40:12.543479 2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Nov 03 00:40:12.543592 ip-10-0-140-107 kubenswrapper[2578]: E1103 00:40:12.543494 2578 projected.go:196] Error preparing data for projected volume kube-api-access-d8l25 for pod openshift-network-diagnostics/network-check-target-w46kd: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 03 00:40:12.543592 ip-10-0-140-107 kubenswrapper[2578]: E1103 00:40:12.543587 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/8cd3ca22-e4ce-4896-ae31-97605af4e4c6-kube-api-access-d8l25 podName:8cd3ca22-e4ce-4896-ae31-97605af4e4c6 nodeName:}" failed. No retries permitted until 2026-11-03 00:40:13.043532855 +0000 UTC m=+2.989112349 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-d8l25" (UniqueName: "kubernetes.io/projected/8cd3ca22-e4ce-4896-ae31-97605af4e4c6-kube-api-access-d8l25") pod "network-check-target-w46kd" (UID: "8cd3ca22-e4ce-4896-ae31-97605af4e4c6") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 03 00:40:12.546205 ip-10-0-140-107 kubenswrapper[2578]: I1103 00:40:12.546175 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-j62gg\" (UniqueName: \"kubernetes.io/projected/00dc2920-34f2-4e0a-892e-7759a16162d5-kube-api-access-j62gg\") pod \"multus-tb9hd\" (UID: \"00dc2920-34f2-4e0a-892e-7759a16162d5\") " pod="openshift-multus/multus-tb9hd" Nov 03 00:40:12.546393 ip-10-0-140-107 kubenswrapper[2578]: I1103 00:40:12.546371 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rkf7s\" (UniqueName: \"kubernetes.io/projected/6650035b-2220-491c-9683-8a03ccb7af6a-kube-api-access-rkf7s\") pod \"multus-additional-cni-plugins-gqppj\" (UID: \"6650035b-2220-491c-9683-8a03ccb7af6a\") " pod="openshift-multus/multus-additional-cni-plugins-gqppj" Nov 03 00:40:12.547202 ip-10-0-140-107 kubenswrapper[2578]: I1103 00:40:12.547178 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2tdmf\" (UniqueName: \"kubernetes.io/projected/c736fd3f-973f-4999-8d4e-c405079b2888-kube-api-access-2tdmf\") pod \"iptables-alerter-5spnk\" (UID: \"c736fd3f-973f-4999-8d4e-c405079b2888\") " pod="openshift-network-operator/iptables-alerter-5spnk" Nov 03 00:40:12.547416 ip-10-0-140-107 kubenswrapper[2578]: I1103 00:40:12.547382 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4bhqh\" (UniqueName: \"kubernetes.io/projected/98ca3972-fd0c-47f5-9d26-569a58e18bd1-kube-api-access-4bhqh\") pod \"ovnkube-node-psqpn\" (UID: \"98ca3972-fd0c-47f5-9d26-569a58e18bd1\") " pod="openshift-ovn-kubernetes/ovnkube-node-psqpn" Nov 03 00:40:12.549736 ip-10-0-140-107 kubenswrapper[2578]: I1103 00:40:12.549714 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-llh2g\" (UniqueName: \"kubernetes.io/projected/136086c0-4846-437a-ad98-3cfef50610a0-kube-api-access-llh2g\") pod \"network-metrics-daemon-rt2hx\" (UID: \"136086c0-4846-437a-ad98-3cfef50610a0\") " pod="openshift-multus/network-metrics-daemon-rt2hx" Nov 03 00:40:12.613255 ip-10-0-140-107 kubenswrapper[2578]: I1103 00:40:12.613198 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-140-107.us-east-2.compute.internal" event={"ID":"a730b11d5a720feea75c07e6e10d3b92","Type":"ContainerStarted","Data":"a012e88fc324e4ecd1bc5f9557af5fe91bc73d451cee98e718e6ecf7cf8da48f"} Nov 03 00:40:12.614680 ip-10-0-140-107 kubenswrapper[2578]: I1103 00:40:12.614651 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-107.us-east-2.compute.internal" event={"ID":"0fb35ee5f976a9435ef306be01bc3e71","Type":"ContainerStarted","Data":"20d31a990fda9ed8069aa304f22413ca1c459b7d9b9b0e268ac30fa12af2d633"} Nov 03 00:40:12.631323 ip-10-0-140-107 kubenswrapper[2578]: I1103 00:40:12.631294 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/37bf96f5-045f-49e8-83e8-410df66bfb1f-sys-fs\") pod \"aws-ebs-csi-driver-node-br8vt\" (UID: \"37bf96f5-045f-49e8-83e8-410df66bfb1f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-br8vt" Nov 03 00:40:12.631452 ip-10-0-140-107 kubenswrapper[2578]: I1103 00:40:12.631334 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/c3ffbe53-7f23-4750-b95d-476e6cfc58a6-konnectivity-ca\") pod \"konnectivity-agent-82x8c\" (UID: \"c3ffbe53-7f23-4750-b95d-476e6cfc58a6\") " pod="kube-system/konnectivity-agent-82x8c" Nov 03 00:40:12.631452 ip-10-0-140-107 kubenswrapper[2578]: I1103 00:40:12.631361 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/b748c4f8-d021-4d37-b390-d395482fcddd-sys\") pod \"tuned-gtwxp\" (UID: \"b748c4f8-d021-4d37-b390-d395482fcddd\") " pod="openshift-cluster-node-tuning-operator/tuned-gtwxp" Nov 03 00:40:12.631452 ip-10-0-140-107 kubenswrapper[2578]: I1103 00:40:12.631387 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/37bf96f5-045f-49e8-83e8-410df66bfb1f-etc-selinux\") pod \"aws-ebs-csi-driver-node-br8vt\" (UID: \"37bf96f5-045f-49e8-83e8-410df66bfb1f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-br8vt" Nov 03 00:40:12.631452 ip-10-0-140-107 kubenswrapper[2578]: I1103 00:40:12.631411 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/b748c4f8-d021-4d37-b390-d395482fcddd-etc-tuned\") pod \"tuned-gtwxp\" (UID: \"b748c4f8-d021-4d37-b390-d395482fcddd\") " pod="openshift-cluster-node-tuning-operator/tuned-gtwxp" Nov 03 00:40:12.631452 ip-10-0-140-107 kubenswrapper[2578]: I1103 00:40:12.631428 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/37bf96f5-045f-49e8-83e8-410df66bfb1f-sys-fs\") pod \"aws-ebs-csi-driver-node-br8vt\" (UID: \"37bf96f5-045f-49e8-83e8-410df66bfb1f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-br8vt" Nov 03 00:40:12.631452 ip-10-0-140-107 kubenswrapper[2578]: I1103 00:40:12.631440 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/c3ffbe53-7f23-4750-b95d-476e6cfc58a6-agent-certs\") pod \"konnectivity-agent-82x8c\" (UID: \"c3ffbe53-7f23-4750-b95d-476e6cfc58a6\") " pod="kube-system/konnectivity-agent-82x8c" Nov 03 00:40:12.631782 ip-10-0-140-107 kubenswrapper[2578]: I1103 00:40:12.631469 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hsvqw\" (UniqueName: \"kubernetes.io/projected/b748c4f8-d021-4d37-b390-d395482fcddd-kube-api-access-hsvqw\") pod \"tuned-gtwxp\" (UID: \"b748c4f8-d021-4d37-b390-d395482fcddd\") " pod="openshift-cluster-node-tuning-operator/tuned-gtwxp" Nov 03 00:40:12.631782 ip-10-0-140-107 kubenswrapper[2578]: I1103 00:40:12.631499 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/37bf96f5-045f-49e8-83e8-410df66bfb1f-kubelet-dir\") pod \"aws-ebs-csi-driver-node-br8vt\" (UID: \"37bf96f5-045f-49e8-83e8-410df66bfb1f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-br8vt" Nov 03 00:40:12.631782 ip-10-0-140-107 kubenswrapper[2578]: I1103 00:40:12.631529 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/37bf96f5-045f-49e8-83e8-410df66bfb1f-registration-dir\") pod \"aws-ebs-csi-driver-node-br8vt\" (UID: \"37bf96f5-045f-49e8-83e8-410df66bfb1f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-br8vt" Nov 03 00:40:12.631782 ip-10-0-140-107 kubenswrapper[2578]: I1103 00:40:12.631548 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/37bf96f5-045f-49e8-83e8-410df66bfb1f-etc-selinux\") pod \"aws-ebs-csi-driver-node-br8vt\" (UID: \"37bf96f5-045f-49e8-83e8-410df66bfb1f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-br8vt" Nov 03 00:40:12.631782 ip-10-0-140-107 kubenswrapper[2578]: I1103 00:40:12.631557 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/ad5bf442-729d-411c-96e4-2e5105ebadcf-hosts-file\") pod \"node-resolver-cjj2h\" (UID: \"ad5bf442-729d-411c-96e4-2e5105ebadcf\") " pod="openshift-dns/node-resolver-cjj2h" Nov 03 00:40:12.631782 ip-10-0-140-107 kubenswrapper[2578]: I1103 00:40:12.631606 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/37bf96f5-045f-49e8-83e8-410df66bfb1f-socket-dir\") pod \"aws-ebs-csi-driver-node-br8vt\" (UID: \"37bf96f5-045f-49e8-83e8-410df66bfb1f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-br8vt" Nov 03 00:40:12.631782 ip-10-0-140-107 kubenswrapper[2578]: I1103 00:40:12.631609 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/37bf96f5-045f-49e8-83e8-410df66bfb1f-kubelet-dir\") pod \"aws-ebs-csi-driver-node-br8vt\" (UID: \"37bf96f5-045f-49e8-83e8-410df66bfb1f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-br8vt" Nov 03 00:40:12.631782 ip-10-0-140-107 kubenswrapper[2578]: I1103 00:40:12.631635 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lp26w\" (UniqueName: \"kubernetes.io/projected/37bf96f5-045f-49e8-83e8-410df66bfb1f-kube-api-access-lp26w\") pod \"aws-ebs-csi-driver-node-br8vt\" (UID: \"37bf96f5-045f-49e8-83e8-410df66bfb1f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-br8vt" Nov 03 00:40:12.631782 ip-10-0-140-107 kubenswrapper[2578]: I1103 00:40:12.631670 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/460779b2-3903-4b23-9b4d-e25142782a21-host\") pod \"node-ca-bbk2h\" (UID: \"460779b2-3903-4b23-9b4d-e25142782a21\") " pod="openshift-image-registry/node-ca-bbk2h" Nov 03 00:40:12.631782 ip-10-0-140-107 kubenswrapper[2578]: I1103 00:40:12.631695 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q4b9t\" (UniqueName: \"kubernetes.io/projected/460779b2-3903-4b23-9b4d-e25142782a21-kube-api-access-q4b9t\") pod \"node-ca-bbk2h\" (UID: \"460779b2-3903-4b23-9b4d-e25142782a21\") " pod="openshift-image-registry/node-ca-bbk2h" Nov 03 00:40:12.631782 ip-10-0-140-107 kubenswrapper[2578]: I1103 00:40:12.631732 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ad5bf442-729d-411c-96e4-2e5105ebadcf-tmp-dir\") pod \"node-resolver-cjj2h\" (UID: \"ad5bf442-729d-411c-96e4-2e5105ebadcf\") " pod="openshift-dns/node-resolver-cjj2h" Nov 03 00:40:12.631782 ip-10-0-140-107 kubenswrapper[2578]: I1103 00:40:12.631753 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/b748c4f8-d021-4d37-b390-d395482fcddd-etc-systemd\") pod \"tuned-gtwxp\" (UID: \"b748c4f8-d021-4d37-b390-d395482fcddd\") " pod="openshift-cluster-node-tuning-operator/tuned-gtwxp" Nov 03 00:40:12.631782 ip-10-0-140-107 kubenswrapper[2578]: I1103 00:40:12.631769 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/b748c4f8-d021-4d37-b390-d395482fcddd-etc-modprobe-d\") pod \"tuned-gtwxp\" (UID: \"b748c4f8-d021-4d37-b390-d395482fcddd\") " pod="openshift-cluster-node-tuning-operator/tuned-gtwxp" Nov 03 00:40:12.631782 ip-10-0-140-107 kubenswrapper[2578]: I1103 00:40:12.631783 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/b748c4f8-d021-4d37-b390-d395482fcddd-tmp\") pod \"tuned-gtwxp\" (UID: \"b748c4f8-d021-4d37-b390-d395482fcddd\") " pod="openshift-cluster-node-tuning-operator/tuned-gtwxp" Nov 03 00:40:12.632435 ip-10-0-140-107 kubenswrapper[2578]: I1103 00:40:12.631801 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/b748c4f8-d021-4d37-b390-d395482fcddd-run\") pod \"tuned-gtwxp\" (UID: \"b748c4f8-d021-4d37-b390-d395482fcddd\") " pod="openshift-cluster-node-tuning-operator/tuned-gtwxp" Nov 03 00:40:12.632435 ip-10-0-140-107 kubenswrapper[2578]: I1103 00:40:12.631821 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/37bf96f5-045f-49e8-83e8-410df66bfb1f-device-dir\") pod \"aws-ebs-csi-driver-node-br8vt\" (UID: \"37bf96f5-045f-49e8-83e8-410df66bfb1f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-br8vt" Nov 03 00:40:12.632435 ip-10-0-140-107 kubenswrapper[2578]: I1103 00:40:12.631847 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/b748c4f8-d021-4d37-b390-d395482fcddd-etc-kubernetes\") pod \"tuned-gtwxp\" (UID: \"b748c4f8-d021-4d37-b390-d395482fcddd\") " pod="openshift-cluster-node-tuning-operator/tuned-gtwxp" Nov 03 00:40:12.632435 ip-10-0-140-107 kubenswrapper[2578]: I1103 00:40:12.631872 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/cabd402c-79c5-4699-9c85-d415d362b4d0-kubelet-config\") pod \"global-pull-secret-syncer-jv7cw\" (UID: \"cabd402c-79c5-4699-9c85-d415d362b4d0\") " pod="kube-system/global-pull-secret-syncer-jv7cw" Nov 03 00:40:12.632435 ip-10-0-140-107 kubenswrapper[2578]: I1103 00:40:12.631892 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/b748c4f8-d021-4d37-b390-d395482fcddd-etc-sysconfig\") pod \"tuned-gtwxp\" (UID: \"b748c4f8-d021-4d37-b390-d395482fcddd\") " pod="openshift-cluster-node-tuning-operator/tuned-gtwxp" Nov 03 00:40:12.632435 ip-10-0-140-107 kubenswrapper[2578]: I1103 00:40:12.631966 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/b748c4f8-d021-4d37-b390-d395482fcddd-etc-sysctl-conf\") pod \"tuned-gtwxp\" (UID: \"b748c4f8-d021-4d37-b390-d395482fcddd\") " pod="openshift-cluster-node-tuning-operator/tuned-gtwxp" Nov 03 00:40:12.632435 ip-10-0-140-107 kubenswrapper[2578]: I1103 00:40:12.632024 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/b748c4f8-d021-4d37-b390-d395482fcddd-var-lib-kubelet\") pod \"tuned-gtwxp\" (UID: \"b748c4f8-d021-4d37-b390-d395482fcddd\") " pod="openshift-cluster-node-tuning-operator/tuned-gtwxp" Nov 03 00:40:12.632435 ip-10-0-140-107 kubenswrapper[2578]: I1103 00:40:12.632021 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/b748c4f8-d021-4d37-b390-d395482fcddd-run\") pod \"tuned-gtwxp\" (UID: \"b748c4f8-d021-4d37-b390-d395482fcddd\") " pod="openshift-cluster-node-tuning-operator/tuned-gtwxp" Nov 03 00:40:12.632435 ip-10-0-140-107 kubenswrapper[2578]: I1103 00:40:12.632036 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/c3ffbe53-7f23-4750-b95d-476e6cfc58a6-konnectivity-ca\") pod \"konnectivity-agent-82x8c\" (UID: \"c3ffbe53-7f23-4750-b95d-476e6cfc58a6\") " pod="kube-system/konnectivity-agent-82x8c" Nov 03 00:40:12.632696 ip-10-0-140-107 kubenswrapper[2578]: I1103 00:40:12.632536 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/b748c4f8-d021-4d37-b390-d395482fcddd-etc-systemd\") pod \"tuned-gtwxp\" (UID: \"b748c4f8-d021-4d37-b390-d395482fcddd\") " pod="openshift-cluster-node-tuning-operator/tuned-gtwxp" Nov 03 00:40:12.632696 ip-10-0-140-107 kubenswrapper[2578]: I1103 00:40:12.632664 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/b748c4f8-d021-4d37-b390-d395482fcddd-etc-modprobe-d\") pod \"tuned-gtwxp\" (UID: \"b748c4f8-d021-4d37-b390-d395482fcddd\") " pod="openshift-cluster-node-tuning-operator/tuned-gtwxp" Nov 03 00:40:12.633414 ip-10-0-140-107 kubenswrapper[2578]: I1103 00:40:12.633381 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sfdp9\" (UniqueName: \"kubernetes.io/projected/ad5bf442-729d-411c-96e4-2e5105ebadcf-kube-api-access-sfdp9\") pod \"node-resolver-cjj2h\" (UID: \"ad5bf442-729d-411c-96e4-2e5105ebadcf\") " pod="openshift-dns/node-resolver-cjj2h" Nov 03 00:40:12.633528 ip-10-0-140-107 kubenswrapper[2578]: I1103 00:40:12.633444 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/b748c4f8-d021-4d37-b390-d395482fcddd-lib-modules\") pod \"tuned-gtwxp\" (UID: \"b748c4f8-d021-4d37-b390-d395482fcddd\") " pod="openshift-cluster-node-tuning-operator/tuned-gtwxp" Nov 03 00:40:12.633528 ip-10-0-140-107 kubenswrapper[2578]: I1103 00:40:12.633486 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/b748c4f8-d021-4d37-b390-d395482fcddd-host\") pod \"tuned-gtwxp\" (UID: \"b748c4f8-d021-4d37-b390-d395482fcddd\") " pod="openshift-cluster-node-tuning-operator/tuned-gtwxp" Nov 03 00:40:12.633595 ip-10-0-140-107 kubenswrapper[2578]: I1103 00:40:12.633572 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/cabd402c-79c5-4699-9c85-d415d362b4d0-original-pull-secret\") pod \"global-pull-secret-syncer-jv7cw\" (UID: \"cabd402c-79c5-4699-9c85-d415d362b4d0\") " pod="kube-system/global-pull-secret-syncer-jv7cw" Nov 03 00:40:12.633652 ip-10-0-140-107 kubenswrapper[2578]: I1103 00:40:12.633634 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/cabd402c-79c5-4699-9c85-d415d362b4d0-dbus\") pod \"global-pull-secret-syncer-jv7cw\" (UID: \"cabd402c-79c5-4699-9c85-d415d362b4d0\") " pod="kube-system/global-pull-secret-syncer-jv7cw" Nov 03 00:40:12.633691 ip-10-0-140-107 kubenswrapper[2578]: I1103 00:40:12.633678 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/b748c4f8-d021-4d37-b390-d395482fcddd-etc-sysctl-d\") pod \"tuned-gtwxp\" (UID: \"b748c4f8-d021-4d37-b390-d395482fcddd\") " pod="openshift-cluster-node-tuning-operator/tuned-gtwxp" Nov 03 00:40:12.633765 ip-10-0-140-107 kubenswrapper[2578]: I1103 00:40:12.633750 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/b748c4f8-d021-4d37-b390-d395482fcddd-host\") pod \"tuned-gtwxp\" (UID: \"b748c4f8-d021-4d37-b390-d395482fcddd\") " pod="openshift-cluster-node-tuning-operator/tuned-gtwxp" Nov 03 00:40:12.633828 ip-10-0-140-107 kubenswrapper[2578]: I1103 00:40:12.633813 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/b748c4f8-d021-4d37-b390-d395482fcddd-var-lib-kubelet\") pod \"tuned-gtwxp\" (UID: \"b748c4f8-d021-4d37-b390-d395482fcddd\") " pod="openshift-cluster-node-tuning-operator/tuned-gtwxp" Nov 03 00:40:12.633887 ip-10-0-140-107 kubenswrapper[2578]: I1103 00:40:12.633872 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/37bf96f5-045f-49e8-83e8-410df66bfb1f-device-dir\") pod \"aws-ebs-csi-driver-node-br8vt\" (UID: \"37bf96f5-045f-49e8-83e8-410df66bfb1f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-br8vt" Nov 03 00:40:12.633945 ip-10-0-140-107 kubenswrapper[2578]: I1103 00:40:12.633932 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/b748c4f8-d021-4d37-b390-d395482fcddd-etc-kubernetes\") pod \"tuned-gtwxp\" (UID: \"b748c4f8-d021-4d37-b390-d395482fcddd\") " pod="openshift-cluster-node-tuning-operator/tuned-gtwxp" Nov 03 00:40:12.634012 ip-10-0-140-107 kubenswrapper[2578]: I1103 00:40:12.633997 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/cabd402c-79c5-4699-9c85-d415d362b4d0-kubelet-config\") pod \"global-pull-secret-syncer-jv7cw\" (UID: \"cabd402c-79c5-4699-9c85-d415d362b4d0\") " pod="kube-system/global-pull-secret-syncer-jv7cw" Nov 03 00:40:12.634073 ip-10-0-140-107 kubenswrapper[2578]: I1103 00:40:12.634054 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/b748c4f8-d021-4d37-b390-d395482fcddd-etc-sysconfig\") pod \"tuned-gtwxp\" (UID: \"b748c4f8-d021-4d37-b390-d395482fcddd\") " pod="openshift-cluster-node-tuning-operator/tuned-gtwxp" Nov 03 00:40:12.634454 ip-10-0-140-107 kubenswrapper[2578]: I1103 00:40:12.634433 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/b748c4f8-d021-4d37-b390-d395482fcddd-lib-modules\") pod \"tuned-gtwxp\" (UID: \"b748c4f8-d021-4d37-b390-d395482fcddd\") " pod="openshift-cluster-node-tuning-operator/tuned-gtwxp" Nov 03 00:40:12.634579 ip-10-0-140-107 kubenswrapper[2578]: E1103 00:40:12.634563 2578 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Nov 03 00:40:12.634665 ip-10-0-140-107 kubenswrapper[2578]: I1103 00:40:12.634648 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/cabd402c-79c5-4699-9c85-d415d362b4d0-dbus\") pod \"global-pull-secret-syncer-jv7cw\" (UID: \"cabd402c-79c5-4699-9c85-d415d362b4d0\") " pod="kube-system/global-pull-secret-syncer-jv7cw" Nov 03 00:40:12.634800 ip-10-0-140-107 kubenswrapper[2578]: I1103 00:40:12.634782 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/b748c4f8-d021-4d37-b390-d395482fcddd-etc-sysctl-d\") pod \"tuned-gtwxp\" (UID: \"b748c4f8-d021-4d37-b390-d395482fcddd\") " pod="openshift-cluster-node-tuning-operator/tuned-gtwxp" Nov 03 00:40:12.634899 ip-10-0-140-107 kubenswrapper[2578]: E1103 00:40:12.634887 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cabd402c-79c5-4699-9c85-d415d362b4d0-original-pull-secret podName:cabd402c-79c5-4699-9c85-d415d362b4d0 nodeName:}" failed. No retries permitted until 2026-11-03 00:40:13.134867123 +0000 UTC m=+3.080446617 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/cabd402c-79c5-4699-9c85-d415d362b4d0-original-pull-secret") pod "global-pull-secret-syncer-jv7cw" (UID: "cabd402c-79c5-4699-9c85-d415d362b4d0") : object "kube-system"/"original-pull-secret" not registered Nov 03 00:40:12.635126 ip-10-0-140-107 kubenswrapper[2578]: I1103 00:40:12.635101 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/460779b2-3903-4b23-9b4d-e25142782a21-serviceca\") pod \"node-ca-bbk2h\" (UID: \"460779b2-3903-4b23-9b4d-e25142782a21\") " pod="openshift-image-registry/node-ca-bbk2h" Nov 03 00:40:12.635578 ip-10-0-140-107 kubenswrapper[2578]: I1103 00:40:12.635541 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/460779b2-3903-4b23-9b4d-e25142782a21-serviceca\") pod \"node-ca-bbk2h\" (UID: \"460779b2-3903-4b23-9b4d-e25142782a21\") " pod="openshift-image-registry/node-ca-bbk2h" Nov 03 00:40:12.635701 ip-10-0-140-107 kubenswrapper[2578]: I1103 00:40:12.635676 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ad5bf442-729d-411c-96e4-2e5105ebadcf-tmp-dir\") pod \"node-resolver-cjj2h\" (UID: \"ad5bf442-729d-411c-96e4-2e5105ebadcf\") " pod="openshift-dns/node-resolver-cjj2h" Nov 03 00:40:12.636344 ip-10-0-140-107 kubenswrapper[2578]: I1103 00:40:12.636291 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/b748c4f8-d021-4d37-b390-d395482fcddd-etc-sysctl-conf\") pod \"tuned-gtwxp\" (UID: \"b748c4f8-d021-4d37-b390-d395482fcddd\") " pod="openshift-cluster-node-tuning-operator/tuned-gtwxp" Nov 03 00:40:12.636477 ip-10-0-140-107 kubenswrapper[2578]: I1103 00:40:12.636458 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/37bf96f5-045f-49e8-83e8-410df66bfb1f-socket-dir\") pod \"aws-ebs-csi-driver-node-br8vt\" (UID: \"37bf96f5-045f-49e8-83e8-410df66bfb1f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-br8vt" Nov 03 00:40:12.636558 ip-10-0-140-107 kubenswrapper[2578]: I1103 00:40:12.631482 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/b748c4f8-d021-4d37-b390-d395482fcddd-sys\") pod \"tuned-gtwxp\" (UID: \"b748c4f8-d021-4d37-b390-d395482fcddd\") " pod="openshift-cluster-node-tuning-operator/tuned-gtwxp" Nov 03 00:40:12.636558 ip-10-0-140-107 kubenswrapper[2578]: I1103 00:40:12.636548 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/b748c4f8-d021-4d37-b390-d395482fcddd-tmp\") pod \"tuned-gtwxp\" (UID: \"b748c4f8-d021-4d37-b390-d395482fcddd\") " pod="openshift-cluster-node-tuning-operator/tuned-gtwxp" Nov 03 00:40:12.636659 ip-10-0-140-107 kubenswrapper[2578]: I1103 00:40:12.636551 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/37bf96f5-045f-49e8-83e8-410df66bfb1f-registration-dir\") pod \"aws-ebs-csi-driver-node-br8vt\" (UID: \"37bf96f5-045f-49e8-83e8-410df66bfb1f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-br8vt" Nov 03 00:40:12.636719 ip-10-0-140-107 kubenswrapper[2578]: I1103 00:40:12.636655 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/460779b2-3903-4b23-9b4d-e25142782a21-host\") pod \"node-ca-bbk2h\" (UID: \"460779b2-3903-4b23-9b4d-e25142782a21\") " pod="openshift-image-registry/node-ca-bbk2h" Nov 03 00:40:12.636776 ip-10-0-140-107 kubenswrapper[2578]: I1103 00:40:12.636727 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/ad5bf442-729d-411c-96e4-2e5105ebadcf-hosts-file\") pod \"node-resolver-cjj2h\" (UID: \"ad5bf442-729d-411c-96e4-2e5105ebadcf\") " pod="openshift-dns/node-resolver-cjj2h" Nov 03 00:40:12.637471 ip-10-0-140-107 kubenswrapper[2578]: I1103 00:40:12.637442 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/b748c4f8-d021-4d37-b390-d395482fcddd-etc-tuned\") pod \"tuned-gtwxp\" (UID: \"b748c4f8-d021-4d37-b390-d395482fcddd\") " pod="openshift-cluster-node-tuning-operator/tuned-gtwxp" Nov 03 00:40:12.640452 ip-10-0-140-107 kubenswrapper[2578]: I1103 00:40:12.640428 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/c3ffbe53-7f23-4750-b95d-476e6cfc58a6-agent-certs\") pod \"konnectivity-agent-82x8c\" (UID: \"c3ffbe53-7f23-4750-b95d-476e6cfc58a6\") " pod="kube-system/konnectivity-agent-82x8c" Nov 03 00:40:12.647820 ip-10-0-140-107 kubenswrapper[2578]: I1103 00:40:12.647795 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hsvqw\" (UniqueName: \"kubernetes.io/projected/b748c4f8-d021-4d37-b390-d395482fcddd-kube-api-access-hsvqw\") pod \"tuned-gtwxp\" (UID: \"b748c4f8-d021-4d37-b390-d395482fcddd\") " pod="openshift-cluster-node-tuning-operator/tuned-gtwxp" Nov 03 00:40:12.647975 ip-10-0-140-107 kubenswrapper[2578]: I1103 00:40:12.647950 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sfdp9\" (UniqueName: \"kubernetes.io/projected/ad5bf442-729d-411c-96e4-2e5105ebadcf-kube-api-access-sfdp9\") pod \"node-resolver-cjj2h\" (UID: \"ad5bf442-729d-411c-96e4-2e5105ebadcf\") " pod="openshift-dns/node-resolver-cjj2h" Nov 03 00:40:12.649380 ip-10-0-140-107 kubenswrapper[2578]: I1103 00:40:12.649292 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-q4b9t\" (UniqueName: \"kubernetes.io/projected/460779b2-3903-4b23-9b4d-e25142782a21-kube-api-access-q4b9t\") pod \"node-ca-bbk2h\" (UID: \"460779b2-3903-4b23-9b4d-e25142782a21\") " pod="openshift-image-registry/node-ca-bbk2h" Nov 03 00:40:12.649715 ip-10-0-140-107 kubenswrapper[2578]: I1103 00:40:12.649693 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lp26w\" (UniqueName: \"kubernetes.io/projected/37bf96f5-045f-49e8-83e8-410df66bfb1f-kube-api-access-lp26w\") pod \"aws-ebs-csi-driver-node-br8vt\" (UID: \"37bf96f5-045f-49e8-83e8-410df66bfb1f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-br8vt" Nov 03 00:40:12.724272 ip-10-0-140-107 kubenswrapper[2578]: I1103 00:40:12.724186 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-5spnk" Nov 03 00:40:12.737370 ip-10-0-140-107 kubenswrapper[2578]: W1103 00:40:12.737324 2578 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc736fd3f_973f_4999_8d4e_c405079b2888.slice/crio-3f1b02020b063d5bec095ef4de846f7d7590f5dec7e9630c6f92094ff6c01081 WatchSource:0}: Error finding container 3f1b02020b063d5bec095ef4de846f7d7590f5dec7e9630c6f92094ff6c01081: Status 404 returned error can't find the container with id 3f1b02020b063d5bec095ef4de846f7d7590f5dec7e9630c6f92094ff6c01081 Nov 03 00:40:12.752457 ip-10-0-140-107 kubenswrapper[2578]: I1103 00:40:12.752431 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-tb9hd" Nov 03 00:40:12.755338 ip-10-0-140-107 kubenswrapper[2578]: I1103 00:40:12.755315 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-gqppj" Nov 03 00:40:12.764087 ip-10-0-140-107 kubenswrapper[2578]: W1103 00:40:12.764063 2578 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod00dc2920_34f2_4e0a_892e_7759a16162d5.slice/crio-4112437df1446eb838fd35176a32848f7974fa73d27056b56cfcd766cd9894da WatchSource:0}: Error finding container 4112437df1446eb838fd35176a32848f7974fa73d27056b56cfcd766cd9894da: Status 404 returned error can't find the container with id 4112437df1446eb838fd35176a32848f7974fa73d27056b56cfcd766cd9894da Nov 03 00:40:12.767558 ip-10-0-140-107 kubenswrapper[2578]: W1103 00:40:12.767534 2578 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6650035b_2220_491c_9683_8a03ccb7af6a.slice/crio-8867ce85967697dcfba6c103bcc04489490e98decd0f1994a213282757db1130 WatchSource:0}: Error finding container 8867ce85967697dcfba6c103bcc04489490e98decd0f1994a213282757db1130: Status 404 returned error can't find the container with id 8867ce85967697dcfba6c103bcc04489490e98decd0f1994a213282757db1130 Nov 03 00:40:12.777499 ip-10-0-140-107 kubenswrapper[2578]: I1103 00:40:12.777476 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-82x8c" Nov 03 00:40:12.779699 ip-10-0-140-107 kubenswrapper[2578]: I1103 00:40:12.779665 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-br8vt" Nov 03 00:40:12.787929 ip-10-0-140-107 kubenswrapper[2578]: I1103 00:40:12.787907 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-cjj2h" Nov 03 00:40:12.792166 ip-10-0-140-107 kubenswrapper[2578]: I1103 00:40:12.792150 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-bbk2h" Nov 03 00:40:12.794631 ip-10-0-140-107 kubenswrapper[2578]: I1103 00:40:12.794346 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-psqpn" Nov 03 00:40:12.798370 ip-10-0-140-107 kubenswrapper[2578]: W1103 00:40:12.798336 2578 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podad5bf442_729d_411c_96e4_2e5105ebadcf.slice/crio-70a1f3b0a3f91bb331a7be1e85fed083f6f1bd0252c8c3c3c47a6dc14cf781c5 WatchSource:0}: Error finding container 70a1f3b0a3f91bb331a7be1e85fed083f6f1bd0252c8c3c3c47a6dc14cf781c5: Status 404 returned error can't find the container with id 70a1f3b0a3f91bb331a7be1e85fed083f6f1bd0252c8c3c3c47a6dc14cf781c5 Nov 03 00:40:12.801928 ip-10-0-140-107 kubenswrapper[2578]: I1103 00:40:12.801751 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-gtwxp" Nov 03 00:40:12.806128 ip-10-0-140-107 kubenswrapper[2578]: W1103 00:40:12.806094 2578 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod460779b2_3903_4b23_9b4d_e25142782a21.slice/crio-e0c24e4a15e4f03b8a6d417c965c9bee757e311cc8ce503619d167866b0b6daa WatchSource:0}: Error finding container e0c24e4a15e4f03b8a6d417c965c9bee757e311cc8ce503619d167866b0b6daa: Status 404 returned error can't find the container with id e0c24e4a15e4f03b8a6d417c965c9bee757e311cc8ce503619d167866b0b6daa Nov 03 00:40:12.810040 ip-10-0-140-107 kubenswrapper[2578]: W1103 00:40:12.809991 2578 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod98ca3972_fd0c_47f5_9d26_569a58e18bd1.slice/crio-c5d2d73eebd8feaeb150e256ec01b80911dcc4ec8baffc65888eb14551386136 WatchSource:0}: Error finding container c5d2d73eebd8feaeb150e256ec01b80911dcc4ec8baffc65888eb14551386136: Status 404 returned error can't find the container with id c5d2d73eebd8feaeb150e256ec01b80911dcc4ec8baffc65888eb14551386136 Nov 03 00:40:12.812944 ip-10-0-140-107 kubenswrapper[2578]: W1103 00:40:12.812893 2578 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb748c4f8_d021_4d37_b390_d395482fcddd.slice/crio-d53c015a0548e079f2530a41c467870dbe83375bd0feafa3ca7e5cea801e0c24 WatchSource:0}: Error finding container d53c015a0548e079f2530a41c467870dbe83375bd0feafa3ca7e5cea801e0c24: Status 404 returned error can't find the container with id d53c015a0548e079f2530a41c467870dbe83375bd0feafa3ca7e5cea801e0c24 Nov 03 00:40:13.038427 ip-10-0-140-107 kubenswrapper[2578]: I1103 00:40:13.038245 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/136086c0-4846-437a-ad98-3cfef50610a0-metrics-certs\") pod \"network-metrics-daemon-rt2hx\" (UID: \"136086c0-4846-437a-ad98-3cfef50610a0\") " pod="openshift-multus/network-metrics-daemon-rt2hx" Nov 03 00:40:13.038427 ip-10-0-140-107 kubenswrapper[2578]: E1103 00:40:13.038422 2578 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Nov 03 00:40:13.038667 ip-10-0-140-107 kubenswrapper[2578]: E1103 00:40:13.038493 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/136086c0-4846-437a-ad98-3cfef50610a0-metrics-certs podName:136086c0-4846-437a-ad98-3cfef50610a0 nodeName:}" failed. No retries permitted until 2026-11-03 00:40:14.038472421 +0000 UTC m=+3.984051902 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/136086c0-4846-437a-ad98-3cfef50610a0-metrics-certs") pod "network-metrics-daemon-rt2hx" (UID: "136086c0-4846-437a-ad98-3cfef50610a0") : object "openshift-multus"/"metrics-daemon-secret" not registered Nov 03 00:40:13.139599 ip-10-0-140-107 kubenswrapper[2578]: I1103 00:40:13.139558 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/cabd402c-79c5-4699-9c85-d415d362b4d0-original-pull-secret\") pod \"global-pull-secret-syncer-jv7cw\" (UID: \"cabd402c-79c5-4699-9c85-d415d362b4d0\") " pod="kube-system/global-pull-secret-syncer-jv7cw" Nov 03 00:40:13.139800 ip-10-0-140-107 kubenswrapper[2578]: I1103 00:40:13.139628 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d8l25\" (UniqueName: \"kubernetes.io/projected/8cd3ca22-e4ce-4896-ae31-97605af4e4c6-kube-api-access-d8l25\") pod \"network-check-target-w46kd\" (UID: \"8cd3ca22-e4ce-4896-ae31-97605af4e4c6\") " pod="openshift-network-diagnostics/network-check-target-w46kd" Nov 03 00:40:13.139800 ip-10-0-140-107 kubenswrapper[2578]: E1103 00:40:13.139735 2578 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Nov 03 00:40:13.139800 ip-10-0-140-107 kubenswrapper[2578]: E1103 00:40:13.139763 2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Nov 03 00:40:13.139800 ip-10-0-140-107 kubenswrapper[2578]: E1103 00:40:13.139780 2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Nov 03 00:40:13.139995 ip-10-0-140-107 kubenswrapper[2578]: E1103 00:40:13.139805 2578 projected.go:196] Error preparing data for projected volume kube-api-access-d8l25 for pod openshift-network-diagnostics/network-check-target-w46kd: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 03 00:40:13.139995 ip-10-0-140-107 kubenswrapper[2578]: E1103 00:40:13.139813 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cabd402c-79c5-4699-9c85-d415d362b4d0-original-pull-secret podName:cabd402c-79c5-4699-9c85-d415d362b4d0 nodeName:}" failed. No retries permitted until 2026-11-03 00:40:14.139793226 +0000 UTC m=+4.085372704 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/cabd402c-79c5-4699-9c85-d415d362b4d0-original-pull-secret") pod "global-pull-secret-syncer-jv7cw" (UID: "cabd402c-79c5-4699-9c85-d415d362b4d0") : object "kube-system"/"original-pull-secret" not registered Nov 03 00:40:13.139995 ip-10-0-140-107 kubenswrapper[2578]: E1103 00:40:13.139859 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/8cd3ca22-e4ce-4896-ae31-97605af4e4c6-kube-api-access-d8l25 podName:8cd3ca22-e4ce-4896-ae31-97605af4e4c6 nodeName:}" failed. No retries permitted until 2026-11-03 00:40:14.139841223 +0000 UTC m=+4.085420721 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-d8l25" (UniqueName: "kubernetes.io/projected/8cd3ca22-e4ce-4896-ae31-97605af4e4c6-kube-api-access-d8l25") pod "network-check-target-w46kd" (UID: "8cd3ca22-e4ce-4896-ae31-97605af4e4c6") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 03 00:40:13.466547 ip-10-0-140-107 kubenswrapper[2578]: I1103 00:40:13.466459 2578 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-11-02 00:35:12 +0000 UTC" deadline="2028-06-15 21:03:05.601497898 +0000 UTC" Nov 03 00:40:13.466547 ip-10-0-140-107 kubenswrapper[2578]: I1103 00:40:13.466523 2578 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="14180h22m52.13497852s" Nov 03 00:40:13.605565 ip-10-0-140-107 kubenswrapper[2578]: I1103 00:40:13.605487 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-rt2hx" Nov 03 00:40:13.605724 ip-10-0-140-107 kubenswrapper[2578]: E1103 00:40:13.605659 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-rt2hx" podUID="136086c0-4846-437a-ad98-3cfef50610a0" Nov 03 00:40:13.621500 ip-10-0-140-107 kubenswrapper[2578]: I1103 00:40:13.621410 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-gtwxp" event={"ID":"b748c4f8-d021-4d37-b390-d395482fcddd","Type":"ContainerStarted","Data":"d53c015a0548e079f2530a41c467870dbe83375bd0feafa3ca7e5cea801e0c24"} Nov 03 00:40:13.623560 ip-10-0-140-107 kubenswrapper[2578]: I1103 00:40:13.623483 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-bbk2h" event={"ID":"460779b2-3903-4b23-9b4d-e25142782a21","Type":"ContainerStarted","Data":"e0c24e4a15e4f03b8a6d417c965c9bee757e311cc8ce503619d167866b0b6daa"} Nov 03 00:40:13.628792 ip-10-0-140-107 kubenswrapper[2578]: I1103 00:40:13.628756 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-cjj2h" event={"ID":"ad5bf442-729d-411c-96e4-2e5105ebadcf","Type":"ContainerStarted","Data":"70a1f3b0a3f91bb331a7be1e85fed083f6f1bd0252c8c3c3c47a6dc14cf781c5"} Nov 03 00:40:13.655297 ip-10-0-140-107 kubenswrapper[2578]: I1103 00:40:13.655259 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-br8vt" event={"ID":"37bf96f5-045f-49e8-83e8-410df66bfb1f","Type":"ContainerStarted","Data":"1c5c6a32bfd30ad97469ed1fb0359d4cf5bc1d6c7254eef2ad99974885969c36"} Nov 03 00:40:13.660342 ip-10-0-140-107 kubenswrapper[2578]: I1103 00:40:13.660306 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-82x8c" event={"ID":"c3ffbe53-7f23-4750-b95d-476e6cfc58a6","Type":"ContainerStarted","Data":"18b88862137befca64ff94878901feb875f51616dd9b95cd34057759a64e55c1"} Nov 03 00:40:13.663023 ip-10-0-140-107 kubenswrapper[2578]: I1103 00:40:13.662948 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-gqppj" event={"ID":"6650035b-2220-491c-9683-8a03ccb7af6a","Type":"ContainerStarted","Data":"8867ce85967697dcfba6c103bcc04489490e98decd0f1994a213282757db1130"} Nov 03 00:40:13.665422 ip-10-0-140-107 kubenswrapper[2578]: I1103 00:40:13.665395 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-psqpn" event={"ID":"98ca3972-fd0c-47f5-9d26-569a58e18bd1","Type":"ContainerStarted","Data":"c5d2d73eebd8feaeb150e256ec01b80911dcc4ec8baffc65888eb14551386136"} Nov 03 00:40:13.668025 ip-10-0-140-107 kubenswrapper[2578]: I1103 00:40:13.667982 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-tb9hd" event={"ID":"00dc2920-34f2-4e0a-892e-7759a16162d5","Type":"ContainerStarted","Data":"4112437df1446eb838fd35176a32848f7974fa73d27056b56cfcd766cd9894da"} Nov 03 00:40:13.669380 ip-10-0-140-107 kubenswrapper[2578]: I1103 00:40:13.669354 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-5spnk" event={"ID":"c736fd3f-973f-4999-8d4e-c405079b2888","Type":"ContainerStarted","Data":"3f1b02020b063d5bec095ef4de846f7d7590f5dec7e9630c6f92094ff6c01081"} Nov 03 00:40:14.047263 ip-10-0-140-107 kubenswrapper[2578]: I1103 00:40:14.047231 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/136086c0-4846-437a-ad98-3cfef50610a0-metrics-certs\") pod \"network-metrics-daemon-rt2hx\" (UID: \"136086c0-4846-437a-ad98-3cfef50610a0\") " pod="openshift-multus/network-metrics-daemon-rt2hx" Nov 03 00:40:14.047489 ip-10-0-140-107 kubenswrapper[2578]: E1103 00:40:14.047470 2578 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Nov 03 00:40:14.047619 ip-10-0-140-107 kubenswrapper[2578]: E1103 00:40:14.047557 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/136086c0-4846-437a-ad98-3cfef50610a0-metrics-certs podName:136086c0-4846-437a-ad98-3cfef50610a0 nodeName:}" failed. No retries permitted until 2026-11-03 00:40:16.047537493 +0000 UTC m=+5.993116987 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/136086c0-4846-437a-ad98-3cfef50610a0-metrics-certs") pod "network-metrics-daemon-rt2hx" (UID: "136086c0-4846-437a-ad98-3cfef50610a0") : object "openshift-multus"/"metrics-daemon-secret" not registered Nov 03 00:40:14.148130 ip-10-0-140-107 kubenswrapper[2578]: I1103 00:40:14.148088 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/cabd402c-79c5-4699-9c85-d415d362b4d0-original-pull-secret\") pod \"global-pull-secret-syncer-jv7cw\" (UID: \"cabd402c-79c5-4699-9c85-d415d362b4d0\") " pod="kube-system/global-pull-secret-syncer-jv7cw" Nov 03 00:40:14.148293 ip-10-0-140-107 kubenswrapper[2578]: I1103 00:40:14.148170 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d8l25\" (UniqueName: \"kubernetes.io/projected/8cd3ca22-e4ce-4896-ae31-97605af4e4c6-kube-api-access-d8l25\") pod \"network-check-target-w46kd\" (UID: \"8cd3ca22-e4ce-4896-ae31-97605af4e4c6\") " pod="openshift-network-diagnostics/network-check-target-w46kd" Nov 03 00:40:14.148362 ip-10-0-140-107 kubenswrapper[2578]: E1103 00:40:14.148346 2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Nov 03 00:40:14.148406 ip-10-0-140-107 kubenswrapper[2578]: E1103 00:40:14.148372 2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Nov 03 00:40:14.148406 ip-10-0-140-107 kubenswrapper[2578]: E1103 00:40:14.148387 2578 projected.go:196] Error preparing data for projected volume kube-api-access-d8l25 for pod openshift-network-diagnostics/network-check-target-w46kd: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 03 00:40:14.148464 ip-10-0-140-107 kubenswrapper[2578]: E1103 00:40:14.148449 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/8cd3ca22-e4ce-4896-ae31-97605af4e4c6-kube-api-access-d8l25 podName:8cd3ca22-e4ce-4896-ae31-97605af4e4c6 nodeName:}" failed. No retries permitted until 2026-11-03 00:40:16.148430872 +0000 UTC m=+6.094010370 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-d8l25" (UniqueName: "kubernetes.io/projected/8cd3ca22-e4ce-4896-ae31-97605af4e4c6-kube-api-access-d8l25") pod "network-check-target-w46kd" (UID: "8cd3ca22-e4ce-4896-ae31-97605af4e4c6") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 03 00:40:14.148879 ip-10-0-140-107 kubenswrapper[2578]: E1103 00:40:14.148861 2578 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Nov 03 00:40:14.148938 ip-10-0-140-107 kubenswrapper[2578]: E1103 00:40:14.148924 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cabd402c-79c5-4699-9c85-d415d362b4d0-original-pull-secret podName:cabd402c-79c5-4699-9c85-d415d362b4d0 nodeName:}" failed. No retries permitted until 2026-11-03 00:40:16.14890734 +0000 UTC m=+6.094486836 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/cabd402c-79c5-4699-9c85-d415d362b4d0-original-pull-secret") pod "global-pull-secret-syncer-jv7cw" (UID: "cabd402c-79c5-4699-9c85-d415d362b4d0") : object "kube-system"/"original-pull-secret" not registered Nov 03 00:40:14.467341 ip-10-0-140-107 kubenswrapper[2578]: I1103 00:40:14.467292 2578 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-11-02 00:35:12 +0000 UTC" deadline="2028-04-19 01:33:36.887043182 +0000 UTC" Nov 03 00:40:14.467790 ip-10-0-140-107 kubenswrapper[2578]: I1103 00:40:14.467340 2578 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="12792h53m22.419709161s" Nov 03 00:40:14.608611 ip-10-0-140-107 kubenswrapper[2578]: I1103 00:40:14.607881 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-w46kd" Nov 03 00:40:14.608611 ip-10-0-140-107 kubenswrapper[2578]: E1103 00:40:14.608053 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-w46kd" podUID="8cd3ca22-e4ce-4896-ae31-97605af4e4c6" Nov 03 00:40:14.608611 ip-10-0-140-107 kubenswrapper[2578]: I1103 00:40:14.608146 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-jv7cw" Nov 03 00:40:14.608611 ip-10-0-140-107 kubenswrapper[2578]: E1103 00:40:14.608226 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-jv7cw" podUID="cabd402c-79c5-4699-9c85-d415d362b4d0" Nov 03 00:40:14.678219 ip-10-0-140-107 kubenswrapper[2578]: I1103 00:40:14.677409 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-140-107.us-east-2.compute.internal" event={"ID":"a730b11d5a720feea75c07e6e10d3b92","Type":"ContainerStarted","Data":"808947fe85aaddb159920b9c10697b79d54be1af13b0fb84d76663d9faca7b9b"} Nov 03 00:40:14.709601 ip-10-0-140-107 kubenswrapper[2578]: I1103 00:40:14.709534 2578 generic.go:356] "Generic (PLEG): container finished" podID="0fb35ee5f976a9435ef306be01bc3e71" containerID="c303e752f251d21c490ca1d8c26f71a79a56276d00a75842417a646b16661e86" exitCode=0 Nov 03 00:40:14.709784 ip-10-0-140-107 kubenswrapper[2578]: I1103 00:40:14.709621 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-107.us-east-2.compute.internal" event={"ID":"0fb35ee5f976a9435ef306be01bc3e71","Type":"ContainerDied","Data":"c303e752f251d21c490ca1d8c26f71a79a56276d00a75842417a646b16661e86"} Nov 03 00:40:14.746179 ip-10-0-140-107 kubenswrapper[2578]: I1103 00:40:14.745212 2578 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-140-107.us-east-2.compute.internal" podStartSLOduration=2.745190367 podStartE2EDuration="2.745190367s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-03 00:40:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:40:14.702435008 +0000 UTC m=+4.648014511" watchObservedRunningTime="2026-11-03 00:40:14.745190367 +0000 UTC m=+4.690769871" Nov 03 00:40:15.605150 ip-10-0-140-107 kubenswrapper[2578]: I1103 00:40:15.605111 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-rt2hx" Nov 03 00:40:15.605769 ip-10-0-140-107 kubenswrapper[2578]: E1103 00:40:15.605267 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-rt2hx" podUID="136086c0-4846-437a-ad98-3cfef50610a0" Nov 03 00:40:15.748374 ip-10-0-140-107 kubenswrapper[2578]: I1103 00:40:15.748315 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-107.us-east-2.compute.internal" event={"ID":"0fb35ee5f976a9435ef306be01bc3e71","Type":"ContainerStarted","Data":"0099f1b1a3a4dc99c6aa41edc76e67c8425e216ae53b77b6a6b6dcabacac6cac"} Nov 03 00:40:16.072201 ip-10-0-140-107 kubenswrapper[2578]: I1103 00:40:16.072108 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/136086c0-4846-437a-ad98-3cfef50610a0-metrics-certs\") pod \"network-metrics-daemon-rt2hx\" (UID: \"136086c0-4846-437a-ad98-3cfef50610a0\") " pod="openshift-multus/network-metrics-daemon-rt2hx" Nov 03 00:40:16.072382 ip-10-0-140-107 kubenswrapper[2578]: E1103 00:40:16.072308 2578 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Nov 03 00:40:16.072460 ip-10-0-140-107 kubenswrapper[2578]: E1103 00:40:16.072390 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/136086c0-4846-437a-ad98-3cfef50610a0-metrics-certs podName:136086c0-4846-437a-ad98-3cfef50610a0 nodeName:}" failed. No retries permitted until 2026-11-03 00:40:20.072369338 +0000 UTC m=+10.017948835 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/136086c0-4846-437a-ad98-3cfef50610a0-metrics-certs") pod "network-metrics-daemon-rt2hx" (UID: "136086c0-4846-437a-ad98-3cfef50610a0") : object "openshift-multus"/"metrics-daemon-secret" not registered Nov 03 00:40:16.172622 ip-10-0-140-107 kubenswrapper[2578]: I1103 00:40:16.172580 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/cabd402c-79c5-4699-9c85-d415d362b4d0-original-pull-secret\") pod \"global-pull-secret-syncer-jv7cw\" (UID: \"cabd402c-79c5-4699-9c85-d415d362b4d0\") " pod="kube-system/global-pull-secret-syncer-jv7cw" Nov 03 00:40:16.172779 ip-10-0-140-107 kubenswrapper[2578]: I1103 00:40:16.172658 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d8l25\" (UniqueName: \"kubernetes.io/projected/8cd3ca22-e4ce-4896-ae31-97605af4e4c6-kube-api-access-d8l25\") pod \"network-check-target-w46kd\" (UID: \"8cd3ca22-e4ce-4896-ae31-97605af4e4c6\") " pod="openshift-network-diagnostics/network-check-target-w46kd" Nov 03 00:40:16.172919 ip-10-0-140-107 kubenswrapper[2578]: E1103 00:40:16.172844 2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Nov 03 00:40:16.172919 ip-10-0-140-107 kubenswrapper[2578]: E1103 00:40:16.172911 2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Nov 03 00:40:16.173043 ip-10-0-140-107 kubenswrapper[2578]: E1103 00:40:16.172926 2578 projected.go:196] Error preparing data for projected volume kube-api-access-d8l25 for pod openshift-network-diagnostics/network-check-target-w46kd: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 03 00:40:16.173043 ip-10-0-140-107 kubenswrapper[2578]: E1103 00:40:16.173011 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/8cd3ca22-e4ce-4896-ae31-97605af4e4c6-kube-api-access-d8l25 podName:8cd3ca22-e4ce-4896-ae31-97605af4e4c6 nodeName:}" failed. No retries permitted until 2026-11-03 00:40:20.172990514 +0000 UTC m=+10.118570011 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-d8l25" (UniqueName: "kubernetes.io/projected/8cd3ca22-e4ce-4896-ae31-97605af4e4c6-kube-api-access-d8l25") pod "network-check-target-w46kd" (UID: "8cd3ca22-e4ce-4896-ae31-97605af4e4c6") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 03 00:40:16.173559 ip-10-0-140-107 kubenswrapper[2578]: E1103 00:40:16.173538 2578 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Nov 03 00:40:16.173612 ip-10-0-140-107 kubenswrapper[2578]: E1103 00:40:16.173600 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cabd402c-79c5-4699-9c85-d415d362b4d0-original-pull-secret podName:cabd402c-79c5-4699-9c85-d415d362b4d0 nodeName:}" failed. No retries permitted until 2026-11-03 00:40:20.173585229 +0000 UTC m=+10.119164708 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/cabd402c-79c5-4699-9c85-d415d362b4d0-original-pull-secret") pod "global-pull-secret-syncer-jv7cw" (UID: "cabd402c-79c5-4699-9c85-d415d362b4d0") : object "kube-system"/"original-pull-secret" not registered Nov 03 00:40:16.607330 ip-10-0-140-107 kubenswrapper[2578]: I1103 00:40:16.607294 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-w46kd" Nov 03 00:40:16.607809 ip-10-0-140-107 kubenswrapper[2578]: E1103 00:40:16.607437 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-w46kd" podUID="8cd3ca22-e4ce-4896-ae31-97605af4e4c6" Nov 03 00:40:16.607809 ip-10-0-140-107 kubenswrapper[2578]: I1103 00:40:16.607527 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-jv7cw" Nov 03 00:40:16.607809 ip-10-0-140-107 kubenswrapper[2578]: E1103 00:40:16.607591 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-jv7cw" podUID="cabd402c-79c5-4699-9c85-d415d362b4d0" Nov 03 00:40:16.764338 ip-10-0-140-107 kubenswrapper[2578]: I1103 00:40:16.764305 2578 generic.go:356] "Generic (PLEG): container finished" podID="6650035b-2220-491c-9683-8a03ccb7af6a" containerID="d61acd42b9668fea1316a57002b752f2a500e975f2eca1afedafac05104db6fd" exitCode=0 Nov 03 00:40:16.765310 ip-10-0-140-107 kubenswrapper[2578]: I1103 00:40:16.765277 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-gqppj" event={"ID":"6650035b-2220-491c-9683-8a03ccb7af6a","Type":"ContainerDied","Data":"d61acd42b9668fea1316a57002b752f2a500e975f2eca1afedafac05104db6fd"} Nov 03 00:40:16.794966 ip-10-0-140-107 kubenswrapper[2578]: I1103 00:40:16.794885 2578 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-107.us-east-2.compute.internal" podStartSLOduration=4.794864947 podStartE2EDuration="4.794864947s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-03 00:40:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:40:15.766156786 +0000 UTC m=+5.711736290" watchObservedRunningTime="2026-11-03 00:40:16.794864947 +0000 UTC m=+6.740444443" Nov 03 00:40:17.604970 ip-10-0-140-107 kubenswrapper[2578]: I1103 00:40:17.604929 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-rt2hx" Nov 03 00:40:17.605170 ip-10-0-140-107 kubenswrapper[2578]: E1103 00:40:17.605148 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-rt2hx" podUID="136086c0-4846-437a-ad98-3cfef50610a0" Nov 03 00:40:17.781030 ip-10-0-140-107 kubenswrapper[2578]: I1103 00:40:17.780991 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-82x8c" event={"ID":"c3ffbe53-7f23-4750-b95d-476e6cfc58a6","Type":"ContainerStarted","Data":"ea906622d917ab724ec9a1bfbf5aed4153e0948afe901922bde11e9857d7ff43"} Nov 03 00:40:18.606935 ip-10-0-140-107 kubenswrapper[2578]: I1103 00:40:18.605040 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-w46kd" Nov 03 00:40:18.606935 ip-10-0-140-107 kubenswrapper[2578]: E1103 00:40:18.605165 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-w46kd" podUID="8cd3ca22-e4ce-4896-ae31-97605af4e4c6" Nov 03 00:40:18.606935 ip-10-0-140-107 kubenswrapper[2578]: I1103 00:40:18.605255 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-jv7cw" Nov 03 00:40:18.606935 ip-10-0-140-107 kubenswrapper[2578]: E1103 00:40:18.605382 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-jv7cw" podUID="cabd402c-79c5-4699-9c85-d415d362b4d0" Nov 03 00:40:18.786524 ip-10-0-140-107 kubenswrapper[2578]: I1103 00:40:18.786465 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-bbk2h" event={"ID":"460779b2-3903-4b23-9b4d-e25142782a21","Type":"ContainerStarted","Data":"8dddf80f04900fd5830b4af93e2967da16f7e95c9f85c1739684d0f25ed7af53"} Nov 03 00:40:18.802709 ip-10-0-140-107 kubenswrapper[2578]: I1103 00:40:18.802657 2578 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/konnectivity-agent-82x8c" podStartSLOduration=3.7615258799999998 podStartE2EDuration="7.802636107s" totalImagesPullingTime="4.041110227s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:40:11 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:40:17.798680758 +0000 UTC m=+7.744260257" watchObservedRunningTime="2026-11-03 00:40:18.802636107 +0000 UTC m=+8.748215607" Nov 03 00:40:18.802872 ip-10-0-140-107 kubenswrapper[2578]: I1103 00:40:18.802813 2578 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-image-registry/node-ca-bbk2h" podStartSLOduration=2.59560429 podStartE2EDuration="7.802805202s" totalImagesPullingTime="5.207200912s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-03 00:40:11 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:40:18.802720133 +0000 UTC m=+8.748299645" watchObservedRunningTime="2026-11-03 00:40:18.802805202 +0000 UTC m=+8.748384702" Nov 03 00:40:19.605383 ip-10-0-140-107 kubenswrapper[2578]: I1103 00:40:19.605344 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-rt2hx" Nov 03 00:40:19.605583 ip-10-0-140-107 kubenswrapper[2578]: E1103 00:40:19.605490 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-rt2hx" podUID="136086c0-4846-437a-ad98-3cfef50610a0" Nov 03 00:40:19.794749 ip-10-0-140-107 kubenswrapper[2578]: I1103 00:40:19.794711 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-br8vt" event={"ID":"37bf96f5-045f-49e8-83e8-410df66bfb1f","Type":"ContainerStarted","Data":"8dc76675fff8954cec3110cc6eebb0c96120d9974dc38ed1b4f8844957fcc2e6"} Nov 03 00:40:20.107526 ip-10-0-140-107 kubenswrapper[2578]: I1103 00:40:20.106939 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/136086c0-4846-437a-ad98-3cfef50610a0-metrics-certs\") pod \"network-metrics-daemon-rt2hx\" (UID: \"136086c0-4846-437a-ad98-3cfef50610a0\") " pod="openshift-multus/network-metrics-daemon-rt2hx" Nov 03 00:40:20.107526 ip-10-0-140-107 kubenswrapper[2578]: E1103 00:40:20.107115 2578 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Nov 03 00:40:20.107526 ip-10-0-140-107 kubenswrapper[2578]: E1103 00:40:20.107183 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/136086c0-4846-437a-ad98-3cfef50610a0-metrics-certs podName:136086c0-4846-437a-ad98-3cfef50610a0 nodeName:}" failed. No retries permitted until 2026-11-03 00:40:28.107165465 +0000 UTC m=+18.052744957 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/136086c0-4846-437a-ad98-3cfef50610a0-metrics-certs") pod "network-metrics-daemon-rt2hx" (UID: "136086c0-4846-437a-ad98-3cfef50610a0") : object "openshift-multus"/"metrics-daemon-secret" not registered Nov 03 00:40:20.208802 ip-10-0-140-107 kubenswrapper[2578]: I1103 00:40:20.207812 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/cabd402c-79c5-4699-9c85-d415d362b4d0-original-pull-secret\") pod \"global-pull-secret-syncer-jv7cw\" (UID: \"cabd402c-79c5-4699-9c85-d415d362b4d0\") " pod="kube-system/global-pull-secret-syncer-jv7cw" Nov 03 00:40:20.208802 ip-10-0-140-107 kubenswrapper[2578]: I1103 00:40:20.207877 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d8l25\" (UniqueName: \"kubernetes.io/projected/8cd3ca22-e4ce-4896-ae31-97605af4e4c6-kube-api-access-d8l25\") pod \"network-check-target-w46kd\" (UID: \"8cd3ca22-e4ce-4896-ae31-97605af4e4c6\") " pod="openshift-network-diagnostics/network-check-target-w46kd" Nov 03 00:40:20.208802 ip-10-0-140-107 kubenswrapper[2578]: E1103 00:40:20.208036 2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Nov 03 00:40:20.208802 ip-10-0-140-107 kubenswrapper[2578]: E1103 00:40:20.208063 2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Nov 03 00:40:20.208802 ip-10-0-140-107 kubenswrapper[2578]: E1103 00:40:20.208076 2578 projected.go:196] Error preparing data for projected volume kube-api-access-d8l25 for pod openshift-network-diagnostics/network-check-target-w46kd: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 03 00:40:20.208802 ip-10-0-140-107 kubenswrapper[2578]: E1103 00:40:20.208138 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/8cd3ca22-e4ce-4896-ae31-97605af4e4c6-kube-api-access-d8l25 podName:8cd3ca22-e4ce-4896-ae31-97605af4e4c6 nodeName:}" failed. No retries permitted until 2026-11-03 00:40:28.208119949 +0000 UTC m=+18.153699449 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-d8l25" (UniqueName: "kubernetes.io/projected/8cd3ca22-e4ce-4896-ae31-97605af4e4c6-kube-api-access-d8l25") pod "network-check-target-w46kd" (UID: "8cd3ca22-e4ce-4896-ae31-97605af4e4c6") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 03 00:40:20.208802 ip-10-0-140-107 kubenswrapper[2578]: E1103 00:40:20.208539 2578 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Nov 03 00:40:20.208802 ip-10-0-140-107 kubenswrapper[2578]: E1103 00:40:20.208590 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cabd402c-79c5-4699-9c85-d415d362b4d0-original-pull-secret podName:cabd402c-79c5-4699-9c85-d415d362b4d0 nodeName:}" failed. No retries permitted until 2026-11-03 00:40:28.208575532 +0000 UTC m=+18.154155041 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/cabd402c-79c5-4699-9c85-d415d362b4d0-original-pull-secret") pod "global-pull-secret-syncer-jv7cw" (UID: "cabd402c-79c5-4699-9c85-d415d362b4d0") : object "kube-system"/"original-pull-secret" not registered Nov 03 00:40:20.609930 ip-10-0-140-107 kubenswrapper[2578]: I1103 00:40:20.609880 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-jv7cw" Nov 03 00:40:20.610160 ip-10-0-140-107 kubenswrapper[2578]: E1103 00:40:20.610128 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-jv7cw" podUID="cabd402c-79c5-4699-9c85-d415d362b4d0" Nov 03 00:40:20.610535 ip-10-0-140-107 kubenswrapper[2578]: I1103 00:40:20.610512 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-w46kd" Nov 03 00:40:20.610692 ip-10-0-140-107 kubenswrapper[2578]: E1103 00:40:20.610669 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-w46kd" podUID="8cd3ca22-e4ce-4896-ae31-97605af4e4c6" Nov 03 00:40:20.631152 ip-10-0-140-107 kubenswrapper[2578]: I1103 00:40:20.631107 2578 plugin_watcher.go:201] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock" Nov 03 00:40:20.737557 ip-10-0-140-107 kubenswrapper[2578]: I1103 00:40:20.737317 2578 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-82x8c" Nov 03 00:40:20.738570 ip-10-0-140-107 kubenswrapper[2578]: I1103 00:40:20.738391 2578 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-82x8c" Nov 03 00:40:20.818655 ip-10-0-140-107 kubenswrapper[2578]: I1103 00:40:20.818572 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-br8vt" event={"ID":"37bf96f5-045f-49e8-83e8-410df66bfb1f","Type":"ContainerStarted","Data":"588095045e9ec87288eb52800846c3fbf44ad4b35e7f4dd923a1ff840c7838d2"} Nov 03 00:40:21.475150 ip-10-0-140-107 kubenswrapper[2578]: I1103 00:40:21.474982 2578 reconciler.go:167] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-11-03T00:40:20.631127395Z","UUID":"f06e45a4-7e9d-47df-9d71-249b1d1d9d71","Handler":null,"Name":"","Endpoint":""} Nov 03 00:40:21.476936 ip-10-0-140-107 kubenswrapper[2578]: I1103 00:40:21.476898 2578 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0 Nov 03 00:40:21.477081 ip-10-0-140-107 kubenswrapper[2578]: I1103 00:40:21.476945 2578 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock Nov 03 00:40:21.605536 ip-10-0-140-107 kubenswrapper[2578]: I1103 00:40:21.605482 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-rt2hx" Nov 03 00:40:21.605728 ip-10-0-140-107 kubenswrapper[2578]: E1103 00:40:21.605603 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-rt2hx" podUID="136086c0-4846-437a-ad98-3cfef50610a0" Nov 03 00:40:21.823147 ip-10-0-140-107 kubenswrapper[2578]: I1103 00:40:21.822923 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-cjj2h" event={"ID":"ad5bf442-729d-411c-96e4-2e5105ebadcf","Type":"ContainerStarted","Data":"6093ad1bf9cc316b2432951d8731bc6c52dee2e766d36473b4fec4afb78923f9"} Nov 03 00:40:21.825324 ip-10-0-140-107 kubenswrapper[2578]: I1103 00:40:21.825295 2578 generic.go:356] "Generic (PLEG): container finished" podID="6650035b-2220-491c-9683-8a03ccb7af6a" containerID="1a0f2e8768244bbda03f5da9c38e1b80b1c6fd39b37c041ab81be2bd010691d2" exitCode=0 Nov 03 00:40:21.825439 ip-10-0-140-107 kubenswrapper[2578]: I1103 00:40:21.825387 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-gqppj" event={"ID":"6650035b-2220-491c-9683-8a03ccb7af6a","Type":"ContainerDied","Data":"1a0f2e8768244bbda03f5da9c38e1b80b1c6fd39b37c041ab81be2bd010691d2"} Nov 03 00:40:21.827557 ip-10-0-140-107 kubenswrapper[2578]: I1103 00:40:21.827530 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-tb9hd" event={"ID":"00dc2920-34f2-4e0a-892e-7759a16162d5","Type":"ContainerStarted","Data":"6ad7c5e70c6a540a28cd47dfe8555f0a81b49945f635731de68552f1541b78ca"} Nov 03 00:40:21.839442 ip-10-0-140-107 kubenswrapper[2578]: I1103 00:40:21.839232 2578 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-dns/node-resolver-cjj2h" podStartSLOduration=2.416367031 podStartE2EDuration="10.83921402s" totalImagesPullingTime="8.422846989s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-03 00:40:11 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:40:21.838392252 +0000 UTC m=+11.783971752" watchObservedRunningTime="2026-11-03 00:40:21.83921402 +0000 UTC m=+11.784793521" Nov 03 00:40:21.856733 ip-10-0-140-107 kubenswrapper[2578]: I1103 00:40:21.856685 2578 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-multus/multus-tb9hd" podStartSLOduration=2.938376991 podStartE2EDuration="10.856672754s" totalImagesPullingTime="7.918295763s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-03 00:40:11 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:40:21.856374885 +0000 UTC m=+11.801954384" watchObservedRunningTime="2026-11-03 00:40:21.856672754 +0000 UTC m=+11.802252279" Nov 03 00:40:22.605622 ip-10-0-140-107 kubenswrapper[2578]: I1103 00:40:22.605582 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-w46kd" Nov 03 00:40:22.605782 ip-10-0-140-107 kubenswrapper[2578]: E1103 00:40:22.605712 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-w46kd" podUID="8cd3ca22-e4ce-4896-ae31-97605af4e4c6" Nov 03 00:40:22.606164 ip-10-0-140-107 kubenswrapper[2578]: I1103 00:40:22.606141 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-jv7cw" Nov 03 00:40:22.606271 ip-10-0-140-107 kubenswrapper[2578]: E1103 00:40:22.606252 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-jv7cw" podUID="cabd402c-79c5-4699-9c85-d415d362b4d0" Nov 03 00:40:22.832045 ip-10-0-140-107 kubenswrapper[2578]: I1103 00:40:22.832008 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-br8vt" event={"ID":"37bf96f5-045f-49e8-83e8-410df66bfb1f","Type":"ContainerStarted","Data":"bc994ef308daccab7f6dac7d06d7c1603dba30dc06c86cb422f3da66a737f1ad"} Nov 03 00:40:22.834903 ip-10-0-140-107 kubenswrapper[2578]: I1103 00:40:22.834873 2578 generic.go:356] "Generic (PLEG): container finished" podID="6650035b-2220-491c-9683-8a03ccb7af6a" containerID="1cdfe4721f48faf8427b35d8affa3b2cb01cffb3bd2cce6fad597b03c758beae" exitCode=0 Nov 03 00:40:22.834903 ip-10-0-140-107 kubenswrapper[2578]: I1103 00:40:22.834960 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-gqppj" event={"ID":"6650035b-2220-491c-9683-8a03ccb7af6a","Type":"ContainerDied","Data":"1cdfe4721f48faf8427b35d8affa3b2cb01cffb3bd2cce6fad597b03c758beae"} Nov 03 00:40:22.837046 ip-10-0-140-107 kubenswrapper[2578]: I1103 00:40:22.837020 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-5spnk" event={"ID":"c736fd3f-973f-4999-8d4e-c405079b2888","Type":"ContainerStarted","Data":"b79e7b8762fab75dbeeb06c352523b03bf61ecbcb08750dc692d03aade805934"} Nov 03 00:40:22.851256 ip-10-0-140-107 kubenswrapper[2578]: I1103 00:40:22.851211 2578 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-br8vt" podStartSLOduration=3.202303553 podStartE2EDuration="11.851195504s" totalImagesPullingTime="8.648891951s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-03 00:40:11 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:40:22.851143078 +0000 UTC m=+12.796722576" watchObservedRunningTime="2026-11-03 00:40:22.851195504 +0000 UTC m=+12.796774999" Nov 03 00:40:22.865492 ip-10-0-140-107 kubenswrapper[2578]: I1103 00:40:22.865386 2578 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-5spnk" podStartSLOduration=4.677959305 podStartE2EDuration="12.865368509s" totalImagesPullingTime="8.187409204s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-03 00:40:10 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:40:22.864710184 +0000 UTC m=+12.810289684" watchObservedRunningTime="2026-11-03 00:40:22.865368509 +0000 UTC m=+12.810948009" Nov 03 00:40:23.379868 ip-10-0-140-107 kubenswrapper[2578]: I1103 00:40:23.379836 2578 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-82x8c" Nov 03 00:40:23.380050 ip-10-0-140-107 kubenswrapper[2578]: I1103 00:40:23.380000 2578 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness" Nov 03 00:40:23.380492 ip-10-0-140-107 kubenswrapper[2578]: I1103 00:40:23.380471 2578 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-82x8c" Nov 03 00:40:23.604578 ip-10-0-140-107 kubenswrapper[2578]: I1103 00:40:23.604542 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-rt2hx" Nov 03 00:40:23.604755 ip-10-0-140-107 kubenswrapper[2578]: E1103 00:40:23.604682 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-rt2hx" podUID="136086c0-4846-437a-ad98-3cfef50610a0" Nov 03 00:40:23.842153 ip-10-0-140-107 kubenswrapper[2578]: I1103 00:40:23.842123 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-gqppj" event={"ID":"6650035b-2220-491c-9683-8a03ccb7af6a","Type":"ContainerStarted","Data":"21751d6aa882b867fb24aa2aed5f52dc918892c4a80cbfe5d7a2227018b3f4ad"} Nov 03 00:40:23.844321 ip-10-0-140-107 kubenswrapper[2578]: I1103 00:40:23.844290 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-gtwxp" event={"ID":"b748c4f8-d021-4d37-b390-d395482fcddd","Type":"ContainerStarted","Data":"e958f8ce22cca2da0a5e0704e6a105b67f88b4e362f9de8bf84e2785daae434b"} Nov 03 00:40:23.881412 ip-10-0-140-107 kubenswrapper[2578]: I1103 00:40:23.881353 2578 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-gtwxp" podStartSLOduration=2.750363239 podStartE2EDuration="12.881334802s" totalImagesPullingTime="10.130971563s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-03 00:40:11 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:40:23.880719895 +0000 UTC m=+13.826299395" watchObservedRunningTime="2026-11-03 00:40:23.881334802 +0000 UTC m=+13.826914299" Nov 03 00:40:24.605557 ip-10-0-140-107 kubenswrapper[2578]: I1103 00:40:24.605524 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-w46kd" Nov 03 00:40:24.605738 ip-10-0-140-107 kubenswrapper[2578]: I1103 00:40:24.605571 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-jv7cw" Nov 03 00:40:24.605738 ip-10-0-140-107 kubenswrapper[2578]: E1103 00:40:24.605676 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-w46kd" podUID="8cd3ca22-e4ce-4896-ae31-97605af4e4c6" Nov 03 00:40:24.605840 ip-10-0-140-107 kubenswrapper[2578]: E1103 00:40:24.605798 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-jv7cw" podUID="cabd402c-79c5-4699-9c85-d415d362b4d0" Nov 03 00:40:24.849137 ip-10-0-140-107 kubenswrapper[2578]: I1103 00:40:24.849100 2578 generic.go:356] "Generic (PLEG): container finished" podID="6650035b-2220-491c-9683-8a03ccb7af6a" containerID="21751d6aa882b867fb24aa2aed5f52dc918892c4a80cbfe5d7a2227018b3f4ad" exitCode=0 Nov 03 00:40:24.849732 ip-10-0-140-107 kubenswrapper[2578]: I1103 00:40:24.849163 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-gqppj" event={"ID":"6650035b-2220-491c-9683-8a03ccb7af6a","Type":"ContainerDied","Data":"21751d6aa882b867fb24aa2aed5f52dc918892c4a80cbfe5d7a2227018b3f4ad"} Nov 03 00:40:25.605034 ip-10-0-140-107 kubenswrapper[2578]: I1103 00:40:25.604998 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-rt2hx" Nov 03 00:40:25.605203 ip-10-0-140-107 kubenswrapper[2578]: E1103 00:40:25.605148 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-rt2hx" podUID="136086c0-4846-437a-ad98-3cfef50610a0" Nov 03 00:40:26.605832 ip-10-0-140-107 kubenswrapper[2578]: I1103 00:40:26.605160 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-w46kd" Nov 03 00:40:26.605832 ip-10-0-140-107 kubenswrapper[2578]: E1103 00:40:26.605455 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-w46kd" podUID="8cd3ca22-e4ce-4896-ae31-97605af4e4c6" Nov 03 00:40:26.605832 ip-10-0-140-107 kubenswrapper[2578]: I1103 00:40:26.605340 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-jv7cw" Nov 03 00:40:26.605832 ip-10-0-140-107 kubenswrapper[2578]: E1103 00:40:26.605543 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-jv7cw" podUID="cabd402c-79c5-4699-9c85-d415d362b4d0" Nov 03 00:40:27.605139 ip-10-0-140-107 kubenswrapper[2578]: I1103 00:40:27.605113 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-rt2hx" Nov 03 00:40:27.605319 ip-10-0-140-107 kubenswrapper[2578]: E1103 00:40:27.605212 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-rt2hx" podUID="136086c0-4846-437a-ad98-3cfef50610a0" Nov 03 00:40:27.858792 ip-10-0-140-107 kubenswrapper[2578]: I1103 00:40:27.858713 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-gqppj" event={"ID":"6650035b-2220-491c-9683-8a03ccb7af6a","Type":"ContainerStarted","Data":"7c9c0e3ad6073ffe6b8d8d2a7d5bb1bb86e1732b9f514a7d6db018f593926e78"} Nov 03 00:40:28.170147 ip-10-0-140-107 kubenswrapper[2578]: I1103 00:40:28.169953 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/136086c0-4846-437a-ad98-3cfef50610a0-metrics-certs\") pod \"network-metrics-daemon-rt2hx\" (UID: \"136086c0-4846-437a-ad98-3cfef50610a0\") " pod="openshift-multus/network-metrics-daemon-rt2hx" Nov 03 00:40:28.170242 ip-10-0-140-107 kubenswrapper[2578]: E1103 00:40:28.170095 2578 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Nov 03 00:40:28.170282 ip-10-0-140-107 kubenswrapper[2578]: E1103 00:40:28.170243 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/136086c0-4846-437a-ad98-3cfef50610a0-metrics-certs podName:136086c0-4846-437a-ad98-3cfef50610a0 nodeName:}" failed. No retries permitted until 2026-11-03 00:40:44.170226728 +0000 UTC m=+34.115806222 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/136086c0-4846-437a-ad98-3cfef50610a0-metrics-certs") pod "network-metrics-daemon-rt2hx" (UID: "136086c0-4846-437a-ad98-3cfef50610a0") : object "openshift-multus"/"metrics-daemon-secret" not registered Nov 03 00:40:28.270753 ip-10-0-140-107 kubenswrapper[2578]: I1103 00:40:28.270714 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d8l25\" (UniqueName: \"kubernetes.io/projected/8cd3ca22-e4ce-4896-ae31-97605af4e4c6-kube-api-access-d8l25\") pod \"network-check-target-w46kd\" (UID: \"8cd3ca22-e4ce-4896-ae31-97605af4e4c6\") " pod="openshift-network-diagnostics/network-check-target-w46kd" Nov 03 00:40:28.270880 ip-10-0-140-107 kubenswrapper[2578]: I1103 00:40:28.270819 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/cabd402c-79c5-4699-9c85-d415d362b4d0-original-pull-secret\") pod \"global-pull-secret-syncer-jv7cw\" (UID: \"cabd402c-79c5-4699-9c85-d415d362b4d0\") " pod="kube-system/global-pull-secret-syncer-jv7cw" Nov 03 00:40:28.270880 ip-10-0-140-107 kubenswrapper[2578]: E1103 00:40:28.270864 2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Nov 03 00:40:28.270978 ip-10-0-140-107 kubenswrapper[2578]: E1103 00:40:28.270887 2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Nov 03 00:40:28.270978 ip-10-0-140-107 kubenswrapper[2578]: E1103 00:40:28.270897 2578 projected.go:196] Error preparing data for projected volume kube-api-access-d8l25 for pod openshift-network-diagnostics/network-check-target-w46kd: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 03 00:40:28.270978 ip-10-0-140-107 kubenswrapper[2578]: E1103 00:40:28.270966 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/8cd3ca22-e4ce-4896-ae31-97605af4e4c6-kube-api-access-d8l25 podName:8cd3ca22-e4ce-4896-ae31-97605af4e4c6 nodeName:}" failed. No retries permitted until 2026-11-03 00:40:44.270951714 +0000 UTC m=+34.216531199 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-d8l25" (UniqueName: "kubernetes.io/projected/8cd3ca22-e4ce-4896-ae31-97605af4e4c6-kube-api-access-d8l25") pod "network-check-target-w46kd" (UID: "8cd3ca22-e4ce-4896-ae31-97605af4e4c6") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 03 00:40:28.270978 ip-10-0-140-107 kubenswrapper[2578]: E1103 00:40:28.270966 2578 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Nov 03 00:40:28.271107 ip-10-0-140-107 kubenswrapper[2578]: E1103 00:40:28.271020 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cabd402c-79c5-4699-9c85-d415d362b4d0-original-pull-secret podName:cabd402c-79c5-4699-9c85-d415d362b4d0 nodeName:}" failed. No retries permitted until 2026-11-03 00:40:44.271004806 +0000 UTC m=+34.216584308 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/cabd402c-79c5-4699-9c85-d415d362b4d0-original-pull-secret") pod "global-pull-secret-syncer-jv7cw" (UID: "cabd402c-79c5-4699-9c85-d415d362b4d0") : object "kube-system"/"original-pull-secret" not registered Nov 03 00:40:28.605265 ip-10-0-140-107 kubenswrapper[2578]: I1103 00:40:28.605223 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-jv7cw" Nov 03 00:40:28.605265 ip-10-0-140-107 kubenswrapper[2578]: I1103 00:40:28.605263 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-w46kd" Nov 03 00:40:28.605474 ip-10-0-140-107 kubenswrapper[2578]: E1103 00:40:28.605376 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-jv7cw" podUID="cabd402c-79c5-4699-9c85-d415d362b4d0" Nov 03 00:40:28.605474 ip-10-0-140-107 kubenswrapper[2578]: E1103 00:40:28.605454 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-w46kd" podUID="8cd3ca22-e4ce-4896-ae31-97605af4e4c6" Nov 03 00:40:28.863545 ip-10-0-140-107 kubenswrapper[2578]: I1103 00:40:28.863463 2578 generic.go:356] "Generic (PLEG): container finished" podID="6650035b-2220-491c-9683-8a03ccb7af6a" containerID="7c9c0e3ad6073ffe6b8d8d2a7d5bb1bb86e1732b9f514a7d6db018f593926e78" exitCode=0 Nov 03 00:40:28.864001 ip-10-0-140-107 kubenswrapper[2578]: I1103 00:40:28.863559 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-gqppj" event={"ID":"6650035b-2220-491c-9683-8a03ccb7af6a","Type":"ContainerDied","Data":"7c9c0e3ad6073ffe6b8d8d2a7d5bb1bb86e1732b9f514a7d6db018f593926e78"} Nov 03 00:40:28.866626 ip-10-0-140-107 kubenswrapper[2578]: I1103 00:40:28.866605 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-psqpn" event={"ID":"98ca3972-fd0c-47f5-9d26-569a58e18bd1","Type":"ContainerStarted","Data":"a7a78b727ace8748f934b51e915759d575a5ec27906a51b36d464598e79519b3"} Nov 03 00:40:28.866695 ip-10-0-140-107 kubenswrapper[2578]: I1103 00:40:28.866682 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-psqpn" event={"ID":"98ca3972-fd0c-47f5-9d26-569a58e18bd1","Type":"ContainerStarted","Data":"aba30af3985637cd3cd65ae15d510e9e74c202e8aef3b4be3019298cbd86e8a0"} Nov 03 00:40:28.866765 ip-10-0-140-107 kubenswrapper[2578]: I1103 00:40:28.866735 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-psqpn" event={"ID":"98ca3972-fd0c-47f5-9d26-569a58e18bd1","Type":"ContainerStarted","Data":"eb213ff68b71d8050a06d7c462064fda4da0ccda860d70bc293f1b40eb54f7a5"} Nov 03 00:40:28.866816 ip-10-0-140-107 kubenswrapper[2578]: I1103 00:40:28.866784 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-psqpn" event={"ID":"98ca3972-fd0c-47f5-9d26-569a58e18bd1","Type":"ContainerStarted","Data":"f486507bd210e8dbfbfffde60bd768434ea940170df66f21dc9c316c2c1ef9b0"} Nov 03 00:40:28.866869 ip-10-0-140-107 kubenswrapper[2578]: I1103 00:40:28.866833 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-psqpn" event={"ID":"98ca3972-fd0c-47f5-9d26-569a58e18bd1","Type":"ContainerStarted","Data":"a1f435654ba8315a0f76a59ef98f914e97b352a04ea4fcb6039363f0ab9d9688"} Nov 03 00:40:28.866936 ip-10-0-140-107 kubenswrapper[2578]: I1103 00:40:28.866879 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-psqpn" event={"ID":"98ca3972-fd0c-47f5-9d26-569a58e18bd1","Type":"ContainerStarted","Data":"ae3a3173bbf53274bfdd7fd717538b63f77c56dbc34d69a6c425dd925d13edd8"} Nov 03 00:40:29.604700 ip-10-0-140-107 kubenswrapper[2578]: I1103 00:40:29.604664 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-rt2hx" Nov 03 00:40:29.604866 ip-10-0-140-107 kubenswrapper[2578]: E1103 00:40:29.604793 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-rt2hx" podUID="136086c0-4846-437a-ad98-3cfef50610a0" Nov 03 00:40:29.871540 ip-10-0-140-107 kubenswrapper[2578]: I1103 00:40:29.871458 2578 generic.go:356] "Generic (PLEG): container finished" podID="6650035b-2220-491c-9683-8a03ccb7af6a" containerID="b13bf6e4a5929d974571b6a56d8155a56640694d56ac85a500cb6d953cf72f6f" exitCode=0 Nov 03 00:40:29.871540 ip-10-0-140-107 kubenswrapper[2578]: I1103 00:40:29.871506 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-gqppj" event={"ID":"6650035b-2220-491c-9683-8a03ccb7af6a","Type":"ContainerDied","Data":"b13bf6e4a5929d974571b6a56d8155a56640694d56ac85a500cb6d953cf72f6f"} Nov 03 00:40:30.605692 ip-10-0-140-107 kubenswrapper[2578]: I1103 00:40:30.605655 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-jv7cw" Nov 03 00:40:30.605837 ip-10-0-140-107 kubenswrapper[2578]: E1103 00:40:30.605749 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-jv7cw" podUID="cabd402c-79c5-4699-9c85-d415d362b4d0" Nov 03 00:40:30.605837 ip-10-0-140-107 kubenswrapper[2578]: I1103 00:40:30.605797 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-w46kd" Nov 03 00:40:30.605921 ip-10-0-140-107 kubenswrapper[2578]: E1103 00:40:30.605889 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-w46kd" podUID="8cd3ca22-e4ce-4896-ae31-97605af4e4c6" Nov 03 00:40:30.876563 ip-10-0-140-107 kubenswrapper[2578]: I1103 00:40:30.876477 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-gqppj" event={"ID":"6650035b-2220-491c-9683-8a03ccb7af6a","Type":"ContainerStarted","Data":"1997ee4e8e4a4ca95f212b0797e8d34a504ff9276eb92accb17a9a4a1d1fb169"} Nov 03 00:40:30.879507 ip-10-0-140-107 kubenswrapper[2578]: I1103 00:40:30.879482 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-psqpn" event={"ID":"98ca3972-fd0c-47f5-9d26-569a58e18bd1","Type":"ContainerStarted","Data":"968fb8d2550370992a376017aeccd3d81c53e9af0b582337ee8d0e3a09aa31a8"} Nov 03 00:40:30.899212 ip-10-0-140-107 kubenswrapper[2578]: I1103 00:40:30.899166 2578 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-gqppj" podStartSLOduration=8.296364091 podStartE2EDuration="19.899153312s" totalImagesPullingTime="11.602789221s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-03 00:40:11 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:40:30.899098175 +0000 UTC m=+20.844677674" watchObservedRunningTime="2026-11-03 00:40:30.899153312 +0000 UTC m=+20.844732810" Nov 03 00:40:31.604984 ip-10-0-140-107 kubenswrapper[2578]: I1103 00:40:31.604954 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-rt2hx" Nov 03 00:40:31.605120 ip-10-0-140-107 kubenswrapper[2578]: E1103 00:40:31.605055 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-rt2hx" podUID="136086c0-4846-437a-ad98-3cfef50610a0" Nov 03 00:40:32.605138 ip-10-0-140-107 kubenswrapper[2578]: I1103 00:40:32.604964 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-w46kd" Nov 03 00:40:32.605974 ip-10-0-140-107 kubenswrapper[2578]: I1103 00:40:32.604977 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-jv7cw" Nov 03 00:40:32.605974 ip-10-0-140-107 kubenswrapper[2578]: E1103 00:40:32.605227 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-w46kd" podUID="8cd3ca22-e4ce-4896-ae31-97605af4e4c6" Nov 03 00:40:32.605974 ip-10-0-140-107 kubenswrapper[2578]: E1103 00:40:32.605271 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-jv7cw" podUID="cabd402c-79c5-4699-9c85-d415d362b4d0" Nov 03 00:40:32.887083 ip-10-0-140-107 kubenswrapper[2578]: I1103 00:40:32.887049 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-psqpn" event={"ID":"98ca3972-fd0c-47f5-9d26-569a58e18bd1","Type":"ContainerStarted","Data":"3b72114f2e3c3f0c8b0d0baad80c80f05b834ab7f76ba5424989d77440c2465c"} Nov 03 00:40:32.887312 ip-10-0-140-107 kubenswrapper[2578]: I1103 00:40:32.887284 2578 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-psqpn" Nov 03 00:40:32.887438 ip-10-0-140-107 kubenswrapper[2578]: I1103 00:40:32.887349 2578 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-psqpn" Nov 03 00:40:32.902716 ip-10-0-140-107 kubenswrapper[2578]: I1103 00:40:32.902692 2578 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-psqpn" Nov 03 00:40:32.902840 ip-10-0-140-107 kubenswrapper[2578]: I1103 00:40:32.902783 2578 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-psqpn" Nov 03 00:40:32.916386 ip-10-0-140-107 kubenswrapper[2578]: I1103 00:40:32.916350 2578 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-psqpn" podStartSLOduration=6.783829666 podStartE2EDuration="21.91633852s" totalImagesPullingTime="15.132508854s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-03 00:40:11 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:40:32.916084369 +0000 UTC m=+22.861663868" watchObservedRunningTime="2026-11-03 00:40:32.91633852 +0000 UTC m=+22.861918020" Nov 03 00:40:33.608032 ip-10-0-140-107 kubenswrapper[2578]: I1103 00:40:33.607993 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-rt2hx" Nov 03 00:40:33.608502 ip-10-0-140-107 kubenswrapper[2578]: E1103 00:40:33.608144 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-rt2hx" podUID="136086c0-4846-437a-ad98-3cfef50610a0" Nov 03 00:40:33.890227 ip-10-0-140-107 kubenswrapper[2578]: I1103 00:40:33.890161 2578 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness" Nov 03 00:40:34.038985 ip-10-0-140-107 kubenswrapper[2578]: I1103 00:40:34.038954 2578 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-w46kd"] Nov 03 00:40:34.039135 ip-10-0-140-107 kubenswrapper[2578]: I1103 00:40:34.039056 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-w46kd" Nov 03 00:40:34.039189 ip-10-0-140-107 kubenswrapper[2578]: E1103 00:40:34.039135 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-w46kd" podUID="8cd3ca22-e4ce-4896-ae31-97605af4e4c6" Nov 03 00:40:34.042132 ip-10-0-140-107 kubenswrapper[2578]: I1103 00:40:34.042099 2578 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-rt2hx"] Nov 03 00:40:34.042266 ip-10-0-140-107 kubenswrapper[2578]: I1103 00:40:34.042215 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-rt2hx" Nov 03 00:40:34.042350 ip-10-0-140-107 kubenswrapper[2578]: E1103 00:40:34.042329 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-rt2hx" podUID="136086c0-4846-437a-ad98-3cfef50610a0" Nov 03 00:40:34.042735 ip-10-0-140-107 kubenswrapper[2578]: I1103 00:40:34.042716 2578 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-jv7cw"] Nov 03 00:40:34.042801 ip-10-0-140-107 kubenswrapper[2578]: I1103 00:40:34.042792 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-jv7cw" Nov 03 00:40:34.042885 ip-10-0-140-107 kubenswrapper[2578]: E1103 00:40:34.042869 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-jv7cw" podUID="cabd402c-79c5-4699-9c85-d415d362b4d0" Nov 03 00:40:34.892027 ip-10-0-140-107 kubenswrapper[2578]: I1103 00:40:34.891999 2578 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness" Nov 03 00:40:35.605344 ip-10-0-140-107 kubenswrapper[2578]: I1103 00:40:35.605310 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-w46kd" Nov 03 00:40:35.605344 ip-10-0-140-107 kubenswrapper[2578]: I1103 00:40:35.605338 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-jv7cw" Nov 03 00:40:35.605574 ip-10-0-140-107 kubenswrapper[2578]: E1103 00:40:35.605409 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-w46kd" podUID="8cd3ca22-e4ce-4896-ae31-97605af4e4c6" Nov 03 00:40:35.605574 ip-10-0-140-107 kubenswrapper[2578]: I1103 00:40:35.605486 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-rt2hx" Nov 03 00:40:35.605678 ip-10-0-140-107 kubenswrapper[2578]: E1103 00:40:35.605601 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-rt2hx" podUID="136086c0-4846-437a-ad98-3cfef50610a0" Nov 03 00:40:35.605732 ip-10-0-140-107 kubenswrapper[2578]: E1103 00:40:35.605696 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-jv7cw" podUID="cabd402c-79c5-4699-9c85-d415d362b4d0" Nov 03 00:40:37.604981 ip-10-0-140-107 kubenswrapper[2578]: I1103 00:40:37.604950 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-w46kd" Nov 03 00:40:37.605422 ip-10-0-140-107 kubenswrapper[2578]: I1103 00:40:37.604949 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-jv7cw" Nov 03 00:40:37.605422 ip-10-0-140-107 kubenswrapper[2578]: E1103 00:40:37.605066 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-w46kd" podUID="8cd3ca22-e4ce-4896-ae31-97605af4e4c6" Nov 03 00:40:37.605422 ip-10-0-140-107 kubenswrapper[2578]: I1103 00:40:37.604950 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-rt2hx" Nov 03 00:40:37.605422 ip-10-0-140-107 kubenswrapper[2578]: E1103 00:40:37.605137 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-jv7cw" podUID="cabd402c-79c5-4699-9c85-d415d362b4d0" Nov 03 00:40:37.605422 ip-10-0-140-107 kubenswrapper[2578]: E1103 00:40:37.605205 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-rt2hx" podUID="136086c0-4846-437a-ad98-3cfef50610a0" Nov 03 00:40:39.604743 ip-10-0-140-107 kubenswrapper[2578]: I1103 00:40:39.604711 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-rt2hx" Nov 03 00:40:39.604743 ip-10-0-140-107 kubenswrapper[2578]: I1103 00:40:39.604732 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-w46kd" Nov 03 00:40:39.605255 ip-10-0-140-107 kubenswrapper[2578]: I1103 00:40:39.604749 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-jv7cw" Nov 03 00:40:39.605255 ip-10-0-140-107 kubenswrapper[2578]: E1103 00:40:39.604823 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-rt2hx" podUID="136086c0-4846-437a-ad98-3cfef50610a0" Nov 03 00:40:39.605255 ip-10-0-140-107 kubenswrapper[2578]: E1103 00:40:39.604981 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-w46kd" podUID="8cd3ca22-e4ce-4896-ae31-97605af4e4c6" Nov 03 00:40:39.605255 ip-10-0-140-107 kubenswrapper[2578]: E1103 00:40:39.605083 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-jv7cw" podUID="cabd402c-79c5-4699-9c85-d415d362b4d0" Nov 03 00:40:40.344385 ip-10-0-140-107 kubenswrapper[2578]: I1103 00:40:40.344308 2578 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-psqpn" Nov 03 00:40:40.344560 ip-10-0-140-107 kubenswrapper[2578]: I1103 00:40:40.344535 2578 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness" Nov 03 00:40:40.355009 ip-10-0-140-107 kubenswrapper[2578]: I1103 00:40:40.354836 2578 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-psqpn" Nov 03 00:40:40.391735 ip-10-0-140-107 kubenswrapper[2578]: I1103 00:40:40.391710 2578 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-140-107.us-east-2.compute.internal" event="NodeReady" Nov 03 00:40:40.391886 ip-10-0-140-107 kubenswrapper[2578]: I1103 00:40:40.391846 2578 kubelet_node_status.go:477] "Fast updating node status as it just became ready" Nov 03 00:40:40.430899 ip-10-0-140-107 kubenswrapper[2578]: I1103 00:40:40.430869 2578 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-zsd49"] Nov 03 00:40:40.432558 ip-10-0-140-107 kubenswrapper[2578]: I1103 00:40:40.432537 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kas-connection-checker-685895c7c9-zsd49" Nov 03 00:40:40.433969 ip-10-0-140-107 kubenswrapper[2578]: I1103 00:40:40.433946 2578 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-xst6j"] Nov 03 00:40:40.435622 ip-10-0-140-107 kubenswrapper[2578]: I1103 00:40:40.435609 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kas-connection-checker-685895c7c9-xst6j" Nov 03 00:40:40.442260 ip-10-0-140-107 kubenswrapper[2578]: I1103 00:40:40.442198 2578 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-ntqw6"] Nov 03 00:40:40.443831 ip-10-0-140-107 kubenswrapper[2578]: I1103 00:40:40.443812 2578 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/router-default-d7798ccf5-442kt"] Nov 03 00:40:40.444501 ip-10-0-140-107 kubenswrapper[2578]: I1103 00:40:40.444479 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kas-connection-checker-685895c7c9-ntqw6" Nov 03 00:40:40.445933 ip-10-0-140-107 kubenswrapper[2578]: I1103 00:40:40.445890 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"kube-root-ca.crt\"" Nov 03 00:40:40.446056 ip-10-0-140-107 kubenswrapper[2578]: I1103 00:40:40.445941 2578 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-65f77c855d-6tnc2"] Nov 03 00:40:40.446056 ip-10-0-140-107 kubenswrapper[2578]: I1103 00:40:40.446032 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"openshift-service-ca.crt\"" Nov 03 00:40:40.446172 ip-10-0-140-107 kubenswrapper[2578]: I1103 00:40:40.446110 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/router-default-d7798ccf5-442kt" Nov 03 00:40:40.447623 ip-10-0-140-107 kubenswrapper[2578]: I1103 00:40:40.447464 2578 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-2ngwx"] Nov 03 00:40:40.447623 ip-10-0-140-107 kubenswrapper[2578]: I1103 00:40:40.447567 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-65f77c855d-6tnc2" Nov 03 00:40:40.449062 ip-10-0-140-107 kubenswrapper[2578]: I1103 00:40:40.449035 2578 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-khmv7"] Nov 03 00:40:40.449159 ip-10-0-140-107 kubenswrapper[2578]: I1103 00:40:40.449106 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-2ngwx" Nov 03 00:40:40.451369 ip-10-0-140-107 kubenswrapper[2578]: I1103 00:40:40.450896 2578 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-884576c57-4bm4f"] Nov 03 00:40:40.451505 ip-10-0-140-107 kubenswrapper[2578]: I1103 00:40:40.451371 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-khmv7" Nov 03 00:40:40.452523 ip-10-0-140-107 kubenswrapper[2578]: I1103 00:40:40.452507 2578 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-console-operator/console-operator-7cd56b8dbc-n42s6"] Nov 03 00:40:40.452685 ip-10-0-140-107 kubenswrapper[2578]: I1103 00:40:40.452588 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/cluster-monitoring-operator-884576c57-4bm4f" Nov 03 00:40:40.456336 ip-10-0-140-107 kubenswrapper[2578]: I1103 00:40:40.456311 2578 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-operator-7c4986d449-xprcf"] Nov 03 00:40:40.459747 ip-10-0-140-107 kubenswrapper[2578]: I1103 00:40:40.459043 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console-operator/console-operator-7cd56b8dbc-n42s6" Nov 03 00:40:40.460258 ip-10-0-140-107 kubenswrapper[2578]: I1103 00:40:40.460236 2578 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-6669dc8887-ws544"] Nov 03 00:40:40.460366 ip-10-0-140-107 kubenswrapper[2578]: I1103 00:40:40.460351 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-operator-7c4986d449-xprcf" Nov 03 00:40:40.462011 ip-10-0-140-107 kubenswrapper[2578]: I1103 00:40:40.461994 2578 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-network-console/networking-console-plugin-86c478bb88-z2qgj"] Nov 03 00:40:40.462142 ip-10-0-140-107 kubenswrapper[2578]: I1103 00:40:40.462123 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-6669dc8887-ws544" Nov 03 00:40:40.463680 ip-10-0-140-107 kubenswrapper[2578]: I1103 00:40:40.463665 2578 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-w4xtw"] Nov 03 00:40:40.463789 ip-10-0-140-107 kubenswrapper[2578]: I1103 00:40:40.463775 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-86c478bb88-z2qgj" Nov 03 00:40:40.465281 ip-10-0-140-107 kubenswrapper[2578]: I1103 00:40:40.465262 2578 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-network-diagnostics/network-check-source-6f78d59f99-z5f9s"] Nov 03 00:40:40.465358 ip-10-0-140-107 kubenswrapper[2578]: I1103 00:40:40.465344 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-w4xtw" Nov 03 00:40:40.466990 ip-10-0-140-107 kubenswrapper[2578]: I1103 00:40:40.466963 2578 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-service-ca-operator/service-ca-operator-544d749c8c-7lqbv"] Nov 03 00:40:40.469007 ip-10-0-140-107 kubenswrapper[2578]: I1103 00:40:40.468986 2578 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-xst6j"] Nov 03 00:40:40.469105 ip-10-0-140-107 kubenswrapper[2578]: I1103 00:40:40.469030 2578 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-zsd49"] Nov 03 00:40:40.469105 ip-10-0-140-107 kubenswrapper[2578]: I1103 00:40:40.469041 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-source-6f78d59f99-z5f9s" Nov 03 00:40:40.469213 ip-10-0-140-107 kubenswrapper[2578]: I1103 00:40:40.469180 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-7lqbv" Nov 03 00:40:40.469581 ip-10-0-140-107 kubenswrapper[2578]: I1103 00:40:40.469565 2578 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-ntqw6"] Nov 03 00:40:40.470187 ip-10-0-140-107 kubenswrapper[2578]: I1103 00:40:40.470171 2578 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-khmv7"] Nov 03 00:40:40.476546 ip-10-0-140-107 kubenswrapper[2578]: I1103 00:40:40.476527 2578 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-266ht"] Nov 03 00:40:40.478391 ip-10-0-140-107 kubenswrapper[2578]: I1103 00:40:40.478374 2578 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-s9hbz"] Nov 03 00:40:40.478565 ip-10-0-140-107 kubenswrapper[2578]: I1103 00:40:40.478546 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-266ht" Nov 03 00:40:40.479074 ip-10-0-140-107 kubenswrapper[2578]: I1103 00:40:40.479056 2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-metrics-certs-default\"" Nov 03 00:40:40.479260 ip-10-0-140-107 kubenswrapper[2578]: I1103 00:40:40.479241 2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"default-ingress-cert\"" Nov 03 00:40:40.479354 ip-10-0-140-107 kubenswrapper[2578]: I1103 00:40:40.479138 2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"serving-cert\"" Nov 03 00:40:40.479773 ip-10-0-140-107 kubenswrapper[2578]: I1103 00:40:40.479532 2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"openshift-insights-serving-cert\"" Nov 03 00:40:40.479773 ip-10-0-140-107 kubenswrapper[2578]: I1103 00:40:40.479643 2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-tls\"" Nov 03 00:40:40.479773 ip-10-0-140-107 kubenswrapper[2578]: I1103 00:40:40.479744 2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"networking-console-plugin-cert\"" Nov 03 00:40:40.480386 ip-10-0-140-107 kubenswrapper[2578]: I1103 00:40:40.480369 2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-stats-default\"" Nov 03 00:40:40.480575 ip-10-0-140-107 kubenswrapper[2578]: I1103 00:40:40.480508 2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console-operator\"/\"serving-cert\"" Nov 03 00:40:40.480714 ip-10-0-140-107 kubenswrapper[2578]: I1103 00:40:40.480696 2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\"" Nov 03 00:40:40.480790 ip-10-0-140-107 kubenswrapper[2578]: I1103 00:40:40.480769 2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"kube-storage-version-migrator-operator-dockercfg-clzxq\"" Nov 03 00:40:40.480849 ip-10-0-140-107 kubenswrapper[2578]: I1103 00:40:40.480792 2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\"" Nov 03 00:40:40.480957 ip-10-0-140-107 kubenswrapper[2578]: I1103 00:40:40.480940 2578 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-2ngwx"] Nov 03 00:40:40.481029 ip-10-0-140-107 kubenswrapper[2578]: I1103 00:40:40.480969 2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"samples-operator-tls\"" Nov 03 00:40:40.481029 ip-10-0-140-107 kubenswrapper[2578]: I1103 00:40:40.480995 2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-grbz8\"" Nov 03 00:40:40.481124 ip-10-0-140-107 kubenswrapper[2578]: I1103 00:40:40.481041 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-s9hbz" Nov 03 00:40:40.481124 ip-10-0-140-107 kubenswrapper[2578]: I1103 00:40:40.481063 2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\"" Nov 03 00:40:40.481124 ip-10-0-140-107 kubenswrapper[2578]: I1103 00:40:40.481072 2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-storage-operator\"/\"volume-data-source-validator-dockercfg-6hxf5\"" Nov 03 00:40:40.481265 ip-10-0-140-107 kubenswrapper[2578]: I1103 00:40:40.481194 2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"networking-console-plugin-dockercfg-sfsbw\"" Nov 03 00:40:40.482018 ip-10-0-140-107 kubenswrapper[2578]: I1103 00:40:40.481953 2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"cluster-samples-operator-dockercfg-qzkgg\"" Nov 03 00:40:40.482018 ip-10-0-140-107 kubenswrapper[2578]: I1103 00:40:40.481985 2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console-operator\"/\"console-operator-dockercfg-cl7zw\"" Nov 03 00:40:40.482165 ip-10-0-140-107 kubenswrapper[2578]: I1103 00:40:40.482018 2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"operator-dockercfg-tg82s\"" Nov 03 00:40:40.482165 ip-10-0-140-107 kubenswrapper[2578]: I1103 00:40:40.482057 2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-dockercfg-rc6ml\"" Nov 03 00:40:40.482349 ip-10-0-140-107 kubenswrapper[2578]: I1103 00:40:40.482321 2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-dockercfg-sfmcg\"" Nov 03 00:40:40.488379 ip-10-0-140-107 kubenswrapper[2578]: I1103 00:40:40.488355 2578 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/router-default-d7798ccf5-442kt"] Nov 03 00:40:40.502365 ip-10-0-140-107 kubenswrapper[2578]: I1103 00:40:40.502345 2578 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-console/networking-console-plugin-86c478bb88-z2qgj"] Nov 03 00:40:40.523834 ip-10-0-140-107 kubenswrapper[2578]: I1103 00:40:40.523788 2578 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console-operator/console-operator-7cd56b8dbc-n42s6"] Nov 03 00:40:40.524031 ip-10-0-140-107 kubenswrapper[2578]: I1103 00:40:40.523870 2578 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-source-6f78d59f99-z5f9s"] Nov 03 00:40:40.531742 ip-10-0-140-107 kubenswrapper[2578]: I1103 00:40:40.531711 2578 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-6669dc8887-ws544"] Nov 03 00:40:40.535361 ip-10-0-140-107 kubenswrapper[2578]: I1103 00:40:40.535335 2578 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca-operator/service-ca-operator-544d749c8c-7lqbv"] Nov 03 00:40:40.540187 ip-10-0-140-107 kubenswrapper[2578]: I1103 00:40:40.539996 2578 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-65f77c855d-6tnc2"] Nov 03 00:40:40.541305 ip-10-0-140-107 kubenswrapper[2578]: I1103 00:40:40.541031 2578 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-s9hbz"] Nov 03 00:40:40.542123 ip-10-0-140-107 kubenswrapper[2578]: I1103 00:40:40.542101 2578 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-w4xtw"] Nov 03 00:40:40.544195 ip-10-0-140-107 kubenswrapper[2578]: I1103 00:40:40.544170 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"kube-root-ca.crt\"" Nov 03 00:40:40.544298 ip-10-0-140-107 kubenswrapper[2578]: I1103 00:40:40.544263 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\"" Nov 03 00:40:40.544361 ip-10-0-140-107 kubenswrapper[2578]: I1103 00:40:40.544325 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\"" Nov 03 00:40:40.544416 ip-10-0-140-107 kubenswrapper[2578]: I1103 00:40:40.544392 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"console-operator-config\"" Nov 03 00:40:40.544573 ip-10-0-140-107 kubenswrapper[2578]: I1103 00:40:40.544555 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\"" Nov 03 00:40:40.544663 ip-10-0-140-107 kubenswrapper[2578]: I1103 00:40:40.544649 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"service-ca-bundle\"" Nov 03 00:40:40.546077 ip-10-0-140-107 kubenswrapper[2578]: I1103 00:40:40.546057 2578 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-266ht"] Nov 03 00:40:40.546234 ip-10-0-140-107 kubenswrapper[2578]: I1103 00:40:40.546213 2578 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-884576c57-4bm4f"] Nov 03 00:40:40.546336 ip-10-0-140-107 kubenswrapper[2578]: I1103 00:40:40.546255 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\"" Nov 03 00:40:40.546395 ip-10-0-140-107 kubenswrapper[2578]: I1103 00:40:40.546355 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"openshift-service-ca.crt\"" Nov 03 00:40:40.546395 ip-10-0-140-107 kubenswrapper[2578]: I1103 00:40:40.546373 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-console\"/\"networking-console-plugin\"" Nov 03 00:40:40.546502 ip-10-0-140-107 kubenswrapper[2578]: I1103 00:40:40.546426 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"service-ca-bundle\"" Nov 03 00:40:40.546552 ip-10-0-140-107 kubenswrapper[2578]: I1103 00:40:40.546534 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"kube-root-ca.crt\"" Nov 03 00:40:40.546552 ip-10-0-140-107 kubenswrapper[2578]: I1103 00:40:40.546537 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\"" Nov 03 00:40:40.546655 ip-10-0-140-107 kubenswrapper[2578]: I1103 00:40:40.546630 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"config\"" Nov 03 00:40:40.546705 ip-10-0-140-107 kubenswrapper[2578]: I1103 00:40:40.546678 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"samples-operator-config\"" Nov 03 00:40:40.546705 ip-10-0-140-107 kubenswrapper[2578]: I1103 00:40:40.546695 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"kube-root-ca.crt\"" Nov 03 00:40:40.546805 ip-10-0-140-107 kubenswrapper[2578]: I1103 00:40:40.546730 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"service-ca-operator-config\"" Nov 03 00:40:40.546805 ip-10-0-140-107 kubenswrapper[2578]: I1103 00:40:40.546742 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"openshift-service-ca.crt\"" Nov 03 00:40:40.546805 ip-10-0-140-107 kubenswrapper[2578]: I1103 00:40:40.546746 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"openshift-service-ca.crt\"" Nov 03 00:40:40.546805 ip-10-0-140-107 kubenswrapper[2578]: I1103 00:40:40.546750 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"openshift-service-ca.crt\"" Nov 03 00:40:40.546805 ip-10-0-140-107 kubenswrapper[2578]: I1103 00:40:40.546734 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\"" Nov 03 00:40:40.546805 ip-10-0-140-107 kubenswrapper[2578]: I1103 00:40:40.546786 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Nov 03 00:40:40.546805 ip-10-0-140-107 kubenswrapper[2578]: I1103 00:40:40.546787 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"kube-root-ca.crt\"" Nov 03 00:40:40.546805 ip-10-0-140-107 kubenswrapper[2578]: I1103 00:40:40.546806 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"openshift-service-ca.crt\"" Nov 03 00:40:40.547140 ip-10-0-140-107 kubenswrapper[2578]: I1103 00:40:40.546808 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\"" Nov 03 00:40:40.547140 ip-10-0-140-107 kubenswrapper[2578]: I1103 00:40:40.546845 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"kube-root-ca.crt\"" Nov 03 00:40:40.547140 ip-10-0-140-107 kubenswrapper[2578]: I1103 00:40:40.547062 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\"" Nov 03 00:40:40.548832 ip-10-0-140-107 kubenswrapper[2578]: I1103 00:40:40.548735 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Nov 03 00:40:40.548832 ip-10-0-140-107 kubenswrapper[2578]: I1103 00:40:40.548780 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\"" Nov 03 00:40:40.549144 ip-10-0-140-107 kubenswrapper[2578]: I1103 00:40:40.549125 2578 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-operator-7c4986d449-xprcf"] Nov 03 00:40:40.551753 ip-10-0-140-107 kubenswrapper[2578]: I1103 00:40:40.551731 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemetry-config\"" Nov 03 00:40:40.559570 ip-10-0-140-107 kubenswrapper[2578]: I1103 00:40:40.559538 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/bc513cba-d924-4719-a2ba-711529175ee0-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-z2qgj\" (UID: \"bc513cba-d924-4719-a2ba-711529175ee0\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-z2qgj" Nov 03 00:40:40.559677 ip-10-0-140-107 kubenswrapper[2578]: I1103 00:40:40.559586 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/ca0f883c-c2b7-4bf6-a90b-ffdbeac161a8-tmp\") pod \"insights-operator-7c4986d449-xprcf\" (UID: \"ca0f883c-c2b7-4bf6-a90b-ffdbeac161a8\") " pod="openshift-insights/insights-operator-7c4986d449-xprcf" Nov 03 00:40:40.559677 ip-10-0-140-107 kubenswrapper[2578]: I1103 00:40:40.559658 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/52a19842-3750-4d2b-bcb2-aebddee81097-installation-pull-secrets\") pod \"image-registry-6669dc8887-ws544\" (UID: \"52a19842-3750-4d2b-bcb2-aebddee81097\") " pod="openshift-image-registry/image-registry-6669dc8887-ws544" Nov 03 00:40:40.559765 ip-10-0-140-107 kubenswrapper[2578]: I1103 00:40:40.559717 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qcvwz\" (UniqueName: \"kubernetes.io/projected/ca0f883c-c2b7-4bf6-a90b-ffdbeac161a8-kube-api-access-qcvwz\") pod \"insights-operator-7c4986d449-xprcf\" (UID: \"ca0f883c-c2b7-4bf6-a90b-ffdbeac161a8\") " pod="openshift-insights/insights-operator-7c4986d449-xprcf" Nov 03 00:40:40.559765 ip-10-0-140-107 kubenswrapper[2578]: I1103 00:40:40.559758 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8j6kr\" (UniqueName: \"kubernetes.io/projected/65d134ac-8a25-4c67-b6c7-cd752a098daf-kube-api-access-8j6kr\") pod \"console-operator-7cd56b8dbc-n42s6\" (UID: \"65d134ac-8a25-4c67-b6c7-cd752a098daf\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-n42s6" Nov 03 00:40:40.559836 ip-10-0-140-107 kubenswrapper[2578]: I1103 00:40:40.559774 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nswxk\" (UniqueName: \"kubernetes.io/projected/4237805a-d475-4bd7-865c-c6ef371c0e11-kube-api-access-nswxk\") pod \"kas-connection-checker-685895c7c9-zsd49\" (UID: \"4237805a-d475-4bd7-865c-c6ef371c0e11\") " pod="kube-system/kas-connection-checker-685895c7c9-zsd49" Nov 03 00:40:40.559836 ip-10-0-140-107 kubenswrapper[2578]: I1103 00:40:40.559789 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tc9rq\" (UniqueName: \"kubernetes.io/projected/52a19842-3750-4d2b-bcb2-aebddee81097-kube-api-access-tc9rq\") pod \"image-registry-6669dc8887-ws544\" (UID: \"52a19842-3750-4d2b-bcb2-aebddee81097\") " pod="openshift-image-registry/image-registry-6669dc8887-ws544" Nov 03 00:40:40.559836 ip-10-0-140-107 kubenswrapper[2578]: I1103 00:40:40.559810 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/69b087f8-4a23-4ec8-808e-b61d2e16f67b-metrics-tls\") pod \"dns-default-266ht\" (UID: \"69b087f8-4a23-4ec8-808e-b61d2e16f67b\") " pod="openshift-dns/dns-default-266ht" Nov 03 00:40:40.559836 ip-10-0-140-107 kubenswrapper[2578]: I1103 00:40:40.559826 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/dd0cd8a1-4d6b-424c-ac39-5f5ddb9cd87e-installation-pull-secrets\") pod \"image-registry-65f77c855d-6tnc2\" (UID: \"dd0cd8a1-4d6b-424c-ac39-5f5ddb9cd87e\") " pod="openshift-image-registry/image-registry-65f77c855d-6tnc2" Nov 03 00:40:40.560019 ip-10-0-140-107 kubenswrapper[2578]: I1103 00:40:40.559841 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/69b087f8-4a23-4ec8-808e-b61d2e16f67b-config-volume\") pod \"dns-default-266ht\" (UID: \"69b087f8-4a23-4ec8-808e-b61d2e16f67b\") " pod="openshift-dns/dns-default-266ht" Nov 03 00:40:40.560019 ip-10-0-140-107 kubenswrapper[2578]: I1103 00:40:40.559855 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pj6ft\" (UniqueName: \"kubernetes.io/projected/69b087f8-4a23-4ec8-808e-b61d2e16f67b-kube-api-access-pj6ft\") pod \"dns-default-266ht\" (UID: \"69b087f8-4a23-4ec8-808e-b61d2e16f67b\") " pod="openshift-dns/dns-default-266ht" Nov 03 00:40:40.560019 ip-10-0-140-107 kubenswrapper[2578]: I1103 00:40:40.559870 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/dd0cd8a1-4d6b-424c-ac39-5f5ddb9cd87e-ca-trust-extracted\") pod \"image-registry-65f77c855d-6tnc2\" (UID: \"dd0cd8a1-4d6b-424c-ac39-5f5ddb9cd87e\") " pod="openshift-image-registry/image-registry-65f77c855d-6tnc2" Nov 03 00:40:40.560019 ip-10-0-140-107 kubenswrapper[2578]: I1103 00:40:40.559888 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/36b9d258-cb2c-4d2b-880f-ae93b065034c-serving-cert\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-khmv7\" (UID: \"36b9d258-cb2c-4d2b-880f-ae93b065034c\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-khmv7" Nov 03 00:40:40.560019 ip-10-0-140-107 kubenswrapper[2578]: I1103 00:40:40.559966 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/36b9d258-cb2c-4d2b-880f-ae93b065034c-config\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-khmv7\" (UID: \"36b9d258-cb2c-4d2b-880f-ae93b065034c\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-khmv7" Nov 03 00:40:40.560204 ip-10-0-140-107 kubenswrapper[2578]: I1103 00:40:40.560058 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/52a19842-3750-4d2b-bcb2-aebddee81097-trusted-ca\") pod \"image-registry-6669dc8887-ws544\" (UID: \"52a19842-3750-4d2b-bcb2-aebddee81097\") " pod="openshift-image-registry/image-registry-6669dc8887-ws544" Nov 03 00:40:40.560204 ip-10-0-140-107 kubenswrapper[2578]: I1103 00:40:40.560125 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qmkgl\" (UniqueName: \"kubernetes.io/projected/75481915-3cbe-475b-abe7-cb4e81c93427-kube-api-access-qmkgl\") pod \"ingress-canary-s9hbz\" (UID: \"75481915-3cbe-475b-abe7-cb4e81c93427\") " pod="openshift-ingress-canary/ingress-canary-s9hbz" Nov 03 00:40:40.560204 ip-10-0-140-107 kubenswrapper[2578]: I1103 00:40:40.560160 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/bc513cba-d924-4719-a2ba-711529175ee0-nginx-conf\") pod \"networking-console-plugin-86c478bb88-z2qgj\" (UID: \"bc513cba-d924-4719-a2ba-711529175ee0\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-z2qgj" Nov 03 00:40:40.560322 ip-10-0-140-107 kubenswrapper[2578]: I1103 00:40:40.560238 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cf217a21-c1bd-48be-b203-f59743bceb33-metrics-certs\") pod \"router-default-d7798ccf5-442kt\" (UID: \"cf217a21-c1bd-48be-b203-f59743bceb33\") " pod="openshift-ingress/router-default-d7798ccf5-442kt" Nov 03 00:40:40.560322 ip-10-0-140-107 kubenswrapper[2578]: I1103 00:40:40.560257 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ca0f883c-c2b7-4bf6-a90b-ffdbeac161a8-service-ca-bundle\") pod \"insights-operator-7c4986d449-xprcf\" (UID: \"ca0f883c-c2b7-4bf6-a90b-ffdbeac161a8\") " pod="openshift-insights/insights-operator-7c4986d449-xprcf" Nov 03 00:40:40.560322 ip-10-0-140-107 kubenswrapper[2578]: I1103 00:40:40.560274 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/69b087f8-4a23-4ec8-808e-b61d2e16f67b-tmp-dir\") pod \"dns-default-266ht\" (UID: \"69b087f8-4a23-4ec8-808e-b61d2e16f67b\") " pod="openshift-dns/dns-default-266ht" Nov 03 00:40:40.560482 ip-10-0-140-107 kubenswrapper[2578]: I1103 00:40:40.560463 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6kcsm\" (UniqueName: \"kubernetes.io/projected/6eafe6f6-cf90-446f-a8d5-620cd2e02775-kube-api-access-6kcsm\") pod \"cluster-samples-operator-6ddb6c8b94-2ngwx\" (UID: \"6eafe6f6-cf90-446f-a8d5-620cd2e02775\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-2ngwx" Nov 03 00:40:40.560524 ip-10-0-140-107 kubenswrapper[2578]: I1103 00:40:40.560498 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/dd0cd8a1-4d6b-424c-ac39-5f5ddb9cd87e-image-registry-private-configuration\") pod \"image-registry-65f77c855d-6tnc2\" (UID: \"dd0cd8a1-4d6b-424c-ac39-5f5ddb9cd87e\") " pod="openshift-image-registry/image-registry-65f77c855d-6tnc2" Nov 03 00:40:40.560612 ip-10-0-140-107 kubenswrapper[2578]: I1103 00:40:40.560592 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/dd0cd8a1-4d6b-424c-ac39-5f5ddb9cd87e-registry-tls\") pod \"image-registry-65f77c855d-6tnc2\" (UID: \"dd0cd8a1-4d6b-424c-ac39-5f5ddb9cd87e\") " pod="openshift-image-registry/image-registry-65f77c855d-6tnc2" Nov 03 00:40:40.560696 ip-10-0-140-107 kubenswrapper[2578]: I1103 00:40:40.560680 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/dd0cd8a1-4d6b-424c-ac39-5f5ddb9cd87e-registry-certificates\") pod \"image-registry-65f77c855d-6tnc2\" (UID: \"dd0cd8a1-4d6b-424c-ac39-5f5ddb9cd87e\") " pod="openshift-image-registry/image-registry-65f77c855d-6tnc2" Nov 03 00:40:40.560744 ip-10-0-140-107 kubenswrapper[2578]: I1103 00:40:40.560724 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fdrwp\" (UniqueName: \"kubernetes.io/projected/cf217a21-c1bd-48be-b203-f59743bceb33-kube-api-access-fdrwp\") pod \"router-default-d7798ccf5-442kt\" (UID: \"cf217a21-c1bd-48be-b203-f59743bceb33\") " pod="openshift-ingress/router-default-d7798ccf5-442kt" Nov 03 00:40:40.560800 ip-10-0-140-107 kubenswrapper[2578]: I1103 00:40:40.560751 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-79khw\" (UniqueName: \"kubernetes.io/projected/36b9d258-cb2c-4d2b-880f-ae93b065034c-kube-api-access-79khw\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-khmv7\" (UID: \"36b9d258-cb2c-4d2b-880f-ae93b065034c\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-khmv7" Nov 03 00:40:40.560800 ip-10-0-140-107 kubenswrapper[2578]: I1103 00:40:40.560786 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/06973556-f51c-4d1c-ae55-fb77699bc5b3-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-4bm4f\" (UID: \"06973556-f51c-4d1c-ae55-fb77699bc5b3\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-4bm4f" Nov 03 00:40:40.560937 ip-10-0-140-107 kubenswrapper[2578]: I1103 00:40:40.560864 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jbfhc\" (UniqueName: \"kubernetes.io/projected/dd0cd8a1-4d6b-424c-ac39-5f5ddb9cd87e-kube-api-access-jbfhc\") pod \"image-registry-65f77c855d-6tnc2\" (UID: \"dd0cd8a1-4d6b-424c-ac39-5f5ddb9cd87e\") " pod="openshift-image-registry/image-registry-65f77c855d-6tnc2" Nov 03 00:40:40.560937 ip-10-0-140-107 kubenswrapper[2578]: I1103 00:40:40.560885 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sbn8x\" (UniqueName: \"kubernetes.io/projected/525dc47d-afde-4019-ad01-80614291f282-kube-api-access-sbn8x\") pod \"network-check-source-6f78d59f99-z5f9s\" (UID: \"525dc47d-afde-4019-ad01-80614291f282\") " pod="openshift-network-diagnostics/network-check-source-6f78d59f99-z5f9s" Nov 03 00:40:40.561041 ip-10-0-140-107 kubenswrapper[2578]: I1103 00:40:40.561013 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/ca0f883c-c2b7-4bf6-a90b-ffdbeac161a8-serving-cert\") pod \"insights-operator-7c4986d449-xprcf\" (UID: \"ca0f883c-c2b7-4bf6-a90b-ffdbeac161a8\") " pod="openshift-insights/insights-operator-7c4986d449-xprcf" Nov 03 00:40:40.561084 ip-10-0-140-107 kubenswrapper[2578]: I1103 00:40:40.561058 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/65d134ac-8a25-4c67-b6c7-cd752a098daf-trusted-ca\") pod \"console-operator-7cd56b8dbc-n42s6\" (UID: \"65d134ac-8a25-4c67-b6c7-cd752a098daf\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-n42s6" Nov 03 00:40:40.561125 ip-10-0-140-107 kubenswrapper[2578]: I1103 00:40:40.561087 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/6eafe6f6-cf90-446f-a8d5-620cd2e02775-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-2ngwx\" (UID: \"6eafe6f6-cf90-446f-a8d5-620cd2e02775\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-2ngwx" Nov 03 00:40:40.561125 ip-10-0-140-107 kubenswrapper[2578]: I1103 00:40:40.561111 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/52a19842-3750-4d2b-bcb2-aebddee81097-registry-certificates\") pod \"image-registry-6669dc8887-ws544\" (UID: \"52a19842-3750-4d2b-bcb2-aebddee81097\") " pod="openshift-image-registry/image-registry-6669dc8887-ws544" Nov 03 00:40:40.561262 ip-10-0-140-107 kubenswrapper[2578]: I1103 00:40:40.561141 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/65d134ac-8a25-4c67-b6c7-cd752a098daf-serving-cert\") pod \"console-operator-7cd56b8dbc-n42s6\" (UID: \"65d134ac-8a25-4c67-b6c7-cd752a098daf\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-n42s6" Nov 03 00:40:40.561262 ip-10-0-140-107 kubenswrapper[2578]: I1103 00:40:40.561182 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/cf217a21-c1bd-48be-b203-f59743bceb33-service-ca-bundle\") pod \"router-default-d7798ccf5-442kt\" (UID: \"cf217a21-c1bd-48be-b203-f59743bceb33\") " pod="openshift-ingress/router-default-d7798ccf5-442kt" Nov 03 00:40:40.561262 ip-10-0-140-107 kubenswrapper[2578]: I1103 00:40:40.561214 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h58rt\" (UniqueName: \"kubernetes.io/projected/56261beb-517f-4c88-b59a-42ab41e278fd-kube-api-access-h58rt\") pod \"volume-data-source-validator-58c9f5f46d-w4xtw\" (UID: \"56261beb-517f-4c88-b59a-42ab41e278fd\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-w4xtw" Nov 03 00:40:40.561415 ip-10-0-140-107 kubenswrapper[2578]: I1103 00:40:40.561314 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wzln7\" (UniqueName: \"kubernetes.io/projected/2b050b18-31f6-4a22-9902-0435d11018b9-kube-api-access-wzln7\") pod \"service-ca-operator-544d749c8c-7lqbv\" (UID: \"2b050b18-31f6-4a22-9902-0435d11018b9\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-7lqbv" Nov 03 00:40:40.561415 ip-10-0-140-107 kubenswrapper[2578]: I1103 00:40:40.561351 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/52a19842-3750-4d2b-bcb2-aebddee81097-ca-trust-extracted\") pod \"image-registry-6669dc8887-ws544\" (UID: \"52a19842-3750-4d2b-bcb2-aebddee81097\") " pod="openshift-image-registry/image-registry-6669dc8887-ws544" Nov 03 00:40:40.561415 ip-10-0-140-107 kubenswrapper[2578]: I1103 00:40:40.561390 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/52a19842-3750-4d2b-bcb2-aebddee81097-bound-sa-token\") pod \"image-registry-6669dc8887-ws544\" (UID: \"52a19842-3750-4d2b-bcb2-aebddee81097\") " pod="openshift-image-registry/image-registry-6669dc8887-ws544" Nov 03 00:40:40.561507 ip-10-0-140-107 kubenswrapper[2578]: I1103 00:40:40.561431 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/75481915-3cbe-475b-abe7-cb4e81c93427-cert\") pod \"ingress-canary-s9hbz\" (UID: \"75481915-3cbe-475b-abe7-cb4e81c93427\") " pod="openshift-ingress-canary/ingress-canary-s9hbz" Nov 03 00:40:40.561507 ip-10-0-140-107 kubenswrapper[2578]: I1103 00:40:40.561473 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/2b050b18-31f6-4a22-9902-0435d11018b9-config\") pod \"service-ca-operator-544d749c8c-7lqbv\" (UID: \"2b050b18-31f6-4a22-9902-0435d11018b9\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-7lqbv" Nov 03 00:40:40.561507 ip-10-0-140-107 kubenswrapper[2578]: I1103 00:40:40.561499 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/ca0f883c-c2b7-4bf6-a90b-ffdbeac161a8-snapshots\") pod \"insights-operator-7c4986d449-xprcf\" (UID: \"ca0f883c-c2b7-4bf6-a90b-ffdbeac161a8\") " pod="openshift-insights/insights-operator-7c4986d449-xprcf" Nov 03 00:40:40.561643 ip-10-0-140-107 kubenswrapper[2578]: I1103 00:40:40.561520 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/52a19842-3750-4d2b-bcb2-aebddee81097-registry-tls\") pod \"image-registry-6669dc8887-ws544\" (UID: \"52a19842-3750-4d2b-bcb2-aebddee81097\") " pod="openshift-image-registry/image-registry-6669dc8887-ws544" Nov 03 00:40:40.561643 ip-10-0-140-107 kubenswrapper[2578]: I1103 00:40:40.561609 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/cf217a21-c1bd-48be-b203-f59743bceb33-default-certificate\") pod \"router-default-d7798ccf5-442kt\" (UID: \"cf217a21-c1bd-48be-b203-f59743bceb33\") " pod="openshift-ingress/router-default-d7798ccf5-442kt" Nov 03 00:40:40.561701 ip-10-0-140-107 kubenswrapper[2578]: I1103 00:40:40.561664 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/dd0cd8a1-4d6b-424c-ac39-5f5ddb9cd87e-bound-sa-token\") pod \"image-registry-65f77c855d-6tnc2\" (UID: \"dd0cd8a1-4d6b-424c-ac39-5f5ddb9cd87e\") " pod="openshift-image-registry/image-registry-65f77c855d-6tnc2" Nov 03 00:40:40.561736 ip-10-0-140-107 kubenswrapper[2578]: I1103 00:40:40.561695 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pdmr4\" (UniqueName: \"kubernetes.io/projected/29a64a08-67ef-4c82-8a82-8cd4a4675327-kube-api-access-pdmr4\") pod \"kas-connection-checker-685895c7c9-xst6j\" (UID: \"29a64a08-67ef-4c82-8a82-8cd4a4675327\") " pod="kube-system/kas-connection-checker-685895c7c9-xst6j" Nov 03 00:40:40.561736 ip-10-0-140-107 kubenswrapper[2578]: I1103 00:40:40.561722 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/06973556-f51c-4d1c-ae55-fb77699bc5b3-telemetry-config\") pod \"cluster-monitoring-operator-884576c57-4bm4f\" (UID: \"06973556-f51c-4d1c-ae55-fb77699bc5b3\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-4bm4f" Nov 03 00:40:40.561829 ip-10-0-140-107 kubenswrapper[2578]: I1103 00:40:40.561779 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/2b050b18-31f6-4a22-9902-0435d11018b9-serving-cert\") pod \"service-ca-operator-544d749c8c-7lqbv\" (UID: \"2b050b18-31f6-4a22-9902-0435d11018b9\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-7lqbv" Nov 03 00:40:40.561829 ip-10-0-140-107 kubenswrapper[2578]: I1103 00:40:40.561807 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"samples-operator-config\" (UniqueName: \"kubernetes.io/configmap/6eafe6f6-cf90-446f-a8d5-620cd2e02775-samples-operator-config\") pod \"cluster-samples-operator-6ddb6c8b94-2ngwx\" (UID: \"6eafe6f6-cf90-446f-a8d5-620cd2e02775\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-2ngwx" Nov 03 00:40:40.561944 ip-10-0-140-107 kubenswrapper[2578]: I1103 00:40:40.561834 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4znwf\" (UniqueName: \"kubernetes.io/projected/06973556-f51c-4d1c-ae55-fb77699bc5b3-kube-api-access-4znwf\") pod \"cluster-monitoring-operator-884576c57-4bm4f\" (UID: \"06973556-f51c-4d1c-ae55-fb77699bc5b3\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-4bm4f" Nov 03 00:40:40.561944 ip-10-0-140-107 kubenswrapper[2578]: I1103 00:40:40.561860 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/65d134ac-8a25-4c67-b6c7-cd752a098daf-config\") pod \"console-operator-7cd56b8dbc-n42s6\" (UID: \"65d134ac-8a25-4c67-b6c7-cd752a098daf\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-n42s6" Nov 03 00:40:40.561944 ip-10-0-140-107 kubenswrapper[2578]: I1103 00:40:40.561886 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7zxdt\" (UniqueName: \"kubernetes.io/projected/b0bd4bd5-12ca-49df-a3d9-4243f4bd642e-kube-api-access-7zxdt\") pod \"kas-connection-checker-685895c7c9-ntqw6\" (UID: \"b0bd4bd5-12ca-49df-a3d9-4243f4bd642e\") " pod="kube-system/kas-connection-checker-685895c7c9-ntqw6" Nov 03 00:40:40.562053 ip-10-0-140-107 kubenswrapper[2578]: I1103 00:40:40.561955 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ca0f883c-c2b7-4bf6-a90b-ffdbeac161a8-trusted-ca-bundle\") pod \"insights-operator-7c4986d449-xprcf\" (UID: \"ca0f883c-c2b7-4bf6-a90b-ffdbeac161a8\") " pod="openshift-insights/insights-operator-7c4986d449-xprcf" Nov 03 00:40:40.562053 ip-10-0-140-107 kubenswrapper[2578]: I1103 00:40:40.562025 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/dd0cd8a1-4d6b-424c-ac39-5f5ddb9cd87e-trusted-ca\") pod \"image-registry-65f77c855d-6tnc2\" (UID: \"dd0cd8a1-4d6b-424c-ac39-5f5ddb9cd87e\") " pod="openshift-image-registry/image-registry-65f77c855d-6tnc2" Nov 03 00:40:40.562116 ip-10-0-140-107 kubenswrapper[2578]: I1103 00:40:40.562073 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/52a19842-3750-4d2b-bcb2-aebddee81097-image-registry-private-configuration\") pod \"image-registry-6669dc8887-ws544\" (UID: \"52a19842-3750-4d2b-bcb2-aebddee81097\") " pod="openshift-image-registry/image-registry-6669dc8887-ws544" Nov 03 00:40:40.562116 ip-10-0-140-107 kubenswrapper[2578]: I1103 00:40:40.562107 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/cf217a21-c1bd-48be-b203-f59743bceb33-stats-auth\") pod \"router-default-d7798ccf5-442kt\" (UID: \"cf217a21-c1bd-48be-b203-f59743bceb33\") " pod="openshift-ingress/router-default-d7798ccf5-442kt" Nov 03 00:40:40.562681 ip-10-0-140-107 kubenswrapper[2578]: I1103 00:40:40.562658 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"trusted-ca-bundle\"" Nov 03 00:40:40.563034 ip-10-0-140-107 kubenswrapper[2578]: I1103 00:40:40.563016 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"trusted-ca\"" Nov 03 00:40:40.563565 ip-10-0-140-107 kubenswrapper[2578]: I1103 00:40:40.563544 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\"" Nov 03 00:40:40.574525 ip-10-0-140-107 kubenswrapper[2578]: I1103 00:40:40.574506 2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca-operator\"/\"serving-cert\"" Nov 03 00:40:40.574809 ip-10-0-140-107 kubenswrapper[2578]: I1103 00:40:40.574792 2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca-operator\"/\"service-ca-operator-dockercfg-4kqf6\"" Nov 03 00:40:40.574869 ip-10-0-140-107 kubenswrapper[2578]: I1103 00:40:40.574831 2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\"" Nov 03 00:40:40.574869 ip-10-0-140-107 kubenswrapper[2578]: I1103 00:40:40.574854 2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\"" Nov 03 00:40:40.574991 ip-10-0-140-107 kubenswrapper[2578]: I1103 00:40:40.574978 2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-diagnostics-dockercfg-7vjx2\"" Nov 03 00:40:40.575041 ip-10-0-140-107 kubenswrapper[2578]: I1103 00:40:40.575017 2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-vz52v\"" Nov 03 00:40:40.575563 ip-10-0-140-107 kubenswrapper[2578]: I1103 00:40:40.575544 2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"ingress-canary-dockercfg-k7bjp\"" Nov 03 00:40:40.639574 ip-10-0-140-107 kubenswrapper[2578]: I1103 00:40:40.639515 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 00:40:40.662601 ip-10-0-140-107 kubenswrapper[2578]: I1103 00:40:40.662579 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/52a19842-3750-4d2b-bcb2-aebddee81097-installation-pull-secrets\") pod \"image-registry-6669dc8887-ws544\" (UID: \"52a19842-3750-4d2b-bcb2-aebddee81097\") " pod="openshift-image-registry/image-registry-6669dc8887-ws544" Nov 03 00:40:40.662755 ip-10-0-140-107 kubenswrapper[2578]: I1103 00:40:40.662611 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qcvwz\" (UniqueName: \"kubernetes.io/projected/ca0f883c-c2b7-4bf6-a90b-ffdbeac161a8-kube-api-access-qcvwz\") pod \"insights-operator-7c4986d449-xprcf\" (UID: \"ca0f883c-c2b7-4bf6-a90b-ffdbeac161a8\") " pod="openshift-insights/insights-operator-7c4986d449-xprcf" Nov 03 00:40:40.662755 ip-10-0-140-107 kubenswrapper[2578]: I1103 00:40:40.662631 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8j6kr\" (UniqueName: \"kubernetes.io/projected/65d134ac-8a25-4c67-b6c7-cd752a098daf-kube-api-access-8j6kr\") pod \"console-operator-7cd56b8dbc-n42s6\" (UID: \"65d134ac-8a25-4c67-b6c7-cd752a098daf\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-n42s6" Nov 03 00:40:40.662755 ip-10-0-140-107 kubenswrapper[2578]: I1103 00:40:40.662653 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nswxk\" (UniqueName: \"kubernetes.io/projected/4237805a-d475-4bd7-865c-c6ef371c0e11-kube-api-access-nswxk\") pod \"kas-connection-checker-685895c7c9-zsd49\" (UID: \"4237805a-d475-4bd7-865c-c6ef371c0e11\") " pod="kube-system/kas-connection-checker-685895c7c9-zsd49" Nov 03 00:40:40.662999 ip-10-0-140-107 kubenswrapper[2578]: I1103 00:40:40.662982 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tc9rq\" (UniqueName: \"kubernetes.io/projected/52a19842-3750-4d2b-bcb2-aebddee81097-kube-api-access-tc9rq\") pod \"image-registry-6669dc8887-ws544\" (UID: \"52a19842-3750-4d2b-bcb2-aebddee81097\") " pod="openshift-image-registry/image-registry-6669dc8887-ws544" Nov 03 00:40:40.663067 ip-10-0-140-107 kubenswrapper[2578]: I1103 00:40:40.663018 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/69b087f8-4a23-4ec8-808e-b61d2e16f67b-metrics-tls\") pod \"dns-default-266ht\" (UID: \"69b087f8-4a23-4ec8-808e-b61d2e16f67b\") " pod="openshift-dns/dns-default-266ht" Nov 03 00:40:40.663067 ip-10-0-140-107 kubenswrapper[2578]: I1103 00:40:40.663039 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/dd0cd8a1-4d6b-424c-ac39-5f5ddb9cd87e-installation-pull-secrets\") pod \"image-registry-65f77c855d-6tnc2\" (UID: \"dd0cd8a1-4d6b-424c-ac39-5f5ddb9cd87e\") " pod="openshift-image-registry/image-registry-65f77c855d-6tnc2" Nov 03 00:40:40.663067 ip-10-0-140-107 kubenswrapper[2578]: I1103 00:40:40.663056 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/69b087f8-4a23-4ec8-808e-b61d2e16f67b-config-volume\") pod \"dns-default-266ht\" (UID: \"69b087f8-4a23-4ec8-808e-b61d2e16f67b\") " pod="openshift-dns/dns-default-266ht" Nov 03 00:40:40.663458 ip-10-0-140-107 kubenswrapper[2578]: I1103 00:40:40.663072 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pj6ft\" (UniqueName: \"kubernetes.io/projected/69b087f8-4a23-4ec8-808e-b61d2e16f67b-kube-api-access-pj6ft\") pod \"dns-default-266ht\" (UID: \"69b087f8-4a23-4ec8-808e-b61d2e16f67b\") " pod="openshift-dns/dns-default-266ht" Nov 03 00:40:40.663575 ip-10-0-140-107 kubenswrapper[2578]: I1103 00:40:40.663489 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/dd0cd8a1-4d6b-424c-ac39-5f5ddb9cd87e-ca-trust-extracted\") pod \"image-registry-65f77c855d-6tnc2\" (UID: \"dd0cd8a1-4d6b-424c-ac39-5f5ddb9cd87e\") " pod="openshift-image-registry/image-registry-65f77c855d-6tnc2" Nov 03 00:40:40.663575 ip-10-0-140-107 kubenswrapper[2578]: I1103 00:40:40.663539 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/36b9d258-cb2c-4d2b-880f-ae93b065034c-serving-cert\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-khmv7\" (UID: \"36b9d258-cb2c-4d2b-880f-ae93b065034c\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-khmv7" Nov 03 00:40:40.663691 ip-10-0-140-107 kubenswrapper[2578]: I1103 00:40:40.663574 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/36b9d258-cb2c-4d2b-880f-ae93b065034c-config\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-khmv7\" (UID: \"36b9d258-cb2c-4d2b-880f-ae93b065034c\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-khmv7" Nov 03 00:40:40.663691 ip-10-0-140-107 kubenswrapper[2578]: I1103 00:40:40.663614 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/52a19842-3750-4d2b-bcb2-aebddee81097-trusted-ca\") pod \"image-registry-6669dc8887-ws544\" (UID: \"52a19842-3750-4d2b-bcb2-aebddee81097\") " pod="openshift-image-registry/image-registry-6669dc8887-ws544" Nov 03 00:40:40.663691 ip-10-0-140-107 kubenswrapper[2578]: I1103 00:40:40.663642 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qmkgl\" (UniqueName: \"kubernetes.io/projected/75481915-3cbe-475b-abe7-cb4e81c93427-kube-api-access-qmkgl\") pod \"ingress-canary-s9hbz\" (UID: \"75481915-3cbe-475b-abe7-cb4e81c93427\") " pod="openshift-ingress-canary/ingress-canary-s9hbz" Nov 03 00:40:40.664382 ip-10-0-140-107 kubenswrapper[2578]: I1103 00:40:40.664361 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/36b9d258-cb2c-4d2b-880f-ae93b065034c-config\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-khmv7\" (UID: \"36b9d258-cb2c-4d2b-880f-ae93b065034c\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-khmv7" Nov 03 00:40:40.664491 ip-10-0-140-107 kubenswrapper[2578]: I1103 00:40:40.664397 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/69b087f8-4a23-4ec8-808e-b61d2e16f67b-config-volume\") pod \"dns-default-266ht\" (UID: \"69b087f8-4a23-4ec8-808e-b61d2e16f67b\") " pod="openshift-dns/dns-default-266ht" Nov 03 00:40:40.665235 ip-10-0-140-107 kubenswrapper[2578]: E1103 00:40:40.664806 2578 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Nov 03 00:40:40.665235 ip-10-0-140-107 kubenswrapper[2578]: E1103 00:40:40.665059 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/69b087f8-4a23-4ec8-808e-b61d2e16f67b-metrics-tls podName:69b087f8-4a23-4ec8-808e-b61d2e16f67b nodeName:}" failed. No retries permitted until 2026-11-03 00:40:41.165040556 +0000 UTC m=+31.110620035 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/69b087f8-4a23-4ec8-808e-b61d2e16f67b-metrics-tls") pod "dns-default-266ht" (UID: "69b087f8-4a23-4ec8-808e-b61d2e16f67b") : secret "dns-default-metrics-tls" not found Nov 03 00:40:40.665235 ip-10-0-140-107 kubenswrapper[2578]: I1103 00:40:40.665229 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/bc513cba-d924-4719-a2ba-711529175ee0-nginx-conf\") pod \"networking-console-plugin-86c478bb88-z2qgj\" (UID: \"bc513cba-d924-4719-a2ba-711529175ee0\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-z2qgj" Nov 03 00:40:40.665415 ip-10-0-140-107 kubenswrapper[2578]: I1103 00:40:40.665381 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/52a19842-3750-4d2b-bcb2-aebddee81097-trusted-ca\") pod \"image-registry-6669dc8887-ws544\" (UID: \"52a19842-3750-4d2b-bcb2-aebddee81097\") " pod="openshift-image-registry/image-registry-6669dc8887-ws544" Nov 03 00:40:40.665484 ip-10-0-140-107 kubenswrapper[2578]: I1103 00:40:40.665464 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cf217a21-c1bd-48be-b203-f59743bceb33-metrics-certs\") pod \"router-default-d7798ccf5-442kt\" (UID: \"cf217a21-c1bd-48be-b203-f59743bceb33\") " pod="openshift-ingress/router-default-d7798ccf5-442kt" Nov 03 00:40:40.665534 ip-10-0-140-107 kubenswrapper[2578]: I1103 00:40:40.665507 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ca0f883c-c2b7-4bf6-a90b-ffdbeac161a8-service-ca-bundle\") pod \"insights-operator-7c4986d449-xprcf\" (UID: \"ca0f883c-c2b7-4bf6-a90b-ffdbeac161a8\") " pod="openshift-insights/insights-operator-7c4986d449-xprcf" Nov 03 00:40:40.665579 ip-10-0-140-107 kubenswrapper[2578]: I1103 00:40:40.665541 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/69b087f8-4a23-4ec8-808e-b61d2e16f67b-tmp-dir\") pod \"dns-default-266ht\" (UID: \"69b087f8-4a23-4ec8-808e-b61d2e16f67b\") " pod="openshift-dns/dns-default-266ht" Nov 03 00:40:40.666513 ip-10-0-140-107 kubenswrapper[2578]: I1103 00:40:40.665645 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6kcsm\" (UniqueName: \"kubernetes.io/projected/6eafe6f6-cf90-446f-a8d5-620cd2e02775-kube-api-access-6kcsm\") pod \"cluster-samples-operator-6ddb6c8b94-2ngwx\" (UID: \"6eafe6f6-cf90-446f-a8d5-620cd2e02775\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-2ngwx" Nov 03 00:40:40.666513 ip-10-0-140-107 kubenswrapper[2578]: I1103 00:40:40.665686 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/dd0cd8a1-4d6b-424c-ac39-5f5ddb9cd87e-image-registry-private-configuration\") pod \"image-registry-65f77c855d-6tnc2\" (UID: \"dd0cd8a1-4d6b-424c-ac39-5f5ddb9cd87e\") " pod="openshift-image-registry/image-registry-65f77c855d-6tnc2" Nov 03 00:40:40.666513 ip-10-0-140-107 kubenswrapper[2578]: I1103 00:40:40.665713 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/dd0cd8a1-4d6b-424c-ac39-5f5ddb9cd87e-registry-tls\") pod \"image-registry-65f77c855d-6tnc2\" (UID: \"dd0cd8a1-4d6b-424c-ac39-5f5ddb9cd87e\") " pod="openshift-image-registry/image-registry-65f77c855d-6tnc2" Nov 03 00:40:40.666513 ip-10-0-140-107 kubenswrapper[2578]: I1103 00:40:40.665754 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/dd0cd8a1-4d6b-424c-ac39-5f5ddb9cd87e-registry-certificates\") pod \"image-registry-65f77c855d-6tnc2\" (UID: \"dd0cd8a1-4d6b-424c-ac39-5f5ddb9cd87e\") " pod="openshift-image-registry/image-registry-65f77c855d-6tnc2" Nov 03 00:40:40.666513 ip-10-0-140-107 kubenswrapper[2578]: I1103 00:40:40.665814 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fdrwp\" (UniqueName: \"kubernetes.io/projected/cf217a21-c1bd-48be-b203-f59743bceb33-kube-api-access-fdrwp\") pod \"router-default-d7798ccf5-442kt\" (UID: \"cf217a21-c1bd-48be-b203-f59743bceb33\") " pod="openshift-ingress/router-default-d7798ccf5-442kt" Nov 03 00:40:40.666513 ip-10-0-140-107 kubenswrapper[2578]: I1103 00:40:40.665850 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-79khw\" (UniqueName: \"kubernetes.io/projected/36b9d258-cb2c-4d2b-880f-ae93b065034c-kube-api-access-79khw\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-khmv7\" (UID: \"36b9d258-cb2c-4d2b-880f-ae93b065034c\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-khmv7" Nov 03 00:40:40.666513 ip-10-0-140-107 kubenswrapper[2578]: I1103 00:40:40.665932 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/06973556-f51c-4d1c-ae55-fb77699bc5b3-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-4bm4f\" (UID: \"06973556-f51c-4d1c-ae55-fb77699bc5b3\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-4bm4f" Nov 03 00:40:40.666513 ip-10-0-140-107 kubenswrapper[2578]: I1103 00:40:40.665972 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jbfhc\" (UniqueName: \"kubernetes.io/projected/dd0cd8a1-4d6b-424c-ac39-5f5ddb9cd87e-kube-api-access-jbfhc\") pod \"image-registry-65f77c855d-6tnc2\" (UID: \"dd0cd8a1-4d6b-424c-ac39-5f5ddb9cd87e\") " pod="openshift-image-registry/image-registry-65f77c855d-6tnc2" Nov 03 00:40:40.666513 ip-10-0-140-107 kubenswrapper[2578]: I1103 00:40:40.666008 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sbn8x\" (UniqueName: \"kubernetes.io/projected/525dc47d-afde-4019-ad01-80614291f282-kube-api-access-sbn8x\") pod \"network-check-source-6f78d59f99-z5f9s\" (UID: \"525dc47d-afde-4019-ad01-80614291f282\") " pod="openshift-network-diagnostics/network-check-source-6f78d59f99-z5f9s" Nov 03 00:40:40.666513 ip-10-0-140-107 kubenswrapper[2578]: I1103 00:40:40.666059 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ca0f883c-c2b7-4bf6-a90b-ffdbeac161a8-service-ca-bundle\") pod \"insights-operator-7c4986d449-xprcf\" (UID: \"ca0f883c-c2b7-4bf6-a90b-ffdbeac161a8\") " pod="openshift-insights/insights-operator-7c4986d449-xprcf" Nov 03 00:40:40.666513 ip-10-0-140-107 kubenswrapper[2578]: E1103 00:40:40.666160 2578 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Nov 03 00:40:40.666513 ip-10-0-140-107 kubenswrapper[2578]: E1103 00:40:40.666341 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cf217a21-c1bd-48be-b203-f59743bceb33-metrics-certs podName:cf217a21-c1bd-48be-b203-f59743bceb33 nodeName:}" failed. No retries permitted until 2026-11-03 00:40:41.166323434 +0000 UTC m=+31.111902924 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/cf217a21-c1bd-48be-b203-f59743bceb33-metrics-certs") pod "router-default-d7798ccf5-442kt" (UID: "cf217a21-c1bd-48be-b203-f59743bceb33") : secret "router-metrics-certs-default" not found Nov 03 00:40:40.666513 ip-10-0-140-107 kubenswrapper[2578]: I1103 00:40:40.666159 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/ca0f883c-c2b7-4bf6-a90b-ffdbeac161a8-serving-cert\") pod \"insights-operator-7c4986d449-xprcf\" (UID: \"ca0f883c-c2b7-4bf6-a90b-ffdbeac161a8\") " pod="openshift-insights/insights-operator-7c4986d449-xprcf" Nov 03 00:40:40.666513 ip-10-0-140-107 kubenswrapper[2578]: I1103 00:40:40.666398 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/65d134ac-8a25-4c67-b6c7-cd752a098daf-trusted-ca\") pod \"console-operator-7cd56b8dbc-n42s6\" (UID: \"65d134ac-8a25-4c67-b6c7-cd752a098daf\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-n42s6" Nov 03 00:40:40.666513 ip-10-0-140-107 kubenswrapper[2578]: I1103 00:40:40.666431 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/6eafe6f6-cf90-446f-a8d5-620cd2e02775-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-2ngwx\" (UID: \"6eafe6f6-cf90-446f-a8d5-620cd2e02775\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-2ngwx" Nov 03 00:40:40.667290 ip-10-0-140-107 kubenswrapper[2578]: I1103 00:40:40.666437 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/bc513cba-d924-4719-a2ba-711529175ee0-nginx-conf\") pod \"networking-console-plugin-86c478bb88-z2qgj\" (UID: \"bc513cba-d924-4719-a2ba-711529175ee0\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-z2qgj" Nov 03 00:40:40.667290 ip-10-0-140-107 kubenswrapper[2578]: I1103 00:40:40.666466 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/52a19842-3750-4d2b-bcb2-aebddee81097-registry-certificates\") pod \"image-registry-6669dc8887-ws544\" (UID: \"52a19842-3750-4d2b-bcb2-aebddee81097\") " pod="openshift-image-registry/image-registry-6669dc8887-ws544" Nov 03 00:40:40.667290 ip-10-0-140-107 kubenswrapper[2578]: I1103 00:40:40.666518 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/65d134ac-8a25-4c67-b6c7-cd752a098daf-serving-cert\") pod \"console-operator-7cd56b8dbc-n42s6\" (UID: \"65d134ac-8a25-4c67-b6c7-cd752a098daf\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-n42s6" Nov 03 00:40:40.667290 ip-10-0-140-107 kubenswrapper[2578]: I1103 00:40:40.666549 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/cf217a21-c1bd-48be-b203-f59743bceb33-service-ca-bundle\") pod \"router-default-d7798ccf5-442kt\" (UID: \"cf217a21-c1bd-48be-b203-f59743bceb33\") " pod="openshift-ingress/router-default-d7798ccf5-442kt" Nov 03 00:40:40.667290 ip-10-0-140-107 kubenswrapper[2578]: I1103 00:40:40.666577 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h58rt\" (UniqueName: \"kubernetes.io/projected/56261beb-517f-4c88-b59a-42ab41e278fd-kube-api-access-h58rt\") pod \"volume-data-source-validator-58c9f5f46d-w4xtw\" (UID: \"56261beb-517f-4c88-b59a-42ab41e278fd\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-w4xtw" Nov 03 00:40:40.667290 ip-10-0-140-107 kubenswrapper[2578]: I1103 00:40:40.666612 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wzln7\" (UniqueName: \"kubernetes.io/projected/2b050b18-31f6-4a22-9902-0435d11018b9-kube-api-access-wzln7\") pod \"service-ca-operator-544d749c8c-7lqbv\" (UID: \"2b050b18-31f6-4a22-9902-0435d11018b9\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-7lqbv" Nov 03 00:40:40.667290 ip-10-0-140-107 kubenswrapper[2578]: I1103 00:40:40.666648 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/52a19842-3750-4d2b-bcb2-aebddee81097-ca-trust-extracted\") pod \"image-registry-6669dc8887-ws544\" (UID: \"52a19842-3750-4d2b-bcb2-aebddee81097\") " pod="openshift-image-registry/image-registry-6669dc8887-ws544" Nov 03 00:40:40.667290 ip-10-0-140-107 kubenswrapper[2578]: I1103 00:40:40.666677 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/52a19842-3750-4d2b-bcb2-aebddee81097-bound-sa-token\") pod \"image-registry-6669dc8887-ws544\" (UID: \"52a19842-3750-4d2b-bcb2-aebddee81097\") " pod="openshift-image-registry/image-registry-6669dc8887-ws544" Nov 03 00:40:40.667290 ip-10-0-140-107 kubenswrapper[2578]: I1103 00:40:40.666705 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/75481915-3cbe-475b-abe7-cb4e81c93427-cert\") pod \"ingress-canary-s9hbz\" (UID: \"75481915-3cbe-475b-abe7-cb4e81c93427\") " pod="openshift-ingress-canary/ingress-canary-s9hbz" Nov 03 00:40:40.667290 ip-10-0-140-107 kubenswrapper[2578]: I1103 00:40:40.666730 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/2b050b18-31f6-4a22-9902-0435d11018b9-config\") pod \"service-ca-operator-544d749c8c-7lqbv\" (UID: \"2b050b18-31f6-4a22-9902-0435d11018b9\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-7lqbv" Nov 03 00:40:40.667290 ip-10-0-140-107 kubenswrapper[2578]: I1103 00:40:40.666759 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/ca0f883c-c2b7-4bf6-a90b-ffdbeac161a8-snapshots\") pod \"insights-operator-7c4986d449-xprcf\" (UID: \"ca0f883c-c2b7-4bf6-a90b-ffdbeac161a8\") " pod="openshift-insights/insights-operator-7c4986d449-xprcf" Nov 03 00:40:40.667290 ip-10-0-140-107 kubenswrapper[2578]: I1103 00:40:40.666788 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/52a19842-3750-4d2b-bcb2-aebddee81097-registry-tls\") pod \"image-registry-6669dc8887-ws544\" (UID: \"52a19842-3750-4d2b-bcb2-aebddee81097\") " pod="openshift-image-registry/image-registry-6669dc8887-ws544" Nov 03 00:40:40.667290 ip-10-0-140-107 kubenswrapper[2578]: I1103 00:40:40.666822 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/cf217a21-c1bd-48be-b203-f59743bceb33-default-certificate\") pod \"router-default-d7798ccf5-442kt\" (UID: \"cf217a21-c1bd-48be-b203-f59743bceb33\") " pod="openshift-ingress/router-default-d7798ccf5-442kt" Nov 03 00:40:40.667290 ip-10-0-140-107 kubenswrapper[2578]: I1103 00:40:40.666854 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/dd0cd8a1-4d6b-424c-ac39-5f5ddb9cd87e-bound-sa-token\") pod \"image-registry-65f77c855d-6tnc2\" (UID: \"dd0cd8a1-4d6b-424c-ac39-5f5ddb9cd87e\") " pod="openshift-image-registry/image-registry-65f77c855d-6tnc2" Nov 03 00:40:40.667290 ip-10-0-140-107 kubenswrapper[2578]: I1103 00:40:40.666878 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/69b087f8-4a23-4ec8-808e-b61d2e16f67b-tmp-dir\") pod \"dns-default-266ht\" (UID: \"69b087f8-4a23-4ec8-808e-b61d2e16f67b\") " pod="openshift-dns/dns-default-266ht" Nov 03 00:40:40.667290 ip-10-0-140-107 kubenswrapper[2578]: I1103 00:40:40.666883 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pdmr4\" (UniqueName: \"kubernetes.io/projected/29a64a08-67ef-4c82-8a82-8cd4a4675327-kube-api-access-pdmr4\") pod \"kas-connection-checker-685895c7c9-xst6j\" (UID: \"29a64a08-67ef-4c82-8a82-8cd4a4675327\") " pod="kube-system/kas-connection-checker-685895c7c9-xst6j" Nov 03 00:40:40.667995 ip-10-0-140-107 kubenswrapper[2578]: I1103 00:40:40.666973 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/06973556-f51c-4d1c-ae55-fb77699bc5b3-telemetry-config\") pod \"cluster-monitoring-operator-884576c57-4bm4f\" (UID: \"06973556-f51c-4d1c-ae55-fb77699bc5b3\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-4bm4f" Nov 03 00:40:40.667995 ip-10-0-140-107 kubenswrapper[2578]: I1103 00:40:40.667017 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/2b050b18-31f6-4a22-9902-0435d11018b9-serving-cert\") pod \"service-ca-operator-544d749c8c-7lqbv\" (UID: \"2b050b18-31f6-4a22-9902-0435d11018b9\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-7lqbv" Nov 03 00:40:40.667995 ip-10-0-140-107 kubenswrapper[2578]: I1103 00:40:40.667059 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-config\" (UniqueName: \"kubernetes.io/configmap/6eafe6f6-cf90-446f-a8d5-620cd2e02775-samples-operator-config\") pod \"cluster-samples-operator-6ddb6c8b94-2ngwx\" (UID: \"6eafe6f6-cf90-446f-a8d5-620cd2e02775\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-2ngwx" Nov 03 00:40:40.667995 ip-10-0-140-107 kubenswrapper[2578]: I1103 00:40:40.667093 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4znwf\" (UniqueName: \"kubernetes.io/projected/06973556-f51c-4d1c-ae55-fb77699bc5b3-kube-api-access-4znwf\") pod \"cluster-monitoring-operator-884576c57-4bm4f\" (UID: \"06973556-f51c-4d1c-ae55-fb77699bc5b3\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-4bm4f" Nov 03 00:40:40.667995 ip-10-0-140-107 kubenswrapper[2578]: I1103 00:40:40.667129 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/65d134ac-8a25-4c67-b6c7-cd752a098daf-config\") pod \"console-operator-7cd56b8dbc-n42s6\" (UID: \"65d134ac-8a25-4c67-b6c7-cd752a098daf\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-n42s6" Nov 03 00:40:40.667995 ip-10-0-140-107 kubenswrapper[2578]: I1103 00:40:40.667164 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7zxdt\" (UniqueName: \"kubernetes.io/projected/b0bd4bd5-12ca-49df-a3d9-4243f4bd642e-kube-api-access-7zxdt\") pod \"kas-connection-checker-685895c7c9-ntqw6\" (UID: \"b0bd4bd5-12ca-49df-a3d9-4243f4bd642e\") " pod="kube-system/kas-connection-checker-685895c7c9-ntqw6" Nov 03 00:40:40.667995 ip-10-0-140-107 kubenswrapper[2578]: I1103 00:40:40.665541 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/dd0cd8a1-4d6b-424c-ac39-5f5ddb9cd87e-ca-trust-extracted\") pod \"image-registry-65f77c855d-6tnc2\" (UID: \"dd0cd8a1-4d6b-424c-ac39-5f5ddb9cd87e\") " pod="openshift-image-registry/image-registry-65f77c855d-6tnc2" Nov 03 00:40:40.668329 ip-10-0-140-107 kubenswrapper[2578]: I1103 00:40:40.668191 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/36b9d258-cb2c-4d2b-880f-ae93b065034c-serving-cert\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-khmv7\" (UID: \"36b9d258-cb2c-4d2b-880f-ae93b065034c\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-khmv7" Nov 03 00:40:40.668388 ip-10-0-140-107 kubenswrapper[2578]: I1103 00:40:40.668326 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/52a19842-3750-4d2b-bcb2-aebddee81097-installation-pull-secrets\") pod \"image-registry-6669dc8887-ws544\" (UID: \"52a19842-3750-4d2b-bcb2-aebddee81097\") " pod="openshift-image-registry/image-registry-6669dc8887-ws544" Nov 03 00:40:40.669003 ip-10-0-140-107 kubenswrapper[2578]: I1103 00:40:40.668506 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/dd0cd8a1-4d6b-424c-ac39-5f5ddb9cd87e-installation-pull-secrets\") pod \"image-registry-65f77c855d-6tnc2\" (UID: \"dd0cd8a1-4d6b-424c-ac39-5f5ddb9cd87e\") " pod="openshift-image-registry/image-registry-65f77c855d-6tnc2" Nov 03 00:40:40.669003 ip-10-0-140-107 kubenswrapper[2578]: I1103 00:40:40.668657 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/52a19842-3750-4d2b-bcb2-aebddee81097-ca-trust-extracted\") pod \"image-registry-6669dc8887-ws544\" (UID: \"52a19842-3750-4d2b-bcb2-aebddee81097\") " pod="openshift-image-registry/image-registry-6669dc8887-ws544" Nov 03 00:40:40.669003 ip-10-0-140-107 kubenswrapper[2578]: I1103 00:40:40.668711 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/65d134ac-8a25-4c67-b6c7-cd752a098daf-trusted-ca\") pod \"console-operator-7cd56b8dbc-n42s6\" (UID: \"65d134ac-8a25-4c67-b6c7-cd752a098daf\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-n42s6" Nov 03 00:40:40.669003 ip-10-0-140-107 kubenswrapper[2578]: I1103 00:40:40.668770 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ca0f883c-c2b7-4bf6-a90b-ffdbeac161a8-trusted-ca-bundle\") pod \"insights-operator-7c4986d449-xprcf\" (UID: \"ca0f883c-c2b7-4bf6-a90b-ffdbeac161a8\") " pod="openshift-insights/insights-operator-7c4986d449-xprcf" Nov 03 00:40:40.669003 ip-10-0-140-107 kubenswrapper[2578]: E1103 00:40:40.668782 2578 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Nov 03 00:40:40.669003 ip-10-0-140-107 kubenswrapper[2578]: I1103 00:40:40.668808 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/dd0cd8a1-4d6b-424c-ac39-5f5ddb9cd87e-trusted-ca\") pod \"image-registry-65f77c855d-6tnc2\" (UID: \"dd0cd8a1-4d6b-424c-ac39-5f5ddb9cd87e\") " pod="openshift-image-registry/image-registry-65f77c855d-6tnc2" Nov 03 00:40:40.669003 ip-10-0-140-107 kubenswrapper[2578]: I1103 00:40:40.668842 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/52a19842-3750-4d2b-bcb2-aebddee81097-image-registry-private-configuration\") pod \"image-registry-6669dc8887-ws544\" (UID: \"52a19842-3750-4d2b-bcb2-aebddee81097\") " pod="openshift-image-registry/image-registry-6669dc8887-ws544" Nov 03 00:40:40.669003 ip-10-0-140-107 kubenswrapper[2578]: E1103 00:40:40.668882 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6eafe6f6-cf90-446f-a8d5-620cd2e02775-samples-operator-tls podName:6eafe6f6-cf90-446f-a8d5-620cd2e02775 nodeName:}" failed. No retries permitted until 2026-11-03 00:40:41.168864617 +0000 UTC m=+31.114444104 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/6eafe6f6-cf90-446f-a8d5-620cd2e02775-samples-operator-tls") pod "cluster-samples-operator-6ddb6c8b94-2ngwx" (UID: "6eafe6f6-cf90-446f-a8d5-620cd2e02775") : secret "samples-operator-tls" not found Nov 03 00:40:40.670272 ip-10-0-140-107 kubenswrapper[2578]: I1103 00:40:40.669386 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/52a19842-3750-4d2b-bcb2-aebddee81097-registry-certificates\") pod \"image-registry-6669dc8887-ws544\" (UID: \"52a19842-3750-4d2b-bcb2-aebddee81097\") " pod="openshift-image-registry/image-registry-6669dc8887-ws544" Nov 03 00:40:40.670272 ip-10-0-140-107 kubenswrapper[2578]: I1103 00:40:40.669490 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"samples-operator-config\" (UniqueName: \"kubernetes.io/configmap/6eafe6f6-cf90-446f-a8d5-620cd2e02775-samples-operator-config\") pod \"cluster-samples-operator-6ddb6c8b94-2ngwx\" (UID: \"6eafe6f6-cf90-446f-a8d5-620cd2e02775\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-2ngwx" Nov 03 00:40:40.670272 ip-10-0-140-107 kubenswrapper[2578]: I1103 00:40:40.669567 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/cf217a21-c1bd-48be-b203-f59743bceb33-stats-auth\") pod \"router-default-d7798ccf5-442kt\" (UID: \"cf217a21-c1bd-48be-b203-f59743bceb33\") " pod="openshift-ingress/router-default-d7798ccf5-442kt" Nov 03 00:40:40.670272 ip-10-0-140-107 kubenswrapper[2578]: I1103 00:40:40.669604 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/bc513cba-d924-4719-a2ba-711529175ee0-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-z2qgj\" (UID: \"bc513cba-d924-4719-a2ba-711529175ee0\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-z2qgj" Nov 03 00:40:40.670272 ip-10-0-140-107 kubenswrapper[2578]: I1103 00:40:40.669644 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/ca0f883c-c2b7-4bf6-a90b-ffdbeac161a8-tmp\") pod \"insights-operator-7c4986d449-xprcf\" (UID: \"ca0f883c-c2b7-4bf6-a90b-ffdbeac161a8\") " pod="openshift-insights/insights-operator-7c4986d449-xprcf" Nov 03 00:40:40.670272 ip-10-0-140-107 kubenswrapper[2578]: I1103 00:40:40.669972 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/dd0cd8a1-4d6b-424c-ac39-5f5ddb9cd87e-registry-certificates\") pod \"image-registry-65f77c855d-6tnc2\" (UID: \"dd0cd8a1-4d6b-424c-ac39-5f5ddb9cd87e\") " pod="openshift-image-registry/image-registry-65f77c855d-6tnc2" Nov 03 00:40:40.670272 ip-10-0-140-107 kubenswrapper[2578]: I1103 00:40:40.670037 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/ca0f883c-c2b7-4bf6-a90b-ffdbeac161a8-tmp\") pod \"insights-operator-7c4986d449-xprcf\" (UID: \"ca0f883c-c2b7-4bf6-a90b-ffdbeac161a8\") " pod="openshift-insights/insights-operator-7c4986d449-xprcf" Nov 03 00:40:40.670272 ip-10-0-140-107 kubenswrapper[2578]: E1103 00:40:40.670152 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/cf217a21-c1bd-48be-b203-f59743bceb33-service-ca-bundle podName:cf217a21-c1bd-48be-b203-f59743bceb33 nodeName:}" failed. No retries permitted until 2026-11-03 00:40:41.170130487 +0000 UTC m=+31.115709966 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/cf217a21-c1bd-48be-b203-f59743bceb33-service-ca-bundle") pod "router-default-d7798ccf5-442kt" (UID: "cf217a21-c1bd-48be-b203-f59743bceb33") : configmap references non-existent config key: service-ca.crt Nov 03 00:40:40.673043 ip-10-0-140-107 kubenswrapper[2578]: E1103 00:40:40.670643 2578 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Nov 03 00:40:40.673043 ip-10-0-140-107 kubenswrapper[2578]: E1103 00:40:40.670710 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/06973556-f51c-4d1c-ae55-fb77699bc5b3-cluster-monitoring-operator-tls podName:06973556-f51c-4d1c-ae55-fb77699bc5b3 nodeName:}" failed. No retries permitted until 2026-11-03 00:40:41.170686793 +0000 UTC m=+31.116266290 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/06973556-f51c-4d1c-ae55-fb77699bc5b3-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-884576c57-4bm4f" (UID: "06973556-f51c-4d1c-ae55-fb77699bc5b3") : secret "cluster-monitoring-operator-tls" not found Nov 03 00:40:40.673043 ip-10-0-140-107 kubenswrapper[2578]: E1103 00:40:40.670732 2578 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Nov 03 00:40:40.673043 ip-10-0-140-107 kubenswrapper[2578]: E1103 00:40:40.671187 2578 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Nov 03 00:40:40.673043 ip-10-0-140-107 kubenswrapper[2578]: E1103 00:40:40.671243 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/bc513cba-d924-4719-a2ba-711529175ee0-networking-console-plugin-cert podName:bc513cba-d924-4719-a2ba-711529175ee0 nodeName:}" failed. No retries permitted until 2026-11-03 00:40:41.17122883 +0000 UTC m=+31.116808320 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/bc513cba-d924-4719-a2ba-711529175ee0-networking-console-plugin-cert") pod "networking-console-plugin-86c478bb88-z2qgj" (UID: "bc513cba-d924-4719-a2ba-711529175ee0") : secret "networking-console-plugin-cert" not found Nov 03 00:40:40.673043 ip-10-0-140-107 kubenswrapper[2578]: E1103 00:40:40.671495 2578 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Nov 03 00:40:40.673043 ip-10-0-140-107 kubenswrapper[2578]: E1103 00:40:40.671511 2578 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-6669dc8887-ws544: secret "image-registry-tls" not found Nov 03 00:40:40.673043 ip-10-0-140-107 kubenswrapper[2578]: E1103 00:40:40.671574 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/52a19842-3750-4d2b-bcb2-aebddee81097-registry-tls podName:52a19842-3750-4d2b-bcb2-aebddee81097 nodeName:}" failed. No retries permitted until 2026-11-03 00:40:41.171557074 +0000 UTC m=+31.117136567 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/52a19842-3750-4d2b-bcb2-aebddee81097-registry-tls") pod "image-registry-6669dc8887-ws544" (UID: "52a19842-3750-4d2b-bcb2-aebddee81097") : secret "image-registry-tls" not found Nov 03 00:40:40.673043 ip-10-0-140-107 kubenswrapper[2578]: I1103 00:40:40.671607 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/06973556-f51c-4d1c-ae55-fb77699bc5b3-telemetry-config\") pod \"cluster-monitoring-operator-884576c57-4bm4f\" (UID: \"06973556-f51c-4d1c-ae55-fb77699bc5b3\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-4bm4f" Nov 03 00:40:40.673043 ip-10-0-140-107 kubenswrapper[2578]: I1103 00:40:40.671811 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/2b050b18-31f6-4a22-9902-0435d11018b9-config\") pod \"service-ca-operator-544d749c8c-7lqbv\" (UID: \"2b050b18-31f6-4a22-9902-0435d11018b9\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-7lqbv" Nov 03 00:40:40.673043 ip-10-0-140-107 kubenswrapper[2578]: E1103 00:40:40.670751 2578 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-65f77c855d-6tnc2: secret "image-registry-tls" not found Nov 03 00:40:40.673043 ip-10-0-140-107 kubenswrapper[2578]: E1103 00:40:40.672067 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/dd0cd8a1-4d6b-424c-ac39-5f5ddb9cd87e-registry-tls podName:dd0cd8a1-4d6b-424c-ac39-5f5ddb9cd87e nodeName:}" failed. No retries permitted until 2026-11-03 00:40:41.172051246 +0000 UTC m=+31.117630740 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/dd0cd8a1-4d6b-424c-ac39-5f5ddb9cd87e-registry-tls") pod "image-registry-65f77c855d-6tnc2" (UID: "dd0cd8a1-4d6b-424c-ac39-5f5ddb9cd87e") : secret "image-registry-tls" not found Nov 03 00:40:40.673043 ip-10-0-140-107 kubenswrapper[2578]: I1103 00:40:40.672504 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/dd0cd8a1-4d6b-424c-ac39-5f5ddb9cd87e-trusted-ca\") pod \"image-registry-65f77c855d-6tnc2\" (UID: \"dd0cd8a1-4d6b-424c-ac39-5f5ddb9cd87e\") " pod="openshift-image-registry/image-registry-65f77c855d-6tnc2" Nov 03 00:40:40.675141 ip-10-0-140-107 kubenswrapper[2578]: I1103 00:40:40.673357 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/ca0f883c-c2b7-4bf6-a90b-ffdbeac161a8-snapshots\") pod \"insights-operator-7c4986d449-xprcf\" (UID: \"ca0f883c-c2b7-4bf6-a90b-ffdbeac161a8\") " pod="openshift-insights/insights-operator-7c4986d449-xprcf" Nov 03 00:40:40.675141 ip-10-0-140-107 kubenswrapper[2578]: I1103 00:40:40.673867 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/65d134ac-8a25-4c67-b6c7-cd752a098daf-config\") pod \"console-operator-7cd56b8dbc-n42s6\" (UID: \"65d134ac-8a25-4c67-b6c7-cd752a098daf\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-n42s6" Nov 03 00:40:40.675141 ip-10-0-140-107 kubenswrapper[2578]: I1103 00:40:40.673956 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/52a19842-3750-4d2b-bcb2-aebddee81097-image-registry-private-configuration\") pod \"image-registry-6669dc8887-ws544\" (UID: \"52a19842-3750-4d2b-bcb2-aebddee81097\") " pod="openshift-image-registry/image-registry-6669dc8887-ws544" Nov 03 00:40:40.675141 ip-10-0-140-107 kubenswrapper[2578]: I1103 00:40:40.674147 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/2b050b18-31f6-4a22-9902-0435d11018b9-serving-cert\") pod \"service-ca-operator-544d749c8c-7lqbv\" (UID: \"2b050b18-31f6-4a22-9902-0435d11018b9\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-7lqbv" Nov 03 00:40:40.675141 ip-10-0-140-107 kubenswrapper[2578]: I1103 00:40:40.674357 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ca0f883c-c2b7-4bf6-a90b-ffdbeac161a8-trusted-ca-bundle\") pod \"insights-operator-7c4986d449-xprcf\" (UID: \"ca0f883c-c2b7-4bf6-a90b-ffdbeac161a8\") " pod="openshift-insights/insights-operator-7c4986d449-xprcf" Nov 03 00:40:40.675141 ip-10-0-140-107 kubenswrapper[2578]: I1103 00:40:40.674455 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/dd0cd8a1-4d6b-424c-ac39-5f5ddb9cd87e-image-registry-private-configuration\") pod \"image-registry-65f77c855d-6tnc2\" (UID: \"dd0cd8a1-4d6b-424c-ac39-5f5ddb9cd87e\") " pod="openshift-image-registry/image-registry-65f77c855d-6tnc2" Nov 03 00:40:40.676233 ip-10-0-140-107 kubenswrapper[2578]: I1103 00:40:40.676111 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/cf217a21-c1bd-48be-b203-f59743bceb33-default-certificate\") pod \"router-default-d7798ccf5-442kt\" (UID: \"cf217a21-c1bd-48be-b203-f59743bceb33\") " pod="openshift-ingress/router-default-d7798ccf5-442kt" Nov 03 00:40:40.676533 ip-10-0-140-107 kubenswrapper[2578]: I1103 00:40:40.676480 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/ca0f883c-c2b7-4bf6-a90b-ffdbeac161a8-serving-cert\") pod \"insights-operator-7c4986d449-xprcf\" (UID: \"ca0f883c-c2b7-4bf6-a90b-ffdbeac161a8\") " pod="openshift-insights/insights-operator-7c4986d449-xprcf" Nov 03 00:40:40.677071 ip-10-0-140-107 kubenswrapper[2578]: I1103 00:40:40.677046 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8j6kr\" (UniqueName: \"kubernetes.io/projected/65d134ac-8a25-4c67-b6c7-cd752a098daf-kube-api-access-8j6kr\") pod \"console-operator-7cd56b8dbc-n42s6\" (UID: \"65d134ac-8a25-4c67-b6c7-cd752a098daf\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-n42s6" Nov 03 00:40:40.677226 ip-10-0-140-107 kubenswrapper[2578]: I1103 00:40:40.677203 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tc9rq\" (UniqueName: \"kubernetes.io/projected/52a19842-3750-4d2b-bcb2-aebddee81097-kube-api-access-tc9rq\") pod \"image-registry-6669dc8887-ws544\" (UID: \"52a19842-3750-4d2b-bcb2-aebddee81097\") " pod="openshift-image-registry/image-registry-6669dc8887-ws544" Nov 03 00:40:40.677430 ip-10-0-140-107 kubenswrapper[2578]: I1103 00:40:40.677411 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/65d134ac-8a25-4c67-b6c7-cd752a098daf-serving-cert\") pod \"console-operator-7cd56b8dbc-n42s6\" (UID: \"65d134ac-8a25-4c67-b6c7-cd752a098daf\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-n42s6" Nov 03 00:40:40.678235 ip-10-0-140-107 kubenswrapper[2578]: I1103 00:40:40.678188 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/75481915-3cbe-475b-abe7-cb4e81c93427-cert\") pod \"ingress-canary-s9hbz\" (UID: \"75481915-3cbe-475b-abe7-cb4e81c93427\") " pod="openshift-ingress-canary/ingress-canary-s9hbz" Nov 03 00:40:40.679323 ip-10-0-140-107 kubenswrapper[2578]: I1103 00:40:40.679295 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pj6ft\" (UniqueName: \"kubernetes.io/projected/69b087f8-4a23-4ec8-808e-b61d2e16f67b-kube-api-access-pj6ft\") pod \"dns-default-266ht\" (UID: \"69b087f8-4a23-4ec8-808e-b61d2e16f67b\") " pod="openshift-dns/dns-default-266ht" Nov 03 00:40:40.679414 ip-10-0-140-107 kubenswrapper[2578]: I1103 00:40:40.679298 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pdmr4\" (UniqueName: \"kubernetes.io/projected/29a64a08-67ef-4c82-8a82-8cd4a4675327-kube-api-access-pdmr4\") pod \"kas-connection-checker-685895c7c9-xst6j\" (UID: \"29a64a08-67ef-4c82-8a82-8cd4a4675327\") " pod="kube-system/kas-connection-checker-685895c7c9-xst6j" Nov 03 00:40:40.679414 ip-10-0-140-107 kubenswrapper[2578]: I1103 00:40:40.679340 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/cf217a21-c1bd-48be-b203-f59743bceb33-stats-auth\") pod \"router-default-d7798ccf5-442kt\" (UID: \"cf217a21-c1bd-48be-b203-f59743bceb33\") " pod="openshift-ingress/router-default-d7798ccf5-442kt" Nov 03 00:40:40.680593 ip-10-0-140-107 kubenswrapper[2578]: I1103 00:40:40.680567 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6kcsm\" (UniqueName: \"kubernetes.io/projected/6eafe6f6-cf90-446f-a8d5-620cd2e02775-kube-api-access-6kcsm\") pod \"cluster-samples-operator-6ddb6c8b94-2ngwx\" (UID: \"6eafe6f6-cf90-446f-a8d5-620cd2e02775\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-2ngwx" Nov 03 00:40:40.680975 ip-10-0-140-107 kubenswrapper[2578]: I1103 00:40:40.680956 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7zxdt\" (UniqueName: \"kubernetes.io/projected/b0bd4bd5-12ca-49df-a3d9-4243f4bd642e-kube-api-access-7zxdt\") pod \"kas-connection-checker-685895c7c9-ntqw6\" (UID: \"b0bd4bd5-12ca-49df-a3d9-4243f4bd642e\") " pod="kube-system/kas-connection-checker-685895c7c9-ntqw6" Nov 03 00:40:40.681058 ip-10-0-140-107 kubenswrapper[2578]: I1103 00:40:40.681031 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nswxk\" (UniqueName: \"kubernetes.io/projected/4237805a-d475-4bd7-865c-c6ef371c0e11-kube-api-access-nswxk\") pod \"kas-connection-checker-685895c7c9-zsd49\" (UID: \"4237805a-d475-4bd7-865c-c6ef371c0e11\") " pod="kube-system/kas-connection-checker-685895c7c9-zsd49" Nov 03 00:40:40.681102 ip-10-0-140-107 kubenswrapper[2578]: I1103 00:40:40.681050 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qcvwz\" (UniqueName: \"kubernetes.io/projected/ca0f883c-c2b7-4bf6-a90b-ffdbeac161a8-kube-api-access-qcvwz\") pod \"insights-operator-7c4986d449-xprcf\" (UID: \"ca0f883c-c2b7-4bf6-a90b-ffdbeac161a8\") " pod="openshift-insights/insights-operator-7c4986d449-xprcf" Nov 03 00:40:40.681716 ip-10-0-140-107 kubenswrapper[2578]: I1103 00:40:40.681695 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qmkgl\" (UniqueName: \"kubernetes.io/projected/75481915-3cbe-475b-abe7-cb4e81c93427-kube-api-access-qmkgl\") pod \"ingress-canary-s9hbz\" (UID: \"75481915-3cbe-475b-abe7-cb4e81c93427\") " pod="openshift-ingress-canary/ingress-canary-s9hbz" Nov 03 00:40:40.682395 ip-10-0-140-107 kubenswrapper[2578]: I1103 00:40:40.682374 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-h58rt\" (UniqueName: \"kubernetes.io/projected/56261beb-517f-4c88-b59a-42ab41e278fd-kube-api-access-h58rt\") pod \"volume-data-source-validator-58c9f5f46d-w4xtw\" (UID: \"56261beb-517f-4c88-b59a-42ab41e278fd\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-w4xtw" Nov 03 00:40:40.688348 ip-10-0-140-107 kubenswrapper[2578]: I1103 00:40:40.688322 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4znwf\" (UniqueName: \"kubernetes.io/projected/06973556-f51c-4d1c-ae55-fb77699bc5b3-kube-api-access-4znwf\") pod \"cluster-monitoring-operator-884576c57-4bm4f\" (UID: \"06973556-f51c-4d1c-ae55-fb77699bc5b3\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-4bm4f" Nov 03 00:40:40.688851 ip-10-0-140-107 kubenswrapper[2578]: I1103 00:40:40.688808 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/dd0cd8a1-4d6b-424c-ac39-5f5ddb9cd87e-bound-sa-token\") pod \"image-registry-65f77c855d-6tnc2\" (UID: \"dd0cd8a1-4d6b-424c-ac39-5f5ddb9cd87e\") " pod="openshift-image-registry/image-registry-65f77c855d-6tnc2" Nov 03 00:40:40.688975 ip-10-0-140-107 kubenswrapper[2578]: I1103 00:40:40.688952 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jbfhc\" (UniqueName: \"kubernetes.io/projected/dd0cd8a1-4d6b-424c-ac39-5f5ddb9cd87e-kube-api-access-jbfhc\") pod \"image-registry-65f77c855d-6tnc2\" (UID: \"dd0cd8a1-4d6b-424c-ac39-5f5ddb9cd87e\") " pod="openshift-image-registry/image-registry-65f77c855d-6tnc2" Nov 03 00:40:40.688975 ip-10-0-140-107 kubenswrapper[2578]: I1103 00:40:40.688959 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sbn8x\" (UniqueName: \"kubernetes.io/projected/525dc47d-afde-4019-ad01-80614291f282-kube-api-access-sbn8x\") pod \"network-check-source-6f78d59f99-z5f9s\" (UID: \"525dc47d-afde-4019-ad01-80614291f282\") " pod="openshift-network-diagnostics/network-check-source-6f78d59f99-z5f9s" Nov 03 00:40:40.689115 ip-10-0-140-107 kubenswrapper[2578]: I1103 00:40:40.689097 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wzln7\" (UniqueName: \"kubernetes.io/projected/2b050b18-31f6-4a22-9902-0435d11018b9-kube-api-access-wzln7\") pod \"service-ca-operator-544d749c8c-7lqbv\" (UID: \"2b050b18-31f6-4a22-9902-0435d11018b9\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-7lqbv" Nov 03 00:40:40.689544 ip-10-0-140-107 kubenswrapper[2578]: I1103 00:40:40.689525 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-79khw\" (UniqueName: \"kubernetes.io/projected/36b9d258-cb2c-4d2b-880f-ae93b065034c-kube-api-access-79khw\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-khmv7\" (UID: \"36b9d258-cb2c-4d2b-880f-ae93b065034c\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-khmv7" Nov 03 00:40:40.690359 ip-10-0-140-107 kubenswrapper[2578]: I1103 00:40:40.690341 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/52a19842-3750-4d2b-bcb2-aebddee81097-bound-sa-token\") pod \"image-registry-6669dc8887-ws544\" (UID: \"52a19842-3750-4d2b-bcb2-aebddee81097\") " pod="openshift-image-registry/image-registry-6669dc8887-ws544" Nov 03 00:40:40.690682 ip-10-0-140-107 kubenswrapper[2578]: I1103 00:40:40.690666 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fdrwp\" (UniqueName: \"kubernetes.io/projected/cf217a21-c1bd-48be-b203-f59743bceb33-kube-api-access-fdrwp\") pod \"router-default-d7798ccf5-442kt\" (UID: \"cf217a21-c1bd-48be-b203-f59743bceb33\") " pod="openshift-ingress/router-default-d7798ccf5-442kt" Nov 03 00:40:40.744925 ip-10-0-140-107 kubenswrapper[2578]: I1103 00:40:40.744882 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kas-connection-checker-685895c7c9-zsd49" Nov 03 00:40:40.753875 ip-10-0-140-107 kubenswrapper[2578]: I1103 00:40:40.753851 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kas-connection-checker-685895c7c9-xst6j" Nov 03 00:40:40.761377 ip-10-0-140-107 kubenswrapper[2578]: I1103 00:40:40.761356 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kas-connection-checker-685895c7c9-ntqw6" Nov 03 00:40:40.797879 ip-10-0-140-107 kubenswrapper[2578]: I1103 00:40:40.797841 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-khmv7" Nov 03 00:40:40.801182 ip-10-0-140-107 kubenswrapper[2578]: I1103 00:40:40.801115 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 00:40:40.814580 ip-10-0-140-107 kubenswrapper[2578]: I1103 00:40:40.813325 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console-operator/console-operator-7cd56b8dbc-n42s6" Nov 03 00:40:40.821252 ip-10-0-140-107 kubenswrapper[2578]: I1103 00:40:40.821099 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-operator-7c4986d449-xprcf" Nov 03 00:40:40.859864 ip-10-0-140-107 kubenswrapper[2578]: I1103 00:40:40.859831 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-w4xtw" Nov 03 00:40:40.867958 ip-10-0-140-107 kubenswrapper[2578]: I1103 00:40:40.867111 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-source-6f78d59f99-z5f9s" Nov 03 00:40:40.874653 ip-10-0-140-107 kubenswrapper[2578]: I1103 00:40:40.874454 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-7lqbv" Nov 03 00:40:40.886025 ip-10-0-140-107 kubenswrapper[2578]: I1103 00:40:40.885998 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-s9hbz" Nov 03 00:40:40.978642 ip-10-0-140-107 kubenswrapper[2578]: I1103 00:40:40.978590 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 00:40:41.036584 ip-10-0-140-107 kubenswrapper[2578]: I1103 00:40:41.036560 2578 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-zsd49"] Nov 03 00:40:41.039278 ip-10-0-140-107 kubenswrapper[2578]: W1103 00:40:41.039250 2578 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4237805a_d475_4bd7_865c_c6ef371c0e11.slice/crio-6708f186ee33a9346160371d6e1ddfb5e3a0cbdec1bf050f2ee75b910e994ce9 WatchSource:0}: Error finding container 6708f186ee33a9346160371d6e1ddfb5e3a0cbdec1bf050f2ee75b910e994ce9: Status 404 returned error can't find the container with id 6708f186ee33a9346160371d6e1ddfb5e3a0cbdec1bf050f2ee75b910e994ce9 Nov 03 00:40:41.042594 ip-10-0-140-107 kubenswrapper[2578]: I1103 00:40:41.042525 2578 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-ntqw6"] Nov 03 00:40:41.045565 ip-10-0-140-107 kubenswrapper[2578]: W1103 00:40:41.045536 2578 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb0bd4bd5_12ca_49df_a3d9_4243f4bd642e.slice/crio-d3e8e3bcd4255307bff4cba939cd812eaaae2e8342e455d9fa4ede17d02790c5 WatchSource:0}: Error finding container d3e8e3bcd4255307bff4cba939cd812eaaae2e8342e455d9fa4ede17d02790c5: Status 404 returned error can't find the container with id d3e8e3bcd4255307bff4cba939cd812eaaae2e8342e455d9fa4ede17d02790c5 Nov 03 00:40:41.048830 ip-10-0-140-107 kubenswrapper[2578]: I1103 00:40:41.048808 2578 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-xst6j"] Nov 03 00:40:41.051900 ip-10-0-140-107 kubenswrapper[2578]: W1103 00:40:41.051878 2578 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod29a64a08_67ef_4c82_8a82_8cd4a4675327.slice/crio-ad4f86165f311454a0c13817aa2b917537300761a1b4dd713a21743be3bbc553 WatchSource:0}: Error finding container ad4f86165f311454a0c13817aa2b917537300761a1b4dd713a21743be3bbc553: Status 404 returned error can't find the container with id ad4f86165f311454a0c13817aa2b917537300761a1b4dd713a21743be3bbc553 Nov 03 00:40:41.099085 ip-10-0-140-107 kubenswrapper[2578]: I1103 00:40:41.099051 2578 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-khmv7"] Nov 03 00:40:41.108987 ip-10-0-140-107 kubenswrapper[2578]: W1103 00:40:41.108950 2578 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod36b9d258_cb2c_4d2b_880f_ae93b065034c.slice/crio-487ea3f04a7119c9c89cde91e9dda8b21902247978fb80fbfd356adf8237b019 WatchSource:0}: Error finding container 487ea3f04a7119c9c89cde91e9dda8b21902247978fb80fbfd356adf8237b019: Status 404 returned error can't find the container with id 487ea3f04a7119c9c89cde91e9dda8b21902247978fb80fbfd356adf8237b019 Nov 03 00:40:41.134840 ip-10-0-140-107 kubenswrapper[2578]: I1103 00:40:41.134806 2578 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-operator-7c4986d449-xprcf"] Nov 03 00:40:41.143120 ip-10-0-140-107 kubenswrapper[2578]: I1103 00:40:41.143054 2578 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console-operator/console-operator-7cd56b8dbc-n42s6"] Nov 03 00:40:41.150414 ip-10-0-140-107 kubenswrapper[2578]: W1103 00:40:41.150177 2578 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podca0f883c_c2b7_4bf6_a90b_ffdbeac161a8.slice/crio-662fed273c35eab0aa9fe8026120a6defc961c8ef53105c04feef133bbb54108 WatchSource:0}: Error finding container 662fed273c35eab0aa9fe8026120a6defc961c8ef53105c04feef133bbb54108: Status 404 returned error can't find the container with id 662fed273c35eab0aa9fe8026120a6defc961c8ef53105c04feef133bbb54108 Nov 03 00:40:41.170935 ip-10-0-140-107 kubenswrapper[2578]: W1103 00:40:41.170889 2578 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod65d134ac_8a25_4c67_b6c7_cd752a098daf.slice/crio-0444234d1c00351a0c0dec655e17f704fbb70c110f8904dc87014c9c860c40db WatchSource:0}: Error finding container 0444234d1c00351a0c0dec655e17f704fbb70c110f8904dc87014c9c860c40db: Status 404 returned error can't find the container with id 0444234d1c00351a0c0dec655e17f704fbb70c110f8904dc87014c9c860c40db Nov 03 00:40:41.175452 ip-10-0-140-107 kubenswrapper[2578]: I1103 00:40:41.175425 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/06973556-f51c-4d1c-ae55-fb77699bc5b3-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-4bm4f\" (UID: \"06973556-f51c-4d1c-ae55-fb77699bc5b3\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-4bm4f" Nov 03 00:40:41.175562 ip-10-0-140-107 kubenswrapper[2578]: I1103 00:40:41.175478 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/6eafe6f6-cf90-446f-a8d5-620cd2e02775-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-2ngwx\" (UID: \"6eafe6f6-cf90-446f-a8d5-620cd2e02775\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-2ngwx" Nov 03 00:40:41.175562 ip-10-0-140-107 kubenswrapper[2578]: I1103 00:40:41.175517 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/cf217a21-c1bd-48be-b203-f59743bceb33-service-ca-bundle\") pod \"router-default-d7798ccf5-442kt\" (UID: \"cf217a21-c1bd-48be-b203-f59743bceb33\") " pod="openshift-ingress/router-default-d7798ccf5-442kt" Nov 03 00:40:41.175671 ip-10-0-140-107 kubenswrapper[2578]: I1103 00:40:41.175561 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/52a19842-3750-4d2b-bcb2-aebddee81097-registry-tls\") pod \"image-registry-6669dc8887-ws544\" (UID: \"52a19842-3750-4d2b-bcb2-aebddee81097\") " pod="openshift-image-registry/image-registry-6669dc8887-ws544" Nov 03 00:40:41.175671 ip-10-0-140-107 kubenswrapper[2578]: E1103 00:40:41.175587 2578 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Nov 03 00:40:41.175671 ip-10-0-140-107 kubenswrapper[2578]: I1103 00:40:41.175617 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/bc513cba-d924-4719-a2ba-711529175ee0-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-z2qgj\" (UID: \"bc513cba-d924-4719-a2ba-711529175ee0\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-z2qgj" Nov 03 00:40:41.175671 ip-10-0-140-107 kubenswrapper[2578]: E1103 00:40:41.175659 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/06973556-f51c-4d1c-ae55-fb77699bc5b3-cluster-monitoring-operator-tls podName:06973556-f51c-4d1c-ae55-fb77699bc5b3 nodeName:}" failed. No retries permitted until 2026-11-03 00:40:42.175634798 +0000 UTC m=+32.121214292 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/06973556-f51c-4d1c-ae55-fb77699bc5b3-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-884576c57-4bm4f" (UID: "06973556-f51c-4d1c-ae55-fb77699bc5b3") : secret "cluster-monitoring-operator-tls" not found Nov 03 00:40:41.175991 ip-10-0-140-107 kubenswrapper[2578]: E1103 00:40:41.175705 2578 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Nov 03 00:40:41.175991 ip-10-0-140-107 kubenswrapper[2578]: I1103 00:40:41.175719 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/69b087f8-4a23-4ec8-808e-b61d2e16f67b-metrics-tls\") pod \"dns-default-266ht\" (UID: \"69b087f8-4a23-4ec8-808e-b61d2e16f67b\") " pod="openshift-dns/dns-default-266ht" Nov 03 00:40:41.175991 ip-10-0-140-107 kubenswrapper[2578]: E1103 00:40:41.175752 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/bc513cba-d924-4719-a2ba-711529175ee0-networking-console-plugin-cert podName:bc513cba-d924-4719-a2ba-711529175ee0 nodeName:}" failed. No retries permitted until 2026-11-03 00:40:42.175737808 +0000 UTC m=+32.121317305 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/bc513cba-d924-4719-a2ba-711529175ee0-networking-console-plugin-cert") pod "networking-console-plugin-86c478bb88-z2qgj" (UID: "bc513cba-d924-4719-a2ba-711529175ee0") : secret "networking-console-plugin-cert" not found Nov 03 00:40:41.175991 ip-10-0-140-107 kubenswrapper[2578]: E1103 00:40:41.175793 2578 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Nov 03 00:40:41.175991 ip-10-0-140-107 kubenswrapper[2578]: I1103 00:40:41.175806 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cf217a21-c1bd-48be-b203-f59743bceb33-metrics-certs\") pod \"router-default-d7798ccf5-442kt\" (UID: \"cf217a21-c1bd-48be-b203-f59743bceb33\") " pod="openshift-ingress/router-default-d7798ccf5-442kt" Nov 03 00:40:41.175991 ip-10-0-140-107 kubenswrapper[2578]: E1103 00:40:41.175834 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/69b087f8-4a23-4ec8-808e-b61d2e16f67b-metrics-tls podName:69b087f8-4a23-4ec8-808e-b61d2e16f67b nodeName:}" failed. No retries permitted until 2026-11-03 00:40:42.175820856 +0000 UTC m=+32.121400349 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/69b087f8-4a23-4ec8-808e-b61d2e16f67b-metrics-tls") pod "dns-default-266ht" (UID: "69b087f8-4a23-4ec8-808e-b61d2e16f67b") : secret "dns-default-metrics-tls" not found Nov 03 00:40:41.175991 ip-10-0-140-107 kubenswrapper[2578]: E1103 00:40:41.175868 2578 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Nov 03 00:40:41.175991 ip-10-0-140-107 kubenswrapper[2578]: I1103 00:40:41.175867 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/dd0cd8a1-4d6b-424c-ac39-5f5ddb9cd87e-registry-tls\") pod \"image-registry-65f77c855d-6tnc2\" (UID: \"dd0cd8a1-4d6b-424c-ac39-5f5ddb9cd87e\") " pod="openshift-image-registry/image-registry-65f77c855d-6tnc2" Nov 03 00:40:41.175991 ip-10-0-140-107 kubenswrapper[2578]: E1103 00:40:41.175921 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cf217a21-c1bd-48be-b203-f59743bceb33-metrics-certs podName:cf217a21-c1bd-48be-b203-f59743bceb33 nodeName:}" failed. No retries permitted until 2026-11-03 00:40:42.175890187 +0000 UTC m=+32.121469667 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/cf217a21-c1bd-48be-b203-f59743bceb33-metrics-certs") pod "router-default-d7798ccf5-442kt" (UID: "cf217a21-c1bd-48be-b203-f59743bceb33") : secret "router-metrics-certs-default" not found Nov 03 00:40:41.175991 ip-10-0-140-107 kubenswrapper[2578]: E1103 00:40:41.175978 2578 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Nov 03 00:40:41.175991 ip-10-0-140-107 kubenswrapper[2578]: E1103 00:40:41.175990 2578 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Nov 03 00:40:41.176484 ip-10-0-140-107 kubenswrapper[2578]: E1103 00:40:41.176004 2578 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-65f77c855d-6tnc2: secret "image-registry-tls" not found Nov 03 00:40:41.176484 ip-10-0-140-107 kubenswrapper[2578]: E1103 00:40:41.176015 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6eafe6f6-cf90-446f-a8d5-620cd2e02775-samples-operator-tls podName:6eafe6f6-cf90-446f-a8d5-620cd2e02775 nodeName:}" failed. No retries permitted until 2026-11-03 00:40:42.176003962 +0000 UTC m=+32.121583455 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/6eafe6f6-cf90-446f-a8d5-620cd2e02775-samples-operator-tls") pod "cluster-samples-operator-6ddb6c8b94-2ngwx" (UID: "6eafe6f6-cf90-446f-a8d5-620cd2e02775") : secret "samples-operator-tls" not found Nov 03 00:40:41.176484 ip-10-0-140-107 kubenswrapper[2578]: E1103 00:40:41.176042 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/dd0cd8a1-4d6b-424c-ac39-5f5ddb9cd87e-registry-tls podName:dd0cd8a1-4d6b-424c-ac39-5f5ddb9cd87e nodeName:}" failed. No retries permitted until 2026-11-03 00:40:42.176029691 +0000 UTC m=+32.121609182 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/dd0cd8a1-4d6b-424c-ac39-5f5ddb9cd87e-registry-tls") pod "image-registry-65f77c855d-6tnc2" (UID: "dd0cd8a1-4d6b-424c-ac39-5f5ddb9cd87e") : secret "image-registry-tls" not found Nov 03 00:40:41.176484 ip-10-0-140-107 kubenswrapper[2578]: E1103 00:40:41.176079 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/cf217a21-c1bd-48be-b203-f59743bceb33-service-ca-bundle podName:cf217a21-c1bd-48be-b203-f59743bceb33 nodeName:}" failed. No retries permitted until 2026-11-03 00:40:42.176068247 +0000 UTC m=+32.121647741 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/cf217a21-c1bd-48be-b203-f59743bceb33-service-ca-bundle") pod "router-default-d7798ccf5-442kt" (UID: "cf217a21-c1bd-48be-b203-f59743bceb33") : configmap references non-existent config key: service-ca.crt Nov 03 00:40:41.176484 ip-10-0-140-107 kubenswrapper[2578]: E1103 00:40:41.176120 2578 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Nov 03 00:40:41.176484 ip-10-0-140-107 kubenswrapper[2578]: E1103 00:40:41.176131 2578 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-6669dc8887-ws544: secret "image-registry-tls" not found Nov 03 00:40:41.176484 ip-10-0-140-107 kubenswrapper[2578]: E1103 00:40:41.176172 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/52a19842-3750-4d2b-bcb2-aebddee81097-registry-tls podName:52a19842-3750-4d2b-bcb2-aebddee81097 nodeName:}" failed. No retries permitted until 2026-11-03 00:40:42.176160657 +0000 UTC m=+32.121740150 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/52a19842-3750-4d2b-bcb2-aebddee81097-registry-tls") pod "image-registry-6669dc8887-ws544" (UID: "52a19842-3750-4d2b-bcb2-aebddee81097") : secret "image-registry-tls" not found Nov 03 00:40:41.181100 ip-10-0-140-107 kubenswrapper[2578]: I1103 00:40:41.179346 2578 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-w4xtw"] Nov 03 00:40:41.187676 ip-10-0-140-107 kubenswrapper[2578]: W1103 00:40:41.187650 2578 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod56261beb_517f_4c88_b59a_42ab41e278fd.slice/crio-b11da0d7baa29ab225e3891350eb46858d211cf3575d1b66e0c76ad4b382e2fa WatchSource:0}: Error finding container b11da0d7baa29ab225e3891350eb46858d211cf3575d1b66e0c76ad4b382e2fa: Status 404 returned error can't find the container with id b11da0d7baa29ab225e3891350eb46858d211cf3575d1b66e0c76ad4b382e2fa Nov 03 00:40:41.196750 ip-10-0-140-107 kubenswrapper[2578]: I1103 00:40:41.196720 2578 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca-operator/service-ca-operator-544d749c8c-7lqbv"] Nov 03 00:40:41.215735 ip-10-0-140-107 kubenswrapper[2578]: I1103 00:40:41.215713 2578 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-source-6f78d59f99-z5f9s"] Nov 03 00:40:41.217959 ip-10-0-140-107 kubenswrapper[2578]: I1103 00:40:41.217723 2578 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-s9hbz"] Nov 03 00:40:41.219358 ip-10-0-140-107 kubenswrapper[2578]: W1103 00:40:41.219334 2578 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod525dc47d_afde_4019_ad01_80614291f282.slice/crio-05e3d4718d6fafed62c11717499adc9949e4df03b37a914bd49d38181d2f088f WatchSource:0}: Error finding container 05e3d4718d6fafed62c11717499adc9949e4df03b37a914bd49d38181d2f088f: Status 404 returned error can't find the container with id 05e3d4718d6fafed62c11717499adc9949e4df03b37a914bd49d38181d2f088f Nov 03 00:40:41.220645 ip-10-0-140-107 kubenswrapper[2578]: W1103 00:40:41.220624 2578 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod75481915_3cbe_475b_abe7_cb4e81c93427.slice/crio-e86455a72def567d132d9ca3c0f5e9449cfde0bf23efbff663c0c38506834a86 WatchSource:0}: Error finding container e86455a72def567d132d9ca3c0f5e9449cfde0bf23efbff663c0c38506834a86: Status 404 returned error can't find the container with id e86455a72def567d132d9ca3c0f5e9449cfde0bf23efbff663c0c38506834a86 Nov 03 00:40:41.299051 ip-10-0-140-107 kubenswrapper[2578]: I1103 00:40:41.299021 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 00:40:41.429409 ip-10-0-140-107 kubenswrapper[2578]: I1103 00:40:41.429323 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 00:40:41.605003 ip-10-0-140-107 kubenswrapper[2578]: I1103 00:40:41.604977 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-jv7cw" Nov 03 00:40:41.605133 ip-10-0-140-107 kubenswrapper[2578]: I1103 00:40:41.605073 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-rt2hx" Nov 03 00:40:41.605133 ip-10-0-140-107 kubenswrapper[2578]: I1103 00:40:41.605108 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-w46kd" Nov 03 00:40:41.609463 ip-10-0-140-107 kubenswrapper[2578]: I1103 00:40:41.609437 2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Nov 03 00:40:41.609597 ip-10-0-140-107 kubenswrapper[2578]: I1103 00:40:41.609546 2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-check-target-dockercfg-7trdq\"" Nov 03 00:40:41.609687 ip-10-0-140-107 kubenswrapper[2578]: I1103 00:40:41.609673 2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-28tgg\"" Nov 03 00:40:41.609775 ip-10-0-140-107 kubenswrapper[2578]: I1103 00:40:41.609763 2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"global-pull-secret-syncer-dockercfg-vmfsq\"" Nov 03 00:40:41.610852 ip-10-0-140-107 kubenswrapper[2578]: I1103 00:40:41.610829 2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Nov 03 00:40:41.922634 ip-10-0-140-107 kubenswrapper[2578]: I1103 00:40:41.922596 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-w4xtw" event={"ID":"56261beb-517f-4c88-b59a-42ab41e278fd","Type":"ContainerStarted","Data":"b11da0d7baa29ab225e3891350eb46858d211cf3575d1b66e0c76ad4b382e2fa"} Nov 03 00:40:41.924956 ip-10-0-140-107 kubenswrapper[2578]: I1103 00:40:41.924927 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-7cd56b8dbc-n42s6" event={"ID":"65d134ac-8a25-4c67-b6c7-cd752a098daf","Type":"ContainerStarted","Data":"0444234d1c00351a0c0dec655e17f704fbb70c110f8904dc87014c9c860c40db"} Nov 03 00:40:41.929895 ip-10-0-140-107 kubenswrapper[2578]: I1103 00:40:41.929867 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kas-connection-checker-685895c7c9-xst6j" event={"ID":"29a64a08-67ef-4c82-8a82-8cd4a4675327","Type":"ContainerStarted","Data":"662ad5c59408c2b6640dbb04bc0a7f7f03e78072e830b82dcc9a2a14d61f51dd"} Nov 03 00:40:41.930008 ip-10-0-140-107 kubenswrapper[2578]: I1103 00:40:41.929936 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kas-connection-checker-685895c7c9-xst6j" event={"ID":"29a64a08-67ef-4c82-8a82-8cd4a4675327","Type":"ContainerStarted","Data":"ad4f86165f311454a0c13817aa2b917537300761a1b4dd713a21743be3bbc553"} Nov 03 00:40:41.932603 ip-10-0-140-107 kubenswrapper[2578]: I1103 00:40:41.932542 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-source-6f78d59f99-z5f9s" event={"ID":"525dc47d-afde-4019-ad01-80614291f282","Type":"ContainerStarted","Data":"05e3d4718d6fafed62c11717499adc9949e4df03b37a914bd49d38181d2f088f"} Nov 03 00:40:41.938933 ip-10-0-140-107 kubenswrapper[2578]: I1103 00:40:41.938584 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kas-connection-checker-685895c7c9-ntqw6" event={"ID":"b0bd4bd5-12ca-49df-a3d9-4243f4bd642e","Type":"ContainerStarted","Data":"a281ed0c7a6fd69c35e0a50d4408a452c999f8a01167160dd87ff02978b68384"} Nov 03 00:40:41.938933 ip-10-0-140-107 kubenswrapper[2578]: I1103 00:40:41.938625 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kas-connection-checker-685895c7c9-ntqw6" event={"ID":"b0bd4bd5-12ca-49df-a3d9-4243f4bd642e","Type":"ContainerStarted","Data":"d3e8e3bcd4255307bff4cba939cd812eaaae2e8342e455d9fa4ede17d02790c5"} Nov 03 00:40:41.941135 ip-10-0-140-107 kubenswrapper[2578]: I1103 00:40:41.941047 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-s9hbz" event={"ID":"75481915-3cbe-475b-abe7-cb4e81c93427","Type":"ContainerStarted","Data":"e86455a72def567d132d9ca3c0f5e9449cfde0bf23efbff663c0c38506834a86"} Nov 03 00:40:41.942931 ip-10-0-140-107 kubenswrapper[2578]: I1103 00:40:41.942871 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-khmv7" event={"ID":"36b9d258-cb2c-4d2b-880f-ae93b065034c","Type":"ContainerStarted","Data":"487ea3f04a7119c9c89cde91e9dda8b21902247978fb80fbfd356adf8237b019"} Nov 03 00:40:41.946580 ip-10-0-140-107 kubenswrapper[2578]: I1103 00:40:41.946087 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kas-connection-checker-685895c7c9-zsd49" event={"ID":"4237805a-d475-4bd7-865c-c6ef371c0e11","Type":"ContainerStarted","Data":"ec48575c1ac084f036763e795d34b2d26b0c0e0b8c81fd08ce6698d79a2ca4c4"} Nov 03 00:40:41.946580 ip-10-0-140-107 kubenswrapper[2578]: I1103 00:40:41.946131 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kas-connection-checker-685895c7c9-zsd49" event={"ID":"4237805a-d475-4bd7-865c-c6ef371c0e11","Type":"ContainerStarted","Data":"6708f186ee33a9346160371d6e1ddfb5e3a0cbdec1bf050f2ee75b910e994ce9"} Nov 03 00:40:41.952214 ip-10-0-140-107 kubenswrapper[2578]: I1103 00:40:41.950652 2578 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/kas-connection-checker-685895c7c9-xst6j" podStartSLOduration=381.950637415 podStartE2EDuration="6m21.950637415s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:34:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:40:41.950526061 +0000 UTC m=+31.896105561" watchObservedRunningTime="2026-11-03 00:40:41.950637415 +0000 UTC m=+31.896216915" Nov 03 00:40:41.952214 ip-10-0-140-107 kubenswrapper[2578]: I1103 00:40:41.950889 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-7lqbv" event={"ID":"2b050b18-31f6-4a22-9902-0435d11018b9","Type":"ContainerStarted","Data":"b942f9e92ab3f2cf912cab48cffd6b46e23dd27024f8c374d7e54e710cdd082f"} Nov 03 00:40:41.953398 ip-10-0-140-107 kubenswrapper[2578]: I1103 00:40:41.953341 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-7c4986d449-xprcf" event={"ID":"ca0f883c-c2b7-4bf6-a90b-ffdbeac161a8","Type":"ContainerStarted","Data":"662fed273c35eab0aa9fe8026120a6defc961c8ef53105c04feef133bbb54108"} Nov 03 00:40:41.970500 ip-10-0-140-107 kubenswrapper[2578]: I1103 00:40:41.970450 2578 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/kas-connection-checker-685895c7c9-ntqw6" podStartSLOduration=381.970434565 podStartE2EDuration="6m21.970434565s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:34:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:40:41.969051594 +0000 UTC m=+31.914631094" watchObservedRunningTime="2026-11-03 00:40:41.970434565 +0000 UTC m=+31.916014066" Nov 03 00:40:41.990559 ip-10-0-140-107 kubenswrapper[2578]: I1103 00:40:41.990504 2578 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/kas-connection-checker-685895c7c9-zsd49" podStartSLOduration=381.990446439 podStartE2EDuration="6m21.990446439s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:34:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:40:41.989088588 +0000 UTC m=+31.934668089" watchObservedRunningTime="2026-11-03 00:40:41.990446439 +0000 UTC m=+31.936025939" Nov 03 00:40:42.185997 ip-10-0-140-107 kubenswrapper[2578]: I1103 00:40:42.185904 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/69b087f8-4a23-4ec8-808e-b61d2e16f67b-metrics-tls\") pod \"dns-default-266ht\" (UID: \"69b087f8-4a23-4ec8-808e-b61d2e16f67b\") " pod="openshift-dns/dns-default-266ht" Nov 03 00:40:42.186152 ip-10-0-140-107 kubenswrapper[2578]: I1103 00:40:42.185997 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cf217a21-c1bd-48be-b203-f59743bceb33-metrics-certs\") pod \"router-default-d7798ccf5-442kt\" (UID: \"cf217a21-c1bd-48be-b203-f59743bceb33\") " pod="openshift-ingress/router-default-d7798ccf5-442kt" Nov 03 00:40:42.186152 ip-10-0-140-107 kubenswrapper[2578]: I1103 00:40:42.186033 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/dd0cd8a1-4d6b-424c-ac39-5f5ddb9cd87e-registry-tls\") pod \"image-registry-65f77c855d-6tnc2\" (UID: \"dd0cd8a1-4d6b-424c-ac39-5f5ddb9cd87e\") " pod="openshift-image-registry/image-registry-65f77c855d-6tnc2" Nov 03 00:40:42.186152 ip-10-0-140-107 kubenswrapper[2578]: I1103 00:40:42.186084 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/06973556-f51c-4d1c-ae55-fb77699bc5b3-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-4bm4f\" (UID: \"06973556-f51c-4d1c-ae55-fb77699bc5b3\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-4bm4f" Nov 03 00:40:42.186152 ip-10-0-140-107 kubenswrapper[2578]: I1103 00:40:42.186120 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/6eafe6f6-cf90-446f-a8d5-620cd2e02775-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-2ngwx\" (UID: \"6eafe6f6-cf90-446f-a8d5-620cd2e02775\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-2ngwx" Nov 03 00:40:42.186152 ip-10-0-140-107 kubenswrapper[2578]: I1103 00:40:42.186150 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/cf217a21-c1bd-48be-b203-f59743bceb33-service-ca-bundle\") pod \"router-default-d7798ccf5-442kt\" (UID: \"cf217a21-c1bd-48be-b203-f59743bceb33\") " pod="openshift-ingress/router-default-d7798ccf5-442kt" Nov 03 00:40:42.186392 ip-10-0-140-107 kubenswrapper[2578]: I1103 00:40:42.186185 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/52a19842-3750-4d2b-bcb2-aebddee81097-registry-tls\") pod \"image-registry-6669dc8887-ws544\" (UID: \"52a19842-3750-4d2b-bcb2-aebddee81097\") " pod="openshift-image-registry/image-registry-6669dc8887-ws544" Nov 03 00:40:42.186392 ip-10-0-140-107 kubenswrapper[2578]: I1103 00:40:42.186240 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/bc513cba-d924-4719-a2ba-711529175ee0-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-z2qgj\" (UID: \"bc513cba-d924-4719-a2ba-711529175ee0\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-z2qgj" Nov 03 00:40:42.186392 ip-10-0-140-107 kubenswrapper[2578]: E1103 00:40:42.186378 2578 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Nov 03 00:40:42.186532 ip-10-0-140-107 kubenswrapper[2578]: E1103 00:40:42.186444 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/bc513cba-d924-4719-a2ba-711529175ee0-networking-console-plugin-cert podName:bc513cba-d924-4719-a2ba-711529175ee0 nodeName:}" failed. No retries permitted until 2026-11-03 00:40:44.186426656 +0000 UTC m=+34.132006137 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/bc513cba-d924-4719-a2ba-711529175ee0-networking-console-plugin-cert") pod "networking-console-plugin-86c478bb88-z2qgj" (UID: "bc513cba-d924-4719-a2ba-711529175ee0") : secret "networking-console-plugin-cert" not found Nov 03 00:40:42.186532 ip-10-0-140-107 kubenswrapper[2578]: E1103 00:40:42.186509 2578 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Nov 03 00:40:42.186628 ip-10-0-140-107 kubenswrapper[2578]: E1103 00:40:42.186540 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/69b087f8-4a23-4ec8-808e-b61d2e16f67b-metrics-tls podName:69b087f8-4a23-4ec8-808e-b61d2e16f67b nodeName:}" failed. No retries permitted until 2026-11-03 00:40:44.186529861 +0000 UTC m=+34.132109337 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/69b087f8-4a23-4ec8-808e-b61d2e16f67b-metrics-tls") pod "dns-default-266ht" (UID: "69b087f8-4a23-4ec8-808e-b61d2e16f67b") : secret "dns-default-metrics-tls" not found Nov 03 00:40:42.186628 ip-10-0-140-107 kubenswrapper[2578]: E1103 00:40:42.186583 2578 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Nov 03 00:40:42.186628 ip-10-0-140-107 kubenswrapper[2578]: E1103 00:40:42.186605 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cf217a21-c1bd-48be-b203-f59743bceb33-metrics-certs podName:cf217a21-c1bd-48be-b203-f59743bceb33 nodeName:}" failed. No retries permitted until 2026-11-03 00:40:44.18659925 +0000 UTC m=+34.132178726 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/cf217a21-c1bd-48be-b203-f59743bceb33-metrics-certs") pod "router-default-d7798ccf5-442kt" (UID: "cf217a21-c1bd-48be-b203-f59743bceb33") : secret "router-metrics-certs-default" not found Nov 03 00:40:42.186775 ip-10-0-140-107 kubenswrapper[2578]: E1103 00:40:42.186658 2578 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Nov 03 00:40:42.186775 ip-10-0-140-107 kubenswrapper[2578]: E1103 00:40:42.186669 2578 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-65f77c855d-6tnc2: secret "image-registry-tls" not found Nov 03 00:40:42.186775 ip-10-0-140-107 kubenswrapper[2578]: E1103 00:40:42.186690 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/dd0cd8a1-4d6b-424c-ac39-5f5ddb9cd87e-registry-tls podName:dd0cd8a1-4d6b-424c-ac39-5f5ddb9cd87e nodeName:}" failed. No retries permitted until 2026-11-03 00:40:44.186684247 +0000 UTC m=+34.132263724 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/dd0cd8a1-4d6b-424c-ac39-5f5ddb9cd87e-registry-tls") pod "image-registry-65f77c855d-6tnc2" (UID: "dd0cd8a1-4d6b-424c-ac39-5f5ddb9cd87e") : secret "image-registry-tls" not found Nov 03 00:40:42.186775 ip-10-0-140-107 kubenswrapper[2578]: E1103 00:40:42.186741 2578 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Nov 03 00:40:42.186775 ip-10-0-140-107 kubenswrapper[2578]: E1103 00:40:42.186763 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/06973556-f51c-4d1c-ae55-fb77699bc5b3-cluster-monitoring-operator-tls podName:06973556-f51c-4d1c-ae55-fb77699bc5b3 nodeName:}" failed. No retries permitted until 2026-11-03 00:40:44.186757391 +0000 UTC m=+34.132336868 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/06973556-f51c-4d1c-ae55-fb77699bc5b3-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-884576c57-4bm4f" (UID: "06973556-f51c-4d1c-ae55-fb77699bc5b3") : secret "cluster-monitoring-operator-tls" not found Nov 03 00:40:42.187062 ip-10-0-140-107 kubenswrapper[2578]: E1103 00:40:42.186800 2578 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Nov 03 00:40:42.187062 ip-10-0-140-107 kubenswrapper[2578]: E1103 00:40:42.186824 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6eafe6f6-cf90-446f-a8d5-620cd2e02775-samples-operator-tls podName:6eafe6f6-cf90-446f-a8d5-620cd2e02775 nodeName:}" failed. No retries permitted until 2026-11-03 00:40:44.18681795 +0000 UTC m=+34.132397427 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/6eafe6f6-cf90-446f-a8d5-620cd2e02775-samples-operator-tls") pod "cluster-samples-operator-6ddb6c8b94-2ngwx" (UID: "6eafe6f6-cf90-446f-a8d5-620cd2e02775") : secret "samples-operator-tls" not found Nov 03 00:40:42.187062 ip-10-0-140-107 kubenswrapper[2578]: E1103 00:40:42.186863 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/cf217a21-c1bd-48be-b203-f59743bceb33-service-ca-bundle podName:cf217a21-c1bd-48be-b203-f59743bceb33 nodeName:}" failed. No retries permitted until 2026-11-03 00:40:44.186858308 +0000 UTC m=+34.132437785 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/cf217a21-c1bd-48be-b203-f59743bceb33-service-ca-bundle") pod "router-default-d7798ccf5-442kt" (UID: "cf217a21-c1bd-48be-b203-f59743bceb33") : configmap references non-existent config key: service-ca.crt Nov 03 00:40:42.187062 ip-10-0-140-107 kubenswrapper[2578]: E1103 00:40:42.186921 2578 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Nov 03 00:40:42.187062 ip-10-0-140-107 kubenswrapper[2578]: E1103 00:40:42.186932 2578 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-6669dc8887-ws544: secret "image-registry-tls" not found Nov 03 00:40:42.187062 ip-10-0-140-107 kubenswrapper[2578]: E1103 00:40:42.186960 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/52a19842-3750-4d2b-bcb2-aebddee81097-registry-tls podName:52a19842-3750-4d2b-bcb2-aebddee81097 nodeName:}" failed. No retries permitted until 2026-11-03 00:40:44.186950342 +0000 UTC m=+34.132529833 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/52a19842-3750-4d2b-bcb2-aebddee81097-registry-tls") pod "image-registry-6669dc8887-ws544" (UID: "52a19842-3750-4d2b-bcb2-aebddee81097") : secret "image-registry-tls" not found Nov 03 00:40:43.965421 ip-10-0-140-107 kubenswrapper[2578]: I1103 00:40:43.965381 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-w4xtw" event={"ID":"56261beb-517f-4c88-b59a-42ab41e278fd","Type":"ContainerStarted","Data":"3b83975dc694997e571f740cd940b719eda84f63f7c533d6d861181ff5cbd683"} Nov 03 00:40:43.994641 ip-10-0-140-107 kubenswrapper[2578]: I1103 00:40:43.993733 2578 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-w4xtw" podStartSLOduration=309.334468557 podStartE2EDuration="5m10.993714273s" totalImagesPullingTime="1.659245716s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:35:33 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:40:43.992376438 +0000 UTC m=+33.937955939" watchObservedRunningTime="2026-11-03 00:40:43.993714273 +0000 UTC m=+33.939293772" Nov 03 00:40:44.183307 ip-10-0-140-107 kubenswrapper[2578]: I1103 00:40:44.183261 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 00:40:44.215545 ip-10-0-140-107 kubenswrapper[2578]: I1103 00:40:44.215507 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/69b087f8-4a23-4ec8-808e-b61d2e16f67b-metrics-tls\") pod \"dns-default-266ht\" (UID: \"69b087f8-4a23-4ec8-808e-b61d2e16f67b\") " pod="openshift-dns/dns-default-266ht" Nov 03 00:40:44.215708 ip-10-0-140-107 kubenswrapper[2578]: I1103 00:40:44.215590 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cf217a21-c1bd-48be-b203-f59743bceb33-metrics-certs\") pod \"router-default-d7798ccf5-442kt\" (UID: \"cf217a21-c1bd-48be-b203-f59743bceb33\") " pod="openshift-ingress/router-default-d7798ccf5-442kt" Nov 03 00:40:44.215708 ip-10-0-140-107 kubenswrapper[2578]: I1103 00:40:44.215625 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/dd0cd8a1-4d6b-424c-ac39-5f5ddb9cd87e-registry-tls\") pod \"image-registry-65f77c855d-6tnc2\" (UID: \"dd0cd8a1-4d6b-424c-ac39-5f5ddb9cd87e\") " pod="openshift-image-registry/image-registry-65f77c855d-6tnc2" Nov 03 00:40:44.215708 ip-10-0-140-107 kubenswrapper[2578]: I1103 00:40:44.215673 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/06973556-f51c-4d1c-ae55-fb77699bc5b3-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-4bm4f\" (UID: \"06973556-f51c-4d1c-ae55-fb77699bc5b3\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-4bm4f" Nov 03 00:40:44.215977 ip-10-0-140-107 kubenswrapper[2578]: I1103 00:40:44.215707 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/6eafe6f6-cf90-446f-a8d5-620cd2e02775-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-2ngwx\" (UID: \"6eafe6f6-cf90-446f-a8d5-620cd2e02775\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-2ngwx" Nov 03 00:40:44.215977 ip-10-0-140-107 kubenswrapper[2578]: I1103 00:40:44.215737 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/cf217a21-c1bd-48be-b203-f59743bceb33-service-ca-bundle\") pod \"router-default-d7798ccf5-442kt\" (UID: \"cf217a21-c1bd-48be-b203-f59743bceb33\") " pod="openshift-ingress/router-default-d7798ccf5-442kt" Nov 03 00:40:44.215977 ip-10-0-140-107 kubenswrapper[2578]: I1103 00:40:44.215762 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/52a19842-3750-4d2b-bcb2-aebddee81097-registry-tls\") pod \"image-registry-6669dc8887-ws544\" (UID: \"52a19842-3750-4d2b-bcb2-aebddee81097\") " pod="openshift-image-registry/image-registry-6669dc8887-ws544" Nov 03 00:40:44.215977 ip-10-0-140-107 kubenswrapper[2578]: I1103 00:40:44.215787 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/136086c0-4846-437a-ad98-3cfef50610a0-metrics-certs\") pod \"network-metrics-daemon-rt2hx\" (UID: \"136086c0-4846-437a-ad98-3cfef50610a0\") " pod="openshift-multus/network-metrics-daemon-rt2hx" Nov 03 00:40:44.215977 ip-10-0-140-107 kubenswrapper[2578]: I1103 00:40:44.215804 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/bc513cba-d924-4719-a2ba-711529175ee0-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-z2qgj\" (UID: \"bc513cba-d924-4719-a2ba-711529175ee0\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-z2qgj" Nov 03 00:40:44.215977 ip-10-0-140-107 kubenswrapper[2578]: E1103 00:40:44.215938 2578 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Nov 03 00:40:44.216256 ip-10-0-140-107 kubenswrapper[2578]: E1103 00:40:44.215989 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/bc513cba-d924-4719-a2ba-711529175ee0-networking-console-plugin-cert podName:bc513cba-d924-4719-a2ba-711529175ee0 nodeName:}" failed. No retries permitted until 2026-11-03 00:40:48.215976321 +0000 UTC m=+38.161555799 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/bc513cba-d924-4719-a2ba-711529175ee0-networking-console-plugin-cert") pod "networking-console-plugin-86c478bb88-z2qgj" (UID: "bc513cba-d924-4719-a2ba-711529175ee0") : secret "networking-console-plugin-cert" not found Nov 03 00:40:44.216312 ip-10-0-140-107 kubenswrapper[2578]: E1103 00:40:44.216304 2578 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Nov 03 00:40:44.216397 ip-10-0-140-107 kubenswrapper[2578]: E1103 00:40:44.216348 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/69b087f8-4a23-4ec8-808e-b61d2e16f67b-metrics-tls podName:69b087f8-4a23-4ec8-808e-b61d2e16f67b nodeName:}" failed. No retries permitted until 2026-11-03 00:40:48.216335639 +0000 UTC m=+38.161915118 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/69b087f8-4a23-4ec8-808e-b61d2e16f67b-metrics-tls") pod "dns-default-266ht" (UID: "69b087f8-4a23-4ec8-808e-b61d2e16f67b") : secret "dns-default-metrics-tls" not found Nov 03 00:40:44.216397 ip-10-0-140-107 kubenswrapper[2578]: E1103 00:40:44.216388 2578 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Nov 03 00:40:44.216481 ip-10-0-140-107 kubenswrapper[2578]: E1103 00:40:44.216410 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cf217a21-c1bd-48be-b203-f59743bceb33-metrics-certs podName:cf217a21-c1bd-48be-b203-f59743bceb33 nodeName:}" failed. No retries permitted until 2026-11-03 00:40:48.216401816 +0000 UTC m=+38.161981294 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/cf217a21-c1bd-48be-b203-f59743bceb33-metrics-certs") pod "router-default-d7798ccf5-442kt" (UID: "cf217a21-c1bd-48be-b203-f59743bceb33") : secret "router-metrics-certs-default" not found Nov 03 00:40:44.216481 ip-10-0-140-107 kubenswrapper[2578]: E1103 00:40:44.216459 2578 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Nov 03 00:40:44.216481 ip-10-0-140-107 kubenswrapper[2578]: E1103 00:40:44.216467 2578 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-65f77c855d-6tnc2: secret "image-registry-tls" not found Nov 03 00:40:44.216578 ip-10-0-140-107 kubenswrapper[2578]: E1103 00:40:44.216489 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/dd0cd8a1-4d6b-424c-ac39-5f5ddb9cd87e-registry-tls podName:dd0cd8a1-4d6b-424c-ac39-5f5ddb9cd87e nodeName:}" failed. No retries permitted until 2026-11-03 00:40:48.216481742 +0000 UTC m=+38.162061219 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/dd0cd8a1-4d6b-424c-ac39-5f5ddb9cd87e-registry-tls") pod "image-registry-65f77c855d-6tnc2" (UID: "dd0cd8a1-4d6b-424c-ac39-5f5ddb9cd87e") : secret "image-registry-tls" not found Nov 03 00:40:44.216578 ip-10-0-140-107 kubenswrapper[2578]: E1103 00:40:44.216530 2578 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Nov 03 00:40:44.216578 ip-10-0-140-107 kubenswrapper[2578]: E1103 00:40:44.216550 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/06973556-f51c-4d1c-ae55-fb77699bc5b3-cluster-monitoring-operator-tls podName:06973556-f51c-4d1c-ae55-fb77699bc5b3 nodeName:}" failed. No retries permitted until 2026-11-03 00:40:48.216544877 +0000 UTC m=+38.162124354 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/06973556-f51c-4d1c-ae55-fb77699bc5b3-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-884576c57-4bm4f" (UID: "06973556-f51c-4d1c-ae55-fb77699bc5b3") : secret "cluster-monitoring-operator-tls" not found Nov 03 00:40:44.216677 ip-10-0-140-107 kubenswrapper[2578]: E1103 00:40:44.216583 2578 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Nov 03 00:40:44.216677 ip-10-0-140-107 kubenswrapper[2578]: E1103 00:40:44.216608 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6eafe6f6-cf90-446f-a8d5-620cd2e02775-samples-operator-tls podName:6eafe6f6-cf90-446f-a8d5-620cd2e02775 nodeName:}" failed. No retries permitted until 2026-11-03 00:40:48.216601322 +0000 UTC m=+38.162180799 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/6eafe6f6-cf90-446f-a8d5-620cd2e02775-samples-operator-tls") pod "cluster-samples-operator-6ddb6c8b94-2ngwx" (UID: "6eafe6f6-cf90-446f-a8d5-620cd2e02775") : secret "samples-operator-tls" not found Nov 03 00:40:44.216677 ip-10-0-140-107 kubenswrapper[2578]: E1103 00:40:44.216650 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/cf217a21-c1bd-48be-b203-f59743bceb33-service-ca-bundle podName:cf217a21-c1bd-48be-b203-f59743bceb33 nodeName:}" failed. No retries permitted until 2026-11-03 00:40:48.216645171 +0000 UTC m=+38.162224648 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/cf217a21-c1bd-48be-b203-f59743bceb33-service-ca-bundle") pod "router-default-d7798ccf5-442kt" (UID: "cf217a21-c1bd-48be-b203-f59743bceb33") : configmap references non-existent config key: service-ca.crt Nov 03 00:40:44.216771 ip-10-0-140-107 kubenswrapper[2578]: E1103 00:40:44.216687 2578 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Nov 03 00:40:44.216771 ip-10-0-140-107 kubenswrapper[2578]: E1103 00:40:44.216694 2578 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-6669dc8887-ws544: secret "image-registry-tls" not found Nov 03 00:40:44.216771 ip-10-0-140-107 kubenswrapper[2578]: E1103 00:40:44.216710 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/52a19842-3750-4d2b-bcb2-aebddee81097-registry-tls podName:52a19842-3750-4d2b-bcb2-aebddee81097 nodeName:}" failed. No retries permitted until 2026-11-03 00:40:48.216705522 +0000 UTC m=+38.162285000 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/52a19842-3750-4d2b-bcb2-aebddee81097-registry-tls") pod "image-registry-6669dc8887-ws544" (UID: "52a19842-3750-4d2b-bcb2-aebddee81097") : secret "image-registry-tls" not found Nov 03 00:40:44.216771 ip-10-0-140-107 kubenswrapper[2578]: E1103 00:40:44.216755 2578 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found Nov 03 00:40:44.216881 ip-10-0-140-107 kubenswrapper[2578]: E1103 00:40:44.216774 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/136086c0-4846-437a-ad98-3cfef50610a0-metrics-certs podName:136086c0-4846-437a-ad98-3cfef50610a0 nodeName:}" failed. No retries permitted until 2026-11-03 00:41:16.216768318 +0000 UTC m=+66.162347794 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/136086c0-4846-437a-ad98-3cfef50610a0-metrics-certs") pod "network-metrics-daemon-rt2hx" (UID: "136086c0-4846-437a-ad98-3cfef50610a0") : secret "metrics-daemon-secret" not found Nov 03 00:40:44.317536 ip-10-0-140-107 kubenswrapper[2578]: I1103 00:40:44.317266 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/cabd402c-79c5-4699-9c85-d415d362b4d0-original-pull-secret\") pod \"global-pull-secret-syncer-jv7cw\" (UID: \"cabd402c-79c5-4699-9c85-d415d362b4d0\") " pod="kube-system/global-pull-secret-syncer-jv7cw" Nov 03 00:40:44.317536 ip-10-0-140-107 kubenswrapper[2578]: I1103 00:40:44.317351 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d8l25\" (UniqueName: \"kubernetes.io/projected/8cd3ca22-e4ce-4896-ae31-97605af4e4c6-kube-api-access-d8l25\") pod \"network-check-target-w46kd\" (UID: \"8cd3ca22-e4ce-4896-ae31-97605af4e4c6\") " pod="openshift-network-diagnostics/network-check-target-w46kd" Nov 03 00:40:44.320440 ip-10-0-140-107 kubenswrapper[2578]: I1103 00:40:44.320377 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/cabd402c-79c5-4699-9c85-d415d362b4d0-original-pull-secret\") pod \"global-pull-secret-syncer-jv7cw\" (UID: \"cabd402c-79c5-4699-9c85-d415d362b4d0\") " pod="kube-system/global-pull-secret-syncer-jv7cw" Nov 03 00:40:44.334787 ip-10-0-140-107 kubenswrapper[2578]: I1103 00:40:44.334759 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-d8l25\" (UniqueName: \"kubernetes.io/projected/8cd3ca22-e4ce-4896-ae31-97605af4e4c6-kube-api-access-d8l25\") pod \"network-check-target-w46kd\" (UID: \"8cd3ca22-e4ce-4896-ae31-97605af4e4c6\") " pod="openshift-network-diagnostics/network-check-target-w46kd" Nov 03 00:40:44.335511 ip-10-0-140-107 kubenswrapper[2578]: I1103 00:40:44.335440 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 00:40:44.617463 ip-10-0-140-107 kubenswrapper[2578]: I1103 00:40:44.617436 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-jv7cw" Nov 03 00:40:44.628310 ip-10-0-140-107 kubenswrapper[2578]: I1103 00:40:44.628249 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-w46kd" Nov 03 00:40:44.963864 ip-10-0-140-107 kubenswrapper[2578]: I1103 00:40:44.963768 2578 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-jv7cw"] Nov 03 00:40:44.963864 ip-10-0-140-107 kubenswrapper[2578]: I1103 00:40:44.963831 2578 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-w46kd"] Nov 03 00:40:44.966304 ip-10-0-140-107 kubenswrapper[2578]: W1103 00:40:44.966277 2578 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8cd3ca22_e4ce_4896_ae31_97605af4e4c6.slice/crio-a5114a1c8f2cd400e64362ba4b984bd6b439505b63ac0a12be3add35ec5a48e7 WatchSource:0}: Error finding container a5114a1c8f2cd400e64362ba4b984bd6b439505b63ac0a12be3add35ec5a48e7: Status 404 returned error can't find the container with id a5114a1c8f2cd400e64362ba4b984bd6b439505b63ac0a12be3add35ec5a48e7 Nov 03 00:40:44.972272 ip-10-0-140-107 kubenswrapper[2578]: W1103 00:40:44.972236 2578 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcabd402c_79c5_4699_9c85_d415d362b4d0.slice/crio-a7619cc5d7ec9ca7f22e4b1d299dfaec1ec361e238eaabfd63843c47d0bc74ea WatchSource:0}: Error finding container a7619cc5d7ec9ca7f22e4b1d299dfaec1ec361e238eaabfd63843c47d0bc74ea: Status 404 returned error can't find the container with id a7619cc5d7ec9ca7f22e4b1d299dfaec1ec361e238eaabfd63843c47d0bc74ea Nov 03 00:40:44.978393 ip-10-0-140-107 kubenswrapper[2578]: I1103 00:40:44.978373 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-n42s6_65d134ac-8a25-4c67-b6c7-cd752a098daf/console-operator/0.log" Nov 03 00:40:44.978499 ip-10-0-140-107 kubenswrapper[2578]: I1103 00:40:44.978410 2578 generic.go:356] "Generic (PLEG): container finished" podID="65d134ac-8a25-4c67-b6c7-cd752a098daf" containerID="456acf53d21314b5d37edf3c6d37c8c1d20e00673851be8128340827b88b73bd" exitCode=255 Nov 03 00:40:44.978744 ip-10-0-140-107 kubenswrapper[2578]: I1103 00:40:44.978722 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-7cd56b8dbc-n42s6" event={"ID":"65d134ac-8a25-4c67-b6c7-cd752a098daf","Type":"ContainerDied","Data":"456acf53d21314b5d37edf3c6d37c8c1d20e00673851be8128340827b88b73bd"} Nov 03 00:40:44.978744 ip-10-0-140-107 kubenswrapper[2578]: I1103 00:40:44.978735 2578 scope.go:122] "RemoveContainer" containerID="456acf53d21314b5d37edf3c6d37c8c1d20e00673851be8128340827b88b73bd" Nov 03 00:40:45.984901 ip-10-0-140-107 kubenswrapper[2578]: I1103 00:40:45.984862 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-khmv7" event={"ID":"36b9d258-cb2c-4d2b-880f-ae93b065034c","Type":"ContainerStarted","Data":"00b1e2838a30893cde164c35041c848e8bd12adedde9a1f06b814d763f2b70dd"} Nov 03 00:40:45.986686 ip-10-0-140-107 kubenswrapper[2578]: I1103 00:40:45.986660 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-7c4986d449-xprcf" event={"ID":"ca0f883c-c2b7-4bf6-a90b-ffdbeac161a8","Type":"ContainerStarted","Data":"90ac319dc159fa4199add901b66cd1e68b419e073b0586dd5ee7fdd7ce194c3d"} Nov 03 00:40:45.989810 ip-10-0-140-107 kubenswrapper[2578]: I1103 00:40:45.989790 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-n42s6_65d134ac-8a25-4c67-b6c7-cd752a098daf/console-operator/1.log" Nov 03 00:40:45.990329 ip-10-0-140-107 kubenswrapper[2578]: I1103 00:40:45.990311 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-n42s6_65d134ac-8a25-4c67-b6c7-cd752a098daf/console-operator/0.log" Nov 03 00:40:45.990411 ip-10-0-140-107 kubenswrapper[2578]: I1103 00:40:45.990347 2578 generic.go:356] "Generic (PLEG): container finished" podID="65d134ac-8a25-4c67-b6c7-cd752a098daf" containerID="8fbbed43b6d1c11a77523d9a59cdf45ffb9e7d1661d1a2290ee4a1ddd7dc0fb7" exitCode=255 Nov 03 00:40:45.990604 ip-10-0-140-107 kubenswrapper[2578]: I1103 00:40:45.990583 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-7cd56b8dbc-n42s6" event={"ID":"65d134ac-8a25-4c67-b6c7-cd752a098daf","Type":"ContainerDied","Data":"8fbbed43b6d1c11a77523d9a59cdf45ffb9e7d1661d1a2290ee4a1ddd7dc0fb7"} Nov 03 00:40:45.990673 ip-10-0-140-107 kubenswrapper[2578]: I1103 00:40:45.990638 2578 scope.go:122] "RemoveContainer" containerID="456acf53d21314b5d37edf3c6d37c8c1d20e00673851be8128340827b88b73bd" Nov 03 00:40:45.990673 ip-10-0-140-107 kubenswrapper[2578]: I1103 00:40:45.990659 2578 scope.go:122] "RemoveContainer" containerID="8fbbed43b6d1c11a77523d9a59cdf45ffb9e7d1661d1a2290ee4a1ddd7dc0fb7" Nov 03 00:40:45.990859 ip-10-0-140-107 kubenswrapper[2578]: E1103 00:40:45.990835 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-7cd56b8dbc-n42s6_openshift-console-operator(65d134ac-8a25-4c67-b6c7-cd752a098daf)\"" pod="openshift-console-operator/console-operator-7cd56b8dbc-n42s6" podUID="65d134ac-8a25-4c67-b6c7-cd752a098daf" Nov 03 00:40:45.992006 ip-10-0-140-107 kubenswrapper[2578]: I1103 00:40:45.991830 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-w46kd" event={"ID":"8cd3ca22-e4ce-4896-ae31-97605af4e4c6","Type":"ContainerStarted","Data":"a5114a1c8f2cd400e64362ba4b984bd6b439505b63ac0a12be3add35ec5a48e7"} Nov 03 00:40:45.992988 ip-10-0-140-107 kubenswrapper[2578]: I1103 00:40:45.992967 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-jv7cw" event={"ID":"cabd402c-79c5-4699-9c85-d415d362b4d0","Type":"ContainerStarted","Data":"a7619cc5d7ec9ca7f22e4b1d299dfaec1ec361e238eaabfd63843c47d0bc74ea"} Nov 03 00:40:46.007241 ip-10-0-140-107 kubenswrapper[2578]: I1103 00:40:46.006314 2578 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-khmv7" podStartSLOduration=331.312740576 podStartE2EDuration="5m36.006302486s" totalImagesPullingTime="4.69356191s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:35:10 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:40:46.003807426 +0000 UTC m=+35.949386927" watchObservedRunningTime="2026-11-03 00:40:46.006302486 +0000 UTC m=+35.951881985" Nov 03 00:40:46.053110 ip-10-0-140-107 kubenswrapper[2578]: I1103 00:40:46.053022 2578 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-insights/insights-operator-7c4986d449-xprcf" podStartSLOduration=337.408109857 podStartE2EDuration="5m42.053004155s" totalImagesPullingTime="4.644894298s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:35:04 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:40:46.049886671 +0000 UTC m=+35.995466172" watchObservedRunningTime="2026-11-03 00:40:46.053004155 +0000 UTC m=+35.998583656" Nov 03 00:40:46.212943 ip-10-0-140-107 kubenswrapper[2578]: I1103 00:40:46.212887 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 00:40:46.361654 ip-10-0-140-107 kubenswrapper[2578]: I1103 00:40:46.361501 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 00:40:46.677851 ip-10-0-140-107 kubenswrapper[2578]: I1103 00:40:46.676562 2578 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-kube-storage-version-migrator/migrator-66df5d496b-m7cxm"] Nov 03 00:40:46.684525 ip-10-0-140-107 kubenswrapper[2578]: I1103 00:40:46.684482 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 00:40:46.694362 ip-10-0-140-107 kubenswrapper[2578]: I1103 00:40:46.693161 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-m7cxm" Nov 03 00:40:46.696759 ip-10-0-140-107 kubenswrapper[2578]: I1103 00:40:46.696698 2578 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-66df5d496b-m7cxm"] Nov 03 00:40:46.745134 ip-10-0-140-107 kubenswrapper[2578]: I1103 00:40:46.745100 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7wzbd\" (UniqueName: \"kubernetes.io/projected/68658033-8a49-4430-869f-a7eb79603484-kube-api-access-7wzbd\") pod \"migrator-66df5d496b-m7cxm\" (UID: \"68658033-8a49-4430-869f-a7eb79603484\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-m7cxm" Nov 03 00:40:46.786971 ip-10-0-140-107 kubenswrapper[2578]: I1103 00:40:46.786936 2578 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-sthjh"] Nov 03 00:40:46.792687 ip-10-0-140-107 kubenswrapper[2578]: I1103 00:40:46.792643 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"openshift-service-ca.crt\"" Nov 03 00:40:46.792840 ip-10-0-140-107 kubenswrapper[2578]: I1103 00:40:46.792780 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-root-ca.crt\"" Nov 03 00:40:46.793086 ip-10-0-140-107 kubenswrapper[2578]: I1103 00:40:46.793063 2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-storage-version-migrator-sa-dockercfg-dx4w2\"" Nov 03 00:40:46.812063 ip-10-0-140-107 kubenswrapper[2578]: I1103 00:40:46.811485 2578 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-sthjh"] Nov 03 00:40:46.812063 ip-10-0-140-107 kubenswrapper[2578]: I1103 00:40:46.811615 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-sthjh" Nov 03 00:40:46.846425 ip-10-0-140-107 kubenswrapper[2578]: I1103 00:40:46.846393 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/945da51a-32cb-4769-900d-1aeb266c7565-data-volume\") pod \"insights-runtime-extractor-sthjh\" (UID: \"945da51a-32cb-4769-900d-1aeb266c7565\") " pod="openshift-insights/insights-runtime-extractor-sthjh" Nov 03 00:40:46.846563 ip-10-0-140-107 kubenswrapper[2578]: I1103 00:40:46.846441 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/945da51a-32cb-4769-900d-1aeb266c7565-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-sthjh\" (UID: \"945da51a-32cb-4769-900d-1aeb266c7565\") " pod="openshift-insights/insights-runtime-extractor-sthjh" Nov 03 00:40:46.846563 ip-10-0-140-107 kubenswrapper[2578]: I1103 00:40:46.846530 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/945da51a-32cb-4769-900d-1aeb266c7565-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-sthjh\" (UID: \"945da51a-32cb-4769-900d-1aeb266c7565\") " pod="openshift-insights/insights-runtime-extractor-sthjh" Nov 03 00:40:46.846675 ip-10-0-140-107 kubenswrapper[2578]: I1103 00:40:46.846562 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/945da51a-32cb-4769-900d-1aeb266c7565-crio-socket\") pod \"insights-runtime-extractor-sthjh\" (UID: \"945da51a-32cb-4769-900d-1aeb266c7565\") " pod="openshift-insights/insights-runtime-extractor-sthjh" Nov 03 00:40:46.846675 ip-10-0-140-107 kubenswrapper[2578]: I1103 00:40:46.846603 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7wzbd\" (UniqueName: \"kubernetes.io/projected/68658033-8a49-4430-869f-a7eb79603484-kube-api-access-7wzbd\") pod \"migrator-66df5d496b-m7cxm\" (UID: \"68658033-8a49-4430-869f-a7eb79603484\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-m7cxm" Nov 03 00:40:46.846675 ip-10-0-140-107 kubenswrapper[2578]: I1103 00:40:46.846632 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4q9wl\" (UniqueName: \"kubernetes.io/projected/945da51a-32cb-4769-900d-1aeb266c7565-kube-api-access-4q9wl\") pod \"insights-runtime-extractor-sthjh\" (UID: \"945da51a-32cb-4769-900d-1aeb266c7565\") " pod="openshift-insights/insights-runtime-extractor-sthjh" Nov 03 00:40:46.856553 ip-10-0-140-107 kubenswrapper[2578]: I1103 00:40:46.856527 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7wzbd\" (UniqueName: \"kubernetes.io/projected/68658033-8a49-4430-869f-a7eb79603484-kube-api-access-7wzbd\") pod \"migrator-66df5d496b-m7cxm\" (UID: \"68658033-8a49-4430-869f-a7eb79603484\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-m7cxm" Nov 03 00:40:46.857005 ip-10-0-140-107 kubenswrapper[2578]: I1103 00:40:46.856977 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 00:40:46.890187 ip-10-0-140-107 kubenswrapper[2578]: I1103 00:40:46.890164 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\"" Nov 03 00:40:46.893000 ip-10-0-140-107 kubenswrapper[2578]: I1103 00:40:46.892785 2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\"" Nov 03 00:40:46.893000 ip-10-0-140-107 kubenswrapper[2578]: I1103 00:40:46.892956 2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-42dxl\"" Nov 03 00:40:46.947563 ip-10-0-140-107 kubenswrapper[2578]: I1103 00:40:46.947536 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/945da51a-32cb-4769-900d-1aeb266c7565-data-volume\") pod \"insights-runtime-extractor-sthjh\" (UID: \"945da51a-32cb-4769-900d-1aeb266c7565\") " pod="openshift-insights/insights-runtime-extractor-sthjh" Nov 03 00:40:46.947717 ip-10-0-140-107 kubenswrapper[2578]: I1103 00:40:46.947576 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/945da51a-32cb-4769-900d-1aeb266c7565-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-sthjh\" (UID: \"945da51a-32cb-4769-900d-1aeb266c7565\") " pod="openshift-insights/insights-runtime-extractor-sthjh" Nov 03 00:40:46.947717 ip-10-0-140-107 kubenswrapper[2578]: I1103 00:40:46.947650 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/945da51a-32cb-4769-900d-1aeb266c7565-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-sthjh\" (UID: \"945da51a-32cb-4769-900d-1aeb266c7565\") " pod="openshift-insights/insights-runtime-extractor-sthjh" Nov 03 00:40:46.947717 ip-10-0-140-107 kubenswrapper[2578]: I1103 00:40:46.947680 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/945da51a-32cb-4769-900d-1aeb266c7565-crio-socket\") pod \"insights-runtime-extractor-sthjh\" (UID: \"945da51a-32cb-4769-900d-1aeb266c7565\") " pod="openshift-insights/insights-runtime-extractor-sthjh" Nov 03 00:40:46.947864 ip-10-0-140-107 kubenswrapper[2578]: E1103 00:40:46.947733 2578 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Nov 03 00:40:46.947864 ip-10-0-140-107 kubenswrapper[2578]: E1103 00:40:46.947823 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/945da51a-32cb-4769-900d-1aeb266c7565-kube-rbac-proxy-cm podName:945da51a-32cb-4769-900d-1aeb266c7565 nodeName:}" failed. No retries permitted until 2026-11-03 00:40:47.447798162 +0000 UTC m=+37.393377653 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/945da51a-32cb-4769-900d-1aeb266c7565-kube-rbac-proxy-cm") pod "insights-runtime-extractor-sthjh" (UID: "945da51a-32cb-4769-900d-1aeb266c7565") : configmap "kube-rbac-proxy" not found Nov 03 00:40:46.947994 ip-10-0-140-107 kubenswrapper[2578]: I1103 00:40:46.947898 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/945da51a-32cb-4769-900d-1aeb266c7565-data-volume\") pod \"insights-runtime-extractor-sthjh\" (UID: \"945da51a-32cb-4769-900d-1aeb266c7565\") " pod="openshift-insights/insights-runtime-extractor-sthjh" Nov 03 00:40:46.947994 ip-10-0-140-107 kubenswrapper[2578]: I1103 00:40:46.947745 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4q9wl\" (UniqueName: \"kubernetes.io/projected/945da51a-32cb-4769-900d-1aeb266c7565-kube-api-access-4q9wl\") pod \"insights-runtime-extractor-sthjh\" (UID: \"945da51a-32cb-4769-900d-1aeb266c7565\") " pod="openshift-insights/insights-runtime-extractor-sthjh" Nov 03 00:40:46.948092 ip-10-0-140-107 kubenswrapper[2578]: I1103 00:40:46.947996 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/945da51a-32cb-4769-900d-1aeb266c7565-crio-socket\") pod \"insights-runtime-extractor-sthjh\" (UID: \"945da51a-32cb-4769-900d-1aeb266c7565\") " pod="openshift-insights/insights-runtime-extractor-sthjh" Nov 03 00:40:46.948092 ip-10-0-140-107 kubenswrapper[2578]: E1103 00:40:46.948021 2578 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Nov 03 00:40:46.948092 ip-10-0-140-107 kubenswrapper[2578]: E1103 00:40:46.948076 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/945da51a-32cb-4769-900d-1aeb266c7565-insights-runtime-extractor-tls podName:945da51a-32cb-4769-900d-1aeb266c7565 nodeName:}" failed. No retries permitted until 2026-11-03 00:40:47.448063516 +0000 UTC m=+37.393643021 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/945da51a-32cb-4769-900d-1aeb266c7565-insights-runtime-extractor-tls") pod "insights-runtime-extractor-sthjh" (UID: "945da51a-32cb-4769-900d-1aeb266c7565") : secret "insights-runtime-extractor-tls" not found Nov 03 00:40:46.958202 ip-10-0-140-107 kubenswrapper[2578]: I1103 00:40:46.958173 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4q9wl\" (UniqueName: \"kubernetes.io/projected/945da51a-32cb-4769-900d-1aeb266c7565-kube-api-access-4q9wl\") pod \"insights-runtime-extractor-sthjh\" (UID: \"945da51a-32cb-4769-900d-1aeb266c7565\") " pod="openshift-insights/insights-runtime-extractor-sthjh" Nov 03 00:40:46.991091 ip-10-0-140-107 kubenswrapper[2578]: I1103 00:40:46.991058 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 00:40:46.999936 ip-10-0-140-107 kubenswrapper[2578]: I1103 00:40:46.999687 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-s9hbz" event={"ID":"75481915-3cbe-475b-abe7-cb4e81c93427","Type":"ContainerStarted","Data":"1bdb6bd8cf6dd207bc1dbbee7b19db5aac76cfb6b7c1391fdbdbe23d38b1758d"} Nov 03 00:40:47.003945 ip-10-0-140-107 kubenswrapper[2578]: I1103 00:40:47.003742 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-7lqbv" event={"ID":"2b050b18-31f6-4a22-9902-0435d11018b9","Type":"ContainerStarted","Data":"c1e2afb719a4e07feb8109267cac6124f4dca189e377407a93c4e97946eb5255"} Nov 03 00:40:47.010965 ip-10-0-140-107 kubenswrapper[2578]: I1103 00:40:47.006777 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-n42s6_65d134ac-8a25-4c67-b6c7-cd752a098daf/console-operator/1.log" Nov 03 00:40:47.010965 ip-10-0-140-107 kubenswrapper[2578]: I1103 00:40:47.007150 2578 scope.go:122] "RemoveContainer" containerID="8fbbed43b6d1c11a77523d9a59cdf45ffb9e7d1661d1a2290ee4a1ddd7dc0fb7" Nov 03 00:40:47.010965 ip-10-0-140-107 kubenswrapper[2578]: E1103 00:40:47.007312 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-7cd56b8dbc-n42s6_openshift-console-operator(65d134ac-8a25-4c67-b6c7-cd752a098daf)\"" pod="openshift-console-operator/console-operator-7cd56b8dbc-n42s6" podUID="65d134ac-8a25-4c67-b6c7-cd752a098daf" Nov 03 00:40:47.010965 ip-10-0-140-107 kubenswrapper[2578]: I1103 00:40:47.010486 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-source-6f78d59f99-z5f9s" event={"ID":"525dc47d-afde-4019-ad01-80614291f282","Type":"ContainerStarted","Data":"2f000bcd035e3c6dc11e792cd5d6cfb9aa6fb71d418a053d7213485af99a2c13"} Nov 03 00:40:47.021296 ip-10-0-140-107 kubenswrapper[2578]: I1103 00:40:47.021246 2578 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-s9hbz" podStartSLOduration=1.87923237 podStartE2EDuration="7.021234318s" totalImagesPullingTime="5.142001948s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:40:40 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:40:47.019102796 +0000 UTC m=+36.964682297" watchObservedRunningTime="2026-11-03 00:40:47.021234318 +0000 UTC m=+36.966813858" Nov 03 00:40:47.042461 ip-10-0-140-107 kubenswrapper[2578]: I1103 00:40:47.042266 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-m7cxm" Nov 03 00:40:47.056472 ip-10-0-140-107 kubenswrapper[2578]: I1103 00:40:47.056415 2578 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-source-6f78d59f99-z5f9s" podStartSLOduration=222.402698592 podStartE2EDuration="3m48.056402955s" totalImagesPullingTime="5.653704363s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:36:59 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:40:47.038486753 +0000 UTC m=+36.984066254" watchObservedRunningTime="2026-11-03 00:40:47.056402955 +0000 UTC m=+37.001982514" Nov 03 00:40:47.085438 ip-10-0-140-107 kubenswrapper[2578]: I1103 00:40:47.084817 2578 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-7lqbv" podStartSLOduration=337.938838865 podStartE2EDuration="5m43.084805304s" totalImagesPullingTime="5.145966439s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:35:04 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:40:47.058503732 +0000 UTC m=+37.004083231" watchObservedRunningTime="2026-11-03 00:40:47.084805304 +0000 UTC m=+37.030384803" Nov 03 00:40:47.195405 ip-10-0-140-107 kubenswrapper[2578]: I1103 00:40:47.195259 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 00:40:47.377730 ip-10-0-140-107 kubenswrapper[2578]: I1103 00:40:47.377702 2578 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-66df5d496b-m7cxm"] Nov 03 00:40:47.381901 ip-10-0-140-107 kubenswrapper[2578]: I1103 00:40:47.381846 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 00:40:47.389842 ip-10-0-140-107 kubenswrapper[2578]: W1103 00:40:47.389810 2578 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod68658033_8a49_4430_869f_a7eb79603484.slice/crio-2b3856ef34d4cbba8d89162e727bb1a5ed2a806920a8e038711a17170afd7bc6 WatchSource:0}: Error finding container 2b3856ef34d4cbba8d89162e727bb1a5ed2a806920a8e038711a17170afd7bc6: Status 404 returned error can't find the container with id 2b3856ef34d4cbba8d89162e727bb1a5ed2a806920a8e038711a17170afd7bc6 Nov 03 00:40:47.453118 ip-10-0-140-107 kubenswrapper[2578]: I1103 00:40:47.453081 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/945da51a-32cb-4769-900d-1aeb266c7565-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-sthjh\" (UID: \"945da51a-32cb-4769-900d-1aeb266c7565\") " pod="openshift-insights/insights-runtime-extractor-sthjh" Nov 03 00:40:47.453279 ip-10-0-140-107 kubenswrapper[2578]: E1103 00:40:47.453231 2578 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Nov 03 00:40:47.453348 ip-10-0-140-107 kubenswrapper[2578]: E1103 00:40:47.453303 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/945da51a-32cb-4769-900d-1aeb266c7565-kube-rbac-proxy-cm podName:945da51a-32cb-4769-900d-1aeb266c7565 nodeName:}" failed. No retries permitted until 2026-11-03 00:40:48.45328466 +0000 UTC m=+38.398864190 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/945da51a-32cb-4769-900d-1aeb266c7565-kube-rbac-proxy-cm") pod "insights-runtime-extractor-sthjh" (UID: "945da51a-32cb-4769-900d-1aeb266c7565") : configmap "kube-rbac-proxy" not found Nov 03 00:40:47.453348 ip-10-0-140-107 kubenswrapper[2578]: I1103 00:40:47.453235 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/945da51a-32cb-4769-900d-1aeb266c7565-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-sthjh\" (UID: \"945da51a-32cb-4769-900d-1aeb266c7565\") " pod="openshift-insights/insights-runtime-extractor-sthjh" Nov 03 00:40:47.453454 ip-10-0-140-107 kubenswrapper[2578]: E1103 00:40:47.453367 2578 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Nov 03 00:40:47.453454 ip-10-0-140-107 kubenswrapper[2578]: E1103 00:40:47.453424 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/945da51a-32cb-4769-900d-1aeb266c7565-insights-runtime-extractor-tls podName:945da51a-32cb-4769-900d-1aeb266c7565 nodeName:}" failed. No retries permitted until 2026-11-03 00:40:48.453407088 +0000 UTC m=+38.398986589 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/945da51a-32cb-4769-900d-1aeb266c7565-insights-runtime-extractor-tls") pod "insights-runtime-extractor-sthjh" (UID: "945da51a-32cb-4769-900d-1aeb266c7565") : secret "insights-runtime-extractor-tls" not found Nov 03 00:40:47.664748 ip-10-0-140-107 kubenswrapper[2578]: I1103 00:40:47.664666 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 00:40:47.801617 ip-10-0-140-107 kubenswrapper[2578]: I1103 00:40:47.801581 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 00:40:48.016098 ip-10-0-140-107 kubenswrapper[2578]: I1103 00:40:48.015965 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-m7cxm" event={"ID":"68658033-8a49-4430-869f-a7eb79603484","Type":"ContainerStarted","Data":"2b3856ef34d4cbba8d89162e727bb1a5ed2a806920a8e038711a17170afd7bc6"} Nov 03 00:40:48.263489 ip-10-0-140-107 kubenswrapper[2578]: I1103 00:40:48.262068 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/69b087f8-4a23-4ec8-808e-b61d2e16f67b-metrics-tls\") pod \"dns-default-266ht\" (UID: \"69b087f8-4a23-4ec8-808e-b61d2e16f67b\") " pod="openshift-dns/dns-default-266ht" Nov 03 00:40:48.263489 ip-10-0-140-107 kubenswrapper[2578]: E1103 00:40:48.262229 2578 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Nov 03 00:40:48.263489 ip-10-0-140-107 kubenswrapper[2578]: E1103 00:40:48.262291 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cf217a21-c1bd-48be-b203-f59743bceb33-metrics-certs podName:cf217a21-c1bd-48be-b203-f59743bceb33 nodeName:}" failed. No retries permitted until 2026-11-03 00:40:56.262277466 +0000 UTC m=+46.207856962 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/cf217a21-c1bd-48be-b203-f59743bceb33-metrics-certs") pod "router-default-d7798ccf5-442kt" (UID: "cf217a21-c1bd-48be-b203-f59743bceb33") : secret "router-metrics-certs-default" not found Nov 03 00:40:48.263489 ip-10-0-140-107 kubenswrapper[2578]: E1103 00:40:48.262682 2578 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Nov 03 00:40:48.263489 ip-10-0-140-107 kubenswrapper[2578]: E1103 00:40:48.262728 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/69b087f8-4a23-4ec8-808e-b61d2e16f67b-metrics-tls podName:69b087f8-4a23-4ec8-808e-b61d2e16f67b nodeName:}" failed. No retries permitted until 2026-11-03 00:40:56.262713201 +0000 UTC m=+46.208292690 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/69b087f8-4a23-4ec8-808e-b61d2e16f67b-metrics-tls") pod "dns-default-266ht" (UID: "69b087f8-4a23-4ec8-808e-b61d2e16f67b") : secret "dns-default-metrics-tls" not found Nov 03 00:40:48.263489 ip-10-0-140-107 kubenswrapper[2578]: I1103 00:40:48.262750 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cf217a21-c1bd-48be-b203-f59743bceb33-metrics-certs\") pod \"router-default-d7798ccf5-442kt\" (UID: \"cf217a21-c1bd-48be-b203-f59743bceb33\") " pod="openshift-ingress/router-default-d7798ccf5-442kt" Nov 03 00:40:48.263489 ip-10-0-140-107 kubenswrapper[2578]: I1103 00:40:48.262786 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/dd0cd8a1-4d6b-424c-ac39-5f5ddb9cd87e-registry-tls\") pod \"image-registry-65f77c855d-6tnc2\" (UID: \"dd0cd8a1-4d6b-424c-ac39-5f5ddb9cd87e\") " pod="openshift-image-registry/image-registry-65f77c855d-6tnc2" Nov 03 00:40:48.263489 ip-10-0-140-107 kubenswrapper[2578]: I1103 00:40:48.262823 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/06973556-f51c-4d1c-ae55-fb77699bc5b3-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-4bm4f\" (UID: \"06973556-f51c-4d1c-ae55-fb77699bc5b3\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-4bm4f" Nov 03 00:40:48.263489 ip-10-0-140-107 kubenswrapper[2578]: I1103 00:40:48.262846 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/6eafe6f6-cf90-446f-a8d5-620cd2e02775-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-2ngwx\" (UID: \"6eafe6f6-cf90-446f-a8d5-620cd2e02775\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-2ngwx" Nov 03 00:40:48.263489 ip-10-0-140-107 kubenswrapper[2578]: I1103 00:40:48.262867 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/cf217a21-c1bd-48be-b203-f59743bceb33-service-ca-bundle\") pod \"router-default-d7798ccf5-442kt\" (UID: \"cf217a21-c1bd-48be-b203-f59743bceb33\") " pod="openshift-ingress/router-default-d7798ccf5-442kt" Nov 03 00:40:48.263489 ip-10-0-140-107 kubenswrapper[2578]: I1103 00:40:48.262926 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/52a19842-3750-4d2b-bcb2-aebddee81097-registry-tls\") pod \"image-registry-6669dc8887-ws544\" (UID: \"52a19842-3750-4d2b-bcb2-aebddee81097\") " pod="openshift-image-registry/image-registry-6669dc8887-ws544" Nov 03 00:40:48.263489 ip-10-0-140-107 kubenswrapper[2578]: I1103 00:40:48.262967 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/bc513cba-d924-4719-a2ba-711529175ee0-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-z2qgj\" (UID: \"bc513cba-d924-4719-a2ba-711529175ee0\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-z2qgj" Nov 03 00:40:48.263489 ip-10-0-140-107 kubenswrapper[2578]: E1103 00:40:48.263071 2578 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Nov 03 00:40:48.263489 ip-10-0-140-107 kubenswrapper[2578]: E1103 00:40:48.263105 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/bc513cba-d924-4719-a2ba-711529175ee0-networking-console-plugin-cert podName:bc513cba-d924-4719-a2ba-711529175ee0 nodeName:}" failed. No retries permitted until 2026-11-03 00:40:56.263093713 +0000 UTC m=+46.208673204 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/bc513cba-d924-4719-a2ba-711529175ee0-networking-console-plugin-cert") pod "networking-console-plugin-86c478bb88-z2qgj" (UID: "bc513cba-d924-4719-a2ba-711529175ee0") : secret "networking-console-plugin-cert" not found Nov 03 00:40:48.263489 ip-10-0-140-107 kubenswrapper[2578]: E1103 00:40:48.263156 2578 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Nov 03 00:40:48.263489 ip-10-0-140-107 kubenswrapper[2578]: E1103 00:40:48.263165 2578 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-65f77c855d-6tnc2: secret "image-registry-tls" not found Nov 03 00:40:48.264263 ip-10-0-140-107 kubenswrapper[2578]: E1103 00:40:48.263192 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/dd0cd8a1-4d6b-424c-ac39-5f5ddb9cd87e-registry-tls podName:dd0cd8a1-4d6b-424c-ac39-5f5ddb9cd87e nodeName:}" failed. No retries permitted until 2026-11-03 00:40:56.263184557 +0000 UTC m=+46.208764048 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/dd0cd8a1-4d6b-424c-ac39-5f5ddb9cd87e-registry-tls") pod "image-registry-65f77c855d-6tnc2" (UID: "dd0cd8a1-4d6b-424c-ac39-5f5ddb9cd87e") : secret "image-registry-tls" not found Nov 03 00:40:48.264263 ip-10-0-140-107 kubenswrapper[2578]: E1103 00:40:48.263238 2578 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Nov 03 00:40:48.264263 ip-10-0-140-107 kubenswrapper[2578]: E1103 00:40:48.263264 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/06973556-f51c-4d1c-ae55-fb77699bc5b3-cluster-monitoring-operator-tls podName:06973556-f51c-4d1c-ae55-fb77699bc5b3 nodeName:}" failed. No retries permitted until 2026-11-03 00:40:56.263257033 +0000 UTC m=+46.208836524 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/06973556-f51c-4d1c-ae55-fb77699bc5b3-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-884576c57-4bm4f" (UID: "06973556-f51c-4d1c-ae55-fb77699bc5b3") : secret "cluster-monitoring-operator-tls" not found Nov 03 00:40:48.264263 ip-10-0-140-107 kubenswrapper[2578]: E1103 00:40:48.263310 2578 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Nov 03 00:40:48.264263 ip-10-0-140-107 kubenswrapper[2578]: E1103 00:40:48.263334 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6eafe6f6-cf90-446f-a8d5-620cd2e02775-samples-operator-tls podName:6eafe6f6-cf90-446f-a8d5-620cd2e02775 nodeName:}" failed. No retries permitted until 2026-11-03 00:40:56.263327 +0000 UTC m=+46.208906492 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/6eafe6f6-cf90-446f-a8d5-620cd2e02775-samples-operator-tls") pod "cluster-samples-operator-6ddb6c8b94-2ngwx" (UID: "6eafe6f6-cf90-446f-a8d5-620cd2e02775") : secret "samples-operator-tls" not found Nov 03 00:40:48.264263 ip-10-0-140-107 kubenswrapper[2578]: E1103 00:40:48.263388 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/cf217a21-c1bd-48be-b203-f59743bceb33-service-ca-bundle podName:cf217a21-c1bd-48be-b203-f59743bceb33 nodeName:}" failed. No retries permitted until 2026-11-03 00:40:56.263379981 +0000 UTC m=+46.208959472 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/cf217a21-c1bd-48be-b203-f59743bceb33-service-ca-bundle") pod "router-default-d7798ccf5-442kt" (UID: "cf217a21-c1bd-48be-b203-f59743bceb33") : configmap references non-existent config key: service-ca.crt Nov 03 00:40:48.264263 ip-10-0-140-107 kubenswrapper[2578]: E1103 00:40:48.263433 2578 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Nov 03 00:40:48.264263 ip-10-0-140-107 kubenswrapper[2578]: E1103 00:40:48.263440 2578 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-6669dc8887-ws544: secret "image-registry-tls" not found Nov 03 00:40:48.264263 ip-10-0-140-107 kubenswrapper[2578]: E1103 00:40:48.263462 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/52a19842-3750-4d2b-bcb2-aebddee81097-registry-tls podName:52a19842-3750-4d2b-bcb2-aebddee81097 nodeName:}" failed. No retries permitted until 2026-11-03 00:40:56.263454582 +0000 UTC m=+46.209034073 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/52a19842-3750-4d2b-bcb2-aebddee81097-registry-tls") pod "image-registry-6669dc8887-ws544" (UID: "52a19842-3750-4d2b-bcb2-aebddee81097") : secret "image-registry-tls" not found Nov 03 00:40:48.465429 ip-10-0-140-107 kubenswrapper[2578]: I1103 00:40:48.465393 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/945da51a-32cb-4769-900d-1aeb266c7565-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-sthjh\" (UID: \"945da51a-32cb-4769-900d-1aeb266c7565\") " pod="openshift-insights/insights-runtime-extractor-sthjh" Nov 03 00:40:48.465582 ip-10-0-140-107 kubenswrapper[2578]: I1103 00:40:48.465459 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/945da51a-32cb-4769-900d-1aeb266c7565-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-sthjh\" (UID: \"945da51a-32cb-4769-900d-1aeb266c7565\") " pod="openshift-insights/insights-runtime-extractor-sthjh" Nov 03 00:40:48.465582 ip-10-0-140-107 kubenswrapper[2578]: E1103 00:40:48.465522 2578 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Nov 03 00:40:48.465582 ip-10-0-140-107 kubenswrapper[2578]: E1103 00:40:48.465567 2578 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Nov 03 00:40:48.465682 ip-10-0-140-107 kubenswrapper[2578]: E1103 00:40:48.465595 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/945da51a-32cb-4769-900d-1aeb266c7565-kube-rbac-proxy-cm podName:945da51a-32cb-4769-900d-1aeb266c7565 nodeName:}" failed. No retries permitted until 2026-11-03 00:40:50.465578945 +0000 UTC m=+40.411158443 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/945da51a-32cb-4769-900d-1aeb266c7565-kube-rbac-proxy-cm") pod "insights-runtime-extractor-sthjh" (UID: "945da51a-32cb-4769-900d-1aeb266c7565") : configmap "kube-rbac-proxy" not found Nov 03 00:40:48.465682 ip-10-0-140-107 kubenswrapper[2578]: E1103 00:40:48.465611 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/945da51a-32cb-4769-900d-1aeb266c7565-insights-runtime-extractor-tls podName:945da51a-32cb-4769-900d-1aeb266c7565 nodeName:}" failed. No retries permitted until 2026-11-03 00:40:50.465605289 +0000 UTC m=+40.411184766 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/945da51a-32cb-4769-900d-1aeb266c7565-insights-runtime-extractor-tls") pod "insights-runtime-extractor-sthjh" (UID: "945da51a-32cb-4769-900d-1aeb266c7565") : secret "insights-runtime-extractor-tls" not found Nov 03 00:40:48.827054 ip-10-0-140-107 kubenswrapper[2578]: I1103 00:40:48.826980 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 00:40:49.020473 ip-10-0-140-107 kubenswrapper[2578]: I1103 00:40:49.020433 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-w46kd" event={"ID":"8cd3ca22-e4ce-4896-ae31-97605af4e4c6","Type":"ContainerStarted","Data":"a41fe73d0340be078ba05a0d8e1f8f7a0649396575fd564d380ed544ea0b4808"} Nov 03 00:40:49.020899 ip-10-0-140-107 kubenswrapper[2578]: I1103 00:40:49.020528 2578 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-w46kd" Nov 03 00:40:49.041252 ip-10-0-140-107 kubenswrapper[2578]: I1103 00:40:49.041202 2578 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-w46kd" podStartSLOduration=35.479135587 podStartE2EDuration="39.041185744s" totalImagesPullingTime="3.562050157s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:40:10 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:40:49.038933708 +0000 UTC m=+38.984513205" watchObservedRunningTime="2026-11-03 00:40:49.041185744 +0000 UTC m=+38.986765274" Nov 03 00:40:50.025472 ip-10-0-140-107 kubenswrapper[2578]: I1103 00:40:50.025435 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-m7cxm" event={"ID":"68658033-8a49-4430-869f-a7eb79603484","Type":"ContainerStarted","Data":"d390645b88c24cdc073b62e8a81d019140e79bdd39e27c27deabfa12d6363217"} Nov 03 00:40:50.044865 ip-10-0-140-107 kubenswrapper[2578]: I1103 00:40:50.044817 2578 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-m7cxm" podStartSLOduration=2.244629333 podStartE2EDuration="4.044801361s" totalImagesPullingTime="1.800172028s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:40:46 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:40:50.043495513 +0000 UTC m=+39.989075015" watchObservedRunningTime="2026-11-03 00:40:50.044801361 +0000 UTC m=+39.990380860" Nov 03 00:40:50.486392 ip-10-0-140-107 kubenswrapper[2578]: I1103 00:40:50.486354 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/945da51a-32cb-4769-900d-1aeb266c7565-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-sthjh\" (UID: \"945da51a-32cb-4769-900d-1aeb266c7565\") " pod="openshift-insights/insights-runtime-extractor-sthjh" Nov 03 00:40:50.486575 ip-10-0-140-107 kubenswrapper[2578]: I1103 00:40:50.486450 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/945da51a-32cb-4769-900d-1aeb266c7565-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-sthjh\" (UID: \"945da51a-32cb-4769-900d-1aeb266c7565\") " pod="openshift-insights/insights-runtime-extractor-sthjh" Nov 03 00:40:50.486575 ip-10-0-140-107 kubenswrapper[2578]: E1103 00:40:50.486466 2578 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Nov 03 00:40:50.486575 ip-10-0-140-107 kubenswrapper[2578]: E1103 00:40:50.486530 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/945da51a-32cb-4769-900d-1aeb266c7565-kube-rbac-proxy-cm podName:945da51a-32cb-4769-900d-1aeb266c7565 nodeName:}" failed. No retries permitted until 2026-11-03 00:40:54.486511553 +0000 UTC m=+44.432091053 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/945da51a-32cb-4769-900d-1aeb266c7565-kube-rbac-proxy-cm") pod "insights-runtime-extractor-sthjh" (UID: "945da51a-32cb-4769-900d-1aeb266c7565") : configmap "kube-rbac-proxy" not found Nov 03 00:40:50.486575 ip-10-0-140-107 kubenswrapper[2578]: E1103 00:40:50.486564 2578 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Nov 03 00:40:50.486791 ip-10-0-140-107 kubenswrapper[2578]: E1103 00:40:50.486603 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/945da51a-32cb-4769-900d-1aeb266c7565-insights-runtime-extractor-tls podName:945da51a-32cb-4769-900d-1aeb266c7565 nodeName:}" failed. No retries permitted until 2026-11-03 00:40:54.486592432 +0000 UTC m=+44.432171914 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/945da51a-32cb-4769-900d-1aeb266c7565-insights-runtime-extractor-tls") pod "insights-runtime-extractor-sthjh" (UID: "945da51a-32cb-4769-900d-1aeb266c7565") : secret "insights-runtime-extractor-tls" not found Nov 03 00:40:50.818633 ip-10-0-140-107 kubenswrapper[2578]: I1103 00:40:50.818554 2578 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-7cd56b8dbc-n42s6" Nov 03 00:40:50.818633 ip-10-0-140-107 kubenswrapper[2578]: I1103 00:40:50.818622 2578 kubelet.go:2727] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="openshift-console-operator/console-operator-7cd56b8dbc-n42s6" Nov 03 00:40:50.819054 ip-10-0-140-107 kubenswrapper[2578]: I1103 00:40:50.819038 2578 scope.go:122] "RemoveContainer" containerID="8fbbed43b6d1c11a77523d9a59cdf45ffb9e7d1661d1a2290ee4a1ddd7dc0fb7" Nov 03 00:40:50.819292 ip-10-0-140-107 kubenswrapper[2578]: E1103 00:40:50.819269 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-7cd56b8dbc-n42s6_openshift-console-operator(65d134ac-8a25-4c67-b6c7-cd752a098daf)\"" pod="openshift-console-operator/console-operator-7cd56b8dbc-n42s6" podUID="65d134ac-8a25-4c67-b6c7-cd752a098daf" Nov 03 00:40:51.667118 ip-10-0-140-107 kubenswrapper[2578]: I1103 00:40:51.666940 2578 dynamic_cafile_content.go:175] "Shutting down controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Nov 03 00:40:51.667140 ip-10-0-140-107 systemd[1]: Stopping Kubernetes Kubelet... Nov 03 00:40:51.685421 ip-10-0-140-107 systemd[1]: kubelet.service: Deactivated successfully. Nov 03 00:40:51.685673 ip-10-0-140-107 systemd[1]: Stopped Kubernetes Kubelet. Nov 03 00:40:51.685740 ip-10-0-140-107 systemd[1]: kubelet.service: Consumed 3.109s CPU time. Nov 03 00:40:51.705373 ip-10-0-140-107 systemd[1]: Starting Kubernetes Kubelet... Nov 03 00:40:51.778990 ip-10-0-140-107 kubenswrapper[4939]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Nov 03 00:40:51.778990 ip-10-0-140-107 kubenswrapper[4939]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version. Nov 03 00:40:51.778990 ip-10-0-140-107 kubenswrapper[4939]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Nov 03 00:40:51.778990 ip-10-0-140-107 kubenswrapper[4939]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Nov 03 00:40:51.781102 ip-10-0-140-107 kubenswrapper[4939]: W1103 00:40:51.781023 4939 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 03 00:40:51.781102 ip-10-0-140-107 kubenswrapper[4939]: W1103 00:40:51.781102 4939 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 03 00:40:51.781179 ip-10-0-140-107 kubenswrapper[4939]: W1103 00:40:51.781107 4939 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 03 00:40:51.781179 ip-10-0-140-107 kubenswrapper[4939]: W1103 00:40:51.781110 4939 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 03 00:40:51.781179 ip-10-0-140-107 kubenswrapper[4939]: W1103 00:40:51.781113 4939 feature_gate.go:386] unrecognized feature gate: Example Nov 03 00:40:51.781179 ip-10-0-140-107 kubenswrapper[4939]: W1103 00:40:51.781118 4939 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 03 00:40:51.781179 ip-10-0-140-107 kubenswrapper[4939]: W1103 00:40:51.781122 4939 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 03 00:40:51.781179 ip-10-0-140-107 kubenswrapper[4939]: W1103 00:40:51.781125 4939 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 03 00:40:51.781179 ip-10-0-140-107 kubenswrapper[4939]: W1103 00:40:51.781129 4939 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 03 00:40:51.781179 ip-10-0-140-107 kubenswrapper[4939]: W1103 00:40:51.781132 4939 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 03 00:40:51.781179 ip-10-0-140-107 kubenswrapper[4939]: W1103 00:40:51.781135 4939 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 03 00:40:51.781179 ip-10-0-140-107 kubenswrapper[4939]: W1103 00:40:51.781138 4939 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 03 00:40:51.781179 ip-10-0-140-107 kubenswrapper[4939]: W1103 00:40:51.781141 4939 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 03 00:40:51.781179 ip-10-0-140-107 kubenswrapper[4939]: W1103 00:40:51.781144 4939 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 03 00:40:51.781179 ip-10-0-140-107 kubenswrapper[4939]: W1103 00:40:51.781146 4939 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 03 00:40:51.781179 ip-10-0-140-107 kubenswrapper[4939]: W1103 00:40:51.781149 4939 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 03 00:40:51.781179 ip-10-0-140-107 kubenswrapper[4939]: W1103 00:40:51.781152 4939 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 03 00:40:51.781179 ip-10-0-140-107 kubenswrapper[4939]: W1103 00:40:51.781155 4939 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 03 00:40:51.781179 ip-10-0-140-107 kubenswrapper[4939]: W1103 00:40:51.781158 4939 feature_gate.go:386] unrecognized feature gate: EVPN Nov 03 00:40:51.781179 ip-10-0-140-107 kubenswrapper[4939]: W1103 00:40:51.781161 4939 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 03 00:40:51.781179 ip-10-0-140-107 kubenswrapper[4939]: W1103 00:40:51.781164 4939 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 03 00:40:51.781633 ip-10-0-140-107 kubenswrapper[4939]: W1103 00:40:51.781166 4939 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 03 00:40:51.781633 ip-10-0-140-107 kubenswrapper[4939]: W1103 00:40:51.781169 4939 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 03 00:40:51.781633 ip-10-0-140-107 kubenswrapper[4939]: W1103 00:40:51.781171 4939 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 03 00:40:51.781633 ip-10-0-140-107 kubenswrapper[4939]: W1103 00:40:51.781174 4939 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 03 00:40:51.781633 ip-10-0-140-107 kubenswrapper[4939]: W1103 00:40:51.781177 4939 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 03 00:40:51.781633 ip-10-0-140-107 kubenswrapper[4939]: W1103 00:40:51.781185 4939 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 03 00:40:51.781633 ip-10-0-140-107 kubenswrapper[4939]: W1103 00:40:51.781188 4939 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 03 00:40:51.781633 ip-10-0-140-107 kubenswrapper[4939]: W1103 00:40:51.781190 4939 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 03 00:40:51.781633 ip-10-0-140-107 kubenswrapper[4939]: W1103 00:40:51.781193 4939 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 03 00:40:51.781633 ip-10-0-140-107 kubenswrapper[4939]: W1103 00:40:51.781196 4939 feature_gate.go:386] unrecognized feature gate: Example2 Nov 03 00:40:51.781633 ip-10-0-140-107 kubenswrapper[4939]: W1103 00:40:51.781199 4939 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 03 00:40:51.781633 ip-10-0-140-107 kubenswrapper[4939]: W1103 00:40:51.781201 4939 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 03 00:40:51.781633 ip-10-0-140-107 kubenswrapper[4939]: W1103 00:40:51.781204 4939 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 03 00:40:51.781633 ip-10-0-140-107 kubenswrapper[4939]: W1103 00:40:51.781207 4939 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 03 00:40:51.781633 ip-10-0-140-107 kubenswrapper[4939]: W1103 00:40:51.781209 4939 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 03 00:40:51.781633 ip-10-0-140-107 kubenswrapper[4939]: W1103 00:40:51.781212 4939 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 03 00:40:51.781633 ip-10-0-140-107 kubenswrapper[4939]: W1103 00:40:51.781214 4939 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 03 00:40:51.781633 ip-10-0-140-107 kubenswrapper[4939]: W1103 00:40:51.781217 4939 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 03 00:40:51.781633 ip-10-0-140-107 kubenswrapper[4939]: W1103 00:40:51.781220 4939 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 03 00:40:51.781633 ip-10-0-140-107 kubenswrapper[4939]: W1103 00:40:51.781222 4939 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 03 00:40:51.782147 ip-10-0-140-107 kubenswrapper[4939]: W1103 00:40:51.781225 4939 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 03 00:40:51.782147 ip-10-0-140-107 kubenswrapper[4939]: W1103 00:40:51.781227 4939 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 03 00:40:51.782147 ip-10-0-140-107 kubenswrapper[4939]: W1103 00:40:51.781230 4939 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 03 00:40:51.782147 ip-10-0-140-107 kubenswrapper[4939]: W1103 00:40:51.781233 4939 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 03 00:40:51.782147 ip-10-0-140-107 kubenswrapper[4939]: W1103 00:40:51.781235 4939 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 03 00:40:51.782147 ip-10-0-140-107 kubenswrapper[4939]: W1103 00:40:51.781238 4939 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 03 00:40:51.782147 ip-10-0-140-107 kubenswrapper[4939]: W1103 00:40:51.781241 4939 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 03 00:40:51.782147 ip-10-0-140-107 kubenswrapper[4939]: W1103 00:40:51.781243 4939 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 03 00:40:51.782147 ip-10-0-140-107 kubenswrapper[4939]: W1103 00:40:51.781246 4939 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 03 00:40:51.782147 ip-10-0-140-107 kubenswrapper[4939]: W1103 00:40:51.781248 4939 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 03 00:40:51.782147 ip-10-0-140-107 kubenswrapper[4939]: W1103 00:40:51.781251 4939 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 03 00:40:51.782147 ip-10-0-140-107 kubenswrapper[4939]: W1103 00:40:51.781254 4939 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 03 00:40:51.782147 ip-10-0-140-107 kubenswrapper[4939]: W1103 00:40:51.781257 4939 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 03 00:40:51.782147 ip-10-0-140-107 kubenswrapper[4939]: W1103 00:40:51.781260 4939 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 03 00:40:51.782147 ip-10-0-140-107 kubenswrapper[4939]: W1103 00:40:51.781262 4939 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 03 00:40:51.782147 ip-10-0-140-107 kubenswrapper[4939]: W1103 00:40:51.781265 4939 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 03 00:40:51.782147 ip-10-0-140-107 kubenswrapper[4939]: W1103 00:40:51.781269 4939 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 03 00:40:51.782147 ip-10-0-140-107 kubenswrapper[4939]: W1103 00:40:51.781272 4939 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 03 00:40:51.782147 ip-10-0-140-107 kubenswrapper[4939]: W1103 00:40:51.781275 4939 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 03 00:40:51.782147 ip-10-0-140-107 kubenswrapper[4939]: W1103 00:40:51.781277 4939 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 03 00:40:51.782634 ip-10-0-140-107 kubenswrapper[4939]: W1103 00:40:51.781279 4939 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 03 00:40:51.782634 ip-10-0-140-107 kubenswrapper[4939]: W1103 00:40:51.781282 4939 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 03 00:40:51.782634 ip-10-0-140-107 kubenswrapper[4939]: W1103 00:40:51.781285 4939 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 03 00:40:51.782634 ip-10-0-140-107 kubenswrapper[4939]: W1103 00:40:51.781288 4939 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 03 00:40:51.782634 ip-10-0-140-107 kubenswrapper[4939]: W1103 00:40:51.781290 4939 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 03 00:40:51.782634 ip-10-0-140-107 kubenswrapper[4939]: W1103 00:40:51.781293 4939 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 03 00:40:51.782634 ip-10-0-140-107 kubenswrapper[4939]: W1103 00:40:51.781296 4939 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 03 00:40:51.782634 ip-10-0-140-107 kubenswrapper[4939]: W1103 00:40:51.781299 4939 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 03 00:40:51.782634 ip-10-0-140-107 kubenswrapper[4939]: W1103 00:40:51.781301 4939 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 03 00:40:51.782634 ip-10-0-140-107 kubenswrapper[4939]: W1103 00:40:51.781304 4939 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 03 00:40:51.782634 ip-10-0-140-107 kubenswrapper[4939]: W1103 00:40:51.781306 4939 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 03 00:40:51.782634 ip-10-0-140-107 kubenswrapper[4939]: W1103 00:40:51.781309 4939 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 03 00:40:51.782634 ip-10-0-140-107 kubenswrapper[4939]: W1103 00:40:51.781312 4939 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 03 00:40:51.782634 ip-10-0-140-107 kubenswrapper[4939]: W1103 00:40:51.781314 4939 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 03 00:40:51.782634 ip-10-0-140-107 kubenswrapper[4939]: W1103 00:40:51.781317 4939 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 03 00:40:51.782634 ip-10-0-140-107 kubenswrapper[4939]: W1103 00:40:51.781320 4939 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 03 00:40:51.782634 ip-10-0-140-107 kubenswrapper[4939]: W1103 00:40:51.781322 4939 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 03 00:40:51.782634 ip-10-0-140-107 kubenswrapper[4939]: W1103 00:40:51.781325 4939 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 03 00:40:51.782634 ip-10-0-140-107 kubenswrapper[4939]: W1103 00:40:51.781328 4939 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 03 00:40:51.783151 ip-10-0-140-107 kubenswrapper[4939]: W1103 00:40:51.781330 4939 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 03 00:40:51.783151 ip-10-0-140-107 kubenswrapper[4939]: W1103 00:40:51.781333 4939 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 03 00:40:51.783151 ip-10-0-140-107 kubenswrapper[4939]: W1103 00:40:51.781336 4939 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 03 00:40:51.783151 ip-10-0-140-107 kubenswrapper[4939]: W1103 00:40:51.781339 4939 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 03 00:40:51.783151 ip-10-0-140-107 kubenswrapper[4939]: W1103 00:40:51.781342 4939 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 03 00:40:51.783151 ip-10-0-140-107 kubenswrapper[4939]: W1103 00:40:51.781344 4939 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 03 00:40:51.783151 ip-10-0-140-107 kubenswrapper[4939]: W1103 00:40:51.781347 4939 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 03 00:40:51.783151 ip-10-0-140-107 kubenswrapper[4939]: W1103 00:40:51.781350 4939 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 03 00:40:51.783151 ip-10-0-140-107 kubenswrapper[4939]: W1103 00:40:51.781352 4939 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 03 00:40:51.783151 ip-10-0-140-107 kubenswrapper[4939]: W1103 00:40:51.781355 4939 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 03 00:40:51.783151 ip-10-0-140-107 kubenswrapper[4939]: W1103 00:40:51.781358 4939 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 03 00:40:51.783151 ip-10-0-140-107 kubenswrapper[4939]: W1103 00:40:51.781361 4939 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 03 00:40:51.783151 ip-10-0-140-107 kubenswrapper[4939]: W1103 00:40:51.781363 4939 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 03 00:40:51.783151 ip-10-0-140-107 kubenswrapper[4939]: W1103 00:40:51.781366 4939 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 03 00:40:51.783151 ip-10-0-140-107 kubenswrapper[4939]: W1103 00:40:51.781368 4939 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 03 00:40:51.783151 ip-10-0-140-107 kubenswrapper[4939]: W1103 00:40:51.781373 4939 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 03 00:40:51.783151 ip-10-0-140-107 kubenswrapper[4939]: W1103 00:40:51.781377 4939 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 03 00:40:51.783151 ip-10-0-140-107 kubenswrapper[4939]: W1103 00:40:51.781381 4939 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 03 00:40:51.783151 ip-10-0-140-107 kubenswrapper[4939]: W1103 00:40:51.781383 4939 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 03 00:40:51.783609 ip-10-0-140-107 kubenswrapper[4939]: W1103 00:40:51.781387 4939 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 03 00:40:51.783609 ip-10-0-140-107 kubenswrapper[4939]: W1103 00:40:51.781389 4939 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 03 00:40:51.783609 ip-10-0-140-107 kubenswrapper[4939]: W1103 00:40:51.781392 4939 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 03 00:40:51.783609 ip-10-0-140-107 kubenswrapper[4939]: W1103 00:40:51.781394 4939 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 03 00:40:51.783609 ip-10-0-140-107 kubenswrapper[4939]: W1103 00:40:51.781397 4939 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 03 00:40:51.783609 ip-10-0-140-107 kubenswrapper[4939]: W1103 00:40:51.781399 4939 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 03 00:40:51.783609 ip-10-0-140-107 kubenswrapper[4939]: W1103 00:40:51.781402 4939 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 03 00:40:51.783609 ip-10-0-140-107 kubenswrapper[4939]: W1103 00:40:51.782159 4939 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 03 00:40:51.783609 ip-10-0-140-107 kubenswrapper[4939]: W1103 00:40:51.782168 4939 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 03 00:40:51.783609 ip-10-0-140-107 kubenswrapper[4939]: W1103 00:40:51.782171 4939 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 03 00:40:51.783609 ip-10-0-140-107 kubenswrapper[4939]: W1103 00:40:51.782174 4939 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 03 00:40:51.783609 ip-10-0-140-107 kubenswrapper[4939]: W1103 00:40:51.782177 4939 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 03 00:40:51.783609 ip-10-0-140-107 kubenswrapper[4939]: W1103 00:40:51.782180 4939 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 03 00:40:51.783609 ip-10-0-140-107 kubenswrapper[4939]: W1103 00:40:51.782182 4939 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 03 00:40:51.783609 ip-10-0-140-107 kubenswrapper[4939]: W1103 00:40:51.782185 4939 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 03 00:40:51.783609 ip-10-0-140-107 kubenswrapper[4939]: W1103 00:40:51.782189 4939 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 03 00:40:51.783609 ip-10-0-140-107 kubenswrapper[4939]: W1103 00:40:51.782192 4939 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 03 00:40:51.783609 ip-10-0-140-107 kubenswrapper[4939]: W1103 00:40:51.782195 4939 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 03 00:40:51.783609 ip-10-0-140-107 kubenswrapper[4939]: W1103 00:40:51.782197 4939 feature_gate.go:386] unrecognized feature gate: Example2 Nov 03 00:40:51.783609 ip-10-0-140-107 kubenswrapper[4939]: W1103 00:40:51.782200 4939 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 03 00:40:51.784110 ip-10-0-140-107 kubenswrapper[4939]: W1103 00:40:51.782203 4939 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 03 00:40:51.784110 ip-10-0-140-107 kubenswrapper[4939]: W1103 00:40:51.782206 4939 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 03 00:40:51.784110 ip-10-0-140-107 kubenswrapper[4939]: W1103 00:40:51.782208 4939 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 03 00:40:51.784110 ip-10-0-140-107 kubenswrapper[4939]: W1103 00:40:51.782211 4939 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 03 00:40:51.784110 ip-10-0-140-107 kubenswrapper[4939]: W1103 00:40:51.782213 4939 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 03 00:40:51.784110 ip-10-0-140-107 kubenswrapper[4939]: W1103 00:40:51.782217 4939 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 03 00:40:51.784110 ip-10-0-140-107 kubenswrapper[4939]: W1103 00:40:51.782220 4939 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 03 00:40:51.784110 ip-10-0-140-107 kubenswrapper[4939]: W1103 00:40:51.782222 4939 feature_gate.go:386] unrecognized feature gate: EVPN Nov 03 00:40:51.784110 ip-10-0-140-107 kubenswrapper[4939]: W1103 00:40:51.782225 4939 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 03 00:40:51.784110 ip-10-0-140-107 kubenswrapper[4939]: W1103 00:40:51.782228 4939 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 03 00:40:51.784110 ip-10-0-140-107 kubenswrapper[4939]: W1103 00:40:51.782230 4939 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 03 00:40:51.784110 ip-10-0-140-107 kubenswrapper[4939]: W1103 00:40:51.782233 4939 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 03 00:40:51.784110 ip-10-0-140-107 kubenswrapper[4939]: W1103 00:40:51.782236 4939 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 03 00:40:51.784110 ip-10-0-140-107 kubenswrapper[4939]: W1103 00:40:51.782238 4939 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 03 00:40:51.784110 ip-10-0-140-107 kubenswrapper[4939]: W1103 00:40:51.782242 4939 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 03 00:40:51.784110 ip-10-0-140-107 kubenswrapper[4939]: W1103 00:40:51.782246 4939 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 03 00:40:51.784110 ip-10-0-140-107 kubenswrapper[4939]: W1103 00:40:51.782250 4939 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 03 00:40:51.784110 ip-10-0-140-107 kubenswrapper[4939]: W1103 00:40:51.782252 4939 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 03 00:40:51.784110 ip-10-0-140-107 kubenswrapper[4939]: W1103 00:40:51.782255 4939 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 03 00:40:51.784597 ip-10-0-140-107 kubenswrapper[4939]: W1103 00:40:51.782258 4939 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 03 00:40:51.784597 ip-10-0-140-107 kubenswrapper[4939]: W1103 00:40:51.782261 4939 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 03 00:40:51.784597 ip-10-0-140-107 kubenswrapper[4939]: W1103 00:40:51.782264 4939 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 03 00:40:51.784597 ip-10-0-140-107 kubenswrapper[4939]: W1103 00:40:51.782267 4939 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 03 00:40:51.784597 ip-10-0-140-107 kubenswrapper[4939]: W1103 00:40:51.782270 4939 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 03 00:40:51.784597 ip-10-0-140-107 kubenswrapper[4939]: W1103 00:40:51.782272 4939 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 03 00:40:51.784597 ip-10-0-140-107 kubenswrapper[4939]: W1103 00:40:51.782275 4939 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 03 00:40:51.784597 ip-10-0-140-107 kubenswrapper[4939]: W1103 00:40:51.782278 4939 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 03 00:40:51.784597 ip-10-0-140-107 kubenswrapper[4939]: W1103 00:40:51.782281 4939 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 03 00:40:51.784597 ip-10-0-140-107 kubenswrapper[4939]: W1103 00:40:51.782283 4939 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 03 00:40:51.784597 ip-10-0-140-107 kubenswrapper[4939]: W1103 00:40:51.782286 4939 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 03 00:40:51.784597 ip-10-0-140-107 kubenswrapper[4939]: W1103 00:40:51.782288 4939 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 03 00:40:51.784597 ip-10-0-140-107 kubenswrapper[4939]: W1103 00:40:51.782291 4939 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 03 00:40:51.784597 ip-10-0-140-107 kubenswrapper[4939]: W1103 00:40:51.782294 4939 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 03 00:40:51.784597 ip-10-0-140-107 kubenswrapper[4939]: W1103 00:40:51.782297 4939 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 03 00:40:51.784597 ip-10-0-140-107 kubenswrapper[4939]: W1103 00:40:51.782300 4939 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 03 00:40:51.784597 ip-10-0-140-107 kubenswrapper[4939]: W1103 00:40:51.782302 4939 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 03 00:40:51.784597 ip-10-0-140-107 kubenswrapper[4939]: W1103 00:40:51.782306 4939 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 03 00:40:51.784597 ip-10-0-140-107 kubenswrapper[4939]: W1103 00:40:51.782309 4939 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 03 00:40:51.785077 ip-10-0-140-107 kubenswrapper[4939]: W1103 00:40:51.782312 4939 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 03 00:40:51.785077 ip-10-0-140-107 kubenswrapper[4939]: W1103 00:40:51.782315 4939 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 03 00:40:51.785077 ip-10-0-140-107 kubenswrapper[4939]: W1103 00:40:51.782318 4939 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 03 00:40:51.785077 ip-10-0-140-107 kubenswrapper[4939]: W1103 00:40:51.782320 4939 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 03 00:40:51.785077 ip-10-0-140-107 kubenswrapper[4939]: W1103 00:40:51.782323 4939 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 03 00:40:51.785077 ip-10-0-140-107 kubenswrapper[4939]: W1103 00:40:51.782326 4939 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 03 00:40:51.785077 ip-10-0-140-107 kubenswrapper[4939]: W1103 00:40:51.782329 4939 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 03 00:40:51.785077 ip-10-0-140-107 kubenswrapper[4939]: W1103 00:40:51.782332 4939 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 03 00:40:51.785077 ip-10-0-140-107 kubenswrapper[4939]: W1103 00:40:51.782334 4939 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 03 00:40:51.785077 ip-10-0-140-107 kubenswrapper[4939]: W1103 00:40:51.782337 4939 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 03 00:40:51.785077 ip-10-0-140-107 kubenswrapper[4939]: W1103 00:40:51.782340 4939 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 03 00:40:51.785077 ip-10-0-140-107 kubenswrapper[4939]: W1103 00:40:51.782343 4939 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 03 00:40:51.785077 ip-10-0-140-107 kubenswrapper[4939]: W1103 00:40:51.782346 4939 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 03 00:40:51.785077 ip-10-0-140-107 kubenswrapper[4939]: W1103 00:40:51.782348 4939 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 03 00:40:51.785077 ip-10-0-140-107 kubenswrapper[4939]: W1103 00:40:51.782353 4939 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 03 00:40:51.785077 ip-10-0-140-107 kubenswrapper[4939]: W1103 00:40:51.782357 4939 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 03 00:40:51.785077 ip-10-0-140-107 kubenswrapper[4939]: W1103 00:40:51.782360 4939 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 03 00:40:51.785077 ip-10-0-140-107 kubenswrapper[4939]: W1103 00:40:51.782362 4939 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 03 00:40:51.785077 ip-10-0-140-107 kubenswrapper[4939]: W1103 00:40:51.782365 4939 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 03 00:40:51.785537 ip-10-0-140-107 kubenswrapper[4939]: W1103 00:40:51.782368 4939 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 03 00:40:51.785537 ip-10-0-140-107 kubenswrapper[4939]: W1103 00:40:51.782371 4939 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 03 00:40:51.785537 ip-10-0-140-107 kubenswrapper[4939]: W1103 00:40:51.782373 4939 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 03 00:40:51.785537 ip-10-0-140-107 kubenswrapper[4939]: W1103 00:40:51.782376 4939 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 03 00:40:51.785537 ip-10-0-140-107 kubenswrapper[4939]: W1103 00:40:51.782378 4939 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 03 00:40:51.785537 ip-10-0-140-107 kubenswrapper[4939]: W1103 00:40:51.782381 4939 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 03 00:40:51.785537 ip-10-0-140-107 kubenswrapper[4939]: W1103 00:40:51.782384 4939 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 03 00:40:51.785537 ip-10-0-140-107 kubenswrapper[4939]: W1103 00:40:51.782386 4939 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 03 00:40:51.785537 ip-10-0-140-107 kubenswrapper[4939]: W1103 00:40:51.782389 4939 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 03 00:40:51.785537 ip-10-0-140-107 kubenswrapper[4939]: W1103 00:40:51.782391 4939 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 03 00:40:51.785537 ip-10-0-140-107 kubenswrapper[4939]: W1103 00:40:51.782394 4939 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 03 00:40:51.785537 ip-10-0-140-107 kubenswrapper[4939]: W1103 00:40:51.782396 4939 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 03 00:40:51.785537 ip-10-0-140-107 kubenswrapper[4939]: W1103 00:40:51.782400 4939 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 03 00:40:51.785537 ip-10-0-140-107 kubenswrapper[4939]: W1103 00:40:51.782403 4939 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 03 00:40:51.785537 ip-10-0-140-107 kubenswrapper[4939]: W1103 00:40:51.782405 4939 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 03 00:40:51.785537 ip-10-0-140-107 kubenswrapper[4939]: W1103 00:40:51.782409 4939 feature_gate.go:386] unrecognized feature gate: Example Nov 03 00:40:51.785537 ip-10-0-140-107 kubenswrapper[4939]: W1103 00:40:51.782412 4939 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 03 00:40:51.785537 ip-10-0-140-107 kubenswrapper[4939]: W1103 00:40:51.782414 4939 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 03 00:40:51.785537 ip-10-0-140-107 kubenswrapper[4939]: W1103 00:40:51.782417 4939 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 03 00:40:51.785537 ip-10-0-140-107 kubenswrapper[4939]: W1103 00:40:51.782420 4939 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 03 00:40:51.786037 ip-10-0-140-107 kubenswrapper[4939]: W1103 00:40:51.782422 4939 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 03 00:40:51.786037 ip-10-0-140-107 kubenswrapper[4939]: W1103 00:40:51.782425 4939 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 03 00:40:51.786037 ip-10-0-140-107 kubenswrapper[4939]: W1103 00:40:51.782427 4939 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 03 00:40:51.786037 ip-10-0-140-107 kubenswrapper[4939]: W1103 00:40:51.782430 4939 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 03 00:40:51.786037 ip-10-0-140-107 kubenswrapper[4939]: W1103 00:40:51.782432 4939 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 03 00:40:51.786037 ip-10-0-140-107 kubenswrapper[4939]: W1103 00:40:51.782435 4939 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 03 00:40:51.786037 ip-10-0-140-107 kubenswrapper[4939]: W1103 00:40:51.782437 4939 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 03 00:40:51.786037 ip-10-0-140-107 kubenswrapper[4939]: W1103 00:40:51.782440 4939 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 03 00:40:51.786037 ip-10-0-140-107 kubenswrapper[4939]: W1103 00:40:51.782442 4939 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 03 00:40:51.786037 ip-10-0-140-107 kubenswrapper[4939]: W1103 00:40:51.782446 4939 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 03 00:40:51.786037 ip-10-0-140-107 kubenswrapper[4939]: W1103 00:40:51.782449 4939 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 03 00:40:51.786037 ip-10-0-140-107 kubenswrapper[4939]: W1103 00:40:51.782451 4939 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 03 00:40:51.786037 ip-10-0-140-107 kubenswrapper[4939]: W1103 00:40:51.782454 4939 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 03 00:40:51.786037 ip-10-0-140-107 kubenswrapper[4939]: W1103 00:40:51.782456 4939 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 03 00:40:51.786037 ip-10-0-140-107 kubenswrapper[4939]: W1103 00:40:51.782459 4939 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 03 00:40:51.786037 ip-10-0-140-107 kubenswrapper[4939]: W1103 00:40:51.782461 4939 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 03 00:40:51.786037 ip-10-0-140-107 kubenswrapper[4939]: W1103 00:40:51.782941 4939 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 03 00:40:51.786037 ip-10-0-140-107 kubenswrapper[4939]: W1103 00:40:51.782947 4939 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 03 00:40:51.786037 ip-10-0-140-107 kubenswrapper[4939]: W1103 00:40:51.782950 4939 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 03 00:40:51.786037 ip-10-0-140-107 kubenswrapper[4939]: W1103 00:40:51.782952 4939 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 03 00:40:51.786520 ip-10-0-140-107 kubenswrapper[4939]: W1103 00:40:51.782955 4939 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 03 00:40:51.786520 ip-10-0-140-107 kubenswrapper[4939]: W1103 00:40:51.782958 4939 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 03 00:40:51.786520 ip-10-0-140-107 kubenswrapper[4939]: W1103 00:40:51.782960 4939 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 03 00:40:51.786520 ip-10-0-140-107 kubenswrapper[4939]: W1103 00:40:51.782964 4939 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 03 00:40:51.786520 ip-10-0-140-107 kubenswrapper[4939]: W1103 00:40:51.782966 4939 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 03 00:40:51.786520 ip-10-0-140-107 kubenswrapper[4939]: W1103 00:40:51.782969 4939 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 03 00:40:51.786520 ip-10-0-140-107 kubenswrapper[4939]: W1103 00:40:51.782973 4939 feature_gate.go:386] unrecognized feature gate: EVPN Nov 03 00:40:51.786520 ip-10-0-140-107 kubenswrapper[4939]: W1103 00:40:51.782976 4939 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 03 00:40:51.786520 ip-10-0-140-107 kubenswrapper[4939]: W1103 00:40:51.782979 4939 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 03 00:40:51.786520 ip-10-0-140-107 kubenswrapper[4939]: W1103 00:40:51.782981 4939 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 03 00:40:51.786520 ip-10-0-140-107 kubenswrapper[4939]: W1103 00:40:51.782984 4939 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 03 00:40:51.786520 ip-10-0-140-107 kubenswrapper[4939]: W1103 00:40:51.782987 4939 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 03 00:40:51.786520 ip-10-0-140-107 kubenswrapper[4939]: W1103 00:40:51.782990 4939 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 03 00:40:51.786520 ip-10-0-140-107 kubenswrapper[4939]: W1103 00:40:51.782992 4939 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 03 00:40:51.786520 ip-10-0-140-107 kubenswrapper[4939]: W1103 00:40:51.782995 4939 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 03 00:40:51.786520 ip-10-0-140-107 kubenswrapper[4939]: W1103 00:40:51.782998 4939 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 03 00:40:51.786520 ip-10-0-140-107 kubenswrapper[4939]: W1103 00:40:51.783000 4939 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 03 00:40:51.786520 ip-10-0-140-107 kubenswrapper[4939]: W1103 00:40:51.783003 4939 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 03 00:40:51.786520 ip-10-0-140-107 kubenswrapper[4939]: W1103 00:40:51.783005 4939 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 03 00:40:51.786520 ip-10-0-140-107 kubenswrapper[4939]: W1103 00:40:51.783008 4939 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 03 00:40:51.787018 ip-10-0-140-107 kubenswrapper[4939]: W1103 00:40:51.783010 4939 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 03 00:40:51.787018 ip-10-0-140-107 kubenswrapper[4939]: W1103 00:40:51.783013 4939 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 03 00:40:51.787018 ip-10-0-140-107 kubenswrapper[4939]: W1103 00:40:51.783016 4939 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 03 00:40:51.787018 ip-10-0-140-107 kubenswrapper[4939]: W1103 00:40:51.783019 4939 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 03 00:40:51.787018 ip-10-0-140-107 kubenswrapper[4939]: W1103 00:40:51.783021 4939 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 03 00:40:51.787018 ip-10-0-140-107 kubenswrapper[4939]: W1103 00:40:51.783024 4939 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 03 00:40:51.787018 ip-10-0-140-107 kubenswrapper[4939]: W1103 00:40:51.783027 4939 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 03 00:40:51.787018 ip-10-0-140-107 kubenswrapper[4939]: W1103 00:40:51.783029 4939 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 03 00:40:51.787018 ip-10-0-140-107 kubenswrapper[4939]: W1103 00:40:51.783032 4939 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 03 00:40:51.787018 ip-10-0-140-107 kubenswrapper[4939]: W1103 00:40:51.783035 4939 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 03 00:40:51.787018 ip-10-0-140-107 kubenswrapper[4939]: W1103 00:40:51.783038 4939 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 03 00:40:51.787018 ip-10-0-140-107 kubenswrapper[4939]: W1103 00:40:51.783040 4939 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 03 00:40:51.787018 ip-10-0-140-107 kubenswrapper[4939]: W1103 00:40:51.783043 4939 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 03 00:40:51.787018 ip-10-0-140-107 kubenswrapper[4939]: W1103 00:40:51.783046 4939 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 03 00:40:51.787018 ip-10-0-140-107 kubenswrapper[4939]: W1103 00:40:51.783048 4939 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 03 00:40:51.787018 ip-10-0-140-107 kubenswrapper[4939]: W1103 00:40:51.783051 4939 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 03 00:40:51.787018 ip-10-0-140-107 kubenswrapper[4939]: W1103 00:40:51.783054 4939 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 03 00:40:51.787018 ip-10-0-140-107 kubenswrapper[4939]: W1103 00:40:51.783056 4939 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 03 00:40:51.787018 ip-10-0-140-107 kubenswrapper[4939]: W1103 00:40:51.783062 4939 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 03 00:40:51.787541 ip-10-0-140-107 kubenswrapper[4939]: W1103 00:40:51.783066 4939 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 03 00:40:51.787541 ip-10-0-140-107 kubenswrapper[4939]: W1103 00:40:51.783068 4939 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 03 00:40:51.787541 ip-10-0-140-107 kubenswrapper[4939]: W1103 00:40:51.783071 4939 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 03 00:40:51.787541 ip-10-0-140-107 kubenswrapper[4939]: W1103 00:40:51.783074 4939 feature_gate.go:386] unrecognized feature gate: Example2 Nov 03 00:40:51.787541 ip-10-0-140-107 kubenswrapper[4939]: W1103 00:40:51.783076 4939 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 03 00:40:51.787541 ip-10-0-140-107 kubenswrapper[4939]: W1103 00:40:51.783079 4939 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 03 00:40:51.787541 ip-10-0-140-107 kubenswrapper[4939]: W1103 00:40:51.783082 4939 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 03 00:40:51.787541 ip-10-0-140-107 kubenswrapper[4939]: W1103 00:40:51.783085 4939 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 03 00:40:51.787541 ip-10-0-140-107 kubenswrapper[4939]: W1103 00:40:51.783087 4939 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 03 00:40:51.787541 ip-10-0-140-107 kubenswrapper[4939]: W1103 00:40:51.783090 4939 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 03 00:40:51.787541 ip-10-0-140-107 kubenswrapper[4939]: W1103 00:40:51.783094 4939 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 03 00:40:51.787541 ip-10-0-140-107 kubenswrapper[4939]: W1103 00:40:51.783099 4939 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 03 00:40:51.787541 ip-10-0-140-107 kubenswrapper[4939]: W1103 00:40:51.783102 4939 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 03 00:40:51.787541 ip-10-0-140-107 kubenswrapper[4939]: W1103 00:40:51.783106 4939 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 03 00:40:51.787541 ip-10-0-140-107 kubenswrapper[4939]: W1103 00:40:51.783109 4939 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 03 00:40:51.787541 ip-10-0-140-107 kubenswrapper[4939]: W1103 00:40:51.783111 4939 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 03 00:40:51.787541 ip-10-0-140-107 kubenswrapper[4939]: W1103 00:40:51.783115 4939 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 03 00:40:51.787541 ip-10-0-140-107 kubenswrapper[4939]: W1103 00:40:51.783120 4939 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 03 00:40:51.787541 ip-10-0-140-107 kubenswrapper[4939]: W1103 00:40:51.783124 4939 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 03 00:40:51.788349 ip-10-0-140-107 kubenswrapper[4939]: W1103 00:40:51.783127 4939 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 03 00:40:51.788349 ip-10-0-140-107 kubenswrapper[4939]: W1103 00:40:51.783130 4939 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 03 00:40:51.788349 ip-10-0-140-107 kubenswrapper[4939]: W1103 00:40:51.783134 4939 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 03 00:40:51.788349 ip-10-0-140-107 kubenswrapper[4939]: W1103 00:40:51.783137 4939 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 03 00:40:51.788349 ip-10-0-140-107 kubenswrapper[4939]: W1103 00:40:51.783140 4939 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 03 00:40:51.788349 ip-10-0-140-107 kubenswrapper[4939]: W1103 00:40:51.783142 4939 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 03 00:40:51.788349 ip-10-0-140-107 kubenswrapper[4939]: W1103 00:40:51.783145 4939 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 03 00:40:51.788349 ip-10-0-140-107 kubenswrapper[4939]: W1103 00:40:51.783148 4939 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 03 00:40:51.788349 ip-10-0-140-107 kubenswrapper[4939]: W1103 00:40:51.783150 4939 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 03 00:40:51.788349 ip-10-0-140-107 kubenswrapper[4939]: W1103 00:40:51.783153 4939 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 03 00:40:51.788349 ip-10-0-140-107 kubenswrapper[4939]: W1103 00:40:51.783156 4939 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 03 00:40:51.788349 ip-10-0-140-107 kubenswrapper[4939]: W1103 00:40:51.783159 4939 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 03 00:40:51.788349 ip-10-0-140-107 kubenswrapper[4939]: W1103 00:40:51.783165 4939 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 03 00:40:51.788349 ip-10-0-140-107 kubenswrapper[4939]: W1103 00:40:51.783168 4939 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 03 00:40:51.788349 ip-10-0-140-107 kubenswrapper[4939]: W1103 00:40:51.783171 4939 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 03 00:40:51.788349 ip-10-0-140-107 kubenswrapper[4939]: W1103 00:40:51.783174 4939 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 03 00:40:51.788349 ip-10-0-140-107 kubenswrapper[4939]: W1103 00:40:51.783177 4939 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 03 00:40:51.788349 ip-10-0-140-107 kubenswrapper[4939]: W1103 00:40:51.783179 4939 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 03 00:40:51.788349 ip-10-0-140-107 kubenswrapper[4939]: W1103 00:40:51.783182 4939 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 03 00:40:51.788877 ip-10-0-140-107 kubenswrapper[4939]: W1103 00:40:51.783185 4939 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 03 00:40:51.788877 ip-10-0-140-107 kubenswrapper[4939]: W1103 00:40:51.783187 4939 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 03 00:40:51.788877 ip-10-0-140-107 kubenswrapper[4939]: W1103 00:40:51.783190 4939 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 03 00:40:51.788877 ip-10-0-140-107 kubenswrapper[4939]: W1103 00:40:51.783192 4939 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 03 00:40:51.788877 ip-10-0-140-107 kubenswrapper[4939]: W1103 00:40:51.783195 4939 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 03 00:40:51.788877 ip-10-0-140-107 kubenswrapper[4939]: W1103 00:40:51.783198 4939 feature_gate.go:386] unrecognized feature gate: Example Nov 03 00:40:51.788877 ip-10-0-140-107 kubenswrapper[4939]: W1103 00:40:51.783200 4939 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 03 00:40:51.788877 ip-10-0-140-107 kubenswrapper[4939]: W1103 00:40:51.783203 4939 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 03 00:40:51.788877 ip-10-0-140-107 kubenswrapper[4939]: W1103 00:40:51.783205 4939 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 03 00:40:51.788877 ip-10-0-140-107 kubenswrapper[4939]: W1103 00:40:51.783208 4939 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 03 00:40:51.788877 ip-10-0-140-107 kubenswrapper[4939]: W1103 00:40:51.783211 4939 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 03 00:40:51.788877 ip-10-0-140-107 kubenswrapper[4939]: W1103 00:40:51.783213 4939 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 03 00:40:51.788877 ip-10-0-140-107 kubenswrapper[4939]: W1103 00:40:51.783216 4939 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 03 00:40:51.788877 ip-10-0-140-107 kubenswrapper[4939]: W1103 00:40:51.783218 4939 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 03 00:40:51.788877 ip-10-0-140-107 kubenswrapper[4939]: W1103 00:40:51.783221 4939 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 03 00:40:51.788877 ip-10-0-140-107 kubenswrapper[4939]: W1103 00:40:51.783224 4939 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 03 00:40:51.788877 ip-10-0-140-107 kubenswrapper[4939]: W1103 00:40:51.783226 4939 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 03 00:40:51.788877 ip-10-0-140-107 kubenswrapper[4939]: W1103 00:40:51.783229 4939 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 03 00:40:51.788877 ip-10-0-140-107 kubenswrapper[4939]: W1103 00:40:51.783231 4939 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 03 00:40:51.788877 ip-10-0-140-107 kubenswrapper[4939]: W1103 00:40:51.783234 4939 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 03 00:40:51.789413 ip-10-0-140-107 kubenswrapper[4939]: W1103 00:40:51.783239 4939 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 03 00:40:51.789413 ip-10-0-140-107 kubenswrapper[4939]: W1103 00:40:51.783242 4939 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 03 00:40:51.789413 ip-10-0-140-107 kubenswrapper[4939]: W1103 00:40:51.783245 4939 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 03 00:40:51.789413 ip-10-0-140-107 kubenswrapper[4939]: W1103 00:40:51.783247 4939 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 03 00:40:51.789413 ip-10-0-140-107 kubenswrapper[4939]: W1103 00:40:51.783250 4939 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 03 00:40:51.789413 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:40:51.783341 4939 flags.go:64] FLAG: --address="0.0.0.0" Nov 03 00:40:51.789413 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:40:51.783349 4939 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]" Nov 03 00:40:51.789413 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:40:51.783356 4939 flags.go:64] FLAG: --anonymous-auth="true" Nov 03 00:40:51.789413 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:40:51.783365 4939 flags.go:64] FLAG: --application-metrics-count-limit="100" Nov 03 00:40:51.789413 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:40:51.783370 4939 flags.go:64] FLAG: --authentication-token-webhook="false" Nov 03 00:40:51.789413 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:40:51.783374 4939 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s" Nov 03 00:40:51.789413 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:40:51.783378 4939 flags.go:64] FLAG: --authorization-mode="AlwaysAllow" Nov 03 00:40:51.789413 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:40:51.783382 4939 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s" Nov 03 00:40:51.789413 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:40:51.783385 4939 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s" Nov 03 00:40:51.789413 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:40:51.783388 4939 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id" Nov 03 00:40:51.789413 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:40:51.783392 4939 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig" Nov 03 00:40:51.789413 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:40:51.783396 4939 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki" Nov 03 00:40:51.789413 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:40:51.783399 4939 flags.go:64] FLAG: --cgroup-driver="cgroupfs" Nov 03 00:40:51.789413 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:40:51.783402 4939 flags.go:64] FLAG: --cgroup-root="" Nov 03 00:40:51.789413 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:40:51.783405 4939 flags.go:64] FLAG: --cgroups-per-qos="true" Nov 03 00:40:51.789413 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:40:51.783409 4939 flags.go:64] FLAG: --client-ca-file="" Nov 03 00:40:51.789413 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:40:51.783411 4939 flags.go:64] FLAG: --cloud-provider="external" Nov 03 00:40:51.789413 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:40:51.783414 4939 flags.go:64] FLAG: --cluster-dns="[]" Nov 03 00:40:51.789980 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:40:51.783419 4939 flags.go:64] FLAG: --cluster-domain="" Nov 03 00:40:51.789980 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:40:51.783422 4939 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf" Nov 03 00:40:51.789980 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:40:51.783425 4939 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d" Nov 03 00:40:51.789980 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:40:51.783428 4939 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json" Nov 03 00:40:51.789980 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:40:51.783432 4939 flags.go:64] FLAG: --container-log-max-files="5" Nov 03 00:40:51.789980 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:40:51.783436 4939 flags.go:64] FLAG: --container-log-max-size="10Mi" Nov 03 00:40:51.789980 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:40:51.783439 4939 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock" Nov 03 00:40:51.789980 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:40:51.783442 4939 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock" Nov 03 00:40:51.789980 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:40:51.783445 4939 flags.go:64] FLAG: --containerd-namespace="k8s.io" Nov 03 00:40:51.789980 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:40:51.783450 4939 flags.go:64] FLAG: --contention-profiling="false" Nov 03 00:40:51.789980 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:40:51.783454 4939 flags.go:64] FLAG: --cpu-cfs-quota="true" Nov 03 00:40:51.789980 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:40:51.783456 4939 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms" Nov 03 00:40:51.789980 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:40:51.783460 4939 flags.go:64] FLAG: --cpu-manager-policy="none" Nov 03 00:40:51.789980 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:40:51.783463 4939 flags.go:64] FLAG: --cpu-manager-policy-options="" Nov 03 00:40:51.789980 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:40:51.783468 4939 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s" Nov 03 00:40:51.789980 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:40:51.783471 4939 flags.go:64] FLAG: --enable-controller-attach-detach="true" Nov 03 00:40:51.789980 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:40:51.783474 4939 flags.go:64] FLAG: --enable-debugging-handlers="true" Nov 03 00:40:51.789980 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:40:51.783477 4939 flags.go:64] FLAG: --enable-load-reader="false" Nov 03 00:40:51.789980 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:40:51.783480 4939 flags.go:64] FLAG: --enable-server="true" Nov 03 00:40:51.789980 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:40:51.783484 4939 flags.go:64] FLAG: --enforce-node-allocatable="[pods]" Nov 03 00:40:51.789980 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:40:51.783487 4939 flags.go:64] FLAG: --event-burst="100" Nov 03 00:40:51.789980 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:40:51.783490 4939 flags.go:64] FLAG: --event-qps="50" Nov 03 00:40:51.789980 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:40:51.783493 4939 flags.go:64] FLAG: --event-storage-age-limit="default=0" Nov 03 00:40:51.789980 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:40:51.783496 4939 flags.go:64] FLAG: --event-storage-event-limit="default=0" Nov 03 00:40:51.790623 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:40:51.783500 4939 flags.go:64] FLAG: --eviction-hard="" Nov 03 00:40:51.790623 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:40:51.783504 4939 flags.go:64] FLAG: --eviction-max-pod-grace-period="0" Nov 03 00:40:51.790623 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:40:51.783507 4939 flags.go:64] FLAG: --eviction-minimum-reclaim="" Nov 03 00:40:51.790623 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:40:51.783510 4939 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s" Nov 03 00:40:51.790623 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:40:51.783513 4939 flags.go:64] FLAG: --eviction-soft="" Nov 03 00:40:51.790623 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:40:51.783516 4939 flags.go:64] FLAG: --eviction-soft-grace-period="" Nov 03 00:40:51.790623 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:40:51.783520 4939 flags.go:64] FLAG: --exit-on-lock-contention="false" Nov 03 00:40:51.790623 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:40:51.783523 4939 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false" Nov 03 00:40:51.790623 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:40:51.783526 4939 flags.go:64] FLAG: --experimental-mounter-path="" Nov 03 00:40:51.790623 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:40:51.783529 4939 flags.go:64] FLAG: --fail-cgroupv1="true" Nov 03 00:40:51.790623 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:40:51.783532 4939 flags.go:64] FLAG: --fail-swap-on="true" Nov 03 00:40:51.790623 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:40:51.783535 4939 flags.go:64] FLAG: --feature-gates="" Nov 03 00:40:51.790623 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:40:51.783539 4939 flags.go:64] FLAG: --file-check-frequency="20s" Nov 03 00:40:51.790623 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:40:51.783542 4939 flags.go:64] FLAG: --global-housekeeping-interval="1m0s" Nov 03 00:40:51.790623 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:40:51.783545 4939 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge" Nov 03 00:40:51.790623 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:40:51.783549 4939 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1" Nov 03 00:40:51.790623 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:40:51.783552 4939 flags.go:64] FLAG: --healthz-port="10248" Nov 03 00:40:51.790623 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:40:51.783557 4939 flags.go:64] FLAG: --help="false" Nov 03 00:40:51.790623 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:40:51.783560 4939 flags.go:64] FLAG: --hostname-override="ip-10-0-140-107.us-east-2.compute.internal" Nov 03 00:40:51.790623 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:40:51.783563 4939 flags.go:64] FLAG: --housekeeping-interval="10s" Nov 03 00:40:51.790623 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:40:51.783566 4939 flags.go:64] FLAG: --http-check-frequency="20s" Nov 03 00:40:51.790623 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:40:51.783569 4939 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins" Nov 03 00:40:51.790623 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:40:51.783574 4939 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml" Nov 03 00:40:51.791185 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:40:51.783578 4939 flags.go:64] FLAG: --image-gc-high-threshold="85" Nov 03 00:40:51.791185 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:40:51.783581 4939 flags.go:64] FLAG: --image-gc-low-threshold="80" Nov 03 00:40:51.791185 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:40:51.783584 4939 flags.go:64] FLAG: --image-service-endpoint="" Nov 03 00:40:51.791185 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:40:51.783587 4939 flags.go:64] FLAG: --kernel-memcg-notification="false" Nov 03 00:40:51.791185 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:40:51.783591 4939 flags.go:64] FLAG: --kube-api-burst="100" Nov 03 00:40:51.791185 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:40:51.783594 4939 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf" Nov 03 00:40:51.791185 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:40:51.783597 4939 flags.go:64] FLAG: --kube-api-qps="50" Nov 03 00:40:51.791185 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:40:51.783600 4939 flags.go:64] FLAG: --kube-reserved="" Nov 03 00:40:51.791185 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:40:51.783603 4939 flags.go:64] FLAG: --kube-reserved-cgroup="" Nov 03 00:40:51.791185 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:40:51.783606 4939 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig" Nov 03 00:40:51.791185 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:40:51.783609 4939 flags.go:64] FLAG: --kubelet-cgroups="" Nov 03 00:40:51.791185 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:40:51.783612 4939 flags.go:64] FLAG: --local-storage-capacity-isolation="true" Nov 03 00:40:51.791185 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:40:51.783616 4939 flags.go:64] FLAG: --lock-file="" Nov 03 00:40:51.791185 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:40:51.783619 4939 flags.go:64] FLAG: --log-cadvisor-usage="false" Nov 03 00:40:51.791185 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:40:51.783622 4939 flags.go:64] FLAG: --log-flush-frequency="5s" Nov 03 00:40:51.791185 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:40:51.783625 4939 flags.go:64] FLAG: --log-json-info-buffer-size="0" Nov 03 00:40:51.791185 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:40:51.783630 4939 flags.go:64] FLAG: --log-json-split-stream="false" Nov 03 00:40:51.791185 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:40:51.783633 4939 flags.go:64] FLAG: --log-text-info-buffer-size="0" Nov 03 00:40:51.791185 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:40:51.783637 4939 flags.go:64] FLAG: --log-text-split-stream="false" Nov 03 00:40:51.791185 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:40:51.783640 4939 flags.go:64] FLAG: --logging-format="text" Nov 03 00:40:51.791185 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:40:51.783643 4939 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id" Nov 03 00:40:51.791185 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:40:51.783646 4939 flags.go:64] FLAG: --make-iptables-util-chains="true" Nov 03 00:40:51.791185 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:40:51.783649 4939 flags.go:64] FLAG: --manifest-url="" Nov 03 00:40:51.791185 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:40:51.783652 4939 flags.go:64] FLAG: --manifest-url-header="" Nov 03 00:40:51.791185 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:40:51.783657 4939 flags.go:64] FLAG: --max-housekeeping-interval="15s" Nov 03 00:40:51.791790 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:40:51.783660 4939 flags.go:64] FLAG: --max-open-files="1000000" Nov 03 00:40:51.791790 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:40:51.783665 4939 flags.go:64] FLAG: --max-pods="110" Nov 03 00:40:51.791790 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:40:51.783668 4939 flags.go:64] FLAG: --maximum-dead-containers="-1" Nov 03 00:40:51.791790 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:40:51.783671 4939 flags.go:64] FLAG: --maximum-dead-containers-per-container="1" Nov 03 00:40:51.791790 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:40:51.783674 4939 flags.go:64] FLAG: --memory-manager-policy="None" Nov 03 00:40:51.791790 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:40:51.783677 4939 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s" Nov 03 00:40:51.791790 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:40:51.783681 4939 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s" Nov 03 00:40:51.791790 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:40:51.783684 4939 flags.go:64] FLAG: --node-ip="0.0.0.0" Nov 03 00:40:51.791790 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:40:51.783687 4939 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel" Nov 03 00:40:51.791790 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:40:51.783694 4939 flags.go:64] FLAG: --node-status-max-images="50" Nov 03 00:40:51.791790 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:40:51.783697 4939 flags.go:64] FLAG: --node-status-update-frequency="10s" Nov 03 00:40:51.791790 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:40:51.783700 4939 flags.go:64] FLAG: --oom-score-adj="-999" Nov 03 00:40:51.791790 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:40:51.783732 4939 flags.go:64] FLAG: --pod-cidr="" Nov 03 00:40:51.791790 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:40:51.783735 4939 flags.go:64] FLAG: --pod-manifest-path="" Nov 03 00:40:51.791790 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:40:51.783739 4939 flags.go:64] FLAG: --pod-max-pids="-1" Nov 03 00:40:51.791790 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:40:51.783742 4939 flags.go:64] FLAG: --pods-per-core="0" Nov 03 00:40:51.791790 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:40:51.783745 4939 flags.go:64] FLAG: --port="10250" Nov 03 00:40:51.791790 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:40:51.783749 4939 flags.go:64] FLAG: --protect-kernel-defaults="false" Nov 03 00:40:51.791790 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:40:51.783752 4939 flags.go:64] FLAG: --provider-id="aws:///us-east-2a/i-02caf7f9962a007b4" Nov 03 00:40:51.791790 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:40:51.783755 4939 flags.go:64] FLAG: --qos-reserved="" Nov 03 00:40:51.791790 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:40:51.783758 4939 flags.go:64] FLAG: --read-only-port="10255" Nov 03 00:40:51.791790 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:40:51.783761 4939 flags.go:64] FLAG: --register-node="true" Nov 03 00:40:51.791790 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:40:51.783764 4939 flags.go:64] FLAG: --register-with-taints="" Nov 03 00:40:51.791790 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:40:51.783767 4939 flags.go:64] FLAG: --registry-burst="10" Nov 03 00:40:51.791790 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:40:51.783770 4939 flags.go:64] FLAG: --registry-qps="5" Nov 03 00:40:51.791790 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:40:51.783773 4939 flags.go:64] FLAG: --reserved-cpus="" Nov 03 00:40:51.792430 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:40:51.783776 4939 flags.go:64] FLAG: --reserved-memory="" Nov 03 00:40:51.792430 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:40:51.783780 4939 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf" Nov 03 00:40:51.792430 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:40:51.783783 4939 flags.go:64] FLAG: --root-dir="/var/lib/kubelet" Nov 03 00:40:51.792430 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:40:51.783786 4939 flags.go:64] FLAG: --rotate-certificates="false" Nov 03 00:40:51.792430 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:40:51.783790 4939 flags.go:64] FLAG: --rotate-server-certificates="false" Nov 03 00:40:51.792430 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:40:51.783793 4939 flags.go:64] FLAG: --runonce="false" Nov 03 00:40:51.792430 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:40:51.783796 4939 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service" Nov 03 00:40:51.792430 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:40:51.783801 4939 flags.go:64] FLAG: --runtime-request-timeout="2m0s" Nov 03 00:40:51.792430 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:40:51.783804 4939 flags.go:64] FLAG: --seccomp-default="false" Nov 03 00:40:51.792430 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:40:51.783807 4939 flags.go:64] FLAG: --serialize-image-pulls="true" Nov 03 00:40:51.792430 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:40:51.783810 4939 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s" Nov 03 00:40:51.792430 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:40:51.783813 4939 flags.go:64] FLAG: --storage-driver-db="cadvisor" Nov 03 00:40:51.792430 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:40:51.783817 4939 flags.go:64] FLAG: --storage-driver-host="localhost:8086" Nov 03 00:40:51.792430 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:40:51.783821 4939 flags.go:64] FLAG: --storage-driver-password="root" Nov 03 00:40:51.792430 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:40:51.783824 4939 flags.go:64] FLAG: --storage-driver-secure="false" Nov 03 00:40:51.792430 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:40:51.783827 4939 flags.go:64] FLAG: --storage-driver-table="stats" Nov 03 00:40:51.792430 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:40:51.783830 4939 flags.go:64] FLAG: --storage-driver-user="root" Nov 03 00:40:51.792430 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:40:51.783833 4939 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s" Nov 03 00:40:51.792430 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:40:51.783837 4939 flags.go:64] FLAG: --sync-frequency="1m0s" Nov 03 00:40:51.792430 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:40:51.783840 4939 flags.go:64] FLAG: --system-cgroups="" Nov 03 00:40:51.792430 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:40:51.783843 4939 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi" Nov 03 00:40:51.792430 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:40:51.783849 4939 flags.go:64] FLAG: --system-reserved-cgroup="" Nov 03 00:40:51.792430 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:40:51.783852 4939 flags.go:64] FLAG: --tls-cert-file="" Nov 03 00:40:51.792430 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:40:51.783855 4939 flags.go:64] FLAG: --tls-cipher-suites="[]" Nov 03 00:40:51.792430 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:40:51.783859 4939 flags.go:64] FLAG: --tls-min-version="" Nov 03 00:40:51.793064 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:40:51.783862 4939 flags.go:64] FLAG: --tls-private-key-file="" Nov 03 00:40:51.793064 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:40:51.783865 4939 flags.go:64] FLAG: --topology-manager-policy="none" Nov 03 00:40:51.793064 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:40:51.783868 4939 flags.go:64] FLAG: --topology-manager-policy-options="" Nov 03 00:40:51.793064 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:40:51.783871 4939 flags.go:64] FLAG: --topology-manager-scope="container" Nov 03 00:40:51.793064 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:40:51.783874 4939 flags.go:64] FLAG: --v="2" Nov 03 00:40:51.793064 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:40:51.783878 4939 flags.go:64] FLAG: --version="false" Nov 03 00:40:51.793064 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:40:51.783883 4939 flags.go:64] FLAG: --vmodule="" Nov 03 00:40:51.793064 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:40:51.783887 4939 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec" Nov 03 00:40:51.793064 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:40:51.783890 4939 flags.go:64] FLAG: --volume-stats-agg-period="1m0s" Nov 03 00:40:51.793064 ip-10-0-140-107 kubenswrapper[4939]: W1103 00:40:51.784050 4939 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 03 00:40:51.793064 ip-10-0-140-107 kubenswrapper[4939]: W1103 00:40:51.784055 4939 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 03 00:40:51.793064 ip-10-0-140-107 kubenswrapper[4939]: W1103 00:40:51.784059 4939 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 03 00:40:51.793064 ip-10-0-140-107 kubenswrapper[4939]: W1103 00:40:51.784061 4939 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 03 00:40:51.793064 ip-10-0-140-107 kubenswrapper[4939]: W1103 00:40:51.784064 4939 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 03 00:40:51.793064 ip-10-0-140-107 kubenswrapper[4939]: W1103 00:40:51.784068 4939 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 03 00:40:51.793064 ip-10-0-140-107 kubenswrapper[4939]: W1103 00:40:51.784071 4939 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 03 00:40:51.793064 ip-10-0-140-107 kubenswrapper[4939]: W1103 00:40:51.784074 4939 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 03 00:40:51.793064 ip-10-0-140-107 kubenswrapper[4939]: W1103 00:40:51.784076 4939 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 03 00:40:51.793064 ip-10-0-140-107 kubenswrapper[4939]: W1103 00:40:51.784078 4939 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 03 00:40:51.793064 ip-10-0-140-107 kubenswrapper[4939]: W1103 00:40:51.784081 4939 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 03 00:40:51.793064 ip-10-0-140-107 kubenswrapper[4939]: W1103 00:40:51.784084 4939 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 03 00:40:51.793064 ip-10-0-140-107 kubenswrapper[4939]: W1103 00:40:51.784086 4939 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 03 00:40:51.793593 ip-10-0-140-107 kubenswrapper[4939]: W1103 00:40:51.784089 4939 feature_gate.go:386] unrecognized feature gate: Example Nov 03 00:40:51.793593 ip-10-0-140-107 kubenswrapper[4939]: W1103 00:40:51.784091 4939 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 03 00:40:51.793593 ip-10-0-140-107 kubenswrapper[4939]: W1103 00:40:51.784094 4939 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 03 00:40:51.793593 ip-10-0-140-107 kubenswrapper[4939]: W1103 00:40:51.784096 4939 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 03 00:40:51.793593 ip-10-0-140-107 kubenswrapper[4939]: W1103 00:40:51.784099 4939 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 03 00:40:51.793593 ip-10-0-140-107 kubenswrapper[4939]: W1103 00:40:51.784102 4939 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 03 00:40:51.793593 ip-10-0-140-107 kubenswrapper[4939]: W1103 00:40:51.784105 4939 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 03 00:40:51.793593 ip-10-0-140-107 kubenswrapper[4939]: W1103 00:40:51.784107 4939 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 03 00:40:51.793593 ip-10-0-140-107 kubenswrapper[4939]: W1103 00:40:51.784110 4939 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 03 00:40:51.793593 ip-10-0-140-107 kubenswrapper[4939]: W1103 00:40:51.784113 4939 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 03 00:40:51.793593 ip-10-0-140-107 kubenswrapper[4939]: W1103 00:40:51.784116 4939 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 03 00:40:51.793593 ip-10-0-140-107 kubenswrapper[4939]: W1103 00:40:51.784119 4939 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 03 00:40:51.793593 ip-10-0-140-107 kubenswrapper[4939]: W1103 00:40:51.784122 4939 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 03 00:40:51.793593 ip-10-0-140-107 kubenswrapper[4939]: W1103 00:40:51.784124 4939 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 03 00:40:51.793593 ip-10-0-140-107 kubenswrapper[4939]: W1103 00:40:51.784127 4939 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 03 00:40:51.793593 ip-10-0-140-107 kubenswrapper[4939]: W1103 00:40:51.784129 4939 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 03 00:40:51.793593 ip-10-0-140-107 kubenswrapper[4939]: W1103 00:40:51.784132 4939 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 03 00:40:51.793593 ip-10-0-140-107 kubenswrapper[4939]: W1103 00:40:51.784135 4939 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 03 00:40:51.793593 ip-10-0-140-107 kubenswrapper[4939]: W1103 00:40:51.784137 4939 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 03 00:40:51.793593 ip-10-0-140-107 kubenswrapper[4939]: W1103 00:40:51.784140 4939 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 03 00:40:51.794098 ip-10-0-140-107 kubenswrapper[4939]: W1103 00:40:51.784143 4939 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 03 00:40:51.794098 ip-10-0-140-107 kubenswrapper[4939]: W1103 00:40:51.784145 4939 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 03 00:40:51.794098 ip-10-0-140-107 kubenswrapper[4939]: W1103 00:40:51.784148 4939 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 03 00:40:51.794098 ip-10-0-140-107 kubenswrapper[4939]: W1103 00:40:51.784150 4939 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 03 00:40:51.794098 ip-10-0-140-107 kubenswrapper[4939]: W1103 00:40:51.784155 4939 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 03 00:40:51.794098 ip-10-0-140-107 kubenswrapper[4939]: W1103 00:40:51.784158 4939 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 03 00:40:51.794098 ip-10-0-140-107 kubenswrapper[4939]: W1103 00:40:51.784161 4939 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 03 00:40:51.794098 ip-10-0-140-107 kubenswrapper[4939]: W1103 00:40:51.784163 4939 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 03 00:40:51.794098 ip-10-0-140-107 kubenswrapper[4939]: W1103 00:40:51.784166 4939 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 03 00:40:51.794098 ip-10-0-140-107 kubenswrapper[4939]: W1103 00:40:51.784168 4939 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 03 00:40:51.794098 ip-10-0-140-107 kubenswrapper[4939]: W1103 00:40:51.784171 4939 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 03 00:40:51.794098 ip-10-0-140-107 kubenswrapper[4939]: W1103 00:40:51.784174 4939 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 03 00:40:51.794098 ip-10-0-140-107 kubenswrapper[4939]: W1103 00:40:51.784176 4939 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 03 00:40:51.794098 ip-10-0-140-107 kubenswrapper[4939]: W1103 00:40:51.784179 4939 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 03 00:40:51.794098 ip-10-0-140-107 kubenswrapper[4939]: W1103 00:40:51.784182 4939 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 03 00:40:51.794098 ip-10-0-140-107 kubenswrapper[4939]: W1103 00:40:51.784184 4939 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 03 00:40:51.794098 ip-10-0-140-107 kubenswrapper[4939]: W1103 00:40:51.784187 4939 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 03 00:40:51.794098 ip-10-0-140-107 kubenswrapper[4939]: W1103 00:40:51.784190 4939 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 03 00:40:51.794098 ip-10-0-140-107 kubenswrapper[4939]: W1103 00:40:51.784193 4939 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 03 00:40:51.794649 ip-10-0-140-107 kubenswrapper[4939]: W1103 00:40:51.784196 4939 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 03 00:40:51.794649 ip-10-0-140-107 kubenswrapper[4939]: W1103 00:40:51.784199 4939 feature_gate.go:386] unrecognized feature gate: EVPN Nov 03 00:40:51.794649 ip-10-0-140-107 kubenswrapper[4939]: W1103 00:40:51.784201 4939 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 03 00:40:51.794649 ip-10-0-140-107 kubenswrapper[4939]: W1103 00:40:51.784204 4939 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 03 00:40:51.794649 ip-10-0-140-107 kubenswrapper[4939]: W1103 00:40:51.784206 4939 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 03 00:40:51.794649 ip-10-0-140-107 kubenswrapper[4939]: W1103 00:40:51.784209 4939 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 03 00:40:51.794649 ip-10-0-140-107 kubenswrapper[4939]: W1103 00:40:51.784211 4939 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 03 00:40:51.794649 ip-10-0-140-107 kubenswrapper[4939]: W1103 00:40:51.784214 4939 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 03 00:40:51.794649 ip-10-0-140-107 kubenswrapper[4939]: W1103 00:40:51.784217 4939 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 03 00:40:51.794649 ip-10-0-140-107 kubenswrapper[4939]: W1103 00:40:51.784221 4939 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 03 00:40:51.794649 ip-10-0-140-107 kubenswrapper[4939]: W1103 00:40:51.784224 4939 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 03 00:40:51.794649 ip-10-0-140-107 kubenswrapper[4939]: W1103 00:40:51.784229 4939 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 03 00:40:51.794649 ip-10-0-140-107 kubenswrapper[4939]: W1103 00:40:51.784232 4939 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 03 00:40:51.794649 ip-10-0-140-107 kubenswrapper[4939]: W1103 00:40:51.784235 4939 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 03 00:40:51.794649 ip-10-0-140-107 kubenswrapper[4939]: W1103 00:40:51.784238 4939 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 03 00:40:51.794649 ip-10-0-140-107 kubenswrapper[4939]: W1103 00:40:51.784240 4939 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 03 00:40:51.794649 ip-10-0-140-107 kubenswrapper[4939]: W1103 00:40:51.784243 4939 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 03 00:40:51.794649 ip-10-0-140-107 kubenswrapper[4939]: W1103 00:40:51.784248 4939 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 03 00:40:51.794649 ip-10-0-140-107 kubenswrapper[4939]: W1103 00:40:51.784250 4939 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 03 00:40:51.795256 ip-10-0-140-107 kubenswrapper[4939]: W1103 00:40:51.784253 4939 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 03 00:40:51.795256 ip-10-0-140-107 kubenswrapper[4939]: W1103 00:40:51.784256 4939 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 03 00:40:51.795256 ip-10-0-140-107 kubenswrapper[4939]: W1103 00:40:51.784258 4939 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 03 00:40:51.795256 ip-10-0-140-107 kubenswrapper[4939]: W1103 00:40:51.784261 4939 feature_gate.go:386] unrecognized feature gate: Example2 Nov 03 00:40:51.795256 ip-10-0-140-107 kubenswrapper[4939]: W1103 00:40:51.784264 4939 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 03 00:40:51.795256 ip-10-0-140-107 kubenswrapper[4939]: W1103 00:40:51.784266 4939 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 03 00:40:51.795256 ip-10-0-140-107 kubenswrapper[4939]: W1103 00:40:51.784269 4939 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 03 00:40:51.795256 ip-10-0-140-107 kubenswrapper[4939]: W1103 00:40:51.784271 4939 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 03 00:40:51.795256 ip-10-0-140-107 kubenswrapper[4939]: W1103 00:40:51.784274 4939 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 03 00:40:51.795256 ip-10-0-140-107 kubenswrapper[4939]: W1103 00:40:51.784277 4939 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 03 00:40:51.795256 ip-10-0-140-107 kubenswrapper[4939]: W1103 00:40:51.784279 4939 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 03 00:40:51.795256 ip-10-0-140-107 kubenswrapper[4939]: W1103 00:40:51.784282 4939 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 03 00:40:51.795256 ip-10-0-140-107 kubenswrapper[4939]: W1103 00:40:51.784284 4939 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 03 00:40:51.795256 ip-10-0-140-107 kubenswrapper[4939]: W1103 00:40:51.784287 4939 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 03 00:40:51.795256 ip-10-0-140-107 kubenswrapper[4939]: W1103 00:40:51.784290 4939 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 03 00:40:51.795256 ip-10-0-140-107 kubenswrapper[4939]: W1103 00:40:51.784293 4939 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 03 00:40:51.795256 ip-10-0-140-107 kubenswrapper[4939]: W1103 00:40:51.784296 4939 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 03 00:40:51.795256 ip-10-0-140-107 kubenswrapper[4939]: W1103 00:40:51.784298 4939 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 03 00:40:51.795256 ip-10-0-140-107 kubenswrapper[4939]: W1103 00:40:51.784301 4939 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 03 00:40:51.795256 ip-10-0-140-107 kubenswrapper[4939]: W1103 00:40:51.784304 4939 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 03 00:40:51.795904 ip-10-0-140-107 kubenswrapper[4939]: W1103 00:40:51.784306 4939 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 03 00:40:51.795904 ip-10-0-140-107 kubenswrapper[4939]: W1103 00:40:51.784309 4939 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 03 00:40:51.795904 ip-10-0-140-107 kubenswrapper[4939]: W1103 00:40:51.784311 4939 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 03 00:40:51.795904 ip-10-0-140-107 kubenswrapper[4939]: W1103 00:40:51.784314 4939 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 03 00:40:51.795904 ip-10-0-140-107 kubenswrapper[4939]: W1103 00:40:51.784318 4939 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 03 00:40:51.795904 ip-10-0-140-107 kubenswrapper[4939]: W1103 00:40:51.784321 4939 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 03 00:40:51.795904 ip-10-0-140-107 kubenswrapper[4939]: W1103 00:40:51.784324 4939 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 03 00:40:51.795904 ip-10-0-140-107 kubenswrapper[4939]: W1103 00:40:51.784328 4939 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 03 00:40:51.795904 ip-10-0-140-107 kubenswrapper[4939]: W1103 00:40:51.784331 4939 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 03 00:40:51.795904 ip-10-0-140-107 kubenswrapper[4939]: W1103 00:40:51.784334 4939 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 03 00:40:51.795904 ip-10-0-140-107 kubenswrapper[4939]: W1103 00:40:51.784338 4939 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 03 00:40:51.795904 ip-10-0-140-107 kubenswrapper[4939]: W1103 00:40:51.784341 4939 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 03 00:40:51.795904 ip-10-0-140-107 kubenswrapper[4939]: W1103 00:40:51.784343 4939 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 03 00:40:51.795904 ip-10-0-140-107 kubenswrapper[4939]: W1103 00:40:51.784346 4939 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 03 00:40:51.795904 ip-10-0-140-107 kubenswrapper[4939]: W1103 00:40:51.784348 4939 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 03 00:40:51.795904 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:40:51.784377 4939 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Nov 03 00:40:51.796402 ip-10-0-140-107 kubenswrapper[4939]: W1103 00:40:51.784416 4939 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 03 00:40:51.796402 ip-10-0-140-107 kubenswrapper[4939]: W1103 00:40:51.784421 4939 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 03 00:40:51.796402 ip-10-0-140-107 kubenswrapper[4939]: W1103 00:40:51.784424 4939 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 03 00:40:51.796402 ip-10-0-140-107 kubenswrapper[4939]: W1103 00:40:51.784427 4939 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 03 00:40:51.796402 ip-10-0-140-107 kubenswrapper[4939]: W1103 00:40:51.784430 4939 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 03 00:40:51.796402 ip-10-0-140-107 kubenswrapper[4939]: W1103 00:40:51.784433 4939 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 03 00:40:51.796402 ip-10-0-140-107 kubenswrapper[4939]: W1103 00:40:51.784435 4939 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 03 00:40:51.796402 ip-10-0-140-107 kubenswrapper[4939]: W1103 00:40:51.784438 4939 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 03 00:40:51.796402 ip-10-0-140-107 kubenswrapper[4939]: W1103 00:40:51.784440 4939 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 03 00:40:51.796402 ip-10-0-140-107 kubenswrapper[4939]: W1103 00:40:51.784443 4939 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 03 00:40:51.796402 ip-10-0-140-107 kubenswrapper[4939]: W1103 00:40:51.784446 4939 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 03 00:40:51.796402 ip-10-0-140-107 kubenswrapper[4939]: W1103 00:40:51.784459 4939 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 03 00:40:51.796402 ip-10-0-140-107 kubenswrapper[4939]: W1103 00:40:51.784462 4939 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 03 00:40:51.796402 ip-10-0-140-107 kubenswrapper[4939]: W1103 00:40:51.784465 4939 feature_gate.go:386] unrecognized feature gate: Example Nov 03 00:40:51.796402 ip-10-0-140-107 kubenswrapper[4939]: W1103 00:40:51.784468 4939 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 03 00:40:51.796402 ip-10-0-140-107 kubenswrapper[4939]: W1103 00:40:51.784471 4939 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 03 00:40:51.796402 ip-10-0-140-107 kubenswrapper[4939]: W1103 00:40:51.784473 4939 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 03 00:40:51.796402 ip-10-0-140-107 kubenswrapper[4939]: W1103 00:40:51.784476 4939 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 03 00:40:51.796402 ip-10-0-140-107 kubenswrapper[4939]: W1103 00:40:51.784479 4939 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 03 00:40:51.796402 ip-10-0-140-107 kubenswrapper[4939]: W1103 00:40:51.784481 4939 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 03 00:40:51.796975 ip-10-0-140-107 kubenswrapper[4939]: W1103 00:40:51.784484 4939 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 03 00:40:51.796975 ip-10-0-140-107 kubenswrapper[4939]: W1103 00:40:51.784489 4939 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 03 00:40:51.796975 ip-10-0-140-107 kubenswrapper[4939]: W1103 00:40:51.784492 4939 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 03 00:40:51.796975 ip-10-0-140-107 kubenswrapper[4939]: W1103 00:40:51.784494 4939 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 03 00:40:51.796975 ip-10-0-140-107 kubenswrapper[4939]: W1103 00:40:51.784497 4939 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 03 00:40:51.796975 ip-10-0-140-107 kubenswrapper[4939]: W1103 00:40:51.784500 4939 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 03 00:40:51.796975 ip-10-0-140-107 kubenswrapper[4939]: W1103 00:40:51.784503 4939 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 03 00:40:51.796975 ip-10-0-140-107 kubenswrapper[4939]: W1103 00:40:51.784506 4939 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 03 00:40:51.796975 ip-10-0-140-107 kubenswrapper[4939]: W1103 00:40:51.784509 4939 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 03 00:40:51.796975 ip-10-0-140-107 kubenswrapper[4939]: W1103 00:40:51.784512 4939 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 03 00:40:51.796975 ip-10-0-140-107 kubenswrapper[4939]: W1103 00:40:51.784514 4939 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 03 00:40:51.796975 ip-10-0-140-107 kubenswrapper[4939]: W1103 00:40:51.784517 4939 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 03 00:40:51.796975 ip-10-0-140-107 kubenswrapper[4939]: W1103 00:40:51.784520 4939 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 03 00:40:51.796975 ip-10-0-140-107 kubenswrapper[4939]: W1103 00:40:51.784522 4939 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 03 00:40:51.796975 ip-10-0-140-107 kubenswrapper[4939]: W1103 00:40:51.784525 4939 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 03 00:40:51.796975 ip-10-0-140-107 kubenswrapper[4939]: W1103 00:40:51.784528 4939 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 03 00:40:51.796975 ip-10-0-140-107 kubenswrapper[4939]: W1103 00:40:51.784530 4939 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 03 00:40:51.796975 ip-10-0-140-107 kubenswrapper[4939]: W1103 00:40:51.784533 4939 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 03 00:40:51.796975 ip-10-0-140-107 kubenswrapper[4939]: W1103 00:40:51.784535 4939 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 03 00:40:51.796975 ip-10-0-140-107 kubenswrapper[4939]: W1103 00:40:51.784538 4939 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 03 00:40:51.797558 ip-10-0-140-107 kubenswrapper[4939]: W1103 00:40:51.784541 4939 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 03 00:40:51.797558 ip-10-0-140-107 kubenswrapper[4939]: W1103 00:40:51.784544 4939 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 03 00:40:51.797558 ip-10-0-140-107 kubenswrapper[4939]: W1103 00:40:51.784546 4939 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 03 00:40:51.797558 ip-10-0-140-107 kubenswrapper[4939]: W1103 00:40:51.784550 4939 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 03 00:40:51.797558 ip-10-0-140-107 kubenswrapper[4939]: W1103 00:40:51.784552 4939 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 03 00:40:51.797558 ip-10-0-140-107 kubenswrapper[4939]: W1103 00:40:51.784555 4939 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 03 00:40:51.797558 ip-10-0-140-107 kubenswrapper[4939]: W1103 00:40:51.784558 4939 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 03 00:40:51.797558 ip-10-0-140-107 kubenswrapper[4939]: W1103 00:40:51.784561 4939 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 03 00:40:51.797558 ip-10-0-140-107 kubenswrapper[4939]: W1103 00:40:51.784563 4939 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 03 00:40:51.797558 ip-10-0-140-107 kubenswrapper[4939]: W1103 00:40:51.784566 4939 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 03 00:40:51.797558 ip-10-0-140-107 kubenswrapper[4939]: W1103 00:40:51.784568 4939 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 03 00:40:51.797558 ip-10-0-140-107 kubenswrapper[4939]: W1103 00:40:51.784571 4939 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 03 00:40:51.797558 ip-10-0-140-107 kubenswrapper[4939]: W1103 00:40:51.784573 4939 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 03 00:40:51.797558 ip-10-0-140-107 kubenswrapper[4939]: W1103 00:40:51.784577 4939 feature_gate.go:386] unrecognized feature gate: EVPN Nov 03 00:40:51.797558 ip-10-0-140-107 kubenswrapper[4939]: W1103 00:40:51.784580 4939 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 03 00:40:51.797558 ip-10-0-140-107 kubenswrapper[4939]: W1103 00:40:51.784582 4939 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 03 00:40:51.797558 ip-10-0-140-107 kubenswrapper[4939]: W1103 00:40:51.784585 4939 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 03 00:40:51.797558 ip-10-0-140-107 kubenswrapper[4939]: W1103 00:40:51.784588 4939 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 03 00:40:51.797558 ip-10-0-140-107 kubenswrapper[4939]: W1103 00:40:51.784592 4939 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 03 00:40:51.798297 ip-10-0-140-107 kubenswrapper[4939]: W1103 00:40:51.784595 4939 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 03 00:40:51.798297 ip-10-0-140-107 kubenswrapper[4939]: W1103 00:40:51.784597 4939 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 03 00:40:51.798297 ip-10-0-140-107 kubenswrapper[4939]: W1103 00:40:51.784602 4939 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 03 00:40:51.798297 ip-10-0-140-107 kubenswrapper[4939]: W1103 00:40:51.784606 4939 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 03 00:40:51.798297 ip-10-0-140-107 kubenswrapper[4939]: W1103 00:40:51.784609 4939 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 03 00:40:51.798297 ip-10-0-140-107 kubenswrapper[4939]: W1103 00:40:51.784612 4939 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 03 00:40:51.798297 ip-10-0-140-107 kubenswrapper[4939]: W1103 00:40:51.784615 4939 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 03 00:40:51.798297 ip-10-0-140-107 kubenswrapper[4939]: W1103 00:40:51.784617 4939 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 03 00:40:51.798297 ip-10-0-140-107 kubenswrapper[4939]: W1103 00:40:51.784620 4939 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 03 00:40:51.798297 ip-10-0-140-107 kubenswrapper[4939]: W1103 00:40:51.784622 4939 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 03 00:40:51.798297 ip-10-0-140-107 kubenswrapper[4939]: W1103 00:40:51.784625 4939 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 03 00:40:51.798297 ip-10-0-140-107 kubenswrapper[4939]: W1103 00:40:51.784628 4939 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 03 00:40:51.798297 ip-10-0-140-107 kubenswrapper[4939]: W1103 00:40:51.784630 4939 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 03 00:40:51.798297 ip-10-0-140-107 kubenswrapper[4939]: W1103 00:40:51.784633 4939 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 03 00:40:51.798297 ip-10-0-140-107 kubenswrapper[4939]: W1103 00:40:51.784636 4939 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 03 00:40:51.798297 ip-10-0-140-107 kubenswrapper[4939]: W1103 00:40:51.784638 4939 feature_gate.go:386] unrecognized feature gate: Example2 Nov 03 00:40:51.798297 ip-10-0-140-107 kubenswrapper[4939]: W1103 00:40:51.784641 4939 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 03 00:40:51.798297 ip-10-0-140-107 kubenswrapper[4939]: W1103 00:40:51.784644 4939 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 03 00:40:51.798297 ip-10-0-140-107 kubenswrapper[4939]: W1103 00:40:51.784647 4939 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 03 00:40:51.799158 ip-10-0-140-107 kubenswrapper[4939]: W1103 00:40:51.784649 4939 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 03 00:40:51.799158 ip-10-0-140-107 kubenswrapper[4939]: W1103 00:40:51.784652 4939 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 03 00:40:51.799158 ip-10-0-140-107 kubenswrapper[4939]: W1103 00:40:51.784654 4939 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 03 00:40:51.799158 ip-10-0-140-107 kubenswrapper[4939]: W1103 00:40:51.784657 4939 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 03 00:40:51.799158 ip-10-0-140-107 kubenswrapper[4939]: W1103 00:40:51.784659 4939 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 03 00:40:51.799158 ip-10-0-140-107 kubenswrapper[4939]: W1103 00:40:51.784662 4939 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 03 00:40:51.799158 ip-10-0-140-107 kubenswrapper[4939]: W1103 00:40:51.784665 4939 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 03 00:40:51.799158 ip-10-0-140-107 kubenswrapper[4939]: W1103 00:40:51.784668 4939 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 03 00:40:51.799158 ip-10-0-140-107 kubenswrapper[4939]: W1103 00:40:51.784672 4939 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 03 00:40:51.799158 ip-10-0-140-107 kubenswrapper[4939]: W1103 00:40:51.784674 4939 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 03 00:40:51.799158 ip-10-0-140-107 kubenswrapper[4939]: W1103 00:40:51.784677 4939 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 03 00:40:51.799158 ip-10-0-140-107 kubenswrapper[4939]: W1103 00:40:51.784680 4939 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 03 00:40:51.799158 ip-10-0-140-107 kubenswrapper[4939]: W1103 00:40:51.784683 4939 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 03 00:40:51.799158 ip-10-0-140-107 kubenswrapper[4939]: W1103 00:40:51.784686 4939 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 03 00:40:51.799158 ip-10-0-140-107 kubenswrapper[4939]: W1103 00:40:51.784689 4939 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 03 00:40:51.799158 ip-10-0-140-107 kubenswrapper[4939]: W1103 00:40:51.784691 4939 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 03 00:40:51.799158 ip-10-0-140-107 kubenswrapper[4939]: W1103 00:40:51.784694 4939 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 03 00:40:51.799158 ip-10-0-140-107 kubenswrapper[4939]: W1103 00:40:51.784697 4939 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 03 00:40:51.799158 ip-10-0-140-107 kubenswrapper[4939]: W1103 00:40:51.784700 4939 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 03 00:40:51.799158 ip-10-0-140-107 kubenswrapper[4939]: W1103 00:40:51.784702 4939 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 03 00:40:51.799998 ip-10-0-140-107 kubenswrapper[4939]: W1103 00:40:51.784706 4939 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 03 00:40:51.799998 ip-10-0-140-107 kubenswrapper[4939]: W1103 00:40:51.784710 4939 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 03 00:40:51.799998 ip-10-0-140-107 kubenswrapper[4939]: W1103 00:40:51.784713 4939 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 03 00:40:51.799998 ip-10-0-140-107 kubenswrapper[4939]: W1103 00:40:51.784715 4939 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 03 00:40:51.799998 ip-10-0-140-107 kubenswrapper[4939]: W1103 00:40:51.784718 4939 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 03 00:40:51.799998 ip-10-0-140-107 kubenswrapper[4939]: W1103 00:40:51.784721 4939 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 03 00:40:51.799998 ip-10-0-140-107 kubenswrapper[4939]: W1103 00:40:51.784724 4939 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 03 00:40:51.799998 ip-10-0-140-107 kubenswrapper[4939]: W1103 00:40:51.784727 4939 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 03 00:40:51.799998 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:40:51.789836 4939 server.go:525] "Kubelet version" kubeletVersion="v1.35.9" Nov 03 00:40:51.799998 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:40:51.789853 4939 server.go:527] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK="" Nov 03 00:40:51.799998 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:40:51.789876 4939 watchdog_linux.go:95] "Systemd watchdog is not enabled" Nov 03 00:40:51.799998 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:40:51.789881 4939 watchdog_linux.go:138] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started." Nov 03 00:40:51.799998 ip-10-0-140-107 kubenswrapper[4939]: W1103 00:40:51.789952 4939 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 03 00:40:51.799998 ip-10-0-140-107 kubenswrapper[4939]: W1103 00:40:51.789958 4939 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 03 00:40:51.799998 ip-10-0-140-107 kubenswrapper[4939]: W1103 00:40:51.789963 4939 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 03 00:40:51.799998 ip-10-0-140-107 kubenswrapper[4939]: W1103 00:40:51.789967 4939 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 03 00:40:51.799998 ip-10-0-140-107 kubenswrapper[4939]: W1103 00:40:51.789970 4939 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 03 00:40:51.799998 ip-10-0-140-107 kubenswrapper[4939]: W1103 00:40:51.789972 4939 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 03 00:40:51.799998 ip-10-0-140-107 kubenswrapper[4939]: W1103 00:40:51.789975 4939 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 03 00:40:51.800783 ip-10-0-140-107 kubenswrapper[4939]: W1103 00:40:51.789978 4939 feature_gate.go:386] unrecognized feature gate: Example2 Nov 03 00:40:51.800783 ip-10-0-140-107 kubenswrapper[4939]: W1103 00:40:51.789981 4939 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 03 00:40:51.800783 ip-10-0-140-107 kubenswrapper[4939]: W1103 00:40:51.789984 4939 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 03 00:40:51.800783 ip-10-0-140-107 kubenswrapper[4939]: W1103 00:40:51.789987 4939 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 03 00:40:51.800783 ip-10-0-140-107 kubenswrapper[4939]: W1103 00:40:51.789990 4939 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 03 00:40:51.800783 ip-10-0-140-107 kubenswrapper[4939]: W1103 00:40:51.789993 4939 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 03 00:40:51.800783 ip-10-0-140-107 kubenswrapper[4939]: W1103 00:40:51.789996 4939 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 03 00:40:51.800783 ip-10-0-140-107 kubenswrapper[4939]: W1103 00:40:51.789999 4939 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 03 00:40:51.800783 ip-10-0-140-107 kubenswrapper[4939]: W1103 00:40:51.790002 4939 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 03 00:40:51.800783 ip-10-0-140-107 kubenswrapper[4939]: W1103 00:40:51.790005 4939 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 03 00:40:51.800783 ip-10-0-140-107 kubenswrapper[4939]: W1103 00:40:51.790008 4939 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 03 00:40:51.800783 ip-10-0-140-107 kubenswrapper[4939]: W1103 00:40:51.790011 4939 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 03 00:40:51.800783 ip-10-0-140-107 kubenswrapper[4939]: W1103 00:40:51.790013 4939 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 03 00:40:51.800783 ip-10-0-140-107 kubenswrapper[4939]: W1103 00:40:51.790016 4939 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 03 00:40:51.800783 ip-10-0-140-107 kubenswrapper[4939]: W1103 00:40:51.790018 4939 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 03 00:40:51.800783 ip-10-0-140-107 kubenswrapper[4939]: W1103 00:40:51.790021 4939 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 03 00:40:51.800783 ip-10-0-140-107 kubenswrapper[4939]: W1103 00:40:51.790024 4939 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 03 00:40:51.800783 ip-10-0-140-107 kubenswrapper[4939]: W1103 00:40:51.790027 4939 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 03 00:40:51.800783 ip-10-0-140-107 kubenswrapper[4939]: W1103 00:40:51.790030 4939 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 03 00:40:51.800783 ip-10-0-140-107 kubenswrapper[4939]: W1103 00:40:51.790032 4939 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 03 00:40:51.801662 ip-10-0-140-107 kubenswrapper[4939]: W1103 00:40:51.790036 4939 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 03 00:40:51.801662 ip-10-0-140-107 kubenswrapper[4939]: W1103 00:40:51.790039 4939 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 03 00:40:51.801662 ip-10-0-140-107 kubenswrapper[4939]: W1103 00:40:51.790041 4939 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 03 00:40:51.801662 ip-10-0-140-107 kubenswrapper[4939]: W1103 00:40:51.790044 4939 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 03 00:40:51.801662 ip-10-0-140-107 kubenswrapper[4939]: W1103 00:40:51.790047 4939 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 03 00:40:51.801662 ip-10-0-140-107 kubenswrapper[4939]: W1103 00:40:51.790049 4939 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 03 00:40:51.801662 ip-10-0-140-107 kubenswrapper[4939]: W1103 00:40:51.790052 4939 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 03 00:40:51.801662 ip-10-0-140-107 kubenswrapper[4939]: W1103 00:40:51.790055 4939 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 03 00:40:51.801662 ip-10-0-140-107 kubenswrapper[4939]: W1103 00:40:51.790057 4939 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 03 00:40:51.801662 ip-10-0-140-107 kubenswrapper[4939]: W1103 00:40:51.790060 4939 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 03 00:40:51.801662 ip-10-0-140-107 kubenswrapper[4939]: W1103 00:40:51.790062 4939 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 03 00:40:51.801662 ip-10-0-140-107 kubenswrapper[4939]: W1103 00:40:51.790065 4939 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 03 00:40:51.801662 ip-10-0-140-107 kubenswrapper[4939]: W1103 00:40:51.790067 4939 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 03 00:40:51.801662 ip-10-0-140-107 kubenswrapper[4939]: W1103 00:40:51.790070 4939 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 03 00:40:51.801662 ip-10-0-140-107 kubenswrapper[4939]: W1103 00:40:51.790073 4939 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 03 00:40:51.801662 ip-10-0-140-107 kubenswrapper[4939]: W1103 00:40:51.790076 4939 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 03 00:40:51.801662 ip-10-0-140-107 kubenswrapper[4939]: W1103 00:40:51.790079 4939 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 03 00:40:51.801662 ip-10-0-140-107 kubenswrapper[4939]: W1103 00:40:51.790082 4939 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 03 00:40:51.801662 ip-10-0-140-107 kubenswrapper[4939]: W1103 00:40:51.790084 4939 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 03 00:40:51.801662 ip-10-0-140-107 kubenswrapper[4939]: W1103 00:40:51.790087 4939 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 03 00:40:51.802453 ip-10-0-140-107 kubenswrapper[4939]: W1103 00:40:51.790090 4939 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 03 00:40:51.802453 ip-10-0-140-107 kubenswrapper[4939]: W1103 00:40:51.790093 4939 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 03 00:40:51.802453 ip-10-0-140-107 kubenswrapper[4939]: W1103 00:40:51.790096 4939 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 03 00:40:51.802453 ip-10-0-140-107 kubenswrapper[4939]: W1103 00:40:51.790099 4939 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 03 00:40:51.802453 ip-10-0-140-107 kubenswrapper[4939]: W1103 00:40:51.790101 4939 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 03 00:40:51.802453 ip-10-0-140-107 kubenswrapper[4939]: W1103 00:40:51.790104 4939 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 03 00:40:51.802453 ip-10-0-140-107 kubenswrapper[4939]: W1103 00:40:51.790107 4939 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 03 00:40:51.802453 ip-10-0-140-107 kubenswrapper[4939]: W1103 00:40:51.790110 4939 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 03 00:40:51.802453 ip-10-0-140-107 kubenswrapper[4939]: W1103 00:40:51.790112 4939 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 03 00:40:51.802453 ip-10-0-140-107 kubenswrapper[4939]: W1103 00:40:51.790115 4939 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 03 00:40:51.802453 ip-10-0-140-107 kubenswrapper[4939]: W1103 00:40:51.790117 4939 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 03 00:40:51.802453 ip-10-0-140-107 kubenswrapper[4939]: W1103 00:40:51.790120 4939 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 03 00:40:51.802453 ip-10-0-140-107 kubenswrapper[4939]: W1103 00:40:51.790123 4939 feature_gate.go:386] unrecognized feature gate: EVPN Nov 03 00:40:51.802453 ip-10-0-140-107 kubenswrapper[4939]: W1103 00:40:51.790126 4939 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 03 00:40:51.802453 ip-10-0-140-107 kubenswrapper[4939]: W1103 00:40:51.790128 4939 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 03 00:40:51.802453 ip-10-0-140-107 kubenswrapper[4939]: W1103 00:40:51.790131 4939 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 03 00:40:51.802453 ip-10-0-140-107 kubenswrapper[4939]: W1103 00:40:51.790133 4939 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 03 00:40:51.802453 ip-10-0-140-107 kubenswrapper[4939]: W1103 00:40:51.790136 4939 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 03 00:40:51.802453 ip-10-0-140-107 kubenswrapper[4939]: W1103 00:40:51.790138 4939 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 03 00:40:51.802453 ip-10-0-140-107 kubenswrapper[4939]: W1103 00:40:51.790141 4939 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 03 00:40:51.803311 ip-10-0-140-107 kubenswrapper[4939]: W1103 00:40:51.790144 4939 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 03 00:40:51.803311 ip-10-0-140-107 kubenswrapper[4939]: W1103 00:40:51.790147 4939 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 03 00:40:51.803311 ip-10-0-140-107 kubenswrapper[4939]: W1103 00:40:51.790150 4939 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 03 00:40:51.803311 ip-10-0-140-107 kubenswrapper[4939]: W1103 00:40:51.790152 4939 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 03 00:40:51.803311 ip-10-0-140-107 kubenswrapper[4939]: W1103 00:40:51.790155 4939 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 03 00:40:51.803311 ip-10-0-140-107 kubenswrapper[4939]: W1103 00:40:51.790158 4939 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 03 00:40:51.803311 ip-10-0-140-107 kubenswrapper[4939]: W1103 00:40:51.790160 4939 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 03 00:40:51.803311 ip-10-0-140-107 kubenswrapper[4939]: W1103 00:40:51.790163 4939 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 03 00:40:51.803311 ip-10-0-140-107 kubenswrapper[4939]: W1103 00:40:51.790166 4939 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 03 00:40:51.803311 ip-10-0-140-107 kubenswrapper[4939]: W1103 00:40:51.790168 4939 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 03 00:40:51.803311 ip-10-0-140-107 kubenswrapper[4939]: W1103 00:40:51.790171 4939 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 03 00:40:51.803311 ip-10-0-140-107 kubenswrapper[4939]: W1103 00:40:51.790173 4939 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 03 00:40:51.803311 ip-10-0-140-107 kubenswrapper[4939]: W1103 00:40:51.790177 4939 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 03 00:40:51.803311 ip-10-0-140-107 kubenswrapper[4939]: W1103 00:40:51.790180 4939 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 03 00:40:51.803311 ip-10-0-140-107 kubenswrapper[4939]: W1103 00:40:51.790183 4939 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 03 00:40:51.803311 ip-10-0-140-107 kubenswrapper[4939]: W1103 00:40:51.790185 4939 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 03 00:40:51.803311 ip-10-0-140-107 kubenswrapper[4939]: W1103 00:40:51.790188 4939 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 03 00:40:51.803311 ip-10-0-140-107 kubenswrapper[4939]: W1103 00:40:51.790190 4939 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 03 00:40:51.803311 ip-10-0-140-107 kubenswrapper[4939]: W1103 00:40:51.790193 4939 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 03 00:40:51.804142 ip-10-0-140-107 kubenswrapper[4939]: W1103 00:40:51.790195 4939 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 03 00:40:51.804142 ip-10-0-140-107 kubenswrapper[4939]: W1103 00:40:51.790198 4939 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 03 00:40:51.804142 ip-10-0-140-107 kubenswrapper[4939]: W1103 00:40:51.790200 4939 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 03 00:40:51.804142 ip-10-0-140-107 kubenswrapper[4939]: W1103 00:40:51.790203 4939 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 03 00:40:51.804142 ip-10-0-140-107 kubenswrapper[4939]: W1103 00:40:51.790206 4939 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 03 00:40:51.804142 ip-10-0-140-107 kubenswrapper[4939]: W1103 00:40:51.790209 4939 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 03 00:40:51.804142 ip-10-0-140-107 kubenswrapper[4939]: W1103 00:40:51.790212 4939 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 03 00:40:51.804142 ip-10-0-140-107 kubenswrapper[4939]: W1103 00:40:51.790215 4939 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 03 00:40:51.804142 ip-10-0-140-107 kubenswrapper[4939]: W1103 00:40:51.790217 4939 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 03 00:40:51.804142 ip-10-0-140-107 kubenswrapper[4939]: W1103 00:40:51.790220 4939 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 03 00:40:51.804142 ip-10-0-140-107 kubenswrapper[4939]: W1103 00:40:51.790222 4939 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 03 00:40:51.804142 ip-10-0-140-107 kubenswrapper[4939]: W1103 00:40:51.790226 4939 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 03 00:40:51.804142 ip-10-0-140-107 kubenswrapper[4939]: W1103 00:40:51.790230 4939 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 03 00:40:51.804142 ip-10-0-140-107 kubenswrapper[4939]: W1103 00:40:51.790233 4939 feature_gate.go:386] unrecognized feature gate: Example Nov 03 00:40:51.804142 ip-10-0-140-107 kubenswrapper[4939]: W1103 00:40:51.790236 4939 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 03 00:40:51.804142 ip-10-0-140-107 kubenswrapper[4939]: W1103 00:40:51.790239 4939 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 03 00:40:51.804142 ip-10-0-140-107 kubenswrapper[4939]: W1103 00:40:51.790241 4939 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 03 00:40:51.804142 ip-10-0-140-107 kubenswrapper[4939]: W1103 00:40:51.790244 4939 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 03 00:40:51.804142 ip-10-0-140-107 kubenswrapper[4939]: W1103 00:40:51.790247 4939 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 03 00:40:51.805193 ip-10-0-140-107 kubenswrapper[4939]: W1103 00:40:51.790249 4939 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 03 00:40:51.805193 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:40:51.790277 4939 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Nov 03 00:40:51.805193 ip-10-0-140-107 kubenswrapper[4939]: W1103 00:40:51.790425 4939 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 03 00:40:51.805193 ip-10-0-140-107 kubenswrapper[4939]: W1103 00:40:51.790430 4939 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 03 00:40:51.805193 ip-10-0-140-107 kubenswrapper[4939]: W1103 00:40:51.790433 4939 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 03 00:40:51.805193 ip-10-0-140-107 kubenswrapper[4939]: W1103 00:40:51.790436 4939 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 03 00:40:51.805193 ip-10-0-140-107 kubenswrapper[4939]: W1103 00:40:51.790439 4939 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 03 00:40:51.805193 ip-10-0-140-107 kubenswrapper[4939]: W1103 00:40:51.790442 4939 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 03 00:40:51.805193 ip-10-0-140-107 kubenswrapper[4939]: W1103 00:40:51.790445 4939 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 03 00:40:51.805193 ip-10-0-140-107 kubenswrapper[4939]: W1103 00:40:51.790448 4939 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 03 00:40:51.805193 ip-10-0-140-107 kubenswrapper[4939]: W1103 00:40:51.790451 4939 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 03 00:40:51.805193 ip-10-0-140-107 kubenswrapper[4939]: W1103 00:40:51.790453 4939 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 03 00:40:51.805193 ip-10-0-140-107 kubenswrapper[4939]: W1103 00:40:51.790456 4939 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 03 00:40:51.805193 ip-10-0-140-107 kubenswrapper[4939]: W1103 00:40:51.790459 4939 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 03 00:40:51.805193 ip-10-0-140-107 kubenswrapper[4939]: W1103 00:40:51.790461 4939 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 03 00:40:51.805193 ip-10-0-140-107 kubenswrapper[4939]: W1103 00:40:51.790464 4939 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 03 00:40:51.805850 ip-10-0-140-107 kubenswrapper[4939]: W1103 00:40:51.790467 4939 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 03 00:40:51.805850 ip-10-0-140-107 kubenswrapper[4939]: W1103 00:40:51.790470 4939 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 03 00:40:51.805850 ip-10-0-140-107 kubenswrapper[4939]: W1103 00:40:51.790472 4939 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 03 00:40:51.805850 ip-10-0-140-107 kubenswrapper[4939]: W1103 00:40:51.790475 4939 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 03 00:40:51.805850 ip-10-0-140-107 kubenswrapper[4939]: W1103 00:40:51.790478 4939 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 03 00:40:51.805850 ip-10-0-140-107 kubenswrapper[4939]: W1103 00:40:51.790481 4939 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 03 00:40:51.805850 ip-10-0-140-107 kubenswrapper[4939]: W1103 00:40:51.790483 4939 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 03 00:40:51.805850 ip-10-0-140-107 kubenswrapper[4939]: W1103 00:40:51.790486 4939 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 03 00:40:51.805850 ip-10-0-140-107 kubenswrapper[4939]: W1103 00:40:51.790488 4939 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 03 00:40:51.805850 ip-10-0-140-107 kubenswrapper[4939]: W1103 00:40:51.790491 4939 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 03 00:40:51.805850 ip-10-0-140-107 kubenswrapper[4939]: W1103 00:40:51.790494 4939 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 03 00:40:51.805850 ip-10-0-140-107 kubenswrapper[4939]: W1103 00:40:51.790497 4939 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 03 00:40:51.805850 ip-10-0-140-107 kubenswrapper[4939]: W1103 00:40:51.790499 4939 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 03 00:40:51.805850 ip-10-0-140-107 kubenswrapper[4939]: W1103 00:40:51.790502 4939 feature_gate.go:386] unrecognized feature gate: Example Nov 03 00:40:51.805850 ip-10-0-140-107 kubenswrapper[4939]: W1103 00:40:51.790504 4939 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 03 00:40:51.805850 ip-10-0-140-107 kubenswrapper[4939]: W1103 00:40:51.790507 4939 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 03 00:40:51.805850 ip-10-0-140-107 kubenswrapper[4939]: W1103 00:40:51.790510 4939 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 03 00:40:51.805850 ip-10-0-140-107 kubenswrapper[4939]: W1103 00:40:51.790512 4939 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 03 00:40:51.805850 ip-10-0-140-107 kubenswrapper[4939]: W1103 00:40:51.790515 4939 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 03 00:40:51.805850 ip-10-0-140-107 kubenswrapper[4939]: W1103 00:40:51.790518 4939 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 03 00:40:51.806728 ip-10-0-140-107 kubenswrapper[4939]: W1103 00:40:51.790520 4939 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 03 00:40:51.806728 ip-10-0-140-107 kubenswrapper[4939]: W1103 00:40:51.790523 4939 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 03 00:40:51.806728 ip-10-0-140-107 kubenswrapper[4939]: W1103 00:40:51.790526 4939 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 03 00:40:51.806728 ip-10-0-140-107 kubenswrapper[4939]: W1103 00:40:51.790528 4939 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 03 00:40:51.806728 ip-10-0-140-107 kubenswrapper[4939]: W1103 00:40:51.790534 4939 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 03 00:40:51.806728 ip-10-0-140-107 kubenswrapper[4939]: W1103 00:40:51.790537 4939 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 03 00:40:51.806728 ip-10-0-140-107 kubenswrapper[4939]: W1103 00:40:51.790540 4939 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 03 00:40:51.806728 ip-10-0-140-107 kubenswrapper[4939]: W1103 00:40:51.790542 4939 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 03 00:40:51.806728 ip-10-0-140-107 kubenswrapper[4939]: W1103 00:40:51.790545 4939 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 03 00:40:51.806728 ip-10-0-140-107 kubenswrapper[4939]: W1103 00:40:51.790547 4939 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 03 00:40:51.806728 ip-10-0-140-107 kubenswrapper[4939]: W1103 00:40:51.790550 4939 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 03 00:40:51.806728 ip-10-0-140-107 kubenswrapper[4939]: W1103 00:40:51.790553 4939 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 03 00:40:51.806728 ip-10-0-140-107 kubenswrapper[4939]: W1103 00:40:51.790555 4939 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 03 00:40:51.806728 ip-10-0-140-107 kubenswrapper[4939]: W1103 00:40:51.790558 4939 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 03 00:40:51.806728 ip-10-0-140-107 kubenswrapper[4939]: W1103 00:40:51.790560 4939 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 03 00:40:51.806728 ip-10-0-140-107 kubenswrapper[4939]: W1103 00:40:51.790563 4939 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 03 00:40:51.806728 ip-10-0-140-107 kubenswrapper[4939]: W1103 00:40:51.790566 4939 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 03 00:40:51.806728 ip-10-0-140-107 kubenswrapper[4939]: W1103 00:40:51.790568 4939 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 03 00:40:51.806728 ip-10-0-140-107 kubenswrapper[4939]: W1103 00:40:51.790571 4939 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 03 00:40:51.806728 ip-10-0-140-107 kubenswrapper[4939]: W1103 00:40:51.790574 4939 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 03 00:40:51.807644 ip-10-0-140-107 kubenswrapper[4939]: W1103 00:40:51.790576 4939 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 03 00:40:51.807644 ip-10-0-140-107 kubenswrapper[4939]: W1103 00:40:51.790579 4939 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 03 00:40:51.807644 ip-10-0-140-107 kubenswrapper[4939]: W1103 00:40:51.790582 4939 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 03 00:40:51.807644 ip-10-0-140-107 kubenswrapper[4939]: W1103 00:40:51.790584 4939 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 03 00:40:51.807644 ip-10-0-140-107 kubenswrapper[4939]: W1103 00:40:51.790587 4939 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 03 00:40:51.807644 ip-10-0-140-107 kubenswrapper[4939]: W1103 00:40:51.790590 4939 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 03 00:40:51.807644 ip-10-0-140-107 kubenswrapper[4939]: W1103 00:40:51.790592 4939 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 03 00:40:51.807644 ip-10-0-140-107 kubenswrapper[4939]: W1103 00:40:51.790595 4939 feature_gate.go:386] unrecognized feature gate: Example2 Nov 03 00:40:51.807644 ip-10-0-140-107 kubenswrapper[4939]: W1103 00:40:51.790597 4939 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 03 00:40:51.807644 ip-10-0-140-107 kubenswrapper[4939]: W1103 00:40:51.790600 4939 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 03 00:40:51.807644 ip-10-0-140-107 kubenswrapper[4939]: W1103 00:40:51.790603 4939 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 03 00:40:51.807644 ip-10-0-140-107 kubenswrapper[4939]: W1103 00:40:51.790605 4939 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 03 00:40:51.807644 ip-10-0-140-107 kubenswrapper[4939]: W1103 00:40:51.790608 4939 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 03 00:40:51.807644 ip-10-0-140-107 kubenswrapper[4939]: W1103 00:40:51.790611 4939 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 03 00:40:51.807644 ip-10-0-140-107 kubenswrapper[4939]: W1103 00:40:51.790613 4939 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 03 00:40:51.807644 ip-10-0-140-107 kubenswrapper[4939]: W1103 00:40:51.790616 4939 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 03 00:40:51.807644 ip-10-0-140-107 kubenswrapper[4939]: W1103 00:40:51.790618 4939 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 03 00:40:51.807644 ip-10-0-140-107 kubenswrapper[4939]: W1103 00:40:51.790626 4939 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 03 00:40:51.807644 ip-10-0-140-107 kubenswrapper[4939]: W1103 00:40:51.790629 4939 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 03 00:40:51.808535 ip-10-0-140-107 kubenswrapper[4939]: W1103 00:40:51.790632 4939 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 03 00:40:51.808535 ip-10-0-140-107 kubenswrapper[4939]: W1103 00:40:51.790635 4939 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 03 00:40:51.808535 ip-10-0-140-107 kubenswrapper[4939]: W1103 00:40:51.790638 4939 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 03 00:40:51.808535 ip-10-0-140-107 kubenswrapper[4939]: W1103 00:40:51.790642 4939 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 03 00:40:51.808535 ip-10-0-140-107 kubenswrapper[4939]: W1103 00:40:51.790646 4939 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 03 00:40:51.808535 ip-10-0-140-107 kubenswrapper[4939]: W1103 00:40:51.790649 4939 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 03 00:40:51.808535 ip-10-0-140-107 kubenswrapper[4939]: W1103 00:40:51.790652 4939 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 03 00:40:51.808535 ip-10-0-140-107 kubenswrapper[4939]: W1103 00:40:51.790655 4939 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 03 00:40:51.808535 ip-10-0-140-107 kubenswrapper[4939]: W1103 00:40:51.790658 4939 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 03 00:40:51.808535 ip-10-0-140-107 kubenswrapper[4939]: W1103 00:40:51.790661 4939 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 03 00:40:51.808535 ip-10-0-140-107 kubenswrapper[4939]: W1103 00:40:51.790663 4939 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 03 00:40:51.808535 ip-10-0-140-107 kubenswrapper[4939]: W1103 00:40:51.790666 4939 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 03 00:40:51.808535 ip-10-0-140-107 kubenswrapper[4939]: W1103 00:40:51.790668 4939 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 03 00:40:51.808535 ip-10-0-140-107 kubenswrapper[4939]: W1103 00:40:51.790671 4939 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 03 00:40:51.808535 ip-10-0-140-107 kubenswrapper[4939]: W1103 00:40:51.790674 4939 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 03 00:40:51.808535 ip-10-0-140-107 kubenswrapper[4939]: W1103 00:40:51.790677 4939 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 03 00:40:51.808535 ip-10-0-140-107 kubenswrapper[4939]: W1103 00:40:51.790679 4939 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 03 00:40:51.808535 ip-10-0-140-107 kubenswrapper[4939]: W1103 00:40:51.790682 4939 feature_gate.go:386] unrecognized feature gate: EVPN Nov 03 00:40:51.808535 ip-10-0-140-107 kubenswrapper[4939]: W1103 00:40:51.790684 4939 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 03 00:40:51.809390 ip-10-0-140-107 kubenswrapper[4939]: W1103 00:40:51.790687 4939 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 03 00:40:51.809390 ip-10-0-140-107 kubenswrapper[4939]: W1103 00:40:51.790690 4939 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 03 00:40:51.809390 ip-10-0-140-107 kubenswrapper[4939]: W1103 00:40:51.790692 4939 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 03 00:40:51.809390 ip-10-0-140-107 kubenswrapper[4939]: W1103 00:40:51.790695 4939 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 03 00:40:51.809390 ip-10-0-140-107 kubenswrapper[4939]: W1103 00:40:51.790697 4939 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 03 00:40:51.809390 ip-10-0-140-107 kubenswrapper[4939]: W1103 00:40:51.790701 4939 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 03 00:40:51.809390 ip-10-0-140-107 kubenswrapper[4939]: W1103 00:40:51.790704 4939 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 03 00:40:51.809390 ip-10-0-140-107 kubenswrapper[4939]: W1103 00:40:51.790707 4939 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 03 00:40:51.809390 ip-10-0-140-107 kubenswrapper[4939]: W1103 00:40:51.790710 4939 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 03 00:40:51.809390 ip-10-0-140-107 kubenswrapper[4939]: W1103 00:40:51.790713 4939 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 03 00:40:51.809390 ip-10-0-140-107 kubenswrapper[4939]: W1103 00:40:51.790715 4939 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 03 00:40:51.809390 ip-10-0-140-107 kubenswrapper[4939]: W1103 00:40:51.790718 4939 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 03 00:40:51.809390 ip-10-0-140-107 kubenswrapper[4939]: W1103 00:40:51.790724 4939 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 03 00:40:51.809390 ip-10-0-140-107 kubenswrapper[4939]: W1103 00:40:51.790727 4939 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 03 00:40:51.809390 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:40:51.790751 4939 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Nov 03 00:40:51.809857 ip-10-0-140-107 kubenswrapper[4939]: W1103 00:40:51.790781 4939 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 03 00:40:51.809857 ip-10-0-140-107 kubenswrapper[4939]: W1103 00:40:51.790785 4939 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 03 00:40:51.809857 ip-10-0-140-107 kubenswrapper[4939]: W1103 00:40:51.790789 4939 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 03 00:40:51.809857 ip-10-0-140-107 kubenswrapper[4939]: W1103 00:40:51.790791 4939 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 03 00:40:51.809857 ip-10-0-140-107 kubenswrapper[4939]: W1103 00:40:51.790794 4939 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 03 00:40:51.809857 ip-10-0-140-107 kubenswrapper[4939]: W1103 00:40:51.790797 4939 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 03 00:40:51.809857 ip-10-0-140-107 kubenswrapper[4939]: W1103 00:40:51.790800 4939 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 03 00:40:51.809857 ip-10-0-140-107 kubenswrapper[4939]: W1103 00:40:51.790803 4939 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 03 00:40:51.809857 ip-10-0-140-107 kubenswrapper[4939]: W1103 00:40:51.790805 4939 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 03 00:40:51.809857 ip-10-0-140-107 kubenswrapper[4939]: W1103 00:40:51.790808 4939 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 03 00:40:51.809857 ip-10-0-140-107 kubenswrapper[4939]: W1103 00:40:51.790810 4939 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 03 00:40:51.809857 ip-10-0-140-107 kubenswrapper[4939]: W1103 00:40:51.790813 4939 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 03 00:40:51.809857 ip-10-0-140-107 kubenswrapper[4939]: W1103 00:40:51.790815 4939 feature_gate.go:386] unrecognized feature gate: Example2 Nov 03 00:40:51.809857 ip-10-0-140-107 kubenswrapper[4939]: W1103 00:40:51.790818 4939 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 03 00:40:51.809857 ip-10-0-140-107 kubenswrapper[4939]: W1103 00:40:51.790821 4939 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 03 00:40:51.809857 ip-10-0-140-107 kubenswrapper[4939]: W1103 00:40:51.790824 4939 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 03 00:40:51.809857 ip-10-0-140-107 kubenswrapper[4939]: W1103 00:40:51.790826 4939 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 03 00:40:51.809857 ip-10-0-140-107 kubenswrapper[4939]: W1103 00:40:51.790829 4939 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 03 00:40:51.809857 ip-10-0-140-107 kubenswrapper[4939]: W1103 00:40:51.790832 4939 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 03 00:40:51.809857 ip-10-0-140-107 kubenswrapper[4939]: W1103 00:40:51.790834 4939 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 03 00:40:51.810557 ip-10-0-140-107 kubenswrapper[4939]: W1103 00:40:51.790837 4939 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 03 00:40:51.810557 ip-10-0-140-107 kubenswrapper[4939]: W1103 00:40:51.790839 4939 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 03 00:40:51.810557 ip-10-0-140-107 kubenswrapper[4939]: W1103 00:40:51.790842 4939 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 03 00:40:51.810557 ip-10-0-140-107 kubenswrapper[4939]: W1103 00:40:51.790845 4939 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 03 00:40:51.810557 ip-10-0-140-107 kubenswrapper[4939]: W1103 00:40:51.790847 4939 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 03 00:40:51.810557 ip-10-0-140-107 kubenswrapper[4939]: W1103 00:40:51.790850 4939 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 03 00:40:51.810557 ip-10-0-140-107 kubenswrapper[4939]: W1103 00:40:51.790852 4939 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 03 00:40:51.810557 ip-10-0-140-107 kubenswrapper[4939]: W1103 00:40:51.790857 4939 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 03 00:40:51.810557 ip-10-0-140-107 kubenswrapper[4939]: W1103 00:40:51.790860 4939 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 03 00:40:51.810557 ip-10-0-140-107 kubenswrapper[4939]: W1103 00:40:51.790863 4939 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 03 00:40:51.810557 ip-10-0-140-107 kubenswrapper[4939]: W1103 00:40:51.790867 4939 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 03 00:40:51.810557 ip-10-0-140-107 kubenswrapper[4939]: W1103 00:40:51.790870 4939 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 03 00:40:51.810557 ip-10-0-140-107 kubenswrapper[4939]: W1103 00:40:51.790873 4939 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 03 00:40:51.810557 ip-10-0-140-107 kubenswrapper[4939]: W1103 00:40:51.790875 4939 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 03 00:40:51.810557 ip-10-0-140-107 kubenswrapper[4939]: W1103 00:40:51.790878 4939 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 03 00:40:51.810557 ip-10-0-140-107 kubenswrapper[4939]: W1103 00:40:51.790881 4939 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 03 00:40:51.810557 ip-10-0-140-107 kubenswrapper[4939]: W1103 00:40:51.790883 4939 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 03 00:40:51.810557 ip-10-0-140-107 kubenswrapper[4939]: W1103 00:40:51.790886 4939 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 03 00:40:51.810557 ip-10-0-140-107 kubenswrapper[4939]: W1103 00:40:51.790888 4939 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 03 00:40:51.811087 ip-10-0-140-107 kubenswrapper[4939]: W1103 00:40:51.790891 4939 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 03 00:40:51.811087 ip-10-0-140-107 kubenswrapper[4939]: W1103 00:40:51.790894 4939 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 03 00:40:51.811087 ip-10-0-140-107 kubenswrapper[4939]: W1103 00:40:51.790896 4939 feature_gate.go:386] unrecognized feature gate: EVPN Nov 03 00:40:51.811087 ip-10-0-140-107 kubenswrapper[4939]: W1103 00:40:51.790899 4939 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 03 00:40:51.811087 ip-10-0-140-107 kubenswrapper[4939]: W1103 00:40:51.790901 4939 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 03 00:40:51.811087 ip-10-0-140-107 kubenswrapper[4939]: W1103 00:40:51.790904 4939 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 03 00:40:51.811087 ip-10-0-140-107 kubenswrapper[4939]: W1103 00:40:51.790917 4939 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 03 00:40:51.811087 ip-10-0-140-107 kubenswrapper[4939]: W1103 00:40:51.790920 4939 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 03 00:40:51.811087 ip-10-0-140-107 kubenswrapper[4939]: W1103 00:40:51.790923 4939 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 03 00:40:51.811087 ip-10-0-140-107 kubenswrapper[4939]: W1103 00:40:51.790927 4939 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 03 00:40:51.811087 ip-10-0-140-107 kubenswrapper[4939]: W1103 00:40:51.790930 4939 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 03 00:40:51.811087 ip-10-0-140-107 kubenswrapper[4939]: W1103 00:40:51.790934 4939 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 03 00:40:51.811087 ip-10-0-140-107 kubenswrapper[4939]: W1103 00:40:51.790937 4939 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 03 00:40:51.811087 ip-10-0-140-107 kubenswrapper[4939]: W1103 00:40:51.790940 4939 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 03 00:40:51.811087 ip-10-0-140-107 kubenswrapper[4939]: W1103 00:40:51.790942 4939 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 03 00:40:51.811087 ip-10-0-140-107 kubenswrapper[4939]: W1103 00:40:51.790945 4939 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 03 00:40:51.811087 ip-10-0-140-107 kubenswrapper[4939]: W1103 00:40:51.790947 4939 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 03 00:40:51.811087 ip-10-0-140-107 kubenswrapper[4939]: W1103 00:40:51.790951 4939 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 03 00:40:51.811087 ip-10-0-140-107 kubenswrapper[4939]: W1103 00:40:51.790953 4939 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 03 00:40:51.811530 ip-10-0-140-107 kubenswrapper[4939]: W1103 00:40:51.790956 4939 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 03 00:40:51.811530 ip-10-0-140-107 kubenswrapper[4939]: W1103 00:40:51.790962 4939 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 03 00:40:51.811530 ip-10-0-140-107 kubenswrapper[4939]: W1103 00:40:51.790964 4939 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 03 00:40:51.811530 ip-10-0-140-107 kubenswrapper[4939]: W1103 00:40:51.790967 4939 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 03 00:40:51.811530 ip-10-0-140-107 kubenswrapper[4939]: W1103 00:40:51.790970 4939 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 03 00:40:51.811530 ip-10-0-140-107 kubenswrapper[4939]: W1103 00:40:51.790973 4939 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 03 00:40:51.811530 ip-10-0-140-107 kubenswrapper[4939]: W1103 00:40:51.790976 4939 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 03 00:40:51.811530 ip-10-0-140-107 kubenswrapper[4939]: W1103 00:40:51.790979 4939 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 03 00:40:51.811530 ip-10-0-140-107 kubenswrapper[4939]: W1103 00:40:51.790981 4939 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 03 00:40:51.811530 ip-10-0-140-107 kubenswrapper[4939]: W1103 00:40:51.790984 4939 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 03 00:40:51.811530 ip-10-0-140-107 kubenswrapper[4939]: W1103 00:40:51.790987 4939 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 03 00:40:51.811530 ip-10-0-140-107 kubenswrapper[4939]: W1103 00:40:51.790990 4939 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 03 00:40:51.811530 ip-10-0-140-107 kubenswrapper[4939]: W1103 00:40:51.790992 4939 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 03 00:40:51.811530 ip-10-0-140-107 kubenswrapper[4939]: W1103 00:40:51.790995 4939 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 03 00:40:51.811530 ip-10-0-140-107 kubenswrapper[4939]: W1103 00:40:51.790997 4939 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 03 00:40:51.811530 ip-10-0-140-107 kubenswrapper[4939]: W1103 00:40:51.791000 4939 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 03 00:40:51.811530 ip-10-0-140-107 kubenswrapper[4939]: W1103 00:40:51.791003 4939 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 03 00:40:51.811530 ip-10-0-140-107 kubenswrapper[4939]: W1103 00:40:51.791005 4939 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 03 00:40:51.811530 ip-10-0-140-107 kubenswrapper[4939]: W1103 00:40:51.791008 4939 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 03 00:40:51.811530 ip-10-0-140-107 kubenswrapper[4939]: W1103 00:40:51.791010 4939 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 03 00:40:51.812030 ip-10-0-140-107 kubenswrapper[4939]: W1103 00:40:51.791013 4939 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 03 00:40:51.812030 ip-10-0-140-107 kubenswrapper[4939]: W1103 00:40:51.791016 4939 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 03 00:40:51.812030 ip-10-0-140-107 kubenswrapper[4939]: W1103 00:40:51.791018 4939 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 03 00:40:51.812030 ip-10-0-140-107 kubenswrapper[4939]: W1103 00:40:51.791021 4939 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 03 00:40:51.812030 ip-10-0-140-107 kubenswrapper[4939]: W1103 00:40:51.791024 4939 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 03 00:40:51.812030 ip-10-0-140-107 kubenswrapper[4939]: W1103 00:40:51.791026 4939 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 03 00:40:51.812030 ip-10-0-140-107 kubenswrapper[4939]: W1103 00:40:51.791029 4939 feature_gate.go:386] unrecognized feature gate: Example Nov 03 00:40:51.812030 ip-10-0-140-107 kubenswrapper[4939]: W1103 00:40:51.791032 4939 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 03 00:40:51.812030 ip-10-0-140-107 kubenswrapper[4939]: W1103 00:40:51.791034 4939 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 03 00:40:51.812030 ip-10-0-140-107 kubenswrapper[4939]: W1103 00:40:51.791037 4939 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 03 00:40:51.812030 ip-10-0-140-107 kubenswrapper[4939]: W1103 00:40:51.791039 4939 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 03 00:40:51.812030 ip-10-0-140-107 kubenswrapper[4939]: W1103 00:40:51.791042 4939 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 03 00:40:51.812030 ip-10-0-140-107 kubenswrapper[4939]: W1103 00:40:51.791044 4939 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 03 00:40:51.812030 ip-10-0-140-107 kubenswrapper[4939]: W1103 00:40:51.791048 4939 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 03 00:40:51.812030 ip-10-0-140-107 kubenswrapper[4939]: W1103 00:40:51.791051 4939 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 03 00:40:51.812030 ip-10-0-140-107 kubenswrapper[4939]: W1103 00:40:51.791055 4939 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 03 00:40:51.812030 ip-10-0-140-107 kubenswrapper[4939]: W1103 00:40:51.791058 4939 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 03 00:40:51.812030 ip-10-0-140-107 kubenswrapper[4939]: W1103 00:40:51.791060 4939 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 03 00:40:51.812030 ip-10-0-140-107 kubenswrapper[4939]: W1103 00:40:51.791064 4939 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 03 00:40:51.812030 ip-10-0-140-107 kubenswrapper[4939]: W1103 00:40:51.791066 4939 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 03 00:40:51.812557 ip-10-0-140-107 kubenswrapper[4939]: W1103 00:40:51.791069 4939 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 03 00:40:51.812557 ip-10-0-140-107 kubenswrapper[4939]: W1103 00:40:51.791072 4939 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 03 00:40:51.812557 ip-10-0-140-107 kubenswrapper[4939]: W1103 00:40:51.791075 4939 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 03 00:40:51.812557 ip-10-0-140-107 kubenswrapper[4939]: W1103 00:40:51.791077 4939 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 03 00:40:51.812557 ip-10-0-140-107 kubenswrapper[4939]: W1103 00:40:51.791080 4939 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 03 00:40:51.812557 ip-10-0-140-107 kubenswrapper[4939]: W1103 00:40:51.791083 4939 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 03 00:40:51.812557 ip-10-0-140-107 kubenswrapper[4939]: W1103 00:40:51.791085 4939 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 03 00:40:51.812557 ip-10-0-140-107 kubenswrapper[4939]: W1103 00:40:51.791088 4939 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 03 00:40:51.812557 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:40:51.791144 4939 server.go:957] "Client rotation is on, will bootstrap in background" Nov 03 00:40:51.812557 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:40:51.792110 4939 bootstrap.go:85] "Current kubeconfig file contents are still valid, no bootstrap necessary" Nov 03 00:40:51.812557 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:40:51.792163 4939 certificate_store.go:147] "Loading cert/key pair from a file" filePath="/var/lib/kubelet/pki/kubelet-client-current.pem" Nov 03 00:40:51.812557 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:40:51.792335 4939 server.go:1014] "Starting client certificate rotation" Nov 03 00:40:51.812557 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:40:51.792489 4939 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet" Nov 03 00:40:51.812557 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:40:51.792535 4939 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-11-02 00:35:10 +0000 UTC" deadline="2028-05-12 02:56:55.706312801 +0000 UTC" Nov 03 00:40:51.812557 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:40:51.792585 4939 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="13346h16m3.913729533s" Nov 03 00:40:51.812557 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:40:51.792830 4939 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Nov 03 00:40:51.812964 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:40:51.793917 4939 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Nov 03 00:40:51.812964 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:40:51.795230 4939 log.go:25] "Validated CRI v1 runtime API" Nov 03 00:40:51.812964 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:40:51.797024 4939 log.go:25] "Validated CRI v1 image API" Nov 03 00:40:51.812964 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:40:51.797733 4939 server.go:1424] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd" Nov 03 00:40:51.812964 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:40:51.800796 4939 fs.go:135] Filesystem UUIDs: map[2ebfe389-d801-4aca-8bd7-d371bb69eba5:/dev/nvme0n1p3 7B77-95E7:/dev/nvme0n1p2 b51bc7d0-7d6e-464a-bc11-e10a6cd8889d:/dev/nvme0n1p4] Nov 03 00:40:51.813101 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:40:51.800821 4939 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:24 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:26 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/0444234d1c00351a0c0dec655e17f704fbb70c110f8904dc87014c9c860c40db/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/0444234d1c00351a0c0dec655e17f704fbb70c110f8904dc87014c9c860c40db/userdata/shm major:0 minor:236 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/05e3d4718d6fafed62c11717499adc9949e4df03b37a914bd49d38181d2f088f/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/05e3d4718d6fafed62c11717499adc9949e4df03b37a914bd49d38181d2f088f/userdata/shm major:0 minor:242 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/18b88862137befca64ff94878901feb875f51616dd9b95cd34057759a64e55c1/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/18b88862137befca64ff94878901feb875f51616dd9b95cd34057759a64e55c1/userdata/shm major:0 minor:80 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/1c5c6a32bfd30ad97469ed1fb0359d4cf5bc1d6c7254eef2ad99974885969c36/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/1c5c6a32bfd30ad97469ed1fb0359d4cf5bc1d6c7254eef2ad99974885969c36/userdata/shm major:0 minor:81 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/20d31a990fda9ed8069aa304f22413ca1c459b7d9b9b0e268ac30fa12af2d633/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/20d31a990fda9ed8069aa304f22413ca1c459b7d9b9b0e268ac30fa12af2d633/userdata/shm major:0 minor:47 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/2b3856ef34d4cbba8d89162e727bb1a5ed2a806920a8e038711a17170afd7bc6/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/2b3856ef34d4cbba8d89162e727bb1a5ed2a806920a8e038711a17170afd7bc6/userdata/shm major:0 minor:329 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/3f1b02020b063d5bec095ef4de846f7d7590f5dec7e9630c6f92094ff6c01081/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/3f1b02020b063d5bec095ef4de846f7d7590f5dec7e9630c6f92094ff6c01081/userdata/shm major:0 minor:68 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/4112437df1446eb838fd35176a32848f7974fa73d27056b56cfcd766cd9894da/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/4112437df1446eb838fd35176a32848f7974fa73d27056b56cfcd766cd9894da/userdata/shm major:0 minor:72 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/487ea3f04a7119c9c89cde91e9dda8b21902247978fb80fbfd356adf8237b019/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/487ea3f04a7119c9c89cde91e9dda8b21902247978fb80fbfd356adf8237b019/userdata/shm major:0 minor:234 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/662fed273c35eab0aa9fe8026120a6defc961c8ef53105c04feef133bbb54108/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/662fed273c35eab0aa9fe8026120a6defc961c8ef53105c04feef133bbb54108/userdata/shm major:0 minor:238 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/6708f186ee33a9346160371d6e1ddfb5e3a0cbdec1bf050f2ee75b910e994ce9/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/6708f186ee33a9346160371d6e1ddfb5e3a0cbdec1bf050f2ee75b910e994ce9/userdata/shm major:0 minor:228 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/70a1f3b0a3f91bb331a7be1e85fed083f6f1bd0252c8c3c3c47a6dc14cf781c5/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/70a1f3b0a3f91bb331a7be1e85fed083f6f1bd0252c8c3c3c47a6dc14cf781c5/userdata/shm major:0 minor:88 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/8867ce85967697dcfba6c103bcc04489490e98decd0f1994a213282757db1130/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/8867ce85967697dcfba6c103bcc04489490e98decd0f1994a213282757db1130/userdata/shm major:0 minor:76 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/a012e88fc324e4ecd1bc5f9557af5fe91bc73d451cee98e718e6ecf7cf8da48f/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/a012e88fc324e4ecd1bc5f9557af5fe91bc73d451cee98e718e6ecf7cf8da48f/userdata/shm major:0 minor:51 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/a5114a1c8f2cd400e64362ba4b984bd6b439505b63ac0a12be3add35ec5a48e7/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/a5114a1c8f2cd400e64362ba4b984bd6b439505b63ac0a12be3add35ec5a48e7/userdata/shm major:0 minor:297 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/a7619cc5d7ec9ca7f22e4b1d299dfaec1ec361e238eaabfd63843c47d0bc74ea/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/a7619cc5d7ec9ca7f22e4b1d299dfaec1ec361e238eaabfd63843c47d0bc74ea/userdata/shm major:0 minor:296 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/ad4f86165f311454a0c13817aa2b917537300761a1b4dd713a21743be3bbc553/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/ad4f86165f311454a0c13817aa2b917537300761a1b4dd713a21743be3bbc553/userdata/shm major:0 minor:230 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/b11da0d7baa29ab225e3891350eb46858d211cf3575d1b66e0c76ad4b382e2fa/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/b11da0d7baa29ab225e3891350eb46858d211cf3575d1b66e0c76ad4b382e2fa/userdata/shm major:0 minor:240 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/b942f9e92ab3f2cf912cab48cffd6b46e23dd27024f8c374d7e54e710cdd082f/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/b942f9e92ab3f2cf912cab48cffd6b46e23dd27024f8c374d7e54e710cdd082f/userdata/shm major:0 minor:243 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/c5d2d73eebd8feaeb150e256ec01b80911dcc4ec8baffc65888eb14551386136/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/c5d2d73eebd8feaeb150e256ec01b80911dcc4ec8baffc65888eb14551386136/userdata/shm major:0 minor:96 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/d3e8e3bcd4255307bff4cba939cd812eaaae2e8342e455d9fa4ede17d02790c5/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/d3e8e3bcd4255307bff4cba939cd812eaaae2e8342e455d9fa4ede17d02790c5/userdata/shm major:0 minor:231 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/e0c24e4a15e4f03b8a6d417c965c9bee757e311cc8ce503619d167866b0b6daa/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/e0c24e4a15e4f03b8a6d417c965c9bee757e311cc8ce503619d167866b0b6daa/userdata/shm major:0 minor:92 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/e86455a72def567d132d9ca3c0f5e9449cfde0bf23efbff663c0c38506834a86/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/e86455a72def567d132d9ca3c0f5e9449cfde0bf23efbff663c0c38506834a86/userdata/shm major:0 minor:246 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:32 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/00dc2920-34f2-4e0a-892e-7759a16162d5/volumes/kubernetes.io~projected/kube-api-access-j62gg:{mountpoint:/var/lib/kubelet/pods/00dc2920-34f2-4e0a-892e-7759a16162d5/volumes/kubernetes.io~projected/kube-api-access-j62gg major:0 minor:56 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/06973556-f51c-4d1c-ae55-fb77699bc5b3/volumes/kubernetes.io~projected/kube-api-access-4znwf:{mountpoint:/var/lib/kubelet/pods/06973556-f51c-4d1c-ae55-fb77699bc5b3/volumes/kubernetes.io~projected/kube-api-access-4znwf major:0 minor:220 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/136086c0-4846-437a-ad98-3cfef50610a0/volumes/kubernetes.io~projected/kube-api-access-llh2g:{mountpoint:/var/lib/kubelet/pods/136086c0-4846-437a-ad98-3cfef50610a0/volumes/kubernetes.io~projected/kube-api-access-llh2g major:0 minor:60 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/29a64a08-67ef-4c82-8a82-8cd4a4675327/volumes/kubernetes.io~projected/kube-api-access-pdmr4:{mountpoint:/var/lib/kubelet/pods/29a64a08-67ef-4c82-8a82-8cd4a4675327/volumes/kubernetes.io~projected/kube-api-access-pdmr4 major:0 minor:212 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/2b050b18-31f6-4a22-9902-0435d11018b9/volumes/kubernetes.io~projected/kube-api-access-wzln7:{mountpoint:/var/lib/kubelet/pods/2b050b18-31f6-4a22-9902-0435d11018b9/volumes/kubernetes.io~projected/kube-api-access-wzln7 major:0 minor:221 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/2b050b18-31f6-4a22-9902-0435d11018b9/volumes/kubernetes.io~secret/serving-cert:{mountpoint:/var/lib/kubelet/pods/2b050b18-31f6-4a22-9902-0435d11018b9/volumes/kubernetes.io~secret/serving-cert major:0 minor:205 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/36b9d258-cb2c-4d2b-880f-ae93b065034c/volumes/kubernetes.io~projected/kube-api-access-79khw:{mountpoint:/var/lib/kubelet/pods/36b9d258-cb2c-4d2b-880f-ae93b065034c/volumes/kubernetes.io~projected/kube-api-access-79khw major:0 minor:225 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/36b9d258-cb2c-4d2b-880f-ae93b065034c/volumes/kubernetes.io~secret/serving-cert:{mountpoint:/var/lib/kubelet/pods/36b9d258-cb2c-4d2b-880f-ae93b065034c/volumes/kubernetes.io~secret/serving-cert major:0 minor:197 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/37bf96f5-045f-49e8-83e8-410df66bfb1f/volumes/kubernetes.io~projected/kube-api-access-lp26w:{mountpoint:/var/lib/kubelet/pods/37bf96f5-045f-49e8-83e8-410df66bfb1f/volumes/kubernetes.io~projected/kube-api-access-lp26w major:0 minor:67 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/4237805a-d475-4bd7-865c-c6ef371c0e11/volumes/kubernetes.io~projected/kube-api-access-nswxk:{mountpoint:/var/lib/kubelet/pods/4237805a-d475-4bd7-865c-c6ef371c0e11/volumes/kubernetes.io~projected/kube-api-access-nswxk major:0 minor:214 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/460779b2-3903-4b23-9b4d-e25142782a21/volumes/kubernetes.io~projected/kube-api-access-q4b9t:{mountpoint:/var/lib/kubelet/pods/460779b2-3903-4b23-9b4d-e25142782a21/volumes/kubernetes.io~projected/kube-api-access-q4b9t major:0 minor:66 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/525dc47d-afde-4019-ad01-80614291f282/volumes/kubernetes.io~projected/kube-api-access-sbn8x:{mountpoint:/var/lib/kubelet/pods/525dc47d-afde-4019-ad01-80614291f282/volumes/kubernetes.io~projected/kube-api-access-sbn8x major:0 minor:222 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/52a19842-3750-4d2b-bcb2-aebddee81097/volumes/kubernetes.io~projected/bound-sa-token:{mountpoint:/var/lib/kubelet/pods/52a19842-3750-4d2b-bcb2-aebddee81097/volumes/kubernetes.io~projected/bound-sa-token major:0 minor:226 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/52a19842-3750-4d2b-bcb2-aebddee81097/volumes/kubernetes.io~projected/kube-api-access-tc9rq:{mountpoint:/var/lib/kubelet/pods/52a19842-3750-4d2b-bcb2-aebddee81097/volumes/kubernetes.io~projected/kube-api-access-tc9rq major:0 minor:208 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/52a19842-3750-4d2b-bcb2-aebddee81097/volumes/kubernetes.io~secret/image-registry-private-configuration:{mountpoint:/var/lib/kubelet/pods/52a19842-3750-4d2b-bcb2-aebddee81097/volumes/kubernetes.io~secret/image-registry-private-configuration major:0 minor:203 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/52a19842-3750-4d2b-bcb2-aebddee81097/volumes/kubernetes.io~secret/installation-pull-secrets:{mountpoint:/var/lib/kubelet/pods/52a19842-3750-4d2b-bcb2-aebddee81097/volumes/kubernetes.io~secret/installation-pull-secrets major:0 minor:45 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/56261beb-517f-4c88-b59a-42ab41e278fd/volumes/kubernetes.io~projected/kube-api-access-h58rt:{mountpoint:/var/lib/kubelet/pods/56261beb-517f-4c88-b59a-42ab41e278fd/volumes/kubernetes.io~projected/kube-api-access-h58rt major:0 minor:219 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/65d134ac-8a25-4c67-b6c7-cd752a098daf/volumes/kubernetes.io~projected/kube-api-access-8j6kr:{mountpoint:/var/lib/kubelet/pods/65d134ac-8a25-4c67-b6c7-cd752a098daf/volumes/kubernetes.io~projected/kube-api-access-8j6kr major:0 minor:209 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/65d134ac-8a25-4c67-b6c7-cd752a098daf/volumes/kubernetes.io~secret/serving-cert:{mountpoint:/var/lib/kubelet/pods/65d134ac-8a25-4c67-b6c7-cd752a098daf/volumes/kubernetes.io~secret/serving-cert major:0 minor:210 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/6650035b-2220-491c-9683-8a03ccb7af6a/volumes/kubernetes.io~projected/kube-api-access-rkf7s:{mountpoint:/var/lib/kubelet/pods/6650035b-2220-491c-9683-8a03ccb7af6a/volumes/kubernetes.io~projected/kube-api-access-rkf7s major:0 minor:58 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/68658033-8a49-4430-869f-a7eb79603484/volumes/kubernetes.io~projected/kube-api-access-7wzbd:{mountpoint:/var/lib/kubelet/pods/68658033-8a49-4430-869f-a7eb79603484/volumes/kubernetes.io~projected/kube-api-access-7wzbd major:0 minor:318 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/69b087f8-4a23-4ec8-808e-b61d2e16f67b/volumes/kubernetes.io~projected/kube-api-access-pj6ft:{mountpoint:/var/lib/kubelet/pods/69b087f8-4a23-4ec8-808e-b61d2e16f67b/volumes/kubernetes.io~projected/kube-api-access-pj6ft major:0 minor:211 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/6eafe6f6-cf90-446f-a8d5-620cd2e02775/volumes/kubernetes.io~projected/kube-api-access-6kcsm:{mountpoint:/var/lib/kubelet/pods/6eafe6f6-cf90-446f-a8d5-620cd2e02775/volumes/kubernetes.io~projected/kube-api-access-6kcsm major:0 minor:215 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/75481915-3cbe-475b-abe7-cb4e81c93427/volumes/kubernetes.io~projected/kube-api-access-qmkgl:{mountpoint:/var/lib/kubelet/pods/75481915-3cbe-475b-abe7-cb4e81c93427/volumes/kubernetes.io~projected/kube-api-access-qmkgl major:0 minor:218 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/75481915-3cbe-475b-abe7-cb4e81c93427/volumes/kubernetes.io~secret/cert:{mountpoint:/var/lib/kubelet/pods/75481915-3cbe-475b-abe7-cb4e81c93427/volumes/kubernetes.io~secret/cert major:0 minor:207 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/8cd3ca22-e4ce-4896-ae31-97605af4e4c6/volumes/kubernetes.io~projected/kube-api-access-d8l25:{mountpoint:/var/lib/kubelet/pods/8cd3ca22-e4ce-4896-ae31-97605af4e4c6/volumes/kubernetes.io~projected/kube-api-access-d8l25 major:0 minor:293 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/945da51a-32cb-4769-900d-1aeb266c7565/volumes/kubernetes.io~projected/kube-api-access-4q9wl:{mountpoint:/var/lib/kubelet/pods/945da51a-32cb-4769-900d-1aeb266c7565/volumes/kubernetes.io~projected/kube-api-access-4q9wl major:0 minor:334 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/98ca3972-fd0c-47f5-9d26-569a58e18bd1/volume-subpaths/run-systemd/ovnkube-controller/6:{mountpoint:/var/lib/kubelet/pods/98ca3972-fd0c-47f5-9d26-569a58e18bd1/volume-subpaths/run-systemd/ovnkube-controller/6 major:0 minor:26 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/98ca3972-fd0c-47f5-9d26-569a58e18bd1/volumes/kubernetes.io~projected/kube-api-access-4bhqh:{mountpoint:/var/lib/kubelet/pods/98ca3972-fd0c-47f5-9d26-569a58e18bd1/volumes/kubernetes.io~projected/kube-api-access-4bhqh major:0 minor:57 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/98ca3972-fd0c-47f5-9d26-569a58e18bd1/volumes/kubernetes.io~secret/ovn-node-metrics-cert:{mountpoint:/var/lib/kubelet/pods/98ca3972-fd0c-47f5-9d26-569a58e18bd1/volumes/kubernetes.io~secret/ovn-node-metrics-cert major:0 minor:55 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/ad5bf442-729d-411c-96e4-2e5105ebadcf/volumes/kubernetes.io~projected/kube-api-access-sfdp9:{mountpoint:/var/lib/kubelet/pods/ad5bf442-729d-411c-96e4-2e5105ebadcf/volumes/kubernetes.io~projected/kube-api-access-sfdp9 major:0 minor:65 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/b0bd4bd5-12ca-49df-a3d9-4243f4bd642e/volumes/kubernetes.io~projected/kube-api-access-7zxdt:{mountpoint:/var/lib/kubelet/pods/b0bd4bd5-12ca-49df-a3d9-4243f4bd642e/volumes/kubernetes.io~projected/kube-api-access-7zxdt major:0 minor:217 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/b748c4f8-d021-4d37-b390-d395482fcddd/volumes/kubernetes.io~empty-dir/etc-tuned:{mountpoint:/var/lib/kubelet/pods/b748c4f8-d021-4d37-b390-d395482fcddd/volumes/kubernetes.io~empty-dir/etc-tuned major:0 minor:63 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/b748c4f8-d021-4d37-b390-d395482fcddd/volumes/kubernetes.io~empty-dir/tmp:{mountpoint:/var/lib/kubelet/pods/b748c4f8-d021-4d37-b390-d395482fcddd/volumes/kubernetes.io~empty-dir/tmp major:0 minor:62 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/b748c4f8-d021-4d37-b390-d395482fcddd/volumes/kubernetes.io~projected/kube-api-access-hsvqw:{mountpoint:/var/lib/kubelet/pods/b748c4f8-d021-4d37-b390-d395482fcddd/volumes/kubernetes.io~projected/kube-api-access-hsvqw major:0 minor:64 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/c3ffbe53-7f23-4750-b95d-476e6cfc58a6/volumes/kubernetes.io~secret/agent-certs:{mountpoint:/var/lib/kubelet/pods/c3ffbe53-7f23-4750-b95d-476e6cfc58a6/volumes/kubernetes.io~secret/agent-certs major:0 minor:61 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/c736fd3f-973f-4999-8d4e-c405079b2888/volumes/kubernetes.io~projected/kube-api-access-2tdmf:{mountpoint:/var/lib/kubelet/pods/c736fd3f-973f-4999-8d4e-c405079b2888/volumes/kubernetes.io~projected/kube-api-access-2tdmf major:0 minor:59 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/ca0f883c-c2b7-4bf6-a90b-ffdbeac161a8/volumes/kubernetes.io~projected/kube-api-access-qcvwz:{mountpoint:/var/lib/kubelet/pods/ca0f883c-c2b7-4bf6-a90b-ffdbeac161a8/volumes/kubernetes.io~projected/kube-api-access-qcvwz major:0 minor:216 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/ca0f883c-c2b7-4bf6-a90b-ffdbeac161a8/volumes/kubernetes.io~secret/serving-cert:{mountpoint:/var/lib/kubelet/pods/ca0f883c-c2b7-4bf6-a90b-ffdbeac161a8/volumes/kubernetes.io~secret/serving-cert major:0 minor:204 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/cabd402c-79c5-4699-9c85-d415d362b4d0/volumes/kubernetes.io~secret/original-pull-secret:{mountpoint:/var/lib/kubelet/pods/cabd402c-79c5-4699-9c85-d415d362b4d0/volumes/kubernetes.io~secret/original-pull-secret major:0 minor:288 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/cf217a21-c1bd-48be-b203-f59743bceb33/volumes/kubernetes.io~projected/kube-api-access-fdrwp:{mountpoint:/var/lib/kubelet/pods/cf217a21-c1bd-48be-b203-f59743bceb33/volumes/kubernetes.io~projected/kube-api-access-fdrwp major:0 minor:227 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/cf217a21-c1bd-48be-b203-f59743bceb33/volumes/kubernetes.io~secret/default-certificate:{mountpoint:/var/lib/kubelet/pods/cf217a21-c1bd-48be-b203-f59743bceb33/volumes/kubernetes.io~secret/default-certificate major:0 minor:206 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/cf217a21-c1bd-48be-b203-f59743bceb33/volumes/kubernetes.io~secret/stats-auth:{mountpoint:/var/lib/kubelet/pods/cf217a21-c1bd-48be-b203-f59743bceb33/volumes/kubernetes.io~secret/stats-auth major:0 minor:213 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/dd0cd8a1-4d6b-424c-ac39-5f5ddb9cd87e/volumes/kubernetes.io~projected/bound-sa-token:{mountpoint:/var/lib/kubelet/pods/dd0cd8a1-4d6b-424c-ac39-5f5ddb9cd87e/volumes/kubernetes.io~projected/bound-sa-token major:0 minor:224 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/dd0cd8a1-4d6b-424c-ac39-5f5ddb9cd87e/volumes/kubernetes.io~projected/kube-api-access-jbfhc:{mountpoint:/var/lib/kubelet/pods/dd0cd8a1-4d6b-424c-ac39-5f5ddb9cd87e/volumes/kubernetes.io~projected/kube-api-access-jbfhc major:0 minor:223 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/dd0cd8a1-4d6b-424c-ac39-5f5ddb9cd87e/volumes/kubernetes.io~secret/image-registry-private-configuration:{mountpoint:/var/lib/kubelet/pods/dd0cd8a1-4d6b-424c-ac39-5f5ddb9cd87e/volumes/kubernetes.io~secret/image-registry-private-configuration major:0 minor:202 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/dd0cd8a1-4d6b-424c-ac39-5f5ddb9cd87e/volumes/kubernetes.io~secret/installation-pull-secrets:{mountpoint:/var/lib/kubelet/pods/dd0cd8a1-4d6b-424c-ac39-5f5ddb9cd87e/volumes/kubernetes.io~secret/installation-pull-secrets major:0 minor:201 fsType:tmpfs blockSize:0} composefs_0-34:{mountpoint:/ major:0 minor:34 fsType:overlay blockSize:0} overlay_0-100:{mountpoint:/var/lib/containers/storage/overlay/c816c217a5e21e84f74250fcd17b08f30eb7e86e65bd1af265ecccb4d238aa40/merged major:0 minor:100 fsType:overlay blockSize:0} overlay_0-102:{mountpoint:/var/lib/containers/storage/overlay/67261615c1ee14d595336180ec3b9dbacf6f2433a070143d5ac6da18bebc57e4/merged major:0 minor:102 fsType:overlay blockSize:0} overlay_0-104:{mountpoint:/var/lib/containers/storage/overlay/8f7f5838ae93db75cf0ccef228c9fa8b6d6b61359c85f90c739b3496bf84a30c/merged major:0 minor:104 fsType:overlay blockSize:0} overlay_0-106:{mountpoint:/var/lib/containers/storage/overlay/18f4547a4237efb9bb18c7447405a806fb8d471b1bddcda9019249eb880d6647/merged major:0 minor:106 fsType:overlay blockSize:0} overlay_0-114:{mountpoint:/var/lib/containers/storage/overlay/a4cd687cfa554048955415376ade36dd5113c6286a3881ac2ce8229c7014fefa/merged major:0 minor:114 fsType:overlay blockSize:0} overlay_0-119:{mountpoint:/var/lib/containers/storage/overlay/2fb3f3dd8e6584b906e8bbdd4d1b8481f454047b5cce0c0e691f9b58c5b55b9a/merged major:0 minor:119 fsType:overlay blockSize:0} overlay_0-124:{mountpoint:/var/lib/containers/storage/overlay/efa24fec1c44d6920bada502b25f2ef185b378d8b2744f99d255c4d693587243/merged major:0 minor:124 fsType:overlay blockSize:0} overlay_0-127:{mountpoint:/var/lib/containers/storage/overlay/2cc5c11855dcf5675ef863317fd80b3dae141b759d48ed103da92efa5d8d5709/merged major:0 minor:127 fsType:overlay blockSize:0} overlay_0-132:{mountpoint:/var/lib/containers/storage/overlay/3973e2937b08167642c333b73b790eede956d4b0114f801aca352f853bf04b3e/merged major:0 minor:132 fsType:overlay blockSize:0} overlay_0-139:{mountpoint:/var/lib/containers/storage/overlay/f987d84054cd743840dd77df863333609c9345ab6b400a0f961e66c0ce166d1d/merged major:0 minor:139 fsType:overlay blockSize:0} overlay_0-141:{mountpoint:/var/lib/containers/storage/overlay/f9808602dccbe714412e1e17cae8ede6c465dd4ebc21019c6a5272eaf4dd556f/merged major:0 minor:141 fsType:overlay blockSize:0} overlay_0-143:{mountpoint:/var/lib/containers/storage/overlay/7f330621fa8e384beb9ba87facf54f5768f697392433dc993e4fa63a4f27faf6/merged major:0 minor:143 fsType:overlay blockSize:0} overlay_0-152:{mountpoint:/var/lib/containers/storage/overlay/41bb162d1a3185ce268970c5e1aed4eb2db1b5c51714d16197b077074dbc0fb5/merged major:0 minor:152 fsType:overlay blockSize:0} overlay_0-157:{mountpoint:/var/lib/containers/storage/overlay/e6b82b5b35c1add35d87fd15ad205a12f2df26bcb3e04341bec8934c9a6b3ab1/merged major:0 minor:157 fsType:overlay blockSize:0} overlay_0-162:{mountpoint:/var/lib/containers/storage/overlay/1711b849ec6831a544b810d11615db38bb6f0810c0820468369593928d8d34b5/merged major:0 minor:162 fsType:overlay blockSize:0} overlay_0-167:{mountpoint:/var/lib/containers/storage/overlay/d745f4b4f8b8d2f79367a5d4e76b238f4e415f3a772090a60a48cacf33d106f4/merged major:0 minor:167 fsType:overlay blockSize:0} overlay_0-172:{mountpoint:/var/lib/containers/storage/overlay/41a787071d0f594a61e7cabc5046a5724c5053ceaaeed298d5255e4964c20511/merged major:0 minor:172 fsType:overlay blockSize:0} overlay_0-177:{mountpoint:/var/lib/containers/storage/overlay/201516641cc920f4cfc14e56678085db488fdb9ac27f97a5bd810809c85ec83b/merged major:0 minor:177 fsType:overlay blockSize:0} overlay_0-182:{mountpoint:/var/lib/containers/storage/overlay/788bab799fcf7c05aa1b477f3e622503acc6bdc16a5339fee99a0d07d53cee66/merged major:0 minor:182 fsType:overlay blockSize:0} overlay_0-187:{mountpoint:/var/lib/containers/storage/overlay/70531873645b34788214d154601139ab5f113bd996066ec61e9588d32117b7f3/merged major:0 minor:187 fsType:overlay blockSize:0} overlay_0-192:{mountpoint:/var/lib/containers/storage/overlay/6d76c2d821464523f65cb25e3a589d6684cfaf84bdd6f6bc73b9ea8ed82742a6/merged major:0 minor:192 fsType:overlay blockSize:0} overlay_0-248:{mountpoint:/var/lib/containers/storage/overlay/c5b56e713bc945a8167cd4306ec930ecd4cd5bb18825acd132ea033a6b5ebaa3/merged major:0 minor:248 fsType:overlay blockSize:0} overlay_0-250:{mountpoint:/var/lib/containers/storage/overlay/57c666d5a4051af4be5ee282dc333dab4b9af523cf5e8654d45baf57cfac43d8/merged major:0 minor:250 fsType:overlay blockSize:0} overlay_0-252:{mountpoint:/var/lib/containers/storage/overlay/8494522a26a4d97f876d153dbcb794a1af911745e97e9cfa4849a73c3f887126/merged major:0 minor:252 fsType:overlay blockSize:0} overlay_0-254:{mountpoint:/var/lib/containers/storage/overlay/7ed5fbade038eb7db64a3b3a0df79f50c3296e0d5b6a0eda7084b24263ad6925/merged major:0 minor:254 fsType:overlay blockSize:0} overlay_0-256:{mountpoint:/var/lib/containers/storage/overlay/2e8277821a2981461b9e55aac12bea3abebbe1eee7b9a73b090c307dbde0bd53/merged major:0 minor:256 fsType:overlay blockSize:0} overlay_0-258:{mountpoint:/var/lib/containers/storage/overlay/094cb5ca6df16abbcd09f4fcaae0836c2b840e08763be324d721864640ad5fca/merged major:0 minor:258 fsType:overlay blockSize:0} overlay_0-260:{mountpoint:/var/lib/containers/storage/overlay/2ece7bf3e002928d787d93d8785a4bfe2fd92d74d67be88b0842a4d3a1535fa2/merged major:0 minor:260 fsType:overlay blockSize:0} overlay_0-262:{mountpoint:/var/lib/containers/storage/overlay/37f7156dc9fc3c370ce04c1c19d32baf3b4dc8761ac015f2c0178d230bb74143/merged major:0 minor:262 fsType:overlay blockSize:0} overlay_0-264:{mountpoint:/var/lib/containers/storage/overlay/e9ff3eb89f7036c4e0416939597970858cd80fa803b17cb20d4c6d7837f0e236/merged major:0 minor:264 fsType:overlay blockSize:0} overlay_0-266:{mountpoint:/var/lib/containers/storage/overlay/abbc22803e553526889cb3f3db0b6845153bdb7331d38219774b54e87a321eab/merged major:0 minor:266 fsType:overlay blockSize:0} overlay_0-268:{mountpoint:/var/lib/containers/storage/overlay/75fa2e3ad4254a1c1e9f7994318b773734d540af1a109bba15af851e0c111be5/merged major:0 minor:268 fsType:overlay blockSize:0} overlay_0-281:{mountpoint:/var/lib/containers/storage/overlay/491c8bc722e473138256e231045ef216cdc601fdea7ccaffbd671a2ced3baaac/merged major:0 minor:281 fsType:overlay blockSize:0} overlay_0-284:{mountpoint:/var/lib/containers/storage/overlay/7349ec0bf1a8cd12aacddf9129245cc8a760b8e2924fcd072e598da4db5d97db/merged major:0 minor:284 fsType:overlay blockSize:0} overlay_0-286:{mountpoint:/var/lib/containers/storage/overlay/a3f59df5c1495235644b247310a0f8704086e98e81437057bcef75ff095f5c9c/merged major:0 minor:286 fsType:overlay blockSize:0} overlay_0-294:{mountpoint:/var/lib/containers/storage/overlay/6cc708481a21029832f73280969a8b927a5bfaa0b4b5bff9d0dfec6d7eb09cd3/merged major:0 minor:294 fsType:overlay blockSize:0} overlay_0-300:{mountpoint:/var/lib/containers/storage/overlay/f88e8eab54101733a50499d66cf9f4fc22ef7771ee2a25879c8b4cf5ead59936/merged major:0 minor:300 fsType:overlay blockSize:0} overlay_0-302:{mountpoint:/var/lib/containers/storage/overlay/0711c62c3cf4f8d9e4885d0a3b529747e5a2baf3eb260ff69098aa8cef23f8fc/merged major:0 minor:302 fsType:overlay blockSize:0} overlay_0-304:{mountpoint:/var/lib/containers/storage/overlay/ca004d7f3443b581f4218ae067a830d84da9623d73ce4b76ce682f6c46f20a76/merged major:0 minor:304 fsType:overlay blockSize:0} overlay_0-306:{mountpoint:/var/lib/containers/storage/overlay/d90205acd9d620630b6bf3af3e5637a1a3c7aaab59537b5b3189e56c7beba5cc/merged major:0 minor:306 fsType:overlay blockSize:0} overlay_0-316:{mountpoint:/var/lib/containers/storage/overlay/16cd4a75733e57126d77adca8cbe490e975fdbb9a5db11bfd51ef12fdb11ae13/merged major:0 minor:316 fsType:overlay blockSize:0} overlay_0-319:{mountpoint:/var/lib/containers/storage/overlay/649bc780d1c30391ed517bb47cc2f97ea37ab5dbfcff1d57f4056d106008c7b4/merged major:0 minor:319 fsType:overlay blockSize:0} overlay_0-336:{mountpoint:/var/lib/containers/storage/overlay/1ebd9a88c6f96fc1e2ae85ea9f1c6b6ed83afc1e2f85105c82cf461f546a7c05/merged major:0 minor:336 fsType:overlay blockSize:0} overlay_0-338:{mountpoint:/var/lib/containers/storage/overlay/555b0f46ef458a401ebf21b96806623e820b674b69b7b49a01915b78d2f814da/merged major:0 minor:338 fsType:overlay blockSize:0} overlay_0-340:{mountpoint:/var/lib/containers/storage/overlay/f2b6a5247637983c8598b82b86eb4996d3ba3ebcef54f2b48467d106ddf37abe/merged major:0 minor:340 fsType:overlay blockSize:0} overlay_0-346:{mountpoint:/var/lib/containers/storage/overlay/92e8294b01a6f83a0d23a9f5284a24629a765ec206a91d1cfb5ddc9ce72cb905/merged major:0 minor:346 fsType:overlay blockSize:0} overlay_0-49:{mountpoint:/var/lib/containers/storage/overlay/9f7e6a954cfe08bdd99bff3f48a10925c226ce80bffca5c29d048f16606a5f0a/merged major:0 minor:49 fsType:overlay blockSize:0} overlay_0-53:{mountpoint:/var/lib/containers/storage/overlay/03061259050e0d64bb2282aa4e4a4e3d3a185192f01b352df091db243baed41a/merged major:0 minor:53 fsType:overlay blockSize:0} overlay_0-70:{mountpoint:/var/lib/containers/storage/overlay/a539bc73c0c1292dc3a6459f677af1693f712f6c428703554dc34de03afaf7c6/merged major:0 minor:70 fsType:overlay blockSize:0} overlay_0-74:{mountpoint:/var/lib/containers/storage/overlay/9c2cd866f0dc4534af53655c6d2f66ca3f40707ff0a58925d62cf9d2474b230c/merged major:0 minor:74 fsType:overlay blockSize:0} overlay_0-78:{mountpoint:/var/lib/containers/storage/overlay/2ab49cfc84d5c5f70a0ecb7c45133c2fee4d411e87b86430f15cb17ee135c94f/merged major:0 minor:78 fsType:overlay blockSize:0} overlay_0-83:{mountpoint:/var/lib/containers/storage/overlay/820b749dcf3d7d19c9f3af5abfa581d1de486483e7ea90ea8eeac13e48e31ecd/merged major:0 minor:83 fsType:overlay blockSize:0} overlay_0-86:{mountpoint:/var/lib/containers/storage/overlay/c94d80eb0ad149d75496e5bddbd8376799592cff8d8df9601d0dda278d7cf71a/merged major:0 minor:86 fsType:overlay blockSize:0} overlay_0-90:{mountpoint:/var/lib/containers/storage/overlay/ee7ab4749708c2d81ce609a34154f9a877cbd5ca71918344672501f2a970e2b4/merged major:0 minor:90 fsType:overlay blockSize:0} overlay_0-94:{mountpoint:/var/lib/containers/storage/overlay/cc47d7c45994b07cdeb86f0f145848c38b66790c4e699f6415ee195a7553f732/merged major:0 minor:94 fsType:overlay blockSize:0} overlay_0-98:{mountpoint:/var/lib/containers/storage/overlay/496105d8d6fd02d4e3a8c62fd6d48ba027bfebe1b9fab34c8768adb1b23b6374/merged major:0 minor:98 fsType:overlay blockSize:0}] Nov 03 00:40:51.813328 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:40:51.812772 4939 manager.go:217] Machine: {Timestamp:2026-11-03 00:40:51.812306399 +0000 UTC m=+0.076641340 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3101781 MemoryCapacity:33164337152 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec24d6a3924334a334030f111abe3d55 SystemUUID:ec24d6a3-9243-34a3-3403-0f111abe3d55 BootID:c386f19e-6a05-4b96-a313-40a9fad63fe5 Filesystems:[{Device:/run/containers/storage/overlay-containers/8867ce85967697dcfba6c103bcc04489490e98decd0f1994a213282757db1130/userdata/shm DeviceMajor:0 DeviceMinor:76 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-74 DeviceMajor:0 DeviceMinor:74 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-139 DeviceMajor:0 DeviceMinor:139 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-281 DeviceMajor:0 DeviceMinor:281 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/00dc2920-34f2-4e0a-892e-7759a16162d5/volumes/kubernetes.io~projected/kube-api-access-j62gg DeviceMajor:0 DeviceMinor:56 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/460779b2-3903-4b23-9b4d-e25142782a21/volumes/kubernetes.io~projected/kube-api-access-q4b9t DeviceMajor:0 DeviceMinor:66 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-182 DeviceMajor:0 DeviceMinor:182 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/ca0f883c-c2b7-4bf6-a90b-ffdbeac161a8/volumes/kubernetes.io~projected/kube-api-access-qcvwz DeviceMajor:0 DeviceMinor:216 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-268 DeviceMajor:0 DeviceMinor:268 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-90 DeviceMajor:0 DeviceMinor:90 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/e0c24e4a15e4f03b8a6d417c965c9bee757e311cc8ce503619d167866b0b6daa/userdata/shm DeviceMajor:0 DeviceMinor:92 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/ca0f883c-c2b7-4bf6-a90b-ffdbeac161a8/volumes/kubernetes.io~secret/serving-cert DeviceMajor:0 DeviceMinor:204 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/05e3d4718d6fafed62c11717499adc9949e4df03b37a914bd49d38181d2f088f/userdata/shm DeviceMajor:0 DeviceMinor:242 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-119 DeviceMajor:0 DeviceMinor:119 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-260 DeviceMajor:0 DeviceMinor:260 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/cf217a21-c1bd-48be-b203-f59743bceb33/volumes/kubernetes.io~secret/stats-auth DeviceMajor:0 DeviceMinor:213 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-78 DeviceMajor:0 DeviceMinor:78 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-98 DeviceMajor:0 DeviceMinor:98 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-157 DeviceMajor:0 DeviceMinor:157 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-248 DeviceMajor:0 DeviceMinor:248 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/cabd402c-79c5-4699-9c85-d415d362b4d0/volumes/kubernetes.io~secret/original-pull-secret DeviceMajor:0 DeviceMinor:288 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/a7619cc5d7ec9ca7f22e4b1d299dfaec1ec361e238eaabfd63843c47d0bc74ea/userdata/shm DeviceMajor:0 DeviceMinor:296 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-319 DeviceMajor:0 DeviceMinor:319 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/70a1f3b0a3f91bb331a7be1e85fed083f6f1bd0252c8c3c3c47a6dc14cf781c5/userdata/shm DeviceMajor:0 DeviceMinor:88 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-152 DeviceMajor:0 DeviceMinor:152 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:26 Capacity:6632869888 Type:vfs Inodes:819200 HasInodes:true} {Device:overlay_0-83 DeviceMajor:0 DeviceMinor:83 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-114 DeviceMajor:0 DeviceMinor:114 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/52a19842-3750-4d2b-bcb2-aebddee81097/volumes/kubernetes.io~projected/bound-sa-token DeviceMajor:0 DeviceMinor:226 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-262 DeviceMajor:0 DeviceMinor:262 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-102 DeviceMajor:0 DeviceMinor:102 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-172 DeviceMajor:0 DeviceMinor:172 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/cf217a21-c1bd-48be-b203-f59743bceb33/volumes/kubernetes.io~secret/default-certificate DeviceMajor:0 DeviceMinor:206 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/75481915-3cbe-475b-abe7-cb4e81c93427/volumes/kubernetes.io~projected/kube-api-access-qmkgl DeviceMajor:0 DeviceMinor:218 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/56261beb-517f-4c88-b59a-42ab41e278fd/volumes/kubernetes.io~projected/kube-api-access-h58rt DeviceMajor:0 DeviceMinor:219 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/2b050b18-31f6-4a22-9902-0435d11018b9/volumes/kubernetes.io~projected/kube-api-access-wzln7 DeviceMajor:0 DeviceMinor:221 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/dd0cd8a1-4d6b-424c-ac39-5f5ddb9cd87e/volumes/kubernetes.io~projected/bound-sa-token DeviceMajor:0 DeviceMinor:224 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-250 DeviceMajor:0 DeviceMinor:250 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/b748c4f8-d021-4d37-b390-d395482fcddd/volumes/kubernetes.io~empty-dir/tmp DeviceMajor:0 DeviceMinor:62 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/4112437df1446eb838fd35176a32848f7974fa73d27056b56cfcd766cd9894da/userdata/shm DeviceMajor:0 DeviceMinor:72 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-86 DeviceMajor:0 DeviceMinor:86 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-94 DeviceMajor:0 DeviceMinor:94 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-141 DeviceMajor:0 DeviceMinor:141 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/945da51a-32cb-4769-900d-1aeb266c7565/volumes/kubernetes.io~projected/kube-api-access-4q9wl DeviceMajor:0 DeviceMinor:334 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-53 DeviceMajor:0 DeviceMinor:53 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/1c5c6a32bfd30ad97469ed1fb0359d4cf5bc1d6c7254eef2ad99974885969c36/userdata/shm DeviceMajor:0 DeviceMinor:81 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/65d134ac-8a25-4c67-b6c7-cd752a098daf/volumes/kubernetes.io~secret/serving-cert DeviceMajor:0 DeviceMinor:210 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/b0bd4bd5-12ca-49df-a3d9-4243f4bd642e/volumes/kubernetes.io~projected/kube-api-access-7zxdt DeviceMajor:0 DeviceMinor:217 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/06973556-f51c-4d1c-ae55-fb77699bc5b3/volumes/kubernetes.io~projected/kube-api-access-4znwf DeviceMajor:0 DeviceMinor:220 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/36b9d258-cb2c-4d2b-880f-ae93b065034c/volumes/kubernetes.io~projected/kube-api-access-79khw DeviceMajor:0 DeviceMinor:225 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/d3e8e3bcd4255307bff4cba939cd812eaaae2e8342e455d9fa4ede17d02790c5/userdata/shm DeviceMajor:0 DeviceMinor:231 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/662fed273c35eab0aa9fe8026120a6defc961c8ef53105c04feef133bbb54108/userdata/shm DeviceMajor:0 DeviceMinor:238 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:24 Capacity:16582168576 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/98ca3972-fd0c-47f5-9d26-569a58e18bd1/volumes/kubernetes.io~secret/ovn-node-metrics-cert DeviceMajor:0 DeviceMinor:55 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/b748c4f8-d021-4d37-b390-d395482fcddd/volumes/kubernetes.io~projected/kube-api-access-hsvqw DeviceMajor:0 DeviceMinor:64 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/2b050b18-31f6-4a22-9902-0435d11018b9/volumes/kubernetes.io~secret/serving-cert DeviceMajor:0 DeviceMinor:205 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/b942f9e92ab3f2cf912cab48cffd6b46e23dd27024f8c374d7e54e710cdd082f/userdata/shm DeviceMajor:0 DeviceMinor:243 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-252 DeviceMajor:0 DeviceMinor:252 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-286 DeviceMajor:0 DeviceMinor:286 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-187 DeviceMajor:0 DeviceMinor:187 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/75481915-3cbe-475b-abe7-cb4e81c93427/volumes/kubernetes.io~secret/cert DeviceMajor:0 DeviceMinor:207 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/e86455a72def567d132d9ca3c0f5e9449cfde0bf23efbff663c0c38506834a86/userdata/shm DeviceMajor:0 DeviceMinor:246 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-264 DeviceMajor:0 DeviceMinor:264 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-104 DeviceMajor:0 DeviceMinor:104 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/36b9d258-cb2c-4d2b-880f-ae93b065034c/volumes/kubernetes.io~secret/serving-cert DeviceMajor:0 DeviceMinor:197 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/dd0cd8a1-4d6b-424c-ac39-5f5ddb9cd87e/volumes/kubernetes.io~secret/image-registry-private-configuration DeviceMajor:0 DeviceMinor:202 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/cf217a21-c1bd-48be-b203-f59743bceb33/volumes/kubernetes.io~projected/kube-api-access-fdrwp DeviceMajor:0 DeviceMinor:227 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-300 DeviceMajor:0 DeviceMinor:300 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-304 DeviceMajor:0 DeviceMinor:304 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-340 DeviceMajor:0 DeviceMinor:340 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/20d31a990fda9ed8069aa304f22413ca1c459b7d9b9b0e268ac30fa12af2d633/userdata/shm DeviceMajor:0 DeviceMinor:47 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/c3ffbe53-7f23-4750-b95d-476e6cfc58a6/volumes/kubernetes.io~secret/agent-certs DeviceMajor:0 DeviceMinor:61 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/b748c4f8-d021-4d37-b390-d395482fcddd/volumes/kubernetes.io~empty-dir/etc-tuned DeviceMajor:0 DeviceMinor:63 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/c5d2d73eebd8feaeb150e256ec01b80911dcc4ec8baffc65888eb14551386136/userdata/shm DeviceMajor:0 DeviceMinor:96 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-106 DeviceMajor:0 DeviceMinor:106 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-167 DeviceMajor:0 DeviceMinor:167 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/98ca3972-fd0c-47f5-9d26-569a58e18bd1/volume-subpaths/run-systemd/ovnkube-controller/6 DeviceMajor:0 DeviceMinor:26 Capacity:6632869888 Type:vfs Inodes:819200 HasInodes:true} {Device:/run/containers/storage/overlay-containers/ad4f86165f311454a0c13817aa2b917537300761a1b4dd713a21743be3bbc553/userdata/shm DeviceMajor:0 DeviceMinor:230 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/18b88862137befca64ff94878901feb875f51616dd9b95cd34057759a64e55c1/userdata/shm DeviceMajor:0 DeviceMinor:80 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-124 DeviceMajor:0 DeviceMinor:124 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-127 DeviceMajor:0 DeviceMinor:127 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/dd0cd8a1-4d6b-424c-ac39-5f5ddb9cd87e/volumes/kubernetes.io~secret/installation-pull-secrets DeviceMajor:0 DeviceMinor:201 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-254 DeviceMajor:0 DeviceMinor:254 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-266 DeviceMajor:0 DeviceMinor:266 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/8cd3ca22-e4ce-4896-ae31-97605af4e4c6/volumes/kubernetes.io~projected/kube-api-access-d8l25 DeviceMajor:0 DeviceMinor:293 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-294 DeviceMajor:0 DeviceMinor:294 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/dd0cd8a1-4d6b-424c-ac39-5f5ddb9cd87e/volumes/kubernetes.io~projected/kube-api-access-jbfhc DeviceMajor:0 DeviceMinor:223 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-338 DeviceMajor:0 DeviceMinor:338 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/136086c0-4846-437a-ad98-3cfef50610a0/volumes/kubernetes.io~projected/kube-api-access-llh2g DeviceMajor:0 DeviceMinor:60 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/37bf96f5-045f-49e8-83e8-410df66bfb1f/volumes/kubernetes.io~projected/kube-api-access-lp26w DeviceMajor:0 DeviceMinor:67 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/0444234d1c00351a0c0dec655e17f704fbb70c110f8904dc87014c9c860c40db/userdata/shm DeviceMajor:0 DeviceMinor:236 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-316 DeviceMajor:0 DeviceMinor:316 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/68658033-8a49-4430-869f-a7eb79603484/volumes/kubernetes.io~projected/kube-api-access-7wzbd DeviceMajor:0 DeviceMinor:318 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:32 Capacity:16582168576 Type:vfs Inodes:1048576 HasInodes:true} {Device:overlay_0-49 DeviceMajor:0 DeviceMinor:49 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-143 DeviceMajor:0 DeviceMinor:143 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/52a19842-3750-4d2b-bcb2-aebddee81097/volumes/kubernetes.io~secret/installation-pull-secrets DeviceMajor:0 DeviceMinor:45 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-284 DeviceMajor:0 DeviceMinor:284 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-346 DeviceMajor:0 DeviceMinor:346 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-302 DeviceMajor:0 DeviceMinor:302 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/c736fd3f-973f-4999-8d4e-c405079b2888/volumes/kubernetes.io~projected/kube-api-access-2tdmf DeviceMajor:0 DeviceMinor:59 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-162 DeviceMajor:0 DeviceMinor:162 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/65d134ac-8a25-4c67-b6c7-cd752a098daf/volumes/kubernetes.io~projected/kube-api-access-8j6kr DeviceMajor:0 DeviceMinor:209 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/6eafe6f6-cf90-446f-a8d5-620cd2e02775/volumes/kubernetes.io~projected/kube-api-access-6kcsm DeviceMajor:0 DeviceMinor:215 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/52a19842-3750-4d2b-bcb2-aebddee81097/volumes/kubernetes.io~secret/image-registry-private-configuration DeviceMajor:0 DeviceMinor:203 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/29a64a08-67ef-4c82-8a82-8cd4a4675327/volumes/kubernetes.io~projected/kube-api-access-pdmr4 DeviceMajor:0 DeviceMinor:212 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/var/lib/kubelet/pods/98ca3972-fd0c-47f5-9d26-569a58e18bd1/volumes/kubernetes.io~projected/kube-api-access-4bhqh DeviceMajor:0 DeviceMinor:57 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/b11da0d7baa29ab225e3891350eb46858d211cf3575d1b66e0c76ad4b382e2fa/userdata/shm DeviceMajor:0 DeviceMinor:240 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:composefs_0-34 DeviceMajor:0 DeviceMinor:34 Capacity:6422528 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:overlay_0-100 DeviceMajor:0 DeviceMinor:100 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-177 DeviceMajor:0 DeviceMinor:177 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-258 DeviceMajor:0 DeviceMinor:258 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-70 DeviceMajor:0 DeviceMinor:70 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/52a19842-3750-4d2b-bcb2-aebddee81097/volumes/kubernetes.io~projected/kube-api-access-tc9rq DeviceMajor:0 DeviceMinor:208 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/6650035b-2220-491c-9683-8a03ccb7af6a/volumes/kubernetes.io~projected/kube-api-access-rkf7s DeviceMajor:0 DeviceMinor:58 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/ad5bf442-729d-411c-96e4-2e5105ebadcf/volumes/kubernetes.io~projected/kube-api-access-sfdp9 DeviceMajor:0 DeviceMinor:65 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-192 DeviceMajor:0 DeviceMinor:192 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/4237805a-d475-4bd7-865c-c6ef371c0e11/volumes/kubernetes.io~projected/kube-api-access-nswxk DeviceMajor:0 DeviceMinor:214 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-256 DeviceMajor:0 DeviceMinor:256 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/2b3856ef34d4cbba8d89162e727bb1a5ed2a806920a8e038711a17170afd7bc6/userdata/shm DeviceMajor:0 DeviceMinor:329 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-132 DeviceMajor:0 DeviceMinor:132 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/6708f186ee33a9346160371d6e1ddfb5e3a0cbdec1bf050f2ee75b910e994ce9/userdata/shm DeviceMajor:0 DeviceMinor:228 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/487ea3f04a7119c9c89cde91e9dda8b21902247978fb80fbfd356adf8237b019/userdata/shm DeviceMajor:0 DeviceMinor:234 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/a012e88fc324e4ecd1bc5f9557af5fe91bc73d451cee98e718e6ecf7cf8da48f/userdata/shm DeviceMajor:0 DeviceMinor:51 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/69b087f8-4a23-4ec8-808e-b61d2e16f67b/volumes/kubernetes.io~projected/kube-api-access-pj6ft DeviceMajor:0 DeviceMinor:211 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-306 DeviceMajor:0 DeviceMinor:306 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/3f1b02020b063d5bec095ef4de846f7d7590f5dec7e9630c6f92094ff6c01081/userdata/shm DeviceMajor:0 DeviceMinor:68 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/525dc47d-afde-4019-ad01-80614291f282/volumes/kubernetes.io~projected/kube-api-access-sbn8x DeviceMajor:0 DeviceMinor:222 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/a5114a1c8f2cd400e64362ba4b984bd6b439505b63ac0a12be3add35ec5a48e7/userdata/shm DeviceMajor:0 DeviceMinor:297 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-336 DeviceMajor:0 DeviceMinor:336 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:0444234d1c00351 MacAddress:9a:bb:b6:c4:bd:e5 Speed:10000 Mtu:8901} {Name:05e3d4718d6fafe MacAddress:72:cd:57:20:e0:09 Speed:10000 Mtu:8901} {Name:2b3856ef34d4cbb MacAddress:3e:ed:0f:d3:75:f7 Speed:10000 Mtu:8901} {Name:487ea3f04a7119c MacAddress:0a:94:5f:8c:53:01 Speed:10000 Mtu:8901} {Name:662fed273c35eab MacAddress:f2:32:a8:e3:21:fe Speed:10000 Mtu:8901} {Name:6708f186ee33a93 MacAddress:0a:c8:f4:2d:d1:de Speed:10000 Mtu:8901} {Name:a5114a1c8f2cd40 MacAddress:fa:9b:3c:9f:43:55 Speed:10000 Mtu:8901} {Name:a7619cc5d7ec9ca MacAddress:66:00:a0:89:55:0e Speed:10000 Mtu:8901} {Name:ad4f86165f31145 MacAddress:3e:41:9c:18:89:0c Speed:10000 Mtu:8901} {Name:b11da0d7baa29ab MacAddress:fa:1a:d0:43:ab:9e Speed:10000 Mtu:8901} {Name:b942f9e92ab3f2c MacAddress:12:7b:30:ab:df:a8 Speed:10000 Mtu:8901} {Name:br-ex MacAddress:02:ff:f4:66:da:9b Speed:0 Mtu:9001} {Name:br-int MacAddress:5a:36:90:eb:17:1e Speed:0 Mtu:8901} {Name:d3e8e3bcd425530 MacAddress:d6:e5:b2:07:d7:e2 Speed:10000 Mtu:8901} {Name:e86455a72def567 MacAddress:46:df:73:08:59:c2 Speed:10000 Mtu:8901} {Name:ens5 MacAddress:02:ff:f4:66:da:9b Speed:0 Mtu:9001} {Name:genev_sys_6081 MacAddress:42:74:a1:5f:59:f4 Speed:0 Mtu:65000} {Name:ovn-k8s-mp0 MacAddress:0a:58:0a:84:00:02 Speed:0 Mtu:8901} {Name:ovs-system MacAddress:42:b8:eb:2d:a2:38 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164337152 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None} Nov 03 00:40:51.813328 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:40:51.813182 4939 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available. Nov 03 00:40:51.813328 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:40:51.813230 4939 manager.go:233] Version: {KernelVersion:5.14.0-687.60.1.el9_8.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.8.20261023-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:} Nov 03 00:40:51.813583 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:40:51.813510 4939 container_manager_linux.go:275] "Container manager verified user specified cgroup-root exists" cgroupRoot=[] Nov 03 00:40:51.813683 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:40:51.813535 4939 container_manager_linux.go:280] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-140-107.us-east-2.compute.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"/system.slice","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{},"system-reserved-compressible":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2} Nov 03 00:40:51.813724 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:40:51.813686 4939 topology_manager.go:143] "Creating topology manager with none policy" Nov 03 00:40:51.813724 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:40:51.813695 4939 container_manager_linux.go:311] "Creating device plugin manager" Nov 03 00:40:51.813724 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:40:51.813705 4939 manager.go:144] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock" Nov 03 00:40:51.813808 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:40:51.813724 4939 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock" Nov 03 00:40:51.813808 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:40:51.813743 4939 container_manager_linux.go:320] "Creating Dynamic Resource Allocation (DRA) manager" Nov 03 00:40:51.813895 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:40:51.813885 4939 state_mem.go:41] "Initialized" logger="CPUManager state memory" Nov 03 00:40:51.814042 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:40:51.814032 4939 server.go:1235] "Using root directory" path="/var/lib/kubelet" Nov 03 00:40:51.814075 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:40:51.814068 4939 kubelet.go:493] "Attempting to sync node with API server" Nov 03 00:40:51.814102 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:40:51.814078 4939 kubelet.go:394] "Adding static pod path" path="/etc/kubernetes/manifests" Nov 03 00:40:51.814102 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:40:51.814091 4939 file.go:69] "Watching path" path="/etc/kubernetes/manifests" Nov 03 00:40:51.814102 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:40:51.814101 4939 kubelet.go:405] "Adding apiserver pod source" Nov 03 00:40:51.814183 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:40:51.814108 4939 apiserver.go:42] "Waiting for node sync before watching apiserver pods" Nov 03 00:40:51.814987 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:40:51.814972 4939 state_checkpoint.go:81] "State checkpoint: restored pod resource state from checkpoint" Nov 03 00:40:51.814987 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:40:51.814985 4939 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Nov 03 00:40:51.815451 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:40:51.815431 4939 kuberuntime_manager.go:304] "Container runtime initialized" containerRuntime="cri-o" version="1.35.10-3.rhaos4.22.git9383502.el9" apiVersion="v1" Nov 03 00:40:51.817954 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:40:51.817938 4939 state_checkpoint.go:81] "State checkpoint: restored pod resource state from checkpoint" Nov 03 00:40:51.817954 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:40:51.817952 4939 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Nov 03 00:40:51.818106 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:40:51.818080 4939 certificate_store.go:147] "Loading cert/key pair from a file" filePath="/var/lib/kubelet/pki/kubelet-server-current.pem" Nov 03 00:40:51.818307 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:40:51.818291 4939 kubelet.go:963] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled" Nov 03 00:40:51.818376 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:40:51.818332 4939 kubelet.go:990] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled" Nov 03 00:40:51.818456 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:40:51.818445 4939 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume" Nov 03 00:40:51.818515 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:40:51.818464 4939 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir" Nov 03 00:40:51.818515 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:40:51.818476 4939 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo" Nov 03 00:40:51.818515 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:40:51.818486 4939 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path" Nov 03 00:40:51.818515 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:40:51.818496 4939 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs" Nov 03 00:40:51.818692 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:40:51.818521 4939 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret" Nov 03 00:40:51.818692 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:40:51.818531 4939 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi" Nov 03 00:40:51.818692 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:40:51.818541 4939 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api" Nov 03 00:40:51.818692 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:40:51.818552 4939 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc" Nov 03 00:40:51.818692 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:40:51.818561 4939 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap" Nov 03 00:40:51.818692 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:40:51.818579 4939 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected" Nov 03 00:40:51.818692 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:40:51.818595 4939 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume" Nov 03 00:40:51.818692 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:40:51.818635 4939 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi" Nov 03 00:40:51.818692 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:40:51.818644 4939 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image" Nov 03 00:40:51.822182 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:40:51.822168 4939 server.go:1263] "Started kubelet" Nov 03 00:40:51.822336 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:40:51.822303 4939 server.go:182] "Starting to listen" address="0.0.0.0" port=10250 Nov 03 00:40:51.822520 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:40:51.822475 4939 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10 Nov 03 00:40:51.822570 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:40:51.822542 4939 server_v1.go:49] "podresources" method="list" useActivePods=true Nov 03 00:40:51.822792 ip-10-0-140-107 systemd[1]: Started Kubernetes Kubelet. Nov 03 00:40:51.822964 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:40:51.822948 4939 server.go:254] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock" Nov 03 00:40:51.823832 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:40:51.823814 4939 server.go:317] "Adding debug handlers to kubelet server" Nov 03 00:40:51.833185 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:40:51.833163 4939 fs_resource_analyzer.go:69] "Starting FS ResourceAnalyzer" Nov 03 00:40:51.835414 ip-10-0-140-107 kubenswrapper[4939]: E1103 00:40:51.833260 4939 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-140-107.us-east-2.compute.internal\" not found" Nov 03 00:40:51.835414 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:40:51.833266 4939 volume_manager.go:309] "The desired_state_of_world populator starts" Nov 03 00:40:51.835414 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:40:51.833283 4939 desired_state_of_world_populator.go:146] "Desired state populator starts to run" Nov 03 00:40:51.835414 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:40:51.833286 4939 volume_manager.go:311] "Starting Kubelet Volume Manager" Nov 03 00:40:51.835414 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:40:51.833304 4939 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving" Nov 03 00:40:51.835414 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:40:51.833495 4939 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-11-02 00:35:12 +0000 UTC" deadline="2028-05-14 18:49:15.964862892 +0000 UTC" Nov 03 00:40:51.835414 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:40:51.833517 4939 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="13410h8m24.131348641s" Nov 03 00:40:51.835414 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:40:51.833833 4939 factory.go:55] Registering systemd factory Nov 03 00:40:51.835414 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:40:51.833868 4939 factory.go:223] Registration of the systemd container factory successfully Nov 03 00:40:51.835414 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:40:51.834733 4939 factory.go:153] Registering CRI-O factory Nov 03 00:40:51.835414 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:40:51.834749 4939 factory.go:223] Registration of the crio container factory successfully Nov 03 00:40:51.835414 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:40:51.834801 4939 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory Nov 03 00:40:51.835414 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:40:51.834820 4939 factory.go:103] Registering Raw factory Nov 03 00:40:51.835414 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:40:51.834834 4939 manager.go:1199] Started watching for new ooms in manager Nov 03 00:40:51.835414 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:40:51.835219 4939 manager.go:319] Starting recovery of all containers Nov 03 00:40:51.836652 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:40:51.836580 4939 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="2b050b18-31f6-4a22-9902-0435d11018b9" volumeName="kubernetes.io/projected/2b050b18-31f6-4a22-9902-0435d11018b9-kube-api-access-wzln7" seLinuxMountContext="" Nov 03 00:40:51.836753 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:40:51.836662 4939 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="2b050b18-31f6-4a22-9902-0435d11018b9" volumeName="kubernetes.io/secret/2b050b18-31f6-4a22-9902-0435d11018b9-serving-cert" seLinuxMountContext="" Nov 03 00:40:51.836753 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:40:51.836692 4939 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="4237805a-d475-4bd7-865c-c6ef371c0e11" volumeName="kubernetes.io/projected/4237805a-d475-4bd7-865c-c6ef371c0e11-kube-api-access-nswxk" seLinuxMountContext="" Nov 03 00:40:51.836753 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:40:51.836704 4939 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="460779b2-3903-4b23-9b4d-e25142782a21" volumeName="kubernetes.io/configmap/460779b2-3903-4b23-9b4d-e25142782a21-serviceca" seLinuxMountContext="" Nov 03 00:40:51.836753 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:40:51.836718 4939 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="ad5bf442-729d-411c-96e4-2e5105ebadcf" volumeName="kubernetes.io/projected/ad5bf442-729d-411c-96e4-2e5105ebadcf-kube-api-access-sfdp9" seLinuxMountContext="" Nov 03 00:40:51.836753 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:40:51.836730 4939 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="06973556-f51c-4d1c-ae55-fb77699bc5b3" volumeName="kubernetes.io/configmap/06973556-f51c-4d1c-ae55-fb77699bc5b3-telemetry-config" seLinuxMountContext="" Nov 03 00:40:51.836753 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:40:51.836741 4939 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="136086c0-4846-437a-ad98-3cfef50610a0" volumeName="kubernetes.io/projected/136086c0-4846-437a-ad98-3cfef50610a0-kube-api-access-llh2g" seLinuxMountContext="" Nov 03 00:40:51.836753 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:40:51.836752 4939 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="6650035b-2220-491c-9683-8a03ccb7af6a" volumeName="kubernetes.io/configmap/6650035b-2220-491c-9683-8a03ccb7af6a-cni-binary-copy" seLinuxMountContext="" Nov 03 00:40:51.837173 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:40:51.836766 4939 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="6650035b-2220-491c-9683-8a03ccb7af6a" volumeName="kubernetes.io/projected/6650035b-2220-491c-9683-8a03ccb7af6a-kube-api-access-rkf7s" seLinuxMountContext="" Nov 03 00:40:51.837173 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:40:51.836786 4939 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="ca0f883c-c2b7-4bf6-a90b-ffdbeac161a8" volumeName="kubernetes.io/secret/ca0f883c-c2b7-4bf6-a90b-ffdbeac161a8-serving-cert" seLinuxMountContext="" Nov 03 00:40:51.837173 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:40:51.836799 4939 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="69b087f8-4a23-4ec8-808e-b61d2e16f67b" volumeName="kubernetes.io/projected/69b087f8-4a23-4ec8-808e-b61d2e16f67b-kube-api-access-pj6ft" seLinuxMountContext="" Nov 03 00:40:51.837173 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:40:51.836828 4939 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="460779b2-3903-4b23-9b4d-e25142782a21" volumeName="kubernetes.io/projected/460779b2-3903-4b23-9b4d-e25142782a21-kube-api-access-q4b9t" seLinuxMountContext="" Nov 03 00:40:51.837173 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:40:51.836840 4939 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="525dc47d-afde-4019-ad01-80614291f282" volumeName="kubernetes.io/projected/525dc47d-afde-4019-ad01-80614291f282-kube-api-access-sbn8x" seLinuxMountContext="" Nov 03 00:40:51.837173 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:40:51.836861 4939 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="2b050b18-31f6-4a22-9902-0435d11018b9" volumeName="kubernetes.io/configmap/2b050b18-31f6-4a22-9902-0435d11018b9-config" seLinuxMountContext="" Nov 03 00:40:51.837173 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:40:51.836875 4939 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="bc513cba-d924-4719-a2ba-711529175ee0" volumeName="kubernetes.io/configmap/bc513cba-d924-4719-a2ba-711529175ee0-nginx-conf" seLinuxMountContext="" Nov 03 00:40:51.837173 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:40:51.836894 4939 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="dd0cd8a1-4d6b-424c-ac39-5f5ddb9cd87e" volumeName="kubernetes.io/projected/dd0cd8a1-4d6b-424c-ac39-5f5ddb9cd87e-bound-sa-token" seLinuxMountContext="" Nov 03 00:40:51.837173 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:40:51.836926 4939 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="dd0cd8a1-4d6b-424c-ac39-5f5ddb9cd87e" volumeName="kubernetes.io/secret/dd0cd8a1-4d6b-424c-ac39-5f5ddb9cd87e-installation-pull-secrets" seLinuxMountContext="" Nov 03 00:40:51.837173 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:40:51.836947 4939 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="6650035b-2220-491c-9683-8a03ccb7af6a" volumeName="kubernetes.io/configmap/6650035b-2220-491c-9683-8a03ccb7af6a-whereabouts-flatfile-configmap" seLinuxMountContext="" Nov 03 00:40:51.837173 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:40:51.836959 4939 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="65d134ac-8a25-4c67-b6c7-cd752a098daf" volumeName="kubernetes.io/configmap/65d134ac-8a25-4c67-b6c7-cd752a098daf-config" seLinuxMountContext="" Nov 03 00:40:51.837173 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:40:51.836970 4939 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="36b9d258-cb2c-4d2b-880f-ae93b065034c" volumeName="kubernetes.io/configmap/36b9d258-cb2c-4d2b-880f-ae93b065034c-config" seLinuxMountContext="" Nov 03 00:40:51.837173 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:40:51.836981 4939 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="98ca3972-fd0c-47f5-9d26-569a58e18bd1" volumeName="kubernetes.io/configmap/98ca3972-fd0c-47f5-9d26-569a58e18bd1-env-overrides" seLinuxMountContext="" Nov 03 00:40:51.837173 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:40:51.836992 4939 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="c3ffbe53-7f23-4750-b95d-476e6cfc58a6" volumeName="kubernetes.io/secret/c3ffbe53-7f23-4750-b95d-476e6cfc58a6-agent-certs" seLinuxMountContext="" Nov 03 00:40:51.837173 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:40:51.837005 4939 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="dd0cd8a1-4d6b-424c-ac39-5f5ddb9cd87e" volumeName="kubernetes.io/secret/dd0cd8a1-4d6b-424c-ac39-5f5ddb9cd87e-image-registry-private-configuration" seLinuxMountContext="" Nov 03 00:40:51.837173 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:40:51.837015 4939 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="52a19842-3750-4d2b-bcb2-aebddee81097" volumeName="kubernetes.io/secret/52a19842-3750-4d2b-bcb2-aebddee81097-installation-pull-secrets" seLinuxMountContext="" Nov 03 00:40:51.837173 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:40:51.837027 4939 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="56261beb-517f-4c88-b59a-42ab41e278fd" volumeName="kubernetes.io/projected/56261beb-517f-4c88-b59a-42ab41e278fd-kube-api-access-h58rt" seLinuxMountContext="" Nov 03 00:40:51.837173 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:40:51.837048 4939 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="65d134ac-8a25-4c67-b6c7-cd752a098daf" volumeName="kubernetes.io/configmap/65d134ac-8a25-4c67-b6c7-cd752a098daf-trusted-ca" seLinuxMountContext="" Nov 03 00:40:51.837173 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:40:51.837061 4939 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="75481915-3cbe-475b-abe7-cb4e81c93427" volumeName="kubernetes.io/secret/75481915-3cbe-475b-abe7-cb4e81c93427-cert" seLinuxMountContext="" Nov 03 00:40:51.837173 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:40:51.837073 4939 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="8cd3ca22-e4ce-4896-ae31-97605af4e4c6" volumeName="kubernetes.io/projected/8cd3ca22-e4ce-4896-ae31-97605af4e4c6-kube-api-access-d8l25" seLinuxMountContext="" Nov 03 00:40:51.837173 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:40:51.837090 4939 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="ca0f883c-c2b7-4bf6-a90b-ffdbeac161a8" volumeName="kubernetes.io/empty-dir/ca0f883c-c2b7-4bf6-a90b-ffdbeac161a8-snapshots" seLinuxMountContext="" Nov 03 00:40:51.837173 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:40:51.837103 4939 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="98ca3972-fd0c-47f5-9d26-569a58e18bd1" volumeName="kubernetes.io/configmap/98ca3972-fd0c-47f5-9d26-569a58e18bd1-ovnkube-script-lib" seLinuxMountContext="" Nov 03 00:40:51.837173 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:40:51.837133 4939 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="98ca3972-fd0c-47f5-9d26-569a58e18bd1" volumeName="kubernetes.io/configmap/98ca3972-fd0c-47f5-9d26-569a58e18bd1-ovnkube-config" seLinuxMountContext="" Nov 03 00:40:51.837173 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:40:51.837158 4939 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="52a19842-3750-4d2b-bcb2-aebddee81097" volumeName="kubernetes.io/configmap/52a19842-3750-4d2b-bcb2-aebddee81097-trusted-ca" seLinuxMountContext="" Nov 03 00:40:51.837173 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:40:51.837171 4939 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="52a19842-3750-4d2b-bcb2-aebddee81097" volumeName="kubernetes.io/empty-dir/52a19842-3750-4d2b-bcb2-aebddee81097-ca-trust-extracted" seLinuxMountContext="" Nov 03 00:40:51.838307 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:40:51.837193 4939 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="52a19842-3750-4d2b-bcb2-aebddee81097" volumeName="kubernetes.io/projected/52a19842-3750-4d2b-bcb2-aebddee81097-kube-api-access-tc9rq" seLinuxMountContext="" Nov 03 00:40:51.838307 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:40:51.837216 4939 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="945da51a-32cb-4769-900d-1aeb266c7565" volumeName="kubernetes.io/projected/945da51a-32cb-4769-900d-1aeb266c7565-kube-api-access-4q9wl" seLinuxMountContext="" Nov 03 00:40:51.838307 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:40:51.837229 4939 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="ca0f883c-c2b7-4bf6-a90b-ffdbeac161a8" volumeName="kubernetes.io/configmap/ca0f883c-c2b7-4bf6-a90b-ffdbeac161a8-service-ca-bundle" seLinuxMountContext="" Nov 03 00:40:51.838307 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:40:51.837240 4939 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="ca0f883c-c2b7-4bf6-a90b-ffdbeac161a8" volumeName="kubernetes.io/projected/ca0f883c-c2b7-4bf6-a90b-ffdbeac161a8-kube-api-access-qcvwz" seLinuxMountContext="" Nov 03 00:40:51.838307 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:40:51.837260 4939 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="b748c4f8-d021-4d37-b390-d395482fcddd" volumeName="kubernetes.io/empty-dir/b748c4f8-d021-4d37-b390-d395482fcddd-tmp" seLinuxMountContext="" Nov 03 00:40:51.838307 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:40:51.837269 4939 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="b748c4f8-d021-4d37-b390-d395482fcddd" volumeName="kubernetes.io/projected/b748c4f8-d021-4d37-b390-d395482fcddd-kube-api-access-hsvqw" seLinuxMountContext="" Nov 03 00:40:51.838307 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:40:51.837277 4939 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="00dc2920-34f2-4e0a-892e-7759a16162d5" volumeName="kubernetes.io/projected/00dc2920-34f2-4e0a-892e-7759a16162d5-kube-api-access-j62gg" seLinuxMountContext="" Nov 03 00:40:51.838307 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:40:51.837291 4939 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="37bf96f5-045f-49e8-83e8-410df66bfb1f" volumeName="kubernetes.io/projected/37bf96f5-045f-49e8-83e8-410df66bfb1f-kube-api-access-lp26w" seLinuxMountContext="" Nov 03 00:40:51.838307 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:40:51.837305 4939 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="52a19842-3750-4d2b-bcb2-aebddee81097" volumeName="kubernetes.io/secret/52a19842-3750-4d2b-bcb2-aebddee81097-image-registry-private-configuration" seLinuxMountContext="" Nov 03 00:40:51.838307 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:40:51.837312 4939 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="68658033-8a49-4430-869f-a7eb79603484" volumeName="kubernetes.io/projected/68658033-8a49-4430-869f-a7eb79603484-kube-api-access-7wzbd" seLinuxMountContext="" Nov 03 00:40:51.838307 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:40:51.837329 4939 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="c736fd3f-973f-4999-8d4e-c405079b2888" volumeName="kubernetes.io/configmap/c736fd3f-973f-4999-8d4e-c405079b2888-iptables-alerter-script" seLinuxMountContext="" Nov 03 00:40:51.838307 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:40:51.837339 4939 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="dd0cd8a1-4d6b-424c-ac39-5f5ddb9cd87e" volumeName="kubernetes.io/empty-dir/dd0cd8a1-4d6b-424c-ac39-5f5ddb9cd87e-ca-trust-extracted" seLinuxMountContext="" Nov 03 00:40:51.838307 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:40:51.837346 4939 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="69b087f8-4a23-4ec8-808e-b61d2e16f67b" volumeName="kubernetes.io/configmap/69b087f8-4a23-4ec8-808e-b61d2e16f67b-config-volume" seLinuxMountContext="" Nov 03 00:40:51.838307 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:40:51.837354 4939 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="98ca3972-fd0c-47f5-9d26-569a58e18bd1" volumeName="kubernetes.io/projected/98ca3972-fd0c-47f5-9d26-569a58e18bd1-kube-api-access-4bhqh" seLinuxMountContext="" Nov 03 00:40:51.838307 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:40:51.837368 4939 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="98ca3972-fd0c-47f5-9d26-569a58e18bd1" volumeName="kubernetes.io/secret/98ca3972-fd0c-47f5-9d26-569a58e18bd1-ovn-node-metrics-cert" seLinuxMountContext="" Nov 03 00:40:51.838307 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:40:51.837375 4939 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="52a19842-3750-4d2b-bcb2-aebddee81097" volumeName="kubernetes.io/projected/52a19842-3750-4d2b-bcb2-aebddee81097-bound-sa-token" seLinuxMountContext="" Nov 03 00:40:51.838307 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:40:51.837383 4939 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="00dc2920-34f2-4e0a-892e-7759a16162d5" volumeName="kubernetes.io/configmap/00dc2920-34f2-4e0a-892e-7759a16162d5-cni-binary-copy" seLinuxMountContext="" Nov 03 00:40:51.838307 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:40:51.837391 4939 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="29a64a08-67ef-4c82-8a82-8cd4a4675327" volumeName="kubernetes.io/projected/29a64a08-67ef-4c82-8a82-8cd4a4675327-kube-api-access-pdmr4" seLinuxMountContext="" Nov 03 00:40:51.838307 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:40:51.837399 4939 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="36b9d258-cb2c-4d2b-880f-ae93b065034c" volumeName="kubernetes.io/projected/36b9d258-cb2c-4d2b-880f-ae93b065034c-kube-api-access-79khw" seLinuxMountContext="" Nov 03 00:40:51.838307 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:40:51.837406 4939 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="52a19842-3750-4d2b-bcb2-aebddee81097" volumeName="kubernetes.io/configmap/52a19842-3750-4d2b-bcb2-aebddee81097-registry-certificates" seLinuxMountContext="" Nov 03 00:40:51.838307 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:40:51.837413 4939 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="36b9d258-cb2c-4d2b-880f-ae93b065034c" volumeName="kubernetes.io/secret/36b9d258-cb2c-4d2b-880f-ae93b065034c-serving-cert" seLinuxMountContext="" Nov 03 00:40:51.838307 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:40:51.837421 4939 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="69b087f8-4a23-4ec8-808e-b61d2e16f67b" volumeName="kubernetes.io/empty-dir/69b087f8-4a23-4ec8-808e-b61d2e16f67b-tmp-dir" seLinuxMountContext="" Nov 03 00:40:51.838307 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:40:51.837428 4939 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="6eafe6f6-cf90-446f-a8d5-620cd2e02775" volumeName="kubernetes.io/projected/6eafe6f6-cf90-446f-a8d5-620cd2e02775-kube-api-access-6kcsm" seLinuxMountContext="" Nov 03 00:40:51.838307 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:40:51.837442 4939 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="b748c4f8-d021-4d37-b390-d395482fcddd" volumeName="kubernetes.io/empty-dir/b748c4f8-d021-4d37-b390-d395482fcddd-etc-tuned" seLinuxMountContext="" Nov 03 00:40:51.838307 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:40:51.837450 4939 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="65d134ac-8a25-4c67-b6c7-cd752a098daf" volumeName="kubernetes.io/secret/65d134ac-8a25-4c67-b6c7-cd752a098daf-serving-cert" seLinuxMountContext="" Nov 03 00:40:51.838307 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:40:51.837459 4939 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="c3ffbe53-7f23-4750-b95d-476e6cfc58a6" volumeName="kubernetes.io/configmap/c3ffbe53-7f23-4750-b95d-476e6cfc58a6-konnectivity-ca" seLinuxMountContext="" Nov 03 00:40:51.838307 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:40:51.837466 4939 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="c736fd3f-973f-4999-8d4e-c405079b2888" volumeName="kubernetes.io/projected/c736fd3f-973f-4999-8d4e-c405079b2888-kube-api-access-2tdmf" seLinuxMountContext="" Nov 03 00:40:51.838307 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:40:51.837473 4939 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="ca0f883c-c2b7-4bf6-a90b-ffdbeac161a8" volumeName="kubernetes.io/empty-dir/ca0f883c-c2b7-4bf6-a90b-ffdbeac161a8-tmp" seLinuxMountContext="" Nov 03 00:40:51.838307 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:40:51.837480 4939 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="06973556-f51c-4d1c-ae55-fb77699bc5b3" volumeName="kubernetes.io/projected/06973556-f51c-4d1c-ae55-fb77699bc5b3-kube-api-access-4znwf" seLinuxMountContext="" Nov 03 00:40:51.838307 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:40:51.837487 4939 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="6eafe6f6-cf90-446f-a8d5-620cd2e02775" volumeName="kubernetes.io/configmap/6eafe6f6-cf90-446f-a8d5-620cd2e02775-samples-operator-config" seLinuxMountContext="" Nov 03 00:40:51.838307 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:40:51.837495 4939 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="945da51a-32cb-4769-900d-1aeb266c7565" volumeName="kubernetes.io/empty-dir/945da51a-32cb-4769-900d-1aeb266c7565-data-volume" seLinuxMountContext="" Nov 03 00:40:51.838307 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:40:51.837504 4939 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="ad5bf442-729d-411c-96e4-2e5105ebadcf" volumeName="kubernetes.io/empty-dir/ad5bf442-729d-411c-96e4-2e5105ebadcf-tmp-dir" seLinuxMountContext="" Nov 03 00:40:51.838307 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:40:51.837512 4939 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="b0bd4bd5-12ca-49df-a3d9-4243f4bd642e" volumeName="kubernetes.io/projected/b0bd4bd5-12ca-49df-a3d9-4243f4bd642e-kube-api-access-7zxdt" seLinuxMountContext="" Nov 03 00:40:51.838307 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:40:51.837520 4939 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="cabd402c-79c5-4699-9c85-d415d362b4d0" volumeName="kubernetes.io/secret/cabd402c-79c5-4699-9c85-d415d362b4d0-original-pull-secret" seLinuxMountContext="" Nov 03 00:40:51.838307 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:40:51.837532 4939 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="65d134ac-8a25-4c67-b6c7-cd752a098daf" volumeName="kubernetes.io/projected/65d134ac-8a25-4c67-b6c7-cd752a098daf-kube-api-access-8j6kr" seLinuxMountContext="" Nov 03 00:40:51.838307 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:40:51.837539 4939 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="cf217a21-c1bd-48be-b203-f59743bceb33" volumeName="kubernetes.io/projected/cf217a21-c1bd-48be-b203-f59743bceb33-kube-api-access-fdrwp" seLinuxMountContext="" Nov 03 00:40:51.838307 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:40:51.837546 4939 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="dd0cd8a1-4d6b-424c-ac39-5f5ddb9cd87e" volumeName="kubernetes.io/configmap/dd0cd8a1-4d6b-424c-ac39-5f5ddb9cd87e-registry-certificates" seLinuxMountContext="" Nov 03 00:40:51.838307 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:40:51.837554 4939 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="dd0cd8a1-4d6b-424c-ac39-5f5ddb9cd87e" volumeName="kubernetes.io/configmap/dd0cd8a1-4d6b-424c-ac39-5f5ddb9cd87e-trusted-ca" seLinuxMountContext="" Nov 03 00:40:51.838307 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:40:51.837561 4939 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="dd0cd8a1-4d6b-424c-ac39-5f5ddb9cd87e" volumeName="kubernetes.io/projected/dd0cd8a1-4d6b-424c-ac39-5f5ddb9cd87e-kube-api-access-jbfhc" seLinuxMountContext="" Nov 03 00:40:51.838307 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:40:51.837568 4939 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="00dc2920-34f2-4e0a-892e-7759a16162d5" volumeName="kubernetes.io/configmap/00dc2920-34f2-4e0a-892e-7759a16162d5-multus-daemon-config" seLinuxMountContext="" Nov 03 00:40:51.838307 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:40:51.837576 4939 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="cf217a21-c1bd-48be-b203-f59743bceb33" volumeName="kubernetes.io/secret/cf217a21-c1bd-48be-b203-f59743bceb33-stats-auth" seLinuxMountContext="" Nov 03 00:40:51.838307 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:40:51.837584 4939 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="cf217a21-c1bd-48be-b203-f59743bceb33" volumeName="kubernetes.io/secret/cf217a21-c1bd-48be-b203-f59743bceb33-default-certificate" seLinuxMountContext="" Nov 03 00:40:51.838307 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:40:51.837591 4939 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="6650035b-2220-491c-9683-8a03ccb7af6a" volumeName="kubernetes.io/configmap/6650035b-2220-491c-9683-8a03ccb7af6a-cni-sysctl-allowlist" seLinuxMountContext="" Nov 03 00:40:51.838307 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:40:51.837598 4939 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="75481915-3cbe-475b-abe7-cb4e81c93427" volumeName="kubernetes.io/projected/75481915-3cbe-475b-abe7-cb4e81c93427-kube-api-access-qmkgl" seLinuxMountContext="" Nov 03 00:40:51.838307 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:40:51.837606 4939 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="ca0f883c-c2b7-4bf6-a90b-ffdbeac161a8" volumeName="kubernetes.io/configmap/ca0f883c-c2b7-4bf6-a90b-ffdbeac161a8-trusted-ca-bundle" seLinuxMountContext="" Nov 03 00:40:51.838307 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:40:51.837614 4939 reconstruct.go:97] "Volume reconstruction finished" Nov 03 00:40:51.838307 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:40:51.837620 4939 reconciler.go:29] "Reconciler: start to sync state" Nov 03 00:40:51.838307 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:40:51.837864 4939 reflector.go:446] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:161" Nov 03 00:40:51.840563 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:40:51.838673 4939 reflector.go:446] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:161" Nov 03 00:40:51.840563 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:40:51.840371 4939 reflector.go:446] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:161" Nov 03 00:40:51.842335 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:40:51.842308 4939 reconstruct.go:205] "DevicePaths of reconstructed volumes updated" Nov 03 00:40:51.851126 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:40:51.851103 4939 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-n42s6_65d134ac-8a25-4c67-b6c7-cd752a098daf/console-operator/1.log" Nov 03 00:40:51.852031 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:40:51.852010 4939 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-n42s6_65d134ac-8a25-4c67-b6c7-cd752a098daf/console-operator/1.log" Nov 03 00:40:51.858663 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:40:51.858634 4939 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4" Nov 03 00:40:51.860027 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:40:51.860006 4939 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6" Nov 03 00:40:51.860027 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:40:51.860028 4939 status_manager.go:249] "Starting to sync pod status with apiserver" Nov 03 00:40:51.860158 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:40:51.860048 4939 kubelet.go:2521] "Starting kubelet main sync loop" Nov 03 00:40:51.860158 ip-10-0-140-107 kubenswrapper[4939]: E1103 00:40:51.860088 4939 kubelet.go:2545] "Skipping pod synchronization" err="[container runtime status check may not have completed yet, PLEG is not healthy: pleg has yet to be successful]" Nov 03 00:40:51.868513 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:40:51.868496 4939 reflector.go:446] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:161" Nov 03 00:40:51.873671 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:40:51.873641 4939 generic.go:356] "Generic (PLEG): container finished" podID="0fb35ee5f976a9435ef306be01bc3e71" containerID="c303e752f251d21c490ca1d8c26f71a79a56276d00a75842417a646b16661e86" exitCode=0 Nov 03 00:40:51.890898 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:40:51.890880 4939 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-n42s6_65d134ac-8a25-4c67-b6c7-cd752a098daf/console-operator/1.log" Nov 03 00:40:51.890992 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:40:51.890930 4939 generic.go:356] "Generic (PLEG): container finished" podID="65d134ac-8a25-4c67-b6c7-cd752a098daf" containerID="8fbbed43b6d1c11a77523d9a59cdf45ffb9e7d1661d1a2290ee4a1ddd7dc0fb7" exitCode=255 Nov 03 00:40:51.901428 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:40:51.901404 4939 generic.go:356] "Generic (PLEG): container finished" podID="6650035b-2220-491c-9683-8a03ccb7af6a" containerID="b13bf6e4a5929d974571b6a56d8155a56640694d56ac85a500cb6d953cf72f6f" exitCode=0 Nov 03 00:40:51.901428 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:40:51.901428 4939 generic.go:356] "Generic (PLEG): container finished" podID="6650035b-2220-491c-9683-8a03ccb7af6a" containerID="7c9c0e3ad6073ffe6b8d8d2a7d5bb1bb86e1732b9f514a7d6db018f593926e78" exitCode=0 Nov 03 00:40:51.901552 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:40:51.901437 4939 generic.go:356] "Generic (PLEG): container finished" podID="6650035b-2220-491c-9683-8a03ccb7af6a" containerID="21751d6aa882b867fb24aa2aed5f52dc918892c4a80cbfe5d7a2227018b3f4ad" exitCode=0 Nov 03 00:40:51.901552 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:40:51.901448 4939 generic.go:356] "Generic (PLEG): container finished" podID="6650035b-2220-491c-9683-8a03ccb7af6a" containerID="1cdfe4721f48faf8427b35d8affa3b2cb01cffb3bd2cce6fad597b03c758beae" exitCode=0 Nov 03 00:40:51.901552 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:40:51.901457 4939 generic.go:356] "Generic (PLEG): container finished" podID="6650035b-2220-491c-9683-8a03ccb7af6a" containerID="1a0f2e8768244bbda03f5da9c38e1b80b1c6fd39b37c041ab81be2bd010691d2" exitCode=0 Nov 03 00:40:51.901552 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:40:51.901464 4939 generic.go:356] "Generic (PLEG): container finished" podID="6650035b-2220-491c-9683-8a03ccb7af6a" containerID="d61acd42b9668fea1316a57002b752f2a500e975f2eca1afedafac05104db6fd" exitCode=0 Nov 03 00:40:51.926078 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:40:51.926032 4939 manager.go:324] Recovery completed Nov 03 00:40:51.945216 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:40:51.945193 4939 cpu_manager.go:226] "Starting" policy="none" Nov 03 00:40:51.945216 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:40:51.945208 4939 cpu_manager.go:227] "Reconciling" reconcilePeriod="10s" Nov 03 00:40:51.945362 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:40:51.945229 4939 state_mem.go:41] "Initialized" logger="CPUManager state checkpoint.CPUManager state memory" Nov 03 00:40:51.945412 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:40:51.945382 4939 state_mem.go:94] "Updated default CPUSet" logger="CPUManager state checkpoint.CPUManager state memory" cpuSet="" Nov 03 00:40:51.945460 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:40:51.945397 4939 state_mem.go:102] "Updated CPUSet assignments" logger="CPUManager state checkpoint.CPUManager state memory" assignments={} Nov 03 00:40:51.945460 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:40:51.945421 4939 state_checkpoint.go:142] "restored state from checkpoint" logger="CPUManager state checkpoint" Nov 03 00:40:51.945460 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:40:51.945430 4939 state_checkpoint.go:143] "defaultCPUSet" logger="CPUManager state checkpoint" defaultCpuSet="" Nov 03 00:40:51.945460 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:40:51.945438 4939 policy_none.go:50] "Start" Nov 03 00:40:51.945460 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:40:51.945450 4939 memory_manager.go:187] "Starting memorymanager" policy="None" Nov 03 00:40:51.945687 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:40:51.945463 4939 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint" Nov 03 00:40:51.945687 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:40:51.945641 4939 state_mem.go:77] "Updated machine memory state" logger="Memory Manager state checkpoint" Nov 03 00:40:51.945687 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:40:51.945653 4939 state_checkpoint.go:85] "State checkpoint: restored state from checkpoint" logger="Memory Manager state checkpoint" Nov 03 00:40:51.945687 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:40:51.945661 4939 policy_none.go:44] "Start" Nov 03 00:40:51.949863 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:40:51.949833 4939 node_container_manager_linux.go:130] "Enforcing system reserved compressible on cgroup" cgroupName="/system.slice" limits={"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"} Nov 03 00:40:51.952891 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:40:51.952877 4939 manager.go:341] "Starting Device Plugin manager" Nov 03 00:40:51.953739 ip-10-0-140-107 kubenswrapper[4939]: E1103 00:40:51.952901 4939 manager.go:525] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint" Nov 03 00:40:51.953813 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:40:51.953773 4939 server.go:87] "Starting device plugin registration server" Nov 03 00:40:51.954161 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:40:51.954141 4939 eviction_manager.go:194] "Eviction manager: starting control loop" Nov 03 00:40:51.954231 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:40:51.954165 4939 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s" Nov 03 00:40:51.954839 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:40:51.954822 4939 plugin_watcher.go:53] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry" Nov 03 00:40:51.954972 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:40:51.954939 4939 plugin_watcher.go:201] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock" Nov 03 00:40:51.955031 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:40:51.954988 4939 plugin_manager.go:119] "The desired_state_of_world populator (plugin watcher) starts" Nov 03 00:40:51.955031 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:40:51.954996 4939 plugin_manager.go:121] "Starting Kubelet Plugin Manager" Nov 03 00:40:51.955187 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:40:51.955083 4939 reconciler.go:167] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-11-03T00:40:51.954968587Z","UUID":"65b1f82e-912e-4d22-94bc-39baa9281f8f","Handler":null,"Name":"","Endpoint":""} Nov 03 00:40:51.956820 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:40:51.956800 4939 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0 Nov 03 00:40:51.956921 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:40:51.956844 4939 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock Nov 03 00:40:51.957734 ip-10-0-140-107 kubenswrapper[4939]: E1103 00:40:51.957713 4939 eviction_manager.go:272] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\"" Nov 03 00:40:51.960198 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:40:51.960160 4939 kubelet.go:2608] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-107.us-east-2.compute.internal","kube-system/kube-apiserver-proxy-ip-10-0-140-107.us-east-2.compute.internal"] Nov 03 00:40:51.960874 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:40:51.960854 4939 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-107.us-east-2.compute.internal" Nov 03 00:40:51.960972 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:40:51.960896 4939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-107.us-east-2.compute.internal" event={"ID":"0fb35ee5f976a9435ef306be01bc3e71","Type":"ContainerStarted","Data":"0099f1b1a3a4dc99c6aa41edc76e67c8425e216ae53b77b6a6b6dcabacac6cac"} Nov 03 00:40:51.960972 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:40:51.960951 4939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-107.us-east-2.compute.internal" event={"ID":"0fb35ee5f976a9435ef306be01bc3e71","Type":"ContainerDied","Data":"c303e752f251d21c490ca1d8c26f71a79a56276d00a75842417a646b16661e86"} Nov 03 00:40:51.961045 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:40:51.960971 4939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-107.us-east-2.compute.internal" event={"ID":"0fb35ee5f976a9435ef306be01bc3e71","Type":"ContainerStarted","Data":"20d31a990fda9ed8069aa304f22413ca1c459b7d9b9b0e268ac30fa12af2d633"} Nov 03 00:40:51.961104 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:40:51.961088 4939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-140-107.us-east-2.compute.internal" event={"ID":"a730b11d5a720feea75c07e6e10d3b92","Type":"ContainerStarted","Data":"808947fe85aaddb159920b9c10697b79d54be1af13b0fb84d76663d9faca7b9b"} Nov 03 00:40:51.961142 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:40:51.961119 4939 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-140-107.us-east-2.compute.internal" Nov 03 00:40:51.961219 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:40:51.961116 4939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-140-107.us-east-2.compute.internal" event={"ID":"a730b11d5a720feea75c07e6e10d3b92","Type":"ContainerStarted","Data":"a012e88fc324e4ecd1bc5f9557af5fe91bc73d451cee98e718e6ecf7cf8da48f"} Nov 03 00:40:51.975773 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:40:51.975748 4939 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Nov 03 00:40:51.975850 ip-10-0-140-107 kubenswrapper[4939]: E1103 00:40:51.975795 4939 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-apiserver-proxy-ip-10-0-140-107.us-east-2.compute.internal\" already exists" pod="kube-system/kube-apiserver-proxy-ip-10-0-140-107.us-east-2.compute.internal" Nov 03 00:40:51.976359 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:40:51.976344 4939 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Nov 03 00:40:51.976430 ip-10-0-140-107 kubenswrapper[4939]: E1103 00:40:51.976410 4939 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-rbac-proxy-crio-ip-10-0-140-107.us-east-2.compute.internal\" already exists" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-107.us-east-2.compute.internal" Nov 03 00:40:52.042918 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:40:52.042891 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/0fb35ee5f976a9435ef306be01bc3e71-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-140-107.us-east-2.compute.internal\" (UID: \"0fb35ee5f976a9435ef306be01bc3e71\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-107.us-east-2.compute.internal" Nov 03 00:40:52.043032 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:40:52.042936 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/0fb35ee5f976a9435ef306be01bc3e71-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-140-107.us-east-2.compute.internal\" (UID: \"0fb35ee5f976a9435ef306be01bc3e71\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-107.us-east-2.compute.internal" Nov 03 00:40:52.043032 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:40:52.042954 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/a730b11d5a720feea75c07e6e10d3b92-config\") pod \"kube-apiserver-proxy-ip-10-0-140-107.us-east-2.compute.internal\" (UID: \"a730b11d5a720feea75c07e6e10d3b92\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-140-107.us-east-2.compute.internal" Nov 03 00:40:52.055110 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:40:52.055092 4939 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Nov 03 00:40:52.056583 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:40:52.056563 4939 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-140-107.us-east-2.compute.internal" event="NodeHasSufficientMemory" Nov 03 00:40:52.056672 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:40:52.056597 4939 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-140-107.us-east-2.compute.internal" event="NodeHasNoDiskPressure" Nov 03 00:40:52.056672 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:40:52.056608 4939 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-140-107.us-east-2.compute.internal" event="NodeHasSufficientPID" Nov 03 00:40:52.056672 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:40:52.056648 4939 kubelet_node_status.go:77] "Attempting to register node" node="ip-10-0-140-107.us-east-2.compute.internal" Nov 03 00:40:52.071151 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:40:52.071138 4939 kubelet_node_status.go:126] "Node was previously registered" node="ip-10-0-140-107.us-east-2.compute.internal" Nov 03 00:40:52.071217 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:40:52.071190 4939 kubelet_node_status.go:80] "Successfully registered node" node="ip-10-0-140-107.us-east-2.compute.internal" Nov 03 00:40:52.143317 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:40:52.143291 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/0fb35ee5f976a9435ef306be01bc3e71-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-140-107.us-east-2.compute.internal\" (UID: \"0fb35ee5f976a9435ef306be01bc3e71\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-107.us-east-2.compute.internal" Nov 03 00:40:52.143428 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:40:52.143322 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/0fb35ee5f976a9435ef306be01bc3e71-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-140-107.us-east-2.compute.internal\" (UID: \"0fb35ee5f976a9435ef306be01bc3e71\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-107.us-east-2.compute.internal" Nov 03 00:40:52.143428 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:40:52.143341 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/a730b11d5a720feea75c07e6e10d3b92-config\") pod \"kube-apiserver-proxy-ip-10-0-140-107.us-east-2.compute.internal\" (UID: \"a730b11d5a720feea75c07e6e10d3b92\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-140-107.us-east-2.compute.internal" Nov 03 00:40:52.143428 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:40:52.143382 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/a730b11d5a720feea75c07e6e10d3b92-config\") pod \"kube-apiserver-proxy-ip-10-0-140-107.us-east-2.compute.internal\" (UID: \"a730b11d5a720feea75c07e6e10d3b92\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-140-107.us-east-2.compute.internal" Nov 03 00:40:52.143428 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:40:52.143398 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/0fb35ee5f976a9435ef306be01bc3e71-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-140-107.us-east-2.compute.internal\" (UID: \"0fb35ee5f976a9435ef306be01bc3e71\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-107.us-east-2.compute.internal" Nov 03 00:40:52.143428 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:40:52.143416 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/0fb35ee5f976a9435ef306be01bc3e71-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-140-107.us-east-2.compute.internal\" (UID: \"0fb35ee5f976a9435ef306be01bc3e71\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-107.us-east-2.compute.internal" Nov 03 00:40:52.815128 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:40:52.815050 4939 apiserver.go:52] "Watching apiserver" Nov 03 00:40:52.823805 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:40:52.823769 4939 reflector.go:446] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66" Nov 03 00:40:52.824569 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:40:52.824520 4939 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-6669dc8887-ws544","openshift-service-ca-operator/service-ca-operator-544d749c8c-7lqbv","openshift-ingress/router-default-d7798ccf5-442kt","openshift-network-diagnostics/network-check-source-6f78d59f99-z5f9s","kube-system/global-pull-secret-syncer-jv7cw","openshift-multus/multus-tb9hd","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-107.us-east-2.compute.internal","openshift-monitoring/cluster-monitoring-operator-884576c57-4bm4f","kube-system/kube-apiserver-proxy-ip-10-0-140-107.us-east-2.compute.internal","openshift-dns/node-resolver-cjj2h","openshift-multus/network-metrics-daemon-rt2hx","openshift-console-operator/console-operator-7cd56b8dbc-n42s6","kube-system/kas-connection-checker-685895c7c9-zsd49","openshift-cluster-node-tuning-operator/tuned-gtwxp","openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-w4xtw","openshift-insights/insights-runtime-extractor-sthjh","openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-khmv7","openshift-image-registry/image-registry-65f77c855d-6tnc2","kube-system/konnectivity-agent-82x8c","openshift-ingress-canary/ingress-canary-s9hbz","openshift-network-diagnostics/network-check-target-w46kd","openshift-network-console/networking-console-plugin-86c478bb88-z2qgj","openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-2ngwx","openshift-dns/dns-default-266ht","kube-system/kas-connection-checker-685895c7c9-xst6j","openshift-ovn-kubernetes/ovnkube-node-psqpn","kube-system/kas-connection-checker-685895c7c9-ntqw6","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-br8vt","openshift-network-operator/iptables-alerter-5spnk","openshift-insights/insights-operator-7c4986d449-xprcf","openshift-multus/multus-additional-cni-plugins-gqppj","openshift-image-registry/node-ca-bbk2h","openshift-kube-storage-version-migrator/migrator-66df5d496b-m7cxm"] Nov 03 00:40:52.826560 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:40:52.826538 4939 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-2ngwx" Nov 03 00:40:52.827139 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:40:52.827120 4939 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/cluster-monitoring-operator-884576c57-4bm4f" Nov 03 00:40:52.828206 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:40:52.828189 4939 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/router-default-d7798ccf5-442kt" Nov 03 00:40:52.828582 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:40:52.828563 4939 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-65f77c855d-6tnc2" Nov 03 00:40:52.829398 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:40:52.829378 4939 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-86c478bb88-z2qgj" Nov 03 00:40:52.831174 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:40:52.831155 4939 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-rt2hx" Nov 03 00:40:52.834220 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:40:52.834201 4939 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"openshift-service-ca.crt\"" Nov 03 00:40:52.834220 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:40:52.834202 4939 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"service-ca-operator-config\"" Nov 03 00:40:52.834344 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:40:52.834285 4939 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"kube-root-ca.crt\"" Nov 03 00:40:52.834859 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:40:52.834835 4939 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"operator-dockercfg-tg82s\"" Nov 03 00:40:52.835050 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:40:52.835031 4939 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"kube-root-ca.crt\"" Nov 03 00:40:52.839819 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:40:52.839794 4939 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"console-operator-config\"" Nov 03 00:40:52.839969 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:40:52.839829 4939 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\"" Nov 03 00:40:52.839969 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:40:52.839860 4939 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-diagnostics-dockercfg-7vjx2\"" Nov 03 00:40:52.839969 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:40:52.839891 4939 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\"" Nov 03 00:40:52.839969 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:40:52.839899 4939 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"openshift-service-ca.crt\"" Nov 03 00:40:52.839969 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:40:52.839932 4939 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"trusted-ca-bundle\"" Nov 03 00:40:52.839969 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:40:52.839961 4939 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"kube-root-ca.crt\"" Nov 03 00:40:52.840243 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:40:52.839974 4939 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca-operator\"/\"serving-cert\"" Nov 03 00:40:52.840243 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:40:52.839991 4939 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"service-ca-bundle\"" Nov 03 00:40:52.840243 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:40:52.840096 4939 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca-operator\"/\"service-ca-operator-dockercfg-4kqf6\"" Nov 03 00:40:52.840524 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:40:52.840502 4939 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"openshift-insights-serving-cert\"" Nov 03 00:40:52.840922 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:40:52.840887 4939 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-tls\"" Nov 03 00:40:52.841005 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:40:52.840923 4939 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"default-ingress-cert\"" Nov 03 00:40:52.841005 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:40:52.840983 4939 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\"" Nov 03 00:40:52.841005 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:40:52.840995 4939 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"networking-console-plugin-cert\"" Nov 03 00:40:52.841160 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:40:52.841037 4939 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-9v27r\"" Nov 03 00:40:52.841160 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:40:52.841062 4939 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\"" Nov 03 00:40:52.841160 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:40:52.841096 4939 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-metrics-certs-default\"" Nov 03 00:40:52.841160 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:40:52.841122 4939 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-dockercfg-rc6ml\"" Nov 03 00:40:52.841160 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:40:52.841147 4939 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\"" Nov 03 00:40:52.841160 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:40:52.841151 4939 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-stats-default\"" Nov 03 00:40:52.841425 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:40:52.841248 4939 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-dockercfg-sfmcg\"" Nov 03 00:40:52.841425 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:40:52.841351 4939 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"trusted-ca\"" Nov 03 00:40:52.841530 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:40:52.841509 4939 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-w9l2l\"" Nov 03 00:40:52.841584 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:40:52.841565 4939 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-kp9k4\"" Nov 03 00:40:52.841688 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:40:52.841668 4939 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"networking-console-plugin-dockercfg-sfsbw\"" Nov 03 00:40:52.841741 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:40:52.841716 4939 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-grbz8\"" Nov 03 00:40:52.841811 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:40:52.841679 4939 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"kube-storage-version-migrator-operator-dockercfg-clzxq\"" Nov 03 00:40:52.841811 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:40:52.841808 4939 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Nov 03 00:40:52.842007 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:40:52.841817 4939 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\"" Nov 03 00:40:52.842007 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:40:52.841827 4939 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"samples-operator-tls\"" Nov 03 00:40:52.842007 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:40:52.841857 4939 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"serving-cert\"" Nov 03 00:40:52.842007 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:40:52.841872 4939 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console-operator\"/\"serving-cert\"" Nov 03 00:40:52.842007 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:40:52.841872 4939 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\"" Nov 03 00:40:52.842007 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:40:52.841899 4939 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-d54vs\"" Nov 03 00:40:52.842007 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:40:52.841987 4939 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-6669dc8887-ws544" Nov 03 00:40:52.842390 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:40:52.842148 4939 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console-operator\"/\"console-operator-dockercfg-cl7zw\"" Nov 03 00:40:52.842390 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:40:52.842175 4939 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-97hfk\"" Nov 03 00:40:52.842390 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:40:52.842234 4939 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-4zb6t\"" Nov 03 00:40:52.842390 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:40:52.842254 4939 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-28tgg\"" Nov 03 00:40:52.842390 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:40:52.842269 4939 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-node-identity-dockercfg-229lw\"" Nov 03 00:40:52.842390 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:40:52.842304 4939 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent-dockercfg-tq982\"" Nov 03 00:40:52.842390 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:40:52.842349 4939 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-storage-operator\"/\"volume-data-source-validator-dockercfg-6hxf5\"" Nov 03 00:40:52.842390 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:40:52.842362 4939 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"cluster-samples-operator-dockercfg-qzkgg\"" Nov 03 00:40:52.842612 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:40:52.842398 4939 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-check-target-dockercfg-7trdq\"" Nov 03 00:40:52.842846 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:40:52.842830 4939 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-266ht" Nov 03 00:40:52.843587 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:40:52.843567 4939 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"global-pull-secret-syncer-dockercfg-vmfsq\"" Nov 03 00:40:52.844245 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:40:52.844231 4939 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-sthjh" Nov 03 00:40:52.844375 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:40:52.844360 4939 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Nov 03 00:40:52.847013 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:40:52.846998 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wzln7\" (UniqueName: \"kubernetes.io/projected/2b050b18-31f6-4a22-9902-0435d11018b9-kube-api-access-wzln7\") pod \"service-ca-operator-544d749c8c-7lqbv\" (UID: \"2b050b18-31f6-4a22-9902-0435d11018b9\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-7lqbv" Nov 03 00:40:52.847059 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:40:52.847024 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cf217a21-c1bd-48be-b203-f59743bceb33-metrics-certs\") pod \"router-default-d7798ccf5-442kt\" (UID: \"cf217a21-c1bd-48be-b203-f59743bceb33\") " pod="openshift-ingress/router-default-d7798ccf5-442kt" Nov 03 00:40:52.847059 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:40:52.847043 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/98ca3972-fd0c-47f5-9d26-569a58e18bd1-host-run-netns\") pod \"ovnkube-node-psqpn\" (UID: \"98ca3972-fd0c-47f5-9d26-569a58e18bd1\") " pod="openshift-ovn-kubernetes/ovnkube-node-psqpn" Nov 03 00:40:52.847115 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:40:52.847066 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/98ca3972-fd0c-47f5-9d26-569a58e18bd1-etc-openvswitch\") pod \"ovnkube-node-psqpn\" (UID: \"98ca3972-fd0c-47f5-9d26-569a58e18bd1\") " pod="openshift-ovn-kubernetes/ovnkube-node-psqpn" Nov 03 00:40:52.847189 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:40:52.847174 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/dd0cd8a1-4d6b-424c-ac39-5f5ddb9cd87e-bound-sa-token\") pod \"image-registry-65f77c855d-6tnc2\" (UID: \"dd0cd8a1-4d6b-424c-ac39-5f5ddb9cd87e\") " pod="openshift-image-registry/image-registry-65f77c855d-6tnc2" Nov 03 00:40:52.847234 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:40:52.847223 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/b748c4f8-d021-4d37-b390-d395482fcddd-host\") pod \"tuned-gtwxp\" (UID: \"b748c4f8-d021-4d37-b390-d395482fcddd\") " pod="openshift-cluster-node-tuning-operator/tuned-gtwxp" Nov 03 00:40:52.847316 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:40:52.847300 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ca0f883c-c2b7-4bf6-a90b-ffdbeac161a8-trusted-ca-bundle\") pod \"insights-operator-7c4986d449-xprcf\" (UID: \"ca0f883c-c2b7-4bf6-a90b-ffdbeac161a8\") " pod="openshift-insights/insights-operator-7c4986d449-xprcf" Nov 03 00:40:52.847346 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:40:52.847337 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-79khw\" (UniqueName: \"kubernetes.io/projected/36b9d258-cb2c-4d2b-880f-ae93b065034c-kube-api-access-79khw\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-khmv7\" (UID: \"36b9d258-cb2c-4d2b-880f-ae93b065034c\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-khmv7" Nov 03 00:40:52.847378 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:40:52.847357 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/cf217a21-c1bd-48be-b203-f59743bceb33-default-certificate\") pod \"router-default-d7798ccf5-442kt\" (UID: \"cf217a21-c1bd-48be-b203-f59743bceb33\") " pod="openshift-ingress/router-default-d7798ccf5-442kt" Nov 03 00:40:52.847415 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:40:52.847396 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-llh2g\" (UniqueName: \"kubernetes.io/projected/136086c0-4846-437a-ad98-3cfef50610a0-kube-api-access-llh2g\") pod \"network-metrics-daemon-rt2hx\" (UID: \"136086c0-4846-437a-ad98-3cfef50610a0\") " pod="openshift-multus/network-metrics-daemon-rt2hx" Nov 03 00:40:52.847445 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:40:52.847432 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/98ca3972-fd0c-47f5-9d26-569a58e18bd1-host-cni-netd\") pod \"ovnkube-node-psqpn\" (UID: \"98ca3972-fd0c-47f5-9d26-569a58e18bd1\") " pod="openshift-ovn-kubernetes/ovnkube-node-psqpn" Nov 03 00:40:52.847487 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:40:52.847467 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/98ca3972-fd0c-47f5-9d26-569a58e18bd1-ovn-node-metrics-cert\") pod \"ovnkube-node-psqpn\" (UID: \"98ca3972-fd0c-47f5-9d26-569a58e18bd1\") " pod="openshift-ovn-kubernetes/ovnkube-node-psqpn" Nov 03 00:40:52.847529 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:40:52.847511 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q4b9t\" (UniqueName: \"kubernetes.io/projected/460779b2-3903-4b23-9b4d-e25142782a21-kube-api-access-q4b9t\") pod \"node-ca-bbk2h\" (UID: \"460779b2-3903-4b23-9b4d-e25142782a21\") " pod="openshift-image-registry/node-ca-bbk2h" Nov 03 00:40:52.847593 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:40:52.847571 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/6650035b-2220-491c-9683-8a03ccb7af6a-tuning-conf-dir\") pod \"multus-additional-cni-plugins-gqppj\" (UID: \"6650035b-2220-491c-9683-8a03ccb7af6a\") " pod="openshift-multus/multus-additional-cni-plugins-gqppj" Nov 03 00:40:52.847649 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:40:52.847616 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/b748c4f8-d021-4d37-b390-d395482fcddd-lib-modules\") pod \"tuned-gtwxp\" (UID: \"b748c4f8-d021-4d37-b390-d395482fcddd\") " pod="openshift-cluster-node-tuning-operator/tuned-gtwxp" Nov 03 00:40:52.847697 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:40:52.847651 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/2b050b18-31f6-4a22-9902-0435d11018b9-config\") pod \"service-ca-operator-544d749c8c-7lqbv\" (UID: \"2b050b18-31f6-4a22-9902-0435d11018b9\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-7lqbv" Nov 03 00:40:52.847697 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:40:52.847681 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nswxk\" (UniqueName: \"kubernetes.io/projected/4237805a-d475-4bd7-865c-c6ef371c0e11-kube-api-access-nswxk\") pod \"kas-connection-checker-685895c7c9-zsd49\" (UID: \"4237805a-d475-4bd7-865c-c6ef371c0e11\") " pod="kube-system/kas-connection-checker-685895c7c9-zsd49" Nov 03 00:40:52.847787 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:40:52.847711 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/98ca3972-fd0c-47f5-9d26-569a58e18bd1-systemd-units\") pod \"ovnkube-node-psqpn\" (UID: \"98ca3972-fd0c-47f5-9d26-569a58e18bd1\") " pod="openshift-ovn-kubernetes/ovnkube-node-psqpn" Nov 03 00:40:52.847787 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:40:52.847742 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/b748c4f8-d021-4d37-b390-d395482fcddd-etc-kubernetes\") pod \"tuned-gtwxp\" (UID: \"b748c4f8-d021-4d37-b390-d395482fcddd\") " pod="openshift-cluster-node-tuning-operator/tuned-gtwxp" Nov 03 00:40:52.847882 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:40:52.847783 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/cf217a21-c1bd-48be-b203-f59743bceb33-default-certificate\") pod \"router-default-d7798ccf5-442kt\" (UID: \"cf217a21-c1bd-48be-b203-f59743bceb33\") " pod="openshift-ingress/router-default-d7798ccf5-442kt" Nov 03 00:40:52.847882 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:40:52.847850 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/98ca3972-fd0c-47f5-9d26-569a58e18bd1-ovn-node-metrics-cert\") pod \"ovnkube-node-psqpn\" (UID: \"98ca3972-fd0c-47f5-9d26-569a58e18bd1\") " pod="openshift-ovn-kubernetes/ovnkube-node-psqpn" Nov 03 00:40:52.847982 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:40:52.847939 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/6650035b-2220-491c-9683-8a03ccb7af6a-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-gqppj\" (UID: \"6650035b-2220-491c-9683-8a03ccb7af6a\") " pod="openshift-multus/multus-additional-cni-plugins-gqppj" Nov 03 00:40:52.848021 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:40:52.847995 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/b748c4f8-d021-4d37-b390-d395482fcddd-sys\") pod \"tuned-gtwxp\" (UID: \"b748c4f8-d021-4d37-b390-d395482fcddd\") " pod="openshift-cluster-node-tuning-operator/tuned-gtwxp" Nov 03 00:40:52.848052 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:40:52.848025 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/98ca3972-fd0c-47f5-9d26-569a58e18bd1-ovnkube-config\") pod \"ovnkube-node-psqpn\" (UID: \"98ca3972-fd0c-47f5-9d26-569a58e18bd1\") " pod="openshift-ovn-kubernetes/ovnkube-node-psqpn" Nov 03 00:40:52.848052 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:40:52.848042 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4bhqh\" (UniqueName: \"kubernetes.io/projected/98ca3972-fd0c-47f5-9d26-569a58e18bd1-kube-api-access-4bhqh\") pod \"ovnkube-node-psqpn\" (UID: \"98ca3972-fd0c-47f5-9d26-569a58e18bd1\") " pod="openshift-ovn-kubernetes/ovnkube-node-psqpn" Nov 03 00:40:52.848119 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:40:52.848018 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/2b050b18-31f6-4a22-9902-0435d11018b9-config\") pod \"service-ca-operator-544d749c8c-7lqbv\" (UID: \"2b050b18-31f6-4a22-9902-0435d11018b9\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-7lqbv" Nov 03 00:40:52.848119 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:40:52.848072 4939 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-7mmgk\"" Nov 03 00:40:52.848119 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:40:52.848072 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/c3ffbe53-7f23-4750-b95d-476e6cfc58a6-konnectivity-ca\") pod \"konnectivity-agent-82x8c\" (UID: \"c3ffbe53-7f23-4750-b95d-476e6cfc58a6\") " pod="kube-system/konnectivity-agent-82x8c" Nov 03 00:40:52.848208 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:40:52.848145 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/b748c4f8-d021-4d37-b390-d395482fcddd-etc-sysctl-conf\") pod \"tuned-gtwxp\" (UID: \"b748c4f8-d021-4d37-b390-d395482fcddd\") " pod="openshift-cluster-node-tuning-operator/tuned-gtwxp" Nov 03 00:40:52.848208 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:40:52.848180 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pdmr4\" (UniqueName: \"kubernetes.io/projected/29a64a08-67ef-4c82-8a82-8cd4a4675327-kube-api-access-pdmr4\") pod \"kas-connection-checker-685895c7c9-xst6j\" (UID: \"29a64a08-67ef-4c82-8a82-8cd4a4675327\") " pod="kube-system/kas-connection-checker-685895c7c9-xst6j" Nov 03 00:40:52.848266 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:40:52.848206 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/c736fd3f-973f-4999-8d4e-c405079b2888-host-slash\") pod \"iptables-alerter-5spnk\" (UID: \"c736fd3f-973f-4999-8d4e-c405079b2888\") " pod="openshift-network-operator/iptables-alerter-5spnk" Nov 03 00:40:52.848399 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:40:52.848375 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/cf217a21-c1bd-48be-b203-f59743bceb33-stats-auth\") pod \"router-default-d7798ccf5-442kt\" (UID: \"cf217a21-c1bd-48be-b203-f59743bceb33\") " pod="openshift-ingress/router-default-d7798ccf5-442kt" Nov 03 00:40:52.848440 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:40:52.848424 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fdrwp\" (UniqueName: \"kubernetes.io/projected/cf217a21-c1bd-48be-b203-f59743bceb33-kube-api-access-fdrwp\") pod \"router-default-d7798ccf5-442kt\" (UID: \"cf217a21-c1bd-48be-b203-f59743bceb33\") " pod="openshift-ingress/router-default-d7798ccf5-442kt" Nov 03 00:40:52.848482 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:40:52.848465 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/98ca3972-fd0c-47f5-9d26-569a58e18bd1-env-overrides\") pod \"ovnkube-node-psqpn\" (UID: \"98ca3972-fd0c-47f5-9d26-569a58e18bd1\") " pod="openshift-ovn-kubernetes/ovnkube-node-psqpn" Nov 03 00:40:52.848526 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:40:52.848493 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/dd0cd8a1-4d6b-424c-ac39-5f5ddb9cd87e-registry-tls\") pod \"image-registry-65f77c855d-6tnc2\" (UID: \"dd0cd8a1-4d6b-424c-ac39-5f5ddb9cd87e\") " pod="openshift-image-registry/image-registry-65f77c855d-6tnc2" Nov 03 00:40:52.848526 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:40:52.848515 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/b748c4f8-d021-4d37-b390-d395482fcddd-etc-sysctl-d\") pod \"tuned-gtwxp\" (UID: \"b748c4f8-d021-4d37-b390-d395482fcddd\") " pod="openshift-cluster-node-tuning-operator/tuned-gtwxp" Nov 03 00:40:52.848602 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:40:52.848515 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ca0f883c-c2b7-4bf6-a90b-ffdbeac161a8-trusted-ca-bundle\") pod \"insights-operator-7c4986d449-xprcf\" (UID: \"ca0f883c-c2b7-4bf6-a90b-ffdbeac161a8\") " pod="openshift-insights/insights-operator-7c4986d449-xprcf" Nov 03 00:40:52.848602 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:40:52.848541 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/b748c4f8-d021-4d37-b390-d395482fcddd-etc-tuned\") pod \"tuned-gtwxp\" (UID: \"b748c4f8-d021-4d37-b390-d395482fcddd\") " pod="openshift-cluster-node-tuning-operator/tuned-gtwxp" Nov 03 00:40:52.848659 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:40:52.848596 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/ca0f883c-c2b7-4bf6-a90b-ffdbeac161a8-serving-cert\") pod \"insights-operator-7c4986d449-xprcf\" (UID: \"ca0f883c-c2b7-4bf6-a90b-ffdbeac161a8\") " pod="openshift-insights/insights-operator-7c4986d449-xprcf" Nov 03 00:40:52.848659 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:40:52.848630 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/cf217a21-c1bd-48be-b203-f59743bceb33-stats-auth\") pod \"router-default-d7798ccf5-442kt\" (UID: \"cf217a21-c1bd-48be-b203-f59743bceb33\") " pod="openshift-ingress/router-default-d7798ccf5-442kt" Nov 03 00:40:52.848659 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:40:52.848611 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/b748c4f8-d021-4d37-b390-d395482fcddd-etc-tuned\") pod \"tuned-gtwxp\" (UID: \"b748c4f8-d021-4d37-b390-d395482fcddd\") " pod="openshift-cluster-node-tuning-operator/tuned-gtwxp" Nov 03 00:40:52.848742 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:40:52.848648 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/98ca3972-fd0c-47f5-9d26-569a58e18bd1-host-cni-bin\") pod \"ovnkube-node-psqpn\" (UID: \"98ca3972-fd0c-47f5-9d26-569a58e18bd1\") " pod="openshift-ovn-kubernetes/ovnkube-node-psqpn" Nov 03 00:40:52.848742 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:40:52.848683 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/dd0cd8a1-4d6b-424c-ac39-5f5ddb9cd87e-image-registry-private-configuration\") pod \"image-registry-65f77c855d-6tnc2\" (UID: \"dd0cd8a1-4d6b-424c-ac39-5f5ddb9cd87e\") " pod="openshift-image-registry/image-registry-65f77c855d-6tnc2" Nov 03 00:40:52.848814 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:40:52.848747 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/dd0cd8a1-4d6b-424c-ac39-5f5ddb9cd87e-ca-trust-extracted\") pod \"image-registry-65f77c855d-6tnc2\" (UID: \"dd0cd8a1-4d6b-424c-ac39-5f5ddb9cd87e\") " pod="openshift-image-registry/image-registry-65f77c855d-6tnc2" Nov 03 00:40:52.848814 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:40:52.848802 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/dd0cd8a1-4d6b-424c-ac39-5f5ddb9cd87e-ca-trust-extracted\") pod \"image-registry-65f77c855d-6tnc2\" (UID: \"dd0cd8a1-4d6b-424c-ac39-5f5ddb9cd87e\") " pod="openshift-image-registry/image-registry-65f77c855d-6tnc2" Nov 03 00:40:52.848814 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:40:52.848811 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/dd0cd8a1-4d6b-424c-ac39-5f5ddb9cd87e-image-registry-private-configuration\") pod \"image-registry-65f77c855d-6tnc2\" (UID: \"dd0cd8a1-4d6b-424c-ac39-5f5ddb9cd87e\") " pod="openshift-image-registry/image-registry-65f77c855d-6tnc2" Nov 03 00:40:52.848974 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:40:52.848835 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/dd0cd8a1-4d6b-424c-ac39-5f5ddb9cd87e-trusted-ca\") pod \"image-registry-65f77c855d-6tnc2\" (UID: \"dd0cd8a1-4d6b-424c-ac39-5f5ddb9cd87e\") " pod="openshift-image-registry/image-registry-65f77c855d-6tnc2" Nov 03 00:40:52.848974 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:40:52.848870 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/ca0f883c-c2b7-4bf6-a90b-ffdbeac161a8-snapshots\") pod \"insights-operator-7c4986d449-xprcf\" (UID: \"ca0f883c-c2b7-4bf6-a90b-ffdbeac161a8\") " pod="openshift-insights/insights-operator-7c4986d449-xprcf" Nov 03 00:40:52.848974 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:40:52.848900 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/6eafe6f6-cf90-446f-a8d5-620cd2e02775-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-2ngwx\" (UID: \"6eafe6f6-cf90-446f-a8d5-620cd2e02775\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-2ngwx" Nov 03 00:40:52.848974 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:40:52.848949 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-config\" (UniqueName: \"kubernetes.io/configmap/6eafe6f6-cf90-446f-a8d5-620cd2e02775-samples-operator-config\") pod \"cluster-samples-operator-6ddb6c8b94-2ngwx\" (UID: \"6eafe6f6-cf90-446f-a8d5-620cd2e02775\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-2ngwx" Nov 03 00:40:52.848974 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:40:52.848961 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/ca0f883c-c2b7-4bf6-a90b-ffdbeac161a8-snapshots\") pod \"insights-operator-7c4986d449-xprcf\" (UID: \"ca0f883c-c2b7-4bf6-a90b-ffdbeac161a8\") " pod="openshift-insights/insights-operator-7c4986d449-xprcf" Nov 03 00:40:52.849216 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:40:52.848964 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/ca0f883c-c2b7-4bf6-a90b-ffdbeac161a8-serving-cert\") pod \"insights-operator-7c4986d449-xprcf\" (UID: \"ca0f883c-c2b7-4bf6-a90b-ffdbeac161a8\") " pod="openshift-insights/insights-operator-7c4986d449-xprcf" Nov 03 00:40:52.849216 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:40:52.848978 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/98ca3972-fd0c-47f5-9d26-569a58e18bd1-run-openvswitch\") pod \"ovnkube-node-psqpn\" (UID: \"98ca3972-fd0c-47f5-9d26-569a58e18bd1\") " pod="openshift-ovn-kubernetes/ovnkube-node-psqpn" Nov 03 00:40:52.849216 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:40:52.849003 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/98ca3972-fd0c-47f5-9d26-569a58e18bd1-node-log\") pod \"ovnkube-node-psqpn\" (UID: \"98ca3972-fd0c-47f5-9d26-569a58e18bd1\") " pod="openshift-ovn-kubernetes/ovnkube-node-psqpn" Nov 03 00:40:52.849216 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:40:52.849029 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h58rt\" (UniqueName: \"kubernetes.io/projected/56261beb-517f-4c88-b59a-42ab41e278fd-kube-api-access-h58rt\") pod \"volume-data-source-validator-58c9f5f46d-w4xtw\" (UID: \"56261beb-517f-4c88-b59a-42ab41e278fd\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-w4xtw" Nov 03 00:40:52.849216 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:40:52.849054 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rkf7s\" (UniqueName: \"kubernetes.io/projected/6650035b-2220-491c-9683-8a03ccb7af6a-kube-api-access-rkf7s\") pod \"multus-additional-cni-plugins-gqppj\" (UID: \"6650035b-2220-491c-9683-8a03ccb7af6a\") " pod="openshift-multus/multus-additional-cni-plugins-gqppj" Nov 03 00:40:52.849216 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:40:52.849086 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/06973556-f51c-4d1c-ae55-fb77699bc5b3-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-4bm4f\" (UID: \"06973556-f51c-4d1c-ae55-fb77699bc5b3\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-4bm4f" Nov 03 00:40:52.849216 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:40:52.849114 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/98ca3972-fd0c-47f5-9d26-569a58e18bd1-host-slash\") pod \"ovnkube-node-psqpn\" (UID: \"98ca3972-fd0c-47f5-9d26-569a58e18bd1\") " pod="openshift-ovn-kubernetes/ovnkube-node-psqpn" Nov 03 00:40:52.849216 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:40:52.849134 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/98ca3972-fd0c-47f5-9d26-569a58e18bd1-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-psqpn\" (UID: \"98ca3972-fd0c-47f5-9d26-569a58e18bd1\") " pod="openshift-ovn-kubernetes/ovnkube-node-psqpn" Nov 03 00:40:52.849216 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:40:52.849191 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jbfhc\" (UniqueName: \"kubernetes.io/projected/dd0cd8a1-4d6b-424c-ac39-5f5ddb9cd87e-kube-api-access-jbfhc\") pod \"image-registry-65f77c855d-6tnc2\" (UID: \"dd0cd8a1-4d6b-424c-ac39-5f5ddb9cd87e\") " pod="openshift-image-registry/image-registry-65f77c855d-6tnc2" Nov 03 00:40:52.849216 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:40:52.849217 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/bc513cba-d924-4719-a2ba-711529175ee0-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-z2qgj\" (UID: \"bc513cba-d924-4719-a2ba-711529175ee0\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-z2qgj" Nov 03 00:40:52.849532 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:40:52.849239 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/c736fd3f-973f-4999-8d4e-c405079b2888-iptables-alerter-script\") pod \"iptables-alerter-5spnk\" (UID: \"c736fd3f-973f-4999-8d4e-c405079b2888\") " pod="openshift-network-operator/iptables-alerter-5spnk" Nov 03 00:40:52.849532 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:40:52.849461 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/65d134ac-8a25-4c67-b6c7-cd752a098daf-config\") pod \"console-operator-7cd56b8dbc-n42s6\" (UID: \"65d134ac-8a25-4c67-b6c7-cd752a098daf\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-n42s6" Nov 03 00:40:52.849532 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:40:52.849497 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/65d134ac-8a25-4c67-b6c7-cd752a098daf-config\") pod \"console-operator-7cd56b8dbc-n42s6\" (UID: \"65d134ac-8a25-4c67-b6c7-cd752a098daf\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-n42s6" Nov 03 00:40:52.849532 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:40:52.849506 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d8l25\" (UniqueName: \"kubernetes.io/projected/8cd3ca22-e4ce-4896-ae31-97605af4e4c6-kube-api-access-d8l25\") pod \"network-check-target-w46kd\" (UID: \"8cd3ca22-e4ce-4896-ae31-97605af4e4c6\") " pod="openshift-network-diagnostics/network-check-target-w46kd" Nov 03 00:40:52.849710 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:40:52.849534 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/dd0cd8a1-4d6b-424c-ac39-5f5ddb9cd87e-installation-pull-secrets\") pod \"image-registry-65f77c855d-6tnc2\" (UID: \"dd0cd8a1-4d6b-424c-ac39-5f5ddb9cd87e\") " pod="openshift-image-registry/image-registry-65f77c855d-6tnc2" Nov 03 00:40:52.849710 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:40:52.849551 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/460779b2-3903-4b23-9b4d-e25142782a21-host\") pod \"node-ca-bbk2h\" (UID: \"460779b2-3903-4b23-9b4d-e25142782a21\") " pod="openshift-image-registry/node-ca-bbk2h" Nov 03 00:40:52.849710 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:40:52.849566 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/b748c4f8-d021-4d37-b390-d395482fcddd-var-lib-kubelet\") pod \"tuned-gtwxp\" (UID: \"b748c4f8-d021-4d37-b390-d395482fcddd\") " pod="openshift-cluster-node-tuning-operator/tuned-gtwxp" Nov 03 00:40:52.849710 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:40:52.849587 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/b748c4f8-d021-4d37-b390-d395482fcddd-tmp\") pod \"tuned-gtwxp\" (UID: \"b748c4f8-d021-4d37-b390-d395482fcddd\") " pod="openshift-cluster-node-tuning-operator/tuned-gtwxp" Nov 03 00:40:52.849710 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:40:52.849618 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/65d134ac-8a25-4c67-b6c7-cd752a098daf-serving-cert\") pod \"console-operator-7cd56b8dbc-n42s6\" (UID: \"65d134ac-8a25-4c67-b6c7-cd752a098daf\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-n42s6" Nov 03 00:40:52.849946 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:40:52.849715 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/b748c4f8-d021-4d37-b390-d395482fcddd-tmp\") pod \"tuned-gtwxp\" (UID: \"b748c4f8-d021-4d37-b390-d395482fcddd\") " pod="openshift-cluster-node-tuning-operator/tuned-gtwxp" Nov 03 00:40:52.849946 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:40:52.849731 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/6650035b-2220-491c-9683-8a03ccb7af6a-system-cni-dir\") pod \"multus-additional-cni-plugins-gqppj\" (UID: \"6650035b-2220-491c-9683-8a03ccb7af6a\") " pod="openshift-multus/multus-additional-cni-plugins-gqppj" Nov 03 00:40:52.849946 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:40:52.849752 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/b748c4f8-d021-4d37-b390-d395482fcddd-etc-systemd\") pod \"tuned-gtwxp\" (UID: \"b748c4f8-d021-4d37-b390-d395482fcddd\") " pod="openshift-cluster-node-tuning-operator/tuned-gtwxp" Nov 03 00:40:52.849946 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:40:52.849755 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/dd0cd8a1-4d6b-424c-ac39-5f5ddb9cd87e-installation-pull-secrets\") pod \"image-registry-65f77c855d-6tnc2\" (UID: \"dd0cd8a1-4d6b-424c-ac39-5f5ddb9cd87e\") " pod="openshift-image-registry/image-registry-65f77c855d-6tnc2" Nov 03 00:40:52.849946 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:40:52.849766 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/b748c4f8-d021-4d37-b390-d395482fcddd-run\") pod \"tuned-gtwxp\" (UID: \"b748c4f8-d021-4d37-b390-d395482fcddd\") " pod="openshift-cluster-node-tuning-operator/tuned-gtwxp" Nov 03 00:40:52.849946 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:40:52.849783 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6kcsm\" (UniqueName: \"kubernetes.io/projected/6eafe6f6-cf90-446f-a8d5-620cd2e02775-kube-api-access-6kcsm\") pod \"cluster-samples-operator-6ddb6c8b94-2ngwx\" (UID: \"6eafe6f6-cf90-446f-a8d5-620cd2e02775\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-2ngwx" Nov 03 00:40:52.849946 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:40:52.849792 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/65d134ac-8a25-4c67-b6c7-cd752a098daf-serving-cert\") pod \"console-operator-7cd56b8dbc-n42s6\" (UID: \"65d134ac-8a25-4c67-b6c7-cd752a098daf\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-n42s6" Nov 03 00:40:52.849946 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:40:52.849863 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/98ca3972-fd0c-47f5-9d26-569a58e18bd1-log-socket\") pod \"ovnkube-node-psqpn\" (UID: \"98ca3972-fd0c-47f5-9d26-569a58e18bd1\") " pod="openshift-ovn-kubernetes/ovnkube-node-psqpn" Nov 03 00:40:52.850307 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:40:52.849958 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/98ca3972-fd0c-47f5-9d26-569a58e18bd1-ovnkube-script-lib\") pod \"ovnkube-node-psqpn\" (UID: \"98ca3972-fd0c-47f5-9d26-569a58e18bd1\") " pod="openshift-ovn-kubernetes/ovnkube-node-psqpn" Nov 03 00:40:52.850307 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:40:52.850003 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/dd0cd8a1-4d6b-424c-ac39-5f5ddb9cd87e-registry-certificates\") pod \"image-registry-65f77c855d-6tnc2\" (UID: \"dd0cd8a1-4d6b-424c-ac39-5f5ddb9cd87e\") " pod="openshift-image-registry/image-registry-65f77c855d-6tnc2" Nov 03 00:40:52.850307 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:40:52.850032 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/c3ffbe53-7f23-4750-b95d-476e6cfc58a6-agent-certs\") pod \"konnectivity-agent-82x8c\" (UID: \"c3ffbe53-7f23-4750-b95d-476e6cfc58a6\") " pod="kube-system/konnectivity-agent-82x8c" Nov 03 00:40:52.850307 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:40:52.850051 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hsvqw\" (UniqueName: \"kubernetes.io/projected/b748c4f8-d021-4d37-b390-d395482fcddd-kube-api-access-hsvqw\") pod \"tuned-gtwxp\" (UID: \"b748c4f8-d021-4d37-b390-d395482fcddd\") " pod="openshift-cluster-node-tuning-operator/tuned-gtwxp" Nov 03 00:40:52.850307 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:40:52.850075 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ca0f883c-c2b7-4bf6-a90b-ffdbeac161a8-service-ca-bundle\") pod \"insights-operator-7c4986d449-xprcf\" (UID: \"ca0f883c-c2b7-4bf6-a90b-ffdbeac161a8\") " pod="openshift-insights/insights-operator-7c4986d449-xprcf" Nov 03 00:40:52.850307 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:40:52.850100 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/36b9d258-cb2c-4d2b-880f-ae93b065034c-serving-cert\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-khmv7\" (UID: \"36b9d258-cb2c-4d2b-880f-ae93b065034c\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-khmv7" Nov 03 00:40:52.850307 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:40:52.850137 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2tdmf\" (UniqueName: \"kubernetes.io/projected/c736fd3f-973f-4999-8d4e-c405079b2888-kube-api-access-2tdmf\") pod \"iptables-alerter-5spnk\" (UID: \"c736fd3f-973f-4999-8d4e-c405079b2888\") " pod="openshift-network-operator/iptables-alerter-5spnk" Nov 03 00:40:52.850307 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:40:52.850175 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7zxdt\" (UniqueName: \"kubernetes.io/projected/b0bd4bd5-12ca-49df-a3d9-4243f4bd642e-kube-api-access-7zxdt\") pod \"kas-connection-checker-685895c7c9-ntqw6\" (UID: \"b0bd4bd5-12ca-49df-a3d9-4243f4bd642e\") " pod="kube-system/kas-connection-checker-685895c7c9-ntqw6" Nov 03 00:40:52.850307 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:40:52.850217 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/cf217a21-c1bd-48be-b203-f59743bceb33-service-ca-bundle\") pod \"router-default-d7798ccf5-442kt\" (UID: \"cf217a21-c1bd-48be-b203-f59743bceb33\") " pod="openshift-ingress/router-default-d7798ccf5-442kt" Nov 03 00:40:52.850307 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:40:52.850220 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ca0f883c-c2b7-4bf6-a90b-ffdbeac161a8-service-ca-bundle\") pod \"insights-operator-7c4986d449-xprcf\" (UID: \"ca0f883c-c2b7-4bf6-a90b-ffdbeac161a8\") " pod="openshift-insights/insights-operator-7c4986d449-xprcf" Nov 03 00:40:52.850307 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:40:52.850266 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/06973556-f51c-4d1c-ae55-fb77699bc5b3-telemetry-config\") pod \"cluster-monitoring-operator-884576c57-4bm4f\" (UID: \"06973556-f51c-4d1c-ae55-fb77699bc5b3\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-4bm4f" Nov 03 00:40:52.850732 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:40:52.850326 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/36b9d258-cb2c-4d2b-880f-ae93b065034c-serving-cert\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-khmv7\" (UID: \"36b9d258-cb2c-4d2b-880f-ae93b065034c\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-khmv7" Nov 03 00:40:52.850732 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:40:52.850405 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4znwf\" (UniqueName: \"kubernetes.io/projected/06973556-f51c-4d1c-ae55-fb77699bc5b3-kube-api-access-4znwf\") pod \"cluster-monitoring-operator-884576c57-4bm4f\" (UID: \"06973556-f51c-4d1c-ae55-fb77699bc5b3\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-4bm4f" Nov 03 00:40:52.850732 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:40:52.850414 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/c3ffbe53-7f23-4750-b95d-476e6cfc58a6-agent-certs\") pod \"konnectivity-agent-82x8c\" (UID: \"c3ffbe53-7f23-4750-b95d-476e6cfc58a6\") " pod="kube-system/konnectivity-agent-82x8c" Nov 03 00:40:52.850732 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:40:52.850424 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/65d134ac-8a25-4c67-b6c7-cd752a098daf-trusted-ca\") pod \"console-operator-7cd56b8dbc-n42s6\" (UID: \"65d134ac-8a25-4c67-b6c7-cd752a098daf\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-n42s6" Nov 03 00:40:52.850732 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:40:52.850467 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/98ca3972-fd0c-47f5-9d26-569a58e18bd1-run-ovn\") pod \"ovnkube-node-psqpn\" (UID: \"98ca3972-fd0c-47f5-9d26-569a58e18bd1\") " pod="openshift-ovn-kubernetes/ovnkube-node-psqpn" Nov 03 00:40:52.850732 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:40:52.850499 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/b748c4f8-d021-4d37-b390-d395482fcddd-etc-modprobe-d\") pod \"tuned-gtwxp\" (UID: \"b748c4f8-d021-4d37-b390-d395482fcddd\") " pod="openshift-cluster-node-tuning-operator/tuned-gtwxp" Nov 03 00:40:52.850953 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:40:52.850749 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/2b050b18-31f6-4a22-9902-0435d11018b9-serving-cert\") pod \"service-ca-operator-544d749c8c-7lqbv\" (UID: \"2b050b18-31f6-4a22-9902-0435d11018b9\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-7lqbv" Nov 03 00:40:52.850953 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:40:52.850826 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/98ca3972-fd0c-47f5-9d26-569a58e18bd1-host-kubelet\") pod \"ovnkube-node-psqpn\" (UID: \"98ca3972-fd0c-47f5-9d26-569a58e18bd1\") " pod="openshift-ovn-kubernetes/ovnkube-node-psqpn" Nov 03 00:40:52.850953 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:40:52.850828 4939 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\"" Nov 03 00:40:52.850953 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:40:52.850841 4939 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\"" Nov 03 00:40:52.850953 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:40:52.850877 4939 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\"" Nov 03 00:40:52.850953 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:40:52.850902 4939 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\"" Nov 03 00:40:52.851220 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:40:52.850952 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/98ca3972-fd0c-47f5-9d26-569a58e18bd1-var-lib-openvswitch\") pod \"ovnkube-node-psqpn\" (UID: \"98ca3972-fd0c-47f5-9d26-569a58e18bd1\") " pod="openshift-ovn-kubernetes/ovnkube-node-psqpn" Nov 03 00:40:52.851220 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:40:52.850994 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/98ca3972-fd0c-47f5-9d26-569a58e18bd1-host-run-ovn-kubernetes\") pod \"ovnkube-node-psqpn\" (UID: \"98ca3972-fd0c-47f5-9d26-569a58e18bd1\") " pod="openshift-ovn-kubernetes/ovnkube-node-psqpn" Nov 03 00:40:52.851220 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:40:52.851053 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/6650035b-2220-491c-9683-8a03ccb7af6a-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-gqppj\" (UID: \"6650035b-2220-491c-9683-8a03ccb7af6a\") " pod="openshift-multus/multus-additional-cni-plugins-gqppj" Nov 03 00:40:52.851220 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:40:52.851082 4939 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-vz52v\"" Nov 03 00:40:52.851220 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:40:52.851094 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qcvwz\" (UniqueName: \"kubernetes.io/projected/ca0f883c-c2b7-4bf6-a90b-ffdbeac161a8-kube-api-access-qcvwz\") pod \"insights-operator-7c4986d449-xprcf\" (UID: \"ca0f883c-c2b7-4bf6-a90b-ffdbeac161a8\") " pod="openshift-insights/insights-operator-7c4986d449-xprcf" Nov 03 00:40:52.851220 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:40:52.851118 4939 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"ingress-canary-dockercfg-k7bjp\"" Nov 03 00:40:52.851220 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:40:52.851133 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8j6kr\" (UniqueName: \"kubernetes.io/projected/65d134ac-8a25-4c67-b6c7-cd752a098daf-kube-api-access-8j6kr\") pod \"console-operator-7cd56b8dbc-n42s6\" (UID: \"65d134ac-8a25-4c67-b6c7-cd752a098daf\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-n42s6" Nov 03 00:40:52.851220 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:40:52.851158 4939 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-42dxl\"" Nov 03 00:40:52.851220 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:40:52.851166 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/136086c0-4846-437a-ad98-3cfef50610a0-metrics-certs\") pod \"network-metrics-daemon-rt2hx\" (UID: \"136086c0-4846-437a-ad98-3cfef50610a0\") " pod="openshift-multus/network-metrics-daemon-rt2hx" Nov 03 00:40:52.851220 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:40:52.851203 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/98ca3972-fd0c-47f5-9d26-569a58e18bd1-run-systemd\") pod \"ovnkube-node-psqpn\" (UID: \"98ca3972-fd0c-47f5-9d26-569a58e18bd1\") " pod="openshift-ovn-kubernetes/ovnkube-node-psqpn" Nov 03 00:40:52.851768 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:40:52.851232 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/460779b2-3903-4b23-9b4d-e25142782a21-serviceca\") pod \"node-ca-bbk2h\" (UID: \"460779b2-3903-4b23-9b4d-e25142782a21\") " pod="openshift-image-registry/node-ca-bbk2h" Nov 03 00:40:52.851768 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:40:52.851265 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/6650035b-2220-491c-9683-8a03ccb7af6a-cnibin\") pod \"multus-additional-cni-plugins-gqppj\" (UID: \"6650035b-2220-491c-9683-8a03ccb7af6a\") " pod="openshift-multus/multus-additional-cni-plugins-gqppj" Nov 03 00:40:52.851768 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:40:52.851301 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/6650035b-2220-491c-9683-8a03ccb7af6a-cni-binary-copy\") pod \"multus-additional-cni-plugins-gqppj\" (UID: \"6650035b-2220-491c-9683-8a03ccb7af6a\") " pod="openshift-multus/multus-additional-cni-plugins-gqppj" Nov 03 00:40:52.851768 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:40:52.851335 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/ca0f883c-c2b7-4bf6-a90b-ffdbeac161a8-tmp\") pod \"insights-operator-7c4986d449-xprcf\" (UID: \"ca0f883c-c2b7-4bf6-a90b-ffdbeac161a8\") " pod="openshift-insights/insights-operator-7c4986d449-xprcf" Nov 03 00:40:52.851768 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:40:52.851355 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/2b050b18-31f6-4a22-9902-0435d11018b9-serving-cert\") pod \"service-ca-operator-544d749c8c-7lqbv\" (UID: \"2b050b18-31f6-4a22-9902-0435d11018b9\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-7lqbv" Nov 03 00:40:52.851768 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:40:52.851369 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/36b9d258-cb2c-4d2b-880f-ae93b065034c-config\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-khmv7\" (UID: \"36b9d258-cb2c-4d2b-880f-ae93b065034c\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-khmv7" Nov 03 00:40:52.851768 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:40:52.851423 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/ca0f883c-c2b7-4bf6-a90b-ffdbeac161a8-tmp\") pod \"insights-operator-7c4986d449-xprcf\" (UID: \"ca0f883c-c2b7-4bf6-a90b-ffdbeac161a8\") " pod="openshift-insights/insights-operator-7c4986d449-xprcf" Nov 03 00:40:52.851768 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:40:52.851425 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/b748c4f8-d021-4d37-b390-d395482fcddd-etc-sysconfig\") pod \"tuned-gtwxp\" (UID: \"b748c4f8-d021-4d37-b390-d395482fcddd\") " pod="openshift-cluster-node-tuning-operator/tuned-gtwxp" Nov 03 00:40:52.851768 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:40:52.851464 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sbn8x\" (UniqueName: \"kubernetes.io/projected/525dc47d-afde-4019-ad01-80614291f282-kube-api-access-sbn8x\") pod \"network-check-source-6f78d59f99-z5f9s\" (UID: \"525dc47d-afde-4019-ad01-80614291f282\") " pod="openshift-network-diagnostics/network-check-source-6f78d59f99-z5f9s" Nov 03 00:40:52.851768 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:40:52.851501 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/bc513cba-d924-4719-a2ba-711529175ee0-nginx-conf\") pod \"networking-console-plugin-86c478bb88-z2qgj\" (UID: \"bc513cba-d924-4719-a2ba-711529175ee0\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-z2qgj" Nov 03 00:40:52.851768 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:40:52.851631 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/65d134ac-8a25-4c67-b6c7-cd752a098daf-trusted-ca\") pod \"console-operator-7cd56b8dbc-n42s6\" (UID: \"65d134ac-8a25-4c67-b6c7-cd752a098daf\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-n42s6" Nov 03 00:40:52.855642 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:40:52.855619 4939 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\"" Nov 03 00:40:52.855723 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:40:52.855676 4939 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\"" Nov 03 00:40:52.855723 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:40:52.855681 4939 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\"" Nov 03 00:40:52.855918 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:40:52.855893 4939 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\"" Nov 03 00:40:52.855997 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:40:52.855895 4939 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\"" Nov 03 00:40:52.855997 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:40:52.855935 4939 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\"" Nov 03 00:40:52.855997 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:40:52.855951 4939 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\"" Nov 03 00:40:52.856138 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:40:52.856028 4939 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\"" Nov 03 00:40:52.856188 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:40:52.856147 4939 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\"" Nov 03 00:40:52.856188 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:40:52.856157 4939 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\"" Nov 03 00:40:52.856188 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:40:52.856179 4939 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-console\"/\"networking-console-plugin\"" Nov 03 00:40:52.856289 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:40:52.856190 4939 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\"" Nov 03 00:40:52.856289 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:40:52.856220 4939 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\"" Nov 03 00:40:52.856289 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:40:52.856269 4939 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\"" Nov 03 00:40:52.856387 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:40:52.856289 4939 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"kube-root-ca.crt\"" Nov 03 00:40:52.856387 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:40:52.856310 4939 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"kube-root-ca.crt\"" Nov 03 00:40:52.856387 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:40:52.856337 4939 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\"" Nov 03 00:40:52.856387 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:40:52.856379 4939 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\"" Nov 03 00:40:52.856638 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:40:52.856620 4939 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\"" Nov 03 00:40:52.856706 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:40:52.856644 4939 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\"" Nov 03 00:40:52.856706 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:40:52.856668 4939 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"service-ca-bundle\"" Nov 03 00:40:52.856706 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:40:52.856625 4939 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"openshift-service-ca.crt\"" Nov 03 00:40:52.856706 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:40:52.856681 4939 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"openshift-service-ca.crt\"" Nov 03 00:40:52.856706 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:40:52.856670 4939 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\"" Nov 03 00:40:52.856706 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:40:52.856669 4939 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\"" Nov 03 00:40:52.856706 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:40:52.856667 4939 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"openshift-service-ca.crt\"" Nov 03 00:40:52.856706 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:40:52.856705 4939 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\"" Nov 03 00:40:52.856706 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:40:52.856710 4939 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-root-ca.crt\"" Nov 03 00:40:52.857101 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:40:52.856717 4939 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\"" Nov 03 00:40:52.857101 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:40:52.856694 4939 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"kube-root-ca.crt\"" Nov 03 00:40:52.857101 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:40:52.856847 4939 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\"" Nov 03 00:40:52.857101 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:40:52.856849 4939 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"openshift-service-ca.crt\"" Nov 03 00:40:52.857101 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:40:52.856864 4939 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Nov 03 00:40:52.857101 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:40:52.856876 4939 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\"" Nov 03 00:40:52.857101 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:40:52.856868 4939 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Nov 03 00:40:52.857101 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:40:52.856900 4939 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\"" Nov 03 00:40:52.857101 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:40:52.856934 4939 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\"" Nov 03 00:40:52.857101 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:40:52.856981 4939 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\"" Nov 03 00:40:52.857101 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:40:52.856989 4939 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\"" Nov 03 00:40:52.857390 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:40:52.857376 4939 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\"" Nov 03 00:40:52.858666 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:40:52.858148 4939 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"config\"" Nov 03 00:40:52.858666 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:40:52.858205 4939 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\"" Nov 03 00:40:52.858666 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:40:52.858219 4939 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\"" Nov 03 00:40:52.858666 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:40:52.858395 4939 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"samples-operator-config\"" Nov 03 00:40:52.858666 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:40:52.858454 4939 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"openshift-service-ca.crt\"" Nov 03 00:40:52.858666 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:40:52.858481 4939 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\"" Nov 03 00:40:52.858666 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:40:52.858557 4939 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\"" Nov 03 00:40:52.858666 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:40:52.858623 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/6650035b-2220-491c-9683-8a03ccb7af6a-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-gqppj\" (UID: \"6650035b-2220-491c-9683-8a03ccb7af6a\") " pod="openshift-multus/multus-additional-cni-plugins-gqppj" Nov 03 00:40:52.859107 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:40:52.858733 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/98ca3972-fd0c-47f5-9d26-569a58e18bd1-ovnkube-config\") pod \"ovnkube-node-psqpn\" (UID: \"98ca3972-fd0c-47f5-9d26-569a58e18bd1\") " pod="openshift-ovn-kubernetes/ovnkube-node-psqpn" Nov 03 00:40:52.859107 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:40:52.858824 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/98ca3972-fd0c-47f5-9d26-569a58e18bd1-env-overrides\") pod \"ovnkube-node-psqpn\" (UID: \"98ca3972-fd0c-47f5-9d26-569a58e18bd1\") " pod="openshift-ovn-kubernetes/ovnkube-node-psqpn" Nov 03 00:40:52.859107 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:40:52.858896 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/c3ffbe53-7f23-4750-b95d-476e6cfc58a6-konnectivity-ca\") pod \"konnectivity-agent-82x8c\" (UID: \"c3ffbe53-7f23-4750-b95d-476e6cfc58a6\") " pod="kube-system/konnectivity-agent-82x8c" Nov 03 00:40:52.859107 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:40:52.858943 4939 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemetry-config\"" Nov 03 00:40:52.862057 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:40:52.860106 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"samples-operator-config\" (UniqueName: \"kubernetes.io/configmap/6eafe6f6-cf90-446f-a8d5-620cd2e02775-samples-operator-config\") pod \"cluster-samples-operator-6ddb6c8b94-2ngwx\" (UID: \"6eafe6f6-cf90-446f-a8d5-620cd2e02775\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-2ngwx" Nov 03 00:40:52.862057 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:40:52.860197 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/c736fd3f-973f-4999-8d4e-c405079b2888-iptables-alerter-script\") pod \"iptables-alerter-5spnk\" (UID: \"c736fd3f-973f-4999-8d4e-c405079b2888\") " pod="openshift-network-operator/iptables-alerter-5spnk" Nov 03 00:40:52.862057 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:40:52.860271 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/dd0cd8a1-4d6b-424c-ac39-5f5ddb9cd87e-registry-certificates\") pod \"image-registry-65f77c855d-6tnc2\" (UID: \"dd0cd8a1-4d6b-424c-ac39-5f5ddb9cd87e\") " pod="openshift-image-registry/image-registry-65f77c855d-6tnc2" Nov 03 00:40:52.862057 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:40:52.860417 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/98ca3972-fd0c-47f5-9d26-569a58e18bd1-ovnkube-script-lib\") pod \"ovnkube-node-psqpn\" (UID: \"98ca3972-fd0c-47f5-9d26-569a58e18bd1\") " pod="openshift-ovn-kubernetes/ovnkube-node-psqpn" Nov 03 00:40:52.862057 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:40:52.860593 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/06973556-f51c-4d1c-ae55-fb77699bc5b3-telemetry-config\") pod \"cluster-monitoring-operator-884576c57-4bm4f\" (UID: \"06973556-f51c-4d1c-ae55-fb77699bc5b3\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-4bm4f" Nov 03 00:40:52.862057 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:40:52.860639 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/dd0cd8a1-4d6b-424c-ac39-5f5ddb9cd87e-bound-sa-token\") pod \"image-registry-65f77c855d-6tnc2\" (UID: \"dd0cd8a1-4d6b-424c-ac39-5f5ddb9cd87e\") " pod="openshift-image-registry/image-registry-65f77c855d-6tnc2" Nov 03 00:40:52.862057 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:40:52.860748 4939 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\"" Nov 03 00:40:52.862430 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:40:52.862152 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/460779b2-3903-4b23-9b4d-e25142782a21-serviceca\") pod \"node-ca-bbk2h\" (UID: \"460779b2-3903-4b23-9b4d-e25142782a21\") " pod="openshift-image-registry/node-ca-bbk2h" Nov 03 00:40:52.862430 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:40:52.862195 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/6650035b-2220-491c-9683-8a03ccb7af6a-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-gqppj\" (UID: \"6650035b-2220-491c-9683-8a03ccb7af6a\") " pod="openshift-multus/multus-additional-cni-plugins-gqppj" Nov 03 00:40:52.862430 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:40:52.862242 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jbfhc\" (UniqueName: \"kubernetes.io/projected/dd0cd8a1-4d6b-424c-ac39-5f5ddb9cd87e-kube-api-access-jbfhc\") pod \"image-registry-65f77c855d-6tnc2\" (UID: \"dd0cd8a1-4d6b-424c-ac39-5f5ddb9cd87e\") " pod="openshift-image-registry/image-registry-65f77c855d-6tnc2" Nov 03 00:40:52.862430 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:40:52.862296 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/6650035b-2220-491c-9683-8a03ccb7af6a-cni-binary-copy\") pod \"multus-additional-cni-plugins-gqppj\" (UID: \"6650035b-2220-491c-9683-8a03ccb7af6a\") " pod="openshift-multus/multus-additional-cni-plugins-gqppj" Nov 03 00:40:52.862430 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:40:52.862396 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/bc513cba-d924-4719-a2ba-711529175ee0-nginx-conf\") pod \"networking-console-plugin-86c478bb88-z2qgj\" (UID: \"bc513cba-d924-4719-a2ba-711529175ee0\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-z2qgj" Nov 03 00:40:52.862686 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:40:52.862587 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/36b9d258-cb2c-4d2b-880f-ae93b065034c-config\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-khmv7\" (UID: \"36b9d258-cb2c-4d2b-880f-ae93b065034c\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-khmv7" Nov 03 00:40:52.862686 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:40:52.862637 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fdrwp\" (UniqueName: \"kubernetes.io/projected/cf217a21-c1bd-48be-b203-f59743bceb33-kube-api-access-fdrwp\") pod \"router-default-d7798ccf5-442kt\" (UID: \"cf217a21-c1bd-48be-b203-f59743bceb33\") " pod="openshift-ingress/router-default-d7798ccf5-442kt" Nov 03 00:40:52.862787 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:40:52.862739 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-79khw\" (UniqueName: \"kubernetes.io/projected/36b9d258-cb2c-4d2b-880f-ae93b065034c-kube-api-access-79khw\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-khmv7\" (UID: \"36b9d258-cb2c-4d2b-880f-ae93b065034c\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-khmv7" Nov 03 00:40:52.862990 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:40:52.862973 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8j6kr\" (UniqueName: \"kubernetes.io/projected/65d134ac-8a25-4c67-b6c7-cd752a098daf-kube-api-access-8j6kr\") pod \"console-operator-7cd56b8dbc-n42s6\" (UID: \"65d134ac-8a25-4c67-b6c7-cd752a098daf\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-n42s6" Nov 03 00:40:52.863336 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:40:52.863318 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qcvwz\" (UniqueName: \"kubernetes.io/projected/ca0f883c-c2b7-4bf6-a90b-ffdbeac161a8-kube-api-access-qcvwz\") pod \"insights-operator-7c4986d449-xprcf\" (UID: \"ca0f883c-c2b7-4bf6-a90b-ffdbeac161a8\") " pod="openshift-insights/insights-operator-7c4986d449-xprcf" Nov 03 00:40:52.863466 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:40:52.863448 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sbn8x\" (UniqueName: \"kubernetes.io/projected/525dc47d-afde-4019-ad01-80614291f282-kube-api-access-sbn8x\") pod \"network-check-source-6f78d59f99-z5f9s\" (UID: \"525dc47d-afde-4019-ad01-80614291f282\") " pod="openshift-network-diagnostics/network-check-source-6f78d59f99-z5f9s" Nov 03 00:40:52.865784 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:40:52.865767 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-q4b9t\" (UniqueName: \"kubernetes.io/projected/460779b2-3903-4b23-9b4d-e25142782a21-kube-api-access-q4b9t\") pod \"node-ca-bbk2h\" (UID: \"460779b2-3903-4b23-9b4d-e25142782a21\") " pod="openshift-image-registry/node-ca-bbk2h" Nov 03 00:40:52.866455 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:40:52.866438 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wzln7\" (UniqueName: \"kubernetes.io/projected/2b050b18-31f6-4a22-9902-0435d11018b9-kube-api-access-wzln7\") pod \"service-ca-operator-544d749c8c-7lqbv\" (UID: \"2b050b18-31f6-4a22-9902-0435d11018b9\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-7lqbv" Nov 03 00:40:52.866903 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:40:52.866884 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4znwf\" (UniqueName: \"kubernetes.io/projected/06973556-f51c-4d1c-ae55-fb77699bc5b3-kube-api-access-4znwf\") pod \"cluster-monitoring-operator-884576c57-4bm4f\" (UID: \"06973556-f51c-4d1c-ae55-fb77699bc5b3\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-4bm4f" Nov 03 00:40:52.867313 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:40:52.867291 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7zxdt\" (UniqueName: \"kubernetes.io/projected/b0bd4bd5-12ca-49df-a3d9-4243f4bd642e-kube-api-access-7zxdt\") pod \"kas-connection-checker-685895c7c9-ntqw6\" (UID: \"b0bd4bd5-12ca-49df-a3d9-4243f4bd642e\") " pod="kube-system/kas-connection-checker-685895c7c9-ntqw6" Nov 03 00:40:52.867686 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:40:52.867668 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hsvqw\" (UniqueName: \"kubernetes.io/projected/b748c4f8-d021-4d37-b390-d395482fcddd-kube-api-access-hsvqw\") pod \"tuned-gtwxp\" (UID: \"b748c4f8-d021-4d37-b390-d395482fcddd\") " pod="openshift-cluster-node-tuning-operator/tuned-gtwxp" Nov 03 00:40:52.867742 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:40:52.867706 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-h58rt\" (UniqueName: \"kubernetes.io/projected/56261beb-517f-4c88-b59a-42ab41e278fd-kube-api-access-h58rt\") pod \"volume-data-source-validator-58c9f5f46d-w4xtw\" (UID: \"56261beb-517f-4c88-b59a-42ab41e278fd\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-w4xtw" Nov 03 00:40:52.868423 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:40:52.868408 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2tdmf\" (UniqueName: \"kubernetes.io/projected/c736fd3f-973f-4999-8d4e-c405079b2888-kube-api-access-2tdmf\") pod \"iptables-alerter-5spnk\" (UID: \"c736fd3f-973f-4999-8d4e-c405079b2888\") " pod="openshift-network-operator/iptables-alerter-5spnk" Nov 03 00:40:52.868477 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:40:52.868447 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-llh2g\" (UniqueName: \"kubernetes.io/projected/136086c0-4846-437a-ad98-3cfef50610a0-kube-api-access-llh2g\") pod \"network-metrics-daemon-rt2hx\" (UID: \"136086c0-4846-437a-ad98-3cfef50610a0\") " pod="openshift-multus/network-metrics-daemon-rt2hx" Nov 03 00:40:52.868863 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:40:52.868848 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6kcsm\" (UniqueName: \"kubernetes.io/projected/6eafe6f6-cf90-446f-a8d5-620cd2e02775-kube-api-access-6kcsm\") pod \"cluster-samples-operator-6ddb6c8b94-2ngwx\" (UID: \"6eafe6f6-cf90-446f-a8d5-620cd2e02775\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-2ngwx" Nov 03 00:40:52.869230 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:40:52.869210 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/dd0cd8a1-4d6b-424c-ac39-5f5ddb9cd87e-trusted-ca\") pod \"image-registry-65f77c855d-6tnc2\" (UID: \"dd0cd8a1-4d6b-424c-ac39-5f5ddb9cd87e\") " pod="openshift-image-registry/image-registry-65f77c855d-6tnc2" Nov 03 00:40:52.869683 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:40:52.869668 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4bhqh\" (UniqueName: \"kubernetes.io/projected/98ca3972-fd0c-47f5-9d26-569a58e18bd1-kube-api-access-4bhqh\") pod \"ovnkube-node-psqpn\" (UID: \"98ca3972-fd0c-47f5-9d26-569a58e18bd1\") " pod="openshift-ovn-kubernetes/ovnkube-node-psqpn" Nov 03 00:40:52.869748 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:40:52.869730 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-d8l25\" (UniqueName: \"kubernetes.io/projected/8cd3ca22-e4ce-4896-ae31-97605af4e4c6-kube-api-access-d8l25\") pod \"network-check-target-w46kd\" (UID: \"8cd3ca22-e4ce-4896-ae31-97605af4e4c6\") " pod="openshift-network-diagnostics/network-check-target-w46kd" Nov 03 00:40:52.869748 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:40:52.869742 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nswxk\" (UniqueName: \"kubernetes.io/projected/4237805a-d475-4bd7-865c-c6ef371c0e11-kube-api-access-nswxk\") pod \"kas-connection-checker-685895c7c9-zsd49\" (UID: \"4237805a-d475-4bd7-865c-c6ef371c0e11\") " pod="kube-system/kas-connection-checker-685895c7c9-zsd49" Nov 03 00:40:52.870185 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:40:52.870168 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pdmr4\" (UniqueName: \"kubernetes.io/projected/29a64a08-67ef-4c82-8a82-8cd4a4675327-kube-api-access-pdmr4\") pod \"kas-connection-checker-685895c7c9-xst6j\" (UID: \"29a64a08-67ef-4c82-8a82-8cd4a4675327\") " pod="kube-system/kas-connection-checker-685895c7c9-xst6j" Nov 03 00:40:52.870454 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:40:52.870440 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rkf7s\" (UniqueName: \"kubernetes.io/projected/6650035b-2220-491c-9683-8a03ccb7af6a-kube-api-access-rkf7s\") pod \"multus-additional-cni-plugins-gqppj\" (UID: \"6650035b-2220-491c-9683-8a03ccb7af6a\") " pod="openshift-multus/multus-additional-cni-plugins-gqppj" Nov 03 00:40:52.896811 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:40:52.896788 4939 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-storage-version-migrator-sa-dockercfg-dx4w2\"" Nov 03 00:40:52.912801 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:40:52.912781 4939 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-140-107.us-east-2.compute.internal" Nov 03 00:40:52.912950 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:40:52.912936 4939 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-107.us-east-2.compute.internal" Nov 03 00:40:52.925992 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:40:52.925978 4939 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Nov 03 00:40:52.926061 ip-10-0-140-107 kubenswrapper[4939]: E1103 00:40:52.926013 4939 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-apiserver-proxy-ip-10-0-140-107.us-east-2.compute.internal\" already exists" pod="kube-system/kube-apiserver-proxy-ip-10-0-140-107.us-east-2.compute.internal" Nov 03 00:40:52.927039 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:40:52.927025 4939 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Nov 03 00:40:52.927074 ip-10-0-140-107 kubenswrapper[4939]: E1103 00:40:52.927057 4939 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-rbac-proxy-crio-ip-10-0-140-107.us-east-2.compute.internal\" already exists" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-107.us-east-2.compute.internal" Nov 03 00:40:52.934257 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:40:52.934239 4939 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world" Nov 03 00:40:52.952535 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:40:52.952515 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/00dc2920-34f2-4e0a-892e-7759a16162d5-hostroot\") pod \"multus-tb9hd\" (UID: \"00dc2920-34f2-4e0a-892e-7759a16162d5\") " pod="openshift-multus/multus-tb9hd" Nov 03 00:40:52.952603 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:40:52.952542 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/69b087f8-4a23-4ec8-808e-b61d2e16f67b-config-volume\") pod \"dns-default-266ht\" (UID: \"69b087f8-4a23-4ec8-808e-b61d2e16f67b\") " pod="openshift-dns/dns-default-266ht" Nov 03 00:40:52.952603 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:40:52.952560 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/dd0cd8a1-4d6b-424c-ac39-5f5ddb9cd87e-registry-tls\") pod \"image-registry-65f77c855d-6tnc2\" (UID: \"dd0cd8a1-4d6b-424c-ac39-5f5ddb9cd87e\") " pod="openshift-image-registry/image-registry-65f77c855d-6tnc2" Nov 03 00:40:52.952603 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:40:52.952575 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/b748c4f8-d021-4d37-b390-d395482fcddd-etc-sysctl-d\") pod \"tuned-gtwxp\" (UID: \"b748c4f8-d021-4d37-b390-d395482fcddd\") " pod="openshift-cluster-node-tuning-operator/tuned-gtwxp" Nov 03 00:40:52.952603 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:40:52.952590 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/98ca3972-fd0c-47f5-9d26-569a58e18bd1-host-cni-bin\") pod \"ovnkube-node-psqpn\" (UID: \"98ca3972-fd0c-47f5-9d26-569a58e18bd1\") " pod="openshift-ovn-kubernetes/ovnkube-node-psqpn" Nov 03 00:40:52.952802 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:40:52.952605 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/00dc2920-34f2-4e0a-892e-7759a16162d5-multus-conf-dir\") pod \"multus-tb9hd\" (UID: \"00dc2920-34f2-4e0a-892e-7759a16162d5\") " pod="openshift-multus/multus-tb9hd" Nov 03 00:40:52.952802 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:40:52.952712 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/52a19842-3750-4d2b-bcb2-aebddee81097-trusted-ca\") pod \"image-registry-6669dc8887-ws544\" (UID: \"52a19842-3750-4d2b-bcb2-aebddee81097\") " pod="openshift-image-registry/image-registry-6669dc8887-ws544" Nov 03 00:40:52.952802 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:40:52.952757 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/6eafe6f6-cf90-446f-a8d5-620cd2e02775-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-2ngwx\" (UID: \"6eafe6f6-cf90-446f-a8d5-620cd2e02775\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-2ngwx" Nov 03 00:40:52.952958 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:40:52.952813 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/69b087f8-4a23-4ec8-808e-b61d2e16f67b-config-volume\") pod \"dns-default-266ht\" (UID: \"69b087f8-4a23-4ec8-808e-b61d2e16f67b\") " pod="openshift-dns/dns-default-266ht" Nov 03 00:40:52.952958 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:40:52.952829 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/98ca3972-fd0c-47f5-9d26-569a58e18bd1-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-psqpn\" (UID: \"98ca3972-fd0c-47f5-9d26-569a58e18bd1\") " pod="openshift-ovn-kubernetes/ovnkube-node-psqpn" Nov 03 00:40:52.952958 ip-10-0-140-107 kubenswrapper[4939]: E1103 00:40:52.952845 4939 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Nov 03 00:40:52.952958 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:40:52.952869 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4q9wl\" (UniqueName: \"kubernetes.io/projected/945da51a-32cb-4769-900d-1aeb266c7565-kube-api-access-4q9wl\") pod \"insights-runtime-extractor-sthjh\" (UID: \"945da51a-32cb-4769-900d-1aeb266c7565\") " pod="openshift-insights/insights-runtime-extractor-sthjh" Nov 03 00:40:52.952958 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:40:52.952876 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/98ca3972-fd0c-47f5-9d26-569a58e18bd1-host-cni-bin\") pod \"ovnkube-node-psqpn\" (UID: \"98ca3972-fd0c-47f5-9d26-569a58e18bd1\") " pod="openshift-ovn-kubernetes/ovnkube-node-psqpn" Nov 03 00:40:52.952958 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:40:52.952895 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/bc513cba-d924-4719-a2ba-711529175ee0-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-z2qgj\" (UID: \"bc513cba-d924-4719-a2ba-711529175ee0\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-z2qgj" Nov 03 00:40:52.952958 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:40:52.952895 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/b748c4f8-d021-4d37-b390-d395482fcddd-etc-sysctl-d\") pod \"tuned-gtwxp\" (UID: \"b748c4f8-d021-4d37-b390-d395482fcddd\") " pod="openshift-cluster-node-tuning-operator/tuned-gtwxp" Nov 03 00:40:52.952958 ip-10-0-140-107 kubenswrapper[4939]: E1103 00:40:52.952960 4939 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6eafe6f6-cf90-446f-a8d5-620cd2e02775-samples-operator-tls podName:6eafe6f6-cf90-446f-a8d5-620cd2e02775 nodeName:}" failed. No retries permitted until 2026-11-03 00:40:53.452938847 +0000 UTC m=+1.717273789 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/6eafe6f6-cf90-446f-a8d5-620cd2e02775-samples-operator-tls") pod "cluster-samples-operator-6ddb6c8b94-2ngwx" (UID: "6eafe6f6-cf90-446f-a8d5-620cd2e02775") : secret "samples-operator-tls" not found Nov 03 00:40:52.953299 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:40:52.952957 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/98ca3972-fd0c-47f5-9d26-569a58e18bd1-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-psqpn\" (UID: \"98ca3972-fd0c-47f5-9d26-569a58e18bd1\") " pod="openshift-ovn-kubernetes/ovnkube-node-psqpn" Nov 03 00:40:52.953299 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:40:52.952981 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/460779b2-3903-4b23-9b4d-e25142782a21-host\") pod \"node-ca-bbk2h\" (UID: \"460779b2-3903-4b23-9b4d-e25142782a21\") " pod="openshift-image-registry/node-ca-bbk2h" Nov 03 00:40:52.953299 ip-10-0-140-107 kubenswrapper[4939]: E1103 00:40:52.952997 4939 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Nov 03 00:40:52.953299 ip-10-0-140-107 kubenswrapper[4939]: E1103 00:40:52.953019 4939 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-65f77c855d-6tnc2: secret "image-registry-tls" not found Nov 03 00:40:52.953299 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:40:52.953025 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/460779b2-3903-4b23-9b4d-e25142782a21-host\") pod \"node-ca-bbk2h\" (UID: \"460779b2-3903-4b23-9b4d-e25142782a21\") " pod="openshift-image-registry/node-ca-bbk2h" Nov 03 00:40:52.953299 ip-10-0-140-107 kubenswrapper[4939]: E1103 00:40:52.953042 4939 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Nov 03 00:40:52.953299 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:40:52.953041 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/cabd402c-79c5-4699-9c85-d415d362b4d0-dbus\") pod \"global-pull-secret-syncer-jv7cw\" (UID: \"cabd402c-79c5-4699-9c85-d415d362b4d0\") " pod="kube-system/global-pull-secret-syncer-jv7cw" Nov 03 00:40:52.953299 ip-10-0-140-107 kubenswrapper[4939]: E1103 00:40:52.953097 4939 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/dd0cd8a1-4d6b-424c-ac39-5f5ddb9cd87e-registry-tls podName:dd0cd8a1-4d6b-424c-ac39-5f5ddb9cd87e nodeName:}" failed. No retries permitted until 2026-11-03 00:40:53.453078958 +0000 UTC m=+1.717413904 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/dd0cd8a1-4d6b-424c-ac39-5f5ddb9cd87e-registry-tls") pod "image-registry-65f77c855d-6tnc2" (UID: "dd0cd8a1-4d6b-424c-ac39-5f5ddb9cd87e") : secret "image-registry-tls" not found Nov 03 00:40:52.953299 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:40:52.953097 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/37bf96f5-045f-49e8-83e8-410df66bfb1f-registration-dir\") pod \"aws-ebs-csi-driver-node-br8vt\" (UID: \"37bf96f5-045f-49e8-83e8-410df66bfb1f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-br8vt" Nov 03 00:40:52.953299 ip-10-0-140-107 kubenswrapper[4939]: E1103 00:40:52.953116 4939 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/bc513cba-d924-4719-a2ba-711529175ee0-networking-console-plugin-cert podName:bc513cba-d924-4719-a2ba-711529175ee0 nodeName:}" failed. No retries permitted until 2026-11-03 00:40:53.453106862 +0000 UTC m=+1.717441785 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/bc513cba-d924-4719-a2ba-711529175ee0-networking-console-plugin-cert") pod "networking-console-plugin-86c478bb88-z2qgj" (UID: "bc513cba-d924-4719-a2ba-711529175ee0") : secret "networking-console-plugin-cert" not found Nov 03 00:40:52.953299 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:40:52.953136 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/52a19842-3750-4d2b-bcb2-aebddee81097-trusted-ca\") pod \"image-registry-6669dc8887-ws544\" (UID: \"52a19842-3750-4d2b-bcb2-aebddee81097\") " pod="openshift-image-registry/image-registry-6669dc8887-ws544" Nov 03 00:40:52.953299 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:40:52.953152 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sfdp9\" (UniqueName: \"kubernetes.io/projected/ad5bf442-729d-411c-96e4-2e5105ebadcf-kube-api-access-sfdp9\") pod \"node-resolver-cjj2h\" (UID: \"ad5bf442-729d-411c-96e4-2e5105ebadcf\") " pod="openshift-dns/node-resolver-cjj2h" Nov 03 00:40:52.953299 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:40:52.953176 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/00dc2920-34f2-4e0a-892e-7759a16162d5-multus-socket-dir-parent\") pod \"multus-tb9hd\" (UID: \"00dc2920-34f2-4e0a-892e-7759a16162d5\") " pod="openshift-multus/multus-tb9hd" Nov 03 00:40:52.953299 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:40:52.953192 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/00dc2920-34f2-4e0a-892e-7759a16162d5-host-var-lib-cni-bin\") pod \"multus-tb9hd\" (UID: \"00dc2920-34f2-4e0a-892e-7759a16162d5\") " pod="openshift-multus/multus-tb9hd" Nov 03 00:40:52.953299 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:40:52.953208 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/b748c4f8-d021-4d37-b390-d395482fcddd-run\") pod \"tuned-gtwxp\" (UID: \"b748c4f8-d021-4d37-b390-d395482fcddd\") " pod="openshift-cluster-node-tuning-operator/tuned-gtwxp" Nov 03 00:40:52.953299 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:40:52.953270 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/b748c4f8-d021-4d37-b390-d395482fcddd-run\") pod \"tuned-gtwxp\" (UID: \"b748c4f8-d021-4d37-b390-d395482fcddd\") " pod="openshift-cluster-node-tuning-operator/tuned-gtwxp" Nov 03 00:40:52.953797 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:40:52.953330 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/00dc2920-34f2-4e0a-892e-7759a16162d5-etc-kubernetes\") pod \"multus-tb9hd\" (UID: \"00dc2920-34f2-4e0a-892e-7759a16162d5\") " pod="openshift-multus/multus-tb9hd" Nov 03 00:40:52.953797 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:40:52.953361 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tc9rq\" (UniqueName: \"kubernetes.io/projected/52a19842-3750-4d2b-bcb2-aebddee81097-kube-api-access-tc9rq\") pod \"image-registry-6669dc8887-ws544\" (UID: \"52a19842-3750-4d2b-bcb2-aebddee81097\") " pod="openshift-image-registry/image-registry-6669dc8887-ws544" Nov 03 00:40:52.953797 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:40:52.953391 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/cf217a21-c1bd-48be-b203-f59743bceb33-service-ca-bundle\") pod \"router-default-d7798ccf5-442kt\" (UID: \"cf217a21-c1bd-48be-b203-f59743bceb33\") " pod="openshift-ingress/router-default-d7798ccf5-442kt" Nov 03 00:40:52.953797 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:40:52.953409 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/98ca3972-fd0c-47f5-9d26-569a58e18bd1-run-ovn\") pod \"ovnkube-node-psqpn\" (UID: \"98ca3972-fd0c-47f5-9d26-569a58e18bd1\") " pod="openshift-ovn-kubernetes/ovnkube-node-psqpn" Nov 03 00:40:52.953797 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:40:52.953425 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qmkgl\" (UniqueName: \"kubernetes.io/projected/75481915-3cbe-475b-abe7-cb4e81c93427-kube-api-access-qmkgl\") pod \"ingress-canary-s9hbz\" (UID: \"75481915-3cbe-475b-abe7-cb4e81c93427\") " pod="openshift-ingress-canary/ingress-canary-s9hbz" Nov 03 00:40:52.953797 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:40:52.953441 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/37bf96f5-045f-49e8-83e8-410df66bfb1f-device-dir\") pod \"aws-ebs-csi-driver-node-br8vt\" (UID: \"37bf96f5-045f-49e8-83e8-410df66bfb1f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-br8vt" Nov 03 00:40:52.953797 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:40:52.953458 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/b748c4f8-d021-4d37-b390-d395482fcddd-etc-modprobe-d\") pod \"tuned-gtwxp\" (UID: \"b748c4f8-d021-4d37-b390-d395482fcddd\") " pod="openshift-cluster-node-tuning-operator/tuned-gtwxp" Nov 03 00:40:52.953797 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:40:52.953476 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/98ca3972-fd0c-47f5-9d26-569a58e18bd1-run-ovn\") pod \"ovnkube-node-psqpn\" (UID: \"98ca3972-fd0c-47f5-9d26-569a58e18bd1\") " pod="openshift-ovn-kubernetes/ovnkube-node-psqpn" Nov 03 00:40:52.953797 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:40:52.953479 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/98ca3972-fd0c-47f5-9d26-569a58e18bd1-host-run-ovn-kubernetes\") pod \"ovnkube-node-psqpn\" (UID: \"98ca3972-fd0c-47f5-9d26-569a58e18bd1\") " pod="openshift-ovn-kubernetes/ovnkube-node-psqpn" Nov 03 00:40:52.953797 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:40:52.953525 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/37bf96f5-045f-49e8-83e8-410df66bfb1f-etc-selinux\") pod \"aws-ebs-csi-driver-node-br8vt\" (UID: \"37bf96f5-045f-49e8-83e8-410df66bfb1f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-br8vt" Nov 03 00:40:52.953797 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:40:52.953561 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/b748c4f8-d021-4d37-b390-d395482fcddd-etc-modprobe-d\") pod \"tuned-gtwxp\" (UID: \"b748c4f8-d021-4d37-b390-d395482fcddd\") " pod="openshift-cluster-node-tuning-operator/tuned-gtwxp" Nov 03 00:40:52.953797 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:40:52.953565 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/136086c0-4846-437a-ad98-3cfef50610a0-metrics-certs\") pod \"network-metrics-daemon-rt2hx\" (UID: \"136086c0-4846-437a-ad98-3cfef50610a0\") " pod="openshift-multus/network-metrics-daemon-rt2hx" Nov 03 00:40:52.953797 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:40:52.953568 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/98ca3972-fd0c-47f5-9d26-569a58e18bd1-host-run-ovn-kubernetes\") pod \"ovnkube-node-psqpn\" (UID: \"98ca3972-fd0c-47f5-9d26-569a58e18bd1\") " pod="openshift-ovn-kubernetes/ovnkube-node-psqpn" Nov 03 00:40:52.953797 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:40:52.953600 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/98ca3972-fd0c-47f5-9d26-569a58e18bd1-run-systemd\") pod \"ovnkube-node-psqpn\" (UID: \"98ca3972-fd0c-47f5-9d26-569a58e18bd1\") " pod="openshift-ovn-kubernetes/ovnkube-node-psqpn" Nov 03 00:40:52.953797 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:40:52.953628 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/ad5bf442-729d-411c-96e4-2e5105ebadcf-hosts-file\") pod \"node-resolver-cjj2h\" (UID: \"ad5bf442-729d-411c-96e4-2e5105ebadcf\") " pod="openshift-dns/node-resolver-cjj2h" Nov 03 00:40:52.953797 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:40:52.953661 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/52a19842-3750-4d2b-bcb2-aebddee81097-registry-certificates\") pod \"image-registry-6669dc8887-ws544\" (UID: \"52a19842-3750-4d2b-bcb2-aebddee81097\") " pod="openshift-image-registry/image-registry-6669dc8887-ws544" Nov 03 00:40:52.953797 ip-10-0-140-107 kubenswrapper[4939]: E1103 00:40:52.953706 4939 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found Nov 03 00:40:52.953797 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:40:52.953731 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/98ca3972-fd0c-47f5-9d26-569a58e18bd1-run-systemd\") pod \"ovnkube-node-psqpn\" (UID: \"98ca3972-fd0c-47f5-9d26-569a58e18bd1\") " pod="openshift-ovn-kubernetes/ovnkube-node-psqpn" Nov 03 00:40:52.953797 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:40:52.953742 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/6650035b-2220-491c-9683-8a03ccb7af6a-cnibin\") pod \"multus-additional-cni-plugins-gqppj\" (UID: \"6650035b-2220-491c-9683-8a03ccb7af6a\") " pod="openshift-multus/multus-additional-cni-plugins-gqppj" Nov 03 00:40:52.953797 ip-10-0-140-107 kubenswrapper[4939]: E1103 00:40:52.953768 4939 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/136086c0-4846-437a-ad98-3cfef50610a0-metrics-certs podName:136086c0-4846-437a-ad98-3cfef50610a0 nodeName:}" failed. No retries permitted until 2026-11-03 00:40:53.453752735 +0000 UTC m=+1.718087678 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/136086c0-4846-437a-ad98-3cfef50610a0-metrics-certs") pod "network-metrics-daemon-rt2hx" (UID: "136086c0-4846-437a-ad98-3cfef50610a0") : secret "metrics-daemon-secret" not found Nov 03 00:40:52.953797 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:40:52.953773 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/6650035b-2220-491c-9683-8a03ccb7af6a-cnibin\") pod \"multus-additional-cni-plugins-gqppj\" (UID: \"6650035b-2220-491c-9683-8a03ccb7af6a\") " pod="openshift-multus/multus-additional-cni-plugins-gqppj" Nov 03 00:40:52.953797 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:40:52.953798 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/52a19842-3750-4d2b-bcb2-aebddee81097-registry-certificates\") pod \"image-registry-6669dc8887-ws544\" (UID: \"52a19842-3750-4d2b-bcb2-aebddee81097\") " pod="openshift-image-registry/image-registry-6669dc8887-ws544" Nov 03 00:40:52.953797 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:40:52.953805 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/98ca3972-fd0c-47f5-9d26-569a58e18bd1-etc-openvswitch\") pod \"ovnkube-node-psqpn\" (UID: \"98ca3972-fd0c-47f5-9d26-569a58e18bd1\") " pod="openshift-ovn-kubernetes/ovnkube-node-psqpn" Nov 03 00:40:52.954497 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:40:52.953830 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/98ca3972-fd0c-47f5-9d26-569a58e18bd1-etc-openvswitch\") pod \"ovnkube-node-psqpn\" (UID: \"98ca3972-fd0c-47f5-9d26-569a58e18bd1\") " pod="openshift-ovn-kubernetes/ovnkube-node-psqpn" Nov 03 00:40:52.954497 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:40:52.953831 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ad5bf442-729d-411c-96e4-2e5105ebadcf-tmp-dir\") pod \"node-resolver-cjj2h\" (UID: \"ad5bf442-729d-411c-96e4-2e5105ebadcf\") " pod="openshift-dns/node-resolver-cjj2h" Nov 03 00:40:52.954497 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:40:52.953871 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ad5bf442-729d-411c-96e4-2e5105ebadcf-tmp-dir\") pod \"node-resolver-cjj2h\" (UID: \"ad5bf442-729d-411c-96e4-2e5105ebadcf\") " pod="openshift-dns/node-resolver-cjj2h" Nov 03 00:40:52.954497 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:40:52.953877 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/945da51a-32cb-4769-900d-1aeb266c7565-data-volume\") pod \"insights-runtime-extractor-sthjh\" (UID: \"945da51a-32cb-4769-900d-1aeb266c7565\") " pod="openshift-insights/insights-runtime-extractor-sthjh" Nov 03 00:40:52.954497 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:40:52.953895 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/00dc2920-34f2-4e0a-892e-7759a16162d5-cnibin\") pod \"multus-tb9hd\" (UID: \"00dc2920-34f2-4e0a-892e-7759a16162d5\") " pod="openshift-multus/multus-tb9hd" Nov 03 00:40:52.954497 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:40:52.953927 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/00dc2920-34f2-4e0a-892e-7759a16162d5-host-run-netns\") pod \"multus-tb9hd\" (UID: \"00dc2920-34f2-4e0a-892e-7759a16162d5\") " pod="openshift-multus/multus-tb9hd" Nov 03 00:40:52.954497 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:40:52.953961 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/52a19842-3750-4d2b-bcb2-aebddee81097-bound-sa-token\") pod \"image-registry-6669dc8887-ws544\" (UID: \"52a19842-3750-4d2b-bcb2-aebddee81097\") " pod="openshift-image-registry/image-registry-6669dc8887-ws544" Nov 03 00:40:52.954497 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:40:52.953978 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pj6ft\" (UniqueName: \"kubernetes.io/projected/69b087f8-4a23-4ec8-808e-b61d2e16f67b-kube-api-access-pj6ft\") pod \"dns-default-266ht\" (UID: \"69b087f8-4a23-4ec8-808e-b61d2e16f67b\") " pod="openshift-dns/dns-default-266ht" Nov 03 00:40:52.954497 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:40:52.953977 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/945da51a-32cb-4769-900d-1aeb266c7565-data-volume\") pod \"insights-runtime-extractor-sthjh\" (UID: \"945da51a-32cb-4769-900d-1aeb266c7565\") " pod="openshift-insights/insights-runtime-extractor-sthjh" Nov 03 00:40:52.954497 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:40:52.954066 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/cf217a21-c1bd-48be-b203-f59743bceb33-service-ca-bundle\") pod \"router-default-d7798ccf5-442kt\" (UID: \"cf217a21-c1bd-48be-b203-f59743bceb33\") " pod="openshift-ingress/router-default-d7798ccf5-442kt" Nov 03 00:40:52.954497 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:40:52.954071 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/6650035b-2220-491c-9683-8a03ccb7af6a-tuning-conf-dir\") pod \"multus-additional-cni-plugins-gqppj\" (UID: \"6650035b-2220-491c-9683-8a03ccb7af6a\") " pod="openshift-multus/multus-additional-cni-plugins-gqppj" Nov 03 00:40:52.954497 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:40:52.954104 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/6650035b-2220-491c-9683-8a03ccb7af6a-tuning-conf-dir\") pod \"multus-additional-cni-plugins-gqppj\" (UID: \"6650035b-2220-491c-9683-8a03ccb7af6a\") " pod="openshift-multus/multus-additional-cni-plugins-gqppj" Nov 03 00:40:52.954497 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:40:52.954114 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/98ca3972-fd0c-47f5-9d26-569a58e18bd1-systemd-units\") pod \"ovnkube-node-psqpn\" (UID: \"98ca3972-fd0c-47f5-9d26-569a58e18bd1\") " pod="openshift-ovn-kubernetes/ovnkube-node-psqpn" Nov 03 00:40:52.954497 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:40:52.954132 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/b748c4f8-d021-4d37-b390-d395482fcddd-sys\") pod \"tuned-gtwxp\" (UID: \"b748c4f8-d021-4d37-b390-d395482fcddd\") " pod="openshift-cluster-node-tuning-operator/tuned-gtwxp" Nov 03 00:40:52.954497 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:40:52.954153 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/cabd402c-79c5-4699-9c85-d415d362b4d0-kubelet-config\") pod \"global-pull-secret-syncer-jv7cw\" (UID: \"cabd402c-79c5-4699-9c85-d415d362b4d0\") " pod="kube-system/global-pull-secret-syncer-jv7cw" Nov 03 00:40:52.954497 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:40:52.954162 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/98ca3972-fd0c-47f5-9d26-569a58e18bd1-systemd-units\") pod \"ovnkube-node-psqpn\" (UID: \"98ca3972-fd0c-47f5-9d26-569a58e18bd1\") " pod="openshift-ovn-kubernetes/ovnkube-node-psqpn" Nov 03 00:40:52.954497 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:40:52.954168 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/52a19842-3750-4d2b-bcb2-aebddee81097-registry-tls\") pod \"image-registry-6669dc8887-ws544\" (UID: \"52a19842-3750-4d2b-bcb2-aebddee81097\") " pod="openshift-image-registry/image-registry-6669dc8887-ws544" Nov 03 00:40:52.954497 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:40:52.954186 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/b748c4f8-d021-4d37-b390-d395482fcddd-etc-sysctl-conf\") pod \"tuned-gtwxp\" (UID: \"b748c4f8-d021-4d37-b390-d395482fcddd\") " pod="openshift-cluster-node-tuning-operator/tuned-gtwxp" Nov 03 00:40:52.954497 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:40:52.954201 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/b748c4f8-d021-4d37-b390-d395482fcddd-sys\") pod \"tuned-gtwxp\" (UID: \"b748c4f8-d021-4d37-b390-d395482fcddd\") " pod="openshift-cluster-node-tuning-operator/tuned-gtwxp" Nov 03 00:40:52.954497 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:40:52.954283 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/b748c4f8-d021-4d37-b390-d395482fcddd-etc-sysctl-conf\") pod \"tuned-gtwxp\" (UID: \"b748c4f8-d021-4d37-b390-d395482fcddd\") " pod="openshift-cluster-node-tuning-operator/tuned-gtwxp" Nov 03 00:40:52.954497 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:40:52.954297 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/c736fd3f-973f-4999-8d4e-c405079b2888-host-slash\") pod \"iptables-alerter-5spnk\" (UID: \"c736fd3f-973f-4999-8d4e-c405079b2888\") " pod="openshift-network-operator/iptables-alerter-5spnk" Nov 03 00:40:52.954497 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:40:52.954334 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/52a19842-3750-4d2b-bcb2-aebddee81097-installation-pull-secrets\") pod \"image-registry-6669dc8887-ws544\" (UID: \"52a19842-3750-4d2b-bcb2-aebddee81097\") " pod="openshift-image-registry/image-registry-6669dc8887-ws544" Nov 03 00:40:52.954497 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:40:52.954354 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/37bf96f5-045f-49e8-83e8-410df66bfb1f-socket-dir\") pod \"aws-ebs-csi-driver-node-br8vt\" (UID: \"37bf96f5-045f-49e8-83e8-410df66bfb1f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-br8vt" Nov 03 00:40:52.954497 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:40:52.954360 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/c736fd3f-973f-4999-8d4e-c405079b2888-host-slash\") pod \"iptables-alerter-5spnk\" (UID: \"c736fd3f-973f-4999-8d4e-c405079b2888\") " pod="openshift-network-operator/iptables-alerter-5spnk" Nov 03 00:40:52.954497 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:40:52.954375 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/945da51a-32cb-4769-900d-1aeb266c7565-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-sthjh\" (UID: \"945da51a-32cb-4769-900d-1aeb266c7565\") " pod="openshift-insights/insights-runtime-extractor-sthjh" Nov 03 00:40:52.954497 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:40:52.954409 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/98ca3972-fd0c-47f5-9d26-569a58e18bd1-run-openvswitch\") pod \"ovnkube-node-psqpn\" (UID: \"98ca3972-fd0c-47f5-9d26-569a58e18bd1\") " pod="openshift-ovn-kubernetes/ovnkube-node-psqpn" Nov 03 00:40:52.954497 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:40:52.954427 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/98ca3972-fd0c-47f5-9d26-569a58e18bd1-node-log\") pod \"ovnkube-node-psqpn\" (UID: \"98ca3972-fd0c-47f5-9d26-569a58e18bd1\") " pod="openshift-ovn-kubernetes/ovnkube-node-psqpn" Nov 03 00:40:52.954497 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:40:52.954464 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/98ca3972-fd0c-47f5-9d26-569a58e18bd1-run-openvswitch\") pod \"ovnkube-node-psqpn\" (UID: \"98ca3972-fd0c-47f5-9d26-569a58e18bd1\") " pod="openshift-ovn-kubernetes/ovnkube-node-psqpn" Nov 03 00:40:52.954497 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:40:52.954467 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/06973556-f51c-4d1c-ae55-fb77699bc5b3-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-4bm4f\" (UID: \"06973556-f51c-4d1c-ae55-fb77699bc5b3\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-4bm4f" Nov 03 00:40:52.954497 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:40:52.954488 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/98ca3972-fd0c-47f5-9d26-569a58e18bd1-host-slash\") pod \"ovnkube-node-psqpn\" (UID: \"98ca3972-fd0c-47f5-9d26-569a58e18bd1\") " pod="openshift-ovn-kubernetes/ovnkube-node-psqpn" Nov 03 00:40:52.954497 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:40:52.954510 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/b748c4f8-d021-4d37-b390-d395482fcddd-var-lib-kubelet\") pod \"tuned-gtwxp\" (UID: \"b748c4f8-d021-4d37-b390-d395482fcddd\") " pod="openshift-cluster-node-tuning-operator/tuned-gtwxp" Nov 03 00:40:52.955385 ip-10-0-140-107 kubenswrapper[4939]: E1103 00:40:52.954528 4939 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Nov 03 00:40:52.955385 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:40:52.954538 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/00dc2920-34f2-4e0a-892e-7759a16162d5-cni-binary-copy\") pod \"multus-tb9hd\" (UID: \"00dc2920-34f2-4e0a-892e-7759a16162d5\") " pod="openshift-multus/multus-tb9hd" Nov 03 00:40:52.955385 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:40:52.954546 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/98ca3972-fd0c-47f5-9d26-569a58e18bd1-host-slash\") pod \"ovnkube-node-psqpn\" (UID: \"98ca3972-fd0c-47f5-9d26-569a58e18bd1\") " pod="openshift-ovn-kubernetes/ovnkube-node-psqpn" Nov 03 00:40:52.955385 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:40:52.954566 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/00dc2920-34f2-4e0a-892e-7759a16162d5-host-var-lib-kubelet\") pod \"multus-tb9hd\" (UID: \"00dc2920-34f2-4e0a-892e-7759a16162d5\") " pod="openshift-multus/multus-tb9hd" Nov 03 00:40:52.955385 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:40:52.954539 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/98ca3972-fd0c-47f5-9d26-569a58e18bd1-node-log\") pod \"ovnkube-node-psqpn\" (UID: \"98ca3972-fd0c-47f5-9d26-569a58e18bd1\") " pod="openshift-ovn-kubernetes/ovnkube-node-psqpn" Nov 03 00:40:52.955385 ip-10-0-140-107 kubenswrapper[4939]: E1103 00:40:52.954579 4939 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/06973556-f51c-4d1c-ae55-fb77699bc5b3-cluster-monitoring-operator-tls podName:06973556-f51c-4d1c-ae55-fb77699bc5b3 nodeName:}" failed. No retries permitted until 2026-11-03 00:40:53.45456528 +0000 UTC m=+1.718900203 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/06973556-f51c-4d1c-ae55-fb77699bc5b3-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-884576c57-4bm4f" (UID: "06973556-f51c-4d1c-ae55-fb77699bc5b3") : secret "cluster-monitoring-operator-tls" not found Nov 03 00:40:52.955385 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:40:52.954568 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/52a19842-3750-4d2b-bcb2-aebddee81097-installation-pull-secrets\") pod \"image-registry-6669dc8887-ws544\" (UID: \"52a19842-3750-4d2b-bcb2-aebddee81097\") " pod="openshift-image-registry/image-registry-6669dc8887-ws544" Nov 03 00:40:52.955385 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:40:52.954584 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/b748c4f8-d021-4d37-b390-d395482fcddd-var-lib-kubelet\") pod \"tuned-gtwxp\" (UID: \"b748c4f8-d021-4d37-b390-d395482fcddd\") " pod="openshift-cluster-node-tuning-operator/tuned-gtwxp" Nov 03 00:40:52.955385 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:40:52.954611 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/6650035b-2220-491c-9683-8a03ccb7af6a-system-cni-dir\") pod \"multus-additional-cni-plugins-gqppj\" (UID: \"6650035b-2220-491c-9683-8a03ccb7af6a\") " pod="openshift-multus/multus-additional-cni-plugins-gqppj" Nov 03 00:40:52.955385 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:40:52.954634 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/b748c4f8-d021-4d37-b390-d395482fcddd-etc-systemd\") pod \"tuned-gtwxp\" (UID: \"b748c4f8-d021-4d37-b390-d395482fcddd\") " pod="openshift-cluster-node-tuning-operator/tuned-gtwxp" Nov 03 00:40:52.955385 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:40:52.954649 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/98ca3972-fd0c-47f5-9d26-569a58e18bd1-log-socket\") pod \"ovnkube-node-psqpn\" (UID: \"98ca3972-fd0c-47f5-9d26-569a58e18bd1\") " pod="openshift-ovn-kubernetes/ovnkube-node-psqpn" Nov 03 00:40:52.955385 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:40:52.954684 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/6650035b-2220-491c-9683-8a03ccb7af6a-system-cni-dir\") pod \"multus-additional-cni-plugins-gqppj\" (UID: \"6650035b-2220-491c-9683-8a03ccb7af6a\") " pod="openshift-multus/multus-additional-cni-plugins-gqppj" Nov 03 00:40:52.955385 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:40:52.954692 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7wzbd\" (UniqueName: \"kubernetes.io/projected/68658033-8a49-4430-869f-a7eb79603484-kube-api-access-7wzbd\") pod \"migrator-66df5d496b-m7cxm\" (UID: \"68658033-8a49-4430-869f-a7eb79603484\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-m7cxm" Nov 03 00:40:52.955385 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:40:52.954697 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/98ca3972-fd0c-47f5-9d26-569a58e18bd1-log-socket\") pod \"ovnkube-node-psqpn\" (UID: \"98ca3972-fd0c-47f5-9d26-569a58e18bd1\") " pod="openshift-ovn-kubernetes/ovnkube-node-psqpn" Nov 03 00:40:52.955385 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:40:52.954702 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/b748c4f8-d021-4d37-b390-d395482fcddd-etc-systemd\") pod \"tuned-gtwxp\" (UID: \"b748c4f8-d021-4d37-b390-d395482fcddd\") " pod="openshift-cluster-node-tuning-operator/tuned-gtwxp" Nov 03 00:40:52.955385 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:40:52.954720 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/00dc2920-34f2-4e0a-892e-7759a16162d5-system-cni-dir\") pod \"multus-tb9hd\" (UID: \"00dc2920-34f2-4e0a-892e-7759a16162d5\") " pod="openshift-multus/multus-tb9hd" Nov 03 00:40:52.955385 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:40:52.954804 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/00dc2920-34f2-4e0a-892e-7759a16162d5-multus-cni-dir\") pod \"multus-tb9hd\" (UID: \"00dc2920-34f2-4e0a-892e-7759a16162d5\") " pod="openshift-multus/multus-tb9hd" Nov 03 00:40:52.955385 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:40:52.954834 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/00dc2920-34f2-4e0a-892e-7759a16162d5-host-run-k8s-cni-cncf-io\") pod \"multus-tb9hd\" (UID: \"00dc2920-34f2-4e0a-892e-7759a16162d5\") " pod="openshift-multus/multus-tb9hd" Nov 03 00:40:52.955385 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:40:52.954846 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/00dc2920-34f2-4e0a-892e-7759a16162d5-cni-binary-copy\") pod \"multus-tb9hd\" (UID: \"00dc2920-34f2-4e0a-892e-7759a16162d5\") " pod="openshift-multus/multus-tb9hd" Nov 03 00:40:52.955385 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:40:52.954864 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j62gg\" (UniqueName: \"kubernetes.io/projected/00dc2920-34f2-4e0a-892e-7759a16162d5-kube-api-access-j62gg\") pod \"multus-tb9hd\" (UID: \"00dc2920-34f2-4e0a-892e-7759a16162d5\") " pod="openshift-multus/multus-tb9hd" Nov 03 00:40:52.955385 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:40:52.954893 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/69b087f8-4a23-4ec8-808e-b61d2e16f67b-metrics-tls\") pod \"dns-default-266ht\" (UID: \"69b087f8-4a23-4ec8-808e-b61d2e16f67b\") " pod="openshift-dns/dns-default-266ht" Nov 03 00:40:52.955385 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:40:52.954937 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/98ca3972-fd0c-47f5-9d26-569a58e18bd1-host-kubelet\") pod \"ovnkube-node-psqpn\" (UID: \"98ca3972-fd0c-47f5-9d26-569a58e18bd1\") " pod="openshift-ovn-kubernetes/ovnkube-node-psqpn" Nov 03 00:40:52.955385 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:40:52.954972 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/98ca3972-fd0c-47f5-9d26-569a58e18bd1-host-kubelet\") pod \"ovnkube-node-psqpn\" (UID: \"98ca3972-fd0c-47f5-9d26-569a58e18bd1\") " pod="openshift-ovn-kubernetes/ovnkube-node-psqpn" Nov 03 00:40:52.955385 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:40:52.954999 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/98ca3972-fd0c-47f5-9d26-569a58e18bd1-var-lib-openvswitch\") pod \"ovnkube-node-psqpn\" (UID: \"98ca3972-fd0c-47f5-9d26-569a58e18bd1\") " pod="openshift-ovn-kubernetes/ovnkube-node-psqpn" Nov 03 00:40:52.955385 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:40:52.955042 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/98ca3972-fd0c-47f5-9d26-569a58e18bd1-var-lib-openvswitch\") pod \"ovnkube-node-psqpn\" (UID: \"98ca3972-fd0c-47f5-9d26-569a58e18bd1\") " pod="openshift-ovn-kubernetes/ovnkube-node-psqpn" Nov 03 00:40:52.955385 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:40:52.955066 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/37bf96f5-045f-49e8-83e8-410df66bfb1f-kubelet-dir\") pod \"aws-ebs-csi-driver-node-br8vt\" (UID: \"37bf96f5-045f-49e8-83e8-410df66bfb1f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-br8vt" Nov 03 00:40:52.955385 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:40:52.955087 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/00dc2920-34f2-4e0a-892e-7759a16162d5-host-var-lib-cni-multus\") pod \"multus-tb9hd\" (UID: \"00dc2920-34f2-4e0a-892e-7759a16162d5\") " pod="openshift-multus/multus-tb9hd" Nov 03 00:40:52.955385 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:40:52.955104 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/00dc2920-34f2-4e0a-892e-7759a16162d5-multus-daemon-config\") pod \"multus-tb9hd\" (UID: \"00dc2920-34f2-4e0a-892e-7759a16162d5\") " pod="openshift-multus/multus-tb9hd" Nov 03 00:40:52.955385 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:40:52.955121 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/52a19842-3750-4d2b-bcb2-aebddee81097-ca-trust-extracted\") pod \"image-registry-6669dc8887-ws544\" (UID: \"52a19842-3750-4d2b-bcb2-aebddee81097\") " pod="openshift-image-registry/image-registry-6669dc8887-ws544" Nov 03 00:40:52.955385 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:40:52.955160 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/00dc2920-34f2-4e0a-892e-7759a16162d5-host-run-multus-certs\") pod \"multus-tb9hd\" (UID: \"00dc2920-34f2-4e0a-892e-7759a16162d5\") " pod="openshift-multus/multus-tb9hd" Nov 03 00:40:52.955385 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:40:52.955180 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/69b087f8-4a23-4ec8-808e-b61d2e16f67b-tmp-dir\") pod \"dns-default-266ht\" (UID: \"69b087f8-4a23-4ec8-808e-b61d2e16f67b\") " pod="openshift-dns/dns-default-266ht" Nov 03 00:40:52.955385 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:40:52.955204 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/cabd402c-79c5-4699-9c85-d415d362b4d0-original-pull-secret\") pod \"global-pull-secret-syncer-jv7cw\" (UID: \"cabd402c-79c5-4699-9c85-d415d362b4d0\") " pod="kube-system/global-pull-secret-syncer-jv7cw" Nov 03 00:40:52.955385 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:40:52.955220 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/37bf96f5-045f-49e8-83e8-410df66bfb1f-sys-fs\") pod \"aws-ebs-csi-driver-node-br8vt\" (UID: \"37bf96f5-045f-49e8-83e8-410df66bfb1f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-br8vt" Nov 03 00:40:52.955385 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:40:52.955254 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/52a19842-3750-4d2b-bcb2-aebddee81097-ca-trust-extracted\") pod \"image-registry-6669dc8887-ws544\" (UID: \"52a19842-3750-4d2b-bcb2-aebddee81097\") " pod="openshift-image-registry/image-registry-6669dc8887-ws544" Nov 03 00:40:52.955385 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:40:52.955262 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/945da51a-32cb-4769-900d-1aeb266c7565-crio-socket\") pod \"insights-runtime-extractor-sthjh\" (UID: \"945da51a-32cb-4769-900d-1aeb266c7565\") " pod="openshift-insights/insights-runtime-extractor-sthjh" Nov 03 00:40:52.955385 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:40:52.955269 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/00dc2920-34f2-4e0a-892e-7759a16162d5-multus-daemon-config\") pod \"multus-tb9hd\" (UID: \"00dc2920-34f2-4e0a-892e-7759a16162d5\") " pod="openshift-multus/multus-tb9hd" Nov 03 00:40:52.955385 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:40:52.955296 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/69b087f8-4a23-4ec8-808e-b61d2e16f67b-tmp-dir\") pod \"dns-default-266ht\" (UID: \"69b087f8-4a23-4ec8-808e-b61d2e16f67b\") " pod="openshift-dns/dns-default-266ht" Nov 03 00:40:52.955385 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:40:52.955308 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/945da51a-32cb-4769-900d-1aeb266c7565-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-sthjh\" (UID: \"945da51a-32cb-4769-900d-1aeb266c7565\") " pod="openshift-insights/insights-runtime-extractor-sthjh" Nov 03 00:40:52.955385 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:40:52.955332 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/b748c4f8-d021-4d37-b390-d395482fcddd-etc-sysconfig\") pod \"tuned-gtwxp\" (UID: \"b748c4f8-d021-4d37-b390-d395482fcddd\") " pod="openshift-cluster-node-tuning-operator/tuned-gtwxp" Nov 03 00:40:52.955385 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:40:52.955374 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/b748c4f8-d021-4d37-b390-d395482fcddd-etc-sysconfig\") pod \"tuned-gtwxp\" (UID: \"b748c4f8-d021-4d37-b390-d395482fcddd\") " pod="openshift-cluster-node-tuning-operator/tuned-gtwxp" Nov 03 00:40:52.955385 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:40:52.955392 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cf217a21-c1bd-48be-b203-f59743bceb33-metrics-certs\") pod \"router-default-d7798ccf5-442kt\" (UID: \"cf217a21-c1bd-48be-b203-f59743bceb33\") " pod="openshift-ingress/router-default-d7798ccf5-442kt" Nov 03 00:40:52.955385 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:40:52.955412 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/98ca3972-fd0c-47f5-9d26-569a58e18bd1-host-run-netns\") pod \"ovnkube-node-psqpn\" (UID: \"98ca3972-fd0c-47f5-9d26-569a58e18bd1\") " pod="openshift-ovn-kubernetes/ovnkube-node-psqpn" Nov 03 00:40:52.956573 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:40:52.955438 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lp26w\" (UniqueName: \"kubernetes.io/projected/37bf96f5-045f-49e8-83e8-410df66bfb1f-kube-api-access-lp26w\") pod \"aws-ebs-csi-driver-node-br8vt\" (UID: \"37bf96f5-045f-49e8-83e8-410df66bfb1f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-br8vt" Nov 03 00:40:52.956573 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:40:52.955461 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/b748c4f8-d021-4d37-b390-d395482fcddd-host\") pod \"tuned-gtwxp\" (UID: \"b748c4f8-d021-4d37-b390-d395482fcddd\") " pod="openshift-cluster-node-tuning-operator/tuned-gtwxp" Nov 03 00:40:52.956573 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:40:52.955465 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/98ca3972-fd0c-47f5-9d26-569a58e18bd1-host-run-netns\") pod \"ovnkube-node-psqpn\" (UID: \"98ca3972-fd0c-47f5-9d26-569a58e18bd1\") " pod="openshift-ovn-kubernetes/ovnkube-node-psqpn" Nov 03 00:40:52.956573 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:40:52.955485 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/98ca3972-fd0c-47f5-9d26-569a58e18bd1-host-cni-netd\") pod \"ovnkube-node-psqpn\" (UID: \"98ca3972-fd0c-47f5-9d26-569a58e18bd1\") " pod="openshift-ovn-kubernetes/ovnkube-node-psqpn" Nov 03 00:40:52.956573 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:40:52.955511 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/52a19842-3750-4d2b-bcb2-aebddee81097-image-registry-private-configuration\") pod \"image-registry-6669dc8887-ws544\" (UID: \"52a19842-3750-4d2b-bcb2-aebddee81097\") " pod="openshift-image-registry/image-registry-6669dc8887-ws544" Nov 03 00:40:52.956573 ip-10-0-140-107 kubenswrapper[4939]: E1103 00:40:52.955521 4939 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Nov 03 00:40:52.956573 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:40:52.955527 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/75481915-3cbe-475b-abe7-cb4e81c93427-cert\") pod \"ingress-canary-s9hbz\" (UID: \"75481915-3cbe-475b-abe7-cb4e81c93427\") " pod="openshift-ingress-canary/ingress-canary-s9hbz" Nov 03 00:40:52.956573 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:40:52.955528 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/b748c4f8-d021-4d37-b390-d395482fcddd-host\") pod \"tuned-gtwxp\" (UID: \"b748c4f8-d021-4d37-b390-d395482fcddd\") " pod="openshift-cluster-node-tuning-operator/tuned-gtwxp" Nov 03 00:40:52.956573 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:40:52.955540 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/b748c4f8-d021-4d37-b390-d395482fcddd-lib-modules\") pod \"tuned-gtwxp\" (UID: \"b748c4f8-d021-4d37-b390-d395482fcddd\") " pod="openshift-cluster-node-tuning-operator/tuned-gtwxp" Nov 03 00:40:52.956573 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:40:52.955539 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/cabd402c-79c5-4699-9c85-d415d362b4d0-original-pull-secret\") pod \"global-pull-secret-syncer-jv7cw\" (UID: \"cabd402c-79c5-4699-9c85-d415d362b4d0\") " pod="kube-system/global-pull-secret-syncer-jv7cw" Nov 03 00:40:52.956573 ip-10-0-140-107 kubenswrapper[4939]: E1103 00:40:52.955553 4939 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cf217a21-c1bd-48be-b203-f59743bceb33-metrics-certs podName:cf217a21-c1bd-48be-b203-f59743bceb33 nodeName:}" failed. No retries permitted until 2026-11-03 00:40:53.455544166 +0000 UTC m=+1.719879123 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/cf217a21-c1bd-48be-b203-f59743bceb33-metrics-certs") pod "router-default-d7798ccf5-442kt" (UID: "cf217a21-c1bd-48be-b203-f59743bceb33") : secret "router-metrics-certs-default" not found Nov 03 00:40:52.956573 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:40:52.955570 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/98ca3972-fd0c-47f5-9d26-569a58e18bd1-host-cni-netd\") pod \"ovnkube-node-psqpn\" (UID: \"98ca3972-fd0c-47f5-9d26-569a58e18bd1\") " pod="openshift-ovn-kubernetes/ovnkube-node-psqpn" Nov 03 00:40:52.956573 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:40:52.955584 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/b748c4f8-d021-4d37-b390-d395482fcddd-etc-kubernetes\") pod \"tuned-gtwxp\" (UID: \"b748c4f8-d021-4d37-b390-d395482fcddd\") " pod="openshift-cluster-node-tuning-operator/tuned-gtwxp" Nov 03 00:40:52.956573 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:40:52.955644 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/b748c4f8-d021-4d37-b390-d395482fcddd-lib-modules\") pod \"tuned-gtwxp\" (UID: \"b748c4f8-d021-4d37-b390-d395482fcddd\") " pod="openshift-cluster-node-tuning-operator/tuned-gtwxp" Nov 03 00:40:52.956573 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:40:52.955661 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/b748c4f8-d021-4d37-b390-d395482fcddd-etc-kubernetes\") pod \"tuned-gtwxp\" (UID: \"b748c4f8-d021-4d37-b390-d395482fcddd\") " pod="openshift-cluster-node-tuning-operator/tuned-gtwxp" Nov 03 00:40:52.956573 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:40:52.955718 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/52a19842-3750-4d2b-bcb2-aebddee81097-image-registry-private-configuration\") pod \"image-registry-6669dc8887-ws544\" (UID: \"52a19842-3750-4d2b-bcb2-aebddee81097\") " pod="openshift-image-registry/image-registry-6669dc8887-ws544" Nov 03 00:40:52.956573 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:40:52.955729 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/75481915-3cbe-475b-abe7-cb4e81c93427-cert\") pod \"ingress-canary-s9hbz\" (UID: \"75481915-3cbe-475b-abe7-cb4e81c93427\") " pod="openshift-ingress-canary/ingress-canary-s9hbz" Nov 03 00:40:52.962992 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:40:52.962963 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 00:40:52.965390 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:40:52.965368 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7wzbd\" (UniqueName: \"kubernetes.io/projected/68658033-8a49-4430-869f-a7eb79603484-kube-api-access-7wzbd\") pod \"migrator-66df5d496b-m7cxm\" (UID: \"68658033-8a49-4430-869f-a7eb79603484\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-m7cxm" Nov 03 00:40:52.965631 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:40:52.965616 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tc9rq\" (UniqueName: \"kubernetes.io/projected/52a19842-3750-4d2b-bcb2-aebddee81097-kube-api-access-tc9rq\") pod \"image-registry-6669dc8887-ws544\" (UID: \"52a19842-3750-4d2b-bcb2-aebddee81097\") " pod="openshift-image-registry/image-registry-6669dc8887-ws544" Nov 03 00:40:52.965783 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:40:52.965762 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sfdp9\" (UniqueName: \"kubernetes.io/projected/ad5bf442-729d-411c-96e4-2e5105ebadcf-kube-api-access-sfdp9\") pod \"node-resolver-cjj2h\" (UID: \"ad5bf442-729d-411c-96e4-2e5105ebadcf\") " pod="openshift-dns/node-resolver-cjj2h" Nov 03 00:40:52.965977 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:40:52.965962 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pj6ft\" (UniqueName: \"kubernetes.io/projected/69b087f8-4a23-4ec8-808e-b61d2e16f67b-kube-api-access-pj6ft\") pod \"dns-default-266ht\" (UID: \"69b087f8-4a23-4ec8-808e-b61d2e16f67b\") " pod="openshift-dns/dns-default-266ht" Nov 03 00:40:52.966380 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:40:52.966360 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/52a19842-3750-4d2b-bcb2-aebddee81097-bound-sa-token\") pod \"image-registry-6669dc8887-ws544\" (UID: \"52a19842-3750-4d2b-bcb2-aebddee81097\") " pod="openshift-image-registry/image-registry-6669dc8887-ws544" Nov 03 00:40:52.966730 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:40:52.966710 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qmkgl\" (UniqueName: \"kubernetes.io/projected/75481915-3cbe-475b-abe7-cb4e81c93427-kube-api-access-qmkgl\") pod \"ingress-canary-s9hbz\" (UID: \"75481915-3cbe-475b-abe7-cb4e81c93427\") " pod="openshift-ingress-canary/ingress-canary-s9hbz" Nov 03 00:40:52.966899 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:40:52.966879 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4q9wl\" (UniqueName: \"kubernetes.io/projected/945da51a-32cb-4769-900d-1aeb266c7565-kube-api-access-4q9wl\") pod \"insights-runtime-extractor-sthjh\" (UID: \"945da51a-32cb-4769-900d-1aeb266c7565\") " pod="openshift-insights/insights-runtime-extractor-sthjh" Nov 03 00:40:52.966974 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:40:52.966923 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-j62gg\" (UniqueName: \"kubernetes.io/projected/00dc2920-34f2-4e0a-892e-7759a16162d5-kube-api-access-j62gg\") pod \"multus-tb9hd\" (UID: \"00dc2920-34f2-4e0a-892e-7759a16162d5\") " pod="openshift-multus/multus-tb9hd" Nov 03 00:40:52.966974 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:40:52.966935 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lp26w\" (UniqueName: \"kubernetes.io/projected/37bf96f5-045f-49e8-83e8-410df66bfb1f-kube-api-access-lp26w\") pod \"aws-ebs-csi-driver-node-br8vt\" (UID: \"37bf96f5-045f-49e8-83e8-410df66bfb1f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-br8vt" Nov 03 00:40:53.056925 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:40:53.056878 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/cabd402c-79c5-4699-9c85-d415d362b4d0-kubelet-config\") pod \"global-pull-secret-syncer-jv7cw\" (UID: \"cabd402c-79c5-4699-9c85-d415d362b4d0\") " pod="kube-system/global-pull-secret-syncer-jv7cw" Nov 03 00:40:53.056925 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:40:53.056922 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/52a19842-3750-4d2b-bcb2-aebddee81097-registry-tls\") pod \"image-registry-6669dc8887-ws544\" (UID: \"52a19842-3750-4d2b-bcb2-aebddee81097\") " pod="openshift-image-registry/image-registry-6669dc8887-ws544" Nov 03 00:40:53.057126 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:40:53.056945 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/37bf96f5-045f-49e8-83e8-410df66bfb1f-socket-dir\") pod \"aws-ebs-csi-driver-node-br8vt\" (UID: \"37bf96f5-045f-49e8-83e8-410df66bfb1f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-br8vt" Nov 03 00:40:53.057126 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:40:53.056969 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/945da51a-32cb-4769-900d-1aeb266c7565-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-sthjh\" (UID: \"945da51a-32cb-4769-900d-1aeb266c7565\") " pod="openshift-insights/insights-runtime-extractor-sthjh" Nov 03 00:40:53.057126 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:40:53.057005 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/cabd402c-79c5-4699-9c85-d415d362b4d0-kubelet-config\") pod \"global-pull-secret-syncer-jv7cw\" (UID: \"cabd402c-79c5-4699-9c85-d415d362b4d0\") " pod="kube-system/global-pull-secret-syncer-jv7cw" Nov 03 00:40:53.057126 ip-10-0-140-107 kubenswrapper[4939]: E1103 00:40:53.057035 4939 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Nov 03 00:40:53.057126 ip-10-0-140-107 kubenswrapper[4939]: E1103 00:40:53.057051 4939 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-6669dc8887-ws544: secret "image-registry-tls" not found Nov 03 00:40:53.057126 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:40:53.057078 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/00dc2920-34f2-4e0a-892e-7759a16162d5-host-var-lib-kubelet\") pod \"multus-tb9hd\" (UID: \"00dc2920-34f2-4e0a-892e-7759a16162d5\") " pod="openshift-multus/multus-tb9hd" Nov 03 00:40:53.057126 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:40:53.057032 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/37bf96f5-045f-49e8-83e8-410df66bfb1f-socket-dir\") pod \"aws-ebs-csi-driver-node-br8vt\" (UID: \"37bf96f5-045f-49e8-83e8-410df66bfb1f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-br8vt" Nov 03 00:40:53.057126 ip-10-0-140-107 kubenswrapper[4939]: E1103 00:40:53.057097 4939 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/52a19842-3750-4d2b-bcb2-aebddee81097-registry-tls podName:52a19842-3750-4d2b-bcb2-aebddee81097 nodeName:}" failed. No retries permitted until 2026-11-03 00:40:53.557082269 +0000 UTC m=+1.821417207 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/52a19842-3750-4d2b-bcb2-aebddee81097-registry-tls") pod "image-registry-6669dc8887-ws544" (UID: "52a19842-3750-4d2b-bcb2-aebddee81097") : secret "image-registry-tls" not found Nov 03 00:40:53.057126 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:40:53.057037 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/00dc2920-34f2-4e0a-892e-7759a16162d5-host-var-lib-kubelet\") pod \"multus-tb9hd\" (UID: \"00dc2920-34f2-4e0a-892e-7759a16162d5\") " pod="openshift-multus/multus-tb9hd" Nov 03 00:40:53.057549 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:40:53.057147 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/00dc2920-34f2-4e0a-892e-7759a16162d5-system-cni-dir\") pod \"multus-tb9hd\" (UID: \"00dc2920-34f2-4e0a-892e-7759a16162d5\") " pod="openshift-multus/multus-tb9hd" Nov 03 00:40:53.057549 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:40:53.057189 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/00dc2920-34f2-4e0a-892e-7759a16162d5-multus-cni-dir\") pod \"multus-tb9hd\" (UID: \"00dc2920-34f2-4e0a-892e-7759a16162d5\") " pod="openshift-multus/multus-tb9hd" Nov 03 00:40:53.057549 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:40:53.057235 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/00dc2920-34f2-4e0a-892e-7759a16162d5-host-run-k8s-cni-cncf-io\") pod \"multus-tb9hd\" (UID: \"00dc2920-34f2-4e0a-892e-7759a16162d5\") " pod="openshift-multus/multus-tb9hd" Nov 03 00:40:53.057549 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:40:53.057255 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/00dc2920-34f2-4e0a-892e-7759a16162d5-system-cni-dir\") pod \"multus-tb9hd\" (UID: \"00dc2920-34f2-4e0a-892e-7759a16162d5\") " pod="openshift-multus/multus-tb9hd" Nov 03 00:40:53.057549 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:40:53.057278 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/69b087f8-4a23-4ec8-808e-b61d2e16f67b-metrics-tls\") pod \"dns-default-266ht\" (UID: \"69b087f8-4a23-4ec8-808e-b61d2e16f67b\") " pod="openshift-dns/dns-default-266ht" Nov 03 00:40:53.057549 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:40:53.057284 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/00dc2920-34f2-4e0a-892e-7759a16162d5-multus-cni-dir\") pod \"multus-tb9hd\" (UID: \"00dc2920-34f2-4e0a-892e-7759a16162d5\") " pod="openshift-multus/multus-tb9hd" Nov 03 00:40:53.057549 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:40:53.057289 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/00dc2920-34f2-4e0a-892e-7759a16162d5-host-run-k8s-cni-cncf-io\") pod \"multus-tb9hd\" (UID: \"00dc2920-34f2-4e0a-892e-7759a16162d5\") " pod="openshift-multus/multus-tb9hd" Nov 03 00:40:53.057549 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:40:53.057320 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/37bf96f5-045f-49e8-83e8-410df66bfb1f-kubelet-dir\") pod \"aws-ebs-csi-driver-node-br8vt\" (UID: \"37bf96f5-045f-49e8-83e8-410df66bfb1f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-br8vt" Nov 03 00:40:53.057549 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:40:53.057357 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/00dc2920-34f2-4e0a-892e-7759a16162d5-host-var-lib-cni-multus\") pod \"multus-tb9hd\" (UID: \"00dc2920-34f2-4e0a-892e-7759a16162d5\") " pod="openshift-multus/multus-tb9hd" Nov 03 00:40:53.057549 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:40:53.057384 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/00dc2920-34f2-4e0a-892e-7759a16162d5-host-run-multus-certs\") pod \"multus-tb9hd\" (UID: \"00dc2920-34f2-4e0a-892e-7759a16162d5\") " pod="openshift-multus/multus-tb9hd" Nov 03 00:40:53.057549 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:40:53.057397 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/37bf96f5-045f-49e8-83e8-410df66bfb1f-kubelet-dir\") pod \"aws-ebs-csi-driver-node-br8vt\" (UID: \"37bf96f5-045f-49e8-83e8-410df66bfb1f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-br8vt" Nov 03 00:40:53.057549 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:40:53.057417 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/37bf96f5-045f-49e8-83e8-410df66bfb1f-sys-fs\") pod \"aws-ebs-csi-driver-node-br8vt\" (UID: \"37bf96f5-045f-49e8-83e8-410df66bfb1f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-br8vt" Nov 03 00:40:53.057549 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:40:53.057439 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/00dc2920-34f2-4e0a-892e-7759a16162d5-host-var-lib-cni-multus\") pod \"multus-tb9hd\" (UID: \"00dc2920-34f2-4e0a-892e-7759a16162d5\") " pod="openshift-multus/multus-tb9hd" Nov 03 00:40:53.057549 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:40:53.057445 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/945da51a-32cb-4769-900d-1aeb266c7565-crio-socket\") pod \"insights-runtime-extractor-sthjh\" (UID: \"945da51a-32cb-4769-900d-1aeb266c7565\") " pod="openshift-insights/insights-runtime-extractor-sthjh" Nov 03 00:40:53.057549 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:40:53.057465 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/00dc2920-34f2-4e0a-892e-7759a16162d5-host-run-multus-certs\") pod \"multus-tb9hd\" (UID: \"00dc2920-34f2-4e0a-892e-7759a16162d5\") " pod="openshift-multus/multus-tb9hd" Nov 03 00:40:53.057549 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:40:53.057470 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/945da51a-32cb-4769-900d-1aeb266c7565-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-sthjh\" (UID: \"945da51a-32cb-4769-900d-1aeb266c7565\") " pod="openshift-insights/insights-runtime-extractor-sthjh" Nov 03 00:40:53.057549 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:40:53.057498 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/37bf96f5-045f-49e8-83e8-410df66bfb1f-sys-fs\") pod \"aws-ebs-csi-driver-node-br8vt\" (UID: \"37bf96f5-045f-49e8-83e8-410df66bfb1f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-br8vt" Nov 03 00:40:53.057549 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:40:53.057501 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/945da51a-32cb-4769-900d-1aeb266c7565-crio-socket\") pod \"insights-runtime-extractor-sthjh\" (UID: \"945da51a-32cb-4769-900d-1aeb266c7565\") " pod="openshift-insights/insights-runtime-extractor-sthjh" Nov 03 00:40:53.057549 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:40:53.057519 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/945da51a-32cb-4769-900d-1aeb266c7565-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-sthjh\" (UID: \"945da51a-32cb-4769-900d-1aeb266c7565\") " pod="openshift-insights/insights-runtime-extractor-sthjh" Nov 03 00:40:53.070175 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:40:53.057580 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/00dc2920-34f2-4e0a-892e-7759a16162d5-hostroot\") pod \"multus-tb9hd\" (UID: \"00dc2920-34f2-4e0a-892e-7759a16162d5\") " pod="openshift-multus/multus-tb9hd" Nov 03 00:40:53.070175 ip-10-0-140-107 kubenswrapper[4939]: E1103 00:40:53.057611 4939 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Nov 03 00:40:53.070175 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:40:53.057630 4939 swap_util.go:78] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory" Nov 03 00:40:53.070175 ip-10-0-140-107 kubenswrapper[4939]: E1103 00:40:53.057661 4939 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/945da51a-32cb-4769-900d-1aeb266c7565-insights-runtime-extractor-tls podName:945da51a-32cb-4769-900d-1aeb266c7565 nodeName:}" failed. No retries permitted until 2026-11-03 00:40:53.55764618 +0000 UTC m=+1.821981127 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/945da51a-32cb-4769-900d-1aeb266c7565-insights-runtime-extractor-tls") pod "insights-runtime-extractor-sthjh" (UID: "945da51a-32cb-4769-900d-1aeb266c7565") : secret "insights-runtime-extractor-tls" not found Nov 03 00:40:53.070175 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:40:53.057659 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/00dc2920-34f2-4e0a-892e-7759a16162d5-hostroot\") pod \"multus-tb9hd\" (UID: \"00dc2920-34f2-4e0a-892e-7759a16162d5\") " pod="openshift-multus/multus-tb9hd" Nov 03 00:40:53.070175 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:40:53.057667 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/00dc2920-34f2-4e0a-892e-7759a16162d5-multus-conf-dir\") pod \"multus-tb9hd\" (UID: \"00dc2920-34f2-4e0a-892e-7759a16162d5\") " pod="openshift-multus/multus-tb9hd" Nov 03 00:40:53.070175 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:40:53.057635 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/00dc2920-34f2-4e0a-892e-7759a16162d5-multus-conf-dir\") pod \"multus-tb9hd\" (UID: \"00dc2920-34f2-4e0a-892e-7759a16162d5\") " pod="openshift-multus/multus-tb9hd" Nov 03 00:40:53.070175 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:40:53.057733 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/cabd402c-79c5-4699-9c85-d415d362b4d0-dbus\") pod \"global-pull-secret-syncer-jv7cw\" (UID: \"cabd402c-79c5-4699-9c85-d415d362b4d0\") " pod="kube-system/global-pull-secret-syncer-jv7cw" Nov 03 00:40:53.070175 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:40:53.057761 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/37bf96f5-045f-49e8-83e8-410df66bfb1f-registration-dir\") pod \"aws-ebs-csi-driver-node-br8vt\" (UID: \"37bf96f5-045f-49e8-83e8-410df66bfb1f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-br8vt" Nov 03 00:40:53.070175 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:40:53.057770 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/cabd402c-79c5-4699-9c85-d415d362b4d0-dbus\") pod \"global-pull-secret-syncer-jv7cw\" (UID: \"cabd402c-79c5-4699-9c85-d415d362b4d0\") " pod="kube-system/global-pull-secret-syncer-jv7cw" Nov 03 00:40:53.070175 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:40:53.057783 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/00dc2920-34f2-4e0a-892e-7759a16162d5-multus-socket-dir-parent\") pod \"multus-tb9hd\" (UID: \"00dc2920-34f2-4e0a-892e-7759a16162d5\") " pod="openshift-multus/multus-tb9hd" Nov 03 00:40:53.070175 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:40:53.057805 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/00dc2920-34f2-4e0a-892e-7759a16162d5-host-var-lib-cni-bin\") pod \"multus-tb9hd\" (UID: \"00dc2920-34f2-4e0a-892e-7759a16162d5\") " pod="openshift-multus/multus-tb9hd" Nov 03 00:40:53.070175 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:40:53.057814 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/37bf96f5-045f-49e8-83e8-410df66bfb1f-registration-dir\") pod \"aws-ebs-csi-driver-node-br8vt\" (UID: \"37bf96f5-045f-49e8-83e8-410df66bfb1f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-br8vt" Nov 03 00:40:53.070175 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:40:53.057820 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/00dc2920-34f2-4e0a-892e-7759a16162d5-etc-kubernetes\") pod \"multus-tb9hd\" (UID: \"00dc2920-34f2-4e0a-892e-7759a16162d5\") " pod="openshift-multus/multus-tb9hd" Nov 03 00:40:53.070175 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:40:53.057841 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/37bf96f5-045f-49e8-83e8-410df66bfb1f-device-dir\") pod \"aws-ebs-csi-driver-node-br8vt\" (UID: \"37bf96f5-045f-49e8-83e8-410df66bfb1f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-br8vt" Nov 03 00:40:53.070175 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:40:53.057850 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/00dc2920-34f2-4e0a-892e-7759a16162d5-multus-socket-dir-parent\") pod \"multus-tb9hd\" (UID: \"00dc2920-34f2-4e0a-892e-7759a16162d5\") " pod="openshift-multus/multus-tb9hd" Nov 03 00:40:53.070175 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:40:53.057857 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/37bf96f5-045f-49e8-83e8-410df66bfb1f-etc-selinux\") pod \"aws-ebs-csi-driver-node-br8vt\" (UID: \"37bf96f5-045f-49e8-83e8-410df66bfb1f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-br8vt" Nov 03 00:40:53.070175 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:40:53.057899 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/37bf96f5-045f-49e8-83e8-410df66bfb1f-device-dir\") pod \"aws-ebs-csi-driver-node-br8vt\" (UID: \"37bf96f5-045f-49e8-83e8-410df66bfb1f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-br8vt" Nov 03 00:40:53.070175 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:40:53.057902 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/37bf96f5-045f-49e8-83e8-410df66bfb1f-etc-selinux\") pod \"aws-ebs-csi-driver-node-br8vt\" (UID: \"37bf96f5-045f-49e8-83e8-410df66bfb1f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-br8vt" Nov 03 00:40:53.070175 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:40:53.057904 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/00dc2920-34f2-4e0a-892e-7759a16162d5-host-var-lib-cni-bin\") pod \"multus-tb9hd\" (UID: \"00dc2920-34f2-4e0a-892e-7759a16162d5\") " pod="openshift-multus/multus-tb9hd" Nov 03 00:40:53.070175 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:40:53.057900 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/00dc2920-34f2-4e0a-892e-7759a16162d5-etc-kubernetes\") pod \"multus-tb9hd\" (UID: \"00dc2920-34f2-4e0a-892e-7759a16162d5\") " pod="openshift-multus/multus-tb9hd" Nov 03 00:40:53.070175 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:40:53.057948 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/ad5bf442-729d-411c-96e4-2e5105ebadcf-hosts-file\") pod \"node-resolver-cjj2h\" (UID: \"ad5bf442-729d-411c-96e4-2e5105ebadcf\") " pod="openshift-dns/node-resolver-cjj2h" Nov 03 00:40:53.070175 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:40:53.057972 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/00dc2920-34f2-4e0a-892e-7759a16162d5-cnibin\") pod \"multus-tb9hd\" (UID: \"00dc2920-34f2-4e0a-892e-7759a16162d5\") " pod="openshift-multus/multus-tb9hd" Nov 03 00:40:53.070175 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:40:53.057995 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/00dc2920-34f2-4e0a-892e-7759a16162d5-host-run-netns\") pod \"multus-tb9hd\" (UID: \"00dc2920-34f2-4e0a-892e-7759a16162d5\") " pod="openshift-multus/multus-tb9hd" Nov 03 00:40:53.070175 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:40:53.058025 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/ad5bf442-729d-411c-96e4-2e5105ebadcf-hosts-file\") pod \"node-resolver-cjj2h\" (UID: \"ad5bf442-729d-411c-96e4-2e5105ebadcf\") " pod="openshift-dns/node-resolver-cjj2h" Nov 03 00:40:53.070175 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:40:53.058040 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/00dc2920-34f2-4e0a-892e-7759a16162d5-cnibin\") pod \"multus-tb9hd\" (UID: \"00dc2920-34f2-4e0a-892e-7759a16162d5\") " pod="openshift-multus/multus-tb9hd" Nov 03 00:40:53.070175 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:40:53.058063 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/00dc2920-34f2-4e0a-892e-7759a16162d5-host-run-netns\") pod \"multus-tb9hd\" (UID: \"00dc2920-34f2-4e0a-892e-7759a16162d5\") " pod="openshift-multus/multus-tb9hd" Nov 03 00:40:53.070175 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:40:53.059634 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/69b087f8-4a23-4ec8-808e-b61d2e16f67b-metrics-tls\") pod \"dns-default-266ht\" (UID: \"69b087f8-4a23-4ec8-808e-b61d2e16f67b\") " pod="openshift-dns/dns-default-266ht" Nov 03 00:40:53.127805 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:40:53.127776 4939 scope.go:122] "RemoveContainer" containerID="8fbbed43b6d1c11a77523d9a59cdf45ffb9e7d1661d1a2290ee4a1ddd7dc0fb7" Nov 03 00:40:53.143709 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:40:53.143657 4939 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-266ht" Nov 03 00:40:53.301422 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:40:53.301394 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 00:40:53.384319 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:40:53.384292 4939 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-cjj2h_ad5bf442-729d-411c-96e4-2e5105ebadcf/dns-node-resolver/0.log" Nov 03 00:40:53.450425 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:40:53.450399 4939 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-266ht"] Nov 03 00:40:53.453624 ip-10-0-140-107 kubenswrapper[4939]: W1103 00:40:53.453594 4939 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod69b087f8_4a23_4ec8_808e_b61d2e16f67b.slice/crio-629ff02ec6e47a4ec0e1bbc446e7778640f8368b14000bd4fc641a2512acd529 WatchSource:0}: Error finding container 629ff02ec6e47a4ec0e1bbc446e7778640f8368b14000bd4fc641a2512acd529: Status 404 returned error can't find the container with id 629ff02ec6e47a4ec0e1bbc446e7778640f8368b14000bd4fc641a2512acd529 Nov 03 00:40:53.455596 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:40:53.455582 4939 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Nov 03 00:40:53.461729 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:40:53.461704 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/136086c0-4846-437a-ad98-3cfef50610a0-metrics-certs\") pod \"network-metrics-daemon-rt2hx\" (UID: \"136086c0-4846-437a-ad98-3cfef50610a0\") " pod="openshift-multus/network-metrics-daemon-rt2hx" Nov 03 00:40:53.461799 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:40:53.461757 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/06973556-f51c-4d1c-ae55-fb77699bc5b3-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-4bm4f\" (UID: \"06973556-f51c-4d1c-ae55-fb77699bc5b3\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-4bm4f" Nov 03 00:40:53.461799 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:40:53.461794 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cf217a21-c1bd-48be-b203-f59743bceb33-metrics-certs\") pod \"router-default-d7798ccf5-442kt\" (UID: \"cf217a21-c1bd-48be-b203-f59743bceb33\") " pod="openshift-ingress/router-default-d7798ccf5-442kt" Nov 03 00:40:53.461859 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:40:53.461816 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/dd0cd8a1-4d6b-424c-ac39-5f5ddb9cd87e-registry-tls\") pod \"image-registry-65f77c855d-6tnc2\" (UID: \"dd0cd8a1-4d6b-424c-ac39-5f5ddb9cd87e\") " pod="openshift-image-registry/image-registry-65f77c855d-6tnc2" Nov 03 00:40:53.461859 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:40:53.461835 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/6eafe6f6-cf90-446f-a8d5-620cd2e02775-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-2ngwx\" (UID: \"6eafe6f6-cf90-446f-a8d5-620cd2e02775\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-2ngwx" Nov 03 00:40:53.461974 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:40:53.461867 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/bc513cba-d924-4719-a2ba-711529175ee0-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-z2qgj\" (UID: \"bc513cba-d924-4719-a2ba-711529175ee0\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-z2qgj" Nov 03 00:40:53.461974 ip-10-0-140-107 kubenswrapper[4939]: E1103 00:40:53.461896 4939 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Nov 03 00:40:53.461974 ip-10-0-140-107 kubenswrapper[4939]: E1103 00:40:53.461932 4939 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Nov 03 00:40:53.461974 ip-10-0-140-107 kubenswrapper[4939]: E1103 00:40:53.461967 4939 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Nov 03 00:40:53.462109 ip-10-0-140-107 kubenswrapper[4939]: E1103 00:40:53.461979 4939 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Nov 03 00:40:53.462109 ip-10-0-140-107 kubenswrapper[4939]: E1103 00:40:53.461993 4939 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Nov 03 00:40:53.462109 ip-10-0-140-107 kubenswrapper[4939]: E1103 00:40:53.461987 4939 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-65f77c855d-6tnc2: secret "image-registry-tls" not found Nov 03 00:40:53.462109 ip-10-0-140-107 kubenswrapper[4939]: E1103 00:40:53.461976 4939 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/06973556-f51c-4d1c-ae55-fb77699bc5b3-cluster-monitoring-operator-tls podName:06973556-f51c-4d1c-ae55-fb77699bc5b3 nodeName:}" failed. No retries permitted until 2026-11-03 00:40:54.461957058 +0000 UTC m=+2.726291981 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/06973556-f51c-4d1c-ae55-fb77699bc5b3-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-884576c57-4bm4f" (UID: "06973556-f51c-4d1c-ae55-fb77699bc5b3") : secret "cluster-monitoring-operator-tls" not found Nov 03 00:40:53.462109 ip-10-0-140-107 kubenswrapper[4939]: E1103 00:40:53.462066 4939 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cf217a21-c1bd-48be-b203-f59743bceb33-metrics-certs podName:cf217a21-c1bd-48be-b203-f59743bceb33 nodeName:}" failed. No retries permitted until 2026-11-03 00:40:54.462053842 +0000 UTC m=+2.726388765 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/cf217a21-c1bd-48be-b203-f59743bceb33-metrics-certs") pod "router-default-d7798ccf5-442kt" (UID: "cf217a21-c1bd-48be-b203-f59743bceb33") : secret "router-metrics-certs-default" not found Nov 03 00:40:53.462109 ip-10-0-140-107 kubenswrapper[4939]: E1103 00:40:53.462076 4939 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6eafe6f6-cf90-446f-a8d5-620cd2e02775-samples-operator-tls podName:6eafe6f6-cf90-446f-a8d5-620cd2e02775 nodeName:}" failed. No retries permitted until 2026-11-03 00:40:54.462070588 +0000 UTC m=+2.726405512 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/6eafe6f6-cf90-446f-a8d5-620cd2e02775-samples-operator-tls") pod "cluster-samples-operator-6ddb6c8b94-2ngwx" (UID: "6eafe6f6-cf90-446f-a8d5-620cd2e02775") : secret "samples-operator-tls" not found Nov 03 00:40:53.462109 ip-10-0-140-107 kubenswrapper[4939]: E1103 00:40:53.462087 4939 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/dd0cd8a1-4d6b-424c-ac39-5f5ddb9cd87e-registry-tls podName:dd0cd8a1-4d6b-424c-ac39-5f5ddb9cd87e nodeName:}" failed. No retries permitted until 2026-11-03 00:40:54.462080483 +0000 UTC m=+2.726415406 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/dd0cd8a1-4d6b-424c-ac39-5f5ddb9cd87e-registry-tls") pod "image-registry-65f77c855d-6tnc2" (UID: "dd0cd8a1-4d6b-424c-ac39-5f5ddb9cd87e") : secret "image-registry-tls" not found Nov 03 00:40:53.462109 ip-10-0-140-107 kubenswrapper[4939]: E1103 00:40:53.462099 4939 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/bc513cba-d924-4719-a2ba-711529175ee0-networking-console-plugin-cert podName:bc513cba-d924-4719-a2ba-711529175ee0 nodeName:}" failed. No retries permitted until 2026-11-03 00:40:54.462093597 +0000 UTC m=+2.726428525 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/bc513cba-d924-4719-a2ba-711529175ee0-networking-console-plugin-cert") pod "networking-console-plugin-86c478bb88-z2qgj" (UID: "bc513cba-d924-4719-a2ba-711529175ee0") : secret "networking-console-plugin-cert" not found Nov 03 00:40:53.463888 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:40:53.463871 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/136086c0-4846-437a-ad98-3cfef50610a0-metrics-certs\") pod \"network-metrics-daemon-rt2hx\" (UID: \"136086c0-4846-437a-ad98-3cfef50610a0\") " pod="openshift-multus/network-metrics-daemon-rt2hx" Nov 03 00:40:53.563057 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:40:53.563023 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/52a19842-3750-4d2b-bcb2-aebddee81097-registry-tls\") pod \"image-registry-6669dc8887-ws544\" (UID: \"52a19842-3750-4d2b-bcb2-aebddee81097\") " pod="openshift-image-registry/image-registry-6669dc8887-ws544" Nov 03 00:40:53.563201 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:40:53.563093 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/945da51a-32cb-4769-900d-1aeb266c7565-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-sthjh\" (UID: \"945da51a-32cb-4769-900d-1aeb266c7565\") " pod="openshift-insights/insights-runtime-extractor-sthjh" Nov 03 00:40:53.563201 ip-10-0-140-107 kubenswrapper[4939]: E1103 00:40:53.563167 4939 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Nov 03 00:40:53.563201 ip-10-0-140-107 kubenswrapper[4939]: E1103 00:40:53.563183 4939 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-6669dc8887-ws544: secret "image-registry-tls" not found Nov 03 00:40:53.563201 ip-10-0-140-107 kubenswrapper[4939]: E1103 00:40:53.563183 4939 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Nov 03 00:40:53.563320 ip-10-0-140-107 kubenswrapper[4939]: E1103 00:40:53.563236 4939 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/52a19842-3750-4d2b-bcb2-aebddee81097-registry-tls podName:52a19842-3750-4d2b-bcb2-aebddee81097 nodeName:}" failed. No retries permitted until 2026-11-03 00:40:54.563219171 +0000 UTC m=+2.827554114 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/52a19842-3750-4d2b-bcb2-aebddee81097-registry-tls") pod "image-registry-6669dc8887-ws544" (UID: "52a19842-3750-4d2b-bcb2-aebddee81097") : secret "image-registry-tls" not found Nov 03 00:40:53.563320 ip-10-0-140-107 kubenswrapper[4939]: E1103 00:40:53.563250 4939 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/945da51a-32cb-4769-900d-1aeb266c7565-insights-runtime-extractor-tls podName:945da51a-32cb-4769-900d-1aeb266c7565 nodeName:}" failed. No retries permitted until 2026-11-03 00:40:54.563244256 +0000 UTC m=+2.827579179 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/945da51a-32cb-4769-900d-1aeb266c7565-insights-runtime-extractor-tls") pod "insights-runtime-extractor-sthjh" (UID: "945da51a-32cb-4769-900d-1aeb266c7565") : secret "insights-runtime-extractor-tls" not found Nov 03 00:40:53.732198 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:40:53.732126 4939 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-rt2hx" Nov 03 00:40:53.834087 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:40:53.834060 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 00:40:53.917584 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:40:53.917549 4939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-266ht" event={"ID":"69b087f8-4a23-4ec8-808e-b61d2e16f67b","Type":"ContainerStarted","Data":"629ff02ec6e47a4ec0e1bbc446e7778640f8368b14000bd4fc641a2512acd529"} Nov 03 00:40:53.919084 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:40:53.919065 4939 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-n42s6_65d134ac-8a25-4c67-b6c7-cd752a098daf/console-operator/2.log" Nov 03 00:40:53.919475 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:40:53.919462 4939 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-n42s6_65d134ac-8a25-4c67-b6c7-cd752a098daf/console-operator/1.log" Nov 03 00:40:53.919518 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:40:53.919490 4939 generic.go:356] "Generic (PLEG): container finished" podID="65d134ac-8a25-4c67-b6c7-cd752a098daf" containerID="c46396ca66a985cafde6bb5bf19c88ae25c1d22ec41700045bc30c81059c31cb" exitCode=255 Nov 03 00:40:53.919590 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:40:53.919567 4939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-7cd56b8dbc-n42s6" event={"ID":"65d134ac-8a25-4c67-b6c7-cd752a098daf","Type":"ContainerDied","Data":"c46396ca66a985cafde6bb5bf19c88ae25c1d22ec41700045bc30c81059c31cb"} Nov 03 00:40:53.919642 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:40:53.919627 4939 scope.go:122] "RemoveContainer" containerID="8fbbed43b6d1c11a77523d9a59cdf45ffb9e7d1661d1a2290ee4a1ddd7dc0fb7" Nov 03 00:40:53.920070 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:40:53.920054 4939 scope.go:122] "RemoveContainer" containerID="c46396ca66a985cafde6bb5bf19c88ae25c1d22ec41700045bc30c81059c31cb" Nov 03 00:40:53.920223 ip-10-0-140-107 kubenswrapper[4939]: E1103 00:40:53.920202 4939 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-7cd56b8dbc-n42s6_openshift-console-operator(65d134ac-8a25-4c67-b6c7-cd752a098daf)\"" pod="openshift-console-operator/console-operator-7cd56b8dbc-n42s6" podUID="65d134ac-8a25-4c67-b6c7-cd752a098daf" Nov 03 00:40:53.991718 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:40:53.991655 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 00:40:54.027876 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:40:54.027843 4939 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-rt2hx"] Nov 03 00:40:54.030875 ip-10-0-140-107 kubenswrapper[4939]: W1103 00:40:54.030852 4939 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod136086c0_4846_437a_ad98_3cfef50610a0.slice/crio-c826e0119a0e3fd8c7a60c5f4aa39a121824e46daf48cc72da384bf46810bb2f WatchSource:0}: Error finding container c826e0119a0e3fd8c7a60c5f4aa39a121824e46daf48cc72da384bf46810bb2f: Status 404 returned error can't find the container with id c826e0119a0e3fd8c7a60c5f4aa39a121824e46daf48cc72da384bf46810bb2f Nov 03 00:40:54.122261 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:40:54.122228 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 00:40:54.145671 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:40:54.145646 4939 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-psqpn" Nov 03 00:40:54.392946 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:40:54.392900 4939 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-bbk2h_460779b2-3903-4b23-9b4d-e25142782a21/node-ca/0.log" Nov 03 00:40:54.471185 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:40:54.471156 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/06973556-f51c-4d1c-ae55-fb77699bc5b3-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-4bm4f\" (UID: \"06973556-f51c-4d1c-ae55-fb77699bc5b3\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-4bm4f" Nov 03 00:40:54.471356 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:40:54.471225 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cf217a21-c1bd-48be-b203-f59743bceb33-metrics-certs\") pod \"router-default-d7798ccf5-442kt\" (UID: \"cf217a21-c1bd-48be-b203-f59743bceb33\") " pod="openshift-ingress/router-default-d7798ccf5-442kt" Nov 03 00:40:54.471356 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:40:54.471253 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/dd0cd8a1-4d6b-424c-ac39-5f5ddb9cd87e-registry-tls\") pod \"image-registry-65f77c855d-6tnc2\" (UID: \"dd0cd8a1-4d6b-424c-ac39-5f5ddb9cd87e\") " pod="openshift-image-registry/image-registry-65f77c855d-6tnc2" Nov 03 00:40:54.471356 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:40:54.471271 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/6eafe6f6-cf90-446f-a8d5-620cd2e02775-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-2ngwx\" (UID: \"6eafe6f6-cf90-446f-a8d5-620cd2e02775\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-2ngwx" Nov 03 00:40:54.471356 ip-10-0-140-107 kubenswrapper[4939]: E1103 00:40:54.471280 4939 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Nov 03 00:40:54.471356 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:40:54.471302 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/bc513cba-d924-4719-a2ba-711529175ee0-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-z2qgj\" (UID: \"bc513cba-d924-4719-a2ba-711529175ee0\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-z2qgj" Nov 03 00:40:54.471564 ip-10-0-140-107 kubenswrapper[4939]: E1103 00:40:54.471366 4939 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Nov 03 00:40:54.471564 ip-10-0-140-107 kubenswrapper[4939]: E1103 00:40:54.471371 4939 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Nov 03 00:40:54.471564 ip-10-0-140-107 kubenswrapper[4939]: E1103 00:40:54.471389 4939 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-65f77c855d-6tnc2: secret "image-registry-tls" not found Nov 03 00:40:54.471564 ip-10-0-140-107 kubenswrapper[4939]: E1103 00:40:54.471399 4939 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Nov 03 00:40:54.471564 ip-10-0-140-107 kubenswrapper[4939]: E1103 00:40:54.471426 4939 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cf217a21-c1bd-48be-b203-f59743bceb33-metrics-certs podName:cf217a21-c1bd-48be-b203-f59743bceb33 nodeName:}" failed. No retries permitted until 2026-11-03 00:40:56.471407992 +0000 UTC m=+4.735742934 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/cf217a21-c1bd-48be-b203-f59743bceb33-metrics-certs") pod "router-default-d7798ccf5-442kt" (UID: "cf217a21-c1bd-48be-b203-f59743bceb33") : secret "router-metrics-certs-default" not found Nov 03 00:40:54.471564 ip-10-0-140-107 kubenswrapper[4939]: E1103 00:40:54.471403 4939 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Nov 03 00:40:54.471564 ip-10-0-140-107 kubenswrapper[4939]: E1103 00:40:54.471441 4939 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/06973556-f51c-4d1c-ae55-fb77699bc5b3-cluster-monitoring-operator-tls podName:06973556-f51c-4d1c-ae55-fb77699bc5b3 nodeName:}" failed. No retries permitted until 2026-11-03 00:40:56.47143501 +0000 UTC m=+4.735769932 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/06973556-f51c-4d1c-ae55-fb77699bc5b3-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-884576c57-4bm4f" (UID: "06973556-f51c-4d1c-ae55-fb77699bc5b3") : secret "cluster-monitoring-operator-tls" not found Nov 03 00:40:54.471564 ip-10-0-140-107 kubenswrapper[4939]: E1103 00:40:54.471489 4939 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/dd0cd8a1-4d6b-424c-ac39-5f5ddb9cd87e-registry-tls podName:dd0cd8a1-4d6b-424c-ac39-5f5ddb9cd87e nodeName:}" failed. No retries permitted until 2026-11-03 00:40:56.471471559 +0000 UTC m=+4.735806488 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/dd0cd8a1-4d6b-424c-ac39-5f5ddb9cd87e-registry-tls") pod "image-registry-65f77c855d-6tnc2" (UID: "dd0cd8a1-4d6b-424c-ac39-5f5ddb9cd87e") : secret "image-registry-tls" not found Nov 03 00:40:54.471564 ip-10-0-140-107 kubenswrapper[4939]: E1103 00:40:54.471510 4939 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6eafe6f6-cf90-446f-a8d5-620cd2e02775-samples-operator-tls podName:6eafe6f6-cf90-446f-a8d5-620cd2e02775 nodeName:}" failed. No retries permitted until 2026-11-03 00:40:56.471498868 +0000 UTC m=+4.735833796 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/6eafe6f6-cf90-446f-a8d5-620cd2e02775-samples-operator-tls") pod "cluster-samples-operator-6ddb6c8b94-2ngwx" (UID: "6eafe6f6-cf90-446f-a8d5-620cd2e02775") : secret "samples-operator-tls" not found Nov 03 00:40:54.471564 ip-10-0-140-107 kubenswrapper[4939]: E1103 00:40:54.471531 4939 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/bc513cba-d924-4719-a2ba-711529175ee0-networking-console-plugin-cert podName:bc513cba-d924-4719-a2ba-711529175ee0 nodeName:}" failed. No retries permitted until 2026-11-03 00:40:56.47152274 +0000 UTC m=+4.735857664 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/bc513cba-d924-4719-a2ba-711529175ee0-networking-console-plugin-cert") pod "networking-console-plugin-86c478bb88-z2qgj" (UID: "bc513cba-d924-4719-a2ba-711529175ee0") : secret "networking-console-plugin-cert" not found Nov 03 00:40:54.517759 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:40:54.517730 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 00:40:54.572110 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:40:54.572076 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/945da51a-32cb-4769-900d-1aeb266c7565-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-sthjh\" (UID: \"945da51a-32cb-4769-900d-1aeb266c7565\") " pod="openshift-insights/insights-runtime-extractor-sthjh" Nov 03 00:40:54.572293 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:40:54.572173 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/52a19842-3750-4d2b-bcb2-aebddee81097-registry-tls\") pod \"image-registry-6669dc8887-ws544\" (UID: \"52a19842-3750-4d2b-bcb2-aebddee81097\") " pod="openshift-image-registry/image-registry-6669dc8887-ws544" Nov 03 00:40:54.572293 ip-10-0-140-107 kubenswrapper[4939]: E1103 00:40:54.572244 4939 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Nov 03 00:40:54.572293 ip-10-0-140-107 kubenswrapper[4939]: E1103 00:40:54.572280 4939 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Nov 03 00:40:54.572403 ip-10-0-140-107 kubenswrapper[4939]: E1103 00:40:54.572296 4939 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-6669dc8887-ws544: secret "image-registry-tls" not found Nov 03 00:40:54.572403 ip-10-0-140-107 kubenswrapper[4939]: E1103 00:40:54.572312 4939 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/945da51a-32cb-4769-900d-1aeb266c7565-insights-runtime-extractor-tls podName:945da51a-32cb-4769-900d-1aeb266c7565 nodeName:}" failed. No retries permitted until 2026-11-03 00:40:56.572296825 +0000 UTC m=+4.836631777 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/945da51a-32cb-4769-900d-1aeb266c7565-insights-runtime-extractor-tls") pod "insights-runtime-extractor-sthjh" (UID: "945da51a-32cb-4769-900d-1aeb266c7565") : secret "insights-runtime-extractor-tls" not found Nov 03 00:40:54.572403 ip-10-0-140-107 kubenswrapper[4939]: E1103 00:40:54.572352 4939 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/52a19842-3750-4d2b-bcb2-aebddee81097-registry-tls podName:52a19842-3750-4d2b-bcb2-aebddee81097 nodeName:}" failed. No retries permitted until 2026-11-03 00:40:56.572333773 +0000 UTC m=+4.836668715 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/52a19842-3750-4d2b-bcb2-aebddee81097-registry-tls") pod "image-registry-6669dc8887-ws544" (UID: "52a19842-3750-4d2b-bcb2-aebddee81097") : secret "image-registry-tls" not found Nov 03 00:40:54.721207 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:40:54.721123 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 00:40:54.923892 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:40:54.923819 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 00:40:54.924712 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:40:54.924680 4939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-rt2hx" event={"ID":"136086c0-4846-437a-ad98-3cfef50610a0","Type":"ContainerStarted","Data":"c826e0119a0e3fd8c7a60c5f4aa39a121824e46daf48cc72da384bf46810bb2f"} Nov 03 00:40:54.927279 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:40:54.927249 4939 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-n42s6_65d134ac-8a25-4c67-b6c7-cd752a098daf/console-operator/2.log" Nov 03 00:40:55.011394 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:40:55.011324 4939 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-psqpn" Nov 03 00:40:55.011586 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:40:55.011497 4939 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness" Nov 03 00:40:55.311588 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:40:55.311498 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 00:40:55.383498 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:40:55.383468 4939 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-s9hbz_75481915-3cbe-475b-abe7-cb4e81c93427/serve-healthcheck-canary/0.log" Nov 03 00:40:55.783298 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:40:55.783276 4939 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator_migrator-66df5d496b-m7cxm_68658033-8a49-4430-869f-a7eb79603484/migrator/0.log" Nov 03 00:40:55.932317 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:40:55.932235 4939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-rt2hx" event={"ID":"136086c0-4846-437a-ad98-3cfef50610a0","Type":"ContainerStarted","Data":"7518a429c1f9d8e15ea8ef438dba40a81b875e9c5193eb6f9d1b66c7ef2d9643"} Nov 03 00:40:55.932317 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:40:55.932289 4939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-rt2hx" event={"ID":"136086c0-4846-437a-ad98-3cfef50610a0","Type":"ContainerStarted","Data":"c4054e91d2b1b2365f6e1ca31bbacfc23edc9a4820a6d87269d0a2355eda20cd"} Nov 03 00:40:55.964155 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:40:55.964131 4939 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-7cd56b8dbc-n42s6" Nov 03 00:40:55.964447 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:40:55.964435 4939 scope.go:122] "RemoveContainer" containerID="c46396ca66a985cafde6bb5bf19c88ae25c1d22ec41700045bc30c81059c31cb" Nov 03 00:40:55.964631 ip-10-0-140-107 kubenswrapper[4939]: E1103 00:40:55.964614 4939 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-7cd56b8dbc-n42s6_openshift-console-operator(65d134ac-8a25-4c67-b6c7-cd752a098daf)\"" pod="openshift-console-operator/console-operator-7cd56b8dbc-n42s6" podUID="65d134ac-8a25-4c67-b6c7-cd752a098daf" Nov 03 00:40:56.104498 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:40:56.104427 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 00:40:56.189607 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:40:56.189532 4939 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator-operator_kube-storage-version-migrator-operator-7d4bc45dd6-khmv7_36b9d258-cb2c-4d2b-880f-ae93b065034c/kube-storage-version-migrator-operator/0.log" Nov 03 00:40:56.330286 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:40:56.330224 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 00:40:56.464326 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:40:56.464247 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 00:40:56.493458 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:40:56.493430 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/06973556-f51c-4d1c-ae55-fb77699bc5b3-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-4bm4f\" (UID: \"06973556-f51c-4d1c-ae55-fb77699bc5b3\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-4bm4f" Nov 03 00:40:56.493631 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:40:56.493481 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cf217a21-c1bd-48be-b203-f59743bceb33-metrics-certs\") pod \"router-default-d7798ccf5-442kt\" (UID: \"cf217a21-c1bd-48be-b203-f59743bceb33\") " pod="openshift-ingress/router-default-d7798ccf5-442kt" Nov 03 00:40:56.493695 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:40:56.493657 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/dd0cd8a1-4d6b-424c-ac39-5f5ddb9cd87e-registry-tls\") pod \"image-registry-65f77c855d-6tnc2\" (UID: \"dd0cd8a1-4d6b-424c-ac39-5f5ddb9cd87e\") " pod="openshift-image-registry/image-registry-65f77c855d-6tnc2" Nov 03 00:40:56.493751 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:40:56.493698 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/6eafe6f6-cf90-446f-a8d5-620cd2e02775-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-2ngwx\" (UID: \"6eafe6f6-cf90-446f-a8d5-620cd2e02775\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-2ngwx" Nov 03 00:40:56.493751 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:40:56.493721 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/bc513cba-d924-4719-a2ba-711529175ee0-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-z2qgj\" (UID: \"bc513cba-d924-4719-a2ba-711529175ee0\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-z2qgj" Nov 03 00:40:56.493863 ip-10-0-140-107 kubenswrapper[4939]: E1103 00:40:56.493751 4939 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Nov 03 00:40:56.493863 ip-10-0-140-107 kubenswrapper[4939]: E1103 00:40:56.493772 4939 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-65f77c855d-6tnc2: secret "image-registry-tls" not found Nov 03 00:40:56.493863 ip-10-0-140-107 kubenswrapper[4939]: E1103 00:40:56.493824 4939 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Nov 03 00:40:56.493863 ip-10-0-140-107 kubenswrapper[4939]: E1103 00:40:56.493835 4939 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Nov 03 00:40:56.494040 ip-10-0-140-107 kubenswrapper[4939]: E1103 00:40:56.493840 4939 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/dd0cd8a1-4d6b-424c-ac39-5f5ddb9cd87e-registry-tls podName:dd0cd8a1-4d6b-424c-ac39-5f5ddb9cd87e nodeName:}" failed. No retries permitted until 2026-11-03 00:41:00.493820407 +0000 UTC m=+8.758155336 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/dd0cd8a1-4d6b-424c-ac39-5f5ddb9cd87e-registry-tls") pod "image-registry-65f77c855d-6tnc2" (UID: "dd0cd8a1-4d6b-424c-ac39-5f5ddb9cd87e") : secret "image-registry-tls" not found Nov 03 00:40:56.494040 ip-10-0-140-107 kubenswrapper[4939]: E1103 00:40:56.493957 4939 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/bc513cba-d924-4719-a2ba-711529175ee0-networking-console-plugin-cert podName:bc513cba-d924-4719-a2ba-711529175ee0 nodeName:}" failed. No retries permitted until 2026-11-03 00:41:00.49390505 +0000 UTC m=+8.758239979 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/bc513cba-d924-4719-a2ba-711529175ee0-networking-console-plugin-cert") pod "networking-console-plugin-86c478bb88-z2qgj" (UID: "bc513cba-d924-4719-a2ba-711529175ee0") : secret "networking-console-plugin-cert" not found Nov 03 00:40:56.494040 ip-10-0-140-107 kubenswrapper[4939]: E1103 00:40:56.493986 4939 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6eafe6f6-cf90-446f-a8d5-620cd2e02775-samples-operator-tls podName:6eafe6f6-cf90-446f-a8d5-620cd2e02775 nodeName:}" failed. No retries permitted until 2026-11-03 00:41:00.493973725 +0000 UTC m=+8.758308650 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/6eafe6f6-cf90-446f-a8d5-620cd2e02775-samples-operator-tls") pod "cluster-samples-operator-6ddb6c8b94-2ngwx" (UID: "6eafe6f6-cf90-446f-a8d5-620cd2e02775") : secret "samples-operator-tls" not found Nov 03 00:40:56.496056 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:40:56.496029 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/06973556-f51c-4d1c-ae55-fb77699bc5b3-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-4bm4f\" (UID: \"06973556-f51c-4d1c-ae55-fb77699bc5b3\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-4bm4f" Nov 03 00:40:56.496161 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:40:56.496061 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cf217a21-c1bd-48be-b203-f59743bceb33-metrics-certs\") pod \"router-default-d7798ccf5-442kt\" (UID: \"cf217a21-c1bd-48be-b203-f59743bceb33\") " pod="openshift-ingress/router-default-d7798ccf5-442kt" Nov 03 00:40:56.594689 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:40:56.594649 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/52a19842-3750-4d2b-bcb2-aebddee81097-registry-tls\") pod \"image-registry-6669dc8887-ws544\" (UID: \"52a19842-3750-4d2b-bcb2-aebddee81097\") " pod="openshift-image-registry/image-registry-6669dc8887-ws544" Nov 03 00:40:56.594837 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:40:56.594707 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/945da51a-32cb-4769-900d-1aeb266c7565-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-sthjh\" (UID: \"945da51a-32cb-4769-900d-1aeb266c7565\") " pod="openshift-insights/insights-runtime-extractor-sthjh" Nov 03 00:40:56.594837 ip-10-0-140-107 kubenswrapper[4939]: E1103 00:40:56.594790 4939 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Nov 03 00:40:56.594837 ip-10-0-140-107 kubenswrapper[4939]: E1103 00:40:56.594811 4939 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-6669dc8887-ws544: secret "image-registry-tls" not found Nov 03 00:40:56.594996 ip-10-0-140-107 kubenswrapper[4939]: E1103 00:40:56.594847 4939 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Nov 03 00:40:56.594996 ip-10-0-140-107 kubenswrapper[4939]: E1103 00:40:56.594875 4939 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/52a19842-3750-4d2b-bcb2-aebddee81097-registry-tls podName:52a19842-3750-4d2b-bcb2-aebddee81097 nodeName:}" failed. No retries permitted until 2026-11-03 00:41:00.594859163 +0000 UTC m=+8.859194098 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/52a19842-3750-4d2b-bcb2-aebddee81097-registry-tls") pod "image-registry-6669dc8887-ws544" (UID: "52a19842-3750-4d2b-bcb2-aebddee81097") : secret "image-registry-tls" not found Nov 03 00:40:56.594996 ip-10-0-140-107 kubenswrapper[4939]: E1103 00:40:56.594891 4939 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/945da51a-32cb-4769-900d-1aeb266c7565-insights-runtime-extractor-tls podName:945da51a-32cb-4769-900d-1aeb266c7565 nodeName:}" failed. No retries permitted until 2026-11-03 00:41:00.594883802 +0000 UTC m=+8.859218725 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/945da51a-32cb-4769-900d-1aeb266c7565-insights-runtime-extractor-tls") pod "insights-runtime-extractor-sthjh" (UID: "945da51a-32cb-4769-900d-1aeb266c7565") : secret "insights-runtime-extractor-tls" not found Nov 03 00:40:56.613530 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:40:56.613489 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 00:40:56.727845 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:40:56.727764 4939 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/cluster-monitoring-operator-884576c57-4bm4f" Nov 03 00:40:56.728844 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:40:56.728817 4939 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/router-default-d7798ccf5-442kt" Nov 03 00:40:56.758681 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:40:56.758651 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 00:40:56.898440 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:40:56.898409 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 00:40:56.937084 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:40:56.937056 4939 generic.go:356] "Generic (PLEG): container finished" podID="ca0f883c-c2b7-4bf6-a90b-ffdbeac161a8" containerID="90ac319dc159fa4199add901b66cd1e68b419e073b0586dd5ee7fdd7ce194c3d" exitCode=0 Nov 03 00:40:56.937470 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:40:56.937134 4939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-7c4986d449-xprcf" event={"ID":"ca0f883c-c2b7-4bf6-a90b-ffdbeac161a8","Type":"ContainerDied","Data":"90ac319dc159fa4199add901b66cd1e68b419e073b0586dd5ee7fdd7ce194c3d"} Nov 03 00:40:56.937545 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:40:56.937501 4939 scope.go:122] "RemoveContainer" containerID="90ac319dc159fa4199add901b66cd1e68b419e073b0586dd5ee7fdd7ce194c3d" Nov 03 00:40:56.938985 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:40:56.938963 4939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-266ht" event={"ID":"69b087f8-4a23-4ec8-808e-b61d2e16f67b","Type":"ContainerStarted","Data":"cd9ad5d9319f252929b8ebc978757ac3b817737d98ac4a614c2a93ec5dcbf6e9"} Nov 03 00:40:56.939106 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:40:56.939009 4939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-266ht" event={"ID":"69b087f8-4a23-4ec8-808e-b61d2e16f67b","Type":"ContainerStarted","Data":"716cd2dd2b83c7df6e48eff1d185e37eaa068d4d97d7f7a95f7dec18393f11bb"} Nov 03 00:40:56.939106 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:40:56.939040 4939 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-266ht" Nov 03 00:40:56.940479 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:40:56.940450 4939 generic.go:356] "Generic (PLEG): container finished" podID="2b050b18-31f6-4a22-9902-0435d11018b9" containerID="c1e2afb719a4e07feb8109267cac6124f4dca189e377407a93c4e97946eb5255" exitCode=0 Nov 03 00:40:56.940575 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:40:56.940525 4939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-7lqbv" event={"ID":"2b050b18-31f6-4a22-9902-0435d11018b9","Type":"ContainerDied","Data":"c1e2afb719a4e07feb8109267cac6124f4dca189e377407a93c4e97946eb5255"} Nov 03 00:40:56.941032 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:40:56.941013 4939 scope.go:122] "RemoveContainer" containerID="c1e2afb719a4e07feb8109267cac6124f4dca189e377407a93c4e97946eb5255" Nov 03 00:40:57.030999 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:40:57.030755 4939 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-884576c57-4bm4f"] Nov 03 00:40:57.037130 ip-10-0-140-107 kubenswrapper[4939]: W1103 00:40:57.037097 4939 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod06973556_f51c_4d1c_ae55_fb77699bc5b3.slice/crio-4302bb8d5d267306fc3f95dd2d1ece5330fb5b88b20408ee964b9c639c895725 WatchSource:0}: Error finding container 4302bb8d5d267306fc3f95dd2d1ece5330fb5b88b20408ee964b9c639c895725: Status 404 returned error can't find the container with id 4302bb8d5d267306fc3f95dd2d1ece5330fb5b88b20408ee964b9c639c895725 Nov 03 00:40:57.069572 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:40:57.069538 4939 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/router-default-d7798ccf5-442kt"] Nov 03 00:40:57.072952 ip-10-0-140-107 kubenswrapper[4939]: W1103 00:40:57.072900 4939 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcf217a21_c1bd_48be_b203_f59743bceb33.slice/crio-f2b3de4db2d99eb62e51686bf271e88d04192feb3b790c460fbcbf01b4a124d2 WatchSource:0}: Error finding container f2b3de4db2d99eb62e51686bf271e88d04192feb3b790c460fbcbf01b4a124d2: Status 404 returned error can't find the container with id f2b3de4db2d99eb62e51686bf271e88d04192feb3b790c460fbcbf01b4a124d2 Nov 03 00:40:57.309679 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:40:57.309600 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 00:40:57.946599 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:40:57.946508 4939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-7c4986d449-xprcf" event={"ID":"ca0f883c-c2b7-4bf6-a90b-ffdbeac161a8","Type":"ContainerStarted","Data":"03003b0a16b86bfd908b95260e3a4239e99705e71bed47a1b9fa49354eeab6db"} Nov 03 00:40:57.948205 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:40:57.948173 4939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/router-default-d7798ccf5-442kt" event={"ID":"cf217a21-c1bd-48be-b203-f59743bceb33","Type":"ContainerStarted","Data":"4aa02db9e0caecc89f5f6b84931043da65aebaff3a6dbb8bfd6079b2bca0cbd5"} Nov 03 00:40:57.948337 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:40:57.948224 4939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/router-default-d7798ccf5-442kt" event={"ID":"cf217a21-c1bd-48be-b203-f59743bceb33","Type":"ContainerStarted","Data":"f2b3de4db2d99eb62e51686bf271e88d04192feb3b790c460fbcbf01b4a124d2"} Nov 03 00:40:57.949878 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:40:57.949855 4939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-7lqbv" event={"ID":"2b050b18-31f6-4a22-9902-0435d11018b9","Type":"ContainerStarted","Data":"aa25b17342664a810bb2a3adf9a2d9dabdc4518c9f63edd82df3bfd932021078"} Nov 03 00:40:57.950927 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:40:57.950892 4939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/cluster-monitoring-operator-884576c57-4bm4f" event={"ID":"06973556-f51c-4d1c-ae55-fb77699bc5b3","Type":"ContainerStarted","Data":"4302bb8d5d267306fc3f95dd2d1ece5330fb5b88b20408ee964b9c639c895725"} Nov 03 00:40:58.106093 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:40:58.106058 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 00:40:58.272586 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:40:58.272457 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 00:40:58.513262 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:40:58.513219 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 00:40:58.716109 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:40:58.716075 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 00:40:58.729048 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:40:58.729012 4939 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/router-default-d7798ccf5-442kt" Nov 03 00:40:58.731856 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:40:58.731833 4939 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/router-default-d7798ccf5-442kt" Nov 03 00:40:58.891465 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:40:58.891427 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 00:40:59.033314 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:40:59.033292 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 00:40:59.316644 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:40:59.316568 4939 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/router-default-d7798ccf5-442kt" Nov 03 00:40:59.317812 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:40:59.317794 4939 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/router-default-d7798ccf5-442kt" Nov 03 00:40:59.353313 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:40:59.353278 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 00:40:59.564999 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:40:59.564970 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 00:40:59.709506 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:40:59.709474 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 00:40:59.959869 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:40:59.959770 4939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/cluster-monitoring-operator-884576c57-4bm4f" event={"ID":"06973556-f51c-4d1c-ae55-fb77699bc5b3","Type":"ContainerStarted","Data":"70e3ddb997fdce636da4c5f681493e284f676ad64bd51032e30a2b12f67d0b3f"} Nov 03 00:41:00.100340 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:41:00.100301 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 00:41:00.527963 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:41:00.527930 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 00:41:00.539136 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:41:00.539104 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/dd0cd8a1-4d6b-424c-ac39-5f5ddb9cd87e-registry-tls\") pod \"image-registry-65f77c855d-6tnc2\" (UID: \"dd0cd8a1-4d6b-424c-ac39-5f5ddb9cd87e\") " pod="openshift-image-registry/image-registry-65f77c855d-6tnc2" Nov 03 00:41:00.539225 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:41:00.539147 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/6eafe6f6-cf90-446f-a8d5-620cd2e02775-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-2ngwx\" (UID: \"6eafe6f6-cf90-446f-a8d5-620cd2e02775\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-2ngwx" Nov 03 00:41:00.539293 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:41:00.539271 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/bc513cba-d924-4719-a2ba-711529175ee0-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-z2qgj\" (UID: \"bc513cba-d924-4719-a2ba-711529175ee0\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-z2qgj" Nov 03 00:41:00.541680 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:41:00.541648 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/6eafe6f6-cf90-446f-a8d5-620cd2e02775-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-2ngwx\" (UID: \"6eafe6f6-cf90-446f-a8d5-620cd2e02775\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-2ngwx" Nov 03 00:41:00.541680 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:41:00.541651 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/bc513cba-d924-4719-a2ba-711529175ee0-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-z2qgj\" (UID: \"bc513cba-d924-4719-a2ba-711529175ee0\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-z2qgj" Nov 03 00:41:00.541836 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:41:00.541720 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/dd0cd8a1-4d6b-424c-ac39-5f5ddb9cd87e-registry-tls\") pod \"image-registry-65f77c855d-6tnc2\" (UID: \"dd0cd8a1-4d6b-424c-ac39-5f5ddb9cd87e\") " pod="openshift-image-registry/image-registry-65f77c855d-6tnc2" Nov 03 00:41:00.627766 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:41:00.627731 4939 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-2ngwx" Nov 03 00:41:00.629469 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:41:00.629443 4939 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-65f77c855d-6tnc2" Nov 03 00:41:00.629883 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:41:00.629587 4939 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-86c478bb88-z2qgj" Nov 03 00:41:00.640204 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:41:00.640180 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/52a19842-3750-4d2b-bcb2-aebddee81097-registry-tls\") pod \"image-registry-6669dc8887-ws544\" (UID: \"52a19842-3750-4d2b-bcb2-aebddee81097\") " pod="openshift-image-registry/image-registry-6669dc8887-ws544" Nov 03 00:41:00.640293 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:41:00.640225 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/945da51a-32cb-4769-900d-1aeb266c7565-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-sthjh\" (UID: \"945da51a-32cb-4769-900d-1aeb266c7565\") " pod="openshift-insights/insights-runtime-extractor-sthjh" Nov 03 00:41:00.640338 ip-10-0-140-107 kubenswrapper[4939]: E1103 00:41:00.640326 4939 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Nov 03 00:41:00.640389 ip-10-0-140-107 kubenswrapper[4939]: E1103 00:41:00.640380 4939 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/945da51a-32cb-4769-900d-1aeb266c7565-insights-runtime-extractor-tls podName:945da51a-32cb-4769-900d-1aeb266c7565 nodeName:}" failed. No retries permitted until 2026-11-03 00:41:08.640364411 +0000 UTC m=+16.904699334 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/945da51a-32cb-4769-900d-1aeb266c7565-insights-runtime-extractor-tls") pod "insights-runtime-extractor-sthjh" (UID: "945da51a-32cb-4769-900d-1aeb266c7565") : secret "insights-runtime-extractor-tls" not found Nov 03 00:41:00.642935 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:41:00.642901 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/52a19842-3750-4d2b-bcb2-aebddee81097-registry-tls\") pod \"image-registry-6669dc8887-ws544\" (UID: \"52a19842-3750-4d2b-bcb2-aebddee81097\") " pod="openshift-image-registry/image-registry-6669dc8887-ws544" Nov 03 00:41:00.650186 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:41:00.650161 4939 kubelet.go:2727] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="openshift-console-operator/console-operator-7cd56b8dbc-n42s6" Nov 03 00:41:00.650659 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:41:00.650638 4939 scope.go:122] "RemoveContainer" containerID="c46396ca66a985cafde6bb5bf19c88ae25c1d22ec41700045bc30c81059c31cb" Nov 03 00:41:00.651176 ip-10-0-140-107 kubenswrapper[4939]: E1103 00:41:00.650868 4939 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-7cd56b8dbc-n42s6_openshift-console-operator(65d134ac-8a25-4c67-b6c7-cd752a098daf)\"" pod="openshift-console-operator/console-operator-7cd56b8dbc-n42s6" podUID="65d134ac-8a25-4c67-b6c7-cd752a098daf" Nov 03 00:41:00.667941 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:41:00.667295 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 00:41:00.788539 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:41:00.788458 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 00:41:00.917549 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:41:00.917519 4939 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-2ngwx"] Nov 03 00:41:00.920844 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:41:00.920795 4939 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-console/networking-console-plugin-86c478bb88-z2qgj"] Nov 03 00:41:00.921979 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:41:00.921953 4939 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-65f77c855d-6tnc2"] Nov 03 00:41:00.925378 ip-10-0-140-107 kubenswrapper[4939]: W1103 00:41:00.925353 4939 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddd0cd8a1_4d6b_424c_ac39_5f5ddb9cd87e.slice/crio-66e078530d58e4adef651cba6a19bf8779383083a0ca7d293b4e569c3c0df87d WatchSource:0}: Error finding container 66e078530d58e4adef651cba6a19bf8779383083a0ca7d293b4e569c3c0df87d: Status 404 returned error can't find the container with id 66e078530d58e4adef651cba6a19bf8779383083a0ca7d293b4e569c3c0df87d Nov 03 00:41:00.929279 ip-10-0-140-107 kubenswrapper[4939]: W1103 00:41:00.928570 4939 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbc513cba_d924_4719_a2ba_711529175ee0.slice/crio-0751ba1624cff8c106948936fe34594be36c7cdd5f3f3a4fff43808d65474258 WatchSource:0}: Error finding container 0751ba1624cff8c106948936fe34594be36c7cdd5f3f3a4fff43808d65474258: Status 404 returned error can't find the container with id 0751ba1624cff8c106948936fe34594be36c7cdd5f3f3a4fff43808d65474258 Nov 03 00:41:00.929633 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:41:00.929554 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 00:41:00.942850 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:41:00.942830 4939 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-6669dc8887-ws544" Nov 03 00:41:00.965359 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:41:00.965333 4939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-65f77c855d-6tnc2" event={"ID":"dd0cd8a1-4d6b-424c-ac39-5f5ddb9cd87e","Type":"ContainerStarted","Data":"66e078530d58e4adef651cba6a19bf8779383083a0ca7d293b4e569c3c0df87d"} Nov 03 00:41:00.966881 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:41:00.966841 4939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-console/networking-console-plugin-86c478bb88-z2qgj" event={"ID":"bc513cba-d924-4719-a2ba-711529175ee0","Type":"ContainerStarted","Data":"0751ba1624cff8c106948936fe34594be36c7cdd5f3f3a4fff43808d65474258"} Nov 03 00:41:00.968278 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:41:00.968253 4939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-2ngwx" event={"ID":"6eafe6f6-cf90-446f-a8d5-620cd2e02775","Type":"ContainerStarted","Data":"27499b4ad9049357a91a201adb6f2c69d3233a194a1e405fe4b0f161013d7499"} Nov 03 00:41:01.226088 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:41:01.226055 4939 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-6669dc8887-ws544"] Nov 03 00:41:01.229057 ip-10-0-140-107 kubenswrapper[4939]: W1103 00:41:01.229028 4939 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod52a19842_3750_4d2b_bcb2_aebddee81097.slice/crio-5932df9ab8e5f4e60993932d77de8e89b0058d1d12084f9c72538cf51c805fd0 WatchSource:0}: Error finding container 5932df9ab8e5f4e60993932d77de8e89b0058d1d12084f9c72538cf51c805fd0: Status 404 returned error can't find the container with id 5932df9ab8e5f4e60993932d77de8e89b0058d1d12084f9c72538cf51c805fd0 Nov 03 00:41:01.308302 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:41:01.308276 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 00:41:01.890028 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:41:01.889989 4939 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-w46kd" Nov 03 00:41:01.973855 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:41:01.973720 4939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-6669dc8887-ws544" event={"ID":"52a19842-3750-4d2b-bcb2-aebddee81097","Type":"ContainerStarted","Data":"3630dbb3130ca380f93aa67ccddf7d735254b2bd02fc3dbbcbb37d070881e0a3"} Nov 03 00:41:01.973855 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:41:01.973817 4939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-6669dc8887-ws544" event={"ID":"52a19842-3750-4d2b-bcb2-aebddee81097","Type":"ContainerStarted","Data":"5932df9ab8e5f4e60993932d77de8e89b0058d1d12084f9c72538cf51c805fd0"} Nov 03 00:41:01.975671 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:41:01.975640 4939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-65f77c855d-6tnc2" event={"ID":"dd0cd8a1-4d6b-424c-ac39-5f5ddb9cd87e","Type":"ContainerStarted","Data":"80cbb4a91ce3ce0b438f284c9fd8cd746d0c50a49e40a26e8fb62931e7d11b72"} Nov 03 00:41:02.090671 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:41:02.090636 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 00:41:02.491373 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:41:02.491297 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 00:41:02.689825 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:41:02.689798 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 00:41:02.926797 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:41:02.926754 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 00:41:02.980957 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:41:02.980896 4939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-console/networking-console-plugin-86c478bb88-z2qgj" event={"ID":"bc513cba-d924-4719-a2ba-711529175ee0","Type":"ContainerStarted","Data":"bee706055b9a4713dd12b4e41be6fa5231ddcead6349df41e4618f5e6947293a"} Nov 03 00:41:02.982774 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:41:02.982749 4939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-2ngwx" event={"ID":"6eafe6f6-cf90-446f-a8d5-620cd2e02775","Type":"ContainerStarted","Data":"47bc79d41fe98c31c67cbb6adb0f6dc6a6c4456768d8627f61ee324ab3d7fac1"} Nov 03 00:41:02.982886 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:41:02.982788 4939 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness" Nov 03 00:41:02.982886 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:41:02.982792 4939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-2ngwx" event={"ID":"6eafe6f6-cf90-446f-a8d5-620cd2e02775","Type":"ContainerStarted","Data":"c20a50d08accc5afa319985318aab2b57690d4b6b4370e1562ad3cc0d48dedd9"} Nov 03 00:41:02.982886 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:41:02.982757 4939 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness" Nov 03 00:41:03.306261 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:41:03.306227 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 00:41:03.492479 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:41:03.492439 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 00:41:04.208440 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:41:04.208409 4939 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-82x8c" Nov 03 00:41:04.861276 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:41:04.861245 4939 scope.go:122] "RemoveContainer" containerID="c46396ca66a985cafde6bb5bf19c88ae25c1d22ec41700045bc30c81059c31cb" Nov 03 00:41:04.995234 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:41:04.995204 4939 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-n42s6_65d134ac-8a25-4c67-b6c7-cd752a098daf/console-operator/2.log" Nov 03 00:41:04.995383 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:41:04.995327 4939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-7cd56b8dbc-n42s6" event={"ID":"65d134ac-8a25-4c67-b6c7-cd752a098daf","Type":"ContainerStarted","Data":"c9396e7e16d5af165413d19628825b01a7d4f590f90519ec32fcdbf8ef59757c"} Nov 03 00:41:04.995623 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:41:04.995607 4939 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-7cd56b8dbc-n42s6" Nov 03 00:41:05.117669 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:41:05.117595 4939 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console-operator/console-operator-7cd56b8dbc-n42s6" Nov 03 00:41:05.384185 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:41:05.384100 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 00:41:05.421217 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:41:05.421188 4939 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-6669dc8887-ws544" Nov 03 00:41:05.540047 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:41:05.540018 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 00:41:05.675604 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:41:05.675508 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 00:41:05.821514 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:41:05.821482 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 00:41:05.984065 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:41:05.983997 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 00:41:06.128834 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:41:06.128798 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 00:41:06.953401 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:41:06.953369 4939 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-266ht" Nov 03 00:41:08.485464 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:41:08.485427 4939 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-65f77c855d-6tnc2" Nov 03 00:41:08.712770 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:41:08.712732 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/945da51a-32cb-4769-900d-1aeb266c7565-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-sthjh\" (UID: \"945da51a-32cb-4769-900d-1aeb266c7565\") " pod="openshift-insights/insights-runtime-extractor-sthjh" Nov 03 00:41:08.715116 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:41:08.715090 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/945da51a-32cb-4769-900d-1aeb266c7565-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-sthjh\" (UID: \"945da51a-32cb-4769-900d-1aeb266c7565\") " pod="openshift-insights/insights-runtime-extractor-sthjh" Nov 03 00:41:08.745281 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:41:08.745202 4939 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-sthjh" Nov 03 00:41:09.021358 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:41:09.021261 4939 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-sthjh"] Nov 03 00:41:09.024064 ip-10-0-140-107 kubenswrapper[4939]: W1103 00:41:09.024032 4939 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod945da51a_32cb_4769_900d_1aeb266c7565.slice/crio-2173fdd46b342c12784e797395b34ebc2795895a896d547b54c15efe60ab2319 WatchSource:0}: Error finding container 2173fdd46b342c12784e797395b34ebc2795895a896d547b54c15efe60ab2319: Status 404 returned error can't find the container with id 2173fdd46b342c12784e797395b34ebc2795895a896d547b54c15efe60ab2319 Nov 03 00:41:09.926767 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:41:09.926737 4939 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-65f77c855d-6tnc2"] Nov 03 00:41:10.016791 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:41:10.016696 4939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-sthjh" event={"ID":"945da51a-32cb-4769-900d-1aeb266c7565","Type":"ContainerStarted","Data":"89abe98d59861e40caa0308c81aae7f477e7edec737f1668a13b698b01195b1a"} Nov 03 00:41:10.016791 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:41:10.016755 4939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-sthjh" event={"ID":"945da51a-32cb-4769-900d-1aeb266c7565","Type":"ContainerStarted","Data":"d1d98eee4f48e133651334b3efbcf7b9fea01bbf7b3dca9e1bca16491f6b51dd"} Nov 03 00:41:10.016791 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:41:10.016776 4939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-sthjh" event={"ID":"945da51a-32cb-4769-900d-1aeb266c7565","Type":"ContainerStarted","Data":"2173fdd46b342c12784e797395b34ebc2795895a896d547b54c15efe60ab2319"} Nov 03 00:41:10.018374 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:41:10.018347 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 00:41:10.241069 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:41:10.241030 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 00:41:10.377133 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:41:10.377099 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 00:41:10.591231 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:41:10.591196 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 00:41:11.253476 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:41:11.253449 4939 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-266ht_69b087f8-4a23-4ec8-808e-b61d2e16f67b/dns/0.log" Nov 03 00:41:11.296199 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:41:11.296167 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 00:41:11.453805 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:41:11.453778 4939 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-266ht_69b087f8-4a23-4ec8-808e-b61d2e16f67b/kube-rbac-proxy/0.log" Nov 03 00:41:11.494681 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:41:11.494650 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 00:41:12.026798 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:41:12.026762 4939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-sthjh" event={"ID":"945da51a-32cb-4769-900d-1aeb266c7565","Type":"ContainerStarted","Data":"911e216555e1386f77992290f0e4a458a9255a60b7d3115573e85098b693a019"} Nov 03 00:41:12.453449 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:41:12.453422 4939 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-cjj2h_ad5bf442-729d-411c-96e4-2e5105ebadcf/dns-node-resolver/0.log" Nov 03 00:41:12.740002 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:41:12.739899 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 00:41:12.892420 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:41:12.892391 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 00:41:13.253470 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:41:13.253441 4939 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_image-registry-65f77c855d-6tnc2_dd0cd8a1-4d6b-424c-ac39-5f5ddb9cd87e/registry/0.log" Nov 03 00:41:13.454785 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:41:13.454754 4939 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_image-registry-6669dc8887-ws544_52a19842-3750-4d2b-bcb2-aebddee81097/registry/0.log" Nov 03 00:41:13.653979 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:41:13.653954 4939 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-bbk2h_460779b2-3903-4b23-9b4d-e25142782a21/node-ca/0.log" Nov 03 00:41:14.259578 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:41:14.259549 4939 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-d7798ccf5-442kt_cf217a21-c1bd-48be-b203-f59743bceb33/router/0.log" Nov 03 00:41:14.654090 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:41:14.654061 4939 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-s9hbz_75481915-3cbe-475b-abe7-cb4e81c93427/serve-healthcheck-canary/0.log" Nov 03 00:41:14.933883 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:41:14.933804 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 00:41:15.886523 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:41:15.886493 4939 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-psqpn" Nov 03 00:41:16.524775 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:41:16.524738 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 00:41:16.681921 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:41:16.681886 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 00:41:16.998192 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:41:16.998159 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 00:41:17.168736 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:41:17.168697 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 00:41:17.328120 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:41:17.328043 4939 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-rstd6"] Nov 03 00:41:17.333010 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:41:17.332993 4939 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-rstd6" Nov 03 00:41:17.360981 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:41:17.360958 4939 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\"" Nov 03 00:41:17.361195 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:41:17.361182 4939 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\"" Nov 03 00:41:17.365650 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:41:17.365628 4939 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-dxq4b\"" Nov 03 00:41:17.402059 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:41:17.402028 4939 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\"" Nov 03 00:41:17.402059 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:41:17.402040 4939 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\"" Nov 03 00:41:17.415582 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:41:17.415554 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 00:41:17.487287 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:41:17.487258 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/49e75d56-a78d-4c7f-88e3-c96e81dc026e-metrics-client-ca\") pod \"node-exporter-rstd6\" (UID: \"49e75d56-a78d-4c7f-88e3-c96e81dc026e\") " pod="openshift-monitoring/node-exporter-rstd6" Nov 03 00:41:17.487430 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:41:17.487299 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/49e75d56-a78d-4c7f-88e3-c96e81dc026e-node-exporter-textfile\") pod \"node-exporter-rstd6\" (UID: \"49e75d56-a78d-4c7f-88e3-c96e81dc026e\") " pod="openshift-monitoring/node-exporter-rstd6" Nov 03 00:41:17.487430 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:41:17.487318 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/49e75d56-a78d-4c7f-88e3-c96e81dc026e-node-exporter-accelerators-collector-config\") pod \"node-exporter-rstd6\" (UID: \"49e75d56-a78d-4c7f-88e3-c96e81dc026e\") " pod="openshift-monitoring/node-exporter-rstd6" Nov 03 00:41:17.487526 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:41:17.487432 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hkwmt\" (UniqueName: \"kubernetes.io/projected/49e75d56-a78d-4c7f-88e3-c96e81dc026e-kube-api-access-hkwmt\") pod \"node-exporter-rstd6\" (UID: \"49e75d56-a78d-4c7f-88e3-c96e81dc026e\") " pod="openshift-monitoring/node-exporter-rstd6" Nov 03 00:41:17.487526 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:41:17.487471 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/49e75d56-a78d-4c7f-88e3-c96e81dc026e-sys\") pod \"node-exporter-rstd6\" (UID: \"49e75d56-a78d-4c7f-88e3-c96e81dc026e\") " pod="openshift-monitoring/node-exporter-rstd6" Nov 03 00:41:17.487526 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:41:17.487489 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/49e75d56-a78d-4c7f-88e3-c96e81dc026e-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-rstd6\" (UID: \"49e75d56-a78d-4c7f-88e3-c96e81dc026e\") " pod="openshift-monitoring/node-exporter-rstd6" Nov 03 00:41:17.487630 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:41:17.487570 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/49e75d56-a78d-4c7f-88e3-c96e81dc026e-node-exporter-tls\") pod \"node-exporter-rstd6\" (UID: \"49e75d56-a78d-4c7f-88e3-c96e81dc026e\") " pod="openshift-monitoring/node-exporter-rstd6" Nov 03 00:41:17.487630 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:41:17.487588 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/49e75d56-a78d-4c7f-88e3-c96e81dc026e-root\") pod \"node-exporter-rstd6\" (UID: \"49e75d56-a78d-4c7f-88e3-c96e81dc026e\") " pod="openshift-monitoring/node-exporter-rstd6" Nov 03 00:41:17.487630 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:41:17.487607 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/49e75d56-a78d-4c7f-88e3-c96e81dc026e-node-exporter-wtmp\") pod \"node-exporter-rstd6\" (UID: \"49e75d56-a78d-4c7f-88e3-c96e81dc026e\") " pod="openshift-monitoring/node-exporter-rstd6" Nov 03 00:41:17.582321 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:41:17.582292 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 00:41:17.587945 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:41:17.587921 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/49e75d56-a78d-4c7f-88e3-c96e81dc026e-sys\") pod \"node-exporter-rstd6\" (UID: \"49e75d56-a78d-4c7f-88e3-c96e81dc026e\") " pod="openshift-monitoring/node-exporter-rstd6" Nov 03 00:41:17.588107 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:41:17.587956 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/49e75d56-a78d-4c7f-88e3-c96e81dc026e-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-rstd6\" (UID: \"49e75d56-a78d-4c7f-88e3-c96e81dc026e\") " pod="openshift-monitoring/node-exporter-rstd6" Nov 03 00:41:17.588107 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:41:17.587990 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/49e75d56-a78d-4c7f-88e3-c96e81dc026e-node-exporter-tls\") pod \"node-exporter-rstd6\" (UID: \"49e75d56-a78d-4c7f-88e3-c96e81dc026e\") " pod="openshift-monitoring/node-exporter-rstd6" Nov 03 00:41:17.588107 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:41:17.588010 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/49e75d56-a78d-4c7f-88e3-c96e81dc026e-root\") pod \"node-exporter-rstd6\" (UID: \"49e75d56-a78d-4c7f-88e3-c96e81dc026e\") " pod="openshift-monitoring/node-exporter-rstd6" Nov 03 00:41:17.588107 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:41:17.588031 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/49e75d56-a78d-4c7f-88e3-c96e81dc026e-node-exporter-wtmp\") pod \"node-exporter-rstd6\" (UID: \"49e75d56-a78d-4c7f-88e3-c96e81dc026e\") " pod="openshift-monitoring/node-exporter-rstd6" Nov 03 00:41:17.588107 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:41:17.588041 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/49e75d56-a78d-4c7f-88e3-c96e81dc026e-sys\") pod \"node-exporter-rstd6\" (UID: \"49e75d56-a78d-4c7f-88e3-c96e81dc026e\") " pod="openshift-monitoring/node-exporter-rstd6" Nov 03 00:41:17.588107 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:41:17.588055 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/49e75d56-a78d-4c7f-88e3-c96e81dc026e-metrics-client-ca\") pod \"node-exporter-rstd6\" (UID: \"49e75d56-a78d-4c7f-88e3-c96e81dc026e\") " pod="openshift-monitoring/node-exporter-rstd6" Nov 03 00:41:17.588107 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:41:17.588104 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/49e75d56-a78d-4c7f-88e3-c96e81dc026e-root\") pod \"node-exporter-rstd6\" (UID: \"49e75d56-a78d-4c7f-88e3-c96e81dc026e\") " pod="openshift-monitoring/node-exporter-rstd6" Nov 03 00:41:17.588448 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:41:17.588153 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/49e75d56-a78d-4c7f-88e3-c96e81dc026e-node-exporter-textfile\") pod \"node-exporter-rstd6\" (UID: \"49e75d56-a78d-4c7f-88e3-c96e81dc026e\") " pod="openshift-monitoring/node-exporter-rstd6" Nov 03 00:41:17.588448 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:41:17.588165 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/49e75d56-a78d-4c7f-88e3-c96e81dc026e-node-exporter-wtmp\") pod \"node-exporter-rstd6\" (UID: \"49e75d56-a78d-4c7f-88e3-c96e81dc026e\") " pod="openshift-monitoring/node-exporter-rstd6" Nov 03 00:41:17.588448 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:41:17.588172 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/49e75d56-a78d-4c7f-88e3-c96e81dc026e-node-exporter-accelerators-collector-config\") pod \"node-exporter-rstd6\" (UID: \"49e75d56-a78d-4c7f-88e3-c96e81dc026e\") " pod="openshift-monitoring/node-exporter-rstd6" Nov 03 00:41:17.588448 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:41:17.588313 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hkwmt\" (UniqueName: \"kubernetes.io/projected/49e75d56-a78d-4c7f-88e3-c96e81dc026e-kube-api-access-hkwmt\") pod \"node-exporter-rstd6\" (UID: \"49e75d56-a78d-4c7f-88e3-c96e81dc026e\") " pod="openshift-monitoring/node-exporter-rstd6" Nov 03 00:41:17.588629 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:41:17.588572 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/49e75d56-a78d-4c7f-88e3-c96e81dc026e-node-exporter-textfile\") pod \"node-exporter-rstd6\" (UID: \"49e75d56-a78d-4c7f-88e3-c96e81dc026e\") " pod="openshift-monitoring/node-exporter-rstd6" Nov 03 00:41:17.588771 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:41:17.588744 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/49e75d56-a78d-4c7f-88e3-c96e81dc026e-metrics-client-ca\") pod \"node-exporter-rstd6\" (UID: \"49e75d56-a78d-4c7f-88e3-c96e81dc026e\") " pod="openshift-monitoring/node-exporter-rstd6" Nov 03 00:41:17.588825 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:41:17.588744 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/49e75d56-a78d-4c7f-88e3-c96e81dc026e-node-exporter-accelerators-collector-config\") pod \"node-exporter-rstd6\" (UID: \"49e75d56-a78d-4c7f-88e3-c96e81dc026e\") " pod="openshift-monitoring/node-exporter-rstd6" Nov 03 00:41:17.590331 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:41:17.590309 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/49e75d56-a78d-4c7f-88e3-c96e81dc026e-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-rstd6\" (UID: \"49e75d56-a78d-4c7f-88e3-c96e81dc026e\") " pod="openshift-monitoring/node-exporter-rstd6" Nov 03 00:41:17.590427 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:41:17.590414 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/49e75d56-a78d-4c7f-88e3-c96e81dc026e-node-exporter-tls\") pod \"node-exporter-rstd6\" (UID: \"49e75d56-a78d-4c7f-88e3-c96e81dc026e\") " pod="openshift-monitoring/node-exporter-rstd6" Nov 03 00:41:17.598533 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:41:17.598510 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hkwmt\" (UniqueName: \"kubernetes.io/projected/49e75d56-a78d-4c7f-88e3-c96e81dc026e-kube-api-access-hkwmt\") pod \"node-exporter-rstd6\" (UID: \"49e75d56-a78d-4c7f-88e3-c96e81dc026e\") " pod="openshift-monitoring/node-exporter-rstd6" Nov 03 00:41:17.649399 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:41:17.649371 4939 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-rstd6" Nov 03 00:41:17.661545 ip-10-0-140-107 kubenswrapper[4939]: W1103 00:41:17.661517 4939 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod49e75d56_a78d_4c7f_88e3_c96e81dc026e.slice/crio-47d932394beb51fe468d27ec3e5358d6c1faf9efe9012f5ed47299eade7fda0f WatchSource:0}: Error finding container 47d932394beb51fe468d27ec3e5358d6c1faf9efe9012f5ed47299eade7fda0f: Status 404 returned error can't find the container with id 47d932394beb51fe468d27ec3e5358d6c1faf9efe9012f5ed47299eade7fda0f Nov 03 00:41:17.714174 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:41:17.714136 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 00:41:17.896319 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:41:17.896244 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 00:41:18.028742 ip-10-0-140-107 kubenswrapper[4939]: E1103 00:41:18.028701 4939 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-sthjh" containerName="extractor" Nov 03 00:41:18.050090 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:41:18.050053 4939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-rstd6" event={"ID":"49e75d56-a78d-4c7f-88e3-c96e81dc026e","Type":"ContainerStarted","Data":"47d932394beb51fe468d27ec3e5358d6c1faf9efe9012f5ed47299eade7fda0f"} Nov 03 00:41:18.395584 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:41:18.395549 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 00:41:18.604833 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:41:18.604798 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 00:41:18.809222 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:41:18.809199 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 00:41:19.056271 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:41:19.056189 4939 generic.go:356] "Generic (PLEG): container finished" podID="49e75d56-a78d-4c7f-88e3-c96e81dc026e" containerID="d9ee419576aae74f10a593e4ed91943824b10846aa7ea4d4cb8f3e52fe02f9ea" exitCode=0 Nov 03 00:41:19.056271 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:41:19.056232 4939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-rstd6" event={"ID":"49e75d56-a78d-4c7f-88e3-c96e81dc026e","Type":"ContainerDied","Data":"d9ee419576aae74f10a593e4ed91943824b10846aa7ea4d4cb8f3e52fe02f9ea"} Nov 03 00:41:19.181707 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:41:19.181683 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 00:41:19.398020 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:41:19.397986 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 00:41:19.932875 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:41:19.932845 4939 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-65f77c855d-6tnc2" Nov 03 00:41:20.062846 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:41:20.062814 4939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-rstd6" event={"ID":"49e75d56-a78d-4c7f-88e3-c96e81dc026e","Type":"ContainerStarted","Data":"6a30862c2aa7806fecc371ff80bb55b67fe06eb765e4743fd3ef4224c1dd1a08"} Nov 03 00:41:20.063233 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:41:20.062872 4939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-rstd6" event={"ID":"49e75d56-a78d-4c7f-88e3-c96e81dc026e","Type":"ContainerStarted","Data":"83873ab76adf874d15ca350f0115840cf63d74977d56bbb8f211d01c49767543"} Nov 03 00:41:20.098086 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:41:20.098034 4939 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-rstd6" podStartSLOduration=1.958085944 podStartE2EDuration="3.098020525s" totalImagesPullingTime="1.072862636s" totalInitContainerRuntime="67.071945ms" isStatefulPod=true podCreationTimestamp="2026-11-03 00:41:17 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:41:20.096491118 +0000 UTC m=+28.360826064" watchObservedRunningTime="2026-11-03 00:41:20.098020525 +0000 UTC m=+28.362355514" Nov 03 00:41:20.303857 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:41:20.303775 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 00:41:20.459663 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:41:20.459618 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 00:41:21.893116 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:41:21.893083 4939 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-w46kd" Nov 03 00:41:21.977217 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:41:21.977186 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 00:41:22.155301 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:41:22.155191 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 00:41:22.317059 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:41:22.317023 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 00:41:22.490192 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:41:22.490111 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 00:41:23.608262 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:41:23.608228 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 00:41:23.798842 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:41:23.798809 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 00:41:25.286724 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:41:25.286696 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 00:41:25.422244 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:41:25.422213 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 00:41:25.425288 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:41:25.425268 4939 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-6669dc8887-ws544" Nov 03 00:41:25.650468 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:41:25.650437 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 00:41:25.791043 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:41:25.790989 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 00:41:26.235025 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:41:26.234988 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 00:41:26.366105 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:41:26.366070 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 00:41:28.560120 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:41:28.560084 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 00:41:28.714774 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:41:28.714743 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 00:41:29.243138 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:41:29.243104 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 00:41:29.451276 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:41:29.451245 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 00:41:31.114719 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:41:31.114677 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 00:41:31.276420 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:41:31.276388 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 00:41:32.188342 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:41:32.188311 4939 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-6669dc8887-ws544"] Nov 03 00:41:32.357132 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:41:32.357102 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 00:41:32.552677 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:41:32.552542 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 00:41:33.706251 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:41:33.706175 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 00:41:33.844419 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:41:33.844381 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 00:41:34.947766 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:41:34.947721 4939 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-image-registry/image-registry-65f77c855d-6tnc2" podUID="dd0cd8a1-4d6b-424c-ac39-5f5ddb9cd87e" containerName="registry" containerID="cri-o://80cbb4a91ce3ce0b438f284c9fd8cd746d0c50a49e40a26e8fb62931e7d11b72" gracePeriod=30 Nov 03 00:41:35.139378 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:41:35.139334 4939 generic.go:356] "Generic (PLEG): container finished" podID="dd0cd8a1-4d6b-424c-ac39-5f5ddb9cd87e" containerID="80cbb4a91ce3ce0b438f284c9fd8cd746d0c50a49e40a26e8fb62931e7d11b72" exitCode=0 Nov 03 00:41:35.139530 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:41:35.139390 4939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-65f77c855d-6tnc2" event={"ID":"dd0cd8a1-4d6b-424c-ac39-5f5ddb9cd87e","Type":"ContainerDied","Data":"80cbb4a91ce3ce0b438f284c9fd8cd746d0c50a49e40a26e8fb62931e7d11b72"} Nov 03 00:41:35.191763 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:41:35.191740 4939 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-65f77c855d-6tnc2" Nov 03 00:41:35.225976 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:41:35.225903 4939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/dd0cd8a1-4d6b-424c-ac39-5f5ddb9cd87e-installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/dd0cd8a1-4d6b-424c-ac39-5f5ddb9cd87e-installation-pull-secrets\") pod \"dd0cd8a1-4d6b-424c-ac39-5f5ddb9cd87e\" (UID: \"dd0cd8a1-4d6b-424c-ac39-5f5ddb9cd87e\") " Nov 03 00:41:35.226114 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:41:35.225975 4939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dd0cd8a1-4d6b-424c-ac39-5f5ddb9cd87e-kube-api-access-jbfhc\" (UniqueName: \"kubernetes.io/projected/dd0cd8a1-4d6b-424c-ac39-5f5ddb9cd87e-kube-api-access-jbfhc\") pod \"dd0cd8a1-4d6b-424c-ac39-5f5ddb9cd87e\" (UID: \"dd0cd8a1-4d6b-424c-ac39-5f5ddb9cd87e\") " Nov 03 00:41:35.226357 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:41:35.226334 4939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/dd0cd8a1-4d6b-424c-ac39-5f5ddb9cd87e-registry-certificates\" (UniqueName: \"kubernetes.io/configmap/dd0cd8a1-4d6b-424c-ac39-5f5ddb9cd87e-registry-certificates\") pod \"dd0cd8a1-4d6b-424c-ac39-5f5ddb9cd87e\" (UID: \"dd0cd8a1-4d6b-424c-ac39-5f5ddb9cd87e\") " Nov 03 00:41:35.226458 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:41:35.226394 4939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/dd0cd8a1-4d6b-424c-ac39-5f5ddb9cd87e-image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/dd0cd8a1-4d6b-424c-ac39-5f5ddb9cd87e-image-registry-private-configuration\") pod \"dd0cd8a1-4d6b-424c-ac39-5f5ddb9cd87e\" (UID: \"dd0cd8a1-4d6b-424c-ac39-5f5ddb9cd87e\") " Nov 03 00:41:35.226458 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:41:35.226429 4939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/dd0cd8a1-4d6b-424c-ac39-5f5ddb9cd87e-trusted-ca\" (UniqueName: \"kubernetes.io/configmap/dd0cd8a1-4d6b-424c-ac39-5f5ddb9cd87e-trusted-ca\") pod \"dd0cd8a1-4d6b-424c-ac39-5f5ddb9cd87e\" (UID: \"dd0cd8a1-4d6b-424c-ac39-5f5ddb9cd87e\") " Nov 03 00:41:35.226567 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:41:35.226477 4939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dd0cd8a1-4d6b-424c-ac39-5f5ddb9cd87e-registry-tls\" (UniqueName: \"kubernetes.io/projected/dd0cd8a1-4d6b-424c-ac39-5f5ddb9cd87e-registry-tls\") pod \"dd0cd8a1-4d6b-424c-ac39-5f5ddb9cd87e\" (UID: \"dd0cd8a1-4d6b-424c-ac39-5f5ddb9cd87e\") " Nov 03 00:41:35.226567 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:41:35.226516 4939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dd0cd8a1-4d6b-424c-ac39-5f5ddb9cd87e-ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/dd0cd8a1-4d6b-424c-ac39-5f5ddb9cd87e-ca-trust-extracted\") pod \"dd0cd8a1-4d6b-424c-ac39-5f5ddb9cd87e\" (UID: \"dd0cd8a1-4d6b-424c-ac39-5f5ddb9cd87e\") " Nov 03 00:41:35.226567 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:41:35.226553 4939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dd0cd8a1-4d6b-424c-ac39-5f5ddb9cd87e-bound-sa-token\" (UniqueName: \"kubernetes.io/projected/dd0cd8a1-4d6b-424c-ac39-5f5ddb9cd87e-bound-sa-token\") pod \"dd0cd8a1-4d6b-424c-ac39-5f5ddb9cd87e\" (UID: \"dd0cd8a1-4d6b-424c-ac39-5f5ddb9cd87e\") " Nov 03 00:41:35.227144 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:41:35.227109 4939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dd0cd8a1-4d6b-424c-ac39-5f5ddb9cd87e-trusted-ca" pod "dd0cd8a1-4d6b-424c-ac39-5f5ddb9cd87e" (UID: "dd0cd8a1-4d6b-424c-ac39-5f5ddb9cd87e"). InnerVolumeSpecName "trusted-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 00:41:35.227459 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:41:35.227433 4939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dd0cd8a1-4d6b-424c-ac39-5f5ddb9cd87e-registry-certificates" pod "dd0cd8a1-4d6b-424c-ac39-5f5ddb9cd87e" (UID: "dd0cd8a1-4d6b-424c-ac39-5f5ddb9cd87e"). InnerVolumeSpecName "registry-certificates". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 00:41:35.229810 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:41:35.229784 4939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dd0cd8a1-4d6b-424c-ac39-5f5ddb9cd87e-registry-tls" pod "dd0cd8a1-4d6b-424c-ac39-5f5ddb9cd87e" (UID: "dd0cd8a1-4d6b-424c-ac39-5f5ddb9cd87e"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 00:41:35.229947 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:41:35.229810 4939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/dd0cd8a1-4d6b-424c-ac39-5f5ddb9cd87e-image-registry-private-configuration" pod "dd0cd8a1-4d6b-424c-ac39-5f5ddb9cd87e" (UID: "dd0cd8a1-4d6b-424c-ac39-5f5ddb9cd87e"). InnerVolumeSpecName "image-registry-private-configuration". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 00:41:35.229947 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:41:35.229836 4939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dd0cd8a1-4d6b-424c-ac39-5f5ddb9cd87e-kube-api-access-jbfhc" pod "dd0cd8a1-4d6b-424c-ac39-5f5ddb9cd87e" (UID: "dd0cd8a1-4d6b-424c-ac39-5f5ddb9cd87e"). InnerVolumeSpecName "kube-api-access-jbfhc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 00:41:35.230078 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:41:35.230028 4939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dd0cd8a1-4d6b-424c-ac39-5f5ddb9cd87e-bound-sa-token" pod "dd0cd8a1-4d6b-424c-ac39-5f5ddb9cd87e" (UID: "dd0cd8a1-4d6b-424c-ac39-5f5ddb9cd87e"). InnerVolumeSpecName "bound-sa-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 00:41:35.236970 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:41:35.236940 4939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dd0cd8a1-4d6b-424c-ac39-5f5ddb9cd87e-ca-trust-extracted" pod "dd0cd8a1-4d6b-424c-ac39-5f5ddb9cd87e" (UID: "dd0cd8a1-4d6b-424c-ac39-5f5ddb9cd87e"). InnerVolumeSpecName "ca-trust-extracted". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 00:41:35.238296 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:41:35.238267 4939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/dd0cd8a1-4d6b-424c-ac39-5f5ddb9cd87e-installation-pull-secrets" pod "dd0cd8a1-4d6b-424c-ac39-5f5ddb9cd87e" (UID: "dd0cd8a1-4d6b-424c-ac39-5f5ddb9cd87e"). InnerVolumeSpecName "installation-pull-secrets". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 00:41:35.307553 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:41:35.307524 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 00:41:35.327678 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:41:35.327650 4939 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/dd0cd8a1-4d6b-424c-ac39-5f5ddb9cd87e-registry-tls\") on node \"ip-10-0-140-107.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 00:41:35.327678 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:41:35.327677 4939 reconciler_common.go:299] "Volume detached for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/dd0cd8a1-4d6b-424c-ac39-5f5ddb9cd87e-ca-trust-extracted\") on node \"ip-10-0-140-107.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 00:41:35.327845 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:41:35.327687 4939 reconciler_common.go:299] "Volume detached for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/dd0cd8a1-4d6b-424c-ac39-5f5ddb9cd87e-bound-sa-token\") on node \"ip-10-0-140-107.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 00:41:35.327845 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:41:35.327696 4939 reconciler_common.go:299] "Volume detached for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/dd0cd8a1-4d6b-424c-ac39-5f5ddb9cd87e-installation-pull-secrets\") on node \"ip-10-0-140-107.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 00:41:35.327845 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:41:35.327706 4939 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jbfhc\" (UniqueName: \"kubernetes.io/projected/dd0cd8a1-4d6b-424c-ac39-5f5ddb9cd87e-kube-api-access-jbfhc\") on node \"ip-10-0-140-107.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 00:41:35.327845 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:41:35.327715 4939 reconciler_common.go:299] "Volume detached for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/dd0cd8a1-4d6b-424c-ac39-5f5ddb9cd87e-registry-certificates\") on node \"ip-10-0-140-107.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 00:41:35.327845 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:41:35.327726 4939 reconciler_common.go:299] "Volume detached for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/dd0cd8a1-4d6b-424c-ac39-5f5ddb9cd87e-image-registry-private-configuration\") on node \"ip-10-0-140-107.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 00:41:35.327845 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:41:35.327734 4939 reconciler_common.go:299] "Volume detached for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/dd0cd8a1-4d6b-424c-ac39-5f5ddb9cd87e-trusted-ca\") on node \"ip-10-0-140-107.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 00:41:35.508644 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:41:35.508565 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 00:41:36.144577 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:41:36.144493 4939 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-65f77c855d-6tnc2" Nov 03 00:41:36.144577 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:41:36.144517 4939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-65f77c855d-6tnc2" event={"ID":"dd0cd8a1-4d6b-424c-ac39-5f5ddb9cd87e","Type":"ContainerDied","Data":"66e078530d58e4adef651cba6a19bf8779383083a0ca7d293b4e569c3c0df87d"} Nov 03 00:41:36.145049 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:41:36.144590 4939 scope.go:122] "RemoveContainer" containerID="80cbb4a91ce3ce0b438f284c9fd8cd746d0c50a49e40a26e8fb62931e7d11b72" Nov 03 00:41:36.166588 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:41:36.166566 4939 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-65f77c855d-6tnc2"] Nov 03 00:41:36.175267 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:41:36.175238 4939 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-image-registry/image-registry-65f77c855d-6tnc2"] Nov 03 00:41:37.868599 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:41:37.868560 4939 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="dd0cd8a1-4d6b-424c-ac39-5f5ddb9cd87e" path="/var/lib/kubelet/pods/dd0cd8a1-4d6b-424c-ac39-5f5ddb9cd87e/volumes" Nov 03 00:41:41.306036 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:41:41.305999 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 00:41:41.445531 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:41:41.445493 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 00:41:41.619077 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:41:41.619028 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 00:41:48.028839 ip-10-0-140-107 kubenswrapper[4939]: E1103 00:41:48.028800 4939 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-sthjh" containerName="extractor" Nov 03 00:41:54.228108 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:41:54.228076 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 00:41:56.279489 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:41:56.279453 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 00:41:56.441822 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:41:56.441790 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 00:41:57.208707 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:41:57.208661 4939 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-image-registry/image-registry-6669dc8887-ws544" podUID="52a19842-3750-4d2b-bcb2-aebddee81097" containerName="registry" containerID="cri-o://3630dbb3130ca380f93aa67ccddf7d735254b2bd02fc3dbbcbb37d070881e0a3" gracePeriod=30 Nov 03 00:41:57.452805 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:41:57.452778 4939 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-6669dc8887-ws544" Nov 03 00:41:57.499337 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:41:57.499260 4939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/52a19842-3750-4d2b-bcb2-aebddee81097-bound-sa-token\" (UniqueName: \"kubernetes.io/projected/52a19842-3750-4d2b-bcb2-aebddee81097-bound-sa-token\") pod \"52a19842-3750-4d2b-bcb2-aebddee81097\" (UID: \"52a19842-3750-4d2b-bcb2-aebddee81097\") " Nov 03 00:41:57.499337 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:41:57.499306 4939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/52a19842-3750-4d2b-bcb2-aebddee81097-registry-tls\" (UniqueName: \"kubernetes.io/projected/52a19842-3750-4d2b-bcb2-aebddee81097-registry-tls\") pod \"52a19842-3750-4d2b-bcb2-aebddee81097\" (UID: \"52a19842-3750-4d2b-bcb2-aebddee81097\") " Nov 03 00:41:57.499337 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:41:57.499330 4939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/52a19842-3750-4d2b-bcb2-aebddee81097-kube-api-access-tc9rq\" (UniqueName: \"kubernetes.io/projected/52a19842-3750-4d2b-bcb2-aebddee81097-kube-api-access-tc9rq\") pod \"52a19842-3750-4d2b-bcb2-aebddee81097\" (UID: \"52a19842-3750-4d2b-bcb2-aebddee81097\") " Nov 03 00:41:57.499584 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:41:57.499361 4939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/52a19842-3750-4d2b-bcb2-aebddee81097-installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/52a19842-3750-4d2b-bcb2-aebddee81097-installation-pull-secrets\") pod \"52a19842-3750-4d2b-bcb2-aebddee81097\" (UID: \"52a19842-3750-4d2b-bcb2-aebddee81097\") " Nov 03 00:41:57.499584 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:41:57.499506 4939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/52a19842-3750-4d2b-bcb2-aebddee81097-registry-certificates\" (UniqueName: \"kubernetes.io/configmap/52a19842-3750-4d2b-bcb2-aebddee81097-registry-certificates\") pod \"52a19842-3750-4d2b-bcb2-aebddee81097\" (UID: \"52a19842-3750-4d2b-bcb2-aebddee81097\") " Nov 03 00:41:57.499584 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:41:57.499566 4939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/52a19842-3750-4d2b-bcb2-aebddee81097-image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/52a19842-3750-4d2b-bcb2-aebddee81097-image-registry-private-configuration\") pod \"52a19842-3750-4d2b-bcb2-aebddee81097\" (UID: \"52a19842-3750-4d2b-bcb2-aebddee81097\") " Nov 03 00:41:57.499736 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:41:57.499624 4939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/52a19842-3750-4d2b-bcb2-aebddee81097-trusted-ca\" (UniqueName: \"kubernetes.io/configmap/52a19842-3750-4d2b-bcb2-aebddee81097-trusted-ca\") pod \"52a19842-3750-4d2b-bcb2-aebddee81097\" (UID: \"52a19842-3750-4d2b-bcb2-aebddee81097\") " Nov 03 00:41:57.499736 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:41:57.499662 4939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/52a19842-3750-4d2b-bcb2-aebddee81097-ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/52a19842-3750-4d2b-bcb2-aebddee81097-ca-trust-extracted\") pod \"52a19842-3750-4d2b-bcb2-aebddee81097\" (UID: \"52a19842-3750-4d2b-bcb2-aebddee81097\") " Nov 03 00:41:57.500254 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:41:57.500198 4939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/52a19842-3750-4d2b-bcb2-aebddee81097-registry-certificates" pod "52a19842-3750-4d2b-bcb2-aebddee81097" (UID: "52a19842-3750-4d2b-bcb2-aebddee81097"). InnerVolumeSpecName "registry-certificates". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 00:41:57.500254 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:41:57.500211 4939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/52a19842-3750-4d2b-bcb2-aebddee81097-trusted-ca" pod "52a19842-3750-4d2b-bcb2-aebddee81097" (UID: "52a19842-3750-4d2b-bcb2-aebddee81097"). InnerVolumeSpecName "trusted-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 00:41:57.503179 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:41:57.503150 4939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/52a19842-3750-4d2b-bcb2-aebddee81097-image-registry-private-configuration" pod "52a19842-3750-4d2b-bcb2-aebddee81097" (UID: "52a19842-3750-4d2b-bcb2-aebddee81097"). InnerVolumeSpecName "image-registry-private-configuration". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 00:41:57.503355 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:41:57.503330 4939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/52a19842-3750-4d2b-bcb2-aebddee81097-installation-pull-secrets" pod "52a19842-3750-4d2b-bcb2-aebddee81097" (UID: "52a19842-3750-4d2b-bcb2-aebddee81097"). InnerVolumeSpecName "installation-pull-secrets". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 00:41:57.503421 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:41:57.503364 4939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/52a19842-3750-4d2b-bcb2-aebddee81097-bound-sa-token" pod "52a19842-3750-4d2b-bcb2-aebddee81097" (UID: "52a19842-3750-4d2b-bcb2-aebddee81097"). InnerVolumeSpecName "bound-sa-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 00:41:57.503421 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:41:57.503396 4939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/52a19842-3750-4d2b-bcb2-aebddee81097-registry-tls" pod "52a19842-3750-4d2b-bcb2-aebddee81097" (UID: "52a19842-3750-4d2b-bcb2-aebddee81097"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 00:41:57.503484 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:41:57.503453 4939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/52a19842-3750-4d2b-bcb2-aebddee81097-kube-api-access-tc9rq" pod "52a19842-3750-4d2b-bcb2-aebddee81097" (UID: "52a19842-3750-4d2b-bcb2-aebddee81097"). InnerVolumeSpecName "kube-api-access-tc9rq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 00:41:57.507318 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:41:57.507295 4939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/52a19842-3750-4d2b-bcb2-aebddee81097-ca-trust-extracted" pod "52a19842-3750-4d2b-bcb2-aebddee81097" (UID: "52a19842-3750-4d2b-bcb2-aebddee81097"). InnerVolumeSpecName "ca-trust-extracted". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 00:41:57.600897 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:41:57.600863 4939 reconciler_common.go:299] "Volume detached for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/52a19842-3750-4d2b-bcb2-aebddee81097-trusted-ca\") on node \"ip-10-0-140-107.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 00:41:57.600897 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:41:57.600892 4939 reconciler_common.go:299] "Volume detached for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/52a19842-3750-4d2b-bcb2-aebddee81097-ca-trust-extracted\") on node \"ip-10-0-140-107.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 00:41:57.600897 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:41:57.600902 4939 reconciler_common.go:299] "Volume detached for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/52a19842-3750-4d2b-bcb2-aebddee81097-bound-sa-token\") on node \"ip-10-0-140-107.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 00:41:57.601133 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:41:57.600931 4939 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/52a19842-3750-4d2b-bcb2-aebddee81097-registry-tls\") on node \"ip-10-0-140-107.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 00:41:57.601133 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:41:57.600940 4939 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tc9rq\" (UniqueName: \"kubernetes.io/projected/52a19842-3750-4d2b-bcb2-aebddee81097-kube-api-access-tc9rq\") on node \"ip-10-0-140-107.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 00:41:57.601133 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:41:57.600951 4939 reconciler_common.go:299] "Volume detached for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/52a19842-3750-4d2b-bcb2-aebddee81097-installation-pull-secrets\") on node \"ip-10-0-140-107.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 00:41:57.601133 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:41:57.600960 4939 reconciler_common.go:299] "Volume detached for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/52a19842-3750-4d2b-bcb2-aebddee81097-registry-certificates\") on node \"ip-10-0-140-107.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 00:41:57.601133 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:41:57.600969 4939 reconciler_common.go:299] "Volume detached for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/52a19842-3750-4d2b-bcb2-aebddee81097-image-registry-private-configuration\") on node \"ip-10-0-140-107.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 00:41:58.235642 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:41:58.235548 4939 generic.go:356] "Generic (PLEG): container finished" podID="52a19842-3750-4d2b-bcb2-aebddee81097" containerID="3630dbb3130ca380f93aa67ccddf7d735254b2bd02fc3dbbcbb37d070881e0a3" exitCode=0 Nov 03 00:41:58.235642 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:41:58.235602 4939 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-6669dc8887-ws544" Nov 03 00:41:58.235814 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:41:58.235635 4939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-6669dc8887-ws544" event={"ID":"52a19842-3750-4d2b-bcb2-aebddee81097","Type":"ContainerDied","Data":"3630dbb3130ca380f93aa67ccddf7d735254b2bd02fc3dbbcbb37d070881e0a3"} Nov 03 00:41:58.235814 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:41:58.235717 4939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-6669dc8887-ws544" event={"ID":"52a19842-3750-4d2b-bcb2-aebddee81097","Type":"ContainerDied","Data":"5932df9ab8e5f4e60993932d77de8e89b0058d1d12084f9c72538cf51c805fd0"} Nov 03 00:41:58.235814 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:41:58.235742 4939 scope.go:122] "RemoveContainer" containerID="3630dbb3130ca380f93aa67ccddf7d735254b2bd02fc3dbbcbb37d070881e0a3" Nov 03 00:41:58.243687 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:41:58.243670 4939 scope.go:122] "RemoveContainer" containerID="3630dbb3130ca380f93aa67ccddf7d735254b2bd02fc3dbbcbb37d070881e0a3" Nov 03 00:41:58.243981 ip-10-0-140-107 kubenswrapper[4939]: E1103 00:41:58.243961 4939 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3630dbb3130ca380f93aa67ccddf7d735254b2bd02fc3dbbcbb37d070881e0a3\": container with ID starting with 3630dbb3130ca380f93aa67ccddf7d735254b2bd02fc3dbbcbb37d070881e0a3 not found: ID does not exist" containerID="3630dbb3130ca380f93aa67ccddf7d735254b2bd02fc3dbbcbb37d070881e0a3" Nov 03 00:41:58.244039 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:41:58.243989 4939 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3630dbb3130ca380f93aa67ccddf7d735254b2bd02fc3dbbcbb37d070881e0a3"} err="failed to get container status \"3630dbb3130ca380f93aa67ccddf7d735254b2bd02fc3dbbcbb37d070881e0a3\": rpc error: code = NotFound desc = could not find container \"3630dbb3130ca380f93aa67ccddf7d735254b2bd02fc3dbbcbb37d070881e0a3\": container with ID starting with 3630dbb3130ca380f93aa67ccddf7d735254b2bd02fc3dbbcbb37d070881e0a3 not found: ID does not exist" Nov 03 00:41:58.257134 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:41:58.257109 4939 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-6669dc8887-ws544"] Nov 03 00:41:58.260386 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:41:58.260363 4939 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-image-registry/image-registry-6669dc8887-ws544"] Nov 03 00:41:59.865189 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:41:59.865154 4939 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="52a19842-3750-4d2b-bcb2-aebddee81097" path="/var/lib/kubelet/pods/52a19842-3750-4d2b-bcb2-aebddee81097/volumes" Nov 03 00:42:02.252249 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:42:02.252210 4939 generic.go:356] "Generic (PLEG): container finished" podID="ca0f883c-c2b7-4bf6-a90b-ffdbeac161a8" containerID="03003b0a16b86bfd908b95260e3a4239e99705e71bed47a1b9fa49354eeab6db" exitCode=0 Nov 03 00:42:02.252618 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:42:02.252277 4939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-7c4986d449-xprcf" event={"ID":"ca0f883c-c2b7-4bf6-a90b-ffdbeac161a8","Type":"ContainerDied","Data":"03003b0a16b86bfd908b95260e3a4239e99705e71bed47a1b9fa49354eeab6db"} Nov 03 00:42:02.252618 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:42:02.252339 4939 scope.go:122] "RemoveContainer" containerID="90ac319dc159fa4199add901b66cd1e68b419e073b0586dd5ee7fdd7ce194c3d" Nov 03 00:42:02.252792 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:42:02.252638 4939 scope.go:122] "RemoveContainer" containerID="03003b0a16b86bfd908b95260e3a4239e99705e71bed47a1b9fa49354eeab6db" Nov 03 00:42:02.252853 ip-10-0-140-107 kubenswrapper[4939]: E1103 00:42:02.252828 4939 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"insights-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=insights-operator pod=insights-operator-7c4986d449-xprcf_openshift-insights(ca0f883c-c2b7-4bf6-a90b-ffdbeac161a8)\"" pod="openshift-insights/insights-operator-7c4986d449-xprcf" podUID="ca0f883c-c2b7-4bf6-a90b-ffdbeac161a8" Nov 03 00:42:03.465861 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:42:03.465820 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 00:42:03.597672 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:42:03.597636 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 00:42:04.743856 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:42:04.743817 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 00:42:04.873433 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:42:04.873398 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 00:42:05.102511 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:42:05.102482 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 00:42:05.232621 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:42:05.232591 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 00:42:14.861493 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:42:14.861461 4939 scope.go:122] "RemoveContainer" containerID="03003b0a16b86bfd908b95260e3a4239e99705e71bed47a1b9fa49354eeab6db" Nov 03 00:42:15.306562 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:42:15.306482 4939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-7c4986d449-xprcf" event={"ID":"ca0f883c-c2b7-4bf6-a90b-ffdbeac161a8","Type":"ContainerStarted","Data":"9b7d3e48090ace0311101ddda02a66271bea16c6ee79194bfd74d8cbd99f9062"} Nov 03 00:42:16.115007 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:42:16.114971 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 00:42:16.232337 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:42:16.232300 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 00:42:18.029279 ip-10-0-140-107 kubenswrapper[4939]: E1103 00:42:18.029243 4939 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-sthjh" containerName="extractor" Nov 03 00:42:26.295747 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:42:26.295710 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 00:42:26.351163 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:42:26.351128 4939 generic.go:356] "Generic (PLEG): container finished" podID="36b9d258-cb2c-4d2b-880f-ae93b065034c" containerID="00b1e2838a30893cde164c35041c848e8bd12adedde9a1f06b814d763f2b70dd" exitCode=0 Nov 03 00:42:26.351297 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:42:26.351199 4939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-khmv7" event={"ID":"36b9d258-cb2c-4d2b-880f-ae93b065034c","Type":"ContainerDied","Data":"00b1e2838a30893cde164c35041c848e8bd12adedde9a1f06b814d763f2b70dd"} Nov 03 00:42:26.351526 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:42:26.351508 4939 scope.go:122] "RemoveContainer" containerID="00b1e2838a30893cde164c35041c848e8bd12adedde9a1f06b814d763f2b70dd" Nov 03 00:42:26.422590 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:42:26.422554 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 00:42:27.356643 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:42:27.356612 4939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-khmv7" event={"ID":"36b9d258-cb2c-4d2b-880f-ae93b065034c","Type":"ContainerStarted","Data":"4f4f8f348289d6135f0bada7dd209ad91611174e39baa976bdb730e12da958f6"} Nov 03 00:42:41.410205 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:42:41.410166 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 00:42:48.028980 ip-10-0-140-107 kubenswrapper[4939]: E1103 00:42:48.028935 4939 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-sthjh" containerName="extractor" Nov 03 00:43:05.754981 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:43:05.754876 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 00:43:18.029993 ip-10-0-140-107 kubenswrapper[4939]: E1103 00:43:18.029951 4939 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-sthjh" containerName="extractor" Nov 03 00:43:41.475097 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:43:41.475064 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 00:43:48.028486 ip-10-0-140-107 kubenswrapper[4939]: E1103 00:43:48.028451 4939 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-sthjh" containerName="extractor" Nov 03 00:44:07.715035 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:44:07.714976 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 00:44:07.923401 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:44:07.923360 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 00:44:08.083780 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:44:08.083753 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 00:44:09.241756 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:44:09.241725 4939 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7876bfcb74-jbfv6"] Nov 03 00:44:09.242138 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:44:09.242097 4939 cpu_manager.go:414] "containerMap: removing container" podUID="dd0cd8a1-4d6b-424c-ac39-5f5ddb9cd87e" containerName="registry" Nov 03 00:44:09.242138 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:44:09.242111 4939 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dd0cd8a1-4d6b-424c-ac39-5f5ddb9cd87e" containerName="registry" Nov 03 00:44:09.242138 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:44:09.242138 4939 cpu_manager.go:414] "containerMap: removing container" podUID="52a19842-3750-4d2b-bcb2-aebddee81097" containerName="registry" Nov 03 00:44:09.242232 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:44:09.242143 4939 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="52a19842-3750-4d2b-bcb2-aebddee81097" containerName="registry" Nov 03 00:44:09.242267 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:44:09.242231 4939 memory_manager.go:365] "RemoveStaleState removing state" podUID="52a19842-3750-4d2b-bcb2-aebddee81097" containerName="registry" Nov 03 00:44:09.242267 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:44:09.242244 4939 memory_manager.go:365] "RemoveStaleState removing state" podUID="dd0cd8a1-4d6b-424c-ac39-5f5ddb9cd87e" containerName="registry" Nov 03 00:44:09.245119 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:44:09.245104 4939 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-jbfv6" Nov 03 00:44:09.252670 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:44:09.252648 4939 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istio-kubeconfig\"" Nov 03 00:44:09.252782 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:44:09.252666 4939 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-tls\"" Nov 03 00:44:09.252782 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:44:09.252681 4939 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\"" Nov 03 00:44:09.252782 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:44:09.252667 4939 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"cacerts\"" Nov 03 00:44:09.252782 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:44:09.252703 4939 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-openshift-gateway-dockercfg-9vdqv\"" Nov 03 00:44:09.267846 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:44:09.267815 4939 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7876bfcb74-jbfv6"] Nov 03 00:44:09.307670 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:44:09.307636 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/16fe2330-9a44-44c9-b41b-ff22f7648ad8-local-certs\") pod \"istiod-openshift-gateway-7876bfcb74-jbfv6\" (UID: \"16fe2330-9a44-44c9-b41b-ff22f7648ad8\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-jbfv6" Nov 03 00:44:09.307835 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:44:09.307683 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/16fe2330-9a44-44c9-b41b-ff22f7648ad8-istio-kubeconfig\") pod \"istiod-openshift-gateway-7876bfcb74-jbfv6\" (UID: \"16fe2330-9a44-44c9-b41b-ff22f7648ad8\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-jbfv6" Nov 03 00:44:09.307835 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:44:09.307702 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/16fe2330-9a44-44c9-b41b-ff22f7648ad8-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7876bfcb74-jbfv6\" (UID: \"16fe2330-9a44-44c9-b41b-ff22f7648ad8\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-jbfv6" Nov 03 00:44:09.307835 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:44:09.307804 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/16fe2330-9a44-44c9-b41b-ff22f7648ad8-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7876bfcb74-jbfv6\" (UID: \"16fe2330-9a44-44c9-b41b-ff22f7648ad8\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-jbfv6" Nov 03 00:44:09.308027 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:44:09.307842 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/16fe2330-9a44-44c9-b41b-ff22f7648ad8-istio-token\") pod \"istiod-openshift-gateway-7876bfcb74-jbfv6\" (UID: \"16fe2330-9a44-44c9-b41b-ff22f7648ad8\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-jbfv6" Nov 03 00:44:09.308027 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:44:09.307993 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bscbp\" (UniqueName: \"kubernetes.io/projected/16fe2330-9a44-44c9-b41b-ff22f7648ad8-kube-api-access-bscbp\") pod \"istiod-openshift-gateway-7876bfcb74-jbfv6\" (UID: \"16fe2330-9a44-44c9-b41b-ff22f7648ad8\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-jbfv6" Nov 03 00:44:09.308117 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:44:09.308033 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/16fe2330-9a44-44c9-b41b-ff22f7648ad8-cacerts\") pod \"istiod-openshift-gateway-7876bfcb74-jbfv6\" (UID: \"16fe2330-9a44-44c9-b41b-ff22f7648ad8\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-jbfv6" Nov 03 00:44:09.323130 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:44:09.323104 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 00:44:09.409260 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:44:09.409222 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bscbp\" (UniqueName: \"kubernetes.io/projected/16fe2330-9a44-44c9-b41b-ff22f7648ad8-kube-api-access-bscbp\") pod \"istiod-openshift-gateway-7876bfcb74-jbfv6\" (UID: \"16fe2330-9a44-44c9-b41b-ff22f7648ad8\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-jbfv6" Nov 03 00:44:09.409424 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:44:09.409266 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/16fe2330-9a44-44c9-b41b-ff22f7648ad8-cacerts\") pod \"istiod-openshift-gateway-7876bfcb74-jbfv6\" (UID: \"16fe2330-9a44-44c9-b41b-ff22f7648ad8\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-jbfv6" Nov 03 00:44:09.409424 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:44:09.409311 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/16fe2330-9a44-44c9-b41b-ff22f7648ad8-local-certs\") pod \"istiod-openshift-gateway-7876bfcb74-jbfv6\" (UID: \"16fe2330-9a44-44c9-b41b-ff22f7648ad8\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-jbfv6" Nov 03 00:44:09.409424 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:44:09.409350 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/16fe2330-9a44-44c9-b41b-ff22f7648ad8-istio-kubeconfig\") pod \"istiod-openshift-gateway-7876bfcb74-jbfv6\" (UID: \"16fe2330-9a44-44c9-b41b-ff22f7648ad8\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-jbfv6" Nov 03 00:44:09.409424 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:44:09.409374 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/16fe2330-9a44-44c9-b41b-ff22f7648ad8-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7876bfcb74-jbfv6\" (UID: \"16fe2330-9a44-44c9-b41b-ff22f7648ad8\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-jbfv6" Nov 03 00:44:09.409424 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:44:09.409410 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/16fe2330-9a44-44c9-b41b-ff22f7648ad8-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7876bfcb74-jbfv6\" (UID: \"16fe2330-9a44-44c9-b41b-ff22f7648ad8\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-jbfv6" Nov 03 00:44:09.409665 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:44:09.409598 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/16fe2330-9a44-44c9-b41b-ff22f7648ad8-istio-token\") pod \"istiod-openshift-gateway-7876bfcb74-jbfv6\" (UID: \"16fe2330-9a44-44c9-b41b-ff22f7648ad8\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-jbfv6" Nov 03 00:44:09.410088 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:44:09.410061 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/16fe2330-9a44-44c9-b41b-ff22f7648ad8-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7876bfcb74-jbfv6\" (UID: \"16fe2330-9a44-44c9-b41b-ff22f7648ad8\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-jbfv6" Nov 03 00:44:09.411841 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:44:09.411809 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/16fe2330-9a44-44c9-b41b-ff22f7648ad8-cacerts\") pod \"istiod-openshift-gateway-7876bfcb74-jbfv6\" (UID: \"16fe2330-9a44-44c9-b41b-ff22f7648ad8\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-jbfv6" Nov 03 00:44:09.412057 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:44:09.411859 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/16fe2330-9a44-44c9-b41b-ff22f7648ad8-istio-kubeconfig\") pod \"istiod-openshift-gateway-7876bfcb74-jbfv6\" (UID: \"16fe2330-9a44-44c9-b41b-ff22f7648ad8\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-jbfv6" Nov 03 00:44:09.412110 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:44:09.412006 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/16fe2330-9a44-44c9-b41b-ff22f7648ad8-local-certs\") pod \"istiod-openshift-gateway-7876bfcb74-jbfv6\" (UID: \"16fe2330-9a44-44c9-b41b-ff22f7648ad8\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-jbfv6" Nov 03 00:44:09.412149 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:44:09.412129 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/16fe2330-9a44-44c9-b41b-ff22f7648ad8-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7876bfcb74-jbfv6\" (UID: \"16fe2330-9a44-44c9-b41b-ff22f7648ad8\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-jbfv6" Nov 03 00:44:09.421313 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:44:09.421292 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/16fe2330-9a44-44c9-b41b-ff22f7648ad8-istio-token\") pod \"istiod-openshift-gateway-7876bfcb74-jbfv6\" (UID: \"16fe2330-9a44-44c9-b41b-ff22f7648ad8\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-jbfv6" Nov 03 00:44:09.421394 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:44:09.421344 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bscbp\" (UniqueName: \"kubernetes.io/projected/16fe2330-9a44-44c9-b41b-ff22f7648ad8-kube-api-access-bscbp\") pod \"istiod-openshift-gateway-7876bfcb74-jbfv6\" (UID: \"16fe2330-9a44-44c9-b41b-ff22f7648ad8\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-jbfv6" Nov 03 00:44:09.472829 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:44:09.472799 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 00:44:09.555686 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:44:09.555604 4939 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-jbfv6" Nov 03 00:44:09.836075 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:44:09.836046 4939 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7876bfcb74-jbfv6"] Nov 03 00:44:09.838997 ip-10-0-140-107 kubenswrapper[4939]: W1103 00:44:09.838966 4939 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod16fe2330_9a44_44c9_b41b_ff22f7648ad8.slice/crio-f77aa67c45a93e00736ca89243cf71e787a2e2046d00761c9e0831d51c177558 WatchSource:0}: Error finding container f77aa67c45a93e00736ca89243cf71e787a2e2046d00761c9e0831d51c177558: Status 404 returned error can't find the container with id f77aa67c45a93e00736ca89243cf71e787a2e2046d00761c9e0831d51c177558 Nov 03 00:44:10.592210 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:44:10.592150 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 00:44:10.723070 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:44:10.723011 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 00:44:10.726372 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:44:10.726345 4939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-jbfv6" event={"ID":"16fe2330-9a44-44c9-b41b-ff22f7648ad8","Type":"ContainerStarted","Data":"f77aa67c45a93e00736ca89243cf71e787a2e2046d00761c9e0831d51c177558"} Nov 03 00:44:12.873551 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:44:12.873521 4939 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 00:44:12.873800 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:44:12.873586 4939 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 00:44:13.282440 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:44:13.282357 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 00:44:13.412041 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:44:13.412005 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 00:44:13.618849 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:44:13.618812 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 00:44:13.738215 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:44:13.738173 4939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-jbfv6" event={"ID":"16fe2330-9a44-44c9-b41b-ff22f7648ad8","Type":"ContainerStarted","Data":"1b1dda8832529b7eac7da9762fc7ebfc6b552dbdb34478b49817b21e8f860e67"} Nov 03 00:44:13.738407 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:44:13.738383 4939 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-jbfv6" Nov 03 00:44:13.739971 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:44:13.739942 4939 patch_prober.go:29] interesting pod/istiod-openshift-gateway-7876bfcb74-jbfv6 container/discovery namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 03 00:44:13.740094 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:44:13.739991 4939 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-jbfv6" podUID="16fe2330-9a44-44c9-b41b-ff22f7648ad8" containerName="discovery" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 03 00:44:13.746712 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:44:13.746665 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 00:44:13.762979 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:44:13.762939 4939 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-jbfv6" podStartSLOduration=1.7310338650000001 podStartE2EDuration="4.762927585s" totalImagesPullingTime="3.03189372s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:44:09 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:44:13.76134469 +0000 UTC m=+202.025679634" watchObservedRunningTime="2026-11-03 00:44:13.762927585 +0000 UTC m=+202.027262523" Nov 03 00:44:13.877544 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:44:13.877419 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 00:44:14.742739 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:44:14.742714 4939 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-jbfv6" Nov 03 00:44:16.309349 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:44:16.309314 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 00:44:18.029078 ip-10-0-140-107 kubenswrapper[4939]: E1103 00:44:18.029046 4939 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-sthjh" containerName="extractor" Nov 03 00:44:41.499090 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:44:41.499005 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 00:44:48.028503 ip-10-0-140-107 kubenswrapper[4939]: E1103 00:44:48.028463 4939 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-sthjh" containerName="extractor" Nov 03 00:44:50.308730 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:44:50.308694 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 00:44:50.456510 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:44:50.456481 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 00:44:50.895182 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:44:50.895147 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 00:44:51.110291 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:44:51.110259 4939 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub-operator-system/opendatahub-operator-controller-manager-7cc9488979-jh6tz"] Nov 03 00:44:51.119805 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:44:51.119788 4939 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-7cc9488979-jh6tz" Nov 03 00:44:51.149968 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:44:51.149891 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 00:44:51.185515 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:44:51.185493 4939 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-controller-manager-service-cert\"" Nov 03 00:44:51.185515 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:44:51.185495 4939 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-controller-webhook-cert\"" Nov 03 00:44:51.185701 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:44:51.185620 4939 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-controller-manager-dockercfg-p7qvx\"" Nov 03 00:44:51.185863 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:44:51.185760 4939 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-metrics-tls\"" Nov 03 00:44:51.187990 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:44:51.187971 4939 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub-operator-system\"/\"openshift-service-ca.crt\"" Nov 03 00:44:51.188104 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:44:51.188090 4939 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub-operator-system\"/\"kube-root-ca.crt\"" Nov 03 00:44:51.208975 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:44:51.208939 4939 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub-operator-system/opendatahub-operator-controller-manager-7cc9488979-jh6tz"] Nov 03 00:44:51.233592 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:44:51.233566 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/f90f3499-7ad4-43f7-9b32-238b34c422c1-manifests\") pod \"opendatahub-operator-controller-manager-7cc9488979-jh6tz\" (UID: \"f90f3499-7ad4-43f7-9b32-238b34c422c1\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-7cc9488979-jh6tz" Nov 03 00:44:51.233707 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:44:51.233608 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/f90f3499-7ad4-43f7-9b32-238b34c422c1-metrics-cert\") pod \"opendatahub-operator-controller-manager-7cc9488979-jh6tz\" (UID: \"f90f3499-7ad4-43f7-9b32-238b34c422c1\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-7cc9488979-jh6tz" Nov 03 00:44:51.233707 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:44:51.233636 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/f90f3499-7ad4-43f7-9b32-238b34c422c1-webhook-cert\") pod \"opendatahub-operator-controller-manager-7cc9488979-jh6tz\" (UID: \"f90f3499-7ad4-43f7-9b32-238b34c422c1\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-7cc9488979-jh6tz" Nov 03 00:44:51.233813 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:44:51.233726 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/f90f3499-7ad4-43f7-9b32-238b34c422c1-tmp\") pod \"opendatahub-operator-controller-manager-7cc9488979-jh6tz\" (UID: \"f90f3499-7ad4-43f7-9b32-238b34c422c1\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-7cc9488979-jh6tz" Nov 03 00:44:51.233870 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:44:51.233841 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/f90f3499-7ad4-43f7-9b32-238b34c422c1-apiservice-cert\") pod \"opendatahub-operator-controller-manager-7cc9488979-jh6tz\" (UID: \"f90f3499-7ad4-43f7-9b32-238b34c422c1\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-7cc9488979-jh6tz" Nov 03 00:44:51.233939 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:44:51.233888 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nmbgt\" (UniqueName: \"kubernetes.io/projected/f90f3499-7ad4-43f7-9b32-238b34c422c1-kube-api-access-nmbgt\") pod \"opendatahub-operator-controller-manager-7cc9488979-jh6tz\" (UID: \"f90f3499-7ad4-43f7-9b32-238b34c422c1\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-7cc9488979-jh6tz" Nov 03 00:44:51.335092 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:44:51.335059 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/f90f3499-7ad4-43f7-9b32-238b34c422c1-metrics-cert\") pod \"opendatahub-operator-controller-manager-7cc9488979-jh6tz\" (UID: \"f90f3499-7ad4-43f7-9b32-238b34c422c1\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-7cc9488979-jh6tz" Nov 03 00:44:51.335092 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:44:51.335095 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/f90f3499-7ad4-43f7-9b32-238b34c422c1-webhook-cert\") pod \"opendatahub-operator-controller-manager-7cc9488979-jh6tz\" (UID: \"f90f3499-7ad4-43f7-9b32-238b34c422c1\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-7cc9488979-jh6tz" Nov 03 00:44:51.335581 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:44:51.335114 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/f90f3499-7ad4-43f7-9b32-238b34c422c1-tmp\") pod \"opendatahub-operator-controller-manager-7cc9488979-jh6tz\" (UID: \"f90f3499-7ad4-43f7-9b32-238b34c422c1\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-7cc9488979-jh6tz" Nov 03 00:44:51.335581 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:44:51.335164 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/f90f3499-7ad4-43f7-9b32-238b34c422c1-apiservice-cert\") pod \"opendatahub-operator-controller-manager-7cc9488979-jh6tz\" (UID: \"f90f3499-7ad4-43f7-9b32-238b34c422c1\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-7cc9488979-jh6tz" Nov 03 00:44:51.335581 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:44:51.335199 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nmbgt\" (UniqueName: \"kubernetes.io/projected/f90f3499-7ad4-43f7-9b32-238b34c422c1-kube-api-access-nmbgt\") pod \"opendatahub-operator-controller-manager-7cc9488979-jh6tz\" (UID: \"f90f3499-7ad4-43f7-9b32-238b34c422c1\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-7cc9488979-jh6tz" Nov 03 00:44:51.335581 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:44:51.335245 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/f90f3499-7ad4-43f7-9b32-238b34c422c1-manifests\") pod \"opendatahub-operator-controller-manager-7cc9488979-jh6tz\" (UID: \"f90f3499-7ad4-43f7-9b32-238b34c422c1\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-7cc9488979-jh6tz" Nov 03 00:44:51.335792 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:44:51.335640 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/f90f3499-7ad4-43f7-9b32-238b34c422c1-manifests\") pod \"opendatahub-operator-controller-manager-7cc9488979-jh6tz\" (UID: \"f90f3499-7ad4-43f7-9b32-238b34c422c1\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-7cc9488979-jh6tz" Nov 03 00:44:51.337665 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:44:51.337634 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/f90f3499-7ad4-43f7-9b32-238b34c422c1-tmp\") pod \"opendatahub-operator-controller-manager-7cc9488979-jh6tz\" (UID: \"f90f3499-7ad4-43f7-9b32-238b34c422c1\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-7cc9488979-jh6tz" Nov 03 00:44:51.337803 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:44:51.337686 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/f90f3499-7ad4-43f7-9b32-238b34c422c1-apiservice-cert\") pod \"opendatahub-operator-controller-manager-7cc9488979-jh6tz\" (UID: \"f90f3499-7ad4-43f7-9b32-238b34c422c1\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-7cc9488979-jh6tz" Nov 03 00:44:51.337876 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:44:51.337800 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/f90f3499-7ad4-43f7-9b32-238b34c422c1-metrics-cert\") pod \"opendatahub-operator-controller-manager-7cc9488979-jh6tz\" (UID: \"f90f3499-7ad4-43f7-9b32-238b34c422c1\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-7cc9488979-jh6tz" Nov 03 00:44:51.337992 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:44:51.337893 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/f90f3499-7ad4-43f7-9b32-238b34c422c1-webhook-cert\") pod \"opendatahub-operator-controller-manager-7cc9488979-jh6tz\" (UID: \"f90f3499-7ad4-43f7-9b32-238b34c422c1\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-7cc9488979-jh6tz" Nov 03 00:44:51.347199 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:44:51.347178 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nmbgt\" (UniqueName: \"kubernetes.io/projected/f90f3499-7ad4-43f7-9b32-238b34c422c1-kube-api-access-nmbgt\") pod \"opendatahub-operator-controller-manager-7cc9488979-jh6tz\" (UID: \"f90f3499-7ad4-43f7-9b32-238b34c422c1\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-7cc9488979-jh6tz" Nov 03 00:44:51.360483 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:44:51.360459 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 00:44:51.435320 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:44:51.435263 4939 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-7cc9488979-jh6tz" Nov 03 00:44:51.491903 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:44:51.491846 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 00:44:51.628654 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:44:51.628581 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 00:44:51.734555 ip-10-0-140-107 kubenswrapper[4939]: W1103 00:44:51.734527 4939 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf90f3499_7ad4_43f7_9b32_238b34c422c1.slice/crio-87ab950e52f806eae440fef8cfc1c91c2779f26fffc09520d98937912edf0623 WatchSource:0}: Error finding container 87ab950e52f806eae440fef8cfc1c91c2779f26fffc09520d98937912edf0623: Status 404 returned error can't find the container with id 87ab950e52f806eae440fef8cfc1c91c2779f26fffc09520d98937912edf0623 Nov 03 00:44:51.735885 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:44:51.735864 4939 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub-operator-system/opendatahub-operator-controller-manager-7cc9488979-jh6tz"] Nov 03 00:44:51.820279 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:44:51.820249 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 00:44:51.873966 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:44:51.873931 4939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-7cc9488979-jh6tz" event={"ID":"f90f3499-7ad4-43f7-9b32-238b34c422c1","Type":"ContainerStarted","Data":"87ab950e52f806eae440fef8cfc1c91c2779f26fffc09520d98937912edf0623"} Nov 03 00:44:53.673395 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:44:53.673363 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 00:44:53.793653 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:44:53.793617 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 00:44:54.919397 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:44:54.919347 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 00:44:55.099657 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:44:55.099626 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 00:44:58.547375 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:44:58.547337 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 00:44:58.740879 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:44:58.740846 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 00:44:59.377856 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:44:59.377813 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 00:44:59.904405 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:44:59.904311 4939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-7cc9488979-jh6tz" event={"ID":"f90f3499-7ad4-43f7-9b32-238b34c422c1","Type":"ContainerStarted","Data":"27f214ef763362dd9b961158d364f405bdb89249bc8e160da83259bd15057ca6"} Nov 03 00:45:00.909502 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:45:00.909470 4939 generic.go:356] "Generic (PLEG): container finished" podID="f90f3499-7ad4-43f7-9b32-238b34c422c1" containerID="27f214ef763362dd9b961158d364f405bdb89249bc8e160da83259bd15057ca6" exitCode=0 Nov 03 00:45:00.910020 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:45:00.909532 4939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-7cc9488979-jh6tz" event={"ID":"f90f3499-7ad4-43f7-9b32-238b34c422c1","Type":"ContainerDied","Data":"27f214ef763362dd9b961158d364f405bdb89249bc8e160da83259bd15057ca6"} Nov 03 00:45:01.915108 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:45:01.915072 4939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-7cc9488979-jh6tz" event={"ID":"f90f3499-7ad4-43f7-9b32-238b34c422c1","Type":"ContainerStarted","Data":"ab917425545b6ffb1d4fdf178d3aff3f275c499279be1f48b072af04d7eb8607"} Nov 03 00:45:01.915573 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:45:01.915165 4939 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-7cc9488979-jh6tz" Nov 03 00:45:01.966251 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:45:01.966183 4939 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-7cc9488979-jh6tz" podStartSLOduration=2.141459399 podStartE2EDuration="10.96616676s" totalImagesPullingTime="8.373680494s" totalInitContainerRuntime="451.026867ms" isStatefulPod=false podCreationTimestamp="2026-11-03 00:44:51 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:45:01.965190004 +0000 UTC m=+250.229524948" watchObservedRunningTime="2026-11-03 00:45:01.96616676 +0000 UTC m=+250.230501705" Nov 03 00:45:03.243930 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:45:03.243882 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 00:45:03.730669 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:45:03.730631 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 00:45:04.482756 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:45:04.482720 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 00:45:05.652879 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:45:05.652848 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 00:45:08.934819 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:45:08.934784 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 00:45:10.850334 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:45:10.850304 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 00:45:11.040924 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:45:11.040867 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 00:45:11.151474 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:45:11.151399 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 00:45:11.857337 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:45:11.857287 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 00:45:12.467681 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:45:12.467643 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 00:45:12.644104 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:45:12.644071 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 00:45:12.824170 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:45:12.824096 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 00:45:12.920826 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:45:12.920799 4939 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-7cc9488979-jh6tz" Nov 03 00:45:15.027875 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:45:15.027845 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 00:45:15.480010 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:45:15.479971 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 00:45:16.471072 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:45:16.471023 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 00:45:18.029407 ip-10-0-140-107 kubenswrapper[4939]: E1103 00:45:18.029368 4939 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-sthjh" containerName="extractor" Nov 03 00:45:18.949595 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:45:18.949559 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 00:45:19.100739 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:45:19.100709 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 00:45:19.239739 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:45:19.239661 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 00:45:19.350002 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:45:19.349970 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 00:45:19.873428 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:45:19.873392 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 00:45:19.997466 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:45:19.997407 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 00:45:20.864877 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:45:20.864833 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 00:45:21.283468 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:45:21.283400 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 00:45:22.480716 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:45:22.480687 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 00:45:22.602306 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:45:22.602271 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 00:45:22.721365 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:45:22.721318 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 00:45:23.274100 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:45:23.274072 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 00:45:27.238488 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:45:27.238452 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 00:45:27.353216 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:45:27.353186 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 00:45:28.455475 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:45:28.455444 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 00:45:28.597672 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:45:28.597639 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 00:45:29.693747 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:45:29.693696 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 00:45:29.827110 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:45:29.827067 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 00:45:30.538939 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:45:30.538876 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 00:45:30.691738 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:45:30.691703 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 00:45:30.754902 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:45:30.754871 4939 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/kube-auth-proxy-594b68584c-km4bh"] Nov 03 00:45:30.758322 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:45:30.758305 4939 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-594b68584c-km4bh" Nov 03 00:45:30.776445 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:45:30.776419 4939 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-594b68584c-km4bh"] Nov 03 00:45:30.782056 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:45:30.782024 4939 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-tls\"" Nov 03 00:45:30.782180 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:45:30.782076 4939 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-creds\"" Nov 03 00:45:30.782240 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:45:30.782231 4939 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-dockercfg-xfr5n\"" Nov 03 00:45:30.846291 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:45:30.846260 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 00:45:30.860403 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:45:30.860375 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6dvt2\" (UniqueName: \"kubernetes.io/projected/5acaee44-b6e2-4504-aebc-aa853d707830-kube-api-access-6dvt2\") pod \"kube-auth-proxy-594b68584c-km4bh\" (UID: \"5acaee44-b6e2-4504-aebc-aa853d707830\") " pod="openshift-ingress/kube-auth-proxy-594b68584c-km4bh" Nov 03 00:45:30.860533 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:45:30.860414 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5acaee44-b6e2-4504-aebc-aa853d707830-tls-certs\") pod \"kube-auth-proxy-594b68584c-km4bh\" (UID: \"5acaee44-b6e2-4504-aebc-aa853d707830\") " pod="openshift-ingress/kube-auth-proxy-594b68584c-km4bh" Nov 03 00:45:30.860533 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:45:30.860476 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/5acaee44-b6e2-4504-aebc-aa853d707830-tmp\") pod \"kube-auth-proxy-594b68584c-km4bh\" (UID: \"5acaee44-b6e2-4504-aebc-aa853d707830\") " pod="openshift-ingress/kube-auth-proxy-594b68584c-km4bh" Nov 03 00:45:30.961305 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:45:30.961274 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6dvt2\" (UniqueName: \"kubernetes.io/projected/5acaee44-b6e2-4504-aebc-aa853d707830-kube-api-access-6dvt2\") pod \"kube-auth-proxy-594b68584c-km4bh\" (UID: \"5acaee44-b6e2-4504-aebc-aa853d707830\") " pod="openshift-ingress/kube-auth-proxy-594b68584c-km4bh" Nov 03 00:45:30.961475 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:45:30.961317 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5acaee44-b6e2-4504-aebc-aa853d707830-tls-certs\") pod \"kube-auth-proxy-594b68584c-km4bh\" (UID: \"5acaee44-b6e2-4504-aebc-aa853d707830\") " pod="openshift-ingress/kube-auth-proxy-594b68584c-km4bh" Nov 03 00:45:30.961475 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:45:30.961373 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/5acaee44-b6e2-4504-aebc-aa853d707830-tmp\") pod \"kube-auth-proxy-594b68584c-km4bh\" (UID: \"5acaee44-b6e2-4504-aebc-aa853d707830\") " pod="openshift-ingress/kube-auth-proxy-594b68584c-km4bh" Nov 03 00:45:30.963729 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:45:30.963702 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/5acaee44-b6e2-4504-aebc-aa853d707830-tmp\") pod \"kube-auth-proxy-594b68584c-km4bh\" (UID: \"5acaee44-b6e2-4504-aebc-aa853d707830\") " pod="openshift-ingress/kube-auth-proxy-594b68584c-km4bh" Nov 03 00:45:30.963992 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:45:30.963975 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5acaee44-b6e2-4504-aebc-aa853d707830-tls-certs\") pod \"kube-auth-proxy-594b68584c-km4bh\" (UID: \"5acaee44-b6e2-4504-aebc-aa853d707830\") " pod="openshift-ingress/kube-auth-proxy-594b68584c-km4bh" Nov 03 00:45:30.981415 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:45:30.981387 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6dvt2\" (UniqueName: \"kubernetes.io/projected/5acaee44-b6e2-4504-aebc-aa853d707830-kube-api-access-6dvt2\") pod \"kube-auth-proxy-594b68584c-km4bh\" (UID: \"5acaee44-b6e2-4504-aebc-aa853d707830\") " pod="openshift-ingress/kube-auth-proxy-594b68584c-km4bh" Nov 03 00:45:30.982797 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:45:30.982775 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 00:45:31.070443 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:45:31.070412 4939 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-594b68584c-km4bh" Nov 03 00:45:31.112166 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:45:31.112070 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 00:45:31.358508 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:45:31.358463 4939 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-594b68584c-km4bh"] Nov 03 00:45:31.361295 ip-10-0-140-107 kubenswrapper[4939]: W1103 00:45:31.361252 4939 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5acaee44_b6e2_4504_aebc_aa853d707830.slice/crio-6512ffeb833b190fd1177c3579584db59c6ad4a68019d6948b081a06cf7c666c WatchSource:0}: Error finding container 6512ffeb833b190fd1177c3579584db59c6ad4a68019d6948b081a06cf7c666c: Status 404 returned error can't find the container with id 6512ffeb833b190fd1177c3579584db59c6ad4a68019d6948b081a06cf7c666c Nov 03 00:45:31.656761 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:45:31.656672 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 00:45:31.791234 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:45:31.791196 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 00:45:32.030818 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:45:32.030739 4939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-594b68584c-km4bh" event={"ID":"5acaee44-b6e2-4504-aebc-aa853d707830","Type":"ContainerStarted","Data":"6512ffeb833b190fd1177c3579584db59c6ad4a68019d6948b081a06cf7c666c"} Nov 03 00:45:32.422252 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:45:32.422216 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 00:45:32.545354 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:45:32.545319 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 00:45:32.953075 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:45:32.953036 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 00:45:34.041104 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:45:34.041066 4939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-594b68584c-km4bh" event={"ID":"5acaee44-b6e2-4504-aebc-aa853d707830","Type":"ContainerStarted","Data":"5e05cd3db01de43866eb977b560d8aac588c8d6e12aa302d0bcbe8c537795188"} Nov 03 00:45:34.067294 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:45:34.067221 4939 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/kube-auth-proxy-594b68584c-km4bh" podStartSLOduration=1.5216453699999999 podStartE2EDuration="4.067207622s" totalImagesPullingTime="2.545562252s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:45:30 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:45:34.066096664 +0000 UTC m=+282.330431617" watchObservedRunningTime="2026-11-03 00:45:34.067207622 +0000 UTC m=+282.331542567" Nov 03 00:45:38.077106 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:45:38.077067 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 00:45:38.202938 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:45:38.202868 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 00:45:41.054391 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:45:41.054358 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 00:45:41.176482 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:45:41.176448 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 00:45:41.524965 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:45:41.524928 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 00:45:43.507322 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:45:43.507284 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 00:45:43.628005 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:45:43.627943 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 00:45:46.024981 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:45:46.024950 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 00:45:46.148399 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:45:46.148358 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 00:45:48.029065 ip-10-0-140-107 kubenswrapper[4939]: E1103 00:45:48.029024 4939 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-sthjh" containerName="extractor" Nov 03 00:45:50.251979 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:45:50.251943 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 00:45:50.362032 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:45:50.361999 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 00:45:50.647836 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:45:50.647803 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 00:45:50.769355 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:45:50.769322 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 00:45:50.889904 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:45:50.889872 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 00:45:51.056723 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:45:51.056641 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 00:45:51.163033 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:45:51.163001 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 00:45:51.885230 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:45:51.885203 4939 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-n42s6_65d134ac-8a25-4c67-b6c7-cd752a098daf/console-operator/2.log" Nov 03 00:45:51.885877 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:45:51.885855 4939 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-n42s6_65d134ac-8a25-4c67-b6c7-cd752a098daf/console-operator/2.log" Nov 03 00:45:59.476345 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:45:59.476310 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 00:45:59.546282 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:45:59.546242 4939 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-4pw7v"] Nov 03 00:45:59.549981 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:45:59.549959 4939 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-7c9b87d574-4pw7v" Nov 03 00:45:59.555589 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:45:59.555567 4939 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"default-dockercfg-w9pxr\"" Nov 03 00:45:59.557726 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:45:59.557693 4939 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"limitador-limits-config-limitador\"" Nov 03 00:45:59.557883 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:45:59.557797 4939 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\"" Nov 03 00:45:59.557978 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:45:59.557884 4939 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\"" Nov 03 00:45:59.559094 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:45:59.559074 4939 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-4pw7v"] Nov 03 00:45:59.576070 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:45:59.576045 4939 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-4pw7v"] Nov 03 00:45:59.580440 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:45:59.580422 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/e1b47e77-a4ab-4d61-ace8-deeea43ea944-config-file\") pod \"limitador-limitador-7c9b87d574-4pw7v\" (UID: \"e1b47e77-a4ab-4d61-ace8-deeea43ea944\") " pod="kuadrant-system/limitador-limitador-7c9b87d574-4pw7v" Nov 03 00:45:59.580502 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:45:59.580473 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9jj68\" (UniqueName: \"kubernetes.io/projected/e1b47e77-a4ab-4d61-ace8-deeea43ea944-kube-api-access-9jj68\") pod \"limitador-limitador-7c9b87d574-4pw7v\" (UID: \"e1b47e77-a4ab-4d61-ace8-deeea43ea944\") " pod="kuadrant-system/limitador-limitador-7c9b87d574-4pw7v" Nov 03 00:45:59.599280 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:45:59.599254 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 00:45:59.680871 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:45:59.680849 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9jj68\" (UniqueName: \"kubernetes.io/projected/e1b47e77-a4ab-4d61-ace8-deeea43ea944-kube-api-access-9jj68\") pod \"limitador-limitador-7c9b87d574-4pw7v\" (UID: \"e1b47e77-a4ab-4d61-ace8-deeea43ea944\") " pod="kuadrant-system/limitador-limitador-7c9b87d574-4pw7v" Nov 03 00:45:59.680997 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:45:59.680895 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/e1b47e77-a4ab-4d61-ace8-deeea43ea944-config-file\") pod \"limitador-limitador-7c9b87d574-4pw7v\" (UID: \"e1b47e77-a4ab-4d61-ace8-deeea43ea944\") " pod="kuadrant-system/limitador-limitador-7c9b87d574-4pw7v" Nov 03 00:45:59.681525 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:45:59.681501 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/e1b47e77-a4ab-4d61-ace8-deeea43ea944-config-file\") pod \"limitador-limitador-7c9b87d574-4pw7v\" (UID: \"e1b47e77-a4ab-4d61-ace8-deeea43ea944\") " pod="kuadrant-system/limitador-limitador-7c9b87d574-4pw7v" Nov 03 00:45:59.688749 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:45:59.688728 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9jj68\" (UniqueName: \"kubernetes.io/projected/e1b47e77-a4ab-4d61-ace8-deeea43ea944-kube-api-access-9jj68\") pod \"limitador-limitador-7c9b87d574-4pw7v\" (UID: \"e1b47e77-a4ab-4d61-ace8-deeea43ea944\") " pod="kuadrant-system/limitador-limitador-7c9b87d574-4pw7v" Nov 03 00:45:59.862610 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:45:59.862405 4939 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-7c9b87d574-4pw7v" Nov 03 00:46:00.136679 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:46:00.136643 4939 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-4pw7v"] Nov 03 00:46:00.139330 ip-10-0-140-107 kubenswrapper[4939]: W1103 00:46:00.139305 4939 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode1b47e77_a4ab_4d61_ace8_deeea43ea944.slice/crio-bdf40307e3a8db1146eec7e39f56ad730d089f344ba677332c23858cc04fe8b0 WatchSource:0}: Error finding container bdf40307e3a8db1146eec7e39f56ad730d089f344ba677332c23858cc04fe8b0: Status 404 returned error can't find the container with id bdf40307e3a8db1146eec7e39f56ad730d089f344ba677332c23858cc04fe8b0 Nov 03 00:46:00.141260 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:46:00.141241 4939 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Nov 03 00:46:01.143465 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:46:01.143429 4939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-7c9b87d574-4pw7v" event={"ID":"e1b47e77-a4ab-4d61-ace8-deeea43ea944","Type":"ContainerStarted","Data":"bdf40307e3a8db1146eec7e39f56ad730d089f344ba677332c23858cc04fe8b0"} Nov 03 00:46:02.473269 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:46:02.473236 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 00:46:02.593270 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:46:02.593232 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 00:46:03.153514 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:46:03.153476 4939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-7c9b87d574-4pw7v" event={"ID":"e1b47e77-a4ab-4d61-ace8-deeea43ea944","Type":"ContainerStarted","Data":"e2d4d31340bdb34b8b0e7ed077d3b5eb9b911658cbe642c4f2a937c90eeb5e40"} Nov 03 00:46:03.153684 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:46:03.153649 4939 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-limitador-7c9b87d574-4pw7v" Nov 03 00:46:03.169340 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:46:03.169283 4939 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/limitador-limitador-7c9b87d574-4pw7v" podStartSLOduration=1.338462682 podStartE2EDuration="4.16926946s" totalImagesPullingTime="2.830806778s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:45:59 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:46:03.168556721 +0000 UTC m=+311.432891668" watchObservedRunningTime="2026-11-03 00:46:03.16926946 +0000 UTC m=+311.433604402" Nov 03 00:46:04.907127 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:46:04.907076 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 00:46:05.013250 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:46:05.013223 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 00:46:05.367519 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:46:05.367490 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 00:46:08.762337 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:46:08.762304 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 00:46:14.159082 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:46:14.159054 4939 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-limitador-7c9b87d574-4pw7v" Nov 03 00:46:18.028728 ip-10-0-140-107 kubenswrapper[4939]: E1103 00:46:18.028690 4939 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-sthjh" containerName="extractor" Nov 03 00:46:18.927847 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:46:18.927808 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 00:46:19.121080 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:46:19.121045 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 00:46:19.682255 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:46:19.682213 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 00:46:25.861205 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:46:25.861170 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 00:46:25.931428 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:46:25.931395 4939 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-jkd9f"] Nov 03 00:46:25.938337 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:46:25.938305 4939 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-jkd9f" Nov 03 00:46:25.956250 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:46:25.956223 4939 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-jkd9f"] Nov 03 00:46:25.965001 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:46:25.964968 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/d9dfeb7b-ac37-456e-8239-43505d7f0ccb-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-jkd9f\" (UID: \"d9dfeb7b-ac37-456e-8239-43505d7f0ccb\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-jkd9f" Nov 03 00:46:25.965111 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:46:25.965053 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d9dfeb7b-ac37-456e-8239-43505d7f0ccb-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-jkd9f\" (UID: \"d9dfeb7b-ac37-456e-8239-43505d7f0ccb\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-jkd9f" Nov 03 00:46:25.965111 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:46:25.965100 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/d9dfeb7b-ac37-456e-8239-43505d7f0ccb-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-jkd9f\" (UID: \"d9dfeb7b-ac37-456e-8239-43505d7f0ccb\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-jkd9f" Nov 03 00:46:25.965211 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:46:25.965167 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/d9dfeb7b-ac37-456e-8239-43505d7f0ccb-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-jkd9f\" (UID: \"d9dfeb7b-ac37-456e-8239-43505d7f0ccb\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-jkd9f" Nov 03 00:46:25.965277 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:46:25.965258 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/d9dfeb7b-ac37-456e-8239-43505d7f0ccb-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-jkd9f\" (UID: \"d9dfeb7b-ac37-456e-8239-43505d7f0ccb\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-jkd9f" Nov 03 00:46:25.965343 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:46:25.965322 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tg2kx\" (UniqueName: \"kubernetes.io/projected/d9dfeb7b-ac37-456e-8239-43505d7f0ccb-kube-api-access-tg2kx\") pod \"istiod-openshift-gateway-7ddd9cfdc8-jkd9f\" (UID: \"d9dfeb7b-ac37-456e-8239-43505d7f0ccb\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-jkd9f" Nov 03 00:46:25.965454 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:46:25.965357 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/d9dfeb7b-ac37-456e-8239-43505d7f0ccb-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-jkd9f\" (UID: \"d9dfeb7b-ac37-456e-8239-43505d7f0ccb\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-jkd9f" Nov 03 00:46:26.015043 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:46:26.015007 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 00:46:26.066418 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:46:26.066383 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/d9dfeb7b-ac37-456e-8239-43505d7f0ccb-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-jkd9f\" (UID: \"d9dfeb7b-ac37-456e-8239-43505d7f0ccb\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-jkd9f" Nov 03 00:46:26.066418 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:46:26.066422 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d9dfeb7b-ac37-456e-8239-43505d7f0ccb-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-jkd9f\" (UID: \"d9dfeb7b-ac37-456e-8239-43505d7f0ccb\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-jkd9f" Nov 03 00:46:26.066618 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:46:26.066452 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/d9dfeb7b-ac37-456e-8239-43505d7f0ccb-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-jkd9f\" (UID: \"d9dfeb7b-ac37-456e-8239-43505d7f0ccb\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-jkd9f" Nov 03 00:46:26.066618 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:46:26.066477 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/d9dfeb7b-ac37-456e-8239-43505d7f0ccb-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-jkd9f\" (UID: \"d9dfeb7b-ac37-456e-8239-43505d7f0ccb\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-jkd9f" Nov 03 00:46:26.066618 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:46:26.066503 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/d9dfeb7b-ac37-456e-8239-43505d7f0ccb-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-jkd9f\" (UID: \"d9dfeb7b-ac37-456e-8239-43505d7f0ccb\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-jkd9f" Nov 03 00:46:26.066618 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:46:26.066520 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tg2kx\" (UniqueName: \"kubernetes.io/projected/d9dfeb7b-ac37-456e-8239-43505d7f0ccb-kube-api-access-tg2kx\") pod \"istiod-openshift-gateway-7ddd9cfdc8-jkd9f\" (UID: \"d9dfeb7b-ac37-456e-8239-43505d7f0ccb\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-jkd9f" Nov 03 00:46:26.066618 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:46:26.066547 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/d9dfeb7b-ac37-456e-8239-43505d7f0ccb-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-jkd9f\" (UID: \"d9dfeb7b-ac37-456e-8239-43505d7f0ccb\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-jkd9f" Nov 03 00:46:26.067427 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:46:26.067399 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/d9dfeb7b-ac37-456e-8239-43505d7f0ccb-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-jkd9f\" (UID: \"d9dfeb7b-ac37-456e-8239-43505d7f0ccb\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-jkd9f" Nov 03 00:46:26.068948 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:46:26.068925 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/d9dfeb7b-ac37-456e-8239-43505d7f0ccb-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-jkd9f\" (UID: \"d9dfeb7b-ac37-456e-8239-43505d7f0ccb\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-jkd9f" Nov 03 00:46:26.069234 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:46:26.069204 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/d9dfeb7b-ac37-456e-8239-43505d7f0ccb-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-jkd9f\" (UID: \"d9dfeb7b-ac37-456e-8239-43505d7f0ccb\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-jkd9f" Nov 03 00:46:26.070149 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:46:26.070124 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/d9dfeb7b-ac37-456e-8239-43505d7f0ccb-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-jkd9f\" (UID: \"d9dfeb7b-ac37-456e-8239-43505d7f0ccb\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-jkd9f" Nov 03 00:46:26.070230 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:46:26.070132 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/d9dfeb7b-ac37-456e-8239-43505d7f0ccb-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-jkd9f\" (UID: \"d9dfeb7b-ac37-456e-8239-43505d7f0ccb\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-jkd9f" Nov 03 00:46:26.085683 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:46:26.085661 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tg2kx\" (UniqueName: \"kubernetes.io/projected/d9dfeb7b-ac37-456e-8239-43505d7f0ccb-kube-api-access-tg2kx\") pod \"istiod-openshift-gateway-7ddd9cfdc8-jkd9f\" (UID: \"d9dfeb7b-ac37-456e-8239-43505d7f0ccb\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-jkd9f" Nov 03 00:46:26.088174 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:46:26.088148 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d9dfeb7b-ac37-456e-8239-43505d7f0ccb-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-jkd9f\" (UID: \"d9dfeb7b-ac37-456e-8239-43505d7f0ccb\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-jkd9f" Nov 03 00:46:26.250310 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:46:26.250232 4939 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-jkd9f" Nov 03 00:46:26.313513 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:46:26.313481 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 00:46:26.522955 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:46:26.522871 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 00:46:26.528949 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:46:26.528817 4939 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-jkd9f"] Nov 03 00:46:26.531557 ip-10-0-140-107 kubenswrapper[4939]: W1103 00:46:26.531532 4939 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd9dfeb7b_ac37_456e_8239_43505d7f0ccb.slice/crio-93a51d6f5c825ec6b726b666357336e9f643d50b36585541564fbd29f6a981f7 WatchSource:0}: Error finding container 93a51d6f5c825ec6b726b666357336e9f643d50b36585541564fbd29f6a981f7: Status 404 returned error can't find the container with id 93a51d6f5c825ec6b726b666357336e9f643d50b36585541564fbd29f6a981f7 Nov 03 00:46:26.533560 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:46:26.533531 4939 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 00:46:26.533624 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:46:26.533589 4939 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 00:46:27.250340 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:46:27.250305 4939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-jkd9f" event={"ID":"d9dfeb7b-ac37-456e-8239-43505d7f0ccb","Type":"ContainerStarted","Data":"ee938f663c868f116b92cef7e4ceefa16fbe8d43f87e0e96382b6b8ab1172c9f"} Nov 03 00:46:27.250809 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:46:27.250355 4939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-jkd9f" event={"ID":"d9dfeb7b-ac37-456e-8239-43505d7f0ccb","Type":"ContainerStarted","Data":"93a51d6f5c825ec6b726b666357336e9f643d50b36585541564fbd29f6a981f7"} Nov 03 00:46:27.250809 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:46:27.250627 4939 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-jkd9f" Nov 03 00:46:27.252274 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:46:27.252249 4939 patch_prober.go:29] interesting pod/istiod-openshift-gateway-7ddd9cfdc8-jkd9f container/discovery namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 03 00:46:27.252404 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:46:27.252291 4939 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-jkd9f" podUID="d9dfeb7b-ac37-456e-8239-43505d7f0ccb" containerName="discovery" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 03 00:46:27.275593 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:46:27.275551 4939 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-jkd9f" podStartSLOduration=2.275540327 podStartE2EDuration="2.275540327s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:46:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:46:27.273629561 +0000 UTC m=+335.537964509" watchObservedRunningTime="2026-11-03 00:46:27.275540327 +0000 UTC m=+335.539875271" Nov 03 00:46:27.454537 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:46:27.454501 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 00:46:27.567682 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:46:27.567590 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 00:46:27.684631 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:46:27.684599 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 00:46:28.255154 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:46:28.255126 4939 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-jkd9f" Nov 03 00:46:28.306790 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:46:28.306756 4939 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7876bfcb74-jbfv6"] Nov 03 00:46:28.307161 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:46:28.307131 4939 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-jbfv6" podUID="16fe2330-9a44-44c9-b41b-ff22f7648ad8" containerName="discovery" containerID="cri-o://1b1dda8832529b7eac7da9762fc7ebfc6b552dbdb34478b49817b21e8f860e67" gracePeriod=30 Nov 03 00:46:28.548972 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:46:28.548940 4939 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-jbfv6" Nov 03 00:46:28.587248 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:46:28.587219 4939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/16fe2330-9a44-44c9-b41b-ff22f7648ad8-istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/16fe2330-9a44-44c9-b41b-ff22f7648ad8-istio-kubeconfig\") pod \"16fe2330-9a44-44c9-b41b-ff22f7648ad8\" (UID: \"16fe2330-9a44-44c9-b41b-ff22f7648ad8\") " Nov 03 00:46:28.587397 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:46:28.587283 4939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/16fe2330-9a44-44c9-b41b-ff22f7648ad8-local-certs\" (UniqueName: \"kubernetes.io/empty-dir/16fe2330-9a44-44c9-b41b-ff22f7648ad8-local-certs\") pod \"16fe2330-9a44-44c9-b41b-ff22f7648ad8\" (UID: \"16fe2330-9a44-44c9-b41b-ff22f7648ad8\") " Nov 03 00:46:28.587397 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:46:28.587339 4939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/16fe2330-9a44-44c9-b41b-ff22f7648ad8-istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/16fe2330-9a44-44c9-b41b-ff22f7648ad8-istio-csr-dns-cert\") pod \"16fe2330-9a44-44c9-b41b-ff22f7648ad8\" (UID: \"16fe2330-9a44-44c9-b41b-ff22f7648ad8\") " Nov 03 00:46:28.587397 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:46:28.587369 4939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/16fe2330-9a44-44c9-b41b-ff22f7648ad8-kube-api-access-bscbp\" (UniqueName: \"kubernetes.io/projected/16fe2330-9a44-44c9-b41b-ff22f7648ad8-kube-api-access-bscbp\") pod \"16fe2330-9a44-44c9-b41b-ff22f7648ad8\" (UID: \"16fe2330-9a44-44c9-b41b-ff22f7648ad8\") " Nov 03 00:46:28.587556 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:46:28.587412 4939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/16fe2330-9a44-44c9-b41b-ff22f7648ad8-cacerts\" (UniqueName: \"kubernetes.io/secret/16fe2330-9a44-44c9-b41b-ff22f7648ad8-cacerts\") pod \"16fe2330-9a44-44c9-b41b-ff22f7648ad8\" (UID: \"16fe2330-9a44-44c9-b41b-ff22f7648ad8\") " Nov 03 00:46:28.587556 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:46:28.587443 4939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/16fe2330-9a44-44c9-b41b-ff22f7648ad8-istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/16fe2330-9a44-44c9-b41b-ff22f7648ad8-istio-csr-ca-configmap\") pod \"16fe2330-9a44-44c9-b41b-ff22f7648ad8\" (UID: \"16fe2330-9a44-44c9-b41b-ff22f7648ad8\") " Nov 03 00:46:28.587556 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:46:28.587500 4939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/16fe2330-9a44-44c9-b41b-ff22f7648ad8-istio-token\" (UniqueName: \"kubernetes.io/projected/16fe2330-9a44-44c9-b41b-ff22f7648ad8-istio-token\") pod \"16fe2330-9a44-44c9-b41b-ff22f7648ad8\" (UID: \"16fe2330-9a44-44c9-b41b-ff22f7648ad8\") " Nov 03 00:46:28.588215 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:46:28.588164 4939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/16fe2330-9a44-44c9-b41b-ff22f7648ad8-istio-csr-ca-configmap" pod "16fe2330-9a44-44c9-b41b-ff22f7648ad8" (UID: "16fe2330-9a44-44c9-b41b-ff22f7648ad8"). InnerVolumeSpecName "istio-csr-ca-configmap". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 00:46:28.589808 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:46:28.589777 4939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/16fe2330-9a44-44c9-b41b-ff22f7648ad8-kube-api-access-bscbp" pod "16fe2330-9a44-44c9-b41b-ff22f7648ad8" (UID: "16fe2330-9a44-44c9-b41b-ff22f7648ad8"). InnerVolumeSpecName "kube-api-access-bscbp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 00:46:28.589999 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:46:28.589970 4939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/16fe2330-9a44-44c9-b41b-ff22f7648ad8-local-certs" pod "16fe2330-9a44-44c9-b41b-ff22f7648ad8" (UID: "16fe2330-9a44-44c9-b41b-ff22f7648ad8"). InnerVolumeSpecName "local-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 00:46:28.590087 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:46:28.590054 4939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/16fe2330-9a44-44c9-b41b-ff22f7648ad8-istio-token" pod "16fe2330-9a44-44c9-b41b-ff22f7648ad8" (UID: "16fe2330-9a44-44c9-b41b-ff22f7648ad8"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 00:46:28.590159 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:46:28.590061 4939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/16fe2330-9a44-44c9-b41b-ff22f7648ad8-cacerts" pod "16fe2330-9a44-44c9-b41b-ff22f7648ad8" (UID: "16fe2330-9a44-44c9-b41b-ff22f7648ad8"). InnerVolumeSpecName "cacerts". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 00:46:28.590159 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:46:28.590122 4939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/16fe2330-9a44-44c9-b41b-ff22f7648ad8-istio-csr-dns-cert" pod "16fe2330-9a44-44c9-b41b-ff22f7648ad8" (UID: "16fe2330-9a44-44c9-b41b-ff22f7648ad8"). InnerVolumeSpecName "istio-csr-dns-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 00:46:28.590259 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:46:28.590209 4939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/16fe2330-9a44-44c9-b41b-ff22f7648ad8-istio-kubeconfig" pod "16fe2330-9a44-44c9-b41b-ff22f7648ad8" (UID: "16fe2330-9a44-44c9-b41b-ff22f7648ad8"). InnerVolumeSpecName "istio-kubeconfig". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 00:46:28.688228 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:46:28.688198 4939 reconciler_common.go:299] "Volume detached for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/16fe2330-9a44-44c9-b41b-ff22f7648ad8-local-certs\") on node \"ip-10-0-140-107.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 00:46:28.688228 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:46:28.688225 4939 reconciler_common.go:299] "Volume detached for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/16fe2330-9a44-44c9-b41b-ff22f7648ad8-istio-csr-dns-cert\") on node \"ip-10-0-140-107.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 00:46:28.688411 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:46:28.688235 4939 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bscbp\" (UniqueName: \"kubernetes.io/projected/16fe2330-9a44-44c9-b41b-ff22f7648ad8-kube-api-access-bscbp\") on node \"ip-10-0-140-107.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 00:46:28.688411 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:46:28.688245 4939 reconciler_common.go:299] "Volume detached for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/16fe2330-9a44-44c9-b41b-ff22f7648ad8-cacerts\") on node \"ip-10-0-140-107.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 00:46:28.688411 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:46:28.688254 4939 reconciler_common.go:299] "Volume detached for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/16fe2330-9a44-44c9-b41b-ff22f7648ad8-istio-csr-ca-configmap\") on node \"ip-10-0-140-107.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 00:46:28.688411 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:46:28.688262 4939 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/16fe2330-9a44-44c9-b41b-ff22f7648ad8-istio-token\") on node \"ip-10-0-140-107.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 00:46:28.688411 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:46:28.688271 4939 reconciler_common.go:299] "Volume detached for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/16fe2330-9a44-44c9-b41b-ff22f7648ad8-istio-kubeconfig\") on node \"ip-10-0-140-107.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 00:46:29.259556 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:46:29.259523 4939 generic.go:356] "Generic (PLEG): container finished" podID="16fe2330-9a44-44c9-b41b-ff22f7648ad8" containerID="1b1dda8832529b7eac7da9762fc7ebfc6b552dbdb34478b49817b21e8f860e67" exitCode=0 Nov 03 00:46:29.260009 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:46:29.259585 4939 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-jbfv6" Nov 03 00:46:29.260009 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:46:29.259603 4939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-jbfv6" event={"ID":"16fe2330-9a44-44c9-b41b-ff22f7648ad8","Type":"ContainerDied","Data":"1b1dda8832529b7eac7da9762fc7ebfc6b552dbdb34478b49817b21e8f860e67"} Nov 03 00:46:29.260009 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:46:29.259655 4939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-jbfv6" event={"ID":"16fe2330-9a44-44c9-b41b-ff22f7648ad8","Type":"ContainerDied","Data":"f77aa67c45a93e00736ca89243cf71e787a2e2046d00761c9e0831d51c177558"} Nov 03 00:46:29.260009 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:46:29.259679 4939 scope.go:122] "RemoveContainer" containerID="1b1dda8832529b7eac7da9762fc7ebfc6b552dbdb34478b49817b21e8f860e67" Nov 03 00:46:29.268946 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:46:29.268928 4939 scope.go:122] "RemoveContainer" containerID="1b1dda8832529b7eac7da9762fc7ebfc6b552dbdb34478b49817b21e8f860e67" Nov 03 00:46:29.269218 ip-10-0-140-107 kubenswrapper[4939]: E1103 00:46:29.269199 4939 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1b1dda8832529b7eac7da9762fc7ebfc6b552dbdb34478b49817b21e8f860e67\": container with ID starting with 1b1dda8832529b7eac7da9762fc7ebfc6b552dbdb34478b49817b21e8f860e67 not found: ID does not exist" containerID="1b1dda8832529b7eac7da9762fc7ebfc6b552dbdb34478b49817b21e8f860e67" Nov 03 00:46:29.269272 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:46:29.269229 4939 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1b1dda8832529b7eac7da9762fc7ebfc6b552dbdb34478b49817b21e8f860e67"} err="failed to get container status \"1b1dda8832529b7eac7da9762fc7ebfc6b552dbdb34478b49817b21e8f860e67\": rpc error: code = NotFound desc = could not find container \"1b1dda8832529b7eac7da9762fc7ebfc6b552dbdb34478b49817b21e8f860e67\": container with ID starting with 1b1dda8832529b7eac7da9762fc7ebfc6b552dbdb34478b49817b21e8f860e67 not found: ID does not exist" Nov 03 00:46:29.287537 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:46:29.287513 4939 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7876bfcb74-jbfv6"] Nov 03 00:46:29.292702 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:46:29.292674 4939 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7876bfcb74-jbfv6"] Nov 03 00:46:29.866193 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:46:29.866160 4939 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="16fe2330-9a44-44c9-b41b-ff22f7648ad8" path="/var/lib/kubelet/pods/16fe2330-9a44-44c9-b41b-ff22f7648ad8/volumes" Nov 03 00:46:41.597285 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:46:41.597249 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 00:46:41.927002 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:46:41.926890 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 00:46:41.976082 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:46:41.976054 4939 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-59fd54f5bc-lqb6l"] Nov 03 00:46:41.976407 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:46:41.976394 4939 cpu_manager.go:414] "containerMap: removing container" podUID="16fe2330-9a44-44c9-b41b-ff22f7648ad8" containerName="discovery" Nov 03 00:46:41.976461 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:46:41.976408 4939 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="16fe2330-9a44-44c9-b41b-ff22f7648ad8" containerName="discovery" Nov 03 00:46:41.976502 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:46:41.976473 4939 memory_manager.go:365] "RemoveStaleState removing state" podUID="16fe2330-9a44-44c9-b41b-ff22f7648ad8" containerName="discovery" Nov 03 00:46:41.983859 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:46:41.983839 4939 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-59fd54f5bc-lqb6l" Nov 03 00:46:41.989176 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:46:41.989147 4939 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-59fd54f5bc-lqb6l"] Nov 03 00:46:41.989867 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:46:41.989824 4939 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-metrics-tls\"" Nov 03 00:46:41.990133 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:46:41.990112 4939 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"maas-parameters\"" Nov 03 00:46:41.990251 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:46:41.990152 4939 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-webhook-cert\"" Nov 03 00:46:41.990251 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:46:41.990180 4939 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-dockercfg-2xnh6\"" Nov 03 00:46:41.990251 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:46:41.990186 4939 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\"" Nov 03 00:46:41.990429 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:46:41.990195 4939 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\"" Nov 03 00:46:42.060819 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:46:42.060785 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 00:46:42.089958 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:46:42.089929 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g6h6q\" (UniqueName: \"kubernetes.io/projected/85bcbe6a-5881-4b40-acd4-1e3b6a560710-kube-api-access-g6h6q\") pod \"maas-controller-59fd54f5bc-lqb6l\" (UID: \"85bcbe6a-5881-4b40-acd4-1e3b6a560710\") " pod="opendatahub/maas-controller-59fd54f5bc-lqb6l" Nov 03 00:46:42.090083 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:46:42.089966 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/85bcbe6a-5881-4b40-acd4-1e3b6a560710-webhook-cert\") pod \"maas-controller-59fd54f5bc-lqb6l\" (UID: \"85bcbe6a-5881-4b40-acd4-1e3b6a560710\") " pod="opendatahub/maas-controller-59fd54f5bc-lqb6l" Nov 03 00:46:42.090083 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:46:42.089983 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/85bcbe6a-5881-4b40-acd4-1e3b6a560710-metrics-certs\") pod \"maas-controller-59fd54f5bc-lqb6l\" (UID: \"85bcbe6a-5881-4b40-acd4-1e3b6a560710\") " pod="opendatahub/maas-controller-59fd54f5bc-lqb6l" Nov 03 00:46:42.191386 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:46:42.191296 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g6h6q\" (UniqueName: \"kubernetes.io/projected/85bcbe6a-5881-4b40-acd4-1e3b6a560710-kube-api-access-g6h6q\") pod \"maas-controller-59fd54f5bc-lqb6l\" (UID: \"85bcbe6a-5881-4b40-acd4-1e3b6a560710\") " pod="opendatahub/maas-controller-59fd54f5bc-lqb6l" Nov 03 00:46:42.191386 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:46:42.191344 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/85bcbe6a-5881-4b40-acd4-1e3b6a560710-webhook-cert\") pod \"maas-controller-59fd54f5bc-lqb6l\" (UID: \"85bcbe6a-5881-4b40-acd4-1e3b6a560710\") " pod="opendatahub/maas-controller-59fd54f5bc-lqb6l" Nov 03 00:46:42.191386 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:46:42.191371 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/85bcbe6a-5881-4b40-acd4-1e3b6a560710-metrics-certs\") pod \"maas-controller-59fd54f5bc-lqb6l\" (UID: \"85bcbe6a-5881-4b40-acd4-1e3b6a560710\") " pod="opendatahub/maas-controller-59fd54f5bc-lqb6l" Nov 03 00:46:42.193871 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:46:42.193848 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/85bcbe6a-5881-4b40-acd4-1e3b6a560710-webhook-cert\") pod \"maas-controller-59fd54f5bc-lqb6l\" (UID: \"85bcbe6a-5881-4b40-acd4-1e3b6a560710\") " pod="opendatahub/maas-controller-59fd54f5bc-lqb6l" Nov 03 00:46:42.193999 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:46:42.193894 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/85bcbe6a-5881-4b40-acd4-1e3b6a560710-metrics-certs\") pod \"maas-controller-59fd54f5bc-lqb6l\" (UID: \"85bcbe6a-5881-4b40-acd4-1e3b6a560710\") " pod="opendatahub/maas-controller-59fd54f5bc-lqb6l" Nov 03 00:46:42.200043 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:46:42.200018 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-g6h6q\" (UniqueName: \"kubernetes.io/projected/85bcbe6a-5881-4b40-acd4-1e3b6a560710-kube-api-access-g6h6q\") pod \"maas-controller-59fd54f5bc-lqb6l\" (UID: \"85bcbe6a-5881-4b40-acd4-1e3b6a560710\") " pod="opendatahub/maas-controller-59fd54f5bc-lqb6l" Nov 03 00:46:42.200186 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:46:42.200166 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 00:46:42.298384 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:46:42.298354 4939 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-59fd54f5bc-lqb6l" Nov 03 00:46:42.303414 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:46:42.303393 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 00:46:42.488319 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:46:42.488241 4939 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-7b85f9594f-9gw68"] Nov 03 00:46:42.491903 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:46:42.491886 4939 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-7b85f9594f-9gw68" Nov 03 00:46:42.501984 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:46:42.501965 4939 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-7b85f9594f-9gw68"] Nov 03 00:46:42.567675 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:46:42.567647 4939 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-59fd54f5bc-lqb6l"] Nov 03 00:46:42.570611 ip-10-0-140-107 kubenswrapper[4939]: W1103 00:46:42.570585 4939 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod85bcbe6a_5881_4b40_acd4_1e3b6a560710.slice/crio-0762ff318f2cbb0233563b818a46d7dda56c321064dbab11da768db4f39aaeee WatchSource:0}: Error finding container 0762ff318f2cbb0233563b818a46d7dda56c321064dbab11da768db4f39aaeee: Status 404 returned error can't find the container with id 0762ff318f2cbb0233563b818a46d7dda56c321064dbab11da768db4f39aaeee Nov 03 00:46:42.593577 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:46:42.593549 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/071dd59f-2a89-48c6-aea0-2a552f5bae08-metrics-certs\") pod \"maas-controller-7b85f9594f-9gw68\" (UID: \"071dd59f-2a89-48c6-aea0-2a552f5bae08\") " pod="opendatahub/maas-controller-7b85f9594f-9gw68" Nov 03 00:46:42.593691 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:46:42.593669 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/071dd59f-2a89-48c6-aea0-2a552f5bae08-webhook-cert\") pod \"maas-controller-7b85f9594f-9gw68\" (UID: \"071dd59f-2a89-48c6-aea0-2a552f5bae08\") " pod="opendatahub/maas-controller-7b85f9594f-9gw68" Nov 03 00:46:42.593752 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:46:42.593718 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cgzlc\" (UniqueName: \"kubernetes.io/projected/071dd59f-2a89-48c6-aea0-2a552f5bae08-kube-api-access-cgzlc\") pod \"maas-controller-7b85f9594f-9gw68\" (UID: \"071dd59f-2a89-48c6-aea0-2a552f5bae08\") " pod="opendatahub/maas-controller-7b85f9594f-9gw68" Nov 03 00:46:42.694711 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:46:42.694687 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/071dd59f-2a89-48c6-aea0-2a552f5bae08-webhook-cert\") pod \"maas-controller-7b85f9594f-9gw68\" (UID: \"071dd59f-2a89-48c6-aea0-2a552f5bae08\") " pod="opendatahub/maas-controller-7b85f9594f-9gw68" Nov 03 00:46:42.695132 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:46:42.694729 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cgzlc\" (UniqueName: \"kubernetes.io/projected/071dd59f-2a89-48c6-aea0-2a552f5bae08-kube-api-access-cgzlc\") pod \"maas-controller-7b85f9594f-9gw68\" (UID: \"071dd59f-2a89-48c6-aea0-2a552f5bae08\") " pod="opendatahub/maas-controller-7b85f9594f-9gw68" Nov 03 00:46:42.695132 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:46:42.694776 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/071dd59f-2a89-48c6-aea0-2a552f5bae08-metrics-certs\") pod \"maas-controller-7b85f9594f-9gw68\" (UID: \"071dd59f-2a89-48c6-aea0-2a552f5bae08\") " pod="opendatahub/maas-controller-7b85f9594f-9gw68" Nov 03 00:46:42.697207 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:46:42.697181 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/071dd59f-2a89-48c6-aea0-2a552f5bae08-webhook-cert\") pod \"maas-controller-7b85f9594f-9gw68\" (UID: \"071dd59f-2a89-48c6-aea0-2a552f5bae08\") " pod="opendatahub/maas-controller-7b85f9594f-9gw68" Nov 03 00:46:42.697293 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:46:42.697181 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/071dd59f-2a89-48c6-aea0-2a552f5bae08-metrics-certs\") pod \"maas-controller-7b85f9594f-9gw68\" (UID: \"071dd59f-2a89-48c6-aea0-2a552f5bae08\") " pod="opendatahub/maas-controller-7b85f9594f-9gw68" Nov 03 00:46:42.702701 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:46:42.702682 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cgzlc\" (UniqueName: \"kubernetes.io/projected/071dd59f-2a89-48c6-aea0-2a552f5bae08-kube-api-access-cgzlc\") pod \"maas-controller-7b85f9594f-9gw68\" (UID: \"071dd59f-2a89-48c6-aea0-2a552f5bae08\") " pod="opendatahub/maas-controller-7b85f9594f-9gw68" Nov 03 00:46:42.804958 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:46:42.804881 4939 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-7b85f9594f-9gw68" Nov 03 00:46:43.074027 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:46:43.073827 4939 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-7b85f9594f-9gw68"] Nov 03 00:46:43.076139 ip-10-0-140-107 kubenswrapper[4939]: W1103 00:46:43.076112 4939 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod071dd59f_2a89_48c6_aea0_2a552f5bae08.slice/crio-a50cc92868a148c2492344807eba85ef2e3a675a2c8f721b6b220f945ab5e343 WatchSource:0}: Error finding container a50cc92868a148c2492344807eba85ef2e3a675a2c8f721b6b220f945ab5e343: Status 404 returned error can't find the container with id a50cc92868a148c2492344807eba85ef2e3a675a2c8f721b6b220f945ab5e343 Nov 03 00:46:43.320143 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:46:43.320108 4939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-59fd54f5bc-lqb6l" event={"ID":"85bcbe6a-5881-4b40-acd4-1e3b6a560710","Type":"ContainerStarted","Data":"0762ff318f2cbb0233563b818a46d7dda56c321064dbab11da768db4f39aaeee"} Nov 03 00:46:43.321371 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:46:43.321346 4939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-7b85f9594f-9gw68" event={"ID":"071dd59f-2a89-48c6-aea0-2a552f5bae08","Type":"ContainerStarted","Data":"a50cc92868a148c2492344807eba85ef2e3a675a2c8f721b6b220f945ab5e343"} Nov 03 00:46:45.961527 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:46:45.961499 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 00:46:46.106729 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:46:46.106696 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 00:46:46.338373 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:46:46.338340 4939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-59fd54f5bc-lqb6l" event={"ID":"85bcbe6a-5881-4b40-acd4-1e3b6a560710","Type":"ContainerStarted","Data":"727b5e8dc501b261bfa7598dc1ee334e8979b5e0a0a08279c49be4a4ca783ea8"} Nov 03 00:46:46.338572 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:46:46.338448 4939 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-59fd54f5bc-lqb6l" Nov 03 00:46:46.340049 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:46:46.340016 4939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-7b85f9594f-9gw68" event={"ID":"071dd59f-2a89-48c6-aea0-2a552f5bae08","Type":"ContainerStarted","Data":"aa982ab5e129d0ec9dbefb32ca09899961f0b1c2328c2dd029a141f3912d1c69"} Nov 03 00:46:46.340151 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:46:46.340136 4939 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-7b85f9594f-9gw68" Nov 03 00:46:46.358154 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:46:46.358079 4939 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/maas-controller-59fd54f5bc-lqb6l" podStartSLOduration=2.173018855 podStartE2EDuration="5.358067711s" totalImagesPullingTime="3.185048856s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:46:41 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:46:46.357107967 +0000 UTC m=+354.621442913" watchObservedRunningTime="2026-11-03 00:46:46.358067711 +0000 UTC m=+354.622402648" Nov 03 00:46:46.379034 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:46:46.378994 4939 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/maas-controller-7b85f9594f-9gw68" podStartSLOduration=1.5939462899999999 podStartE2EDuration="4.378982533s" totalImagesPullingTime="2.785036243s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:46:42 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:46:46.378055655 +0000 UTC m=+354.642390600" watchObservedRunningTime="2026-11-03 00:46:46.378982533 +0000 UTC m=+354.643317475" Nov 03 00:46:47.402064 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:46:47.402032 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 00:46:47.524839 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:46:47.524793 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 00:46:48.028692 ip-10-0-140-107 kubenswrapper[4939]: E1103 00:46:48.028656 4939 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-sthjh" containerName="extractor" Nov 03 00:46:57.346694 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:46:57.346663 4939 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-7b85f9594f-9gw68" Nov 03 00:46:57.347128 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:46:57.346745 4939 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-59fd54f5bc-lqb6l" Nov 03 00:46:57.411827 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:46:57.411788 4939 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-59fd54f5bc-lqb6l"] Nov 03 00:46:57.412155 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:46:57.412124 4939 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/maas-controller-59fd54f5bc-lqb6l" podUID="85bcbe6a-5881-4b40-acd4-1e3b6a560710" containerName="manager" containerID="cri-o://727b5e8dc501b261bfa7598dc1ee334e8979b5e0a0a08279c49be4a4ca783ea8" gracePeriod=30 Nov 03 00:46:57.652351 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:46:57.652319 4939 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-59fd54f5bc-lqb6l" Nov 03 00:46:57.724633 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:46:57.724599 4939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/85bcbe6a-5881-4b40-acd4-1e3b6a560710-kube-api-access-g6h6q\" (UniqueName: \"kubernetes.io/projected/85bcbe6a-5881-4b40-acd4-1e3b6a560710-kube-api-access-g6h6q\") pod \"85bcbe6a-5881-4b40-acd4-1e3b6a560710\" (UID: \"85bcbe6a-5881-4b40-acd4-1e3b6a560710\") " Nov 03 00:46:57.724633 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:46:57.724638 4939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/85bcbe6a-5881-4b40-acd4-1e3b6a560710-webhook-cert\" (UniqueName: \"kubernetes.io/secret/85bcbe6a-5881-4b40-acd4-1e3b6a560710-webhook-cert\") pod \"85bcbe6a-5881-4b40-acd4-1e3b6a560710\" (UID: \"85bcbe6a-5881-4b40-acd4-1e3b6a560710\") " Nov 03 00:46:57.724846 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:46:57.724673 4939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/85bcbe6a-5881-4b40-acd4-1e3b6a560710-metrics-certs\" (UniqueName: \"kubernetes.io/secret/85bcbe6a-5881-4b40-acd4-1e3b6a560710-metrics-certs\") pod \"85bcbe6a-5881-4b40-acd4-1e3b6a560710\" (UID: \"85bcbe6a-5881-4b40-acd4-1e3b6a560710\") " Nov 03 00:46:57.726904 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:46:57.726866 4939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/85bcbe6a-5881-4b40-acd4-1e3b6a560710-webhook-cert" pod "85bcbe6a-5881-4b40-acd4-1e3b6a560710" (UID: "85bcbe6a-5881-4b40-acd4-1e3b6a560710"). InnerVolumeSpecName "webhook-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 00:46:57.726904 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:46:57.726884 4939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/85bcbe6a-5881-4b40-acd4-1e3b6a560710-metrics-certs" pod "85bcbe6a-5881-4b40-acd4-1e3b6a560710" (UID: "85bcbe6a-5881-4b40-acd4-1e3b6a560710"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 00:46:57.727048 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:46:57.726943 4939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/85bcbe6a-5881-4b40-acd4-1e3b6a560710-kube-api-access-g6h6q" pod "85bcbe6a-5881-4b40-acd4-1e3b6a560710" (UID: "85bcbe6a-5881-4b40-acd4-1e3b6a560710"). InnerVolumeSpecName "kube-api-access-g6h6q". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 00:46:57.825926 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:46:57.825856 4939 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-g6h6q\" (UniqueName: \"kubernetes.io/projected/85bcbe6a-5881-4b40-acd4-1e3b6a560710-kube-api-access-g6h6q\") on node \"ip-10-0-140-107.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 00:46:57.825926 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:46:57.825898 4939 reconciler_common.go:299] "Volume detached for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/85bcbe6a-5881-4b40-acd4-1e3b6a560710-webhook-cert\") on node \"ip-10-0-140-107.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 00:46:57.825926 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:46:57.825926 4939 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/85bcbe6a-5881-4b40-acd4-1e3b6a560710-metrics-certs\") on node \"ip-10-0-140-107.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 00:46:58.390081 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:46:58.390041 4939 generic.go:356] "Generic (PLEG): container finished" podID="85bcbe6a-5881-4b40-acd4-1e3b6a560710" containerID="727b5e8dc501b261bfa7598dc1ee334e8979b5e0a0a08279c49be4a4ca783ea8" exitCode=0 Nov 03 00:46:58.390512 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:46:58.390088 4939 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-59fd54f5bc-lqb6l" Nov 03 00:46:58.390512 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:46:58.390091 4939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-59fd54f5bc-lqb6l" event={"ID":"85bcbe6a-5881-4b40-acd4-1e3b6a560710","Type":"ContainerDied","Data":"727b5e8dc501b261bfa7598dc1ee334e8979b5e0a0a08279c49be4a4ca783ea8"} Nov 03 00:46:58.390512 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:46:58.390198 4939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-59fd54f5bc-lqb6l" event={"ID":"85bcbe6a-5881-4b40-acd4-1e3b6a560710","Type":"ContainerDied","Data":"0762ff318f2cbb0233563b818a46d7dda56c321064dbab11da768db4f39aaeee"} Nov 03 00:46:58.390512 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:46:58.390218 4939 scope.go:122] "RemoveContainer" containerID="727b5e8dc501b261bfa7598dc1ee334e8979b5e0a0a08279c49be4a4ca783ea8" Nov 03 00:46:58.398734 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:46:58.398715 4939 scope.go:122] "RemoveContainer" containerID="727b5e8dc501b261bfa7598dc1ee334e8979b5e0a0a08279c49be4a4ca783ea8" Nov 03 00:46:58.399015 ip-10-0-140-107 kubenswrapper[4939]: E1103 00:46:58.398992 4939 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"727b5e8dc501b261bfa7598dc1ee334e8979b5e0a0a08279c49be4a4ca783ea8\": container with ID starting with 727b5e8dc501b261bfa7598dc1ee334e8979b5e0a0a08279c49be4a4ca783ea8 not found: ID does not exist" containerID="727b5e8dc501b261bfa7598dc1ee334e8979b5e0a0a08279c49be4a4ca783ea8" Nov 03 00:46:58.399089 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:46:58.399027 4939 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"727b5e8dc501b261bfa7598dc1ee334e8979b5e0a0a08279c49be4a4ca783ea8"} err="failed to get container status \"727b5e8dc501b261bfa7598dc1ee334e8979b5e0a0a08279c49be4a4ca783ea8\": rpc error: code = NotFound desc = could not find container \"727b5e8dc501b261bfa7598dc1ee334e8979b5e0a0a08279c49be4a4ca783ea8\": container with ID starting with 727b5e8dc501b261bfa7598dc1ee334e8979b5e0a0a08279c49be4a4ca783ea8 not found: ID does not exist" Nov 03 00:46:58.408036 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:46:58.408014 4939 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-59fd54f5bc-lqb6l"] Nov 03 00:46:58.411828 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:46:58.411806 4939 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-controller-59fd54f5bc-lqb6l"] Nov 03 00:46:59.865823 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:46:59.865787 4939 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="85bcbe6a-5881-4b40-acd4-1e3b6a560710" path="/var/lib/kubelet/pods/85bcbe6a-5881-4b40-acd4-1e3b6a560710/volumes" Nov 03 00:47:15.861153 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:47:15.861116 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 00:47:17.179002 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:47:17.178963 4939 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-66fb685bcd-wmhbg"] Nov 03 00:47:17.179477 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:47:17.179336 4939 cpu_manager.go:414] "containerMap: removing container" podUID="85bcbe6a-5881-4b40-acd4-1e3b6a560710" containerName="manager" Nov 03 00:47:17.179477 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:47:17.179348 4939 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="85bcbe6a-5881-4b40-acd4-1e3b6a560710" containerName="manager" Nov 03 00:47:17.179477 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:47:17.179419 4939 memory_manager.go:365] "RemoveStaleState removing state" podUID="85bcbe6a-5881-4b40-acd4-1e3b6a560710" containerName="manager" Nov 03 00:47:17.182353 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:47:17.182329 4939 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-66fb685bcd-wmhbg" Nov 03 00:47:17.197405 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:47:17.197100 4939 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 03 00:47:17.197405 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:47:17.197142 4939 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-nfcxl\"" Nov 03 00:47:17.197405 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:47:17.197235 4939 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-metrics-tls\"" Nov 03 00:47:17.197405 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:47:17.197349 4939 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 03 00:47:17.197405 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:47:17.197380 4939 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-serving-cert\"" Nov 03 00:47:17.198785 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:47:17.198756 4939 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-66fb685bcd-wmhbg"] Nov 03 00:47:17.279062 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:47:17.279033 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gh9fj\" (UniqueName: \"kubernetes.io/projected/8da5e4e6-0a92-46fb-b179-b89e52f0e7cc-kube-api-access-gh9fj\") pod \"maas-api-66fb685bcd-wmhbg\" (UID: \"8da5e4e6-0a92-46fb-b179-b89e52f0e7cc\") " pod="odh-ai-gateway-infra/maas-api-66fb685bcd-wmhbg" Nov 03 00:47:17.279213 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:47:17.279077 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8da5e4e6-0a92-46fb-b179-b89e52f0e7cc-maas-api-tls\") pod \"maas-api-66fb685bcd-wmhbg\" (UID: \"8da5e4e6-0a92-46fb-b179-b89e52f0e7cc\") " pod="odh-ai-gateway-infra/maas-api-66fb685bcd-wmhbg" Nov 03 00:47:17.279213 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:47:17.279107 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8da5e4e6-0a92-46fb-b179-b89e52f0e7cc-metrics-certs\") pod \"maas-api-66fb685bcd-wmhbg\" (UID: \"8da5e4e6-0a92-46fb-b179-b89e52f0e7cc\") " pod="odh-ai-gateway-infra/maas-api-66fb685bcd-wmhbg" Nov 03 00:47:17.279213 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:47:17.279189 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8da5e4e6-0a92-46fb-b179-b89e52f0e7cc-openshift-service-ca\") pod \"maas-api-66fb685bcd-wmhbg\" (UID: \"8da5e4e6-0a92-46fb-b179-b89e52f0e7cc\") " pod="odh-ai-gateway-infra/maas-api-66fb685bcd-wmhbg" Nov 03 00:47:17.380201 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:47:17.380172 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gh9fj\" (UniqueName: \"kubernetes.io/projected/8da5e4e6-0a92-46fb-b179-b89e52f0e7cc-kube-api-access-gh9fj\") pod \"maas-api-66fb685bcd-wmhbg\" (UID: \"8da5e4e6-0a92-46fb-b179-b89e52f0e7cc\") " pod="odh-ai-gateway-infra/maas-api-66fb685bcd-wmhbg" Nov 03 00:47:17.380353 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:47:17.380213 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8da5e4e6-0a92-46fb-b179-b89e52f0e7cc-maas-api-tls\") pod \"maas-api-66fb685bcd-wmhbg\" (UID: \"8da5e4e6-0a92-46fb-b179-b89e52f0e7cc\") " pod="odh-ai-gateway-infra/maas-api-66fb685bcd-wmhbg" Nov 03 00:47:17.380353 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:47:17.380269 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8da5e4e6-0a92-46fb-b179-b89e52f0e7cc-metrics-certs\") pod \"maas-api-66fb685bcd-wmhbg\" (UID: \"8da5e4e6-0a92-46fb-b179-b89e52f0e7cc\") " pod="odh-ai-gateway-infra/maas-api-66fb685bcd-wmhbg" Nov 03 00:47:17.380353 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:47:17.380309 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8da5e4e6-0a92-46fb-b179-b89e52f0e7cc-openshift-service-ca\") pod \"maas-api-66fb685bcd-wmhbg\" (UID: \"8da5e4e6-0a92-46fb-b179-b89e52f0e7cc\") " pod="odh-ai-gateway-infra/maas-api-66fb685bcd-wmhbg" Nov 03 00:47:17.380987 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:47:17.380965 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8da5e4e6-0a92-46fb-b179-b89e52f0e7cc-openshift-service-ca\") pod \"maas-api-66fb685bcd-wmhbg\" (UID: \"8da5e4e6-0a92-46fb-b179-b89e52f0e7cc\") " pod="odh-ai-gateway-infra/maas-api-66fb685bcd-wmhbg" Nov 03 00:47:17.382732 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:47:17.382708 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8da5e4e6-0a92-46fb-b179-b89e52f0e7cc-metrics-certs\") pod \"maas-api-66fb685bcd-wmhbg\" (UID: \"8da5e4e6-0a92-46fb-b179-b89e52f0e7cc\") " pod="odh-ai-gateway-infra/maas-api-66fb685bcd-wmhbg" Nov 03 00:47:17.382826 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:47:17.382753 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8da5e4e6-0a92-46fb-b179-b89e52f0e7cc-maas-api-tls\") pod \"maas-api-66fb685bcd-wmhbg\" (UID: \"8da5e4e6-0a92-46fb-b179-b89e52f0e7cc\") " pod="odh-ai-gateway-infra/maas-api-66fb685bcd-wmhbg" Nov 03 00:47:17.388388 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:47:17.388366 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gh9fj\" (UniqueName: \"kubernetes.io/projected/8da5e4e6-0a92-46fb-b179-b89e52f0e7cc-kube-api-access-gh9fj\") pod \"maas-api-66fb685bcd-wmhbg\" (UID: \"8da5e4e6-0a92-46fb-b179-b89e52f0e7cc\") " pod="odh-ai-gateway-infra/maas-api-66fb685bcd-wmhbg" Nov 03 00:47:17.426615 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:47:17.426574 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 00:47:17.495833 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:47:17.495754 4939 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-66fb685bcd-wmhbg" Nov 03 00:47:17.768884 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:47:17.768860 4939 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-66fb685bcd-wmhbg"] Nov 03 00:47:17.771400 ip-10-0-140-107 kubenswrapper[4939]: W1103 00:47:17.771358 4939 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8da5e4e6_0a92_46fb_b179_b89e52f0e7cc.slice/crio-26608fa6b6fd48bb4bc103f452f1bd2b2c855efacd4b5743f9672da84ccb3330 WatchSource:0}: Error finding container 26608fa6b6fd48bb4bc103f452f1bd2b2c855efacd4b5743f9672da84ccb3330: Status 404 returned error can't find the container with id 26608fa6b6fd48bb4bc103f452f1bd2b2c855efacd4b5743f9672da84ccb3330 Nov 03 00:47:17.971017 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:47:17.970983 4939 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-4pw7v"] Nov 03 00:47:18.030371 ip-10-0-140-107 kubenswrapper[4939]: E1103 00:47:18.030293 4939 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-sthjh" containerName="extractor" Nov 03 00:47:18.470689 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:47:18.470654 4939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-66fb685bcd-wmhbg" event={"ID":"8da5e4e6-0a92-46fb-b179-b89e52f0e7cc","Type":"ContainerStarted","Data":"26608fa6b6fd48bb4bc103f452f1bd2b2c855efacd4b5743f9672da84ccb3330"} Nov 03 00:47:20.480520 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:47:20.480418 4939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-66fb685bcd-wmhbg" event={"ID":"8da5e4e6-0a92-46fb-b179-b89e52f0e7cc","Type":"ContainerStarted","Data":"fa4883dd21f4e72580629978870b5a5b01c47f9c180c864836d6722351c9cd05"} Nov 03 00:47:20.480965 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:47:20.480515 4939 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-66fb685bcd-wmhbg" Nov 03 00:47:20.499860 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:47:20.499809 4939 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-66fb685bcd-wmhbg" podStartSLOduration=1.04861438 podStartE2EDuration="3.499796272s" totalImagesPullingTime="2.451181892s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:47:17 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:47:20.498247133 +0000 UTC m=+388.762582078" watchObservedRunningTime="2026-11-03 00:47:20.499796272 +0000 UTC m=+388.764131214" Nov 03 00:47:26.491144 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:47:26.491109 4939 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-66fb685bcd-wmhbg" Nov 03 00:47:26.920787 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:47:26.920750 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 00:47:27.217709 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:47:27.217624 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 00:47:39.356524 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:47:39.356436 4939 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-66fb685bcd-wmhbg"] Nov 03 00:47:39.357025 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:47:39.356654 4939 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-66fb685bcd-wmhbg" podUID="8da5e4e6-0a92-46fb-b179-b89e52f0e7cc" containerName="maas-api" containerID="cri-o://fa4883dd21f4e72580629978870b5a5b01c47f9c180c864836d6722351c9cd05" gracePeriod=30 Nov 03 00:47:39.560720 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:47:39.560682 4939 generic.go:356] "Generic (PLEG): container finished" podID="8da5e4e6-0a92-46fb-b179-b89e52f0e7cc" containerID="fa4883dd21f4e72580629978870b5a5b01c47f9c180c864836d6722351c9cd05" exitCode=0 Nov 03 00:47:39.560888 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:47:39.560761 4939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-66fb685bcd-wmhbg" event={"ID":"8da5e4e6-0a92-46fb-b179-b89e52f0e7cc","Type":"ContainerDied","Data":"fa4883dd21f4e72580629978870b5a5b01c47f9c180c864836d6722351c9cd05"} Nov 03 00:47:39.594961 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:47:39.594936 4939 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-66fb685bcd-wmhbg" Nov 03 00:47:39.650162 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:47:39.650097 4939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8da5e4e6-0a92-46fb-b179-b89e52f0e7cc-kube-api-access-gh9fj\" (UniqueName: \"kubernetes.io/projected/8da5e4e6-0a92-46fb-b179-b89e52f0e7cc-kube-api-access-gh9fj\") pod \"8da5e4e6-0a92-46fb-b179-b89e52f0e7cc\" (UID: \"8da5e4e6-0a92-46fb-b179-b89e52f0e7cc\") " Nov 03 00:47:39.650162 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:47:39.650140 4939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8da5e4e6-0a92-46fb-b179-b89e52f0e7cc-metrics-certs\" (UniqueName: \"kubernetes.io/secret/8da5e4e6-0a92-46fb-b179-b89e52f0e7cc-metrics-certs\") pod \"8da5e4e6-0a92-46fb-b179-b89e52f0e7cc\" (UID: \"8da5e4e6-0a92-46fb-b179-b89e52f0e7cc\") " Nov 03 00:47:39.650368 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:47:39.650171 4939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8da5e4e6-0a92-46fb-b179-b89e52f0e7cc-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8da5e4e6-0a92-46fb-b179-b89e52f0e7cc-maas-api-tls\") pod \"8da5e4e6-0a92-46fb-b179-b89e52f0e7cc\" (UID: \"8da5e4e6-0a92-46fb-b179-b89e52f0e7cc\") " Nov 03 00:47:39.650368 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:47:39.650191 4939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8da5e4e6-0a92-46fb-b179-b89e52f0e7cc-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8da5e4e6-0a92-46fb-b179-b89e52f0e7cc-openshift-service-ca\") pod \"8da5e4e6-0a92-46fb-b179-b89e52f0e7cc\" (UID: \"8da5e4e6-0a92-46fb-b179-b89e52f0e7cc\") " Nov 03 00:47:39.650687 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:47:39.650655 4939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8da5e4e6-0a92-46fb-b179-b89e52f0e7cc-openshift-service-ca" pod "8da5e4e6-0a92-46fb-b179-b89e52f0e7cc" (UID: "8da5e4e6-0a92-46fb-b179-b89e52f0e7cc"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 00:47:39.652369 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:47:39.652338 4939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8da5e4e6-0a92-46fb-b179-b89e52f0e7cc-metrics-certs" pod "8da5e4e6-0a92-46fb-b179-b89e52f0e7cc" (UID: "8da5e4e6-0a92-46fb-b179-b89e52f0e7cc"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 00:47:39.652722 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:47:39.652696 4939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8da5e4e6-0a92-46fb-b179-b89e52f0e7cc-kube-api-access-gh9fj" pod "8da5e4e6-0a92-46fb-b179-b89e52f0e7cc" (UID: "8da5e4e6-0a92-46fb-b179-b89e52f0e7cc"). InnerVolumeSpecName "kube-api-access-gh9fj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 00:47:39.652807 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:47:39.652778 4939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8da5e4e6-0a92-46fb-b179-b89e52f0e7cc-maas-api-tls" pod "8da5e4e6-0a92-46fb-b179-b89e52f0e7cc" (UID: "8da5e4e6-0a92-46fb-b179-b89e52f0e7cc"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 00:47:39.751296 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:47:39.751261 4939 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gh9fj\" (UniqueName: \"kubernetes.io/projected/8da5e4e6-0a92-46fb-b179-b89e52f0e7cc-kube-api-access-gh9fj\") on node \"ip-10-0-140-107.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 00:47:39.751296 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:47:39.751290 4939 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8da5e4e6-0a92-46fb-b179-b89e52f0e7cc-metrics-certs\") on node \"ip-10-0-140-107.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 00:47:39.751296 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:47:39.751304 4939 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8da5e4e6-0a92-46fb-b179-b89e52f0e7cc-maas-api-tls\") on node \"ip-10-0-140-107.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 00:47:39.751533 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:47:39.751312 4939 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8da5e4e6-0a92-46fb-b179-b89e52f0e7cc-openshift-service-ca\") on node \"ip-10-0-140-107.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 00:47:40.566963 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:47:40.566926 4939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-66fb685bcd-wmhbg" event={"ID":"8da5e4e6-0a92-46fb-b179-b89e52f0e7cc","Type":"ContainerDied","Data":"26608fa6b6fd48bb4bc103f452f1bd2b2c855efacd4b5743f9672da84ccb3330"} Nov 03 00:47:40.566963 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:47:40.566959 4939 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-66fb685bcd-wmhbg" Nov 03 00:47:40.567447 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:47:40.566987 4939 scope.go:122] "RemoveContainer" containerID="fa4883dd21f4e72580629978870b5a5b01c47f9c180c864836d6722351c9cd05" Nov 03 00:47:40.586803 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:47:40.586776 4939 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-66fb685bcd-wmhbg"] Nov 03 00:47:40.592933 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:47:40.592897 4939 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-66fb685bcd-wmhbg"] Nov 03 00:47:41.617420 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:47:41.617386 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 00:47:41.870496 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:47:41.870419 4939 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8da5e4e6-0a92-46fb-b179-b89e52f0e7cc" path="/var/lib/kubelet/pods/8da5e4e6-0a92-46fb-b179-b89e52f0e7cc/volumes" Nov 03 00:47:48.028730 ip-10-0-140-107 kubenswrapper[4939]: E1103 00:47:48.028691 4939 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-sthjh" containerName="extractor" Nov 03 00:48:04.451293 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:48:04.451255 4939 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/ai-gateway-controller-5d8dddb676-9fw7q"] Nov 03 00:48:04.451753 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:48:04.451609 4939 cpu_manager.go:414] "containerMap: removing container" podUID="8da5e4e6-0a92-46fb-b179-b89e52f0e7cc" containerName="maas-api" Nov 03 00:48:04.451753 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:48:04.451621 4939 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8da5e4e6-0a92-46fb-b179-b89e52f0e7cc" containerName="maas-api" Nov 03 00:48:04.451753 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:48:04.451689 4939 memory_manager.go:365] "RemoveStaleState removing state" podUID="8da5e4e6-0a92-46fb-b179-b89e52f0e7cc" containerName="maas-api" Nov 03 00:48:04.454611 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:48:04.454594 4939 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/ai-gateway-controller-5d8dddb676-9fw7q" Nov 03 00:48:04.460394 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:48:04.460364 4939 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"ai-gateway-controller-parameters\"" Nov 03 00:48:04.460394 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:48:04.460382 4939 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"ai-gateway-controller-metrics-tls\"" Nov 03 00:48:04.460558 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:48:04.460382 4939 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"ai-gateway-controller-dockercfg-8s5dl\"" Nov 03 00:48:04.467812 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:48:04.467780 4939 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/ai-gateway-controller-5d8dddb676-9fw7q"] Nov 03 00:48:04.498984 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:48:04.498949 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 00:48:04.544903 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:48:04.544867 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/46e2ee17-5a32-4025-a744-fca710d58acc-metrics-certs\") pod \"ai-gateway-controller-5d8dddb676-9fw7q\" (UID: \"46e2ee17-5a32-4025-a744-fca710d58acc\") " pod="opendatahub/ai-gateway-controller-5d8dddb676-9fw7q" Nov 03 00:48:04.545075 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:48:04.544996 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h5gdn\" (UniqueName: \"kubernetes.io/projected/46e2ee17-5a32-4025-a744-fca710d58acc-kube-api-access-h5gdn\") pod \"ai-gateway-controller-5d8dddb676-9fw7q\" (UID: \"46e2ee17-5a32-4025-a744-fca710d58acc\") " pod="opendatahub/ai-gateway-controller-5d8dddb676-9fw7q" Nov 03 00:48:04.646066 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:48:04.646033 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h5gdn\" (UniqueName: \"kubernetes.io/projected/46e2ee17-5a32-4025-a744-fca710d58acc-kube-api-access-h5gdn\") pod \"ai-gateway-controller-5d8dddb676-9fw7q\" (UID: \"46e2ee17-5a32-4025-a744-fca710d58acc\") " pod="opendatahub/ai-gateway-controller-5d8dddb676-9fw7q" Nov 03 00:48:04.646230 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:48:04.646089 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/46e2ee17-5a32-4025-a744-fca710d58acc-metrics-certs\") pod \"ai-gateway-controller-5d8dddb676-9fw7q\" (UID: \"46e2ee17-5a32-4025-a744-fca710d58acc\") " pod="opendatahub/ai-gateway-controller-5d8dddb676-9fw7q" Nov 03 00:48:04.648493 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:48:04.648463 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/46e2ee17-5a32-4025-a744-fca710d58acc-metrics-certs\") pod \"ai-gateway-controller-5d8dddb676-9fw7q\" (UID: \"46e2ee17-5a32-4025-a744-fca710d58acc\") " pod="opendatahub/ai-gateway-controller-5d8dddb676-9fw7q" Nov 03 00:48:04.656319 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:48:04.656297 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-h5gdn\" (UniqueName: \"kubernetes.io/projected/46e2ee17-5a32-4025-a744-fca710d58acc-kube-api-access-h5gdn\") pod \"ai-gateway-controller-5d8dddb676-9fw7q\" (UID: \"46e2ee17-5a32-4025-a744-fca710d58acc\") " pod="opendatahub/ai-gateway-controller-5d8dddb676-9fw7q" Nov 03 00:48:04.668525 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:48:04.668499 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 00:48:04.767827 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:48:04.767743 4939 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/ai-gateway-controller-5d8dddb676-9fw7q" Nov 03 00:48:05.040611 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:48:05.040409 4939 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/ai-gateway-controller-5d8dddb676-9fw7q"] Nov 03 00:48:05.043108 ip-10-0-140-107 kubenswrapper[4939]: W1103 00:48:05.043079 4939 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod46e2ee17_5a32_4025_a744_fca710d58acc.slice/crio-52b274ba62413220e0b5dbb916309dc3e039a33977a396b0ae91a72a692c6d8b WatchSource:0}: Error finding container 52b274ba62413220e0b5dbb916309dc3e039a33977a396b0ae91a72a692c6d8b: Status 404 returned error can't find the container with id 52b274ba62413220e0b5dbb916309dc3e039a33977a396b0ae91a72a692c6d8b Nov 03 00:48:05.667182 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:48:05.667142 4939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/ai-gateway-controller-5d8dddb676-9fw7q" event={"ID":"46e2ee17-5a32-4025-a744-fca710d58acc","Type":"ContainerStarted","Data":"52b274ba62413220e0b5dbb916309dc3e039a33977a396b0ae91a72a692c6d8b"} Nov 03 00:48:07.677746 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:48:07.677708 4939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/ai-gateway-controller-5d8dddb676-9fw7q" event={"ID":"46e2ee17-5a32-4025-a744-fca710d58acc","Type":"ContainerStarted","Data":"20eac6fd6c4d366759508d430a25818948c3537b715726f2f76b2babe9a2463e"} Nov 03 00:48:07.678159 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:48:07.677843 4939 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/ai-gateway-controller-5d8dddb676-9fw7q" Nov 03 00:48:07.696428 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:48:07.696382 4939 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/ai-gateway-controller-5d8dddb676-9fw7q" podStartSLOduration=1.683908804 podStartE2EDuration="3.696362728s" totalImagesPullingTime="2.012453924s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:48:04 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:48:07.695400436 +0000 UTC m=+435.959735403" watchObservedRunningTime="2026-11-03 00:48:07.696362728 +0000 UTC m=+435.960697705" Nov 03 00:48:08.847287 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:48:08.847251 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 00:48:08.853426 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:48:08.853399 4939 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-7b98bb678c-6q4gn"] Nov 03 00:48:08.856875 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:48:08.856855 4939 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-7b98bb678c-6q4gn" Nov 03 00:48:08.866016 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:48:08.865994 4939 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins\"" Nov 03 00:48:08.866260 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:48:08.866244 4939 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-dockercfg-l45h2\"" Nov 03 00:48:08.873995 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:48:08.873965 4939 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-7b98bb678c-6q4gn"] Nov 03 00:48:08.912082 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:48:08.912049 4939 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-55fbbbf9dc-qvj7w"] Nov 03 00:48:08.915424 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:48:08.915407 4939 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-55fbbbf9dc-qvj7w" Nov 03 00:48:08.926276 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:48:08.926249 4939 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-55fbbbf9dc-qvj7w"] Nov 03 00:48:08.980824 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:48:08.980785 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7cab38b8-5158-4678-a429-350af5e2f72f-plugins-config-volume\") pod \"payload-processing-55fbbbf9dc-qvj7w\" (UID: \"7cab38b8-5158-4678-a429-350af5e2f72f\") " pod="openshift-ingress/payload-processing-55fbbbf9dc-qvj7w" Nov 03 00:48:08.980824 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:48:08.980826 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lpcts\" (UniqueName: \"kubernetes.io/projected/7cab38b8-5158-4678-a429-350af5e2f72f-kube-api-access-lpcts\") pod \"payload-processing-55fbbbf9dc-qvj7w\" (UID: \"7cab38b8-5158-4678-a429-350af5e2f72f\") " pod="openshift-ingress/payload-processing-55fbbbf9dc-qvj7w" Nov 03 00:48:08.981044 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:48:08.980946 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6dde687f-2bc9-4f1f-9dac-9b7557f0327e-plugins-config-volume\") pod \"payload-pre-processing-7b98bb678c-6q4gn\" (UID: \"6dde687f-2bc9-4f1f-9dac-9b7557f0327e\") " pod="openshift-ingress/payload-pre-processing-7b98bb678c-6q4gn" Nov 03 00:48:08.981095 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:48:08.981050 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8l8g4\" (UniqueName: \"kubernetes.io/projected/6dde687f-2bc9-4f1f-9dac-9b7557f0327e-kube-api-access-8l8g4\") pod \"payload-pre-processing-7b98bb678c-6q4gn\" (UID: \"6dde687f-2bc9-4f1f-9dac-9b7557f0327e\") " pod="openshift-ingress/payload-pre-processing-7b98bb678c-6q4gn" Nov 03 00:48:09.000637 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:48:09.000604 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 00:48:09.082105 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:48:09.082067 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8l8g4\" (UniqueName: \"kubernetes.io/projected/6dde687f-2bc9-4f1f-9dac-9b7557f0327e-kube-api-access-8l8g4\") pod \"payload-pre-processing-7b98bb678c-6q4gn\" (UID: \"6dde687f-2bc9-4f1f-9dac-9b7557f0327e\") " pod="openshift-ingress/payload-pre-processing-7b98bb678c-6q4gn" Nov 03 00:48:09.082295 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:48:09.082134 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7cab38b8-5158-4678-a429-350af5e2f72f-plugins-config-volume\") pod \"payload-processing-55fbbbf9dc-qvj7w\" (UID: \"7cab38b8-5158-4678-a429-350af5e2f72f\") " pod="openshift-ingress/payload-processing-55fbbbf9dc-qvj7w" Nov 03 00:48:09.082295 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:48:09.082286 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lpcts\" (UniqueName: \"kubernetes.io/projected/7cab38b8-5158-4678-a429-350af5e2f72f-kube-api-access-lpcts\") pod \"payload-processing-55fbbbf9dc-qvj7w\" (UID: \"7cab38b8-5158-4678-a429-350af5e2f72f\") " pod="openshift-ingress/payload-processing-55fbbbf9dc-qvj7w" Nov 03 00:48:09.082422 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:48:09.082334 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6dde687f-2bc9-4f1f-9dac-9b7557f0327e-plugins-config-volume\") pod \"payload-pre-processing-7b98bb678c-6q4gn\" (UID: \"6dde687f-2bc9-4f1f-9dac-9b7557f0327e\") " pod="openshift-ingress/payload-pre-processing-7b98bb678c-6q4gn" Nov 03 00:48:09.082928 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:48:09.082895 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7cab38b8-5158-4678-a429-350af5e2f72f-plugins-config-volume\") pod \"payload-processing-55fbbbf9dc-qvj7w\" (UID: \"7cab38b8-5158-4678-a429-350af5e2f72f\") " pod="openshift-ingress/payload-processing-55fbbbf9dc-qvj7w" Nov 03 00:48:09.083068 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:48:09.083049 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6dde687f-2bc9-4f1f-9dac-9b7557f0327e-plugins-config-volume\") pod \"payload-pre-processing-7b98bb678c-6q4gn\" (UID: \"6dde687f-2bc9-4f1f-9dac-9b7557f0327e\") " pod="openshift-ingress/payload-pre-processing-7b98bb678c-6q4gn" Nov 03 00:48:09.093967 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:48:09.093948 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lpcts\" (UniqueName: \"kubernetes.io/projected/7cab38b8-5158-4678-a429-350af5e2f72f-kube-api-access-lpcts\") pod \"payload-processing-55fbbbf9dc-qvj7w\" (UID: \"7cab38b8-5158-4678-a429-350af5e2f72f\") " pod="openshift-ingress/payload-processing-55fbbbf9dc-qvj7w" Nov 03 00:48:09.094043 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:48:09.093952 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8l8g4\" (UniqueName: \"kubernetes.io/projected/6dde687f-2bc9-4f1f-9dac-9b7557f0327e-kube-api-access-8l8g4\") pod \"payload-pre-processing-7b98bb678c-6q4gn\" (UID: \"6dde687f-2bc9-4f1f-9dac-9b7557f0327e\") " pod="openshift-ingress/payload-pre-processing-7b98bb678c-6q4gn" Nov 03 00:48:09.169035 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:48:09.168945 4939 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-7b98bb678c-6q4gn" Nov 03 00:48:09.230318 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:48:09.230282 4939 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-55fbbbf9dc-qvj7w" Nov 03 00:48:09.447297 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:48:09.447150 4939 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-7b98bb678c-6q4gn"] Nov 03 00:48:09.449681 ip-10-0-140-107 kubenswrapper[4939]: W1103 00:48:09.449645 4939 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6dde687f_2bc9_4f1f_9dac_9b7557f0327e.slice/crio-26ec763c6b27b76bf345e225d6abcbd02f5686849482a97da4c4527a6fb4411d WatchSource:0}: Error finding container 26ec763c6b27b76bf345e225d6abcbd02f5686849482a97da4c4527a6fb4411d: Status 404 returned error can't find the container with id 26ec763c6b27b76bf345e225d6abcbd02f5686849482a97da4c4527a6fb4411d Nov 03 00:48:09.508936 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:48:09.508897 4939 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-55fbbbf9dc-qvj7w"] Nov 03 00:48:09.511323 ip-10-0-140-107 kubenswrapper[4939]: W1103 00:48:09.511294 4939 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7cab38b8_5158_4678_a429_350af5e2f72f.slice/crio-fe1d1261ba4a7fac93fa31f0a0a52007d46dd64d6ec353e3f71b4b074e57eb5f WatchSource:0}: Error finding container fe1d1261ba4a7fac93fa31f0a0a52007d46dd64d6ec353e3f71b4b074e57eb5f: Status 404 returned error can't find the container with id fe1d1261ba4a7fac93fa31f0a0a52007d46dd64d6ec353e3f71b4b074e57eb5f Nov 03 00:48:09.687282 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:48:09.687239 4939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-7b98bb678c-6q4gn" event={"ID":"6dde687f-2bc9-4f1f-9dac-9b7557f0327e","Type":"ContainerStarted","Data":"26ec763c6b27b76bf345e225d6abcbd02f5686849482a97da4c4527a6fb4411d"} Nov 03 00:48:09.688502 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:48:09.688472 4939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-55fbbbf9dc-qvj7w" event={"ID":"7cab38b8-5158-4678-a429-350af5e2f72f","Type":"ContainerStarted","Data":"fe1d1261ba4a7fac93fa31f0a0a52007d46dd64d6ec353e3f71b4b074e57eb5f"} Nov 03 00:48:12.707467 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:48:12.707428 4939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-7b98bb678c-6q4gn" event={"ID":"6dde687f-2bc9-4f1f-9dac-9b7557f0327e","Type":"ContainerStarted","Data":"bd29709ad3952ca1b1796127296c134e3de3dfbf4aaa56ca17f8c8c5843788b8"} Nov 03 00:48:12.707979 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:48:12.707582 4939 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-7b98bb678c-6q4gn" Nov 03 00:48:12.709080 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:48:12.709057 4939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-55fbbbf9dc-qvj7w" event={"ID":"7cab38b8-5158-4678-a429-350af5e2f72f","Type":"ContainerStarted","Data":"616536dff201b76ae71b9d490fc4aee724e32dd83b0d45b81bfcdd86208c24db"} Nov 03 00:48:12.709187 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:48:12.709144 4939 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-55fbbbf9dc-qvj7w" Nov 03 00:48:12.745974 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:48:12.745930 4939 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-7b98bb678c-6q4gn" podStartSLOduration=2.335141633 podStartE2EDuration="4.745898316s" totalImagesPullingTime="2.410756683s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:48:08 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:48:12.727774099 +0000 UTC m=+440.992109048" watchObservedRunningTime="2026-11-03 00:48:12.745898316 +0000 UTC m=+441.010233326" Nov 03 00:48:12.746325 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:48:12.746299 4939 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-55fbbbf9dc-qvj7w" podStartSLOduration=2.3627036710000002 podStartE2EDuration="4.746292437s" totalImagesPullingTime="2.383588766s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:48:08 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:48:12.745065207 +0000 UTC m=+441.009400153" watchObservedRunningTime="2026-11-03 00:48:12.746292437 +0000 UTC m=+441.010627382" Nov 03 00:48:18.029179 ip-10-0-140-107 kubenswrapper[4939]: E1103 00:48:18.029139 4939 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-sthjh" containerName="extractor" Nov 03 00:48:18.684622 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:48:18.684586 4939 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/ai-gateway-controller-5d8dddb676-9fw7q" Nov 03 00:48:18.942756 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:48:18.942669 4939 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/ai-gateway-controller-75bbf6968c-fz65x"] Nov 03 00:48:18.946213 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:48:18.946191 4939 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/ai-gateway-controller-75bbf6968c-fz65x" Nov 03 00:48:18.953853 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:48:18.953826 4939 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/ai-gateway-controller-75bbf6968c-fz65x"] Nov 03 00:48:19.068998 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:48:19.068966 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5n2hc\" (UniqueName: \"kubernetes.io/projected/f690a187-3701-4c28-a425-e5e1f447b650-kube-api-access-5n2hc\") pod \"ai-gateway-controller-75bbf6968c-fz65x\" (UID: \"f690a187-3701-4c28-a425-e5e1f447b650\") " pod="opendatahub/ai-gateway-controller-75bbf6968c-fz65x" Nov 03 00:48:19.069375 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:48:19.069011 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f690a187-3701-4c28-a425-e5e1f447b650-metrics-certs\") pod \"ai-gateway-controller-75bbf6968c-fz65x\" (UID: \"f690a187-3701-4c28-a425-e5e1f447b650\") " pod="opendatahub/ai-gateway-controller-75bbf6968c-fz65x" Nov 03 00:48:19.169475 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:48:19.169442 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5n2hc\" (UniqueName: \"kubernetes.io/projected/f690a187-3701-4c28-a425-e5e1f447b650-kube-api-access-5n2hc\") pod \"ai-gateway-controller-75bbf6968c-fz65x\" (UID: \"f690a187-3701-4c28-a425-e5e1f447b650\") " pod="opendatahub/ai-gateway-controller-75bbf6968c-fz65x" Nov 03 00:48:19.169623 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:48:19.169486 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f690a187-3701-4c28-a425-e5e1f447b650-metrics-certs\") pod \"ai-gateway-controller-75bbf6968c-fz65x\" (UID: \"f690a187-3701-4c28-a425-e5e1f447b650\") " pod="opendatahub/ai-gateway-controller-75bbf6968c-fz65x" Nov 03 00:48:19.171978 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:48:19.171953 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f690a187-3701-4c28-a425-e5e1f447b650-metrics-certs\") pod \"ai-gateway-controller-75bbf6968c-fz65x\" (UID: \"f690a187-3701-4c28-a425-e5e1f447b650\") " pod="opendatahub/ai-gateway-controller-75bbf6968c-fz65x" Nov 03 00:48:19.178637 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:48:19.178616 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5n2hc\" (UniqueName: \"kubernetes.io/projected/f690a187-3701-4c28-a425-e5e1f447b650-kube-api-access-5n2hc\") pod \"ai-gateway-controller-75bbf6968c-fz65x\" (UID: \"f690a187-3701-4c28-a425-e5e1f447b650\") " pod="opendatahub/ai-gateway-controller-75bbf6968c-fz65x" Nov 03 00:48:19.260574 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:48:19.260487 4939 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/ai-gateway-controller-75bbf6968c-fz65x" Nov 03 00:48:19.539582 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:48:19.539552 4939 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/ai-gateway-controller-75bbf6968c-fz65x"] Nov 03 00:48:19.541465 ip-10-0-140-107 kubenswrapper[4939]: W1103 00:48:19.541432 4939 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf690a187_3701_4c28_a425_e5e1f447b650.slice/crio-85444c708cb39a8f908cf72c06a7324924a65fc6ec2b2391cf961d0dff809bcb WatchSource:0}: Error finding container 85444c708cb39a8f908cf72c06a7324924a65fc6ec2b2391cf961d0dff809bcb: Status 404 returned error can't find the container with id 85444c708cb39a8f908cf72c06a7324924a65fc6ec2b2391cf961d0dff809bcb Nov 03 00:48:19.737957 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:48:19.737894 4939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/ai-gateway-controller-75bbf6968c-fz65x" event={"ID":"f690a187-3701-4c28-a425-e5e1f447b650","Type":"ContainerStarted","Data":"85444c708cb39a8f908cf72c06a7324924a65fc6ec2b2391cf961d0dff809bcb"} Nov 03 00:48:20.743160 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:48:20.743123 4939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/ai-gateway-controller-75bbf6968c-fz65x" event={"ID":"f690a187-3701-4c28-a425-e5e1f447b650","Type":"ContainerStarted","Data":"5e5f359ff16e82b564e09f9f7e3b5609be03f6b710bfd691a782d1a522ba85c7"} Nov 03 00:48:20.743615 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:48:20.743258 4939 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/ai-gateway-controller-75bbf6968c-fz65x" Nov 03 00:48:20.763357 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:48:20.763305 4939 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/ai-gateway-controller-75bbf6968c-fz65x" podStartSLOduration=2.368838621 podStartE2EDuration="2.763290687s" totalImagesPullingTime="394.452066ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:48:18 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:48:20.762687149 +0000 UTC m=+449.027022093" watchObservedRunningTime="2026-11-03 00:48:20.763290687 +0000 UTC m=+449.027625642" Nov 03 00:48:23.717173 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:48:23.717140 4939 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-55fbbbf9dc-qvj7w" Nov 03 00:48:23.717632 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:48:23.717207 4939 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-7b98bb678c-6q4gn" Nov 03 00:48:28.379866 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:48:28.379819 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 00:48:28.522258 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:48:28.522222 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 00:48:31.750428 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:48:31.750392 4939 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/ai-gateway-controller-75bbf6968c-fz65x" Nov 03 00:48:31.790520 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:48:31.790494 4939 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/ai-gateway-controller-5d8dddb676-9fw7q"] Nov 03 00:48:31.790743 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:48:31.790704 4939 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/ai-gateway-controller-5d8dddb676-9fw7q" podUID="46e2ee17-5a32-4025-a744-fca710d58acc" containerName="manager" containerID="cri-o://20eac6fd6c4d366759508d430a25818948c3537b715726f2f76b2babe9a2463e" gracePeriod=30 Nov 03 00:48:32.037066 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:48:32.037040 4939 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/ai-gateway-controller-5d8dddb676-9fw7q" Nov 03 00:48:32.178972 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:48:32.178935 4939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/46e2ee17-5a32-4025-a744-fca710d58acc-kube-api-access-h5gdn\" (UniqueName: \"kubernetes.io/projected/46e2ee17-5a32-4025-a744-fca710d58acc-kube-api-access-h5gdn\") pod \"46e2ee17-5a32-4025-a744-fca710d58acc\" (UID: \"46e2ee17-5a32-4025-a744-fca710d58acc\") " Nov 03 00:48:32.179168 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:48:32.178991 4939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/46e2ee17-5a32-4025-a744-fca710d58acc-metrics-certs\" (UniqueName: \"kubernetes.io/secret/46e2ee17-5a32-4025-a744-fca710d58acc-metrics-certs\") pod \"46e2ee17-5a32-4025-a744-fca710d58acc\" (UID: \"46e2ee17-5a32-4025-a744-fca710d58acc\") " Nov 03 00:48:32.181081 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:48:32.181054 4939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/46e2ee17-5a32-4025-a744-fca710d58acc-metrics-certs" pod "46e2ee17-5a32-4025-a744-fca710d58acc" (UID: "46e2ee17-5a32-4025-a744-fca710d58acc"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 00:48:32.181187 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:48:32.181112 4939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/46e2ee17-5a32-4025-a744-fca710d58acc-kube-api-access-h5gdn" pod "46e2ee17-5a32-4025-a744-fca710d58acc" (UID: "46e2ee17-5a32-4025-a744-fca710d58acc"). InnerVolumeSpecName "kube-api-access-h5gdn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 00:48:32.280248 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:48:32.280179 4939 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h5gdn\" (UniqueName: \"kubernetes.io/projected/46e2ee17-5a32-4025-a744-fca710d58acc-kube-api-access-h5gdn\") on node \"ip-10-0-140-107.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 00:48:32.280248 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:48:32.280216 4939 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/46e2ee17-5a32-4025-a744-fca710d58acc-metrics-certs\") on node \"ip-10-0-140-107.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 00:48:32.797253 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:48:32.797216 4939 generic.go:356] "Generic (PLEG): container finished" podID="46e2ee17-5a32-4025-a744-fca710d58acc" containerID="20eac6fd6c4d366759508d430a25818948c3537b715726f2f76b2babe9a2463e" exitCode=0 Nov 03 00:48:32.797667 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:48:32.797260 4939 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/ai-gateway-controller-5d8dddb676-9fw7q" Nov 03 00:48:32.797667 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:48:32.797298 4939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/ai-gateway-controller-5d8dddb676-9fw7q" event={"ID":"46e2ee17-5a32-4025-a744-fca710d58acc","Type":"ContainerDied","Data":"20eac6fd6c4d366759508d430a25818948c3537b715726f2f76b2babe9a2463e"} Nov 03 00:48:32.797667 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:48:32.797352 4939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/ai-gateway-controller-5d8dddb676-9fw7q" event={"ID":"46e2ee17-5a32-4025-a744-fca710d58acc","Type":"ContainerDied","Data":"52b274ba62413220e0b5dbb916309dc3e039a33977a396b0ae91a72a692c6d8b"} Nov 03 00:48:32.797667 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:48:32.797376 4939 scope.go:122] "RemoveContainer" containerID="20eac6fd6c4d366759508d430a25818948c3537b715726f2f76b2babe9a2463e" Nov 03 00:48:32.806060 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:48:32.806034 4939 scope.go:122] "RemoveContainer" containerID="20eac6fd6c4d366759508d430a25818948c3537b715726f2f76b2babe9a2463e" Nov 03 00:48:32.806309 ip-10-0-140-107 kubenswrapper[4939]: E1103 00:48:32.806293 4939 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"20eac6fd6c4d366759508d430a25818948c3537b715726f2f76b2babe9a2463e\": container with ID starting with 20eac6fd6c4d366759508d430a25818948c3537b715726f2f76b2babe9a2463e not found: ID does not exist" containerID="20eac6fd6c4d366759508d430a25818948c3537b715726f2f76b2babe9a2463e" Nov 03 00:48:32.806362 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:48:32.806319 4939 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"20eac6fd6c4d366759508d430a25818948c3537b715726f2f76b2babe9a2463e"} err="failed to get container status \"20eac6fd6c4d366759508d430a25818948c3537b715726f2f76b2babe9a2463e\": rpc error: code = NotFound desc = could not find container \"20eac6fd6c4d366759508d430a25818948c3537b715726f2f76b2babe9a2463e\": container with ID starting with 20eac6fd6c4d366759508d430a25818948c3537b715726f2f76b2babe9a2463e not found: ID does not exist" Nov 03 00:48:32.818354 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:48:32.818330 4939 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/ai-gateway-controller-5d8dddb676-9fw7q"] Nov 03 00:48:32.822151 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:48:32.822130 4939 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/ai-gateway-controller-5d8dddb676-9fw7q"] Nov 03 00:48:33.401870 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:48:33.401828 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 00:48:33.803542 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:48:33.803469 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 00:48:33.866175 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:48:33.866135 4939 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="46e2ee17-5a32-4025-a744-fca710d58acc" path="/var/lib/kubelet/pods/46e2ee17-5a32-4025-a744-fca710d58acc/volumes" Nov 03 00:48:35.011617 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:48:35.011584 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 00:48:35.132840 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:48:35.132804 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 00:48:38.813900 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:48:38.813867 4939 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/e2e-distinct-2-simulated-kserve-cbd4f5586-kksqt"] Nov 03 00:48:38.814435 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:48:38.814414 4939 cpu_manager.go:414] "containerMap: removing container" podUID="46e2ee17-5a32-4025-a744-fca710d58acc" containerName="manager" Nov 03 00:48:38.814513 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:48:38.814437 4939 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="46e2ee17-5a32-4025-a744-fca710d58acc" containerName="manager" Nov 03 00:48:38.814569 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:48:38.814535 4939 memory_manager.go:365] "RemoveStaleState removing state" podUID="46e2ee17-5a32-4025-a744-fca710d58acc" containerName="manager" Nov 03 00:48:38.817535 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:48:38.817513 4939 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-kksqt" Nov 03 00:48:38.823235 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:48:38.823214 4939 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"kube-root-ca.crt\"" Nov 03 00:48:38.824712 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:48:38.824688 4939 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"openshift-service-ca.crt\"" Nov 03 00:48:38.824845 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:48:38.824744 4939 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-distinct-2-simulated-kserve-self-signed-certs\"" Nov 03 00:48:38.824845 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:48:38.824781 4939 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"default-dockercfg-jhxdz\"" Nov 03 00:48:38.826555 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:48:38.826271 4939 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-2-simulated-kserve-cbd4f5586-kksqt"] Nov 03 00:48:38.836000 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:48:38.835976 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7q8s2\" (UniqueName: \"kubernetes.io/projected/08eb2c0e-2bd2-4a9e-815e-da525dbd5f32-kube-api-access-7q8s2\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-kksqt\" (UID: \"08eb2c0e-2bd2-4a9e-815e-da525dbd5f32\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-kksqt" Nov 03 00:48:38.836092 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:48:38.836015 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/08eb2c0e-2bd2-4a9e-815e-da525dbd5f32-home\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-kksqt\" (UID: \"08eb2c0e-2bd2-4a9e-815e-da525dbd5f32\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-kksqt" Nov 03 00:48:38.836169 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:48:38.836147 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/08eb2c0e-2bd2-4a9e-815e-da525dbd5f32-tls-certs\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-kksqt\" (UID: \"08eb2c0e-2bd2-4a9e-815e-da525dbd5f32\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-kksqt" Nov 03 00:48:38.836225 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:48:38.836212 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/08eb2c0e-2bd2-4a9e-815e-da525dbd5f32-tmp-dir\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-kksqt\" (UID: \"08eb2c0e-2bd2-4a9e-815e-da525dbd5f32\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-kksqt" Nov 03 00:48:38.836292 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:48:38.836276 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/08eb2c0e-2bd2-4a9e-815e-da525dbd5f32-model-cache\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-kksqt\" (UID: \"08eb2c0e-2bd2-4a9e-815e-da525dbd5f32\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-kksqt" Nov 03 00:48:38.836333 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:48:38.836325 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/08eb2c0e-2bd2-4a9e-815e-da525dbd5f32-dshm\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-kksqt\" (UID: \"08eb2c0e-2bd2-4a9e-815e-da525dbd5f32\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-kksqt" Nov 03 00:48:38.936027 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:48:38.935995 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 00:48:38.936942 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:48:38.936899 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/08eb2c0e-2bd2-4a9e-815e-da525dbd5f32-tls-certs\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-kksqt\" (UID: \"08eb2c0e-2bd2-4a9e-815e-da525dbd5f32\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-kksqt" Nov 03 00:48:38.937051 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:48:38.936973 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/08eb2c0e-2bd2-4a9e-815e-da525dbd5f32-tmp-dir\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-kksqt\" (UID: \"08eb2c0e-2bd2-4a9e-815e-da525dbd5f32\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-kksqt" Nov 03 00:48:38.937051 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:48:38.936999 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/08eb2c0e-2bd2-4a9e-815e-da525dbd5f32-model-cache\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-kksqt\" (UID: \"08eb2c0e-2bd2-4a9e-815e-da525dbd5f32\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-kksqt" Nov 03 00:48:38.937051 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:48:38.937048 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/08eb2c0e-2bd2-4a9e-815e-da525dbd5f32-dshm\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-kksqt\" (UID: \"08eb2c0e-2bd2-4a9e-815e-da525dbd5f32\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-kksqt" Nov 03 00:48:38.937300 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:48:38.937089 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7q8s2\" (UniqueName: \"kubernetes.io/projected/08eb2c0e-2bd2-4a9e-815e-da525dbd5f32-kube-api-access-7q8s2\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-kksqt\" (UID: \"08eb2c0e-2bd2-4a9e-815e-da525dbd5f32\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-kksqt" Nov 03 00:48:38.937300 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:48:38.937121 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/08eb2c0e-2bd2-4a9e-815e-da525dbd5f32-home\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-kksqt\" (UID: \"08eb2c0e-2bd2-4a9e-815e-da525dbd5f32\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-kksqt" Nov 03 00:48:38.937458 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:48:38.937394 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/08eb2c0e-2bd2-4a9e-815e-da525dbd5f32-model-cache\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-kksqt\" (UID: \"08eb2c0e-2bd2-4a9e-815e-da525dbd5f32\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-kksqt" Nov 03 00:48:38.937517 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:48:38.937461 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/08eb2c0e-2bd2-4a9e-815e-da525dbd5f32-tmp-dir\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-kksqt\" (UID: \"08eb2c0e-2bd2-4a9e-815e-da525dbd5f32\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-kksqt" Nov 03 00:48:38.937517 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:48:38.937489 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/08eb2c0e-2bd2-4a9e-815e-da525dbd5f32-home\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-kksqt\" (UID: \"08eb2c0e-2bd2-4a9e-815e-da525dbd5f32\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-kksqt" Nov 03 00:48:38.939439 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:48:38.939417 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/08eb2c0e-2bd2-4a9e-815e-da525dbd5f32-dshm\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-kksqt\" (UID: \"08eb2c0e-2bd2-4a9e-815e-da525dbd5f32\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-kksqt" Nov 03 00:48:38.939572 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:48:38.939556 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/08eb2c0e-2bd2-4a9e-815e-da525dbd5f32-tls-certs\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-kksqt\" (UID: \"08eb2c0e-2bd2-4a9e-815e-da525dbd5f32\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-kksqt" Nov 03 00:48:38.946767 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:48:38.946747 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7q8s2\" (UniqueName: \"kubernetes.io/projected/08eb2c0e-2bd2-4a9e-815e-da525dbd5f32-kube-api-access-7q8s2\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-kksqt\" (UID: \"08eb2c0e-2bd2-4a9e-815e-da525dbd5f32\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-kksqt" Nov 03 00:48:39.134058 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:48:39.134027 4939 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-kksqt" Nov 03 00:48:39.389900 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:48:39.389828 4939 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-4pw7v"] Nov 03 00:48:39.407717 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:48:39.407696 4939 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-2-simulated-kserve-cbd4f5586-kksqt"] Nov 03 00:48:39.410403 ip-10-0-140-107 kubenswrapper[4939]: W1103 00:48:39.410373 4939 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod08eb2c0e_2bd2_4a9e_815e_da525dbd5f32.slice/crio-026fa70f56767ae2d264c958cc5843c2d49459073201544b9cdb8b4139a144bf WatchSource:0}: Error finding container 026fa70f56767ae2d264c958cc5843c2d49459073201544b9cdb8b4139a144bf: Status 404 returned error can't find the container with id 026fa70f56767ae2d264c958cc5843c2d49459073201544b9cdb8b4139a144bf Nov 03 00:48:39.829385 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:48:39.829293 4939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-kksqt" event={"ID":"08eb2c0e-2bd2-4a9e-815e-da525dbd5f32","Type":"ContainerStarted","Data":"026fa70f56767ae2d264c958cc5843c2d49459073201544b9cdb8b4139a144bf"} Nov 03 00:48:41.660968 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:48:41.660937 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 00:48:41.838079 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:48:41.838048 4939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-kksqt" event={"ID":"08eb2c0e-2bd2-4a9e-815e-da525dbd5f32","Type":"ContainerStarted","Data":"ae40add442c985a8d9a88e583416ae55781f318fbb73dcee4363e34d568a0176"} Nov 03 00:48:41.857419 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:48:41.857374 4939 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-kksqt" podStartSLOduration=2.483861501 podStartE2EDuration="3.857359502s" totalImagesPullingTime="1.373498001s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:48:38 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:48:41.855403443 +0000 UTC m=+470.119738413" watchObservedRunningTime="2026-11-03 00:48:41.857359502 +0000 UTC m=+470.121694447" Nov 03 00:48:42.134607 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:48:42.134564 4939 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-kksqt" Nov 03 00:48:42.163191 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:48:42.163151 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 00:48:42.647683 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:48:42.647637 4939 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-4pw7v"] Nov 03 00:48:43.408492 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:48:43.408462 4939 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/e2e-embedding-simulated-kserve-67bf594cd6-fp9vz"] Nov 03 00:48:43.412020 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:48:43.411995 4939 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-fp9vz" Nov 03 00:48:43.420829 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:48:43.420808 4939 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-embedding-simulated-kserve-self-signed-certs\"" Nov 03 00:48:43.420963 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:48:43.420884 4939 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-embedding-simulated-kserve-67bf594cd6-fp9vz"] Nov 03 00:48:43.472381 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:48:43.472347 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 00:48:43.474577 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:48:43.474557 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2e990a12-7eac-4d1a-84f1-05fe60a93761-tmp-dir\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-fp9vz\" (UID: \"2e990a12-7eac-4d1a-84f1-05fe60a93761\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-fp9vz" Nov 03 00:48:43.474633 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:48:43.474604 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2e990a12-7eac-4d1a-84f1-05fe60a93761-model-cache\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-fp9vz\" (UID: \"2e990a12-7eac-4d1a-84f1-05fe60a93761\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-fp9vz" Nov 03 00:48:43.474702 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:48:43.474687 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2e990a12-7eac-4d1a-84f1-05fe60a93761-tls-certs\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-fp9vz\" (UID: \"2e990a12-7eac-4d1a-84f1-05fe60a93761\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-fp9vz" Nov 03 00:48:43.474736 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:48:43.474714 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f4zp4\" (UniqueName: \"kubernetes.io/projected/2e990a12-7eac-4d1a-84f1-05fe60a93761-kube-api-access-f4zp4\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-fp9vz\" (UID: \"2e990a12-7eac-4d1a-84f1-05fe60a93761\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-fp9vz" Nov 03 00:48:43.474770 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:48:43.474743 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2e990a12-7eac-4d1a-84f1-05fe60a93761-home\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-fp9vz\" (UID: \"2e990a12-7eac-4d1a-84f1-05fe60a93761\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-fp9vz" Nov 03 00:48:43.474812 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:48:43.474801 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2e990a12-7eac-4d1a-84f1-05fe60a93761-dshm\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-fp9vz\" (UID: \"2e990a12-7eac-4d1a-84f1-05fe60a93761\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-fp9vz" Nov 03 00:48:43.576053 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:48:43.576018 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2e990a12-7eac-4d1a-84f1-05fe60a93761-home\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-fp9vz\" (UID: \"2e990a12-7eac-4d1a-84f1-05fe60a93761\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-fp9vz" Nov 03 00:48:43.576200 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:48:43.576061 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2e990a12-7eac-4d1a-84f1-05fe60a93761-dshm\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-fp9vz\" (UID: \"2e990a12-7eac-4d1a-84f1-05fe60a93761\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-fp9vz" Nov 03 00:48:43.576200 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:48:43.576097 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2e990a12-7eac-4d1a-84f1-05fe60a93761-tmp-dir\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-fp9vz\" (UID: \"2e990a12-7eac-4d1a-84f1-05fe60a93761\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-fp9vz" Nov 03 00:48:43.576200 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:48:43.576118 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2e990a12-7eac-4d1a-84f1-05fe60a93761-model-cache\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-fp9vz\" (UID: \"2e990a12-7eac-4d1a-84f1-05fe60a93761\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-fp9vz" Nov 03 00:48:43.576200 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:48:43.576151 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2e990a12-7eac-4d1a-84f1-05fe60a93761-tls-certs\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-fp9vz\" (UID: \"2e990a12-7eac-4d1a-84f1-05fe60a93761\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-fp9vz" Nov 03 00:48:43.576200 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:48:43.576169 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f4zp4\" (UniqueName: \"kubernetes.io/projected/2e990a12-7eac-4d1a-84f1-05fe60a93761-kube-api-access-f4zp4\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-fp9vz\" (UID: \"2e990a12-7eac-4d1a-84f1-05fe60a93761\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-fp9vz" Nov 03 00:48:43.576525 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:48:43.576498 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2e990a12-7eac-4d1a-84f1-05fe60a93761-home\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-fp9vz\" (UID: \"2e990a12-7eac-4d1a-84f1-05fe60a93761\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-fp9vz" Nov 03 00:48:43.576593 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:48:43.576534 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2e990a12-7eac-4d1a-84f1-05fe60a93761-tmp-dir\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-fp9vz\" (UID: \"2e990a12-7eac-4d1a-84f1-05fe60a93761\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-fp9vz" Nov 03 00:48:43.576593 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:48:43.576559 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2e990a12-7eac-4d1a-84f1-05fe60a93761-model-cache\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-fp9vz\" (UID: \"2e990a12-7eac-4d1a-84f1-05fe60a93761\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-fp9vz" Nov 03 00:48:43.578351 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:48:43.578327 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2e990a12-7eac-4d1a-84f1-05fe60a93761-dshm\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-fp9vz\" (UID: \"2e990a12-7eac-4d1a-84f1-05fe60a93761\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-fp9vz" Nov 03 00:48:43.578636 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:48:43.578614 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2e990a12-7eac-4d1a-84f1-05fe60a93761-tls-certs\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-fp9vz\" (UID: \"2e990a12-7eac-4d1a-84f1-05fe60a93761\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-fp9vz" Nov 03 00:48:43.586715 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:48:43.586691 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-f4zp4\" (UniqueName: \"kubernetes.io/projected/2e990a12-7eac-4d1a-84f1-05fe60a93761-kube-api-access-f4zp4\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-fp9vz\" (UID: \"2e990a12-7eac-4d1a-84f1-05fe60a93761\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-fp9vz" Nov 03 00:48:43.726154 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:48:43.726087 4939 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-fp9vz" Nov 03 00:48:43.998932 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:48:43.998894 4939 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-embedding-simulated-kserve-67bf594cd6-fp9vz"] Nov 03 00:48:44.001149 ip-10-0-140-107 kubenswrapper[4939]: W1103 00:48:44.001113 4939 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2e990a12_7eac_4d1a_84f1_05fe60a93761.slice/crio-03e7291fb0e5c7f86f4721f51b66bcf85cee56c4647985460b818fee4a3f8f3b WatchSource:0}: Error finding container 03e7291fb0e5c7f86f4721f51b66bcf85cee56c4647985460b818fee4a3f8f3b: Status 404 returned error can't find the container with id 03e7291fb0e5c7f86f4721f51b66bcf85cee56c4647985460b818fee4a3f8f3b Nov 03 00:48:44.852536 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:48:44.852494 4939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-fp9vz" event={"ID":"2e990a12-7eac-4d1a-84f1-05fe60a93761","Type":"ContainerStarted","Data":"f018323b18d6dfe755024dbe7d9cc03dd53a1477a1de862fe1c6d552022ff2d1"} Nov 03 00:48:44.853000 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:48:44.852552 4939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-fp9vz" event={"ID":"2e990a12-7eac-4d1a-84f1-05fe60a93761","Type":"ContainerStarted","Data":"03e7291fb0e5c7f86f4721f51b66bcf85cee56c4647985460b818fee4a3f8f3b"} Nov 03 00:48:44.870486 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:48:44.870438 4939 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-fp9vz" podStartSLOduration=1.5920186680000001 podStartE2EDuration="1.870424734s" totalImagesPullingTime="278.406066ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:48:43 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:48:44.869277254 +0000 UTC m=+473.133612202" watchObservedRunningTime="2026-11-03 00:48:44.870424734 +0000 UTC m=+473.134759678" Nov 03 00:48:44.893315 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:48:44.893289 4939 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-4pw7v"] Nov 03 00:48:45.726572 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:48:45.726534 4939 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-fp9vz" Nov 03 00:48:48.028709 ip-10-0-140-107 kubenswrapper[4939]: E1103 00:48:48.028675 4939 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-sthjh" containerName="extractor" Nov 03 00:48:48.459148 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:48:48.459116 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 00:48:49.134242 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:48:49.134210 4939 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-kksqt" Nov 03 00:48:49.146636 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:48:49.146616 4939 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-kksqt" Nov 03 00:48:49.207506 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:48:49.207478 4939 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-4pw7v"] Nov 03 00:48:49.239374 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:48:49.239347 4939 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8dx487"] Nov 03 00:48:49.247107 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:48:49.247087 4939 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8dx487" Nov 03 00:48:49.254831 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:48:49.254807 4939 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8dx487"] Nov 03 00:48:49.255190 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:48:49.255159 4939 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-unab60ef4d3a239b5143b412cab04acac3-kserve-self-signed-certs\"" Nov 03 00:48:49.328255 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:48:49.328224 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/33fe945b-76e3-4201-970d-235ff8f719a3-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8dx487\" (UID: \"33fe945b-76e3-4201-970d-235ff8f719a3\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8dx487" Nov 03 00:48:49.328255 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:48:49.328256 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/33fe945b-76e3-4201-970d-235ff8f719a3-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8dx487\" (UID: \"33fe945b-76e3-4201-970d-235ff8f719a3\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8dx487" Nov 03 00:48:49.328486 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:48:49.328292 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s8mxp\" (UniqueName: \"kubernetes.io/projected/33fe945b-76e3-4201-970d-235ff8f719a3-kube-api-access-s8mxp\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8dx487\" (UID: \"33fe945b-76e3-4201-970d-235ff8f719a3\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8dx487" Nov 03 00:48:49.328486 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:48:49.328343 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/33fe945b-76e3-4201-970d-235ff8f719a3-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8dx487\" (UID: \"33fe945b-76e3-4201-970d-235ff8f719a3\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8dx487" Nov 03 00:48:49.328486 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:48:49.328405 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/33fe945b-76e3-4201-970d-235ff8f719a3-tmp-dir\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8dx487\" (UID: \"33fe945b-76e3-4201-970d-235ff8f719a3\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8dx487" Nov 03 00:48:49.328598 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:48:49.328503 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/33fe945b-76e3-4201-970d-235ff8f719a3-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8dx487\" (UID: \"33fe945b-76e3-4201-970d-235ff8f719a3\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8dx487" Nov 03 00:48:49.334682 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:48:49.334656 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 00:48:49.429872 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:48:49.429786 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s8mxp\" (UniqueName: \"kubernetes.io/projected/33fe945b-76e3-4201-970d-235ff8f719a3-kube-api-access-s8mxp\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8dx487\" (UID: \"33fe945b-76e3-4201-970d-235ff8f719a3\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8dx487" Nov 03 00:48:49.429872 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:48:49.429833 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/33fe945b-76e3-4201-970d-235ff8f719a3-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8dx487\" (UID: \"33fe945b-76e3-4201-970d-235ff8f719a3\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8dx487" Nov 03 00:48:49.429872 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:48:49.429862 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/33fe945b-76e3-4201-970d-235ff8f719a3-tmp-dir\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8dx487\" (UID: \"33fe945b-76e3-4201-970d-235ff8f719a3\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8dx487" Nov 03 00:48:49.430235 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:48:49.429993 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/33fe945b-76e3-4201-970d-235ff8f719a3-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8dx487\" (UID: \"33fe945b-76e3-4201-970d-235ff8f719a3\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8dx487" Nov 03 00:48:49.430235 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:48:49.430106 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/33fe945b-76e3-4201-970d-235ff8f719a3-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8dx487\" (UID: \"33fe945b-76e3-4201-970d-235ff8f719a3\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8dx487" Nov 03 00:48:49.430235 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:48:49.430130 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/33fe945b-76e3-4201-970d-235ff8f719a3-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8dx487\" (UID: \"33fe945b-76e3-4201-970d-235ff8f719a3\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8dx487" Nov 03 00:48:49.430673 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:48:49.430638 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/33fe945b-76e3-4201-970d-235ff8f719a3-tmp-dir\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8dx487\" (UID: \"33fe945b-76e3-4201-970d-235ff8f719a3\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8dx487" Nov 03 00:48:49.431080 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:48:49.431048 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/33fe945b-76e3-4201-970d-235ff8f719a3-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8dx487\" (UID: \"33fe945b-76e3-4201-970d-235ff8f719a3\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8dx487" Nov 03 00:48:49.431474 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:48:49.431264 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/33fe945b-76e3-4201-970d-235ff8f719a3-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8dx487\" (UID: \"33fe945b-76e3-4201-970d-235ff8f719a3\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8dx487" Nov 03 00:48:49.433129 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:48:49.433104 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/33fe945b-76e3-4201-970d-235ff8f719a3-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8dx487\" (UID: \"33fe945b-76e3-4201-970d-235ff8f719a3\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8dx487" Nov 03 00:48:49.433262 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:48:49.433240 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/33fe945b-76e3-4201-970d-235ff8f719a3-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8dx487\" (UID: \"33fe945b-76e3-4201-970d-235ff8f719a3\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8dx487" Nov 03 00:48:49.438580 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:48:49.438554 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-s8mxp\" (UniqueName: \"kubernetes.io/projected/33fe945b-76e3-4201-970d-235ff8f719a3-kube-api-access-s8mxp\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8dx487\" (UID: \"33fe945b-76e3-4201-970d-235ff8f719a3\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8dx487" Nov 03 00:48:49.563490 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:48:49.563461 4939 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8dx487" Nov 03 00:48:49.841483 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:48:49.841458 4939 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8dx487"] Nov 03 00:48:49.844259 ip-10-0-140-107 kubenswrapper[4939]: W1103 00:48:49.844225 4939 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod33fe945b_76e3_4201_970d_235ff8f719a3.slice/crio-40b082a4a4e079a12e005dcfd6c4c83b70a64623c1cbf8b58cb2a8658de06914 WatchSource:0}: Error finding container 40b082a4a4e079a12e005dcfd6c4c83b70a64623c1cbf8b58cb2a8658de06914: Status 404 returned error can't find the container with id 40b082a4a4e079a12e005dcfd6c4c83b70a64623c1cbf8b58cb2a8658de06914 Nov 03 00:48:49.874718 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:48:49.874687 4939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8dx487" event={"ID":"33fe945b-76e3-4201-970d-235ff8f719a3","Type":"ContainerStarted","Data":"40b082a4a4e079a12e005dcfd6c4c83b70a64623c1cbf8b58cb2a8658de06914"} Nov 03 00:48:49.886036 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:48:49.886009 4939 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-kksqt" Nov 03 00:48:50.881774 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:48:50.881741 4939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8dx487" event={"ID":"33fe945b-76e3-4201-970d-235ff8f719a3","Type":"ContainerStarted","Data":"629187cdc8430dd065495ca15f9e7fb360aa061a26d56f606f2b1016ce32c363"} Nov 03 00:48:50.904101 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:48:50.904045 4939 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8dx487" podStartSLOduration=1.595369051 podStartE2EDuration="1.904033134s" totalImagesPullingTime="308.664083ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:48:49 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:48:50.902834559 +0000 UTC m=+479.167169504" watchObservedRunningTime="2026-11-03 00:48:50.904033134 +0000 UTC m=+479.168368079" Nov 03 00:48:51.322954 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:48:51.322813 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 00:48:51.564060 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:48:51.564031 4939 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8dx487" Nov 03 00:48:52.211443 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:48:52.211363 4939 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-4pw7v"] Nov 03 00:48:53.118507 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:48:53.118475 4939 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/facebook-opt-125m-simulated-kserve-84b4ff656c-xggrt"] Nov 03 00:48:53.123677 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:48:53.123653 4939 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-xggrt" Nov 03 00:48:53.136404 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:48:53.136384 4939 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"facebook-opt-125m-simulated-kserve-self-signed-certs\"" Nov 03 00:48:53.138420 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:48:53.138401 4939 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/facebook-opt-125m-simulated-kserve-84b4ff656c-xggrt"] Nov 03 00:48:53.161592 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:48:53.161563 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9f0874ba-e3aa-4036-ad25-b2b97bf5ddc4-model-cache\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-xggrt\" (UID: \"9f0874ba-e3aa-4036-ad25-b2b97bf5ddc4\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-xggrt" Nov 03 00:48:53.161712 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:48:53.161608 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dghzl\" (UniqueName: \"kubernetes.io/projected/9f0874ba-e3aa-4036-ad25-b2b97bf5ddc4-kube-api-access-dghzl\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-xggrt\" (UID: \"9f0874ba-e3aa-4036-ad25-b2b97bf5ddc4\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-xggrt" Nov 03 00:48:53.161778 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:48:53.161699 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9f0874ba-e3aa-4036-ad25-b2b97bf5ddc4-home\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-xggrt\" (UID: \"9f0874ba-e3aa-4036-ad25-b2b97bf5ddc4\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-xggrt" Nov 03 00:48:53.161778 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:48:53.161751 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9f0874ba-e3aa-4036-ad25-b2b97bf5ddc4-tmp-dir\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-xggrt\" (UID: \"9f0874ba-e3aa-4036-ad25-b2b97bf5ddc4\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-xggrt" Nov 03 00:48:53.161932 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:48:53.161890 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9f0874ba-e3aa-4036-ad25-b2b97bf5ddc4-dshm\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-xggrt\" (UID: \"9f0874ba-e3aa-4036-ad25-b2b97bf5ddc4\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-xggrt" Nov 03 00:48:53.161981 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:48:53.161932 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9f0874ba-e3aa-4036-ad25-b2b97bf5ddc4-tls-certs\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-xggrt\" (UID: \"9f0874ba-e3aa-4036-ad25-b2b97bf5ddc4\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-xggrt" Nov 03 00:48:53.200612 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:48:53.200582 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 00:48:53.265138 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:48:53.265096 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9f0874ba-e3aa-4036-ad25-b2b97bf5ddc4-dshm\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-xggrt\" (UID: \"9f0874ba-e3aa-4036-ad25-b2b97bf5ddc4\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-xggrt" Nov 03 00:48:53.265590 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:48:53.265173 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9f0874ba-e3aa-4036-ad25-b2b97bf5ddc4-tls-certs\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-xggrt\" (UID: \"9f0874ba-e3aa-4036-ad25-b2b97bf5ddc4\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-xggrt" Nov 03 00:48:53.266073 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:48:53.266028 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9f0874ba-e3aa-4036-ad25-b2b97bf5ddc4-model-cache\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-xggrt\" (UID: \"9f0874ba-e3aa-4036-ad25-b2b97bf5ddc4\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-xggrt" Nov 03 00:48:53.266384 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:48:53.266283 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dghzl\" (UniqueName: \"kubernetes.io/projected/9f0874ba-e3aa-4036-ad25-b2b97bf5ddc4-kube-api-access-dghzl\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-xggrt\" (UID: \"9f0874ba-e3aa-4036-ad25-b2b97bf5ddc4\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-xggrt" Nov 03 00:48:53.266541 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:48:53.266482 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9f0874ba-e3aa-4036-ad25-b2b97bf5ddc4-home\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-xggrt\" (UID: \"9f0874ba-e3aa-4036-ad25-b2b97bf5ddc4\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-xggrt" Nov 03 00:48:53.266659 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:48:53.266636 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9f0874ba-e3aa-4036-ad25-b2b97bf5ddc4-tmp-dir\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-xggrt\" (UID: \"9f0874ba-e3aa-4036-ad25-b2b97bf5ddc4\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-xggrt" Nov 03 00:48:53.266961 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:48:53.266940 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9f0874ba-e3aa-4036-ad25-b2b97bf5ddc4-model-cache\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-xggrt\" (UID: \"9f0874ba-e3aa-4036-ad25-b2b97bf5ddc4\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-xggrt" Nov 03 00:48:53.267051 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:48:53.266983 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9f0874ba-e3aa-4036-ad25-b2b97bf5ddc4-tmp-dir\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-xggrt\" (UID: \"9f0874ba-e3aa-4036-ad25-b2b97bf5ddc4\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-xggrt" Nov 03 00:48:53.267051 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:48:53.267010 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9f0874ba-e3aa-4036-ad25-b2b97bf5ddc4-home\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-xggrt\" (UID: \"9f0874ba-e3aa-4036-ad25-b2b97bf5ddc4\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-xggrt" Nov 03 00:48:53.269553 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:48:53.269525 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9f0874ba-e3aa-4036-ad25-b2b97bf5ddc4-dshm\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-xggrt\" (UID: \"9f0874ba-e3aa-4036-ad25-b2b97bf5ddc4\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-xggrt" Nov 03 00:48:53.269726 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:48:53.269708 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9f0874ba-e3aa-4036-ad25-b2b97bf5ddc4-tls-certs\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-xggrt\" (UID: \"9f0874ba-e3aa-4036-ad25-b2b97bf5ddc4\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-xggrt" Nov 03 00:48:53.275394 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:48:53.275371 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dghzl\" (UniqueName: \"kubernetes.io/projected/9f0874ba-e3aa-4036-ad25-b2b97bf5ddc4-kube-api-access-dghzl\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-xggrt\" (UID: \"9f0874ba-e3aa-4036-ad25-b2b97bf5ddc4\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-xggrt" Nov 03 00:48:53.443106 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:48:53.443027 4939 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-xggrt" Nov 03 00:48:53.715480 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:48:53.715452 4939 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/facebook-opt-125m-simulated-kserve-84b4ff656c-xggrt"] Nov 03 00:48:53.717859 ip-10-0-140-107 kubenswrapper[4939]: W1103 00:48:53.717823 4939 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9f0874ba_e3aa_4036_ad25_b2b97bf5ddc4.slice/crio-1401dc91b5da47fc69aca7d125b92be7933315206d37b9f9a8871505243e61a0 WatchSource:0}: Error finding container 1401dc91b5da47fc69aca7d125b92be7933315206d37b9f9a8871505243e61a0: Status 404 returned error can't find the container with id 1401dc91b5da47fc69aca7d125b92be7933315206d37b9f9a8871505243e61a0 Nov 03 00:48:53.727169 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:48:53.727145 4939 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-fp9vz" Nov 03 00:48:53.739617 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:48:53.739595 4939 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-fp9vz" Nov 03 00:48:53.895318 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:48:53.895289 4939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-xggrt" event={"ID":"9f0874ba-e3aa-4036-ad25-b2b97bf5ddc4","Type":"ContainerStarted","Data":"1401dc91b5da47fc69aca7d125b92be7933315206d37b9f9a8871505243e61a0"} Nov 03 00:48:53.906448 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:48:53.906425 4939 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-fp9vz" Nov 03 00:48:54.002894 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:48:54.002794 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 00:48:54.901336 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:48:54.901299 4939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-xggrt" event={"ID":"9f0874ba-e3aa-4036-ad25-b2b97bf5ddc4","Type":"ContainerStarted","Data":"e7a3aaf5cafc88fb6da6ad721d60d377f84c8b6fe52cc3132fa5bcadc6755029"} Nov 03 00:48:54.926401 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:48:54.926354 4939 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-xggrt" podStartSLOduration=1.642950705 podStartE2EDuration="1.926339674s" totalImagesPullingTime="283.388969ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:48:53 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:48:54.925016921 +0000 UTC m=+483.189351889" watchObservedRunningTime="2026-11-03 00:48:54.926339674 +0000 UTC m=+483.190674616" Nov 03 00:48:55.444065 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:48:55.444032 4939 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-xggrt" Nov 03 00:48:56.683401 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:48:56.683369 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 00:48:56.797265 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:48:56.797228 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 00:48:58.000537 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:48:58.000505 4939 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-4pw7v"] Nov 03 00:48:58.945712 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:48:58.941889 4939 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-9jvbb"] Nov 03 00:48:58.947435 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:48:58.947412 4939 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-9jvbb" Nov 03 00:48:58.954817 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:48:58.954792 4939 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"maas-default-gateway-openshift-default-dockercfg-pjnsk\"" Nov 03 00:48:58.955303 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:48:58.954933 4939 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-9jvbb"] Nov 03 00:48:59.015990 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:48:59.015961 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/512d26b5-8ad9-4cca-8687-cd06d31d8369-workload-certs\") pod \"maas-default-gateway-openshift-default-596dc9545f-9jvbb\" (UID: \"512d26b5-8ad9-4cca-8687-cd06d31d8369\") " pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-9jvbb" Nov 03 00:48:59.016356 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:48:59.016002 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/512d26b5-8ad9-4cca-8687-cd06d31d8369-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-596dc9545f-9jvbb\" (UID: \"512d26b5-8ad9-4cca-8687-cd06d31d8369\") " pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-9jvbb" Nov 03 00:48:59.016356 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:48:59.016027 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/512d26b5-8ad9-4cca-8687-cd06d31d8369-istio-token\") pod \"maas-default-gateway-openshift-default-596dc9545f-9jvbb\" (UID: \"512d26b5-8ad9-4cca-8687-cd06d31d8369\") " pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-9jvbb" Nov 03 00:48:59.016356 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:48:59.016152 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/512d26b5-8ad9-4cca-8687-cd06d31d8369-istio-data\") pod \"maas-default-gateway-openshift-default-596dc9545f-9jvbb\" (UID: \"512d26b5-8ad9-4cca-8687-cd06d31d8369\") " pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-9jvbb" Nov 03 00:48:59.016356 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:48:59.016185 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/512d26b5-8ad9-4cca-8687-cd06d31d8369-workload-socket\") pod \"maas-default-gateway-openshift-default-596dc9545f-9jvbb\" (UID: \"512d26b5-8ad9-4cca-8687-cd06d31d8369\") " pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-9jvbb" Nov 03 00:48:59.016356 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:48:59.016284 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/512d26b5-8ad9-4cca-8687-cd06d31d8369-istio-envoy\") pod \"maas-default-gateway-openshift-default-596dc9545f-9jvbb\" (UID: \"512d26b5-8ad9-4cca-8687-cd06d31d8369\") " pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-9jvbb" Nov 03 00:48:59.016356 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:48:59.016347 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/512d26b5-8ad9-4cca-8687-cd06d31d8369-istio-podinfo\") pod \"maas-default-gateway-openshift-default-596dc9545f-9jvbb\" (UID: \"512d26b5-8ad9-4cca-8687-cd06d31d8369\") " pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-9jvbb" Nov 03 00:48:59.016561 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:48:59.016379 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tnk5h\" (UniqueName: \"kubernetes.io/projected/512d26b5-8ad9-4cca-8687-cd06d31d8369-kube-api-access-tnk5h\") pod \"maas-default-gateway-openshift-default-596dc9545f-9jvbb\" (UID: \"512d26b5-8ad9-4cca-8687-cd06d31d8369\") " pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-9jvbb" Nov 03 00:48:59.016561 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:48:59.016399 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/512d26b5-8ad9-4cca-8687-cd06d31d8369-credential-socket\") pod \"maas-default-gateway-openshift-default-596dc9545f-9jvbb\" (UID: \"512d26b5-8ad9-4cca-8687-cd06d31d8369\") " pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-9jvbb" Nov 03 00:48:59.117700 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:48:59.117676 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/512d26b5-8ad9-4cca-8687-cd06d31d8369-workload-certs\") pod \"maas-default-gateway-openshift-default-596dc9545f-9jvbb\" (UID: \"512d26b5-8ad9-4cca-8687-cd06d31d8369\") " pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-9jvbb" Nov 03 00:48:59.117845 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:48:59.117708 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/512d26b5-8ad9-4cca-8687-cd06d31d8369-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-596dc9545f-9jvbb\" (UID: \"512d26b5-8ad9-4cca-8687-cd06d31d8369\") " pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-9jvbb" Nov 03 00:48:59.117845 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:48:59.117732 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/512d26b5-8ad9-4cca-8687-cd06d31d8369-istio-token\") pod \"maas-default-gateway-openshift-default-596dc9545f-9jvbb\" (UID: \"512d26b5-8ad9-4cca-8687-cd06d31d8369\") " pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-9jvbb" Nov 03 00:48:59.117845 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:48:59.117800 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/512d26b5-8ad9-4cca-8687-cd06d31d8369-istio-data\") pod \"maas-default-gateway-openshift-default-596dc9545f-9jvbb\" (UID: \"512d26b5-8ad9-4cca-8687-cd06d31d8369\") " pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-9jvbb" Nov 03 00:48:59.117845 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:48:59.117840 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/512d26b5-8ad9-4cca-8687-cd06d31d8369-workload-socket\") pod \"maas-default-gateway-openshift-default-596dc9545f-9jvbb\" (UID: \"512d26b5-8ad9-4cca-8687-cd06d31d8369\") " pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-9jvbb" Nov 03 00:48:59.118064 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:48:59.117887 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/512d26b5-8ad9-4cca-8687-cd06d31d8369-istio-envoy\") pod \"maas-default-gateway-openshift-default-596dc9545f-9jvbb\" (UID: \"512d26b5-8ad9-4cca-8687-cd06d31d8369\") " pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-9jvbb" Nov 03 00:48:59.118064 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:48:59.117942 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/512d26b5-8ad9-4cca-8687-cd06d31d8369-istio-podinfo\") pod \"maas-default-gateway-openshift-default-596dc9545f-9jvbb\" (UID: \"512d26b5-8ad9-4cca-8687-cd06d31d8369\") " pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-9jvbb" Nov 03 00:48:59.118064 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:48:59.117966 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tnk5h\" (UniqueName: \"kubernetes.io/projected/512d26b5-8ad9-4cca-8687-cd06d31d8369-kube-api-access-tnk5h\") pod \"maas-default-gateway-openshift-default-596dc9545f-9jvbb\" (UID: \"512d26b5-8ad9-4cca-8687-cd06d31d8369\") " pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-9jvbb" Nov 03 00:48:59.118064 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:48:59.117994 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/512d26b5-8ad9-4cca-8687-cd06d31d8369-credential-socket\") pod \"maas-default-gateway-openshift-default-596dc9545f-9jvbb\" (UID: \"512d26b5-8ad9-4cca-8687-cd06d31d8369\") " pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-9jvbb" Nov 03 00:48:59.118267 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:48:59.118143 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/512d26b5-8ad9-4cca-8687-cd06d31d8369-workload-certs\") pod \"maas-default-gateway-openshift-default-596dc9545f-9jvbb\" (UID: \"512d26b5-8ad9-4cca-8687-cd06d31d8369\") " pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-9jvbb" Nov 03 00:48:59.118267 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:48:59.118210 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/512d26b5-8ad9-4cca-8687-cd06d31d8369-istio-data\") pod \"maas-default-gateway-openshift-default-596dc9545f-9jvbb\" (UID: \"512d26b5-8ad9-4cca-8687-cd06d31d8369\") " pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-9jvbb" Nov 03 00:48:59.118555 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:48:59.118524 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/512d26b5-8ad9-4cca-8687-cd06d31d8369-credential-socket\") pod \"maas-default-gateway-openshift-default-596dc9545f-9jvbb\" (UID: \"512d26b5-8ad9-4cca-8687-cd06d31d8369\") " pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-9jvbb" Nov 03 00:48:59.118679 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:48:59.118564 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/512d26b5-8ad9-4cca-8687-cd06d31d8369-workload-socket\") pod \"maas-default-gateway-openshift-default-596dc9545f-9jvbb\" (UID: \"512d26b5-8ad9-4cca-8687-cd06d31d8369\") " pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-9jvbb" Nov 03 00:48:59.118751 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:48:59.118694 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/512d26b5-8ad9-4cca-8687-cd06d31d8369-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-596dc9545f-9jvbb\" (UID: \"512d26b5-8ad9-4cca-8687-cd06d31d8369\") " pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-9jvbb" Nov 03 00:48:59.120322 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:48:59.120297 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/512d26b5-8ad9-4cca-8687-cd06d31d8369-istio-envoy\") pod \"maas-default-gateway-openshift-default-596dc9545f-9jvbb\" (UID: \"512d26b5-8ad9-4cca-8687-cd06d31d8369\") " pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-9jvbb" Nov 03 00:48:59.120518 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:48:59.120501 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/512d26b5-8ad9-4cca-8687-cd06d31d8369-istio-podinfo\") pod \"maas-default-gateway-openshift-default-596dc9545f-9jvbb\" (UID: \"512d26b5-8ad9-4cca-8687-cd06d31d8369\") " pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-9jvbb" Nov 03 00:48:59.126103 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:48:59.126077 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/512d26b5-8ad9-4cca-8687-cd06d31d8369-istio-token\") pod \"maas-default-gateway-openshift-default-596dc9545f-9jvbb\" (UID: \"512d26b5-8ad9-4cca-8687-cd06d31d8369\") " pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-9jvbb" Nov 03 00:48:59.126186 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:48:59.126153 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tnk5h\" (UniqueName: \"kubernetes.io/projected/512d26b5-8ad9-4cca-8687-cd06d31d8369-kube-api-access-tnk5h\") pod \"maas-default-gateway-openshift-default-596dc9545f-9jvbb\" (UID: \"512d26b5-8ad9-4cca-8687-cd06d31d8369\") " pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-9jvbb" Nov 03 00:48:59.266045 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:48:59.265972 4939 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-9jvbb" Nov 03 00:48:59.539761 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:48:59.539727 4939 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-9jvbb"] Nov 03 00:48:59.541170 ip-10-0-140-107 kubenswrapper[4939]: W1103 00:48:59.541143 4939 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod512d26b5_8ad9_4cca_8687_cd06d31d8369.slice/crio-b685d4a8d687327af043a0588832e5c7a92ee0594191271836f6f640cecf670e WatchSource:0}: Error finding container b685d4a8d687327af043a0588832e5c7a92ee0594191271836f6f640cecf670e: Status 404 returned error can't find the container with id b685d4a8d687327af043a0588832e5c7a92ee0594191271836f6f640cecf670e Nov 03 00:48:59.563952 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:48:59.563928 4939 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8dx487" Nov 03 00:48:59.576231 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:48:59.576210 4939 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8dx487" Nov 03 00:48:59.924170 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:48:59.924138 4939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-9jvbb" event={"ID":"512d26b5-8ad9-4cca-8687-cd06d31d8369","Type":"ContainerStarted","Data":"b685d4a8d687327af043a0588832e5c7a92ee0594191271836f6f640cecf670e"} Nov 03 00:48:59.935536 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:48:59.935509 4939 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8dx487" Nov 03 00:49:03.119506 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:49:03.119469 4939 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 00:49:03.119793 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:49:03.119540 4939 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 00:49:03.119793 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:49:03.119598 4939 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 00:49:03.444273 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:49:03.444197 4939 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-xggrt" Nov 03 00:49:03.456424 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:49:03.456403 4939 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-xggrt" Nov 03 00:49:03.944178 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:49:03.944139 4939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-9jvbb" event={"ID":"512d26b5-8ad9-4cca-8687-cd06d31d8369","Type":"ContainerStarted","Data":"58f0c7581af96da3526eb5eb790d9852b5743d5142588ad248ceb4dbf797a08b"} Nov 03 00:49:03.966931 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:49:03.966587 4939 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-xggrt" Nov 03 00:49:03.982382 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:49:03.982314 4939 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-9jvbb" podStartSLOduration=2.406644548 podStartE2EDuration="5.982292059s" totalImagesPullingTime="3.575647511s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:48:58 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:49:03.981289481 +0000 UTC m=+492.245624428" watchObservedRunningTime="2026-11-03 00:49:03.982292059 +0000 UTC m=+492.246627005" Nov 03 00:49:04.271252 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:49:04.271162 4939 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-9jvbb" Nov 03 00:49:04.272994 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:49:04.272962 4939 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-596dc9545f-9jvbb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.41:15021/healthz/ready\": dial tcp 10.132.0.41:15021: connect: connection refused" start-of-body= Nov 03 00:49:04.273114 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:49:04.273016 4939 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-9jvbb" podUID="512d26b5-8ad9-4cca-8687-cd06d31d8369" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.41:15021/healthz/ready\": dial tcp 10.132.0.41:15021: connect: connection refused" Nov 03 00:49:05.271312 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:49:05.271274 4939 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-596dc9545f-9jvbb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.41:15021/healthz/ready\": dial tcp 10.132.0.41:15021: connect: connection refused" start-of-body= Nov 03 00:49:05.271818 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:49:05.271342 4939 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-9jvbb" podUID="512d26b5-8ad9-4cca-8687-cd06d31d8369" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.41:15021/healthz/ready\": dial tcp 10.132.0.41:15021: connect: connection refused" Nov 03 00:49:06.271330 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:49:06.271291 4939 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-596dc9545f-9jvbb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.41:15021/healthz/ready\": dial tcp 10.132.0.41:15021: connect: connection refused" start-of-body= Nov 03 00:49:06.271709 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:49:06.271349 4939 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-9jvbb" podUID="512d26b5-8ad9-4cca-8687-cd06d31d8369" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.41:15021/healthz/ready\": dial tcp 10.132.0.41:15021: connect: connection refused" Nov 03 00:49:07.267356 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:49:07.267317 4939 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-596dc9545f-9jvbb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.41:15021/healthz/ready\": dial tcp 10.132.0.41:15021: connect: connection refused" start-of-body= Nov 03 00:49:07.267651 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:49:07.267622 4939 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-9jvbb" podUID="512d26b5-8ad9-4cca-8687-cd06d31d8369" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.41:15021/healthz/ready\": dial tcp 10.132.0.41:15021: connect: connection refused" Nov 03 00:49:08.271370 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:49:08.271324 4939 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-596dc9545f-9jvbb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.41:15021/healthz/ready\": dial tcp 10.132.0.41:15021: connect: connection refused" start-of-body= Nov 03 00:49:08.271947 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:49:08.271397 4939 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-9jvbb" podUID="512d26b5-8ad9-4cca-8687-cd06d31d8369" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.41:15021/healthz/ready\": dial tcp 10.132.0.41:15021: connect: connection refused" Nov 03 00:49:09.266643 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:49:09.266604 4939 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-596dc9545f-9jvbb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.41:15021/healthz/ready\": dial tcp 10.132.0.41:15021: connect: connection refused" start-of-body= Nov 03 00:49:09.266823 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:49:09.266665 4939 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-9jvbb" podUID="512d26b5-8ad9-4cca-8687-cd06d31d8369" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.41:15021/healthz/ready\": dial tcp 10.132.0.41:15021: connect: connection refused" Nov 03 00:49:10.222500 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:49:10.222463 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 00:49:10.266453 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:49:10.266421 4939 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-596dc9545f-9jvbb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.41:15021/healthz/ready\": dial tcp 10.132.0.41:15021: connect: connection refused" start-of-body= Nov 03 00:49:10.266616 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:49:10.266482 4939 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-9jvbb" podUID="512d26b5-8ad9-4cca-8687-cd06d31d8369" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.41:15021/healthz/ready\": dial tcp 10.132.0.41:15021: connect: connection refused" Nov 03 00:49:10.440542 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:49:10.440469 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 00:49:11.383979 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:49:11.383946 4939 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-9jvbb" Nov 03 00:49:11.384869 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:49:11.384843 4939 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-9jvbb" Nov 03 00:49:11.385577 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:49:11.385558 4939 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-9jvbb" Nov 03 00:49:11.721322 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:49:11.721237 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 00:49:11.886231 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:49:11.886195 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 00:49:13.964724 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:49:13.964691 4939 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-pcw82"] Nov 03 00:49:13.971530 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:49:13.971510 4939 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-pcw82" Nov 03 00:49:13.982747 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:49:13.982719 4939 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-pcw82"] Nov 03 00:49:14.065828 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:49:14.065791 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/48186e38-73f5-47c6-a90a-d82afeac4a52-credential-socket\") pod \"maas-default-gateway-openshift-default-596dc9545f-pcw82\" (UID: \"48186e38-73f5-47c6-a90a-d82afeac4a52\") " pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-pcw82" Nov 03 00:49:14.065828 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:49:14.065831 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/48186e38-73f5-47c6-a90a-d82afeac4a52-workload-certs\") pod \"maas-default-gateway-openshift-default-596dc9545f-pcw82\" (UID: \"48186e38-73f5-47c6-a90a-d82afeac4a52\") " pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-pcw82" Nov 03 00:49:14.066052 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:49:14.065859 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/48186e38-73f5-47c6-a90a-d82afeac4a52-istio-envoy\") pod \"maas-default-gateway-openshift-default-596dc9545f-pcw82\" (UID: \"48186e38-73f5-47c6-a90a-d82afeac4a52\") " pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-pcw82" Nov 03 00:49:14.066052 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:49:14.066004 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/48186e38-73f5-47c6-a90a-d82afeac4a52-workload-socket\") pod \"maas-default-gateway-openshift-default-596dc9545f-pcw82\" (UID: \"48186e38-73f5-47c6-a90a-d82afeac4a52\") " pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-pcw82" Nov 03 00:49:14.066052 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:49:14.066039 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lwv87\" (UniqueName: \"kubernetes.io/projected/48186e38-73f5-47c6-a90a-d82afeac4a52-kube-api-access-lwv87\") pod \"maas-default-gateway-openshift-default-596dc9545f-pcw82\" (UID: \"48186e38-73f5-47c6-a90a-d82afeac4a52\") " pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-pcw82" Nov 03 00:49:14.066150 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:49:14.066079 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/48186e38-73f5-47c6-a90a-d82afeac4a52-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-596dc9545f-pcw82\" (UID: \"48186e38-73f5-47c6-a90a-d82afeac4a52\") " pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-pcw82" Nov 03 00:49:14.066150 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:49:14.066098 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/48186e38-73f5-47c6-a90a-d82afeac4a52-istio-token\") pod \"maas-default-gateway-openshift-default-596dc9545f-pcw82\" (UID: \"48186e38-73f5-47c6-a90a-d82afeac4a52\") " pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-pcw82" Nov 03 00:49:14.066226 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:49:14.066204 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/48186e38-73f5-47c6-a90a-d82afeac4a52-istio-data\") pod \"maas-default-gateway-openshift-default-596dc9545f-pcw82\" (UID: \"48186e38-73f5-47c6-a90a-d82afeac4a52\") " pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-pcw82" Nov 03 00:49:14.066272 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:49:14.066259 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/48186e38-73f5-47c6-a90a-d82afeac4a52-istio-podinfo\") pod \"maas-default-gateway-openshift-default-596dc9545f-pcw82\" (UID: \"48186e38-73f5-47c6-a90a-d82afeac4a52\") " pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-pcw82" Nov 03 00:49:14.167690 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:49:14.167659 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/48186e38-73f5-47c6-a90a-d82afeac4a52-istio-podinfo\") pod \"maas-default-gateway-openshift-default-596dc9545f-pcw82\" (UID: \"48186e38-73f5-47c6-a90a-d82afeac4a52\") " pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-pcw82" Nov 03 00:49:14.167844 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:49:14.167701 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/48186e38-73f5-47c6-a90a-d82afeac4a52-credential-socket\") pod \"maas-default-gateway-openshift-default-596dc9545f-pcw82\" (UID: \"48186e38-73f5-47c6-a90a-d82afeac4a52\") " pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-pcw82" Nov 03 00:49:14.167844 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:49:14.167717 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/48186e38-73f5-47c6-a90a-d82afeac4a52-workload-certs\") pod \"maas-default-gateway-openshift-default-596dc9545f-pcw82\" (UID: \"48186e38-73f5-47c6-a90a-d82afeac4a52\") " pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-pcw82" Nov 03 00:49:14.167999 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:49:14.167974 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/48186e38-73f5-47c6-a90a-d82afeac4a52-istio-envoy\") pod \"maas-default-gateway-openshift-default-596dc9545f-pcw82\" (UID: \"48186e38-73f5-47c6-a90a-d82afeac4a52\") " pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-pcw82" Nov 03 00:49:14.168057 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:49:14.168031 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/48186e38-73f5-47c6-a90a-d82afeac4a52-workload-socket\") pod \"maas-default-gateway-openshift-default-596dc9545f-pcw82\" (UID: \"48186e38-73f5-47c6-a90a-d82afeac4a52\") " pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-pcw82" Nov 03 00:49:14.168117 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:49:14.168067 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lwv87\" (UniqueName: \"kubernetes.io/projected/48186e38-73f5-47c6-a90a-d82afeac4a52-kube-api-access-lwv87\") pod \"maas-default-gateway-openshift-default-596dc9545f-pcw82\" (UID: \"48186e38-73f5-47c6-a90a-d82afeac4a52\") " pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-pcw82" Nov 03 00:49:14.168173 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:49:14.168137 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/48186e38-73f5-47c6-a90a-d82afeac4a52-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-596dc9545f-pcw82\" (UID: \"48186e38-73f5-47c6-a90a-d82afeac4a52\") " pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-pcw82" Nov 03 00:49:14.168173 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:49:14.168148 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/48186e38-73f5-47c6-a90a-d82afeac4a52-credential-socket\") pod \"maas-default-gateway-openshift-default-596dc9545f-pcw82\" (UID: \"48186e38-73f5-47c6-a90a-d82afeac4a52\") " pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-pcw82" Nov 03 00:49:14.168173 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:49:14.168167 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/48186e38-73f5-47c6-a90a-d82afeac4a52-istio-token\") pod \"maas-default-gateway-openshift-default-596dc9545f-pcw82\" (UID: \"48186e38-73f5-47c6-a90a-d82afeac4a52\") " pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-pcw82" Nov 03 00:49:14.168326 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:49:14.168195 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/48186e38-73f5-47c6-a90a-d82afeac4a52-workload-certs\") pod \"maas-default-gateway-openshift-default-596dc9545f-pcw82\" (UID: \"48186e38-73f5-47c6-a90a-d82afeac4a52\") " pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-pcw82" Nov 03 00:49:14.168326 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:49:14.168250 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/48186e38-73f5-47c6-a90a-d82afeac4a52-istio-data\") pod \"maas-default-gateway-openshift-default-596dc9545f-pcw82\" (UID: \"48186e38-73f5-47c6-a90a-d82afeac4a52\") " pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-pcw82" Nov 03 00:49:14.168465 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:49:14.168431 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/48186e38-73f5-47c6-a90a-d82afeac4a52-workload-socket\") pod \"maas-default-gateway-openshift-default-596dc9545f-pcw82\" (UID: \"48186e38-73f5-47c6-a90a-d82afeac4a52\") " pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-pcw82" Nov 03 00:49:14.168623 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:49:14.168600 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/48186e38-73f5-47c6-a90a-d82afeac4a52-istio-data\") pod \"maas-default-gateway-openshift-default-596dc9545f-pcw82\" (UID: \"48186e38-73f5-47c6-a90a-d82afeac4a52\") " pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-pcw82" Nov 03 00:49:14.168753 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:49:14.168737 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/48186e38-73f5-47c6-a90a-d82afeac4a52-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-596dc9545f-pcw82\" (UID: \"48186e38-73f5-47c6-a90a-d82afeac4a52\") " pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-pcw82" Nov 03 00:49:14.170067 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:49:14.170044 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/48186e38-73f5-47c6-a90a-d82afeac4a52-istio-envoy\") pod \"maas-default-gateway-openshift-default-596dc9545f-pcw82\" (UID: \"48186e38-73f5-47c6-a90a-d82afeac4a52\") " pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-pcw82" Nov 03 00:49:14.170247 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:49:14.170232 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/48186e38-73f5-47c6-a90a-d82afeac4a52-istio-podinfo\") pod \"maas-default-gateway-openshift-default-596dc9545f-pcw82\" (UID: \"48186e38-73f5-47c6-a90a-d82afeac4a52\") " pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-pcw82" Nov 03 00:49:14.175946 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:49:14.175902 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/48186e38-73f5-47c6-a90a-d82afeac4a52-istio-token\") pod \"maas-default-gateway-openshift-default-596dc9545f-pcw82\" (UID: \"48186e38-73f5-47c6-a90a-d82afeac4a52\") " pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-pcw82" Nov 03 00:49:14.176046 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:49:14.175997 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lwv87\" (UniqueName: \"kubernetes.io/projected/48186e38-73f5-47c6-a90a-d82afeac4a52-kube-api-access-lwv87\") pod \"maas-default-gateway-openshift-default-596dc9545f-pcw82\" (UID: \"48186e38-73f5-47c6-a90a-d82afeac4a52\") " pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-pcw82" Nov 03 00:49:14.287433 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:49:14.287335 4939 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-pcw82" Nov 03 00:49:14.574795 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:49:14.574718 4939 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-pcw82"] Nov 03 00:49:14.578462 ip-10-0-140-107 kubenswrapper[4939]: W1103 00:49:14.578433 4939 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod48186e38_73f5_47c6_a90a_d82afeac4a52.slice/crio-edc25129fc4e843a9b9190be34b30951598bb92acf83d50c2dfacb27b37d2292 WatchSource:0}: Error finding container edc25129fc4e843a9b9190be34b30951598bb92acf83d50c2dfacb27b37d2292: Status 404 returned error can't find the container with id edc25129fc4e843a9b9190be34b30951598bb92acf83d50c2dfacb27b37d2292 Nov 03 00:49:14.581922 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:49:14.581879 4939 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 00:49:14.582029 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:49:14.581974 4939 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 00:49:14.582029 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:49:14.582020 4939 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 00:49:15.019826 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:49:15.019788 4939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-pcw82" event={"ID":"48186e38-73f5-47c6-a90a-d82afeac4a52","Type":"ContainerStarted","Data":"2caa716486ef738a8bc9e623ec10c415d19ed7bfc79d016ab466e6c2da482fde"} Nov 03 00:49:15.020310 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:49:15.019846 4939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-pcw82" event={"ID":"48186e38-73f5-47c6-a90a-d82afeac4a52","Type":"ContainerStarted","Data":"edc25129fc4e843a9b9190be34b30951598bb92acf83d50c2dfacb27b37d2292"} Nov 03 00:49:15.093530 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:49:15.093442 4939 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-pcw82" podStartSLOduration=2.093422333 podStartE2EDuration="2.093422333s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:49:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:49:15.093380925 +0000 UTC m=+503.357715869" watchObservedRunningTime="2026-11-03 00:49:15.093422333 +0000 UTC m=+503.357757278" Nov 03 00:49:15.287820 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:49:15.287737 4939 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-pcw82" Nov 03 00:49:15.289450 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:49:15.289423 4939 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-596dc9545f-pcw82 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.42:15021/healthz/ready\": dial tcp 10.132.0.42:15021: connect: connection refused" start-of-body= Nov 03 00:49:15.289561 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:49:15.289471 4939 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-pcw82" podUID="48186e38-73f5-47c6-a90a-d82afeac4a52" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.42:15021/healthz/ready\": dial tcp 10.132.0.42:15021: connect: connection refused" Nov 03 00:49:16.288534 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:49:16.288495 4939 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-596dc9545f-pcw82 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.42:15021/healthz/ready\": dial tcp 10.132.0.42:15021: connect: connection refused" start-of-body= Nov 03 00:49:16.289019 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:49:16.288564 4939 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-pcw82" podUID="48186e38-73f5-47c6-a90a-d82afeac4a52" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.42:15021/healthz/ready\": dial tcp 10.132.0.42:15021: connect: connection refused" Nov 03 00:49:17.288474 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:49:17.288435 4939 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-596dc9545f-pcw82 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.42:15021/healthz/ready\": dial tcp 10.132.0.42:15021: connect: connection refused" start-of-body= Nov 03 00:49:17.288623 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:49:17.288497 4939 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-pcw82" podUID="48186e38-73f5-47c6-a90a-d82afeac4a52" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.42:15021/healthz/ready\": dial tcp 10.132.0.42:15021: connect: connection refused" Nov 03 00:49:18.029340 ip-10-0-140-107 kubenswrapper[4939]: E1103 00:49:18.029302 4939 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-sthjh" containerName="extractor" Nov 03 00:49:18.288600 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:49:18.288508 4939 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-596dc9545f-pcw82 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.42:15021/healthz/ready\": dial tcp 10.132.0.42:15021: connect: connection refused" start-of-body= Nov 03 00:49:18.288600 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:49:18.288568 4939 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-pcw82" podUID="48186e38-73f5-47c6-a90a-d82afeac4a52" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.42:15021/healthz/ready\": dial tcp 10.132.0.42:15021: connect: connection refused" Nov 03 00:49:19.288464 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:49:19.288425 4939 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-596dc9545f-pcw82 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.42:15021/healthz/ready\": dial tcp 10.132.0.42:15021: connect: connection refused" start-of-body= Nov 03 00:49:19.288669 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:49:19.288483 4939 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-pcw82" podUID="48186e38-73f5-47c6-a90a-d82afeac4a52" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.42:15021/healthz/ready\": dial tcp 10.132.0.42:15021: connect: connection refused" Nov 03 00:49:20.288654 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:49:20.288612 4939 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-596dc9545f-pcw82 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.42:15021/healthz/ready\": dial tcp 10.132.0.42:15021: connect: connection refused" start-of-body= Nov 03 00:49:20.289104 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:49:20.288675 4939 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-pcw82" podUID="48186e38-73f5-47c6-a90a-d82afeac4a52" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.42:15021/healthz/ready\": dial tcp 10.132.0.42:15021: connect: connection refused" Nov 03 00:49:21.287871 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:49:21.287824 4939 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-596dc9545f-pcw82 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.42:15021/healthz/ready\": dial tcp 10.132.0.42:15021: connect: connection refused" start-of-body= Nov 03 00:49:21.288069 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:49:21.287902 4939 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-pcw82" podUID="48186e38-73f5-47c6-a90a-d82afeac4a52" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.42:15021/healthz/ready\": dial tcp 10.132.0.42:15021: connect: connection refused" Nov 03 00:49:22.764589 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:49:22.764552 4939 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-pcw82" Nov 03 00:49:22.765059 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:49:22.764935 4939 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-pcw82" Nov 03 00:49:22.765826 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:49:22.765806 4939 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-pcw82" Nov 03 00:49:29.004135 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:49:29.004098 4939 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-l9ccf"] Nov 03 00:49:29.008095 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:49:29.008074 4939 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-l9ccf" Nov 03 00:49:29.026601 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:49:29.026574 4939 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-l9ccf"] Nov 03 00:49:29.115161 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:49:29.115126 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c10d48f9-d5fc-4c4c-96c4-6531114191b8-istio-podinfo\") pod \"maas-default-gateway-openshift-default-596dc9545f-l9ccf\" (UID: \"c10d48f9-d5fc-4c4c-96c4-6531114191b8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-l9ccf" Nov 03 00:49:29.115333 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:49:29.115189 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c10d48f9-d5fc-4c4c-96c4-6531114191b8-istio-data\") pod \"maas-default-gateway-openshift-default-596dc9545f-l9ccf\" (UID: \"c10d48f9-d5fc-4c4c-96c4-6531114191b8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-l9ccf" Nov 03 00:49:29.115333 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:49:29.115284 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c10d48f9-d5fc-4c4c-96c4-6531114191b8-credential-socket\") pod \"maas-default-gateway-openshift-default-596dc9545f-l9ccf\" (UID: \"c10d48f9-d5fc-4c4c-96c4-6531114191b8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-l9ccf" Nov 03 00:49:29.115333 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:49:29.115309 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c10d48f9-d5fc-4c4c-96c4-6531114191b8-istio-token\") pod \"maas-default-gateway-openshift-default-596dc9545f-l9ccf\" (UID: \"c10d48f9-d5fc-4c4c-96c4-6531114191b8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-l9ccf" Nov 03 00:49:29.115445 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:49:29.115342 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c10d48f9-d5fc-4c4c-96c4-6531114191b8-istio-envoy\") pod \"maas-default-gateway-openshift-default-596dc9545f-l9ccf\" (UID: \"c10d48f9-d5fc-4c4c-96c4-6531114191b8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-l9ccf" Nov 03 00:49:29.115487 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:49:29.115444 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c10d48f9-d5fc-4c4c-96c4-6531114191b8-workload-certs\") pod \"maas-default-gateway-openshift-default-596dc9545f-l9ccf\" (UID: \"c10d48f9-d5fc-4c4c-96c4-6531114191b8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-l9ccf" Nov 03 00:49:29.115522 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:49:29.115487 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c10d48f9-d5fc-4c4c-96c4-6531114191b8-workload-socket\") pod \"maas-default-gateway-openshift-default-596dc9545f-l9ccf\" (UID: \"c10d48f9-d5fc-4c4c-96c4-6531114191b8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-l9ccf" Nov 03 00:49:29.115522 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:49:29.115505 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dv4tg\" (UniqueName: \"kubernetes.io/projected/c10d48f9-d5fc-4c4c-96c4-6531114191b8-kube-api-access-dv4tg\") pod \"maas-default-gateway-openshift-default-596dc9545f-l9ccf\" (UID: \"c10d48f9-d5fc-4c4c-96c4-6531114191b8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-l9ccf" Nov 03 00:49:29.115595 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:49:29.115533 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c10d48f9-d5fc-4c4c-96c4-6531114191b8-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-596dc9545f-l9ccf\" (UID: \"c10d48f9-d5fc-4c4c-96c4-6531114191b8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-l9ccf" Nov 03 00:49:29.216126 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:49:29.216090 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c10d48f9-d5fc-4c4c-96c4-6531114191b8-istio-data\") pod \"maas-default-gateway-openshift-default-596dc9545f-l9ccf\" (UID: \"c10d48f9-d5fc-4c4c-96c4-6531114191b8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-l9ccf" Nov 03 00:49:29.216303 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:49:29.216135 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c10d48f9-d5fc-4c4c-96c4-6531114191b8-credential-socket\") pod \"maas-default-gateway-openshift-default-596dc9545f-l9ccf\" (UID: \"c10d48f9-d5fc-4c4c-96c4-6531114191b8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-l9ccf" Nov 03 00:49:29.216303 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:49:29.216153 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c10d48f9-d5fc-4c4c-96c4-6531114191b8-istio-token\") pod \"maas-default-gateway-openshift-default-596dc9545f-l9ccf\" (UID: \"c10d48f9-d5fc-4c4c-96c4-6531114191b8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-l9ccf" Nov 03 00:49:29.216303 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:49:29.216178 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c10d48f9-d5fc-4c4c-96c4-6531114191b8-istio-envoy\") pod \"maas-default-gateway-openshift-default-596dc9545f-l9ccf\" (UID: \"c10d48f9-d5fc-4c4c-96c4-6531114191b8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-l9ccf" Nov 03 00:49:29.216303 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:49:29.216196 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c10d48f9-d5fc-4c4c-96c4-6531114191b8-workload-certs\") pod \"maas-default-gateway-openshift-default-596dc9545f-l9ccf\" (UID: \"c10d48f9-d5fc-4c4c-96c4-6531114191b8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-l9ccf" Nov 03 00:49:29.216303 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:49:29.216223 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c10d48f9-d5fc-4c4c-96c4-6531114191b8-workload-socket\") pod \"maas-default-gateway-openshift-default-596dc9545f-l9ccf\" (UID: \"c10d48f9-d5fc-4c4c-96c4-6531114191b8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-l9ccf" Nov 03 00:49:29.216303 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:49:29.216240 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dv4tg\" (UniqueName: \"kubernetes.io/projected/c10d48f9-d5fc-4c4c-96c4-6531114191b8-kube-api-access-dv4tg\") pod \"maas-default-gateway-openshift-default-596dc9545f-l9ccf\" (UID: \"c10d48f9-d5fc-4c4c-96c4-6531114191b8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-l9ccf" Nov 03 00:49:29.216303 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:49:29.216277 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c10d48f9-d5fc-4c4c-96c4-6531114191b8-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-596dc9545f-l9ccf\" (UID: \"c10d48f9-d5fc-4c4c-96c4-6531114191b8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-l9ccf" Nov 03 00:49:29.216303 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:49:29.216306 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c10d48f9-d5fc-4c4c-96c4-6531114191b8-istio-podinfo\") pod \"maas-default-gateway-openshift-default-596dc9545f-l9ccf\" (UID: \"c10d48f9-d5fc-4c4c-96c4-6531114191b8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-l9ccf" Nov 03 00:49:29.216729 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:49:29.216595 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c10d48f9-d5fc-4c4c-96c4-6531114191b8-credential-socket\") pod \"maas-default-gateway-openshift-default-596dc9545f-l9ccf\" (UID: \"c10d48f9-d5fc-4c4c-96c4-6531114191b8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-l9ccf" Nov 03 00:49:29.216729 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:49:29.216674 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c10d48f9-d5fc-4c4c-96c4-6531114191b8-istio-data\") pod \"maas-default-gateway-openshift-default-596dc9545f-l9ccf\" (UID: \"c10d48f9-d5fc-4c4c-96c4-6531114191b8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-l9ccf" Nov 03 00:49:29.216846 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:49:29.216728 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c10d48f9-d5fc-4c4c-96c4-6531114191b8-workload-socket\") pod \"maas-default-gateway-openshift-default-596dc9545f-l9ccf\" (UID: \"c10d48f9-d5fc-4c4c-96c4-6531114191b8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-l9ccf" Nov 03 00:49:29.216846 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:49:29.216753 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c10d48f9-d5fc-4c4c-96c4-6531114191b8-workload-certs\") pod \"maas-default-gateway-openshift-default-596dc9545f-l9ccf\" (UID: \"c10d48f9-d5fc-4c4c-96c4-6531114191b8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-l9ccf" Nov 03 00:49:29.217319 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:49:29.217288 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c10d48f9-d5fc-4c4c-96c4-6531114191b8-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-596dc9545f-l9ccf\" (UID: \"c10d48f9-d5fc-4c4c-96c4-6531114191b8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-l9ccf" Nov 03 00:49:29.218479 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:49:29.218459 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c10d48f9-d5fc-4c4c-96c4-6531114191b8-istio-envoy\") pod \"maas-default-gateway-openshift-default-596dc9545f-l9ccf\" (UID: \"c10d48f9-d5fc-4c4c-96c4-6531114191b8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-l9ccf" Nov 03 00:49:29.218608 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:49:29.218590 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c10d48f9-d5fc-4c4c-96c4-6531114191b8-istio-podinfo\") pod \"maas-default-gateway-openshift-default-596dc9545f-l9ccf\" (UID: \"c10d48f9-d5fc-4c4c-96c4-6531114191b8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-l9ccf" Nov 03 00:49:29.225334 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:49:29.225315 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dv4tg\" (UniqueName: \"kubernetes.io/projected/c10d48f9-d5fc-4c4c-96c4-6531114191b8-kube-api-access-dv4tg\") pod \"maas-default-gateway-openshift-default-596dc9545f-l9ccf\" (UID: \"c10d48f9-d5fc-4c4c-96c4-6531114191b8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-l9ccf" Nov 03 00:49:29.225595 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:49:29.225574 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c10d48f9-d5fc-4c4c-96c4-6531114191b8-istio-token\") pod \"maas-default-gateway-openshift-default-596dc9545f-l9ccf\" (UID: \"c10d48f9-d5fc-4c4c-96c4-6531114191b8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-l9ccf" Nov 03 00:49:29.321640 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:49:29.321563 4939 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-l9ccf" Nov 03 00:49:29.601102 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:49:29.601076 4939 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-l9ccf"] Nov 03 00:49:29.603114 ip-10-0-140-107 kubenswrapper[4939]: W1103 00:49:29.603089 4939 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc10d48f9_d5fc_4c4c_96c4_6531114191b8.slice/crio-1d323cbd6f0e8504b37ae689678082cc85dc007f9b2f0c554b82350ed0d2be2e WatchSource:0}: Error finding container 1d323cbd6f0e8504b37ae689678082cc85dc007f9b2f0c554b82350ed0d2be2e: Status 404 returned error can't find the container with id 1d323cbd6f0e8504b37ae689678082cc85dc007f9b2f0c554b82350ed0d2be2e Nov 03 00:49:29.605690 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:49:29.605658 4939 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 00:49:29.605765 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:49:29.605720 4939 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 00:49:29.605765 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:49:29.605749 4939 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 00:49:30.089994 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:49:30.089954 4939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-l9ccf" event={"ID":"c10d48f9-d5fc-4c4c-96c4-6531114191b8","Type":"ContainerStarted","Data":"7e92a0e30c6a10dcdccb126ec9c43ef3c7d1f0fa48b0880ee2428c2c57ea0346"} Nov 03 00:49:30.090492 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:49:30.090007 4939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-l9ccf" event={"ID":"c10d48f9-d5fc-4c4c-96c4-6531114191b8","Type":"ContainerStarted","Data":"1d323cbd6f0e8504b37ae689678082cc85dc007f9b2f0c554b82350ed0d2be2e"} Nov 03 00:49:30.119586 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:49:30.119534 4939 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-l9ccf" podStartSLOduration=2.119519455 podStartE2EDuration="2.119519455s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:49:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:49:30.119449031 +0000 UTC m=+518.383783976" watchObservedRunningTime="2026-11-03 00:49:30.119519455 +0000 UTC m=+518.383854434" Nov 03 00:49:30.322031 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:49:30.321987 4939 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-l9ccf" Nov 03 00:49:30.323707 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:49:30.323681 4939 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-596dc9545f-l9ccf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.43:15021/healthz/ready\": dial tcp 10.132.0.43:15021: connect: connection refused" start-of-body= Nov 03 00:49:30.323827 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:49:30.323731 4939 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-l9ccf" podUID="c10d48f9-d5fc-4c4c-96c4-6531114191b8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.43:15021/healthz/ready\": dial tcp 10.132.0.43:15021: connect: connection refused" Nov 03 00:49:31.322876 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:49:31.322838 4939 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-596dc9545f-l9ccf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.43:15021/healthz/ready\": dial tcp 10.132.0.43:15021: connect: connection refused" start-of-body= Nov 03 00:49:31.323288 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:49:31.322896 4939 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-l9ccf" podUID="c10d48f9-d5fc-4c4c-96c4-6531114191b8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.43:15021/healthz/ready\": dial tcp 10.132.0.43:15021: connect: connection refused" Nov 03 00:49:31.739383 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:49:31.739345 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 00:49:32.322873 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:49:32.322837 4939 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-596dc9545f-l9ccf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.43:15021/healthz/ready\": dial tcp 10.132.0.43:15021: connect: connection refused" start-of-body= Nov 03 00:49:32.323041 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:49:32.322895 4939 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-l9ccf" podUID="c10d48f9-d5fc-4c4c-96c4-6531114191b8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.43:15021/healthz/ready\": dial tcp 10.132.0.43:15021: connect: connection refused" Nov 03 00:49:33.322830 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:49:33.322793 4939 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-596dc9545f-l9ccf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.43:15021/healthz/ready\": dial tcp 10.132.0.43:15021: connect: connection refused" start-of-body= Nov 03 00:49:33.323058 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:49:33.322859 4939 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-l9ccf" podUID="c10d48f9-d5fc-4c4c-96c4-6531114191b8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.43:15021/healthz/ready\": dial tcp 10.132.0.43:15021: connect: connection refused" Nov 03 00:49:34.322646 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:49:34.322608 4939 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-596dc9545f-l9ccf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.43:15021/healthz/ready\": dial tcp 10.132.0.43:15021: connect: connection refused" start-of-body= Nov 03 00:49:34.322835 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:49:34.322664 4939 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-l9ccf" podUID="c10d48f9-d5fc-4c4c-96c4-6531114191b8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.43:15021/healthz/ready\": dial tcp 10.132.0.43:15021: connect: connection refused" Nov 03 00:49:35.322042 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:49:35.322004 4939 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-596dc9545f-l9ccf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.43:15021/healthz/ready\": dial tcp 10.132.0.43:15021: connect: connection refused" start-of-body= Nov 03 00:49:35.322437 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:49:35.322062 4939 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-l9ccf" podUID="c10d48f9-d5fc-4c4c-96c4-6531114191b8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.43:15021/healthz/ready\": dial tcp 10.132.0.43:15021: connect: connection refused" Nov 03 00:49:36.322389 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:49:36.322349 4939 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-596dc9545f-l9ccf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.43:15021/healthz/ready\": dial tcp 10.132.0.43:15021: connect: connection refused" start-of-body= Nov 03 00:49:36.322773 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:49:36.322407 4939 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-l9ccf" podUID="c10d48f9-d5fc-4c4c-96c4-6531114191b8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.43:15021/healthz/ready\": dial tcp 10.132.0.43:15021: connect: connection refused" Nov 03 00:49:37.783638 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:49:37.783602 4939 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-l9ccf" Nov 03 00:49:37.784178 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:49:37.784153 4939 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-l9ccf" Nov 03 00:49:37.784832 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:49:37.784812 4939 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-l9ccf" Nov 03 00:49:41.697214 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:49:41.697172 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 00:49:48.029254 ip-10-0-140-107 kubenswrapper[4939]: E1103 00:49:48.029217 4939 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-sthjh" containerName="extractor" Nov 03 00:49:51.751306 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:49:51.751272 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 00:49:51.935526 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:49:51.935492 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 00:49:52.128130 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:49:52.128097 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 00:49:52.321362 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:49:52.321311 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 00:49:52.322544 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:49:52.322519 4939 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ait-15abbf02-openshift-default-65d8997ff-dtl78"] Nov 03 00:49:52.327316 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:49:52.327293 4939 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-15abbf02-openshift-default-65d8997ff-dtl78" Nov 03 00:49:52.344250 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:49:52.344227 4939 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ait-15abbf02-openshift-default-dockercfg-9vsf9\"" Nov 03 00:49:52.347840 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:49:52.347815 4939 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-15abbf02-openshift-default-65d8997ff-dtl78"] Nov 03 00:49:52.447321 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:49:52.447241 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b65bcc4a-131b-422b-b407-207ec0f75f26-istiod-ca-cert\") pod \"e2e-ait-15abbf02-openshift-default-65d8997ff-dtl78\" (UID: \"b65bcc4a-131b-422b-b407-207ec0f75f26\") " pod="openshift-ingress/e2e-ait-15abbf02-openshift-default-65d8997ff-dtl78" Nov 03 00:49:52.447321 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:49:52.447301 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b65bcc4a-131b-422b-b407-207ec0f75f26-workload-certs\") pod \"e2e-ait-15abbf02-openshift-default-65d8997ff-dtl78\" (UID: \"b65bcc4a-131b-422b-b407-207ec0f75f26\") " pod="openshift-ingress/e2e-ait-15abbf02-openshift-default-65d8997ff-dtl78" Nov 03 00:49:52.447531 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:49:52.447370 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b65bcc4a-131b-422b-b407-207ec0f75f26-istio-token\") pod \"e2e-ait-15abbf02-openshift-default-65d8997ff-dtl78\" (UID: \"b65bcc4a-131b-422b-b407-207ec0f75f26\") " pod="openshift-ingress/e2e-ait-15abbf02-openshift-default-65d8997ff-dtl78" Nov 03 00:49:52.447531 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:49:52.447399 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b65bcc4a-131b-422b-b407-207ec0f75f26-istio-data\") pod \"e2e-ait-15abbf02-openshift-default-65d8997ff-dtl78\" (UID: \"b65bcc4a-131b-422b-b407-207ec0f75f26\") " pod="openshift-ingress/e2e-ait-15abbf02-openshift-default-65d8997ff-dtl78" Nov 03 00:49:52.447531 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:49:52.447420 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b65bcc4a-131b-422b-b407-207ec0f75f26-credential-socket\") pod \"e2e-ait-15abbf02-openshift-default-65d8997ff-dtl78\" (UID: \"b65bcc4a-131b-422b-b407-207ec0f75f26\") " pod="openshift-ingress/e2e-ait-15abbf02-openshift-default-65d8997ff-dtl78" Nov 03 00:49:52.447645 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:49:52.447602 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2v59r\" (UniqueName: \"kubernetes.io/projected/b65bcc4a-131b-422b-b407-207ec0f75f26-kube-api-access-2v59r\") pod \"e2e-ait-15abbf02-openshift-default-65d8997ff-dtl78\" (UID: \"b65bcc4a-131b-422b-b407-207ec0f75f26\") " pod="openshift-ingress/e2e-ait-15abbf02-openshift-default-65d8997ff-dtl78" Nov 03 00:49:52.447746 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:49:52.447722 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b65bcc4a-131b-422b-b407-207ec0f75f26-istio-envoy\") pod \"e2e-ait-15abbf02-openshift-default-65d8997ff-dtl78\" (UID: \"b65bcc4a-131b-422b-b407-207ec0f75f26\") " pod="openshift-ingress/e2e-ait-15abbf02-openshift-default-65d8997ff-dtl78" Nov 03 00:49:52.447783 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:49:52.447771 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b65bcc4a-131b-422b-b407-207ec0f75f26-istio-podinfo\") pod \"e2e-ait-15abbf02-openshift-default-65d8997ff-dtl78\" (UID: \"b65bcc4a-131b-422b-b407-207ec0f75f26\") " pod="openshift-ingress/e2e-ait-15abbf02-openshift-default-65d8997ff-dtl78" Nov 03 00:49:52.447838 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:49:52.447806 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b65bcc4a-131b-422b-b407-207ec0f75f26-workload-socket\") pod \"e2e-ait-15abbf02-openshift-default-65d8997ff-dtl78\" (UID: \"b65bcc4a-131b-422b-b407-207ec0f75f26\") " pod="openshift-ingress/e2e-ait-15abbf02-openshift-default-65d8997ff-dtl78" Nov 03 00:49:52.461098 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:49:52.461067 4939 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ait-331bcaa1-openshift-default-86476dcd55-7z4mx"] Nov 03 00:49:52.464483 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:49:52.464460 4939 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-331bcaa1-openshift-default-86476dcd55-7z4mx" Nov 03 00:49:52.478272 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:49:52.478249 4939 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ait-331bcaa1-openshift-default-dockercfg-hrdfl\"" Nov 03 00:49:52.481077 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:49:52.481052 4939 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-331bcaa1-openshift-default-86476dcd55-7z4mx"] Nov 03 00:49:52.534086 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:49:52.534047 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 00:49:52.548498 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:49:52.548464 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f710f312-c05c-4028-92ec-c4636998926f-istio-token\") pod \"e2e-ait-331bcaa1-openshift-default-86476dcd55-7z4mx\" (UID: \"f710f312-c05c-4028-92ec-c4636998926f\") " pod="openshift-ingress/e2e-ait-331bcaa1-openshift-default-86476dcd55-7z4mx" Nov 03 00:49:52.548708 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:49:52.548518 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2v59r\" (UniqueName: \"kubernetes.io/projected/b65bcc4a-131b-422b-b407-207ec0f75f26-kube-api-access-2v59r\") pod \"e2e-ait-15abbf02-openshift-default-65d8997ff-dtl78\" (UID: \"b65bcc4a-131b-422b-b407-207ec0f75f26\") " pod="openshift-ingress/e2e-ait-15abbf02-openshift-default-65d8997ff-dtl78" Nov 03 00:49:52.548708 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:49:52.548599 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f710f312-c05c-4028-92ec-c4636998926f-istiod-ca-cert\") pod \"e2e-ait-331bcaa1-openshift-default-86476dcd55-7z4mx\" (UID: \"f710f312-c05c-4028-92ec-c4636998926f\") " pod="openshift-ingress/e2e-ait-331bcaa1-openshift-default-86476dcd55-7z4mx" Nov 03 00:49:52.548708 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:49:52.548639 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b65bcc4a-131b-422b-b407-207ec0f75f26-istio-envoy\") pod \"e2e-ait-15abbf02-openshift-default-65d8997ff-dtl78\" (UID: \"b65bcc4a-131b-422b-b407-207ec0f75f26\") " pod="openshift-ingress/e2e-ait-15abbf02-openshift-default-65d8997ff-dtl78" Nov 03 00:49:52.548708 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:49:52.548661 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f710f312-c05c-4028-92ec-c4636998926f-workload-certs\") pod \"e2e-ait-331bcaa1-openshift-default-86476dcd55-7z4mx\" (UID: \"f710f312-c05c-4028-92ec-c4636998926f\") " pod="openshift-ingress/e2e-ait-331bcaa1-openshift-default-86476dcd55-7z4mx" Nov 03 00:49:52.549061 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:49:52.548824 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b65bcc4a-131b-422b-b407-207ec0f75f26-istio-podinfo\") pod \"e2e-ait-15abbf02-openshift-default-65d8997ff-dtl78\" (UID: \"b65bcc4a-131b-422b-b407-207ec0f75f26\") " pod="openshift-ingress/e2e-ait-15abbf02-openshift-default-65d8997ff-dtl78" Nov 03 00:49:52.549061 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:49:52.548854 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b65bcc4a-131b-422b-b407-207ec0f75f26-workload-socket\") pod \"e2e-ait-15abbf02-openshift-default-65d8997ff-dtl78\" (UID: \"b65bcc4a-131b-422b-b407-207ec0f75f26\") " pod="openshift-ingress/e2e-ait-15abbf02-openshift-default-65d8997ff-dtl78" Nov 03 00:49:52.549061 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:49:52.548884 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wd4w8\" (UniqueName: \"kubernetes.io/projected/f710f312-c05c-4028-92ec-c4636998926f-kube-api-access-wd4w8\") pod \"e2e-ait-331bcaa1-openshift-default-86476dcd55-7z4mx\" (UID: \"f710f312-c05c-4028-92ec-c4636998926f\") " pod="openshift-ingress/e2e-ait-331bcaa1-openshift-default-86476dcd55-7z4mx" Nov 03 00:49:52.549061 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:49:52.548986 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b65bcc4a-131b-422b-b407-207ec0f75f26-istiod-ca-cert\") pod \"e2e-ait-15abbf02-openshift-default-65d8997ff-dtl78\" (UID: \"b65bcc4a-131b-422b-b407-207ec0f75f26\") " pod="openshift-ingress/e2e-ait-15abbf02-openshift-default-65d8997ff-dtl78" Nov 03 00:49:52.549061 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:49:52.549015 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f710f312-c05c-4028-92ec-c4636998926f-credential-socket\") pod \"e2e-ait-331bcaa1-openshift-default-86476dcd55-7z4mx\" (UID: \"f710f312-c05c-4028-92ec-c4636998926f\") " pod="openshift-ingress/e2e-ait-331bcaa1-openshift-default-86476dcd55-7z4mx" Nov 03 00:49:52.549389 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:49:52.549156 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f710f312-c05c-4028-92ec-c4636998926f-istio-envoy\") pod \"e2e-ait-331bcaa1-openshift-default-86476dcd55-7z4mx\" (UID: \"f710f312-c05c-4028-92ec-c4636998926f\") " pod="openshift-ingress/e2e-ait-331bcaa1-openshift-default-86476dcd55-7z4mx" Nov 03 00:49:52.549389 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:49:52.549187 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b65bcc4a-131b-422b-b407-207ec0f75f26-workload-certs\") pod \"e2e-ait-15abbf02-openshift-default-65d8997ff-dtl78\" (UID: \"b65bcc4a-131b-422b-b407-207ec0f75f26\") " pod="openshift-ingress/e2e-ait-15abbf02-openshift-default-65d8997ff-dtl78" Nov 03 00:49:52.549389 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:49:52.549229 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f710f312-c05c-4028-92ec-c4636998926f-workload-socket\") pod \"e2e-ait-331bcaa1-openshift-default-86476dcd55-7z4mx\" (UID: \"f710f312-c05c-4028-92ec-c4636998926f\") " pod="openshift-ingress/e2e-ait-331bcaa1-openshift-default-86476dcd55-7z4mx" Nov 03 00:49:52.549389 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:49:52.549275 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b65bcc4a-131b-422b-b407-207ec0f75f26-workload-socket\") pod \"e2e-ait-15abbf02-openshift-default-65d8997ff-dtl78\" (UID: \"b65bcc4a-131b-422b-b407-207ec0f75f26\") " pod="openshift-ingress/e2e-ait-15abbf02-openshift-default-65d8997ff-dtl78" Nov 03 00:49:52.549598 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:49:52.549472 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b65bcc4a-131b-422b-b407-207ec0f75f26-workload-certs\") pod \"e2e-ait-15abbf02-openshift-default-65d8997ff-dtl78\" (UID: \"b65bcc4a-131b-422b-b407-207ec0f75f26\") " pod="openshift-ingress/e2e-ait-15abbf02-openshift-default-65d8997ff-dtl78" Nov 03 00:49:52.549598 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:49:52.549558 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b65bcc4a-131b-422b-b407-207ec0f75f26-istio-token\") pod \"e2e-ait-15abbf02-openshift-default-65d8997ff-dtl78\" (UID: \"b65bcc4a-131b-422b-b407-207ec0f75f26\") " pod="openshift-ingress/e2e-ait-15abbf02-openshift-default-65d8997ff-dtl78" Nov 03 00:49:52.549598 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:49:52.549590 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b65bcc4a-131b-422b-b407-207ec0f75f26-istio-data\") pod \"e2e-ait-15abbf02-openshift-default-65d8997ff-dtl78\" (UID: \"b65bcc4a-131b-422b-b407-207ec0f75f26\") " pod="openshift-ingress/e2e-ait-15abbf02-openshift-default-65d8997ff-dtl78" Nov 03 00:49:52.549753 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:49:52.549611 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f710f312-c05c-4028-92ec-c4636998926f-istio-data\") pod \"e2e-ait-331bcaa1-openshift-default-86476dcd55-7z4mx\" (UID: \"f710f312-c05c-4028-92ec-c4636998926f\") " pod="openshift-ingress/e2e-ait-331bcaa1-openshift-default-86476dcd55-7z4mx" Nov 03 00:49:52.549753 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:49:52.549660 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f710f312-c05c-4028-92ec-c4636998926f-istio-podinfo\") pod \"e2e-ait-331bcaa1-openshift-default-86476dcd55-7z4mx\" (UID: \"f710f312-c05c-4028-92ec-c4636998926f\") " pod="openshift-ingress/e2e-ait-331bcaa1-openshift-default-86476dcd55-7z4mx" Nov 03 00:49:52.549753 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:49:52.549724 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b65bcc4a-131b-422b-b407-207ec0f75f26-istiod-ca-cert\") pod \"e2e-ait-15abbf02-openshift-default-65d8997ff-dtl78\" (UID: \"b65bcc4a-131b-422b-b407-207ec0f75f26\") " pod="openshift-ingress/e2e-ait-15abbf02-openshift-default-65d8997ff-dtl78" Nov 03 00:49:52.549923 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:49:52.549798 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b65bcc4a-131b-422b-b407-207ec0f75f26-credential-socket\") pod \"e2e-ait-15abbf02-openshift-default-65d8997ff-dtl78\" (UID: \"b65bcc4a-131b-422b-b407-207ec0f75f26\") " pod="openshift-ingress/e2e-ait-15abbf02-openshift-default-65d8997ff-dtl78" Nov 03 00:49:52.549923 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:49:52.549880 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b65bcc4a-131b-422b-b407-207ec0f75f26-istio-data\") pod \"e2e-ait-15abbf02-openshift-default-65d8997ff-dtl78\" (UID: \"b65bcc4a-131b-422b-b407-207ec0f75f26\") " pod="openshift-ingress/e2e-ait-15abbf02-openshift-default-65d8997ff-dtl78" Nov 03 00:49:52.550231 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:49:52.550209 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b65bcc4a-131b-422b-b407-207ec0f75f26-credential-socket\") pod \"e2e-ait-15abbf02-openshift-default-65d8997ff-dtl78\" (UID: \"b65bcc4a-131b-422b-b407-207ec0f75f26\") " pod="openshift-ingress/e2e-ait-15abbf02-openshift-default-65d8997ff-dtl78" Nov 03 00:49:52.552065 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:49:52.552032 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b65bcc4a-131b-422b-b407-207ec0f75f26-istio-podinfo\") pod \"e2e-ait-15abbf02-openshift-default-65d8997ff-dtl78\" (UID: \"b65bcc4a-131b-422b-b407-207ec0f75f26\") " pod="openshift-ingress/e2e-ait-15abbf02-openshift-default-65d8997ff-dtl78" Nov 03 00:49:52.558203 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:49:52.558173 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b65bcc4a-131b-422b-b407-207ec0f75f26-istio-envoy\") pod \"e2e-ait-15abbf02-openshift-default-65d8997ff-dtl78\" (UID: \"b65bcc4a-131b-422b-b407-207ec0f75f26\") " pod="openshift-ingress/e2e-ait-15abbf02-openshift-default-65d8997ff-dtl78" Nov 03 00:49:52.558976 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:49:52.558950 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2v59r\" (UniqueName: \"kubernetes.io/projected/b65bcc4a-131b-422b-b407-207ec0f75f26-kube-api-access-2v59r\") pod \"e2e-ait-15abbf02-openshift-default-65d8997ff-dtl78\" (UID: \"b65bcc4a-131b-422b-b407-207ec0f75f26\") " pod="openshift-ingress/e2e-ait-15abbf02-openshift-default-65d8997ff-dtl78" Nov 03 00:49:52.559082 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:49:52.559047 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b65bcc4a-131b-422b-b407-207ec0f75f26-istio-token\") pod \"e2e-ait-15abbf02-openshift-default-65d8997ff-dtl78\" (UID: \"b65bcc4a-131b-422b-b407-207ec0f75f26\") " pod="openshift-ingress/e2e-ait-15abbf02-openshift-default-65d8997ff-dtl78" Nov 03 00:49:52.640422 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:49:52.640383 4939 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-15abbf02-openshift-default-65d8997ff-dtl78" Nov 03 00:49:52.650594 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:49:52.650561 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f710f312-c05c-4028-92ec-c4636998926f-istio-data\") pod \"e2e-ait-331bcaa1-openshift-default-86476dcd55-7z4mx\" (UID: \"f710f312-c05c-4028-92ec-c4636998926f\") " pod="openshift-ingress/e2e-ait-331bcaa1-openshift-default-86476dcd55-7z4mx" Nov 03 00:49:52.650740 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:49:52.650599 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f710f312-c05c-4028-92ec-c4636998926f-istio-podinfo\") pod \"e2e-ait-331bcaa1-openshift-default-86476dcd55-7z4mx\" (UID: \"f710f312-c05c-4028-92ec-c4636998926f\") " pod="openshift-ingress/e2e-ait-331bcaa1-openshift-default-86476dcd55-7z4mx" Nov 03 00:49:52.650740 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:49:52.650630 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f710f312-c05c-4028-92ec-c4636998926f-istio-token\") pod \"e2e-ait-331bcaa1-openshift-default-86476dcd55-7z4mx\" (UID: \"f710f312-c05c-4028-92ec-c4636998926f\") " pod="openshift-ingress/e2e-ait-331bcaa1-openshift-default-86476dcd55-7z4mx" Nov 03 00:49:52.650850 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:49:52.650795 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f710f312-c05c-4028-92ec-c4636998926f-istiod-ca-cert\") pod \"e2e-ait-331bcaa1-openshift-default-86476dcd55-7z4mx\" (UID: \"f710f312-c05c-4028-92ec-c4636998926f\") " pod="openshift-ingress/e2e-ait-331bcaa1-openshift-default-86476dcd55-7z4mx" Nov 03 00:49:52.650938 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:49:52.650849 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f710f312-c05c-4028-92ec-c4636998926f-workload-certs\") pod \"e2e-ait-331bcaa1-openshift-default-86476dcd55-7z4mx\" (UID: \"f710f312-c05c-4028-92ec-c4636998926f\") " pod="openshift-ingress/e2e-ait-331bcaa1-openshift-default-86476dcd55-7z4mx" Nov 03 00:49:52.650938 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:49:52.650890 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wd4w8\" (UniqueName: \"kubernetes.io/projected/f710f312-c05c-4028-92ec-c4636998926f-kube-api-access-wd4w8\") pod \"e2e-ait-331bcaa1-openshift-default-86476dcd55-7z4mx\" (UID: \"f710f312-c05c-4028-92ec-c4636998926f\") " pod="openshift-ingress/e2e-ait-331bcaa1-openshift-default-86476dcd55-7z4mx" Nov 03 00:49:52.651074 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:49:52.650954 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f710f312-c05c-4028-92ec-c4636998926f-credential-socket\") pod \"e2e-ait-331bcaa1-openshift-default-86476dcd55-7z4mx\" (UID: \"f710f312-c05c-4028-92ec-c4636998926f\") " pod="openshift-ingress/e2e-ait-331bcaa1-openshift-default-86476dcd55-7z4mx" Nov 03 00:49:52.651074 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:49:52.651051 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f710f312-c05c-4028-92ec-c4636998926f-istio-envoy\") pod \"e2e-ait-331bcaa1-openshift-default-86476dcd55-7z4mx\" (UID: \"f710f312-c05c-4028-92ec-c4636998926f\") " pod="openshift-ingress/e2e-ait-331bcaa1-openshift-default-86476dcd55-7z4mx" Nov 03 00:49:52.651175 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:49:52.651071 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f710f312-c05c-4028-92ec-c4636998926f-istio-data\") pod \"e2e-ait-331bcaa1-openshift-default-86476dcd55-7z4mx\" (UID: \"f710f312-c05c-4028-92ec-c4636998926f\") " pod="openshift-ingress/e2e-ait-331bcaa1-openshift-default-86476dcd55-7z4mx" Nov 03 00:49:52.651175 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:49:52.651081 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f710f312-c05c-4028-92ec-c4636998926f-workload-socket\") pod \"e2e-ait-331bcaa1-openshift-default-86476dcd55-7z4mx\" (UID: \"f710f312-c05c-4028-92ec-c4636998926f\") " pod="openshift-ingress/e2e-ait-331bcaa1-openshift-default-86476dcd55-7z4mx" Nov 03 00:49:52.651279 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:49:52.651216 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f710f312-c05c-4028-92ec-c4636998926f-workload-certs\") pod \"e2e-ait-331bcaa1-openshift-default-86476dcd55-7z4mx\" (UID: \"f710f312-c05c-4028-92ec-c4636998926f\") " pod="openshift-ingress/e2e-ait-331bcaa1-openshift-default-86476dcd55-7z4mx" Nov 03 00:49:52.651565 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:49:52.651541 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f710f312-c05c-4028-92ec-c4636998926f-istiod-ca-cert\") pod \"e2e-ait-331bcaa1-openshift-default-86476dcd55-7z4mx\" (UID: \"f710f312-c05c-4028-92ec-c4636998926f\") " pod="openshift-ingress/e2e-ait-331bcaa1-openshift-default-86476dcd55-7z4mx" Nov 03 00:49:52.651665 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:49:52.651570 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f710f312-c05c-4028-92ec-c4636998926f-workload-socket\") pod \"e2e-ait-331bcaa1-openshift-default-86476dcd55-7z4mx\" (UID: \"f710f312-c05c-4028-92ec-c4636998926f\") " pod="openshift-ingress/e2e-ait-331bcaa1-openshift-default-86476dcd55-7z4mx" Nov 03 00:49:52.651665 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:49:52.651628 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f710f312-c05c-4028-92ec-c4636998926f-credential-socket\") pod \"e2e-ait-331bcaa1-openshift-default-86476dcd55-7z4mx\" (UID: \"f710f312-c05c-4028-92ec-c4636998926f\") " pod="openshift-ingress/e2e-ait-331bcaa1-openshift-default-86476dcd55-7z4mx" Nov 03 00:49:52.653299 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:49:52.653279 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f710f312-c05c-4028-92ec-c4636998926f-istio-envoy\") pod \"e2e-ait-331bcaa1-openshift-default-86476dcd55-7z4mx\" (UID: \"f710f312-c05c-4028-92ec-c4636998926f\") " pod="openshift-ingress/e2e-ait-331bcaa1-openshift-default-86476dcd55-7z4mx" Nov 03 00:49:52.653622 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:49:52.653600 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f710f312-c05c-4028-92ec-c4636998926f-istio-podinfo\") pod \"e2e-ait-331bcaa1-openshift-default-86476dcd55-7z4mx\" (UID: \"f710f312-c05c-4028-92ec-c4636998926f\") " pod="openshift-ingress/e2e-ait-331bcaa1-openshift-default-86476dcd55-7z4mx" Nov 03 00:49:52.665377 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:49:52.665352 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f710f312-c05c-4028-92ec-c4636998926f-istio-token\") pod \"e2e-ait-331bcaa1-openshift-default-86476dcd55-7z4mx\" (UID: \"f710f312-c05c-4028-92ec-c4636998926f\") " pod="openshift-ingress/e2e-ait-331bcaa1-openshift-default-86476dcd55-7z4mx" Nov 03 00:49:52.665509 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:49:52.665491 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wd4w8\" (UniqueName: \"kubernetes.io/projected/f710f312-c05c-4028-92ec-c4636998926f-kube-api-access-wd4w8\") pod \"e2e-ait-331bcaa1-openshift-default-86476dcd55-7z4mx\" (UID: \"f710f312-c05c-4028-92ec-c4636998926f\") " pod="openshift-ingress/e2e-ait-331bcaa1-openshift-default-86476dcd55-7z4mx" Nov 03 00:49:52.695741 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:49:52.695691 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 00:49:52.777510 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:49:52.777474 4939 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-331bcaa1-openshift-default-86476dcd55-7z4mx" Nov 03 00:49:52.867827 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:49:52.867787 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 00:49:52.932017 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:49:52.931980 4939 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-15abbf02-openshift-default-65d8997ff-dtl78"] Nov 03 00:49:52.936237 ip-10-0-140-107 kubenswrapper[4939]: W1103 00:49:52.936205 4939 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb65bcc4a_131b_422b_b407_207ec0f75f26.slice/crio-2a29ea8c90c43e12ef117efb8a28d4015160cd253b176f8f2a3af31adbade27d WatchSource:0}: Error finding container 2a29ea8c90c43e12ef117efb8a28d4015160cd253b176f8f2a3af31adbade27d: Status 404 returned error can't find the container with id 2a29ea8c90c43e12ef117efb8a28d4015160cd253b176f8f2a3af31adbade27d Nov 03 00:49:52.938710 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:49:52.938678 4939 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 00:49:52.938788 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:49:52.938751 4939 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 00:49:52.938788 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:49:52.938776 4939 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 00:49:52.992522 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:49:52.992488 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 00:49:53.058449 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:49:53.058421 4939 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-331bcaa1-openshift-default-86476dcd55-7z4mx"] Nov 03 00:49:53.061059 ip-10-0-140-107 kubenswrapper[4939]: W1103 00:49:53.061028 4939 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf710f312_c05c_4028_92ec_c4636998926f.slice/crio-1d53322cd2d2072e81df9b898e6b4ea3c880c30d4b300eaa07b3ece5ef4839ee WatchSource:0}: Error finding container 1d53322cd2d2072e81df9b898e6b4ea3c880c30d4b300eaa07b3ece5ef4839ee: Status 404 returned error can't find the container with id 1d53322cd2d2072e81df9b898e6b4ea3c880c30d4b300eaa07b3ece5ef4839ee Nov 03 00:49:53.063357 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:49:53.063323 4939 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 00:49:53.063432 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:49:53.063390 4939 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 00:49:53.063432 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:49:53.063420 4939 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 00:49:53.198886 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:49:53.198840 4939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-15abbf02-openshift-default-65d8997ff-dtl78" event={"ID":"b65bcc4a-131b-422b-b407-207ec0f75f26","Type":"ContainerStarted","Data":"30cf59b9da3be8b2dabadecf37d011b47d92e5f95ac2e969a65d60eeeee838c5"} Nov 03 00:49:53.199064 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:49:53.198924 4939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-15abbf02-openshift-default-65d8997ff-dtl78" event={"ID":"b65bcc4a-131b-422b-b407-207ec0f75f26","Type":"ContainerStarted","Data":"2a29ea8c90c43e12ef117efb8a28d4015160cd253b176f8f2a3af31adbade27d"} Nov 03 00:49:53.200800 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:49:53.200770 4939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-331bcaa1-openshift-default-86476dcd55-7z4mx" event={"ID":"f710f312-c05c-4028-92ec-c4636998926f","Type":"ContainerStarted","Data":"42c34511bef545ccb1cfae72aa8efc91dae542368483c76d90c98c619ae89366"} Nov 03 00:49:53.200933 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:49:53.200820 4939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-331bcaa1-openshift-default-86476dcd55-7z4mx" event={"ID":"f710f312-c05c-4028-92ec-c4636998926f","Type":"ContainerStarted","Data":"1d53322cd2d2072e81df9b898e6b4ea3c880c30d4b300eaa07b3ece5ef4839ee"} Nov 03 00:49:53.220667 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:49:53.220573 4939 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ait-15abbf02-openshift-default-65d8997ff-dtl78" podStartSLOduration=1.220559091 podStartE2EDuration="1.220559091s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:49:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:49:53.219328019 +0000 UTC m=+541.483662989" watchObservedRunningTime="2026-11-03 00:49:53.220559091 +0000 UTC m=+541.484894088" Nov 03 00:49:53.241598 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:49:53.241531 4939 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ait-331bcaa1-openshift-default-86476dcd55-7z4mx" podStartSLOduration=1.241512117 podStartE2EDuration="1.241512117s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:49:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:49:53.240696414 +0000 UTC m=+541.505031360" watchObservedRunningTime="2026-11-03 00:49:53.241512117 +0000 UTC m=+541.505847062" Nov 03 00:49:53.640847 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:49:53.640805 4939 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-15abbf02-openshift-default-65d8997ff-dtl78" Nov 03 00:49:53.647182 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:49:53.647151 4939 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-15abbf02-openshift-default-65d8997ff-dtl78" Nov 03 00:49:53.777954 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:49:53.777890 4939 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-331bcaa1-openshift-default-86476dcd55-7z4mx" Nov 03 00:49:53.921566 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:49:53.921483 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 00:49:54.167296 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:49:54.167242 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 00:49:54.211111 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:49:54.211011 4939 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-15abbf02-openshift-default-65d8997ff-dtl78" Nov 03 00:49:54.212289 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:49:54.212259 4939 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-15abbf02-openshift-default-65d8997ff-dtl78" Nov 03 00:49:54.314157 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:49:54.314126 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 00:49:54.784325 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:49:54.784294 4939 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-331bcaa1-openshift-default-86476dcd55-7z4mx" Nov 03 00:49:54.824327 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:49:54.824291 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 00:49:55.028493 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:49:55.028465 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 00:49:55.219257 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:49:55.219222 4939 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-331bcaa1-openshift-default-86476dcd55-7z4mx" Nov 03 00:49:55.221607 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:49:55.221577 4939 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-331bcaa1-openshift-default-86476dcd55-7z4mx" Nov 03 00:49:55.538003 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:49:55.537900 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 00:49:55.696557 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:49:55.696518 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 00:49:56.090166 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:49:56.090128 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 00:49:56.119776 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:49:56.119738 4939 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-e9b9924e-5646b597cf-cgg6n"] Nov 03 00:49:56.123589 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:49:56.123567 4939 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-migrate-e9b9924e-5646b597cf-cgg6n" Nov 03 00:49:56.129015 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:49:56.128978 4939 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-migrate-e9b9924e\"" Nov 03 00:49:56.140187 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:49:56.140146 4939 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-e9b9924e-5646b597cf-cgg6n"] Nov 03 00:49:56.147525 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:49:56.147498 4939 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-e9b9924e-79d64d74c5-r6l2z"] Nov 03 00:49:56.151941 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:49:56.151902 4939 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-migrate-e9b9924e-79d64d74c5-r6l2z" Nov 03 00:49:56.167056 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:49:56.165332 4939 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-migrate-e9b9924e-dockercfg-dx4bk\"" Nov 03 00:49:56.167824 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:49:56.167789 4939 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-e9b9924e-79d64d74c5-r6l2z"] Nov 03 00:49:56.199602 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:49:56.199561 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5vglb\" (UniqueName: \"kubernetes.io/projected/761d585c-40b1-40c8-b499-6abd8689a794-kube-api-access-5vglb\") pod \"payload-pre-processing-e2e-migrate-e9b9924e-5646b597cf-cgg6n\" (UID: \"761d585c-40b1-40c8-b499-6abd8689a794\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-e9b9924e-5646b597cf-cgg6n" Nov 03 00:49:56.199774 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:49:56.199613 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/761d585c-40b1-40c8-b499-6abd8689a794-plugins-config-volume\") pod \"payload-pre-processing-e2e-migrate-e9b9924e-5646b597cf-cgg6n\" (UID: \"761d585c-40b1-40c8-b499-6abd8689a794\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-e9b9924e-5646b597cf-cgg6n" Nov 03 00:49:56.262169 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:49:56.262131 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 00:49:56.300220 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:49:56.300183 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wj2l4\" (UniqueName: \"kubernetes.io/projected/2366920b-4d83-4d4b-a83d-15e9abf48aac-kube-api-access-wj2l4\") pod \"payload-processing-e2e-migrate-e9b9924e-79d64d74c5-r6l2z\" (UID: \"2366920b-4d83-4d4b-a83d-15e9abf48aac\") " pod="openshift-ingress/payload-processing-e2e-migrate-e9b9924e-79d64d74c5-r6l2z" Nov 03 00:49:56.300396 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:49:56.300292 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5vglb\" (UniqueName: \"kubernetes.io/projected/761d585c-40b1-40c8-b499-6abd8689a794-kube-api-access-5vglb\") pod \"payload-pre-processing-e2e-migrate-e9b9924e-5646b597cf-cgg6n\" (UID: \"761d585c-40b1-40c8-b499-6abd8689a794\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-e9b9924e-5646b597cf-cgg6n" Nov 03 00:49:56.300396 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:49:56.300336 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/761d585c-40b1-40c8-b499-6abd8689a794-plugins-config-volume\") pod \"payload-pre-processing-e2e-migrate-e9b9924e-5646b597cf-cgg6n\" (UID: \"761d585c-40b1-40c8-b499-6abd8689a794\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-e9b9924e-5646b597cf-cgg6n" Nov 03 00:49:56.300396 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:49:56.300389 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2366920b-4d83-4d4b-a83d-15e9abf48aac-plugins-config-volume\") pod \"payload-processing-e2e-migrate-e9b9924e-79d64d74c5-r6l2z\" (UID: \"2366920b-4d83-4d4b-a83d-15e9abf48aac\") " pod="openshift-ingress/payload-processing-e2e-migrate-e9b9924e-79d64d74c5-r6l2z" Nov 03 00:49:56.301329 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:49:56.301306 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/761d585c-40b1-40c8-b499-6abd8689a794-plugins-config-volume\") pod \"payload-pre-processing-e2e-migrate-e9b9924e-5646b597cf-cgg6n\" (UID: \"761d585c-40b1-40c8-b499-6abd8689a794\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-e9b9924e-5646b597cf-cgg6n" Nov 03 00:49:56.309225 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:49:56.309204 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5vglb\" (UniqueName: \"kubernetes.io/projected/761d585c-40b1-40c8-b499-6abd8689a794-kube-api-access-5vglb\") pod \"payload-pre-processing-e2e-migrate-e9b9924e-5646b597cf-cgg6n\" (UID: \"761d585c-40b1-40c8-b499-6abd8689a794\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-e9b9924e-5646b597cf-cgg6n" Nov 03 00:49:56.401650 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:49:56.401561 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wj2l4\" (UniqueName: \"kubernetes.io/projected/2366920b-4d83-4d4b-a83d-15e9abf48aac-kube-api-access-wj2l4\") pod \"payload-processing-e2e-migrate-e9b9924e-79d64d74c5-r6l2z\" (UID: \"2366920b-4d83-4d4b-a83d-15e9abf48aac\") " pod="openshift-ingress/payload-processing-e2e-migrate-e9b9924e-79d64d74c5-r6l2z" Nov 03 00:49:56.401650 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:49:56.401627 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2366920b-4d83-4d4b-a83d-15e9abf48aac-plugins-config-volume\") pod \"payload-processing-e2e-migrate-e9b9924e-79d64d74c5-r6l2z\" (UID: \"2366920b-4d83-4d4b-a83d-15e9abf48aac\") " pod="openshift-ingress/payload-processing-e2e-migrate-e9b9924e-79d64d74c5-r6l2z" Nov 03 00:49:56.402308 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:49:56.402290 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2366920b-4d83-4d4b-a83d-15e9abf48aac-plugins-config-volume\") pod \"payload-processing-e2e-migrate-e9b9924e-79d64d74c5-r6l2z\" (UID: \"2366920b-4d83-4d4b-a83d-15e9abf48aac\") " pod="openshift-ingress/payload-processing-e2e-migrate-e9b9924e-79d64d74c5-r6l2z" Nov 03 00:49:56.410871 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:49:56.410843 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wj2l4\" (UniqueName: \"kubernetes.io/projected/2366920b-4d83-4d4b-a83d-15e9abf48aac-kube-api-access-wj2l4\") pod \"payload-processing-e2e-migrate-e9b9924e-79d64d74c5-r6l2z\" (UID: \"2366920b-4d83-4d4b-a83d-15e9abf48aac\") " pod="openshift-ingress/payload-processing-e2e-migrate-e9b9924e-79d64d74c5-r6l2z" Nov 03 00:49:56.436695 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:49:56.436662 4939 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-migrate-e9b9924e-5646b597cf-cgg6n" Nov 03 00:49:56.443695 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:49:56.443671 4939 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-331bcaa1-7596774966-n5kcr"] Nov 03 00:49:56.447312 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:49:56.447295 4939 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-331bcaa1-7596774966-n5kcr" Nov 03 00:49:56.456172 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:49:56.456147 4939 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-nfcxl\"" Nov 03 00:49:56.456301 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:49:56.456237 4939 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 03 00:49:56.456390 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:49:56.456324 4939 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 03 00:49:56.456456 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:49:56.456398 4939 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ait-331bcaa1-serving-cert\"" Nov 03 00:49:56.456820 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:49:56.456794 4939 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-metrics-tls\"" Nov 03 00:49:56.458247 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:49:56.458216 4939 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-331bcaa1-7596774966-n5kcr"] Nov 03 00:49:56.469811 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:49:56.469757 4939 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-migrate-e9b9924e-79d64d74c5-r6l2z" Nov 03 00:49:56.502652 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:49:56.502186 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zx77z\" (UniqueName: \"kubernetes.io/projected/8383fa89-7f1f-4500-8f3f-6af09701a70a-kube-api-access-zx77z\") pod \"maas-api-e2e-ait-331bcaa1-7596774966-n5kcr\" (UID: \"8383fa89-7f1f-4500-8f3f-6af09701a70a\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-331bcaa1-7596774966-n5kcr" Nov 03 00:49:56.502652 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:49:56.502229 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8383fa89-7f1f-4500-8f3f-6af09701a70a-metrics-certs\") pod \"maas-api-e2e-ait-331bcaa1-7596774966-n5kcr\" (UID: \"8383fa89-7f1f-4500-8f3f-6af09701a70a\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-331bcaa1-7596774966-n5kcr" Nov 03 00:49:56.502652 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:49:56.502263 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8383fa89-7f1f-4500-8f3f-6af09701a70a-openshift-service-ca\") pod \"maas-api-e2e-ait-331bcaa1-7596774966-n5kcr\" (UID: \"8383fa89-7f1f-4500-8f3f-6af09701a70a\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-331bcaa1-7596774966-n5kcr" Nov 03 00:49:56.502652 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:49:56.502291 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8383fa89-7f1f-4500-8f3f-6af09701a70a-maas-api-tls\") pod \"maas-api-e2e-ait-331bcaa1-7596774966-n5kcr\" (UID: \"8383fa89-7f1f-4500-8f3f-6af09701a70a\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-331bcaa1-7596774966-n5kcr" Nov 03 00:49:56.603496 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:49:56.603449 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8383fa89-7f1f-4500-8f3f-6af09701a70a-maas-api-tls\") pod \"maas-api-e2e-ait-331bcaa1-7596774966-n5kcr\" (UID: \"8383fa89-7f1f-4500-8f3f-6af09701a70a\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-331bcaa1-7596774966-n5kcr" Nov 03 00:49:56.603666 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:49:56.603615 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zx77z\" (UniqueName: \"kubernetes.io/projected/8383fa89-7f1f-4500-8f3f-6af09701a70a-kube-api-access-zx77z\") pod \"maas-api-e2e-ait-331bcaa1-7596774966-n5kcr\" (UID: \"8383fa89-7f1f-4500-8f3f-6af09701a70a\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-331bcaa1-7596774966-n5kcr" Nov 03 00:49:56.603666 ip-10-0-140-107 kubenswrapper[4939]: E1103 00:49:56.603656 4939 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-ait-331bcaa1-serving-cert: secret "maas-api-e2e-ait-331bcaa1-serving-cert" not found Nov 03 00:49:56.603782 ip-10-0-140-107 kubenswrapper[4939]: E1103 00:49:56.603740 4939 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8383fa89-7f1f-4500-8f3f-6af09701a70a-maas-api-tls podName:8383fa89-7f1f-4500-8f3f-6af09701a70a nodeName:}" failed. No retries permitted until 2026-11-03 00:49:57.10371825 +0000 UTC m=+545.368053185 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/8383fa89-7f1f-4500-8f3f-6af09701a70a-maas-api-tls") pod "maas-api-e2e-ait-331bcaa1-7596774966-n5kcr" (UID: "8383fa89-7f1f-4500-8f3f-6af09701a70a") : secret "maas-api-e2e-ait-331bcaa1-serving-cert" not found Nov 03 00:49:56.603845 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:49:56.603781 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8383fa89-7f1f-4500-8f3f-6af09701a70a-metrics-certs\") pod \"maas-api-e2e-ait-331bcaa1-7596774966-n5kcr\" (UID: \"8383fa89-7f1f-4500-8f3f-6af09701a70a\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-331bcaa1-7596774966-n5kcr" Nov 03 00:49:56.603895 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:49:56.603839 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8383fa89-7f1f-4500-8f3f-6af09701a70a-openshift-service-ca\") pod \"maas-api-e2e-ait-331bcaa1-7596774966-n5kcr\" (UID: \"8383fa89-7f1f-4500-8f3f-6af09701a70a\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-331bcaa1-7596774966-n5kcr" Nov 03 00:49:56.604589 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:49:56.604567 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8383fa89-7f1f-4500-8f3f-6af09701a70a-openshift-service-ca\") pod \"maas-api-e2e-ait-331bcaa1-7596774966-n5kcr\" (UID: \"8383fa89-7f1f-4500-8f3f-6af09701a70a\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-331bcaa1-7596774966-n5kcr" Nov 03 00:49:56.607103 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:49:56.607078 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8383fa89-7f1f-4500-8f3f-6af09701a70a-metrics-certs\") pod \"maas-api-e2e-ait-331bcaa1-7596774966-n5kcr\" (UID: \"8383fa89-7f1f-4500-8f3f-6af09701a70a\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-331bcaa1-7596774966-n5kcr" Nov 03 00:49:56.610529 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:49:56.610507 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 00:49:56.643443 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:49:56.643370 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zx77z\" (UniqueName: \"kubernetes.io/projected/8383fa89-7f1f-4500-8f3f-6af09701a70a-kube-api-access-zx77z\") pod \"maas-api-e2e-ait-331bcaa1-7596774966-n5kcr\" (UID: \"8383fa89-7f1f-4500-8f3f-6af09701a70a\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-331bcaa1-7596774966-n5kcr" Nov 03 00:49:56.739288 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:49:56.739264 4939 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-e9b9924e-5646b597cf-cgg6n"] Nov 03 00:49:56.741368 ip-10-0-140-107 kubenswrapper[4939]: W1103 00:49:56.741331 4939 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod761d585c_40b1_40c8_b499_6abd8689a794.slice/crio-b34e1aa1e944bc23e611d989425b2e1edc317e8377cfc42f81daf5695c913b1d WatchSource:0}: Error finding container b34e1aa1e944bc23e611d989425b2e1edc317e8377cfc42f81daf5695c913b1d: Status 404 returned error can't find the container with id b34e1aa1e944bc23e611d989425b2e1edc317e8377cfc42f81daf5695c913b1d Nov 03 00:49:56.784791 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:49:56.784758 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 00:49:56.791497 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:49:56.791477 4939 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-e9b9924e-79d64d74c5-r6l2z"] Nov 03 00:49:56.793718 ip-10-0-140-107 kubenswrapper[4939]: W1103 00:49:56.793687 4939 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2366920b_4d83_4d4b_a83d_15e9abf48aac.slice/crio-1f71433883ce3d93a22bd04b2f444074e686ea255d93e46d41fa61fa02dce64a WatchSource:0}: Error finding container 1f71433883ce3d93a22bd04b2f444074e686ea255d93e46d41fa61fa02dce64a: Status 404 returned error can't find the container with id 1f71433883ce3d93a22bd04b2f444074e686ea255d93e46d41fa61fa02dce64a Nov 03 00:49:56.951793 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:49:56.951751 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 00:49:57.107520 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:49:57.107477 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 00:49:57.110353 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:49:57.110316 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8383fa89-7f1f-4500-8f3f-6af09701a70a-maas-api-tls\") pod \"maas-api-e2e-ait-331bcaa1-7596774966-n5kcr\" (UID: \"8383fa89-7f1f-4500-8f3f-6af09701a70a\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-331bcaa1-7596774966-n5kcr" Nov 03 00:49:57.115692 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:49:57.115628 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8383fa89-7f1f-4500-8f3f-6af09701a70a-maas-api-tls\") pod \"maas-api-e2e-ait-331bcaa1-7596774966-n5kcr\" (UID: \"8383fa89-7f1f-4500-8f3f-6af09701a70a\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-331bcaa1-7596774966-n5kcr" Nov 03 00:49:57.259872 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:49:57.259836 4939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-migrate-e9b9924e-5646b597cf-cgg6n" event={"ID":"761d585c-40b1-40c8-b499-6abd8689a794","Type":"ContainerStarted","Data":"5e8c60632dc6ab481ec4073115b06c3e2f83a8edfcfc05a98bfc8b44a6ce7a7a"} Nov 03 00:49:57.260091 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:49:57.260017 4939 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-migrate-e9b9924e-5646b597cf-cgg6n" Nov 03 00:49:57.260091 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:49:57.260057 4939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-migrate-e9b9924e-5646b597cf-cgg6n" event={"ID":"761d585c-40b1-40c8-b499-6abd8689a794","Type":"ContainerStarted","Data":"b34e1aa1e944bc23e611d989425b2e1edc317e8377cfc42f81daf5695c913b1d"} Nov 03 00:49:57.261611 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:49:57.261584 4939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-migrate-e9b9924e-79d64d74c5-r6l2z" event={"ID":"2366920b-4d83-4d4b-a83d-15e9abf48aac","Type":"ContainerStarted","Data":"a8f476211eef1abd28f3e66059ae6516c321b95a4f6a58f261ef4f3a05f6d1ca"} Nov 03 00:49:57.261738 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:49:57.261628 4939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-migrate-e9b9924e-79d64d74c5-r6l2z" event={"ID":"2366920b-4d83-4d4b-a83d-15e9abf48aac","Type":"ContainerStarted","Data":"1f71433883ce3d93a22bd04b2f444074e686ea255d93e46d41fa61fa02dce64a"} Nov 03 00:49:57.261777 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:49:57.261735 4939 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-migrate-e9b9924e-79d64d74c5-r6l2z" Nov 03 00:49:57.285667 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:49:57.285615 4939 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-migrate-e9b9924e-5646b597cf-cgg6n" podStartSLOduration=1.285600329 podStartE2EDuration="1.285600329s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:49:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:49:57.284093535 +0000 UTC m=+545.548428491" watchObservedRunningTime="2026-11-03 00:49:57.285600329 +0000 UTC m=+545.549935319" Nov 03 00:49:57.312769 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:49:57.312699 4939 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-migrate-e9b9924e-79d64d74c5-r6l2z" podStartSLOduration=1.31267884 podStartE2EDuration="1.31267884s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:49:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:49:57.310271397 +0000 UTC m=+545.574606342" watchObservedRunningTime="2026-11-03 00:49:57.31267884 +0000 UTC m=+545.577013786" Nov 03 00:49:57.384553 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:49:57.384467 4939 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-331bcaa1-7596774966-n5kcr" Nov 03 00:49:57.504470 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:49:57.504434 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 00:49:57.519809 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:49:57.519782 4939 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-15abbf02-85bb978ffc-q9krm"] Nov 03 00:49:57.527756 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:49:57.527727 4939 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-15abbf02-85bb978ffc-q9krm" Nov 03 00:49:57.544588 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:49:57.544565 4939 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-15abbf02-85bb978ffc-q9krm"] Nov 03 00:49:57.544735 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:49:57.544721 4939 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ait-15abbf02\"" Nov 03 00:49:57.615641 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:49:57.615602 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/006cb224-99e3-4d1b-90ec-e1a0e73ae181-plugins-config-volume\") pod \"payload-processing-e2e-ait-15abbf02-85bb978ffc-q9krm\" (UID: \"006cb224-99e3-4d1b-90ec-e1a0e73ae181\") " pod="openshift-ingress/payload-processing-e2e-ait-15abbf02-85bb978ffc-q9krm" Nov 03 00:49:57.615814 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:49:57.615672 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dgrpp\" (UniqueName: \"kubernetes.io/projected/006cb224-99e3-4d1b-90ec-e1a0e73ae181-kube-api-access-dgrpp\") pod \"payload-processing-e2e-ait-15abbf02-85bb978ffc-q9krm\" (UID: \"006cb224-99e3-4d1b-90ec-e1a0e73ae181\") " pod="openshift-ingress/payload-processing-e2e-ait-15abbf02-85bb978ffc-q9krm" Nov 03 00:49:57.651627 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:49:57.651539 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 00:49:57.665805 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:49:57.665772 4939 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-331bcaa1-7596774966-n5kcr"] Nov 03 00:49:57.667343 ip-10-0-140-107 kubenswrapper[4939]: W1103 00:49:57.667305 4939 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8383fa89_7f1f_4500_8f3f_6af09701a70a.slice/crio-defe59decf0aa8bfda0a4159b103b4292c53f3582782dd8da2ceeb4b643518ff WatchSource:0}: Error finding container defe59decf0aa8bfda0a4159b103b4292c53f3582782dd8da2ceeb4b643518ff: Status 404 returned error can't find the container with id defe59decf0aa8bfda0a4159b103b4292c53f3582782dd8da2ceeb4b643518ff Nov 03 00:49:57.717312 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:49:57.717272 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/006cb224-99e3-4d1b-90ec-e1a0e73ae181-plugins-config-volume\") pod \"payload-processing-e2e-ait-15abbf02-85bb978ffc-q9krm\" (UID: \"006cb224-99e3-4d1b-90ec-e1a0e73ae181\") " pod="openshift-ingress/payload-processing-e2e-ait-15abbf02-85bb978ffc-q9krm" Nov 03 00:49:57.717495 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:49:57.717366 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dgrpp\" (UniqueName: \"kubernetes.io/projected/006cb224-99e3-4d1b-90ec-e1a0e73ae181-kube-api-access-dgrpp\") pod \"payload-processing-e2e-ait-15abbf02-85bb978ffc-q9krm\" (UID: \"006cb224-99e3-4d1b-90ec-e1a0e73ae181\") " pod="openshift-ingress/payload-processing-e2e-ait-15abbf02-85bb978ffc-q9krm" Nov 03 00:49:57.718630 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:49:57.718568 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/006cb224-99e3-4d1b-90ec-e1a0e73ae181-plugins-config-volume\") pod \"payload-processing-e2e-ait-15abbf02-85bb978ffc-q9krm\" (UID: \"006cb224-99e3-4d1b-90ec-e1a0e73ae181\") " pod="openshift-ingress/payload-processing-e2e-ait-15abbf02-85bb978ffc-q9krm" Nov 03 00:49:57.739203 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:49:57.739168 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dgrpp\" (UniqueName: \"kubernetes.io/projected/006cb224-99e3-4d1b-90ec-e1a0e73ae181-kube-api-access-dgrpp\") pod \"payload-processing-e2e-ait-15abbf02-85bb978ffc-q9krm\" (UID: \"006cb224-99e3-4d1b-90ec-e1a0e73ae181\") " pod="openshift-ingress/payload-processing-e2e-ait-15abbf02-85bb978ffc-q9krm" Nov 03 00:49:57.841279 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:49:57.841243 4939 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-15abbf02-85bb978ffc-q9krm" Nov 03 00:49:57.983283 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:49:57.983235 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 00:49:58.126547 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:49:58.126511 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 00:49:58.132435 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:49:58.132405 4939 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-15abbf02-85bb978ffc-q9krm"] Nov 03 00:49:58.136635 ip-10-0-140-107 kubenswrapper[4939]: W1103 00:49:58.136603 4939 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod006cb224_99e3_4d1b_90ec_e1a0e73ae181.slice/crio-c5e2dd32d871f51a111a3f4c592af5aad7036439ff3460e48259f2f3df245b51 WatchSource:0}: Error finding container c5e2dd32d871f51a111a3f4c592af5aad7036439ff3460e48259f2f3df245b51: Status 404 returned error can't find the container with id c5e2dd32d871f51a111a3f4c592af5aad7036439ff3460e48259f2f3df245b51 Nov 03 00:49:58.252557 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:49:58.252522 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 00:49:58.267731 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:49:58.267700 4939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-15abbf02-85bb978ffc-q9krm" event={"ID":"006cb224-99e3-4d1b-90ec-e1a0e73ae181","Type":"ContainerStarted","Data":"c9b11043e7d8dac79abbbb1de86d90a55625b07a178eaf615137c7d84b67fe07"} Nov 03 00:49:58.267869 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:49:58.267756 4939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-15abbf02-85bb978ffc-q9krm" event={"ID":"006cb224-99e3-4d1b-90ec-e1a0e73ae181","Type":"ContainerStarted","Data":"c5e2dd32d871f51a111a3f4c592af5aad7036439ff3460e48259f2f3df245b51"} Nov 03 00:49:58.267869 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:49:58.267789 4939 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ait-15abbf02-85bb978ffc-q9krm" Nov 03 00:49:58.269098 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:49:58.269070 4939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-331bcaa1-7596774966-n5kcr" event={"ID":"8383fa89-7f1f-4500-8f3f-6af09701a70a","Type":"ContainerStarted","Data":"defe59decf0aa8bfda0a4159b103b4292c53f3582782dd8da2ceeb4b643518ff"} Nov 03 00:49:58.293523 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:49:58.293478 4939 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ait-15abbf02-85bb978ffc-q9krm" podStartSLOduration=1.293463016 podStartE2EDuration="1.293463016s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:49:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:49:58.290737571 +0000 UTC m=+546.555072516" watchObservedRunningTime="2026-11-03 00:49:58.293463016 +0000 UTC m=+546.557797961" Nov 03 00:49:58.940258 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:49:58.940229 4939 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-e9b9924e-76848c4649-7d998"] Nov 03 00:49:58.943960 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:49:58.943937 4939 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-e9b9924e-76848c4649-7d998" Nov 03 00:49:58.960802 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:49:58.960003 4939 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-migrate-e9b9924e-serving-cert\"" Nov 03 00:49:58.962361 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:49:58.962336 4939 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-e9b9924e-76848c4649-7d998"] Nov 03 00:49:59.032338 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:49:59.032303 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9150215d-1d55-4fa1-ad85-056b5bdf1b2c-metrics-certs\") pod \"maas-api-e2e-migrate-e9b9924e-76848c4649-7d998\" (UID: \"9150215d-1d55-4fa1-ad85-056b5bdf1b2c\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-e9b9924e-76848c4649-7d998" Nov 03 00:49:59.032498 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:49:59.032405 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9150215d-1d55-4fa1-ad85-056b5bdf1b2c-maas-api-tls\") pod \"maas-api-e2e-migrate-e9b9924e-76848c4649-7d998\" (UID: \"9150215d-1d55-4fa1-ad85-056b5bdf1b2c\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-e9b9924e-76848c4649-7d998" Nov 03 00:49:59.032538 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:49:59.032499 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9150215d-1d55-4fa1-ad85-056b5bdf1b2c-openshift-service-ca\") pod \"maas-api-e2e-migrate-e9b9924e-76848c4649-7d998\" (UID: \"9150215d-1d55-4fa1-ad85-056b5bdf1b2c\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-e9b9924e-76848c4649-7d998" Nov 03 00:49:59.032594 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:49:59.032531 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fmxdb\" (UniqueName: \"kubernetes.io/projected/9150215d-1d55-4fa1-ad85-056b5bdf1b2c-kube-api-access-fmxdb\") pod \"maas-api-e2e-migrate-e9b9924e-76848c4649-7d998\" (UID: \"9150215d-1d55-4fa1-ad85-056b5bdf1b2c\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-e9b9924e-76848c4649-7d998" Nov 03 00:49:59.074361 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:49:59.074318 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 00:49:59.134307 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:49:59.134264 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9150215d-1d55-4fa1-ad85-056b5bdf1b2c-maas-api-tls\") pod \"maas-api-e2e-migrate-e9b9924e-76848c4649-7d998\" (UID: \"9150215d-1d55-4fa1-ad85-056b5bdf1b2c\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-e9b9924e-76848c4649-7d998" Nov 03 00:49:59.134800 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:49:59.134461 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9150215d-1d55-4fa1-ad85-056b5bdf1b2c-openshift-service-ca\") pod \"maas-api-e2e-migrate-e9b9924e-76848c4649-7d998\" (UID: \"9150215d-1d55-4fa1-ad85-056b5bdf1b2c\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-e9b9924e-76848c4649-7d998" Nov 03 00:49:59.134800 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:49:59.134501 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fmxdb\" (UniqueName: \"kubernetes.io/projected/9150215d-1d55-4fa1-ad85-056b5bdf1b2c-kube-api-access-fmxdb\") pod \"maas-api-e2e-migrate-e9b9924e-76848c4649-7d998\" (UID: \"9150215d-1d55-4fa1-ad85-056b5bdf1b2c\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-e9b9924e-76848c4649-7d998" Nov 03 00:49:59.134800 ip-10-0-140-107 kubenswrapper[4939]: E1103 00:49:59.134612 4939 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-migrate-e9b9924e-serving-cert: secret "maas-api-e2e-migrate-e9b9924e-serving-cert" not found Nov 03 00:49:59.134800 ip-10-0-140-107 kubenswrapper[4939]: E1103 00:49:59.134688 4939 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9150215d-1d55-4fa1-ad85-056b5bdf1b2c-maas-api-tls podName:9150215d-1d55-4fa1-ad85-056b5bdf1b2c nodeName:}" failed. No retries permitted until 2026-11-03 00:49:59.6346668 +0000 UTC m=+547.899001728 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/9150215d-1d55-4fa1-ad85-056b5bdf1b2c-maas-api-tls") pod "maas-api-e2e-migrate-e9b9924e-76848c4649-7d998" (UID: "9150215d-1d55-4fa1-ad85-056b5bdf1b2c") : secret "maas-api-e2e-migrate-e9b9924e-serving-cert" not found Nov 03 00:49:59.135086 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:49:59.134810 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9150215d-1d55-4fa1-ad85-056b5bdf1b2c-metrics-certs\") pod \"maas-api-e2e-migrate-e9b9924e-76848c4649-7d998\" (UID: \"9150215d-1d55-4fa1-ad85-056b5bdf1b2c\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-e9b9924e-76848c4649-7d998" Nov 03 00:49:59.135810 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:49:59.135748 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9150215d-1d55-4fa1-ad85-056b5bdf1b2c-openshift-service-ca\") pod \"maas-api-e2e-migrate-e9b9924e-76848c4649-7d998\" (UID: \"9150215d-1d55-4fa1-ad85-056b5bdf1b2c\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-e9b9924e-76848c4649-7d998" Nov 03 00:49:59.138501 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:49:59.138474 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9150215d-1d55-4fa1-ad85-056b5bdf1b2c-metrics-certs\") pod \"maas-api-e2e-migrate-e9b9924e-76848c4649-7d998\" (UID: \"9150215d-1d55-4fa1-ad85-056b5bdf1b2c\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-e9b9924e-76848c4649-7d998" Nov 03 00:49:59.147420 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:49:59.147397 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fmxdb\" (UniqueName: \"kubernetes.io/projected/9150215d-1d55-4fa1-ad85-056b5bdf1b2c-kube-api-access-fmxdb\") pod \"maas-api-e2e-migrate-e9b9924e-76848c4649-7d998\" (UID: \"9150215d-1d55-4fa1-ad85-056b5bdf1b2c\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-e9b9924e-76848c4649-7d998" Nov 03 00:49:59.235682 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:49:59.235604 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 00:49:59.277078 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:49:59.277044 4939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-331bcaa1-7596774966-n5kcr" event={"ID":"8383fa89-7f1f-4500-8f3f-6af09701a70a","Type":"ContainerStarted","Data":"3ac02630ec183f6aaaac065a614780e328cf0eee5efc5b7c4c0f3bf730f1243b"} Nov 03 00:49:59.277266 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:49:59.277118 4939 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-331bcaa1-7596774966-n5kcr" Nov 03 00:49:59.298992 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:49:59.298003 4939 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ait-331bcaa1-7596774966-n5kcr" podStartSLOduration=2.768380309 podStartE2EDuration="3.29798158s" totalImagesPullingTime="529.601271ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:49:56 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:49:59.29643687 +0000 UTC m=+547.560771814" watchObservedRunningTime="2026-11-03 00:49:59.29798158 +0000 UTC m=+547.562316516" Nov 03 00:49:59.358399 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:49:59.358368 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 00:49:59.639820 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:49:59.639781 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9150215d-1d55-4fa1-ad85-056b5bdf1b2c-maas-api-tls\") pod \"maas-api-e2e-migrate-e9b9924e-76848c4649-7d998\" (UID: \"9150215d-1d55-4fa1-ad85-056b5bdf1b2c\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-e9b9924e-76848c4649-7d998" Nov 03 00:49:59.642647 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:49:59.642616 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9150215d-1d55-4fa1-ad85-056b5bdf1b2c-maas-api-tls\") pod \"maas-api-e2e-migrate-e9b9924e-76848c4649-7d998\" (UID: \"9150215d-1d55-4fa1-ad85-056b5bdf1b2c\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-e9b9924e-76848c4649-7d998" Nov 03 00:49:59.701306 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:49:59.701270 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 00:49:59.818724 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:49:59.818688 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 00:49:59.857839 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:49:59.857801 4939 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-e9b9924e-76848c4649-7d998" Nov 03 00:50:00.142531 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:50:00.142503 4939 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-e9b9924e-76848c4649-7d998"] Nov 03 00:50:00.146201 ip-10-0-140-107 kubenswrapper[4939]: W1103 00:50:00.146096 4939 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9150215d_1d55_4fa1_ad85_056b5bdf1b2c.slice/crio-1c20504083f55c3a15b390fec34cbfbfcd3ca0b004cc6373706d44633b9aed83 WatchSource:0}: Error finding container 1c20504083f55c3a15b390fec34cbfbfcd3ca0b004cc6373706d44633b9aed83: Status 404 returned error can't find the container with id 1c20504083f55c3a15b390fec34cbfbfcd3ca0b004cc6373706d44633b9aed83 Nov 03 00:50:00.207989 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:50:00.207857 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 00:50:00.232342 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:50:00.232305 4939 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-4pw7v"] Nov 03 00:50:00.282264 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:50:00.282231 4939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-e9b9924e-76848c4649-7d998" event={"ID":"9150215d-1d55-4fa1-ad85-056b5bdf1b2c","Type":"ContainerStarted","Data":"1c20504083f55c3a15b390fec34cbfbfcd3ca0b004cc6373706d44633b9aed83"} Nov 03 00:50:00.789440 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:50:00.789405 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 00:50:01.000166 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:50:01.000133 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 00:50:01.127974 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:50:01.127942 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 00:50:01.293106 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:50:01.293006 4939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-e9b9924e-76848c4649-7d998" event={"ID":"9150215d-1d55-4fa1-ad85-056b5bdf1b2c","Type":"ContainerStarted","Data":"e58fd89d37806367369ea3103b3ac821c3948d94ae55f352543b898f2da61124"} Nov 03 00:50:01.293106 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:50:01.293090 4939 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-e9b9924e-76848c4649-7d998" Nov 03 00:50:01.308618 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:50:01.308579 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 00:50:01.444038 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:50:01.443992 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 00:50:01.601932 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:50:01.590851 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 00:50:02.285837 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:50:02.285802 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 00:50:03.537404 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:50:03.537363 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 00:50:04.122387 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:50:04.122354 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 00:50:04.260933 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:50:04.258368 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 00:50:04.750931 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:50:04.747601 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 00:50:05.302935 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:50:05.302112 4939 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-331bcaa1-7596774966-n5kcr" Nov 03 00:50:05.323743 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:50:05.323692 4939 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-e9b9924e-76848c4649-7d998" podStartSLOduration=6.92207104 podStartE2EDuration="7.323678129s" totalImagesPullingTime="401.607089ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:49:58 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:50:01.323777652 +0000 UTC m=+549.588112596" watchObservedRunningTime="2026-11-03 00:50:05.323678129 +0000 UTC m=+553.588013073" Nov 03 00:50:05.458934 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:50:05.458402 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 00:50:05.544043 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:50:05.541148 4939 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-4aec6006-75dd64d785-ndfg6"] Nov 03 00:50:05.545573 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:50:05.545551 4939 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-derive-4aec6006-75dd64d785-ndfg6" Nov 03 00:50:05.576684 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:50:05.575897 4939 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-derive-4aec6006-dockercfg-6895z\"" Nov 03 00:50:05.576684 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:50:05.576020 4939 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-derive-4aec6006\"" Nov 03 00:50:05.618932 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:50:05.610015 4939 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-4aec6006-75dd64d785-ndfg6"] Nov 03 00:50:05.625082 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:50:05.624857 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 00:50:05.643278 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:50:05.643249 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v8ffl\" (UniqueName: \"kubernetes.io/projected/cf2f1c18-14f6-42fa-a6ff-949c1269c429-kube-api-access-v8ffl\") pod \"payload-processing-e2e-derive-4aec6006-75dd64d785-ndfg6\" (UID: \"cf2f1c18-14f6-42fa-a6ff-949c1269c429\") " pod="openshift-ingress/payload-processing-e2e-derive-4aec6006-75dd64d785-ndfg6" Nov 03 00:50:05.643425 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:50:05.643328 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cf2f1c18-14f6-42fa-a6ff-949c1269c429-plugins-config-volume\") pod \"payload-processing-e2e-derive-4aec6006-75dd64d785-ndfg6\" (UID: \"cf2f1c18-14f6-42fa-a6ff-949c1269c429\") " pod="openshift-ingress/payload-processing-e2e-derive-4aec6006-75dd64d785-ndfg6" Nov 03 00:50:05.744479 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:50:05.744437 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v8ffl\" (UniqueName: \"kubernetes.io/projected/cf2f1c18-14f6-42fa-a6ff-949c1269c429-kube-api-access-v8ffl\") pod \"payload-processing-e2e-derive-4aec6006-75dd64d785-ndfg6\" (UID: \"cf2f1c18-14f6-42fa-a6ff-949c1269c429\") " pod="openshift-ingress/payload-processing-e2e-derive-4aec6006-75dd64d785-ndfg6" Nov 03 00:50:05.744651 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:50:05.744532 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cf2f1c18-14f6-42fa-a6ff-949c1269c429-plugins-config-volume\") pod \"payload-processing-e2e-derive-4aec6006-75dd64d785-ndfg6\" (UID: \"cf2f1c18-14f6-42fa-a6ff-949c1269c429\") " pod="openshift-ingress/payload-processing-e2e-derive-4aec6006-75dd64d785-ndfg6" Nov 03 00:50:05.745565 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:50:05.745537 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cf2f1c18-14f6-42fa-a6ff-949c1269c429-plugins-config-volume\") pod \"payload-processing-e2e-derive-4aec6006-75dd64d785-ndfg6\" (UID: \"cf2f1c18-14f6-42fa-a6ff-949c1269c429\") " pod="openshift-ingress/payload-processing-e2e-derive-4aec6006-75dd64d785-ndfg6" Nov 03 00:50:05.772928 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:50:05.772397 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-v8ffl\" (UniqueName: \"kubernetes.io/projected/cf2f1c18-14f6-42fa-a6ff-949c1269c429-kube-api-access-v8ffl\") pod \"payload-processing-e2e-derive-4aec6006-75dd64d785-ndfg6\" (UID: \"cf2f1c18-14f6-42fa-a6ff-949c1269c429\") " pod="openshift-ingress/payload-processing-e2e-derive-4aec6006-75dd64d785-ndfg6" Nov 03 00:50:05.962933 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:50:05.962110 4939 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-derive-4aec6006-75dd64d785-ndfg6" Nov 03 00:50:06.123935 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:50:06.123877 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 00:50:06.337075 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:50:06.337048 4939 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-4aec6006-75dd64d785-ndfg6"] Nov 03 00:50:06.339411 ip-10-0-140-107 kubenswrapper[4939]: W1103 00:50:06.339384 4939 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcf2f1c18_14f6_42fa_a6ff_949c1269c429.slice/crio-52e001e5d3cf4cf92bd05e613e8dfbff6c62608fb726ab8232ef39f69e73bbb9 WatchSource:0}: Error finding container 52e001e5d3cf4cf92bd05e613e8dfbff6c62608fb726ab8232ef39f69e73bbb9: Status 404 returned error can't find the container with id 52e001e5d3cf4cf92bd05e613e8dfbff6c62608fb726ab8232ef39f69e73bbb9 Nov 03 00:50:07.069467 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:50:07.069435 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 00:50:07.315132 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:50:07.315059 4939 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-e9b9924e-76848c4649-7d998" Nov 03 00:50:07.343600 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:50:07.343567 4939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-derive-4aec6006-75dd64d785-ndfg6" event={"ID":"cf2f1c18-14f6-42fa-a6ff-949c1269c429","Type":"ContainerStarted","Data":"fbd949163f8db5e57e3d945d6a0bbacb91ad74dea35b94963d518866b5625f2c"} Nov 03 00:50:07.343745 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:50:07.343625 4939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-derive-4aec6006-75dd64d785-ndfg6" event={"ID":"cf2f1c18-14f6-42fa-a6ff-949c1269c429","Type":"ContainerStarted","Data":"52e001e5d3cf4cf92bd05e613e8dfbff6c62608fb726ab8232ef39f69e73bbb9"} Nov 03 00:50:07.343745 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:50:07.343693 4939 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-derive-4aec6006-75dd64d785-ndfg6" Nov 03 00:50:07.783740 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:50:07.783705 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 00:50:08.172747 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:50:08.172689 4939 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-derive-4aec6006-75dd64d785-ndfg6" podStartSLOduration=3.172671092 podStartE2EDuration="3.172671092s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:50:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:50:07.365463625 +0000 UTC m=+555.629798572" watchObservedRunningTime="2026-11-03 00:50:08.172671092 +0000 UTC m=+556.437006015" Nov 03 00:50:08.173175 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:50:08.172840 4939 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-4pw7v"] Nov 03 00:50:08.273274 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:50:08.273239 4939 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-migrate-e9b9924e-79d64d74c5-r6l2z" Nov 03 00:50:08.273621 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:50:08.273601 4939 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-migrate-e9b9924e-5646b597cf-cgg6n" Nov 03 00:50:08.647449 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:50:08.645264 4939 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-4aec6006-7cdf66f59-5sqpc"] Nov 03 00:50:08.654924 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:50:08.654873 4939 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-4aec6006-7cdf66f59-5sqpc" Nov 03 00:50:08.660294 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:50:08.660261 4939 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-derive-4aec6006-serving-cert\"" Nov 03 00:50:08.673186 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:50:08.673147 4939 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-4aec6006-7cdf66f59-5sqpc"] Nov 03 00:50:08.692123 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:50:08.692089 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 00:50:08.750147 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:50:08.750110 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8nnjb\" (UniqueName: \"kubernetes.io/projected/e11c6e1b-c77d-49b5-a7d0-32c0f3e55e7b-kube-api-access-8nnjb\") pod \"maas-api-e2e-derive-4aec6006-7cdf66f59-5sqpc\" (UID: \"e11c6e1b-c77d-49b5-a7d0-32c0f3e55e7b\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-4aec6006-7cdf66f59-5sqpc" Nov 03 00:50:08.750356 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:50:08.750211 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e11c6e1b-c77d-49b5-a7d0-32c0f3e55e7b-maas-api-tls\") pod \"maas-api-e2e-derive-4aec6006-7cdf66f59-5sqpc\" (UID: \"e11c6e1b-c77d-49b5-a7d0-32c0f3e55e7b\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-4aec6006-7cdf66f59-5sqpc" Nov 03 00:50:08.750356 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:50:08.750271 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e11c6e1b-c77d-49b5-a7d0-32c0f3e55e7b-metrics-certs\") pod \"maas-api-e2e-derive-4aec6006-7cdf66f59-5sqpc\" (UID: \"e11c6e1b-c77d-49b5-a7d0-32c0f3e55e7b\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-4aec6006-7cdf66f59-5sqpc" Nov 03 00:50:08.750356 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:50:08.750307 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e11c6e1b-c77d-49b5-a7d0-32c0f3e55e7b-openshift-service-ca\") pod \"maas-api-e2e-derive-4aec6006-7cdf66f59-5sqpc\" (UID: \"e11c6e1b-c77d-49b5-a7d0-32c0f3e55e7b\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-4aec6006-7cdf66f59-5sqpc" Nov 03 00:50:08.814739 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:50:08.814707 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 00:50:08.850959 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:50:08.850886 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e11c6e1b-c77d-49b5-a7d0-32c0f3e55e7b-maas-api-tls\") pod \"maas-api-e2e-derive-4aec6006-7cdf66f59-5sqpc\" (UID: \"e11c6e1b-c77d-49b5-a7d0-32c0f3e55e7b\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-4aec6006-7cdf66f59-5sqpc" Nov 03 00:50:08.851180 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:50:08.851014 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e11c6e1b-c77d-49b5-a7d0-32c0f3e55e7b-metrics-certs\") pod \"maas-api-e2e-derive-4aec6006-7cdf66f59-5sqpc\" (UID: \"e11c6e1b-c77d-49b5-a7d0-32c0f3e55e7b\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-4aec6006-7cdf66f59-5sqpc" Nov 03 00:50:08.851180 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:50:08.851057 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e11c6e1b-c77d-49b5-a7d0-32c0f3e55e7b-openshift-service-ca\") pod \"maas-api-e2e-derive-4aec6006-7cdf66f59-5sqpc\" (UID: \"e11c6e1b-c77d-49b5-a7d0-32c0f3e55e7b\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-4aec6006-7cdf66f59-5sqpc" Nov 03 00:50:08.851180 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:50:08.851137 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8nnjb\" (UniqueName: \"kubernetes.io/projected/e11c6e1b-c77d-49b5-a7d0-32c0f3e55e7b-kube-api-access-8nnjb\") pod \"maas-api-e2e-derive-4aec6006-7cdf66f59-5sqpc\" (UID: \"e11c6e1b-c77d-49b5-a7d0-32c0f3e55e7b\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-4aec6006-7cdf66f59-5sqpc" Nov 03 00:50:08.852136 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:50:08.852109 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e11c6e1b-c77d-49b5-a7d0-32c0f3e55e7b-openshift-service-ca\") pod \"maas-api-e2e-derive-4aec6006-7cdf66f59-5sqpc\" (UID: \"e11c6e1b-c77d-49b5-a7d0-32c0f3e55e7b\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-4aec6006-7cdf66f59-5sqpc" Nov 03 00:50:08.853942 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:50:08.853900 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e11c6e1b-c77d-49b5-a7d0-32c0f3e55e7b-maas-api-tls\") pod \"maas-api-e2e-derive-4aec6006-7cdf66f59-5sqpc\" (UID: \"e11c6e1b-c77d-49b5-a7d0-32c0f3e55e7b\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-4aec6006-7cdf66f59-5sqpc" Nov 03 00:50:08.854168 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:50:08.854144 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e11c6e1b-c77d-49b5-a7d0-32c0f3e55e7b-metrics-certs\") pod \"maas-api-e2e-derive-4aec6006-7cdf66f59-5sqpc\" (UID: \"e11c6e1b-c77d-49b5-a7d0-32c0f3e55e7b\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-4aec6006-7cdf66f59-5sqpc" Nov 03 00:50:08.862942 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:50:08.862882 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8nnjb\" (UniqueName: \"kubernetes.io/projected/e11c6e1b-c77d-49b5-a7d0-32c0f3e55e7b-kube-api-access-8nnjb\") pod \"maas-api-e2e-derive-4aec6006-7cdf66f59-5sqpc\" (UID: \"e11c6e1b-c77d-49b5-a7d0-32c0f3e55e7b\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-4aec6006-7cdf66f59-5sqpc" Nov 03 00:50:08.944817 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:50:08.944733 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 00:50:08.993616 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:50:08.993573 4939 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-4aec6006-7cdf66f59-5sqpc" Nov 03 00:50:09.275656 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:50:09.275627 4939 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-4aec6006-7cdf66f59-5sqpc"] Nov 03 00:50:09.279571 ip-10-0-140-107 kubenswrapper[4939]: W1103 00:50:09.279498 4939 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode11c6e1b_c77d_49b5_a7d0_32c0f3e55e7b.slice/crio-41ee755c173ad88da952d76fd53abb39e2c06c36457a1b188e6b09ffdbe6fa00 WatchSource:0}: Error finding container 41ee755c173ad88da952d76fd53abb39e2c06c36457a1b188e6b09ffdbe6fa00: Status 404 returned error can't find the container with id 41ee755c173ad88da952d76fd53abb39e2c06c36457a1b188e6b09ffdbe6fa00 Nov 03 00:50:09.280327 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:50:09.280305 4939 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ait-15abbf02-85bb978ffc-q9krm" Nov 03 00:50:09.361880 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:50:09.361837 4939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-4aec6006-7cdf66f59-5sqpc" event={"ID":"e11c6e1b-c77d-49b5-a7d0-32c0f3e55e7b","Type":"ContainerStarted","Data":"41ee755c173ad88da952d76fd53abb39e2c06c36457a1b188e6b09ffdbe6fa00"} Nov 03 00:50:09.729357 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:50:09.729258 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 00:50:10.253083 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:50:10.253046 4939 patch_prober.go:29] interesting pod/e2e-ait-15abbf02-openshift-default-65d8997ff-dtl78 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.44:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 00:50:10.253243 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:50:10.253107 4939 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ait-15abbf02-openshift-default-65d8997ff-dtl78" podUID="b65bcc4a-131b-422b-b407-207ec0f75f26" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.44:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 00:50:10.369976 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:50:10.369929 4939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-4aec6006-7cdf66f59-5sqpc" event={"ID":"e11c6e1b-c77d-49b5-a7d0-32c0f3e55e7b","Type":"ContainerStarted","Data":"cf5a9700382f32d07d2a0cd029c7d23b13e5369923c6a07ec59ff2b83067f4ab"} Nov 03 00:50:10.370482 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:50:10.370115 4939 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-derive-4aec6006-7cdf66f59-5sqpc" Nov 03 00:50:10.408450 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:50:10.408385 4939 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-derive-4aec6006-7cdf66f59-5sqpc" podStartSLOduration=1.906812864 podStartE2EDuration="2.408358233s" totalImagesPullingTime="501.545369ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:50:08 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:50:10.407509641 +0000 UTC m=+558.671844596" watchObservedRunningTime="2026-11-03 00:50:10.408358233 +0000 UTC m=+558.672693181" Nov 03 00:50:10.485817 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:50:10.485770 4939 patch_prober.go:29] interesting pod/e2e-ait-331bcaa1-openshift-default-86476dcd55-7z4mx container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.45:15021/healthz/ready\": read tcp 10.132.0.2:45084->10.132.0.45:15021: read: connection reset by peer" start-of-body= Nov 03 00:50:10.485959 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:50:10.485840 4939 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ait-331bcaa1-openshift-default-86476dcd55-7z4mx" podUID="f710f312-c05c-4028-92ec-c4636998926f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.45:15021/healthz/ready\": read tcp 10.132.0.2:45084->10.132.0.45:15021: read: connection reset by peer" Nov 03 00:50:10.628739 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:50:10.628701 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 00:50:11.379596 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:50:11.379564 4939 generic.go:356] "Generic (PLEG): container finished" podID="b65bcc4a-131b-422b-b407-207ec0f75f26" containerID="30cf59b9da3be8b2dabadecf37d011b47d92e5f95ac2e969a65d60eeeee838c5" exitCode=0 Nov 03 00:50:11.380053 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:50:11.379637 4939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-15abbf02-openshift-default-65d8997ff-dtl78" event={"ID":"b65bcc4a-131b-422b-b407-207ec0f75f26","Type":"ContainerDied","Data":"30cf59b9da3be8b2dabadecf37d011b47d92e5f95ac2e969a65d60eeeee838c5"} Nov 03 00:50:11.380115 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:50:11.380085 4939 scope.go:122] "RemoveContainer" containerID="30cf59b9da3be8b2dabadecf37d011b47d92e5f95ac2e969a65d60eeeee838c5" Nov 03 00:50:11.381287 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:50:11.381267 4939 generic.go:356] "Generic (PLEG): container finished" podID="f710f312-c05c-4028-92ec-c4636998926f" containerID="42c34511bef545ccb1cfae72aa8efc91dae542368483c76d90c98c619ae89366" exitCode=0 Nov 03 00:50:11.381403 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:50:11.381330 4939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-331bcaa1-openshift-default-86476dcd55-7z4mx" event={"ID":"f710f312-c05c-4028-92ec-c4636998926f","Type":"ContainerDied","Data":"42c34511bef545ccb1cfae72aa8efc91dae542368483c76d90c98c619ae89366"} Nov 03 00:50:11.381456 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:50:11.381418 4939 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 00:50:11.381509 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:50:11.381492 4939 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 00:50:11.381553 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:50:11.381542 4939 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 00:50:11.381787 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:50:11.381774 4939 scope.go:122] "RemoveContainer" containerID="42c34511bef545ccb1cfae72aa8efc91dae542368483c76d90c98c619ae89366" Nov 03 00:50:11.382861 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:50:11.382835 4939 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 00:50:11.383001 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:50:11.382923 4939 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 00:50:11.383001 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:50:11.382977 4939 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 00:50:11.586334 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:50:11.586299 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 00:50:12.372932 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:50:12.372494 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 00:50:12.425054 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:50:12.424345 4939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-15abbf02-openshift-default-65d8997ff-dtl78" event={"ID":"b65bcc4a-131b-422b-b407-207ec0f75f26","Type":"ContainerStarted","Data":"7069432c5e0f3f92df9063f70cb7588fca184580437788a66e2b7f1cac695f19"} Nov 03 00:50:12.426976 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:50:12.426947 4939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-331bcaa1-openshift-default-86476dcd55-7z4mx" event={"ID":"f710f312-c05c-4028-92ec-c4636998926f","Type":"ContainerStarted","Data":"8dbe55abae5c74cdb23cb4c060bbc4a67c97f3fce9e56af3a86b1d538b39a717"} Nov 03 00:50:12.641578 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:50:12.641487 4939 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-15abbf02-openshift-default-65d8997ff-dtl78" Nov 03 00:50:12.641776 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:50:12.641750 4939 patch_prober.go:29] interesting pod/e2e-ait-15abbf02-openshift-default-65d8997ff-dtl78 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.44:15021/healthz/ready\": dial tcp 10.132.0.44:15021: connect: connection refused" start-of-body= Nov 03 00:50:12.641854 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:50:12.641818 4939 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-15abbf02-openshift-default-65d8997ff-dtl78" podUID="b65bcc4a-131b-422b-b407-207ec0f75f26" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.44:15021/healthz/ready\": dial tcp 10.132.0.44:15021: connect: connection refused" Nov 03 00:50:12.778361 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:50:12.778327 4939 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-331bcaa1-openshift-default-86476dcd55-7z4mx" Nov 03 00:50:12.778757 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:50:12.778661 4939 patch_prober.go:29] interesting pod/e2e-ait-331bcaa1-openshift-default-86476dcd55-7z4mx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.45:15021/healthz/ready\": dial tcp 10.132.0.45:15021: connect: connection refused" start-of-body= Nov 03 00:50:12.778950 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:50:12.778896 4939 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-331bcaa1-openshift-default-86476dcd55-7z4mx" podUID="f710f312-c05c-4028-92ec-c4636998926f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.45:15021/healthz/ready\": dial tcp 10.132.0.45:15021: connect: connection refused" Nov 03 00:50:13.292538 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:50:13.292499 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 00:50:13.640856 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:50:13.640818 4939 patch_prober.go:29] interesting pod/e2e-ait-15abbf02-openshift-default-65d8997ff-dtl78 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.44:15021/healthz/ready\": dial tcp 10.132.0.44:15021: connect: connection refused" start-of-body= Nov 03 00:50:13.642281 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:50:13.641963 4939 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-15abbf02-openshift-default-65d8997ff-dtl78" podUID="b65bcc4a-131b-422b-b407-207ec0f75f26" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.44:15021/healthz/ready\": dial tcp 10.132.0.44:15021: connect: connection refused" Nov 03 00:50:13.800932 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:50:13.778331 4939 patch_prober.go:29] interesting pod/e2e-ait-331bcaa1-openshift-default-86476dcd55-7z4mx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.45:15021/healthz/ready\": dial tcp 10.132.0.45:15021: connect: connection refused" start-of-body= Nov 03 00:50:13.800932 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:50:13.778395 4939 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-331bcaa1-openshift-default-86476dcd55-7z4mx" podUID="f710f312-c05c-4028-92ec-c4636998926f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.45:15021/healthz/ready\": dial tcp 10.132.0.45:15021: connect: connection refused" Nov 03 00:50:14.438934 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:50:14.438875 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 00:50:14.641370 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:50:14.641334 4939 patch_prober.go:29] interesting pod/e2e-ait-15abbf02-openshift-default-65d8997ff-dtl78 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.44:15021/healthz/ready\": dial tcp 10.132.0.44:15021: connect: connection refused" start-of-body= Nov 03 00:50:14.641846 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:50:14.641394 4939 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-15abbf02-openshift-default-65d8997ff-dtl78" podUID="b65bcc4a-131b-422b-b407-207ec0f75f26" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.44:15021/healthz/ready\": dial tcp 10.132.0.44:15021: connect: connection refused" Nov 03 00:50:14.779027 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:50:14.778497 4939 patch_prober.go:29] interesting pod/e2e-ait-331bcaa1-openshift-default-86476dcd55-7z4mx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.45:15021/healthz/ready\": dial tcp 10.132.0.45:15021: connect: connection refused" start-of-body= Nov 03 00:50:14.779027 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:50:14.778593 4939 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-331bcaa1-openshift-default-86476dcd55-7z4mx" podUID="f710f312-c05c-4028-92ec-c4636998926f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.45:15021/healthz/ready\": dial tcp 10.132.0.45:15021: connect: connection refused" Nov 03 00:50:14.808205 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:50:14.808168 4939 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-4pw7v"] Nov 03 00:50:15.286934 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:50:15.285834 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 00:50:15.641659 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:50:15.641628 4939 patch_prober.go:29] interesting pod/e2e-ait-15abbf02-openshift-default-65d8997ff-dtl78 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.44:15021/healthz/ready\": dial tcp 10.132.0.44:15021: connect: connection refused" start-of-body= Nov 03 00:50:15.642058 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:50:15.641679 4939 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-15abbf02-openshift-default-65d8997ff-dtl78" podUID="b65bcc4a-131b-422b-b407-207ec0f75f26" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.44:15021/healthz/ready\": dial tcp 10.132.0.44:15021: connect: connection refused" Nov 03 00:50:15.783939 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:50:15.783309 4939 patch_prober.go:29] interesting pod/e2e-ait-331bcaa1-openshift-default-86476dcd55-7z4mx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.45:15021/healthz/ready\": dial tcp 10.132.0.45:15021: connect: connection refused" start-of-body= Nov 03 00:50:15.783939 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:50:15.783378 4939 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-331bcaa1-openshift-default-86476dcd55-7z4mx" podUID="f710f312-c05c-4028-92ec-c4636998926f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.45:15021/healthz/ready\": dial tcp 10.132.0.45:15021: connect: connection refused" Nov 03 00:50:16.397934 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:50:16.397840 4939 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-derive-4aec6006-7cdf66f59-5sqpc" Nov 03 00:50:16.634655 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:50:16.634137 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 00:50:16.641579 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:50:16.641544 4939 patch_prober.go:29] interesting pod/e2e-ait-15abbf02-openshift-default-65d8997ff-dtl78 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.44:15021/healthz/ready\": dial tcp 10.132.0.44:15021: connect: connection refused" start-of-body= Nov 03 00:50:16.641838 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:50:16.641812 4939 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-15abbf02-openshift-default-65d8997ff-dtl78" podUID="b65bcc4a-131b-422b-b407-207ec0f75f26" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.44:15021/healthz/ready\": dial tcp 10.132.0.44:15021: connect: connection refused" Nov 03 00:50:16.780551 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:50:16.780450 4939 patch_prober.go:29] interesting pod/e2e-ait-331bcaa1-openshift-default-86476dcd55-7z4mx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.45:15021/healthz/ready\": dial tcp 10.132.0.45:15021: connect: connection refused" start-of-body= Nov 03 00:50:16.780847 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:50:16.780802 4939 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-331bcaa1-openshift-default-86476dcd55-7z4mx" podUID="f710f312-c05c-4028-92ec-c4636998926f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.45:15021/healthz/ready\": dial tcp 10.132.0.45:15021: connect: connection refused" Nov 03 00:50:17.641534 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:50:17.641213 4939 patch_prober.go:29] interesting pod/e2e-ait-15abbf02-openshift-default-65d8997ff-dtl78 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.44:15021/healthz/ready\": dial tcp 10.132.0.44:15021: connect: connection refused" start-of-body= Nov 03 00:50:17.641534 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:50:17.641277 4939 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-15abbf02-openshift-default-65d8997ff-dtl78" podUID="b65bcc4a-131b-422b-b407-207ec0f75f26" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.44:15021/healthz/ready\": dial tcp 10.132.0.44:15021: connect: connection refused" Nov 03 00:50:17.677663 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:50:17.677624 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 00:50:17.778925 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:50:17.778875 4939 patch_prober.go:29] interesting pod/e2e-ait-331bcaa1-openshift-default-86476dcd55-7z4mx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.45:15021/healthz/ready\": dial tcp 10.132.0.45:15021: connect: connection refused" start-of-body= Nov 03 00:50:17.779140 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:50:17.778957 4939 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-331bcaa1-openshift-default-86476dcd55-7z4mx" podUID="f710f312-c05c-4028-92ec-c4636998926f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.45:15021/healthz/ready\": dial tcp 10.132.0.45:15021: connect: connection refused" Nov 03 00:50:18.035701 ip-10-0-140-107 kubenswrapper[4939]: E1103 00:50:18.035611 4939 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-sthjh" containerName="extractor" Nov 03 00:50:18.355180 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:50:18.355151 4939 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-derive-4aec6006-75dd64d785-ndfg6" Nov 03 00:50:18.615088 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:50:18.614981 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 00:50:18.644458 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:50:18.644428 4939 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-15abbf02-openshift-default-65d8997ff-dtl78" Nov 03 00:50:18.644749 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:50:18.644728 4939 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-15abbf02-openshift-default-65d8997ff-dtl78" Nov 03 00:50:18.645504 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:50:18.645476 4939 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-15abbf02-openshift-default-65d8997ff-dtl78" Nov 03 00:50:18.781793 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:50:18.781755 4939 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-331bcaa1-openshift-default-86476dcd55-7z4mx" Nov 03 00:50:18.782209 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:50:18.782070 4939 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-331bcaa1-openshift-default-86476dcd55-7z4mx" Nov 03 00:50:18.782918 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:50:18.782891 4939 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-331bcaa1-openshift-default-86476dcd55-7z4mx" Nov 03 00:50:19.429302 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:50:19.429263 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 00:50:20.118719 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:50:20.118680 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 00:50:21.052201 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:50:21.052167 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 00:50:21.222732 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:50:21.222697 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 00:50:21.794870 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:50:21.794834 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 00:50:22.395211 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:50:22.395182 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 00:50:23.093270 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:50:23.093226 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 00:50:23.553860 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:50:23.553767 4939 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-331bcaa1-7596774966-n5kcr"] Nov 03 00:50:23.554352 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:50:23.554017 4939 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ait-331bcaa1-7596774966-n5kcr" podUID="8383fa89-7f1f-4500-8f3f-6af09701a70a" containerName="maas-api" containerID="cri-o://3ac02630ec183f6aaaac065a614780e328cf0eee5efc5b7c4c0f3bf730f1243b" gracePeriod=30 Nov 03 00:50:23.630782 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:50:23.630750 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 00:50:23.793257 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:50:23.786358 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 00:50:23.900693 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:50:23.900664 4939 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-331bcaa1-7596774966-n5kcr" Nov 03 00:50:23.901726 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:50:23.901700 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 00:50:23.932648 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:50:23.932611 4939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8383fa89-7f1f-4500-8f3f-6af09701a70a-kube-api-access-zx77z\" (UniqueName: \"kubernetes.io/projected/8383fa89-7f1f-4500-8f3f-6af09701a70a-kube-api-access-zx77z\") pod \"8383fa89-7f1f-4500-8f3f-6af09701a70a\" (UID: \"8383fa89-7f1f-4500-8f3f-6af09701a70a\") " Nov 03 00:50:23.932811 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:50:23.932754 4939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8383fa89-7f1f-4500-8f3f-6af09701a70a-metrics-certs\" (UniqueName: \"kubernetes.io/secret/8383fa89-7f1f-4500-8f3f-6af09701a70a-metrics-certs\") pod \"8383fa89-7f1f-4500-8f3f-6af09701a70a\" (UID: \"8383fa89-7f1f-4500-8f3f-6af09701a70a\") " Nov 03 00:50:23.932811 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:50:23.932814 4939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8383fa89-7f1f-4500-8f3f-6af09701a70a-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8383fa89-7f1f-4500-8f3f-6af09701a70a-maas-api-tls\") pod \"8383fa89-7f1f-4500-8f3f-6af09701a70a\" (UID: \"8383fa89-7f1f-4500-8f3f-6af09701a70a\") " Nov 03 00:50:23.933010 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:50:23.932853 4939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8383fa89-7f1f-4500-8f3f-6af09701a70a-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8383fa89-7f1f-4500-8f3f-6af09701a70a-openshift-service-ca\") pod \"8383fa89-7f1f-4500-8f3f-6af09701a70a\" (UID: \"8383fa89-7f1f-4500-8f3f-6af09701a70a\") " Nov 03 00:50:23.933452 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:50:23.933213 4939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8383fa89-7f1f-4500-8f3f-6af09701a70a-openshift-service-ca" pod "8383fa89-7f1f-4500-8f3f-6af09701a70a" (UID: "8383fa89-7f1f-4500-8f3f-6af09701a70a"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 00:50:23.933452 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:50:23.933381 4939 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8383fa89-7f1f-4500-8f3f-6af09701a70a-openshift-service-ca\") on node \"ip-10-0-140-107.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 00:50:23.935540 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:50:23.935508 4939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8383fa89-7f1f-4500-8f3f-6af09701a70a-kube-api-access-zx77z" pod "8383fa89-7f1f-4500-8f3f-6af09701a70a" (UID: "8383fa89-7f1f-4500-8f3f-6af09701a70a"). InnerVolumeSpecName "kube-api-access-zx77z". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 00:50:23.935540 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:50:23.935528 4939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8383fa89-7f1f-4500-8f3f-6af09701a70a-metrics-certs" pod "8383fa89-7f1f-4500-8f3f-6af09701a70a" (UID: "8383fa89-7f1f-4500-8f3f-6af09701a70a"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 00:50:23.935726 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:50:23.935711 4939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8383fa89-7f1f-4500-8f3f-6af09701a70a-maas-api-tls" pod "8383fa89-7f1f-4500-8f3f-6af09701a70a" (UID: "8383fa89-7f1f-4500-8f3f-6af09701a70a"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 00:50:24.025632 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:50:24.025600 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 00:50:24.050153 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:50:24.050102 4939 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8383fa89-7f1f-4500-8f3f-6af09701a70a-metrics-certs\") on node \"ip-10-0-140-107.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 00:50:24.050153 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:50:24.050147 4939 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8383fa89-7f1f-4500-8f3f-6af09701a70a-maas-api-tls\") on node \"ip-10-0-140-107.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 00:50:24.050402 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:50:24.050165 4939 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zx77z\" (UniqueName: \"kubernetes.io/projected/8383fa89-7f1f-4500-8f3f-6af09701a70a-kube-api-access-zx77z\") on node \"ip-10-0-140-107.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 00:50:24.554168 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:50:24.554129 4939 generic.go:356] "Generic (PLEG): container finished" podID="8383fa89-7f1f-4500-8f3f-6af09701a70a" containerID="3ac02630ec183f6aaaac065a614780e328cf0eee5efc5b7c4c0f3bf730f1243b" exitCode=0 Nov 03 00:50:24.554610 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:50:24.554170 4939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-331bcaa1-7596774966-n5kcr" event={"ID":"8383fa89-7f1f-4500-8f3f-6af09701a70a","Type":"ContainerDied","Data":"3ac02630ec183f6aaaac065a614780e328cf0eee5efc5b7c4c0f3bf730f1243b"} Nov 03 00:50:24.554610 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:50:24.554200 4939 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-331bcaa1-7596774966-n5kcr" Nov 03 00:50:24.554610 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:50:24.554239 4939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-331bcaa1-7596774966-n5kcr" event={"ID":"8383fa89-7f1f-4500-8f3f-6af09701a70a","Type":"ContainerDied","Data":"defe59decf0aa8bfda0a4159b103b4292c53f3582782dd8da2ceeb4b643518ff"} Nov 03 00:50:24.554610 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:50:24.554269 4939 scope.go:122] "RemoveContainer" containerID="3ac02630ec183f6aaaac065a614780e328cf0eee5efc5b7c4c0f3bf730f1243b" Nov 03 00:50:24.567814 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:50:24.567796 4939 scope.go:122] "RemoveContainer" containerID="3ac02630ec183f6aaaac065a614780e328cf0eee5efc5b7c4c0f3bf730f1243b" Nov 03 00:50:24.568229 ip-10-0-140-107 kubenswrapper[4939]: E1103 00:50:24.568207 4939 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3ac02630ec183f6aaaac065a614780e328cf0eee5efc5b7c4c0f3bf730f1243b\": container with ID starting with 3ac02630ec183f6aaaac065a614780e328cf0eee5efc5b7c4c0f3bf730f1243b not found: ID does not exist" containerID="3ac02630ec183f6aaaac065a614780e328cf0eee5efc5b7c4c0f3bf730f1243b" Nov 03 00:50:24.568349 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:50:24.568235 4939 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3ac02630ec183f6aaaac065a614780e328cf0eee5efc5b7c4c0f3bf730f1243b"} err="failed to get container status \"3ac02630ec183f6aaaac065a614780e328cf0eee5efc5b7c4c0f3bf730f1243b\": rpc error: code = NotFound desc = could not find container \"3ac02630ec183f6aaaac065a614780e328cf0eee5efc5b7c4c0f3bf730f1243b\": container with ID starting with 3ac02630ec183f6aaaac065a614780e328cf0eee5efc5b7c4c0f3bf730f1243b not found: ID does not exist" Nov 03 00:50:24.587689 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:50:24.587660 4939 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-331bcaa1-7596774966-n5kcr"] Nov 03 00:50:24.599517 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:50:24.599491 4939 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-331bcaa1-7596774966-n5kcr"] Nov 03 00:50:24.963320 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:50:24.963285 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 00:50:25.383924 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:50:25.381702 4939 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-331bcaa1-openshift-default-86476dcd55-7z4mx"] Nov 03 00:50:25.383924 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:50:25.381976 4939 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ait-331bcaa1-openshift-default-86476dcd55-7z4mx" podUID="f710f312-c05c-4028-92ec-c4636998926f" containerName="istio-proxy" containerID="cri-o://8dbe55abae5c74cdb23cb4c060bbc4a67c97f3fce9e56af3a86b1d538b39a717" gracePeriod=30 Nov 03 00:50:25.441121 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:50:25.441087 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 00:50:25.550382 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:50:25.550345 4939 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-e9b9924e-79d64d74c5-r6l2z"] Nov 03 00:50:25.563662 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:50:25.563620 4939 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-e9b9924e-5646b597cf-cgg6n"] Nov 03 00:50:25.651235 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:50:25.651156 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 00:50:25.718200 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:50:25.718164 4939 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-4pw7v"] Nov 03 00:50:25.769237 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:50:25.769206 4939 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-15abbf02-85bb978ffc-q9krm"] Nov 03 00:50:25.790938 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:50:25.790894 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 00:50:25.867387 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:50:25.867351 4939 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8383fa89-7f1f-4500-8f3f-6af09701a70a" path="/var/lib/kubelet/pods/8383fa89-7f1f-4500-8f3f-6af09701a70a/volumes" Nov 03 00:50:25.917551 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:50:25.917473 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 00:50:26.271011 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:50:26.270927 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 00:50:26.431195 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:50:26.431148 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 00:50:26.611678 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:50:26.611645 4939 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-650be0-openshift-default-5c7c5fb7d9-tkzdb"] Nov 03 00:50:26.612202 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:50:26.612173 4939 cpu_manager.go:414] "containerMap: removing container" podUID="8383fa89-7f1f-4500-8f3f-6af09701a70a" containerName="maas-api" Nov 03 00:50:26.612202 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:50:26.612197 4939 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8383fa89-7f1f-4500-8f3f-6af09701a70a" containerName="maas-api" Nov 03 00:50:26.612445 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:50:26.612363 4939 memory_manager.go:365] "RemoveStaleState removing state" podUID="8383fa89-7f1f-4500-8f3f-6af09701a70a" containerName="maas-api" Nov 03 00:50:26.616502 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:50:26.616479 4939 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-650be0-openshift-default-5c7c5fb7d9-tkzdb" Nov 03 00:50:26.621621 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:50:26.621603 4939 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w4-650be0-openshift-default-dockercfg-l27n6\"" Nov 03 00:50:26.632722 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:50:26.632694 4939 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-650be0-openshift-default-5c7c5fb7d9-tkzdb"] Nov 03 00:50:26.676438 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:50:26.676402 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4da03a65-9706-40b4-955f-6e628901b6be-istiod-ca-cert\") pod \"e2e-worker-w4-650be0-openshift-default-5c7c5fb7d9-tkzdb\" (UID: \"4da03a65-9706-40b4-955f-6e628901b6be\") " pod="openshift-ingress/e2e-worker-w4-650be0-openshift-default-5c7c5fb7d9-tkzdb" Nov 03 00:50:26.676633 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:50:26.676454 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4da03a65-9706-40b4-955f-6e628901b6be-workload-socket\") pod \"e2e-worker-w4-650be0-openshift-default-5c7c5fb7d9-tkzdb\" (UID: \"4da03a65-9706-40b4-955f-6e628901b6be\") " pod="openshift-ingress/e2e-worker-w4-650be0-openshift-default-5c7c5fb7d9-tkzdb" Nov 03 00:50:26.676633 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:50:26.676594 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4da03a65-9706-40b4-955f-6e628901b6be-credential-socket\") pod \"e2e-worker-w4-650be0-openshift-default-5c7c5fb7d9-tkzdb\" (UID: \"4da03a65-9706-40b4-955f-6e628901b6be\") " pod="openshift-ingress/e2e-worker-w4-650be0-openshift-default-5c7c5fb7d9-tkzdb" Nov 03 00:50:26.676740 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:50:26.676635 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4da03a65-9706-40b4-955f-6e628901b6be-istio-podinfo\") pod \"e2e-worker-w4-650be0-openshift-default-5c7c5fb7d9-tkzdb\" (UID: \"4da03a65-9706-40b4-955f-6e628901b6be\") " pod="openshift-ingress/e2e-worker-w4-650be0-openshift-default-5c7c5fb7d9-tkzdb" Nov 03 00:50:26.676740 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:50:26.676666 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4da03a65-9706-40b4-955f-6e628901b6be-istio-envoy\") pod \"e2e-worker-w4-650be0-openshift-default-5c7c5fb7d9-tkzdb\" (UID: \"4da03a65-9706-40b4-955f-6e628901b6be\") " pod="openshift-ingress/e2e-worker-w4-650be0-openshift-default-5c7c5fb7d9-tkzdb" Nov 03 00:50:26.676740 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:50:26.676713 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4da03a65-9706-40b4-955f-6e628901b6be-istio-data\") pod \"e2e-worker-w4-650be0-openshift-default-5c7c5fb7d9-tkzdb\" (UID: \"4da03a65-9706-40b4-955f-6e628901b6be\") " pod="openshift-ingress/e2e-worker-w4-650be0-openshift-default-5c7c5fb7d9-tkzdb" Nov 03 00:50:26.677117 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:50:26.676745 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4da03a65-9706-40b4-955f-6e628901b6be-istio-token\") pod \"e2e-worker-w4-650be0-openshift-default-5c7c5fb7d9-tkzdb\" (UID: \"4da03a65-9706-40b4-955f-6e628901b6be\") " pod="openshift-ingress/e2e-worker-w4-650be0-openshift-default-5c7c5fb7d9-tkzdb" Nov 03 00:50:26.677117 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:50:26.676763 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4da03a65-9706-40b4-955f-6e628901b6be-workload-certs\") pod \"e2e-worker-w4-650be0-openshift-default-5c7c5fb7d9-tkzdb\" (UID: \"4da03a65-9706-40b4-955f-6e628901b6be\") " pod="openshift-ingress/e2e-worker-w4-650be0-openshift-default-5c7c5fb7d9-tkzdb" Nov 03 00:50:26.677117 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:50:26.676796 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7lbj6\" (UniqueName: \"kubernetes.io/projected/4da03a65-9706-40b4-955f-6e628901b6be-kube-api-access-7lbj6\") pod \"e2e-worker-w4-650be0-openshift-default-5c7c5fb7d9-tkzdb\" (UID: \"4da03a65-9706-40b4-955f-6e628901b6be\") " pod="openshift-ingress/e2e-worker-w4-650be0-openshift-default-5c7c5fb7d9-tkzdb" Nov 03 00:50:26.683578 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:50:26.683556 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 00:50:26.778205 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:50:26.778170 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7lbj6\" (UniqueName: \"kubernetes.io/projected/4da03a65-9706-40b4-955f-6e628901b6be-kube-api-access-7lbj6\") pod \"e2e-worker-w4-650be0-openshift-default-5c7c5fb7d9-tkzdb\" (UID: \"4da03a65-9706-40b4-955f-6e628901b6be\") " pod="openshift-ingress/e2e-worker-w4-650be0-openshift-default-5c7c5fb7d9-tkzdb" Nov 03 00:50:26.778413 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:50:26.778229 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4da03a65-9706-40b4-955f-6e628901b6be-istiod-ca-cert\") pod \"e2e-worker-w4-650be0-openshift-default-5c7c5fb7d9-tkzdb\" (UID: \"4da03a65-9706-40b4-955f-6e628901b6be\") " pod="openshift-ingress/e2e-worker-w4-650be0-openshift-default-5c7c5fb7d9-tkzdb" Nov 03 00:50:26.778413 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:50:26.778269 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4da03a65-9706-40b4-955f-6e628901b6be-workload-socket\") pod \"e2e-worker-w4-650be0-openshift-default-5c7c5fb7d9-tkzdb\" (UID: \"4da03a65-9706-40b4-955f-6e628901b6be\") " pod="openshift-ingress/e2e-worker-w4-650be0-openshift-default-5c7c5fb7d9-tkzdb" Nov 03 00:50:26.778413 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:50:26.778314 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4da03a65-9706-40b4-955f-6e628901b6be-credential-socket\") pod \"e2e-worker-w4-650be0-openshift-default-5c7c5fb7d9-tkzdb\" (UID: \"4da03a65-9706-40b4-955f-6e628901b6be\") " pod="openshift-ingress/e2e-worker-w4-650be0-openshift-default-5c7c5fb7d9-tkzdb" Nov 03 00:50:26.778413 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:50:26.778333 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4da03a65-9706-40b4-955f-6e628901b6be-istio-podinfo\") pod \"e2e-worker-w4-650be0-openshift-default-5c7c5fb7d9-tkzdb\" (UID: \"4da03a65-9706-40b4-955f-6e628901b6be\") " pod="openshift-ingress/e2e-worker-w4-650be0-openshift-default-5c7c5fb7d9-tkzdb" Nov 03 00:50:26.778413 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:50:26.778353 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4da03a65-9706-40b4-955f-6e628901b6be-istio-envoy\") pod \"e2e-worker-w4-650be0-openshift-default-5c7c5fb7d9-tkzdb\" (UID: \"4da03a65-9706-40b4-955f-6e628901b6be\") " pod="openshift-ingress/e2e-worker-w4-650be0-openshift-default-5c7c5fb7d9-tkzdb" Nov 03 00:50:26.778413 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:50:26.778380 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4da03a65-9706-40b4-955f-6e628901b6be-istio-data\") pod \"e2e-worker-w4-650be0-openshift-default-5c7c5fb7d9-tkzdb\" (UID: \"4da03a65-9706-40b4-955f-6e628901b6be\") " pod="openshift-ingress/e2e-worker-w4-650be0-openshift-default-5c7c5fb7d9-tkzdb" Nov 03 00:50:26.778840 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:50:26.778435 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4da03a65-9706-40b4-955f-6e628901b6be-istio-token\") pod \"e2e-worker-w4-650be0-openshift-default-5c7c5fb7d9-tkzdb\" (UID: \"4da03a65-9706-40b4-955f-6e628901b6be\") " pod="openshift-ingress/e2e-worker-w4-650be0-openshift-default-5c7c5fb7d9-tkzdb" Nov 03 00:50:26.778840 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:50:26.778748 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4da03a65-9706-40b4-955f-6e628901b6be-workload-certs\") pod \"e2e-worker-w4-650be0-openshift-default-5c7c5fb7d9-tkzdb\" (UID: \"4da03a65-9706-40b4-955f-6e628901b6be\") " pod="openshift-ingress/e2e-worker-w4-650be0-openshift-default-5c7c5fb7d9-tkzdb" Nov 03 00:50:26.778974 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:50:26.778847 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4da03a65-9706-40b4-955f-6e628901b6be-workload-socket\") pod \"e2e-worker-w4-650be0-openshift-default-5c7c5fb7d9-tkzdb\" (UID: \"4da03a65-9706-40b4-955f-6e628901b6be\") " pod="openshift-ingress/e2e-worker-w4-650be0-openshift-default-5c7c5fb7d9-tkzdb" Nov 03 00:50:26.778974 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:50:26.778880 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4da03a65-9706-40b4-955f-6e628901b6be-credential-socket\") pod \"e2e-worker-w4-650be0-openshift-default-5c7c5fb7d9-tkzdb\" (UID: \"4da03a65-9706-40b4-955f-6e628901b6be\") " pod="openshift-ingress/e2e-worker-w4-650be0-openshift-default-5c7c5fb7d9-tkzdb" Nov 03 00:50:26.779044 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:50:26.778979 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4da03a65-9706-40b4-955f-6e628901b6be-istio-data\") pod \"e2e-worker-w4-650be0-openshift-default-5c7c5fb7d9-tkzdb\" (UID: \"4da03a65-9706-40b4-955f-6e628901b6be\") " pod="openshift-ingress/e2e-worker-w4-650be0-openshift-default-5c7c5fb7d9-tkzdb" Nov 03 00:50:26.779077 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:50:26.779052 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4da03a65-9706-40b4-955f-6e628901b6be-istiod-ca-cert\") pod \"e2e-worker-w4-650be0-openshift-default-5c7c5fb7d9-tkzdb\" (UID: \"4da03a65-9706-40b4-955f-6e628901b6be\") " pod="openshift-ingress/e2e-worker-w4-650be0-openshift-default-5c7c5fb7d9-tkzdb" Nov 03 00:50:26.779110 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:50:26.779094 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4da03a65-9706-40b4-955f-6e628901b6be-workload-certs\") pod \"e2e-worker-w4-650be0-openshift-default-5c7c5fb7d9-tkzdb\" (UID: \"4da03a65-9706-40b4-955f-6e628901b6be\") " pod="openshift-ingress/e2e-worker-w4-650be0-openshift-default-5c7c5fb7d9-tkzdb" Nov 03 00:50:26.781341 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:50:26.781318 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4da03a65-9706-40b4-955f-6e628901b6be-istio-envoy\") pod \"e2e-worker-w4-650be0-openshift-default-5c7c5fb7d9-tkzdb\" (UID: \"4da03a65-9706-40b4-955f-6e628901b6be\") " pod="openshift-ingress/e2e-worker-w4-650be0-openshift-default-5c7c5fb7d9-tkzdb" Nov 03 00:50:26.781733 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:50:26.781714 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4da03a65-9706-40b4-955f-6e628901b6be-istio-podinfo\") pod \"e2e-worker-w4-650be0-openshift-default-5c7c5fb7d9-tkzdb\" (UID: \"4da03a65-9706-40b4-955f-6e628901b6be\") " pod="openshift-ingress/e2e-worker-w4-650be0-openshift-default-5c7c5fb7d9-tkzdb" Nov 03 00:50:26.786819 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:50:26.786797 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4da03a65-9706-40b4-955f-6e628901b6be-istio-token\") pod \"e2e-worker-w4-650be0-openshift-default-5c7c5fb7d9-tkzdb\" (UID: \"4da03a65-9706-40b4-955f-6e628901b6be\") " pod="openshift-ingress/e2e-worker-w4-650be0-openshift-default-5c7c5fb7d9-tkzdb" Nov 03 00:50:26.787146 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:50:26.787126 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7lbj6\" (UniqueName: \"kubernetes.io/projected/4da03a65-9706-40b4-955f-6e628901b6be-kube-api-access-7lbj6\") pod \"e2e-worker-w4-650be0-openshift-default-5c7c5fb7d9-tkzdb\" (UID: \"4da03a65-9706-40b4-955f-6e628901b6be\") " pod="openshift-ingress/e2e-worker-w4-650be0-openshift-default-5c7c5fb7d9-tkzdb" Nov 03 00:50:26.850919 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:50:26.850877 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 00:50:26.937344 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:50:26.937259 4939 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-650be0-openshift-default-5c7c5fb7d9-tkzdb" Nov 03 00:50:27.217672 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:50:27.217647 4939 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-650be0-openshift-default-5c7c5fb7d9-tkzdb"] Nov 03 00:50:27.219596 ip-10-0-140-107 kubenswrapper[4939]: W1103 00:50:27.219566 4939 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4da03a65_9706_40b4_955f_6e628901b6be.slice/crio-5818e364320fa9d73d59c4aabc9bef4eb863b9bee7304a6ef449cdcf823f2d40 WatchSource:0}: Error finding container 5818e364320fa9d73d59c4aabc9bef4eb863b9bee7304a6ef449cdcf823f2d40: Status 404 returned error can't find the container with id 5818e364320fa9d73d59c4aabc9bef4eb863b9bee7304a6ef449cdcf823f2d40 Nov 03 00:50:27.222068 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:50:27.222035 4939 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 00:50:27.222133 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:50:27.222104 4939 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 00:50:27.222169 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:50:27.222135 4939 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 00:50:27.239712 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:50:27.239677 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 00:50:27.576149 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:50:27.576062 4939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-650be0-openshift-default-5c7c5fb7d9-tkzdb" event={"ID":"4da03a65-9706-40b4-955f-6e628901b6be","Type":"ContainerStarted","Data":"b60c499c7f01628148ff6755b3ae78bf771351dcc2d9253dfbd6c45015e55519"} Nov 03 00:50:27.576149 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:50:27.576111 4939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-650be0-openshift-default-5c7c5fb7d9-tkzdb" event={"ID":"4da03a65-9706-40b4-955f-6e628901b6be","Type":"ContainerStarted","Data":"5818e364320fa9d73d59c4aabc9bef4eb863b9bee7304a6ef449cdcf823f2d40"} Nov 03 00:50:27.598753 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:50:27.598707 4939 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-650be0-openshift-default-5c7c5fb7d9-tkzdb" podStartSLOduration=1.5986936790000001 podStartE2EDuration="1.598693679s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:50:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:50:27.595761919 +0000 UTC m=+575.860096867" watchObservedRunningTime="2026-11-03 00:50:27.598693679 +0000 UTC m=+575.863028625" Nov 03 00:50:27.894258 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:50:27.894221 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 00:50:27.938047 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:50:27.938002 4939 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-650be0-openshift-default-5c7c5fb7d9-tkzdb" Nov 03 00:50:28.223925 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:50:28.223827 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 00:50:28.324236 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:50:28.324202 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 00:50:28.449651 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:50:28.449621 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 00:50:28.515542 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:50:28.515457 4939 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-e9b9924e-76848c4649-7d998"] Nov 03 00:50:28.515742 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:50:28.515717 4939 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-e9b9924e-76848c4649-7d998" podUID="9150215d-1d55-4fa1-ad85-056b5bdf1b2c" containerName="maas-api" containerID="cri-o://e58fd89d37806367369ea3103b3ac821c3948d94ae55f352543b898f2da61124" gracePeriod=30 Nov 03 00:50:28.594283 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:50:28.594246 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 00:50:28.753240 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:50:28.751315 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 00:50:28.839329 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:50:28.839300 4939 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-15abbf02-openshift-default-65d8997ff-dtl78"] Nov 03 00:50:28.839510 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:50:28.839489 4939 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ait-15abbf02-openshift-default-65d8997ff-dtl78" podUID="b65bcc4a-131b-422b-b407-207ec0f75f26" containerName="istio-proxy" containerID="cri-o://7069432c5e0f3f92df9063f70cb7588fca184580437788a66e2b7f1cac695f19" gracePeriod=30 Nov 03 00:50:28.848346 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:50:28.848322 4939 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-e9b9924e-76848c4649-7d998" Nov 03 00:50:28.940945 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:50:28.939058 4939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9150215d-1d55-4fa1-ad85-056b5bdf1b2c-kube-api-access-fmxdb\" (UniqueName: \"kubernetes.io/projected/9150215d-1d55-4fa1-ad85-056b5bdf1b2c-kube-api-access-fmxdb\") pod \"9150215d-1d55-4fa1-ad85-056b5bdf1b2c\" (UID: \"9150215d-1d55-4fa1-ad85-056b5bdf1b2c\") " Nov 03 00:50:28.965651 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:50:28.939160 4939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9150215d-1d55-4fa1-ad85-056b5bdf1b2c-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9150215d-1d55-4fa1-ad85-056b5bdf1b2c-openshift-service-ca\") pod \"9150215d-1d55-4fa1-ad85-056b5bdf1b2c\" (UID: \"9150215d-1d55-4fa1-ad85-056b5bdf1b2c\") " Nov 03 00:50:28.965651 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:50:28.958116 4939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9150215d-1d55-4fa1-ad85-056b5bdf1b2c-openshift-service-ca" pod "9150215d-1d55-4fa1-ad85-056b5bdf1b2c" (UID: "9150215d-1d55-4fa1-ad85-056b5bdf1b2c"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 00:50:28.965651 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:50:28.958769 4939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9150215d-1d55-4fa1-ad85-056b5bdf1b2c-metrics-certs\" (UniqueName: \"kubernetes.io/secret/9150215d-1d55-4fa1-ad85-056b5bdf1b2c-metrics-certs\") pod \"9150215d-1d55-4fa1-ad85-056b5bdf1b2c\" (UID: \"9150215d-1d55-4fa1-ad85-056b5bdf1b2c\") " Nov 03 00:50:28.965651 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:50:28.958857 4939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9150215d-1d55-4fa1-ad85-056b5bdf1b2c-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9150215d-1d55-4fa1-ad85-056b5bdf1b2c-maas-api-tls\") pod \"9150215d-1d55-4fa1-ad85-056b5bdf1b2c\" (UID: \"9150215d-1d55-4fa1-ad85-056b5bdf1b2c\") " Nov 03 00:50:28.965651 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:50:28.965297 4939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9150215d-1d55-4fa1-ad85-056b5bdf1b2c-kube-api-access-fmxdb" pod "9150215d-1d55-4fa1-ad85-056b5bdf1b2c" (UID: "9150215d-1d55-4fa1-ad85-056b5bdf1b2c"). InnerVolumeSpecName "kube-api-access-fmxdb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 00:50:28.975952 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:50:28.971819 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 00:50:28.975952 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:50:28.972472 4939 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fmxdb\" (UniqueName: \"kubernetes.io/projected/9150215d-1d55-4fa1-ad85-056b5bdf1b2c-kube-api-access-fmxdb\") on node \"ip-10-0-140-107.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 00:50:28.975952 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:50:28.972508 4939 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9150215d-1d55-4fa1-ad85-056b5bdf1b2c-openshift-service-ca\") on node \"ip-10-0-140-107.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 00:50:28.975952 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:50:28.975591 4939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9150215d-1d55-4fa1-ad85-056b5bdf1b2c-metrics-certs" pod "9150215d-1d55-4fa1-ad85-056b5bdf1b2c" (UID: "9150215d-1d55-4fa1-ad85-056b5bdf1b2c"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 00:50:28.976310 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:50:28.976093 4939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9150215d-1d55-4fa1-ad85-056b5bdf1b2c-maas-api-tls" pod "9150215d-1d55-4fa1-ad85-056b5bdf1b2c" (UID: "9150215d-1d55-4fa1-ad85-056b5bdf1b2c"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 00:50:28.976310 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:50:28.976184 4939 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-650be0-openshift-default-5c7c5fb7d9-tkzdb" Nov 03 00:50:29.073510 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:50:29.073473 4939 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9150215d-1d55-4fa1-ad85-056b5bdf1b2c-metrics-certs\") on node \"ip-10-0-140-107.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 00:50:29.073510 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:50:29.073503 4939 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9150215d-1d55-4fa1-ad85-056b5bdf1b2c-maas-api-tls\") on node \"ip-10-0-140-107.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 00:50:29.080708 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:50:29.080637 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 00:50:29.224793 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:50:29.224749 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 00:50:29.587325 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:50:29.587290 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 00:50:29.588143 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:50:29.588120 4939 generic.go:356] "Generic (PLEG): container finished" podID="9150215d-1d55-4fa1-ad85-056b5bdf1b2c" containerID="e58fd89d37806367369ea3103b3ac821c3948d94ae55f352543b898f2da61124" exitCode=0 Nov 03 00:50:29.588258 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:50:29.588174 4939 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-e9b9924e-76848c4649-7d998" Nov 03 00:50:29.588258 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:50:29.588203 4939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-e9b9924e-76848c4649-7d998" event={"ID":"9150215d-1d55-4fa1-ad85-056b5bdf1b2c","Type":"ContainerDied","Data":"e58fd89d37806367369ea3103b3ac821c3948d94ae55f352543b898f2da61124"} Nov 03 00:50:29.588373 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:50:29.588290 4939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-e9b9924e-76848c4649-7d998" event={"ID":"9150215d-1d55-4fa1-ad85-056b5bdf1b2c","Type":"ContainerDied","Data":"1c20504083f55c3a15b390fec34cbfbfcd3ca0b004cc6373706d44633b9aed83"} Nov 03 00:50:29.588373 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:50:29.588327 4939 scope.go:122] "RemoveContainer" containerID="e58fd89d37806367369ea3103b3ac821c3948d94ae55f352543b898f2da61124" Nov 03 00:50:29.588589 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:50:29.588563 4939 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-650be0-openshift-default-5c7c5fb7d9-tkzdb" Nov 03 00:50:29.589807 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:50:29.589766 4939 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-650be0-openshift-default-5c7c5fb7d9-tkzdb" Nov 03 00:50:29.598797 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:50:29.598775 4939 scope.go:122] "RemoveContainer" containerID="e58fd89d37806367369ea3103b3ac821c3948d94ae55f352543b898f2da61124" Nov 03 00:50:29.599082 ip-10-0-140-107 kubenswrapper[4939]: E1103 00:50:29.599062 4939 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e58fd89d37806367369ea3103b3ac821c3948d94ae55f352543b898f2da61124\": container with ID starting with e58fd89d37806367369ea3103b3ac821c3948d94ae55f352543b898f2da61124 not found: ID does not exist" containerID="e58fd89d37806367369ea3103b3ac821c3948d94ae55f352543b898f2da61124" Nov 03 00:50:29.599155 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:50:29.599089 4939 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e58fd89d37806367369ea3103b3ac821c3948d94ae55f352543b898f2da61124"} err="failed to get container status \"e58fd89d37806367369ea3103b3ac821c3948d94ae55f352543b898f2da61124\": rpc error: code = NotFound desc = could not find container \"e58fd89d37806367369ea3103b3ac821c3948d94ae55f352543b898f2da61124\": container with ID starting with e58fd89d37806367369ea3103b3ac821c3948d94ae55f352543b898f2da61124 not found: ID does not exist" Nov 03 00:50:29.634847 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:50:29.633995 4939 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-e9b9924e-76848c4649-7d998"] Nov 03 00:50:29.637671 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:50:29.637643 4939 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-e9b9924e-76848c4649-7d998"] Nov 03 00:50:29.762208 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:50:29.762175 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 00:50:29.868971 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:50:29.868873 4939 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9150215d-1d55-4fa1-ad85-056b5bdf1b2c" path="/var/lib/kubelet/pods/9150215d-1d55-4fa1-ad85-056b5bdf1b2c/volumes" Nov 03 00:50:30.001027 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:50:30.000996 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 00:50:30.139817 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:50:30.139719 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 00:50:30.271614 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:50:30.271580 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 00:50:30.407690 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:50:30.407661 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 00:50:30.466551 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:50:30.466519 4939 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-83755a-openshift-default-c86f9565c-shrff"] Nov 03 00:50:30.467024 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:50:30.467005 4939 cpu_manager.go:414] "containerMap: removing container" podUID="9150215d-1d55-4fa1-ad85-056b5bdf1b2c" containerName="maas-api" Nov 03 00:50:30.467024 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:50:30.467022 4939 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9150215d-1d55-4fa1-ad85-056b5bdf1b2c" containerName="maas-api" Nov 03 00:50:30.467196 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:50:30.467110 4939 memory_manager.go:365] "RemoveStaleState removing state" podUID="9150215d-1d55-4fa1-ad85-056b5bdf1b2c" containerName="maas-api" Nov 03 00:50:30.470536 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:50:30.470513 4939 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-83755a-openshift-default-c86f9565c-shrff" Nov 03 00:50:30.483060 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:50:30.483033 4939 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w3-83755a-openshift-default-dockercfg-qhtrs\"" Nov 03 00:50:30.487674 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:50:30.487649 4939 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-83755a-openshift-default-c86f9565c-shrff"] Nov 03 00:50:30.539654 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:50:30.539622 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 00:50:30.561321 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:50:30.560983 4939 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-migrate-e9b9924e-79d64d74c5-r6l2z" podUID="2366920b-4d83-4d4b-a83d-15e9abf48aac" containerName="payload-processing" containerID="cri-o://a8f476211eef1abd28f3e66059ae6516c321b95a4f6a58f261ef4f3a05f6d1ca" gracePeriod=25 Nov 03 00:50:30.568696 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:50:30.568009 4939 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-migrate-e9b9924e-5646b597cf-cgg6n" podUID="761d585c-40b1-40c8-b499-6abd8689a794" containerName="payload-pre-processing" containerID="cri-o://5e8c60632dc6ab481ec4073115b06c3e2f83a8edfcfc05a98bfc8b44a6ce7a7a" gracePeriod=25 Nov 03 00:50:30.592304 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:50:30.590009 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4nvwn\" (UniqueName: \"kubernetes.io/projected/30435a8a-8666-4929-b02e-02a6bb02d878-kube-api-access-4nvwn\") pod \"e2e-worker-w3-83755a-openshift-default-c86f9565c-shrff\" (UID: \"30435a8a-8666-4929-b02e-02a6bb02d878\") " pod="openshift-ingress/e2e-worker-w3-83755a-openshift-default-c86f9565c-shrff" Nov 03 00:50:30.592304 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:50:30.590068 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/30435a8a-8666-4929-b02e-02a6bb02d878-workload-certs\") pod \"e2e-worker-w3-83755a-openshift-default-c86f9565c-shrff\" (UID: \"30435a8a-8666-4929-b02e-02a6bb02d878\") " pod="openshift-ingress/e2e-worker-w3-83755a-openshift-default-c86f9565c-shrff" Nov 03 00:50:30.592304 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:50:30.590100 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/30435a8a-8666-4929-b02e-02a6bb02d878-workload-socket\") pod \"e2e-worker-w3-83755a-openshift-default-c86f9565c-shrff\" (UID: \"30435a8a-8666-4929-b02e-02a6bb02d878\") " pod="openshift-ingress/e2e-worker-w3-83755a-openshift-default-c86f9565c-shrff" Nov 03 00:50:30.592304 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:50:30.590142 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/30435a8a-8666-4929-b02e-02a6bb02d878-istiod-ca-cert\") pod \"e2e-worker-w3-83755a-openshift-default-c86f9565c-shrff\" (UID: \"30435a8a-8666-4929-b02e-02a6bb02d878\") " pod="openshift-ingress/e2e-worker-w3-83755a-openshift-default-c86f9565c-shrff" Nov 03 00:50:30.592304 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:50:30.590178 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/30435a8a-8666-4929-b02e-02a6bb02d878-istio-envoy\") pod \"e2e-worker-w3-83755a-openshift-default-c86f9565c-shrff\" (UID: \"30435a8a-8666-4929-b02e-02a6bb02d878\") " pod="openshift-ingress/e2e-worker-w3-83755a-openshift-default-c86f9565c-shrff" Nov 03 00:50:30.592304 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:50:30.590208 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/30435a8a-8666-4929-b02e-02a6bb02d878-istio-podinfo\") pod \"e2e-worker-w3-83755a-openshift-default-c86f9565c-shrff\" (UID: \"30435a8a-8666-4929-b02e-02a6bb02d878\") " pod="openshift-ingress/e2e-worker-w3-83755a-openshift-default-c86f9565c-shrff" Nov 03 00:50:30.592304 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:50:30.590269 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/30435a8a-8666-4929-b02e-02a6bb02d878-istio-token\") pod \"e2e-worker-w3-83755a-openshift-default-c86f9565c-shrff\" (UID: \"30435a8a-8666-4929-b02e-02a6bb02d878\") " pod="openshift-ingress/e2e-worker-w3-83755a-openshift-default-c86f9565c-shrff" Nov 03 00:50:30.592304 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:50:30.590296 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/30435a8a-8666-4929-b02e-02a6bb02d878-istio-data\") pod \"e2e-worker-w3-83755a-openshift-default-c86f9565c-shrff\" (UID: \"30435a8a-8666-4929-b02e-02a6bb02d878\") " pod="openshift-ingress/e2e-worker-w3-83755a-openshift-default-c86f9565c-shrff" Nov 03 00:50:30.592304 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:50:30.590333 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/30435a8a-8666-4929-b02e-02a6bb02d878-credential-socket\") pod \"e2e-worker-w3-83755a-openshift-default-c86f9565c-shrff\" (UID: \"30435a8a-8666-4929-b02e-02a6bb02d878\") " pod="openshift-ingress/e2e-worker-w3-83755a-openshift-default-c86f9565c-shrff" Nov 03 00:50:30.620225 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:50:30.619468 4939 generic.go:356] "Generic (PLEG): container finished" podID="f710f312-c05c-4028-92ec-c4636998926f" containerID="8dbe55abae5c74cdb23cb4c060bbc4a67c97f3fce9e56af3a86b1d538b39a717" exitCode=0 Nov 03 00:50:30.620225 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:50:30.619870 4939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-331bcaa1-openshift-default-86476dcd55-7z4mx" event={"ID":"f710f312-c05c-4028-92ec-c4636998926f","Type":"ContainerDied","Data":"8dbe55abae5c74cdb23cb4c060bbc4a67c97f3fce9e56af3a86b1d538b39a717"} Nov 03 00:50:30.620225 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:50:30.620141 4939 scope.go:122] "RemoveContainer" containerID="42c34511bef545ccb1cfae72aa8efc91dae542368483c76d90c98c619ae89366" Nov 03 00:50:30.677988 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:50:30.677008 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 00:50:30.691364 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:50:30.691338 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/30435a8a-8666-4929-b02e-02a6bb02d878-workload-certs\") pod \"e2e-worker-w3-83755a-openshift-default-c86f9565c-shrff\" (UID: \"30435a8a-8666-4929-b02e-02a6bb02d878\") " pod="openshift-ingress/e2e-worker-w3-83755a-openshift-default-c86f9565c-shrff" Nov 03 00:50:30.691485 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:50:30.691380 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/30435a8a-8666-4929-b02e-02a6bb02d878-workload-socket\") pod \"e2e-worker-w3-83755a-openshift-default-c86f9565c-shrff\" (UID: \"30435a8a-8666-4929-b02e-02a6bb02d878\") " pod="openshift-ingress/e2e-worker-w3-83755a-openshift-default-c86f9565c-shrff" Nov 03 00:50:30.691485 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:50:30.691434 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/30435a8a-8666-4929-b02e-02a6bb02d878-istiod-ca-cert\") pod \"e2e-worker-w3-83755a-openshift-default-c86f9565c-shrff\" (UID: \"30435a8a-8666-4929-b02e-02a6bb02d878\") " pod="openshift-ingress/e2e-worker-w3-83755a-openshift-default-c86f9565c-shrff" Nov 03 00:50:30.691485 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:50:30.691474 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/30435a8a-8666-4929-b02e-02a6bb02d878-istio-envoy\") pod \"e2e-worker-w3-83755a-openshift-default-c86f9565c-shrff\" (UID: \"30435a8a-8666-4929-b02e-02a6bb02d878\") " pod="openshift-ingress/e2e-worker-w3-83755a-openshift-default-c86f9565c-shrff" Nov 03 00:50:30.691651 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:50:30.691506 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/30435a8a-8666-4929-b02e-02a6bb02d878-istio-podinfo\") pod \"e2e-worker-w3-83755a-openshift-default-c86f9565c-shrff\" (UID: \"30435a8a-8666-4929-b02e-02a6bb02d878\") " pod="openshift-ingress/e2e-worker-w3-83755a-openshift-default-c86f9565c-shrff" Nov 03 00:50:30.691651 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:50:30.691572 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/30435a8a-8666-4929-b02e-02a6bb02d878-istio-token\") pod \"e2e-worker-w3-83755a-openshift-default-c86f9565c-shrff\" (UID: \"30435a8a-8666-4929-b02e-02a6bb02d878\") " pod="openshift-ingress/e2e-worker-w3-83755a-openshift-default-c86f9565c-shrff" Nov 03 00:50:30.691651 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:50:30.691599 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/30435a8a-8666-4929-b02e-02a6bb02d878-istio-data\") pod \"e2e-worker-w3-83755a-openshift-default-c86f9565c-shrff\" (UID: \"30435a8a-8666-4929-b02e-02a6bb02d878\") " pod="openshift-ingress/e2e-worker-w3-83755a-openshift-default-c86f9565c-shrff" Nov 03 00:50:30.691651 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:50:30.691639 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/30435a8a-8666-4929-b02e-02a6bb02d878-credential-socket\") pod \"e2e-worker-w3-83755a-openshift-default-c86f9565c-shrff\" (UID: \"30435a8a-8666-4929-b02e-02a6bb02d878\") " pod="openshift-ingress/e2e-worker-w3-83755a-openshift-default-c86f9565c-shrff" Nov 03 00:50:30.691848 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:50:30.691766 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/30435a8a-8666-4929-b02e-02a6bb02d878-workload-certs\") pod \"e2e-worker-w3-83755a-openshift-default-c86f9565c-shrff\" (UID: \"30435a8a-8666-4929-b02e-02a6bb02d878\") " pod="openshift-ingress/e2e-worker-w3-83755a-openshift-default-c86f9565c-shrff" Nov 03 00:50:30.691848 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:50:30.691768 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4nvwn\" (UniqueName: \"kubernetes.io/projected/30435a8a-8666-4929-b02e-02a6bb02d878-kube-api-access-4nvwn\") pod \"e2e-worker-w3-83755a-openshift-default-c86f9565c-shrff\" (UID: \"30435a8a-8666-4929-b02e-02a6bb02d878\") " pod="openshift-ingress/e2e-worker-w3-83755a-openshift-default-c86f9565c-shrff" Nov 03 00:50:30.692249 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:50:30.692230 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/30435a8a-8666-4929-b02e-02a6bb02d878-istiod-ca-cert\") pod \"e2e-worker-w3-83755a-openshift-default-c86f9565c-shrff\" (UID: \"30435a8a-8666-4929-b02e-02a6bb02d878\") " pod="openshift-ingress/e2e-worker-w3-83755a-openshift-default-c86f9565c-shrff" Nov 03 00:50:30.692428 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:50:30.692405 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/30435a8a-8666-4929-b02e-02a6bb02d878-istio-data\") pod \"e2e-worker-w3-83755a-openshift-default-c86f9565c-shrff\" (UID: \"30435a8a-8666-4929-b02e-02a6bb02d878\") " pod="openshift-ingress/e2e-worker-w3-83755a-openshift-default-c86f9565c-shrff" Nov 03 00:50:30.692492 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:50:30.692435 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/30435a8a-8666-4929-b02e-02a6bb02d878-credential-socket\") pod \"e2e-worker-w3-83755a-openshift-default-c86f9565c-shrff\" (UID: \"30435a8a-8666-4929-b02e-02a6bb02d878\") " pod="openshift-ingress/e2e-worker-w3-83755a-openshift-default-c86f9565c-shrff" Nov 03 00:50:30.692608 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:50:30.692589 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/30435a8a-8666-4929-b02e-02a6bb02d878-workload-socket\") pod \"e2e-worker-w3-83755a-openshift-default-c86f9565c-shrff\" (UID: \"30435a8a-8666-4929-b02e-02a6bb02d878\") " pod="openshift-ingress/e2e-worker-w3-83755a-openshift-default-c86f9565c-shrff" Nov 03 00:50:30.694292 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:50:30.694274 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/30435a8a-8666-4929-b02e-02a6bb02d878-istio-envoy\") pod \"e2e-worker-w3-83755a-openshift-default-c86f9565c-shrff\" (UID: \"30435a8a-8666-4929-b02e-02a6bb02d878\") " pod="openshift-ingress/e2e-worker-w3-83755a-openshift-default-c86f9565c-shrff" Nov 03 00:50:30.694863 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:50:30.694844 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/30435a8a-8666-4929-b02e-02a6bb02d878-istio-podinfo\") pod \"e2e-worker-w3-83755a-openshift-default-c86f9565c-shrff\" (UID: \"30435a8a-8666-4929-b02e-02a6bb02d878\") " pod="openshift-ingress/e2e-worker-w3-83755a-openshift-default-c86f9565c-shrff" Nov 03 00:50:30.700208 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:50:30.700180 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/30435a8a-8666-4929-b02e-02a6bb02d878-istio-token\") pod \"e2e-worker-w3-83755a-openshift-default-c86f9565c-shrff\" (UID: \"30435a8a-8666-4929-b02e-02a6bb02d878\") " pod="openshift-ingress/e2e-worker-w3-83755a-openshift-default-c86f9565c-shrff" Nov 03 00:50:30.700627 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:50:30.700608 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4nvwn\" (UniqueName: \"kubernetes.io/projected/30435a8a-8666-4929-b02e-02a6bb02d878-kube-api-access-4nvwn\") pod \"e2e-worker-w3-83755a-openshift-default-c86f9565c-shrff\" (UID: \"30435a8a-8666-4929-b02e-02a6bb02d878\") " pod="openshift-ingress/e2e-worker-w3-83755a-openshift-default-c86f9565c-shrff" Nov 03 00:50:30.736652 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:50:30.736617 4939 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-331bcaa1-openshift-default-86476dcd55-7z4mx" Nov 03 00:50:30.769639 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:50:30.769475 4939 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ait-15abbf02-85bb978ffc-q9krm" podUID="006cb224-99e3-4d1b-90ec-e1a0e73ae181" containerName="payload-processing" containerID="cri-o://c9b11043e7d8dac79abbbb1de86d90a55625b07a178eaf615137c7d84b67fe07" gracePeriod=25 Nov 03 00:50:30.791551 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:50:30.791469 4939 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-83755a-openshift-default-c86f9565c-shrff" Nov 03 00:50:30.836030 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:50:30.835949 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 00:50:30.901707 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:50:30.901521 4939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f710f312-c05c-4028-92ec-c4636998926f-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f710f312-c05c-4028-92ec-c4636998926f-istiod-ca-cert\") pod \"f710f312-c05c-4028-92ec-c4636998926f\" (UID: \"f710f312-c05c-4028-92ec-c4636998926f\") " Nov 03 00:50:30.901707 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:50:30.901578 4939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f710f312-c05c-4028-92ec-c4636998926f-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f710f312-c05c-4028-92ec-c4636998926f-credential-socket\") pod \"f710f312-c05c-4028-92ec-c4636998926f\" (UID: \"f710f312-c05c-4028-92ec-c4636998926f\") " Nov 03 00:50:30.901707 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:50:30.901624 4939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f710f312-c05c-4028-92ec-c4636998926f-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f710f312-c05c-4028-92ec-c4636998926f-istio-envoy\") pod \"f710f312-c05c-4028-92ec-c4636998926f\" (UID: \"f710f312-c05c-4028-92ec-c4636998926f\") " Nov 03 00:50:30.906218 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:50:30.905512 4939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f710f312-c05c-4028-92ec-c4636998926f-credential-socket" pod "f710f312-c05c-4028-92ec-c4636998926f" (UID: "f710f312-c05c-4028-92ec-c4636998926f"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 00:50:30.906218 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:50:30.905973 4939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f710f312-c05c-4028-92ec-c4636998926f-istiod-ca-cert" pod "f710f312-c05c-4028-92ec-c4636998926f" (UID: "f710f312-c05c-4028-92ec-c4636998926f"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 00:50:30.924092 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:50:30.923095 4939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/f710f312-c05c-4028-92ec-c4636998926f-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f710f312-c05c-4028-92ec-c4636998926f-istio-podinfo\") pod \"f710f312-c05c-4028-92ec-c4636998926f\" (UID: \"f710f312-c05c-4028-92ec-c4636998926f\") " Nov 03 00:50:30.924092 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:50:30.923161 4939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f710f312-c05c-4028-92ec-c4636998926f-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f710f312-c05c-4028-92ec-c4636998926f-workload-certs\") pod \"f710f312-c05c-4028-92ec-c4636998926f\" (UID: \"f710f312-c05c-4028-92ec-c4636998926f\") " Nov 03 00:50:30.924092 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:50:30.923207 4939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f710f312-c05c-4028-92ec-c4636998926f-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f710f312-c05c-4028-92ec-c4636998926f-workload-socket\") pod \"f710f312-c05c-4028-92ec-c4636998926f\" (UID: \"f710f312-c05c-4028-92ec-c4636998926f\") " Nov 03 00:50:30.924092 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:50:30.923275 4939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f710f312-c05c-4028-92ec-c4636998926f-istio-token\" (UniqueName: \"kubernetes.io/projected/f710f312-c05c-4028-92ec-c4636998926f-istio-token\") pod \"f710f312-c05c-4028-92ec-c4636998926f\" (UID: \"f710f312-c05c-4028-92ec-c4636998926f\") " Nov 03 00:50:30.924092 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:50:30.923320 4939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f710f312-c05c-4028-92ec-c4636998926f-kube-api-access-wd4w8\" (UniqueName: \"kubernetes.io/projected/f710f312-c05c-4028-92ec-c4636998926f-kube-api-access-wd4w8\") pod \"f710f312-c05c-4028-92ec-c4636998926f\" (UID: \"f710f312-c05c-4028-92ec-c4636998926f\") " Nov 03 00:50:30.924092 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:50:30.923393 4939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f710f312-c05c-4028-92ec-c4636998926f-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f710f312-c05c-4028-92ec-c4636998926f-istio-data\") pod \"f710f312-c05c-4028-92ec-c4636998926f\" (UID: \"f710f312-c05c-4028-92ec-c4636998926f\") " Nov 03 00:50:30.924092 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:50:30.923818 4939 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f710f312-c05c-4028-92ec-c4636998926f-istiod-ca-cert\") on node \"ip-10-0-140-107.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 00:50:30.924092 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:50:30.923847 4939 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f710f312-c05c-4028-92ec-c4636998926f-credential-socket\") on node \"ip-10-0-140-107.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 00:50:30.924548 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:50:30.924135 4939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f710f312-c05c-4028-92ec-c4636998926f-workload-socket" pod "f710f312-c05c-4028-92ec-c4636998926f" (UID: "f710f312-c05c-4028-92ec-c4636998926f"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 00:50:30.926970 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:50:30.926930 4939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f710f312-c05c-4028-92ec-c4636998926f-istio-data" pod "f710f312-c05c-4028-92ec-c4636998926f" (UID: "f710f312-c05c-4028-92ec-c4636998926f"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 00:50:30.935619 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:50:30.935592 4939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f710f312-c05c-4028-92ec-c4636998926f-workload-certs" pod "f710f312-c05c-4028-92ec-c4636998926f" (UID: "f710f312-c05c-4028-92ec-c4636998926f"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 00:50:30.936129 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:50:30.936001 4939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/f710f312-c05c-4028-92ec-c4636998926f-istio-podinfo" pod "f710f312-c05c-4028-92ec-c4636998926f" (UID: "f710f312-c05c-4028-92ec-c4636998926f"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 00:50:30.942174 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:50:30.942141 4939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f710f312-c05c-4028-92ec-c4636998926f-istio-envoy" pod "f710f312-c05c-4028-92ec-c4636998926f" (UID: "f710f312-c05c-4028-92ec-c4636998926f"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 00:50:30.943285 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:50:30.943215 4939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f710f312-c05c-4028-92ec-c4636998926f-kube-api-access-wd4w8" pod "f710f312-c05c-4028-92ec-c4636998926f" (UID: "f710f312-c05c-4028-92ec-c4636998926f"). InnerVolumeSpecName "kube-api-access-wd4w8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 00:50:30.943998 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:50:30.943575 4939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f710f312-c05c-4028-92ec-c4636998926f-istio-token" pod "f710f312-c05c-4028-92ec-c4636998926f" (UID: "f710f312-c05c-4028-92ec-c4636998926f"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 00:50:30.948256 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:50:30.948186 4939 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-migrate-e9b9924e-5646b597cf-cgg6n" Nov 03 00:50:30.956877 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:50:30.956846 4939 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-migrate-e9b9924e-79d64d74c5-r6l2z" Nov 03 00:50:31.024902 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:50:31.024868 4939 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f710f312-c05c-4028-92ec-c4636998926f-workload-socket\") on node \"ip-10-0-140-107.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 00:50:31.027987 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:50:31.027962 4939 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f710f312-c05c-4028-92ec-c4636998926f-istio-token\") on node \"ip-10-0-140-107.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 00:50:31.028098 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:50:31.027994 4939 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wd4w8\" (UniqueName: \"kubernetes.io/projected/f710f312-c05c-4028-92ec-c4636998926f-kube-api-access-wd4w8\") on node \"ip-10-0-140-107.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 00:50:31.028098 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:50:31.028006 4939 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f710f312-c05c-4028-92ec-c4636998926f-istio-data\") on node \"ip-10-0-140-107.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 00:50:31.028098 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:50:31.028015 4939 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f710f312-c05c-4028-92ec-c4636998926f-istio-envoy\") on node \"ip-10-0-140-107.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 00:50:31.028098 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:50:31.028025 4939 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f710f312-c05c-4028-92ec-c4636998926f-istio-podinfo\") on node \"ip-10-0-140-107.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 00:50:31.028098 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:50:31.028035 4939 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f710f312-c05c-4028-92ec-c4636998926f-workload-certs\") on node \"ip-10-0-140-107.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 00:50:31.110144 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:50:31.110118 4939 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-83755a-openshift-default-c86f9565c-shrff"] Nov 03 00:50:31.112640 ip-10-0-140-107 kubenswrapper[4939]: W1103 00:50:31.112610 4939 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod30435a8a_8666_4929_b02e_02a6bb02d878.slice/crio-c52fd75c0012a45c1cfc0e40f217fbf71fa6ca5ad8f0fbed96dce072f1f6014a WatchSource:0}: Error finding container c52fd75c0012a45c1cfc0e40f217fbf71fa6ca5ad8f0fbed96dce072f1f6014a: Status 404 returned error can't find the container with id c52fd75c0012a45c1cfc0e40f217fbf71fa6ca5ad8f0fbed96dce072f1f6014a Nov 03 00:50:31.115343 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:50:31.115308 4939 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 00:50:31.115426 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:50:31.115398 4939 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 00:50:31.115499 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:50:31.115447 4939 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 00:50:31.129070 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:50:31.129048 4939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/761d585c-40b1-40c8-b499-6abd8689a794-kube-api-access-5vglb\" (UniqueName: \"kubernetes.io/projected/761d585c-40b1-40c8-b499-6abd8689a794-kube-api-access-5vglb\") pod \"761d585c-40b1-40c8-b499-6abd8689a794\" (UID: \"761d585c-40b1-40c8-b499-6abd8689a794\") " Nov 03 00:50:31.129156 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:50:31.129129 4939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2366920b-4d83-4d4b-a83d-15e9abf48aac-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2366920b-4d83-4d4b-a83d-15e9abf48aac-plugins-config-volume\") pod \"2366920b-4d83-4d4b-a83d-15e9abf48aac\" (UID: \"2366920b-4d83-4d4b-a83d-15e9abf48aac\") " Nov 03 00:50:31.129207 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:50:31.129177 4939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2366920b-4d83-4d4b-a83d-15e9abf48aac-kube-api-access-wj2l4\" (UniqueName: \"kubernetes.io/projected/2366920b-4d83-4d4b-a83d-15e9abf48aac-kube-api-access-wj2l4\") pod \"2366920b-4d83-4d4b-a83d-15e9abf48aac\" (UID: \"2366920b-4d83-4d4b-a83d-15e9abf48aac\") " Nov 03 00:50:31.129264 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:50:31.129205 4939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/761d585c-40b1-40c8-b499-6abd8689a794-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/761d585c-40b1-40c8-b499-6abd8689a794-plugins-config-volume\") pod \"761d585c-40b1-40c8-b499-6abd8689a794\" (UID: \"761d585c-40b1-40c8-b499-6abd8689a794\") " Nov 03 00:50:31.129865 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:50:31.129837 4939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/761d585c-40b1-40c8-b499-6abd8689a794-plugins-config-volume" pod "761d585c-40b1-40c8-b499-6abd8689a794" (UID: "761d585c-40b1-40c8-b499-6abd8689a794"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 00:50:31.129978 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:50:31.129888 4939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2366920b-4d83-4d4b-a83d-15e9abf48aac-plugins-config-volume" pod "2366920b-4d83-4d4b-a83d-15e9abf48aac" (UID: "2366920b-4d83-4d4b-a83d-15e9abf48aac"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 00:50:31.131799 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:50:31.131776 4939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/761d585c-40b1-40c8-b499-6abd8689a794-kube-api-access-5vglb" pod "761d585c-40b1-40c8-b499-6abd8689a794" (UID: "761d585c-40b1-40c8-b499-6abd8689a794"). InnerVolumeSpecName "kube-api-access-5vglb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 00:50:31.131885 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:50:31.131812 4939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2366920b-4d83-4d4b-a83d-15e9abf48aac-kube-api-access-wj2l4" pod "2366920b-4d83-4d4b-a83d-15e9abf48aac" (UID: "2366920b-4d83-4d4b-a83d-15e9abf48aac"). InnerVolumeSpecName "kube-api-access-wj2l4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 00:50:31.140790 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:50:31.140769 4939 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-15abbf02-85bb978ffc-q9krm" Nov 03 00:50:31.225636 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:50:31.225599 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 00:50:31.246036 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:50:31.245265 4939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/006cb224-99e3-4d1b-90ec-e1a0e73ae181-kube-api-access-dgrpp\" (UniqueName: \"kubernetes.io/projected/006cb224-99e3-4d1b-90ec-e1a0e73ae181-kube-api-access-dgrpp\") pod \"006cb224-99e3-4d1b-90ec-e1a0e73ae181\" (UID: \"006cb224-99e3-4d1b-90ec-e1a0e73ae181\") " Nov 03 00:50:31.246036 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:50:31.245368 4939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/006cb224-99e3-4d1b-90ec-e1a0e73ae181-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/006cb224-99e3-4d1b-90ec-e1a0e73ae181-plugins-config-volume\") pod \"006cb224-99e3-4d1b-90ec-e1a0e73ae181\" (UID: \"006cb224-99e3-4d1b-90ec-e1a0e73ae181\") " Nov 03 00:50:31.246036 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:50:31.245721 4939 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2366920b-4d83-4d4b-a83d-15e9abf48aac-plugins-config-volume\") on node \"ip-10-0-140-107.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 00:50:31.246036 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:50:31.245740 4939 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wj2l4\" (UniqueName: \"kubernetes.io/projected/2366920b-4d83-4d4b-a83d-15e9abf48aac-kube-api-access-wj2l4\") on node \"ip-10-0-140-107.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 00:50:31.246036 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:50:31.245756 4939 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/761d585c-40b1-40c8-b499-6abd8689a794-plugins-config-volume\") on node \"ip-10-0-140-107.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 00:50:31.246036 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:50:31.245770 4939 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5vglb\" (UniqueName: \"kubernetes.io/projected/761d585c-40b1-40c8-b499-6abd8689a794-kube-api-access-5vglb\") on node \"ip-10-0-140-107.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 00:50:31.246994 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:50:31.246644 4939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/006cb224-99e3-4d1b-90ec-e1a0e73ae181-plugins-config-volume" pod "006cb224-99e3-4d1b-90ec-e1a0e73ae181" (UID: "006cb224-99e3-4d1b-90ec-e1a0e73ae181"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 00:50:31.250320 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:50:31.250276 4939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/006cb224-99e3-4d1b-90ec-e1a0e73ae181-kube-api-access-dgrpp" pod "006cb224-99e3-4d1b-90ec-e1a0e73ae181" (UID: "006cb224-99e3-4d1b-90ec-e1a0e73ae181"). InnerVolumeSpecName "kube-api-access-dgrpp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 00:50:31.347007 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:50:31.346975 4939 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dgrpp\" (UniqueName: \"kubernetes.io/projected/006cb224-99e3-4d1b-90ec-e1a0e73ae181-kube-api-access-dgrpp\") on node \"ip-10-0-140-107.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 00:50:31.347007 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:50:31.347004 4939 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/006cb224-99e3-4d1b-90ec-e1a0e73ae181-plugins-config-volume\") on node \"ip-10-0-140-107.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 00:50:31.524042 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:50:31.524007 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 00:50:31.537888 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:50:31.537861 4939 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w4-650be0-58fc9b6455-9jl7f"] Nov 03 00:50:31.538334 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:50:31.538320 4939 cpu_manager.go:414] "containerMap: removing container" podUID="f710f312-c05c-4028-92ec-c4636998926f" containerName="istio-proxy" Nov 03 00:50:31.538378 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:50:31.538336 4939 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f710f312-c05c-4028-92ec-c4636998926f" containerName="istio-proxy" Nov 03 00:50:31.538378 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:50:31.538359 4939 cpu_manager.go:414] "containerMap: removing container" podUID="2366920b-4d83-4d4b-a83d-15e9abf48aac" containerName="payload-processing" Nov 03 00:50:31.538378 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:50:31.538367 4939 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2366920b-4d83-4d4b-a83d-15e9abf48aac" containerName="payload-processing" Nov 03 00:50:31.538468 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:50:31.538380 4939 cpu_manager.go:414] "containerMap: removing container" podUID="006cb224-99e3-4d1b-90ec-e1a0e73ae181" containerName="payload-processing" Nov 03 00:50:31.538468 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:50:31.538385 4939 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="006cb224-99e3-4d1b-90ec-e1a0e73ae181" containerName="payload-processing" Nov 03 00:50:31.538468 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:50:31.538401 4939 cpu_manager.go:414] "containerMap: removing container" podUID="761d585c-40b1-40c8-b499-6abd8689a794" containerName="payload-pre-processing" Nov 03 00:50:31.538468 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:50:31.538406 4939 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="761d585c-40b1-40c8-b499-6abd8689a794" containerName="payload-pre-processing" Nov 03 00:50:31.538468 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:50:31.538419 4939 cpu_manager.go:414] "containerMap: removing container" podUID="f710f312-c05c-4028-92ec-c4636998926f" containerName="istio-proxy" Nov 03 00:50:31.538468 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:50:31.538424 4939 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f710f312-c05c-4028-92ec-c4636998926f" containerName="istio-proxy" Nov 03 00:50:31.538634 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:50:31.538488 4939 memory_manager.go:365] "RemoveStaleState removing state" podUID="2366920b-4d83-4d4b-a83d-15e9abf48aac" containerName="payload-processing" Nov 03 00:50:31.538634 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:50:31.538498 4939 memory_manager.go:365] "RemoveStaleState removing state" podUID="006cb224-99e3-4d1b-90ec-e1a0e73ae181" containerName="payload-processing" Nov 03 00:50:31.538634 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:50:31.538505 4939 memory_manager.go:365] "RemoveStaleState removing state" podUID="761d585c-40b1-40c8-b499-6abd8689a794" containerName="payload-pre-processing" Nov 03 00:50:31.538634 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:50:31.538514 4939 memory_manager.go:365] "RemoveStaleState removing state" podUID="f710f312-c05c-4028-92ec-c4636998926f" containerName="istio-proxy" Nov 03 00:50:31.538634 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:50:31.538525 4939 memory_manager.go:365] "RemoveStaleState removing state" podUID="f710f312-c05c-4028-92ec-c4636998926f" containerName="istio-proxy" Nov 03 00:50:31.541871 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:50:31.541855 4939 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-650be0-58fc9b6455-9jl7f" Nov 03 00:50:31.550238 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:50:31.550199 4939 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w4-650be0\"" Nov 03 00:50:31.553773 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:50:31.553751 4939 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w4-650be0-58fc9b6455-9jl7f"] Nov 03 00:50:31.605010 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:50:31.604970 4939 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w4-650be0-86d95657b7-vm62r"] Nov 03 00:50:31.608654 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:50:31.608629 4939 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w4-650be0-86d95657b7-vm62r" Nov 03 00:50:31.614788 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:50:31.614764 4939 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w4-650be0-dockercfg-9qf2l\"" Nov 03 00:50:31.617246 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:50:31.617219 4939 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w4-650be0-86d95657b7-vm62r"] Nov 03 00:50:31.643356 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:50:31.643316 4939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-83755a-openshift-default-c86f9565c-shrff" event={"ID":"30435a8a-8666-4929-b02e-02a6bb02d878","Type":"ContainerStarted","Data":"84a6e26ec7fe983ad1671971374185c5be4b64c87d96563859ff05d313517926"} Nov 03 00:50:31.643511 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:50:31.643415 4939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-83755a-openshift-default-c86f9565c-shrff" event={"ID":"30435a8a-8666-4929-b02e-02a6bb02d878","Type":"ContainerStarted","Data":"c52fd75c0012a45c1cfc0e40f217fbf71fa6ca5ad8f0fbed96dce072f1f6014a"} Nov 03 00:50:31.646121 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:50:31.646095 4939 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-331bcaa1-openshift-default-86476dcd55-7z4mx" Nov 03 00:50:31.646366 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:50:31.646099 4939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-331bcaa1-openshift-default-86476dcd55-7z4mx" event={"ID":"f710f312-c05c-4028-92ec-c4636998926f","Type":"ContainerDied","Data":"1d53322cd2d2072e81df9b898e6b4ea3c880c30d4b300eaa07b3ece5ef4839ee"} Nov 03 00:50:31.646484 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:50:31.646416 4939 scope.go:122] "RemoveContainer" containerID="8dbe55abae5c74cdb23cb4c060bbc4a67c97f3fce9e56af3a86b1d538b39a717" Nov 03 00:50:31.647396 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:50:31.647369 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 00:50:31.648760 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:50:31.648735 4939 generic.go:356] "Generic (PLEG): container finished" podID="761d585c-40b1-40c8-b499-6abd8689a794" containerID="5e8c60632dc6ab481ec4073115b06c3e2f83a8edfcfc05a98bfc8b44a6ce7a7a" exitCode=0 Nov 03 00:50:31.648993 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:50:31.648832 4939 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-migrate-e9b9924e-5646b597cf-cgg6n" Nov 03 00:50:31.649464 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:50:31.648847 4939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-migrate-e9b9924e-5646b597cf-cgg6n" event={"ID":"761d585c-40b1-40c8-b499-6abd8689a794","Type":"ContainerDied","Data":"5e8c60632dc6ab481ec4073115b06c3e2f83a8edfcfc05a98bfc8b44a6ce7a7a"} Nov 03 00:50:31.649585 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:50:31.649543 4939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-migrate-e9b9924e-5646b597cf-cgg6n" event={"ID":"761d585c-40b1-40c8-b499-6abd8689a794","Type":"ContainerDied","Data":"b34e1aa1e944bc23e611d989425b2e1edc317e8377cfc42f81daf5695c913b1d"} Nov 03 00:50:31.650793 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:50:31.650747 4939 generic.go:356] "Generic (PLEG): container finished" podID="006cb224-99e3-4d1b-90ec-e1a0e73ae181" containerID="c9b11043e7d8dac79abbbb1de86d90a55625b07a178eaf615137c7d84b67fe07" exitCode=0 Nov 03 00:50:31.650878 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:50:31.650794 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ppcrv\" (UniqueName: \"kubernetes.io/projected/c72f9b11-1218-4c3f-be7c-6541feeb1ed2-kube-api-access-ppcrv\") pod \"payload-pre-processing-e2e-worker-w4-650be0-58fc9b6455-9jl7f\" (UID: \"c72f9b11-1218-4c3f-be7c-6541feeb1ed2\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-650be0-58fc9b6455-9jl7f" Nov 03 00:50:31.650878 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:50:31.650856 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qnlh8\" (UniqueName: \"kubernetes.io/projected/58875207-cd7a-4da9-aee7-f538e7f44963-kube-api-access-qnlh8\") pod \"payload-processing-e2e-worker-w4-650be0-86d95657b7-vm62r\" (UID: \"58875207-cd7a-4da9-aee7-f538e7f44963\") " pod="openshift-ingress/payload-processing-e2e-worker-w4-650be0-86d95657b7-vm62r" Nov 03 00:50:31.650878 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:50:31.650863 4939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-15abbf02-85bb978ffc-q9krm" event={"ID":"006cb224-99e3-4d1b-90ec-e1a0e73ae181","Type":"ContainerDied","Data":"c9b11043e7d8dac79abbbb1de86d90a55625b07a178eaf615137c7d84b67fe07"} Nov 03 00:50:31.651075 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:50:31.650929 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/58875207-cd7a-4da9-aee7-f538e7f44963-plugins-config-volume\") pod \"payload-processing-e2e-worker-w4-650be0-86d95657b7-vm62r\" (UID: \"58875207-cd7a-4da9-aee7-f538e7f44963\") " pod="openshift-ingress/payload-processing-e2e-worker-w4-650be0-86d95657b7-vm62r" Nov 03 00:50:31.651075 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:50:31.650901 4939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-15abbf02-85bb978ffc-q9krm" event={"ID":"006cb224-99e3-4d1b-90ec-e1a0e73ae181","Type":"ContainerDied","Data":"c5e2dd32d871f51a111a3f4c592af5aad7036439ff3460e48259f2f3df245b51"} Nov 03 00:50:31.651075 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:50:31.651007 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c72f9b11-1218-4c3f-be7c-6541feeb1ed2-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w4-650be0-58fc9b6455-9jl7f\" (UID: \"c72f9b11-1218-4c3f-be7c-6541feeb1ed2\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-650be0-58fc9b6455-9jl7f" Nov 03 00:50:31.651075 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:50:31.651054 4939 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-15abbf02-85bb978ffc-q9krm" Nov 03 00:50:31.654171 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:50:31.654146 4939 generic.go:356] "Generic (PLEG): container finished" podID="2366920b-4d83-4d4b-a83d-15e9abf48aac" containerID="a8f476211eef1abd28f3e66059ae6516c321b95a4f6a58f261ef4f3a05f6d1ca" exitCode=0 Nov 03 00:50:31.676015 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:50:31.654838 4939 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-migrate-e9b9924e-79d64d74c5-r6l2z" Nov 03 00:50:31.676015 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:50:31.655972 4939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-migrate-e9b9924e-79d64d74c5-r6l2z" event={"ID":"2366920b-4d83-4d4b-a83d-15e9abf48aac","Type":"ContainerDied","Data":"a8f476211eef1abd28f3e66059ae6516c321b95a4f6a58f261ef4f3a05f6d1ca"} Nov 03 00:50:31.676015 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:50:31.656017 4939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-migrate-e9b9924e-79d64d74c5-r6l2z" event={"ID":"2366920b-4d83-4d4b-a83d-15e9abf48aac","Type":"ContainerDied","Data":"1f71433883ce3d93a22bd04b2f444074e686ea255d93e46d41fa61fa02dce64a"} Nov 03 00:50:31.676015 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:50:31.660096 4939 scope.go:122] "RemoveContainer" containerID="5e8c60632dc6ab481ec4073115b06c3e2f83a8edfcfc05a98bfc8b44a6ce7a7a" Nov 03 00:50:31.676015 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:50:31.670596 4939 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-83755a-openshift-default-c86f9565c-shrff" podStartSLOduration=1.6705773750000001 podStartE2EDuration="1.670577375s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:50:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:50:31.665085211 +0000 UTC m=+579.929420157" watchObservedRunningTime="2026-11-03 00:50:31.670577375 +0000 UTC m=+579.934912321" Nov 03 00:50:31.699378 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:50:31.694991 4939 scope.go:122] "RemoveContainer" containerID="5e8c60632dc6ab481ec4073115b06c3e2f83a8edfcfc05a98bfc8b44a6ce7a7a" Nov 03 00:50:31.706502 ip-10-0-140-107 kubenswrapper[4939]: E1103 00:50:31.706452 4939 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5e8c60632dc6ab481ec4073115b06c3e2f83a8edfcfc05a98bfc8b44a6ce7a7a\": container with ID starting with 5e8c60632dc6ab481ec4073115b06c3e2f83a8edfcfc05a98bfc8b44a6ce7a7a not found: ID does not exist" containerID="5e8c60632dc6ab481ec4073115b06c3e2f83a8edfcfc05a98bfc8b44a6ce7a7a" Nov 03 00:50:31.710255 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:50:31.710219 4939 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5e8c60632dc6ab481ec4073115b06c3e2f83a8edfcfc05a98bfc8b44a6ce7a7a"} err="failed to get container status \"5e8c60632dc6ab481ec4073115b06c3e2f83a8edfcfc05a98bfc8b44a6ce7a7a\": rpc error: code = NotFound desc = could not find container \"5e8c60632dc6ab481ec4073115b06c3e2f83a8edfcfc05a98bfc8b44a6ce7a7a\": container with ID starting with 5e8c60632dc6ab481ec4073115b06c3e2f83a8edfcfc05a98bfc8b44a6ce7a7a not found: ID does not exist" Nov 03 00:50:31.712556 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:50:31.712532 4939 scope.go:122] "RemoveContainer" containerID="c9b11043e7d8dac79abbbb1de86d90a55625b07a178eaf615137c7d84b67fe07" Nov 03 00:50:31.726995 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:50:31.726144 4939 scope.go:122] "RemoveContainer" containerID="c9b11043e7d8dac79abbbb1de86d90a55625b07a178eaf615137c7d84b67fe07" Nov 03 00:50:31.726995 ip-10-0-140-107 kubenswrapper[4939]: E1103 00:50:31.726549 4939 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c9b11043e7d8dac79abbbb1de86d90a55625b07a178eaf615137c7d84b67fe07\": container with ID starting with c9b11043e7d8dac79abbbb1de86d90a55625b07a178eaf615137c7d84b67fe07 not found: ID does not exist" containerID="c9b11043e7d8dac79abbbb1de86d90a55625b07a178eaf615137c7d84b67fe07" Nov 03 00:50:31.726995 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:50:31.726584 4939 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c9b11043e7d8dac79abbbb1de86d90a55625b07a178eaf615137c7d84b67fe07"} err="failed to get container status \"c9b11043e7d8dac79abbbb1de86d90a55625b07a178eaf615137c7d84b67fe07\": rpc error: code = NotFound desc = could not find container \"c9b11043e7d8dac79abbbb1de86d90a55625b07a178eaf615137c7d84b67fe07\": container with ID starting with c9b11043e7d8dac79abbbb1de86d90a55625b07a178eaf615137c7d84b67fe07 not found: ID does not exist" Nov 03 00:50:31.726995 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:50:31.726608 4939 scope.go:122] "RemoveContainer" containerID="a8f476211eef1abd28f3e66059ae6516c321b95a4f6a58f261ef4f3a05f6d1ca" Nov 03 00:50:31.732699 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:50:31.732677 4939 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-e9b9924e-5646b597cf-cgg6n"] Nov 03 00:50:31.742275 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:50:31.742248 4939 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-e9b9924e-5646b597cf-cgg6n"] Nov 03 00:50:31.751865 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:50:31.751827 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c72f9b11-1218-4c3f-be7c-6541feeb1ed2-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w4-650be0-58fc9b6455-9jl7f\" (UID: \"c72f9b11-1218-4c3f-be7c-6541feeb1ed2\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-650be0-58fc9b6455-9jl7f" Nov 03 00:50:31.752012 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:50:31.751902 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ppcrv\" (UniqueName: \"kubernetes.io/projected/c72f9b11-1218-4c3f-be7c-6541feeb1ed2-kube-api-access-ppcrv\") pod \"payload-pre-processing-e2e-worker-w4-650be0-58fc9b6455-9jl7f\" (UID: \"c72f9b11-1218-4c3f-be7c-6541feeb1ed2\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-650be0-58fc9b6455-9jl7f" Nov 03 00:50:31.752012 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:50:31.751997 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qnlh8\" (UniqueName: \"kubernetes.io/projected/58875207-cd7a-4da9-aee7-f538e7f44963-kube-api-access-qnlh8\") pod \"payload-processing-e2e-worker-w4-650be0-86d95657b7-vm62r\" (UID: \"58875207-cd7a-4da9-aee7-f538e7f44963\") " pod="openshift-ingress/payload-processing-e2e-worker-w4-650be0-86d95657b7-vm62r" Nov 03 00:50:31.752109 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:50:31.752098 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/58875207-cd7a-4da9-aee7-f538e7f44963-plugins-config-volume\") pod \"payload-processing-e2e-worker-w4-650be0-86d95657b7-vm62r\" (UID: \"58875207-cd7a-4da9-aee7-f538e7f44963\") " pod="openshift-ingress/payload-processing-e2e-worker-w4-650be0-86d95657b7-vm62r" Nov 03 00:50:31.752581 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:50:31.752552 4939 scope.go:122] "RemoveContainer" containerID="a8f476211eef1abd28f3e66059ae6516c321b95a4f6a58f261ef4f3a05f6d1ca" Nov 03 00:50:31.753016 ip-10-0-140-107 kubenswrapper[4939]: E1103 00:50:31.752994 4939 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a8f476211eef1abd28f3e66059ae6516c321b95a4f6a58f261ef4f3a05f6d1ca\": container with ID starting with a8f476211eef1abd28f3e66059ae6516c321b95a4f6a58f261ef4f3a05f6d1ca not found: ID does not exist" containerID="a8f476211eef1abd28f3e66059ae6516c321b95a4f6a58f261ef4f3a05f6d1ca" Nov 03 00:50:31.753115 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:50:31.753026 4939 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a8f476211eef1abd28f3e66059ae6516c321b95a4f6a58f261ef4f3a05f6d1ca"} err="failed to get container status \"a8f476211eef1abd28f3e66059ae6516c321b95a4f6a58f261ef4f3a05f6d1ca\": rpc error: code = NotFound desc = could not find container \"a8f476211eef1abd28f3e66059ae6516c321b95a4f6a58f261ef4f3a05f6d1ca\": container with ID starting with a8f476211eef1abd28f3e66059ae6516c321b95a4f6a58f261ef4f3a05f6d1ca not found: ID does not exist" Nov 03 00:50:31.753115 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:50:31.753073 4939 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-331bcaa1-openshift-default-86476dcd55-7z4mx"] Nov 03 00:50:31.753600 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:50:31.753574 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c72f9b11-1218-4c3f-be7c-6541feeb1ed2-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w4-650be0-58fc9b6455-9jl7f\" (UID: \"c72f9b11-1218-4c3f-be7c-6541feeb1ed2\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-650be0-58fc9b6455-9jl7f" Nov 03 00:50:31.753659 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:50:31.753642 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/58875207-cd7a-4da9-aee7-f538e7f44963-plugins-config-volume\") pod \"payload-processing-e2e-worker-w4-650be0-86d95657b7-vm62r\" (UID: \"58875207-cd7a-4da9-aee7-f538e7f44963\") " pod="openshift-ingress/payload-processing-e2e-worker-w4-650be0-86d95657b7-vm62r" Nov 03 00:50:31.754880 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:50:31.754861 4939 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ait-331bcaa1-openshift-default-86476dcd55-7z4mx"] Nov 03 00:50:31.759983 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:50:31.759962 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ppcrv\" (UniqueName: \"kubernetes.io/projected/c72f9b11-1218-4c3f-be7c-6541feeb1ed2-kube-api-access-ppcrv\") pod \"payload-pre-processing-e2e-worker-w4-650be0-58fc9b6455-9jl7f\" (UID: \"c72f9b11-1218-4c3f-be7c-6541feeb1ed2\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-650be0-58fc9b6455-9jl7f" Nov 03 00:50:31.760132 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:50:31.760114 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qnlh8\" (UniqueName: \"kubernetes.io/projected/58875207-cd7a-4da9-aee7-f538e7f44963-kube-api-access-qnlh8\") pod \"payload-processing-e2e-worker-w4-650be0-86d95657b7-vm62r\" (UID: \"58875207-cd7a-4da9-aee7-f538e7f44963\") " pod="openshift-ingress/payload-processing-e2e-worker-w4-650be0-86d95657b7-vm62r" Nov 03 00:50:31.764064 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:50:31.764044 4939 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-e9b9924e-79d64d74c5-r6l2z"] Nov 03 00:50:31.766739 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:50:31.766721 4939 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-e9b9924e-79d64d74c5-r6l2z"] Nov 03 00:50:31.769058 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:50:31.769039 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 00:50:31.791795 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:50:31.791756 4939 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-83755a-openshift-default-c86f9565c-shrff" Nov 03 00:50:31.792920 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:50:31.792873 4939 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-15abbf02-85bb978ffc-q9krm"] Nov 03 00:50:31.796192 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:50:31.795298 4939 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-15abbf02-85bb978ffc-q9krm"] Nov 03 00:50:31.856804 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:50:31.856769 4939 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-650be0-58fc9b6455-9jl7f" Nov 03 00:50:31.867558 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:50:31.867531 4939 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="006cb224-99e3-4d1b-90ec-e1a0e73ae181" path="/var/lib/kubelet/pods/006cb224-99e3-4d1b-90ec-e1a0e73ae181/volumes" Nov 03 00:50:31.867933 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:50:31.867893 4939 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2366920b-4d83-4d4b-a83d-15e9abf48aac" path="/var/lib/kubelet/pods/2366920b-4d83-4d4b-a83d-15e9abf48aac/volumes" Nov 03 00:50:31.868278 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:50:31.868259 4939 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="761d585c-40b1-40c8-b499-6abd8689a794" path="/var/lib/kubelet/pods/761d585c-40b1-40c8-b499-6abd8689a794/volumes" Nov 03 00:50:31.868963 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:50:31.868946 4939 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f710f312-c05c-4028-92ec-c4636998926f" path="/var/lib/kubelet/pods/f710f312-c05c-4028-92ec-c4636998926f/volumes" Nov 03 00:50:31.930017 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:50:31.929950 4939 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w4-650be0-86d95657b7-vm62r" Nov 03 00:50:32.025683 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:50:32.025649 4939 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-2f9a3a-openshift-default-7f6f74cf97-dxffl"] Nov 03 00:50:32.030621 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:50:32.030598 4939 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-2f9a3a-openshift-default-7f6f74cf97-dxffl" Nov 03 00:50:32.047825 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:50:32.047734 4939 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-2f9a3a-openshift-default-7f6f74cf97-dxffl"] Nov 03 00:50:32.066722 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:50:32.066692 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/80e0f915-1a1a-4214-92f0-5eeb806be96e-credential-socket\") pod \"e2e-worker-w0-2f9a3a-openshift-default-7f6f74cf97-dxffl\" (UID: \"80e0f915-1a1a-4214-92f0-5eeb806be96e\") " pod="openshift-ingress/e2e-worker-w0-2f9a3a-openshift-default-7f6f74cf97-dxffl" Nov 03 00:50:32.066896 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:50:32.066742 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/80e0f915-1a1a-4214-92f0-5eeb806be96e-workload-certs\") pod \"e2e-worker-w0-2f9a3a-openshift-default-7f6f74cf97-dxffl\" (UID: \"80e0f915-1a1a-4214-92f0-5eeb806be96e\") " pod="openshift-ingress/e2e-worker-w0-2f9a3a-openshift-default-7f6f74cf97-dxffl" Nov 03 00:50:32.066896 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:50:32.066811 4939 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w0-2f9a3a-openshift-default-dockercfg-gdpt5\"" Nov 03 00:50:32.066896 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:50:32.066827 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/80e0f915-1a1a-4214-92f0-5eeb806be96e-workload-socket\") pod \"e2e-worker-w0-2f9a3a-openshift-default-7f6f74cf97-dxffl\" (UID: \"80e0f915-1a1a-4214-92f0-5eeb806be96e\") " pod="openshift-ingress/e2e-worker-w0-2f9a3a-openshift-default-7f6f74cf97-dxffl" Nov 03 00:50:32.067114 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:50:32.066957 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/80e0f915-1a1a-4214-92f0-5eeb806be96e-istiod-ca-cert\") pod \"e2e-worker-w0-2f9a3a-openshift-default-7f6f74cf97-dxffl\" (UID: \"80e0f915-1a1a-4214-92f0-5eeb806be96e\") " pod="openshift-ingress/e2e-worker-w0-2f9a3a-openshift-default-7f6f74cf97-dxffl" Nov 03 00:50:32.067114 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:50:32.067034 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/80e0f915-1a1a-4214-92f0-5eeb806be96e-istio-data\") pod \"e2e-worker-w0-2f9a3a-openshift-default-7f6f74cf97-dxffl\" (UID: \"80e0f915-1a1a-4214-92f0-5eeb806be96e\") " pod="openshift-ingress/e2e-worker-w0-2f9a3a-openshift-default-7f6f74cf97-dxffl" Nov 03 00:50:32.067114 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:50:32.067071 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2qzls\" (UniqueName: \"kubernetes.io/projected/80e0f915-1a1a-4214-92f0-5eeb806be96e-kube-api-access-2qzls\") pod \"e2e-worker-w0-2f9a3a-openshift-default-7f6f74cf97-dxffl\" (UID: \"80e0f915-1a1a-4214-92f0-5eeb806be96e\") " pod="openshift-ingress/e2e-worker-w0-2f9a3a-openshift-default-7f6f74cf97-dxffl" Nov 03 00:50:32.067114 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:50:32.067111 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/80e0f915-1a1a-4214-92f0-5eeb806be96e-istio-envoy\") pod \"e2e-worker-w0-2f9a3a-openshift-default-7f6f74cf97-dxffl\" (UID: \"80e0f915-1a1a-4214-92f0-5eeb806be96e\") " pod="openshift-ingress/e2e-worker-w0-2f9a3a-openshift-default-7f6f74cf97-dxffl" Nov 03 00:50:32.067311 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:50:32.067165 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/80e0f915-1a1a-4214-92f0-5eeb806be96e-istio-token\") pod \"e2e-worker-w0-2f9a3a-openshift-default-7f6f74cf97-dxffl\" (UID: \"80e0f915-1a1a-4214-92f0-5eeb806be96e\") " pod="openshift-ingress/e2e-worker-w0-2f9a3a-openshift-default-7f6f74cf97-dxffl" Nov 03 00:50:32.067311 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:50:32.067212 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/80e0f915-1a1a-4214-92f0-5eeb806be96e-istio-podinfo\") pod \"e2e-worker-w0-2f9a3a-openshift-default-7f6f74cf97-dxffl\" (UID: \"80e0f915-1a1a-4214-92f0-5eeb806be96e\") " pod="openshift-ingress/e2e-worker-w0-2f9a3a-openshift-default-7f6f74cf97-dxffl" Nov 03 00:50:32.099543 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:50:32.099513 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 00:50:32.138337 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:50:32.138312 4939 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w4-650be0-58fc9b6455-9jl7f"] Nov 03 00:50:32.139985 ip-10-0-140-107 kubenswrapper[4939]: W1103 00:50:32.139956 4939 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc72f9b11_1218_4c3f_be7c_6541feeb1ed2.slice/crio-f464f01b23f98abff545fc1f7bf400e86eb8bf6b808deb40742362525b3e0004 WatchSource:0}: Error finding container f464f01b23f98abff545fc1f7bf400e86eb8bf6b808deb40742362525b3e0004: Status 404 returned error can't find the container with id f464f01b23f98abff545fc1f7bf400e86eb8bf6b808deb40742362525b3e0004 Nov 03 00:50:32.168352 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:50:32.168325 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/80e0f915-1a1a-4214-92f0-5eeb806be96e-istiod-ca-cert\") pod \"e2e-worker-w0-2f9a3a-openshift-default-7f6f74cf97-dxffl\" (UID: \"80e0f915-1a1a-4214-92f0-5eeb806be96e\") " pod="openshift-ingress/e2e-worker-w0-2f9a3a-openshift-default-7f6f74cf97-dxffl" Nov 03 00:50:32.168472 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:50:32.168372 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/80e0f915-1a1a-4214-92f0-5eeb806be96e-istio-data\") pod \"e2e-worker-w0-2f9a3a-openshift-default-7f6f74cf97-dxffl\" (UID: \"80e0f915-1a1a-4214-92f0-5eeb806be96e\") " pod="openshift-ingress/e2e-worker-w0-2f9a3a-openshift-default-7f6f74cf97-dxffl" Nov 03 00:50:32.168472 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:50:32.168419 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2qzls\" (UniqueName: \"kubernetes.io/projected/80e0f915-1a1a-4214-92f0-5eeb806be96e-kube-api-access-2qzls\") pod \"e2e-worker-w0-2f9a3a-openshift-default-7f6f74cf97-dxffl\" (UID: \"80e0f915-1a1a-4214-92f0-5eeb806be96e\") " pod="openshift-ingress/e2e-worker-w0-2f9a3a-openshift-default-7f6f74cf97-dxffl" Nov 03 00:50:32.168592 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:50:32.168470 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/80e0f915-1a1a-4214-92f0-5eeb806be96e-istio-envoy\") pod \"e2e-worker-w0-2f9a3a-openshift-default-7f6f74cf97-dxffl\" (UID: \"80e0f915-1a1a-4214-92f0-5eeb806be96e\") " pod="openshift-ingress/e2e-worker-w0-2f9a3a-openshift-default-7f6f74cf97-dxffl" Nov 03 00:50:32.168592 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:50:32.168506 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/80e0f915-1a1a-4214-92f0-5eeb806be96e-istio-token\") pod \"e2e-worker-w0-2f9a3a-openshift-default-7f6f74cf97-dxffl\" (UID: \"80e0f915-1a1a-4214-92f0-5eeb806be96e\") " pod="openshift-ingress/e2e-worker-w0-2f9a3a-openshift-default-7f6f74cf97-dxffl" Nov 03 00:50:32.168592 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:50:32.168537 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/80e0f915-1a1a-4214-92f0-5eeb806be96e-istio-podinfo\") pod \"e2e-worker-w0-2f9a3a-openshift-default-7f6f74cf97-dxffl\" (UID: \"80e0f915-1a1a-4214-92f0-5eeb806be96e\") " pod="openshift-ingress/e2e-worker-w0-2f9a3a-openshift-default-7f6f74cf97-dxffl" Nov 03 00:50:32.168750 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:50:32.168642 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/80e0f915-1a1a-4214-92f0-5eeb806be96e-credential-socket\") pod \"e2e-worker-w0-2f9a3a-openshift-default-7f6f74cf97-dxffl\" (UID: \"80e0f915-1a1a-4214-92f0-5eeb806be96e\") " pod="openshift-ingress/e2e-worker-w0-2f9a3a-openshift-default-7f6f74cf97-dxffl" Nov 03 00:50:32.168750 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:50:32.168681 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/80e0f915-1a1a-4214-92f0-5eeb806be96e-workload-certs\") pod \"e2e-worker-w0-2f9a3a-openshift-default-7f6f74cf97-dxffl\" (UID: \"80e0f915-1a1a-4214-92f0-5eeb806be96e\") " pod="openshift-ingress/e2e-worker-w0-2f9a3a-openshift-default-7f6f74cf97-dxffl" Nov 03 00:50:32.168750 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:50:32.168716 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/80e0f915-1a1a-4214-92f0-5eeb806be96e-workload-socket\") pod \"e2e-worker-w0-2f9a3a-openshift-default-7f6f74cf97-dxffl\" (UID: \"80e0f915-1a1a-4214-92f0-5eeb806be96e\") " pod="openshift-ingress/e2e-worker-w0-2f9a3a-openshift-default-7f6f74cf97-dxffl" Nov 03 00:50:32.168893 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:50:32.168819 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/80e0f915-1a1a-4214-92f0-5eeb806be96e-istio-data\") pod \"e2e-worker-w0-2f9a3a-openshift-default-7f6f74cf97-dxffl\" (UID: \"80e0f915-1a1a-4214-92f0-5eeb806be96e\") " pod="openshift-ingress/e2e-worker-w0-2f9a3a-openshift-default-7f6f74cf97-dxffl" Nov 03 00:50:32.169067 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:50:32.169047 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/80e0f915-1a1a-4214-92f0-5eeb806be96e-istiod-ca-cert\") pod \"e2e-worker-w0-2f9a3a-openshift-default-7f6f74cf97-dxffl\" (UID: \"80e0f915-1a1a-4214-92f0-5eeb806be96e\") " pod="openshift-ingress/e2e-worker-w0-2f9a3a-openshift-default-7f6f74cf97-dxffl" Nov 03 00:50:32.169317 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:50:32.169293 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/80e0f915-1a1a-4214-92f0-5eeb806be96e-workload-certs\") pod \"e2e-worker-w0-2f9a3a-openshift-default-7f6f74cf97-dxffl\" (UID: \"80e0f915-1a1a-4214-92f0-5eeb806be96e\") " pod="openshift-ingress/e2e-worker-w0-2f9a3a-openshift-default-7f6f74cf97-dxffl" Nov 03 00:50:32.169378 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:50:32.169323 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/80e0f915-1a1a-4214-92f0-5eeb806be96e-credential-socket\") pod \"e2e-worker-w0-2f9a3a-openshift-default-7f6f74cf97-dxffl\" (UID: \"80e0f915-1a1a-4214-92f0-5eeb806be96e\") " pod="openshift-ingress/e2e-worker-w0-2f9a3a-openshift-default-7f6f74cf97-dxffl" Nov 03 00:50:32.169432 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:50:32.169411 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/80e0f915-1a1a-4214-92f0-5eeb806be96e-workload-socket\") pod \"e2e-worker-w0-2f9a3a-openshift-default-7f6f74cf97-dxffl\" (UID: \"80e0f915-1a1a-4214-92f0-5eeb806be96e\") " pod="openshift-ingress/e2e-worker-w0-2f9a3a-openshift-default-7f6f74cf97-dxffl" Nov 03 00:50:32.171446 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:50:32.171378 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/80e0f915-1a1a-4214-92f0-5eeb806be96e-istio-podinfo\") pod \"e2e-worker-w0-2f9a3a-openshift-default-7f6f74cf97-dxffl\" (UID: \"80e0f915-1a1a-4214-92f0-5eeb806be96e\") " pod="openshift-ingress/e2e-worker-w0-2f9a3a-openshift-default-7f6f74cf97-dxffl" Nov 03 00:50:32.171544 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:50:32.171523 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/80e0f915-1a1a-4214-92f0-5eeb806be96e-istio-envoy\") pod \"e2e-worker-w0-2f9a3a-openshift-default-7f6f74cf97-dxffl\" (UID: \"80e0f915-1a1a-4214-92f0-5eeb806be96e\") " pod="openshift-ingress/e2e-worker-w0-2f9a3a-openshift-default-7f6f74cf97-dxffl" Nov 03 00:50:32.177069 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:50:32.177044 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/80e0f915-1a1a-4214-92f0-5eeb806be96e-istio-token\") pod \"e2e-worker-w0-2f9a3a-openshift-default-7f6f74cf97-dxffl\" (UID: \"80e0f915-1a1a-4214-92f0-5eeb806be96e\") " pod="openshift-ingress/e2e-worker-w0-2f9a3a-openshift-default-7f6f74cf97-dxffl" Nov 03 00:50:32.177239 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:50:32.177223 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2qzls\" (UniqueName: \"kubernetes.io/projected/80e0f915-1a1a-4214-92f0-5eeb806be96e-kube-api-access-2qzls\") pod \"e2e-worker-w0-2f9a3a-openshift-default-7f6f74cf97-dxffl\" (UID: \"80e0f915-1a1a-4214-92f0-5eeb806be96e\") " pod="openshift-ingress/e2e-worker-w0-2f9a3a-openshift-default-7f6f74cf97-dxffl" Nov 03 00:50:32.217922 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:50:32.217881 4939 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w4-650be0-86d95657b7-vm62r"] Nov 03 00:50:32.219692 ip-10-0-140-107 kubenswrapper[4939]: W1103 00:50:32.219671 4939 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod58875207_cd7a_4da9_aee7_f538e7f44963.slice/crio-4ad2081b3d72b46f9fea2179cfd4fd66089cc26c3954c47f2cabb7959843af3d WatchSource:0}: Error finding container 4ad2081b3d72b46f9fea2179cfd4fd66089cc26c3954c47f2cabb7959843af3d: Status 404 returned error can't find the container with id 4ad2081b3d72b46f9fea2179cfd4fd66089cc26c3954c47f2cabb7959843af3d Nov 03 00:50:32.233569 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:50:32.233545 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 00:50:32.343327 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:50:32.343296 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 00:50:32.353531 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:50:32.353506 4939 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-2f9a3a-openshift-default-7f6f74cf97-dxffl" Nov 03 00:50:32.632188 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:50:32.632163 4939 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-2f9a3a-openshift-default-7f6f74cf97-dxffl"] Nov 03 00:50:32.634170 ip-10-0-140-107 kubenswrapper[4939]: W1103 00:50:32.634142 4939 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod80e0f915_1a1a_4214_92f0_5eeb806be96e.slice/crio-af8a98c9f3db734673618a17fc8ef34b517999f2ed9dff5a0c6fc75ff1503d43 WatchSource:0}: Error finding container af8a98c9f3db734673618a17fc8ef34b517999f2ed9dff5a0c6fc75ff1503d43: Status 404 returned error can't find the container with id af8a98c9f3db734673618a17fc8ef34b517999f2ed9dff5a0c6fc75ff1503d43 Nov 03 00:50:32.636590 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:50:32.636556 4939 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 00:50:32.636681 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:50:32.636637 4939 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 00:50:32.636750 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:50:32.636683 4939 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 00:50:32.661228 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:50:32.661198 4939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-650be0-58fc9b6455-9jl7f" event={"ID":"c72f9b11-1218-4c3f-be7c-6541feeb1ed2","Type":"ContainerStarted","Data":"4438a1bb491548b813040ebf4eeb229485ebbe25ba7dfe0c4027433520ddd46e"} Nov 03 00:50:32.661351 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:50:32.661258 4939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-650be0-58fc9b6455-9jl7f" event={"ID":"c72f9b11-1218-4c3f-be7c-6541feeb1ed2","Type":"ContainerStarted","Data":"f464f01b23f98abff545fc1f7bf400e86eb8bf6b808deb40742362525b3e0004"} Nov 03 00:50:32.661351 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:50:32.661291 4939 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-650be0-58fc9b6455-9jl7f" Nov 03 00:50:32.665518 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:50:32.665496 4939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w4-650be0-86d95657b7-vm62r" event={"ID":"58875207-cd7a-4da9-aee7-f538e7f44963","Type":"ContainerStarted","Data":"368faaefacd433a21be99f2ad8a3867bc4ce409dac664ba52af1d6fd56e3ce4c"} Nov 03 00:50:32.665620 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:50:32.665532 4939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w4-650be0-86d95657b7-vm62r" event={"ID":"58875207-cd7a-4da9-aee7-f538e7f44963","Type":"ContainerStarted","Data":"4ad2081b3d72b46f9fea2179cfd4fd66089cc26c3954c47f2cabb7959843af3d"} Nov 03 00:50:32.665620 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:50:32.665552 4939 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-w4-650be0-86d95657b7-vm62r" Nov 03 00:50:32.667591 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:50:32.667572 4939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-2f9a3a-openshift-default-7f6f74cf97-dxffl" event={"ID":"80e0f915-1a1a-4214-92f0-5eeb806be96e","Type":"ContainerStarted","Data":"af8a98c9f3db734673618a17fc8ef34b517999f2ed9dff5a0c6fc75ff1503d43"} Nov 03 00:50:32.680315 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:50:32.680278 4939 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-650be0-58fc9b6455-9jl7f" podStartSLOduration=1.6802650049999999 podStartE2EDuration="1.680265005s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:50:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:50:32.678456715 +0000 UTC m=+580.942791664" watchObservedRunningTime="2026-11-03 00:50:32.680265005 +0000 UTC m=+580.944599950" Nov 03 00:50:32.697490 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:50:32.697450 4939 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-worker-w4-650be0-86d95657b7-vm62r" podStartSLOduration=1.697438349 podStartE2EDuration="1.697438349s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:50:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:50:32.696145644 +0000 UTC m=+580.960480590" watchObservedRunningTime="2026-11-03 00:50:32.697438349 +0000 UTC m=+580.961773316" Nov 03 00:50:32.798118 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:50:32.798083 4939 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-83755a-openshift-default-c86f9565c-shrff" Nov 03 00:50:33.064338 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:50:33.064216 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 00:50:33.295605 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:50:33.295570 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 00:50:33.430576 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:50:33.430541 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 00:50:33.647346 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:50:33.647308 4939 patch_prober.go:29] interesting pod/e2e-ait-15abbf02-openshift-default-65d8997ff-dtl78 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 03 00:50:33.647496 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:50:33.647385 4939 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ait-15abbf02-openshift-default-65d8997ff-dtl78" podUID="b65bcc4a-131b-422b-b407-207ec0f75f26" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 03 00:50:33.677803 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:50:33.677758 4939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-2f9a3a-openshift-default-7f6f74cf97-dxffl" event={"ID":"80e0f915-1a1a-4214-92f0-5eeb806be96e","Type":"ContainerStarted","Data":"daffb5100933e1f21d8c999ff6a9c75480633c4dfc1f5014ebff8b4f61f3e10a"} Nov 03 00:50:33.678334 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:50:33.678308 4939 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-83755a-openshift-default-c86f9565c-shrff" Nov 03 00:50:33.678944 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:50:33.678895 4939 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-83755a-openshift-default-c86f9565c-shrff" Nov 03 00:50:33.701457 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:50:33.701380 4939 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-4pw7v"] Nov 03 00:50:33.703344 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:50:33.703299 4939 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-2f9a3a-openshift-default-7f6f74cf97-dxffl" podStartSLOduration=2.703285976 podStartE2EDuration="2.703285976s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:50:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:50:33.702879067 +0000 UTC m=+581.967214033" watchObservedRunningTime="2026-11-03 00:50:33.703285976 +0000 UTC m=+581.967620964" Nov 03 00:50:34.022864 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:50:34.022837 4939 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-15abbf02-openshift-default-65d8997ff-dtl78" Nov 03 00:50:34.093593 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:50:34.093557 4939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b65bcc4a-131b-422b-b407-207ec0f75f26-istio-token\" (UniqueName: \"kubernetes.io/projected/b65bcc4a-131b-422b-b407-207ec0f75f26-istio-token\") pod \"b65bcc4a-131b-422b-b407-207ec0f75f26\" (UID: \"b65bcc4a-131b-422b-b407-207ec0f75f26\") " Nov 03 00:50:34.094099 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:50:34.093654 4939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b65bcc4a-131b-422b-b407-207ec0f75f26-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b65bcc4a-131b-422b-b407-207ec0f75f26-istio-data\") pod \"b65bcc4a-131b-422b-b407-207ec0f75f26\" (UID: \"b65bcc4a-131b-422b-b407-207ec0f75f26\") " Nov 03 00:50:34.094099 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:50:34.093692 4939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b65bcc4a-131b-422b-b407-207ec0f75f26-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b65bcc4a-131b-422b-b407-207ec0f75f26-workload-socket\") pod \"b65bcc4a-131b-422b-b407-207ec0f75f26\" (UID: \"b65bcc4a-131b-422b-b407-207ec0f75f26\") " Nov 03 00:50:34.094099 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:50:34.093735 4939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b65bcc4a-131b-422b-b407-207ec0f75f26-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b65bcc4a-131b-422b-b407-207ec0f75f26-workload-certs\") pod \"b65bcc4a-131b-422b-b407-207ec0f75f26\" (UID: \"b65bcc4a-131b-422b-b407-207ec0f75f26\") " Nov 03 00:50:34.094099 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:50:34.093795 4939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b65bcc4a-131b-422b-b407-207ec0f75f26-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b65bcc4a-131b-422b-b407-207ec0f75f26-istiod-ca-cert\") pod \"b65bcc4a-131b-422b-b407-207ec0f75f26\" (UID: \"b65bcc4a-131b-422b-b407-207ec0f75f26\") " Nov 03 00:50:34.094099 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:50:34.093846 4939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/b65bcc4a-131b-422b-b407-207ec0f75f26-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b65bcc4a-131b-422b-b407-207ec0f75f26-istio-podinfo\") pod \"b65bcc4a-131b-422b-b407-207ec0f75f26\" (UID: \"b65bcc4a-131b-422b-b407-207ec0f75f26\") " Nov 03 00:50:34.094099 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:50:34.093882 4939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b65bcc4a-131b-422b-b407-207ec0f75f26-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b65bcc4a-131b-422b-b407-207ec0f75f26-credential-socket\") pod \"b65bcc4a-131b-422b-b407-207ec0f75f26\" (UID: \"b65bcc4a-131b-422b-b407-207ec0f75f26\") " Nov 03 00:50:34.094099 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:50:34.093949 4939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b65bcc4a-131b-422b-b407-207ec0f75f26-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b65bcc4a-131b-422b-b407-207ec0f75f26-istio-envoy\") pod \"b65bcc4a-131b-422b-b407-207ec0f75f26\" (UID: \"b65bcc4a-131b-422b-b407-207ec0f75f26\") " Nov 03 00:50:34.094099 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:50:34.094006 4939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b65bcc4a-131b-422b-b407-207ec0f75f26-kube-api-access-2v59r\" (UniqueName: \"kubernetes.io/projected/b65bcc4a-131b-422b-b407-207ec0f75f26-kube-api-access-2v59r\") pod \"b65bcc4a-131b-422b-b407-207ec0f75f26\" (UID: \"b65bcc4a-131b-422b-b407-207ec0f75f26\") " Nov 03 00:50:34.094500 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:50:34.093945 4939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b65bcc4a-131b-422b-b407-207ec0f75f26-istio-data" pod "b65bcc4a-131b-422b-b407-207ec0f75f26" (UID: "b65bcc4a-131b-422b-b407-207ec0f75f26"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 00:50:34.094500 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:50:34.094027 4939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b65bcc4a-131b-422b-b407-207ec0f75f26-workload-socket" pod "b65bcc4a-131b-422b-b407-207ec0f75f26" (UID: "b65bcc4a-131b-422b-b407-207ec0f75f26"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 00:50:34.094500 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:50:34.094057 4939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b65bcc4a-131b-422b-b407-207ec0f75f26-workload-certs" pod "b65bcc4a-131b-422b-b407-207ec0f75f26" (UID: "b65bcc4a-131b-422b-b407-207ec0f75f26"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 00:50:34.094500 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:50:34.094179 4939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b65bcc4a-131b-422b-b407-207ec0f75f26-credential-socket" pod "b65bcc4a-131b-422b-b407-207ec0f75f26" (UID: "b65bcc4a-131b-422b-b407-207ec0f75f26"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 00:50:34.094500 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:50:34.094381 4939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b65bcc4a-131b-422b-b407-207ec0f75f26-istiod-ca-cert" pod "b65bcc4a-131b-422b-b407-207ec0f75f26" (UID: "b65bcc4a-131b-422b-b407-207ec0f75f26"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 00:50:34.094500 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:50:34.094460 4939 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b65bcc4a-131b-422b-b407-207ec0f75f26-istio-data\") on node \"ip-10-0-140-107.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 00:50:34.094500 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:50:34.094479 4939 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b65bcc4a-131b-422b-b407-207ec0f75f26-workload-socket\") on node \"ip-10-0-140-107.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 00:50:34.094500 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:50:34.094494 4939 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b65bcc4a-131b-422b-b407-207ec0f75f26-workload-certs\") on node \"ip-10-0-140-107.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 00:50:34.094500 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:50:34.094506 4939 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b65bcc4a-131b-422b-b407-207ec0f75f26-istiod-ca-cert\") on node \"ip-10-0-140-107.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 00:50:34.094996 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:50:34.094516 4939 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b65bcc4a-131b-422b-b407-207ec0f75f26-credential-socket\") on node \"ip-10-0-140-107.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 00:50:34.095951 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:50:34.095924 4939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b65bcc4a-131b-422b-b407-207ec0f75f26-istio-token" pod "b65bcc4a-131b-422b-b407-207ec0f75f26" (UID: "b65bcc4a-131b-422b-b407-207ec0f75f26"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 00:50:34.096277 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:50:34.096258 4939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/b65bcc4a-131b-422b-b407-207ec0f75f26-istio-podinfo" pod "b65bcc4a-131b-422b-b407-207ec0f75f26" (UID: "b65bcc4a-131b-422b-b407-207ec0f75f26"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 00:50:34.096461 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:50:34.096436 4939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b65bcc4a-131b-422b-b407-207ec0f75f26-kube-api-access-2v59r" pod "b65bcc4a-131b-422b-b407-207ec0f75f26" (UID: "b65bcc4a-131b-422b-b407-207ec0f75f26"). InnerVolumeSpecName "kube-api-access-2v59r". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 00:50:34.096543 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:50:34.096488 4939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b65bcc4a-131b-422b-b407-207ec0f75f26-istio-envoy" pod "b65bcc4a-131b-422b-b407-207ec0f75f26" (UID: "b65bcc4a-131b-422b-b407-207ec0f75f26"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 00:50:34.194617 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:50:34.194563 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 00:50:34.195051 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:50:34.195028 4939 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b65bcc4a-131b-422b-b407-207ec0f75f26-istio-podinfo\") on node \"ip-10-0-140-107.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 00:50:34.195051 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:50:34.195050 4939 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b65bcc4a-131b-422b-b407-207ec0f75f26-istio-envoy\") on node \"ip-10-0-140-107.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 00:50:34.195228 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:50:34.195061 4939 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2v59r\" (UniqueName: \"kubernetes.io/projected/b65bcc4a-131b-422b-b407-207ec0f75f26-kube-api-access-2v59r\") on node \"ip-10-0-140-107.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 00:50:34.195228 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:50:34.195070 4939 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b65bcc4a-131b-422b-b407-207ec0f75f26-istio-token\") on node \"ip-10-0-140-107.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 00:50:34.316982 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:50:34.316293 4939 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-4aec6006-75dd64d785-ndfg6"] Nov 03 00:50:34.355076 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:50:34.354767 4939 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-2f9a3a-openshift-default-7f6f74cf97-dxffl" Nov 03 00:50:34.367423 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:50:34.367385 4939 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-2f9a3a-openshift-default-7f6f74cf97-dxffl" Nov 03 00:50:34.424473 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:50:34.424417 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 00:50:34.615688 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:50:34.615654 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 00:50:34.685188 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:50:34.685156 4939 generic.go:356] "Generic (PLEG): container finished" podID="b65bcc4a-131b-422b-b407-207ec0f75f26" containerID="7069432c5e0f3f92df9063f70cb7588fca184580437788a66e2b7f1cac695f19" exitCode=0 Nov 03 00:50:34.685330 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:50:34.685222 4939 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-15abbf02-openshift-default-65d8997ff-dtl78" Nov 03 00:50:34.685330 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:50:34.685243 4939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-15abbf02-openshift-default-65d8997ff-dtl78" event={"ID":"b65bcc4a-131b-422b-b407-207ec0f75f26","Type":"ContainerDied","Data":"7069432c5e0f3f92df9063f70cb7588fca184580437788a66e2b7f1cac695f19"} Nov 03 00:50:34.685330 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:50:34.685296 4939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-15abbf02-openshift-default-65d8997ff-dtl78" event={"ID":"b65bcc4a-131b-422b-b407-207ec0f75f26","Type":"ContainerDied","Data":"2a29ea8c90c43e12ef117efb8a28d4015160cd253b176f8f2a3af31adbade27d"} Nov 03 00:50:34.685330 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:50:34.685327 4939 scope.go:122] "RemoveContainer" containerID="7069432c5e0f3f92df9063f70cb7588fca184580437788a66e2b7f1cac695f19" Nov 03 00:50:34.686922 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:50:34.686757 4939 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-2f9a3a-openshift-default-7f6f74cf97-dxffl" Nov 03 00:50:34.687789 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:50:34.687768 4939 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-2f9a3a-openshift-default-7f6f74cf97-dxffl" Nov 03 00:50:34.695984 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:50:34.695961 4939 scope.go:122] "RemoveContainer" containerID="30cf59b9da3be8b2dabadecf37d011b47d92e5f95ac2e969a65d60eeeee838c5" Nov 03 00:50:34.705371 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:50:34.705353 4939 scope.go:122] "RemoveContainer" containerID="7069432c5e0f3f92df9063f70cb7588fca184580437788a66e2b7f1cac695f19" Nov 03 00:50:34.705609 ip-10-0-140-107 kubenswrapper[4939]: E1103 00:50:34.705590 4939 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7069432c5e0f3f92df9063f70cb7588fca184580437788a66e2b7f1cac695f19\": container with ID starting with 7069432c5e0f3f92df9063f70cb7588fca184580437788a66e2b7f1cac695f19 not found: ID does not exist" containerID="7069432c5e0f3f92df9063f70cb7588fca184580437788a66e2b7f1cac695f19" Nov 03 00:50:34.705664 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:50:34.705620 4939 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7069432c5e0f3f92df9063f70cb7588fca184580437788a66e2b7f1cac695f19"} err="failed to get container status \"7069432c5e0f3f92df9063f70cb7588fca184580437788a66e2b7f1cac695f19\": rpc error: code = NotFound desc = could not find container \"7069432c5e0f3f92df9063f70cb7588fca184580437788a66e2b7f1cac695f19\": container with ID starting with 7069432c5e0f3f92df9063f70cb7588fca184580437788a66e2b7f1cac695f19 not found: ID does not exist" Nov 03 00:50:34.705664 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:50:34.705642 4939 scope.go:122] "RemoveContainer" containerID="30cf59b9da3be8b2dabadecf37d011b47d92e5f95ac2e969a65d60eeeee838c5" Nov 03 00:50:34.705958 ip-10-0-140-107 kubenswrapper[4939]: E1103 00:50:34.705898 4939 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"30cf59b9da3be8b2dabadecf37d011b47d92e5f95ac2e969a65d60eeeee838c5\": container with ID starting with 30cf59b9da3be8b2dabadecf37d011b47d92e5f95ac2e969a65d60eeeee838c5 not found: ID does not exist" containerID="30cf59b9da3be8b2dabadecf37d011b47d92e5f95ac2e969a65d60eeeee838c5" Nov 03 00:50:34.706060 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:50:34.705949 4939 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"30cf59b9da3be8b2dabadecf37d011b47d92e5f95ac2e969a65d60eeeee838c5"} err="failed to get container status \"30cf59b9da3be8b2dabadecf37d011b47d92e5f95ac2e969a65d60eeeee838c5\": rpc error: code = NotFound desc = could not find container \"30cf59b9da3be8b2dabadecf37d011b47d92e5f95ac2e969a65d60eeeee838c5\": container with ID starting with 30cf59b9da3be8b2dabadecf37d011b47d92e5f95ac2e969a65d60eeeee838c5 not found: ID does not exist" Nov 03 00:50:34.736345 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:50:34.736310 4939 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-15abbf02-openshift-default-65d8997ff-dtl78"] Nov 03 00:50:34.743955 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:50:34.743930 4939 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ait-15abbf02-openshift-default-65d8997ff-dtl78"] Nov 03 00:50:34.905272 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:50:34.905183 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 00:50:35.035044 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:50:35.035006 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 00:50:35.315653 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:50:35.315566 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 00:50:35.439610 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:50:35.439432 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 00:50:35.751782 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:50:35.751749 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 00:50:35.862211 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:50:35.862176 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 00:50:35.871512 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:50:35.871477 4939 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b65bcc4a-131b-422b-b407-207ec0f75f26" path="/var/lib/kubelet/pods/b65bcc4a-131b-422b-b407-207ec0f75f26/volumes" Nov 03 00:50:36.124656 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:50:36.124615 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 00:50:36.247511 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:50:36.246897 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 00:50:36.333671 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:50:36.333631 4939 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-4aec6006-7cdf66f59-5sqpc"] Nov 03 00:50:36.334142 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:50:36.333994 4939 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-derive-4aec6006-7cdf66f59-5sqpc" podUID="e11c6e1b-c77d-49b5-a7d0-32c0f3e55e7b" containerName="maas-api" containerID="cri-o://cf5a9700382f32d07d2a0cd029c7d23b13e5369923c6a07ec59ff2b83067f4ab" gracePeriod=30 Nov 03 00:50:36.382283 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:50:36.382211 4939 prober.go:122] "Probe failed" probeType="Readiness" pod="odh-ai-gateway-infra/maas-api-e2e-derive-4aec6006-7cdf66f59-5sqpc" podUID="e11c6e1b-c77d-49b5-a7d0-32c0f3e55e7b" containerName="maas-api" probeResult="failure" output="Get \"https://10.132.0.52:8443/health\": dial tcp 10.132.0.52:8443: connect: connection refused" Nov 03 00:50:36.422922 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:50:36.422875 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 00:50:36.576896 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:50:36.576840 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 00:50:36.659387 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:50:36.659357 4939 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-4aec6006-7cdf66f59-5sqpc" Nov 03 00:50:36.700191 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:50:36.700152 4939 generic.go:356] "Generic (PLEG): container finished" podID="e11c6e1b-c77d-49b5-a7d0-32c0f3e55e7b" containerID="cf5a9700382f32d07d2a0cd029c7d23b13e5369923c6a07ec59ff2b83067f4ab" exitCode=0 Nov 03 00:50:36.700382 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:50:36.700211 4939 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-4aec6006-7cdf66f59-5sqpc" Nov 03 00:50:36.700382 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:50:36.700233 4939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-4aec6006-7cdf66f59-5sqpc" event={"ID":"e11c6e1b-c77d-49b5-a7d0-32c0f3e55e7b","Type":"ContainerDied","Data":"cf5a9700382f32d07d2a0cd029c7d23b13e5369923c6a07ec59ff2b83067f4ab"} Nov 03 00:50:36.700382 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:50:36.700289 4939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-4aec6006-7cdf66f59-5sqpc" event={"ID":"e11c6e1b-c77d-49b5-a7d0-32c0f3e55e7b","Type":"ContainerDied","Data":"41ee755c173ad88da952d76fd53abb39e2c06c36457a1b188e6b09ffdbe6fa00"} Nov 03 00:50:36.700382 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:50:36.700320 4939 scope.go:122] "RemoveContainer" containerID="cf5a9700382f32d07d2a0cd029c7d23b13e5369923c6a07ec59ff2b83067f4ab" Nov 03 00:50:36.709781 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:50:36.709758 4939 scope.go:122] "RemoveContainer" containerID="cf5a9700382f32d07d2a0cd029c7d23b13e5369923c6a07ec59ff2b83067f4ab" Nov 03 00:50:36.710132 ip-10-0-140-107 kubenswrapper[4939]: E1103 00:50:36.710109 4939 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cf5a9700382f32d07d2a0cd029c7d23b13e5369923c6a07ec59ff2b83067f4ab\": container with ID starting with cf5a9700382f32d07d2a0cd029c7d23b13e5369923c6a07ec59ff2b83067f4ab not found: ID does not exist" containerID="cf5a9700382f32d07d2a0cd029c7d23b13e5369923c6a07ec59ff2b83067f4ab" Nov 03 00:50:36.710245 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:50:36.710142 4939 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cf5a9700382f32d07d2a0cd029c7d23b13e5369923c6a07ec59ff2b83067f4ab"} err="failed to get container status \"cf5a9700382f32d07d2a0cd029c7d23b13e5369923c6a07ec59ff2b83067f4ab\": rpc error: code = NotFound desc = could not find container \"cf5a9700382f32d07d2a0cd029c7d23b13e5369923c6a07ec59ff2b83067f4ab\": container with ID starting with cf5a9700382f32d07d2a0cd029c7d23b13e5369923c6a07ec59ff2b83067f4ab not found: ID does not exist" Nov 03 00:50:36.718565 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:50:36.718530 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 00:50:36.731893 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:50:36.731862 4939 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w3-83755a-84f6d788fc-bgk7t"] Nov 03 00:50:36.732555 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:50:36.732528 4939 cpu_manager.go:414] "containerMap: removing container" podUID="b65bcc4a-131b-422b-b407-207ec0f75f26" containerName="istio-proxy" Nov 03 00:50:36.732555 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:50:36.732552 4939 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b65bcc4a-131b-422b-b407-207ec0f75f26" containerName="istio-proxy" Nov 03 00:50:36.732755 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:50:36.732571 4939 cpu_manager.go:414] "containerMap: removing container" podUID="e11c6e1b-c77d-49b5-a7d0-32c0f3e55e7b" containerName="maas-api" Nov 03 00:50:36.732755 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:50:36.732578 4939 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e11c6e1b-c77d-49b5-a7d0-32c0f3e55e7b" containerName="maas-api" Nov 03 00:50:36.732755 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:50:36.732608 4939 cpu_manager.go:414] "containerMap: removing container" podUID="b65bcc4a-131b-422b-b407-207ec0f75f26" containerName="istio-proxy" Nov 03 00:50:36.732755 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:50:36.732616 4939 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b65bcc4a-131b-422b-b407-207ec0f75f26" containerName="istio-proxy" Nov 03 00:50:36.732984 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:50:36.732755 4939 memory_manager.go:365] "RemoveStaleState removing state" podUID="b65bcc4a-131b-422b-b407-207ec0f75f26" containerName="istio-proxy" Nov 03 00:50:36.732984 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:50:36.732768 4939 memory_manager.go:365] "RemoveStaleState removing state" podUID="e11c6e1b-c77d-49b5-a7d0-32c0f3e55e7b" containerName="maas-api" Nov 03 00:50:36.732984 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:50:36.732780 4939 memory_manager.go:365] "RemoveStaleState removing state" podUID="b65bcc4a-131b-422b-b407-207ec0f75f26" containerName="istio-proxy" Nov 03 00:50:36.738258 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:50:36.738237 4939 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-83755a-84f6d788fc-bgk7t" Nov 03 00:50:36.741787 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:50:36.741759 4939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e11c6e1b-c77d-49b5-a7d0-32c0f3e55e7b-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e11c6e1b-c77d-49b5-a7d0-32c0f3e55e7b-openshift-service-ca\") pod \"e11c6e1b-c77d-49b5-a7d0-32c0f3e55e7b\" (UID: \"e11c6e1b-c77d-49b5-a7d0-32c0f3e55e7b\") " Nov 03 00:50:36.741902 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:50:36.741887 4939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e11c6e1b-c77d-49b5-a7d0-32c0f3e55e7b-kube-api-access-8nnjb\" (UniqueName: \"kubernetes.io/projected/e11c6e1b-c77d-49b5-a7d0-32c0f3e55e7b-kube-api-access-8nnjb\") pod \"e11c6e1b-c77d-49b5-a7d0-32c0f3e55e7b\" (UID: \"e11c6e1b-c77d-49b5-a7d0-32c0f3e55e7b\") " Nov 03 00:50:36.742001 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:50:36.741983 4939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e11c6e1b-c77d-49b5-a7d0-32c0f3e55e7b-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e11c6e1b-c77d-49b5-a7d0-32c0f3e55e7b-maas-api-tls\") pod \"e11c6e1b-c77d-49b5-a7d0-32c0f3e55e7b\" (UID: \"e11c6e1b-c77d-49b5-a7d0-32c0f3e55e7b\") " Nov 03 00:50:36.742082 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:50:36.742065 4939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e11c6e1b-c77d-49b5-a7d0-32c0f3e55e7b-metrics-certs\" (UniqueName: \"kubernetes.io/secret/e11c6e1b-c77d-49b5-a7d0-32c0f3e55e7b-metrics-certs\") pod \"e11c6e1b-c77d-49b5-a7d0-32c0f3e55e7b\" (UID: \"e11c6e1b-c77d-49b5-a7d0-32c0f3e55e7b\") " Nov 03 00:50:36.742429 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:50:36.742178 4939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e11c6e1b-c77d-49b5-a7d0-32c0f3e55e7b-openshift-service-ca" pod "e11c6e1b-c77d-49b5-a7d0-32c0f3e55e7b" (UID: "e11c6e1b-c77d-49b5-a7d0-32c0f3e55e7b"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 00:50:36.742510 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:50:36.742434 4939 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e11c6e1b-c77d-49b5-a7d0-32c0f3e55e7b-openshift-service-ca\") on node \"ip-10-0-140-107.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 00:50:36.745259 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:50:36.745173 4939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e11c6e1b-c77d-49b5-a7d0-32c0f3e55e7b-metrics-certs" pod "e11c6e1b-c77d-49b5-a7d0-32c0f3e55e7b" (UID: "e11c6e1b-c77d-49b5-a7d0-32c0f3e55e7b"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 00:50:36.745259 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:50:36.745172 4939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e11c6e1b-c77d-49b5-a7d0-32c0f3e55e7b-kube-api-access-8nnjb" pod "e11c6e1b-c77d-49b5-a7d0-32c0f3e55e7b" (UID: "e11c6e1b-c77d-49b5-a7d0-32c0f3e55e7b"). InnerVolumeSpecName "kube-api-access-8nnjb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 00:50:36.745259 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:50:36.745235 4939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e11c6e1b-c77d-49b5-a7d0-32c0f3e55e7b-maas-api-tls" pod "e11c6e1b-c77d-49b5-a7d0-32c0f3e55e7b" (UID: "e11c6e1b-c77d-49b5-a7d0-32c0f3e55e7b"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 00:50:36.751801 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:50:36.751763 4939 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-worker-w3-83755a-serving-cert\"" Nov 03 00:50:36.753674 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:50:36.753654 4939 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w3-83755a-84f6d788fc-bgk7t"] Nov 03 00:50:36.843152 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:50:36.843116 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/68597924-b259-47a2-b006-0d94dcb6315f-maas-api-tls\") pod \"maas-api-e2e-worker-w3-83755a-84f6d788fc-bgk7t\" (UID: \"68597924-b259-47a2-b006-0d94dcb6315f\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-83755a-84f6d788fc-bgk7t" Nov 03 00:50:36.843303 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:50:36.843254 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/68597924-b259-47a2-b006-0d94dcb6315f-metrics-certs\") pod \"maas-api-e2e-worker-w3-83755a-84f6d788fc-bgk7t\" (UID: \"68597924-b259-47a2-b006-0d94dcb6315f\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-83755a-84f6d788fc-bgk7t" Nov 03 00:50:36.843362 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:50:36.843316 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/68597924-b259-47a2-b006-0d94dcb6315f-openshift-service-ca\") pod \"maas-api-e2e-worker-w3-83755a-84f6d788fc-bgk7t\" (UID: \"68597924-b259-47a2-b006-0d94dcb6315f\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-83755a-84f6d788fc-bgk7t" Nov 03 00:50:36.843696 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:50:36.843674 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z5jrq\" (UniqueName: \"kubernetes.io/projected/68597924-b259-47a2-b006-0d94dcb6315f-kube-api-access-z5jrq\") pod \"maas-api-e2e-worker-w3-83755a-84f6d788fc-bgk7t\" (UID: \"68597924-b259-47a2-b006-0d94dcb6315f\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-83755a-84f6d788fc-bgk7t" Nov 03 00:50:36.843804 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:50:36.843788 4939 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8nnjb\" (UniqueName: \"kubernetes.io/projected/e11c6e1b-c77d-49b5-a7d0-32c0f3e55e7b-kube-api-access-8nnjb\") on node \"ip-10-0-140-107.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 00:50:36.843853 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:50:36.843808 4939 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e11c6e1b-c77d-49b5-a7d0-32c0f3e55e7b-maas-api-tls\") on node \"ip-10-0-140-107.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 00:50:36.843853 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:50:36.843818 4939 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e11c6e1b-c77d-49b5-a7d0-32c0f3e55e7b-metrics-certs\") on node \"ip-10-0-140-107.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 00:50:36.859288 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:50:36.859258 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 00:50:36.944931 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:50:36.944835 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/68597924-b259-47a2-b006-0d94dcb6315f-maas-api-tls\") pod \"maas-api-e2e-worker-w3-83755a-84f6d788fc-bgk7t\" (UID: \"68597924-b259-47a2-b006-0d94dcb6315f\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-83755a-84f6d788fc-bgk7t" Nov 03 00:50:36.945098 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:50:36.944969 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/68597924-b259-47a2-b006-0d94dcb6315f-metrics-certs\") pod \"maas-api-e2e-worker-w3-83755a-84f6d788fc-bgk7t\" (UID: \"68597924-b259-47a2-b006-0d94dcb6315f\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-83755a-84f6d788fc-bgk7t" Nov 03 00:50:36.945098 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:50:36.945006 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/68597924-b259-47a2-b006-0d94dcb6315f-openshift-service-ca\") pod \"maas-api-e2e-worker-w3-83755a-84f6d788fc-bgk7t\" (UID: \"68597924-b259-47a2-b006-0d94dcb6315f\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-83755a-84f6d788fc-bgk7t" Nov 03 00:50:36.945098 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:50:36.945071 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z5jrq\" (UniqueName: \"kubernetes.io/projected/68597924-b259-47a2-b006-0d94dcb6315f-kube-api-access-z5jrq\") pod \"maas-api-e2e-worker-w3-83755a-84f6d788fc-bgk7t\" (UID: \"68597924-b259-47a2-b006-0d94dcb6315f\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-83755a-84f6d788fc-bgk7t" Nov 03 00:50:36.946097 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:50:36.946063 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/68597924-b259-47a2-b006-0d94dcb6315f-openshift-service-ca\") pod \"maas-api-e2e-worker-w3-83755a-84f6d788fc-bgk7t\" (UID: \"68597924-b259-47a2-b006-0d94dcb6315f\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-83755a-84f6d788fc-bgk7t" Nov 03 00:50:36.947496 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:50:36.947477 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/68597924-b259-47a2-b006-0d94dcb6315f-metrics-certs\") pod \"maas-api-e2e-worker-w3-83755a-84f6d788fc-bgk7t\" (UID: \"68597924-b259-47a2-b006-0d94dcb6315f\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-83755a-84f6d788fc-bgk7t" Nov 03 00:50:36.947586 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:50:36.947520 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/68597924-b259-47a2-b006-0d94dcb6315f-maas-api-tls\") pod \"maas-api-e2e-worker-w3-83755a-84f6d788fc-bgk7t\" (UID: \"68597924-b259-47a2-b006-0d94dcb6315f\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-83755a-84f6d788fc-bgk7t" Nov 03 00:50:36.967366 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:50:36.967333 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-z5jrq\" (UniqueName: \"kubernetes.io/projected/68597924-b259-47a2-b006-0d94dcb6315f-kube-api-access-z5jrq\") pod \"maas-api-e2e-worker-w3-83755a-84f6d788fc-bgk7t\" (UID: \"68597924-b259-47a2-b006-0d94dcb6315f\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-83755a-84f6d788fc-bgk7t" Nov 03 00:50:37.036987 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:50:37.036844 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 00:50:37.070741 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:50:37.070706 4939 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-83755a-84f6d788fc-bgk7t" Nov 03 00:50:37.076414 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:50:37.076381 4939 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-4aec6006-7cdf66f59-5sqpc"] Nov 03 00:50:37.091865 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:50:37.091836 4939 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-4aec6006-7cdf66f59-5sqpc"] Nov 03 00:50:37.362609 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:50:37.362575 4939 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w3-83755a-84f6d788fc-bgk7t"] Nov 03 00:50:37.364859 ip-10-0-140-107 kubenswrapper[4939]: W1103 00:50:37.364297 4939 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod68597924_b259_47a2_b006_0d94dcb6315f.slice/crio-4bd4619b5544726517f2ead4e9ca162d1d45e189c2eee69c8786a4dfea8f599a WatchSource:0}: Error finding container 4bd4619b5544726517f2ead4e9ca162d1d45e189c2eee69c8786a4dfea8f599a: Status 404 returned error can't find the container with id 4bd4619b5544726517f2ead4e9ca162d1d45e189c2eee69c8786a4dfea8f599a Nov 03 00:50:37.437072 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:50:37.437035 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 00:50:37.660753 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:50:37.660550 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 00:50:37.718864 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:50:37.718824 4939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-83755a-84f6d788fc-bgk7t" event={"ID":"68597924-b259-47a2-b006-0d94dcb6315f","Type":"ContainerStarted","Data":"4bd4619b5544726517f2ead4e9ca162d1d45e189c2eee69c8786a4dfea8f599a"} Nov 03 00:50:37.840087 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:50:37.840045 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 00:50:37.867701 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:50:37.867675 4939 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e11c6e1b-c77d-49b5-a7d0-32c0f3e55e7b" path="/var/lib/kubelet/pods/e11c6e1b-c77d-49b5-a7d0-32c0f3e55e7b/volumes" Nov 03 00:50:38.186087 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:50:38.185987 4939 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-88bbf7-openshift-default-574f47fff7-rf5cn"] Nov 03 00:50:38.190411 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:50:38.190387 4939 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-88bbf7-openshift-default-574f47fff7-rf5cn" Nov 03 00:50:38.199263 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:50:38.199236 4939 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w1-88bbf7-openshift-default-dockercfg-h6n25\"" Nov 03 00:50:38.199998 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:50:38.199972 4939 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-88bbf7-openshift-default-574f47fff7-rf5cn"] Nov 03 00:50:38.263227 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:50:38.263190 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 00:50:38.267366 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:50:38.267340 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/08f62205-b31d-4863-ab9e-3b13b15fbef5-istiod-ca-cert\") pod \"e2e-worker-w1-88bbf7-openshift-default-574f47fff7-rf5cn\" (UID: \"08f62205-b31d-4863-ab9e-3b13b15fbef5\") " pod="openshift-ingress/e2e-worker-w1-88bbf7-openshift-default-574f47fff7-rf5cn" Nov 03 00:50:38.267486 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:50:38.267378 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/08f62205-b31d-4863-ab9e-3b13b15fbef5-istio-token\") pod \"e2e-worker-w1-88bbf7-openshift-default-574f47fff7-rf5cn\" (UID: \"08f62205-b31d-4863-ab9e-3b13b15fbef5\") " pod="openshift-ingress/e2e-worker-w1-88bbf7-openshift-default-574f47fff7-rf5cn" Nov 03 00:50:38.267521 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:50:38.267488 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/08f62205-b31d-4863-ab9e-3b13b15fbef5-workload-socket\") pod \"e2e-worker-w1-88bbf7-openshift-default-574f47fff7-rf5cn\" (UID: \"08f62205-b31d-4863-ab9e-3b13b15fbef5\") " pod="openshift-ingress/e2e-worker-w1-88bbf7-openshift-default-574f47fff7-rf5cn" Nov 03 00:50:38.267571 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:50:38.267521 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/08f62205-b31d-4863-ab9e-3b13b15fbef5-istio-data\") pod \"e2e-worker-w1-88bbf7-openshift-default-574f47fff7-rf5cn\" (UID: \"08f62205-b31d-4863-ab9e-3b13b15fbef5\") " pod="openshift-ingress/e2e-worker-w1-88bbf7-openshift-default-574f47fff7-rf5cn" Nov 03 00:50:38.267607 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:50:38.267582 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/08f62205-b31d-4863-ab9e-3b13b15fbef5-istio-envoy\") pod \"e2e-worker-w1-88bbf7-openshift-default-574f47fff7-rf5cn\" (UID: \"08f62205-b31d-4863-ab9e-3b13b15fbef5\") " pod="openshift-ingress/e2e-worker-w1-88bbf7-openshift-default-574f47fff7-rf5cn" Nov 03 00:50:38.267649 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:50:38.267627 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/08f62205-b31d-4863-ab9e-3b13b15fbef5-workload-certs\") pod \"e2e-worker-w1-88bbf7-openshift-default-574f47fff7-rf5cn\" (UID: \"08f62205-b31d-4863-ab9e-3b13b15fbef5\") " pod="openshift-ingress/e2e-worker-w1-88bbf7-openshift-default-574f47fff7-rf5cn" Nov 03 00:50:38.267681 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:50:38.267664 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fq5xx\" (UniqueName: \"kubernetes.io/projected/08f62205-b31d-4863-ab9e-3b13b15fbef5-kube-api-access-fq5xx\") pod \"e2e-worker-w1-88bbf7-openshift-default-574f47fff7-rf5cn\" (UID: \"08f62205-b31d-4863-ab9e-3b13b15fbef5\") " pod="openshift-ingress/e2e-worker-w1-88bbf7-openshift-default-574f47fff7-rf5cn" Nov 03 00:50:38.267715 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:50:38.267693 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/08f62205-b31d-4863-ab9e-3b13b15fbef5-credential-socket\") pod \"e2e-worker-w1-88bbf7-openshift-default-574f47fff7-rf5cn\" (UID: \"08f62205-b31d-4863-ab9e-3b13b15fbef5\") " pod="openshift-ingress/e2e-worker-w1-88bbf7-openshift-default-574f47fff7-rf5cn" Nov 03 00:50:38.267819 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:50:38.267807 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/08f62205-b31d-4863-ab9e-3b13b15fbef5-istio-podinfo\") pod \"e2e-worker-w1-88bbf7-openshift-default-574f47fff7-rf5cn\" (UID: \"08f62205-b31d-4863-ab9e-3b13b15fbef5\") " pod="openshift-ingress/e2e-worker-w1-88bbf7-openshift-default-574f47fff7-rf5cn" Nov 03 00:50:38.374937 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:50:38.370476 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/08f62205-b31d-4863-ab9e-3b13b15fbef5-workload-socket\") pod \"e2e-worker-w1-88bbf7-openshift-default-574f47fff7-rf5cn\" (UID: \"08f62205-b31d-4863-ab9e-3b13b15fbef5\") " pod="openshift-ingress/e2e-worker-w1-88bbf7-openshift-default-574f47fff7-rf5cn" Nov 03 00:50:38.374937 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:50:38.370539 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/08f62205-b31d-4863-ab9e-3b13b15fbef5-istio-data\") pod \"e2e-worker-w1-88bbf7-openshift-default-574f47fff7-rf5cn\" (UID: \"08f62205-b31d-4863-ab9e-3b13b15fbef5\") " pod="openshift-ingress/e2e-worker-w1-88bbf7-openshift-default-574f47fff7-rf5cn" Nov 03 00:50:38.374937 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:50:38.370588 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/08f62205-b31d-4863-ab9e-3b13b15fbef5-istio-envoy\") pod \"e2e-worker-w1-88bbf7-openshift-default-574f47fff7-rf5cn\" (UID: \"08f62205-b31d-4863-ab9e-3b13b15fbef5\") " pod="openshift-ingress/e2e-worker-w1-88bbf7-openshift-default-574f47fff7-rf5cn" Nov 03 00:50:38.374937 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:50:38.370625 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/08f62205-b31d-4863-ab9e-3b13b15fbef5-workload-certs\") pod \"e2e-worker-w1-88bbf7-openshift-default-574f47fff7-rf5cn\" (UID: \"08f62205-b31d-4863-ab9e-3b13b15fbef5\") " pod="openshift-ingress/e2e-worker-w1-88bbf7-openshift-default-574f47fff7-rf5cn" Nov 03 00:50:38.374937 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:50:38.370660 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fq5xx\" (UniqueName: \"kubernetes.io/projected/08f62205-b31d-4863-ab9e-3b13b15fbef5-kube-api-access-fq5xx\") pod \"e2e-worker-w1-88bbf7-openshift-default-574f47fff7-rf5cn\" (UID: \"08f62205-b31d-4863-ab9e-3b13b15fbef5\") " pod="openshift-ingress/e2e-worker-w1-88bbf7-openshift-default-574f47fff7-rf5cn" Nov 03 00:50:38.374937 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:50:38.370755 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/08f62205-b31d-4863-ab9e-3b13b15fbef5-credential-socket\") pod \"e2e-worker-w1-88bbf7-openshift-default-574f47fff7-rf5cn\" (UID: \"08f62205-b31d-4863-ab9e-3b13b15fbef5\") " pod="openshift-ingress/e2e-worker-w1-88bbf7-openshift-default-574f47fff7-rf5cn" Nov 03 00:50:38.374937 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:50:38.370838 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/08f62205-b31d-4863-ab9e-3b13b15fbef5-istio-podinfo\") pod \"e2e-worker-w1-88bbf7-openshift-default-574f47fff7-rf5cn\" (UID: \"08f62205-b31d-4863-ab9e-3b13b15fbef5\") " pod="openshift-ingress/e2e-worker-w1-88bbf7-openshift-default-574f47fff7-rf5cn" Nov 03 00:50:38.374937 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:50:38.370874 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/08f62205-b31d-4863-ab9e-3b13b15fbef5-istiod-ca-cert\") pod \"e2e-worker-w1-88bbf7-openshift-default-574f47fff7-rf5cn\" (UID: \"08f62205-b31d-4863-ab9e-3b13b15fbef5\") " pod="openshift-ingress/e2e-worker-w1-88bbf7-openshift-default-574f47fff7-rf5cn" Nov 03 00:50:38.374937 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:50:38.371894 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/08f62205-b31d-4863-ab9e-3b13b15fbef5-istio-token\") pod \"e2e-worker-w1-88bbf7-openshift-default-574f47fff7-rf5cn\" (UID: \"08f62205-b31d-4863-ab9e-3b13b15fbef5\") " pod="openshift-ingress/e2e-worker-w1-88bbf7-openshift-default-574f47fff7-rf5cn" Nov 03 00:50:38.374937 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:50:38.372748 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/08f62205-b31d-4863-ab9e-3b13b15fbef5-workload-socket\") pod \"e2e-worker-w1-88bbf7-openshift-default-574f47fff7-rf5cn\" (UID: \"08f62205-b31d-4863-ab9e-3b13b15fbef5\") " pod="openshift-ingress/e2e-worker-w1-88bbf7-openshift-default-574f47fff7-rf5cn" Nov 03 00:50:38.374937 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:50:38.373021 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/08f62205-b31d-4863-ab9e-3b13b15fbef5-istio-data\") pod \"e2e-worker-w1-88bbf7-openshift-default-574f47fff7-rf5cn\" (UID: \"08f62205-b31d-4863-ab9e-3b13b15fbef5\") " pod="openshift-ingress/e2e-worker-w1-88bbf7-openshift-default-574f47fff7-rf5cn" Nov 03 00:50:38.374937 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:50:38.374451 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 00:50:38.374937 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:50:38.374755 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/08f62205-b31d-4863-ab9e-3b13b15fbef5-credential-socket\") pod \"e2e-worker-w1-88bbf7-openshift-default-574f47fff7-rf5cn\" (UID: \"08f62205-b31d-4863-ab9e-3b13b15fbef5\") " pod="openshift-ingress/e2e-worker-w1-88bbf7-openshift-default-574f47fff7-rf5cn" Nov 03 00:50:38.376133 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:50:38.375071 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/08f62205-b31d-4863-ab9e-3b13b15fbef5-workload-certs\") pod \"e2e-worker-w1-88bbf7-openshift-default-574f47fff7-rf5cn\" (UID: \"08f62205-b31d-4863-ab9e-3b13b15fbef5\") " pod="openshift-ingress/e2e-worker-w1-88bbf7-openshift-default-574f47fff7-rf5cn" Nov 03 00:50:38.395948 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:50:38.377839 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/08f62205-b31d-4863-ab9e-3b13b15fbef5-istiod-ca-cert\") pod \"e2e-worker-w1-88bbf7-openshift-default-574f47fff7-rf5cn\" (UID: \"08f62205-b31d-4863-ab9e-3b13b15fbef5\") " pod="openshift-ingress/e2e-worker-w1-88bbf7-openshift-default-574f47fff7-rf5cn" Nov 03 00:50:38.395948 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:50:38.382156 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/08f62205-b31d-4863-ab9e-3b13b15fbef5-istio-podinfo\") pod \"e2e-worker-w1-88bbf7-openshift-default-574f47fff7-rf5cn\" (UID: \"08f62205-b31d-4863-ab9e-3b13b15fbef5\") " pod="openshift-ingress/e2e-worker-w1-88bbf7-openshift-default-574f47fff7-rf5cn" Nov 03 00:50:38.395948 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:50:38.393598 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fq5xx\" (UniqueName: \"kubernetes.io/projected/08f62205-b31d-4863-ab9e-3b13b15fbef5-kube-api-access-fq5xx\") pod \"e2e-worker-w1-88bbf7-openshift-default-574f47fff7-rf5cn\" (UID: \"08f62205-b31d-4863-ab9e-3b13b15fbef5\") " pod="openshift-ingress/e2e-worker-w1-88bbf7-openshift-default-574f47fff7-rf5cn" Nov 03 00:50:38.410933 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:50:38.407322 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/08f62205-b31d-4863-ab9e-3b13b15fbef5-istio-envoy\") pod \"e2e-worker-w1-88bbf7-openshift-default-574f47fff7-rf5cn\" (UID: \"08f62205-b31d-4863-ab9e-3b13b15fbef5\") " pod="openshift-ingress/e2e-worker-w1-88bbf7-openshift-default-574f47fff7-rf5cn" Nov 03 00:50:38.421933 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:50:38.411340 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/08f62205-b31d-4863-ab9e-3b13b15fbef5-istio-token\") pod \"e2e-worker-w1-88bbf7-openshift-default-574f47fff7-rf5cn\" (UID: \"08f62205-b31d-4863-ab9e-3b13b15fbef5\") " pod="openshift-ingress/e2e-worker-w1-88bbf7-openshift-default-574f47fff7-rf5cn" Nov 03 00:50:38.507148 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:50:38.507054 4939 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-88bbf7-openshift-default-574f47fff7-rf5cn" Nov 03 00:50:38.725134 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:50:38.725101 4939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-83755a-84f6d788fc-bgk7t" event={"ID":"68597924-b259-47a2-b006-0d94dcb6315f","Type":"ContainerStarted","Data":"c47b18d53a2f3cfa9a35572473600e99d0a508ff19b12d7209d16df6fced9a52"} Nov 03 00:50:38.725278 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:50:38.725162 4939 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-83755a-84f6d788fc-bgk7t" Nov 03 00:50:38.744230 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:50:38.744178 4939 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-83755a-84f6d788fc-bgk7t" podStartSLOduration=2.2549794260000002 podStartE2EDuration="2.744162917s" totalImagesPullingTime="489.183491ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:50:36 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:50:38.743503151 +0000 UTC m=+587.007838096" watchObservedRunningTime="2026-11-03 00:50:38.744162917 +0000 UTC m=+587.008497862" Nov 03 00:50:38.795388 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:50:38.795357 4939 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-88bbf7-openshift-default-574f47fff7-rf5cn"] Nov 03 00:50:38.796854 ip-10-0-140-107 kubenswrapper[4939]: W1103 00:50:38.796821 4939 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod08f62205_b31d_4863_ab9e_3b13b15fbef5.slice/crio-11fc00cb2213c4a3846cc8e0d4aacd2ab70d7747e57f04e0d64c759393dff938 WatchSource:0}: Error finding container 11fc00cb2213c4a3846cc8e0d4aacd2ab70d7747e57f04e0d64c759393dff938: Status 404 returned error can't find the container with id 11fc00cb2213c4a3846cc8e0d4aacd2ab70d7747e57f04e0d64c759393dff938 Nov 03 00:50:38.799228 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:50:38.799187 4939 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 00:50:38.799332 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:50:38.799314 4939 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 00:50:38.799410 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:50:38.799362 4939 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 00:50:39.235750 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:50:39.235716 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 00:50:39.317397 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:50:39.317328 4939 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-derive-4aec6006-75dd64d785-ndfg6" podUID="cf2f1c18-14f6-42fa-a6ff-949c1269c429" containerName="payload-processing" containerID="cri-o://fbd949163f8db5e57e3d945d6a0bbacb91ad74dea35b94963d518866b5625f2c" gracePeriod=25 Nov 03 00:50:39.610796 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:50:39.610764 4939 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-derive-4aec6006-75dd64d785-ndfg6" Nov 03 00:50:39.697303 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:50:39.697270 4939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cf2f1c18-14f6-42fa-a6ff-949c1269c429-kube-api-access-v8ffl\" (UniqueName: \"kubernetes.io/projected/cf2f1c18-14f6-42fa-a6ff-949c1269c429-kube-api-access-v8ffl\") pod \"cf2f1c18-14f6-42fa-a6ff-949c1269c429\" (UID: \"cf2f1c18-14f6-42fa-a6ff-949c1269c429\") " Nov 03 00:50:39.697491 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:50:39.697362 4939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cf2f1c18-14f6-42fa-a6ff-949c1269c429-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cf2f1c18-14f6-42fa-a6ff-949c1269c429-plugins-config-volume\") pod \"cf2f1c18-14f6-42fa-a6ff-949c1269c429\" (UID: \"cf2f1c18-14f6-42fa-a6ff-949c1269c429\") " Nov 03 00:50:39.697841 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:50:39.697813 4939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cf2f1c18-14f6-42fa-a6ff-949c1269c429-plugins-config-volume" pod "cf2f1c18-14f6-42fa-a6ff-949c1269c429" (UID: "cf2f1c18-14f6-42fa-a6ff-949c1269c429"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 00:50:39.699637 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:50:39.699613 4939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cf2f1c18-14f6-42fa-a6ff-949c1269c429-kube-api-access-v8ffl" pod "cf2f1c18-14f6-42fa-a6ff-949c1269c429" (UID: "cf2f1c18-14f6-42fa-a6ff-949c1269c429"). InnerVolumeSpecName "kube-api-access-v8ffl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 00:50:39.732276 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:50:39.732239 4939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-88bbf7-openshift-default-574f47fff7-rf5cn" event={"ID":"08f62205-b31d-4863-ab9e-3b13b15fbef5","Type":"ContainerStarted","Data":"9d890d4ea180cbf4f646aedeabe5f01cf348d5f8e216255c3d18cca4804707e6"} Nov 03 00:50:39.732435 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:50:39.732302 4939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-88bbf7-openshift-default-574f47fff7-rf5cn" event={"ID":"08f62205-b31d-4863-ab9e-3b13b15fbef5","Type":"ContainerStarted","Data":"11fc00cb2213c4a3846cc8e0d4aacd2ab70d7747e57f04e0d64c759393dff938"} Nov 03 00:50:39.733608 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:50:39.733576 4939 generic.go:356] "Generic (PLEG): container finished" podID="cf2f1c18-14f6-42fa-a6ff-949c1269c429" containerID="fbd949163f8db5e57e3d945d6a0bbacb91ad74dea35b94963d518866b5625f2c" exitCode=0 Nov 03 00:50:39.733730 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:50:39.733624 4939 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-derive-4aec6006-75dd64d785-ndfg6" Nov 03 00:50:39.733730 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:50:39.733653 4939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-derive-4aec6006-75dd64d785-ndfg6" event={"ID":"cf2f1c18-14f6-42fa-a6ff-949c1269c429","Type":"ContainerDied","Data":"fbd949163f8db5e57e3d945d6a0bbacb91ad74dea35b94963d518866b5625f2c"} Nov 03 00:50:39.733730 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:50:39.733689 4939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-derive-4aec6006-75dd64d785-ndfg6" event={"ID":"cf2f1c18-14f6-42fa-a6ff-949c1269c429","Type":"ContainerDied","Data":"52e001e5d3cf4cf92bd05e613e8dfbff6c62608fb726ab8232ef39f69e73bbb9"} Nov 03 00:50:39.733730 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:50:39.733709 4939 scope.go:122] "RemoveContainer" containerID="fbd949163f8db5e57e3d945d6a0bbacb91ad74dea35b94963d518866b5625f2c" Nov 03 00:50:39.742821 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:50:39.742804 4939 scope.go:122] "RemoveContainer" containerID="fbd949163f8db5e57e3d945d6a0bbacb91ad74dea35b94963d518866b5625f2c" Nov 03 00:50:39.743119 ip-10-0-140-107 kubenswrapper[4939]: E1103 00:50:39.743100 4939 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fbd949163f8db5e57e3d945d6a0bbacb91ad74dea35b94963d518866b5625f2c\": container with ID starting with fbd949163f8db5e57e3d945d6a0bbacb91ad74dea35b94963d518866b5625f2c not found: ID does not exist" containerID="fbd949163f8db5e57e3d945d6a0bbacb91ad74dea35b94963d518866b5625f2c" Nov 03 00:50:39.743176 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:50:39.743126 4939 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fbd949163f8db5e57e3d945d6a0bbacb91ad74dea35b94963d518866b5625f2c"} err="failed to get container status \"fbd949163f8db5e57e3d945d6a0bbacb91ad74dea35b94963d518866b5625f2c\": rpc error: code = NotFound desc = could not find container \"fbd949163f8db5e57e3d945d6a0bbacb91ad74dea35b94963d518866b5625f2c\": container with ID starting with fbd949163f8db5e57e3d945d6a0bbacb91ad74dea35b94963d518866b5625f2c not found: ID does not exist" Nov 03 00:50:39.774582 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:50:39.774485 4939 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-88bbf7-openshift-default-574f47fff7-rf5cn" podStartSLOduration=1.774463106 podStartE2EDuration="1.774463106s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:50:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:50:39.772509928 +0000 UTC m=+588.036844875" watchObservedRunningTime="2026-11-03 00:50:39.774463106 +0000 UTC m=+588.038798051" Nov 03 00:50:39.789802 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:50:39.789772 4939 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-4aec6006-75dd64d785-ndfg6"] Nov 03 00:50:39.794941 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:50:39.794898 4939 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-4aec6006-75dd64d785-ndfg6"] Nov 03 00:50:39.799237 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:50:39.799211 4939 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cf2f1c18-14f6-42fa-a6ff-949c1269c429-plugins-config-volume\") on node \"ip-10-0-140-107.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 00:50:39.799237 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:50:39.799233 4939 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v8ffl\" (UniqueName: \"kubernetes.io/projected/cf2f1c18-14f6-42fa-a6ff-949c1269c429-kube-api-access-v8ffl\") on node \"ip-10-0-140-107.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 00:50:39.919989 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:50:39.919955 4939 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cf2f1c18-14f6-42fa-a6ff-949c1269c429" path="/var/lib/kubelet/pods/cf2f1c18-14f6-42fa-a6ff-949c1269c429/volumes" Nov 03 00:50:40.013747 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:50:40.013712 4939 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w0-2f9a3a-7b485fd99-hjcmm"] Nov 03 00:50:40.014343 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:50:40.014259 4939 cpu_manager.go:414] "containerMap: removing container" podUID="cf2f1c18-14f6-42fa-a6ff-949c1269c429" containerName="payload-processing" Nov 03 00:50:40.014343 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:50:40.014281 4939 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cf2f1c18-14f6-42fa-a6ff-949c1269c429" containerName="payload-processing" Nov 03 00:50:40.014519 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:50:40.014418 4939 memory_manager.go:365] "RemoveStaleState removing state" podUID="cf2f1c18-14f6-42fa-a6ff-949c1269c429" containerName="payload-processing" Nov 03 00:50:40.019034 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:50:40.019013 4939 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-2f9a3a-7b485fd99-hjcmm" Nov 03 00:50:40.028337 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:50:40.028261 4939 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-worker-w0-2f9a3a-serving-cert\"" Nov 03 00:50:40.030966 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:50:40.030942 4939 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w0-2f9a3a-7b485fd99-hjcmm"] Nov 03 00:50:40.104595 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:50:40.104558 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5240e2be-d3c2-45ef-8165-aec738684748-openshift-service-ca\") pod \"maas-api-e2e-worker-w0-2f9a3a-7b485fd99-hjcmm\" (UID: \"5240e2be-d3c2-45ef-8165-aec738684748\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-2f9a3a-7b485fd99-hjcmm" Nov 03 00:50:40.104746 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:50:40.104680 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zl5rf\" (UniqueName: \"kubernetes.io/projected/5240e2be-d3c2-45ef-8165-aec738684748-kube-api-access-zl5rf\") pod \"maas-api-e2e-worker-w0-2f9a3a-7b485fd99-hjcmm\" (UID: \"5240e2be-d3c2-45ef-8165-aec738684748\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-2f9a3a-7b485fd99-hjcmm" Nov 03 00:50:40.104799 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:50:40.104757 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5240e2be-d3c2-45ef-8165-aec738684748-maas-api-tls\") pod \"maas-api-e2e-worker-w0-2f9a3a-7b485fd99-hjcmm\" (UID: \"5240e2be-d3c2-45ef-8165-aec738684748\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-2f9a3a-7b485fd99-hjcmm" Nov 03 00:50:40.104860 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:50:40.104843 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5240e2be-d3c2-45ef-8165-aec738684748-metrics-certs\") pod \"maas-api-e2e-worker-w0-2f9a3a-7b485fd99-hjcmm\" (UID: \"5240e2be-d3c2-45ef-8165-aec738684748\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-2f9a3a-7b485fd99-hjcmm" Nov 03 00:50:40.106361 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:50:40.106335 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 00:50:40.205595 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:50:40.205554 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5240e2be-d3c2-45ef-8165-aec738684748-openshift-service-ca\") pod \"maas-api-e2e-worker-w0-2f9a3a-7b485fd99-hjcmm\" (UID: \"5240e2be-d3c2-45ef-8165-aec738684748\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-2f9a3a-7b485fd99-hjcmm" Nov 03 00:50:40.205982 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:50:40.205955 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zl5rf\" (UniqueName: \"kubernetes.io/projected/5240e2be-d3c2-45ef-8165-aec738684748-kube-api-access-zl5rf\") pod \"maas-api-e2e-worker-w0-2f9a3a-7b485fd99-hjcmm\" (UID: \"5240e2be-d3c2-45ef-8165-aec738684748\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-2f9a3a-7b485fd99-hjcmm" Nov 03 00:50:40.206078 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:50:40.206043 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5240e2be-d3c2-45ef-8165-aec738684748-maas-api-tls\") pod \"maas-api-e2e-worker-w0-2f9a3a-7b485fd99-hjcmm\" (UID: \"5240e2be-d3c2-45ef-8165-aec738684748\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-2f9a3a-7b485fd99-hjcmm" Nov 03 00:50:40.206121 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:50:40.206092 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5240e2be-d3c2-45ef-8165-aec738684748-metrics-certs\") pod \"maas-api-e2e-worker-w0-2f9a3a-7b485fd99-hjcmm\" (UID: \"5240e2be-d3c2-45ef-8165-aec738684748\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-2f9a3a-7b485fd99-hjcmm" Nov 03 00:50:40.206337 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:50:40.206314 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5240e2be-d3c2-45ef-8165-aec738684748-openshift-service-ca\") pod \"maas-api-e2e-worker-w0-2f9a3a-7b485fd99-hjcmm\" (UID: \"5240e2be-d3c2-45ef-8165-aec738684748\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-2f9a3a-7b485fd99-hjcmm" Nov 03 00:50:40.208771 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:50:40.208752 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5240e2be-d3c2-45ef-8165-aec738684748-maas-api-tls\") pod \"maas-api-e2e-worker-w0-2f9a3a-7b485fd99-hjcmm\" (UID: \"5240e2be-d3c2-45ef-8165-aec738684748\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-2f9a3a-7b485fd99-hjcmm" Nov 03 00:50:40.208943 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:50:40.208923 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5240e2be-d3c2-45ef-8165-aec738684748-metrics-certs\") pod \"maas-api-e2e-worker-w0-2f9a3a-7b485fd99-hjcmm\" (UID: \"5240e2be-d3c2-45ef-8165-aec738684748\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-2f9a3a-7b485fd99-hjcmm" Nov 03 00:50:40.213755 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:50:40.213729 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zl5rf\" (UniqueName: \"kubernetes.io/projected/5240e2be-d3c2-45ef-8165-aec738684748-kube-api-access-zl5rf\") pod \"maas-api-e2e-worker-w0-2f9a3a-7b485fd99-hjcmm\" (UID: \"5240e2be-d3c2-45ef-8165-aec738684748\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-2f9a3a-7b485fd99-hjcmm" Nov 03 00:50:40.223742 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:50:40.223714 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 00:50:40.329704 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:50:40.329602 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 00:50:40.336283 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:50:40.336259 4939 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-2f9a3a-7b485fd99-hjcmm" Nov 03 00:50:40.508298 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:50:40.508259 4939 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-88bbf7-openshift-default-574f47fff7-rf5cn" Nov 03 00:50:40.513439 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:50:40.513408 4939 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-88bbf7-openshift-default-574f47fff7-rf5cn" Nov 03 00:50:40.636823 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:50:40.636783 4939 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w0-2f9a3a-7b485fd99-hjcmm"] Nov 03 00:50:40.637723 ip-10-0-140-107 kubenswrapper[4939]: W1103 00:50:40.637695 4939 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5240e2be_d3c2_45ef_8165_aec738684748.slice/crio-67aca44a2d6b34ef859f70bbe52c36c2344a99802ebba096023b894fdb3fd2d4 WatchSource:0}: Error finding container 67aca44a2d6b34ef859f70bbe52c36c2344a99802ebba096023b894fdb3fd2d4: Status 404 returned error can't find the container with id 67aca44a2d6b34ef859f70bbe52c36c2344a99802ebba096023b894fdb3fd2d4 Nov 03 00:50:40.773932 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:50:40.765111 4939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-2f9a3a-7b485fd99-hjcmm" event={"ID":"5240e2be-d3c2-45ef-8165-aec738684748","Type":"ContainerStarted","Data":"67aca44a2d6b34ef859f70bbe52c36c2344a99802ebba096023b894fdb3fd2d4"} Nov 03 00:50:40.773932 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:50:40.771723 4939 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-88bbf7-openshift-default-574f47fff7-rf5cn" Nov 03 00:50:40.784957 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:50:40.783217 4939 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-88bbf7-openshift-default-574f47fff7-rf5cn" Nov 03 00:50:41.075804 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:50:41.075722 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 00:50:41.325424 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:50:41.325388 4939 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-4pw7v"] Nov 03 00:50:41.800056 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:50:41.799061 4939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-2f9a3a-7b485fd99-hjcmm" event={"ID":"5240e2be-d3c2-45ef-8165-aec738684748","Type":"ContainerStarted","Data":"b8846f80d56673691d0d98e1a01abf8a1bd755ec0ffe0b7f95cfa9d40f91724b"} Nov 03 00:50:41.800056 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:50:41.799127 4939 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-2f9a3a-7b485fd99-hjcmm" Nov 03 00:50:41.810531 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:50:41.810499 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 00:50:41.817554 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:50:41.817499 4939 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-2f9a3a-7b485fd99-hjcmm" podStartSLOduration=2.35491884 podStartE2EDuration="2.817483248s" totalImagesPullingTime="462.564408ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:50:39 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:50:41.816177026 +0000 UTC m=+590.080511972" watchObservedRunningTime="2026-11-03 00:50:41.817483248 +0000 UTC m=+590.081818196" Nov 03 00:50:42.136202 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:50:42.136174 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 00:50:42.363064 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:50:42.363032 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 00:50:42.503045 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:50:42.502925 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 00:50:43.430163 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:50:43.430104 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 00:50:43.669935 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:50:43.668427 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 00:50:43.718992 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:50:43.716246 4939 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-650be0-58fc9b6455-9jl7f" Nov 03 00:50:43.719413 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:50:43.719385 4939 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-worker-w4-650be0-86d95657b7-vm62r" Nov 03 00:50:43.905094 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:50:43.905050 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 00:50:44.669570 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:50:44.669369 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 00:50:44.740574 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:50:44.740535 4939 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-83755a-84f6d788fc-bgk7t" Nov 03 00:50:45.630129 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:50:45.630079 4939 patch_prober.go:29] interesting pod/e2e-worker-w4-650be0-openshift-default-5c7c5fb7d9-tkzdb container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.53:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 00:50:45.630408 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:50:45.630157 4939 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w4-650be0-openshift-default-5c7c5fb7d9-tkzdb" podUID="4da03a65-9706-40b4-955f-6e628901b6be" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.53:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 00:50:45.695517 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:50:45.695477 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 00:50:46.918205 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:50:46.918169 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 00:50:47.035503 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:50:47.035466 4939 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-4pw7v"] Nov 03 00:50:47.096574 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:50:47.096540 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 00:50:47.305598 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:50:47.305522 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 00:50:47.813451 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:50:47.813336 4939 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-2f9a3a-7b485fd99-hjcmm" Nov 03 00:50:48.029659 ip-10-0-140-107 kubenswrapper[4939]: E1103 00:50:48.029592 4939 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-sthjh" containerName="extractor" Nov 03 00:50:48.224658 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:50:48.224615 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 00:50:49.379541 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:50:49.379456 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 00:50:49.721100 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:50:49.721019 4939 patch_prober.go:29] interesting pod/e2e-worker-w3-83755a-openshift-default-c86f9565c-shrff container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.54:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 00:50:49.722238 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:50:49.721097 4939 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w3-83755a-openshift-default-c86f9565c-shrff" podUID="30435a8a-8666-4929-b02e-02a6bb02d878" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.54:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 00:50:49.857077 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:50:49.855703 4939 generic.go:356] "Generic (PLEG): container finished" podID="4da03a65-9706-40b4-955f-6e628901b6be" containerID="b60c499c7f01628148ff6755b3ae78bf771351dcc2d9253dfbd6c45015e55519" exitCode=0 Nov 03 00:50:49.857077 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:50:49.855850 4939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-650be0-openshift-default-5c7c5fb7d9-tkzdb" event={"ID":"4da03a65-9706-40b4-955f-6e628901b6be","Type":"ContainerDied","Data":"b60c499c7f01628148ff6755b3ae78bf771351dcc2d9253dfbd6c45015e55519"} Nov 03 00:50:49.857077 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:50:49.856363 4939 scope.go:122] "RemoveContainer" containerID="b60c499c7f01628148ff6755b3ae78bf771351dcc2d9253dfbd6c45015e55519" Nov 03 00:50:49.859768 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:50:49.858492 4939 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 00:50:49.859768 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:50:49.858585 4939 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 00:50:49.859768 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:50:49.858638 4939 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 00:50:50.327004 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:50:50.317819 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 00:50:50.564061 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:50:50.558499 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 00:50:50.790934 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:50:50.787483 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 00:50:50.886932 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:50:50.886562 4939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-650be0-openshift-default-5c7c5fb7d9-tkzdb" event={"ID":"4da03a65-9706-40b4-955f-6e628901b6be","Type":"ContainerStarted","Data":"30bc7f034396870ec020bebe35ea11831db41e782506d32f1b37ace2348c0a85"} Nov 03 00:50:50.935867 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:50:50.933769 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 00:50:50.938883 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:50:50.938856 4939 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-650be0-openshift-default-5c7c5fb7d9-tkzdb" Nov 03 00:50:50.940817 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:50:50.939142 4939 patch_prober.go:29] interesting pod/e2e-worker-w4-650be0-openshift-default-5c7c5fb7d9-tkzdb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.53:15021/healthz/ready\": dial tcp 10.132.0.53:15021: connect: connection refused" start-of-body= Nov 03 00:50:50.940817 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:50:50.939254 4939 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-650be0-openshift-default-5c7c5fb7d9-tkzdb" podUID="4da03a65-9706-40b4-955f-6e628901b6be" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.53:15021/healthz/ready\": dial tcp 10.132.0.53:15021: connect: connection refused" Nov 03 00:50:51.106933 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:50:51.106719 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 00:50:51.827039 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:50:51.826894 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 00:50:51.937949 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:50:51.937901 4939 patch_prober.go:29] interesting pod/e2e-worker-w4-650be0-openshift-default-5c7c5fb7d9-tkzdb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.53:15021/healthz/ready\": dial tcp 10.132.0.53:15021: connect: connection refused" start-of-body= Nov 03 00:50:51.938227 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:50:51.938203 4939 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-650be0-openshift-default-5c7c5fb7d9-tkzdb" podUID="4da03a65-9706-40b4-955f-6e628901b6be" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.53:15021/healthz/ready\": dial tcp 10.132.0.53:15021: connect: connection refused" Nov 03 00:50:52.077164 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:50:52.045613 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 00:50:52.077994 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:50:52.077892 4939 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-n42s6_65d134ac-8a25-4c67-b6c7-cd752a098daf/console-operator/2.log" Nov 03 00:50:52.113097 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:50:52.110714 4939 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-n42s6_65d134ac-8a25-4c67-b6c7-cd752a098daf/console-operator/2.log" Nov 03 00:50:52.366934 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:50:52.359198 4939 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-4pw7v"] Nov 03 00:50:52.938739 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:50:52.938699 4939 patch_prober.go:29] interesting pod/e2e-worker-w4-650be0-openshift-default-5c7c5fb7d9-tkzdb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.53:15021/healthz/ready\": dial tcp 10.132.0.53:15021: connect: connection refused" start-of-body= Nov 03 00:50:52.939231 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:50:52.938763 4939 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-650be0-openshift-default-5c7c5fb7d9-tkzdb" podUID="4da03a65-9706-40b4-955f-6e628901b6be" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.53:15021/healthz/ready\": dial tcp 10.132.0.53:15021: connect: connection refused" Nov 03 00:50:53.211984 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:50:53.211096 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 00:50:53.367490 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:50:53.367367 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 00:50:53.938392 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:50:53.937932 4939 patch_prober.go:29] interesting pod/e2e-worker-w4-650be0-openshift-default-5c7c5fb7d9-tkzdb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.53:15021/healthz/ready\": dial tcp 10.132.0.53:15021: connect: connection refused" start-of-body= Nov 03 00:50:53.938392 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:50:53.937996 4939 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-650be0-openshift-default-5c7c5fb7d9-tkzdb" podUID="4da03a65-9706-40b4-955f-6e628901b6be" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.53:15021/healthz/ready\": dial tcp 10.132.0.53:15021: connect: connection refused" Nov 03 00:50:53.944248 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:50:53.944210 4939 generic.go:356] "Generic (PLEG): container finished" podID="30435a8a-8666-4929-b02e-02a6bb02d878" containerID="84a6e26ec7fe983ad1671971374185c5be4b64c87d96563859ff05d313517926" exitCode=0 Nov 03 00:50:53.944747 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:50:53.944272 4939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-83755a-openshift-default-c86f9565c-shrff" event={"ID":"30435a8a-8666-4929-b02e-02a6bb02d878","Type":"ContainerDied","Data":"84a6e26ec7fe983ad1671971374185c5be4b64c87d96563859ff05d313517926"} Nov 03 00:50:53.944747 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:50:53.944714 4939 scope.go:122] "RemoveContainer" containerID="84a6e26ec7fe983ad1671971374185c5be4b64c87d96563859ff05d313517926" Nov 03 00:50:53.947247 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:50:53.946367 4939 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 00:50:53.947247 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:50:53.946457 4939 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 00:50:53.947247 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:50:53.946510 4939 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 00:50:53.985847 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:50:53.985736 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 00:50:54.114944 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:50:54.114395 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 00:50:54.265932 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:50:54.265752 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 00:50:54.427364 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:50:54.427329 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 00:50:54.815951 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:50:54.807791 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 00:50:54.937933 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:50:54.937615 4939 patch_prober.go:29] interesting pod/e2e-worker-w4-650be0-openshift-default-5c7c5fb7d9-tkzdb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.53:15021/healthz/ready\": dial tcp 10.132.0.53:15021: connect: connection refused" start-of-body= Nov 03 00:50:54.937933 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:50:54.937677 4939 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-650be0-openshift-default-5c7c5fb7d9-tkzdb" podUID="4da03a65-9706-40b4-955f-6e628901b6be" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.53:15021/healthz/ready\": dial tcp 10.132.0.53:15021: connect: connection refused" Nov 03 00:50:54.998934 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:50:54.998299 4939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-83755a-openshift-default-c86f9565c-shrff" event={"ID":"30435a8a-8666-4929-b02e-02a6bb02d878","Type":"ContainerStarted","Data":"93d16e09d35069a5b5b048f130ba15a456d72265fe9958895510c5111f8a4eda"} Nov 03 00:50:55.019933 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:50:55.015968 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 00:50:55.808591 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:50:55.808553 4939 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-83755a-openshift-default-c86f9565c-shrff" Nov 03 00:50:55.809946 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:50:55.809290 4939 patch_prober.go:29] interesting pod/e2e-worker-w3-83755a-openshift-default-c86f9565c-shrff container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.54:15021/healthz/ready\": dial tcp 10.132.0.54:15021: connect: connection refused" start-of-body= Nov 03 00:50:55.809946 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:50:55.809342 4939 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-83755a-openshift-default-c86f9565c-shrff" podUID="30435a8a-8666-4929-b02e-02a6bb02d878" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.54:15021/healthz/ready\": dial tcp 10.132.0.54:15021: connect: connection refused" Nov 03 00:50:55.937719 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:50:55.937686 4939 patch_prober.go:29] interesting pod/e2e-worker-w4-650be0-openshift-default-5c7c5fb7d9-tkzdb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.53:15021/healthz/ready\": dial tcp 10.132.0.53:15021: connect: connection refused" start-of-body= Nov 03 00:50:55.938007 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:50:55.937875 4939 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-650be0-openshift-default-5c7c5fb7d9-tkzdb" podUID="4da03a65-9706-40b4-955f-6e628901b6be" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.53:15021/healthz/ready\": dial tcp 10.132.0.53:15021: connect: connection refused" Nov 03 00:50:56.192549 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:50:56.192467 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 00:50:56.478700 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:50:56.478468 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 00:50:56.792041 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:50:56.791715 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 00:50:56.792041 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:50:56.791852 4939 patch_prober.go:29] interesting pod/e2e-worker-w3-83755a-openshift-default-c86f9565c-shrff container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.54:15021/healthz/ready\": dial tcp 10.132.0.54:15021: connect: connection refused" start-of-body= Nov 03 00:50:56.792041 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:50:56.791945 4939 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-83755a-openshift-default-c86f9565c-shrff" podUID="30435a8a-8666-4929-b02e-02a6bb02d878" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.54:15021/healthz/ready\": dial tcp 10.132.0.54:15021: connect: connection refused" Nov 03 00:50:56.913992 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:50:56.913960 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 00:50:57.032989 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:50:57.032884 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 00:50:57.247286 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:50:57.247242 4939 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-650be0-openshift-default-5c7c5fb7d9-tkzdb" Nov 03 00:50:57.248953 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:50:57.248857 4939 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-650be0-openshift-default-5c7c5fb7d9-tkzdb" Nov 03 00:50:57.252993 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:50:57.252923 4939 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-650be0-openshift-default-5c7c5fb7d9-tkzdb" Nov 03 00:50:57.578166 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:50:57.578010 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 00:50:57.767550 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:50:57.767449 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 00:50:57.796847 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:50:57.796337 4939 patch_prober.go:29] interesting pod/e2e-worker-w3-83755a-openshift-default-c86f9565c-shrff container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.54:15021/healthz/ready\": dial tcp 10.132.0.54:15021: connect: connection refused" start-of-body= Nov 03 00:50:57.796847 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:50:57.796447 4939 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-83755a-openshift-default-c86f9565c-shrff" podUID="30435a8a-8666-4929-b02e-02a6bb02d878" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.54:15021/healthz/ready\": dial tcp 10.132.0.54:15021: connect: connection refused" Nov 03 00:50:57.889760 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:50:57.889708 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 00:50:57.920982 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:50:57.920945 4939 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-4pw7v"] Nov 03 00:50:58.730057 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:50:58.730025 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 00:50:58.792526 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:50:58.792486 4939 patch_prober.go:29] interesting pod/e2e-worker-w3-83755a-openshift-default-c86f9565c-shrff container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.54:15021/healthz/ready\": dial tcp 10.132.0.54:15021: connect: connection refused" start-of-body= Nov 03 00:50:58.792704 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:50:58.792547 4939 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-83755a-openshift-default-c86f9565c-shrff" podUID="30435a8a-8666-4929-b02e-02a6bb02d878" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.54:15021/healthz/ready\": dial tcp 10.132.0.54:15021: connect: connection refused" Nov 03 00:50:59.035486 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:50:59.035404 4939 generic.go:356] "Generic (PLEG): container finished" podID="80e0f915-1a1a-4214-92f0-5eeb806be96e" containerID="daffb5100933e1f21d8c999ff6a9c75480633c4dfc1f5014ebff8b4f61f3e10a" exitCode=0 Nov 03 00:50:59.036014 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:50:59.035989 4939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-2f9a3a-openshift-default-7f6f74cf97-dxffl" event={"ID":"80e0f915-1a1a-4214-92f0-5eeb806be96e","Type":"ContainerDied","Data":"daffb5100933e1f21d8c999ff6a9c75480633c4dfc1f5014ebff8b4f61f3e10a"} Nov 03 00:50:59.036386 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:50:59.036366 4939 scope.go:122] "RemoveContainer" containerID="daffb5100933e1f21d8c999ff6a9c75480633c4dfc1f5014ebff8b4f61f3e10a" Nov 03 00:50:59.037743 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:50:59.037714 4939 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 00:50:59.037817 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:50:59.037780 4939 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 00:50:59.037817 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:50:59.037812 4939 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 00:50:59.472942 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:50:59.470767 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 00:50:59.605212 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:50:59.605074 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 00:50:59.799983 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:50:59.799363 4939 patch_prober.go:29] interesting pod/e2e-worker-w3-83755a-openshift-default-c86f9565c-shrff container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.54:15021/healthz/ready\": dial tcp 10.132.0.54:15021: connect: connection refused" start-of-body= Nov 03 00:50:59.799983 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:50:59.799428 4939 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-83755a-openshift-default-c86f9565c-shrff" podUID="30435a8a-8666-4929-b02e-02a6bb02d878" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.54:15021/healthz/ready\": dial tcp 10.132.0.54:15021: connect: connection refused" Nov 03 00:51:00.042560 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:51:00.042526 4939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-2f9a3a-openshift-default-7f6f74cf97-dxffl" event={"ID":"80e0f915-1a1a-4214-92f0-5eeb806be96e","Type":"ContainerStarted","Data":"b9db464c8699b13b1cdd17a9bb0329c298b15fccb73d53481ccb96658a2c707f"} Nov 03 00:51:00.368931 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:51:00.365536 4939 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-2f9a3a-openshift-default-7f6f74cf97-dxffl" Nov 03 00:51:00.368931 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:51:00.365684 4939 patch_prober.go:29] interesting pod/e2e-worker-w0-2f9a3a-openshift-default-7f6f74cf97-dxffl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.57:15021/healthz/ready\": dial tcp 10.132.0.57:15021: connect: connection refused" start-of-body= Nov 03 00:51:00.368931 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:51:00.365730 4939 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-2f9a3a-openshift-default-7f6f74cf97-dxffl" podUID="80e0f915-1a1a-4214-92f0-5eeb806be96e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.57:15021/healthz/ready\": dial tcp 10.132.0.57:15021: connect: connection refused" Nov 03 00:51:00.415278 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:51:00.415241 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 00:51:00.886715 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:51:00.886682 4939 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-83755a-openshift-default-c86f9565c-shrff" Nov 03 00:51:00.889776 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:51:00.887171 4939 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-83755a-openshift-default-c86f9565c-shrff" Nov 03 00:51:00.889776 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:51:00.888522 4939 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-83755a-openshift-default-c86f9565c-shrff" Nov 03 00:51:01.238309 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:51:01.238197 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 00:51:01.358982 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:51:01.358501 4939 patch_prober.go:29] interesting pod/e2e-worker-w0-2f9a3a-openshift-default-7f6f74cf97-dxffl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.57:15021/healthz/ready\": dial tcp 10.132.0.57:15021: connect: connection refused" start-of-body= Nov 03 00:51:01.358982 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:51:01.358564 4939 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-2f9a3a-openshift-default-7f6f74cf97-dxffl" podUID="80e0f915-1a1a-4214-92f0-5eeb806be96e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.57:15021/healthz/ready\": dial tcp 10.132.0.57:15021: connect: connection refused" Nov 03 00:51:01.437985 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:51:01.437950 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 00:51:02.085747 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:51:02.085711 4939 generic.go:356] "Generic (PLEG): container finished" podID="08f62205-b31d-4863-ab9e-3b13b15fbef5" containerID="9d890d4ea180cbf4f646aedeabe5f01cf348d5f8e216255c3d18cca4804707e6" exitCode=0 Nov 03 00:51:02.086189 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:51:02.085786 4939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-88bbf7-openshift-default-574f47fff7-rf5cn" event={"ID":"08f62205-b31d-4863-ab9e-3b13b15fbef5","Type":"ContainerDied","Data":"9d890d4ea180cbf4f646aedeabe5f01cf348d5f8e216255c3d18cca4804707e6"} Nov 03 00:51:02.086403 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:51:02.086384 4939 scope.go:122] "RemoveContainer" containerID="9d890d4ea180cbf4f646aedeabe5f01cf348d5f8e216255c3d18cca4804707e6" Nov 03 00:51:02.087598 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:51:02.087569 4939 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 00:51:02.087653 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:51:02.087641 4939 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 00:51:02.087698 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:51:02.087673 4939 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 00:51:02.259396 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:51:02.259324 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 00:51:02.367000 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:51:02.363314 4939 patch_prober.go:29] interesting pod/e2e-worker-w0-2f9a3a-openshift-default-7f6f74cf97-dxffl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.57:15021/healthz/ready\": dial tcp 10.132.0.57:15021: connect: connection refused" start-of-body= Nov 03 00:51:02.375965 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:51:02.372999 4939 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-2f9a3a-openshift-default-7f6f74cf97-dxffl" podUID="80e0f915-1a1a-4214-92f0-5eeb806be96e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.57:15021/healthz/ready\": dial tcp 10.132.0.57:15021: connect: connection refused" Nov 03 00:51:03.015786 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:51:03.015752 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 00:51:03.110934 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:51:03.110034 4939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-88bbf7-openshift-default-574f47fff7-rf5cn" event={"ID":"08f62205-b31d-4863-ab9e-3b13b15fbef5","Type":"ContainerStarted","Data":"b0f0b67f3e3bf35f2888c554bb137203035916673d4c5e7d02f47cb5dbb416db"} Nov 03 00:51:03.357524 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:51:03.357484 4939 patch_prober.go:29] interesting pod/e2e-worker-w0-2f9a3a-openshift-default-7f6f74cf97-dxffl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.57:15021/healthz/ready\": dial tcp 10.132.0.57:15021: connect: connection refused" start-of-body= Nov 03 00:51:03.357830 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:51:03.357785 4939 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-2f9a3a-openshift-default-7f6f74cf97-dxffl" podUID="80e0f915-1a1a-4214-92f0-5eeb806be96e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.57:15021/healthz/ready\": dial tcp 10.132.0.57:15021: connect: connection refused" Nov 03 00:51:03.508095 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:51:03.508057 4939 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-88bbf7-openshift-default-574f47fff7-rf5cn" Nov 03 00:51:03.508295 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:51:03.508274 4939 patch_prober.go:29] interesting pod/e2e-worker-w1-88bbf7-openshift-default-574f47fff7-rf5cn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.59:15021/healthz/ready\": dial tcp 10.132.0.59:15021: connect: connection refused" start-of-body= Nov 03 00:51:03.508352 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:51:03.508323 4939 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-88bbf7-openshift-default-574f47fff7-rf5cn" podUID="08f62205-b31d-4863-ab9e-3b13b15fbef5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.59:15021/healthz/ready\": dial tcp 10.132.0.59:15021: connect: connection refused" Nov 03 00:51:03.921208 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:51:03.921178 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 00:51:04.360932 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:51:04.358300 4939 patch_prober.go:29] interesting pod/e2e-worker-w0-2f9a3a-openshift-default-7f6f74cf97-dxffl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.57:15021/healthz/ready\": dial tcp 10.132.0.57:15021: connect: connection refused" start-of-body= Nov 03 00:51:04.360932 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:51:04.358366 4939 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-2f9a3a-openshift-default-7f6f74cf97-dxffl" podUID="80e0f915-1a1a-4214-92f0-5eeb806be96e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.57:15021/healthz/ready\": dial tcp 10.132.0.57:15021: connect: connection refused" Nov 03 00:51:04.507788 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:51:04.507747 4939 patch_prober.go:29] interesting pod/e2e-worker-w1-88bbf7-openshift-default-574f47fff7-rf5cn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.59:15021/healthz/ready\": dial tcp 10.132.0.59:15021: connect: connection refused" start-of-body= Nov 03 00:51:04.507991 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:51:04.507800 4939 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-88bbf7-openshift-default-574f47fff7-rf5cn" podUID="08f62205-b31d-4863-ab9e-3b13b15fbef5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.59:15021/healthz/ready\": dial tcp 10.132.0.59:15021: connect: connection refused" Nov 03 00:51:04.586933 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:51:04.570346 4939 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w0-2f9a3a/simulator-w0-2f9a3a-kserve-9596bbb8f-2trgz"] Nov 03 00:51:04.607888 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:51:04.607859 4939 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-2f9a3a/simulator-w0-2f9a3a-kserve-9596bbb8f-2trgz" Nov 03 00:51:04.610742 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:51:04.607753 4939 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-2f9a3a/simulator-w0-2f9a3a-kserve-9596bbb8f-2trgz"] Nov 03 00:51:04.615662 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:51:04.615501 4939 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-2f9a3a\"/\"simulator-w0-2f9a3a-kserve-self-signed-certs\"" Nov 03 00:51:04.617382 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:51:04.617317 4939 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w0-2f9a3a\"/\"kube-root-ca.crt\"" Nov 03 00:51:04.617382 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:51:04.617362 4939 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w0-2f9a3a\"/\"openshift-service-ca.crt\"" Nov 03 00:51:04.618023 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:51:04.618004 4939 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-2f9a3a\"/\"default-dockercfg-tf6nn\"" Nov 03 00:51:04.639049 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:51:04.639023 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 00:51:04.689935 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:51:04.689341 4939 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-2f9a3a-openshift-default-7f6f74cf97-dxffl" Nov 03 00:51:04.708067 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:51:04.707931 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4qvwx\" (UniqueName: \"kubernetes.io/projected/207e2207-fb6c-4076-95df-72e95671c261-kube-api-access-4qvwx\") pod \"simulator-w0-2f9a3a-kserve-9596bbb8f-2trgz\" (UID: \"207e2207-fb6c-4076-95df-72e95671c261\") " pod="e2e-models-e2e-worker-w0-2f9a3a/simulator-w0-2f9a3a-kserve-9596bbb8f-2trgz" Nov 03 00:51:04.708067 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:51:04.707999 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/207e2207-fb6c-4076-95df-72e95671c261-tls-certs\") pod \"simulator-w0-2f9a3a-kserve-9596bbb8f-2trgz\" (UID: \"207e2207-fb6c-4076-95df-72e95671c261\") " pod="e2e-models-e2e-worker-w0-2f9a3a/simulator-w0-2f9a3a-kserve-9596bbb8f-2trgz" Nov 03 00:51:04.708330 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:51:04.708156 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/207e2207-fb6c-4076-95df-72e95671c261-model-cache\") pod \"simulator-w0-2f9a3a-kserve-9596bbb8f-2trgz\" (UID: \"207e2207-fb6c-4076-95df-72e95671c261\") " pod="e2e-models-e2e-worker-w0-2f9a3a/simulator-w0-2f9a3a-kserve-9596bbb8f-2trgz" Nov 03 00:51:04.708330 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:51:04.708192 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/207e2207-fb6c-4076-95df-72e95671c261-dshm\") pod \"simulator-w0-2f9a3a-kserve-9596bbb8f-2trgz\" (UID: \"207e2207-fb6c-4076-95df-72e95671c261\") " pod="e2e-models-e2e-worker-w0-2f9a3a/simulator-w0-2f9a3a-kserve-9596bbb8f-2trgz" Nov 03 00:51:04.708330 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:51:04.708248 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/207e2207-fb6c-4076-95df-72e95671c261-home\") pod \"simulator-w0-2f9a3a-kserve-9596bbb8f-2trgz\" (UID: \"207e2207-fb6c-4076-95df-72e95671c261\") " pod="e2e-models-e2e-worker-w0-2f9a3a/simulator-w0-2f9a3a-kserve-9596bbb8f-2trgz" Nov 03 00:51:04.708330 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:51:04.708276 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/207e2207-fb6c-4076-95df-72e95671c261-tmp-dir\") pod \"simulator-w0-2f9a3a-kserve-9596bbb8f-2trgz\" (UID: \"207e2207-fb6c-4076-95df-72e95671c261\") " pod="e2e-models-e2e-worker-w0-2f9a3a/simulator-w0-2f9a3a-kserve-9596bbb8f-2trgz" Nov 03 00:51:04.782480 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:51:04.782325 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 00:51:04.813542 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:51:04.813501 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/207e2207-fb6c-4076-95df-72e95671c261-tls-certs\") pod \"simulator-w0-2f9a3a-kserve-9596bbb8f-2trgz\" (UID: \"207e2207-fb6c-4076-95df-72e95671c261\") " pod="e2e-models-e2e-worker-w0-2f9a3a/simulator-w0-2f9a3a-kserve-9596bbb8f-2trgz" Nov 03 00:51:04.813954 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:51:04.813877 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/207e2207-fb6c-4076-95df-72e95671c261-model-cache\") pod \"simulator-w0-2f9a3a-kserve-9596bbb8f-2trgz\" (UID: \"207e2207-fb6c-4076-95df-72e95671c261\") " pod="e2e-models-e2e-worker-w0-2f9a3a/simulator-w0-2f9a3a-kserve-9596bbb8f-2trgz" Nov 03 00:51:04.814124 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:51:04.814106 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/207e2207-fb6c-4076-95df-72e95671c261-dshm\") pod \"simulator-w0-2f9a3a-kserve-9596bbb8f-2trgz\" (UID: \"207e2207-fb6c-4076-95df-72e95671c261\") " pod="e2e-models-e2e-worker-w0-2f9a3a/simulator-w0-2f9a3a-kserve-9596bbb8f-2trgz" Nov 03 00:51:04.814301 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:51:04.814285 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/207e2207-fb6c-4076-95df-72e95671c261-home\") pod \"simulator-w0-2f9a3a-kserve-9596bbb8f-2trgz\" (UID: \"207e2207-fb6c-4076-95df-72e95671c261\") " pod="e2e-models-e2e-worker-w0-2f9a3a/simulator-w0-2f9a3a-kserve-9596bbb8f-2trgz" Nov 03 00:51:04.814414 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:51:04.814400 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/207e2207-fb6c-4076-95df-72e95671c261-tmp-dir\") pod \"simulator-w0-2f9a3a-kserve-9596bbb8f-2trgz\" (UID: \"207e2207-fb6c-4076-95df-72e95671c261\") " pod="e2e-models-e2e-worker-w0-2f9a3a/simulator-w0-2f9a3a-kserve-9596bbb8f-2trgz" Nov 03 00:51:04.814565 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:51:04.814548 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4qvwx\" (UniqueName: \"kubernetes.io/projected/207e2207-fb6c-4076-95df-72e95671c261-kube-api-access-4qvwx\") pod \"simulator-w0-2f9a3a-kserve-9596bbb8f-2trgz\" (UID: \"207e2207-fb6c-4076-95df-72e95671c261\") " pod="e2e-models-e2e-worker-w0-2f9a3a/simulator-w0-2f9a3a-kserve-9596bbb8f-2trgz" Nov 03 00:51:04.816167 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:51:04.816144 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/207e2207-fb6c-4076-95df-72e95671c261-model-cache\") pod \"simulator-w0-2f9a3a-kserve-9596bbb8f-2trgz\" (UID: \"207e2207-fb6c-4076-95df-72e95671c261\") " pod="e2e-models-e2e-worker-w0-2f9a3a/simulator-w0-2f9a3a-kserve-9596bbb8f-2trgz" Nov 03 00:51:04.826031 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:51:04.816540 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/207e2207-fb6c-4076-95df-72e95671c261-tmp-dir\") pod \"simulator-w0-2f9a3a-kserve-9596bbb8f-2trgz\" (UID: \"207e2207-fb6c-4076-95df-72e95671c261\") " pod="e2e-models-e2e-worker-w0-2f9a3a/simulator-w0-2f9a3a-kserve-9596bbb8f-2trgz" Nov 03 00:51:04.826205 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:51:04.816772 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/207e2207-fb6c-4076-95df-72e95671c261-home\") pod \"simulator-w0-2f9a3a-kserve-9596bbb8f-2trgz\" (UID: \"207e2207-fb6c-4076-95df-72e95671c261\") " pod="e2e-models-e2e-worker-w0-2f9a3a/simulator-w0-2f9a3a-kserve-9596bbb8f-2trgz" Nov 03 00:51:04.826306 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:51:04.818854 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/207e2207-fb6c-4076-95df-72e95671c261-tls-certs\") pod \"simulator-w0-2f9a3a-kserve-9596bbb8f-2trgz\" (UID: \"207e2207-fb6c-4076-95df-72e95671c261\") " pod="e2e-models-e2e-worker-w0-2f9a3a/simulator-w0-2f9a3a-kserve-9596bbb8f-2trgz" Nov 03 00:51:04.831562 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:51:04.831538 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4qvwx\" (UniqueName: \"kubernetes.io/projected/207e2207-fb6c-4076-95df-72e95671c261-kube-api-access-4qvwx\") pod \"simulator-w0-2f9a3a-kserve-9596bbb8f-2trgz\" (UID: \"207e2207-fb6c-4076-95df-72e95671c261\") " pod="e2e-models-e2e-worker-w0-2f9a3a/simulator-w0-2f9a3a-kserve-9596bbb8f-2trgz" Nov 03 00:51:04.838808 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:51:04.838787 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/207e2207-fb6c-4076-95df-72e95671c261-dshm\") pod \"simulator-w0-2f9a3a-kserve-9596bbb8f-2trgz\" (UID: \"207e2207-fb6c-4076-95df-72e95671c261\") " pod="e2e-models-e2e-worker-w0-2f9a3a/simulator-w0-2f9a3a-kserve-9596bbb8f-2trgz" Nov 03 00:51:04.934647 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:51:04.934564 4939 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-2f9a3a/simulator-w0-2f9a3a-kserve-9596bbb8f-2trgz" Nov 03 00:51:05.280360 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:51:05.280330 4939 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-2f9a3a/simulator-w0-2f9a3a-kserve-9596bbb8f-2trgz"] Nov 03 00:51:05.488761 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:51:05.488722 4939 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-2f9a3a-openshift-default-7f6f74cf97-dxffl" Nov 03 00:51:05.491682 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:51:05.491658 4939 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-2f9a3a-openshift-default-7f6f74cf97-dxffl" Nov 03 00:51:05.508304 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:51:05.508270 4939 patch_prober.go:29] interesting pod/e2e-worker-w1-88bbf7-openshift-default-574f47fff7-rf5cn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.59:15021/healthz/ready\": dial tcp 10.132.0.59:15021: connect: connection refused" start-of-body= Nov 03 00:51:05.508513 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:51:05.508329 4939 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-88bbf7-openshift-default-574f47fff7-rf5cn" podUID="08f62205-b31d-4863-ab9e-3b13b15fbef5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.59:15021/healthz/ready\": dial tcp 10.132.0.59:15021: connect: connection refused" Nov 03 00:51:05.600159 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:51:05.600129 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 00:51:06.127736 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:51:06.127696 4939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-2f9a3a/simulator-w0-2f9a3a-kserve-9596bbb8f-2trgz" event={"ID":"207e2207-fb6c-4076-95df-72e95671c261","Type":"ContainerStarted","Data":"dfef807da95ad9c22322d8175eec57c184708ee6486d8d5363ac97e094c8deb4"} Nov 03 00:51:06.127736 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:51:06.127742 4939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-2f9a3a/simulator-w0-2f9a3a-kserve-9596bbb8f-2trgz" event={"ID":"207e2207-fb6c-4076-95df-72e95671c261","Type":"ContainerStarted","Data":"d5ac9e84b2bffd4fe57f914fd389bbda4c03c22e4a04644567e20f8221feaa93"} Nov 03 00:51:06.145459 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:51:06.145411 4939 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w0-2f9a3a/simulator-w0-2f9a3a-kserve-9596bbb8f-2trgz" podStartSLOduration=2.145397447 podStartE2EDuration="2.145397447s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:51:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:51:06.14401157 +0000 UTC m=+614.408346515" watchObservedRunningTime="2026-11-03 00:51:06.145397447 +0000 UTC m=+614.409732391" Nov 03 00:51:06.317574 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:51:06.317537 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 00:51:06.508474 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:51:06.508382 4939 patch_prober.go:29] interesting pod/e2e-worker-w1-88bbf7-openshift-default-574f47fff7-rf5cn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.59:15021/healthz/ready\": dial tcp 10.132.0.59:15021: connect: connection refused" start-of-body= Nov 03 00:51:06.508474 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:51:06.508449 4939 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-88bbf7-openshift-default-574f47fff7-rf5cn" podUID="08f62205-b31d-4863-ab9e-3b13b15fbef5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.59:15021/healthz/ready\": dial tcp 10.132.0.59:15021: connect: connection refused" Nov 03 00:51:06.835456 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:51:06.835411 4939 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w1-88bbf7/simulator-w1-88bbf7-kserve-5fbb669f8f-42hgd"] Nov 03 00:51:06.838847 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:51:06.838830 4939 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-88bbf7/simulator-w1-88bbf7-kserve-5fbb669f8f-42hgd" Nov 03 00:51:06.875020 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:51:06.867153 4939 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w1-88bbf7\"/\"default-dockercfg-j2wgx\"" Nov 03 00:51:06.875020 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:51:06.867829 4939 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w1-88bbf7\"/\"kube-root-ca.crt\"" Nov 03 00:51:06.879415 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:51:06.879389 4939 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w1-88bbf7\"/\"openshift-service-ca.crt\"" Nov 03 00:51:06.879652 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:51:06.879626 4939 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w1-88bbf7\"/\"simulator-w1-88bbf7-kserve-self-signed-certs\"" Nov 03 00:51:06.882096 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:51:06.882068 4939 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w1-88bbf7/simulator-w1-88bbf7-kserve-5fbb669f8f-42hgd"] Nov 03 00:51:06.923740 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:51:06.923713 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 00:51:06.935674 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:51:06.935653 4939 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-2f9a3a/simulator-w0-2f9a3a-kserve-9596bbb8f-2trgz" Nov 03 00:51:06.939463 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:51:06.939431 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/83b38883-2e7a-444d-aa68-52be4a25ae66-model-cache\") pod \"simulator-w1-88bbf7-kserve-5fbb669f8f-42hgd\" (UID: \"83b38883-2e7a-444d-aa68-52be4a25ae66\") " pod="e2e-models-e2e-worker-w1-88bbf7/simulator-w1-88bbf7-kserve-5fbb669f8f-42hgd" Nov 03 00:51:06.939581 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:51:06.939536 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/83b38883-2e7a-444d-aa68-52be4a25ae66-home\") pod \"simulator-w1-88bbf7-kserve-5fbb669f8f-42hgd\" (UID: \"83b38883-2e7a-444d-aa68-52be4a25ae66\") " pod="e2e-models-e2e-worker-w1-88bbf7/simulator-w1-88bbf7-kserve-5fbb669f8f-42hgd" Nov 03 00:51:06.939767 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:51:06.939745 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/83b38883-2e7a-444d-aa68-52be4a25ae66-dshm\") pod \"simulator-w1-88bbf7-kserve-5fbb669f8f-42hgd\" (UID: \"83b38883-2e7a-444d-aa68-52be4a25ae66\") " pod="e2e-models-e2e-worker-w1-88bbf7/simulator-w1-88bbf7-kserve-5fbb669f8f-42hgd" Nov 03 00:51:06.939946 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:51:06.939898 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9qxj6\" (UniqueName: \"kubernetes.io/projected/83b38883-2e7a-444d-aa68-52be4a25ae66-kube-api-access-9qxj6\") pod \"simulator-w1-88bbf7-kserve-5fbb669f8f-42hgd\" (UID: \"83b38883-2e7a-444d-aa68-52be4a25ae66\") " pod="e2e-models-e2e-worker-w1-88bbf7/simulator-w1-88bbf7-kserve-5fbb669f8f-42hgd" Nov 03 00:51:06.939999 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:51:06.939985 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/83b38883-2e7a-444d-aa68-52be4a25ae66-tls-certs\") pod \"simulator-w1-88bbf7-kserve-5fbb669f8f-42hgd\" (UID: \"83b38883-2e7a-444d-aa68-52be4a25ae66\") " pod="e2e-models-e2e-worker-w1-88bbf7/simulator-w1-88bbf7-kserve-5fbb669f8f-42hgd" Nov 03 00:51:06.940043 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:51:06.940030 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/83b38883-2e7a-444d-aa68-52be4a25ae66-tmp-dir\") pod \"simulator-w1-88bbf7-kserve-5fbb669f8f-42hgd\" (UID: \"83b38883-2e7a-444d-aa68-52be4a25ae66\") " pod="e2e-models-e2e-worker-w1-88bbf7/simulator-w1-88bbf7-kserve-5fbb669f8f-42hgd" Nov 03 00:51:07.060962 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:51:07.043632 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/83b38883-2e7a-444d-aa68-52be4a25ae66-tmp-dir\") pod \"simulator-w1-88bbf7-kserve-5fbb669f8f-42hgd\" (UID: \"83b38883-2e7a-444d-aa68-52be4a25ae66\") " pod="e2e-models-e2e-worker-w1-88bbf7/simulator-w1-88bbf7-kserve-5fbb669f8f-42hgd" Nov 03 00:51:07.060962 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:51:07.043691 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/83b38883-2e7a-444d-aa68-52be4a25ae66-model-cache\") pod \"simulator-w1-88bbf7-kserve-5fbb669f8f-42hgd\" (UID: \"83b38883-2e7a-444d-aa68-52be4a25ae66\") " pod="e2e-models-e2e-worker-w1-88bbf7/simulator-w1-88bbf7-kserve-5fbb669f8f-42hgd" Nov 03 00:51:07.060962 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:51:07.043742 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/83b38883-2e7a-444d-aa68-52be4a25ae66-home\") pod \"simulator-w1-88bbf7-kserve-5fbb669f8f-42hgd\" (UID: \"83b38883-2e7a-444d-aa68-52be4a25ae66\") " pod="e2e-models-e2e-worker-w1-88bbf7/simulator-w1-88bbf7-kserve-5fbb669f8f-42hgd" Nov 03 00:51:07.060962 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:51:07.043810 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/83b38883-2e7a-444d-aa68-52be4a25ae66-dshm\") pod \"simulator-w1-88bbf7-kserve-5fbb669f8f-42hgd\" (UID: \"83b38883-2e7a-444d-aa68-52be4a25ae66\") " pod="e2e-models-e2e-worker-w1-88bbf7/simulator-w1-88bbf7-kserve-5fbb669f8f-42hgd" Nov 03 00:51:07.060962 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:51:07.043957 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9qxj6\" (UniqueName: \"kubernetes.io/projected/83b38883-2e7a-444d-aa68-52be4a25ae66-kube-api-access-9qxj6\") pod \"simulator-w1-88bbf7-kserve-5fbb669f8f-42hgd\" (UID: \"83b38883-2e7a-444d-aa68-52be4a25ae66\") " pod="e2e-models-e2e-worker-w1-88bbf7/simulator-w1-88bbf7-kserve-5fbb669f8f-42hgd" Nov 03 00:51:07.060962 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:51:07.043996 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/83b38883-2e7a-444d-aa68-52be4a25ae66-tls-certs\") pod \"simulator-w1-88bbf7-kserve-5fbb669f8f-42hgd\" (UID: \"83b38883-2e7a-444d-aa68-52be4a25ae66\") " pod="e2e-models-e2e-worker-w1-88bbf7/simulator-w1-88bbf7-kserve-5fbb669f8f-42hgd" Nov 03 00:51:07.060962 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:51:07.044614 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/83b38883-2e7a-444d-aa68-52be4a25ae66-home\") pod \"simulator-w1-88bbf7-kserve-5fbb669f8f-42hgd\" (UID: \"83b38883-2e7a-444d-aa68-52be4a25ae66\") " pod="e2e-models-e2e-worker-w1-88bbf7/simulator-w1-88bbf7-kserve-5fbb669f8f-42hgd" Nov 03 00:51:07.060962 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:51:07.044887 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/83b38883-2e7a-444d-aa68-52be4a25ae66-tmp-dir\") pod \"simulator-w1-88bbf7-kserve-5fbb669f8f-42hgd\" (UID: \"83b38883-2e7a-444d-aa68-52be4a25ae66\") " pod="e2e-models-e2e-worker-w1-88bbf7/simulator-w1-88bbf7-kserve-5fbb669f8f-42hgd" Nov 03 00:51:07.060962 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:51:07.045193 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/83b38883-2e7a-444d-aa68-52be4a25ae66-model-cache\") pod \"simulator-w1-88bbf7-kserve-5fbb669f8f-42hgd\" (UID: \"83b38883-2e7a-444d-aa68-52be4a25ae66\") " pod="e2e-models-e2e-worker-w1-88bbf7/simulator-w1-88bbf7-kserve-5fbb669f8f-42hgd" Nov 03 00:51:07.062316 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:51:07.062282 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/83b38883-2e7a-444d-aa68-52be4a25ae66-dshm\") pod \"simulator-w1-88bbf7-kserve-5fbb669f8f-42hgd\" (UID: \"83b38883-2e7a-444d-aa68-52be4a25ae66\") " pod="e2e-models-e2e-worker-w1-88bbf7/simulator-w1-88bbf7-kserve-5fbb669f8f-42hgd" Nov 03 00:51:07.090048 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:51:07.067532 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/83b38883-2e7a-444d-aa68-52be4a25ae66-tls-certs\") pod \"simulator-w1-88bbf7-kserve-5fbb669f8f-42hgd\" (UID: \"83b38883-2e7a-444d-aa68-52be4a25ae66\") " pod="e2e-models-e2e-worker-w1-88bbf7/simulator-w1-88bbf7-kserve-5fbb669f8f-42hgd" Nov 03 00:51:07.090048 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:51:07.074132 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9qxj6\" (UniqueName: \"kubernetes.io/projected/83b38883-2e7a-444d-aa68-52be4a25ae66-kube-api-access-9qxj6\") pod \"simulator-w1-88bbf7-kserve-5fbb669f8f-42hgd\" (UID: \"83b38883-2e7a-444d-aa68-52be4a25ae66\") " pod="e2e-models-e2e-worker-w1-88bbf7/simulator-w1-88bbf7-kserve-5fbb669f8f-42hgd" Nov 03 00:51:07.139809 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:51:07.139769 4939 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-4pw7v"] Nov 03 00:51:07.184278 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:51:07.184243 4939 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-88bbf7/simulator-w1-88bbf7-kserve-5fbb669f8f-42hgd" Nov 03 00:51:07.218333 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:51:07.218300 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 00:51:07.415179 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:51:07.415097 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 00:51:07.504411 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:51:07.504381 4939 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w1-88bbf7/simulator-w1-88bbf7-kserve-5fbb669f8f-42hgd"] Nov 03 00:51:07.507071 ip-10-0-140-107 kubenswrapper[4939]: W1103 00:51:07.507042 4939 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod83b38883_2e7a_444d_aa68_52be4a25ae66.slice/crio-c12115237becab031644b60fce23c7384f55c7176eb70bc9a37d3bd6bd484b1e WatchSource:0}: Error finding container c12115237becab031644b60fce23c7384f55c7176eb70bc9a37d3bd6bd484b1e: Status 404 returned error can't find the container with id c12115237becab031644b60fce23c7384f55c7176eb70bc9a37d3bd6bd484b1e Nov 03 00:51:07.507960 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:51:07.507934 4939 patch_prober.go:29] interesting pod/e2e-worker-w1-88bbf7-openshift-default-574f47fff7-rf5cn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.59:15021/healthz/ready\": dial tcp 10.132.0.59:15021: connect: connection refused" start-of-body= Nov 03 00:51:07.508062 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:51:07.507985 4939 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-88bbf7-openshift-default-574f47fff7-rf5cn" podUID="08f62205-b31d-4863-ab9e-3b13b15fbef5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.59:15021/healthz/ready\": dial tcp 10.132.0.59:15021: connect: connection refused" Nov 03 00:51:08.140318 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:51:08.140277 4939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-88bbf7/simulator-w1-88bbf7-kserve-5fbb669f8f-42hgd" event={"ID":"83b38883-2e7a-444d-aa68-52be4a25ae66","Type":"ContainerStarted","Data":"f6ef9c5f02772524826ae140c107b0209f1ee2cb412948574b7ba2a235ab0860"} Nov 03 00:51:08.140318 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:51:08.140323 4939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-88bbf7/simulator-w1-88bbf7-kserve-5fbb669f8f-42hgd" event={"ID":"83b38883-2e7a-444d-aa68-52be4a25ae66","Type":"ContainerStarted","Data":"c12115237becab031644b60fce23c7384f55c7176eb70bc9a37d3bd6bd484b1e"} Nov 03 00:51:08.191933 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:51:08.191688 4939 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-88bbf7/simulator-w1-88bbf7-kserve-5fbb669f8f-42hgd" Nov 03 00:51:08.284931 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:51:08.284435 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 00:51:08.507496 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:51:08.507407 4939 patch_prober.go:29] interesting pod/e2e-worker-w1-88bbf7-openshift-default-574f47fff7-rf5cn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.59:15021/healthz/ready\": dial tcp 10.132.0.59:15021: connect: connection refused" start-of-body= Nov 03 00:51:08.507496 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:51:08.507464 4939 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-88bbf7-openshift-default-574f47fff7-rf5cn" podUID="08f62205-b31d-4863-ab9e-3b13b15fbef5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.59:15021/healthz/ready\": dial tcp 10.132.0.59:15021: connect: connection refused" Nov 03 00:51:09.018389 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:51:09.018355 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 00:51:09.511973 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:51:09.511939 4939 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-88bbf7-openshift-default-574f47fff7-rf5cn" Nov 03 00:51:09.512357 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:51:09.512253 4939 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-88bbf7-openshift-default-574f47fff7-rf5cn" Nov 03 00:51:09.512923 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:51:09.512896 4939 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-88bbf7-openshift-default-574f47fff7-rf5cn" Nov 03 00:51:09.536432 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:51:09.536380 4939 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w1-88bbf7/simulator-w1-88bbf7-kserve-5fbb669f8f-42hgd" podStartSLOduration=3.536370037 podStartE2EDuration="3.536370037s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:51:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:51:08.173830705 +0000 UTC m=+616.438165651" watchObservedRunningTime="2026-11-03 00:51:09.536370037 +0000 UTC m=+617.800705023" Nov 03 00:51:09.871672 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:51:09.871630 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 00:51:10.564148 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:51:10.564099 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 00:51:11.204483 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:51:11.204443 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 00:51:11.917679 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:51:11.917647 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 00:51:12.598190 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:51:12.598158 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 00:51:13.233614 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:51:13.233574 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 00:51:13.353843 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:51:13.353809 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 00:51:14.010393 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:51:14.010357 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 00:51:14.898511 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:51:14.898474 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 00:51:14.937458 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:51:14.937418 4939 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w0-2f9a3a/simulator-w0-2f9a3a-kserve-9596bbb8f-2trgz" Nov 03 00:51:14.964016 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:51:14.963988 4939 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w0-2f9a3a/simulator-w0-2f9a3a-kserve-9596bbb8f-2trgz" Nov 03 00:51:15.188112 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:51:15.188032 4939 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-2f9a3a/simulator-w0-2f9a3a-kserve-9596bbb8f-2trgz" Nov 03 00:51:15.588765 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:51:15.588730 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 00:51:16.498274 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:51:16.498240 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 00:51:16.632187 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:51:16.632156 4939 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-4pw7v"] Nov 03 00:51:17.184993 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:51:17.184963 4939 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w1-88bbf7/simulator-w1-88bbf7-kserve-5fbb669f8f-42hgd" Nov 03 00:51:17.198756 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:51:17.198729 4939 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w1-88bbf7/simulator-w1-88bbf7-kserve-5fbb669f8f-42hgd" Nov 03 00:51:17.337428 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:51:17.337391 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 00:51:17.994885 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:51:17.994840 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 00:51:18.048475 ip-10-0-140-107 kubenswrapper[4939]: E1103 00:51:18.048432 4939 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-sthjh" containerName="extractor" Nov 03 00:51:18.200166 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:51:18.200135 4939 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-88bbf7/simulator-w1-88bbf7-kserve-5fbb669f8f-42hgd" Nov 03 00:51:18.608568 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:51:18.608537 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 00:51:19.381407 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:51:19.381356 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 00:51:20.091686 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:51:20.091652 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 00:51:20.635941 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:51:20.635891 4939 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w2-f7680d/premium-w2-f7680d-kserve-76565b8cf9-hqbbt"] Nov 03 00:51:20.641725 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:51:20.641702 4939 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-f7680d/premium-w2-f7680d-kserve-76565b8cf9-hqbbt" Nov 03 00:51:20.646950 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:51:20.646625 4939 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w2-f7680d\"/\"default-dockercfg-zmrzj\"" Nov 03 00:51:20.648099 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:51:20.648078 4939 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w2-f7680d/premium-w2-f7680d-kserve-76565b8cf9-hqbbt"] Nov 03 00:51:20.650391 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:51:20.650369 4939 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w2-f7680d\"/\"openshift-service-ca.crt\"" Nov 03 00:51:20.650508 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:51:20.650379 4939 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w2-f7680d\"/\"kube-root-ca.crt\"" Nov 03 00:51:20.700846 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:51:20.700814 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 00:51:20.747264 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:51:20.747229 4939 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w2-f7680d\"/\"premium-w2-f7680d-kserve-self-signed-certs\"" Nov 03 00:51:20.763771 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:51:20.763738 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c6lhc\" (UniqueName: \"kubernetes.io/projected/4712e708-18a2-4a13-a12c-a5669397b2c3-kube-api-access-c6lhc\") pod \"premium-w2-f7680d-kserve-76565b8cf9-hqbbt\" (UID: \"4712e708-18a2-4a13-a12c-a5669397b2c3\") " pod="e2e-models-e2e-worker-w2-f7680d/premium-w2-f7680d-kserve-76565b8cf9-hqbbt" Nov 03 00:51:20.763897 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:51:20.763781 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4712e708-18a2-4a13-a12c-a5669397b2c3-model-cache\") pod \"premium-w2-f7680d-kserve-76565b8cf9-hqbbt\" (UID: \"4712e708-18a2-4a13-a12c-a5669397b2c3\") " pod="e2e-models-e2e-worker-w2-f7680d/premium-w2-f7680d-kserve-76565b8cf9-hqbbt" Nov 03 00:51:20.763897 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:51:20.763805 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4712e708-18a2-4a13-a12c-a5669397b2c3-home\") pod \"premium-w2-f7680d-kserve-76565b8cf9-hqbbt\" (UID: \"4712e708-18a2-4a13-a12c-a5669397b2c3\") " pod="e2e-models-e2e-worker-w2-f7680d/premium-w2-f7680d-kserve-76565b8cf9-hqbbt" Nov 03 00:51:20.763997 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:51:20.763985 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4712e708-18a2-4a13-a12c-a5669397b2c3-dshm\") pod \"premium-w2-f7680d-kserve-76565b8cf9-hqbbt\" (UID: \"4712e708-18a2-4a13-a12c-a5669397b2c3\") " pod="e2e-models-e2e-worker-w2-f7680d/premium-w2-f7680d-kserve-76565b8cf9-hqbbt" Nov 03 00:51:20.764070 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:51:20.764038 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4712e708-18a2-4a13-a12c-a5669397b2c3-tls-certs\") pod \"premium-w2-f7680d-kserve-76565b8cf9-hqbbt\" (UID: \"4712e708-18a2-4a13-a12c-a5669397b2c3\") " pod="e2e-models-e2e-worker-w2-f7680d/premium-w2-f7680d-kserve-76565b8cf9-hqbbt" Nov 03 00:51:20.764208 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:51:20.764112 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4712e708-18a2-4a13-a12c-a5669397b2c3-tmp-dir\") pod \"premium-w2-f7680d-kserve-76565b8cf9-hqbbt\" (UID: \"4712e708-18a2-4a13-a12c-a5669397b2c3\") " pod="e2e-models-e2e-worker-w2-f7680d/premium-w2-f7680d-kserve-76565b8cf9-hqbbt" Nov 03 00:51:20.865041 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:51:20.864989 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4712e708-18a2-4a13-a12c-a5669397b2c3-tmp-dir\") pod \"premium-w2-f7680d-kserve-76565b8cf9-hqbbt\" (UID: \"4712e708-18a2-4a13-a12c-a5669397b2c3\") " pod="e2e-models-e2e-worker-w2-f7680d/premium-w2-f7680d-kserve-76565b8cf9-hqbbt" Nov 03 00:51:20.865400 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:51:20.865375 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c6lhc\" (UniqueName: \"kubernetes.io/projected/4712e708-18a2-4a13-a12c-a5669397b2c3-kube-api-access-c6lhc\") pod \"premium-w2-f7680d-kserve-76565b8cf9-hqbbt\" (UID: \"4712e708-18a2-4a13-a12c-a5669397b2c3\") " pod="e2e-models-e2e-worker-w2-f7680d/premium-w2-f7680d-kserve-76565b8cf9-hqbbt" Nov 03 00:51:20.865534 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:51:20.865518 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4712e708-18a2-4a13-a12c-a5669397b2c3-model-cache\") pod \"premium-w2-f7680d-kserve-76565b8cf9-hqbbt\" (UID: \"4712e708-18a2-4a13-a12c-a5669397b2c3\") " pod="e2e-models-e2e-worker-w2-f7680d/premium-w2-f7680d-kserve-76565b8cf9-hqbbt" Nov 03 00:51:20.865677 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:51:20.865644 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4712e708-18a2-4a13-a12c-a5669397b2c3-home\") pod \"premium-w2-f7680d-kserve-76565b8cf9-hqbbt\" (UID: \"4712e708-18a2-4a13-a12c-a5669397b2c3\") " pod="e2e-models-e2e-worker-w2-f7680d/premium-w2-f7680d-kserve-76565b8cf9-hqbbt" Nov 03 00:51:20.865677 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:51:20.865392 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4712e708-18a2-4a13-a12c-a5669397b2c3-tmp-dir\") pod \"premium-w2-f7680d-kserve-76565b8cf9-hqbbt\" (UID: \"4712e708-18a2-4a13-a12c-a5669397b2c3\") " pod="e2e-models-e2e-worker-w2-f7680d/premium-w2-f7680d-kserve-76565b8cf9-hqbbt" Nov 03 00:51:20.865891 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:51:20.865758 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4712e708-18a2-4a13-a12c-a5669397b2c3-dshm\") pod \"premium-w2-f7680d-kserve-76565b8cf9-hqbbt\" (UID: \"4712e708-18a2-4a13-a12c-a5669397b2c3\") " pod="e2e-models-e2e-worker-w2-f7680d/premium-w2-f7680d-kserve-76565b8cf9-hqbbt" Nov 03 00:51:20.865891 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:51:20.865818 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4712e708-18a2-4a13-a12c-a5669397b2c3-tls-certs\") pod \"premium-w2-f7680d-kserve-76565b8cf9-hqbbt\" (UID: \"4712e708-18a2-4a13-a12c-a5669397b2c3\") " pod="e2e-models-e2e-worker-w2-f7680d/premium-w2-f7680d-kserve-76565b8cf9-hqbbt" Nov 03 00:51:20.866050 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:51:20.865985 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4712e708-18a2-4a13-a12c-a5669397b2c3-home\") pod \"premium-w2-f7680d-kserve-76565b8cf9-hqbbt\" (UID: \"4712e708-18a2-4a13-a12c-a5669397b2c3\") " pod="e2e-models-e2e-worker-w2-f7680d/premium-w2-f7680d-kserve-76565b8cf9-hqbbt" Nov 03 00:51:20.866249 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:51:20.866229 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4712e708-18a2-4a13-a12c-a5669397b2c3-model-cache\") pod \"premium-w2-f7680d-kserve-76565b8cf9-hqbbt\" (UID: \"4712e708-18a2-4a13-a12c-a5669397b2c3\") " pod="e2e-models-e2e-worker-w2-f7680d/premium-w2-f7680d-kserve-76565b8cf9-hqbbt" Nov 03 00:51:20.869537 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:51:20.869516 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4712e708-18a2-4a13-a12c-a5669397b2c3-tls-certs\") pod \"premium-w2-f7680d-kserve-76565b8cf9-hqbbt\" (UID: \"4712e708-18a2-4a13-a12c-a5669397b2c3\") " pod="e2e-models-e2e-worker-w2-f7680d/premium-w2-f7680d-kserve-76565b8cf9-hqbbt" Nov 03 00:51:20.869812 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:51:20.869792 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4712e708-18a2-4a13-a12c-a5669397b2c3-dshm\") pod \"premium-w2-f7680d-kserve-76565b8cf9-hqbbt\" (UID: \"4712e708-18a2-4a13-a12c-a5669397b2c3\") " pod="e2e-models-e2e-worker-w2-f7680d/premium-w2-f7680d-kserve-76565b8cf9-hqbbt" Nov 03 00:51:20.899225 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:51:20.899163 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-c6lhc\" (UniqueName: \"kubernetes.io/projected/4712e708-18a2-4a13-a12c-a5669397b2c3-kube-api-access-c6lhc\") pod \"premium-w2-f7680d-kserve-76565b8cf9-hqbbt\" (UID: \"4712e708-18a2-4a13-a12c-a5669397b2c3\") " pod="e2e-models-e2e-worker-w2-f7680d/premium-w2-f7680d-kserve-76565b8cf9-hqbbt" Nov 03 00:51:20.959530 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:51:20.959498 4939 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-f7680d/premium-w2-f7680d-kserve-76565b8cf9-hqbbt" Nov 03 00:51:21.117177 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:51:21.117139 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 00:51:21.234299 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:51:21.234276 4939 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w2-f7680d/premium-w2-f7680d-kserve-76565b8cf9-hqbbt"] Nov 03 00:51:21.236444 ip-10-0-140-107 kubenswrapper[4939]: W1103 00:51:21.236412 4939 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod4712e708_18a2_4a13_a12c_a5669397b2c3.slice/crio-6789d8930e178842740e13fd1a143c9f06fb93885382f253e83b404486842a22 WatchSource:0}: Error finding container 6789d8930e178842740e13fd1a143c9f06fb93885382f253e83b404486842a22: Status 404 returned error can't find the container with id 6789d8930e178842740e13fd1a143c9f06fb93885382f253e83b404486842a22 Nov 03 00:51:21.738565 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:51:21.730454 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 00:51:22.232234 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:51:22.232189 4939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-f7680d/premium-w2-f7680d-kserve-76565b8cf9-hqbbt" event={"ID":"4712e708-18a2-4a13-a12c-a5669397b2c3","Type":"ContainerStarted","Data":"031486de41eb4ed29f41d79f4243ab5a68a82bb031b3d1042ca44cd3d215aa04"} Nov 03 00:51:22.232550 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:51:22.232525 4939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-f7680d/premium-w2-f7680d-kserve-76565b8cf9-hqbbt" event={"ID":"4712e708-18a2-4a13-a12c-a5669397b2c3","Type":"ContainerStarted","Data":"6789d8930e178842740e13fd1a143c9f06fb93885382f253e83b404486842a22"} Nov 03 00:51:22.250195 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:51:22.250144 4939 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w2-f7680d/premium-w2-f7680d-kserve-76565b8cf9-hqbbt" podStartSLOduration=2.250131194 podStartE2EDuration="2.250131194s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:51:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:51:22.248839823 +0000 UTC m=+630.513174768" watchObservedRunningTime="2026-11-03 00:51:22.250131194 +0000 UTC m=+630.514466117" Nov 03 00:51:22.515212 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:51:22.515117 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 00:51:22.960527 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:51:22.960498 4939 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-f7680d/premium-w2-f7680d-kserve-76565b8cf9-hqbbt" Nov 03 00:51:23.282896 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:51:23.282807 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 00:51:24.093100 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:51:24.093061 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 00:51:24.723882 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:51:24.723845 4939 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-4pw7v"] Nov 03 00:51:24.791379 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:51:24.791340 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 00:51:25.576406 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:51:25.576364 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 00:51:26.021477 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:51:26.021424 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 00:51:26.291387 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:51:26.291299 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 00:51:26.905916 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:51:26.905866 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 00:51:27.591329 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:51:27.591298 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 00:51:28.382382 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:51:28.382350 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 00:51:28.718162 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:51:28.718071 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 00:51:28.829179 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:51:28.829134 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 00:51:29.092587 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:51:29.092548 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 00:51:29.706958 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:51:29.706887 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 00:51:29.815230 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:51:29.815197 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 00:51:30.500098 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:51:30.500051 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 00:51:30.959678 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:51:30.959646 4939 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w2-f7680d/premium-w2-f7680d-kserve-76565b8cf9-hqbbt" Nov 03 00:51:30.972230 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:51:30.972203 4939 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w2-f7680d/premium-w2-f7680d-kserve-76565b8cf9-hqbbt" Nov 03 00:51:31.193591 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:51:31.193558 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 00:51:31.290342 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:51:31.290258 4939 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-f7680d/premium-w2-f7680d-kserve-76565b8cf9-hqbbt" Nov 03 00:51:31.934180 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:51:31.934148 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 00:51:32.555547 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:51:32.555518 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 00:51:33.209578 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:51:33.209536 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 00:51:34.131896 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:51:34.131854 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 00:51:34.331838 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:51:34.331805 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 00:51:34.851470 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:51:34.851438 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 00:51:35.515067 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:51:35.515028 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 00:51:36.205270 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:51:36.205228 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 00:51:36.982664 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:51:36.982625 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 00:51:37.605707 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:51:37.605671 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 00:51:37.639758 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:51:37.639716 4939 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-4pw7v"] Nov 03 00:51:38.599559 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:51:38.599522 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 00:51:39.501105 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:51:39.501017 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 00:51:40.198569 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:51:40.198530 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 00:51:40.885334 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:51:40.885298 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 00:51:41.602018 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:51:41.601984 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 00:51:41.871044 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:51:41.870966 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 00:51:42.376477 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:51:42.376438 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 00:51:42.977252 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:51:42.977216 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 00:51:43.692874 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:51:43.692817 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 00:51:44.391148 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:51:44.391112 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 00:51:44.989085 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:51:44.989033 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 00:51:45.740875 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:51:45.740831 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 00:51:46.589889 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:51:46.589853 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 00:51:47.303816 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:51:47.303777 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 00:51:48.029435 ip-10-0-140-107 kubenswrapper[4939]: E1103 00:51:48.029390 4939 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-sthjh" containerName="extractor" Nov 03 00:51:48.150955 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:51:48.150923 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 00:51:49.028677 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:51:49.028633 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 00:51:49.753427 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:51:49.753393 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 00:51:50.603960 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:51:50.603930 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 00:51:51.298332 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:51:51.298298 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 00:51:52.088172 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:51:52.088133 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 00:51:52.777794 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:51:52.777759 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 00:51:53.391368 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:51:53.391331 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 00:51:54.089633 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:51:54.089600 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 00:51:54.791075 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:51:54.791043 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 00:51:55.553846 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:51:55.553813 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 00:51:56.392726 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:51:56.392688 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 00:51:57.084240 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:51:57.084207 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 00:51:57.708308 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:51:57.708274 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 00:51:58.427167 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:51:58.427128 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 00:51:58.523354 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:51:58.523319 4939 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-4pw7v"] Nov 03 00:51:59.293598 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:51:59.293562 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 00:52:00.187298 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:52:00.187260 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 00:52:01.130133 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:52:01.130094 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 00:52:01.821941 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:52:01.820555 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 00:52:02.481705 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:52:02.481672 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 00:52:03.217963 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:52:03.217832 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 00:52:03.899297 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:52:03.899253 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 00:52:04.627611 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:52:04.627575 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 00:52:04.735539 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:52:04.735508 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 00:52:05.303332 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:52:05.303303 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 00:52:06.004782 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:52:06.004749 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 00:52:06.610011 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:52:06.609978 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 00:52:07.385898 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:52:07.385860 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 00:52:08.459507 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:52:08.459451 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 00:52:09.307504 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:52:09.307460 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 00:52:09.912595 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:52:09.912565 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 00:52:10.629886 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:52:10.629830 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 00:52:11.049533 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:52:11.049452 4939 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f9962d74-ln86t"] Nov 03 00:52:11.053148 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:52:11.053132 4939 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f9962d74-ln86t" Nov 03 00:52:11.059311 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:52:11.059290 4939 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 00:52:11.059851 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:52:11.059828 4939 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-6q4xp\"" Nov 03 00:52:11.062479 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:52:11.062456 4939 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f9962d74-ln86t"] Nov 03 00:52:11.132541 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:52:11.132504 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/84a1ac77-cd42-4963-b8a0-b941878e5c10-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f9962d74-ln86t\" (UID: \"84a1ac77-cd42-4963-b8a0-b941878e5c10\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f9962d74-ln86t" Nov 03 00:52:11.132720 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:52:11.132564 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/84a1ac77-cd42-4963-b8a0-b941878e5c10-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f9962d74-ln86t\" (UID: \"84a1ac77-cd42-4963-b8a0-b941878e5c10\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f9962d74-ln86t" Nov 03 00:52:11.132720 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:52:11.132652 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fltdg\" (UniqueName: \"kubernetes.io/projected/84a1ac77-cd42-4963-b8a0-b941878e5c10-kube-api-access-fltdg\") pod \"maas-api-revoke-sub-f9962d74-ln86t\" (UID: \"84a1ac77-cd42-4963-b8a0-b941878e5c10\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f9962d74-ln86t" Nov 03 00:52:11.233402 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:52:11.233370 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/84a1ac77-cd42-4963-b8a0-b941878e5c10-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f9962d74-ln86t\" (UID: \"84a1ac77-cd42-4963-b8a0-b941878e5c10\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f9962d74-ln86t" Nov 03 00:52:11.233584 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:52:11.233426 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/84a1ac77-cd42-4963-b8a0-b941878e5c10-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f9962d74-ln86t\" (UID: \"84a1ac77-cd42-4963-b8a0-b941878e5c10\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f9962d74-ln86t" Nov 03 00:52:11.233584 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:52:11.233475 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fltdg\" (UniqueName: \"kubernetes.io/projected/84a1ac77-cd42-4963-b8a0-b941878e5c10-kube-api-access-fltdg\") pod \"maas-api-revoke-sub-f9962d74-ln86t\" (UID: \"84a1ac77-cd42-4963-b8a0-b941878e5c10\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f9962d74-ln86t" Nov 03 00:52:11.234159 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:52:11.234139 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/84a1ac77-cd42-4963-b8a0-b941878e5c10-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f9962d74-ln86t\" (UID: \"84a1ac77-cd42-4963-b8a0-b941878e5c10\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f9962d74-ln86t" Nov 03 00:52:11.235828 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:52:11.235805 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/84a1ac77-cd42-4963-b8a0-b941878e5c10-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f9962d74-ln86t\" (UID: \"84a1ac77-cd42-4963-b8a0-b941878e5c10\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f9962d74-ln86t" Nov 03 00:52:11.241613 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:52:11.241584 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fltdg\" (UniqueName: \"kubernetes.io/projected/84a1ac77-cd42-4963-b8a0-b941878e5c10-kube-api-access-fltdg\") pod \"maas-api-revoke-sub-f9962d74-ln86t\" (UID: \"84a1ac77-cd42-4963-b8a0-b941878e5c10\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f9962d74-ln86t" Nov 03 00:52:11.369706 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:52:11.369670 4939 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f9962d74-ln86t" Nov 03 00:52:11.402610 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:52:11.402581 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 00:52:11.644453 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:52:11.644419 4939 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f9962d74-ln86t"] Nov 03 00:52:11.646488 ip-10-0-140-107 kubenswrapper[4939]: W1103 00:52:11.646457 4939 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod84a1ac77_cd42_4963_b8a0_b941878e5c10.slice/crio-52a32a73e7afc8c325f66a65bc38af974297a5c51c3d36055b505768f7ec934e WatchSource:0}: Error finding container 52a32a73e7afc8c325f66a65bc38af974297a5c51c3d36055b505768f7ec934e: Status 404 returned error can't find the container with id 52a32a73e7afc8c325f66a65bc38af974297a5c51c3d36055b505768f7ec934e Nov 03 00:52:11.648453 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:52:11.648436 4939 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Nov 03 00:52:12.111816 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:52:12.111756 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 00:52:12.592294 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:52:12.592255 4939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f9962d74-ln86t" event={"ID":"84a1ac77-cd42-4963-b8a0-b941878e5c10","Type":"ContainerStarted","Data":"52a32a73e7afc8c325f66a65bc38af974297a5c51c3d36055b505768f7ec934e"} Nov 03 00:52:12.791399 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:52:12.791361 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 00:52:13.398342 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:52:13.398295 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 00:52:14.202742 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:52:14.202707 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 00:52:14.891229 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:52:14.891193 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 00:52:15.122504 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:52:15.122471 4939 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-f7680d-openshift-default-c4488d5c9-h5j8d"] Nov 03 00:52:15.128010 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:52:15.127983 4939 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-f7680d-openshift-default-c4488d5c9-h5j8d" Nov 03 00:52:15.136007 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:52:15.135576 4939 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w2-f7680d-openshift-default-dockercfg-fr542\"" Nov 03 00:52:15.140103 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:52:15.140075 4939 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-f7680d-openshift-default-c4488d5c9-h5j8d"] Nov 03 00:52:15.191175 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:52:15.191140 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4fe351d2-d66c-4cd4-8bf7-264a40d2551c-istio-token\") pod \"e2e-worker-w2-f7680d-openshift-default-c4488d5c9-h5j8d\" (UID: \"4fe351d2-d66c-4cd4-8bf7-264a40d2551c\") " pod="openshift-ingress/e2e-worker-w2-f7680d-openshift-default-c4488d5c9-h5j8d" Nov 03 00:52:15.191375 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:52:15.191202 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4fe351d2-d66c-4cd4-8bf7-264a40d2551c-istio-data\") pod \"e2e-worker-w2-f7680d-openshift-default-c4488d5c9-h5j8d\" (UID: \"4fe351d2-d66c-4cd4-8bf7-264a40d2551c\") " pod="openshift-ingress/e2e-worker-w2-f7680d-openshift-default-c4488d5c9-h5j8d" Nov 03 00:52:15.191375 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:52:15.191226 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4fe351d2-d66c-4cd4-8bf7-264a40d2551c-istiod-ca-cert\") pod \"e2e-worker-w2-f7680d-openshift-default-c4488d5c9-h5j8d\" (UID: \"4fe351d2-d66c-4cd4-8bf7-264a40d2551c\") " pod="openshift-ingress/e2e-worker-w2-f7680d-openshift-default-c4488d5c9-h5j8d" Nov 03 00:52:15.191505 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:52:15.191380 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4fe351d2-d66c-4cd4-8bf7-264a40d2551c-istio-envoy\") pod \"e2e-worker-w2-f7680d-openshift-default-c4488d5c9-h5j8d\" (UID: \"4fe351d2-d66c-4cd4-8bf7-264a40d2551c\") " pod="openshift-ingress/e2e-worker-w2-f7680d-openshift-default-c4488d5c9-h5j8d" Nov 03 00:52:15.191505 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:52:15.191439 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4fe351d2-d66c-4cd4-8bf7-264a40d2551c-credential-socket\") pod \"e2e-worker-w2-f7680d-openshift-default-c4488d5c9-h5j8d\" (UID: \"4fe351d2-d66c-4cd4-8bf7-264a40d2551c\") " pod="openshift-ingress/e2e-worker-w2-f7680d-openshift-default-c4488d5c9-h5j8d" Nov 03 00:52:15.191505 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:52:15.191491 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4fe351d2-d66c-4cd4-8bf7-264a40d2551c-workload-certs\") pod \"e2e-worker-w2-f7680d-openshift-default-c4488d5c9-h5j8d\" (UID: \"4fe351d2-d66c-4cd4-8bf7-264a40d2551c\") " pod="openshift-ingress/e2e-worker-w2-f7680d-openshift-default-c4488d5c9-h5j8d" Nov 03 00:52:15.191648 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:52:15.191559 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4fe351d2-d66c-4cd4-8bf7-264a40d2551c-istio-podinfo\") pod \"e2e-worker-w2-f7680d-openshift-default-c4488d5c9-h5j8d\" (UID: \"4fe351d2-d66c-4cd4-8bf7-264a40d2551c\") " pod="openshift-ingress/e2e-worker-w2-f7680d-openshift-default-c4488d5c9-h5j8d" Nov 03 00:52:15.191648 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:52:15.191634 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4fe351d2-d66c-4cd4-8bf7-264a40d2551c-workload-socket\") pod \"e2e-worker-w2-f7680d-openshift-default-c4488d5c9-h5j8d\" (UID: \"4fe351d2-d66c-4cd4-8bf7-264a40d2551c\") " pod="openshift-ingress/e2e-worker-w2-f7680d-openshift-default-c4488d5c9-h5j8d" Nov 03 00:52:15.191787 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:52:15.191766 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s7cqg\" (UniqueName: \"kubernetes.io/projected/4fe351d2-d66c-4cd4-8bf7-264a40d2551c-kube-api-access-s7cqg\") pod \"e2e-worker-w2-f7680d-openshift-default-c4488d5c9-h5j8d\" (UID: \"4fe351d2-d66c-4cd4-8bf7-264a40d2551c\") " pod="openshift-ingress/e2e-worker-w2-f7680d-openshift-default-c4488d5c9-h5j8d" Nov 03 00:52:15.295166 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:52:15.295064 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4fe351d2-d66c-4cd4-8bf7-264a40d2551c-istio-token\") pod \"e2e-worker-w2-f7680d-openshift-default-c4488d5c9-h5j8d\" (UID: \"4fe351d2-d66c-4cd4-8bf7-264a40d2551c\") " pod="openshift-ingress/e2e-worker-w2-f7680d-openshift-default-c4488d5c9-h5j8d" Nov 03 00:52:15.295488 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:52:15.295466 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4fe351d2-d66c-4cd4-8bf7-264a40d2551c-istio-data\") pod \"e2e-worker-w2-f7680d-openshift-default-c4488d5c9-h5j8d\" (UID: \"4fe351d2-d66c-4cd4-8bf7-264a40d2551c\") " pod="openshift-ingress/e2e-worker-w2-f7680d-openshift-default-c4488d5c9-h5j8d" Nov 03 00:52:15.295624 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:52:15.295609 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4fe351d2-d66c-4cd4-8bf7-264a40d2551c-istiod-ca-cert\") pod \"e2e-worker-w2-f7680d-openshift-default-c4488d5c9-h5j8d\" (UID: \"4fe351d2-d66c-4cd4-8bf7-264a40d2551c\") " pod="openshift-ingress/e2e-worker-w2-f7680d-openshift-default-c4488d5c9-h5j8d" Nov 03 00:52:15.295754 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:52:15.295739 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4fe351d2-d66c-4cd4-8bf7-264a40d2551c-istio-envoy\") pod \"e2e-worker-w2-f7680d-openshift-default-c4488d5c9-h5j8d\" (UID: \"4fe351d2-d66c-4cd4-8bf7-264a40d2551c\") " pod="openshift-ingress/e2e-worker-w2-f7680d-openshift-default-c4488d5c9-h5j8d" Nov 03 00:52:15.295863 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:52:15.295850 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4fe351d2-d66c-4cd4-8bf7-264a40d2551c-credential-socket\") pod \"e2e-worker-w2-f7680d-openshift-default-c4488d5c9-h5j8d\" (UID: \"4fe351d2-d66c-4cd4-8bf7-264a40d2551c\") " pod="openshift-ingress/e2e-worker-w2-f7680d-openshift-default-c4488d5c9-h5j8d" Nov 03 00:52:15.295988 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:52:15.295971 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4fe351d2-d66c-4cd4-8bf7-264a40d2551c-workload-certs\") pod \"e2e-worker-w2-f7680d-openshift-default-c4488d5c9-h5j8d\" (UID: \"4fe351d2-d66c-4cd4-8bf7-264a40d2551c\") " pod="openshift-ingress/e2e-worker-w2-f7680d-openshift-default-c4488d5c9-h5j8d" Nov 03 00:52:15.296148 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:52:15.296131 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4fe351d2-d66c-4cd4-8bf7-264a40d2551c-istio-podinfo\") pod \"e2e-worker-w2-f7680d-openshift-default-c4488d5c9-h5j8d\" (UID: \"4fe351d2-d66c-4cd4-8bf7-264a40d2551c\") " pod="openshift-ingress/e2e-worker-w2-f7680d-openshift-default-c4488d5c9-h5j8d" Nov 03 00:52:15.296257 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:52:15.296243 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4fe351d2-d66c-4cd4-8bf7-264a40d2551c-workload-socket\") pod \"e2e-worker-w2-f7680d-openshift-default-c4488d5c9-h5j8d\" (UID: \"4fe351d2-d66c-4cd4-8bf7-264a40d2551c\") " pod="openshift-ingress/e2e-worker-w2-f7680d-openshift-default-c4488d5c9-h5j8d" Nov 03 00:52:15.296434 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:52:15.296416 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s7cqg\" (UniqueName: \"kubernetes.io/projected/4fe351d2-d66c-4cd4-8bf7-264a40d2551c-kube-api-access-s7cqg\") pod \"e2e-worker-w2-f7680d-openshift-default-c4488d5c9-h5j8d\" (UID: \"4fe351d2-d66c-4cd4-8bf7-264a40d2551c\") " pod="openshift-ingress/e2e-worker-w2-f7680d-openshift-default-c4488d5c9-h5j8d" Nov 03 00:52:15.300378 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:52:15.297510 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4fe351d2-d66c-4cd4-8bf7-264a40d2551c-istio-data\") pod \"e2e-worker-w2-f7680d-openshift-default-c4488d5c9-h5j8d\" (UID: \"4fe351d2-d66c-4cd4-8bf7-264a40d2551c\") " pod="openshift-ingress/e2e-worker-w2-f7680d-openshift-default-c4488d5c9-h5j8d" Nov 03 00:52:15.301349 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:52:15.301323 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4fe351d2-d66c-4cd4-8bf7-264a40d2551c-istiod-ca-cert\") pod \"e2e-worker-w2-f7680d-openshift-default-c4488d5c9-h5j8d\" (UID: \"4fe351d2-d66c-4cd4-8bf7-264a40d2551c\") " pod="openshift-ingress/e2e-worker-w2-f7680d-openshift-default-c4488d5c9-h5j8d" Nov 03 00:52:15.302143 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:52:15.302105 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4fe351d2-d66c-4cd4-8bf7-264a40d2551c-workload-certs\") pod \"e2e-worker-w2-f7680d-openshift-default-c4488d5c9-h5j8d\" (UID: \"4fe351d2-d66c-4cd4-8bf7-264a40d2551c\") " pod="openshift-ingress/e2e-worker-w2-f7680d-openshift-default-c4488d5c9-h5j8d" Nov 03 00:52:15.302565 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:52:15.302543 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4fe351d2-d66c-4cd4-8bf7-264a40d2551c-credential-socket\") pod \"e2e-worker-w2-f7680d-openshift-default-c4488d5c9-h5j8d\" (UID: \"4fe351d2-d66c-4cd4-8bf7-264a40d2551c\") " pod="openshift-ingress/e2e-worker-w2-f7680d-openshift-default-c4488d5c9-h5j8d" Nov 03 00:52:15.303619 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:52:15.303596 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4fe351d2-d66c-4cd4-8bf7-264a40d2551c-workload-socket\") pod \"e2e-worker-w2-f7680d-openshift-default-c4488d5c9-h5j8d\" (UID: \"4fe351d2-d66c-4cd4-8bf7-264a40d2551c\") " pod="openshift-ingress/e2e-worker-w2-f7680d-openshift-default-c4488d5c9-h5j8d" Nov 03 00:52:15.310800 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:52:15.310774 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4fe351d2-d66c-4cd4-8bf7-264a40d2551c-istio-token\") pod \"e2e-worker-w2-f7680d-openshift-default-c4488d5c9-h5j8d\" (UID: \"4fe351d2-d66c-4cd4-8bf7-264a40d2551c\") " pod="openshift-ingress/e2e-worker-w2-f7680d-openshift-default-c4488d5c9-h5j8d" Nov 03 00:52:15.311556 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:52:15.311531 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-s7cqg\" (UniqueName: \"kubernetes.io/projected/4fe351d2-d66c-4cd4-8bf7-264a40d2551c-kube-api-access-s7cqg\") pod \"e2e-worker-w2-f7680d-openshift-default-c4488d5c9-h5j8d\" (UID: \"4fe351d2-d66c-4cd4-8bf7-264a40d2551c\") " pod="openshift-ingress/e2e-worker-w2-f7680d-openshift-default-c4488d5c9-h5j8d" Nov 03 00:52:15.315306 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:52:15.315282 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4fe351d2-d66c-4cd4-8bf7-264a40d2551c-istio-podinfo\") pod \"e2e-worker-w2-f7680d-openshift-default-c4488d5c9-h5j8d\" (UID: \"4fe351d2-d66c-4cd4-8bf7-264a40d2551c\") " pod="openshift-ingress/e2e-worker-w2-f7680d-openshift-default-c4488d5c9-h5j8d" Nov 03 00:52:15.318026 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:52:15.317963 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4fe351d2-d66c-4cd4-8bf7-264a40d2551c-istio-envoy\") pod \"e2e-worker-w2-f7680d-openshift-default-c4488d5c9-h5j8d\" (UID: \"4fe351d2-d66c-4cd4-8bf7-264a40d2551c\") " pod="openshift-ingress/e2e-worker-w2-f7680d-openshift-default-c4488d5c9-h5j8d" Nov 03 00:52:15.445743 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:52:15.445654 4939 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-f7680d-openshift-default-c4488d5c9-h5j8d" Nov 03 00:52:15.616864 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:52:15.616829 4939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f9962d74-ln86t" event={"ID":"84a1ac77-cd42-4963-b8a0-b941878e5c10","Type":"ContainerStarted","Data":"2917d956307fc73e5dac205783ce6e7058366d9cc6c4ca12c77f067e0e4462c5"} Nov 03 00:52:15.640073 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:52:15.640012 4939 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f9962d74-ln86t" podStartSLOduration=1.464248665 podStartE2EDuration="4.639993681s" totalImagesPullingTime="3.175745016s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:52:11 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:52:15.639240679 +0000 UTC m=+683.903575623" watchObservedRunningTime="2026-11-03 00:52:15.639993681 +0000 UTC m=+683.904328626" Nov 03 00:52:15.711535 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:52:15.711444 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 00:52:15.769053 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:52:15.768948 4939 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-f7680d-openshift-default-c4488d5c9-h5j8d"] Nov 03 00:52:15.781618 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:52:15.781573 4939 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 00:52:15.781835 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:52:15.781809 4939 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 00:52:15.782100 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:52:15.782077 4939 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 00:52:16.526078 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:52:16.526048 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 00:52:16.623442 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:52:16.623408 4939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-f7680d-openshift-default-c4488d5c9-h5j8d" event={"ID":"4fe351d2-d66c-4cd4-8bf7-264a40d2551c","Type":"ContainerStarted","Data":"40fc2a8fbd662b472eaf98c301d6481da7424cd5121b415342c40e5637d2665b"} Nov 03 00:52:16.623622 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:52:16.623464 4939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-f7680d-openshift-default-c4488d5c9-h5j8d" event={"ID":"4fe351d2-d66c-4cd4-8bf7-264a40d2551c","Type":"ContainerStarted","Data":"b3d49abb8aca5b0ba9793afa1d699942589b243091048f2c663a526f5b22f305"} Nov 03 00:52:16.624956 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:52:16.624925 4939 generic.go:356] "Generic (PLEG): container finished" podID="84a1ac77-cd42-4963-b8a0-b941878e5c10" containerID="2917d956307fc73e5dac205783ce6e7058366d9cc6c4ca12c77f067e0e4462c5" exitCode=0 Nov 03 00:52:16.625078 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:52:16.625008 4939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f9962d74-ln86t" event={"ID":"84a1ac77-cd42-4963-b8a0-b941878e5c10","Type":"ContainerDied","Data":"2917d956307fc73e5dac205783ce6e7058366d9cc6c4ca12c77f067e0e4462c5"} Nov 03 00:52:16.644390 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:52:16.644335 4939 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-f7680d-openshift-default-c4488d5c9-h5j8d" podStartSLOduration=1.644322931 podStartE2EDuration="1.644322931s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:52:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:52:16.641661458 +0000 UTC m=+684.905996417" watchObservedRunningTime="2026-11-03 00:52:16.644322931 +0000 UTC m=+684.908657876" Nov 03 00:52:17.345454 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:52:17.345416 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 00:52:17.463933 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:52:17.462771 4939 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-f7680d-openshift-default-c4488d5c9-h5j8d" Nov 03 00:52:17.463933 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:52:17.462892 4939 patch_prober.go:29] interesting pod/e2e-worker-w2-f7680d-openshift-default-c4488d5c9-h5j8d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.65:15021/healthz/ready\": dial tcp 10.132.0.65:15021: connect: connection refused" start-of-body= Nov 03 00:52:17.463933 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:52:17.462968 4939 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-f7680d-openshift-default-c4488d5c9-h5j8d" podUID="4fe351d2-d66c-4cd4-8bf7-264a40d2551c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.65:15021/healthz/ready\": dial tcp 10.132.0.65:15021: connect: connection refused" Nov 03 00:52:17.817274 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:52:17.817245 4939 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f9962d74-ln86t" Nov 03 00:52:17.944106 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:52:17.944010 4939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/84a1ac77-cd42-4963-b8a0-b941878e5c10-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/84a1ac77-cd42-4963-b8a0-b941878e5c10-maas-api-xks-ca\") pod \"84a1ac77-cd42-4963-b8a0-b941878e5c10\" (UID: \"84a1ac77-cd42-4963-b8a0-b941878e5c10\") " Nov 03 00:52:17.944106 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:52:17.944081 4939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/84a1ac77-cd42-4963-b8a0-b941878e5c10-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/84a1ac77-cd42-4963-b8a0-b941878e5c10-maas-api-openshift-ca\") pod \"84a1ac77-cd42-4963-b8a0-b941878e5c10\" (UID: \"84a1ac77-cd42-4963-b8a0-b941878e5c10\") " Nov 03 00:52:17.944340 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:52:17.944137 4939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/84a1ac77-cd42-4963-b8a0-b941878e5c10-kube-api-access-fltdg\" (UniqueName: \"kubernetes.io/projected/84a1ac77-cd42-4963-b8a0-b941878e5c10-kube-api-access-fltdg\") pod \"84a1ac77-cd42-4963-b8a0-b941878e5c10\" (UID: \"84a1ac77-cd42-4963-b8a0-b941878e5c10\") " Nov 03 00:52:17.944470 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:52:17.944444 4939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/84a1ac77-cd42-4963-b8a0-b941878e5c10-maas-api-openshift-ca" pod "84a1ac77-cd42-4963-b8a0-b941878e5c10" (UID: "84a1ac77-cd42-4963-b8a0-b941878e5c10"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 00:52:17.944730 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:52:17.944672 4939 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/84a1ac77-cd42-4963-b8a0-b941878e5c10-maas-api-openshift-ca\") on node \"ip-10-0-140-107.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 00:52:17.976934 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:52:17.958224 4939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/84a1ac77-cd42-4963-b8a0-b941878e5c10-kube-api-access-fltdg" pod "84a1ac77-cd42-4963-b8a0-b941878e5c10" (UID: "84a1ac77-cd42-4963-b8a0-b941878e5c10"). InnerVolumeSpecName "kube-api-access-fltdg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 00:52:17.976934 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:52:17.966484 4939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/84a1ac77-cd42-4963-b8a0-b941878e5c10-maas-api-xks-ca" pod "84a1ac77-cd42-4963-b8a0-b941878e5c10" (UID: "84a1ac77-cd42-4963-b8a0-b941878e5c10"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 00:52:18.033060 ip-10-0-140-107 kubenswrapper[4939]: E1103 00:52:18.033023 4939 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-sthjh" containerName="extractor" Nov 03 00:52:18.046065 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:52:18.046005 4939 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fltdg\" (UniqueName: \"kubernetes.io/projected/84a1ac77-cd42-4963-b8a0-b941878e5c10-kube-api-access-fltdg\") on node \"ip-10-0-140-107.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 00:52:18.046065 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:52:18.046036 4939 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/84a1ac77-cd42-4963-b8a0-b941878e5c10-maas-api-xks-ca\") on node \"ip-10-0-140-107.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 00:52:18.450778 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:52:18.450321 4939 patch_prober.go:29] interesting pod/e2e-worker-w2-f7680d-openshift-default-c4488d5c9-h5j8d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.65:15021/healthz/ready\": dial tcp 10.132.0.65:15021: connect: connection refused" start-of-body= Nov 03 00:52:18.450778 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:52:18.450389 4939 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-f7680d-openshift-default-c4488d5c9-h5j8d" podUID="4fe351d2-d66c-4cd4-8bf7-264a40d2551c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.65:15021/healthz/ready\": dial tcp 10.132.0.65:15021: connect: connection refused" Nov 03 00:52:18.558631 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:52:18.558545 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 00:52:18.638460 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:52:18.638428 4939 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f9962d74-ln86t" Nov 03 00:52:18.638460 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:52:18.638448 4939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f9962d74-ln86t" event={"ID":"84a1ac77-cd42-4963-b8a0-b941878e5c10","Type":"ContainerDied","Data":"52a32a73e7afc8c325f66a65bc38af974297a5c51c3d36055b505768f7ec934e"} Nov 03 00:52:18.638674 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:52:18.638496 4939 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="52a32a73e7afc8c325f66a65bc38af974297a5c51c3d36055b505768f7ec934e" Nov 03 00:52:18.922850 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:52:18.922811 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 00:52:19.459946 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:52:19.459664 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 00:52:19.460277 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:52:19.460226 4939 patch_prober.go:29] interesting pod/e2e-worker-w2-f7680d-openshift-default-c4488d5c9-h5j8d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.65:15021/healthz/ready\": dial tcp 10.132.0.65:15021: connect: connection refused" start-of-body= Nov 03 00:52:19.460471 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:52:19.460444 4939 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-f7680d-openshift-default-c4488d5c9-h5j8d" podUID="4fe351d2-d66c-4cd4-8bf7-264a40d2551c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.65:15021/healthz/ready\": dial tcp 10.132.0.65:15021: connect: connection refused" Nov 03 00:52:19.573680 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:52:19.573625 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 00:52:19.996432 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:52:19.996378 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 00:52:20.129718 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:52:20.129678 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 00:52:20.203994 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:52:20.203965 4939 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["models-as-a-service/payload-processing-external-model-6889df475b-4kdg9"] Nov 03 00:52:20.204455 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:52:20.204442 4939 cpu_manager.go:414] "containerMap: removing container" podUID="84a1ac77-cd42-4963-b8a0-b941878e5c10" containerName="revoke-keys" Nov 03 00:52:20.204504 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:52:20.204456 4939 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="84a1ac77-cd42-4963-b8a0-b941878e5c10" containerName="revoke-keys" Nov 03 00:52:20.204574 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:52:20.204564 4939 memory_manager.go:365] "RemoveStaleState removing state" podUID="84a1ac77-cd42-4963-b8a0-b941878e5c10" containerName="revoke-keys" Nov 03 00:52:20.208082 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:52:20.208064 4939 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-6889df475b-4kdg9" Nov 03 00:52:20.216489 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:52:20.216458 4939 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"models-as-a-service\"/\"payload-processing-external-model-plugins\"" Nov 03 00:52:20.216616 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:52:20.216495 4939 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"models-as-a-service\"/\"routing-overlay\"" Nov 03 00:52:20.216616 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:52:20.216586 4939 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"models-as-a-service\"/\"payload-processing-external-model-dockercfg-r9jtj\"" Nov 03 00:52:20.216742 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:52:20.216679 4939 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"models-as-a-service\"/\"openshift-service-ca.crt\"" Nov 03 00:52:20.216801 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:52:20.216741 4939 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"models-as-a-service\"/\"agc-cred-a60c08844a3e\"" Nov 03 00:52:20.216859 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:52:20.216848 4939 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"models-as-a-service\"/\"kube-root-ca.crt\"" Nov 03 00:52:20.216948 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:52:20.216884 4939 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"models-as-a-service\"/\"agc-cred-48d9fe8f921f\"" Nov 03 00:52:20.219619 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:52:20.219597 4939 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["models-as-a-service/payload-processing-external-model-6889df475b-4kdg9"] Nov 03 00:52:20.295984 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:52:20.295624 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/81fff666-afd7-4fc5-b20a-c3573dc23d38-provider-credentials\") pod \"payload-processing-external-model-6889df475b-4kdg9\" (UID: \"81fff666-afd7-4fc5-b20a-c3573dc23d38\") " pod="models-as-a-service/payload-processing-external-model-6889df475b-4kdg9" Nov 03 00:52:20.295984 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:52:20.295717 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/81fff666-afd7-4fc5-b20a-c3573dc23d38-routing-overlay-volume\") pod \"payload-processing-external-model-6889df475b-4kdg9\" (UID: \"81fff666-afd7-4fc5-b20a-c3573dc23d38\") " pod="models-as-a-service/payload-processing-external-model-6889df475b-4kdg9" Nov 03 00:52:20.295984 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:52:20.295757 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/81fff666-afd7-4fc5-b20a-c3573dc23d38-plugins-config-volume\") pod \"payload-processing-external-model-6889df475b-4kdg9\" (UID: \"81fff666-afd7-4fc5-b20a-c3573dc23d38\") " pod="models-as-a-service/payload-processing-external-model-6889df475b-4kdg9" Nov 03 00:52:20.295984 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:52:20.295852 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n68z9\" (UniqueName: \"kubernetes.io/projected/81fff666-afd7-4fc5-b20a-c3573dc23d38-kube-api-access-n68z9\") pod \"payload-processing-external-model-6889df475b-4kdg9\" (UID: \"81fff666-afd7-4fc5-b20a-c3573dc23d38\") " pod="models-as-a-service/payload-processing-external-model-6889df475b-4kdg9" Nov 03 00:52:20.396932 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:52:20.396541 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/81fff666-afd7-4fc5-b20a-c3573dc23d38-routing-overlay-volume\") pod \"payload-processing-external-model-6889df475b-4kdg9\" (UID: \"81fff666-afd7-4fc5-b20a-c3573dc23d38\") " pod="models-as-a-service/payload-processing-external-model-6889df475b-4kdg9" Nov 03 00:52:20.396932 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:52:20.396599 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/81fff666-afd7-4fc5-b20a-c3573dc23d38-plugins-config-volume\") pod \"payload-processing-external-model-6889df475b-4kdg9\" (UID: \"81fff666-afd7-4fc5-b20a-c3573dc23d38\") " pod="models-as-a-service/payload-processing-external-model-6889df475b-4kdg9" Nov 03 00:52:20.396932 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:52:20.396696 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n68z9\" (UniqueName: \"kubernetes.io/projected/81fff666-afd7-4fc5-b20a-c3573dc23d38-kube-api-access-n68z9\") pod \"payload-processing-external-model-6889df475b-4kdg9\" (UID: \"81fff666-afd7-4fc5-b20a-c3573dc23d38\") " pod="models-as-a-service/payload-processing-external-model-6889df475b-4kdg9" Nov 03 00:52:20.396932 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:52:20.396794 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/81fff666-afd7-4fc5-b20a-c3573dc23d38-provider-credentials\") pod \"payload-processing-external-model-6889df475b-4kdg9\" (UID: \"81fff666-afd7-4fc5-b20a-c3573dc23d38\") " pod="models-as-a-service/payload-processing-external-model-6889df475b-4kdg9" Nov 03 00:52:20.399958 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:52:20.398457 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/81fff666-afd7-4fc5-b20a-c3573dc23d38-routing-overlay-volume\") pod \"payload-processing-external-model-6889df475b-4kdg9\" (UID: \"81fff666-afd7-4fc5-b20a-c3573dc23d38\") " pod="models-as-a-service/payload-processing-external-model-6889df475b-4kdg9" Nov 03 00:52:20.400830 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:52:20.400547 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/81fff666-afd7-4fc5-b20a-c3573dc23d38-plugins-config-volume\") pod \"payload-processing-external-model-6889df475b-4kdg9\" (UID: \"81fff666-afd7-4fc5-b20a-c3573dc23d38\") " pod="models-as-a-service/payload-processing-external-model-6889df475b-4kdg9" Nov 03 00:52:20.402964 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:52:20.402595 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/81fff666-afd7-4fc5-b20a-c3573dc23d38-provider-credentials\") pod \"payload-processing-external-model-6889df475b-4kdg9\" (UID: \"81fff666-afd7-4fc5-b20a-c3573dc23d38\") " pod="models-as-a-service/payload-processing-external-model-6889df475b-4kdg9" Nov 03 00:52:20.406024 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:52:20.406002 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-n68z9\" (UniqueName: \"kubernetes.io/projected/81fff666-afd7-4fc5-b20a-c3573dc23d38-kube-api-access-n68z9\") pod \"payload-processing-external-model-6889df475b-4kdg9\" (UID: \"81fff666-afd7-4fc5-b20a-c3573dc23d38\") " pod="models-as-a-service/payload-processing-external-model-6889df475b-4kdg9" Nov 03 00:52:20.466934 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:52:20.456546 4939 patch_prober.go:29] interesting pod/e2e-worker-w2-f7680d-openshift-default-c4488d5c9-h5j8d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.65:15021/healthz/ready\": dial tcp 10.132.0.65:15021: connect: connection refused" start-of-body= Nov 03 00:52:20.466934 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:52:20.456621 4939 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-f7680d-openshift-default-c4488d5c9-h5j8d" podUID="4fe351d2-d66c-4cd4-8bf7-264a40d2551c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.65:15021/healthz/ready\": dial tcp 10.132.0.65:15021: connect: connection refused" Nov 03 00:52:20.523057 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:52:20.523010 4939 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-6889df475b-4kdg9" Nov 03 00:52:21.041817 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:52:21.041781 4939 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["models-as-a-service/payload-processing-external-model-6889df475b-4kdg9"] Nov 03 00:52:21.043610 ip-10-0-140-107 kubenswrapper[4939]: W1103 00:52:21.043584 4939 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod81fff666_afd7_4fc5_b20a_c3573dc23d38.slice/crio-7a2d1d230334e58109733996394372170ebf3ba659d9dce450d3c22e65198f9d WatchSource:0}: Error finding container 7a2d1d230334e58109733996394372170ebf3ba659d9dce450d3c22e65198f9d: Status 404 returned error can't find the container with id 7a2d1d230334e58109733996394372170ebf3ba659d9dce450d3c22e65198f9d Nov 03 00:52:21.160823 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:52:21.160788 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 00:52:21.453934 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:52:21.450329 4939 patch_prober.go:29] interesting pod/e2e-worker-w2-f7680d-openshift-default-c4488d5c9-h5j8d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.65:15021/healthz/ready\": dial tcp 10.132.0.65:15021: connect: connection refused" start-of-body= Nov 03 00:52:21.453934 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:52:21.450398 4939 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-f7680d-openshift-default-c4488d5c9-h5j8d" podUID="4fe351d2-d66c-4cd4-8bf7-264a40d2551c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.65:15021/healthz/ready\": dial tcp 10.132.0.65:15021: connect: connection refused" Nov 03 00:52:21.757991 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:52:21.754212 4939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-6889df475b-4kdg9" event={"ID":"81fff666-afd7-4fc5-b20a-c3573dc23d38","Type":"ContainerStarted","Data":"2d9e5706d1d2cb71dd7a4a1683999e2cd0afaa100689fafd037c24268c6f579c"} Nov 03 00:52:21.757991 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:52:21.754281 4939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-6889df475b-4kdg9" event={"ID":"81fff666-afd7-4fc5-b20a-c3573dc23d38","Type":"ContainerStarted","Data":"7a2d1d230334e58109733996394372170ebf3ba659d9dce450d3c22e65198f9d"} Nov 03 00:52:21.757991 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:52:21.755210 4939 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="models-as-a-service/payload-processing-external-model-6889df475b-4kdg9" Nov 03 00:52:22.345727 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:52:22.345684 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 00:52:22.447931 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:52:22.447316 4939 patch_prober.go:29] interesting pod/e2e-worker-w2-f7680d-openshift-default-c4488d5c9-h5j8d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.65:15021/healthz/ready\": dial tcp 10.132.0.65:15021: connect: connection refused" start-of-body= Nov 03 00:52:22.447931 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:52:22.447388 4939 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-f7680d-openshift-default-c4488d5c9-h5j8d" podUID="4fe351d2-d66c-4cd4-8bf7-264a40d2551c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.65:15021/healthz/ready\": dial tcp 10.132.0.65:15021: connect: connection refused" Nov 03 00:52:23.516498 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:52:23.516460 4939 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-f7680d-openshift-default-c4488d5c9-h5j8d" Nov 03 00:52:23.517289 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:52:23.517265 4939 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-f7680d-openshift-default-c4488d5c9-h5j8d" Nov 03 00:52:23.518696 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:52:23.518674 4939 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-f7680d-openshift-default-c4488d5c9-h5j8d" Nov 03 00:52:23.549271 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:52:23.547854 4939 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="models-as-a-service/payload-processing-external-model-6889df475b-4kdg9" podStartSLOduration=3.547835838 podStartE2EDuration="3.547835838s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:52:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:52:21.795088245 +0000 UTC m=+690.059423216" watchObservedRunningTime="2026-11-03 00:52:23.547835838 +0000 UTC m=+691.812170784" Nov 03 00:52:23.639028 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:52:23.638744 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 00:52:24.393379 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:52:24.393340 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 00:52:25.092663 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:52:25.092625 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 00:52:25.970851 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:52:25.970813 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 00:52:26.609465 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:52:26.609430 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 00:52:26.796685 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:52:26.796650 4939 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-650be0-openshift-default-5c7c5fb7d9-vj4qt"] Nov 03 00:52:26.800833 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:52:26.800811 4939 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-650be0-openshift-default-5c7c5fb7d9-vj4qt" Nov 03 00:52:26.816786 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:52:26.816753 4939 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-650be0-openshift-default-5c7c5fb7d9-vj4qt"] Nov 03 00:52:26.854517 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:52:26.854483 4939 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-4pw7v"] Nov 03 00:52:26.938312 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:52:26.938227 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5b6c2a38-1b62-41a5-803f-3111bd896857-istio-podinfo\") pod \"e2e-worker-w4-650be0-openshift-default-5c7c5fb7d9-vj4qt\" (UID: \"5b6c2a38-1b62-41a5-803f-3111bd896857\") " pod="openshift-ingress/e2e-worker-w4-650be0-openshift-default-5c7c5fb7d9-vj4qt" Nov 03 00:52:26.938312 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:52:26.938268 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5b6c2a38-1b62-41a5-803f-3111bd896857-workload-socket\") pod \"e2e-worker-w4-650be0-openshift-default-5c7c5fb7d9-vj4qt\" (UID: \"5b6c2a38-1b62-41a5-803f-3111bd896857\") " pod="openshift-ingress/e2e-worker-w4-650be0-openshift-default-5c7c5fb7d9-vj4qt" Nov 03 00:52:26.938509 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:52:26.938370 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5b6c2a38-1b62-41a5-803f-3111bd896857-credential-socket\") pod \"e2e-worker-w4-650be0-openshift-default-5c7c5fb7d9-vj4qt\" (UID: \"5b6c2a38-1b62-41a5-803f-3111bd896857\") " pod="openshift-ingress/e2e-worker-w4-650be0-openshift-default-5c7c5fb7d9-vj4qt" Nov 03 00:52:26.938509 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:52:26.938422 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zn9t7\" (UniqueName: \"kubernetes.io/projected/5b6c2a38-1b62-41a5-803f-3111bd896857-kube-api-access-zn9t7\") pod \"e2e-worker-w4-650be0-openshift-default-5c7c5fb7d9-vj4qt\" (UID: \"5b6c2a38-1b62-41a5-803f-3111bd896857\") " pod="openshift-ingress/e2e-worker-w4-650be0-openshift-default-5c7c5fb7d9-vj4qt" Nov 03 00:52:26.938587 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:52:26.938541 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5b6c2a38-1b62-41a5-803f-3111bd896857-istio-envoy\") pod \"e2e-worker-w4-650be0-openshift-default-5c7c5fb7d9-vj4qt\" (UID: \"5b6c2a38-1b62-41a5-803f-3111bd896857\") " pod="openshift-ingress/e2e-worker-w4-650be0-openshift-default-5c7c5fb7d9-vj4qt" Nov 03 00:52:26.938648 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:52:26.938590 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5b6c2a38-1b62-41a5-803f-3111bd896857-istiod-ca-cert\") pod \"e2e-worker-w4-650be0-openshift-default-5c7c5fb7d9-vj4qt\" (UID: \"5b6c2a38-1b62-41a5-803f-3111bd896857\") " pod="openshift-ingress/e2e-worker-w4-650be0-openshift-default-5c7c5fb7d9-vj4qt" Nov 03 00:52:26.938847 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:52:26.938831 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5b6c2a38-1b62-41a5-803f-3111bd896857-istio-data\") pod \"e2e-worker-w4-650be0-openshift-default-5c7c5fb7d9-vj4qt\" (UID: \"5b6c2a38-1b62-41a5-803f-3111bd896857\") " pod="openshift-ingress/e2e-worker-w4-650be0-openshift-default-5c7c5fb7d9-vj4qt" Nov 03 00:52:26.938946 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:52:26.938868 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5b6c2a38-1b62-41a5-803f-3111bd896857-istio-token\") pod \"e2e-worker-w4-650be0-openshift-default-5c7c5fb7d9-vj4qt\" (UID: \"5b6c2a38-1b62-41a5-803f-3111bd896857\") " pod="openshift-ingress/e2e-worker-w4-650be0-openshift-default-5c7c5fb7d9-vj4qt" Nov 03 00:52:26.939017 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:52:26.938989 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5b6c2a38-1b62-41a5-803f-3111bd896857-workload-certs\") pod \"e2e-worker-w4-650be0-openshift-default-5c7c5fb7d9-vj4qt\" (UID: \"5b6c2a38-1b62-41a5-803f-3111bd896857\") " pod="openshift-ingress/e2e-worker-w4-650be0-openshift-default-5c7c5fb7d9-vj4qt" Nov 03 00:52:27.039874 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:52:27.039838 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zn9t7\" (UniqueName: \"kubernetes.io/projected/5b6c2a38-1b62-41a5-803f-3111bd896857-kube-api-access-zn9t7\") pod \"e2e-worker-w4-650be0-openshift-default-5c7c5fb7d9-vj4qt\" (UID: \"5b6c2a38-1b62-41a5-803f-3111bd896857\") " pod="openshift-ingress/e2e-worker-w4-650be0-openshift-default-5c7c5fb7d9-vj4qt" Nov 03 00:52:27.040061 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:52:27.039884 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5b6c2a38-1b62-41a5-803f-3111bd896857-istio-envoy\") pod \"e2e-worker-w4-650be0-openshift-default-5c7c5fb7d9-vj4qt\" (UID: \"5b6c2a38-1b62-41a5-803f-3111bd896857\") " pod="openshift-ingress/e2e-worker-w4-650be0-openshift-default-5c7c5fb7d9-vj4qt" Nov 03 00:52:27.040061 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:52:27.039945 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5b6c2a38-1b62-41a5-803f-3111bd896857-istiod-ca-cert\") pod \"e2e-worker-w4-650be0-openshift-default-5c7c5fb7d9-vj4qt\" (UID: \"5b6c2a38-1b62-41a5-803f-3111bd896857\") " pod="openshift-ingress/e2e-worker-w4-650be0-openshift-default-5c7c5fb7d9-vj4qt" Nov 03 00:52:27.040061 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:52:27.040018 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5b6c2a38-1b62-41a5-803f-3111bd896857-istio-data\") pod \"e2e-worker-w4-650be0-openshift-default-5c7c5fb7d9-vj4qt\" (UID: \"5b6c2a38-1b62-41a5-803f-3111bd896857\") " pod="openshift-ingress/e2e-worker-w4-650be0-openshift-default-5c7c5fb7d9-vj4qt" Nov 03 00:52:27.040061 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:52:27.040056 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5b6c2a38-1b62-41a5-803f-3111bd896857-istio-token\") pod \"e2e-worker-w4-650be0-openshift-default-5c7c5fb7d9-vj4qt\" (UID: \"5b6c2a38-1b62-41a5-803f-3111bd896857\") " pod="openshift-ingress/e2e-worker-w4-650be0-openshift-default-5c7c5fb7d9-vj4qt" Nov 03 00:52:27.040263 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:52:27.040093 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5b6c2a38-1b62-41a5-803f-3111bd896857-workload-certs\") pod \"e2e-worker-w4-650be0-openshift-default-5c7c5fb7d9-vj4qt\" (UID: \"5b6c2a38-1b62-41a5-803f-3111bd896857\") " pod="openshift-ingress/e2e-worker-w4-650be0-openshift-default-5c7c5fb7d9-vj4qt" Nov 03 00:52:27.040263 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:52:27.040194 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5b6c2a38-1b62-41a5-803f-3111bd896857-istio-podinfo\") pod \"e2e-worker-w4-650be0-openshift-default-5c7c5fb7d9-vj4qt\" (UID: \"5b6c2a38-1b62-41a5-803f-3111bd896857\") " pod="openshift-ingress/e2e-worker-w4-650be0-openshift-default-5c7c5fb7d9-vj4qt" Nov 03 00:52:27.040263 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:52:27.040225 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5b6c2a38-1b62-41a5-803f-3111bd896857-workload-socket\") pod \"e2e-worker-w4-650be0-openshift-default-5c7c5fb7d9-vj4qt\" (UID: \"5b6c2a38-1b62-41a5-803f-3111bd896857\") " pod="openshift-ingress/e2e-worker-w4-650be0-openshift-default-5c7c5fb7d9-vj4qt" Nov 03 00:52:27.040263 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:52:27.040258 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5b6c2a38-1b62-41a5-803f-3111bd896857-credential-socket\") pod \"e2e-worker-w4-650be0-openshift-default-5c7c5fb7d9-vj4qt\" (UID: \"5b6c2a38-1b62-41a5-803f-3111bd896857\") " pod="openshift-ingress/e2e-worker-w4-650be0-openshift-default-5c7c5fb7d9-vj4qt" Nov 03 00:52:27.040619 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:52:27.040569 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5b6c2a38-1b62-41a5-803f-3111bd896857-workload-certs\") pod \"e2e-worker-w4-650be0-openshift-default-5c7c5fb7d9-vj4qt\" (UID: \"5b6c2a38-1b62-41a5-803f-3111bd896857\") " pod="openshift-ingress/e2e-worker-w4-650be0-openshift-default-5c7c5fb7d9-vj4qt" Nov 03 00:52:27.040619 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:52:27.040583 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5b6c2a38-1b62-41a5-803f-3111bd896857-istio-data\") pod \"e2e-worker-w4-650be0-openshift-default-5c7c5fb7d9-vj4qt\" (UID: \"5b6c2a38-1b62-41a5-803f-3111bd896857\") " pod="openshift-ingress/e2e-worker-w4-650be0-openshift-default-5c7c5fb7d9-vj4qt" Nov 03 00:52:27.040768 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:52:27.040622 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5b6c2a38-1b62-41a5-803f-3111bd896857-workload-socket\") pod \"e2e-worker-w4-650be0-openshift-default-5c7c5fb7d9-vj4qt\" (UID: \"5b6c2a38-1b62-41a5-803f-3111bd896857\") " pod="openshift-ingress/e2e-worker-w4-650be0-openshift-default-5c7c5fb7d9-vj4qt" Nov 03 00:52:27.040768 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:52:27.040669 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5b6c2a38-1b62-41a5-803f-3111bd896857-credential-socket\") pod \"e2e-worker-w4-650be0-openshift-default-5c7c5fb7d9-vj4qt\" (UID: \"5b6c2a38-1b62-41a5-803f-3111bd896857\") " pod="openshift-ingress/e2e-worker-w4-650be0-openshift-default-5c7c5fb7d9-vj4qt" Nov 03 00:52:27.041072 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:52:27.041055 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5b6c2a38-1b62-41a5-803f-3111bd896857-istiod-ca-cert\") pod \"e2e-worker-w4-650be0-openshift-default-5c7c5fb7d9-vj4qt\" (UID: \"5b6c2a38-1b62-41a5-803f-3111bd896857\") " pod="openshift-ingress/e2e-worker-w4-650be0-openshift-default-5c7c5fb7d9-vj4qt" Nov 03 00:52:27.042308 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:52:27.042286 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5b6c2a38-1b62-41a5-803f-3111bd896857-istio-envoy\") pod \"e2e-worker-w4-650be0-openshift-default-5c7c5fb7d9-vj4qt\" (UID: \"5b6c2a38-1b62-41a5-803f-3111bd896857\") " pod="openshift-ingress/e2e-worker-w4-650be0-openshift-default-5c7c5fb7d9-vj4qt" Nov 03 00:52:27.042685 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:52:27.042664 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5b6c2a38-1b62-41a5-803f-3111bd896857-istio-podinfo\") pod \"e2e-worker-w4-650be0-openshift-default-5c7c5fb7d9-vj4qt\" (UID: \"5b6c2a38-1b62-41a5-803f-3111bd896857\") " pod="openshift-ingress/e2e-worker-w4-650be0-openshift-default-5c7c5fb7d9-vj4qt" Nov 03 00:52:27.054344 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:52:27.054299 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5b6c2a38-1b62-41a5-803f-3111bd896857-istio-token\") pod \"e2e-worker-w4-650be0-openshift-default-5c7c5fb7d9-vj4qt\" (UID: \"5b6c2a38-1b62-41a5-803f-3111bd896857\") " pod="openshift-ingress/e2e-worker-w4-650be0-openshift-default-5c7c5fb7d9-vj4qt" Nov 03 00:52:27.054936 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:52:27.054874 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zn9t7\" (UniqueName: \"kubernetes.io/projected/5b6c2a38-1b62-41a5-803f-3111bd896857-kube-api-access-zn9t7\") pod \"e2e-worker-w4-650be0-openshift-default-5c7c5fb7d9-vj4qt\" (UID: \"5b6c2a38-1b62-41a5-803f-3111bd896857\") " pod="openshift-ingress/e2e-worker-w4-650be0-openshift-default-5c7c5fb7d9-vj4qt" Nov 03 00:52:27.118860 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:52:27.118822 4939 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-650be0-openshift-default-5c7c5fb7d9-vj4qt" Nov 03 00:52:27.335475 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:52:27.335438 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 00:52:27.440045 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:52:27.436120 4939 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-650be0-openshift-default-5c7c5fb7d9-vj4qt"] Nov 03 00:52:27.441318 ip-10-0-140-107 kubenswrapper[4939]: W1103 00:52:27.441286 4939 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5b6c2a38_1b62_41a5_803f_3111bd896857.slice/crio-c227f263df4a59967d13b1ab7b3030f9f80daf4da817e13e0a794290190ff69f WatchSource:0}: Error finding container c227f263df4a59967d13b1ab7b3030f9f80daf4da817e13e0a794290190ff69f: Status 404 returned error can't find the container with id c227f263df4a59967d13b1ab7b3030f9f80daf4da817e13e0a794290190ff69f Nov 03 00:52:27.453286 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:52:27.447690 4939 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 00:52:27.453286 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:52:27.447780 4939 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 00:52:27.453286 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:52:27.447825 4939 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 00:52:27.827081 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:52:27.826983 4939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-650be0-openshift-default-5c7c5fb7d9-vj4qt" event={"ID":"5b6c2a38-1b62-41a5-803f-3111bd896857","Type":"ContainerStarted","Data":"2bb3d70abeafaa14cc7a438cb90ec1c4c3d30ee30fb4c12a87d7eb1804af321c"} Nov 03 00:52:27.827891 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:52:27.827861 4939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-650be0-openshift-default-5c7c5fb7d9-vj4qt" event={"ID":"5b6c2a38-1b62-41a5-803f-3111bd896857","Type":"ContainerStarted","Data":"c227f263df4a59967d13b1ab7b3030f9f80daf4da817e13e0a794290190ff69f"} Nov 03 00:52:27.865962 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:52:27.865651 4939 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-650be0-openshift-default-5c7c5fb7d9-vj4qt" podStartSLOduration=1.865629795 podStartE2EDuration="1.865629795s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:52:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:52:27.859546768 +0000 UTC m=+696.123881736" watchObservedRunningTime="2026-11-03 00:52:27.865629795 +0000 UTC m=+696.129964740" Nov 03 00:52:28.121934 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:52:28.121750 4939 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-650be0-openshift-default-5c7c5fb7d9-vj4qt" Nov 03 00:52:28.122148 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:52:28.121938 4939 patch_prober.go:29] interesting pod/e2e-worker-w4-650be0-openshift-default-5c7c5fb7d9-vj4qt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.67:15021/healthz/ready\": dial tcp 10.132.0.67:15021: connect: connection refused" start-of-body= Nov 03 00:52:28.122148 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:52:28.121987 4939 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-650be0-openshift-default-5c7c5fb7d9-vj4qt" podUID="5b6c2a38-1b62-41a5-803f-3111bd896857" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.67:15021/healthz/ready\": dial tcp 10.132.0.67:15021: connect: connection refused" Nov 03 00:52:28.261003 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:52:28.256342 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 00:52:28.376033 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:52:28.370495 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 00:52:28.746098 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:52:28.744171 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 00:52:28.798006 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:52:28.796599 4939 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w4-650be0/e2e-conflict-a-45480c9d-kserve-5c75dbc7bb-tjrmz"] Nov 03 00:52:28.812688 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:52:28.812615 4939 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-650be0/e2e-conflict-a-45480c9d-kserve-5c75dbc7bb-tjrmz" Nov 03 00:52:28.812688 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:52:28.812638 4939 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-650be0/e2e-conflict-a-45480c9d-kserve-5c75dbc7bb-tjrmz"] Nov 03 00:52:28.829774 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:52:28.829743 4939 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w4-650be0\"/\"openshift-service-ca.crt\"" Nov 03 00:52:28.830241 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:52:28.829930 4939 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-650be0\"/\"e2e-conflict-a-45480c9d-kserve-self-signed-certs\"" Nov 03 00:52:28.830241 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:52:28.830031 4939 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w4-650be0\"/\"kube-root-ca.crt\"" Nov 03 00:52:28.830241 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:52:28.830146 4939 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-650be0\"/\"default-dockercfg-ggdgp\"" Nov 03 00:52:28.886957 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:52:28.886751 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 00:52:29.025750 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:52:29.025648 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e6e864bc-d2bc-4c9f-81f8-0bed22c00883-tls-certs\") pod \"e2e-conflict-a-45480c9d-kserve-5c75dbc7bb-tjrmz\" (UID: \"e6e864bc-d2bc-4c9f-81f8-0bed22c00883\") " pod="e2e-models-e2e-worker-w4-650be0/e2e-conflict-a-45480c9d-kserve-5c75dbc7bb-tjrmz" Nov 03 00:52:29.025963 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:52:29.025761 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e6e864bc-d2bc-4c9f-81f8-0bed22c00883-dshm\") pod \"e2e-conflict-a-45480c9d-kserve-5c75dbc7bb-tjrmz\" (UID: \"e6e864bc-d2bc-4c9f-81f8-0bed22c00883\") " pod="e2e-models-e2e-worker-w4-650be0/e2e-conflict-a-45480c9d-kserve-5c75dbc7bb-tjrmz" Nov 03 00:52:29.025963 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:52:29.025820 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e6e864bc-d2bc-4c9f-81f8-0bed22c00883-tmp-dir\") pod \"e2e-conflict-a-45480c9d-kserve-5c75dbc7bb-tjrmz\" (UID: \"e6e864bc-d2bc-4c9f-81f8-0bed22c00883\") " pod="e2e-models-e2e-worker-w4-650be0/e2e-conflict-a-45480c9d-kserve-5c75dbc7bb-tjrmz" Nov 03 00:52:29.025963 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:52:29.025853 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e6e864bc-d2bc-4c9f-81f8-0bed22c00883-model-cache\") pod \"e2e-conflict-a-45480c9d-kserve-5c75dbc7bb-tjrmz\" (UID: \"e6e864bc-d2bc-4c9f-81f8-0bed22c00883\") " pod="e2e-models-e2e-worker-w4-650be0/e2e-conflict-a-45480c9d-kserve-5c75dbc7bb-tjrmz" Nov 03 00:52:29.026140 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:52:29.025977 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e6e864bc-d2bc-4c9f-81f8-0bed22c00883-home\") pod \"e2e-conflict-a-45480c9d-kserve-5c75dbc7bb-tjrmz\" (UID: \"e6e864bc-d2bc-4c9f-81f8-0bed22c00883\") " pod="e2e-models-e2e-worker-w4-650be0/e2e-conflict-a-45480c9d-kserve-5c75dbc7bb-tjrmz" Nov 03 00:52:29.026190 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:52:29.026136 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nm85n\" (UniqueName: \"kubernetes.io/projected/e6e864bc-d2bc-4c9f-81f8-0bed22c00883-kube-api-access-nm85n\") pod \"e2e-conflict-a-45480c9d-kserve-5c75dbc7bb-tjrmz\" (UID: \"e6e864bc-d2bc-4c9f-81f8-0bed22c00883\") " pod="e2e-models-e2e-worker-w4-650be0/e2e-conflict-a-45480c9d-kserve-5c75dbc7bb-tjrmz" Nov 03 00:52:29.120090 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:52:29.120030 4939 patch_prober.go:29] interesting pod/e2e-worker-w4-650be0-openshift-default-5c7c5fb7d9-vj4qt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.67:15021/healthz/ready\": dial tcp 10.132.0.67:15021: connect: connection refused" start-of-body= Nov 03 00:52:29.120279 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:52:29.120088 4939 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-650be0-openshift-default-5c7c5fb7d9-vj4qt" podUID="5b6c2a38-1b62-41a5-803f-3111bd896857" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.67:15021/healthz/ready\": dial tcp 10.132.0.67:15021: connect: connection refused" Nov 03 00:52:29.127294 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:52:29.127251 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e6e864bc-d2bc-4c9f-81f8-0bed22c00883-home\") pod \"e2e-conflict-a-45480c9d-kserve-5c75dbc7bb-tjrmz\" (UID: \"e6e864bc-d2bc-4c9f-81f8-0bed22c00883\") " pod="e2e-models-e2e-worker-w4-650be0/e2e-conflict-a-45480c9d-kserve-5c75dbc7bb-tjrmz" Nov 03 00:52:29.127455 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:52:29.127419 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nm85n\" (UniqueName: \"kubernetes.io/projected/e6e864bc-d2bc-4c9f-81f8-0bed22c00883-kube-api-access-nm85n\") pod \"e2e-conflict-a-45480c9d-kserve-5c75dbc7bb-tjrmz\" (UID: \"e6e864bc-d2bc-4c9f-81f8-0bed22c00883\") " pod="e2e-models-e2e-worker-w4-650be0/e2e-conflict-a-45480c9d-kserve-5c75dbc7bb-tjrmz" Nov 03 00:52:29.127528 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:52:29.127494 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e6e864bc-d2bc-4c9f-81f8-0bed22c00883-tls-certs\") pod \"e2e-conflict-a-45480c9d-kserve-5c75dbc7bb-tjrmz\" (UID: \"e6e864bc-d2bc-4c9f-81f8-0bed22c00883\") " pod="e2e-models-e2e-worker-w4-650be0/e2e-conflict-a-45480c9d-kserve-5c75dbc7bb-tjrmz" Nov 03 00:52:29.127630 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:52:29.127563 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e6e864bc-d2bc-4c9f-81f8-0bed22c00883-dshm\") pod \"e2e-conflict-a-45480c9d-kserve-5c75dbc7bb-tjrmz\" (UID: \"e6e864bc-d2bc-4c9f-81f8-0bed22c00883\") " pod="e2e-models-e2e-worker-w4-650be0/e2e-conflict-a-45480c9d-kserve-5c75dbc7bb-tjrmz" Nov 03 00:52:29.127630 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:52:29.127601 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e6e864bc-d2bc-4c9f-81f8-0bed22c00883-tmp-dir\") pod \"e2e-conflict-a-45480c9d-kserve-5c75dbc7bb-tjrmz\" (UID: \"e6e864bc-d2bc-4c9f-81f8-0bed22c00883\") " pod="e2e-models-e2e-worker-w4-650be0/e2e-conflict-a-45480c9d-kserve-5c75dbc7bb-tjrmz" Nov 03 00:52:29.127744 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:52:29.127645 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e6e864bc-d2bc-4c9f-81f8-0bed22c00883-model-cache\") pod \"e2e-conflict-a-45480c9d-kserve-5c75dbc7bb-tjrmz\" (UID: \"e6e864bc-d2bc-4c9f-81f8-0bed22c00883\") " pod="e2e-models-e2e-worker-w4-650be0/e2e-conflict-a-45480c9d-kserve-5c75dbc7bb-tjrmz" Nov 03 00:52:29.128208 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:52:29.128178 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e6e864bc-d2bc-4c9f-81f8-0bed22c00883-model-cache\") pod \"e2e-conflict-a-45480c9d-kserve-5c75dbc7bb-tjrmz\" (UID: \"e6e864bc-d2bc-4c9f-81f8-0bed22c00883\") " pod="e2e-models-e2e-worker-w4-650be0/e2e-conflict-a-45480c9d-kserve-5c75dbc7bb-tjrmz" Nov 03 00:52:29.128981 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:52:29.128959 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e6e864bc-d2bc-4c9f-81f8-0bed22c00883-home\") pod \"e2e-conflict-a-45480c9d-kserve-5c75dbc7bb-tjrmz\" (UID: \"e6e864bc-d2bc-4c9f-81f8-0bed22c00883\") " pod="e2e-models-e2e-worker-w4-650be0/e2e-conflict-a-45480c9d-kserve-5c75dbc7bb-tjrmz" Nov 03 00:52:29.129275 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:52:29.129244 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e6e864bc-d2bc-4c9f-81f8-0bed22c00883-tmp-dir\") pod \"e2e-conflict-a-45480c9d-kserve-5c75dbc7bb-tjrmz\" (UID: \"e6e864bc-d2bc-4c9f-81f8-0bed22c00883\") " pod="e2e-models-e2e-worker-w4-650be0/e2e-conflict-a-45480c9d-kserve-5c75dbc7bb-tjrmz" Nov 03 00:52:29.145715 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:52:29.143436 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e6e864bc-d2bc-4c9f-81f8-0bed22c00883-dshm\") pod \"e2e-conflict-a-45480c9d-kserve-5c75dbc7bb-tjrmz\" (UID: \"e6e864bc-d2bc-4c9f-81f8-0bed22c00883\") " pod="e2e-models-e2e-worker-w4-650be0/e2e-conflict-a-45480c9d-kserve-5c75dbc7bb-tjrmz" Nov 03 00:52:29.145837 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:52:29.145772 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e6e864bc-d2bc-4c9f-81f8-0bed22c00883-tls-certs\") pod \"e2e-conflict-a-45480c9d-kserve-5c75dbc7bb-tjrmz\" (UID: \"e6e864bc-d2bc-4c9f-81f8-0bed22c00883\") " pod="e2e-models-e2e-worker-w4-650be0/e2e-conflict-a-45480c9d-kserve-5c75dbc7bb-tjrmz" Nov 03 00:52:29.153561 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:52:29.153531 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nm85n\" (UniqueName: \"kubernetes.io/projected/e6e864bc-d2bc-4c9f-81f8-0bed22c00883-kube-api-access-nm85n\") pod \"e2e-conflict-a-45480c9d-kserve-5c75dbc7bb-tjrmz\" (UID: \"e6e864bc-d2bc-4c9f-81f8-0bed22c00883\") " pod="e2e-models-e2e-worker-w4-650be0/e2e-conflict-a-45480c9d-kserve-5c75dbc7bb-tjrmz" Nov 03 00:52:29.158668 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:52:29.158115 4939 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-650be0/e2e-conflict-a-45480c9d-kserve-5c75dbc7bb-tjrmz" Nov 03 00:52:29.296994 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:52:29.271441 4939 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-v5p6s"] Nov 03 00:52:29.296994 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:52:29.296473 4939 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-v5p6s" Nov 03 00:52:29.318443 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:52:29.316523 4939 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-v5p6s"] Nov 03 00:52:29.345936 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:52:29.340970 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e6aa4af1-9a04-4116-9d8a-1d8ef5b04958-istio-data\") pod \"maas-default-gateway-openshift-default-596dc9545f-v5p6s\" (UID: \"e6aa4af1-9a04-4116-9d8a-1d8ef5b04958\") " pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-v5p6s" Nov 03 00:52:29.345936 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:52:29.341057 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e6aa4af1-9a04-4116-9d8a-1d8ef5b04958-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-596dc9545f-v5p6s\" (UID: \"e6aa4af1-9a04-4116-9d8a-1d8ef5b04958\") " pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-v5p6s" Nov 03 00:52:29.345936 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:52:29.341117 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e6aa4af1-9a04-4116-9d8a-1d8ef5b04958-istio-envoy\") pod \"maas-default-gateway-openshift-default-596dc9545f-v5p6s\" (UID: \"e6aa4af1-9a04-4116-9d8a-1d8ef5b04958\") " pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-v5p6s" Nov 03 00:52:29.345936 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:52:29.341141 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e6aa4af1-9a04-4116-9d8a-1d8ef5b04958-istio-podinfo\") pod \"maas-default-gateway-openshift-default-596dc9545f-v5p6s\" (UID: \"e6aa4af1-9a04-4116-9d8a-1d8ef5b04958\") " pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-v5p6s" Nov 03 00:52:29.345936 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:52:29.341197 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e6aa4af1-9a04-4116-9d8a-1d8ef5b04958-workload-socket\") pod \"maas-default-gateway-openshift-default-596dc9545f-v5p6s\" (UID: \"e6aa4af1-9a04-4116-9d8a-1d8ef5b04958\") " pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-v5p6s" Nov 03 00:52:29.345936 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:52:29.341232 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lcmrv\" (UniqueName: \"kubernetes.io/projected/e6aa4af1-9a04-4116-9d8a-1d8ef5b04958-kube-api-access-lcmrv\") pod \"maas-default-gateway-openshift-default-596dc9545f-v5p6s\" (UID: \"e6aa4af1-9a04-4116-9d8a-1d8ef5b04958\") " pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-v5p6s" Nov 03 00:52:29.345936 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:52:29.341268 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e6aa4af1-9a04-4116-9d8a-1d8ef5b04958-credential-socket\") pod \"maas-default-gateway-openshift-default-596dc9545f-v5p6s\" (UID: \"e6aa4af1-9a04-4116-9d8a-1d8ef5b04958\") " pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-v5p6s" Nov 03 00:52:29.345936 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:52:29.341318 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e6aa4af1-9a04-4116-9d8a-1d8ef5b04958-workload-certs\") pod \"maas-default-gateway-openshift-default-596dc9545f-v5p6s\" (UID: \"e6aa4af1-9a04-4116-9d8a-1d8ef5b04958\") " pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-v5p6s" Nov 03 00:52:29.345936 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:52:29.341377 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e6aa4af1-9a04-4116-9d8a-1d8ef5b04958-istio-token\") pod \"maas-default-gateway-openshift-default-596dc9545f-v5p6s\" (UID: \"e6aa4af1-9a04-4116-9d8a-1d8ef5b04958\") " pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-v5p6s" Nov 03 00:52:29.605981 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:52:29.599589 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e6aa4af1-9a04-4116-9d8a-1d8ef5b04958-istio-envoy\") pod \"maas-default-gateway-openshift-default-596dc9545f-v5p6s\" (UID: \"e6aa4af1-9a04-4116-9d8a-1d8ef5b04958\") " pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-v5p6s" Nov 03 00:52:29.605981 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:52:29.599639 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e6aa4af1-9a04-4116-9d8a-1d8ef5b04958-istio-podinfo\") pod \"maas-default-gateway-openshift-default-596dc9545f-v5p6s\" (UID: \"e6aa4af1-9a04-4116-9d8a-1d8ef5b04958\") " pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-v5p6s" Nov 03 00:52:29.605981 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:52:29.599706 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e6aa4af1-9a04-4116-9d8a-1d8ef5b04958-workload-socket\") pod \"maas-default-gateway-openshift-default-596dc9545f-v5p6s\" (UID: \"e6aa4af1-9a04-4116-9d8a-1d8ef5b04958\") " pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-v5p6s" Nov 03 00:52:29.605981 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:52:29.599743 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lcmrv\" (UniqueName: \"kubernetes.io/projected/e6aa4af1-9a04-4116-9d8a-1d8ef5b04958-kube-api-access-lcmrv\") pod \"maas-default-gateway-openshift-default-596dc9545f-v5p6s\" (UID: \"e6aa4af1-9a04-4116-9d8a-1d8ef5b04958\") " pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-v5p6s" Nov 03 00:52:29.605981 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:52:29.599784 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e6aa4af1-9a04-4116-9d8a-1d8ef5b04958-credential-socket\") pod \"maas-default-gateway-openshift-default-596dc9545f-v5p6s\" (UID: \"e6aa4af1-9a04-4116-9d8a-1d8ef5b04958\") " pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-v5p6s" Nov 03 00:52:29.605981 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:52:29.599838 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e6aa4af1-9a04-4116-9d8a-1d8ef5b04958-workload-certs\") pod \"maas-default-gateway-openshift-default-596dc9545f-v5p6s\" (UID: \"e6aa4af1-9a04-4116-9d8a-1d8ef5b04958\") " pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-v5p6s" Nov 03 00:52:29.605981 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:52:29.599901 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e6aa4af1-9a04-4116-9d8a-1d8ef5b04958-istio-token\") pod \"maas-default-gateway-openshift-default-596dc9545f-v5p6s\" (UID: \"e6aa4af1-9a04-4116-9d8a-1d8ef5b04958\") " pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-v5p6s" Nov 03 00:52:29.605981 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:52:29.599987 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e6aa4af1-9a04-4116-9d8a-1d8ef5b04958-istio-data\") pod \"maas-default-gateway-openshift-default-596dc9545f-v5p6s\" (UID: \"e6aa4af1-9a04-4116-9d8a-1d8ef5b04958\") " pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-v5p6s" Nov 03 00:52:29.605981 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:52:29.600072 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e6aa4af1-9a04-4116-9d8a-1d8ef5b04958-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-596dc9545f-v5p6s\" (UID: \"e6aa4af1-9a04-4116-9d8a-1d8ef5b04958\") " pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-v5p6s" Nov 03 00:52:29.605981 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:52:29.600976 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e6aa4af1-9a04-4116-9d8a-1d8ef5b04958-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-596dc9545f-v5p6s\" (UID: \"e6aa4af1-9a04-4116-9d8a-1d8ef5b04958\") " pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-v5p6s" Nov 03 00:52:29.643568 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:52:29.643110 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e6aa4af1-9a04-4116-9d8a-1d8ef5b04958-credential-socket\") pod \"maas-default-gateway-openshift-default-596dc9545f-v5p6s\" (UID: \"e6aa4af1-9a04-4116-9d8a-1d8ef5b04958\") " pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-v5p6s" Nov 03 00:52:29.645899 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:52:29.645872 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e6aa4af1-9a04-4116-9d8a-1d8ef5b04958-workload-socket\") pod \"maas-default-gateway-openshift-default-596dc9545f-v5p6s\" (UID: \"e6aa4af1-9a04-4116-9d8a-1d8ef5b04958\") " pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-v5p6s" Nov 03 00:52:29.646706 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:52:29.646681 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e6aa4af1-9a04-4116-9d8a-1d8ef5b04958-workload-certs\") pod \"maas-default-gateway-openshift-default-596dc9545f-v5p6s\" (UID: \"e6aa4af1-9a04-4116-9d8a-1d8ef5b04958\") " pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-v5p6s" Nov 03 00:52:29.697945 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:52:29.648774 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e6aa4af1-9a04-4116-9d8a-1d8ef5b04958-istio-data\") pod \"maas-default-gateway-openshift-default-596dc9545f-v5p6s\" (UID: \"e6aa4af1-9a04-4116-9d8a-1d8ef5b04958\") " pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-v5p6s" Nov 03 00:52:29.697945 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:52:29.674302 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e6aa4af1-9a04-4116-9d8a-1d8ef5b04958-istio-envoy\") pod \"maas-default-gateway-openshift-default-596dc9545f-v5p6s\" (UID: \"e6aa4af1-9a04-4116-9d8a-1d8ef5b04958\") " pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-v5p6s" Nov 03 00:52:29.697945 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:52:29.689520 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e6aa4af1-9a04-4116-9d8a-1d8ef5b04958-istio-podinfo\") pod \"maas-default-gateway-openshift-default-596dc9545f-v5p6s\" (UID: \"e6aa4af1-9a04-4116-9d8a-1d8ef5b04958\") " pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-v5p6s" Nov 03 00:52:29.706831 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:52:29.705449 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e6aa4af1-9a04-4116-9d8a-1d8ef5b04958-istio-token\") pod \"maas-default-gateway-openshift-default-596dc9545f-v5p6s\" (UID: \"e6aa4af1-9a04-4116-9d8a-1d8ef5b04958\") " pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-v5p6s" Nov 03 00:52:29.745972 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:52:29.730738 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lcmrv\" (UniqueName: \"kubernetes.io/projected/e6aa4af1-9a04-4116-9d8a-1d8ef5b04958-kube-api-access-lcmrv\") pod \"maas-default-gateway-openshift-default-596dc9545f-v5p6s\" (UID: \"e6aa4af1-9a04-4116-9d8a-1d8ef5b04958\") " pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-v5p6s" Nov 03 00:52:29.929990 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:52:29.925218 4939 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-v5p6s" Nov 03 00:52:30.033451 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:52:30.033357 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 00:52:30.126321 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:52:30.126286 4939 patch_prober.go:29] interesting pod/e2e-worker-w4-650be0-openshift-default-5c7c5fb7d9-vj4qt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.67:15021/healthz/ready\": dial tcp 10.132.0.67:15021: connect: connection refused" start-of-body= Nov 03 00:52:30.126555 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:52:30.126526 4939 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-650be0-openshift-default-5c7c5fb7d9-vj4qt" podUID="5b6c2a38-1b62-41a5-803f-3111bd896857" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.67:15021/healthz/ready\": dial tcp 10.132.0.67:15021: connect: connection refused" Nov 03 00:52:31.138959 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:52:31.138497 4939 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-650be0/e2e-conflict-a-45480c9d-kserve-5c75dbc7bb-tjrmz"] Nov 03 00:52:31.229267 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:52:31.157594 4939 patch_prober.go:29] interesting pod/e2e-worker-w4-650be0-openshift-default-5c7c5fb7d9-vj4qt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.67:15021/healthz/ready\": dial tcp 10.132.0.67:15021: connect: connection refused" start-of-body= Nov 03 00:52:31.229503 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:52:31.229468 4939 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-650be0-openshift-default-5c7c5fb7d9-vj4qt" podUID="5b6c2a38-1b62-41a5-803f-3111bd896857" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.67:15021/healthz/ready\": dial tcp 10.132.0.67:15021: connect: connection refused" Nov 03 00:52:31.442884 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:52:31.442841 4939 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 00:52:31.445855 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:52:31.445825 4939 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 00:52:31.468364 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:52:31.468326 4939 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 00:52:31.469015 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:52:31.445004 4939 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-v5p6s"] Nov 03 00:52:31.708771 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:52:31.708660 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 00:52:32.119610 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:52:32.119573 4939 patch_prober.go:29] interesting pod/e2e-worker-w4-650be0-openshift-default-5c7c5fb7d9-vj4qt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.67:15021/healthz/ready\": dial tcp 10.132.0.67:15021: connect: connection refused" start-of-body= Nov 03 00:52:32.119790 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:52:32.119628 4939 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-650be0-openshift-default-5c7c5fb7d9-vj4qt" podUID="5b6c2a38-1b62-41a5-803f-3111bd896857" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.67:15021/healthz/ready\": dial tcp 10.132.0.67:15021: connect: connection refused" Nov 03 00:52:32.404591 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:52:32.404497 4939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-v5p6s" event={"ID":"e6aa4af1-9a04-4116-9d8a-1d8ef5b04958","Type":"ContainerStarted","Data":"711c3fe33e81559763d3b8cb65e4ebc9ce079fd44a89a33770d461e47788ab8d"} Nov 03 00:52:32.404591 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:52:32.404566 4939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-v5p6s" event={"ID":"e6aa4af1-9a04-4116-9d8a-1d8ef5b04958","Type":"ContainerStarted","Data":"7b3ea74c929b86b7994a4d419fafe16a4831c286632729ce86ad7648f5a0549b"} Nov 03 00:52:32.406749 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:52:32.406357 4939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-650be0/e2e-conflict-a-45480c9d-kserve-5c75dbc7bb-tjrmz" event={"ID":"e6e864bc-d2bc-4c9f-81f8-0bed22c00883","Type":"ContainerStarted","Data":"4ff35624f3f7a2c5f8283325b4420facbff1ccc760eba36146da5bcf048dca59"} Nov 03 00:52:32.406749 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:52:32.406397 4939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-650be0/e2e-conflict-a-45480c9d-kserve-5c75dbc7bb-tjrmz" event={"ID":"e6e864bc-d2bc-4c9f-81f8-0bed22c00883","Type":"ContainerStarted","Data":"9d753ec2458f819759c729f73b6334068d1ded722b1c634db6d8e446c8d06d57"} Nov 03 00:52:32.485931 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:52:32.485096 4939 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-v5p6s" podStartSLOduration=3.48507525 podStartE2EDuration="3.48507525s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:52:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:52:32.46941952 +0000 UTC m=+700.733754468" watchObservedRunningTime="2026-11-03 00:52:32.48507525 +0000 UTC m=+700.749410197" Nov 03 00:52:32.516275 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:52:32.516224 4939 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w4-650be0/e2e-conflict-a-45480c9d-kserve-5c75dbc7bb-tjrmz" podStartSLOduration=4.516208988 podStartE2EDuration="4.516208988s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:52:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:52:32.514204585 +0000 UTC m=+700.778539530" watchObservedRunningTime="2026-11-03 00:52:32.516208988 +0000 UTC m=+700.780543925" Nov 03 00:52:32.668982 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:52:32.667976 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 00:52:32.930497 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:52:32.930416 4939 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-v5p6s" Nov 03 00:52:32.932077 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:52:32.932055 4939 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-596dc9545f-v5p6s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.69:15021/healthz/ready\": dial tcp 10.132.0.69:15021: connect: connection refused" start-of-body= Nov 03 00:52:32.932169 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:52:32.932102 4939 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-v5p6s" podUID="e6aa4af1-9a04-4116-9d8a-1d8ef5b04958" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.69:15021/healthz/ready\": dial tcp 10.132.0.69:15021: connect: connection refused" Nov 03 00:52:33.119617 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:52:33.119574 4939 patch_prober.go:29] interesting pod/e2e-worker-w4-650be0-openshift-default-5c7c5fb7d9-vj4qt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.67:15021/healthz/ready\": dial tcp 10.132.0.67:15021: connect: connection refused" start-of-body= Nov 03 00:52:33.119804 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:52:33.119632 4939 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-650be0-openshift-default-5c7c5fb7d9-vj4qt" podUID="5b6c2a38-1b62-41a5-803f-3111bd896857" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.67:15021/healthz/ready\": dial tcp 10.132.0.67:15021: connect: connection refused" Nov 03 00:52:33.166933 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:52:33.166043 4939 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-650be0/e2e-conflict-a-45480c9d-kserve-5c75dbc7bb-tjrmz" Nov 03 00:52:33.562953 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:52:33.562455 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 00:52:33.798420 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:52:33.798312 4939 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="models-as-a-service/payload-processing-external-model-6889df475b-4kdg9" Nov 03 00:52:33.930336 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:52:33.930300 4939 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-596dc9545f-v5p6s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.69:15021/healthz/ready\": dial tcp 10.132.0.69:15021: connect: connection refused" start-of-body= Nov 03 00:52:33.930502 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:52:33.930363 4939 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-v5p6s" podUID="e6aa4af1-9a04-4116-9d8a-1d8ef5b04958" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.69:15021/healthz/ready\": dial tcp 10.132.0.69:15021: connect: connection refused" Nov 03 00:52:34.119833 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:52:34.119784 4939 patch_prober.go:29] interesting pod/e2e-worker-w4-650be0-openshift-default-5c7c5fb7d9-vj4qt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.67:15021/healthz/ready\": dial tcp 10.132.0.67:15021: connect: connection refused" start-of-body= Nov 03 00:52:34.120070 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:52:34.119843 4939 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-650be0-openshift-default-5c7c5fb7d9-vj4qt" podUID="5b6c2a38-1b62-41a5-803f-3111bd896857" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.67:15021/healthz/ready\": dial tcp 10.132.0.67:15021: connect: connection refused" Nov 03 00:52:34.308678 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:52:34.308581 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 00:52:34.930561 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:52:34.930515 4939 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-596dc9545f-v5p6s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.69:15021/healthz/ready\": dial tcp 10.132.0.69:15021: connect: connection refused" start-of-body= Nov 03 00:52:34.931048 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:52:34.930591 4939 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-v5p6s" podUID="e6aa4af1-9a04-4116-9d8a-1d8ef5b04958" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.69:15021/healthz/ready\": dial tcp 10.132.0.69:15021: connect: connection refused" Nov 03 00:52:35.108755 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:52:35.108706 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 00:52:35.124009 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:52:35.123980 4939 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-650be0-openshift-default-5c7c5fb7d9-vj4qt" Nov 03 00:52:35.124373 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:52:35.124349 4939 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-650be0-openshift-default-5c7c5fb7d9-vj4qt" Nov 03 00:52:35.125124 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:52:35.125105 4939 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-650be0-openshift-default-5c7c5fb7d9-vj4qt" Nov 03 00:52:35.908566 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:52:35.908537 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 00:52:35.930707 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:52:35.930676 4939 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-596dc9545f-v5p6s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.69:15021/healthz/ready\": dial tcp 10.132.0.69:15021: connect: connection refused" start-of-body= Nov 03 00:52:35.931174 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:52:35.930733 4939 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-v5p6s" podUID="e6aa4af1-9a04-4116-9d8a-1d8ef5b04958" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.69:15021/healthz/ready\": dial tcp 10.132.0.69:15021: connect: connection refused" Nov 03 00:52:36.681827 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:52:36.681792 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 00:52:36.930333 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:52:36.930293 4939 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-596dc9545f-v5p6s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.69:15021/healthz/ready\": dial tcp 10.132.0.69:15021: connect: connection refused" start-of-body= Nov 03 00:52:36.930527 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:52:36.930360 4939 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-v5p6s" podUID="e6aa4af1-9a04-4116-9d8a-1d8ef5b04958" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.69:15021/healthz/ready\": dial tcp 10.132.0.69:15021: connect: connection refused" Nov 03 00:52:37.630062 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:52:37.630024 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 00:52:37.930301 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:52:37.930218 4939 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-596dc9545f-v5p6s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.69:15021/healthz/ready\": dial tcp 10.132.0.69:15021: connect: connection refused" start-of-body= Nov 03 00:52:37.930301 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:52:37.930277 4939 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-v5p6s" podUID="e6aa4af1-9a04-4116-9d8a-1d8ef5b04958" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.69:15021/healthz/ready\": dial tcp 10.132.0.69:15021: connect: connection refused" Nov 03 00:52:38.539600 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:52:38.539546 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 00:52:38.937933 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:52:38.937033 4939 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-596dc9545f-v5p6s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.69:15021/healthz/ready\": dial tcp 10.132.0.69:15021: connect: connection refused" start-of-body= Nov 03 00:52:38.937933 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:52:38.937099 4939 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-v5p6s" podUID="e6aa4af1-9a04-4116-9d8a-1d8ef5b04958" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.69:15021/healthz/ready\": dial tcp 10.132.0.69:15021: connect: connection refused" Nov 03 00:52:39.163454 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:52:39.163417 4939 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w4-650be0/e2e-conflict-a-45480c9d-kserve-5c75dbc7bb-tjrmz" Nov 03 00:52:39.931012 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:52:39.922793 4939 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w4-650be0/e2e-conflict-a-45480c9d-kserve-5c75dbc7bb-tjrmz" Nov 03 00:52:39.931713 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:52:39.931673 4939 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-596dc9545f-v5p6s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.69:15021/healthz/ready\": dial tcp 10.132.0.69:15021: connect: connection refused" start-of-body= Nov 03 00:52:39.931845 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:52:39.931736 4939 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-v5p6s" podUID="e6aa4af1-9a04-4116-9d8a-1d8ef5b04958" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.69:15021/healthz/ready\": dial tcp 10.132.0.69:15021: connect: connection refused" Nov 03 00:52:40.142932 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:52:40.142658 4939 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-650be0/e2e-conflict-a-45480c9d-kserve-5c75dbc7bb-tjrmz" Nov 03 00:52:40.162932 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:52:40.157977 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 00:52:41.823807 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:52:41.823718 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 00:52:41.979590 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:52:41.979551 4939 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-596dc9545f-v5p6s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.69:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 00:52:41.979877 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:52:41.979847 4939 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-v5p6s" podUID="e6aa4af1-9a04-4116-9d8a-1d8ef5b04958" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.69:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 00:52:42.032634 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:52:42.032601 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 00:52:42.428926 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:52:42.428884 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 00:52:42.534666 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:52:42.534629 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 00:52:42.980718 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:52:42.980680 4939 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-596dc9545f-v5p6s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.69:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 00:52:42.981191 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:52:42.980746 4939 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-v5p6s" podUID="e6aa4af1-9a04-4116-9d8a-1d8ef5b04958" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.69:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 00:52:43.422030 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:52:43.417582 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 00:52:44.022082 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:52:44.022040 4939 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-596dc9545f-v5p6s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.69:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 00:52:44.022570 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:52:44.022150 4939 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-v5p6s" podUID="e6aa4af1-9a04-4116-9d8a-1d8ef5b04958" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.69:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 00:52:44.312503 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:52:44.312454 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 00:52:44.461775 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:52:44.461742 4939 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-v5p6s" Nov 03 00:52:44.462272 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:52:44.462245 4939 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-v5p6s" Nov 03 00:52:44.463172 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:52:44.463149 4939 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-v5p6s" Nov 03 00:52:44.684934 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:52:44.684869 4939 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-4pw7v"] Nov 03 00:52:44.989640 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:52:44.989559 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 00:52:45.691289 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:52:45.691248 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 00:52:45.822934 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:52:45.822579 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 00:52:46.599394 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:52:46.599360 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 00:52:47.302272 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:52:47.302242 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 00:52:48.031861 ip-10-0-140-107 kubenswrapper[4939]: E1103 00:52:48.031825 4939 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-sthjh" containerName="extractor" Nov 03 00:52:48.103688 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:52:48.103642 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 00:52:48.968217 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:52:48.968186 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 00:52:49.649692 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:52:49.649656 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 00:52:50.486807 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:52:50.486765 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 00:52:51.086161 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:52:51.086117 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 00:52:51.743351 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:52:51.743315 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 00:52:52.866532 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:52:52.866489 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 00:52:53.816035 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:52:53.815996 4939 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-596dc9545f-pcw82 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.42:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 00:52:53.816372 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:52:53.816338 4939 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-pcw82" podUID="48186e38-73f5-47c6-a90a-d82afeac4a52" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.42:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 00:52:53.841052 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:52:53.841013 4939 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-596dc9545f-l9ccf container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.43:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 00:52:53.841359 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:52:53.841329 4939 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-l9ccf" podUID="c10d48f9-d5fc-4c4c-96c4-6531114191b8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.43:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 00:52:54.445941 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:52:54.445122 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 00:52:55.404548 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:52:55.404508 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 00:52:55.433399 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:52:55.433362 4939 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["models-as-a-service/payload-processing-external-model-6889df475b-4kdg9"] Nov 03 00:52:56.189227 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:52:56.189193 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 00:52:56.681989 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:52:56.681944 4939 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w2-f7680d/distinct-w2-f7680d-kserve-756f874b98-qnpv8"] Nov 03 00:52:56.690004 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:52:56.689970 4939 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-f7680d/distinct-w2-f7680d-kserve-756f874b98-qnpv8" Nov 03 00:52:56.695957 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:52:56.695904 4939 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w2-f7680d\"/\"distinct-w2-f7680d-kserve-self-signed-certs\"" Nov 03 00:52:56.713825 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:52:56.713781 4939 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w2-f7680d/distinct-w2-f7680d-kserve-756f874b98-qnpv8"] Nov 03 00:52:56.750271 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:52:56.750230 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 00:52:56.860497 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:52:56.860460 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/836753fc-f9ef-4f61-b3fe-e5b92b7ee27b-tmp-dir\") pod \"distinct-w2-f7680d-kserve-756f874b98-qnpv8\" (UID: \"836753fc-f9ef-4f61-b3fe-e5b92b7ee27b\") " pod="e2e-models-e2e-worker-w2-f7680d/distinct-w2-f7680d-kserve-756f874b98-qnpv8" Nov 03 00:52:56.861479 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:52:56.861454 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/836753fc-f9ef-4f61-b3fe-e5b92b7ee27b-tls-certs\") pod \"distinct-w2-f7680d-kserve-756f874b98-qnpv8\" (UID: \"836753fc-f9ef-4f61-b3fe-e5b92b7ee27b\") " pod="e2e-models-e2e-worker-w2-f7680d/distinct-w2-f7680d-kserve-756f874b98-qnpv8" Nov 03 00:52:56.861625 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:52:56.861509 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jcp5z\" (UniqueName: \"kubernetes.io/projected/836753fc-f9ef-4f61-b3fe-e5b92b7ee27b-kube-api-access-jcp5z\") pod \"distinct-w2-f7680d-kserve-756f874b98-qnpv8\" (UID: \"836753fc-f9ef-4f61-b3fe-e5b92b7ee27b\") " pod="e2e-models-e2e-worker-w2-f7680d/distinct-w2-f7680d-kserve-756f874b98-qnpv8" Nov 03 00:52:56.861625 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:52:56.861540 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/836753fc-f9ef-4f61-b3fe-e5b92b7ee27b-dshm\") pod \"distinct-w2-f7680d-kserve-756f874b98-qnpv8\" (UID: \"836753fc-f9ef-4f61-b3fe-e5b92b7ee27b\") " pod="e2e-models-e2e-worker-w2-f7680d/distinct-w2-f7680d-kserve-756f874b98-qnpv8" Nov 03 00:52:56.861625 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:52:56.861580 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/836753fc-f9ef-4f61-b3fe-e5b92b7ee27b-home\") pod \"distinct-w2-f7680d-kserve-756f874b98-qnpv8\" (UID: \"836753fc-f9ef-4f61-b3fe-e5b92b7ee27b\") " pod="e2e-models-e2e-worker-w2-f7680d/distinct-w2-f7680d-kserve-756f874b98-qnpv8" Nov 03 00:52:56.861795 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:52:56.861630 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/836753fc-f9ef-4f61-b3fe-e5b92b7ee27b-model-cache\") pod \"distinct-w2-f7680d-kserve-756f874b98-qnpv8\" (UID: \"836753fc-f9ef-4f61-b3fe-e5b92b7ee27b\") " pod="e2e-models-e2e-worker-w2-f7680d/distinct-w2-f7680d-kserve-756f874b98-qnpv8" Nov 03 00:52:56.896349 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:52:56.896310 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 00:52:56.969002 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:52:56.968839 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/836753fc-f9ef-4f61-b3fe-e5b92b7ee27b-tls-certs\") pod \"distinct-w2-f7680d-kserve-756f874b98-qnpv8\" (UID: \"836753fc-f9ef-4f61-b3fe-e5b92b7ee27b\") " pod="e2e-models-e2e-worker-w2-f7680d/distinct-w2-f7680d-kserve-756f874b98-qnpv8" Nov 03 00:52:56.969279 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:52:56.969256 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jcp5z\" (UniqueName: \"kubernetes.io/projected/836753fc-f9ef-4f61-b3fe-e5b92b7ee27b-kube-api-access-jcp5z\") pod \"distinct-w2-f7680d-kserve-756f874b98-qnpv8\" (UID: \"836753fc-f9ef-4f61-b3fe-e5b92b7ee27b\") " pod="e2e-models-e2e-worker-w2-f7680d/distinct-w2-f7680d-kserve-756f874b98-qnpv8" Nov 03 00:52:56.969398 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:52:56.969383 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/836753fc-f9ef-4f61-b3fe-e5b92b7ee27b-dshm\") pod \"distinct-w2-f7680d-kserve-756f874b98-qnpv8\" (UID: \"836753fc-f9ef-4f61-b3fe-e5b92b7ee27b\") " pod="e2e-models-e2e-worker-w2-f7680d/distinct-w2-f7680d-kserve-756f874b98-qnpv8" Nov 03 00:52:56.969507 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:52:56.969496 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/836753fc-f9ef-4f61-b3fe-e5b92b7ee27b-home\") pod \"distinct-w2-f7680d-kserve-756f874b98-qnpv8\" (UID: \"836753fc-f9ef-4f61-b3fe-e5b92b7ee27b\") " pod="e2e-models-e2e-worker-w2-f7680d/distinct-w2-f7680d-kserve-756f874b98-qnpv8" Nov 03 00:52:56.969657 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:52:56.969645 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/836753fc-f9ef-4f61-b3fe-e5b92b7ee27b-model-cache\") pod \"distinct-w2-f7680d-kserve-756f874b98-qnpv8\" (UID: \"836753fc-f9ef-4f61-b3fe-e5b92b7ee27b\") " pod="e2e-models-e2e-worker-w2-f7680d/distinct-w2-f7680d-kserve-756f874b98-qnpv8" Nov 03 00:52:56.969847 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:52:56.969828 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/836753fc-f9ef-4f61-b3fe-e5b92b7ee27b-tmp-dir\") pod \"distinct-w2-f7680d-kserve-756f874b98-qnpv8\" (UID: \"836753fc-f9ef-4f61-b3fe-e5b92b7ee27b\") " pod="e2e-models-e2e-worker-w2-f7680d/distinct-w2-f7680d-kserve-756f874b98-qnpv8" Nov 03 00:52:56.970496 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:52:56.970474 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/836753fc-f9ef-4f61-b3fe-e5b92b7ee27b-tmp-dir\") pod \"distinct-w2-f7680d-kserve-756f874b98-qnpv8\" (UID: \"836753fc-f9ef-4f61-b3fe-e5b92b7ee27b\") " pod="e2e-models-e2e-worker-w2-f7680d/distinct-w2-f7680d-kserve-756f874b98-qnpv8" Nov 03 00:52:56.976747 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:52:56.976718 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/836753fc-f9ef-4f61-b3fe-e5b92b7ee27b-model-cache\") pod \"distinct-w2-f7680d-kserve-756f874b98-qnpv8\" (UID: \"836753fc-f9ef-4f61-b3fe-e5b92b7ee27b\") " pod="e2e-models-e2e-worker-w2-f7680d/distinct-w2-f7680d-kserve-756f874b98-qnpv8" Nov 03 00:52:56.981640 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:52:56.981610 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/836753fc-f9ef-4f61-b3fe-e5b92b7ee27b-home\") pod \"distinct-w2-f7680d-kserve-756f874b98-qnpv8\" (UID: \"836753fc-f9ef-4f61-b3fe-e5b92b7ee27b\") " pod="e2e-models-e2e-worker-w2-f7680d/distinct-w2-f7680d-kserve-756f874b98-qnpv8" Nov 03 00:52:56.986284 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:52:56.985626 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/836753fc-f9ef-4f61-b3fe-e5b92b7ee27b-tls-certs\") pod \"distinct-w2-f7680d-kserve-756f874b98-qnpv8\" (UID: \"836753fc-f9ef-4f61-b3fe-e5b92b7ee27b\") " pod="e2e-models-e2e-worker-w2-f7680d/distinct-w2-f7680d-kserve-756f874b98-qnpv8" Nov 03 00:52:56.986668 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:52:56.986646 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/836753fc-f9ef-4f61-b3fe-e5b92b7ee27b-dshm\") pod \"distinct-w2-f7680d-kserve-756f874b98-qnpv8\" (UID: \"836753fc-f9ef-4f61-b3fe-e5b92b7ee27b\") " pod="e2e-models-e2e-worker-w2-f7680d/distinct-w2-f7680d-kserve-756f874b98-qnpv8" Nov 03 00:52:56.991712 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:52:56.991686 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jcp5z\" (UniqueName: \"kubernetes.io/projected/836753fc-f9ef-4f61-b3fe-e5b92b7ee27b-kube-api-access-jcp5z\") pod \"distinct-w2-f7680d-kserve-756f874b98-qnpv8\" (UID: \"836753fc-f9ef-4f61-b3fe-e5b92b7ee27b\") " pod="e2e-models-e2e-worker-w2-f7680d/distinct-w2-f7680d-kserve-756f874b98-qnpv8" Nov 03 00:52:57.015243 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:52:57.015214 4939 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-f7680d/distinct-w2-f7680d-kserve-756f874b98-qnpv8" Nov 03 00:52:57.291712 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:52:57.291683 4939 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w2-f7680d/distinct-w2-f7680d-kserve-756f874b98-qnpv8"] Nov 03 00:52:57.292861 ip-10-0-140-107 kubenswrapper[4939]: W1103 00:52:57.292833 4939 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod836753fc_f9ef_4f61_b3fe_e5b92b7ee27b.slice/crio-8b3d5b6db611466c6dc5c2d2e82dc2be9814cac4ab327a523754b0dc99f2e1dd WatchSource:0}: Error finding container 8b3d5b6db611466c6dc5c2d2e82dc2be9814cac4ab327a523754b0dc99f2e1dd: Status 404 returned error can't find the container with id 8b3d5b6db611466c6dc5c2d2e82dc2be9814cac4ab327a523754b0dc99f2e1dd Nov 03 00:52:57.693104 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:52:57.693070 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 00:52:57.867946 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:52:57.867891 4939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-f7680d/distinct-w2-f7680d-kserve-756f874b98-qnpv8" event={"ID":"836753fc-f9ef-4f61-b3fe-e5b92b7ee27b","Type":"ContainerStarted","Data":"d334fd3c1f721eecec1716833695512720ecab73858e5add00260032237f4f93"} Nov 03 00:52:57.867946 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:52:57.867951 4939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-f7680d/distinct-w2-f7680d-kserve-756f874b98-qnpv8" event={"ID":"836753fc-f9ef-4f61-b3fe-e5b92b7ee27b","Type":"ContainerStarted","Data":"8b3d5b6db611466c6dc5c2d2e82dc2be9814cac4ab327a523754b0dc99f2e1dd"} Nov 03 00:52:57.882689 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:52:57.882641 4939 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w2-f7680d/distinct-w2-f7680d-kserve-756f874b98-qnpv8" podStartSLOduration=1.8826268499999999 podStartE2EDuration="1.88262685s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:52:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:52:57.880899052 +0000 UTC m=+726.145234014" watchObservedRunningTime="2026-11-03 00:52:57.88262685 +0000 UTC m=+726.146961794" Nov 03 00:52:58.019157 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:52:58.019036 4939 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-f7680d/distinct-w2-f7680d-kserve-756f874b98-qnpv8" Nov 03 00:52:58.397992 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:52:58.397937 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 00:52:59.196499 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:52:59.196465 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 00:52:59.969845 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:52:59.969807 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 00:53:00.434675 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:53:00.434631 4939 kuberuntime_container.go:898] "Killing container with a grace period" pod="models-as-a-service/payload-processing-external-model-6889df475b-4kdg9" podUID="81fff666-afd7-4fc5-b20a-c3573dc23d38" containerName="payload-processing" containerID="cri-o://2d9e5706d1d2cb71dd7a4a1683999e2cd0afaa100689fafd037c24268c6f579c" gracePeriod=25 Nov 03 00:53:00.585379 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:53:00.585326 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 00:53:00.719331 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:53:00.719252 4939 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-83755a-openshift-default-c86f9565c-qlgb7"] Nov 03 00:53:00.724377 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:53:00.724351 4939 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-83755a-openshift-default-c86f9565c-qlgb7" Nov 03 00:53:00.737349 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:53:00.737282 4939 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-83755a-openshift-default-c86f9565c-qlgb7"] Nov 03 00:53:00.749424 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:53:00.749387 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ba28e265-fab5-43ba-94b2-dc5b7dd5029f-istio-envoy\") pod \"e2e-worker-w3-83755a-openshift-default-c86f9565c-qlgb7\" (UID: \"ba28e265-fab5-43ba-94b2-dc5b7dd5029f\") " pod="openshift-ingress/e2e-worker-w3-83755a-openshift-default-c86f9565c-qlgb7" Nov 03 00:53:00.749578 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:53:00.749453 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ba28e265-fab5-43ba-94b2-dc5b7dd5029f-workload-certs\") pod \"e2e-worker-w3-83755a-openshift-default-c86f9565c-qlgb7\" (UID: \"ba28e265-fab5-43ba-94b2-dc5b7dd5029f\") " pod="openshift-ingress/e2e-worker-w3-83755a-openshift-default-c86f9565c-qlgb7" Nov 03 00:53:00.749578 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:53:00.749488 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ba28e265-fab5-43ba-94b2-dc5b7dd5029f-istio-token\") pod \"e2e-worker-w3-83755a-openshift-default-c86f9565c-qlgb7\" (UID: \"ba28e265-fab5-43ba-94b2-dc5b7dd5029f\") " pod="openshift-ingress/e2e-worker-w3-83755a-openshift-default-c86f9565c-qlgb7" Nov 03 00:53:00.749578 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:53:00.749525 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ba28e265-fab5-43ba-94b2-dc5b7dd5029f-istio-data\") pod \"e2e-worker-w3-83755a-openshift-default-c86f9565c-qlgb7\" (UID: \"ba28e265-fab5-43ba-94b2-dc5b7dd5029f\") " pod="openshift-ingress/e2e-worker-w3-83755a-openshift-default-c86f9565c-qlgb7" Nov 03 00:53:00.749750 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:53:00.749650 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ba28e265-fab5-43ba-94b2-dc5b7dd5029f-istio-podinfo\") pod \"e2e-worker-w3-83755a-openshift-default-c86f9565c-qlgb7\" (UID: \"ba28e265-fab5-43ba-94b2-dc5b7dd5029f\") " pod="openshift-ingress/e2e-worker-w3-83755a-openshift-default-c86f9565c-qlgb7" Nov 03 00:53:00.749750 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:53:00.749675 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xg7pd\" (UniqueName: \"kubernetes.io/projected/ba28e265-fab5-43ba-94b2-dc5b7dd5029f-kube-api-access-xg7pd\") pod \"e2e-worker-w3-83755a-openshift-default-c86f9565c-qlgb7\" (UID: \"ba28e265-fab5-43ba-94b2-dc5b7dd5029f\") " pod="openshift-ingress/e2e-worker-w3-83755a-openshift-default-c86f9565c-qlgb7" Nov 03 00:53:00.749750 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:53:00.749707 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ba28e265-fab5-43ba-94b2-dc5b7dd5029f-credential-socket\") pod \"e2e-worker-w3-83755a-openshift-default-c86f9565c-qlgb7\" (UID: \"ba28e265-fab5-43ba-94b2-dc5b7dd5029f\") " pod="openshift-ingress/e2e-worker-w3-83755a-openshift-default-c86f9565c-qlgb7" Nov 03 00:53:00.749946 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:53:00.749765 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ba28e265-fab5-43ba-94b2-dc5b7dd5029f-istiod-ca-cert\") pod \"e2e-worker-w3-83755a-openshift-default-c86f9565c-qlgb7\" (UID: \"ba28e265-fab5-43ba-94b2-dc5b7dd5029f\") " pod="openshift-ingress/e2e-worker-w3-83755a-openshift-default-c86f9565c-qlgb7" Nov 03 00:53:00.749946 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:53:00.749823 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ba28e265-fab5-43ba-94b2-dc5b7dd5029f-workload-socket\") pod \"e2e-worker-w3-83755a-openshift-default-c86f9565c-qlgb7\" (UID: \"ba28e265-fab5-43ba-94b2-dc5b7dd5029f\") " pod="openshift-ingress/e2e-worker-w3-83755a-openshift-default-c86f9565c-qlgb7" Nov 03 00:53:00.752032 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:53:00.751703 4939 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-6889df475b-4kdg9" Nov 03 00:53:00.855895 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:53:00.855861 4939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/81fff666-afd7-4fc5-b20a-c3573dc23d38-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/81fff666-afd7-4fc5-b20a-c3573dc23d38-routing-overlay-volume\") pod \"81fff666-afd7-4fc5-b20a-c3573dc23d38\" (UID: \"81fff666-afd7-4fc5-b20a-c3573dc23d38\") " Nov 03 00:53:00.856123 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:53:00.855953 4939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/81fff666-afd7-4fc5-b20a-c3573dc23d38-provider-credentials\" (UniqueName: \"kubernetes.io/projected/81fff666-afd7-4fc5-b20a-c3573dc23d38-provider-credentials\") pod \"81fff666-afd7-4fc5-b20a-c3573dc23d38\" (UID: \"81fff666-afd7-4fc5-b20a-c3573dc23d38\") " Nov 03 00:53:00.856123 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:53:00.856025 4939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/81fff666-afd7-4fc5-b20a-c3573dc23d38-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/81fff666-afd7-4fc5-b20a-c3573dc23d38-plugins-config-volume\") pod \"81fff666-afd7-4fc5-b20a-c3573dc23d38\" (UID: \"81fff666-afd7-4fc5-b20a-c3573dc23d38\") " Nov 03 00:53:00.856264 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:53:00.856132 4939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/81fff666-afd7-4fc5-b20a-c3573dc23d38-kube-api-access-n68z9\" (UniqueName: \"kubernetes.io/projected/81fff666-afd7-4fc5-b20a-c3573dc23d38-kube-api-access-n68z9\") pod \"81fff666-afd7-4fc5-b20a-c3573dc23d38\" (UID: \"81fff666-afd7-4fc5-b20a-c3573dc23d38\") " Nov 03 00:53:00.856339 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:53:00.856314 4939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/81fff666-afd7-4fc5-b20a-c3573dc23d38-routing-overlay-volume" pod "81fff666-afd7-4fc5-b20a-c3573dc23d38" (UID: "81fff666-afd7-4fc5-b20a-c3573dc23d38"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 00:53:00.856393 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:53:00.856349 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ba28e265-fab5-43ba-94b2-dc5b7dd5029f-istio-envoy\") pod \"e2e-worker-w3-83755a-openshift-default-c86f9565c-qlgb7\" (UID: \"ba28e265-fab5-43ba-94b2-dc5b7dd5029f\") " pod="openshift-ingress/e2e-worker-w3-83755a-openshift-default-c86f9565c-qlgb7" Nov 03 00:53:00.856457 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:53:00.856393 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ba28e265-fab5-43ba-94b2-dc5b7dd5029f-workload-certs\") pod \"e2e-worker-w3-83755a-openshift-default-c86f9565c-qlgb7\" (UID: \"ba28e265-fab5-43ba-94b2-dc5b7dd5029f\") " pod="openshift-ingress/e2e-worker-w3-83755a-openshift-default-c86f9565c-qlgb7" Nov 03 00:53:00.856457 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:53:00.856423 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ba28e265-fab5-43ba-94b2-dc5b7dd5029f-istio-token\") pod \"e2e-worker-w3-83755a-openshift-default-c86f9565c-qlgb7\" (UID: \"ba28e265-fab5-43ba-94b2-dc5b7dd5029f\") " pod="openshift-ingress/e2e-worker-w3-83755a-openshift-default-c86f9565c-qlgb7" Nov 03 00:53:00.856457 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:53:00.856445 4939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/81fff666-afd7-4fc5-b20a-c3573dc23d38-plugins-config-volume" pod "81fff666-afd7-4fc5-b20a-c3573dc23d38" (UID: "81fff666-afd7-4fc5-b20a-c3573dc23d38"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 00:53:00.856610 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:53:00.856457 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ba28e265-fab5-43ba-94b2-dc5b7dd5029f-istio-data\") pod \"e2e-worker-w3-83755a-openshift-default-c86f9565c-qlgb7\" (UID: \"ba28e265-fab5-43ba-94b2-dc5b7dd5029f\") " pod="openshift-ingress/e2e-worker-w3-83755a-openshift-default-c86f9565c-qlgb7" Nov 03 00:53:00.856610 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:53:00.856598 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ba28e265-fab5-43ba-94b2-dc5b7dd5029f-istio-podinfo\") pod \"e2e-worker-w3-83755a-openshift-default-c86f9565c-qlgb7\" (UID: \"ba28e265-fab5-43ba-94b2-dc5b7dd5029f\") " pod="openshift-ingress/e2e-worker-w3-83755a-openshift-default-c86f9565c-qlgb7" Nov 03 00:53:00.856711 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:53:00.856697 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xg7pd\" (UniqueName: \"kubernetes.io/projected/ba28e265-fab5-43ba-94b2-dc5b7dd5029f-kube-api-access-xg7pd\") pod \"e2e-worker-w3-83755a-openshift-default-c86f9565c-qlgb7\" (UID: \"ba28e265-fab5-43ba-94b2-dc5b7dd5029f\") " pod="openshift-ingress/e2e-worker-w3-83755a-openshift-default-c86f9565c-qlgb7" Nov 03 00:53:00.856765 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:53:00.856734 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ba28e265-fab5-43ba-94b2-dc5b7dd5029f-credential-socket\") pod \"e2e-worker-w3-83755a-openshift-default-c86f9565c-qlgb7\" (UID: \"ba28e265-fab5-43ba-94b2-dc5b7dd5029f\") " pod="openshift-ingress/e2e-worker-w3-83755a-openshift-default-c86f9565c-qlgb7" Nov 03 00:53:00.856823 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:53:00.856787 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ba28e265-fab5-43ba-94b2-dc5b7dd5029f-istiod-ca-cert\") pod \"e2e-worker-w3-83755a-openshift-default-c86f9565c-qlgb7\" (UID: \"ba28e265-fab5-43ba-94b2-dc5b7dd5029f\") " pod="openshift-ingress/e2e-worker-w3-83755a-openshift-default-c86f9565c-qlgb7" Nov 03 00:53:00.856876 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:53:00.856841 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ba28e265-fab5-43ba-94b2-dc5b7dd5029f-workload-socket\") pod \"e2e-worker-w3-83755a-openshift-default-c86f9565c-qlgb7\" (UID: \"ba28e265-fab5-43ba-94b2-dc5b7dd5029f\") " pod="openshift-ingress/e2e-worker-w3-83755a-openshift-default-c86f9565c-qlgb7" Nov 03 00:53:00.856955 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:53:00.856871 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ba28e265-fab5-43ba-94b2-dc5b7dd5029f-istio-data\") pod \"e2e-worker-w3-83755a-openshift-default-c86f9565c-qlgb7\" (UID: \"ba28e265-fab5-43ba-94b2-dc5b7dd5029f\") " pod="openshift-ingress/e2e-worker-w3-83755a-openshift-default-c86f9565c-qlgb7" Nov 03 00:53:00.857121 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:53:00.857019 4939 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/81fff666-afd7-4fc5-b20a-c3573dc23d38-plugins-config-volume\") on node \"ip-10-0-140-107.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 00:53:00.857121 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:53:00.857045 4939 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/81fff666-afd7-4fc5-b20a-c3573dc23d38-routing-overlay-volume\") on node \"ip-10-0-140-107.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 00:53:00.857247 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:53:00.857148 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ba28e265-fab5-43ba-94b2-dc5b7dd5029f-credential-socket\") pod \"e2e-worker-w3-83755a-openshift-default-c86f9565c-qlgb7\" (UID: \"ba28e265-fab5-43ba-94b2-dc5b7dd5029f\") " pod="openshift-ingress/e2e-worker-w3-83755a-openshift-default-c86f9565c-qlgb7" Nov 03 00:53:00.857247 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:53:00.857181 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ba28e265-fab5-43ba-94b2-dc5b7dd5029f-workload-socket\") pod \"e2e-worker-w3-83755a-openshift-default-c86f9565c-qlgb7\" (UID: \"ba28e265-fab5-43ba-94b2-dc5b7dd5029f\") " pod="openshift-ingress/e2e-worker-w3-83755a-openshift-default-c86f9565c-qlgb7" Nov 03 00:53:00.857532 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:53:00.857503 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ba28e265-fab5-43ba-94b2-dc5b7dd5029f-workload-certs\") pod \"e2e-worker-w3-83755a-openshift-default-c86f9565c-qlgb7\" (UID: \"ba28e265-fab5-43ba-94b2-dc5b7dd5029f\") " pod="openshift-ingress/e2e-worker-w3-83755a-openshift-default-c86f9565c-qlgb7" Nov 03 00:53:00.858034 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:53:00.858009 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ba28e265-fab5-43ba-94b2-dc5b7dd5029f-istiod-ca-cert\") pod \"e2e-worker-w3-83755a-openshift-default-c86f9565c-qlgb7\" (UID: \"ba28e265-fab5-43ba-94b2-dc5b7dd5029f\") " pod="openshift-ingress/e2e-worker-w3-83755a-openshift-default-c86f9565c-qlgb7" Nov 03 00:53:00.858626 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:53:00.858606 4939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/81fff666-afd7-4fc5-b20a-c3573dc23d38-provider-credentials" pod "81fff666-afd7-4fc5-b20a-c3573dc23d38" (UID: "81fff666-afd7-4fc5-b20a-c3573dc23d38"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 00:53:00.858689 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:53:00.858659 4939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/81fff666-afd7-4fc5-b20a-c3573dc23d38-kube-api-access-n68z9" pod "81fff666-afd7-4fc5-b20a-c3573dc23d38" (UID: "81fff666-afd7-4fc5-b20a-c3573dc23d38"). InnerVolumeSpecName "kube-api-access-n68z9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 00:53:00.859114 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:53:00.859091 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ba28e265-fab5-43ba-94b2-dc5b7dd5029f-istio-envoy\") pod \"e2e-worker-w3-83755a-openshift-default-c86f9565c-qlgb7\" (UID: \"ba28e265-fab5-43ba-94b2-dc5b7dd5029f\") " pod="openshift-ingress/e2e-worker-w3-83755a-openshift-default-c86f9565c-qlgb7" Nov 03 00:53:00.859199 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:53:00.859147 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ba28e265-fab5-43ba-94b2-dc5b7dd5029f-istio-podinfo\") pod \"e2e-worker-w3-83755a-openshift-default-c86f9565c-qlgb7\" (UID: \"ba28e265-fab5-43ba-94b2-dc5b7dd5029f\") " pod="openshift-ingress/e2e-worker-w3-83755a-openshift-default-c86f9565c-qlgb7" Nov 03 00:53:00.864994 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:53:00.864969 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ba28e265-fab5-43ba-94b2-dc5b7dd5029f-istio-token\") pod \"e2e-worker-w3-83755a-openshift-default-c86f9565c-qlgb7\" (UID: \"ba28e265-fab5-43ba-94b2-dc5b7dd5029f\") " pod="openshift-ingress/e2e-worker-w3-83755a-openshift-default-c86f9565c-qlgb7" Nov 03 00:53:00.865458 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:53:00.865438 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xg7pd\" (UniqueName: \"kubernetes.io/projected/ba28e265-fab5-43ba-94b2-dc5b7dd5029f-kube-api-access-xg7pd\") pod \"e2e-worker-w3-83755a-openshift-default-c86f9565c-qlgb7\" (UID: \"ba28e265-fab5-43ba-94b2-dc5b7dd5029f\") " pod="openshift-ingress/e2e-worker-w3-83755a-openshift-default-c86f9565c-qlgb7" Nov 03 00:53:00.890785 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:53:00.890753 4939 generic.go:356] "Generic (PLEG): container finished" podID="81fff666-afd7-4fc5-b20a-c3573dc23d38" containerID="2d9e5706d1d2cb71dd7a4a1683999e2cd0afaa100689fafd037c24268c6f579c" exitCode=0 Nov 03 00:53:00.890953 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:53:00.890797 4939 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-6889df475b-4kdg9" Nov 03 00:53:00.890953 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:53:00.890830 4939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-6889df475b-4kdg9" event={"ID":"81fff666-afd7-4fc5-b20a-c3573dc23d38","Type":"ContainerDied","Data":"2d9e5706d1d2cb71dd7a4a1683999e2cd0afaa100689fafd037c24268c6f579c"} Nov 03 00:53:00.890953 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:53:00.890881 4939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-6889df475b-4kdg9" event={"ID":"81fff666-afd7-4fc5-b20a-c3573dc23d38","Type":"ContainerDied","Data":"7a2d1d230334e58109733996394372170ebf3ba659d9dce450d3c22e65198f9d"} Nov 03 00:53:00.890953 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:53:00.890903 4939 scope.go:122] "RemoveContainer" containerID="2d9e5706d1d2cb71dd7a4a1683999e2cd0afaa100689fafd037c24268c6f579c" Nov 03 00:53:00.905246 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:53:00.905220 4939 scope.go:122] "RemoveContainer" containerID="2d9e5706d1d2cb71dd7a4a1683999e2cd0afaa100689fafd037c24268c6f579c" Nov 03 00:53:00.905629 ip-10-0-140-107 kubenswrapper[4939]: E1103 00:53:00.905602 4939 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2d9e5706d1d2cb71dd7a4a1683999e2cd0afaa100689fafd037c24268c6f579c\": container with ID starting with 2d9e5706d1d2cb71dd7a4a1683999e2cd0afaa100689fafd037c24268c6f579c not found: ID does not exist" containerID="2d9e5706d1d2cb71dd7a4a1683999e2cd0afaa100689fafd037c24268c6f579c" Nov 03 00:53:00.905735 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:53:00.905634 4939 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2d9e5706d1d2cb71dd7a4a1683999e2cd0afaa100689fafd037c24268c6f579c"} err="failed to get container status \"2d9e5706d1d2cb71dd7a4a1683999e2cd0afaa100689fafd037c24268c6f579c\": rpc error: code = NotFound desc = could not find container \"2d9e5706d1d2cb71dd7a4a1683999e2cd0afaa100689fafd037c24268c6f579c\": container with ID starting with 2d9e5706d1d2cb71dd7a4a1683999e2cd0afaa100689fafd037c24268c6f579c not found: ID does not exist" Nov 03 00:53:00.917357 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:53:00.917332 4939 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["models-as-a-service/payload-processing-external-model-6889df475b-4kdg9"] Nov 03 00:53:00.924133 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:53:00.924104 4939 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["models-as-a-service/payload-processing-external-model-6889df475b-4kdg9"] Nov 03 00:53:00.958805 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:53:00.958767 4939 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/81fff666-afd7-4fc5-b20a-c3573dc23d38-provider-credentials\") on node \"ip-10-0-140-107.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 00:53:00.959019 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:53:00.958935 4939 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-n68z9\" (UniqueName: \"kubernetes.io/projected/81fff666-afd7-4fc5-b20a-c3573dc23d38-kube-api-access-n68z9\") on node \"ip-10-0-140-107.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 00:53:01.067374 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:53:01.067283 4939 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-83755a-openshift-default-c86f9565c-qlgb7" Nov 03 00:53:01.351446 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:53:01.351412 4939 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-83755a-openshift-default-c86f9565c-qlgb7"] Nov 03 00:53:01.353710 ip-10-0-140-107 kubenswrapper[4939]: W1103 00:53:01.353678 4939 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podba28e265_fab5_43ba_94b2_dc5b7dd5029f.slice/crio-dcee058b87d0628003e1115acf538db0ef71c6e3326e04ecfee850cccbd04f11 WatchSource:0}: Error finding container dcee058b87d0628003e1115acf538db0ef71c6e3326e04ecfee850cccbd04f11: Status 404 returned error can't find the container with id dcee058b87d0628003e1115acf538db0ef71c6e3326e04ecfee850cccbd04f11 Nov 03 00:53:01.356068 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:53:01.356035 4939 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 00:53:01.356137 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:53:01.356101 4939 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 00:53:01.356137 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:53:01.356131 4939 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 00:53:01.378145 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:53:01.378120 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 00:53:01.876340 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:53:01.876302 4939 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="81fff666-afd7-4fc5-b20a-c3573dc23d38" path="/var/lib/kubelet/pods/81fff666-afd7-4fc5-b20a-c3573dc23d38/volumes" Nov 03 00:53:01.898714 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:53:01.898682 4939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-83755a-openshift-default-c86f9565c-qlgb7" event={"ID":"ba28e265-fab5-43ba-94b2-dc5b7dd5029f","Type":"ContainerStarted","Data":"417abf9a2d53f92307b997d0dd67ef2989b4a47fbb9f30f80d4fcc23d92ab776"} Nov 03 00:53:01.898931 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:53:01.898733 4939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-83755a-openshift-default-c86f9565c-qlgb7" event={"ID":"ba28e265-fab5-43ba-94b2-dc5b7dd5029f","Type":"ContainerStarted","Data":"dcee058b87d0628003e1115acf538db0ef71c6e3326e04ecfee850cccbd04f11"} Nov 03 00:53:01.925186 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:53:01.925131 4939 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-83755a-openshift-default-c86f9565c-qlgb7" podStartSLOduration=1.925113918 podStartE2EDuration="1.925113918s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:53:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:53:01.922721436 +0000 UTC m=+730.187056381" watchObservedRunningTime="2026-11-03 00:53:01.925113918 +0000 UTC m=+730.189448864" Nov 03 00:53:02.072934 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:53:02.072018 4939 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-83755a-openshift-default-c86f9565c-qlgb7" Nov 03 00:53:02.072934 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:53:02.072722 4939 patch_prober.go:29] interesting pod/e2e-worker-w3-83755a-openshift-default-c86f9565c-qlgb7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.71:15021/healthz/ready\": dial tcp 10.132.0.71:15021: connect: connection refused" start-of-body= Nov 03 00:53:02.072934 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:53:02.072766 4939 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-83755a-openshift-default-c86f9565c-qlgb7" podUID="ba28e265-fab5-43ba-94b2-dc5b7dd5029f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.71:15021/healthz/ready\": dial tcp 10.132.0.71:15021: connect: connection refused" Nov 03 00:53:02.209583 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:53:02.209501 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 00:53:02.252933 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:53:02.246518 4939 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-4pw7v"] Nov 03 00:53:03.051932 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:53:03.051396 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 00:53:03.070789 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:53:03.070310 4939 patch_prober.go:29] interesting pod/e2e-worker-w3-83755a-openshift-default-c86f9565c-qlgb7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.71:15021/healthz/ready\": dial tcp 10.132.0.71:15021: connect: connection refused" start-of-body= Nov 03 00:53:03.070789 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:53:03.070379 4939 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-83755a-openshift-default-c86f9565c-qlgb7" podUID="ba28e265-fab5-43ba-94b2-dc5b7dd5029f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.71:15021/healthz/ready\": dial tcp 10.132.0.71:15021: connect: connection refused" Nov 03 00:53:04.068825 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:53:04.068325 4939 patch_prober.go:29] interesting pod/e2e-worker-w3-83755a-openshift-default-c86f9565c-qlgb7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.71:15021/healthz/ready\": dial tcp 10.132.0.71:15021: connect: connection refused" start-of-body= Nov 03 00:53:04.068825 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:53:04.068409 4939 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-83755a-openshift-default-c86f9565c-qlgb7" podUID="ba28e265-fab5-43ba-94b2-dc5b7dd5029f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.71:15021/healthz/ready\": dial tcp 10.132.0.71:15021: connect: connection refused" Nov 03 00:53:04.239901 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:53:04.239858 4939 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w3-83755a/distinct-2-w3-83755a-kserve-6cf66b995d-2l47b"] Nov 03 00:53:04.244622 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:53:04.244590 4939 cpu_manager.go:414] "containerMap: removing container" podUID="81fff666-afd7-4fc5-b20a-c3573dc23d38" containerName="payload-processing" Nov 03 00:53:04.244844 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:53:04.244826 4939 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="81fff666-afd7-4fc5-b20a-c3573dc23d38" containerName="payload-processing" Nov 03 00:53:04.245158 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:53:04.245141 4939 memory_manager.go:365] "RemoveStaleState removing state" podUID="81fff666-afd7-4fc5-b20a-c3573dc23d38" containerName="payload-processing" Nov 03 00:53:04.250639 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:53:04.250617 4939 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-83755a/distinct-2-w3-83755a-kserve-6cf66b995d-2l47b" Nov 03 00:53:04.262977 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:53:04.260965 4939 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w3-83755a\"/\"kube-root-ca.crt\"" Nov 03 00:53:04.262977 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:53:04.262262 4939 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w3-83755a\"/\"distinct-2-w3-83755a-kserve-self-signed-certs\"" Nov 03 00:53:04.262977 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:53:04.262316 4939 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w3-83755a\"/\"openshift-service-ca.crt\"" Nov 03 00:53:04.262977 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:53:04.262363 4939 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w3-83755a\"/\"default-dockercfg-tmlfc\"" Nov 03 00:53:04.286943 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:53:04.271476 4939 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w3-83755a/distinct-2-w3-83755a-kserve-6cf66b995d-2l47b"] Nov 03 00:53:04.340935 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:53:04.340880 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 00:53:04.474957 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:53:04.461154 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c3b05e88-5d5b-4d03-b28d-832f22e3c868-dshm\") pod \"distinct-2-w3-83755a-kserve-6cf66b995d-2l47b\" (UID: \"c3b05e88-5d5b-4d03-b28d-832f22e3c868\") " pod="e2e-models-e2e-worker-w3-83755a/distinct-2-w3-83755a-kserve-6cf66b995d-2l47b" Nov 03 00:53:04.474957 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:53:04.461225 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c3b05e88-5d5b-4d03-b28d-832f22e3c868-home\") pod \"distinct-2-w3-83755a-kserve-6cf66b995d-2l47b\" (UID: \"c3b05e88-5d5b-4d03-b28d-832f22e3c868\") " pod="e2e-models-e2e-worker-w3-83755a/distinct-2-w3-83755a-kserve-6cf66b995d-2l47b" Nov 03 00:53:04.474957 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:53:04.461257 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c3b05e88-5d5b-4d03-b28d-832f22e3c868-tmp-dir\") pod \"distinct-2-w3-83755a-kserve-6cf66b995d-2l47b\" (UID: \"c3b05e88-5d5b-4d03-b28d-832f22e3c868\") " pod="e2e-models-e2e-worker-w3-83755a/distinct-2-w3-83755a-kserve-6cf66b995d-2l47b" Nov 03 00:53:04.474957 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:53:04.461304 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c3b05e88-5d5b-4d03-b28d-832f22e3c868-model-cache\") pod \"distinct-2-w3-83755a-kserve-6cf66b995d-2l47b\" (UID: \"c3b05e88-5d5b-4d03-b28d-832f22e3c868\") " pod="e2e-models-e2e-worker-w3-83755a/distinct-2-w3-83755a-kserve-6cf66b995d-2l47b" Nov 03 00:53:04.474957 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:53:04.461348 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2s8z2\" (UniqueName: \"kubernetes.io/projected/c3b05e88-5d5b-4d03-b28d-832f22e3c868-kube-api-access-2s8z2\") pod \"distinct-2-w3-83755a-kserve-6cf66b995d-2l47b\" (UID: \"c3b05e88-5d5b-4d03-b28d-832f22e3c868\") " pod="e2e-models-e2e-worker-w3-83755a/distinct-2-w3-83755a-kserve-6cf66b995d-2l47b" Nov 03 00:53:04.474957 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:53:04.461404 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c3b05e88-5d5b-4d03-b28d-832f22e3c868-tls-certs\") pod \"distinct-2-w3-83755a-kserve-6cf66b995d-2l47b\" (UID: \"c3b05e88-5d5b-4d03-b28d-832f22e3c868\") " pod="e2e-models-e2e-worker-w3-83755a/distinct-2-w3-83755a-kserve-6cf66b995d-2l47b" Nov 03 00:53:04.593038 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:53:04.569835 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c3b05e88-5d5b-4d03-b28d-832f22e3c868-model-cache\") pod \"distinct-2-w3-83755a-kserve-6cf66b995d-2l47b\" (UID: \"c3b05e88-5d5b-4d03-b28d-832f22e3c868\") " pod="e2e-models-e2e-worker-w3-83755a/distinct-2-w3-83755a-kserve-6cf66b995d-2l47b" Nov 03 00:53:04.602523 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:53:04.602482 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c3b05e88-5d5b-4d03-b28d-832f22e3c868-model-cache\") pod \"distinct-2-w3-83755a-kserve-6cf66b995d-2l47b\" (UID: \"c3b05e88-5d5b-4d03-b28d-832f22e3c868\") " pod="e2e-models-e2e-worker-w3-83755a/distinct-2-w3-83755a-kserve-6cf66b995d-2l47b" Nov 03 00:53:04.629230 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:53:04.569895 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2s8z2\" (UniqueName: \"kubernetes.io/projected/c3b05e88-5d5b-4d03-b28d-832f22e3c868-kube-api-access-2s8z2\") pod \"distinct-2-w3-83755a-kserve-6cf66b995d-2l47b\" (UID: \"c3b05e88-5d5b-4d03-b28d-832f22e3c868\") " pod="e2e-models-e2e-worker-w3-83755a/distinct-2-w3-83755a-kserve-6cf66b995d-2l47b" Nov 03 00:53:04.629601 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:53:04.629571 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c3b05e88-5d5b-4d03-b28d-832f22e3c868-tls-certs\") pod \"distinct-2-w3-83755a-kserve-6cf66b995d-2l47b\" (UID: \"c3b05e88-5d5b-4d03-b28d-832f22e3c868\") " pod="e2e-models-e2e-worker-w3-83755a/distinct-2-w3-83755a-kserve-6cf66b995d-2l47b" Nov 03 00:53:04.640038 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:53:04.630413 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c3b05e88-5d5b-4d03-b28d-832f22e3c868-dshm\") pod \"distinct-2-w3-83755a-kserve-6cf66b995d-2l47b\" (UID: \"c3b05e88-5d5b-4d03-b28d-832f22e3c868\") " pod="e2e-models-e2e-worker-w3-83755a/distinct-2-w3-83755a-kserve-6cf66b995d-2l47b" Nov 03 00:53:04.641140 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:53:04.641108 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c3b05e88-5d5b-4d03-b28d-832f22e3c868-home\") pod \"distinct-2-w3-83755a-kserve-6cf66b995d-2l47b\" (UID: \"c3b05e88-5d5b-4d03-b28d-832f22e3c868\") " pod="e2e-models-e2e-worker-w3-83755a/distinct-2-w3-83755a-kserve-6cf66b995d-2l47b" Nov 03 00:53:04.641477 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:53:04.641461 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c3b05e88-5d5b-4d03-b28d-832f22e3c868-tmp-dir\") pod \"distinct-2-w3-83755a-kserve-6cf66b995d-2l47b\" (UID: \"c3b05e88-5d5b-4d03-b28d-832f22e3c868\") " pod="e2e-models-e2e-worker-w3-83755a/distinct-2-w3-83755a-kserve-6cf66b995d-2l47b" Nov 03 00:53:04.657960 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:53:04.647293 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c3b05e88-5d5b-4d03-b28d-832f22e3c868-home\") pod \"distinct-2-w3-83755a-kserve-6cf66b995d-2l47b\" (UID: \"c3b05e88-5d5b-4d03-b28d-832f22e3c868\") " pod="e2e-models-e2e-worker-w3-83755a/distinct-2-w3-83755a-kserve-6cf66b995d-2l47b" Nov 03 00:53:04.657960 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:53:04.648563 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c3b05e88-5d5b-4d03-b28d-832f22e3c868-tmp-dir\") pod \"distinct-2-w3-83755a-kserve-6cf66b995d-2l47b\" (UID: \"c3b05e88-5d5b-4d03-b28d-832f22e3c868\") " pod="e2e-models-e2e-worker-w3-83755a/distinct-2-w3-83755a-kserve-6cf66b995d-2l47b" Nov 03 00:53:04.833883 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:53:04.833810 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c3b05e88-5d5b-4d03-b28d-832f22e3c868-tls-certs\") pod \"distinct-2-w3-83755a-kserve-6cf66b995d-2l47b\" (UID: \"c3b05e88-5d5b-4d03-b28d-832f22e3c868\") " pod="e2e-models-e2e-worker-w3-83755a/distinct-2-w3-83755a-kserve-6cf66b995d-2l47b" Nov 03 00:53:04.838940 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:53:04.834656 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2s8z2\" (UniqueName: \"kubernetes.io/projected/c3b05e88-5d5b-4d03-b28d-832f22e3c868-kube-api-access-2s8z2\") pod \"distinct-2-w3-83755a-kserve-6cf66b995d-2l47b\" (UID: \"c3b05e88-5d5b-4d03-b28d-832f22e3c868\") " pod="e2e-models-e2e-worker-w3-83755a/distinct-2-w3-83755a-kserve-6cf66b995d-2l47b" Nov 03 00:53:04.866547 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:53:04.866469 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c3b05e88-5d5b-4d03-b28d-832f22e3c868-dshm\") pod \"distinct-2-w3-83755a-kserve-6cf66b995d-2l47b\" (UID: \"c3b05e88-5d5b-4d03-b28d-832f22e3c868\") " pod="e2e-models-e2e-worker-w3-83755a/distinct-2-w3-83755a-kserve-6cf66b995d-2l47b" Nov 03 00:53:05.057994 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:53:05.039891 4939 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-83755a/distinct-2-w3-83755a-kserve-6cf66b995d-2l47b" Nov 03 00:53:05.119987 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:53:05.115871 4939 patch_prober.go:29] interesting pod/e2e-worker-w3-83755a-openshift-default-c86f9565c-qlgb7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.71:15021/healthz/ready\": dial tcp 10.132.0.71:15021: connect: connection refused" start-of-body= Nov 03 00:53:05.119987 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:53:05.115957 4939 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-83755a-openshift-default-c86f9565c-qlgb7" podUID="ba28e265-fab5-43ba-94b2-dc5b7dd5029f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.71:15021/healthz/ready\": dial tcp 10.132.0.71:15021: connect: connection refused" Nov 03 00:53:05.220530 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:53:05.195224 4939 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-88bbf7/simulator-w1-88bbf7-kserve-5fbb669f8f-42hgd" podUID="83b38883-2e7a-444d-aa68-52be4a25ae66" containerName="main" probeResult="failure" output="Get \"https://10.132.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 00:53:05.979973 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:53:05.944107 4939 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-2f9a3a/simulator-w0-2f9a3a-kserve-9596bbb8f-2trgz" podUID="207e2207-fb6c-4076-95df-72e95671c261" containerName="main" probeResult="failure" output="Get \"https://10.132.0.61:8000/health\": context deadline exceeded" Nov 03 00:53:05.979973 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:53:05.947663 4939 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-650be0/e2e-conflict-a-45480c9d-kserve-5c75dbc7bb-tjrmz" podUID="e6e864bc-d2bc-4c9f-81f8-0bed22c00883" containerName="main" probeResult="failure" output="Get \"https://10.132.0.68:8000/ready\": context deadline exceeded" Nov 03 00:53:06.096990 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:53:06.090275 4939 patch_prober.go:29] interesting pod/e2e-worker-w3-83755a-openshift-default-c86f9565c-qlgb7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.71:15021/healthz/ready\": dial tcp 10.132.0.71:15021: connect: connection refused" start-of-body= Nov 03 00:53:06.096990 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:53:06.090369 4939 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-83755a-openshift-default-c86f9565c-qlgb7" podUID="ba28e265-fab5-43ba-94b2-dc5b7dd5029f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.71:15021/healthz/ready\": dial tcp 10.132.0.71:15021: connect: connection refused" Nov 03 00:53:06.158739 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:53:06.157124 4939 patch_prober.go:29] interesting pod/e2e-worker-w4-650be0-openshift-default-5c7c5fb7d9-vj4qt container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.67:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 00:53:06.158739 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:53:06.157199 4939 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w4-650be0-openshift-default-5c7c5fb7d9-vj4qt" podUID="5b6c2a38-1b62-41a5-803f-3111bd896857" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.67:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 00:53:06.190935 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:53:06.190828 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 00:53:06.207744 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:53:06.207300 4939 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-88bbf7/simulator-w1-88bbf7-kserve-5fbb669f8f-42hgd" podUID="83b38883-2e7a-444d-aa68-52be4a25ae66" containerName="main" probeResult="failure" output="Get \"https://10.132.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 00:53:06.210414 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:53:06.208038 4939 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-88bbf7/simulator-w1-88bbf7-kserve-5fbb669f8f-42hgd" Nov 03 00:53:06.366992 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:53:06.330284 4939 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-f7680d/premium-w2-f7680d-kserve-76565b8cf9-hqbbt" podUID="4712e708-18a2-4a13-a12c-a5669397b2c3" containerName="main" probeResult="failure" output="Get \"https://10.132.0.63:8000/ready\": context deadline exceeded" Nov 03 00:53:06.366992 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:53:06.348462 4939 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-88bbf7/simulator-w1-88bbf7-kserve-5fbb669f8f-42hgd" Nov 03 00:53:06.425930 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:53:06.407247 4939 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w3-83755a/distinct-2-w3-83755a-kserve-6cf66b995d-2l47b"] Nov 03 00:53:07.102520 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:53:07.102398 4939 patch_prober.go:29] interesting pod/e2e-worker-w3-83755a-openshift-default-c86f9565c-qlgb7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.71:15021/healthz/ready\": dial tcp 10.132.0.71:15021: connect: connection refused" start-of-body= Nov 03 00:53:07.103001 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:53:07.102874 4939 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-83755a-openshift-default-c86f9565c-qlgb7" podUID="ba28e265-fab5-43ba-94b2-dc5b7dd5029f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.71:15021/healthz/ready\": dial tcp 10.132.0.71:15021: connect: connection refused" Nov 03 00:53:07.103831 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:53:07.103810 4939 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w2-f7680d/distinct-w2-f7680d-kserve-756f874b98-qnpv8" Nov 03 00:53:07.210010 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:53:07.190969 4939 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-2f9a3a/simulator-w0-2f9a3a-kserve-9596bbb8f-2trgz" podUID="207e2207-fb6c-4076-95df-72e95671c261" containerName="main" probeResult="failure" output="Get \"https://10.132.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 00:53:07.333942 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:53:07.332008 4939 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-f7680d/premium-w2-f7680d-kserve-76565b8cf9-hqbbt" podUID="4712e708-18a2-4a13-a12c-a5669397b2c3" containerName="main" probeResult="failure" output="Get \"https://10.132.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 00:53:07.333942 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:53:07.332713 4939 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-f7680d/premium-w2-f7680d-kserve-76565b8cf9-hqbbt" Nov 03 00:53:07.399097 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:53:07.399052 4939 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-88bbf7/simulator-w1-88bbf7-kserve-5fbb669f8f-42hgd" podUID="83b38883-2e7a-444d-aa68-52be4a25ae66" containerName="main" probeResult="failure" output="Get \"https://10.132.0.62:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Nov 03 00:53:07.618933 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:53:07.598085 4939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-83755a/distinct-2-w3-83755a-kserve-6cf66b995d-2l47b" event={"ID":"c3b05e88-5d5b-4d03-b28d-832f22e3c868","Type":"ContainerStarted","Data":"d8d6c22ddb44f0dbd29527555b4ca12f953f6492cd1a9ce2ab50fe7e33f1ad03"} Nov 03 00:53:07.852933 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:53:07.852607 4939 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-f7680d/premium-w2-f7680d-kserve-76565b8cf9-hqbbt" Nov 03 00:53:07.954983 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:53:07.954311 4939 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-650be0/e2e-conflict-a-45480c9d-kserve-5c75dbc7bb-tjrmz" podUID="e6e864bc-d2bc-4c9f-81f8-0bed22c00883" containerName="main" probeResult="failure" output="Get \"https://10.132.0.68:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 00:53:08.039064 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:53:08.028524 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 00:53:08.092725 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:53:08.092590 4939 prober.go:122] "Probe failed" probeType="Startup" pod="e2e-models-e2e-worker-w2-f7680d/distinct-w2-f7680d-kserve-756f874b98-qnpv8" podUID="836753fc-f9ef-4f61-b3fe-e5b92b7ee27b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.70:8000/health\": context deadline exceeded" Nov 03 00:53:08.101934 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:53:08.097407 4939 patch_prober.go:29] interesting pod/e2e-worker-w3-83755a-openshift-default-c86f9565c-qlgb7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.71:15021/healthz/ready\": dial tcp 10.132.0.71:15021: connect: connection refused" start-of-body= Nov 03 00:53:08.102219 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:53:08.102180 4939 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-83755a-openshift-default-c86f9565c-qlgb7" podUID="ba28e265-fab5-43ba-94b2-dc5b7dd5029f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.71:15021/healthz/ready\": dial tcp 10.132.0.71:15021: connect: connection refused" Nov 03 00:53:08.198959 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:53:08.192043 4939 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-88bbf7/simulator-w1-88bbf7-kserve-5fbb669f8f-42hgd" podUID="83b38883-2e7a-444d-aa68-52be4a25ae66" containerName="main" probeResult="failure" output="Get \"https://10.132.0.62:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 00:53:08.243730 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:53:08.243523 4939 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-2f9a3a/simulator-w0-2f9a3a-kserve-9596bbb8f-2trgz" podUID="207e2207-fb6c-4076-95df-72e95671c261" containerName="main" probeResult="failure" output="Get \"https://10.132.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 00:53:08.244897 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:53:08.244861 4939 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-2f9a3a/simulator-w0-2f9a3a-kserve-9596bbb8f-2trgz" Nov 03 00:53:08.418223 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:53:08.418162 4939 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-88bbf7/simulator-w1-88bbf7-kserve-5fbb669f8f-42hgd" podUID="83b38883-2e7a-444d-aa68-52be4a25ae66" containerName="main" probeResult="failure" output="Get \"https://10.132.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 00:53:08.419168 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:53:08.419143 4939 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-88bbf7/simulator-w1-88bbf7-kserve-5fbb669f8f-42hgd" Nov 03 00:53:08.432793 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:53:08.432767 4939 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-2f9a3a/simulator-w0-2f9a3a-kserve-9596bbb8f-2trgz" Nov 03 00:53:08.670981 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:53:08.582242 4939 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-88bbf7/simulator-w1-88bbf7-kserve-5fbb669f8f-42hgd" Nov 03 00:53:08.880246 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:53:08.880090 4939 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-f7680d/premium-w2-f7680d-kserve-76565b8cf9-hqbbt" podUID="4712e708-18a2-4a13-a12c-a5669397b2c3" containerName="main" probeResult="failure" output="Get \"https://10.132.0.63:8000/ready\": context deadline exceeded" Nov 03 00:53:08.976009 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:53:08.974939 4939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-83755a/distinct-2-w3-83755a-kserve-6cf66b995d-2l47b" event={"ID":"c3b05e88-5d5b-4d03-b28d-832f22e3c868","Type":"ContainerStarted","Data":"bdbe879f0d2fc0f251c3802a5e1e8b6dab2a240c43fe6064b6c071717ef02875"} Nov 03 00:53:08.999965 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:53:08.986419 4939 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-596dc9545f-pcw82 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.42:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 00:53:08.999965 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:53:08.986489 4939 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-pcw82" podUID="48186e38-73f5-47c6-a90a-d82afeac4a52" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.42:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 00:53:09.042039 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:53:09.041980 4939 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w3-83755a/distinct-2-w3-83755a-kserve-6cf66b995d-2l47b" podStartSLOduration=5.041963115 podStartE2EDuration="5.041963115s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:53:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:53:09.041295123 +0000 UTC m=+737.305630073" watchObservedRunningTime="2026-11-03 00:53:09.041963115 +0000 UTC m=+737.306298059" Nov 03 00:53:09.080935 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:53:09.080058 4939 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-83755a/distinct-2-w3-83755a-kserve-6cf66b995d-2l47b" Nov 03 00:53:09.081937 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:53:09.081608 4939 patch_prober.go:29] interesting pod/e2e-worker-w3-83755a-openshift-default-c86f9565c-qlgb7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.71:15021/healthz/ready\": dial tcp 10.132.0.71:15021: connect: connection refused" start-of-body= Nov 03 00:53:09.081937 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:53:09.081663 4939 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-83755a-openshift-default-c86f9565c-qlgb7" podUID="ba28e265-fab5-43ba-94b2-dc5b7dd5029f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.71:15021/healthz/ready\": dial tcp 10.132.0.71:15021: connect: connection refused" Nov 03 00:53:09.532791 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:53:09.532755 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 00:53:10.068286 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:53:10.068247 4939 patch_prober.go:29] interesting pod/e2e-worker-w3-83755a-openshift-default-c86f9565c-qlgb7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.71:15021/healthz/ready\": dial tcp 10.132.0.71:15021: connect: connection refused" start-of-body= Nov 03 00:53:10.068470 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:53:10.068303 4939 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-83755a-openshift-default-c86f9565c-qlgb7" podUID="ba28e265-fab5-43ba-94b2-dc5b7dd5029f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.71:15021/healthz/ready\": dial tcp 10.132.0.71:15021: connect: connection refused" Nov 03 00:53:10.283709 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:53:10.283676 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 00:53:10.894683 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:53:10.894649 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 00:53:11.073245 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:53:11.073215 4939 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-83755a-openshift-default-c86f9565c-qlgb7" Nov 03 00:53:11.073628 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:53:11.073608 4939 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-83755a-openshift-default-c86f9565c-qlgb7" Nov 03 00:53:11.074432 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:53:11.074414 4939 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-83755a-openshift-default-c86f9565c-qlgb7" Nov 03 00:53:11.520696 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:53:11.520657 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 00:53:11.710972 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:53:11.710903 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 00:53:12.388813 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:53:12.388770 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 00:53:13.155244 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:53:13.155206 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 00:53:13.788824 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:53:13.788785 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 00:53:14.479902 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:53:14.479870 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 00:53:15.075502 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:53:15.075470 4939 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w3-83755a/distinct-2-w3-83755a-kserve-6cf66b995d-2l47b" Nov 03 00:53:15.088387 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:53:15.088360 4939 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w3-83755a/distinct-2-w3-83755a-kserve-6cf66b995d-2l47b" Nov 03 00:53:15.102522 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:53:15.102494 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 00:53:15.228709 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:53:15.228674 4939 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-f7680d-openshift-default-c4488d5c9-5nv8c"] Nov 03 00:53:15.232858 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:53:15.232835 4939 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-f7680d-openshift-default-c4488d5c9-5nv8c" Nov 03 00:53:15.246102 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:53:15.246079 4939 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-f7680d-openshift-default-c4488d5c9-5nv8c"] Nov 03 00:53:15.248626 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:53:15.248602 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/38dfb6cb-588c-4ca9-916d-0b3fdc7910bb-workload-socket\") pod \"e2e-worker-w2-f7680d-openshift-default-c4488d5c9-5nv8c\" (UID: \"38dfb6cb-588c-4ca9-916d-0b3fdc7910bb\") " pod="openshift-ingress/e2e-worker-w2-f7680d-openshift-default-c4488d5c9-5nv8c" Nov 03 00:53:15.248761 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:53:15.248635 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/38dfb6cb-588c-4ca9-916d-0b3fdc7910bb-credential-socket\") pod \"e2e-worker-w2-f7680d-openshift-default-c4488d5c9-5nv8c\" (UID: \"38dfb6cb-588c-4ca9-916d-0b3fdc7910bb\") " pod="openshift-ingress/e2e-worker-w2-f7680d-openshift-default-c4488d5c9-5nv8c" Nov 03 00:53:15.248761 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:53:15.248667 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/38dfb6cb-588c-4ca9-916d-0b3fdc7910bb-istio-envoy\") pod \"e2e-worker-w2-f7680d-openshift-default-c4488d5c9-5nv8c\" (UID: \"38dfb6cb-588c-4ca9-916d-0b3fdc7910bb\") " pod="openshift-ingress/e2e-worker-w2-f7680d-openshift-default-c4488d5c9-5nv8c" Nov 03 00:53:15.248877 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:53:15.248813 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/38dfb6cb-588c-4ca9-916d-0b3fdc7910bb-istio-data\") pod \"e2e-worker-w2-f7680d-openshift-default-c4488d5c9-5nv8c\" (UID: \"38dfb6cb-588c-4ca9-916d-0b3fdc7910bb\") " pod="openshift-ingress/e2e-worker-w2-f7680d-openshift-default-c4488d5c9-5nv8c" Nov 03 00:53:15.248877 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:53:15.248853 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/38dfb6cb-588c-4ca9-916d-0b3fdc7910bb-istio-podinfo\") pod \"e2e-worker-w2-f7680d-openshift-default-c4488d5c9-5nv8c\" (UID: \"38dfb6cb-588c-4ca9-916d-0b3fdc7910bb\") " pod="openshift-ingress/e2e-worker-w2-f7680d-openshift-default-c4488d5c9-5nv8c" Nov 03 00:53:15.249050 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:53:15.249024 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/38dfb6cb-588c-4ca9-916d-0b3fdc7910bb-istio-token\") pod \"e2e-worker-w2-f7680d-openshift-default-c4488d5c9-5nv8c\" (UID: \"38dfb6cb-588c-4ca9-916d-0b3fdc7910bb\") " pod="openshift-ingress/e2e-worker-w2-f7680d-openshift-default-c4488d5c9-5nv8c" Nov 03 00:53:15.249178 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:53:15.249161 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/38dfb6cb-588c-4ca9-916d-0b3fdc7910bb-istiod-ca-cert\") pod \"e2e-worker-w2-f7680d-openshift-default-c4488d5c9-5nv8c\" (UID: \"38dfb6cb-588c-4ca9-916d-0b3fdc7910bb\") " pod="openshift-ingress/e2e-worker-w2-f7680d-openshift-default-c4488d5c9-5nv8c" Nov 03 00:53:15.249231 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:53:15.249207 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/38dfb6cb-588c-4ca9-916d-0b3fdc7910bb-workload-certs\") pod \"e2e-worker-w2-f7680d-openshift-default-c4488d5c9-5nv8c\" (UID: \"38dfb6cb-588c-4ca9-916d-0b3fdc7910bb\") " pod="openshift-ingress/e2e-worker-w2-f7680d-openshift-default-c4488d5c9-5nv8c" Nov 03 00:53:15.249281 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:53:15.249232 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-75rh4\" (UniqueName: \"kubernetes.io/projected/38dfb6cb-588c-4ca9-916d-0b3fdc7910bb-kube-api-access-75rh4\") pod \"e2e-worker-w2-f7680d-openshift-default-c4488d5c9-5nv8c\" (UID: \"38dfb6cb-588c-4ca9-916d-0b3fdc7910bb\") " pod="openshift-ingress/e2e-worker-w2-f7680d-openshift-default-c4488d5c9-5nv8c" Nov 03 00:53:15.313632 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:53:15.313598 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 00:53:15.350225 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:53:15.350194 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/38dfb6cb-588c-4ca9-916d-0b3fdc7910bb-istio-token\") pod \"e2e-worker-w2-f7680d-openshift-default-c4488d5c9-5nv8c\" (UID: \"38dfb6cb-588c-4ca9-916d-0b3fdc7910bb\") " pod="openshift-ingress/e2e-worker-w2-f7680d-openshift-default-c4488d5c9-5nv8c" Nov 03 00:53:15.350382 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:53:15.350268 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/38dfb6cb-588c-4ca9-916d-0b3fdc7910bb-istiod-ca-cert\") pod \"e2e-worker-w2-f7680d-openshift-default-c4488d5c9-5nv8c\" (UID: \"38dfb6cb-588c-4ca9-916d-0b3fdc7910bb\") " pod="openshift-ingress/e2e-worker-w2-f7680d-openshift-default-c4488d5c9-5nv8c" Nov 03 00:53:15.350382 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:53:15.350294 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/38dfb6cb-588c-4ca9-916d-0b3fdc7910bb-workload-certs\") pod \"e2e-worker-w2-f7680d-openshift-default-c4488d5c9-5nv8c\" (UID: \"38dfb6cb-588c-4ca9-916d-0b3fdc7910bb\") " pod="openshift-ingress/e2e-worker-w2-f7680d-openshift-default-c4488d5c9-5nv8c" Nov 03 00:53:15.350382 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:53:15.350311 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-75rh4\" (UniqueName: \"kubernetes.io/projected/38dfb6cb-588c-4ca9-916d-0b3fdc7910bb-kube-api-access-75rh4\") pod \"e2e-worker-w2-f7680d-openshift-default-c4488d5c9-5nv8c\" (UID: \"38dfb6cb-588c-4ca9-916d-0b3fdc7910bb\") " pod="openshift-ingress/e2e-worker-w2-f7680d-openshift-default-c4488d5c9-5nv8c" Nov 03 00:53:15.350382 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:53:15.350348 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/38dfb6cb-588c-4ca9-916d-0b3fdc7910bb-workload-socket\") pod \"e2e-worker-w2-f7680d-openshift-default-c4488d5c9-5nv8c\" (UID: \"38dfb6cb-588c-4ca9-916d-0b3fdc7910bb\") " pod="openshift-ingress/e2e-worker-w2-f7680d-openshift-default-c4488d5c9-5nv8c" Nov 03 00:53:15.350382 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:53:15.350375 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/38dfb6cb-588c-4ca9-916d-0b3fdc7910bb-credential-socket\") pod \"e2e-worker-w2-f7680d-openshift-default-c4488d5c9-5nv8c\" (UID: \"38dfb6cb-588c-4ca9-916d-0b3fdc7910bb\") " pod="openshift-ingress/e2e-worker-w2-f7680d-openshift-default-c4488d5c9-5nv8c" Nov 03 00:53:15.350644 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:53:15.350409 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/38dfb6cb-588c-4ca9-916d-0b3fdc7910bb-istio-envoy\") pod \"e2e-worker-w2-f7680d-openshift-default-c4488d5c9-5nv8c\" (UID: \"38dfb6cb-588c-4ca9-916d-0b3fdc7910bb\") " pod="openshift-ingress/e2e-worker-w2-f7680d-openshift-default-c4488d5c9-5nv8c" Nov 03 00:53:15.350644 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:53:15.350461 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/38dfb6cb-588c-4ca9-916d-0b3fdc7910bb-istio-data\") pod \"e2e-worker-w2-f7680d-openshift-default-c4488d5c9-5nv8c\" (UID: \"38dfb6cb-588c-4ca9-916d-0b3fdc7910bb\") " pod="openshift-ingress/e2e-worker-w2-f7680d-openshift-default-c4488d5c9-5nv8c" Nov 03 00:53:15.350644 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:53:15.350530 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/38dfb6cb-588c-4ca9-916d-0b3fdc7910bb-istio-podinfo\") pod \"e2e-worker-w2-f7680d-openshift-default-c4488d5c9-5nv8c\" (UID: \"38dfb6cb-588c-4ca9-916d-0b3fdc7910bb\") " pod="openshift-ingress/e2e-worker-w2-f7680d-openshift-default-c4488d5c9-5nv8c" Nov 03 00:53:15.350964 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:53:15.350942 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/38dfb6cb-588c-4ca9-916d-0b3fdc7910bb-istio-data\") pod \"e2e-worker-w2-f7680d-openshift-default-c4488d5c9-5nv8c\" (UID: \"38dfb6cb-588c-4ca9-916d-0b3fdc7910bb\") " pod="openshift-ingress/e2e-worker-w2-f7680d-openshift-default-c4488d5c9-5nv8c" Nov 03 00:53:15.350964 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:53:15.350954 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/38dfb6cb-588c-4ca9-916d-0b3fdc7910bb-workload-socket\") pod \"e2e-worker-w2-f7680d-openshift-default-c4488d5c9-5nv8c\" (UID: \"38dfb6cb-588c-4ca9-916d-0b3fdc7910bb\") " pod="openshift-ingress/e2e-worker-w2-f7680d-openshift-default-c4488d5c9-5nv8c" Nov 03 00:53:15.351110 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:53:15.351013 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/38dfb6cb-588c-4ca9-916d-0b3fdc7910bb-credential-socket\") pod \"e2e-worker-w2-f7680d-openshift-default-c4488d5c9-5nv8c\" (UID: \"38dfb6cb-588c-4ca9-916d-0b3fdc7910bb\") " pod="openshift-ingress/e2e-worker-w2-f7680d-openshift-default-c4488d5c9-5nv8c" Nov 03 00:53:15.351110 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:53:15.351046 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/38dfb6cb-588c-4ca9-916d-0b3fdc7910bb-workload-certs\") pod \"e2e-worker-w2-f7680d-openshift-default-c4488d5c9-5nv8c\" (UID: \"38dfb6cb-588c-4ca9-916d-0b3fdc7910bb\") " pod="openshift-ingress/e2e-worker-w2-f7680d-openshift-default-c4488d5c9-5nv8c" Nov 03 00:53:15.351257 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:53:15.351238 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/38dfb6cb-588c-4ca9-916d-0b3fdc7910bb-istiod-ca-cert\") pod \"e2e-worker-w2-f7680d-openshift-default-c4488d5c9-5nv8c\" (UID: \"38dfb6cb-588c-4ca9-916d-0b3fdc7910bb\") " pod="openshift-ingress/e2e-worker-w2-f7680d-openshift-default-c4488d5c9-5nv8c" Nov 03 00:53:15.353212 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:53:15.353192 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/38dfb6cb-588c-4ca9-916d-0b3fdc7910bb-istio-podinfo\") pod \"e2e-worker-w2-f7680d-openshift-default-c4488d5c9-5nv8c\" (UID: \"38dfb6cb-588c-4ca9-916d-0b3fdc7910bb\") " pod="openshift-ingress/e2e-worker-w2-f7680d-openshift-default-c4488d5c9-5nv8c" Nov 03 00:53:15.354004 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:53:15.353981 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/38dfb6cb-588c-4ca9-916d-0b3fdc7910bb-istio-envoy\") pod \"e2e-worker-w2-f7680d-openshift-default-c4488d5c9-5nv8c\" (UID: \"38dfb6cb-588c-4ca9-916d-0b3fdc7910bb\") " pod="openshift-ingress/e2e-worker-w2-f7680d-openshift-default-c4488d5c9-5nv8c" Nov 03 00:53:15.358374 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:53:15.358350 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-75rh4\" (UniqueName: \"kubernetes.io/projected/38dfb6cb-588c-4ca9-916d-0b3fdc7910bb-kube-api-access-75rh4\") pod \"e2e-worker-w2-f7680d-openshift-default-c4488d5c9-5nv8c\" (UID: \"38dfb6cb-588c-4ca9-916d-0b3fdc7910bb\") " pod="openshift-ingress/e2e-worker-w2-f7680d-openshift-default-c4488d5c9-5nv8c" Nov 03 00:53:15.358759 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:53:15.358741 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/38dfb6cb-588c-4ca9-916d-0b3fdc7910bb-istio-token\") pod \"e2e-worker-w2-f7680d-openshift-default-c4488d5c9-5nv8c\" (UID: \"38dfb6cb-588c-4ca9-916d-0b3fdc7910bb\") " pod="openshift-ingress/e2e-worker-w2-f7680d-openshift-default-c4488d5c9-5nv8c" Nov 03 00:53:15.546298 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:53:15.546265 4939 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-f7680d-openshift-default-c4488d5c9-5nv8c" Nov 03 00:53:15.842600 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:53:15.842559 4939 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-f7680d-openshift-default-c4488d5c9-5nv8c"] Nov 03 00:53:15.847047 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:53:15.846998 4939 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 00:53:15.847170 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:53:15.847142 4939 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 00:53:15.847273 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:53:15.847249 4939 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 00:53:15.887003 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:53:15.886975 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 00:53:16.000866 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:53:16.000836 4939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-f7680d-openshift-default-c4488d5c9-5nv8c" event={"ID":"38dfb6cb-588c-4ca9-916d-0b3fdc7910bb","Type":"ContainerStarted","Data":"a7bbc7d481c2d6a20649e5356e8ce1f69c26ed3fe3ca7cc1670745a892cbb3de"} Nov 03 00:53:16.001014 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:53:16.000888 4939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-f7680d-openshift-default-c4488d5c9-5nv8c" event={"ID":"38dfb6cb-588c-4ca9-916d-0b3fdc7910bb","Type":"ContainerStarted","Data":"62c74d342840303f44f0a6db2e3022c6017252e17c173168ea4c6473cc3a24c1"} Nov 03 00:53:16.012847 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:53:16.012815 4939 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-83755a/distinct-2-w3-83755a-kserve-6cf66b995d-2l47b" Nov 03 00:53:16.020799 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:53:16.020740 4939 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-f7680d-openshift-default-c4488d5c9-5nv8c" podStartSLOduration=1.020727282 podStartE2EDuration="1.020727282s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:53:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:53:16.020210794 +0000 UTC m=+744.284545735" watchObservedRunningTime="2026-11-03 00:53:16.020727282 +0000 UTC m=+744.285062227" Nov 03 00:53:16.560719 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:53:16.559286 4939 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-f7680d-openshift-default-c4488d5c9-5nv8c" Nov 03 00:53:16.565145 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:53:16.565108 4939 patch_prober.go:29] interesting pod/e2e-worker-w2-f7680d-openshift-default-c4488d5c9-5nv8c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.73:15021/healthz/ready\": dial tcp 10.132.0.73:15021: connect: connection refused" start-of-body= Nov 03 00:53:16.565636 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:53:16.565594 4939 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-f7680d-openshift-default-c4488d5c9-5nv8c" podUID="38dfb6cb-588c-4ca9-916d-0b3fdc7910bb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.73:15021/healthz/ready\": dial tcp 10.132.0.73:15021: connect: connection refused" Nov 03 00:53:16.610727 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:53:16.610693 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 00:53:17.027646 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:53:17.027616 4939 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w2-f7680d/distinct-w2-f7680d-kserve-756f874b98-qnpv8" Nov 03 00:53:17.039199 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:53:17.039172 4939 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-f7680d/distinct-w2-f7680d-kserve-756f874b98-qnpv8" Nov 03 00:53:17.397113 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:53:17.397080 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 00:53:17.551341 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:53:17.551304 4939 patch_prober.go:29] interesting pod/e2e-worker-w2-f7680d-openshift-default-c4488d5c9-5nv8c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.73:15021/healthz/ready\": dial tcp 10.132.0.73:15021: connect: connection refused" start-of-body= Nov 03 00:53:17.551615 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:53:17.551587 4939 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-f7680d-openshift-default-c4488d5c9-5nv8c" podUID="38dfb6cb-588c-4ca9-916d-0b3fdc7910bb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.73:15021/healthz/ready\": dial tcp 10.132.0.73:15021: connect: connection refused" Nov 03 00:53:18.031376 ip-10-0-140-107 kubenswrapper[4939]: E1103 00:53:18.031338 4939 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-sthjh" containerName="extractor" Nov 03 00:53:18.127684 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:53:18.127650 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 00:53:18.571106 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:53:18.571061 4939 patch_prober.go:29] interesting pod/e2e-worker-w2-f7680d-openshift-default-c4488d5c9-5nv8c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.73:15021/healthz/ready\": dial tcp 10.132.0.73:15021: connect: connection refused" start-of-body= Nov 03 00:53:18.571299 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:53:18.571133 4939 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-f7680d-openshift-default-c4488d5c9-5nv8c" podUID="38dfb6cb-588c-4ca9-916d-0b3fdc7910bb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.73:15021/healthz/ready\": dial tcp 10.132.0.73:15021: connect: connection refused" Nov 03 00:53:18.892974 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:53:18.892897 4939 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-650be0/e2e-conflict-a-45480c9d-kserve-5c75dbc7bb-tjrmz"] Nov 03 00:53:18.924947 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:53:18.924898 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 00:53:19.199308 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:53:19.199215 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 00:53:19.549986 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:53:19.549360 4939 patch_prober.go:29] interesting pod/e2e-worker-w2-f7680d-openshift-default-c4488d5c9-5nv8c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.73:15021/healthz/ready\": dial tcp 10.132.0.73:15021: connect: connection refused" start-of-body= Nov 03 00:53:19.549986 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:53:19.549442 4939 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-f7680d-openshift-default-c4488d5c9-5nv8c" podUID="38dfb6cb-588c-4ca9-916d-0b3fdc7910bb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.73:15021/healthz/ready\": dial tcp 10.132.0.73:15021: connect: connection refused" Nov 03 00:53:20.184596 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:53:20.184492 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 00:53:20.551567 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:53:20.551474 4939 patch_prober.go:29] interesting pod/e2e-worker-w2-f7680d-openshift-default-c4488d5c9-5nv8c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.73:15021/healthz/ready\": dial tcp 10.132.0.73:15021: connect: connection refused" start-of-body= Nov 03 00:53:20.552181 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:53:20.552147 4939 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-f7680d-openshift-default-c4488d5c9-5nv8c" podUID="38dfb6cb-588c-4ca9-916d-0b3fdc7910bb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.73:15021/healthz/ready\": dial tcp 10.132.0.73:15021: connect: connection refused" Nov 03 00:53:21.239202 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:53:21.239166 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 00:53:21.550389 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:53:21.550293 4939 patch_prober.go:29] interesting pod/e2e-worker-w2-f7680d-openshift-default-c4488d5c9-5nv8c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.73:15021/healthz/ready\": dial tcp 10.132.0.73:15021: connect: connection refused" start-of-body= Nov 03 00:53:21.555044 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:53:21.555002 4939 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-f7680d-openshift-default-c4488d5c9-5nv8c" podUID="38dfb6cb-588c-4ca9-916d-0b3fdc7910bb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.73:15021/healthz/ready\": dial tcp 10.132.0.73:15021: connect: connection refused" Nov 03 00:53:21.981776 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:53:21.981740 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 00:53:22.547153 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:53:22.547117 4939 patch_prober.go:29] interesting pod/e2e-worker-w2-f7680d-openshift-default-c4488d5c9-5nv8c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.73:15021/healthz/ready\": dial tcp 10.132.0.73:15021: connect: connection refused" start-of-body= Nov 03 00:53:22.547341 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:53:22.547173 4939 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-f7680d-openshift-default-c4488d5c9-5nv8c" podUID="38dfb6cb-588c-4ca9-916d-0b3fdc7910bb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.73:15021/healthz/ready\": dial tcp 10.132.0.73:15021: connect: connection refused" Nov 03 00:53:22.675802 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:53:22.675763 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 00:53:23.374445 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:53:23.374404 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 00:53:23.550984 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:53:23.550950 4939 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-f7680d-openshift-default-c4488d5c9-5nv8c" Nov 03 00:53:23.551456 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:53:23.551429 4939 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-f7680d-openshift-default-c4488d5c9-5nv8c" Nov 03 00:53:23.552027 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:53:23.552007 4939 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-f7680d-openshift-default-c4488d5c9-5nv8c" Nov 03 00:53:24.089057 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:53:24.089022 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 00:53:24.777404 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:53:24.777372 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 00:53:25.438524 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:53:25.438486 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 00:53:26.167746 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:53:26.167705 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 00:53:26.839201 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:53:26.839164 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 00:53:26.962502 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:53:26.962451 4939 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["models-as-a-service/payload-processing-external-model-6889df475b-r2899"] Nov 03 00:53:26.979582 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:53:26.979548 4939 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-6889df475b-r2899" Nov 03 00:53:27.014646 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:53:27.014577 4939 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"models-as-a-service\"/\"agc-cred-a60c08844a3e\"" Nov 03 00:53:27.014980 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:53:27.014942 4939 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"models-as-a-service\"/\"payload-processing-external-model-dockercfg-r9jtj\"" Nov 03 00:53:27.025950 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:53:27.023051 4939 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"models-as-a-service\"/\"agc-cred-48d9fe8f921f\"" Nov 03 00:53:27.025950 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:53:27.023158 4939 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"models-as-a-service\"/\"openshift-service-ca.crt\"" Nov 03 00:53:27.025950 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:53:27.023205 4939 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"models-as-a-service\"/\"kube-root-ca.crt\"" Nov 03 00:53:27.025950 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:53:27.023294 4939 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"models-as-a-service\"/\"routing-overlay\"" Nov 03 00:53:27.025950 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:53:27.023351 4939 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"models-as-a-service\"/\"payload-processing-external-model-plugins\"" Nov 03 00:53:27.032030 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:53:27.027234 4939 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["models-as-a-service/payload-processing-external-model-6889df475b-r2899"] Nov 03 00:53:27.088072 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:53:27.082773 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 00:53:27.134773 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:53:27.134693 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/b2c06545-43b5-40a2-becc-cb8754bc68e2-routing-overlay-volume\") pod \"payload-processing-external-model-6889df475b-r2899\" (UID: \"b2c06545-43b5-40a2-becc-cb8754bc68e2\") " pod="models-as-a-service/payload-processing-external-model-6889df475b-r2899" Nov 03 00:53:27.134773 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:53:27.134762 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/b2c06545-43b5-40a2-becc-cb8754bc68e2-provider-credentials\") pod \"payload-processing-external-model-6889df475b-r2899\" (UID: \"b2c06545-43b5-40a2-becc-cb8754bc68e2\") " pod="models-as-a-service/payload-processing-external-model-6889df475b-r2899" Nov 03 00:53:27.134978 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:53:27.134949 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wvr8h\" (UniqueName: \"kubernetes.io/projected/b2c06545-43b5-40a2-becc-cb8754bc68e2-kube-api-access-wvr8h\") pod \"payload-processing-external-model-6889df475b-r2899\" (UID: \"b2c06545-43b5-40a2-becc-cb8754bc68e2\") " pod="models-as-a-service/payload-processing-external-model-6889df475b-r2899" Nov 03 00:53:27.135030 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:53:27.135012 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b2c06545-43b5-40a2-becc-cb8754bc68e2-plugins-config-volume\") pod \"payload-processing-external-model-6889df475b-r2899\" (UID: \"b2c06545-43b5-40a2-becc-cb8754bc68e2\") " pod="models-as-a-service/payload-processing-external-model-6889df475b-r2899" Nov 03 00:53:27.217639 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:53:27.217589 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 00:53:27.237929 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:53:27.237874 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/b2c06545-43b5-40a2-becc-cb8754bc68e2-routing-overlay-volume\") pod \"payload-processing-external-model-6889df475b-r2899\" (UID: \"b2c06545-43b5-40a2-becc-cb8754bc68e2\") " pod="models-as-a-service/payload-processing-external-model-6889df475b-r2899" Nov 03 00:53:27.238301 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:53:27.238273 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/b2c06545-43b5-40a2-becc-cb8754bc68e2-provider-credentials\") pod \"payload-processing-external-model-6889df475b-r2899\" (UID: \"b2c06545-43b5-40a2-becc-cb8754bc68e2\") " pod="models-as-a-service/payload-processing-external-model-6889df475b-r2899" Nov 03 00:53:27.238600 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:53:27.238583 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wvr8h\" (UniqueName: \"kubernetes.io/projected/b2c06545-43b5-40a2-becc-cb8754bc68e2-kube-api-access-wvr8h\") pod \"payload-processing-external-model-6889df475b-r2899\" (UID: \"b2c06545-43b5-40a2-becc-cb8754bc68e2\") " pod="models-as-a-service/payload-processing-external-model-6889df475b-r2899" Nov 03 00:53:27.238789 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:53:27.238774 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b2c06545-43b5-40a2-becc-cb8754bc68e2-plugins-config-volume\") pod \"payload-processing-external-model-6889df475b-r2899\" (UID: \"b2c06545-43b5-40a2-becc-cb8754bc68e2\") " pod="models-as-a-service/payload-processing-external-model-6889df475b-r2899" Nov 03 00:53:27.251237 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:53:27.240005 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b2c06545-43b5-40a2-becc-cb8754bc68e2-plugins-config-volume\") pod \"payload-processing-external-model-6889df475b-r2899\" (UID: \"b2c06545-43b5-40a2-becc-cb8754bc68e2\") " pod="models-as-a-service/payload-processing-external-model-6889df475b-r2899" Nov 03 00:53:27.252270 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:53:27.252240 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/b2c06545-43b5-40a2-becc-cb8754bc68e2-routing-overlay-volume\") pod \"payload-processing-external-model-6889df475b-r2899\" (UID: \"b2c06545-43b5-40a2-becc-cb8754bc68e2\") " pod="models-as-a-service/payload-processing-external-model-6889df475b-r2899" Nov 03 00:53:27.257447 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:53:27.257420 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/b2c06545-43b5-40a2-becc-cb8754bc68e2-provider-credentials\") pod \"payload-processing-external-model-6889df475b-r2899\" (UID: \"b2c06545-43b5-40a2-becc-cb8754bc68e2\") " pod="models-as-a-service/payload-processing-external-model-6889df475b-r2899" Nov 03 00:53:27.270939 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:53:27.270883 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wvr8h\" (UniqueName: \"kubernetes.io/projected/b2c06545-43b5-40a2-becc-cb8754bc68e2-kube-api-access-wvr8h\") pod \"payload-processing-external-model-6889df475b-r2899\" (UID: \"b2c06545-43b5-40a2-becc-cb8754bc68e2\") " pod="models-as-a-service/payload-processing-external-model-6889df475b-r2899" Nov 03 00:53:27.320929 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:53:27.320591 4939 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-6889df475b-r2899" Nov 03 00:53:27.712598 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:53:27.712565 4939 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["models-as-a-service/payload-processing-external-model-6889df475b-r2899"] Nov 03 00:53:27.714321 ip-10-0-140-107 kubenswrapper[4939]: W1103 00:53:27.714279 4939 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb2c06545_43b5_40a2_becc_cb8754bc68e2.slice/crio-b0f4ab9f51ca12aaa02c1566c44675e9ce7329d0d51742663b2f7bcc186b6de8 WatchSource:0}: Error finding container b0f4ab9f51ca12aaa02c1566c44675e9ce7329d0d51742663b2f7bcc186b6de8: Status 404 returned error can't find the container with id b0f4ab9f51ca12aaa02c1566c44675e9ce7329d0d51742663b2f7bcc186b6de8 Nov 03 00:53:27.720167 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:53:27.720142 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 00:53:28.163250 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:53:28.163208 4939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-6889df475b-r2899" event={"ID":"b2c06545-43b5-40a2-becc-cb8754bc68e2","Type":"ContainerStarted","Data":"9e6d684ea1257d09651f30c3c98c455ca86e7bbb17e7e7ce849b04ccf269acfb"} Nov 03 00:53:28.163673 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:53:28.163272 4939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-6889df475b-r2899" event={"ID":"b2c06545-43b5-40a2-becc-cb8754bc68e2","Type":"ContainerStarted","Data":"b0f4ab9f51ca12aaa02c1566c44675e9ce7329d0d51742663b2f7bcc186b6de8"} Nov 03 00:53:28.163673 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:53:28.163302 4939 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="models-as-a-service/payload-processing-external-model-6889df475b-r2899" Nov 03 00:53:28.182042 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:53:28.181991 4939 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="models-as-a-service/payload-processing-external-model-6889df475b-r2899" podStartSLOduration=2.181975648 podStartE2EDuration="2.181975648s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:53:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:53:28.180024019 +0000 UTC m=+756.444358965" watchObservedRunningTime="2026-11-03 00:53:28.181975648 +0000 UTC m=+756.446310593" Nov 03 00:53:28.189846 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:53:28.189819 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 00:53:28.889052 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:53:28.889022 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 00:53:29.606605 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:53:29.606570 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 00:53:30.291643 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:53:30.291608 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 00:53:30.623442 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:53:30.623408 4939 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w2-f7680d/distinct-2-w2-f7680d-kserve-7d7c4564bf-cbv2c"] Nov 03 00:53:30.628517 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:53:30.628492 4939 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-f7680d/distinct-2-w2-f7680d-kserve-7d7c4564bf-cbv2c" Nov 03 00:53:30.640922 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:53:30.640865 4939 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w2-f7680d/distinct-2-w2-f7680d-kserve-7d7c4564bf-cbv2c"] Nov 03 00:53:30.642839 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:53:30.642813 4939 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w2-f7680d\"/\"distinct-2-w2-f7680d-kserve-self-signed-certs\"" Nov 03 00:53:30.684598 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:53:30.684526 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 00:53:30.716356 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:53:30.716322 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2de18086-761b-4773-b9fc-15ddc4a220fc-tmp-dir\") pod \"distinct-2-w2-f7680d-kserve-7d7c4564bf-cbv2c\" (UID: \"2de18086-761b-4773-b9fc-15ddc4a220fc\") " pod="e2e-models-e2e-worker-w2-f7680d/distinct-2-w2-f7680d-kserve-7d7c4564bf-cbv2c" Nov 03 00:53:30.716537 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:53:30.716491 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2de18086-761b-4773-b9fc-15ddc4a220fc-tls-certs\") pod \"distinct-2-w2-f7680d-kserve-7d7c4564bf-cbv2c\" (UID: \"2de18086-761b-4773-b9fc-15ddc4a220fc\") " pod="e2e-models-e2e-worker-w2-f7680d/distinct-2-w2-f7680d-kserve-7d7c4564bf-cbv2c" Nov 03 00:53:30.716682 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:53:30.716661 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zjb7h\" (UniqueName: \"kubernetes.io/projected/2de18086-761b-4773-b9fc-15ddc4a220fc-kube-api-access-zjb7h\") pod \"distinct-2-w2-f7680d-kserve-7d7c4564bf-cbv2c\" (UID: \"2de18086-761b-4773-b9fc-15ddc4a220fc\") " pod="e2e-models-e2e-worker-w2-f7680d/distinct-2-w2-f7680d-kserve-7d7c4564bf-cbv2c" Nov 03 00:53:30.716757 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:53:30.716701 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2de18086-761b-4773-b9fc-15ddc4a220fc-model-cache\") pod \"distinct-2-w2-f7680d-kserve-7d7c4564bf-cbv2c\" (UID: \"2de18086-761b-4773-b9fc-15ddc4a220fc\") " pod="e2e-models-e2e-worker-w2-f7680d/distinct-2-w2-f7680d-kserve-7d7c4564bf-cbv2c" Nov 03 00:53:30.716757 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:53:30.716737 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2de18086-761b-4773-b9fc-15ddc4a220fc-dshm\") pod \"distinct-2-w2-f7680d-kserve-7d7c4564bf-cbv2c\" (UID: \"2de18086-761b-4773-b9fc-15ddc4a220fc\") " pod="e2e-models-e2e-worker-w2-f7680d/distinct-2-w2-f7680d-kserve-7d7c4564bf-cbv2c" Nov 03 00:53:30.716868 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:53:30.716846 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2de18086-761b-4773-b9fc-15ddc4a220fc-home\") pod \"distinct-2-w2-f7680d-kserve-7d7c4564bf-cbv2c\" (UID: \"2de18086-761b-4773-b9fc-15ddc4a220fc\") " pod="e2e-models-e2e-worker-w2-f7680d/distinct-2-w2-f7680d-kserve-7d7c4564bf-cbv2c" Nov 03 00:53:30.822956 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:53:30.819766 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2de18086-761b-4773-b9fc-15ddc4a220fc-tls-certs\") pod \"distinct-2-w2-f7680d-kserve-7d7c4564bf-cbv2c\" (UID: \"2de18086-761b-4773-b9fc-15ddc4a220fc\") " pod="e2e-models-e2e-worker-w2-f7680d/distinct-2-w2-f7680d-kserve-7d7c4564bf-cbv2c" Nov 03 00:53:30.822956 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:53:30.819892 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zjb7h\" (UniqueName: \"kubernetes.io/projected/2de18086-761b-4773-b9fc-15ddc4a220fc-kube-api-access-zjb7h\") pod \"distinct-2-w2-f7680d-kserve-7d7c4564bf-cbv2c\" (UID: \"2de18086-761b-4773-b9fc-15ddc4a220fc\") " pod="e2e-models-e2e-worker-w2-f7680d/distinct-2-w2-f7680d-kserve-7d7c4564bf-cbv2c" Nov 03 00:53:30.822956 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:53:30.819944 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2de18086-761b-4773-b9fc-15ddc4a220fc-model-cache\") pod \"distinct-2-w2-f7680d-kserve-7d7c4564bf-cbv2c\" (UID: \"2de18086-761b-4773-b9fc-15ddc4a220fc\") " pod="e2e-models-e2e-worker-w2-f7680d/distinct-2-w2-f7680d-kserve-7d7c4564bf-cbv2c" Nov 03 00:53:30.822956 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:53:30.819978 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2de18086-761b-4773-b9fc-15ddc4a220fc-dshm\") pod \"distinct-2-w2-f7680d-kserve-7d7c4564bf-cbv2c\" (UID: \"2de18086-761b-4773-b9fc-15ddc4a220fc\") " pod="e2e-models-e2e-worker-w2-f7680d/distinct-2-w2-f7680d-kserve-7d7c4564bf-cbv2c" Nov 03 00:53:30.822956 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:53:30.820036 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2de18086-761b-4773-b9fc-15ddc4a220fc-home\") pod \"distinct-2-w2-f7680d-kserve-7d7c4564bf-cbv2c\" (UID: \"2de18086-761b-4773-b9fc-15ddc4a220fc\") " pod="e2e-models-e2e-worker-w2-f7680d/distinct-2-w2-f7680d-kserve-7d7c4564bf-cbv2c" Nov 03 00:53:30.822956 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:53:30.820132 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2de18086-761b-4773-b9fc-15ddc4a220fc-tmp-dir\") pod \"distinct-2-w2-f7680d-kserve-7d7c4564bf-cbv2c\" (UID: \"2de18086-761b-4773-b9fc-15ddc4a220fc\") " pod="e2e-models-e2e-worker-w2-f7680d/distinct-2-w2-f7680d-kserve-7d7c4564bf-cbv2c" Nov 03 00:53:30.822956 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:53:30.820847 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2de18086-761b-4773-b9fc-15ddc4a220fc-tmp-dir\") pod \"distinct-2-w2-f7680d-kserve-7d7c4564bf-cbv2c\" (UID: \"2de18086-761b-4773-b9fc-15ddc4a220fc\") " pod="e2e-models-e2e-worker-w2-f7680d/distinct-2-w2-f7680d-kserve-7d7c4564bf-cbv2c" Nov 03 00:53:30.822956 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:53:30.821655 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2de18086-761b-4773-b9fc-15ddc4a220fc-model-cache\") pod \"distinct-2-w2-f7680d-kserve-7d7c4564bf-cbv2c\" (UID: \"2de18086-761b-4773-b9fc-15ddc4a220fc\") " pod="e2e-models-e2e-worker-w2-f7680d/distinct-2-w2-f7680d-kserve-7d7c4564bf-cbv2c" Nov 03 00:53:30.822956 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:53:30.822637 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2de18086-761b-4773-b9fc-15ddc4a220fc-home\") pod \"distinct-2-w2-f7680d-kserve-7d7c4564bf-cbv2c\" (UID: \"2de18086-761b-4773-b9fc-15ddc4a220fc\") " pod="e2e-models-e2e-worker-w2-f7680d/distinct-2-w2-f7680d-kserve-7d7c4564bf-cbv2c" Nov 03 00:53:30.863928 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:53:30.848509 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2de18086-761b-4773-b9fc-15ddc4a220fc-dshm\") pod \"distinct-2-w2-f7680d-kserve-7d7c4564bf-cbv2c\" (UID: \"2de18086-761b-4773-b9fc-15ddc4a220fc\") " pod="e2e-models-e2e-worker-w2-f7680d/distinct-2-w2-f7680d-kserve-7d7c4564bf-cbv2c" Nov 03 00:53:30.879505 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:53:30.876372 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2de18086-761b-4773-b9fc-15ddc4a220fc-tls-certs\") pod \"distinct-2-w2-f7680d-kserve-7d7c4564bf-cbv2c\" (UID: \"2de18086-761b-4773-b9fc-15ddc4a220fc\") " pod="e2e-models-e2e-worker-w2-f7680d/distinct-2-w2-f7680d-kserve-7d7c4564bf-cbv2c" Nov 03 00:53:30.879505 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:53:30.879343 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zjb7h\" (UniqueName: \"kubernetes.io/projected/2de18086-761b-4773-b9fc-15ddc4a220fc-kube-api-access-zjb7h\") pod \"distinct-2-w2-f7680d-kserve-7d7c4564bf-cbv2c\" (UID: \"2de18086-761b-4773-b9fc-15ddc4a220fc\") " pod="e2e-models-e2e-worker-w2-f7680d/distinct-2-w2-f7680d-kserve-7d7c4564bf-cbv2c" Nov 03 00:53:30.955101 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:53:30.955070 4939 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-f7680d/distinct-2-w2-f7680d-kserve-7d7c4564bf-cbv2c" Nov 03 00:53:31.102224 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:53:31.102169 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 00:53:31.235714 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:53:31.235688 4939 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w2-f7680d/distinct-2-w2-f7680d-kserve-7d7c4564bf-cbv2c"] Nov 03 00:53:31.237617 ip-10-0-140-107 kubenswrapper[4939]: W1103 00:53:31.237593 4939 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod2de18086_761b_4773_b9fc_15ddc4a220fc.slice/crio-298e21e863aa9fdf46bfe9391a1719c66f790b930026380395fca1218f96e34d WatchSource:0}: Error finding container 298e21e863aa9fdf46bfe9391a1719c66f790b930026380395fca1218f96e34d: Status 404 returned error can't find the container with id 298e21e863aa9fdf46bfe9391a1719c66f790b930026380395fca1218f96e34d Nov 03 00:53:31.904456 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:53:31.904420 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 00:53:32.195341 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:53:32.195251 4939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-f7680d/distinct-2-w2-f7680d-kserve-7d7c4564bf-cbv2c" event={"ID":"2de18086-761b-4773-b9fc-15ddc4a220fc","Type":"ContainerStarted","Data":"e47719852a6e462cb0947c01e78ec27e6a6dfa8d40a4f01271c3a12a16113800"} Nov 03 00:53:32.195341 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:53:32.195295 4939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-f7680d/distinct-2-w2-f7680d-kserve-7d7c4564bf-cbv2c" event={"ID":"2de18086-761b-4773-b9fc-15ddc4a220fc","Type":"ContainerStarted","Data":"298e21e863aa9fdf46bfe9391a1719c66f790b930026380395fca1218f96e34d"} Nov 03 00:53:32.212456 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:53:32.212411 4939 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w2-f7680d/distinct-2-w2-f7680d-kserve-7d7c4564bf-cbv2c" podStartSLOduration=2.212391139 podStartE2EDuration="2.212391139s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:53:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:53:32.210336533 +0000 UTC m=+760.474671477" watchObservedRunningTime="2026-11-03 00:53:32.212391139 +0000 UTC m=+760.476726083" Nov 03 00:53:32.691643 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:53:32.691599 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 00:53:32.918575 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:53:32.918533 4939 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-4pw7v"] Nov 03 00:53:32.955863 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:53:32.955758 4939 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-f7680d/distinct-2-w2-f7680d-kserve-7d7c4564bf-cbv2c" Nov 03 00:53:33.394086 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:53:33.394048 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 00:53:33.963932 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:53:33.963082 4939 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w4-650be0/e2e-conflict-a-45480c9d-kserve-5c75dbc7bb-tjrmz" podUID="e6e864bc-d2bc-4c9f-81f8-0bed22c00883" containerName="main" containerID="cri-o://4ff35624f3f7a2c5f8283325b4420facbff1ccc760eba36146da5bcf048dca59" gracePeriod=45 Nov 03 00:53:34.219948 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:53:34.219261 4939 generic.go:356] "Generic (PLEG): container finished" podID="e6e864bc-d2bc-4c9f-81f8-0bed22c00883" containerID="4ff35624f3f7a2c5f8283325b4420facbff1ccc760eba36146da5bcf048dca59" exitCode=0 Nov 03 00:53:34.220214 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:53:34.220149 4939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-650be0/e2e-conflict-a-45480c9d-kserve-5c75dbc7bb-tjrmz" event={"ID":"e6e864bc-d2bc-4c9f-81f8-0bed22c00883","Type":"ContainerDied","Data":"4ff35624f3f7a2c5f8283325b4420facbff1ccc760eba36146da5bcf048dca59"} Nov 03 00:53:34.271576 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:53:34.271546 4939 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-650be0/e2e-conflict-a-45480c9d-kserve-5c75dbc7bb-tjrmz" Nov 03 00:53:34.300227 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:53:34.300192 4939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e6e864bc-d2bc-4c9f-81f8-0bed22c00883-dshm\" (UniqueName: \"kubernetes.io/empty-dir/e6e864bc-d2bc-4c9f-81f8-0bed22c00883-dshm\") pod \"e6e864bc-d2bc-4c9f-81f8-0bed22c00883\" (UID: \"e6e864bc-d2bc-4c9f-81f8-0bed22c00883\") " Nov 03 00:53:34.300412 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:53:34.300259 4939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e6e864bc-d2bc-4c9f-81f8-0bed22c00883-tls-certs\" (UniqueName: \"kubernetes.io/secret/e6e864bc-d2bc-4c9f-81f8-0bed22c00883-tls-certs\") pod \"e6e864bc-d2bc-4c9f-81f8-0bed22c00883\" (UID: \"e6e864bc-d2bc-4c9f-81f8-0bed22c00883\") " Nov 03 00:53:34.300412 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:53:34.300324 4939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e6e864bc-d2bc-4c9f-81f8-0bed22c00883-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e6e864bc-d2bc-4c9f-81f8-0bed22c00883-tmp-dir\") pod \"e6e864bc-d2bc-4c9f-81f8-0bed22c00883\" (UID: \"e6e864bc-d2bc-4c9f-81f8-0bed22c00883\") " Nov 03 00:53:34.301992 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:53:34.301682 4939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e6e864bc-d2bc-4c9f-81f8-0bed22c00883-tmp-dir" pod "e6e864bc-d2bc-4c9f-81f8-0bed22c00883" (UID: "e6e864bc-d2bc-4c9f-81f8-0bed22c00883"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 00:53:34.304266 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:53:34.303967 4939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e6e864bc-d2bc-4c9f-81f8-0bed22c00883-dshm" pod "e6e864bc-d2bc-4c9f-81f8-0bed22c00883" (UID: "e6e864bc-d2bc-4c9f-81f8-0bed22c00883"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 00:53:34.304266 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:53:34.304143 4939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e6e864bc-d2bc-4c9f-81f8-0bed22c00883-tls-certs" pod "e6e864bc-d2bc-4c9f-81f8-0bed22c00883" (UID: "e6e864bc-d2bc-4c9f-81f8-0bed22c00883"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 00:53:34.377184 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:53:34.377149 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 00:53:34.423982 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:53:34.412318 4939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e6e864bc-d2bc-4c9f-81f8-0bed22c00883-kube-api-access-nm85n\" (UniqueName: \"kubernetes.io/projected/e6e864bc-d2bc-4c9f-81f8-0bed22c00883-kube-api-access-nm85n\") pod \"e6e864bc-d2bc-4c9f-81f8-0bed22c00883\" (UID: \"e6e864bc-d2bc-4c9f-81f8-0bed22c00883\") " Nov 03 00:53:34.424367 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:53:34.424329 4939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e6e864bc-d2bc-4c9f-81f8-0bed22c00883-home\" (UniqueName: \"kubernetes.io/empty-dir/e6e864bc-d2bc-4c9f-81f8-0bed22c00883-home\") pod \"e6e864bc-d2bc-4c9f-81f8-0bed22c00883\" (UID: \"e6e864bc-d2bc-4c9f-81f8-0bed22c00883\") " Nov 03 00:53:34.428212 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:53:34.428187 4939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e6e864bc-d2bc-4c9f-81f8-0bed22c00883-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e6e864bc-d2bc-4c9f-81f8-0bed22c00883-model-cache\") pod \"e6e864bc-d2bc-4c9f-81f8-0bed22c00883\" (UID: \"e6e864bc-d2bc-4c9f-81f8-0bed22c00883\") " Nov 03 00:53:34.434509 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:53:34.428034 4939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e6e864bc-d2bc-4c9f-81f8-0bed22c00883-home" pod "e6e864bc-d2bc-4c9f-81f8-0bed22c00883" (UID: "e6e864bc-d2bc-4c9f-81f8-0bed22c00883"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 00:53:34.437352 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:53:34.434955 4939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e6e864bc-d2bc-4c9f-81f8-0bed22c00883-model-cache" pod "e6e864bc-d2bc-4c9f-81f8-0bed22c00883" (UID: "e6e864bc-d2bc-4c9f-81f8-0bed22c00883"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 00:53:34.437882 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:53:34.437855 4939 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e6e864bc-d2bc-4c9f-81f8-0bed22c00883-home\") on node \"ip-10-0-140-107.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 00:53:34.438038 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:53:34.438020 4939 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e6e864bc-d2bc-4c9f-81f8-0bed22c00883-model-cache\") on node \"ip-10-0-140-107.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 00:53:34.470984 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:53:34.446195 4939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e6e864bc-d2bc-4c9f-81f8-0bed22c00883-kube-api-access-nm85n" pod "e6e864bc-d2bc-4c9f-81f8-0bed22c00883" (UID: "e6e864bc-d2bc-4c9f-81f8-0bed22c00883"). InnerVolumeSpecName "kube-api-access-nm85n". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 00:53:34.470984 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:53:34.470771 4939 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e6e864bc-d2bc-4c9f-81f8-0bed22c00883-dshm\") on node \"ip-10-0-140-107.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 00:53:34.470984 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:53:34.470792 4939 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e6e864bc-d2bc-4c9f-81f8-0bed22c00883-tls-certs\") on node \"ip-10-0-140-107.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 00:53:34.470984 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:53:34.470806 4939 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e6e864bc-d2bc-4c9f-81f8-0bed22c00883-tmp-dir\") on node \"ip-10-0-140-107.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 00:53:34.572470 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:53:34.572378 4939 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nm85n\" (UniqueName: \"kubernetes.io/projected/e6e864bc-d2bc-4c9f-81f8-0bed22c00883-kube-api-access-nm85n\") on node \"ip-10-0-140-107.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 00:53:35.173829 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:53:35.173788 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 00:53:35.251238 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:53:35.249018 4939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-650be0/e2e-conflict-a-45480c9d-kserve-5c75dbc7bb-tjrmz" event={"ID":"e6e864bc-d2bc-4c9f-81f8-0bed22c00883","Type":"ContainerDied","Data":"9d753ec2458f819759c729f73b6334068d1ded722b1c634db6d8e446c8d06d57"} Nov 03 00:53:35.251238 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:53:35.249081 4939 scope.go:122] "RemoveContainer" containerID="4ff35624f3f7a2c5f8283325b4420facbff1ccc760eba36146da5bcf048dca59" Nov 03 00:53:35.251238 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:53:35.249271 4939 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-650be0/e2e-conflict-a-45480c9d-kserve-5c75dbc7bb-tjrmz" Nov 03 00:53:35.287134 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:53:35.287106 4939 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-650be0/e2e-conflict-a-45480c9d-kserve-5c75dbc7bb-tjrmz"] Nov 03 00:53:35.290992 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:53:35.290969 4939 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w4-650be0/e2e-conflict-a-45480c9d-kserve-5c75dbc7bb-tjrmz"] Nov 03 00:53:35.784278 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:53:35.784230 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 00:53:35.887677 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:53:35.887639 4939 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e6e864bc-d2bc-4c9f-81f8-0bed22c00883" path="/var/lib/kubelet/pods/e6e864bc-d2bc-4c9f-81f8-0bed22c00883/volumes" Nov 03 00:53:36.390200 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:53:36.390164 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 00:53:36.990448 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:53:36.990413 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 00:53:37.640604 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:53:37.640573 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 00:53:38.307931 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:53:38.307872 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 00:53:38.677837 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:53:38.677801 4939 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["models-as-a-service/payload-processing-external-model-6889df475b-r2899"] Nov 03 00:53:38.695284 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:53:38.695254 4939 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="models-as-a-service/payload-processing-external-model-6889df475b-r2899" Nov 03 00:53:38.756256 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:53:38.756225 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 00:53:38.881051 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:53:38.879885 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 00:53:39.289376 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:53:39.289342 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 00:53:40.184082 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:53:40.184052 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 00:53:40.956130 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:53:40.956088 4939 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w2-f7680d/distinct-2-w2-f7680d-kserve-7d7c4564bf-cbv2c" Nov 03 00:53:40.965731 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:53:40.965700 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 00:53:40.969295 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:53:40.969274 4939 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w2-f7680d/distinct-2-w2-f7680d-kserve-7d7c4564bf-cbv2c" Nov 03 00:53:41.299942 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:53:41.299837 4939 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-f7680d/distinct-2-w2-f7680d-kserve-7d7c4564bf-cbv2c" Nov 03 00:53:41.713119 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:53:41.713081 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 00:53:42.093713 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:53:42.093671 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 00:53:42.206710 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:53:42.206677 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 00:53:42.491801 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:53:42.491721 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 00:53:42.904217 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:53:42.904180 4939 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w0-2f9a3a/distinct-w0-2f9a3a-kserve-555565cfcf-jtcr9"] Nov 03 00:53:42.905131 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:53:42.905109 4939 cpu_manager.go:414] "containerMap: removing container" podUID="e6e864bc-d2bc-4c9f-81f8-0bed22c00883" containerName="main" Nov 03 00:53:42.905347 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:53:42.905324 4939 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e6e864bc-d2bc-4c9f-81f8-0bed22c00883" containerName="main" Nov 03 00:53:42.905540 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:53:42.905527 4939 memory_manager.go:365] "RemoveStaleState removing state" podUID="e6e864bc-d2bc-4c9f-81f8-0bed22c00883" containerName="main" Nov 03 00:53:42.909651 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:53:42.909627 4939 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-2f9a3a/distinct-w0-2f9a3a-kserve-555565cfcf-jtcr9" Nov 03 00:53:42.918173 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:53:42.918146 4939 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-2f9a3a/distinct-w0-2f9a3a-kserve-555565cfcf-jtcr9"] Nov 03 00:53:42.918480 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:53:42.918458 4939 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-2f9a3a\"/\"distinct-w0-2f9a3a-kserve-self-signed-certs\"" Nov 03 00:53:42.974673 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:53:42.974611 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 00:53:43.072941 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:53:43.071642 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ac0ece06-8718-463f-915f-5315988d9711-tls-certs\") pod \"distinct-w0-2f9a3a-kserve-555565cfcf-jtcr9\" (UID: \"ac0ece06-8718-463f-915f-5315988d9711\") " pod="e2e-models-e2e-worker-w0-2f9a3a/distinct-w0-2f9a3a-kserve-555565cfcf-jtcr9" Nov 03 00:53:43.072941 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:53:43.071708 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-28rcq\" (UniqueName: \"kubernetes.io/projected/ac0ece06-8718-463f-915f-5315988d9711-kube-api-access-28rcq\") pod \"distinct-w0-2f9a3a-kserve-555565cfcf-jtcr9\" (UID: \"ac0ece06-8718-463f-915f-5315988d9711\") " pod="e2e-models-e2e-worker-w0-2f9a3a/distinct-w0-2f9a3a-kserve-555565cfcf-jtcr9" Nov 03 00:53:43.072941 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:53:43.071809 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ac0ece06-8718-463f-915f-5315988d9711-tmp-dir\") pod \"distinct-w0-2f9a3a-kserve-555565cfcf-jtcr9\" (UID: \"ac0ece06-8718-463f-915f-5315988d9711\") " pod="e2e-models-e2e-worker-w0-2f9a3a/distinct-w0-2f9a3a-kserve-555565cfcf-jtcr9" Nov 03 00:53:43.072941 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:53:43.071848 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ac0ece06-8718-463f-915f-5315988d9711-home\") pod \"distinct-w0-2f9a3a-kserve-555565cfcf-jtcr9\" (UID: \"ac0ece06-8718-463f-915f-5315988d9711\") " pod="e2e-models-e2e-worker-w0-2f9a3a/distinct-w0-2f9a3a-kserve-555565cfcf-jtcr9" Nov 03 00:53:43.072941 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:53:43.071988 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ac0ece06-8718-463f-915f-5315988d9711-dshm\") pod \"distinct-w0-2f9a3a-kserve-555565cfcf-jtcr9\" (UID: \"ac0ece06-8718-463f-915f-5315988d9711\") " pod="e2e-models-e2e-worker-w0-2f9a3a/distinct-w0-2f9a3a-kserve-555565cfcf-jtcr9" Nov 03 00:53:43.072941 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:53:43.072014 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ac0ece06-8718-463f-915f-5315988d9711-model-cache\") pod \"distinct-w0-2f9a3a-kserve-555565cfcf-jtcr9\" (UID: \"ac0ece06-8718-463f-915f-5315988d9711\") " pod="e2e-models-e2e-worker-w0-2f9a3a/distinct-w0-2f9a3a-kserve-555565cfcf-jtcr9" Nov 03 00:53:43.173755 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:53:43.173657 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ac0ece06-8718-463f-915f-5315988d9711-dshm\") pod \"distinct-w0-2f9a3a-kserve-555565cfcf-jtcr9\" (UID: \"ac0ece06-8718-463f-915f-5315988d9711\") " pod="e2e-models-e2e-worker-w0-2f9a3a/distinct-w0-2f9a3a-kserve-555565cfcf-jtcr9" Nov 03 00:53:43.173755 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:53:43.173695 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ac0ece06-8718-463f-915f-5315988d9711-model-cache\") pod \"distinct-w0-2f9a3a-kserve-555565cfcf-jtcr9\" (UID: \"ac0ece06-8718-463f-915f-5315988d9711\") " pod="e2e-models-e2e-worker-w0-2f9a3a/distinct-w0-2f9a3a-kserve-555565cfcf-jtcr9" Nov 03 00:53:43.174052 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:53:43.173765 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ac0ece06-8718-463f-915f-5315988d9711-tls-certs\") pod \"distinct-w0-2f9a3a-kserve-555565cfcf-jtcr9\" (UID: \"ac0ece06-8718-463f-915f-5315988d9711\") " pod="e2e-models-e2e-worker-w0-2f9a3a/distinct-w0-2f9a3a-kserve-555565cfcf-jtcr9" Nov 03 00:53:43.174052 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:53:43.173788 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-28rcq\" (UniqueName: \"kubernetes.io/projected/ac0ece06-8718-463f-915f-5315988d9711-kube-api-access-28rcq\") pod \"distinct-w0-2f9a3a-kserve-555565cfcf-jtcr9\" (UID: \"ac0ece06-8718-463f-915f-5315988d9711\") " pod="e2e-models-e2e-worker-w0-2f9a3a/distinct-w0-2f9a3a-kserve-555565cfcf-jtcr9" Nov 03 00:53:43.174052 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:53:43.173847 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ac0ece06-8718-463f-915f-5315988d9711-tmp-dir\") pod \"distinct-w0-2f9a3a-kserve-555565cfcf-jtcr9\" (UID: \"ac0ece06-8718-463f-915f-5315988d9711\") " pod="e2e-models-e2e-worker-w0-2f9a3a/distinct-w0-2f9a3a-kserve-555565cfcf-jtcr9" Nov 03 00:53:43.174052 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:53:43.173868 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ac0ece06-8718-463f-915f-5315988d9711-home\") pod \"distinct-w0-2f9a3a-kserve-555565cfcf-jtcr9\" (UID: \"ac0ece06-8718-463f-915f-5315988d9711\") " pod="e2e-models-e2e-worker-w0-2f9a3a/distinct-w0-2f9a3a-kserve-555565cfcf-jtcr9" Nov 03 00:53:43.174310 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:53:43.174286 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ac0ece06-8718-463f-915f-5315988d9711-home\") pod \"distinct-w0-2f9a3a-kserve-555565cfcf-jtcr9\" (UID: \"ac0ece06-8718-463f-915f-5315988d9711\") " pod="e2e-models-e2e-worker-w0-2f9a3a/distinct-w0-2f9a3a-kserve-555565cfcf-jtcr9" Nov 03 00:53:43.174401 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:53:43.174380 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ac0ece06-8718-463f-915f-5315988d9711-tmp-dir\") pod \"distinct-w0-2f9a3a-kserve-555565cfcf-jtcr9\" (UID: \"ac0ece06-8718-463f-915f-5315988d9711\") " pod="e2e-models-e2e-worker-w0-2f9a3a/distinct-w0-2f9a3a-kserve-555565cfcf-jtcr9" Nov 03 00:53:43.174564 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:53:43.174539 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ac0ece06-8718-463f-915f-5315988d9711-model-cache\") pod \"distinct-w0-2f9a3a-kserve-555565cfcf-jtcr9\" (UID: \"ac0ece06-8718-463f-915f-5315988d9711\") " pod="e2e-models-e2e-worker-w0-2f9a3a/distinct-w0-2f9a3a-kserve-555565cfcf-jtcr9" Nov 03 00:53:43.176281 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:53:43.176254 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ac0ece06-8718-463f-915f-5315988d9711-dshm\") pod \"distinct-w0-2f9a3a-kserve-555565cfcf-jtcr9\" (UID: \"ac0ece06-8718-463f-915f-5315988d9711\") " pod="e2e-models-e2e-worker-w0-2f9a3a/distinct-w0-2f9a3a-kserve-555565cfcf-jtcr9" Nov 03 00:53:43.176708 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:53:43.176689 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ac0ece06-8718-463f-915f-5315988d9711-tls-certs\") pod \"distinct-w0-2f9a3a-kserve-555565cfcf-jtcr9\" (UID: \"ac0ece06-8718-463f-915f-5315988d9711\") " pod="e2e-models-e2e-worker-w0-2f9a3a/distinct-w0-2f9a3a-kserve-555565cfcf-jtcr9" Nov 03 00:53:43.182194 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:53:43.182164 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-28rcq\" (UniqueName: \"kubernetes.io/projected/ac0ece06-8718-463f-915f-5315988d9711-kube-api-access-28rcq\") pod \"distinct-w0-2f9a3a-kserve-555565cfcf-jtcr9\" (UID: \"ac0ece06-8718-463f-915f-5315988d9711\") " pod="e2e-models-e2e-worker-w0-2f9a3a/distinct-w0-2f9a3a-kserve-555565cfcf-jtcr9" Nov 03 00:53:43.229181 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:53:43.229147 4939 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-2f9a3a/distinct-w0-2f9a3a-kserve-555565cfcf-jtcr9" Nov 03 00:53:43.418243 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:53:43.418207 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 00:53:43.513007 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:53:43.512975 4939 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-2f9a3a/distinct-w0-2f9a3a-kserve-555565cfcf-jtcr9"] Nov 03 00:53:43.514084 ip-10-0-140-107 kubenswrapper[4939]: W1103 00:53:43.514057 4939 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podac0ece06_8718_463f_915f_5315988d9711.slice/crio-3835b2b377d056f2885d14046786441888c8c501770596213edbfc7f6f4c1ad5 WatchSource:0}: Error finding container 3835b2b377d056f2885d14046786441888c8c501770596213edbfc7f6f4c1ad5: Status 404 returned error can't find the container with id 3835b2b377d056f2885d14046786441888c8c501770596213edbfc7f6f4c1ad5 Nov 03 00:53:43.679296 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:53:43.679249 4939 kuberuntime_container.go:898] "Killing container with a grace period" pod="models-as-a-service/payload-processing-external-model-6889df475b-r2899" podUID="b2c06545-43b5-40a2-becc-cb8754bc68e2" containerName="payload-processing" containerID="cri-o://9e6d684ea1257d09651f30c3c98c455ca86e7bbb17e7e7ce849b04ccf269acfb" gracePeriod=25 Nov 03 00:53:43.935260 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:53:43.935236 4939 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-6889df475b-r2899" Nov 03 00:53:44.083648 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:53:44.083609 4939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b2c06545-43b5-40a2-becc-cb8754bc68e2-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b2c06545-43b5-40a2-becc-cb8754bc68e2-plugins-config-volume\") pod \"b2c06545-43b5-40a2-becc-cb8754bc68e2\" (UID: \"b2c06545-43b5-40a2-becc-cb8754bc68e2\") " Nov 03 00:53:44.083843 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:53:44.083714 4939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b2c06545-43b5-40a2-becc-cb8754bc68e2-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/b2c06545-43b5-40a2-becc-cb8754bc68e2-routing-overlay-volume\") pod \"b2c06545-43b5-40a2-becc-cb8754bc68e2\" (UID: \"b2c06545-43b5-40a2-becc-cb8754bc68e2\") " Nov 03 00:53:44.083918 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:53:44.083839 4939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b2c06545-43b5-40a2-becc-cb8754bc68e2-kube-api-access-wvr8h\" (UniqueName: \"kubernetes.io/projected/b2c06545-43b5-40a2-becc-cb8754bc68e2-kube-api-access-wvr8h\") pod \"b2c06545-43b5-40a2-becc-cb8754bc68e2\" (UID: \"b2c06545-43b5-40a2-becc-cb8754bc68e2\") " Nov 03 00:53:44.084030 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:53:44.084010 4939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b2c06545-43b5-40a2-becc-cb8754bc68e2-provider-credentials\" (UniqueName: \"kubernetes.io/projected/b2c06545-43b5-40a2-becc-cb8754bc68e2-provider-credentials\") pod \"b2c06545-43b5-40a2-becc-cb8754bc68e2\" (UID: \"b2c06545-43b5-40a2-becc-cb8754bc68e2\") " Nov 03 00:53:44.084123 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:53:44.084102 4939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b2c06545-43b5-40a2-becc-cb8754bc68e2-plugins-config-volume" pod "b2c06545-43b5-40a2-becc-cb8754bc68e2" (UID: "b2c06545-43b5-40a2-becc-cb8754bc68e2"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 00:53:44.084186 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:53:44.084146 4939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b2c06545-43b5-40a2-becc-cb8754bc68e2-routing-overlay-volume" pod "b2c06545-43b5-40a2-becc-cb8754bc68e2" (UID: "b2c06545-43b5-40a2-becc-cb8754bc68e2"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 00:53:44.084443 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:53:44.084425 4939 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b2c06545-43b5-40a2-becc-cb8754bc68e2-plugins-config-volume\") on node \"ip-10-0-140-107.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 00:53:44.084527 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:53:44.084444 4939 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/b2c06545-43b5-40a2-becc-cb8754bc68e2-routing-overlay-volume\") on node \"ip-10-0-140-107.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 00:53:44.086030 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:53:44.086009 4939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b2c06545-43b5-40a2-becc-cb8754bc68e2-kube-api-access-wvr8h" pod "b2c06545-43b5-40a2-becc-cb8754bc68e2" (UID: "b2c06545-43b5-40a2-becc-cb8754bc68e2"). InnerVolumeSpecName "kube-api-access-wvr8h". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 00:53:44.086030 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:53:44.086019 4939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b2c06545-43b5-40a2-becc-cb8754bc68e2-provider-credentials" pod "b2c06545-43b5-40a2-becc-cb8754bc68e2" (UID: "b2c06545-43b5-40a2-becc-cb8754bc68e2"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 00:53:44.135459 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:53:44.135415 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 00:53:44.195932 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:53:44.189130 4939 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wvr8h\" (UniqueName: \"kubernetes.io/projected/b2c06545-43b5-40a2-becc-cb8754bc68e2-kube-api-access-wvr8h\") on node \"ip-10-0-140-107.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 00:53:44.195932 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:53:44.189169 4939 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/b2c06545-43b5-40a2-becc-cb8754bc68e2-provider-credentials\") on node \"ip-10-0-140-107.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 00:53:44.315870 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:53:44.315837 4939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-2f9a3a/distinct-w0-2f9a3a-kserve-555565cfcf-jtcr9" event={"ID":"ac0ece06-8718-463f-915f-5315988d9711","Type":"ContainerStarted","Data":"80286a14e895c6364341c44e7045505b4d8a73753b16649945598afd3730fd5e"} Nov 03 00:53:44.315870 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:53:44.315878 4939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-2f9a3a/distinct-w0-2f9a3a-kserve-555565cfcf-jtcr9" event={"ID":"ac0ece06-8718-463f-915f-5315988d9711","Type":"ContainerStarted","Data":"3835b2b377d056f2885d14046786441888c8c501770596213edbfc7f6f4c1ad5"} Nov 03 00:53:44.317236 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:53:44.317212 4939 generic.go:356] "Generic (PLEG): container finished" podID="b2c06545-43b5-40a2-becc-cb8754bc68e2" containerID="9e6d684ea1257d09651f30c3c98c455ca86e7bbb17e7e7ce849b04ccf269acfb" exitCode=0 Nov 03 00:53:44.317347 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:53:44.317241 4939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-6889df475b-r2899" event={"ID":"b2c06545-43b5-40a2-becc-cb8754bc68e2","Type":"ContainerDied","Data":"9e6d684ea1257d09651f30c3c98c455ca86e7bbb17e7e7ce849b04ccf269acfb"} Nov 03 00:53:44.317347 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:53:44.317258 4939 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-6889df475b-r2899" Nov 03 00:53:44.317347 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:53:44.317290 4939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-6889df475b-r2899" event={"ID":"b2c06545-43b5-40a2-becc-cb8754bc68e2","Type":"ContainerDied","Data":"b0f4ab9f51ca12aaa02c1566c44675e9ce7329d0d51742663b2f7bcc186b6de8"} Nov 03 00:53:44.317347 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:53:44.317311 4939 scope.go:122] "RemoveContainer" containerID="9e6d684ea1257d09651f30c3c98c455ca86e7bbb17e7e7ce849b04ccf269acfb" Nov 03 00:53:44.327791 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:53:44.327776 4939 scope.go:122] "RemoveContainer" containerID="9e6d684ea1257d09651f30c3c98c455ca86e7bbb17e7e7ce849b04ccf269acfb" Nov 03 00:53:44.328069 ip-10-0-140-107 kubenswrapper[4939]: E1103 00:53:44.328050 4939 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9e6d684ea1257d09651f30c3c98c455ca86e7bbb17e7e7ce849b04ccf269acfb\": container with ID starting with 9e6d684ea1257d09651f30c3c98c455ca86e7bbb17e7e7ce849b04ccf269acfb not found: ID does not exist" containerID="9e6d684ea1257d09651f30c3c98c455ca86e7bbb17e7e7ce849b04ccf269acfb" Nov 03 00:53:44.328148 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:53:44.328079 4939 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9e6d684ea1257d09651f30c3c98c455ca86e7bbb17e7e7ce849b04ccf269acfb"} err="failed to get container status \"9e6d684ea1257d09651f30c3c98c455ca86e7bbb17e7e7ce849b04ccf269acfb\": rpc error: code = NotFound desc = could not find container \"9e6d684ea1257d09651f30c3c98c455ca86e7bbb17e7e7ce849b04ccf269acfb\": container with ID starting with 9e6d684ea1257d09651f30c3c98c455ca86e7bbb17e7e7ce849b04ccf269acfb not found: ID does not exist" Nov 03 00:53:44.335634 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:53:44.335568 4939 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w0-2f9a3a/distinct-w0-2f9a3a-kserve-555565cfcf-jtcr9" podStartSLOduration=2.335558369 podStartE2EDuration="2.335558369s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:53:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:53:44.334247299 +0000 UTC m=+772.598582243" watchObservedRunningTime="2026-11-03 00:53:44.335558369 +0000 UTC m=+772.599893326" Nov 03 00:53:44.348929 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:53:44.348880 4939 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["models-as-a-service/payload-processing-external-model-6889df475b-r2899"] Nov 03 00:53:44.354378 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:53:44.354358 4939 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["models-as-a-service/payload-processing-external-model-6889df475b-r2899"] Nov 03 00:53:44.912286 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:53:44.912252 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 00:53:45.234986 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:53:45.233048 4939 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-2f9a3a/distinct-w0-2f9a3a-kserve-555565cfcf-jtcr9" Nov 03 00:53:45.804944 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:53:45.803528 4939 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-83755a-openshift-default-c86f9565c-rwxw4"] Nov 03 00:53:45.835674 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:53:45.835611 4939 cpu_manager.go:414] "containerMap: removing container" podUID="b2c06545-43b5-40a2-becc-cb8754bc68e2" containerName="payload-processing" Nov 03 00:53:45.835674 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:53:45.835645 4939 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b2c06545-43b5-40a2-becc-cb8754bc68e2" containerName="payload-processing" Nov 03 00:53:45.836030 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:53:45.835809 4939 memory_manager.go:365] "RemoveStaleState removing state" podUID="b2c06545-43b5-40a2-becc-cb8754bc68e2" containerName="payload-processing" Nov 03 00:53:45.905125 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:53:45.905092 4939 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-83755a-openshift-default-c86f9565c-rwxw4" Nov 03 00:53:46.089948 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:53:46.089896 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-62w7v\" (UniqueName: \"kubernetes.io/projected/4adcd893-4e34-4860-bafb-a32ee8b956f0-kube-api-access-62w7v\") pod \"e2e-worker-w3-83755a-openshift-default-c86f9565c-rwxw4\" (UID: \"4adcd893-4e34-4860-bafb-a32ee8b956f0\") " pod="openshift-ingress/e2e-worker-w3-83755a-openshift-default-c86f9565c-rwxw4" Nov 03 00:53:46.200978 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:53:46.175254 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4adcd893-4e34-4860-bafb-a32ee8b956f0-workload-certs\") pod \"e2e-worker-w3-83755a-openshift-default-c86f9565c-rwxw4\" (UID: \"4adcd893-4e34-4860-bafb-a32ee8b956f0\") " pod="openshift-ingress/e2e-worker-w3-83755a-openshift-default-c86f9565c-rwxw4" Nov 03 00:53:46.200978 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:53:46.177318 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 00:53:46.214952 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:53:46.211287 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4adcd893-4e34-4860-bafb-a32ee8b956f0-workload-socket\") pod \"e2e-worker-w3-83755a-openshift-default-c86f9565c-rwxw4\" (UID: \"4adcd893-4e34-4860-bafb-a32ee8b956f0\") " pod="openshift-ingress/e2e-worker-w3-83755a-openshift-default-c86f9565c-rwxw4" Nov 03 00:53:46.246996 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:53:46.236979 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4adcd893-4e34-4860-bafb-a32ee8b956f0-istio-data\") pod \"e2e-worker-w3-83755a-openshift-default-c86f9565c-rwxw4\" (UID: \"4adcd893-4e34-4860-bafb-a32ee8b956f0\") " pod="openshift-ingress/e2e-worker-w3-83755a-openshift-default-c86f9565c-rwxw4" Nov 03 00:53:46.246996 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:53:46.237095 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4adcd893-4e34-4860-bafb-a32ee8b956f0-istio-token\") pod \"e2e-worker-w3-83755a-openshift-default-c86f9565c-rwxw4\" (UID: \"4adcd893-4e34-4860-bafb-a32ee8b956f0\") " pod="openshift-ingress/e2e-worker-w3-83755a-openshift-default-c86f9565c-rwxw4" Nov 03 00:53:46.246996 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:53:46.237193 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4adcd893-4e34-4860-bafb-a32ee8b956f0-credential-socket\") pod \"e2e-worker-w3-83755a-openshift-default-c86f9565c-rwxw4\" (UID: \"4adcd893-4e34-4860-bafb-a32ee8b956f0\") " pod="openshift-ingress/e2e-worker-w3-83755a-openshift-default-c86f9565c-rwxw4" Nov 03 00:53:46.246996 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:53:46.237226 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4adcd893-4e34-4860-bafb-a32ee8b956f0-istiod-ca-cert\") pod \"e2e-worker-w3-83755a-openshift-default-c86f9565c-rwxw4\" (UID: \"4adcd893-4e34-4860-bafb-a32ee8b956f0\") " pod="openshift-ingress/e2e-worker-w3-83755a-openshift-default-c86f9565c-rwxw4" Nov 03 00:53:46.246996 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:53:46.237351 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4adcd893-4e34-4860-bafb-a32ee8b956f0-istio-podinfo\") pod \"e2e-worker-w3-83755a-openshift-default-c86f9565c-rwxw4\" (UID: \"4adcd893-4e34-4860-bafb-a32ee8b956f0\") " pod="openshift-ingress/e2e-worker-w3-83755a-openshift-default-c86f9565c-rwxw4" Nov 03 00:53:46.246996 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:53:46.237414 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4adcd893-4e34-4860-bafb-a32ee8b956f0-istio-envoy\") pod \"e2e-worker-w3-83755a-openshift-default-c86f9565c-rwxw4\" (UID: \"4adcd893-4e34-4860-bafb-a32ee8b956f0\") " pod="openshift-ingress/e2e-worker-w3-83755a-openshift-default-c86f9565c-rwxw4" Nov 03 00:53:46.295988 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:53:46.275808 4939 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b2c06545-43b5-40a2-becc-cb8754bc68e2" path="/var/lib/kubelet/pods/b2c06545-43b5-40a2-becc-cb8754bc68e2/volumes" Nov 03 00:53:46.297026 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:53:46.296983 4939 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-83755a-openshift-default-c86f9565c-rwxw4"] Nov 03 00:53:46.320017 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:53:46.297238 4939 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-83755a-openshift-default-c86f9565c-nzmzn"] Nov 03 00:53:46.374981 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:53:46.343785 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4adcd893-4e34-4860-bafb-a32ee8b956f0-istio-podinfo\") pod \"e2e-worker-w3-83755a-openshift-default-c86f9565c-rwxw4\" (UID: \"4adcd893-4e34-4860-bafb-a32ee8b956f0\") " pod="openshift-ingress/e2e-worker-w3-83755a-openshift-default-c86f9565c-rwxw4" Nov 03 00:53:46.403510 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:53:46.403430 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4adcd893-4e34-4860-bafb-a32ee8b956f0-istio-envoy\") pod \"e2e-worker-w3-83755a-openshift-default-c86f9565c-rwxw4\" (UID: \"4adcd893-4e34-4860-bafb-a32ee8b956f0\") " pod="openshift-ingress/e2e-worker-w3-83755a-openshift-default-c86f9565c-rwxw4" Nov 03 00:53:46.431492 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:53:46.428239 4939 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-83755a-openshift-default-c86f9565c-nzmzn" Nov 03 00:53:46.431797 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:53:46.428131 4939 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-83755a-openshift-default-c86f9565c-nzmzn"] Nov 03 00:53:46.446932 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:53:46.443770 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-62w7v\" (UniqueName: \"kubernetes.io/projected/4adcd893-4e34-4860-bafb-a32ee8b956f0-kube-api-access-62w7v\") pod \"e2e-worker-w3-83755a-openshift-default-c86f9565c-rwxw4\" (UID: \"4adcd893-4e34-4860-bafb-a32ee8b956f0\") " pod="openshift-ingress/e2e-worker-w3-83755a-openshift-default-c86f9565c-rwxw4" Nov 03 00:53:46.451041 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:53:46.451009 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4adcd893-4e34-4860-bafb-a32ee8b956f0-workload-certs\") pod \"e2e-worker-w3-83755a-openshift-default-c86f9565c-rwxw4\" (UID: \"4adcd893-4e34-4860-bafb-a32ee8b956f0\") " pod="openshift-ingress/e2e-worker-w3-83755a-openshift-default-c86f9565c-rwxw4" Nov 03 00:53:46.451237 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:53:46.451218 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4adcd893-4e34-4860-bafb-a32ee8b956f0-workload-socket\") pod \"e2e-worker-w3-83755a-openshift-default-c86f9565c-rwxw4\" (UID: \"4adcd893-4e34-4860-bafb-a32ee8b956f0\") " pod="openshift-ingress/e2e-worker-w3-83755a-openshift-default-c86f9565c-rwxw4" Nov 03 00:53:46.455013 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:53:46.454986 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4adcd893-4e34-4860-bafb-a32ee8b956f0-istio-data\") pod \"e2e-worker-w3-83755a-openshift-default-c86f9565c-rwxw4\" (UID: \"4adcd893-4e34-4860-bafb-a32ee8b956f0\") " pod="openshift-ingress/e2e-worker-w3-83755a-openshift-default-c86f9565c-rwxw4" Nov 03 00:53:46.458929 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:53:46.455202 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4adcd893-4e34-4860-bafb-a32ee8b956f0-istio-token\") pod \"e2e-worker-w3-83755a-openshift-default-c86f9565c-rwxw4\" (UID: \"4adcd893-4e34-4860-bafb-a32ee8b956f0\") " pod="openshift-ingress/e2e-worker-w3-83755a-openshift-default-c86f9565c-rwxw4" Nov 03 00:53:46.459171 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:53:46.459150 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4adcd893-4e34-4860-bafb-a32ee8b956f0-credential-socket\") pod \"e2e-worker-w3-83755a-openshift-default-c86f9565c-rwxw4\" (UID: \"4adcd893-4e34-4860-bafb-a32ee8b956f0\") " pod="openshift-ingress/e2e-worker-w3-83755a-openshift-default-c86f9565c-rwxw4" Nov 03 00:53:46.463003 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:53:46.462965 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4adcd893-4e34-4860-bafb-a32ee8b956f0-istiod-ca-cert\") pod \"e2e-worker-w3-83755a-openshift-default-c86f9565c-rwxw4\" (UID: \"4adcd893-4e34-4860-bafb-a32ee8b956f0\") " pod="openshift-ingress/e2e-worker-w3-83755a-openshift-default-c86f9565c-rwxw4" Nov 03 00:53:46.488933 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:53:46.453521 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4adcd893-4e34-4860-bafb-a32ee8b956f0-workload-socket\") pod \"e2e-worker-w3-83755a-openshift-default-c86f9565c-rwxw4\" (UID: \"4adcd893-4e34-4860-bafb-a32ee8b956f0\") " pod="openshift-ingress/e2e-worker-w3-83755a-openshift-default-c86f9565c-rwxw4" Nov 03 00:53:46.490026 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:53:46.453286 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4adcd893-4e34-4860-bafb-a32ee8b956f0-workload-certs\") pod \"e2e-worker-w3-83755a-openshift-default-c86f9565c-rwxw4\" (UID: \"4adcd893-4e34-4860-bafb-a32ee8b956f0\") " pod="openshift-ingress/e2e-worker-w3-83755a-openshift-default-c86f9565c-rwxw4" Nov 03 00:53:46.490208 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:53:46.475573 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4adcd893-4e34-4860-bafb-a32ee8b956f0-istio-envoy\") pod \"e2e-worker-w3-83755a-openshift-default-c86f9565c-rwxw4\" (UID: \"4adcd893-4e34-4860-bafb-a32ee8b956f0\") " pod="openshift-ingress/e2e-worker-w3-83755a-openshift-default-c86f9565c-rwxw4" Nov 03 00:53:46.490318 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:53:46.458981 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4adcd893-4e34-4860-bafb-a32ee8b956f0-istio-data\") pod \"e2e-worker-w3-83755a-openshift-default-c86f9565c-rwxw4\" (UID: \"4adcd893-4e34-4860-bafb-a32ee8b956f0\") " pod="openshift-ingress/e2e-worker-w3-83755a-openshift-default-c86f9565c-rwxw4" Nov 03 00:53:46.490415 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:53:46.477264 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4adcd893-4e34-4860-bafb-a32ee8b956f0-credential-socket\") pod \"e2e-worker-w3-83755a-openshift-default-c86f9565c-rwxw4\" (UID: \"4adcd893-4e34-4860-bafb-a32ee8b956f0\") " pod="openshift-ingress/e2e-worker-w3-83755a-openshift-default-c86f9565c-rwxw4" Nov 03 00:53:46.490510 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:53:46.477937 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4adcd893-4e34-4860-bafb-a32ee8b956f0-istiod-ca-cert\") pod \"e2e-worker-w3-83755a-openshift-default-c86f9565c-rwxw4\" (UID: \"4adcd893-4e34-4860-bafb-a32ee8b956f0\") " pod="openshift-ingress/e2e-worker-w3-83755a-openshift-default-c86f9565c-rwxw4" Nov 03 00:53:46.490615 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:53:46.444987 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4adcd893-4e34-4860-bafb-a32ee8b956f0-istio-podinfo\") pod \"e2e-worker-w3-83755a-openshift-default-c86f9565c-rwxw4\" (UID: \"4adcd893-4e34-4860-bafb-a32ee8b956f0\") " pod="openshift-ingress/e2e-worker-w3-83755a-openshift-default-c86f9565c-rwxw4" Nov 03 00:53:46.535960 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:53:46.527486 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4adcd893-4e34-4860-bafb-a32ee8b956f0-istio-token\") pod \"e2e-worker-w3-83755a-openshift-default-c86f9565c-rwxw4\" (UID: \"4adcd893-4e34-4860-bafb-a32ee8b956f0\") " pod="openshift-ingress/e2e-worker-w3-83755a-openshift-default-c86f9565c-rwxw4" Nov 03 00:53:46.539960 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:53:46.536254 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-62w7v\" (UniqueName: \"kubernetes.io/projected/4adcd893-4e34-4860-bafb-a32ee8b956f0-kube-api-access-62w7v\") pod \"e2e-worker-w3-83755a-openshift-default-c86f9565c-rwxw4\" (UID: \"4adcd893-4e34-4860-bafb-a32ee8b956f0\") " pod="openshift-ingress/e2e-worker-w3-83755a-openshift-default-c86f9565c-rwxw4" Nov 03 00:53:46.582456 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:53:46.564668 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/44fe729d-d7d1-41c2-99df-60e955909b8b-workload-socket\") pod \"e2e-worker-w3-83755a-openshift-default-c86f9565c-nzmzn\" (UID: \"44fe729d-d7d1-41c2-99df-60e955909b8b\") " pod="openshift-ingress/e2e-worker-w3-83755a-openshift-default-c86f9565c-nzmzn" Nov 03 00:53:46.582456 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:53:46.564758 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/44fe729d-d7d1-41c2-99df-60e955909b8b-istio-envoy\") pod \"e2e-worker-w3-83755a-openshift-default-c86f9565c-nzmzn\" (UID: \"44fe729d-d7d1-41c2-99df-60e955909b8b\") " pod="openshift-ingress/e2e-worker-w3-83755a-openshift-default-c86f9565c-nzmzn" Nov 03 00:53:46.582456 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:53:46.564791 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/44fe729d-d7d1-41c2-99df-60e955909b8b-istio-podinfo\") pod \"e2e-worker-w3-83755a-openshift-default-c86f9565c-nzmzn\" (UID: \"44fe729d-d7d1-41c2-99df-60e955909b8b\") " pod="openshift-ingress/e2e-worker-w3-83755a-openshift-default-c86f9565c-nzmzn" Nov 03 00:53:46.582456 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:53:46.564873 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/44fe729d-d7d1-41c2-99df-60e955909b8b-istio-token\") pod \"e2e-worker-w3-83755a-openshift-default-c86f9565c-nzmzn\" (UID: \"44fe729d-d7d1-41c2-99df-60e955909b8b\") " pod="openshift-ingress/e2e-worker-w3-83755a-openshift-default-c86f9565c-nzmzn" Nov 03 00:53:46.582456 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:53:46.564988 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/44fe729d-d7d1-41c2-99df-60e955909b8b-credential-socket\") pod \"e2e-worker-w3-83755a-openshift-default-c86f9565c-nzmzn\" (UID: \"44fe729d-d7d1-41c2-99df-60e955909b8b\") " pod="openshift-ingress/e2e-worker-w3-83755a-openshift-default-c86f9565c-nzmzn" Nov 03 00:53:46.582456 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:53:46.565026 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5f6cd\" (UniqueName: \"kubernetes.io/projected/44fe729d-d7d1-41c2-99df-60e955909b8b-kube-api-access-5f6cd\") pod \"e2e-worker-w3-83755a-openshift-default-c86f9565c-nzmzn\" (UID: \"44fe729d-d7d1-41c2-99df-60e955909b8b\") " pod="openshift-ingress/e2e-worker-w3-83755a-openshift-default-c86f9565c-nzmzn" Nov 03 00:53:46.582456 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:53:46.565106 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/44fe729d-d7d1-41c2-99df-60e955909b8b-istiod-ca-cert\") pod \"e2e-worker-w3-83755a-openshift-default-c86f9565c-nzmzn\" (UID: \"44fe729d-d7d1-41c2-99df-60e955909b8b\") " pod="openshift-ingress/e2e-worker-w3-83755a-openshift-default-c86f9565c-nzmzn" Nov 03 00:53:46.582456 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:53:46.565141 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/44fe729d-d7d1-41c2-99df-60e955909b8b-istio-data\") pod \"e2e-worker-w3-83755a-openshift-default-c86f9565c-nzmzn\" (UID: \"44fe729d-d7d1-41c2-99df-60e955909b8b\") " pod="openshift-ingress/e2e-worker-w3-83755a-openshift-default-c86f9565c-nzmzn" Nov 03 00:53:46.582456 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:53:46.565175 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/44fe729d-d7d1-41c2-99df-60e955909b8b-workload-certs\") pod \"e2e-worker-w3-83755a-openshift-default-c86f9565c-nzmzn\" (UID: \"44fe729d-d7d1-41c2-99df-60e955909b8b\") " pod="openshift-ingress/e2e-worker-w3-83755a-openshift-default-c86f9565c-nzmzn" Nov 03 00:53:46.660005 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:53:46.643091 4939 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-83755a-openshift-default-c86f9565c-rwxw4" Nov 03 00:53:46.693809 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:53:46.668737 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5f6cd\" (UniqueName: \"kubernetes.io/projected/44fe729d-d7d1-41c2-99df-60e955909b8b-kube-api-access-5f6cd\") pod \"e2e-worker-w3-83755a-openshift-default-c86f9565c-nzmzn\" (UID: \"44fe729d-d7d1-41c2-99df-60e955909b8b\") " pod="openshift-ingress/e2e-worker-w3-83755a-openshift-default-c86f9565c-nzmzn" Nov 03 00:53:46.693809 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:53:46.668834 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/44fe729d-d7d1-41c2-99df-60e955909b8b-istiod-ca-cert\") pod \"e2e-worker-w3-83755a-openshift-default-c86f9565c-nzmzn\" (UID: \"44fe729d-d7d1-41c2-99df-60e955909b8b\") " pod="openshift-ingress/e2e-worker-w3-83755a-openshift-default-c86f9565c-nzmzn" Nov 03 00:53:46.693809 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:53:46.668866 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/44fe729d-d7d1-41c2-99df-60e955909b8b-istio-data\") pod \"e2e-worker-w3-83755a-openshift-default-c86f9565c-nzmzn\" (UID: \"44fe729d-d7d1-41c2-99df-60e955909b8b\") " pod="openshift-ingress/e2e-worker-w3-83755a-openshift-default-c86f9565c-nzmzn" Nov 03 00:53:46.693809 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:53:46.668900 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/44fe729d-d7d1-41c2-99df-60e955909b8b-workload-certs\") pod \"e2e-worker-w3-83755a-openshift-default-c86f9565c-nzmzn\" (UID: \"44fe729d-d7d1-41c2-99df-60e955909b8b\") " pod="openshift-ingress/e2e-worker-w3-83755a-openshift-default-c86f9565c-nzmzn" Nov 03 00:53:46.693809 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:53:46.669031 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/44fe729d-d7d1-41c2-99df-60e955909b8b-workload-socket\") pod \"e2e-worker-w3-83755a-openshift-default-c86f9565c-nzmzn\" (UID: \"44fe729d-d7d1-41c2-99df-60e955909b8b\") " pod="openshift-ingress/e2e-worker-w3-83755a-openshift-default-c86f9565c-nzmzn" Nov 03 00:53:46.693809 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:53:46.669085 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/44fe729d-d7d1-41c2-99df-60e955909b8b-istio-envoy\") pod \"e2e-worker-w3-83755a-openshift-default-c86f9565c-nzmzn\" (UID: \"44fe729d-d7d1-41c2-99df-60e955909b8b\") " pod="openshift-ingress/e2e-worker-w3-83755a-openshift-default-c86f9565c-nzmzn" Nov 03 00:53:46.693809 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:53:46.669125 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/44fe729d-d7d1-41c2-99df-60e955909b8b-istio-podinfo\") pod \"e2e-worker-w3-83755a-openshift-default-c86f9565c-nzmzn\" (UID: \"44fe729d-d7d1-41c2-99df-60e955909b8b\") " pod="openshift-ingress/e2e-worker-w3-83755a-openshift-default-c86f9565c-nzmzn" Nov 03 00:53:46.693809 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:53:46.669183 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/44fe729d-d7d1-41c2-99df-60e955909b8b-istio-token\") pod \"e2e-worker-w3-83755a-openshift-default-c86f9565c-nzmzn\" (UID: \"44fe729d-d7d1-41c2-99df-60e955909b8b\") " pod="openshift-ingress/e2e-worker-w3-83755a-openshift-default-c86f9565c-nzmzn" Nov 03 00:53:46.693809 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:53:46.669286 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/44fe729d-d7d1-41c2-99df-60e955909b8b-credential-socket\") pod \"e2e-worker-w3-83755a-openshift-default-c86f9565c-nzmzn\" (UID: \"44fe729d-d7d1-41c2-99df-60e955909b8b\") " pod="openshift-ingress/e2e-worker-w3-83755a-openshift-default-c86f9565c-nzmzn" Nov 03 00:53:46.693809 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:53:46.669719 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/44fe729d-d7d1-41c2-99df-60e955909b8b-credential-socket\") pod \"e2e-worker-w3-83755a-openshift-default-c86f9565c-nzmzn\" (UID: \"44fe729d-d7d1-41c2-99df-60e955909b8b\") " pod="openshift-ingress/e2e-worker-w3-83755a-openshift-default-c86f9565c-nzmzn" Nov 03 00:53:46.693809 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:53:46.670724 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/44fe729d-d7d1-41c2-99df-60e955909b8b-istiod-ca-cert\") pod \"e2e-worker-w3-83755a-openshift-default-c86f9565c-nzmzn\" (UID: \"44fe729d-d7d1-41c2-99df-60e955909b8b\") " pod="openshift-ingress/e2e-worker-w3-83755a-openshift-default-c86f9565c-nzmzn" Nov 03 00:53:46.693809 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:53:46.671002 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/44fe729d-d7d1-41c2-99df-60e955909b8b-istio-data\") pod \"e2e-worker-w3-83755a-openshift-default-c86f9565c-nzmzn\" (UID: \"44fe729d-d7d1-41c2-99df-60e955909b8b\") " pod="openshift-ingress/e2e-worker-w3-83755a-openshift-default-c86f9565c-nzmzn" Nov 03 00:53:46.693809 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:53:46.671260 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/44fe729d-d7d1-41c2-99df-60e955909b8b-workload-certs\") pod \"e2e-worker-w3-83755a-openshift-default-c86f9565c-nzmzn\" (UID: \"44fe729d-d7d1-41c2-99df-60e955909b8b\") " pod="openshift-ingress/e2e-worker-w3-83755a-openshift-default-c86f9565c-nzmzn" Nov 03 00:53:46.693809 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:53:46.671507 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/44fe729d-d7d1-41c2-99df-60e955909b8b-workload-socket\") pod \"e2e-worker-w3-83755a-openshift-default-c86f9565c-nzmzn\" (UID: \"44fe729d-d7d1-41c2-99df-60e955909b8b\") " pod="openshift-ingress/e2e-worker-w3-83755a-openshift-default-c86f9565c-nzmzn" Nov 03 00:53:46.712812 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:53:46.712763 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5f6cd\" (UniqueName: \"kubernetes.io/projected/44fe729d-d7d1-41c2-99df-60e955909b8b-kube-api-access-5f6cd\") pod \"e2e-worker-w3-83755a-openshift-default-c86f9565c-nzmzn\" (UID: \"44fe729d-d7d1-41c2-99df-60e955909b8b\") " pod="openshift-ingress/e2e-worker-w3-83755a-openshift-default-c86f9565c-nzmzn" Nov 03 00:53:46.717982 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:53:46.717676 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/44fe729d-d7d1-41c2-99df-60e955909b8b-istio-podinfo\") pod \"e2e-worker-w3-83755a-openshift-default-c86f9565c-nzmzn\" (UID: \"44fe729d-d7d1-41c2-99df-60e955909b8b\") " pod="openshift-ingress/e2e-worker-w3-83755a-openshift-default-c86f9565c-nzmzn" Nov 03 00:53:46.773874 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:53:46.772808 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/44fe729d-d7d1-41c2-99df-60e955909b8b-istio-token\") pod \"e2e-worker-w3-83755a-openshift-default-c86f9565c-nzmzn\" (UID: \"44fe729d-d7d1-41c2-99df-60e955909b8b\") " pod="openshift-ingress/e2e-worker-w3-83755a-openshift-default-c86f9565c-nzmzn" Nov 03 00:53:46.774341 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:53:46.774314 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/44fe729d-d7d1-41c2-99df-60e955909b8b-istio-envoy\") pod \"e2e-worker-w3-83755a-openshift-default-c86f9565c-nzmzn\" (UID: \"44fe729d-d7d1-41c2-99df-60e955909b8b\") " pod="openshift-ingress/e2e-worker-w3-83755a-openshift-default-c86f9565c-nzmzn" Nov 03 00:53:46.938988 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:53:46.938359 4939 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-83755a-openshift-default-c86f9565c-nzmzn" Nov 03 00:53:47.322997 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:53:47.313000 4939 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-f7680d/distinct-2-w2-f7680d-kserve-7d7c4564bf-cbv2c" podUID="2de18086-761b-4773-b9fc-15ddc4a220fc" containerName="main" probeResult="failure" output="Get \"https://10.132.0.75:8000/ready\": context deadline exceeded" Nov 03 00:53:47.451979 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:53:47.451021 4939 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-2f9a3a/simulator-w0-2f9a3a-kserve-9596bbb8f-2trgz" podUID="207e2207-fb6c-4076-95df-72e95671c261" containerName="main" probeResult="failure" output="Get \"https://10.132.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 00:53:47.564837 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:53:47.564793 4939 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-88bbf7/simulator-w1-88bbf7-kserve-5fbb669f8f-42hgd" podUID="83b38883-2e7a-444d-aa68-52be4a25ae66" containerName="main" probeResult="failure" output="Get \"https://10.132.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 00:53:48.000989 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:53:47.984977 4939 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-f7680d/premium-w2-f7680d-kserve-76565b8cf9-hqbbt" podUID="4712e708-18a2-4a13-a12c-a5669397b2c3" containerName="main" probeResult="failure" output="Get \"https://10.132.0.63:8000/ready\": context deadline exceeded" Nov 03 00:53:48.101951 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:53:48.086694 4939 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-f7680d/distinct-w2-f7680d-kserve-756f874b98-qnpv8" podUID="836753fc-f9ef-4f61-b3fe-e5b92b7ee27b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.70:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 00:53:48.101951 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:53:48.087634 4939 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-83755a/distinct-2-w3-83755a-kserve-6cf66b995d-2l47b" podUID="c3b05e88-5d5b-4d03-b28d-832f22e3c868" containerName="main" probeResult="failure" output="Get \"https://10.132.0.72:8000/ready\": context deadline exceeded" Nov 03 00:53:48.236233 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:53:48.194029 4939 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-f7680d/distinct-w2-f7680d-kserve-756f874b98-qnpv8" podUID="836753fc-f9ef-4f61-b3fe-e5b92b7ee27b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.70:8000/ready\": context deadline exceeded" Nov 03 00:53:48.236233 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:53:48.231754 4939 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-88bbf7/simulator-w1-88bbf7-kserve-5fbb669f8f-42hgd" podUID="83b38883-2e7a-444d-aa68-52be4a25ae66" containerName="main" probeResult="failure" output="Get \"https://10.132.0.62:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 00:53:48.334984 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:53:48.313269 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 00:53:48.334984 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:53:48.313755 4939 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-f7680d/distinct-2-w2-f7680d-kserve-7d7c4564bf-cbv2c" podUID="2de18086-761b-4773-b9fc-15ddc4a220fc" containerName="main" probeResult="failure" output="Get \"https://10.132.0.75:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 00:53:48.493935 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:53:48.468591 4939 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-f7680d/distinct-2-w2-f7680d-kserve-7d7c4564bf-cbv2c" Nov 03 00:53:48.538579 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:53:48.479476 4939 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-2f9a3a/simulator-w0-2f9a3a-kserve-9596bbb8f-2trgz" podUID="207e2207-fb6c-4076-95df-72e95671c261" containerName="main" probeResult="failure" output="Get \"https://10.132.0.61:8000/ready\": context deadline exceeded" Nov 03 00:53:48.539439 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:53:48.539415 4939 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-2f9a3a/simulator-w0-2f9a3a-kserve-9596bbb8f-2trgz" Nov 03 00:53:48.544864 ip-10-0-140-107 kubenswrapper[4939]: W1103 00:53:48.543338 4939 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4adcd893_4e34_4860_bafb_a32ee8b956f0.slice/crio-dc4ec1e404447874693445ec056c5a2049949417ecf31cb0dbd4a0b470bebccb WatchSource:0}: Error finding container dc4ec1e404447874693445ec056c5a2049949417ecf31cb0dbd4a0b470bebccb: Status 404 returned error can't find the container with id dc4ec1e404447874693445ec056c5a2049949417ecf31cb0dbd4a0b470bebccb Nov 03 00:53:48.584963 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:53:48.584873 4939 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 00:53:48.585734 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:53:48.585660 4939 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 00:53:48.594100 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:53:48.594058 4939 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 00:53:48.634106 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:53:48.629538 4939 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-88bbf7/simulator-w1-88bbf7-kserve-5fbb669f8f-42hgd" podUID="83b38883-2e7a-444d-aa68-52be4a25ae66" containerName="main" probeResult="failure" output="Get \"https://10.132.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 00:53:48.635991 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:53:48.635958 4939 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-83755a-openshift-default-c86f9565c-nzmzn"] Nov 03 00:53:48.636371 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:53:48.636348 4939 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-88bbf7/simulator-w1-88bbf7-kserve-5fbb669f8f-42hgd" Nov 03 00:53:48.957928 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:53:48.957532 4939 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 00:53:48.958109 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:53:48.957968 4939 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 00:53:48.959465 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:53:48.958228 4939 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 00:53:49.056933 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:53:49.056425 4939 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-f7680d/premium-w2-f7680d-kserve-76565b8cf9-hqbbt" podUID="4712e708-18a2-4a13-a12c-a5669397b2c3" containerName="main" probeResult="failure" output="Get \"https://10.132.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 00:53:49.339938 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:53:49.327064 4939 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-f7680d/distinct-2-w2-f7680d-kserve-7d7c4564bf-cbv2c" podUID="2de18086-761b-4773-b9fc-15ddc4a220fc" containerName="main" probeResult="failure" output="Get \"https://10.132.0.75:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 00:53:49.339938 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:53:49.337899 4939 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-f7680d/premium-w2-f7680d-kserve-76565b8cf9-hqbbt" Nov 03 00:53:49.339938 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:53:49.338963 4939 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-83755a/distinct-2-w3-83755a-kserve-6cf66b995d-2l47b" podUID="c3b05e88-5d5b-4d03-b28d-832f22e3c868" containerName="main" probeResult="failure" output="Get \"https://10.132.0.72:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 00:53:49.375929 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:53:49.374710 4939 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-83755a/distinct-2-w3-83755a-kserve-6cf66b995d-2l47b" Nov 03 00:53:49.524457 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:53:49.496616 4939 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-83755a-openshift-default-c86f9565c-rwxw4"] Nov 03 00:53:49.595930 ip-10-0-140-107 kubenswrapper[4939]: E1103 00:53:49.580669 4939 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-sthjh" containerName="extractor" Nov 03 00:53:49.616209 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:53:49.597641 4939 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-88bbf7/simulator-w1-88bbf7-kserve-5fbb669f8f-42hgd" Nov 03 00:53:49.616979 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:53:49.616945 4939 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-2f9a3a/simulator-w0-2f9a3a-kserve-9596bbb8f-2trgz" podUID="207e2207-fb6c-4076-95df-72e95671c261" containerName="main" probeResult="failure" output="Get \"https://10.132.0.61:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Nov 03 00:53:49.647535 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:53:49.647506 4939 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-f7680d/premium-w2-f7680d-kserve-76565b8cf9-hqbbt" Nov 03 00:53:49.680129 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:53:49.680100 4939 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-83755a/distinct-2-w3-83755a-kserve-6cf66b995d-2l47b" Nov 03 00:53:49.680493 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:53:49.680448 4939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-83755a-openshift-default-c86f9565c-rwxw4" event={"ID":"4adcd893-4e34-4860-bafb-a32ee8b956f0","Type":"ContainerStarted","Data":"dc4ec1e404447874693445ec056c5a2049949417ecf31cb0dbd4a0b470bebccb"} Nov 03 00:53:49.724945 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:53:49.718174 4939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-83755a-openshift-default-c86f9565c-nzmzn" event={"ID":"44fe729d-d7d1-41c2-99df-60e955909b8b","Type":"ContainerStarted","Data":"0d7a5a36d2f696ca5bb42ac53e4f8a2d178d8b5ba57e7098cdd74fe68c23d412"} Nov 03 00:53:49.850401 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:53:49.850370 4939 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-2f9a3a/simulator-w0-2f9a3a-kserve-9596bbb8f-2trgz" Nov 03 00:53:49.911180 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:53:49.911144 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 00:53:49.925689 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:53:49.925657 4939 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-f7680d/distinct-2-w2-f7680d-kserve-7d7c4564bf-cbv2c" Nov 03 00:53:50.727400 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:53:50.727361 4939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-83755a-openshift-default-c86f9565c-nzmzn" event={"ID":"44fe729d-d7d1-41c2-99df-60e955909b8b","Type":"ContainerStarted","Data":"74d2ce58c7345c8b44395dda007f1db79217e50d329cd8e7791d0422eed2d738"} Nov 03 00:53:50.729226 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:53:50.729200 4939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-83755a-openshift-default-c86f9565c-rwxw4" event={"ID":"4adcd893-4e34-4860-bafb-a32ee8b956f0","Type":"ContainerStarted","Data":"d63e6ba311f7c32592f99e41dd118860a99b3a9e540667037881f767af6eec95"} Nov 03 00:53:50.774934 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:53:50.753464 4939 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-83755a-openshift-default-c86f9565c-nzmzn" podStartSLOduration=5.753443055 podStartE2EDuration="5.753443055s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:53:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:53:50.750101322 +0000 UTC m=+779.014436270" watchObservedRunningTime="2026-11-03 00:53:50.753443055 +0000 UTC m=+779.017778002" Nov 03 00:53:50.778457 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:53:50.778392 4939 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-83755a-openshift-default-c86f9565c-rwxw4" podStartSLOduration=5.778370924 podStartE2EDuration="5.778370924s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:53:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:53:50.777405064 +0000 UTC m=+779.041740017" watchObservedRunningTime="2026-11-03 00:53:50.778370924 +0000 UTC m=+779.042705869" Nov 03 00:53:50.939361 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:53:50.939329 4939 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-83755a-openshift-default-c86f9565c-nzmzn" Nov 03 00:53:51.225737 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:53:51.225673 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 00:53:51.684933 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:53:51.684100 4939 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-83755a-openshift-default-c86f9565c-rwxw4" Nov 03 00:53:51.684933 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:53:51.684247 4939 patch_prober.go:29] interesting pod/e2e-worker-w3-83755a-openshift-default-c86f9565c-rwxw4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.77:15021/healthz/ready\": dial tcp 10.132.0.77:15021: connect: connection refused" start-of-body= Nov 03 00:53:51.684933 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:53:51.684293 4939 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-83755a-openshift-default-c86f9565c-rwxw4" podUID="4adcd893-4e34-4860-bafb-a32ee8b956f0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.77:15021/healthz/ready\": dial tcp 10.132.0.77:15021: connect: connection refused" Nov 03 00:53:51.941556 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:53:51.941467 4939 patch_prober.go:29] interesting pod/e2e-worker-w3-83755a-openshift-default-c86f9565c-nzmzn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.78:15021/healthz/ready\": dial tcp 10.132.0.78:15021: connect: connection refused" start-of-body= Nov 03 00:53:51.941556 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:53:51.941527 4939 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-83755a-openshift-default-c86f9565c-nzmzn" podUID="44fe729d-d7d1-41c2-99df-60e955909b8b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.78:15021/healthz/ready\": dial tcp 10.132.0.78:15021: connect: connection refused" Nov 03 00:53:51.975146 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:53:51.969837 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 00:53:52.674825 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:53:52.674787 4939 patch_prober.go:29] interesting pod/e2e-worker-w3-83755a-openshift-default-c86f9565c-rwxw4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.77:15021/healthz/ready\": dial tcp 10.132.0.77:15021: connect: connection refused" start-of-body= Nov 03 00:53:52.675152 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:53:52.675119 4939 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-83755a-openshift-default-c86f9565c-rwxw4" podUID="4adcd893-4e34-4860-bafb-a32ee8b956f0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.77:15021/healthz/ready\": dial tcp 10.132.0.77:15021: connect: connection refused" Nov 03 00:53:52.745577 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:53:52.745548 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 00:53:52.940285 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:53:52.940203 4939 patch_prober.go:29] interesting pod/e2e-worker-w3-83755a-openshift-default-c86f9565c-nzmzn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.78:15021/healthz/ready\": dial tcp 10.132.0.78:15021: connect: connection refused" start-of-body= Nov 03 00:53:52.940285 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:53:52.940251 4939 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-83755a-openshift-default-c86f9565c-nzmzn" podUID="44fe729d-d7d1-41c2-99df-60e955909b8b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.78:15021/healthz/ready\": dial tcp 10.132.0.78:15021: connect: connection refused" Nov 03 00:53:53.230037 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:53:53.229945 4939 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w0-2f9a3a/distinct-w0-2f9a3a-kserve-555565cfcf-jtcr9" Nov 03 00:53:53.243881 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:53:53.243852 4939 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w0-2f9a3a/distinct-w0-2f9a3a-kserve-555565cfcf-jtcr9" Nov 03 00:53:53.631967 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:53:53.631895 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 00:53:53.655346 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:53:53.655305 4939 patch_prober.go:29] interesting pod/e2e-worker-w3-83755a-openshift-default-c86f9565c-rwxw4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.77:15021/healthz/ready\": dial tcp 10.132.0.77:15021: connect: connection refused" start-of-body= Nov 03 00:53:53.655611 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:53:53.655583 4939 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-83755a-openshift-default-c86f9565c-rwxw4" podUID="4adcd893-4e34-4860-bafb-a32ee8b956f0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.77:15021/healthz/ready\": dial tcp 10.132.0.77:15021: connect: connection refused" Nov 03 00:53:53.918056 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:53:53.917978 4939 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-2f9a3a/distinct-w0-2f9a3a-kserve-555565cfcf-jtcr9" Nov 03 00:53:53.940174 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:53:53.940143 4939 patch_prober.go:29] interesting pod/e2e-worker-w3-83755a-openshift-default-c86f9565c-nzmzn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.78:15021/healthz/ready\": dial tcp 10.132.0.78:15021: connect: connection refused" start-of-body= Nov 03 00:53:53.940676 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:53:53.940360 4939 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-83755a-openshift-default-c86f9565c-nzmzn" podUID="44fe729d-d7d1-41c2-99df-60e955909b8b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.78:15021/healthz/ready\": dial tcp 10.132.0.78:15021: connect: connection refused" Nov 03 00:53:54.101540 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:53:54.101482 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 00:53:54.643865 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:53:54.643820 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 00:53:54.657933 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:53:54.655109 4939 patch_prober.go:29] interesting pod/e2e-worker-w3-83755a-openshift-default-c86f9565c-rwxw4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.77:15021/healthz/ready\": dial tcp 10.132.0.77:15021: connect: connection refused" start-of-body= Nov 03 00:53:54.657933 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:53:54.655182 4939 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-83755a-openshift-default-c86f9565c-rwxw4" podUID="4adcd893-4e34-4860-bafb-a32ee8b956f0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.77:15021/healthz/ready\": dial tcp 10.132.0.77:15021: connect: connection refused" Nov 03 00:53:54.940318 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:53:54.940233 4939 patch_prober.go:29] interesting pod/e2e-worker-w3-83755a-openshift-default-c86f9565c-nzmzn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.78:15021/healthz/ready\": dial tcp 10.132.0.78:15021: connect: connection refused" start-of-body= Nov 03 00:53:54.940318 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:53:54.940298 4939 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-83755a-openshift-default-c86f9565c-nzmzn" podUID="44fe729d-d7d1-41c2-99df-60e955909b8b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.78:15021/healthz/ready\": dial tcp 10.132.0.78:15021: connect: connection refused" Nov 03 00:53:55.503007 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:53:55.502932 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 00:53:55.672930 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:53:55.672128 4939 patch_prober.go:29] interesting pod/e2e-worker-w3-83755a-openshift-default-c86f9565c-rwxw4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.77:15021/healthz/ready\": dial tcp 10.132.0.77:15021: connect: connection refused" start-of-body= Nov 03 00:53:55.672930 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:53:55.672211 4939 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-83755a-openshift-default-c86f9565c-rwxw4" podUID="4adcd893-4e34-4860-bafb-a32ee8b956f0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.77:15021/healthz/ready\": dial tcp 10.132.0.77:15021: connect: connection refused" Nov 03 00:53:55.939663 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:53:55.939629 4939 patch_prober.go:29] interesting pod/e2e-worker-w3-83755a-openshift-default-c86f9565c-nzmzn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.78:15021/healthz/ready\": dial tcp 10.132.0.78:15021: connect: connection refused" start-of-body= Nov 03 00:53:55.939843 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:53:55.939685 4939 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-83755a-openshift-default-c86f9565c-nzmzn" podUID="44fe729d-d7d1-41c2-99df-60e955909b8b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.78:15021/healthz/ready\": dial tcp 10.132.0.78:15021: connect: connection refused" Nov 03 00:53:56.387825 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:53:56.387783 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 00:53:56.654242 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:53:56.654151 4939 patch_prober.go:29] interesting pod/e2e-worker-w3-83755a-openshift-default-c86f9565c-rwxw4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.77:15021/healthz/ready\": dial tcp 10.132.0.77:15021: connect: connection refused" start-of-body= Nov 03 00:53:56.654242 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:53:56.654218 4939 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-83755a-openshift-default-c86f9565c-rwxw4" podUID="4adcd893-4e34-4860-bafb-a32ee8b956f0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.77:15021/healthz/ready\": dial tcp 10.132.0.77:15021: connect: connection refused" Nov 03 00:53:56.939768 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:53:56.939680 4939 patch_prober.go:29] interesting pod/e2e-worker-w3-83755a-openshift-default-c86f9565c-nzmzn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.78:15021/healthz/ready\": dial tcp 10.132.0.78:15021: connect: connection refused" start-of-body= Nov 03 00:53:56.939768 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:53:56.939731 4939 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-83755a-openshift-default-c86f9565c-nzmzn" podUID="44fe729d-d7d1-41c2-99df-60e955909b8b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.78:15021/healthz/ready\": dial tcp 10.132.0.78:15021: connect: connection refused" Nov 03 00:53:57.237462 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:53:57.237375 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 00:53:57.885765 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:53:57.885447 4939 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-83755a-openshift-default-c86f9565c-rwxw4" Nov 03 00:53:57.886182 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:53:57.886153 4939 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-83755a-openshift-default-c86f9565c-rwxw4" Nov 03 00:53:57.887258 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:53:57.887235 4939 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-83755a-openshift-default-c86f9565c-rwxw4" Nov 03 00:53:57.944439 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:53:57.944387 4939 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-83755a-openshift-default-c86f9565c-nzmzn" Nov 03 00:53:57.944933 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:53:57.944919 4939 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-83755a-openshift-default-c86f9565c-nzmzn" Nov 03 00:53:57.945892 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:53:57.945860 4939 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-83755a-openshift-default-c86f9565c-nzmzn" Nov 03 00:53:58.082115 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:53:58.082060 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 00:53:58.877866 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:53:58.877830 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 00:53:59.590555 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:53:59.590514 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 00:54:00.299074 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:54:00.299034 4939 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-f7680d-openshift-default-c4488d5c9-qstvv"] Nov 03 00:54:00.306376 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:54:00.306349 4939 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-f7680d-openshift-default-c4488d5c9-qstvv" Nov 03 00:54:00.318405 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:54:00.318129 4939 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-f7680d-openshift-default-c4488d5c9-qstvv"] Nov 03 00:54:00.351333 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:54:00.351205 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 00:54:00.453248 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:54:00.453209 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ccf63a28-a2d8-4686-8555-968e54429210-istio-data\") pod \"e2e-worker-w2-f7680d-openshift-default-c4488d5c9-qstvv\" (UID: \"ccf63a28-a2d8-4686-8555-968e54429210\") " pod="openshift-ingress/e2e-worker-w2-f7680d-openshift-default-c4488d5c9-qstvv" Nov 03 00:54:00.453436 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:54:00.453295 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ccf63a28-a2d8-4686-8555-968e54429210-istiod-ca-cert\") pod \"e2e-worker-w2-f7680d-openshift-default-c4488d5c9-qstvv\" (UID: \"ccf63a28-a2d8-4686-8555-968e54429210\") " pod="openshift-ingress/e2e-worker-w2-f7680d-openshift-default-c4488d5c9-qstvv" Nov 03 00:54:00.453436 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:54:00.453373 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ccf63a28-a2d8-4686-8555-968e54429210-workload-socket\") pod \"e2e-worker-w2-f7680d-openshift-default-c4488d5c9-qstvv\" (UID: \"ccf63a28-a2d8-4686-8555-968e54429210\") " pod="openshift-ingress/e2e-worker-w2-f7680d-openshift-default-c4488d5c9-qstvv" Nov 03 00:54:00.453436 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:54:00.453426 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2nvmc\" (UniqueName: \"kubernetes.io/projected/ccf63a28-a2d8-4686-8555-968e54429210-kube-api-access-2nvmc\") pod \"e2e-worker-w2-f7680d-openshift-default-c4488d5c9-qstvv\" (UID: \"ccf63a28-a2d8-4686-8555-968e54429210\") " pod="openshift-ingress/e2e-worker-w2-f7680d-openshift-default-c4488d5c9-qstvv" Nov 03 00:54:00.453578 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:54:00.453490 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ccf63a28-a2d8-4686-8555-968e54429210-istio-envoy\") pod \"e2e-worker-w2-f7680d-openshift-default-c4488d5c9-qstvv\" (UID: \"ccf63a28-a2d8-4686-8555-968e54429210\") " pod="openshift-ingress/e2e-worker-w2-f7680d-openshift-default-c4488d5c9-qstvv" Nov 03 00:54:00.453578 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:54:00.453508 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ccf63a28-a2d8-4686-8555-968e54429210-credential-socket\") pod \"e2e-worker-w2-f7680d-openshift-default-c4488d5c9-qstvv\" (UID: \"ccf63a28-a2d8-4686-8555-968e54429210\") " pod="openshift-ingress/e2e-worker-w2-f7680d-openshift-default-c4488d5c9-qstvv" Nov 03 00:54:00.453578 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:54:00.453529 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ccf63a28-a2d8-4686-8555-968e54429210-workload-certs\") pod \"e2e-worker-w2-f7680d-openshift-default-c4488d5c9-qstvv\" (UID: \"ccf63a28-a2d8-4686-8555-968e54429210\") " pod="openshift-ingress/e2e-worker-w2-f7680d-openshift-default-c4488d5c9-qstvv" Nov 03 00:54:00.453683 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:54:00.453659 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ccf63a28-a2d8-4686-8555-968e54429210-istio-token\") pod \"e2e-worker-w2-f7680d-openshift-default-c4488d5c9-qstvv\" (UID: \"ccf63a28-a2d8-4686-8555-968e54429210\") " pod="openshift-ingress/e2e-worker-w2-f7680d-openshift-default-c4488d5c9-qstvv" Nov 03 00:54:00.453729 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:54:00.453701 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ccf63a28-a2d8-4686-8555-968e54429210-istio-podinfo\") pod \"e2e-worker-w2-f7680d-openshift-default-c4488d5c9-qstvv\" (UID: \"ccf63a28-a2d8-4686-8555-968e54429210\") " pod="openshift-ingress/e2e-worker-w2-f7680d-openshift-default-c4488d5c9-qstvv" Nov 03 00:54:00.491385 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:54:00.491345 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 00:54:00.559835 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:54:00.559737 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ccf63a28-a2d8-4686-8555-968e54429210-istiod-ca-cert\") pod \"e2e-worker-w2-f7680d-openshift-default-c4488d5c9-qstvv\" (UID: \"ccf63a28-a2d8-4686-8555-968e54429210\") " pod="openshift-ingress/e2e-worker-w2-f7680d-openshift-default-c4488d5c9-qstvv" Nov 03 00:54:00.561114 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:54:00.561091 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ccf63a28-a2d8-4686-8555-968e54429210-workload-socket\") pod \"e2e-worker-w2-f7680d-openshift-default-c4488d5c9-qstvv\" (UID: \"ccf63a28-a2d8-4686-8555-968e54429210\") " pod="openshift-ingress/e2e-worker-w2-f7680d-openshift-default-c4488d5c9-qstvv" Nov 03 00:54:00.562013 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:54:00.561992 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2nvmc\" (UniqueName: \"kubernetes.io/projected/ccf63a28-a2d8-4686-8555-968e54429210-kube-api-access-2nvmc\") pod \"e2e-worker-w2-f7680d-openshift-default-c4488d5c9-qstvv\" (UID: \"ccf63a28-a2d8-4686-8555-968e54429210\") " pod="openshift-ingress/e2e-worker-w2-f7680d-openshift-default-c4488d5c9-qstvv" Nov 03 00:54:00.562569 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:54:00.562549 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ccf63a28-a2d8-4686-8555-968e54429210-istio-envoy\") pod \"e2e-worker-w2-f7680d-openshift-default-c4488d5c9-qstvv\" (UID: \"ccf63a28-a2d8-4686-8555-968e54429210\") " pod="openshift-ingress/e2e-worker-w2-f7680d-openshift-default-c4488d5c9-qstvv" Nov 03 00:54:00.563091 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:54:00.563072 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ccf63a28-a2d8-4686-8555-968e54429210-credential-socket\") pod \"e2e-worker-w2-f7680d-openshift-default-c4488d5c9-qstvv\" (UID: \"ccf63a28-a2d8-4686-8555-968e54429210\") " pod="openshift-ingress/e2e-worker-w2-f7680d-openshift-default-c4488d5c9-qstvv" Nov 03 00:54:00.563583 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:54:00.563566 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ccf63a28-a2d8-4686-8555-968e54429210-workload-certs\") pod \"e2e-worker-w2-f7680d-openshift-default-c4488d5c9-qstvv\" (UID: \"ccf63a28-a2d8-4686-8555-968e54429210\") " pod="openshift-ingress/e2e-worker-w2-f7680d-openshift-default-c4488d5c9-qstvv" Nov 03 00:54:00.564086 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:54:00.564067 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ccf63a28-a2d8-4686-8555-968e54429210-istio-token\") pod \"e2e-worker-w2-f7680d-openshift-default-c4488d5c9-qstvv\" (UID: \"ccf63a28-a2d8-4686-8555-968e54429210\") " pod="openshift-ingress/e2e-worker-w2-f7680d-openshift-default-c4488d5c9-qstvv" Nov 03 00:54:00.564745 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:54:00.564718 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ccf63a28-a2d8-4686-8555-968e54429210-istio-podinfo\") pod \"e2e-worker-w2-f7680d-openshift-default-c4488d5c9-qstvv\" (UID: \"ccf63a28-a2d8-4686-8555-968e54429210\") " pod="openshift-ingress/e2e-worker-w2-f7680d-openshift-default-c4488d5c9-qstvv" Nov 03 00:54:00.565005 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:54:00.564987 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ccf63a28-a2d8-4686-8555-968e54429210-istio-data\") pod \"e2e-worker-w2-f7680d-openshift-default-c4488d5c9-qstvv\" (UID: \"ccf63a28-a2d8-4686-8555-968e54429210\") " pod="openshift-ingress/e2e-worker-w2-f7680d-openshift-default-c4488d5c9-qstvv" Nov 03 00:54:00.565968 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:54:00.561018 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ccf63a28-a2d8-4686-8555-968e54429210-istiod-ca-cert\") pod \"e2e-worker-w2-f7680d-openshift-default-c4488d5c9-qstvv\" (UID: \"ccf63a28-a2d8-4686-8555-968e54429210\") " pod="openshift-ingress/e2e-worker-w2-f7680d-openshift-default-c4488d5c9-qstvv" Nov 03 00:54:00.566106 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:54:00.564618 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ccf63a28-a2d8-4686-8555-968e54429210-workload-certs\") pod \"e2e-worker-w2-f7680d-openshift-default-c4488d5c9-qstvv\" (UID: \"ccf63a28-a2d8-4686-8555-968e54429210\") " pod="openshift-ingress/e2e-worker-w2-f7680d-openshift-default-c4488d5c9-qstvv" Nov 03 00:54:00.566206 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:54:00.563515 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ccf63a28-a2d8-4686-8555-968e54429210-credential-socket\") pod \"e2e-worker-w2-f7680d-openshift-default-c4488d5c9-qstvv\" (UID: \"ccf63a28-a2d8-4686-8555-968e54429210\") " pod="openshift-ingress/e2e-worker-w2-f7680d-openshift-default-c4488d5c9-qstvv" Nov 03 00:54:00.566300 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:54:00.561517 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ccf63a28-a2d8-4686-8555-968e54429210-workload-socket\") pod \"e2e-worker-w2-f7680d-openshift-default-c4488d5c9-qstvv\" (UID: \"ccf63a28-a2d8-4686-8555-968e54429210\") " pod="openshift-ingress/e2e-worker-w2-f7680d-openshift-default-c4488d5c9-qstvv" Nov 03 00:54:00.566385 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:54:00.565745 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ccf63a28-a2d8-4686-8555-968e54429210-istio-data\") pod \"e2e-worker-w2-f7680d-openshift-default-c4488d5c9-qstvv\" (UID: \"ccf63a28-a2d8-4686-8555-968e54429210\") " pod="openshift-ingress/e2e-worker-w2-f7680d-openshift-default-c4488d5c9-qstvv" Nov 03 00:54:00.570246 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:54:00.570225 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ccf63a28-a2d8-4686-8555-968e54429210-istio-podinfo\") pod \"e2e-worker-w2-f7680d-openshift-default-c4488d5c9-qstvv\" (UID: \"ccf63a28-a2d8-4686-8555-968e54429210\") " pod="openshift-ingress/e2e-worker-w2-f7680d-openshift-default-c4488d5c9-qstvv" Nov 03 00:54:00.580490 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:54:00.580450 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ccf63a28-a2d8-4686-8555-968e54429210-istio-envoy\") pod \"e2e-worker-w2-f7680d-openshift-default-c4488d5c9-qstvv\" (UID: \"ccf63a28-a2d8-4686-8555-968e54429210\") " pod="openshift-ingress/e2e-worker-w2-f7680d-openshift-default-c4488d5c9-qstvv" Nov 03 00:54:00.582710 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:54:00.582490 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2nvmc\" (UniqueName: \"kubernetes.io/projected/ccf63a28-a2d8-4686-8555-968e54429210-kube-api-access-2nvmc\") pod \"e2e-worker-w2-f7680d-openshift-default-c4488d5c9-qstvv\" (UID: \"ccf63a28-a2d8-4686-8555-968e54429210\") " pod="openshift-ingress/e2e-worker-w2-f7680d-openshift-default-c4488d5c9-qstvv" Nov 03 00:54:00.584775 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:54:00.584751 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ccf63a28-a2d8-4686-8555-968e54429210-istio-token\") pod \"e2e-worker-w2-f7680d-openshift-default-c4488d5c9-qstvv\" (UID: \"ccf63a28-a2d8-4686-8555-968e54429210\") " pod="openshift-ingress/e2e-worker-w2-f7680d-openshift-default-c4488d5c9-qstvv" Nov 03 00:54:00.645935 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:54:00.639110 4939 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-4pw7v"] Nov 03 00:54:00.669081 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:54:00.669048 4939 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-f7680d-openshift-default-c4488d5c9-qstvv" Nov 03 00:54:00.968541 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:54:00.967801 4939 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-f7680d-openshift-default-c4488d5c9-qstvv"] Nov 03 00:54:00.971278 ip-10-0-140-107 kubenswrapper[4939]: W1103 00:54:00.971250 4939 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podccf63a28_a2d8_4686_8555_968e54429210.slice/crio-0ff751b36761ef753b4d27625c2cffe978c6f5af2a54a83e4b72ef7c76c9df97 WatchSource:0}: Error finding container 0ff751b36761ef753b4d27625c2cffe978c6f5af2a54a83e4b72ef7c76c9df97: Status 404 returned error can't find the container with id 0ff751b36761ef753b4d27625c2cffe978c6f5af2a54a83e4b72ef7c76c9df97 Nov 03 00:54:00.974782 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:54:00.974748 4939 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 00:54:00.974882 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:54:00.974837 4939 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 00:54:00.974985 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:54:00.974883 4939 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 00:54:01.195727 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:54:01.195670 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 00:54:02.008933 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:54:02.008675 4939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-f7680d-openshift-default-c4488d5c9-qstvv" event={"ID":"ccf63a28-a2d8-4686-8555-968e54429210","Type":"ContainerStarted","Data":"5607cb9bdc1566797246d0558e8a8a2611dd45b7a214459c511d4a8025ac6e3b"} Nov 03 00:54:02.008933 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:54:02.008746 4939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-f7680d-openshift-default-c4488d5c9-qstvv" event={"ID":"ccf63a28-a2d8-4686-8555-968e54429210","Type":"ContainerStarted","Data":"0ff751b36761ef753b4d27625c2cffe978c6f5af2a54a83e4b72ef7c76c9df97"} Nov 03 00:54:02.064929 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:54:02.064236 4939 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-f7680d-openshift-default-c4488d5c9-qstvv" podStartSLOduration=2.06421522 podStartE2EDuration="2.06421522s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:54:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:54:02.036488614 +0000 UTC m=+790.300823562" watchObservedRunningTime="2026-11-03 00:54:02.06421522 +0000 UTC m=+790.328550163" Nov 03 00:54:02.681944 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:54:02.679593 4939 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-f7680d-openshift-default-c4488d5c9-qstvv" Nov 03 00:54:02.681944 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:54:02.679802 4939 patch_prober.go:29] interesting pod/e2e-worker-w2-f7680d-openshift-default-c4488d5c9-qstvv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.79:15021/healthz/ready\": dial tcp 10.132.0.79:15021: connect: connection refused" start-of-body= Nov 03 00:54:02.681944 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:54:02.679851 4939 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-f7680d-openshift-default-c4488d5c9-qstvv" podUID="ccf63a28-a2d8-4686-8555-968e54429210" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.79:15021/healthz/ready\": dial tcp 10.132.0.79:15021: connect: connection refused" Nov 03 00:54:02.862946 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:54:02.862131 4939 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-2f9a3a/distinct-w0-2f9a3a-kserve-555565cfcf-jtcr9" podUID="ac0ece06-8718-463f-915f-5315988d9711" containerName="main" probeResult="failure" output="Get \"https://10.132.0.76:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 00:54:03.048713 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:54:03.048615 4939 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-f7680d/distinct-w2-f7680d-kserve-756f874b98-qnpv8" podUID="836753fc-f9ef-4f61-b3fe-e5b92b7ee27b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.70:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 00:54:03.617003 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:54:03.616369 4939 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-88bbf7/simulator-w1-88bbf7-kserve-5fbb669f8f-42hgd" podUID="83b38883-2e7a-444d-aa68-52be4a25ae66" containerName="main" probeResult="failure" output="Get \"https://10.132.0.62:8000/ready\": context deadline exceeded" Nov 03 00:54:03.655220 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:54:03.654001 4939 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-f7680d/premium-w2-f7680d-kserve-76565b8cf9-hqbbt" podUID="4712e708-18a2-4a13-a12c-a5669397b2c3" containerName="main" probeResult="failure" output="Get \"https://10.132.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 00:54:03.666956 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:54:03.666224 4939 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-83755a/distinct-2-w3-83755a-kserve-6cf66b995d-2l47b" podUID="c3b05e88-5d5b-4d03-b28d-832f22e3c868" containerName="main" probeResult="failure" output="Get \"https://10.132.0.72:8000/ready\": context deadline exceeded" Nov 03 00:54:03.679361 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:54:03.679323 4939 patch_prober.go:29] interesting pod/e2e-worker-w2-f7680d-openshift-default-c4488d5c9-qstvv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.79:15021/healthz/ready\": dial tcp 10.132.0.79:15021: connect: connection refused" start-of-body= Nov 03 00:54:03.679636 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:54:03.679600 4939 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-f7680d-openshift-default-c4488d5c9-qstvv" podUID="ccf63a28-a2d8-4686-8555-968e54429210" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.79:15021/healthz/ready\": dial tcp 10.132.0.79:15021: connect: connection refused" Nov 03 00:54:03.863096 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:54:03.863042 4939 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-2f9a3a/distinct-w0-2f9a3a-kserve-555565cfcf-jtcr9" podUID="ac0ece06-8718-463f-915f-5315988d9711" containerName="main" probeResult="failure" output="Get \"https://10.132.0.76:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 00:54:03.863758 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:54:03.863726 4939 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-2f9a3a/simulator-w0-2f9a3a-kserve-9596bbb8f-2trgz" podUID="207e2207-fb6c-4076-95df-72e95671c261" containerName="main" probeResult="failure" output="Get \"https://10.132.0.61:8000/ready\": context deadline exceeded" Nov 03 00:54:03.864749 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:54:03.864728 4939 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-2f9a3a/distinct-w0-2f9a3a-kserve-555565cfcf-jtcr9" Nov 03 00:54:03.935160 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:54:03.935059 4939 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-f7680d/distinct-2-w2-f7680d-kserve-7d7c4564bf-cbv2c" podUID="2de18086-761b-4773-b9fc-15ddc4a220fc" containerName="main" probeResult="failure" output="Get \"https://10.132.0.75:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 00:54:04.175938 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:54:04.175642 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 00:54:04.273047 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:54:04.270553 4939 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-2f9a3a/distinct-w0-2f9a3a-kserve-555565cfcf-jtcr9" podUID="ac0ece06-8718-463f-915f-5315988d9711" containerName="main" probeResult="failure" output="Get \"https://10.132.0.76:8000/health\": context deadline exceeded" Nov 03 00:54:04.632315 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:54:04.632207 4939 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-88bbf7/simulator-w1-88bbf7-kserve-5fbb669f8f-42hgd" podUID="83b38883-2e7a-444d-aa68-52be4a25ae66" containerName="main" probeResult="failure" output="Get \"https://10.132.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 00:54:04.697969 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:54:04.686414 4939 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-f7680d/premium-w2-f7680d-kserve-76565b8cf9-hqbbt" podUID="4712e708-18a2-4a13-a12c-a5669397b2c3" containerName="main" probeResult="failure" output="Get \"https://10.132.0.63:8000/ready\": context deadline exceeded" Nov 03 00:54:04.697969 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:54:04.688360 4939 patch_prober.go:29] interesting pod/e2e-worker-w2-f7680d-openshift-default-c4488d5c9-qstvv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.79:15021/healthz/ready\": dial tcp 10.132.0.79:15021: connect: connection refused" start-of-body= Nov 03 00:54:04.697969 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:54:04.688416 4939 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-f7680d-openshift-default-c4488d5c9-qstvv" podUID="ccf63a28-a2d8-4686-8555-968e54429210" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.79:15021/healthz/ready\": dial tcp 10.132.0.79:15021: connect: connection refused" Nov 03 00:54:04.726932 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:54:04.725853 4939 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-88bbf7/simulator-w1-88bbf7-kserve-5fbb669f8f-42hgd" Nov 03 00:54:04.726932 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:54:04.725940 4939 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-f7680d/premium-w2-f7680d-kserve-76565b8cf9-hqbbt" Nov 03 00:54:04.865039 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:54:04.864994 4939 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-2f9a3a/simulator-w0-2f9a3a-kserve-9596bbb8f-2trgz" podUID="207e2207-fb6c-4076-95df-72e95671c261" containerName="main" probeResult="failure" output="Get \"https://10.132.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 00:54:04.871016 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:54:04.870973 4939 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-2f9a3a/distinct-w0-2f9a3a-kserve-555565cfcf-jtcr9" podUID="ac0ece06-8718-463f-915f-5315988d9711" containerName="main" probeResult="failure" output="Get \"https://10.132.0.76:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 00:54:04.903991 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:54:04.879767 4939 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-2f9a3a/simulator-w0-2f9a3a-kserve-9596bbb8f-2trgz" Nov 03 00:54:04.998936 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:54:04.990123 4939 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-f7680d/distinct-2-w2-f7680d-kserve-7d7c4564bf-cbv2c" podUID="2de18086-761b-4773-b9fc-15ddc4a220fc" containerName="main" probeResult="failure" output="Get \"https://10.132.0.75:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 00:54:05.015938 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:54:05.015648 4939 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-f7680d/distinct-2-w2-f7680d-kserve-7d7c4564bf-cbv2c" Nov 03 00:54:05.069657 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:54:05.067925 4939 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-f7680d/distinct-w2-f7680d-kserve-756f874b98-qnpv8" podUID="836753fc-f9ef-4f61-b3fe-e5b92b7ee27b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.70:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 00:54:05.634557 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:54:05.634514 4939 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-88bbf7/simulator-w1-88bbf7-kserve-5fbb669f8f-42hgd" podUID="83b38883-2e7a-444d-aa68-52be4a25ae66" containerName="main" probeResult="failure" output="Get \"https://10.132.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 00:54:05.671930 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:54:05.671313 4939 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-83755a/distinct-2-w3-83755a-kserve-6cf66b995d-2l47b" podUID="c3b05e88-5d5b-4d03-b28d-832f22e3c868" containerName="main" probeResult="failure" output="Get \"https://10.132.0.72:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 00:54:05.708946 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:54:05.691269 4939 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-f7680d/premium-w2-f7680d-kserve-76565b8cf9-hqbbt" podUID="4712e708-18a2-4a13-a12c-a5669397b2c3" containerName="main" probeResult="failure" output="Get \"https://10.132.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 00:54:05.719683 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:54:05.719079 4939 patch_prober.go:29] interesting pod/e2e-worker-w2-f7680d-openshift-default-c4488d5c9-qstvv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.79:15021/healthz/ready\": dial tcp 10.132.0.79:15021: connect: connection refused" start-of-body= Nov 03 00:54:05.719683 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:54:05.719143 4939 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-f7680d-openshift-default-c4488d5c9-qstvv" podUID="ccf63a28-a2d8-4686-8555-968e54429210" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.79:15021/healthz/ready\": dial tcp 10.132.0.79:15021: connect: connection refused" Nov 03 00:54:05.767009 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:54:05.766290 4939 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-f7680d/distinct-2-w2-f7680d-kserve-7d7c4564bf-cbv2c" Nov 03 00:54:05.811110 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:54:05.811078 4939 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-f7680d/premium-w2-f7680d-kserve-76565b8cf9-hqbbt" Nov 03 00:54:05.849971 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:54:05.846503 4939 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-2f9a3a/simulator-w0-2f9a3a-kserve-9596bbb8f-2trgz" Nov 03 00:54:05.897159 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:54:05.893111 4939 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-2f9a3a/distinct-w0-2f9a3a-kserve-555565cfcf-jtcr9" podUID="ac0ece06-8718-463f-915f-5315988d9711" containerName="main" probeResult="failure" output="Get \"https://10.132.0.76:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 00:54:05.916939 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:54:05.916877 4939 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-88bbf7/simulator-w1-88bbf7-kserve-5fbb669f8f-42hgd" Nov 03 00:54:05.933433 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:54:05.933407 4939 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-2f9a3a/distinct-w0-2f9a3a-kserve-555565cfcf-jtcr9" Nov 03 00:54:06.033871 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:54:06.033787 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 00:54:06.671442 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:54:06.671310 4939 patch_prober.go:29] interesting pod/e2e-worker-w2-f7680d-openshift-default-c4488d5c9-qstvv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.79:15021/healthz/ready\": dial tcp 10.132.0.79:15021: connect: connection refused" start-of-body= Nov 03 00:54:06.671442 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:54:06.671383 4939 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-f7680d-openshift-default-c4488d5c9-qstvv" podUID="ccf63a28-a2d8-4686-8555-968e54429210" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.79:15021/healthz/ready\": dial tcp 10.132.0.79:15021: connect: connection refused" Nov 03 00:54:07.118466 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:54:07.118432 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 00:54:07.671406 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:54:07.671370 4939 patch_prober.go:29] interesting pod/e2e-worker-w2-f7680d-openshift-default-c4488d5c9-qstvv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.79:15021/healthz/ready\": dial tcp 10.132.0.79:15021: connect: connection refused" start-of-body= Nov 03 00:54:07.671740 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:54:07.671705 4939 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-f7680d-openshift-default-c4488d5c9-qstvv" podUID="ccf63a28-a2d8-4686-8555-968e54429210" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.79:15021/healthz/ready\": dial tcp 10.132.0.79:15021: connect: connection refused" Nov 03 00:54:07.835063 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:54:07.835030 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 00:54:08.600015 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:54:08.599977 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 00:54:08.680930 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:54:08.674251 4939 patch_prober.go:29] interesting pod/e2e-worker-w2-f7680d-openshift-default-c4488d5c9-qstvv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.79:15021/healthz/ready\": dial tcp 10.132.0.79:15021: connect: connection refused" start-of-body= Nov 03 00:54:08.708085 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:54:08.708016 4939 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-f7680d-openshift-default-c4488d5c9-qstvv" podUID="ccf63a28-a2d8-4686-8555-968e54429210" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.79:15021/healthz/ready\": dial tcp 10.132.0.79:15021: connect: connection refused" Nov 03 00:54:09.402657 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:54:09.402617 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 00:54:09.671457 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:54:09.671344 4939 patch_prober.go:29] interesting pod/e2e-worker-w2-f7680d-openshift-default-c4488d5c9-qstvv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.79:15021/healthz/ready\": dial tcp 10.132.0.79:15021: connect: connection refused" start-of-body= Nov 03 00:54:09.671457 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:54:09.671419 4939 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-f7680d-openshift-default-c4488d5c9-qstvv" podUID="ccf63a28-a2d8-4686-8555-968e54429210" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.79:15021/healthz/ready\": dial tcp 10.132.0.79:15021: connect: connection refused" Nov 03 00:54:10.086413 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:54:10.086383 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 00:54:10.670232 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:54:10.670196 4939 patch_prober.go:29] interesting pod/e2e-worker-w2-f7680d-openshift-default-c4488d5c9-qstvv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.79:15021/healthz/ready\": dial tcp 10.132.0.79:15021: connect: connection refused" start-of-body= Nov 03 00:54:10.670403 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:54:10.670262 4939 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-f7680d-openshift-default-c4488d5c9-qstvv" podUID="ccf63a28-a2d8-4686-8555-968e54429210" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.79:15021/healthz/ready\": dial tcp 10.132.0.79:15021: connect: connection refused" Nov 03 00:54:10.773851 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:54:10.773813 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 00:54:11.605987 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:54:11.605944 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 00:54:11.686854 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:54:11.686065 4939 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-f7680d-openshift-default-c4488d5c9-qstvv" Nov 03 00:54:11.687544 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:54:11.687522 4939 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-f7680d-openshift-default-c4488d5c9-qstvv" Nov 03 00:54:11.688784 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:54:11.688761 4939 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-f7680d-openshift-default-c4488d5c9-qstvv" Nov 03 00:54:11.925846 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:54:11.925767 4939 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-650be0-openshift-default-5c7c5fb7d9-84qlq"] Nov 03 00:54:11.930095 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:54:11.930072 4939 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-650be0-openshift-default-5c7c5fb7d9-84qlq" Nov 03 00:54:11.951252 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:54:11.951222 4939 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-650be0-openshift-default-5c7c5fb7d9-84qlq"] Nov 03 00:54:11.968938 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:54:11.968880 4939 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-650be0-openshift-default-5c7c5fb7d9-267h5"] Nov 03 00:54:11.973212 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:54:11.973185 4939 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-650be0-openshift-default-5c7c5fb7d9-267h5" Nov 03 00:54:11.984482 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:54:11.984452 4939 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-650be0-openshift-default-5c7c5fb7d9-267h5"] Nov 03 00:54:12.022856 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:54:12.022824 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d2dabe94-37c9-4a3f-bf9a-15e4f53e05d0-istiod-ca-cert\") pod \"e2e-worker-w4-650be0-openshift-default-5c7c5fb7d9-84qlq\" (UID: \"d2dabe94-37c9-4a3f-bf9a-15e4f53e05d0\") " pod="openshift-ingress/e2e-worker-w4-650be0-openshift-default-5c7c5fb7d9-84qlq" Nov 03 00:54:12.023072 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:54:12.023030 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d2dabe94-37c9-4a3f-bf9a-15e4f53e05d0-istio-envoy\") pod \"e2e-worker-w4-650be0-openshift-default-5c7c5fb7d9-84qlq\" (UID: \"d2dabe94-37c9-4a3f-bf9a-15e4f53e05d0\") " pod="openshift-ingress/e2e-worker-w4-650be0-openshift-default-5c7c5fb7d9-84qlq" Nov 03 00:54:12.023149 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:54:12.023071 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d2dabe94-37c9-4a3f-bf9a-15e4f53e05d0-workload-socket\") pod \"e2e-worker-w4-650be0-openshift-default-5c7c5fb7d9-84qlq\" (UID: \"d2dabe94-37c9-4a3f-bf9a-15e4f53e05d0\") " pod="openshift-ingress/e2e-worker-w4-650be0-openshift-default-5c7c5fb7d9-84qlq" Nov 03 00:54:12.023334 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:54:12.023313 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5tj7h\" (UniqueName: \"kubernetes.io/projected/d2dabe94-37c9-4a3f-bf9a-15e4f53e05d0-kube-api-access-5tj7h\") pod \"e2e-worker-w4-650be0-openshift-default-5c7c5fb7d9-84qlq\" (UID: \"d2dabe94-37c9-4a3f-bf9a-15e4f53e05d0\") " pod="openshift-ingress/e2e-worker-w4-650be0-openshift-default-5c7c5fb7d9-84qlq" Nov 03 00:54:12.023448 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:54:12.023433 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d2dabe94-37c9-4a3f-bf9a-15e4f53e05d0-istio-data\") pod \"e2e-worker-w4-650be0-openshift-default-5c7c5fb7d9-84qlq\" (UID: \"d2dabe94-37c9-4a3f-bf9a-15e4f53e05d0\") " pod="openshift-ingress/e2e-worker-w4-650be0-openshift-default-5c7c5fb7d9-84qlq" Nov 03 00:54:12.023528 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:54:12.023515 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d2dabe94-37c9-4a3f-bf9a-15e4f53e05d0-workload-certs\") pod \"e2e-worker-w4-650be0-openshift-default-5c7c5fb7d9-84qlq\" (UID: \"d2dabe94-37c9-4a3f-bf9a-15e4f53e05d0\") " pod="openshift-ingress/e2e-worker-w4-650be0-openshift-default-5c7c5fb7d9-84qlq" Nov 03 00:54:12.023594 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:54:12.023552 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d2dabe94-37c9-4a3f-bf9a-15e4f53e05d0-istio-podinfo\") pod \"e2e-worker-w4-650be0-openshift-default-5c7c5fb7d9-84qlq\" (UID: \"d2dabe94-37c9-4a3f-bf9a-15e4f53e05d0\") " pod="openshift-ingress/e2e-worker-w4-650be0-openshift-default-5c7c5fb7d9-84qlq" Nov 03 00:54:12.023594 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:54:12.023573 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d2dabe94-37c9-4a3f-bf9a-15e4f53e05d0-istio-token\") pod \"e2e-worker-w4-650be0-openshift-default-5c7c5fb7d9-84qlq\" (UID: \"d2dabe94-37c9-4a3f-bf9a-15e4f53e05d0\") " pod="openshift-ingress/e2e-worker-w4-650be0-openshift-default-5c7c5fb7d9-84qlq" Nov 03 00:54:12.023712 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:54:12.023696 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d2dabe94-37c9-4a3f-bf9a-15e4f53e05d0-credential-socket\") pod \"e2e-worker-w4-650be0-openshift-default-5c7c5fb7d9-84qlq\" (UID: \"d2dabe94-37c9-4a3f-bf9a-15e4f53e05d0\") " pod="openshift-ingress/e2e-worker-w4-650be0-openshift-default-5c7c5fb7d9-84qlq" Nov 03 00:54:12.125175 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:54:12.125122 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3c4531e2-d1cc-430f-8245-f129e486d172-istiod-ca-cert\") pod \"e2e-worker-w4-650be0-openshift-default-5c7c5fb7d9-267h5\" (UID: \"3c4531e2-d1cc-430f-8245-f129e486d172\") " pod="openshift-ingress/e2e-worker-w4-650be0-openshift-default-5c7c5fb7d9-267h5" Nov 03 00:54:12.125379 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:54:12.125183 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d2dabe94-37c9-4a3f-bf9a-15e4f53e05d0-workload-certs\") pod \"e2e-worker-w4-650be0-openshift-default-5c7c5fb7d9-84qlq\" (UID: \"d2dabe94-37c9-4a3f-bf9a-15e4f53e05d0\") " pod="openshift-ingress/e2e-worker-w4-650be0-openshift-default-5c7c5fb7d9-84qlq" Nov 03 00:54:12.125379 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:54:12.125271 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3c4531e2-d1cc-430f-8245-f129e486d172-istio-token\") pod \"e2e-worker-w4-650be0-openshift-default-5c7c5fb7d9-267h5\" (UID: \"3c4531e2-d1cc-430f-8245-f129e486d172\") " pod="openshift-ingress/e2e-worker-w4-650be0-openshift-default-5c7c5fb7d9-267h5" Nov 03 00:54:12.125379 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:54:12.125309 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d2dabe94-37c9-4a3f-bf9a-15e4f53e05d0-istio-podinfo\") pod \"e2e-worker-w4-650be0-openshift-default-5c7c5fb7d9-84qlq\" (UID: \"d2dabe94-37c9-4a3f-bf9a-15e4f53e05d0\") " pod="openshift-ingress/e2e-worker-w4-650be0-openshift-default-5c7c5fb7d9-84qlq" Nov 03 00:54:12.125379 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:54:12.125333 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d2dabe94-37c9-4a3f-bf9a-15e4f53e05d0-istio-token\") pod \"e2e-worker-w4-650be0-openshift-default-5c7c5fb7d9-84qlq\" (UID: \"d2dabe94-37c9-4a3f-bf9a-15e4f53e05d0\") " pod="openshift-ingress/e2e-worker-w4-650be0-openshift-default-5c7c5fb7d9-84qlq" Nov 03 00:54:12.125611 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:54:12.125434 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d2dabe94-37c9-4a3f-bf9a-15e4f53e05d0-credential-socket\") pod \"e2e-worker-w4-650be0-openshift-default-5c7c5fb7d9-84qlq\" (UID: \"d2dabe94-37c9-4a3f-bf9a-15e4f53e05d0\") " pod="openshift-ingress/e2e-worker-w4-650be0-openshift-default-5c7c5fb7d9-84qlq" Nov 03 00:54:12.125611 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:54:12.125471 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d2dabe94-37c9-4a3f-bf9a-15e4f53e05d0-istiod-ca-cert\") pod \"e2e-worker-w4-650be0-openshift-default-5c7c5fb7d9-84qlq\" (UID: \"d2dabe94-37c9-4a3f-bf9a-15e4f53e05d0\") " pod="openshift-ingress/e2e-worker-w4-650be0-openshift-default-5c7c5fb7d9-84qlq" Nov 03 00:54:12.125611 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:54:12.125549 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d2dabe94-37c9-4a3f-bf9a-15e4f53e05d0-istio-envoy\") pod \"e2e-worker-w4-650be0-openshift-default-5c7c5fb7d9-84qlq\" (UID: \"d2dabe94-37c9-4a3f-bf9a-15e4f53e05d0\") " pod="openshift-ingress/e2e-worker-w4-650be0-openshift-default-5c7c5fb7d9-84qlq" Nov 03 00:54:12.125611 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:54:12.125572 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d2dabe94-37c9-4a3f-bf9a-15e4f53e05d0-workload-socket\") pod \"e2e-worker-w4-650be0-openshift-default-5c7c5fb7d9-84qlq\" (UID: \"d2dabe94-37c9-4a3f-bf9a-15e4f53e05d0\") " pod="openshift-ingress/e2e-worker-w4-650be0-openshift-default-5c7c5fb7d9-84qlq" Nov 03 00:54:12.125611 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:54:12.125597 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3c4531e2-d1cc-430f-8245-f129e486d172-workload-certs\") pod \"e2e-worker-w4-650be0-openshift-default-5c7c5fb7d9-267h5\" (UID: \"3c4531e2-d1cc-430f-8245-f129e486d172\") " pod="openshift-ingress/e2e-worker-w4-650be0-openshift-default-5c7c5fb7d9-267h5" Nov 03 00:54:12.125873 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:54:12.125652 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-thsvt\" (UniqueName: \"kubernetes.io/projected/3c4531e2-d1cc-430f-8245-f129e486d172-kube-api-access-thsvt\") pod \"e2e-worker-w4-650be0-openshift-default-5c7c5fb7d9-267h5\" (UID: \"3c4531e2-d1cc-430f-8245-f129e486d172\") " pod="openshift-ingress/e2e-worker-w4-650be0-openshift-default-5c7c5fb7d9-267h5" Nov 03 00:54:12.125873 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:54:12.125653 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d2dabe94-37c9-4a3f-bf9a-15e4f53e05d0-workload-certs\") pod \"e2e-worker-w4-650be0-openshift-default-5c7c5fb7d9-84qlq\" (UID: \"d2dabe94-37c9-4a3f-bf9a-15e4f53e05d0\") " pod="openshift-ingress/e2e-worker-w4-650be0-openshift-default-5c7c5fb7d9-84qlq" Nov 03 00:54:12.125873 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:54:12.125730 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5tj7h\" (UniqueName: \"kubernetes.io/projected/d2dabe94-37c9-4a3f-bf9a-15e4f53e05d0-kube-api-access-5tj7h\") pod \"e2e-worker-w4-650be0-openshift-default-5c7c5fb7d9-84qlq\" (UID: \"d2dabe94-37c9-4a3f-bf9a-15e4f53e05d0\") " pod="openshift-ingress/e2e-worker-w4-650be0-openshift-default-5c7c5fb7d9-84qlq" Nov 03 00:54:12.125873 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:54:12.125750 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d2dabe94-37c9-4a3f-bf9a-15e4f53e05d0-credential-socket\") pod \"e2e-worker-w4-650be0-openshift-default-5c7c5fb7d9-84qlq\" (UID: \"d2dabe94-37c9-4a3f-bf9a-15e4f53e05d0\") " pod="openshift-ingress/e2e-worker-w4-650be0-openshift-default-5c7c5fb7d9-84qlq" Nov 03 00:54:12.125873 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:54:12.125806 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3c4531e2-d1cc-430f-8245-f129e486d172-credential-socket\") pod \"e2e-worker-w4-650be0-openshift-default-5c7c5fb7d9-267h5\" (UID: \"3c4531e2-d1cc-430f-8245-f129e486d172\") " pod="openshift-ingress/e2e-worker-w4-650be0-openshift-default-5c7c5fb7d9-267h5" Nov 03 00:54:12.125873 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:54:12.125857 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3c4531e2-d1cc-430f-8245-f129e486d172-istio-data\") pod \"e2e-worker-w4-650be0-openshift-default-5c7c5fb7d9-267h5\" (UID: \"3c4531e2-d1cc-430f-8245-f129e486d172\") " pod="openshift-ingress/e2e-worker-w4-650be0-openshift-default-5c7c5fb7d9-267h5" Nov 03 00:54:12.126202 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:54:12.125988 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3c4531e2-d1cc-430f-8245-f129e486d172-istio-envoy\") pod \"e2e-worker-w4-650be0-openshift-default-5c7c5fb7d9-267h5\" (UID: \"3c4531e2-d1cc-430f-8245-f129e486d172\") " pod="openshift-ingress/e2e-worker-w4-650be0-openshift-default-5c7c5fb7d9-267h5" Nov 03 00:54:12.126202 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:54:12.126038 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3c4531e2-d1cc-430f-8245-f129e486d172-workload-socket\") pod \"e2e-worker-w4-650be0-openshift-default-5c7c5fb7d9-267h5\" (UID: \"3c4531e2-d1cc-430f-8245-f129e486d172\") " pod="openshift-ingress/e2e-worker-w4-650be0-openshift-default-5c7c5fb7d9-267h5" Nov 03 00:54:12.126202 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:54:12.126069 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d2dabe94-37c9-4a3f-bf9a-15e4f53e05d0-workload-socket\") pod \"e2e-worker-w4-650be0-openshift-default-5c7c5fb7d9-84qlq\" (UID: \"d2dabe94-37c9-4a3f-bf9a-15e4f53e05d0\") " pod="openshift-ingress/e2e-worker-w4-650be0-openshift-default-5c7c5fb7d9-84qlq" Nov 03 00:54:12.126202 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:54:12.126115 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d2dabe94-37c9-4a3f-bf9a-15e4f53e05d0-istio-data\") pod \"e2e-worker-w4-650be0-openshift-default-5c7c5fb7d9-84qlq\" (UID: \"d2dabe94-37c9-4a3f-bf9a-15e4f53e05d0\") " pod="openshift-ingress/e2e-worker-w4-650be0-openshift-default-5c7c5fb7d9-84qlq" Nov 03 00:54:12.126202 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:54:12.126144 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3c4531e2-d1cc-430f-8245-f129e486d172-istio-podinfo\") pod \"e2e-worker-w4-650be0-openshift-default-5c7c5fb7d9-267h5\" (UID: \"3c4531e2-d1cc-430f-8245-f129e486d172\") " pod="openshift-ingress/e2e-worker-w4-650be0-openshift-default-5c7c5fb7d9-267h5" Nov 03 00:54:12.126472 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:54:12.126305 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d2dabe94-37c9-4a3f-bf9a-15e4f53e05d0-istiod-ca-cert\") pod \"e2e-worker-w4-650be0-openshift-default-5c7c5fb7d9-84qlq\" (UID: \"d2dabe94-37c9-4a3f-bf9a-15e4f53e05d0\") " pod="openshift-ingress/e2e-worker-w4-650be0-openshift-default-5c7c5fb7d9-84qlq" Nov 03 00:54:12.126472 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:54:12.126370 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d2dabe94-37c9-4a3f-bf9a-15e4f53e05d0-istio-data\") pod \"e2e-worker-w4-650be0-openshift-default-5c7c5fb7d9-84qlq\" (UID: \"d2dabe94-37c9-4a3f-bf9a-15e4f53e05d0\") " pod="openshift-ingress/e2e-worker-w4-650be0-openshift-default-5c7c5fb7d9-84qlq" Nov 03 00:54:12.128474 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:54:12.128446 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d2dabe94-37c9-4a3f-bf9a-15e4f53e05d0-istio-envoy\") pod \"e2e-worker-w4-650be0-openshift-default-5c7c5fb7d9-84qlq\" (UID: \"d2dabe94-37c9-4a3f-bf9a-15e4f53e05d0\") " pod="openshift-ingress/e2e-worker-w4-650be0-openshift-default-5c7c5fb7d9-84qlq" Nov 03 00:54:12.128648 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:54:12.128627 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d2dabe94-37c9-4a3f-bf9a-15e4f53e05d0-istio-podinfo\") pod \"e2e-worker-w4-650be0-openshift-default-5c7c5fb7d9-84qlq\" (UID: \"d2dabe94-37c9-4a3f-bf9a-15e4f53e05d0\") " pod="openshift-ingress/e2e-worker-w4-650be0-openshift-default-5c7c5fb7d9-84qlq" Nov 03 00:54:12.134518 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:54:12.134496 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5tj7h\" (UniqueName: \"kubernetes.io/projected/d2dabe94-37c9-4a3f-bf9a-15e4f53e05d0-kube-api-access-5tj7h\") pod \"e2e-worker-w4-650be0-openshift-default-5c7c5fb7d9-84qlq\" (UID: \"d2dabe94-37c9-4a3f-bf9a-15e4f53e05d0\") " pod="openshift-ingress/e2e-worker-w4-650be0-openshift-default-5c7c5fb7d9-84qlq" Nov 03 00:54:12.134622 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:54:12.134526 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d2dabe94-37c9-4a3f-bf9a-15e4f53e05d0-istio-token\") pod \"e2e-worker-w4-650be0-openshift-default-5c7c5fb7d9-84qlq\" (UID: \"d2dabe94-37c9-4a3f-bf9a-15e4f53e05d0\") " pod="openshift-ingress/e2e-worker-w4-650be0-openshift-default-5c7c5fb7d9-84qlq" Nov 03 00:54:12.227066 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:54:12.226975 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3c4531e2-d1cc-430f-8245-f129e486d172-istiod-ca-cert\") pod \"e2e-worker-w4-650be0-openshift-default-5c7c5fb7d9-267h5\" (UID: \"3c4531e2-d1cc-430f-8245-f129e486d172\") " pod="openshift-ingress/e2e-worker-w4-650be0-openshift-default-5c7c5fb7d9-267h5" Nov 03 00:54:12.227223 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:54:12.227128 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3c4531e2-d1cc-430f-8245-f129e486d172-istio-token\") pod \"e2e-worker-w4-650be0-openshift-default-5c7c5fb7d9-267h5\" (UID: \"3c4531e2-d1cc-430f-8245-f129e486d172\") " pod="openshift-ingress/e2e-worker-w4-650be0-openshift-default-5c7c5fb7d9-267h5" Nov 03 00:54:12.227309 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:54:12.227281 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3c4531e2-d1cc-430f-8245-f129e486d172-workload-certs\") pod \"e2e-worker-w4-650be0-openshift-default-5c7c5fb7d9-267h5\" (UID: \"3c4531e2-d1cc-430f-8245-f129e486d172\") " pod="openshift-ingress/e2e-worker-w4-650be0-openshift-default-5c7c5fb7d9-267h5" Nov 03 00:54:12.227391 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:54:12.227335 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-thsvt\" (UniqueName: \"kubernetes.io/projected/3c4531e2-d1cc-430f-8245-f129e486d172-kube-api-access-thsvt\") pod \"e2e-worker-w4-650be0-openshift-default-5c7c5fb7d9-267h5\" (UID: \"3c4531e2-d1cc-430f-8245-f129e486d172\") " pod="openshift-ingress/e2e-worker-w4-650be0-openshift-default-5c7c5fb7d9-267h5" Nov 03 00:54:12.227391 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:54:12.227378 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3c4531e2-d1cc-430f-8245-f129e486d172-credential-socket\") pod \"e2e-worker-w4-650be0-openshift-default-5c7c5fb7d9-267h5\" (UID: \"3c4531e2-d1cc-430f-8245-f129e486d172\") " pod="openshift-ingress/e2e-worker-w4-650be0-openshift-default-5c7c5fb7d9-267h5" Nov 03 00:54:12.227510 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:54:12.227430 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3c4531e2-d1cc-430f-8245-f129e486d172-istio-data\") pod \"e2e-worker-w4-650be0-openshift-default-5c7c5fb7d9-267h5\" (UID: \"3c4531e2-d1cc-430f-8245-f129e486d172\") " pod="openshift-ingress/e2e-worker-w4-650be0-openshift-default-5c7c5fb7d9-267h5" Nov 03 00:54:12.227510 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:54:12.227476 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3c4531e2-d1cc-430f-8245-f129e486d172-istio-envoy\") pod \"e2e-worker-w4-650be0-openshift-default-5c7c5fb7d9-267h5\" (UID: \"3c4531e2-d1cc-430f-8245-f129e486d172\") " pod="openshift-ingress/e2e-worker-w4-650be0-openshift-default-5c7c5fb7d9-267h5" Nov 03 00:54:12.227611 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:54:12.227515 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3c4531e2-d1cc-430f-8245-f129e486d172-workload-socket\") pod \"e2e-worker-w4-650be0-openshift-default-5c7c5fb7d9-267h5\" (UID: \"3c4531e2-d1cc-430f-8245-f129e486d172\") " pod="openshift-ingress/e2e-worker-w4-650be0-openshift-default-5c7c5fb7d9-267h5" Nov 03 00:54:12.227611 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:54:12.227574 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3c4531e2-d1cc-430f-8245-f129e486d172-istio-podinfo\") pod \"e2e-worker-w4-650be0-openshift-default-5c7c5fb7d9-267h5\" (UID: \"3c4531e2-d1cc-430f-8245-f129e486d172\") " pod="openshift-ingress/e2e-worker-w4-650be0-openshift-default-5c7c5fb7d9-267h5" Nov 03 00:54:12.227718 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:54:12.227626 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3c4531e2-d1cc-430f-8245-f129e486d172-workload-certs\") pod \"e2e-worker-w4-650be0-openshift-default-5c7c5fb7d9-267h5\" (UID: \"3c4531e2-d1cc-430f-8245-f129e486d172\") " pod="openshift-ingress/e2e-worker-w4-650be0-openshift-default-5c7c5fb7d9-267h5" Nov 03 00:54:12.227820 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:54:12.227788 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3c4531e2-d1cc-430f-8245-f129e486d172-istiod-ca-cert\") pod \"e2e-worker-w4-650be0-openshift-default-5c7c5fb7d9-267h5\" (UID: \"3c4531e2-d1cc-430f-8245-f129e486d172\") " pod="openshift-ingress/e2e-worker-w4-650be0-openshift-default-5c7c5fb7d9-267h5" Nov 03 00:54:12.227820 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:54:12.227819 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3c4531e2-d1cc-430f-8245-f129e486d172-credential-socket\") pod \"e2e-worker-w4-650be0-openshift-default-5c7c5fb7d9-267h5\" (UID: \"3c4531e2-d1cc-430f-8245-f129e486d172\") " pod="openshift-ingress/e2e-worker-w4-650be0-openshift-default-5c7c5fb7d9-267h5" Nov 03 00:54:12.228033 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:54:12.228005 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3c4531e2-d1cc-430f-8245-f129e486d172-workload-socket\") pod \"e2e-worker-w4-650be0-openshift-default-5c7c5fb7d9-267h5\" (UID: \"3c4531e2-d1cc-430f-8245-f129e486d172\") " pod="openshift-ingress/e2e-worker-w4-650be0-openshift-default-5c7c5fb7d9-267h5" Nov 03 00:54:12.228408 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:54:12.228379 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3c4531e2-d1cc-430f-8245-f129e486d172-istio-data\") pod \"e2e-worker-w4-650be0-openshift-default-5c7c5fb7d9-267h5\" (UID: \"3c4531e2-d1cc-430f-8245-f129e486d172\") " pod="openshift-ingress/e2e-worker-w4-650be0-openshift-default-5c7c5fb7d9-267h5" Nov 03 00:54:12.230184 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:54:12.230161 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3c4531e2-d1cc-430f-8245-f129e486d172-istio-envoy\") pod \"e2e-worker-w4-650be0-openshift-default-5c7c5fb7d9-267h5\" (UID: \"3c4531e2-d1cc-430f-8245-f129e486d172\") " pod="openshift-ingress/e2e-worker-w4-650be0-openshift-default-5c7c5fb7d9-267h5" Nov 03 00:54:12.230391 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:54:12.230366 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3c4531e2-d1cc-430f-8245-f129e486d172-istio-podinfo\") pod \"e2e-worker-w4-650be0-openshift-default-5c7c5fb7d9-267h5\" (UID: \"3c4531e2-d1cc-430f-8245-f129e486d172\") " pod="openshift-ingress/e2e-worker-w4-650be0-openshift-default-5c7c5fb7d9-267h5" Nov 03 00:54:12.235845 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:54:12.235824 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3c4531e2-d1cc-430f-8245-f129e486d172-istio-token\") pod \"e2e-worker-w4-650be0-openshift-default-5c7c5fb7d9-267h5\" (UID: \"3c4531e2-d1cc-430f-8245-f129e486d172\") " pod="openshift-ingress/e2e-worker-w4-650be0-openshift-default-5c7c5fb7d9-267h5" Nov 03 00:54:12.235995 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:54:12.235979 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-thsvt\" (UniqueName: \"kubernetes.io/projected/3c4531e2-d1cc-430f-8245-f129e486d172-kube-api-access-thsvt\") pod \"e2e-worker-w4-650be0-openshift-default-5c7c5fb7d9-267h5\" (UID: \"3c4531e2-d1cc-430f-8245-f129e486d172\") " pod="openshift-ingress/e2e-worker-w4-650be0-openshift-default-5c7c5fb7d9-267h5" Nov 03 00:54:12.244824 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:54:12.244809 4939 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-650be0-openshift-default-5c7c5fb7d9-84qlq" Nov 03 00:54:12.290775 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:54:12.290743 4939 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-650be0-openshift-default-5c7c5fb7d9-267h5" Nov 03 00:54:12.526450 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:54:12.526364 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 00:54:12.548214 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:54:12.548188 4939 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-650be0-openshift-default-5c7c5fb7d9-84qlq"] Nov 03 00:54:12.549228 ip-10-0-140-107 kubenswrapper[4939]: W1103 00:54:12.549208 4939 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd2dabe94_37c9_4a3f_bf9a_15e4f53e05d0.slice/crio-59229abf1249ba042d9b842f2814fd31ff087502d8a464bb514a33b0b86121e4 WatchSource:0}: Error finding container 59229abf1249ba042d9b842f2814fd31ff087502d8a464bb514a33b0b86121e4: Status 404 returned error can't find the container with id 59229abf1249ba042d9b842f2814fd31ff087502d8a464bb514a33b0b86121e4 Nov 03 00:54:12.551741 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:54:12.551710 4939 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 00:54:12.551807 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:54:12.551771 4939 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 00:54:12.551845 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:54:12.551808 4939 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 00:54:12.596478 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:54:12.596449 4939 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-650be0-openshift-default-5c7c5fb7d9-267h5"] Nov 03 00:54:12.597940 ip-10-0-140-107 kubenswrapper[4939]: W1103 00:54:12.597886 4939 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3c4531e2_d1cc_430f_8245_f129e486d172.slice/crio-ca19798c771aa143c820f7c5ccce887042c5ef2f35a4d5415b8e9de4fa1526c5 WatchSource:0}: Error finding container ca19798c771aa143c820f7c5ccce887042c5ef2f35a4d5415b8e9de4fa1526c5: Status 404 returned error can't find the container with id ca19798c771aa143c820f7c5ccce887042c5ef2f35a4d5415b8e9de4fa1526c5 Nov 03 00:54:12.600737 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:54:12.600696 4939 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 00:54:12.600826 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:54:12.600775 4939 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 00:54:12.600826 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:54:12.600804 4939 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 00:54:12.744141 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:54:12.744090 4939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-650be0-openshift-default-5c7c5fb7d9-84qlq" event={"ID":"d2dabe94-37c9-4a3f-bf9a-15e4f53e05d0","Type":"ContainerStarted","Data":"463608114ded4c17d374f75c240e2b2f1556c617d103054903616e64d1a01bc6"} Nov 03 00:54:12.744661 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:54:12.744172 4939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-650be0-openshift-default-5c7c5fb7d9-84qlq" event={"ID":"d2dabe94-37c9-4a3f-bf9a-15e4f53e05d0","Type":"ContainerStarted","Data":"59229abf1249ba042d9b842f2814fd31ff087502d8a464bb514a33b0b86121e4"} Nov 03 00:54:12.746355 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:54:12.746324 4939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-650be0-openshift-default-5c7c5fb7d9-267h5" event={"ID":"3c4531e2-d1cc-430f-8245-f129e486d172","Type":"ContainerStarted","Data":"61535bce52894eba09afa6c0ff603a245743d80cdd1df53f4d77b402ba744723"} Nov 03 00:54:12.746574 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:54:12.746528 4939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-650be0-openshift-default-5c7c5fb7d9-267h5" event={"ID":"3c4531e2-d1cc-430f-8245-f129e486d172","Type":"ContainerStarted","Data":"ca19798c771aa143c820f7c5ccce887042c5ef2f35a4d5415b8e9de4fa1526c5"} Nov 03 00:54:12.766609 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:54:12.766549 4939 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-650be0-openshift-default-5c7c5fb7d9-84qlq" podStartSLOduration=1.766531968 podStartE2EDuration="1.766531968s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:54:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:54:12.76362976 +0000 UTC m=+801.027964705" watchObservedRunningTime="2026-11-03 00:54:12.766531968 +0000 UTC m=+801.030866914" Nov 03 00:54:13.181329 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:54:13.181294 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 00:54:13.250587 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:54:13.245312 4939 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-650be0-openshift-default-5c7c5fb7d9-84qlq" Nov 03 00:54:13.256852 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:54:13.256038 4939 patch_prober.go:29] interesting pod/e2e-worker-w4-650be0-openshift-default-5c7c5fb7d9-84qlq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.80:15021/healthz/ready\": dial tcp 10.132.0.80:15021: connect: connection refused" start-of-body= Nov 03 00:54:13.256852 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:54:13.256104 4939 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-650be0-openshift-default-5c7c5fb7d9-84qlq" podUID="d2dabe94-37c9-4a3f-bf9a-15e4f53e05d0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.80:15021/healthz/ready\": dial tcp 10.132.0.80:15021: connect: connection refused" Nov 03 00:54:13.291572 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:54:13.291540 4939 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-650be0-openshift-default-5c7c5fb7d9-267h5" Nov 03 00:54:13.293230 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:54:13.293200 4939 patch_prober.go:29] interesting pod/e2e-worker-w4-650be0-openshift-default-5c7c5fb7d9-267h5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.81:15021/healthz/ready\": dial tcp 10.132.0.81:15021: connect: connection refused" start-of-body= Nov 03 00:54:13.293383 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:54:13.293251 4939 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-650be0-openshift-default-5c7c5fb7d9-267h5" podUID="3c4531e2-d1cc-430f-8245-f129e486d172" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.81:15021/healthz/ready\": dial tcp 10.132.0.81:15021: connect: connection refused" Nov 03 00:54:13.904720 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:54:13.904680 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 00:54:14.245951 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:54:14.245846 4939 patch_prober.go:29] interesting pod/e2e-worker-w4-650be0-openshift-default-5c7c5fb7d9-84qlq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.80:15021/healthz/ready\": dial tcp 10.132.0.80:15021: connect: connection refused" start-of-body= Nov 03 00:54:14.245951 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:54:14.245936 4939 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-650be0-openshift-default-5c7c5fb7d9-84qlq" podUID="d2dabe94-37c9-4a3f-bf9a-15e4f53e05d0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.80:15021/healthz/ready\": dial tcp 10.132.0.80:15021: connect: connection refused" Nov 03 00:54:14.295403 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:54:14.295357 4939 patch_prober.go:29] interesting pod/e2e-worker-w4-650be0-openshift-default-5c7c5fb7d9-267h5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.81:15021/healthz/ready\": dial tcp 10.132.0.81:15021: connect: connection refused" start-of-body= Nov 03 00:54:14.295623 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:54:14.295429 4939 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-650be0-openshift-default-5c7c5fb7d9-267h5" podUID="3c4531e2-d1cc-430f-8245-f129e486d172" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.81:15021/healthz/ready\": dial tcp 10.132.0.81:15021: connect: connection refused" Nov 03 00:54:14.932246 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:54:14.932202 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 00:54:15.245601 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:54:15.245523 4939 patch_prober.go:29] interesting pod/e2e-worker-w4-650be0-openshift-default-5c7c5fb7d9-84qlq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.80:15021/healthz/ready\": dial tcp 10.132.0.80:15021: connect: connection refused" start-of-body= Nov 03 00:54:15.245601 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:54:15.245578 4939 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-650be0-openshift-default-5c7c5fb7d9-84qlq" podUID="d2dabe94-37c9-4a3f-bf9a-15e4f53e05d0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.80:15021/healthz/ready\": dial tcp 10.132.0.80:15021: connect: connection refused" Nov 03 00:54:15.292345 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:54:15.292308 4939 patch_prober.go:29] interesting pod/e2e-worker-w4-650be0-openshift-default-5c7c5fb7d9-267h5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.81:15021/healthz/ready\": dial tcp 10.132.0.81:15021: connect: connection refused" start-of-body= Nov 03 00:54:15.292628 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:54:15.292597 4939 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-650be0-openshift-default-5c7c5fb7d9-267h5" podUID="3c4531e2-d1cc-430f-8245-f129e486d172" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.81:15021/healthz/ready\": dial tcp 10.132.0.81:15021: connect: connection refused" Nov 03 00:54:15.330964 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:54:15.330918 4939 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-650be0-openshift-default-5c7c5fb7d9-267h5" podStartSLOduration=4.330892295 podStartE2EDuration="4.330892295s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:54:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:54:12.784068501 +0000 UTC m=+801.048403444" watchObservedRunningTime="2026-11-03 00:54:15.330892295 +0000 UTC m=+803.595227235" Nov 03 00:54:15.332101 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:54:15.332077 4939 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-f7680d-openshift-default-c4488d5c9-ldr7q"] Nov 03 00:54:15.336548 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:54:15.336533 4939 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-f7680d-openshift-default-c4488d5c9-ldr7q" Nov 03 00:54:15.414028 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:54:15.413997 4939 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-f7680d-openshift-default-c4488d5c9-ldr7q"] Nov 03 00:54:15.427168 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:54:15.426014 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9b5bb111-d57c-478b-b990-db75ac2df998-istio-envoy\") pod \"e2e-worker-w2-f7680d-openshift-default-c4488d5c9-ldr7q\" (UID: \"9b5bb111-d57c-478b-b990-db75ac2df998\") " pod="openshift-ingress/e2e-worker-w2-f7680d-openshift-default-c4488d5c9-ldr7q" Nov 03 00:54:15.427168 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:54:15.426063 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9b5bb111-d57c-478b-b990-db75ac2df998-istio-data\") pod \"e2e-worker-w2-f7680d-openshift-default-c4488d5c9-ldr7q\" (UID: \"9b5bb111-d57c-478b-b990-db75ac2df998\") " pod="openshift-ingress/e2e-worker-w2-f7680d-openshift-default-c4488d5c9-ldr7q" Nov 03 00:54:15.427168 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:54:15.426149 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9b5bb111-d57c-478b-b990-db75ac2df998-workload-socket\") pod \"e2e-worker-w2-f7680d-openshift-default-c4488d5c9-ldr7q\" (UID: \"9b5bb111-d57c-478b-b990-db75ac2df998\") " pod="openshift-ingress/e2e-worker-w2-f7680d-openshift-default-c4488d5c9-ldr7q" Nov 03 00:54:15.427168 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:54:15.426183 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9b5bb111-d57c-478b-b990-db75ac2df998-workload-certs\") pod \"e2e-worker-w2-f7680d-openshift-default-c4488d5c9-ldr7q\" (UID: \"9b5bb111-d57c-478b-b990-db75ac2df998\") " pod="openshift-ingress/e2e-worker-w2-f7680d-openshift-default-c4488d5c9-ldr7q" Nov 03 00:54:15.427168 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:54:15.426200 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9b5bb111-d57c-478b-b990-db75ac2df998-istio-podinfo\") pod \"e2e-worker-w2-f7680d-openshift-default-c4488d5c9-ldr7q\" (UID: \"9b5bb111-d57c-478b-b990-db75ac2df998\") " pod="openshift-ingress/e2e-worker-w2-f7680d-openshift-default-c4488d5c9-ldr7q" Nov 03 00:54:15.427168 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:54:15.426215 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9b5bb111-d57c-478b-b990-db75ac2df998-istiod-ca-cert\") pod \"e2e-worker-w2-f7680d-openshift-default-c4488d5c9-ldr7q\" (UID: \"9b5bb111-d57c-478b-b990-db75ac2df998\") " pod="openshift-ingress/e2e-worker-w2-f7680d-openshift-default-c4488d5c9-ldr7q" Nov 03 00:54:15.427168 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:54:15.426356 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wlbpj\" (UniqueName: \"kubernetes.io/projected/9b5bb111-d57c-478b-b990-db75ac2df998-kube-api-access-wlbpj\") pod \"e2e-worker-w2-f7680d-openshift-default-c4488d5c9-ldr7q\" (UID: \"9b5bb111-d57c-478b-b990-db75ac2df998\") " pod="openshift-ingress/e2e-worker-w2-f7680d-openshift-default-c4488d5c9-ldr7q" Nov 03 00:54:15.427168 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:54:15.426378 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9b5bb111-d57c-478b-b990-db75ac2df998-istio-token\") pod \"e2e-worker-w2-f7680d-openshift-default-c4488d5c9-ldr7q\" (UID: \"9b5bb111-d57c-478b-b990-db75ac2df998\") " pod="openshift-ingress/e2e-worker-w2-f7680d-openshift-default-c4488d5c9-ldr7q" Nov 03 00:54:15.427168 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:54:15.426414 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9b5bb111-d57c-478b-b990-db75ac2df998-credential-socket\") pod \"e2e-worker-w2-f7680d-openshift-default-c4488d5c9-ldr7q\" (UID: \"9b5bb111-d57c-478b-b990-db75ac2df998\") " pod="openshift-ingress/e2e-worker-w2-f7680d-openshift-default-c4488d5c9-ldr7q" Nov 03 00:54:15.528020 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:54:15.527904 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9b5bb111-d57c-478b-b990-db75ac2df998-istio-envoy\") pod \"e2e-worker-w2-f7680d-openshift-default-c4488d5c9-ldr7q\" (UID: \"9b5bb111-d57c-478b-b990-db75ac2df998\") " pod="openshift-ingress/e2e-worker-w2-f7680d-openshift-default-c4488d5c9-ldr7q" Nov 03 00:54:15.528020 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:54:15.527986 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9b5bb111-d57c-478b-b990-db75ac2df998-istio-data\") pod \"e2e-worker-w2-f7680d-openshift-default-c4488d5c9-ldr7q\" (UID: \"9b5bb111-d57c-478b-b990-db75ac2df998\") " pod="openshift-ingress/e2e-worker-w2-f7680d-openshift-default-c4488d5c9-ldr7q" Nov 03 00:54:15.528225 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:54:15.528084 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9b5bb111-d57c-478b-b990-db75ac2df998-workload-socket\") pod \"e2e-worker-w2-f7680d-openshift-default-c4488d5c9-ldr7q\" (UID: \"9b5bb111-d57c-478b-b990-db75ac2df998\") " pod="openshift-ingress/e2e-worker-w2-f7680d-openshift-default-c4488d5c9-ldr7q" Nov 03 00:54:15.528225 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:54:15.528118 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9b5bb111-d57c-478b-b990-db75ac2df998-workload-certs\") pod \"e2e-worker-w2-f7680d-openshift-default-c4488d5c9-ldr7q\" (UID: \"9b5bb111-d57c-478b-b990-db75ac2df998\") " pod="openshift-ingress/e2e-worker-w2-f7680d-openshift-default-c4488d5c9-ldr7q" Nov 03 00:54:15.528225 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:54:15.528134 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9b5bb111-d57c-478b-b990-db75ac2df998-istio-podinfo\") pod \"e2e-worker-w2-f7680d-openshift-default-c4488d5c9-ldr7q\" (UID: \"9b5bb111-d57c-478b-b990-db75ac2df998\") " pod="openshift-ingress/e2e-worker-w2-f7680d-openshift-default-c4488d5c9-ldr7q" Nov 03 00:54:15.528225 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:54:15.528150 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9b5bb111-d57c-478b-b990-db75ac2df998-istiod-ca-cert\") pod \"e2e-worker-w2-f7680d-openshift-default-c4488d5c9-ldr7q\" (UID: \"9b5bb111-d57c-478b-b990-db75ac2df998\") " pod="openshift-ingress/e2e-worker-w2-f7680d-openshift-default-c4488d5c9-ldr7q" Nov 03 00:54:15.528440 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:54:15.528235 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wlbpj\" (UniqueName: \"kubernetes.io/projected/9b5bb111-d57c-478b-b990-db75ac2df998-kube-api-access-wlbpj\") pod \"e2e-worker-w2-f7680d-openshift-default-c4488d5c9-ldr7q\" (UID: \"9b5bb111-d57c-478b-b990-db75ac2df998\") " pod="openshift-ingress/e2e-worker-w2-f7680d-openshift-default-c4488d5c9-ldr7q" Nov 03 00:54:15.528440 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:54:15.528252 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9b5bb111-d57c-478b-b990-db75ac2df998-istio-token\") pod \"e2e-worker-w2-f7680d-openshift-default-c4488d5c9-ldr7q\" (UID: \"9b5bb111-d57c-478b-b990-db75ac2df998\") " pod="openshift-ingress/e2e-worker-w2-f7680d-openshift-default-c4488d5c9-ldr7q" Nov 03 00:54:15.528440 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:54:15.528284 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9b5bb111-d57c-478b-b990-db75ac2df998-credential-socket\") pod \"e2e-worker-w2-f7680d-openshift-default-c4488d5c9-ldr7q\" (UID: \"9b5bb111-d57c-478b-b990-db75ac2df998\") " pod="openshift-ingress/e2e-worker-w2-f7680d-openshift-default-c4488d5c9-ldr7q" Nov 03 00:54:15.528660 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:54:15.528635 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9b5bb111-d57c-478b-b990-db75ac2df998-workload-certs\") pod \"e2e-worker-w2-f7680d-openshift-default-c4488d5c9-ldr7q\" (UID: \"9b5bb111-d57c-478b-b990-db75ac2df998\") " pod="openshift-ingress/e2e-worker-w2-f7680d-openshift-default-c4488d5c9-ldr7q" Nov 03 00:54:15.528740 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:54:15.528717 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9b5bb111-d57c-478b-b990-db75ac2df998-credential-socket\") pod \"e2e-worker-w2-f7680d-openshift-default-c4488d5c9-ldr7q\" (UID: \"9b5bb111-d57c-478b-b990-db75ac2df998\") " pod="openshift-ingress/e2e-worker-w2-f7680d-openshift-default-c4488d5c9-ldr7q" Nov 03 00:54:15.528832 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:54:15.528808 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9b5bb111-d57c-478b-b990-db75ac2df998-istio-data\") pod \"e2e-worker-w2-f7680d-openshift-default-c4488d5c9-ldr7q\" (UID: \"9b5bb111-d57c-478b-b990-db75ac2df998\") " pod="openshift-ingress/e2e-worker-w2-f7680d-openshift-default-c4488d5c9-ldr7q" Nov 03 00:54:15.529259 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:54:15.529238 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9b5bb111-d57c-478b-b990-db75ac2df998-workload-socket\") pod \"e2e-worker-w2-f7680d-openshift-default-c4488d5c9-ldr7q\" (UID: \"9b5bb111-d57c-478b-b990-db75ac2df998\") " pod="openshift-ingress/e2e-worker-w2-f7680d-openshift-default-c4488d5c9-ldr7q" Nov 03 00:54:15.529613 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:54:15.529594 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9b5bb111-d57c-478b-b990-db75ac2df998-istiod-ca-cert\") pod \"e2e-worker-w2-f7680d-openshift-default-c4488d5c9-ldr7q\" (UID: \"9b5bb111-d57c-478b-b990-db75ac2df998\") " pod="openshift-ingress/e2e-worker-w2-f7680d-openshift-default-c4488d5c9-ldr7q" Nov 03 00:54:15.531053 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:54:15.531034 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9b5bb111-d57c-478b-b990-db75ac2df998-istio-envoy\") pod \"e2e-worker-w2-f7680d-openshift-default-c4488d5c9-ldr7q\" (UID: \"9b5bb111-d57c-478b-b990-db75ac2df998\") " pod="openshift-ingress/e2e-worker-w2-f7680d-openshift-default-c4488d5c9-ldr7q" Nov 03 00:54:15.533369 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:54:15.533330 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9b5bb111-d57c-478b-b990-db75ac2df998-istio-podinfo\") pod \"e2e-worker-w2-f7680d-openshift-default-c4488d5c9-ldr7q\" (UID: \"9b5bb111-d57c-478b-b990-db75ac2df998\") " pod="openshift-ingress/e2e-worker-w2-f7680d-openshift-default-c4488d5c9-ldr7q" Nov 03 00:54:15.536837 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:54:15.536814 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9b5bb111-d57c-478b-b990-db75ac2df998-istio-token\") pod \"e2e-worker-w2-f7680d-openshift-default-c4488d5c9-ldr7q\" (UID: \"9b5bb111-d57c-478b-b990-db75ac2df998\") " pod="openshift-ingress/e2e-worker-w2-f7680d-openshift-default-c4488d5c9-ldr7q" Nov 03 00:54:15.537277 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:54:15.537261 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wlbpj\" (UniqueName: \"kubernetes.io/projected/9b5bb111-d57c-478b-b990-db75ac2df998-kube-api-access-wlbpj\") pod \"e2e-worker-w2-f7680d-openshift-default-c4488d5c9-ldr7q\" (UID: \"9b5bb111-d57c-478b-b990-db75ac2df998\") " pod="openshift-ingress/e2e-worker-w2-f7680d-openshift-default-c4488d5c9-ldr7q" Nov 03 00:54:15.715718 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:54:15.715688 4939 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-f7680d-openshift-default-c4488d5c9-ldr7q" Nov 03 00:54:16.199456 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:54:16.178139 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 00:54:16.203854 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:54:16.203801 4939 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 00:54:16.203993 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:54:16.203925 4939 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 00:54:16.203993 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:54:16.203984 4939 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 00:54:16.204844 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:54:16.204808 4939 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-f7680d-openshift-default-c4488d5c9-ldr7q"] Nov 03 00:54:16.245579 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:54:16.245261 4939 patch_prober.go:29] interesting pod/e2e-worker-w4-650be0-openshift-default-5c7c5fb7d9-84qlq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.80:15021/healthz/ready\": dial tcp 10.132.0.80:15021: connect: connection refused" start-of-body= Nov 03 00:54:16.245579 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:54:16.245304 4939 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-650be0-openshift-default-5c7c5fb7d9-84qlq" podUID="d2dabe94-37c9-4a3f-bf9a-15e4f53e05d0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.80:15021/healthz/ready\": dial tcp 10.132.0.80:15021: connect: connection refused" Nov 03 00:54:16.300974 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:54:16.297329 4939 patch_prober.go:29] interesting pod/e2e-worker-w4-650be0-openshift-default-5c7c5fb7d9-267h5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.81:15021/healthz/ready\": dial tcp 10.132.0.81:15021: connect: connection refused" start-of-body= Nov 03 00:54:16.300974 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:54:16.297387 4939 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-650be0-openshift-default-5c7c5fb7d9-267h5" podUID="3c4531e2-d1cc-430f-8245-f129e486d172" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.81:15021/healthz/ready\": dial tcp 10.132.0.81:15021: connect: connection refused" Nov 03 00:54:16.829746 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:54:16.829711 4939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-f7680d-openshift-default-c4488d5c9-ldr7q" event={"ID":"9b5bb111-d57c-478b-b990-db75ac2df998","Type":"ContainerStarted","Data":"e5ff86a1fadda7447aaa59b5b3ece9e64d7f369acf0fe17c71c037b90c523ffd"} Nov 03 00:54:16.829898 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:54:16.829776 4939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-f7680d-openshift-default-c4488d5c9-ldr7q" event={"ID":"9b5bb111-d57c-478b-b990-db75ac2df998","Type":"ContainerStarted","Data":"141c582ac526d7ce0827c274d1319e13ddaa01ce5e1951198e9778c25110b152"} Nov 03 00:54:16.834058 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:54:16.833992 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 00:54:16.870928 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:54:16.870839 4939 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-f7680d-openshift-default-c4488d5c9-ldr7q" podStartSLOduration=1.8708172109999999 podStartE2EDuration="1.870817211s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:54:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:54:16.8515047 +0000 UTC m=+805.115839659" watchObservedRunningTime="2026-11-03 00:54:16.870817211 +0000 UTC m=+805.135152157" Nov 03 00:54:17.246039 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:54:17.246001 4939 patch_prober.go:29] interesting pod/e2e-worker-w4-650be0-openshift-default-5c7c5fb7d9-84qlq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.80:15021/healthz/ready\": dial tcp 10.132.0.80:15021: connect: connection refused" start-of-body= Nov 03 00:54:17.246546 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:54:17.246063 4939 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-650be0-openshift-default-5c7c5fb7d9-84qlq" podUID="d2dabe94-37c9-4a3f-bf9a-15e4f53e05d0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.80:15021/healthz/ready\": dial tcp 10.132.0.80:15021: connect: connection refused" Nov 03 00:54:17.304934 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:54:17.299360 4939 patch_prober.go:29] interesting pod/e2e-worker-w4-650be0-openshift-default-5c7c5fb7d9-267h5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.81:15021/healthz/ready\": dial tcp 10.132.0.81:15021: connect: connection refused" start-of-body= Nov 03 00:54:17.304934 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:54:17.299435 4939 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-650be0-openshift-default-5c7c5fb7d9-267h5" podUID="3c4531e2-d1cc-430f-8245-f129e486d172" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.81:15021/healthz/ready\": dial tcp 10.132.0.81:15021: connect: connection refused" Nov 03 00:54:17.718936 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:54:17.718156 4939 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-f7680d-openshift-default-c4488d5c9-ldr7q" Nov 03 00:54:17.718936 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:54:17.718330 4939 patch_prober.go:29] interesting pod/e2e-worker-w2-f7680d-openshift-default-c4488d5c9-ldr7q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.82:15021/healthz/ready\": dial tcp 10.132.0.82:15021: connect: connection refused" start-of-body= Nov 03 00:54:17.718936 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:54:17.718374 4939 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-f7680d-openshift-default-c4488d5c9-ldr7q" podUID="9b5bb111-d57c-478b-b990-db75ac2df998" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.82:15021/healthz/ready\": dial tcp 10.132.0.82:15021: connect: connection refused" Nov 03 00:54:17.832252 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:54:17.832181 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 00:54:18.212933 ip-10-0-140-107 kubenswrapper[4939]: E1103 00:54:18.210437 4939 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-sthjh" containerName="extractor" Nov 03 00:54:18.246933 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:54:18.246508 4939 patch_prober.go:29] interesting pod/e2e-worker-w4-650be0-openshift-default-5c7c5fb7d9-84qlq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.80:15021/healthz/ready\": dial tcp 10.132.0.80:15021: connect: connection refused" start-of-body= Nov 03 00:54:18.246933 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:54:18.246569 4939 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-650be0-openshift-default-5c7c5fb7d9-84qlq" podUID="d2dabe94-37c9-4a3f-bf9a-15e4f53e05d0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.80:15021/healthz/ready\": dial tcp 10.132.0.80:15021: connect: connection refused" Nov 03 00:54:18.296933 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:54:18.296298 4939 patch_prober.go:29] interesting pod/e2e-worker-w4-650be0-openshift-default-5c7c5fb7d9-267h5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.81:15021/healthz/ready\": dial tcp 10.132.0.81:15021: connect: connection refused" start-of-body= Nov 03 00:54:18.296933 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:54:18.296362 4939 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-650be0-openshift-default-5c7c5fb7d9-267h5" podUID="3c4531e2-d1cc-430f-8245-f129e486d172" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.81:15021/healthz/ready\": dial tcp 10.132.0.81:15021: connect: connection refused" Nov 03 00:54:18.725839 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:54:18.718352 4939 patch_prober.go:29] interesting pod/e2e-worker-w2-f7680d-openshift-default-c4488d5c9-ldr7q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.82:15021/healthz/ready\": dial tcp 10.132.0.82:15021: connect: connection refused" start-of-body= Nov 03 00:54:18.725839 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:54:18.718415 4939 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-f7680d-openshift-default-c4488d5c9-ldr7q" podUID="9b5bb111-d57c-478b-b990-db75ac2df998" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.82:15021/healthz/ready\": dial tcp 10.132.0.82:15021: connect: connection refused" Nov 03 00:54:18.919928 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:54:18.918984 4939 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-88bbf7/simulator-w1-88bbf7-kserve-5fbb669f8f-42hgd" podUID="83b38883-2e7a-444d-aa68-52be4a25ae66" containerName="main" probeResult="failure" output="Get \"https://10.132.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 00:54:19.245931 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:54:19.245577 4939 patch_prober.go:29] interesting pod/e2e-worker-w4-650be0-openshift-default-5c7c5fb7d9-84qlq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.80:15021/healthz/ready\": dial tcp 10.132.0.80:15021: connect: connection refused" start-of-body= Nov 03 00:54:19.245931 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:54:19.245655 4939 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-650be0-openshift-default-5c7c5fb7d9-84qlq" podUID="d2dabe94-37c9-4a3f-bf9a-15e4f53e05d0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.80:15021/healthz/ready\": dial tcp 10.132.0.80:15021: connect: connection refused" Nov 03 00:54:19.294420 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:54:19.294382 4939 patch_prober.go:29] interesting pod/e2e-worker-w4-650be0-openshift-default-5c7c5fb7d9-267h5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.81:15021/healthz/ready\": dial tcp 10.132.0.81:15021: connect: connection refused" start-of-body= Nov 03 00:54:19.295028 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:54:19.294985 4939 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-650be0-openshift-default-5c7c5fb7d9-267h5" podUID="3c4531e2-d1cc-430f-8245-f129e486d172" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.81:15021/healthz/ready\": dial tcp 10.132.0.81:15021: connect: connection refused" Nov 03 00:54:19.676057 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:54:19.676009 4939 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-83755a/distinct-2-w3-83755a-kserve-6cf66b995d-2l47b" podUID="c3b05e88-5d5b-4d03-b28d-832f22e3c868" containerName="main" probeResult="failure" output="Get \"https://10.132.0.72:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 00:54:19.720318 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:54:19.720277 4939 patch_prober.go:29] interesting pod/e2e-worker-w2-f7680d-openshift-default-c4488d5c9-ldr7q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.82:15021/healthz/ready\": dial tcp 10.132.0.82:15021: connect: connection refused" start-of-body= Nov 03 00:54:19.720618 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:54:19.720582 4939 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-f7680d-openshift-default-c4488d5c9-ldr7q" podUID="9b5bb111-d57c-478b-b990-db75ac2df998" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.82:15021/healthz/ready\": dial tcp 10.132.0.82:15021: connect: connection refused" Nov 03 00:54:19.772933 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:54:19.772206 4939 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-f7680d/distinct-2-w2-f7680d-kserve-7d7c4564bf-cbv2c" podUID="2de18086-761b-4773-b9fc-15ddc4a220fc" containerName="main" probeResult="failure" output="Get \"https://10.132.0.75:8000/ready\": context deadline exceeded" Nov 03 00:54:19.817046 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:54:19.816998 4939 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-f7680d/premium-w2-f7680d-kserve-76565b8cf9-hqbbt" podUID="4712e708-18a2-4a13-a12c-a5669397b2c3" containerName="main" probeResult="failure" output="Get \"https://10.132.0.63:8000/ready\": context deadline exceeded" Nov 03 00:54:19.872209 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:54:19.872168 4939 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-2f9a3a/simulator-w0-2f9a3a-kserve-9596bbb8f-2trgz" podUID="207e2207-fb6c-4076-95df-72e95671c261" containerName="main" probeResult="failure" output="Get \"https://10.132.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 00:54:19.905938 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:54:19.902036 4939 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-2f9a3a/distinct-w0-2f9a3a-kserve-555565cfcf-jtcr9" podUID="ac0ece06-8718-463f-915f-5315988d9711" containerName="main" probeResult="failure" output="Get \"https://10.132.0.76:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 00:54:19.934994 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:54:19.920018 4939 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-88bbf7/simulator-w1-88bbf7-kserve-5fbb669f8f-42hgd" podUID="83b38883-2e7a-444d-aa68-52be4a25ae66" containerName="main" probeResult="failure" output="Get \"https://10.132.0.62:8000/ready\": context deadline exceeded" Nov 03 00:54:19.934994 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:54:19.920962 4939 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-88bbf7/simulator-w1-88bbf7-kserve-5fbb669f8f-42hgd" Nov 03 00:54:20.079065 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:54:20.079013 4939 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-f7680d/distinct-w2-f7680d-kserve-756f874b98-qnpv8" podUID="836753fc-f9ef-4f61-b3fe-e5b92b7ee27b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.70:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 00:54:20.303986 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:54:20.303695 4939 patch_prober.go:29] interesting pod/e2e-worker-w4-650be0-openshift-default-5c7c5fb7d9-84qlq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.80:15021/healthz/ready\": dial tcp 10.132.0.80:15021: connect: connection refused" start-of-body= Nov 03 00:54:20.303986 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:54:20.303764 4939 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-650be0-openshift-default-5c7c5fb7d9-84qlq" podUID="d2dabe94-37c9-4a3f-bf9a-15e4f53e05d0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.80:15021/healthz/ready\": dial tcp 10.132.0.80:15021: connect: connection refused" Nov 03 00:54:20.304544 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:54:20.304431 4939 patch_prober.go:29] interesting pod/e2e-worker-w4-650be0-openshift-default-5c7c5fb7d9-267h5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.81:15021/healthz/ready\": dial tcp 10.132.0.81:15021: connect: connection refused" start-of-body= Nov 03 00:54:20.304544 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:54:20.304470 4939 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-650be0-openshift-default-5c7c5fb7d9-267h5" podUID="3c4531e2-d1cc-430f-8245-f129e486d172" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.81:15021/healthz/ready\": dial tcp 10.132.0.81:15021: connect: connection refused" Nov 03 00:54:20.682421 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:54:20.682358 4939 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-83755a/distinct-2-w3-83755a-kserve-6cf66b995d-2l47b" podUID="c3b05e88-5d5b-4d03-b28d-832f22e3c868" containerName="main" probeResult="failure" output="Get \"https://10.132.0.72:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 00:54:20.683006 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:54:20.682983 4939 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-83755a/distinct-2-w3-83755a-kserve-6cf66b995d-2l47b" Nov 03 00:54:20.909931 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:54:20.886003 4939 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-2f9a3a/simulator-w0-2f9a3a-kserve-9596bbb8f-2trgz" podUID="207e2207-fb6c-4076-95df-72e95671c261" containerName="main" probeResult="failure" output="Get \"https://10.132.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 00:54:20.910482 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:54:20.910425 4939 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-2f9a3a/distinct-w0-2f9a3a-kserve-555565cfcf-jtcr9" podUID="ac0ece06-8718-463f-915f-5315988d9711" containerName="main" probeResult="failure" output="Get \"https://10.132.0.76:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 00:54:21.199111 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:54:20.911130 4939 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-2f9a3a/simulator-w0-2f9a3a-kserve-9596bbb8f-2trgz" Nov 03 00:54:21.199111 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:54:21.198206 4939 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-2f9a3a/distinct-w0-2f9a3a-kserve-555565cfcf-jtcr9" Nov 03 00:54:21.268302 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:54:20.958849 4939 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-f7680d/distinct-2-w2-f7680d-kserve-7d7c4564bf-cbv2c" podUID="2de18086-761b-4773-b9fc-15ddc4a220fc" containerName="main" probeResult="failure" output="Get \"https://10.132.0.75:8000/ready\": context deadline exceeded" Nov 03 00:54:21.268302 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:54:21.069777 4939 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-f7680d/premium-w2-f7680d-kserve-76565b8cf9-hqbbt" podUID="4712e708-18a2-4a13-a12c-a5669397b2c3" containerName="main" probeResult="failure" output="Get \"https://10.132.0.63:8000/ready\": context deadline exceeded" Nov 03 00:54:21.268302 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:54:21.255939 4939 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-f7680d/distinct-2-w2-f7680d-kserve-7d7c4564bf-cbv2c" Nov 03 00:54:21.268302 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:54:21.256005 4939 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-f7680d/premium-w2-f7680d-kserve-76565b8cf9-hqbbt" Nov 03 00:54:21.268302 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:54:21.096021 4939 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-f7680d/distinct-w2-f7680d-kserve-756f874b98-qnpv8" podUID="836753fc-f9ef-4f61-b3fe-e5b92b7ee27b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.70:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 00:54:21.277592 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:54:21.070787 4939 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-88bbf7/simulator-w1-88bbf7-kserve-5fbb669f8f-42hgd" podUID="83b38883-2e7a-444d-aa68-52be4a25ae66" containerName="main" probeResult="failure" output="Get \"https://10.132.0.62:8000/ready\": context deadline exceeded" Nov 03 00:54:21.305934 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:54:20.980874 4939 patch_prober.go:29] interesting pod/e2e-worker-w2-f7680d-openshift-default-c4488d5c9-ldr7q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.82:15021/healthz/ready\": dial tcp 10.132.0.82:15021: connect: connection refused" start-of-body= Nov 03 00:54:21.306567 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:54:21.306522 4939 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-f7680d-openshift-default-c4488d5c9-ldr7q" podUID="9b5bb111-d57c-478b-b990-db75ac2df998" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.82:15021/healthz/ready\": dial tcp 10.132.0.82:15021: connect: connection refused" Nov 03 00:54:21.416213 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:54:21.415987 4939 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-f7680d/distinct-w2-f7680d-kserve-756f874b98-qnpv8" Nov 03 00:54:21.471010 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:54:21.448702 4939 patch_prober.go:29] interesting pod/e2e-worker-w4-650be0-openshift-default-5c7c5fb7d9-267h5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.81:15021/healthz/ready\": dial tcp 10.132.0.81:15021: connect: connection refused" start-of-body= Nov 03 00:54:21.471010 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:54:21.448766 4939 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-650be0-openshift-default-5c7c5fb7d9-267h5" podUID="3c4531e2-d1cc-430f-8245-f129e486d172" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.81:15021/healthz/ready\": dial tcp 10.132.0.81:15021: connect: connection refused" Nov 03 00:54:21.582040 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:54:21.581113 4939 patch_prober.go:29] interesting pod/e2e-worker-w0-2f9a3a-openshift-default-7f6f74cf97-dxffl container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.57:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 00:54:21.582040 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:54:21.581179 4939 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w0-2f9a3a-openshift-default-7f6f74cf97-dxffl" podUID="80e0f915-1a1a-4214-92f0-5eeb806be96e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.57:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 00:54:21.686866 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:54:21.686817 4939 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-83755a/distinct-2-w3-83755a-kserve-6cf66b995d-2l47b" podUID="c3b05e88-5d5b-4d03-b28d-832f22e3c868" containerName="main" probeResult="failure" output="Get \"https://10.132.0.72:8000/ready\": context deadline exceeded" Nov 03 00:54:21.755002 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:54:21.728166 4939 patch_prober.go:29] interesting pod/e2e-worker-w2-f7680d-openshift-default-c4488d5c9-ldr7q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.82:15021/healthz/ready\": dial tcp 10.132.0.82:15021: connect: connection refused" start-of-body= Nov 03 00:54:21.755301 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:54:21.755257 4939 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-f7680d-openshift-default-c4488d5c9-ldr7q" podUID="9b5bb111-d57c-478b-b990-db75ac2df998" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.82:15021/healthz/ready\": dial tcp 10.132.0.82:15021: connect: connection refused" Nov 03 00:54:22.194935 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:54:22.147480 4939 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-f7680d/distinct-2-w2-f7680d-kserve-7d7c4564bf-cbv2c" podUID="2de18086-761b-4773-b9fc-15ddc4a220fc" containerName="main" probeResult="failure" output="Get \"https://10.132.0.75:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 00:54:22.201634 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:54:22.201594 4939 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-f7680d/premium-w2-f7680d-kserve-76565b8cf9-hqbbt" podUID="4712e708-18a2-4a13-a12c-a5669397b2c3" containerName="main" probeResult="failure" output="Get \"https://10.132.0.63:8000/health\": context deadline exceeded" Nov 03 00:54:22.264522 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:54:22.264466 4939 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-f7680d/distinct-2-w2-f7680d-kserve-7d7c4564bf-cbv2c" podUID="2de18086-761b-4773-b9fc-15ddc4a220fc" containerName="main" probeResult="failure" output="Get \"https://10.132.0.75:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 00:54:22.273980 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:54:22.270524 4939 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-88bbf7/simulator-w1-88bbf7-kserve-5fbb669f8f-42hgd" podUID="83b38883-2e7a-444d-aa68-52be4a25ae66" containerName="main" probeResult="failure" output="Get \"https://10.132.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 00:54:22.304217 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:54:22.304151 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 00:54:22.351082 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:54:22.306464 4939 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-f7680d/distinct-w2-f7680d-kserve-756f874b98-qnpv8" podUID="836753fc-f9ef-4f61-b3fe-e5b92b7ee27b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.70:8000/ready\": context deadline exceeded" Nov 03 00:54:22.391539 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:54:22.332070 4939 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-2f9a3a/distinct-w0-2f9a3a-kserve-555565cfcf-jtcr9" podUID="ac0ece06-8718-463f-915f-5315988d9711" containerName="main" probeResult="failure" output="Get \"https://10.132.0.76:8000/ready\": context deadline exceeded" Nov 03 00:54:22.399937 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:54:22.337221 4939 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-f7680d/premium-w2-f7680d-kserve-76565b8cf9-hqbbt" podUID="4712e708-18a2-4a13-a12c-a5669397b2c3" containerName="main" probeResult="failure" output="Get \"https://10.132.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 00:54:22.440933 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:54:22.403255 4939 patch_prober.go:29] interesting pod/e2e-worker-w4-650be0-openshift-default-5c7c5fb7d9-267h5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.81:15021/healthz/ready\": dial tcp 10.132.0.81:15021: connect: connection refused" start-of-body= Nov 03 00:54:22.476811 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:54:22.476714 4939 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-650be0-openshift-default-5c7c5fb7d9-267h5" podUID="3c4531e2-d1cc-430f-8245-f129e486d172" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.81:15021/healthz/ready\": dial tcp 10.132.0.81:15021: connect: connection refused" Nov 03 00:54:22.513442 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:54:22.408344 4939 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-2f9a3a/simulator-w0-2f9a3a-kserve-9596bbb8f-2trgz" podUID="207e2207-fb6c-4076-95df-72e95671c261" containerName="main" probeResult="failure" output="Get \"https://10.132.0.61:8000/ready\": context deadline exceeded" Nov 03 00:54:22.515166 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:54:22.452001 4939 patch_prober.go:29] interesting pod/e2e-worker-w4-650be0-openshift-default-5c7c5fb7d9-84qlq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.80:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 00:54:22.515368 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:54:22.515339 4939 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-650be0-openshift-default-5c7c5fb7d9-84qlq" podUID="d2dabe94-37c9-4a3f-bf9a-15e4f53e05d0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.80:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 00:54:22.655656 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:54:22.655624 4939 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-2f9a3a/simulator-w0-2f9a3a-kserve-9596bbb8f-2trgz" Nov 03 00:54:22.683455 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:54:22.683408 4939 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-650be0-openshift-default-5c7c5fb7d9-84qlq" Nov 03 00:54:22.684776 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:54:22.684743 4939 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-650be0-openshift-default-5c7c5fb7d9-84qlq" Nov 03 00:54:22.714689 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:54:22.692078 4939 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-83755a/distinct-2-w3-83755a-kserve-6cf66b995d-2l47b" podUID="c3b05e88-5d5b-4d03-b28d-832f22e3c868" containerName="main" probeResult="failure" output="Get \"https://10.132.0.72:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 00:54:22.723714 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:54:22.723681 4939 patch_prober.go:29] interesting pod/e2e-worker-w2-f7680d-openshift-default-c4488d5c9-ldr7q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.82:15021/healthz/ready\": dial tcp 10.132.0.82:15021: connect: connection refused" start-of-body= Nov 03 00:54:22.723977 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:54:22.723950 4939 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-f7680d-openshift-default-c4488d5c9-ldr7q" podUID="9b5bb111-d57c-478b-b990-db75ac2df998" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.82:15021/healthz/ready\": dial tcp 10.132.0.82:15021: connect: connection refused" Nov 03 00:54:22.731249 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:54:22.731162 4939 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-650be0-openshift-default-5c7c5fb7d9-84qlq" Nov 03 00:54:22.760304 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:54:22.760274 4939 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-88bbf7/simulator-w1-88bbf7-kserve-5fbb669f8f-42hgd" Nov 03 00:54:22.787652 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:54:22.787622 4939 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-83755a/distinct-2-w3-83755a-kserve-6cf66b995d-2l47b" Nov 03 00:54:22.983449 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:54:22.983316 4939 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-f7680d/premium-w2-f7680d-kserve-76565b8cf9-hqbbt" Nov 03 00:54:23.038925 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:54:23.036734 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 00:54:23.039575 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:54:23.039557 4939 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-2f9a3a/distinct-w0-2f9a3a-kserve-555565cfcf-jtcr9" Nov 03 00:54:23.052416 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:54:23.052377 4939 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-f7680d/distinct-w2-f7680d-kserve-756f874b98-qnpv8" Nov 03 00:54:23.052685 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:54:23.052667 4939 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-f7680d/distinct-2-w2-f7680d-kserve-7d7c4564bf-cbv2c" Nov 03 00:54:23.434772 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:54:23.434735 4939 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-650be0-openshift-default-5c7c5fb7d9-267h5" Nov 03 00:54:23.435246 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:54:23.435164 4939 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-650be0-openshift-default-5c7c5fb7d9-267h5" Nov 03 00:54:23.436242 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:54:23.436219 4939 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-650be0-openshift-default-5c7c5fb7d9-267h5" Nov 03 00:54:23.717043 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:54:23.716958 4939 patch_prober.go:29] interesting pod/e2e-worker-w2-f7680d-openshift-default-c4488d5c9-ldr7q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.82:15021/healthz/ready\": dial tcp 10.132.0.82:15021: connect: connection refused" start-of-body= Nov 03 00:54:23.717043 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:54:23.717006 4939 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-f7680d-openshift-default-c4488d5c9-ldr7q" podUID="9b5bb111-d57c-478b-b990-db75ac2df998" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.82:15021/healthz/ready\": dial tcp 10.132.0.82:15021: connect: connection refused" Nov 03 00:54:23.994015 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:54:23.993928 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 00:54:24.717289 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:54:24.717252 4939 patch_prober.go:29] interesting pod/e2e-worker-w2-f7680d-openshift-default-c4488d5c9-ldr7q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.82:15021/healthz/ready\": dial tcp 10.132.0.82:15021: connect: connection refused" start-of-body= Nov 03 00:54:24.717716 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:54:24.717311 4939 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-f7680d-openshift-default-c4488d5c9-ldr7q" podUID="9b5bb111-d57c-478b-b990-db75ac2df998" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.82:15021/healthz/ready\": dial tcp 10.132.0.82:15021: connect: connection refused" Nov 03 00:54:24.770067 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:54:24.770029 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 00:54:25.488216 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:54:25.488175 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 00:54:25.717295 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:54:25.717259 4939 patch_prober.go:29] interesting pod/e2e-worker-w2-f7680d-openshift-default-c4488d5c9-ldr7q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.82:15021/healthz/ready\": dial tcp 10.132.0.82:15021: connect: connection refused" start-of-body= Nov 03 00:54:25.717681 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:54:25.717330 4939 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-f7680d-openshift-default-c4488d5c9-ldr7q" podUID="9b5bb111-d57c-478b-b990-db75ac2df998" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.82:15021/healthz/ready\": dial tcp 10.132.0.82:15021: connect: connection refused" Nov 03 00:54:26.294411 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:54:26.294347 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 00:54:26.720963 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:54:26.720895 4939 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-f7680d-openshift-default-c4488d5c9-ldr7q" Nov 03 00:54:26.721435 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:54:26.721417 4939 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-f7680d-openshift-default-c4488d5c9-ldr7q" Nov 03 00:54:26.722098 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:54:26.722080 4939 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-f7680d-openshift-default-c4488d5c9-ldr7q" Nov 03 00:54:27.139550 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:54:27.139512 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 00:54:27.882814 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:54:27.882760 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 00:54:28.417879 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:54:28.417846 4939 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-4pw7v"] Nov 03 00:54:28.691897 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:54:28.691819 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 00:54:29.483558 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:54:29.483526 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 00:54:30.664933 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:54:30.662828 4939 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-2f9a3a/simulator-w0-2f9a3a-kserve-9596bbb8f-2trgz" podUID="207e2207-fb6c-4076-95df-72e95671c261" containerName="main" probeResult="failure" output="Get \"https://10.132.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 00:54:30.797933 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:54:30.794027 4939 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-83755a/distinct-2-w3-83755a-kserve-6cf66b995d-2l47b" podUID="c3b05e88-5d5b-4d03-b28d-832f22e3c868" containerName="main" probeResult="failure" output="Get \"https://10.132.0.72:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 00:54:30.833944 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:54:30.833889 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 00:54:31.058090 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:54:31.016674 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 00:54:31.058090 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:54:31.016818 4939 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-f7680d/premium-w2-f7680d-kserve-76565b8cf9-hqbbt" podUID="4712e708-18a2-4a13-a12c-a5669397b2c3" containerName="main" probeResult="failure" output="Get \"https://10.132.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 00:54:31.058090 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:54:31.049530 4939 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-f7680d/distinct-2-w2-f7680d-kserve-7d7c4564bf-cbv2c" podUID="2de18086-761b-4773-b9fc-15ddc4a220fc" containerName="main" probeResult="failure" output="Get \"https://10.132.0.75:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 00:54:31.290105 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:54:31.290023 4939 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-88bbf7/simulator-w1-88bbf7-kserve-5fbb669f8f-42hgd" podUID="83b38883-2e7a-444d-aa68-52be4a25ae66" containerName="main" probeResult="failure" output="Get \"https://10.132.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 00:54:31.405095 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:54:31.405031 4939 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-f7680d/distinct-w2-f7680d-kserve-756f874b98-qnpv8" podUID="836753fc-f9ef-4f61-b3fe-e5b92b7ee27b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.70:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 00:54:31.405836 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:54:31.405788 4939 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-2f9a3a/distinct-w0-2f9a3a-kserve-555565cfcf-jtcr9" podUID="ac0ece06-8718-463f-915f-5315988d9711" containerName="main" probeResult="failure" output="Get \"https://10.132.0.76:8000/ready\": context deadline exceeded" Nov 03 00:54:31.676137 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:54:31.676028 4939 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-2f9a3a/simulator-w0-2f9a3a-kserve-9596bbb8f-2trgz" podUID="207e2207-fb6c-4076-95df-72e95671c261" containerName="main" probeResult="failure" output="Get \"https://10.132.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 00:54:31.677424 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:54:31.677398 4939 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-2f9a3a/simulator-w0-2f9a3a-kserve-9596bbb8f-2trgz" Nov 03 00:54:31.991219 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:54:31.991082 4939 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-83755a/distinct-2-w3-83755a-kserve-6cf66b995d-2l47b" podUID="c3b05e88-5d5b-4d03-b28d-832f22e3c868" containerName="main" probeResult="failure" output="Get \"https://10.132.0.72:8000/ready\": context deadline exceeded" Nov 03 00:54:32.167506 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:54:32.056583 4939 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-f7680d/distinct-2-w2-f7680d-kserve-7d7c4564bf-cbv2c" podUID="2de18086-761b-4773-b9fc-15ddc4a220fc" containerName="main" probeResult="failure" output="Get \"https://10.132.0.75:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 00:54:32.201800 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:54:32.124649 4939 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-f7680d/premium-w2-f7680d-kserve-76565b8cf9-hqbbt" podUID="4712e708-18a2-4a13-a12c-a5669397b2c3" containerName="main" probeResult="failure" output="Get \"https://10.132.0.63:8000/ready\": context deadline exceeded" Nov 03 00:54:32.264985 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:54:32.244009 4939 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-f7680d/premium-w2-f7680d-kserve-76565b8cf9-hqbbt" podUID="4712e708-18a2-4a13-a12c-a5669397b2c3" containerName="main" probeResult="failure" output="Get \"https://10.132.0.63:8000/health\": context deadline exceeded" Nov 03 00:54:32.290934 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:54:32.290619 4939 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-f7680d/distinct-2-w2-f7680d-kserve-7d7c4564bf-cbv2c" podUID="2de18086-761b-4773-b9fc-15ddc4a220fc" containerName="main" probeResult="failure" output="Get \"https://10.132.0.75:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 00:54:32.291122 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:54:32.291005 4939 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-88bbf7/simulator-w1-88bbf7-kserve-5fbb669f8f-42hgd" podUID="83b38883-2e7a-444d-aa68-52be4a25ae66" containerName="main" probeResult="failure" output="Get \"https://10.132.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 00:54:32.491934 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:54:32.433195 4939 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-2f9a3a/distinct-w0-2f9a3a-kserve-555565cfcf-jtcr9" podUID="ac0ece06-8718-463f-915f-5315988d9711" containerName="main" probeResult="failure" output="Get \"https://10.132.0.76:8000/ready\": context deadline exceeded" Nov 03 00:54:32.774229 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:54:32.774158 4939 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-2f9a3a/simulator-w0-2f9a3a-kserve-9596bbb8f-2trgz" podUID="207e2207-fb6c-4076-95df-72e95671c261" containerName="main" probeResult="failure" output="Get \"https://10.132.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 00:54:32.849539 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:54:32.849501 4939 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-83755a/distinct-2-w3-83755a-kserve-6cf66b995d-2l47b" Nov 03 00:54:32.859393 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:54:32.859346 4939 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-f7680d/distinct-2-w2-f7680d-kserve-7d7c4564bf-cbv2c" Nov 03 00:54:32.859796 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:54:32.859757 4939 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-f7680d/premium-w2-f7680d-kserve-76565b8cf9-hqbbt" Nov 03 00:54:32.871038 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:54:32.870987 4939 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-88bbf7/simulator-w1-88bbf7-kserve-5fbb669f8f-42hgd" Nov 03 00:54:32.871416 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:54:32.871378 4939 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-2f9a3a/distinct-w0-2f9a3a-kserve-555565cfcf-jtcr9" Nov 03 00:54:33.028302 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:54:33.028202 4939 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-83755a/distinct-2-w3-83755a-kserve-6cf66b995d-2l47b" podUID="c3b05e88-5d5b-4d03-b28d-832f22e3c868" containerName="main" probeResult="failure" output="Get \"https://10.132.0.72:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 00:54:33.260957 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:54:33.230406 4939 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-f7680d/distinct-2-w2-f7680d-kserve-7d7c4564bf-cbv2c" podUID="2de18086-761b-4773-b9fc-15ddc4a220fc" containerName="main" probeResult="failure" output="Get \"https://10.132.0.75:8000/ready\": context deadline exceeded" Nov 03 00:54:33.262101 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:54:33.262050 4939 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-f7680d/premium-w2-f7680d-kserve-76565b8cf9-hqbbt" podUID="4712e708-18a2-4a13-a12c-a5669397b2c3" containerName="main" probeResult="failure" output="Get \"https://10.132.0.63:8000/ready\": context deadline exceeded" Nov 03 00:54:33.287129 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:54:33.287042 4939 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-2f9a3a-openshift-default-7f6f74cf97-j8q97"] Nov 03 00:54:33.313743 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:54:33.301808 4939 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-88bbf7/simulator-w1-88bbf7-kserve-5fbb669f8f-42hgd" podUID="83b38883-2e7a-444d-aa68-52be4a25ae66" containerName="main" probeResult="failure" output="Get \"https://10.132.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 00:54:33.517939 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:54:33.505286 4939 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-2f9a3a-openshift-default-7f6f74cf97-j8q97" Nov 03 00:54:33.526081 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:54:33.526039 4939 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-2f9a3a-openshift-default-7f6f74cf97-j8q97"] Nov 03 00:54:33.656423 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:54:33.656381 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 00:54:33.675934 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:54:33.672208 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d96472d8-fbad-44c0-815d-bf4d20cce46a-istio-envoy\") pod \"e2e-worker-w0-2f9a3a-openshift-default-7f6f74cf97-j8q97\" (UID: \"d96472d8-fbad-44c0-815d-bf4d20cce46a\") " pod="openshift-ingress/e2e-worker-w0-2f9a3a-openshift-default-7f6f74cf97-j8q97" Nov 03 00:54:33.680088 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:54:33.680048 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d96472d8-fbad-44c0-815d-bf4d20cce46a-istio-podinfo\") pod \"e2e-worker-w0-2f9a3a-openshift-default-7f6f74cf97-j8q97\" (UID: \"d96472d8-fbad-44c0-815d-bf4d20cce46a\") " pod="openshift-ingress/e2e-worker-w0-2f9a3a-openshift-default-7f6f74cf97-j8q97" Nov 03 00:54:33.680350 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:54:33.680327 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d96472d8-fbad-44c0-815d-bf4d20cce46a-istiod-ca-cert\") pod \"e2e-worker-w0-2f9a3a-openshift-default-7f6f74cf97-j8q97\" (UID: \"d96472d8-fbad-44c0-815d-bf4d20cce46a\") " pod="openshift-ingress/e2e-worker-w0-2f9a3a-openshift-default-7f6f74cf97-j8q97" Nov 03 00:54:33.680513 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:54:33.680493 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d96472d8-fbad-44c0-815d-bf4d20cce46a-credential-socket\") pod \"e2e-worker-w0-2f9a3a-openshift-default-7f6f74cf97-j8q97\" (UID: \"d96472d8-fbad-44c0-815d-bf4d20cce46a\") " pod="openshift-ingress/e2e-worker-w0-2f9a3a-openshift-default-7f6f74cf97-j8q97" Nov 03 00:54:33.680741 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:54:33.680722 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-srpmf\" (UniqueName: \"kubernetes.io/projected/d96472d8-fbad-44c0-815d-bf4d20cce46a-kube-api-access-srpmf\") pod \"e2e-worker-w0-2f9a3a-openshift-default-7f6f74cf97-j8q97\" (UID: \"d96472d8-fbad-44c0-815d-bf4d20cce46a\") " pod="openshift-ingress/e2e-worker-w0-2f9a3a-openshift-default-7f6f74cf97-j8q97" Nov 03 00:54:33.680966 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:54:33.680945 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d96472d8-fbad-44c0-815d-bf4d20cce46a-istio-data\") pod \"e2e-worker-w0-2f9a3a-openshift-default-7f6f74cf97-j8q97\" (UID: \"d96472d8-fbad-44c0-815d-bf4d20cce46a\") " pod="openshift-ingress/e2e-worker-w0-2f9a3a-openshift-default-7f6f74cf97-j8q97" Nov 03 00:54:33.685037 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:54:33.685007 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d96472d8-fbad-44c0-815d-bf4d20cce46a-workload-socket\") pod \"e2e-worker-w0-2f9a3a-openshift-default-7f6f74cf97-j8q97\" (UID: \"d96472d8-fbad-44c0-815d-bf4d20cce46a\") " pod="openshift-ingress/e2e-worker-w0-2f9a3a-openshift-default-7f6f74cf97-j8q97" Nov 03 00:54:33.685234 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:54:33.685216 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d96472d8-fbad-44c0-815d-bf4d20cce46a-workload-certs\") pod \"e2e-worker-w0-2f9a3a-openshift-default-7f6f74cf97-j8q97\" (UID: \"d96472d8-fbad-44c0-815d-bf4d20cce46a\") " pod="openshift-ingress/e2e-worker-w0-2f9a3a-openshift-default-7f6f74cf97-j8q97" Nov 03 00:54:33.706935 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:54:33.685386 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d96472d8-fbad-44c0-815d-bf4d20cce46a-istio-token\") pod \"e2e-worker-w0-2f9a3a-openshift-default-7f6f74cf97-j8q97\" (UID: \"d96472d8-fbad-44c0-815d-bf4d20cce46a\") " pod="openshift-ingress/e2e-worker-w0-2f9a3a-openshift-default-7f6f74cf97-j8q97" Nov 03 00:54:33.727902 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:54:33.727860 4939 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-f7680d/distinct-w2-f7680d-kserve-756f874b98-qnpv8" podUID="836753fc-f9ef-4f61-b3fe-e5b92b7ee27b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.70:8000/ready\": context deadline exceeded" Nov 03 00:54:33.743831 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:54:33.743788 4939 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-2f9a3a/distinct-w0-2f9a3a-kserve-555565cfcf-jtcr9" podUID="ac0ece06-8718-463f-915f-5315988d9711" containerName="main" probeResult="failure" output="Get \"https://10.132.0.76:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 00:54:33.790335 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:54:33.790290 4939 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-2f9a3a/simulator-w0-2f9a3a-kserve-9596bbb8f-2trgz" podUID="207e2207-fb6c-4076-95df-72e95671c261" containerName="main" probeResult="failure" output="Get \"https://10.132.0.61:8000/ready\": context deadline exceeded" Nov 03 00:54:33.819811 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:54:33.819767 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-srpmf\" (UniqueName: \"kubernetes.io/projected/d96472d8-fbad-44c0-815d-bf4d20cce46a-kube-api-access-srpmf\") pod \"e2e-worker-w0-2f9a3a-openshift-default-7f6f74cf97-j8q97\" (UID: \"d96472d8-fbad-44c0-815d-bf4d20cce46a\") " pod="openshift-ingress/e2e-worker-w0-2f9a3a-openshift-default-7f6f74cf97-j8q97" Nov 03 00:54:33.827668 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:54:33.827637 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d96472d8-fbad-44c0-815d-bf4d20cce46a-istio-data\") pod \"e2e-worker-w0-2f9a3a-openshift-default-7f6f74cf97-j8q97\" (UID: \"d96472d8-fbad-44c0-815d-bf4d20cce46a\") " pod="openshift-ingress/e2e-worker-w0-2f9a3a-openshift-default-7f6f74cf97-j8q97" Nov 03 00:54:33.835324 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:54:33.835293 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d96472d8-fbad-44c0-815d-bf4d20cce46a-workload-socket\") pod \"e2e-worker-w0-2f9a3a-openshift-default-7f6f74cf97-j8q97\" (UID: \"d96472d8-fbad-44c0-815d-bf4d20cce46a\") " pod="openshift-ingress/e2e-worker-w0-2f9a3a-openshift-default-7f6f74cf97-j8q97" Nov 03 00:54:33.895212 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:54:33.828767 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d96472d8-fbad-44c0-815d-bf4d20cce46a-istio-data\") pod \"e2e-worker-w0-2f9a3a-openshift-default-7f6f74cf97-j8q97\" (UID: \"d96472d8-fbad-44c0-815d-bf4d20cce46a\") " pod="openshift-ingress/e2e-worker-w0-2f9a3a-openshift-default-7f6f74cf97-j8q97" Nov 03 00:54:33.922138 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:54:33.847686 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d96472d8-fbad-44c0-815d-bf4d20cce46a-workload-socket\") pod \"e2e-worker-w0-2f9a3a-openshift-default-7f6f74cf97-j8q97\" (UID: \"d96472d8-fbad-44c0-815d-bf4d20cce46a\") " pod="openshift-ingress/e2e-worker-w0-2f9a3a-openshift-default-7f6f74cf97-j8q97" Nov 03 00:54:33.922372 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:54:33.906964 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d96472d8-fbad-44c0-815d-bf4d20cce46a-workload-certs\") pod \"e2e-worker-w0-2f9a3a-openshift-default-7f6f74cf97-j8q97\" (UID: \"d96472d8-fbad-44c0-815d-bf4d20cce46a\") " pod="openshift-ingress/e2e-worker-w0-2f9a3a-openshift-default-7f6f74cf97-j8q97" Nov 03 00:54:33.945082 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:54:33.945044 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d96472d8-fbad-44c0-815d-bf4d20cce46a-istio-token\") pod \"e2e-worker-w0-2f9a3a-openshift-default-7f6f74cf97-j8q97\" (UID: \"d96472d8-fbad-44c0-815d-bf4d20cce46a\") " pod="openshift-ingress/e2e-worker-w0-2f9a3a-openshift-default-7f6f74cf97-j8q97" Nov 03 00:54:33.945431 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:54:33.945408 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d96472d8-fbad-44c0-815d-bf4d20cce46a-istio-envoy\") pod \"e2e-worker-w0-2f9a3a-openshift-default-7f6f74cf97-j8q97\" (UID: \"d96472d8-fbad-44c0-815d-bf4d20cce46a\") " pod="openshift-ingress/e2e-worker-w0-2f9a3a-openshift-default-7f6f74cf97-j8q97" Nov 03 00:54:33.945628 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:54:33.945611 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d96472d8-fbad-44c0-815d-bf4d20cce46a-istio-podinfo\") pod \"e2e-worker-w0-2f9a3a-openshift-default-7f6f74cf97-j8q97\" (UID: \"d96472d8-fbad-44c0-815d-bf4d20cce46a\") " pod="openshift-ingress/e2e-worker-w0-2f9a3a-openshift-default-7f6f74cf97-j8q97" Nov 03 00:54:33.945775 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:54:33.945756 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d96472d8-fbad-44c0-815d-bf4d20cce46a-istiod-ca-cert\") pod \"e2e-worker-w0-2f9a3a-openshift-default-7f6f74cf97-j8q97\" (UID: \"d96472d8-fbad-44c0-815d-bf4d20cce46a\") " pod="openshift-ingress/e2e-worker-w0-2f9a3a-openshift-default-7f6f74cf97-j8q97" Nov 03 00:54:33.945935 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:54:33.945899 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d96472d8-fbad-44c0-815d-bf4d20cce46a-credential-socket\") pod \"e2e-worker-w0-2f9a3a-openshift-default-7f6f74cf97-j8q97\" (UID: \"d96472d8-fbad-44c0-815d-bf4d20cce46a\") " pod="openshift-ingress/e2e-worker-w0-2f9a3a-openshift-default-7f6f74cf97-j8q97" Nov 03 00:54:33.946575 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:54:33.946552 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d96472d8-fbad-44c0-815d-bf4d20cce46a-credential-socket\") pod \"e2e-worker-w0-2f9a3a-openshift-default-7f6f74cf97-j8q97\" (UID: \"d96472d8-fbad-44c0-815d-bf4d20cce46a\") " pod="openshift-ingress/e2e-worker-w0-2f9a3a-openshift-default-7f6f74cf97-j8q97" Nov 03 00:54:33.973006 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:54:33.907283 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d96472d8-fbad-44c0-815d-bf4d20cce46a-workload-certs\") pod \"e2e-worker-w0-2f9a3a-openshift-default-7f6f74cf97-j8q97\" (UID: \"d96472d8-fbad-44c0-815d-bf4d20cce46a\") " pod="openshift-ingress/e2e-worker-w0-2f9a3a-openshift-default-7f6f74cf97-j8q97" Nov 03 00:54:33.973237 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:54:33.956898 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d96472d8-fbad-44c0-815d-bf4d20cce46a-istiod-ca-cert\") pod \"e2e-worker-w0-2f9a3a-openshift-default-7f6f74cf97-j8q97\" (UID: \"d96472d8-fbad-44c0-815d-bf4d20cce46a\") " pod="openshift-ingress/e2e-worker-w0-2f9a3a-openshift-default-7f6f74cf97-j8q97" Nov 03 00:54:33.996659 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:54:33.996517 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d96472d8-fbad-44c0-815d-bf4d20cce46a-istio-token\") pod \"e2e-worker-w0-2f9a3a-openshift-default-7f6f74cf97-j8q97\" (UID: \"d96472d8-fbad-44c0-815d-bf4d20cce46a\") " pod="openshift-ingress/e2e-worker-w0-2f9a3a-openshift-default-7f6f74cf97-j8q97" Nov 03 00:54:34.032421 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:54:34.032364 4939 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-83755a/distinct-2-w3-83755a-kserve-6cf66b995d-2l47b" podUID="c3b05e88-5d5b-4d03-b28d-832f22e3c868" containerName="main" probeResult="failure" output="Get \"https://10.132.0.72:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 00:54:34.048548 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:54:34.048511 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d96472d8-fbad-44c0-815d-bf4d20cce46a-istio-envoy\") pod \"e2e-worker-w0-2f9a3a-openshift-default-7f6f74cf97-j8q97\" (UID: \"d96472d8-fbad-44c0-815d-bf4d20cce46a\") " pod="openshift-ingress/e2e-worker-w0-2f9a3a-openshift-default-7f6f74cf97-j8q97" Nov 03 00:54:34.114983 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:54:34.114889 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-srpmf\" (UniqueName: \"kubernetes.io/projected/d96472d8-fbad-44c0-815d-bf4d20cce46a-kube-api-access-srpmf\") pod \"e2e-worker-w0-2f9a3a-openshift-default-7f6f74cf97-j8q97\" (UID: \"d96472d8-fbad-44c0-815d-bf4d20cce46a\") " pod="openshift-ingress/e2e-worker-w0-2f9a3a-openshift-default-7f6f74cf97-j8q97" Nov 03 00:54:34.127889 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:54:34.127853 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d96472d8-fbad-44c0-815d-bf4d20cce46a-istio-podinfo\") pod \"e2e-worker-w0-2f9a3a-openshift-default-7f6f74cf97-j8q97\" (UID: \"d96472d8-fbad-44c0-815d-bf4d20cce46a\") " pod="openshift-ingress/e2e-worker-w0-2f9a3a-openshift-default-7f6f74cf97-j8q97" Nov 03 00:54:34.274015 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:54:34.250040 4939 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-f7680d/distinct-2-w2-f7680d-kserve-7d7c4564bf-cbv2c" podUID="2de18086-761b-4773-b9fc-15ddc4a220fc" containerName="main" probeResult="failure" output="Get \"https://10.132.0.75:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 00:54:34.307935 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:54:34.276388 4939 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-2f9a3a/distinct-w0-2f9a3a-kserve-555565cfcf-jtcr9" podUID="ac0ece06-8718-463f-915f-5315988d9711" containerName="main" probeResult="failure" output="Get \"https://10.132.0.76:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 00:54:34.333238 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:54:34.289348 4939 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-f7680d/premium-w2-f7680d-kserve-76565b8cf9-hqbbt" podUID="4712e708-18a2-4a13-a12c-a5669397b2c3" containerName="main" probeResult="failure" output="Get \"https://10.132.0.63:8000/ready\": context deadline exceeded" Nov 03 00:54:34.354934 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:54:34.308299 4939 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-88bbf7/simulator-w1-88bbf7-kserve-5fbb669f8f-42hgd" podUID="83b38883-2e7a-444d-aa68-52be4a25ae66" containerName="main" probeResult="failure" output="Get \"https://10.132.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 00:54:34.380343 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:54:34.345454 4939 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-2f9a3a-openshift-default-7f6f74cf97-j8q97" Nov 03 00:54:34.734021 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:54:34.733888 4939 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-f7680d/distinct-w2-f7680d-kserve-756f874b98-qnpv8" podUID="836753fc-f9ef-4f61-b3fe-e5b92b7ee27b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.70:8000/ready\": context deadline exceeded" Nov 03 00:54:34.735309 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:54:34.735279 4939 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-f7680d/distinct-w2-f7680d-kserve-756f874b98-qnpv8" Nov 03 00:54:34.791936 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:54:34.791249 4939 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-2f9a3a/distinct-w0-2f9a3a-kserve-555565cfcf-jtcr9" podUID="ac0ece06-8718-463f-915f-5315988d9711" containerName="main" probeResult="failure" output="Get \"https://10.132.0.76:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 00:54:34.796678 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:54:34.796638 4939 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-2f9a3a/simulator-w0-2f9a3a-kserve-9596bbb8f-2trgz" podUID="207e2207-fb6c-4076-95df-72e95671c261" containerName="main" probeResult="failure" output="Get \"https://10.132.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 00:54:35.044189 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:54:35.044143 4939 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-83755a/distinct-2-w3-83755a-kserve-6cf66b995d-2l47b" podUID="c3b05e88-5d5b-4d03-b28d-832f22e3c868" containerName="main" probeResult="failure" output="Get \"https://10.132.0.72:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 00:54:35.275055 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:54:35.274976 4939 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-f7680d/distinct-2-w2-f7680d-kserve-7d7c4564bf-cbv2c" podUID="2de18086-761b-4773-b9fc-15ddc4a220fc" containerName="main" probeResult="failure" output="Get \"https://10.132.0.75:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 00:54:35.335181 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:54:35.334010 4939 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-f7680d/premium-w2-f7680d-kserve-76565b8cf9-hqbbt" podUID="4712e708-18a2-4a13-a12c-a5669397b2c3" containerName="main" probeResult="failure" output="Get \"https://10.132.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 00:54:35.389986 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:54:35.383008 4939 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-88bbf7/simulator-w1-88bbf7-kserve-5fbb669f8f-42hgd" podUID="83b38883-2e7a-444d-aa68-52be4a25ae66" containerName="main" probeResult="failure" output="Get \"https://10.132.0.62:8000/ready\": context deadline exceeded" Nov 03 00:54:35.421935 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:54:35.421257 4939 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-f7680d/distinct-w2-f7680d-kserve-756f874b98-qnpv8" Nov 03 00:54:35.592216 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:54:35.592189 4939 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-2f9a3a/distinct-w0-2f9a3a-kserve-555565cfcf-jtcr9" Nov 03 00:54:35.702874 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:54:35.701177 4939 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-83755a/distinct-2-w3-83755a-kserve-6cf66b995d-2l47b" Nov 03 00:54:35.708209 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:54:35.706078 4939 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-88bbf7/simulator-w1-88bbf7-kserve-5fbb669f8f-42hgd" Nov 03 00:54:35.710973 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:54:35.710943 4939 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-f7680d/distinct-2-w2-f7680d-kserve-7d7c4564bf-cbv2c" Nov 03 00:54:35.718748 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:54:35.718726 4939 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-f7680d/premium-w2-f7680d-kserve-76565b8cf9-hqbbt" Nov 03 00:54:35.731675 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:54:35.731647 4939 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-2f9a3a/simulator-w0-2f9a3a-kserve-9596bbb8f-2trgz" Nov 03 00:54:35.848377 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:54:35.848350 4939 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-2f9a3a-openshift-default-7f6f74cf97-j8q97"] Nov 03 00:54:35.849646 ip-10-0-140-107 kubenswrapper[4939]: W1103 00:54:35.849615 4939 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd96472d8_fbad_44c0_815d_bf4d20cce46a.slice/crio-0981a3c88659ffd9728cd5def1d780ab4f6fe94232f1971ff22fb86b58b7b4ea WatchSource:0}: Error finding container 0981a3c88659ffd9728cd5def1d780ab4f6fe94232f1971ff22fb86b58b7b4ea: Status 404 returned error can't find the container with id 0981a3c88659ffd9728cd5def1d780ab4f6fe94232f1971ff22fb86b58b7b4ea Nov 03 00:54:35.852436 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:54:35.852388 4939 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 00:54:35.852529 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:54:35.852499 4939 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 00:54:35.852615 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:54:35.852552 4939 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 00:54:36.084195 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:54:36.084138 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 00:54:36.322072 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:54:36.321974 4939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-2f9a3a-openshift-default-7f6f74cf97-j8q97" event={"ID":"d96472d8-fbad-44c0-815d-bf4d20cce46a","Type":"ContainerStarted","Data":"1f2e9fa18ac7e6c6c904f4612ece64a27ae0a4c622fa0970018363af08e3bc92"} Nov 03 00:54:36.322242 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:54:36.322086 4939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-2f9a3a-openshift-default-7f6f74cf97-j8q97" event={"ID":"d96472d8-fbad-44c0-815d-bf4d20cce46a","Type":"ContainerStarted","Data":"0981a3c88659ffd9728cd5def1d780ab4f6fe94232f1971ff22fb86b58b7b4ea"} Nov 03 00:54:36.342706 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:54:36.342519 4939 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-2f9a3a-openshift-default-7f6f74cf97-j8q97" podStartSLOduration=4.342499304 podStartE2EDuration="4.342499304s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:54:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:54:36.341933326 +0000 UTC m=+824.606268272" watchObservedRunningTime="2026-11-03 00:54:36.342499304 +0000 UTC m=+824.606834250" Nov 03 00:54:36.346021 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:54:36.345955 4939 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-2f9a3a-openshift-default-7f6f74cf97-j8q97" Nov 03 00:54:36.347283 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:54:36.347258 4939 patch_prober.go:29] interesting pod/e2e-worker-w0-2f9a3a-openshift-default-7f6f74cf97-j8q97 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.83:15021/healthz/ready\": dial tcp 10.132.0.83:15021: connect: connection refused" start-of-body= Nov 03 00:54:36.347394 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:54:36.347305 4939 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-2f9a3a-openshift-default-7f6f74cf97-j8q97" podUID="d96472d8-fbad-44c0-815d-bf4d20cce46a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.83:15021/healthz/ready\": dial tcp 10.132.0.83:15021: connect: connection refused" Nov 03 00:54:36.504149 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:54:36.504109 4939 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-05e6bd51-ht7qq"] Nov 03 00:54:36.508367 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:54:36.508341 4939 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-05e6bd51-ht7qq" Nov 03 00:54:36.512372 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:54:36.512337 4939 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 00:54:36.512532 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:54:36.512381 4939 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-6q4xp\"" Nov 03 00:54:36.515851 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:54:36.515826 4939 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-05e6bd51-ht7qq"] Nov 03 00:54:36.545304 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:54:36.545264 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jxs98\" (UniqueName: \"kubernetes.io/projected/3be51f99-989b-4efd-a563-e2a985522a3e-kube-api-access-jxs98\") pod \"maas-api-revoke-sub-05e6bd51-ht7qq\" (UID: \"3be51f99-989b-4efd-a563-e2a985522a3e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-05e6bd51-ht7qq" Nov 03 00:54:36.545485 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:54:36.545341 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3be51f99-989b-4efd-a563-e2a985522a3e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-05e6bd51-ht7qq\" (UID: \"3be51f99-989b-4efd-a563-e2a985522a3e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-05e6bd51-ht7qq" Nov 03 00:54:36.545555 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:54:36.545534 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3be51f99-989b-4efd-a563-e2a985522a3e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-05e6bd51-ht7qq\" (UID: \"3be51f99-989b-4efd-a563-e2a985522a3e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-05e6bd51-ht7qq" Nov 03 00:54:36.647690 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:54:36.647650 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3be51f99-989b-4efd-a563-e2a985522a3e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-05e6bd51-ht7qq\" (UID: \"3be51f99-989b-4efd-a563-e2a985522a3e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-05e6bd51-ht7qq" Nov 03 00:54:36.648852 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:54:36.648828 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jxs98\" (UniqueName: \"kubernetes.io/projected/3be51f99-989b-4efd-a563-e2a985522a3e-kube-api-access-jxs98\") pod \"maas-api-revoke-sub-05e6bd51-ht7qq\" (UID: \"3be51f99-989b-4efd-a563-e2a985522a3e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-05e6bd51-ht7qq" Nov 03 00:54:36.656428 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:54:36.656387 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3be51f99-989b-4efd-a563-e2a985522a3e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-05e6bd51-ht7qq\" (UID: \"3be51f99-989b-4efd-a563-e2a985522a3e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-05e6bd51-ht7qq" Nov 03 00:54:36.681248 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:54:36.648684 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3be51f99-989b-4efd-a563-e2a985522a3e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-05e6bd51-ht7qq\" (UID: \"3be51f99-989b-4efd-a563-e2a985522a3e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-05e6bd51-ht7qq" Nov 03 00:54:36.681248 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:54:36.680631 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3be51f99-989b-4efd-a563-e2a985522a3e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-05e6bd51-ht7qq\" (UID: \"3be51f99-989b-4efd-a563-e2a985522a3e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-05e6bd51-ht7qq" Nov 03 00:54:36.681407 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:54:36.681331 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jxs98\" (UniqueName: \"kubernetes.io/projected/3be51f99-989b-4efd-a563-e2a985522a3e-kube-api-access-jxs98\") pod \"maas-api-revoke-sub-05e6bd51-ht7qq\" (UID: \"3be51f99-989b-4efd-a563-e2a985522a3e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-05e6bd51-ht7qq" Nov 03 00:54:36.825391 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:54:36.825357 4939 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-05e6bd51-ht7qq" Nov 03 00:54:36.999609 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:54:36.999576 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 00:54:37.138120 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:54:37.138093 4939 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-05e6bd51-ht7qq"] Nov 03 00:54:37.139549 ip-10-0-140-107 kubenswrapper[4939]: W1103 00:54:37.139523 4939 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3be51f99_989b_4efd_a563_e2a985522a3e.slice/crio-99bdd3712429a4dae95a3a5a8e785de522a036acc78be6ef4c846e2aeb4019be WatchSource:0}: Error finding container 99bdd3712429a4dae95a3a5a8e785de522a036acc78be6ef4c846e2aeb4019be: Status 404 returned error can't find the container with id 99bdd3712429a4dae95a3a5a8e785de522a036acc78be6ef4c846e2aeb4019be Nov 03 00:54:37.329673 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:54:37.329631 4939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-05e6bd51-ht7qq" event={"ID":"3be51f99-989b-4efd-a563-e2a985522a3e","Type":"ContainerStarted","Data":"99bdd3712429a4dae95a3a5a8e785de522a036acc78be6ef4c846e2aeb4019be"} Nov 03 00:54:37.371146 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:54:37.371105 4939 patch_prober.go:29] interesting pod/e2e-worker-w0-2f9a3a-openshift-default-7f6f74cf97-j8q97 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.83:15021/healthz/ready\": dial tcp 10.132.0.83:15021: connect: connection refused" start-of-body= Nov 03 00:54:37.371389 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:54:37.371362 4939 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-2f9a3a-openshift-default-7f6f74cf97-j8q97" podUID="d96472d8-fbad-44c0-815d-bf4d20cce46a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.83:15021/healthz/ready\": dial tcp 10.132.0.83:15021: connect: connection refused" Nov 03 00:54:37.873398 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:54:37.873347 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 00:54:38.344399 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:54:38.344355 4939 generic.go:356] "Generic (PLEG): container finished" podID="3be51f99-989b-4efd-a563-e2a985522a3e" containerID="172c051c8692aa520fdb4336cf232a2c6826a830ed044c0a56c765fc2ee2fa1c" exitCode=0 Nov 03 00:54:38.344792 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:54:38.344437 4939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-05e6bd51-ht7qq" event={"ID":"3be51f99-989b-4efd-a563-e2a985522a3e","Type":"ContainerDied","Data":"172c051c8692aa520fdb4336cf232a2c6826a830ed044c0a56c765fc2ee2fa1c"} Nov 03 00:54:38.367931 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:54:38.345972 4939 patch_prober.go:29] interesting pod/e2e-worker-w0-2f9a3a-openshift-default-7f6f74cf97-j8q97 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.83:15021/healthz/ready\": dial tcp 10.132.0.83:15021: connect: connection refused" start-of-body= Nov 03 00:54:38.367931 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:54:38.346051 4939 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-2f9a3a-openshift-default-7f6f74cf97-j8q97" podUID="d96472d8-fbad-44c0-815d-bf4d20cce46a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.83:15021/healthz/ready\": dial tcp 10.132.0.83:15021: connect: connection refused" Nov 03 00:54:38.622687 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:54:38.622651 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 00:54:39.362993 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:54:39.347352 4939 patch_prober.go:29] interesting pod/e2e-worker-w0-2f9a3a-openshift-default-7f6f74cf97-j8q97 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.83:15021/healthz/ready\": dial tcp 10.132.0.83:15021: connect: connection refused" start-of-body= Nov 03 00:54:39.362993 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:54:39.347411 4939 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-2f9a3a-openshift-default-7f6f74cf97-j8q97" podUID="d96472d8-fbad-44c0-815d-bf4d20cce46a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.83:15021/healthz/ready\": dial tcp 10.132.0.83:15021: connect: connection refused" Nov 03 00:54:39.401236 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:54:39.401203 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 00:54:39.678923 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:54:39.678889 4939 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-05e6bd51-ht7qq" Nov 03 00:54:39.687678 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:54:39.687651 4939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3be51f99-989b-4efd-a563-e2a985522a3e-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3be51f99-989b-4efd-a563-e2a985522a3e-maas-api-xks-ca\") pod \"3be51f99-989b-4efd-a563-e2a985522a3e\" (UID: \"3be51f99-989b-4efd-a563-e2a985522a3e\") " Nov 03 00:54:39.687778 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:54:39.687744 4939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3be51f99-989b-4efd-a563-e2a985522a3e-kube-api-access-jxs98\" (UniqueName: \"kubernetes.io/projected/3be51f99-989b-4efd-a563-e2a985522a3e-kube-api-access-jxs98\") pod \"3be51f99-989b-4efd-a563-e2a985522a3e\" (UID: \"3be51f99-989b-4efd-a563-e2a985522a3e\") " Nov 03 00:54:39.687872 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:54:39.687861 4939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3be51f99-989b-4efd-a563-e2a985522a3e-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3be51f99-989b-4efd-a563-e2a985522a3e-maas-api-openshift-ca\") pod \"3be51f99-989b-4efd-a563-e2a985522a3e\" (UID: \"3be51f99-989b-4efd-a563-e2a985522a3e\") " Nov 03 00:54:39.688264 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:54:39.688237 4939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3be51f99-989b-4efd-a563-e2a985522a3e-maas-api-openshift-ca" pod "3be51f99-989b-4efd-a563-e2a985522a3e" (UID: "3be51f99-989b-4efd-a563-e2a985522a3e"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 00:54:39.688422 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:54:39.688406 4939 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3be51f99-989b-4efd-a563-e2a985522a3e-maas-api-openshift-ca\") on node \"ip-10-0-140-107.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 00:54:39.689765 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:54:39.689742 4939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3be51f99-989b-4efd-a563-e2a985522a3e-maas-api-xks-ca" pod "3be51f99-989b-4efd-a563-e2a985522a3e" (UID: "3be51f99-989b-4efd-a563-e2a985522a3e"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 00:54:39.689946 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:54:39.689896 4939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3be51f99-989b-4efd-a563-e2a985522a3e-kube-api-access-jxs98" pod "3be51f99-989b-4efd-a563-e2a985522a3e" (UID: "3be51f99-989b-4efd-a563-e2a985522a3e"). InnerVolumeSpecName "kube-api-access-jxs98". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 00:54:39.789222 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:54:39.789186 4939 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3be51f99-989b-4efd-a563-e2a985522a3e-maas-api-xks-ca\") on node \"ip-10-0-140-107.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 00:54:39.789222 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:54:39.789217 4939 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jxs98\" (UniqueName: \"kubernetes.io/projected/3be51f99-989b-4efd-a563-e2a985522a3e-kube-api-access-jxs98\") on node \"ip-10-0-140-107.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 00:54:40.305006 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:54:40.304900 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 00:54:40.355529 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:54:40.355050 4939 patch_prober.go:29] interesting pod/e2e-worker-w0-2f9a3a-openshift-default-7f6f74cf97-j8q97 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.83:15021/healthz/ready\": dial tcp 10.132.0.83:15021: connect: connection refused" start-of-body= Nov 03 00:54:40.355529 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:54:40.355123 4939 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-2f9a3a-openshift-default-7f6f74cf97-j8q97" podUID="d96472d8-fbad-44c0-815d-bf4d20cce46a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.83:15021/healthz/ready\": dial tcp 10.132.0.83:15021: connect: connection refused" Nov 03 00:54:40.431972 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:54:40.430615 4939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-05e6bd51-ht7qq" event={"ID":"3be51f99-989b-4efd-a563-e2a985522a3e","Type":"ContainerDied","Data":"99bdd3712429a4dae95a3a5a8e785de522a036acc78be6ef4c846e2aeb4019be"} Nov 03 00:54:40.431972 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:54:40.430712 4939 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="99bdd3712429a4dae95a3a5a8e785de522a036acc78be6ef4c846e2aeb4019be" Nov 03 00:54:40.431972 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:54:40.430786 4939 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-05e6bd51-ht7qq" Nov 03 00:54:41.016939 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:54:41.016678 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 00:54:41.366933 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:54:41.346005 4939 patch_prober.go:29] interesting pod/e2e-worker-w0-2f9a3a-openshift-default-7f6f74cf97-j8q97 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.83:15021/healthz/ready\": dial tcp 10.132.0.83:15021: connect: connection refused" start-of-body= Nov 03 00:54:41.366933 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:54:41.346067 4939 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-2f9a3a-openshift-default-7f6f74cf97-j8q97" podUID="d96472d8-fbad-44c0-815d-bf4d20cce46a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.83:15021/healthz/ready\": dial tcp 10.132.0.83:15021: connect: connection refused" Nov 03 00:54:41.695252 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:54:41.695166 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 00:54:42.164609 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:54:42.164575 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 00:54:42.346009 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:54:42.345967 4939 patch_prober.go:29] interesting pod/e2e-worker-w0-2f9a3a-openshift-default-7f6f74cf97-j8q97 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.83:15021/healthz/ready\": dial tcp 10.132.0.83:15021: connect: connection refused" start-of-body= Nov 03 00:54:42.346192 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:54:42.346034 4939 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-2f9a3a-openshift-default-7f6f74cf97-j8q97" podUID="d96472d8-fbad-44c0-815d-bf4d20cce46a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.83:15021/healthz/ready\": dial tcp 10.132.0.83:15021: connect: connection refused" Nov 03 00:54:42.485291 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:54:42.485197 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 00:54:43.219440 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:54:43.219378 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 00:54:43.358420 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:54:43.358389 4939 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-2f9a3a-openshift-default-7f6f74cf97-j8q97" Nov 03 00:54:43.358808 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:54:43.358780 4939 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-2f9a3a-openshift-default-7f6f74cf97-j8q97" Nov 03 00:54:43.360861 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:54:43.360837 4939 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-2f9a3a-openshift-default-7f6f74cf97-j8q97" Nov 03 00:54:43.893628 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:54:43.893599 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 00:54:44.055998 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:54:44.055964 4939 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w1-88bbf7/distinct-2-w1-88bbf7-kserve-5457cdcdbd-l6tfd"] Nov 03 00:54:44.056570 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:54:44.056552 4939 cpu_manager.go:414] "containerMap: removing container" podUID="3be51f99-989b-4efd-a563-e2a985522a3e" containerName="revoke-keys" Nov 03 00:54:44.056570 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:54:44.056567 4939 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3be51f99-989b-4efd-a563-e2a985522a3e" containerName="revoke-keys" Nov 03 00:54:44.056743 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:54:44.056655 4939 memory_manager.go:365] "RemoveStaleState removing state" podUID="3be51f99-989b-4efd-a563-e2a985522a3e" containerName="revoke-keys" Nov 03 00:54:44.060482 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:54:44.060389 4939 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-88bbf7/distinct-2-w1-88bbf7-kserve-5457cdcdbd-l6tfd" Nov 03 00:54:44.082166 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:54:44.081592 4939 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w1-88bbf7\"/\"distinct-2-w1-88bbf7-kserve-self-signed-certs\"" Nov 03 00:54:44.082538 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:54:44.082512 4939 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w1-88bbf7/distinct-2-w1-88bbf7-kserve-5457cdcdbd-l6tfd"] Nov 03 00:54:44.095390 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:54:44.095356 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a397931b-faa9-4c2e-8290-491350546663-dshm\") pod \"distinct-2-w1-88bbf7-kserve-5457cdcdbd-l6tfd\" (UID: \"a397931b-faa9-4c2e-8290-491350546663\") " pod="e2e-models-e2e-worker-w1-88bbf7/distinct-2-w1-88bbf7-kserve-5457cdcdbd-l6tfd" Nov 03 00:54:44.095512 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:54:44.095428 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a397931b-faa9-4c2e-8290-491350546663-tls-certs\") pod \"distinct-2-w1-88bbf7-kserve-5457cdcdbd-l6tfd\" (UID: \"a397931b-faa9-4c2e-8290-491350546663\") " pod="e2e-models-e2e-worker-w1-88bbf7/distinct-2-w1-88bbf7-kserve-5457cdcdbd-l6tfd" Nov 03 00:54:44.095569 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:54:44.095554 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a397931b-faa9-4c2e-8290-491350546663-tmp-dir\") pod \"distinct-2-w1-88bbf7-kserve-5457cdcdbd-l6tfd\" (UID: \"a397931b-faa9-4c2e-8290-491350546663\") " pod="e2e-models-e2e-worker-w1-88bbf7/distinct-2-w1-88bbf7-kserve-5457cdcdbd-l6tfd" Nov 03 00:54:44.098244 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:54:44.095687 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vnpdj\" (UniqueName: \"kubernetes.io/projected/a397931b-faa9-4c2e-8290-491350546663-kube-api-access-vnpdj\") pod \"distinct-2-w1-88bbf7-kserve-5457cdcdbd-l6tfd\" (UID: \"a397931b-faa9-4c2e-8290-491350546663\") " pod="e2e-models-e2e-worker-w1-88bbf7/distinct-2-w1-88bbf7-kserve-5457cdcdbd-l6tfd" Nov 03 00:54:44.098244 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:54:44.095828 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a397931b-faa9-4c2e-8290-491350546663-model-cache\") pod \"distinct-2-w1-88bbf7-kserve-5457cdcdbd-l6tfd\" (UID: \"a397931b-faa9-4c2e-8290-491350546663\") " pod="e2e-models-e2e-worker-w1-88bbf7/distinct-2-w1-88bbf7-kserve-5457cdcdbd-l6tfd" Nov 03 00:54:44.098244 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:54:44.095944 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a397931b-faa9-4c2e-8290-491350546663-home\") pod \"distinct-2-w1-88bbf7-kserve-5457cdcdbd-l6tfd\" (UID: \"a397931b-faa9-4c2e-8290-491350546663\") " pod="e2e-models-e2e-worker-w1-88bbf7/distinct-2-w1-88bbf7-kserve-5457cdcdbd-l6tfd" Nov 03 00:54:44.127639 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:54:44.127593 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 00:54:44.211857 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:54:44.211707 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vnpdj\" (UniqueName: \"kubernetes.io/projected/a397931b-faa9-4c2e-8290-491350546663-kube-api-access-vnpdj\") pod \"distinct-2-w1-88bbf7-kserve-5457cdcdbd-l6tfd\" (UID: \"a397931b-faa9-4c2e-8290-491350546663\") " pod="e2e-models-e2e-worker-w1-88bbf7/distinct-2-w1-88bbf7-kserve-5457cdcdbd-l6tfd" Nov 03 00:54:44.212184 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:54:44.212158 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a397931b-faa9-4c2e-8290-491350546663-model-cache\") pod \"distinct-2-w1-88bbf7-kserve-5457cdcdbd-l6tfd\" (UID: \"a397931b-faa9-4c2e-8290-491350546663\") " pod="e2e-models-e2e-worker-w1-88bbf7/distinct-2-w1-88bbf7-kserve-5457cdcdbd-l6tfd" Nov 03 00:54:44.212488 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:54:44.212432 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a397931b-faa9-4c2e-8290-491350546663-home\") pod \"distinct-2-w1-88bbf7-kserve-5457cdcdbd-l6tfd\" (UID: \"a397931b-faa9-4c2e-8290-491350546663\") " pod="e2e-models-e2e-worker-w1-88bbf7/distinct-2-w1-88bbf7-kserve-5457cdcdbd-l6tfd" Nov 03 00:54:44.212774 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:54:44.212752 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a397931b-faa9-4c2e-8290-491350546663-dshm\") pod \"distinct-2-w1-88bbf7-kserve-5457cdcdbd-l6tfd\" (UID: \"a397931b-faa9-4c2e-8290-491350546663\") " pod="e2e-models-e2e-worker-w1-88bbf7/distinct-2-w1-88bbf7-kserve-5457cdcdbd-l6tfd" Nov 03 00:54:44.212950 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:54:44.212933 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a397931b-faa9-4c2e-8290-491350546663-tls-certs\") pod \"distinct-2-w1-88bbf7-kserve-5457cdcdbd-l6tfd\" (UID: \"a397931b-faa9-4c2e-8290-491350546663\") " pod="e2e-models-e2e-worker-w1-88bbf7/distinct-2-w1-88bbf7-kserve-5457cdcdbd-l6tfd" Nov 03 00:54:44.218928 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:54:44.213168 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a397931b-faa9-4c2e-8290-491350546663-tmp-dir\") pod \"distinct-2-w1-88bbf7-kserve-5457cdcdbd-l6tfd\" (UID: \"a397931b-faa9-4c2e-8290-491350546663\") " pod="e2e-models-e2e-worker-w1-88bbf7/distinct-2-w1-88bbf7-kserve-5457cdcdbd-l6tfd" Nov 03 00:54:44.228821 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:54:44.219482 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a397931b-faa9-4c2e-8290-491350546663-home\") pod \"distinct-2-w1-88bbf7-kserve-5457cdcdbd-l6tfd\" (UID: \"a397931b-faa9-4c2e-8290-491350546663\") " pod="e2e-models-e2e-worker-w1-88bbf7/distinct-2-w1-88bbf7-kserve-5457cdcdbd-l6tfd" Nov 03 00:54:44.243931 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:54:44.220105 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a397931b-faa9-4c2e-8290-491350546663-model-cache\") pod \"distinct-2-w1-88bbf7-kserve-5457cdcdbd-l6tfd\" (UID: \"a397931b-faa9-4c2e-8290-491350546663\") " pod="e2e-models-e2e-worker-w1-88bbf7/distinct-2-w1-88bbf7-kserve-5457cdcdbd-l6tfd" Nov 03 00:54:44.243931 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:54:44.229318 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a397931b-faa9-4c2e-8290-491350546663-tmp-dir\") pod \"distinct-2-w1-88bbf7-kserve-5457cdcdbd-l6tfd\" (UID: \"a397931b-faa9-4c2e-8290-491350546663\") " pod="e2e-models-e2e-worker-w1-88bbf7/distinct-2-w1-88bbf7-kserve-5457cdcdbd-l6tfd" Nov 03 00:54:44.254461 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:54:44.254429 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a397931b-faa9-4c2e-8290-491350546663-dshm\") pod \"distinct-2-w1-88bbf7-kserve-5457cdcdbd-l6tfd\" (UID: \"a397931b-faa9-4c2e-8290-491350546663\") " pod="e2e-models-e2e-worker-w1-88bbf7/distinct-2-w1-88bbf7-kserve-5457cdcdbd-l6tfd" Nov 03 00:54:44.255722 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:54:44.255695 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vnpdj\" (UniqueName: \"kubernetes.io/projected/a397931b-faa9-4c2e-8290-491350546663-kube-api-access-vnpdj\") pod \"distinct-2-w1-88bbf7-kserve-5457cdcdbd-l6tfd\" (UID: \"a397931b-faa9-4c2e-8290-491350546663\") " pod="e2e-models-e2e-worker-w1-88bbf7/distinct-2-w1-88bbf7-kserve-5457cdcdbd-l6tfd" Nov 03 00:54:44.266625 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:54:44.266589 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a397931b-faa9-4c2e-8290-491350546663-tls-certs\") pod \"distinct-2-w1-88bbf7-kserve-5457cdcdbd-l6tfd\" (UID: \"a397931b-faa9-4c2e-8290-491350546663\") " pod="e2e-models-e2e-worker-w1-88bbf7/distinct-2-w1-88bbf7-kserve-5457cdcdbd-l6tfd" Nov 03 00:54:44.398628 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:54:44.398594 4939 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-88bbf7/distinct-2-w1-88bbf7-kserve-5457cdcdbd-l6tfd" Nov 03 00:54:44.679473 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:54:44.679442 4939 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w1-88bbf7/distinct-2-w1-88bbf7-kserve-5457cdcdbd-l6tfd"] Nov 03 00:54:44.679946 ip-10-0-140-107 kubenswrapper[4939]: W1103 00:54:44.679893 4939 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-poda397931b_faa9_4c2e_8290_491350546663.slice/crio-04c5c3048d85531cb0f5ce68c456ca67a406d641e5cd4d43f6f3dad0173ed240 WatchSource:0}: Error finding container 04c5c3048d85531cb0f5ce68c456ca67a406d641e5cd4d43f6f3dad0173ed240: Status 404 returned error can't find the container with id 04c5c3048d85531cb0f5ce68c456ca67a406d641e5cd4d43f6f3dad0173ed240 Nov 03 00:54:44.781971 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:54:44.781936 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 00:54:45.489576 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:54:45.489448 4939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-88bbf7/distinct-2-w1-88bbf7-kserve-5457cdcdbd-l6tfd" event={"ID":"a397931b-faa9-4c2e-8290-491350546663","Type":"ContainerStarted","Data":"4d89519b7718515ab659a365ce964abdde58e902233a6a73f99bb5a317f2eb19"} Nov 03 00:54:45.489576 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:54:45.489493 4939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-88bbf7/distinct-2-w1-88bbf7-kserve-5457cdcdbd-l6tfd" event={"ID":"a397931b-faa9-4c2e-8290-491350546663","Type":"ContainerStarted","Data":"04c5c3048d85531cb0f5ce68c456ca67a406d641e5cd4d43f6f3dad0173ed240"} Nov 03 00:54:45.512178 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:54:45.512123 4939 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w1-88bbf7/distinct-2-w1-88bbf7-kserve-5457cdcdbd-l6tfd" podStartSLOduration=1.512108172 podStartE2EDuration="1.512108172s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:54:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:54:45.510276407 +0000 UTC m=+833.774611353" watchObservedRunningTime="2026-11-03 00:54:45.512108172 +0000 UTC m=+833.776443175" Nov 03 00:54:45.615372 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:54:45.615328 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 00:54:46.407932 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:54:46.401003 4939 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-88bbf7/distinct-2-w1-88bbf7-kserve-5457cdcdbd-l6tfd" Nov 03 00:54:47.238262 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:54:47.233013 4939 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-f7680d/distinct-2-w2-f7680d-kserve-7d7c4564bf-cbv2c" podUID="2de18086-761b-4773-b9fc-15ddc4a220fc" containerName="main" probeResult="failure" output="Get \"https://10.132.0.75:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 00:54:48.049935 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:54:48.022033 4939 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-f7680d/distinct-w2-f7680d-kserve-756f874b98-qnpv8" podUID="836753fc-f9ef-4f61-b3fe-e5b92b7ee27b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.70:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 00:54:48.049935 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:54:48.003697 4939 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-2f9a3a/simulator-w0-2f9a3a-kserve-9596bbb8f-2trgz" podUID="207e2207-fb6c-4076-95df-72e95671c261" containerName="main" probeResult="failure" output="Get \"https://10.132.0.61:8000/ready\": context deadline exceeded" Nov 03 00:54:48.049935 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:54:48.047988 4939 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-83755a/distinct-2-w3-83755a-kserve-6cf66b995d-2l47b" podUID="c3b05e88-5d5b-4d03-b28d-832f22e3c868" containerName="main" probeResult="failure" output="Get \"https://10.132.0.72:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 00:54:48.222935 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:54:48.218362 4939 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-88bbf7/simulator-w1-88bbf7-kserve-5fbb669f8f-42hgd" podUID="83b38883-2e7a-444d-aa68-52be4a25ae66" containerName="main" probeResult="failure" output="Get \"https://10.132.0.62:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 00:54:48.263045 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:54:48.250305 4939 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-f7680d/distinct-2-w2-f7680d-kserve-7d7c4564bf-cbv2c" podUID="2de18086-761b-4773-b9fc-15ddc4a220fc" containerName="main" probeResult="failure" output="Get \"https://10.132.0.75:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 00:54:48.263045 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:54:48.252933 4939 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-f7680d/distinct-2-w2-f7680d-kserve-7d7c4564bf-cbv2c" Nov 03 00:54:48.414954 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:54:48.389994 4939 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-88bbf7/simulator-w1-88bbf7-kserve-5fbb669f8f-42hgd" podUID="83b38883-2e7a-444d-aa68-52be4a25ae66" containerName="main" probeResult="failure" output="Get \"https://10.132.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 00:54:48.459843 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:54:48.437524 4939 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-f7680d/distinct-w2-f7680d-kserve-756f874b98-qnpv8" podUID="836753fc-f9ef-4f61-b3fe-e5b92b7ee27b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.70:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 00:54:48.794990 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:54:48.784009 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 00:54:48.933945 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:54:48.928487 4939 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-f7680d/premium-w2-f7680d-kserve-76565b8cf9-hqbbt" podUID="4712e708-18a2-4a13-a12c-a5669397b2c3" containerName="main" probeResult="failure" output="Get \"https://10.132.0.63:8000/ready\": context deadline exceeded" Nov 03 00:54:49.041547 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:54:49.033973 4939 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-2f9a3a/simulator-w0-2f9a3a-kserve-9596bbb8f-2trgz" podUID="207e2207-fb6c-4076-95df-72e95671c261" containerName="main" probeResult="failure" output="Get \"https://10.132.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 00:54:49.041547 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:54:49.034702 4939 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-2f9a3a/simulator-w0-2f9a3a-kserve-9596bbb8f-2trgz" Nov 03 00:54:49.041547 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:54:49.036429 4939 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-2f9a3a/distinct-w0-2f9a3a-kserve-555565cfcf-jtcr9" podUID="ac0ece06-8718-463f-915f-5315988d9711" containerName="main" probeResult="failure" output="Get \"https://10.132.0.76:8000/ready\": context deadline exceeded" Nov 03 00:54:49.060103 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:54:49.060001 4939 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-83755a/distinct-2-w3-83755a-kserve-6cf66b995d-2l47b" podUID="c3b05e88-5d5b-4d03-b28d-832f22e3c868" containerName="main" probeResult="failure" output="Get \"https://10.132.0.72:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 00:54:49.060676 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:54:49.060648 4939 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-83755a/distinct-2-w3-83755a-kserve-6cf66b995d-2l47b" Nov 03 00:54:49.091351 ip-10-0-140-107 kubenswrapper[4939]: E1103 00:54:49.091312 4939 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-sthjh" containerName="extractor" Nov 03 00:54:49.293948 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:54:49.272749 4939 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-f7680d/distinct-2-w2-f7680d-kserve-7d7c4564bf-cbv2c" podUID="2de18086-761b-4773-b9fc-15ddc4a220fc" containerName="main" probeResult="failure" output="Get \"https://10.132.0.75:8000/ready\": context deadline exceeded" Nov 03 00:54:49.421369 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:54:49.421317 4939 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-88bbf7/simulator-w1-88bbf7-kserve-5fbb669f8f-42hgd" podUID="83b38883-2e7a-444d-aa68-52be4a25ae66" containerName="main" probeResult="failure" output="Get \"https://10.132.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 00:54:49.432930 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:54:49.422355 4939 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-88bbf7/simulator-w1-88bbf7-kserve-5fbb669f8f-42hgd" Nov 03 00:54:49.453130 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:54:49.437992 4939 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-f7680d/distinct-w2-f7680d-kserve-756f874b98-qnpv8" podUID="836753fc-f9ef-4f61-b3fe-e5b92b7ee27b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.70:8000/ready\": context deadline exceeded" Nov 03 00:54:49.503567 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:54:49.503536 4939 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-f7680d/distinct-w2-f7680d-kserve-756f874b98-qnpv8" Nov 03 00:54:49.980934 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:54:49.939846 4939 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-f7680d/premium-w2-f7680d-kserve-76565b8cf9-hqbbt" podUID="4712e708-18a2-4a13-a12c-a5669397b2c3" containerName="main" probeResult="failure" output="Get \"https://10.132.0.63:8000/ready\": context deadline exceeded" Nov 03 00:54:50.009241 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:54:49.992076 4939 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-88bbf7/simulator-w1-88bbf7-kserve-5fbb669f8f-42hgd" Nov 03 00:54:50.009241 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:54:49.992143 4939 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-f7680d/premium-w2-f7680d-kserve-76565b8cf9-hqbbt" Nov 03 00:54:50.009241 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:54:49.992177 4939 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-2f9a3a/simulator-w0-2f9a3a-kserve-9596bbb8f-2trgz" Nov 03 00:54:50.036595 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:54:50.036541 4939 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-2f9a3a/distinct-w0-2f9a3a-kserve-555565cfcf-jtcr9" podUID="ac0ece06-8718-463f-915f-5315988d9711" containerName="main" probeResult="failure" output="Get \"https://10.132.0.76:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 00:54:50.037500 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:54:50.037475 4939 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-2f9a3a/distinct-w0-2f9a3a-kserve-555565cfcf-jtcr9" Nov 03 00:54:50.071234 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:54:50.071195 4939 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-83755a/distinct-2-w3-83755a-kserve-6cf66b995d-2l47b" podUID="c3b05e88-5d5b-4d03-b28d-832f22e3c868" containerName="main" probeResult="failure" output="Get \"https://10.132.0.72:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 00:54:50.279718 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:54:50.276156 4939 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-f7680d/distinct-2-w2-f7680d-kserve-7d7c4564bf-cbv2c" podUID="2de18086-761b-4773-b9fc-15ddc4a220fc" containerName="main" probeResult="failure" output="Get \"https://10.132.0.75:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 00:54:50.309342 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:54:50.309310 4939 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-f7680d/premium-w2-f7680d-kserve-76565b8cf9-hqbbt" Nov 03 00:54:50.571232 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:54:50.570947 4939 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-f7680d/distinct-w2-f7680d-kserve-756f874b98-qnpv8" podUID="836753fc-f9ef-4f61-b3fe-e5b92b7ee27b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.70:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 00:54:50.628981 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:54:50.611035 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 00:54:50.647151 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:54:50.633933 4939 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-f7680d/distinct-w2-f7680d-kserve-756f874b98-qnpv8" Nov 03 00:54:50.647426 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:54:50.647406 4939 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-2f9a3a/distinct-w0-2f9a3a-kserve-555565cfcf-jtcr9" Nov 03 00:54:50.725761 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:54:50.725717 4939 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-f7680d/distinct-2-w2-f7680d-kserve-7d7c4564bf-cbv2c" Nov 03 00:54:50.734877 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:54:50.734848 4939 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-83755a/distinct-2-w3-83755a-kserve-6cf66b995d-2l47b" Nov 03 00:54:51.294047 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:54:51.294016 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 00:54:52.216327 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:54:52.216291 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 00:54:52.997754 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:54:52.997719 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 00:54:53.596550 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:54:53.596520 4939 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-88bbf7-openshift-default-574f47fff7-k87gb"] Nov 03 00:54:53.601522 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:54:53.601498 4939 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-88bbf7-openshift-default-574f47fff7-k87gb" Nov 03 00:54:53.613952 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:54:53.613926 4939 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-88bbf7-openshift-default-574f47fff7-k87gb"] Nov 03 00:54:53.727517 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:54:53.727470 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 00:54:53.770938 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:54:53.768333 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c7907eaa-df92-4d28-aa19-ba6f03cb54b1-istio-envoy\") pod \"e2e-worker-w1-88bbf7-openshift-default-574f47fff7-k87gb\" (UID: \"c7907eaa-df92-4d28-aa19-ba6f03cb54b1\") " pod="openshift-ingress/e2e-worker-w1-88bbf7-openshift-default-574f47fff7-k87gb" Nov 03 00:54:53.770938 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:54:53.768404 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c7907eaa-df92-4d28-aa19-ba6f03cb54b1-istio-data\") pod \"e2e-worker-w1-88bbf7-openshift-default-574f47fff7-k87gb\" (UID: \"c7907eaa-df92-4d28-aa19-ba6f03cb54b1\") " pod="openshift-ingress/e2e-worker-w1-88bbf7-openshift-default-574f47fff7-k87gb" Nov 03 00:54:53.770938 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:54:53.768441 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c7907eaa-df92-4d28-aa19-ba6f03cb54b1-workload-socket\") pod \"e2e-worker-w1-88bbf7-openshift-default-574f47fff7-k87gb\" (UID: \"c7907eaa-df92-4d28-aa19-ba6f03cb54b1\") " pod="openshift-ingress/e2e-worker-w1-88bbf7-openshift-default-574f47fff7-k87gb" Nov 03 00:54:53.770938 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:54:53.768472 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r84h9\" (UniqueName: \"kubernetes.io/projected/c7907eaa-df92-4d28-aa19-ba6f03cb54b1-kube-api-access-r84h9\") pod \"e2e-worker-w1-88bbf7-openshift-default-574f47fff7-k87gb\" (UID: \"c7907eaa-df92-4d28-aa19-ba6f03cb54b1\") " pod="openshift-ingress/e2e-worker-w1-88bbf7-openshift-default-574f47fff7-k87gb" Nov 03 00:54:53.770938 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:54:53.768576 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c7907eaa-df92-4d28-aa19-ba6f03cb54b1-istio-podinfo\") pod \"e2e-worker-w1-88bbf7-openshift-default-574f47fff7-k87gb\" (UID: \"c7907eaa-df92-4d28-aa19-ba6f03cb54b1\") " pod="openshift-ingress/e2e-worker-w1-88bbf7-openshift-default-574f47fff7-k87gb" Nov 03 00:54:53.770938 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:54:53.768726 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c7907eaa-df92-4d28-aa19-ba6f03cb54b1-istiod-ca-cert\") pod \"e2e-worker-w1-88bbf7-openshift-default-574f47fff7-k87gb\" (UID: \"c7907eaa-df92-4d28-aa19-ba6f03cb54b1\") " pod="openshift-ingress/e2e-worker-w1-88bbf7-openshift-default-574f47fff7-k87gb" Nov 03 00:54:53.770938 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:54:53.768761 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c7907eaa-df92-4d28-aa19-ba6f03cb54b1-istio-token\") pod \"e2e-worker-w1-88bbf7-openshift-default-574f47fff7-k87gb\" (UID: \"c7907eaa-df92-4d28-aa19-ba6f03cb54b1\") " pod="openshift-ingress/e2e-worker-w1-88bbf7-openshift-default-574f47fff7-k87gb" Nov 03 00:54:53.770938 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:54:53.768804 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c7907eaa-df92-4d28-aa19-ba6f03cb54b1-workload-certs\") pod \"e2e-worker-w1-88bbf7-openshift-default-574f47fff7-k87gb\" (UID: \"c7907eaa-df92-4d28-aa19-ba6f03cb54b1\") " pod="openshift-ingress/e2e-worker-w1-88bbf7-openshift-default-574f47fff7-k87gb" Nov 03 00:54:53.770938 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:54:53.768847 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c7907eaa-df92-4d28-aa19-ba6f03cb54b1-credential-socket\") pod \"e2e-worker-w1-88bbf7-openshift-default-574f47fff7-k87gb\" (UID: \"c7907eaa-df92-4d28-aa19-ba6f03cb54b1\") " pod="openshift-ingress/e2e-worker-w1-88bbf7-openshift-default-574f47fff7-k87gb" Nov 03 00:54:53.869790 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:54:53.869707 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c7907eaa-df92-4d28-aa19-ba6f03cb54b1-istio-envoy\") pod \"e2e-worker-w1-88bbf7-openshift-default-574f47fff7-k87gb\" (UID: \"c7907eaa-df92-4d28-aa19-ba6f03cb54b1\") " pod="openshift-ingress/e2e-worker-w1-88bbf7-openshift-default-574f47fff7-k87gb" Nov 03 00:54:53.869790 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:54:53.869746 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c7907eaa-df92-4d28-aa19-ba6f03cb54b1-istio-data\") pod \"e2e-worker-w1-88bbf7-openshift-default-574f47fff7-k87gb\" (UID: \"c7907eaa-df92-4d28-aa19-ba6f03cb54b1\") " pod="openshift-ingress/e2e-worker-w1-88bbf7-openshift-default-574f47fff7-k87gb" Nov 03 00:54:53.869790 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:54:53.869767 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c7907eaa-df92-4d28-aa19-ba6f03cb54b1-workload-socket\") pod \"e2e-worker-w1-88bbf7-openshift-default-574f47fff7-k87gb\" (UID: \"c7907eaa-df92-4d28-aa19-ba6f03cb54b1\") " pod="openshift-ingress/e2e-worker-w1-88bbf7-openshift-default-574f47fff7-k87gb" Nov 03 00:54:53.869790 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:54:53.869785 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r84h9\" (UniqueName: \"kubernetes.io/projected/c7907eaa-df92-4d28-aa19-ba6f03cb54b1-kube-api-access-r84h9\") pod \"e2e-worker-w1-88bbf7-openshift-default-574f47fff7-k87gb\" (UID: \"c7907eaa-df92-4d28-aa19-ba6f03cb54b1\") " pod="openshift-ingress/e2e-worker-w1-88bbf7-openshift-default-574f47fff7-k87gb" Nov 03 00:54:53.870197 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:54:53.869845 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c7907eaa-df92-4d28-aa19-ba6f03cb54b1-istio-podinfo\") pod \"e2e-worker-w1-88bbf7-openshift-default-574f47fff7-k87gb\" (UID: \"c7907eaa-df92-4d28-aa19-ba6f03cb54b1\") " pod="openshift-ingress/e2e-worker-w1-88bbf7-openshift-default-574f47fff7-k87gb" Nov 03 00:54:53.870197 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:54:53.869979 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c7907eaa-df92-4d28-aa19-ba6f03cb54b1-istiod-ca-cert\") pod \"e2e-worker-w1-88bbf7-openshift-default-574f47fff7-k87gb\" (UID: \"c7907eaa-df92-4d28-aa19-ba6f03cb54b1\") " pod="openshift-ingress/e2e-worker-w1-88bbf7-openshift-default-574f47fff7-k87gb" Nov 03 00:54:53.870197 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:54:53.870015 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c7907eaa-df92-4d28-aa19-ba6f03cb54b1-istio-token\") pod \"e2e-worker-w1-88bbf7-openshift-default-574f47fff7-k87gb\" (UID: \"c7907eaa-df92-4d28-aa19-ba6f03cb54b1\") " pod="openshift-ingress/e2e-worker-w1-88bbf7-openshift-default-574f47fff7-k87gb" Nov 03 00:54:53.870197 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:54:53.870056 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c7907eaa-df92-4d28-aa19-ba6f03cb54b1-workload-certs\") pod \"e2e-worker-w1-88bbf7-openshift-default-574f47fff7-k87gb\" (UID: \"c7907eaa-df92-4d28-aa19-ba6f03cb54b1\") " pod="openshift-ingress/e2e-worker-w1-88bbf7-openshift-default-574f47fff7-k87gb" Nov 03 00:54:53.870197 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:54:53.870099 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c7907eaa-df92-4d28-aa19-ba6f03cb54b1-credential-socket\") pod \"e2e-worker-w1-88bbf7-openshift-default-574f47fff7-k87gb\" (UID: \"c7907eaa-df92-4d28-aa19-ba6f03cb54b1\") " pod="openshift-ingress/e2e-worker-w1-88bbf7-openshift-default-574f47fff7-k87gb" Nov 03 00:54:53.870197 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:54:53.870148 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c7907eaa-df92-4d28-aa19-ba6f03cb54b1-istio-data\") pod \"e2e-worker-w1-88bbf7-openshift-default-574f47fff7-k87gb\" (UID: \"c7907eaa-df92-4d28-aa19-ba6f03cb54b1\") " pod="openshift-ingress/e2e-worker-w1-88bbf7-openshift-default-574f47fff7-k87gb" Nov 03 00:54:53.870496 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:54:53.870204 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c7907eaa-df92-4d28-aa19-ba6f03cb54b1-workload-socket\") pod \"e2e-worker-w1-88bbf7-openshift-default-574f47fff7-k87gb\" (UID: \"c7907eaa-df92-4d28-aa19-ba6f03cb54b1\") " pod="openshift-ingress/e2e-worker-w1-88bbf7-openshift-default-574f47fff7-k87gb" Nov 03 00:54:53.870496 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:54:53.870447 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c7907eaa-df92-4d28-aa19-ba6f03cb54b1-workload-certs\") pod \"e2e-worker-w1-88bbf7-openshift-default-574f47fff7-k87gb\" (UID: \"c7907eaa-df92-4d28-aa19-ba6f03cb54b1\") " pod="openshift-ingress/e2e-worker-w1-88bbf7-openshift-default-574f47fff7-k87gb" Nov 03 00:54:53.870609 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:54:53.870535 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c7907eaa-df92-4d28-aa19-ba6f03cb54b1-credential-socket\") pod \"e2e-worker-w1-88bbf7-openshift-default-574f47fff7-k87gb\" (UID: \"c7907eaa-df92-4d28-aa19-ba6f03cb54b1\") " pod="openshift-ingress/e2e-worker-w1-88bbf7-openshift-default-574f47fff7-k87gb" Nov 03 00:54:53.870844 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:54:53.870823 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c7907eaa-df92-4d28-aa19-ba6f03cb54b1-istiod-ca-cert\") pod \"e2e-worker-w1-88bbf7-openshift-default-574f47fff7-k87gb\" (UID: \"c7907eaa-df92-4d28-aa19-ba6f03cb54b1\") " pod="openshift-ingress/e2e-worker-w1-88bbf7-openshift-default-574f47fff7-k87gb" Nov 03 00:54:53.872102 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:54:53.872079 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c7907eaa-df92-4d28-aa19-ba6f03cb54b1-istio-envoy\") pod \"e2e-worker-w1-88bbf7-openshift-default-574f47fff7-k87gb\" (UID: \"c7907eaa-df92-4d28-aa19-ba6f03cb54b1\") " pod="openshift-ingress/e2e-worker-w1-88bbf7-openshift-default-574f47fff7-k87gb" Nov 03 00:54:53.872405 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:54:53.872375 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c7907eaa-df92-4d28-aa19-ba6f03cb54b1-istio-podinfo\") pod \"e2e-worker-w1-88bbf7-openshift-default-574f47fff7-k87gb\" (UID: \"c7907eaa-df92-4d28-aa19-ba6f03cb54b1\") " pod="openshift-ingress/e2e-worker-w1-88bbf7-openshift-default-574f47fff7-k87gb" Nov 03 00:54:53.877617 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:54:53.877595 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c7907eaa-df92-4d28-aa19-ba6f03cb54b1-istio-token\") pod \"e2e-worker-w1-88bbf7-openshift-default-574f47fff7-k87gb\" (UID: \"c7907eaa-df92-4d28-aa19-ba6f03cb54b1\") " pod="openshift-ingress/e2e-worker-w1-88bbf7-openshift-default-574f47fff7-k87gb" Nov 03 00:54:53.878063 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:54:53.878042 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-r84h9\" (UniqueName: \"kubernetes.io/projected/c7907eaa-df92-4d28-aa19-ba6f03cb54b1-kube-api-access-r84h9\") pod \"e2e-worker-w1-88bbf7-openshift-default-574f47fff7-k87gb\" (UID: \"c7907eaa-df92-4d28-aa19-ba6f03cb54b1\") " pod="openshift-ingress/e2e-worker-w1-88bbf7-openshift-default-574f47fff7-k87gb" Nov 03 00:54:53.923800 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:54:53.923766 4939 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-88bbf7-openshift-default-574f47fff7-k87gb" Nov 03 00:54:54.203794 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:54:54.203762 4939 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-88bbf7-openshift-default-574f47fff7-k87gb"] Nov 03 00:54:54.204951 ip-10-0-140-107 kubenswrapper[4939]: W1103 00:54:54.204920 4939 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc7907eaa_df92_4d28_aa19_ba6f03cb54b1.slice/crio-4f16624bb13bdc4cb203774ad42595d9e1a471f145e687306027ba3a15cb2a80 WatchSource:0}: Error finding container 4f16624bb13bdc4cb203774ad42595d9e1a471f145e687306027ba3a15cb2a80: Status 404 returned error can't find the container with id 4f16624bb13bdc4cb203774ad42595d9e1a471f145e687306027ba3a15cb2a80 Nov 03 00:54:54.207456 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:54:54.207419 4939 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 00:54:54.207543 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:54:54.207487 4939 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 00:54:54.207543 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:54:54.207519 4939 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 00:54:54.399177 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:54:54.399143 4939 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w1-88bbf7/distinct-2-w1-88bbf7-kserve-5457cdcdbd-l6tfd" Nov 03 00:54:54.412439 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:54:54.412413 4939 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w1-88bbf7/distinct-2-w1-88bbf7-kserve-5457cdcdbd-l6tfd" Nov 03 00:54:54.587980 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:54:54.587950 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 00:54:54.933083 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:54:54.932975 4939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-88bbf7-openshift-default-574f47fff7-k87gb" event={"ID":"c7907eaa-df92-4d28-aa19-ba6f03cb54b1","Type":"ContainerStarted","Data":"dddb3be79f5aa8b4fa1455c37072e16e4a1b3338a22817f7840fb711772b6885"} Nov 03 00:54:54.933083 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:54:54.933059 4939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-88bbf7-openshift-default-574f47fff7-k87gb" event={"ID":"c7907eaa-df92-4d28-aa19-ba6f03cb54b1","Type":"ContainerStarted","Data":"4f16624bb13bdc4cb203774ad42595d9e1a471f145e687306027ba3a15cb2a80"} Nov 03 00:54:54.944725 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:54:54.944698 4939 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-88bbf7/distinct-2-w1-88bbf7-kserve-5457cdcdbd-l6tfd" Nov 03 00:54:54.960950 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:54:54.960888 4939 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-88bbf7-openshift-default-574f47fff7-k87gb" podStartSLOduration=1.960874523 podStartE2EDuration="1.960874523s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:54:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:54:54.957789386 +0000 UTC m=+843.222124324" watchObservedRunningTime="2026-11-03 00:54:54.960874523 +0000 UTC m=+843.225209464" Nov 03 00:54:55.409368 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:54:55.409332 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 00:54:55.924481 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:54:55.924445 4939 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-88bbf7-openshift-default-574f47fff7-k87gb" Nov 03 00:54:55.926070 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:54:55.926042 4939 patch_prober.go:29] interesting pod/e2e-worker-w1-88bbf7-openshift-default-574f47fff7-k87gb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.86:15021/healthz/ready\": dial tcp 10.132.0.86:15021: connect: connection refused" start-of-body= Nov 03 00:54:55.926218 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:54:55.926099 4939 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-88bbf7-openshift-default-574f47fff7-k87gb" podUID="c7907eaa-df92-4d28-aa19-ba6f03cb54b1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.86:15021/healthz/ready\": dial tcp 10.132.0.86:15021: connect: connection refused" Nov 03 00:54:56.220287 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:54:56.220204 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 00:54:56.924818 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:54:56.924780 4939 patch_prober.go:29] interesting pod/e2e-worker-w1-88bbf7-openshift-default-574f47fff7-k87gb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.86:15021/healthz/ready\": dial tcp 10.132.0.86:15021: connect: connection refused" start-of-body= Nov 03 00:54:56.925129 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:54:56.925096 4939 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-88bbf7-openshift-default-574f47fff7-k87gb" podUID="c7907eaa-df92-4d28-aa19-ba6f03cb54b1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.86:15021/healthz/ready\": dial tcp 10.132.0.86:15021: connect: connection refused" Nov 03 00:54:56.927224 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:54:56.927196 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 00:54:57.882521 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:54:57.882488 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 00:54:57.977520 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:54:57.966464 4939 patch_prober.go:29] interesting pod/e2e-worker-w1-88bbf7-openshift-default-574f47fff7-k87gb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.86:15021/healthz/ready\": dial tcp 10.132.0.86:15021: connect: connection refused" start-of-body= Nov 03 00:54:57.977520 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:54:57.966527 4939 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-88bbf7-openshift-default-574f47fff7-k87gb" podUID="c7907eaa-df92-4d28-aa19-ba6f03cb54b1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.86:15021/healthz/ready\": dial tcp 10.132.0.86:15021: connect: connection refused" Nov 03 00:54:58.698988 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:54:58.698956 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 00:54:58.924166 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:54:58.924125 4939 patch_prober.go:29] interesting pod/e2e-worker-w1-88bbf7-openshift-default-574f47fff7-k87gb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.86:15021/healthz/ready\": dial tcp 10.132.0.86:15021: connect: connection refused" start-of-body= Nov 03 00:54:58.924677 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:54:58.924187 4939 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-88bbf7-openshift-default-574f47fff7-k87gb" podUID="c7907eaa-df92-4d28-aa19-ba6f03cb54b1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.86:15021/healthz/ready\": dial tcp 10.132.0.86:15021: connect: connection refused" Nov 03 00:54:59.594428 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:54:59.594388 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 00:54:59.770260 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:54:59.770222 4939 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-4pw7v"] Nov 03 00:54:59.924742 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:54:59.924703 4939 patch_prober.go:29] interesting pod/e2e-worker-w1-88bbf7-openshift-default-574f47fff7-k87gb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.86:15021/healthz/ready\": dial tcp 10.132.0.86:15021: connect: connection refused" start-of-body= Nov 03 00:54:59.925148 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:54:59.924784 4939 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-88bbf7-openshift-default-574f47fff7-k87gb" podUID="c7907eaa-df92-4d28-aa19-ba6f03cb54b1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.86:15021/healthz/ready\": dial tcp 10.132.0.86:15021: connect: connection refused" Nov 03 00:55:00.425866 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:55:00.425816 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 00:55:00.927462 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:55:00.927421 4939 patch_prober.go:29] interesting pod/e2e-worker-w1-88bbf7-openshift-default-574f47fff7-k87gb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.86:15021/healthz/ready\": dial tcp 10.132.0.86:15021: connect: connection refused" start-of-body= Nov 03 00:55:00.928080 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:55:00.928043 4939 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-88bbf7-openshift-default-574f47fff7-k87gb" podUID="c7907eaa-df92-4d28-aa19-ba6f03cb54b1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.86:15021/healthz/ready\": dial tcp 10.132.0.86:15021: connect: connection refused" Nov 03 00:55:01.660123 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:55:01.660061 4939 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-f7680d/distinct-w2-f7680d-kserve-756f874b98-qnpv8" podUID="836753fc-f9ef-4f61-b3fe-e5b92b7ee27b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.70:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 00:55:01.902935 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:55:01.900409 4939 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-88bbf7/simulator-w1-88bbf7-kserve-5fbb669f8f-42hgd" podUID="83b38883-2e7a-444d-aa68-52be4a25ae66" containerName="main" probeResult="failure" output="Get \"https://10.132.0.62:8000/ready\": context deadline exceeded" Nov 03 00:55:01.955072 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:55:01.954979 4939 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-88bbf7/distinct-2-w1-88bbf7-kserve-5457cdcdbd-l6tfd" podUID="a397931b-faa9-4c2e-8290-491350546663" containerName="main" probeResult="failure" output="Get \"https://10.132.0.85:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 00:55:01.967942 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:55:01.951893 4939 patch_prober.go:29] interesting pod/e2e-worker-w1-88bbf7-openshift-default-574f47fff7-k87gb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.86:15021/healthz/ready\": dial tcp 10.132.0.86:15021: connect: connection refused" start-of-body= Nov 03 00:55:01.967942 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:55:01.964000 4939 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-88bbf7-openshift-default-574f47fff7-k87gb" podUID="c7907eaa-df92-4d28-aa19-ba6f03cb54b1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.86:15021/healthz/ready\": dial tcp 10.132.0.86:15021: connect: connection refused" Nov 03 00:55:01.967942 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:55:01.966459 4939 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-f7680d/distinct-2-w2-f7680d-kserve-7d7c4564bf-cbv2c" podUID="2de18086-761b-4773-b9fc-15ddc4a220fc" containerName="main" probeResult="failure" output="Get \"https://10.132.0.75:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 00:55:01.967942 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:55:01.966529 4939 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-f7680d/premium-w2-f7680d-kserve-76565b8cf9-hqbbt" podUID="4712e708-18a2-4a13-a12c-a5669397b2c3" containerName="main" probeResult="failure" output="Get \"https://10.132.0.63:8000/health\": context deadline exceeded" Nov 03 00:55:01.985572 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:55:01.985524 4939 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-2f9a3a/simulator-w0-2f9a3a-kserve-9596bbb8f-2trgz" podUID="207e2207-fb6c-4076-95df-72e95671c261" containerName="main" probeResult="failure" output="Get \"https://10.132.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 00:55:02.084096 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:55:02.084037 4939 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-83755a/distinct-2-w3-83755a-kserve-6cf66b995d-2l47b" podUID="c3b05e88-5d5b-4d03-b28d-832f22e3c868" containerName="main" probeResult="failure" output="Get \"https://10.132.0.72:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 00:55:02.283170 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:55:02.283053 4939 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-f7680d/distinct-2-w2-f7680d-kserve-7d7c4564bf-cbv2c" podUID="2de18086-761b-4773-b9fc-15ddc4a220fc" containerName="main" probeResult="failure" output="Get \"https://10.132.0.75:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 00:55:02.326523 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:55:02.326472 4939 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-f7680d/premium-w2-f7680d-kserve-76565b8cf9-hqbbt" podUID="4712e708-18a2-4a13-a12c-a5669397b2c3" containerName="main" probeResult="failure" output="Get \"https://10.132.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 00:55:02.648799 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:55:02.648744 4939 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-2f9a3a/distinct-w0-2f9a3a-kserve-555565cfcf-jtcr9" podUID="ac0ece06-8718-463f-915f-5315988d9711" containerName="main" probeResult="failure" output="Get \"https://10.132.0.76:8000/ready\": context deadline exceeded" Nov 03 00:55:02.674934 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:55:02.666887 4939 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-f7680d/distinct-w2-f7680d-kserve-756f874b98-qnpv8" podUID="836753fc-f9ef-4f61-b3fe-e5b92b7ee27b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.70:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 00:55:02.674934 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:55:02.667513 4939 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-f7680d/distinct-w2-f7680d-kserve-756f874b98-qnpv8" Nov 03 00:55:03.035978 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:55:03.006442 4939 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-88bbf7/distinct-2-w1-88bbf7-kserve-5457cdcdbd-l6tfd" podUID="a397931b-faa9-4c2e-8290-491350546663" containerName="main" probeResult="failure" output="Get \"https://10.132.0.85:8000/ready\": context deadline exceeded" Nov 03 00:55:03.093132 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:55:03.067076 4939 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-88bbf7/distinct-2-w1-88bbf7-kserve-5457cdcdbd-l6tfd" Nov 03 00:55:03.093132 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:55:03.046289 4939 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-88bbf7/simulator-w1-88bbf7-kserve-5fbb669f8f-42hgd" podUID="83b38883-2e7a-444d-aa68-52be4a25ae66" containerName="main" probeResult="failure" output="Get \"https://10.132.0.62:8000/ready\": context deadline exceeded" Nov 03 00:55:03.093132 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:55:03.068373 4939 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-88bbf7/simulator-w1-88bbf7-kserve-5fbb669f8f-42hgd" Nov 03 00:55:03.093132 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:55:03.091935 4939 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-83755a/distinct-2-w3-83755a-kserve-6cf66b995d-2l47b" podUID="c3b05e88-5d5b-4d03-b28d-832f22e3c868" containerName="main" probeResult="failure" output="Get \"https://10.132.0.72:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 00:55:03.093132 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:55:03.092363 4939 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-83755a/distinct-2-w3-83755a-kserve-6cf66b995d-2l47b" Nov 03 00:55:03.094094 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:55:03.038562 4939 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-2f9a3a/simulator-w0-2f9a3a-kserve-9596bbb8f-2trgz" podUID="207e2207-fb6c-4076-95df-72e95671c261" containerName="main" probeResult="failure" output="Get \"https://10.132.0.61:8000/ready\": context deadline exceeded" Nov 03 00:55:03.094498 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:55:03.094464 4939 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-2f9a3a/simulator-w0-2f9a3a-kserve-9596bbb8f-2trgz" Nov 03 00:55:03.296983 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:55:03.288087 4939 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-f7680d/distinct-2-w2-f7680d-kserve-7d7c4564bf-cbv2c" podUID="2de18086-761b-4773-b9fc-15ddc4a220fc" containerName="main" probeResult="failure" output="Get \"https://10.132.0.75:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 00:55:03.297503 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:55:03.297474 4939 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-f7680d/distinct-2-w2-f7680d-kserve-7d7c4564bf-cbv2c" Nov 03 00:55:03.354712 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:55:03.354661 4939 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-f7680d/premium-w2-f7680d-kserve-76565b8cf9-hqbbt" podUID="4712e708-18a2-4a13-a12c-a5669397b2c3" containerName="main" probeResult="failure" output="Get \"https://10.132.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 00:55:03.388935 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:55:03.388595 4939 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-f7680d/premium-w2-f7680d-kserve-76565b8cf9-hqbbt" Nov 03 00:55:03.650294 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:55:03.649500 4939 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-2f9a3a/distinct-w0-2f9a3a-kserve-555565cfcf-jtcr9" podUID="ac0ece06-8718-463f-915f-5315988d9711" containerName="main" probeResult="failure" output="Get \"https://10.132.0.76:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 00:55:03.650294 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:55:03.649779 4939 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-2f9a3a/distinct-w0-2f9a3a-kserve-555565cfcf-jtcr9" Nov 03 00:55:03.702059 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:55:03.702008 4939 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-f7680d/distinct-w2-f7680d-kserve-756f874b98-qnpv8" podUID="836753fc-f9ef-4f61-b3fe-e5b92b7ee27b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.70:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 00:55:04.068991 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:55:04.067996 4939 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-88bbf7/distinct-2-w1-88bbf7-kserve-5457cdcdbd-l6tfd" podUID="a397931b-faa9-4c2e-8290-491350546663" containerName="main" probeResult="failure" output="Get \"https://10.132.0.85:8000/ready\": context deadline exceeded" Nov 03 00:55:04.097932 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:55:04.097258 4939 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-2f9a3a/simulator-w0-2f9a3a-kserve-9596bbb8f-2trgz" podUID="207e2207-fb6c-4076-95df-72e95671c261" containerName="main" probeResult="failure" output="Get \"https://10.132.0.61:8000/ready\": context deadline exceeded" Nov 03 00:55:04.105106 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:55:04.099750 4939 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-88bbf7/simulator-w1-88bbf7-kserve-5fbb669f8f-42hgd" podUID="83b38883-2e7a-444d-aa68-52be4a25ae66" containerName="main" probeResult="failure" output="Get \"https://10.132.0.62:8000/ready\": context deadline exceeded" Nov 03 00:55:04.122451 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:55:04.098233 4939 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-83755a/distinct-2-w3-83755a-kserve-6cf66b995d-2l47b" podUID="c3b05e88-5d5b-4d03-b28d-832f22e3c868" containerName="main" probeResult="failure" output="Get \"https://10.132.0.72:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 00:55:04.201955 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:55:04.193471 4939 patch_prober.go:29] interesting pod/e2e-worker-w1-88bbf7-openshift-default-574f47fff7-k87gb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.86:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 00:55:04.201955 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:55:04.193602 4939 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-88bbf7-openshift-default-574f47fff7-k87gb" podUID="c7907eaa-df92-4d28-aa19-ba6f03cb54b1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.86:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 00:55:04.242246 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:55:04.242199 4939 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-2f9a3a/distinct-w0-2f9a3a-kserve-555565cfcf-jtcr9" podUID="ac0ece06-8718-463f-915f-5315988d9711" containerName="main" probeResult="failure" output="Get \"https://10.132.0.76:8000/health\": context deadline exceeded" Nov 03 00:55:04.299225 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:55:04.299178 4939 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-f7680d/distinct-2-w2-f7680d-kserve-7d7c4564bf-cbv2c" podUID="2de18086-761b-4773-b9fc-15ddc4a220fc" containerName="main" probeResult="failure" output="Get \"https://10.132.0.75:8000/ready\": context deadline exceeded" Nov 03 00:55:04.368930 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:55:04.367135 4939 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-f7680d/premium-w2-f7680d-kserve-76565b8cf9-hqbbt" podUID="4712e708-18a2-4a13-a12c-a5669397b2c3" containerName="main" probeResult="failure" output="Get \"https://10.132.0.63:8000/ready\": context deadline exceeded" Nov 03 00:55:04.661737 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:55:04.658008 4939 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-2f9a3a/distinct-w0-2f9a3a-kserve-555565cfcf-jtcr9" podUID="ac0ece06-8718-463f-915f-5315988d9711" containerName="main" probeResult="failure" output="Get \"https://10.132.0.76:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 00:55:04.690475 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:55:04.689291 4939 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-83755a/distinct-2-w3-83755a-kserve-6cf66b995d-2l47b" Nov 03 00:55:04.699739 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:55:04.698464 4939 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-88bbf7/simulator-w1-88bbf7-kserve-5fbb669f8f-42hgd" Nov 03 00:55:04.729714 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:55:04.729666 4939 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-f7680d/distinct-w2-f7680d-kserve-756f874b98-qnpv8" podUID="836753fc-f9ef-4f61-b3fe-e5b92b7ee27b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.70:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 00:55:05.185966 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:55:05.163426 4939 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-88bbf7/distinct-2-w1-88bbf7-kserve-5457cdcdbd-l6tfd" podUID="a397931b-faa9-4c2e-8290-491350546663" containerName="main" probeResult="failure" output="Get \"https://10.132.0.85:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 00:55:05.257945 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:55:05.228316 4939 patch_prober.go:29] interesting pod/e2e-worker-w1-88bbf7-openshift-default-574f47fff7-k87gb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.86:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 00:55:05.257945 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:55:05.228385 4939 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-88bbf7-openshift-default-574f47fff7-k87gb" podUID="c7907eaa-df92-4d28-aa19-ba6f03cb54b1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.86:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 00:55:05.300359 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:55:05.300312 4939 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-2f9a3a/simulator-w0-2f9a3a-kserve-9596bbb8f-2trgz" podUID="207e2207-fb6c-4076-95df-72e95671c261" containerName="main" probeResult="failure" output="Get \"https://10.132.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 00:55:05.334931 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:55:05.302285 4939 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-f7680d/distinct-2-w2-f7680d-kserve-7d7c4564bf-cbv2c" podUID="2de18086-761b-4773-b9fc-15ddc4a220fc" containerName="main" probeResult="failure" output="Get \"https://10.132.0.75:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 00:55:05.387763 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:55:05.348523 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 00:55:05.387763 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:55:05.369980 4939 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-f7680d/premium-w2-f7680d-kserve-76565b8cf9-hqbbt" podUID="4712e708-18a2-4a13-a12c-a5669397b2c3" containerName="main" probeResult="failure" output="Get \"https://10.132.0.63:8000/ready\": context deadline exceeded" Nov 03 00:55:05.410045 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:55:05.409995 4939 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-88bbf7/distinct-2-w1-88bbf7-kserve-5457cdcdbd-l6tfd" podUID="a397931b-faa9-4c2e-8290-491350546663" containerName="main" probeResult="failure" output="Get \"https://10.132.0.85:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 00:55:05.471694 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:55:05.471583 4939 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-2f9a3a/simulator-w0-2f9a3a-kserve-9596bbb8f-2trgz" Nov 03 00:55:05.541965 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:55:05.529805 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 00:55:05.698851 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:55:05.698801 4939 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-88bbf7/simulator-w1-88bbf7-kserve-5fbb669f8f-42hgd" podUID="83b38883-2e7a-444d-aa68-52be4a25ae66" containerName="main" probeResult="failure" output="Get \"https://10.132.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 00:55:05.752973 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:55:05.752862 4939 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-f7680d/distinct-w2-f7680d-kserve-756f874b98-qnpv8" podUID="836753fc-f9ef-4f61-b3fe-e5b92b7ee27b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.70:8000/ready\": context deadline exceeded" Nov 03 00:55:05.753347 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:55:05.753323 4939 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-f7680d/distinct-2-w2-f7680d-kserve-7d7c4564bf-cbv2c" Nov 03 00:55:05.755052 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:55:05.755019 4939 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-88bbf7/distinct-2-w1-88bbf7-kserve-5457cdcdbd-l6tfd" Nov 03 00:55:05.755898 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:55:05.755856 4939 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-2f9a3a/distinct-w0-2f9a3a-kserve-555565cfcf-jtcr9" podUID="ac0ece06-8718-463f-915f-5315988d9711" containerName="main" probeResult="failure" output="Get \"https://10.132.0.76:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Nov 03 00:55:05.774659 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:55:05.774628 4939 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-2f9a3a/distinct-w0-2f9a3a-kserve-555565cfcf-jtcr9" Nov 03 00:55:05.823761 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:55:05.823732 4939 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-f7680d/premium-w2-f7680d-kserve-76565b8cf9-hqbbt" Nov 03 00:55:05.973236 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:55:05.973206 4939 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-f7680d/distinct-w2-f7680d-kserve-756f874b98-qnpv8" Nov 03 00:55:06.270468 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:55:06.270428 4939 patch_prober.go:29] interesting pod/e2e-worker-w1-88bbf7-openshift-default-574f47fff7-k87gb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.86:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 00:55:06.271053 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:55:06.270499 4939 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-88bbf7-openshift-default-574f47fff7-k87gb" podUID="c7907eaa-df92-4d28-aa19-ba6f03cb54b1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.86:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 00:55:06.595900 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:55:06.595857 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 00:55:06.827258 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:55:06.827229 4939 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-88bbf7-openshift-default-574f47fff7-k87gb" Nov 03 00:55:06.828432 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:55:06.828406 4939 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-88bbf7-openshift-default-574f47fff7-k87gb" Nov 03 00:55:06.829684 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:55:06.829657 4939 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-88bbf7-openshift-default-574f47fff7-k87gb" Nov 03 00:55:07.467643 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:55:07.467595 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 00:55:08.176791 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:55:08.176762 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 00:55:08.908145 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:55:08.908107 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 00:55:09.689725 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:55:09.689679 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 00:55:09.830946 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:55:09.828324 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 00:55:09.950275 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:55:09.950199 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 00:55:10.603542 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:55:10.603463 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 00:55:10.779180 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:55:10.779145 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 00:55:10.892710 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:55:10.892620 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 00:55:11.004827 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:55:11.004772 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 00:55:11.369548 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:55:11.369513 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 00:55:11.485617 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:55:11.485577 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 00:55:11.775417 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:55:11.775330 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 00:55:12.077333 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:55:12.077236 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 00:55:12.400192 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:55:12.400158 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 00:55:13.193595 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:55:13.193556 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 00:55:13.800188 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:55:13.800155 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 00:55:14.598104 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:55:14.598066 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 00:55:15.083805 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:55:15.082897 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 00:55:15.199817 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:55:15.199784 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 00:55:15.354819 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:55:15.354726 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 00:55:15.969904 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:55:15.969873 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 00:55:16.258047 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:55:16.257953 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 00:55:16.336078 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:55:16.336046 4939 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-dd03e395-779465d96c-tckp6"] Nov 03 00:55:16.341678 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:55:16.341656 4939 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-dd03e395-779465d96c-tckp6" Nov 03 00:55:16.347866 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:55:16.347842 4939 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-dd03e395-779465d96c-tckp6"] Nov 03 00:55:16.348132 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:55:16.348115 4939 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-mt-dd03e395\"" Nov 03 00:55:16.349632 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:55:16.349610 4939 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-mt-dd03e395-dockercfg-7277c\"" Nov 03 00:55:16.380376 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:55:16.380341 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 00:55:16.382464 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:55:16.382436 4939 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-dd03e395-b8464b49-p8872"] Nov 03 00:55:16.392313 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:55:16.392279 4939 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-dd03e395-b8464b49-p8872" Nov 03 00:55:16.393780 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:55:16.393754 4939 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-dd03e395-b8464b49-p8872"] Nov 03 00:55:16.464934 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:55:16.460290 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f9c818e4-813f-40dd-9311-906a517eb42a-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-dd03e395-779465d96c-tckp6\" (UID: \"f9c818e4-813f-40dd-9311-906a517eb42a\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-dd03e395-779465d96c-tckp6" Nov 03 00:55:16.464934 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:55:16.460388 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f975j\" (UniqueName: \"kubernetes.io/projected/f9c818e4-813f-40dd-9311-906a517eb42a-kube-api-access-f975j\") pod \"payload-pre-processing-e2e-mt-dd03e395-779465d96c-tckp6\" (UID: \"f9c818e4-813f-40dd-9311-906a517eb42a\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-dd03e395-779465d96c-tckp6" Nov 03 00:55:16.464934 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:55:16.460435 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dfcf5830-ce86-40fb-bdfb-3564175ac08a-plugins-config-volume\") pod \"payload-processing-e2e-mt-dd03e395-b8464b49-p8872\" (UID: \"dfcf5830-ce86-40fb-bdfb-3564175ac08a\") " pod="openshift-ingress/payload-processing-e2e-mt-dd03e395-b8464b49-p8872" Nov 03 00:55:16.464934 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:55:16.460516 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8btt9\" (UniqueName: \"kubernetes.io/projected/dfcf5830-ce86-40fb-bdfb-3564175ac08a-kube-api-access-8btt9\") pod \"payload-processing-e2e-mt-dd03e395-b8464b49-p8872\" (UID: \"dfcf5830-ce86-40fb-bdfb-3564175ac08a\") " pod="openshift-ingress/payload-processing-e2e-mt-dd03e395-b8464b49-p8872" Nov 03 00:55:16.514995 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:55:16.512589 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 00:55:16.573930 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:55:16.572893 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f975j\" (UniqueName: \"kubernetes.io/projected/f9c818e4-813f-40dd-9311-906a517eb42a-kube-api-access-f975j\") pod \"payload-pre-processing-e2e-mt-dd03e395-779465d96c-tckp6\" (UID: \"f9c818e4-813f-40dd-9311-906a517eb42a\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-dd03e395-779465d96c-tckp6" Nov 03 00:55:16.573930 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:55:16.572977 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dfcf5830-ce86-40fb-bdfb-3564175ac08a-plugins-config-volume\") pod \"payload-processing-e2e-mt-dd03e395-b8464b49-p8872\" (UID: \"dfcf5830-ce86-40fb-bdfb-3564175ac08a\") " pod="openshift-ingress/payload-processing-e2e-mt-dd03e395-b8464b49-p8872" Nov 03 00:55:16.573930 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:55:16.573064 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8btt9\" (UniqueName: \"kubernetes.io/projected/dfcf5830-ce86-40fb-bdfb-3564175ac08a-kube-api-access-8btt9\") pod \"payload-processing-e2e-mt-dd03e395-b8464b49-p8872\" (UID: \"dfcf5830-ce86-40fb-bdfb-3564175ac08a\") " pod="openshift-ingress/payload-processing-e2e-mt-dd03e395-b8464b49-p8872" Nov 03 00:55:16.573930 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:55:16.573292 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f9c818e4-813f-40dd-9311-906a517eb42a-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-dd03e395-779465d96c-tckp6\" (UID: \"f9c818e4-813f-40dd-9311-906a517eb42a\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-dd03e395-779465d96c-tckp6" Nov 03 00:55:16.610930 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:55:16.609405 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f9c818e4-813f-40dd-9311-906a517eb42a-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-dd03e395-779465d96c-tckp6\" (UID: \"f9c818e4-813f-40dd-9311-906a517eb42a\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-dd03e395-779465d96c-tckp6" Nov 03 00:55:16.611129 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:55:16.611067 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dfcf5830-ce86-40fb-bdfb-3564175ac08a-plugins-config-volume\") pod \"payload-processing-e2e-mt-dd03e395-b8464b49-p8872\" (UID: \"dfcf5830-ce86-40fb-bdfb-3564175ac08a\") " pod="openshift-ingress/payload-processing-e2e-mt-dd03e395-b8464b49-p8872" Nov 03 00:55:16.655619 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:55:16.655514 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8btt9\" (UniqueName: \"kubernetes.io/projected/dfcf5830-ce86-40fb-bdfb-3564175ac08a-kube-api-access-8btt9\") pod \"payload-processing-e2e-mt-dd03e395-b8464b49-p8872\" (UID: \"dfcf5830-ce86-40fb-bdfb-3564175ac08a\") " pod="openshift-ingress/payload-processing-e2e-mt-dd03e395-b8464b49-p8872" Nov 03 00:55:16.668821 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:55:16.668771 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-f975j\" (UniqueName: \"kubernetes.io/projected/f9c818e4-813f-40dd-9311-906a517eb42a-kube-api-access-f975j\") pod \"payload-pre-processing-e2e-mt-dd03e395-779465d96c-tckp6\" (UID: \"f9c818e4-813f-40dd-9311-906a517eb42a\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-dd03e395-779465d96c-tckp6" Nov 03 00:55:16.948506 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:55:16.948474 4939 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-dd03e395-b8464b49-p8872" Nov 03 00:55:16.959807 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:55:16.959783 4939 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-dd03e395-779465d96c-tckp6" Nov 03 00:55:17.237279 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:55:17.237251 4939 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-dd03e395-b8464b49-p8872"] Nov 03 00:55:17.239142 ip-10-0-140-107 kubenswrapper[4939]: W1103 00:55:17.239116 4939 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddfcf5830_ce86_40fb_bdfb_3564175ac08a.slice/crio-41328b17babe6447b177ce6b242c6f18234a4305f5e5e8e9bee925ee2a55cdeb WatchSource:0}: Error finding container 41328b17babe6447b177ce6b242c6f18234a4305f5e5e8e9bee925ee2a55cdeb: Status 404 returned error can't find the container with id 41328b17babe6447b177ce6b242c6f18234a4305f5e5e8e9bee925ee2a55cdeb Nov 03 00:55:17.246955 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:55:17.246753 4939 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-dd03e395-779465d96c-tckp6"] Nov 03 00:55:17.250050 ip-10-0-140-107 kubenswrapper[4939]: W1103 00:55:17.250014 4939 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf9c818e4_813f_40dd_9311_906a517eb42a.slice/crio-2d1e4798ddf894d5878c75f4ad1ced70419275649d2105d63a80858ba03f8d9f WatchSource:0}: Error finding container 2d1e4798ddf894d5878c75f4ad1ced70419275649d2105d63a80858ba03f8d9f: Status 404 returned error can't find the container with id 2d1e4798ddf894d5878c75f4ad1ced70419275649d2105d63a80858ba03f8d9f Nov 03 00:55:17.466366 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:55:17.466309 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 00:55:18.032999 ip-10-0-140-107 kubenswrapper[4939]: E1103 00:55:18.032960 4939 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-sthjh" containerName="extractor" Nov 03 00:55:18.108983 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:55:18.108943 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 00:55:18.120941 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:55:18.120881 4939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-dd03e395-b8464b49-p8872" event={"ID":"dfcf5830-ce86-40fb-bdfb-3564175ac08a","Type":"ContainerStarted","Data":"f10b27fe436ff74ddfd71c25ff4c67ea2f10276544f729b75b518d51d4cd4930"} Nov 03 00:55:18.121133 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:55:18.120966 4939 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-mt-dd03e395-b8464b49-p8872" Nov 03 00:55:18.121133 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:55:18.120988 4939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-dd03e395-b8464b49-p8872" event={"ID":"dfcf5830-ce86-40fb-bdfb-3564175ac08a","Type":"ContainerStarted","Data":"41328b17babe6447b177ce6b242c6f18234a4305f5e5e8e9bee925ee2a55cdeb"} Nov 03 00:55:18.122535 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:55:18.122509 4939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-dd03e395-779465d96c-tckp6" event={"ID":"f9c818e4-813f-40dd-9311-906a517eb42a","Type":"ContainerStarted","Data":"fd6221e9ac6ab31991e76723f19e79802ad1765ded99357a8dcf8ee7c0bd3ae0"} Nov 03 00:55:18.122648 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:55:18.122550 4939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-dd03e395-779465d96c-tckp6" event={"ID":"f9c818e4-813f-40dd-9311-906a517eb42a","Type":"ContainerStarted","Data":"2d1e4798ddf894d5878c75f4ad1ced70419275649d2105d63a80858ba03f8d9f"} Nov 03 00:55:18.122648 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:55:18.122569 4939 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-dd03e395-779465d96c-tckp6" Nov 03 00:55:18.191093 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:55:18.191026 4939 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-mt-dd03e395-b8464b49-p8872" podStartSLOduration=2.191003391 podStartE2EDuration="2.191003391s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:55:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:55:18.151441685 +0000 UTC m=+866.415776632" watchObservedRunningTime="2026-11-03 00:55:18.191003391 +0000 UTC m=+866.455338338" Nov 03 00:55:18.192361 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:55:18.192309 4939 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-mt-dd03e395-779465d96c-tckp6" podStartSLOduration=2.192295948 podStartE2EDuration="2.192295948s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:55:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:55:18.171676735 +0000 UTC m=+866.436011683" watchObservedRunningTime="2026-11-03 00:55:18.192295948 +0000 UTC m=+866.456630892" Nov 03 00:55:18.807122 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:55:18.807085 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 00:55:18.915467 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:55:18.915421 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 00:55:20.375933 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:55:20.375575 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 00:55:20.764993 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:55:20.764020 4939 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-2f9a3a/distinct-w0-2f9a3a-kserve-555565cfcf-jtcr9" podUID="ac0ece06-8718-463f-915f-5315988d9711" containerName="main" probeResult="failure" output="Get \"https://10.132.0.76:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 00:55:21.123585 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:55:21.123534 4939 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-83755a/distinct-2-w3-83755a-kserve-6cf66b995d-2l47b" podUID="c3b05e88-5d5b-4d03-b28d-832f22e3c868" containerName="main" probeResult="failure" output="Get \"https://10.132.0.72:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 00:55:21.153081 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:55:21.153035 4939 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-2f9a3a/simulator-w0-2f9a3a-kserve-9596bbb8f-2trgz" podUID="207e2207-fb6c-4076-95df-72e95671c261" containerName="main" probeResult="failure" output="Get \"https://10.132.0.61:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 00:55:21.174054 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:55:21.174009 4939 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-88bbf7/distinct-2-w1-88bbf7-kserve-5457cdcdbd-l6tfd" podUID="a397931b-faa9-4c2e-8290-491350546663" containerName="main" probeResult="failure" output="Get \"https://10.132.0.85:8000/ready\": context deadline exceeded" Nov 03 00:55:21.393120 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:55:21.393009 4939 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-f7680d/distinct-2-w2-f7680d-kserve-7d7c4564bf-cbv2c" podUID="2de18086-761b-4773-b9fc-15ddc4a220fc" containerName="main" probeResult="failure" output="Get \"https://10.132.0.75:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 00:55:21.435134 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:55:21.435085 4939 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-f7680d/premium-w2-f7680d-kserve-76565b8cf9-hqbbt" podUID="4712e708-18a2-4a13-a12c-a5669397b2c3" containerName="main" probeResult="failure" output="Get \"https://10.132.0.63:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 00:55:21.704127 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:55:21.704011 4939 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-88bbf7/simulator-w1-88bbf7-kserve-5fbb669f8f-42hgd" podUID="83b38883-2e7a-444d-aa68-52be4a25ae66" containerName="main" probeResult="failure" output="Get \"https://10.132.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 00:55:21.739062 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:55:21.739010 4939 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-f7680d/distinct-w2-f7680d-kserve-756f874b98-qnpv8" podUID="836753fc-f9ef-4f61-b3fe-e5b92b7ee27b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.70:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 00:55:21.781300 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:55:21.774008 4939 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-2f9a3a/distinct-w0-2f9a3a-kserve-555565cfcf-jtcr9" podUID="ac0ece06-8718-463f-915f-5315988d9711" containerName="main" probeResult="failure" output="Get \"https://10.132.0.76:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 00:55:21.781300 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:55:21.774500 4939 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-2f9a3a/distinct-w0-2f9a3a-kserve-555565cfcf-jtcr9" Nov 03 00:55:21.969900 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:55:21.969794 4939 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-f7680d/premium-w2-f7680d-kserve-76565b8cf9-hqbbt" podUID="4712e708-18a2-4a13-a12c-a5669397b2c3" containerName="main" probeResult="failure" output="Get \"https://10.132.0.63:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 00:55:22.087436 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:55:22.038644 4939 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-f7680d/distinct-2-w2-f7680d-kserve-7d7c4564bf-cbv2c" podUID="2de18086-761b-4773-b9fc-15ddc4a220fc" containerName="main" probeResult="failure" output="Get \"https://10.132.0.75:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 00:55:22.188936 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:55:22.176013 4939 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-88bbf7/distinct-2-w1-88bbf7-kserve-5457cdcdbd-l6tfd" podUID="a397931b-faa9-4c2e-8290-491350546663" containerName="main" probeResult="failure" output="Get \"https://10.132.0.85:8000/ready\": context deadline exceeded" Nov 03 00:55:22.188936 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:55:22.176867 4939 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-2f9a3a/simulator-w0-2f9a3a-kserve-9596bbb8f-2trgz" podUID="207e2207-fb6c-4076-95df-72e95671c261" containerName="main" probeResult="failure" output="Get \"https://10.132.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 00:55:22.398473 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:55:22.398423 4939 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-f7680d/distinct-2-w2-f7680d-kserve-7d7c4564bf-cbv2c" podUID="2de18086-761b-4773-b9fc-15ddc4a220fc" containerName="main" probeResult="failure" output="Get \"https://10.132.0.75:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 00:55:22.402211 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:55:22.402170 4939 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-83755a/distinct-2-w3-83755a-kserve-6cf66b995d-2l47b" podUID="c3b05e88-5d5b-4d03-b28d-832f22e3c868" containerName="main" probeResult="failure" output="Get \"https://10.132.0.72:8000/ready\": context deadline exceeded" Nov 03 00:55:22.457336 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:55:22.457299 4939 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-88bbf7/distinct-2-w1-88bbf7-kserve-5457cdcdbd-l6tfd" Nov 03 00:55:22.516675 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:55:22.457598 4939 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-f7680d/premium-w2-f7680d-kserve-76565b8cf9-hqbbt" podUID="4712e708-18a2-4a13-a12c-a5669397b2c3" containerName="main" probeResult="failure" output="Get \"https://10.132.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 00:55:22.540936 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:55:22.517292 4939 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-2f9a3a/simulator-w0-2f9a3a-kserve-9596bbb8f-2trgz" Nov 03 00:55:22.606687 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:55:22.564155 4939 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-f7680d/distinct-2-w2-f7680d-kserve-7d7c4564bf-cbv2c" Nov 03 00:55:22.608360 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:55:22.608339 4939 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-83755a/distinct-2-w3-83755a-kserve-6cf66b995d-2l47b" Nov 03 00:55:22.608853 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:55:22.608827 4939 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-f7680d/premium-w2-f7680d-kserve-76565b8cf9-hqbbt" Nov 03 00:55:22.742135 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:55:22.742021 4939 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-f7680d/distinct-w2-f7680d-kserve-756f874b98-qnpv8" podUID="836753fc-f9ef-4f61-b3fe-e5b92b7ee27b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.70:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 00:55:22.824944 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:55:22.771422 4939 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-88bbf7/simulator-w1-88bbf7-kserve-5fbb669f8f-42hgd" podUID="83b38883-2e7a-444d-aa68-52be4a25ae66" containerName="main" probeResult="failure" output="Get \"https://10.132.0.62:8000/ready\": context deadline exceeded" Nov 03 00:55:22.824944 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:55:22.772176 4939 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-f7680d/distinct-w2-f7680d-kserve-756f874b98-qnpv8" Nov 03 00:55:22.824944 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:55:22.772238 4939 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-88bbf7/simulator-w1-88bbf7-kserve-5fbb669f8f-42hgd" Nov 03 00:55:22.824944 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:55:22.781897 4939 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-2f9a3a/distinct-w0-2f9a3a-kserve-555565cfcf-jtcr9" podUID="ac0ece06-8718-463f-915f-5315988d9711" containerName="main" probeResult="failure" output="Get \"https://10.132.0.76:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 00:55:23.221936 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:55:23.221181 4939 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-2f9a3a/simulator-w0-2f9a3a-kserve-9596bbb8f-2trgz" podUID="207e2207-fb6c-4076-95df-72e95671c261" containerName="main" probeResult="failure" output="Get \"https://10.132.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 00:55:23.274486 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:55:23.274004 4939 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-88bbf7/distinct-2-w1-88bbf7-kserve-5457cdcdbd-l6tfd" podUID="a397931b-faa9-4c2e-8290-491350546663" containerName="main" probeResult="failure" output="Get \"https://10.132.0.85:8000/ready\": context deadline exceeded" Nov 03 00:55:23.455084 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:55:23.455039 4939 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-83755a/distinct-2-w3-83755a-kserve-6cf66b995d-2l47b" podUID="c3b05e88-5d5b-4d03-b28d-832f22e3c868" containerName="main" probeResult="failure" output="Get \"https://10.132.0.72:8000/ready\": context deadline exceeded" Nov 03 00:55:23.491996 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:55:23.491074 4939 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-f7680d/distinct-2-w2-f7680d-kserve-7d7c4564bf-cbv2c" podUID="2de18086-761b-4773-b9fc-15ddc4a220fc" containerName="main" probeResult="failure" output="Get \"https://10.132.0.75:8000/ready\": context deadline exceeded" Nov 03 00:55:23.528315 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:55:23.526626 4939 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-f7680d/premium-w2-f7680d-kserve-76565b8cf9-hqbbt" podUID="4712e708-18a2-4a13-a12c-a5669397b2c3" containerName="main" probeResult="failure" output="Get \"https://10.132.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 00:55:23.768985 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:55:23.753422 4939 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-f7680d/distinct-w2-f7680d-kserve-756f874b98-qnpv8" podUID="836753fc-f9ef-4f61-b3fe-e5b92b7ee27b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.70:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 00:55:23.808951 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:55:23.776498 4939 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-88bbf7/simulator-w1-88bbf7-kserve-5fbb669f8f-42hgd" podUID="83b38883-2e7a-444d-aa68-52be4a25ae66" containerName="main" probeResult="failure" output="Get \"https://10.132.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 00:55:23.845938 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:55:23.796464 4939 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-2f9a3a/distinct-w0-2f9a3a-kserve-555565cfcf-jtcr9" podUID="ac0ece06-8718-463f-915f-5315988d9711" containerName="main" probeResult="failure" output="Get \"https://10.132.0.76:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 00:55:24.333524 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:55:24.333479 4939 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-88bbf7/distinct-2-w1-88bbf7-kserve-5457cdcdbd-l6tfd" podUID="a397931b-faa9-4c2e-8290-491350546663" containerName="main" probeResult="failure" output="Get \"https://10.132.0.85:8000/ready\": context deadline exceeded" Nov 03 00:55:24.335098 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:55:24.335060 4939 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-2f9a3a/simulator-w0-2f9a3a-kserve-9596bbb8f-2trgz" podUID="207e2207-fb6c-4076-95df-72e95671c261" containerName="main" probeResult="failure" output="Get \"https://10.132.0.61:8000/ready\": context deadline exceeded" Nov 03 00:55:24.345683 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:55:24.333314 4939 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-2f9a3a/distinct-w0-2f9a3a-kserve-555565cfcf-jtcr9" podUID="ac0ece06-8718-463f-915f-5315988d9711" containerName="main" probeResult="failure" output="Get \"https://10.132.0.76:8000/health\": context deadline exceeded" Nov 03 00:55:24.375101 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:55:24.375061 4939 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-f7680d/distinct-2-w2-f7680d-kserve-7d7c4564bf-cbv2c" Nov 03 00:55:24.474935 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:55:24.469438 4939 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-83755a/distinct-2-w3-83755a-kserve-6cf66b995d-2l47b" podUID="c3b05e88-5d5b-4d03-b28d-832f22e3c868" containerName="main" probeResult="failure" output="Get \"https://10.132.0.72:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 00:55:24.697020 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:55:24.678888 4939 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-f7680d/premium-w2-f7680d-kserve-76565b8cf9-hqbbt" podUID="4712e708-18a2-4a13-a12c-a5669397b2c3" containerName="main" probeResult="failure" output="Get \"https://10.132.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 00:55:24.764966 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:55:24.756697 4939 patch_prober.go:29] interesting pod/e2e-worker-w2-f7680d-openshift-default-c4488d5c9-h5j8d container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.65:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 00:55:24.764966 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:55:24.756783 4939 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w2-f7680d-openshift-default-c4488d5c9-h5j8d" podUID="4fe351d2-d66c-4cd4-8bf7-264a40d2551c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.65:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 00:55:24.796330 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:55:24.796276 4939 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-f7680d/distinct-w2-f7680d-kserve-756f874b98-qnpv8" podUID="836753fc-f9ef-4f61-b3fe-e5b92b7ee27b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.70:8000/ready\": context deadline exceeded" Nov 03 00:55:24.825577 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:55:24.824814 4939 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-88bbf7/simulator-w1-88bbf7-kserve-5fbb669f8f-42hgd" podUID="83b38883-2e7a-444d-aa68-52be4a25ae66" containerName="main" probeResult="failure" output="Get \"https://10.132.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 00:55:24.859355 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:55:24.858134 4939 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-2f9a3a/distinct-w0-2f9a3a-kserve-555565cfcf-jtcr9" podUID="ac0ece06-8718-463f-915f-5315988d9711" containerName="main" probeResult="failure" output="Get \"https://10.132.0.76:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 00:55:25.063577 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:55:25.063483 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 00:55:25.283535 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:55:25.281725 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 00:55:25.437166 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:55:25.437096 4939 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-88bbf7/distinct-2-w1-88bbf7-kserve-5457cdcdbd-l6tfd" podUID="a397931b-faa9-4c2e-8290-491350546663" containerName="main" probeResult="failure" output="Get \"https://10.132.0.85:8000/ready\": context deadline exceeded" Nov 03 00:55:25.500226 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:55:25.457735 4939 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-2f9a3a/simulator-w0-2f9a3a-kserve-9596bbb8f-2trgz" podUID="207e2207-fb6c-4076-95df-72e95671c261" containerName="main" probeResult="failure" output="Get \"https://10.132.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 00:55:25.501152 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:55:25.461543 4939 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-f7680d/distinct-2-w2-f7680d-kserve-7d7c4564bf-cbv2c" podUID="2de18086-761b-4773-b9fc-15ddc4a220fc" containerName="main" probeResult="failure" output="Get \"https://10.132.0.75:8000/ready\": context deadline exceeded" Nov 03 00:55:25.509944 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:55:25.461625 4939 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-88bbf7/distinct-2-w1-88bbf7-kserve-5457cdcdbd-l6tfd" podUID="a397931b-faa9-4c2e-8290-491350546663" containerName="main" probeResult="failure" output="Get \"https://10.132.0.85:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 00:55:25.521932 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:55:25.488456 4939 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-83755a/distinct-2-w3-83755a-kserve-6cf66b995d-2l47b" podUID="c3b05e88-5d5b-4d03-b28d-832f22e3c868" containerName="main" probeResult="failure" output="Get \"https://10.132.0.72:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 00:55:25.680259 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:55:25.680211 4939 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-f7680d/premium-w2-f7680d-kserve-76565b8cf9-hqbbt" podUID="4712e708-18a2-4a13-a12c-a5669397b2c3" containerName="main" probeResult="failure" output="Get \"https://10.132.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 00:55:25.802988 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:55:25.793171 4939 patch_prober.go:29] interesting pod/e2e-worker-w1-88bbf7-openshift-default-574f47fff7-rf5cn container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.59:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 00:55:25.802988 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:55:25.793237 4939 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w1-88bbf7-openshift-default-574f47fff7-rf5cn" podUID="08f62205-b31d-4863-ab9e-3b13b15fbef5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.59:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 00:55:25.853936 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:55:25.830818 4939 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-f7680d/distinct-w2-f7680d-kserve-756f874b98-qnpv8" podUID="836753fc-f9ef-4f61-b3fe-e5b92b7ee27b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.70:8000/ready\": context deadline exceeded" Nov 03 00:55:25.853936 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:55:25.832389 4939 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-88bbf7/simulator-w1-88bbf7-kserve-5fbb669f8f-42hgd" podUID="83b38883-2e7a-444d-aa68-52be4a25ae66" containerName="main" probeResult="failure" output="Get \"https://10.132.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 00:55:25.863257 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:55:25.863212 4939 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-2f9a3a/distinct-w0-2f9a3a-kserve-555565cfcf-jtcr9" podUID="ac0ece06-8718-463f-915f-5315988d9711" containerName="main" probeResult="failure" output="Get \"https://10.132.0.76:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 00:55:26.000049 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:55:26.000001 4939 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-2f9a3a/simulator-w0-2f9a3a-kserve-9596bbb8f-2trgz" podUID="207e2207-fb6c-4076-95df-72e95671c261" containerName="main" probeResult="failure" output="Get \"https://10.132.0.61:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 00:55:26.252849 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:55:26.163645 4939 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-83755a/distinct-2-w3-83755a-kserve-6cf66b995d-2l47b" podUID="c3b05e88-5d5b-4d03-b28d-832f22e3c868" containerName="main" probeResult="failure" output="Get \"https://10.132.0.72:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 00:55:26.453083 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:55:26.453033 4939 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-88bbf7/distinct-2-w1-88bbf7-kserve-5457cdcdbd-l6tfd" podUID="a397931b-faa9-4c2e-8290-491350546663" containerName="main" probeResult="failure" output="Get \"https://10.132.0.85:8000/ready\": context deadline exceeded" Nov 03 00:55:26.505152 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:55:26.505049 4939 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-f7680d/distinct-2-w2-f7680d-kserve-7d7c4564bf-cbv2c" podUID="2de18086-761b-4773-b9fc-15ddc4a220fc" containerName="main" probeResult="failure" output="Get \"https://10.132.0.75:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 00:55:26.514006 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:55:26.513966 4939 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-2f9a3a/simulator-w0-2f9a3a-kserve-9596bbb8f-2trgz" podUID="207e2207-fb6c-4076-95df-72e95671c261" containerName="main" probeResult="failure" output="Get \"https://10.132.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 00:55:26.517765 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:55:26.517737 4939 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-f7680d/distinct-2-w2-f7680d-kserve-7d7c4564bf-cbv2c" Nov 03 00:55:26.548935 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:55:26.527247 4939 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-83755a/distinct-2-w3-83755a-kserve-6cf66b995d-2l47b" podUID="c3b05e88-5d5b-4d03-b28d-832f22e3c868" containerName="main" probeResult="failure" output="Get \"https://10.132.0.72:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 00:55:26.721421 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:55:26.708011 4939 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-f7680d/premium-w2-f7680d-kserve-76565b8cf9-hqbbt" podUID="4712e708-18a2-4a13-a12c-a5669397b2c3" containerName="main" probeResult="failure" output="Get \"https://10.132.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 00:55:26.858942 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:55:26.836022 4939 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-88bbf7/simulator-w1-88bbf7-kserve-5fbb669f8f-42hgd" podUID="83b38883-2e7a-444d-aa68-52be4a25ae66" containerName="main" probeResult="failure" output="Get \"https://10.132.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 00:55:26.858942 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:55:26.842984 4939 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-f7680d/distinct-w2-f7680d-kserve-756f874b98-qnpv8" podUID="836753fc-f9ef-4f61-b3fe-e5b92b7ee27b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.70:8000/ready\": context deadline exceeded" Nov 03 00:55:26.890943 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:55:26.884028 4939 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-2f9a3a/distinct-w0-2f9a3a-kserve-555565cfcf-jtcr9" podUID="ac0ece06-8718-463f-915f-5315988d9711" containerName="main" probeResult="failure" output="Get \"https://10.132.0.76:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 00:55:27.143999 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:55:27.143850 4939 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-f7680d/distinct-2-w2-f7680d-kserve-7d7c4564bf-cbv2c" Nov 03 00:55:27.446481 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:55:27.446054 4939 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-596dc9545f-9jvbb container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.41:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 00:55:27.446481 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:55:27.446123 4939 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-9jvbb" podUID="512d26b5-8ad9-4cca-8687-cd06d31d8369" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.41:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 00:55:27.466079 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:55:27.466037 4939 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-88bbf7/distinct-2-w1-88bbf7-kserve-5457cdcdbd-l6tfd" podUID="a397931b-faa9-4c2e-8290-491350546663" containerName="main" probeResult="failure" output="Get \"https://10.132.0.85:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 00:55:27.518852 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:55:27.518774 4939 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-2f9a3a/simulator-w0-2f9a3a-kserve-9596bbb8f-2trgz" podUID="207e2207-fb6c-4076-95df-72e95671c261" containerName="main" probeResult="failure" output="Get \"https://10.132.0.61:8000/ready\": context deadline exceeded" Nov 03 00:55:27.549678 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:55:27.549635 4939 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-83755a/distinct-2-w3-83755a-kserve-6cf66b995d-2l47b" podUID="c3b05e88-5d5b-4d03-b28d-832f22e3c868" containerName="main" probeResult="failure" output="Get \"https://10.132.0.72:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 00:55:27.684307 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:55:27.684274 4939 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-f7680d/distinct-w2-f7680d-kserve-756f874b98-qnpv8" Nov 03 00:55:27.719503 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:55:27.718525 4939 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-f7680d/premium-w2-f7680d-kserve-76565b8cf9-hqbbt" podUID="4712e708-18a2-4a13-a12c-a5669397b2c3" containerName="main" probeResult="failure" output="Get \"https://10.132.0.63:8000/ready\": context deadline exceeded" Nov 03 00:55:27.720833 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:55:27.720757 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 00:55:27.838930 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:55:27.838061 4939 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-88bbf7/simulator-w1-88bbf7-kserve-5fbb669f8f-42hgd" podUID="83b38883-2e7a-444d-aa68-52be4a25ae66" containerName="main" probeResult="failure" output="Get \"https://10.132.0.62:8000/ready\": context deadline exceeded" Nov 03 00:55:27.853055 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:55:27.852965 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 00:55:27.900333 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:55:27.897370 4939 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-2f9a3a/distinct-w0-2f9a3a-kserve-555565cfcf-jtcr9" podUID="ac0ece06-8718-463f-915f-5315988d9711" containerName="main" probeResult="failure" output="Get \"https://10.132.0.76:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 00:55:28.079681 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:55:28.079594 4939 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-f7680d/premium-w2-f7680d-kserve-76565b8cf9-hqbbt" Nov 03 00:55:28.193221 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:55:28.192736 4939 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-2f9a3a/distinct-w0-2f9a3a-kserve-555565cfcf-jtcr9" Nov 03 00:55:28.194407 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:55:28.194377 4939 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-88bbf7/simulator-w1-88bbf7-kserve-5fbb669f8f-42hgd" podUID="83b38883-2e7a-444d-aa68-52be4a25ae66" containerName="main" probeResult="failure" output="Get \"https://10.132.0.62:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 00:55:28.215818 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:55:28.215789 4939 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-88bbf7/simulator-w1-88bbf7-kserve-5fbb669f8f-42hgd" Nov 03 00:55:28.222319 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:55:28.222288 4939 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-2f9a3a/simulator-w0-2f9a3a-kserve-9596bbb8f-2trgz" Nov 03 00:55:28.281932 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:55:28.281822 4939 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-83755a/distinct-2-w3-83755a-kserve-6cf66b995d-2l47b" Nov 03 00:55:28.319586 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:55:28.306792 4939 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-88bbf7/distinct-2-w1-88bbf7-kserve-5457cdcdbd-l6tfd" Nov 03 00:55:28.780527 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:55:28.780492 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 00:55:29.132759 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:55:29.132731 4939 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-mt-dd03e395-b8464b49-p8872" Nov 03 00:55:29.132957 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:55:29.132793 4939 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-dd03e395-779465d96c-tckp6" Nov 03 00:55:29.479390 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:55:29.479301 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 00:55:30.199189 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:55:30.199152 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 00:55:30.872206 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:55:30.872170 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 00:55:31.011519 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:55:31.009451 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 00:55:31.840550 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:55:31.840518 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 00:55:32.655235 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:55:32.655203 4939 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-2f9a3a-openshift-default-7f6f74cf97-ph469"] Nov 03 00:55:32.690208 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:55:32.690164 4939 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-2f9a3a-openshift-default-7f6f74cf97-ph469" Nov 03 00:55:32.695608 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:55:32.695580 4939 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-2f9a3a-openshift-default-7f6f74cf97-ckb7r"] Nov 03 00:55:32.700042 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:55:32.700009 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5ca6f6a0-78ff-4b3f-ad1f-b94428b9178e-workload-certs\") pod \"e2e-worker-w0-2f9a3a-openshift-default-7f6f74cf97-ph469\" (UID: \"5ca6f6a0-78ff-4b3f-ad1f-b94428b9178e\") " pod="openshift-ingress/e2e-worker-w0-2f9a3a-openshift-default-7f6f74cf97-ph469" Nov 03 00:55:32.700188 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:55:32.700067 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5ca6f6a0-78ff-4b3f-ad1f-b94428b9178e-istio-data\") pod \"e2e-worker-w0-2f9a3a-openshift-default-7f6f74cf97-ph469\" (UID: \"5ca6f6a0-78ff-4b3f-ad1f-b94428b9178e\") " pod="openshift-ingress/e2e-worker-w0-2f9a3a-openshift-default-7f6f74cf97-ph469" Nov 03 00:55:32.700188 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:55:32.700167 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5ca6f6a0-78ff-4b3f-ad1f-b94428b9178e-istiod-ca-cert\") pod \"e2e-worker-w0-2f9a3a-openshift-default-7f6f74cf97-ph469\" (UID: \"5ca6f6a0-78ff-4b3f-ad1f-b94428b9178e\") " pod="openshift-ingress/e2e-worker-w0-2f9a3a-openshift-default-7f6f74cf97-ph469" Nov 03 00:55:32.700294 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:55:32.700230 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5ca6f6a0-78ff-4b3f-ad1f-b94428b9178e-istio-envoy\") pod \"e2e-worker-w0-2f9a3a-openshift-default-7f6f74cf97-ph469\" (UID: \"5ca6f6a0-78ff-4b3f-ad1f-b94428b9178e\") " pod="openshift-ingress/e2e-worker-w0-2f9a3a-openshift-default-7f6f74cf97-ph469" Nov 03 00:55:32.700345 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:55:32.700311 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5ca6f6a0-78ff-4b3f-ad1f-b94428b9178e-credential-socket\") pod \"e2e-worker-w0-2f9a3a-openshift-default-7f6f74cf97-ph469\" (UID: \"5ca6f6a0-78ff-4b3f-ad1f-b94428b9178e\") " pod="openshift-ingress/e2e-worker-w0-2f9a3a-openshift-default-7f6f74cf97-ph469" Nov 03 00:55:32.700407 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:55:32.700364 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9kjbw\" (UniqueName: \"kubernetes.io/projected/5ca6f6a0-78ff-4b3f-ad1f-b94428b9178e-kube-api-access-9kjbw\") pod \"e2e-worker-w0-2f9a3a-openshift-default-7f6f74cf97-ph469\" (UID: \"5ca6f6a0-78ff-4b3f-ad1f-b94428b9178e\") " pod="openshift-ingress/e2e-worker-w0-2f9a3a-openshift-default-7f6f74cf97-ph469" Nov 03 00:55:32.700499 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:55:32.700483 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5ca6f6a0-78ff-4b3f-ad1f-b94428b9178e-workload-socket\") pod \"e2e-worker-w0-2f9a3a-openshift-default-7f6f74cf97-ph469\" (UID: \"5ca6f6a0-78ff-4b3f-ad1f-b94428b9178e\") " pod="openshift-ingress/e2e-worker-w0-2f9a3a-openshift-default-7f6f74cf97-ph469" Nov 03 00:55:32.700557 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:55:32.700545 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5ca6f6a0-78ff-4b3f-ad1f-b94428b9178e-istio-podinfo\") pod \"e2e-worker-w0-2f9a3a-openshift-default-7f6f74cf97-ph469\" (UID: \"5ca6f6a0-78ff-4b3f-ad1f-b94428b9178e\") " pod="openshift-ingress/e2e-worker-w0-2f9a3a-openshift-default-7f6f74cf97-ph469" Nov 03 00:55:32.700612 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:55:32.700568 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5ca6f6a0-78ff-4b3f-ad1f-b94428b9178e-istio-token\") pod \"e2e-worker-w0-2f9a3a-openshift-default-7f6f74cf97-ph469\" (UID: \"5ca6f6a0-78ff-4b3f-ad1f-b94428b9178e\") " pod="openshift-ingress/e2e-worker-w0-2f9a3a-openshift-default-7f6f74cf97-ph469" Nov 03 00:55:32.700856 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:55:32.700822 4939 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-2f9a3a-openshift-default-7f6f74cf97-ph469"] Nov 03 00:55:32.701337 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:55:32.700967 4939 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-2f9a3a-openshift-default-7f6f74cf97-ckb7r" Nov 03 00:55:32.796337 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:55:32.796305 4939 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-2f9a3a-openshift-default-7f6f74cf97-ckb7r"] Nov 03 00:55:32.803047 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:55:32.801578 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5ca6f6a0-78ff-4b3f-ad1f-b94428b9178e-workload-socket\") pod \"e2e-worker-w0-2f9a3a-openshift-default-7f6f74cf97-ph469\" (UID: \"5ca6f6a0-78ff-4b3f-ad1f-b94428b9178e\") " pod="openshift-ingress/e2e-worker-w0-2f9a3a-openshift-default-7f6f74cf97-ph469" Nov 03 00:55:32.803047 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:55:32.801631 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5ca6f6a0-78ff-4b3f-ad1f-b94428b9178e-istiod-ca-cert\") pod \"e2e-worker-w0-2f9a3a-openshift-default-7f6f74cf97-ph469\" (UID: \"5ca6f6a0-78ff-4b3f-ad1f-b94428b9178e\") " pod="openshift-ingress/e2e-worker-w0-2f9a3a-openshift-default-7f6f74cf97-ph469" Nov 03 00:55:32.803047 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:55:32.801663 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ee090f53-4805-4a1a-a79f-450b8bc77079-credential-socket\") pod \"e2e-worker-w0-2f9a3a-openshift-default-7f6f74cf97-ckb7r\" (UID: \"ee090f53-4805-4a1a-a79f-450b8bc77079\") " pod="openshift-ingress/e2e-worker-w0-2f9a3a-openshift-default-7f6f74cf97-ckb7r" Nov 03 00:55:32.803047 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:55:32.801693 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ee090f53-4805-4a1a-a79f-450b8bc77079-istio-podinfo\") pod \"e2e-worker-w0-2f9a3a-openshift-default-7f6f74cf97-ckb7r\" (UID: \"ee090f53-4805-4a1a-a79f-450b8bc77079\") " pod="openshift-ingress/e2e-worker-w0-2f9a3a-openshift-default-7f6f74cf97-ckb7r" Nov 03 00:55:32.803047 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:55:32.801748 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ee090f53-4805-4a1a-a79f-450b8bc77079-workload-socket\") pod \"e2e-worker-w0-2f9a3a-openshift-default-7f6f74cf97-ckb7r\" (UID: \"ee090f53-4805-4a1a-a79f-450b8bc77079\") " pod="openshift-ingress/e2e-worker-w0-2f9a3a-openshift-default-7f6f74cf97-ckb7r" Nov 03 00:55:32.803047 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:55:32.801771 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ee090f53-4805-4a1a-a79f-450b8bc77079-istio-token\") pod \"e2e-worker-w0-2f9a3a-openshift-default-7f6f74cf97-ckb7r\" (UID: \"ee090f53-4805-4a1a-a79f-450b8bc77079\") " pod="openshift-ingress/e2e-worker-w0-2f9a3a-openshift-default-7f6f74cf97-ckb7r" Nov 03 00:55:32.803047 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:55:32.801801 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5ca6f6a0-78ff-4b3f-ad1f-b94428b9178e-workload-certs\") pod \"e2e-worker-w0-2f9a3a-openshift-default-7f6f74cf97-ph469\" (UID: \"5ca6f6a0-78ff-4b3f-ad1f-b94428b9178e\") " pod="openshift-ingress/e2e-worker-w0-2f9a3a-openshift-default-7f6f74cf97-ph469" Nov 03 00:55:32.803047 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:55:32.801843 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5ca6f6a0-78ff-4b3f-ad1f-b94428b9178e-istio-data\") pod \"e2e-worker-w0-2f9a3a-openshift-default-7f6f74cf97-ph469\" (UID: \"5ca6f6a0-78ff-4b3f-ad1f-b94428b9178e\") " pod="openshift-ingress/e2e-worker-w0-2f9a3a-openshift-default-7f6f74cf97-ph469" Nov 03 00:55:32.803047 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:55:32.801878 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9kjbw\" (UniqueName: \"kubernetes.io/projected/5ca6f6a0-78ff-4b3f-ad1f-b94428b9178e-kube-api-access-9kjbw\") pod \"e2e-worker-w0-2f9a3a-openshift-default-7f6f74cf97-ph469\" (UID: \"5ca6f6a0-78ff-4b3f-ad1f-b94428b9178e\") " pod="openshift-ingress/e2e-worker-w0-2f9a3a-openshift-default-7f6f74cf97-ph469" Nov 03 00:55:32.803047 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:55:32.801938 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5ca6f6a0-78ff-4b3f-ad1f-b94428b9178e-credential-socket\") pod \"e2e-worker-w0-2f9a3a-openshift-default-7f6f74cf97-ph469\" (UID: \"5ca6f6a0-78ff-4b3f-ad1f-b94428b9178e\") " pod="openshift-ingress/e2e-worker-w0-2f9a3a-openshift-default-7f6f74cf97-ph469" Nov 03 00:55:32.803047 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:55:32.801995 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jkrsh\" (UniqueName: \"kubernetes.io/projected/ee090f53-4805-4a1a-a79f-450b8bc77079-kube-api-access-jkrsh\") pod \"e2e-worker-w0-2f9a3a-openshift-default-7f6f74cf97-ckb7r\" (UID: \"ee090f53-4805-4a1a-a79f-450b8bc77079\") " pod="openshift-ingress/e2e-worker-w0-2f9a3a-openshift-default-7f6f74cf97-ckb7r" Nov 03 00:55:32.803047 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:55:32.802028 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ee090f53-4805-4a1a-a79f-450b8bc77079-istio-data\") pod \"e2e-worker-w0-2f9a3a-openshift-default-7f6f74cf97-ckb7r\" (UID: \"ee090f53-4805-4a1a-a79f-450b8bc77079\") " pod="openshift-ingress/e2e-worker-w0-2f9a3a-openshift-default-7f6f74cf97-ckb7r" Nov 03 00:55:32.803047 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:55:32.802103 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ee090f53-4805-4a1a-a79f-450b8bc77079-istio-envoy\") pod \"e2e-worker-w0-2f9a3a-openshift-default-7f6f74cf97-ckb7r\" (UID: \"ee090f53-4805-4a1a-a79f-450b8bc77079\") " pod="openshift-ingress/e2e-worker-w0-2f9a3a-openshift-default-7f6f74cf97-ckb7r" Nov 03 00:55:32.803047 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:55:32.802130 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5ca6f6a0-78ff-4b3f-ad1f-b94428b9178e-istio-token\") pod \"e2e-worker-w0-2f9a3a-openshift-default-7f6f74cf97-ph469\" (UID: \"5ca6f6a0-78ff-4b3f-ad1f-b94428b9178e\") " pod="openshift-ingress/e2e-worker-w0-2f9a3a-openshift-default-7f6f74cf97-ph469" Nov 03 00:55:32.803047 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:55:32.802156 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5ca6f6a0-78ff-4b3f-ad1f-b94428b9178e-istio-podinfo\") pod \"e2e-worker-w0-2f9a3a-openshift-default-7f6f74cf97-ph469\" (UID: \"5ca6f6a0-78ff-4b3f-ad1f-b94428b9178e\") " pod="openshift-ingress/e2e-worker-w0-2f9a3a-openshift-default-7f6f74cf97-ph469" Nov 03 00:55:32.803047 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:55:32.802206 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ee090f53-4805-4a1a-a79f-450b8bc77079-workload-certs\") pod \"e2e-worker-w0-2f9a3a-openshift-default-7f6f74cf97-ckb7r\" (UID: \"ee090f53-4805-4a1a-a79f-450b8bc77079\") " pod="openshift-ingress/e2e-worker-w0-2f9a3a-openshift-default-7f6f74cf97-ckb7r" Nov 03 00:55:32.803047 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:55:32.802248 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ee090f53-4805-4a1a-a79f-450b8bc77079-istiod-ca-cert\") pod \"e2e-worker-w0-2f9a3a-openshift-default-7f6f74cf97-ckb7r\" (UID: \"ee090f53-4805-4a1a-a79f-450b8bc77079\") " pod="openshift-ingress/e2e-worker-w0-2f9a3a-openshift-default-7f6f74cf97-ckb7r" Nov 03 00:55:32.803047 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:55:32.802285 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5ca6f6a0-78ff-4b3f-ad1f-b94428b9178e-istio-envoy\") pod \"e2e-worker-w0-2f9a3a-openshift-default-7f6f74cf97-ph469\" (UID: \"5ca6f6a0-78ff-4b3f-ad1f-b94428b9178e\") " pod="openshift-ingress/e2e-worker-w0-2f9a3a-openshift-default-7f6f74cf97-ph469" Nov 03 00:55:32.804789 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:55:32.804762 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5ca6f6a0-78ff-4b3f-ad1f-b94428b9178e-workload-socket\") pod \"e2e-worker-w0-2f9a3a-openshift-default-7f6f74cf97-ph469\" (UID: \"5ca6f6a0-78ff-4b3f-ad1f-b94428b9178e\") " pod="openshift-ingress/e2e-worker-w0-2f9a3a-openshift-default-7f6f74cf97-ph469" Nov 03 00:55:32.807449 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:55:32.805665 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5ca6f6a0-78ff-4b3f-ad1f-b94428b9178e-istiod-ca-cert\") pod \"e2e-worker-w0-2f9a3a-openshift-default-7f6f74cf97-ph469\" (UID: \"5ca6f6a0-78ff-4b3f-ad1f-b94428b9178e\") " pod="openshift-ingress/e2e-worker-w0-2f9a3a-openshift-default-7f6f74cf97-ph469" Nov 03 00:55:32.807633 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:55:32.806098 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5ca6f6a0-78ff-4b3f-ad1f-b94428b9178e-workload-certs\") pod \"e2e-worker-w0-2f9a3a-openshift-default-7f6f74cf97-ph469\" (UID: \"5ca6f6a0-78ff-4b3f-ad1f-b94428b9178e\") " pod="openshift-ingress/e2e-worker-w0-2f9a3a-openshift-default-7f6f74cf97-ph469" Nov 03 00:55:32.807777 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:55:32.806343 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5ca6f6a0-78ff-4b3f-ad1f-b94428b9178e-istio-data\") pod \"e2e-worker-w0-2f9a3a-openshift-default-7f6f74cf97-ph469\" (UID: \"5ca6f6a0-78ff-4b3f-ad1f-b94428b9178e\") " pod="openshift-ingress/e2e-worker-w0-2f9a3a-openshift-default-7f6f74cf97-ph469" Nov 03 00:55:32.807873 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:55:32.806636 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5ca6f6a0-78ff-4b3f-ad1f-b94428b9178e-credential-socket\") pod \"e2e-worker-w0-2f9a3a-openshift-default-7f6f74cf97-ph469\" (UID: \"5ca6f6a0-78ff-4b3f-ad1f-b94428b9178e\") " pod="openshift-ingress/e2e-worker-w0-2f9a3a-openshift-default-7f6f74cf97-ph469" Nov 03 00:55:32.852651 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:55:32.852609 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 00:55:32.854201 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:55:32.854117 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5ca6f6a0-78ff-4b3f-ad1f-b94428b9178e-istio-token\") pod \"e2e-worker-w0-2f9a3a-openshift-default-7f6f74cf97-ph469\" (UID: \"5ca6f6a0-78ff-4b3f-ad1f-b94428b9178e\") " pod="openshift-ingress/e2e-worker-w0-2f9a3a-openshift-default-7f6f74cf97-ph469" Nov 03 00:55:32.858796 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:55:32.858769 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9kjbw\" (UniqueName: \"kubernetes.io/projected/5ca6f6a0-78ff-4b3f-ad1f-b94428b9178e-kube-api-access-9kjbw\") pod \"e2e-worker-w0-2f9a3a-openshift-default-7f6f74cf97-ph469\" (UID: \"5ca6f6a0-78ff-4b3f-ad1f-b94428b9178e\") " pod="openshift-ingress/e2e-worker-w0-2f9a3a-openshift-default-7f6f74cf97-ph469" Nov 03 00:55:32.859297 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:55:32.859239 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5ca6f6a0-78ff-4b3f-ad1f-b94428b9178e-istio-envoy\") pod \"e2e-worker-w0-2f9a3a-openshift-default-7f6f74cf97-ph469\" (UID: \"5ca6f6a0-78ff-4b3f-ad1f-b94428b9178e\") " pod="openshift-ingress/e2e-worker-w0-2f9a3a-openshift-default-7f6f74cf97-ph469" Nov 03 00:55:32.859481 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:55:32.859462 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5ca6f6a0-78ff-4b3f-ad1f-b94428b9178e-istio-podinfo\") pod \"e2e-worker-w0-2f9a3a-openshift-default-7f6f74cf97-ph469\" (UID: \"5ca6f6a0-78ff-4b3f-ad1f-b94428b9178e\") " pod="openshift-ingress/e2e-worker-w0-2f9a3a-openshift-default-7f6f74cf97-ph469" Nov 03 00:55:32.903219 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:55:32.903180 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ee090f53-4805-4a1a-a79f-450b8bc77079-credential-socket\") pod \"e2e-worker-w0-2f9a3a-openshift-default-7f6f74cf97-ckb7r\" (UID: \"ee090f53-4805-4a1a-a79f-450b8bc77079\") " pod="openshift-ingress/e2e-worker-w0-2f9a3a-openshift-default-7f6f74cf97-ckb7r" Nov 03 00:55:32.903219 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:55:32.903231 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ee090f53-4805-4a1a-a79f-450b8bc77079-istio-podinfo\") pod \"e2e-worker-w0-2f9a3a-openshift-default-7f6f74cf97-ckb7r\" (UID: \"ee090f53-4805-4a1a-a79f-450b8bc77079\") " pod="openshift-ingress/e2e-worker-w0-2f9a3a-openshift-default-7f6f74cf97-ckb7r" Nov 03 00:55:32.903473 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:55:32.903403 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ee090f53-4805-4a1a-a79f-450b8bc77079-workload-socket\") pod \"e2e-worker-w0-2f9a3a-openshift-default-7f6f74cf97-ckb7r\" (UID: \"ee090f53-4805-4a1a-a79f-450b8bc77079\") " pod="openshift-ingress/e2e-worker-w0-2f9a3a-openshift-default-7f6f74cf97-ckb7r" Nov 03 00:55:32.903473 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:55:32.903438 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ee090f53-4805-4a1a-a79f-450b8bc77079-istio-token\") pod \"e2e-worker-w0-2f9a3a-openshift-default-7f6f74cf97-ckb7r\" (UID: \"ee090f53-4805-4a1a-a79f-450b8bc77079\") " pod="openshift-ingress/e2e-worker-w0-2f9a3a-openshift-default-7f6f74cf97-ckb7r" Nov 03 00:55:32.903901 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:55:32.903725 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jkrsh\" (UniqueName: \"kubernetes.io/projected/ee090f53-4805-4a1a-a79f-450b8bc77079-kube-api-access-jkrsh\") pod \"e2e-worker-w0-2f9a3a-openshift-default-7f6f74cf97-ckb7r\" (UID: \"ee090f53-4805-4a1a-a79f-450b8bc77079\") " pod="openshift-ingress/e2e-worker-w0-2f9a3a-openshift-default-7f6f74cf97-ckb7r" Nov 03 00:55:32.903901 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:55:32.903783 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ee090f53-4805-4a1a-a79f-450b8bc77079-istio-data\") pod \"e2e-worker-w0-2f9a3a-openshift-default-7f6f74cf97-ckb7r\" (UID: \"ee090f53-4805-4a1a-a79f-450b8bc77079\") " pod="openshift-ingress/e2e-worker-w0-2f9a3a-openshift-default-7f6f74cf97-ckb7r" Nov 03 00:55:32.903901 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:55:32.903828 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ee090f53-4805-4a1a-a79f-450b8bc77079-credential-socket\") pod \"e2e-worker-w0-2f9a3a-openshift-default-7f6f74cf97-ckb7r\" (UID: \"ee090f53-4805-4a1a-a79f-450b8bc77079\") " pod="openshift-ingress/e2e-worker-w0-2f9a3a-openshift-default-7f6f74cf97-ckb7r" Nov 03 00:55:32.903901 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:55:32.903845 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ee090f53-4805-4a1a-a79f-450b8bc77079-workload-socket\") pod \"e2e-worker-w0-2f9a3a-openshift-default-7f6f74cf97-ckb7r\" (UID: \"ee090f53-4805-4a1a-a79f-450b8bc77079\") " pod="openshift-ingress/e2e-worker-w0-2f9a3a-openshift-default-7f6f74cf97-ckb7r" Nov 03 00:55:32.905092 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:55:32.905064 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ee090f53-4805-4a1a-a79f-450b8bc77079-istio-envoy\") pod \"e2e-worker-w0-2f9a3a-openshift-default-7f6f74cf97-ckb7r\" (UID: \"ee090f53-4805-4a1a-a79f-450b8bc77079\") " pod="openshift-ingress/e2e-worker-w0-2f9a3a-openshift-default-7f6f74cf97-ckb7r" Nov 03 00:55:32.905807 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:55:32.905282 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ee090f53-4805-4a1a-a79f-450b8bc77079-workload-certs\") pod \"e2e-worker-w0-2f9a3a-openshift-default-7f6f74cf97-ckb7r\" (UID: \"ee090f53-4805-4a1a-a79f-450b8bc77079\") " pod="openshift-ingress/e2e-worker-w0-2f9a3a-openshift-default-7f6f74cf97-ckb7r" Nov 03 00:55:32.905807 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:55:32.905331 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ee090f53-4805-4a1a-a79f-450b8bc77079-istiod-ca-cert\") pod \"e2e-worker-w0-2f9a3a-openshift-default-7f6f74cf97-ckb7r\" (UID: \"ee090f53-4805-4a1a-a79f-450b8bc77079\") " pod="openshift-ingress/e2e-worker-w0-2f9a3a-openshift-default-7f6f74cf97-ckb7r" Nov 03 00:55:32.905807 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:55:32.905512 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ee090f53-4805-4a1a-a79f-450b8bc77079-istio-data\") pod \"e2e-worker-w0-2f9a3a-openshift-default-7f6f74cf97-ckb7r\" (UID: \"ee090f53-4805-4a1a-a79f-450b8bc77079\") " pod="openshift-ingress/e2e-worker-w0-2f9a3a-openshift-default-7f6f74cf97-ckb7r" Nov 03 00:55:32.906077 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:55:32.906056 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ee090f53-4805-4a1a-a79f-450b8bc77079-istiod-ca-cert\") pod \"e2e-worker-w0-2f9a3a-openshift-default-7f6f74cf97-ckb7r\" (UID: \"ee090f53-4805-4a1a-a79f-450b8bc77079\") " pod="openshift-ingress/e2e-worker-w0-2f9a3a-openshift-default-7f6f74cf97-ckb7r" Nov 03 00:55:32.906296 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:55:32.906268 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ee090f53-4805-4a1a-a79f-450b8bc77079-istio-podinfo\") pod \"e2e-worker-w0-2f9a3a-openshift-default-7f6f74cf97-ckb7r\" (UID: \"ee090f53-4805-4a1a-a79f-450b8bc77079\") " pod="openshift-ingress/e2e-worker-w0-2f9a3a-openshift-default-7f6f74cf97-ckb7r" Nov 03 00:55:32.906609 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:55:32.906585 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ee090f53-4805-4a1a-a79f-450b8bc77079-workload-certs\") pod \"e2e-worker-w0-2f9a3a-openshift-default-7f6f74cf97-ckb7r\" (UID: \"ee090f53-4805-4a1a-a79f-450b8bc77079\") " pod="openshift-ingress/e2e-worker-w0-2f9a3a-openshift-default-7f6f74cf97-ckb7r" Nov 03 00:55:32.908339 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:55:32.908318 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ee090f53-4805-4a1a-a79f-450b8bc77079-istio-envoy\") pod \"e2e-worker-w0-2f9a3a-openshift-default-7f6f74cf97-ckb7r\" (UID: \"ee090f53-4805-4a1a-a79f-450b8bc77079\") " pod="openshift-ingress/e2e-worker-w0-2f9a3a-openshift-default-7f6f74cf97-ckb7r" Nov 03 00:55:32.917346 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:55:32.917298 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ee090f53-4805-4a1a-a79f-450b8bc77079-istio-token\") pod \"e2e-worker-w0-2f9a3a-openshift-default-7f6f74cf97-ckb7r\" (UID: \"ee090f53-4805-4a1a-a79f-450b8bc77079\") " pod="openshift-ingress/e2e-worker-w0-2f9a3a-openshift-default-7f6f74cf97-ckb7r" Nov 03 00:55:32.919120 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:55:32.919095 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jkrsh\" (UniqueName: \"kubernetes.io/projected/ee090f53-4805-4a1a-a79f-450b8bc77079-kube-api-access-jkrsh\") pod \"e2e-worker-w0-2f9a3a-openshift-default-7f6f74cf97-ckb7r\" (UID: \"ee090f53-4805-4a1a-a79f-450b8bc77079\") " pod="openshift-ingress/e2e-worker-w0-2f9a3a-openshift-default-7f6f74cf97-ckb7r" Nov 03 00:55:33.013702 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:55:33.013664 4939 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-2f9a3a-openshift-default-7f6f74cf97-ph469" Nov 03 00:55:33.058430 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:55:33.058393 4939 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-2f9a3a-openshift-default-7f6f74cf97-ckb7r" Nov 03 00:55:33.299764 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:55:33.299738 4939 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-2f9a3a-openshift-default-7f6f74cf97-ph469"] Nov 03 00:55:33.302382 ip-10-0-140-107 kubenswrapper[4939]: W1103 00:55:33.302354 4939 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5ca6f6a0_78ff_4b3f_ad1f_b94428b9178e.slice/crio-1a8e2b89707096775e4ed562b80204df0cefe937399955353e8fdbf96eca9977 WatchSource:0}: Error finding container 1a8e2b89707096775e4ed562b80204df0cefe937399955353e8fdbf96eca9977: Status 404 returned error can't find the container with id 1a8e2b89707096775e4ed562b80204df0cefe937399955353e8fdbf96eca9977 Nov 03 00:55:33.305087 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:55:33.305055 4939 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 00:55:33.305175 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:55:33.305118 4939 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 00:55:33.305175 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:55:33.305147 4939 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 00:55:33.371437 ip-10-0-140-107 kubenswrapper[4939]: W1103 00:55:33.371408 4939 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podee090f53_4805_4a1a_a79f_450b8bc77079.slice/crio-1cd68f6d15017a3b45ce57f376a034757dede003cd515b06b803622d33645b88 WatchSource:0}: Error finding container 1cd68f6d15017a3b45ce57f376a034757dede003cd515b06b803622d33645b88: Status 404 returned error can't find the container with id 1cd68f6d15017a3b45ce57f376a034757dede003cd515b06b803622d33645b88 Nov 03 00:55:33.373542 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:55:33.373506 4939 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-2f9a3a-openshift-default-7f6f74cf97-ckb7r"] Nov 03 00:55:33.374262 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:55:33.374231 4939 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 00:55:33.374328 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:55:33.374296 4939 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 00:55:33.374360 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:55:33.374324 4939 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 00:55:33.676988 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:55:33.675894 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 00:55:34.222558 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:55:34.222522 4939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-2f9a3a-openshift-default-7f6f74cf97-ph469" event={"ID":"5ca6f6a0-78ff-4b3f-ad1f-b94428b9178e","Type":"ContainerStarted","Data":"19473538a2c75199337c996b9d3339e90e813285dc8c6eeb33c78557b70e09fd"} Nov 03 00:55:34.223167 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:55:34.223141 4939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-2f9a3a-openshift-default-7f6f74cf97-ph469" event={"ID":"5ca6f6a0-78ff-4b3f-ad1f-b94428b9178e","Type":"ContainerStarted","Data":"1a8e2b89707096775e4ed562b80204df0cefe937399955353e8fdbf96eca9977"} Nov 03 00:55:34.225683 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:55:34.225645 4939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-2f9a3a-openshift-default-7f6f74cf97-ckb7r" event={"ID":"ee090f53-4805-4a1a-a79f-450b8bc77079","Type":"ContainerStarted","Data":"9ce5ab3f82c59a07fcab8e231933c24e8c586c6194d2db303bcb6d65033bf2f8"} Nov 03 00:55:34.225785 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:55:34.225706 4939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-2f9a3a-openshift-default-7f6f74cf97-ckb7r" event={"ID":"ee090f53-4805-4a1a-a79f-450b8bc77079","Type":"ContainerStarted","Data":"1cd68f6d15017a3b45ce57f376a034757dede003cd515b06b803622d33645b88"} Nov 03 00:55:34.259934 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:55:34.246004 4939 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-2f9a3a-openshift-default-7f6f74cf97-ph469" podStartSLOduration=2.245983024 podStartE2EDuration="2.245983024s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:55:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:55:34.2425157 +0000 UTC m=+882.506850645" watchObservedRunningTime="2026-11-03 00:55:34.245983024 +0000 UTC m=+882.510317970" Nov 03 00:55:34.302870 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:55:34.302793 4939 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-2f9a3a-openshift-default-7f6f74cf97-ckb7r" podStartSLOduration=2.302772406 podStartE2EDuration="2.302772406s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:55:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:55:34.287986239 +0000 UTC m=+882.552321186" watchObservedRunningTime="2026-11-03 00:55:34.302772406 +0000 UTC m=+882.567107331" Nov 03 00:55:34.529818 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:55:34.529731 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 00:55:35.010681 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:55:35.010644 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 00:55:35.013971 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:55:35.013944 4939 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-2f9a3a-openshift-default-7f6f74cf97-ph469" Nov 03 00:55:35.016013 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:55:35.015979 4939 patch_prober.go:29] interesting pod/e2e-worker-w0-2f9a3a-openshift-default-7f6f74cf97-ph469 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.89:15021/healthz/ready\": dial tcp 10.132.0.89:15021: connect: connection refused" start-of-body= Nov 03 00:55:35.016127 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:55:35.016052 4939 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-2f9a3a-openshift-default-7f6f74cf97-ph469" podUID="5ca6f6a0-78ff-4b3f-ad1f-b94428b9178e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.89:15021/healthz/ready\": dial tcp 10.132.0.89:15021: connect: connection refused" Nov 03 00:55:35.065929 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:55:35.061609 4939 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-2f9a3a-openshift-default-7f6f74cf97-ckb7r" Nov 03 00:55:35.065929 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:55:35.061743 4939 patch_prober.go:29] interesting pod/e2e-worker-w0-2f9a3a-openshift-default-7f6f74cf97-ckb7r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.90:15021/healthz/ready\": dial tcp 10.132.0.90:15021: connect: connection refused" start-of-body= Nov 03 00:55:35.065929 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:55:35.061804 4939 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-2f9a3a-openshift-default-7f6f74cf97-ckb7r" podUID="ee090f53-4805-4a1a-a79f-450b8bc77079" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.90:15021/healthz/ready\": dial tcp 10.132.0.90:15021: connect: connection refused" Nov 03 00:55:35.638812 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:55:35.638775 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 00:55:36.014523 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:55:36.014434 4939 patch_prober.go:29] interesting pod/e2e-worker-w0-2f9a3a-openshift-default-7f6f74cf97-ph469 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.89:15021/healthz/ready\": dial tcp 10.132.0.89:15021: connect: connection refused" start-of-body= Nov 03 00:55:36.014690 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:55:36.014525 4939 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-2f9a3a-openshift-default-7f6f74cf97-ph469" podUID="5ca6f6a0-78ff-4b3f-ad1f-b94428b9178e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.89:15021/healthz/ready\": dial tcp 10.132.0.89:15021: connect: connection refused" Nov 03 00:55:36.065935 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:55:36.060462 4939 patch_prober.go:29] interesting pod/e2e-worker-w0-2f9a3a-openshift-default-7f6f74cf97-ckb7r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.90:15021/healthz/ready\": dial tcp 10.132.0.90:15021: connect: connection refused" start-of-body= Nov 03 00:55:36.065935 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:55:36.060539 4939 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-2f9a3a-openshift-default-7f6f74cf97-ckb7r" podUID="ee090f53-4805-4a1a-a79f-450b8bc77079" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.90:15021/healthz/ready\": dial tcp 10.132.0.90:15021: connect: connection refused" Nov 03 00:55:36.510783 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:55:36.510751 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 00:55:37.014643 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:55:37.014608 4939 patch_prober.go:29] interesting pod/e2e-worker-w0-2f9a3a-openshift-default-7f6f74cf97-ph469 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.89:15021/healthz/ready\": dial tcp 10.132.0.89:15021: connect: connection refused" start-of-body= Nov 03 00:55:37.015087 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:55:37.014675 4939 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-2f9a3a-openshift-default-7f6f74cf97-ph469" podUID="5ca6f6a0-78ff-4b3f-ad1f-b94428b9178e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.89:15021/healthz/ready\": dial tcp 10.132.0.89:15021: connect: connection refused" Nov 03 00:55:37.063932 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:55:37.059313 4939 patch_prober.go:29] interesting pod/e2e-worker-w0-2f9a3a-openshift-default-7f6f74cf97-ckb7r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.90:15021/healthz/ready\": dial tcp 10.132.0.90:15021: connect: connection refused" start-of-body= Nov 03 00:55:37.063932 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:55:37.059392 4939 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-2f9a3a-openshift-default-7f6f74cf97-ckb7r" podUID="ee090f53-4805-4a1a-a79f-450b8bc77079" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.90:15021/healthz/ready\": dial tcp 10.132.0.90:15021: connect: connection refused" Nov 03 00:55:37.216752 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:55:37.216717 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 00:55:37.487813 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:55:37.487777 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 00:55:37.516156 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:55:37.516123 4939 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-mt-dd03e395/e2e-lifecycle-model-dd03e395-kserve-78979759df-b8v9c"] Nov 03 00:55:37.520853 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:55:37.520824 4939 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-mt-dd03e395/e2e-lifecycle-model-dd03e395-kserve-78979759df-b8v9c"] Nov 03 00:55:37.521019 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:55:37.520959 4939 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-dd03e395/e2e-lifecycle-model-dd03e395-kserve-78979759df-b8v9c" Nov 03 00:55:37.527069 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:55:37.526894 4939 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-mt-dd03e395\"/\"openshift-service-ca.crt\"" Nov 03 00:55:37.527069 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:55:37.527060 4939 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-mt-dd03e395\"/\"kube-root-ca.crt\"" Nov 03 00:55:37.527294 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:55:37.527147 4939 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-mt-dd03e395\"/\"e2e-lifecycle-model-dd03e395-kserve-self-signed-certs\"" Nov 03 00:55:37.527294 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:55:37.527210 4939 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-mt-dd03e395\"/\"default-dockercfg-cdw68\"" Nov 03 00:55:37.601850 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:55:37.601788 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 00:55:37.618392 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:55:37.618358 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e4803d7d-4603-4b14-950d-018f88294285-tls-certs\") pod \"e2e-lifecycle-model-dd03e395-kserve-78979759df-b8v9c\" (UID: \"e4803d7d-4603-4b14-950d-018f88294285\") " pod="ai-tenant-e2e-mt-dd03e395/e2e-lifecycle-model-dd03e395-kserve-78979759df-b8v9c" Nov 03 00:55:37.618535 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:55:37.618447 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e4803d7d-4603-4b14-950d-018f88294285-dshm\") pod \"e2e-lifecycle-model-dd03e395-kserve-78979759df-b8v9c\" (UID: \"e4803d7d-4603-4b14-950d-018f88294285\") " pod="ai-tenant-e2e-mt-dd03e395/e2e-lifecycle-model-dd03e395-kserve-78979759df-b8v9c" Nov 03 00:55:37.618535 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:55:37.618469 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e4803d7d-4603-4b14-950d-018f88294285-tmp-dir\") pod \"e2e-lifecycle-model-dd03e395-kserve-78979759df-b8v9c\" (UID: \"e4803d7d-4603-4b14-950d-018f88294285\") " pod="ai-tenant-e2e-mt-dd03e395/e2e-lifecycle-model-dd03e395-kserve-78979759df-b8v9c" Nov 03 00:55:37.618659 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:55:37.618644 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e4803d7d-4603-4b14-950d-018f88294285-home\") pod \"e2e-lifecycle-model-dd03e395-kserve-78979759df-b8v9c\" (UID: \"e4803d7d-4603-4b14-950d-018f88294285\") " pod="ai-tenant-e2e-mt-dd03e395/e2e-lifecycle-model-dd03e395-kserve-78979759df-b8v9c" Nov 03 00:55:37.618693 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:55:37.618675 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e4803d7d-4603-4b14-950d-018f88294285-model-cache\") pod \"e2e-lifecycle-model-dd03e395-kserve-78979759df-b8v9c\" (UID: \"e4803d7d-4603-4b14-950d-018f88294285\") " pod="ai-tenant-e2e-mt-dd03e395/e2e-lifecycle-model-dd03e395-kserve-78979759df-b8v9c" Nov 03 00:55:37.618726 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:55:37.618702 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qg4mj\" (UniqueName: \"kubernetes.io/projected/e4803d7d-4603-4b14-950d-018f88294285-kube-api-access-qg4mj\") pod \"e2e-lifecycle-model-dd03e395-kserve-78979759df-b8v9c\" (UID: \"e4803d7d-4603-4b14-950d-018f88294285\") " pod="ai-tenant-e2e-mt-dd03e395/e2e-lifecycle-model-dd03e395-kserve-78979759df-b8v9c" Nov 03 00:55:37.722011 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:55:37.721970 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e4803d7d-4603-4b14-950d-018f88294285-tls-certs\") pod \"e2e-lifecycle-model-dd03e395-kserve-78979759df-b8v9c\" (UID: \"e4803d7d-4603-4b14-950d-018f88294285\") " pod="ai-tenant-e2e-mt-dd03e395/e2e-lifecycle-model-dd03e395-kserve-78979759df-b8v9c" Nov 03 00:55:37.722329 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:55:37.722310 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e4803d7d-4603-4b14-950d-018f88294285-dshm\") pod \"e2e-lifecycle-model-dd03e395-kserve-78979759df-b8v9c\" (UID: \"e4803d7d-4603-4b14-950d-018f88294285\") " pod="ai-tenant-e2e-mt-dd03e395/e2e-lifecycle-model-dd03e395-kserve-78979759df-b8v9c" Nov 03 00:55:37.722517 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:55:37.722500 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e4803d7d-4603-4b14-950d-018f88294285-tmp-dir\") pod \"e2e-lifecycle-model-dd03e395-kserve-78979759df-b8v9c\" (UID: \"e4803d7d-4603-4b14-950d-018f88294285\") " pod="ai-tenant-e2e-mt-dd03e395/e2e-lifecycle-model-dd03e395-kserve-78979759df-b8v9c" Nov 03 00:55:37.722769 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:55:37.722751 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e4803d7d-4603-4b14-950d-018f88294285-home\") pod \"e2e-lifecycle-model-dd03e395-kserve-78979759df-b8v9c\" (UID: \"e4803d7d-4603-4b14-950d-018f88294285\") " pod="ai-tenant-e2e-mt-dd03e395/e2e-lifecycle-model-dd03e395-kserve-78979759df-b8v9c" Nov 03 00:55:37.722878 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:55:37.722864 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e4803d7d-4603-4b14-950d-018f88294285-model-cache\") pod \"e2e-lifecycle-model-dd03e395-kserve-78979759df-b8v9c\" (UID: \"e4803d7d-4603-4b14-950d-018f88294285\") " pod="ai-tenant-e2e-mt-dd03e395/e2e-lifecycle-model-dd03e395-kserve-78979759df-b8v9c" Nov 03 00:55:37.723019 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:55:37.723003 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qg4mj\" (UniqueName: \"kubernetes.io/projected/e4803d7d-4603-4b14-950d-018f88294285-kube-api-access-qg4mj\") pod \"e2e-lifecycle-model-dd03e395-kserve-78979759df-b8v9c\" (UID: \"e4803d7d-4603-4b14-950d-018f88294285\") " pod="ai-tenant-e2e-mt-dd03e395/e2e-lifecycle-model-dd03e395-kserve-78979759df-b8v9c" Nov 03 00:55:37.741984 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:55:37.726560 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e4803d7d-4603-4b14-950d-018f88294285-home\") pod \"e2e-lifecycle-model-dd03e395-kserve-78979759df-b8v9c\" (UID: \"e4803d7d-4603-4b14-950d-018f88294285\") " pod="ai-tenant-e2e-mt-dd03e395/e2e-lifecycle-model-dd03e395-kserve-78979759df-b8v9c" Nov 03 00:55:37.742543 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:55:37.742513 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e4803d7d-4603-4b14-950d-018f88294285-tmp-dir\") pod \"e2e-lifecycle-model-dd03e395-kserve-78979759df-b8v9c\" (UID: \"e4803d7d-4603-4b14-950d-018f88294285\") " pod="ai-tenant-e2e-mt-dd03e395/e2e-lifecycle-model-dd03e395-kserve-78979759df-b8v9c" Nov 03 00:55:37.743054 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:55:37.742939 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e4803d7d-4603-4b14-950d-018f88294285-model-cache\") pod \"e2e-lifecycle-model-dd03e395-kserve-78979759df-b8v9c\" (UID: \"e4803d7d-4603-4b14-950d-018f88294285\") " pod="ai-tenant-e2e-mt-dd03e395/e2e-lifecycle-model-dd03e395-kserve-78979759df-b8v9c" Nov 03 00:55:37.746587 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:55:37.746561 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e4803d7d-4603-4b14-950d-018f88294285-tls-certs\") pod \"e2e-lifecycle-model-dd03e395-kserve-78979759df-b8v9c\" (UID: \"e4803d7d-4603-4b14-950d-018f88294285\") " pod="ai-tenant-e2e-mt-dd03e395/e2e-lifecycle-model-dd03e395-kserve-78979759df-b8v9c" Nov 03 00:55:37.763128 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:55:37.757100 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e4803d7d-4603-4b14-950d-018f88294285-dshm\") pod \"e2e-lifecycle-model-dd03e395-kserve-78979759df-b8v9c\" (UID: \"e4803d7d-4603-4b14-950d-018f88294285\") " pod="ai-tenant-e2e-mt-dd03e395/e2e-lifecycle-model-dd03e395-kserve-78979759df-b8v9c" Nov 03 00:55:37.763128 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:55:37.759040 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qg4mj\" (UniqueName: \"kubernetes.io/projected/e4803d7d-4603-4b14-950d-018f88294285-kube-api-access-qg4mj\") pod \"e2e-lifecycle-model-dd03e395-kserve-78979759df-b8v9c\" (UID: \"e4803d7d-4603-4b14-950d-018f88294285\") " pod="ai-tenant-e2e-mt-dd03e395/e2e-lifecycle-model-dd03e395-kserve-78979759df-b8v9c" Nov 03 00:55:37.882699 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:55:37.880124 4939 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-dd03e395/e2e-lifecycle-model-dd03e395-kserve-78979759df-b8v9c" Nov 03 00:55:38.015128 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:55:38.014573 4939 patch_prober.go:29] interesting pod/e2e-worker-w0-2f9a3a-openshift-default-7f6f74cf97-ph469 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.89:15021/healthz/ready\": dial tcp 10.132.0.89:15021: connect: connection refused" start-of-body= Nov 03 00:55:38.015128 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:55:38.014652 4939 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-2f9a3a-openshift-default-7f6f74cf97-ph469" podUID="5ca6f6a0-78ff-4b3f-ad1f-b94428b9178e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.89:15021/healthz/ready\": dial tcp 10.132.0.89:15021: connect: connection refused" Nov 03 00:55:38.068813 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:55:38.068328 4939 patch_prober.go:29] interesting pod/e2e-worker-w0-2f9a3a-openshift-default-7f6f74cf97-ckb7r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.90:15021/healthz/ready\": dial tcp 10.132.0.90:15021: connect: connection refused" start-of-body= Nov 03 00:55:38.068813 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:55:38.068395 4939 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-2f9a3a-openshift-default-7f6f74cf97-ckb7r" podUID="ee090f53-4805-4a1a-a79f-450b8bc77079" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.90:15021/healthz/ready\": dial tcp 10.132.0.90:15021: connect: connection refused" Nov 03 00:55:38.330963 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:55:38.330939 4939 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-mt-dd03e395/e2e-lifecycle-model-dd03e395-kserve-78979759df-b8v9c"] Nov 03 00:55:38.332672 ip-10-0-140-107 kubenswrapper[4939]: W1103 00:55:38.332636 4939 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pode4803d7d_4603_4b14_950d_018f88294285.slice/crio-fb22740ea0e6dae41c9ee796a703215fcdd325bec32f365ebd8c24a6a399c2aa WatchSource:0}: Error finding container fb22740ea0e6dae41c9ee796a703215fcdd325bec32f365ebd8c24a6a399c2aa: Status 404 returned error can't find the container with id fb22740ea0e6dae41c9ee796a703215fcdd325bec32f365ebd8c24a6a399c2aa Nov 03 00:55:38.505374 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:55:38.505325 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 00:55:39.014626 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:55:39.014592 4939 patch_prober.go:29] interesting pod/e2e-worker-w0-2f9a3a-openshift-default-7f6f74cf97-ph469 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.89:15021/healthz/ready\": dial tcp 10.132.0.89:15021: connect: connection refused" start-of-body= Nov 03 00:55:39.014626 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:55:39.014643 4939 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-2f9a3a-openshift-default-7f6f74cf97-ph469" podUID="5ca6f6a0-78ff-4b3f-ad1f-b94428b9178e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.89:15021/healthz/ready\": dial tcp 10.132.0.89:15021: connect: connection refused" Nov 03 00:55:39.087934 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:55:39.087040 4939 patch_prober.go:29] interesting pod/e2e-worker-w0-2f9a3a-openshift-default-7f6f74cf97-ckb7r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.90:15021/healthz/ready\": dial tcp 10.132.0.90:15021: connect: connection refused" start-of-body= Nov 03 00:55:39.087934 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:55:39.087105 4939 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-2f9a3a-openshift-default-7f6f74cf97-ckb7r" podUID="ee090f53-4805-4a1a-a79f-450b8bc77079" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.90:15021/healthz/ready\": dial tcp 10.132.0.90:15021: connect: connection refused" Nov 03 00:55:39.321093 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:55:39.320971 4939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-dd03e395/e2e-lifecycle-model-dd03e395-kserve-78979759df-b8v9c" event={"ID":"e4803d7d-4603-4b14-950d-018f88294285","Type":"ContainerStarted","Data":"80328c9985d797195afba46b194886bdb01d5ba00303ded6defced0175dcf411"} Nov 03 00:55:39.321093 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:55:39.321047 4939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-dd03e395/e2e-lifecycle-model-dd03e395-kserve-78979759df-b8v9c" event={"ID":"e4803d7d-4603-4b14-950d-018f88294285","Type":"ContainerStarted","Data":"fb22740ea0e6dae41c9ee796a703215fcdd325bec32f365ebd8c24a6a399c2aa"} Nov 03 00:55:39.341737 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:55:39.341666 4939 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-mt-dd03e395/e2e-lifecycle-model-dd03e395-kserve-78979759df-b8v9c" podStartSLOduration=2.341647504 podStartE2EDuration="2.341647504s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:55:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:55:39.338010196 +0000 UTC m=+887.602345141" watchObservedRunningTime="2026-11-03 00:55:39.341647504 +0000 UTC m=+887.605982450" Nov 03 00:55:39.504763 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:55:39.504727 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 00:55:39.881693 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:55:39.881656 4939 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-mt-dd03e395/e2e-lifecycle-model-dd03e395-kserve-78979759df-b8v9c" Nov 03 00:55:40.014506 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:55:40.014469 4939 patch_prober.go:29] interesting pod/e2e-worker-w0-2f9a3a-openshift-default-7f6f74cf97-ph469 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.89:15021/healthz/ready\": dial tcp 10.132.0.89:15021: connect: connection refused" start-of-body= Nov 03 00:55:40.014694 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:55:40.014526 4939 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-2f9a3a-openshift-default-7f6f74cf97-ph469" podUID="5ca6f6a0-78ff-4b3f-ad1f-b94428b9178e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.89:15021/healthz/ready\": dial tcp 10.132.0.89:15021: connect: connection refused" Nov 03 00:55:40.058837 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:55:40.058796 4939 patch_prober.go:29] interesting pod/e2e-worker-w0-2f9a3a-openshift-default-7f6f74cf97-ckb7r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.90:15021/healthz/ready\": dial tcp 10.132.0.90:15021: connect: connection refused" start-of-body= Nov 03 00:55:40.059023 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:55:40.058863 4939 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-2f9a3a-openshift-default-7f6f74cf97-ckb7r" podUID="ee090f53-4805-4a1a-a79f-450b8bc77079" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.90:15021/healthz/ready\": dial tcp 10.132.0.90:15021: connect: connection refused" Nov 03 00:55:40.286398 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:55:40.286301 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 00:55:41.072045 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:55:41.072004 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 00:55:41.290991 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:55:41.290962 4939 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-2f9a3a-openshift-default-7f6f74cf97-ph469" Nov 03 00:55:41.291920 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:55:41.291876 4939 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-2f9a3a-openshift-default-7f6f74cf97-ph469" Nov 03 00:55:41.292798 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:55:41.292782 4939 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-2f9a3a-openshift-default-7f6f74cf97-ph469" Nov 03 00:55:41.306223 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:55:41.306184 4939 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-2f9a3a-openshift-default-7f6f74cf97-ckb7r" Nov 03 00:55:41.307468 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:55:41.307443 4939 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-2f9a3a-openshift-default-7f6f74cf97-ckb7r" Nov 03 00:55:41.308245 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:55:41.308219 4939 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-2f9a3a-openshift-default-7f6f74cf97-ckb7r" Nov 03 00:55:41.886286 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:55:41.886248 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 00:55:42.249803 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:55:42.249721 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 00:55:42.584728 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:55:42.584697 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 00:55:42.895381 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:55:42.895302 4939 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-4pw7v"] Nov 03 00:55:43.437975 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:55:43.437940 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 00:55:43.561932 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:55:43.558027 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 00:55:44.882442 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:55:44.882407 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 00:55:45.280141 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:55:45.280024 4939 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-88bbf7/distinct-2-w1-88bbf7-kserve-5457cdcdbd-l6tfd" podUID="a397931b-faa9-4c2e-8290-491350546663" containerName="main" probeResult="failure" output="Get \"https://10.132.0.85:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 00:55:45.353056 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:55:45.353014 4939 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-2f9a3a/simulator-w0-2f9a3a-kserve-9596bbb8f-2trgz" podUID="207e2207-fb6c-4076-95df-72e95671c261" containerName="main" probeResult="failure" output="Get \"https://10.132.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 00:55:45.406058 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:55:45.406004 4939 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-88bbf7/distinct-2-w1-88bbf7-kserve-5457cdcdbd-l6tfd" podUID="a397931b-faa9-4c2e-8290-491350546663" containerName="main" probeResult="failure" output="Get \"https://10.132.0.85:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 00:55:45.500052 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:55:45.500006 4939 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-83755a/distinct-2-w3-83755a-kserve-6cf66b995d-2l47b" podUID="c3b05e88-5d5b-4d03-b28d-832f22e3c868" containerName="main" probeResult="failure" output="Get \"https://10.132.0.72:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 00:55:45.776129 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:55:45.776068 4939 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-f7680d/distinct-w2-f7680d-kserve-756f874b98-qnpv8" podUID="836753fc-f9ef-4f61-b3fe-e5b92b7ee27b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.70:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 00:55:45.825542 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:55:45.825496 4939 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-88bbf7/simulator-w1-88bbf7-kserve-5fbb669f8f-42hgd" podUID="83b38883-2e7a-444d-aa68-52be4a25ae66" containerName="main" probeResult="failure" output="Get \"https://10.132.0.62:8000/ready\": context deadline exceeded" Nov 03 00:55:45.937932 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:55:45.937781 4939 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-2f9a3a/distinct-w0-2f9a3a-kserve-555565cfcf-jtcr9" podUID="ac0ece06-8718-463f-915f-5315988d9711" containerName="main" probeResult="failure" output="Get \"https://10.132.0.76:8000/ready\": context deadline exceeded" Nov 03 00:55:45.974935 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:55:45.974054 4939 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-2f9a3a/simulator-w0-2f9a3a-kserve-9596bbb8f-2trgz" podUID="207e2207-fb6c-4076-95df-72e95671c261" containerName="main" probeResult="failure" output="Get \"https://10.132.0.61:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 00:55:46.089032 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:55:46.083610 4939 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-83755a/distinct-2-w3-83755a-kserve-6cf66b995d-2l47b" podUID="c3b05e88-5d5b-4d03-b28d-832f22e3c868" containerName="main" probeResult="failure" output="Get \"https://10.132.0.72:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 00:55:46.178062 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:55:46.178008 4939 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-f7680d/distinct-2-w2-f7680d-kserve-7d7c4564bf-cbv2c" podUID="2de18086-761b-4773-b9fc-15ddc4a220fc" containerName="main" probeResult="failure" output="Get \"https://10.132.0.75:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 00:55:46.426125 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:55:46.426015 4939 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-2f9a3a/simulator-w0-2f9a3a-kserve-9596bbb8f-2trgz" podUID="207e2207-fb6c-4076-95df-72e95671c261" containerName="main" probeResult="failure" output="Get \"https://10.132.0.61:8000/ready\": context deadline exceeded" Nov 03 00:55:46.515286 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:55:46.515254 4939 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-2f9a3a/simulator-w0-2f9a3a-kserve-9596bbb8f-2trgz" Nov 03 00:55:46.569605 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:55:46.568119 4939 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-83755a/distinct-2-w3-83755a-kserve-6cf66b995d-2l47b" podUID="c3b05e88-5d5b-4d03-b28d-832f22e3c868" containerName="main" probeResult="failure" output="Get \"https://10.132.0.72:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Nov 03 00:55:46.569605 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:55:46.568595 4939 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-83755a/distinct-2-w3-83755a-kserve-6cf66b995d-2l47b" Nov 03 00:55:46.648862 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:55:46.648832 4939 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-2f9a3a/simulator-w0-2f9a3a-kserve-9596bbb8f-2trgz" Nov 03 00:55:46.834930 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:55:46.833240 4939 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-f7680d/distinct-w2-f7680d-kserve-756f874b98-qnpv8" podUID="836753fc-f9ef-4f61-b3fe-e5b92b7ee27b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.70:8000/ready\": context deadline exceeded" Nov 03 00:55:46.847948 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:55:46.837090 4939 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-f7680d/distinct-w2-f7680d-kserve-756f874b98-qnpv8" Nov 03 00:55:46.901970 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:55:46.882029 4939 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-88bbf7/simulator-w1-88bbf7-kserve-5fbb669f8f-42hgd" podUID="83b38883-2e7a-444d-aa68-52be4a25ae66" containerName="main" probeResult="failure" output="Get \"https://10.132.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 00:55:46.901970 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:55:46.882354 4939 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-88bbf7/simulator-w1-88bbf7-kserve-5fbb669f8f-42hgd" Nov 03 00:55:47.116567 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:55:47.116464 4939 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-2f9a3a/distinct-w0-2f9a3a-kserve-555565cfcf-jtcr9" podUID="ac0ece06-8718-463f-915f-5315988d9711" containerName="main" probeResult="failure" output="Get \"https://10.132.0.76:8000/ready\": context deadline exceeded" Nov 03 00:55:47.255932 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:55:47.183218 4939 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-f7680d/distinct-2-w2-f7680d-kserve-7d7c4564bf-cbv2c" podUID="2de18086-761b-4773-b9fc-15ddc4a220fc" containerName="main" probeResult="failure" output="Get \"https://10.132.0.75:8000/ready\": context deadline exceeded" Nov 03 00:55:47.255932 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:55:47.255821 4939 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-2f9a3a/distinct-w0-2f9a3a-kserve-555565cfcf-jtcr9" Nov 03 00:55:47.437631 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:55:47.437533 4939 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-f7680d/distinct-2-w2-f7680d-kserve-7d7c4564bf-cbv2c" Nov 03 00:55:47.605327 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:55:47.605276 4939 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-83755a/distinct-2-w3-83755a-kserve-6cf66b995d-2l47b" podUID="c3b05e88-5d5b-4d03-b28d-832f22e3c868" containerName="main" probeResult="failure" output="Get \"https://10.132.0.72:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 00:55:47.707997 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:55:47.707690 4939 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-f7680d/premium-w2-f7680d-kserve-76565b8cf9-hqbbt" podUID="4712e708-18a2-4a13-a12c-a5669397b2c3" containerName="main" probeResult="failure" output="Get \"https://10.132.0.63:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Nov 03 00:55:47.747933 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:55:47.740993 4939 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-2f9a3a/simulator-w0-2f9a3a-kserve-9596bbb8f-2trgz" podUID="207e2207-fb6c-4076-95df-72e95671c261" containerName="main" probeResult="failure" output="Get \"https://10.132.0.61:8000/ready\": context deadline exceeded" Nov 03 00:55:47.885044 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:55:47.884993 4939 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-88bbf7/simulator-w1-88bbf7-kserve-5fbb669f8f-42hgd" podUID="83b38883-2e7a-444d-aa68-52be4a25ae66" containerName="main" probeResult="failure" output="Get \"https://10.132.0.62:8000/ready\": context deadline exceeded" Nov 03 00:55:47.899933 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:55:47.886405 4939 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-f7680d/distinct-w2-f7680d-kserve-756f874b98-qnpv8" podUID="836753fc-f9ef-4f61-b3fe-e5b92b7ee27b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.70:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 00:55:48.030741 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:55:48.030640 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 00:55:48.090955 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:55:48.082617 4939 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-f7680d/distinct-w2-f7680d-kserve-756f874b98-qnpv8" podUID="836753fc-f9ef-4f61-b3fe-e5b92b7ee27b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.70:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 00:55:48.267074 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:55:48.267027 4939 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-2f9a3a/distinct-w0-2f9a3a-kserve-555565cfcf-jtcr9" podUID="ac0ece06-8718-463f-915f-5315988d9711" containerName="main" probeResult="failure" output="Get \"https://10.132.0.76:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 00:55:48.294557 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:55:48.267649 4939 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-88bbf7/simulator-w1-88bbf7-kserve-5fbb669f8f-42hgd" podUID="83b38883-2e7a-444d-aa68-52be4a25ae66" containerName="main" probeResult="failure" output="Get \"https://10.132.0.62:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 00:55:48.411088 ip-10-0-140-107 kubenswrapper[4939]: E1103 00:55:48.405233 4939 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-sthjh" containerName="extractor" Nov 03 00:55:48.445145 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:55:48.445091 4939 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-88bbf7/distinct-2-w1-88bbf7-kserve-5457cdcdbd-l6tfd" podUID="a397931b-faa9-4c2e-8290-491350546663" containerName="main" probeResult="failure" output="Get \"https://10.132.0.85:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 00:55:48.494015 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:55:48.493970 4939 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-mt-dd03e395/e2e-lifecycle-model-dd03e395-kserve-78979759df-b8v9c" Nov 03 00:55:48.606100 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:55:48.606044 4939 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-83755a/distinct-2-w3-83755a-kserve-6cf66b995d-2l47b" podUID="c3b05e88-5d5b-4d03-b28d-832f22e3c868" containerName="main" probeResult="failure" output="Get \"https://10.132.0.72:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 00:55:48.607980 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:55:48.607957 4939 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-f7680d/distinct-2-w2-f7680d-kserve-7d7c4564bf-cbv2c" Nov 03 00:55:48.725024 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:55:48.724967 4939 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-f7680d/premium-w2-f7680d-kserve-76565b8cf9-hqbbt" podUID="4712e708-18a2-4a13-a12c-a5669397b2c3" containerName="main" probeResult="failure" output="Get \"https://10.132.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 00:55:48.725852 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:55:48.725827 4939 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-f7680d/premium-w2-f7680d-kserve-76565b8cf9-hqbbt" Nov 03 00:55:48.762113 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:55:48.762069 4939 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-2f9a3a/simulator-w0-2f9a3a-kserve-9596bbb8f-2trgz" podUID="207e2207-fb6c-4076-95df-72e95671c261" containerName="main" probeResult="failure" output="Get \"https://10.132.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 00:55:48.789736 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:55:48.789704 4939 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-2f9a3a/simulator-w0-2f9a3a-kserve-9596bbb8f-2trgz" Nov 03 00:55:48.887177 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:55:48.887059 4939 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-88bbf7/simulator-w1-88bbf7-kserve-5fbb669f8f-42hgd" podUID="83b38883-2e7a-444d-aa68-52be4a25ae66" containerName="main" probeResult="failure" output="Get \"https://10.132.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 00:55:48.911933 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:55:48.910309 4939 prober.go:122] "Probe failed" probeType="Startup" pod="ai-tenant-e2e-mt-dd03e395/e2e-lifecycle-model-dd03e395-kserve-78979759df-b8v9c" podUID="e4803d7d-4603-4b14-950d-018f88294285" containerName="main" probeResult="failure" output="Get \"https://10.132.0.91:8000/health\": context deadline exceeded" Nov 03 00:55:48.913106 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:55:48.913059 4939 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-f7680d/distinct-w2-f7680d-kserve-756f874b98-qnpv8" podUID="836753fc-f9ef-4f61-b3fe-e5b92b7ee27b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.70:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 00:55:49.115267 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:55:49.115205 4939 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-f7680d/distinct-2-w2-f7680d-kserve-7d7c4564bf-cbv2c" podUID="2de18086-761b-4773-b9fc-15ddc4a220fc" containerName="main" probeResult="failure" output="Get \"https://10.132.0.75:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 00:55:49.151754 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:55:49.151661 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 00:55:49.327935 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:55:49.327160 4939 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-2f9a3a/distinct-w0-2f9a3a-kserve-555565cfcf-jtcr9" podUID="ac0ece06-8718-463f-915f-5315988d9711" containerName="main" probeResult="failure" output="Get \"https://10.132.0.76:8000/ready\": context deadline exceeded" Nov 03 00:55:49.439569 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:55:49.403447 4939 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-83755a/distinct-2-w3-83755a-kserve-6cf66b995d-2l47b" Nov 03 00:55:49.483956 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:55:49.469151 4939 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-88bbf7/distinct-2-w1-88bbf7-kserve-5457cdcdbd-l6tfd" podUID="a397931b-faa9-4c2e-8290-491350546663" containerName="main" probeResult="failure" output="Get \"https://10.132.0.85:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 00:55:49.568970 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:55:49.552571 4939 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-88bbf7/distinct-2-w1-88bbf7-kserve-5457cdcdbd-l6tfd" Nov 03 00:55:49.593110 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:55:49.593065 4939 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-2f9a3a/simulator-w0-2f9a3a-kserve-9596bbb8f-2trgz" Nov 03 00:55:49.803982 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:55:49.803071 4939 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-f7680d/premium-w2-f7680d-kserve-76565b8cf9-hqbbt" podUID="4712e708-18a2-4a13-a12c-a5669397b2c3" containerName="main" probeResult="failure" output="Get \"https://10.132.0.63:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Nov 03 00:55:49.889460 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:55:49.889415 4939 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-88bbf7/simulator-w1-88bbf7-kserve-5fbb669f8f-42hgd" podUID="83b38883-2e7a-444d-aa68-52be4a25ae66" containerName="main" probeResult="failure" output="Get \"https://10.132.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 00:55:49.921957 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:55:49.918053 4939 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-f7680d/distinct-w2-f7680d-kserve-756f874b98-qnpv8" podUID="836753fc-f9ef-4f61-b3fe-e5b92b7ee27b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.70:8000/ready\": context deadline exceeded" Nov 03 00:55:49.923954 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:55:49.923925 4939 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-2f9a3a/distinct-w0-2f9a3a-kserve-555565cfcf-jtcr9" Nov 03 00:55:49.924167 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:55:49.924150 4939 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-88bbf7/distinct-2-w1-88bbf7-kserve-5457cdcdbd-l6tfd" Nov 03 00:55:49.945696 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:55:49.944885 4939 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-f7680d/premium-w2-f7680d-kserve-76565b8cf9-hqbbt" Nov 03 00:55:49.945696 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:55:49.946545 4939 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-f7680d/distinct-w2-f7680d-kserve-756f874b98-qnpv8" Nov 03 00:55:49.964964 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:55:49.961716 4939 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-88bbf7/simulator-w1-88bbf7-kserve-5fbb669f8f-42hgd" Nov 03 00:55:50.287969 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:55:50.287937 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 00:55:50.993343 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:55:50.993301 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 00:55:51.673375 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:55:51.673330 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 00:55:52.158250 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:55:52.158223 4939 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-n42s6_65d134ac-8a25-4c67-b6c7-cd752a098daf/console-operator/2.log" Nov 03 00:55:52.158670 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:55:52.158373 4939 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-n42s6_65d134ac-8a25-4c67-b6c7-cd752a098daf/console-operator/2.log" Nov 03 00:55:52.389492 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:55:52.389460 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 00:55:53.088656 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:55:53.088614 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 00:55:53.744005 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:55:53.743972 4939 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-88bbf7-openshift-default-574f47fff7-b229j"] Nov 03 00:55:53.750227 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:55:53.750197 4939 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-88bbf7-openshift-default-574f47fff7-b229j" Nov 03 00:55:53.774839 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:55:53.774805 4939 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-88bbf7-openshift-default-574f47fff7-b229j"] Nov 03 00:55:53.805935 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:55:53.805873 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c7753242-f913-42a4-89d5-bc0ded362646-istio-envoy\") pod \"e2e-worker-w1-88bbf7-openshift-default-574f47fff7-b229j\" (UID: \"c7753242-f913-42a4-89d5-bc0ded362646\") " pod="openshift-ingress/e2e-worker-w1-88bbf7-openshift-default-574f47fff7-b229j" Nov 03 00:55:53.805935 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:55:53.805925 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c7753242-f913-42a4-89d5-bc0ded362646-istio-token\") pod \"e2e-worker-w1-88bbf7-openshift-default-574f47fff7-b229j\" (UID: \"c7753242-f913-42a4-89d5-bc0ded362646\") " pod="openshift-ingress/e2e-worker-w1-88bbf7-openshift-default-574f47fff7-b229j" Nov 03 00:55:53.806132 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:55:53.806035 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c7753242-f913-42a4-89d5-bc0ded362646-istiod-ca-cert\") pod \"e2e-worker-w1-88bbf7-openshift-default-574f47fff7-b229j\" (UID: \"c7753242-f913-42a4-89d5-bc0ded362646\") " pod="openshift-ingress/e2e-worker-w1-88bbf7-openshift-default-574f47fff7-b229j" Nov 03 00:55:53.806132 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:55:53.806064 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c7753242-f913-42a4-89d5-bc0ded362646-istio-data\") pod \"e2e-worker-w1-88bbf7-openshift-default-574f47fff7-b229j\" (UID: \"c7753242-f913-42a4-89d5-bc0ded362646\") " pod="openshift-ingress/e2e-worker-w1-88bbf7-openshift-default-574f47fff7-b229j" Nov 03 00:55:53.806212 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:55:53.806143 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c7753242-f913-42a4-89d5-bc0ded362646-workload-socket\") pod \"e2e-worker-w1-88bbf7-openshift-default-574f47fff7-b229j\" (UID: \"c7753242-f913-42a4-89d5-bc0ded362646\") " pod="openshift-ingress/e2e-worker-w1-88bbf7-openshift-default-574f47fff7-b229j" Nov 03 00:55:53.806212 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:55:53.806204 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c7753242-f913-42a4-89d5-bc0ded362646-credential-socket\") pod \"e2e-worker-w1-88bbf7-openshift-default-574f47fff7-b229j\" (UID: \"c7753242-f913-42a4-89d5-bc0ded362646\") " pod="openshift-ingress/e2e-worker-w1-88bbf7-openshift-default-574f47fff7-b229j" Nov 03 00:55:53.806279 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:55:53.806234 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c7753242-f913-42a4-89d5-bc0ded362646-istio-podinfo\") pod \"e2e-worker-w1-88bbf7-openshift-default-574f47fff7-b229j\" (UID: \"c7753242-f913-42a4-89d5-bc0ded362646\") " pod="openshift-ingress/e2e-worker-w1-88bbf7-openshift-default-574f47fff7-b229j" Nov 03 00:55:53.806334 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:55:53.806311 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6f9zg\" (UniqueName: \"kubernetes.io/projected/c7753242-f913-42a4-89d5-bc0ded362646-kube-api-access-6f9zg\") pod \"e2e-worker-w1-88bbf7-openshift-default-574f47fff7-b229j\" (UID: \"c7753242-f913-42a4-89d5-bc0ded362646\") " pod="openshift-ingress/e2e-worker-w1-88bbf7-openshift-default-574f47fff7-b229j" Nov 03 00:55:53.806466 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:55:53.806444 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c7753242-f913-42a4-89d5-bc0ded362646-workload-certs\") pod \"e2e-worker-w1-88bbf7-openshift-default-574f47fff7-b229j\" (UID: \"c7753242-f913-42a4-89d5-bc0ded362646\") " pod="openshift-ingress/e2e-worker-w1-88bbf7-openshift-default-574f47fff7-b229j" Nov 03 00:55:53.907202 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:55:53.907172 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c7753242-f913-42a4-89d5-bc0ded362646-istiod-ca-cert\") pod \"e2e-worker-w1-88bbf7-openshift-default-574f47fff7-b229j\" (UID: \"c7753242-f913-42a4-89d5-bc0ded362646\") " pod="openshift-ingress/e2e-worker-w1-88bbf7-openshift-default-574f47fff7-b229j" Nov 03 00:55:53.907386 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:55:53.907223 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c7753242-f913-42a4-89d5-bc0ded362646-istio-data\") pod \"e2e-worker-w1-88bbf7-openshift-default-574f47fff7-b229j\" (UID: \"c7753242-f913-42a4-89d5-bc0ded362646\") " pod="openshift-ingress/e2e-worker-w1-88bbf7-openshift-default-574f47fff7-b229j" Nov 03 00:55:53.907386 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:55:53.907316 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c7753242-f913-42a4-89d5-bc0ded362646-workload-socket\") pod \"e2e-worker-w1-88bbf7-openshift-default-574f47fff7-b229j\" (UID: \"c7753242-f913-42a4-89d5-bc0ded362646\") " pod="openshift-ingress/e2e-worker-w1-88bbf7-openshift-default-574f47fff7-b229j" Nov 03 00:55:53.907509 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:55:53.907404 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c7753242-f913-42a4-89d5-bc0ded362646-credential-socket\") pod \"e2e-worker-w1-88bbf7-openshift-default-574f47fff7-b229j\" (UID: \"c7753242-f913-42a4-89d5-bc0ded362646\") " pod="openshift-ingress/e2e-worker-w1-88bbf7-openshift-default-574f47fff7-b229j" Nov 03 00:55:53.907509 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:55:53.907469 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c7753242-f913-42a4-89d5-bc0ded362646-istio-podinfo\") pod \"e2e-worker-w1-88bbf7-openshift-default-574f47fff7-b229j\" (UID: \"c7753242-f913-42a4-89d5-bc0ded362646\") " pod="openshift-ingress/e2e-worker-w1-88bbf7-openshift-default-574f47fff7-b229j" Nov 03 00:55:53.907588 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:55:53.907535 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6f9zg\" (UniqueName: \"kubernetes.io/projected/c7753242-f913-42a4-89d5-bc0ded362646-kube-api-access-6f9zg\") pod \"e2e-worker-w1-88bbf7-openshift-default-574f47fff7-b229j\" (UID: \"c7753242-f913-42a4-89d5-bc0ded362646\") " pod="openshift-ingress/e2e-worker-w1-88bbf7-openshift-default-574f47fff7-b229j" Nov 03 00:55:53.907623 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:55:53.907596 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c7753242-f913-42a4-89d5-bc0ded362646-istio-data\") pod \"e2e-worker-w1-88bbf7-openshift-default-574f47fff7-b229j\" (UID: \"c7753242-f913-42a4-89d5-bc0ded362646\") " pod="openshift-ingress/e2e-worker-w1-88bbf7-openshift-default-574f47fff7-b229j" Nov 03 00:55:53.907737 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:55:53.907715 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c7753242-f913-42a4-89d5-bc0ded362646-workload-certs\") pod \"e2e-worker-w1-88bbf7-openshift-default-574f47fff7-b229j\" (UID: \"c7753242-f913-42a4-89d5-bc0ded362646\") " pod="openshift-ingress/e2e-worker-w1-88bbf7-openshift-default-574f47fff7-b229j" Nov 03 00:55:53.907806 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:55:53.907719 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c7753242-f913-42a4-89d5-bc0ded362646-credential-socket\") pod \"e2e-worker-w1-88bbf7-openshift-default-574f47fff7-b229j\" (UID: \"c7753242-f913-42a4-89d5-bc0ded362646\") " pod="openshift-ingress/e2e-worker-w1-88bbf7-openshift-default-574f47fff7-b229j" Nov 03 00:55:53.907848 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:55:53.907799 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c7753242-f913-42a4-89d5-bc0ded362646-istio-envoy\") pod \"e2e-worker-w1-88bbf7-openshift-default-574f47fff7-b229j\" (UID: \"c7753242-f913-42a4-89d5-bc0ded362646\") " pod="openshift-ingress/e2e-worker-w1-88bbf7-openshift-default-574f47fff7-b229j" Nov 03 00:55:53.907892 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:55:53.907855 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c7753242-f913-42a4-89d5-bc0ded362646-istio-token\") pod \"e2e-worker-w1-88bbf7-openshift-default-574f47fff7-b229j\" (UID: \"c7753242-f913-42a4-89d5-bc0ded362646\") " pod="openshift-ingress/e2e-worker-w1-88bbf7-openshift-default-574f47fff7-b229j" Nov 03 00:55:53.907950 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:55:53.907935 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c7753242-f913-42a4-89d5-bc0ded362646-istiod-ca-cert\") pod \"e2e-worker-w1-88bbf7-openshift-default-574f47fff7-b229j\" (UID: \"c7753242-f913-42a4-89d5-bc0ded362646\") " pod="openshift-ingress/e2e-worker-w1-88bbf7-openshift-default-574f47fff7-b229j" Nov 03 00:55:53.908121 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:55:53.908097 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c7753242-f913-42a4-89d5-bc0ded362646-workload-certs\") pod \"e2e-worker-w1-88bbf7-openshift-default-574f47fff7-b229j\" (UID: \"c7753242-f913-42a4-89d5-bc0ded362646\") " pod="openshift-ingress/e2e-worker-w1-88bbf7-openshift-default-574f47fff7-b229j" Nov 03 00:55:53.908201 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:55:53.908181 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c7753242-f913-42a4-89d5-bc0ded362646-workload-socket\") pod \"e2e-worker-w1-88bbf7-openshift-default-574f47fff7-b229j\" (UID: \"c7753242-f913-42a4-89d5-bc0ded362646\") " pod="openshift-ingress/e2e-worker-w1-88bbf7-openshift-default-574f47fff7-b229j" Nov 03 00:55:53.910129 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:55:53.910101 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c7753242-f913-42a4-89d5-bc0ded362646-istio-envoy\") pod \"e2e-worker-w1-88bbf7-openshift-default-574f47fff7-b229j\" (UID: \"c7753242-f913-42a4-89d5-bc0ded362646\") " pod="openshift-ingress/e2e-worker-w1-88bbf7-openshift-default-574f47fff7-b229j" Nov 03 00:55:53.910219 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:55:53.910148 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c7753242-f913-42a4-89d5-bc0ded362646-istio-podinfo\") pod \"e2e-worker-w1-88bbf7-openshift-default-574f47fff7-b229j\" (UID: \"c7753242-f913-42a4-89d5-bc0ded362646\") " pod="openshift-ingress/e2e-worker-w1-88bbf7-openshift-default-574f47fff7-b229j" Nov 03 00:55:53.911068 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:55:53.911048 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 00:55:53.915595 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:55:53.915571 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6f9zg\" (UniqueName: \"kubernetes.io/projected/c7753242-f913-42a4-89d5-bc0ded362646-kube-api-access-6f9zg\") pod \"e2e-worker-w1-88bbf7-openshift-default-574f47fff7-b229j\" (UID: \"c7753242-f913-42a4-89d5-bc0ded362646\") " pod="openshift-ingress/e2e-worker-w1-88bbf7-openshift-default-574f47fff7-b229j" Nov 03 00:55:53.915726 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:55:53.915705 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c7753242-f913-42a4-89d5-bc0ded362646-istio-token\") pod \"e2e-worker-w1-88bbf7-openshift-default-574f47fff7-b229j\" (UID: \"c7753242-f913-42a4-89d5-bc0ded362646\") " pod="openshift-ingress/e2e-worker-w1-88bbf7-openshift-default-574f47fff7-b229j" Nov 03 00:55:54.084105 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:55:54.084075 4939 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-88bbf7-openshift-default-574f47fff7-b229j" Nov 03 00:55:54.370150 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:55:54.370117 4939 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-88bbf7-openshift-default-574f47fff7-b229j"] Nov 03 00:55:54.371247 ip-10-0-140-107 kubenswrapper[4939]: W1103 00:55:54.371203 4939 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc7753242_f913_42a4_89d5_bc0ded362646.slice/crio-00d10162a6027715228d61dc08fbde06cd4a8e6ee8def426d6c70e804388444e WatchSource:0}: Error finding container 00d10162a6027715228d61dc08fbde06cd4a8e6ee8def426d6c70e804388444e: Status 404 returned error can't find the container with id 00d10162a6027715228d61dc08fbde06cd4a8e6ee8def426d6c70e804388444e Nov 03 00:55:54.374351 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:55:54.374305 4939 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 00:55:54.374484 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:55:54.374402 4939 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 00:55:54.374484 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:55:54.374478 4939 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 00:55:54.411810 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:55:54.411779 4939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-88bbf7-openshift-default-574f47fff7-b229j" event={"ID":"c7753242-f913-42a4-89d5-bc0ded362646","Type":"ContainerStarted","Data":"00d10162a6027715228d61dc08fbde06cd4a8e6ee8def426d6c70e804388444e"} Nov 03 00:55:54.593098 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:55:54.593055 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 00:55:55.319024 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:55:55.318975 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 00:55:55.480930 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:55:55.480588 4939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-88bbf7-openshift-default-574f47fff7-b229j" event={"ID":"c7753242-f913-42a4-89d5-bc0ded362646","Type":"ContainerStarted","Data":"077d31059183a7267fe7e4c9d7ad94c6ae65ddb25bb63b933d33c400dd67bfa7"} Nov 03 00:55:56.107931 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:55:56.097123 4939 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-88bbf7-openshift-default-574f47fff7-b229j" Nov 03 00:55:56.107931 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:55:56.097262 4939 patch_prober.go:29] interesting pod/e2e-worker-w1-88bbf7-openshift-default-574f47fff7-b229j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.92:15021/healthz/ready\": dial tcp 10.132.0.92:15021: connect: connection refused" start-of-body= Nov 03 00:55:56.107931 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:55:56.097322 4939 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-88bbf7-openshift-default-574f47fff7-b229j" podUID="c7753242-f913-42a4-89d5-bc0ded362646" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.92:15021/healthz/ready\": dial tcp 10.132.0.92:15021: connect: connection refused" Nov 03 00:55:56.589023 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:55:56.574224 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 00:55:56.815964 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:55:56.815150 4939 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-88bbf7-openshift-default-574f47fff7-b229j" podStartSLOduration=3.81512647 podStartE2EDuration="3.81512647s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:55:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:55:55.506655158 +0000 UTC m=+903.770990103" watchObservedRunningTime="2026-11-03 00:55:56.81512647 +0000 UTC m=+905.079461416" Nov 03 00:55:56.822938 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:55:56.820649 4939 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-dd03e395-openshift-default-75799f6fc-4gdnp"] Nov 03 00:55:56.826271 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:55:56.825614 4939 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-dd03e395-openshift-default-75799f6fc-4gdnp" Nov 03 00:55:56.834281 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:55:56.834253 4939 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-mt-dd03e395-openshift-default-dockercfg-9rst4\"" Nov 03 00:55:56.841411 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:55:56.841336 4939 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-dd03e395-openshift-default-75799f6fc-4gdnp"] Nov 03 00:55:56.941022 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:55:56.939092 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dd1a7eae-e782-4a82-bdac-c45c75614357-istio-data\") pod \"e2e-mt-dd03e395-openshift-default-75799f6fc-4gdnp\" (UID: \"dd1a7eae-e782-4a82-bdac-c45c75614357\") " pod="openshift-ingress/e2e-mt-dd03e395-openshift-default-75799f6fc-4gdnp" Nov 03 00:55:56.941022 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:55:56.939185 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dd1a7eae-e782-4a82-bdac-c45c75614357-istiod-ca-cert\") pod \"e2e-mt-dd03e395-openshift-default-75799f6fc-4gdnp\" (UID: \"dd1a7eae-e782-4a82-bdac-c45c75614357\") " pod="openshift-ingress/e2e-mt-dd03e395-openshift-default-75799f6fc-4gdnp" Nov 03 00:55:56.941022 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:55:56.939354 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dd1a7eae-e782-4a82-bdac-c45c75614357-istio-envoy\") pod \"e2e-mt-dd03e395-openshift-default-75799f6fc-4gdnp\" (UID: \"dd1a7eae-e782-4a82-bdac-c45c75614357\") " pod="openshift-ingress/e2e-mt-dd03e395-openshift-default-75799f6fc-4gdnp" Nov 03 00:55:56.941022 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:55:56.939519 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dd1a7eae-e782-4a82-bdac-c45c75614357-workload-socket\") pod \"e2e-mt-dd03e395-openshift-default-75799f6fc-4gdnp\" (UID: \"dd1a7eae-e782-4a82-bdac-c45c75614357\") " pod="openshift-ingress/e2e-mt-dd03e395-openshift-default-75799f6fc-4gdnp" Nov 03 00:55:56.941022 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:55:56.939582 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dd1a7eae-e782-4a82-bdac-c45c75614357-workload-certs\") pod \"e2e-mt-dd03e395-openshift-default-75799f6fc-4gdnp\" (UID: \"dd1a7eae-e782-4a82-bdac-c45c75614357\") " pod="openshift-ingress/e2e-mt-dd03e395-openshift-default-75799f6fc-4gdnp" Nov 03 00:55:56.941022 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:55:56.939639 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dd1a7eae-e782-4a82-bdac-c45c75614357-credential-socket\") pod \"e2e-mt-dd03e395-openshift-default-75799f6fc-4gdnp\" (UID: \"dd1a7eae-e782-4a82-bdac-c45c75614357\") " pod="openshift-ingress/e2e-mt-dd03e395-openshift-default-75799f6fc-4gdnp" Nov 03 00:55:56.941022 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:55:56.939665 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dd1a7eae-e782-4a82-bdac-c45c75614357-istio-token\") pod \"e2e-mt-dd03e395-openshift-default-75799f6fc-4gdnp\" (UID: \"dd1a7eae-e782-4a82-bdac-c45c75614357\") " pod="openshift-ingress/e2e-mt-dd03e395-openshift-default-75799f6fc-4gdnp" Nov 03 00:55:56.941022 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:55:56.939710 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wp9xr\" (UniqueName: \"kubernetes.io/projected/dd1a7eae-e782-4a82-bdac-c45c75614357-kube-api-access-wp9xr\") pod \"e2e-mt-dd03e395-openshift-default-75799f6fc-4gdnp\" (UID: \"dd1a7eae-e782-4a82-bdac-c45c75614357\") " pod="openshift-ingress/e2e-mt-dd03e395-openshift-default-75799f6fc-4gdnp" Nov 03 00:55:56.941022 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:55:56.939738 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dd1a7eae-e782-4a82-bdac-c45c75614357-istio-podinfo\") pod \"e2e-mt-dd03e395-openshift-default-75799f6fc-4gdnp\" (UID: \"dd1a7eae-e782-4a82-bdac-c45c75614357\") " pod="openshift-ingress/e2e-mt-dd03e395-openshift-default-75799f6fc-4gdnp" Nov 03 00:55:57.045519 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:55:57.043889 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dd1a7eae-e782-4a82-bdac-c45c75614357-workload-socket\") pod \"e2e-mt-dd03e395-openshift-default-75799f6fc-4gdnp\" (UID: \"dd1a7eae-e782-4a82-bdac-c45c75614357\") " pod="openshift-ingress/e2e-mt-dd03e395-openshift-default-75799f6fc-4gdnp" Nov 03 00:55:57.045519 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:55:57.043997 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dd1a7eae-e782-4a82-bdac-c45c75614357-workload-certs\") pod \"e2e-mt-dd03e395-openshift-default-75799f6fc-4gdnp\" (UID: \"dd1a7eae-e782-4a82-bdac-c45c75614357\") " pod="openshift-ingress/e2e-mt-dd03e395-openshift-default-75799f6fc-4gdnp" Nov 03 00:55:57.045519 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:55:57.044038 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dd1a7eae-e782-4a82-bdac-c45c75614357-credential-socket\") pod \"e2e-mt-dd03e395-openshift-default-75799f6fc-4gdnp\" (UID: \"dd1a7eae-e782-4a82-bdac-c45c75614357\") " pod="openshift-ingress/e2e-mt-dd03e395-openshift-default-75799f6fc-4gdnp" Nov 03 00:55:57.045519 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:55:57.044062 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dd1a7eae-e782-4a82-bdac-c45c75614357-istio-token\") pod \"e2e-mt-dd03e395-openshift-default-75799f6fc-4gdnp\" (UID: \"dd1a7eae-e782-4a82-bdac-c45c75614357\") " pod="openshift-ingress/e2e-mt-dd03e395-openshift-default-75799f6fc-4gdnp" Nov 03 00:55:57.045519 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:55:57.044099 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wp9xr\" (UniqueName: \"kubernetes.io/projected/dd1a7eae-e782-4a82-bdac-c45c75614357-kube-api-access-wp9xr\") pod \"e2e-mt-dd03e395-openshift-default-75799f6fc-4gdnp\" (UID: \"dd1a7eae-e782-4a82-bdac-c45c75614357\") " pod="openshift-ingress/e2e-mt-dd03e395-openshift-default-75799f6fc-4gdnp" Nov 03 00:55:57.045519 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:55:57.044124 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dd1a7eae-e782-4a82-bdac-c45c75614357-istio-podinfo\") pod \"e2e-mt-dd03e395-openshift-default-75799f6fc-4gdnp\" (UID: \"dd1a7eae-e782-4a82-bdac-c45c75614357\") " pod="openshift-ingress/e2e-mt-dd03e395-openshift-default-75799f6fc-4gdnp" Nov 03 00:55:57.045519 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:55:57.044165 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dd1a7eae-e782-4a82-bdac-c45c75614357-istio-data\") pod \"e2e-mt-dd03e395-openshift-default-75799f6fc-4gdnp\" (UID: \"dd1a7eae-e782-4a82-bdac-c45c75614357\") " pod="openshift-ingress/e2e-mt-dd03e395-openshift-default-75799f6fc-4gdnp" Nov 03 00:55:57.045519 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:55:57.044228 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dd1a7eae-e782-4a82-bdac-c45c75614357-istiod-ca-cert\") pod \"e2e-mt-dd03e395-openshift-default-75799f6fc-4gdnp\" (UID: \"dd1a7eae-e782-4a82-bdac-c45c75614357\") " pod="openshift-ingress/e2e-mt-dd03e395-openshift-default-75799f6fc-4gdnp" Nov 03 00:55:57.045519 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:55:57.044368 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dd1a7eae-e782-4a82-bdac-c45c75614357-istio-envoy\") pod \"e2e-mt-dd03e395-openshift-default-75799f6fc-4gdnp\" (UID: \"dd1a7eae-e782-4a82-bdac-c45c75614357\") " pod="openshift-ingress/e2e-mt-dd03e395-openshift-default-75799f6fc-4gdnp" Nov 03 00:55:57.050974 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:55:57.050940 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dd1a7eae-e782-4a82-bdac-c45c75614357-workload-socket\") pod \"e2e-mt-dd03e395-openshift-default-75799f6fc-4gdnp\" (UID: \"dd1a7eae-e782-4a82-bdac-c45c75614357\") " pod="openshift-ingress/e2e-mt-dd03e395-openshift-default-75799f6fc-4gdnp" Nov 03 00:55:57.051877 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:55:57.051433 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dd1a7eae-e782-4a82-bdac-c45c75614357-workload-certs\") pod \"e2e-mt-dd03e395-openshift-default-75799f6fc-4gdnp\" (UID: \"dd1a7eae-e782-4a82-bdac-c45c75614357\") " pod="openshift-ingress/e2e-mt-dd03e395-openshift-default-75799f6fc-4gdnp" Nov 03 00:55:57.052383 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:55:57.052360 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dd1a7eae-e782-4a82-bdac-c45c75614357-credential-socket\") pod \"e2e-mt-dd03e395-openshift-default-75799f6fc-4gdnp\" (UID: \"dd1a7eae-e782-4a82-bdac-c45c75614357\") " pod="openshift-ingress/e2e-mt-dd03e395-openshift-default-75799f6fc-4gdnp" Nov 03 00:55:57.053056 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:55:57.053034 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dd1a7eae-e782-4a82-bdac-c45c75614357-istio-data\") pod \"e2e-mt-dd03e395-openshift-default-75799f6fc-4gdnp\" (UID: \"dd1a7eae-e782-4a82-bdac-c45c75614357\") " pod="openshift-ingress/e2e-mt-dd03e395-openshift-default-75799f6fc-4gdnp" Nov 03 00:55:57.053833 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:55:57.053811 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dd1a7eae-e782-4a82-bdac-c45c75614357-istiod-ca-cert\") pod \"e2e-mt-dd03e395-openshift-default-75799f6fc-4gdnp\" (UID: \"dd1a7eae-e782-4a82-bdac-c45c75614357\") " pod="openshift-ingress/e2e-mt-dd03e395-openshift-default-75799f6fc-4gdnp" Nov 03 00:55:57.054641 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:55:57.054595 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dd1a7eae-e782-4a82-bdac-c45c75614357-istio-envoy\") pod \"e2e-mt-dd03e395-openshift-default-75799f6fc-4gdnp\" (UID: \"dd1a7eae-e782-4a82-bdac-c45c75614357\") " pod="openshift-ingress/e2e-mt-dd03e395-openshift-default-75799f6fc-4gdnp" Nov 03 00:55:57.118989 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:55:57.116652 4939 patch_prober.go:29] interesting pod/e2e-worker-w1-88bbf7-openshift-default-574f47fff7-b229j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.92:15021/healthz/ready\": dial tcp 10.132.0.92:15021: connect: connection refused" start-of-body= Nov 03 00:55:57.118989 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:55:57.116713 4939 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-88bbf7-openshift-default-574f47fff7-b229j" podUID="c7753242-f913-42a4-89d5-bc0ded362646" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.92:15021/healthz/ready\": dial tcp 10.132.0.92:15021: connect: connection refused" Nov 03 00:55:57.180670 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:55:57.140276 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dd1a7eae-e782-4a82-bdac-c45c75614357-istio-token\") pod \"e2e-mt-dd03e395-openshift-default-75799f6fc-4gdnp\" (UID: \"dd1a7eae-e782-4a82-bdac-c45c75614357\") " pod="openshift-ingress/e2e-mt-dd03e395-openshift-default-75799f6fc-4gdnp" Nov 03 00:55:57.180670 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:55:57.145694 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dd1a7eae-e782-4a82-bdac-c45c75614357-istio-podinfo\") pod \"e2e-mt-dd03e395-openshift-default-75799f6fc-4gdnp\" (UID: \"dd1a7eae-e782-4a82-bdac-c45c75614357\") " pod="openshift-ingress/e2e-mt-dd03e395-openshift-default-75799f6fc-4gdnp" Nov 03 00:55:57.190575 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:55:57.190287 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wp9xr\" (UniqueName: \"kubernetes.io/projected/dd1a7eae-e782-4a82-bdac-c45c75614357-kube-api-access-wp9xr\") pod \"e2e-mt-dd03e395-openshift-default-75799f6fc-4gdnp\" (UID: \"dd1a7eae-e782-4a82-bdac-c45c75614357\") " pod="openshift-ingress/e2e-mt-dd03e395-openshift-default-75799f6fc-4gdnp" Nov 03 00:55:57.331822 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:55:57.331785 4939 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-dd03e395-openshift-default-75799f6fc-4gdnp" Nov 03 00:55:57.544341 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:55:57.543675 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 00:55:57.719450 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:55:57.719318 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 00:55:57.916135 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:55:57.916104 4939 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-dd03e395-openshift-default-75799f6fc-4gdnp"] Nov 03 00:55:57.930655 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:55:57.930600 4939 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 00:55:57.930810 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:55:57.930696 4939 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 00:55:57.930810 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:55:57.930737 4939 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 00:55:57.933831 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:55:57.933802 4939 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-mt-dd03e395/e2e-lifecycle-model-dd03e395-kserve-78979759df-b8v9c" Nov 03 00:55:57.975155 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:55:57.974602 4939 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-mt-dd03e395/e2e-lifecycle-model-dd03e395-kserve-78979759df-b8v9c" Nov 03 00:55:58.042561 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:55:58.042476 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 00:55:58.084697 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:55:58.084669 4939 patch_prober.go:29] interesting pod/e2e-worker-w1-88bbf7-openshift-default-574f47fff7-b229j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.92:15021/healthz/ready\": dial tcp 10.132.0.92:15021: connect: connection refused" start-of-body= Nov 03 00:55:58.084870 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:55:58.084735 4939 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-88bbf7-openshift-default-574f47fff7-b229j" podUID="c7753242-f913-42a4-89d5-bc0ded362646" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.92:15021/healthz/ready\": dial tcp 10.132.0.92:15021: connect: connection refused" Nov 03 00:55:58.226985 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:55:58.225808 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 00:55:58.707707 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:55:58.707659 4939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-dd03e395-openshift-default-75799f6fc-4gdnp" event={"ID":"dd1a7eae-e782-4a82-bdac-c45c75614357","Type":"ContainerStarted","Data":"a3457590cb89b41b7165836f93cdce29fd1ad3f88f6139cc1ba7799be0513756"} Nov 03 00:55:58.707942 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:55:58.707759 4939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-dd03e395-openshift-default-75799f6fc-4gdnp" event={"ID":"dd1a7eae-e782-4a82-bdac-c45c75614357","Type":"ContainerStarted","Data":"57dd79d1405d00ca9315750aacdfbddd9fb8ae55ca5167b9a0fa80f211d30416"} Nov 03 00:55:58.730208 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:55:58.730154 4939 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-dd03e395-openshift-default-75799f6fc-4gdnp" podStartSLOduration=2.730140247 podStartE2EDuration="2.730140247s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:55:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:55:58.727646457 +0000 UTC m=+906.991981403" watchObservedRunningTime="2026-11-03 00:55:58.730140247 +0000 UTC m=+906.994475192" Nov 03 00:55:58.995939 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:55:58.995817 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 00:55:59.085486 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:55:59.085448 4939 patch_prober.go:29] interesting pod/e2e-worker-w1-88bbf7-openshift-default-574f47fff7-b229j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.92:15021/healthz/ready\": dial tcp 10.132.0.92:15021: connect: connection refused" start-of-body= Nov 03 00:55:59.085782 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:55:59.085745 4939 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-88bbf7-openshift-default-574f47fff7-b229j" podUID="c7753242-f913-42a4-89d5-bc0ded362646" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.92:15021/healthz/ready\": dial tcp 10.132.0.92:15021: connect: connection refused" Nov 03 00:55:59.332615 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:55:59.332586 4939 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-dd03e395-openshift-default-75799f6fc-4gdnp" Nov 03 00:55:59.334100 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:55:59.334080 4939 patch_prober.go:29] interesting pod/e2e-mt-dd03e395-openshift-default-75799f6fc-4gdnp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.93:15021/healthz/ready\": dial tcp 10.132.0.93:15021: connect: connection refused" start-of-body= Nov 03 00:55:59.334164 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:55:59.334126 4939 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-dd03e395-openshift-default-75799f6fc-4gdnp" podUID="dd1a7eae-e782-4a82-bdac-c45c75614357" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.93:15021/healthz/ready\": dial tcp 10.132.0.93:15021: connect: connection refused" Nov 03 00:55:59.983943 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:55:59.983330 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 00:56:00.094932 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:56:00.093612 4939 patch_prober.go:29] interesting pod/e2e-worker-w1-88bbf7-openshift-default-574f47fff7-b229j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.92:15021/healthz/ready\": dial tcp 10.132.0.92:15021: connect: connection refused" start-of-body= Nov 03 00:56:00.094932 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:56:00.093815 4939 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-88bbf7-openshift-default-574f47fff7-b229j" podUID="c7753242-f913-42a4-89d5-bc0ded362646" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.92:15021/healthz/ready\": dial tcp 10.132.0.92:15021: connect: connection refused" Nov 03 00:56:00.333244 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:56:00.333211 4939 patch_prober.go:29] interesting pod/e2e-mt-dd03e395-openshift-default-75799f6fc-4gdnp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.93:15021/healthz/ready\": dial tcp 10.132.0.93:15021: connect: connection refused" start-of-body= Nov 03 00:56:00.333489 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:56:00.333465 4939 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-dd03e395-openshift-default-75799f6fc-4gdnp" podUID="dd1a7eae-e782-4a82-bdac-c45c75614357" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.93:15021/healthz/ready\": dial tcp 10.132.0.93:15021: connect: connection refused" Nov 03 00:56:01.084764 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:56:01.084732 4939 patch_prober.go:29] interesting pod/e2e-worker-w1-88bbf7-openshift-default-574f47fff7-b229j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.92:15021/healthz/ready\": dial tcp 10.132.0.92:15021: connect: connection refused" start-of-body= Nov 03 00:56:01.097112 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:56:01.084789 4939 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-88bbf7-openshift-default-574f47fff7-b229j" podUID="c7753242-f913-42a4-89d5-bc0ded362646" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.92:15021/healthz/ready\": dial tcp 10.132.0.92:15021: connect: connection refused" Nov 03 00:56:01.179585 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:56:01.179547 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 00:56:01.333231 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:56:01.333193 4939 patch_prober.go:29] interesting pod/e2e-mt-dd03e395-openshift-default-75799f6fc-4gdnp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.93:15021/healthz/ready\": dial tcp 10.132.0.93:15021: connect: connection refused" start-of-body= Nov 03 00:56:01.333432 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:56:01.333254 4939 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-dd03e395-openshift-default-75799f6fc-4gdnp" podUID="dd1a7eae-e782-4a82-bdac-c45c75614357" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.93:15021/healthz/ready\": dial tcp 10.132.0.93:15021: connect: connection refused" Nov 03 00:56:01.805320 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:56:01.805287 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 00:56:01.945827 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:56:01.945780 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 00:56:02.076954 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:56:02.076847 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 00:56:02.288249 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:56:02.288210 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 00:56:02.342518 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:56:02.342479 4939 patch_prober.go:29] interesting pod/e2e-mt-dd03e395-openshift-default-75799f6fc-4gdnp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.93:15021/healthz/ready\": dial tcp 10.132.0.93:15021: connect: connection refused" start-of-body= Nov 03 00:56:02.342863 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:56:02.342812 4939 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-dd03e395-openshift-default-75799f6fc-4gdnp" podUID="dd1a7eae-e782-4a82-bdac-c45c75614357" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.93:15021/healthz/ready\": dial tcp 10.132.0.93:15021: connect: connection refused" Nov 03 00:56:02.995349 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:56:02.995315 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 00:56:03.085758 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:56:03.085723 4939 patch_prober.go:29] interesting pod/e2e-worker-w1-88bbf7-openshift-default-574f47fff7-b229j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.92:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 00:56:03.086133 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:56:03.086097 4939 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-88bbf7-openshift-default-574f47fff7-b229j" podUID="c7753242-f913-42a4-89d5-bc0ded362646" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.92:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 00:56:03.113645 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:56:03.113613 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 00:56:03.280712 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:56:03.280518 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 00:56:03.353948 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:56:03.345298 4939 patch_prober.go:29] interesting pod/e2e-mt-dd03e395-openshift-default-75799f6fc-4gdnp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.93:15021/healthz/ready\": dial tcp 10.132.0.93:15021: connect: connection refused" start-of-body= Nov 03 00:56:03.353948 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:56:03.345358 4939 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-dd03e395-openshift-default-75799f6fc-4gdnp" podUID="dd1a7eae-e782-4a82-bdac-c45c75614357" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.93:15021/healthz/ready\": dial tcp 10.132.0.93:15021: connect: connection refused" Nov 03 00:56:04.109329 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:56:04.109298 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 00:56:04.129068 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:56:04.129028 4939 patch_prober.go:29] interesting pod/e2e-worker-w1-88bbf7-openshift-default-574f47fff7-b229j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.92:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 00:56:04.129249 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:56:04.129109 4939 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-88bbf7-openshift-default-574f47fff7-b229j" podUID="c7753242-f913-42a4-89d5-bc0ded362646" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.92:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 00:56:04.334254 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:56:04.334218 4939 patch_prober.go:29] interesting pod/e2e-mt-dd03e395-openshift-default-75799f6fc-4gdnp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.93:15021/healthz/ready\": dial tcp 10.132.0.93:15021: connect: connection refused" start-of-body= Nov 03 00:56:04.334462 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:56:04.334276 4939 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-dd03e395-openshift-default-75799f6fc-4gdnp" podUID="dd1a7eae-e782-4a82-bdac-c45c75614357" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.93:15021/healthz/ready\": dial tcp 10.132.0.93:15021: connect: connection refused" Nov 03 00:56:04.422792 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:56:04.422707 4939 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-88bbf7-openshift-default-574f47fff7-b229j" Nov 03 00:56:04.423183 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:56:04.423155 4939 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-88bbf7-openshift-default-574f47fff7-b229j" Nov 03 00:56:04.424014 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:56:04.423996 4939 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-88bbf7-openshift-default-574f47fff7-b229j" Nov 03 00:56:05.029006 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:56:05.028874 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 00:56:05.442230 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:56:05.442200 4939 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-dd03e395-openshift-default-75799f6fc-4gdnp" Nov 03 00:56:05.442848 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:56:05.442825 4939 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-dd03e395-openshift-default-75799f6fc-4gdnp" Nov 03 00:56:05.444447 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:56:05.444412 4939 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-dd03e395-openshift-default-75799f6fc-4gdnp" Nov 03 00:56:05.669675 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:56:05.669647 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 00:56:06.283422 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:56:06.283384 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 00:56:06.993992 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:56:06.993961 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 00:56:07.107931 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:56:07.106105 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 00:56:08.091677 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:56:08.091642 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 00:56:08.700870 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:56:08.700828 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 00:56:09.114701 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:56:09.114660 4939 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-4pw7v"] Nov 03 00:56:09.453011 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:56:09.452902 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 00:56:10.420051 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:56:10.420000 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 00:56:11.356899 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:56:11.356856 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 00:56:12.175460 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:56:12.175423 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 00:56:13.012778 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:56:13.012744 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 00:56:13.780256 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:56:13.780223 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 00:56:14.505617 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:56:14.505581 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 00:56:15.311669 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:56:15.311635 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 00:56:16.007287 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:56:16.007256 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 00:56:16.800454 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:56:16.800426 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 00:56:17.506556 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:56:17.506521 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 00:56:18.031073 ip-10-0-140-107 kubenswrapper[4939]: E1103 00:56:18.031036 4939 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-sthjh" containerName="extractor" Nov 03 00:56:18.170305 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:56:18.170272 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 00:56:18.780305 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:56:18.780264 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 00:56:19.377874 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:56:19.377840 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 00:56:20.384440 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:56:20.384411 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 00:56:21.145565 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:56:21.145530 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 00:56:21.779931 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:56:21.779871 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 00:56:21.907170 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:56:21.907100 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 00:56:22.481534 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:56:22.481503 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 00:56:22.597048 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:56:22.597012 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 00:56:22.878071 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:56:22.878036 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 00:56:23.235085 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:56:23.235006 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 00:56:23.486996 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:56:23.486885 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 00:56:24.090626 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:56:24.090592 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 00:56:24.751339 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:56:24.751308 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 00:56:25.688531 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:56:25.688485 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 00:56:26.186098 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:56:26.186059 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 00:56:26.303669 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:56:26.303635 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 00:56:26.442949 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:56:26.442843 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 00:56:27.111790 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:56:27.111754 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 00:56:27.443805 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:56:27.443725 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 00:56:27.462299 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:56:27.462266 4939 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-65e5ec-64858685d7-dt99f"] Nov 03 00:56:27.468011 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:56:27.467986 4939 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-65e5ec-64858685d7-dt99f" Nov 03 00:56:27.479610 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:56:27.477776 4939 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-b-65e5ec\"" Nov 03 00:56:27.479937 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:56:27.479879 4939 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-65e5ec-64858685d7-dt99f"] Nov 03 00:56:27.480269 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:56:27.480242 4939 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-b-65e5ec-dockercfg-lc477\"" Nov 03 00:56:27.493974 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:56:27.493901 4939 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-65e5ec-5f46866b59-knkfz"] Nov 03 00:56:27.499288 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:56:27.499260 4939 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-65e5ec-5f46866b59-knkfz" Nov 03 00:56:27.506596 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:56:27.506564 4939 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-65e5ec-5f46866b59-knkfz"] Nov 03 00:56:27.591933 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:56:27.573088 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2gx95\" (UniqueName: \"kubernetes.io/projected/bf1852d1-20e7-4e98-be2a-bc444c04c938-kube-api-access-2gx95\") pod \"payload-pre-processing-e2e-api-b-65e5ec-5f46866b59-knkfz\" (UID: \"bf1852d1-20e7-4e98-be2a-bc444c04c938\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-65e5ec-5f46866b59-knkfz" Nov 03 00:56:27.592258 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:56:27.592223 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zq2pw\" (UniqueName: \"kubernetes.io/projected/a8ce4315-416e-4663-b9bc-d4ef6714c4c1-kube-api-access-zq2pw\") pod \"payload-processing-e2e-api-b-65e5ec-64858685d7-dt99f\" (UID: \"a8ce4315-416e-4663-b9bc-d4ef6714c4c1\") " pod="openshift-ingress/payload-processing-e2e-api-b-65e5ec-64858685d7-dt99f" Nov 03 00:56:27.592442 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:56:27.592420 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bf1852d1-20e7-4e98-be2a-bc444c04c938-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-65e5ec-5f46866b59-knkfz\" (UID: \"bf1852d1-20e7-4e98-be2a-bc444c04c938\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-65e5ec-5f46866b59-knkfz" Nov 03 00:56:27.592963 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:56:27.592935 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a8ce4315-416e-4663-b9bc-d4ef6714c4c1-plugins-config-volume\") pod \"payload-processing-e2e-api-b-65e5ec-64858685d7-dt99f\" (UID: \"a8ce4315-416e-4663-b9bc-d4ef6714c4c1\") " pod="openshift-ingress/payload-processing-e2e-api-b-65e5ec-64858685d7-dt99f" Nov 03 00:56:27.602934 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:56:27.593882 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 00:56:27.696048 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:56:27.695954 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a8ce4315-416e-4663-b9bc-d4ef6714c4c1-plugins-config-volume\") pod \"payload-processing-e2e-api-b-65e5ec-64858685d7-dt99f\" (UID: \"a8ce4315-416e-4663-b9bc-d4ef6714c4c1\") " pod="openshift-ingress/payload-processing-e2e-api-b-65e5ec-64858685d7-dt99f" Nov 03 00:56:27.697356 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:56:27.697326 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2gx95\" (UniqueName: \"kubernetes.io/projected/bf1852d1-20e7-4e98-be2a-bc444c04c938-kube-api-access-2gx95\") pod \"payload-pre-processing-e2e-api-b-65e5ec-5f46866b59-knkfz\" (UID: \"bf1852d1-20e7-4e98-be2a-bc444c04c938\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-65e5ec-5f46866b59-knkfz" Nov 03 00:56:27.697887 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:56:27.697864 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zq2pw\" (UniqueName: \"kubernetes.io/projected/a8ce4315-416e-4663-b9bc-d4ef6714c4c1-kube-api-access-zq2pw\") pod \"payload-processing-e2e-api-b-65e5ec-64858685d7-dt99f\" (UID: \"a8ce4315-416e-4663-b9bc-d4ef6714c4c1\") " pod="openshift-ingress/payload-processing-e2e-api-b-65e5ec-64858685d7-dt99f" Nov 03 00:56:27.704679 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:56:27.704652 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bf1852d1-20e7-4e98-be2a-bc444c04c938-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-65e5ec-5f46866b59-knkfz\" (UID: \"bf1852d1-20e7-4e98-be2a-bc444c04c938\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-65e5ec-5f46866b59-knkfz" Nov 03 00:56:27.714897 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:56:27.705684 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bf1852d1-20e7-4e98-be2a-bc444c04c938-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-65e5ec-5f46866b59-knkfz\" (UID: \"bf1852d1-20e7-4e98-be2a-bc444c04c938\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-65e5ec-5f46866b59-knkfz" Nov 03 00:56:27.715151 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:56:27.697157 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a8ce4315-416e-4663-b9bc-d4ef6714c4c1-plugins-config-volume\") pod \"payload-processing-e2e-api-b-65e5ec-64858685d7-dt99f\" (UID: \"a8ce4315-416e-4663-b9bc-d4ef6714c4c1\") " pod="openshift-ingress/payload-processing-e2e-api-b-65e5ec-64858685d7-dt99f" Nov 03 00:56:27.732397 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:56:27.732347 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zq2pw\" (UniqueName: \"kubernetes.io/projected/a8ce4315-416e-4663-b9bc-d4ef6714c4c1-kube-api-access-zq2pw\") pod \"payload-processing-e2e-api-b-65e5ec-64858685d7-dt99f\" (UID: \"a8ce4315-416e-4663-b9bc-d4ef6714c4c1\") " pod="openshift-ingress/payload-processing-e2e-api-b-65e5ec-64858685d7-dt99f" Nov 03 00:56:27.764270 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:56:27.764234 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2gx95\" (UniqueName: \"kubernetes.io/projected/bf1852d1-20e7-4e98-be2a-bc444c04c938-kube-api-access-2gx95\") pod \"payload-pre-processing-e2e-api-b-65e5ec-5f46866b59-knkfz\" (UID: \"bf1852d1-20e7-4e98-be2a-bc444c04c938\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-65e5ec-5f46866b59-knkfz" Nov 03 00:56:27.793250 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:56:27.793218 4939 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-65e5ec-64858685d7-dt99f" Nov 03 00:56:27.839945 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:56:27.839201 4939 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-65e5ec-5f46866b59-knkfz" Nov 03 00:56:28.226058 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:56:28.225220 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 00:56:28.268792 ip-10-0-140-107 kubenswrapper[4939]: W1103 00:56:28.266714 4939 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda8ce4315_416e_4663_b9bc_d4ef6714c4c1.slice/crio-1294a358a4dff4883fbb7d4af5e5c1cfc5b81da0d1224b059a7bc03ead6f00a0 WatchSource:0}: Error finding container 1294a358a4dff4883fbb7d4af5e5c1cfc5b81da0d1224b059a7bc03ead6f00a0: Status 404 returned error can't find the container with id 1294a358a4dff4883fbb7d4af5e5c1cfc5b81da0d1224b059a7bc03ead6f00a0 Nov 03 00:56:28.272033 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:56:28.271370 4939 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-65e5ec-64858685d7-dt99f"] Nov 03 00:56:28.297300 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:56:28.297268 4939 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-65e5ec-5f46866b59-knkfz"] Nov 03 00:56:28.313746 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:56:28.313713 4939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-65e5ec-5f46866b59-knkfz" event={"ID":"bf1852d1-20e7-4e98-be2a-bc444c04c938","Type":"ContainerStarted","Data":"5832b194f3a8bdf99aedfd644b7391820bdb7ed0966de7ad4af042fbaad70e79"} Nov 03 00:56:28.315150 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:56:28.315126 4939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-65e5ec-64858685d7-dt99f" event={"ID":"a8ce4315-416e-4663-b9bc-d4ef6714c4c1","Type":"ContainerStarted","Data":"1294a358a4dff4883fbb7d4af5e5c1cfc5b81da0d1224b059a7bc03ead6f00a0"} Nov 03 00:56:29.014456 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:56:29.014419 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 00:56:29.324641 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:56:29.324554 4939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-65e5ec-5f46866b59-knkfz" event={"ID":"bf1852d1-20e7-4e98-be2a-bc444c04c938","Type":"ContainerStarted","Data":"622f3e9b4f23392f14772534a1d9f8dbb64a963364c339df4c0acc5a35890132"} Nov 03 00:56:29.324641 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:56:29.324622 4939 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-api-b-65e5ec-5f46866b59-knkfz" Nov 03 00:56:29.326414 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:56:29.326382 4939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-65e5ec-64858685d7-dt99f" event={"ID":"a8ce4315-416e-4663-b9bc-d4ef6714c4c1","Type":"ContainerStarted","Data":"b3b6e002ba5578fa145a1553b8bc939e69df122e1fc3e74390a6438019f58824"} Nov 03 00:56:29.326536 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:56:29.326488 4939 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-api-b-65e5ec-64858685d7-dt99f" Nov 03 00:56:29.342055 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:56:29.342011 4939 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-api-b-65e5ec-5f46866b59-knkfz" podStartSLOduration=2.34199802 podStartE2EDuration="2.34199802s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:56:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:56:29.340348274 +0000 UTC m=+937.604683219" watchObservedRunningTime="2026-11-03 00:56:29.34199802 +0000 UTC m=+937.606332965" Nov 03 00:56:29.357766 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:56:29.357724 4939 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-api-b-65e5ec-64858685d7-dt99f" podStartSLOduration=2.3577129709999998 podStartE2EDuration="2.357712971s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:56:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:56:29.355630016 +0000 UTC m=+937.619964963" watchObservedRunningTime="2026-11-03 00:56:29.357712971 +0000 UTC m=+937.622047915" Nov 03 00:56:29.681903 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:56:29.681868 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 00:56:30.284525 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:56:30.284494 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 00:56:30.909289 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:56:30.909252 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 00:56:31.596971 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:56:31.596918 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 00:56:31.690707 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:56:31.690674 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 00:56:32.779246 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:56:32.779216 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 00:56:33.669569 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:56:33.669532 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 00:56:34.583257 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:56:34.583198 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 00:56:35.265372 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:56:35.265335 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 00:56:35.877895 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:56:35.877864 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 00:56:36.485790 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:56:36.485756 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 00:56:37.088765 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:56:37.088732 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 00:56:37.781084 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:56:37.781052 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 00:56:38.486137 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:56:38.486069 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 00:56:39.181960 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:56:39.181891 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 00:56:39.784138 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:56:39.784105 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 00:56:40.396769 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:56:40.396741 4939 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-api-b-65e5ec-64858685d7-dt99f" Nov 03 00:56:40.396949 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:56:40.396806 4939 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-api-b-65e5ec-5f46866b59-knkfz" Nov 03 00:56:40.502546 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:56:40.502511 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 00:56:41.178180 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:56:41.178144 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 00:56:41.788745 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:56:41.788694 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 00:56:42.282920 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:56:42.282886 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 00:56:42.500164 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:56:42.500132 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 00:56:43.166047 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:56:43.166012 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 00:56:43.789791 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:56:43.789752 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 00:56:44.399006 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:56:44.398972 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 00:56:45.346814 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:56:45.346780 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 00:56:46.100689 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:56:46.100656 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 00:56:46.240739 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:56:46.240710 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 00:56:46.881130 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:56:46.881096 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 00:56:47.639173 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:56:47.639131 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 00:56:47.741434 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:56:47.741400 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 00:56:48.031199 ip-10-0-140-107 kubenswrapper[4939]: E1103 00:56:48.031115 4939 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-sthjh" containerName="extractor" Nov 03 00:56:48.278255 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:56:48.278222 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 00:56:48.879194 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:56:48.879158 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 00:56:49.581641 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:56:49.581611 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 00:56:50.183250 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:56:50.183211 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 00:56:50.805454 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:56:50.805419 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 00:56:51.485802 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:56:51.485750 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 00:56:52.201662 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:56:52.201630 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 00:56:52.340352 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:56:52.340318 4939 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-4pw7v"] Nov 03 00:56:53.314040 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:56:53.314007 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 00:56:54.068549 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:56:54.068492 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 00:56:54.185825 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:56:54.185794 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 00:56:54.321627 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:56:54.321534 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 00:56:55.239536 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:56:55.239501 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 00:56:55.995783 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:56:55.995748 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 00:56:56.791970 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:56:56.791930 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 00:56:57.296338 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:56:57.296302 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 00:56:57.894776 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:56:57.894734 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 00:56:58.409837 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:56:58.409797 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 00:56:58.885232 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:56:58.885201 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 00:56:59.663088 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:56:59.663055 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 00:57:00.160734 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:57:00.160703 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 00:57:00.280510 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:57:00.280475 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 00:57:00.480024 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:57:00.479887 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 00:57:00.740273 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:57:00.740200 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 00:57:01.004362 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:57:01.004277 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 00:57:01.530782 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:57:01.530752 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 00:57:01.768773 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:57:01.768740 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 00:57:02.166656 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:57:02.166625 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 00:57:02.635686 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:57:02.635654 4939 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7034-65817a"] Nov 03 00:57:02.641783 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:57:02.641747 4939 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7034-65817a" Nov 03 00:57:02.645352 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:57:02.645329 4939 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-7l8r6\"" Nov 03 00:57:02.647080 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:57:02.647048 4939 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7034-65817a"] Nov 03 00:57:02.742693 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:57:02.742650 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jp5lw\" (UniqueName: \"kubernetes.io/projected/cb6e4163-f7ee-47f8-8517-a37405e25d5a-kube-api-access-jp5lw\") pod \"test-curl-7034-65817a\" (UID: \"cb6e4163-f7ee-47f8-8517-a37405e25d5a\") " pod="opendatahub/test-curl-7034-65817a" Nov 03 00:57:02.802751 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:57:02.802715 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 00:57:02.863933 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:57:02.844677 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jp5lw\" (UniqueName: \"kubernetes.io/projected/cb6e4163-f7ee-47f8-8517-a37405e25d5a-kube-api-access-jp5lw\") pod \"test-curl-7034-65817a\" (UID: \"cb6e4163-f7ee-47f8-8517-a37405e25d5a\") " pod="opendatahub/test-curl-7034-65817a" Nov 03 00:57:02.891410 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:57:02.891326 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jp5lw\" (UniqueName: \"kubernetes.io/projected/cb6e4163-f7ee-47f8-8517-a37405e25d5a-kube-api-access-jp5lw\") pod \"test-curl-7034-65817a\" (UID: \"cb6e4163-f7ee-47f8-8517-a37405e25d5a\") " pod="opendatahub/test-curl-7034-65817a" Nov 03 00:57:02.962809 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:57:02.962779 4939 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7034-65817a" Nov 03 00:57:03.240117 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:57:03.240086 4939 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7034-65817a"] Nov 03 00:57:03.241409 ip-10-0-140-107 kubenswrapper[4939]: W1103 00:57:03.241383 4939 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podcb6e4163_f7ee_47f8_8517_a37405e25d5a.slice/crio-35525d2b3101b477177d6530c47d528849f1c93c1c9d1424cd07c330d694be22 WatchSource:0}: Error finding container 35525d2b3101b477177d6530c47d528849f1c93c1c9d1424cd07c330d694be22: Status 404 returned error can't find the container with id 35525d2b3101b477177d6530c47d528849f1c93c1c9d1424cd07c330d694be22 Nov 03 00:57:03.449539 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:57:03.449494 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 00:57:03.813973 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:57:03.813939 4939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7034-65817a" event={"ID":"cb6e4163-f7ee-47f8-8517-a37405e25d5a","Type":"ContainerStarted","Data":"35525d2b3101b477177d6530c47d528849f1c93c1c9d1424cd07c330d694be22"} Nov 03 00:57:04.071003 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:57:04.070897 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 00:57:04.687133 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:57:04.687101 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 00:57:05.394779 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:57:05.394739 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 00:57:05.609373 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:57:05.609331 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 00:57:06.121723 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:57:06.121679 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 00:57:06.815604 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:57:06.815522 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 00:57:06.961945 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:57:06.961892 4939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7034-65817a" event={"ID":"cb6e4163-f7ee-47f8-8517-a37405e25d5a","Type":"ContainerStarted","Data":"c20c4ed89a030ac649eb888cf226736a9685d4fc80124332688614dca078860d"} Nov 03 00:57:06.980405 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:57:06.980340 4939 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/test-curl-7034-65817a" podStartSLOduration=1.859422052 podStartE2EDuration="4.980321117s" totalImagesPullingTime="3.120899065s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:57:02 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:57:06.978442823 +0000 UTC m=+975.242777769" watchObservedRunningTime="2026-11-03 00:57:06.980321117 +0000 UTC m=+975.244656063" Nov 03 00:57:07.509367 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:57:07.509339 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 00:57:07.608428 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:57:07.608388 4939 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-65e5ec-64858685d7-dt99f"] Nov 03 00:57:07.615360 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:57:07.615326 4939 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7034-65817a"] Nov 03 00:57:07.646978 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:57:07.641444 4939 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7034-65817a"] Nov 03 00:57:07.646978 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:57:07.642018 4939 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-65e5ec-5f46866b59-knkfz"] Nov 03 00:57:07.738026 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:57:07.737970 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 00:57:07.859676 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:57:07.859638 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 00:57:07.978999 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:57:07.978966 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 00:57:08.371182 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:57:08.371146 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 00:57:08.497807 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:57:08.497763 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 00:57:08.607564 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:57:08.606746 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 00:57:09.111207 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:57:09.111173 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 00:57:09.216890 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:57:09.216849 4939 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/test-curl-7034-65817a" podUID="cb6e4163-f7ee-47f8-8517-a37405e25d5a" containerName="test-curl-7034-65817a" containerID="cri-o://c20c4ed89a030ac649eb888cf226736a9685d4fc80124332688614dca078860d" gracePeriod=2 Nov 03 00:57:09.873675 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:57:09.873620 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 00:57:10.418828 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:57:10.418784 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 00:57:10.927971 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:57:10.927434 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 00:57:11.396890 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:57:11.396862 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 00:57:11.443477 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:57:11.443451 4939 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7034-65817a" Nov 03 00:57:11.533218 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:57:11.533136 4939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cb6e4163-f7ee-47f8-8517-a37405e25d5a-kube-api-access-jp5lw\" (UniqueName: \"kubernetes.io/projected/cb6e4163-f7ee-47f8-8517-a37405e25d5a-kube-api-access-jp5lw\") pod \"cb6e4163-f7ee-47f8-8517-a37405e25d5a\" (UID: \"cb6e4163-f7ee-47f8-8517-a37405e25d5a\") " Nov 03 00:57:11.535257 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:57:11.535231 4939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cb6e4163-f7ee-47f8-8517-a37405e25d5a-kube-api-access-jp5lw" pod "cb6e4163-f7ee-47f8-8517-a37405e25d5a" (UID: "cb6e4163-f7ee-47f8-8517-a37405e25d5a"). InnerVolumeSpecName "kube-api-access-jp5lw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 00:57:11.635453 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:57:11.635415 4939 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jp5lw\" (UniqueName: \"kubernetes.io/projected/cb6e4163-f7ee-47f8-8517-a37405e25d5a-kube-api-access-jp5lw\") on node \"ip-10-0-140-107.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 00:57:11.870563 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:57:11.870531 4939 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cb6e4163-f7ee-47f8-8517-a37405e25d5a" path="/var/lib/kubelet/pods/cb6e4163-f7ee-47f8-8517-a37405e25d5a/volumes" Nov 03 00:57:11.955374 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:57:11.955339 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 00:57:12.186672 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:57:12.186588 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 00:57:12.297942 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:57:12.266966 4939 generic.go:356] "Generic (PLEG): container finished" podID="cb6e4163-f7ee-47f8-8517-a37405e25d5a" containerID="c20c4ed89a030ac649eb888cf226736a9685d4fc80124332688614dca078860d" exitCode=137 Nov 03 00:57:12.297942 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:57:12.267183 4939 scope.go:122] "RemoveContainer" containerID="c20c4ed89a030ac649eb888cf226736a9685d4fc80124332688614dca078860d" Nov 03 00:57:12.297942 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:57:12.267351 4939 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7034-65817a" Nov 03 00:57:12.566174 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:57:12.566138 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 00:57:12.575674 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:57:12.575644 4939 scope.go:122] "RemoveContainer" containerID="c20c4ed89a030ac649eb888cf226736a9685d4fc80124332688614dca078860d" Nov 03 00:57:12.576099 ip-10-0-140-107 kubenswrapper[4939]: E1103 00:57:12.576069 4939 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c20c4ed89a030ac649eb888cf226736a9685d4fc80124332688614dca078860d\": container with ID starting with c20c4ed89a030ac649eb888cf226736a9685d4fc80124332688614dca078860d not found: ID does not exist" containerID="c20c4ed89a030ac649eb888cf226736a9685d4fc80124332688614dca078860d" Nov 03 00:57:12.576206 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:57:12.576107 4939 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c20c4ed89a030ac649eb888cf226736a9685d4fc80124332688614dca078860d"} err="failed to get container status \"c20c4ed89a030ac649eb888cf226736a9685d4fc80124332688614dca078860d\": rpc error: code = NotFound desc = could not find container \"c20c4ed89a030ac649eb888cf226736a9685d4fc80124332688614dca078860d\": container with ID starting with c20c4ed89a030ac649eb888cf226736a9685d4fc80124332688614dca078860d not found: ID does not exist" Nov 03 00:57:12.609165 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:57:12.609111 4939 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-api-b-65e5ec-64858685d7-dt99f" podUID="a8ce4315-416e-4663-b9bc-d4ef6714c4c1" containerName="payload-processing" containerID="cri-o://b3b6e002ba5578fa145a1553b8bc939e69df122e1fc3e74390a6438019f58824" gracePeriod=25 Nov 03 00:57:12.642803 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:57:12.642745 4939 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-api-b-65e5ec-5f46866b59-knkfz" podUID="bf1852d1-20e7-4e98-be2a-bc444c04c938" containerName="payload-pre-processing" containerID="cri-o://622f3e9b4f23392f14772534a1d9f8dbb64a963364c339df4c0acc5a35890132" gracePeriod=25 Nov 03 00:57:12.963621 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:57:12.963592 4939 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-65e5ec-64858685d7-dt99f" Nov 03 00:57:12.994631 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:57:12.994607 4939 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-65e5ec-5f46866b59-knkfz" Nov 03 00:57:13.146780 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:57:13.146747 4939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bf1852d1-20e7-4e98-be2a-bc444c04c938-kube-api-access-2gx95\" (UniqueName: \"kubernetes.io/projected/bf1852d1-20e7-4e98-be2a-bc444c04c938-kube-api-access-2gx95\") pod \"bf1852d1-20e7-4e98-be2a-bc444c04c938\" (UID: \"bf1852d1-20e7-4e98-be2a-bc444c04c938\") " Nov 03 00:57:13.146956 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:57:13.146857 4939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bf1852d1-20e7-4e98-be2a-bc444c04c938-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bf1852d1-20e7-4e98-be2a-bc444c04c938-plugins-config-volume\") pod \"bf1852d1-20e7-4e98-be2a-bc444c04c938\" (UID: \"bf1852d1-20e7-4e98-be2a-bc444c04c938\") " Nov 03 00:57:13.146956 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:57:13.146885 4939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a8ce4315-416e-4663-b9bc-d4ef6714c4c1-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a8ce4315-416e-4663-b9bc-d4ef6714c4c1-plugins-config-volume\") pod \"a8ce4315-416e-4663-b9bc-d4ef6714c4c1\" (UID: \"a8ce4315-416e-4663-b9bc-d4ef6714c4c1\") " Nov 03 00:57:13.147113 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:57:13.147049 4939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a8ce4315-416e-4663-b9bc-d4ef6714c4c1-kube-api-access-zq2pw\" (UniqueName: \"kubernetes.io/projected/a8ce4315-416e-4663-b9bc-d4ef6714c4c1-kube-api-access-zq2pw\") pod \"a8ce4315-416e-4663-b9bc-d4ef6714c4c1\" (UID: \"a8ce4315-416e-4663-b9bc-d4ef6714c4c1\") " Nov 03 00:57:13.147315 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:57:13.147281 4939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bf1852d1-20e7-4e98-be2a-bc444c04c938-plugins-config-volume" pod "bf1852d1-20e7-4e98-be2a-bc444c04c938" (UID: "bf1852d1-20e7-4e98-be2a-bc444c04c938"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 00:57:13.147450 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:57:13.147357 4939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a8ce4315-416e-4663-b9bc-d4ef6714c4c1-plugins-config-volume" pod "a8ce4315-416e-4663-b9bc-d4ef6714c4c1" (UID: "a8ce4315-416e-4663-b9bc-d4ef6714c4c1"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 00:57:13.147885 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:57:13.147860 4939 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bf1852d1-20e7-4e98-be2a-bc444c04c938-plugins-config-volume\") on node \"ip-10-0-140-107.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 00:57:13.148024 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:57:13.147890 4939 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a8ce4315-416e-4663-b9bc-d4ef6714c4c1-plugins-config-volume\") on node \"ip-10-0-140-107.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 00:57:13.149143 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:57:13.149120 4939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a8ce4315-416e-4663-b9bc-d4ef6714c4c1-kube-api-access-zq2pw" pod "a8ce4315-416e-4663-b9bc-d4ef6714c4c1" (UID: "a8ce4315-416e-4663-b9bc-d4ef6714c4c1"). InnerVolumeSpecName "kube-api-access-zq2pw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 00:57:13.149219 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:57:13.149146 4939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bf1852d1-20e7-4e98-be2a-bc444c04c938-kube-api-access-2gx95" pod "bf1852d1-20e7-4e98-be2a-bc444c04c938" (UID: "bf1852d1-20e7-4e98-be2a-bc444c04c938"). InnerVolumeSpecName "kube-api-access-2gx95". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 00:57:13.249495 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:57:13.249460 4939 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2gx95\" (UniqueName: \"kubernetes.io/projected/bf1852d1-20e7-4e98-be2a-bc444c04c938-kube-api-access-2gx95\") on node \"ip-10-0-140-107.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 00:57:13.249495 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:57:13.249491 4939 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zq2pw\" (UniqueName: \"kubernetes.io/projected/a8ce4315-416e-4663-b9bc-d4ef6714c4c1-kube-api-access-zq2pw\") on node \"ip-10-0-140-107.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 00:57:13.275538 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:57:13.275502 4939 generic.go:356] "Generic (PLEG): container finished" podID="bf1852d1-20e7-4e98-be2a-bc444c04c938" containerID="622f3e9b4f23392f14772534a1d9f8dbb64a963364c339df4c0acc5a35890132" exitCode=0 Nov 03 00:57:13.275714 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:57:13.275552 4939 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-65e5ec-5f46866b59-knkfz" Nov 03 00:57:13.275714 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:57:13.275591 4939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-65e5ec-5f46866b59-knkfz" event={"ID":"bf1852d1-20e7-4e98-be2a-bc444c04c938","Type":"ContainerDied","Data":"622f3e9b4f23392f14772534a1d9f8dbb64a963364c339df4c0acc5a35890132"} Nov 03 00:57:13.275714 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:57:13.275643 4939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-65e5ec-5f46866b59-knkfz" event={"ID":"bf1852d1-20e7-4e98-be2a-bc444c04c938","Type":"ContainerDied","Data":"5832b194f3a8bdf99aedfd644b7391820bdb7ed0966de7ad4af042fbaad70e79"} Nov 03 00:57:13.275714 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:57:13.275672 4939 scope.go:122] "RemoveContainer" containerID="622f3e9b4f23392f14772534a1d9f8dbb64a963364c339df4c0acc5a35890132" Nov 03 00:57:13.277314 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:57:13.277292 4939 generic.go:356] "Generic (PLEG): container finished" podID="a8ce4315-416e-4663-b9bc-d4ef6714c4c1" containerID="b3b6e002ba5578fa145a1553b8bc939e69df122e1fc3e74390a6438019f58824" exitCode=0 Nov 03 00:57:13.277425 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:57:13.277352 4939 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-65e5ec-64858685d7-dt99f" Nov 03 00:57:13.277488 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:57:13.277362 4939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-65e5ec-64858685d7-dt99f" event={"ID":"a8ce4315-416e-4663-b9bc-d4ef6714c4c1","Type":"ContainerDied","Data":"b3b6e002ba5578fa145a1553b8bc939e69df122e1fc3e74390a6438019f58824"} Nov 03 00:57:13.277561 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:57:13.277510 4939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-65e5ec-64858685d7-dt99f" event={"ID":"a8ce4315-416e-4663-b9bc-d4ef6714c4c1","Type":"ContainerDied","Data":"1294a358a4dff4883fbb7d4af5e5c1cfc5b81da0d1224b059a7bc03ead6f00a0"} Nov 03 00:57:13.287532 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:57:13.287514 4939 scope.go:122] "RemoveContainer" containerID="622f3e9b4f23392f14772534a1d9f8dbb64a963364c339df4c0acc5a35890132" Nov 03 00:57:13.287829 ip-10-0-140-107 kubenswrapper[4939]: E1103 00:57:13.287802 4939 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"622f3e9b4f23392f14772534a1d9f8dbb64a963364c339df4c0acc5a35890132\": container with ID starting with 622f3e9b4f23392f14772534a1d9f8dbb64a963364c339df4c0acc5a35890132 not found: ID does not exist" containerID="622f3e9b4f23392f14772534a1d9f8dbb64a963364c339df4c0acc5a35890132" Nov 03 00:57:13.287938 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:57:13.287845 4939 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"622f3e9b4f23392f14772534a1d9f8dbb64a963364c339df4c0acc5a35890132"} err="failed to get container status \"622f3e9b4f23392f14772534a1d9f8dbb64a963364c339df4c0acc5a35890132\": rpc error: code = NotFound desc = could not find container \"622f3e9b4f23392f14772534a1d9f8dbb64a963364c339df4c0acc5a35890132\": container with ID starting with 622f3e9b4f23392f14772534a1d9f8dbb64a963364c339df4c0acc5a35890132 not found: ID does not exist" Nov 03 00:57:13.287938 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:57:13.287870 4939 scope.go:122] "RemoveContainer" containerID="b3b6e002ba5578fa145a1553b8bc939e69df122e1fc3e74390a6438019f58824" Nov 03 00:57:13.298149 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:57:13.298129 4939 scope.go:122] "RemoveContainer" containerID="b3b6e002ba5578fa145a1553b8bc939e69df122e1fc3e74390a6438019f58824" Nov 03 00:57:13.298541 ip-10-0-140-107 kubenswrapper[4939]: E1103 00:57:13.298520 4939 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b3b6e002ba5578fa145a1553b8bc939e69df122e1fc3e74390a6438019f58824\": container with ID starting with b3b6e002ba5578fa145a1553b8bc939e69df122e1fc3e74390a6438019f58824 not found: ID does not exist" containerID="b3b6e002ba5578fa145a1553b8bc939e69df122e1fc3e74390a6438019f58824" Nov 03 00:57:13.298624 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:57:13.298549 4939 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b3b6e002ba5578fa145a1553b8bc939e69df122e1fc3e74390a6438019f58824"} err="failed to get container status \"b3b6e002ba5578fa145a1553b8bc939e69df122e1fc3e74390a6438019f58824\": rpc error: code = NotFound desc = could not find container \"b3b6e002ba5578fa145a1553b8bc939e69df122e1fc3e74390a6438019f58824\": container with ID starting with b3b6e002ba5578fa145a1553b8bc939e69df122e1fc3e74390a6438019f58824 not found: ID does not exist" Nov 03 00:57:13.309178 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:57:13.309155 4939 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-65e5ec-5f46866b59-knkfz"] Nov 03 00:57:13.315050 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:57:13.315028 4939 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-65e5ec-5f46866b59-knkfz"] Nov 03 00:57:13.361934 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:57:13.354816 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 00:57:13.369955 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:57:13.363977 4939 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-65e5ec-64858685d7-dt99f"] Nov 03 00:57:13.388816 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:57:13.386526 4939 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-65e5ec-64858685d7-dt99f"] Nov 03 00:57:13.545214 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:57:13.545126 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 00:57:13.593169 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:57:13.593138 4939 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-54df68-5d998485f9-pknfz"] Nov 03 00:57:13.593824 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:57:13.593808 4939 cpu_manager.go:414] "containerMap: removing container" podUID="a8ce4315-416e-4663-b9bc-d4ef6714c4c1" containerName="payload-processing" Nov 03 00:57:13.593884 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:57:13.593827 4939 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a8ce4315-416e-4663-b9bc-d4ef6714c4c1" containerName="payload-processing" Nov 03 00:57:13.593884 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:57:13.593838 4939 cpu_manager.go:414] "containerMap: removing container" podUID="bf1852d1-20e7-4e98-be2a-bc444c04c938" containerName="payload-pre-processing" Nov 03 00:57:13.593884 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:57:13.593843 4939 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bf1852d1-20e7-4e98-be2a-bc444c04c938" containerName="payload-pre-processing" Nov 03 00:57:13.593884 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:57:13.593856 4939 cpu_manager.go:414] "containerMap: removing container" podUID="cb6e4163-f7ee-47f8-8517-a37405e25d5a" containerName="test-curl-7034-65817a" Nov 03 00:57:13.593884 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:57:13.593861 4939 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cb6e4163-f7ee-47f8-8517-a37405e25d5a" containerName="test-curl-7034-65817a" Nov 03 00:57:13.594041 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:57:13.593963 4939 memory_manager.go:365] "RemoveStaleState removing state" podUID="bf1852d1-20e7-4e98-be2a-bc444c04c938" containerName="payload-pre-processing" Nov 03 00:57:13.594041 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:57:13.593971 4939 memory_manager.go:365] "RemoveStaleState removing state" podUID="cb6e4163-f7ee-47f8-8517-a37405e25d5a" containerName="test-curl-7034-65817a" Nov 03 00:57:13.594041 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:57:13.593988 4939 memory_manager.go:365] "RemoveStaleState removing state" podUID="a8ce4315-416e-4663-b9bc-d4ef6714c4c1" containerName="payload-processing" Nov 03 00:57:13.599365 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:57:13.599347 4939 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-54df68-5d998485f9-pknfz" Nov 03 00:57:13.606141 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:57:13.606118 4939 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-a-54df68\"" Nov 03 00:57:13.606714 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:57:13.606694 4939 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-a-54df68-dockercfg-2xhd6\"" Nov 03 00:57:13.610369 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:57:13.610342 4939 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-54df68-5d998485f9-pknfz"] Nov 03 00:57:13.665975 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:57:13.665947 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 00:57:13.822981 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:57:13.805569 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8rjnv\" (UniqueName: \"kubernetes.io/projected/fbcbf75f-badb-46a6-9f2d-3204afc53907-kube-api-access-8rjnv\") pod \"payload-pre-processing-e2e-api-a-54df68-5d998485f9-pknfz\" (UID: \"fbcbf75f-badb-46a6-9f2d-3204afc53907\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-54df68-5d998485f9-pknfz" Nov 03 00:57:13.830935 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:57:13.823368 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fbcbf75f-badb-46a6-9f2d-3204afc53907-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-54df68-5d998485f9-pknfz\" (UID: \"fbcbf75f-badb-46a6-9f2d-3204afc53907\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-54df68-5d998485f9-pknfz" Nov 03 00:57:13.987545 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:57:13.971338 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8rjnv\" (UniqueName: \"kubernetes.io/projected/fbcbf75f-badb-46a6-9f2d-3204afc53907-kube-api-access-8rjnv\") pod \"payload-pre-processing-e2e-api-a-54df68-5d998485f9-pknfz\" (UID: \"fbcbf75f-badb-46a6-9f2d-3204afc53907\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-54df68-5d998485f9-pknfz" Nov 03 00:57:13.987545 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:57:13.971546 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fbcbf75f-badb-46a6-9f2d-3204afc53907-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-54df68-5d998485f9-pknfz\" (UID: \"fbcbf75f-badb-46a6-9f2d-3204afc53907\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-54df68-5d998485f9-pknfz" Nov 03 00:57:13.987545 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:57:13.973790 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fbcbf75f-badb-46a6-9f2d-3204afc53907-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-54df68-5d998485f9-pknfz\" (UID: \"fbcbf75f-badb-46a6-9f2d-3204afc53907\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-54df68-5d998485f9-pknfz" Nov 03 00:57:13.991560 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:57:13.991529 4939 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a8ce4315-416e-4663-b9bc-d4ef6714c4c1" path="/var/lib/kubelet/pods/a8ce4315-416e-4663-b9bc-d4ef6714c4c1/volumes" Nov 03 00:57:13.992159 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:57:13.992139 4939 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bf1852d1-20e7-4e98-be2a-bc444c04c938" path="/var/lib/kubelet/pods/bf1852d1-20e7-4e98-be2a-bc444c04c938/volumes" Nov 03 00:57:14.017420 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:57:14.017383 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8rjnv\" (UniqueName: \"kubernetes.io/projected/fbcbf75f-badb-46a6-9f2d-3204afc53907-kube-api-access-8rjnv\") pod \"payload-pre-processing-e2e-api-a-54df68-5d998485f9-pknfz\" (UID: \"fbcbf75f-badb-46a6-9f2d-3204afc53907\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-54df68-5d998485f9-pknfz" Nov 03 00:57:14.218383 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:57:14.218349 4939 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-54df68-5d998485f9-pknfz" Nov 03 00:57:14.389370 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:57:14.389338 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 00:57:14.522426 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:57:14.522396 4939 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-54df68-5d998485f9-pknfz"] Nov 03 00:57:14.523260 ip-10-0-140-107 kubenswrapper[4939]: W1103 00:57:14.523237 4939 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfbcbf75f_badb_46a6_9f2d_3204afc53907.slice/crio-9161e814ac4c38448cb4bdb64ef5a0d7c237905131aec2547016b5b41c2535cf WatchSource:0}: Error finding container 9161e814ac4c38448cb4bdb64ef5a0d7c237905131aec2547016b5b41c2535cf: Status 404 returned error can't find the container with id 9161e814ac4c38448cb4bdb64ef5a0d7c237905131aec2547016b5b41c2535cf Nov 03 00:57:15.152053 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:57:15.152021 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 00:57:15.279941 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:57:15.277760 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 00:57:15.314360 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:57:15.314050 4939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-54df68-5d998485f9-pknfz" event={"ID":"fbcbf75f-badb-46a6-9f2d-3204afc53907","Type":"ContainerStarted","Data":"eefb2c57c0c16e9c524aa60947ee2e2410a8e592f6e7634894e70f3ffaa06169"} Nov 03 00:57:15.314360 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:57:15.314119 4939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-54df68-5d998485f9-pknfz" event={"ID":"fbcbf75f-badb-46a6-9f2d-3204afc53907","Type":"ContainerStarted","Data":"9161e814ac4c38448cb4bdb64ef5a0d7c237905131aec2547016b5b41c2535cf"} Nov 03 00:57:15.317478 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:57:15.315231 4939 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-api-a-54df68-5d998485f9-pknfz" Nov 03 00:57:15.342053 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:57:15.341995 4939 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-api-a-54df68-5d998485f9-pknfz" podStartSLOduration=2.341978637 podStartE2EDuration="2.341978637s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:57:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:57:15.3364522 +0000 UTC m=+983.600787149" watchObservedRunningTime="2026-11-03 00:57:15.341978637 +0000 UTC m=+983.606313582" Nov 03 00:57:15.682772 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:57:15.682742 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 00:57:16.106765 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:57:16.106711 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 00:57:16.276942 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:57:16.273570 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 00:57:16.950425 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:57:16.950387 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 00:57:17.585085 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:57:17.585044 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 00:57:18.025821 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:57:18.025723 4939 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f9962d74-ln86t"] Nov 03 00:57:18.029268 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:57:18.029238 4939 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f9962d74-ln86t"] Nov 03 00:57:18.033416 ip-10-0-140-107 kubenswrapper[4939]: E1103 00:57:18.033388 4939 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-sthjh" containerName="extractor" Nov 03 00:57:18.307060 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:57:18.306978 4939 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-54df68-64c7497868-srpmr"] Nov 03 00:57:18.313463 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:57:18.313438 4939 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-54df68-64c7497868-srpmr" Nov 03 00:57:18.318242 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:57:18.318214 4939 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-54df68-64c7497868-srpmr"] Nov 03 00:57:18.322978 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:57:18.322119 4939 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-a-54df68-serving-cert\"" Nov 03 00:57:18.382057 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:57:18.382012 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 00:57:18.390635 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:57:18.390605 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ad1a1a5c-cb63-49e9-af61-26a6cd45cb8a-openshift-service-ca\") pod \"maas-api-e2e-api-a-54df68-64c7497868-srpmr\" (UID: \"ad1a1a5c-cb63-49e9-af61-26a6cd45cb8a\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-54df68-64c7497868-srpmr" Nov 03 00:57:18.390774 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:57:18.390649 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mcgpd\" (UniqueName: \"kubernetes.io/projected/ad1a1a5c-cb63-49e9-af61-26a6cd45cb8a-kube-api-access-mcgpd\") pod \"maas-api-e2e-api-a-54df68-64c7497868-srpmr\" (UID: \"ad1a1a5c-cb63-49e9-af61-26a6cd45cb8a\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-54df68-64c7497868-srpmr" Nov 03 00:57:18.390958 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:57:18.390937 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ad1a1a5c-cb63-49e9-af61-26a6cd45cb8a-maas-api-tls\") pod \"maas-api-e2e-api-a-54df68-64c7497868-srpmr\" (UID: \"ad1a1a5c-cb63-49e9-af61-26a6cd45cb8a\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-54df68-64c7497868-srpmr" Nov 03 00:57:18.391028 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:57:18.390971 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ad1a1a5c-cb63-49e9-af61-26a6cd45cb8a-metrics-certs\") pod \"maas-api-e2e-api-a-54df68-64c7497868-srpmr\" (UID: \"ad1a1a5c-cb63-49e9-af61-26a6cd45cb8a\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-54df68-64c7497868-srpmr" Nov 03 00:57:18.508930 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:57:18.499530 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 00:57:18.509423 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:57:18.499727 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ad1a1a5c-cb63-49e9-af61-26a6cd45cb8a-maas-api-tls\") pod \"maas-api-e2e-api-a-54df68-64c7497868-srpmr\" (UID: \"ad1a1a5c-cb63-49e9-af61-26a6cd45cb8a\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-54df68-64c7497868-srpmr" Nov 03 00:57:18.509667 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:57:18.509644 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ad1a1a5c-cb63-49e9-af61-26a6cd45cb8a-metrics-certs\") pod \"maas-api-e2e-api-a-54df68-64c7497868-srpmr\" (UID: \"ad1a1a5c-cb63-49e9-af61-26a6cd45cb8a\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-54df68-64c7497868-srpmr" Nov 03 00:57:18.510081 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:57:18.510059 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ad1a1a5c-cb63-49e9-af61-26a6cd45cb8a-openshift-service-ca\") pod \"maas-api-e2e-api-a-54df68-64c7497868-srpmr\" (UID: \"ad1a1a5c-cb63-49e9-af61-26a6cd45cb8a\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-54df68-64c7497868-srpmr" Nov 03 00:57:18.510243 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:57:18.510227 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mcgpd\" (UniqueName: \"kubernetes.io/projected/ad1a1a5c-cb63-49e9-af61-26a6cd45cb8a-kube-api-access-mcgpd\") pod \"maas-api-e2e-api-a-54df68-64c7497868-srpmr\" (UID: \"ad1a1a5c-cb63-49e9-af61-26a6cd45cb8a\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-54df68-64c7497868-srpmr" Nov 03 00:57:18.519941 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:57:18.512227 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ad1a1a5c-cb63-49e9-af61-26a6cd45cb8a-openshift-service-ca\") pod \"maas-api-e2e-api-a-54df68-64c7497868-srpmr\" (UID: \"ad1a1a5c-cb63-49e9-af61-26a6cd45cb8a\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-54df68-64c7497868-srpmr" Nov 03 00:57:18.562043 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:57:18.549333 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ad1a1a5c-cb63-49e9-af61-26a6cd45cb8a-metrics-certs\") pod \"maas-api-e2e-api-a-54df68-64c7497868-srpmr\" (UID: \"ad1a1a5c-cb63-49e9-af61-26a6cd45cb8a\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-54df68-64c7497868-srpmr" Nov 03 00:57:18.574964 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:57:18.565559 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ad1a1a5c-cb63-49e9-af61-26a6cd45cb8a-maas-api-tls\") pod \"maas-api-e2e-api-a-54df68-64c7497868-srpmr\" (UID: \"ad1a1a5c-cb63-49e9-af61-26a6cd45cb8a\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-54df68-64c7497868-srpmr" Nov 03 00:57:18.574964 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:57:18.566844 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mcgpd\" (UniqueName: \"kubernetes.io/projected/ad1a1a5c-cb63-49e9-af61-26a6cd45cb8a-kube-api-access-mcgpd\") pod \"maas-api-e2e-api-a-54df68-64c7497868-srpmr\" (UID: \"ad1a1a5c-cb63-49e9-af61-26a6cd45cb8a\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-54df68-64c7497868-srpmr" Nov 03 00:57:18.574964 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:57:18.573449 4939 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-dd03e395-b8464b49-p8872"] Nov 03 00:57:18.599938 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:57:18.597487 4939 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-dd03e395-779465d96c-tckp6"] Nov 03 00:57:18.639286 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:57:18.638315 4939 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-54df68-64c7497868-srpmr" Nov 03 00:57:18.971536 ip-10-0-140-107 kubenswrapper[4939]: W1103 00:57:18.971490 4939 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podad1a1a5c_cb63_49e9_af61_26a6cd45cb8a.slice/crio-f427c42b7a50a898a08fd3090cbd4f50d4061153a903d0d642ee626ea341037c WatchSource:0}: Error finding container f427c42b7a50a898a08fd3090cbd4f50d4061153a903d0d642ee626ea341037c: Status 404 returned error can't find the container with id f427c42b7a50a898a08fd3090cbd4f50d4061153a903d0d642ee626ea341037c Nov 03 00:57:18.971701 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:57:18.971570 4939 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-54df68-64c7497868-srpmr"] Nov 03 00:57:18.973229 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:57:18.973209 4939 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Nov 03 00:57:19.369885 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:57:19.369845 4939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-54df68-64c7497868-srpmr" event={"ID":"ad1a1a5c-cb63-49e9-af61-26a6cd45cb8a","Type":"ContainerStarted","Data":"f427c42b7a50a898a08fd3090cbd4f50d4061153a903d0d642ee626ea341037c"} Nov 03 00:57:19.870713 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:57:19.870681 4939 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="84a1ac77-cd42-4963-b8a0-b941878e5c10" path="/var/lib/kubelet/pods/84a1ac77-cd42-4963-b8a0-b941878e5c10/volumes" Nov 03 00:57:20.378296 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:57:20.378257 4939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-54df68-64c7497868-srpmr" event={"ID":"ad1a1a5c-cb63-49e9-af61-26a6cd45cb8a","Type":"ContainerStarted","Data":"a8fcc422f8ec8b907eeda9f408123c3e9d30cda568707cbe3414b0afcf6e59ff"} Nov 03 00:57:20.378461 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:57:20.378348 4939 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-54df68-64c7497868-srpmr" Nov 03 00:57:20.396223 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:57:20.396172 4939 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-54df68-64c7497868-srpmr" podStartSLOduration=1.833717643 podStartE2EDuration="2.396158476s" totalImagesPullingTime="562.440833ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:57:18 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:57:20.39508218 +0000 UTC m=+988.659417129" watchObservedRunningTime="2026-11-03 00:57:20.396158476 +0000 UTC m=+988.660493421" Nov 03 00:57:20.433779 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:57:20.433752 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 00:57:20.911398 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:57:20.911360 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 00:57:21.396485 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:57:21.396451 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 00:57:22.098704 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:57:22.098661 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 00:57:22.786457 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:57:22.786356 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 00:57:23.499740 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:57:23.499687 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 00:57:23.582931 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:57:23.578527 4939 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-mt-dd03e395-b8464b49-p8872" podUID="dfcf5830-ce86-40fb-bdfb-3564175ac08a" containerName="payload-processing" containerID="cri-o://f10b27fe436ff74ddfd71c25ff4c67ea2f10276544f729b75b518d51d4cd4930" gracePeriod=25 Nov 03 00:57:23.609932 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:57:23.607892 4939 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-mt-dd03e395-779465d96c-tckp6" podUID="f9c818e4-813f-40dd-9311-906a517eb42a" containerName="payload-pre-processing" containerID="cri-o://fd6221e9ac6ab31991e76723f19e79802ad1765ded99357a8dcf8ee7c0bd3ae0" gracePeriod=25 Nov 03 00:57:23.856798 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:57:23.856759 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 00:57:24.075732 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:57:24.075694 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 00:57:24.771942 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:57:24.771036 4939 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-4pw7v"] Nov 03 00:57:24.796995 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:57:24.795796 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 00:57:25.364999 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:57:25.364954 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 00:57:26.289102 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:57:26.289048 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 00:57:26.410134 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:57:26.410100 4939 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-54df68-64c7497868-srpmr" Nov 03 00:57:27.390114 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:57:27.390079 4939 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-api-a-54df68-5d998485f9-pknfz" Nov 03 00:57:27.398336 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:57:27.398291 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 00:57:27.967936 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:57:27.964042 4939 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-f7680d/distinct-w2-f7680d-kserve-756f874b98-qnpv8" podUID="836753fc-f9ef-4f61-b3fe-e5b92b7ee27b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.70:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 00:57:28.186934 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:57:28.186000 4939 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-88bbf7/simulator-w1-88bbf7-kserve-5fbb669f8f-42hgd" podUID="83b38883-2e7a-444d-aa68-52be4a25ae66" containerName="main" probeResult="failure" output="Get \"https://10.132.0.62:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 00:57:28.300072 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:57:28.299977 4939 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-2f9a3a/distinct-w0-2f9a3a-kserve-555565cfcf-jtcr9" podUID="ac0ece06-8718-463f-915f-5315988d9711" containerName="main" probeResult="failure" output="Get \"https://10.132.0.76:8000/ready\": context deadline exceeded" Nov 03 00:57:28.691088 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:57:28.691034 4939 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-83755a/distinct-2-w3-83755a-kserve-6cf66b995d-2l47b" podUID="c3b05e88-5d5b-4d03-b28d-832f22e3c868" containerName="main" probeResult="failure" output="Get \"https://10.132.0.72:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 00:57:28.695316 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:57:28.691768 4939 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-2f9a3a/simulator-w0-2f9a3a-kserve-9596bbb8f-2trgz" podUID="207e2207-fb6c-4076-95df-72e95671c261" containerName="main" probeResult="failure" output="Get \"https://10.132.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 00:57:28.916490 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:57:28.916452 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 00:57:29.172039 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:57:29.171994 4939 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-mt-dd03e395-b8464b49-p8872" podUID="dfcf5830-ce86-40fb-bdfb-3564175ac08a" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 03 00:57:29.176791 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:57:29.176739 4939 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-mt-dd03e395-779465d96c-tckp6" podUID="f9c818e4-813f-40dd-9311-906a517eb42a" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 03 00:57:29.724235 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:57:29.724194 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 00:57:30.653830 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:57:30.653789 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 00:57:31.405204 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:57:31.405164 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 00:57:32.178626 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:57:32.178597 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 00:57:32.295487 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:57:32.295450 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 00:57:32.798444 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:57:32.798411 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 00:57:32.915616 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:57:32.915580 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 00:57:33.214892 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:57:33.214858 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 00:57:33.607350 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:57:33.607318 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 00:57:33.791959 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:57:33.770656 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 00:57:33.943266 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:57:33.938794 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 00:57:34.098331 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:57:34.098302 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 00:57:34.202048 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:57:34.201957 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 00:57:35.198213 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:57:35.198185 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 00:57:36.040104 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:57:36.040058 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 00:57:36.695378 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:57:36.695341 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 00:57:36.856660 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:57:36.856618 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 00:57:37.592130 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:57:37.592099 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 00:57:37.932885 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:57:37.932804 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 00:57:37.935825 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:57:37.935800 4939 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-a2d8e5-5bb4464ff9-fpt8c"] Nov 03 00:57:37.941708 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:57:37.941687 4939 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-a2d8e5-5bb4464ff9-fpt8c" Nov 03 00:57:37.953528 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:57:37.953505 4939 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-b-a2d8e5\"" Nov 03 00:57:37.957859 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:57:37.957836 4939 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-a2d8e5-5bb4464ff9-fpt8c"] Nov 03 00:57:38.051319 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:57:38.051281 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 00:57:38.130644 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:57:38.130608 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/916506b5-c34f-479e-a192-1a4c2e5714ee-plugins-config-volume\") pod \"payload-processing-e2e-api-b-a2d8e5-5bb4464ff9-fpt8c\" (UID: \"916506b5-c34f-479e-a192-1a4c2e5714ee\") " pod="openshift-ingress/payload-processing-e2e-api-b-a2d8e5-5bb4464ff9-fpt8c" Nov 03 00:57:38.131088 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:57:38.131064 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mjq48\" (UniqueName: \"kubernetes.io/projected/916506b5-c34f-479e-a192-1a4c2e5714ee-kube-api-access-mjq48\") pod \"payload-processing-e2e-api-b-a2d8e5-5bb4464ff9-fpt8c\" (UID: \"916506b5-c34f-479e-a192-1a4c2e5714ee\") " pod="openshift-ingress/payload-processing-e2e-api-b-a2d8e5-5bb4464ff9-fpt8c" Nov 03 00:57:38.341503 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:57:38.341463 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/916506b5-c34f-479e-a192-1a4c2e5714ee-plugins-config-volume\") pod \"payload-processing-e2e-api-b-a2d8e5-5bb4464ff9-fpt8c\" (UID: \"916506b5-c34f-479e-a192-1a4c2e5714ee\") " pod="openshift-ingress/payload-processing-e2e-api-b-a2d8e5-5bb4464ff9-fpt8c" Nov 03 00:57:38.341983 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:57:38.341955 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mjq48\" (UniqueName: \"kubernetes.io/projected/916506b5-c34f-479e-a192-1a4c2e5714ee-kube-api-access-mjq48\") pod \"payload-processing-e2e-api-b-a2d8e5-5bb4464ff9-fpt8c\" (UID: \"916506b5-c34f-479e-a192-1a4c2e5714ee\") " pod="openshift-ingress/payload-processing-e2e-api-b-a2d8e5-5bb4464ff9-fpt8c" Nov 03 00:57:38.344335 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:57:38.344311 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/916506b5-c34f-479e-a192-1a4c2e5714ee-plugins-config-volume\") pod \"payload-processing-e2e-api-b-a2d8e5-5bb4464ff9-fpt8c\" (UID: \"916506b5-c34f-479e-a192-1a4c2e5714ee\") " pod="openshift-ingress/payload-processing-e2e-api-b-a2d8e5-5bb4464ff9-fpt8c" Nov 03 00:57:38.359975 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:57:38.359942 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mjq48\" (UniqueName: \"kubernetes.io/projected/916506b5-c34f-479e-a192-1a4c2e5714ee-kube-api-access-mjq48\") pod \"payload-processing-e2e-api-b-a2d8e5-5bb4464ff9-fpt8c\" (UID: \"916506b5-c34f-479e-a192-1a4c2e5714ee\") " pod="openshift-ingress/payload-processing-e2e-api-b-a2d8e5-5bb4464ff9-fpt8c" Nov 03 00:57:38.569149 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:57:38.569117 4939 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-a2d8e5-5bb4464ff9-fpt8c" Nov 03 00:57:38.583542 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:57:38.583485 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 00:57:38.938817 ip-10-0-140-107 kubenswrapper[4939]: W1103 00:57:38.938784 4939 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod916506b5_c34f_479e_a192_1a4c2e5714ee.slice/crio-4bd2739b9b5a129e58e81eecdc29b43f0ba295f2f1333c55058ba214472b74e1 WatchSource:0}: Error finding container 4bd2739b9b5a129e58e81eecdc29b43f0ba295f2f1333c55058ba214472b74e1: Status 404 returned error can't find the container with id 4bd2739b9b5a129e58e81eecdc29b43f0ba295f2f1333c55058ba214472b74e1 Nov 03 00:57:38.939273 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:57:38.939252 4939 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-a2d8e5-5bb4464ff9-fpt8c"] Nov 03 00:57:39.131463 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:57:39.131416 4939 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-mt-dd03e395-779465d96c-tckp6" podUID="f9c818e4-813f-40dd-9311-906a517eb42a" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 03 00:57:39.131635 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:57:39.131477 4939 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-mt-dd03e395-b8464b49-p8872" podUID="dfcf5830-ce86-40fb-bdfb-3564175ac08a" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 03 00:57:39.166209 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:57:39.166166 4939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-a2d8e5-5bb4464ff9-fpt8c" event={"ID":"916506b5-c34f-479e-a192-1a4c2e5714ee","Type":"ContainerStarted","Data":"e8eb3e2eb562ca97580a86e3b1f1029dc27471fca9c41e286f494aab12aee0a0"} Nov 03 00:57:39.166400 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:57:39.166236 4939 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-api-b-a2d8e5-5bb4464ff9-fpt8c" Nov 03 00:57:39.166400 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:57:39.166262 4939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-a2d8e5-5bb4464ff9-fpt8c" event={"ID":"916506b5-c34f-479e-a192-1a4c2e5714ee","Type":"ContainerStarted","Data":"4bd2739b9b5a129e58e81eecdc29b43f0ba295f2f1333c55058ba214472b74e1"} Nov 03 00:57:39.185449 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:57:39.185401 4939 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-api-b-a2d8e5-5bb4464ff9-fpt8c" podStartSLOduration=2.18538535 podStartE2EDuration="2.18538535s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:57:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:57:39.183994366 +0000 UTC m=+1007.448329311" watchObservedRunningTime="2026-11-03 00:57:39.18538535 +0000 UTC m=+1007.449720295" Nov 03 00:57:39.288887 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:57:39.288803 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 00:57:40.034685 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:57:40.034648 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 00:57:40.866132 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:57:40.866096 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 00:57:41.008935 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:57:41.003052 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 00:57:41.208982 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:57:41.208893 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 00:57:41.502205 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:57:41.502131 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 00:57:41.503034 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:57:41.503013 4939 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-dd03e395-b8464b49-7rlqj"] Nov 03 00:57:41.510299 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:57:41.510276 4939 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-dd03e395-b8464b49-7rlqj" Nov 03 00:57:41.520400 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:57:41.520365 4939 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-dd03e395-b8464b49-7rlqj"] Nov 03 00:57:41.630563 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:57:41.630532 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 00:57:41.645615 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:57:41.645581 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z8sfd\" (UniqueName: \"kubernetes.io/projected/dbe5252d-5569-4e97-8ede-e8943326dcd8-kube-api-access-z8sfd\") pod \"payload-processing-e2e-mt-dd03e395-b8464b49-7rlqj\" (UID: \"dbe5252d-5569-4e97-8ede-e8943326dcd8\") " pod="openshift-ingress/payload-processing-e2e-mt-dd03e395-b8464b49-7rlqj" Nov 03 00:57:41.645856 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:57:41.645835 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dbe5252d-5569-4e97-8ede-e8943326dcd8-plugins-config-volume\") pod \"payload-processing-e2e-mt-dd03e395-b8464b49-7rlqj\" (UID: \"dbe5252d-5569-4e97-8ede-e8943326dcd8\") " pod="openshift-ingress/payload-processing-e2e-mt-dd03e395-b8464b49-7rlqj" Nov 03 00:57:41.743049 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:57:41.742995 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 00:57:41.748042 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:57:41.748005 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z8sfd\" (UniqueName: \"kubernetes.io/projected/dbe5252d-5569-4e97-8ede-e8943326dcd8-kube-api-access-z8sfd\") pod \"payload-processing-e2e-mt-dd03e395-b8464b49-7rlqj\" (UID: \"dbe5252d-5569-4e97-8ede-e8943326dcd8\") " pod="openshift-ingress/payload-processing-e2e-mt-dd03e395-b8464b49-7rlqj" Nov 03 00:57:41.749184 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:57:41.749125 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dbe5252d-5569-4e97-8ede-e8943326dcd8-plugins-config-volume\") pod \"payload-processing-e2e-mt-dd03e395-b8464b49-7rlqj\" (UID: \"dbe5252d-5569-4e97-8ede-e8943326dcd8\") " pod="openshift-ingress/payload-processing-e2e-mt-dd03e395-b8464b49-7rlqj" Nov 03 00:57:41.762221 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:57:41.750778 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dbe5252d-5569-4e97-8ede-e8943326dcd8-plugins-config-volume\") pod \"payload-processing-e2e-mt-dd03e395-b8464b49-7rlqj\" (UID: \"dbe5252d-5569-4e97-8ede-e8943326dcd8\") " pod="openshift-ingress/payload-processing-e2e-mt-dd03e395-b8464b49-7rlqj" Nov 03 00:57:41.764952 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:57:41.764861 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-z8sfd\" (UniqueName: \"kubernetes.io/projected/dbe5252d-5569-4e97-8ede-e8943326dcd8-kube-api-access-z8sfd\") pod \"payload-processing-e2e-mt-dd03e395-b8464b49-7rlqj\" (UID: \"dbe5252d-5569-4e97-8ede-e8943326dcd8\") " pod="openshift-ingress/payload-processing-e2e-mt-dd03e395-b8464b49-7rlqj" Nov 03 00:57:41.837672 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:57:41.836183 4939 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-dd03e395-b8464b49-7rlqj" Nov 03 00:57:42.247728 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:57:42.247696 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 00:57:42.313408 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:57:42.313375 4939 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-dd03e395-b8464b49-7rlqj"] Nov 03 00:57:42.314110 ip-10-0-140-107 kubenswrapper[4939]: W1103 00:57:42.314079 4939 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddbe5252d_5569_4e97_8ede_e8943326dcd8.slice/crio-e1f790b22bf9c0f254c67b7197525951a53f1f80c1123513c991d09bfc8b6d4e WatchSource:0}: Error finding container e1f790b22bf9c0f254c67b7197525951a53f1f80c1123513c991d09bfc8b6d4e: Status 404 returned error can't find the container with id e1f790b22bf9c0f254c67b7197525951a53f1f80c1123513c991d09bfc8b6d4e Nov 03 00:57:42.372694 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:57:42.372659 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 00:57:42.490465 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:57:42.490434 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 00:57:43.182955 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:57:43.178868 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 00:57:43.233294 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:57:43.228152 4939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-dd03e395-b8464b49-7rlqj" event={"ID":"dbe5252d-5569-4e97-8ede-e8943326dcd8","Type":"ContainerStarted","Data":"131551666e7f1a0a1a6f9371f1d161aba288e4866587b847f29cf4a9d4cc50b3"} Nov 03 00:57:43.233294 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:57:43.228220 4939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-dd03e395-b8464b49-7rlqj" event={"ID":"dbe5252d-5569-4e97-8ede-e8943326dcd8","Type":"ContainerStarted","Data":"e1f790b22bf9c0f254c67b7197525951a53f1f80c1123513c991d09bfc8b6d4e"} Nov 03 00:57:43.233294 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:57:43.229013 4939 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-mt-dd03e395-b8464b49-7rlqj" Nov 03 00:57:44.450937 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:57:44.443996 4939 generic.go:356] "Generic (PLEG): container finished" podID="dfcf5830-ce86-40fb-bdfb-3564175ac08a" containerID="f10b27fe436ff74ddfd71c25ff4c67ea2f10276544f729b75b518d51d4cd4930" exitCode=0 Nov 03 00:57:44.451815 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:57:44.451127 4939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-dd03e395-b8464b49-p8872" event={"ID":"dfcf5830-ce86-40fb-bdfb-3564175ac08a","Type":"ContainerDied","Data":"f10b27fe436ff74ddfd71c25ff4c67ea2f10276544f729b75b518d51d4cd4930"} Nov 03 00:57:44.520943 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:57:44.506740 4939 generic.go:356] "Generic (PLEG): container finished" podID="f9c818e4-813f-40dd-9311-906a517eb42a" containerID="fd6221e9ac6ab31991e76723f19e79802ad1765ded99357a8dcf8ee7c0bd3ae0" exitCode=0 Nov 03 00:57:44.520943 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:57:44.515127 4939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-dd03e395-779465d96c-tckp6" event={"ID":"f9c818e4-813f-40dd-9311-906a517eb42a","Type":"ContainerDied","Data":"fd6221e9ac6ab31991e76723f19e79802ad1765ded99357a8dcf8ee7c0bd3ae0"} Nov 03 00:57:44.562182 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:57:44.562153 4939 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-dd03e395-b8464b49-p8872" Nov 03 00:57:44.565600 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:57:44.565578 4939 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-dd03e395-779465d96c-tckp6" Nov 03 00:57:44.608666 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:57:44.608613 4939 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-mt-dd03e395-b8464b49-7rlqj" podStartSLOduration=3.608598386 podStartE2EDuration="3.608598386s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:57:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:57:43.269275581 +0000 UTC m=+1011.533610529" watchObservedRunningTime="2026-11-03 00:57:44.608598386 +0000 UTC m=+1012.872933356" Nov 03 00:57:44.624991 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:57:44.624859 4939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/dfcf5830-ce86-40fb-bdfb-3564175ac08a-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dfcf5830-ce86-40fb-bdfb-3564175ac08a-plugins-config-volume\") pod \"dfcf5830-ce86-40fb-bdfb-3564175ac08a\" (UID: \"dfcf5830-ce86-40fb-bdfb-3564175ac08a\") " Nov 03 00:57:44.625142 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:57:44.625014 4939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f9c818e4-813f-40dd-9311-906a517eb42a-kube-api-access-f975j\" (UniqueName: \"kubernetes.io/projected/f9c818e4-813f-40dd-9311-906a517eb42a-kube-api-access-f975j\") pod \"f9c818e4-813f-40dd-9311-906a517eb42a\" (UID: \"f9c818e4-813f-40dd-9311-906a517eb42a\") " Nov 03 00:57:44.625262 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:57:44.625235 4939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dfcf5830-ce86-40fb-bdfb-3564175ac08a-kube-api-access-8btt9\" (UniqueName: \"kubernetes.io/projected/dfcf5830-ce86-40fb-bdfb-3564175ac08a-kube-api-access-8btt9\") pod \"dfcf5830-ce86-40fb-bdfb-3564175ac08a\" (UID: \"dfcf5830-ce86-40fb-bdfb-3564175ac08a\") " Nov 03 00:57:44.625424 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:57:44.625266 4939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dfcf5830-ce86-40fb-bdfb-3564175ac08a-plugins-config-volume" pod "dfcf5830-ce86-40fb-bdfb-3564175ac08a" (UID: "dfcf5830-ce86-40fb-bdfb-3564175ac08a"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 00:57:44.625424 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:57:44.625342 4939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f9c818e4-813f-40dd-9311-906a517eb42a-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f9c818e4-813f-40dd-9311-906a517eb42a-plugins-config-volume\") pod \"f9c818e4-813f-40dd-9311-906a517eb42a\" (UID: \"f9c818e4-813f-40dd-9311-906a517eb42a\") " Nov 03 00:57:44.626192 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:57:44.626168 4939 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dfcf5830-ce86-40fb-bdfb-3564175ac08a-plugins-config-volume\") on node \"ip-10-0-140-107.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 00:57:44.626306 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:57:44.626257 4939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f9c818e4-813f-40dd-9311-906a517eb42a-plugins-config-volume" pod "f9c818e4-813f-40dd-9311-906a517eb42a" (UID: "f9c818e4-813f-40dd-9311-906a517eb42a"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 00:57:44.628171 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:57:44.628144 4939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f9c818e4-813f-40dd-9311-906a517eb42a-kube-api-access-f975j" pod "f9c818e4-813f-40dd-9311-906a517eb42a" (UID: "f9c818e4-813f-40dd-9311-906a517eb42a"). InnerVolumeSpecName "kube-api-access-f975j". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 00:57:44.629067 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:57:44.629042 4939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dfcf5830-ce86-40fb-bdfb-3564175ac08a-kube-api-access-8btt9" pod "dfcf5830-ce86-40fb-bdfb-3564175ac08a" (UID: "dfcf5830-ce86-40fb-bdfb-3564175ac08a"). InnerVolumeSpecName "kube-api-access-8btt9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 00:57:44.728015 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:57:44.727980 4939 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-f975j\" (UniqueName: \"kubernetes.io/projected/f9c818e4-813f-40dd-9311-906a517eb42a-kube-api-access-f975j\") on node \"ip-10-0-140-107.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 00:57:44.728015 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:57:44.728012 4939 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8btt9\" (UniqueName: \"kubernetes.io/projected/dfcf5830-ce86-40fb-bdfb-3564175ac08a-kube-api-access-8btt9\") on node \"ip-10-0-140-107.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 00:57:44.728015 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:57:44.728023 4939 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f9c818e4-813f-40dd-9311-906a517eb42a-plugins-config-volume\") on node \"ip-10-0-140-107.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 00:57:44.863226 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:57:44.863181 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 00:57:45.129614 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:57:45.129576 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 00:57:45.271382 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:57:45.271346 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 00:57:45.515883 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:57:45.515801 4939 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-dd03e395-b8464b49-p8872" Nov 03 00:57:45.516365 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:57:45.515800 4939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-dd03e395-b8464b49-p8872" event={"ID":"dfcf5830-ce86-40fb-bdfb-3564175ac08a","Type":"ContainerDied","Data":"41328b17babe6447b177ce6b242c6f18234a4305f5e5e8e9bee925ee2a55cdeb"} Nov 03 00:57:45.516365 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:57:45.516001 4939 scope.go:122] "RemoveContainer" containerID="f10b27fe436ff74ddfd71c25ff4c67ea2f10276544f729b75b518d51d4cd4930" Nov 03 00:57:45.517559 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:57:45.517538 4939 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-dd03e395-779465d96c-tckp6" Nov 03 00:57:45.517686 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:57:45.517537 4939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-dd03e395-779465d96c-tckp6" event={"ID":"f9c818e4-813f-40dd-9311-906a517eb42a","Type":"ContainerDied","Data":"2d1e4798ddf894d5878c75f4ad1ced70419275649d2105d63a80858ba03f8d9f"} Nov 03 00:57:45.530822 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:57:45.530801 4939 scope.go:122] "RemoveContainer" containerID="fd6221e9ac6ab31991e76723f19e79802ad1765ded99357a8dcf8ee7c0bd3ae0" Nov 03 00:57:45.546950 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:57:45.546890 4939 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-dd03e395-779465d96c-tckp6"] Nov 03 00:57:45.550162 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:57:45.550133 4939 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-dd03e395-779465d96c-tckp6"] Nov 03 00:57:45.557189 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:57:45.557169 4939 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-dd03e395-b8464b49-p8872"] Nov 03 00:57:45.561037 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:57:45.561018 4939 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-dd03e395-b8464b49-p8872"] Nov 03 00:57:45.717704 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:57:45.717670 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 00:57:45.868687 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:57:45.868654 4939 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="dfcf5830-ce86-40fb-bdfb-3564175ac08a" path="/var/lib/kubelet/pods/dfcf5830-ce86-40fb-bdfb-3564175ac08a/volumes" Nov 03 00:57:45.869169 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:57:45.869152 4939 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f9c818e4-813f-40dd-9311-906a517eb42a" path="/var/lib/kubelet/pods/f9c818e4-813f-40dd-9311-906a517eb42a/volumes" Nov 03 00:57:46.318006 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:57:46.317871 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 00:57:46.356435 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:57:46.356372 4939 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-340b96-86744887cf-swn7k"] Nov 03 00:57:46.357229 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:57:46.357211 4939 cpu_manager.go:414] "containerMap: removing container" podUID="dfcf5830-ce86-40fb-bdfb-3564175ac08a" containerName="payload-processing" Nov 03 00:57:46.357318 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:57:46.357230 4939 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dfcf5830-ce86-40fb-bdfb-3564175ac08a" containerName="payload-processing" Nov 03 00:57:46.357318 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:57:46.357263 4939 cpu_manager.go:414] "containerMap: removing container" podUID="f9c818e4-813f-40dd-9311-906a517eb42a" containerName="payload-pre-processing" Nov 03 00:57:46.357318 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:57:46.357269 4939 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f9c818e4-813f-40dd-9311-906a517eb42a" containerName="payload-pre-processing" Nov 03 00:57:46.357416 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:57:46.357383 4939 memory_manager.go:365] "RemoveStaleState removing state" podUID="f9c818e4-813f-40dd-9311-906a517eb42a" containerName="payload-pre-processing" Nov 03 00:57:46.357416 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:57:46.357398 4939 memory_manager.go:365] "RemoveStaleState removing state" podUID="dfcf5830-ce86-40fb-bdfb-3564175ac08a" containerName="payload-processing" Nov 03 00:57:46.362293 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:57:46.362274 4939 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-340b96-86744887cf-swn7k" Nov 03 00:57:46.369490 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:57:46.369085 4939 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-a-340b96\"" Nov 03 00:57:46.369490 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:57:46.369137 4939 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-a-340b96-dockercfg-cpl4g\"" Nov 03 00:57:46.369490 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:57:46.369312 4939 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-340b96-86744887cf-swn7k"] Nov 03 00:57:46.406814 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:57:46.406783 4939 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-340b96-c984ff695-nz2kt"] Nov 03 00:57:46.412663 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:57:46.412640 4939 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-340b96-c984ff695-nz2kt" Nov 03 00:57:46.418924 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:57:46.418889 4939 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-340b96-c984ff695-nz2kt"] Nov 03 00:57:46.481668 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:57:46.479114 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c4hdr\" (UniqueName: \"kubernetes.io/projected/a6a06134-dd53-4f93-8a78-7d26fedd2ab3-kube-api-access-c4hdr\") pod \"payload-processing-e2e-api-a-340b96-c984ff695-nz2kt\" (UID: \"a6a06134-dd53-4f93-8a78-7d26fedd2ab3\") " pod="openshift-ingress/payload-processing-e2e-api-a-340b96-c984ff695-nz2kt" Nov 03 00:57:46.481668 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:57:46.479174 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vcn4w\" (UniqueName: \"kubernetes.io/projected/350de0d1-756f-4e15-8cc6-e5b8474571b2-kube-api-access-vcn4w\") pod \"payload-pre-processing-e2e-api-a-340b96-86744887cf-swn7k\" (UID: \"350de0d1-756f-4e15-8cc6-e5b8474571b2\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-340b96-86744887cf-swn7k" Nov 03 00:57:46.481668 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:57:46.479267 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a6a06134-dd53-4f93-8a78-7d26fedd2ab3-plugins-config-volume\") pod \"payload-processing-e2e-api-a-340b96-c984ff695-nz2kt\" (UID: \"a6a06134-dd53-4f93-8a78-7d26fedd2ab3\") " pod="openshift-ingress/payload-processing-e2e-api-a-340b96-c984ff695-nz2kt" Nov 03 00:57:46.481668 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:57:46.479462 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/350de0d1-756f-4e15-8cc6-e5b8474571b2-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-340b96-86744887cf-swn7k\" (UID: \"350de0d1-756f-4e15-8cc6-e5b8474571b2\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-340b96-86744887cf-swn7k" Nov 03 00:57:46.501367 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:57:46.483320 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 00:57:46.662933 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:57:46.662515 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a6a06134-dd53-4f93-8a78-7d26fedd2ab3-plugins-config-volume\") pod \"payload-processing-e2e-api-a-340b96-c984ff695-nz2kt\" (UID: \"a6a06134-dd53-4f93-8a78-7d26fedd2ab3\") " pod="openshift-ingress/payload-processing-e2e-api-a-340b96-c984ff695-nz2kt" Nov 03 00:57:46.662933 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:57:46.662754 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/350de0d1-756f-4e15-8cc6-e5b8474571b2-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-340b96-86744887cf-swn7k\" (UID: \"350de0d1-756f-4e15-8cc6-e5b8474571b2\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-340b96-86744887cf-swn7k" Nov 03 00:57:46.663502 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:57:46.662956 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c4hdr\" (UniqueName: \"kubernetes.io/projected/a6a06134-dd53-4f93-8a78-7d26fedd2ab3-kube-api-access-c4hdr\") pod \"payload-processing-e2e-api-a-340b96-c984ff695-nz2kt\" (UID: \"a6a06134-dd53-4f93-8a78-7d26fedd2ab3\") " pod="openshift-ingress/payload-processing-e2e-api-a-340b96-c984ff695-nz2kt" Nov 03 00:57:46.663502 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:57:46.662991 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vcn4w\" (UniqueName: \"kubernetes.io/projected/350de0d1-756f-4e15-8cc6-e5b8474571b2-kube-api-access-vcn4w\") pod \"payload-pre-processing-e2e-api-a-340b96-86744887cf-swn7k\" (UID: \"350de0d1-756f-4e15-8cc6-e5b8474571b2\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-340b96-86744887cf-swn7k" Nov 03 00:57:46.697069 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:57:46.690897 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/350de0d1-756f-4e15-8cc6-e5b8474571b2-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-340b96-86744887cf-swn7k\" (UID: \"350de0d1-756f-4e15-8cc6-e5b8474571b2\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-340b96-86744887cf-swn7k" Nov 03 00:57:46.711929 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:57:46.697512 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a6a06134-dd53-4f93-8a78-7d26fedd2ab3-plugins-config-volume\") pod \"payload-processing-e2e-api-a-340b96-c984ff695-nz2kt\" (UID: \"a6a06134-dd53-4f93-8a78-7d26fedd2ab3\") " pod="openshift-ingress/payload-processing-e2e-api-a-340b96-c984ff695-nz2kt" Nov 03 00:57:46.767560 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:57:46.767521 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-c4hdr\" (UniqueName: \"kubernetes.io/projected/a6a06134-dd53-4f93-8a78-7d26fedd2ab3-kube-api-access-c4hdr\") pod \"payload-processing-e2e-api-a-340b96-c984ff695-nz2kt\" (UID: \"a6a06134-dd53-4f93-8a78-7d26fedd2ab3\") " pod="openshift-ingress/payload-processing-e2e-api-a-340b96-c984ff695-nz2kt" Nov 03 00:57:46.768952 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:57:46.768927 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vcn4w\" (UniqueName: \"kubernetes.io/projected/350de0d1-756f-4e15-8cc6-e5b8474571b2-kube-api-access-vcn4w\") pod \"payload-pre-processing-e2e-api-a-340b96-86744887cf-swn7k\" (UID: \"350de0d1-756f-4e15-8cc6-e5b8474571b2\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-340b96-86744887cf-swn7k" Nov 03 00:57:46.927175 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:57:46.927094 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 00:57:47.001063 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:57:46.994465 4939 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-340b96-86744887cf-swn7k" Nov 03 00:57:47.047765 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:57:47.047735 4939 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-340b96-c984ff695-nz2kt" Nov 03 00:57:47.328997 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:57:47.328955 4939 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-340b96-86744887cf-swn7k"] Nov 03 00:57:47.329145 ip-10-0-140-107 kubenswrapper[4939]: W1103 00:57:47.329123 4939 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod350de0d1_756f_4e15_8cc6_e5b8474571b2.slice/crio-fc805066da0cb9281e663c323585ef4bbaadb9c8d7564ba0d005fea75938ee48 WatchSource:0}: Error finding container fc805066da0cb9281e663c323585ef4bbaadb9c8d7564ba0d005fea75938ee48: Status 404 returned error can't find the container with id fc805066da0cb9281e663c323585ef4bbaadb9c8d7564ba0d005fea75938ee48 Nov 03 00:57:47.341581 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:57:47.341553 4939 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-340b96-c984ff695-nz2kt"] Nov 03 00:57:47.344173 ip-10-0-140-107 kubenswrapper[4939]: W1103 00:57:47.343150 4939 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda6a06134_dd53_4f93_8a78_7d26fedd2ab3.slice/crio-a941f65684ad8465dca8b2e51c0ff17ae9bdd3276b4411af6e6926c6f1f9e1cc WatchSource:0}: Error finding container a941f65684ad8465dca8b2e51c0ff17ae9bdd3276b4411af6e6926c6f1f9e1cc: Status 404 returned error can't find the container with id a941f65684ad8465dca8b2e51c0ff17ae9bdd3276b4411af6e6926c6f1f9e1cc Nov 03 00:57:47.586965 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:57:47.586936 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 00:57:47.778694 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:57:47.778637 4939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-340b96-86744887cf-swn7k" event={"ID":"350de0d1-756f-4e15-8cc6-e5b8474571b2","Type":"ContainerStarted","Data":"87933e8f795dd3c4c8815d3376bfaeae21fe9967d3532f65b0242b082c49c545"} Nov 03 00:57:47.779199 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:57:47.778726 4939 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-api-a-340b96-86744887cf-swn7k" Nov 03 00:57:47.779199 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:57:47.778749 4939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-340b96-86744887cf-swn7k" event={"ID":"350de0d1-756f-4e15-8cc6-e5b8474571b2","Type":"ContainerStarted","Data":"fc805066da0cb9281e663c323585ef4bbaadb9c8d7564ba0d005fea75938ee48"} Nov 03 00:57:47.780273 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:57:47.780244 4939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-340b96-c984ff695-nz2kt" event={"ID":"a6a06134-dd53-4f93-8a78-7d26fedd2ab3","Type":"ContainerStarted","Data":"9f50fc61cb95cb3e9c29e3a408c393961c75d1d8f6bf4411670b752aa42c45a7"} Nov 03 00:57:47.780432 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:57:47.780288 4939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-340b96-c984ff695-nz2kt" event={"ID":"a6a06134-dd53-4f93-8a78-7d26fedd2ab3","Type":"ContainerStarted","Data":"a941f65684ad8465dca8b2e51c0ff17ae9bdd3276b4411af6e6926c6f1f9e1cc"} Nov 03 00:57:47.780432 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:57:47.780330 4939 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-api-a-340b96-c984ff695-nz2kt" Nov 03 00:57:47.797668 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:57:47.797617 4939 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-api-a-340b96-86744887cf-swn7k" podStartSLOduration=1.797603687 podStartE2EDuration="1.797603687s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:57:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:57:47.794886833 +0000 UTC m=+1016.059221778" watchObservedRunningTime="2026-11-03 00:57:47.797603687 +0000 UTC m=+1016.061938632" Nov 03 00:57:47.813154 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:57:47.813094 4939 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-api-a-340b96-c984ff695-nz2kt" podStartSLOduration=1.813076422 podStartE2EDuration="1.813076422s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:57:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:57:47.812114274 +0000 UTC m=+1016.076449221" watchObservedRunningTime="2026-11-03 00:57:47.813076422 +0000 UTC m=+1016.077411367" Nov 03 00:57:48.065289 ip-10-0-140-107 kubenswrapper[4939]: E1103 00:57:48.065189 4939 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-sthjh" containerName="extractor" Nov 03 00:57:48.396885 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:57:48.396853 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 00:57:48.513947 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:57:48.512578 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 00:57:49.382373 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:57:49.382334 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 00:57:50.117843 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:57:50.117810 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 00:57:50.217930 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:57:50.217125 4939 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-api-b-a2d8e5-5bb4464ff9-fpt8c" Nov 03 00:57:50.890038 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:57:50.889990 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 00:57:51.845874 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:57:51.845845 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 00:57:52.420837 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:57:52.420799 4939 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-340b96-69fb46f475-f4qbm"] Nov 03 00:57:52.427853 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:57:52.427832 4939 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-340b96-69fb46f475-f4qbm" Nov 03 00:57:52.438840 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:57:52.438774 4939 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-340b96-69fb46f475-f4qbm"] Nov 03 00:57:52.439308 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:57:52.439285 4939 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-a-340b96-serving-cert\"" Nov 03 00:57:52.502039 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:57:52.502008 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b911ac20-6027-4445-8ccc-c622715708dc-maas-api-tls\") pod \"maas-api-e2e-api-a-340b96-69fb46f475-f4qbm\" (UID: \"b911ac20-6027-4445-8ccc-c622715708dc\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-340b96-69fb46f475-f4qbm" Nov 03 00:57:52.502211 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:57:52.502060 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fskdg\" (UniqueName: \"kubernetes.io/projected/b911ac20-6027-4445-8ccc-c622715708dc-kube-api-access-fskdg\") pod \"maas-api-e2e-api-a-340b96-69fb46f475-f4qbm\" (UID: \"b911ac20-6027-4445-8ccc-c622715708dc\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-340b96-69fb46f475-f4qbm" Nov 03 00:57:52.502250 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:57:52.502204 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b911ac20-6027-4445-8ccc-c622715708dc-metrics-certs\") pod \"maas-api-e2e-api-a-340b96-69fb46f475-f4qbm\" (UID: \"b911ac20-6027-4445-8ccc-c622715708dc\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-340b96-69fb46f475-f4qbm" Nov 03 00:57:52.502343 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:57:52.502328 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b911ac20-6027-4445-8ccc-c622715708dc-openshift-service-ca\") pod \"maas-api-e2e-api-a-340b96-69fb46f475-f4qbm\" (UID: \"b911ac20-6027-4445-8ccc-c622715708dc\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-340b96-69fb46f475-f4qbm" Nov 03 00:57:52.577160 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:57:52.577055 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 00:57:52.625552 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:57:52.624477 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b911ac20-6027-4445-8ccc-c622715708dc-maas-api-tls\") pod \"maas-api-e2e-api-a-340b96-69fb46f475-f4qbm\" (UID: \"b911ac20-6027-4445-8ccc-c622715708dc\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-340b96-69fb46f475-f4qbm" Nov 03 00:57:52.625552 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:57:52.624553 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fskdg\" (UniqueName: \"kubernetes.io/projected/b911ac20-6027-4445-8ccc-c622715708dc-kube-api-access-fskdg\") pod \"maas-api-e2e-api-a-340b96-69fb46f475-f4qbm\" (UID: \"b911ac20-6027-4445-8ccc-c622715708dc\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-340b96-69fb46f475-f4qbm" Nov 03 00:57:52.625552 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:57:52.624745 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b911ac20-6027-4445-8ccc-c622715708dc-metrics-certs\") pod \"maas-api-e2e-api-a-340b96-69fb46f475-f4qbm\" (UID: \"b911ac20-6027-4445-8ccc-c622715708dc\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-340b96-69fb46f475-f4qbm" Nov 03 00:57:52.625552 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:57:52.624881 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b911ac20-6027-4445-8ccc-c622715708dc-openshift-service-ca\") pod \"maas-api-e2e-api-a-340b96-69fb46f475-f4qbm\" (UID: \"b911ac20-6027-4445-8ccc-c622715708dc\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-340b96-69fb46f475-f4qbm" Nov 03 00:57:52.625896 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:57:52.625867 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b911ac20-6027-4445-8ccc-c622715708dc-openshift-service-ca\") pod \"maas-api-e2e-api-a-340b96-69fb46f475-f4qbm\" (UID: \"b911ac20-6027-4445-8ccc-c622715708dc\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-340b96-69fb46f475-f4qbm" Nov 03 00:57:52.650462 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:57:52.649768 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b911ac20-6027-4445-8ccc-c622715708dc-maas-api-tls\") pod \"maas-api-e2e-api-a-340b96-69fb46f475-f4qbm\" (UID: \"b911ac20-6027-4445-8ccc-c622715708dc\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-340b96-69fb46f475-f4qbm" Nov 03 00:57:52.651206 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:57:52.651179 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b911ac20-6027-4445-8ccc-c622715708dc-metrics-certs\") pod \"maas-api-e2e-api-a-340b96-69fb46f475-f4qbm\" (UID: \"b911ac20-6027-4445-8ccc-c622715708dc\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-340b96-69fb46f475-f4qbm" Nov 03 00:57:52.673717 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:57:52.673681 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fskdg\" (UniqueName: \"kubernetes.io/projected/b911ac20-6027-4445-8ccc-c622715708dc-kube-api-access-fskdg\") pod \"maas-api-e2e-api-a-340b96-69fb46f475-f4qbm\" (UID: \"b911ac20-6027-4445-8ccc-c622715708dc\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-340b96-69fb46f475-f4qbm" Nov 03 00:57:52.685162 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:57:52.685128 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 00:57:52.756665 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:57:52.756635 4939 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-340b96-69fb46f475-f4qbm" Nov 03 00:57:53.040395 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:57:53.040358 4939 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-340b96-69fb46f475-f4qbm"] Nov 03 00:57:53.041280 ip-10-0-140-107 kubenswrapper[4939]: W1103 00:57:53.041254 4939 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb911ac20_6027_4445_8ccc_c622715708dc.slice/crio-7b5b7d629366d885311d8dcb71c6da92c5baa6e106902a2bbf780f0f68000b2c WatchSource:0}: Error finding container 7b5b7d629366d885311d8dcb71c6da92c5baa6e106902a2bbf780f0f68000b2c: Status 404 returned error can't find the container with id 7b5b7d629366d885311d8dcb71c6da92c5baa6e106902a2bbf780f0f68000b2c Nov 03 00:57:53.311017 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:57:53.298817 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 00:57:53.830234 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:57:53.830137 4939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-340b96-69fb46f475-f4qbm" event={"ID":"b911ac20-6027-4445-8ccc-c622715708dc","Type":"ContainerStarted","Data":"6e116fc6af05bd396ac09d78e6196e0c02f97eb2b7cc230c09042ea42ce77071"} Nov 03 00:57:53.830234 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:57:53.830202 4939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-340b96-69fb46f475-f4qbm" event={"ID":"b911ac20-6027-4445-8ccc-c622715708dc","Type":"ContainerStarted","Data":"7b5b7d629366d885311d8dcb71c6da92c5baa6e106902a2bbf780f0f68000b2c"} Nov 03 00:57:53.830656 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:57:53.830342 4939 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-340b96-69fb46f475-f4qbm" Nov 03 00:57:53.851568 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:57:53.851514 4939 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-340b96-69fb46f475-f4qbm" podStartSLOduration=1.346783286 podStartE2EDuration="1.851500405s" totalImagesPullingTime="504.717119ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:57:52 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:57:53.849430742 +0000 UTC m=+1022.113765686" watchObservedRunningTime="2026-11-03 00:57:53.851500405 +0000 UTC m=+1022.115835350" Nov 03 00:57:54.192191 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:57:54.192141 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 00:57:54.517812 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:57:54.517731 4939 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-mt-dd03e395-b8464b49-7rlqj" Nov 03 00:57:54.879744 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:57:54.879696 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 00:57:55.599516 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:57:55.599479 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 00:57:56.267073 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:57:56.267040 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 00:57:56.781452 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:57:56.781408 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 00:57:57.021761 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:57:57.021734 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 00:57:57.645809 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:57:57.645741 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 00:57:58.400392 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:57:58.400346 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 00:57:58.790948 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:57:58.790853 4939 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-api-a-340b96-c984ff695-nz2kt" Nov 03 00:57:58.791378 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:57:58.790968 4939 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-api-a-340b96-86744887cf-swn7k" Nov 03 00:57:58.979950 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:57:58.979885 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 00:57:59.093880 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:57:59.093846 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 00:57:59.694413 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:57:59.694372 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 00:57:59.843266 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:57:59.843237 4939 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-340b96-69fb46f475-f4qbm" Nov 03 00:58:00.656820 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:58:00.656787 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 00:58:01.349108 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:58:01.349068 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 00:58:02.113138 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:58:02.113107 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 00:58:02.825122 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:58:02.825063 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 00:58:03.026250 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:58:03.026165 4939 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-4pw7v"] Nov 03 00:58:03.670332 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:58:03.670255 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 00:58:04.928074 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:58:04.928033 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 00:58:05.070616 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:58:05.069407 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 00:58:05.249831 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:58:05.249337 4939 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-54df68-5d998485f9-pknfz"] Nov 03 00:58:05.403877 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:58:05.403845 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 00:58:05.652286 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:58:05.652254 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 00:58:05.795930 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:58:05.795464 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 00:58:06.152110 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:58:06.152033 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 00:58:06.448029 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:58:06.444973 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 00:58:06.597121 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:58:06.597086 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 00:58:07.442934 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:58:07.441928 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 00:58:07.975107 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:58:07.944072 4939 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-mt-dd03e395/e2e-lifecycle-model-dd03e395-kserve-78979759df-b8v9c" podUID="e4803d7d-4603-4b14-950d-018f88294285" containerName="main" probeResult="failure" output="Get \"https://10.132.0.91:8000/ready\": context deadline exceeded" Nov 03 00:58:07.975107 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:58:07.955256 4939 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-f7680d/distinct-w2-f7680d-kserve-756f874b98-qnpv8" podUID="836753fc-f9ef-4f61-b3fe-e5b92b7ee27b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.70:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 00:58:08.021937 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:58:08.018980 4939 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-f7680d/distinct-w2-f7680d-kserve-756f874b98-qnpv8" podUID="836753fc-f9ef-4f61-b3fe-e5b92b7ee27b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.70:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 00:58:08.123934 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:58:08.123713 4939 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-f7680d/distinct-2-w2-f7680d-kserve-7d7c4564bf-cbv2c" podUID="2de18086-761b-4773-b9fc-15ddc4a220fc" containerName="main" probeResult="failure" output="Get \"https://10.132.0.75:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 00:58:08.297036 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:58:08.296926 4939 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-88bbf7/simulator-w1-88bbf7-kserve-5fbb669f8f-42hgd" podUID="83b38883-2e7a-444d-aa68-52be4a25ae66" containerName="main" probeResult="failure" output="Get \"https://10.132.0.62:8000/health\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Nov 03 00:58:08.302674 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:58:08.302550 4939 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-2f9a3a/distinct-w0-2f9a3a-kserve-555565cfcf-jtcr9" podUID="ac0ece06-8718-463f-915f-5315988d9711" containerName="main" probeResult="failure" output="Get \"https://10.132.0.76:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 00:58:08.605229 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:58:08.604039 4939 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-2f9a3a/simulator-w0-2f9a3a-kserve-9596bbb8f-2trgz" podUID="207e2207-fb6c-4076-95df-72e95671c261" containerName="main" probeResult="failure" output="Get \"https://10.132.0.61:8000/ready\": context deadline exceeded" Nov 03 00:58:08.639281 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:58:08.637584 4939 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-83755a/distinct-2-w3-83755a-kserve-6cf66b995d-2l47b" podUID="c3b05e88-5d5b-4d03-b28d-832f22e3c868" containerName="main" probeResult="failure" output="Get \"https://10.132.0.72:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 00:58:08.795436 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:58:08.795404 4939 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7034-3d9226"] Nov 03 00:58:08.845205 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:58:08.840505 4939 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-596dc9545f-l9ccf container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.43:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 00:58:08.845205 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:58:08.840589 4939 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-l9ccf" podUID="c10d48f9-d5fc-4c4c-96c4-6531114191b8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.43:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 00:58:08.893638 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:58:08.893518 4939 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-88bbf7/simulator-w1-88bbf7-kserve-5fbb669f8f-42hgd" podUID="83b38883-2e7a-444d-aa68-52be4a25ae66" containerName="main" probeResult="failure" output="Get \"https://10.132.0.62:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Nov 03 00:58:08.907940 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:58:08.907092 4939 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7034-3d9226"] Nov 03 00:58:08.907940 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:58:08.907215 4939 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7034-3d9226" Nov 03 00:58:08.913776 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:58:08.913719 4939 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-7l8r6\"" Nov 03 00:58:08.931860 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:58:08.930662 4939 prober.go:122] "Probe failed" probeType="Liveness" pod="ai-tenant-e2e-mt-dd03e395/e2e-lifecycle-model-dd03e395-kserve-78979759df-b8v9c" podUID="e4803d7d-4603-4b14-950d-018f88294285" containerName="main" probeResult="failure" output="Get \"https://10.132.0.91:8000/health\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Nov 03 00:58:08.937652 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:58:08.937518 4939 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-88bbf7/distinct-2-w1-88bbf7-kserve-5457cdcdbd-l6tfd" podUID="a397931b-faa9-4c2e-8290-491350546663" containerName="main" probeResult="failure" output="Get \"https://10.132.0.85:8000/ready\": context deadline exceeded" Nov 03 00:58:08.959933 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:58:08.955310 4939 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-f7680d/premium-w2-f7680d-kserve-76565b8cf9-hqbbt" podUID="4712e708-18a2-4a13-a12c-a5669397b2c3" containerName="main" probeResult="failure" output="Get \"https://10.132.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 00:58:09.005063 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:58:09.005018 4939 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-mt-dd03e395/e2e-lifecycle-model-dd03e395-kserve-78979759df-b8v9c" podUID="e4803d7d-4603-4b14-950d-018f88294285" containerName="main" probeResult="failure" output="Get \"https://10.132.0.91:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Nov 03 00:58:09.017929 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:58:09.011977 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hc9rl\" (UniqueName: \"kubernetes.io/projected/cc7c990a-c21b-4046-8c45-aba9b48114e4-kube-api-access-hc9rl\") pod \"test-curl-7034-3d9226\" (UID: \"cc7c990a-c21b-4046-8c45-aba9b48114e4\") " pod="opendatahub/test-curl-7034-3d9226" Nov 03 00:58:09.017929 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:58:09.016664 4939 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-mt-dd03e395/e2e-lifecycle-model-dd03e395-kserve-78979759df-b8v9c" Nov 03 00:58:09.040976 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:58:09.035409 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 00:58:09.152117 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:58:09.152025 4939 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-f7680d/distinct-2-w2-f7680d-kserve-7d7c4564bf-cbv2c" podUID="2de18086-761b-4773-b9fc-15ddc4a220fc" containerName="main" probeResult="failure" output="Get \"https://10.132.0.75:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Nov 03 00:58:09.152987 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:58:09.152964 4939 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-f7680d/distinct-2-w2-f7680d-kserve-7d7c4564bf-cbv2c" Nov 03 00:58:09.215753 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:58:09.215724 4939 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-f7680d/distinct-2-w2-f7680d-kserve-7d7c4564bf-cbv2c" Nov 03 00:58:09.232978 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:58:09.220357 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hc9rl\" (UniqueName: \"kubernetes.io/projected/cc7c990a-c21b-4046-8c45-aba9b48114e4-kube-api-access-hc9rl\") pod \"test-curl-7034-3d9226\" (UID: \"cc7c990a-c21b-4046-8c45-aba9b48114e4\") " pod="opendatahub/test-curl-7034-3d9226" Nov 03 00:58:09.232978 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:58:09.222818 4939 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-mt-dd03e395/e2e-lifecycle-model-dd03e395-kserve-78979759df-b8v9c" Nov 03 00:58:09.264784 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:58:09.264753 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hc9rl\" (UniqueName: \"kubernetes.io/projected/cc7c990a-c21b-4046-8c45-aba9b48114e4-kube-api-access-hc9rl\") pod \"test-curl-7034-3d9226\" (UID: \"cc7c990a-c21b-4046-8c45-aba9b48114e4\") " pod="opendatahub/test-curl-7034-3d9226" Nov 03 00:58:09.453902 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:58:09.453804 4939 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7034-3d9226" Nov 03 00:58:09.657597 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:58:09.657564 4939 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-54df68-64c7497868-srpmr"] Nov 03 00:58:09.658443 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:58:09.658390 4939 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-54df68-64c7497868-srpmr" podUID="ad1a1a5c-cb63-49e9-af61-26a6cd45cb8a" containerName="maas-api" containerID="cri-o://a8fcc422f8ec8b907eeda9f408123c3e9d30cda568707cbe3414b0afcf6e59ff" gracePeriod=30 Nov 03 00:58:09.728650 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:58:09.728566 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 00:58:09.798977 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:58:09.798925 4939 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7034-3d9226"] Nov 03 00:58:09.799558 ip-10-0-140-107 kubenswrapper[4939]: W1103 00:58:09.799488 4939 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podcc7c990a_c21b_4046_8c45_aba9b48114e4.slice/crio-69fa7a4c3693cf34235460e5030cf61fb4896197234e4aae0ff36dde8ef2dd95 WatchSource:0}: Error finding container 69fa7a4c3693cf34235460e5030cf61fb4896197234e4aae0ff36dde8ef2dd95: Status 404 returned error can't find the container with id 69fa7a4c3693cf34235460e5030cf61fb4896197234e4aae0ff36dde8ef2dd95 Nov 03 00:58:10.016441 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:58:09.996509 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 00:58:10.298966 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:58:10.281237 4939 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-api-a-54df68-5d998485f9-pknfz" podUID="fbcbf75f-badb-46a6-9f2d-3204afc53907" containerName="payload-pre-processing" containerID="cri-o://eefb2c57c0c16e9c524aa60947ee2e2410a8e592f6e7634894e70f3ffaa06169" gracePeriod=25 Nov 03 00:58:10.309456 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:58:10.309355 4939 generic.go:356] "Generic (PLEG): container finished" podID="ad1a1a5c-cb63-49e9-af61-26a6cd45cb8a" containerID="a8fcc422f8ec8b907eeda9f408123c3e9d30cda568707cbe3414b0afcf6e59ff" exitCode=0 Nov 03 00:58:10.309655 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:58:10.309585 4939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-54df68-64c7497868-srpmr" event={"ID":"ad1a1a5c-cb63-49e9-af61-26a6cd45cb8a","Type":"ContainerDied","Data":"a8fcc422f8ec8b907eeda9f408123c3e9d30cda568707cbe3414b0afcf6e59ff"} Nov 03 00:58:10.312299 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:58:10.312277 4939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7034-3d9226" event={"ID":"cc7c990a-c21b-4046-8c45-aba9b48114e4","Type":"ContainerStarted","Data":"69fa7a4c3693cf34235460e5030cf61fb4896197234e4aae0ff36dde8ef2dd95"} Nov 03 00:58:10.414774 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:58:10.414746 4939 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-54df68-64c7497868-srpmr" Nov 03 00:58:10.503411 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:58:10.503373 4939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ad1a1a5c-cb63-49e9-af61-26a6cd45cb8a-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ad1a1a5c-cb63-49e9-af61-26a6cd45cb8a-openshift-service-ca\") pod \"ad1a1a5c-cb63-49e9-af61-26a6cd45cb8a\" (UID: \"ad1a1a5c-cb63-49e9-af61-26a6cd45cb8a\") " Nov 03 00:58:10.503579 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:58:10.503430 4939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ad1a1a5c-cb63-49e9-af61-26a6cd45cb8a-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ad1a1a5c-cb63-49e9-af61-26a6cd45cb8a-maas-api-tls\") pod \"ad1a1a5c-cb63-49e9-af61-26a6cd45cb8a\" (UID: \"ad1a1a5c-cb63-49e9-af61-26a6cd45cb8a\") " Nov 03 00:58:10.503621 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:58:10.503596 4939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ad1a1a5c-cb63-49e9-af61-26a6cd45cb8a-kube-api-access-mcgpd\" (UniqueName: \"kubernetes.io/projected/ad1a1a5c-cb63-49e9-af61-26a6cd45cb8a-kube-api-access-mcgpd\") pod \"ad1a1a5c-cb63-49e9-af61-26a6cd45cb8a\" (UID: \"ad1a1a5c-cb63-49e9-af61-26a6cd45cb8a\") " Nov 03 00:58:10.503709 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:58:10.503692 4939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ad1a1a5c-cb63-49e9-af61-26a6cd45cb8a-metrics-certs\" (UniqueName: \"kubernetes.io/secret/ad1a1a5c-cb63-49e9-af61-26a6cd45cb8a-metrics-certs\") pod \"ad1a1a5c-cb63-49e9-af61-26a6cd45cb8a\" (UID: \"ad1a1a5c-cb63-49e9-af61-26a6cd45cb8a\") " Nov 03 00:58:10.503763 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:58:10.503718 4939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ad1a1a5c-cb63-49e9-af61-26a6cd45cb8a-openshift-service-ca" pod "ad1a1a5c-cb63-49e9-af61-26a6cd45cb8a" (UID: "ad1a1a5c-cb63-49e9-af61-26a6cd45cb8a"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 00:58:10.504677 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:58:10.504369 4939 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ad1a1a5c-cb63-49e9-af61-26a6cd45cb8a-openshift-service-ca\") on node \"ip-10-0-140-107.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 00:58:10.505828 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:58:10.505808 4939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ad1a1a5c-cb63-49e9-af61-26a6cd45cb8a-kube-api-access-mcgpd" pod "ad1a1a5c-cb63-49e9-af61-26a6cd45cb8a" (UID: "ad1a1a5c-cb63-49e9-af61-26a6cd45cb8a"). InnerVolumeSpecName "kube-api-access-mcgpd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 00:58:10.506146 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:58:10.506130 4939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ad1a1a5c-cb63-49e9-af61-26a6cd45cb8a-metrics-certs" pod "ad1a1a5c-cb63-49e9-af61-26a6cd45cb8a" (UID: "ad1a1a5c-cb63-49e9-af61-26a6cd45cb8a"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 00:58:10.506211 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:58:10.506198 4939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ad1a1a5c-cb63-49e9-af61-26a6cd45cb8a-maas-api-tls" pod "ad1a1a5c-cb63-49e9-af61-26a6cd45cb8a" (UID: "ad1a1a5c-cb63-49e9-af61-26a6cd45cb8a"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 00:58:10.605741 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:58:10.605710 4939 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mcgpd\" (UniqueName: \"kubernetes.io/projected/ad1a1a5c-cb63-49e9-af61-26a6cd45cb8a-kube-api-access-mcgpd\") on node \"ip-10-0-140-107.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 00:58:10.605741 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:58:10.605744 4939 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ad1a1a5c-cb63-49e9-af61-26a6cd45cb8a-metrics-certs\") on node \"ip-10-0-140-107.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 00:58:10.606218 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:58:10.605760 4939 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ad1a1a5c-cb63-49e9-af61-26a6cd45cb8a-maas-api-tls\") on node \"ip-10-0-140-107.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 00:58:10.612587 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:58:10.612563 4939 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-54df68-5d998485f9-pknfz" Nov 03 00:58:10.706749 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:58:10.706712 4939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fbcbf75f-badb-46a6-9f2d-3204afc53907-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fbcbf75f-badb-46a6-9f2d-3204afc53907-plugins-config-volume\") pod \"fbcbf75f-badb-46a6-9f2d-3204afc53907\" (UID: \"fbcbf75f-badb-46a6-9f2d-3204afc53907\") " Nov 03 00:58:10.707279 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:58:10.706861 4939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fbcbf75f-badb-46a6-9f2d-3204afc53907-kube-api-access-8rjnv\" (UniqueName: \"kubernetes.io/projected/fbcbf75f-badb-46a6-9f2d-3204afc53907-kube-api-access-8rjnv\") pod \"fbcbf75f-badb-46a6-9f2d-3204afc53907\" (UID: \"fbcbf75f-badb-46a6-9f2d-3204afc53907\") " Nov 03 00:58:10.707279 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:58:10.707216 4939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fbcbf75f-badb-46a6-9f2d-3204afc53907-plugins-config-volume" pod "fbcbf75f-badb-46a6-9f2d-3204afc53907" (UID: "fbcbf75f-badb-46a6-9f2d-3204afc53907"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 00:58:10.707411 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:58:10.707391 4939 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fbcbf75f-badb-46a6-9f2d-3204afc53907-plugins-config-volume\") on node \"ip-10-0-140-107.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 00:58:10.708932 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:58:10.708886 4939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fbcbf75f-badb-46a6-9f2d-3204afc53907-kube-api-access-8rjnv" pod "fbcbf75f-badb-46a6-9f2d-3204afc53907" (UID: "fbcbf75f-badb-46a6-9f2d-3204afc53907"). InnerVolumeSpecName "kube-api-access-8rjnv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 00:58:10.808592 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:58:10.808561 4939 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8rjnv\" (UniqueName: \"kubernetes.io/projected/fbcbf75f-badb-46a6-9f2d-3204afc53907-kube-api-access-8rjnv\") on node \"ip-10-0-140-107.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 00:58:10.879839 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:58:10.879748 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 00:58:11.094801 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:58:11.094771 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 00:58:11.322591 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:58:11.322547 4939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7034-3d9226" event={"ID":"cc7c990a-c21b-4046-8c45-aba9b48114e4","Type":"ContainerStarted","Data":"166013e6cce52f91e840e14cd10348a90f81c7c5478549744474f543059257a3"} Nov 03 00:58:11.324183 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:58:11.324153 4939 generic.go:356] "Generic (PLEG): container finished" podID="fbcbf75f-badb-46a6-9f2d-3204afc53907" containerID="eefb2c57c0c16e9c524aa60947ee2e2410a8e592f6e7634894e70f3ffaa06169" exitCode=0 Nov 03 00:58:11.324306 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:58:11.324190 4939 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-54df68-5d998485f9-pknfz" Nov 03 00:58:11.324306 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:58:11.324229 4939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-54df68-5d998485f9-pknfz" event={"ID":"fbcbf75f-badb-46a6-9f2d-3204afc53907","Type":"ContainerDied","Data":"eefb2c57c0c16e9c524aa60947ee2e2410a8e592f6e7634894e70f3ffaa06169"} Nov 03 00:58:11.324306 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:58:11.324281 4939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-54df68-5d998485f9-pknfz" event={"ID":"fbcbf75f-badb-46a6-9f2d-3204afc53907","Type":"ContainerDied","Data":"9161e814ac4c38448cb4bdb64ef5a0d7c237905131aec2547016b5b41c2535cf"} Nov 03 00:58:11.324306 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:58:11.324305 4939 scope.go:122] "RemoveContainer" containerID="eefb2c57c0c16e9c524aa60947ee2e2410a8e592f6e7634894e70f3ffaa06169" Nov 03 00:58:11.326055 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:58:11.326029 4939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-54df68-64c7497868-srpmr" event={"ID":"ad1a1a5c-cb63-49e9-af61-26a6cd45cb8a","Type":"ContainerDied","Data":"f427c42b7a50a898a08fd3090cbd4f50d4061153a903d0d642ee626ea341037c"} Nov 03 00:58:11.326175 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:58:11.326117 4939 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-54df68-64c7497868-srpmr" Nov 03 00:58:11.340035 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:58:11.340003 4939 scope.go:122] "RemoveContainer" containerID="eefb2c57c0c16e9c524aa60947ee2e2410a8e592f6e7634894e70f3ffaa06169" Nov 03 00:58:11.340348 ip-10-0-140-107 kubenswrapper[4939]: E1103 00:58:11.340326 4939 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"eefb2c57c0c16e9c524aa60947ee2e2410a8e592f6e7634894e70f3ffaa06169\": container with ID starting with eefb2c57c0c16e9c524aa60947ee2e2410a8e592f6e7634894e70f3ffaa06169 not found: ID does not exist" containerID="eefb2c57c0c16e9c524aa60947ee2e2410a8e592f6e7634894e70f3ffaa06169" Nov 03 00:58:11.340511 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:58:11.340360 4939 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"eefb2c57c0c16e9c524aa60947ee2e2410a8e592f6e7634894e70f3ffaa06169"} err="failed to get container status \"eefb2c57c0c16e9c524aa60947ee2e2410a8e592f6e7634894e70f3ffaa06169\": rpc error: code = NotFound desc = could not find container \"eefb2c57c0c16e9c524aa60947ee2e2410a8e592f6e7634894e70f3ffaa06169\": container with ID starting with eefb2c57c0c16e9c524aa60947ee2e2410a8e592f6e7634894e70f3ffaa06169 not found: ID does not exist" Nov 03 00:58:11.340511 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:58:11.340382 4939 scope.go:122] "RemoveContainer" containerID="a8fcc422f8ec8b907eeda9f408123c3e9d30cda568707cbe3414b0afcf6e59ff" Nov 03 00:58:11.340711 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:58:11.340670 4939 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/test-curl-7034-3d9226" podStartSLOduration=3.340655759 podStartE2EDuration="3.340655759s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:58:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:58:11.340023756 +0000 UTC m=+1039.604358702" watchObservedRunningTime="2026-11-03 00:58:11.340655759 +0000 UTC m=+1039.604990707" Nov 03 00:58:11.363231 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:58:11.363206 4939 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-54df68-64c7497868-srpmr"] Nov 03 00:58:11.366559 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:58:11.366539 4939 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-54df68-64c7497868-srpmr"] Nov 03 00:58:11.379209 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:58:11.379186 4939 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-54df68-5d998485f9-pknfz"] Nov 03 00:58:11.388162 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:58:11.388138 4939 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-54df68-5d998485f9-pknfz"] Nov 03 00:58:11.497882 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:58:11.497846 4939 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-a-07f5f6-openshift-default-98746c89f-vz567"] Nov 03 00:58:11.498524 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:58:11.498507 4939 cpu_manager.go:414] "containerMap: removing container" podUID="fbcbf75f-badb-46a6-9f2d-3204afc53907" containerName="payload-pre-processing" Nov 03 00:58:11.498524 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:58:11.498524 4939 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fbcbf75f-badb-46a6-9f2d-3204afc53907" containerName="payload-pre-processing" Nov 03 00:58:11.498683 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:58:11.498533 4939 cpu_manager.go:414] "containerMap: removing container" podUID="ad1a1a5c-cb63-49e9-af61-26a6cd45cb8a" containerName="maas-api" Nov 03 00:58:11.498683 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:58:11.498538 4939 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ad1a1a5c-cb63-49e9-af61-26a6cd45cb8a" containerName="maas-api" Nov 03 00:58:11.498683 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:58:11.498673 4939 memory_manager.go:365] "RemoveStaleState removing state" podUID="fbcbf75f-badb-46a6-9f2d-3204afc53907" containerName="payload-pre-processing" Nov 03 00:58:11.498683 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:58:11.498683 4939 memory_manager.go:365] "RemoveStaleState removing state" podUID="ad1a1a5c-cb63-49e9-af61-26a6cd45cb8a" containerName="maas-api" Nov 03 00:58:11.503817 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:58:11.503798 4939 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-07f5f6-openshift-default-98746c89f-vz567" Nov 03 00:58:11.526270 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:58:11.526189 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ef4163bb-8632-4675-a678-11ec6d58e737-istio-data\") pod \"e2e-ipp-a-07f5f6-openshift-default-98746c89f-vz567\" (UID: \"ef4163bb-8632-4675-a678-11ec6d58e737\") " pod="openshift-ingress/e2e-ipp-a-07f5f6-openshift-default-98746c89f-vz567" Nov 03 00:58:11.526270 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:58:11.526245 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zccwt\" (UniqueName: \"kubernetes.io/projected/ef4163bb-8632-4675-a678-11ec6d58e737-kube-api-access-zccwt\") pod \"e2e-ipp-a-07f5f6-openshift-default-98746c89f-vz567\" (UID: \"ef4163bb-8632-4675-a678-11ec6d58e737\") " pod="openshift-ingress/e2e-ipp-a-07f5f6-openshift-default-98746c89f-vz567" Nov 03 00:58:11.526429 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:58:11.526412 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ef4163bb-8632-4675-a678-11ec6d58e737-credential-socket\") pod \"e2e-ipp-a-07f5f6-openshift-default-98746c89f-vz567\" (UID: \"ef4163bb-8632-4675-a678-11ec6d58e737\") " pod="openshift-ingress/e2e-ipp-a-07f5f6-openshift-default-98746c89f-vz567" Nov 03 00:58:11.526517 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:58:11.526441 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ef4163bb-8632-4675-a678-11ec6d58e737-workload-certs\") pod \"e2e-ipp-a-07f5f6-openshift-default-98746c89f-vz567\" (UID: \"ef4163bb-8632-4675-a678-11ec6d58e737\") " pod="openshift-ingress/e2e-ipp-a-07f5f6-openshift-default-98746c89f-vz567" Nov 03 00:58:11.526576 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:58:11.526556 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ef4163bb-8632-4675-a678-11ec6d58e737-istio-envoy\") pod \"e2e-ipp-a-07f5f6-openshift-default-98746c89f-vz567\" (UID: \"ef4163bb-8632-4675-a678-11ec6d58e737\") " pod="openshift-ingress/e2e-ipp-a-07f5f6-openshift-default-98746c89f-vz567" Nov 03 00:58:11.526705 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:58:11.526685 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ef4163bb-8632-4675-a678-11ec6d58e737-workload-socket\") pod \"e2e-ipp-a-07f5f6-openshift-default-98746c89f-vz567\" (UID: \"ef4163bb-8632-4675-a678-11ec6d58e737\") " pod="openshift-ingress/e2e-ipp-a-07f5f6-openshift-default-98746c89f-vz567" Nov 03 00:58:11.526774 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:58:11.526727 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ef4163bb-8632-4675-a678-11ec6d58e737-istio-token\") pod \"e2e-ipp-a-07f5f6-openshift-default-98746c89f-vz567\" (UID: \"ef4163bb-8632-4675-a678-11ec6d58e737\") " pod="openshift-ingress/e2e-ipp-a-07f5f6-openshift-default-98746c89f-vz567" Nov 03 00:58:11.526774 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:58:11.526761 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ef4163bb-8632-4675-a678-11ec6d58e737-istiod-ca-cert\") pod \"e2e-ipp-a-07f5f6-openshift-default-98746c89f-vz567\" (UID: \"ef4163bb-8632-4675-a678-11ec6d58e737\") " pod="openshift-ingress/e2e-ipp-a-07f5f6-openshift-default-98746c89f-vz567" Nov 03 00:58:11.526882 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:58:11.526787 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ef4163bb-8632-4675-a678-11ec6d58e737-istio-podinfo\") pod \"e2e-ipp-a-07f5f6-openshift-default-98746c89f-vz567\" (UID: \"ef4163bb-8632-4675-a678-11ec6d58e737\") " pod="openshift-ingress/e2e-ipp-a-07f5f6-openshift-default-98746c89f-vz567" Nov 03 00:58:11.531575 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:58:11.531552 4939 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ipp-a-07f5f6-openshift-default-dockercfg-6m8gj\"" Nov 03 00:58:11.533899 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:58:11.533876 4939 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-07f5f6-openshift-default-98746c89f-vz567"] Nov 03 00:58:11.647933 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:58:11.641869 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ef4163bb-8632-4675-a678-11ec6d58e737-credential-socket\") pod \"e2e-ipp-a-07f5f6-openshift-default-98746c89f-vz567\" (UID: \"ef4163bb-8632-4675-a678-11ec6d58e737\") " pod="openshift-ingress/e2e-ipp-a-07f5f6-openshift-default-98746c89f-vz567" Nov 03 00:58:11.647933 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:58:11.641939 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ef4163bb-8632-4675-a678-11ec6d58e737-workload-certs\") pod \"e2e-ipp-a-07f5f6-openshift-default-98746c89f-vz567\" (UID: \"ef4163bb-8632-4675-a678-11ec6d58e737\") " pod="openshift-ingress/e2e-ipp-a-07f5f6-openshift-default-98746c89f-vz567" Nov 03 00:58:11.647933 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:58:11.642039 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ef4163bb-8632-4675-a678-11ec6d58e737-istio-envoy\") pod \"e2e-ipp-a-07f5f6-openshift-default-98746c89f-vz567\" (UID: \"ef4163bb-8632-4675-a678-11ec6d58e737\") " pod="openshift-ingress/e2e-ipp-a-07f5f6-openshift-default-98746c89f-vz567" Nov 03 00:58:11.647933 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:58:11.642166 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ef4163bb-8632-4675-a678-11ec6d58e737-workload-socket\") pod \"e2e-ipp-a-07f5f6-openshift-default-98746c89f-vz567\" (UID: \"ef4163bb-8632-4675-a678-11ec6d58e737\") " pod="openshift-ingress/e2e-ipp-a-07f5f6-openshift-default-98746c89f-vz567" Nov 03 00:58:11.647933 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:58:11.642202 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ef4163bb-8632-4675-a678-11ec6d58e737-istio-token\") pod \"e2e-ipp-a-07f5f6-openshift-default-98746c89f-vz567\" (UID: \"ef4163bb-8632-4675-a678-11ec6d58e737\") " pod="openshift-ingress/e2e-ipp-a-07f5f6-openshift-default-98746c89f-vz567" Nov 03 00:58:11.647933 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:58:11.642233 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ef4163bb-8632-4675-a678-11ec6d58e737-istiod-ca-cert\") pod \"e2e-ipp-a-07f5f6-openshift-default-98746c89f-vz567\" (UID: \"ef4163bb-8632-4675-a678-11ec6d58e737\") " pod="openshift-ingress/e2e-ipp-a-07f5f6-openshift-default-98746c89f-vz567" Nov 03 00:58:11.647933 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:58:11.642257 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ef4163bb-8632-4675-a678-11ec6d58e737-istio-podinfo\") pod \"e2e-ipp-a-07f5f6-openshift-default-98746c89f-vz567\" (UID: \"ef4163bb-8632-4675-a678-11ec6d58e737\") " pod="openshift-ingress/e2e-ipp-a-07f5f6-openshift-default-98746c89f-vz567" Nov 03 00:58:11.647933 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:58:11.642350 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ef4163bb-8632-4675-a678-11ec6d58e737-istio-data\") pod \"e2e-ipp-a-07f5f6-openshift-default-98746c89f-vz567\" (UID: \"ef4163bb-8632-4675-a678-11ec6d58e737\") " pod="openshift-ingress/e2e-ipp-a-07f5f6-openshift-default-98746c89f-vz567" Nov 03 00:58:11.647933 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:58:11.642418 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zccwt\" (UniqueName: \"kubernetes.io/projected/ef4163bb-8632-4675-a678-11ec6d58e737-kube-api-access-zccwt\") pod \"e2e-ipp-a-07f5f6-openshift-default-98746c89f-vz567\" (UID: \"ef4163bb-8632-4675-a678-11ec6d58e737\") " pod="openshift-ingress/e2e-ipp-a-07f5f6-openshift-default-98746c89f-vz567" Nov 03 00:58:11.673740 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:58:11.671955 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 00:58:11.673740 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:58:11.672407 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ef4163bb-8632-4675-a678-11ec6d58e737-credential-socket\") pod \"e2e-ipp-a-07f5f6-openshift-default-98746c89f-vz567\" (UID: \"ef4163bb-8632-4675-a678-11ec6d58e737\") " pod="openshift-ingress/e2e-ipp-a-07f5f6-openshift-default-98746c89f-vz567" Nov 03 00:58:11.673740 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:58:11.672792 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ef4163bb-8632-4675-a678-11ec6d58e737-workload-certs\") pod \"e2e-ipp-a-07f5f6-openshift-default-98746c89f-vz567\" (UID: \"ef4163bb-8632-4675-a678-11ec6d58e737\") " pod="openshift-ingress/e2e-ipp-a-07f5f6-openshift-default-98746c89f-vz567" Nov 03 00:58:11.673740 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:58:11.673167 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ef4163bb-8632-4675-a678-11ec6d58e737-istiod-ca-cert\") pod \"e2e-ipp-a-07f5f6-openshift-default-98746c89f-vz567\" (UID: \"ef4163bb-8632-4675-a678-11ec6d58e737\") " pod="openshift-ingress/e2e-ipp-a-07f5f6-openshift-default-98746c89f-vz567" Nov 03 00:58:11.673740 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:58:11.673425 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ef4163bb-8632-4675-a678-11ec6d58e737-workload-socket\") pod \"e2e-ipp-a-07f5f6-openshift-default-98746c89f-vz567\" (UID: \"ef4163bb-8632-4675-a678-11ec6d58e737\") " pod="openshift-ingress/e2e-ipp-a-07f5f6-openshift-default-98746c89f-vz567" Nov 03 00:58:11.673740 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:58:11.673684 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ef4163bb-8632-4675-a678-11ec6d58e737-istio-data\") pod \"e2e-ipp-a-07f5f6-openshift-default-98746c89f-vz567\" (UID: \"ef4163bb-8632-4675-a678-11ec6d58e737\") " pod="openshift-ingress/e2e-ipp-a-07f5f6-openshift-default-98746c89f-vz567" Nov 03 00:58:11.723151 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:58:11.717985 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ef4163bb-8632-4675-a678-11ec6d58e737-istio-podinfo\") pod \"e2e-ipp-a-07f5f6-openshift-default-98746c89f-vz567\" (UID: \"ef4163bb-8632-4675-a678-11ec6d58e737\") " pod="openshift-ingress/e2e-ipp-a-07f5f6-openshift-default-98746c89f-vz567" Nov 03 00:58:11.759944 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:58:11.746083 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ef4163bb-8632-4675-a678-11ec6d58e737-istio-token\") pod \"e2e-ipp-a-07f5f6-openshift-default-98746c89f-vz567\" (UID: \"ef4163bb-8632-4675-a678-11ec6d58e737\") " pod="openshift-ingress/e2e-ipp-a-07f5f6-openshift-default-98746c89f-vz567" Nov 03 00:58:11.760726 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:58:11.760661 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ef4163bb-8632-4675-a678-11ec6d58e737-istio-envoy\") pod \"e2e-ipp-a-07f5f6-openshift-default-98746c89f-vz567\" (UID: \"ef4163bb-8632-4675-a678-11ec6d58e737\") " pod="openshift-ingress/e2e-ipp-a-07f5f6-openshift-default-98746c89f-vz567" Nov 03 00:58:11.768926 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:58:11.765862 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zccwt\" (UniqueName: \"kubernetes.io/projected/ef4163bb-8632-4675-a678-11ec6d58e737-kube-api-access-zccwt\") pod \"e2e-ipp-a-07f5f6-openshift-default-98746c89f-vz567\" (UID: \"ef4163bb-8632-4675-a678-11ec6d58e737\") " pod="openshift-ingress/e2e-ipp-a-07f5f6-openshift-default-98746c89f-vz567" Nov 03 00:58:11.819855 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:58:11.819494 4939 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-07f5f6-openshift-default-98746c89f-vz567" Nov 03 00:58:11.850688 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:58:11.850572 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 00:58:11.989873 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:58:11.989840 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 00:58:12.051957 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:58:12.048624 4939 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ad1a1a5c-cb63-49e9-af61-26a6cd45cb8a" path="/var/lib/kubelet/pods/ad1a1a5c-cb63-49e9-af61-26a6cd45cb8a/volumes" Nov 03 00:58:12.051957 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:58:12.049426 4939 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fbcbf75f-badb-46a6-9f2d-3204afc53907" path="/var/lib/kubelet/pods/fbcbf75f-badb-46a6-9f2d-3204afc53907/volumes" Nov 03 00:58:12.319236 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:58:12.319206 4939 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-07f5f6-openshift-default-98746c89f-vz567"] Nov 03 00:58:12.320086 ip-10-0-140-107 kubenswrapper[4939]: W1103 00:58:12.320058 4939 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podef4163bb_8632_4675_a678_11ec6d58e737.slice/crio-55f7f089ab857a160ad99afb348a8f5471ddc494f3adf5e72c56d6d613631322 WatchSource:0}: Error finding container 55f7f089ab857a160ad99afb348a8f5471ddc494f3adf5e72c56d6d613631322: Status 404 returned error can't find the container with id 55f7f089ab857a160ad99afb348a8f5471ddc494f3adf5e72c56d6d613631322 Nov 03 00:58:12.323176 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:58:12.323140 4939 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 00:58:12.323286 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:58:12.323205 4939 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 00:58:12.323286 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:58:12.323237 4939 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 00:58:12.357760 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:58:12.357723 4939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-07f5f6-openshift-default-98746c89f-vz567" event={"ID":"ef4163bb-8632-4675-a678-11ec6d58e737","Type":"ContainerStarted","Data":"55f7f089ab857a160ad99afb348a8f5471ddc494f3adf5e72c56d6d613631322"} Nov 03 00:58:12.469406 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:58:12.469374 4939 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7034-3d9226"] Nov 03 00:58:12.475689 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:58:12.475603 4939 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7034-3d9226"] Nov 03 00:58:12.968234 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:58:12.968203 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 00:58:13.090627 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:58:13.090590 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 00:58:13.243150 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:58:13.243081 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 00:58:13.367541 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:58:13.367504 4939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-07f5f6-openshift-default-98746c89f-vz567" event={"ID":"ef4163bb-8632-4675-a678-11ec6d58e737","Type":"ContainerStarted","Data":"e5e40ad69efe3af5e0379a278b807446fbeb02e1055b85c53a413a32b985a78b"} Nov 03 00:58:13.367810 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:58:13.367780 4939 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/test-curl-7034-3d9226" podUID="cc7c990a-c21b-4046-8c45-aba9b48114e4" containerName="test-curl-7034-3d9226" containerID="cri-o://166013e6cce52f91e840e14cd10348a90f81c7c5478549744474f543059257a3" gracePeriod=2 Nov 03 00:58:13.373149 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:58:13.373124 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 00:58:13.391268 ip-10-0-140-107 kubenswrapper[4939]: E1103 00:58:13.391240 4939 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7034-3d9226\" is forbidden: User \"system:node:ip-10-0-140-107.us-east-2.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-140-107.us-east-2.compute.internal' and this object" podUID="cc7c990a-c21b-4046-8c45-aba9b48114e4" pod="opendatahub/test-curl-7034-3d9226" Nov 03 00:58:13.392517 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:58:13.392481 4939 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-a-07f5f6-openshift-default-98746c89f-vz567" podStartSLOduration=2.392469272 podStartE2EDuration="2.392469272s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:58:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:58:13.388462978 +0000 UTC m=+1041.652797926" watchObservedRunningTime="2026-11-03 00:58:13.392469272 +0000 UTC m=+1041.656804218" Nov 03 00:58:13.786769 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:58:13.786734 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 00:58:13.829874 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:58:13.822931 4939 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-a-07f5f6-openshift-default-98746c89f-vz567" Nov 03 00:58:13.850942 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:58:13.845245 4939 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-a-07f5f6-openshift-default-98746c89f-vz567" Nov 03 00:58:13.877421 ip-10-0-140-107 kubenswrapper[4939]: E1103 00:58:13.875346 4939 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7034-3d9226\" is forbidden: User \"system:node:ip-10-0-140-107.us-east-2.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-140-107.us-east-2.compute.internal' and this object" podUID="cc7c990a-c21b-4046-8c45-aba9b48114e4" pod="opendatahub/test-curl-7034-3d9226" Nov 03 00:58:13.946898 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:58:13.946861 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 00:58:14.082469 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:58:14.082438 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 00:58:14.212931 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:58:14.208820 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 00:58:14.423065 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:58:14.410191 4939 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-a-07f5f6-openshift-default-98746c89f-vz567" Nov 03 00:58:14.464928 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:58:14.452250 4939 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-a-07f5f6-openshift-default-98746c89f-vz567" Nov 03 00:58:14.879199 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:58:14.879168 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 00:58:15.249131 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:58:15.249053 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 00:58:15.345189 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:58:15.345151 4939 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-eed369-7d94fb8545-hdldv"] Nov 03 00:58:15.346238 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:58:15.346213 4939 cpu_manager.go:414] "containerMap: removing container" podUID="cc7c990a-c21b-4046-8c45-aba9b48114e4" containerName="test-curl-7034-3d9226" Nov 03 00:58:15.346238 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:58:15.346235 4939 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cc7c990a-c21b-4046-8c45-aba9b48114e4" containerName="test-curl-7034-3d9226" Nov 03 00:58:15.346452 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:58:15.346437 4939 memory_manager.go:365] "RemoveStaleState removing state" podUID="cc7c990a-c21b-4046-8c45-aba9b48114e4" containerName="test-curl-7034-3d9226" Nov 03 00:58:15.351563 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:58:15.351276 4939 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-eed369-7d94fb8545-hdldv" Nov 03 00:58:15.357873 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:58:15.357392 4939 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-eed369-7d94fb8545-hdldv"] Nov 03 00:58:15.459935 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:58:15.437712 4939 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-b-eed369-dockercfg-z5xf5\"" Nov 03 00:58:15.459935 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:58:15.437809 4939 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-b-eed369\"" Nov 03 00:58:15.520988 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:58:15.519652 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 00:58:15.540197 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:58:15.532719 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7r269\" (UniqueName: \"kubernetes.io/projected/0ae23a7b-9f49-479e-bfd5-7732c28052aa-kube-api-access-7r269\") pod \"payload-processing-e2e-api-b-eed369-7d94fb8545-hdldv\" (UID: \"0ae23a7b-9f49-479e-bfd5-7732c28052aa\") " pod="openshift-ingress/payload-processing-e2e-api-b-eed369-7d94fb8545-hdldv" Nov 03 00:58:15.540197 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:58:15.533006 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0ae23a7b-9f49-479e-bfd5-7732c28052aa-plugins-config-volume\") pod \"payload-processing-e2e-api-b-eed369-7d94fb8545-hdldv\" (UID: \"0ae23a7b-9f49-479e-bfd5-7732c28052aa\") " pod="openshift-ingress/payload-processing-e2e-api-b-eed369-7d94fb8545-hdldv" Nov 03 00:58:15.681446 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:58:15.681404 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0ae23a7b-9f49-479e-bfd5-7732c28052aa-plugins-config-volume\") pod \"payload-processing-e2e-api-b-eed369-7d94fb8545-hdldv\" (UID: \"0ae23a7b-9f49-479e-bfd5-7732c28052aa\") " pod="openshift-ingress/payload-processing-e2e-api-b-eed369-7d94fb8545-hdldv" Nov 03 00:58:15.681992 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:58:15.681963 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7r269\" (UniqueName: \"kubernetes.io/projected/0ae23a7b-9f49-479e-bfd5-7732c28052aa-kube-api-access-7r269\") pod \"payload-processing-e2e-api-b-eed369-7d94fb8545-hdldv\" (UID: \"0ae23a7b-9f49-479e-bfd5-7732c28052aa\") " pod="openshift-ingress/payload-processing-e2e-api-b-eed369-7d94fb8545-hdldv" Nov 03 00:58:15.684091 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:58:15.684054 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0ae23a7b-9f49-479e-bfd5-7732c28052aa-plugins-config-volume\") pod \"payload-processing-e2e-api-b-eed369-7d94fb8545-hdldv\" (UID: \"0ae23a7b-9f49-479e-bfd5-7732c28052aa\") " pod="openshift-ingress/payload-processing-e2e-api-b-eed369-7d94fb8545-hdldv" Nov 03 00:58:15.738377 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:58:15.720551 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7r269\" (UniqueName: \"kubernetes.io/projected/0ae23a7b-9f49-479e-bfd5-7732c28052aa-kube-api-access-7r269\") pod \"payload-processing-e2e-api-b-eed369-7d94fb8545-hdldv\" (UID: \"0ae23a7b-9f49-479e-bfd5-7732c28052aa\") " pod="openshift-ingress/payload-processing-e2e-api-b-eed369-7d94fb8545-hdldv" Nov 03 00:58:15.774135 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:58:15.774028 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 00:58:15.903849 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:58:15.903818 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 00:58:15.904685 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:58:15.904666 4939 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7034-3d9226" Nov 03 00:58:15.970976 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:58:15.970946 4939 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-eed369-7d94fb8545-hdldv" Nov 03 00:58:15.986467 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:58:15.986439 4939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cc7c990a-c21b-4046-8c45-aba9b48114e4-kube-api-access-hc9rl\" (UniqueName: \"kubernetes.io/projected/cc7c990a-c21b-4046-8c45-aba9b48114e4-kube-api-access-hc9rl\") pod \"cc7c990a-c21b-4046-8c45-aba9b48114e4\" (UID: \"cc7c990a-c21b-4046-8c45-aba9b48114e4\") " Nov 03 00:58:15.988528 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:58:15.988500 4939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cc7c990a-c21b-4046-8c45-aba9b48114e4-kube-api-access-hc9rl" pod "cc7c990a-c21b-4046-8c45-aba9b48114e4" (UID: "cc7c990a-c21b-4046-8c45-aba9b48114e4"). InnerVolumeSpecName "kube-api-access-hc9rl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 00:58:16.089411 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:58:16.089369 4939 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hc9rl\" (UniqueName: \"kubernetes.io/projected/cc7c990a-c21b-4046-8c45-aba9b48114e4-kube-api-access-hc9rl\") on node \"ip-10-0-140-107.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 00:58:16.251734 ip-10-0-140-107 kubenswrapper[4939]: W1103 00:58:16.251706 4939 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0ae23a7b_9f49_479e_bfd5_7732c28052aa.slice/crio-c69a6cac3df3c53965f661d4ccd09ded7a1de7acabbf48aa18c34a4d8e1446e9 WatchSource:0}: Error finding container c69a6cac3df3c53965f661d4ccd09ded7a1de7acabbf48aa18c34a4d8e1446e9: Status 404 returned error can't find the container with id c69a6cac3df3c53965f661d4ccd09ded7a1de7acabbf48aa18c34a4d8e1446e9 Nov 03 00:58:16.252223 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:58:16.252195 4939 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-eed369-7d94fb8545-hdldv"] Nov 03 00:58:16.402502 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:58:16.402466 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 00:58:16.593932 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:58:16.585650 4939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-eed369-7d94fb8545-hdldv" event={"ID":"0ae23a7b-9f49-479e-bfd5-7732c28052aa","Type":"ContainerStarted","Data":"eb5f271d302c42f074bf45c32cdcdfb927cc0ddd6bb8a344638107571d67bed6"} Nov 03 00:58:16.593932 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:58:16.585700 4939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-eed369-7d94fb8545-hdldv" event={"ID":"0ae23a7b-9f49-479e-bfd5-7732c28052aa","Type":"ContainerStarted","Data":"c69a6cac3df3c53965f661d4ccd09ded7a1de7acabbf48aa18c34a4d8e1446e9"} Nov 03 00:58:16.593932 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:58:16.585812 4939 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-api-b-eed369-7d94fb8545-hdldv" Nov 03 00:58:16.593932 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:58:16.587369 4939 generic.go:356] "Generic (PLEG): container finished" podID="cc7c990a-c21b-4046-8c45-aba9b48114e4" containerID="166013e6cce52f91e840e14cd10348a90f81c7c5478549744474f543059257a3" exitCode=137 Nov 03 00:58:16.593932 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:58:16.588657 4939 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7034-3d9226" Nov 03 00:58:16.593932 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:58:16.592760 4939 scope.go:122] "RemoveContainer" containerID="166013e6cce52f91e840e14cd10348a90f81c7c5478549744474f543059257a3" Nov 03 00:58:16.615838 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:58:16.615784 4939 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-api-b-eed369-7d94fb8545-hdldv" podStartSLOduration=1.615765515 podStartE2EDuration="1.615765515s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:58:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:58:16.608859759 +0000 UTC m=+1044.873194706" watchObservedRunningTime="2026-11-03 00:58:16.615765515 +0000 UTC m=+1044.880100461" Nov 03 00:58:16.617859 ip-10-0-140-107 kubenswrapper[4939]: E1103 00:58:16.617826 4939 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7034-3d9226\" is forbidden: User \"system:node:ip-10-0-140-107.us-east-2.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-140-107.us-east-2.compute.internal' and this object" podUID="cc7c990a-c21b-4046-8c45-aba9b48114e4" pod="opendatahub/test-curl-7034-3d9226" Nov 03 00:58:16.618953 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:58:16.618932 4939 scope.go:122] "RemoveContainer" containerID="166013e6cce52f91e840e14cd10348a90f81c7c5478549744474f543059257a3" Nov 03 00:58:16.619352 ip-10-0-140-107 kubenswrapper[4939]: E1103 00:58:16.619330 4939 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"166013e6cce52f91e840e14cd10348a90f81c7c5478549744474f543059257a3\": container with ID starting with 166013e6cce52f91e840e14cd10348a90f81c7c5478549744474f543059257a3 not found: ID does not exist" containerID="166013e6cce52f91e840e14cd10348a90f81c7c5478549744474f543059257a3" Nov 03 00:58:16.619426 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:58:16.619365 4939 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"166013e6cce52f91e840e14cd10348a90f81c7c5478549744474f543059257a3"} err="failed to get container status \"166013e6cce52f91e840e14cd10348a90f81c7c5478549744474f543059257a3\": rpc error: code = NotFound desc = could not find container \"166013e6cce52f91e840e14cd10348a90f81c7c5478549744474f543059257a3\": container with ID starting with 166013e6cce52f91e840e14cd10348a90f81c7c5478549744474f543059257a3 not found: ID does not exist" Nov 03 00:58:17.062697 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:58:17.062606 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 00:58:17.071136 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:58:17.071105 4939 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-07f5f6-f97497957-m28k9"] Nov 03 00:58:17.076820 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:58:17.076799 4939 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-07f5f6-f97497957-m28k9" Nov 03 00:58:17.090739 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:58:17.090717 4939 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-a-07f5f6-dockercfg-q7wz8\"" Nov 03 00:58:17.095816 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:58:17.095793 4939 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-a-07f5f6\"" Nov 03 00:58:17.095959 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:58:17.095823 4939 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-07f5f6-f97497957-m28k9"] Nov 03 00:58:17.105043 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:58:17.105014 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ncjht\" (UniqueName: \"kubernetes.io/projected/090b138f-b622-4f99-bd4a-16d49430ba5e-kube-api-access-ncjht\") pod \"payload-processing-e2e-ipp-a-07f5f6-f97497957-m28k9\" (UID: \"090b138f-b622-4f99-bd4a-16d49430ba5e\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-07f5f6-f97497957-m28k9" Nov 03 00:58:17.105235 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:58:17.105213 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/090b138f-b622-4f99-bd4a-16d49430ba5e-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-07f5f6-f97497957-m28k9\" (UID: \"090b138f-b622-4f99-bd4a-16d49430ba5e\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-07f5f6-f97497957-m28k9" Nov 03 00:58:17.116276 ip-10-0-140-107 kubenswrapper[4939]: E1103 00:58:17.116244 4939 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7034-3d9226\" is forbidden: User \"system:node:ip-10-0-140-107.us-east-2.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-140-107.us-east-2.compute.internal' and this object" podUID="cc7c990a-c21b-4046-8c45-aba9b48114e4" pod="opendatahub/test-curl-7034-3d9226" Nov 03 00:58:17.258930 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:58:17.255288 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ncjht\" (UniqueName: \"kubernetes.io/projected/090b138f-b622-4f99-bd4a-16d49430ba5e-kube-api-access-ncjht\") pod \"payload-processing-e2e-ipp-a-07f5f6-f97497957-m28k9\" (UID: \"090b138f-b622-4f99-bd4a-16d49430ba5e\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-07f5f6-f97497957-m28k9" Nov 03 00:58:17.258930 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:58:17.255519 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/090b138f-b622-4f99-bd4a-16d49430ba5e-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-07f5f6-f97497957-m28k9\" (UID: \"090b138f-b622-4f99-bd4a-16d49430ba5e\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-07f5f6-f97497957-m28k9" Nov 03 00:58:17.258930 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:58:17.256471 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/090b138f-b622-4f99-bd4a-16d49430ba5e-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-07f5f6-f97497957-m28k9\" (UID: \"090b138f-b622-4f99-bd4a-16d49430ba5e\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-07f5f6-f97497957-m28k9" Nov 03 00:58:17.258930 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:58:17.257947 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 00:58:17.333976 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:58:17.333629 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ncjht\" (UniqueName: \"kubernetes.io/projected/090b138f-b622-4f99-bd4a-16d49430ba5e-kube-api-access-ncjht\") pod \"payload-processing-e2e-ipp-a-07f5f6-f97497957-m28k9\" (UID: \"090b138f-b622-4f99-bd4a-16d49430ba5e\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-07f5f6-f97497957-m28k9" Nov 03 00:58:17.391237 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:58:17.391201 4939 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-07f5f6-f97497957-m28k9" Nov 03 00:58:17.672481 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:58:17.672359 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 00:58:17.796822 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:58:17.796791 4939 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-07f5f6-f97497957-m28k9"] Nov 03 00:58:17.797074 ip-10-0-140-107 kubenswrapper[4939]: W1103 00:58:17.797051 4939 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod090b138f_b622_4f99_bd4a_16d49430ba5e.slice/crio-9b50d6056d3020d36ae223d7b12a1307417069cdf70297de58dc1ac512090974 WatchSource:0}: Error finding container 9b50d6056d3020d36ae223d7b12a1307417069cdf70297de58dc1ac512090974: Status 404 returned error can't find the container with id 9b50d6056d3020d36ae223d7b12a1307417069cdf70297de58dc1ac512090974 Nov 03 00:58:17.869646 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:58:17.869615 4939 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cc7c990a-c21b-4046-8c45-aba9b48114e4" path="/var/lib/kubelet/pods/cc7c990a-c21b-4046-8c45-aba9b48114e4/volumes" Nov 03 00:58:17.926051 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:58:17.925968 4939 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-mt-dd03e395/e2e-lifecycle-model-dd03e395-kserve-78979759df-b8v9c"] Nov 03 00:58:18.021783 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:58:18.021753 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 00:58:18.054334 ip-10-0-140-107 kubenswrapper[4939]: E1103 00:58:18.054293 4939 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-sthjh" containerName="extractor" Nov 03 00:58:18.433180 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:58:18.433151 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 00:58:18.619002 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:58:18.618967 4939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-07f5f6-f97497957-m28k9" event={"ID":"090b138f-b622-4f99-bd4a-16d49430ba5e","Type":"ContainerStarted","Data":"b92dd3a75fcc687c734ce7f2f54491cc13aea28b8c7135a76ecafe5bbdd674fe"} Nov 03 00:58:18.619200 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:58:18.619025 4939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-07f5f6-f97497957-m28k9" event={"ID":"090b138f-b622-4f99-bd4a-16d49430ba5e","Type":"ContainerStarted","Data":"9b50d6056d3020d36ae223d7b12a1307417069cdf70297de58dc1ac512090974"} Nov 03 00:58:18.619200 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:58:18.619057 4939 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-07f5f6-f97497957-m28k9" Nov 03 00:58:18.635776 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:58:18.635719 4939 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-a-07f5f6-f97497957-m28k9" podStartSLOduration=1.635702353 podStartE2EDuration="1.635702353s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:58:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:58:18.634785138 +0000 UTC m=+1046.899120075" watchObservedRunningTime="2026-11-03 00:58:18.635702353 +0000 UTC m=+1046.900037301" Nov 03 00:58:19.271919 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:58:19.271873 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 00:58:19.940768 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:58:19.940688 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 00:58:20.511689 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:58:20.511651 4939 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-a2d8e5-5bb4464ff9-fpt8c"] Nov 03 00:58:20.614433 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:58:20.614397 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 00:58:20.731737 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:58:20.731702 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 00:58:21.485542 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:58:21.485508 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 00:58:21.593893 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:58:21.593862 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 00:58:21.866180 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:58:21.866149 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 00:58:22.531730 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:58:22.531697 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 00:58:23.222236 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:58:23.222197 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 00:58:23.900108 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:58:23.900077 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 00:58:24.041597 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:58:24.041545 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 00:58:24.152713 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:58:24.152586 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 00:58:24.658432 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:58:24.658394 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 00:58:25.349077 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:58:25.349045 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 00:58:25.514103 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:58:25.514059 4939 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-api-b-a2d8e5-5bb4464ff9-fpt8c" podUID="916506b5-c34f-479e-a192-1a4c2e5714ee" containerName="payload-processing" containerID="cri-o://e8eb3e2eb562ca97580a86e3b1f1029dc27471fca9c41e286f494aab12aee0a0" gracePeriod=25 Nov 03 00:58:25.762617 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:58:25.762578 4939 generic.go:356] "Generic (PLEG): container finished" podID="916506b5-c34f-479e-a192-1a4c2e5714ee" containerID="e8eb3e2eb562ca97580a86e3b1f1029dc27471fca9c41e286f494aab12aee0a0" exitCode=0 Nov 03 00:58:25.762788 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:58:25.762674 4939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-a2d8e5-5bb4464ff9-fpt8c" event={"ID":"916506b5-c34f-479e-a192-1a4c2e5714ee","Type":"ContainerDied","Data":"e8eb3e2eb562ca97580a86e3b1f1029dc27471fca9c41e286f494aab12aee0a0"} Nov 03 00:58:25.809537 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:58:25.809504 4939 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-a2d8e5-5bb4464ff9-fpt8c" Nov 03 00:58:25.896573 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:58:25.896493 4939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/916506b5-c34f-479e-a192-1a4c2e5714ee-kube-api-access-mjq48\" (UniqueName: \"kubernetes.io/projected/916506b5-c34f-479e-a192-1a4c2e5714ee-kube-api-access-mjq48\") pod \"916506b5-c34f-479e-a192-1a4c2e5714ee\" (UID: \"916506b5-c34f-479e-a192-1a4c2e5714ee\") " Nov 03 00:58:25.896743 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:58:25.896650 4939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/916506b5-c34f-479e-a192-1a4c2e5714ee-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/916506b5-c34f-479e-a192-1a4c2e5714ee-plugins-config-volume\") pod \"916506b5-c34f-479e-a192-1a4c2e5714ee\" (UID: \"916506b5-c34f-479e-a192-1a4c2e5714ee\") " Nov 03 00:58:25.897077 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:58:25.897050 4939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/916506b5-c34f-479e-a192-1a4c2e5714ee-plugins-config-volume" pod "916506b5-c34f-479e-a192-1a4c2e5714ee" (UID: "916506b5-c34f-479e-a192-1a4c2e5714ee"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 00:58:25.897477 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:58:25.897456 4939 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/916506b5-c34f-479e-a192-1a4c2e5714ee-plugins-config-volume\") on node \"ip-10-0-140-107.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 00:58:25.898702 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:58:25.898680 4939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/916506b5-c34f-479e-a192-1a4c2e5714ee-kube-api-access-mjq48" pod "916506b5-c34f-479e-a192-1a4c2e5714ee" (UID: "916506b5-c34f-479e-a192-1a4c2e5714ee"). InnerVolumeSpecName "kube-api-access-mjq48". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 00:58:25.999117 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:58:25.999085 4939 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mjq48\" (UniqueName: \"kubernetes.io/projected/916506b5-c34f-479e-a192-1a4c2e5714ee-kube-api-access-mjq48\") on node \"ip-10-0-140-107.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 00:58:26.027808 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:58:26.027777 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 00:58:26.566808 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:58:26.566760 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 00:58:26.803168 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:58:26.803129 4939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-a2d8e5-5bb4464ff9-fpt8c" event={"ID":"916506b5-c34f-479e-a192-1a4c2e5714ee","Type":"ContainerDied","Data":"4bd2739b9b5a129e58e81eecdc29b43f0ba295f2f1333c55058ba214472b74e1"} Nov 03 00:58:26.803490 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:58:26.803467 4939 scope.go:122] "RemoveContainer" containerID="e8eb3e2eb562ca97580a86e3b1f1029dc27471fca9c41e286f494aab12aee0a0" Nov 03 00:58:26.803791 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:58:26.803772 4939 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-a2d8e5-5bb4464ff9-fpt8c" Nov 03 00:58:26.895217 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:58:26.894779 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 00:58:26.912201 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:58:26.912164 4939 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-a2d8e5-5bb4464ff9-fpt8c"] Nov 03 00:58:26.917397 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:58:26.917171 4939 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-a2d8e5-5bb4464ff9-fpt8c"] Nov 03 00:58:27.050928 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:58:27.035547 4939 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-4pw7v"] Nov 03 00:58:27.612311 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:58:27.612284 4939 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-api-b-eed369-7d94fb8545-hdldv" Nov 03 00:58:27.691203 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:58:27.691168 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 00:58:27.870886 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:58:27.870809 4939 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="916506b5-c34f-479e-a192-1a4c2e5714ee" path="/var/lib/kubelet/pods/916506b5-c34f-479e-a192-1a4c2e5714ee/volumes" Nov 03 00:58:27.875182 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:58:27.875147 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 00:58:27.988557 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:58:27.988517 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 00:58:28.689937 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:58:28.686395 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 00:58:29.030475 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:58:29.030380 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 00:58:29.672615 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:58:29.672569 4939 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-2f9a3a/simulator-w0-2f9a3a-kserve-9596bbb8f-2trgz" podUID="207e2207-fb6c-4076-95df-72e95671c261" containerName="main" probeResult="failure" output="Get \"https://10.132.0.61:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Nov 03 00:58:29.700106 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:58:29.700059 4939 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-83755a/distinct-2-w3-83755a-kserve-6cf66b995d-2l47b" podUID="c3b05e88-5d5b-4d03-b28d-832f22e3c868" containerName="main" probeResult="failure" output="Get \"https://10.132.0.72:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Nov 03 00:58:29.780490 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:58:29.780401 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 00:58:29.793308 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:58:29.793274 4939 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-07f5f6-f97497957-m28k9" Nov 03 00:58:29.932982 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:58:29.926946 4939 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-88bbf7/distinct-2-w1-88bbf7-kserve-5457cdcdbd-l6tfd" podUID="a397931b-faa9-4c2e-8290-491350546663" containerName="main" probeResult="failure" output="Get \"https://10.132.0.85:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 00:58:29.948171 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:58:29.948115 4939 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-f7680d/distinct-w2-f7680d-kserve-756f874b98-qnpv8" podUID="836753fc-f9ef-4f61-b3fe-e5b92b7ee27b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.70:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 00:58:30.124667 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:58:29.948448 4939 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-f7680d/premium-w2-f7680d-kserve-76565b8cf9-hqbbt" podUID="4712e708-18a2-4a13-a12c-a5669397b2c3" containerName="main" probeResult="failure" output="Get \"https://10.132.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 00:58:30.125547 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:58:30.104875 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 00:58:30.400440 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:58:30.400274 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 00:58:30.614938 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:58:30.610949 4939 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-596dc9545f-v5p6s container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.69:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 00:58:30.614938 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:58:30.611740 4939 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-v5p6s" podUID="e6aa4af1-9a04-4116-9d8a-1d8ef5b04958" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.69:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 00:58:30.940246 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:58:30.940188 4939 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-88bbf7/distinct-2-w1-88bbf7-kserve-5457cdcdbd-l6tfd" podUID="a397931b-faa9-4c2e-8290-491350546663" containerName="main" probeResult="failure" output="Get \"https://10.132.0.85:8000/ready\": context deadline exceeded" Nov 03 00:58:30.942268 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:58:30.942230 4939 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-88bbf7/distinct-2-w1-88bbf7-kserve-5457cdcdbd-l6tfd" Nov 03 00:58:30.976933 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:58:30.970455 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 00:58:31.469390 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:58:31.469357 4939 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-88bbf7/distinct-2-w1-88bbf7-kserve-5457cdcdbd-l6tfd" Nov 03 00:58:31.620363 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:58:31.620335 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 00:58:32.647454 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:58:32.647421 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 00:58:32.949643 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:58:32.949523 4939 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-mt-dd03e395/e2e-lifecycle-model-dd03e395-kserve-78979759df-b8v9c" podUID="e4803d7d-4603-4b14-950d-018f88294285" containerName="main" containerID="cri-o://80328c9985d797195afba46b194886bdb01d5ba00303ded6defced0175dcf411" gracePeriod=45 Nov 03 00:58:33.240071 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:58:33.240044 4939 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-dd03e395/e2e-lifecycle-model-dd03e395-kserve-78979759df-b8v9c" Nov 03 00:58:33.242175 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:58:33.242152 4939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e4803d7d-4603-4b14-950d-018f88294285-kube-api-access-qg4mj\" (UniqueName: \"kubernetes.io/projected/e4803d7d-4603-4b14-950d-018f88294285-kube-api-access-qg4mj\") pod \"e4803d7d-4603-4b14-950d-018f88294285\" (UID: \"e4803d7d-4603-4b14-950d-018f88294285\") " Nov 03 00:58:33.242272 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:58:33.242219 4939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e4803d7d-4603-4b14-950d-018f88294285-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e4803d7d-4603-4b14-950d-018f88294285-model-cache\") pod \"e4803d7d-4603-4b14-950d-018f88294285\" (UID: \"e4803d7d-4603-4b14-950d-018f88294285\") " Nov 03 00:58:33.242332 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:58:33.242266 4939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e4803d7d-4603-4b14-950d-018f88294285-tls-certs\" (UniqueName: \"kubernetes.io/secret/e4803d7d-4603-4b14-950d-018f88294285-tls-certs\") pod \"e4803d7d-4603-4b14-950d-018f88294285\" (UID: \"e4803d7d-4603-4b14-950d-018f88294285\") " Nov 03 00:58:33.242455 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:58:33.242438 4939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e4803d7d-4603-4b14-950d-018f88294285-home\" (UniqueName: \"kubernetes.io/empty-dir/e4803d7d-4603-4b14-950d-018f88294285-home\") pod \"e4803d7d-4603-4b14-950d-018f88294285\" (UID: \"e4803d7d-4603-4b14-950d-018f88294285\") " Nov 03 00:58:33.242519 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:58:33.242476 4939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e4803d7d-4603-4b14-950d-018f88294285-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e4803d7d-4603-4b14-950d-018f88294285-tmp-dir\") pod \"e4803d7d-4603-4b14-950d-018f88294285\" (UID: \"e4803d7d-4603-4b14-950d-018f88294285\") " Nov 03 00:58:33.242519 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:58:33.242494 4939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e4803d7d-4603-4b14-950d-018f88294285-dshm\" (UniqueName: \"kubernetes.io/empty-dir/e4803d7d-4603-4b14-950d-018f88294285-dshm\") pod \"e4803d7d-4603-4b14-950d-018f88294285\" (UID: \"e4803d7d-4603-4b14-950d-018f88294285\") " Nov 03 00:58:33.242623 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:58:33.242518 4939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e4803d7d-4603-4b14-950d-018f88294285-model-cache" pod "e4803d7d-4603-4b14-950d-018f88294285" (UID: "e4803d7d-4603-4b14-950d-018f88294285"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 00:58:33.243033 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:58:33.242740 4939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e4803d7d-4603-4b14-950d-018f88294285-home" pod "e4803d7d-4603-4b14-950d-018f88294285" (UID: "e4803d7d-4603-4b14-950d-018f88294285"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 00:58:33.243033 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:58:33.242768 4939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e4803d7d-4603-4b14-950d-018f88294285-tmp-dir" pod "e4803d7d-4603-4b14-950d-018f88294285" (UID: "e4803d7d-4603-4b14-950d-018f88294285"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 00:58:33.243250 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:58:33.243228 4939 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e4803d7d-4603-4b14-950d-018f88294285-tmp-dir\") on node \"ip-10-0-140-107.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 00:58:33.243313 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:58:33.243258 4939 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e4803d7d-4603-4b14-950d-018f88294285-model-cache\") on node \"ip-10-0-140-107.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 00:58:33.243313 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:58:33.243275 4939 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e4803d7d-4603-4b14-950d-018f88294285-home\") on node \"ip-10-0-140-107.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 00:58:33.244679 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:58:33.244654 4939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e4803d7d-4603-4b14-950d-018f88294285-kube-api-access-qg4mj" pod "e4803d7d-4603-4b14-950d-018f88294285" (UID: "e4803d7d-4603-4b14-950d-018f88294285"). InnerVolumeSpecName "kube-api-access-qg4mj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 00:58:33.244892 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:58:33.244871 4939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e4803d7d-4603-4b14-950d-018f88294285-tls-certs" pod "e4803d7d-4603-4b14-950d-018f88294285" (UID: "e4803d7d-4603-4b14-950d-018f88294285"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 00:58:33.244892 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:58:33.244875 4939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e4803d7d-4603-4b14-950d-018f88294285-dshm" pod "e4803d7d-4603-4b14-950d-018f88294285" (UID: "e4803d7d-4603-4b14-950d-018f88294285"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 00:58:33.345137 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:58:33.345109 4939 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e4803d7d-4603-4b14-950d-018f88294285-tls-certs\") on node \"ip-10-0-140-107.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 00:58:33.345137 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:58:33.345132 4939 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e4803d7d-4603-4b14-950d-018f88294285-dshm\") on node \"ip-10-0-140-107.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 00:58:33.345137 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:58:33.345143 4939 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qg4mj\" (UniqueName: \"kubernetes.io/projected/e4803d7d-4603-4b14-950d-018f88294285-kube-api-access-qg4mj\") on node \"ip-10-0-140-107.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 00:58:33.353298 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:58:33.353262 4939 generic.go:356] "Generic (PLEG): container finished" podID="e4803d7d-4603-4b14-950d-018f88294285" containerID="80328c9985d797195afba46b194886bdb01d5ba00303ded6defced0175dcf411" exitCode=0 Nov 03 00:58:33.353473 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:58:33.353308 4939 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-dd03e395/e2e-lifecycle-model-dd03e395-kserve-78979759df-b8v9c" Nov 03 00:58:33.353473 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:58:33.353349 4939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-dd03e395/e2e-lifecycle-model-dd03e395-kserve-78979759df-b8v9c" event={"ID":"e4803d7d-4603-4b14-950d-018f88294285","Type":"ContainerDied","Data":"80328c9985d797195afba46b194886bdb01d5ba00303ded6defced0175dcf411"} Nov 03 00:58:33.353473 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:58:33.353406 4939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-dd03e395/e2e-lifecycle-model-dd03e395-kserve-78979759df-b8v9c" event={"ID":"e4803d7d-4603-4b14-950d-018f88294285","Type":"ContainerDied","Data":"fb22740ea0e6dae41c9ee796a703215fcdd325bec32f365ebd8c24a6a399c2aa"} Nov 03 00:58:33.353473 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:58:33.353431 4939 scope.go:122] "RemoveContainer" containerID="80328c9985d797195afba46b194886bdb01d5ba00303ded6defced0175dcf411" Nov 03 00:58:33.365439 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:58:33.365422 4939 scope.go:122] "RemoveContainer" containerID="80328c9985d797195afba46b194886bdb01d5ba00303ded6defced0175dcf411" Nov 03 00:58:33.365726 ip-10-0-140-107 kubenswrapper[4939]: E1103 00:58:33.365704 4939 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"80328c9985d797195afba46b194886bdb01d5ba00303ded6defced0175dcf411\": container with ID starting with 80328c9985d797195afba46b194886bdb01d5ba00303ded6defced0175dcf411 not found: ID does not exist" containerID="80328c9985d797195afba46b194886bdb01d5ba00303ded6defced0175dcf411" Nov 03 00:58:33.365828 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:58:33.365733 4939 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"80328c9985d797195afba46b194886bdb01d5ba00303ded6defced0175dcf411"} err="failed to get container status \"80328c9985d797195afba46b194886bdb01d5ba00303ded6defced0175dcf411\": rpc error: code = NotFound desc = could not find container \"80328c9985d797195afba46b194886bdb01d5ba00303ded6defced0175dcf411\": container with ID starting with 80328c9985d797195afba46b194886bdb01d5ba00303ded6defced0175dcf411 not found: ID does not exist" Nov 03 00:58:33.387281 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:58:33.384388 4939 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-mt-dd03e395/e2e-lifecycle-model-dd03e395-kserve-78979759df-b8v9c"] Nov 03 00:58:33.388706 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:58:33.388680 4939 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-mt-dd03e395/e2e-lifecycle-model-dd03e395-kserve-78979759df-b8v9c"] Nov 03 00:58:33.547360 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:58:33.547261 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 00:58:33.784467 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:58:33.784436 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 00:58:33.884846 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:58:33.884814 4939 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e4803d7d-4603-4b14-950d-018f88294285" path="/var/lib/kubelet/pods/e4803d7d-4603-4b14-950d-018f88294285/volumes" Nov 03 00:58:34.223283 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:58:34.223197 4939 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-mt-dd03e395/e2e-lifecycle-model-dd03e395-kserve-78979759df-b8v9c" podUID="e4803d7d-4603-4b14-950d-018f88294285" containerName="main" probeResult="failure" output="Get \"https://10.132.0.91:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 00:58:34.372834 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:58:34.372783 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 00:58:34.516611 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:58:34.516489 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 00:58:34.878114 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:58:34.878043 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 00:58:35.388802 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:58:35.388738 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 00:58:35.949640 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:58:35.949587 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 00:58:35.997694 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:58:35.997660 4939 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-7f43f9-69b998759f-zzpgn"] Nov 03 00:58:35.998703 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:58:35.998682 4939 cpu_manager.go:414] "containerMap: removing container" podUID="e4803d7d-4603-4b14-950d-018f88294285" containerName="main" Nov 03 00:58:35.998767 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:58:35.998708 4939 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e4803d7d-4603-4b14-950d-018f88294285" containerName="main" Nov 03 00:58:35.998767 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:58:35.998759 4939 cpu_manager.go:414] "containerMap: removing container" podUID="916506b5-c34f-479e-a192-1a4c2e5714ee" containerName="payload-processing" Nov 03 00:58:35.998842 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:58:35.998767 4939 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="916506b5-c34f-479e-a192-1a4c2e5714ee" containerName="payload-processing" Nov 03 00:58:35.999065 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:58:35.999048 4939 memory_manager.go:365] "RemoveStaleState removing state" podUID="e4803d7d-4603-4b14-950d-018f88294285" containerName="main" Nov 03 00:58:35.999120 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:58:35.999076 4939 memory_manager.go:365] "RemoveStaleState removing state" podUID="916506b5-c34f-479e-a192-1a4c2e5714ee" containerName="payload-processing" Nov 03 00:58:36.008896 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:58:36.008743 4939 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-7f43f9-69b998759f-zzpgn" Nov 03 00:58:36.009194 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:58:36.009155 4939 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-7f43f9-69b998759f-zzpgn"] Nov 03 00:58:36.092933 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:58:36.088299 4939 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-a-7f43f9\"" Nov 03 00:58:36.092933 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:58:36.088456 4939 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-a-7f43f9-dockercfg-jxxrr\"" Nov 03 00:58:36.092933 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:58:36.092804 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 00:58:36.229980 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:58:36.228317 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-97z7t\" (UniqueName: \"kubernetes.io/projected/979a369e-4ab3-4565-ac44-d3574751bed7-kube-api-access-97z7t\") pod \"payload-pre-processing-e2e-ipp-a-7f43f9-69b998759f-zzpgn\" (UID: \"979a369e-4ab3-4565-ac44-d3574751bed7\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-7f43f9-69b998759f-zzpgn" Nov 03 00:58:36.229980 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:58:36.228531 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/979a369e-4ab3-4565-ac44-d3574751bed7-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-7f43f9-69b998759f-zzpgn\" (UID: \"979a369e-4ab3-4565-ac44-d3574751bed7\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-7f43f9-69b998759f-zzpgn" Nov 03 00:58:36.333041 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:58:36.332977 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-97z7t\" (UniqueName: \"kubernetes.io/projected/979a369e-4ab3-4565-ac44-d3574751bed7-kube-api-access-97z7t\") pod \"payload-pre-processing-e2e-ipp-a-7f43f9-69b998759f-zzpgn\" (UID: \"979a369e-4ab3-4565-ac44-d3574751bed7\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-7f43f9-69b998759f-zzpgn" Nov 03 00:58:36.334584 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:58:36.334560 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/979a369e-4ab3-4565-ac44-d3574751bed7-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-7f43f9-69b998759f-zzpgn\" (UID: \"979a369e-4ab3-4565-ac44-d3574751bed7\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-7f43f9-69b998759f-zzpgn" Nov 03 00:58:36.335864 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:58:36.335642 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/979a369e-4ab3-4565-ac44-d3574751bed7-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-7f43f9-69b998759f-zzpgn\" (UID: \"979a369e-4ab3-4565-ac44-d3574751bed7\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-7f43f9-69b998759f-zzpgn" Nov 03 00:58:36.367000 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:58:36.364203 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-97z7t\" (UniqueName: \"kubernetes.io/projected/979a369e-4ab3-4565-ac44-d3574751bed7-kube-api-access-97z7t\") pod \"payload-pre-processing-e2e-ipp-a-7f43f9-69b998759f-zzpgn\" (UID: \"979a369e-4ab3-4565-ac44-d3574751bed7\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-7f43f9-69b998759f-zzpgn" Nov 03 00:58:36.387265 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:58:36.387230 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 00:58:36.578079 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:58:36.577989 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 00:58:36.649931 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:58:36.642195 4939 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-7f43f9-69b998759f-zzpgn" Nov 03 00:58:36.765885 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:58:36.763746 4939 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-a-80f70e-openshift-default-5984948b65-ck87x"] Nov 03 00:58:36.771957 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:58:36.770048 4939 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-80f70e-openshift-default-5984948b65-ck87x" Nov 03 00:58:36.780076 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:58:36.780030 4939 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-a-80f70e-openshift-default-dockercfg-xqz8h\"" Nov 03 00:58:36.787525 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:58:36.787505 4939 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-80f70e-openshift-default-5984948b65-ck87x"] Nov 03 00:58:36.819450 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:58:36.819415 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 00:58:36.949893 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:58:36.949860 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 00:58:36.973929 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:58:36.971034 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bbecba1a-367a-4239-9e6c-fdfb4972ccf1-istio-data\") pod \"e2e-api-a-80f70e-openshift-default-5984948b65-ck87x\" (UID: \"bbecba1a-367a-4239-9e6c-fdfb4972ccf1\") " pod="openshift-ingress/e2e-api-a-80f70e-openshift-default-5984948b65-ck87x" Nov 03 00:58:37.004931 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:58:37.002037 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bbecba1a-367a-4239-9e6c-fdfb4972ccf1-istiod-ca-cert\") pod \"e2e-api-a-80f70e-openshift-default-5984948b65-ck87x\" (UID: \"bbecba1a-367a-4239-9e6c-fdfb4972ccf1\") " pod="openshift-ingress/e2e-api-a-80f70e-openshift-default-5984948b65-ck87x" Nov 03 00:58:37.004931 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:58:37.002134 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bbecba1a-367a-4239-9e6c-fdfb4972ccf1-istio-envoy\") pod \"e2e-api-a-80f70e-openshift-default-5984948b65-ck87x\" (UID: \"bbecba1a-367a-4239-9e6c-fdfb4972ccf1\") " pod="openshift-ingress/e2e-api-a-80f70e-openshift-default-5984948b65-ck87x" Nov 03 00:58:37.004931 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:58:37.002166 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bbecba1a-367a-4239-9e6c-fdfb4972ccf1-istio-podinfo\") pod \"e2e-api-a-80f70e-openshift-default-5984948b65-ck87x\" (UID: \"bbecba1a-367a-4239-9e6c-fdfb4972ccf1\") " pod="openshift-ingress/e2e-api-a-80f70e-openshift-default-5984948b65-ck87x" Nov 03 00:58:37.004931 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:58:37.002215 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bbecba1a-367a-4239-9e6c-fdfb4972ccf1-workload-certs\") pod \"e2e-api-a-80f70e-openshift-default-5984948b65-ck87x\" (UID: \"bbecba1a-367a-4239-9e6c-fdfb4972ccf1\") " pod="openshift-ingress/e2e-api-a-80f70e-openshift-default-5984948b65-ck87x" Nov 03 00:58:37.004931 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:58:37.002266 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bbecba1a-367a-4239-9e6c-fdfb4972ccf1-istio-token\") pod \"e2e-api-a-80f70e-openshift-default-5984948b65-ck87x\" (UID: \"bbecba1a-367a-4239-9e6c-fdfb4972ccf1\") " pod="openshift-ingress/e2e-api-a-80f70e-openshift-default-5984948b65-ck87x" Nov 03 00:58:37.004931 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:58:37.002367 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bbecba1a-367a-4239-9e6c-fdfb4972ccf1-workload-socket\") pod \"e2e-api-a-80f70e-openshift-default-5984948b65-ck87x\" (UID: \"bbecba1a-367a-4239-9e6c-fdfb4972ccf1\") " pod="openshift-ingress/e2e-api-a-80f70e-openshift-default-5984948b65-ck87x" Nov 03 00:58:37.004931 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:58:37.002546 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g44q5\" (UniqueName: \"kubernetes.io/projected/bbecba1a-367a-4239-9e6c-fdfb4972ccf1-kube-api-access-g44q5\") pod \"e2e-api-a-80f70e-openshift-default-5984948b65-ck87x\" (UID: \"bbecba1a-367a-4239-9e6c-fdfb4972ccf1\") " pod="openshift-ingress/e2e-api-a-80f70e-openshift-default-5984948b65-ck87x" Nov 03 00:58:37.004931 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:58:37.002660 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bbecba1a-367a-4239-9e6c-fdfb4972ccf1-credential-socket\") pod \"e2e-api-a-80f70e-openshift-default-5984948b65-ck87x\" (UID: \"bbecba1a-367a-4239-9e6c-fdfb4972ccf1\") " pod="openshift-ingress/e2e-api-a-80f70e-openshift-default-5984948b65-ck87x" Nov 03 00:58:37.180289 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:58:37.134212 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bbecba1a-367a-4239-9e6c-fdfb4972ccf1-istio-data\") pod \"e2e-api-a-80f70e-openshift-default-5984948b65-ck87x\" (UID: \"bbecba1a-367a-4239-9e6c-fdfb4972ccf1\") " pod="openshift-ingress/e2e-api-a-80f70e-openshift-default-5984948b65-ck87x" Nov 03 00:58:37.180289 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:58:37.134420 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bbecba1a-367a-4239-9e6c-fdfb4972ccf1-istiod-ca-cert\") pod \"e2e-api-a-80f70e-openshift-default-5984948b65-ck87x\" (UID: \"bbecba1a-367a-4239-9e6c-fdfb4972ccf1\") " pod="openshift-ingress/e2e-api-a-80f70e-openshift-default-5984948b65-ck87x" Nov 03 00:58:37.180289 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:58:37.134468 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bbecba1a-367a-4239-9e6c-fdfb4972ccf1-istio-envoy\") pod \"e2e-api-a-80f70e-openshift-default-5984948b65-ck87x\" (UID: \"bbecba1a-367a-4239-9e6c-fdfb4972ccf1\") " pod="openshift-ingress/e2e-api-a-80f70e-openshift-default-5984948b65-ck87x" Nov 03 00:58:37.180289 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:58:37.134496 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bbecba1a-367a-4239-9e6c-fdfb4972ccf1-istio-podinfo\") pod \"e2e-api-a-80f70e-openshift-default-5984948b65-ck87x\" (UID: \"bbecba1a-367a-4239-9e6c-fdfb4972ccf1\") " pod="openshift-ingress/e2e-api-a-80f70e-openshift-default-5984948b65-ck87x" Nov 03 00:58:37.180289 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:58:37.134529 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bbecba1a-367a-4239-9e6c-fdfb4972ccf1-workload-certs\") pod \"e2e-api-a-80f70e-openshift-default-5984948b65-ck87x\" (UID: \"bbecba1a-367a-4239-9e6c-fdfb4972ccf1\") " pod="openshift-ingress/e2e-api-a-80f70e-openshift-default-5984948b65-ck87x" Nov 03 00:58:37.180289 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:58:37.134563 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bbecba1a-367a-4239-9e6c-fdfb4972ccf1-istio-token\") pod \"e2e-api-a-80f70e-openshift-default-5984948b65-ck87x\" (UID: \"bbecba1a-367a-4239-9e6c-fdfb4972ccf1\") " pod="openshift-ingress/e2e-api-a-80f70e-openshift-default-5984948b65-ck87x" Nov 03 00:58:37.180289 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:58:37.134628 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bbecba1a-367a-4239-9e6c-fdfb4972ccf1-workload-socket\") pod \"e2e-api-a-80f70e-openshift-default-5984948b65-ck87x\" (UID: \"bbecba1a-367a-4239-9e6c-fdfb4972ccf1\") " pod="openshift-ingress/e2e-api-a-80f70e-openshift-default-5984948b65-ck87x" Nov 03 00:58:37.180289 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:58:37.134747 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g44q5\" (UniqueName: \"kubernetes.io/projected/bbecba1a-367a-4239-9e6c-fdfb4972ccf1-kube-api-access-g44q5\") pod \"e2e-api-a-80f70e-openshift-default-5984948b65-ck87x\" (UID: \"bbecba1a-367a-4239-9e6c-fdfb4972ccf1\") " pod="openshift-ingress/e2e-api-a-80f70e-openshift-default-5984948b65-ck87x" Nov 03 00:58:37.180289 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:58:37.134817 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bbecba1a-367a-4239-9e6c-fdfb4972ccf1-credential-socket\") pod \"e2e-api-a-80f70e-openshift-default-5984948b65-ck87x\" (UID: \"bbecba1a-367a-4239-9e6c-fdfb4972ccf1\") " pod="openshift-ingress/e2e-api-a-80f70e-openshift-default-5984948b65-ck87x" Nov 03 00:58:37.180289 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:58:37.135287 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bbecba1a-367a-4239-9e6c-fdfb4972ccf1-credential-socket\") pod \"e2e-api-a-80f70e-openshift-default-5984948b65-ck87x\" (UID: \"bbecba1a-367a-4239-9e6c-fdfb4972ccf1\") " pod="openshift-ingress/e2e-api-a-80f70e-openshift-default-5984948b65-ck87x" Nov 03 00:58:37.180289 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:58:37.136296 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bbecba1a-367a-4239-9e6c-fdfb4972ccf1-workload-certs\") pod \"e2e-api-a-80f70e-openshift-default-5984948b65-ck87x\" (UID: \"bbecba1a-367a-4239-9e6c-fdfb4972ccf1\") " pod="openshift-ingress/e2e-api-a-80f70e-openshift-default-5984948b65-ck87x" Nov 03 00:58:37.180289 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:58:37.137123 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bbecba1a-367a-4239-9e6c-fdfb4972ccf1-istiod-ca-cert\") pod \"e2e-api-a-80f70e-openshift-default-5984948b65-ck87x\" (UID: \"bbecba1a-367a-4239-9e6c-fdfb4972ccf1\") " pod="openshift-ingress/e2e-api-a-80f70e-openshift-default-5984948b65-ck87x" Nov 03 00:58:37.180289 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:58:37.137716 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bbecba1a-367a-4239-9e6c-fdfb4972ccf1-istio-data\") pod \"e2e-api-a-80f70e-openshift-default-5984948b65-ck87x\" (UID: \"bbecba1a-367a-4239-9e6c-fdfb4972ccf1\") " pod="openshift-ingress/e2e-api-a-80f70e-openshift-default-5984948b65-ck87x" Nov 03 00:58:37.180289 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:58:37.138298 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bbecba1a-367a-4239-9e6c-fdfb4972ccf1-workload-socket\") pod \"e2e-api-a-80f70e-openshift-default-5984948b65-ck87x\" (UID: \"bbecba1a-367a-4239-9e6c-fdfb4972ccf1\") " pod="openshift-ingress/e2e-api-a-80f70e-openshift-default-5984948b65-ck87x" Nov 03 00:58:37.180289 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:58:37.141666 4939 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-7f43f9-69b998759f-zzpgn"] Nov 03 00:58:37.180289 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:58:37.148994 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-g44q5\" (UniqueName: \"kubernetes.io/projected/bbecba1a-367a-4239-9e6c-fdfb4972ccf1-kube-api-access-g44q5\") pod \"e2e-api-a-80f70e-openshift-default-5984948b65-ck87x\" (UID: \"bbecba1a-367a-4239-9e6c-fdfb4972ccf1\") " pod="openshift-ingress/e2e-api-a-80f70e-openshift-default-5984948b65-ck87x" Nov 03 00:58:37.180289 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:58:37.150355 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bbecba1a-367a-4239-9e6c-fdfb4972ccf1-istio-token\") pod \"e2e-api-a-80f70e-openshift-default-5984948b65-ck87x\" (UID: \"bbecba1a-367a-4239-9e6c-fdfb4972ccf1\") " pod="openshift-ingress/e2e-api-a-80f70e-openshift-default-5984948b65-ck87x" Nov 03 00:58:37.180289 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:58:37.150776 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bbecba1a-367a-4239-9e6c-fdfb4972ccf1-istio-envoy\") pod \"e2e-api-a-80f70e-openshift-default-5984948b65-ck87x\" (UID: \"bbecba1a-367a-4239-9e6c-fdfb4972ccf1\") " pod="openshift-ingress/e2e-api-a-80f70e-openshift-default-5984948b65-ck87x" Nov 03 00:58:37.180289 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:58:37.153104 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bbecba1a-367a-4239-9e6c-fdfb4972ccf1-istio-podinfo\") pod \"e2e-api-a-80f70e-openshift-default-5984948b65-ck87x\" (UID: \"bbecba1a-367a-4239-9e6c-fdfb4972ccf1\") " pod="openshift-ingress/e2e-api-a-80f70e-openshift-default-5984948b65-ck87x" Nov 03 00:58:37.392688 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:58:37.392651 4939 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-80f70e-openshift-default-5984948b65-ck87x" Nov 03 00:58:37.404922 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:58:37.404779 4939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-7f43f9-69b998759f-zzpgn" event={"ID":"979a369e-4ab3-4565-ac44-d3574751bed7","Type":"ContainerStarted","Data":"b1a18ce06d85317b3dfa38223068df318f1971d2734269998254a049c7c6f625"} Nov 03 00:58:37.404922 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:58:37.404841 4939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-7f43f9-69b998759f-zzpgn" event={"ID":"979a369e-4ab3-4565-ac44-d3574751bed7","Type":"ContainerStarted","Data":"99f8af298712c1f1cf78fd5db7717296f05e51f2e3d6cc9b1079a16ee2defda7"} Nov 03 00:58:37.404922 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:58:37.404867 4939 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-7f43f9-69b998759f-zzpgn" Nov 03 00:58:37.482933 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:58:37.478262 4939 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-7f43f9-69b998759f-zzpgn" podStartSLOduration=2.478239283 podStartE2EDuration="2.478239283s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:58:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:58:37.44468102 +0000 UTC m=+1065.709015981" watchObservedRunningTime="2026-11-03 00:58:37.478239283 +0000 UTC m=+1065.742574229" Nov 03 00:58:37.621416 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:58:37.621382 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 00:58:37.771021 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:58:37.770981 4939 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-80f70e-openshift-default-5984948b65-ck87x"] Nov 03 00:58:37.772256 ip-10-0-140-107 kubenswrapper[4939]: W1103 00:58:37.772228 4939 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbbecba1a_367a_4239_9e6c_fdfb4972ccf1.slice/crio-8c320fd95987ad7a5f426ea6351cf908a2a2797548a7bb070c5c2f539a44bccc WatchSource:0}: Error finding container 8c320fd95987ad7a5f426ea6351cf908a2a2797548a7bb070c5c2f539a44bccc: Status 404 returned error can't find the container with id 8c320fd95987ad7a5f426ea6351cf908a2a2797548a7bb070c5c2f539a44bccc Nov 03 00:58:37.775379 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:58:37.775347 4939 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 00:58:37.775472 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:58:37.775425 4939 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 00:58:37.775472 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:58:37.775458 4939 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 00:58:38.378051 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:58:38.377996 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 00:58:38.448189 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:58:38.448154 4939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-80f70e-openshift-default-5984948b65-ck87x" event={"ID":"bbecba1a-367a-4239-9e6c-fdfb4972ccf1","Type":"ContainerStarted","Data":"9213c015a67724a51db25ba1367b7f35a54128dcbc010e1f9c06ccb7aa2a6556"} Nov 03 00:58:38.448516 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:58:38.448492 4939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-80f70e-openshift-default-5984948b65-ck87x" event={"ID":"bbecba1a-367a-4239-9e6c-fdfb4972ccf1","Type":"ContainerStarted","Data":"8c320fd95987ad7a5f426ea6351cf908a2a2797548a7bb070c5c2f539a44bccc"} Nov 03 00:58:38.491541 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:58:38.491471 4939 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-api-a-80f70e-openshift-default-5984948b65-ck87x" podStartSLOduration=2.491450278 podStartE2EDuration="2.491450278s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:58:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:58:38.466540429 +0000 UTC m=+1066.730875374" watchObservedRunningTime="2026-11-03 00:58:38.491450278 +0000 UTC m=+1066.755785225" Nov 03 00:58:39.082162 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:58:39.082130 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 00:58:39.393236 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:58:39.393153 4939 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-80f70e-openshift-default-5984948b65-ck87x" Nov 03 00:58:39.398546 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:58:39.398506 4939 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-80f70e-openshift-default-5984948b65-ck87x" Nov 03 00:58:39.453576 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:58:39.453544 4939 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-80f70e-openshift-default-5984948b65-ck87x" Nov 03 00:58:39.454549 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:58:39.454524 4939 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-80f70e-openshift-default-5984948b65-ck87x" Nov 03 00:58:39.963922 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:58:39.963882 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 00:58:40.759987 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:58:40.759944 4939 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-07f5f6-f97497957-m28k9"] Nov 03 00:58:40.844511 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:58:40.844471 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 00:58:41.135933 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:58:41.105101 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 00:58:41.267879 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:58:41.267839 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 00:58:41.954491 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:58:41.954454 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 00:58:42.090311 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:58:42.090257 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 00:58:42.230986 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:58:42.223469 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 00:58:42.269552 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:58:42.269481 4939 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-340b96-c984ff695-nz2kt"] Nov 03 00:58:42.289752 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:58:42.289714 4939 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-340b96-86744887cf-swn7k"] Nov 03 00:58:42.355432 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:58:42.355401 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 00:58:42.550756 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:58:42.550627 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 00:58:42.667537 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:58:42.667506 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 00:58:43.418552 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:58:43.418520 4939 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-7f43f9-668f86957-vqsff"] Nov 03 00:58:43.422770 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:58:43.422754 4939 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-7f43f9-668f86957-vqsff" Nov 03 00:58:43.428864 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:58:43.428829 4939 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ipp-a-7f43f9-serving-cert\"" Nov 03 00:58:43.434491 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:58:43.434466 4939 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-7f43f9-668f86957-vqsff"] Nov 03 00:58:43.472029 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:58:43.471996 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e2f30569-2ce1-469c-a8dc-87ac06c45698-maas-api-tls\") pod \"maas-api-e2e-ipp-a-7f43f9-668f86957-vqsff\" (UID: \"e2f30569-2ce1-469c-a8dc-87ac06c45698\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-7f43f9-668f86957-vqsff" Nov 03 00:58:43.472168 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:58:43.472143 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e2f30569-2ce1-469c-a8dc-87ac06c45698-metrics-certs\") pod \"maas-api-e2e-ipp-a-7f43f9-668f86957-vqsff\" (UID: \"e2f30569-2ce1-469c-a8dc-87ac06c45698\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-7f43f9-668f86957-vqsff" Nov 03 00:58:43.472313 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:58:43.472296 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e2f30569-2ce1-469c-a8dc-87ac06c45698-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-7f43f9-668f86957-vqsff\" (UID: \"e2f30569-2ce1-469c-a8dc-87ac06c45698\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-7f43f9-668f86957-vqsff" Nov 03 00:58:43.472366 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:58:43.472352 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-667sm\" (UniqueName: \"kubernetes.io/projected/e2f30569-2ce1-469c-a8dc-87ac06c45698-kube-api-access-667sm\") pod \"maas-api-e2e-ipp-a-7f43f9-668f86957-vqsff\" (UID: \"e2f30569-2ce1-469c-a8dc-87ac06c45698\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-7f43f9-668f86957-vqsff" Nov 03 00:58:43.556764 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:58:43.556728 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 00:58:43.573519 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:58:43.573484 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e2f30569-2ce1-469c-a8dc-87ac06c45698-metrics-certs\") pod \"maas-api-e2e-ipp-a-7f43f9-668f86957-vqsff\" (UID: \"e2f30569-2ce1-469c-a8dc-87ac06c45698\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-7f43f9-668f86957-vqsff" Nov 03 00:58:43.573708 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:58:43.573645 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e2f30569-2ce1-469c-a8dc-87ac06c45698-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-7f43f9-668f86957-vqsff\" (UID: \"e2f30569-2ce1-469c-a8dc-87ac06c45698\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-7f43f9-668f86957-vqsff" Nov 03 00:58:43.573708 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:58:43.573703 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-667sm\" (UniqueName: \"kubernetes.io/projected/e2f30569-2ce1-469c-a8dc-87ac06c45698-kube-api-access-667sm\") pod \"maas-api-e2e-ipp-a-7f43f9-668f86957-vqsff\" (UID: \"e2f30569-2ce1-469c-a8dc-87ac06c45698\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-7f43f9-668f86957-vqsff" Nov 03 00:58:43.573952 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:58:43.573934 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e2f30569-2ce1-469c-a8dc-87ac06c45698-maas-api-tls\") pod \"maas-api-e2e-ipp-a-7f43f9-668f86957-vqsff\" (UID: \"e2f30569-2ce1-469c-a8dc-87ac06c45698\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-7f43f9-668f86957-vqsff" Nov 03 00:58:43.574492 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:58:43.574464 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e2f30569-2ce1-469c-a8dc-87ac06c45698-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-7f43f9-668f86957-vqsff\" (UID: \"e2f30569-2ce1-469c-a8dc-87ac06c45698\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-7f43f9-668f86957-vqsff" Nov 03 00:58:43.576711 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:58:43.576689 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e2f30569-2ce1-469c-a8dc-87ac06c45698-metrics-certs\") pod \"maas-api-e2e-ipp-a-7f43f9-668f86957-vqsff\" (UID: \"e2f30569-2ce1-469c-a8dc-87ac06c45698\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-7f43f9-668f86957-vqsff" Nov 03 00:58:43.580733 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:58:43.580691 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e2f30569-2ce1-469c-a8dc-87ac06c45698-maas-api-tls\") pod \"maas-api-e2e-ipp-a-7f43f9-668f86957-vqsff\" (UID: \"e2f30569-2ce1-469c-a8dc-87ac06c45698\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-7f43f9-668f86957-vqsff" Nov 03 00:58:43.582132 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:58:43.582106 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-667sm\" (UniqueName: \"kubernetes.io/projected/e2f30569-2ce1-469c-a8dc-87ac06c45698-kube-api-access-667sm\") pod \"maas-api-e2e-ipp-a-7f43f9-668f86957-vqsff\" (UID: \"e2f30569-2ce1-469c-a8dc-87ac06c45698\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-7f43f9-668f86957-vqsff" Nov 03 00:58:43.694040 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:58:43.689561 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 00:58:43.744381 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:58:43.743206 4939 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-7f43f9-668f86957-vqsff" Nov 03 00:58:44.098220 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:58:44.098190 4939 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-7f43f9-668f86957-vqsff"] Nov 03 00:58:44.098805 ip-10-0-140-107 kubenswrapper[4939]: W1103 00:58:44.098767 4939 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode2f30569_2ce1_469c_a8dc_87ac06c45698.slice/crio-d41413268dbe658f2f722c79bce6ec1c866c63dea861307fa2af65bfea54cd18 WatchSource:0}: Error finding container d41413268dbe658f2f722c79bce6ec1c866c63dea861307fa2af65bfea54cd18: Status 404 returned error can't find the container with id d41413268dbe658f2f722c79bce6ec1c866c63dea861307fa2af65bfea54cd18 Nov 03 00:58:44.625711 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:58:44.625675 4939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-7f43f9-668f86957-vqsff" event={"ID":"e2f30569-2ce1-469c-a8dc-87ac06c45698","Type":"ContainerStarted","Data":"d41413268dbe658f2f722c79bce6ec1c866c63dea861307fa2af65bfea54cd18"} Nov 03 00:58:44.783015 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:58:44.782984 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 00:58:45.401031 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:58:45.400990 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 00:58:45.635246 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:58:45.635208 4939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-7f43f9-668f86957-vqsff" event={"ID":"e2f30569-2ce1-469c-a8dc-87ac06c45698","Type":"ContainerStarted","Data":"12d0dc8d226d8b6cc9f19c1e2ef05e2721b39285a2986fa00aa2acf7ecabb7e8"} Nov 03 00:58:45.635669 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:58:45.635419 4939 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-7f43f9-668f86957-vqsff" Nov 03 00:58:45.654574 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:58:45.654485 4939 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-7f43f9-668f86957-vqsff" podStartSLOduration=2.122874456 podStartE2EDuration="2.654469729s" totalImagesPullingTime="531.595273ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:58:43 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:58:45.651224597 +0000 UTC m=+1073.915559541" watchObservedRunningTime="2026-11-03 00:58:45.654469729 +0000 UTC m=+1073.918804673" Nov 03 00:58:45.760434 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:58:45.760397 4939 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-a-07f5f6-f97497957-m28k9" podUID="090b138f-b622-4f99-bd4a-16d49430ba5e" containerName="payload-processing" containerID="cri-o://b92dd3a75fcc687c734ce7f2f54491cc13aea28b8c7135a76ecafe5bbdd674fe" gracePeriod=25 Nov 03 00:58:45.920420 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:58:45.920221 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 00:58:46.032421 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:58:46.032382 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 00:58:46.089841 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:58:46.089814 4939 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-07f5f6-f97497957-m28k9" Nov 03 00:58:46.189831 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:58:46.186516 4939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/090b138f-b622-4f99-bd4a-16d49430ba5e-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/090b138f-b622-4f99-bd4a-16d49430ba5e-plugins-config-volume\") pod \"090b138f-b622-4f99-bd4a-16d49430ba5e\" (UID: \"090b138f-b622-4f99-bd4a-16d49430ba5e\") " Nov 03 00:58:46.189831 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:58:46.186600 4939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/090b138f-b622-4f99-bd4a-16d49430ba5e-kube-api-access-ncjht\" (UniqueName: \"kubernetes.io/projected/090b138f-b622-4f99-bd4a-16d49430ba5e-kube-api-access-ncjht\") pod \"090b138f-b622-4f99-bd4a-16d49430ba5e\" (UID: \"090b138f-b622-4f99-bd4a-16d49430ba5e\") " Nov 03 00:58:46.229126 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:58:46.225253 4939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/090b138f-b622-4f99-bd4a-16d49430ba5e-plugins-config-volume" pod "090b138f-b622-4f99-bd4a-16d49430ba5e" (UID: "090b138f-b622-4f99-bd4a-16d49430ba5e"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 00:58:46.229126 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:58:46.227076 4939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/090b138f-b622-4f99-bd4a-16d49430ba5e-kube-api-access-ncjht" pod "090b138f-b622-4f99-bd4a-16d49430ba5e" (UID: "090b138f-b622-4f99-bd4a-16d49430ba5e"). InnerVolumeSpecName "kube-api-access-ncjht". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 00:58:46.295932 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:58:46.292604 4939 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/090b138f-b622-4f99-bd4a-16d49430ba5e-plugins-config-volume\") on node \"ip-10-0-140-107.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 00:58:46.295932 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:58:46.292647 4939 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ncjht\" (UniqueName: \"kubernetes.io/projected/090b138f-b622-4f99-bd4a-16d49430ba5e-kube-api-access-ncjht\") on node \"ip-10-0-140-107.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 00:58:46.643641 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:58:46.643605 4939 generic.go:356] "Generic (PLEG): container finished" podID="090b138f-b622-4f99-bd4a-16d49430ba5e" containerID="b92dd3a75fcc687c734ce7f2f54491cc13aea28b8c7135a76ecafe5bbdd674fe" exitCode=0 Nov 03 00:58:46.644232 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:58:46.643655 4939 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-07f5f6-f97497957-m28k9" Nov 03 00:58:46.644232 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:58:46.643692 4939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-07f5f6-f97497957-m28k9" event={"ID":"090b138f-b622-4f99-bd4a-16d49430ba5e","Type":"ContainerDied","Data":"b92dd3a75fcc687c734ce7f2f54491cc13aea28b8c7135a76ecafe5bbdd674fe"} Nov 03 00:58:46.644232 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:58:46.643762 4939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-07f5f6-f97497957-m28k9" event={"ID":"090b138f-b622-4f99-bd4a-16d49430ba5e","Type":"ContainerDied","Data":"9b50d6056d3020d36ae223d7b12a1307417069cdf70297de58dc1ac512090974"} Nov 03 00:58:46.644232 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:58:46.643800 4939 scope.go:122] "RemoveContainer" containerID="b92dd3a75fcc687c734ce7f2f54491cc13aea28b8c7135a76ecafe5bbdd674fe" Nov 03 00:58:46.655198 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:58:46.655180 4939 scope.go:122] "RemoveContainer" containerID="b92dd3a75fcc687c734ce7f2f54491cc13aea28b8c7135a76ecafe5bbdd674fe" Nov 03 00:58:46.655471 ip-10-0-140-107 kubenswrapper[4939]: E1103 00:58:46.655453 4939 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b92dd3a75fcc687c734ce7f2f54491cc13aea28b8c7135a76ecafe5bbdd674fe\": container with ID starting with b92dd3a75fcc687c734ce7f2f54491cc13aea28b8c7135a76ecafe5bbdd674fe not found: ID does not exist" containerID="b92dd3a75fcc687c734ce7f2f54491cc13aea28b8c7135a76ecafe5bbdd674fe" Nov 03 00:58:46.655553 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:58:46.655482 4939 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b92dd3a75fcc687c734ce7f2f54491cc13aea28b8c7135a76ecafe5bbdd674fe"} err="failed to get container status \"b92dd3a75fcc687c734ce7f2f54491cc13aea28b8c7135a76ecafe5bbdd674fe\": rpc error: code = NotFound desc = could not find container \"b92dd3a75fcc687c734ce7f2f54491cc13aea28b8c7135a76ecafe5bbdd674fe\": container with ID starting with b92dd3a75fcc687c734ce7f2f54491cc13aea28b8c7135a76ecafe5bbdd674fe not found: ID does not exist" Nov 03 00:58:46.668772 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:58:46.668741 4939 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-07f5f6-f97497957-m28k9"] Nov 03 00:58:46.672878 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:58:46.672857 4939 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-07f5f6-f97497957-m28k9"] Nov 03 00:58:46.673314 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:58:46.673298 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 00:58:47.270154 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:58:47.270114 4939 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-api-a-340b96-c984ff695-nz2kt" podUID="a6a06134-dd53-4f93-8a78-7d26fedd2ab3" containerName="payload-processing" containerID="cri-o://9f50fc61cb95cb3e9c29e3a408c393961c75d1d8f6bf4411670b752aa42c45a7" gracePeriod=25 Nov 03 00:58:47.291930 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:58:47.291887 4939 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-api-a-340b96-86744887cf-swn7k" podUID="350de0d1-756f-4e15-8cc6-e5b8474571b2" containerName="payload-pre-processing" containerID="cri-o://87933e8f795dd3c4c8815d3376bfaeae21fe9967d3532f65b0242b082c49c545" gracePeriod=25 Nov 03 00:58:47.366643 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:58:47.365068 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 00:58:47.652957 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:58:47.652897 4939 generic.go:356] "Generic (PLEG): container finished" podID="350de0d1-756f-4e15-8cc6-e5b8474571b2" containerID="87933e8f795dd3c4c8815d3376bfaeae21fe9967d3532f65b0242b082c49c545" exitCode=0 Nov 03 00:58:47.653323 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:58:47.652971 4939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-340b96-86744887cf-swn7k" event={"ID":"350de0d1-756f-4e15-8cc6-e5b8474571b2","Type":"ContainerDied","Data":"87933e8f795dd3c4c8815d3376bfaeae21fe9967d3532f65b0242b082c49c545"} Nov 03 00:58:47.653323 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:58:47.653027 4939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-340b96-86744887cf-swn7k" event={"ID":"350de0d1-756f-4e15-8cc6-e5b8474571b2","Type":"ContainerDied","Data":"fc805066da0cb9281e663c323585ef4bbaadb9c8d7564ba0d005fea75938ee48"} Nov 03 00:58:47.653323 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:58:47.653044 4939 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="fc805066da0cb9281e663c323585ef4bbaadb9c8d7564ba0d005fea75938ee48" Nov 03 00:58:47.654438 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:58:47.654418 4939 generic.go:356] "Generic (PLEG): container finished" podID="a6a06134-dd53-4f93-8a78-7d26fedd2ab3" containerID="9f50fc61cb95cb3e9c29e3a408c393961c75d1d8f6bf4411670b752aa42c45a7" exitCode=0 Nov 03 00:58:47.654566 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:58:47.654496 4939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-340b96-c984ff695-nz2kt" event={"ID":"a6a06134-dd53-4f93-8a78-7d26fedd2ab3","Type":"ContainerDied","Data":"9f50fc61cb95cb3e9c29e3a408c393961c75d1d8f6bf4411670b752aa42c45a7"} Nov 03 00:58:47.654566 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:58:47.654547 4939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-340b96-c984ff695-nz2kt" event={"ID":"a6a06134-dd53-4f93-8a78-7d26fedd2ab3","Type":"ContainerDied","Data":"a941f65684ad8465dca8b2e51c0ff17ae9bdd3276b4411af6e6926c6f1f9e1cc"} Nov 03 00:58:47.654654 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:58:47.654572 4939 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a941f65684ad8465dca8b2e51c0ff17ae9bdd3276b4411af6e6926c6f1f9e1cc" Nov 03 00:58:47.668754 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:58:47.668734 4939 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-340b96-c984ff695-nz2kt" Nov 03 00:58:47.672538 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:58:47.672518 4939 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-340b96-86744887cf-swn7k" Nov 03 00:58:47.773960 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:58:47.773896 4939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a6a06134-dd53-4f93-8a78-7d26fedd2ab3-kube-api-access-c4hdr\" (UniqueName: \"kubernetes.io/projected/a6a06134-dd53-4f93-8a78-7d26fedd2ab3-kube-api-access-c4hdr\") pod \"a6a06134-dd53-4f93-8a78-7d26fedd2ab3\" (UID: \"a6a06134-dd53-4f93-8a78-7d26fedd2ab3\") " Nov 03 00:58:47.774162 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:58:47.774147 4939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a6a06134-dd53-4f93-8a78-7d26fedd2ab3-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a6a06134-dd53-4f93-8a78-7d26fedd2ab3-plugins-config-volume\") pod \"a6a06134-dd53-4f93-8a78-7d26fedd2ab3\" (UID: \"a6a06134-dd53-4f93-8a78-7d26fedd2ab3\") " Nov 03 00:58:47.774241 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:58:47.774228 4939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/350de0d1-756f-4e15-8cc6-e5b8474571b2-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/350de0d1-756f-4e15-8cc6-e5b8474571b2-plugins-config-volume\") pod \"350de0d1-756f-4e15-8cc6-e5b8474571b2\" (UID: \"350de0d1-756f-4e15-8cc6-e5b8474571b2\") " Nov 03 00:58:47.774285 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:58:47.774273 4939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/350de0d1-756f-4e15-8cc6-e5b8474571b2-kube-api-access-vcn4w\" (UniqueName: \"kubernetes.io/projected/350de0d1-756f-4e15-8cc6-e5b8474571b2-kube-api-access-vcn4w\") pod \"350de0d1-756f-4e15-8cc6-e5b8474571b2\" (UID: \"350de0d1-756f-4e15-8cc6-e5b8474571b2\") " Nov 03 00:58:47.774590 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:58:47.774559 4939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a6a06134-dd53-4f93-8a78-7d26fedd2ab3-plugins-config-volume" pod "a6a06134-dd53-4f93-8a78-7d26fedd2ab3" (UID: "a6a06134-dd53-4f93-8a78-7d26fedd2ab3"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 00:58:47.774716 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:58:47.774622 4939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/350de0d1-756f-4e15-8cc6-e5b8474571b2-plugins-config-volume" pod "350de0d1-756f-4e15-8cc6-e5b8474571b2" (UID: "350de0d1-756f-4e15-8cc6-e5b8474571b2"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 00:58:47.774831 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:58:47.774817 4939 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a6a06134-dd53-4f93-8a78-7d26fedd2ab3-plugins-config-volume\") on node \"ip-10-0-140-107.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 00:58:47.774874 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:58:47.774835 4939 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/350de0d1-756f-4e15-8cc6-e5b8474571b2-plugins-config-volume\") on node \"ip-10-0-140-107.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 00:58:47.776019 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:58:47.775997 4939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a6a06134-dd53-4f93-8a78-7d26fedd2ab3-kube-api-access-c4hdr" pod "a6a06134-dd53-4f93-8a78-7d26fedd2ab3" (UID: "a6a06134-dd53-4f93-8a78-7d26fedd2ab3"). InnerVolumeSpecName "kube-api-access-c4hdr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 00:58:47.776339 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:58:47.776317 4939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/350de0d1-756f-4e15-8cc6-e5b8474571b2-kube-api-access-vcn4w" pod "350de0d1-756f-4e15-8cc6-e5b8474571b2" (UID: "350de0d1-756f-4e15-8cc6-e5b8474571b2"). InnerVolumeSpecName "kube-api-access-vcn4w". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 00:58:47.876185 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:58:47.876146 4939 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vcn4w\" (UniqueName: \"kubernetes.io/projected/350de0d1-756f-4e15-8cc6-e5b8474571b2-kube-api-access-vcn4w\") on node \"ip-10-0-140-107.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 00:58:47.876185 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:58:47.876184 4939 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-c4hdr\" (UniqueName: \"kubernetes.io/projected/a6a06134-dd53-4f93-8a78-7d26fedd2ab3-kube-api-access-c4hdr\") on node \"ip-10-0-140-107.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 00:58:47.881064 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:58:47.881023 4939 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="090b138f-b622-4f99-bd4a-16d49430ba5e" path="/var/lib/kubelet/pods/090b138f-b622-4f99-bd4a-16d49430ba5e/volumes" Nov 03 00:58:48.032197 ip-10-0-140-107 kubenswrapper[4939]: E1103 00:58:48.032108 4939 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-sthjh" containerName="extractor" Nov 03 00:58:48.080696 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:58:48.080664 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 00:58:48.463959 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:58:48.463426 4939 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-7f43f9-69b998759f-zzpgn" Nov 03 00:58:48.662720 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:58:48.662690 4939 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-340b96-c984ff695-nz2kt" Nov 03 00:58:48.663193 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:58:48.662690 4939 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-340b96-86744887cf-swn7k" Nov 03 00:58:48.688163 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:58:48.688131 4939 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-340b96-c984ff695-nz2kt"] Nov 03 00:58:48.694490 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:58:48.694467 4939 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-340b96-c984ff695-nz2kt"] Nov 03 00:58:48.707745 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:58:48.707723 4939 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-340b96-86744887cf-swn7k"] Nov 03 00:58:48.715437 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:58:48.715414 4939 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-340b96-86744887cf-swn7k"] Nov 03 00:58:48.835640 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:58:48.835608 4939 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-340b96-69fb46f475-f4qbm"] Nov 03 00:58:48.835922 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:58:48.835869 4939 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-340b96-69fb46f475-f4qbm" podUID="b911ac20-6027-4445-8ccc-c622715708dc" containerName="maas-api" containerID="cri-o://6e116fc6af05bd396ac09d78e6196e0c02f97eb2b7cc230c09042ea42ce77071" gracePeriod=30 Nov 03 00:58:48.911036 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:58:48.910990 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 00:58:49.118676 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:58:49.118644 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 00:58:49.126209 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:58:49.126187 4939 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-340b96-69fb46f475-f4qbm" Nov 03 00:58:49.280028 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:58:49.267569 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 00:58:49.299821 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:58:49.299787 4939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b911ac20-6027-4445-8ccc-c622715708dc-kube-api-access-fskdg\" (UniqueName: \"kubernetes.io/projected/b911ac20-6027-4445-8ccc-c622715708dc-kube-api-access-fskdg\") pod \"b911ac20-6027-4445-8ccc-c622715708dc\" (UID: \"b911ac20-6027-4445-8ccc-c622715708dc\") " Nov 03 00:58:49.300114 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:58:49.300090 4939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b911ac20-6027-4445-8ccc-c622715708dc-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b911ac20-6027-4445-8ccc-c622715708dc-maas-api-tls\") pod \"b911ac20-6027-4445-8ccc-c622715708dc\" (UID: \"b911ac20-6027-4445-8ccc-c622715708dc\") " Nov 03 00:58:49.300439 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:58:49.300414 4939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b911ac20-6027-4445-8ccc-c622715708dc-metrics-certs\" (UniqueName: \"kubernetes.io/secret/b911ac20-6027-4445-8ccc-c622715708dc-metrics-certs\") pod \"b911ac20-6027-4445-8ccc-c622715708dc\" (UID: \"b911ac20-6027-4445-8ccc-c622715708dc\") " Nov 03 00:58:49.300627 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:58:49.300610 4939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b911ac20-6027-4445-8ccc-c622715708dc-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b911ac20-6027-4445-8ccc-c622715708dc-openshift-service-ca\") pod \"b911ac20-6027-4445-8ccc-c622715708dc\" (UID: \"b911ac20-6027-4445-8ccc-c622715708dc\") " Nov 03 00:58:49.312504 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:58:49.312473 4939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b911ac20-6027-4445-8ccc-c622715708dc-openshift-service-ca" pod "b911ac20-6027-4445-8ccc-c622715708dc" (UID: "b911ac20-6027-4445-8ccc-c622715708dc"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 00:58:49.377681 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:58:49.377624 4939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b911ac20-6027-4445-8ccc-c622715708dc-kube-api-access-fskdg" pod "b911ac20-6027-4445-8ccc-c622715708dc" (UID: "b911ac20-6027-4445-8ccc-c622715708dc"). InnerVolumeSpecName "kube-api-access-fskdg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 00:58:49.426455 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:58:49.402047 4939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b911ac20-6027-4445-8ccc-c622715708dc-maas-api-tls" pod "b911ac20-6027-4445-8ccc-c622715708dc" (UID: "b911ac20-6027-4445-8ccc-c622715708dc"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 00:58:49.426455 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:58:49.403451 4939 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b911ac20-6027-4445-8ccc-c622715708dc-openshift-service-ca\") on node \"ip-10-0-140-107.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 00:58:49.426455 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:58:49.403481 4939 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fskdg\" (UniqueName: \"kubernetes.io/projected/b911ac20-6027-4445-8ccc-c622715708dc-kube-api-access-fskdg\") on node \"ip-10-0-140-107.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 00:58:49.426455 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:58:49.403499 4939 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b911ac20-6027-4445-8ccc-c622715708dc-maas-api-tls\") on node \"ip-10-0-140-107.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 00:58:49.426455 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:58:49.403635 4939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b911ac20-6027-4445-8ccc-c622715708dc-metrics-certs" pod "b911ac20-6027-4445-8ccc-c622715708dc" (UID: "b911ac20-6027-4445-8ccc-c622715708dc"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 00:58:49.505126 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:58:49.505097 4939 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b911ac20-6027-4445-8ccc-c622715708dc-metrics-certs\") on node \"ip-10-0-140-107.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 00:58:49.671059 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:58:49.671021 4939 generic.go:356] "Generic (PLEG): container finished" podID="b911ac20-6027-4445-8ccc-c622715708dc" containerID="6e116fc6af05bd396ac09d78e6196e0c02f97eb2b7cc230c09042ea42ce77071" exitCode=0 Nov 03 00:58:49.671494 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:58:49.671072 4939 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-340b96-69fb46f475-f4qbm" Nov 03 00:58:49.671494 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:58:49.671094 4939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-340b96-69fb46f475-f4qbm" event={"ID":"b911ac20-6027-4445-8ccc-c622715708dc","Type":"ContainerDied","Data":"6e116fc6af05bd396ac09d78e6196e0c02f97eb2b7cc230c09042ea42ce77071"} Nov 03 00:58:49.671494 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:58:49.671162 4939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-340b96-69fb46f475-f4qbm" event={"ID":"b911ac20-6027-4445-8ccc-c622715708dc","Type":"ContainerDied","Data":"7b5b7d629366d885311d8dcb71c6da92c5baa6e106902a2bbf780f0f68000b2c"} Nov 03 00:58:49.671494 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:58:49.671194 4939 scope.go:122] "RemoveContainer" containerID="6e116fc6af05bd396ac09d78e6196e0c02f97eb2b7cc230c09042ea42ce77071" Nov 03 00:58:49.682477 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:58:49.682460 4939 scope.go:122] "RemoveContainer" containerID="6e116fc6af05bd396ac09d78e6196e0c02f97eb2b7cc230c09042ea42ce77071" Nov 03 00:58:49.682730 ip-10-0-140-107 kubenswrapper[4939]: E1103 00:58:49.682710 4939 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6e116fc6af05bd396ac09d78e6196e0c02f97eb2b7cc230c09042ea42ce77071\": container with ID starting with 6e116fc6af05bd396ac09d78e6196e0c02f97eb2b7cc230c09042ea42ce77071 not found: ID does not exist" containerID="6e116fc6af05bd396ac09d78e6196e0c02f97eb2b7cc230c09042ea42ce77071" Nov 03 00:58:49.682799 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:58:49.682743 4939 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6e116fc6af05bd396ac09d78e6196e0c02f97eb2b7cc230c09042ea42ce77071"} err="failed to get container status \"6e116fc6af05bd396ac09d78e6196e0c02f97eb2b7cc230c09042ea42ce77071\": rpc error: code = NotFound desc = could not find container \"6e116fc6af05bd396ac09d78e6196e0c02f97eb2b7cc230c09042ea42ce77071\": container with ID starting with 6e116fc6af05bd396ac09d78e6196e0c02f97eb2b7cc230c09042ea42ce77071 not found: ID does not exist" Nov 03 00:58:49.697103 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:58:49.697079 4939 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-340b96-69fb46f475-f4qbm"] Nov 03 00:58:49.702295 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:58:49.702270 4939 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-340b96-69fb46f475-f4qbm"] Nov 03 00:58:49.809121 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:58:49.809088 4939 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-80f70e-659f4d7d6f-69755"] Nov 03 00:58:49.810020 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:58:49.809997 4939 cpu_manager.go:414] "containerMap: removing container" podUID="a6a06134-dd53-4f93-8a78-7d26fedd2ab3" containerName="payload-processing" Nov 03 00:58:49.810020 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:58:49.810019 4939 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a6a06134-dd53-4f93-8a78-7d26fedd2ab3" containerName="payload-processing" Nov 03 00:58:49.810190 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:58:49.810052 4939 cpu_manager.go:414] "containerMap: removing container" podUID="b911ac20-6027-4445-8ccc-c622715708dc" containerName="maas-api" Nov 03 00:58:49.810190 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:58:49.810058 4939 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b911ac20-6027-4445-8ccc-c622715708dc" containerName="maas-api" Nov 03 00:58:49.810190 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:58:49.810067 4939 cpu_manager.go:414] "containerMap: removing container" podUID="350de0d1-756f-4e15-8cc6-e5b8474571b2" containerName="payload-pre-processing" Nov 03 00:58:49.810190 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:58:49.810073 4939 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="350de0d1-756f-4e15-8cc6-e5b8474571b2" containerName="payload-pre-processing" Nov 03 00:58:49.810190 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:58:49.810085 4939 cpu_manager.go:414] "containerMap: removing container" podUID="090b138f-b622-4f99-bd4a-16d49430ba5e" containerName="payload-processing" Nov 03 00:58:49.810190 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:58:49.810091 4939 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="090b138f-b622-4f99-bd4a-16d49430ba5e" containerName="payload-processing" Nov 03 00:58:49.810438 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:58:49.810210 4939 memory_manager.go:365] "RemoveStaleState removing state" podUID="a6a06134-dd53-4f93-8a78-7d26fedd2ab3" containerName="payload-processing" Nov 03 00:58:49.810438 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:58:49.810222 4939 memory_manager.go:365] "RemoveStaleState removing state" podUID="b911ac20-6027-4445-8ccc-c622715708dc" containerName="maas-api" Nov 03 00:58:49.810438 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:58:49.810230 4939 memory_manager.go:365] "RemoveStaleState removing state" podUID="350de0d1-756f-4e15-8cc6-e5b8474571b2" containerName="payload-pre-processing" Nov 03 00:58:49.810438 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:58:49.810236 4939 memory_manager.go:365] "RemoveStaleState removing state" podUID="090b138f-b622-4f99-bd4a-16d49430ba5e" containerName="payload-processing" Nov 03 00:58:49.815743 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:58:49.815718 4939 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-80f70e-659f4d7d6f-69755" Nov 03 00:58:49.825241 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:58:49.825214 4939 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-80f70e-659f4d7d6f-69755"] Nov 03 00:58:49.825704 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:58:49.825620 4939 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-a-80f70e-serving-cert\"" Nov 03 00:58:49.870948 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:58:49.870922 4939 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="350de0d1-756f-4e15-8cc6-e5b8474571b2" path="/var/lib/kubelet/pods/350de0d1-756f-4e15-8cc6-e5b8474571b2/volumes" Nov 03 00:58:49.871325 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:58:49.871312 4939 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a6a06134-dd53-4f93-8a78-7d26fedd2ab3" path="/var/lib/kubelet/pods/a6a06134-dd53-4f93-8a78-7d26fedd2ab3/volumes" Nov 03 00:58:49.871715 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:58:49.871695 4939 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b911ac20-6027-4445-8ccc-c622715708dc" path="/var/lib/kubelet/pods/b911ac20-6027-4445-8ccc-c622715708dc/volumes" Nov 03 00:58:49.890648 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:58:49.890629 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 00:58:49.911289 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:58:49.911260 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bdmhl\" (UniqueName: \"kubernetes.io/projected/d102e320-316d-4577-b088-9b39a27e8794-kube-api-access-bdmhl\") pod \"maas-api-e2e-api-a-80f70e-659f4d7d6f-69755\" (UID: \"d102e320-316d-4577-b088-9b39a27e8794\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-80f70e-659f4d7d6f-69755" Nov 03 00:58:49.911413 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:58:49.911394 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d102e320-316d-4577-b088-9b39a27e8794-openshift-service-ca\") pod \"maas-api-e2e-api-a-80f70e-659f4d7d6f-69755\" (UID: \"d102e320-316d-4577-b088-9b39a27e8794\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-80f70e-659f4d7d6f-69755" Nov 03 00:58:49.911492 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:58:49.911479 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d102e320-316d-4577-b088-9b39a27e8794-metrics-certs\") pod \"maas-api-e2e-api-a-80f70e-659f4d7d6f-69755\" (UID: \"d102e320-316d-4577-b088-9b39a27e8794\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-80f70e-659f4d7d6f-69755" Nov 03 00:58:49.911621 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:58:49.911605 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d102e320-316d-4577-b088-9b39a27e8794-maas-api-tls\") pod \"maas-api-e2e-api-a-80f70e-659f4d7d6f-69755\" (UID: \"d102e320-316d-4577-b088-9b39a27e8794\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-80f70e-659f4d7d6f-69755" Nov 03 00:58:50.019490 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:58:50.018866 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 00:58:50.037962 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:58:50.034074 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d102e320-316d-4577-b088-9b39a27e8794-maas-api-tls\") pod \"maas-api-e2e-api-a-80f70e-659f4d7d6f-69755\" (UID: \"d102e320-316d-4577-b088-9b39a27e8794\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-80f70e-659f4d7d6f-69755" Nov 03 00:58:50.037962 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:58:50.034317 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bdmhl\" (UniqueName: \"kubernetes.io/projected/d102e320-316d-4577-b088-9b39a27e8794-kube-api-access-bdmhl\") pod \"maas-api-e2e-api-a-80f70e-659f4d7d6f-69755\" (UID: \"d102e320-316d-4577-b088-9b39a27e8794\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-80f70e-659f4d7d6f-69755" Nov 03 00:58:50.037962 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:58:50.034650 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d102e320-316d-4577-b088-9b39a27e8794-openshift-service-ca\") pod \"maas-api-e2e-api-a-80f70e-659f4d7d6f-69755\" (UID: \"d102e320-316d-4577-b088-9b39a27e8794\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-80f70e-659f4d7d6f-69755" Nov 03 00:58:50.037962 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:58:50.034969 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d102e320-316d-4577-b088-9b39a27e8794-metrics-certs\") pod \"maas-api-e2e-api-a-80f70e-659f4d7d6f-69755\" (UID: \"d102e320-316d-4577-b088-9b39a27e8794\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-80f70e-659f4d7d6f-69755" Nov 03 00:58:50.037962 ip-10-0-140-107 kubenswrapper[4939]: E1103 00:58:50.036026 4939 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-api-a-80f70e-serving-cert: secret "maas-api-e2e-api-a-80f70e-serving-cert" not found Nov 03 00:58:50.037962 ip-10-0-140-107 kubenswrapper[4939]: E1103 00:58:50.036092 4939 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d102e320-316d-4577-b088-9b39a27e8794-maas-api-tls podName:d102e320-316d-4577-b088-9b39a27e8794 nodeName:}" failed. No retries permitted until 2026-11-03 00:58:50.53607131 +0000 UTC m=+1078.800406248 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/d102e320-316d-4577-b088-9b39a27e8794-maas-api-tls") pod "maas-api-e2e-api-a-80f70e-659f4d7d6f-69755" (UID: "d102e320-316d-4577-b088-9b39a27e8794") : secret "maas-api-e2e-api-a-80f70e-serving-cert" not found Nov 03 00:58:50.039046 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:58:50.039021 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d102e320-316d-4577-b088-9b39a27e8794-openshift-service-ca\") pod \"maas-api-e2e-api-a-80f70e-659f4d7d6f-69755\" (UID: \"d102e320-316d-4577-b088-9b39a27e8794\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-80f70e-659f4d7d6f-69755" Nov 03 00:58:50.053411 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:58:50.043763 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d102e320-316d-4577-b088-9b39a27e8794-metrics-certs\") pod \"maas-api-e2e-api-a-80f70e-659f4d7d6f-69755\" (UID: \"d102e320-316d-4577-b088-9b39a27e8794\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-80f70e-659f4d7d6f-69755" Nov 03 00:58:50.053411 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:58:50.053150 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bdmhl\" (UniqueName: \"kubernetes.io/projected/d102e320-316d-4577-b088-9b39a27e8794-kube-api-access-bdmhl\") pod \"maas-api-e2e-api-a-80f70e-659f4d7d6f-69755\" (UID: \"d102e320-316d-4577-b088-9b39a27e8794\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-80f70e-659f4d7d6f-69755" Nov 03 00:58:50.251811 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:58:50.251778 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 00:58:50.546188 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:58:50.546072 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d102e320-316d-4577-b088-9b39a27e8794-maas-api-tls\") pod \"maas-api-e2e-api-a-80f70e-659f4d7d6f-69755\" (UID: \"d102e320-316d-4577-b088-9b39a27e8794\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-80f70e-659f4d7d6f-69755" Nov 03 00:58:50.586932 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:58:50.577607 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d102e320-316d-4577-b088-9b39a27e8794-maas-api-tls\") pod \"maas-api-e2e-api-a-80f70e-659f4d7d6f-69755\" (UID: \"d102e320-316d-4577-b088-9b39a27e8794\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-80f70e-659f4d7d6f-69755" Nov 03 00:58:50.735945 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:58:50.735068 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 00:58:50.743946 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:58:50.743248 4939 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-80f70e-659f4d7d6f-69755" Nov 03 00:58:51.178021 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:58:51.177987 4939 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-80f70e-659f4d7d6f-69755"] Nov 03 00:58:51.178157 ip-10-0-140-107 kubenswrapper[4939]: W1103 00:58:51.178071 4939 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd102e320_316d_4577_b088_9b39a27e8794.slice/crio-0fc511f219a4f86b00f66d6d403ce7b78a2822c8bf4e1af03a42ce186b2aff56 WatchSource:0}: Error finding container 0fc511f219a4f86b00f66d6d403ce7b78a2822c8bf4e1af03a42ce186b2aff56: Status 404 returned error can't find the container with id 0fc511f219a4f86b00f66d6d403ce7b78a2822c8bf4e1af03a42ce186b2aff56 Nov 03 00:58:51.497095 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:58:51.497012 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 00:58:51.650457 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:58:51.650432 4939 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-7f43f9-668f86957-vqsff" Nov 03 00:58:51.744268 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:58:51.744236 4939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-80f70e-659f4d7d6f-69755" event={"ID":"d102e320-316d-4577-b088-9b39a27e8794","Type":"ContainerStarted","Data":"68516ddd670620157dcbb71927055247038344c2a29683bc162d157e0d35c5cc"} Nov 03 00:58:51.744647 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:58:51.744298 4939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-80f70e-659f4d7d6f-69755" event={"ID":"d102e320-316d-4577-b088-9b39a27e8794","Type":"ContainerStarted","Data":"0fc511f219a4f86b00f66d6d403ce7b78a2822c8bf4e1af03a42ce186b2aff56"} Nov 03 00:58:52.198568 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:58:52.198535 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 00:58:52.342980 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:58:52.342948 4939 scope.go:122] "RemoveContainer" containerID="2917d956307fc73e5dac205783ce6e7058366d9cc6c4ca12c77f067e0e4462c5" Nov 03 00:58:52.751192 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:58:52.751158 4939 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-80f70e-659f4d7d6f-69755" Nov 03 00:58:52.771789 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:58:52.771736 4939 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-80f70e-659f4d7d6f-69755" podStartSLOduration=3.305421188 podStartE2EDuration="3.77172383s" totalImagesPullingTime="466.302642ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:58:49 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:58:52.767717198 +0000 UTC m=+1081.032052145" watchObservedRunningTime="2026-11-03 00:58:52.77172383 +0000 UTC m=+1081.036058774" Nov 03 00:58:52.886382 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:58:52.886345 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 00:58:53.922000 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:58:53.921959 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 00:58:54.511579 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:58:54.509293 4939 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-7f43f9-69b998759f-zzpgn"] Nov 03 00:58:54.587946 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:58:54.585852 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 00:58:54.707718 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:58:54.707684 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 00:58:54.820792 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:58:54.820710 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 00:58:55.011933 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:58:55.010899 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 00:58:55.464419 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:58:55.464376 4939 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-dd03e395-b8464b49-7rlqj"] Nov 03 00:58:55.577239 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:58:55.577208 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 00:58:55.701307 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:58:55.701279 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 00:58:55.860956 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:58:55.854525 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 00:58:56.228771 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:58:56.228668 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 00:58:56.380757 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:58:56.380717 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 00:58:56.804566 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:58:56.804531 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 00:58:56.925118 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:58:56.925080 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 00:58:57.258555 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:58:57.258520 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 00:58:57.678269 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:58:57.678219 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 00:58:57.978010 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:58:57.977289 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 00:58:58.534842 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:58:58.534804 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 00:58:58.829135 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:58:58.829055 4939 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-80f70e-659f4d7d6f-69755" Nov 03 00:58:59.204018 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:58:59.203985 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 00:58:59.510367 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:58:59.510269 4939 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-7f43f9-69b998759f-zzpgn" podUID="979a369e-4ab3-4565-ac44-d3574751bed7" containerName="payload-pre-processing" containerID="cri-o://b1a18ce06d85317b3dfa38223068df318f1971d2734269998254a049c7c6f625" gracePeriod=25 Nov 03 00:58:59.885120 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:58:59.885094 4939 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-7f43f9-69b998759f-zzpgn" Nov 03 00:58:59.934930 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:58:59.934885 4939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/979a369e-4ab3-4565-ac44-d3574751bed7-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/979a369e-4ab3-4565-ac44-d3574751bed7-plugins-config-volume\") pod \"979a369e-4ab3-4565-ac44-d3574751bed7\" (UID: \"979a369e-4ab3-4565-ac44-d3574751bed7\") " Nov 03 00:58:59.935138 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:58:59.935040 4939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/979a369e-4ab3-4565-ac44-d3574751bed7-kube-api-access-97z7t\" (UniqueName: \"kubernetes.io/projected/979a369e-4ab3-4565-ac44-d3574751bed7-kube-api-access-97z7t\") pod \"979a369e-4ab3-4565-ac44-d3574751bed7\" (UID: \"979a369e-4ab3-4565-ac44-d3574751bed7\") " Nov 03 00:58:59.935340 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:58:59.935313 4939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/979a369e-4ab3-4565-ac44-d3574751bed7-plugins-config-volume" pod "979a369e-4ab3-4565-ac44-d3574751bed7" (UID: "979a369e-4ab3-4565-ac44-d3574751bed7"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 00:58:59.935651 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:58:59.935634 4939 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/979a369e-4ab3-4565-ac44-d3574751bed7-plugins-config-volume\") on node \"ip-10-0-140-107.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 00:58:59.937179 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:58:59.937154 4939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/979a369e-4ab3-4565-ac44-d3574751bed7-kube-api-access-97z7t" pod "979a369e-4ab3-4565-ac44-d3574751bed7" (UID: "979a369e-4ab3-4565-ac44-d3574751bed7"). InnerVolumeSpecName "kube-api-access-97z7t". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 00:59:00.006200 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:59:00.006168 4939 generic.go:356] "Generic (PLEG): container finished" podID="979a369e-4ab3-4565-ac44-d3574751bed7" containerID="b1a18ce06d85317b3dfa38223068df318f1971d2734269998254a049c7c6f625" exitCode=0 Nov 03 00:59:00.006363 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:59:00.006222 4939 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-7f43f9-69b998759f-zzpgn" Nov 03 00:59:00.006363 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:59:00.006227 4939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-7f43f9-69b998759f-zzpgn" event={"ID":"979a369e-4ab3-4565-ac44-d3574751bed7","Type":"ContainerDied","Data":"b1a18ce06d85317b3dfa38223068df318f1971d2734269998254a049c7c6f625"} Nov 03 00:59:00.006363 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:59:00.006292 4939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-7f43f9-69b998759f-zzpgn" event={"ID":"979a369e-4ab3-4565-ac44-d3574751bed7","Type":"ContainerDied","Data":"99f8af298712c1f1cf78fd5db7717296f05e51f2e3d6cc9b1079a16ee2defda7"} Nov 03 00:59:00.006363 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:59:00.006319 4939 scope.go:122] "RemoveContainer" containerID="b1a18ce06d85317b3dfa38223068df318f1971d2734269998254a049c7c6f625" Nov 03 00:59:00.017235 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:59:00.017216 4939 scope.go:122] "RemoveContainer" containerID="b1a18ce06d85317b3dfa38223068df318f1971d2734269998254a049c7c6f625" Nov 03 00:59:00.017516 ip-10-0-140-107 kubenswrapper[4939]: E1103 00:59:00.017496 4939 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b1a18ce06d85317b3dfa38223068df318f1971d2734269998254a049c7c6f625\": container with ID starting with b1a18ce06d85317b3dfa38223068df318f1971d2734269998254a049c7c6f625 not found: ID does not exist" containerID="b1a18ce06d85317b3dfa38223068df318f1971d2734269998254a049c7c6f625" Nov 03 00:59:00.017564 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:59:00.017524 4939 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b1a18ce06d85317b3dfa38223068df318f1971d2734269998254a049c7c6f625"} err="failed to get container status \"b1a18ce06d85317b3dfa38223068df318f1971d2734269998254a049c7c6f625\": rpc error: code = NotFound desc = could not find container \"b1a18ce06d85317b3dfa38223068df318f1971d2734269998254a049c7c6f625\": container with ID starting with b1a18ce06d85317b3dfa38223068df318f1971d2734269998254a049c7c6f625 not found: ID does not exist" Nov 03 00:59:00.032879 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:59:00.032850 4939 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-7f43f9-69b998759f-zzpgn"] Nov 03 00:59:00.036500 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:59:00.036469 4939 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-7f43f9-69b998759f-zzpgn"] Nov 03 00:59:00.037367 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:59:00.037348 4939 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-97z7t\" (UniqueName: \"kubernetes.io/projected/979a369e-4ab3-4565-ac44-d3574751bed7-kube-api-access-97z7t\") on node \"ip-10-0-140-107.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 00:59:00.075266 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:59:00.075181 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 00:59:00.471049 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:59:00.470994 4939 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-mt-dd03e395-b8464b49-7rlqj" podUID="dbe5252d-5569-4e97-8ede-e8943326dcd8" containerName="payload-processing" containerID="cri-o://131551666e7f1a0a1a6f9371f1d161aba288e4866587b847f29cf4a9d4cc50b3" gracePeriod=25 Nov 03 00:59:01.136237 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:59:01.136195 4939 generic.go:356] "Generic (PLEG): container finished" podID="dbe5252d-5569-4e97-8ede-e8943326dcd8" containerID="131551666e7f1a0a1a6f9371f1d161aba288e4866587b847f29cf4a9d4cc50b3" exitCode=0 Nov 03 00:59:01.136733 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:59:01.136367 4939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-dd03e395-b8464b49-7rlqj" event={"ID":"dbe5252d-5569-4e97-8ede-e8943326dcd8","Type":"ContainerDied","Data":"131551666e7f1a0a1a6f9371f1d161aba288e4866587b847f29cf4a9d4cc50b3"} Nov 03 00:59:01.181284 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:59:01.181255 4939 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-dd03e395-b8464b49-7rlqj" Nov 03 00:59:01.239449 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:59:01.239410 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 00:59:01.333984 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:59:01.330616 4939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dbe5252d-5569-4e97-8ede-e8943326dcd8-kube-api-access-z8sfd\" (UniqueName: \"kubernetes.io/projected/dbe5252d-5569-4e97-8ede-e8943326dcd8-kube-api-access-z8sfd\") pod \"dbe5252d-5569-4e97-8ede-e8943326dcd8\" (UID: \"dbe5252d-5569-4e97-8ede-e8943326dcd8\") " Nov 03 00:59:01.333984 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:59:01.330761 4939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/dbe5252d-5569-4e97-8ede-e8943326dcd8-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dbe5252d-5569-4e97-8ede-e8943326dcd8-plugins-config-volume\") pod \"dbe5252d-5569-4e97-8ede-e8943326dcd8\" (UID: \"dbe5252d-5569-4e97-8ede-e8943326dcd8\") " Nov 03 00:59:01.377980 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:59:01.372420 4939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dbe5252d-5569-4e97-8ede-e8943326dcd8-plugins-config-volume" pod "dbe5252d-5569-4e97-8ede-e8943326dcd8" (UID: "dbe5252d-5569-4e97-8ede-e8943326dcd8"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 00:59:01.414705 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:59:01.414650 4939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dbe5252d-5569-4e97-8ede-e8943326dcd8-kube-api-access-z8sfd" pod "dbe5252d-5569-4e97-8ede-e8943326dcd8" (UID: "dbe5252d-5569-4e97-8ede-e8943326dcd8"). InnerVolumeSpecName "kube-api-access-z8sfd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 00:59:01.433252 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:59:01.433217 4939 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dbe5252d-5569-4e97-8ede-e8943326dcd8-plugins-config-volume\") on node \"ip-10-0-140-107.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 00:59:01.433252 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:59:01.433249 4939 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-z8sfd\" (UniqueName: \"kubernetes.io/projected/dbe5252d-5569-4e97-8ede-e8943326dcd8-kube-api-access-z8sfd\") on node \"ip-10-0-140-107.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 00:59:01.884922 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:59:01.884888 4939 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="979a369e-4ab3-4565-ac44-d3574751bed7" path="/var/lib/kubelet/pods/979a369e-4ab3-4565-ac44-d3574751bed7/volumes" Nov 03 00:59:02.090332 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:59:02.090265 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 00:59:02.213988 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:59:02.213553 4939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-dd03e395-b8464b49-7rlqj" event={"ID":"dbe5252d-5569-4e97-8ede-e8943326dcd8","Type":"ContainerDied","Data":"e1f790b22bf9c0f254c67b7197525951a53f1f80c1123513c991d09bfc8b6d4e"} Nov 03 00:59:02.213988 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:59:02.213631 4939 scope.go:122] "RemoveContainer" containerID="131551666e7f1a0a1a6f9371f1d161aba288e4866587b847f29cf4a9d4cc50b3" Nov 03 00:59:02.213988 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:59:02.213798 4939 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-dd03e395-b8464b49-7rlqj" Nov 03 00:59:02.266987 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:59:02.266958 4939 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-dd03e395-b8464b49-7rlqj"] Nov 03 00:59:02.274938 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:59:02.274823 4939 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-dd03e395-b8464b49-7rlqj"] Nov 03 00:59:03.004882 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:59:03.004846 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 00:59:03.148583 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:59:03.148324 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 00:59:03.572180 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:59:03.572110 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 00:59:03.877738 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:59:03.877706 4939 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="dbe5252d-5569-4e97-8ede-e8943326dcd8" path="/var/lib/kubelet/pods/dbe5252d-5569-4e97-8ede-e8943326dcd8/volumes" Nov 03 00:59:03.878121 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:59:03.878102 4939 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-dd03e395-openshift-default-75799f6fc-4gdnp"] Nov 03 00:59:03.878307 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:59:03.878284 4939 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-dd03e395-openshift-default-75799f6fc-4gdnp" podUID="dd1a7eae-e782-4a82-bdac-c45c75614357" containerName="istio-proxy" containerID="cri-o://a3457590cb89b41b7165836f93cdce29fd1ad3f88f6139cc1ba7799be0513756" gracePeriod=30 Nov 03 00:59:03.900239 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:59:03.900213 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 00:59:04.040277 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:59:04.040191 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 00:59:04.288613 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:59:04.288537 4939 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-eed369-7d94fb8545-hdldv"] Nov 03 00:59:04.400837 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:59:04.399405 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 00:59:04.521056 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:59:04.521013 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 00:59:04.691375 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:59:04.691295 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 00:59:04.824981 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:59:04.823754 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 00:59:04.949482 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:59:04.949368 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 00:59:05.036678 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:59:05.036650 4939 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-a-7406a9-openshift-default-74c9f8d955-rqrs4"] Nov 03 00:59:05.037631 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:59:05.037606 4939 cpu_manager.go:414] "containerMap: removing container" podUID="979a369e-4ab3-4565-ac44-d3574751bed7" containerName="payload-pre-processing" Nov 03 00:59:05.037816 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:59:05.037799 4939 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="979a369e-4ab3-4565-ac44-d3574751bed7" containerName="payload-pre-processing" Nov 03 00:59:05.037984 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:59:05.037970 4939 cpu_manager.go:414] "containerMap: removing container" podUID="dbe5252d-5569-4e97-8ede-e8943326dcd8" containerName="payload-processing" Nov 03 00:59:05.038084 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:59:05.038061 4939 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dbe5252d-5569-4e97-8ede-e8943326dcd8" containerName="payload-processing" Nov 03 00:59:05.038335 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:59:05.038317 4939 memory_manager.go:365] "RemoveStaleState removing state" podUID="dbe5252d-5569-4e97-8ede-e8943326dcd8" containerName="payload-processing" Nov 03 00:59:05.038388 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:59:05.038350 4939 memory_manager.go:365] "RemoveStaleState removing state" podUID="979a369e-4ab3-4565-ac44-d3574751bed7" containerName="payload-pre-processing" Nov 03 00:59:05.047619 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:59:05.047590 4939 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-7406a9-openshift-default-74c9f8d955-rqrs4" Nov 03 00:59:05.054756 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:59:05.054730 4939 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-7406a9-openshift-default-74c9f8d955-rqrs4"] Nov 03 00:59:05.058637 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:59:05.058611 4939 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-a-7406a9-openshift-default-dockercfg-57vph\"" Nov 03 00:59:05.089605 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:59:05.088544 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 00:59:05.165059 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:59:05.151976 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6vd5r\" (UniqueName: \"kubernetes.io/projected/ab2a5b74-e0ad-4a13-9d46-7b78d5ef8aa7-kube-api-access-6vd5r\") pod \"e2e-api-a-7406a9-openshift-default-74c9f8d955-rqrs4\" (UID: \"ab2a5b74-e0ad-4a13-9d46-7b78d5ef8aa7\") " pod="openshift-ingress/e2e-api-a-7406a9-openshift-default-74c9f8d955-rqrs4" Nov 03 00:59:05.165059 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:59:05.152119 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ab2a5b74-e0ad-4a13-9d46-7b78d5ef8aa7-workload-socket\") pod \"e2e-api-a-7406a9-openshift-default-74c9f8d955-rqrs4\" (UID: \"ab2a5b74-e0ad-4a13-9d46-7b78d5ef8aa7\") " pod="openshift-ingress/e2e-api-a-7406a9-openshift-default-74c9f8d955-rqrs4" Nov 03 00:59:05.165059 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:59:05.152200 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ab2a5b74-e0ad-4a13-9d46-7b78d5ef8aa7-istio-data\") pod \"e2e-api-a-7406a9-openshift-default-74c9f8d955-rqrs4\" (UID: \"ab2a5b74-e0ad-4a13-9d46-7b78d5ef8aa7\") " pod="openshift-ingress/e2e-api-a-7406a9-openshift-default-74c9f8d955-rqrs4" Nov 03 00:59:05.165059 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:59:05.152228 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ab2a5b74-e0ad-4a13-9d46-7b78d5ef8aa7-istiod-ca-cert\") pod \"e2e-api-a-7406a9-openshift-default-74c9f8d955-rqrs4\" (UID: \"ab2a5b74-e0ad-4a13-9d46-7b78d5ef8aa7\") " pod="openshift-ingress/e2e-api-a-7406a9-openshift-default-74c9f8d955-rqrs4" Nov 03 00:59:05.165059 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:59:05.152530 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ab2a5b74-e0ad-4a13-9d46-7b78d5ef8aa7-credential-socket\") pod \"e2e-api-a-7406a9-openshift-default-74c9f8d955-rqrs4\" (UID: \"ab2a5b74-e0ad-4a13-9d46-7b78d5ef8aa7\") " pod="openshift-ingress/e2e-api-a-7406a9-openshift-default-74c9f8d955-rqrs4" Nov 03 00:59:05.165059 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:59:05.152566 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ab2a5b74-e0ad-4a13-9d46-7b78d5ef8aa7-workload-certs\") pod \"e2e-api-a-7406a9-openshift-default-74c9f8d955-rqrs4\" (UID: \"ab2a5b74-e0ad-4a13-9d46-7b78d5ef8aa7\") " pod="openshift-ingress/e2e-api-a-7406a9-openshift-default-74c9f8d955-rqrs4" Nov 03 00:59:05.165059 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:59:05.152644 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ab2a5b74-e0ad-4a13-9d46-7b78d5ef8aa7-istio-token\") pod \"e2e-api-a-7406a9-openshift-default-74c9f8d955-rqrs4\" (UID: \"ab2a5b74-e0ad-4a13-9d46-7b78d5ef8aa7\") " pod="openshift-ingress/e2e-api-a-7406a9-openshift-default-74c9f8d955-rqrs4" Nov 03 00:59:05.165059 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:59:05.152837 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ab2a5b74-e0ad-4a13-9d46-7b78d5ef8aa7-istio-podinfo\") pod \"e2e-api-a-7406a9-openshift-default-74c9f8d955-rqrs4\" (UID: \"ab2a5b74-e0ad-4a13-9d46-7b78d5ef8aa7\") " pod="openshift-ingress/e2e-api-a-7406a9-openshift-default-74c9f8d955-rqrs4" Nov 03 00:59:05.165059 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:59:05.153003 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ab2a5b74-e0ad-4a13-9d46-7b78d5ef8aa7-istio-envoy\") pod \"e2e-api-a-7406a9-openshift-default-74c9f8d955-rqrs4\" (UID: \"ab2a5b74-e0ad-4a13-9d46-7b78d5ef8aa7\") " pod="openshift-ingress/e2e-api-a-7406a9-openshift-default-74c9f8d955-rqrs4" Nov 03 00:59:05.225887 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:59:05.225800 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 00:59:05.284093 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:59:05.283939 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ab2a5b74-e0ad-4a13-9d46-7b78d5ef8aa7-credential-socket\") pod \"e2e-api-a-7406a9-openshift-default-74c9f8d955-rqrs4\" (UID: \"ab2a5b74-e0ad-4a13-9d46-7b78d5ef8aa7\") " pod="openshift-ingress/e2e-api-a-7406a9-openshift-default-74c9f8d955-rqrs4" Nov 03 00:59:05.284093 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:59:05.283997 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ab2a5b74-e0ad-4a13-9d46-7b78d5ef8aa7-workload-certs\") pod \"e2e-api-a-7406a9-openshift-default-74c9f8d955-rqrs4\" (UID: \"ab2a5b74-e0ad-4a13-9d46-7b78d5ef8aa7\") " pod="openshift-ingress/e2e-api-a-7406a9-openshift-default-74c9f8d955-rqrs4" Nov 03 00:59:05.284346 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:59:05.284099 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ab2a5b74-e0ad-4a13-9d46-7b78d5ef8aa7-istio-token\") pod \"e2e-api-a-7406a9-openshift-default-74c9f8d955-rqrs4\" (UID: \"ab2a5b74-e0ad-4a13-9d46-7b78d5ef8aa7\") " pod="openshift-ingress/e2e-api-a-7406a9-openshift-default-74c9f8d955-rqrs4" Nov 03 00:59:05.284346 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:59:05.284211 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ab2a5b74-e0ad-4a13-9d46-7b78d5ef8aa7-istio-podinfo\") pod \"e2e-api-a-7406a9-openshift-default-74c9f8d955-rqrs4\" (UID: \"ab2a5b74-e0ad-4a13-9d46-7b78d5ef8aa7\") " pod="openshift-ingress/e2e-api-a-7406a9-openshift-default-74c9f8d955-rqrs4" Nov 03 00:59:05.284346 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:59:05.284307 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ab2a5b74-e0ad-4a13-9d46-7b78d5ef8aa7-istio-envoy\") pod \"e2e-api-a-7406a9-openshift-default-74c9f8d955-rqrs4\" (UID: \"ab2a5b74-e0ad-4a13-9d46-7b78d5ef8aa7\") " pod="openshift-ingress/e2e-api-a-7406a9-openshift-default-74c9f8d955-rqrs4" Nov 03 00:59:05.284514 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:59:05.284362 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6vd5r\" (UniqueName: \"kubernetes.io/projected/ab2a5b74-e0ad-4a13-9d46-7b78d5ef8aa7-kube-api-access-6vd5r\") pod \"e2e-api-a-7406a9-openshift-default-74c9f8d955-rqrs4\" (UID: \"ab2a5b74-e0ad-4a13-9d46-7b78d5ef8aa7\") " pod="openshift-ingress/e2e-api-a-7406a9-openshift-default-74c9f8d955-rqrs4" Nov 03 00:59:05.284514 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:59:05.284404 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ab2a5b74-e0ad-4a13-9d46-7b78d5ef8aa7-workload-socket\") pod \"e2e-api-a-7406a9-openshift-default-74c9f8d955-rqrs4\" (UID: \"ab2a5b74-e0ad-4a13-9d46-7b78d5ef8aa7\") " pod="openshift-ingress/e2e-api-a-7406a9-openshift-default-74c9f8d955-rqrs4" Nov 03 00:59:05.284514 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:59:05.284436 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ab2a5b74-e0ad-4a13-9d46-7b78d5ef8aa7-istio-data\") pod \"e2e-api-a-7406a9-openshift-default-74c9f8d955-rqrs4\" (UID: \"ab2a5b74-e0ad-4a13-9d46-7b78d5ef8aa7\") " pod="openshift-ingress/e2e-api-a-7406a9-openshift-default-74c9f8d955-rqrs4" Nov 03 00:59:05.284514 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:59:05.284464 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ab2a5b74-e0ad-4a13-9d46-7b78d5ef8aa7-istiod-ca-cert\") pod \"e2e-api-a-7406a9-openshift-default-74c9f8d955-rqrs4\" (UID: \"ab2a5b74-e0ad-4a13-9d46-7b78d5ef8aa7\") " pod="openshift-ingress/e2e-api-a-7406a9-openshift-default-74c9f8d955-rqrs4" Nov 03 00:59:05.287935 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:59:05.285418 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ab2a5b74-e0ad-4a13-9d46-7b78d5ef8aa7-istiod-ca-cert\") pod \"e2e-api-a-7406a9-openshift-default-74c9f8d955-rqrs4\" (UID: \"ab2a5b74-e0ad-4a13-9d46-7b78d5ef8aa7\") " pod="openshift-ingress/e2e-api-a-7406a9-openshift-default-74c9f8d955-rqrs4" Nov 03 00:59:05.287935 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:59:05.285679 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ab2a5b74-e0ad-4a13-9d46-7b78d5ef8aa7-credential-socket\") pod \"e2e-api-a-7406a9-openshift-default-74c9f8d955-rqrs4\" (UID: \"ab2a5b74-e0ad-4a13-9d46-7b78d5ef8aa7\") " pod="openshift-ingress/e2e-api-a-7406a9-openshift-default-74c9f8d955-rqrs4" Nov 03 00:59:05.287935 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:59:05.285889 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ab2a5b74-e0ad-4a13-9d46-7b78d5ef8aa7-workload-certs\") pod \"e2e-api-a-7406a9-openshift-default-74c9f8d955-rqrs4\" (UID: \"ab2a5b74-e0ad-4a13-9d46-7b78d5ef8aa7\") " pod="openshift-ingress/e2e-api-a-7406a9-openshift-default-74c9f8d955-rqrs4" Nov 03 00:59:05.297942 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:59:05.291422 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ab2a5b74-e0ad-4a13-9d46-7b78d5ef8aa7-workload-socket\") pod \"e2e-api-a-7406a9-openshift-default-74c9f8d955-rqrs4\" (UID: \"ab2a5b74-e0ad-4a13-9d46-7b78d5ef8aa7\") " pod="openshift-ingress/e2e-api-a-7406a9-openshift-default-74c9f8d955-rqrs4" Nov 03 00:59:05.297942 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:59:05.291939 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ab2a5b74-e0ad-4a13-9d46-7b78d5ef8aa7-istio-data\") pod \"e2e-api-a-7406a9-openshift-default-74c9f8d955-rqrs4\" (UID: \"ab2a5b74-e0ad-4a13-9d46-7b78d5ef8aa7\") " pod="openshift-ingress/e2e-api-a-7406a9-openshift-default-74c9f8d955-rqrs4" Nov 03 00:59:05.297942 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:59:05.297516 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ab2a5b74-e0ad-4a13-9d46-7b78d5ef8aa7-istio-envoy\") pod \"e2e-api-a-7406a9-openshift-default-74c9f8d955-rqrs4\" (UID: \"ab2a5b74-e0ad-4a13-9d46-7b78d5ef8aa7\") " pod="openshift-ingress/e2e-api-a-7406a9-openshift-default-74c9f8d955-rqrs4" Nov 03 00:59:05.355927 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:59:05.354670 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ab2a5b74-e0ad-4a13-9d46-7b78d5ef8aa7-istio-token\") pod \"e2e-api-a-7406a9-openshift-default-74c9f8d955-rqrs4\" (UID: \"ab2a5b74-e0ad-4a13-9d46-7b78d5ef8aa7\") " pod="openshift-ingress/e2e-api-a-7406a9-openshift-default-74c9f8d955-rqrs4" Nov 03 00:59:05.375753 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:59:05.375702 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 00:59:05.403930 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:59:05.399568 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ab2a5b74-e0ad-4a13-9d46-7b78d5ef8aa7-istio-podinfo\") pod \"e2e-api-a-7406a9-openshift-default-74c9f8d955-rqrs4\" (UID: \"ab2a5b74-e0ad-4a13-9d46-7b78d5ef8aa7\") " pod="openshift-ingress/e2e-api-a-7406a9-openshift-default-74c9f8d955-rqrs4" Nov 03 00:59:05.403930 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:59:05.400628 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6vd5r\" (UniqueName: \"kubernetes.io/projected/ab2a5b74-e0ad-4a13-9d46-7b78d5ef8aa7-kube-api-access-6vd5r\") pod \"e2e-api-a-7406a9-openshift-default-74c9f8d955-rqrs4\" (UID: \"ab2a5b74-e0ad-4a13-9d46-7b78d5ef8aa7\") " pod="openshift-ingress/e2e-api-a-7406a9-openshift-default-74c9f8d955-rqrs4" Nov 03 00:59:05.485105 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:59:05.481547 4939 patch_prober.go:29] interesting pod/e2e-mt-dd03e395-openshift-default-75799f6fc-4gdnp container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 03 00:59:05.485105 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:59:05.481637 4939 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-mt-dd03e395-openshift-default-75799f6fc-4gdnp" podUID="dd1a7eae-e782-4a82-bdac-c45c75614357" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 03 00:59:05.675639 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:59:05.675600 4939 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-7406a9-openshift-default-74c9f8d955-rqrs4" Nov 03 00:59:05.756255 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:59:05.756186 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 00:59:05.885387 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:59:05.885149 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 00:59:05.981198 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:59:05.979897 4939 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-7406a9-openshift-default-74c9f8d955-rqrs4"] Nov 03 00:59:05.989829 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:59:05.989746 4939 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 00:59:05.989973 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:59:05.989899 4939 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 00:59:05.990037 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:59:05.989988 4939 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 00:59:06.357957 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:59:06.357921 4939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-7406a9-openshift-default-74c9f8d955-rqrs4" event={"ID":"ab2a5b74-e0ad-4a13-9d46-7b78d5ef8aa7","Type":"ContainerStarted","Data":"a6df97fdc2b980db01fd7357a1e20e13cbe9006cb42e82f275f50fe8a992651a"} Nov 03 00:59:06.358124 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:59:06.357990 4939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-7406a9-openshift-default-74c9f8d955-rqrs4" event={"ID":"ab2a5b74-e0ad-4a13-9d46-7b78d5ef8aa7","Type":"ContainerStarted","Data":"a12d574c8798148e0f5242f7c103db97f295572fa048fd637a45f0c59e3937e0"} Nov 03 00:59:06.379647 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:59:06.379596 4939 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-api-a-7406a9-openshift-default-74c9f8d955-rqrs4" podStartSLOduration=1.379582445 podStartE2EDuration="1.379582445s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:59:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:59:06.379265746 +0000 UTC m=+1094.643600692" watchObservedRunningTime="2026-11-03 00:59:06.379582445 +0000 UTC m=+1094.643917390" Nov 03 00:59:06.542934 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:59:06.532149 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 00:59:06.676372 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:59:06.676297 4939 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-7406a9-openshift-default-74c9f8d955-rqrs4" Nov 03 00:59:06.704384 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:59:06.704352 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 00:59:06.994328 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:59:06.994243 4939 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-b-47369d-openshift-default-696ccdd997-9mqtd"] Nov 03 00:59:06.999220 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:59:06.999200 4939 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-47369d-openshift-default-696ccdd997-9mqtd" Nov 03 00:59:07.013012 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:59:07.012989 4939 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-b-47369d-openshift-default-dockercfg-gr2cj\"" Nov 03 00:59:07.015162 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:59:07.015134 4939 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-47369d-openshift-default-696ccdd997-9mqtd"] Nov 03 00:59:07.083722 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:59:07.083678 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 00:59:07.199932 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:59:07.194940 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/49df4a8a-81d6-4c9e-baf7-40c0f7cbaae1-istio-podinfo\") pod \"e2e-api-b-47369d-openshift-default-696ccdd997-9mqtd\" (UID: \"49df4a8a-81d6-4c9e-baf7-40c0f7cbaae1\") " pod="openshift-ingress/e2e-api-b-47369d-openshift-default-696ccdd997-9mqtd" Nov 03 00:59:07.199932 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:59:07.195029 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/49df4a8a-81d6-4c9e-baf7-40c0f7cbaae1-istio-token\") pod \"e2e-api-b-47369d-openshift-default-696ccdd997-9mqtd\" (UID: \"49df4a8a-81d6-4c9e-baf7-40c0f7cbaae1\") " pod="openshift-ingress/e2e-api-b-47369d-openshift-default-696ccdd997-9mqtd" Nov 03 00:59:07.199932 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:59:07.195152 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/49df4a8a-81d6-4c9e-baf7-40c0f7cbaae1-workload-certs\") pod \"e2e-api-b-47369d-openshift-default-696ccdd997-9mqtd\" (UID: \"49df4a8a-81d6-4c9e-baf7-40c0f7cbaae1\") " pod="openshift-ingress/e2e-api-b-47369d-openshift-default-696ccdd997-9mqtd" Nov 03 00:59:07.199932 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:59:07.195192 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/49df4a8a-81d6-4c9e-baf7-40c0f7cbaae1-istio-envoy\") pod \"e2e-api-b-47369d-openshift-default-696ccdd997-9mqtd\" (UID: \"49df4a8a-81d6-4c9e-baf7-40c0f7cbaae1\") " pod="openshift-ingress/e2e-api-b-47369d-openshift-default-696ccdd997-9mqtd" Nov 03 00:59:07.199932 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:59:07.195245 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gnbms\" (UniqueName: \"kubernetes.io/projected/49df4a8a-81d6-4c9e-baf7-40c0f7cbaae1-kube-api-access-gnbms\") pod \"e2e-api-b-47369d-openshift-default-696ccdd997-9mqtd\" (UID: \"49df4a8a-81d6-4c9e-baf7-40c0f7cbaae1\") " pod="openshift-ingress/e2e-api-b-47369d-openshift-default-696ccdd997-9mqtd" Nov 03 00:59:07.199932 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:59:07.195345 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/49df4a8a-81d6-4c9e-baf7-40c0f7cbaae1-istiod-ca-cert\") pod \"e2e-api-b-47369d-openshift-default-696ccdd997-9mqtd\" (UID: \"49df4a8a-81d6-4c9e-baf7-40c0f7cbaae1\") " pod="openshift-ingress/e2e-api-b-47369d-openshift-default-696ccdd997-9mqtd" Nov 03 00:59:07.199932 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:59:07.195379 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/49df4a8a-81d6-4c9e-baf7-40c0f7cbaae1-workload-socket\") pod \"e2e-api-b-47369d-openshift-default-696ccdd997-9mqtd\" (UID: \"49df4a8a-81d6-4c9e-baf7-40c0f7cbaae1\") " pod="openshift-ingress/e2e-api-b-47369d-openshift-default-696ccdd997-9mqtd" Nov 03 00:59:07.199932 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:59:07.195535 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/49df4a8a-81d6-4c9e-baf7-40c0f7cbaae1-istio-data\") pod \"e2e-api-b-47369d-openshift-default-696ccdd997-9mqtd\" (UID: \"49df4a8a-81d6-4c9e-baf7-40c0f7cbaae1\") " pod="openshift-ingress/e2e-api-b-47369d-openshift-default-696ccdd997-9mqtd" Nov 03 00:59:07.199932 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:59:07.195565 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/49df4a8a-81d6-4c9e-baf7-40c0f7cbaae1-credential-socket\") pod \"e2e-api-b-47369d-openshift-default-696ccdd997-9mqtd\" (UID: \"49df4a8a-81d6-4c9e-baf7-40c0f7cbaae1\") " pod="openshift-ingress/e2e-api-b-47369d-openshift-default-696ccdd997-9mqtd" Nov 03 00:59:07.304977 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:59:07.303799 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/49df4a8a-81d6-4c9e-baf7-40c0f7cbaae1-workload-socket\") pod \"e2e-api-b-47369d-openshift-default-696ccdd997-9mqtd\" (UID: \"49df4a8a-81d6-4c9e-baf7-40c0f7cbaae1\") " pod="openshift-ingress/e2e-api-b-47369d-openshift-default-696ccdd997-9mqtd" Nov 03 00:59:07.304977 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:59:07.303971 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/49df4a8a-81d6-4c9e-baf7-40c0f7cbaae1-istio-data\") pod \"e2e-api-b-47369d-openshift-default-696ccdd997-9mqtd\" (UID: \"49df4a8a-81d6-4c9e-baf7-40c0f7cbaae1\") " pod="openshift-ingress/e2e-api-b-47369d-openshift-default-696ccdd997-9mqtd" Nov 03 00:59:07.304977 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:59:07.304009 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/49df4a8a-81d6-4c9e-baf7-40c0f7cbaae1-credential-socket\") pod \"e2e-api-b-47369d-openshift-default-696ccdd997-9mqtd\" (UID: \"49df4a8a-81d6-4c9e-baf7-40c0f7cbaae1\") " pod="openshift-ingress/e2e-api-b-47369d-openshift-default-696ccdd997-9mqtd" Nov 03 00:59:07.304977 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:59:07.304056 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/49df4a8a-81d6-4c9e-baf7-40c0f7cbaae1-istio-podinfo\") pod \"e2e-api-b-47369d-openshift-default-696ccdd997-9mqtd\" (UID: \"49df4a8a-81d6-4c9e-baf7-40c0f7cbaae1\") " pod="openshift-ingress/e2e-api-b-47369d-openshift-default-696ccdd997-9mqtd" Nov 03 00:59:07.304977 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:59:07.304123 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/49df4a8a-81d6-4c9e-baf7-40c0f7cbaae1-istio-token\") pod \"e2e-api-b-47369d-openshift-default-696ccdd997-9mqtd\" (UID: \"49df4a8a-81d6-4c9e-baf7-40c0f7cbaae1\") " pod="openshift-ingress/e2e-api-b-47369d-openshift-default-696ccdd997-9mqtd" Nov 03 00:59:07.304977 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:59:07.304268 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/49df4a8a-81d6-4c9e-baf7-40c0f7cbaae1-workload-certs\") pod \"e2e-api-b-47369d-openshift-default-696ccdd997-9mqtd\" (UID: \"49df4a8a-81d6-4c9e-baf7-40c0f7cbaae1\") " pod="openshift-ingress/e2e-api-b-47369d-openshift-default-696ccdd997-9mqtd" Nov 03 00:59:07.304977 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:59:07.304316 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/49df4a8a-81d6-4c9e-baf7-40c0f7cbaae1-istio-envoy\") pod \"e2e-api-b-47369d-openshift-default-696ccdd997-9mqtd\" (UID: \"49df4a8a-81d6-4c9e-baf7-40c0f7cbaae1\") " pod="openshift-ingress/e2e-api-b-47369d-openshift-default-696ccdd997-9mqtd" Nov 03 00:59:07.304977 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:59:07.304364 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gnbms\" (UniqueName: \"kubernetes.io/projected/49df4a8a-81d6-4c9e-baf7-40c0f7cbaae1-kube-api-access-gnbms\") pod \"e2e-api-b-47369d-openshift-default-696ccdd997-9mqtd\" (UID: \"49df4a8a-81d6-4c9e-baf7-40c0f7cbaae1\") " pod="openshift-ingress/e2e-api-b-47369d-openshift-default-696ccdd997-9mqtd" Nov 03 00:59:07.304977 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:59:07.304463 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/49df4a8a-81d6-4c9e-baf7-40c0f7cbaae1-istiod-ca-cert\") pod \"e2e-api-b-47369d-openshift-default-696ccdd997-9mqtd\" (UID: \"49df4a8a-81d6-4c9e-baf7-40c0f7cbaae1\") " pod="openshift-ingress/e2e-api-b-47369d-openshift-default-696ccdd997-9mqtd" Nov 03 00:59:07.305496 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:59:07.305372 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/49df4a8a-81d6-4c9e-baf7-40c0f7cbaae1-istiod-ca-cert\") pod \"e2e-api-b-47369d-openshift-default-696ccdd997-9mqtd\" (UID: \"49df4a8a-81d6-4c9e-baf7-40c0f7cbaae1\") " pod="openshift-ingress/e2e-api-b-47369d-openshift-default-696ccdd997-9mqtd" Nov 03 00:59:07.318930 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:59:07.305716 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/49df4a8a-81d6-4c9e-baf7-40c0f7cbaae1-workload-socket\") pod \"e2e-api-b-47369d-openshift-default-696ccdd997-9mqtd\" (UID: \"49df4a8a-81d6-4c9e-baf7-40c0f7cbaae1\") " pod="openshift-ingress/e2e-api-b-47369d-openshift-default-696ccdd997-9mqtd" Nov 03 00:59:07.318930 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:59:07.305985 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/49df4a8a-81d6-4c9e-baf7-40c0f7cbaae1-istio-data\") pod \"e2e-api-b-47369d-openshift-default-696ccdd997-9mqtd\" (UID: \"49df4a8a-81d6-4c9e-baf7-40c0f7cbaae1\") " pod="openshift-ingress/e2e-api-b-47369d-openshift-default-696ccdd997-9mqtd" Nov 03 00:59:07.318930 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:59:07.306262 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/49df4a8a-81d6-4c9e-baf7-40c0f7cbaae1-credential-socket\") pod \"e2e-api-b-47369d-openshift-default-696ccdd997-9mqtd\" (UID: \"49df4a8a-81d6-4c9e-baf7-40c0f7cbaae1\") " pod="openshift-ingress/e2e-api-b-47369d-openshift-default-696ccdd997-9mqtd" Nov 03 00:59:07.319328 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:59:07.319120 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/49df4a8a-81d6-4c9e-baf7-40c0f7cbaae1-workload-certs\") pod \"e2e-api-b-47369d-openshift-default-696ccdd997-9mqtd\" (UID: \"49df4a8a-81d6-4c9e-baf7-40c0f7cbaae1\") " pod="openshift-ingress/e2e-api-b-47369d-openshift-default-696ccdd997-9mqtd" Nov 03 00:59:07.319996 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:59:07.319969 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/49df4a8a-81d6-4c9e-baf7-40c0f7cbaae1-istio-podinfo\") pod \"e2e-api-b-47369d-openshift-default-696ccdd997-9mqtd\" (UID: \"49df4a8a-81d6-4c9e-baf7-40c0f7cbaae1\") " pod="openshift-ingress/e2e-api-b-47369d-openshift-default-696ccdd997-9mqtd" Nov 03 00:59:07.344927 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:59:07.340706 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/49df4a8a-81d6-4c9e-baf7-40c0f7cbaae1-istio-envoy\") pod \"e2e-api-b-47369d-openshift-default-696ccdd997-9mqtd\" (UID: \"49df4a8a-81d6-4c9e-baf7-40c0f7cbaae1\") " pod="openshift-ingress/e2e-api-b-47369d-openshift-default-696ccdd997-9mqtd" Nov 03 00:59:07.360940 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:59:07.349786 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gnbms\" (UniqueName: \"kubernetes.io/projected/49df4a8a-81d6-4c9e-baf7-40c0f7cbaae1-kube-api-access-gnbms\") pod \"e2e-api-b-47369d-openshift-default-696ccdd997-9mqtd\" (UID: \"49df4a8a-81d6-4c9e-baf7-40c0f7cbaae1\") " pod="openshift-ingress/e2e-api-b-47369d-openshift-default-696ccdd997-9mqtd" Nov 03 00:59:07.370333 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:59:07.361492 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/49df4a8a-81d6-4c9e-baf7-40c0f7cbaae1-istio-token\") pod \"e2e-api-b-47369d-openshift-default-696ccdd997-9mqtd\" (UID: \"49df4a8a-81d6-4c9e-baf7-40c0f7cbaae1\") " pod="openshift-ingress/e2e-api-b-47369d-openshift-default-696ccdd997-9mqtd" Nov 03 00:59:07.463969 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:59:07.460680 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 00:59:07.630951 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:59:07.621397 4939 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-47369d-openshift-default-696ccdd997-9mqtd" Nov 03 00:59:07.686525 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:59:07.686494 4939 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-7406a9-openshift-default-74c9f8d955-rqrs4" Nov 03 00:59:07.738618 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:59:07.738568 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 00:59:07.946924 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:59:07.946869 4939 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-47369d-openshift-default-696ccdd997-9mqtd"] Nov 03 00:59:07.948728 ip-10-0-140-107 kubenswrapper[4939]: W1103 00:59:07.948700 4939 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod49df4a8a_81d6_4c9e_baf7_40c0f7cbaae1.slice/crio-cd0e55f9205d8d8122f7868cd8cae0c9c807a0881bfab6b65bee314cbcb6ad9d WatchSource:0}: Error finding container cd0e55f9205d8d8122f7868cd8cae0c9c807a0881bfab6b65bee314cbcb6ad9d: Status 404 returned error can't find the container with id cd0e55f9205d8d8122f7868cd8cae0c9c807a0881bfab6b65bee314cbcb6ad9d Nov 03 00:59:07.953892 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:59:07.953853 4939 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 00:59:07.954012 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:59:07.953988 4939 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 00:59:07.954068 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:59:07.954037 4939 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 00:59:08.419695 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:59:08.410595 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 00:59:08.538096 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:59:08.521197 4939 generic.go:356] "Generic (PLEG): container finished" podID="ef4163bb-8632-4675-a678-11ec6d58e737" containerID="e5e40ad69efe3af5e0379a278b807446fbeb02e1055b85c53a413a32b985a78b" exitCode=0 Nov 03 00:59:08.538096 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:59:08.521404 4939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-07f5f6-openshift-default-98746c89f-vz567" event={"ID":"ef4163bb-8632-4675-a678-11ec6d58e737","Type":"ContainerDied","Data":"e5e40ad69efe3af5e0379a278b807446fbeb02e1055b85c53a413a32b985a78b"} Nov 03 00:59:08.538096 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:59:08.522066 4939 scope.go:122] "RemoveContainer" containerID="e5e40ad69efe3af5e0379a278b807446fbeb02e1055b85c53a413a32b985a78b" Nov 03 00:59:08.538096 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:59:08.525594 4939 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 00:59:08.538096 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:59:08.525699 4939 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 00:59:08.538096 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:59:08.525745 4939 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 00:59:08.538096 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:59:08.530567 4939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-47369d-openshift-default-696ccdd997-9mqtd" event={"ID":"49df4a8a-81d6-4c9e-baf7-40c0f7cbaae1","Type":"ContainerStarted","Data":"2a7c98c6f2826c0cc792ea5c299ab7bf08c0753c213be3dced1475e3c2330a95"} Nov 03 00:59:08.538096 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:59:08.530622 4939 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-7406a9-openshift-default-74c9f8d955-rqrs4" Nov 03 00:59:08.538096 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:59:08.530650 4939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-47369d-openshift-default-696ccdd997-9mqtd" event={"ID":"49df4a8a-81d6-4c9e-baf7-40c0f7cbaae1","Type":"ContainerStarted","Data":"cd0e55f9205d8d8122f7868cd8cae0c9c807a0881bfab6b65bee314cbcb6ad9d"} Nov 03 00:59:08.637861 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:59:08.630650 4939 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-7406a9-openshift-default-74c9f8d955-rqrs4" Nov 03 00:59:08.637861 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:59:08.632671 4939 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-47369d-openshift-default-696ccdd997-9mqtd" Nov 03 00:59:08.637861 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:59:08.634015 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 00:59:08.724495 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:59:08.724415 4939 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-api-b-47369d-openshift-default-696ccdd997-9mqtd" podStartSLOduration=2.724382433 podStartE2EDuration="2.724382433s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:59:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:59:08.622773871 +0000 UTC m=+1096.887108819" watchObservedRunningTime="2026-11-03 00:59:08.724382433 +0000 UTC m=+1096.988717381" Nov 03 00:59:08.877712 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:59:08.877665 4939 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-4pw7v"] Nov 03 00:59:08.905552 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:59:08.905451 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 00:59:09.169003 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:59:09.116025 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 00:59:09.304655 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:59:09.304586 4939 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-api-b-eed369-7d94fb8545-hdldv" podUID="0ae23a7b-9f49-479e-bfd5-7732c28052aa" containerName="payload-processing" containerID="cri-o://eb5f271d302c42f074bf45c32cdcdfb927cc0ddd6bb8a344638107571d67bed6" gracePeriod=25 Nov 03 00:59:09.719812 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:59:09.719444 4939 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-47369d-openshift-default-696ccdd997-9mqtd" Nov 03 00:59:09.798133 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:59:09.798104 4939 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-dd03e395-openshift-default-75799f6fc-4gdnp" Nov 03 00:59:09.831210 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:59:09.831172 4939 generic.go:356] "Generic (PLEG): container finished" podID="dd1a7eae-e782-4a82-bdac-c45c75614357" containerID="a3457590cb89b41b7165836f93cdce29fd1ad3f88f6139cc1ba7799be0513756" exitCode=0 Nov 03 00:59:09.831447 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:59:09.831220 4939 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-dd03e395-openshift-default-75799f6fc-4gdnp" Nov 03 00:59:09.831630 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:59:09.831238 4939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-dd03e395-openshift-default-75799f6fc-4gdnp" event={"ID":"dd1a7eae-e782-4a82-bdac-c45c75614357","Type":"ContainerDied","Data":"a3457590cb89b41b7165836f93cdce29fd1ad3f88f6139cc1ba7799be0513756"} Nov 03 00:59:09.831692 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:59:09.831676 4939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-dd03e395-openshift-default-75799f6fc-4gdnp" event={"ID":"dd1a7eae-e782-4a82-bdac-c45c75614357","Type":"ContainerDied","Data":"57dd79d1405d00ca9315750aacdfbddd9fb8ae55ca5167b9a0fa80f211d30416"} Nov 03 00:59:09.832490 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:59:09.831710 4939 scope.go:122] "RemoveContainer" containerID="a3457590cb89b41b7165836f93cdce29fd1ad3f88f6139cc1ba7799be0513756" Nov 03 00:59:09.837349 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:59:09.837325 4939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-07f5f6-openshift-default-98746c89f-vz567" event={"ID":"ef4163bb-8632-4675-a678-11ec6d58e737","Type":"ContainerStarted","Data":"52e11dadb127b4b182eca770f7958e3db330130f547828282776635d0e75bb2f"} Nov 03 00:59:09.841548 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:59:09.841528 4939 generic.go:356] "Generic (PLEG): container finished" podID="0ae23a7b-9f49-479e-bfd5-7732c28052aa" containerID="eb5f271d302c42f074bf45c32cdcdfb927cc0ddd6bb8a344638107571d67bed6" exitCode=0 Nov 03 00:59:09.843876 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:59:09.843074 4939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-eed369-7d94fb8545-hdldv" event={"ID":"0ae23a7b-9f49-479e-bfd5-7732c28052aa","Type":"ContainerDied","Data":"eb5f271d302c42f074bf45c32cdcdfb927cc0ddd6bb8a344638107571d67bed6"} Nov 03 00:59:09.845196 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:59:09.845177 4939 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-47369d-openshift-default-696ccdd997-9mqtd" Nov 03 00:59:09.856292 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:59:09.850494 4939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dd1a7eae-e782-4a82-bdac-c45c75614357-kube-api-access-wp9xr\" (UniqueName: \"kubernetes.io/projected/dd1a7eae-e782-4a82-bdac-c45c75614357-kube-api-access-wp9xr\") pod \"dd1a7eae-e782-4a82-bdac-c45c75614357\" (UID: \"dd1a7eae-e782-4a82-bdac-c45c75614357\") " Nov 03 00:59:09.856292 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:59:09.850548 4939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dd1a7eae-e782-4a82-bdac-c45c75614357-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dd1a7eae-e782-4a82-bdac-c45c75614357-credential-socket\") pod \"dd1a7eae-e782-4a82-bdac-c45c75614357\" (UID: \"dd1a7eae-e782-4a82-bdac-c45c75614357\") " Nov 03 00:59:09.856292 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:59:09.850705 4939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dd1a7eae-e782-4a82-bdac-c45c75614357-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dd1a7eae-e782-4a82-bdac-c45c75614357-istio-data\") pod \"dd1a7eae-e782-4a82-bdac-c45c75614357\" (UID: \"dd1a7eae-e782-4a82-bdac-c45c75614357\") " Nov 03 00:59:09.856292 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:59:09.850750 4939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dd1a7eae-e782-4a82-bdac-c45c75614357-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dd1a7eae-e782-4a82-bdac-c45c75614357-istio-envoy\") pod \"dd1a7eae-e782-4a82-bdac-c45c75614357\" (UID: \"dd1a7eae-e782-4a82-bdac-c45c75614357\") " Nov 03 00:59:09.856292 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:59:09.850783 4939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/dd1a7eae-e782-4a82-bdac-c45c75614357-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dd1a7eae-e782-4a82-bdac-c45c75614357-istio-podinfo\") pod \"dd1a7eae-e782-4a82-bdac-c45c75614357\" (UID: \"dd1a7eae-e782-4a82-bdac-c45c75614357\") " Nov 03 00:59:09.856292 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:59:09.850834 4939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/dd1a7eae-e782-4a82-bdac-c45c75614357-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dd1a7eae-e782-4a82-bdac-c45c75614357-istiod-ca-cert\") pod \"dd1a7eae-e782-4a82-bdac-c45c75614357\" (UID: \"dd1a7eae-e782-4a82-bdac-c45c75614357\") " Nov 03 00:59:09.856292 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:59:09.851227 4939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dd1a7eae-e782-4a82-bdac-c45c75614357-credential-socket" pod "dd1a7eae-e782-4a82-bdac-c45c75614357" (UID: "dd1a7eae-e782-4a82-bdac-c45c75614357"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 00:59:09.856292 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:59:09.854051 4939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dd1a7eae-e782-4a82-bdac-c45c75614357-istio-data" pod "dd1a7eae-e782-4a82-bdac-c45c75614357" (UID: "dd1a7eae-e782-4a82-bdac-c45c75614357"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 00:59:09.856292 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:59:09.854367 4939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dd1a7eae-e782-4a82-bdac-c45c75614357-istiod-ca-cert" pod "dd1a7eae-e782-4a82-bdac-c45c75614357" (UID: "dd1a7eae-e782-4a82-bdac-c45c75614357"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 00:59:09.861109 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:59:09.858136 4939 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-47369d-openshift-default-696ccdd997-9mqtd" Nov 03 00:59:09.861109 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:59:09.858509 4939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dd1a7eae-e782-4a82-bdac-c45c75614357-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dd1a7eae-e782-4a82-bdac-c45c75614357-workload-socket\") pod \"dd1a7eae-e782-4a82-bdac-c45c75614357\" (UID: \"dd1a7eae-e782-4a82-bdac-c45c75614357\") " Nov 03 00:59:09.861109 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:59:09.858752 4939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dd1a7eae-e782-4a82-bdac-c45c75614357-workload-socket" pod "dd1a7eae-e782-4a82-bdac-c45c75614357" (UID: "dd1a7eae-e782-4a82-bdac-c45c75614357"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 00:59:09.861109 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:59:09.859057 4939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dd1a7eae-e782-4a82-bdac-c45c75614357-workload-certs" pod "dd1a7eae-e782-4a82-bdac-c45c75614357" (UID: "dd1a7eae-e782-4a82-bdac-c45c75614357"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 00:59:09.861109 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:59:09.858841 4939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dd1a7eae-e782-4a82-bdac-c45c75614357-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dd1a7eae-e782-4a82-bdac-c45c75614357-workload-certs\") pod \"dd1a7eae-e782-4a82-bdac-c45c75614357\" (UID: \"dd1a7eae-e782-4a82-bdac-c45c75614357\") " Nov 03 00:59:09.861109 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:59:09.859143 4939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dd1a7eae-e782-4a82-bdac-c45c75614357-istio-token\" (UniqueName: \"kubernetes.io/projected/dd1a7eae-e782-4a82-bdac-c45c75614357-istio-token\") pod \"dd1a7eae-e782-4a82-bdac-c45c75614357\" (UID: \"dd1a7eae-e782-4a82-bdac-c45c75614357\") " Nov 03 00:59:10.185294 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:59:10.181293 4939 scope.go:122] "RemoveContainer" containerID="a3457590cb89b41b7165836f93cdce29fd1ad3f88f6139cc1ba7799be0513756" Nov 03 00:59:10.195602 ip-10-0-140-107 kubenswrapper[4939]: E1103 00:59:10.195567 4939 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a3457590cb89b41b7165836f93cdce29fd1ad3f88f6139cc1ba7799be0513756\": container with ID starting with a3457590cb89b41b7165836f93cdce29fd1ad3f88f6139cc1ba7799be0513756 not found: ID does not exist" containerID="a3457590cb89b41b7165836f93cdce29fd1ad3f88f6139cc1ba7799be0513756" Nov 03 00:59:10.195856 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:59:10.195826 4939 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a3457590cb89b41b7165836f93cdce29fd1ad3f88f6139cc1ba7799be0513756"} err="failed to get container status \"a3457590cb89b41b7165836f93cdce29fd1ad3f88f6139cc1ba7799be0513756\": rpc error: code = NotFound desc = could not find container \"a3457590cb89b41b7165836f93cdce29fd1ad3f88f6139cc1ba7799be0513756\": container with ID starting with a3457590cb89b41b7165836f93cdce29fd1ad3f88f6139cc1ba7799be0513756 not found: ID does not exist" Nov 03 00:59:10.221959 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:59:10.211436 4939 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dd1a7eae-e782-4a82-bdac-c45c75614357-istio-data\") on node \"ip-10-0-140-107.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 00:59:10.221959 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:59:10.212807 4939 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dd1a7eae-e782-4a82-bdac-c45c75614357-istiod-ca-cert\") on node \"ip-10-0-140-107.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 00:59:10.221959 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:59:10.212833 4939 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dd1a7eae-e782-4a82-bdac-c45c75614357-workload-socket\") on node \"ip-10-0-140-107.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 00:59:10.221959 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:59:10.212846 4939 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dd1a7eae-e782-4a82-bdac-c45c75614357-workload-certs\") on node \"ip-10-0-140-107.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 00:59:10.221959 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:59:10.212859 4939 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dd1a7eae-e782-4a82-bdac-c45c75614357-credential-socket\") on node \"ip-10-0-140-107.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 00:59:10.263229 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:59:10.263192 4939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/dd1a7eae-e782-4a82-bdac-c45c75614357-istio-podinfo" pod "dd1a7eae-e782-4a82-bdac-c45c75614357" (UID: "dd1a7eae-e782-4a82-bdac-c45c75614357"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 00:59:10.264116 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:59:10.264084 4939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dd1a7eae-e782-4a82-bdac-c45c75614357-istio-envoy" pod "dd1a7eae-e782-4a82-bdac-c45c75614357" (UID: "dd1a7eae-e782-4a82-bdac-c45c75614357"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 00:59:10.269071 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:59:10.268095 4939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dd1a7eae-e782-4a82-bdac-c45c75614357-istio-token" pod "dd1a7eae-e782-4a82-bdac-c45c75614357" (UID: "dd1a7eae-e782-4a82-bdac-c45c75614357"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 00:59:10.270432 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:59:10.270385 4939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dd1a7eae-e782-4a82-bdac-c45c75614357-kube-api-access-wp9xr" pod "dd1a7eae-e782-4a82-bdac-c45c75614357" (UID: "dd1a7eae-e782-4a82-bdac-c45c75614357"). InnerVolumeSpecName "kube-api-access-wp9xr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 00:59:10.314931 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:59:10.313379 4939 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-eed369-7d94fb8545-hdldv" Nov 03 00:59:10.314931 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:59:10.314684 4939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0ae23a7b-9f49-479e-bfd5-7732c28052aa-kube-api-access-7r269\" (UniqueName: \"kubernetes.io/projected/0ae23a7b-9f49-479e-bfd5-7732c28052aa-kube-api-access-7r269\") pod \"0ae23a7b-9f49-479e-bfd5-7732c28052aa\" (UID: \"0ae23a7b-9f49-479e-bfd5-7732c28052aa\") " Nov 03 00:59:10.315284 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:59:10.315257 4939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0ae23a7b-9f49-479e-bfd5-7732c28052aa-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0ae23a7b-9f49-479e-bfd5-7732c28052aa-plugins-config-volume\") pod \"0ae23a7b-9f49-479e-bfd5-7732c28052aa\" (UID: \"0ae23a7b-9f49-479e-bfd5-7732c28052aa\") " Nov 03 00:59:10.315745 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:59:10.315636 4939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0ae23a7b-9f49-479e-bfd5-7732c28052aa-plugins-config-volume" pod "0ae23a7b-9f49-479e-bfd5-7732c28052aa" (UID: "0ae23a7b-9f49-479e-bfd5-7732c28052aa"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 00:59:10.317114 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:59:10.317088 4939 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wp9xr\" (UniqueName: \"kubernetes.io/projected/dd1a7eae-e782-4a82-bdac-c45c75614357-kube-api-access-wp9xr\") on node \"ip-10-0-140-107.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 00:59:10.317225 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:59:10.317117 4939 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0ae23a7b-9f49-479e-bfd5-7732c28052aa-plugins-config-volume\") on node \"ip-10-0-140-107.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 00:59:10.317225 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:59:10.317135 4939 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dd1a7eae-e782-4a82-bdac-c45c75614357-istio-envoy\") on node \"ip-10-0-140-107.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 00:59:10.317225 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:59:10.317150 4939 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dd1a7eae-e782-4a82-bdac-c45c75614357-istio-podinfo\") on node \"ip-10-0-140-107.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 00:59:10.317225 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:59:10.317165 4939 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dd1a7eae-e782-4a82-bdac-c45c75614357-istio-token\") on node \"ip-10-0-140-107.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 00:59:10.319456 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:59:10.319424 4939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0ae23a7b-9f49-479e-bfd5-7732c28052aa-kube-api-access-7r269" pod "0ae23a7b-9f49-479e-bfd5-7732c28052aa" (UID: "0ae23a7b-9f49-479e-bfd5-7732c28052aa"). InnerVolumeSpecName "kube-api-access-7r269". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 00:59:10.421230 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:59:10.420608 4939 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7r269\" (UniqueName: \"kubernetes.io/projected/0ae23a7b-9f49-479e-bfd5-7732c28052aa-kube-api-access-7r269\") on node \"ip-10-0-140-107.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 00:59:10.509304 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:59:10.507767 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 00:59:10.518658 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:59:10.518372 4939 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-7406a9-78cc96967f-6wksw"] Nov 03 00:59:10.520272 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:59:10.520244 4939 cpu_manager.go:414] "containerMap: removing container" podUID="0ae23a7b-9f49-479e-bfd5-7732c28052aa" containerName="payload-processing" Nov 03 00:59:10.520373 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:59:10.520272 4939 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0ae23a7b-9f49-479e-bfd5-7732c28052aa" containerName="payload-processing" Nov 03 00:59:10.520373 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:59:10.520297 4939 cpu_manager.go:414] "containerMap: removing container" podUID="dd1a7eae-e782-4a82-bdac-c45c75614357" containerName="istio-proxy" Nov 03 00:59:10.520373 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:59:10.520306 4939 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dd1a7eae-e782-4a82-bdac-c45c75614357" containerName="istio-proxy" Nov 03 00:59:10.520588 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:59:10.520574 4939 memory_manager.go:365] "RemoveStaleState removing state" podUID="0ae23a7b-9f49-479e-bfd5-7732c28052aa" containerName="payload-processing" Nov 03 00:59:10.520637 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:59:10.520596 4939 memory_manager.go:365] "RemoveStaleState removing state" podUID="dd1a7eae-e782-4a82-bdac-c45c75614357" containerName="istio-proxy" Nov 03 00:59:10.525243 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:59:10.525219 4939 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-7406a9-78cc96967f-6wksw" Nov 03 00:59:10.530644 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:59:10.530611 4939 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-a-7406a9\"" Nov 03 00:59:10.531271 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:59:10.531245 4939 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-a-7406a9-dockercfg-pq72m\"" Nov 03 00:59:10.533407 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:59:10.533381 4939 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-7406a9-78cc96967f-6wksw"] Nov 03 00:59:10.569167 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:59:10.546614 4939 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-dd03e395-openshift-default-75799f6fc-4gdnp"] Nov 03 00:59:10.621699 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:59:10.610259 4939 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-dd03e395-openshift-default-75799f6fc-4gdnp"] Nov 03 00:59:10.628880 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:59:10.628835 4939 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-7406a9-5cd77855fc-n4s94"] Nov 03 00:59:10.629494 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:59:10.628725 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jbnx9\" (UniqueName: \"kubernetes.io/projected/12ba5035-bd74-4162-9c8e-9353527f757f-kube-api-access-jbnx9\") pod \"payload-pre-processing-e2e-api-a-7406a9-78cc96967f-6wksw\" (UID: \"12ba5035-bd74-4162-9c8e-9353527f757f\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-7406a9-78cc96967f-6wksw" Nov 03 00:59:10.629614 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:59:10.629503 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/12ba5035-bd74-4162-9c8e-9353527f757f-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-7406a9-78cc96967f-6wksw\" (UID: \"12ba5035-bd74-4162-9c8e-9353527f757f\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-7406a9-78cc96967f-6wksw" Nov 03 00:59:10.801983 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:59:10.796141 4939 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-7406a9-5cd77855fc-n4s94"] Nov 03 00:59:10.851104 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:59:10.832017 4939 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-a-07f5f6-openshift-default-98746c89f-vz567" Nov 03 00:59:10.851104 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:59:10.803103 4939 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-7406a9-5cd77855fc-n4s94" Nov 03 00:59:10.947934 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:59:10.925094 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/12ba5035-bd74-4162-9c8e-9353527f757f-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-7406a9-78cc96967f-6wksw\" (UID: \"12ba5035-bd74-4162-9c8e-9353527f757f\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-7406a9-78cc96967f-6wksw" Nov 03 00:59:11.101990 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:59:11.031451 4939 patch_prober.go:29] interesting pod/e2e-ipp-a-07f5f6-openshift-default-98746c89f-vz567 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.105:15021/healthz/ready\": dial tcp 10.132.0.105:15021: connect: connection refused" start-of-body= Nov 03 00:59:11.101990 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:59:11.054114 4939 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-07f5f6-openshift-default-98746c89f-vz567" podUID="ef4163bb-8632-4675-a678-11ec6d58e737" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.105:15021/healthz/ready\": dial tcp 10.132.0.105:15021: connect: connection refused" Nov 03 00:59:11.101990 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:59:11.042366 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 00:59:11.101990 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:59:11.031924 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/12ba5035-bd74-4162-9c8e-9353527f757f-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-7406a9-78cc96967f-6wksw\" (UID: \"12ba5035-bd74-4162-9c8e-9353527f757f\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-7406a9-78cc96967f-6wksw" Nov 03 00:59:11.113935 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:59:11.113422 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d5b08c7b-b02e-434f-8b32-32a502a92e69-plugins-config-volume\") pod \"payload-processing-e2e-api-a-7406a9-5cd77855fc-n4s94\" (UID: \"d5b08c7b-b02e-434f-8b32-32a502a92e69\") " pod="openshift-ingress/payload-processing-e2e-api-a-7406a9-5cd77855fc-n4s94" Nov 03 00:59:11.113935 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:59:11.113484 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qfcfj\" (UniqueName: \"kubernetes.io/projected/d5b08c7b-b02e-434f-8b32-32a502a92e69-kube-api-access-qfcfj\") pod \"payload-processing-e2e-api-a-7406a9-5cd77855fc-n4s94\" (UID: \"d5b08c7b-b02e-434f-8b32-32a502a92e69\") " pod="openshift-ingress/payload-processing-e2e-api-a-7406a9-5cd77855fc-n4s94" Nov 03 00:59:11.113935 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:59:11.113653 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jbnx9\" (UniqueName: \"kubernetes.io/projected/12ba5035-bd74-4162-9c8e-9353527f757f-kube-api-access-jbnx9\") pod \"payload-pre-processing-e2e-api-a-7406a9-78cc96967f-6wksw\" (UID: \"12ba5035-bd74-4162-9c8e-9353527f757f\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-7406a9-78cc96967f-6wksw" Nov 03 00:59:11.197933 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:59:11.181142 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jbnx9\" (UniqueName: \"kubernetes.io/projected/12ba5035-bd74-4162-9c8e-9353527f757f-kube-api-access-jbnx9\") pod \"payload-pre-processing-e2e-api-a-7406a9-78cc96967f-6wksw\" (UID: \"12ba5035-bd74-4162-9c8e-9353527f757f\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-7406a9-78cc96967f-6wksw" Nov 03 00:59:11.197933 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:59:11.181893 4939 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-7406a9-78cc96967f-6wksw" Nov 03 00:59:11.218984 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:59:11.218686 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d5b08c7b-b02e-434f-8b32-32a502a92e69-plugins-config-volume\") pod \"payload-processing-e2e-api-a-7406a9-5cd77855fc-n4s94\" (UID: \"d5b08c7b-b02e-434f-8b32-32a502a92e69\") " pod="openshift-ingress/payload-processing-e2e-api-a-7406a9-5cd77855fc-n4s94" Nov 03 00:59:11.218984 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:59:11.218748 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qfcfj\" (UniqueName: \"kubernetes.io/projected/d5b08c7b-b02e-434f-8b32-32a502a92e69-kube-api-access-qfcfj\") pod \"payload-processing-e2e-api-a-7406a9-5cd77855fc-n4s94\" (UID: \"d5b08c7b-b02e-434f-8b32-32a502a92e69\") " pod="openshift-ingress/payload-processing-e2e-api-a-7406a9-5cd77855fc-n4s94" Nov 03 00:59:11.221520 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:59:11.221334 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d5b08c7b-b02e-434f-8b32-32a502a92e69-plugins-config-volume\") pod \"payload-processing-e2e-api-a-7406a9-5cd77855fc-n4s94\" (UID: \"d5b08c7b-b02e-434f-8b32-32a502a92e69\") " pod="openshift-ingress/payload-processing-e2e-api-a-7406a9-5cd77855fc-n4s94" Nov 03 00:59:11.229500 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:59:11.229473 4939 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-eed369-7d94fb8545-hdldv" Nov 03 00:59:11.229641 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:59:11.229483 4939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-eed369-7d94fb8545-hdldv" event={"ID":"0ae23a7b-9f49-479e-bfd5-7732c28052aa","Type":"ContainerDied","Data":"c69a6cac3df3c53965f661d4ccd09ded7a1de7acabbf48aa18c34a4d8e1446e9"} Nov 03 00:59:11.229641 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:59:11.229604 4939 scope.go:122] "RemoveContainer" containerID="eb5f271d302c42f074bf45c32cdcdfb927cc0ddd6bb8a344638107571d67bed6" Nov 03 00:59:11.237860 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:59:11.237824 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qfcfj\" (UniqueName: \"kubernetes.io/projected/d5b08c7b-b02e-434f-8b32-32a502a92e69-kube-api-access-qfcfj\") pod \"payload-processing-e2e-api-a-7406a9-5cd77855fc-n4s94\" (UID: \"d5b08c7b-b02e-434f-8b32-32a502a92e69\") " pod="openshift-ingress/payload-processing-e2e-api-a-7406a9-5cd77855fc-n4s94" Nov 03 00:59:11.344022 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:59:11.343989 4939 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-eed369-7d94fb8545-hdldv"] Nov 03 00:59:11.428749 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:59:11.428715 4939 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-eed369-7d94fb8545-hdldv"] Nov 03 00:59:11.529528 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:59:11.529487 4939 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-7406a9-5cd77855fc-n4s94" Nov 03 00:59:11.710897 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:59:11.706847 4939 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-7406a9-78cc96967f-6wksw"] Nov 03 00:59:11.821566 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:59:11.821511 4939 patch_prober.go:29] interesting pod/e2e-ipp-a-07f5f6-openshift-default-98746c89f-vz567 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.105:15021/healthz/ready\": dial tcp 10.132.0.105:15021: connect: connection refused" start-of-body= Nov 03 00:59:11.822004 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:59:11.821561 4939 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-07f5f6-openshift-default-98746c89f-vz567" podUID="ef4163bb-8632-4675-a678-11ec6d58e737" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.105:15021/healthz/ready\": dial tcp 10.132.0.105:15021: connect: connection refused" Nov 03 00:59:11.930941 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:59:11.930894 4939 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0ae23a7b-9f49-479e-bfd5-7732c28052aa" path="/var/lib/kubelet/pods/0ae23a7b-9f49-479e-bfd5-7732c28052aa/volumes" Nov 03 00:59:11.932359 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:59:11.932320 4939 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="dd1a7eae-e782-4a82-bdac-c45c75614357" path="/var/lib/kubelet/pods/dd1a7eae-e782-4a82-bdac-c45c75614357/volumes" Nov 03 00:59:11.940549 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:59:11.940527 4939 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-7406a9-5cd77855fc-n4s94"] Nov 03 00:59:12.063500 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:59:12.063443 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 00:59:12.218826 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:59:12.218784 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 00:59:12.240509 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:59:12.240408 4939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-7406a9-5cd77855fc-n4s94" event={"ID":"d5b08c7b-b02e-434f-8b32-32a502a92e69","Type":"ContainerStarted","Data":"a06ce95b4e30f0381b53f1c01cd0c3fdffd07fa2b8974d540122e7d722532242"} Nov 03 00:59:12.240509 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:59:12.240467 4939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-7406a9-5cd77855fc-n4s94" event={"ID":"d5b08c7b-b02e-434f-8b32-32a502a92e69","Type":"ContainerStarted","Data":"0a654fd5deffcacc8ee16dfb863a8c862e1ec7c83b9450034abdf0bfe0ffca77"} Nov 03 00:59:12.240509 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:59:12.240503 4939 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-api-a-7406a9-5cd77855fc-n4s94" Nov 03 00:59:12.242209 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:59:12.242184 4939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-7406a9-78cc96967f-6wksw" event={"ID":"12ba5035-bd74-4162-9c8e-9353527f757f","Type":"ContainerStarted","Data":"2aa61ee297ea7c024943a7060c9d7c0ee4db97995216b2d5909ea5f09be1ca10"} Nov 03 00:59:12.242344 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:59:12.242225 4939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-7406a9-78cc96967f-6wksw" event={"ID":"12ba5035-bd74-4162-9c8e-9353527f757f","Type":"ContainerStarted","Data":"4cf41df5205975e7f3d32f20a353cce8bb348c913236673a10b1a548d402eb1d"} Nov 03 00:59:12.242344 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:59:12.242280 4939 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-api-a-7406a9-78cc96967f-6wksw" Nov 03 00:59:12.284001 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:59:12.264463 4939 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-api-a-7406a9-5cd77855fc-n4s94" podStartSLOduration=2.264444556 podStartE2EDuration="2.264444556s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:59:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:59:12.262874286 +0000 UTC m=+1100.527209234" watchObservedRunningTime="2026-11-03 00:59:12.264444556 +0000 UTC m=+1100.528779502" Nov 03 00:59:12.422616 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:59:12.422577 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 00:59:12.820607 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:59:12.820572 4939 patch_prober.go:29] interesting pod/e2e-ipp-a-07f5f6-openshift-default-98746c89f-vz567 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.105:15021/healthz/ready\": dial tcp 10.132.0.105:15021: connect: connection refused" start-of-body= Nov 03 00:59:12.820777 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:59:12.820637 4939 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-07f5f6-openshift-default-98746c89f-vz567" podUID="ef4163bb-8632-4675-a678-11ec6d58e737" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.105:15021/healthz/ready\": dial tcp 10.132.0.105:15021: connect: connection refused" Nov 03 00:59:12.900340 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:59:12.900269 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 00:59:13.100934 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:59:13.100582 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 00:59:13.506121 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:59:13.506028 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 00:59:13.792987 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:59:13.792792 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 00:59:13.834933 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:59:13.834062 4939 patch_prober.go:29] interesting pod/e2e-ipp-a-07f5f6-openshift-default-98746c89f-vz567 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.105:15021/healthz/ready\": dial tcp 10.132.0.105:15021: connect: connection refused" start-of-body= Nov 03 00:59:13.834933 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:59:13.834118 4939 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-07f5f6-openshift-default-98746c89f-vz567" podUID="ef4163bb-8632-4675-a678-11ec6d58e737" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.105:15021/healthz/ready\": dial tcp 10.132.0.105:15021: connect: connection refused" Nov 03 00:59:14.192259 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:59:14.192205 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 00:59:14.400946 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:59:14.397013 4939 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-a-07f5f6-openshift-default-98746c89f-vz567" Nov 03 00:59:14.805954 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:59:14.804981 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 00:59:14.820796 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:59:14.820759 4939 patch_prober.go:29] interesting pod/e2e-ipp-a-07f5f6-openshift-default-98746c89f-vz567 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.105:15021/healthz/ready\": dial tcp 10.132.0.105:15021: connect: connection refused" start-of-body= Nov 03 00:59:14.821010 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:59:14.820830 4939 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-07f5f6-openshift-default-98746c89f-vz567" podUID="ef4163bb-8632-4675-a678-11ec6d58e737" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.105:15021/healthz/ready\": dial tcp 10.132.0.105:15021: connect: connection refused" Nov 03 00:59:15.110402 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:59:15.109789 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 00:59:15.206635 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:59:15.206577 4939 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-api-a-7406a9-78cc96967f-6wksw" podStartSLOduration=5.206560272 podStartE2EDuration="5.206560272s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:59:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:59:12.306808835 +0000 UTC m=+1100.571143790" watchObservedRunningTime="2026-11-03 00:59:15.206560272 +0000 UTC m=+1103.470895216" Nov 03 00:59:15.213855 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:59:15.210054 4939 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-db5422-68786bc84f-9nclv"] Nov 03 00:59:15.217221 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:59:15.216361 4939 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-db5422-68786bc84f-9nclv" Nov 03 00:59:15.222988 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:59:15.222960 4939 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-db5422-68786bc84f-9nclv"] Nov 03 00:59:15.309930 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:59:15.280193 4939 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-b-db5422\"" Nov 03 00:59:15.309930 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:59:15.297745 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 00:59:15.309930 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:59:15.305229 4939 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-b-db5422-dockercfg-plzgj\"" Nov 03 00:59:15.664934 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:59:15.637351 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zf8vc\" (UniqueName: \"kubernetes.io/projected/c650f474-5078-4466-99b1-4d2ea2c11093-kube-api-access-zf8vc\") pod \"payload-processing-e2e-ipp-b-db5422-68786bc84f-9nclv\" (UID: \"c650f474-5078-4466-99b1-4d2ea2c11093\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-db5422-68786bc84f-9nclv" Nov 03 00:59:15.691944 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:59:15.665377 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c650f474-5078-4466-99b1-4d2ea2c11093-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-db5422-68786bc84f-9nclv\" (UID: \"c650f474-5078-4466-99b1-4d2ea2c11093\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-db5422-68786bc84f-9nclv" Nov 03 00:59:15.779929 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:59:15.776259 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zf8vc\" (UniqueName: \"kubernetes.io/projected/c650f474-5078-4466-99b1-4d2ea2c11093-kube-api-access-zf8vc\") pod \"payload-processing-e2e-ipp-b-db5422-68786bc84f-9nclv\" (UID: \"c650f474-5078-4466-99b1-4d2ea2c11093\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-db5422-68786bc84f-9nclv" Nov 03 00:59:15.779929 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:59:15.776740 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c650f474-5078-4466-99b1-4d2ea2c11093-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-db5422-68786bc84f-9nclv\" (UID: \"c650f474-5078-4466-99b1-4d2ea2c11093\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-db5422-68786bc84f-9nclv" Nov 03 00:59:15.779929 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:59:15.778129 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c650f474-5078-4466-99b1-4d2ea2c11093-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-db5422-68786bc84f-9nclv\" (UID: \"c650f474-5078-4466-99b1-4d2ea2c11093\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-db5422-68786bc84f-9nclv" Nov 03 00:59:15.808935 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:59:15.808635 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zf8vc\" (UniqueName: \"kubernetes.io/projected/c650f474-5078-4466-99b1-4d2ea2c11093-kube-api-access-zf8vc\") pod \"payload-processing-e2e-ipp-b-db5422-68786bc84f-9nclv\" (UID: \"c650f474-5078-4466-99b1-4d2ea2c11093\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-db5422-68786bc84f-9nclv" Nov 03 00:59:15.822722 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:59:15.822243 4939 patch_prober.go:29] interesting pod/e2e-ipp-a-07f5f6-openshift-default-98746c89f-vz567 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.105:15021/healthz/ready\": dial tcp 10.132.0.105:15021: connect: connection refused" start-of-body= Nov 03 00:59:15.822722 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:59:15.822322 4939 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-07f5f6-openshift-default-98746c89f-vz567" podUID="ef4163bb-8632-4675-a678-11ec6d58e737" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.105:15021/healthz/ready\": dial tcp 10.132.0.105:15021: connect: connection refused" Nov 03 00:59:15.841589 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:59:15.840341 4939 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-db5422-68786bc84f-9nclv" Nov 03 00:59:16.042609 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:59:16.042260 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 00:59:16.503375 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:59:16.503334 4939 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-db5422-68786bc84f-9nclv"] Nov 03 00:59:16.505524 ip-10-0-140-107 kubenswrapper[4939]: W1103 00:59:16.505493 4939 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc650f474_5078_4466_99b1_4d2ea2c11093.slice/crio-dcb729e0dbff9115f6d41e101ef0044903cb8130c56b96700f507346f239c94e WatchSource:0}: Error finding container dcb729e0dbff9115f6d41e101ef0044903cb8130c56b96700f507346f239c94e: Status 404 returned error can't find the container with id dcb729e0dbff9115f6d41e101ef0044903cb8130c56b96700f507346f239c94e Nov 03 00:59:16.639185 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:59:16.639147 4939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-db5422-68786bc84f-9nclv" event={"ID":"c650f474-5078-4466-99b1-4d2ea2c11093","Type":"ContainerStarted","Data":"dcb729e0dbff9115f6d41e101ef0044903cb8130c56b96700f507346f239c94e"} Nov 03 00:59:16.825615 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:59:16.825535 4939 patch_prober.go:29] interesting pod/e2e-ipp-a-07f5f6-openshift-default-98746c89f-vz567 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.105:15021/healthz/ready\": dial tcp 10.132.0.105:15021: connect: connection refused" start-of-body= Nov 03 00:59:16.825847 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:59:16.825818 4939 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-07f5f6-openshift-default-98746c89f-vz567" podUID="ef4163bb-8632-4675-a678-11ec6d58e737" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.105:15021/healthz/ready\": dial tcp 10.132.0.105:15021: connect: connection refused" Nov 03 00:59:16.910498 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:59:16.910457 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 00:59:17.443337 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:59:17.443303 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 00:59:17.647889 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:59:17.647798 4939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-db5422-68786bc84f-9nclv" event={"ID":"c650f474-5078-4466-99b1-4d2ea2c11093","Type":"ContainerStarted","Data":"a70f79cf1ffff54da53b7c55ba0553d17a6b758e96b3bfc7467c7b99fb6c957e"} Nov 03 00:59:17.647889 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:59:17.647856 4939 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-db5422-68786bc84f-9nclv" Nov 03 00:59:17.668745 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:59:17.668693 4939 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-b-db5422-68786bc84f-9nclv" podStartSLOduration=2.668677216 podStartE2EDuration="2.668677216s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:59:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:59:17.667211964 +0000 UTC m=+1105.931546909" watchObservedRunningTime="2026-11-03 00:59:17.668677216 +0000 UTC m=+1105.933012163" Nov 03 00:59:17.895975 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:59:17.892796 4939 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-a-07f5f6-openshift-default-98746c89f-vz567" Nov 03 00:59:17.906965 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:59:17.903959 4939 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-a-07f5f6-openshift-default-98746c89f-vz567" Nov 03 00:59:18.051952 ip-10-0-140-107 kubenswrapper[4939]: E1103 00:59:18.050166 4939 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-sthjh" containerName="extractor" Nov 03 00:59:18.293671 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:59:18.293582 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 00:59:19.386197 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:59:19.386158 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 00:59:20.107824 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:59:20.107785 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 00:59:20.314732 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:59:20.314641 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 00:59:20.741163 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:59:20.741115 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 00:59:20.960934 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:59:20.960738 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 00:59:21.304213 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:59:21.304183 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 00:59:21.463931 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:59:21.463448 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 00:59:21.838857 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:59:21.838787 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 00:59:21.853065 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:59:21.853030 4939 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-b-def754-openshift-default-544466b787-9msx9"] Nov 03 00:59:21.869937 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:59:21.869537 4939 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-def754-openshift-default-544466b787-9msx9" Nov 03 00:59:21.886666 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:59:21.885618 4939 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ipp-b-def754-openshift-default-dockercfg-xh7pp\"" Nov 03 00:59:21.886993 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:59:21.886969 4939 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-b-def754-openshift-default-544466b787-9msx9"] Nov 03 00:59:21.904542 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:59:21.904514 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8c763d19-e7ff-4638-8d21-3e977502a520-credential-socket\") pod \"e2e-ipp-b-def754-openshift-default-544466b787-9msx9\" (UID: \"8c763d19-e7ff-4638-8d21-3e977502a520\") " pod="openshift-ingress/e2e-ipp-b-def754-openshift-default-544466b787-9msx9" Nov 03 00:59:21.904710 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:59:21.904557 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8c763d19-e7ff-4638-8d21-3e977502a520-istio-token\") pod \"e2e-ipp-b-def754-openshift-default-544466b787-9msx9\" (UID: \"8c763d19-e7ff-4638-8d21-3e977502a520\") " pod="openshift-ingress/e2e-ipp-b-def754-openshift-default-544466b787-9msx9" Nov 03 00:59:21.904710 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:59:21.904593 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8c763d19-e7ff-4638-8d21-3e977502a520-istio-podinfo\") pod \"e2e-ipp-b-def754-openshift-default-544466b787-9msx9\" (UID: \"8c763d19-e7ff-4638-8d21-3e977502a520\") " pod="openshift-ingress/e2e-ipp-b-def754-openshift-default-544466b787-9msx9" Nov 03 00:59:21.904710 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:59:21.904686 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8c763d19-e7ff-4638-8d21-3e977502a520-istio-data\") pod \"e2e-ipp-b-def754-openshift-default-544466b787-9msx9\" (UID: \"8c763d19-e7ff-4638-8d21-3e977502a520\") " pod="openshift-ingress/e2e-ipp-b-def754-openshift-default-544466b787-9msx9" Nov 03 00:59:21.904973 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:59:21.904819 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8c763d19-e7ff-4638-8d21-3e977502a520-workload-certs\") pod \"e2e-ipp-b-def754-openshift-default-544466b787-9msx9\" (UID: \"8c763d19-e7ff-4638-8d21-3e977502a520\") " pod="openshift-ingress/e2e-ipp-b-def754-openshift-default-544466b787-9msx9" Nov 03 00:59:21.904973 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:59:21.904902 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wqrqg\" (UniqueName: \"kubernetes.io/projected/8c763d19-e7ff-4638-8d21-3e977502a520-kube-api-access-wqrqg\") pod \"e2e-ipp-b-def754-openshift-default-544466b787-9msx9\" (UID: \"8c763d19-e7ff-4638-8d21-3e977502a520\") " pod="openshift-ingress/e2e-ipp-b-def754-openshift-default-544466b787-9msx9" Nov 03 00:59:21.905129 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:59:21.905056 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8c763d19-e7ff-4638-8d21-3e977502a520-workload-socket\") pod \"e2e-ipp-b-def754-openshift-default-544466b787-9msx9\" (UID: \"8c763d19-e7ff-4638-8d21-3e977502a520\") " pod="openshift-ingress/e2e-ipp-b-def754-openshift-default-544466b787-9msx9" Nov 03 00:59:21.905129 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:59:21.905090 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8c763d19-e7ff-4638-8d21-3e977502a520-istiod-ca-cert\") pod \"e2e-ipp-b-def754-openshift-default-544466b787-9msx9\" (UID: \"8c763d19-e7ff-4638-8d21-3e977502a520\") " pod="openshift-ingress/e2e-ipp-b-def754-openshift-default-544466b787-9msx9" Nov 03 00:59:21.905339 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:59:21.905314 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8c763d19-e7ff-4638-8d21-3e977502a520-istio-envoy\") pod \"e2e-ipp-b-def754-openshift-default-544466b787-9msx9\" (UID: \"8c763d19-e7ff-4638-8d21-3e977502a520\") " pod="openshift-ingress/e2e-ipp-b-def754-openshift-default-544466b787-9msx9" Nov 03 00:59:22.051776 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:59:22.051738 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 00:59:22.086987 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:59:22.061624 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8c763d19-e7ff-4638-8d21-3e977502a520-credential-socket\") pod \"e2e-ipp-b-def754-openshift-default-544466b787-9msx9\" (UID: \"8c763d19-e7ff-4638-8d21-3e977502a520\") " pod="openshift-ingress/e2e-ipp-b-def754-openshift-default-544466b787-9msx9" Nov 03 00:59:22.086987 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:59:22.061680 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8c763d19-e7ff-4638-8d21-3e977502a520-istio-token\") pod \"e2e-ipp-b-def754-openshift-default-544466b787-9msx9\" (UID: \"8c763d19-e7ff-4638-8d21-3e977502a520\") " pod="openshift-ingress/e2e-ipp-b-def754-openshift-default-544466b787-9msx9" Nov 03 00:59:22.086987 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:59:22.061721 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8c763d19-e7ff-4638-8d21-3e977502a520-istio-podinfo\") pod \"e2e-ipp-b-def754-openshift-default-544466b787-9msx9\" (UID: \"8c763d19-e7ff-4638-8d21-3e977502a520\") " pod="openshift-ingress/e2e-ipp-b-def754-openshift-default-544466b787-9msx9" Nov 03 00:59:22.086987 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:59:22.061858 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8c763d19-e7ff-4638-8d21-3e977502a520-istio-data\") pod \"e2e-ipp-b-def754-openshift-default-544466b787-9msx9\" (UID: \"8c763d19-e7ff-4638-8d21-3e977502a520\") " pod="openshift-ingress/e2e-ipp-b-def754-openshift-default-544466b787-9msx9" Nov 03 00:59:22.086987 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:59:22.061965 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8c763d19-e7ff-4638-8d21-3e977502a520-workload-certs\") pod \"e2e-ipp-b-def754-openshift-default-544466b787-9msx9\" (UID: \"8c763d19-e7ff-4638-8d21-3e977502a520\") " pod="openshift-ingress/e2e-ipp-b-def754-openshift-default-544466b787-9msx9" Nov 03 00:59:22.086987 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:59:22.062066 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wqrqg\" (UniqueName: \"kubernetes.io/projected/8c763d19-e7ff-4638-8d21-3e977502a520-kube-api-access-wqrqg\") pod \"e2e-ipp-b-def754-openshift-default-544466b787-9msx9\" (UID: \"8c763d19-e7ff-4638-8d21-3e977502a520\") " pod="openshift-ingress/e2e-ipp-b-def754-openshift-default-544466b787-9msx9" Nov 03 00:59:22.086987 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:59:22.062161 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8c763d19-e7ff-4638-8d21-3e977502a520-workload-socket\") pod \"e2e-ipp-b-def754-openshift-default-544466b787-9msx9\" (UID: \"8c763d19-e7ff-4638-8d21-3e977502a520\") " pod="openshift-ingress/e2e-ipp-b-def754-openshift-default-544466b787-9msx9" Nov 03 00:59:22.086987 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:59:22.062204 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8c763d19-e7ff-4638-8d21-3e977502a520-istiod-ca-cert\") pod \"e2e-ipp-b-def754-openshift-default-544466b787-9msx9\" (UID: \"8c763d19-e7ff-4638-8d21-3e977502a520\") " pod="openshift-ingress/e2e-ipp-b-def754-openshift-default-544466b787-9msx9" Nov 03 00:59:22.086987 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:59:22.062485 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8c763d19-e7ff-4638-8d21-3e977502a520-istio-envoy\") pod \"e2e-ipp-b-def754-openshift-default-544466b787-9msx9\" (UID: \"8c763d19-e7ff-4638-8d21-3e977502a520\") " pod="openshift-ingress/e2e-ipp-b-def754-openshift-default-544466b787-9msx9" Nov 03 00:59:22.086987 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:59:22.072457 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8c763d19-e7ff-4638-8d21-3e977502a520-istio-data\") pod \"e2e-ipp-b-def754-openshift-default-544466b787-9msx9\" (UID: \"8c763d19-e7ff-4638-8d21-3e977502a520\") " pod="openshift-ingress/e2e-ipp-b-def754-openshift-default-544466b787-9msx9" Nov 03 00:59:22.086987 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:59:22.072743 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8c763d19-e7ff-4638-8d21-3e977502a520-credential-socket\") pod \"e2e-ipp-b-def754-openshift-default-544466b787-9msx9\" (UID: \"8c763d19-e7ff-4638-8d21-3e977502a520\") " pod="openshift-ingress/e2e-ipp-b-def754-openshift-default-544466b787-9msx9" Nov 03 00:59:22.099471 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:59:22.099393 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8c763d19-e7ff-4638-8d21-3e977502a520-workload-certs\") pod \"e2e-ipp-b-def754-openshift-default-544466b787-9msx9\" (UID: \"8c763d19-e7ff-4638-8d21-3e977502a520\") " pod="openshift-ingress/e2e-ipp-b-def754-openshift-default-544466b787-9msx9" Nov 03 00:59:22.100085 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:59:22.100022 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8c763d19-e7ff-4638-8d21-3e977502a520-workload-socket\") pod \"e2e-ipp-b-def754-openshift-default-544466b787-9msx9\" (UID: \"8c763d19-e7ff-4638-8d21-3e977502a520\") " pod="openshift-ingress/e2e-ipp-b-def754-openshift-default-544466b787-9msx9" Nov 03 00:59:22.100801 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:59:22.100774 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8c763d19-e7ff-4638-8d21-3e977502a520-istiod-ca-cert\") pod \"e2e-ipp-b-def754-openshift-default-544466b787-9msx9\" (UID: \"8c763d19-e7ff-4638-8d21-3e977502a520\") " pod="openshift-ingress/e2e-ipp-b-def754-openshift-default-544466b787-9msx9" Nov 03 00:59:22.130519 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:59:22.130484 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8c763d19-e7ff-4638-8d21-3e977502a520-istio-token\") pod \"e2e-ipp-b-def754-openshift-default-544466b787-9msx9\" (UID: \"8c763d19-e7ff-4638-8d21-3e977502a520\") " pod="openshift-ingress/e2e-ipp-b-def754-openshift-default-544466b787-9msx9" Nov 03 00:59:22.135929 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:59:22.133421 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8c763d19-e7ff-4638-8d21-3e977502a520-istio-podinfo\") pod \"e2e-ipp-b-def754-openshift-default-544466b787-9msx9\" (UID: \"8c763d19-e7ff-4638-8d21-3e977502a520\") " pod="openshift-ingress/e2e-ipp-b-def754-openshift-default-544466b787-9msx9" Nov 03 00:59:22.135929 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:59:22.134619 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8c763d19-e7ff-4638-8d21-3e977502a520-istio-envoy\") pod \"e2e-ipp-b-def754-openshift-default-544466b787-9msx9\" (UID: \"8c763d19-e7ff-4638-8d21-3e977502a520\") " pod="openshift-ingress/e2e-ipp-b-def754-openshift-default-544466b787-9msx9" Nov 03 00:59:22.138937 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:59:22.137294 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wqrqg\" (UniqueName: \"kubernetes.io/projected/8c763d19-e7ff-4638-8d21-3e977502a520-kube-api-access-wqrqg\") pod \"e2e-ipp-b-def754-openshift-default-544466b787-9msx9\" (UID: \"8c763d19-e7ff-4638-8d21-3e977502a520\") " pod="openshift-ingress/e2e-ipp-b-def754-openshift-default-544466b787-9msx9" Nov 03 00:59:22.216532 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:59:22.215235 4939 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-def754-openshift-default-544466b787-9msx9" Nov 03 00:59:22.539975 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:59:22.539936 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 00:59:22.661901 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:59:22.661860 4939 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-b-def754-openshift-default-544466b787-9msx9"] Nov 03 00:59:22.665172 ip-10-0-140-107 kubenswrapper[4939]: W1103 00:59:22.665143 4939 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8c763d19_e7ff_4638_8d21_3e977502a520.slice/crio-d4edfb00f2d587730e7b261aeb16e22647c701c8d1952f13bdd4d368625fc951 WatchSource:0}: Error finding container d4edfb00f2d587730e7b261aeb16e22647c701c8d1952f13bdd4d368625fc951: Status 404 returned error can't find the container with id d4edfb00f2d587730e7b261aeb16e22647c701c8d1952f13bdd4d368625fc951 Nov 03 00:59:22.667998 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:59:22.667962 4939 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 00:59:22.668091 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:59:22.668030 4939 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 00:59:22.668091 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:59:22.668063 4939 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 00:59:22.694741 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:59:22.694716 4939 generic.go:356] "Generic (PLEG): container finished" podID="bbecba1a-367a-4239-9e6c-fdfb4972ccf1" containerID="9213c015a67724a51db25ba1367b7f35a54128dcbc010e1f9c06ccb7aa2a6556" exitCode=0 Nov 03 00:59:22.694858 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:59:22.694788 4939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-80f70e-openshift-default-5984948b65-ck87x" event={"ID":"bbecba1a-367a-4239-9e6c-fdfb4972ccf1","Type":"ContainerDied","Data":"9213c015a67724a51db25ba1367b7f35a54128dcbc010e1f9c06ccb7aa2a6556"} Nov 03 00:59:22.695343 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:59:22.695324 4939 scope.go:122] "RemoveContainer" containerID="9213c015a67724a51db25ba1367b7f35a54128dcbc010e1f9c06ccb7aa2a6556" Nov 03 00:59:22.696221 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:59:22.696172 4939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-def754-openshift-default-544466b787-9msx9" event={"ID":"8c763d19-e7ff-4638-8d21-3e977502a520","Type":"ContainerStarted","Data":"d4edfb00f2d587730e7b261aeb16e22647c701c8d1952f13bdd4d368625fc951"} Nov 03 00:59:22.697025 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:59:22.696995 4939 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 00:59:22.697130 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:59:22.697063 4939 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 00:59:22.697130 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:59:22.697102 4939 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 00:59:23.100504 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:59:23.100470 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 00:59:23.288653 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:59:23.288620 4939 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-api-a-7406a9-5cd77855fc-n4s94" Nov 03 00:59:23.292085 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:59:23.291065 4939 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-api-a-7406a9-78cc96967f-6wksw" Nov 03 00:59:23.507610 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:59:23.507533 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 00:59:23.708942 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:59:23.706203 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 00:59:23.741934 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:59:23.736012 4939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-80f70e-openshift-default-5984948b65-ck87x" event={"ID":"bbecba1a-367a-4239-9e6c-fdfb4972ccf1","Type":"ContainerStarted","Data":"05464c4081c9d2df9c356d204b4e11297f66ffca65156416e56a9032be54e1d6"} Nov 03 00:59:23.758980 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:59:23.754424 4939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-def754-openshift-default-544466b787-9msx9" event={"ID":"8c763d19-e7ff-4638-8d21-3e977502a520","Type":"ContainerStarted","Data":"32da52f1f2bddcb5fdad799643b3a1bfafca75e8e99547378cb528abcbd073b7"} Nov 03 00:59:23.865934 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:59:23.858302 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 00:59:23.892928 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:59:23.876619 4939 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-b-def754-openshift-default-544466b787-9msx9" podStartSLOduration=2.876598222 podStartE2EDuration="2.876598222s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:59:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:59:23.854960799 +0000 UTC m=+1112.119295749" watchObservedRunningTime="2026-11-03 00:59:23.876598222 +0000 UTC m=+1112.140933168" Nov 03 00:59:24.230933 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:59:24.226506 4939 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-b-def754-openshift-default-544466b787-9msx9" Nov 03 00:59:24.249934 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:59:24.243330 4939 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-b-def754-openshift-default-544466b787-9msx9" Nov 03 00:59:24.393354 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:59:24.393317 4939 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-80f70e-openshift-default-5984948b65-ck87x" Nov 03 00:59:24.394809 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:59:24.394781 4939 patch_prober.go:29] interesting pod/e2e-api-a-80f70e-openshift-default-5984948b65-ck87x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.109:15021/healthz/ready\": dial tcp 10.132.0.109:15021: connect: connection refused" start-of-body= Nov 03 00:59:24.394942 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:59:24.394842 4939 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-80f70e-openshift-default-5984948b65-ck87x" podUID="bbecba1a-367a-4239-9e6c-fdfb4972ccf1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.109:15021/healthz/ready\": dial tcp 10.132.0.109:15021: connect: connection refused" Nov 03 00:59:24.462931 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:59:24.459258 4939 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-80f70e-openshift-default-5984948b65-ck87x" Nov 03 00:59:24.781534 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:59:24.781500 4939 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-b-def754-openshift-default-544466b787-9msx9" Nov 03 00:59:24.783770 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:59:24.783745 4939 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-b-def754-openshift-default-544466b787-9msx9" Nov 03 00:59:25.221076 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:59:25.212293 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 00:59:25.339406 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:59:25.339376 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 00:59:25.393876 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:59:25.393844 4939 patch_prober.go:29] interesting pod/e2e-api-a-80f70e-openshift-default-5984948b65-ck87x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.109:15021/healthz/ready\": dial tcp 10.132.0.109:15021: connect: connection refused" start-of-body= Nov 03 00:59:25.394071 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:59:25.393894 4939 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-80f70e-openshift-default-5984948b65-ck87x" podUID="bbecba1a-367a-4239-9e6c-fdfb4972ccf1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.109:15021/healthz/ready\": dial tcp 10.132.0.109:15021: connect: connection refused" Nov 03 00:59:26.395447 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:59:26.395410 4939 patch_prober.go:29] interesting pod/e2e-api-a-80f70e-openshift-default-5984948b65-ck87x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.109:15021/healthz/ready\": dial tcp 10.132.0.109:15021: connect: connection refused" start-of-body= Nov 03 00:59:26.396110 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:59:26.396077 4939 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-80f70e-openshift-default-5984948b65-ck87x" podUID="bbecba1a-367a-4239-9e6c-fdfb4972ccf1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.109:15021/healthz/ready\": dial tcp 10.132.0.109:15021: connect: connection refused" Nov 03 00:59:26.815437 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:59:26.815321 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 00:59:26.982832 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:59:26.980438 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 00:59:27.398413 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:59:27.398376 4939 patch_prober.go:29] interesting pod/e2e-api-a-80f70e-openshift-default-5984948b65-ck87x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.109:15021/healthz/ready\": dial tcp 10.132.0.109:15021: connect: connection refused" start-of-body= Nov 03 00:59:27.399263 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:59:27.399210 4939 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-80f70e-openshift-default-5984948b65-ck87x" podUID="bbecba1a-367a-4239-9e6c-fdfb4972ccf1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.109:15021/healthz/ready\": dial tcp 10.132.0.109:15021: connect: connection refused" Nov 03 00:59:27.489119 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:59:27.487923 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 00:59:27.673980 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:59:27.673479 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 00:59:28.269388 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:59:28.269356 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 00:59:28.381356 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:59:28.381323 4939 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-def754-969c7dd8c-rb4dt"] Nov 03 00:59:28.387094 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:59:28.387067 4939 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-def754-969c7dd8c-rb4dt" Nov 03 00:59:28.395063 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:59:28.393797 4939 patch_prober.go:29] interesting pod/e2e-api-a-80f70e-openshift-default-5984948b65-ck87x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.109:15021/healthz/ready\": dial tcp 10.132.0.109:15021: connect: connection refused" start-of-body= Nov 03 00:59:28.395063 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:59:28.393869 4939 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-80f70e-openshift-default-5984948b65-ck87x" podUID="bbecba1a-367a-4239-9e6c-fdfb4972ccf1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.109:15021/healthz/ready\": dial tcp 10.132.0.109:15021: connect: connection refused" Nov 03 00:59:28.407584 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:59:28.407554 4939 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-def754-969c7dd8c-rb4dt"] Nov 03 00:59:28.435199 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:59:28.435167 4939 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-b-def754-dockercfg-c66w7\"" Nov 03 00:59:28.442679 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:59:28.442655 4939 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-b-def754\"" Nov 03 00:59:28.493204 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:59:28.493172 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6cc03f20-f3ea-491f-b0be-e3bc1b9c3ac1-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-def754-969c7dd8c-rb4dt\" (UID: \"6cc03f20-f3ea-491f-b0be-e3bc1b9c3ac1\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-def754-969c7dd8c-rb4dt" Nov 03 00:59:28.493409 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:59:28.493396 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8z4fs\" (UniqueName: \"kubernetes.io/projected/6cc03f20-f3ea-491f-b0be-e3bc1b9c3ac1-kube-api-access-8z4fs\") pod \"payload-pre-processing-e2e-ipp-b-def754-969c7dd8c-rb4dt\" (UID: \"6cc03f20-f3ea-491f-b0be-e3bc1b9c3ac1\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-def754-969c7dd8c-rb4dt" Nov 03 00:59:28.515461 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:59:28.515430 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 00:59:28.597160 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:59:28.597119 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6cc03f20-f3ea-491f-b0be-e3bc1b9c3ac1-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-def754-969c7dd8c-rb4dt\" (UID: \"6cc03f20-f3ea-491f-b0be-e3bc1b9c3ac1\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-def754-969c7dd8c-rb4dt" Nov 03 00:59:28.597494 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:59:28.597471 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8z4fs\" (UniqueName: \"kubernetes.io/projected/6cc03f20-f3ea-491f-b0be-e3bc1b9c3ac1-kube-api-access-8z4fs\") pod \"payload-pre-processing-e2e-ipp-b-def754-969c7dd8c-rb4dt\" (UID: \"6cc03f20-f3ea-491f-b0be-e3bc1b9c3ac1\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-def754-969c7dd8c-rb4dt" Nov 03 00:59:28.616931 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:59:28.613416 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6cc03f20-f3ea-491f-b0be-e3bc1b9c3ac1-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-def754-969c7dd8c-rb4dt\" (UID: \"6cc03f20-f3ea-491f-b0be-e3bc1b9c3ac1\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-def754-969c7dd8c-rb4dt" Nov 03 00:59:28.620586 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:59:28.620557 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8z4fs\" (UniqueName: \"kubernetes.io/projected/6cc03f20-f3ea-491f-b0be-e3bc1b9c3ac1-kube-api-access-8z4fs\") pod \"payload-pre-processing-e2e-ipp-b-def754-969c7dd8c-rb4dt\" (UID: \"6cc03f20-f3ea-491f-b0be-e3bc1b9c3ac1\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-def754-969c7dd8c-rb4dt" Nov 03 00:59:28.663440 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:59:28.663406 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 00:59:28.706152 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:59:28.706119 4939 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-db5422-68786bc84f-9nclv" Nov 03 00:59:28.718183 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:59:28.718152 4939 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-def754-969c7dd8c-rb4dt" Nov 03 00:59:29.418930 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:59:29.394760 4939 patch_prober.go:29] interesting pod/e2e-api-a-80f70e-openshift-default-5984948b65-ck87x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.109:15021/healthz/ready\": dial tcp 10.132.0.109:15021: connect: connection refused" start-of-body= Nov 03 00:59:29.418930 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:59:29.394850 4939 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-80f70e-openshift-default-5984948b65-ck87x" podUID="bbecba1a-367a-4239-9e6c-fdfb4972ccf1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.109:15021/healthz/ready\": dial tcp 10.132.0.109:15021: connect: connection refused" Nov 03 00:59:29.631821 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:59:29.631783 4939 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-def754-969c7dd8c-rb4dt"] Nov 03 00:59:29.823929 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:59:29.815632 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 00:59:29.999823 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:59:29.999486 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 00:59:30.010336 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:59:30.010305 4939 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-f88f87-5c6d75c98c-8cjnk"] Nov 03 00:59:30.017361 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:59:30.015691 4939 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-f88f87-5c6d75c98c-8cjnk" Nov 03 00:59:30.017361 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:59:30.016584 4939 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-f88f87-5c6d75c98c-8cjnk"] Nov 03 00:59:30.021248 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:59:30.021225 4939 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-a-f88f87\"" Nov 03 00:59:30.028461 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:59:30.026965 4939 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-a-f88f87-dockercfg-clk5s\"" Nov 03 00:59:30.045405 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:59:30.045369 4939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-def754-969c7dd8c-rb4dt" event={"ID":"6cc03f20-f3ea-491f-b0be-e3bc1b9c3ac1","Type":"ContainerStarted","Data":"15ce4fbbe00c91a18e33fb27084951ae04c083a734dd3ccb3948ae9516283304"} Nov 03 00:59:30.045560 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:59:30.045434 4939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-def754-969c7dd8c-rb4dt" event={"ID":"6cc03f20-f3ea-491f-b0be-e3bc1b9c3ac1","Type":"ContainerStarted","Data":"b371e3f9a98ad068b4bb55663668ba3d0addbc763e6b564e94120904a9b3c3d5"} Nov 03 00:59:30.045771 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:59:30.045660 4939 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-def754-969c7dd8c-rb4dt" Nov 03 00:59:30.147068 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:59:30.130450 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 00:59:30.152800 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:59:30.152340 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0a9269dd-5b6c-4311-8f78-87028a4a7935-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-f88f87-5c6d75c98c-8cjnk\" (UID: \"0a9269dd-5b6c-4311-8f78-87028a4a7935\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-f88f87-5c6d75c98c-8cjnk" Nov 03 00:59:30.174948 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:59:30.156849 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7ws6c\" (UniqueName: \"kubernetes.io/projected/0a9269dd-5b6c-4311-8f78-87028a4a7935-kube-api-access-7ws6c\") pod \"payload-processing-e2e-ipp-a-f88f87-5c6d75c98c-8cjnk\" (UID: \"0a9269dd-5b6c-4311-8f78-87028a4a7935\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-f88f87-5c6d75c98c-8cjnk" Nov 03 00:59:30.194974 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:59:30.182162 4939 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-def754-969c7dd8c-rb4dt" podStartSLOduration=2.182140584 podStartE2EDuration="2.182140584s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:59:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:59:30.101206445 +0000 UTC m=+1118.365541390" watchObservedRunningTime="2026-11-03 00:59:30.182140584 +0000 UTC m=+1118.446475537" Nov 03 00:59:30.306090 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:59:30.275079 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0a9269dd-5b6c-4311-8f78-87028a4a7935-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-f88f87-5c6d75c98c-8cjnk\" (UID: \"0a9269dd-5b6c-4311-8f78-87028a4a7935\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-f88f87-5c6d75c98c-8cjnk" Nov 03 00:59:30.306090 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:59:30.302671 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7ws6c\" (UniqueName: \"kubernetes.io/projected/0a9269dd-5b6c-4311-8f78-87028a4a7935-kube-api-access-7ws6c\") pod \"payload-processing-e2e-ipp-a-f88f87-5c6d75c98c-8cjnk\" (UID: \"0a9269dd-5b6c-4311-8f78-87028a4a7935\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-f88f87-5c6d75c98c-8cjnk" Nov 03 00:59:30.306090 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:59:30.304101 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0a9269dd-5b6c-4311-8f78-87028a4a7935-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-f88f87-5c6d75c98c-8cjnk\" (UID: \"0a9269dd-5b6c-4311-8f78-87028a4a7935\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-f88f87-5c6d75c98c-8cjnk" Nov 03 00:59:30.377927 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:59:30.377560 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7ws6c\" (UniqueName: \"kubernetes.io/projected/0a9269dd-5b6c-4311-8f78-87028a4a7935-kube-api-access-7ws6c\") pod \"payload-processing-e2e-ipp-a-f88f87-5c6d75c98c-8cjnk\" (UID: \"0a9269dd-5b6c-4311-8f78-87028a4a7935\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-f88f87-5c6d75c98c-8cjnk" Nov 03 00:59:30.438932 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:59:30.429041 4939 patch_prober.go:29] interesting pod/e2e-api-a-80f70e-openshift-default-5984948b65-ck87x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.109:15021/healthz/ready\": dial tcp 10.132.0.109:15021: connect: connection refused" start-of-body= Nov 03 00:59:30.438932 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:59:30.429109 4939 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-80f70e-openshift-default-5984948b65-ck87x" podUID="bbecba1a-367a-4239-9e6c-fdfb4972ccf1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.109:15021/healthz/ready\": dial tcp 10.132.0.109:15021: connect: connection refused" Nov 03 00:59:30.642641 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:59:30.641360 4939 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-f88f87-5c6d75c98c-8cjnk" Nov 03 00:59:30.974241 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:59:30.974155 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 00:59:31.095930 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:59:31.087812 4939 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-f88f87-5c6d75c98c-8cjnk"] Nov 03 00:59:31.117864 ip-10-0-140-107 kubenswrapper[4939]: W1103 00:59:31.115546 4939 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0a9269dd_5b6c_4311_8f78_87028a4a7935.slice/crio-6c18058941a15f0b36138239c12c60e865606e83958fa3f8757529457eeb609b WatchSource:0}: Error finding container 6c18058941a15f0b36138239c12c60e865606e83958fa3f8757529457eeb609b: Status 404 returned error can't find the container with id 6c18058941a15f0b36138239c12c60e865606e83958fa3f8757529457eeb609b Nov 03 00:59:31.398182 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:59:31.398153 4939 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-80f70e-openshift-default-5984948b65-ck87x" Nov 03 00:59:31.399321 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:59:31.399302 4939 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-80f70e-openshift-default-5984948b65-ck87x" Nov 03 00:59:31.586252 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:59:31.586220 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 00:59:32.101203 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:59:32.101166 4939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-f88f87-5c6d75c98c-8cjnk" event={"ID":"0a9269dd-5b6c-4311-8f78-87028a4a7935","Type":"ContainerStarted","Data":"4aa28d207392c7d52c00f18ec084eba275ca07282ae6f52dea48f35358eaa71a"} Nov 03 00:59:32.101381 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:59:32.101257 4939 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-f88f87-5c6d75c98c-8cjnk" Nov 03 00:59:32.101381 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:59:32.101280 4939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-f88f87-5c6d75c98c-8cjnk" event={"ID":"0a9269dd-5b6c-4311-8f78-87028a4a7935","Type":"ContainerStarted","Data":"6c18058941a15f0b36138239c12c60e865606e83958fa3f8757529457eeb609b"} Nov 03 00:59:32.140611 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:59:32.140561 4939 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-a-f88f87-5c6d75c98c-8cjnk" podStartSLOduration=3.14054743 podStartE2EDuration="3.14054743s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:59:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:59:32.137655718 +0000 UTC m=+1120.401990664" watchObservedRunningTime="2026-11-03 00:59:32.14054743 +0000 UTC m=+1120.404882374" Nov 03 00:59:32.591444 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:59:32.591410 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 00:59:33.330556 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:59:33.330522 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 00:59:34.105563 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:59:34.105527 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 00:59:34.803098 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:59:34.803063 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 00:59:35.616621 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:59:35.616591 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 00:59:35.977020 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:59:35.976936 4939 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-4pw7v"] Nov 03 00:59:37.001930 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:59:37.000998 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 00:59:38.290483 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:59:38.290452 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 00:59:39.129934 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:59:39.129383 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 00:59:39.468991 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:59:39.446145 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 00:59:39.581932 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:59:39.580547 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 00:59:39.944462 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:59:39.944423 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 00:59:40.068933 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:59:40.061011 4939 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-05e6bd51-ht7qq"] Nov 03 00:59:40.068933 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:59:40.063853 4939 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-05e6bd51-ht7qq"] Nov 03 00:59:40.200015 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:59:40.199892 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 00:59:40.328790 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:59:40.328750 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 00:59:40.896711 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:59:40.896672 4939 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-f88f87-7b9d4c6697-mthjz"] Nov 03 00:59:40.902053 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:59:40.902029 4939 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-f88f87-7b9d4c6697-mthjz"] Nov 03 00:59:40.902195 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:59:40.902128 4939 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-f88f87-7b9d4c6697-mthjz" Nov 03 00:59:40.909189 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:59:40.909163 4939 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ipp-a-f88f87-serving-cert\"" Nov 03 00:59:40.943063 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:59:40.942992 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 00:59:41.166389 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:59:41.166294 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 00:59:41.167850 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:59:41.167818 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/81813dd5-11a4-45b8-990c-159f5f4359be-maas-api-tls\") pod \"maas-api-e2e-ipp-a-f88f87-7b9d4c6697-mthjz\" (UID: \"81813dd5-11a4-45b8-990c-159f5f4359be\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-f88f87-7b9d4c6697-mthjz" Nov 03 00:59:41.168025 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:59:41.168001 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-794t9\" (UniqueName: \"kubernetes.io/projected/81813dd5-11a4-45b8-990c-159f5f4359be-kube-api-access-794t9\") pod \"maas-api-e2e-ipp-a-f88f87-7b9d4c6697-mthjz\" (UID: \"81813dd5-11a4-45b8-990c-159f5f4359be\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-f88f87-7b9d4c6697-mthjz" Nov 03 00:59:41.168206 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:59:41.168183 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/81813dd5-11a4-45b8-990c-159f5f4359be-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-f88f87-7b9d4c6697-mthjz\" (UID: \"81813dd5-11a4-45b8-990c-159f5f4359be\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-f88f87-7b9d4c6697-mthjz" Nov 03 00:59:41.168320 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:59:41.168302 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/81813dd5-11a4-45b8-990c-159f5f4359be-metrics-certs\") pod \"maas-api-e2e-ipp-a-f88f87-7b9d4c6697-mthjz\" (UID: \"81813dd5-11a4-45b8-990c-159f5f4359be\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-f88f87-7b9d4c6697-mthjz" Nov 03 00:59:41.200305 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:59:41.200279 4939 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-def754-969c7dd8c-rb4dt" Nov 03 00:59:41.273933 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:59:41.273345 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-794t9\" (UniqueName: \"kubernetes.io/projected/81813dd5-11a4-45b8-990c-159f5f4359be-kube-api-access-794t9\") pod \"maas-api-e2e-ipp-a-f88f87-7b9d4c6697-mthjz\" (UID: \"81813dd5-11a4-45b8-990c-159f5f4359be\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-f88f87-7b9d4c6697-mthjz" Nov 03 00:59:41.273933 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:59:41.273478 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/81813dd5-11a4-45b8-990c-159f5f4359be-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-f88f87-7b9d4c6697-mthjz\" (UID: \"81813dd5-11a4-45b8-990c-159f5f4359be\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-f88f87-7b9d4c6697-mthjz" Nov 03 00:59:41.273933 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:59:41.273599 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/81813dd5-11a4-45b8-990c-159f5f4359be-metrics-certs\") pod \"maas-api-e2e-ipp-a-f88f87-7b9d4c6697-mthjz\" (UID: \"81813dd5-11a4-45b8-990c-159f5f4359be\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-f88f87-7b9d4c6697-mthjz" Nov 03 00:59:41.273933 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:59:41.273718 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/81813dd5-11a4-45b8-990c-159f5f4359be-maas-api-tls\") pod \"maas-api-e2e-ipp-a-f88f87-7b9d4c6697-mthjz\" (UID: \"81813dd5-11a4-45b8-990c-159f5f4359be\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-f88f87-7b9d4c6697-mthjz" Nov 03 00:59:41.276928 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:59:41.275070 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/81813dd5-11a4-45b8-990c-159f5f4359be-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-f88f87-7b9d4c6697-mthjz\" (UID: \"81813dd5-11a4-45b8-990c-159f5f4359be\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-f88f87-7b9d4c6697-mthjz" Nov 03 00:59:41.278699 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:59:41.278677 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/81813dd5-11a4-45b8-990c-159f5f4359be-maas-api-tls\") pod \"maas-api-e2e-ipp-a-f88f87-7b9d4c6697-mthjz\" (UID: \"81813dd5-11a4-45b8-990c-159f5f4359be\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-f88f87-7b9d4c6697-mthjz" Nov 03 00:59:41.279154 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:59:41.279134 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/81813dd5-11a4-45b8-990c-159f5f4359be-metrics-certs\") pod \"maas-api-e2e-ipp-a-f88f87-7b9d4c6697-mthjz\" (UID: \"81813dd5-11a4-45b8-990c-159f5f4359be\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-f88f87-7b9d4c6697-mthjz" Nov 03 00:59:41.285954 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:59:41.285329 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-794t9\" (UniqueName: \"kubernetes.io/projected/81813dd5-11a4-45b8-990c-159f5f4359be-kube-api-access-794t9\") pod \"maas-api-e2e-ipp-a-f88f87-7b9d4c6697-mthjz\" (UID: \"81813dd5-11a4-45b8-990c-159f5f4359be\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-f88f87-7b9d4c6697-mthjz" Nov 03 00:59:41.294093 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:59:41.294061 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 00:59:41.525385 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:59:41.525304 4939 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-f88f87-7b9d4c6697-mthjz" Nov 03 00:59:41.777982 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:59:41.777781 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 00:59:42.439622 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:59:42.439583 4939 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3be51f99-989b-4efd-a563-e2a985522a3e" path="/var/lib/kubelet/pods/3be51f99-989b-4efd-a563-e2a985522a3e/volumes" Nov 03 00:59:42.440281 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:59:42.440226 4939 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-f88f87-7b9d4c6697-mthjz"] Nov 03 00:59:42.441575 ip-10-0-140-107 kubenswrapper[4939]: W1103 00:59:42.441522 4939 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod81813dd5_11a4_45b8_990c_159f5f4359be.slice/crio-bac7dd94540690381d62b7793b2697b129e6fb790d28266160d18e4420a1bee4 WatchSource:0}: Error finding container bac7dd94540690381d62b7793b2697b129e6fb790d28266160d18e4420a1bee4: Status 404 returned error can't find the container with id bac7dd94540690381d62b7793b2697b129e6fb790d28266160d18e4420a1bee4 Nov 03 00:59:42.518129 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:59:42.518086 4939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-f88f87-7b9d4c6697-mthjz" event={"ID":"81813dd5-11a4-45b8-990c-159f5f4359be","Type":"ContainerStarted","Data":"bac7dd94540690381d62b7793b2697b129e6fb790d28266160d18e4420a1bee4"} Nov 03 00:59:42.645587 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:59:42.645512 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 00:59:42.781989 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:59:42.780085 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 00:59:42.891194 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:59:42.891164 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 00:59:43.113403 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:59:43.113376 4939 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-f88f87-5c6d75c98c-8cjnk" Nov 03 00:59:43.526708 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:59:43.526675 4939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-f88f87-7b9d4c6697-mthjz" event={"ID":"81813dd5-11a4-45b8-990c-159f5f4359be","Type":"ContainerStarted","Data":"8ee3652007076d525b1302aeff97fb35d02540ff3c5a04660d497d7a611e1d59"} Nov 03 00:59:43.527207 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:59:43.527191 4939 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-f88f87-7b9d4c6697-mthjz" Nov 03 00:59:43.568313 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:59:43.568247 4939 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-f88f87-7b9d4c6697-mthjz" podStartSLOduration=3.089432581 podStartE2EDuration="3.568226324s" totalImagesPullingTime="478.793743ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:59:40 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:59:43.563825654 +0000 UTC m=+1131.828160601" watchObservedRunningTime="2026-11-03 00:59:43.568226324 +0000 UTC m=+1131.832561274" Nov 03 00:59:43.715974 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:59:43.715938 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 00:59:44.381203 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:59:44.381169 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 00:59:44.518932 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:59:44.509007 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 00:59:44.793806 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:59:44.793721 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 00:59:45.633439 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:59:45.633396 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 00:59:45.701594 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:59:45.695598 4939 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-ffdea6-6975889bf-cb625"] Nov 03 00:59:45.701594 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:59:45.701523 4939 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-ffdea6-6975889bf-cb625" Nov 03 00:59:45.712377 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:59:45.712344 4939 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-ffdea6-6975889bf-cb625"] Nov 03 00:59:45.790966 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:59:45.786282 4939 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-b-ffdea6-dockercfg-dpfcx\"" Nov 03 00:59:45.790966 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:59:45.786373 4939 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-b-ffdea6\"" Nov 03 00:59:45.815413 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:59:45.808724 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 00:59:45.934996 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:59:45.853754 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g5996\" (UniqueName: \"kubernetes.io/projected/7e00f3bb-5398-415b-bcb3-9db058e00a05-kube-api-access-g5996\") pod \"payload-processing-e2e-api-b-ffdea6-6975889bf-cb625\" (UID: \"7e00f3bb-5398-415b-bcb3-9db058e00a05\") " pod="openshift-ingress/payload-processing-e2e-api-b-ffdea6-6975889bf-cb625" Nov 03 00:59:45.934996 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:59:45.854103 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7e00f3bb-5398-415b-bcb3-9db058e00a05-plugins-config-volume\") pod \"payload-processing-e2e-api-b-ffdea6-6975889bf-cb625\" (UID: \"7e00f3bb-5398-415b-bcb3-9db058e00a05\") " pod="openshift-ingress/payload-processing-e2e-api-b-ffdea6-6975889bf-cb625" Nov 03 00:59:45.943005 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:59:45.941611 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 00:59:45.970418 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:59:45.955575 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7e00f3bb-5398-415b-bcb3-9db058e00a05-plugins-config-volume\") pod \"payload-processing-e2e-api-b-ffdea6-6975889bf-cb625\" (UID: \"7e00f3bb-5398-415b-bcb3-9db058e00a05\") " pod="openshift-ingress/payload-processing-e2e-api-b-ffdea6-6975889bf-cb625" Nov 03 00:59:45.970418 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:59:45.955765 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g5996\" (UniqueName: \"kubernetes.io/projected/7e00f3bb-5398-415b-bcb3-9db058e00a05-kube-api-access-g5996\") pod \"payload-processing-e2e-api-b-ffdea6-6975889bf-cb625\" (UID: \"7e00f3bb-5398-415b-bcb3-9db058e00a05\") " pod="openshift-ingress/payload-processing-e2e-api-b-ffdea6-6975889bf-cb625" Nov 03 00:59:45.970418 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:59:45.968001 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7e00f3bb-5398-415b-bcb3-9db058e00a05-plugins-config-volume\") pod \"payload-processing-e2e-api-b-ffdea6-6975889bf-cb625\" (UID: \"7e00f3bb-5398-415b-bcb3-9db058e00a05\") " pod="openshift-ingress/payload-processing-e2e-api-b-ffdea6-6975889bf-cb625" Nov 03 00:59:45.998691 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:59:45.998645 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-g5996\" (UniqueName: \"kubernetes.io/projected/7e00f3bb-5398-415b-bcb3-9db058e00a05-kube-api-access-g5996\") pod \"payload-processing-e2e-api-b-ffdea6-6975889bf-cb625\" (UID: \"7e00f3bb-5398-415b-bcb3-9db058e00a05\") " pod="openshift-ingress/payload-processing-e2e-api-b-ffdea6-6975889bf-cb625" Nov 03 00:59:46.048619 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:59:46.048218 4939 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-ffdea6-6975889bf-cb625" Nov 03 00:59:46.516866 ip-10-0-140-107 kubenswrapper[4939]: W1103 00:59:46.516824 4939 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7e00f3bb_5398_415b_bcb3_9db058e00a05.slice/crio-88f9fdc188279cc05a660047995b9c103c7a84e68c8f6402f8ee136da990a06c WatchSource:0}: Error finding container 88f9fdc188279cc05a660047995b9c103c7a84e68c8f6402f8ee136da990a06c: Status 404 returned error can't find the container with id 88f9fdc188279cc05a660047995b9c103c7a84e68c8f6402f8ee136da990a06c Nov 03 00:59:46.517298 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:59:46.517274 4939 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-ffdea6-6975889bf-cb625"] Nov 03 00:59:46.593735 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:59:46.593697 4939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-ffdea6-6975889bf-cb625" event={"ID":"7e00f3bb-5398-415b-bcb3-9db058e00a05","Type":"ContainerStarted","Data":"88f9fdc188279cc05a660047995b9c103c7a84e68c8f6402f8ee136da990a06c"} Nov 03 00:59:46.747297 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:59:46.747261 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 00:59:46.870819 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:59:46.870778 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 00:59:47.154026 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:59:47.141472 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 00:59:47.605411 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:59:47.605379 4939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-ffdea6-6975889bf-cb625" event={"ID":"7e00f3bb-5398-415b-bcb3-9db058e00a05","Type":"ContainerStarted","Data":"ea3c68a83dde08f7b23120f46e7905ede68f9bbe36b69c47e2d2a03639dc3560"} Nov 03 00:59:47.605580 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:59:47.605518 4939 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-api-b-ffdea6-6975889bf-cb625" Nov 03 00:59:47.626094 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:59:47.626037 4939 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-api-b-ffdea6-6975889bf-cb625" podStartSLOduration=2.62601955 podStartE2EDuration="2.62601955s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:59:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:59:47.623685823 +0000 UTC m=+1135.888020768" watchObservedRunningTime="2026-11-03 00:59:47.62601955 +0000 UTC m=+1135.890354496" Nov 03 00:59:47.946392 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:59:47.946314 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 00:59:48.035026 ip-10-0-140-107 kubenswrapper[4939]: E1103 00:59:48.033814 4939 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-sthjh" containerName="extractor" Nov 03 00:59:48.166479 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:59:48.166444 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 00:59:48.285993 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:59:48.285886 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 00:59:48.298410 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:59:48.298380 4939 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-db5422-68786bc84f-9nclv"] Nov 03 00:59:48.411642 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:59:48.408559 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 00:59:48.592520 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:59:48.588578 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 00:59:48.830930 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:59:48.828156 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 00:59:49.385387 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:59:49.385349 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 00:59:49.575800 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:59:49.575715 4939 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-f88f87-7b9d4c6697-mthjz" Nov 03 00:59:49.640380 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:59:49.640344 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 00:59:49.721301 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:59:49.721262 4939 generic.go:356] "Generic (PLEG): container finished" podID="49df4a8a-81d6-4c9e-baf7-40c0f7cbaae1" containerID="2a7c98c6f2826c0cc792ea5c299ab7bf08c0753c213be3dced1475e3c2330a95" exitCode=0 Nov 03 00:59:49.721589 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:59:49.721561 4939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-47369d-openshift-default-696ccdd997-9mqtd" event={"ID":"49df4a8a-81d6-4c9e-baf7-40c0f7cbaae1","Type":"ContainerDied","Data":"2a7c98c6f2826c0cc792ea5c299ab7bf08c0753c213be3dced1475e3c2330a95"} Nov 03 00:59:49.722366 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:59:49.722341 4939 scope.go:122] "RemoveContainer" containerID="2a7c98c6f2826c0cc792ea5c299ab7bf08c0753c213be3dced1475e3c2330a95" Nov 03 00:59:49.725376 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:59:49.725345 4939 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 00:59:49.725476 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:59:49.725416 4939 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 00:59:49.725476 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:59:49.725452 4939 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 00:59:50.196742 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:59:50.196708 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 00:59:50.730733 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:59:50.730696 4939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-47369d-openshift-default-696ccdd997-9mqtd" event={"ID":"49df4a8a-81d6-4c9e-baf7-40c0f7cbaae1","Type":"ContainerStarted","Data":"4f759ea70471a8a4d57a74305e9917ce0eedab9b94f41568f1f13462bf9fba8d"} Nov 03 00:59:50.910038 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:59:50.910006 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 00:59:51.622103 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:59:51.622070 4939 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-47369d-openshift-default-696ccdd997-9mqtd" Nov 03 00:59:51.622996 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:59:51.622971 4939 patch_prober.go:29] interesting pod/e2e-api-b-47369d-openshift-default-696ccdd997-9mqtd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.113:15021/healthz/ready\": dial tcp 10.132.0.113:15021: connect: connection refused" start-of-body= Nov 03 00:59:51.623109 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:59:51.623022 4939 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-47369d-openshift-default-696ccdd997-9mqtd" podUID="49df4a8a-81d6-4c9e-baf7-40c0f7cbaae1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.113:15021/healthz/ready\": dial tcp 10.132.0.113:15021: connect: connection refused" Nov 03 00:59:51.823159 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:59:51.823123 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 00:59:52.622209 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:59:52.622180 4939 patch_prober.go:29] interesting pod/e2e-api-b-47369d-openshift-default-696ccdd997-9mqtd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.113:15021/healthz/ready\": dial tcp 10.132.0.113:15021: connect: connection refused" start-of-body= Nov 03 00:59:52.622384 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:59:52.622229 4939 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-47369d-openshift-default-696ccdd997-9mqtd" podUID="49df4a8a-81d6-4c9e-baf7-40c0f7cbaae1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.113:15021/healthz/ready\": dial tcp 10.132.0.113:15021: connect: connection refused" Nov 03 00:59:52.679540 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:59:52.679466 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 00:59:53.299525 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:59:53.299481 4939 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-b-db5422-68786bc84f-9nclv" podUID="c650f474-5078-4466-99b1-4d2ea2c11093" containerName="payload-processing" containerID="cri-o://a70f79cf1ffff54da53b7c55ba0553d17a6b758e96b3bfc7467c7b99fb6c957e" gracePeriod=25 Nov 03 00:59:53.490364 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:59:53.490324 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 00:59:53.641011 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:59:53.639066 4939 patch_prober.go:29] interesting pod/e2e-api-b-47369d-openshift-default-696ccdd997-9mqtd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.113:15021/healthz/ready\": dial tcp 10.132.0.113:15021: connect: connection refused" start-of-body= Nov 03 00:59:53.641011 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:59:53.639214 4939 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-47369d-openshift-default-696ccdd997-9mqtd" podUID="49df4a8a-81d6-4c9e-baf7-40c0f7cbaae1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.113:15021/healthz/ready\": dial tcp 10.132.0.113:15021: connect: connection refused" Nov 03 00:59:53.786661 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:59:53.786623 4939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-db5422-68786bc84f-9nclv" event={"ID":"c650f474-5078-4466-99b1-4d2ea2c11093","Type":"ContainerDied","Data":"a70f79cf1ffff54da53b7c55ba0553d17a6b758e96b3bfc7467c7b99fb6c957e"} Nov 03 00:59:53.786661 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:59:53.786536 4939 generic.go:356] "Generic (PLEG): container finished" podID="c650f474-5078-4466-99b1-4d2ea2c11093" containerID="a70f79cf1ffff54da53b7c55ba0553d17a6b758e96b3bfc7467c7b99fb6c957e" exitCode=0 Nov 03 00:59:53.786893 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:59:53.786708 4939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-db5422-68786bc84f-9nclv" event={"ID":"c650f474-5078-4466-99b1-4d2ea2c11093","Type":"ContainerDied","Data":"dcb729e0dbff9115f6d41e101ef0044903cb8130c56b96700f507346f239c94e"} Nov 03 00:59:53.786893 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:59:53.786735 4939 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="dcb729e0dbff9115f6d41e101ef0044903cb8130c56b96700f507346f239c94e" Nov 03 00:59:53.799958 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:59:53.799314 4939 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-db5422-68786bc84f-9nclv" Nov 03 00:59:53.876230 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:59:53.876193 4939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c650f474-5078-4466-99b1-4d2ea2c11093-kube-api-access-zf8vc\" (UniqueName: \"kubernetes.io/projected/c650f474-5078-4466-99b1-4d2ea2c11093-kube-api-access-zf8vc\") pod \"c650f474-5078-4466-99b1-4d2ea2c11093\" (UID: \"c650f474-5078-4466-99b1-4d2ea2c11093\") " Nov 03 00:59:53.876481 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:59:53.876290 4939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c650f474-5078-4466-99b1-4d2ea2c11093-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c650f474-5078-4466-99b1-4d2ea2c11093-plugins-config-volume\") pod \"c650f474-5078-4466-99b1-4d2ea2c11093\" (UID: \"c650f474-5078-4466-99b1-4d2ea2c11093\") " Nov 03 00:59:53.877632 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:59:53.877519 4939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c650f474-5078-4466-99b1-4d2ea2c11093-plugins-config-volume" pod "c650f474-5078-4466-99b1-4d2ea2c11093" (UID: "c650f474-5078-4466-99b1-4d2ea2c11093"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 00:59:53.888230 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:59:53.883827 4939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c650f474-5078-4466-99b1-4d2ea2c11093-kube-api-access-zf8vc" pod "c650f474-5078-4466-99b1-4d2ea2c11093" (UID: "c650f474-5078-4466-99b1-4d2ea2c11093"). InnerVolumeSpecName "kube-api-access-zf8vc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 00:59:53.978866 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:59:53.978832 4939 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zf8vc\" (UniqueName: \"kubernetes.io/projected/c650f474-5078-4466-99b1-4d2ea2c11093-kube-api-access-zf8vc\") on node \"ip-10-0-140-107.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 00:59:53.978866 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:59:53.978867 4939 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c650f474-5078-4466-99b1-4d2ea2c11093-plugins-config-volume\") on node \"ip-10-0-140-107.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 00:59:54.215969 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:59:54.215868 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 00:59:54.622360 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:59:54.622327 4939 patch_prober.go:29] interesting pod/e2e-api-b-47369d-openshift-default-696ccdd997-9mqtd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.113:15021/healthz/ready\": dial tcp 10.132.0.113:15021: connect: connection refused" start-of-body= Nov 03 00:59:54.622768 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:59:54.622384 4939 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-47369d-openshift-default-696ccdd997-9mqtd" podUID="49df4a8a-81d6-4c9e-baf7-40c0f7cbaae1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.113:15021/healthz/ready\": dial tcp 10.132.0.113:15021: connect: connection refused" Nov 03 00:59:54.794276 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:59:54.794162 4939 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-db5422-68786bc84f-9nclv" Nov 03 00:59:54.836932 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:59:54.835433 4939 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-db5422-68786bc84f-9nclv"] Nov 03 00:59:54.842151 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:59:54.842122 4939 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-db5422-68786bc84f-9nclv"] Nov 03 00:59:54.852933 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:59:54.850020 4939 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-47369d-openshift-default-696ccdd997-9mqtd" Nov 03 00:59:55.294935 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:59:55.292744 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 00:59:55.511133 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:59:55.511098 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 00:59:55.622863 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:59:55.622830 4939 patch_prober.go:29] interesting pod/e2e-api-b-47369d-openshift-default-696ccdd997-9mqtd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.113:15021/healthz/ready\": dial tcp 10.132.0.113:15021: connect: connection refused" start-of-body= Nov 03 00:59:55.623283 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:59:55.622882 4939 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-47369d-openshift-default-696ccdd997-9mqtd" podUID="49df4a8a-81d6-4c9e-baf7-40c0f7cbaae1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.113:15021/healthz/ready\": dial tcp 10.132.0.113:15021: connect: connection refused" Nov 03 00:59:55.713569 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:59:55.713533 4939 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-4pw7v"] Nov 03 00:59:55.943069 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:59:55.928276 4939 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c650f474-5078-4466-99b1-4d2ea2c11093" path="/var/lib/kubelet/pods/c650f474-5078-4466-99b1-4d2ea2c11093/volumes" Nov 03 00:59:56.651967 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:59:56.630765 4939 patch_prober.go:29] interesting pod/e2e-api-b-47369d-openshift-default-696ccdd997-9mqtd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.113:15021/healthz/ready\": dial tcp 10.132.0.113:15021: connect: connection refused" start-of-body= Nov 03 00:59:56.651967 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:59:56.630823 4939 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-47369d-openshift-default-696ccdd997-9mqtd" podUID="49df4a8a-81d6-4c9e-baf7-40c0f7cbaae1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.113:15021/healthz/ready\": dial tcp 10.132.0.113:15021: connect: connection refused" Nov 03 00:59:56.811765 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:59:56.811728 4939 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-def754-969c7dd8c-rb4dt"] Nov 03 00:59:56.918186 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:59:56.918100 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 00:59:57.068620 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:59:57.068571 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 00:59:57.507279 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:59:57.506494 4939 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-f88f87-5c6d75c98c-8cjnk"] Nov 03 00:59:57.627803 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:59:57.627773 4939 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-47369d-openshift-default-696ccdd997-9mqtd" Nov 03 00:59:57.630208 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:59:57.630183 4939 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-47369d-openshift-default-696ccdd997-9mqtd" Nov 03 00:59:57.672561 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:59:57.672514 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 00:59:57.798516 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:59:57.798404 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 00:59:58.598845 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:59:58.598814 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 00:59:58.729009 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:59:58.728977 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 00:59:58.741484 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:59:58.741456 4939 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-api-b-ffdea6-6975889bf-cb625" Nov 03 00:59:58.895320 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:59:58.894455 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 00:59:59.822954 ip-10-0-140-107 kubenswrapper[4939]: I1103 00:59:59.822921 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:00:00.183674 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:00:00.183638 4939 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-650be0-29894460-m8w7r"] Nov 03 01:00:00.184400 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:00:00.184378 4939 cpu_manager.go:414] "containerMap: removing container" podUID="c650f474-5078-4466-99b1-4d2ea2c11093" containerName="payload-processing" Nov 03 01:00:00.184400 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:00:00.184398 4939 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c650f474-5078-4466-99b1-4d2ea2c11093" containerName="payload-processing" Nov 03 01:00:00.184604 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:00:00.184575 4939 memory_manager.go:365] "RemoveStaleState removing state" podUID="c650f474-5078-4466-99b1-4d2ea2c11093" containerName="payload-processing" Nov 03 01:00:00.197477 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:00:00.193448 4939 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-ffdea6-29894460-skg2j"] Nov 03 01:00:00.198274 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:00:00.198250 4939 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-650be0-29894460-m8w7r" Nov 03 01:00:00.209806 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:00:00.209781 4939 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-650be0-29894460-m8w7r"] Nov 03 01:00:00.209942 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:00:00.209829 4939 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-ffdea6-29894460-skg2j"] Nov 03 01:00:00.209942 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:00:00.209856 4939 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-def754-29894460-j7kz4"] Nov 03 01:00:00.209942 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:00:00.209897 4939 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-ffdea6-29894460-skg2j" Nov 03 01:00:00.214162 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:00:00.214058 4939 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-88bbf7-29894460-hqztw"] Nov 03 01:00:00.215486 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:00:00.215467 4939 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-def754-29894460-j7kz4" Nov 03 01:00:00.219666 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:00:00.218793 4939 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-88bbf7-29894460-hqztw" Nov 03 01:00:00.221135 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:00:00.221100 4939 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-47369d-29894460-h5m8c"] Nov 03 01:00:00.224889 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:00:00.224869 4939 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-47369d-29894460-h5m8c" Nov 03 01:00:00.225457 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:00:00.225439 4939 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-def754-29894460-j7kz4"] Nov 03 01:00:00.272935 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:00:00.247431 4939 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-47369d-29894460-h5m8c"] Nov 03 01:00:00.291025 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:00:00.290979 4939 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-6q4xp\"" Nov 03 01:00:00.307059 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:00:00.307031 4939 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-88bbf7-29894460-hqztw"] Nov 03 01:00:00.383038 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:00:00.382976 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d5ee2b64-db54-4185-bbcf-3d294be13551-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w1-88bbf7-29894460-hqztw\" (UID: \"d5ee2b64-db54-4185-bbcf-3d294be13551\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-88bbf7-29894460-hqztw" Nov 03 01:00:00.383038 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:00:00.383026 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bmhss\" (UniqueName: \"kubernetes.io/projected/d5ee2b64-db54-4185-bbcf-3d294be13551-kube-api-access-bmhss\") pod \"maas-api-key-cleanup-e2e-worker-w1-88bbf7-29894460-hqztw\" (UID: \"d5ee2b64-db54-4185-bbcf-3d294be13551\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-88bbf7-29894460-hqztw" Nov 03 01:00:00.383291 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:00:00.383191 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xsscr\" (UniqueName: \"kubernetes.io/projected/9a1dc28b-dbe2-439e-8f7f-c63f6444de42-kube-api-access-xsscr\") pod \"maas-api-key-cleanup-e2e-ipp-b-def754-29894460-j7kz4\" (UID: \"9a1dc28b-dbe2-439e-8f7f-c63f6444de42\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-def754-29894460-j7kz4" Nov 03 01:00:00.383291 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:00:00.383244 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6ec302ce-4f22-4a05-8200-9525580d6cdf-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w4-650be0-29894460-m8w7r\" (UID: \"6ec302ce-4f22-4a05-8200-9525580d6cdf\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-650be0-29894460-m8w7r" Nov 03 01:00:00.383383 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:00:00.383359 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9a1dc28b-dbe2-439e-8f7f-c63f6444de42-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-ipp-b-def754-29894460-j7kz4\" (UID: \"9a1dc28b-dbe2-439e-8f7f-c63f6444de42\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-def754-29894460-j7kz4" Nov 03 01:00:00.383477 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:00:00.383454 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xkvb2\" (UniqueName: \"kubernetes.io/projected/6ec302ce-4f22-4a05-8200-9525580d6cdf-kube-api-access-xkvb2\") pod \"maas-api-key-cleanup-e2e-worker-w4-650be0-29894460-m8w7r\" (UID: \"6ec302ce-4f22-4a05-8200-9525580d6cdf\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-650be0-29894460-m8w7r" Nov 03 01:00:00.383556 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:00:00.383541 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/370396d8-27e0-4320-b0f4-40134e62e18d-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-api-b-47369d-29894460-h5m8c\" (UID: \"370396d8-27e0-4320-b0f4-40134e62e18d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-47369d-29894460-h5m8c" Nov 03 01:00:00.383591 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:00:00.383575 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b1e21399-49c6-4226-aa7f-6de60c486c9a-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-api-b-ffdea6-29894460-skg2j\" (UID: \"b1e21399-49c6-4226-aa7f-6de60c486c9a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-ffdea6-29894460-skg2j" Nov 03 01:00:00.383652 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:00:00.383635 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cs7bg\" (UniqueName: \"kubernetes.io/projected/370396d8-27e0-4320-b0f4-40134e62e18d-kube-api-access-cs7bg\") pod \"maas-api-key-cleanup-e2e-api-b-47369d-29894460-h5m8c\" (UID: \"370396d8-27e0-4320-b0f4-40134e62e18d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-47369d-29894460-h5m8c" Nov 03 01:00:00.383720 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:00:00.383704 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-brdqq\" (UniqueName: \"kubernetes.io/projected/b1e21399-49c6-4226-aa7f-6de60c486c9a-kube-api-access-brdqq\") pod \"maas-api-key-cleanup-e2e-api-b-ffdea6-29894460-skg2j\" (UID: \"b1e21399-49c6-4226-aa7f-6de60c486c9a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-ffdea6-29894460-skg2j" Nov 03 01:00:00.485357 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:00:00.485255 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cs7bg\" (UniqueName: \"kubernetes.io/projected/370396d8-27e0-4320-b0f4-40134e62e18d-kube-api-access-cs7bg\") pod \"maas-api-key-cleanup-e2e-api-b-47369d-29894460-h5m8c\" (UID: \"370396d8-27e0-4320-b0f4-40134e62e18d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-47369d-29894460-h5m8c" Nov 03 01:00:00.485539 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:00:00.485363 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-brdqq\" (UniqueName: \"kubernetes.io/projected/b1e21399-49c6-4226-aa7f-6de60c486c9a-kube-api-access-brdqq\") pod \"maas-api-key-cleanup-e2e-api-b-ffdea6-29894460-skg2j\" (UID: \"b1e21399-49c6-4226-aa7f-6de60c486c9a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-ffdea6-29894460-skg2j" Nov 03 01:00:00.485539 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:00:00.485449 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d5ee2b64-db54-4185-bbcf-3d294be13551-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w1-88bbf7-29894460-hqztw\" (UID: \"d5ee2b64-db54-4185-bbcf-3d294be13551\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-88bbf7-29894460-hqztw" Nov 03 01:00:00.485539 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:00:00.485474 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bmhss\" (UniqueName: \"kubernetes.io/projected/d5ee2b64-db54-4185-bbcf-3d294be13551-kube-api-access-bmhss\") pod \"maas-api-key-cleanup-e2e-worker-w1-88bbf7-29894460-hqztw\" (UID: \"d5ee2b64-db54-4185-bbcf-3d294be13551\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-88bbf7-29894460-hqztw" Nov 03 01:00:00.485716 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:00:00.485592 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xsscr\" (UniqueName: \"kubernetes.io/projected/9a1dc28b-dbe2-439e-8f7f-c63f6444de42-kube-api-access-xsscr\") pod \"maas-api-key-cleanup-e2e-ipp-b-def754-29894460-j7kz4\" (UID: \"9a1dc28b-dbe2-439e-8f7f-c63f6444de42\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-def754-29894460-j7kz4" Nov 03 01:00:00.485716 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:00:00.485651 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6ec302ce-4f22-4a05-8200-9525580d6cdf-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w4-650be0-29894460-m8w7r\" (UID: \"6ec302ce-4f22-4a05-8200-9525580d6cdf\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-650be0-29894460-m8w7r" Nov 03 01:00:00.485824 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:00:00.485739 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9a1dc28b-dbe2-439e-8f7f-c63f6444de42-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-ipp-b-def754-29894460-j7kz4\" (UID: \"9a1dc28b-dbe2-439e-8f7f-c63f6444de42\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-def754-29894460-j7kz4" Nov 03 01:00:00.485824 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:00:00.485801 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xkvb2\" (UniqueName: \"kubernetes.io/projected/6ec302ce-4f22-4a05-8200-9525580d6cdf-kube-api-access-xkvb2\") pod \"maas-api-key-cleanup-e2e-worker-w4-650be0-29894460-m8w7r\" (UID: \"6ec302ce-4f22-4a05-8200-9525580d6cdf\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-650be0-29894460-m8w7r" Nov 03 01:00:00.485993 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:00:00.485870 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/370396d8-27e0-4320-b0f4-40134e62e18d-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-api-b-47369d-29894460-h5m8c\" (UID: \"370396d8-27e0-4320-b0f4-40134e62e18d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-47369d-29894460-h5m8c" Nov 03 01:00:00.485993 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:00:00.485924 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b1e21399-49c6-4226-aa7f-6de60c486c9a-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-api-b-ffdea6-29894460-skg2j\" (UID: \"b1e21399-49c6-4226-aa7f-6de60c486c9a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-ffdea6-29894460-skg2j" Nov 03 01:00:00.486778 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:00:00.486736 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b1e21399-49c6-4226-aa7f-6de60c486c9a-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-api-b-ffdea6-29894460-skg2j\" (UID: \"b1e21399-49c6-4226-aa7f-6de60c486c9a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-ffdea6-29894460-skg2j" Nov 03 01:00:00.487798 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:00:00.487767 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9a1dc28b-dbe2-439e-8f7f-c63f6444de42-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-ipp-b-def754-29894460-j7kz4\" (UID: \"9a1dc28b-dbe2-439e-8f7f-c63f6444de42\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-def754-29894460-j7kz4" Nov 03 01:00:00.487904 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:00:00.487767 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d5ee2b64-db54-4185-bbcf-3d294be13551-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w1-88bbf7-29894460-hqztw\" (UID: \"d5ee2b64-db54-4185-bbcf-3d294be13551\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-88bbf7-29894460-hqztw" Nov 03 01:00:00.488461 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:00:00.488436 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/370396d8-27e0-4320-b0f4-40134e62e18d-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-api-b-47369d-29894460-h5m8c\" (UID: \"370396d8-27e0-4320-b0f4-40134e62e18d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-47369d-29894460-h5m8c" Nov 03 01:00:00.488596 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:00:00.488532 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6ec302ce-4f22-4a05-8200-9525580d6cdf-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w4-650be0-29894460-m8w7r\" (UID: \"6ec302ce-4f22-4a05-8200-9525580d6cdf\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-650be0-29894460-m8w7r" Nov 03 01:00:00.505027 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:00:00.504992 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cs7bg\" (UniqueName: \"kubernetes.io/projected/370396d8-27e0-4320-b0f4-40134e62e18d-kube-api-access-cs7bg\") pod \"maas-api-key-cleanup-e2e-api-b-47369d-29894460-h5m8c\" (UID: \"370396d8-27e0-4320-b0f4-40134e62e18d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-47369d-29894460-h5m8c" Nov 03 01:00:00.505527 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:00:00.505490 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bmhss\" (UniqueName: \"kubernetes.io/projected/d5ee2b64-db54-4185-bbcf-3d294be13551-kube-api-access-bmhss\") pod \"maas-api-key-cleanup-e2e-worker-w1-88bbf7-29894460-hqztw\" (UID: \"d5ee2b64-db54-4185-bbcf-3d294be13551\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-88bbf7-29894460-hqztw" Nov 03 01:00:00.506770 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:00:00.506744 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xkvb2\" (UniqueName: \"kubernetes.io/projected/6ec302ce-4f22-4a05-8200-9525580d6cdf-kube-api-access-xkvb2\") pod \"maas-api-key-cleanup-e2e-worker-w4-650be0-29894460-m8w7r\" (UID: \"6ec302ce-4f22-4a05-8200-9525580d6cdf\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-650be0-29894460-m8w7r" Nov 03 01:00:00.508371 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:00:00.508351 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xsscr\" (UniqueName: \"kubernetes.io/projected/9a1dc28b-dbe2-439e-8f7f-c63f6444de42-kube-api-access-xsscr\") pod \"maas-api-key-cleanup-e2e-ipp-b-def754-29894460-j7kz4\" (UID: \"9a1dc28b-dbe2-439e-8f7f-c63f6444de42\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-def754-29894460-j7kz4" Nov 03 01:00:00.508510 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:00:00.508486 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-brdqq\" (UniqueName: \"kubernetes.io/projected/b1e21399-49c6-4226-aa7f-6de60c486c9a-kube-api-access-brdqq\") pod \"maas-api-key-cleanup-e2e-api-b-ffdea6-29894460-skg2j\" (UID: \"b1e21399-49c6-4226-aa7f-6de60c486c9a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-ffdea6-29894460-skg2j" Nov 03 01:00:00.517585 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:00:00.517558 4939 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-650be0-29894460-m8w7r" Nov 03 01:00:00.541353 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:00:00.541327 4939 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-ffdea6-29894460-skg2j" Nov 03 01:00:00.568931 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:00:00.544784 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:00:00.601382 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:00:00.601183 4939 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-def754-29894460-j7kz4" Nov 03 01:00:00.610899 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:00:00.610707 4939 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-88bbf7-29894460-hqztw" Nov 03 01:00:00.620098 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:00:00.620073 4939 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-47369d-29894460-h5m8c" Nov 03 01:00:00.688022 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:00:00.685900 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:00:01.108433 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:00:01.075053 4939 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-650be0-29894460-m8w7r"] Nov 03 01:00:01.379336 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:00:01.379297 4939 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-88bbf7-29894460-hqztw"] Nov 03 01:00:01.415200 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:00:01.415171 4939 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-def754-29894460-j7kz4"] Nov 03 01:00:01.474485 ip-10-0-140-107 kubenswrapper[4939]: W1103 01:00:01.474368 4939 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd5ee2b64_db54_4185_bbcf_3d294be13551.slice/crio-1c631c2c4f5c23fa4d075ae2dd2a8c02983164b1470e38d06b0571bac4bd4446 WatchSource:0}: Error finding container 1c631c2c4f5c23fa4d075ae2dd2a8c02983164b1470e38d06b0571bac4bd4446: Status 404 returned error can't find the container with id 1c631c2c4f5c23fa4d075ae2dd2a8c02983164b1470e38d06b0571bac4bd4446 Nov 03 01:00:01.475410 ip-10-0-140-107 kubenswrapper[4939]: W1103 01:00:01.475387 4939 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9a1dc28b_dbe2_439e_8f7f_c63f6444de42.slice/crio-0bd86e1ba00c1609eacf46abd52c0b6b7a491711e1a4223844e96cbdc228d610 WatchSource:0}: Error finding container 0bd86e1ba00c1609eacf46abd52c0b6b7a491711e1a4223844e96cbdc228d610: Status 404 returned error can't find the container with id 0bd86e1ba00c1609eacf46abd52c0b6b7a491711e1a4223844e96cbdc228d610 Nov 03 01:00:01.476875 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:00:01.476847 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:00:01.646467 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:00:01.646024 4939 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-ffdea6-29894460-skg2j"] Nov 03 01:00:01.684127 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:00:01.684095 4939 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-47369d-29894460-h5m8c"] Nov 03 01:00:01.812819 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:00:01.812786 4939 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-def754-969c7dd8c-rb4dt" podUID="6cc03f20-f3ea-491f-b0be-e3bc1b9c3ac1" containerName="payload-pre-processing" containerID="cri-o://15ce4fbbe00c91a18e33fb27084951ae04c083a734dd3ccb3948ae9516283304" gracePeriod=25 Nov 03 01:00:01.996780 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:00:01.996748 4939 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-80f70e-659f4d7d6f-69755"] Nov 03 01:00:01.997016 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:00:01.996991 4939 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-80f70e-659f4d7d6f-69755" podUID="d102e320-316d-4577-b088-9b39a27e8794" containerName="maas-api" containerID="cri-o://68516ddd670620157dcbb71927055247038344c2a29683bc162d157e0d35c5cc" gracePeriod=30 Nov 03 01:00:02.079525 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:00:02.079439 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:00:02.202616 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:00:02.202117 4939 generic.go:356] "Generic (PLEG): container finished" podID="d102e320-316d-4577-b088-9b39a27e8794" containerID="68516ddd670620157dcbb71927055247038344c2a29683bc162d157e0d35c5cc" exitCode=0 Nov 03 01:00:02.202616 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:00:02.202196 4939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-80f70e-659f4d7d6f-69755" event={"ID":"d102e320-316d-4577-b088-9b39a27e8794","Type":"ContainerDied","Data":"68516ddd670620157dcbb71927055247038344c2a29683bc162d157e0d35c5cc"} Nov 03 01:00:02.205942 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:00:02.205879 4939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-def754-29894460-j7kz4" event={"ID":"9a1dc28b-dbe2-439e-8f7f-c63f6444de42","Type":"ContainerStarted","Data":"0bd86e1ba00c1609eacf46abd52c0b6b7a491711e1a4223844e96cbdc228d610"} Nov 03 01:00:02.207449 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:00:02.207422 4939 generic.go:356] "Generic (PLEG): container finished" podID="6cc03f20-f3ea-491f-b0be-e3bc1b9c3ac1" containerID="15ce4fbbe00c91a18e33fb27084951ae04c083a734dd3ccb3948ae9516283304" exitCode=0 Nov 03 01:00:02.207779 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:00:02.207569 4939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-def754-969c7dd8c-rb4dt" event={"ID":"6cc03f20-f3ea-491f-b0be-e3bc1b9c3ac1","Type":"ContainerDied","Data":"15ce4fbbe00c91a18e33fb27084951ae04c083a734dd3ccb3948ae9516283304"} Nov 03 01:00:02.213318 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:00:02.213127 4939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-47369d-29894460-h5m8c" event={"ID":"370396d8-27e0-4320-b0f4-40134e62e18d","Type":"ContainerStarted","Data":"4f6d9c82dd9854261b1837cc7256c93f8a3d9327ea782ce260ceff98c118c3b3"} Nov 03 01:00:02.263444 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:00:02.227841 4939 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-def754-969c7dd8c-rb4dt" Nov 03 01:00:02.464054 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:00:02.397855 4939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6cc03f20-f3ea-491f-b0be-e3bc1b9c3ac1-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6cc03f20-f3ea-491f-b0be-e3bc1b9c3ac1-plugins-config-volume\") pod \"6cc03f20-f3ea-491f-b0be-e3bc1b9c3ac1\" (UID: \"6cc03f20-f3ea-491f-b0be-e3bc1b9c3ac1\") " Nov 03 01:00:02.464054 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:00:02.415770 4939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6cc03f20-f3ea-491f-b0be-e3bc1b9c3ac1-kube-api-access-8z4fs\" (UniqueName: \"kubernetes.io/projected/6cc03f20-f3ea-491f-b0be-e3bc1b9c3ac1-kube-api-access-8z4fs\") pod \"6cc03f20-f3ea-491f-b0be-e3bc1b9c3ac1\" (UID: \"6cc03f20-f3ea-491f-b0be-e3bc1b9c3ac1\") " Nov 03 01:00:02.491236 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:00:02.464758 4939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6cc03f20-f3ea-491f-b0be-e3bc1b9c3ac1-plugins-config-volume" pod "6cc03f20-f3ea-491f-b0be-e3bc1b9c3ac1" (UID: "6cc03f20-f3ea-491f-b0be-e3bc1b9c3ac1"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 01:00:02.491236 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:00:02.471076 4939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6cc03f20-f3ea-491f-b0be-e3bc1b9c3ac1-kube-api-access-8z4fs" pod "6cc03f20-f3ea-491f-b0be-e3bc1b9c3ac1" (UID: "6cc03f20-f3ea-491f-b0be-e3bc1b9c3ac1"). InnerVolumeSpecName "kube-api-access-8z4fs". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 01:00:02.511979 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:00:02.491720 4939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-ffdea6-29894460-skg2j" event={"ID":"b1e21399-49c6-4226-aa7f-6de60c486c9a","Type":"ContainerStarted","Data":"32b66444cbc0c4dc6da318b1196b49d8355385e397dd1f6e182c92e64bb93f87"} Nov 03 01:00:02.511979 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:00:02.491792 4939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-ffdea6-29894460-skg2j" event={"ID":"b1e21399-49c6-4226-aa7f-6de60c486c9a","Type":"ContainerStarted","Data":"1f37a36783734982e299f86768f5b7f578a91fb584af58be65d153012fac4a89"} Nov 03 01:00:02.534927 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:00:02.512314 4939 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-a-f88f87-5c6d75c98c-8cjnk" podUID="0a9269dd-5b6c-4311-8f78-87028a4a7935" containerName="payload-processing" containerID="cri-o://4aa28d207392c7d52c00f18ec084eba275ca07282ae6f52dea48f35358eaa71a" gracePeriod=25 Nov 03 01:00:02.685216 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:00:02.630345 4939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-650be0-29894460-m8w7r" event={"ID":"6ec302ce-4f22-4a05-8200-9525580d6cdf","Type":"ContainerStarted","Data":"bff77f597899599a87613800e7ad414baa9fba4a51c6659cb287339a1ee7bb87"} Nov 03 01:00:02.685216 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:00:02.630420 4939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-650be0-29894460-m8w7r" event={"ID":"6ec302ce-4f22-4a05-8200-9525580d6cdf","Type":"ContainerStarted","Data":"a8d104cfa0db11e27139f9eb9d5cc79760892c6c22f9fe3ff2889e96eb7f73c4"} Nov 03 01:00:02.688727 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:00:02.688253 4939 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8z4fs\" (UniqueName: \"kubernetes.io/projected/6cc03f20-f3ea-491f-b0be-e3bc1b9c3ac1-kube-api-access-8z4fs\") on node \"ip-10-0-140-107.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:00:02.688727 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:00:02.688427 4939 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6cc03f20-f3ea-491f-b0be-e3bc1b9c3ac1-plugins-config-volume\") on node \"ip-10-0-140-107.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:00:02.699560 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:00:02.698688 4939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-88bbf7-29894460-hqztw" event={"ID":"d5ee2b64-db54-4185-bbcf-3d294be13551","Type":"ContainerStarted","Data":"1c631c2c4f5c23fa4d075ae2dd2a8c02983164b1470e38d06b0571bac4bd4446"} Nov 03 01:00:02.713960 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:00:02.713375 4939 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-ffdea6-29894460-skg2j" podStartSLOduration=2.713355153 podStartE2EDuration="2.713355153s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 01:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 01:00:02.685899542 +0000 UTC m=+1150.950234489" watchObservedRunningTime="2026-11-03 01:00:02.713355153 +0000 UTC m=+1150.977690101" Nov 03 01:00:02.724088 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:00:02.723215 4939 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-650be0-29894460-m8w7r" podStartSLOduration=2.723197271 podStartE2EDuration="2.723197271s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 01:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 01:00:02.709774377 +0000 UTC m=+1150.974109322" watchObservedRunningTime="2026-11-03 01:00:02.723197271 +0000 UTC m=+1150.987532218" Nov 03 01:00:02.731243 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:00:02.731187 4939 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-88bbf7-29894460-hqztw" podStartSLOduration=2.731161545 podStartE2EDuration="2.731161545s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 01:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 01:00:02.727060489 +0000 UTC m=+1150.991395435" watchObservedRunningTime="2026-11-03 01:00:02.731161545 +0000 UTC m=+1150.995496492" Nov 03 01:00:02.791665 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:00:02.791454 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:00:03.011438 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:00:03.011373 4939 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-80f70e-659f4d7d6f-69755" Nov 03 01:00:03.032262 ip-10-0-140-107 kubenswrapper[4939]: E1103 01:00:03.032221 4939 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd5ee2b64_db54_4185_bbcf_3d294be13551.slice/crio-conmon-773a7864b6e75b779a03ec331dc76bd2d897b0ac218985b2bcbb04f4d11ab112.scope\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd5ee2b64_db54_4185_bbcf_3d294be13551.slice/crio-773a7864b6e75b779a03ec331dc76bd2d897b0ac218985b2bcbb04f4d11ab112.scope\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb1e21399_49c6_4226_aa7f_6de60c486c9a.slice/crio-32b66444cbc0c4dc6da318b1196b49d8355385e397dd1f6e182c92e64bb93f87.scope\": RecentStats: unable to find data in memory cache]" Nov 03 01:00:03.098473 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:00:03.098429 4939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d102e320-316d-4577-b088-9b39a27e8794-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d102e320-316d-4577-b088-9b39a27e8794-maas-api-tls\") pod \"d102e320-316d-4577-b088-9b39a27e8794\" (UID: \"d102e320-316d-4577-b088-9b39a27e8794\") " Nov 03 01:00:03.098645 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:00:03.098501 4939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d102e320-316d-4577-b088-9b39a27e8794-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d102e320-316d-4577-b088-9b39a27e8794-openshift-service-ca\") pod \"d102e320-316d-4577-b088-9b39a27e8794\" (UID: \"d102e320-316d-4577-b088-9b39a27e8794\") " Nov 03 01:00:03.098645 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:00:03.098571 4939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d102e320-316d-4577-b088-9b39a27e8794-kube-api-access-bdmhl\" (UniqueName: \"kubernetes.io/projected/d102e320-316d-4577-b088-9b39a27e8794-kube-api-access-bdmhl\") pod \"d102e320-316d-4577-b088-9b39a27e8794\" (UID: \"d102e320-316d-4577-b088-9b39a27e8794\") " Nov 03 01:00:03.098744 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:00:03.098726 4939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d102e320-316d-4577-b088-9b39a27e8794-metrics-certs\" (UniqueName: \"kubernetes.io/secret/d102e320-316d-4577-b088-9b39a27e8794-metrics-certs\") pod \"d102e320-316d-4577-b088-9b39a27e8794\" (UID: \"d102e320-316d-4577-b088-9b39a27e8794\") " Nov 03 01:00:03.100434 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:00:03.100397 4939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d102e320-316d-4577-b088-9b39a27e8794-openshift-service-ca" pod "d102e320-316d-4577-b088-9b39a27e8794" (UID: "d102e320-316d-4577-b088-9b39a27e8794"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 01:00:03.103365 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:00:03.103173 4939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d102e320-316d-4577-b088-9b39a27e8794-kube-api-access-bdmhl" pod "d102e320-316d-4577-b088-9b39a27e8794" (UID: "d102e320-316d-4577-b088-9b39a27e8794"). InnerVolumeSpecName "kube-api-access-bdmhl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 01:00:03.104017 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:00:03.103932 4939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d102e320-316d-4577-b088-9b39a27e8794-metrics-certs" pod "d102e320-316d-4577-b088-9b39a27e8794" (UID: "d102e320-316d-4577-b088-9b39a27e8794"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 01:00:03.105367 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:00:03.105336 4939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d102e320-316d-4577-b088-9b39a27e8794-maas-api-tls" pod "d102e320-316d-4577-b088-9b39a27e8794" (UID: "d102e320-316d-4577-b088-9b39a27e8794"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 01:00:03.129613 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:00:03.129589 4939 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-f88f87-5c6d75c98c-8cjnk" Nov 03 01:00:03.207890 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:00:03.207849 4939 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d102e320-316d-4577-b088-9b39a27e8794-metrics-certs\") on node \"ip-10-0-140-107.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:00:03.207890 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:00:03.207884 4939 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d102e320-316d-4577-b088-9b39a27e8794-maas-api-tls\") on node \"ip-10-0-140-107.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:00:03.207890 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:00:03.207900 4939 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d102e320-316d-4577-b088-9b39a27e8794-openshift-service-ca\") on node \"ip-10-0-140-107.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:00:03.208582 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:00:03.207925 4939 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bdmhl\" (UniqueName: \"kubernetes.io/projected/d102e320-316d-4577-b088-9b39a27e8794-kube-api-access-bdmhl\") on node \"ip-10-0-140-107.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:00:03.309425 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:00:03.309327 4939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0a9269dd-5b6c-4311-8f78-87028a4a7935-kube-api-access-7ws6c\" (UniqueName: \"kubernetes.io/projected/0a9269dd-5b6c-4311-8f78-87028a4a7935-kube-api-access-7ws6c\") pod \"0a9269dd-5b6c-4311-8f78-87028a4a7935\" (UID: \"0a9269dd-5b6c-4311-8f78-87028a4a7935\") " Nov 03 01:00:03.309425 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:00:03.309378 4939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0a9269dd-5b6c-4311-8f78-87028a4a7935-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0a9269dd-5b6c-4311-8f78-87028a4a7935-plugins-config-volume\") pod \"0a9269dd-5b6c-4311-8f78-87028a4a7935\" (UID: \"0a9269dd-5b6c-4311-8f78-87028a4a7935\") " Nov 03 01:00:03.309756 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:00:03.309735 4939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0a9269dd-5b6c-4311-8f78-87028a4a7935-plugins-config-volume" pod "0a9269dd-5b6c-4311-8f78-87028a4a7935" (UID: "0a9269dd-5b6c-4311-8f78-87028a4a7935"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 01:00:03.310120 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:00:03.310095 4939 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0a9269dd-5b6c-4311-8f78-87028a4a7935-plugins-config-volume\") on node \"ip-10-0-140-107.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:00:03.311602 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:00:03.311573 4939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0a9269dd-5b6c-4311-8f78-87028a4a7935-kube-api-access-7ws6c" pod "0a9269dd-5b6c-4311-8f78-87028a4a7935" (UID: "0a9269dd-5b6c-4311-8f78-87028a4a7935"). InnerVolumeSpecName "kube-api-access-7ws6c". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 01:00:03.412157 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:00:03.412119 4939 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7ws6c\" (UniqueName: \"kubernetes.io/projected/0a9269dd-5b6c-4311-8f78-87028a4a7935-kube-api-access-7ws6c\") on node \"ip-10-0-140-107.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:00:03.512896 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:00:03.512865 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:00:03.710111 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:00:03.710075 4939 generic.go:356] "Generic (PLEG): container finished" podID="b1e21399-49c6-4226-aa7f-6de60c486c9a" containerID="32b66444cbc0c4dc6da318b1196b49d8355385e397dd1f6e182c92e64bb93f87" exitCode=7 Nov 03 01:00:03.710323 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:00:03.710164 4939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-ffdea6-29894460-skg2j" event={"ID":"b1e21399-49c6-4226-aa7f-6de60c486c9a","Type":"ContainerDied","Data":"32b66444cbc0c4dc6da318b1196b49d8355385e397dd1f6e182c92e64bb93f87"} Nov 03 01:00:03.710520 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:00:03.710501 4939 scope.go:122] "RemoveContainer" containerID="32b66444cbc0c4dc6da318b1196b49d8355385e397dd1f6e182c92e64bb93f87" Nov 03 01:00:03.711963 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:00:03.711937 4939 generic.go:356] "Generic (PLEG): container finished" podID="6ec302ce-4f22-4a05-8200-9525580d6cdf" containerID="bff77f597899599a87613800e7ad414baa9fba4a51c6659cb287339a1ee7bb87" exitCode=60 Nov 03 01:00:03.712074 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:00:03.712034 4939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-650be0-29894460-m8w7r" event={"ID":"6ec302ce-4f22-4a05-8200-9525580d6cdf","Type":"ContainerDied","Data":"bff77f597899599a87613800e7ad414baa9fba4a51c6659cb287339a1ee7bb87"} Nov 03 01:00:03.712332 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:00:03.712309 4939 scope.go:122] "RemoveContainer" containerID="bff77f597899599a87613800e7ad414baa9fba4a51c6659cb287339a1ee7bb87" Nov 03 01:00:03.713983 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:00:03.713922 4939 generic.go:356] "Generic (PLEG): container finished" podID="d5ee2b64-db54-4185-bbcf-3d294be13551" containerID="773a7864b6e75b779a03ec331dc76bd2d897b0ac218985b2bcbb04f4d11ab112" exitCode=60 Nov 03 01:00:03.714055 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:00:03.714019 4939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-88bbf7-29894460-hqztw" event={"ID":"d5ee2b64-db54-4185-bbcf-3d294be13551","Type":"ContainerDied","Data":"773a7864b6e75b779a03ec331dc76bd2d897b0ac218985b2bcbb04f4d11ab112"} Nov 03 01:00:03.714325 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:00:03.714301 4939 scope.go:122] "RemoveContainer" containerID="773a7864b6e75b779a03ec331dc76bd2d897b0ac218985b2bcbb04f4d11ab112" Nov 03 01:00:03.716582 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:00:03.716405 4939 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-80f70e-659f4d7d6f-69755" Nov 03 01:00:03.716582 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:00:03.716550 4939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-80f70e-659f4d7d6f-69755" event={"ID":"d102e320-316d-4577-b088-9b39a27e8794","Type":"ContainerDied","Data":"0fc511f219a4f86b00f66d6d403ce7b78a2822c8bf4e1af03a42ce186b2aff56"} Nov 03 01:00:03.716702 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:00:03.716588 4939 scope.go:122] "RemoveContainer" containerID="68516ddd670620157dcbb71927055247038344c2a29683bc162d157e0d35c5cc" Nov 03 01:00:03.718530 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:00:03.718502 4939 generic.go:356] "Generic (PLEG): container finished" podID="9a1dc28b-dbe2-439e-8f7f-c63f6444de42" containerID="279d810be0c6f32a56b74c2e9d66ee39598c854ac215c68b4d268d7caabba636" exitCode=60 Nov 03 01:00:03.718718 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:00:03.718582 4939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-def754-29894460-j7kz4" event={"ID":"9a1dc28b-dbe2-439e-8f7f-c63f6444de42","Type":"ContainerDied","Data":"279d810be0c6f32a56b74c2e9d66ee39598c854ac215c68b4d268d7caabba636"} Nov 03 01:00:03.718871 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:00:03.718849 4939 scope.go:122] "RemoveContainer" containerID="279d810be0c6f32a56b74c2e9d66ee39598c854ac215c68b4d268d7caabba636" Nov 03 01:00:03.720513 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:00:03.720495 4939 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-def754-969c7dd8c-rb4dt" Nov 03 01:00:03.720728 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:00:03.720502 4939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-def754-969c7dd8c-rb4dt" event={"ID":"6cc03f20-f3ea-491f-b0be-e3bc1b9c3ac1","Type":"ContainerDied","Data":"b371e3f9a98ad068b4bb55663668ba3d0addbc763e6b564e94120904a9b3c3d5"} Nov 03 01:00:03.722308 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:00:03.722253 4939 generic.go:356] "Generic (PLEG): container finished" podID="0a9269dd-5b6c-4311-8f78-87028a4a7935" containerID="4aa28d207392c7d52c00f18ec084eba275ca07282ae6f52dea48f35358eaa71a" exitCode=0 Nov 03 01:00:03.722404 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:00:03.722336 4939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-f88f87-5c6d75c98c-8cjnk" event={"ID":"0a9269dd-5b6c-4311-8f78-87028a4a7935","Type":"ContainerDied","Data":"4aa28d207392c7d52c00f18ec084eba275ca07282ae6f52dea48f35358eaa71a"} Nov 03 01:00:03.722404 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:00:03.722361 4939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-f88f87-5c6d75c98c-8cjnk" event={"ID":"0a9269dd-5b6c-4311-8f78-87028a4a7935","Type":"ContainerDied","Data":"6c18058941a15f0b36138239c12c60e865606e83958fa3f8757529457eeb609b"} Nov 03 01:00:03.722522 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:00:03.722411 4939 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-f88f87-5c6d75c98c-8cjnk" Nov 03 01:00:03.731173 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:00:03.731128 4939 generic.go:356] "Generic (PLEG): container finished" podID="370396d8-27e0-4320-b0f4-40134e62e18d" containerID="0832f2f864a69f37fed1accf4c0c89d135212a312c8527d26fcf5726b34bb9b6" exitCode=60 Nov 03 01:00:03.731288 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:00:03.731278 4939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-47369d-29894460-h5m8c" event={"ID":"370396d8-27e0-4320-b0f4-40134e62e18d","Type":"ContainerDied","Data":"0832f2f864a69f37fed1accf4c0c89d135212a312c8527d26fcf5726b34bb9b6"} Nov 03 01:00:03.731609 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:00:03.731589 4939 scope.go:122] "RemoveContainer" containerID="0832f2f864a69f37fed1accf4c0c89d135212a312c8527d26fcf5726b34bb9b6" Nov 03 01:00:03.760545 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:00:03.760522 4939 scope.go:122] "RemoveContainer" containerID="15ce4fbbe00c91a18e33fb27084951ae04c083a734dd3ccb3948ae9516283304" Nov 03 01:00:03.806377 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:00:03.806356 4939 scope.go:122] "RemoveContainer" containerID="4aa28d207392c7d52c00f18ec084eba275ca07282ae6f52dea48f35358eaa71a" Nov 03 01:00:03.840565 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:00:03.840541 4939 scope.go:122] "RemoveContainer" containerID="4aa28d207392c7d52c00f18ec084eba275ca07282ae6f52dea48f35358eaa71a" Nov 03 01:00:03.841477 ip-10-0-140-107 kubenswrapper[4939]: E1103 01:00:03.841439 4939 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4aa28d207392c7d52c00f18ec084eba275ca07282ae6f52dea48f35358eaa71a\": container with ID starting with 4aa28d207392c7d52c00f18ec084eba275ca07282ae6f52dea48f35358eaa71a not found: ID does not exist" containerID="4aa28d207392c7d52c00f18ec084eba275ca07282ae6f52dea48f35358eaa71a" Nov 03 01:00:03.841570 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:00:03.841486 4939 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4aa28d207392c7d52c00f18ec084eba275ca07282ae6f52dea48f35358eaa71a"} err="failed to get container status \"4aa28d207392c7d52c00f18ec084eba275ca07282ae6f52dea48f35358eaa71a\": rpc error: code = NotFound desc = could not find container \"4aa28d207392c7d52c00f18ec084eba275ca07282ae6f52dea48f35358eaa71a\": container with ID starting with 4aa28d207392c7d52c00f18ec084eba275ca07282ae6f52dea48f35358eaa71a not found: ID does not exist" Nov 03 01:00:03.862927 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:00:03.862786 4939 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-80f70e-659f4d7d6f-69755"] Nov 03 01:00:03.880265 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:00:03.880229 4939 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-80f70e-659f4d7d6f-69755"] Nov 03 01:00:03.880402 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:00:03.880288 4939 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-def754-969c7dd8c-rb4dt"] Nov 03 01:00:03.881698 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:00:03.881543 4939 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-def754-969c7dd8c-rb4dt"] Nov 03 01:00:03.909672 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:00:03.909642 4939 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-f88f87-5c6d75c98c-8cjnk"] Nov 03 01:00:03.914473 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:00:03.914445 4939 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-f88f87-5c6d75c98c-8cjnk"] Nov 03 01:00:04.111770 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:00:04.111732 4939 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-ipp-a-f88f87-5c6d75c98c-8cjnk" podUID="0a9269dd-5b6c-4311-8f78-87028a4a7935" containerName="payload-processing" probeResult="failure" output="timeout: failed to connect service \"10.132.0.119:50052\" within 1s: context deadline exceeded" Nov 03 01:00:04.112123 ip-10-0-140-107 kubenswrapper[4939]: W1103 01:00:04.112102 4939 logging.go:55] [core] [Channel #1088 SubChannel #1089]grpc: addrConn.createTransport failed to connect to {Addr: "10.132.0.119:50052", ServerName: "10.132.0.119:50052", BalancerAttributes: {"<%!p(pickfirstleaf.managedByPickfirstKeyType={})>": "<%!p(bool=true)>" }}. Err: connection error: desc = "transport: Error while dialing: dial tcp 10.132.0.119:50052: operation was canceled" Nov 03 01:00:04.323996 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:00:04.323876 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:00:04.744636 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:00:04.744599 4939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-ffdea6-29894460-skg2j" event={"ID":"b1e21399-49c6-4226-aa7f-6de60c486c9a","Type":"ContainerStarted","Data":"3fc762abd3a0645501ccd2889def951ae9a2f1458e068b58d3b427869547084b"} Nov 03 01:00:04.746838 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:00:04.746802 4939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-650be0-29894460-m8w7r" event={"ID":"6ec302ce-4f22-4a05-8200-9525580d6cdf","Type":"ContainerStarted","Data":"44e644f7a08209b4cdc4cb6cede31c9860f7c784b6c616cb5f306bcd21a1005d"} Nov 03 01:00:04.749434 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:00:04.749400 4939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-88bbf7-29894460-hqztw" event={"ID":"d5ee2b64-db54-4185-bbcf-3d294be13551","Type":"ContainerStarted","Data":"8765b051546fa5d279071124161acbf49f49cc29876f4a40fd26055fa3330c0a"} Nov 03 01:00:04.753485 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:00:04.753460 4939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-def754-29894460-j7kz4" event={"ID":"9a1dc28b-dbe2-439e-8f7f-c63f6444de42","Type":"ContainerStarted","Data":"957d817952c82ccccee19bf0d7a80c5866851aff40ebae28904fc73c6ab02966"} Nov 03 01:00:04.758361 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:00:04.758335 4939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-47369d-29894460-h5m8c" event={"ID":"370396d8-27e0-4320-b0f4-40134e62e18d","Type":"ContainerStarted","Data":"bc2ad4555c5de1943c7e3dc8fa16dd28e77bdf84dfad526454916726a15c34ce"} Nov 03 01:00:04.824017 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:00:04.823967 4939 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-47369d-29894460-h5m8c" podStartSLOduration=4.823953866 podStartE2EDuration="4.823953866s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 01:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 01:00:04.820773023 +0000 UTC m=+1153.085107968" watchObservedRunningTime="2026-11-03 01:00:04.823953866 +0000 UTC m=+1153.088288850" Nov 03 01:00:04.853552 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:00:04.853505 4939 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-def754-29894460-j7kz4" podStartSLOduration=4.853492897 podStartE2EDuration="4.853492897s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 01:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 01:00:04.851340283 +0000 UTC m=+1153.115675232" watchObservedRunningTime="2026-11-03 01:00:04.853492897 +0000 UTC m=+1153.117827841" Nov 03 01:00:05.509362 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:00:05.509313 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:00:05.770323 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:00:05.770245 4939 generic.go:356] "Generic (PLEG): container finished" podID="370396d8-27e0-4320-b0f4-40134e62e18d" containerID="bc2ad4555c5de1943c7e3dc8fa16dd28e77bdf84dfad526454916726a15c34ce" exitCode=60 Nov 03 01:00:05.770481 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:00:05.770329 4939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-47369d-29894460-h5m8c" event={"ID":"370396d8-27e0-4320-b0f4-40134e62e18d","Type":"ContainerDied","Data":"bc2ad4555c5de1943c7e3dc8fa16dd28e77bdf84dfad526454916726a15c34ce"} Nov 03 01:00:05.770481 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:00:05.770401 4939 scope.go:122] "RemoveContainer" containerID="0832f2f864a69f37fed1accf4c0c89d135212a312c8527d26fcf5726b34bb9b6" Nov 03 01:00:05.770643 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:00:05.770624 4939 scope.go:122] "RemoveContainer" containerID="bc2ad4555c5de1943c7e3dc8fa16dd28e77bdf84dfad526454916726a15c34ce" Nov 03 01:00:05.770980 ip-10-0-140-107 kubenswrapper[4939]: E1103 01:00:05.770954 4939 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-api-b-47369d-29894460-h5m8c_odh-ai-gateway-infra(370396d8-27e0-4320-b0f4-40134e62e18d)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-47369d-29894460-h5m8c" podUID="370396d8-27e0-4320-b0f4-40134e62e18d" Nov 03 01:00:05.773431 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:00:05.773410 4939 generic.go:356] "Generic (PLEG): container finished" podID="b1e21399-49c6-4226-aa7f-6de60c486c9a" containerID="3fc762abd3a0645501ccd2889def951ae9a2f1458e068b58d3b427869547084b" exitCode=7 Nov 03 01:00:05.773613 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:00:05.773479 4939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-ffdea6-29894460-skg2j" event={"ID":"b1e21399-49c6-4226-aa7f-6de60c486c9a","Type":"ContainerDied","Data":"3fc762abd3a0645501ccd2889def951ae9a2f1458e068b58d3b427869547084b"} Nov 03 01:00:05.773766 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:00:05.773750 4939 scope.go:122] "RemoveContainer" containerID="3fc762abd3a0645501ccd2889def951ae9a2f1458e068b58d3b427869547084b" Nov 03 01:00:05.774103 ip-10-0-140-107 kubenswrapper[4939]: E1103 01:00:05.774080 4939 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-api-b-ffdea6-29894460-skg2j_odh-ai-gateway-infra(b1e21399-49c6-4226-aa7f-6de60c486c9a)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-ffdea6-29894460-skg2j" podUID="b1e21399-49c6-4226-aa7f-6de60c486c9a" Nov 03 01:00:05.775588 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:00:05.775560 4939 generic.go:356] "Generic (PLEG): container finished" podID="6ec302ce-4f22-4a05-8200-9525580d6cdf" containerID="44e644f7a08209b4cdc4cb6cede31c9860f7c784b6c616cb5f306bcd21a1005d" exitCode=60 Nov 03 01:00:05.775706 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:00:05.775594 4939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-650be0-29894460-m8w7r" event={"ID":"6ec302ce-4f22-4a05-8200-9525580d6cdf","Type":"ContainerDied","Data":"44e644f7a08209b4cdc4cb6cede31c9860f7c784b6c616cb5f306bcd21a1005d"} Nov 03 01:00:05.776078 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:00:05.776055 4939 scope.go:122] "RemoveContainer" containerID="44e644f7a08209b4cdc4cb6cede31c9860f7c784b6c616cb5f306bcd21a1005d" Nov 03 01:00:05.776403 ip-10-0-140-107 kubenswrapper[4939]: E1103 01:00:05.776367 4939 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w4-650be0-29894460-m8w7r_odh-ai-gateway-infra(6ec302ce-4f22-4a05-8200-9525580d6cdf)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-650be0-29894460-m8w7r" podUID="6ec302ce-4f22-4a05-8200-9525580d6cdf" Nov 03 01:00:05.778034 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:00:05.778002 4939 generic.go:356] "Generic (PLEG): container finished" podID="d5ee2b64-db54-4185-bbcf-3d294be13551" containerID="8765b051546fa5d279071124161acbf49f49cc29876f4a40fd26055fa3330c0a" exitCode=60 Nov 03 01:00:05.778230 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:00:05.778163 4939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-88bbf7-29894460-hqztw" event={"ID":"d5ee2b64-db54-4185-bbcf-3d294be13551","Type":"ContainerDied","Data":"8765b051546fa5d279071124161acbf49f49cc29876f4a40fd26055fa3330c0a"} Nov 03 01:00:05.778355 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:00:05.778338 4939 scope.go:122] "RemoveContainer" containerID="8765b051546fa5d279071124161acbf49f49cc29876f4a40fd26055fa3330c0a" Nov 03 01:00:05.778637 ip-10-0-140-107 kubenswrapper[4939]: E1103 01:00:05.778613 4939 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w1-88bbf7-29894460-hqztw_odh-ai-gateway-infra(d5ee2b64-db54-4185-bbcf-3d294be13551)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-88bbf7-29894460-hqztw" podUID="d5ee2b64-db54-4185-bbcf-3d294be13551" Nov 03 01:00:05.780151 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:00:05.780125 4939 generic.go:356] "Generic (PLEG): container finished" podID="9a1dc28b-dbe2-439e-8f7f-c63f6444de42" containerID="957d817952c82ccccee19bf0d7a80c5866851aff40ebae28904fc73c6ab02966" exitCode=60 Nov 03 01:00:05.780226 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:00:05.780175 4939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-def754-29894460-j7kz4" event={"ID":"9a1dc28b-dbe2-439e-8f7f-c63f6444de42","Type":"ContainerDied","Data":"957d817952c82ccccee19bf0d7a80c5866851aff40ebae28904fc73c6ab02966"} Nov 03 01:00:05.780419 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:00:05.780398 4939 scope.go:122] "RemoveContainer" containerID="957d817952c82ccccee19bf0d7a80c5866851aff40ebae28904fc73c6ab02966" Nov 03 01:00:05.780701 ip-10-0-140-107 kubenswrapper[4939]: E1103 01:00:05.780682 4939 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-ipp-b-def754-29894460-j7kz4_odh-ai-gateway-infra(9a1dc28b-dbe2-439e-8f7f-c63f6444de42)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-def754-29894460-j7kz4" podUID="9a1dc28b-dbe2-439e-8f7f-c63f6444de42" Nov 03 01:00:05.786470 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:00:05.786454 4939 scope.go:122] "RemoveContainer" containerID="32b66444cbc0c4dc6da318b1196b49d8355385e397dd1f6e182c92e64bb93f87" Nov 03 01:00:05.800199 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:00:05.800174 4939 scope.go:122] "RemoveContainer" containerID="bff77f597899599a87613800e7ad414baa9fba4a51c6659cb287339a1ee7bb87" Nov 03 01:00:05.812328 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:00:05.812307 4939 scope.go:122] "RemoveContainer" containerID="773a7864b6e75b779a03ec331dc76bd2d897b0ac218985b2bcbb04f4d11ab112" Nov 03 01:00:05.824290 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:00:05.824269 4939 scope.go:122] "RemoveContainer" containerID="279d810be0c6f32a56b74c2e9d66ee39598c854ac215c68b4d268d7caabba636" Nov 03 01:00:05.874216 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:00:05.874182 4939 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0a9269dd-5b6c-4311-8f78-87028a4a7935" path="/var/lib/kubelet/pods/0a9269dd-5b6c-4311-8f78-87028a4a7935/volumes" Nov 03 01:00:05.874704 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:00:05.874683 4939 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6cc03f20-f3ea-491f-b0be-e3bc1b9c3ac1" path="/var/lib/kubelet/pods/6cc03f20-f3ea-491f-b0be-e3bc1b9c3ac1/volumes" Nov 03 01:00:05.875285 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:00:05.875260 4939 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d102e320-316d-4577-b088-9b39a27e8794" path="/var/lib/kubelet/pods/d102e320-316d-4577-b088-9b39a27e8794/volumes" Nov 03 01:00:06.433149 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:00:06.433090 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:00:06.790325 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:00:06.790233 4939 scope.go:122] "RemoveContainer" containerID="8765b051546fa5d279071124161acbf49f49cc29876f4a40fd26055fa3330c0a" Nov 03 01:00:06.790756 ip-10-0-140-107 kubenswrapper[4939]: E1103 01:00:06.790576 4939 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w1-88bbf7-29894460-hqztw_odh-ai-gateway-infra(d5ee2b64-db54-4185-bbcf-3d294be13551)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-88bbf7-29894460-hqztw" podUID="d5ee2b64-db54-4185-bbcf-3d294be13551" Nov 03 01:00:06.792315 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:00:06.792291 4939 scope.go:122] "RemoveContainer" containerID="957d817952c82ccccee19bf0d7a80c5866851aff40ebae28904fc73c6ab02966" Nov 03 01:00:06.792598 ip-10-0-140-107 kubenswrapper[4939]: E1103 01:00:06.792571 4939 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-ipp-b-def754-29894460-j7kz4_odh-ai-gateway-infra(9a1dc28b-dbe2-439e-8f7f-c63f6444de42)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-def754-29894460-j7kz4" podUID="9a1dc28b-dbe2-439e-8f7f-c63f6444de42" Nov 03 01:00:06.794727 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:00:06.794703 4939 scope.go:122] "RemoveContainer" containerID="bc2ad4555c5de1943c7e3dc8fa16dd28e77bdf84dfad526454916726a15c34ce" Nov 03 01:00:06.794974 ip-10-0-140-107 kubenswrapper[4939]: E1103 01:00:06.794954 4939 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-api-b-47369d-29894460-h5m8c_odh-ai-gateway-infra(370396d8-27e0-4320-b0f4-40134e62e18d)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-47369d-29894460-h5m8c" podUID="370396d8-27e0-4320-b0f4-40134e62e18d" Nov 03 01:00:06.796392 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:00:06.796375 4939 scope.go:122] "RemoveContainer" containerID="3fc762abd3a0645501ccd2889def951ae9a2f1458e068b58d3b427869547084b" Nov 03 01:00:06.796607 ip-10-0-140-107 kubenswrapper[4939]: E1103 01:00:06.796582 4939 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-api-b-ffdea6-29894460-skg2j_odh-ai-gateway-infra(b1e21399-49c6-4226-aa7f-6de60c486c9a)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-ffdea6-29894460-skg2j" podUID="b1e21399-49c6-4226-aa7f-6de60c486c9a" Nov 03 01:00:06.798065 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:00:06.798049 4939 scope.go:122] "RemoveContainer" containerID="44e644f7a08209b4cdc4cb6cede31c9860f7c784b6c616cb5f306bcd21a1005d" Nov 03 01:00:06.798253 ip-10-0-140-107 kubenswrapper[4939]: E1103 01:00:06.798237 4939 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w4-650be0-29894460-m8w7r_odh-ai-gateway-infra(6ec302ce-4f22-4a05-8200-9525580d6cdf)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-650be0-29894460-m8w7r" podUID="6ec302ce-4f22-4a05-8200-9525580d6cdf" Nov 03 01:00:07.284384 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:00:07.284349 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:00:07.994034 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:00:07.994002 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:00:08.814275 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:00:08.814134 4939 generic.go:356] "Generic (PLEG): container finished" podID="8c763d19-e7ff-4638-8d21-3e977502a520" containerID="32da52f1f2bddcb5fdad799643b3a1bfafca75e8e99547378cb528abcbd073b7" exitCode=0 Nov 03 01:00:08.814275 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:00:08.814207 4939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-def754-openshift-default-544466b787-9msx9" event={"ID":"8c763d19-e7ff-4638-8d21-3e977502a520","Type":"ContainerDied","Data":"32da52f1f2bddcb5fdad799643b3a1bfafca75e8e99547378cb528abcbd073b7"} Nov 03 01:00:08.814736 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:00:08.814719 4939 scope.go:122] "RemoveContainer" containerID="32da52f1f2bddcb5fdad799643b3a1bfafca75e8e99547378cb528abcbd073b7" Nov 03 01:00:08.816482 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:00:08.816449 4939 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 01:00:08.816574 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:00:08.816518 4939 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 01:00:08.816574 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:00:08.816551 4939 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 01:00:08.900289 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:00:08.900255 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:00:09.757670 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:00:09.757462 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:00:09.782975 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:00:09.782105 4939 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-b-def754-openshift-default-544466b787-9msx9" Nov 03 01:00:09.861992 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:00:09.856826 4939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-def754-openshift-default-544466b787-9msx9" event={"ID":"8c763d19-e7ff-4638-8d21-3e977502a520","Type":"ContainerStarted","Data":"0580fbf79b774e762d78d096dda18b1f06f7743894ab28892d60afde59dda780"} Nov 03 01:00:10.215787 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:00:10.215748 4939 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-b-def754-openshift-default-544466b787-9msx9" Nov 03 01:00:10.216892 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:00:10.216863 4939 patch_prober.go:29] interesting pod/e2e-ipp-b-def754-openshift-default-544466b787-9msx9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.117:15021/healthz/ready\": dial tcp 10.132.0.117:15021: connect: connection refused" start-of-body= Nov 03 01:00:10.217072 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:00:10.216935 4939 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-def754-openshift-default-544466b787-9msx9" podUID="8c763d19-e7ff-4638-8d21-3e977502a520" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.117:15021/healthz/ready\": dial tcp 10.132.0.117:15021: connect: connection refused" Nov 03 01:00:10.404071 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:00:10.404033 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:00:10.540637 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:00:10.540556 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:00:11.205408 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:00:11.205374 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:00:11.216589 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:00:11.216556 4939 patch_prober.go:29] interesting pod/e2e-ipp-b-def754-openshift-default-544466b787-9msx9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.117:15021/healthz/ready\": dial tcp 10.132.0.117:15021: connect: connection refused" start-of-body= Nov 03 01:00:11.216753 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:00:11.216614 4939 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-def754-openshift-default-544466b787-9msx9" podUID="8c763d19-e7ff-4638-8d21-3e977502a520" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.117:15021/healthz/ready\": dial tcp 10.132.0.117:15021: connect: connection refused" Nov 03 01:00:11.383932 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:00:11.383345 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:00:11.525097 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:00:11.521448 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:00:12.093166 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:00:12.066368 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:00:12.205002 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:00:12.204970 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:00:12.222102 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:00:12.220942 4939 patch_prober.go:29] interesting pod/e2e-ipp-b-def754-openshift-default-544466b787-9msx9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.117:15021/healthz/ready\": dial tcp 10.132.0.117:15021: connect: connection refused" start-of-body= Nov 03 01:00:12.222102 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:00:12.220994 4939 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-def754-openshift-default-544466b787-9msx9" podUID="8c763d19-e7ff-4638-8d21-3e977502a520" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.117:15021/healthz/ready\": dial tcp 10.132.0.117:15021: connect: connection refused" Nov 03 01:00:12.383035 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:00:12.382957 4939 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-f88f87-67d5f888b7-nf6wt"] Nov 03 01:00:12.385543 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:00:12.384395 4939 cpu_manager.go:414] "containerMap: removing container" podUID="d102e320-316d-4577-b088-9b39a27e8794" containerName="maas-api" Nov 03 01:00:12.385728 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:00:12.385710 4939 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d102e320-316d-4577-b088-9b39a27e8794" containerName="maas-api" Nov 03 01:00:12.385877 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:00:12.385834 4939 cpu_manager.go:414] "containerMap: removing container" podUID="6cc03f20-f3ea-491f-b0be-e3bc1b9c3ac1" containerName="payload-pre-processing" Nov 03 01:00:12.385877 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:00:12.385848 4939 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6cc03f20-f3ea-491f-b0be-e3bc1b9c3ac1" containerName="payload-pre-processing" Nov 03 01:00:12.386104 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:00:12.385956 4939 cpu_manager.go:414] "containerMap: removing container" podUID="0a9269dd-5b6c-4311-8f78-87028a4a7935" containerName="payload-processing" Nov 03 01:00:12.386104 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:00:12.385967 4939 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0a9269dd-5b6c-4311-8f78-87028a4a7935" containerName="payload-processing" Nov 03 01:00:12.386229 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:00:12.386150 4939 memory_manager.go:365] "RemoveStaleState removing state" podUID="0a9269dd-5b6c-4311-8f78-87028a4a7935" containerName="payload-processing" Nov 03 01:00:12.386229 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:00:12.386172 4939 memory_manager.go:365] "RemoveStaleState removing state" podUID="d102e320-316d-4577-b088-9b39a27e8794" containerName="maas-api" Nov 03 01:00:12.386229 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:00:12.386187 4939 memory_manager.go:365] "RemoveStaleState removing state" podUID="6cc03f20-f3ea-491f-b0be-e3bc1b9c3ac1" containerName="payload-pre-processing" Nov 03 01:00:12.389968 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:00:12.389902 4939 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-f88f87-67d5f888b7-nf6wt"] Nov 03 01:00:12.390080 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:00:12.390058 4939 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-f88f87-67d5f888b7-nf6wt" Nov 03 01:00:12.396214 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:00:12.396190 4939 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-a-f88f87\"" Nov 03 01:00:12.396550 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:00:12.396520 4939 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-a-f88f87-dockercfg-mb66r\"" Nov 03 01:00:12.397394 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:00:12.397367 4939 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-80f70e-openshift-default-5984948b65-ck87x"] Nov 03 01:00:12.397605 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:00:12.397578 4939 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-api-a-80f70e-openshift-default-5984948b65-ck87x" podUID="bbecba1a-367a-4239-9e6c-fdfb4972ccf1" containerName="istio-proxy" containerID="cri-o://05464c4081c9d2df9c356d204b4e11297f66ffca65156416e56a9032be54e1d6" gracePeriod=30 Nov 03 01:00:12.426673 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:00:12.426602 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:00:12.478895 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:00:12.478859 4939 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-f88f87-7b579c8df6-qvh6t"] Nov 03 01:00:12.484296 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:00:12.484039 4939 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-f88f87-7b579c8df6-qvh6t" Nov 03 01:00:12.484966 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:00:12.484937 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/46a4dd33-f941-48f6-bdb7-fb2c057aa808-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-f88f87-67d5f888b7-nf6wt\" (UID: \"46a4dd33-f941-48f6-bdb7-fb2c057aa808\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-f88f87-67d5f888b7-nf6wt" Nov 03 01:00:12.485201 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:00:12.485179 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qfxhv\" (UniqueName: \"kubernetes.io/projected/46a4dd33-f941-48f6-bdb7-fb2c057aa808-kube-api-access-qfxhv\") pod \"payload-pre-processing-e2e-ipp-a-f88f87-67d5f888b7-nf6wt\" (UID: \"46a4dd33-f941-48f6-bdb7-fb2c057aa808\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-f88f87-67d5f888b7-nf6wt" Nov 03 01:00:12.510765 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:00:12.506534 4939 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-f88f87-7b579c8df6-qvh6t"] Nov 03 01:00:12.600167 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:00:12.592039 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0a6a1482-5067-471b-88f5-7dcbeb5323cc-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-f88f87-7b579c8df6-qvh6t\" (UID: \"0a6a1482-5067-471b-88f5-7dcbeb5323cc\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-f88f87-7b579c8df6-qvh6t" Nov 03 01:00:12.600167 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:00:12.592153 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/46a4dd33-f941-48f6-bdb7-fb2c057aa808-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-f88f87-67d5f888b7-nf6wt\" (UID: \"46a4dd33-f941-48f6-bdb7-fb2c057aa808\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-f88f87-67d5f888b7-nf6wt" Nov 03 01:00:12.600167 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:00:12.592251 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qfxhv\" (UniqueName: \"kubernetes.io/projected/46a4dd33-f941-48f6-bdb7-fb2c057aa808-kube-api-access-qfxhv\") pod \"payload-pre-processing-e2e-ipp-a-f88f87-67d5f888b7-nf6wt\" (UID: \"46a4dd33-f941-48f6-bdb7-fb2c057aa808\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-f88f87-67d5f888b7-nf6wt" Nov 03 01:00:12.600167 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:00:12.592350 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ctpl7\" (UniqueName: \"kubernetes.io/projected/0a6a1482-5067-471b-88f5-7dcbeb5323cc-kube-api-access-ctpl7\") pod \"payload-processing-e2e-ipp-a-f88f87-7b579c8df6-qvh6t\" (UID: \"0a6a1482-5067-471b-88f5-7dcbeb5323cc\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-f88f87-7b579c8df6-qvh6t" Nov 03 01:00:12.600167 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:00:12.593461 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/46a4dd33-f941-48f6-bdb7-fb2c057aa808-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-f88f87-67d5f888b7-nf6wt\" (UID: \"46a4dd33-f941-48f6-bdb7-fb2c057aa808\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-f88f87-67d5f888b7-nf6wt" Nov 03 01:00:12.718987 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:00:12.699000 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ctpl7\" (UniqueName: \"kubernetes.io/projected/0a6a1482-5067-471b-88f5-7dcbeb5323cc-kube-api-access-ctpl7\") pod \"payload-processing-e2e-ipp-a-f88f87-7b579c8df6-qvh6t\" (UID: \"0a6a1482-5067-471b-88f5-7dcbeb5323cc\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-f88f87-7b579c8df6-qvh6t" Nov 03 01:00:12.718987 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:00:12.699327 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0a6a1482-5067-471b-88f5-7dcbeb5323cc-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-f88f87-7b579c8df6-qvh6t\" (UID: \"0a6a1482-5067-471b-88f5-7dcbeb5323cc\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-f88f87-7b579c8df6-qvh6t" Nov 03 01:00:12.733929 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:00:12.725954 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0a6a1482-5067-471b-88f5-7dcbeb5323cc-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-f88f87-7b579c8df6-qvh6t\" (UID: \"0a6a1482-5067-471b-88f5-7dcbeb5323cc\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-f88f87-7b579c8df6-qvh6t" Nov 03 01:00:12.733929 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:00:12.732554 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qfxhv\" (UniqueName: \"kubernetes.io/projected/46a4dd33-f941-48f6-bdb7-fb2c057aa808-kube-api-access-qfxhv\") pod \"payload-pre-processing-e2e-ipp-a-f88f87-67d5f888b7-nf6wt\" (UID: \"46a4dd33-f941-48f6-bdb7-fb2c057aa808\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-f88f87-67d5f888b7-nf6wt" Nov 03 01:00:12.788933 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:00:12.788744 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ctpl7\" (UniqueName: \"kubernetes.io/projected/0a6a1482-5067-471b-88f5-7dcbeb5323cc-kube-api-access-ctpl7\") pod \"payload-processing-e2e-ipp-a-f88f87-7b579c8df6-qvh6t\" (UID: \"0a6a1482-5067-471b-88f5-7dcbeb5323cc\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-f88f87-7b579c8df6-qvh6t" Nov 03 01:00:12.843889 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:00:12.843854 4939 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-f88f87-7b579c8df6-qvh6t" Nov 03 01:00:13.008499 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:00:13.008403 4939 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-f88f87-67d5f888b7-nf6wt" Nov 03 01:00:13.339550 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:00:13.338962 4939 patch_prober.go:29] interesting pod/e2e-ipp-b-def754-openshift-default-544466b787-9msx9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.117:15021/healthz/ready\": dial tcp 10.132.0.117:15021: connect: connection refused" start-of-body= Nov 03 01:00:13.339550 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:00:13.339032 4939 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-def754-openshift-default-544466b787-9msx9" podUID="8c763d19-e7ff-4638-8d21-3e977502a520" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.117:15021/healthz/ready\": dial tcp 10.132.0.117:15021: connect: connection refused" Nov 03 01:00:13.386948 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:00:13.383402 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:00:13.390693 ip-10-0-140-107 kubenswrapper[4939]: W1103 01:00:13.390622 4939 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0a6a1482_5067_471b_88f5_7dcbeb5323cc.slice/crio-cce2d25f4cbbb3d0294237926453b90c235f5ba0f8e40aeabc2ded2bbdddbc74 WatchSource:0}: Error finding container cce2d25f4cbbb3d0294237926453b90c235f5ba0f8e40aeabc2ded2bbdddbc74: Status 404 returned error can't find the container with id cce2d25f4cbbb3d0294237926453b90c235f5ba0f8e40aeabc2ded2bbdddbc74 Nov 03 01:00:13.392897 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:00:13.392866 4939 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-f88f87-7b579c8df6-qvh6t"] Nov 03 01:00:13.403143 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:00:13.403124 4939 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-f88f87-67d5f888b7-nf6wt"] Nov 03 01:00:13.517823 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:00:13.517789 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:00:14.084630 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:00:14.084462 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:00:14.232661 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:00:14.232630 4939 patch_prober.go:29] interesting pod/e2e-ipp-b-def754-openshift-default-544466b787-9msx9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.117:15021/healthz/ready\": dial tcp 10.132.0.117:15021: connect: connection refused" start-of-body= Nov 03 01:00:14.233293 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:00:14.232891 4939 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-def754-openshift-default-544466b787-9msx9" podUID="8c763d19-e7ff-4638-8d21-3e977502a520" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.117:15021/healthz/ready\": dial tcp 10.132.0.117:15021: connect: connection refused" Nov 03 01:00:14.382004 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:00:14.381880 4939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-f88f87-7b579c8df6-qvh6t" event={"ID":"0a6a1482-5067-471b-88f5-7dcbeb5323cc","Type":"ContainerStarted","Data":"cce2d25f4cbbb3d0294237926453b90c235f5ba0f8e40aeabc2ded2bbdddbc74"} Nov 03 01:00:14.383432 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:00:14.383400 4939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-f88f87-67d5f888b7-nf6wt" event={"ID":"46a4dd33-f941-48f6-bdb7-fb2c057aa808","Type":"ContainerStarted","Data":"4bdaf9e9d1b24e8e674f3244320020a0b2c82abdb209116bcbeecd548b98c6a8"} Nov 03 01:00:15.216375 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:00:15.216320 4939 patch_prober.go:29] interesting pod/e2e-ipp-b-def754-openshift-default-544466b787-9msx9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.117:15021/healthz/ready\": dial tcp 10.132.0.117:15021: connect: connection refused" start-of-body= Nov 03 01:00:15.216375 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:00:15.216389 4939 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-def754-openshift-default-544466b787-9msx9" podUID="8c763d19-e7ff-4638-8d21-3e977502a520" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.117:15021/healthz/ready\": dial tcp 10.132.0.117:15021: connect: connection refused" Nov 03 01:00:15.218801 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:00:15.218768 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:00:16.402395 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:00:16.402347 4939 patch_prober.go:29] interesting pod/e2e-api-a-80f70e-openshift-default-5984948b65-ck87x container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 03 01:00:16.402865 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:00:16.402424 4939 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-api-a-80f70e-openshift-default-5984948b65-ck87x" podUID="bbecba1a-367a-4239-9e6c-fdfb4972ccf1" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 03 01:00:16.485884 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:00:16.485848 4939 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-b-def754-openshift-default-544466b787-9msx9" Nov 03 01:00:16.487529 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:00:16.487503 4939 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-b-def754-openshift-default-544466b787-9msx9" Nov 03 01:00:16.488550 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:00:16.488500 4939 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-b-def754-openshift-default-544466b787-9msx9" Nov 03 01:00:16.564109 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:00:16.563994 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:00:16.862036 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:00:16.862004 4939 scope.go:122] "RemoveContainer" containerID="957d817952c82ccccee19bf0d7a80c5866851aff40ebae28904fc73c6ab02966" Nov 03 01:00:17.409787 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:00:17.409754 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:00:17.532412 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:00:17.530952 4939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-f88f87-7b579c8df6-qvh6t" event={"ID":"0a6a1482-5067-471b-88f5-7dcbeb5323cc","Type":"ContainerStarted","Data":"27ca8337f788b4df0699c115ad9a857f1b2b4524c5d71fbad893ff7666bb80ae"} Nov 03 01:00:17.534898 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:00:17.534423 4939 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-f88f87-7b579c8df6-qvh6t" Nov 03 01:00:17.539552 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:00:17.539519 4939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-f88f87-67d5f888b7-nf6wt" event={"ID":"46a4dd33-f941-48f6-bdb7-fb2c057aa808","Type":"ContainerStarted","Data":"bda32303134b23caa4555af4dec7a2b0790f86e74c2e50f2e1846ace30f186a4"} Nov 03 01:00:17.540229 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:00:17.539960 4939 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-f88f87-67d5f888b7-nf6wt" Nov 03 01:00:17.560072 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:00:17.560023 4939 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-a-f88f87-7b579c8df6-qvh6t" podStartSLOduration=1.9816627 podStartE2EDuration="5.560005776s" totalImagesPullingTime="3.578343076s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 01:00:12 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 01:00:17.557709008 +0000 UTC m=+1165.822043953" watchObservedRunningTime="2026-11-03 01:00:17.560005776 +0000 UTC m=+1165.824340722" Nov 03 01:00:17.589654 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:00:17.589590 4939 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-f88f87-67d5f888b7-nf6wt" podStartSLOduration=2.042564995 podStartE2EDuration="5.58956826s" totalImagesPullingTime="3.547003265s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 01:00:12 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 01:00:17.587068757 +0000 UTC m=+1165.851403702" watchObservedRunningTime="2026-11-03 01:00:17.58956826 +0000 UTC m=+1165.853903206" Nov 03 01:00:17.864207 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:00:17.862488 4939 scope.go:122] "RemoveContainer" containerID="8765b051546fa5d279071124161acbf49f49cc29876f4a40fd26055fa3330c0a" Nov 03 01:00:17.864207 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:00:17.863050 4939 scope.go:122] "RemoveContainer" containerID="44e644f7a08209b4cdc4cb6cede31c9860f7c784b6c616cb5f306bcd21a1005d" Nov 03 01:00:17.864207 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:00:17.863541 4939 scope.go:122] "RemoveContainer" containerID="3fc762abd3a0645501ccd2889def951ae9a2f1458e068b58d3b427869547084b" Nov 03 01:00:17.909792 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:00:17.909762 4939 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-80f70e-openshift-default-5984948b65-ck87x" Nov 03 01:00:18.044060 ip-10-0-140-107 kubenswrapper[4939]: E1103 01:00:18.044020 4939 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-sthjh" containerName="extractor" Nov 03 01:00:18.068052 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:00:18.068005 4939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bbecba1a-367a-4239-9e6c-fdfb4972ccf1-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bbecba1a-367a-4239-9e6c-fdfb4972ccf1-workload-socket\") pod \"bbecba1a-367a-4239-9e6c-fdfb4972ccf1\" (UID: \"bbecba1a-367a-4239-9e6c-fdfb4972ccf1\") " Nov 03 01:00:18.068246 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:00:18.068093 4939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bbecba1a-367a-4239-9e6c-fdfb4972ccf1-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bbecba1a-367a-4239-9e6c-fdfb4972ccf1-istiod-ca-cert\") pod \"bbecba1a-367a-4239-9e6c-fdfb4972ccf1\" (UID: \"bbecba1a-367a-4239-9e6c-fdfb4972ccf1\") " Nov 03 01:00:18.068246 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:00:18.068139 4939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bbecba1a-367a-4239-9e6c-fdfb4972ccf1-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bbecba1a-367a-4239-9e6c-fdfb4972ccf1-workload-certs\") pod \"bbecba1a-367a-4239-9e6c-fdfb4972ccf1\" (UID: \"bbecba1a-367a-4239-9e6c-fdfb4972ccf1\") " Nov 03 01:00:18.068246 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:00:18.068220 4939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bbecba1a-367a-4239-9e6c-fdfb4972ccf1-istio-token\" (UniqueName: \"kubernetes.io/projected/bbecba1a-367a-4239-9e6c-fdfb4972ccf1-istio-token\") pod \"bbecba1a-367a-4239-9e6c-fdfb4972ccf1\" (UID: \"bbecba1a-367a-4239-9e6c-fdfb4972ccf1\") " Nov 03 01:00:18.068430 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:00:18.068253 4939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bbecba1a-367a-4239-9e6c-fdfb4972ccf1-kube-api-access-g44q5\" (UniqueName: \"kubernetes.io/projected/bbecba1a-367a-4239-9e6c-fdfb4972ccf1-kube-api-access-g44q5\") pod \"bbecba1a-367a-4239-9e6c-fdfb4972ccf1\" (UID: \"bbecba1a-367a-4239-9e6c-fdfb4972ccf1\") " Nov 03 01:00:18.068430 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:00:18.068343 4939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bbecba1a-367a-4239-9e6c-fdfb4972ccf1-workload-socket" pod "bbecba1a-367a-4239-9e6c-fdfb4972ccf1" (UID: "bbecba1a-367a-4239-9e6c-fdfb4972ccf1"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 01:00:18.068588 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:00:18.068563 4939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bbecba1a-367a-4239-9e6c-fdfb4972ccf1-istiod-ca-cert" pod "bbecba1a-367a-4239-9e6c-fdfb4972ccf1" (UID: "bbecba1a-367a-4239-9e6c-fdfb4972ccf1"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 01:00:18.068676 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:00:18.068656 4939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bbecba1a-367a-4239-9e6c-fdfb4972ccf1-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bbecba1a-367a-4239-9e6c-fdfb4972ccf1-istio-data\") pod \"bbecba1a-367a-4239-9e6c-fdfb4972ccf1\" (UID: \"bbecba1a-367a-4239-9e6c-fdfb4972ccf1\") " Nov 03 01:00:18.068885 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:00:18.068861 4939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/bbecba1a-367a-4239-9e6c-fdfb4972ccf1-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bbecba1a-367a-4239-9e6c-fdfb4972ccf1-istio-podinfo\") pod \"bbecba1a-367a-4239-9e6c-fdfb4972ccf1\" (UID: \"bbecba1a-367a-4239-9e6c-fdfb4972ccf1\") " Nov 03 01:00:18.068989 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:00:18.068877 4939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bbecba1a-367a-4239-9e6c-fdfb4972ccf1-workload-certs" pod "bbecba1a-367a-4239-9e6c-fdfb4972ccf1" (UID: "bbecba1a-367a-4239-9e6c-fdfb4972ccf1"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 01:00:18.068989 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:00:18.068967 4939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bbecba1a-367a-4239-9e6c-fdfb4972ccf1-istio-data" pod "bbecba1a-367a-4239-9e6c-fdfb4972ccf1" (UID: "bbecba1a-367a-4239-9e6c-fdfb4972ccf1"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 01:00:18.069179 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:00:18.069016 4939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bbecba1a-367a-4239-9e6c-fdfb4972ccf1-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bbecba1a-367a-4239-9e6c-fdfb4972ccf1-istio-envoy\") pod \"bbecba1a-367a-4239-9e6c-fdfb4972ccf1\" (UID: \"bbecba1a-367a-4239-9e6c-fdfb4972ccf1\") " Nov 03 01:00:18.069179 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:00:18.069065 4939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bbecba1a-367a-4239-9e6c-fdfb4972ccf1-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bbecba1a-367a-4239-9e6c-fdfb4972ccf1-credential-socket\") pod \"bbecba1a-367a-4239-9e6c-fdfb4972ccf1\" (UID: \"bbecba1a-367a-4239-9e6c-fdfb4972ccf1\") " Nov 03 01:00:18.069957 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:00:18.069449 4939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bbecba1a-367a-4239-9e6c-fdfb4972ccf1-credential-socket" pod "bbecba1a-367a-4239-9e6c-fdfb4972ccf1" (UID: "bbecba1a-367a-4239-9e6c-fdfb4972ccf1"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 01:00:18.069957 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:00:18.069828 4939 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bbecba1a-367a-4239-9e6c-fdfb4972ccf1-credential-socket\") on node \"ip-10-0-140-107.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:00:18.069957 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:00:18.069849 4939 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bbecba1a-367a-4239-9e6c-fdfb4972ccf1-workload-socket\") on node \"ip-10-0-140-107.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:00:18.069957 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:00:18.069864 4939 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bbecba1a-367a-4239-9e6c-fdfb4972ccf1-istiod-ca-cert\") on node \"ip-10-0-140-107.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:00:18.069957 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:00:18.069878 4939 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bbecba1a-367a-4239-9e6c-fdfb4972ccf1-workload-certs\") on node \"ip-10-0-140-107.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:00:18.069957 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:00:18.069892 4939 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bbecba1a-367a-4239-9e6c-fdfb4972ccf1-istio-data\") on node \"ip-10-0-140-107.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:00:18.150332 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:00:18.149986 4939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bbecba1a-367a-4239-9e6c-fdfb4972ccf1-istio-token" pod "bbecba1a-367a-4239-9e6c-fdfb4972ccf1" (UID: "bbecba1a-367a-4239-9e6c-fdfb4972ccf1"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 01:00:18.150332 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:00:18.150109 4939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bbecba1a-367a-4239-9e6c-fdfb4972ccf1-kube-api-access-g44q5" pod "bbecba1a-367a-4239-9e6c-fdfb4972ccf1" (UID: "bbecba1a-367a-4239-9e6c-fdfb4972ccf1"). InnerVolumeSpecName "kube-api-access-g44q5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 01:00:18.150472 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:00:18.150437 4939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/bbecba1a-367a-4239-9e6c-fdfb4972ccf1-istio-podinfo" pod "bbecba1a-367a-4239-9e6c-fdfb4972ccf1" (UID: "bbecba1a-367a-4239-9e6c-fdfb4972ccf1"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 01:00:18.150678 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:00:18.150651 4939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bbecba1a-367a-4239-9e6c-fdfb4972ccf1-istio-envoy" pod "bbecba1a-367a-4239-9e6c-fdfb4972ccf1" (UID: "bbecba1a-367a-4239-9e6c-fdfb4972ccf1"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 01:00:18.171344 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:00:18.171311 4939 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bbecba1a-367a-4239-9e6c-fdfb4972ccf1-istio-token\") on node \"ip-10-0-140-107.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:00:18.171487 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:00:18.171348 4939 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-g44q5\" (UniqueName: \"kubernetes.io/projected/bbecba1a-367a-4239-9e6c-fdfb4972ccf1-kube-api-access-g44q5\") on node \"ip-10-0-140-107.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:00:18.171487 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:00:18.171364 4939 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bbecba1a-367a-4239-9e6c-fdfb4972ccf1-istio-podinfo\") on node \"ip-10-0-140-107.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:00:18.171487 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:00:18.171378 4939 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bbecba1a-367a-4239-9e6c-fdfb4972ccf1-istio-envoy\") on node \"ip-10-0-140-107.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:00:18.271143 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:00:18.271037 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:00:18.552371 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:00:18.552333 4939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-ffdea6-29894460-skg2j" event={"ID":"b1e21399-49c6-4226-aa7f-6de60c486c9a","Type":"ContainerStarted","Data":"4b56d5825b198deb66661f9a2e997b141c28b59ae7d1b6aa968909eab7c44af9"} Nov 03 01:00:18.554650 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:00:18.554620 4939 generic.go:356] "Generic (PLEG): container finished" podID="bbecba1a-367a-4239-9e6c-fdfb4972ccf1" containerID="05464c4081c9d2df9c356d204b4e11297f66ffca65156416e56a9032be54e1d6" exitCode=0 Nov 03 01:00:18.554807 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:00:18.554665 4939 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-80f70e-openshift-default-5984948b65-ck87x" Nov 03 01:00:18.554807 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:00:18.554707 4939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-80f70e-openshift-default-5984948b65-ck87x" event={"ID":"bbecba1a-367a-4239-9e6c-fdfb4972ccf1","Type":"ContainerDied","Data":"05464c4081c9d2df9c356d204b4e11297f66ffca65156416e56a9032be54e1d6"} Nov 03 01:00:18.554807 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:00:18.554770 4939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-80f70e-openshift-default-5984948b65-ck87x" event={"ID":"bbecba1a-367a-4239-9e6c-fdfb4972ccf1","Type":"ContainerDied","Data":"8c320fd95987ad7a5f426ea6351cf908a2a2797548a7bb070c5c2f539a44bccc"} Nov 03 01:00:18.554807 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:00:18.554805 4939 scope.go:122] "RemoveContainer" containerID="05464c4081c9d2df9c356d204b4e11297f66ffca65156416e56a9032be54e1d6" Nov 03 01:00:18.557968 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:00:18.557380 4939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-88bbf7-29894460-hqztw" event={"ID":"d5ee2b64-db54-4185-bbcf-3d294be13551","Type":"ContainerStarted","Data":"10187440eed1674288ca599257a3509168bf1b4b9d6e5b2a4dd06fe2ab7397b2"} Nov 03 01:00:18.560730 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:00:18.560630 4939 generic.go:356] "Generic (PLEG): container finished" podID="9a1dc28b-dbe2-439e-8f7f-c63f6444de42" containerID="e6434dd9257751376fda22b419e65cbb262977521a405f353d43d5dc970c9477" exitCode=60 Nov 03 01:00:18.562263 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:00:18.562239 4939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-def754-29894460-j7kz4" event={"ID":"9a1dc28b-dbe2-439e-8f7f-c63f6444de42","Type":"ContainerDied","Data":"e6434dd9257751376fda22b419e65cbb262977521a405f353d43d5dc970c9477"} Nov 03 01:00:18.563361 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:00:18.563338 4939 scope.go:122] "RemoveContainer" containerID="e6434dd9257751376fda22b419e65cbb262977521a405f353d43d5dc970c9477" Nov 03 01:00:18.564347 ip-10-0-140-107 kubenswrapper[4939]: E1103 01:00:18.564216 4939 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-ipp-b-def754-29894460-j7kz4_odh-ai-gateway-infra(9a1dc28b-dbe2-439e-8f7f-c63f6444de42)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-def754-29894460-j7kz4" podUID="9a1dc28b-dbe2-439e-8f7f-c63f6444de42" Nov 03 01:00:18.570072 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:00:18.570055 4939 scope.go:122] "RemoveContainer" containerID="9213c015a67724a51db25ba1367b7f35a54128dcbc010e1f9c06ccb7aa2a6556" Nov 03 01:00:18.584156 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:00:18.584128 4939 scope.go:122] "RemoveContainer" containerID="05464c4081c9d2df9c356d204b4e11297f66ffca65156416e56a9032be54e1d6" Nov 03 01:00:18.584507 ip-10-0-140-107 kubenswrapper[4939]: E1103 01:00:18.584483 4939 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"05464c4081c9d2df9c356d204b4e11297f66ffca65156416e56a9032be54e1d6\": container with ID starting with 05464c4081c9d2df9c356d204b4e11297f66ffca65156416e56a9032be54e1d6 not found: ID does not exist" containerID="05464c4081c9d2df9c356d204b4e11297f66ffca65156416e56a9032be54e1d6" Nov 03 01:00:18.584595 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:00:18.584516 4939 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"05464c4081c9d2df9c356d204b4e11297f66ffca65156416e56a9032be54e1d6"} err="failed to get container status \"05464c4081c9d2df9c356d204b4e11297f66ffca65156416e56a9032be54e1d6\": rpc error: code = NotFound desc = could not find container \"05464c4081c9d2df9c356d204b4e11297f66ffca65156416e56a9032be54e1d6\": container with ID starting with 05464c4081c9d2df9c356d204b4e11297f66ffca65156416e56a9032be54e1d6 not found: ID does not exist" Nov 03 01:00:18.584595 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:00:18.584534 4939 scope.go:122] "RemoveContainer" containerID="9213c015a67724a51db25ba1367b7f35a54128dcbc010e1f9c06ccb7aa2a6556" Nov 03 01:00:18.584813 ip-10-0-140-107 kubenswrapper[4939]: E1103 01:00:18.584796 4939 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9213c015a67724a51db25ba1367b7f35a54128dcbc010e1f9c06ccb7aa2a6556\": container with ID starting with 9213c015a67724a51db25ba1367b7f35a54128dcbc010e1f9c06ccb7aa2a6556 not found: ID does not exist" containerID="9213c015a67724a51db25ba1367b7f35a54128dcbc010e1f9c06ccb7aa2a6556" Nov 03 01:00:18.584854 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:00:18.584818 4939 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9213c015a67724a51db25ba1367b7f35a54128dcbc010e1f9c06ccb7aa2a6556"} err="failed to get container status \"9213c015a67724a51db25ba1367b7f35a54128dcbc010e1f9c06ccb7aa2a6556\": rpc error: code = NotFound desc = could not find container \"9213c015a67724a51db25ba1367b7f35a54128dcbc010e1f9c06ccb7aa2a6556\": container with ID starting with 9213c015a67724a51db25ba1367b7f35a54128dcbc010e1f9c06ccb7aa2a6556 not found: ID does not exist" Nov 03 01:00:18.584854 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:00:18.584832 4939 scope.go:122] "RemoveContainer" containerID="957d817952c82ccccee19bf0d7a80c5866851aff40ebae28904fc73c6ab02966" Nov 03 01:00:18.628457 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:00:18.628426 4939 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-80f70e-openshift-default-5984948b65-ck87x"] Nov 03 01:00:18.628600 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:00:18.628472 4939 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-a-80f70e-openshift-default-5984948b65-ck87x"] Nov 03 01:00:18.860798 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:00:18.860767 4939 scope.go:122] "RemoveContainer" containerID="bc2ad4555c5de1943c7e3dc8fa16dd28e77bdf84dfad526454916726a15c34ce" Nov 03 01:00:18.932931 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:00:18.929355 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:00:19.571969 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:00:19.571864 4939 generic.go:356] "Generic (PLEG): container finished" podID="b1e21399-49c6-4226-aa7f-6de60c486c9a" containerID="4b56d5825b198deb66661f9a2e997b141c28b59ae7d1b6aa968909eab7c44af9" exitCode=60 Nov 03 01:00:19.571969 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:00:19.571940 4939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-ffdea6-29894460-skg2j" event={"ID":"b1e21399-49c6-4226-aa7f-6de60c486c9a","Type":"ContainerDied","Data":"4b56d5825b198deb66661f9a2e997b141c28b59ae7d1b6aa968909eab7c44af9"} Nov 03 01:00:19.572545 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:00:19.572009 4939 scope.go:122] "RemoveContainer" containerID="3fc762abd3a0645501ccd2889def951ae9a2f1458e068b58d3b427869547084b" Nov 03 01:00:19.572545 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:00:19.572454 4939 scope.go:122] "RemoveContainer" containerID="4b56d5825b198deb66661f9a2e997b141c28b59ae7d1b6aa968909eab7c44af9" Nov 03 01:00:19.572890 ip-10-0-140-107 kubenswrapper[4939]: E1103 01:00:19.572865 4939 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-api-b-ffdea6-29894460-skg2j_odh-ai-gateway-infra(b1e21399-49c6-4226-aa7f-6de60c486c9a)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-ffdea6-29894460-skg2j" podUID="b1e21399-49c6-4226-aa7f-6de60c486c9a" Nov 03 01:00:19.575222 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:00:19.575199 4939 generic.go:356] "Generic (PLEG): container finished" podID="6ec302ce-4f22-4a05-8200-9525580d6cdf" containerID="4a532fbd070e6032d4307cdb93c6e99fac6a3937e439e8163e525c42463aecd5" exitCode=60 Nov 03 01:00:19.575360 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:00:19.575241 4939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-650be0-29894460-m8w7r" event={"ID":"6ec302ce-4f22-4a05-8200-9525580d6cdf","Type":"ContainerDied","Data":"4a532fbd070e6032d4307cdb93c6e99fac6a3937e439e8163e525c42463aecd5"} Nov 03 01:00:19.575741 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:00:19.575719 4939 scope.go:122] "RemoveContainer" containerID="4a532fbd070e6032d4307cdb93c6e99fac6a3937e439e8163e525c42463aecd5" Nov 03 01:00:19.576158 ip-10-0-140-107 kubenswrapper[4939]: E1103 01:00:19.576134 4939 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w4-650be0-29894460-m8w7r_odh-ai-gateway-infra(6ec302ce-4f22-4a05-8200-9525580d6cdf)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-650be0-29894460-m8w7r" podUID="6ec302ce-4f22-4a05-8200-9525580d6cdf" Nov 03 01:00:19.578556 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:00:19.578536 4939 generic.go:356] "Generic (PLEG): container finished" podID="d5ee2b64-db54-4185-bbcf-3d294be13551" containerID="10187440eed1674288ca599257a3509168bf1b4b9d6e5b2a4dd06fe2ab7397b2" exitCode=60 Nov 03 01:00:19.578650 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:00:19.578605 4939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-88bbf7-29894460-hqztw" event={"ID":"d5ee2b64-db54-4185-bbcf-3d294be13551","Type":"ContainerDied","Data":"10187440eed1674288ca599257a3509168bf1b4b9d6e5b2a4dd06fe2ab7397b2"} Nov 03 01:00:19.579076 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:00:19.579058 4939 scope.go:122] "RemoveContainer" containerID="10187440eed1674288ca599257a3509168bf1b4b9d6e5b2a4dd06fe2ab7397b2" Nov 03 01:00:19.579414 ip-10-0-140-107 kubenswrapper[4939]: E1103 01:00:19.579392 4939 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w1-88bbf7-29894460-hqztw_odh-ai-gateway-infra(d5ee2b64-db54-4185-bbcf-3d294be13551)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-88bbf7-29894460-hqztw" podUID="d5ee2b64-db54-4185-bbcf-3d294be13551" Nov 03 01:00:19.585225 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:00:19.585201 4939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-47369d-29894460-h5m8c" event={"ID":"370396d8-27e0-4320-b0f4-40134e62e18d","Type":"ContainerStarted","Data":"1f09fb83f6ad1d26a166faa09a39d314f9c1b189a182dec3f33d0d9bdd24eb80"} Nov 03 01:00:19.585328 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:00:19.585243 4939 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-ffdea6-29894460-skg2j"] Nov 03 01:00:19.587558 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:00:19.587542 4939 scope.go:122] "RemoveContainer" containerID="44e644f7a08209b4cdc4cb6cede31c9860f7c784b6c616cb5f306bcd21a1005d" Nov 03 01:00:19.602816 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:00:19.602531 4939 scope.go:122] "RemoveContainer" containerID="8765b051546fa5d279071124161acbf49f49cc29876f4a40fd26055fa3330c0a" Nov 03 01:00:19.607262 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:00:19.607238 4939 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-def754-29894460-j7kz4"] Nov 03 01:00:19.625352 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:00:19.625289 4939 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-88bbf7-29894460-hqztw"] Nov 03 01:00:19.815350 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:00:19.815313 4939 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-def754-29894460-j7kz4" Nov 03 01:00:19.876397 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:00:19.876362 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:00:19.879653 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:00:19.879619 4939 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bbecba1a-367a-4239-9e6c-fdfb4972ccf1" path="/var/lib/kubelet/pods/bbecba1a-367a-4239-9e6c-fdfb4972ccf1/volumes" Nov 03 01:00:19.962016 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:00:19.947608 4939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9a1dc28b-dbe2-439e-8f7f-c63f6444de42-kube-api-access-xsscr\" (UniqueName: \"kubernetes.io/projected/9a1dc28b-dbe2-439e-8f7f-c63f6444de42-kube-api-access-xsscr\") pod \"9a1dc28b-dbe2-439e-8f7f-c63f6444de42\" (UID: \"9a1dc28b-dbe2-439e-8f7f-c63f6444de42\") " Nov 03 01:00:19.962372 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:00:19.962345 4939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9a1dc28b-dbe2-439e-8f7f-c63f6444de42-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9a1dc28b-dbe2-439e-8f7f-c63f6444de42-maas-api-service-ca\") pod \"9a1dc28b-dbe2-439e-8f7f-c63f6444de42\" (UID: \"9a1dc28b-dbe2-439e-8f7f-c63f6444de42\") " Nov 03 01:00:19.979949 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:00:19.968163 4939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9a1dc28b-dbe2-439e-8f7f-c63f6444de42-maas-api-service-ca" pod "9a1dc28b-dbe2-439e-8f7f-c63f6444de42" (UID: "9a1dc28b-dbe2-439e-8f7f-c63f6444de42"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 01:00:20.011940 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:00:19.997456 4939 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9a1dc28b-dbe2-439e-8f7f-c63f6444de42-maas-api-service-ca\") on node \"ip-10-0-140-107.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:00:20.019428 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:00:20.018589 4939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9a1dc28b-dbe2-439e-8f7f-c63f6444de42-kube-api-access-xsscr" pod "9a1dc28b-dbe2-439e-8f7f-c63f6444de42" (UID: "9a1dc28b-dbe2-439e-8f7f-c63f6444de42"). InnerVolumeSpecName "kube-api-access-xsscr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 01:00:20.075847 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:00:20.075803 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:00:20.098839 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:00:20.098807 4939 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xsscr\" (UniqueName: \"kubernetes.io/projected/9a1dc28b-dbe2-439e-8f7f-c63f6444de42-kube-api-access-xsscr\") on node \"ip-10-0-140-107.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:00:20.255457 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:00:20.255370 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:00:20.408061 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:00:20.407971 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:00:20.639445 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:00:20.626738 4939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-def754-29894460-j7kz4" event={"ID":"9a1dc28b-dbe2-439e-8f7f-c63f6444de42","Type":"ContainerDied","Data":"0bd86e1ba00c1609eacf46abd52c0b6b7a491711e1a4223844e96cbdc228d610"} Nov 03 01:00:20.639445 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:00:20.626804 4939 scope.go:122] "RemoveContainer" containerID="e6434dd9257751376fda22b419e65cbb262977521a405f353d43d5dc970c9477" Nov 03 01:00:20.639445 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:00:20.628126 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:00:20.639445 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:00:20.631251 4939 generic.go:356] "Generic (PLEG): container finished" podID="370396d8-27e0-4320-b0f4-40134e62e18d" containerID="1f09fb83f6ad1d26a166faa09a39d314f9c1b189a182dec3f33d0d9bdd24eb80" exitCode=60 Nov 03 01:00:20.639445 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:00:20.631452 4939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-47369d-29894460-h5m8c" event={"ID":"370396d8-27e0-4320-b0f4-40134e62e18d","Type":"ContainerDied","Data":"1f09fb83f6ad1d26a166faa09a39d314f9c1b189a182dec3f33d0d9bdd24eb80"} Nov 03 01:00:20.639445 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:00:20.631899 4939 scope.go:122] "RemoveContainer" containerID="1f09fb83f6ad1d26a166faa09a39d314f9c1b189a182dec3f33d0d9bdd24eb80" Nov 03 01:00:20.639445 ip-10-0-140-107 kubenswrapper[4939]: E1103 01:00:20.632317 4939 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-api-b-47369d-29894460-h5m8c_odh-ai-gateway-infra(370396d8-27e0-4320-b0f4-40134e62e18d)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-47369d-29894460-h5m8c" podUID="370396d8-27e0-4320-b0f4-40134e62e18d" Nov 03 01:00:20.639445 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:00:20.637163 4939 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-def754-29894460-j7kz4" Nov 03 01:00:20.721993 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:00:20.721953 4939 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-650be0-29894460-m8w7r"] Nov 03 01:00:20.784204 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:00:20.781662 4939 scope.go:122] "RemoveContainer" containerID="bc2ad4555c5de1943c7e3dc8fa16dd28e77bdf84dfad526454916726a15c34ce" Nov 03 01:00:20.784204 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:00:20.782994 4939 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-47369d-29894460-h5m8c"] Nov 03 01:00:20.813767 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:00:20.813737 4939 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-def754-29894460-j7kz4"] Nov 03 01:00:20.831175 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:00:20.831138 4939 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-def754-29894460-j7kz4"] Nov 03 01:00:20.988080 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:00:20.987266 4939 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-650be0-29894460-m8w7r" Nov 03 01:00:21.139820 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:00:21.139778 4939 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-ffdea6-29894460-skg2j" Nov 03 01:00:21.143246 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:00:21.143226 4939 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-88bbf7-29894460-hqztw" Nov 03 01:00:21.192013 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:00:21.191892 4939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b1e21399-49c6-4226-aa7f-6de60c486c9a-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b1e21399-49c6-4226-aa7f-6de60c486c9a-maas-api-service-ca\") pod \"b1e21399-49c6-4226-aa7f-6de60c486c9a\" (UID: \"b1e21399-49c6-4226-aa7f-6de60c486c9a\") " Nov 03 01:00:21.192013 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:00:21.191986 4939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d5ee2b64-db54-4185-bbcf-3d294be13551-kube-api-access-bmhss\" (UniqueName: \"kubernetes.io/projected/d5ee2b64-db54-4185-bbcf-3d294be13551-kube-api-access-bmhss\") pod \"d5ee2b64-db54-4185-bbcf-3d294be13551\" (UID: \"d5ee2b64-db54-4185-bbcf-3d294be13551\") " Nov 03 01:00:21.192338 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:00:21.192105 4939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6ec302ce-4f22-4a05-8200-9525580d6cdf-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6ec302ce-4f22-4a05-8200-9525580d6cdf-maas-api-service-ca\") pod \"6ec302ce-4f22-4a05-8200-9525580d6cdf\" (UID: \"6ec302ce-4f22-4a05-8200-9525580d6cdf\") " Nov 03 01:00:21.192338 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:00:21.192162 4939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d5ee2b64-db54-4185-bbcf-3d294be13551-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d5ee2b64-db54-4185-bbcf-3d294be13551-maas-api-service-ca\") pod \"d5ee2b64-db54-4185-bbcf-3d294be13551\" (UID: \"d5ee2b64-db54-4185-bbcf-3d294be13551\") " Nov 03 01:00:21.192338 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:00:21.192304 4939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b1e21399-49c6-4226-aa7f-6de60c486c9a-maas-api-service-ca" pod "b1e21399-49c6-4226-aa7f-6de60c486c9a" (UID: "b1e21399-49c6-4226-aa7f-6de60c486c9a"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 01:00:21.192773 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:00:21.192744 4939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b1e21399-49c6-4226-aa7f-6de60c486c9a-kube-api-access-brdqq\" (UniqueName: \"kubernetes.io/projected/b1e21399-49c6-4226-aa7f-6de60c486c9a-kube-api-access-brdqq\") pod \"b1e21399-49c6-4226-aa7f-6de60c486c9a\" (UID: \"b1e21399-49c6-4226-aa7f-6de60c486c9a\") " Nov 03 01:00:21.192988 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:00:21.192963 4939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6ec302ce-4f22-4a05-8200-9525580d6cdf-maas-api-service-ca" pod "6ec302ce-4f22-4a05-8200-9525580d6cdf" (UID: "6ec302ce-4f22-4a05-8200-9525580d6cdf"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 01:00:21.193118 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:00:21.193068 4939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6ec302ce-4f22-4a05-8200-9525580d6cdf-kube-api-access-xkvb2\" (UniqueName: \"kubernetes.io/projected/6ec302ce-4f22-4a05-8200-9525580d6cdf-kube-api-access-xkvb2\") pod \"6ec302ce-4f22-4a05-8200-9525580d6cdf\" (UID: \"6ec302ce-4f22-4a05-8200-9525580d6cdf\") " Nov 03 01:00:21.193296 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:00:21.193275 4939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d5ee2b64-db54-4185-bbcf-3d294be13551-maas-api-service-ca" pod "d5ee2b64-db54-4185-bbcf-3d294be13551" (UID: "d5ee2b64-db54-4185-bbcf-3d294be13551"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 01:00:21.193809 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:00:21.193784 4939 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b1e21399-49c6-4226-aa7f-6de60c486c9a-maas-api-service-ca\") on node \"ip-10-0-140-107.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:00:21.194048 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:00:21.193981 4939 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6ec302ce-4f22-4a05-8200-9525580d6cdf-maas-api-service-ca\") on node \"ip-10-0-140-107.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:00:21.194048 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:00:21.194003 4939 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d5ee2b64-db54-4185-bbcf-3d294be13551-maas-api-service-ca\") on node \"ip-10-0-140-107.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:00:21.195681 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:00:21.195647 4939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b1e21399-49c6-4226-aa7f-6de60c486c9a-kube-api-access-brdqq" pod "b1e21399-49c6-4226-aa7f-6de60c486c9a" (UID: "b1e21399-49c6-4226-aa7f-6de60c486c9a"). InnerVolumeSpecName "kube-api-access-brdqq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 01:00:21.196170 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:00:21.196143 4939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d5ee2b64-db54-4185-bbcf-3d294be13551-kube-api-access-bmhss" pod "d5ee2b64-db54-4185-bbcf-3d294be13551" (UID: "d5ee2b64-db54-4185-bbcf-3d294be13551"). InnerVolumeSpecName "kube-api-access-bmhss". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 01:00:21.198084 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:00:21.198058 4939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6ec302ce-4f22-4a05-8200-9525580d6cdf-kube-api-access-xkvb2" pod "6ec302ce-4f22-4a05-8200-9525580d6cdf" (UID: "6ec302ce-4f22-4a05-8200-9525580d6cdf"). InnerVolumeSpecName "kube-api-access-xkvb2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 01:00:21.270939 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:00:21.270899 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:00:21.295158 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:00:21.295118 4939 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-brdqq\" (UniqueName: \"kubernetes.io/projected/b1e21399-49c6-4226-aa7f-6de60c486c9a-kube-api-access-brdqq\") on node \"ip-10-0-140-107.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:00:21.295158 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:00:21.295147 4939 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xkvb2\" (UniqueName: \"kubernetes.io/projected/6ec302ce-4f22-4a05-8200-9525580d6cdf-kube-api-access-xkvb2\") on node \"ip-10-0-140-107.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:00:21.295158 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:00:21.295157 4939 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bmhss\" (UniqueName: \"kubernetes.io/projected/d5ee2b64-db54-4185-bbcf-3d294be13551-kube-api-access-bmhss\") on node \"ip-10-0-140-107.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:00:21.435484 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:00:21.435445 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:00:21.686765 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:00:21.686714 4939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-ffdea6-29894460-skg2j" event={"ID":"b1e21399-49c6-4226-aa7f-6de60c486c9a","Type":"ContainerDied","Data":"1f37a36783734982e299f86768f5b7f578a91fb584af58be65d153012fac4a89"} Nov 03 01:00:21.686765 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:00:21.686744 4939 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-ffdea6-29894460-skg2j" Nov 03 01:00:21.687374 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:00:21.686804 4939 scope.go:122] "RemoveContainer" containerID="4b56d5825b198deb66661f9a2e997b141c28b59ae7d1b6aa968909eab7c44af9" Nov 03 01:00:21.688864 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:00:21.688842 4939 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-650be0-29894460-m8w7r" Nov 03 01:00:21.689188 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:00:21.688850 4939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-650be0-29894460-m8w7r" event={"ID":"6ec302ce-4f22-4a05-8200-9525580d6cdf","Type":"ContainerDied","Data":"a8d104cfa0db11e27139f9eb9d5cc79760892c6c22f9fe3ff2889e96eb7f73c4"} Nov 03 01:00:21.690622 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:00:21.690592 4939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-88bbf7-29894460-hqztw" event={"ID":"d5ee2b64-db54-4185-bbcf-3d294be13551","Type":"ContainerDied","Data":"1c631c2c4f5c23fa4d075ae2dd2a8c02983164b1470e38d06b0571bac4bd4446"} Nov 03 01:00:21.690740 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:00:21.690690 4939 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-88bbf7-29894460-hqztw" Nov 03 01:00:21.702989 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:00:21.702970 4939 scope.go:122] "RemoveContainer" containerID="4a532fbd070e6032d4307cdb93c6e99fac6a3937e439e8163e525c42463aecd5" Nov 03 01:00:21.714634 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:00:21.714616 4939 scope.go:122] "RemoveContainer" containerID="10187440eed1674288ca599257a3509168bf1b4b9d6e5b2a4dd06fe2ab7397b2" Nov 03 01:00:21.737076 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:00:21.737045 4939 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-88bbf7-29894460-hqztw"] Nov 03 01:00:21.743150 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:00:21.743123 4939 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-88bbf7-29894460-hqztw"] Nov 03 01:00:21.755698 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:00:21.755632 4939 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-ffdea6-29894460-skg2j"] Nov 03 01:00:21.760163 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:00:21.760138 4939 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-ffdea6-29894460-skg2j"] Nov 03 01:00:21.774660 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:00:21.774616 4939 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-650be0-29894460-m8w7r"] Nov 03 01:00:21.796814 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:00:21.794772 4939 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-650be0-29894460-m8w7r"] Nov 03 01:00:21.878003 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:00:21.877969 4939 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6ec302ce-4f22-4a05-8200-9525580d6cdf" path="/var/lib/kubelet/pods/6ec302ce-4f22-4a05-8200-9525580d6cdf/volumes" Nov 03 01:00:21.878748 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:00:21.878724 4939 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9a1dc28b-dbe2-439e-8f7f-c63f6444de42" path="/var/lib/kubelet/pods/9a1dc28b-dbe2-439e-8f7f-c63f6444de42/volumes" Nov 03 01:00:21.879376 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:00:21.879354 4939 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b1e21399-49c6-4226-aa7f-6de60c486c9a" path="/var/lib/kubelet/pods/b1e21399-49c6-4226-aa7f-6de60c486c9a/volumes" Nov 03 01:00:21.880550 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:00:21.880527 4939 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d5ee2b64-db54-4185-bbcf-3d294be13551" path="/var/lib/kubelet/pods/d5ee2b64-db54-4185-bbcf-3d294be13551/volumes" Nov 03 01:00:21.903321 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:00:21.903296 4939 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-47369d-29894460-h5m8c" Nov 03 01:00:22.011736 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:00:22.011652 4939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/370396d8-27e0-4320-b0f4-40134e62e18d-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/370396d8-27e0-4320-b0f4-40134e62e18d-maas-api-service-ca\") pod \"370396d8-27e0-4320-b0f4-40134e62e18d\" (UID: \"370396d8-27e0-4320-b0f4-40134e62e18d\") " Nov 03 01:00:22.011888 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:00:22.011736 4939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/370396d8-27e0-4320-b0f4-40134e62e18d-kube-api-access-cs7bg\" (UniqueName: \"kubernetes.io/projected/370396d8-27e0-4320-b0f4-40134e62e18d-kube-api-access-cs7bg\") pod \"370396d8-27e0-4320-b0f4-40134e62e18d\" (UID: \"370396d8-27e0-4320-b0f4-40134e62e18d\") " Nov 03 01:00:22.012092 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:00:22.012067 4939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/370396d8-27e0-4320-b0f4-40134e62e18d-maas-api-service-ca" pod "370396d8-27e0-4320-b0f4-40134e62e18d" (UID: "370396d8-27e0-4320-b0f4-40134e62e18d"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 01:00:22.012533 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:00:22.012512 4939 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/370396d8-27e0-4320-b0f4-40134e62e18d-maas-api-service-ca\") on node \"ip-10-0-140-107.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:00:22.013992 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:00:22.013968 4939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/370396d8-27e0-4320-b0f4-40134e62e18d-kube-api-access-cs7bg" pod "370396d8-27e0-4320-b0f4-40134e62e18d" (UID: "370396d8-27e0-4320-b0f4-40134e62e18d"). InnerVolumeSpecName "kube-api-access-cs7bg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 01:00:22.114190 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:00:22.114150 4939 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cs7bg\" (UniqueName: \"kubernetes.io/projected/370396d8-27e0-4320-b0f4-40134e62e18d-kube-api-access-cs7bg\") on node \"ip-10-0-140-107.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:00:22.245976 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:00:22.245935 4939 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-ipp-a-07f5f6/ipp-route-07f5f6-kserve-87874869c-4xwpw"] Nov 03 01:00:22.247094 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:00:22.247064 4939 cpu_manager.go:414] "containerMap: removing container" podUID="6ec302ce-4f22-4a05-8200-9525580d6cdf" containerName="cleanup" Nov 03 01:00:22.247094 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:00:22.247089 4939 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6ec302ce-4f22-4a05-8200-9525580d6cdf" containerName="cleanup" Nov 03 01:00:22.247302 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:00:22.247114 4939 cpu_manager.go:414] "containerMap: removing container" podUID="370396d8-27e0-4320-b0f4-40134e62e18d" containerName="cleanup" Nov 03 01:00:22.247302 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:00:22.247123 4939 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="370396d8-27e0-4320-b0f4-40134e62e18d" containerName="cleanup" Nov 03 01:00:22.247302 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:00:22.247146 4939 cpu_manager.go:414] "containerMap: removing container" podUID="370396d8-27e0-4320-b0f4-40134e62e18d" containerName="cleanup" Nov 03 01:00:22.247302 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:00:22.247153 4939 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="370396d8-27e0-4320-b0f4-40134e62e18d" containerName="cleanup" Nov 03 01:00:22.247302 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:00:22.247170 4939 cpu_manager.go:414] "containerMap: removing container" podUID="d5ee2b64-db54-4185-bbcf-3d294be13551" containerName="cleanup" Nov 03 01:00:22.247302 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:00:22.247178 4939 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d5ee2b64-db54-4185-bbcf-3d294be13551" containerName="cleanup" Nov 03 01:00:22.247302 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:00:22.247193 4939 cpu_manager.go:414] "containerMap: removing container" podUID="b1e21399-49c6-4226-aa7f-6de60c486c9a" containerName="cleanup" Nov 03 01:00:22.247302 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:00:22.247200 4939 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b1e21399-49c6-4226-aa7f-6de60c486c9a" containerName="cleanup" Nov 03 01:00:22.247302 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:00:22.247211 4939 cpu_manager.go:414] "containerMap: removing container" podUID="d5ee2b64-db54-4185-bbcf-3d294be13551" containerName="cleanup" Nov 03 01:00:22.247302 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:00:22.247218 4939 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d5ee2b64-db54-4185-bbcf-3d294be13551" containerName="cleanup" Nov 03 01:00:22.247302 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:00:22.247245 4939 cpu_manager.go:414] "containerMap: removing container" podUID="bbecba1a-367a-4239-9e6c-fdfb4972ccf1" containerName="istio-proxy" Nov 03 01:00:22.247302 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:00:22.247252 4939 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bbecba1a-367a-4239-9e6c-fdfb4972ccf1" containerName="istio-proxy" Nov 03 01:00:22.247302 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:00:22.247274 4939 cpu_manager.go:414] "containerMap: removing container" podUID="9a1dc28b-dbe2-439e-8f7f-c63f6444de42" containerName="cleanup" Nov 03 01:00:22.247302 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:00:22.247281 4939 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9a1dc28b-dbe2-439e-8f7f-c63f6444de42" containerName="cleanup" Nov 03 01:00:22.247302 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:00:22.247290 4939 cpu_manager.go:414] "containerMap: removing container" podUID="b1e21399-49c6-4226-aa7f-6de60c486c9a" containerName="cleanup" Nov 03 01:00:22.247302 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:00:22.247297 4939 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b1e21399-49c6-4226-aa7f-6de60c486c9a" containerName="cleanup" Nov 03 01:00:22.247302 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:00:22.247306 4939 cpu_manager.go:414] "containerMap: removing container" podUID="6ec302ce-4f22-4a05-8200-9525580d6cdf" containerName="cleanup" Nov 03 01:00:22.249509 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:00:22.247313 4939 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6ec302ce-4f22-4a05-8200-9525580d6cdf" containerName="cleanup" Nov 03 01:00:22.249509 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:00:22.247354 4939 cpu_manager.go:414] "containerMap: removing container" podUID="9a1dc28b-dbe2-439e-8f7f-c63f6444de42" containerName="cleanup" Nov 03 01:00:22.249509 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:00:22.247361 4939 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9a1dc28b-dbe2-439e-8f7f-c63f6444de42" containerName="cleanup" Nov 03 01:00:22.249509 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:00:22.247375 4939 cpu_manager.go:414] "containerMap: removing container" podUID="bbecba1a-367a-4239-9e6c-fdfb4972ccf1" containerName="istio-proxy" Nov 03 01:00:22.249509 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:00:22.247384 4939 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bbecba1a-367a-4239-9e6c-fdfb4972ccf1" containerName="istio-proxy" Nov 03 01:00:22.249509 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:00:22.247396 4939 cpu_manager.go:414] "containerMap: removing container" podUID="9a1dc28b-dbe2-439e-8f7f-c63f6444de42" containerName="cleanup" Nov 03 01:00:22.249509 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:00:22.247404 4939 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9a1dc28b-dbe2-439e-8f7f-c63f6444de42" containerName="cleanup" Nov 03 01:00:22.249509 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:00:22.247428 4939 cpu_manager.go:414] "containerMap: removing container" podUID="d5ee2b64-db54-4185-bbcf-3d294be13551" containerName="cleanup" Nov 03 01:00:22.249509 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:00:22.247435 4939 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d5ee2b64-db54-4185-bbcf-3d294be13551" containerName="cleanup" Nov 03 01:00:22.249509 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:00:22.248197 4939 memory_manager.go:365] "RemoveStaleState removing state" podUID="370396d8-27e0-4320-b0f4-40134e62e18d" containerName="cleanup" Nov 03 01:00:22.249509 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:00:22.248216 4939 memory_manager.go:365] "RemoveStaleState removing state" podUID="6ec302ce-4f22-4a05-8200-9525580d6cdf" containerName="cleanup" Nov 03 01:00:22.249509 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:00:22.248223 4939 memory_manager.go:365] "RemoveStaleState removing state" podUID="370396d8-27e0-4320-b0f4-40134e62e18d" containerName="cleanup" Nov 03 01:00:22.249509 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:00:22.248231 4939 memory_manager.go:365] "RemoveStaleState removing state" podUID="370396d8-27e0-4320-b0f4-40134e62e18d" containerName="cleanup" Nov 03 01:00:22.249509 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:00:22.248237 4939 memory_manager.go:365] "RemoveStaleState removing state" podUID="d5ee2b64-db54-4185-bbcf-3d294be13551" containerName="cleanup" Nov 03 01:00:22.249509 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:00:22.248245 4939 memory_manager.go:365] "RemoveStaleState removing state" podUID="b1e21399-49c6-4226-aa7f-6de60c486c9a" containerName="cleanup" Nov 03 01:00:22.249509 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:00:22.248251 4939 memory_manager.go:365] "RemoveStaleState removing state" podUID="d5ee2b64-db54-4185-bbcf-3d294be13551" containerName="cleanup" Nov 03 01:00:22.249509 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:00:22.248263 4939 memory_manager.go:365] "RemoveStaleState removing state" podUID="6ec302ce-4f22-4a05-8200-9525580d6cdf" containerName="cleanup" Nov 03 01:00:22.249509 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:00:22.248271 4939 memory_manager.go:365] "RemoveStaleState removing state" podUID="bbecba1a-367a-4239-9e6c-fdfb4972ccf1" containerName="istio-proxy" Nov 03 01:00:22.249509 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:00:22.248276 4939 memory_manager.go:365] "RemoveStaleState removing state" podUID="b1e21399-49c6-4226-aa7f-6de60c486c9a" containerName="cleanup" Nov 03 01:00:22.249509 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:00:22.248282 4939 memory_manager.go:365] "RemoveStaleState removing state" podUID="9a1dc28b-dbe2-439e-8f7f-c63f6444de42" containerName="cleanup" Nov 03 01:00:22.249509 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:00:22.248287 4939 memory_manager.go:365] "RemoveStaleState removing state" podUID="b1e21399-49c6-4226-aa7f-6de60c486c9a" containerName="cleanup" Nov 03 01:00:22.249509 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:00:22.248293 4939 memory_manager.go:365] "RemoveStaleState removing state" podUID="9a1dc28b-dbe2-439e-8f7f-c63f6444de42" containerName="cleanup" Nov 03 01:00:22.249509 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:00:22.248299 4939 memory_manager.go:365] "RemoveStaleState removing state" podUID="bbecba1a-367a-4239-9e6c-fdfb4972ccf1" containerName="istio-proxy" Nov 03 01:00:22.249509 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:00:22.248305 4939 memory_manager.go:365] "RemoveStaleState removing state" podUID="9a1dc28b-dbe2-439e-8f7f-c63f6444de42" containerName="cleanup" Nov 03 01:00:22.252539 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:00:22.252519 4939 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-ipp-a-07f5f6/ipp-route-07f5f6-kserve-87874869c-4xwpw" Nov 03 01:00:22.258153 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:00:22.258128 4939 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-ipp-a-07f5f6\"/\"ipp-route-07f5f6-kserve-self-signed-certs\"" Nov 03 01:00:22.259663 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:00:22.259625 4939 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-ipp-a-07f5f6\"/\"default-dockercfg-8jzp4\"" Nov 03 01:00:22.260526 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:00:22.260495 4939 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-ipp-a-07f5f6/ipp-route-07f5f6-kserve-87874869c-4xwpw"] Nov 03 01:00:22.284824 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:00:22.284787 4939 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-ipp-a-07f5f6\"/\"openshift-service-ca.crt\"" Nov 03 01:00:22.285021 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:00:22.284847 4939 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-ipp-a-07f5f6\"/\"kube-root-ca.crt\"" Nov 03 01:00:22.311699 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:00:22.311669 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:00:22.317373 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:00:22.317347 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/21797ea7-0c52-407b-a34e-0af0aa92295a-tmp-dir\") pod \"ipp-route-07f5f6-kserve-87874869c-4xwpw\" (UID: \"21797ea7-0c52-407b-a34e-0af0aa92295a\") " pod="ai-tenant-e2e-ipp-a-07f5f6/ipp-route-07f5f6-kserve-87874869c-4xwpw" Nov 03 01:00:22.317497 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:00:22.317389 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/21797ea7-0c52-407b-a34e-0af0aa92295a-tls-certs\") pod \"ipp-route-07f5f6-kserve-87874869c-4xwpw\" (UID: \"21797ea7-0c52-407b-a34e-0af0aa92295a\") " pod="ai-tenant-e2e-ipp-a-07f5f6/ipp-route-07f5f6-kserve-87874869c-4xwpw" Nov 03 01:00:22.317571 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:00:22.317533 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/21797ea7-0c52-407b-a34e-0af0aa92295a-home\") pod \"ipp-route-07f5f6-kserve-87874869c-4xwpw\" (UID: \"21797ea7-0c52-407b-a34e-0af0aa92295a\") " pod="ai-tenant-e2e-ipp-a-07f5f6/ipp-route-07f5f6-kserve-87874869c-4xwpw" Nov 03 01:00:22.317624 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:00:22.317593 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hnq6c\" (UniqueName: \"kubernetes.io/projected/21797ea7-0c52-407b-a34e-0af0aa92295a-kube-api-access-hnq6c\") pod \"ipp-route-07f5f6-kserve-87874869c-4xwpw\" (UID: \"21797ea7-0c52-407b-a34e-0af0aa92295a\") " pod="ai-tenant-e2e-ipp-a-07f5f6/ipp-route-07f5f6-kserve-87874869c-4xwpw" Nov 03 01:00:22.317749 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:00:22.317729 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/21797ea7-0c52-407b-a34e-0af0aa92295a-dshm\") pod \"ipp-route-07f5f6-kserve-87874869c-4xwpw\" (UID: \"21797ea7-0c52-407b-a34e-0af0aa92295a\") " pod="ai-tenant-e2e-ipp-a-07f5f6/ipp-route-07f5f6-kserve-87874869c-4xwpw" Nov 03 01:00:22.317879 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:00:22.317854 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/21797ea7-0c52-407b-a34e-0af0aa92295a-model-cache\") pod \"ipp-route-07f5f6-kserve-87874869c-4xwpw\" (UID: \"21797ea7-0c52-407b-a34e-0af0aa92295a\") " pod="ai-tenant-e2e-ipp-a-07f5f6/ipp-route-07f5f6-kserve-87874869c-4xwpw" Nov 03 01:00:22.419234 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:00:22.419199 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/21797ea7-0c52-407b-a34e-0af0aa92295a-dshm\") pod \"ipp-route-07f5f6-kserve-87874869c-4xwpw\" (UID: \"21797ea7-0c52-407b-a34e-0af0aa92295a\") " pod="ai-tenant-e2e-ipp-a-07f5f6/ipp-route-07f5f6-kserve-87874869c-4xwpw" Nov 03 01:00:22.419419 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:00:22.419381 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/21797ea7-0c52-407b-a34e-0af0aa92295a-model-cache\") pod \"ipp-route-07f5f6-kserve-87874869c-4xwpw\" (UID: \"21797ea7-0c52-407b-a34e-0af0aa92295a\") " pod="ai-tenant-e2e-ipp-a-07f5f6/ipp-route-07f5f6-kserve-87874869c-4xwpw" Nov 03 01:00:22.419474 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:00:22.419462 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/21797ea7-0c52-407b-a34e-0af0aa92295a-tmp-dir\") pod \"ipp-route-07f5f6-kserve-87874869c-4xwpw\" (UID: \"21797ea7-0c52-407b-a34e-0af0aa92295a\") " pod="ai-tenant-e2e-ipp-a-07f5f6/ipp-route-07f5f6-kserve-87874869c-4xwpw" Nov 03 01:00:22.419512 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:00:22.419489 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/21797ea7-0c52-407b-a34e-0af0aa92295a-tls-certs\") pod \"ipp-route-07f5f6-kserve-87874869c-4xwpw\" (UID: \"21797ea7-0c52-407b-a34e-0af0aa92295a\") " pod="ai-tenant-e2e-ipp-a-07f5f6/ipp-route-07f5f6-kserve-87874869c-4xwpw" Nov 03 01:00:22.419686 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:00:22.419651 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/21797ea7-0c52-407b-a34e-0af0aa92295a-home\") pod \"ipp-route-07f5f6-kserve-87874869c-4xwpw\" (UID: \"21797ea7-0c52-407b-a34e-0af0aa92295a\") " pod="ai-tenant-e2e-ipp-a-07f5f6/ipp-route-07f5f6-kserve-87874869c-4xwpw" Nov 03 01:00:22.419776 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:00:22.419733 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hnq6c\" (UniqueName: \"kubernetes.io/projected/21797ea7-0c52-407b-a34e-0af0aa92295a-kube-api-access-hnq6c\") pod \"ipp-route-07f5f6-kserve-87874869c-4xwpw\" (UID: \"21797ea7-0c52-407b-a34e-0af0aa92295a\") " pod="ai-tenant-e2e-ipp-a-07f5f6/ipp-route-07f5f6-kserve-87874869c-4xwpw" Nov 03 01:00:22.419833 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:00:22.419787 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/21797ea7-0c52-407b-a34e-0af0aa92295a-model-cache\") pod \"ipp-route-07f5f6-kserve-87874869c-4xwpw\" (UID: \"21797ea7-0c52-407b-a34e-0af0aa92295a\") " pod="ai-tenant-e2e-ipp-a-07f5f6/ipp-route-07f5f6-kserve-87874869c-4xwpw" Nov 03 01:00:22.419833 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:00:22.419813 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/21797ea7-0c52-407b-a34e-0af0aa92295a-tmp-dir\") pod \"ipp-route-07f5f6-kserve-87874869c-4xwpw\" (UID: \"21797ea7-0c52-407b-a34e-0af0aa92295a\") " pod="ai-tenant-e2e-ipp-a-07f5f6/ipp-route-07f5f6-kserve-87874869c-4xwpw" Nov 03 01:00:22.420030 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:00:22.420008 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/21797ea7-0c52-407b-a34e-0af0aa92295a-home\") pod \"ipp-route-07f5f6-kserve-87874869c-4xwpw\" (UID: \"21797ea7-0c52-407b-a34e-0af0aa92295a\") " pod="ai-tenant-e2e-ipp-a-07f5f6/ipp-route-07f5f6-kserve-87874869c-4xwpw" Nov 03 01:00:22.421571 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:00:22.421545 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/21797ea7-0c52-407b-a34e-0af0aa92295a-dshm\") pod \"ipp-route-07f5f6-kserve-87874869c-4xwpw\" (UID: \"21797ea7-0c52-407b-a34e-0af0aa92295a\") " pod="ai-tenant-e2e-ipp-a-07f5f6/ipp-route-07f5f6-kserve-87874869c-4xwpw" Nov 03 01:00:22.422131 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:00:22.422109 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/21797ea7-0c52-407b-a34e-0af0aa92295a-tls-certs\") pod \"ipp-route-07f5f6-kserve-87874869c-4xwpw\" (UID: \"21797ea7-0c52-407b-a34e-0af0aa92295a\") " pod="ai-tenant-e2e-ipp-a-07f5f6/ipp-route-07f5f6-kserve-87874869c-4xwpw" Nov 03 01:00:22.436079 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:00:22.436047 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hnq6c\" (UniqueName: \"kubernetes.io/projected/21797ea7-0c52-407b-a34e-0af0aa92295a-kube-api-access-hnq6c\") pod \"ipp-route-07f5f6-kserve-87874869c-4xwpw\" (UID: \"21797ea7-0c52-407b-a34e-0af0aa92295a\") " pod="ai-tenant-e2e-ipp-a-07f5f6/ipp-route-07f5f6-kserve-87874869c-4xwpw" Nov 03 01:00:22.466933 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:00:22.463251 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:00:22.576479 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:00:22.576334 4939 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-ipp-a-07f5f6/ipp-route-07f5f6-kserve-87874869c-4xwpw" Nov 03 01:00:22.747327 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:00:22.747242 4939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-47369d-29894460-h5m8c" event={"ID":"370396d8-27e0-4320-b0f4-40134e62e18d","Type":"ContainerDied","Data":"4f6d9c82dd9854261b1837cc7256c93f8a3d9327ea782ce260ceff98c118c3b3"} Nov 03 01:00:22.747925 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:00:22.747882 4939 scope.go:122] "RemoveContainer" containerID="1f09fb83f6ad1d26a166faa09a39d314f9c1b189a182dec3f33d0d9bdd24eb80" Nov 03 01:00:22.748236 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:00:22.748219 4939 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-47369d-29894460-h5m8c" Nov 03 01:00:22.784894 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:00:22.783649 4939 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-47369d-29894460-h5m8c"] Nov 03 01:00:22.791526 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:00:22.789016 4939 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-47369d-29894460-h5m8c"] Nov 03 01:00:22.991497 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:00:22.991468 4939 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-ipp-a-07f5f6/ipp-route-07f5f6-kserve-87874869c-4xwpw"] Nov 03 01:00:22.992802 ip-10-0-140-107 kubenswrapper[4939]: W1103 01:00:22.992774 4939 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod21797ea7_0c52_407b_a34e_0af0aa92295a.slice/crio-845b7ff36b7f3e507a800c691a9d3350a9708131d3ee17b0f039834195c9b2fc WatchSource:0}: Error finding container 845b7ff36b7f3e507a800c691a9d3350a9708131d3ee17b0f039834195c9b2fc: Status 404 returned error can't find the container with id 845b7ff36b7f3e507a800c691a9d3350a9708131d3ee17b0f039834195c9b2fc Nov 03 01:00:23.269217 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:00:23.269112 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:00:23.756284 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:00:23.756244 4939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-ipp-a-07f5f6/ipp-route-07f5f6-kserve-87874869c-4xwpw" event={"ID":"21797ea7-0c52-407b-a34e-0af0aa92295a","Type":"ContainerStarted","Data":"d25aad89b759b278c4d2a3fd1f9421e482817bfea1ba703712d7fed2c06de97a"} Nov 03 01:00:23.756284 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:00:23.756283 4939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-ipp-a-07f5f6/ipp-route-07f5f6-kserve-87874869c-4xwpw" event={"ID":"21797ea7-0c52-407b-a34e-0af0aa92295a","Type":"ContainerStarted","Data":"845b7ff36b7f3e507a800c691a9d3350a9708131d3ee17b0f039834195c9b2fc"} Nov 03 01:00:23.779235 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:00:23.779174 4939 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-ipp-a-07f5f6/ipp-route-07f5f6-kserve-87874869c-4xwpw" podStartSLOduration=1.779155323 podStartE2EDuration="1.779155323s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 01:00:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 01:00:23.776090429 +0000 UTC m=+1172.040425398" watchObservedRunningTime="2026-11-03 01:00:23.779155323 +0000 UTC m=+1172.043490269" Nov 03 01:00:23.906751 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:00:23.906705 4939 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="370396d8-27e0-4320-b0f4-40134e62e18d" path="/var/lib/kubelet/pods/370396d8-27e0-4320-b0f4-40134e62e18d/volumes" Nov 03 01:00:23.965042 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:00:23.964979 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:00:24.501475 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:00:24.501433 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:00:24.582560 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:00:24.582521 4939 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-ipp-a-07f5f6/ipp-route-07f5f6-kserve-87874869c-4xwpw" Nov 03 01:00:25.427898 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:00:25.427859 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:00:26.496413 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:00:26.496377 4939 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-7406a9-5cd77855fc-n4s94"] Nov 03 01:00:26.515111 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:00:26.515077 4939 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-7406a9-78cc96967f-6wksw"] Nov 03 01:00:26.537112 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:00:26.537081 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:00:26.668085 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:00:26.668048 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:00:27.531686 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:00:27.531656 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:00:27.586996 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:00:27.586959 4939 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-650be0-openshift-default-5c7c5fb7d9-vj4qt"] Nov 03 01:00:27.587491 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:00:27.587457 4939 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-650be0-openshift-default-5c7c5fb7d9-vj4qt" podUID="5b6c2a38-1b62-41a5-803f-3111bd896857" containerName="istio-proxy" containerID="cri-o://2bb3d70abeafaa14cc7a438cb90ec1c4c3d30ee30fb4c12a87d7eb1804af321c" gracePeriod=30 Nov 03 01:00:27.644931 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:00:27.643405 4939 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-650be0-openshift-default-5c7c5fb7d9-267h5"] Nov 03 01:00:27.644931 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:00:27.643640 4939 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-650be0-openshift-default-5c7c5fb7d9-267h5" podUID="3c4531e2-d1cc-430f-8245-f129e486d172" containerName="istio-proxy" containerID="cri-o://61535bce52894eba09afa6c0ff603a245743d80cdd1df53f4d77b402ba744723" gracePeriod=30 Nov 03 01:00:27.796149 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:00:27.795235 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:00:28.231965 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:00:28.226011 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:00:28.370931 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:00:28.367981 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:00:28.575225 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:00:28.575134 4939 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-f88f87-67d5f888b7-nf6wt" Nov 03 01:00:28.685227 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:00:28.685186 4939 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-47369d-openshift-default-696ccdd997-9mqtd"] Nov 03 01:00:28.685465 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:00:28.685439 4939 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-api-b-47369d-openshift-default-696ccdd997-9mqtd" podUID="49df4a8a-81d6-4c9e-baf7-40c0f7cbaae1" containerName="istio-proxy" containerID="cri-o://4f759ea70471a8a4d57a74305e9917ce0eedab9b94f41568f1f13462bf9fba8d" gracePeriod=30 Nov 03 01:00:28.741952 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:00:28.741890 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:00:28.903933 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:00:28.899493 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:00:29.414694 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:00:29.414651 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:00:29.587862 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:00:29.587815 4939 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-f88f87-7b579c8df6-qvh6t" Nov 03 01:00:29.617741 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:00:29.617712 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:00:30.401513 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:00:30.401470 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:00:30.526934 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:00:30.526448 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:00:30.944848 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:00:30.944812 4939 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-praxis-a-1e352e-openshift-default-75f8bbdbfc-gphv5"] Nov 03 01:00:30.945892 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:00:30.945864 4939 cpu_manager.go:414] "containerMap: removing container" podUID="b1e21399-49c6-4226-aa7f-6de60c486c9a" containerName="cleanup" Nov 03 01:00:30.945892 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:00:30.945891 4939 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b1e21399-49c6-4226-aa7f-6de60c486c9a" containerName="cleanup" Nov 03 01:00:30.968226 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:00:30.945926 4939 cpu_manager.go:414] "containerMap: removing container" podUID="6ec302ce-4f22-4a05-8200-9525580d6cdf" containerName="cleanup" Nov 03 01:00:30.968226 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:00:30.945933 4939 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6ec302ce-4f22-4a05-8200-9525580d6cdf" containerName="cleanup" Nov 03 01:00:30.968226 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:00:30.945997 4939 cpu_manager.go:414] "containerMap: removing container" podUID="370396d8-27e0-4320-b0f4-40134e62e18d" containerName="cleanup" Nov 03 01:00:30.968226 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:00:30.946006 4939 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="370396d8-27e0-4320-b0f4-40134e62e18d" containerName="cleanup" Nov 03 01:00:30.968226 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:00:30.946175 4939 memory_manager.go:365] "RemoveStaleState removing state" podUID="d5ee2b64-db54-4185-bbcf-3d294be13551" containerName="cleanup" Nov 03 01:00:30.968226 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:00:30.946191 4939 memory_manager.go:365] "RemoveStaleState removing state" podUID="6ec302ce-4f22-4a05-8200-9525580d6cdf" containerName="cleanup" Nov 03 01:00:30.976277 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:00:30.976213 4939 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-a-1e352e-openshift-default-75f8bbdbfc-gphv5" Nov 03 01:00:30.987638 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:00:30.987610 4939 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-praxis-a-1e352e-openshift-default-75f8bbdbfc-gphv5"] Nov 03 01:00:30.988115 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:00:30.988097 4939 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-praxis-a-1e352e-openshift-default-dockercfg-dvr6z\"" Nov 03 01:00:31.007248 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:00:31.007210 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1d33734b-7202-4184-9fc0-a58898f68802-istio-podinfo\") pod \"e2e-praxis-a-1e352e-openshift-default-75f8bbdbfc-gphv5\" (UID: \"1d33734b-7202-4184-9fc0-a58898f68802\") " pod="openshift-ingress/e2e-praxis-a-1e352e-openshift-default-75f8bbdbfc-gphv5" Nov 03 01:00:31.007404 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:00:31.007286 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1d33734b-7202-4184-9fc0-a58898f68802-istiod-ca-cert\") pod \"e2e-praxis-a-1e352e-openshift-default-75f8bbdbfc-gphv5\" (UID: \"1d33734b-7202-4184-9fc0-a58898f68802\") " pod="openshift-ingress/e2e-praxis-a-1e352e-openshift-default-75f8bbdbfc-gphv5" Nov 03 01:00:31.007404 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:00:31.007389 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1d33734b-7202-4184-9fc0-a58898f68802-istio-data\") pod \"e2e-praxis-a-1e352e-openshift-default-75f8bbdbfc-gphv5\" (UID: \"1d33734b-7202-4184-9fc0-a58898f68802\") " pod="openshift-ingress/e2e-praxis-a-1e352e-openshift-default-75f8bbdbfc-gphv5" Nov 03 01:00:31.007542 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:00:31.007524 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1d33734b-7202-4184-9fc0-a58898f68802-istio-token\") pod \"e2e-praxis-a-1e352e-openshift-default-75f8bbdbfc-gphv5\" (UID: \"1d33734b-7202-4184-9fc0-a58898f68802\") " pod="openshift-ingress/e2e-praxis-a-1e352e-openshift-default-75f8bbdbfc-gphv5" Nov 03 01:00:31.007605 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:00:31.007563 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1d33734b-7202-4184-9fc0-a58898f68802-credential-socket\") pod \"e2e-praxis-a-1e352e-openshift-default-75f8bbdbfc-gphv5\" (UID: \"1d33734b-7202-4184-9fc0-a58898f68802\") " pod="openshift-ingress/e2e-praxis-a-1e352e-openshift-default-75f8bbdbfc-gphv5" Nov 03 01:00:31.007605 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:00:31.007590 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1d33734b-7202-4184-9fc0-a58898f68802-istio-envoy\") pod \"e2e-praxis-a-1e352e-openshift-default-75f8bbdbfc-gphv5\" (UID: \"1d33734b-7202-4184-9fc0-a58898f68802\") " pod="openshift-ingress/e2e-praxis-a-1e352e-openshift-default-75f8bbdbfc-gphv5" Nov 03 01:00:31.007744 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:00:31.007726 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1d33734b-7202-4184-9fc0-a58898f68802-workload-certs\") pod \"e2e-praxis-a-1e352e-openshift-default-75f8bbdbfc-gphv5\" (UID: \"1d33734b-7202-4184-9fc0-a58898f68802\") " pod="openshift-ingress/e2e-praxis-a-1e352e-openshift-default-75f8bbdbfc-gphv5" Nov 03 01:00:31.007813 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:00:31.007763 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1d33734b-7202-4184-9fc0-a58898f68802-workload-socket\") pod \"e2e-praxis-a-1e352e-openshift-default-75f8bbdbfc-gphv5\" (UID: \"1d33734b-7202-4184-9fc0-a58898f68802\") " pod="openshift-ingress/e2e-praxis-a-1e352e-openshift-default-75f8bbdbfc-gphv5" Nov 03 01:00:31.007868 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:00:31.007814 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-55f8w\" (UniqueName: \"kubernetes.io/projected/1d33734b-7202-4184-9fc0-a58898f68802-kube-api-access-55f8w\") pod \"e2e-praxis-a-1e352e-openshift-default-75f8bbdbfc-gphv5\" (UID: \"1d33734b-7202-4184-9fc0-a58898f68802\") " pod="openshift-ingress/e2e-praxis-a-1e352e-openshift-default-75f8bbdbfc-gphv5" Nov 03 01:00:31.118933 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:00:31.117141 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1d33734b-7202-4184-9fc0-a58898f68802-istio-token\") pod \"e2e-praxis-a-1e352e-openshift-default-75f8bbdbfc-gphv5\" (UID: \"1d33734b-7202-4184-9fc0-a58898f68802\") " pod="openshift-ingress/e2e-praxis-a-1e352e-openshift-default-75f8bbdbfc-gphv5" Nov 03 01:00:31.118933 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:00:31.117209 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1d33734b-7202-4184-9fc0-a58898f68802-credential-socket\") pod \"e2e-praxis-a-1e352e-openshift-default-75f8bbdbfc-gphv5\" (UID: \"1d33734b-7202-4184-9fc0-a58898f68802\") " pod="openshift-ingress/e2e-praxis-a-1e352e-openshift-default-75f8bbdbfc-gphv5" Nov 03 01:00:31.118933 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:00:31.117248 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1d33734b-7202-4184-9fc0-a58898f68802-istio-envoy\") pod \"e2e-praxis-a-1e352e-openshift-default-75f8bbdbfc-gphv5\" (UID: \"1d33734b-7202-4184-9fc0-a58898f68802\") " pod="openshift-ingress/e2e-praxis-a-1e352e-openshift-default-75f8bbdbfc-gphv5" Nov 03 01:00:31.118933 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:00:31.117347 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1d33734b-7202-4184-9fc0-a58898f68802-workload-certs\") pod \"e2e-praxis-a-1e352e-openshift-default-75f8bbdbfc-gphv5\" (UID: \"1d33734b-7202-4184-9fc0-a58898f68802\") " pod="openshift-ingress/e2e-praxis-a-1e352e-openshift-default-75f8bbdbfc-gphv5" Nov 03 01:00:31.118933 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:00:31.117395 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1d33734b-7202-4184-9fc0-a58898f68802-workload-socket\") pod \"e2e-praxis-a-1e352e-openshift-default-75f8bbdbfc-gphv5\" (UID: \"1d33734b-7202-4184-9fc0-a58898f68802\") " pod="openshift-ingress/e2e-praxis-a-1e352e-openshift-default-75f8bbdbfc-gphv5" Nov 03 01:00:31.118933 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:00:31.117428 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-55f8w\" (UniqueName: \"kubernetes.io/projected/1d33734b-7202-4184-9fc0-a58898f68802-kube-api-access-55f8w\") pod \"e2e-praxis-a-1e352e-openshift-default-75f8bbdbfc-gphv5\" (UID: \"1d33734b-7202-4184-9fc0-a58898f68802\") " pod="openshift-ingress/e2e-praxis-a-1e352e-openshift-default-75f8bbdbfc-gphv5" Nov 03 01:00:31.118933 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:00:31.117625 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1d33734b-7202-4184-9fc0-a58898f68802-istio-podinfo\") pod \"e2e-praxis-a-1e352e-openshift-default-75f8bbdbfc-gphv5\" (UID: \"1d33734b-7202-4184-9fc0-a58898f68802\") " pod="openshift-ingress/e2e-praxis-a-1e352e-openshift-default-75f8bbdbfc-gphv5" Nov 03 01:00:31.118933 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:00:31.117689 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1d33734b-7202-4184-9fc0-a58898f68802-istiod-ca-cert\") pod \"e2e-praxis-a-1e352e-openshift-default-75f8bbdbfc-gphv5\" (UID: \"1d33734b-7202-4184-9fc0-a58898f68802\") " pod="openshift-ingress/e2e-praxis-a-1e352e-openshift-default-75f8bbdbfc-gphv5" Nov 03 01:00:31.118933 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:00:31.117731 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1d33734b-7202-4184-9fc0-a58898f68802-istio-data\") pod \"e2e-praxis-a-1e352e-openshift-default-75f8bbdbfc-gphv5\" (UID: \"1d33734b-7202-4184-9fc0-a58898f68802\") " pod="openshift-ingress/e2e-praxis-a-1e352e-openshift-default-75f8bbdbfc-gphv5" Nov 03 01:00:31.118933 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:00:31.118428 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1d33734b-7202-4184-9fc0-a58898f68802-istio-data\") pod \"e2e-praxis-a-1e352e-openshift-default-75f8bbdbfc-gphv5\" (UID: \"1d33734b-7202-4184-9fc0-a58898f68802\") " pod="openshift-ingress/e2e-praxis-a-1e352e-openshift-default-75f8bbdbfc-gphv5" Nov 03 01:00:31.120036 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:00:31.119744 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1d33734b-7202-4184-9fc0-a58898f68802-istiod-ca-cert\") pod \"e2e-praxis-a-1e352e-openshift-default-75f8bbdbfc-gphv5\" (UID: \"1d33734b-7202-4184-9fc0-a58898f68802\") " pod="openshift-ingress/e2e-praxis-a-1e352e-openshift-default-75f8bbdbfc-gphv5" Nov 03 01:00:31.142012 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:00:31.141978 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:00:31.157337 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:00:31.157300 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1d33734b-7202-4184-9fc0-a58898f68802-workload-socket\") pod \"e2e-praxis-a-1e352e-openshift-default-75f8bbdbfc-gphv5\" (UID: \"1d33734b-7202-4184-9fc0-a58898f68802\") " pod="openshift-ingress/e2e-praxis-a-1e352e-openshift-default-75f8bbdbfc-gphv5" Nov 03 01:00:31.157947 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:00:31.157852 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1d33734b-7202-4184-9fc0-a58898f68802-workload-certs\") pod \"e2e-praxis-a-1e352e-openshift-default-75f8bbdbfc-gphv5\" (UID: \"1d33734b-7202-4184-9fc0-a58898f68802\") " pod="openshift-ingress/e2e-praxis-a-1e352e-openshift-default-75f8bbdbfc-gphv5" Nov 03 01:00:31.158955 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:00:31.158892 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1d33734b-7202-4184-9fc0-a58898f68802-istio-token\") pod \"e2e-praxis-a-1e352e-openshift-default-75f8bbdbfc-gphv5\" (UID: \"1d33734b-7202-4184-9fc0-a58898f68802\") " pod="openshift-ingress/e2e-praxis-a-1e352e-openshift-default-75f8bbdbfc-gphv5" Nov 03 01:00:31.161314 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:00:31.161290 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1d33734b-7202-4184-9fc0-a58898f68802-istio-envoy\") pod \"e2e-praxis-a-1e352e-openshift-default-75f8bbdbfc-gphv5\" (UID: \"1d33734b-7202-4184-9fc0-a58898f68802\") " pod="openshift-ingress/e2e-praxis-a-1e352e-openshift-default-75f8bbdbfc-gphv5" Nov 03 01:00:31.176007 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:00:31.161897 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1d33734b-7202-4184-9fc0-a58898f68802-istio-podinfo\") pod \"e2e-praxis-a-1e352e-openshift-default-75f8bbdbfc-gphv5\" (UID: \"1d33734b-7202-4184-9fc0-a58898f68802\") " pod="openshift-ingress/e2e-praxis-a-1e352e-openshift-default-75f8bbdbfc-gphv5" Nov 03 01:00:31.176221 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:00:31.171276 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1d33734b-7202-4184-9fc0-a58898f68802-credential-socket\") pod \"e2e-praxis-a-1e352e-openshift-default-75f8bbdbfc-gphv5\" (UID: \"1d33734b-7202-4184-9fc0-a58898f68802\") " pod="openshift-ingress/e2e-praxis-a-1e352e-openshift-default-75f8bbdbfc-gphv5" Nov 03 01:00:31.186929 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:00:31.177184 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-55f8w\" (UniqueName: \"kubernetes.io/projected/1d33734b-7202-4184-9fc0-a58898f68802-kube-api-access-55f8w\") pod \"e2e-praxis-a-1e352e-openshift-default-75f8bbdbfc-gphv5\" (UID: \"1d33734b-7202-4184-9fc0-a58898f68802\") " pod="openshift-ingress/e2e-praxis-a-1e352e-openshift-default-75f8bbdbfc-gphv5" Nov 03 01:00:31.304722 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:00:31.304638 4939 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-a-1e352e-openshift-default-75f8bbdbfc-gphv5" Nov 03 01:00:31.443504 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:00:31.443451 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:00:31.497155 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:00:31.496983 4939 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-api-a-7406a9-5cd77855fc-n4s94" podUID="d5b08c7b-b02e-434f-8b32-32a502a92e69" containerName="payload-processing" containerID="cri-o://a06ce95b4e30f0381b53f1c01cd0c3fdffd07fa2b8974d540122e7d722532242" gracePeriod=25 Nov 03 01:00:31.517244 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:00:31.517097 4939 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-api-a-7406a9-78cc96967f-6wksw" podUID="12ba5035-bd74-4162-9c8e-9353527f757f" containerName="payload-pre-processing" containerID="cri-o://2aa61ee297ea7c024943a7060c9d7c0ee4db97995216b2d5909ea5f09be1ca10" gracePeriod=25 Nov 03 01:00:31.820518 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:00:31.820482 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:00:32.046829 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:00:32.042260 4939 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 01:00:32.046829 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:00:32.042367 4939 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 01:00:32.046829 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:00:32.042430 4939 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 01:00:32.063193 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:00:32.050683 4939 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-praxis-a-1e352e-openshift-default-75f8bbdbfc-gphv5"] Nov 03 01:00:32.096767 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:00:32.096734 4939 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-7406a9-5cd77855fc-n4s94" Nov 03 01:00:32.108232 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:00:32.108207 4939 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-7406a9-78cc96967f-6wksw" Nov 03 01:00:32.184841 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:00:32.184809 4939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/12ba5035-bd74-4162-9c8e-9353527f757f-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/12ba5035-bd74-4162-9c8e-9353527f757f-plugins-config-volume\") pod \"12ba5035-bd74-4162-9c8e-9353527f757f\" (UID: \"12ba5035-bd74-4162-9c8e-9353527f757f\") " Nov 03 01:00:32.184841 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:00:32.184852 4939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d5b08c7b-b02e-434f-8b32-32a502a92e69-kube-api-access-qfcfj\" (UniqueName: \"kubernetes.io/projected/d5b08c7b-b02e-434f-8b32-32a502a92e69-kube-api-access-qfcfj\") pod \"d5b08c7b-b02e-434f-8b32-32a502a92e69\" (UID: \"d5b08c7b-b02e-434f-8b32-32a502a92e69\") " Nov 03 01:00:32.185110 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:00:32.185066 4939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/12ba5035-bd74-4162-9c8e-9353527f757f-kube-api-access-jbnx9\" (UniqueName: \"kubernetes.io/projected/12ba5035-bd74-4162-9c8e-9353527f757f-kube-api-access-jbnx9\") pod \"12ba5035-bd74-4162-9c8e-9353527f757f\" (UID: \"12ba5035-bd74-4162-9c8e-9353527f757f\") " Nov 03 01:00:32.185110 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:00:32.185104 4939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d5b08c7b-b02e-434f-8b32-32a502a92e69-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d5b08c7b-b02e-434f-8b32-32a502a92e69-plugins-config-volume\") pod \"d5b08c7b-b02e-434f-8b32-32a502a92e69\" (UID: \"d5b08c7b-b02e-434f-8b32-32a502a92e69\") " Nov 03 01:00:32.185306 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:00:32.185284 4939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/12ba5035-bd74-4162-9c8e-9353527f757f-plugins-config-volume" pod "12ba5035-bd74-4162-9c8e-9353527f757f" (UID: "12ba5035-bd74-4162-9c8e-9353527f757f"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 01:00:32.185786 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:00:32.185761 4939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d5b08c7b-b02e-434f-8b32-32a502a92e69-plugins-config-volume" pod "d5b08c7b-b02e-434f-8b32-32a502a92e69" (UID: "d5b08c7b-b02e-434f-8b32-32a502a92e69"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 01:00:32.186184 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:00:32.186161 4939 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/12ba5035-bd74-4162-9c8e-9353527f757f-plugins-config-volume\") on node \"ip-10-0-140-107.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:00:32.186260 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:00:32.186189 4939 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d5b08c7b-b02e-434f-8b32-32a502a92e69-plugins-config-volume\") on node \"ip-10-0-140-107.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:00:32.187667 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:00:32.187641 4939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d5b08c7b-b02e-434f-8b32-32a502a92e69-kube-api-access-qfcfj" pod "d5b08c7b-b02e-434f-8b32-32a502a92e69" (UID: "d5b08c7b-b02e-434f-8b32-32a502a92e69"). InnerVolumeSpecName "kube-api-access-qfcfj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 01:00:32.187735 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:00:32.187685 4939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/12ba5035-bd74-4162-9c8e-9353527f757f-kube-api-access-jbnx9" pod "12ba5035-bd74-4162-9c8e-9353527f757f" (UID: "12ba5035-bd74-4162-9c8e-9353527f757f"). InnerVolumeSpecName "kube-api-access-jbnx9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 01:00:32.287434 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:00:32.287394 4939 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qfcfj\" (UniqueName: \"kubernetes.io/projected/d5b08c7b-b02e-434f-8b32-32a502a92e69-kube-api-access-qfcfj\") on node \"ip-10-0-140-107.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:00:32.287434 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:00:32.287421 4939 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jbnx9\" (UniqueName: \"kubernetes.io/projected/12ba5035-bd74-4162-9c8e-9353527f757f-kube-api-access-jbnx9\") on node \"ip-10-0-140-107.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:00:32.288156 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:00:32.288077 4939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-1e352e-openshift-default-75f8bbdbfc-gphv5" event={"ID":"1d33734b-7202-4184-9fc0-a58898f68802","Type":"ContainerStarted","Data":"e7bc838eb15997b3a1363ee712b7e1769fc4c35e00cdc7ec50b7801791a30994"} Nov 03 01:00:32.288156 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:00:32.288130 4939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-1e352e-openshift-default-75f8bbdbfc-gphv5" event={"ID":"1d33734b-7202-4184-9fc0-a58898f68802","Type":"ContainerStarted","Data":"fc0d634dcb8f8ae0378b0a6e2de6430f238fab05046ae21362ecba8a3e4d9229"} Nov 03 01:00:32.289789 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:00:32.289762 4939 generic.go:356] "Generic (PLEG): container finished" podID="d5b08c7b-b02e-434f-8b32-32a502a92e69" containerID="a06ce95b4e30f0381b53f1c01cd0c3fdffd07fa2b8974d540122e7d722532242" exitCode=0 Nov 03 01:00:32.289878 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:00:32.289800 4939 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-7406a9-5cd77855fc-n4s94" Nov 03 01:00:32.289878 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:00:32.289851 4939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-7406a9-5cd77855fc-n4s94" event={"ID":"d5b08c7b-b02e-434f-8b32-32a502a92e69","Type":"ContainerDied","Data":"a06ce95b4e30f0381b53f1c01cd0c3fdffd07fa2b8974d540122e7d722532242"} Nov 03 01:00:32.289988 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:00:32.289936 4939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-7406a9-5cd77855fc-n4s94" event={"ID":"d5b08c7b-b02e-434f-8b32-32a502a92e69","Type":"ContainerDied","Data":"0a654fd5deffcacc8ee16dfb863a8c862e1ec7c83b9450034abdf0bfe0ffca77"} Nov 03 01:00:32.289988 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:00:32.289965 4939 scope.go:122] "RemoveContainer" containerID="a06ce95b4e30f0381b53f1c01cd0c3fdffd07fa2b8974d540122e7d722532242" Nov 03 01:00:32.291769 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:00:32.291748 4939 generic.go:356] "Generic (PLEG): container finished" podID="12ba5035-bd74-4162-9c8e-9353527f757f" containerID="2aa61ee297ea7c024943a7060c9d7c0ee4db97995216b2d5909ea5f09be1ca10" exitCode=0 Nov 03 01:00:32.291941 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:00:32.291893 4939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-7406a9-78cc96967f-6wksw" event={"ID":"12ba5035-bd74-4162-9c8e-9353527f757f","Type":"ContainerDied","Data":"2aa61ee297ea7c024943a7060c9d7c0ee4db97995216b2d5909ea5f09be1ca10"} Nov 03 01:00:32.292009 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:00:32.291948 4939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-7406a9-78cc96967f-6wksw" event={"ID":"12ba5035-bd74-4162-9c8e-9353527f757f","Type":"ContainerDied","Data":"4cf41df5205975e7f3d32f20a353cce8bb348c913236673a10b1a548d402eb1d"} Nov 03 01:00:32.292077 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:00:32.292017 4939 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-7406a9-78cc96967f-6wksw" Nov 03 01:00:32.304346 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:00:32.304322 4939 scope.go:122] "RemoveContainer" containerID="a06ce95b4e30f0381b53f1c01cd0c3fdffd07fa2b8974d540122e7d722532242" Nov 03 01:00:32.304864 ip-10-0-140-107 kubenswrapper[4939]: E1103 01:00:32.304841 4939 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a06ce95b4e30f0381b53f1c01cd0c3fdffd07fa2b8974d540122e7d722532242\": container with ID starting with a06ce95b4e30f0381b53f1c01cd0c3fdffd07fa2b8974d540122e7d722532242 not found: ID does not exist" containerID="a06ce95b4e30f0381b53f1c01cd0c3fdffd07fa2b8974d540122e7d722532242" Nov 03 01:00:32.304954 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:00:32.304877 4939 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a06ce95b4e30f0381b53f1c01cd0c3fdffd07fa2b8974d540122e7d722532242"} err="failed to get container status \"a06ce95b4e30f0381b53f1c01cd0c3fdffd07fa2b8974d540122e7d722532242\": rpc error: code = NotFound desc = could not find container \"a06ce95b4e30f0381b53f1c01cd0c3fdffd07fa2b8974d540122e7d722532242\": container with ID starting with a06ce95b4e30f0381b53f1c01cd0c3fdffd07fa2b8974d540122e7d722532242 not found: ID does not exist" Nov 03 01:00:32.305008 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:00:32.304899 4939 scope.go:122] "RemoveContainer" containerID="2aa61ee297ea7c024943a7060c9d7c0ee4db97995216b2d5909ea5f09be1ca10" Nov 03 01:00:32.306408 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:00:32.306365 4939 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-praxis-a-1e352e-openshift-default-75f8bbdbfc-gphv5" Nov 03 01:00:32.317151 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:00:32.317108 4939 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-praxis-a-1e352e-openshift-default-75f8bbdbfc-gphv5" podStartSLOduration=2.317096085 podStartE2EDuration="2.317096085s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 01:00:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 01:00:32.312889529 +0000 UTC m=+1180.577224457" watchObservedRunningTime="2026-11-03 01:00:32.317096085 +0000 UTC m=+1180.581431029" Nov 03 01:00:32.319689 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:00:32.319589 4939 scope.go:122] "RemoveContainer" containerID="2aa61ee297ea7c024943a7060c9d7c0ee4db97995216b2d5909ea5f09be1ca10" Nov 03 01:00:32.319878 ip-10-0-140-107 kubenswrapper[4939]: E1103 01:00:32.319860 4939 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2aa61ee297ea7c024943a7060c9d7c0ee4db97995216b2d5909ea5f09be1ca10\": container with ID starting with 2aa61ee297ea7c024943a7060c9d7c0ee4db97995216b2d5909ea5f09be1ca10 not found: ID does not exist" containerID="2aa61ee297ea7c024943a7060c9d7c0ee4db97995216b2d5909ea5f09be1ca10" Nov 03 01:00:32.319980 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:00:32.319887 4939 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2aa61ee297ea7c024943a7060c9d7c0ee4db97995216b2d5909ea5f09be1ca10"} err="failed to get container status \"2aa61ee297ea7c024943a7060c9d7c0ee4db97995216b2d5909ea5f09be1ca10\": rpc error: code = NotFound desc = could not find container \"2aa61ee297ea7c024943a7060c9d7c0ee4db97995216b2d5909ea5f09be1ca10\": container with ID starting with 2aa61ee297ea7c024943a7060c9d7c0ee4db97995216b2d5909ea5f09be1ca10 not found: ID does not exist" Nov 03 01:00:32.331478 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:00:32.331446 4939 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-7406a9-78cc96967f-6wksw"] Nov 03 01:00:32.336500 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:00:32.336474 4939 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-7406a9-78cc96967f-6wksw"] Nov 03 01:00:32.380452 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:00:32.380403 4939 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-7406a9-5cd77855fc-n4s94"] Nov 03 01:00:32.384058 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:00:32.384030 4939 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-7406a9-5cd77855fc-n4s94"] Nov 03 01:00:32.584724 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:00:32.584685 4939 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-ipp-a-07f5f6/ipp-route-07f5f6-kserve-87874869c-4xwpw" Nov 03 01:00:32.608934 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:00:32.608217 4939 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-ipp-a-07f5f6/ipp-route-07f5f6-kserve-87874869c-4xwpw" Nov 03 01:00:32.786960 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:00:32.774720 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:00:33.142701 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:00:33.142672 4939 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-650be0-openshift-default-5c7c5fb7d9-vj4qt" Nov 03 01:00:33.224754 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:00:33.224696 4939 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-650be0-openshift-default-5c7c5fb7d9-267h5" Nov 03 01:00:33.256959 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:00:33.246184 4939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5b6c2a38-1b62-41a5-803f-3111bd896857-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5b6c2a38-1b62-41a5-803f-3111bd896857-workload-socket\") pod \"5b6c2a38-1b62-41a5-803f-3111bd896857\" (UID: \"5b6c2a38-1b62-41a5-803f-3111bd896857\") " Nov 03 01:00:33.257200 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:00:33.250439 4939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5b6c2a38-1b62-41a5-803f-3111bd896857-workload-socket" pod "5b6c2a38-1b62-41a5-803f-3111bd896857" (UID: "5b6c2a38-1b62-41a5-803f-3111bd896857"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 01:00:33.259063 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:00:33.258991 4939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5b6c2a38-1b62-41a5-803f-3111bd896857-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5b6c2a38-1b62-41a5-803f-3111bd896857-istio-envoy\") pod \"5b6c2a38-1b62-41a5-803f-3111bd896857\" (UID: \"5b6c2a38-1b62-41a5-803f-3111bd896857\") " Nov 03 01:00:33.259196 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:00:33.259091 4939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5b6c2a38-1b62-41a5-803f-3111bd896857-kube-api-access-zn9t7\" (UniqueName: \"kubernetes.io/projected/5b6c2a38-1b62-41a5-803f-3111bd896857-kube-api-access-zn9t7\") pod \"5b6c2a38-1b62-41a5-803f-3111bd896857\" (UID: \"5b6c2a38-1b62-41a5-803f-3111bd896857\") " Nov 03 01:00:33.259196 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:00:33.259116 4939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5b6c2a38-1b62-41a5-803f-3111bd896857-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5b6c2a38-1b62-41a5-803f-3111bd896857-credential-socket\") pod \"5b6c2a38-1b62-41a5-803f-3111bd896857\" (UID: \"5b6c2a38-1b62-41a5-803f-3111bd896857\") " Nov 03 01:00:33.259318 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:00:33.259233 4939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5b6c2a38-1b62-41a5-803f-3111bd896857-istio-token\" (UniqueName: \"kubernetes.io/projected/5b6c2a38-1b62-41a5-803f-3111bd896857-istio-token\") pod \"5b6c2a38-1b62-41a5-803f-3111bd896857\" (UID: \"5b6c2a38-1b62-41a5-803f-3111bd896857\") " Nov 03 01:00:33.259318 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:00:33.259283 4939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/5b6c2a38-1b62-41a5-803f-3111bd896857-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5b6c2a38-1b62-41a5-803f-3111bd896857-istio-podinfo\") pod \"5b6c2a38-1b62-41a5-803f-3111bd896857\" (UID: \"5b6c2a38-1b62-41a5-803f-3111bd896857\") " Nov 03 01:00:33.259318 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:00:33.259302 4939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5b6c2a38-1b62-41a5-803f-3111bd896857-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5b6c2a38-1b62-41a5-803f-3111bd896857-workload-certs\") pod \"5b6c2a38-1b62-41a5-803f-3111bd896857\" (UID: \"5b6c2a38-1b62-41a5-803f-3111bd896857\") " Nov 03 01:00:33.259483 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:00:33.259335 4939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5b6c2a38-1b62-41a5-803f-3111bd896857-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5b6c2a38-1b62-41a5-803f-3111bd896857-istio-data\") pod \"5b6c2a38-1b62-41a5-803f-3111bd896857\" (UID: \"5b6c2a38-1b62-41a5-803f-3111bd896857\") " Nov 03 01:00:33.259540 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:00:33.259514 4939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5b6c2a38-1b62-41a5-803f-3111bd896857-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5b6c2a38-1b62-41a5-803f-3111bd896857-istiod-ca-cert\") pod \"5b6c2a38-1b62-41a5-803f-3111bd896857\" (UID: \"5b6c2a38-1b62-41a5-803f-3111bd896857\") " Nov 03 01:00:33.259859 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:00:33.259833 4939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5b6c2a38-1b62-41a5-803f-3111bd896857-workload-certs" pod "5b6c2a38-1b62-41a5-803f-3111bd896857" (UID: "5b6c2a38-1b62-41a5-803f-3111bd896857"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 01:00:33.260317 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:00:33.260291 4939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5b6c2a38-1b62-41a5-803f-3111bd896857-credential-socket" pod "5b6c2a38-1b62-41a5-803f-3111bd896857" (UID: "5b6c2a38-1b62-41a5-803f-3111bd896857"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 01:00:33.260425 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:00:33.260345 4939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5b6c2a38-1b62-41a5-803f-3111bd896857-istiod-ca-cert" pod "5b6c2a38-1b62-41a5-803f-3111bd896857" (UID: "5b6c2a38-1b62-41a5-803f-3111bd896857"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 01:00:33.260797 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:00:33.260515 4939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5b6c2a38-1b62-41a5-803f-3111bd896857-istio-data" pod "5b6c2a38-1b62-41a5-803f-3111bd896857" (UID: "5b6c2a38-1b62-41a5-803f-3111bd896857"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 01:00:33.260797 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:00:33.260676 4939 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5b6c2a38-1b62-41a5-803f-3111bd896857-workload-certs\") on node \"ip-10-0-140-107.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:00:33.260797 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:00:33.260695 4939 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5b6c2a38-1b62-41a5-803f-3111bd896857-istio-data\") on node \"ip-10-0-140-107.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:00:33.260797 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:00:33.260711 4939 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5b6c2a38-1b62-41a5-803f-3111bd896857-istiod-ca-cert\") on node \"ip-10-0-140-107.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:00:33.260797 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:00:33.260725 4939 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5b6c2a38-1b62-41a5-803f-3111bd896857-workload-socket\") on node \"ip-10-0-140-107.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:00:33.260797 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:00:33.260738 4939 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5b6c2a38-1b62-41a5-803f-3111bd896857-credential-socket\") on node \"ip-10-0-140-107.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:00:33.261830 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:00:33.261804 4939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5b6c2a38-1b62-41a5-803f-3111bd896857-istio-envoy" pod "5b6c2a38-1b62-41a5-803f-3111bd896857" (UID: "5b6c2a38-1b62-41a5-803f-3111bd896857"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 01:00:33.262036 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:00:33.262013 4939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5b6c2a38-1b62-41a5-803f-3111bd896857-kube-api-access-zn9t7" pod "5b6c2a38-1b62-41a5-803f-3111bd896857" (UID: "5b6c2a38-1b62-41a5-803f-3111bd896857"). InnerVolumeSpecName "kube-api-access-zn9t7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 01:00:33.263108 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:00:33.263076 4939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5b6c2a38-1b62-41a5-803f-3111bd896857-istio-token" pod "5b6c2a38-1b62-41a5-803f-3111bd896857" (UID: "5b6c2a38-1b62-41a5-803f-3111bd896857"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 01:00:33.263474 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:00:33.263450 4939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/5b6c2a38-1b62-41a5-803f-3111bd896857-istio-podinfo" pod "5b6c2a38-1b62-41a5-803f-3111bd896857" (UID: "5b6c2a38-1b62-41a5-803f-3111bd896857"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 01:00:33.306879 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:00:33.306830 4939 generic.go:356] "Generic (PLEG): container finished" podID="5b6c2a38-1b62-41a5-803f-3111bd896857" containerID="2bb3d70abeafaa14cc7a438cb90ec1c4c3d30ee30fb4c12a87d7eb1804af321c" exitCode=0 Nov 03 01:00:33.307101 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:00:33.306948 4939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-650be0-openshift-default-5c7c5fb7d9-vj4qt" event={"ID":"5b6c2a38-1b62-41a5-803f-3111bd896857","Type":"ContainerDied","Data":"2bb3d70abeafaa14cc7a438cb90ec1c4c3d30ee30fb4c12a87d7eb1804af321c"} Nov 03 01:00:33.307101 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:00:33.306999 4939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-650be0-openshift-default-5c7c5fb7d9-vj4qt" event={"ID":"5b6c2a38-1b62-41a5-803f-3111bd896857","Type":"ContainerDied","Data":"c227f263df4a59967d13b1ab7b3030f9f80daf4da817e13e0a794290190ff69f"} Nov 03 01:00:33.307101 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:00:33.307029 4939 scope.go:122] "RemoveContainer" containerID="2bb3d70abeafaa14cc7a438cb90ec1c4c3d30ee30fb4c12a87d7eb1804af321c" Nov 03 01:00:33.307352 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:00:33.307062 4939 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-650be0-openshift-default-5c7c5fb7d9-vj4qt" Nov 03 01:00:33.313500 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:00:33.313471 4939 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-praxis-a-1e352e-openshift-default-75f8bbdbfc-gphv5" Nov 03 01:00:33.314508 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:00:33.314479 4939 generic.go:356] "Generic (PLEG): container finished" podID="3c4531e2-d1cc-430f-8245-f129e486d172" containerID="61535bce52894eba09afa6c0ff603a245743d80cdd1df53f4d77b402ba744723" exitCode=0 Nov 03 01:00:33.314628 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:00:33.314526 4939 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-650be0-openshift-default-5c7c5fb7d9-267h5" Nov 03 01:00:33.314628 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:00:33.314526 4939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-650be0-openshift-default-5c7c5fb7d9-267h5" event={"ID":"3c4531e2-d1cc-430f-8245-f129e486d172","Type":"ContainerDied","Data":"61535bce52894eba09afa6c0ff603a245743d80cdd1df53f4d77b402ba744723"} Nov 03 01:00:33.314628 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:00:33.314580 4939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-650be0-openshift-default-5c7c5fb7d9-267h5" event={"ID":"3c4531e2-d1cc-430f-8245-f129e486d172","Type":"ContainerDied","Data":"ca19798c771aa143c820f7c5ccce887042c5ef2f35a4d5415b8e9de4fa1526c5"} Nov 03 01:00:33.315008 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:00:33.314990 4939 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-praxis-a-1e352e-openshift-default-75f8bbdbfc-gphv5" Nov 03 01:00:33.316000 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:00:33.315944 4939 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-praxis-a-1e352e-openshift-default-75f8bbdbfc-gphv5" Nov 03 01:00:33.328025 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:00:33.327994 4939 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-ipp-a-07f5f6/ipp-route-07f5f6-kserve-87874869c-4xwpw" Nov 03 01:00:33.329299 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:00:33.329131 4939 scope.go:122] "RemoveContainer" containerID="2bb3d70abeafaa14cc7a438cb90ec1c4c3d30ee30fb4c12a87d7eb1804af321c" Nov 03 01:00:33.329519 ip-10-0-140-107 kubenswrapper[4939]: E1103 01:00:33.329491 4939 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2bb3d70abeafaa14cc7a438cb90ec1c4c3d30ee30fb4c12a87d7eb1804af321c\": container with ID starting with 2bb3d70abeafaa14cc7a438cb90ec1c4c3d30ee30fb4c12a87d7eb1804af321c not found: ID does not exist" containerID="2bb3d70abeafaa14cc7a438cb90ec1c4c3d30ee30fb4c12a87d7eb1804af321c" Nov 03 01:00:33.329619 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:00:33.329529 4939 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2bb3d70abeafaa14cc7a438cb90ec1c4c3d30ee30fb4c12a87d7eb1804af321c"} err="failed to get container status \"2bb3d70abeafaa14cc7a438cb90ec1c4c3d30ee30fb4c12a87d7eb1804af321c\": rpc error: code = NotFound desc = could not find container \"2bb3d70abeafaa14cc7a438cb90ec1c4c3d30ee30fb4c12a87d7eb1804af321c\": container with ID starting with 2bb3d70abeafaa14cc7a438cb90ec1c4c3d30ee30fb4c12a87d7eb1804af321c not found: ID does not exist" Nov 03 01:00:33.329619 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:00:33.329556 4939 scope.go:122] "RemoveContainer" containerID="61535bce52894eba09afa6c0ff603a245743d80cdd1df53f4d77b402ba744723" Nov 03 01:00:33.341188 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:00:33.341168 4939 scope.go:122] "RemoveContainer" containerID="61535bce52894eba09afa6c0ff603a245743d80cdd1df53f4d77b402ba744723" Nov 03 01:00:33.341524 ip-10-0-140-107 kubenswrapper[4939]: E1103 01:00:33.341501 4939 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"61535bce52894eba09afa6c0ff603a245743d80cdd1df53f4d77b402ba744723\": container with ID starting with 61535bce52894eba09afa6c0ff603a245743d80cdd1df53f4d77b402ba744723 not found: ID does not exist" containerID="61535bce52894eba09afa6c0ff603a245743d80cdd1df53f4d77b402ba744723" Nov 03 01:00:33.341626 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:00:33.341531 4939 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"61535bce52894eba09afa6c0ff603a245743d80cdd1df53f4d77b402ba744723"} err="failed to get container status \"61535bce52894eba09afa6c0ff603a245743d80cdd1df53f4d77b402ba744723\": rpc error: code = NotFound desc = could not find container \"61535bce52894eba09afa6c0ff603a245743d80cdd1df53f4d77b402ba744723\": container with ID starting with 61535bce52894eba09afa6c0ff603a245743d80cdd1df53f4d77b402ba744723 not found: ID does not exist" Nov 03 01:00:33.351223 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:00:33.351190 4939 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-650be0-openshift-default-5c7c5fb7d9-vj4qt"] Nov 03 01:00:33.376938 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:00:33.372195 4939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3c4531e2-d1cc-430f-8245-f129e486d172-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3c4531e2-d1cc-430f-8245-f129e486d172-workload-certs\") pod \"3c4531e2-d1cc-430f-8245-f129e486d172\" (UID: \"3c4531e2-d1cc-430f-8245-f129e486d172\") " Nov 03 01:00:33.376938 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:00:33.372364 4939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3c4531e2-d1cc-430f-8245-f129e486d172-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3c4531e2-d1cc-430f-8245-f129e486d172-credential-socket\") pod \"3c4531e2-d1cc-430f-8245-f129e486d172\" (UID: \"3c4531e2-d1cc-430f-8245-f129e486d172\") " Nov 03 01:00:33.376938 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:00:33.372452 4939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3c4531e2-d1cc-430f-8245-f129e486d172-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3c4531e2-d1cc-430f-8245-f129e486d172-istiod-ca-cert\") pod \"3c4531e2-d1cc-430f-8245-f129e486d172\" (UID: \"3c4531e2-d1cc-430f-8245-f129e486d172\") " Nov 03 01:00:33.376938 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:00:33.372478 4939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3c4531e2-d1cc-430f-8245-f129e486d172-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3c4531e2-d1cc-430f-8245-f129e486d172-istio-envoy\") pod \"3c4531e2-d1cc-430f-8245-f129e486d172\" (UID: \"3c4531e2-d1cc-430f-8245-f129e486d172\") " Nov 03 01:00:33.376938 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:00:33.372541 4939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3c4531e2-d1cc-430f-8245-f129e486d172-kube-api-access-thsvt\" (UniqueName: \"kubernetes.io/projected/3c4531e2-d1cc-430f-8245-f129e486d172-kube-api-access-thsvt\") pod \"3c4531e2-d1cc-430f-8245-f129e486d172\" (UID: \"3c4531e2-d1cc-430f-8245-f129e486d172\") " Nov 03 01:00:33.376938 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:00:33.372594 4939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3c4531e2-d1cc-430f-8245-f129e486d172-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3c4531e2-d1cc-430f-8245-f129e486d172-istio-data\") pod \"3c4531e2-d1cc-430f-8245-f129e486d172\" (UID: \"3c4531e2-d1cc-430f-8245-f129e486d172\") " Nov 03 01:00:33.376938 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:00:33.372740 4939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3c4531e2-d1cc-430f-8245-f129e486d172-istio-token\" (UniqueName: \"kubernetes.io/projected/3c4531e2-d1cc-430f-8245-f129e486d172-istio-token\") pod \"3c4531e2-d1cc-430f-8245-f129e486d172\" (UID: \"3c4531e2-d1cc-430f-8245-f129e486d172\") " Nov 03 01:00:33.376938 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:00:33.372810 4939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3c4531e2-d1cc-430f-8245-f129e486d172-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3c4531e2-d1cc-430f-8245-f129e486d172-workload-socket\") pod \"3c4531e2-d1cc-430f-8245-f129e486d172\" (UID: \"3c4531e2-d1cc-430f-8245-f129e486d172\") " Nov 03 01:00:33.376938 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:00:33.372846 4939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/3c4531e2-d1cc-430f-8245-f129e486d172-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3c4531e2-d1cc-430f-8245-f129e486d172-istio-podinfo\") pod \"3c4531e2-d1cc-430f-8245-f129e486d172\" (UID: \"3c4531e2-d1cc-430f-8245-f129e486d172\") " Nov 03 01:00:33.376938 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:00:33.374713 4939 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5b6c2a38-1b62-41a5-803f-3111bd896857-istio-token\") on node \"ip-10-0-140-107.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:00:33.376938 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:00:33.374743 4939 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5b6c2a38-1b62-41a5-803f-3111bd896857-istio-podinfo\") on node \"ip-10-0-140-107.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:00:33.376938 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:00:33.374757 4939 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5b6c2a38-1b62-41a5-803f-3111bd896857-istio-envoy\") on node \"ip-10-0-140-107.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:00:33.376938 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:00:33.374772 4939 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zn9t7\" (UniqueName: \"kubernetes.io/projected/5b6c2a38-1b62-41a5-803f-3111bd896857-kube-api-access-zn9t7\") on node \"ip-10-0-140-107.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:00:33.376938 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:00:33.375751 4939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3c4531e2-d1cc-430f-8245-f129e486d172-istio-data" pod "3c4531e2-d1cc-430f-8245-f129e486d172" (UID: "3c4531e2-d1cc-430f-8245-f129e486d172"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 01:00:33.376938 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:00:33.376126 4939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3c4531e2-d1cc-430f-8245-f129e486d172-credential-socket" pod "3c4531e2-d1cc-430f-8245-f129e486d172" (UID: "3c4531e2-d1cc-430f-8245-f129e486d172"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 01:00:33.376938 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:00:33.376316 4939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3c4531e2-d1cc-430f-8245-f129e486d172-workload-certs" pod "3c4531e2-d1cc-430f-8245-f129e486d172" (UID: "3c4531e2-d1cc-430f-8245-f129e486d172"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 01:00:33.376938 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:00:33.376768 4939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3c4531e2-d1cc-430f-8245-f129e486d172-workload-socket" pod "3c4531e2-d1cc-430f-8245-f129e486d172" (UID: "3c4531e2-d1cc-430f-8245-f129e486d172"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 01:00:33.378769 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:00:33.377111 4939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3c4531e2-d1cc-430f-8245-f129e486d172-istiod-ca-cert" pod "3c4531e2-d1cc-430f-8245-f129e486d172" (UID: "3c4531e2-d1cc-430f-8245-f129e486d172"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 01:00:33.384570 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:00:33.383177 4939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3c4531e2-d1cc-430f-8245-f129e486d172-istio-token" pod "3c4531e2-d1cc-430f-8245-f129e486d172" (UID: "3c4531e2-d1cc-430f-8245-f129e486d172"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 01:00:33.384570 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:00:33.383761 4939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/3c4531e2-d1cc-430f-8245-f129e486d172-istio-podinfo" pod "3c4531e2-d1cc-430f-8245-f129e486d172" (UID: "3c4531e2-d1cc-430f-8245-f129e486d172"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 01:00:33.384570 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:00:33.383893 4939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3c4531e2-d1cc-430f-8245-f129e486d172-kube-api-access-thsvt" pod "3c4531e2-d1cc-430f-8245-f129e486d172" (UID: "3c4531e2-d1cc-430f-8245-f129e486d172"). InnerVolumeSpecName "kube-api-access-thsvt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 01:00:33.384570 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:00:33.384275 4939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3c4531e2-d1cc-430f-8245-f129e486d172-istio-envoy" pod "3c4531e2-d1cc-430f-8245-f129e486d172" (UID: "3c4531e2-d1cc-430f-8245-f129e486d172"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 01:00:33.386129 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:00:33.386065 4939 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-650be0-openshift-default-5c7c5fb7d9-vj4qt"] Nov 03 01:00:33.476485 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:00:33.476355 4939 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3c4531e2-d1cc-430f-8245-f129e486d172-istio-token\") on node \"ip-10-0-140-107.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:00:33.476485 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:00:33.476397 4939 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3c4531e2-d1cc-430f-8245-f129e486d172-workload-socket\") on node \"ip-10-0-140-107.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:00:33.476485 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:00:33.476414 4939 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3c4531e2-d1cc-430f-8245-f129e486d172-istio-podinfo\") on node \"ip-10-0-140-107.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:00:33.476485 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:00:33.476437 4939 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3c4531e2-d1cc-430f-8245-f129e486d172-workload-certs\") on node \"ip-10-0-140-107.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:00:33.476485 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:00:33.476453 4939 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3c4531e2-d1cc-430f-8245-f129e486d172-credential-socket\") on node \"ip-10-0-140-107.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:00:33.476485 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:00:33.476468 4939 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3c4531e2-d1cc-430f-8245-f129e486d172-istiod-ca-cert\") on node \"ip-10-0-140-107.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:00:33.476485 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:00:33.476482 4939 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3c4531e2-d1cc-430f-8245-f129e486d172-istio-envoy\") on node \"ip-10-0-140-107.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:00:33.476485 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:00:33.476499 4939 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-thsvt\" (UniqueName: \"kubernetes.io/projected/3c4531e2-d1cc-430f-8245-f129e486d172-kube-api-access-thsvt\") on node \"ip-10-0-140-107.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:00:33.476485 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:00:33.476513 4939 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3c4531e2-d1cc-430f-8245-f129e486d172-istio-data\") on node \"ip-10-0-140-107.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:00:33.652506 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:00:33.649331 4939 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-650be0-openshift-default-5c7c5fb7d9-267h5"] Nov 03 01:00:33.652506 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:00:33.651744 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:00:33.654652 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:00:33.654622 4939 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-650be0-openshift-default-5c7c5fb7d9-267h5"] Nov 03 01:00:33.878362 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:00:33.877279 4939 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="12ba5035-bd74-4162-9c8e-9353527f757f" path="/var/lib/kubelet/pods/12ba5035-bd74-4162-9c8e-9353527f757f/volumes" Nov 03 01:00:33.878519 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:00:33.878394 4939 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3c4531e2-d1cc-430f-8245-f129e486d172" path="/var/lib/kubelet/pods/3c4531e2-d1cc-430f-8245-f129e486d172/volumes" Nov 03 01:00:33.879647 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:00:33.879161 4939 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5b6c2a38-1b62-41a5-803f-3111bd896857" path="/var/lib/kubelet/pods/5b6c2a38-1b62-41a5-803f-3111bd896857/volumes" Nov 03 01:00:33.882073 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:00:33.880277 4939 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d5b08c7b-b02e-434f-8b32-32a502a92e69" path="/var/lib/kubelet/pods/d5b08c7b-b02e-434f-8b32-32a502a92e69/volumes" Nov 03 01:00:33.986388 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:00:33.986354 4939 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-47369d-openshift-default-696ccdd997-9mqtd" Nov 03 01:00:34.096090 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:00:34.096059 4939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/49df4a8a-81d6-4c9e-baf7-40c0f7cbaae1-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/49df4a8a-81d6-4c9e-baf7-40c0f7cbaae1-workload-socket\") pod \"49df4a8a-81d6-4c9e-baf7-40c0f7cbaae1\" (UID: \"49df4a8a-81d6-4c9e-baf7-40c0f7cbaae1\") " Nov 03 01:00:34.096291 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:00:34.096098 4939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/49df4a8a-81d6-4c9e-baf7-40c0f7cbaae1-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/49df4a8a-81d6-4c9e-baf7-40c0f7cbaae1-istiod-ca-cert\") pod \"49df4a8a-81d6-4c9e-baf7-40c0f7cbaae1\" (UID: \"49df4a8a-81d6-4c9e-baf7-40c0f7cbaae1\") " Nov 03 01:00:34.096291 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:00:34.096204 4939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/49df4a8a-81d6-4c9e-baf7-40c0f7cbaae1-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/49df4a8a-81d6-4c9e-baf7-40c0f7cbaae1-istio-envoy\") pod \"49df4a8a-81d6-4c9e-baf7-40c0f7cbaae1\" (UID: \"49df4a8a-81d6-4c9e-baf7-40c0f7cbaae1\") " Nov 03 01:00:34.096291 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:00:34.096235 4939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/49df4a8a-81d6-4c9e-baf7-40c0f7cbaae1-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/49df4a8a-81d6-4c9e-baf7-40c0f7cbaae1-credential-socket\") pod \"49df4a8a-81d6-4c9e-baf7-40c0f7cbaae1\" (UID: \"49df4a8a-81d6-4c9e-baf7-40c0f7cbaae1\") " Nov 03 01:00:34.096470 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:00:34.096302 4939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/49df4a8a-81d6-4c9e-baf7-40c0f7cbaae1-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/49df4a8a-81d6-4c9e-baf7-40c0f7cbaae1-workload-certs\") pod \"49df4a8a-81d6-4c9e-baf7-40c0f7cbaae1\" (UID: \"49df4a8a-81d6-4c9e-baf7-40c0f7cbaae1\") " Nov 03 01:00:34.096470 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:00:34.096348 4939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/49df4a8a-81d6-4c9e-baf7-40c0f7cbaae1-kube-api-access-gnbms\" (UniqueName: \"kubernetes.io/projected/49df4a8a-81d6-4c9e-baf7-40c0f7cbaae1-kube-api-access-gnbms\") pod \"49df4a8a-81d6-4c9e-baf7-40c0f7cbaae1\" (UID: \"49df4a8a-81d6-4c9e-baf7-40c0f7cbaae1\") " Nov 03 01:00:34.096470 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:00:34.096384 4939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/49df4a8a-81d6-4c9e-baf7-40c0f7cbaae1-workload-socket" pod "49df4a8a-81d6-4c9e-baf7-40c0f7cbaae1" (UID: "49df4a8a-81d6-4c9e-baf7-40c0f7cbaae1"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 01:00:34.096470 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:00:34.096412 4939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/49df4a8a-81d6-4c9e-baf7-40c0f7cbaae1-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/49df4a8a-81d6-4c9e-baf7-40c0f7cbaae1-istio-data\") pod \"49df4a8a-81d6-4c9e-baf7-40c0f7cbaae1\" (UID: \"49df4a8a-81d6-4c9e-baf7-40c0f7cbaae1\") " Nov 03 01:00:34.096685 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:00:34.096484 4939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/49df4a8a-81d6-4c9e-baf7-40c0f7cbaae1-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/49df4a8a-81d6-4c9e-baf7-40c0f7cbaae1-istio-podinfo\") pod \"49df4a8a-81d6-4c9e-baf7-40c0f7cbaae1\" (UID: \"49df4a8a-81d6-4c9e-baf7-40c0f7cbaae1\") " Nov 03 01:00:34.096685 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:00:34.096519 4939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/49df4a8a-81d6-4c9e-baf7-40c0f7cbaae1-istio-token\" (UniqueName: \"kubernetes.io/projected/49df4a8a-81d6-4c9e-baf7-40c0f7cbaae1-istio-token\") pod \"49df4a8a-81d6-4c9e-baf7-40c0f7cbaae1\" (UID: \"49df4a8a-81d6-4c9e-baf7-40c0f7cbaae1\") " Nov 03 01:00:34.096685 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:00:34.096606 4939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/49df4a8a-81d6-4c9e-baf7-40c0f7cbaae1-workload-certs" pod "49df4a8a-81d6-4c9e-baf7-40c0f7cbaae1" (UID: "49df4a8a-81d6-4c9e-baf7-40c0f7cbaae1"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 01:00:34.096685 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:00:34.096620 4939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/49df4a8a-81d6-4c9e-baf7-40c0f7cbaae1-credential-socket" pod "49df4a8a-81d6-4c9e-baf7-40c0f7cbaae1" (UID: "49df4a8a-81d6-4c9e-baf7-40c0f7cbaae1"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 01:00:34.096685 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:00:34.096614 4939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/49df4a8a-81d6-4c9e-baf7-40c0f7cbaae1-istiod-ca-cert" pod "49df4a8a-81d6-4c9e-baf7-40c0f7cbaae1" (UID: "49df4a8a-81d6-4c9e-baf7-40c0f7cbaae1"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 01:00:34.096968 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:00:34.096751 4939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/49df4a8a-81d6-4c9e-baf7-40c0f7cbaae1-istio-data" pod "49df4a8a-81d6-4c9e-baf7-40c0f7cbaae1" (UID: "49df4a8a-81d6-4c9e-baf7-40c0f7cbaae1"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 01:00:34.097321 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:00:34.097300 4939 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/49df4a8a-81d6-4c9e-baf7-40c0f7cbaae1-workload-socket\") on node \"ip-10-0-140-107.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:00:34.097321 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:00:34.097322 4939 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/49df4a8a-81d6-4c9e-baf7-40c0f7cbaae1-istiod-ca-cert\") on node \"ip-10-0-140-107.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:00:34.097477 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:00:34.097331 4939 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/49df4a8a-81d6-4c9e-baf7-40c0f7cbaae1-credential-socket\") on node \"ip-10-0-140-107.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:00:34.097477 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:00:34.097340 4939 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/49df4a8a-81d6-4c9e-baf7-40c0f7cbaae1-workload-certs\") on node \"ip-10-0-140-107.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:00:34.097477 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:00:34.097348 4939 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/49df4a8a-81d6-4c9e-baf7-40c0f7cbaae1-istio-data\") on node \"ip-10-0-140-107.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:00:34.099347 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:00:34.099322 4939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/49df4a8a-81d6-4c9e-baf7-40c0f7cbaae1-istio-token" pod "49df4a8a-81d6-4c9e-baf7-40c0f7cbaae1" (UID: "49df4a8a-81d6-4c9e-baf7-40c0f7cbaae1"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 01:00:34.099456 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:00:34.099351 4939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/49df4a8a-81d6-4c9e-baf7-40c0f7cbaae1-kube-api-access-gnbms" pod "49df4a8a-81d6-4c9e-baf7-40c0f7cbaae1" (UID: "49df4a8a-81d6-4c9e-baf7-40c0f7cbaae1"). InnerVolumeSpecName "kube-api-access-gnbms". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 01:00:34.099456 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:00:34.099345 4939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/49df4a8a-81d6-4c9e-baf7-40c0f7cbaae1-istio-podinfo" pod "49df4a8a-81d6-4c9e-baf7-40c0f7cbaae1" (UID: "49df4a8a-81d6-4c9e-baf7-40c0f7cbaae1"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 01:00:34.099745 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:00:34.099729 4939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/49df4a8a-81d6-4c9e-baf7-40c0f7cbaae1-istio-envoy" pod "49df4a8a-81d6-4c9e-baf7-40c0f7cbaae1" (UID: "49df4a8a-81d6-4c9e-baf7-40c0f7cbaae1"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 01:00:34.199621 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:00:34.199586 4939 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/49df4a8a-81d6-4c9e-baf7-40c0f7cbaae1-istio-envoy\") on node \"ip-10-0-140-107.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:00:34.199989 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:00:34.199624 4939 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gnbms\" (UniqueName: \"kubernetes.io/projected/49df4a8a-81d6-4c9e-baf7-40c0f7cbaae1-kube-api-access-gnbms\") on node \"ip-10-0-140-107.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:00:34.199989 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:00:34.199643 4939 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/49df4a8a-81d6-4c9e-baf7-40c0f7cbaae1-istio-podinfo\") on node \"ip-10-0-140-107.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:00:34.199989 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:00:34.199658 4939 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/49df4a8a-81d6-4c9e-baf7-40c0f7cbaae1-istio-token\") on node \"ip-10-0-140-107.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:00:34.326508 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:00:34.326474 4939 generic.go:356] "Generic (PLEG): container finished" podID="49df4a8a-81d6-4c9e-baf7-40c0f7cbaae1" containerID="4f759ea70471a8a4d57a74305e9917ce0eedab9b94f41568f1f13462bf9fba8d" exitCode=0 Nov 03 01:00:34.326670 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:00:34.326533 4939 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-47369d-openshift-default-696ccdd997-9mqtd" Nov 03 01:00:34.326670 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:00:34.326558 4939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-47369d-openshift-default-696ccdd997-9mqtd" event={"ID":"49df4a8a-81d6-4c9e-baf7-40c0f7cbaae1","Type":"ContainerDied","Data":"4f759ea70471a8a4d57a74305e9917ce0eedab9b94f41568f1f13462bf9fba8d"} Nov 03 01:00:34.326670 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:00:34.326613 4939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-47369d-openshift-default-696ccdd997-9mqtd" event={"ID":"49df4a8a-81d6-4c9e-baf7-40c0f7cbaae1","Type":"ContainerDied","Data":"cd0e55f9205d8d8122f7868cd8cae0c9c807a0881bfab6b65bee314cbcb6ad9d"} Nov 03 01:00:34.326670 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:00:34.326645 4939 scope.go:122] "RemoveContainer" containerID="4f759ea70471a8a4d57a74305e9917ce0eedab9b94f41568f1f13462bf9fba8d" Nov 03 01:00:34.340192 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:00:34.339976 4939 scope.go:122] "RemoveContainer" containerID="2a7c98c6f2826c0cc792ea5c299ab7bf08c0753c213be3dced1475e3c2330a95" Nov 03 01:00:34.353168 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:00:34.353144 4939 scope.go:122] "RemoveContainer" containerID="4f759ea70471a8a4d57a74305e9917ce0eedab9b94f41568f1f13462bf9fba8d" Nov 03 01:00:34.353594 ip-10-0-140-107 kubenswrapper[4939]: E1103 01:00:34.353529 4939 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4f759ea70471a8a4d57a74305e9917ce0eedab9b94f41568f1f13462bf9fba8d\": container with ID starting with 4f759ea70471a8a4d57a74305e9917ce0eedab9b94f41568f1f13462bf9fba8d not found: ID does not exist" containerID="4f759ea70471a8a4d57a74305e9917ce0eedab9b94f41568f1f13462bf9fba8d" Nov 03 01:00:34.353798 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:00:34.353772 4939 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-47369d-openshift-default-696ccdd997-9mqtd"] Nov 03 01:00:34.353798 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:00:34.353730 4939 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4f759ea70471a8a4d57a74305e9917ce0eedab9b94f41568f1f13462bf9fba8d"} err="failed to get container status \"4f759ea70471a8a4d57a74305e9917ce0eedab9b94f41568f1f13462bf9fba8d\": rpc error: code = NotFound desc = could not find container \"4f759ea70471a8a4d57a74305e9917ce0eedab9b94f41568f1f13462bf9fba8d\": container with ID starting with 4f759ea70471a8a4d57a74305e9917ce0eedab9b94f41568f1f13462bf9fba8d not found: ID does not exist" Nov 03 01:00:34.353798 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:00:34.353827 4939 scope.go:122] "RemoveContainer" containerID="2a7c98c6f2826c0cc792ea5c299ab7bf08c0753c213be3dced1475e3c2330a95" Nov 03 01:00:34.354934 ip-10-0-140-107 kubenswrapper[4939]: E1103 01:00:34.354888 4939 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2a7c98c6f2826c0cc792ea5c299ab7bf08c0753c213be3dced1475e3c2330a95\": container with ID starting with 2a7c98c6f2826c0cc792ea5c299ab7bf08c0753c213be3dced1475e3c2330a95 not found: ID does not exist" containerID="2a7c98c6f2826c0cc792ea5c299ab7bf08c0753c213be3dced1475e3c2330a95" Nov 03 01:00:34.355020 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:00:34.354942 4939 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2a7c98c6f2826c0cc792ea5c299ab7bf08c0753c213be3dced1475e3c2330a95"} err="failed to get container status \"2a7c98c6f2826c0cc792ea5c299ab7bf08c0753c213be3dced1475e3c2330a95\": rpc error: code = NotFound desc = could not find container \"2a7c98c6f2826c0cc792ea5c299ab7bf08c0753c213be3dced1475e3c2330a95\": container with ID starting with 2a7c98c6f2826c0cc792ea5c299ab7bf08c0753c213be3dced1475e3c2330a95 not found: ID does not exist" Nov 03 01:00:34.358781 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:00:34.358756 4939 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-b-47369d-openshift-default-696ccdd997-9mqtd"] Nov 03 01:00:34.641657 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:00:34.641562 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:00:34.794678 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:00:34.794581 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:00:35.091984 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:00:35.090995 4939 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-b-02e4c0-openshift-default-c64cd65d9-7l2h4"] Nov 03 01:00:35.092161 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:00:35.092013 4939 cpu_manager.go:414] "containerMap: removing container" podUID="49df4a8a-81d6-4c9e-baf7-40c0f7cbaae1" containerName="istio-proxy" Nov 03 01:00:35.092161 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:00:35.092034 4939 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="49df4a8a-81d6-4c9e-baf7-40c0f7cbaae1" containerName="istio-proxy" Nov 03 01:00:35.092161 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:00:35.092051 4939 cpu_manager.go:414] "containerMap: removing container" podUID="d5b08c7b-b02e-434f-8b32-32a502a92e69" containerName="payload-processing" Nov 03 01:00:35.092161 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:00:35.092058 4939 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d5b08c7b-b02e-434f-8b32-32a502a92e69" containerName="payload-processing" Nov 03 01:00:35.092161 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:00:35.092072 4939 cpu_manager.go:414] "containerMap: removing container" podUID="3c4531e2-d1cc-430f-8245-f129e486d172" containerName="istio-proxy" Nov 03 01:00:35.092161 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:00:35.092079 4939 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3c4531e2-d1cc-430f-8245-f129e486d172" containerName="istio-proxy" Nov 03 01:00:35.092161 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:00:35.092121 4939 cpu_manager.go:414] "containerMap: removing container" podUID="5b6c2a38-1b62-41a5-803f-3111bd896857" containerName="istio-proxy" Nov 03 01:00:35.092161 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:00:35.092129 4939 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5b6c2a38-1b62-41a5-803f-3111bd896857" containerName="istio-proxy" Nov 03 01:00:35.092161 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:00:35.092143 4939 cpu_manager.go:414] "containerMap: removing container" podUID="12ba5035-bd74-4162-9c8e-9353527f757f" containerName="payload-pre-processing" Nov 03 01:00:35.092161 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:00:35.092149 4939 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="12ba5035-bd74-4162-9c8e-9353527f757f" containerName="payload-pre-processing" Nov 03 01:00:35.092659 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:00:35.092327 4939 memory_manager.go:365] "RemoveStaleState removing state" podUID="49df4a8a-81d6-4c9e-baf7-40c0f7cbaae1" containerName="istio-proxy" Nov 03 01:00:35.092659 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:00:35.092343 4939 memory_manager.go:365] "RemoveStaleState removing state" podUID="49df4a8a-81d6-4c9e-baf7-40c0f7cbaae1" containerName="istio-proxy" Nov 03 01:00:35.092659 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:00:35.092356 4939 memory_manager.go:365] "RemoveStaleState removing state" podUID="d5b08c7b-b02e-434f-8b32-32a502a92e69" containerName="payload-processing" Nov 03 01:00:35.092659 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:00:35.092365 4939 memory_manager.go:365] "RemoveStaleState removing state" podUID="3c4531e2-d1cc-430f-8245-f129e486d172" containerName="istio-proxy" Nov 03 01:00:35.092659 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:00:35.092383 4939 memory_manager.go:365] "RemoveStaleState removing state" podUID="5b6c2a38-1b62-41a5-803f-3111bd896857" containerName="istio-proxy" Nov 03 01:00:35.092659 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:00:35.092392 4939 memory_manager.go:365] "RemoveStaleState removing state" podUID="12ba5035-bd74-4162-9c8e-9353527f757f" containerName="payload-pre-processing" Nov 03 01:00:35.103010 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:00:35.102983 4939 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-02e4c0-openshift-default-c64cd65d9-7l2h4" Nov 03 01:00:35.105676 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:00:35.104436 4939 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-b-02e4c0-openshift-default-c64cd65d9-7l2h4"] Nov 03 01:00:35.108621 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:00:35.108025 4939 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ipp-b-02e4c0-openshift-default-dockercfg-nlnl5\"" Nov 03 01:00:35.149433 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:00:35.149398 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fvcp7\" (UniqueName: \"kubernetes.io/projected/2df5fd65-3b2a-448e-92a8-f07a8888692d-kube-api-access-fvcp7\") pod \"e2e-ipp-b-02e4c0-openshift-default-c64cd65d9-7l2h4\" (UID: \"2df5fd65-3b2a-448e-92a8-f07a8888692d\") " pod="openshift-ingress/e2e-ipp-b-02e4c0-openshift-default-c64cd65d9-7l2h4" Nov 03 01:00:35.149672 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:00:35.149567 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2df5fd65-3b2a-448e-92a8-f07a8888692d-workload-certs\") pod \"e2e-ipp-b-02e4c0-openshift-default-c64cd65d9-7l2h4\" (UID: \"2df5fd65-3b2a-448e-92a8-f07a8888692d\") " pod="openshift-ingress/e2e-ipp-b-02e4c0-openshift-default-c64cd65d9-7l2h4" Nov 03 01:00:35.149672 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:00:35.149608 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2df5fd65-3b2a-448e-92a8-f07a8888692d-istio-envoy\") pod \"e2e-ipp-b-02e4c0-openshift-default-c64cd65d9-7l2h4\" (UID: \"2df5fd65-3b2a-448e-92a8-f07a8888692d\") " pod="openshift-ingress/e2e-ipp-b-02e4c0-openshift-default-c64cd65d9-7l2h4" Nov 03 01:00:35.149672 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:00:35.149634 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2df5fd65-3b2a-448e-92a8-f07a8888692d-istio-data\") pod \"e2e-ipp-b-02e4c0-openshift-default-c64cd65d9-7l2h4\" (UID: \"2df5fd65-3b2a-448e-92a8-f07a8888692d\") " pod="openshift-ingress/e2e-ipp-b-02e4c0-openshift-default-c64cd65d9-7l2h4" Nov 03 01:00:35.149796 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:00:35.149711 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2df5fd65-3b2a-448e-92a8-f07a8888692d-istio-podinfo\") pod \"e2e-ipp-b-02e4c0-openshift-default-c64cd65d9-7l2h4\" (UID: \"2df5fd65-3b2a-448e-92a8-f07a8888692d\") " pod="openshift-ingress/e2e-ipp-b-02e4c0-openshift-default-c64cd65d9-7l2h4" Nov 03 01:00:35.149860 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:00:35.149842 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2df5fd65-3b2a-448e-92a8-f07a8888692d-istio-token\") pod \"e2e-ipp-b-02e4c0-openshift-default-c64cd65d9-7l2h4\" (UID: \"2df5fd65-3b2a-448e-92a8-f07a8888692d\") " pod="openshift-ingress/e2e-ipp-b-02e4c0-openshift-default-c64cd65d9-7l2h4" Nov 03 01:00:35.149940 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:00:35.149893 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2df5fd65-3b2a-448e-92a8-f07a8888692d-workload-socket\") pod \"e2e-ipp-b-02e4c0-openshift-default-c64cd65d9-7l2h4\" (UID: \"2df5fd65-3b2a-448e-92a8-f07a8888692d\") " pod="openshift-ingress/e2e-ipp-b-02e4c0-openshift-default-c64cd65d9-7l2h4" Nov 03 01:00:35.149985 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:00:35.149963 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2df5fd65-3b2a-448e-92a8-f07a8888692d-credential-socket\") pod \"e2e-ipp-b-02e4c0-openshift-default-c64cd65d9-7l2h4\" (UID: \"2df5fd65-3b2a-448e-92a8-f07a8888692d\") " pod="openshift-ingress/e2e-ipp-b-02e4c0-openshift-default-c64cd65d9-7l2h4" Nov 03 01:00:35.150134 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:00:35.150114 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2df5fd65-3b2a-448e-92a8-f07a8888692d-istiod-ca-cert\") pod \"e2e-ipp-b-02e4c0-openshift-default-c64cd65d9-7l2h4\" (UID: \"2df5fd65-3b2a-448e-92a8-f07a8888692d\") " pod="openshift-ingress/e2e-ipp-b-02e4c0-openshift-default-c64cd65d9-7l2h4" Nov 03 01:00:35.191492 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:00:35.191453 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:00:35.251227 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:00:35.251188 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2df5fd65-3b2a-448e-92a8-f07a8888692d-istiod-ca-cert\") pod \"e2e-ipp-b-02e4c0-openshift-default-c64cd65d9-7l2h4\" (UID: \"2df5fd65-3b2a-448e-92a8-f07a8888692d\") " pod="openshift-ingress/e2e-ipp-b-02e4c0-openshift-default-c64cd65d9-7l2h4" Nov 03 01:00:35.251678 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:00:35.251269 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fvcp7\" (UniqueName: \"kubernetes.io/projected/2df5fd65-3b2a-448e-92a8-f07a8888692d-kube-api-access-fvcp7\") pod \"e2e-ipp-b-02e4c0-openshift-default-c64cd65d9-7l2h4\" (UID: \"2df5fd65-3b2a-448e-92a8-f07a8888692d\") " pod="openshift-ingress/e2e-ipp-b-02e4c0-openshift-default-c64cd65d9-7l2h4" Nov 03 01:00:35.251678 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:00:35.251362 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2df5fd65-3b2a-448e-92a8-f07a8888692d-workload-certs\") pod \"e2e-ipp-b-02e4c0-openshift-default-c64cd65d9-7l2h4\" (UID: \"2df5fd65-3b2a-448e-92a8-f07a8888692d\") " pod="openshift-ingress/e2e-ipp-b-02e4c0-openshift-default-c64cd65d9-7l2h4" Nov 03 01:00:35.251678 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:00:35.251391 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2df5fd65-3b2a-448e-92a8-f07a8888692d-istio-envoy\") pod \"e2e-ipp-b-02e4c0-openshift-default-c64cd65d9-7l2h4\" (UID: \"2df5fd65-3b2a-448e-92a8-f07a8888692d\") " pod="openshift-ingress/e2e-ipp-b-02e4c0-openshift-default-c64cd65d9-7l2h4" Nov 03 01:00:35.251678 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:00:35.251412 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2df5fd65-3b2a-448e-92a8-f07a8888692d-istio-data\") pod \"e2e-ipp-b-02e4c0-openshift-default-c64cd65d9-7l2h4\" (UID: \"2df5fd65-3b2a-448e-92a8-f07a8888692d\") " pod="openshift-ingress/e2e-ipp-b-02e4c0-openshift-default-c64cd65d9-7l2h4" Nov 03 01:00:35.251678 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:00:35.251449 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2df5fd65-3b2a-448e-92a8-f07a8888692d-istio-podinfo\") pod \"e2e-ipp-b-02e4c0-openshift-default-c64cd65d9-7l2h4\" (UID: \"2df5fd65-3b2a-448e-92a8-f07a8888692d\") " pod="openshift-ingress/e2e-ipp-b-02e4c0-openshift-default-c64cd65d9-7l2h4" Nov 03 01:00:35.251678 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:00:35.251578 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2df5fd65-3b2a-448e-92a8-f07a8888692d-istio-token\") pod \"e2e-ipp-b-02e4c0-openshift-default-c64cd65d9-7l2h4\" (UID: \"2df5fd65-3b2a-448e-92a8-f07a8888692d\") " pod="openshift-ingress/e2e-ipp-b-02e4c0-openshift-default-c64cd65d9-7l2h4" Nov 03 01:00:35.251678 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:00:35.251622 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2df5fd65-3b2a-448e-92a8-f07a8888692d-workload-socket\") pod \"e2e-ipp-b-02e4c0-openshift-default-c64cd65d9-7l2h4\" (UID: \"2df5fd65-3b2a-448e-92a8-f07a8888692d\") " pod="openshift-ingress/e2e-ipp-b-02e4c0-openshift-default-c64cd65d9-7l2h4" Nov 03 01:00:35.252071 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:00:35.251691 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2df5fd65-3b2a-448e-92a8-f07a8888692d-credential-socket\") pod \"e2e-ipp-b-02e4c0-openshift-default-c64cd65d9-7l2h4\" (UID: \"2df5fd65-3b2a-448e-92a8-f07a8888692d\") " pod="openshift-ingress/e2e-ipp-b-02e4c0-openshift-default-c64cd65d9-7l2h4" Nov 03 01:00:35.252071 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:00:35.251780 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2df5fd65-3b2a-448e-92a8-f07a8888692d-workload-certs\") pod \"e2e-ipp-b-02e4c0-openshift-default-c64cd65d9-7l2h4\" (UID: \"2df5fd65-3b2a-448e-92a8-f07a8888692d\") " pod="openshift-ingress/e2e-ipp-b-02e4c0-openshift-default-c64cd65d9-7l2h4" Nov 03 01:00:35.252071 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:00:35.251800 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2df5fd65-3b2a-448e-92a8-f07a8888692d-istio-data\") pod \"e2e-ipp-b-02e4c0-openshift-default-c64cd65d9-7l2h4\" (UID: \"2df5fd65-3b2a-448e-92a8-f07a8888692d\") " pod="openshift-ingress/e2e-ipp-b-02e4c0-openshift-default-c64cd65d9-7l2h4" Nov 03 01:00:35.252239 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:00:35.252151 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2df5fd65-3b2a-448e-92a8-f07a8888692d-workload-socket\") pod \"e2e-ipp-b-02e4c0-openshift-default-c64cd65d9-7l2h4\" (UID: \"2df5fd65-3b2a-448e-92a8-f07a8888692d\") " pod="openshift-ingress/e2e-ipp-b-02e4c0-openshift-default-c64cd65d9-7l2h4" Nov 03 01:00:35.252239 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:00:35.252192 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2df5fd65-3b2a-448e-92a8-f07a8888692d-credential-socket\") pod \"e2e-ipp-b-02e4c0-openshift-default-c64cd65d9-7l2h4\" (UID: \"2df5fd65-3b2a-448e-92a8-f07a8888692d\") " pod="openshift-ingress/e2e-ipp-b-02e4c0-openshift-default-c64cd65d9-7l2h4" Nov 03 01:00:35.252396 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:00:35.252372 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2df5fd65-3b2a-448e-92a8-f07a8888692d-istiod-ca-cert\") pod \"e2e-ipp-b-02e4c0-openshift-default-c64cd65d9-7l2h4\" (UID: \"2df5fd65-3b2a-448e-92a8-f07a8888692d\") " pod="openshift-ingress/e2e-ipp-b-02e4c0-openshift-default-c64cd65d9-7l2h4" Nov 03 01:00:35.254358 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:00:35.254331 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2df5fd65-3b2a-448e-92a8-f07a8888692d-istio-envoy\") pod \"e2e-ipp-b-02e4c0-openshift-default-c64cd65d9-7l2h4\" (UID: \"2df5fd65-3b2a-448e-92a8-f07a8888692d\") " pod="openshift-ingress/e2e-ipp-b-02e4c0-openshift-default-c64cd65d9-7l2h4" Nov 03 01:00:35.254567 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:00:35.254547 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2df5fd65-3b2a-448e-92a8-f07a8888692d-istio-podinfo\") pod \"e2e-ipp-b-02e4c0-openshift-default-c64cd65d9-7l2h4\" (UID: \"2df5fd65-3b2a-448e-92a8-f07a8888692d\") " pod="openshift-ingress/e2e-ipp-b-02e4c0-openshift-default-c64cd65d9-7l2h4" Nov 03 01:00:35.265759 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:00:35.265737 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2df5fd65-3b2a-448e-92a8-f07a8888692d-istio-token\") pod \"e2e-ipp-b-02e4c0-openshift-default-c64cd65d9-7l2h4\" (UID: \"2df5fd65-3b2a-448e-92a8-f07a8888692d\") " pod="openshift-ingress/e2e-ipp-b-02e4c0-openshift-default-c64cd65d9-7l2h4" Nov 03 01:00:35.266135 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:00:35.266068 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fvcp7\" (UniqueName: \"kubernetes.io/projected/2df5fd65-3b2a-448e-92a8-f07a8888692d-kube-api-access-fvcp7\") pod \"e2e-ipp-b-02e4c0-openshift-default-c64cd65d9-7l2h4\" (UID: \"2df5fd65-3b2a-448e-92a8-f07a8888692d\") " pod="openshift-ingress/e2e-ipp-b-02e4c0-openshift-default-c64cd65d9-7l2h4" Nov 03 01:00:35.365978 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:00:35.364194 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:00:35.438933 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:00:35.434370 4939 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-02e4c0-openshift-default-c64cd65d9-7l2h4" Nov 03 01:00:35.811189 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:00:35.811145 4939 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-4pw7v"] Nov 03 01:00:35.870722 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:00:35.870683 4939 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="49df4a8a-81d6-4c9e-baf7-40c0f7cbaae1" path="/var/lib/kubelet/pods/49df4a8a-81d6-4c9e-baf7-40c0f7cbaae1/volumes" Nov 03 01:00:35.879620 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:00:35.879313 4939 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-b-02e4c0-openshift-default-c64cd65d9-7l2h4"] Nov 03 01:00:35.882634 ip-10-0-140-107 kubenswrapper[4939]: W1103 01:00:35.882604 4939 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2df5fd65_3b2a_448e_92a8_f07a8888692d.slice/crio-f7a4cdea8b289934af0481f89ad892a2dae979e81f61f19b4e6a8edac9750fb3 WatchSource:0}: Error finding container f7a4cdea8b289934af0481f89ad892a2dae979e81f61f19b4e6a8edac9750fb3: Status 404 returned error can't find the container with id f7a4cdea8b289934af0481f89ad892a2dae979e81f61f19b4e6a8edac9750fb3 Nov 03 01:00:35.883488 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:00:35.883465 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:00:35.886053 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:00:35.886024 4939 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 01:00:35.886153 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:00:35.886091 4939 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 01:00:35.886153 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:00:35.886126 4939 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 01:00:36.047928 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:00:36.043753 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:00:36.197016 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:00:36.196940 4939 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-1e352e-6bb6bb4d7f-hd98j"] Nov 03 01:00:36.197989 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:00:36.197965 4939 cpu_manager.go:414] "containerMap: removing container" podUID="49df4a8a-81d6-4c9e-baf7-40c0f7cbaae1" containerName="istio-proxy" Nov 03 01:00:36.198060 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:00:36.197990 4939 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="49df4a8a-81d6-4c9e-baf7-40c0f7cbaae1" containerName="istio-proxy" Nov 03 01:00:36.202379 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:00:36.202356 4939 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-1e352e-6bb6bb4d7f-hd98j" Nov 03 01:00:36.212333 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:00:36.212000 4939 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-praxis-a-1e352e\"" Nov 03 01:00:36.219231 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:00:36.219183 4939 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-praxis-a-1e352e-dockercfg-tn9q2\"" Nov 03 01:00:36.219839 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:00:36.219810 4939 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-1e352e-6bb6bb4d7f-hd98j"] Nov 03 01:00:36.221574 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:00:36.220432 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:00:36.228674 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:00:36.226397 4939 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-1e352e-585bf55c69-ddbn9"] Nov 03 01:00:36.260056 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:00:36.258235 4939 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-a-1e352e-585bf55c69-ddbn9" Nov 03 01:00:36.279808 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:00:36.279771 4939 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-1e352e-585bf55c69-ddbn9"] Nov 03 01:00:36.377934 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:00:36.375228 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:00:36.449983 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:00:36.444814 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4372e01e-917c-4de9-b2de-3882fcd6d87f-plugins-config-volume\") pod \"payload-processing-e2e-praxis-a-1e352e-585bf55c69-ddbn9\" (UID: \"4372e01e-917c-4de9-b2de-3882fcd6d87f\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-1e352e-585bf55c69-ddbn9" Nov 03 01:00:36.449983 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:00:36.444877 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w2cc6\" (UniqueName: \"kubernetes.io/projected/4372e01e-917c-4de9-b2de-3882fcd6d87f-kube-api-access-w2cc6\") pod \"payload-processing-e2e-praxis-a-1e352e-585bf55c69-ddbn9\" (UID: \"4372e01e-917c-4de9-b2de-3882fcd6d87f\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-1e352e-585bf55c69-ddbn9" Nov 03 01:00:36.449983 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:00:36.444989 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/04358d76-4e0d-4b13-9307-4734a8891250-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-a-1e352e-6bb6bb4d7f-hd98j\" (UID: \"04358d76-4e0d-4b13-9307-4734a8891250\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-1e352e-6bb6bb4d7f-hd98j" Nov 03 01:00:36.449983 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:00:36.445144 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jxd9d\" (UniqueName: \"kubernetes.io/projected/04358d76-4e0d-4b13-9307-4734a8891250-kube-api-access-jxd9d\") pod \"payload-pre-processing-e2e-praxis-a-1e352e-6bb6bb4d7f-hd98j\" (UID: \"04358d76-4e0d-4b13-9307-4734a8891250\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-1e352e-6bb6bb4d7f-hd98j" Nov 03 01:00:36.520929 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:00:36.518062 4939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-02e4c0-openshift-default-c64cd65d9-7l2h4" event={"ID":"2df5fd65-3b2a-448e-92a8-f07a8888692d","Type":"ContainerStarted","Data":"ea409d7fcc0d44e2d14e7b53da7e4f0377490e01ba0cd67ec8b499c4d77a8b20"} Nov 03 01:00:36.520929 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:00:36.518127 4939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-02e4c0-openshift-default-c64cd65d9-7l2h4" event={"ID":"2df5fd65-3b2a-448e-92a8-f07a8888692d","Type":"ContainerStarted","Data":"f7a4cdea8b289934af0481f89ad892a2dae979e81f61f19b4e6a8edac9750fb3"} Nov 03 01:00:36.549934 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:00:36.546358 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jxd9d\" (UniqueName: \"kubernetes.io/projected/04358d76-4e0d-4b13-9307-4734a8891250-kube-api-access-jxd9d\") pod \"payload-pre-processing-e2e-praxis-a-1e352e-6bb6bb4d7f-hd98j\" (UID: \"04358d76-4e0d-4b13-9307-4734a8891250\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-1e352e-6bb6bb4d7f-hd98j" Nov 03 01:00:36.549934 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:00:36.546625 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4372e01e-917c-4de9-b2de-3882fcd6d87f-plugins-config-volume\") pod \"payload-processing-e2e-praxis-a-1e352e-585bf55c69-ddbn9\" (UID: \"4372e01e-917c-4de9-b2de-3882fcd6d87f\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-1e352e-585bf55c69-ddbn9" Nov 03 01:00:36.549934 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:00:36.546660 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w2cc6\" (UniqueName: \"kubernetes.io/projected/4372e01e-917c-4de9-b2de-3882fcd6d87f-kube-api-access-w2cc6\") pod \"payload-processing-e2e-praxis-a-1e352e-585bf55c69-ddbn9\" (UID: \"4372e01e-917c-4de9-b2de-3882fcd6d87f\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-1e352e-585bf55c69-ddbn9" Nov 03 01:00:36.549934 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:00:36.546733 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/04358d76-4e0d-4b13-9307-4734a8891250-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-a-1e352e-6bb6bb4d7f-hd98j\" (UID: \"04358d76-4e0d-4b13-9307-4734a8891250\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-1e352e-6bb6bb4d7f-hd98j" Nov 03 01:00:36.549934 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:00:36.547695 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/04358d76-4e0d-4b13-9307-4734a8891250-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-a-1e352e-6bb6bb4d7f-hd98j\" (UID: \"04358d76-4e0d-4b13-9307-4734a8891250\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-1e352e-6bb6bb4d7f-hd98j" Nov 03 01:00:36.549934 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:00:36.548786 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4372e01e-917c-4de9-b2de-3882fcd6d87f-plugins-config-volume\") pod \"payload-processing-e2e-praxis-a-1e352e-585bf55c69-ddbn9\" (UID: \"4372e01e-917c-4de9-b2de-3882fcd6d87f\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-1e352e-585bf55c69-ddbn9" Nov 03 01:00:36.595011 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:00:36.584678 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jxd9d\" (UniqueName: \"kubernetes.io/projected/04358d76-4e0d-4b13-9307-4734a8891250-kube-api-access-jxd9d\") pod \"payload-pre-processing-e2e-praxis-a-1e352e-6bb6bb4d7f-hd98j\" (UID: \"04358d76-4e0d-4b13-9307-4734a8891250\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-1e352e-6bb6bb4d7f-hd98j" Nov 03 01:00:36.606932 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:00:36.597628 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-w2cc6\" (UniqueName: \"kubernetes.io/projected/4372e01e-917c-4de9-b2de-3882fcd6d87f-kube-api-access-w2cc6\") pod \"payload-processing-e2e-praxis-a-1e352e-585bf55c69-ddbn9\" (UID: \"4372e01e-917c-4de9-b2de-3882fcd6d87f\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-1e352e-585bf55c69-ddbn9" Nov 03 01:00:36.607096 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:00:36.606898 4939 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-b-02e4c0-openshift-default-c64cd65d9-7l2h4" podStartSLOduration=1.6068765900000002 podStartE2EDuration="1.60687659s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 01:00:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 01:00:36.56815375 +0000 UTC m=+1184.832488719" watchObservedRunningTime="2026-11-03 01:00:36.60687659 +0000 UTC m=+1184.871211550" Nov 03 01:00:36.837943 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:00:36.835385 4939 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-1e352e-6bb6bb4d7f-hd98j" Nov 03 01:00:36.837943 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:00:36.836139 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:00:36.924134 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:00:36.919184 4939 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-a-1e352e-585bf55c69-ddbn9" Nov 03 01:00:37.311966 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:00:37.309543 4939 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-1e352e-6bb6bb4d7f-hd98j"] Nov 03 01:00:37.391174 ip-10-0-140-107 kubenswrapper[4939]: W1103 01:00:37.391141 4939 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4372e01e_917c_4de9_b2de_3882fcd6d87f.slice/crio-45ba2f60218e0605a5a18080ebf964248e7363e351db248486d49b633afa0e3f WatchSource:0}: Error finding container 45ba2f60218e0605a5a18080ebf964248e7363e351db248486d49b633afa0e3f: Status 404 returned error can't find the container with id 45ba2f60218e0605a5a18080ebf964248e7363e351db248486d49b633afa0e3f Nov 03 01:00:37.393284 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:00:37.392653 4939 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-1e352e-585bf55c69-ddbn9"] Nov 03 01:00:37.435200 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:00:37.435176 4939 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-b-02e4c0-openshift-default-c64cd65d9-7l2h4" Nov 03 01:00:37.441521 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:00:37.441499 4939 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-b-02e4c0-openshift-default-c64cd65d9-7l2h4" Nov 03 01:00:37.529974 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:00:37.528928 4939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-1e352e-6bb6bb4d7f-hd98j" event={"ID":"04358d76-4e0d-4b13-9307-4734a8891250","Type":"ContainerStarted","Data":"c9eafbc622358d7ef47203bd8ed798c5814452456c54093c9c5ed4242277be91"} Nov 03 01:00:37.529974 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:00:37.528999 4939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-1e352e-6bb6bb4d7f-hd98j" event={"ID":"04358d76-4e0d-4b13-9307-4734a8891250","Type":"ContainerStarted","Data":"65fadd941576d539b543577554ebc34a5016011f8d388fb8a5804c510487d14c"} Nov 03 01:00:37.529974 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:00:37.529131 4939 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-1e352e-6bb6bb4d7f-hd98j" Nov 03 01:00:37.532688 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:00:37.532638 4939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-a-1e352e-585bf55c69-ddbn9" event={"ID":"4372e01e-917c-4de9-b2de-3882fcd6d87f","Type":"ContainerStarted","Data":"beacdbc898060f4136c58383b5353930702aae7e938e46b411609394b0d20973"} Nov 03 01:00:37.532813 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:00:37.532691 4939 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-praxis-a-1e352e-585bf55c69-ddbn9" Nov 03 01:00:37.532813 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:00:37.532713 4939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-a-1e352e-585bf55c69-ddbn9" event={"ID":"4372e01e-917c-4de9-b2de-3882fcd6d87f","Type":"ContainerStarted","Data":"45ba2f60218e0605a5a18080ebf964248e7363e351db248486d49b633afa0e3f"} Nov 03 01:00:37.533120 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:00:37.533078 4939 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-b-02e4c0-openshift-default-c64cd65d9-7l2h4" Nov 03 01:00:37.534280 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:00:37.534259 4939 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-b-02e4c0-openshift-default-c64cd65d9-7l2h4" Nov 03 01:00:37.569958 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:00:37.566893 4939 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-1e352e-6bb6bb4d7f-hd98j" podStartSLOduration=1.566870433 podStartE2EDuration="1.566870433s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 01:00:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 01:00:37.562384139 +0000 UTC m=+1185.826719088" watchObservedRunningTime="2026-11-03 01:00:37.566870433 +0000 UTC m=+1185.831205378" Nov 03 01:00:37.614053 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:00:37.613865 4939 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-praxis-a-1e352e-585bf55c69-ddbn9" podStartSLOduration=1.613846354 podStartE2EDuration="1.613846354s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 01:00:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 01:00:37.610464616 +0000 UTC m=+1185.874799564" watchObservedRunningTime="2026-11-03 01:00:37.613846354 +0000 UTC m=+1185.878181300" Nov 03 01:00:37.625539 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:00:37.625501 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:00:37.970993 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:00:37.969623 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:00:38.180229 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:00:38.180191 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:00:39.091869 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:00:39.091836 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:00:39.317857 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:00:39.317816 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:00:39.517140 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:00:39.517106 4939 patch_prober.go:29] interesting pod/e2e-api-a-7406a9-openshift-default-74c9f8d955-rqrs4 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.112:15021/healthz/ready\": read tcp 10.132.0.2:44864->10.132.0.112:15021: read: connection reset by peer" start-of-body= Nov 03 01:00:39.517284 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:00:39.517166 4939 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-api-a-7406a9-openshift-default-74c9f8d955-rqrs4" podUID="ab2a5b74-e0ad-4a13-9d46-7b78d5ef8aa7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.112:15021/healthz/ready\": read tcp 10.132.0.2:44864->10.132.0.112:15021: read: connection reset by peer" Nov 03 01:00:39.553641 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:00:39.553569 4939 generic.go:356] "Generic (PLEG): container finished" podID="ab2a5b74-e0ad-4a13-9d46-7b78d5ef8aa7" containerID="a6df97fdc2b980db01fd7357a1e20e13cbe9006cb42e82f275f50fe8a992651a" exitCode=0 Nov 03 01:00:39.553641 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:00:39.553628 4939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-7406a9-openshift-default-74c9f8d955-rqrs4" event={"ID":"ab2a5b74-e0ad-4a13-9d46-7b78d5ef8aa7","Type":"ContainerDied","Data":"a6df97fdc2b980db01fd7357a1e20e13cbe9006cb42e82f275f50fe8a992651a"} Nov 03 01:00:39.554103 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:00:39.554089 4939 scope.go:122] "RemoveContainer" containerID="a6df97fdc2b980db01fd7357a1e20e13cbe9006cb42e82f275f50fe8a992651a" Nov 03 01:00:39.555812 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:00:39.555780 4939 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 01:00:39.555933 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:00:39.555867 4939 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 01:00:39.556006 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:00:39.555936 4939 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 01:00:40.312209 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:00:40.309964 4939 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-02e4c0-57ff55f5cf-72j67"] Nov 03 01:00:40.323051 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:00:40.323001 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:00:40.323722 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:00:40.323675 4939 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-02e4c0-57ff55f5cf-72j67" Nov 03 01:00:40.327228 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:00:40.327200 4939 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-02e4c0-57ff55f5cf-72j67"] Nov 03 01:00:40.327964 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:00:40.327887 4939 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-b-02e4c0\"" Nov 03 01:00:40.392133 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:00:40.392098 4939 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-02e4c0-7796479459-fjkx9"] Nov 03 01:00:40.398711 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:00:40.398679 4939 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-02e4c0-7796479459-fjkx9" Nov 03 01:00:40.412252 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:00:40.412227 4939 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-b-02e4c0-dockercfg-smkzb\"" Nov 03 01:00:40.415112 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:00:40.415089 4939 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-02e4c0-7796479459-fjkx9"] Nov 03 01:00:40.464933 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:00:40.461069 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b4h26\" (UniqueName: \"kubernetes.io/projected/34792239-0f53-474d-adbf-fb1205780987-kube-api-access-b4h26\") pod \"payload-pre-processing-e2e-ipp-b-02e4c0-57ff55f5cf-72j67\" (UID: \"34792239-0f53-474d-adbf-fb1205780987\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-02e4c0-57ff55f5cf-72j67" Nov 03 01:00:40.464933 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:00:40.461262 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/34792239-0f53-474d-adbf-fb1205780987-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-02e4c0-57ff55f5cf-72j67\" (UID: \"34792239-0f53-474d-adbf-fb1205780987\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-02e4c0-57ff55f5cf-72j67" Nov 03 01:00:40.464933 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:00:40.461566 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vgf8p\" (UniqueName: \"kubernetes.io/projected/fd866aeb-ea66-4982-b18d-cdc4a1f8a836-kube-api-access-vgf8p\") pod \"payload-processing-e2e-ipp-b-02e4c0-7796479459-fjkx9\" (UID: \"fd866aeb-ea66-4982-b18d-cdc4a1f8a836\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-02e4c0-7796479459-fjkx9" Nov 03 01:00:40.464933 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:00:40.461610 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fd866aeb-ea66-4982-b18d-cdc4a1f8a836-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-02e4c0-7796479459-fjkx9\" (UID: \"fd866aeb-ea66-4982-b18d-cdc4a1f8a836\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-02e4c0-7796479459-fjkx9" Nov 03 01:00:40.591987 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:00:40.573496 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:00:40.636933 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:00:40.594189 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vgf8p\" (UniqueName: \"kubernetes.io/projected/fd866aeb-ea66-4982-b18d-cdc4a1f8a836-kube-api-access-vgf8p\") pod \"payload-processing-e2e-ipp-b-02e4c0-7796479459-fjkx9\" (UID: \"fd866aeb-ea66-4982-b18d-cdc4a1f8a836\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-02e4c0-7796479459-fjkx9" Nov 03 01:00:40.647008 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:00:40.637233 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fd866aeb-ea66-4982-b18d-cdc4a1f8a836-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-02e4c0-7796479459-fjkx9\" (UID: \"fd866aeb-ea66-4982-b18d-cdc4a1f8a836\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-02e4c0-7796479459-fjkx9" Nov 03 01:00:40.647426 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:00:40.647393 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b4h26\" (UniqueName: \"kubernetes.io/projected/34792239-0f53-474d-adbf-fb1205780987-kube-api-access-b4h26\") pod \"payload-pre-processing-e2e-ipp-b-02e4c0-57ff55f5cf-72j67\" (UID: \"34792239-0f53-474d-adbf-fb1205780987\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-02e4c0-57ff55f5cf-72j67" Nov 03 01:00:40.678933 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:00:40.654144 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/34792239-0f53-474d-adbf-fb1205780987-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-02e4c0-57ff55f5cf-72j67\" (UID: \"34792239-0f53-474d-adbf-fb1205780987\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-02e4c0-57ff55f5cf-72j67" Nov 03 01:00:40.679418 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:00:40.656841 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fd866aeb-ea66-4982-b18d-cdc4a1f8a836-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-02e4c0-7796479459-fjkx9\" (UID: \"fd866aeb-ea66-4982-b18d-cdc4a1f8a836\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-02e4c0-7796479459-fjkx9" Nov 03 01:00:40.679597 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:00:40.677557 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/34792239-0f53-474d-adbf-fb1205780987-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-02e4c0-57ff55f5cf-72j67\" (UID: \"34792239-0f53-474d-adbf-fb1205780987\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-02e4c0-57ff55f5cf-72j67" Nov 03 01:00:40.715731 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:00:40.715692 4939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-7406a9-openshift-default-74c9f8d955-rqrs4" event={"ID":"ab2a5b74-e0ad-4a13-9d46-7b78d5ef8aa7","Type":"ContainerStarted","Data":"653e46bdc0ec8b7a25e2ab7986ac2c3cac776da6eba42b8ef8633f9e12bef9bb"} Nov 03 01:00:40.834948 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:00:40.834378 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vgf8p\" (UniqueName: \"kubernetes.io/projected/fd866aeb-ea66-4982-b18d-cdc4a1f8a836-kube-api-access-vgf8p\") pod \"payload-processing-e2e-ipp-b-02e4c0-7796479459-fjkx9\" (UID: \"fd866aeb-ea66-4982-b18d-cdc4a1f8a836\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-02e4c0-7796479459-fjkx9" Nov 03 01:00:40.859977 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:00:40.858682 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b4h26\" (UniqueName: \"kubernetes.io/projected/34792239-0f53-474d-adbf-fb1205780987-kube-api-access-b4h26\") pod \"payload-pre-processing-e2e-ipp-b-02e4c0-57ff55f5cf-72j67\" (UID: \"34792239-0f53-474d-adbf-fb1205780987\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-02e4c0-57ff55f5cf-72j67" Nov 03 01:00:40.879854 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:00:40.879820 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:00:40.944313 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:00:40.944278 4939 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-02e4c0-57ff55f5cf-72j67" Nov 03 01:00:41.018127 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:00:41.017246 4939 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-02e4c0-7796479459-fjkx9" Nov 03 01:00:41.390746 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:00:41.390721 4939 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-02e4c0-57ff55f5cf-72j67"] Nov 03 01:00:41.392009 ip-10-0-140-107 kubenswrapper[4939]: W1103 01:00:41.391972 4939 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod34792239_0f53_474d_adbf_fb1205780987.slice/crio-37fc5aedb78760eba9bb267537e2fc7745ec0dfa3393512447de22453378e41b WatchSource:0}: Error finding container 37fc5aedb78760eba9bb267537e2fc7745ec0dfa3393512447de22453378e41b: Status 404 returned error can't find the container with id 37fc5aedb78760eba9bb267537e2fc7745ec0dfa3393512447de22453378e41b Nov 03 01:00:41.409719 ip-10-0-140-107 kubenswrapper[4939]: W1103 01:00:41.409692 4939 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfd866aeb_ea66_4982_b18d_cdc4a1f8a836.slice/crio-0d9793d67f18a5f9fed762a8bdc2bb5db9b9ebc16b7a907b4f684392db2f8bf6 WatchSource:0}: Error finding container 0d9793d67f18a5f9fed762a8bdc2bb5db9b9ebc16b7a907b4f684392db2f8bf6: Status 404 returned error can't find the container with id 0d9793d67f18a5f9fed762a8bdc2bb5db9b9ebc16b7a907b4f684392db2f8bf6 Nov 03 01:00:41.411157 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:00:41.411128 4939 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-02e4c0-7796479459-fjkx9"] Nov 03 01:00:41.676466 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:00:41.676369 4939 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-7406a9-openshift-default-74c9f8d955-rqrs4" Nov 03 01:00:41.676639 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:00:41.676573 4939 patch_prober.go:29] interesting pod/e2e-api-a-7406a9-openshift-default-74c9f8d955-rqrs4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.112:15021/healthz/ready\": dial tcp 10.132.0.112:15021: connect: connection refused" start-of-body= Nov 03 01:00:41.676639 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:00:41.676621 4939 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-7406a9-openshift-default-74c9f8d955-rqrs4" podUID="ab2a5b74-e0ad-4a13-9d46-7b78d5ef8aa7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.112:15021/healthz/ready\": dial tcp 10.132.0.112:15021: connect: connection refused" Nov 03 01:00:41.725358 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:00:41.725324 4939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-02e4c0-57ff55f5cf-72j67" event={"ID":"34792239-0f53-474d-adbf-fb1205780987","Type":"ContainerStarted","Data":"77080b26b5af20c8525019f42212cbfe6cc05b1b1b7eaf78ea7a1f9e16a52d95"} Nov 03 01:00:41.725553 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:00:41.725378 4939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-02e4c0-57ff55f5cf-72j67" event={"ID":"34792239-0f53-474d-adbf-fb1205780987","Type":"ContainerStarted","Data":"37fc5aedb78760eba9bb267537e2fc7745ec0dfa3393512447de22453378e41b"} Nov 03 01:00:41.725553 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:00:41.725404 4939 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-02e4c0-57ff55f5cf-72j67" Nov 03 01:00:41.727091 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:00:41.727064 4939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-02e4c0-7796479459-fjkx9" event={"ID":"fd866aeb-ea66-4982-b18d-cdc4a1f8a836","Type":"ContainerStarted","Data":"3f097a20ba5038ce2b70404d07d2ec82bb3b6540d2f02ad524360522382c0e13"} Nov 03 01:00:41.727204 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:00:41.727109 4939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-02e4c0-7796479459-fjkx9" event={"ID":"fd866aeb-ea66-4982-b18d-cdc4a1f8a836","Type":"ContainerStarted","Data":"0d9793d67f18a5f9fed762a8bdc2bb5db9b9ebc16b7a907b4f684392db2f8bf6"} Nov 03 01:00:41.727204 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:00:41.727157 4939 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-02e4c0-7796479459-fjkx9" Nov 03 01:00:41.773616 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:00:41.773563 4939 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-02e4c0-57ff55f5cf-72j67" podStartSLOduration=1.773548946 podStartE2EDuration="1.773548946s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 01:00:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 01:00:41.74467092 +0000 UTC m=+1190.009005868" watchObservedRunningTime="2026-11-03 01:00:41.773548946 +0000 UTC m=+1190.037883891" Nov 03 01:00:41.777608 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:00:41.777559 4939 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-b-02e4c0-7796479459-fjkx9" podStartSLOduration=1.777546319 podStartE2EDuration="1.777546319s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 01:00:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 01:00:41.775373894 +0000 UTC m=+1190.039708840" watchObservedRunningTime="2026-11-03 01:00:41.777546319 +0000 UTC m=+1190.041881265" Nov 03 01:00:42.221630 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:00:42.221599 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:00:42.676289 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:00:42.676258 4939 patch_prober.go:29] interesting pod/e2e-api-a-7406a9-openshift-default-74c9f8d955-rqrs4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.112:15021/healthz/ready\": dial tcp 10.132.0.112:15021: connect: connection refused" start-of-body= Nov 03 01:00:42.676899 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:00:42.676856 4939 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-7406a9-openshift-default-74c9f8d955-rqrs4" podUID="ab2a5b74-e0ad-4a13-9d46-7b78d5ef8aa7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.112:15021/healthz/ready\": dial tcp 10.132.0.112:15021: connect: connection refused" Nov 03 01:00:42.778472 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:00:42.778438 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:00:42.990800 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:00:42.990702 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:00:43.209775 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:00:43.209739 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:00:43.481930 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:00:43.479404 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:00:43.676414 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:00:43.676379 4939 patch_prober.go:29] interesting pod/e2e-api-a-7406a9-openshift-default-74c9f8d955-rqrs4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.112:15021/healthz/ready\": dial tcp 10.132.0.112:15021: connect: connection refused" start-of-body= Nov 03 01:00:43.677024 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:00:43.676988 4939 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-7406a9-openshift-default-74c9f8d955-rqrs4" podUID="ab2a5b74-e0ad-4a13-9d46-7b78d5ef8aa7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.112:15021/healthz/ready\": dial tcp 10.132.0.112:15021: connect: connection refused" Nov 03 01:00:43.710148 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:00:43.710123 4939 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-7406a9-openshift-default-74c9f8d955-rqrs4"] Nov 03 01:00:43.710524 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:00:43.710502 4939 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-api-a-7406a9-openshift-default-74c9f8d955-rqrs4" podUID="ab2a5b74-e0ad-4a13-9d46-7b78d5ef8aa7" containerName="istio-proxy" containerID="cri-o://653e46bdc0ec8b7a25e2ab7986ac2c3cac776da6eba42b8ef8633f9e12bef9bb" gracePeriod=30 Nov 03 01:00:43.790134 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:00:43.790033 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:00:44.414427 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:00:44.414396 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:00:44.560060 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:00:44.560026 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:00:45.324281 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:00:45.324235 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:00:46.128289 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:00:46.128240 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:00:46.498767 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:00:46.498688 4939 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-83755a-openshift-default-c86f9565c-rwxw4"] Nov 03 01:00:46.499206 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:00:46.498943 4939 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-83755a-openshift-default-c86f9565c-rwxw4" podUID="4adcd893-4e34-4860-bafb-a32ee8b956f0" containerName="istio-proxy" containerID="cri-o://d63e6ba311f7c32592f99e41dd118860a99b3a9e540667037881f767af6eec95" gracePeriod=30 Nov 03 01:00:46.502388 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:00:46.502359 4939 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-83755a-openshift-default-c86f9565c-nzmzn"] Nov 03 01:00:46.502595 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:00:46.502562 4939 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-83755a-openshift-default-c86f9565c-nzmzn" podUID="44fe729d-d7d1-41c2-99df-60e955909b8b" containerName="istio-proxy" containerID="cri-o://74d2ce58c7345c8b44395dda007f1db79217e50d329cd8e7791d0422eed2d738" gracePeriod=30 Nov 03 01:00:46.943644 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:00:46.943615 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:00:47.627413 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:00:47.627380 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:00:47.737115 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:00:47.737069 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:00:47.859104 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:00:47.859069 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:00:48.033043 ip-10-0-140-107 kubenswrapper[4939]: E1103 01:00:48.032949 4939 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-sthjh" containerName="extractor" Nov 03 01:00:48.158364 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:00:48.158330 4939 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-2f9a3a-openshift-default-7f6f74cf97-j8q97"] Nov 03 01:00:48.158592 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:00:48.158569 4939 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-2f9a3a-openshift-default-7f6f74cf97-j8q97" podUID="d96472d8-fbad-44c0-815d-bf4d20cce46a" containerName="istio-proxy" containerID="cri-o://1f2e9fa18ac7e6c6c904f4612ece64a27ae0a4c622fa0970018363af08e3bc92" gracePeriod=30 Nov 03 01:00:48.162062 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:00:48.162037 4939 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-2f9a3a-openshift-default-7f6f74cf97-ph469"] Nov 03 01:00:48.162265 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:00:48.162244 4939 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-2f9a3a-openshift-default-7f6f74cf97-ph469" podUID="5ca6f6a0-78ff-4b3f-ad1f-b94428b9178e" containerName="istio-proxy" containerID="cri-o://19473538a2c75199337c996b9d3339e90e813285dc8c6eeb33c78557b70e09fd" gracePeriod=30 Nov 03 01:00:48.228884 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:00:48.228847 4939 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-7f43f9-668f86957-vqsff"] Nov 03 01:00:48.229096 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:00:48.229071 4939 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-7f43f9-668f86957-vqsff" podUID="e2f30569-2ce1-469c-a8dc-87ac06c45698" containerName="maas-api" containerID="cri-o://12d0dc8d226d8b6cc9f19c1e2ef05e2721b39285a2986fa00aa2acf7ecabb7e8" gracePeriod=30 Nov 03 01:00:48.296010 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:00:48.295860 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:00:48.432841 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:00:48.432792 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:00:48.625937 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:00:48.614066 4939 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-1e352e-6bb6bb4d7f-hd98j" Nov 03 01:00:48.650527 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:00:48.650497 4939 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-praxis-a-1e352e-585bf55c69-ddbn9" Nov 03 01:00:48.774187 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:00:48.761378 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:00:48.891997 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:00:48.882453 4939 generic.go:356] "Generic (PLEG): container finished" podID="e2f30569-2ce1-469c-a8dc-87ac06c45698" containerID="12d0dc8d226d8b6cc9f19c1e2ef05e2721b39285a2986fa00aa2acf7ecabb7e8" exitCode=0 Nov 03 01:00:48.891997 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:00:48.882761 4939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-7f43f9-668f86957-vqsff" event={"ID":"e2f30569-2ce1-469c-a8dc-87ac06c45698","Type":"ContainerDied","Data":"12d0dc8d226d8b6cc9f19c1e2ef05e2721b39285a2986fa00aa2acf7ecabb7e8"} Nov 03 01:00:48.891997 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:00:48.882843 4939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-7f43f9-668f86957-vqsff" event={"ID":"e2f30569-2ce1-469c-a8dc-87ac06c45698","Type":"ContainerDied","Data":"d41413268dbe658f2f722c79bce6ec1c866c63dea861307fa2af65bfea54cd18"} Nov 03 01:00:48.891997 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:00:48.882869 4939 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d41413268dbe658f2f722c79bce6ec1c866c63dea861307fa2af65bfea54cd18" Nov 03 01:00:48.953109 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:00:48.953064 4939 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-7f43f9-668f86957-vqsff" Nov 03 01:00:49.144387 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:00:49.144300 4939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e2f30569-2ce1-469c-a8dc-87ac06c45698-metrics-certs\" (UniqueName: \"kubernetes.io/secret/e2f30569-2ce1-469c-a8dc-87ac06c45698-metrics-certs\") pod \"e2f30569-2ce1-469c-a8dc-87ac06c45698\" (UID: \"e2f30569-2ce1-469c-a8dc-87ac06c45698\") " Nov 03 01:00:49.144565 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:00:49.144433 4939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e2f30569-2ce1-469c-a8dc-87ac06c45698-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e2f30569-2ce1-469c-a8dc-87ac06c45698-openshift-service-ca\") pod \"e2f30569-2ce1-469c-a8dc-87ac06c45698\" (UID: \"e2f30569-2ce1-469c-a8dc-87ac06c45698\") " Nov 03 01:00:49.144627 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:00:49.144589 4939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e2f30569-2ce1-469c-a8dc-87ac06c45698-kube-api-access-667sm\" (UniqueName: \"kubernetes.io/projected/e2f30569-2ce1-469c-a8dc-87ac06c45698-kube-api-access-667sm\") pod \"e2f30569-2ce1-469c-a8dc-87ac06c45698\" (UID: \"e2f30569-2ce1-469c-a8dc-87ac06c45698\") " Nov 03 01:00:49.144708 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:00:49.144689 4939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e2f30569-2ce1-469c-a8dc-87ac06c45698-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e2f30569-2ce1-469c-a8dc-87ac06c45698-maas-api-tls\") pod \"e2f30569-2ce1-469c-a8dc-87ac06c45698\" (UID: \"e2f30569-2ce1-469c-a8dc-87ac06c45698\") " Nov 03 01:00:49.144840 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:00:49.144790 4939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e2f30569-2ce1-469c-a8dc-87ac06c45698-openshift-service-ca" pod "e2f30569-2ce1-469c-a8dc-87ac06c45698" (UID: "e2f30569-2ce1-469c-a8dc-87ac06c45698"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 01:00:49.145471 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:00:49.145441 4939 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e2f30569-2ce1-469c-a8dc-87ac06c45698-openshift-service-ca\") on node \"ip-10-0-140-107.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:00:49.146774 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:00:49.146749 4939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e2f30569-2ce1-469c-a8dc-87ac06c45698-kube-api-access-667sm" pod "e2f30569-2ce1-469c-a8dc-87ac06c45698" (UID: "e2f30569-2ce1-469c-a8dc-87ac06c45698"). InnerVolumeSpecName "kube-api-access-667sm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 01:00:49.146774 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:00:49.146765 4939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e2f30569-2ce1-469c-a8dc-87ac06c45698-maas-api-tls" pod "e2f30569-2ce1-469c-a8dc-87ac06c45698" (UID: "e2f30569-2ce1-469c-a8dc-87ac06c45698"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 01:00:49.147057 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:00:49.146819 4939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e2f30569-2ce1-469c-a8dc-87ac06c45698-metrics-certs" pod "e2f30569-2ce1-469c-a8dc-87ac06c45698" (UID: "e2f30569-2ce1-469c-a8dc-87ac06c45698"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 01:00:49.247270 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:00:49.247240 4939 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e2f30569-2ce1-469c-a8dc-87ac06c45698-maas-api-tls\") on node \"ip-10-0-140-107.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:00:49.247270 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:00:49.247265 4939 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e2f30569-2ce1-469c-a8dc-87ac06c45698-metrics-certs\") on node \"ip-10-0-140-107.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:00:49.247270 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:00:49.247276 4939 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-667sm\" (UniqueName: \"kubernetes.io/projected/e2f30569-2ce1-469c-a8dc-87ac06c45698-kube-api-access-667sm\") on node \"ip-10-0-140-107.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:00:49.694402 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:00:49.694365 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:00:49.894445 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:00:49.894411 4939 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-7f43f9-668f86957-vqsff" Nov 03 01:00:49.931999 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:00:49.931970 4939 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-7f43f9-668f86957-vqsff"] Nov 03 01:00:49.935839 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:00:49.935813 4939 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-7f43f9-668f86957-vqsff"] Nov 03 01:00:50.436971 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:00:50.436929 4939 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-02e4c0-575bdc8b8c-ss7s5"] Nov 03 01:00:50.438166 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:00:50.438138 4939 cpu_manager.go:414] "containerMap: removing container" podUID="e2f30569-2ce1-469c-a8dc-87ac06c45698" containerName="maas-api" Nov 03 01:00:50.438355 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:00:50.438328 4939 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e2f30569-2ce1-469c-a8dc-87ac06c45698" containerName="maas-api" Nov 03 01:00:50.438565 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:00:50.438549 4939 memory_manager.go:365] "RemoveStaleState removing state" podUID="e2f30569-2ce1-469c-a8dc-87ac06c45698" containerName="maas-api" Nov 03 01:00:50.445011 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:00:50.444991 4939 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-02e4c0-575bdc8b8c-ss7s5" Nov 03 01:00:50.452783 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:00:50.452762 4939 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ipp-b-02e4c0-serving-cert\"" Nov 03 01:00:50.455596 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:00:50.455574 4939 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-02e4c0-575bdc8b8c-ss7s5"] Nov 03 01:00:50.513876 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:00:50.513834 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:00:50.578367 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:00:50.578336 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/67809b22-2bbf-452b-84fc-b7711dd222d0-maas-api-tls\") pod \"maas-api-e2e-ipp-b-02e4c0-575bdc8b8c-ss7s5\" (UID: \"67809b22-2bbf-452b-84fc-b7711dd222d0\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-02e4c0-575bdc8b8c-ss7s5" Nov 03 01:00:50.578508 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:00:50.578374 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/67809b22-2bbf-452b-84fc-b7711dd222d0-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-02e4c0-575bdc8b8c-ss7s5\" (UID: \"67809b22-2bbf-452b-84fc-b7711dd222d0\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-02e4c0-575bdc8b8c-ss7s5" Nov 03 01:00:50.578563 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:00:50.578514 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/67809b22-2bbf-452b-84fc-b7711dd222d0-metrics-certs\") pod \"maas-api-e2e-ipp-b-02e4c0-575bdc8b8c-ss7s5\" (UID: \"67809b22-2bbf-452b-84fc-b7711dd222d0\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-02e4c0-575bdc8b8c-ss7s5" Nov 03 01:00:50.578637 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:00:50.578623 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rzf6l\" (UniqueName: \"kubernetes.io/projected/67809b22-2bbf-452b-84fc-b7711dd222d0-kube-api-access-rzf6l\") pod \"maas-api-e2e-ipp-b-02e4c0-575bdc8b8c-ss7s5\" (UID: \"67809b22-2bbf-452b-84fc-b7711dd222d0\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-02e4c0-575bdc8b8c-ss7s5" Nov 03 01:00:50.650931 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:00:50.650435 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:00:50.687035 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:00:50.681374 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/67809b22-2bbf-452b-84fc-b7711dd222d0-maas-api-tls\") pod \"maas-api-e2e-ipp-b-02e4c0-575bdc8b8c-ss7s5\" (UID: \"67809b22-2bbf-452b-84fc-b7711dd222d0\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-02e4c0-575bdc8b8c-ss7s5" Nov 03 01:00:50.687035 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:00:50.681431 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/67809b22-2bbf-452b-84fc-b7711dd222d0-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-02e4c0-575bdc8b8c-ss7s5\" (UID: \"67809b22-2bbf-452b-84fc-b7711dd222d0\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-02e4c0-575bdc8b8c-ss7s5" Nov 03 01:00:50.687035 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:00:50.681523 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/67809b22-2bbf-452b-84fc-b7711dd222d0-metrics-certs\") pod \"maas-api-e2e-ipp-b-02e4c0-575bdc8b8c-ss7s5\" (UID: \"67809b22-2bbf-452b-84fc-b7711dd222d0\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-02e4c0-575bdc8b8c-ss7s5" Nov 03 01:00:50.687035 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:00:50.681615 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rzf6l\" (UniqueName: \"kubernetes.io/projected/67809b22-2bbf-452b-84fc-b7711dd222d0-kube-api-access-rzf6l\") pod \"maas-api-e2e-ipp-b-02e4c0-575bdc8b8c-ss7s5\" (UID: \"67809b22-2bbf-452b-84fc-b7711dd222d0\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-02e4c0-575bdc8b8c-ss7s5" Nov 03 01:00:50.687035 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:00:50.682849 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/67809b22-2bbf-452b-84fc-b7711dd222d0-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-02e4c0-575bdc8b8c-ss7s5\" (UID: \"67809b22-2bbf-452b-84fc-b7711dd222d0\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-02e4c0-575bdc8b8c-ss7s5" Nov 03 01:00:50.691656 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:00:50.691622 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/67809b22-2bbf-452b-84fc-b7711dd222d0-maas-api-tls\") pod \"maas-api-e2e-ipp-b-02e4c0-575bdc8b8c-ss7s5\" (UID: \"67809b22-2bbf-452b-84fc-b7711dd222d0\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-02e4c0-575bdc8b8c-ss7s5" Nov 03 01:00:50.702962 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:00:50.692249 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rzf6l\" (UniqueName: \"kubernetes.io/projected/67809b22-2bbf-452b-84fc-b7711dd222d0-kube-api-access-rzf6l\") pod \"maas-api-e2e-ipp-b-02e4c0-575bdc8b8c-ss7s5\" (UID: \"67809b22-2bbf-452b-84fc-b7711dd222d0\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-02e4c0-575bdc8b8c-ss7s5" Nov 03 01:00:50.704126 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:00:50.704091 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/67809b22-2bbf-452b-84fc-b7711dd222d0-metrics-certs\") pod \"maas-api-e2e-ipp-b-02e4c0-575bdc8b8c-ss7s5\" (UID: \"67809b22-2bbf-452b-84fc-b7711dd222d0\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-02e4c0-575bdc8b8c-ss7s5" Nov 03 01:00:50.763500 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:00:50.763467 4939 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-02e4c0-575bdc8b8c-ss7s5" Nov 03 01:00:51.048105 ip-10-0-140-107 kubenswrapper[4939]: W1103 01:00:51.048072 4939 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod67809b22_2bbf_452b_84fc_b7711dd222d0.slice/crio-cd9b365513a95b12ad97690a23fa36779b961cd7249d1915f1f681ef1b199f93 WatchSource:0}: Error finding container cd9b365513a95b12ad97690a23fa36779b961cd7249d1915f1f681ef1b199f93: Status 404 returned error can't find the container with id cd9b365513a95b12ad97690a23fa36779b961cd7249d1915f1f681ef1b199f93 Nov 03 01:00:51.048997 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:00:51.048879 4939 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-02e4c0-575bdc8b8c-ss7s5"] Nov 03 01:00:51.139032 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:00:51.138996 4939 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-ffdea6-6975889bf-cb625"] Nov 03 01:00:51.384933 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:00:51.382543 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:00:51.533954 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:00:51.526451 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:00:51.872128 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:00:51.866556 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:00:52.176939 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:00:52.159277 4939 generic.go:356] "Generic (PLEG): container finished" podID="ab2a5b74-e0ad-4a13-9d46-7b78d5ef8aa7" containerID="653e46bdc0ec8b7a25e2ab7986ac2c3cac776da6eba42b8ef8633f9e12bef9bb" exitCode=0 Nov 03 01:00:52.176939 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:00:52.168329 4939 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e2f30569-2ce1-469c-a8dc-87ac06c45698" path="/var/lib/kubelet/pods/e2f30569-2ce1-469c-a8dc-87ac06c45698/volumes" Nov 03 01:00:52.176939 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:00:52.168759 4939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-7406a9-openshift-default-74c9f8d955-rqrs4" event={"ID":"ab2a5b74-e0ad-4a13-9d46-7b78d5ef8aa7","Type":"ContainerDied","Data":"653e46bdc0ec8b7a25e2ab7986ac2c3cac776da6eba42b8ef8633f9e12bef9bb"} Nov 03 01:00:52.176939 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:00:52.168822 4939 scope.go:122] "RemoveContainer" containerID="a6df97fdc2b980db01fd7357a1e20e13cbe9006cb42e82f275f50fe8a992651a" Nov 03 01:00:52.186302 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:00:52.186245 4939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-02e4c0-575bdc8b8c-ss7s5" event={"ID":"67809b22-2bbf-452b-84fc-b7711dd222d0","Type":"ContainerStarted","Data":"cd9b365513a95b12ad97690a23fa36779b961cd7249d1915f1f681ef1b199f93"} Nov 03 01:00:52.238948 ip-10-0-140-107 kubenswrapper[4939]: E1103 01:00:52.238738 4939 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a6df97fdc2b980db01fd7357a1e20e13cbe9006cb42e82f275f50fe8a992651a\": container with ID starting with a6df97fdc2b980db01fd7357a1e20e13cbe9006cb42e82f275f50fe8a992651a not found: ID does not exist" containerID="a6df97fdc2b980db01fd7357a1e20e13cbe9006cb42e82f275f50fe8a992651a" Nov 03 01:00:52.238948 ip-10-0-140-107 kubenswrapper[4939]: E1103 01:00:52.238793 4939 kubelet.go:1679] "Image garbage collection failed once. Stats initialization may not have completed yet" err="get container status: runtime container status: rpc error: code = NotFound desc = could not find container \"a6df97fdc2b980db01fd7357a1e20e13cbe9006cb42e82f275f50fe8a992651a\": container with ID starting with a6df97fdc2b980db01fd7357a1e20e13cbe9006cb42e82f275f50fe8a992651a not found: ID does not exist" Nov 03 01:00:52.240997 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:00:52.240961 4939 image_gc_manager.go:223] "Failed to monitor images" err="get container status: runtime container status: rpc error: code = NotFound desc = could not find container \"a6df97fdc2b980db01fd7357a1e20e13cbe9006cb42e82f275f50fe8a992651a\": container with ID starting with a6df97fdc2b980db01fd7357a1e20e13cbe9006cb42e82f275f50fe8a992651a not found: ID does not exist" Nov 03 01:00:52.346416 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:00:52.346380 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:00:52.354803 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:00:52.354777 4939 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-7406a9-openshift-default-74c9f8d955-rqrs4" Nov 03 01:00:52.424044 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:00:52.424012 4939 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-83755a-openshift-default-c86f9565c-rwxw4" Nov 03 01:00:52.466134 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:00:52.462865 4939 scope.go:122] "RemoveContainer" containerID="172c051c8692aa520fdb4336cf232a2c6826a830ed044c0a56c765fc2ee2fa1c" Nov 03 01:00:52.466134 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:00:52.465446 4939 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-83755a-openshift-default-c86f9565c-nzmzn" Nov 03 01:00:52.486496 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:00:52.476764 4939 scope.go:122] "RemoveContainer" containerID="74d2ce58c7345c8b44395dda007f1db79217e50d329cd8e7791d0422eed2d738" Nov 03 01:00:52.489358 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:00:52.489334 4939 scope.go:122] "RemoveContainer" containerID="d63e6ba311f7c32592f99e41dd118860a99b3a9e540667037881f767af6eec95" Nov 03 01:00:52.510488 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:00:52.510451 4939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ab2a5b74-e0ad-4a13-9d46-7b78d5ef8aa7-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ab2a5b74-e0ad-4a13-9d46-7b78d5ef8aa7-credential-socket\") pod \"ab2a5b74-e0ad-4a13-9d46-7b78d5ef8aa7\" (UID: \"ab2a5b74-e0ad-4a13-9d46-7b78d5ef8aa7\") " Nov 03 01:00:52.510654 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:00:52.510564 4939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/ab2a5b74-e0ad-4a13-9d46-7b78d5ef8aa7-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ab2a5b74-e0ad-4a13-9d46-7b78d5ef8aa7-istio-podinfo\") pod \"ab2a5b74-e0ad-4a13-9d46-7b78d5ef8aa7\" (UID: \"ab2a5b74-e0ad-4a13-9d46-7b78d5ef8aa7\") " Nov 03 01:00:52.510726 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:00:52.510652 4939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ab2a5b74-e0ad-4a13-9d46-7b78d5ef8aa7-istio-token\" (UniqueName: \"kubernetes.io/projected/ab2a5b74-e0ad-4a13-9d46-7b78d5ef8aa7-istio-token\") pod \"ab2a5b74-e0ad-4a13-9d46-7b78d5ef8aa7\" (UID: \"ab2a5b74-e0ad-4a13-9d46-7b78d5ef8aa7\") " Nov 03 01:00:52.510821 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:00:52.510803 4939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ab2a5b74-e0ad-4a13-9d46-7b78d5ef8aa7-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ab2a5b74-e0ad-4a13-9d46-7b78d5ef8aa7-workload-socket\") pod \"ab2a5b74-e0ad-4a13-9d46-7b78d5ef8aa7\" (UID: \"ab2a5b74-e0ad-4a13-9d46-7b78d5ef8aa7\") " Nov 03 01:00:52.511037 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:00:52.510888 4939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ab2a5b74-e0ad-4a13-9d46-7b78d5ef8aa7-kube-api-access-6vd5r\" (UniqueName: \"kubernetes.io/projected/ab2a5b74-e0ad-4a13-9d46-7b78d5ef8aa7-kube-api-access-6vd5r\") pod \"ab2a5b74-e0ad-4a13-9d46-7b78d5ef8aa7\" (UID: \"ab2a5b74-e0ad-4a13-9d46-7b78d5ef8aa7\") " Nov 03 01:00:52.511261 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:00:52.511209 4939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ab2a5b74-e0ad-4a13-9d46-7b78d5ef8aa7-credential-socket" pod "ab2a5b74-e0ad-4a13-9d46-7b78d5ef8aa7" (UID: "ab2a5b74-e0ad-4a13-9d46-7b78d5ef8aa7"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 01:00:52.511371 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:00:52.511319 4939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ab2a5b74-e0ad-4a13-9d46-7b78d5ef8aa7-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ab2a5b74-e0ad-4a13-9d46-7b78d5ef8aa7-istio-data\") pod \"ab2a5b74-e0ad-4a13-9d46-7b78d5ef8aa7\" (UID: \"ab2a5b74-e0ad-4a13-9d46-7b78d5ef8aa7\") " Nov 03 01:00:52.511435 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:00:52.511369 4939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ab2a5b74-e0ad-4a13-9d46-7b78d5ef8aa7-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ab2a5b74-e0ad-4a13-9d46-7b78d5ef8aa7-workload-certs\") pod \"ab2a5b74-e0ad-4a13-9d46-7b78d5ef8aa7\" (UID: \"ab2a5b74-e0ad-4a13-9d46-7b78d5ef8aa7\") " Nov 03 01:00:52.511737 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:00:52.511472 4939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ab2a5b74-e0ad-4a13-9d46-7b78d5ef8aa7-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ab2a5b74-e0ad-4a13-9d46-7b78d5ef8aa7-istio-envoy\") pod \"ab2a5b74-e0ad-4a13-9d46-7b78d5ef8aa7\" (UID: \"ab2a5b74-e0ad-4a13-9d46-7b78d5ef8aa7\") " Nov 03 01:00:52.511831 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:00:52.511808 4939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ab2a5b74-e0ad-4a13-9d46-7b78d5ef8aa7-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ab2a5b74-e0ad-4a13-9d46-7b78d5ef8aa7-istiod-ca-cert\") pod \"ab2a5b74-e0ad-4a13-9d46-7b78d5ef8aa7\" (UID: \"ab2a5b74-e0ad-4a13-9d46-7b78d5ef8aa7\") " Nov 03 01:00:52.512722 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:00:52.512690 4939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ab2a5b74-e0ad-4a13-9d46-7b78d5ef8aa7-istio-data" pod "ab2a5b74-e0ad-4a13-9d46-7b78d5ef8aa7" (UID: "ab2a5b74-e0ad-4a13-9d46-7b78d5ef8aa7"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 01:00:52.512813 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:00:52.512770 4939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ab2a5b74-e0ad-4a13-9d46-7b78d5ef8aa7-workload-certs" pod "ab2a5b74-e0ad-4a13-9d46-7b78d5ef8aa7" (UID: "ab2a5b74-e0ad-4a13-9d46-7b78d5ef8aa7"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 01:00:52.513722 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:00:52.513693 4939 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ab2a5b74-e0ad-4a13-9d46-7b78d5ef8aa7-credential-socket\") on node \"ip-10-0-140-107.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:00:52.513809 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:00:52.513727 4939 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ab2a5b74-e0ad-4a13-9d46-7b78d5ef8aa7-istio-data\") on node \"ip-10-0-140-107.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:00:52.513809 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:00:52.513725 4939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ab2a5b74-e0ad-4a13-9d46-7b78d5ef8aa7-istiod-ca-cert" pod "ab2a5b74-e0ad-4a13-9d46-7b78d5ef8aa7" (UID: "ab2a5b74-e0ad-4a13-9d46-7b78d5ef8aa7"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 01:00:52.514319 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:00:52.513743 4939 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ab2a5b74-e0ad-4a13-9d46-7b78d5ef8aa7-workload-certs\") on node \"ip-10-0-140-107.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:00:52.514411 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:00:52.513829 4939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ab2a5b74-e0ad-4a13-9d46-7b78d5ef8aa7-workload-socket" pod "ab2a5b74-e0ad-4a13-9d46-7b78d5ef8aa7" (UID: "ab2a5b74-e0ad-4a13-9d46-7b78d5ef8aa7"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 01:00:52.515260 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:00:52.515220 4939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/ab2a5b74-e0ad-4a13-9d46-7b78d5ef8aa7-istio-podinfo" pod "ab2a5b74-e0ad-4a13-9d46-7b78d5ef8aa7" (UID: "ab2a5b74-e0ad-4a13-9d46-7b78d5ef8aa7"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 01:00:52.522340 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:00:52.522314 4939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ab2a5b74-e0ad-4a13-9d46-7b78d5ef8aa7-istio-token" pod "ab2a5b74-e0ad-4a13-9d46-7b78d5ef8aa7" (UID: "ab2a5b74-e0ad-4a13-9d46-7b78d5ef8aa7"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 01:00:52.522463 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:00:52.522355 4939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ab2a5b74-e0ad-4a13-9d46-7b78d5ef8aa7-kube-api-access-6vd5r" pod "ab2a5b74-e0ad-4a13-9d46-7b78d5ef8aa7" (UID: "ab2a5b74-e0ad-4a13-9d46-7b78d5ef8aa7"). InnerVolumeSpecName "kube-api-access-6vd5r". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 01:00:52.526057 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:00:52.526028 4939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ab2a5b74-e0ad-4a13-9d46-7b78d5ef8aa7-istio-envoy" pod "ab2a5b74-e0ad-4a13-9d46-7b78d5ef8aa7" (UID: "ab2a5b74-e0ad-4a13-9d46-7b78d5ef8aa7"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 01:00:52.620192 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:00:52.620153 4939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4adcd893-4e34-4860-bafb-a32ee8b956f0-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4adcd893-4e34-4860-bafb-a32ee8b956f0-credential-socket\") pod \"4adcd893-4e34-4860-bafb-a32ee8b956f0\" (UID: \"4adcd893-4e34-4860-bafb-a32ee8b956f0\") " Nov 03 01:00:52.620425 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:00:52.620214 4939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/44fe729d-d7d1-41c2-99df-60e955909b8b-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/44fe729d-d7d1-41c2-99df-60e955909b8b-istio-envoy\") pod \"44fe729d-d7d1-41c2-99df-60e955909b8b\" (UID: \"44fe729d-d7d1-41c2-99df-60e955909b8b\") " Nov 03 01:00:52.620425 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:00:52.620254 4939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4adcd893-4e34-4860-bafb-a32ee8b956f0-kube-api-access-62w7v\" (UniqueName: \"kubernetes.io/projected/4adcd893-4e34-4860-bafb-a32ee8b956f0-kube-api-access-62w7v\") pod \"4adcd893-4e34-4860-bafb-a32ee8b956f0\" (UID: \"4adcd893-4e34-4860-bafb-a32ee8b956f0\") " Nov 03 01:00:52.620425 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:00:52.620308 4939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/44fe729d-d7d1-41c2-99df-60e955909b8b-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/44fe729d-d7d1-41c2-99df-60e955909b8b-istiod-ca-cert\") pod \"44fe729d-d7d1-41c2-99df-60e955909b8b\" (UID: \"44fe729d-d7d1-41c2-99df-60e955909b8b\") " Nov 03 01:00:52.620425 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:00:52.620353 4939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4adcd893-4e34-4860-bafb-a32ee8b956f0-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4adcd893-4e34-4860-bafb-a32ee8b956f0-workload-socket\") pod \"4adcd893-4e34-4860-bafb-a32ee8b956f0\" (UID: \"4adcd893-4e34-4860-bafb-a32ee8b956f0\") " Nov 03 01:00:52.620425 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:00:52.620421 4939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4adcd893-4e34-4860-bafb-a32ee8b956f0-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4adcd893-4e34-4860-bafb-a32ee8b956f0-istio-data\") pod \"4adcd893-4e34-4860-bafb-a32ee8b956f0\" (UID: \"4adcd893-4e34-4860-bafb-a32ee8b956f0\") " Nov 03 01:00:52.620718 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:00:52.620463 4939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/44fe729d-d7d1-41c2-99df-60e955909b8b-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/44fe729d-d7d1-41c2-99df-60e955909b8b-istio-podinfo\") pod \"44fe729d-d7d1-41c2-99df-60e955909b8b\" (UID: \"44fe729d-d7d1-41c2-99df-60e955909b8b\") " Nov 03 01:00:52.620718 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:00:52.620469 4939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4adcd893-4e34-4860-bafb-a32ee8b956f0-credential-socket" pod "4adcd893-4e34-4860-bafb-a32ee8b956f0" (UID: "4adcd893-4e34-4860-bafb-a32ee8b956f0"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 01:00:52.620718 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:00:52.620506 4939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/44fe729d-d7d1-41c2-99df-60e955909b8b-istio-token\" (UniqueName: \"kubernetes.io/projected/44fe729d-d7d1-41c2-99df-60e955909b8b-istio-token\") pod \"44fe729d-d7d1-41c2-99df-60e955909b8b\" (UID: \"44fe729d-d7d1-41c2-99df-60e955909b8b\") " Nov 03 01:00:52.620718 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:00:52.620550 4939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4adcd893-4e34-4860-bafb-a32ee8b956f0-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4adcd893-4e34-4860-bafb-a32ee8b956f0-istiod-ca-cert\") pod \"4adcd893-4e34-4860-bafb-a32ee8b956f0\" (UID: \"4adcd893-4e34-4860-bafb-a32ee8b956f0\") " Nov 03 01:00:52.620718 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:00:52.620577 4939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/4adcd893-4e34-4860-bafb-a32ee8b956f0-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4adcd893-4e34-4860-bafb-a32ee8b956f0-istio-podinfo\") pod \"4adcd893-4e34-4860-bafb-a32ee8b956f0\" (UID: \"4adcd893-4e34-4860-bafb-a32ee8b956f0\") " Nov 03 01:00:52.620718 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:00:52.620610 4939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/44fe729d-d7d1-41c2-99df-60e955909b8b-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/44fe729d-d7d1-41c2-99df-60e955909b8b-workload-socket\") pod \"44fe729d-d7d1-41c2-99df-60e955909b8b\" (UID: \"44fe729d-d7d1-41c2-99df-60e955909b8b\") " Nov 03 01:00:52.620718 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:00:52.620637 4939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/44fe729d-d7d1-41c2-99df-60e955909b8b-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/44fe729d-d7d1-41c2-99df-60e955909b8b-credential-socket\") pod \"44fe729d-d7d1-41c2-99df-60e955909b8b\" (UID: \"44fe729d-d7d1-41c2-99df-60e955909b8b\") " Nov 03 01:00:52.620718 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:00:52.620703 4939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/44fe729d-d7d1-41c2-99df-60e955909b8b-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/44fe729d-d7d1-41c2-99df-60e955909b8b-istio-data\") pod \"44fe729d-d7d1-41c2-99df-60e955909b8b\" (UID: \"44fe729d-d7d1-41c2-99df-60e955909b8b\") " Nov 03 01:00:52.621461 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:00:52.620745 4939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4adcd893-4e34-4860-bafb-a32ee8b956f0-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4adcd893-4e34-4860-bafb-a32ee8b956f0-workload-certs\") pod \"4adcd893-4e34-4860-bafb-a32ee8b956f0\" (UID: \"4adcd893-4e34-4860-bafb-a32ee8b956f0\") " Nov 03 01:00:52.621461 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:00:52.620804 4939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4adcd893-4e34-4860-bafb-a32ee8b956f0-istio-token\" (UniqueName: \"kubernetes.io/projected/4adcd893-4e34-4860-bafb-a32ee8b956f0-istio-token\") pod \"4adcd893-4e34-4860-bafb-a32ee8b956f0\" (UID: \"4adcd893-4e34-4860-bafb-a32ee8b956f0\") " Nov 03 01:00:52.621461 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:00:52.620846 4939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/44fe729d-d7d1-41c2-99df-60e955909b8b-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/44fe729d-d7d1-41c2-99df-60e955909b8b-workload-certs\") pod \"44fe729d-d7d1-41c2-99df-60e955909b8b\" (UID: \"44fe729d-d7d1-41c2-99df-60e955909b8b\") " Nov 03 01:00:52.621461 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:00:52.620888 4939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/44fe729d-d7d1-41c2-99df-60e955909b8b-kube-api-access-5f6cd\" (UniqueName: \"kubernetes.io/projected/44fe729d-d7d1-41c2-99df-60e955909b8b-kube-api-access-5f6cd\") pod \"44fe729d-d7d1-41c2-99df-60e955909b8b\" (UID: \"44fe729d-d7d1-41c2-99df-60e955909b8b\") " Nov 03 01:00:52.621461 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:00:52.620956 4939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/44fe729d-d7d1-41c2-99df-60e955909b8b-istiod-ca-cert" pod "44fe729d-d7d1-41c2-99df-60e955909b8b" (UID: "44fe729d-d7d1-41c2-99df-60e955909b8b"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 01:00:52.621461 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:00:52.621065 4939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4adcd893-4e34-4860-bafb-a32ee8b956f0-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4adcd893-4e34-4860-bafb-a32ee8b956f0-istio-envoy\") pod \"4adcd893-4e34-4860-bafb-a32ee8b956f0\" (UID: \"4adcd893-4e34-4860-bafb-a32ee8b956f0\") " Nov 03 01:00:52.624398 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:00:52.622709 4939 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ab2a5b74-e0ad-4a13-9d46-7b78d5ef8aa7-workload-socket\") on node \"ip-10-0-140-107.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:00:52.624398 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:00:52.622736 4939 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4adcd893-4e34-4860-bafb-a32ee8b956f0-credential-socket\") on node \"ip-10-0-140-107.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:00:52.624398 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:00:52.622753 4939 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6vd5r\" (UniqueName: \"kubernetes.io/projected/ab2a5b74-e0ad-4a13-9d46-7b78d5ef8aa7-kube-api-access-6vd5r\") on node \"ip-10-0-140-107.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:00:52.624398 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:00:52.622769 4939 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/44fe729d-d7d1-41c2-99df-60e955909b8b-istiod-ca-cert\") on node \"ip-10-0-140-107.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:00:52.624398 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:00:52.622784 4939 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ab2a5b74-e0ad-4a13-9d46-7b78d5ef8aa7-istio-envoy\") on node \"ip-10-0-140-107.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:00:52.624398 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:00:52.622799 4939 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ab2a5b74-e0ad-4a13-9d46-7b78d5ef8aa7-istiod-ca-cert\") on node \"ip-10-0-140-107.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:00:52.624398 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:00:52.623288 4939 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ab2a5b74-e0ad-4a13-9d46-7b78d5ef8aa7-istio-podinfo\") on node \"ip-10-0-140-107.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:00:52.624398 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:00:52.623305 4939 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ab2a5b74-e0ad-4a13-9d46-7b78d5ef8aa7-istio-token\") on node \"ip-10-0-140-107.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:00:52.624398 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:00:52.623727 4939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4adcd893-4e34-4860-bafb-a32ee8b956f0-workload-socket" pod "4adcd893-4e34-4860-bafb-a32ee8b956f0" (UID: "4adcd893-4e34-4860-bafb-a32ee8b956f0"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 01:00:52.624398 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:00:52.623813 4939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4adcd893-4e34-4860-bafb-a32ee8b956f0-workload-certs" pod "4adcd893-4e34-4860-bafb-a32ee8b956f0" (UID: "4adcd893-4e34-4860-bafb-a32ee8b956f0"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 01:00:52.624398 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:00:52.623828 4939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4adcd893-4e34-4860-bafb-a32ee8b956f0-istio-data" pod "4adcd893-4e34-4860-bafb-a32ee8b956f0" (UID: "4adcd893-4e34-4860-bafb-a32ee8b956f0"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 01:00:52.624398 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:00:52.623903 4939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/44fe729d-d7d1-41c2-99df-60e955909b8b-istio-envoy" pod "44fe729d-d7d1-41c2-99df-60e955909b8b" (UID: "44fe729d-d7d1-41c2-99df-60e955909b8b"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 01:00:52.624398 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:00:52.624068 4939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/44fe729d-d7d1-41c2-99df-60e955909b8b-istio-data" pod "44fe729d-d7d1-41c2-99df-60e955909b8b" (UID: "44fe729d-d7d1-41c2-99df-60e955909b8b"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 01:00:52.624398 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:00:52.624085 4939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/44fe729d-d7d1-41c2-99df-60e955909b8b-workload-certs" pod "44fe729d-d7d1-41c2-99df-60e955909b8b" (UID: "44fe729d-d7d1-41c2-99df-60e955909b8b"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 01:00:52.624398 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:00:52.624210 4939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4adcd893-4e34-4860-bafb-a32ee8b956f0-kube-api-access-62w7v" pod "4adcd893-4e34-4860-bafb-a32ee8b956f0" (UID: "4adcd893-4e34-4860-bafb-a32ee8b956f0"). InnerVolumeSpecName "kube-api-access-62w7v". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 01:00:52.624398 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:00:52.624296 4939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/44fe729d-d7d1-41c2-99df-60e955909b8b-workload-socket" pod "44fe729d-d7d1-41c2-99df-60e955909b8b" (UID: "44fe729d-d7d1-41c2-99df-60e955909b8b"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 01:00:52.624398 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:00:52.624306 4939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/4adcd893-4e34-4860-bafb-a32ee8b956f0-istio-podinfo" pod "4adcd893-4e34-4860-bafb-a32ee8b956f0" (UID: "4adcd893-4e34-4860-bafb-a32ee8b956f0"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 01:00:52.624398 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:00:52.624377 4939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4adcd893-4e34-4860-bafb-a32ee8b956f0-istiod-ca-cert" pod "4adcd893-4e34-4860-bafb-a32ee8b956f0" (UID: "4adcd893-4e34-4860-bafb-a32ee8b956f0"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 01:00:52.625675 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:00:52.625648 4939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/44fe729d-d7d1-41c2-99df-60e955909b8b-istio-token" pod "44fe729d-d7d1-41c2-99df-60e955909b8b" (UID: "44fe729d-d7d1-41c2-99df-60e955909b8b"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 01:00:52.626140 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:00:52.626104 4939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/44fe729d-d7d1-41c2-99df-60e955909b8b-credential-socket" pod "44fe729d-d7d1-41c2-99df-60e955909b8b" (UID: "44fe729d-d7d1-41c2-99df-60e955909b8b"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 01:00:52.626480 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:00:52.626449 4939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/44fe729d-d7d1-41c2-99df-60e955909b8b-kube-api-access-5f6cd" pod "44fe729d-d7d1-41c2-99df-60e955909b8b" (UID: "44fe729d-d7d1-41c2-99df-60e955909b8b"). InnerVolumeSpecName "kube-api-access-5f6cd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 01:00:52.626764 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:00:52.626723 4939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4adcd893-4e34-4860-bafb-a32ee8b956f0-istio-envoy" pod "4adcd893-4e34-4860-bafb-a32ee8b956f0" (UID: "4adcd893-4e34-4860-bafb-a32ee8b956f0"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 01:00:52.626848 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:00:52.626757 4939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4adcd893-4e34-4860-bafb-a32ee8b956f0-istio-token" pod "4adcd893-4e34-4860-bafb-a32ee8b956f0" (UID: "4adcd893-4e34-4860-bafb-a32ee8b956f0"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 01:00:52.627844 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:00:52.627818 4939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/44fe729d-d7d1-41c2-99df-60e955909b8b-istio-podinfo" pod "44fe729d-d7d1-41c2-99df-60e955909b8b" (UID: "44fe729d-d7d1-41c2-99df-60e955909b8b"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 01:00:52.725021 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:00:52.724986 4939 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/44fe729d-d7d1-41c2-99df-60e955909b8b-istio-data\") on node \"ip-10-0-140-107.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:00:52.725021 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:00:52.725014 4939 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4adcd893-4e34-4860-bafb-a32ee8b956f0-workload-certs\") on node \"ip-10-0-140-107.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:00:52.725021 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:00:52.725024 4939 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4adcd893-4e34-4860-bafb-a32ee8b956f0-istio-token\") on node \"ip-10-0-140-107.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:00:52.725238 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:00:52.725037 4939 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/44fe729d-d7d1-41c2-99df-60e955909b8b-workload-certs\") on node \"ip-10-0-140-107.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:00:52.725238 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:00:52.725047 4939 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5f6cd\" (UniqueName: \"kubernetes.io/projected/44fe729d-d7d1-41c2-99df-60e955909b8b-kube-api-access-5f6cd\") on node \"ip-10-0-140-107.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:00:52.725238 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:00:52.725057 4939 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4adcd893-4e34-4860-bafb-a32ee8b956f0-istio-envoy\") on node \"ip-10-0-140-107.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:00:52.725238 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:00:52.725066 4939 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/44fe729d-d7d1-41c2-99df-60e955909b8b-istio-envoy\") on node \"ip-10-0-140-107.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:00:52.725238 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:00:52.725075 4939 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-62w7v\" (UniqueName: \"kubernetes.io/projected/4adcd893-4e34-4860-bafb-a32ee8b956f0-kube-api-access-62w7v\") on node \"ip-10-0-140-107.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:00:52.725238 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:00:52.725083 4939 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4adcd893-4e34-4860-bafb-a32ee8b956f0-workload-socket\") on node \"ip-10-0-140-107.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:00:52.725238 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:00:52.725092 4939 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4adcd893-4e34-4860-bafb-a32ee8b956f0-istio-data\") on node \"ip-10-0-140-107.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:00:52.725238 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:00:52.725100 4939 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/44fe729d-d7d1-41c2-99df-60e955909b8b-istio-podinfo\") on node \"ip-10-0-140-107.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:00:52.725238 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:00:52.725109 4939 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/44fe729d-d7d1-41c2-99df-60e955909b8b-istio-token\") on node \"ip-10-0-140-107.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:00:52.725238 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:00:52.725117 4939 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4adcd893-4e34-4860-bafb-a32ee8b956f0-istiod-ca-cert\") on node \"ip-10-0-140-107.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:00:52.725238 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:00:52.725125 4939 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4adcd893-4e34-4860-bafb-a32ee8b956f0-istio-podinfo\") on node \"ip-10-0-140-107.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:00:52.725238 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:00:52.725133 4939 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/44fe729d-d7d1-41c2-99df-60e955909b8b-workload-socket\") on node \"ip-10-0-140-107.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:00:52.725238 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:00:52.725142 4939 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/44fe729d-d7d1-41c2-99df-60e955909b8b-credential-socket\") on node \"ip-10-0-140-107.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:00:52.739046 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:00:52.739018 4939 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-02e4c0-7796479459-fjkx9" Nov 03 01:00:52.739167 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:00:52.739108 4939 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-02e4c0-57ff55f5cf-72j67" Nov 03 01:00:53.038450 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:00:53.038415 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:00:53.195325 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:00:53.195301 4939 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-7406a9-openshift-default-74c9f8d955-rqrs4" Nov 03 01:00:53.195453 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:00:53.195322 4939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-7406a9-openshift-default-74c9f8d955-rqrs4" event={"ID":"ab2a5b74-e0ad-4a13-9d46-7b78d5ef8aa7","Type":"ContainerDied","Data":"a12d574c8798148e0f5242f7c103db97f295572fa048fd637a45f0c59e3937e0"} Nov 03 01:00:53.195453 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:00:53.195383 4939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-83755a-openshift-default-c86f9565c-rwxw4" event={"ID":"4adcd893-4e34-4860-bafb-a32ee8b956f0","Type":"ContainerDied","Data":"d63e6ba311f7c32592f99e41dd118860a99b3a9e540667037881f767af6eec95"} Nov 03 01:00:53.195453 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:00:53.195404 4939 scope.go:122] "RemoveContainer" containerID="653e46bdc0ec8b7a25e2ab7986ac2c3cac776da6eba42b8ef8633f9e12bef9bb" Nov 03 01:00:53.195453 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:00:53.195417 4939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-83755a-openshift-default-c86f9565c-rwxw4" event={"ID":"4adcd893-4e34-4860-bafb-a32ee8b956f0","Type":"ContainerDied","Data":"dc4ec1e404447874693445ec056c5a2049949417ecf31cb0dbd4a0b470bebccb"} Nov 03 01:00:53.195651 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:00:53.195590 4939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-83755a-openshift-default-c86f9565c-nzmzn" event={"ID":"44fe729d-d7d1-41c2-99df-60e955909b8b","Type":"ContainerDied","Data":"74d2ce58c7345c8b44395dda007f1db79217e50d329cd8e7791d0422eed2d738"} Nov 03 01:00:53.195712 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:00:53.195653 4939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-83755a-openshift-default-c86f9565c-nzmzn" event={"ID":"44fe729d-d7d1-41c2-99df-60e955909b8b","Type":"ContainerDied","Data":"0d7a5a36d2f696ca5bb42ac53e4f8a2d178d8b5ba57e7098cdd74fe68c23d412"} Nov 03 01:00:53.197634 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:00:53.197606 4939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-02e4c0-575bdc8b8c-ss7s5" event={"ID":"67809b22-2bbf-452b-84fc-b7711dd222d0","Type":"ContainerStarted","Data":"bb4edc4267f626e3d1d52e06442c0c92b9094c864504cb4c2c7987282da2c3eb"} Nov 03 01:00:53.197765 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:00:53.197645 4939 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-83755a-openshift-default-c86f9565c-rwxw4" Nov 03 01:00:53.197765 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:00:53.197750 4939 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-83755a-openshift-default-c86f9565c-nzmzn" Nov 03 01:00:53.197877 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:00:53.197850 4939 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-02e4c0-575bdc8b8c-ss7s5" Nov 03 01:00:53.225401 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:00:53.225358 4939 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-02e4c0-575bdc8b8c-ss7s5" podStartSLOduration=2.289932732 podStartE2EDuration="3.225346843s" totalImagesPullingTime="935.414111ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 01:00:50 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 01:00:53.223435044 +0000 UTC m=+1201.487769990" watchObservedRunningTime="2026-11-03 01:00:53.225346843 +0000 UTC m=+1201.489681787" Nov 03 01:00:53.325383 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:00:53.325324 4939 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-7406a9-openshift-default-74c9f8d955-rqrs4"] Nov 03 01:00:53.331089 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:00:53.331031 4939 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-a-7406a9-openshift-default-74c9f8d955-rqrs4"] Nov 03 01:00:53.336777 ip-10-0-140-107 kubenswrapper[4939]: E1103 01:00:53.336734 4939 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5ca6f6a0_78ff_4b3f_ad1f_b94428b9178e.slice/crio-19473538a2c75199337c996b9d3339e90e813285dc8c6eeb33c78557b70e09fd.scope\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4adcd893_4e34_4860_bafb_a32ee8b956f0.slice\": RecentStats: unable to find data in memory cache]" Nov 03 01:00:53.432241 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:00:53.363852 4939 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-83755a-openshift-default-c86f9565c-rwxw4"] Nov 03 01:00:53.432241 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:00:53.370452 4939 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-83755a-openshift-default-c86f9565c-rwxw4"] Nov 03 01:00:53.449263 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:00:53.438205 4939 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-83755a-openshift-default-c86f9565c-nzmzn"] Nov 03 01:00:53.449263 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:00:53.438271 4939 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-83755a-openshift-default-c86f9565c-nzmzn"] Nov 03 01:00:53.571113 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:00:53.571078 4939 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-2f9a3a-openshift-default-7f6f74cf97-ph469" Nov 03 01:00:53.592014 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:00:53.591976 4939 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-2f9a3a-openshift-default-7f6f74cf97-j8q97" Nov 03 01:00:53.652129 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:00:53.652101 4939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d96472d8-fbad-44c0-815d-bf4d20cce46a-kube-api-access-srpmf\" (UniqueName: \"kubernetes.io/projected/d96472d8-fbad-44c0-815d-bf4d20cce46a-kube-api-access-srpmf\") pod \"d96472d8-fbad-44c0-815d-bf4d20cce46a\" (UID: \"d96472d8-fbad-44c0-815d-bf4d20cce46a\") " Nov 03 01:00:53.652328 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:00:53.652136 4939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d96472d8-fbad-44c0-815d-bf4d20cce46a-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d96472d8-fbad-44c0-815d-bf4d20cce46a-workload-socket\") pod \"d96472d8-fbad-44c0-815d-bf4d20cce46a\" (UID: \"d96472d8-fbad-44c0-815d-bf4d20cce46a\") " Nov 03 01:00:53.652328 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:00:53.652203 4939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/d96472d8-fbad-44c0-815d-bf4d20cce46a-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d96472d8-fbad-44c0-815d-bf4d20cce46a-istio-podinfo\") pod \"d96472d8-fbad-44c0-815d-bf4d20cce46a\" (UID: \"d96472d8-fbad-44c0-815d-bf4d20cce46a\") " Nov 03 01:00:53.652470 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:00:53.652372 4939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d96472d8-fbad-44c0-815d-bf4d20cce46a-istio-token\" (UniqueName: \"kubernetes.io/projected/d96472d8-fbad-44c0-815d-bf4d20cce46a-istio-token\") pod \"d96472d8-fbad-44c0-815d-bf4d20cce46a\" (UID: \"d96472d8-fbad-44c0-815d-bf4d20cce46a\") " Nov 03 01:00:53.654010 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:00:53.653974 4939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d96472d8-fbad-44c0-815d-bf4d20cce46a-workload-socket" pod "d96472d8-fbad-44c0-815d-bf4d20cce46a" (UID: "d96472d8-fbad-44c0-815d-bf4d20cce46a"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 01:00:53.654597 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:00:53.654571 4939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d96472d8-fbad-44c0-815d-bf4d20cce46a-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d96472d8-fbad-44c0-815d-bf4d20cce46a-istio-data\") pod \"d96472d8-fbad-44c0-815d-bf4d20cce46a\" (UID: \"d96472d8-fbad-44c0-815d-bf4d20cce46a\") " Nov 03 01:00:53.654726 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:00:53.654614 4939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5ca6f6a0-78ff-4b3f-ad1f-b94428b9178e-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5ca6f6a0-78ff-4b3f-ad1f-b94428b9178e-workload-certs\") pod \"5ca6f6a0-78ff-4b3f-ad1f-b94428b9178e\" (UID: \"5ca6f6a0-78ff-4b3f-ad1f-b94428b9178e\") " Nov 03 01:00:53.654726 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:00:53.654660 4939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5ca6f6a0-78ff-4b3f-ad1f-b94428b9178e-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5ca6f6a0-78ff-4b3f-ad1f-b94428b9178e-workload-socket\") pod \"5ca6f6a0-78ff-4b3f-ad1f-b94428b9178e\" (UID: \"5ca6f6a0-78ff-4b3f-ad1f-b94428b9178e\") " Nov 03 01:00:53.654852 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:00:53.654772 4939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5ca6f6a0-78ff-4b3f-ad1f-b94428b9178e-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5ca6f6a0-78ff-4b3f-ad1f-b94428b9178e-istio-envoy\") pod \"5ca6f6a0-78ff-4b3f-ad1f-b94428b9178e\" (UID: \"5ca6f6a0-78ff-4b3f-ad1f-b94428b9178e\") " Nov 03 01:00:53.654852 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:00:53.654820 4939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5ca6f6a0-78ff-4b3f-ad1f-b94428b9178e-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5ca6f6a0-78ff-4b3f-ad1f-b94428b9178e-istio-data\") pod \"5ca6f6a0-78ff-4b3f-ad1f-b94428b9178e\" (UID: \"5ca6f6a0-78ff-4b3f-ad1f-b94428b9178e\") " Nov 03 01:00:53.654852 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:00:53.654818 4939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d96472d8-fbad-44c0-815d-bf4d20cce46a-istio-data" pod "d96472d8-fbad-44c0-815d-bf4d20cce46a" (UID: "d96472d8-fbad-44c0-815d-bf4d20cce46a"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 01:00:53.655072 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:00:53.654853 4939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5ca6f6a0-78ff-4b3f-ad1f-b94428b9178e-kube-api-access-9kjbw\" (UniqueName: \"kubernetes.io/projected/5ca6f6a0-78ff-4b3f-ad1f-b94428b9178e-kube-api-access-9kjbw\") pod \"5ca6f6a0-78ff-4b3f-ad1f-b94428b9178e\" (UID: \"5ca6f6a0-78ff-4b3f-ad1f-b94428b9178e\") " Nov 03 01:00:53.655072 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:00:53.654997 4939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5ca6f6a0-78ff-4b3f-ad1f-b94428b9178e-workload-certs" pod "5ca6f6a0-78ff-4b3f-ad1f-b94428b9178e" (UID: "5ca6f6a0-78ff-4b3f-ad1f-b94428b9178e"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 01:00:53.655072 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:00:53.655000 4939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d96472d8-fbad-44c0-815d-bf4d20cce46a-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d96472d8-fbad-44c0-815d-bf4d20cce46a-credential-socket\") pod \"d96472d8-fbad-44c0-815d-bf4d20cce46a\" (UID: \"d96472d8-fbad-44c0-815d-bf4d20cce46a\") " Nov 03 01:00:53.655072 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:00:53.655063 4939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5ca6f6a0-78ff-4b3f-ad1f-b94428b9178e-istio-token\" (UniqueName: \"kubernetes.io/projected/5ca6f6a0-78ff-4b3f-ad1f-b94428b9178e-istio-token\") pod \"5ca6f6a0-78ff-4b3f-ad1f-b94428b9178e\" (UID: \"5ca6f6a0-78ff-4b3f-ad1f-b94428b9178e\") " Nov 03 01:00:53.655281 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:00:53.655106 4939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d96472d8-fbad-44c0-815d-bf4d20cce46a-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d96472d8-fbad-44c0-815d-bf4d20cce46a-istiod-ca-cert\") pod \"d96472d8-fbad-44c0-815d-bf4d20cce46a\" (UID: \"d96472d8-fbad-44c0-815d-bf4d20cce46a\") " Nov 03 01:00:53.655281 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:00:53.655169 4939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d96472d8-fbad-44c0-815d-bf4d20cce46a-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d96472d8-fbad-44c0-815d-bf4d20cce46a-workload-certs\") pod \"d96472d8-fbad-44c0-815d-bf4d20cce46a\" (UID: \"d96472d8-fbad-44c0-815d-bf4d20cce46a\") " Nov 03 01:00:53.655281 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:00:53.655196 4939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5ca6f6a0-78ff-4b3f-ad1f-b94428b9178e-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5ca6f6a0-78ff-4b3f-ad1f-b94428b9178e-istiod-ca-cert\") pod \"5ca6f6a0-78ff-4b3f-ad1f-b94428b9178e\" (UID: \"5ca6f6a0-78ff-4b3f-ad1f-b94428b9178e\") " Nov 03 01:00:53.655281 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:00:53.655204 4939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5ca6f6a0-78ff-4b3f-ad1f-b94428b9178e-istio-data" pod "5ca6f6a0-78ff-4b3f-ad1f-b94428b9178e" (UID: "5ca6f6a0-78ff-4b3f-ad1f-b94428b9178e"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 01:00:53.655281 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:00:53.655228 4939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d96472d8-fbad-44c0-815d-bf4d20cce46a-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d96472d8-fbad-44c0-815d-bf4d20cce46a-istio-envoy\") pod \"d96472d8-fbad-44c0-815d-bf4d20cce46a\" (UID: \"d96472d8-fbad-44c0-815d-bf4d20cce46a\") " Nov 03 01:00:53.655281 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:00:53.655253 4939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d96472d8-fbad-44c0-815d-bf4d20cce46a-credential-socket" pod "d96472d8-fbad-44c0-815d-bf4d20cce46a" (UID: "d96472d8-fbad-44c0-815d-bf4d20cce46a"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 01:00:53.655281 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:00:53.655271 4939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/5ca6f6a0-78ff-4b3f-ad1f-b94428b9178e-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5ca6f6a0-78ff-4b3f-ad1f-b94428b9178e-istio-podinfo\") pod \"5ca6f6a0-78ff-4b3f-ad1f-b94428b9178e\" (UID: \"5ca6f6a0-78ff-4b3f-ad1f-b94428b9178e\") " Nov 03 01:00:53.655651 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:00:53.655365 4939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5ca6f6a0-78ff-4b3f-ad1f-b94428b9178e-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5ca6f6a0-78ff-4b3f-ad1f-b94428b9178e-credential-socket\") pod \"5ca6f6a0-78ff-4b3f-ad1f-b94428b9178e\" (UID: \"5ca6f6a0-78ff-4b3f-ad1f-b94428b9178e\") " Nov 03 01:00:53.655651 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:00:53.655444 4939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d96472d8-fbad-44c0-815d-bf4d20cce46a-workload-certs" pod "d96472d8-fbad-44c0-815d-bf4d20cce46a" (UID: "d96472d8-fbad-44c0-815d-bf4d20cce46a"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 01:00:53.656366 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:00:53.656339 4939 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d96472d8-fbad-44c0-815d-bf4d20cce46a-workload-certs\") on node \"ip-10-0-140-107.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:00:53.656491 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:00:53.656369 4939 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d96472d8-fbad-44c0-815d-bf4d20cce46a-workload-socket\") on node \"ip-10-0-140-107.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:00:53.656491 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:00:53.656386 4939 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d96472d8-fbad-44c0-815d-bf4d20cce46a-istio-data\") on node \"ip-10-0-140-107.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:00:53.656491 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:00:53.656401 4939 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5ca6f6a0-78ff-4b3f-ad1f-b94428b9178e-workload-certs\") on node \"ip-10-0-140-107.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:00:53.656491 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:00:53.656422 4939 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5ca6f6a0-78ff-4b3f-ad1f-b94428b9178e-istio-data\") on node \"ip-10-0-140-107.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:00:53.656491 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:00:53.656437 4939 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d96472d8-fbad-44c0-815d-bf4d20cce46a-credential-socket\") on node \"ip-10-0-140-107.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:00:53.656749 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:00:53.656500 4939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5ca6f6a0-78ff-4b3f-ad1f-b94428b9178e-istiod-ca-cert" pod "5ca6f6a0-78ff-4b3f-ad1f-b94428b9178e" (UID: "5ca6f6a0-78ff-4b3f-ad1f-b94428b9178e"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 01:00:53.656749 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:00:53.656704 4939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5ca6f6a0-78ff-4b3f-ad1f-b94428b9178e-credential-socket" pod "5ca6f6a0-78ff-4b3f-ad1f-b94428b9178e" (UID: "5ca6f6a0-78ff-4b3f-ad1f-b94428b9178e"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 01:00:53.656963 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:00:53.656919 4939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d96472d8-fbad-44c0-815d-bf4d20cce46a-istiod-ca-cert" pod "d96472d8-fbad-44c0-815d-bf4d20cce46a" (UID: "d96472d8-fbad-44c0-815d-bf4d20cce46a"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 01:00:53.657250 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:00:53.657216 4939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5ca6f6a0-78ff-4b3f-ad1f-b94428b9178e-istio-envoy" pod "5ca6f6a0-78ff-4b3f-ad1f-b94428b9178e" (UID: "5ca6f6a0-78ff-4b3f-ad1f-b94428b9178e"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 01:00:53.657416 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:00:53.657387 4939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5ca6f6a0-78ff-4b3f-ad1f-b94428b9178e-workload-socket" pod "5ca6f6a0-78ff-4b3f-ad1f-b94428b9178e" (UID: "5ca6f6a0-78ff-4b3f-ad1f-b94428b9178e"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 01:00:53.658110 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:00:53.658085 4939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/d96472d8-fbad-44c0-815d-bf4d20cce46a-istio-podinfo" pod "d96472d8-fbad-44c0-815d-bf4d20cce46a" (UID: "d96472d8-fbad-44c0-815d-bf4d20cce46a"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 01:00:53.660033 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:00:53.659962 4939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d96472d8-fbad-44c0-815d-bf4d20cce46a-istio-envoy" pod "d96472d8-fbad-44c0-815d-bf4d20cce46a" (UID: "d96472d8-fbad-44c0-815d-bf4d20cce46a"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 01:00:53.660126 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:00:53.660015 4939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5ca6f6a0-78ff-4b3f-ad1f-b94428b9178e-istio-token" pod "5ca6f6a0-78ff-4b3f-ad1f-b94428b9178e" (UID: "5ca6f6a0-78ff-4b3f-ad1f-b94428b9178e"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 01:00:53.660174 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:00:53.660055 4939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d96472d8-fbad-44c0-815d-bf4d20cce46a-istio-token" pod "d96472d8-fbad-44c0-815d-bf4d20cce46a" (UID: "d96472d8-fbad-44c0-815d-bf4d20cce46a"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 01:00:53.660174 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:00:53.660074 4939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5ca6f6a0-78ff-4b3f-ad1f-b94428b9178e-kube-api-access-9kjbw" pod "5ca6f6a0-78ff-4b3f-ad1f-b94428b9178e" (UID: "5ca6f6a0-78ff-4b3f-ad1f-b94428b9178e"). InnerVolumeSpecName "kube-api-access-9kjbw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 01:00:53.660174 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:00:53.660089 4939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d96472d8-fbad-44c0-815d-bf4d20cce46a-kube-api-access-srpmf" pod "d96472d8-fbad-44c0-815d-bf4d20cce46a" (UID: "d96472d8-fbad-44c0-815d-bf4d20cce46a"). InnerVolumeSpecName "kube-api-access-srpmf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 01:00:53.661203 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:00:53.661177 4939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/5ca6f6a0-78ff-4b3f-ad1f-b94428b9178e-istio-podinfo" pod "5ca6f6a0-78ff-4b3f-ad1f-b94428b9178e" (UID: "5ca6f6a0-78ff-4b3f-ad1f-b94428b9178e"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 01:00:53.757672 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:00:53.757645 4939 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5ca6f6a0-78ff-4b3f-ad1f-b94428b9178e-workload-socket\") on node \"ip-10-0-140-107.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:00:53.757672 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:00:53.757673 4939 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5ca6f6a0-78ff-4b3f-ad1f-b94428b9178e-istio-envoy\") on node \"ip-10-0-140-107.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:00:53.757868 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:00:53.757687 4939 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9kjbw\" (UniqueName: \"kubernetes.io/projected/5ca6f6a0-78ff-4b3f-ad1f-b94428b9178e-kube-api-access-9kjbw\") on node \"ip-10-0-140-107.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:00:53.757868 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:00:53.757700 4939 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5ca6f6a0-78ff-4b3f-ad1f-b94428b9178e-istio-token\") on node \"ip-10-0-140-107.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:00:53.757868 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:00:53.757712 4939 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d96472d8-fbad-44c0-815d-bf4d20cce46a-istiod-ca-cert\") on node \"ip-10-0-140-107.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:00:53.757868 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:00:53.757724 4939 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5ca6f6a0-78ff-4b3f-ad1f-b94428b9178e-istiod-ca-cert\") on node \"ip-10-0-140-107.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:00:53.757868 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:00:53.757736 4939 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d96472d8-fbad-44c0-815d-bf4d20cce46a-istio-envoy\") on node \"ip-10-0-140-107.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:00:53.757868 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:00:53.757748 4939 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5ca6f6a0-78ff-4b3f-ad1f-b94428b9178e-istio-podinfo\") on node \"ip-10-0-140-107.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:00:53.757868 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:00:53.757762 4939 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5ca6f6a0-78ff-4b3f-ad1f-b94428b9178e-credential-socket\") on node \"ip-10-0-140-107.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:00:53.757868 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:00:53.757774 4939 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-srpmf\" (UniqueName: \"kubernetes.io/projected/d96472d8-fbad-44c0-815d-bf4d20cce46a-kube-api-access-srpmf\") on node \"ip-10-0-140-107.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:00:53.757868 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:00:53.757786 4939 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d96472d8-fbad-44c0-815d-bf4d20cce46a-istio-podinfo\") on node \"ip-10-0-140-107.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:00:53.757868 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:00:53.757800 4939 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d96472d8-fbad-44c0-815d-bf4d20cce46a-istio-token\") on node \"ip-10-0-140-107.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:00:53.869271 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:00:53.869237 4939 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="44fe729d-d7d1-41c2-99df-60e955909b8b" path="/var/lib/kubelet/pods/44fe729d-d7d1-41c2-99df-60e955909b8b/volumes" Nov 03 01:00:53.869733 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:00:53.869719 4939 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4adcd893-4e34-4860-bafb-a32ee8b956f0" path="/var/lib/kubelet/pods/4adcd893-4e34-4860-bafb-a32ee8b956f0/volumes" Nov 03 01:00:53.870238 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:00:53.870218 4939 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ab2a5b74-e0ad-4a13-9d46-7b78d5ef8aa7" path="/var/lib/kubelet/pods/ab2a5b74-e0ad-4a13-9d46-7b78d5ef8aa7/volumes" Nov 03 01:00:54.012693 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:00:54.012664 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:00:54.193123 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:00:54.193039 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:00:54.206523 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:00:54.206497 4939 generic.go:356] "Generic (PLEG): container finished" podID="d96472d8-fbad-44c0-815d-bf4d20cce46a" containerID="1f2e9fa18ac7e6c6c904f4612ece64a27ae0a4c622fa0970018363af08e3bc92" exitCode=0 Nov 03 01:00:54.206652 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:00:54.206554 4939 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-2f9a3a-openshift-default-7f6f74cf97-j8q97" Nov 03 01:00:54.206652 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:00:54.206589 4939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-2f9a3a-openshift-default-7f6f74cf97-j8q97" event={"ID":"d96472d8-fbad-44c0-815d-bf4d20cce46a","Type":"ContainerDied","Data":"1f2e9fa18ac7e6c6c904f4612ece64a27ae0a4c622fa0970018363af08e3bc92"} Nov 03 01:00:54.206736 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:00:54.206670 4939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-2f9a3a-openshift-default-7f6f74cf97-j8q97" event={"ID":"d96472d8-fbad-44c0-815d-bf4d20cce46a","Type":"ContainerDied","Data":"0981a3c88659ffd9728cd5def1d780ab4f6fe94232f1971ff22fb86b58b7b4ea"} Nov 03 01:00:54.206736 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:00:54.206693 4939 scope.go:122] "RemoveContainer" containerID="1f2e9fa18ac7e6c6c904f4612ece64a27ae0a4c622fa0970018363af08e3bc92" Nov 03 01:00:54.209274 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:00:54.209247 4939 generic.go:356] "Generic (PLEG): container finished" podID="5ca6f6a0-78ff-4b3f-ad1f-b94428b9178e" containerID="19473538a2c75199337c996b9d3339e90e813285dc8c6eeb33c78557b70e09fd" exitCode=0 Nov 03 01:00:54.209410 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:00:54.209386 4939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-2f9a3a-openshift-default-7f6f74cf97-ph469" event={"ID":"5ca6f6a0-78ff-4b3f-ad1f-b94428b9178e","Type":"ContainerDied","Data":"19473538a2c75199337c996b9d3339e90e813285dc8c6eeb33c78557b70e09fd"} Nov 03 01:00:54.209488 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:00:54.209439 4939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-2f9a3a-openshift-default-7f6f74cf97-ph469" event={"ID":"5ca6f6a0-78ff-4b3f-ad1f-b94428b9178e","Type":"ContainerDied","Data":"1a8e2b89707096775e4ed562b80204df0cefe937399955353e8fdbf96eca9977"} Nov 03 01:00:54.209579 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:00:54.209559 4939 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-2f9a3a-openshift-default-7f6f74cf97-ph469" Nov 03 01:00:54.220047 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:00:54.220025 4939 scope.go:122] "RemoveContainer" containerID="1f2e9fa18ac7e6c6c904f4612ece64a27ae0a4c622fa0970018363af08e3bc92" Nov 03 01:00:54.220362 ip-10-0-140-107 kubenswrapper[4939]: E1103 01:00:54.220335 4939 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1f2e9fa18ac7e6c6c904f4612ece64a27ae0a4c622fa0970018363af08e3bc92\": container with ID starting with 1f2e9fa18ac7e6c6c904f4612ece64a27ae0a4c622fa0970018363af08e3bc92 not found: ID does not exist" containerID="1f2e9fa18ac7e6c6c904f4612ece64a27ae0a4c622fa0970018363af08e3bc92" Nov 03 01:00:54.220464 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:00:54.220368 4939 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1f2e9fa18ac7e6c6c904f4612ece64a27ae0a4c622fa0970018363af08e3bc92"} err="failed to get container status \"1f2e9fa18ac7e6c6c904f4612ece64a27ae0a4c622fa0970018363af08e3bc92\": rpc error: code = NotFound desc = could not find container \"1f2e9fa18ac7e6c6c904f4612ece64a27ae0a4c622fa0970018363af08e3bc92\": container with ID starting with 1f2e9fa18ac7e6c6c904f4612ece64a27ae0a4c622fa0970018363af08e3bc92 not found: ID does not exist" Nov 03 01:00:54.220464 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:00:54.220392 4939 scope.go:122] "RemoveContainer" containerID="19473538a2c75199337c996b9d3339e90e813285dc8c6eeb33c78557b70e09fd" Nov 03 01:00:54.252948 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:00:54.251256 4939 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-2f9a3a-openshift-default-7f6f74cf97-j8q97"] Nov 03 01:00:54.275607 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:00:54.262658 4939 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-2f9a3a-openshift-default-7f6f74cf97-j8q97"] Nov 03 01:00:54.278850 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:00:54.278264 4939 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-2f9a3a-openshift-default-7f6f74cf97-ph469"] Nov 03 01:00:54.284877 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:00:54.284785 4939 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-2f9a3a-openshift-default-7f6f74cf97-ph469"] Nov 03 01:00:54.290217 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:00:54.290195 4939 scope.go:122] "RemoveContainer" containerID="19473538a2c75199337c996b9d3339e90e813285dc8c6eeb33c78557b70e09fd" Nov 03 01:00:54.290531 ip-10-0-140-107 kubenswrapper[4939]: E1103 01:00:54.290511 4939 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"19473538a2c75199337c996b9d3339e90e813285dc8c6eeb33c78557b70e09fd\": container with ID starting with 19473538a2c75199337c996b9d3339e90e813285dc8c6eeb33c78557b70e09fd not found: ID does not exist" containerID="19473538a2c75199337c996b9d3339e90e813285dc8c6eeb33c78557b70e09fd" Nov 03 01:00:54.290576 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:00:54.290540 4939 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"19473538a2c75199337c996b9d3339e90e813285dc8c6eeb33c78557b70e09fd"} err="failed to get container status \"19473538a2c75199337c996b9d3339e90e813285dc8c6eeb33c78557b70e09fd\": rpc error: code = NotFound desc = could not find container \"19473538a2c75199337c996b9d3339e90e813285dc8c6eeb33c78557b70e09fd\": container with ID starting with 19473538a2c75199337c996b9d3339e90e813285dc8c6eeb33c78557b70e09fd not found: ID does not exist" Nov 03 01:00:54.322208 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:00:54.322176 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:00:54.691780 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:00:54.691751 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:00:55.048276 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:00:55.048200 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:00:55.182038 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:00:55.181866 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:00:55.363938 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:00:55.354939 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:00:55.823555 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:00:55.823457 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:00:55.870749 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:00:55.870721 4939 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5ca6f6a0-78ff-4b3f-ad1f-b94428b9178e" path="/var/lib/kubelet/pods/5ca6f6a0-78ff-4b3f-ad1f-b94428b9178e/volumes" Nov 03 01:00:55.871257 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:00:55.871238 4939 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d96472d8-fbad-44c0-815d-bf4d20cce46a" path="/var/lib/kubelet/pods/d96472d8-fbad-44c0-815d-bf4d20cce46a/volumes" Nov 03 01:00:56.139656 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:00:56.139605 4939 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-api-b-ffdea6-6975889bf-cb625" podUID="7e00f3bb-5398-415b-bcb3-9db058e00a05" containerName="payload-processing" containerID="cri-o://ea3c68a83dde08f7b23120f46e7905ede68f9bbe36b69c47e2d2a03639dc3560" gracePeriod=25 Nov 03 01:00:56.201015 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:00:56.200966 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:00:56.547033 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:00:56.521559 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:00:56.587752 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:00:56.587475 4939 generic.go:356] "Generic (PLEG): container finished" podID="7e00f3bb-5398-415b-bcb3-9db058e00a05" containerID="ea3c68a83dde08f7b23120f46e7905ede68f9bbe36b69c47e2d2a03639dc3560" exitCode=0 Nov 03 01:00:56.587752 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:00:56.587594 4939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-ffdea6-6975889bf-cb625" event={"ID":"7e00f3bb-5398-415b-bcb3-9db058e00a05","Type":"ContainerDied","Data":"ea3c68a83dde08f7b23120f46e7905ede68f9bbe36b69c47e2d2a03639dc3560"} Nov 03 01:00:56.730222 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:00:56.730195 4939 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-ffdea6-6975889bf-cb625" Nov 03 01:00:56.907774 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:00:56.907739 4939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7e00f3bb-5398-415b-bcb3-9db058e00a05-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7e00f3bb-5398-415b-bcb3-9db058e00a05-plugins-config-volume\") pod \"7e00f3bb-5398-415b-bcb3-9db058e00a05\" (UID: \"7e00f3bb-5398-415b-bcb3-9db058e00a05\") " Nov 03 01:00:56.907942 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:00:56.907881 4939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7e00f3bb-5398-415b-bcb3-9db058e00a05-kube-api-access-g5996\" (UniqueName: \"kubernetes.io/projected/7e00f3bb-5398-415b-bcb3-9db058e00a05-kube-api-access-g5996\") pod \"7e00f3bb-5398-415b-bcb3-9db058e00a05\" (UID: \"7e00f3bb-5398-415b-bcb3-9db058e00a05\") " Nov 03 01:00:56.908139 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:00:56.908117 4939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7e00f3bb-5398-415b-bcb3-9db058e00a05-plugins-config-volume" pod "7e00f3bb-5398-415b-bcb3-9db058e00a05" (UID: "7e00f3bb-5398-415b-bcb3-9db058e00a05"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 01:00:56.908531 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:00:56.908513 4939 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7e00f3bb-5398-415b-bcb3-9db058e00a05-plugins-config-volume\") on node \"ip-10-0-140-107.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:00:56.910048 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:00:56.910023 4939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7e00f3bb-5398-415b-bcb3-9db058e00a05-kube-api-access-g5996" pod "7e00f3bb-5398-415b-bcb3-9db058e00a05" (UID: "7e00f3bb-5398-415b-bcb3-9db058e00a05"). InnerVolumeSpecName "kube-api-access-g5996". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 01:00:57.010254 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:00:57.010223 4939 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-g5996\" (UniqueName: \"kubernetes.io/projected/7e00f3bb-5398-415b-bcb3-9db058e00a05-kube-api-access-g5996\") on node \"ip-10-0-140-107.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:00:57.463875 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:00:57.463843 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:00:57.596140 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:00:57.596108 4939 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-ffdea6-6975889bf-cb625" Nov 03 01:00:57.596571 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:00:57.596149 4939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-ffdea6-6975889bf-cb625" event={"ID":"7e00f3bb-5398-415b-bcb3-9db058e00a05","Type":"ContainerDied","Data":"88f9fdc188279cc05a660047995b9c103c7a84e68c8f6402f8ee136da990a06c"} Nov 03 01:00:57.596571 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:00:57.596212 4939 scope.go:122] "RemoveContainer" containerID="ea3c68a83dde08f7b23120f46e7905ede68f9bbe36b69c47e2d2a03639dc3560" Nov 03 01:00:57.624775 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:00:57.624754 4939 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-ffdea6-6975889bf-cb625"] Nov 03 01:00:57.636599 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:00:57.636577 4939 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-ffdea6-6975889bf-cb625"] Nov 03 01:00:57.869747 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:00:57.869705 4939 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7e00f3bb-5398-415b-bcb3-9db058e00a05" path="/var/lib/kubelet/pods/7e00f3bb-5398-415b-bcb3-9db058e00a05/volumes" Nov 03 01:00:58.123802 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:00:58.123721 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:00:58.611508 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:00:58.611475 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:00:58.741328 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:00:58.741294 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:00:58.872411 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:00:58.872334 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:00:59.173660 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:00:59.173582 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:00:59.242403 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:00:59.242375 4939 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-02e4c0-575bdc8b8c-ss7s5" Nov 03 01:00:59.562300 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:00:59.562225 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:01:00.182235 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:01:00.182195 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:01:00.517548 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:01:00.517455 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:01:00.900530 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:01:00.900498 4939 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-a-dc0f3f-openshift-default-d9f67c9bc-ks8t6"] Nov 03 01:01:00.901666 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:01:00.901639 4939 cpu_manager.go:414] "containerMap: removing container" podUID="4adcd893-4e34-4860-bafb-a32ee8b956f0" containerName="istio-proxy" Nov 03 01:01:00.901666 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:01:00.901668 4939 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4adcd893-4e34-4860-bafb-a32ee8b956f0" containerName="istio-proxy" Nov 03 01:01:00.901812 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:01:00.901704 4939 cpu_manager.go:414] "containerMap: removing container" podUID="7e00f3bb-5398-415b-bcb3-9db058e00a05" containerName="payload-processing" Nov 03 01:01:00.901812 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:01:00.901712 4939 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7e00f3bb-5398-415b-bcb3-9db058e00a05" containerName="payload-processing" Nov 03 01:01:00.901812 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:01:00.901726 4939 cpu_manager.go:414] "containerMap: removing container" podUID="d96472d8-fbad-44c0-815d-bf4d20cce46a" containerName="istio-proxy" Nov 03 01:01:00.901812 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:01:00.901733 4939 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d96472d8-fbad-44c0-815d-bf4d20cce46a" containerName="istio-proxy" Nov 03 01:01:00.901812 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:01:00.901743 4939 cpu_manager.go:414] "containerMap: removing container" podUID="ab2a5b74-e0ad-4a13-9d46-7b78d5ef8aa7" containerName="istio-proxy" Nov 03 01:01:00.901812 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:01:00.901750 4939 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ab2a5b74-e0ad-4a13-9d46-7b78d5ef8aa7" containerName="istio-proxy" Nov 03 01:01:00.901812 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:01:00.901782 4939 cpu_manager.go:414] "containerMap: removing container" podUID="ab2a5b74-e0ad-4a13-9d46-7b78d5ef8aa7" containerName="istio-proxy" Nov 03 01:01:00.901812 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:01:00.901790 4939 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ab2a5b74-e0ad-4a13-9d46-7b78d5ef8aa7" containerName="istio-proxy" Nov 03 01:01:00.901812 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:01:00.901811 4939 cpu_manager.go:414] "containerMap: removing container" podUID="5ca6f6a0-78ff-4b3f-ad1f-b94428b9178e" containerName="istio-proxy" Nov 03 01:01:00.902134 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:01:00.901819 4939 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5ca6f6a0-78ff-4b3f-ad1f-b94428b9178e" containerName="istio-proxy" Nov 03 01:01:00.902134 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:01:00.901834 4939 cpu_manager.go:414] "containerMap: removing container" podUID="44fe729d-d7d1-41c2-99df-60e955909b8b" containerName="istio-proxy" Nov 03 01:01:00.902134 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:01:00.901841 4939 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="44fe729d-d7d1-41c2-99df-60e955909b8b" containerName="istio-proxy" Nov 03 01:01:00.902134 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:01:00.902028 4939 memory_manager.go:365] "RemoveStaleState removing state" podUID="4adcd893-4e34-4860-bafb-a32ee8b956f0" containerName="istio-proxy" Nov 03 01:01:00.902134 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:01:00.902042 4939 memory_manager.go:365] "RemoveStaleState removing state" podUID="7e00f3bb-5398-415b-bcb3-9db058e00a05" containerName="payload-processing" Nov 03 01:01:00.902134 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:01:00.902052 4939 memory_manager.go:365] "RemoveStaleState removing state" podUID="d96472d8-fbad-44c0-815d-bf4d20cce46a" containerName="istio-proxy" Nov 03 01:01:00.902134 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:01:00.902059 4939 memory_manager.go:365] "RemoveStaleState removing state" podUID="ab2a5b74-e0ad-4a13-9d46-7b78d5ef8aa7" containerName="istio-proxy" Nov 03 01:01:00.902134 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:01:00.902067 4939 memory_manager.go:365] "RemoveStaleState removing state" podUID="ab2a5b74-e0ad-4a13-9d46-7b78d5ef8aa7" containerName="istio-proxy" Nov 03 01:01:00.902134 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:01:00.902078 4939 memory_manager.go:365] "RemoveStaleState removing state" podUID="5ca6f6a0-78ff-4b3f-ad1f-b94428b9178e" containerName="istio-proxy" Nov 03 01:01:00.902134 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:01:00.902090 4939 memory_manager.go:365] "RemoveStaleState removing state" podUID="44fe729d-d7d1-41c2-99df-60e955909b8b" containerName="istio-proxy" Nov 03 01:01:00.911746 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:01:00.911720 4939 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-dc0f3f-openshift-default-d9f67c9bc-ks8t6"] Nov 03 01:01:00.913019 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:01:00.912185 4939 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-dc0f3f-openshift-default-d9f67c9bc-ks8t6" Nov 03 01:01:00.968238 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:01:00.968174 4939 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ipp-a-dc0f3f-openshift-default-dockercfg-z2dfw\"" Nov 03 01:01:00.971923 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:01:00.968630 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:01:01.033624 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:01:01.033590 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6c13fe13-2eda-4d7c-bf31-c522d7f3b9c3-istio-podinfo\") pod \"e2e-ipp-a-dc0f3f-openshift-default-d9f67c9bc-ks8t6\" (UID: \"6c13fe13-2eda-4d7c-bf31-c522d7f3b9c3\") " pod="openshift-ingress/e2e-ipp-a-dc0f3f-openshift-default-d9f67c9bc-ks8t6" Nov 03 01:01:01.033759 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:01:01.033645 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6c13fe13-2eda-4d7c-bf31-c522d7f3b9c3-istio-envoy\") pod \"e2e-ipp-a-dc0f3f-openshift-default-d9f67c9bc-ks8t6\" (UID: \"6c13fe13-2eda-4d7c-bf31-c522d7f3b9c3\") " pod="openshift-ingress/e2e-ipp-a-dc0f3f-openshift-default-d9f67c9bc-ks8t6" Nov 03 01:01:01.033759 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:01:01.033744 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6c13fe13-2eda-4d7c-bf31-c522d7f3b9c3-workload-certs\") pod \"e2e-ipp-a-dc0f3f-openshift-default-d9f67c9bc-ks8t6\" (UID: \"6c13fe13-2eda-4d7c-bf31-c522d7f3b9c3\") " pod="openshift-ingress/e2e-ipp-a-dc0f3f-openshift-default-d9f67c9bc-ks8t6" Nov 03 01:01:01.033887 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:01:01.033774 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6c13fe13-2eda-4d7c-bf31-c522d7f3b9c3-istio-data\") pod \"e2e-ipp-a-dc0f3f-openshift-default-d9f67c9bc-ks8t6\" (UID: \"6c13fe13-2eda-4d7c-bf31-c522d7f3b9c3\") " pod="openshift-ingress/e2e-ipp-a-dc0f3f-openshift-default-d9f67c9bc-ks8t6" Nov 03 01:01:01.033887 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:01:01.033832 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6c13fe13-2eda-4d7c-bf31-c522d7f3b9c3-istiod-ca-cert\") pod \"e2e-ipp-a-dc0f3f-openshift-default-d9f67c9bc-ks8t6\" (UID: \"6c13fe13-2eda-4d7c-bf31-c522d7f3b9c3\") " pod="openshift-ingress/e2e-ipp-a-dc0f3f-openshift-default-d9f67c9bc-ks8t6" Nov 03 01:01:01.034031 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:01:01.033901 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6c13fe13-2eda-4d7c-bf31-c522d7f3b9c3-credential-socket\") pod \"e2e-ipp-a-dc0f3f-openshift-default-d9f67c9bc-ks8t6\" (UID: \"6c13fe13-2eda-4d7c-bf31-c522d7f3b9c3\") " pod="openshift-ingress/e2e-ipp-a-dc0f3f-openshift-default-d9f67c9bc-ks8t6" Nov 03 01:01:01.034031 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:01:01.033946 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6c13fe13-2eda-4d7c-bf31-c522d7f3b9c3-workload-socket\") pod \"e2e-ipp-a-dc0f3f-openshift-default-d9f67c9bc-ks8t6\" (UID: \"6c13fe13-2eda-4d7c-bf31-c522d7f3b9c3\") " pod="openshift-ingress/e2e-ipp-a-dc0f3f-openshift-default-d9f67c9bc-ks8t6" Nov 03 01:01:01.034126 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:01:01.034068 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6c13fe13-2eda-4d7c-bf31-c522d7f3b9c3-istio-token\") pod \"e2e-ipp-a-dc0f3f-openshift-default-d9f67c9bc-ks8t6\" (UID: \"6c13fe13-2eda-4d7c-bf31-c522d7f3b9c3\") " pod="openshift-ingress/e2e-ipp-a-dc0f3f-openshift-default-d9f67c9bc-ks8t6" Nov 03 01:01:01.034222 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:01:01.034203 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6wxjj\" (UniqueName: \"kubernetes.io/projected/6c13fe13-2eda-4d7c-bf31-c522d7f3b9c3-kube-api-access-6wxjj\") pod \"e2e-ipp-a-dc0f3f-openshift-default-d9f67c9bc-ks8t6\" (UID: \"6c13fe13-2eda-4d7c-bf31-c522d7f3b9c3\") " pod="openshift-ingress/e2e-ipp-a-dc0f3f-openshift-default-d9f67c9bc-ks8t6" Nov 03 01:01:01.144284 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:01:01.138051 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6wxjj\" (UniqueName: \"kubernetes.io/projected/6c13fe13-2eda-4d7c-bf31-c522d7f3b9c3-kube-api-access-6wxjj\") pod \"e2e-ipp-a-dc0f3f-openshift-default-d9f67c9bc-ks8t6\" (UID: \"6c13fe13-2eda-4d7c-bf31-c522d7f3b9c3\") " pod="openshift-ingress/e2e-ipp-a-dc0f3f-openshift-default-d9f67c9bc-ks8t6" Nov 03 01:01:01.144284 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:01:01.138176 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6c13fe13-2eda-4d7c-bf31-c522d7f3b9c3-istio-podinfo\") pod \"e2e-ipp-a-dc0f3f-openshift-default-d9f67c9bc-ks8t6\" (UID: \"6c13fe13-2eda-4d7c-bf31-c522d7f3b9c3\") " pod="openshift-ingress/e2e-ipp-a-dc0f3f-openshift-default-d9f67c9bc-ks8t6" Nov 03 01:01:01.144284 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:01:01.138229 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6c13fe13-2eda-4d7c-bf31-c522d7f3b9c3-istio-envoy\") pod \"e2e-ipp-a-dc0f3f-openshift-default-d9f67c9bc-ks8t6\" (UID: \"6c13fe13-2eda-4d7c-bf31-c522d7f3b9c3\") " pod="openshift-ingress/e2e-ipp-a-dc0f3f-openshift-default-d9f67c9bc-ks8t6" Nov 03 01:01:01.144284 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:01:01.138282 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6c13fe13-2eda-4d7c-bf31-c522d7f3b9c3-workload-certs\") pod \"e2e-ipp-a-dc0f3f-openshift-default-d9f67c9bc-ks8t6\" (UID: \"6c13fe13-2eda-4d7c-bf31-c522d7f3b9c3\") " pod="openshift-ingress/e2e-ipp-a-dc0f3f-openshift-default-d9f67c9bc-ks8t6" Nov 03 01:01:01.144284 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:01:01.138332 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6c13fe13-2eda-4d7c-bf31-c522d7f3b9c3-istio-data\") pod \"e2e-ipp-a-dc0f3f-openshift-default-d9f67c9bc-ks8t6\" (UID: \"6c13fe13-2eda-4d7c-bf31-c522d7f3b9c3\") " pod="openshift-ingress/e2e-ipp-a-dc0f3f-openshift-default-d9f67c9bc-ks8t6" Nov 03 01:01:01.144284 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:01:01.138368 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6c13fe13-2eda-4d7c-bf31-c522d7f3b9c3-istiod-ca-cert\") pod \"e2e-ipp-a-dc0f3f-openshift-default-d9f67c9bc-ks8t6\" (UID: \"6c13fe13-2eda-4d7c-bf31-c522d7f3b9c3\") " pod="openshift-ingress/e2e-ipp-a-dc0f3f-openshift-default-d9f67c9bc-ks8t6" Nov 03 01:01:01.144284 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:01:01.138395 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6c13fe13-2eda-4d7c-bf31-c522d7f3b9c3-credential-socket\") pod \"e2e-ipp-a-dc0f3f-openshift-default-d9f67c9bc-ks8t6\" (UID: \"6c13fe13-2eda-4d7c-bf31-c522d7f3b9c3\") " pod="openshift-ingress/e2e-ipp-a-dc0f3f-openshift-default-d9f67c9bc-ks8t6" Nov 03 01:01:01.144284 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:01:01.138423 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6c13fe13-2eda-4d7c-bf31-c522d7f3b9c3-workload-socket\") pod \"e2e-ipp-a-dc0f3f-openshift-default-d9f67c9bc-ks8t6\" (UID: \"6c13fe13-2eda-4d7c-bf31-c522d7f3b9c3\") " pod="openshift-ingress/e2e-ipp-a-dc0f3f-openshift-default-d9f67c9bc-ks8t6" Nov 03 01:01:01.144284 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:01:01.138505 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6c13fe13-2eda-4d7c-bf31-c522d7f3b9c3-istio-token\") pod \"e2e-ipp-a-dc0f3f-openshift-default-d9f67c9bc-ks8t6\" (UID: \"6c13fe13-2eda-4d7c-bf31-c522d7f3b9c3\") " pod="openshift-ingress/e2e-ipp-a-dc0f3f-openshift-default-d9f67c9bc-ks8t6" Nov 03 01:01:01.144929 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:01:01.144344 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6c13fe13-2eda-4d7c-bf31-c522d7f3b9c3-istio-data\") pod \"e2e-ipp-a-dc0f3f-openshift-default-d9f67c9bc-ks8t6\" (UID: \"6c13fe13-2eda-4d7c-bf31-c522d7f3b9c3\") " pod="openshift-ingress/e2e-ipp-a-dc0f3f-openshift-default-d9f67c9bc-ks8t6" Nov 03 01:01:01.152053 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:01:01.145263 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:01:01.152053 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:01:01.145624 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6c13fe13-2eda-4d7c-bf31-c522d7f3b9c3-workload-certs\") pod \"e2e-ipp-a-dc0f3f-openshift-default-d9f67c9bc-ks8t6\" (UID: \"6c13fe13-2eda-4d7c-bf31-c522d7f3b9c3\") " pod="openshift-ingress/e2e-ipp-a-dc0f3f-openshift-default-d9f67c9bc-ks8t6" Nov 03 01:01:01.152053 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:01:01.145894 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6c13fe13-2eda-4d7c-bf31-c522d7f3b9c3-credential-socket\") pod \"e2e-ipp-a-dc0f3f-openshift-default-d9f67c9bc-ks8t6\" (UID: \"6c13fe13-2eda-4d7c-bf31-c522d7f3b9c3\") " pod="openshift-ingress/e2e-ipp-a-dc0f3f-openshift-default-d9f67c9bc-ks8t6" Nov 03 01:01:01.152053 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:01:01.146268 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6c13fe13-2eda-4d7c-bf31-c522d7f3b9c3-istio-podinfo\") pod \"e2e-ipp-a-dc0f3f-openshift-default-d9f67c9bc-ks8t6\" (UID: \"6c13fe13-2eda-4d7c-bf31-c522d7f3b9c3\") " pod="openshift-ingress/e2e-ipp-a-dc0f3f-openshift-default-d9f67c9bc-ks8t6" Nov 03 01:01:01.152053 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:01:01.146542 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6c13fe13-2eda-4d7c-bf31-c522d7f3b9c3-istiod-ca-cert\") pod \"e2e-ipp-a-dc0f3f-openshift-default-d9f67c9bc-ks8t6\" (UID: \"6c13fe13-2eda-4d7c-bf31-c522d7f3b9c3\") " pod="openshift-ingress/e2e-ipp-a-dc0f3f-openshift-default-d9f67c9bc-ks8t6" Nov 03 01:01:01.152053 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:01:01.146623 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6c13fe13-2eda-4d7c-bf31-c522d7f3b9c3-workload-socket\") pod \"e2e-ipp-a-dc0f3f-openshift-default-d9f67c9bc-ks8t6\" (UID: \"6c13fe13-2eda-4d7c-bf31-c522d7f3b9c3\") " pod="openshift-ingress/e2e-ipp-a-dc0f3f-openshift-default-d9f67c9bc-ks8t6" Nov 03 01:01:01.185025 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:01:01.174097 4939 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-f7680d-openshift-default-c4488d5c9-5nv8c"] Nov 03 01:01:01.185025 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:01:01.178803 4939 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-f7680d-openshift-default-c4488d5c9-5nv8c" podUID="38dfb6cb-588c-4ca9-916d-0b3fdc7910bb" containerName="istio-proxy" containerID="cri-o://a7bbc7d481c2d6a20649e5356e8ce1f69c26ed3fe3ca7cc1670745a892cbb3de" gracePeriod=30 Nov 03 01:01:01.276300 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:01:01.276257 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6c13fe13-2eda-4d7c-bf31-c522d7f3b9c3-istio-envoy\") pod \"e2e-ipp-a-dc0f3f-openshift-default-d9f67c9bc-ks8t6\" (UID: \"6c13fe13-2eda-4d7c-bf31-c522d7f3b9c3\") " pod="openshift-ingress/e2e-ipp-a-dc0f3f-openshift-default-d9f67c9bc-ks8t6" Nov 03 01:01:01.338944 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:01:01.337233 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6wxjj\" (UniqueName: \"kubernetes.io/projected/6c13fe13-2eda-4d7c-bf31-c522d7f3b9c3-kube-api-access-6wxjj\") pod \"e2e-ipp-a-dc0f3f-openshift-default-d9f67c9bc-ks8t6\" (UID: \"6c13fe13-2eda-4d7c-bf31-c522d7f3b9c3\") " pod="openshift-ingress/e2e-ipp-a-dc0f3f-openshift-default-d9f67c9bc-ks8t6" Nov 03 01:01:01.363973 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:01:01.360196 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6c13fe13-2eda-4d7c-bf31-c522d7f3b9c3-istio-token\") pod \"e2e-ipp-a-dc0f3f-openshift-default-d9f67c9bc-ks8t6\" (UID: \"6c13fe13-2eda-4d7c-bf31-c522d7f3b9c3\") " pod="openshift-ingress/e2e-ipp-a-dc0f3f-openshift-default-d9f67c9bc-ks8t6" Nov 03 01:01:01.549985 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:01:01.546279 4939 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-dc0f3f-openshift-default-d9f67c9bc-ks8t6" Nov 03 01:01:01.681805 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:01:01.681730 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:01:02.018664 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:01:02.018629 4939 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-dc0f3f-openshift-default-d9f67c9bc-ks8t6"] Nov 03 01:01:02.019107 ip-10-0-140-107 kubenswrapper[4939]: W1103 01:01:02.019081 4939 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6c13fe13_2eda_4d7c_bf31_c522d7f3b9c3.slice/crio-c2a67945e11fae41f902ac8e853f430a56915418619de74ba55cd0010a83475a WatchSource:0}: Error finding container c2a67945e11fae41f902ac8e853f430a56915418619de74ba55cd0010a83475a: Status 404 returned error can't find the container with id c2a67945e11fae41f902ac8e853f430a56915418619de74ba55cd0010a83475a Nov 03 01:01:02.021655 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:01:02.021617 4939 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 01:01:02.021728 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:01:02.021682 4939 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 01:01:02.021728 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:01:02.021711 4939 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 01:01:02.700809 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:01:02.700768 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:01:02.707075 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:01:02.707051 4939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-dc0f3f-openshift-default-d9f67c9bc-ks8t6" event={"ID":"6c13fe13-2eda-4d7c-bf31-c522d7f3b9c3","Type":"ContainerStarted","Data":"24198b03b942e91379154ffbed0ce369eb1b7bea6f0a27a5caadcb65fe09cefd"} Nov 03 01:01:02.707200 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:01:02.707096 4939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-dc0f3f-openshift-default-d9f67c9bc-ks8t6" event={"ID":"6c13fe13-2eda-4d7c-bf31-c522d7f3b9c3","Type":"ContainerStarted","Data":"c2a67945e11fae41f902ac8e853f430a56915418619de74ba55cd0010a83475a"} Nov 03 01:01:02.735080 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:01:02.735039 4939 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-a-dc0f3f-openshift-default-d9f67c9bc-ks8t6" podStartSLOduration=2.735027316 podStartE2EDuration="2.735027316s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 01:01:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 01:01:02.729543557 +0000 UTC m=+1210.993878504" watchObservedRunningTime="2026-11-03 01:01:02.735027316 +0000 UTC m=+1210.999362261" Nov 03 01:01:03.145968 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:01:03.145869 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:01:03.547057 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:01:03.546975 4939 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-a-dc0f3f-openshift-default-d9f67c9bc-ks8t6" Nov 03 01:01:03.552997 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:01:03.552969 4939 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-a-dc0f3f-openshift-default-d9f67c9bc-ks8t6" Nov 03 01:01:03.685088 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:01:03.685058 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:01:03.756488 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:01:03.756459 4939 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-a-dc0f3f-openshift-default-d9f67c9bc-ks8t6" Nov 03 01:01:03.772242 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:01:03.772204 4939 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-a-dc0f3f-openshift-default-d9f67c9bc-ks8t6" Nov 03 01:01:04.616958 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:01:04.613505 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:01:04.910023 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:01:04.908591 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:01:05.151277 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:01:05.151236 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:01:05.293553 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:01:05.293470 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:01:05.543063 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:01:05.543031 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:01:05.684396 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:01:05.684364 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:01:06.017980 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:01:06.013887 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:01:06.142166 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:01:06.142132 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:01:06.424641 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:01:06.424604 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:01:06.525764 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:01:06.525728 4939 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-dc0f3f-747fcc75fc-zvf8h"] Nov 03 01:01:06.530301 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:01:06.530279 4939 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-dc0f3f-5c746d84f9-4scbk"] Nov 03 01:01:06.530409 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:01:06.530389 4939 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-dc0f3f-747fcc75fc-zvf8h" Nov 03 01:01:06.534511 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:01:06.534486 4939 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-dc0f3f-5c746d84f9-4scbk" Nov 03 01:01:06.541018 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:01:06.540842 4939 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-dc0f3f-747fcc75fc-zvf8h"] Nov 03 01:01:06.541018 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:01:06.540988 4939 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-a-dc0f3f\"" Nov 03 01:01:06.544691 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:01:06.544630 4939 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-dc0f3f-5c746d84f9-4scbk"] Nov 03 01:01:06.564741 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:01:06.564717 4939 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-a-dc0f3f-dockercfg-lbrdg\"" Nov 03 01:01:06.628578 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:01:06.628489 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:01:06.732005 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:01:06.724299 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ffa6f2b0-be86-4f99-8d96-e4835139d9cd-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-dc0f3f-5c746d84f9-4scbk\" (UID: \"ffa6f2b0-be86-4f99-8d96-e4835139d9cd\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-dc0f3f-5c746d84f9-4scbk" Nov 03 01:01:06.755934 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:01:06.732298 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8vl5q\" (UniqueName: \"kubernetes.io/projected/9fc6f374-4e0d-4a76-aa72-056b80fee767-kube-api-access-8vl5q\") pod \"payload-processing-e2e-ipp-a-dc0f3f-747fcc75fc-zvf8h\" (UID: \"9fc6f374-4e0d-4a76-aa72-056b80fee767\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-dc0f3f-747fcc75fc-zvf8h" Nov 03 01:01:06.756329 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:01:06.756291 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b7bp4\" (UniqueName: \"kubernetes.io/projected/ffa6f2b0-be86-4f99-8d96-e4835139d9cd-kube-api-access-b7bp4\") pod \"payload-pre-processing-e2e-ipp-a-dc0f3f-5c746d84f9-4scbk\" (UID: \"ffa6f2b0-be86-4f99-8d96-e4835139d9cd\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-dc0f3f-5c746d84f9-4scbk" Nov 03 01:01:06.756586 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:01:06.756560 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9fc6f374-4e0d-4a76-aa72-056b80fee767-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-dc0f3f-747fcc75fc-zvf8h\" (UID: \"9fc6f374-4e0d-4a76-aa72-056b80fee767\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-dc0f3f-747fcc75fc-zvf8h" Nov 03 01:01:06.860992 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:01:06.858591 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b7bp4\" (UniqueName: \"kubernetes.io/projected/ffa6f2b0-be86-4f99-8d96-e4835139d9cd-kube-api-access-b7bp4\") pod \"payload-pre-processing-e2e-ipp-a-dc0f3f-5c746d84f9-4scbk\" (UID: \"ffa6f2b0-be86-4f99-8d96-e4835139d9cd\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-dc0f3f-5c746d84f9-4scbk" Nov 03 01:01:06.860992 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:01:06.858692 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9fc6f374-4e0d-4a76-aa72-056b80fee767-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-dc0f3f-747fcc75fc-zvf8h\" (UID: \"9fc6f374-4e0d-4a76-aa72-056b80fee767\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-dc0f3f-747fcc75fc-zvf8h" Nov 03 01:01:06.860992 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:01:06.858950 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ffa6f2b0-be86-4f99-8d96-e4835139d9cd-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-dc0f3f-5c746d84f9-4scbk\" (UID: \"ffa6f2b0-be86-4f99-8d96-e4835139d9cd\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-dc0f3f-5c746d84f9-4scbk" Nov 03 01:01:06.860992 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:01:06.859027 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8vl5q\" (UniqueName: \"kubernetes.io/projected/9fc6f374-4e0d-4a76-aa72-056b80fee767-kube-api-access-8vl5q\") pod \"payload-processing-e2e-ipp-a-dc0f3f-747fcc75fc-zvf8h\" (UID: \"9fc6f374-4e0d-4a76-aa72-056b80fee767\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-dc0f3f-747fcc75fc-zvf8h" Nov 03 01:01:06.860992 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:01:06.860366 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9fc6f374-4e0d-4a76-aa72-056b80fee767-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-dc0f3f-747fcc75fc-zvf8h\" (UID: \"9fc6f374-4e0d-4a76-aa72-056b80fee767\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-dc0f3f-747fcc75fc-zvf8h" Nov 03 01:01:06.861739 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:01:06.861657 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ffa6f2b0-be86-4f99-8d96-e4835139d9cd-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-dc0f3f-5c746d84f9-4scbk\" (UID: \"ffa6f2b0-be86-4f99-8d96-e4835139d9cd\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-dc0f3f-5c746d84f9-4scbk" Nov 03 01:01:06.891837 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:01:06.890565 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b7bp4\" (UniqueName: \"kubernetes.io/projected/ffa6f2b0-be86-4f99-8d96-e4835139d9cd-kube-api-access-b7bp4\") pod \"payload-pre-processing-e2e-ipp-a-dc0f3f-5c746d84f9-4scbk\" (UID: \"ffa6f2b0-be86-4f99-8d96-e4835139d9cd\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-dc0f3f-5c746d84f9-4scbk" Nov 03 01:01:06.891837 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:01:06.890729 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8vl5q\" (UniqueName: \"kubernetes.io/projected/9fc6f374-4e0d-4a76-aa72-056b80fee767-kube-api-access-8vl5q\") pod \"payload-processing-e2e-ipp-a-dc0f3f-747fcc75fc-zvf8h\" (UID: \"9fc6f374-4e0d-4a76-aa72-056b80fee767\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-dc0f3f-747fcc75fc-zvf8h" Nov 03 01:01:06.913989 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:01:06.913492 4939 generic.go:356] "Generic (PLEG): container finished" podID="38dfb6cb-588c-4ca9-916d-0b3fdc7910bb" containerID="a7bbc7d481c2d6a20649e5356e8ce1f69c26ed3fe3ca7cc1670745a892cbb3de" exitCode=0 Nov 03 01:01:06.913989 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:01:06.913618 4939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-f7680d-openshift-default-c4488d5c9-5nv8c" event={"ID":"38dfb6cb-588c-4ca9-916d-0b3fdc7910bb","Type":"ContainerDied","Data":"a7bbc7d481c2d6a20649e5356e8ce1f69c26ed3fe3ca7cc1670745a892cbb3de"} Nov 03 01:01:07.003566 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:01:07.003517 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:01:07.135170 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:01:07.135116 4939 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-f7680d-openshift-default-c4488d5c9-5nv8c" Nov 03 01:01:07.155250 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:01:07.155224 4939 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-dc0f3f-747fcc75fc-zvf8h" Nov 03 01:01:07.167008 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:01:07.166984 4939 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-dc0f3f-5c746d84f9-4scbk" Nov 03 01:01:07.279054 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:01:07.275115 4939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/38dfb6cb-588c-4ca9-916d-0b3fdc7910bb-kube-api-access-75rh4\" (UniqueName: \"kubernetes.io/projected/38dfb6cb-588c-4ca9-916d-0b3fdc7910bb-kube-api-access-75rh4\") pod \"38dfb6cb-588c-4ca9-916d-0b3fdc7910bb\" (UID: \"38dfb6cb-588c-4ca9-916d-0b3fdc7910bb\") " Nov 03 01:01:07.279054 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:01:07.275259 4939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/38dfb6cb-588c-4ca9-916d-0b3fdc7910bb-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/38dfb6cb-588c-4ca9-916d-0b3fdc7910bb-istiod-ca-cert\") pod \"38dfb6cb-588c-4ca9-916d-0b3fdc7910bb\" (UID: \"38dfb6cb-588c-4ca9-916d-0b3fdc7910bb\") " Nov 03 01:01:07.279054 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:01:07.275416 4939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/38dfb6cb-588c-4ca9-916d-0b3fdc7910bb-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/38dfb6cb-588c-4ca9-916d-0b3fdc7910bb-istio-data\") pod \"38dfb6cb-588c-4ca9-916d-0b3fdc7910bb\" (UID: \"38dfb6cb-588c-4ca9-916d-0b3fdc7910bb\") " Nov 03 01:01:07.279054 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:01:07.275448 4939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/38dfb6cb-588c-4ca9-916d-0b3fdc7910bb-istio-token\" (UniqueName: \"kubernetes.io/projected/38dfb6cb-588c-4ca9-916d-0b3fdc7910bb-istio-token\") pod \"38dfb6cb-588c-4ca9-916d-0b3fdc7910bb\" (UID: \"38dfb6cb-588c-4ca9-916d-0b3fdc7910bb\") " Nov 03 01:01:07.279054 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:01:07.275524 4939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/38dfb6cb-588c-4ca9-916d-0b3fdc7910bb-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/38dfb6cb-588c-4ca9-916d-0b3fdc7910bb-credential-socket\") pod \"38dfb6cb-588c-4ca9-916d-0b3fdc7910bb\" (UID: \"38dfb6cb-588c-4ca9-916d-0b3fdc7910bb\") " Nov 03 01:01:07.279054 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:01:07.275562 4939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/38dfb6cb-588c-4ca9-916d-0b3fdc7910bb-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/38dfb6cb-588c-4ca9-916d-0b3fdc7910bb-istio-envoy\") pod \"38dfb6cb-588c-4ca9-916d-0b3fdc7910bb\" (UID: \"38dfb6cb-588c-4ca9-916d-0b3fdc7910bb\") " Nov 03 01:01:07.279054 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:01:07.275632 4939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/38dfb6cb-588c-4ca9-916d-0b3fdc7910bb-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/38dfb6cb-588c-4ca9-916d-0b3fdc7910bb-workload-socket\") pod \"38dfb6cb-588c-4ca9-916d-0b3fdc7910bb\" (UID: \"38dfb6cb-588c-4ca9-916d-0b3fdc7910bb\") " Nov 03 01:01:07.279054 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:01:07.275755 4939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/38dfb6cb-588c-4ca9-916d-0b3fdc7910bb-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/38dfb6cb-588c-4ca9-916d-0b3fdc7910bb-workload-certs\") pod \"38dfb6cb-588c-4ca9-916d-0b3fdc7910bb\" (UID: \"38dfb6cb-588c-4ca9-916d-0b3fdc7910bb\") " Nov 03 01:01:07.279054 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:01:07.275831 4939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/38dfb6cb-588c-4ca9-916d-0b3fdc7910bb-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/38dfb6cb-588c-4ca9-916d-0b3fdc7910bb-istio-podinfo\") pod \"38dfb6cb-588c-4ca9-916d-0b3fdc7910bb\" (UID: \"38dfb6cb-588c-4ca9-916d-0b3fdc7910bb\") " Nov 03 01:01:07.286477 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:01:07.286266 4939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/38dfb6cb-588c-4ca9-916d-0b3fdc7910bb-workload-certs" pod "38dfb6cb-588c-4ca9-916d-0b3fdc7910bb" (UID: "38dfb6cb-588c-4ca9-916d-0b3fdc7910bb"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 01:01:07.286477 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:01:07.286351 4939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/38dfb6cb-588c-4ca9-916d-0b3fdc7910bb-credential-socket" pod "38dfb6cb-588c-4ca9-916d-0b3fdc7910bb" (UID: "38dfb6cb-588c-4ca9-916d-0b3fdc7910bb"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 01:01:07.286477 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:01:07.286400 4939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/38dfb6cb-588c-4ca9-916d-0b3fdc7910bb-workload-socket" pod "38dfb6cb-588c-4ca9-916d-0b3fdc7910bb" (UID: "38dfb6cb-588c-4ca9-916d-0b3fdc7910bb"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 01:01:07.287529 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:01:07.287417 4939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/38dfb6cb-588c-4ca9-916d-0b3fdc7910bb-istiod-ca-cert" pod "38dfb6cb-588c-4ca9-916d-0b3fdc7910bb" (UID: "38dfb6cb-588c-4ca9-916d-0b3fdc7910bb"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 01:01:07.287529 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:01:07.287510 4939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/38dfb6cb-588c-4ca9-916d-0b3fdc7910bb-istio-data" pod "38dfb6cb-588c-4ca9-916d-0b3fdc7910bb" (UID: "38dfb6cb-588c-4ca9-916d-0b3fdc7910bb"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 01:01:07.288089 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:01:07.288057 4939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/38dfb6cb-588c-4ca9-916d-0b3fdc7910bb-istio-envoy" pod "38dfb6cb-588c-4ca9-916d-0b3fdc7910bb" (UID: "38dfb6cb-588c-4ca9-916d-0b3fdc7910bb"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 01:01:07.288522 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:01:07.288491 4939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/38dfb6cb-588c-4ca9-916d-0b3fdc7910bb-istio-token" pod "38dfb6cb-588c-4ca9-916d-0b3fdc7910bb" (UID: "38dfb6cb-588c-4ca9-916d-0b3fdc7910bb"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 01:01:07.291227 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:01:07.290420 4939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/38dfb6cb-588c-4ca9-916d-0b3fdc7910bb-kube-api-access-75rh4" pod "38dfb6cb-588c-4ca9-916d-0b3fdc7910bb" (UID: "38dfb6cb-588c-4ca9-916d-0b3fdc7910bb"). InnerVolumeSpecName "kube-api-access-75rh4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 01:01:07.295229 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:01:07.295204 4939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/38dfb6cb-588c-4ca9-916d-0b3fdc7910bb-istio-podinfo" pod "38dfb6cb-588c-4ca9-916d-0b3fdc7910bb" (UID: "38dfb6cb-588c-4ca9-916d-0b3fdc7910bb"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 01:01:07.378245 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:01:07.378171 4939 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/38dfb6cb-588c-4ca9-916d-0b3fdc7910bb-workload-socket\") on node \"ip-10-0-140-107.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:01:07.378245 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:01:07.378243 4939 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/38dfb6cb-588c-4ca9-916d-0b3fdc7910bb-workload-certs\") on node \"ip-10-0-140-107.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:01:07.378550 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:01:07.378263 4939 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/38dfb6cb-588c-4ca9-916d-0b3fdc7910bb-istio-podinfo\") on node \"ip-10-0-140-107.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:01:07.378550 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:01:07.378277 4939 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-75rh4\" (UniqueName: \"kubernetes.io/projected/38dfb6cb-588c-4ca9-916d-0b3fdc7910bb-kube-api-access-75rh4\") on node \"ip-10-0-140-107.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:01:07.378550 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:01:07.378296 4939 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/38dfb6cb-588c-4ca9-916d-0b3fdc7910bb-istiod-ca-cert\") on node \"ip-10-0-140-107.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:01:07.378550 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:01:07.378310 4939 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/38dfb6cb-588c-4ca9-916d-0b3fdc7910bb-istio-data\") on node \"ip-10-0-140-107.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:01:07.378550 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:01:07.378323 4939 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/38dfb6cb-588c-4ca9-916d-0b3fdc7910bb-istio-token\") on node \"ip-10-0-140-107.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:01:07.378550 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:01:07.378337 4939 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/38dfb6cb-588c-4ca9-916d-0b3fdc7910bb-credential-socket\") on node \"ip-10-0-140-107.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:01:07.378550 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:01:07.378351 4939 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/38dfb6cb-588c-4ca9-916d-0b3fdc7910bb-istio-envoy\") on node \"ip-10-0-140-107.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:01:07.389866 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:01:07.389783 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:01:07.457754 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:01:07.457718 4939 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-dc0f3f-747fcc75fc-zvf8h"] Nov 03 01:01:07.515156 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:01:07.515128 4939 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-dc0f3f-5c746d84f9-4scbk"] Nov 03 01:01:07.922254 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:01:07.922217 4939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-dc0f3f-5c746d84f9-4scbk" event={"ID":"ffa6f2b0-be86-4f99-8d96-e4835139d9cd","Type":"ContainerStarted","Data":"c29b71509ea2bdd39abc500025e44a79fc354f93c4d29c03d8b010f2277bb9f7"} Nov 03 01:01:07.922436 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:01:07.922277 4939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-dc0f3f-5c746d84f9-4scbk" event={"ID":"ffa6f2b0-be86-4f99-8d96-e4835139d9cd","Type":"ContainerStarted","Data":"d7d29def85899c6b23a87d48317533aefcc9025a33509c1b7913c0a0b25fa17a"} Nov 03 01:01:07.922436 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:01:07.922305 4939 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-dc0f3f-5c746d84f9-4scbk" Nov 03 01:01:07.924075 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:01:07.924049 4939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-dc0f3f-747fcc75fc-zvf8h" event={"ID":"9fc6f374-4e0d-4a76-aa72-056b80fee767","Type":"ContainerStarted","Data":"b03b86b8943b71df62793cecd7dbfc46a1b0789e1d5c55930c3041a0c2917968"} Nov 03 01:01:07.924204 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:01:07.924096 4939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-dc0f3f-747fcc75fc-zvf8h" event={"ID":"9fc6f374-4e0d-4a76-aa72-056b80fee767","Type":"ContainerStarted","Data":"c977d94751cfa99cc7c99cef9e67b06e47d3aa9b2f43333248ffffa407066883"} Nov 03 01:01:07.924204 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:01:07.924126 4939 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-dc0f3f-747fcc75fc-zvf8h" Nov 03 01:01:07.925500 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:01:07.925482 4939 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-f7680d-openshift-default-c4488d5c9-5nv8c" Nov 03 01:01:07.925615 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:01:07.925478 4939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-f7680d-openshift-default-c4488d5c9-5nv8c" event={"ID":"38dfb6cb-588c-4ca9-916d-0b3fdc7910bb","Type":"ContainerDied","Data":"62c74d342840303f44f0a6db2e3022c6017252e17c173168ea4c6473cc3a24c1"} Nov 03 01:01:07.925615 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:01:07.925586 4939 scope.go:122] "RemoveContainer" containerID="a7bbc7d481c2d6a20649e5356e8ce1f69c26ed3fe3ca7cc1670745a892cbb3de" Nov 03 01:01:07.958699 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:01:07.958643 4939 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-dc0f3f-5c746d84f9-4scbk" podStartSLOduration=1.958625709 podStartE2EDuration="1.958625709s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 01:01:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 01:01:07.956271539 +0000 UTC m=+1216.220606484" watchObservedRunningTime="2026-11-03 01:01:07.958625709 +0000 UTC m=+1216.222960654" Nov 03 01:01:07.986776 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:01:07.986657 4939 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-a-dc0f3f-747fcc75fc-zvf8h" podStartSLOduration=1.9866380289999999 podStartE2EDuration="1.986638029s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 01:01:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 01:01:07.980268894 +0000 UTC m=+1216.244603838" watchObservedRunningTime="2026-11-03 01:01:07.986638029 +0000 UTC m=+1216.250972978" Nov 03 01:01:08.000434 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:01:08.000397 4939 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-f7680d-openshift-default-c4488d5c9-5nv8c"] Nov 03 01:01:08.006223 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:01:08.006197 4939 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-f7680d-openshift-default-c4488d5c9-5nv8c"] Nov 03 01:01:08.422150 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:01:08.422118 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:01:08.940163 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:01:08.940128 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:01:09.070879 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:01:09.070840 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:01:09.283224 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:01:09.283145 4939 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-02e4c0-7796479459-fjkx9"] Nov 03 01:01:09.307929 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:01:09.301112 4939 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-02e4c0-57ff55f5cf-72j67"] Nov 03 01:01:09.388376 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:01:09.388344 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:01:09.517430 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:01:09.517397 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:01:09.638408 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:01:09.638375 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:01:09.765517 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:01:09.765476 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:01:10.133708 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:01:10.133658 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:01:10.161157 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:01:10.161119 4939 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="38dfb6cb-588c-4ca9-916d-0b3fdc7910bb" path="/var/lib/kubelet/pods/38dfb6cb-588c-4ca9-916d-0b3fdc7910bb/volumes" Nov 03 01:01:10.161863 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:01:10.161840 4939 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-def754-openshift-default-544466b787-9msx9"] Nov 03 01:01:10.166930 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:01:10.162244 4939 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-b-def754-openshift-default-544466b787-9msx9" podUID="8c763d19-e7ff-4638-8d21-3e977502a520" containerName="istio-proxy" containerID="cri-o://0580fbf79b774e762d78d096dda18b1f06f7743894ab28892d60afde59dda780" gracePeriod=30 Nov 03 01:01:10.304028 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:01:10.303994 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:01:10.853536 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:01:10.834801 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:01:11.331151 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:01:11.331123 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:01:11.363087 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:01:11.363052 4939 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-208a0d-f9cc48655-wfc6q"] Nov 03 01:01:11.363806 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:01:11.363786 4939 cpu_manager.go:414] "containerMap: removing container" podUID="38dfb6cb-588c-4ca9-916d-0b3fdc7910bb" containerName="istio-proxy" Nov 03 01:01:11.363926 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:01:11.363807 4939 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="38dfb6cb-588c-4ca9-916d-0b3fdc7910bb" containerName="istio-proxy" Nov 03 01:01:11.364022 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:01:11.364009 4939 memory_manager.go:365] "RemoveStaleState removing state" podUID="38dfb6cb-588c-4ca9-916d-0b3fdc7910bb" containerName="istio-proxy" Nov 03 01:01:11.368033 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:01:11.368011 4939 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-208a0d-f9cc48655-wfc6q" Nov 03 01:01:11.373503 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:01:11.373479 4939 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-208a0d-6f7c665d67-pc68m"] Nov 03 01:01:11.376331 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:01:11.376303 4939 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-praxis-b-208a0d\"" Nov 03 01:01:11.376812 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:01:11.376793 4939 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-praxis-b-208a0d-dockercfg-mqbj7\"" Nov 03 01:01:11.379076 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:01:11.379054 4939 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-b-208a0d-6f7c665d67-pc68m" Nov 03 01:01:11.380955 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:01:11.380936 4939 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-208a0d-f9cc48655-wfc6q"] Nov 03 01:01:11.397019 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:01:11.396987 4939 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-208a0d-6f7c665d67-pc68m"] Nov 03 01:01:11.503483 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:01:11.503446 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:01:11.577937 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:01:11.577498 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/19d3d3db-71fc-41f2-9976-fa8bcd10bfec-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-b-208a0d-f9cc48655-wfc6q\" (UID: \"19d3d3db-71fc-41f2-9976-fa8bcd10bfec\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-208a0d-f9cc48655-wfc6q" Nov 03 01:01:11.577937 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:01:11.577642 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2626m\" (UniqueName: \"kubernetes.io/projected/19d3d3db-71fc-41f2-9976-fa8bcd10bfec-kube-api-access-2626m\") pod \"payload-pre-processing-e2e-praxis-b-208a0d-f9cc48655-wfc6q\" (UID: \"19d3d3db-71fc-41f2-9976-fa8bcd10bfec\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-208a0d-f9cc48655-wfc6q" Nov 03 01:01:11.577937 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:01:11.577773 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e1d5b015-e51e-4f86-ac94-34b05d8db2d3-plugins-config-volume\") pod \"payload-processing-e2e-praxis-b-208a0d-6f7c665d67-pc68m\" (UID: \"e1d5b015-e51e-4f86-ac94-34b05d8db2d3\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-208a0d-6f7c665d67-pc68m" Nov 03 01:01:11.577937 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:01:11.577844 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2784x\" (UniqueName: \"kubernetes.io/projected/e1d5b015-e51e-4f86-ac94-34b05d8db2d3-kube-api-access-2784x\") pod \"payload-processing-e2e-praxis-b-208a0d-6f7c665d67-pc68m\" (UID: \"e1d5b015-e51e-4f86-ac94-34b05d8db2d3\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-208a0d-6f7c665d67-pc68m" Nov 03 01:01:11.683692 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:01:11.683597 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2626m\" (UniqueName: \"kubernetes.io/projected/19d3d3db-71fc-41f2-9976-fa8bcd10bfec-kube-api-access-2626m\") pod \"payload-pre-processing-e2e-praxis-b-208a0d-f9cc48655-wfc6q\" (UID: \"19d3d3db-71fc-41f2-9976-fa8bcd10bfec\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-208a0d-f9cc48655-wfc6q" Nov 03 01:01:11.691196 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:01:11.691158 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e1d5b015-e51e-4f86-ac94-34b05d8db2d3-plugins-config-volume\") pod \"payload-processing-e2e-praxis-b-208a0d-6f7c665d67-pc68m\" (UID: \"e1d5b015-e51e-4f86-ac94-34b05d8db2d3\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-208a0d-6f7c665d67-pc68m" Nov 03 01:01:11.691545 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:01:11.691494 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2784x\" (UniqueName: \"kubernetes.io/projected/e1d5b015-e51e-4f86-ac94-34b05d8db2d3-kube-api-access-2784x\") pod \"payload-processing-e2e-praxis-b-208a0d-6f7c665d67-pc68m\" (UID: \"e1d5b015-e51e-4f86-ac94-34b05d8db2d3\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-208a0d-6f7c665d67-pc68m" Nov 03 01:01:11.691847 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:01:11.691822 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/19d3d3db-71fc-41f2-9976-fa8bcd10bfec-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-b-208a0d-f9cc48655-wfc6q\" (UID: \"19d3d3db-71fc-41f2-9976-fa8bcd10bfec\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-208a0d-f9cc48655-wfc6q" Nov 03 01:01:11.693630 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:01:11.693571 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/19d3d3db-71fc-41f2-9976-fa8bcd10bfec-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-b-208a0d-f9cc48655-wfc6q\" (UID: \"19d3d3db-71fc-41f2-9976-fa8bcd10bfec\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-208a0d-f9cc48655-wfc6q" Nov 03 01:01:11.694296 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:01:11.694210 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e1d5b015-e51e-4f86-ac94-34b05d8db2d3-plugins-config-volume\") pod \"payload-processing-e2e-praxis-b-208a0d-6f7c665d67-pc68m\" (UID: \"e1d5b015-e51e-4f86-ac94-34b05d8db2d3\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-208a0d-6f7c665d67-pc68m" Nov 03 01:01:11.721259 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:01:11.717001 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2784x\" (UniqueName: \"kubernetes.io/projected/e1d5b015-e51e-4f86-ac94-34b05d8db2d3-kube-api-access-2784x\") pod \"payload-processing-e2e-praxis-b-208a0d-6f7c665d67-pc68m\" (UID: \"e1d5b015-e51e-4f86-ac94-34b05d8db2d3\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-208a0d-6f7c665d67-pc68m" Nov 03 01:01:11.721259 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:01:11.717006 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2626m\" (UniqueName: \"kubernetes.io/projected/19d3d3db-71fc-41f2-9976-fa8bcd10bfec-kube-api-access-2626m\") pod \"payload-pre-processing-e2e-praxis-b-208a0d-f9cc48655-wfc6q\" (UID: \"19d3d3db-71fc-41f2-9976-fa8bcd10bfec\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-208a0d-f9cc48655-wfc6q" Nov 03 01:01:11.871050 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:01:11.871018 4939 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-praxis-a-d5f78f-openshift-default-64f48987d9-rk5hm"] Nov 03 01:01:11.875809 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:01:11.875776 4939 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-a-d5f78f-openshift-default-64f48987d9-rk5hm" Nov 03 01:01:11.887982 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:01:11.887958 4939 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-praxis-a-d5f78f-openshift-default-dockercfg-zdrqv\"" Nov 03 01:01:11.894590 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:01:11.894561 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dfea3192-3541-49c5-a491-ebf7308b74e0-istio-podinfo\") pod \"e2e-praxis-a-d5f78f-openshift-default-64f48987d9-rk5hm\" (UID: \"dfea3192-3541-49c5-a491-ebf7308b74e0\") " pod="openshift-ingress/e2e-praxis-a-d5f78f-openshift-default-64f48987d9-rk5hm" Nov 03 01:01:11.894791 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:01:11.894767 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dfea3192-3541-49c5-a491-ebf7308b74e0-istio-envoy\") pod \"e2e-praxis-a-d5f78f-openshift-default-64f48987d9-rk5hm\" (UID: \"dfea3192-3541-49c5-a491-ebf7308b74e0\") " pod="openshift-ingress/e2e-praxis-a-d5f78f-openshift-default-64f48987d9-rk5hm" Nov 03 01:01:11.894964 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:01:11.894943 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dfea3192-3541-49c5-a491-ebf7308b74e0-credential-socket\") pod \"e2e-praxis-a-d5f78f-openshift-default-64f48987d9-rk5hm\" (UID: \"dfea3192-3541-49c5-a491-ebf7308b74e0\") " pod="openshift-ingress/e2e-praxis-a-d5f78f-openshift-default-64f48987d9-rk5hm" Nov 03 01:01:11.895035 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:01:11.894990 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bgchl\" (UniqueName: \"kubernetes.io/projected/dfea3192-3541-49c5-a491-ebf7308b74e0-kube-api-access-bgchl\") pod \"e2e-praxis-a-d5f78f-openshift-default-64f48987d9-rk5hm\" (UID: \"dfea3192-3541-49c5-a491-ebf7308b74e0\") " pod="openshift-ingress/e2e-praxis-a-d5f78f-openshift-default-64f48987d9-rk5hm" Nov 03 01:01:11.895109 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:01:11.895078 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dfea3192-3541-49c5-a491-ebf7308b74e0-istiod-ca-cert\") pod \"e2e-praxis-a-d5f78f-openshift-default-64f48987d9-rk5hm\" (UID: \"dfea3192-3541-49c5-a491-ebf7308b74e0\") " pod="openshift-ingress/e2e-praxis-a-d5f78f-openshift-default-64f48987d9-rk5hm" Nov 03 01:01:11.895169 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:01:11.895156 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dfea3192-3541-49c5-a491-ebf7308b74e0-workload-certs\") pod \"e2e-praxis-a-d5f78f-openshift-default-64f48987d9-rk5hm\" (UID: \"dfea3192-3541-49c5-a491-ebf7308b74e0\") " pod="openshift-ingress/e2e-praxis-a-d5f78f-openshift-default-64f48987d9-rk5hm" Nov 03 01:01:11.895222 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:01:11.895206 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dfea3192-3541-49c5-a491-ebf7308b74e0-workload-socket\") pod \"e2e-praxis-a-d5f78f-openshift-default-64f48987d9-rk5hm\" (UID: \"dfea3192-3541-49c5-a491-ebf7308b74e0\") " pod="openshift-ingress/e2e-praxis-a-d5f78f-openshift-default-64f48987d9-rk5hm" Nov 03 01:01:11.895380 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:01:11.895365 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dfea3192-3541-49c5-a491-ebf7308b74e0-istio-token\") pod \"e2e-praxis-a-d5f78f-openshift-default-64f48987d9-rk5hm\" (UID: \"dfea3192-3541-49c5-a491-ebf7308b74e0\") " pod="openshift-ingress/e2e-praxis-a-d5f78f-openshift-default-64f48987d9-rk5hm" Nov 03 01:01:11.895442 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:01:11.895388 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dfea3192-3541-49c5-a491-ebf7308b74e0-istio-data\") pod \"e2e-praxis-a-d5f78f-openshift-default-64f48987d9-rk5hm\" (UID: \"dfea3192-3541-49c5-a491-ebf7308b74e0\") " pod="openshift-ingress/e2e-praxis-a-d5f78f-openshift-default-64f48987d9-rk5hm" Nov 03 01:01:11.908016 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:01:11.907991 4939 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-praxis-a-d5f78f-openshift-default-64f48987d9-rk5hm"] Nov 03 01:01:11.953055 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:01:11.952947 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:01:12.024934 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:01:12.004367 4939 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-208a0d-f9cc48655-wfc6q" Nov 03 01:01:12.024934 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:01:12.006049 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dfea3192-3541-49c5-a491-ebf7308b74e0-credential-socket\") pod \"e2e-praxis-a-d5f78f-openshift-default-64f48987d9-rk5hm\" (UID: \"dfea3192-3541-49c5-a491-ebf7308b74e0\") " pod="openshift-ingress/e2e-praxis-a-d5f78f-openshift-default-64f48987d9-rk5hm" Nov 03 01:01:12.024934 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:01:12.006103 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bgchl\" (UniqueName: \"kubernetes.io/projected/dfea3192-3541-49c5-a491-ebf7308b74e0-kube-api-access-bgchl\") pod \"e2e-praxis-a-d5f78f-openshift-default-64f48987d9-rk5hm\" (UID: \"dfea3192-3541-49c5-a491-ebf7308b74e0\") " pod="openshift-ingress/e2e-praxis-a-d5f78f-openshift-default-64f48987d9-rk5hm" Nov 03 01:01:12.024934 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:01:12.006157 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dfea3192-3541-49c5-a491-ebf7308b74e0-istiod-ca-cert\") pod \"e2e-praxis-a-d5f78f-openshift-default-64f48987d9-rk5hm\" (UID: \"dfea3192-3541-49c5-a491-ebf7308b74e0\") " pod="openshift-ingress/e2e-praxis-a-d5f78f-openshift-default-64f48987d9-rk5hm" Nov 03 01:01:12.024934 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:01:12.006227 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dfea3192-3541-49c5-a491-ebf7308b74e0-workload-certs\") pod \"e2e-praxis-a-d5f78f-openshift-default-64f48987d9-rk5hm\" (UID: \"dfea3192-3541-49c5-a491-ebf7308b74e0\") " pod="openshift-ingress/e2e-praxis-a-d5f78f-openshift-default-64f48987d9-rk5hm" Nov 03 01:01:12.024934 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:01:12.006262 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dfea3192-3541-49c5-a491-ebf7308b74e0-workload-socket\") pod \"e2e-praxis-a-d5f78f-openshift-default-64f48987d9-rk5hm\" (UID: \"dfea3192-3541-49c5-a491-ebf7308b74e0\") " pod="openshift-ingress/e2e-praxis-a-d5f78f-openshift-default-64f48987d9-rk5hm" Nov 03 01:01:12.024934 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:01:12.006308 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dfea3192-3541-49c5-a491-ebf7308b74e0-istio-token\") pod \"e2e-praxis-a-d5f78f-openshift-default-64f48987d9-rk5hm\" (UID: \"dfea3192-3541-49c5-a491-ebf7308b74e0\") " pod="openshift-ingress/e2e-praxis-a-d5f78f-openshift-default-64f48987d9-rk5hm" Nov 03 01:01:12.024934 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:01:12.006335 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dfea3192-3541-49c5-a491-ebf7308b74e0-istio-data\") pod \"e2e-praxis-a-d5f78f-openshift-default-64f48987d9-rk5hm\" (UID: \"dfea3192-3541-49c5-a491-ebf7308b74e0\") " pod="openshift-ingress/e2e-praxis-a-d5f78f-openshift-default-64f48987d9-rk5hm" Nov 03 01:01:12.024934 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:01:12.006417 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dfea3192-3541-49c5-a491-ebf7308b74e0-istio-podinfo\") pod \"e2e-praxis-a-d5f78f-openshift-default-64f48987d9-rk5hm\" (UID: \"dfea3192-3541-49c5-a491-ebf7308b74e0\") " pod="openshift-ingress/e2e-praxis-a-d5f78f-openshift-default-64f48987d9-rk5hm" Nov 03 01:01:12.024934 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:01:12.006553 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dfea3192-3541-49c5-a491-ebf7308b74e0-istio-envoy\") pod \"e2e-praxis-a-d5f78f-openshift-default-64f48987d9-rk5hm\" (UID: \"dfea3192-3541-49c5-a491-ebf7308b74e0\") " pod="openshift-ingress/e2e-praxis-a-d5f78f-openshift-default-64f48987d9-rk5hm" Nov 03 01:01:12.024934 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:01:12.008344 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dfea3192-3541-49c5-a491-ebf7308b74e0-workload-certs\") pod \"e2e-praxis-a-d5f78f-openshift-default-64f48987d9-rk5hm\" (UID: \"dfea3192-3541-49c5-a491-ebf7308b74e0\") " pod="openshift-ingress/e2e-praxis-a-d5f78f-openshift-default-64f48987d9-rk5hm" Nov 03 01:01:12.024934 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:01:12.008708 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dfea3192-3541-49c5-a491-ebf7308b74e0-credential-socket\") pod \"e2e-praxis-a-d5f78f-openshift-default-64f48987d9-rk5hm\" (UID: \"dfea3192-3541-49c5-a491-ebf7308b74e0\") " pod="openshift-ingress/e2e-praxis-a-d5f78f-openshift-default-64f48987d9-rk5hm" Nov 03 01:01:12.024934 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:01:12.009621 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dfea3192-3541-49c5-a491-ebf7308b74e0-istiod-ca-cert\") pod \"e2e-praxis-a-d5f78f-openshift-default-64f48987d9-rk5hm\" (UID: \"dfea3192-3541-49c5-a491-ebf7308b74e0\") " pod="openshift-ingress/e2e-praxis-a-d5f78f-openshift-default-64f48987d9-rk5hm" Nov 03 01:01:12.024934 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:01:12.010120 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dfea3192-3541-49c5-a491-ebf7308b74e0-workload-socket\") pod \"e2e-praxis-a-d5f78f-openshift-default-64f48987d9-rk5hm\" (UID: \"dfea3192-3541-49c5-a491-ebf7308b74e0\") " pod="openshift-ingress/e2e-praxis-a-d5f78f-openshift-default-64f48987d9-rk5hm" Nov 03 01:01:12.024934 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:01:12.014137 4939 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-b-208a0d-6f7c665d67-pc68m" Nov 03 01:01:12.084934 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:01:12.084146 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dfea3192-3541-49c5-a491-ebf7308b74e0-istio-data\") pod \"e2e-praxis-a-d5f78f-openshift-default-64f48987d9-rk5hm\" (UID: \"dfea3192-3541-49c5-a491-ebf7308b74e0\") " pod="openshift-ingress/e2e-praxis-a-d5f78f-openshift-default-64f48987d9-rk5hm" Nov 03 01:01:12.085577 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:01:12.085552 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dfea3192-3541-49c5-a491-ebf7308b74e0-istio-envoy\") pod \"e2e-praxis-a-d5f78f-openshift-default-64f48987d9-rk5hm\" (UID: \"dfea3192-3541-49c5-a491-ebf7308b74e0\") " pod="openshift-ingress/e2e-praxis-a-d5f78f-openshift-default-64f48987d9-rk5hm" Nov 03 01:01:12.097839 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:01:12.097761 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dfea3192-3541-49c5-a491-ebf7308b74e0-istio-podinfo\") pod \"e2e-praxis-a-d5f78f-openshift-default-64f48987d9-rk5hm\" (UID: \"dfea3192-3541-49c5-a491-ebf7308b74e0\") " pod="openshift-ingress/e2e-praxis-a-d5f78f-openshift-default-64f48987d9-rk5hm" Nov 03 01:01:12.204584 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:01:12.204451 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:01:12.213796 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:01:12.213762 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dfea3192-3541-49c5-a491-ebf7308b74e0-istio-token\") pod \"e2e-praxis-a-d5f78f-openshift-default-64f48987d9-rk5hm\" (UID: \"dfea3192-3541-49c5-a491-ebf7308b74e0\") " pod="openshift-ingress/e2e-praxis-a-d5f78f-openshift-default-64f48987d9-rk5hm" Nov 03 01:01:12.215798 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:01:12.215764 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bgchl\" (UniqueName: \"kubernetes.io/projected/dfea3192-3541-49c5-a491-ebf7308b74e0-kube-api-access-bgchl\") pod \"e2e-praxis-a-d5f78f-openshift-default-64f48987d9-rk5hm\" (UID: \"dfea3192-3541-49c5-a491-ebf7308b74e0\") " pod="openshift-ingress/e2e-praxis-a-d5f78f-openshift-default-64f48987d9-rk5hm" Nov 03 01:01:12.345965 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:01:12.345324 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:01:12.492086 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:01:12.492009 4939 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-a-d5f78f-openshift-default-64f48987d9-rk5hm" Nov 03 01:01:12.594139 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:01:12.594105 4939 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-208a0d-f9cc48655-wfc6q"] Nov 03 01:01:12.595046 ip-10-0-140-107 kubenswrapper[4939]: W1103 01:01:12.595013 4939 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod19d3d3db_71fc_41f2_9976_fa8bcd10bfec.slice/crio-117451719c7dd4f9dc063a2c959d57244c6357b967191382d3e9857ce56e319c WatchSource:0}: Error finding container 117451719c7dd4f9dc063a2c959d57244c6357b967191382d3e9857ce56e319c: Status 404 returned error can't find the container with id 117451719c7dd4f9dc063a2c959d57244c6357b967191382d3e9857ce56e319c Nov 03 01:01:12.621000 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:01:12.620969 4939 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-208a0d-6f7c665d67-pc68m"] Nov 03 01:01:12.621407 ip-10-0-140-107 kubenswrapper[4939]: W1103 01:01:12.621374 4939 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode1d5b015_e51e_4f86_ac94_34b05d8db2d3.slice/crio-d060e6ec87a45a8b3e8b62b4e645ca0bd618b90b84213b9ab12cb587f4d162c6 WatchSource:0}: Error finding container d060e6ec87a45a8b3e8b62b4e645ca0bd618b90b84213b9ab12cb587f4d162c6: Status 404 returned error can't find the container with id d060e6ec87a45a8b3e8b62b4e645ca0bd618b90b84213b9ab12cb587f4d162c6 Nov 03 01:01:12.776881 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:01:12.776852 4939 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-praxis-a-d5f78f-openshift-default-64f48987d9-rk5hm"] Nov 03 01:01:12.777419 ip-10-0-140-107 kubenswrapper[4939]: W1103 01:01:12.777393 4939 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddfea3192_3541_49c5_a491_ebf7308b74e0.slice/crio-0b0eb9c283a2814232329a608d09e0e4e182142602704add5ffff7f476d5d46d WatchSource:0}: Error finding container 0b0eb9c283a2814232329a608d09e0e4e182142602704add5ffff7f476d5d46d: Status 404 returned error can't find the container with id 0b0eb9c283a2814232329a608d09e0e4e182142602704add5ffff7f476d5d46d Nov 03 01:01:12.780635 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:01:12.780597 4939 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 01:01:12.780709 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:01:12.780691 4939 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 01:01:12.780765 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:01:12.780738 4939 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 01:01:12.789232 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:01:12.789211 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:01:13.224055 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:01:13.223374 4939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-b-208a0d-6f7c665d67-pc68m" event={"ID":"e1d5b015-e51e-4f86-ac94-34b05d8db2d3","Type":"ContainerStarted","Data":"294acea636bb9e10e64a9e3ebf26fe4257515f5bc8a787135e5ab2de4de46f16"} Nov 03 01:01:13.224055 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:01:13.223443 4939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-b-208a0d-6f7c665d67-pc68m" event={"ID":"e1d5b015-e51e-4f86-ac94-34b05d8db2d3","Type":"ContainerStarted","Data":"d060e6ec87a45a8b3e8b62b4e645ca0bd618b90b84213b9ab12cb587f4d162c6"} Nov 03 01:01:13.224055 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:01:13.223604 4939 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-praxis-b-208a0d-6f7c665d67-pc68m" Nov 03 01:01:13.225381 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:01:13.225256 4939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-208a0d-f9cc48655-wfc6q" event={"ID":"19d3d3db-71fc-41f2-9976-fa8bcd10bfec","Type":"ContainerStarted","Data":"f6e857800bf7fcec22a2bf93eebc4b9237f4871fe54a63775de09f4823584346"} Nov 03 01:01:13.225381 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:01:13.225315 4939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-208a0d-f9cc48655-wfc6q" event={"ID":"19d3d3db-71fc-41f2-9976-fa8bcd10bfec","Type":"ContainerStarted","Data":"117451719c7dd4f9dc063a2c959d57244c6357b967191382d3e9857ce56e319c"} Nov 03 01:01:13.225381 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:01:13.225352 4939 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-208a0d-f9cc48655-wfc6q" Nov 03 01:01:13.227218 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:01:13.227196 4939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-d5f78f-openshift-default-64f48987d9-rk5hm" event={"ID":"dfea3192-3541-49c5-a491-ebf7308b74e0","Type":"ContainerStarted","Data":"e481325a728fa0455f38b868b5c3e86b20747ed84501736d04d1bc4267145bbe"} Nov 03 01:01:13.227313 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:01:13.227237 4939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-d5f78f-openshift-default-64f48987d9-rk5hm" event={"ID":"dfea3192-3541-49c5-a491-ebf7308b74e0","Type":"ContainerStarted","Data":"0b0eb9c283a2814232329a608d09e0e4e182142602704add5ffff7f476d5d46d"} Nov 03 01:01:13.248307 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:01:13.248250 4939 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-praxis-b-208a0d-6f7c665d67-pc68m" podStartSLOduration=2.248238134 podStartE2EDuration="2.248238134s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 01:01:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 01:01:13.245697085 +0000 UTC m=+1221.510032057" watchObservedRunningTime="2026-11-03 01:01:13.248238134 +0000 UTC m=+1221.512573078" Nov 03 01:01:13.265505 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:01:13.265455 4939 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-208a0d-f9cc48655-wfc6q" podStartSLOduration=2.265444344 podStartE2EDuration="2.265444344s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 01:01:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 01:01:13.262324866 +0000 UTC m=+1221.526659812" watchObservedRunningTime="2026-11-03 01:01:13.265444344 +0000 UTC m=+1221.529779289" Nov 03 01:01:13.287988 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:01:13.287944 4939 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-praxis-a-d5f78f-openshift-default-64f48987d9-rk5hm" podStartSLOduration=2.287930791 podStartE2EDuration="2.287930791s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 01:01:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 01:01:13.284043044 +0000 UTC m=+1221.548378001" watchObservedRunningTime="2026-11-03 01:01:13.287930791 +0000 UTC m=+1221.552265733" Nov 03 01:01:13.497137 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:01:13.496812 4939 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-praxis-a-d5f78f-openshift-default-64f48987d9-rk5hm" Nov 03 01:01:13.503744 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:01:13.503449 4939 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-praxis-a-d5f78f-openshift-default-64f48987d9-rk5hm" Nov 03 01:01:13.663259 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:01:13.663226 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:01:14.234384 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:01:14.234357 4939 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-praxis-a-d5f78f-openshift-default-64f48987d9-rk5hm" Nov 03 01:01:14.235091 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:01:14.235074 4939 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-praxis-a-d5f78f-openshift-default-64f48987d9-rk5hm" Nov 03 01:01:14.284338 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:01:14.284295 4939 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-b-02e4c0-7796479459-fjkx9" podUID="fd866aeb-ea66-4982-b18d-cdc4a1f8a836" containerName="payload-processing" containerID="cri-o://3f097a20ba5038ce2b70404d07d2ec82bb3b6540d2f02ad524360522382c0e13" gracePeriod=25 Nov 03 01:01:14.302288 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:01:14.302220 4939 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-02e4c0-57ff55f5cf-72j67" podUID="34792239-0f53-474d-adbf-fb1205780987" containerName="payload-pre-processing" containerID="cri-o://77080b26b5af20c8525019f42212cbfe6cc05b1b1b7eaf78ea7a1f9e16a52d95" gracePeriod=25 Nov 03 01:01:14.391817 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:01:14.391791 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:01:14.694578 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:01:14.694558 4939 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-02e4c0-7796479459-fjkx9" Nov 03 01:01:14.717782 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:01:14.717742 4939 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-02e4c0-57ff55f5cf-72j67" Nov 03 01:01:14.803286 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:01:14.803205 4939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fd866aeb-ea66-4982-b18d-cdc4a1f8a836-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fd866aeb-ea66-4982-b18d-cdc4a1f8a836-plugins-config-volume\") pod \"fd866aeb-ea66-4982-b18d-cdc4a1f8a836\" (UID: \"fd866aeb-ea66-4982-b18d-cdc4a1f8a836\") " Nov 03 01:01:14.803459 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:01:14.803299 4939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/34792239-0f53-474d-adbf-fb1205780987-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/34792239-0f53-474d-adbf-fb1205780987-plugins-config-volume\") pod \"34792239-0f53-474d-adbf-fb1205780987\" (UID: \"34792239-0f53-474d-adbf-fb1205780987\") " Nov 03 01:01:14.803459 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:01:14.803358 4939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/34792239-0f53-474d-adbf-fb1205780987-kube-api-access-b4h26\" (UniqueName: \"kubernetes.io/projected/34792239-0f53-474d-adbf-fb1205780987-kube-api-access-b4h26\") pod \"34792239-0f53-474d-adbf-fb1205780987\" (UID: \"34792239-0f53-474d-adbf-fb1205780987\") " Nov 03 01:01:14.803459 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:01:14.803411 4939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fd866aeb-ea66-4982-b18d-cdc4a1f8a836-kube-api-access-vgf8p\" (UniqueName: \"kubernetes.io/projected/fd866aeb-ea66-4982-b18d-cdc4a1f8a836-kube-api-access-vgf8p\") pod \"fd866aeb-ea66-4982-b18d-cdc4a1f8a836\" (UID: \"fd866aeb-ea66-4982-b18d-cdc4a1f8a836\") " Nov 03 01:01:14.803635 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:01:14.803605 4939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fd866aeb-ea66-4982-b18d-cdc4a1f8a836-plugins-config-volume" pod "fd866aeb-ea66-4982-b18d-cdc4a1f8a836" (UID: "fd866aeb-ea66-4982-b18d-cdc4a1f8a836"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 01:01:14.803746 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:01:14.803691 4939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/34792239-0f53-474d-adbf-fb1205780987-plugins-config-volume" pod "34792239-0f53-474d-adbf-fb1205780987" (UID: "34792239-0f53-474d-adbf-fb1205780987"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 01:01:14.804284 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:01:14.804261 4939 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fd866aeb-ea66-4982-b18d-cdc4a1f8a836-plugins-config-volume\") on node \"ip-10-0-140-107.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:01:14.804380 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:01:14.804286 4939 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/34792239-0f53-474d-adbf-fb1205780987-plugins-config-volume\") on node \"ip-10-0-140-107.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:01:14.806170 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:01:14.806146 4939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/34792239-0f53-474d-adbf-fb1205780987-kube-api-access-b4h26" pod "34792239-0f53-474d-adbf-fb1205780987" (UID: "34792239-0f53-474d-adbf-fb1205780987"). InnerVolumeSpecName "kube-api-access-b4h26". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 01:01:14.806243 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:01:14.806174 4939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fd866aeb-ea66-4982-b18d-cdc4a1f8a836-kube-api-access-vgf8p" pod "fd866aeb-ea66-4982-b18d-cdc4a1f8a836" (UID: "fd866aeb-ea66-4982-b18d-cdc4a1f8a836"). InnerVolumeSpecName "kube-api-access-vgf8p". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 01:01:14.906016 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:01:14.905977 4939 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b4h26\" (UniqueName: \"kubernetes.io/projected/34792239-0f53-474d-adbf-fb1205780987-kube-api-access-b4h26\") on node \"ip-10-0-140-107.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:01:14.906016 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:01:14.906020 4939 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vgf8p\" (UniqueName: \"kubernetes.io/projected/fd866aeb-ea66-4982-b18d-cdc4a1f8a836-kube-api-access-vgf8p\") on node \"ip-10-0-140-107.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:01:15.243429 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:01:15.243393 4939 generic.go:356] "Generic (PLEG): container finished" podID="34792239-0f53-474d-adbf-fb1205780987" containerID="77080b26b5af20c8525019f42212cbfe6cc05b1b1b7eaf78ea7a1f9e16a52d95" exitCode=0 Nov 03 01:01:15.243890 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:01:15.243440 4939 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-02e4c0-57ff55f5cf-72j67" Nov 03 01:01:15.243890 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:01:15.243481 4939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-02e4c0-57ff55f5cf-72j67" event={"ID":"34792239-0f53-474d-adbf-fb1205780987","Type":"ContainerDied","Data":"77080b26b5af20c8525019f42212cbfe6cc05b1b1b7eaf78ea7a1f9e16a52d95"} Nov 03 01:01:15.243890 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:01:15.243544 4939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-02e4c0-57ff55f5cf-72j67" event={"ID":"34792239-0f53-474d-adbf-fb1205780987","Type":"ContainerDied","Data":"37fc5aedb78760eba9bb267537e2fc7745ec0dfa3393512447de22453378e41b"} Nov 03 01:01:15.243890 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:01:15.243576 4939 scope.go:122] "RemoveContainer" containerID="77080b26b5af20c8525019f42212cbfe6cc05b1b1b7eaf78ea7a1f9e16a52d95" Nov 03 01:01:15.245099 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:01:15.245074 4939 generic.go:356] "Generic (PLEG): container finished" podID="fd866aeb-ea66-4982-b18d-cdc4a1f8a836" containerID="3f097a20ba5038ce2b70404d07d2ec82bb3b6540d2f02ad524360522382c0e13" exitCode=0 Nov 03 01:01:15.245195 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:01:15.245108 4939 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-02e4c0-7796479459-fjkx9" Nov 03 01:01:15.245195 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:01:15.245135 4939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-02e4c0-7796479459-fjkx9" event={"ID":"fd866aeb-ea66-4982-b18d-cdc4a1f8a836","Type":"ContainerDied","Data":"3f097a20ba5038ce2b70404d07d2ec82bb3b6540d2f02ad524360522382c0e13"} Nov 03 01:01:15.245195 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:01:15.245172 4939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-02e4c0-7796479459-fjkx9" event={"ID":"fd866aeb-ea66-4982-b18d-cdc4a1f8a836","Type":"ContainerDied","Data":"0d9793d67f18a5f9fed762a8bdc2bb5db9b9ebc16b7a907b4f684392db2f8bf6"} Nov 03 01:01:15.278056 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:01:15.278030 4939 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-02e4c0-7796479459-fjkx9"] Nov 03 01:01:15.283728 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:01:15.283689 4939 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-02e4c0-7796479459-fjkx9"] Nov 03 01:01:15.296778 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:01:15.296750 4939 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-02e4c0-57ff55f5cf-72j67"] Nov 03 01:01:15.303817 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:01:15.303792 4939 scope.go:122] "RemoveContainer" containerID="77080b26b5af20c8525019f42212cbfe6cc05b1b1b7eaf78ea7a1f9e16a52d95" Nov 03 01:01:15.304199 ip-10-0-140-107 kubenswrapper[4939]: E1103 01:01:15.304177 4939 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"77080b26b5af20c8525019f42212cbfe6cc05b1b1b7eaf78ea7a1f9e16a52d95\": container with ID starting with 77080b26b5af20c8525019f42212cbfe6cc05b1b1b7eaf78ea7a1f9e16a52d95 not found: ID does not exist" containerID="77080b26b5af20c8525019f42212cbfe6cc05b1b1b7eaf78ea7a1f9e16a52d95" Nov 03 01:01:15.304199 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:01:15.304193 4939 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-02e4c0-57ff55f5cf-72j67"] Nov 03 01:01:15.304380 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:01:15.304210 4939 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"77080b26b5af20c8525019f42212cbfe6cc05b1b1b7eaf78ea7a1f9e16a52d95"} err="failed to get container status \"77080b26b5af20c8525019f42212cbfe6cc05b1b1b7eaf78ea7a1f9e16a52d95\": rpc error: code = NotFound desc = could not find container \"77080b26b5af20c8525019f42212cbfe6cc05b1b1b7eaf78ea7a1f9e16a52d95\": container with ID starting with 77080b26b5af20c8525019f42212cbfe6cc05b1b1b7eaf78ea7a1f9e16a52d95 not found: ID does not exist" Nov 03 01:01:15.304380 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:01:15.304235 4939 scope.go:122] "RemoveContainer" containerID="3f097a20ba5038ce2b70404d07d2ec82bb3b6540d2f02ad524360522382c0e13" Nov 03 01:01:15.316949 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:01:15.316889 4939 scope.go:122] "RemoveContainer" containerID="3f097a20ba5038ce2b70404d07d2ec82bb3b6540d2f02ad524360522382c0e13" Nov 03 01:01:15.317277 ip-10-0-140-107 kubenswrapper[4939]: E1103 01:01:15.317248 4939 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3f097a20ba5038ce2b70404d07d2ec82bb3b6540d2f02ad524360522382c0e13\": container with ID starting with 3f097a20ba5038ce2b70404d07d2ec82bb3b6540d2f02ad524360522382c0e13 not found: ID does not exist" containerID="3f097a20ba5038ce2b70404d07d2ec82bb3b6540d2f02ad524360522382c0e13" Nov 03 01:01:15.317356 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:01:15.317288 4939 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3f097a20ba5038ce2b70404d07d2ec82bb3b6540d2f02ad524360522382c0e13"} err="failed to get container status \"3f097a20ba5038ce2b70404d07d2ec82bb3b6540d2f02ad524360522382c0e13\": rpc error: code = NotFound desc = could not find container \"3f097a20ba5038ce2b70404d07d2ec82bb3b6540d2f02ad524360522382c0e13\": container with ID starting with 3f097a20ba5038ce2b70404d07d2ec82bb3b6540d2f02ad524360522382c0e13 not found: ID does not exist" Nov 03 01:01:15.349965 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:01:15.349935 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:01:15.572250 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:01:15.572225 4939 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-def754-openshift-default-544466b787-9msx9" Nov 03 01:01:15.688669 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:01:15.688627 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:01:15.745975 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:01:15.745900 4939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8c763d19-e7ff-4638-8d21-3e977502a520-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8c763d19-e7ff-4638-8d21-3e977502a520-istio-data\") pod \"8c763d19-e7ff-4638-8d21-3e977502a520\" (UID: \"8c763d19-e7ff-4638-8d21-3e977502a520\") " Nov 03 01:01:15.746165 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:01:15.745983 4939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/8c763d19-e7ff-4638-8d21-3e977502a520-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8c763d19-e7ff-4638-8d21-3e977502a520-istio-podinfo\") pod \"8c763d19-e7ff-4638-8d21-3e977502a520\" (UID: \"8c763d19-e7ff-4638-8d21-3e977502a520\") " Nov 03 01:01:15.746165 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:01:15.746097 4939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8c763d19-e7ff-4638-8d21-3e977502a520-kube-api-access-wqrqg\" (UniqueName: \"kubernetes.io/projected/8c763d19-e7ff-4638-8d21-3e977502a520-kube-api-access-wqrqg\") pod \"8c763d19-e7ff-4638-8d21-3e977502a520\" (UID: \"8c763d19-e7ff-4638-8d21-3e977502a520\") " Nov 03 01:01:15.746165 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:01:15.746129 4939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8c763d19-e7ff-4638-8d21-3e977502a520-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8c763d19-e7ff-4638-8d21-3e977502a520-workload-socket\") pod \"8c763d19-e7ff-4638-8d21-3e977502a520\" (UID: \"8c763d19-e7ff-4638-8d21-3e977502a520\") " Nov 03 01:01:15.746165 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:01:15.746158 4939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8c763d19-e7ff-4638-8d21-3e977502a520-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8c763d19-e7ff-4638-8d21-3e977502a520-istiod-ca-cert\") pod \"8c763d19-e7ff-4638-8d21-3e977502a520\" (UID: \"8c763d19-e7ff-4638-8d21-3e977502a520\") " Nov 03 01:01:15.746385 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:01:15.746159 4939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8c763d19-e7ff-4638-8d21-3e977502a520-istio-data" pod "8c763d19-e7ff-4638-8d21-3e977502a520" (UID: "8c763d19-e7ff-4638-8d21-3e977502a520"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 01:01:15.746385 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:01:15.746255 4939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8c763d19-e7ff-4638-8d21-3e977502a520-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8c763d19-e7ff-4638-8d21-3e977502a520-istio-envoy\") pod \"8c763d19-e7ff-4638-8d21-3e977502a520\" (UID: \"8c763d19-e7ff-4638-8d21-3e977502a520\") " Nov 03 01:01:15.746385 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:01:15.746285 4939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8c763d19-e7ff-4638-8d21-3e977502a520-istio-token\" (UniqueName: \"kubernetes.io/projected/8c763d19-e7ff-4638-8d21-3e977502a520-istio-token\") pod \"8c763d19-e7ff-4638-8d21-3e977502a520\" (UID: \"8c763d19-e7ff-4638-8d21-3e977502a520\") " Nov 03 01:01:15.746385 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:01:15.746347 4939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8c763d19-e7ff-4638-8d21-3e977502a520-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8c763d19-e7ff-4638-8d21-3e977502a520-workload-certs\") pod \"8c763d19-e7ff-4638-8d21-3e977502a520\" (UID: \"8c763d19-e7ff-4638-8d21-3e977502a520\") " Nov 03 01:01:15.746588 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:01:15.746435 4939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8c763d19-e7ff-4638-8d21-3e977502a520-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8c763d19-e7ff-4638-8d21-3e977502a520-credential-socket\") pod \"8c763d19-e7ff-4638-8d21-3e977502a520\" (UID: \"8c763d19-e7ff-4638-8d21-3e977502a520\") " Nov 03 01:01:15.746588 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:01:15.746445 4939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8c763d19-e7ff-4638-8d21-3e977502a520-workload-socket" pod "8c763d19-e7ff-4638-8d21-3e977502a520" (UID: "8c763d19-e7ff-4638-8d21-3e977502a520"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 01:01:15.746588 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:01:15.746566 4939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8c763d19-e7ff-4638-8d21-3e977502a520-istiod-ca-cert" pod "8c763d19-e7ff-4638-8d21-3e977502a520" (UID: "8c763d19-e7ff-4638-8d21-3e977502a520"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 01:01:15.747039 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:01:15.746836 4939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8c763d19-e7ff-4638-8d21-3e977502a520-workload-certs" pod "8c763d19-e7ff-4638-8d21-3e977502a520" (UID: "8c763d19-e7ff-4638-8d21-3e977502a520"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 01:01:15.747169 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:01:15.747074 4939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8c763d19-e7ff-4638-8d21-3e977502a520-credential-socket" pod "8c763d19-e7ff-4638-8d21-3e977502a520" (UID: "8c763d19-e7ff-4638-8d21-3e977502a520"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 01:01:15.747336 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:01:15.747315 4939 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8c763d19-e7ff-4638-8d21-3e977502a520-istio-data\") on node \"ip-10-0-140-107.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:01:15.747410 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:01:15.747341 4939 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8c763d19-e7ff-4638-8d21-3e977502a520-workload-socket\") on node \"ip-10-0-140-107.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:01:15.747410 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:01:15.747355 4939 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8c763d19-e7ff-4638-8d21-3e977502a520-istiod-ca-cert\") on node \"ip-10-0-140-107.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:01:15.747410 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:01:15.747370 4939 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8c763d19-e7ff-4638-8d21-3e977502a520-workload-certs\") on node \"ip-10-0-140-107.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:01:15.747410 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:01:15.747384 4939 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8c763d19-e7ff-4638-8d21-3e977502a520-credential-socket\") on node \"ip-10-0-140-107.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:01:15.748751 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:01:15.748724 4939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8c763d19-e7ff-4638-8d21-3e977502a520-istio-token" pod "8c763d19-e7ff-4638-8d21-3e977502a520" (UID: "8c763d19-e7ff-4638-8d21-3e977502a520"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 01:01:15.748861 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:01:15.748772 4939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8c763d19-e7ff-4638-8d21-3e977502a520-istio-envoy" pod "8c763d19-e7ff-4638-8d21-3e977502a520" (UID: "8c763d19-e7ff-4638-8d21-3e977502a520"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 01:01:15.748861 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:01:15.748789 4939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8c763d19-e7ff-4638-8d21-3e977502a520-kube-api-access-wqrqg" pod "8c763d19-e7ff-4638-8d21-3e977502a520" (UID: "8c763d19-e7ff-4638-8d21-3e977502a520"). InnerVolumeSpecName "kube-api-access-wqrqg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 01:01:15.748861 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:01:15.748806 4939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/8c763d19-e7ff-4638-8d21-3e977502a520-istio-podinfo" pod "8c763d19-e7ff-4638-8d21-3e977502a520" (UID: "8c763d19-e7ff-4638-8d21-3e977502a520"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 01:01:15.824271 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:01:15.824192 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:01:15.848871 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:01:15.848838 4939 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8c763d19-e7ff-4638-8d21-3e977502a520-istio-podinfo\") on node \"ip-10-0-140-107.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:01:15.848871 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:01:15.848869 4939 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wqrqg\" (UniqueName: \"kubernetes.io/projected/8c763d19-e7ff-4638-8d21-3e977502a520-kube-api-access-wqrqg\") on node \"ip-10-0-140-107.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:01:15.849094 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:01:15.848881 4939 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8c763d19-e7ff-4638-8d21-3e977502a520-istio-envoy\") on node \"ip-10-0-140-107.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:01:15.849094 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:01:15.848891 4939 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8c763d19-e7ff-4638-8d21-3e977502a520-istio-token\") on node \"ip-10-0-140-107.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:01:15.871524 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:01:15.871493 4939 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="34792239-0f53-474d-adbf-fb1205780987" path="/var/lib/kubelet/pods/34792239-0f53-474d-adbf-fb1205780987/volumes" Nov 03 01:01:15.871972 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:01:15.871959 4939 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fd866aeb-ea66-4982-b18d-cdc4a1f8a836" path="/var/lib/kubelet/pods/fd866aeb-ea66-4982-b18d-cdc4a1f8a836/volumes" Nov 03 01:01:16.096199 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:01:16.096168 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:01:16.237374 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:01:16.237343 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:01:16.252061 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:01:16.252031 4939 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-02e4c0-597548c69f-ccdnj"] Nov 03 01:01:16.252858 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:01:16.252835 4939 cpu_manager.go:414] "containerMap: removing container" podUID="8c763d19-e7ff-4638-8d21-3e977502a520" containerName="istio-proxy" Nov 03 01:01:16.252955 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:01:16.252862 4939 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8c763d19-e7ff-4638-8d21-3e977502a520" containerName="istio-proxy" Nov 03 01:01:16.252955 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:01:16.252926 4939 cpu_manager.go:414] "containerMap: removing container" podUID="34792239-0f53-474d-adbf-fb1205780987" containerName="payload-pre-processing" Nov 03 01:01:16.252955 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:01:16.252936 4939 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="34792239-0f53-474d-adbf-fb1205780987" containerName="payload-pre-processing" Nov 03 01:01:16.252955 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:01:16.252953 4939 cpu_manager.go:414] "containerMap: removing container" podUID="8c763d19-e7ff-4638-8d21-3e977502a520" containerName="istio-proxy" Nov 03 01:01:16.253072 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:01:16.252962 4939 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8c763d19-e7ff-4638-8d21-3e977502a520" containerName="istio-proxy" Nov 03 01:01:16.253072 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:01:16.253014 4939 cpu_manager.go:414] "containerMap: removing container" podUID="fd866aeb-ea66-4982-b18d-cdc4a1f8a836" containerName="payload-processing" Nov 03 01:01:16.253072 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:01:16.253023 4939 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fd866aeb-ea66-4982-b18d-cdc4a1f8a836" containerName="payload-processing" Nov 03 01:01:16.253207 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:01:16.253193 4939 memory_manager.go:365] "RemoveStaleState removing state" podUID="8c763d19-e7ff-4638-8d21-3e977502a520" containerName="istio-proxy" Nov 03 01:01:16.253239 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:01:16.253218 4939 memory_manager.go:365] "RemoveStaleState removing state" podUID="34792239-0f53-474d-adbf-fb1205780987" containerName="payload-pre-processing" Nov 03 01:01:16.253239 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:01:16.253230 4939 memory_manager.go:365] "RemoveStaleState removing state" podUID="8c763d19-e7ff-4638-8d21-3e977502a520" containerName="istio-proxy" Nov 03 01:01:16.253303 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:01:16.253248 4939 memory_manager.go:365] "RemoveStaleState removing state" podUID="fd866aeb-ea66-4982-b18d-cdc4a1f8a836" containerName="payload-processing" Nov 03 01:01:16.258690 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:01:16.258662 4939 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-02e4c0-597548c69f-ccdnj" Nov 03 01:01:16.259208 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:01:16.259181 4939 generic.go:356] "Generic (PLEG): container finished" podID="8c763d19-e7ff-4638-8d21-3e977502a520" containerID="0580fbf79b774e762d78d096dda18b1f06f7743894ab28892d60afde59dda780" exitCode=0 Nov 03 01:01:16.259327 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:01:16.259230 4939 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-def754-openshift-default-544466b787-9msx9" Nov 03 01:01:16.259327 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:01:16.259227 4939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-def754-openshift-default-544466b787-9msx9" event={"ID":"8c763d19-e7ff-4638-8d21-3e977502a520","Type":"ContainerDied","Data":"0580fbf79b774e762d78d096dda18b1f06f7743894ab28892d60afde59dda780"} Nov 03 01:01:16.259438 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:01:16.259349 4939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-def754-openshift-default-544466b787-9msx9" event={"ID":"8c763d19-e7ff-4638-8d21-3e977502a520","Type":"ContainerDied","Data":"d4edfb00f2d587730e7b261aeb16e22647c701c8d1952f13bdd4d368625fc951"} Nov 03 01:01:16.259438 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:01:16.259373 4939 scope.go:122] "RemoveContainer" containerID="0580fbf79b774e762d78d096dda18b1f06f7743894ab28892d60afde59dda780" Nov 03 01:01:16.267431 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:01:16.267409 4939 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-b-02e4c0\"" Nov 03 01:01:16.267538 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:01:16.267515 4939 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-b-02e4c0-dockercfg-tr4rb\"" Nov 03 01:01:16.271463 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:01:16.271440 4939 scope.go:122] "RemoveContainer" containerID="32da52f1f2bddcb5fdad799643b3a1bfafca75e8e99547378cb528abcbd073b7" Nov 03 01:01:16.280489 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:01:16.280464 4939 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-02e4c0-597548c69f-ccdnj"] Nov 03 01:01:16.289102 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:01:16.288998 4939 scope.go:122] "RemoveContainer" containerID="0580fbf79b774e762d78d096dda18b1f06f7743894ab28892d60afde59dda780" Nov 03 01:01:16.290632 ip-10-0-140-107 kubenswrapper[4939]: E1103 01:01:16.290598 4939 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0580fbf79b774e762d78d096dda18b1f06f7743894ab28892d60afde59dda780\": container with ID starting with 0580fbf79b774e762d78d096dda18b1f06f7743894ab28892d60afde59dda780 not found: ID does not exist" containerID="0580fbf79b774e762d78d096dda18b1f06f7743894ab28892d60afde59dda780" Nov 03 01:01:16.290743 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:01:16.290644 4939 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0580fbf79b774e762d78d096dda18b1f06f7743894ab28892d60afde59dda780"} err="failed to get container status \"0580fbf79b774e762d78d096dda18b1f06f7743894ab28892d60afde59dda780\": rpc error: code = NotFound desc = could not find container \"0580fbf79b774e762d78d096dda18b1f06f7743894ab28892d60afde59dda780\": container with ID starting with 0580fbf79b774e762d78d096dda18b1f06f7743894ab28892d60afde59dda780 not found: ID does not exist" Nov 03 01:01:16.290743 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:01:16.290669 4939 scope.go:122] "RemoveContainer" containerID="32da52f1f2bddcb5fdad799643b3a1bfafca75e8e99547378cb528abcbd073b7" Nov 03 01:01:16.291066 ip-10-0-140-107 kubenswrapper[4939]: E1103 01:01:16.291041 4939 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"32da52f1f2bddcb5fdad799643b3a1bfafca75e8e99547378cb528abcbd073b7\": container with ID starting with 32da52f1f2bddcb5fdad799643b3a1bfafca75e8e99547378cb528abcbd073b7 not found: ID does not exist" containerID="32da52f1f2bddcb5fdad799643b3a1bfafca75e8e99547378cb528abcbd073b7" Nov 03 01:01:16.291176 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:01:16.291072 4939 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"32da52f1f2bddcb5fdad799643b3a1bfafca75e8e99547378cb528abcbd073b7"} err="failed to get container status \"32da52f1f2bddcb5fdad799643b3a1bfafca75e8e99547378cb528abcbd073b7\": rpc error: code = NotFound desc = could not find container \"32da52f1f2bddcb5fdad799643b3a1bfafca75e8e99547378cb528abcbd073b7\": container with ID starting with 32da52f1f2bddcb5fdad799643b3a1bfafca75e8e99547378cb528abcbd073b7 not found: ID does not exist" Nov 03 01:01:16.310483 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:01:16.310461 4939 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-def754-openshift-default-544466b787-9msx9"] Nov 03 01:01:16.423983 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:01:16.387321 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r5cdr\" (UniqueName: \"kubernetes.io/projected/ef1640af-2448-4ae7-b3e5-67a5113de620-kube-api-access-r5cdr\") pod \"payload-pre-processing-e2e-ipp-b-02e4c0-597548c69f-ccdnj\" (UID: \"ef1640af-2448-4ae7-b3e5-67a5113de620\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-02e4c0-597548c69f-ccdnj" Nov 03 01:01:16.423983 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:01:16.387477 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ef1640af-2448-4ae7-b3e5-67a5113de620-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-02e4c0-597548c69f-ccdnj\" (UID: \"ef1640af-2448-4ae7-b3e5-67a5113de620\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-02e4c0-597548c69f-ccdnj" Nov 03 01:01:16.546930 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:01:16.544711 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ef1640af-2448-4ae7-b3e5-67a5113de620-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-02e4c0-597548c69f-ccdnj\" (UID: \"ef1640af-2448-4ae7-b3e5-67a5113de620\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-02e4c0-597548c69f-ccdnj" Nov 03 01:01:16.546930 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:01:16.545057 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r5cdr\" (UniqueName: \"kubernetes.io/projected/ef1640af-2448-4ae7-b3e5-67a5113de620-kube-api-access-r5cdr\") pod \"payload-pre-processing-e2e-ipp-b-02e4c0-597548c69f-ccdnj\" (UID: \"ef1640af-2448-4ae7-b3e5-67a5113de620\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-02e4c0-597548c69f-ccdnj" Nov 03 01:01:16.546930 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:01:16.546745 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ef1640af-2448-4ae7-b3e5-67a5113de620-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-02e4c0-597548c69f-ccdnj\" (UID: \"ef1640af-2448-4ae7-b3e5-67a5113de620\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-02e4c0-597548c69f-ccdnj" Nov 03 01:01:16.616661 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:01:16.616614 4939 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-b-def754-openshift-default-544466b787-9msx9"] Nov 03 01:01:16.620811 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:01:16.620779 4939 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-02e4c0-68d4d67f9b-ncxlh"] Nov 03 01:01:16.628143 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:01:16.628109 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-r5cdr\" (UniqueName: \"kubernetes.io/projected/ef1640af-2448-4ae7-b3e5-67a5113de620-kube-api-access-r5cdr\") pod \"payload-pre-processing-e2e-ipp-b-02e4c0-597548c69f-ccdnj\" (UID: \"ef1640af-2448-4ae7-b3e5-67a5113de620\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-02e4c0-597548c69f-ccdnj" Nov 03 01:01:16.633515 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:01:16.633475 4939 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-02e4c0-68d4d67f9b-ncxlh"] Nov 03 01:01:16.633721 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:01:16.633618 4939 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-02e4c0-68d4d67f9b-ncxlh" Nov 03 01:01:16.747326 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:01:16.747230 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6svcf\" (UniqueName: \"kubernetes.io/projected/58ffd442-39d5-4e22-a8a2-1103aad2867e-kube-api-access-6svcf\") pod \"payload-processing-e2e-ipp-b-02e4c0-68d4d67f9b-ncxlh\" (UID: \"58ffd442-39d5-4e22-a8a2-1103aad2867e\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-02e4c0-68d4d67f9b-ncxlh" Nov 03 01:01:16.747326 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:01:16.747286 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/58ffd442-39d5-4e22-a8a2-1103aad2867e-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-02e4c0-68d4d67f9b-ncxlh\" (UID: \"58ffd442-39d5-4e22-a8a2-1103aad2867e\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-02e4c0-68d4d67f9b-ncxlh" Nov 03 01:01:16.835142 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:01:16.835108 4939 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-dc0f3f-656999868b-4bk2z"] Nov 03 01:01:16.840825 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:01:16.840599 4939 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-dc0f3f-656999868b-4bk2z" Nov 03 01:01:16.846096 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:01:16.846063 4939 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ipp-a-dc0f3f-serving-cert\"" Nov 03 01:01:16.849800 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:01:16.849768 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/58ffd442-39d5-4e22-a8a2-1103aad2867e-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-02e4c0-68d4d67f9b-ncxlh\" (UID: \"58ffd442-39d5-4e22-a8a2-1103aad2867e\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-02e4c0-68d4d67f9b-ncxlh" Nov 03 01:01:16.849986 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:01:16.849823 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6stmq\" (UniqueName: \"kubernetes.io/projected/61c842b8-bf22-49a5-b873-0b22bd049dd8-kube-api-access-6stmq\") pod \"maas-api-e2e-ipp-a-dc0f3f-656999868b-4bk2z\" (UID: \"61c842b8-bf22-49a5-b873-0b22bd049dd8\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-dc0f3f-656999868b-4bk2z" Nov 03 01:01:16.850053 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:01:16.849996 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/61c842b8-bf22-49a5-b873-0b22bd049dd8-maas-api-tls\") pod \"maas-api-e2e-ipp-a-dc0f3f-656999868b-4bk2z\" (UID: \"61c842b8-bf22-49a5-b873-0b22bd049dd8\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-dc0f3f-656999868b-4bk2z" Nov 03 01:01:16.850100 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:01:16.850085 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/61c842b8-bf22-49a5-b873-0b22bd049dd8-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-dc0f3f-656999868b-4bk2z\" (UID: \"61c842b8-bf22-49a5-b873-0b22bd049dd8\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-dc0f3f-656999868b-4bk2z" Nov 03 01:01:16.850159 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:01:16.850105 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/61c842b8-bf22-49a5-b873-0b22bd049dd8-metrics-certs\") pod \"maas-api-e2e-ipp-a-dc0f3f-656999868b-4bk2z\" (UID: \"61c842b8-bf22-49a5-b873-0b22bd049dd8\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-dc0f3f-656999868b-4bk2z" Nov 03 01:01:16.850159 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:01:16.850144 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6svcf\" (UniqueName: \"kubernetes.io/projected/58ffd442-39d5-4e22-a8a2-1103aad2867e-kube-api-access-6svcf\") pod \"payload-processing-e2e-ipp-b-02e4c0-68d4d67f9b-ncxlh\" (UID: \"58ffd442-39d5-4e22-a8a2-1103aad2867e\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-02e4c0-68d4d67f9b-ncxlh" Nov 03 01:01:16.850528 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:01:16.850508 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/58ffd442-39d5-4e22-a8a2-1103aad2867e-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-02e4c0-68d4d67f9b-ncxlh\" (UID: \"58ffd442-39d5-4e22-a8a2-1103aad2867e\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-02e4c0-68d4d67f9b-ncxlh" Nov 03 01:01:16.851408 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:01:16.851386 4939 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-dc0f3f-656999868b-4bk2z"] Nov 03 01:01:16.865688 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:01:16.865665 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6svcf\" (UniqueName: \"kubernetes.io/projected/58ffd442-39d5-4e22-a8a2-1103aad2867e-kube-api-access-6svcf\") pod \"payload-processing-e2e-ipp-b-02e4c0-68d4d67f9b-ncxlh\" (UID: \"58ffd442-39d5-4e22-a8a2-1103aad2867e\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-02e4c0-68d4d67f9b-ncxlh" Nov 03 01:01:16.876662 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:01:16.876636 4939 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-02e4c0-597548c69f-ccdnj" Nov 03 01:01:16.898472 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:01:16.898361 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:01:17.066736 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:01:17.066461 4939 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-02e4c0-68d4d67f9b-ncxlh" Nov 03 01:01:17.071080 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:01:17.070409 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/61c842b8-bf22-49a5-b873-0b22bd049dd8-maas-api-tls\") pod \"maas-api-e2e-ipp-a-dc0f3f-656999868b-4bk2z\" (UID: \"61c842b8-bf22-49a5-b873-0b22bd049dd8\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-dc0f3f-656999868b-4bk2z" Nov 03 01:01:17.071080 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:01:17.070614 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/61c842b8-bf22-49a5-b873-0b22bd049dd8-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-dc0f3f-656999868b-4bk2z\" (UID: \"61c842b8-bf22-49a5-b873-0b22bd049dd8\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-dc0f3f-656999868b-4bk2z" Nov 03 01:01:17.071080 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:01:17.070649 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/61c842b8-bf22-49a5-b873-0b22bd049dd8-metrics-certs\") pod \"maas-api-e2e-ipp-a-dc0f3f-656999868b-4bk2z\" (UID: \"61c842b8-bf22-49a5-b873-0b22bd049dd8\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-dc0f3f-656999868b-4bk2z" Nov 03 01:01:17.071080 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:01:17.070760 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6stmq\" (UniqueName: \"kubernetes.io/projected/61c842b8-bf22-49a5-b873-0b22bd049dd8-kube-api-access-6stmq\") pod \"maas-api-e2e-ipp-a-dc0f3f-656999868b-4bk2z\" (UID: \"61c842b8-bf22-49a5-b873-0b22bd049dd8\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-dc0f3f-656999868b-4bk2z" Nov 03 01:01:17.076560 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:01:17.072704 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/61c842b8-bf22-49a5-b873-0b22bd049dd8-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-dc0f3f-656999868b-4bk2z\" (UID: \"61c842b8-bf22-49a5-b873-0b22bd049dd8\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-dc0f3f-656999868b-4bk2z" Nov 03 01:01:17.097391 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:01:17.094512 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/61c842b8-bf22-49a5-b873-0b22bd049dd8-maas-api-tls\") pod \"maas-api-e2e-ipp-a-dc0f3f-656999868b-4bk2z\" (UID: \"61c842b8-bf22-49a5-b873-0b22bd049dd8\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-dc0f3f-656999868b-4bk2z" Nov 03 01:01:17.105317 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:01:17.104458 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6stmq\" (UniqueName: \"kubernetes.io/projected/61c842b8-bf22-49a5-b873-0b22bd049dd8-kube-api-access-6stmq\") pod \"maas-api-e2e-ipp-a-dc0f3f-656999868b-4bk2z\" (UID: \"61c842b8-bf22-49a5-b873-0b22bd049dd8\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-dc0f3f-656999868b-4bk2z" Nov 03 01:01:17.112150 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:01:17.111944 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/61c842b8-bf22-49a5-b873-0b22bd049dd8-metrics-certs\") pod \"maas-api-e2e-ipp-a-dc0f3f-656999868b-4bk2z\" (UID: \"61c842b8-bf22-49a5-b873-0b22bd049dd8\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-dc0f3f-656999868b-4bk2z" Nov 03 01:01:17.159896 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:01:17.159853 4939 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-dc0f3f-656999868b-4bk2z" Nov 03 01:01:17.192449 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:01:17.192382 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:01:17.244833 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:01:17.244788 4939 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-02e4c0-597548c69f-ccdnj"] Nov 03 01:01:17.254381 ip-10-0-140-107 kubenswrapper[4939]: W1103 01:01:17.254354 4939 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podef1640af_2448_4ae7_b3e5_67a5113de620.slice/crio-053c493b687c3457aef33e0db94225f6c0308ac3eb3432b557aef11bcf3f479d WatchSource:0}: Error finding container 053c493b687c3457aef33e0db94225f6c0308ac3eb3432b557aef11bcf3f479d: Status 404 returned error can't find the container with id 053c493b687c3457aef33e0db94225f6c0308ac3eb3432b557aef11bcf3f479d Nov 03 01:01:17.287893 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:01:17.287856 4939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-02e4c0-597548c69f-ccdnj" event={"ID":"ef1640af-2448-4ae7-b3e5-67a5113de620","Type":"ContainerStarted","Data":"053c493b687c3457aef33e0db94225f6c0308ac3eb3432b557aef11bcf3f479d"} Nov 03 01:01:17.350379 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:01:17.350329 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:01:17.405040 ip-10-0-140-107 kubenswrapper[4939]: W1103 01:01:17.405013 4939 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod58ffd442_39d5_4e22_a8a2_1103aad2867e.slice/crio-b2f32ac1ed6165834bd8e157ad8013528d0b1e9b00ea3a813e98d965aef9b1fe WatchSource:0}: Error finding container b2f32ac1ed6165834bd8e157ad8013528d0b1e9b00ea3a813e98d965aef9b1fe: Status 404 returned error can't find the container with id b2f32ac1ed6165834bd8e157ad8013528d0b1e9b00ea3a813e98d965aef9b1fe Nov 03 01:01:17.405226 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:01:17.405199 4939 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-02e4c0-68d4d67f9b-ncxlh"] Nov 03 01:01:17.487305 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:01:17.487263 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:01:17.496902 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:01:17.496823 4939 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-dc0f3f-656999868b-4bk2z"] Nov 03 01:01:17.914974 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:01:17.909855 4939 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8c763d19-e7ff-4638-8d21-3e977502a520" path="/var/lib/kubelet/pods/8c763d19-e7ff-4638-8d21-3e977502a520/volumes" Nov 03 01:01:18.142941 ip-10-0-140-107 kubenswrapper[4939]: E1103 01:01:18.142041 4939 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-sthjh" containerName="extractor" Nov 03 01:01:18.287275 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:01:18.287239 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:01:18.302647 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:01:18.302607 4939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-02e4c0-68d4d67f9b-ncxlh" event={"ID":"58ffd442-39d5-4e22-a8a2-1103aad2867e","Type":"ContainerStarted","Data":"4682a5c130dfdba6a6212e0d52adfc98520ae7e010667691c3d8bbf7bc84c93f"} Nov 03 01:01:18.302927 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:01:18.302891 4939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-02e4c0-68d4d67f9b-ncxlh" event={"ID":"58ffd442-39d5-4e22-a8a2-1103aad2867e","Type":"ContainerStarted","Data":"b2f32ac1ed6165834bd8e157ad8013528d0b1e9b00ea3a813e98d965aef9b1fe"} Nov 03 01:01:18.304188 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:01:18.304163 4939 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-02e4c0-68d4d67f9b-ncxlh" Nov 03 01:01:18.310061 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:01:18.310029 4939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-02e4c0-597548c69f-ccdnj" event={"ID":"ef1640af-2448-4ae7-b3e5-67a5113de620","Type":"ContainerStarted","Data":"3718cafe7a448d97a5568cfbc5104542f3276df70f76fa3482e2437f45743476"} Nov 03 01:01:18.311427 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:01:18.311404 4939 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-02e4c0-597548c69f-ccdnj" Nov 03 01:01:18.314271 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:01:18.314234 4939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-dc0f3f-656999868b-4bk2z" event={"ID":"61c842b8-bf22-49a5-b873-0b22bd049dd8","Type":"ContainerStarted","Data":"d317352158b7a37af0263136970f841a17c8e59cbb98371fbe6cf526755a98d9"} Nov 03 01:01:18.314468 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:01:18.314448 4939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-dc0f3f-656999868b-4bk2z" event={"ID":"61c842b8-bf22-49a5-b873-0b22bd049dd8","Type":"ContainerStarted","Data":"9447a1951212b276d05062a44245124a79947c7732483119a2a59e3a01b71aa3"} Nov 03 01:01:18.315894 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:01:18.315868 4939 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-dc0f3f-656999868b-4bk2z" Nov 03 01:01:18.344937 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:01:18.335360 4939 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-b-02e4c0-68d4d67f9b-ncxlh" podStartSLOduration=2.335338927 podStartE2EDuration="2.335338927s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 01:01:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 01:01:18.326545778 +0000 UTC m=+1226.590880724" watchObservedRunningTime="2026-11-03 01:01:18.335338927 +0000 UTC m=+1226.599673873" Nov 03 01:01:18.356334 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:01:18.356280 4939 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-dc0f3f-656999868b-4bk2z" podStartSLOduration=1.711824338 podStartE2EDuration="2.356263863s" totalImagesPullingTime="644.439525ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 01:01:16 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 01:01:18.353208903 +0000 UTC m=+1226.617543849" watchObservedRunningTime="2026-11-03 01:01:18.356263863 +0000 UTC m=+1226.620598807" Nov 03 01:01:18.940870 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:01:18.940842 4939 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-dc0f3f-747fcc75fc-zvf8h" Nov 03 01:01:18.941086 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:01:18.940961 4939 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-dc0f3f-5c746d84f9-4scbk" Nov 03 01:01:18.963799 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:01:18.963736 4939 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-02e4c0-597548c69f-ccdnj" podStartSLOduration=2.963717441 podStartE2EDuration="2.963717441s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 01:01:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 01:01:18.394890594 +0000 UTC m=+1226.659225541" watchObservedRunningTime="2026-11-03 01:01:18.963717441 +0000 UTC m=+1227.228052386" Nov 03 01:01:19.205638 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:01:19.205479 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:01:20.086903 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:01:20.086870 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:01:20.758181 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:01:20.758140 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:01:21.696481 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:01:21.696411 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:01:21.818108 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:01:21.818060 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:01:21.933737 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:01:21.933704 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:01:22.449953 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:01:22.448555 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:01:23.313369 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:01:23.313328 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:01:24.102898 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:01:24.102859 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:01:24.237410 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:01:24.237380 4939 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-praxis-b-208a0d-6f7c665d67-pc68m" Nov 03 01:01:24.237603 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:01:24.237449 4939 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-208a0d-f9cc48655-wfc6q" Nov 03 01:01:24.459807 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:01:24.459729 4939 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-4pw7v"] Nov 03 01:01:25.011015 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:01:25.010963 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:01:25.375547 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:01:25.373081 4939 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-dc0f3f-656999868b-4bk2z" Nov 03 01:01:25.920128 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:01:25.920096 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:01:26.619133 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:01:26.619098 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:01:27.405359 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:01:27.405328 4939 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-d5f78f-7c9668888d-dwcfx"] Nov 03 01:01:27.411849 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:01:27.411824 4939 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-d5f78f-7c9668888d-dwcfx" Nov 03 01:01:27.419520 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:01:27.419490 4939 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-praxis-a-d5f78f-serving-cert\"" Nov 03 01:01:27.428198 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:01:27.428170 4939 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-d5f78f-7c9668888d-dwcfx"] Nov 03 01:01:27.491355 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:01:27.491322 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:01:27.514157 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:01:27.514128 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-snmpr\" (UniqueName: \"kubernetes.io/projected/3b61a830-b96e-4ad7-a5b9-567553bc47c9-kube-api-access-snmpr\") pod \"maas-api-e2e-praxis-a-d5f78f-7c9668888d-dwcfx\" (UID: \"3b61a830-b96e-4ad7-a5b9-567553bc47c9\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-d5f78f-7c9668888d-dwcfx" Nov 03 01:01:27.514274 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:01:27.514175 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3b61a830-b96e-4ad7-a5b9-567553bc47c9-openshift-service-ca\") pod \"maas-api-e2e-praxis-a-d5f78f-7c9668888d-dwcfx\" (UID: \"3b61a830-b96e-4ad7-a5b9-567553bc47c9\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-d5f78f-7c9668888d-dwcfx" Nov 03 01:01:27.514335 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:01:27.514316 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3b61a830-b96e-4ad7-a5b9-567553bc47c9-metrics-certs\") pod \"maas-api-e2e-praxis-a-d5f78f-7c9668888d-dwcfx\" (UID: \"3b61a830-b96e-4ad7-a5b9-567553bc47c9\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-d5f78f-7c9668888d-dwcfx" Nov 03 01:01:27.514395 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:01:27.514381 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3b61a830-b96e-4ad7-a5b9-567553bc47c9-maas-api-tls\") pod \"maas-api-e2e-praxis-a-d5f78f-7c9668888d-dwcfx\" (UID: \"3b61a830-b96e-4ad7-a5b9-567553bc47c9\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-d5f78f-7c9668888d-dwcfx" Nov 03 01:01:27.620507 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:01:27.620426 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:01:27.626652 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:01:27.624154 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-snmpr\" (UniqueName: \"kubernetes.io/projected/3b61a830-b96e-4ad7-a5b9-567553bc47c9-kube-api-access-snmpr\") pod \"maas-api-e2e-praxis-a-d5f78f-7c9668888d-dwcfx\" (UID: \"3b61a830-b96e-4ad7-a5b9-567553bc47c9\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-d5f78f-7c9668888d-dwcfx" Nov 03 01:01:27.626652 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:01:27.624234 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3b61a830-b96e-4ad7-a5b9-567553bc47c9-openshift-service-ca\") pod \"maas-api-e2e-praxis-a-d5f78f-7c9668888d-dwcfx\" (UID: \"3b61a830-b96e-4ad7-a5b9-567553bc47c9\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-d5f78f-7c9668888d-dwcfx" Nov 03 01:01:27.626652 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:01:27.624426 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3b61a830-b96e-4ad7-a5b9-567553bc47c9-metrics-certs\") pod \"maas-api-e2e-praxis-a-d5f78f-7c9668888d-dwcfx\" (UID: \"3b61a830-b96e-4ad7-a5b9-567553bc47c9\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-d5f78f-7c9668888d-dwcfx" Nov 03 01:01:27.628267 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:01:27.628189 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3b61a830-b96e-4ad7-a5b9-567553bc47c9-openshift-service-ca\") pod \"maas-api-e2e-praxis-a-d5f78f-7c9668888d-dwcfx\" (UID: \"3b61a830-b96e-4ad7-a5b9-567553bc47c9\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-d5f78f-7c9668888d-dwcfx" Nov 03 01:01:27.632845 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:01:27.632792 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3b61a830-b96e-4ad7-a5b9-567553bc47c9-maas-api-tls\") pod \"maas-api-e2e-praxis-a-d5f78f-7c9668888d-dwcfx\" (UID: \"3b61a830-b96e-4ad7-a5b9-567553bc47c9\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-d5f78f-7c9668888d-dwcfx" Nov 03 01:01:27.635515 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:01:27.635492 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3b61a830-b96e-4ad7-a5b9-567553bc47c9-metrics-certs\") pod \"maas-api-e2e-praxis-a-d5f78f-7c9668888d-dwcfx\" (UID: \"3b61a830-b96e-4ad7-a5b9-567553bc47c9\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-d5f78f-7c9668888d-dwcfx" Nov 03 01:01:27.639759 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:01:27.639728 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3b61a830-b96e-4ad7-a5b9-567553bc47c9-maas-api-tls\") pod \"maas-api-e2e-praxis-a-d5f78f-7c9668888d-dwcfx\" (UID: \"3b61a830-b96e-4ad7-a5b9-567553bc47c9\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-d5f78f-7c9668888d-dwcfx" Nov 03 01:01:27.640877 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:01:27.640857 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-snmpr\" (UniqueName: \"kubernetes.io/projected/3b61a830-b96e-4ad7-a5b9-567553bc47c9-kube-api-access-snmpr\") pod \"maas-api-e2e-praxis-a-d5f78f-7c9668888d-dwcfx\" (UID: \"3b61a830-b96e-4ad7-a5b9-567553bc47c9\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-d5f78f-7c9668888d-dwcfx" Nov 03 01:01:27.731594 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:01:27.731528 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:01:27.734379 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:01:27.734357 4939 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-d5f78f-7c9668888d-dwcfx" Nov 03 01:01:28.020059 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:01:28.020028 4939 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-d5f78f-7c9668888d-dwcfx"] Nov 03 01:01:28.021215 ip-10-0-140-107 kubenswrapper[4939]: W1103 01:01:28.021185 4939 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3b61a830_b96e_4ad7_a5b9_567553bc47c9.slice/crio-5c58b2cc7f452e180d50ee4953ee7d8588f242eb1478aca41ae929fc7cbe678f WatchSource:0}: Error finding container 5c58b2cc7f452e180d50ee4953ee7d8588f242eb1478aca41ae929fc7cbe678f: Status 404 returned error can't find the container with id 5c58b2cc7f452e180d50ee4953ee7d8588f242eb1478aca41ae929fc7cbe678f Nov 03 01:01:28.386543 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:01:28.386514 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:01:28.671203 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:01:28.671162 4939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-d5f78f-7c9668888d-dwcfx" event={"ID":"3b61a830-b96e-4ad7-a5b9-567553bc47c9","Type":"ContainerStarted","Data":"a2914c22a0119d48fce1a9a0d718c91addaae49147b2d0ac323d65ca3014ac6f"} Nov 03 01:01:28.671583 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:01:28.671259 4939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-d5f78f-7c9668888d-dwcfx" event={"ID":"3b61a830-b96e-4ad7-a5b9-567553bc47c9","Type":"ContainerStarted","Data":"5c58b2cc7f452e180d50ee4953ee7d8588f242eb1478aca41ae929fc7cbe678f"} Nov 03 01:01:28.671583 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:01:28.671380 4939 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-d5f78f-7c9668888d-dwcfx" Nov 03 01:01:28.693486 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:01:28.693442 4939 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-d5f78f-7c9668888d-dwcfx" podStartSLOduration=1.147551642 podStartE2EDuration="1.693430068s" totalImagesPullingTime="545.878426ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 01:01:27 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 01:01:28.68979218 +0000 UTC m=+1236.954127162" watchObservedRunningTime="2026-11-03 01:01:28.693430068 +0000 UTC m=+1236.957765013" Nov 03 01:01:29.579123 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:01:29.579061 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:01:30.369939 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:01:30.353064 4939 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-02e4c0-597548c69f-ccdnj" Nov 03 01:01:30.374112 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:01:30.374078 4939 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-02e4c0-68d4d67f9b-ncxlh" Nov 03 01:01:30.690658 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:01:30.690554 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:01:31.684653 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:01:31.684620 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:01:32.590040 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:01:32.590008 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:01:33.440349 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:01:33.440313 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:01:33.896280 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:01:33.896251 4939 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-55fbbbf9dc-qvj7w_7cab38b8-5158-4678-a429-350af5e2f72f/payload-processing/0.log" Nov 03 01:01:34.097471 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:01:34.097446 4939 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-55fbbbf9dc-qvj7w_7cab38b8-5158-4678-a429-350af5e2f72f/payload-processing/0.log" Nov 03 01:01:34.272865 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:01:34.272744 4939 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-e2e-ipp-b-02e4c0-68d4d67f9b-ncxlh_58ffd442-39d5-4e22-a8a2-1103aad2867e/payload-processing/0.log" Nov 03 01:01:34.355559 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:01:34.355481 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:01:34.704936 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:01:34.704893 4939 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-d5f78f-7c9668888d-dwcfx" Nov 03 01:01:35.267004 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:01:35.266903 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:01:35.388237 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:01:35.388203 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:01:35.398673 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:01:35.398632 4939 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-dc0f3f-747fcc75fc-zvf8h"] Nov 03 01:01:35.433928 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:01:35.424688 4939 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-dc0f3f-5c746d84f9-4scbk"] Nov 03 01:01:35.528625 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:01:35.528539 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:01:35.658271 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:01:35.658228 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:01:35.789982 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:01:35.778704 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:01:36.308423 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:01:36.308381 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:01:36.997801 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:01:36.997759 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:01:37.624289 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:01:37.624255 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:01:38.358519 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:01:38.358485 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:01:38.576036 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:01:38.575998 4939 patch_prober.go:29] interesting pod/e2e-ipp-b-02e4c0-openshift-default-c64cd65d9-7l2h4 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.131:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 01:01:38.576223 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:01:38.576071 4939 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ipp-b-02e4c0-openshift-default-c64cd65d9-7l2h4" podUID="2df5fd65-3b2a-448e-92a8-f07a8888692d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.131:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 01:01:39.025605 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:01:39.025561 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:01:39.801838 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:01:39.801802 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:01:39.935241 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:01:39.935150 4939 generic.go:356] "Generic (PLEG): container finished" podID="2df5fd65-3b2a-448e-92a8-f07a8888692d" containerID="ea409d7fcc0d44e2d14e7b53da7e4f0377490e01ba0cd67ec8b499c4d77a8b20" exitCode=0 Nov 03 01:01:39.935405 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:01:39.935229 4939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-02e4c0-openshift-default-c64cd65d9-7l2h4" event={"ID":"2df5fd65-3b2a-448e-92a8-f07a8888692d","Type":"ContainerDied","Data":"ea409d7fcc0d44e2d14e7b53da7e4f0377490e01ba0cd67ec8b499c4d77a8b20"} Nov 03 01:01:39.935767 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:01:39.935751 4939 scope.go:122] "RemoveContainer" containerID="ea409d7fcc0d44e2d14e7b53da7e4f0377490e01ba0cd67ec8b499c4d77a8b20" Nov 03 01:01:39.937480 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:01:39.937454 4939 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 01:01:39.937560 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:01:39.937522 4939 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 01:01:39.937560 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:01:39.937556 4939 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 01:01:40.177444 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:01:40.177358 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:01:40.399286 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:01:40.399233 4939 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-a-dc0f3f-747fcc75fc-zvf8h" podUID="9fc6f374-4e0d-4a76-aa72-056b80fee767" containerName="payload-processing" containerID="cri-o://b03b86b8943b71df62793cecd7dbfc46a1b0789e1d5c55930c3041a0c2917968" gracePeriod=25 Nov 03 01:01:40.426075 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:01:40.425961 4939 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-dc0f3f-5c746d84f9-4scbk" podUID="ffa6f2b0-be86-4f99-8d96-e4835139d9cd" containerName="payload-pre-processing" containerID="cri-o://c29b71509ea2bdd39abc500025e44a79fc354f93c4d29c03d8b010f2277bb9f7" gracePeriod=25 Nov 03 01:01:40.909890 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:01:40.909210 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:01:40.938891 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:01:40.938117 4939 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-dc0f3f-747fcc75fc-zvf8h" Nov 03 01:01:40.948988 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:01:40.948902 4939 generic.go:356] "Generic (PLEG): container finished" podID="9fc6f374-4e0d-4a76-aa72-056b80fee767" containerID="b03b86b8943b71df62793cecd7dbfc46a1b0789e1d5c55930c3041a0c2917968" exitCode=0 Nov 03 01:01:40.949158 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:01:40.949043 4939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-dc0f3f-747fcc75fc-zvf8h" event={"ID":"9fc6f374-4e0d-4a76-aa72-056b80fee767","Type":"ContainerDied","Data":"b03b86b8943b71df62793cecd7dbfc46a1b0789e1d5c55930c3041a0c2917968"} Nov 03 01:01:40.949158 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:01:40.949111 4939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-dc0f3f-747fcc75fc-zvf8h" event={"ID":"9fc6f374-4e0d-4a76-aa72-056b80fee767","Type":"ContainerDied","Data":"c977d94751cfa99cc7c99cef9e67b06e47d3aa9b2f43333248ffffa407066883"} Nov 03 01:01:40.949158 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:01:40.949140 4939 scope.go:122] "RemoveContainer" containerID="b03b86b8943b71df62793cecd7dbfc46a1b0789e1d5c55930c3041a0c2917968" Nov 03 01:01:40.949348 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:01:40.949334 4939 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-dc0f3f-747fcc75fc-zvf8h" Nov 03 01:01:40.953792 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:01:40.953703 4939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-02e4c0-openshift-default-c64cd65d9-7l2h4" event={"ID":"2df5fd65-3b2a-448e-92a8-f07a8888692d","Type":"ContainerStarted","Data":"153561948d7bc4acf758d26763ac2b0e3a6e4165173e85bf546f7b0b4314fd27"} Nov 03 01:01:40.957416 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:01:40.957387 4939 generic.go:356] "Generic (PLEG): container finished" podID="ffa6f2b0-be86-4f99-8d96-e4835139d9cd" containerID="c29b71509ea2bdd39abc500025e44a79fc354f93c4d29c03d8b010f2277bb9f7" exitCode=0 Nov 03 01:01:40.957521 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:01:40.957468 4939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-dc0f3f-5c746d84f9-4scbk" event={"ID":"ffa6f2b0-be86-4f99-8d96-e4835139d9cd","Type":"ContainerDied","Data":"c29b71509ea2bdd39abc500025e44a79fc354f93c4d29c03d8b010f2277bb9f7"} Nov 03 01:01:40.979643 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:01:40.979624 4939 scope.go:122] "RemoveContainer" containerID="b03b86b8943b71df62793cecd7dbfc46a1b0789e1d5c55930c3041a0c2917968" Nov 03 01:01:40.979946 ip-10-0-140-107 kubenswrapper[4939]: E1103 01:01:40.979902 4939 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b03b86b8943b71df62793cecd7dbfc46a1b0789e1d5c55930c3041a0c2917968\": container with ID starting with b03b86b8943b71df62793cecd7dbfc46a1b0789e1d5c55930c3041a0c2917968 not found: ID does not exist" containerID="b03b86b8943b71df62793cecd7dbfc46a1b0789e1d5c55930c3041a0c2917968" Nov 03 01:01:40.980018 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:01:40.979957 4939 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b03b86b8943b71df62793cecd7dbfc46a1b0789e1d5c55930c3041a0c2917968"} err="failed to get container status \"b03b86b8943b71df62793cecd7dbfc46a1b0789e1d5c55930c3041a0c2917968\": rpc error: code = NotFound desc = could not find container \"b03b86b8943b71df62793cecd7dbfc46a1b0789e1d5c55930c3041a0c2917968\": container with ID starting with b03b86b8943b71df62793cecd7dbfc46a1b0789e1d5c55930c3041a0c2917968 not found: ID does not exist" Nov 03 01:01:41.018876 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:01:41.018822 4939 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-dc0f3f-5c746d84f9-4scbk" Nov 03 01:01:41.106867 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:01:41.106831 4939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9fc6f374-4e0d-4a76-aa72-056b80fee767-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9fc6f374-4e0d-4a76-aa72-056b80fee767-plugins-config-volume\") pod \"9fc6f374-4e0d-4a76-aa72-056b80fee767\" (UID: \"9fc6f374-4e0d-4a76-aa72-056b80fee767\") " Nov 03 01:01:41.107065 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:01:41.107030 4939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9fc6f374-4e0d-4a76-aa72-056b80fee767-kube-api-access-8vl5q\" (UniqueName: \"kubernetes.io/projected/9fc6f374-4e0d-4a76-aa72-056b80fee767-kube-api-access-8vl5q\") pod \"9fc6f374-4e0d-4a76-aa72-056b80fee767\" (UID: \"9fc6f374-4e0d-4a76-aa72-056b80fee767\") " Nov 03 01:01:41.107512 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:01:41.107426 4939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9fc6f374-4e0d-4a76-aa72-056b80fee767-plugins-config-volume" pod "9fc6f374-4e0d-4a76-aa72-056b80fee767" (UID: "9fc6f374-4e0d-4a76-aa72-056b80fee767"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 01:01:41.107709 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:01:41.107686 4939 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9fc6f374-4e0d-4a76-aa72-056b80fee767-plugins-config-volume\") on node \"ip-10-0-140-107.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:01:41.109877 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:01:41.109853 4939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9fc6f374-4e0d-4a76-aa72-056b80fee767-kube-api-access-8vl5q" pod "9fc6f374-4e0d-4a76-aa72-056b80fee767" (UID: "9fc6f374-4e0d-4a76-aa72-056b80fee767"). InnerVolumeSpecName "kube-api-access-8vl5q". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 01:01:41.209212 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:01:41.209176 4939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ffa6f2b0-be86-4f99-8d96-e4835139d9cd-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ffa6f2b0-be86-4f99-8d96-e4835139d9cd-plugins-config-volume\") pod \"ffa6f2b0-be86-4f99-8d96-e4835139d9cd\" (UID: \"ffa6f2b0-be86-4f99-8d96-e4835139d9cd\") " Nov 03 01:01:41.209394 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:01:41.209239 4939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ffa6f2b0-be86-4f99-8d96-e4835139d9cd-kube-api-access-b7bp4\" (UniqueName: \"kubernetes.io/projected/ffa6f2b0-be86-4f99-8d96-e4835139d9cd-kube-api-access-b7bp4\") pod \"ffa6f2b0-be86-4f99-8d96-e4835139d9cd\" (UID: \"ffa6f2b0-be86-4f99-8d96-e4835139d9cd\") " Nov 03 01:01:41.209569 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:01:41.209542 4939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ffa6f2b0-be86-4f99-8d96-e4835139d9cd-plugins-config-volume" pod "ffa6f2b0-be86-4f99-8d96-e4835139d9cd" (UID: "ffa6f2b0-be86-4f99-8d96-e4835139d9cd"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 01:01:41.209951 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:01:41.209898 4939 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8vl5q\" (UniqueName: \"kubernetes.io/projected/9fc6f374-4e0d-4a76-aa72-056b80fee767-kube-api-access-8vl5q\") on node \"ip-10-0-140-107.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:01:41.210013 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:01:41.209954 4939 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ffa6f2b0-be86-4f99-8d96-e4835139d9cd-plugins-config-volume\") on node \"ip-10-0-140-107.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:01:41.211434 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:01:41.211410 4939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ffa6f2b0-be86-4f99-8d96-e4835139d9cd-kube-api-access-b7bp4" pod "ffa6f2b0-be86-4f99-8d96-e4835139d9cd" (UID: "ffa6f2b0-be86-4f99-8d96-e4835139d9cd"). InnerVolumeSpecName "kube-api-access-b7bp4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 01:01:41.283735 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:01:41.283653 4939 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-dc0f3f-747fcc75fc-zvf8h"] Nov 03 01:01:41.292439 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:01:41.292404 4939 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-dc0f3f-747fcc75fc-zvf8h"] Nov 03 01:01:41.311884 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:01:41.311857 4939 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b7bp4\" (UniqueName: \"kubernetes.io/projected/ffa6f2b0-be86-4f99-8d96-e4835139d9cd-kube-api-access-b7bp4\") on node \"ip-10-0-140-107.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:01:41.435506 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:01:41.435475 4939 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-b-02e4c0-openshift-default-c64cd65d9-7l2h4" Nov 03 01:01:41.435762 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:01:41.435734 4939 patch_prober.go:29] interesting pod/e2e-ipp-b-02e4c0-openshift-default-c64cd65d9-7l2h4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.131:15021/healthz/ready\": dial tcp 10.132.0.131:15021: connect: connection refused" start-of-body= Nov 03 01:01:41.435812 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:01:41.435788 4939 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-02e4c0-openshift-default-c64cd65d9-7l2h4" podUID="2df5fd65-3b2a-448e-92a8-f07a8888692d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.131:15021/healthz/ready\": dial tcp 10.132.0.131:15021: connect: connection refused" Nov 03 01:01:41.887094 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:01:41.887055 4939 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9fc6f374-4e0d-4a76-aa72-056b80fee767" path="/var/lib/kubelet/pods/9fc6f374-4e0d-4a76-aa72-056b80fee767/volumes" Nov 03 01:01:41.930696 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:01:41.930669 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:01:42.011955 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:01:42.005179 4939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-dc0f3f-5c746d84f9-4scbk" event={"ID":"ffa6f2b0-be86-4f99-8d96-e4835139d9cd","Type":"ContainerDied","Data":"d7d29def85899c6b23a87d48317533aefcc9025a33509c1b7913c0a0b25fa17a"} Nov 03 01:01:42.011955 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:01:42.005253 4939 scope.go:122] "RemoveContainer" containerID="c29b71509ea2bdd39abc500025e44a79fc354f93c4d29c03d8b010f2277bb9f7" Nov 03 01:01:42.011955 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:01:42.005369 4939 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-dc0f3f-5c746d84f9-4scbk" Nov 03 01:01:42.083006 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:01:42.082967 4939 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-dc0f3f-5c746d84f9-4scbk"] Nov 03 01:01:42.112623 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:01:42.112596 4939 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-dc0f3f-5c746d84f9-4scbk"] Nov 03 01:01:42.435342 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:01:42.435263 4939 patch_prober.go:29] interesting pod/e2e-ipp-b-02e4c0-openshift-default-c64cd65d9-7l2h4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.131:15021/healthz/ready\": dial tcp 10.132.0.131:15021: connect: connection refused" start-of-body= Nov 03 01:01:42.435342 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:01:42.435325 4939 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-02e4c0-openshift-default-c64cd65d9-7l2h4" podUID="2df5fd65-3b2a-448e-92a8-f07a8888692d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.131:15021/healthz/ready\": dial tcp 10.132.0.131:15021: connect: connection refused" Nov 03 01:01:42.580448 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:01:42.579808 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:01:42.707028 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:01:42.706870 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:01:42.895193 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:01:42.885880 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:01:43.012981 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:01:43.007685 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:01:43.440025 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:01:43.438290 4939 patch_prober.go:29] interesting pod/e2e-ipp-b-02e4c0-openshift-default-c64cd65d9-7l2h4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.131:15021/healthz/ready\": dial tcp 10.132.0.131:15021: connect: connection refused" start-of-body= Nov 03 01:01:43.440025 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:01:43.438352 4939 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-02e4c0-openshift-default-c64cd65d9-7l2h4" podUID="2df5fd65-3b2a-448e-92a8-f07a8888692d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.131:15021/healthz/ready\": dial tcp 10.132.0.131:15021: connect: connection refused" Nov 03 01:01:43.464933 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:01:43.461628 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:01:43.498031 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:01:43.497970 4939 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-praxis-a-1e352e/praxis-route-1e352e-kserve-67cb6ccfdc-k25pd"] Nov 03 01:01:43.525935 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:01:43.525290 4939 cpu_manager.go:414] "containerMap: removing container" podUID="ffa6f2b0-be86-4f99-8d96-e4835139d9cd" containerName="payload-pre-processing" Nov 03 01:01:43.525935 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:01:43.525324 4939 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ffa6f2b0-be86-4f99-8d96-e4835139d9cd" containerName="payload-pre-processing" Nov 03 01:01:43.525935 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:01:43.525366 4939 cpu_manager.go:414] "containerMap: removing container" podUID="9fc6f374-4e0d-4a76-aa72-056b80fee767" containerName="payload-processing" Nov 03 01:01:43.525935 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:01:43.525377 4939 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9fc6f374-4e0d-4a76-aa72-056b80fee767" containerName="payload-processing" Nov 03 01:01:43.525935 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:01:43.525592 4939 memory_manager.go:365] "RemoveStaleState removing state" podUID="ffa6f2b0-be86-4f99-8d96-e4835139d9cd" containerName="payload-pre-processing" Nov 03 01:01:43.525935 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:01:43.525612 4939 memory_manager.go:365] "RemoveStaleState removing state" podUID="9fc6f374-4e0d-4a76-aa72-056b80fee767" containerName="payload-processing" Nov 03 01:01:43.540039 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:01:43.540007 4939 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-praxis-a-1e352e/praxis-route-1e352e-kserve-67cb6ccfdc-k25pd"] Nov 03 01:01:43.540189 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:01:43.540126 4939 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-praxis-a-1e352e/praxis-route-1e352e-kserve-67cb6ccfdc-k25pd" Nov 03 01:01:43.545572 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:01:43.545547 4939 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-praxis-a-1e352e\"/\"default-dockercfg-k225g\"" Nov 03 01:01:43.545735 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:01:43.545655 4939 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-praxis-a-1e352e\"/\"kube-root-ca.crt\"" Nov 03 01:01:43.545735 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:01:43.545696 4939 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-praxis-a-1e352e\"/\"praxis-route-1e352e-kserve-self-signed-certs\"" Nov 03 01:01:43.545925 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:01:43.545775 4939 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-praxis-a-1e352e\"/\"openshift-service-ca.crt\"" Nov 03 01:01:43.579334 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:01:43.579284 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b645db90-ca61-4e9c-b2d7-1e57ebd22e64-dshm\") pod \"praxis-route-1e352e-kserve-67cb6ccfdc-k25pd\" (UID: \"b645db90-ca61-4e9c-b2d7-1e57ebd22e64\") " pod="ai-tenant-e2e-praxis-a-1e352e/praxis-route-1e352e-kserve-67cb6ccfdc-k25pd" Nov 03 01:01:43.579582 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:01:43.579561 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-26fmb\" (UniqueName: \"kubernetes.io/projected/b645db90-ca61-4e9c-b2d7-1e57ebd22e64-kube-api-access-26fmb\") pod \"praxis-route-1e352e-kserve-67cb6ccfdc-k25pd\" (UID: \"b645db90-ca61-4e9c-b2d7-1e57ebd22e64\") " pod="ai-tenant-e2e-praxis-a-1e352e/praxis-route-1e352e-kserve-67cb6ccfdc-k25pd" Nov 03 01:01:43.579737 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:01:43.579720 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b645db90-ca61-4e9c-b2d7-1e57ebd22e64-home\") pod \"praxis-route-1e352e-kserve-67cb6ccfdc-k25pd\" (UID: \"b645db90-ca61-4e9c-b2d7-1e57ebd22e64\") " pod="ai-tenant-e2e-praxis-a-1e352e/praxis-route-1e352e-kserve-67cb6ccfdc-k25pd" Nov 03 01:01:43.579962 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:01:43.579944 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b645db90-ca61-4e9c-b2d7-1e57ebd22e64-tmp-dir\") pod \"praxis-route-1e352e-kserve-67cb6ccfdc-k25pd\" (UID: \"b645db90-ca61-4e9c-b2d7-1e57ebd22e64\") " pod="ai-tenant-e2e-praxis-a-1e352e/praxis-route-1e352e-kserve-67cb6ccfdc-k25pd" Nov 03 01:01:43.580129 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:01:43.580114 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b645db90-ca61-4e9c-b2d7-1e57ebd22e64-tls-certs\") pod \"praxis-route-1e352e-kserve-67cb6ccfdc-k25pd\" (UID: \"b645db90-ca61-4e9c-b2d7-1e57ebd22e64\") " pod="ai-tenant-e2e-praxis-a-1e352e/praxis-route-1e352e-kserve-67cb6ccfdc-k25pd" Nov 03 01:01:43.580291 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:01:43.580273 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b645db90-ca61-4e9c-b2d7-1e57ebd22e64-model-cache\") pod \"praxis-route-1e352e-kserve-67cb6ccfdc-k25pd\" (UID: \"b645db90-ca61-4e9c-b2d7-1e57ebd22e64\") " pod="ai-tenant-e2e-praxis-a-1e352e/praxis-route-1e352e-kserve-67cb6ccfdc-k25pd" Nov 03 01:01:43.589767 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:01:43.589682 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:01:43.686751 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:01:43.686709 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b645db90-ca61-4e9c-b2d7-1e57ebd22e64-dshm\") pod \"praxis-route-1e352e-kserve-67cb6ccfdc-k25pd\" (UID: \"b645db90-ca61-4e9c-b2d7-1e57ebd22e64\") " pod="ai-tenant-e2e-praxis-a-1e352e/praxis-route-1e352e-kserve-67cb6ccfdc-k25pd" Nov 03 01:01:43.689319 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:01:43.689292 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-26fmb\" (UniqueName: \"kubernetes.io/projected/b645db90-ca61-4e9c-b2d7-1e57ebd22e64-kube-api-access-26fmb\") pod \"praxis-route-1e352e-kserve-67cb6ccfdc-k25pd\" (UID: \"b645db90-ca61-4e9c-b2d7-1e57ebd22e64\") " pod="ai-tenant-e2e-praxis-a-1e352e/praxis-route-1e352e-kserve-67cb6ccfdc-k25pd" Nov 03 01:01:43.695087 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:01:43.695019 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b645db90-ca61-4e9c-b2d7-1e57ebd22e64-home\") pod \"praxis-route-1e352e-kserve-67cb6ccfdc-k25pd\" (UID: \"b645db90-ca61-4e9c-b2d7-1e57ebd22e64\") " pod="ai-tenant-e2e-praxis-a-1e352e/praxis-route-1e352e-kserve-67cb6ccfdc-k25pd" Nov 03 01:01:43.701438 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:01:43.695878 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b645db90-ca61-4e9c-b2d7-1e57ebd22e64-tmp-dir\") pod \"praxis-route-1e352e-kserve-67cb6ccfdc-k25pd\" (UID: \"b645db90-ca61-4e9c-b2d7-1e57ebd22e64\") " pod="ai-tenant-e2e-praxis-a-1e352e/praxis-route-1e352e-kserve-67cb6ccfdc-k25pd" Nov 03 01:01:43.701670 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:01:43.695696 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b645db90-ca61-4e9c-b2d7-1e57ebd22e64-home\") pod \"praxis-route-1e352e-kserve-67cb6ccfdc-k25pd\" (UID: \"b645db90-ca61-4e9c-b2d7-1e57ebd22e64\") " pod="ai-tenant-e2e-praxis-a-1e352e/praxis-route-1e352e-kserve-67cb6ccfdc-k25pd" Nov 03 01:01:43.701809 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:01:43.700872 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b645db90-ca61-4e9c-b2d7-1e57ebd22e64-tmp-dir\") pod \"praxis-route-1e352e-kserve-67cb6ccfdc-k25pd\" (UID: \"b645db90-ca61-4e9c-b2d7-1e57ebd22e64\") " pod="ai-tenant-e2e-praxis-a-1e352e/praxis-route-1e352e-kserve-67cb6ccfdc-k25pd" Nov 03 01:01:43.702086 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:01:43.702068 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b645db90-ca61-4e9c-b2d7-1e57ebd22e64-tls-certs\") pod \"praxis-route-1e352e-kserve-67cb6ccfdc-k25pd\" (UID: \"b645db90-ca61-4e9c-b2d7-1e57ebd22e64\") " pod="ai-tenant-e2e-praxis-a-1e352e/praxis-route-1e352e-kserve-67cb6ccfdc-k25pd" Nov 03 01:01:43.702858 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:01:43.702837 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b645db90-ca61-4e9c-b2d7-1e57ebd22e64-model-cache\") pod \"praxis-route-1e352e-kserve-67cb6ccfdc-k25pd\" (UID: \"b645db90-ca61-4e9c-b2d7-1e57ebd22e64\") " pod="ai-tenant-e2e-praxis-a-1e352e/praxis-route-1e352e-kserve-67cb6ccfdc-k25pd" Nov 03 01:01:43.703427 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:01:43.703326 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b645db90-ca61-4e9c-b2d7-1e57ebd22e64-model-cache\") pod \"praxis-route-1e352e-kserve-67cb6ccfdc-k25pd\" (UID: \"b645db90-ca61-4e9c-b2d7-1e57ebd22e64\") " pod="ai-tenant-e2e-praxis-a-1e352e/praxis-route-1e352e-kserve-67cb6ccfdc-k25pd" Nov 03 01:01:43.757933 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:01:43.745882 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-26fmb\" (UniqueName: \"kubernetes.io/projected/b645db90-ca61-4e9c-b2d7-1e57ebd22e64-kube-api-access-26fmb\") pod \"praxis-route-1e352e-kserve-67cb6ccfdc-k25pd\" (UID: \"b645db90-ca61-4e9c-b2d7-1e57ebd22e64\") " pod="ai-tenant-e2e-praxis-a-1e352e/praxis-route-1e352e-kserve-67cb6ccfdc-k25pd" Nov 03 01:01:43.757933 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:01:43.750205 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b645db90-ca61-4e9c-b2d7-1e57ebd22e64-dshm\") pod \"praxis-route-1e352e-kserve-67cb6ccfdc-k25pd\" (UID: \"b645db90-ca61-4e9c-b2d7-1e57ebd22e64\") " pod="ai-tenant-e2e-praxis-a-1e352e/praxis-route-1e352e-kserve-67cb6ccfdc-k25pd" Nov 03 01:01:43.857933 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:01:43.857034 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b645db90-ca61-4e9c-b2d7-1e57ebd22e64-tls-certs\") pod \"praxis-route-1e352e-kserve-67cb6ccfdc-k25pd\" (UID: \"b645db90-ca61-4e9c-b2d7-1e57ebd22e64\") " pod="ai-tenant-e2e-praxis-a-1e352e/praxis-route-1e352e-kserve-67cb6ccfdc-k25pd" Nov 03 01:01:43.921298 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:01:43.920862 4939 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-praxis-a-1e352e/praxis-route-1e352e-kserve-67cb6ccfdc-k25pd" Nov 03 01:01:44.003752 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:01:44.003715 4939 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ffa6f2b0-be86-4f99-8d96-e4835139d9cd" path="/var/lib/kubelet/pods/ffa6f2b0-be86-4f99-8d96-e4835139d9cd/volumes" Nov 03 01:01:44.095903 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:01:44.095872 4939 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-f88f87-7b9d4c6697-mthjz"] Nov 03 01:01:44.096352 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:01:44.096084 4939 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-f88f87-7b9d4c6697-mthjz" podUID="81813dd5-11a4-45b8-990c-159f5f4359be" containerName="maas-api" containerID="cri-o://8ee3652007076d525b1302aeff97fb35d02540ff3c5a04660d497d7a611e1d59" gracePeriod=30 Nov 03 01:01:44.186106 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:01:44.186057 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:01:44.188720 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:01:44.187984 4939 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-f88f87-7b579c8df6-qvh6t"] Nov 03 01:01:44.188720 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:01:44.188215 4939 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-a-f88f87-7b579c8df6-qvh6t" podUID="0a6a1482-5067-471b-88f5-7dcbeb5323cc" containerName="payload-processing" containerID="cri-o://27ca8337f788b4df0699c115ad9a857f1b2b4524c5d71fbad893ff7666bb80ae" gracePeriod=30 Nov 03 01:01:44.193093 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:01:44.192662 4939 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-f88f87-67d5f888b7-nf6wt"] Nov 03 01:01:44.193093 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:01:44.192855 4939 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-f88f87-67d5f888b7-nf6wt" podUID="46a4dd33-f941-48f6-bdb7-fb2c057aa808" containerName="payload-pre-processing" containerID="cri-o://bda32303134b23caa4555af4dec7a2b0790f86e74c2e50f2e1846ace30f186a4" gracePeriod=30 Nov 03 01:01:44.292824 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:01:44.292745 4939 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-praxis-a-1e352e/praxis-route-1e352e-kserve-67cb6ccfdc-k25pd"] Nov 03 01:01:44.307295 ip-10-0-140-107 kubenswrapper[4939]: W1103 01:01:44.307228 4939 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podb645db90_ca61_4e9c_b2d7_1e57ebd22e64.slice/crio-58ef122cf732868ebea5f938c39a8163251a6358332dd848fbcafe22781256f7 WatchSource:0}: Error finding container 58ef122cf732868ebea5f938c39a8163251a6358332dd848fbcafe22781256f7: Status 404 returned error can't find the container with id 58ef122cf732868ebea5f938c39a8163251a6358332dd848fbcafe22781256f7 Nov 03 01:01:44.310935 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:01:44.310614 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:01:44.456775 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:01:44.456333 4939 patch_prober.go:29] interesting pod/e2e-ipp-b-02e4c0-openshift-default-c64cd65d9-7l2h4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.131:15021/healthz/ready\": dial tcp 10.132.0.131:15021: connect: connection refused" start-of-body= Nov 03 01:01:44.456775 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:01:44.456391 4939 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-02e4c0-openshift-default-c64cd65d9-7l2h4" podUID="2df5fd65-3b2a-448e-92a8-f07a8888692d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.131:15021/healthz/ready\": dial tcp 10.132.0.131:15021: connect: connection refused" Nov 03 01:01:44.570224 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:01:44.570185 4939 prober.go:122] "Probe failed" probeType="Readiness" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-f88f87-7b9d4c6697-mthjz" podUID="81813dd5-11a4-45b8-990c-159f5f4359be" containerName="maas-api" probeResult="failure" output="Get \"https://10.132.0.120:8443/health\": dial tcp 10.132.0.120:8443: connect: connection refused" Nov 03 01:01:44.740918 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:01:44.740752 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:01:44.995986 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:01:44.995957 4939 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-f88f87-7b9d4c6697-mthjz" Nov 03 01:01:45.070506 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:01:45.070473 4939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/81813dd5-11a4-45b8-990c-159f5f4359be-metrics-certs\" (UniqueName: \"kubernetes.io/secret/81813dd5-11a4-45b8-990c-159f5f4359be-metrics-certs\") pod \"81813dd5-11a4-45b8-990c-159f5f4359be\" (UID: \"81813dd5-11a4-45b8-990c-159f5f4359be\") " Nov 03 01:01:45.070998 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:01:45.070975 4939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/81813dd5-11a4-45b8-990c-159f5f4359be-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/81813dd5-11a4-45b8-990c-159f5f4359be-maas-api-tls\") pod \"81813dd5-11a4-45b8-990c-159f5f4359be\" (UID: \"81813dd5-11a4-45b8-990c-159f5f4359be\") " Nov 03 01:01:45.071615 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:01:45.071595 4939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/81813dd5-11a4-45b8-990c-159f5f4359be-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/81813dd5-11a4-45b8-990c-159f5f4359be-openshift-service-ca\") pod \"81813dd5-11a4-45b8-990c-159f5f4359be\" (UID: \"81813dd5-11a4-45b8-990c-159f5f4359be\") " Nov 03 01:01:45.072667 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:01:45.072646 4939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/81813dd5-11a4-45b8-990c-159f5f4359be-kube-api-access-794t9\" (UniqueName: \"kubernetes.io/projected/81813dd5-11a4-45b8-990c-159f5f4359be-kube-api-access-794t9\") pod \"81813dd5-11a4-45b8-990c-159f5f4359be\" (UID: \"81813dd5-11a4-45b8-990c-159f5f4359be\") " Nov 03 01:01:45.073711 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:01:45.072591 4939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/81813dd5-11a4-45b8-990c-159f5f4359be-openshift-service-ca" pod "81813dd5-11a4-45b8-990c-159f5f4359be" (UID: "81813dd5-11a4-45b8-990c-159f5f4359be"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 01:01:45.083021 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:01:45.079997 4939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/81813dd5-11a4-45b8-990c-159f5f4359be-kube-api-access-794t9" pod "81813dd5-11a4-45b8-990c-159f5f4359be" (UID: "81813dd5-11a4-45b8-990c-159f5f4359be"). InnerVolumeSpecName "kube-api-access-794t9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 01:01:45.083603 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:01:45.083566 4939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/81813dd5-11a4-45b8-990c-159f5f4359be-maas-api-tls" pod "81813dd5-11a4-45b8-990c-159f5f4359be" (UID: "81813dd5-11a4-45b8-990c-159f5f4359be"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 01:01:45.088056 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:01:45.088026 4939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/81813dd5-11a4-45b8-990c-159f5f4359be-metrics-certs" pod "81813dd5-11a4-45b8-990c-159f5f4359be" (UID: "81813dd5-11a4-45b8-990c-159f5f4359be"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 01:01:45.107707 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:01:45.107680 4939 generic.go:356] "Generic (PLEG): container finished" podID="0a6a1482-5067-471b-88f5-7dcbeb5323cc" containerID="27ca8337f788b4df0699c115ad9a857f1b2b4524c5d71fbad893ff7666bb80ae" exitCode=0 Nov 03 01:01:45.107997 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:01:45.107745 4939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-f88f87-7b579c8df6-qvh6t" event={"ID":"0a6a1482-5067-471b-88f5-7dcbeb5323cc","Type":"ContainerDied","Data":"27ca8337f788b4df0699c115ad9a857f1b2b4524c5d71fbad893ff7666bb80ae"} Nov 03 01:01:45.109097 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:01:45.109077 4939 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-f88f87-67d5f888b7-nf6wt" Nov 03 01:01:45.109833 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:01:45.109796 4939 generic.go:356] "Generic (PLEG): container finished" podID="46a4dd33-f941-48f6-bdb7-fb2c057aa808" containerID="bda32303134b23caa4555af4dec7a2b0790f86e74c2e50f2e1846ace30f186a4" exitCode=0 Nov 03 01:01:45.109999 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:01:45.109962 4939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-f88f87-67d5f888b7-nf6wt" event={"ID":"46a4dd33-f941-48f6-bdb7-fb2c057aa808","Type":"ContainerDied","Data":"bda32303134b23caa4555af4dec7a2b0790f86e74c2e50f2e1846ace30f186a4"} Nov 03 01:01:45.110082 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:01:45.110025 4939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-f88f87-67d5f888b7-nf6wt" event={"ID":"46a4dd33-f941-48f6-bdb7-fb2c057aa808","Type":"ContainerDied","Data":"4bdaf9e9d1b24e8e674f3244320020a0b2c82abdb209116bcbeecd548b98c6a8"} Nov 03 01:01:45.110159 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:01:45.110058 4939 scope.go:122] "RemoveContainer" containerID="bda32303134b23caa4555af4dec7a2b0790f86e74c2e50f2e1846ace30f186a4" Nov 03 01:01:45.113835 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:01:45.113775 4939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-praxis-a-1e352e/praxis-route-1e352e-kserve-67cb6ccfdc-k25pd" event={"ID":"b645db90-ca61-4e9c-b2d7-1e57ebd22e64","Type":"ContainerStarted","Data":"45978c90c580f947f1aa640e41cf6cb669f2364cbd5e79162f87b6058777ad4e"} Nov 03 01:01:45.113835 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:01:45.113814 4939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-praxis-a-1e352e/praxis-route-1e352e-kserve-67cb6ccfdc-k25pd" event={"ID":"b645db90-ca61-4e9c-b2d7-1e57ebd22e64","Type":"ContainerStarted","Data":"58ef122cf732868ebea5f938c39a8163251a6358332dd848fbcafe22781256f7"} Nov 03 01:01:45.115415 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:01:45.115391 4939 generic.go:356] "Generic (PLEG): container finished" podID="81813dd5-11a4-45b8-990c-159f5f4359be" containerID="8ee3652007076d525b1302aeff97fb35d02540ff3c5a04660d497d7a611e1d59" exitCode=0 Nov 03 01:01:45.115498 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:01:45.115428 4939 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-f88f87-7b9d4c6697-mthjz" Nov 03 01:01:45.115498 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:01:45.115466 4939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-f88f87-7b9d4c6697-mthjz" event={"ID":"81813dd5-11a4-45b8-990c-159f5f4359be","Type":"ContainerDied","Data":"8ee3652007076d525b1302aeff97fb35d02540ff3c5a04660d497d7a611e1d59"} Nov 03 01:01:45.115575 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:01:45.115505 4939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-f88f87-7b9d4c6697-mthjz" event={"ID":"81813dd5-11a4-45b8-990c-159f5f4359be","Type":"ContainerDied","Data":"bac7dd94540690381d62b7793b2697b129e6fb790d28266160d18e4420a1bee4"} Nov 03 01:01:45.115610 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:01:45.115597 4939 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-f88f87-7b579c8df6-qvh6t" Nov 03 01:01:45.122144 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:01:45.122120 4939 scope.go:122] "RemoveContainer" containerID="bda32303134b23caa4555af4dec7a2b0790f86e74c2e50f2e1846ace30f186a4" Nov 03 01:01:45.122490 ip-10-0-140-107 kubenswrapper[4939]: E1103 01:01:45.122463 4939 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bda32303134b23caa4555af4dec7a2b0790f86e74c2e50f2e1846ace30f186a4\": container with ID starting with bda32303134b23caa4555af4dec7a2b0790f86e74c2e50f2e1846ace30f186a4 not found: ID does not exist" containerID="bda32303134b23caa4555af4dec7a2b0790f86e74c2e50f2e1846ace30f186a4" Nov 03 01:01:45.122566 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:01:45.122499 4939 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bda32303134b23caa4555af4dec7a2b0790f86e74c2e50f2e1846ace30f186a4"} err="failed to get container status \"bda32303134b23caa4555af4dec7a2b0790f86e74c2e50f2e1846ace30f186a4\": rpc error: code = NotFound desc = could not find container \"bda32303134b23caa4555af4dec7a2b0790f86e74c2e50f2e1846ace30f186a4\": container with ID starting with bda32303134b23caa4555af4dec7a2b0790f86e74c2e50f2e1846ace30f186a4 not found: ID does not exist" Nov 03 01:01:45.122566 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:01:45.122540 4939 scope.go:122] "RemoveContainer" containerID="8ee3652007076d525b1302aeff97fb35d02540ff3c5a04660d497d7a611e1d59" Nov 03 01:01:45.179970 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:01:45.171323 4939 scope.go:122] "RemoveContainer" containerID="8ee3652007076d525b1302aeff97fb35d02540ff3c5a04660d497d7a611e1d59" Nov 03 01:01:45.179970 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:01:45.176022 4939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/46a4dd33-f941-48f6-bdb7-fb2c057aa808-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/46a4dd33-f941-48f6-bdb7-fb2c057aa808-plugins-config-volume\") pod \"46a4dd33-f941-48f6-bdb7-fb2c057aa808\" (UID: \"46a4dd33-f941-48f6-bdb7-fb2c057aa808\") " Nov 03 01:01:45.179970 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:01:45.176096 4939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0a6a1482-5067-471b-88f5-7dcbeb5323cc-kube-api-access-ctpl7\" (UniqueName: \"kubernetes.io/projected/0a6a1482-5067-471b-88f5-7dcbeb5323cc-kube-api-access-ctpl7\") pod \"0a6a1482-5067-471b-88f5-7dcbeb5323cc\" (UID: \"0a6a1482-5067-471b-88f5-7dcbeb5323cc\") " Nov 03 01:01:45.179970 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:01:45.176272 4939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/46a4dd33-f941-48f6-bdb7-fb2c057aa808-kube-api-access-qfxhv\" (UniqueName: \"kubernetes.io/projected/46a4dd33-f941-48f6-bdb7-fb2c057aa808-kube-api-access-qfxhv\") pod \"46a4dd33-f941-48f6-bdb7-fb2c057aa808\" (UID: \"46a4dd33-f941-48f6-bdb7-fb2c057aa808\") " Nov 03 01:01:45.179970 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:01:45.176344 4939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0a6a1482-5067-471b-88f5-7dcbeb5323cc-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0a6a1482-5067-471b-88f5-7dcbeb5323cc-plugins-config-volume\") pod \"0a6a1482-5067-471b-88f5-7dcbeb5323cc\" (UID: \"0a6a1482-5067-471b-88f5-7dcbeb5323cc\") " Nov 03 01:01:45.179970 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:01:45.176884 4939 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/81813dd5-11a4-45b8-990c-159f5f4359be-maas-api-tls\") on node \"ip-10-0-140-107.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:01:45.179970 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:01:45.176924 4939 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/81813dd5-11a4-45b8-990c-159f5f4359be-openshift-service-ca\") on node \"ip-10-0-140-107.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:01:45.179970 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:01:45.176939 4939 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-794t9\" (UniqueName: \"kubernetes.io/projected/81813dd5-11a4-45b8-990c-159f5f4359be-kube-api-access-794t9\") on node \"ip-10-0-140-107.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:01:45.179970 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:01:45.176954 4939 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/81813dd5-11a4-45b8-990c-159f5f4359be-metrics-certs\") on node \"ip-10-0-140-107.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:01:45.179970 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:01:45.178512 4939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0a6a1482-5067-471b-88f5-7dcbeb5323cc-plugins-config-volume" pod "0a6a1482-5067-471b-88f5-7dcbeb5323cc" (UID: "0a6a1482-5067-471b-88f5-7dcbeb5323cc"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 01:01:45.179970 ip-10-0-140-107 kubenswrapper[4939]: E1103 01:01:45.178881 4939 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8ee3652007076d525b1302aeff97fb35d02540ff3c5a04660d497d7a611e1d59\": container with ID starting with 8ee3652007076d525b1302aeff97fb35d02540ff3c5a04660d497d7a611e1d59 not found: ID does not exist" containerID="8ee3652007076d525b1302aeff97fb35d02540ff3c5a04660d497d7a611e1d59" Nov 03 01:01:45.179970 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:01:45.178968 4939 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8ee3652007076d525b1302aeff97fb35d02540ff3c5a04660d497d7a611e1d59"} err="failed to get container status \"8ee3652007076d525b1302aeff97fb35d02540ff3c5a04660d497d7a611e1d59\": rpc error: code = NotFound desc = could not find container \"8ee3652007076d525b1302aeff97fb35d02540ff3c5a04660d497d7a611e1d59\": container with ID starting with 8ee3652007076d525b1302aeff97fb35d02540ff3c5a04660d497d7a611e1d59 not found: ID does not exist" Nov 03 01:01:45.179970 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:01:45.179675 4939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/46a4dd33-f941-48f6-bdb7-fb2c057aa808-plugins-config-volume" pod "46a4dd33-f941-48f6-bdb7-fb2c057aa808" (UID: "46a4dd33-f941-48f6-bdb7-fb2c057aa808"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 01:01:45.180524 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:01:45.180260 4939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/46a4dd33-f941-48f6-bdb7-fb2c057aa808-kube-api-access-qfxhv" pod "46a4dd33-f941-48f6-bdb7-fb2c057aa808" (UID: "46a4dd33-f941-48f6-bdb7-fb2c057aa808"). InnerVolumeSpecName "kube-api-access-qfxhv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 01:01:45.181002 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:01:45.180980 4939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0a6a1482-5067-471b-88f5-7dcbeb5323cc-kube-api-access-ctpl7" pod "0a6a1482-5067-471b-88f5-7dcbeb5323cc" (UID: "0a6a1482-5067-471b-88f5-7dcbeb5323cc"). InnerVolumeSpecName "kube-api-access-ctpl7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 01:01:45.187965 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:01:45.187900 4939 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-praxis-a-1e352e/praxis-route-1e352e-kserve-67cb6ccfdc-k25pd" podStartSLOduration=2.187887223 podStartE2EDuration="2.187887223s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 01:01:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 01:01:45.187026866 +0000 UTC m=+1253.451361810" watchObservedRunningTime="2026-11-03 01:01:45.187887223 +0000 UTC m=+1253.452222208" Nov 03 01:01:45.225022 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:01:45.224997 4939 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-f88f87-7b9d4c6697-mthjz"] Nov 03 01:01:45.232530 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:01:45.232508 4939 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-f88f87-7b9d4c6697-mthjz"] Nov 03 01:01:45.278178 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:01:45.278142 4939 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ctpl7\" (UniqueName: \"kubernetes.io/projected/0a6a1482-5067-471b-88f5-7dcbeb5323cc-kube-api-access-ctpl7\") on node \"ip-10-0-140-107.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:01:45.278178 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:01:45.278171 4939 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qfxhv\" (UniqueName: \"kubernetes.io/projected/46a4dd33-f941-48f6-bdb7-fb2c057aa808-kube-api-access-qfxhv\") on node \"ip-10-0-140-107.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:01:45.278371 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:01:45.278188 4939 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0a6a1482-5067-471b-88f5-7dcbeb5323cc-plugins-config-volume\") on node \"ip-10-0-140-107.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:01:45.278371 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:01:45.278201 4939 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/46a4dd33-f941-48f6-bdb7-fb2c057aa808-plugins-config-volume\") on node \"ip-10-0-140-107.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:01:45.436818 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:01:45.436715 4939 patch_prober.go:29] interesting pod/e2e-ipp-b-02e4c0-openshift-default-c64cd65d9-7l2h4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.131:15021/healthz/ready\": dial tcp 10.132.0.131:15021: connect: connection refused" start-of-body= Nov 03 01:01:45.437006 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:01:45.436813 4939 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-02e4c0-openshift-default-c64cd65d9-7l2h4" podUID="2df5fd65-3b2a-448e-92a8-f07a8888692d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.131:15021/healthz/ready\": dial tcp 10.132.0.131:15021: connect: connection refused" Nov 03 01:01:45.780928 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:01:45.776567 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:01:45.970990 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:01:45.967484 4939 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="81813dd5-11a4-45b8-990c-159f5f4359be" path="/var/lib/kubelet/pods/81813dd5-11a4-45b8-990c-159f5f4359be/volumes" Nov 03 01:01:45.970990 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:01:45.968602 4939 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-praxis-a-1e352e/praxis-route-1e352e-kserve-67cb6ccfdc-k25pd" Nov 03 01:01:45.993868 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:01:45.992491 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:01:46.127787 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:01:46.127760 4939 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-f88f87-7b579c8df6-qvh6t" Nov 03 01:01:46.127787 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:01:46.127779 4939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-f88f87-7b579c8df6-qvh6t" event={"ID":"0a6a1482-5067-471b-88f5-7dcbeb5323cc","Type":"ContainerDied","Data":"cce2d25f4cbbb3d0294237926453b90c235f5ba0f8e40aeabc2ded2bbdddbc74"} Nov 03 01:01:46.128219 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:01:46.127839 4939 scope.go:122] "RemoveContainer" containerID="27ca8337f788b4df0699c115ad9a857f1b2b4524c5d71fbad893ff7666bb80ae" Nov 03 01:01:46.129456 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:01:46.129438 4939 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-f88f87-67d5f888b7-nf6wt" Nov 03 01:01:46.173232 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:01:46.173203 4939 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-f88f87-67d5f888b7-nf6wt"] Nov 03 01:01:46.176559 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:01:46.176533 4939 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-f88f87-67d5f888b7-nf6wt"] Nov 03 01:01:46.186472 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:01:46.186441 4939 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-f88f87-7b579c8df6-qvh6t"] Nov 03 01:01:46.192284 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:01:46.192259 4939 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-f88f87-7b579c8df6-qvh6t"] Nov 03 01:01:46.444933 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:01:46.444307 4939 patch_prober.go:29] interesting pod/e2e-ipp-b-02e4c0-openshift-default-c64cd65d9-7l2h4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.131:15021/healthz/ready\": dial tcp 10.132.0.131:15021: connect: connection refused" start-of-body= Nov 03 01:01:46.444933 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:01:46.444367 4939 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-02e4c0-openshift-default-c64cd65d9-7l2h4" podUID="2df5fd65-3b2a-448e-92a8-f07a8888692d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.131:15021/healthz/ready\": dial tcp 10.132.0.131:15021: connect: connection refused" Nov 03 01:01:47.171390 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:01:47.171358 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:01:47.435537 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:01:47.435442 4939 patch_prober.go:29] interesting pod/e2e-ipp-b-02e4c0-openshift-default-c64cd65d9-7l2h4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.131:15021/healthz/ready\": dial tcp 10.132.0.131:15021: connect: connection refused" start-of-body= Nov 03 01:01:47.435537 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:01:47.435511 4939 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-02e4c0-openshift-default-c64cd65d9-7l2h4" podUID="2df5fd65-3b2a-448e-92a8-f07a8888692d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.131:15021/healthz/ready\": dial tcp 10.132.0.131:15021: connect: connection refused" Nov 03 01:01:47.487949 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:01:47.487901 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:01:47.507357 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:01:47.507023 4939 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-praxis-b-46510c-openshift-default-849969b69c-9gxjp"] Nov 03 01:01:47.507819 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:01:47.507801 4939 cpu_manager.go:414] "containerMap: removing container" podUID="0a6a1482-5067-471b-88f5-7dcbeb5323cc" containerName="payload-processing" Nov 03 01:01:47.507819 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:01:47.507819 4939 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0a6a1482-5067-471b-88f5-7dcbeb5323cc" containerName="payload-processing" Nov 03 01:01:47.507990 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:01:47.507832 4939 cpu_manager.go:414] "containerMap: removing container" podUID="46a4dd33-f941-48f6-bdb7-fb2c057aa808" containerName="payload-pre-processing" Nov 03 01:01:47.507990 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:01:47.507837 4939 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="46a4dd33-f941-48f6-bdb7-fb2c057aa808" containerName="payload-pre-processing" Nov 03 01:01:47.507990 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:01:47.507857 4939 cpu_manager.go:414] "containerMap: removing container" podUID="81813dd5-11a4-45b8-990c-159f5f4359be" containerName="maas-api" Nov 03 01:01:47.507990 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:01:47.507862 4939 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="81813dd5-11a4-45b8-990c-159f5f4359be" containerName="maas-api" Nov 03 01:01:47.507990 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:01:47.507977 4939 memory_manager.go:365] "RemoveStaleState removing state" podUID="0a6a1482-5067-471b-88f5-7dcbeb5323cc" containerName="payload-processing" Nov 03 01:01:47.507990 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:01:47.507986 4939 memory_manager.go:365] "RemoveStaleState removing state" podUID="46a4dd33-f941-48f6-bdb7-fb2c057aa808" containerName="payload-pre-processing" Nov 03 01:01:47.508284 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:01:47.507997 4939 memory_manager.go:365] "RemoveStaleState removing state" podUID="81813dd5-11a4-45b8-990c-159f5f4359be" containerName="maas-api" Nov 03 01:01:47.512081 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:01:47.512057 4939 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-b-46510c-openshift-default-849969b69c-9gxjp" Nov 03 01:01:47.517645 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:01:47.517619 4939 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-praxis-b-46510c-openshift-default-849969b69c-9gxjp"] Nov 03 01:01:47.522243 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:01:47.522223 4939 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-praxis-b-46510c-openshift-default-dockercfg-lxshr\"" Nov 03 01:01:47.632692 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:01:47.630636 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:01:47.695983 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:01:47.694873 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a2aede6b-fd70-4df3-bec1-a716350a85c8-workload-socket\") pod \"e2e-praxis-b-46510c-openshift-default-849969b69c-9gxjp\" (UID: \"a2aede6b-fd70-4df3-bec1-a716350a85c8\") " pod="openshift-ingress/e2e-praxis-b-46510c-openshift-default-849969b69c-9gxjp" Nov 03 01:01:47.695983 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:01:47.694958 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a2aede6b-fd70-4df3-bec1-a716350a85c8-istio-podinfo\") pod \"e2e-praxis-b-46510c-openshift-default-849969b69c-9gxjp\" (UID: \"a2aede6b-fd70-4df3-bec1-a716350a85c8\") " pod="openshift-ingress/e2e-praxis-b-46510c-openshift-default-849969b69c-9gxjp" Nov 03 01:01:47.695983 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:01:47.694988 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a2aede6b-fd70-4df3-bec1-a716350a85c8-istiod-ca-cert\") pod \"e2e-praxis-b-46510c-openshift-default-849969b69c-9gxjp\" (UID: \"a2aede6b-fd70-4df3-bec1-a716350a85c8\") " pod="openshift-ingress/e2e-praxis-b-46510c-openshift-default-849969b69c-9gxjp" Nov 03 01:01:47.695983 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:01:47.695026 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nt29x\" (UniqueName: \"kubernetes.io/projected/a2aede6b-fd70-4df3-bec1-a716350a85c8-kube-api-access-nt29x\") pod \"e2e-praxis-b-46510c-openshift-default-849969b69c-9gxjp\" (UID: \"a2aede6b-fd70-4df3-bec1-a716350a85c8\") " pod="openshift-ingress/e2e-praxis-b-46510c-openshift-default-849969b69c-9gxjp" Nov 03 01:01:47.695983 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:01:47.695184 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a2aede6b-fd70-4df3-bec1-a716350a85c8-istio-token\") pod \"e2e-praxis-b-46510c-openshift-default-849969b69c-9gxjp\" (UID: \"a2aede6b-fd70-4df3-bec1-a716350a85c8\") " pod="openshift-ingress/e2e-praxis-b-46510c-openshift-default-849969b69c-9gxjp" Nov 03 01:01:47.695983 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:01:47.695269 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a2aede6b-fd70-4df3-bec1-a716350a85c8-istio-envoy\") pod \"e2e-praxis-b-46510c-openshift-default-849969b69c-9gxjp\" (UID: \"a2aede6b-fd70-4df3-bec1-a716350a85c8\") " pod="openshift-ingress/e2e-praxis-b-46510c-openshift-default-849969b69c-9gxjp" Nov 03 01:01:47.695983 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:01:47.695339 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a2aede6b-fd70-4df3-bec1-a716350a85c8-workload-certs\") pod \"e2e-praxis-b-46510c-openshift-default-849969b69c-9gxjp\" (UID: \"a2aede6b-fd70-4df3-bec1-a716350a85c8\") " pod="openshift-ingress/e2e-praxis-b-46510c-openshift-default-849969b69c-9gxjp" Nov 03 01:01:47.695983 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:01:47.695467 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a2aede6b-fd70-4df3-bec1-a716350a85c8-credential-socket\") pod \"e2e-praxis-b-46510c-openshift-default-849969b69c-9gxjp\" (UID: \"a2aede6b-fd70-4df3-bec1-a716350a85c8\") " pod="openshift-ingress/e2e-praxis-b-46510c-openshift-default-849969b69c-9gxjp" Nov 03 01:01:47.695983 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:01:47.695535 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a2aede6b-fd70-4df3-bec1-a716350a85c8-istio-data\") pod \"e2e-praxis-b-46510c-openshift-default-849969b69c-9gxjp\" (UID: \"a2aede6b-fd70-4df3-bec1-a716350a85c8\") " pod="openshift-ingress/e2e-praxis-b-46510c-openshift-default-849969b69c-9gxjp" Nov 03 01:01:47.799930 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:01:47.791726 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:01:47.799930 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:01:47.797032 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nt29x\" (UniqueName: \"kubernetes.io/projected/a2aede6b-fd70-4df3-bec1-a716350a85c8-kube-api-access-nt29x\") pod \"e2e-praxis-b-46510c-openshift-default-849969b69c-9gxjp\" (UID: \"a2aede6b-fd70-4df3-bec1-a716350a85c8\") " pod="openshift-ingress/e2e-praxis-b-46510c-openshift-default-849969b69c-9gxjp" Nov 03 01:01:47.799930 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:01:47.797245 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a2aede6b-fd70-4df3-bec1-a716350a85c8-istio-token\") pod \"e2e-praxis-b-46510c-openshift-default-849969b69c-9gxjp\" (UID: \"a2aede6b-fd70-4df3-bec1-a716350a85c8\") " pod="openshift-ingress/e2e-praxis-b-46510c-openshift-default-849969b69c-9gxjp" Nov 03 01:01:47.799930 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:01:47.797345 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a2aede6b-fd70-4df3-bec1-a716350a85c8-istio-envoy\") pod \"e2e-praxis-b-46510c-openshift-default-849969b69c-9gxjp\" (UID: \"a2aede6b-fd70-4df3-bec1-a716350a85c8\") " pod="openshift-ingress/e2e-praxis-b-46510c-openshift-default-849969b69c-9gxjp" Nov 03 01:01:47.799930 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:01:47.797424 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a2aede6b-fd70-4df3-bec1-a716350a85c8-workload-certs\") pod \"e2e-praxis-b-46510c-openshift-default-849969b69c-9gxjp\" (UID: \"a2aede6b-fd70-4df3-bec1-a716350a85c8\") " pod="openshift-ingress/e2e-praxis-b-46510c-openshift-default-849969b69c-9gxjp" Nov 03 01:01:47.799930 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:01:47.797494 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a2aede6b-fd70-4df3-bec1-a716350a85c8-credential-socket\") pod \"e2e-praxis-b-46510c-openshift-default-849969b69c-9gxjp\" (UID: \"a2aede6b-fd70-4df3-bec1-a716350a85c8\") " pod="openshift-ingress/e2e-praxis-b-46510c-openshift-default-849969b69c-9gxjp" Nov 03 01:01:47.799930 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:01:47.797568 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a2aede6b-fd70-4df3-bec1-a716350a85c8-istio-data\") pod \"e2e-praxis-b-46510c-openshift-default-849969b69c-9gxjp\" (UID: \"a2aede6b-fd70-4df3-bec1-a716350a85c8\") " pod="openshift-ingress/e2e-praxis-b-46510c-openshift-default-849969b69c-9gxjp" Nov 03 01:01:47.799930 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:01:47.797667 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a2aede6b-fd70-4df3-bec1-a716350a85c8-workload-socket\") pod \"e2e-praxis-b-46510c-openshift-default-849969b69c-9gxjp\" (UID: \"a2aede6b-fd70-4df3-bec1-a716350a85c8\") " pod="openshift-ingress/e2e-praxis-b-46510c-openshift-default-849969b69c-9gxjp" Nov 03 01:01:47.799930 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:01:47.797713 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a2aede6b-fd70-4df3-bec1-a716350a85c8-istio-podinfo\") pod \"e2e-praxis-b-46510c-openshift-default-849969b69c-9gxjp\" (UID: \"a2aede6b-fd70-4df3-bec1-a716350a85c8\") " pod="openshift-ingress/e2e-praxis-b-46510c-openshift-default-849969b69c-9gxjp" Nov 03 01:01:47.799930 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:01:47.797735 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a2aede6b-fd70-4df3-bec1-a716350a85c8-istiod-ca-cert\") pod \"e2e-praxis-b-46510c-openshift-default-849969b69c-9gxjp\" (UID: \"a2aede6b-fd70-4df3-bec1-a716350a85c8\") " pod="openshift-ingress/e2e-praxis-b-46510c-openshift-default-849969b69c-9gxjp" Nov 03 01:01:47.800606 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:01:47.800260 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a2aede6b-fd70-4df3-bec1-a716350a85c8-workload-certs\") pod \"e2e-praxis-b-46510c-openshift-default-849969b69c-9gxjp\" (UID: \"a2aede6b-fd70-4df3-bec1-a716350a85c8\") " pod="openshift-ingress/e2e-praxis-b-46510c-openshift-default-849969b69c-9gxjp" Nov 03 01:01:47.816412 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:01:47.800676 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a2aede6b-fd70-4df3-bec1-a716350a85c8-workload-socket\") pod \"e2e-praxis-b-46510c-openshift-default-849969b69c-9gxjp\" (UID: \"a2aede6b-fd70-4df3-bec1-a716350a85c8\") " pod="openshift-ingress/e2e-praxis-b-46510c-openshift-default-849969b69c-9gxjp" Nov 03 01:01:47.816412 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:01:47.815792 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a2aede6b-fd70-4df3-bec1-a716350a85c8-istio-data\") pod \"e2e-praxis-b-46510c-openshift-default-849969b69c-9gxjp\" (UID: \"a2aede6b-fd70-4df3-bec1-a716350a85c8\") " pod="openshift-ingress/e2e-praxis-b-46510c-openshift-default-849969b69c-9gxjp" Nov 03 01:01:47.848526 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:01:47.848483 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a2aede6b-fd70-4df3-bec1-a716350a85c8-credential-socket\") pod \"e2e-praxis-b-46510c-openshift-default-849969b69c-9gxjp\" (UID: \"a2aede6b-fd70-4df3-bec1-a716350a85c8\") " pod="openshift-ingress/e2e-praxis-b-46510c-openshift-default-849969b69c-9gxjp" Nov 03 01:01:47.849317 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:01:47.849283 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a2aede6b-fd70-4df3-bec1-a716350a85c8-istiod-ca-cert\") pod \"e2e-praxis-b-46510c-openshift-default-849969b69c-9gxjp\" (UID: \"a2aede6b-fd70-4df3-bec1-a716350a85c8\") " pod="openshift-ingress/e2e-praxis-b-46510c-openshift-default-849969b69c-9gxjp" Nov 03 01:01:47.858464 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:01:47.858378 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nt29x\" (UniqueName: \"kubernetes.io/projected/a2aede6b-fd70-4df3-bec1-a716350a85c8-kube-api-access-nt29x\") pod \"e2e-praxis-b-46510c-openshift-default-849969b69c-9gxjp\" (UID: \"a2aede6b-fd70-4df3-bec1-a716350a85c8\") " pod="openshift-ingress/e2e-praxis-b-46510c-openshift-default-849969b69c-9gxjp" Nov 03 01:01:47.973716 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:01:47.970372 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a2aede6b-fd70-4df3-bec1-a716350a85c8-istio-envoy\") pod \"e2e-praxis-b-46510c-openshift-default-849969b69c-9gxjp\" (UID: \"a2aede6b-fd70-4df3-bec1-a716350a85c8\") " pod="openshift-ingress/e2e-praxis-b-46510c-openshift-default-849969b69c-9gxjp" Nov 03 01:01:47.973716 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:01:47.972757 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a2aede6b-fd70-4df3-bec1-a716350a85c8-istio-podinfo\") pod \"e2e-praxis-b-46510c-openshift-default-849969b69c-9gxjp\" (UID: \"a2aede6b-fd70-4df3-bec1-a716350a85c8\") " pod="openshift-ingress/e2e-praxis-b-46510c-openshift-default-849969b69c-9gxjp" Nov 03 01:01:47.973716 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:01:47.973198 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a2aede6b-fd70-4df3-bec1-a716350a85c8-istio-token\") pod \"e2e-praxis-b-46510c-openshift-default-849969b69c-9gxjp\" (UID: \"a2aede6b-fd70-4df3-bec1-a716350a85c8\") " pod="openshift-ingress/e2e-praxis-b-46510c-openshift-default-849969b69c-9gxjp" Nov 03 01:01:48.008580 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:01:48.007575 4939 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0a6a1482-5067-471b-88f5-7dcbeb5323cc" path="/var/lib/kubelet/pods/0a6a1482-5067-471b-88f5-7dcbeb5323cc/volumes" Nov 03 01:01:48.009201 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:01:48.009160 4939 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="46a4dd33-f941-48f6-bdb7-fb2c057aa808" path="/var/lib/kubelet/pods/46a4dd33-f941-48f6-bdb7-fb2c057aa808/volumes" Nov 03 01:01:48.076628 ip-10-0-140-107 kubenswrapper[4939]: E1103 01:01:48.076588 4939 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-sthjh" containerName="extractor" Nov 03 01:01:48.130980 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:01:48.130238 4939 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-b-46510c-openshift-default-849969b69c-9gxjp" Nov 03 01:01:48.441646 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:01:48.441365 4939 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-b-02e4c0-openshift-default-c64cd65d9-7l2h4" Nov 03 01:01:48.444092 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:01:48.443757 4939 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-b-02e4c0-openshift-default-c64cd65d9-7l2h4" Nov 03 01:01:48.444236 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:01:48.444180 4939 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-b-02e4c0-openshift-default-c64cd65d9-7l2h4" Nov 03 01:01:48.540289 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:01:48.539246 4939 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-praxis-b-46510c-openshift-default-849969b69c-9gxjp"] Nov 03 01:01:48.603086 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:01:48.591554 4939 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 01:01:48.603086 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:01:48.591692 4939 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 01:01:48.603086 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:01:48.591744 4939 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 01:01:48.804933 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:01:48.802377 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:01:49.020931 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:01:49.006664 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:01:49.222887 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:01:49.222713 4939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-46510c-openshift-default-849969b69c-9gxjp" event={"ID":"a2aede6b-fd70-4df3-bec1-a716350a85c8","Type":"ContainerStarted","Data":"351ae7d796b0fdfb5006b2caca4fa98abf7cedf4b9ad8b37b1819eca7f3f372a"} Nov 03 01:01:49.222887 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:01:49.222780 4939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-46510c-openshift-default-849969b69c-9gxjp" event={"ID":"a2aede6b-fd70-4df3-bec1-a716350a85c8","Type":"ContainerStarted","Data":"e059e308e00831f2ca532c1bf8266aaf4519f70c7885c54028ad66b781b7b59d"} Nov 03 01:01:49.300935 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:01:49.275822 4939 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-praxis-b-46510c-openshift-default-849969b69c-9gxjp" podStartSLOduration=2.275803738 podStartE2EDuration="2.275803738s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 01:01:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 01:01:49.274574367 +0000 UTC m=+1257.538909316" watchObservedRunningTime="2026-11-03 01:01:49.275803738 +0000 UTC m=+1257.540138684" Nov 03 01:01:49.534983 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:01:49.528978 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:01:49.985933 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:01:49.978180 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:01:50.141930 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:01:50.136190 4939 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-praxis-b-46510c-openshift-default-849969b69c-9gxjp" Nov 03 01:01:50.326995 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:01:50.319798 4939 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-praxis-b-46510c-openshift-default-849969b69c-9gxjp" Nov 03 01:01:50.582994 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:01:50.582397 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:01:50.773934 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:01:50.773449 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:01:51.317266 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:01:51.317083 4939 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-praxis-b-46510c-openshift-default-849969b69c-9gxjp" Nov 03 01:01:51.318321 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:01:51.318293 4939 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-praxis-b-46510c-openshift-default-849969b69c-9gxjp" Nov 03 01:01:51.580984 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:01:51.580346 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:01:51.732929 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:01:51.731806 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:01:52.699930 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:01:52.699448 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:01:52.842028 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:01:52.842000 4939 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-55fbbbf9dc-qvj7w_7cab38b8-5158-4678-a429-350af5e2f72f/payload-processing/0.log" Nov 03 01:01:53.039108 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:01:53.039076 4939 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-55fbbbf9dc-qvj7w_7cab38b8-5158-4678-a429-350af5e2f72f/payload-processing/0.log" Nov 03 01:01:53.108644 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:01:53.108611 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:01:53.113115 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:01:53.109646 4939 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-46510c-d675d977f-ph287"] Nov 03 01:01:53.117492 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:01:53.117471 4939 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-46510c-d675d977f-ph287"] Nov 03 01:01:53.117650 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:01:53.117583 4939 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-46510c-d675d977f-ph287" Nov 03 01:01:53.124234 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:01:53.124207 4939 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-praxis-b-46510c\"" Nov 03 01:01:53.124675 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:01:53.124655 4939 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-praxis-b-46510c-dockercfg-6rlf5\"" Nov 03 01:01:53.136963 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:01:53.136890 4939 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-46510c-5cc495698c-94h2j"] Nov 03 01:01:53.141356 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:01:53.141331 4939 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-b-46510c-5cc495698c-94h2j" Nov 03 01:01:53.234894 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:01:53.215343 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/35fb0ddf-b9bf-4173-ae72-c0100ac754ad-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-b-46510c-d675d977f-ph287\" (UID: \"35fb0ddf-b9bf-4173-ae72-c0100ac754ad\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-46510c-d675d977f-ph287" Nov 03 01:01:53.234894 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:01:53.215445 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bmq6x\" (UniqueName: \"kubernetes.io/projected/35fb0ddf-b9bf-4173-ae72-c0100ac754ad-kube-api-access-bmq6x\") pod \"payload-pre-processing-e2e-praxis-b-46510c-d675d977f-ph287\" (UID: \"35fb0ddf-b9bf-4173-ae72-c0100ac754ad\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-46510c-d675d977f-ph287" Nov 03 01:01:53.346946 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:01:53.344576 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:01:53.468321 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:01:53.468232 4939 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-46510c-5cc495698c-94h2j"] Nov 03 01:01:53.536981 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:01:53.526069 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/35fb0ddf-b9bf-4173-ae72-c0100ac754ad-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-b-46510c-d675d977f-ph287\" (UID: \"35fb0ddf-b9bf-4173-ae72-c0100ac754ad\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-46510c-d675d977f-ph287" Nov 03 01:01:53.537268 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:01:53.537234 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j4xv9\" (UniqueName: \"kubernetes.io/projected/7bcdba6a-dab2-4616-82d1-2b0bf8770b8a-kube-api-access-j4xv9\") pod \"payload-processing-e2e-praxis-b-46510c-5cc495698c-94h2j\" (UID: \"7bcdba6a-dab2-4616-82d1-2b0bf8770b8a\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-46510c-5cc495698c-94h2j" Nov 03 01:01:53.552853 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:01:53.552812 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bmq6x\" (UniqueName: \"kubernetes.io/projected/35fb0ddf-b9bf-4173-ae72-c0100ac754ad-kube-api-access-bmq6x\") pod \"payload-pre-processing-e2e-praxis-b-46510c-d675d977f-ph287\" (UID: \"35fb0ddf-b9bf-4173-ae72-c0100ac754ad\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-46510c-d675d977f-ph287" Nov 03 01:01:53.553311 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:01:53.553287 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7bcdba6a-dab2-4616-82d1-2b0bf8770b8a-plugins-config-volume\") pod \"payload-processing-e2e-praxis-b-46510c-5cc495698c-94h2j\" (UID: \"7bcdba6a-dab2-4616-82d1-2b0bf8770b8a\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-46510c-5cc495698c-94h2j" Nov 03 01:01:53.623698 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:01:53.623646 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/35fb0ddf-b9bf-4173-ae72-c0100ac754ad-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-b-46510c-d675d977f-ph287\" (UID: \"35fb0ddf-b9bf-4173-ae72-c0100ac754ad\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-46510c-d675d977f-ph287" Nov 03 01:01:53.640748 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:01:53.640609 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bmq6x\" (UniqueName: \"kubernetes.io/projected/35fb0ddf-b9bf-4173-ae72-c0100ac754ad-kube-api-access-bmq6x\") pod \"payload-pre-processing-e2e-praxis-b-46510c-d675d977f-ph287\" (UID: \"35fb0ddf-b9bf-4173-ae72-c0100ac754ad\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-46510c-d675d977f-ph287" Nov 03 01:01:53.695077 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:01:53.695038 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j4xv9\" (UniqueName: \"kubernetes.io/projected/7bcdba6a-dab2-4616-82d1-2b0bf8770b8a-kube-api-access-j4xv9\") pod \"payload-processing-e2e-praxis-b-46510c-5cc495698c-94h2j\" (UID: \"7bcdba6a-dab2-4616-82d1-2b0bf8770b8a\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-46510c-5cc495698c-94h2j" Nov 03 01:01:53.697875 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:01:53.697848 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7bcdba6a-dab2-4616-82d1-2b0bf8770b8a-plugins-config-volume\") pod \"payload-processing-e2e-praxis-b-46510c-5cc495698c-94h2j\" (UID: \"7bcdba6a-dab2-4616-82d1-2b0bf8770b8a\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-46510c-5cc495698c-94h2j" Nov 03 01:01:53.698085 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:01:53.696517 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7bcdba6a-dab2-4616-82d1-2b0bf8770b8a-plugins-config-volume\") pod \"payload-processing-e2e-praxis-b-46510c-5cc495698c-94h2j\" (UID: \"7bcdba6a-dab2-4616-82d1-2b0bf8770b8a\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-46510c-5cc495698c-94h2j" Nov 03 01:01:53.705594 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:01:53.705545 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-j4xv9\" (UniqueName: \"kubernetes.io/projected/7bcdba6a-dab2-4616-82d1-2b0bf8770b8a-kube-api-access-j4xv9\") pod \"payload-processing-e2e-praxis-b-46510c-5cc495698c-94h2j\" (UID: \"7bcdba6a-dab2-4616-82d1-2b0bf8770b8a\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-46510c-5cc495698c-94h2j" Nov 03 01:01:53.736945 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:01:53.736685 4939 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-46510c-d675d977f-ph287" Nov 03 01:01:53.973360 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:01:53.973320 4939 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-b-46510c-5cc495698c-94h2j" Nov 03 01:01:54.023573 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:01:54.023529 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:01:54.147625 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:01:54.147594 4939 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-praxis-a-1e352e/praxis-route-1e352e-kserve-67cb6ccfdc-k25pd" Nov 03 01:01:54.147939 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:01:54.147899 4939 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-praxis-a-1e352e/praxis-route-1e352e-kserve-67cb6ccfdc-k25pd" Nov 03 01:01:54.167887 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:01:54.167766 4939 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-praxis-a-1e352e/praxis-route-1e352e-kserve-67cb6ccfdc-k25pd" Nov 03 01:01:54.353815 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:01:54.353783 4939 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-46510c-d675d977f-ph287"] Nov 03 01:01:54.354383 ip-10-0-140-107 kubenswrapper[4939]: W1103 01:01:54.354309 4939 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod35fb0ddf_b9bf_4173_ae72_c0100ac754ad.slice/crio-62d36f4c0ddeca6f9e016b059919994256a182262dd4cfc92661f0a966545264 WatchSource:0}: Error finding container 62d36f4c0ddeca6f9e016b059919994256a182262dd4cfc92661f0a966545264: Status 404 returned error can't find the container with id 62d36f4c0ddeca6f9e016b059919994256a182262dd4cfc92661f0a966545264 Nov 03 01:01:54.439139 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:01:54.439069 4939 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-46510c-5cc495698c-94h2j"] Nov 03 01:01:54.503164 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:01:54.503138 4939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-46510c-d675d977f-ph287" event={"ID":"35fb0ddf-b9bf-4173-ae72-c0100ac754ad","Type":"ContainerStarted","Data":"a161972dae9d5c2d0f423fd73962af3fe1601734033cdf81552413a9b9ad8cae"} Nov 03 01:01:54.503345 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:01:54.503329 4939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-46510c-d675d977f-ph287" event={"ID":"35fb0ddf-b9bf-4173-ae72-c0100ac754ad","Type":"ContainerStarted","Data":"62d36f4c0ddeca6f9e016b059919994256a182262dd4cfc92661f0a966545264"} Nov 03 01:01:54.503443 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:01:54.503433 4939 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-46510c-d675d977f-ph287" Nov 03 01:01:54.507402 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:01:54.507382 4939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-b-46510c-5cc495698c-94h2j" event={"ID":"7bcdba6a-dab2-4616-82d1-2b0bf8770b8a","Type":"ContainerStarted","Data":"bfe999920304164d52ac01f85355657fd0f72c1e8aca149fa59353c34bb0579a"} Nov 03 01:01:54.509753 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:01:54.509707 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:01:54.526589 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:01:54.526524 4939 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-46510c-d675d977f-ph287" podStartSLOduration=1.526502351 podStartE2EDuration="1.526502351s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 01:01:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 01:01:54.525036132 +0000 UTC m=+1262.789371089" watchObservedRunningTime="2026-11-03 01:01:54.526502351 +0000 UTC m=+1262.790837297" Nov 03 01:01:54.899340 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:01:54.899300 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:01:55.520006 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:01:55.519956 4939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-b-46510c-5cc495698c-94h2j" event={"ID":"7bcdba6a-dab2-4616-82d1-2b0bf8770b8a","Type":"ContainerStarted","Data":"453ddfee6a7ee44a0dd8929959c1674de6c70a77b625a2af0c3fad235acb7ae5"} Nov 03 01:01:55.520507 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:01:55.520483 4939 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-praxis-b-46510c-5cc495698c-94h2j" Nov 03 01:01:55.538837 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:01:55.538771 4939 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-praxis-b-46510c-5cc495698c-94h2j" podStartSLOduration=2.538751405 podStartE2EDuration="2.538751405s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 01:01:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 01:01:55.537771428 +0000 UTC m=+1263.802106374" watchObservedRunningTime="2026-11-03 01:01:55.538751405 +0000 UTC m=+1263.803086351" Nov 03 01:01:55.805031 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:01:55.804575 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:01:56.653831 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:01:56.653797 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:01:57.117333 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:01:57.117299 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:01:57.235229 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:01:57.235178 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:01:57.691340 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:01:57.691298 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:01:58.141545 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:01:58.141008 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:01:58.319536 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:01:58.318491 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:01:59.051951 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:01:59.040196 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:01:59.235875 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:01:59.235783 4939 patch_prober.go:29] interesting pod/e2e-praxis-a-d5f78f-openshift-default-64f48987d9-rk5hm container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.142:15021/healthz/ready\": dial tcp 10.132.0.142:15021: connect: connection refused" start-of-body= Nov 03 01:01:59.235875 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:01:59.235838 4939 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-praxis-a-d5f78f-openshift-default-64f48987d9-rk5hm" podUID="dfea3192-3541-49c5-a491-ebf7308b74e0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.142:15021/healthz/ready\": dial tcp 10.132.0.142:15021: connect: connection refused" Nov 03 01:01:59.437663 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:01:59.437625 4939 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-ipp-a-07f5f6/ipp-route-07f5f6-kserve-87874869c-4xwpw"] Nov 03 01:01:59.556300 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:01:59.556215 4939 generic.go:356] "Generic (PLEG): container finished" podID="dfea3192-3541-49c5-a491-ebf7308b74e0" containerID="e481325a728fa0455f38b868b5c3e86b20747ed84501736d04d1bc4267145bbe" exitCode=0 Nov 03 01:01:59.556300 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:01:59.556286 4939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-d5f78f-openshift-default-64f48987d9-rk5hm" event={"ID":"dfea3192-3541-49c5-a491-ebf7308b74e0","Type":"ContainerDied","Data":"e481325a728fa0455f38b868b5c3e86b20747ed84501736d04d1bc4267145bbe"} Nov 03 01:01:59.556811 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:01:59.556782 4939 scope.go:122] "RemoveContainer" containerID="e481325a728fa0455f38b868b5c3e86b20747ed84501736d04d1bc4267145bbe" Nov 03 01:01:59.558082 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:01:59.558064 4939 generic.go:356] "Generic (PLEG): container finished" podID="6c13fe13-2eda-4d7c-bf31-c522d7f3b9c3" containerID="24198b03b942e91379154ffbed0ce369eb1b7bea6f0a27a5caadcb65fe09cefd" exitCode=0 Nov 03 01:01:59.558177 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:01:59.558136 4939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-dc0f3f-openshift-default-d9f67c9bc-ks8t6" event={"ID":"6c13fe13-2eda-4d7c-bf31-c522d7f3b9c3","Type":"ContainerDied","Data":"24198b03b942e91379154ffbed0ce369eb1b7bea6f0a27a5caadcb65fe09cefd"} Nov 03 01:01:59.558515 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:01:59.558501 4939 scope.go:122] "RemoveContainer" containerID="24198b03b942e91379154ffbed0ce369eb1b7bea6f0a27a5caadcb65fe09cefd" Nov 03 01:01:59.558573 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:01:59.558548 4939 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 01:01:59.558645 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:01:59.558632 4939 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 01:01:59.558684 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:01:59.558668 4939 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 01:01:59.559965 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:01:59.559936 4939 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 01:01:59.560087 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:01:59.560013 4939 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 01:01:59.560087 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:01:59.560044 4939 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 01:01:59.592744 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:01:59.592714 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:02:00.119580 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:02:00.119494 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:02:00.608835 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:02:00.607748 4939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-dc0f3f-openshift-default-d9f67c9bc-ks8t6" event={"ID":"6c13fe13-2eda-4d7c-bf31-c522d7f3b9c3","Type":"ContainerStarted","Data":"e57037fbf8ed3717bc1c08ade218920cf420ae675039878ca1e37e672842c7b2"} Nov 03 01:02:00.610849 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:02:00.610825 4939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-d5f78f-openshift-default-64f48987d9-rk5hm" event={"ID":"dfea3192-3541-49c5-a491-ebf7308b74e0","Type":"ContainerStarted","Data":"dc8aa51c76110652e10097113c76c7c518140ac83f2a5a9cda0ace159a85b59a"} Nov 03 01:02:00.814349 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:02:00.814313 4939 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-4pw7v"] Nov 03 01:02:00.911293 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:02:00.911206 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:02:01.495932 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:02:01.495490 4939 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-praxis-a-d5f78f-openshift-default-64f48987d9-rk5hm" Nov 03 01:02:01.495932 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:02:01.495622 4939 patch_prober.go:29] interesting pod/e2e-praxis-a-d5f78f-openshift-default-64f48987d9-rk5hm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.142:15021/healthz/ready\": dial tcp 10.132.0.142:15021: connect: connection refused" start-of-body= Nov 03 01:02:01.495932 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:02:01.495671 4939 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-d5f78f-openshift-default-64f48987d9-rk5hm" podUID="dfea3192-3541-49c5-a491-ebf7308b74e0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.142:15021/healthz/ready\": dial tcp 10.132.0.142:15021: connect: connection refused" Nov 03 01:02:01.558931 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:02:01.555546 4939 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-a-dc0f3f-openshift-default-d9f67c9bc-ks8t6" Nov 03 01:02:01.558931 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:02:01.558607 4939 patch_prober.go:29] interesting pod/e2e-ipp-a-dc0f3f-openshift-default-d9f67c9bc-ks8t6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.137:15021/healthz/ready\": dial tcp 10.132.0.137:15021: connect: connection refused" start-of-body= Nov 03 01:02:01.558931 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:02:01.558675 4939 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-dc0f3f-openshift-default-d9f67c9bc-ks8t6" podUID="6c13fe13-2eda-4d7c-bf31-c522d7f3b9c3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.137:15021/healthz/ready\": dial tcp 10.132.0.137:15021: connect: connection refused" Nov 03 01:02:02.233422 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:02:02.233387 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:02:02.345708 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:02:02.345639 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:02:02.496583 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:02:02.496480 4939 patch_prober.go:29] interesting pod/e2e-praxis-a-d5f78f-openshift-default-64f48987d9-rk5hm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.142:15021/healthz/ready\": dial tcp 10.132.0.142:15021: connect: connection refused" start-of-body= Nov 03 01:02:02.497238 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:02:02.497187 4939 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-d5f78f-openshift-default-64f48987d9-rk5hm" podUID="dfea3192-3541-49c5-a491-ebf7308b74e0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.142:15021/healthz/ready\": dial tcp 10.132.0.142:15021: connect: connection refused" Nov 03 01:02:02.550321 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:02:02.550283 4939 patch_prober.go:29] interesting pod/e2e-ipp-a-dc0f3f-openshift-default-d9f67c9bc-ks8t6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.137:15021/healthz/ready\": dial tcp 10.132.0.137:15021: connect: connection refused" start-of-body= Nov 03 01:02:02.550601 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:02:02.550574 4939 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-dc0f3f-openshift-default-d9f67c9bc-ks8t6" podUID="6c13fe13-2eda-4d7c-bf31-c522d7f3b9c3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.137:15021/healthz/ready\": dial tcp 10.132.0.137:15021: connect: connection refused" Nov 03 01:02:02.933070 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:02:02.933032 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:02:03.497947 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:02:03.497308 4939 patch_prober.go:29] interesting pod/e2e-praxis-a-d5f78f-openshift-default-64f48987d9-rk5hm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.142:15021/healthz/ready\": dial tcp 10.132.0.142:15021: connect: connection refused" start-of-body= Nov 03 01:02:03.497947 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:02:03.497372 4939 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-d5f78f-openshift-default-64f48987d9-rk5hm" podUID="dfea3192-3541-49c5-a491-ebf7308b74e0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.142:15021/healthz/ready\": dial tcp 10.132.0.142:15021: connect: connection refused" Nov 03 01:02:03.546816 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:02:03.546778 4939 patch_prober.go:29] interesting pod/e2e-ipp-a-dc0f3f-openshift-default-d9f67c9bc-ks8t6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.137:15021/healthz/ready\": dial tcp 10.132.0.137:15021: connect: connection refused" start-of-body= Nov 03 01:02:03.550933 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:02:03.547116 4939 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-dc0f3f-openshift-default-d9f67c9bc-ks8t6" podUID="6c13fe13-2eda-4d7c-bf31-c522d7f3b9c3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.137:15021/healthz/ready\": dial tcp 10.132.0.137:15021: connect: connection refused" Nov 03 01:02:03.781102 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:02:03.781006 4939 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-a-dc0f3f-openshift-default-d9f67c9bc-ks8t6" Nov 03 01:02:03.914932 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:02:03.913720 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:02:04.276040 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:02:04.276005 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:02:04.500465 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:02:04.500418 4939 patch_prober.go:29] interesting pod/e2e-praxis-a-d5f78f-openshift-default-64f48987d9-rk5hm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.142:15021/healthz/ready\": dial tcp 10.132.0.142:15021: connect: connection refused" start-of-body= Nov 03 01:02:04.517044 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:02:04.509923 4939 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-d5f78f-openshift-default-64f48987d9-rk5hm" podUID="dfea3192-3541-49c5-a491-ebf7308b74e0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.142:15021/healthz/ready\": dial tcp 10.132.0.142:15021: connect: connection refused" Nov 03 01:02:04.572999 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:02:04.569601 4939 patch_prober.go:29] interesting pod/e2e-ipp-a-dc0f3f-openshift-default-d9f67c9bc-ks8t6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.137:15021/healthz/ready\": dial tcp 10.132.0.137:15021: connect: connection refused" start-of-body= Nov 03 01:02:04.572999 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:02:04.569677 4939 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-dc0f3f-openshift-default-d9f67c9bc-ks8t6" podUID="6c13fe13-2eda-4d7c-bf31-c522d7f3b9c3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.137:15021/healthz/ready\": dial tcp 10.132.0.137:15021: connect: connection refused" Nov 03 01:02:04.723246 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:02:04.723174 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:02:05.543139 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:02:05.543099 4939 patch_prober.go:29] interesting pod/e2e-praxis-a-d5f78f-openshift-default-64f48987d9-rk5hm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.142:15021/healthz/ready\": dial tcp 10.132.0.142:15021: connect: connection refused" start-of-body= Nov 03 01:02:05.543939 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:02:05.543885 4939 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-d5f78f-openshift-default-64f48987d9-rk5hm" podUID="dfea3192-3541-49c5-a491-ebf7308b74e0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.142:15021/healthz/ready\": dial tcp 10.132.0.142:15021: connect: connection refused" Nov 03 01:02:05.554569 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:02:05.554537 4939 patch_prober.go:29] interesting pod/e2e-ipp-a-dc0f3f-openshift-default-d9f67c9bc-ks8t6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.137:15021/healthz/ready\": dial tcp 10.132.0.137:15021: connect: connection refused" start-of-body= Nov 03 01:02:05.554814 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:02:05.554787 4939 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-dc0f3f-openshift-default-d9f67c9bc-ks8t6" podUID="6c13fe13-2eda-4d7c-bf31-c522d7f3b9c3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.137:15021/healthz/ready\": dial tcp 10.132.0.137:15021: connect: connection refused" Nov 03 01:02:05.565215 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:02:05.565184 4939 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-46510c-d675d977f-ph287" Nov 03 01:02:06.244612 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:02:06.244314 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:02:06.246078 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:02:06.246045 4939 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-46510c-77cb98ffcb-4x68b"] Nov 03 01:02:06.311950 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:02:06.303423 4939 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-46510c-77cb98ffcb-4x68b" Nov 03 01:02:06.333732 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:02:06.333700 4939 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-praxis-b-46510c-serving-cert\"" Nov 03 01:02:06.368482 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:02:06.368447 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:02:06.396185 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:02:06.396139 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rvk5w\" (UniqueName: \"kubernetes.io/projected/9cb13f72-a50c-49be-9490-fc347576d0d5-kube-api-access-rvk5w\") pod \"maas-api-e2e-praxis-b-46510c-77cb98ffcb-4x68b\" (UID: \"9cb13f72-a50c-49be-9490-fc347576d0d5\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-46510c-77cb98ffcb-4x68b" Nov 03 01:02:06.396537 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:02:06.396505 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9cb13f72-a50c-49be-9490-fc347576d0d5-maas-api-tls\") pod \"maas-api-e2e-praxis-b-46510c-77cb98ffcb-4x68b\" (UID: \"9cb13f72-a50c-49be-9490-fc347576d0d5\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-46510c-77cb98ffcb-4x68b" Nov 03 01:02:06.396776 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:02:06.396761 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9cb13f72-a50c-49be-9490-fc347576d0d5-metrics-certs\") pod \"maas-api-e2e-praxis-b-46510c-77cb98ffcb-4x68b\" (UID: \"9cb13f72-a50c-49be-9490-fc347576d0d5\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-46510c-77cb98ffcb-4x68b" Nov 03 01:02:06.397172 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:02:06.397147 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9cb13f72-a50c-49be-9490-fc347576d0d5-openshift-service-ca\") pod \"maas-api-e2e-praxis-b-46510c-77cb98ffcb-4x68b\" (UID: \"9cb13f72-a50c-49be-9490-fc347576d0d5\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-46510c-77cb98ffcb-4x68b" Nov 03 01:02:06.452411 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:02:06.435180 4939 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-46510c-77cb98ffcb-4x68b"] Nov 03 01:02:06.516145 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:02:06.516043 4939 patch_prober.go:29] interesting pod/e2e-praxis-a-d5f78f-openshift-default-64f48987d9-rk5hm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.142:15021/healthz/ready\": dial tcp 10.132.0.142:15021: connect: connection refused" start-of-body= Nov 03 01:02:06.516481 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:02:06.516447 4939 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-d5f78f-openshift-default-64f48987d9-rk5hm" podUID="dfea3192-3541-49c5-a491-ebf7308b74e0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.142:15021/healthz/ready\": dial tcp 10.132.0.142:15021: connect: connection refused" Nov 03 01:02:06.517440 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:02:06.517353 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9cb13f72-a50c-49be-9490-fc347576d0d5-openshift-service-ca\") pod \"maas-api-e2e-praxis-b-46510c-77cb98ffcb-4x68b\" (UID: \"9cb13f72-a50c-49be-9490-fc347576d0d5\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-46510c-77cb98ffcb-4x68b" Nov 03 01:02:06.517560 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:02:06.517471 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rvk5w\" (UniqueName: \"kubernetes.io/projected/9cb13f72-a50c-49be-9490-fc347576d0d5-kube-api-access-rvk5w\") pod \"maas-api-e2e-praxis-b-46510c-77cb98ffcb-4x68b\" (UID: \"9cb13f72-a50c-49be-9490-fc347576d0d5\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-46510c-77cb98ffcb-4x68b" Nov 03 01:02:06.517621 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:02:06.517579 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9cb13f72-a50c-49be-9490-fc347576d0d5-maas-api-tls\") pod \"maas-api-e2e-praxis-b-46510c-77cb98ffcb-4x68b\" (UID: \"9cb13f72-a50c-49be-9490-fc347576d0d5\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-46510c-77cb98ffcb-4x68b" Nov 03 01:02:06.517688 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:02:06.517674 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9cb13f72-a50c-49be-9490-fc347576d0d5-metrics-certs\") pod \"maas-api-e2e-praxis-b-46510c-77cb98ffcb-4x68b\" (UID: \"9cb13f72-a50c-49be-9490-fc347576d0d5\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-46510c-77cb98ffcb-4x68b" Nov 03 01:02:06.518761 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:02:06.518735 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9cb13f72-a50c-49be-9490-fc347576d0d5-openshift-service-ca\") pod \"maas-api-e2e-praxis-b-46510c-77cb98ffcb-4x68b\" (UID: \"9cb13f72-a50c-49be-9490-fc347576d0d5\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-46510c-77cb98ffcb-4x68b" Nov 03 01:02:06.542932 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:02:06.539287 4939 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-praxis-b-46510c-5cc495698c-94h2j" Nov 03 01:02:06.567213 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:02:06.565639 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rvk5w\" (UniqueName: \"kubernetes.io/projected/9cb13f72-a50c-49be-9490-fc347576d0d5-kube-api-access-rvk5w\") pod \"maas-api-e2e-praxis-b-46510c-77cb98ffcb-4x68b\" (UID: \"9cb13f72-a50c-49be-9490-fc347576d0d5\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-46510c-77cb98ffcb-4x68b" Nov 03 01:02:06.567213 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:02:06.565948 4939 patch_prober.go:29] interesting pod/e2e-ipp-a-dc0f3f-openshift-default-d9f67c9bc-ks8t6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.137:15021/healthz/ready\": dial tcp 10.132.0.137:15021: connect: connection refused" start-of-body= Nov 03 01:02:06.567213 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:02:06.565998 4939 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-dc0f3f-openshift-default-d9f67c9bc-ks8t6" podUID="6c13fe13-2eda-4d7c-bf31-c522d7f3b9c3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.137:15021/healthz/ready\": dial tcp 10.132.0.137:15021: connect: connection refused" Nov 03 01:02:06.585935 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:02:06.582383 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9cb13f72-a50c-49be-9490-fc347576d0d5-metrics-certs\") pod \"maas-api-e2e-praxis-b-46510c-77cb98ffcb-4x68b\" (UID: \"9cb13f72-a50c-49be-9490-fc347576d0d5\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-46510c-77cb98ffcb-4x68b" Nov 03 01:02:06.629991 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:02:06.629555 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9cb13f72-a50c-49be-9490-fc347576d0d5-maas-api-tls\") pod \"maas-api-e2e-praxis-b-46510c-77cb98ffcb-4x68b\" (UID: \"9cb13f72-a50c-49be-9490-fc347576d0d5\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-46510c-77cb98ffcb-4x68b" Nov 03 01:02:06.802740 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:02:06.802650 4939 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-46510c-77cb98ffcb-4x68b" Nov 03 01:02:07.509964 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:02:07.509904 4939 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-praxis-a-d5f78f-openshift-default-64f48987d9-rk5hm" Nov 03 01:02:07.511139 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:02:07.511107 4939 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-praxis-a-d5f78f-openshift-default-64f48987d9-rk5hm" Nov 03 01:02:07.513061 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:02:07.513038 4939 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-praxis-a-d5f78f-openshift-default-64f48987d9-rk5hm" Nov 03 01:02:07.514416 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:02:07.514392 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:02:07.525293 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:02:07.523970 4939 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-46510c-77cb98ffcb-4x68b"] Nov 03 01:02:07.540385 ip-10-0-140-107 kubenswrapper[4939]: W1103 01:02:07.531119 4939 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9cb13f72_a50c_49be_9490_fc347576d0d5.slice/crio-fff570da721e651ecc97a96ede9ac217465461b5b07aa1aad4fc6e7687276232 WatchSource:0}: Error finding container fff570da721e651ecc97a96ede9ac217465461b5b07aa1aad4fc6e7687276232: Status 404 returned error can't find the container with id fff570da721e651ecc97a96ede9ac217465461b5b07aa1aad4fc6e7687276232 Nov 03 01:02:07.665941 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:02:07.663689 4939 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-a-dc0f3f-openshift-default-d9f67c9bc-ks8t6" Nov 03 01:02:07.765643 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:02:07.765604 4939 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-a-dc0f3f-openshift-default-d9f67c9bc-ks8t6" Nov 03 01:02:08.207654 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:02:08.205664 4939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-46510c-77cb98ffcb-4x68b" event={"ID":"9cb13f72-a50c-49be-9490-fc347576d0d5","Type":"ContainerStarted","Data":"fff570da721e651ecc97a96ede9ac217465461b5b07aa1aad4fc6e7687276232"} Nov 03 01:02:08.403936 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:02:08.401754 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:02:09.227987 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:02:09.227504 4939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-46510c-77cb98ffcb-4x68b" event={"ID":"9cb13f72-a50c-49be-9490-fc347576d0d5","Type":"ContainerStarted","Data":"af773b53aac47ec47715af3ec4f5219c30f62bd2caed59ddfe2543bbfb7f0161"} Nov 03 01:02:09.227987 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:02:09.227633 4939 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-46510c-77cb98ffcb-4x68b" Nov 03 01:02:09.252031 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:02:09.251819 4939 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-46510c-77cb98ffcb-4x68b" podStartSLOduration=2.635823463 podStartE2EDuration="3.25180182s" totalImagesPullingTime="615.978357ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 01:02:06 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 01:02:09.249051008 +0000 UTC m=+1277.513385953" watchObservedRunningTime="2026-11-03 01:02:09.25180182 +0000 UTC m=+1277.516136769" Nov 03 01:02:09.283548 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:02:09.283516 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:02:10.132322 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:02:10.132278 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:02:10.899393 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:02:10.899350 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:02:11.898789 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:02:11.898756 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:02:12.409541 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:02:12.409501 4939 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-02e4c0-575bdc8b8c-ss7s5"] Nov 03 01:02:12.410059 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:02:12.409777 4939 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-02e4c0-575bdc8b8c-ss7s5" podUID="67809b22-2bbf-452b-84fc-b7711dd222d0" containerName="maas-api" containerID="cri-o://bb4edc4267f626e3d1d52e06442c0c92b9094c864504cb4c2c7987282da2c3eb" gracePeriod=30 Nov 03 01:02:12.451465 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:02:12.451434 4939 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-02e4c0-68d4d67f9b-ncxlh"] Nov 03 01:02:12.451684 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:02:12.451659 4939 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-b-02e4c0-68d4d67f9b-ncxlh" podUID="58ffd442-39d5-4e22-a8a2-1103aad2867e" containerName="payload-processing" containerID="cri-o://4682a5c130dfdba6a6212e0d52adfc98520ae7e010667691c3d8bbf7bc84c93f" gracePeriod=30 Nov 03 01:02:12.460680 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:02:12.460651 4939 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-02e4c0-597548c69f-ccdnj"] Nov 03 01:02:12.461932 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:02:12.460932 4939 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-02e4c0-597548c69f-ccdnj" podUID="ef1640af-2448-4ae7-b3e5-67a5113de620" containerName="payload-pre-processing" containerID="cri-o://3718cafe7a448d97a5568cfbc5104542f3276df70f76fa3482e2437f45743476" gracePeriod=30 Nov 03 01:02:12.466176 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:02:12.466148 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:02:12.584620 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:02:12.580816 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:02:13.038809 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:02:13.038780 4939 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-02e4c0-575bdc8b8c-ss7s5" Nov 03 01:02:13.090183 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:02:13.089888 4939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/67809b22-2bbf-452b-84fc-b7711dd222d0-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/67809b22-2bbf-452b-84fc-b7711dd222d0-openshift-service-ca\") pod \"67809b22-2bbf-452b-84fc-b7711dd222d0\" (UID: \"67809b22-2bbf-452b-84fc-b7711dd222d0\") " Nov 03 01:02:13.090183 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:02:13.089976 4939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/67809b22-2bbf-452b-84fc-b7711dd222d0-metrics-certs\" (UniqueName: \"kubernetes.io/secret/67809b22-2bbf-452b-84fc-b7711dd222d0-metrics-certs\") pod \"67809b22-2bbf-452b-84fc-b7711dd222d0\" (UID: \"67809b22-2bbf-452b-84fc-b7711dd222d0\") " Nov 03 01:02:13.090183 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:02:13.090037 4939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/67809b22-2bbf-452b-84fc-b7711dd222d0-kube-api-access-rzf6l\" (UniqueName: \"kubernetes.io/projected/67809b22-2bbf-452b-84fc-b7711dd222d0-kube-api-access-rzf6l\") pod \"67809b22-2bbf-452b-84fc-b7711dd222d0\" (UID: \"67809b22-2bbf-452b-84fc-b7711dd222d0\") " Nov 03 01:02:13.090183 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:02:13.090143 4939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/67809b22-2bbf-452b-84fc-b7711dd222d0-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/67809b22-2bbf-452b-84fc-b7711dd222d0-maas-api-tls\") pod \"67809b22-2bbf-452b-84fc-b7711dd222d0\" (UID: \"67809b22-2bbf-452b-84fc-b7711dd222d0\") " Nov 03 01:02:13.096554 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:02:13.096520 4939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/67809b22-2bbf-452b-84fc-b7711dd222d0-openshift-service-ca" pod "67809b22-2bbf-452b-84fc-b7711dd222d0" (UID: "67809b22-2bbf-452b-84fc-b7711dd222d0"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 01:02:13.097667 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:02:13.097641 4939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/67809b22-2bbf-452b-84fc-b7711dd222d0-maas-api-tls" pod "67809b22-2bbf-452b-84fc-b7711dd222d0" (UID: "67809b22-2bbf-452b-84fc-b7711dd222d0"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 01:02:13.098403 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:02:13.098378 4939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/67809b22-2bbf-452b-84fc-b7711dd222d0-metrics-certs" pod "67809b22-2bbf-452b-84fc-b7711dd222d0" (UID: "67809b22-2bbf-452b-84fc-b7711dd222d0"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 01:02:13.099547 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:02:13.099521 4939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/67809b22-2bbf-452b-84fc-b7711dd222d0-kube-api-access-rzf6l" pod "67809b22-2bbf-452b-84fc-b7711dd222d0" (UID: "67809b22-2bbf-452b-84fc-b7711dd222d0"). InnerVolumeSpecName "kube-api-access-rzf6l". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 01:02:13.126286 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:02:13.126208 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:02:13.145030 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:02:13.144998 4939 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-02e4c0-68d4d67f9b-ncxlh" Nov 03 01:02:13.204932 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:02:13.202480 4939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/58ffd442-39d5-4e22-a8a2-1103aad2867e-kube-api-access-6svcf\" (UniqueName: \"kubernetes.io/projected/58ffd442-39d5-4e22-a8a2-1103aad2867e-kube-api-access-6svcf\") pod \"58ffd442-39d5-4e22-a8a2-1103aad2867e\" (UID: \"58ffd442-39d5-4e22-a8a2-1103aad2867e\") " Nov 03 01:02:13.204932 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:02:13.202897 4939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/58ffd442-39d5-4e22-a8a2-1103aad2867e-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/58ffd442-39d5-4e22-a8a2-1103aad2867e-plugins-config-volume\") pod \"58ffd442-39d5-4e22-a8a2-1103aad2867e\" (UID: \"58ffd442-39d5-4e22-a8a2-1103aad2867e\") " Nov 03 01:02:13.204932 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:02:13.203694 4939 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/67809b22-2bbf-452b-84fc-b7711dd222d0-openshift-service-ca\") on node \"ip-10-0-140-107.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:02:13.204932 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:02:13.203721 4939 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/67809b22-2bbf-452b-84fc-b7711dd222d0-metrics-certs\") on node \"ip-10-0-140-107.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:02:13.204932 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:02:13.203738 4939 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rzf6l\" (UniqueName: \"kubernetes.io/projected/67809b22-2bbf-452b-84fc-b7711dd222d0-kube-api-access-rzf6l\") on node \"ip-10-0-140-107.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:02:13.204932 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:02:13.203752 4939 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/67809b22-2bbf-452b-84fc-b7711dd222d0-maas-api-tls\") on node \"ip-10-0-140-107.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:02:13.261965 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:02:13.259597 4939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/58ffd442-39d5-4e22-a8a2-1103aad2867e-plugins-config-volume" pod "58ffd442-39d5-4e22-a8a2-1103aad2867e" (UID: "58ffd442-39d5-4e22-a8a2-1103aad2867e"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 01:02:13.263992 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:02:13.263953 4939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/58ffd442-39d5-4e22-a8a2-1103aad2867e-kube-api-access-6svcf" pod "58ffd442-39d5-4e22-a8a2-1103aad2867e" (UID: "58ffd442-39d5-4e22-a8a2-1103aad2867e"). InnerVolumeSpecName "kube-api-access-6svcf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 01:02:13.278645 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:02:13.278621 4939 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-02e4c0-597548c69f-ccdnj" Nov 03 01:02:13.296808 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:02:13.296778 4939 generic.go:356] "Generic (PLEG): container finished" podID="58ffd442-39d5-4e22-a8a2-1103aad2867e" containerID="4682a5c130dfdba6a6212e0d52adfc98520ae7e010667691c3d8bbf7bc84c93f" exitCode=0 Nov 03 01:02:13.296958 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:02:13.296824 4939 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-02e4c0-68d4d67f9b-ncxlh" Nov 03 01:02:13.296958 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:02:13.296929 4939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-02e4c0-68d4d67f9b-ncxlh" event={"ID":"58ffd442-39d5-4e22-a8a2-1103aad2867e","Type":"ContainerDied","Data":"4682a5c130dfdba6a6212e0d52adfc98520ae7e010667691c3d8bbf7bc84c93f"} Nov 03 01:02:13.297100 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:02:13.297003 4939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-02e4c0-68d4d67f9b-ncxlh" event={"ID":"58ffd442-39d5-4e22-a8a2-1103aad2867e","Type":"ContainerDied","Data":"b2f32ac1ed6165834bd8e157ad8013528d0b1e9b00ea3a813e98d965aef9b1fe"} Nov 03 01:02:13.297100 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:02:13.297029 4939 scope.go:122] "RemoveContainer" containerID="4682a5c130dfdba6a6212e0d52adfc98520ae7e010667691c3d8bbf7bc84c93f" Nov 03 01:02:13.298724 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:02:13.298695 4939 generic.go:356] "Generic (PLEG): container finished" podID="67809b22-2bbf-452b-84fc-b7711dd222d0" containerID="bb4edc4267f626e3d1d52e06442c0c92b9094c864504cb4c2c7987282da2c3eb" exitCode=0 Nov 03 01:02:13.298835 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:02:13.298780 4939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-02e4c0-575bdc8b8c-ss7s5" event={"ID":"67809b22-2bbf-452b-84fc-b7711dd222d0","Type":"ContainerDied","Data":"bb4edc4267f626e3d1d52e06442c0c92b9094c864504cb4c2c7987282da2c3eb"} Nov 03 01:02:13.298903 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:02:13.298826 4939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-02e4c0-575bdc8b8c-ss7s5" event={"ID":"67809b22-2bbf-452b-84fc-b7711dd222d0","Type":"ContainerDied","Data":"cd9b365513a95b12ad97690a23fa36779b961cd7249d1915f1f681ef1b199f93"} Nov 03 01:02:13.298903 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:02:13.298896 4939 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-02e4c0-575bdc8b8c-ss7s5" Nov 03 01:02:13.311534 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:02:13.310312 4939 generic.go:356] "Generic (PLEG): container finished" podID="ef1640af-2448-4ae7-b3e5-67a5113de620" containerID="3718cafe7a448d97a5568cfbc5104542f3276df70f76fa3482e2437f45743476" exitCode=0 Nov 03 01:02:13.311534 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:02:13.310758 4939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-02e4c0-597548c69f-ccdnj" event={"ID":"ef1640af-2448-4ae7-b3e5-67a5113de620","Type":"ContainerDied","Data":"3718cafe7a448d97a5568cfbc5104542f3276df70f76fa3482e2437f45743476"} Nov 03 01:02:13.311534 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:02:13.310842 4939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-02e4c0-597548c69f-ccdnj" event={"ID":"ef1640af-2448-4ae7-b3e5-67a5113de620","Type":"ContainerDied","Data":"053c493b687c3457aef33e0db94225f6c0308ac3eb3432b557aef11bcf3f479d"} Nov 03 01:02:13.311534 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:02:13.310923 4939 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-02e4c0-597548c69f-ccdnj" Nov 03 01:02:13.311534 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:02:13.311308 4939 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/58ffd442-39d5-4e22-a8a2-1103aad2867e-plugins-config-volume\") on node \"ip-10-0-140-107.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:02:13.311534 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:02:13.311323 4939 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6svcf\" (UniqueName: \"kubernetes.io/projected/58ffd442-39d5-4e22-a8a2-1103aad2867e-kube-api-access-6svcf\") on node \"ip-10-0-140-107.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:02:13.314840 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:02:13.314818 4939 scope.go:122] "RemoveContainer" containerID="4682a5c130dfdba6a6212e0d52adfc98520ae7e010667691c3d8bbf7bc84c93f" Nov 03 01:02:13.316049 ip-10-0-140-107 kubenswrapper[4939]: E1103 01:02:13.316027 4939 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4682a5c130dfdba6a6212e0d52adfc98520ae7e010667691c3d8bbf7bc84c93f\": container with ID starting with 4682a5c130dfdba6a6212e0d52adfc98520ae7e010667691c3d8bbf7bc84c93f not found: ID does not exist" containerID="4682a5c130dfdba6a6212e0d52adfc98520ae7e010667691c3d8bbf7bc84c93f" Nov 03 01:02:13.316187 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:02:13.316164 4939 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4682a5c130dfdba6a6212e0d52adfc98520ae7e010667691c3d8bbf7bc84c93f"} err="failed to get container status \"4682a5c130dfdba6a6212e0d52adfc98520ae7e010667691c3d8bbf7bc84c93f\": rpc error: code = NotFound desc = could not find container \"4682a5c130dfdba6a6212e0d52adfc98520ae7e010667691c3d8bbf7bc84c93f\": container with ID starting with 4682a5c130dfdba6a6212e0d52adfc98520ae7e010667691c3d8bbf7bc84c93f not found: ID does not exist" Nov 03 01:02:13.316293 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:02:13.316281 4939 scope.go:122] "RemoveContainer" containerID="bb4edc4267f626e3d1d52e06442c0c92b9094c864504cb4c2c7987282da2c3eb" Nov 03 01:02:13.330387 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:02:13.330254 4939 scope.go:122] "RemoveContainer" containerID="bb4edc4267f626e3d1d52e06442c0c92b9094c864504cb4c2c7987282da2c3eb" Nov 03 01:02:13.330842 ip-10-0-140-107 kubenswrapper[4939]: E1103 01:02:13.330663 4939 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bb4edc4267f626e3d1d52e06442c0c92b9094c864504cb4c2c7987282da2c3eb\": container with ID starting with bb4edc4267f626e3d1d52e06442c0c92b9094c864504cb4c2c7987282da2c3eb not found: ID does not exist" containerID="bb4edc4267f626e3d1d52e06442c0c92b9094c864504cb4c2c7987282da2c3eb" Nov 03 01:02:13.330842 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:02:13.330737 4939 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bb4edc4267f626e3d1d52e06442c0c92b9094c864504cb4c2c7987282da2c3eb"} err="failed to get container status \"bb4edc4267f626e3d1d52e06442c0c92b9094c864504cb4c2c7987282da2c3eb\": rpc error: code = NotFound desc = could not find container \"bb4edc4267f626e3d1d52e06442c0c92b9094c864504cb4c2c7987282da2c3eb\": container with ID starting with bb4edc4267f626e3d1d52e06442c0c92b9094c864504cb4c2c7987282da2c3eb not found: ID does not exist" Nov 03 01:02:13.330842 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:02:13.330758 4939 scope.go:122] "RemoveContainer" containerID="3718cafe7a448d97a5568cfbc5104542f3276df70f76fa3482e2437f45743476" Nov 03 01:02:13.335731 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:02:13.335706 4939 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-02e4c0-575bdc8b8c-ss7s5"] Nov 03 01:02:13.342105 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:02:13.342082 4939 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-02e4c0-575bdc8b8c-ss7s5"] Nov 03 01:02:13.343673 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:02:13.343651 4939 scope.go:122] "RemoveContainer" containerID="3718cafe7a448d97a5568cfbc5104542f3276df70f76fa3482e2437f45743476" Nov 03 01:02:13.344012 ip-10-0-140-107 kubenswrapper[4939]: E1103 01:02:13.343990 4939 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3718cafe7a448d97a5568cfbc5104542f3276df70f76fa3482e2437f45743476\": container with ID starting with 3718cafe7a448d97a5568cfbc5104542f3276df70f76fa3482e2437f45743476 not found: ID does not exist" containerID="3718cafe7a448d97a5568cfbc5104542f3276df70f76fa3482e2437f45743476" Nov 03 01:02:13.344116 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:02:13.344017 4939 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3718cafe7a448d97a5568cfbc5104542f3276df70f76fa3482e2437f45743476"} err="failed to get container status \"3718cafe7a448d97a5568cfbc5104542f3276df70f76fa3482e2437f45743476\": rpc error: code = NotFound desc = could not find container \"3718cafe7a448d97a5568cfbc5104542f3276df70f76fa3482e2437f45743476\": container with ID starting with 3718cafe7a448d97a5568cfbc5104542f3276df70f76fa3482e2437f45743476 not found: ID does not exist" Nov 03 01:02:13.353300 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:02:13.353279 4939 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-02e4c0-68d4d67f9b-ncxlh"] Nov 03 01:02:13.361761 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:02:13.361739 4939 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-02e4c0-68d4d67f9b-ncxlh"] Nov 03 01:02:13.412581 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:02:13.412556 4939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ef1640af-2448-4ae7-b3e5-67a5113de620-kube-api-access-r5cdr\" (UniqueName: \"kubernetes.io/projected/ef1640af-2448-4ae7-b3e5-67a5113de620-kube-api-access-r5cdr\") pod \"ef1640af-2448-4ae7-b3e5-67a5113de620\" (UID: \"ef1640af-2448-4ae7-b3e5-67a5113de620\") " Nov 03 01:02:13.412993 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:02:13.412724 4939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ef1640af-2448-4ae7-b3e5-67a5113de620-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ef1640af-2448-4ae7-b3e5-67a5113de620-plugins-config-volume\") pod \"ef1640af-2448-4ae7-b3e5-67a5113de620\" (UID: \"ef1640af-2448-4ae7-b3e5-67a5113de620\") " Nov 03 01:02:13.413118 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:02:13.413096 4939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ef1640af-2448-4ae7-b3e5-67a5113de620-plugins-config-volume" pod "ef1640af-2448-4ae7-b3e5-67a5113de620" (UID: "ef1640af-2448-4ae7-b3e5-67a5113de620"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 01:02:13.413467 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:02:13.413451 4939 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ef1640af-2448-4ae7-b3e5-67a5113de620-plugins-config-volume\") on node \"ip-10-0-140-107.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:02:13.414633 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:02:13.414613 4939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ef1640af-2448-4ae7-b3e5-67a5113de620-kube-api-access-r5cdr" pod "ef1640af-2448-4ae7-b3e5-67a5113de620" (UID: "ef1640af-2448-4ae7-b3e5-67a5113de620"). InnerVolumeSpecName "kube-api-access-r5cdr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 01:02:13.514512 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:02:13.514436 4939 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-r5cdr\" (UniqueName: \"kubernetes.io/projected/ef1640af-2448-4ae7-b3e5-67a5113de620-kube-api-access-r5cdr\") on node \"ip-10-0-140-107.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:02:13.638951 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:02:13.638886 4939 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-02e4c0-597548c69f-ccdnj"] Nov 03 01:02:13.642326 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:02:13.642303 4939 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-02e4c0-597548c69f-ccdnj"] Nov 03 01:02:13.874968 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:02:13.872281 4939 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="58ffd442-39d5-4e22-a8a2-1103aad2867e" path="/var/lib/kubelet/pods/58ffd442-39d5-4e22-a8a2-1103aad2867e/volumes" Nov 03 01:02:13.874968 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:02:13.873077 4939 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="67809b22-2bbf-452b-84fc-b7711dd222d0" path="/var/lib/kubelet/pods/67809b22-2bbf-452b-84fc-b7711dd222d0/volumes" Nov 03 01:02:13.874968 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:02:13.873622 4939 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ef1640af-2448-4ae7-b3e5-67a5113de620" path="/var/lib/kubelet/pods/ef1640af-2448-4ae7-b3e5-67a5113de620/volumes" Nov 03 01:02:14.001026 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:02:14.000998 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:02:14.152512 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:02:14.152438 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:02:14.260847 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:02:14.260820 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:02:14.320654 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:02:14.320625 4939 generic.go:356] "Generic (PLEG): container finished" podID="1d33734b-7202-4184-9fc0-a58898f68802" containerID="e7bc838eb15997b3a1363ee712b7e1769fc4c35e00cdc7ec50b7801791a30994" exitCode=0 Nov 03 01:02:14.320834 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:02:14.320694 4939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-1e352e-openshift-default-75f8bbdbfc-gphv5" event={"ID":"1d33734b-7202-4184-9fc0-a58898f68802","Type":"ContainerDied","Data":"e7bc838eb15997b3a1363ee712b7e1769fc4c35e00cdc7ec50b7801791a30994"} Nov 03 01:02:14.321246 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:02:14.321225 4939 scope.go:122] "RemoveContainer" containerID="e7bc838eb15997b3a1363ee712b7e1769fc4c35e00cdc7ec50b7801791a30994" Nov 03 01:02:14.323191 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:02:14.323162 4939 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 01:02:14.323295 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:02:14.323248 4939 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 01:02:14.323295 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:02:14.323289 4939 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 01:02:14.470244 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:02:14.470006 4939 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-ipp-a-07f5f6/ipp-route-07f5f6-kserve-87874869c-4xwpw" podUID="21797ea7-0c52-407b-a34e-0af0aa92295a" containerName="main" containerID="cri-o://d25aad89b759b278c4d2a3fd1f9421e482817bfea1ba703712d7fed2c06de97a" gracePeriod=45 Nov 03 01:02:14.688134 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:02:14.688057 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:02:14.842922 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:02:14.842887 4939 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-ipp-a-07f5f6/ipp-route-07f5f6-kserve-87874869c-4xwpw" Nov 03 01:02:14.994164 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:02:14.994075 4939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/21797ea7-0c52-407b-a34e-0af0aa92295a-dshm\" (UniqueName: \"kubernetes.io/empty-dir/21797ea7-0c52-407b-a34e-0af0aa92295a-dshm\") pod \"21797ea7-0c52-407b-a34e-0af0aa92295a\" (UID: \"21797ea7-0c52-407b-a34e-0af0aa92295a\") " Nov 03 01:02:14.994164 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:02:14.994155 4939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/21797ea7-0c52-407b-a34e-0af0aa92295a-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/21797ea7-0c52-407b-a34e-0af0aa92295a-tmp-dir\") pod \"21797ea7-0c52-407b-a34e-0af0aa92295a\" (UID: \"21797ea7-0c52-407b-a34e-0af0aa92295a\") " Nov 03 01:02:14.994400 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:02:14.994371 4939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/21797ea7-0c52-407b-a34e-0af0aa92295a-home\" (UniqueName: \"kubernetes.io/empty-dir/21797ea7-0c52-407b-a34e-0af0aa92295a-home\") pod \"21797ea7-0c52-407b-a34e-0af0aa92295a\" (UID: \"21797ea7-0c52-407b-a34e-0af0aa92295a\") " Nov 03 01:02:14.994503 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:02:14.994473 4939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/21797ea7-0c52-407b-a34e-0af0aa92295a-tls-certs\" (UniqueName: \"kubernetes.io/secret/21797ea7-0c52-407b-a34e-0af0aa92295a-tls-certs\") pod \"21797ea7-0c52-407b-a34e-0af0aa92295a\" (UID: \"21797ea7-0c52-407b-a34e-0af0aa92295a\") " Nov 03 01:02:15.000053 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:02:14.995313 4939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/21797ea7-0c52-407b-a34e-0af0aa92295a-tmp-dir" pod "21797ea7-0c52-407b-a34e-0af0aa92295a" (UID: "21797ea7-0c52-407b-a34e-0af0aa92295a"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 01:02:15.000223 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:02:14.995624 4939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/21797ea7-0c52-407b-a34e-0af0aa92295a-home" pod "21797ea7-0c52-407b-a34e-0af0aa92295a" (UID: "21797ea7-0c52-407b-a34e-0af0aa92295a"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 01:02:15.000223 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:02:15.000056 4939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/21797ea7-0c52-407b-a34e-0af0aa92295a-kube-api-access-hnq6c\" (UniqueName: \"kubernetes.io/projected/21797ea7-0c52-407b-a34e-0af0aa92295a-kube-api-access-hnq6c\") pod \"21797ea7-0c52-407b-a34e-0af0aa92295a\" (UID: \"21797ea7-0c52-407b-a34e-0af0aa92295a\") " Nov 03 01:02:15.000223 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:02:15.000191 4939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/21797ea7-0c52-407b-a34e-0af0aa92295a-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/21797ea7-0c52-407b-a34e-0af0aa92295a-model-cache\") pod \"21797ea7-0c52-407b-a34e-0af0aa92295a\" (UID: \"21797ea7-0c52-407b-a34e-0af0aa92295a\") " Nov 03 01:02:15.000454 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:02:15.000433 4939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/21797ea7-0c52-407b-a34e-0af0aa92295a-model-cache" pod "21797ea7-0c52-407b-a34e-0af0aa92295a" (UID: "21797ea7-0c52-407b-a34e-0af0aa92295a"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 01:02:15.001312 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:02:15.001290 4939 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/21797ea7-0c52-407b-a34e-0af0aa92295a-home\") on node \"ip-10-0-140-107.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:02:15.001528 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:02:15.001511 4939 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/21797ea7-0c52-407b-a34e-0af0aa92295a-model-cache\") on node \"ip-10-0-140-107.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:02:15.001606 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:02:15.001531 4939 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/21797ea7-0c52-407b-a34e-0af0aa92295a-tmp-dir\") on node \"ip-10-0-140-107.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:02:15.003027 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:02:15.003003 4939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/21797ea7-0c52-407b-a34e-0af0aa92295a-dshm" pod "21797ea7-0c52-407b-a34e-0af0aa92295a" (UID: "21797ea7-0c52-407b-a34e-0af0aa92295a"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 01:02:15.003318 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:02:15.003293 4939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/21797ea7-0c52-407b-a34e-0af0aa92295a-tls-certs" pod "21797ea7-0c52-407b-a34e-0af0aa92295a" (UID: "21797ea7-0c52-407b-a34e-0af0aa92295a"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 01:02:15.003380 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:02:15.003367 4939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/21797ea7-0c52-407b-a34e-0af0aa92295a-kube-api-access-hnq6c" pod "21797ea7-0c52-407b-a34e-0af0aa92295a" (UID: "21797ea7-0c52-407b-a34e-0af0aa92295a"). InnerVolumeSpecName "kube-api-access-hnq6c". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 01:02:15.103188 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:02:15.103153 4939 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/21797ea7-0c52-407b-a34e-0af0aa92295a-tls-certs\") on node \"ip-10-0-140-107.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:02:15.103188 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:02:15.103179 4939 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hnq6c\" (UniqueName: \"kubernetes.io/projected/21797ea7-0c52-407b-a34e-0af0aa92295a-kube-api-access-hnq6c\") on node \"ip-10-0-140-107.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:02:15.103188 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:02:15.103189 4939 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/21797ea7-0c52-407b-a34e-0af0aa92295a-dshm\") on node \"ip-10-0-140-107.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:02:15.317159 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:02:15.317064 4939 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-ff4113-5487d8fb68-mf2c7"] Nov 03 01:02:15.317876 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:02:15.317850 4939 cpu_manager.go:414] "containerMap: removing container" podUID="67809b22-2bbf-452b-84fc-b7711dd222d0" containerName="maas-api" Nov 03 01:02:15.317876 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:02:15.317871 4939 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="67809b22-2bbf-452b-84fc-b7711dd222d0" containerName="maas-api" Nov 03 01:02:15.318071 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:02:15.317950 4939 cpu_manager.go:414] "containerMap: removing container" podUID="58ffd442-39d5-4e22-a8a2-1103aad2867e" containerName="payload-processing" Nov 03 01:02:15.318071 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:02:15.317960 4939 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="58ffd442-39d5-4e22-a8a2-1103aad2867e" containerName="payload-processing" Nov 03 01:02:15.318071 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:02:15.317979 4939 cpu_manager.go:414] "containerMap: removing container" podUID="ef1640af-2448-4ae7-b3e5-67a5113de620" containerName="payload-pre-processing" Nov 03 01:02:15.318071 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:02:15.317987 4939 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ef1640af-2448-4ae7-b3e5-67a5113de620" containerName="payload-pre-processing" Nov 03 01:02:15.318071 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:02:15.318002 4939 cpu_manager.go:414] "containerMap: removing container" podUID="21797ea7-0c52-407b-a34e-0af0aa92295a" containerName="main" Nov 03 01:02:15.318071 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:02:15.318009 4939 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="21797ea7-0c52-407b-a34e-0af0aa92295a" containerName="main" Nov 03 01:02:15.318363 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:02:15.318235 4939 memory_manager.go:365] "RemoveStaleState removing state" podUID="58ffd442-39d5-4e22-a8a2-1103aad2867e" containerName="payload-processing" Nov 03 01:02:15.318363 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:02:15.318252 4939 memory_manager.go:365] "RemoveStaleState removing state" podUID="ef1640af-2448-4ae7-b3e5-67a5113de620" containerName="payload-pre-processing" Nov 03 01:02:15.318363 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:02:15.318266 4939 memory_manager.go:365] "RemoveStaleState removing state" podUID="21797ea7-0c52-407b-a34e-0af0aa92295a" containerName="main" Nov 03 01:02:15.318363 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:02:15.318282 4939 memory_manager.go:365] "RemoveStaleState removing state" podUID="67809b22-2bbf-452b-84fc-b7711dd222d0" containerName="maas-api" Nov 03 01:02:15.322084 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:02:15.322064 4939 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-ff4113-5487d8fb68-mf2c7" Nov 03 01:02:15.333077 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:02:15.332936 4939 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ipp-b-ff4113-serving-cert\"" Nov 03 01:02:15.343475 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:02:15.343443 4939 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-ff4113-5487d8fb68-mf2c7"] Nov 03 01:02:15.392483 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:02:15.392440 4939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-1e352e-openshift-default-75f8bbdbfc-gphv5" event={"ID":"1d33734b-7202-4184-9fc0-a58898f68802","Type":"ContainerStarted","Data":"7eca4f7934c25f38045b05e2902a777f3bc239975a80c5c8412eaab6b3c068ea"} Nov 03 01:02:15.394882 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:02:15.394852 4939 generic.go:356] "Generic (PLEG): container finished" podID="21797ea7-0c52-407b-a34e-0af0aa92295a" containerID="d25aad89b759b278c4d2a3fd1f9421e482817bfea1ba703712d7fed2c06de97a" exitCode=0 Nov 03 01:02:15.395025 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:02:15.394933 4939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-ipp-a-07f5f6/ipp-route-07f5f6-kserve-87874869c-4xwpw" event={"ID":"21797ea7-0c52-407b-a34e-0af0aa92295a","Type":"ContainerDied","Data":"d25aad89b759b278c4d2a3fd1f9421e482817bfea1ba703712d7fed2c06de97a"} Nov 03 01:02:15.395025 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:02:15.394966 4939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-ipp-a-07f5f6/ipp-route-07f5f6-kserve-87874869c-4xwpw" event={"ID":"21797ea7-0c52-407b-a34e-0af0aa92295a","Type":"ContainerDied","Data":"845b7ff36b7f3e507a800c691a9d3350a9708131d3ee17b0f039834195c9b2fc"} Nov 03 01:02:15.395025 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:02:15.394981 4939 scope.go:122] "RemoveContainer" containerID="d25aad89b759b278c4d2a3fd1f9421e482817bfea1ba703712d7fed2c06de97a" Nov 03 01:02:15.395247 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:02:15.395229 4939 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-ipp-a-07f5f6/ipp-route-07f5f6-kserve-87874869c-4xwpw" Nov 03 01:02:15.406717 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:02:15.406701 4939 scope.go:122] "RemoveContainer" containerID="d25aad89b759b278c4d2a3fd1f9421e482817bfea1ba703712d7fed2c06de97a" Nov 03 01:02:15.407015 ip-10-0-140-107 kubenswrapper[4939]: E1103 01:02:15.406991 4939 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d25aad89b759b278c4d2a3fd1f9421e482817bfea1ba703712d7fed2c06de97a\": container with ID starting with d25aad89b759b278c4d2a3fd1f9421e482817bfea1ba703712d7fed2c06de97a not found: ID does not exist" containerID="d25aad89b759b278c4d2a3fd1f9421e482817bfea1ba703712d7fed2c06de97a" Nov 03 01:02:15.407125 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:02:15.407018 4939 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d25aad89b759b278c4d2a3fd1f9421e482817bfea1ba703712d7fed2c06de97a"} err="failed to get container status \"d25aad89b759b278c4d2a3fd1f9421e482817bfea1ba703712d7fed2c06de97a\": rpc error: code = NotFound desc = could not find container \"d25aad89b759b278c4d2a3fd1f9421e482817bfea1ba703712d7fed2c06de97a\": container with ID starting with d25aad89b759b278c4d2a3fd1f9421e482817bfea1ba703712d7fed2c06de97a not found: ID does not exist" Nov 03 01:02:15.416064 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:02:15.416043 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:02:15.509996 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:02:15.509947 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/cf320aba-5bf7-42fe-a4a9-603130676712-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-ff4113-5487d8fb68-mf2c7\" (UID: \"cf320aba-5bf7-42fe-a4a9-603130676712\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-ff4113-5487d8fb68-mf2c7" Nov 03 01:02:15.510556 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:02:15.510401 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hlr9z\" (UniqueName: \"kubernetes.io/projected/cf320aba-5bf7-42fe-a4a9-603130676712-kube-api-access-hlr9z\") pod \"maas-api-e2e-ipp-b-ff4113-5487d8fb68-mf2c7\" (UID: \"cf320aba-5bf7-42fe-a4a9-603130676712\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-ff4113-5487d8fb68-mf2c7" Nov 03 01:02:15.510556 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:02:15.510495 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/cf320aba-5bf7-42fe-a4a9-603130676712-maas-api-tls\") pod \"maas-api-e2e-ipp-b-ff4113-5487d8fb68-mf2c7\" (UID: \"cf320aba-5bf7-42fe-a4a9-603130676712\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-ff4113-5487d8fb68-mf2c7" Nov 03 01:02:15.510684 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:02:15.510587 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cf320aba-5bf7-42fe-a4a9-603130676712-metrics-certs\") pod \"maas-api-e2e-ipp-b-ff4113-5487d8fb68-mf2c7\" (UID: \"cf320aba-5bf7-42fe-a4a9-603130676712\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-ff4113-5487d8fb68-mf2c7" Nov 03 01:02:15.534939 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:02:15.534887 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:02:15.594644 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:02:15.589998 4939 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-ipp-a-07f5f6/ipp-route-07f5f6-kserve-87874869c-4xwpw"] Nov 03 01:02:15.612385 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:02:15.612352 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hlr9z\" (UniqueName: \"kubernetes.io/projected/cf320aba-5bf7-42fe-a4a9-603130676712-kube-api-access-hlr9z\") pod \"maas-api-e2e-ipp-b-ff4113-5487d8fb68-mf2c7\" (UID: \"cf320aba-5bf7-42fe-a4a9-603130676712\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-ff4113-5487d8fb68-mf2c7" Nov 03 01:02:15.612719 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:02:15.612688 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/cf320aba-5bf7-42fe-a4a9-603130676712-maas-api-tls\") pod \"maas-api-e2e-ipp-b-ff4113-5487d8fb68-mf2c7\" (UID: \"cf320aba-5bf7-42fe-a4a9-603130676712\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-ff4113-5487d8fb68-mf2c7" Nov 03 01:02:15.612960 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:02:15.612940 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cf320aba-5bf7-42fe-a4a9-603130676712-metrics-certs\") pod \"maas-api-e2e-ipp-b-ff4113-5487d8fb68-mf2c7\" (UID: \"cf320aba-5bf7-42fe-a4a9-603130676712\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-ff4113-5487d8fb68-mf2c7" Nov 03 01:02:15.613160 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:02:15.613140 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/cf320aba-5bf7-42fe-a4a9-603130676712-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-ff4113-5487d8fb68-mf2c7\" (UID: \"cf320aba-5bf7-42fe-a4a9-603130676712\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-ff4113-5487d8fb68-mf2c7" Nov 03 01:02:15.613899 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:02:15.613878 4939 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-ipp-a-07f5f6/ipp-route-07f5f6-kserve-87874869c-4xwpw"] Nov 03 01:02:15.614777 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:02:15.614758 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/cf320aba-5bf7-42fe-a4a9-603130676712-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-ff4113-5487d8fb68-mf2c7\" (UID: \"cf320aba-5bf7-42fe-a4a9-603130676712\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-ff4113-5487d8fb68-mf2c7" Nov 03 01:02:15.623375 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:02:15.623355 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hlr9z\" (UniqueName: \"kubernetes.io/projected/cf320aba-5bf7-42fe-a4a9-603130676712-kube-api-access-hlr9z\") pod \"maas-api-e2e-ipp-b-ff4113-5487d8fb68-mf2c7\" (UID: \"cf320aba-5bf7-42fe-a4a9-603130676712\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-ff4113-5487d8fb68-mf2c7" Nov 03 01:02:15.624717 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:02:15.624678 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/cf320aba-5bf7-42fe-a4a9-603130676712-maas-api-tls\") pod \"maas-api-e2e-ipp-b-ff4113-5487d8fb68-mf2c7\" (UID: \"cf320aba-5bf7-42fe-a4a9-603130676712\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-ff4113-5487d8fb68-mf2c7" Nov 03 01:02:15.626330 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:02:15.626294 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cf320aba-5bf7-42fe-a4a9-603130676712-metrics-certs\") pod \"maas-api-e2e-ipp-b-ff4113-5487d8fb68-mf2c7\" (UID: \"cf320aba-5bf7-42fe-a4a9-603130676712\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-ff4113-5487d8fb68-mf2c7" Nov 03 01:02:15.642158 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:02:15.642119 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:02:15.686564 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:02:15.686525 4939 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-ff4113-5487d8fb68-mf2c7" Nov 03 01:02:15.883675 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:02:15.883552 4939 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="21797ea7-0c52-407b-a34e-0af0aa92295a" path="/var/lib/kubelet/pods/21797ea7-0c52-407b-a34e-0af0aa92295a/volumes" Nov 03 01:02:15.996278 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:02:15.996250 4939 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-ff4113-5487d8fb68-mf2c7"] Nov 03 01:02:15.998173 ip-10-0-140-107 kubenswrapper[4939]: W1103 01:02:15.998146 4939 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcf320aba_5bf7_42fe_a4a9_603130676712.slice/crio-7606705ca630c6fe6761aea69d9a2c3a88ba0a143018e8206d92b1dee1acd39a WatchSource:0}: Error finding container 7606705ca630c6fe6761aea69d9a2c3a88ba0a143018e8206d92b1dee1acd39a: Status 404 returned error can't find the container with id 7606705ca630c6fe6761aea69d9a2c3a88ba0a143018e8206d92b1dee1acd39a Nov 03 01:02:16.295583 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:02:16.295504 4939 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-46510c-77cb98ffcb-4x68b" Nov 03 01:02:16.306582 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:02:16.306553 4939 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-praxis-a-1e352e-openshift-default-75f8bbdbfc-gphv5" Nov 03 01:02:16.306697 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:02:16.306652 4939 patch_prober.go:29] interesting pod/e2e-praxis-a-1e352e-openshift-default-75f8bbdbfc-gphv5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.130:15021/healthz/ready\": dial tcp 10.132.0.130:15021: connect: connection refused" start-of-body= Nov 03 01:02:16.306740 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:02:16.306693 4939 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-1e352e-openshift-default-75f8bbdbfc-gphv5" podUID="1d33734b-7202-4184-9fc0-a58898f68802" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.130:15021/healthz/ready\": dial tcp 10.132.0.130:15021: connect: connection refused" Nov 03 01:02:16.404316 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:02:16.404280 4939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-ff4113-5487d8fb68-mf2c7" event={"ID":"cf320aba-5bf7-42fe-a4a9-603130676712","Type":"ContainerStarted","Data":"7606705ca630c6fe6761aea69d9a2c3a88ba0a143018e8206d92b1dee1acd39a"} Nov 03 01:02:16.437475 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:02:16.437447 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:02:17.232095 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:02:17.232061 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:02:17.305937 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:02:17.305801 4939 patch_prober.go:29] interesting pod/e2e-praxis-a-1e352e-openshift-default-75f8bbdbfc-gphv5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.130:15021/healthz/ready\": dial tcp 10.132.0.130:15021: connect: connection refused" start-of-body= Nov 03 01:02:17.305937 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:02:17.305867 4939 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-1e352e-openshift-default-75f8bbdbfc-gphv5" podUID="1d33734b-7202-4184-9fc0-a58898f68802" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.130:15021/healthz/ready\": dial tcp 10.132.0.130:15021: connect: connection refused" Nov 03 01:02:17.413454 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:02:17.413418 4939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-ff4113-5487d8fb68-mf2c7" event={"ID":"cf320aba-5bf7-42fe-a4a9-603130676712","Type":"ContainerStarted","Data":"dd4fd811c9b0d73743b448b01c8dcc5095c37c8e5c64c8278451e5cfe5e8ccc0"} Nov 03 01:02:17.413663 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:02:17.413570 4939 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-ff4113-5487d8fb68-mf2c7" Nov 03 01:02:17.434232 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:02:17.434188 4939 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-ff4113-5487d8fb68-mf2c7" podStartSLOduration=1.829231625 podStartE2EDuration="2.434172384s" totalImagesPullingTime="604.940759ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 01:02:15 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 01:02:17.43038461 +0000 UTC m=+1285.694719555" watchObservedRunningTime="2026-11-03 01:02:17.434172384 +0000 UTC m=+1285.698507330" Nov 03 01:02:17.918701 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:02:17.918669 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:02:18.108325 ip-10-0-140-107 kubenswrapper[4939]: E1103 01:02:18.108290 4939 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-sthjh" containerName="extractor" Nov 03 01:02:18.305582 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:02:18.305500 4939 patch_prober.go:29] interesting pod/e2e-praxis-a-1e352e-openshift-default-75f8bbdbfc-gphv5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.130:15021/healthz/ready\": dial tcp 10.132.0.130:15021: connect: connection refused" start-of-body= Nov 03 01:02:18.305582 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:02:18.305553 4939 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-1e352e-openshift-default-75f8bbdbfc-gphv5" podUID="1d33734b-7202-4184-9fc0-a58898f68802" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.130:15021/healthz/ready\": dial tcp 10.132.0.130:15021: connect: connection refused" Nov 03 01:02:18.314902 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:02:18.314876 4939 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-praxis-a-1e352e-openshift-default-75f8bbdbfc-gphv5" Nov 03 01:02:18.727329 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:02:18.727296 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:02:19.321936 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:02:19.312310 4939 patch_prober.go:29] interesting pod/e2e-praxis-a-1e352e-openshift-default-75f8bbdbfc-gphv5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.130:15021/healthz/ready\": dial tcp 10.132.0.130:15021: connect: connection refused" start-of-body= Nov 03 01:02:19.321936 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:02:19.312372 4939 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-1e352e-openshift-default-75f8bbdbfc-gphv5" podUID="1d33734b-7202-4184-9fc0-a58898f68802" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.130:15021/healthz/ready\": dial tcp 10.132.0.130:15021: connect: connection refused" Nov 03 01:02:19.680676 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:02:19.680639 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:02:20.306212 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:02:20.306168 4939 patch_prober.go:29] interesting pod/e2e-praxis-a-1e352e-openshift-default-75f8bbdbfc-gphv5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.130:15021/healthz/ready\": dial tcp 10.132.0.130:15021: connect: connection refused" start-of-body= Nov 03 01:02:20.306419 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:02:20.306238 4939 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-1e352e-openshift-default-75f8bbdbfc-gphv5" podUID="1d33734b-7202-4184-9fc0-a58898f68802" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.130:15021/healthz/ready\": dial tcp 10.132.0.130:15021: connect: connection refused" Nov 03 01:02:20.514022 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:02:20.513981 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:02:21.329710 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:02:21.328085 4939 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-praxis-a-1e352e-openshift-default-75f8bbdbfc-gphv5" Nov 03 01:02:21.331703 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:02:21.331679 4939 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-praxis-a-1e352e-openshift-default-75f8bbdbfc-gphv5" Nov 03 01:02:22.244366 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:02:22.244329 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:02:22.476962 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:02:22.476904 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:02:23.004679 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:02:23.004622 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:02:23.122730 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:02:23.122699 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:02:23.426850 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:02:23.426822 4939 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-ff4113-5487d8fb68-mf2c7" Nov 03 01:02:23.440625 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:02:23.440592 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:02:23.678360 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:02:23.678284 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:02:24.018195 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:02:24.018111 4939 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-02e4c0-openshift-default-c64cd65d9-7l2h4"] Nov 03 01:02:24.018491 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:02:24.018453 4939 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-b-02e4c0-openshift-default-c64cd65d9-7l2h4" podUID="2df5fd65-3b2a-448e-92a8-f07a8888692d" containerName="istio-proxy" containerID="cri-o://153561948d7bc4acf758d26763ac2b0e3a6e4165173e85bf546f7b0b4314fd27" gracePeriod=30 Nov 03 01:02:24.075442 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:02:24.075411 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:02:24.237934 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:02:24.227768 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:02:25.254943 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:02:25.251477 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:02:25.707838 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:02:25.707800 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:02:25.891456 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:02:25.891421 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:02:26.079201 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:02:26.079120 4939 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-praxis-a-b9dad7-openshift-default-994bfcbb9-bw5wc"] Nov 03 01:02:26.086059 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:02:26.086027 4939 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-a-b9dad7-openshift-default-994bfcbb9-bw5wc" Nov 03 01:02:26.094749 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:02:26.094726 4939 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-praxis-a-b9dad7-openshift-default-dockercfg-dklzf\"" Nov 03 01:02:26.097609 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:02:26.097575 4939 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-praxis-a-b9dad7-openshift-default-994bfcbb9-bw5wc"] Nov 03 01:02:26.117295 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:02:26.117263 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:02:26.123807 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:02:26.123779 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/40f81e1f-3df1-4802-9636-b2ab2263f0ae-istio-podinfo\") pod \"e2e-praxis-a-b9dad7-openshift-default-994bfcbb9-bw5wc\" (UID: \"40f81e1f-3df1-4802-9636-b2ab2263f0ae\") " pod="openshift-ingress/e2e-praxis-a-b9dad7-openshift-default-994bfcbb9-bw5wc" Nov 03 01:02:26.123992 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:02:26.123963 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/40f81e1f-3df1-4802-9636-b2ab2263f0ae-istio-data\") pod \"e2e-praxis-a-b9dad7-openshift-default-994bfcbb9-bw5wc\" (UID: \"40f81e1f-3df1-4802-9636-b2ab2263f0ae\") " pod="openshift-ingress/e2e-praxis-a-b9dad7-openshift-default-994bfcbb9-bw5wc" Nov 03 01:02:26.124097 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:02:26.124076 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/40f81e1f-3df1-4802-9636-b2ab2263f0ae-credential-socket\") pod \"e2e-praxis-a-b9dad7-openshift-default-994bfcbb9-bw5wc\" (UID: \"40f81e1f-3df1-4802-9636-b2ab2263f0ae\") " pod="openshift-ingress/e2e-praxis-a-b9dad7-openshift-default-994bfcbb9-bw5wc" Nov 03 01:02:26.124170 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:02:26.124130 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/40f81e1f-3df1-4802-9636-b2ab2263f0ae-istio-envoy\") pod \"e2e-praxis-a-b9dad7-openshift-default-994bfcbb9-bw5wc\" (UID: \"40f81e1f-3df1-4802-9636-b2ab2263f0ae\") " pod="openshift-ingress/e2e-praxis-a-b9dad7-openshift-default-994bfcbb9-bw5wc" Nov 03 01:02:26.124236 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:02:26.124173 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/40f81e1f-3df1-4802-9636-b2ab2263f0ae-istio-token\") pod \"e2e-praxis-a-b9dad7-openshift-default-994bfcbb9-bw5wc\" (UID: \"40f81e1f-3df1-4802-9636-b2ab2263f0ae\") " pod="openshift-ingress/e2e-praxis-a-b9dad7-openshift-default-994bfcbb9-bw5wc" Nov 03 01:02:26.124302 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:02:26.124235 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/40f81e1f-3df1-4802-9636-b2ab2263f0ae-workload-certs\") pod \"e2e-praxis-a-b9dad7-openshift-default-994bfcbb9-bw5wc\" (UID: \"40f81e1f-3df1-4802-9636-b2ab2263f0ae\") " pod="openshift-ingress/e2e-praxis-a-b9dad7-openshift-default-994bfcbb9-bw5wc" Nov 03 01:02:26.124302 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:02:26.124274 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/40f81e1f-3df1-4802-9636-b2ab2263f0ae-workload-socket\") pod \"e2e-praxis-a-b9dad7-openshift-default-994bfcbb9-bw5wc\" (UID: \"40f81e1f-3df1-4802-9636-b2ab2263f0ae\") " pod="openshift-ingress/e2e-praxis-a-b9dad7-openshift-default-994bfcbb9-bw5wc" Nov 03 01:02:26.124411 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:02:26.124300 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/40f81e1f-3df1-4802-9636-b2ab2263f0ae-istiod-ca-cert\") pod \"e2e-praxis-a-b9dad7-openshift-default-994bfcbb9-bw5wc\" (UID: \"40f81e1f-3df1-4802-9636-b2ab2263f0ae\") " pod="openshift-ingress/e2e-praxis-a-b9dad7-openshift-default-994bfcbb9-bw5wc" Nov 03 01:02:26.124480 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:02:26.124462 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2wjcn\" (UniqueName: \"kubernetes.io/projected/40f81e1f-3df1-4802-9636-b2ab2263f0ae-kube-api-access-2wjcn\") pod \"e2e-praxis-a-b9dad7-openshift-default-994bfcbb9-bw5wc\" (UID: \"40f81e1f-3df1-4802-9636-b2ab2263f0ae\") " pod="openshift-ingress/e2e-praxis-a-b9dad7-openshift-default-994bfcbb9-bw5wc" Nov 03 01:02:26.234934 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:02:26.234860 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/40f81e1f-3df1-4802-9636-b2ab2263f0ae-credential-socket\") pod \"e2e-praxis-a-b9dad7-openshift-default-994bfcbb9-bw5wc\" (UID: \"40f81e1f-3df1-4802-9636-b2ab2263f0ae\") " pod="openshift-ingress/e2e-praxis-a-b9dad7-openshift-default-994bfcbb9-bw5wc" Nov 03 01:02:26.235146 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:02:26.234953 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/40f81e1f-3df1-4802-9636-b2ab2263f0ae-istio-envoy\") pod \"e2e-praxis-a-b9dad7-openshift-default-994bfcbb9-bw5wc\" (UID: \"40f81e1f-3df1-4802-9636-b2ab2263f0ae\") " pod="openshift-ingress/e2e-praxis-a-b9dad7-openshift-default-994bfcbb9-bw5wc" Nov 03 01:02:26.235146 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:02:26.235029 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/40f81e1f-3df1-4802-9636-b2ab2263f0ae-istio-token\") pod \"e2e-praxis-a-b9dad7-openshift-default-994bfcbb9-bw5wc\" (UID: \"40f81e1f-3df1-4802-9636-b2ab2263f0ae\") " pod="openshift-ingress/e2e-praxis-a-b9dad7-openshift-default-994bfcbb9-bw5wc" Nov 03 01:02:26.235146 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:02:26.235061 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/40f81e1f-3df1-4802-9636-b2ab2263f0ae-workload-certs\") pod \"e2e-praxis-a-b9dad7-openshift-default-994bfcbb9-bw5wc\" (UID: \"40f81e1f-3df1-4802-9636-b2ab2263f0ae\") " pod="openshift-ingress/e2e-praxis-a-b9dad7-openshift-default-994bfcbb9-bw5wc" Nov 03 01:02:26.235146 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:02:26.235098 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/40f81e1f-3df1-4802-9636-b2ab2263f0ae-workload-socket\") pod \"e2e-praxis-a-b9dad7-openshift-default-994bfcbb9-bw5wc\" (UID: \"40f81e1f-3df1-4802-9636-b2ab2263f0ae\") " pod="openshift-ingress/e2e-praxis-a-b9dad7-openshift-default-994bfcbb9-bw5wc" Nov 03 01:02:26.235146 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:02:26.235124 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/40f81e1f-3df1-4802-9636-b2ab2263f0ae-istiod-ca-cert\") pod \"e2e-praxis-a-b9dad7-openshift-default-994bfcbb9-bw5wc\" (UID: \"40f81e1f-3df1-4802-9636-b2ab2263f0ae\") " pod="openshift-ingress/e2e-praxis-a-b9dad7-openshift-default-994bfcbb9-bw5wc" Nov 03 01:02:26.235441 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:02:26.235203 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2wjcn\" (UniqueName: \"kubernetes.io/projected/40f81e1f-3df1-4802-9636-b2ab2263f0ae-kube-api-access-2wjcn\") pod \"e2e-praxis-a-b9dad7-openshift-default-994bfcbb9-bw5wc\" (UID: \"40f81e1f-3df1-4802-9636-b2ab2263f0ae\") " pod="openshift-ingress/e2e-praxis-a-b9dad7-openshift-default-994bfcbb9-bw5wc" Nov 03 01:02:26.235441 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:02:26.235339 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/40f81e1f-3df1-4802-9636-b2ab2263f0ae-istio-podinfo\") pod \"e2e-praxis-a-b9dad7-openshift-default-994bfcbb9-bw5wc\" (UID: \"40f81e1f-3df1-4802-9636-b2ab2263f0ae\") " pod="openshift-ingress/e2e-praxis-a-b9dad7-openshift-default-994bfcbb9-bw5wc" Nov 03 01:02:26.235561 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:02:26.235490 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/40f81e1f-3df1-4802-9636-b2ab2263f0ae-istio-data\") pod \"e2e-praxis-a-b9dad7-openshift-default-994bfcbb9-bw5wc\" (UID: \"40f81e1f-3df1-4802-9636-b2ab2263f0ae\") " pod="openshift-ingress/e2e-praxis-a-b9dad7-openshift-default-994bfcbb9-bw5wc" Nov 03 01:02:26.238935 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:02:26.236035 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/40f81e1f-3df1-4802-9636-b2ab2263f0ae-istio-data\") pod \"e2e-praxis-a-b9dad7-openshift-default-994bfcbb9-bw5wc\" (UID: \"40f81e1f-3df1-4802-9636-b2ab2263f0ae\") " pod="openshift-ingress/e2e-praxis-a-b9dad7-openshift-default-994bfcbb9-bw5wc" Nov 03 01:02:26.238935 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:02:26.236321 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/40f81e1f-3df1-4802-9636-b2ab2263f0ae-credential-socket\") pod \"e2e-praxis-a-b9dad7-openshift-default-994bfcbb9-bw5wc\" (UID: \"40f81e1f-3df1-4802-9636-b2ab2263f0ae\") " pod="openshift-ingress/e2e-praxis-a-b9dad7-openshift-default-994bfcbb9-bw5wc" Nov 03 01:02:26.243993 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:02:26.239990 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/40f81e1f-3df1-4802-9636-b2ab2263f0ae-istiod-ca-cert\") pod \"e2e-praxis-a-b9dad7-openshift-default-994bfcbb9-bw5wc\" (UID: \"40f81e1f-3df1-4802-9636-b2ab2263f0ae\") " pod="openshift-ingress/e2e-praxis-a-b9dad7-openshift-default-994bfcbb9-bw5wc" Nov 03 01:02:26.254446 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:02:26.254413 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:02:26.267165 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:02:26.255051 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/40f81e1f-3df1-4802-9636-b2ab2263f0ae-workload-socket\") pod \"e2e-praxis-a-b9dad7-openshift-default-994bfcbb9-bw5wc\" (UID: \"40f81e1f-3df1-4802-9636-b2ab2263f0ae\") " pod="openshift-ingress/e2e-praxis-a-b9dad7-openshift-default-994bfcbb9-bw5wc" Nov 03 01:02:26.267724 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:02:26.255289 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/40f81e1f-3df1-4802-9636-b2ab2263f0ae-workload-certs\") pod \"e2e-praxis-a-b9dad7-openshift-default-994bfcbb9-bw5wc\" (UID: \"40f81e1f-3df1-4802-9636-b2ab2263f0ae\") " pod="openshift-ingress/e2e-praxis-a-b9dad7-openshift-default-994bfcbb9-bw5wc" Nov 03 01:02:26.294992 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:02:26.293814 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/40f81e1f-3df1-4802-9636-b2ab2263f0ae-istio-podinfo\") pod \"e2e-praxis-a-b9dad7-openshift-default-994bfcbb9-bw5wc\" (UID: \"40f81e1f-3df1-4802-9636-b2ab2263f0ae\") " pod="openshift-ingress/e2e-praxis-a-b9dad7-openshift-default-994bfcbb9-bw5wc" Nov 03 01:02:26.305936 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:02:26.305888 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/40f81e1f-3df1-4802-9636-b2ab2263f0ae-istio-envoy\") pod \"e2e-praxis-a-b9dad7-openshift-default-994bfcbb9-bw5wc\" (UID: \"40f81e1f-3df1-4802-9636-b2ab2263f0ae\") " pod="openshift-ingress/e2e-praxis-a-b9dad7-openshift-default-994bfcbb9-bw5wc" Nov 03 01:02:26.349282 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:02:26.349211 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/40f81e1f-3df1-4802-9636-b2ab2263f0ae-istio-token\") pod \"e2e-praxis-a-b9dad7-openshift-default-994bfcbb9-bw5wc\" (UID: \"40f81e1f-3df1-4802-9636-b2ab2263f0ae\") " pod="openshift-ingress/e2e-praxis-a-b9dad7-openshift-default-994bfcbb9-bw5wc" Nov 03 01:02:26.357938 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:02:26.354720 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2wjcn\" (UniqueName: \"kubernetes.io/projected/40f81e1f-3df1-4802-9636-b2ab2263f0ae-kube-api-access-2wjcn\") pod \"e2e-praxis-a-b9dad7-openshift-default-994bfcbb9-bw5wc\" (UID: \"40f81e1f-3df1-4802-9636-b2ab2263f0ae\") " pod="openshift-ingress/e2e-praxis-a-b9dad7-openshift-default-994bfcbb9-bw5wc" Nov 03 01:02:26.400753 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:02:26.400601 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:02:26.414856 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:02:26.414400 4939 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-a-b9dad7-openshift-default-994bfcbb9-bw5wc" Nov 03 01:02:26.849185 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:02:26.841601 4939 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 01:02:26.849185 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:02:26.841777 4939 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 01:02:26.849185 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:02:26.841824 4939 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 01:02:26.938930 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:02:26.935796 4939 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-praxis-a-b9dad7-openshift-default-994bfcbb9-bw5wc"] Nov 03 01:02:26.984463 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:02:26.984420 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:02:27.655128 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:02:27.655085 4939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-b9dad7-openshift-default-994bfcbb9-bw5wc" event={"ID":"40f81e1f-3df1-4802-9636-b2ab2263f0ae","Type":"ContainerStarted","Data":"1832905ec9ae5b736eba433d7706628698036e63e972c24219ab929e0b6ff842"} Nov 03 01:02:27.655637 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:02:27.655155 4939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-b9dad7-openshift-default-994bfcbb9-bw5wc" event={"ID":"40f81e1f-3df1-4802-9636-b2ab2263f0ae","Type":"ContainerStarted","Data":"cb9dd7cc876429d260155290c349713abc89df93d941aed80e6eeeb3e2bb5cf1"} Nov 03 01:02:27.693984 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:02:27.693320 4939 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-praxis-a-b9dad7-openshift-default-994bfcbb9-bw5wc" podStartSLOduration=1.693294987 podStartE2EDuration="1.693294987s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 01:02:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 01:02:27.676638826 +0000 UTC m=+1295.940973809" watchObservedRunningTime="2026-11-03 01:02:27.693294987 +0000 UTC m=+1295.957629933" Nov 03 01:02:28.030988 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:02:28.014141 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:02:28.416105 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:02:28.416066 4939 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-praxis-a-b9dad7-openshift-default-994bfcbb9-bw5wc" Nov 03 01:02:28.422475 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:02:28.422434 4939 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-praxis-a-b9dad7-openshift-default-994bfcbb9-bw5wc" Nov 03 01:02:28.679343 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:02:28.679253 4939 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-praxis-a-b9dad7-openshift-default-994bfcbb9-bw5wc" Nov 03 01:02:28.680269 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:02:28.680249 4939 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-praxis-a-b9dad7-openshift-default-994bfcbb9-bw5wc" Nov 03 01:02:29.391896 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:02:29.391869 4939 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-02e4c0-openshift-default-c64cd65d9-7l2h4" Nov 03 01:02:29.522255 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:02:29.522221 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:02:29.575932 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:02:29.575623 4939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2df5fd65-3b2a-448e-92a8-f07a8888692d-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2df5fd65-3b2a-448e-92a8-f07a8888692d-workload-socket\") pod \"2df5fd65-3b2a-448e-92a8-f07a8888692d\" (UID: \"2df5fd65-3b2a-448e-92a8-f07a8888692d\") " Nov 03 01:02:29.575932 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:02:29.575778 4939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2df5fd65-3b2a-448e-92a8-f07a8888692d-istio-token\" (UniqueName: \"kubernetes.io/projected/2df5fd65-3b2a-448e-92a8-f07a8888692d-istio-token\") pod \"2df5fd65-3b2a-448e-92a8-f07a8888692d\" (UID: \"2df5fd65-3b2a-448e-92a8-f07a8888692d\") " Nov 03 01:02:29.575932 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:02:29.575835 4939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2df5fd65-3b2a-448e-92a8-f07a8888692d-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2df5fd65-3b2a-448e-92a8-f07a8888692d-credential-socket\") pod \"2df5fd65-3b2a-448e-92a8-f07a8888692d\" (UID: \"2df5fd65-3b2a-448e-92a8-f07a8888692d\") " Nov 03 01:02:29.576264 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:02:29.575947 4939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2df5fd65-3b2a-448e-92a8-f07a8888692d-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2df5fd65-3b2a-448e-92a8-f07a8888692d-istiod-ca-cert\") pod \"2df5fd65-3b2a-448e-92a8-f07a8888692d\" (UID: \"2df5fd65-3b2a-448e-92a8-f07a8888692d\") " Nov 03 01:02:29.576264 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:02:29.576121 4939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2df5fd65-3b2a-448e-92a8-f07a8888692d-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2df5fd65-3b2a-448e-92a8-f07a8888692d-istio-envoy\") pod \"2df5fd65-3b2a-448e-92a8-f07a8888692d\" (UID: \"2df5fd65-3b2a-448e-92a8-f07a8888692d\") " Nov 03 01:02:29.576264 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:02:29.576230 4939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2df5fd65-3b2a-448e-92a8-f07a8888692d-kube-api-access-fvcp7\" (UniqueName: \"kubernetes.io/projected/2df5fd65-3b2a-448e-92a8-f07a8888692d-kube-api-access-fvcp7\") pod \"2df5fd65-3b2a-448e-92a8-f07a8888692d\" (UID: \"2df5fd65-3b2a-448e-92a8-f07a8888692d\") " Nov 03 01:02:29.581935 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:02:29.578842 4939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/2df5fd65-3b2a-448e-92a8-f07a8888692d-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2df5fd65-3b2a-448e-92a8-f07a8888692d-istio-podinfo\") pod \"2df5fd65-3b2a-448e-92a8-f07a8888692d\" (UID: \"2df5fd65-3b2a-448e-92a8-f07a8888692d\") " Nov 03 01:02:29.581935 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:02:29.578938 4939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2df5fd65-3b2a-448e-92a8-f07a8888692d-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2df5fd65-3b2a-448e-92a8-f07a8888692d-workload-certs\") pod \"2df5fd65-3b2a-448e-92a8-f07a8888692d\" (UID: \"2df5fd65-3b2a-448e-92a8-f07a8888692d\") " Nov 03 01:02:29.581935 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:02:29.579136 4939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2df5fd65-3b2a-448e-92a8-f07a8888692d-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2df5fd65-3b2a-448e-92a8-f07a8888692d-istio-data\") pod \"2df5fd65-3b2a-448e-92a8-f07a8888692d\" (UID: \"2df5fd65-3b2a-448e-92a8-f07a8888692d\") " Nov 03 01:02:29.599981 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:02:29.595627 4939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2df5fd65-3b2a-448e-92a8-f07a8888692d-workload-socket" pod "2df5fd65-3b2a-448e-92a8-f07a8888692d" (UID: "2df5fd65-3b2a-448e-92a8-f07a8888692d"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 01:02:29.599981 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:02:29.596459 4939 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2df5fd65-3b2a-448e-92a8-f07a8888692d-workload-socket\") on node \"ip-10-0-140-107.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:02:29.599981 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:02:29.597172 4939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2df5fd65-3b2a-448e-92a8-f07a8888692d-istio-data" pod "2df5fd65-3b2a-448e-92a8-f07a8888692d" (UID: "2df5fd65-3b2a-448e-92a8-f07a8888692d"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 01:02:29.610524 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:02:29.610489 4939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2df5fd65-3b2a-448e-92a8-f07a8888692d-istio-token" pod "2df5fd65-3b2a-448e-92a8-f07a8888692d" (UID: "2df5fd65-3b2a-448e-92a8-f07a8888692d"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 01:02:29.620057 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:02:29.620019 4939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2df5fd65-3b2a-448e-92a8-f07a8888692d-workload-certs" pod "2df5fd65-3b2a-448e-92a8-f07a8888692d" (UID: "2df5fd65-3b2a-448e-92a8-f07a8888692d"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 01:02:29.621102 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:02:29.621036 4939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2df5fd65-3b2a-448e-92a8-f07a8888692d-credential-socket" pod "2df5fd65-3b2a-448e-92a8-f07a8888692d" (UID: "2df5fd65-3b2a-448e-92a8-f07a8888692d"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 01:02:29.621311 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:02:29.621232 4939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2df5fd65-3b2a-448e-92a8-f07a8888692d-istiod-ca-cert" pod "2df5fd65-3b2a-448e-92a8-f07a8888692d" (UID: "2df5fd65-3b2a-448e-92a8-f07a8888692d"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 01:02:29.636095 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:02:29.636023 4939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2df5fd65-3b2a-448e-92a8-f07a8888692d-istio-envoy" pod "2df5fd65-3b2a-448e-92a8-f07a8888692d" (UID: "2df5fd65-3b2a-448e-92a8-f07a8888692d"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 01:02:29.656720 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:02:29.656664 4939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2df5fd65-3b2a-448e-92a8-f07a8888692d-kube-api-access-fvcp7" pod "2df5fd65-3b2a-448e-92a8-f07a8888692d" (UID: "2df5fd65-3b2a-448e-92a8-f07a8888692d"). InnerVolumeSpecName "kube-api-access-fvcp7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 01:02:29.661098 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:02:29.661036 4939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/2df5fd65-3b2a-448e-92a8-f07a8888692d-istio-podinfo" pod "2df5fd65-3b2a-448e-92a8-f07a8888692d" (UID: "2df5fd65-3b2a-448e-92a8-f07a8888692d"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 01:02:29.697947 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:02:29.697779 4939 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2df5fd65-3b2a-448e-92a8-f07a8888692d-istio-token\") on node \"ip-10-0-140-107.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:02:29.697947 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:02:29.697815 4939 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2df5fd65-3b2a-448e-92a8-f07a8888692d-credential-socket\") on node \"ip-10-0-140-107.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:02:29.697947 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:02:29.697830 4939 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2df5fd65-3b2a-448e-92a8-f07a8888692d-istiod-ca-cert\") on node \"ip-10-0-140-107.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:02:29.697947 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:02:29.697845 4939 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2df5fd65-3b2a-448e-92a8-f07a8888692d-istio-envoy\") on node \"ip-10-0-140-107.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:02:29.697947 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:02:29.697859 4939 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fvcp7\" (UniqueName: \"kubernetes.io/projected/2df5fd65-3b2a-448e-92a8-f07a8888692d-kube-api-access-fvcp7\") on node \"ip-10-0-140-107.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:02:29.697947 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:02:29.697878 4939 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2df5fd65-3b2a-448e-92a8-f07a8888692d-istio-podinfo\") on node \"ip-10-0-140-107.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:02:29.697947 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:02:29.697893 4939 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2df5fd65-3b2a-448e-92a8-f07a8888692d-workload-certs\") on node \"ip-10-0-140-107.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:02:29.697947 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:02:29.697921 4939 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2df5fd65-3b2a-448e-92a8-f07a8888692d-istio-data\") on node \"ip-10-0-140-107.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:02:29.706992 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:02:29.706868 4939 generic.go:356] "Generic (PLEG): container finished" podID="2df5fd65-3b2a-448e-92a8-f07a8888692d" containerID="153561948d7bc4acf758d26763ac2b0e3a6e4165173e85bf546f7b0b4314fd27" exitCode=0 Nov 03 01:02:29.706992 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:02:29.706933 4939 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-02e4c0-openshift-default-c64cd65d9-7l2h4" Nov 03 01:02:29.706992 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:02:29.706957 4939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-02e4c0-openshift-default-c64cd65d9-7l2h4" event={"ID":"2df5fd65-3b2a-448e-92a8-f07a8888692d","Type":"ContainerDied","Data":"153561948d7bc4acf758d26763ac2b0e3a6e4165173e85bf546f7b0b4314fd27"} Nov 03 01:02:29.707217 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:02:29.707029 4939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-02e4c0-openshift-default-c64cd65d9-7l2h4" event={"ID":"2df5fd65-3b2a-448e-92a8-f07a8888692d","Type":"ContainerDied","Data":"f7a4cdea8b289934af0481f89ad892a2dae979e81f61f19b4e6a8edac9750fb3"} Nov 03 01:02:29.707315 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:02:29.707061 4939 scope.go:122] "RemoveContainer" containerID="153561948d7bc4acf758d26763ac2b0e3a6e4165173e85bf546f7b0b4314fd27" Nov 03 01:02:29.721203 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:02:29.721183 4939 scope.go:122] "RemoveContainer" containerID="ea409d7fcc0d44e2d14e7b53da7e4f0377490e01ba0cd67ec8b499c4d77a8b20" Nov 03 01:02:29.731041 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:02:29.731017 4939 scope.go:122] "RemoveContainer" containerID="153561948d7bc4acf758d26763ac2b0e3a6e4165173e85bf546f7b0b4314fd27" Nov 03 01:02:29.731340 ip-10-0-140-107 kubenswrapper[4939]: E1103 01:02:29.731319 4939 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"153561948d7bc4acf758d26763ac2b0e3a6e4165173e85bf546f7b0b4314fd27\": container with ID starting with 153561948d7bc4acf758d26763ac2b0e3a6e4165173e85bf546f7b0b4314fd27 not found: ID does not exist" containerID="153561948d7bc4acf758d26763ac2b0e3a6e4165173e85bf546f7b0b4314fd27" Nov 03 01:02:29.731436 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:02:29.731354 4939 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"153561948d7bc4acf758d26763ac2b0e3a6e4165173e85bf546f7b0b4314fd27"} err="failed to get container status \"153561948d7bc4acf758d26763ac2b0e3a6e4165173e85bf546f7b0b4314fd27\": rpc error: code = NotFound desc = could not find container \"153561948d7bc4acf758d26763ac2b0e3a6e4165173e85bf546f7b0b4314fd27\": container with ID starting with 153561948d7bc4acf758d26763ac2b0e3a6e4165173e85bf546f7b0b4314fd27 not found: ID does not exist" Nov 03 01:02:29.731436 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:02:29.731379 4939 scope.go:122] "RemoveContainer" containerID="ea409d7fcc0d44e2d14e7b53da7e4f0377490e01ba0cd67ec8b499c4d77a8b20" Nov 03 01:02:29.731658 ip-10-0-140-107 kubenswrapper[4939]: E1103 01:02:29.731634 4939 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ea409d7fcc0d44e2d14e7b53da7e4f0377490e01ba0cd67ec8b499c4d77a8b20\": container with ID starting with ea409d7fcc0d44e2d14e7b53da7e4f0377490e01ba0cd67ec8b499c4d77a8b20 not found: ID does not exist" containerID="ea409d7fcc0d44e2d14e7b53da7e4f0377490e01ba0cd67ec8b499c4d77a8b20" Nov 03 01:02:29.731709 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:02:29.731669 4939 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ea409d7fcc0d44e2d14e7b53da7e4f0377490e01ba0cd67ec8b499c4d77a8b20"} err="failed to get container status \"ea409d7fcc0d44e2d14e7b53da7e4f0377490e01ba0cd67ec8b499c4d77a8b20\": rpc error: code = NotFound desc = could not find container \"ea409d7fcc0d44e2d14e7b53da7e4f0377490e01ba0cd67ec8b499c4d77a8b20\": container with ID starting with ea409d7fcc0d44e2d14e7b53da7e4f0377490e01ba0cd67ec8b499c4d77a8b20 not found: ID does not exist" Nov 03 01:02:29.734393 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:02:29.734374 4939 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-02e4c0-openshift-default-c64cd65d9-7l2h4"] Nov 03 01:02:29.740789 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:02:29.740763 4939 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-b-02e4c0-openshift-default-c64cd65d9-7l2h4"] Nov 03 01:02:29.877472 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:02:29.877389 4939 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2df5fd65-3b2a-448e-92a8-f07a8888692d" path="/var/lib/kubelet/pods/2df5fd65-3b2a-448e-92a8-f07a8888692d/volumes" Nov 03 01:02:30.075315 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:02:30.075282 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:02:30.267983 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:02:30.264300 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:02:30.586263 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:02:30.586235 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:02:31.285713 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:02:31.285681 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:02:31.318043 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:02:31.318016 4939 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-b9dad7-7458764995-5pdtn"] Nov 03 01:02:31.318726 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:02:31.318706 4939 cpu_manager.go:414] "containerMap: removing container" podUID="2df5fd65-3b2a-448e-92a8-f07a8888692d" containerName="istio-proxy" Nov 03 01:02:31.318859 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:02:31.318727 4939 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2df5fd65-3b2a-448e-92a8-f07a8888692d" containerName="istio-proxy" Nov 03 01:02:31.318859 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:02:31.318763 4939 cpu_manager.go:414] "containerMap: removing container" podUID="2df5fd65-3b2a-448e-92a8-f07a8888692d" containerName="istio-proxy" Nov 03 01:02:31.318859 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:02:31.318771 4939 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2df5fd65-3b2a-448e-92a8-f07a8888692d" containerName="istio-proxy" Nov 03 01:02:31.319063 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:02:31.319007 4939 memory_manager.go:365] "RemoveStaleState removing state" podUID="2df5fd65-3b2a-448e-92a8-f07a8888692d" containerName="istio-proxy" Nov 03 01:02:31.324358 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:02:31.324337 4939 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-b9dad7-7458764995-5pdtn" Nov 03 01:02:31.329139 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:02:31.328893 4939 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-praxis-a-b9dad7-dockercfg-78h26\"" Nov 03 01:02:31.329590 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:02:31.329575 4939 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-praxis-a-b9dad7\"" Nov 03 01:02:31.331268 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:02:31.331135 4939 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-b9dad7-7458764995-5pdtn"] Nov 03 01:02:31.585161 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:02:31.548535 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e6830dd6-9ccc-47ca-b7a3-3cbb6c9bc8d4-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-a-b9dad7-7458764995-5pdtn\" (UID: \"e6830dd6-9ccc-47ca-b7a3-3cbb6c9bc8d4\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-b9dad7-7458764995-5pdtn" Nov 03 01:02:31.585161 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:02:31.548797 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mq8k5\" (UniqueName: \"kubernetes.io/projected/e6830dd6-9ccc-47ca-b7a3-3cbb6c9bc8d4-kube-api-access-mq8k5\") pod \"payload-pre-processing-e2e-praxis-a-b9dad7-7458764995-5pdtn\" (UID: \"e6830dd6-9ccc-47ca-b7a3-3cbb6c9bc8d4\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-b9dad7-7458764995-5pdtn" Nov 03 01:02:31.585161 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:02:31.550514 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:02:31.780799 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:02:31.772006 4939 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-b9dad7-8569798f65-m9hcd"] Nov 03 01:02:31.782234 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:02:31.782204 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mq8k5\" (UniqueName: \"kubernetes.io/projected/e6830dd6-9ccc-47ca-b7a3-3cbb6c9bc8d4-kube-api-access-mq8k5\") pod \"payload-pre-processing-e2e-praxis-a-b9dad7-7458764995-5pdtn\" (UID: \"e6830dd6-9ccc-47ca-b7a3-3cbb6c9bc8d4\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-b9dad7-7458764995-5pdtn" Nov 03 01:02:31.783693 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:02:31.783669 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e6830dd6-9ccc-47ca-b7a3-3cbb6c9bc8d4-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-a-b9dad7-7458764995-5pdtn\" (UID: \"e6830dd6-9ccc-47ca-b7a3-3cbb6c9bc8d4\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-b9dad7-7458764995-5pdtn" Nov 03 01:02:31.785149 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:02:31.782647 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e6830dd6-9ccc-47ca-b7a3-3cbb6c9bc8d4-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-a-b9dad7-7458764995-5pdtn\" (UID: \"e6830dd6-9ccc-47ca-b7a3-3cbb6c9bc8d4\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-b9dad7-7458764995-5pdtn" Nov 03 01:02:31.790997 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:02:31.790972 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:02:31.792698 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:02:31.792233 4939 memory_manager.go:365] "RemoveStaleState removing state" podUID="2df5fd65-3b2a-448e-92a8-f07a8888692d" containerName="istio-proxy" Nov 03 01:02:31.801383 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:02:31.801234 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mq8k5\" (UniqueName: \"kubernetes.io/projected/e6830dd6-9ccc-47ca-b7a3-3cbb6c9bc8d4-kube-api-access-mq8k5\") pod \"payload-pre-processing-e2e-praxis-a-b9dad7-7458764995-5pdtn\" (UID: \"e6830dd6-9ccc-47ca-b7a3-3cbb6c9bc8d4\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-b9dad7-7458764995-5pdtn" Nov 03 01:02:31.810309 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:02:31.810288 4939 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-b9dad7-8569798f65-m9hcd"] Nov 03 01:02:31.810422 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:02:31.810385 4939 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-a-b9dad7-8569798f65-m9hcd" Nov 03 01:02:31.938594 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:02:31.938497 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nrfm4\" (UniqueName: \"kubernetes.io/projected/2b03c0be-253a-41f3-907b-ec20ddc72a52-kube-api-access-nrfm4\") pod \"payload-processing-e2e-praxis-a-b9dad7-8569798f65-m9hcd\" (UID: \"2b03c0be-253a-41f3-907b-ec20ddc72a52\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-b9dad7-8569798f65-m9hcd" Nov 03 01:02:31.938887 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:02:31.938866 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2b03c0be-253a-41f3-907b-ec20ddc72a52-plugins-config-volume\") pod \"payload-processing-e2e-praxis-a-b9dad7-8569798f65-m9hcd\" (UID: \"2b03c0be-253a-41f3-907b-ec20ddc72a52\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-b9dad7-8569798f65-m9hcd" Nov 03 01:02:31.971340 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:02:31.971304 4939 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-b9dad7-7458764995-5pdtn" Nov 03 01:02:32.066956 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:02:32.045691 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nrfm4\" (UniqueName: \"kubernetes.io/projected/2b03c0be-253a-41f3-907b-ec20ddc72a52-kube-api-access-nrfm4\") pod \"payload-processing-e2e-praxis-a-b9dad7-8569798f65-m9hcd\" (UID: \"2b03c0be-253a-41f3-907b-ec20ddc72a52\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-b9dad7-8569798f65-m9hcd" Nov 03 01:02:32.066956 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:02:32.045762 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2b03c0be-253a-41f3-907b-ec20ddc72a52-plugins-config-volume\") pod \"payload-processing-e2e-praxis-a-b9dad7-8569798f65-m9hcd\" (UID: \"2b03c0be-253a-41f3-907b-ec20ddc72a52\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-b9dad7-8569798f65-m9hcd" Nov 03 01:02:32.066956 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:02:32.060982 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2b03c0be-253a-41f3-907b-ec20ddc72a52-plugins-config-volume\") pod \"payload-processing-e2e-praxis-a-b9dad7-8569798f65-m9hcd\" (UID: \"2b03c0be-253a-41f3-907b-ec20ddc72a52\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-b9dad7-8569798f65-m9hcd" Nov 03 01:02:32.155790 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:02:32.114734 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nrfm4\" (UniqueName: \"kubernetes.io/projected/2b03c0be-253a-41f3-907b-ec20ddc72a52-kube-api-access-nrfm4\") pod \"payload-processing-e2e-praxis-a-b9dad7-8569798f65-m9hcd\" (UID: \"2b03c0be-253a-41f3-907b-ec20ddc72a52\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-b9dad7-8569798f65-m9hcd" Nov 03 01:02:32.164854 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:02:32.164596 4939 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-a-b9dad7-8569798f65-m9hcd" Nov 03 01:02:32.336514 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:02:32.334958 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:02:32.470997 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:02:32.460773 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:02:32.523015 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:02:32.522989 4939 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-b9dad7-7458764995-5pdtn"] Nov 03 01:02:32.538230 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:02:32.537606 4939 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-b9dad7-8569798f65-m9hcd"] Nov 03 01:02:32.541967 ip-10-0-140-107 kubenswrapper[4939]: W1103 01:02:32.540970 4939 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2b03c0be_253a_41f3_907b_ec20ddc72a52.slice/crio-4696c7be42ec5317b3160afbe059e0bb030b95ef4c8bd9cada09cdc6194b70f3 WatchSource:0}: Error finding container 4696c7be42ec5317b3160afbe059e0bb030b95ef4c8bd9cada09cdc6194b70f3: Status 404 returned error can't find the container with id 4696c7be42ec5317b3160afbe059e0bb030b95ef4c8bd9cada09cdc6194b70f3 Nov 03 01:02:32.592716 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:02:32.592583 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:02:32.707154 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:02:32.707119 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:02:32.873416 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:02:32.864672 4939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-a-b9dad7-8569798f65-m9hcd" event={"ID":"2b03c0be-253a-41f3-907b-ec20ddc72a52","Type":"ContainerStarted","Data":"73d99d7b4c933b05257f5b032f0a345d12a06df1c46c832538b61cd6029cf4bc"} Nov 03 01:02:32.873416 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:02:32.864736 4939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-a-b9dad7-8569798f65-m9hcd" event={"ID":"2b03c0be-253a-41f3-907b-ec20ddc72a52","Type":"ContainerStarted","Data":"4696c7be42ec5317b3160afbe059e0bb030b95ef4c8bd9cada09cdc6194b70f3"} Nov 03 01:02:32.873416 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:02:32.867031 4939 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-praxis-a-b9dad7-8569798f65-m9hcd" Nov 03 01:02:32.873416 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:02:32.869512 4939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-b9dad7-7458764995-5pdtn" event={"ID":"e6830dd6-9ccc-47ca-b7a3-3cbb6c9bc8d4","Type":"ContainerStarted","Data":"493d816a46f7a89b30934d47cc9ce25b8c44fd000b264b8cf8f8b553fbeec657"} Nov 03 01:02:32.873416 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:02:32.869566 4939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-b9dad7-7458764995-5pdtn" event={"ID":"e6830dd6-9ccc-47ca-b7a3-3cbb6c9bc8d4","Type":"ContainerStarted","Data":"88344714d3e7ce2fbe5bec9a20680f38cf20c0fd7c4fe8614b56d8e33014cebd"} Nov 03 01:02:32.873416 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:02:32.870484 4939 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-b9dad7-7458764995-5pdtn" Nov 03 01:02:32.920931 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:02:32.916142 4939 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-praxis-a-b9dad7-8569798f65-m9hcd" podStartSLOduration=1.9161221130000001 podStartE2EDuration="1.916122113s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 01:02:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 01:02:32.894256362 +0000 UTC m=+1301.158591321" watchObservedRunningTime="2026-11-03 01:02:32.916122113 +0000 UTC m=+1301.180457059" Nov 03 01:02:32.944002 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:02:32.943254 4939 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-b9dad7-7458764995-5pdtn" podStartSLOduration=1.943236845 podStartE2EDuration="1.943236845s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 01:02:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 01:02:32.911985912 +0000 UTC m=+1301.176320858" watchObservedRunningTime="2026-11-03 01:02:32.943236845 +0000 UTC m=+1301.207571791" Nov 03 01:02:32.990351 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:02:32.970080 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:02:33.319325 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:02:33.319225 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:02:34.038879 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:02:34.038848 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:02:34.711525 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:02:34.711490 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:02:35.491808 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:02:35.491331 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:02:35.883255 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:02:35.883226 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:02:36.249714 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:02:36.249632 4939 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-d5f78f-7c9668888d-dwcfx"] Nov 03 01:02:36.250174 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:02:36.250142 4939 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-d5f78f-7c9668888d-dwcfx" podUID="3b61a830-b96e-4ad7-a5b9-567553bc47c9" containerName="maas-api" containerID="cri-o://a2914c22a0119d48fce1a9a0d718c91addaae49147b2d0ac323d65ca3014ac6f" gracePeriod=30 Nov 03 01:02:36.364271 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:02:36.364228 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:02:36.820511 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:02:36.820486 4939 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-d5f78f-7c9668888d-dwcfx" Nov 03 01:02:36.900585 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:02:36.900549 4939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3b61a830-b96e-4ad7-a5b9-567553bc47c9-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3b61a830-b96e-4ad7-a5b9-567553bc47c9-maas-api-tls\") pod \"3b61a830-b96e-4ad7-a5b9-567553bc47c9\" (UID: \"3b61a830-b96e-4ad7-a5b9-567553bc47c9\") " Nov 03 01:02:36.900769 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:02:36.900609 4939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3b61a830-b96e-4ad7-a5b9-567553bc47c9-kube-api-access-snmpr\" (UniqueName: \"kubernetes.io/projected/3b61a830-b96e-4ad7-a5b9-567553bc47c9-kube-api-access-snmpr\") pod \"3b61a830-b96e-4ad7-a5b9-567553bc47c9\" (UID: \"3b61a830-b96e-4ad7-a5b9-567553bc47c9\") " Nov 03 01:02:36.900769 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:02:36.900735 4939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3b61a830-b96e-4ad7-a5b9-567553bc47c9-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3b61a830-b96e-4ad7-a5b9-567553bc47c9-openshift-service-ca\") pod \"3b61a830-b96e-4ad7-a5b9-567553bc47c9\" (UID: \"3b61a830-b96e-4ad7-a5b9-567553bc47c9\") " Nov 03 01:02:36.901010 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:02:36.900988 4939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3b61a830-b96e-4ad7-a5b9-567553bc47c9-metrics-certs\" (UniqueName: \"kubernetes.io/secret/3b61a830-b96e-4ad7-a5b9-567553bc47c9-metrics-certs\") pod \"3b61a830-b96e-4ad7-a5b9-567553bc47c9\" (UID: \"3b61a830-b96e-4ad7-a5b9-567553bc47c9\") " Nov 03 01:02:36.901394 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:02:36.901367 4939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3b61a830-b96e-4ad7-a5b9-567553bc47c9-openshift-service-ca" pod "3b61a830-b96e-4ad7-a5b9-567553bc47c9" (UID: "3b61a830-b96e-4ad7-a5b9-567553bc47c9"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 01:02:36.901645 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:02:36.901624 4939 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3b61a830-b96e-4ad7-a5b9-567553bc47c9-openshift-service-ca\") on node \"ip-10-0-140-107.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:02:36.903125 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:02:36.903104 4939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3b61a830-b96e-4ad7-a5b9-567553bc47c9-metrics-certs" pod "3b61a830-b96e-4ad7-a5b9-567553bc47c9" (UID: "3b61a830-b96e-4ad7-a5b9-567553bc47c9"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 01:02:36.904252 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:02:36.904222 4939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3b61a830-b96e-4ad7-a5b9-567553bc47c9-maas-api-tls" pod "3b61a830-b96e-4ad7-a5b9-567553bc47c9" (UID: "3b61a830-b96e-4ad7-a5b9-567553bc47c9"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 01:02:36.907106 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:02:36.906186 4939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3b61a830-b96e-4ad7-a5b9-567553bc47c9-kube-api-access-snmpr" pod "3b61a830-b96e-4ad7-a5b9-567553bc47c9" (UID: "3b61a830-b96e-4ad7-a5b9-567553bc47c9"). InnerVolumeSpecName "kube-api-access-snmpr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 01:02:36.914548 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:02:36.914525 4939 generic.go:356] "Generic (PLEG): container finished" podID="3b61a830-b96e-4ad7-a5b9-567553bc47c9" containerID="a2914c22a0119d48fce1a9a0d718c91addaae49147b2d0ac323d65ca3014ac6f" exitCode=0 Nov 03 01:02:36.914696 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:02:36.914676 4939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-d5f78f-7c9668888d-dwcfx" event={"ID":"3b61a830-b96e-4ad7-a5b9-567553bc47c9","Type":"ContainerDied","Data":"a2914c22a0119d48fce1a9a0d718c91addaae49147b2d0ac323d65ca3014ac6f"} Nov 03 01:02:36.914776 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:02:36.914732 4939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-d5f78f-7c9668888d-dwcfx" event={"ID":"3b61a830-b96e-4ad7-a5b9-567553bc47c9","Type":"ContainerDied","Data":"5c58b2cc7f452e180d50ee4953ee7d8588f242eb1478aca41ae929fc7cbe678f"} Nov 03 01:02:36.914776 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:02:36.914766 4939 scope.go:122] "RemoveContainer" containerID="a2914c22a0119d48fce1a9a0d718c91addaae49147b2d0ac323d65ca3014ac6f" Nov 03 01:02:36.914961 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:02:36.914946 4939 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-d5f78f-7c9668888d-dwcfx" Nov 03 01:02:36.940608 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:02:36.940585 4939 scope.go:122] "RemoveContainer" containerID="a2914c22a0119d48fce1a9a0d718c91addaae49147b2d0ac323d65ca3014ac6f" Nov 03 01:02:36.940972 ip-10-0-140-107 kubenswrapper[4939]: E1103 01:02:36.940951 4939 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a2914c22a0119d48fce1a9a0d718c91addaae49147b2d0ac323d65ca3014ac6f\": container with ID starting with a2914c22a0119d48fce1a9a0d718c91addaae49147b2d0ac323d65ca3014ac6f not found: ID does not exist" containerID="a2914c22a0119d48fce1a9a0d718c91addaae49147b2d0ac323d65ca3014ac6f" Nov 03 01:02:36.941047 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:02:36.940979 4939 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a2914c22a0119d48fce1a9a0d718c91addaae49147b2d0ac323d65ca3014ac6f"} err="failed to get container status \"a2914c22a0119d48fce1a9a0d718c91addaae49147b2d0ac323d65ca3014ac6f\": rpc error: code = NotFound desc = could not find container \"a2914c22a0119d48fce1a9a0d718c91addaae49147b2d0ac323d65ca3014ac6f\": container with ID starting with a2914c22a0119d48fce1a9a0d718c91addaae49147b2d0ac323d65ca3014ac6f not found: ID does not exist" Nov 03 01:02:36.973127 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:02:36.964764 4939 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-d5f78f-7c9668888d-dwcfx"] Nov 03 01:02:36.973127 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:02:36.964823 4939 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-d5f78f-7c9668888d-dwcfx"] Nov 03 01:02:36.987492 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:02:36.987464 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:02:37.027969 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:02:37.011530 4939 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3b61a830-b96e-4ad7-a5b9-567553bc47c9-maas-api-tls\") on node \"ip-10-0-140-107.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:02:37.028439 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:02:37.028370 4939 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-snmpr\" (UniqueName: \"kubernetes.io/projected/3b61a830-b96e-4ad7-a5b9-567553bc47c9-kube-api-access-snmpr\") on node \"ip-10-0-140-107.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:02:37.028669 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:02:37.028645 4939 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3b61a830-b96e-4ad7-a5b9-567553bc47c9-metrics-certs\") on node \"ip-10-0-140-107.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:02:37.261231 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:02:37.261198 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:02:37.360638 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:02:37.360597 4939 patch_prober.go:29] interesting pod/e2e-praxis-b-46510c-openshift-default-849969b69c-9gxjp container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.148:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 01:02:37.360809 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:02:37.360656 4939 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-praxis-b-46510c-openshift-default-849969b69c-9gxjp" podUID="a2aede6b-fd70-4df3-bec1-a716350a85c8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.148:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 01:02:37.783966 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:02:37.783903 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:02:37.897674 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:02:37.896626 4939 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3b61a830-b96e-4ad7-a5b9-567553bc47c9" path="/var/lib/kubelet/pods/3b61a830-b96e-4ad7-a5b9-567553bc47c9/volumes" Nov 03 01:02:38.684022 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:02:38.683987 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:02:39.410718 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:02:39.410687 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:02:39.551565 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:02:39.551518 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:02:39.898292 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:02:39.898252 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:02:40.003760 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:02:40.003716 4939 generic.go:356] "Generic (PLEG): container finished" podID="a2aede6b-fd70-4df3-bec1-a716350a85c8" containerID="351ae7d796b0fdfb5006b2caca4fa98abf7cedf4b9ad8b37b1819eca7f3f372a" exitCode=0 Nov 03 01:02:40.004011 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:02:40.003804 4939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-46510c-openshift-default-849969b69c-9gxjp" event={"ID":"a2aede6b-fd70-4df3-bec1-a716350a85c8","Type":"ContainerDied","Data":"351ae7d796b0fdfb5006b2caca4fa98abf7cedf4b9ad8b37b1819eca7f3f372a"} Nov 03 01:02:40.004342 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:02:40.004316 4939 scope.go:122] "RemoveContainer" containerID="351ae7d796b0fdfb5006b2caca4fa98abf7cedf4b9ad8b37b1819eca7f3f372a" Nov 03 01:02:40.007543 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:02:40.007500 4939 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 01:02:40.007922 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:02:40.007880 4939 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 01:02:40.009128 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:02:40.009096 4939 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 01:02:40.081394 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:02:40.080650 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:02:40.235993 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:02:40.229605 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:02:40.242965 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:02:40.242247 4939 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-ff4113-6fd46cb5cf-g5q8p"] Nov 03 01:02:40.244459 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:02:40.244321 4939 cpu_manager.go:414] "containerMap: removing container" podUID="3b61a830-b96e-4ad7-a5b9-567553bc47c9" containerName="maas-api" Nov 03 01:02:40.245109 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:02:40.245082 4939 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3b61a830-b96e-4ad7-a5b9-567553bc47c9" containerName="maas-api" Nov 03 01:02:40.247243 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:02:40.247226 4939 memory_manager.go:365] "RemoveStaleState removing state" podUID="3b61a830-b96e-4ad7-a5b9-567553bc47c9" containerName="maas-api" Nov 03 01:02:40.251192 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:02:40.251167 4939 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-ff4113-6fd46cb5cf-g5q8p" Nov 03 01:02:40.252544 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:02:40.252517 4939 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-praxis-a-d5f78f-openshift-default-64f48987d9-rk5hm"] Nov 03 01:02:40.258307 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:02:40.258287 4939 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-b-ff4113\"" Nov 03 01:02:40.258895 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:02:40.258877 4939 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-b-ff4113-dockercfg-95nfb\"" Nov 03 01:02:40.259076 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:02:40.258977 4939 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-praxis-a-d5f78f-openshift-default-64f48987d9-rk5hm" podUID="dfea3192-3541-49c5-a491-ebf7308b74e0" containerName="istio-proxy" containerID="cri-o://dc8aa51c76110652e10097113c76c7c518140ac83f2a5a9cda0ace159a85b59a" gracePeriod=30 Nov 03 01:02:40.265216 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:02:40.264435 4939 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-ff4113-6fd46cb5cf-g5q8p"] Nov 03 01:02:40.308678 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:02:40.308647 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x5x9z\" (UniqueName: \"kubernetes.io/projected/fb530145-405c-43ea-946c-e7f2c85cec68-kube-api-access-x5x9z\") pod \"payload-pre-processing-e2e-ipp-b-ff4113-6fd46cb5cf-g5q8p\" (UID: \"fb530145-405c-43ea-946c-e7f2c85cec68\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-ff4113-6fd46cb5cf-g5q8p" Nov 03 01:02:40.309633 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:02:40.309607 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fb530145-405c-43ea-946c-e7f2c85cec68-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-ff4113-6fd46cb5cf-g5q8p\" (UID: \"fb530145-405c-43ea-946c-e7f2c85cec68\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-ff4113-6fd46cb5cf-g5q8p" Nov 03 01:02:40.398121 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:02:40.398081 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:02:40.425226 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:02:40.425187 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fb530145-405c-43ea-946c-e7f2c85cec68-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-ff4113-6fd46cb5cf-g5q8p\" (UID: \"fb530145-405c-43ea-946c-e7f2c85cec68\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-ff4113-6fd46cb5cf-g5q8p" Nov 03 01:02:40.449416 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:02:40.449290 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x5x9z\" (UniqueName: \"kubernetes.io/projected/fb530145-405c-43ea-946c-e7f2c85cec68-kube-api-access-x5x9z\") pod \"payload-pre-processing-e2e-ipp-b-ff4113-6fd46cb5cf-g5q8p\" (UID: \"fb530145-405c-43ea-946c-e7f2c85cec68\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-ff4113-6fd46cb5cf-g5q8p" Nov 03 01:02:40.449967 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:02:40.449943 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fb530145-405c-43ea-946c-e7f2c85cec68-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-ff4113-6fd46cb5cf-g5q8p\" (UID: \"fb530145-405c-43ea-946c-e7f2c85cec68\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-ff4113-6fd46cb5cf-g5q8p" Nov 03 01:02:40.510266 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:02:40.510183 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-x5x9z\" (UniqueName: \"kubernetes.io/projected/fb530145-405c-43ea-946c-e7f2c85cec68-kube-api-access-x5x9z\") pod \"payload-pre-processing-e2e-ipp-b-ff4113-6fd46cb5cf-g5q8p\" (UID: \"fb530145-405c-43ea-946c-e7f2c85cec68\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-ff4113-6fd46cb5cf-g5q8p" Nov 03 01:02:40.578232 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:02:40.578187 4939 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-ff4113-6fd46cb5cf-g5q8p" Nov 03 01:02:40.942615 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:02:40.942583 4939 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-ff4113-6fd46cb5cf-g5q8p"] Nov 03 01:02:40.943802 ip-10-0-140-107 kubenswrapper[4939]: W1103 01:02:40.943775 4939 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfb530145_405c_43ea_946c_e7f2c85cec68.slice/crio-f4eb305b2d2eb8fc44fa506fd58cba2ec4b0d08e3c8c52ab4e8db934c8532798 WatchSource:0}: Error finding container f4eb305b2d2eb8fc44fa506fd58cba2ec4b0d08e3c8c52ab4e8db934c8532798: Status 404 returned error can't find the container with id f4eb305b2d2eb8fc44fa506fd58cba2ec4b0d08e3c8c52ab4e8db934c8532798 Nov 03 01:02:41.018773 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:02:41.018698 4939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-ff4113-6fd46cb5cf-g5q8p" event={"ID":"fb530145-405c-43ea-946c-e7f2c85cec68","Type":"ContainerStarted","Data":"f4eb305b2d2eb8fc44fa506fd58cba2ec4b0d08e3c8c52ab4e8db934c8532798"} Nov 03 01:02:41.021028 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:02:41.021003 4939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-46510c-openshift-default-849969b69c-9gxjp" event={"ID":"a2aede6b-fd70-4df3-bec1-a716350a85c8","Type":"ContainerStarted","Data":"25c3f582db8e654ba3d466bfc18be83859e1becd529c2598d0904905836add4b"} Nov 03 01:02:41.045266 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:02:41.045239 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:02:41.197049 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:02:41.197020 4939 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-praxis-b-46510c-openshift-default-849969b69c-9gxjp" Nov 03 01:02:41.199792 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:02:41.199508 4939 patch_prober.go:29] interesting pod/e2e-praxis-b-46510c-openshift-default-849969b69c-9gxjp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.148:15021/healthz/ready\": dial tcp 10.132.0.148:15021: connect: connection refused" start-of-body= Nov 03 01:02:41.199792 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:02:41.199555 4939 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-46510c-openshift-default-849969b69c-9gxjp" podUID="a2aede6b-fd70-4df3-bec1-a716350a85c8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.148:15021/healthz/ready\": dial tcp 10.132.0.148:15021: connect: connection refused" Nov 03 01:02:41.200877 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:02:41.200850 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:02:42.030944 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:02:42.030868 4939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-ff4113-6fd46cb5cf-g5q8p" event={"ID":"fb530145-405c-43ea-946c-e7f2c85cec68","Type":"ContainerStarted","Data":"c8d70cf8657137f023db39f5576efa63158d88286208f0bf4cb3bdd4447d78b4"} Nov 03 01:02:42.031436 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:02:42.031091 4939 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-ff4113-6fd46cb5cf-g5q8p" Nov 03 01:02:42.131717 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:02:42.131683 4939 patch_prober.go:29] interesting pod/e2e-praxis-b-46510c-openshift-default-849969b69c-9gxjp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.148:15021/healthz/ready\": dial tcp 10.132.0.148:15021: connect: connection refused" start-of-body= Nov 03 01:02:42.131859 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:02:42.131735 4939 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-46510c-openshift-default-849969b69c-9gxjp" podUID="a2aede6b-fd70-4df3-bec1-a716350a85c8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.148:15021/healthz/ready\": dial tcp 10.132.0.148:15021: connect: connection refused" Nov 03 01:02:42.164666 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:02:42.164604 4939 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-ff4113-6fd46cb5cf-g5q8p" podStartSLOduration=2.164583307 podStartE2EDuration="2.164583307s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 01:02:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 01:02:42.078005161 +0000 UTC m=+1310.342340107" watchObservedRunningTime="2026-11-03 01:02:42.164583307 +0000 UTC m=+1310.428918253" Nov 03 01:02:42.184779 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:02:42.184748 4939 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-b9dad7-7fbdb867c-zrgdg"] Nov 03 01:02:42.195956 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:02:42.195866 4939 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-b9dad7-7fbdb867c-zrgdg"] Nov 03 01:02:42.196140 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:02:42.196001 4939 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-b9dad7-7fbdb867c-zrgdg" Nov 03 01:02:42.206155 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:02:42.206077 4939 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-praxis-a-b9dad7-serving-cert\"" Nov 03 01:02:42.214035 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:02:42.213933 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/220bbc69-e7e7-4f85-9d3d-8b5b0f6bede3-metrics-certs\") pod \"maas-api-e2e-praxis-a-b9dad7-7fbdb867c-zrgdg\" (UID: \"220bbc69-e7e7-4f85-9d3d-8b5b0f6bede3\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-b9dad7-7fbdb867c-zrgdg" Nov 03 01:02:42.214179 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:02:42.213980 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j8mch\" (UniqueName: \"kubernetes.io/projected/220bbc69-e7e7-4f85-9d3d-8b5b0f6bede3-kube-api-access-j8mch\") pod \"maas-api-e2e-praxis-a-b9dad7-7fbdb867c-zrgdg\" (UID: \"220bbc69-e7e7-4f85-9d3d-8b5b0f6bede3\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-b9dad7-7fbdb867c-zrgdg" Nov 03 01:02:42.214378 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:02:42.214353 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/220bbc69-e7e7-4f85-9d3d-8b5b0f6bede3-openshift-service-ca\") pod \"maas-api-e2e-praxis-a-b9dad7-7fbdb867c-zrgdg\" (UID: \"220bbc69-e7e7-4f85-9d3d-8b5b0f6bede3\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-b9dad7-7fbdb867c-zrgdg" Nov 03 01:02:42.217164 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:02:42.215337 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/220bbc69-e7e7-4f85-9d3d-8b5b0f6bede3-maas-api-tls\") pod \"maas-api-e2e-praxis-a-b9dad7-7fbdb867c-zrgdg\" (UID: \"220bbc69-e7e7-4f85-9d3d-8b5b0f6bede3\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-b9dad7-7fbdb867c-zrgdg" Nov 03 01:02:42.286222 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:02:42.286147 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:02:42.316629 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:02:42.316599 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/220bbc69-e7e7-4f85-9d3d-8b5b0f6bede3-openshift-service-ca\") pod \"maas-api-e2e-praxis-a-b9dad7-7fbdb867c-zrgdg\" (UID: \"220bbc69-e7e7-4f85-9d3d-8b5b0f6bede3\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-b9dad7-7fbdb867c-zrgdg" Nov 03 01:02:42.316804 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:02:42.316660 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/220bbc69-e7e7-4f85-9d3d-8b5b0f6bede3-maas-api-tls\") pod \"maas-api-e2e-praxis-a-b9dad7-7fbdb867c-zrgdg\" (UID: \"220bbc69-e7e7-4f85-9d3d-8b5b0f6bede3\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-b9dad7-7fbdb867c-zrgdg" Nov 03 01:02:42.316804 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:02:42.316744 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/220bbc69-e7e7-4f85-9d3d-8b5b0f6bede3-metrics-certs\") pod \"maas-api-e2e-praxis-a-b9dad7-7fbdb867c-zrgdg\" (UID: \"220bbc69-e7e7-4f85-9d3d-8b5b0f6bede3\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-b9dad7-7fbdb867c-zrgdg" Nov 03 01:02:42.316804 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:02:42.316759 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j8mch\" (UniqueName: \"kubernetes.io/projected/220bbc69-e7e7-4f85-9d3d-8b5b0f6bede3-kube-api-access-j8mch\") pod \"maas-api-e2e-praxis-a-b9dad7-7fbdb867c-zrgdg\" (UID: \"220bbc69-e7e7-4f85-9d3d-8b5b0f6bede3\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-b9dad7-7fbdb867c-zrgdg" Nov 03 01:02:42.317867 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:02:42.317834 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/220bbc69-e7e7-4f85-9d3d-8b5b0f6bede3-openshift-service-ca\") pod \"maas-api-e2e-praxis-a-b9dad7-7fbdb867c-zrgdg\" (UID: \"220bbc69-e7e7-4f85-9d3d-8b5b0f6bede3\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-b9dad7-7fbdb867c-zrgdg" Nov 03 01:02:42.328264 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:02:42.328234 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/220bbc69-e7e7-4f85-9d3d-8b5b0f6bede3-metrics-certs\") pod \"maas-api-e2e-praxis-a-b9dad7-7fbdb867c-zrgdg\" (UID: \"220bbc69-e7e7-4f85-9d3d-8b5b0f6bede3\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-b9dad7-7fbdb867c-zrgdg" Nov 03 01:02:42.328585 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:02:42.328559 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-j8mch\" (UniqueName: \"kubernetes.io/projected/220bbc69-e7e7-4f85-9d3d-8b5b0f6bede3-kube-api-access-j8mch\") pod \"maas-api-e2e-praxis-a-b9dad7-7fbdb867c-zrgdg\" (UID: \"220bbc69-e7e7-4f85-9d3d-8b5b0f6bede3\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-b9dad7-7fbdb867c-zrgdg" Nov 03 01:02:42.330501 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:02:42.330479 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/220bbc69-e7e7-4f85-9d3d-8b5b0f6bede3-maas-api-tls\") pod \"maas-api-e2e-praxis-a-b9dad7-7fbdb867c-zrgdg\" (UID: \"220bbc69-e7e7-4f85-9d3d-8b5b0f6bede3\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-b9dad7-7fbdb867c-zrgdg" Nov 03 01:02:42.409318 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:02:42.409235 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:02:42.532934 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:02:42.526185 4939 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-b9dad7-7fbdb867c-zrgdg" Nov 03 01:02:42.903191 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:02:42.902999 4939 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-b9dad7-7fbdb867c-zrgdg"] Nov 03 01:02:42.906599 ip-10-0-140-107 kubenswrapper[4939]: W1103 01:02:42.906568 4939 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod220bbc69_e7e7_4f85_9d3d_8b5b0f6bede3.slice/crio-490c7b2c9d3bf74d78061ab7f7d0f8b5def9e860f5b7a23cd1dfa3d5fd8732bc WatchSource:0}: Error finding container 490c7b2c9d3bf74d78061ab7f7d0f8b5def9e860f5b7a23cd1dfa3d5fd8732bc: Status 404 returned error can't find the container with id 490c7b2c9d3bf74d78061ab7f7d0f8b5def9e860f5b7a23cd1dfa3d5fd8732bc Nov 03 01:02:42.908530 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:02:42.908513 4939 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Nov 03 01:02:42.978979 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:02:42.970483 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:02:43.073937 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:02:43.072356 4939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-b9dad7-7fbdb867c-zrgdg" event={"ID":"220bbc69-e7e7-4f85-9d3d-8b5b0f6bede3","Type":"ContainerStarted","Data":"490c7b2c9d3bf74d78061ab7f7d0f8b5def9e860f5b7a23cd1dfa3d5fd8732bc"} Nov 03 01:02:43.134930 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:02:43.134312 4939 patch_prober.go:29] interesting pod/e2e-praxis-b-46510c-openshift-default-849969b69c-9gxjp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.148:15021/healthz/ready\": dial tcp 10.132.0.148:15021: connect: connection refused" start-of-body= Nov 03 01:02:43.134930 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:02:43.134389 4939 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-46510c-openshift-default-849969b69c-9gxjp" podUID="a2aede6b-fd70-4df3-bec1-a716350a85c8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.148:15021/healthz/ready\": dial tcp 10.132.0.148:15021: connect: connection refused" Nov 03 01:02:43.490326 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:02:43.490293 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:02:43.888525 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:02:43.888494 4939 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-praxis-a-b9dad7-8569798f65-m9hcd" Nov 03 01:02:44.081421 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:02:44.081388 4939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-b9dad7-7fbdb867c-zrgdg" event={"ID":"220bbc69-e7e7-4f85-9d3d-8b5b0f6bede3","Type":"ContainerStarted","Data":"240d8a12a0ebeeb3eef3d423715387379095093b4755435966d05a225ce4712a"} Nov 03 01:02:44.081902 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:02:44.081571 4939 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-b9dad7-7fbdb867c-zrgdg" Nov 03 01:02:44.130887 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:02:44.130856 4939 patch_prober.go:29] interesting pod/e2e-praxis-b-46510c-openshift-default-849969b69c-9gxjp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.148:15021/healthz/ready\": dial tcp 10.132.0.148:15021: connect: connection refused" start-of-body= Nov 03 01:02:44.131085 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:02:44.130953 4939 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-46510c-openshift-default-849969b69c-9gxjp" podUID="a2aede6b-fd70-4df3-bec1-a716350a85c8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.148:15021/healthz/ready\": dial tcp 10.132.0.148:15021: connect: connection refused" Nov 03 01:02:44.173731 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:02:44.173474 4939 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-b9dad7-7fbdb867c-zrgdg" podStartSLOduration=1.651721435 podStartE2EDuration="2.17345651s" totalImagesPullingTime="521.735075ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 01:02:42 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 01:02:44.157747011 +0000 UTC m=+1312.422081957" watchObservedRunningTime="2026-11-03 01:02:44.17345651 +0000 UTC m=+1312.437791456" Nov 03 01:02:44.439340 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:02:44.439255 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:02:44.590947 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:02:44.579282 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:02:44.897673 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:02:44.897644 4939 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-b9dad7-7458764995-5pdtn" Nov 03 01:02:45.131460 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:02:45.131416 4939 patch_prober.go:29] interesting pod/e2e-praxis-b-46510c-openshift-default-849969b69c-9gxjp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.148:15021/healthz/ready\": dial tcp 10.132.0.148:15021: connect: connection refused" start-of-body= Nov 03 01:02:45.131880 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:02:45.131492 4939 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-46510c-openshift-default-849969b69c-9gxjp" podUID="a2aede6b-fd70-4df3-bec1-a716350a85c8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.148:15021/healthz/ready\": dial tcp 10.132.0.148:15021: connect: connection refused" Nov 03 01:02:45.434084 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:02:45.434053 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:02:46.035774 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:02:46.035712 4939 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-a-d5f78f-openshift-default-64f48987d9-rk5hm" Nov 03 01:02:46.111302 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:02:46.111261 4939 generic.go:356] "Generic (PLEG): container finished" podID="dfea3192-3541-49c5-a491-ebf7308b74e0" containerID="dc8aa51c76110652e10097113c76c7c518140ac83f2a5a9cda0ace159a85b59a" exitCode=0 Nov 03 01:02:46.111469 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:02:46.111312 4939 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-a-d5f78f-openshift-default-64f48987d9-rk5hm" Nov 03 01:02:46.111578 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:02:46.111559 4939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-d5f78f-openshift-default-64f48987d9-rk5hm" event={"ID":"dfea3192-3541-49c5-a491-ebf7308b74e0","Type":"ContainerDied","Data":"dc8aa51c76110652e10097113c76c7c518140ac83f2a5a9cda0ace159a85b59a"} Nov 03 01:02:46.111647 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:02:46.111626 4939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-d5f78f-openshift-default-64f48987d9-rk5hm" event={"ID":"dfea3192-3541-49c5-a491-ebf7308b74e0","Type":"ContainerDied","Data":"0b0eb9c283a2814232329a608d09e0e4e182142602704add5ffff7f476d5d46d"} Nov 03 01:02:46.111700 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:02:46.111675 4939 scope.go:122] "RemoveContainer" containerID="dc8aa51c76110652e10097113c76c7c518140ac83f2a5a9cda0ace159a85b59a" Nov 03 01:02:46.123175 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:02:46.123150 4939 scope.go:122] "RemoveContainer" containerID="e481325a728fa0455f38b868b5c3e86b20747ed84501736d04d1bc4267145bbe" Nov 03 01:02:46.132161 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:02:46.132123 4939 patch_prober.go:29] interesting pod/e2e-praxis-b-46510c-openshift-default-849969b69c-9gxjp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.148:15021/healthz/ready\": dial tcp 10.132.0.148:15021: connect: connection refused" start-of-body= Nov 03 01:02:46.132564 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:02:46.132194 4939 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-46510c-openshift-default-849969b69c-9gxjp" podUID="a2aede6b-fd70-4df3-bec1-a716350a85c8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.148:15021/healthz/ready\": dial tcp 10.132.0.148:15021: connect: connection refused" Nov 03 01:02:46.135812 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:02:46.135780 4939 scope.go:122] "RemoveContainer" containerID="dc8aa51c76110652e10097113c76c7c518140ac83f2a5a9cda0ace159a85b59a" Nov 03 01:02:46.136156 ip-10-0-140-107 kubenswrapper[4939]: E1103 01:02:46.136133 4939 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"dc8aa51c76110652e10097113c76c7c518140ac83f2a5a9cda0ace159a85b59a\": container with ID starting with dc8aa51c76110652e10097113c76c7c518140ac83f2a5a9cda0ace159a85b59a not found: ID does not exist" containerID="dc8aa51c76110652e10097113c76c7c518140ac83f2a5a9cda0ace159a85b59a" Nov 03 01:02:46.136247 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:02:46.136172 4939 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"dc8aa51c76110652e10097113c76c7c518140ac83f2a5a9cda0ace159a85b59a"} err="failed to get container status \"dc8aa51c76110652e10097113c76c7c518140ac83f2a5a9cda0ace159a85b59a\": rpc error: code = NotFound desc = could not find container \"dc8aa51c76110652e10097113c76c7c518140ac83f2a5a9cda0ace159a85b59a\": container with ID starting with dc8aa51c76110652e10097113c76c7c518140ac83f2a5a9cda0ace159a85b59a not found: ID does not exist" Nov 03 01:02:46.136247 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:02:46.136195 4939 scope.go:122] "RemoveContainer" containerID="e481325a728fa0455f38b868b5c3e86b20747ed84501736d04d1bc4267145bbe" Nov 03 01:02:46.136531 ip-10-0-140-107 kubenswrapper[4939]: E1103 01:02:46.136506 4939 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e481325a728fa0455f38b868b5c3e86b20747ed84501736d04d1bc4267145bbe\": container with ID starting with e481325a728fa0455f38b868b5c3e86b20747ed84501736d04d1bc4267145bbe not found: ID does not exist" containerID="e481325a728fa0455f38b868b5c3e86b20747ed84501736d04d1bc4267145bbe" Nov 03 01:02:46.136591 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:02:46.136541 4939 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e481325a728fa0455f38b868b5c3e86b20747ed84501736d04d1bc4267145bbe"} err="failed to get container status \"e481325a728fa0455f38b868b5c3e86b20747ed84501736d04d1bc4267145bbe\": rpc error: code = NotFound desc = could not find container \"e481325a728fa0455f38b868b5c3e86b20747ed84501736d04d1bc4267145bbe\": container with ID starting with e481325a728fa0455f38b868b5c3e86b20747ed84501736d04d1bc4267145bbe not found: ID does not exist" Nov 03 01:02:46.187642 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:02:46.187605 4939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/dfea3192-3541-49c5-a491-ebf7308b74e0-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dfea3192-3541-49c5-a491-ebf7308b74e0-istiod-ca-cert\") pod \"dfea3192-3541-49c5-a491-ebf7308b74e0\" (UID: \"dfea3192-3541-49c5-a491-ebf7308b74e0\") " Nov 03 01:02:46.187811 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:02:46.187668 4939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dfea3192-3541-49c5-a491-ebf7308b74e0-istio-token\" (UniqueName: \"kubernetes.io/projected/dfea3192-3541-49c5-a491-ebf7308b74e0-istio-token\") pod \"dfea3192-3541-49c5-a491-ebf7308b74e0\" (UID: \"dfea3192-3541-49c5-a491-ebf7308b74e0\") " Nov 03 01:02:46.187811 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:02:46.187780 4939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dfea3192-3541-49c5-a491-ebf7308b74e0-kube-api-access-bgchl\" (UniqueName: \"kubernetes.io/projected/dfea3192-3541-49c5-a491-ebf7308b74e0-kube-api-access-bgchl\") pod \"dfea3192-3541-49c5-a491-ebf7308b74e0\" (UID: \"dfea3192-3541-49c5-a491-ebf7308b74e0\") " Nov 03 01:02:46.188009 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:02:46.187979 4939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dfea3192-3541-49c5-a491-ebf7308b74e0-istiod-ca-cert" pod "dfea3192-3541-49c5-a491-ebf7308b74e0" (UID: "dfea3192-3541-49c5-a491-ebf7308b74e0"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 01:02:46.188088 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:02:46.188005 4939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/dfea3192-3541-49c5-a491-ebf7308b74e0-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dfea3192-3541-49c5-a491-ebf7308b74e0-istio-podinfo\") pod \"dfea3192-3541-49c5-a491-ebf7308b74e0\" (UID: \"dfea3192-3541-49c5-a491-ebf7308b74e0\") " Nov 03 01:02:46.188088 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:02:46.188080 4939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dfea3192-3541-49c5-a491-ebf7308b74e0-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dfea3192-3541-49c5-a491-ebf7308b74e0-credential-socket\") pod \"dfea3192-3541-49c5-a491-ebf7308b74e0\" (UID: \"dfea3192-3541-49c5-a491-ebf7308b74e0\") " Nov 03 01:02:46.188207 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:02:46.188108 4939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dfea3192-3541-49c5-a491-ebf7308b74e0-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dfea3192-3541-49c5-a491-ebf7308b74e0-workload-socket\") pod \"dfea3192-3541-49c5-a491-ebf7308b74e0\" (UID: \"dfea3192-3541-49c5-a491-ebf7308b74e0\") " Nov 03 01:02:46.188207 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:02:46.188161 4939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dfea3192-3541-49c5-a491-ebf7308b74e0-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dfea3192-3541-49c5-a491-ebf7308b74e0-workload-certs\") pod \"dfea3192-3541-49c5-a491-ebf7308b74e0\" (UID: \"dfea3192-3541-49c5-a491-ebf7308b74e0\") " Nov 03 01:02:46.188318 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:02:46.188215 4939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dfea3192-3541-49c5-a491-ebf7308b74e0-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dfea3192-3541-49c5-a491-ebf7308b74e0-istio-data\") pod \"dfea3192-3541-49c5-a491-ebf7308b74e0\" (UID: \"dfea3192-3541-49c5-a491-ebf7308b74e0\") " Nov 03 01:02:46.188318 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:02:46.188242 4939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dfea3192-3541-49c5-a491-ebf7308b74e0-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dfea3192-3541-49c5-a491-ebf7308b74e0-istio-envoy\") pod \"dfea3192-3541-49c5-a491-ebf7308b74e0\" (UID: \"dfea3192-3541-49c5-a491-ebf7308b74e0\") " Nov 03 01:02:46.188520 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:02:46.188494 4939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dfea3192-3541-49c5-a491-ebf7308b74e0-credential-socket" pod "dfea3192-3541-49c5-a491-ebf7308b74e0" (UID: "dfea3192-3541-49c5-a491-ebf7308b74e0"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 01:02:46.188605 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:02:46.188531 4939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dfea3192-3541-49c5-a491-ebf7308b74e0-istio-data" pod "dfea3192-3541-49c5-a491-ebf7308b74e0" (UID: "dfea3192-3541-49c5-a491-ebf7308b74e0"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 01:02:46.189014 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:02:46.188988 4939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dfea3192-3541-49c5-a491-ebf7308b74e0-workload-certs" pod "dfea3192-3541-49c5-a491-ebf7308b74e0" (UID: "dfea3192-3541-49c5-a491-ebf7308b74e0"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 01:02:46.189106 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:02:46.189048 4939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dfea3192-3541-49c5-a491-ebf7308b74e0-workload-socket" pod "dfea3192-3541-49c5-a491-ebf7308b74e0" (UID: "dfea3192-3541-49c5-a491-ebf7308b74e0"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 01:02:46.189676 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:02:46.189645 4939 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dfea3192-3541-49c5-a491-ebf7308b74e0-istiod-ca-cert\") on node \"ip-10-0-140-107.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:02:46.189773 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:02:46.189681 4939 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dfea3192-3541-49c5-a491-ebf7308b74e0-credential-socket\") on node \"ip-10-0-140-107.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:02:46.189773 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:02:46.189698 4939 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dfea3192-3541-49c5-a491-ebf7308b74e0-workload-socket\") on node \"ip-10-0-140-107.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:02:46.189773 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:02:46.189713 4939 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dfea3192-3541-49c5-a491-ebf7308b74e0-workload-certs\") on node \"ip-10-0-140-107.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:02:46.189773 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:02:46.189727 4939 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dfea3192-3541-49c5-a491-ebf7308b74e0-istio-data\") on node \"ip-10-0-140-107.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:02:46.190198 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:02:46.190173 4939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dfea3192-3541-49c5-a491-ebf7308b74e0-istio-token" pod "dfea3192-3541-49c5-a491-ebf7308b74e0" (UID: "dfea3192-3541-49c5-a491-ebf7308b74e0"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 01:02:46.190339 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:02:46.190314 4939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dfea3192-3541-49c5-a491-ebf7308b74e0-kube-api-access-bgchl" pod "dfea3192-3541-49c5-a491-ebf7308b74e0" (UID: "dfea3192-3541-49c5-a491-ebf7308b74e0"). InnerVolumeSpecName "kube-api-access-bgchl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 01:02:46.190805 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:02:46.190780 4939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/dfea3192-3541-49c5-a491-ebf7308b74e0-istio-podinfo" pod "dfea3192-3541-49c5-a491-ebf7308b74e0" (UID: "dfea3192-3541-49c5-a491-ebf7308b74e0"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 01:02:46.191073 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:02:46.191054 4939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dfea3192-3541-49c5-a491-ebf7308b74e0-istio-envoy" pod "dfea3192-3541-49c5-a491-ebf7308b74e0" (UID: "dfea3192-3541-49c5-a491-ebf7308b74e0"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 01:02:46.285131 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:02:46.285098 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:02:46.291200 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:02:46.291132 4939 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dfea3192-3541-49c5-a491-ebf7308b74e0-istio-token\") on node \"ip-10-0-140-107.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:02:46.291200 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:02:46.291158 4939 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bgchl\" (UniqueName: \"kubernetes.io/projected/dfea3192-3541-49c5-a491-ebf7308b74e0-kube-api-access-bgchl\") on node \"ip-10-0-140-107.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:02:46.291200 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:02:46.291169 4939 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dfea3192-3541-49c5-a491-ebf7308b74e0-istio-podinfo\") on node \"ip-10-0-140-107.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:02:46.291200 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:02:46.291178 4939 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dfea3192-3541-49c5-a491-ebf7308b74e0-istio-envoy\") on node \"ip-10-0-140-107.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:02:46.528931 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:02:46.528642 4939 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-praxis-a-d5f78f-openshift-default-64f48987d9-rk5hm"] Nov 03 01:02:46.561985 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:02:46.559783 4939 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-praxis-a-d5f78f-openshift-default-64f48987d9-rk5hm"] Nov 03 01:02:47.027191 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:02:47.027155 4939 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-46510c-5cc495698c-94h2j"] Nov 03 01:02:47.037476 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:02:47.037447 4939 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-46510c-d675d977f-ph287"] Nov 03 01:02:47.130995 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:02:47.130959 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:02:47.135138 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:02:47.135111 4939 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-praxis-b-46510c-openshift-default-849969b69c-9gxjp" Nov 03 01:02:47.135746 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:02:47.135510 4939 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-praxis-b-46510c-openshift-default-849969b69c-9gxjp" Nov 03 01:02:47.136384 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:02:47.136361 4939 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-praxis-b-46510c-openshift-default-849969b69c-9gxjp" Nov 03 01:02:47.251293 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:02:47.251261 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:02:47.361980 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:02:47.361947 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:02:47.873185 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:02:47.873144 4939 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="dfea3192-3541-49c5-a491-ebf7308b74e0" path="/var/lib/kubelet/pods/dfea3192-3541-49c5-a491-ebf7308b74e0/volumes" Nov 03 01:02:48.036872 ip-10-0-140-107 kubenswrapper[4939]: E1103 01:02:48.036832 4939 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-sthjh" containerName="extractor" Nov 03 01:02:48.228480 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:02:48.228355 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:02:48.336961 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:02:48.336925 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:02:48.906834 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:02:48.906803 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:02:49.442753 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:02:49.442715 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:02:49.559593 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:02:49.559552 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:02:49.778898 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:02:49.778822 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:02:50.107762 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:02:50.107730 4939 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-b9dad7-7fbdb867c-zrgdg" Nov 03 01:02:50.378647 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:02:50.378567 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:02:50.502992 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:02:50.502897 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:02:50.872855 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:02:50.872821 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:02:51.001361 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:02:51.000455 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:02:51.236307 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:02:51.236230 4939 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-07f5f6-openshift-default-98746c89f-vz567"] Nov 03 01:02:51.236518 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:02:51.236493 4939 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-a-07f5f6-openshift-default-98746c89f-vz567" podUID="ef4163bb-8632-4675-a678-11ec6d58e737" containerName="istio-proxy" containerID="cri-o://52e11dadb127b4b182eca770f7958e3db330130f547828282776635d0e75bb2f" gracePeriod=30 Nov 03 01:02:51.283148 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:02:51.283118 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:02:51.805308 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:02:51.805253 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:02:52.027586 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:02:52.027548 4939 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-praxis-b-46510c-5cc495698c-94h2j" podUID="7bcdba6a-dab2-4616-82d1-2b0bf8770b8a" containerName="payload-processing" containerID="cri-o://453ddfee6a7ee44a0dd8929959c1674de6c70a77b625a2af0c3fad235acb7ae5" gracePeriod=25 Nov 03 01:02:52.038088 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:02:52.038050 4939 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-46510c-d675d977f-ph287" podUID="35fb0ddf-b9bf-4173-ae72-c0100ac754ad" containerName="payload-pre-processing" containerID="cri-o://a161972dae9d5c2d0f423fd73962af3fe1601734033cdf81552413a9b9ad8cae" gracePeriod=25 Nov 03 01:02:52.166573 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:02:52.166536 4939 generic.go:356] "Generic (PLEG): container finished" podID="35fb0ddf-b9bf-4173-ae72-c0100ac754ad" containerID="a161972dae9d5c2d0f423fd73962af3fe1601734033cdf81552413a9b9ad8cae" exitCode=0 Nov 03 01:02:52.166745 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:02:52.166616 4939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-46510c-d675d977f-ph287" event={"ID":"35fb0ddf-b9bf-4173-ae72-c0100ac754ad","Type":"ContainerDied","Data":"a161972dae9d5c2d0f423fd73962af3fe1601734033cdf81552413a9b9ad8cae"} Nov 03 01:02:52.169030 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:02:52.168996 4939 generic.go:356] "Generic (PLEG): container finished" podID="7bcdba6a-dab2-4616-82d1-2b0bf8770b8a" containerID="453ddfee6a7ee44a0dd8929959c1674de6c70a77b625a2af0c3fad235acb7ae5" exitCode=0 Nov 03 01:02:52.169208 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:02:52.169073 4939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-b-46510c-5cc495698c-94h2j" event={"ID":"7bcdba6a-dab2-4616-82d1-2b0bf8770b8a","Type":"ContainerDied","Data":"453ddfee6a7ee44a0dd8929959c1674de6c70a77b625a2af0c3fad235acb7ae5"} Nov 03 01:02:52.326166 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:02:52.326103 4939 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-b-46510c-5cc495698c-94h2j" Nov 03 01:02:52.356098 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:02:52.356073 4939 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-46510c-d675d977f-ph287" Nov 03 01:02:52.434730 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:02:52.434699 4939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7bcdba6a-dab2-4616-82d1-2b0bf8770b8a-kube-api-access-j4xv9\" (UniqueName: \"kubernetes.io/projected/7bcdba6a-dab2-4616-82d1-2b0bf8770b8a-kube-api-access-j4xv9\") pod \"7bcdba6a-dab2-4616-82d1-2b0bf8770b8a\" (UID: \"7bcdba6a-dab2-4616-82d1-2b0bf8770b8a\") " Nov 03 01:02:52.434949 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:02:52.434764 4939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/35fb0ddf-b9bf-4173-ae72-c0100ac754ad-kube-api-access-bmq6x\" (UniqueName: \"kubernetes.io/projected/35fb0ddf-b9bf-4173-ae72-c0100ac754ad-kube-api-access-bmq6x\") pod \"35fb0ddf-b9bf-4173-ae72-c0100ac754ad\" (UID: \"35fb0ddf-b9bf-4173-ae72-c0100ac754ad\") " Nov 03 01:02:52.434949 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:02:52.434867 4939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/35fb0ddf-b9bf-4173-ae72-c0100ac754ad-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/35fb0ddf-b9bf-4173-ae72-c0100ac754ad-plugins-config-volume\") pod \"35fb0ddf-b9bf-4173-ae72-c0100ac754ad\" (UID: \"35fb0ddf-b9bf-4173-ae72-c0100ac754ad\") " Nov 03 01:02:52.435111 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:02:52.434993 4939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7bcdba6a-dab2-4616-82d1-2b0bf8770b8a-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7bcdba6a-dab2-4616-82d1-2b0bf8770b8a-plugins-config-volume\") pod \"7bcdba6a-dab2-4616-82d1-2b0bf8770b8a\" (UID: \"7bcdba6a-dab2-4616-82d1-2b0bf8770b8a\") " Nov 03 01:02:52.435394 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:02:52.435364 4939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/35fb0ddf-b9bf-4173-ae72-c0100ac754ad-plugins-config-volume" pod "35fb0ddf-b9bf-4173-ae72-c0100ac754ad" (UID: "35fb0ddf-b9bf-4173-ae72-c0100ac754ad"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 01:02:52.435511 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:02:52.435388 4939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7bcdba6a-dab2-4616-82d1-2b0bf8770b8a-plugins-config-volume" pod "7bcdba6a-dab2-4616-82d1-2b0bf8770b8a" (UID: "7bcdba6a-dab2-4616-82d1-2b0bf8770b8a"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 01:02:52.435746 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:02:52.435722 4939 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7bcdba6a-dab2-4616-82d1-2b0bf8770b8a-plugins-config-volume\") on node \"ip-10-0-140-107.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:02:52.435824 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:02:52.435750 4939 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/35fb0ddf-b9bf-4173-ae72-c0100ac754ad-plugins-config-volume\") on node \"ip-10-0-140-107.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:02:52.436943 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:02:52.436899 4939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7bcdba6a-dab2-4616-82d1-2b0bf8770b8a-kube-api-access-j4xv9" pod "7bcdba6a-dab2-4616-82d1-2b0bf8770b8a" (UID: "7bcdba6a-dab2-4616-82d1-2b0bf8770b8a"). InnerVolumeSpecName "kube-api-access-j4xv9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 01:02:52.437021 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:02:52.436954 4939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/35fb0ddf-b9bf-4173-ae72-c0100ac754ad-kube-api-access-bmq6x" pod "35fb0ddf-b9bf-4173-ae72-c0100ac754ad" (UID: "35fb0ddf-b9bf-4173-ae72-c0100ac754ad"). InnerVolumeSpecName "kube-api-access-bmq6x". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 01:02:52.537769 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:02:52.537737 4939 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-j4xv9\" (UniqueName: \"kubernetes.io/projected/7bcdba6a-dab2-4616-82d1-2b0bf8770b8a-kube-api-access-j4xv9\") on node \"ip-10-0-140-107.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:02:52.537769 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:02:52.537766 4939 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bmq6x\" (UniqueName: \"kubernetes.io/projected/35fb0ddf-b9bf-4173-ae72-c0100ac754ad-kube-api-access-bmq6x\") on node \"ip-10-0-140-107.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:02:52.617341 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:02:52.617295 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:02:53.075342 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:02:53.075309 4939 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-ff4113-6fd46cb5cf-g5q8p" Nov 03 01:02:53.178283 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:02:53.178250 4939 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-46510c-d675d977f-ph287" Nov 03 01:02:53.178283 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:02:53.178273 4939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-46510c-d675d977f-ph287" event={"ID":"35fb0ddf-b9bf-4173-ae72-c0100ac754ad","Type":"ContainerDied","Data":"62d36f4c0ddeca6f9e016b059919994256a182262dd4cfc92661f0a966545264"} Nov 03 01:02:53.178498 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:02:53.178339 4939 scope.go:122] "RemoveContainer" containerID="a161972dae9d5c2d0f423fd73962af3fe1601734033cdf81552413a9b9ad8cae" Nov 03 01:02:53.180649 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:02:53.180629 4939 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-b-46510c-5cc495698c-94h2j" Nov 03 01:02:53.180649 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:02:53.180629 4939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-b-46510c-5cc495698c-94h2j" event={"ID":"7bcdba6a-dab2-4616-82d1-2b0bf8770b8a","Type":"ContainerDied","Data":"bfe999920304164d52ac01f85355657fd0f72c1e8aca149fa59353c34bb0579a"} Nov 03 01:02:53.189729 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:02:53.189705 4939 scope.go:122] "RemoveContainer" containerID="453ddfee6a7ee44a0dd8929959c1674de6c70a77b625a2af0c3fad235acb7ae5" Nov 03 01:02:53.216471 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:02:53.216438 4939 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-46510c-d675d977f-ph287"] Nov 03 01:02:53.221673 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:02:53.221643 4939 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-46510c-d675d977f-ph287"] Nov 03 01:02:53.226672 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:02:53.226650 4939 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-46510c-5cc495698c-94h2j"] Nov 03 01:02:53.232983 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:02:53.232960 4939 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-46510c-5cc495698c-94h2j"] Nov 03 01:02:53.581814 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:02:53.581785 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:02:53.869849 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:02:53.869757 4939 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="35fb0ddf-b9bf-4173-ae72-c0100ac754ad" path="/var/lib/kubelet/pods/35fb0ddf-b9bf-4173-ae72-c0100ac754ad/volumes" Nov 03 01:02:53.870367 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:02:53.870342 4939 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7bcdba6a-dab2-4616-82d1-2b0bf8770b8a" path="/var/lib/kubelet/pods/7bcdba6a-dab2-4616-82d1-2b0bf8770b8a/volumes" Nov 03 01:02:54.192779 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:02:54.192703 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:02:54.221807 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:02:54.221777 4939 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-4pw7v"] Nov 03 01:02:54.479504 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:02:54.479365 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:02:54.820316 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:02:54.820236 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:02:55.384228 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:02:55.384195 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:02:55.773870 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:02:55.773748 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:02:56.523796 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:02:56.523711 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:02:56.688923 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:02:56.688873 4939 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-07f5f6-openshift-default-98746c89f-vz567" Nov 03 01:02:56.771595 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:02:56.771550 4939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ef4163bb-8632-4675-a678-11ec6d58e737-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ef4163bb-8632-4675-a678-11ec6d58e737-istio-envoy\") pod \"ef4163bb-8632-4675-a678-11ec6d58e737\" (UID: \"ef4163bb-8632-4675-a678-11ec6d58e737\") " Nov 03 01:02:56.771792 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:02:56.771667 4939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ef4163bb-8632-4675-a678-11ec6d58e737-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ef4163bb-8632-4675-a678-11ec6d58e737-workload-socket\") pod \"ef4163bb-8632-4675-a678-11ec6d58e737\" (UID: \"ef4163bb-8632-4675-a678-11ec6d58e737\") " Nov 03 01:02:56.771792 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:02:56.771725 4939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ef4163bb-8632-4675-a678-11ec6d58e737-istio-token\" (UniqueName: \"kubernetes.io/projected/ef4163bb-8632-4675-a678-11ec6d58e737-istio-token\") pod \"ef4163bb-8632-4675-a678-11ec6d58e737\" (UID: \"ef4163bb-8632-4675-a678-11ec6d58e737\") " Nov 03 01:02:56.771792 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:02:56.771777 4939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ef4163bb-8632-4675-a678-11ec6d58e737-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ef4163bb-8632-4675-a678-11ec6d58e737-credential-socket\") pod \"ef4163bb-8632-4675-a678-11ec6d58e737\" (UID: \"ef4163bb-8632-4675-a678-11ec6d58e737\") " Nov 03 01:02:56.772037 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:02:56.771846 4939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ef4163bb-8632-4675-a678-11ec6d58e737-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ef4163bb-8632-4675-a678-11ec6d58e737-istio-data\") pod \"ef4163bb-8632-4675-a678-11ec6d58e737\" (UID: \"ef4163bb-8632-4675-a678-11ec6d58e737\") " Nov 03 01:02:56.772037 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:02:56.771902 4939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ef4163bb-8632-4675-a678-11ec6d58e737-kube-api-access-zccwt\" (UniqueName: \"kubernetes.io/projected/ef4163bb-8632-4675-a678-11ec6d58e737-kube-api-access-zccwt\") pod \"ef4163bb-8632-4675-a678-11ec6d58e737\" (UID: \"ef4163bb-8632-4675-a678-11ec6d58e737\") " Nov 03 01:02:56.772037 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:02:56.772020 4939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ef4163bb-8632-4675-a678-11ec6d58e737-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ef4163bb-8632-4675-a678-11ec6d58e737-workload-certs\") pod \"ef4163bb-8632-4675-a678-11ec6d58e737\" (UID: \"ef4163bb-8632-4675-a678-11ec6d58e737\") " Nov 03 01:02:56.772144 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:02:56.772080 4939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/ef4163bb-8632-4675-a678-11ec6d58e737-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ef4163bb-8632-4675-a678-11ec6d58e737-istio-podinfo\") pod \"ef4163bb-8632-4675-a678-11ec6d58e737\" (UID: \"ef4163bb-8632-4675-a678-11ec6d58e737\") " Nov 03 01:02:56.772182 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:02:56.772172 4939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ef4163bb-8632-4675-a678-11ec6d58e737-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ef4163bb-8632-4675-a678-11ec6d58e737-istiod-ca-cert\") pod \"ef4163bb-8632-4675-a678-11ec6d58e737\" (UID: \"ef4163bb-8632-4675-a678-11ec6d58e737\") " Nov 03 01:02:56.773205 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:02:56.772701 4939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ef4163bb-8632-4675-a678-11ec6d58e737-workload-certs" pod "ef4163bb-8632-4675-a678-11ec6d58e737" (UID: "ef4163bb-8632-4675-a678-11ec6d58e737"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 01:02:56.773205 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:02:56.772994 4939 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ef4163bb-8632-4675-a678-11ec6d58e737-workload-certs\") on node \"ip-10-0-140-107.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:02:56.773205 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:02:56.773139 4939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ef4163bb-8632-4675-a678-11ec6d58e737-credential-socket" pod "ef4163bb-8632-4675-a678-11ec6d58e737" (UID: "ef4163bb-8632-4675-a678-11ec6d58e737"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 01:02:56.773205 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:02:56.773163 4939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ef4163bb-8632-4675-a678-11ec6d58e737-istiod-ca-cert" pod "ef4163bb-8632-4675-a678-11ec6d58e737" (UID: "ef4163bb-8632-4675-a678-11ec6d58e737"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 01:02:56.773205 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:02:56.773175 4939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ef4163bb-8632-4675-a678-11ec6d58e737-istio-data" pod "ef4163bb-8632-4675-a678-11ec6d58e737" (UID: "ef4163bb-8632-4675-a678-11ec6d58e737"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 01:02:56.773562 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:02:56.773541 4939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ef4163bb-8632-4675-a678-11ec6d58e737-workload-socket" pod "ef4163bb-8632-4675-a678-11ec6d58e737" (UID: "ef4163bb-8632-4675-a678-11ec6d58e737"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 01:02:56.776077 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:02:56.775999 4939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ef4163bb-8632-4675-a678-11ec6d58e737-kube-api-access-zccwt" pod "ef4163bb-8632-4675-a678-11ec6d58e737" (UID: "ef4163bb-8632-4675-a678-11ec6d58e737"). InnerVolumeSpecName "kube-api-access-zccwt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 01:02:56.776695 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:02:56.776674 4939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ef4163bb-8632-4675-a678-11ec6d58e737-istio-token" pod "ef4163bb-8632-4675-a678-11ec6d58e737" (UID: "ef4163bb-8632-4675-a678-11ec6d58e737"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 01:02:56.777135 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:02:56.777108 4939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ef4163bb-8632-4675-a678-11ec6d58e737-istio-envoy" pod "ef4163bb-8632-4675-a678-11ec6d58e737" (UID: "ef4163bb-8632-4675-a678-11ec6d58e737"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 01:02:56.777856 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:02:56.777837 4939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/ef4163bb-8632-4675-a678-11ec6d58e737-istio-podinfo" pod "ef4163bb-8632-4675-a678-11ec6d58e737" (UID: "ef4163bb-8632-4675-a678-11ec6d58e737"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 01:02:56.874696 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:02:56.874651 4939 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ef4163bb-8632-4675-a678-11ec6d58e737-credential-socket\") on node \"ip-10-0-140-107.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:02:56.874696 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:02:56.874686 4939 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ef4163bb-8632-4675-a678-11ec6d58e737-istio-data\") on node \"ip-10-0-140-107.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:02:56.874696 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:02:56.874701 4939 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zccwt\" (UniqueName: \"kubernetes.io/projected/ef4163bb-8632-4675-a678-11ec6d58e737-kube-api-access-zccwt\") on node \"ip-10-0-140-107.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:02:56.875023 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:02:56.874716 4939 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ef4163bb-8632-4675-a678-11ec6d58e737-istio-podinfo\") on node \"ip-10-0-140-107.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:02:56.875023 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:02:56.874729 4939 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ef4163bb-8632-4675-a678-11ec6d58e737-istiod-ca-cert\") on node \"ip-10-0-140-107.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:02:56.875023 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:02:56.874743 4939 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ef4163bb-8632-4675-a678-11ec6d58e737-istio-envoy\") on node \"ip-10-0-140-107.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:02:56.875023 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:02:56.874756 4939 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ef4163bb-8632-4675-a678-11ec6d58e737-workload-socket\") on node \"ip-10-0-140-107.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:02:56.875023 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:02:56.874770 4939 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ef4163bb-8632-4675-a678-11ec6d58e737-istio-token\") on node \"ip-10-0-140-107.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:02:57.043395 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:02:57.043305 4939 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-46510c-77cb98ffcb-4x68b"] Nov 03 01:02:57.043816 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:02:57.043788 4939 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-46510c-77cb98ffcb-4x68b" podUID="9cb13f72-a50c-49be-9490-fc347576d0d5" containerName="maas-api" containerID="cri-o://af773b53aac47ec47715af3ec4f5219c30f62bd2caed59ddfe2543bbfb7f0161" gracePeriod=30 Nov 03 01:02:57.084291 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:02:57.084259 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:02:57.201205 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:02:57.201069 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:02:57.214157 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:02:57.214126 4939 generic.go:356] "Generic (PLEG): container finished" podID="ef4163bb-8632-4675-a678-11ec6d58e737" containerID="52e11dadb127b4b182eca770f7958e3db330130f547828282776635d0e75bb2f" exitCode=0 Nov 03 01:02:57.214275 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:02:57.214232 4939 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-07f5f6-openshift-default-98746c89f-vz567" Nov 03 01:02:57.214275 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:02:57.214253 4939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-07f5f6-openshift-default-98746c89f-vz567" event={"ID":"ef4163bb-8632-4675-a678-11ec6d58e737","Type":"ContainerDied","Data":"52e11dadb127b4b182eca770f7958e3db330130f547828282776635d0e75bb2f"} Nov 03 01:02:57.214371 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:02:57.214311 4939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-07f5f6-openshift-default-98746c89f-vz567" event={"ID":"ef4163bb-8632-4675-a678-11ec6d58e737","Type":"ContainerDied","Data":"55f7f089ab857a160ad99afb348a8f5471ddc494f3adf5e72c56d6d613631322"} Nov 03 01:02:57.214371 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:02:57.214340 4939 scope.go:122] "RemoveContainer" containerID="52e11dadb127b4b182eca770f7958e3db330130f547828282776635d0e75bb2f" Nov 03 01:02:57.216311 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:02:57.216285 4939 generic.go:356] "Generic (PLEG): container finished" podID="9cb13f72-a50c-49be-9490-fc347576d0d5" containerID="af773b53aac47ec47715af3ec4f5219c30f62bd2caed59ddfe2543bbfb7f0161" exitCode=0 Nov 03 01:02:57.216443 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:02:57.216375 4939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-46510c-77cb98ffcb-4x68b" event={"ID":"9cb13f72-a50c-49be-9490-fc347576d0d5","Type":"ContainerDied","Data":"af773b53aac47ec47715af3ec4f5219c30f62bd2caed59ddfe2543bbfb7f0161"} Nov 03 01:02:57.256375 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:02:57.226347 4939 scope.go:122] "RemoveContainer" containerID="e5e40ad69efe3af5e0379a278b807446fbeb02e1055b85c53a413a32b985a78b" Nov 03 01:02:57.336226 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:02:57.334070 4939 scope.go:122] "RemoveContainer" containerID="52e11dadb127b4b182eca770f7958e3db330130f547828282776635d0e75bb2f" Nov 03 01:02:57.336226 ip-10-0-140-107 kubenswrapper[4939]: E1103 01:02:57.335048 4939 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"52e11dadb127b4b182eca770f7958e3db330130f547828282776635d0e75bb2f\": container with ID starting with 52e11dadb127b4b182eca770f7958e3db330130f547828282776635d0e75bb2f not found: ID does not exist" containerID="52e11dadb127b4b182eca770f7958e3db330130f547828282776635d0e75bb2f" Nov 03 01:02:57.336226 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:02:57.335086 4939 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"52e11dadb127b4b182eca770f7958e3db330130f547828282776635d0e75bb2f"} err="failed to get container status \"52e11dadb127b4b182eca770f7958e3db330130f547828282776635d0e75bb2f\": rpc error: code = NotFound desc = could not find container \"52e11dadb127b4b182eca770f7958e3db330130f547828282776635d0e75bb2f\": container with ID starting with 52e11dadb127b4b182eca770f7958e3db330130f547828282776635d0e75bb2f not found: ID does not exist" Nov 03 01:02:57.336226 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:02:57.335116 4939 scope.go:122] "RemoveContainer" containerID="e5e40ad69efe3af5e0379a278b807446fbeb02e1055b85c53a413a32b985a78b" Nov 03 01:02:57.354933 ip-10-0-140-107 kubenswrapper[4939]: E1103 01:02:57.352630 4939 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e5e40ad69efe3af5e0379a278b807446fbeb02e1055b85c53a413a32b985a78b\": container with ID starting with e5e40ad69efe3af5e0379a278b807446fbeb02e1055b85c53a413a32b985a78b not found: ID does not exist" containerID="e5e40ad69efe3af5e0379a278b807446fbeb02e1055b85c53a413a32b985a78b" Nov 03 01:02:57.354933 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:02:57.352679 4939 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e5e40ad69efe3af5e0379a278b807446fbeb02e1055b85c53a413a32b985a78b"} err="failed to get container status \"e5e40ad69efe3af5e0379a278b807446fbeb02e1055b85c53a413a32b985a78b\": rpc error: code = NotFound desc = could not find container \"e5e40ad69efe3af5e0379a278b807446fbeb02e1055b85c53a413a32b985a78b\": container with ID starting with e5e40ad69efe3af5e0379a278b807446fbeb02e1055b85c53a413a32b985a78b not found: ID does not exist" Nov 03 01:02:57.368009 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:02:57.367976 4939 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-07f5f6-openshift-default-98746c89f-vz567"] Nov 03 01:02:57.380935 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:02:57.372622 4939 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-a-07f5f6-openshift-default-98746c89f-vz567"] Nov 03 01:02:57.673094 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:02:57.670345 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:02:57.802866 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:02:57.802837 4939 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-46510c-77cb98ffcb-4x68b" Nov 03 01:02:57.883485 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:02:57.883442 4939 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ef4163bb-8632-4675-a678-11ec6d58e737" path="/var/lib/kubelet/pods/ef4163bb-8632-4675-a678-11ec6d58e737/volumes" Nov 03 01:02:57.906665 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:02:57.906637 4939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9cb13f72-a50c-49be-9490-fc347576d0d5-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9cb13f72-a50c-49be-9490-fc347576d0d5-openshift-service-ca\") pod \"9cb13f72-a50c-49be-9490-fc347576d0d5\" (UID: \"9cb13f72-a50c-49be-9490-fc347576d0d5\") " Nov 03 01:02:57.906828 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:02:57.906692 4939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9cb13f72-a50c-49be-9490-fc347576d0d5-metrics-certs\" (UniqueName: \"kubernetes.io/secret/9cb13f72-a50c-49be-9490-fc347576d0d5-metrics-certs\") pod \"9cb13f72-a50c-49be-9490-fc347576d0d5\" (UID: \"9cb13f72-a50c-49be-9490-fc347576d0d5\") " Nov 03 01:02:57.906952 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:02:57.906934 4939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9cb13f72-a50c-49be-9490-fc347576d0d5-kube-api-access-rvk5w\" (UniqueName: \"kubernetes.io/projected/9cb13f72-a50c-49be-9490-fc347576d0d5-kube-api-access-rvk5w\") pod \"9cb13f72-a50c-49be-9490-fc347576d0d5\" (UID: \"9cb13f72-a50c-49be-9490-fc347576d0d5\") " Nov 03 01:02:57.907025 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:02:57.906995 4939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9cb13f72-a50c-49be-9490-fc347576d0d5-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9cb13f72-a50c-49be-9490-fc347576d0d5-maas-api-tls\") pod \"9cb13f72-a50c-49be-9490-fc347576d0d5\" (UID: \"9cb13f72-a50c-49be-9490-fc347576d0d5\") " Nov 03 01:02:57.907142 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:02:57.907115 4939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9cb13f72-a50c-49be-9490-fc347576d0d5-openshift-service-ca" pod "9cb13f72-a50c-49be-9490-fc347576d0d5" (UID: "9cb13f72-a50c-49be-9490-fc347576d0d5"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 01:02:57.907579 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:02:57.907556 4939 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9cb13f72-a50c-49be-9490-fc347576d0d5-openshift-service-ca\") on node \"ip-10-0-140-107.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:02:57.909464 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:02:57.909434 4939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9cb13f72-a50c-49be-9490-fc347576d0d5-metrics-certs" pod "9cb13f72-a50c-49be-9490-fc347576d0d5" (UID: "9cb13f72-a50c-49be-9490-fc347576d0d5"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 01:02:57.909594 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:02:57.909554 4939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9cb13f72-a50c-49be-9490-fc347576d0d5-kube-api-access-rvk5w" pod "9cb13f72-a50c-49be-9490-fc347576d0d5" (UID: "9cb13f72-a50c-49be-9490-fc347576d0d5"). InnerVolumeSpecName "kube-api-access-rvk5w". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 01:02:57.909594 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:02:57.909584 4939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9cb13f72-a50c-49be-9490-fc347576d0d5-maas-api-tls" pod "9cb13f72-a50c-49be-9490-fc347576d0d5" (UID: "9cb13f72-a50c-49be-9490-fc347576d0d5"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 01:02:58.009592 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:02:58.009495 4939 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9cb13f72-a50c-49be-9490-fc347576d0d5-maas-api-tls\") on node \"ip-10-0-140-107.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:02:58.009592 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:02:58.009522 4939 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9cb13f72-a50c-49be-9490-fc347576d0d5-metrics-certs\") on node \"ip-10-0-140-107.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:02:58.009592 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:02:58.009533 4939 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rvk5w\" (UniqueName: \"kubernetes.io/projected/9cb13f72-a50c-49be-9490-fc347576d0d5-kube-api-access-rvk5w\") on node \"ip-10-0-140-107.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:02:58.224755 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:02:58.224717 4939 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-46510c-77cb98ffcb-4x68b" Nov 03 01:02:58.224962 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:02:58.224750 4939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-46510c-77cb98ffcb-4x68b" event={"ID":"9cb13f72-a50c-49be-9490-fc347576d0d5","Type":"ContainerDied","Data":"fff570da721e651ecc97a96ede9ac217465461b5b07aa1aad4fc6e7687276232"} Nov 03 01:02:58.224962 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:02:58.224814 4939 scope.go:122] "RemoveContainer" containerID="af773b53aac47ec47715af3ec4f5219c30f62bd2caed59ddfe2543bbfb7f0161" Nov 03 01:02:58.279556 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:02:58.279486 4939 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-46510c-77cb98ffcb-4x68b"] Nov 03 01:02:58.284406 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:02:58.284381 4939 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-46510c-77cb98ffcb-4x68b"] Nov 03 01:02:58.379561 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:02:58.379526 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:02:59.219796 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:02:59.219761 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:02:59.880399 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:02:59.880362 4939 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9cb13f72-a50c-49be-9490-fc347576d0d5" path="/var/lib/kubelet/pods/9cb13f72-a50c-49be-9490-fc347576d0d5/volumes" Nov 03 01:02:59.961934 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:02:59.960280 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:03:00.400665 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:03:00.400632 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:03:00.789955 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:03:00.789835 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:03:01.044377 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:03:01.044305 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:03:01.267992 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:03:01.267960 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:03:01.584525 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:03:01.584485 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:03:01.706766 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:03:01.706731 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:03:01.891983 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:03:01.890084 4939 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-praxis-b-46510c-openshift-default-849969b69c-9gxjp"] Nov 03 01:03:01.891983 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:03:01.890306 4939 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-praxis-b-46510c-openshift-default-849969b69c-9gxjp" podUID="a2aede6b-fd70-4df3-bec1-a716350a85c8" containerName="istio-proxy" containerID="cri-o://25c3f582db8e654ba3d466bfc18be83859e1becd529c2598d0904905836add4b" gracePeriod=30 Nov 03 01:03:01.897147 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:03:01.897119 4939 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-praxis-b-f14bda-openshift-default-7c65bb6969-rdtlg"] Nov 03 01:03:01.897959 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:03:01.897928 4939 cpu_manager.go:414] "containerMap: removing container" podUID="7bcdba6a-dab2-4616-82d1-2b0bf8770b8a" containerName="payload-processing" Nov 03 01:03:01.897959 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:03:01.897954 4939 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7bcdba6a-dab2-4616-82d1-2b0bf8770b8a" containerName="payload-processing" Nov 03 01:03:01.898139 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:03:01.897975 4939 cpu_manager.go:414] "containerMap: removing container" podUID="35fb0ddf-b9bf-4173-ae72-c0100ac754ad" containerName="payload-pre-processing" Nov 03 01:03:01.898139 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:03:01.897982 4939 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="35fb0ddf-b9bf-4173-ae72-c0100ac754ad" containerName="payload-pre-processing" Nov 03 01:03:01.898139 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:03:01.898003 4939 cpu_manager.go:414] "containerMap: removing container" podUID="ef4163bb-8632-4675-a678-11ec6d58e737" containerName="istio-proxy" Nov 03 01:03:01.898139 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:03:01.898011 4939 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ef4163bb-8632-4675-a678-11ec6d58e737" containerName="istio-proxy" Nov 03 01:03:01.898139 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:03:01.898024 4939 cpu_manager.go:414] "containerMap: removing container" podUID="9cb13f72-a50c-49be-9490-fc347576d0d5" containerName="maas-api" Nov 03 01:03:01.898139 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:03:01.898031 4939 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9cb13f72-a50c-49be-9490-fc347576d0d5" containerName="maas-api" Nov 03 01:03:01.898139 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:03:01.898080 4939 cpu_manager.go:414] "containerMap: removing container" podUID="dfea3192-3541-49c5-a491-ebf7308b74e0" containerName="istio-proxy" Nov 03 01:03:01.898139 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:03:01.898088 4939 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dfea3192-3541-49c5-a491-ebf7308b74e0" containerName="istio-proxy" Nov 03 01:03:01.898139 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:03:01.898116 4939 cpu_manager.go:414] "containerMap: removing container" podUID="dfea3192-3541-49c5-a491-ebf7308b74e0" containerName="istio-proxy" Nov 03 01:03:01.898139 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:03:01.898123 4939 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dfea3192-3541-49c5-a491-ebf7308b74e0" containerName="istio-proxy" Nov 03 01:03:01.898723 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:03:01.898291 4939 memory_manager.go:365] "RemoveStaleState removing state" podUID="ef4163bb-8632-4675-a678-11ec6d58e737" containerName="istio-proxy" Nov 03 01:03:01.898723 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:03:01.898305 4939 memory_manager.go:365] "RemoveStaleState removing state" podUID="dfea3192-3541-49c5-a491-ebf7308b74e0" containerName="istio-proxy" Nov 03 01:03:01.898723 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:03:01.898319 4939 memory_manager.go:365] "RemoveStaleState removing state" podUID="dfea3192-3541-49c5-a491-ebf7308b74e0" containerName="istio-proxy" Nov 03 01:03:01.898723 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:03:01.898332 4939 memory_manager.go:365] "RemoveStaleState removing state" podUID="7bcdba6a-dab2-4616-82d1-2b0bf8770b8a" containerName="payload-processing" Nov 03 01:03:01.898723 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:03:01.898341 4939 memory_manager.go:365] "RemoveStaleState removing state" podUID="35fb0ddf-b9bf-4173-ae72-c0100ac754ad" containerName="payload-pre-processing" Nov 03 01:03:01.898723 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:03:01.898354 4939 memory_manager.go:365] "RemoveStaleState removing state" podUID="9cb13f72-a50c-49be-9490-fc347576d0d5" containerName="maas-api" Nov 03 01:03:01.905550 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:03:01.905523 4939 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-b-f14bda-openshift-default-7c65bb6969-rdtlg" Nov 03 01:03:01.929096 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:03:01.918224 4939 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-praxis-b-f14bda-openshift-default-dockercfg-dv7sl\"" Nov 03 01:03:01.939076 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:03:01.939049 4939 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-praxis-b-f14bda-openshift-default-7c65bb6969-rdtlg"] Nov 03 01:03:01.963271 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:03:01.963236 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:03:02.033957 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:03:02.028925 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mck9s\" (UniqueName: \"kubernetes.io/projected/959b27c0-16c2-41c2-989c-d86f12cedee9-kube-api-access-mck9s\") pod \"e2e-praxis-b-f14bda-openshift-default-7c65bb6969-rdtlg\" (UID: \"959b27c0-16c2-41c2-989c-d86f12cedee9\") " pod="openshift-ingress/e2e-praxis-b-f14bda-openshift-default-7c65bb6969-rdtlg" Nov 03 01:03:02.033957 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:03:02.029027 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/959b27c0-16c2-41c2-989c-d86f12cedee9-istio-envoy\") pod \"e2e-praxis-b-f14bda-openshift-default-7c65bb6969-rdtlg\" (UID: \"959b27c0-16c2-41c2-989c-d86f12cedee9\") " pod="openshift-ingress/e2e-praxis-b-f14bda-openshift-default-7c65bb6969-rdtlg" Nov 03 01:03:02.033957 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:03:02.029062 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/959b27c0-16c2-41c2-989c-d86f12cedee9-istiod-ca-cert\") pod \"e2e-praxis-b-f14bda-openshift-default-7c65bb6969-rdtlg\" (UID: \"959b27c0-16c2-41c2-989c-d86f12cedee9\") " pod="openshift-ingress/e2e-praxis-b-f14bda-openshift-default-7c65bb6969-rdtlg" Nov 03 01:03:02.033957 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:03:02.029198 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/959b27c0-16c2-41c2-989c-d86f12cedee9-workload-socket\") pod \"e2e-praxis-b-f14bda-openshift-default-7c65bb6969-rdtlg\" (UID: \"959b27c0-16c2-41c2-989c-d86f12cedee9\") " pod="openshift-ingress/e2e-praxis-b-f14bda-openshift-default-7c65bb6969-rdtlg" Nov 03 01:03:02.033957 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:03:02.029246 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/959b27c0-16c2-41c2-989c-d86f12cedee9-istio-data\") pod \"e2e-praxis-b-f14bda-openshift-default-7c65bb6969-rdtlg\" (UID: \"959b27c0-16c2-41c2-989c-d86f12cedee9\") " pod="openshift-ingress/e2e-praxis-b-f14bda-openshift-default-7c65bb6969-rdtlg" Nov 03 01:03:02.033957 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:03:02.029287 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/959b27c0-16c2-41c2-989c-d86f12cedee9-credential-socket\") pod \"e2e-praxis-b-f14bda-openshift-default-7c65bb6969-rdtlg\" (UID: \"959b27c0-16c2-41c2-989c-d86f12cedee9\") " pod="openshift-ingress/e2e-praxis-b-f14bda-openshift-default-7c65bb6969-rdtlg" Nov 03 01:03:02.033957 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:03:02.029345 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/959b27c0-16c2-41c2-989c-d86f12cedee9-istio-podinfo\") pod \"e2e-praxis-b-f14bda-openshift-default-7c65bb6969-rdtlg\" (UID: \"959b27c0-16c2-41c2-989c-d86f12cedee9\") " pod="openshift-ingress/e2e-praxis-b-f14bda-openshift-default-7c65bb6969-rdtlg" Nov 03 01:03:02.033957 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:03:02.029402 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/959b27c0-16c2-41c2-989c-d86f12cedee9-istio-token\") pod \"e2e-praxis-b-f14bda-openshift-default-7c65bb6969-rdtlg\" (UID: \"959b27c0-16c2-41c2-989c-d86f12cedee9\") " pod="openshift-ingress/e2e-praxis-b-f14bda-openshift-default-7c65bb6969-rdtlg" Nov 03 01:03:02.033957 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:03:02.029447 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/959b27c0-16c2-41c2-989c-d86f12cedee9-workload-certs\") pod \"e2e-praxis-b-f14bda-openshift-default-7c65bb6969-rdtlg\" (UID: \"959b27c0-16c2-41c2-989c-d86f12cedee9\") " pod="openshift-ingress/e2e-praxis-b-f14bda-openshift-default-7c65bb6969-rdtlg" Nov 03 01:03:02.157980 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:03:02.150749 4939 patch_prober.go:29] interesting pod/e2e-praxis-b-46510c-openshift-default-849969b69c-9gxjp container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=upstream connect error or disconnect/reset before headers. reset reason: remote connection failure, transport failure reason: delayed connect error: Connection refused Nov 03 01:03:02.157980 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:03:02.150835 4939 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-praxis-b-46510c-openshift-default-849969b69c-9gxjp" podUID="a2aede6b-fd70-4df3-bec1-a716350a85c8" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 03 01:03:02.157980 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:03:02.155874 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mck9s\" (UniqueName: \"kubernetes.io/projected/959b27c0-16c2-41c2-989c-d86f12cedee9-kube-api-access-mck9s\") pod \"e2e-praxis-b-f14bda-openshift-default-7c65bb6969-rdtlg\" (UID: \"959b27c0-16c2-41c2-989c-d86f12cedee9\") " pod="openshift-ingress/e2e-praxis-b-f14bda-openshift-default-7c65bb6969-rdtlg" Nov 03 01:03:02.157980 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:03:02.155994 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/959b27c0-16c2-41c2-989c-d86f12cedee9-istio-envoy\") pod \"e2e-praxis-b-f14bda-openshift-default-7c65bb6969-rdtlg\" (UID: \"959b27c0-16c2-41c2-989c-d86f12cedee9\") " pod="openshift-ingress/e2e-praxis-b-f14bda-openshift-default-7c65bb6969-rdtlg" Nov 03 01:03:02.157980 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:03:02.156024 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/959b27c0-16c2-41c2-989c-d86f12cedee9-istiod-ca-cert\") pod \"e2e-praxis-b-f14bda-openshift-default-7c65bb6969-rdtlg\" (UID: \"959b27c0-16c2-41c2-989c-d86f12cedee9\") " pod="openshift-ingress/e2e-praxis-b-f14bda-openshift-default-7c65bb6969-rdtlg" Nov 03 01:03:02.157980 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:03:02.156181 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/959b27c0-16c2-41c2-989c-d86f12cedee9-workload-socket\") pod \"e2e-praxis-b-f14bda-openshift-default-7c65bb6969-rdtlg\" (UID: \"959b27c0-16c2-41c2-989c-d86f12cedee9\") " pod="openshift-ingress/e2e-praxis-b-f14bda-openshift-default-7c65bb6969-rdtlg" Nov 03 01:03:02.157980 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:03:02.156236 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/959b27c0-16c2-41c2-989c-d86f12cedee9-istio-data\") pod \"e2e-praxis-b-f14bda-openshift-default-7c65bb6969-rdtlg\" (UID: \"959b27c0-16c2-41c2-989c-d86f12cedee9\") " pod="openshift-ingress/e2e-praxis-b-f14bda-openshift-default-7c65bb6969-rdtlg" Nov 03 01:03:02.157980 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:03:02.156263 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/959b27c0-16c2-41c2-989c-d86f12cedee9-credential-socket\") pod \"e2e-praxis-b-f14bda-openshift-default-7c65bb6969-rdtlg\" (UID: \"959b27c0-16c2-41c2-989c-d86f12cedee9\") " pod="openshift-ingress/e2e-praxis-b-f14bda-openshift-default-7c65bb6969-rdtlg" Nov 03 01:03:02.157980 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:03:02.156304 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/959b27c0-16c2-41c2-989c-d86f12cedee9-istio-podinfo\") pod \"e2e-praxis-b-f14bda-openshift-default-7c65bb6969-rdtlg\" (UID: \"959b27c0-16c2-41c2-989c-d86f12cedee9\") " pod="openshift-ingress/e2e-praxis-b-f14bda-openshift-default-7c65bb6969-rdtlg" Nov 03 01:03:02.157980 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:03:02.156357 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/959b27c0-16c2-41c2-989c-d86f12cedee9-istio-token\") pod \"e2e-praxis-b-f14bda-openshift-default-7c65bb6969-rdtlg\" (UID: \"959b27c0-16c2-41c2-989c-d86f12cedee9\") " pod="openshift-ingress/e2e-praxis-b-f14bda-openshift-default-7c65bb6969-rdtlg" Nov 03 01:03:02.157980 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:03:02.156401 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/959b27c0-16c2-41c2-989c-d86f12cedee9-workload-certs\") pod \"e2e-praxis-b-f14bda-openshift-default-7c65bb6969-rdtlg\" (UID: \"959b27c0-16c2-41c2-989c-d86f12cedee9\") " pod="openshift-ingress/e2e-praxis-b-f14bda-openshift-default-7c65bb6969-rdtlg" Nov 03 01:03:02.157980 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:03:02.156876 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/959b27c0-16c2-41c2-989c-d86f12cedee9-workload-certs\") pod \"e2e-praxis-b-f14bda-openshift-default-7c65bb6969-rdtlg\" (UID: \"959b27c0-16c2-41c2-989c-d86f12cedee9\") " pod="openshift-ingress/e2e-praxis-b-f14bda-openshift-default-7c65bb6969-rdtlg" Nov 03 01:03:02.162991 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:03:02.160732 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/959b27c0-16c2-41c2-989c-d86f12cedee9-istio-data\") pod \"e2e-praxis-b-f14bda-openshift-default-7c65bb6969-rdtlg\" (UID: \"959b27c0-16c2-41c2-989c-d86f12cedee9\") " pod="openshift-ingress/e2e-praxis-b-f14bda-openshift-default-7c65bb6969-rdtlg" Nov 03 01:03:02.162991 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:03:02.161437 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/959b27c0-16c2-41c2-989c-d86f12cedee9-istiod-ca-cert\") pod \"e2e-praxis-b-f14bda-openshift-default-7c65bb6969-rdtlg\" (UID: \"959b27c0-16c2-41c2-989c-d86f12cedee9\") " pod="openshift-ingress/e2e-praxis-b-f14bda-openshift-default-7c65bb6969-rdtlg" Nov 03 01:03:02.162991 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:03:02.161687 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/959b27c0-16c2-41c2-989c-d86f12cedee9-workload-socket\") pod \"e2e-praxis-b-f14bda-openshift-default-7c65bb6969-rdtlg\" (UID: \"959b27c0-16c2-41c2-989c-d86f12cedee9\") " pod="openshift-ingress/e2e-praxis-b-f14bda-openshift-default-7c65bb6969-rdtlg" Nov 03 01:03:02.164582 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:03:02.164558 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/959b27c0-16c2-41c2-989c-d86f12cedee9-credential-socket\") pod \"e2e-praxis-b-f14bda-openshift-default-7c65bb6969-rdtlg\" (UID: \"959b27c0-16c2-41c2-989c-d86f12cedee9\") " pod="openshift-ingress/e2e-praxis-b-f14bda-openshift-default-7c65bb6969-rdtlg" Nov 03 01:03:02.175492 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:03:02.175431 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mck9s\" (UniqueName: \"kubernetes.io/projected/959b27c0-16c2-41c2-989c-d86f12cedee9-kube-api-access-mck9s\") pod \"e2e-praxis-b-f14bda-openshift-default-7c65bb6969-rdtlg\" (UID: \"959b27c0-16c2-41c2-989c-d86f12cedee9\") " pod="openshift-ingress/e2e-praxis-b-f14bda-openshift-default-7c65bb6969-rdtlg" Nov 03 01:03:02.176453 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:03:02.176381 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:03:02.180256 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:03:02.180213 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/959b27c0-16c2-41c2-989c-d86f12cedee9-istio-token\") pod \"e2e-praxis-b-f14bda-openshift-default-7c65bb6969-rdtlg\" (UID: \"959b27c0-16c2-41c2-989c-d86f12cedee9\") " pod="openshift-ingress/e2e-praxis-b-f14bda-openshift-default-7c65bb6969-rdtlg" Nov 03 01:03:02.208002 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:03:02.205282 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/959b27c0-16c2-41c2-989c-d86f12cedee9-istio-podinfo\") pod \"e2e-praxis-b-f14bda-openshift-default-7c65bb6969-rdtlg\" (UID: \"959b27c0-16c2-41c2-989c-d86f12cedee9\") " pod="openshift-ingress/e2e-praxis-b-f14bda-openshift-default-7c65bb6969-rdtlg" Nov 03 01:03:02.257930 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:03:02.254480 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/959b27c0-16c2-41c2-989c-d86f12cedee9-istio-envoy\") pod \"e2e-praxis-b-f14bda-openshift-default-7c65bb6969-rdtlg\" (UID: \"959b27c0-16c2-41c2-989c-d86f12cedee9\") " pod="openshift-ingress/e2e-praxis-b-f14bda-openshift-default-7c65bb6969-rdtlg" Nov 03 01:03:02.292822 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:03:02.291076 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:03:02.550982 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:03:02.550817 4939 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-b-f14bda-openshift-default-7c65bb6969-rdtlg" Nov 03 01:03:02.843811 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:03:02.843735 4939 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-praxis-b-f14bda-openshift-default-7c65bb6969-rdtlg"] Nov 03 01:03:02.846356 ip-10-0-140-107 kubenswrapper[4939]: W1103 01:03:02.846321 4939 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod959b27c0_16c2_41c2_989c_d86f12cedee9.slice/crio-4f69ecc869bdcb8c53ee6824847f79025e2be8b5e5d152b98e0b397a1229006d WatchSource:0}: Error finding container 4f69ecc869bdcb8c53ee6824847f79025e2be8b5e5d152b98e0b397a1229006d: Status 404 returned error can't find the container with id 4f69ecc869bdcb8c53ee6824847f79025e2be8b5e5d152b98e0b397a1229006d Nov 03 01:03:02.849394 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:03:02.849351 4939 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 01:03:02.849506 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:03:02.849428 4939 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 01:03:02.849506 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:03:02.849462 4939 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 01:03:03.089349 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:03:03.089280 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:03:03.301203 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:03:03.301105 4939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-f14bda-openshift-default-7c65bb6969-rdtlg" event={"ID":"959b27c0-16c2-41c2-989c-d86f12cedee9","Type":"ContainerStarted","Data":"366bf518257472a8114c5031209c632fe88cd3a6bfc6a7cc0175d0edc46192ab"} Nov 03 01:03:03.301203 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:03:03.301168 4939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-f14bda-openshift-default-7c65bb6969-rdtlg" event={"ID":"959b27c0-16c2-41c2-989c-d86f12cedee9","Type":"ContainerStarted","Data":"4f69ecc869bdcb8c53ee6824847f79025e2be8b5e5d152b98e0b397a1229006d"} Nov 03 01:03:03.325965 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:03:03.325890 4939 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-praxis-b-f14bda-openshift-default-7c65bb6969-rdtlg" podStartSLOduration=2.325871581 podStartE2EDuration="2.325871581s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 01:03:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 01:03:03.322444052 +0000 UTC m=+1331.586778999" watchObservedRunningTime="2026-11-03 01:03:03.325871581 +0000 UTC m=+1331.590206527" Nov 03 01:03:03.551337 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:03:03.551249 4939 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-praxis-b-f14bda-openshift-default-7c65bb6969-rdtlg" Nov 03 01:03:03.557386 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:03:03.557358 4939 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-praxis-b-f14bda-openshift-default-7c65bb6969-rdtlg" Nov 03 01:03:03.601307 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:03:03.601259 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:03:03.991222 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:03:03.991188 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:03:04.309116 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:03:04.309031 4939 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-praxis-b-f14bda-openshift-default-7c65bb6969-rdtlg" Nov 03 01:03:04.309982 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:03:04.309959 4939 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-praxis-b-f14bda-openshift-default-7c65bb6969-rdtlg" Nov 03 01:03:04.853566 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:03:04.853483 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:03:04.963219 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:03:04.963186 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:03:05.366037 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:03:05.365994 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:03:05.578501 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:03:05.578443 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:03:05.763852 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:03:05.763759 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:03:06.325299 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:03:06.325249 4939 generic.go:356] "Generic (PLEG): container finished" podID="40f81e1f-3df1-4802-9636-b2ab2263f0ae" containerID="1832905ec9ae5b736eba433d7706628698036e63e972c24219ab929e0b6ff842" exitCode=0 Nov 03 01:03:06.325591 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:03:06.325324 4939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-b9dad7-openshift-default-994bfcbb9-bw5wc" event={"ID":"40f81e1f-3df1-4802-9636-b2ab2263f0ae","Type":"ContainerDied","Data":"1832905ec9ae5b736eba433d7706628698036e63e972c24219ab929e0b6ff842"} Nov 03 01:03:06.326005 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:03:06.325983 4939 scope.go:122] "RemoveContainer" containerID="1832905ec9ae5b736eba433d7706628698036e63e972c24219ab929e0b6ff842" Nov 03 01:03:06.327463 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:03:06.327435 4939 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 01:03:06.327533 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:03:06.327504 4939 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 01:03:06.327572 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:03:06.327534 4939 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 01:03:06.681608 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:03:06.681578 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:03:06.821954 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:03:06.821895 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:03:07.336984 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:03:07.336950 4939 generic.go:356] "Generic (PLEG): container finished" podID="a2aede6b-fd70-4df3-bec1-a716350a85c8" containerID="25c3f582db8e654ba3d466bfc18be83859e1becd529c2598d0904905836add4b" exitCode=0 Nov 03 01:03:07.337174 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:03:07.337025 4939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-46510c-openshift-default-849969b69c-9gxjp" event={"ID":"a2aede6b-fd70-4df3-bec1-a716350a85c8","Type":"ContainerDied","Data":"25c3f582db8e654ba3d466bfc18be83859e1becd529c2598d0904905836add4b"} Nov 03 01:03:07.337174 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:03:07.337114 4939 scope.go:122] "RemoveContainer" containerID="351ae7d796b0fdfb5006b2caca4fa98abf7cedf4b9ad8b37b1819eca7f3f372a" Nov 03 01:03:07.340623 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:03:07.340600 4939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-b9dad7-openshift-default-994bfcbb9-bw5wc" event={"ID":"40f81e1f-3df1-4802-9636-b2ab2263f0ae","Type":"ContainerStarted","Data":"ddbdc7e8f3c9a2d14e8931c42077f4b592c2d0714bc4e4c025b57807a7a51baf"} Nov 03 01:03:07.416547 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:03:07.416518 4939 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-praxis-a-b9dad7-openshift-default-994bfcbb9-bw5wc" Nov 03 01:03:07.417800 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:03:07.417215 4939 patch_prober.go:29] interesting pod/e2e-praxis-a-b9dad7-openshift-default-994bfcbb9-bw5wc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.153:15021/healthz/ready\": dial tcp 10.132.0.153:15021: connect: connection refused" start-of-body= Nov 03 01:03:07.417800 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:03:07.417258 4939 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-b9dad7-openshift-default-994bfcbb9-bw5wc" podUID="40f81e1f-3df1-4802-9636-b2ab2263f0ae" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.153:15021/healthz/ready\": dial tcp 10.132.0.153:15021: connect: connection refused" Nov 03 01:03:07.516176 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:03:07.516152 4939 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-b-46510c-openshift-default-849969b69c-9gxjp" Nov 03 01:03:07.541180 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:03:07.541103 4939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a2aede6b-fd70-4df3-bec1-a716350a85c8-istio-token\" (UniqueName: \"kubernetes.io/projected/a2aede6b-fd70-4df3-bec1-a716350a85c8-istio-token\") pod \"a2aede6b-fd70-4df3-bec1-a716350a85c8\" (UID: \"a2aede6b-fd70-4df3-bec1-a716350a85c8\") " Nov 03 01:03:07.541331 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:03:07.541200 4939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a2aede6b-fd70-4df3-bec1-a716350a85c8-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a2aede6b-fd70-4df3-bec1-a716350a85c8-istio-envoy\") pod \"a2aede6b-fd70-4df3-bec1-a716350a85c8\" (UID: \"a2aede6b-fd70-4df3-bec1-a716350a85c8\") " Nov 03 01:03:07.541331 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:03:07.541222 4939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a2aede6b-fd70-4df3-bec1-a716350a85c8-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a2aede6b-fd70-4df3-bec1-a716350a85c8-workload-socket\") pod \"a2aede6b-fd70-4df3-bec1-a716350a85c8\" (UID: \"a2aede6b-fd70-4df3-bec1-a716350a85c8\") " Nov 03 01:03:07.541331 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:03:07.541264 4939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a2aede6b-fd70-4df3-bec1-a716350a85c8-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a2aede6b-fd70-4df3-bec1-a716350a85c8-istio-data\") pod \"a2aede6b-fd70-4df3-bec1-a716350a85c8\" (UID: \"a2aede6b-fd70-4df3-bec1-a716350a85c8\") " Nov 03 01:03:07.541331 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:03:07.541286 4939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a2aede6b-fd70-4df3-bec1-a716350a85c8-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a2aede6b-fd70-4df3-bec1-a716350a85c8-workload-certs\") pod \"a2aede6b-fd70-4df3-bec1-a716350a85c8\" (UID: \"a2aede6b-fd70-4df3-bec1-a716350a85c8\") " Nov 03 01:03:07.541542 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:03:07.541406 4939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a2aede6b-fd70-4df3-bec1-a716350a85c8-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a2aede6b-fd70-4df3-bec1-a716350a85c8-credential-socket\") pod \"a2aede6b-fd70-4df3-bec1-a716350a85c8\" (UID: \"a2aede6b-fd70-4df3-bec1-a716350a85c8\") " Nov 03 01:03:07.541542 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:03:07.541455 4939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/a2aede6b-fd70-4df3-bec1-a716350a85c8-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a2aede6b-fd70-4df3-bec1-a716350a85c8-istio-podinfo\") pod \"a2aede6b-fd70-4df3-bec1-a716350a85c8\" (UID: \"a2aede6b-fd70-4df3-bec1-a716350a85c8\") " Nov 03 01:03:07.541542 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:03:07.541490 4939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a2aede6b-fd70-4df3-bec1-a716350a85c8-kube-api-access-nt29x\" (UniqueName: \"kubernetes.io/projected/a2aede6b-fd70-4df3-bec1-a716350a85c8-kube-api-access-nt29x\") pod \"a2aede6b-fd70-4df3-bec1-a716350a85c8\" (UID: \"a2aede6b-fd70-4df3-bec1-a716350a85c8\") " Nov 03 01:03:07.541698 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:03:07.541563 4939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a2aede6b-fd70-4df3-bec1-a716350a85c8-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a2aede6b-fd70-4df3-bec1-a716350a85c8-istiod-ca-cert\") pod \"a2aede6b-fd70-4df3-bec1-a716350a85c8\" (UID: \"a2aede6b-fd70-4df3-bec1-a716350a85c8\") " Nov 03 01:03:07.541698 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:03:07.541677 4939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a2aede6b-fd70-4df3-bec1-a716350a85c8-credential-socket" pod "a2aede6b-fd70-4df3-bec1-a716350a85c8" (UID: "a2aede6b-fd70-4df3-bec1-a716350a85c8"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 01:03:07.541923 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:03:07.541865 4939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a2aede6b-fd70-4df3-bec1-a716350a85c8-workload-certs" pod "a2aede6b-fd70-4df3-bec1-a716350a85c8" (UID: "a2aede6b-fd70-4df3-bec1-a716350a85c8"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 01:03:07.542068 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:03:07.542043 4939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a2aede6b-fd70-4df3-bec1-a716350a85c8-istiod-ca-cert" pod "a2aede6b-fd70-4df3-bec1-a716350a85c8" (UID: "a2aede6b-fd70-4df3-bec1-a716350a85c8"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 01:03:07.542518 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:03:07.542492 4939 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a2aede6b-fd70-4df3-bec1-a716350a85c8-workload-certs\") on node \"ip-10-0-140-107.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:03:07.542619 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:03:07.542523 4939 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a2aede6b-fd70-4df3-bec1-a716350a85c8-credential-socket\") on node \"ip-10-0-140-107.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:03:07.542619 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:03:07.542536 4939 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a2aede6b-fd70-4df3-bec1-a716350a85c8-istiod-ca-cert\") on node \"ip-10-0-140-107.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:03:07.542733 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:03:07.542682 4939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a2aede6b-fd70-4df3-bec1-a716350a85c8-workload-socket" pod "a2aede6b-fd70-4df3-bec1-a716350a85c8" (UID: "a2aede6b-fd70-4df3-bec1-a716350a85c8"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 01:03:07.545315 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:03:07.545204 4939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/a2aede6b-fd70-4df3-bec1-a716350a85c8-istio-podinfo" pod "a2aede6b-fd70-4df3-bec1-a716350a85c8" (UID: "a2aede6b-fd70-4df3-bec1-a716350a85c8"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 01:03:07.545315 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:03:07.545221 4939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a2aede6b-fd70-4df3-bec1-a716350a85c8-istio-envoy" pod "a2aede6b-fd70-4df3-bec1-a716350a85c8" (UID: "a2aede6b-fd70-4df3-bec1-a716350a85c8"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 01:03:07.545315 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:03:07.545285 4939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a2aede6b-fd70-4df3-bec1-a716350a85c8-istio-token" pod "a2aede6b-fd70-4df3-bec1-a716350a85c8" (UID: "a2aede6b-fd70-4df3-bec1-a716350a85c8"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 01:03:07.545315 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:03:07.545311 4939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a2aede6b-fd70-4df3-bec1-a716350a85c8-kube-api-access-nt29x" pod "a2aede6b-fd70-4df3-bec1-a716350a85c8" (UID: "a2aede6b-fd70-4df3-bec1-a716350a85c8"). InnerVolumeSpecName "kube-api-access-nt29x". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 01:03:07.545619 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:03:07.545505 4939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a2aede6b-fd70-4df3-bec1-a716350a85c8-istio-data" pod "a2aede6b-fd70-4df3-bec1-a716350a85c8" (UID: "a2aede6b-fd70-4df3-bec1-a716350a85c8"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 01:03:07.644110 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:03:07.644079 4939 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a2aede6b-fd70-4df3-bec1-a716350a85c8-istio-token\") on node \"ip-10-0-140-107.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:03:07.644110 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:03:07.644105 4939 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a2aede6b-fd70-4df3-bec1-a716350a85c8-istio-envoy\") on node \"ip-10-0-140-107.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:03:07.644110 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:03:07.644115 4939 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a2aede6b-fd70-4df3-bec1-a716350a85c8-workload-socket\") on node \"ip-10-0-140-107.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:03:07.644344 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:03:07.644131 4939 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a2aede6b-fd70-4df3-bec1-a716350a85c8-istio-data\") on node \"ip-10-0-140-107.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:03:07.644344 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:03:07.644140 4939 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a2aede6b-fd70-4df3-bec1-a716350a85c8-istio-podinfo\") on node \"ip-10-0-140-107.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:03:07.644344 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:03:07.644149 4939 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nt29x\" (UniqueName: \"kubernetes.io/projected/a2aede6b-fd70-4df3-bec1-a716350a85c8-kube-api-access-nt29x\") on node \"ip-10-0-140-107.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:03:07.782312 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:03:07.782226 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:03:08.374063 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:03:08.374019 4939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-46510c-openshift-default-849969b69c-9gxjp" event={"ID":"a2aede6b-fd70-4df3-bec1-a716350a85c8","Type":"ContainerDied","Data":"e059e308e00831f2ca532c1bf8266aaf4519f70c7885c54028ad66b781b7b59d"} Nov 03 01:03:08.374262 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:03:08.374069 4939 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-b-46510c-openshift-default-849969b69c-9gxjp" Nov 03 01:03:08.374262 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:03:08.374082 4939 scope.go:122] "RemoveContainer" containerID="25c3f582db8e654ba3d466bfc18be83859e1becd529c2598d0904905836add4b" Nov 03 01:03:08.396478 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:03:08.396450 4939 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-praxis-b-46510c-openshift-default-849969b69c-9gxjp"] Nov 03 01:03:08.399587 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:03:08.399562 4939 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-praxis-b-46510c-openshift-default-849969b69c-9gxjp"] Nov 03 01:03:08.416075 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:03:08.416050 4939 patch_prober.go:29] interesting pod/e2e-praxis-a-b9dad7-openshift-default-994bfcbb9-bw5wc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.153:15021/healthz/ready\": dial tcp 10.132.0.153:15021: connect: connection refused" start-of-body= Nov 03 01:03:08.416204 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:03:08.416101 4939 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-b9dad7-openshift-default-994bfcbb9-bw5wc" podUID="40f81e1f-3df1-4802-9636-b2ab2263f0ae" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.153:15021/healthz/ready\": dial tcp 10.132.0.153:15021: connect: connection refused" Nov 03 01:03:08.734476 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:03:08.734392 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:03:09.416330 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:03:09.416290 4939 patch_prober.go:29] interesting pod/e2e-praxis-a-b9dad7-openshift-default-994bfcbb9-bw5wc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.153:15021/healthz/ready\": dial tcp 10.132.0.153:15021: connect: connection refused" start-of-body= Nov 03 01:03:09.416719 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:03:09.416352 4939 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-b9dad7-openshift-default-994bfcbb9-bw5wc" podUID="40f81e1f-3df1-4802-9636-b2ab2263f0ae" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.153:15021/healthz/ready\": dial tcp 10.132.0.153:15021: connect: connection refused" Nov 03 01:03:09.504979 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:03:09.504946 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:03:09.725652 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:03:09.725562 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:03:09.875259 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:03:09.875212 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:03:09.881555 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:03:09.881525 4939 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a2aede6b-fd70-4df3-bec1-a716350a85c8" path="/var/lib/kubelet/pods/a2aede6b-fd70-4df3-bec1-a716350a85c8/volumes" Nov 03 01:03:10.172770 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:03:10.172735 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:03:10.415997 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:03:10.415963 4939 patch_prober.go:29] interesting pod/e2e-praxis-a-b9dad7-openshift-default-994bfcbb9-bw5wc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.153:15021/healthz/ready\": dial tcp 10.132.0.153:15021: connect: connection refused" start-of-body= Nov 03 01:03:10.416162 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:03:10.416023 4939 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-b9dad7-openshift-default-994bfcbb9-bw5wc" podUID="40f81e1f-3df1-4802-9636-b2ab2263f0ae" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.153:15021/healthz/ready\": dial tcp 10.132.0.153:15021: connect: connection refused" Nov 03 01:03:10.532241 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:03:10.532156 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:03:10.966858 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:03:10.966820 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:03:11.416017 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:03:11.415979 4939 patch_prober.go:29] interesting pod/e2e-praxis-a-b9dad7-openshift-default-994bfcbb9-bw5wc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.153:15021/healthz/ready\": dial tcp 10.132.0.153:15021: connect: connection refused" start-of-body= Nov 03 01:03:11.416196 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:03:11.416041 4939 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-b9dad7-openshift-default-994bfcbb9-bw5wc" podUID="40f81e1f-3df1-4802-9636-b2ab2263f0ae" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.153:15021/healthz/ready\": dial tcp 10.132.0.153:15021: connect: connection refused" Nov 03 01:03:11.732315 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:03:11.732234 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:03:12.415973 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:03:12.415941 4939 patch_prober.go:29] interesting pod/e2e-praxis-a-b9dad7-openshift-default-994bfcbb9-bw5wc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.153:15021/healthz/ready\": dial tcp 10.132.0.153:15021: connect: connection refused" start-of-body= Nov 03 01:03:12.416166 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:03:12.415998 4939 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-b9dad7-openshift-default-994bfcbb9-bw5wc" podUID="40f81e1f-3df1-4802-9636-b2ab2263f0ae" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.153:15021/healthz/ready\": dial tcp 10.132.0.153:15021: connect: connection refused" Nov 03 01:03:12.429091 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:03:12.429061 4939 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-dc0f3f-656999868b-4bk2z"] Nov 03 01:03:12.429373 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:03:12.429347 4939 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-dc0f3f-656999868b-4bk2z" podUID="61c842b8-bf22-49a5-b873-0b22bd049dd8" containerName="maas-api" containerID="cri-o://d317352158b7a37af0263136970f841a17c8e59cbb98371fbe6cf526755a98d9" gracePeriod=30 Nov 03 01:03:12.555215 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:03:12.555183 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:03:12.667565 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:03:12.667350 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:03:12.973192 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:03:12.973128 4939 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-dc0f3f-656999868b-4bk2z" Nov 03 01:03:13.076627 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:03:13.076596 4939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/61c842b8-bf22-49a5-b873-0b22bd049dd8-metrics-certs\" (UniqueName: \"kubernetes.io/secret/61c842b8-bf22-49a5-b873-0b22bd049dd8-metrics-certs\") pod \"61c842b8-bf22-49a5-b873-0b22bd049dd8\" (UID: \"61c842b8-bf22-49a5-b873-0b22bd049dd8\") " Nov 03 01:03:13.076827 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:03:13.076753 4939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/61c842b8-bf22-49a5-b873-0b22bd049dd8-kube-api-access-6stmq\" (UniqueName: \"kubernetes.io/projected/61c842b8-bf22-49a5-b873-0b22bd049dd8-kube-api-access-6stmq\") pod \"61c842b8-bf22-49a5-b873-0b22bd049dd8\" (UID: \"61c842b8-bf22-49a5-b873-0b22bd049dd8\") " Nov 03 01:03:13.076902 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:03:13.076858 4939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/61c842b8-bf22-49a5-b873-0b22bd049dd8-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/61c842b8-bf22-49a5-b873-0b22bd049dd8-maas-api-tls\") pod \"61c842b8-bf22-49a5-b873-0b22bd049dd8\" (UID: \"61c842b8-bf22-49a5-b873-0b22bd049dd8\") " Nov 03 01:03:13.077080 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:03:13.077059 4939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/61c842b8-bf22-49a5-b873-0b22bd049dd8-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/61c842b8-bf22-49a5-b873-0b22bd049dd8-openshift-service-ca\") pod \"61c842b8-bf22-49a5-b873-0b22bd049dd8\" (UID: \"61c842b8-bf22-49a5-b873-0b22bd049dd8\") " Nov 03 01:03:13.077823 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:03:13.077791 4939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/61c842b8-bf22-49a5-b873-0b22bd049dd8-openshift-service-ca" pod "61c842b8-bf22-49a5-b873-0b22bd049dd8" (UID: "61c842b8-bf22-49a5-b873-0b22bd049dd8"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 01:03:13.090868 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:03:13.090846 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:03:13.092172 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:03:13.092151 4939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/61c842b8-bf22-49a5-b873-0b22bd049dd8-metrics-certs" pod "61c842b8-bf22-49a5-b873-0b22bd049dd8" (UID: "61c842b8-bf22-49a5-b873-0b22bd049dd8"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 01:03:13.092264 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:03:13.092184 4939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/61c842b8-bf22-49a5-b873-0b22bd049dd8-maas-api-tls" pod "61c842b8-bf22-49a5-b873-0b22bd049dd8" (UID: "61c842b8-bf22-49a5-b873-0b22bd049dd8"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 01:03:13.093874 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:03:13.093849 4939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/61c842b8-bf22-49a5-b873-0b22bd049dd8-kube-api-access-6stmq" pod "61c842b8-bf22-49a5-b873-0b22bd049dd8" (UID: "61c842b8-bf22-49a5-b873-0b22bd049dd8"). InnerVolumeSpecName "kube-api-access-6stmq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 01:03:13.186952 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:03:13.185000 4939 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/61c842b8-bf22-49a5-b873-0b22bd049dd8-openshift-service-ca\") on node \"ip-10-0-140-107.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:03:13.186952 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:03:13.185042 4939 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/61c842b8-bf22-49a5-b873-0b22bd049dd8-metrics-certs\") on node \"ip-10-0-140-107.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:03:13.186952 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:03:13.185057 4939 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6stmq\" (UniqueName: \"kubernetes.io/projected/61c842b8-bf22-49a5-b873-0b22bd049dd8-kube-api-access-6stmq\") on node \"ip-10-0-140-107.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:03:13.186952 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:03:13.185088 4939 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/61c842b8-bf22-49a5-b873-0b22bd049dd8-maas-api-tls\") on node \"ip-10-0-140-107.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:03:13.416519 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:03:13.416475 4939 generic.go:356] "Generic (PLEG): container finished" podID="61c842b8-bf22-49a5-b873-0b22bd049dd8" containerID="d317352158b7a37af0263136970f841a17c8e59cbb98371fbe6cf526755a98d9" exitCode=0 Nov 03 01:03:13.416519 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:03:13.416517 4939 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-dc0f3f-656999868b-4bk2z" Nov 03 01:03:13.416861 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:03:13.416560 4939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-dc0f3f-656999868b-4bk2z" event={"ID":"61c842b8-bf22-49a5-b873-0b22bd049dd8","Type":"ContainerDied","Data":"d317352158b7a37af0263136970f841a17c8e59cbb98371fbe6cf526755a98d9"} Nov 03 01:03:13.416861 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:03:13.416621 4939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-dc0f3f-656999868b-4bk2z" event={"ID":"61c842b8-bf22-49a5-b873-0b22bd049dd8","Type":"ContainerDied","Data":"9447a1951212b276d05062a44245124a79947c7732483119a2a59e3a01b71aa3"} Nov 03 01:03:13.416861 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:03:13.416653 4939 scope.go:122] "RemoveContainer" containerID="d317352158b7a37af0263136970f841a17c8e59cbb98371fbe6cf526755a98d9" Nov 03 01:03:13.421130 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:03:13.420988 4939 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-praxis-a-b9dad7-openshift-default-994bfcbb9-bw5wc" Nov 03 01:03:13.421553 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:03:13.421519 4939 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-praxis-a-b9dad7-openshift-default-994bfcbb9-bw5wc" Nov 03 01:03:13.422430 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:03:13.422412 4939 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-praxis-a-b9dad7-openshift-default-994bfcbb9-bw5wc" Nov 03 01:03:13.428537 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:03:13.428520 4939 scope.go:122] "RemoveContainer" containerID="d317352158b7a37af0263136970f841a17c8e59cbb98371fbe6cf526755a98d9" Nov 03 01:03:13.428832 ip-10-0-140-107 kubenswrapper[4939]: E1103 01:03:13.428800 4939 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d317352158b7a37af0263136970f841a17c8e59cbb98371fbe6cf526755a98d9\": container with ID starting with d317352158b7a37af0263136970f841a17c8e59cbb98371fbe6cf526755a98d9 not found: ID does not exist" containerID="d317352158b7a37af0263136970f841a17c8e59cbb98371fbe6cf526755a98d9" Nov 03 01:03:13.428958 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:03:13.428837 4939 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d317352158b7a37af0263136970f841a17c8e59cbb98371fbe6cf526755a98d9"} err="failed to get container status \"d317352158b7a37af0263136970f841a17c8e59cbb98371fbe6cf526755a98d9\": rpc error: code = NotFound desc = could not find container \"d317352158b7a37af0263136970f841a17c8e59cbb98371fbe6cf526755a98d9\": container with ID starting with d317352158b7a37af0263136970f841a17c8e59cbb98371fbe6cf526755a98d9 not found: ID does not exist" Nov 03 01:03:13.458159 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:03:13.458137 4939 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-dc0f3f-656999868b-4bk2z"] Nov 03 01:03:13.461821 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:03:13.461801 4939 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-dc0f3f-656999868b-4bk2z"] Nov 03 01:03:13.797481 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:03:13.797399 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:03:13.883625 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:03:13.879215 4939 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="61c842b8-bf22-49a5-b873-0b22bd049dd8" path="/var/lib/kubelet/pods/61c842b8-bf22-49a5-b873-0b22bd049dd8/volumes" Nov 03 01:03:14.503464 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:03:14.503430 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:03:15.223622 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:03:15.223591 4939 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-f14bda-fb4769c77-kxzzg"] Nov 03 01:03:15.224332 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:03:15.224313 4939 cpu_manager.go:414] "containerMap: removing container" podUID="a2aede6b-fd70-4df3-bec1-a716350a85c8" containerName="istio-proxy" Nov 03 01:03:15.224332 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:03:15.224333 4939 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a2aede6b-fd70-4df3-bec1-a716350a85c8" containerName="istio-proxy" Nov 03 01:03:15.224451 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:03:15.224349 4939 cpu_manager.go:414] "containerMap: removing container" podUID="a2aede6b-fd70-4df3-bec1-a716350a85c8" containerName="istio-proxy" Nov 03 01:03:15.224451 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:03:15.224358 4939 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a2aede6b-fd70-4df3-bec1-a716350a85c8" containerName="istio-proxy" Nov 03 01:03:15.224451 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:03:15.224396 4939 cpu_manager.go:414] "containerMap: removing container" podUID="61c842b8-bf22-49a5-b873-0b22bd049dd8" containerName="maas-api" Nov 03 01:03:15.224451 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:03:15.224401 4939 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="61c842b8-bf22-49a5-b873-0b22bd049dd8" containerName="maas-api" Nov 03 01:03:15.224451 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:03:15.224419 4939 cpu_manager.go:414] "containerMap: removing container" podUID="ef4163bb-8632-4675-a678-11ec6d58e737" containerName="istio-proxy" Nov 03 01:03:15.224451 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:03:15.224424 4939 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ef4163bb-8632-4675-a678-11ec6d58e737" containerName="istio-proxy" Nov 03 01:03:15.224632 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:03:15.224532 4939 memory_manager.go:365] "RemoveStaleState removing state" podUID="61c842b8-bf22-49a5-b873-0b22bd049dd8" containerName="maas-api" Nov 03 01:03:15.224632 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:03:15.224546 4939 memory_manager.go:365] "RemoveStaleState removing state" podUID="a2aede6b-fd70-4df3-bec1-a716350a85c8" containerName="istio-proxy" Nov 03 01:03:15.224632 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:03:15.224554 4939 memory_manager.go:365] "RemoveStaleState removing state" podUID="a2aede6b-fd70-4df3-bec1-a716350a85c8" containerName="istio-proxy" Nov 03 01:03:15.224632 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:03:15.224561 4939 memory_manager.go:365] "RemoveStaleState removing state" podUID="ef4163bb-8632-4675-a678-11ec6d58e737" containerName="istio-proxy" Nov 03 01:03:15.229604 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:03:15.229583 4939 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-f14bda-fb4769c77-kxzzg" Nov 03 01:03:15.246133 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:03:15.246015 4939 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-praxis-b-f14bda-serving-cert\"" Nov 03 01:03:15.250129 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:03:15.250104 4939 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-f14bda-fb4769c77-kxzzg"] Nov 03 01:03:15.364143 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:03:15.364109 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:03:15.388428 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:03:15.388394 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c629818b-0176-4218-9e31-18bf89542550-openshift-service-ca\") pod \"maas-api-e2e-praxis-b-f14bda-fb4769c77-kxzzg\" (UID: \"c629818b-0176-4218-9e31-18bf89542550\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-f14bda-fb4769c77-kxzzg" Nov 03 01:03:15.388585 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:03:15.388554 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c629818b-0176-4218-9e31-18bf89542550-metrics-certs\") pod \"maas-api-e2e-praxis-b-f14bda-fb4769c77-kxzzg\" (UID: \"c629818b-0176-4218-9e31-18bf89542550\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-f14bda-fb4769c77-kxzzg" Nov 03 01:03:15.388759 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:03:15.388741 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c629818b-0176-4218-9e31-18bf89542550-maas-api-tls\") pod \"maas-api-e2e-praxis-b-f14bda-fb4769c77-kxzzg\" (UID: \"c629818b-0176-4218-9e31-18bf89542550\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-f14bda-fb4769c77-kxzzg" Nov 03 01:03:15.388840 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:03:15.388775 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qmnzc\" (UniqueName: \"kubernetes.io/projected/c629818b-0176-4218-9e31-18bf89542550-kube-api-access-qmnzc\") pod \"maas-api-e2e-praxis-b-f14bda-fb4769c77-kxzzg\" (UID: \"c629818b-0176-4218-9e31-18bf89542550\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-f14bda-fb4769c77-kxzzg" Nov 03 01:03:15.491001 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:03:15.490853 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c629818b-0176-4218-9e31-18bf89542550-maas-api-tls\") pod \"maas-api-e2e-praxis-b-f14bda-fb4769c77-kxzzg\" (UID: \"c629818b-0176-4218-9e31-18bf89542550\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-f14bda-fb4769c77-kxzzg" Nov 03 01:03:15.491001 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:03:15.490929 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qmnzc\" (UniqueName: \"kubernetes.io/projected/c629818b-0176-4218-9e31-18bf89542550-kube-api-access-qmnzc\") pod \"maas-api-e2e-praxis-b-f14bda-fb4769c77-kxzzg\" (UID: \"c629818b-0176-4218-9e31-18bf89542550\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-f14bda-fb4769c77-kxzzg" Nov 03 01:03:15.491240 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:03:15.491031 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c629818b-0176-4218-9e31-18bf89542550-openshift-service-ca\") pod \"maas-api-e2e-praxis-b-f14bda-fb4769c77-kxzzg\" (UID: \"c629818b-0176-4218-9e31-18bf89542550\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-f14bda-fb4769c77-kxzzg" Nov 03 01:03:15.491240 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:03:15.491184 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c629818b-0176-4218-9e31-18bf89542550-metrics-certs\") pod \"maas-api-e2e-praxis-b-f14bda-fb4769c77-kxzzg\" (UID: \"c629818b-0176-4218-9e31-18bf89542550\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-f14bda-fb4769c77-kxzzg" Nov 03 01:03:15.492107 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:03:15.492077 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c629818b-0176-4218-9e31-18bf89542550-openshift-service-ca\") pod \"maas-api-e2e-praxis-b-f14bda-fb4769c77-kxzzg\" (UID: \"c629818b-0176-4218-9e31-18bf89542550\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-f14bda-fb4769c77-kxzzg" Nov 03 01:03:15.503191 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:03:15.502989 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c629818b-0176-4218-9e31-18bf89542550-metrics-certs\") pod \"maas-api-e2e-praxis-b-f14bda-fb4769c77-kxzzg\" (UID: \"c629818b-0176-4218-9e31-18bf89542550\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-f14bda-fb4769c77-kxzzg" Nov 03 01:03:15.528647 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:03:15.528587 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c629818b-0176-4218-9e31-18bf89542550-maas-api-tls\") pod \"maas-api-e2e-praxis-b-f14bda-fb4769c77-kxzzg\" (UID: \"c629818b-0176-4218-9e31-18bf89542550\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-f14bda-fb4769c77-kxzzg" Nov 03 01:03:15.530514 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:03:15.530475 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:03:15.533673 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:03:15.533620 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qmnzc\" (UniqueName: \"kubernetes.io/projected/c629818b-0176-4218-9e31-18bf89542550-kube-api-access-qmnzc\") pod \"maas-api-e2e-praxis-b-f14bda-fb4769c77-kxzzg\" (UID: \"c629818b-0176-4218-9e31-18bf89542550\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-f14bda-fb4769c77-kxzzg" Nov 03 01:03:15.557934 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:03:15.547277 4939 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-f14bda-fb4769c77-kxzzg" Nov 03 01:03:15.876713 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:03:15.876682 4939 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-f14bda-fb4769c77-kxzzg"] Nov 03 01:03:15.877989 ip-10-0-140-107 kubenswrapper[4939]: W1103 01:03:15.877958 4939 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc629818b_0176_4218_9e31_18bf89542550.slice/crio-6e5b4f2810046285a53306e8824331294b984847d9b6d6ab7a3cfa0d2e7d1814 WatchSource:0}: Error finding container 6e5b4f2810046285a53306e8824331294b984847d9b6d6ab7a3cfa0d2e7d1814: Status 404 returned error can't find the container with id 6e5b4f2810046285a53306e8824331294b984847d9b6d6ab7a3cfa0d2e7d1814 Nov 03 01:03:16.348931 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:03:16.348891 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:03:16.450546 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:03:16.450514 4939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-f14bda-fb4769c77-kxzzg" event={"ID":"c629818b-0176-4218-9e31-18bf89542550","Type":"ContainerStarted","Data":"6e5b4f2810046285a53306e8824331294b984847d9b6d6ab7a3cfa0d2e7d1814"} Nov 03 01:03:16.551997 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:03:16.551966 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:03:16.660892 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:03:16.660856 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:03:17.007351 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:03:17.007264 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:03:17.459403 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:03:17.459360 4939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-f14bda-fb4769c77-kxzzg" event={"ID":"c629818b-0176-4218-9e31-18bf89542550","Type":"ContainerStarted","Data":"c95ced104d756526a1ff8cbe9b92a27e683eafc9f5f9fd2b49a7f5a99e005ecb"} Nov 03 01:03:17.459615 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:03:17.459432 4939 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-f14bda-fb4769c77-kxzzg" Nov 03 01:03:17.484385 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:03:17.484334 4939 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-f14bda-fb4769c77-kxzzg" podStartSLOduration=2.04374194 podStartE2EDuration="2.484321065s" totalImagesPullingTime="440.579125ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 01:03:15 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 01:03:17.481434518 +0000 UTC m=+1345.745769463" watchObservedRunningTime="2026-11-03 01:03:17.484321065 +0000 UTC m=+1345.748656009" Nov 03 01:03:17.618050 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:03:17.618015 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:03:18.033013 ip-10-0-140-107 kubenswrapper[4939]: E1103 01:03:18.032977 4939 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-sthjh" containerName="extractor" Nov 03 01:03:18.083967 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:03:18.083932 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:03:18.195670 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:03:18.195634 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:03:18.321706 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:03:18.321630 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:03:18.590723 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:03:18.590686 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:03:18.925413 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:03:18.925315 4939 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-dc0f3f-openshift-default-d9f67c9bc-ks8t6"] Nov 03 01:03:18.925885 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:03:18.925573 4939 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-a-dc0f3f-openshift-default-d9f67c9bc-ks8t6" podUID="6c13fe13-2eda-4d7c-bf31-c522d7f3b9c3" containerName="istio-proxy" containerID="cri-o://e57037fbf8ed3717bc1c08ade218920cf420ae675039878ca1e37e672842c7b2" gracePeriod=30 Nov 03 01:03:19.000333 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:03:19.000300 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:03:19.466613 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:03:19.466581 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:03:19.749962 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:03:19.749876 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:03:19.931555 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:03:19.931519 4939 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-4pw7v"] Nov 03 01:03:20.198477 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:03:20.198447 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:03:21.322639 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:03:21.322603 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:03:22.122861 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:03:22.122830 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:03:22.670359 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:03:22.670317 4939 patch_prober.go:29] interesting pod/e2e-ipp-a-dc0f3f-openshift-default-d9f67c9bc-ks8t6 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 03 01:03:22.670767 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:03:22.670388 4939 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ipp-a-dc0f3f-openshift-default-d9f67c9bc-ks8t6" podUID="6c13fe13-2eda-4d7c-bf31-c522d7f3b9c3" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 03 01:03:22.890007 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:03:22.889959 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:03:23.473082 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:03:23.473025 4939 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-f14bda-fb4769c77-kxzzg" Nov 03 01:03:23.687100 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:03:23.687061 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:03:24.127121 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:03:24.127089 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:03:24.472399 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:03:24.472364 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:03:24.589387 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:03:24.589357 4939 generic.go:356] "Generic (PLEG): container finished" podID="6c13fe13-2eda-4d7c-bf31-c522d7f3b9c3" containerID="e57037fbf8ed3717bc1c08ade218920cf420ae675039878ca1e37e672842c7b2" exitCode=0 Nov 03 01:03:24.589602 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:03:24.589436 4939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-dc0f3f-openshift-default-d9f67c9bc-ks8t6" event={"ID":"6c13fe13-2eda-4d7c-bf31-c522d7f3b9c3","Type":"ContainerDied","Data":"e57037fbf8ed3717bc1c08ade218920cf420ae675039878ca1e37e672842c7b2"} Nov 03 01:03:24.589697 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:03:24.589651 4939 scope.go:122] "RemoveContainer" containerID="24198b03b942e91379154ffbed0ce369eb1b7bea6f0a27a5caadcb65fe09cefd" Nov 03 01:03:24.694817 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:03:24.694754 4939 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-dc0f3f-openshift-default-d9f67c9bc-ks8t6" Nov 03 01:03:24.768072 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:03:24.768035 4939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6c13fe13-2eda-4d7c-bf31-c522d7f3b9c3-kube-api-access-6wxjj\" (UniqueName: \"kubernetes.io/projected/6c13fe13-2eda-4d7c-bf31-c522d7f3b9c3-kube-api-access-6wxjj\") pod \"6c13fe13-2eda-4d7c-bf31-c522d7f3b9c3\" (UID: \"6c13fe13-2eda-4d7c-bf31-c522d7f3b9c3\") " Nov 03 01:03:24.768214 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:03:24.768170 4939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6c13fe13-2eda-4d7c-bf31-c522d7f3b9c3-istio-token\" (UniqueName: \"kubernetes.io/projected/6c13fe13-2eda-4d7c-bf31-c522d7f3b9c3-istio-token\") pod \"6c13fe13-2eda-4d7c-bf31-c522d7f3b9c3\" (UID: \"6c13fe13-2eda-4d7c-bf31-c522d7f3b9c3\") " Nov 03 01:03:24.768286 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:03:24.768231 4939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6c13fe13-2eda-4d7c-bf31-c522d7f3b9c3-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6c13fe13-2eda-4d7c-bf31-c522d7f3b9c3-workload-socket\") pod \"6c13fe13-2eda-4d7c-bf31-c522d7f3b9c3\" (UID: \"6c13fe13-2eda-4d7c-bf31-c522d7f3b9c3\") " Nov 03 01:03:24.768457 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:03:24.768428 4939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6c13fe13-2eda-4d7c-bf31-c522d7f3b9c3-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6c13fe13-2eda-4d7c-bf31-c522d7f3b9c3-istiod-ca-cert\") pod \"6c13fe13-2eda-4d7c-bf31-c522d7f3b9c3\" (UID: \"6c13fe13-2eda-4d7c-bf31-c522d7f3b9c3\") " Nov 03 01:03:24.768561 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:03:24.768492 4939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6c13fe13-2eda-4d7c-bf31-c522d7f3b9c3-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6c13fe13-2eda-4d7c-bf31-c522d7f3b9c3-credential-socket\") pod \"6c13fe13-2eda-4d7c-bf31-c522d7f3b9c3\" (UID: \"6c13fe13-2eda-4d7c-bf31-c522d7f3b9c3\") " Nov 03 01:03:24.768617 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:03:24.768581 4939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6c13fe13-2eda-4d7c-bf31-c522d7f3b9c3-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6c13fe13-2eda-4d7c-bf31-c522d7f3b9c3-istio-envoy\") pod \"6c13fe13-2eda-4d7c-bf31-c522d7f3b9c3\" (UID: \"6c13fe13-2eda-4d7c-bf31-c522d7f3b9c3\") " Nov 03 01:03:24.768666 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:03:24.768628 4939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/6c13fe13-2eda-4d7c-bf31-c522d7f3b9c3-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6c13fe13-2eda-4d7c-bf31-c522d7f3b9c3-istio-podinfo\") pod \"6c13fe13-2eda-4d7c-bf31-c522d7f3b9c3\" (UID: \"6c13fe13-2eda-4d7c-bf31-c522d7f3b9c3\") " Nov 03 01:03:24.768666 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:03:24.768630 4939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6c13fe13-2eda-4d7c-bf31-c522d7f3b9c3-workload-socket" pod "6c13fe13-2eda-4d7c-bf31-c522d7f3b9c3" (UID: "6c13fe13-2eda-4d7c-bf31-c522d7f3b9c3"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 01:03:24.768785 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:03:24.768687 4939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6c13fe13-2eda-4d7c-bf31-c522d7f3b9c3-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6c13fe13-2eda-4d7c-bf31-c522d7f3b9c3-workload-certs\") pod \"6c13fe13-2eda-4d7c-bf31-c522d7f3b9c3\" (UID: \"6c13fe13-2eda-4d7c-bf31-c522d7f3b9c3\") " Nov 03 01:03:24.768785 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:03:24.768715 4939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6c13fe13-2eda-4d7c-bf31-c522d7f3b9c3-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6c13fe13-2eda-4d7c-bf31-c522d7f3b9c3-istio-data\") pod \"6c13fe13-2eda-4d7c-bf31-c522d7f3b9c3\" (UID: \"6c13fe13-2eda-4d7c-bf31-c522d7f3b9c3\") " Nov 03 01:03:24.768891 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:03:24.768839 4939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6c13fe13-2eda-4d7c-bf31-c522d7f3b9c3-credential-socket" pod "6c13fe13-2eda-4d7c-bf31-c522d7f3b9c3" (UID: "6c13fe13-2eda-4d7c-bf31-c522d7f3b9c3"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 01:03:24.769542 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:03:24.769175 4939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6c13fe13-2eda-4d7c-bf31-c522d7f3b9c3-istiod-ca-cert" pod "6c13fe13-2eda-4d7c-bf31-c522d7f3b9c3" (UID: "6c13fe13-2eda-4d7c-bf31-c522d7f3b9c3"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 01:03:24.769542 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:03:24.769416 4939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6c13fe13-2eda-4d7c-bf31-c522d7f3b9c3-istio-data" pod "6c13fe13-2eda-4d7c-bf31-c522d7f3b9c3" (UID: "6c13fe13-2eda-4d7c-bf31-c522d7f3b9c3"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 01:03:24.769542 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:03:24.769511 4939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6c13fe13-2eda-4d7c-bf31-c522d7f3b9c3-workload-certs" pod "6c13fe13-2eda-4d7c-bf31-c522d7f3b9c3" (UID: "6c13fe13-2eda-4d7c-bf31-c522d7f3b9c3"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 01:03:24.770286 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:03:24.769601 4939 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6c13fe13-2eda-4d7c-bf31-c522d7f3b9c3-istiod-ca-cert\") on node \"ip-10-0-140-107.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:03:24.770286 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:03:24.769639 4939 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6c13fe13-2eda-4d7c-bf31-c522d7f3b9c3-credential-socket\") on node \"ip-10-0-140-107.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:03:24.770286 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:03:24.769655 4939 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6c13fe13-2eda-4d7c-bf31-c522d7f3b9c3-istio-data\") on node \"ip-10-0-140-107.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:03:24.770286 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:03:24.769669 4939 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6c13fe13-2eda-4d7c-bf31-c522d7f3b9c3-workload-socket\") on node \"ip-10-0-140-107.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:03:24.771189 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:03:24.771161 4939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6c13fe13-2eda-4d7c-bf31-c522d7f3b9c3-kube-api-access-6wxjj" pod "6c13fe13-2eda-4d7c-bf31-c522d7f3b9c3" (UID: "6c13fe13-2eda-4d7c-bf31-c522d7f3b9c3"). InnerVolumeSpecName "kube-api-access-6wxjj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 01:03:24.771336 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:03:24.771275 4939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6c13fe13-2eda-4d7c-bf31-c522d7f3b9c3-istio-envoy" pod "6c13fe13-2eda-4d7c-bf31-c522d7f3b9c3" (UID: "6c13fe13-2eda-4d7c-bf31-c522d7f3b9c3"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 01:03:24.771618 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:03:24.771591 4939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/6c13fe13-2eda-4d7c-bf31-c522d7f3b9c3-istio-podinfo" pod "6c13fe13-2eda-4d7c-bf31-c522d7f3b9c3" (UID: "6c13fe13-2eda-4d7c-bf31-c522d7f3b9c3"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 01:03:24.772163 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:03:24.772137 4939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6c13fe13-2eda-4d7c-bf31-c522d7f3b9c3-istio-token" pod "6c13fe13-2eda-4d7c-bf31-c522d7f3b9c3" (UID: "6c13fe13-2eda-4d7c-bf31-c522d7f3b9c3"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 01:03:24.871586 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:03:24.871549 4939 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6c13fe13-2eda-4d7c-bf31-c522d7f3b9c3-istio-podinfo\") on node \"ip-10-0-140-107.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:03:24.871751 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:03:24.871590 4939 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6c13fe13-2eda-4d7c-bf31-c522d7f3b9c3-workload-certs\") on node \"ip-10-0-140-107.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:03:24.871751 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:03:24.871609 4939 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6wxjj\" (UniqueName: \"kubernetes.io/projected/6c13fe13-2eda-4d7c-bf31-c522d7f3b9c3-kube-api-access-6wxjj\") on node \"ip-10-0-140-107.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:03:24.871751 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:03:24.871623 4939 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6c13fe13-2eda-4d7c-bf31-c522d7f3b9c3-istio-token\") on node \"ip-10-0-140-107.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:03:24.871751 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:03:24.871638 4939 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6c13fe13-2eda-4d7c-bf31-c522d7f3b9c3-istio-envoy\") on node \"ip-10-0-140-107.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:03:25.425474 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:03:25.412254 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:03:25.569712 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:03:25.569675 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:03:25.661929 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:03:25.661492 4939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-dc0f3f-openshift-default-d9f67c9bc-ks8t6" event={"ID":"6c13fe13-2eda-4d7c-bf31-c522d7f3b9c3","Type":"ContainerDied","Data":"c2a67945e11fae41f902ac8e853f430a56915418619de74ba55cd0010a83475a"} Nov 03 01:03:25.661929 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:03:25.661543 4939 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-dc0f3f-openshift-default-d9f67c9bc-ks8t6" Nov 03 01:03:25.661929 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:03:25.661625 4939 scope.go:122] "RemoveContainer" containerID="e57037fbf8ed3717bc1c08ade218920cf420ae675039878ca1e37e672842c7b2" Nov 03 01:03:25.687678 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:03:25.687649 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:03:25.696102 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:03:25.696073 4939 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-dc0f3f-openshift-default-d9f67c9bc-ks8t6"] Nov 03 01:03:25.700805 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:03:25.700784 4939 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-a-dc0f3f-openshift-default-d9f67c9bc-ks8t6"] Nov 03 01:03:25.871174 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:03:25.871133 4939 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6c13fe13-2eda-4d7c-bf31-c522d7f3b9c3" path="/var/lib/kubelet/pods/6c13fe13-2eda-4d7c-bf31-c522d7f3b9c3/volumes" Nov 03 01:03:25.958151 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:03:25.958069 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:03:26.063623 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:03:26.063586 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:03:26.215822 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:03:26.215720 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:03:26.770166 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:03:26.770133 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:03:27.056860 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:03:27.056758 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:03:27.186239 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:03:27.186204 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:03:27.771261 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:03:27.771223 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:03:28.297359 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:03:28.297326 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:03:28.987126 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:03:28.987095 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:03:29.588005 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:03:29.587971 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:03:30.296569 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:03:30.296534 4939 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-praxis-a-1e352e/praxis-route-1e352e-kserve-67cb6ccfdc-k25pd"] Nov 03 01:03:30.390289 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:03:30.390253 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:03:30.510608 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:03:30.510563 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:03:31.204773 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:03:31.204738 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:03:31.657533 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:03:31.657497 4939 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-ff4113-5487d8fb68-mf2c7"] Nov 03 01:03:31.658291 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:03:31.658257 4939 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-ff4113-5487d8fb68-mf2c7" podUID="cf320aba-5bf7-42fe-a4a9-603130676712" containerName="maas-api" containerID="cri-o://dd4fd811c9b0d73743b448b01c8dcc5095c37c8e5c64c8278451e5cfe5e8ccc0" gracePeriod=30 Nov 03 01:03:31.729218 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:03:31.729183 4939 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-ff4113-6fd46cb5cf-g5q8p"] Nov 03 01:03:31.729477 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:03:31.729447 4939 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-ff4113-6fd46cb5cf-g5q8p" podUID="fb530145-405c-43ea-946c-e7f2c85cec68" containerName="payload-pre-processing" containerID="cri-o://c8d70cf8657137f023db39f5576efa63158d88286208f0bf4cb3bdd4447d78b4" gracePeriod=30 Nov 03 01:03:31.768528 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:03:31.766454 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:03:31.882655 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:03:31.882617 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:03:32.187962 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:03:32.182520 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:03:32.383316 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:03:32.383292 4939 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-ff4113-5487d8fb68-mf2c7" Nov 03 01:03:32.404178 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:03:32.403542 4939 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-ff4113-6fd46cb5cf-g5q8p" Nov 03 01:03:32.499505 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:03:32.499471 4939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cf320aba-5bf7-42fe-a4a9-603130676712-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/cf320aba-5bf7-42fe-a4a9-603130676712-openshift-service-ca\") pod \"cf320aba-5bf7-42fe-a4a9-603130676712\" (UID: \"cf320aba-5bf7-42fe-a4a9-603130676712\") " Nov 03 01:03:32.499683 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:03:32.499518 4939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fb530145-405c-43ea-946c-e7f2c85cec68-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fb530145-405c-43ea-946c-e7f2c85cec68-plugins-config-volume\") pod \"fb530145-405c-43ea-946c-e7f2c85cec68\" (UID: \"fb530145-405c-43ea-946c-e7f2c85cec68\") " Nov 03 01:03:32.499683 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:03:32.499582 4939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fb530145-405c-43ea-946c-e7f2c85cec68-kube-api-access-x5x9z\" (UniqueName: \"kubernetes.io/projected/fb530145-405c-43ea-946c-e7f2c85cec68-kube-api-access-x5x9z\") pod \"fb530145-405c-43ea-946c-e7f2c85cec68\" (UID: \"fb530145-405c-43ea-946c-e7f2c85cec68\") " Nov 03 01:03:32.499683 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:03:32.499614 4939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/cf320aba-5bf7-42fe-a4a9-603130676712-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/cf320aba-5bf7-42fe-a4a9-603130676712-maas-api-tls\") pod \"cf320aba-5bf7-42fe-a4a9-603130676712\" (UID: \"cf320aba-5bf7-42fe-a4a9-603130676712\") " Nov 03 01:03:32.499862 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:03:32.499695 4939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cf320aba-5bf7-42fe-a4a9-603130676712-kube-api-access-hlr9z\" (UniqueName: \"kubernetes.io/projected/cf320aba-5bf7-42fe-a4a9-603130676712-kube-api-access-hlr9z\") pod \"cf320aba-5bf7-42fe-a4a9-603130676712\" (UID: \"cf320aba-5bf7-42fe-a4a9-603130676712\") " Nov 03 01:03:32.499862 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:03:32.499767 4939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/cf320aba-5bf7-42fe-a4a9-603130676712-metrics-certs\" (UniqueName: \"kubernetes.io/secret/cf320aba-5bf7-42fe-a4a9-603130676712-metrics-certs\") pod \"cf320aba-5bf7-42fe-a4a9-603130676712\" (UID: \"cf320aba-5bf7-42fe-a4a9-603130676712\") " Nov 03 01:03:32.499994 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:03:32.499897 4939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cf320aba-5bf7-42fe-a4a9-603130676712-openshift-service-ca" pod "cf320aba-5bf7-42fe-a4a9-603130676712" (UID: "cf320aba-5bf7-42fe-a4a9-603130676712"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 01:03:32.500065 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:03:32.500028 4939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fb530145-405c-43ea-946c-e7f2c85cec68-plugins-config-volume" pod "fb530145-405c-43ea-946c-e7f2c85cec68" (UID: "fb530145-405c-43ea-946c-e7f2c85cec68"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 01:03:32.500705 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:03:32.500670 4939 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/cf320aba-5bf7-42fe-a4a9-603130676712-openshift-service-ca\") on node \"ip-10-0-140-107.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:03:32.500705 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:03:32.500699 4939 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fb530145-405c-43ea-946c-e7f2c85cec68-plugins-config-volume\") on node \"ip-10-0-140-107.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:03:32.502147 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:03:32.502121 4939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cf320aba-5bf7-42fe-a4a9-603130676712-maas-api-tls" pod "cf320aba-5bf7-42fe-a4a9-603130676712" (UID: "cf320aba-5bf7-42fe-a4a9-603130676712"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 01:03:32.502356 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:03:32.502333 4939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cf320aba-5bf7-42fe-a4a9-603130676712-kube-api-access-hlr9z" pod "cf320aba-5bf7-42fe-a4a9-603130676712" (UID: "cf320aba-5bf7-42fe-a4a9-603130676712"). InnerVolumeSpecName "kube-api-access-hlr9z". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 01:03:32.502477 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:03:32.502460 4939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cf320aba-5bf7-42fe-a4a9-603130676712-metrics-certs" pod "cf320aba-5bf7-42fe-a4a9-603130676712" (UID: "cf320aba-5bf7-42fe-a4a9-603130676712"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 01:03:32.502522 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:03:32.502465 4939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fb530145-405c-43ea-946c-e7f2c85cec68-kube-api-access-x5x9z" pod "fb530145-405c-43ea-946c-e7f2c85cec68" (UID: "fb530145-405c-43ea-946c-e7f2c85cec68"). InnerVolumeSpecName "kube-api-access-x5x9z". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 01:03:32.601837 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:03:32.601803 4939 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-x5x9z\" (UniqueName: \"kubernetes.io/projected/fb530145-405c-43ea-946c-e7f2c85cec68-kube-api-access-x5x9z\") on node \"ip-10-0-140-107.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:03:32.601837 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:03:32.601835 4939 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/cf320aba-5bf7-42fe-a4a9-603130676712-maas-api-tls\") on node \"ip-10-0-140-107.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:03:32.601837 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:03:32.601845 4939 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hlr9z\" (UniqueName: \"kubernetes.io/projected/cf320aba-5bf7-42fe-a4a9-603130676712-kube-api-access-hlr9z\") on node \"ip-10-0-140-107.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:03:32.602113 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:03:32.601855 4939 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cf320aba-5bf7-42fe-a4a9-603130676712-metrics-certs\") on node \"ip-10-0-140-107.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:03:32.760881 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:03:32.760842 4939 generic.go:356] "Generic (PLEG): container finished" podID="fb530145-405c-43ea-946c-e7f2c85cec68" containerID="c8d70cf8657137f023db39f5576efa63158d88286208f0bf4cb3bdd4447d78b4" exitCode=0 Nov 03 01:03:32.761313 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:03:32.760901 4939 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-ff4113-6fd46cb5cf-g5q8p" Nov 03 01:03:32.761313 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:03:32.760952 4939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-ff4113-6fd46cb5cf-g5q8p" event={"ID":"fb530145-405c-43ea-946c-e7f2c85cec68","Type":"ContainerDied","Data":"c8d70cf8657137f023db39f5576efa63158d88286208f0bf4cb3bdd4447d78b4"} Nov 03 01:03:32.761313 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:03:32.761010 4939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-ff4113-6fd46cb5cf-g5q8p" event={"ID":"fb530145-405c-43ea-946c-e7f2c85cec68","Type":"ContainerDied","Data":"f4eb305b2d2eb8fc44fa506fd58cba2ec4b0d08e3c8c52ab4e8db934c8532798"} Nov 03 01:03:32.761313 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:03:32.761037 4939 scope.go:122] "RemoveContainer" containerID="c8d70cf8657137f023db39f5576efa63158d88286208f0bf4cb3bdd4447d78b4" Nov 03 01:03:32.762773 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:03:32.762749 4939 generic.go:356] "Generic (PLEG): container finished" podID="cf320aba-5bf7-42fe-a4a9-603130676712" containerID="dd4fd811c9b0d73743b448b01c8dcc5095c37c8e5c64c8278451e5cfe5e8ccc0" exitCode=0 Nov 03 01:03:32.762880 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:03:32.762800 4939 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-ff4113-5487d8fb68-mf2c7" Nov 03 01:03:32.762880 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:03:32.762839 4939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-ff4113-5487d8fb68-mf2c7" event={"ID":"cf320aba-5bf7-42fe-a4a9-603130676712","Type":"ContainerDied","Data":"dd4fd811c9b0d73743b448b01c8dcc5095c37c8e5c64c8278451e5cfe5e8ccc0"} Nov 03 01:03:32.763067 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:03:32.762939 4939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-ff4113-5487d8fb68-mf2c7" event={"ID":"cf320aba-5bf7-42fe-a4a9-603130676712","Type":"ContainerDied","Data":"7606705ca630c6fe6761aea69d9a2c3a88ba0a143018e8206d92b1dee1acd39a"} Nov 03 01:03:32.774119 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:03:32.774103 4939 scope.go:122] "RemoveContainer" containerID="c8d70cf8657137f023db39f5576efa63158d88286208f0bf4cb3bdd4447d78b4" Nov 03 01:03:32.774425 ip-10-0-140-107 kubenswrapper[4939]: E1103 01:03:32.774403 4939 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c8d70cf8657137f023db39f5576efa63158d88286208f0bf4cb3bdd4447d78b4\": container with ID starting with c8d70cf8657137f023db39f5576efa63158d88286208f0bf4cb3bdd4447d78b4 not found: ID does not exist" containerID="c8d70cf8657137f023db39f5576efa63158d88286208f0bf4cb3bdd4447d78b4" Nov 03 01:03:32.774511 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:03:32.774437 4939 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c8d70cf8657137f023db39f5576efa63158d88286208f0bf4cb3bdd4447d78b4"} err="failed to get container status \"c8d70cf8657137f023db39f5576efa63158d88286208f0bf4cb3bdd4447d78b4\": rpc error: code = NotFound desc = could not find container \"c8d70cf8657137f023db39f5576efa63158d88286208f0bf4cb3bdd4447d78b4\": container with ID starting with c8d70cf8657137f023db39f5576efa63158d88286208f0bf4cb3bdd4447d78b4 not found: ID does not exist" Nov 03 01:03:32.774511 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:03:32.774459 4939 scope.go:122] "RemoveContainer" containerID="dd4fd811c9b0d73743b448b01c8dcc5095c37c8e5c64c8278451e5cfe5e8ccc0" Nov 03 01:03:32.784332 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:03:32.784317 4939 scope.go:122] "RemoveContainer" containerID="dd4fd811c9b0d73743b448b01c8dcc5095c37c8e5c64c8278451e5cfe5e8ccc0" Nov 03 01:03:32.784604 ip-10-0-140-107 kubenswrapper[4939]: E1103 01:03:32.784579 4939 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"dd4fd811c9b0d73743b448b01c8dcc5095c37c8e5c64c8278451e5cfe5e8ccc0\": container with ID starting with dd4fd811c9b0d73743b448b01c8dcc5095c37c8e5c64c8278451e5cfe5e8ccc0 not found: ID does not exist" containerID="dd4fd811c9b0d73743b448b01c8dcc5095c37c8e5c64c8278451e5cfe5e8ccc0" Nov 03 01:03:32.784686 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:03:32.784613 4939 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"dd4fd811c9b0d73743b448b01c8dcc5095c37c8e5c64c8278451e5cfe5e8ccc0"} err="failed to get container status \"dd4fd811c9b0d73743b448b01c8dcc5095c37c8e5c64c8278451e5cfe5e8ccc0\": rpc error: code = NotFound desc = could not find container \"dd4fd811c9b0d73743b448b01c8dcc5095c37c8e5c64c8278451e5cfe5e8ccc0\": container with ID starting with dd4fd811c9b0d73743b448b01c8dcc5095c37c8e5c64c8278451e5cfe5e8ccc0 not found: ID does not exist" Nov 03 01:03:32.788204 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:03:32.788182 4939 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-ff4113-6fd46cb5cf-g5q8p"] Nov 03 01:03:32.791990 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:03:32.791966 4939 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-ff4113-6fd46cb5cf-g5q8p"] Nov 03 01:03:32.804622 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:03:32.804600 4939 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-ff4113-5487d8fb68-mf2c7"] Nov 03 01:03:32.808033 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:03:32.808014 4939 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-ff4113-5487d8fb68-mf2c7"] Nov 03 01:03:32.918359 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:03:32.918320 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:03:33.055740 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:03:33.055650 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:03:33.189802 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:03:33.189767 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:03:33.372944 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:03:33.372889 4939 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-b9dad7-8569798f65-m9hcd"] Nov 03 01:03:33.399547 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:03:33.392806 4939 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-b9dad7-7458764995-5pdtn"] Nov 03 01:03:33.476018 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:03:33.472223 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:03:33.590408 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:03:33.590352 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:03:33.699111 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:03:33.699013 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:03:33.915600 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:03:33.912298 4939 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cf320aba-5bf7-42fe-a4a9-603130676712" path="/var/lib/kubelet/pods/cf320aba-5bf7-42fe-a4a9-603130676712/volumes" Nov 03 01:03:33.939928 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:03:33.933939 4939 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fb530145-405c-43ea-946c-e7f2c85cec68" path="/var/lib/kubelet/pods/fb530145-405c-43ea-946c-e7f2c85cec68/volumes" Nov 03 01:03:34.060983 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:03:34.059398 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:03:34.207939 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:03:34.200129 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:03:35.168957 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:03:35.168922 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:03:35.803531 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:03:35.803497 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:03:36.613174 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:03:36.610476 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:03:37.200329 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:03:37.200296 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:03:37.884858 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:03:37.884808 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:03:38.373405 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:03:38.373366 4939 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-praxis-a-b9dad7-8569798f65-m9hcd" podUID="2b03c0be-253a-41f3-907b-ec20ddc72a52" containerName="payload-processing" containerID="cri-o://73d99d7b4c933b05257f5b032f0a345d12a06df1c46c832538b61cd6029cf4bc" gracePeriod=25 Nov 03 01:03:38.393610 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:03:38.393572 4939 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-b9dad7-7458764995-5pdtn" podUID="e6830dd6-9ccc-47ca-b7a3-3cbb6c9bc8d4" containerName="payload-pre-processing" containerID="cri-o://493d816a46f7a89b30934d47cc9ce25b8c44fd000b264b8cf8f8b553fbeec657" gracePeriod=25 Nov 03 01:03:38.672746 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:03:38.672715 4939 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-a-b9dad7-8569798f65-m9hcd" Nov 03 01:03:38.684637 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:03:38.684610 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:03:38.696107 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:03:38.696077 4939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2b03c0be-253a-41f3-907b-ec20ddc72a52-kube-api-access-nrfm4\" (UniqueName: \"kubernetes.io/projected/2b03c0be-253a-41f3-907b-ec20ddc72a52-kube-api-access-nrfm4\") pod \"2b03c0be-253a-41f3-907b-ec20ddc72a52\" (UID: \"2b03c0be-253a-41f3-907b-ec20ddc72a52\") " Nov 03 01:03:38.696264 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:03:38.696124 4939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2b03c0be-253a-41f3-907b-ec20ddc72a52-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2b03c0be-253a-41f3-907b-ec20ddc72a52-plugins-config-volume\") pod \"2b03c0be-253a-41f3-907b-ec20ddc72a52\" (UID: \"2b03c0be-253a-41f3-907b-ec20ddc72a52\") " Nov 03 01:03:38.696784 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:03:38.696759 4939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2b03c0be-253a-41f3-907b-ec20ddc72a52-plugins-config-volume" pod "2b03c0be-253a-41f3-907b-ec20ddc72a52" (UID: "2b03c0be-253a-41f3-907b-ec20ddc72a52"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 01:03:38.699098 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:03:38.699068 4939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2b03c0be-253a-41f3-907b-ec20ddc72a52-kube-api-access-nrfm4" pod "2b03c0be-253a-41f3-907b-ec20ddc72a52" (UID: "2b03c0be-253a-41f3-907b-ec20ddc72a52"). InnerVolumeSpecName "kube-api-access-nrfm4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 01:03:38.769311 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:03:38.768225 4939 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-b9dad7-7458764995-5pdtn" Nov 03 01:03:38.806877 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:03:38.801532 4939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e6830dd6-9ccc-47ca-b7a3-3cbb6c9bc8d4-kube-api-access-mq8k5\" (UniqueName: \"kubernetes.io/projected/e6830dd6-9ccc-47ca-b7a3-3cbb6c9bc8d4-kube-api-access-mq8k5\") pod \"e6830dd6-9ccc-47ca-b7a3-3cbb6c9bc8d4\" (UID: \"e6830dd6-9ccc-47ca-b7a3-3cbb6c9bc8d4\") " Nov 03 01:03:38.806877 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:03:38.801632 4939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e6830dd6-9ccc-47ca-b7a3-3cbb6c9bc8d4-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e6830dd6-9ccc-47ca-b7a3-3cbb6c9bc8d4-plugins-config-volume\") pod \"e6830dd6-9ccc-47ca-b7a3-3cbb6c9bc8d4\" (UID: \"e6830dd6-9ccc-47ca-b7a3-3cbb6c9bc8d4\") " Nov 03 01:03:38.806877 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:03:38.802406 4939 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nrfm4\" (UniqueName: \"kubernetes.io/projected/2b03c0be-253a-41f3-907b-ec20ddc72a52-kube-api-access-nrfm4\") on node \"ip-10-0-140-107.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:03:38.806877 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:03:38.802433 4939 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2b03c0be-253a-41f3-907b-ec20ddc72a52-plugins-config-volume\") on node \"ip-10-0-140-107.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:03:38.806877 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:03:38.802834 4939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e6830dd6-9ccc-47ca-b7a3-3cbb6c9bc8d4-plugins-config-volume" pod "e6830dd6-9ccc-47ca-b7a3-3cbb6c9bc8d4" (UID: "e6830dd6-9ccc-47ca-b7a3-3cbb6c9bc8d4"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 01:03:38.819088 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:03:38.819051 4939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e6830dd6-9ccc-47ca-b7a3-3cbb6c9bc8d4-kube-api-access-mq8k5" pod "e6830dd6-9ccc-47ca-b7a3-3cbb6c9bc8d4" (UID: "e6830dd6-9ccc-47ca-b7a3-3cbb6c9bc8d4"). InnerVolumeSpecName "kube-api-access-mq8k5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 01:03:38.873236 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:03:38.873205 4939 generic.go:356] "Generic (PLEG): container finished" podID="2b03c0be-253a-41f3-907b-ec20ddc72a52" containerID="73d99d7b4c933b05257f5b032f0a345d12a06df1c46c832538b61cd6029cf4bc" exitCode=0 Nov 03 01:03:38.873408 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:03:38.873254 4939 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-a-b9dad7-8569798f65-m9hcd" Nov 03 01:03:38.873408 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:03:38.873293 4939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-a-b9dad7-8569798f65-m9hcd" event={"ID":"2b03c0be-253a-41f3-907b-ec20ddc72a52","Type":"ContainerDied","Data":"73d99d7b4c933b05257f5b032f0a345d12a06df1c46c832538b61cd6029cf4bc"} Nov 03 01:03:38.873408 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:03:38.873342 4939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-a-b9dad7-8569798f65-m9hcd" event={"ID":"2b03c0be-253a-41f3-907b-ec20ddc72a52","Type":"ContainerDied","Data":"4696c7be42ec5317b3160afbe059e0bb030b95ef4c8bd9cada09cdc6194b70f3"} Nov 03 01:03:38.873408 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:03:38.873365 4939 scope.go:122] "RemoveContainer" containerID="73d99d7b4c933b05257f5b032f0a345d12a06df1c46c832538b61cd6029cf4bc" Nov 03 01:03:38.874977 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:03:38.874903 4939 generic.go:356] "Generic (PLEG): container finished" podID="e6830dd6-9ccc-47ca-b7a3-3cbb6c9bc8d4" containerID="493d816a46f7a89b30934d47cc9ce25b8c44fd000b264b8cf8f8b553fbeec657" exitCode=0 Nov 03 01:03:38.874977 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:03:38.874957 4939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-b9dad7-7458764995-5pdtn" event={"ID":"e6830dd6-9ccc-47ca-b7a3-3cbb6c9bc8d4","Type":"ContainerDied","Data":"493d816a46f7a89b30934d47cc9ce25b8c44fd000b264b8cf8f8b553fbeec657"} Nov 03 01:03:38.875139 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:03:38.875028 4939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-b9dad7-7458764995-5pdtn" event={"ID":"e6830dd6-9ccc-47ca-b7a3-3cbb6c9bc8d4","Type":"ContainerDied","Data":"88344714d3e7ce2fbe5bec9a20680f38cf20c0fd7c4fe8614b56d8e33014cebd"} Nov 03 01:03:38.875139 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:03:38.874975 4939 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-b9dad7-7458764995-5pdtn" Nov 03 01:03:38.891342 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:03:38.891321 4939 scope.go:122] "RemoveContainer" containerID="73d99d7b4c933b05257f5b032f0a345d12a06df1c46c832538b61cd6029cf4bc" Nov 03 01:03:38.891666 ip-10-0-140-107 kubenswrapper[4939]: E1103 01:03:38.891645 4939 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"73d99d7b4c933b05257f5b032f0a345d12a06df1c46c832538b61cd6029cf4bc\": container with ID starting with 73d99d7b4c933b05257f5b032f0a345d12a06df1c46c832538b61cd6029cf4bc not found: ID does not exist" containerID="73d99d7b4c933b05257f5b032f0a345d12a06df1c46c832538b61cd6029cf4bc" Nov 03 01:03:38.891737 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:03:38.891678 4939 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"73d99d7b4c933b05257f5b032f0a345d12a06df1c46c832538b61cd6029cf4bc"} err="failed to get container status \"73d99d7b4c933b05257f5b032f0a345d12a06df1c46c832538b61cd6029cf4bc\": rpc error: code = NotFound desc = could not find container \"73d99d7b4c933b05257f5b032f0a345d12a06df1c46c832538b61cd6029cf4bc\": container with ID starting with 73d99d7b4c933b05257f5b032f0a345d12a06df1c46c832538b61cd6029cf4bc not found: ID does not exist" Nov 03 01:03:38.891737 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:03:38.891702 4939 scope.go:122] "RemoveContainer" containerID="493d816a46f7a89b30934d47cc9ce25b8c44fd000b264b8cf8f8b553fbeec657" Nov 03 01:03:38.902685 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:03:38.902665 4939 scope.go:122] "RemoveContainer" containerID="493d816a46f7a89b30934d47cc9ce25b8c44fd000b264b8cf8f8b553fbeec657" Nov 03 01:03:38.903652 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:03:38.903634 4939 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mq8k5\" (UniqueName: \"kubernetes.io/projected/e6830dd6-9ccc-47ca-b7a3-3cbb6c9bc8d4-kube-api-access-mq8k5\") on node \"ip-10-0-140-107.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:03:38.903730 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:03:38.903657 4939 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e6830dd6-9ccc-47ca-b7a3-3cbb6c9bc8d4-plugins-config-volume\") on node \"ip-10-0-140-107.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:03:38.904005 ip-10-0-140-107 kubenswrapper[4939]: E1103 01:03:38.903926 4939 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"493d816a46f7a89b30934d47cc9ce25b8c44fd000b264b8cf8f8b553fbeec657\": container with ID starting with 493d816a46f7a89b30934d47cc9ce25b8c44fd000b264b8cf8f8b553fbeec657 not found: ID does not exist" containerID="493d816a46f7a89b30934d47cc9ce25b8c44fd000b264b8cf8f8b553fbeec657" Nov 03 01:03:38.904005 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:03:38.903958 4939 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"493d816a46f7a89b30934d47cc9ce25b8c44fd000b264b8cf8f8b553fbeec657"} err="failed to get container status \"493d816a46f7a89b30934d47cc9ce25b8c44fd000b264b8cf8f8b553fbeec657\": rpc error: code = NotFound desc = could not find container \"493d816a46f7a89b30934d47cc9ce25b8c44fd000b264b8cf8f8b553fbeec657\": container with ID starting with 493d816a46f7a89b30934d47cc9ce25b8c44fd000b264b8cf8f8b553fbeec657 not found: ID does not exist" Nov 03 01:03:38.905117 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:03:38.905100 4939 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-b9dad7-8569798f65-m9hcd"] Nov 03 01:03:38.908211 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:03:38.908187 4939 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-b9dad7-8569798f65-m9hcd"] Nov 03 01:03:38.918654 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:03:38.918630 4939 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-b9dad7-7458764995-5pdtn"] Nov 03 01:03:38.921479 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:03:38.921457 4939 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-b9dad7-7458764995-5pdtn"] Nov 03 01:03:39.363111 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:03:39.363076 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:03:39.870351 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:03:39.870317 4939 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2b03c0be-253a-41f3-907b-ec20ddc72a52" path="/var/lib/kubelet/pods/2b03c0be-253a-41f3-907b-ec20ddc72a52/volumes" Nov 03 01:03:39.870840 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:03:39.870818 4939 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e6830dd6-9ccc-47ca-b7a3-3cbb6c9bc8d4" path="/var/lib/kubelet/pods/e6830dd6-9ccc-47ca-b7a3-3cbb6c9bc8d4/volumes" Nov 03 01:03:40.070946 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:03:40.070897 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:03:40.313523 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:03:40.313448 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:03:40.770577 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:03:40.770535 4939 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-b9dad7-7fbdb867c-zrgdg"] Nov 03 01:03:40.770821 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:03:40.770792 4939 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-b9dad7-7fbdb867c-zrgdg" podUID="220bbc69-e7e7-4f85-9d3d-8b5b0f6bede3" containerName="maas-api" containerID="cri-o://240d8a12a0ebeeb3eef3d423715387379095093b4755435966d05a225ce4712a" gracePeriod=30 Nov 03 01:03:40.840935 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:03:40.840896 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:03:41.277000 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:03:41.276974 4939 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-b9dad7-7fbdb867c-zrgdg" Nov 03 01:03:41.310601 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:03:41.310531 4939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/220bbc69-e7e7-4f85-9d3d-8b5b0f6bede3-kube-api-access-j8mch\" (UniqueName: \"kubernetes.io/projected/220bbc69-e7e7-4f85-9d3d-8b5b0f6bede3-kube-api-access-j8mch\") pod \"220bbc69-e7e7-4f85-9d3d-8b5b0f6bede3\" (UID: \"220bbc69-e7e7-4f85-9d3d-8b5b0f6bede3\") " Nov 03 01:03:41.310601 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:03:41.310572 4939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/220bbc69-e7e7-4f85-9d3d-8b5b0f6bede3-metrics-certs\" (UniqueName: \"kubernetes.io/secret/220bbc69-e7e7-4f85-9d3d-8b5b0f6bede3-metrics-certs\") pod \"220bbc69-e7e7-4f85-9d3d-8b5b0f6bede3\" (UID: \"220bbc69-e7e7-4f85-9d3d-8b5b0f6bede3\") " Nov 03 01:03:41.310872 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:03:41.310636 4939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/220bbc69-e7e7-4f85-9d3d-8b5b0f6bede3-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/220bbc69-e7e7-4f85-9d3d-8b5b0f6bede3-maas-api-tls\") pod \"220bbc69-e7e7-4f85-9d3d-8b5b0f6bede3\" (UID: \"220bbc69-e7e7-4f85-9d3d-8b5b0f6bede3\") " Nov 03 01:03:41.310872 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:03:41.310704 4939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/220bbc69-e7e7-4f85-9d3d-8b5b0f6bede3-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/220bbc69-e7e7-4f85-9d3d-8b5b0f6bede3-openshift-service-ca\") pod \"220bbc69-e7e7-4f85-9d3d-8b5b0f6bede3\" (UID: \"220bbc69-e7e7-4f85-9d3d-8b5b0f6bede3\") " Nov 03 01:03:41.311424 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:03:41.311391 4939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/220bbc69-e7e7-4f85-9d3d-8b5b0f6bede3-openshift-service-ca" pod "220bbc69-e7e7-4f85-9d3d-8b5b0f6bede3" (UID: "220bbc69-e7e7-4f85-9d3d-8b5b0f6bede3"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 01:03:41.312999 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:03:41.312958 4939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/220bbc69-e7e7-4f85-9d3d-8b5b0f6bede3-maas-api-tls" pod "220bbc69-e7e7-4f85-9d3d-8b5b0f6bede3" (UID: "220bbc69-e7e7-4f85-9d3d-8b5b0f6bede3"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 01:03:41.313331 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:03:41.313291 4939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/220bbc69-e7e7-4f85-9d3d-8b5b0f6bede3-metrics-certs" pod "220bbc69-e7e7-4f85-9d3d-8b5b0f6bede3" (UID: "220bbc69-e7e7-4f85-9d3d-8b5b0f6bede3"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 01:03:41.313673 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:03:41.313648 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:03:41.314768 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:03:41.314741 4939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/220bbc69-e7e7-4f85-9d3d-8b5b0f6bede3-kube-api-access-j8mch" pod "220bbc69-e7e7-4f85-9d3d-8b5b0f6bede3" (UID: "220bbc69-e7e7-4f85-9d3d-8b5b0f6bede3"). InnerVolumeSpecName "kube-api-access-j8mch". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 01:03:41.427930 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:03:41.423037 4939 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/220bbc69-e7e7-4f85-9d3d-8b5b0f6bede3-openshift-service-ca\") on node \"ip-10-0-140-107.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:03:41.427930 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:03:41.423080 4939 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-j8mch\" (UniqueName: \"kubernetes.io/projected/220bbc69-e7e7-4f85-9d3d-8b5b0f6bede3-kube-api-access-j8mch\") on node \"ip-10-0-140-107.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:03:41.427930 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:03:41.423096 4939 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/220bbc69-e7e7-4f85-9d3d-8b5b0f6bede3-metrics-certs\") on node \"ip-10-0-140-107.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:03:41.427930 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:03:41.423112 4939 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/220bbc69-e7e7-4f85-9d3d-8b5b0f6bede3-maas-api-tls\") on node \"ip-10-0-140-107.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:03:41.902012 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:03:41.901976 4939 generic.go:356] "Generic (PLEG): container finished" podID="220bbc69-e7e7-4f85-9d3d-8b5b0f6bede3" containerID="240d8a12a0ebeeb3eef3d423715387379095093b4755435966d05a225ce4712a" exitCode=0 Nov 03 01:03:41.902187 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:03:41.902021 4939 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-b9dad7-7fbdb867c-zrgdg" Nov 03 01:03:41.902187 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:03:41.902056 4939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-b9dad7-7fbdb867c-zrgdg" event={"ID":"220bbc69-e7e7-4f85-9d3d-8b5b0f6bede3","Type":"ContainerDied","Data":"240d8a12a0ebeeb3eef3d423715387379095093b4755435966d05a225ce4712a"} Nov 03 01:03:41.902187 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:03:41.902120 4939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-b9dad7-7fbdb867c-zrgdg" event={"ID":"220bbc69-e7e7-4f85-9d3d-8b5b0f6bede3","Type":"ContainerDied","Data":"490c7b2c9d3bf74d78061ab7f7d0f8b5def9e860f5b7a23cd1dfa3d5fd8732bc"} Nov 03 01:03:41.902187 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:03:41.902148 4939 scope.go:122] "RemoveContainer" containerID="240d8a12a0ebeeb3eef3d423715387379095093b4755435966d05a225ce4712a" Nov 03 01:03:41.912817 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:03:41.912793 4939 scope.go:122] "RemoveContainer" containerID="240d8a12a0ebeeb3eef3d423715387379095093b4755435966d05a225ce4712a" Nov 03 01:03:41.913090 ip-10-0-140-107 kubenswrapper[4939]: E1103 01:03:41.913072 4939 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"240d8a12a0ebeeb3eef3d423715387379095093b4755435966d05a225ce4712a\": container with ID starting with 240d8a12a0ebeeb3eef3d423715387379095093b4755435966d05a225ce4712a not found: ID does not exist" containerID="240d8a12a0ebeeb3eef3d423715387379095093b4755435966d05a225ce4712a" Nov 03 01:03:41.913135 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:03:41.913101 4939 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"240d8a12a0ebeeb3eef3d423715387379095093b4755435966d05a225ce4712a"} err="failed to get container status \"240d8a12a0ebeeb3eef3d423715387379095093b4755435966d05a225ce4712a\": rpc error: code = NotFound desc = could not find container \"240d8a12a0ebeeb3eef3d423715387379095093b4755435966d05a225ce4712a\": container with ID starting with 240d8a12a0ebeeb3eef3d423715387379095093b4755435966d05a225ce4712a not found: ID does not exist" Nov 03 01:03:41.919847 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:03:41.919826 4939 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-b9dad7-7fbdb867c-zrgdg"] Nov 03 01:03:41.922631 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:03:41.922613 4939 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-b9dad7-7fbdb867c-zrgdg"] Nov 03 01:03:42.009606 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:03:42.009573 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:03:42.808269 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:03:42.808238 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:03:42.983589 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:03:42.983224 4939 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-650be0-openshift-default-5c7c5fb7d9-flb5n"] Nov 03 01:03:42.991474 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:03:42.991442 4939 cpu_manager.go:414] "containerMap: removing container" podUID="fb530145-405c-43ea-946c-e7f2c85cec68" containerName="payload-pre-processing" Nov 03 01:03:43.007010 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:03:43.006981 4939 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fb530145-405c-43ea-946c-e7f2c85cec68" containerName="payload-pre-processing" Nov 03 01:03:43.007275 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:03:43.007260 4939 cpu_manager.go:414] "containerMap: removing container" podUID="e6830dd6-9ccc-47ca-b7a3-3cbb6c9bc8d4" containerName="payload-pre-processing" Nov 03 01:03:43.007373 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:03:43.007359 4939 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e6830dd6-9ccc-47ca-b7a3-3cbb6c9bc8d4" containerName="payload-pre-processing" Nov 03 01:03:43.007480 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:03:43.007469 4939 cpu_manager.go:414] "containerMap: removing container" podUID="cf320aba-5bf7-42fe-a4a9-603130676712" containerName="maas-api" Nov 03 01:03:43.007561 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:03:43.007551 4939 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cf320aba-5bf7-42fe-a4a9-603130676712" containerName="maas-api" Nov 03 01:03:43.007643 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:03:43.007633 4939 cpu_manager.go:414] "containerMap: removing container" podUID="220bbc69-e7e7-4f85-9d3d-8b5b0f6bede3" containerName="maas-api" Nov 03 01:03:43.007712 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:03:43.007702 4939 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="220bbc69-e7e7-4f85-9d3d-8b5b0f6bede3" containerName="maas-api" Nov 03 01:03:43.007869 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:03:43.007857 4939 cpu_manager.go:414] "containerMap: removing container" podUID="6c13fe13-2eda-4d7c-bf31-c522d7f3b9c3" containerName="istio-proxy" Nov 03 01:03:43.007971 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:03:43.007958 4939 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6c13fe13-2eda-4d7c-bf31-c522d7f3b9c3" containerName="istio-proxy" Nov 03 01:03:43.008091 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:03:43.008081 4939 cpu_manager.go:414] "containerMap: removing container" podUID="6c13fe13-2eda-4d7c-bf31-c522d7f3b9c3" containerName="istio-proxy" Nov 03 01:03:43.008168 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:03:43.008158 4939 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6c13fe13-2eda-4d7c-bf31-c522d7f3b9c3" containerName="istio-proxy" Nov 03 01:03:43.008249 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:03:43.008240 4939 cpu_manager.go:414] "containerMap: removing container" podUID="2b03c0be-253a-41f3-907b-ec20ddc72a52" containerName="payload-processing" Nov 03 01:03:43.008327 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:03:43.008316 4939 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2b03c0be-253a-41f3-907b-ec20ddc72a52" containerName="payload-processing" Nov 03 01:03:43.008644 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:03:43.008625 4939 memory_manager.go:365] "RemoveStaleState removing state" podUID="e6830dd6-9ccc-47ca-b7a3-3cbb6c9bc8d4" containerName="payload-pre-processing" Nov 03 01:03:43.008765 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:03:43.008754 4939 memory_manager.go:365] "RemoveStaleState removing state" podUID="cf320aba-5bf7-42fe-a4a9-603130676712" containerName="maas-api" Nov 03 01:03:43.008842 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:03:43.008833 4939 memory_manager.go:365] "RemoveStaleState removing state" podUID="220bbc69-e7e7-4f85-9d3d-8b5b0f6bede3" containerName="maas-api" Nov 03 01:03:43.008941 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:03:43.008931 4939 memory_manager.go:365] "RemoveStaleState removing state" podUID="6c13fe13-2eda-4d7c-bf31-c522d7f3b9c3" containerName="istio-proxy" Nov 03 01:03:43.009063 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:03:43.009050 4939 memory_manager.go:365] "RemoveStaleState removing state" podUID="6c13fe13-2eda-4d7c-bf31-c522d7f3b9c3" containerName="istio-proxy" Nov 03 01:03:43.009162 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:03:43.009152 4939 memory_manager.go:365] "RemoveStaleState removing state" podUID="2b03c0be-253a-41f3-907b-ec20ddc72a52" containerName="payload-processing" Nov 03 01:03:43.009251 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:03:43.009241 4939 memory_manager.go:365] "RemoveStaleState removing state" podUID="fb530145-405c-43ea-946c-e7f2c85cec68" containerName="payload-pre-processing" Nov 03 01:03:43.035693 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:03:43.006760 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:03:43.068031 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:03:43.059140 4939 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-650be0-openshift-default-5c7c5fb7d9-flb5n"] Nov 03 01:03:43.068031 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:03:43.059287 4939 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-650be0-openshift-default-5c7c5fb7d9-flb5n" Nov 03 01:03:43.180977 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:03:43.180821 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9146740f-e0d2-46cb-a063-b5942f02b275-credential-socket\") pod \"e2e-worker-w4-650be0-openshift-default-5c7c5fb7d9-flb5n\" (UID: \"9146740f-e0d2-46cb-a063-b5942f02b275\") " pod="openshift-ingress/e2e-worker-w4-650be0-openshift-default-5c7c5fb7d9-flb5n" Nov 03 01:03:43.180977 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:03:43.180885 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9146740f-e0d2-46cb-a063-b5942f02b275-istiod-ca-cert\") pod \"e2e-worker-w4-650be0-openshift-default-5c7c5fb7d9-flb5n\" (UID: \"9146740f-e0d2-46cb-a063-b5942f02b275\") " pod="openshift-ingress/e2e-worker-w4-650be0-openshift-default-5c7c5fb7d9-flb5n" Nov 03 01:03:43.180977 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:03:43.180959 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-clt5s\" (UniqueName: \"kubernetes.io/projected/9146740f-e0d2-46cb-a063-b5942f02b275-kube-api-access-clt5s\") pod \"e2e-worker-w4-650be0-openshift-default-5c7c5fb7d9-flb5n\" (UID: \"9146740f-e0d2-46cb-a063-b5942f02b275\") " pod="openshift-ingress/e2e-worker-w4-650be0-openshift-default-5c7c5fb7d9-flb5n" Nov 03 01:03:43.181255 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:03:43.180999 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9146740f-e0d2-46cb-a063-b5942f02b275-workload-certs\") pod \"e2e-worker-w4-650be0-openshift-default-5c7c5fb7d9-flb5n\" (UID: \"9146740f-e0d2-46cb-a063-b5942f02b275\") " pod="openshift-ingress/e2e-worker-w4-650be0-openshift-default-5c7c5fb7d9-flb5n" Nov 03 01:03:43.181255 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:03:43.181059 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9146740f-e0d2-46cb-a063-b5942f02b275-istio-token\") pod \"e2e-worker-w4-650be0-openshift-default-5c7c5fb7d9-flb5n\" (UID: \"9146740f-e0d2-46cb-a063-b5942f02b275\") " pod="openshift-ingress/e2e-worker-w4-650be0-openshift-default-5c7c5fb7d9-flb5n" Nov 03 01:03:43.181255 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:03:43.181159 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9146740f-e0d2-46cb-a063-b5942f02b275-istio-data\") pod \"e2e-worker-w4-650be0-openshift-default-5c7c5fb7d9-flb5n\" (UID: \"9146740f-e0d2-46cb-a063-b5942f02b275\") " pod="openshift-ingress/e2e-worker-w4-650be0-openshift-default-5c7c5fb7d9-flb5n" Nov 03 01:03:43.181385 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:03:43.181281 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9146740f-e0d2-46cb-a063-b5942f02b275-istio-envoy\") pod \"e2e-worker-w4-650be0-openshift-default-5c7c5fb7d9-flb5n\" (UID: \"9146740f-e0d2-46cb-a063-b5942f02b275\") " pod="openshift-ingress/e2e-worker-w4-650be0-openshift-default-5c7c5fb7d9-flb5n" Nov 03 01:03:43.181447 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:03:43.181418 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9146740f-e0d2-46cb-a063-b5942f02b275-workload-socket\") pod \"e2e-worker-w4-650be0-openshift-default-5c7c5fb7d9-flb5n\" (UID: \"9146740f-e0d2-46cb-a063-b5942f02b275\") " pod="openshift-ingress/e2e-worker-w4-650be0-openshift-default-5c7c5fb7d9-flb5n" Nov 03 01:03:43.181495 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:03:43.181481 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9146740f-e0d2-46cb-a063-b5942f02b275-istio-podinfo\") pod \"e2e-worker-w4-650be0-openshift-default-5c7c5fb7d9-flb5n\" (UID: \"9146740f-e0d2-46cb-a063-b5942f02b275\") " pod="openshift-ingress/e2e-worker-w4-650be0-openshift-default-5c7c5fb7d9-flb5n" Nov 03 01:03:43.283022 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:03:43.282985 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9146740f-e0d2-46cb-a063-b5942f02b275-istio-podinfo\") pod \"e2e-worker-w4-650be0-openshift-default-5c7c5fb7d9-flb5n\" (UID: \"9146740f-e0d2-46cb-a063-b5942f02b275\") " pod="openshift-ingress/e2e-worker-w4-650be0-openshift-default-5c7c5fb7d9-flb5n" Nov 03 01:03:43.283218 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:03:43.283060 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9146740f-e0d2-46cb-a063-b5942f02b275-credential-socket\") pod \"e2e-worker-w4-650be0-openshift-default-5c7c5fb7d9-flb5n\" (UID: \"9146740f-e0d2-46cb-a063-b5942f02b275\") " pod="openshift-ingress/e2e-worker-w4-650be0-openshift-default-5c7c5fb7d9-flb5n" Nov 03 01:03:43.283218 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:03:43.283097 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9146740f-e0d2-46cb-a063-b5942f02b275-istiod-ca-cert\") pod \"e2e-worker-w4-650be0-openshift-default-5c7c5fb7d9-flb5n\" (UID: \"9146740f-e0d2-46cb-a063-b5942f02b275\") " pod="openshift-ingress/e2e-worker-w4-650be0-openshift-default-5c7c5fb7d9-flb5n" Nov 03 01:03:43.283519 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:03:43.283488 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-clt5s\" (UniqueName: \"kubernetes.io/projected/9146740f-e0d2-46cb-a063-b5942f02b275-kube-api-access-clt5s\") pod \"e2e-worker-w4-650be0-openshift-default-5c7c5fb7d9-flb5n\" (UID: \"9146740f-e0d2-46cb-a063-b5942f02b275\") " pod="openshift-ingress/e2e-worker-w4-650be0-openshift-default-5c7c5fb7d9-flb5n" Nov 03 01:03:43.283636 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:03:43.283552 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9146740f-e0d2-46cb-a063-b5942f02b275-workload-certs\") pod \"e2e-worker-w4-650be0-openshift-default-5c7c5fb7d9-flb5n\" (UID: \"9146740f-e0d2-46cb-a063-b5942f02b275\") " pod="openshift-ingress/e2e-worker-w4-650be0-openshift-default-5c7c5fb7d9-flb5n" Nov 03 01:03:43.283636 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:03:43.283602 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9146740f-e0d2-46cb-a063-b5942f02b275-istio-token\") pod \"e2e-worker-w4-650be0-openshift-default-5c7c5fb7d9-flb5n\" (UID: \"9146740f-e0d2-46cb-a063-b5942f02b275\") " pod="openshift-ingress/e2e-worker-w4-650be0-openshift-default-5c7c5fb7d9-flb5n" Nov 03 01:03:43.283749 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:03:43.283625 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9146740f-e0d2-46cb-a063-b5942f02b275-credential-socket\") pod \"e2e-worker-w4-650be0-openshift-default-5c7c5fb7d9-flb5n\" (UID: \"9146740f-e0d2-46cb-a063-b5942f02b275\") " pod="openshift-ingress/e2e-worker-w4-650be0-openshift-default-5c7c5fb7d9-flb5n" Nov 03 01:03:43.283749 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:03:43.283648 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9146740f-e0d2-46cb-a063-b5942f02b275-istio-data\") pod \"e2e-worker-w4-650be0-openshift-default-5c7c5fb7d9-flb5n\" (UID: \"9146740f-e0d2-46cb-a063-b5942f02b275\") " pod="openshift-ingress/e2e-worker-w4-650be0-openshift-default-5c7c5fb7d9-flb5n" Nov 03 01:03:43.283865 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:03:43.283784 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9146740f-e0d2-46cb-a063-b5942f02b275-istio-envoy\") pod \"e2e-worker-w4-650be0-openshift-default-5c7c5fb7d9-flb5n\" (UID: \"9146740f-e0d2-46cb-a063-b5942f02b275\") " pod="openshift-ingress/e2e-worker-w4-650be0-openshift-default-5c7c5fb7d9-flb5n" Nov 03 01:03:43.283972 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:03:43.283870 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9146740f-e0d2-46cb-a063-b5942f02b275-workload-certs\") pod \"e2e-worker-w4-650be0-openshift-default-5c7c5fb7d9-flb5n\" (UID: \"9146740f-e0d2-46cb-a063-b5942f02b275\") " pod="openshift-ingress/e2e-worker-w4-650be0-openshift-default-5c7c5fb7d9-flb5n" Nov 03 01:03:43.284058 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:03:43.283976 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9146740f-e0d2-46cb-a063-b5942f02b275-workload-socket\") pod \"e2e-worker-w4-650be0-openshift-default-5c7c5fb7d9-flb5n\" (UID: \"9146740f-e0d2-46cb-a063-b5942f02b275\") " pod="openshift-ingress/e2e-worker-w4-650be0-openshift-default-5c7c5fb7d9-flb5n" Nov 03 01:03:43.284058 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:03:43.283999 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9146740f-e0d2-46cb-a063-b5942f02b275-istiod-ca-cert\") pod \"e2e-worker-w4-650be0-openshift-default-5c7c5fb7d9-flb5n\" (UID: \"9146740f-e0d2-46cb-a063-b5942f02b275\") " pod="openshift-ingress/e2e-worker-w4-650be0-openshift-default-5c7c5fb7d9-flb5n" Nov 03 01:03:43.284777 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:03:43.284724 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9146740f-e0d2-46cb-a063-b5942f02b275-istio-data\") pod \"e2e-worker-w4-650be0-openshift-default-5c7c5fb7d9-flb5n\" (UID: \"9146740f-e0d2-46cb-a063-b5942f02b275\") " pod="openshift-ingress/e2e-worker-w4-650be0-openshift-default-5c7c5fb7d9-flb5n" Nov 03 01:03:43.284921 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:03:43.284807 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9146740f-e0d2-46cb-a063-b5942f02b275-workload-socket\") pod \"e2e-worker-w4-650be0-openshift-default-5c7c5fb7d9-flb5n\" (UID: \"9146740f-e0d2-46cb-a063-b5942f02b275\") " pod="openshift-ingress/e2e-worker-w4-650be0-openshift-default-5c7c5fb7d9-flb5n" Nov 03 01:03:43.288718 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:03:43.288415 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9146740f-e0d2-46cb-a063-b5942f02b275-istio-envoy\") pod \"e2e-worker-w4-650be0-openshift-default-5c7c5fb7d9-flb5n\" (UID: \"9146740f-e0d2-46cb-a063-b5942f02b275\") " pod="openshift-ingress/e2e-worker-w4-650be0-openshift-default-5c7c5fb7d9-flb5n" Nov 03 01:03:43.288718 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:03:43.288677 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9146740f-e0d2-46cb-a063-b5942f02b275-istio-podinfo\") pod \"e2e-worker-w4-650be0-openshift-default-5c7c5fb7d9-flb5n\" (UID: \"9146740f-e0d2-46cb-a063-b5942f02b275\") " pod="openshift-ingress/e2e-worker-w4-650be0-openshift-default-5c7c5fb7d9-flb5n" Nov 03 01:03:43.296813 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:03:43.296785 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9146740f-e0d2-46cb-a063-b5942f02b275-istio-token\") pod \"e2e-worker-w4-650be0-openshift-default-5c7c5fb7d9-flb5n\" (UID: \"9146740f-e0d2-46cb-a063-b5942f02b275\") " pod="openshift-ingress/e2e-worker-w4-650be0-openshift-default-5c7c5fb7d9-flb5n" Nov 03 01:03:43.297114 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:03:43.297097 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-clt5s\" (UniqueName: \"kubernetes.io/projected/9146740f-e0d2-46cb-a063-b5942f02b275-kube-api-access-clt5s\") pod \"e2e-worker-w4-650be0-openshift-default-5c7c5fb7d9-flb5n\" (UID: \"9146740f-e0d2-46cb-a063-b5942f02b275\") " pod="openshift-ingress/e2e-worker-w4-650be0-openshift-default-5c7c5fb7d9-flb5n" Nov 03 01:03:43.496573 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:03:43.496526 4939 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-650be0-openshift-default-5c7c5fb7d9-flb5n" Nov 03 01:03:43.873004 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:03:43.872958 4939 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="220bbc69-e7e7-4f85-9d3d-8b5b0f6bede3" path="/var/lib/kubelet/pods/220bbc69-e7e7-4f85-9d3d-8b5b0f6bede3/volumes" Nov 03 01:03:43.984314 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:03:43.980250 4939 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-650be0-openshift-default-5c7c5fb7d9-flb5n"] Nov 03 01:03:43.991663 ip-10-0-140-107 kubenswrapper[4939]: W1103 01:03:43.991631 4939 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9146740f_e0d2_46cb_a063_b5942f02b275.slice/crio-46950e63eca92b4fc2960c90e6f76b7348913958ba02e6e3d776783b8a3fecf7 WatchSource:0}: Error finding container 46950e63eca92b4fc2960c90e6f76b7348913958ba02e6e3d776783b8a3fecf7: Status 404 returned error can't find the container with id 46950e63eca92b4fc2960c90e6f76b7348913958ba02e6e3d776783b8a3fecf7 Nov 03 01:03:43.997176 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:03:43.997141 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:03:44.014941 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:03:44.014582 4939 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 01:03:44.014941 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:03:44.014694 4939 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 01:03:44.014941 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:03:44.014749 4939 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 01:03:45.230935 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:03:45.220779 4939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-650be0-openshift-default-5c7c5fb7d9-flb5n" event={"ID":"9146740f-e0d2-46cb-a063-b5942f02b275","Type":"ContainerStarted","Data":"45291eab55af92c5381548b238ac110b1f64a1980acbeeca8868720be4b7caf7"} Nov 03 01:03:45.230935 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:03:45.221817 4939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-650be0-openshift-default-5c7c5fb7d9-flb5n" event={"ID":"9146740f-e0d2-46cb-a063-b5942f02b275","Type":"ContainerStarted","Data":"46950e63eca92b4fc2960c90e6f76b7348913958ba02e6e3d776783b8a3fecf7"} Nov 03 01:03:45.269942 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:03:45.269011 4939 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-650be0-openshift-default-5c7c5fb7d9-flb5n" podStartSLOduration=3.268989392 podStartE2EDuration="3.268989392s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 01:03:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 01:03:45.261429672 +0000 UTC m=+1373.525764620" watchObservedRunningTime="2026-11-03 01:03:45.268989392 +0000 UTC m=+1373.533324339" Nov 03 01:03:45.343427 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:03:45.341600 4939 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-praxis-a-1e352e/praxis-route-1e352e-kserve-67cb6ccfdc-k25pd" podUID="b645db90-ca61-4e9c-b2d7-1e57ebd22e64" containerName="main" containerID="cri-o://45978c90c580f947f1aa640e41cf6cb669f2364cbd5e79162f87b6058777ad4e" gracePeriod=45 Nov 03 01:03:45.489983 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:03:45.489521 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:03:45.511002 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:03:45.510970 4939 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-650be0-openshift-default-5c7c5fb7d9-flb5n" Nov 03 01:03:45.511430 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:03:45.511022 4939 patch_prober.go:29] interesting pod/e2e-worker-w4-650be0-openshift-default-5c7c5fb7d9-flb5n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.160:15021/healthz/ready\": dial tcp 10.132.0.160:15021: connect: connection refused" start-of-body= Nov 03 01:03:45.511589 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:03:45.511568 4939 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-650be0-openshift-default-5c7c5fb7d9-flb5n" podUID="9146740f-e0d2-46cb-a063-b5942f02b275" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.160:15021/healthz/ready\": dial tcp 10.132.0.160:15021: connect: connection refused" Nov 03 01:03:46.158932 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:03:46.154319 4939 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-praxis-a-1e352e/praxis-route-1e352e-kserve-67cb6ccfdc-k25pd" podUID="b645db90-ca61-4e9c-b2d7-1e57ebd22e64" containerName="main" probeResult="failure" output="Get \"https://10.132.0.147:8000/ready\": dial tcp 10.132.0.147:8000: connect: connection refused" Nov 03 01:03:46.240958 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:03:46.236029 4939 generic.go:356] "Generic (PLEG): container finished" podID="b645db90-ca61-4e9c-b2d7-1e57ebd22e64" containerID="45978c90c580f947f1aa640e41cf6cb669f2364cbd5e79162f87b6058777ad4e" exitCode=0 Nov 03 01:03:46.240958 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:03:46.236406 4939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-praxis-a-1e352e/praxis-route-1e352e-kserve-67cb6ccfdc-k25pd" event={"ID":"b645db90-ca61-4e9c-b2d7-1e57ebd22e64","Type":"ContainerDied","Data":"45978c90c580f947f1aa640e41cf6cb669f2364cbd5e79162f87b6058777ad4e"} Nov 03 01:03:46.425816 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:03:46.425785 4939 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-praxis-a-1e352e/praxis-route-1e352e-kserve-67cb6ccfdc-k25pd" Nov 03 01:03:46.502475 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:03:46.502436 4939 patch_prober.go:29] interesting pod/e2e-worker-w4-650be0-openshift-default-5c7c5fb7d9-flb5n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.160:15021/healthz/ready\": dial tcp 10.132.0.160:15021: connect: connection refused" start-of-body= Nov 03 01:03:46.502741 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:03:46.502712 4939 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-650be0-openshift-default-5c7c5fb7d9-flb5n" podUID="9146740f-e0d2-46cb-a063-b5942f02b275" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.160:15021/healthz/ready\": dial tcp 10.132.0.160:15021: connect: connection refused" Nov 03 01:03:46.605937 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:03:46.604724 4939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b645db90-ca61-4e9c-b2d7-1e57ebd22e64-home\" (UniqueName: \"kubernetes.io/empty-dir/b645db90-ca61-4e9c-b2d7-1e57ebd22e64-home\") pod \"b645db90-ca61-4e9c-b2d7-1e57ebd22e64\" (UID: \"b645db90-ca61-4e9c-b2d7-1e57ebd22e64\") " Nov 03 01:03:46.605937 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:03:46.604804 4939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b645db90-ca61-4e9c-b2d7-1e57ebd22e64-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b645db90-ca61-4e9c-b2d7-1e57ebd22e64-tmp-dir\") pod \"b645db90-ca61-4e9c-b2d7-1e57ebd22e64\" (UID: \"b645db90-ca61-4e9c-b2d7-1e57ebd22e64\") " Nov 03 01:03:46.605937 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:03:46.604858 4939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b645db90-ca61-4e9c-b2d7-1e57ebd22e64-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b645db90-ca61-4e9c-b2d7-1e57ebd22e64-model-cache\") pod \"b645db90-ca61-4e9c-b2d7-1e57ebd22e64\" (UID: \"b645db90-ca61-4e9c-b2d7-1e57ebd22e64\") " Nov 03 01:03:46.605937 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:03:46.604924 4939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b645db90-ca61-4e9c-b2d7-1e57ebd22e64-kube-api-access-26fmb\" (UniqueName: \"kubernetes.io/projected/b645db90-ca61-4e9c-b2d7-1e57ebd22e64-kube-api-access-26fmb\") pod \"b645db90-ca61-4e9c-b2d7-1e57ebd22e64\" (UID: \"b645db90-ca61-4e9c-b2d7-1e57ebd22e64\") " Nov 03 01:03:46.605937 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:03:46.604984 4939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b645db90-ca61-4e9c-b2d7-1e57ebd22e64-tls-certs\" (UniqueName: \"kubernetes.io/secret/b645db90-ca61-4e9c-b2d7-1e57ebd22e64-tls-certs\") pod \"b645db90-ca61-4e9c-b2d7-1e57ebd22e64\" (UID: \"b645db90-ca61-4e9c-b2d7-1e57ebd22e64\") " Nov 03 01:03:46.605937 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:03:46.605200 4939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b645db90-ca61-4e9c-b2d7-1e57ebd22e64-dshm\" (UniqueName: \"kubernetes.io/empty-dir/b645db90-ca61-4e9c-b2d7-1e57ebd22e64-dshm\") pod \"b645db90-ca61-4e9c-b2d7-1e57ebd22e64\" (UID: \"b645db90-ca61-4e9c-b2d7-1e57ebd22e64\") " Nov 03 01:03:46.611084 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:03:46.609747 4939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b645db90-ca61-4e9c-b2d7-1e57ebd22e64-home" pod "b645db90-ca61-4e9c-b2d7-1e57ebd22e64" (UID: "b645db90-ca61-4e9c-b2d7-1e57ebd22e64"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 01:03:46.611084 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:03:46.610001 4939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b645db90-ca61-4e9c-b2d7-1e57ebd22e64-dshm" pod "b645db90-ca61-4e9c-b2d7-1e57ebd22e64" (UID: "b645db90-ca61-4e9c-b2d7-1e57ebd22e64"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 01:03:46.611084 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:03:46.610986 4939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b645db90-ca61-4e9c-b2d7-1e57ebd22e64-tmp-dir" pod "b645db90-ca61-4e9c-b2d7-1e57ebd22e64" (UID: "b645db90-ca61-4e9c-b2d7-1e57ebd22e64"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 01:03:46.611893 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:03:46.611840 4939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b645db90-ca61-4e9c-b2d7-1e57ebd22e64-model-cache" pod "b645db90-ca61-4e9c-b2d7-1e57ebd22e64" (UID: "b645db90-ca61-4e9c-b2d7-1e57ebd22e64"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 01:03:46.615016 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:03:46.614776 4939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b645db90-ca61-4e9c-b2d7-1e57ebd22e64-tls-certs" pod "b645db90-ca61-4e9c-b2d7-1e57ebd22e64" (UID: "b645db90-ca61-4e9c-b2d7-1e57ebd22e64"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 01:03:46.615277 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:03:46.615252 4939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b645db90-ca61-4e9c-b2d7-1e57ebd22e64-kube-api-access-26fmb" pod "b645db90-ca61-4e9c-b2d7-1e57ebd22e64" (UID: "b645db90-ca61-4e9c-b2d7-1e57ebd22e64"). InnerVolumeSpecName "kube-api-access-26fmb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 01:03:46.668933 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:03:46.668329 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:03:46.719953 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:03:46.713193 4939 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b645db90-ca61-4e9c-b2d7-1e57ebd22e64-dshm\") on node \"ip-10-0-140-107.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:03:46.719953 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:03:46.713236 4939 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b645db90-ca61-4e9c-b2d7-1e57ebd22e64-home\") on node \"ip-10-0-140-107.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:03:46.719953 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:03:46.713251 4939 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b645db90-ca61-4e9c-b2d7-1e57ebd22e64-tmp-dir\") on node \"ip-10-0-140-107.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:03:46.719953 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:03:46.713266 4939 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b645db90-ca61-4e9c-b2d7-1e57ebd22e64-model-cache\") on node \"ip-10-0-140-107.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:03:46.719953 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:03:46.713282 4939 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-26fmb\" (UniqueName: \"kubernetes.io/projected/b645db90-ca61-4e9c-b2d7-1e57ebd22e64-kube-api-access-26fmb\") on node \"ip-10-0-140-107.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:03:46.719953 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:03:46.713296 4939 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b645db90-ca61-4e9c-b2d7-1e57ebd22e64-tls-certs\") on node \"ip-10-0-140-107.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:03:47.171773 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:03:47.171725 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:03:47.253933 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:03:47.251064 4939 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-praxis-a-1e352e/praxis-route-1e352e-kserve-67cb6ccfdc-k25pd" Nov 03 01:03:47.253933 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:03:47.251959 4939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-praxis-a-1e352e/praxis-route-1e352e-kserve-67cb6ccfdc-k25pd" event={"ID":"b645db90-ca61-4e9c-b2d7-1e57ebd22e64","Type":"ContainerDied","Data":"58ef122cf732868ebea5f938c39a8163251a6358332dd848fbcafe22781256f7"} Nov 03 01:03:47.253933 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:03:47.252016 4939 scope.go:122] "RemoveContainer" containerID="45978c90c580f947f1aa640e41cf6cb669f2364cbd5e79162f87b6058777ad4e" Nov 03 01:03:47.307855 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:03:47.307826 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:03:47.314350 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:03:47.314321 4939 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-praxis-a-1e352e/praxis-route-1e352e-kserve-67cb6ccfdc-k25pd"] Nov 03 01:03:47.319715 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:03:47.319692 4939 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-praxis-a-1e352e/praxis-route-1e352e-kserve-67cb6ccfdc-k25pd"] Nov 03 01:03:47.501490 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:03:47.501396 4939 patch_prober.go:29] interesting pod/e2e-worker-w4-650be0-openshift-default-5c7c5fb7d9-flb5n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.160:15021/healthz/ready\": dial tcp 10.132.0.160:15021: connect: connection refused" start-of-body= Nov 03 01:03:47.501781 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:03:47.501740 4939 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-650be0-openshift-default-5c7c5fb7d9-flb5n" podUID="9146740f-e0d2-46cb-a063-b5942f02b275" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.160:15021/healthz/ready\": dial tcp 10.132.0.160:15021: connect: connection refused" Nov 03 01:03:47.903561 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:03:47.903505 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:03:47.904848 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:03:47.904824 4939 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b645db90-ca61-4e9c-b2d7-1e57ebd22e64" path="/var/lib/kubelet/pods/b645db90-ca61-4e9c-b2d7-1e57ebd22e64/volumes" Nov 03 01:03:48.051231 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:03:48.051149 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:03:48.216600 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:03:48.216508 4939 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-praxis-a-b9dad7-openshift-default-994bfcbb9-bw5wc"] Nov 03 01:03:48.217114 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:03:48.217081 4939 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-praxis-a-b9dad7-openshift-default-994bfcbb9-bw5wc" podUID="40f81e1f-3df1-4802-9636-b2ab2263f0ae" containerName="istio-proxy" containerID="cri-o://ddbdc7e8f3c9a2d14e8931c42077f4b592c2d0714bc4e4c025b57807a7a51baf" gracePeriod=30 Nov 03 01:03:48.233957 ip-10-0-140-107 kubenswrapper[4939]: E1103 01:03:48.233924 4939 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-sthjh" containerName="extractor" Nov 03 01:03:48.418934 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:03:48.389690 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:03:48.603986 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:03:48.603947 4939 patch_prober.go:29] interesting pod/e2e-worker-w4-650be0-openshift-default-5c7c5fb7d9-flb5n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.160:15021/healthz/ready\": dial tcp 10.132.0.160:15021: connect: connection refused" start-of-body= Nov 03 01:03:48.604287 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:03:48.604255 4939 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-650be0-openshift-default-5c7c5fb7d9-flb5n" podUID="9146740f-e0d2-46cb-a063-b5942f02b275" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.160:15021/healthz/ready\": dial tcp 10.132.0.160:15021: connect: connection refused" Nov 03 01:03:49.030703 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:03:49.030626 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:03:49.152458 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:03:49.152414 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:03:49.498633 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:03:49.498595 4939 patch_prober.go:29] interesting pod/e2e-worker-w4-650be0-openshift-default-5c7c5fb7d9-flb5n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.160:15021/healthz/ready\": dial tcp 10.132.0.160:15021: connect: connection refused" start-of-body= Nov 03 01:03:49.499273 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:03:49.499238 4939 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-650be0-openshift-default-5c7c5fb7d9-flb5n" podUID="9146740f-e0d2-46cb-a063-b5942f02b275" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.160:15021/healthz/ready\": dial tcp 10.132.0.160:15021: connect: connection refused" Nov 03 01:03:49.926802 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:03:49.926763 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:03:50.498547 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:03:50.498505 4939 patch_prober.go:29] interesting pod/e2e-worker-w4-650be0-openshift-default-5c7c5fb7d9-flb5n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.160:15021/healthz/ready\": dial tcp 10.132.0.160:15021: connect: connection refused" start-of-body= Nov 03 01:03:50.498721 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:03:50.498573 4939 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-650be0-openshift-default-5c7c5fb7d9-flb5n" podUID="9146740f-e0d2-46cb-a063-b5942f02b275" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.160:15021/healthz/ready\": dial tcp 10.132.0.160:15021: connect: connection refused" Nov 03 01:03:50.693846 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:03:50.693805 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:03:51.402732 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:03:51.402692 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:03:51.501931 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:03:51.498609 4939 patch_prober.go:29] interesting pod/e2e-worker-w4-650be0-openshift-default-5c7c5fb7d9-flb5n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.160:15021/healthz/ready\": dial tcp 10.132.0.160:15021: connect: connection refused" start-of-body= Nov 03 01:03:51.501931 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:03:51.498697 4939 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-650be0-openshift-default-5c7c5fb7d9-flb5n" podUID="9146740f-e0d2-46cb-a063-b5942f02b275" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.160:15021/healthz/ready\": dial tcp 10.132.0.160:15021: connect: connection refused" Nov 03 01:03:52.193025 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:03:52.192994 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:03:52.499081 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:03:52.499000 4939 patch_prober.go:29] interesting pod/e2e-worker-w4-650be0-openshift-default-5c7c5fb7d9-flb5n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.160:15021/healthz/ready\": dial tcp 10.132.0.160:15021: connect: connection refused" start-of-body= Nov 03 01:03:52.499081 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:03:52.499056 4939 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-650be0-openshift-default-5c7c5fb7d9-flb5n" podUID="9146740f-e0d2-46cb-a063-b5942f02b275" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.160:15021/healthz/ready\": dial tcp 10.132.0.160:15021: connect: connection refused" Nov 03 01:03:52.802420 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:03:52.802332 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:03:53.078878 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:03:53.078797 4939 scope.go:122] "RemoveContainer" containerID="87933e8f795dd3c4c8815d3376bfaeae21fe9967d3532f65b0242b082c49c545" Nov 03 01:03:53.089762 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:03:53.089734 4939 scope.go:122] "RemoveContainer" containerID="9f50fc61cb95cb3e9c29e3a408c393961c75d1d8f6bf4411670b752aa42c45a7" Nov 03 01:03:53.485930 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:03:53.461377 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:03:53.594991 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:03:53.593210 4939 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-650be0-openshift-default-5c7c5fb7d9-flb5n" Nov 03 01:03:53.594991 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:03:53.594720 4939 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-650be0-openshift-default-5c7c5fb7d9-flb5n" Nov 03 01:03:53.596108 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:03:53.595365 4939 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-650be0-openshift-default-5c7c5fb7d9-flb5n" Nov 03 01:03:53.648410 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:03:53.648383 4939 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-a-b9dad7-openshift-default-994bfcbb9-bw5wc" Nov 03 01:03:53.651197 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:03:53.651167 4939 generic.go:356] "Generic (PLEG): container finished" podID="40f81e1f-3df1-4802-9636-b2ab2263f0ae" containerID="ddbdc7e8f3c9a2d14e8931c42077f4b592c2d0714bc4e4c025b57807a7a51baf" exitCode=0 Nov 03 01:03:53.651348 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:03:53.651205 4939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-b9dad7-openshift-default-994bfcbb9-bw5wc" event={"ID":"40f81e1f-3df1-4802-9636-b2ab2263f0ae","Type":"ContainerDied","Data":"ddbdc7e8f3c9a2d14e8931c42077f4b592c2d0714bc4e4c025b57807a7a51baf"} Nov 03 01:03:53.651348 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:03:53.651300 4939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-b9dad7-openshift-default-994bfcbb9-bw5wc" event={"ID":"40f81e1f-3df1-4802-9636-b2ab2263f0ae","Type":"ContainerDied","Data":"cb9dd7cc876429d260155290c349713abc89df93d941aed80e6eeeb3e2bb5cf1"} Nov 03 01:03:53.651463 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:03:53.651219 4939 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-a-b9dad7-openshift-default-994bfcbb9-bw5wc" Nov 03 01:03:53.651463 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:03:53.651423 4939 scope.go:122] "RemoveContainer" containerID="ddbdc7e8f3c9a2d14e8931c42077f4b592c2d0714bc4e4c025b57807a7a51baf" Nov 03 01:03:53.668096 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:03:53.668069 4939 scope.go:122] "RemoveContainer" containerID="1832905ec9ae5b736eba433d7706628698036e63e972c24219ab929e0b6ff842" Nov 03 01:03:53.702066 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:03:53.696615 4939 scope.go:122] "RemoveContainer" containerID="ddbdc7e8f3c9a2d14e8931c42077f4b592c2d0714bc4e4c025b57807a7a51baf" Nov 03 01:03:53.702066 ip-10-0-140-107 kubenswrapper[4939]: E1103 01:03:53.698430 4939 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ddbdc7e8f3c9a2d14e8931c42077f4b592c2d0714bc4e4c025b57807a7a51baf\": container with ID starting with ddbdc7e8f3c9a2d14e8931c42077f4b592c2d0714bc4e4c025b57807a7a51baf not found: ID does not exist" containerID="ddbdc7e8f3c9a2d14e8931c42077f4b592c2d0714bc4e4c025b57807a7a51baf" Nov 03 01:03:53.702066 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:03:53.698471 4939 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ddbdc7e8f3c9a2d14e8931c42077f4b592c2d0714bc4e4c025b57807a7a51baf"} err="failed to get container status \"ddbdc7e8f3c9a2d14e8931c42077f4b592c2d0714bc4e4c025b57807a7a51baf\": rpc error: code = NotFound desc = could not find container \"ddbdc7e8f3c9a2d14e8931c42077f4b592c2d0714bc4e4c025b57807a7a51baf\": container with ID starting with ddbdc7e8f3c9a2d14e8931c42077f4b592c2d0714bc4e4c025b57807a7a51baf not found: ID does not exist" Nov 03 01:03:53.702066 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:03:53.698499 4939 scope.go:122] "RemoveContainer" containerID="1832905ec9ae5b736eba433d7706628698036e63e972c24219ab929e0b6ff842" Nov 03 01:03:53.702066 ip-10-0-140-107 kubenswrapper[4939]: E1103 01:03:53.698864 4939 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1832905ec9ae5b736eba433d7706628698036e63e972c24219ab929e0b6ff842\": container with ID starting with 1832905ec9ae5b736eba433d7706628698036e63e972c24219ab929e0b6ff842 not found: ID does not exist" containerID="1832905ec9ae5b736eba433d7706628698036e63e972c24219ab929e0b6ff842" Nov 03 01:03:53.702066 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:03:53.698896 4939 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1832905ec9ae5b736eba433d7706628698036e63e972c24219ab929e0b6ff842"} err="failed to get container status \"1832905ec9ae5b736eba433d7706628698036e63e972c24219ab929e0b6ff842\": rpc error: code = NotFound desc = could not find container \"1832905ec9ae5b736eba433d7706628698036e63e972c24219ab929e0b6ff842\": container with ID starting with 1832905ec9ae5b736eba433d7706628698036e63e972c24219ab929e0b6ff842 not found: ID does not exist" Nov 03 01:03:53.767528 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:03:53.767483 4939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/40f81e1f-3df1-4802-9636-b2ab2263f0ae-kube-api-access-2wjcn\" (UniqueName: \"kubernetes.io/projected/40f81e1f-3df1-4802-9636-b2ab2263f0ae-kube-api-access-2wjcn\") pod \"40f81e1f-3df1-4802-9636-b2ab2263f0ae\" (UID: \"40f81e1f-3df1-4802-9636-b2ab2263f0ae\") " Nov 03 01:03:53.767723 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:03:53.767552 4939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/40f81e1f-3df1-4802-9636-b2ab2263f0ae-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/40f81e1f-3df1-4802-9636-b2ab2263f0ae-istio-data\") pod \"40f81e1f-3df1-4802-9636-b2ab2263f0ae\" (UID: \"40f81e1f-3df1-4802-9636-b2ab2263f0ae\") " Nov 03 01:03:53.767723 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:03:53.767599 4939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/40f81e1f-3df1-4802-9636-b2ab2263f0ae-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/40f81e1f-3df1-4802-9636-b2ab2263f0ae-istio-envoy\") pod \"40f81e1f-3df1-4802-9636-b2ab2263f0ae\" (UID: \"40f81e1f-3df1-4802-9636-b2ab2263f0ae\") " Nov 03 01:03:53.767723 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:03:53.767652 4939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/40f81e1f-3df1-4802-9636-b2ab2263f0ae-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/40f81e1f-3df1-4802-9636-b2ab2263f0ae-istio-podinfo\") pod \"40f81e1f-3df1-4802-9636-b2ab2263f0ae\" (UID: \"40f81e1f-3df1-4802-9636-b2ab2263f0ae\") " Nov 03 01:03:53.767971 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:03:53.767793 4939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/40f81e1f-3df1-4802-9636-b2ab2263f0ae-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/40f81e1f-3df1-4802-9636-b2ab2263f0ae-workload-socket\") pod \"40f81e1f-3df1-4802-9636-b2ab2263f0ae\" (UID: \"40f81e1f-3df1-4802-9636-b2ab2263f0ae\") " Nov 03 01:03:53.767971 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:03:53.767823 4939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/40f81e1f-3df1-4802-9636-b2ab2263f0ae-istio-token\" (UniqueName: \"kubernetes.io/projected/40f81e1f-3df1-4802-9636-b2ab2263f0ae-istio-token\") pod \"40f81e1f-3df1-4802-9636-b2ab2263f0ae\" (UID: \"40f81e1f-3df1-4802-9636-b2ab2263f0ae\") " Nov 03 01:03:53.767971 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:03:53.767920 4939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/40f81e1f-3df1-4802-9636-b2ab2263f0ae-istio-data" pod "40f81e1f-3df1-4802-9636-b2ab2263f0ae" (UID: "40f81e1f-3df1-4802-9636-b2ab2263f0ae"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 01:03:53.768175 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:03:53.768151 4939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/40f81e1f-3df1-4802-9636-b2ab2263f0ae-workload-socket" pod "40f81e1f-3df1-4802-9636-b2ab2263f0ae" (UID: "40f81e1f-3df1-4802-9636-b2ab2263f0ae"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 01:03:53.768238 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:03:53.767882 4939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/40f81e1f-3df1-4802-9636-b2ab2263f0ae-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/40f81e1f-3df1-4802-9636-b2ab2263f0ae-istiod-ca-cert\") pod \"40f81e1f-3df1-4802-9636-b2ab2263f0ae\" (UID: \"40f81e1f-3df1-4802-9636-b2ab2263f0ae\") " Nov 03 01:03:53.768299 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:03:53.768248 4939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/40f81e1f-3df1-4802-9636-b2ab2263f0ae-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/40f81e1f-3df1-4802-9636-b2ab2263f0ae-workload-certs\") pod \"40f81e1f-3df1-4802-9636-b2ab2263f0ae\" (UID: \"40f81e1f-3df1-4802-9636-b2ab2263f0ae\") " Nov 03 01:03:53.768375 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:03:53.768356 4939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/40f81e1f-3df1-4802-9636-b2ab2263f0ae-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/40f81e1f-3df1-4802-9636-b2ab2263f0ae-credential-socket\") pod \"40f81e1f-3df1-4802-9636-b2ab2263f0ae\" (UID: \"40f81e1f-3df1-4802-9636-b2ab2263f0ae\") " Nov 03 01:03:53.768505 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:03:53.768482 4939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/40f81e1f-3df1-4802-9636-b2ab2263f0ae-istiod-ca-cert" pod "40f81e1f-3df1-4802-9636-b2ab2263f0ae" (UID: "40f81e1f-3df1-4802-9636-b2ab2263f0ae"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 01:03:53.769232 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:03:53.769206 4939 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/40f81e1f-3df1-4802-9636-b2ab2263f0ae-istio-data\") on node \"ip-10-0-140-107.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:03:53.769332 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:03:53.769236 4939 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/40f81e1f-3df1-4802-9636-b2ab2263f0ae-workload-socket\") on node \"ip-10-0-140-107.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:03:53.769332 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:03:53.769253 4939 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/40f81e1f-3df1-4802-9636-b2ab2263f0ae-istiod-ca-cert\") on node \"ip-10-0-140-107.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:03:53.770172 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:03:53.770148 4939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/40f81e1f-3df1-4802-9636-b2ab2263f0ae-credential-socket" pod "40f81e1f-3df1-4802-9636-b2ab2263f0ae" (UID: "40f81e1f-3df1-4802-9636-b2ab2263f0ae"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 01:03:53.770502 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:03:53.770481 4939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/40f81e1f-3df1-4802-9636-b2ab2263f0ae-istio-podinfo" pod "40f81e1f-3df1-4802-9636-b2ab2263f0ae" (UID: "40f81e1f-3df1-4802-9636-b2ab2263f0ae"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 01:03:53.770602 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:03:53.770578 4939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/40f81e1f-3df1-4802-9636-b2ab2263f0ae-workload-certs" pod "40f81e1f-3df1-4802-9636-b2ab2263f0ae" (UID: "40f81e1f-3df1-4802-9636-b2ab2263f0ae"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 01:03:53.771159 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:03:53.771118 4939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/40f81e1f-3df1-4802-9636-b2ab2263f0ae-istio-token" pod "40f81e1f-3df1-4802-9636-b2ab2263f0ae" (UID: "40f81e1f-3df1-4802-9636-b2ab2263f0ae"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 01:03:53.772524 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:03:53.772497 4939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/40f81e1f-3df1-4802-9636-b2ab2263f0ae-istio-envoy" pod "40f81e1f-3df1-4802-9636-b2ab2263f0ae" (UID: "40f81e1f-3df1-4802-9636-b2ab2263f0ae"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 01:03:53.773088 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:03:53.773061 4939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/40f81e1f-3df1-4802-9636-b2ab2263f0ae-kube-api-access-2wjcn" pod "40f81e1f-3df1-4802-9636-b2ab2263f0ae" (UID: "40f81e1f-3df1-4802-9636-b2ab2263f0ae"). InnerVolumeSpecName "kube-api-access-2wjcn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 01:03:53.870965 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:03:53.870853 4939 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/40f81e1f-3df1-4802-9636-b2ab2263f0ae-istio-envoy\") on node \"ip-10-0-140-107.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:03:53.870965 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:03:53.870892 4939 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/40f81e1f-3df1-4802-9636-b2ab2263f0ae-istio-podinfo\") on node \"ip-10-0-140-107.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:03:53.870965 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:03:53.870940 4939 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/40f81e1f-3df1-4802-9636-b2ab2263f0ae-istio-token\") on node \"ip-10-0-140-107.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:03:53.870965 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:03:53.870956 4939 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/40f81e1f-3df1-4802-9636-b2ab2263f0ae-workload-certs\") on node \"ip-10-0-140-107.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:03:53.871422 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:03:53.870970 4939 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/40f81e1f-3df1-4802-9636-b2ab2263f0ae-credential-socket\") on node \"ip-10-0-140-107.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:03:53.871422 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:03:53.870984 4939 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2wjcn\" (UniqueName: \"kubernetes.io/projected/40f81e1f-3df1-4802-9636-b2ab2263f0ae-kube-api-access-2wjcn\") on node \"ip-10-0-140-107.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:03:53.980651 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:03:53.980613 4939 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-praxis-a-b9dad7-openshift-default-994bfcbb9-bw5wc"] Nov 03 01:03:53.986032 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:03:53.986001 4939 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-praxis-a-b9dad7-openshift-default-994bfcbb9-bw5wc"] Nov 03 01:03:54.337009 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:03:54.335885 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:03:54.797380 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:03:54.797300 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:03:55.073982 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:03:55.065719 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:03:55.245304 ip-10-0-140-107 kubenswrapper[4939]: E1103 01:03:55.245268 4939 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/system.slice/system-systemd\\\\x2dcoredump.slice/systemd-coredump@17-28211-0.service\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod959b27c0_16c2_41c2_989c_d86f12cedee9.slice/crio-conmon-366bf518257472a8114c5031209c632fe88cd3a6bfc6a7cc0175d0edc46192ab.scope\": RecentStats: unable to find data in memory cache]" Nov 03 01:03:55.385946 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:03:55.384400 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:03:55.667748 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:03:55.667665 4939 generic.go:356] "Generic (PLEG): container finished" podID="959b27c0-16c2-41c2-989c-d86f12cedee9" containerID="366bf518257472a8114c5031209c632fe88cd3a6bfc6a7cc0175d0edc46192ab" exitCode=0 Nov 03 01:03:55.667748 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:03:55.667738 4939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-f14bda-openshift-default-7c65bb6969-rdtlg" event={"ID":"959b27c0-16c2-41c2-989c-d86f12cedee9","Type":"ContainerDied","Data":"366bf518257472a8114c5031209c632fe88cd3a6bfc6a7cc0175d0edc46192ab"} Nov 03 01:03:55.668260 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:03:55.668235 4939 scope.go:122] "RemoveContainer" containerID="366bf518257472a8114c5031209c632fe88cd3a6bfc6a7cc0175d0edc46192ab" Nov 03 01:03:55.669729 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:03:55.669700 4939 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 01:03:55.669804 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:03:55.669790 4939 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 01:03:55.669847 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:03:55.669835 4939 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 01:03:55.870072 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:03:55.870040 4939 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="40f81e1f-3df1-4802-9636-b2ab2263f0ae" path="/var/lib/kubelet/pods/40f81e1f-3df1-4802-9636-b2ab2263f0ae/volumes" Nov 03 01:03:56.059612 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:03:56.059524 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:03:56.185932 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:03:56.185408 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:03:56.677693 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:03:56.677306 4939 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-f14bda-fb4769c77-kxzzg"] Nov 03 01:03:56.677693 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:03:56.677561 4939 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-f14bda-fb4769c77-kxzzg" podUID="c629818b-0176-4218-9e31-18bf89542550" containerName="maas-api" containerID="cri-o://c95ced104d756526a1ff8cbe9b92a27e683eafc9f5f9fd2b49a7f5a99e005ecb" gracePeriod=30 Nov 03 01:03:56.691366 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:03:56.691202 4939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-f14bda-openshift-default-7c65bb6969-rdtlg" event={"ID":"959b27c0-16c2-41c2-989c-d86f12cedee9","Type":"ContainerStarted","Data":"7521a7b1aa8deb7826036d0beab664b06c57ed2240764944788e20f3f405dbc1"} Nov 03 01:03:56.769973 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:03:56.769896 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:03:57.277192 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:03:57.277153 4939 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-f14bda-fb4769c77-kxzzg" Nov 03 01:03:57.366932 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:03:57.345971 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:03:57.416638 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:03:57.416598 4939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c629818b-0176-4218-9e31-18bf89542550-metrics-certs\" (UniqueName: \"kubernetes.io/secret/c629818b-0176-4218-9e31-18bf89542550-metrics-certs\") pod \"c629818b-0176-4218-9e31-18bf89542550\" (UID: \"c629818b-0176-4218-9e31-18bf89542550\") " Nov 03 01:03:57.416850 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:03:57.416663 4939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c629818b-0176-4218-9e31-18bf89542550-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c629818b-0176-4218-9e31-18bf89542550-openshift-service-ca\") pod \"c629818b-0176-4218-9e31-18bf89542550\" (UID: \"c629818b-0176-4218-9e31-18bf89542550\") " Nov 03 01:03:57.416850 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:03:57.416700 4939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c629818b-0176-4218-9e31-18bf89542550-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c629818b-0176-4218-9e31-18bf89542550-maas-api-tls\") pod \"c629818b-0176-4218-9e31-18bf89542550\" (UID: \"c629818b-0176-4218-9e31-18bf89542550\") " Nov 03 01:03:57.416850 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:03:57.416836 4939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c629818b-0176-4218-9e31-18bf89542550-kube-api-access-qmnzc\" (UniqueName: \"kubernetes.io/projected/c629818b-0176-4218-9e31-18bf89542550-kube-api-access-qmnzc\") pod \"c629818b-0176-4218-9e31-18bf89542550\" (UID: \"c629818b-0176-4218-9e31-18bf89542550\") " Nov 03 01:03:57.417120 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:03:57.417094 4939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c629818b-0176-4218-9e31-18bf89542550-openshift-service-ca" pod "c629818b-0176-4218-9e31-18bf89542550" (UID: "c629818b-0176-4218-9e31-18bf89542550"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 01:03:57.417437 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:03:57.417412 4939 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c629818b-0176-4218-9e31-18bf89542550-openshift-service-ca\") on node \"ip-10-0-140-107.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:03:57.418954 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:03:57.418899 4939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c629818b-0176-4218-9e31-18bf89542550-metrics-certs" pod "c629818b-0176-4218-9e31-18bf89542550" (UID: "c629818b-0176-4218-9e31-18bf89542550"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 01:03:57.419048 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:03:57.418951 4939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c629818b-0176-4218-9e31-18bf89542550-maas-api-tls" pod "c629818b-0176-4218-9e31-18bf89542550" (UID: "c629818b-0176-4218-9e31-18bf89542550"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 01:03:57.419090 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:03:57.419076 4939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c629818b-0176-4218-9e31-18bf89542550-kube-api-access-qmnzc" pod "c629818b-0176-4218-9e31-18bf89542550" (UID: "c629818b-0176-4218-9e31-18bf89542550"). InnerVolumeSpecName "kube-api-access-qmnzc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 01:03:57.518757 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:03:57.518662 4939 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c629818b-0176-4218-9e31-18bf89542550-metrics-certs\") on node \"ip-10-0-140-107.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:03:57.525324 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:03:57.518998 4939 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c629818b-0176-4218-9e31-18bf89542550-maas-api-tls\") on node \"ip-10-0-140-107.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:03:57.525324 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:03:57.525262 4939 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qmnzc\" (UniqueName: \"kubernetes.io/projected/c629818b-0176-4218-9e31-18bf89542550-kube-api-access-qmnzc\") on node \"ip-10-0-140-107.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:03:57.554885 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:03:57.554210 4939 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-praxis-b-f14bda-openshift-default-7c65bb6969-rdtlg" Nov 03 01:03:57.554885 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:03:57.554542 4939 patch_prober.go:29] interesting pod/e2e-praxis-b-f14bda-openshift-default-7c65bb6969-rdtlg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.158:15021/healthz/ready\": dial tcp 10.132.0.158:15021: connect: connection refused" start-of-body= Nov 03 01:03:57.554885 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:03:57.554591 4939 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-f14bda-openshift-default-7c65bb6969-rdtlg" podUID="959b27c0-16c2-41c2-989c-d86f12cedee9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.158:15021/healthz/ready\": dial tcp 10.132.0.158:15021: connect: connection refused" Nov 03 01:03:57.700335 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:03:57.700298 4939 generic.go:356] "Generic (PLEG): container finished" podID="c629818b-0176-4218-9e31-18bf89542550" containerID="c95ced104d756526a1ff8cbe9b92a27e683eafc9f5f9fd2b49a7f5a99e005ecb" exitCode=0 Nov 03 01:03:57.700530 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:03:57.700345 4939 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-f14bda-fb4769c77-kxzzg" Nov 03 01:03:57.700530 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:03:57.700374 4939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-f14bda-fb4769c77-kxzzg" event={"ID":"c629818b-0176-4218-9e31-18bf89542550","Type":"ContainerDied","Data":"c95ced104d756526a1ff8cbe9b92a27e683eafc9f5f9fd2b49a7f5a99e005ecb"} Nov 03 01:03:57.700530 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:03:57.700429 4939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-f14bda-fb4769c77-kxzzg" event={"ID":"c629818b-0176-4218-9e31-18bf89542550","Type":"ContainerDied","Data":"6e5b4f2810046285a53306e8824331294b984847d9b6d6ab7a3cfa0d2e7d1814"} Nov 03 01:03:57.700530 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:03:57.700461 4939 scope.go:122] "RemoveContainer" containerID="c95ced104d756526a1ff8cbe9b92a27e683eafc9f5f9fd2b49a7f5a99e005ecb" Nov 03 01:03:57.712272 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:03:57.712250 4939 scope.go:122] "RemoveContainer" containerID="c95ced104d756526a1ff8cbe9b92a27e683eafc9f5f9fd2b49a7f5a99e005ecb" Nov 03 01:03:57.712634 ip-10-0-140-107 kubenswrapper[4939]: E1103 01:03:57.712582 4939 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c95ced104d756526a1ff8cbe9b92a27e683eafc9f5f9fd2b49a7f5a99e005ecb\": container with ID starting with c95ced104d756526a1ff8cbe9b92a27e683eafc9f5f9fd2b49a7f5a99e005ecb not found: ID does not exist" containerID="c95ced104d756526a1ff8cbe9b92a27e683eafc9f5f9fd2b49a7f5a99e005ecb" Nov 03 01:03:57.712729 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:03:57.712642 4939 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c95ced104d756526a1ff8cbe9b92a27e683eafc9f5f9fd2b49a7f5a99e005ecb"} err="failed to get container status \"c95ced104d756526a1ff8cbe9b92a27e683eafc9f5f9fd2b49a7f5a99e005ecb\": rpc error: code = NotFound desc = could not find container \"c95ced104d756526a1ff8cbe9b92a27e683eafc9f5f9fd2b49a7f5a99e005ecb\": container with ID starting with c95ced104d756526a1ff8cbe9b92a27e683eafc9f5f9fd2b49a7f5a99e005ecb not found: ID does not exist" Nov 03 01:03:57.726777 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:03:57.726740 4939 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-f14bda-fb4769c77-kxzzg"] Nov 03 01:03:57.732236 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:03:57.732207 4939 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-f14bda-fb4769c77-kxzzg"] Nov 03 01:03:57.883721 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:03:57.883689 4939 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c629818b-0176-4218-9e31-18bf89542550" path="/var/lib/kubelet/pods/c629818b-0176-4218-9e31-18bf89542550/volumes" Nov 03 01:03:58.204390 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:03:58.204308 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:03:58.572420 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:03:58.572318 4939 patch_prober.go:29] interesting pod/e2e-praxis-b-f14bda-openshift-default-7c65bb6969-rdtlg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.158:15021/healthz/ready\": dial tcp 10.132.0.158:15021: connect: connection refused" start-of-body= Nov 03 01:03:58.572420 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:03:58.572381 4939 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-f14bda-openshift-default-7c65bb6969-rdtlg" podUID="959b27c0-16c2-41c2-989c-d86f12cedee9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.158:15021/healthz/ready\": dial tcp 10.132.0.158:15021: connect: connection refused" Nov 03 01:03:59.082544 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:03:59.082517 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:03:59.570354 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:03:59.570258 4939 patch_prober.go:29] interesting pod/e2e-praxis-b-f14bda-openshift-default-7c65bb6969-rdtlg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.158:15021/healthz/ready\": dial tcp 10.132.0.158:15021: connect: connection refused" start-of-body= Nov 03 01:03:59.570654 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:03:59.570624 4939 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-f14bda-openshift-default-7c65bb6969-rdtlg" podUID="959b27c0-16c2-41c2-989c-d86f12cedee9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.158:15021/healthz/ready\": dial tcp 10.132.0.158:15021: connect: connection refused" Nov 03 01:03:59.799362 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:03:59.799326 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:04:00.323554 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:04:00.323519 4939 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-pcw82"] Nov 03 01:04:00.323795 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:04:00.323770 4939 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-pcw82" podUID="48186e38-73f5-47c6-a90a-d82afeac4a52" containerName="istio-proxy" containerID="cri-o://2caa716486ef738a8bc9e623ec10c415d19ed7bfc79d016ab466e6c2da482fde" gracePeriod=30 Nov 03 01:04:00.569934 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:04:00.554320 4939 patch_prober.go:29] interesting pod/e2e-praxis-b-f14bda-openshift-default-7c65bb6969-rdtlg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.158:15021/healthz/ready\": dial tcp 10.132.0.158:15021: connect: connection refused" start-of-body= Nov 03 01:04:00.570290 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:04:00.570250 4939 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-f14bda-openshift-default-7c65bb6969-rdtlg" podUID="959b27c0-16c2-41c2-989c-d86f12cedee9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.158:15021/healthz/ready\": dial tcp 10.132.0.158:15021: connect: connection refused" Nov 03 01:04:00.703696 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:04:00.703663 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:04:01.402503 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:04:01.402464 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:04:01.568936 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:04:01.568301 4939 patch_prober.go:29] interesting pod/e2e-praxis-b-f14bda-openshift-default-7c65bb6969-rdtlg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.158:15021/healthz/ready\": dial tcp 10.132.0.158:15021: connect: connection refused" start-of-body= Nov 03 01:04:01.568936 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:04:01.568366 4939 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-f14bda-openshift-default-7c65bb6969-rdtlg" podUID="959b27c0-16c2-41c2-989c-d86f12cedee9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.158:15021/healthz/ready\": dial tcp 10.132.0.158:15021: connect: connection refused" Nov 03 01:04:02.182378 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:04:02.182344 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:04:02.300632 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:04:02.300594 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:04:02.556572 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:04:02.556487 4939 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-praxis-b-f14bda-openshift-default-7c65bb6969-rdtlg" Nov 03 01:04:02.557090 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:04:02.556786 4939 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-praxis-b-f14bda-openshift-default-7c65bb6969-rdtlg" Nov 03 01:04:02.557583 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:04:02.557567 4939 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-praxis-b-f14bda-openshift-default-7c65bb6969-rdtlg" Nov 03 01:04:02.624559 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:04:02.624515 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:04:02.907184 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:04:02.907149 4939 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-praxis-b-f14bda-openshift-default-7c65bb6969-rdtlg"] Nov 03 01:04:02.960830 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:04:02.960752 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:04:03.338170 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:04:03.338127 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:04:03.795659 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:04:03.795584 4939 kubelet_pods.go:1082] "Unable to retrieve pull secret, the image pull may not succeed." pod="openshift-ingress/e2e-praxis-b-f14bda-openshift-default-7c65bb6969-rdtlg" secret="" err="secret \"e2e-praxis-b-f14bda-openshift-default-dockercfg-dv7sl\" not found" Nov 03 01:04:04.345446 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:04:04.345403 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:04:04.801556 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:04:04.801463 4939 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-praxis-b-f14bda-openshift-default-7c65bb6969-rdtlg" podUID="959b27c0-16c2-41c2-989c-d86f12cedee9" containerName="istio-proxy" containerID="cri-o://7521a7b1aa8deb7826036d0beab664b06c57ed2240764944788e20f3f405dbc1" gracePeriod=30 Nov 03 01:04:05.001209 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:04:05.001176 4939 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-4pw7v"] Nov 03 01:04:05.363009 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:04:05.356597 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:04:05.805683 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:04:05.805657 4939 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-pcw82" Nov 03 01:04:05.810488 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:04:05.810455 4939 generic.go:356] "Generic (PLEG): container finished" podID="48186e38-73f5-47c6-a90a-d82afeac4a52" containerID="2caa716486ef738a8bc9e623ec10c415d19ed7bfc79d016ab466e6c2da482fde" exitCode=0 Nov 03 01:04:05.810643 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:04:05.810511 4939 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-pcw82" Nov 03 01:04:05.810643 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:04:05.810537 4939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-pcw82" event={"ID":"48186e38-73f5-47c6-a90a-d82afeac4a52","Type":"ContainerDied","Data":"2caa716486ef738a8bc9e623ec10c415d19ed7bfc79d016ab466e6c2da482fde"} Nov 03 01:04:05.810643 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:04:05.810591 4939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-pcw82" event={"ID":"48186e38-73f5-47c6-a90a-d82afeac4a52","Type":"ContainerDied","Data":"edc25129fc4e843a9b9190be34b30951598bb92acf83d50c2dfacb27b37d2292"} Nov 03 01:04:05.810643 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:04:05.810615 4939 scope.go:122] "RemoveContainer" containerID="2caa716486ef738a8bc9e623ec10c415d19ed7bfc79d016ab466e6c2da482fde" Nov 03 01:04:05.820444 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:04:05.820418 4939 scope.go:122] "RemoveContainer" containerID="2caa716486ef738a8bc9e623ec10c415d19ed7bfc79d016ab466e6c2da482fde" Nov 03 01:04:05.820720 ip-10-0-140-107 kubenswrapper[4939]: E1103 01:04:05.820701 4939 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2caa716486ef738a8bc9e623ec10c415d19ed7bfc79d016ab466e6c2da482fde\": container with ID starting with 2caa716486ef738a8bc9e623ec10c415d19ed7bfc79d016ab466e6c2da482fde not found: ID does not exist" containerID="2caa716486ef738a8bc9e623ec10c415d19ed7bfc79d016ab466e6c2da482fde" Nov 03 01:04:05.820793 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:04:05.820727 4939 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2caa716486ef738a8bc9e623ec10c415d19ed7bfc79d016ab466e6c2da482fde"} err="failed to get container status \"2caa716486ef738a8bc9e623ec10c415d19ed7bfc79d016ab466e6c2da482fde\": rpc error: code = NotFound desc = could not find container \"2caa716486ef738a8bc9e623ec10c415d19ed7bfc79d016ab466e6c2da482fde\": container with ID starting with 2caa716486ef738a8bc9e623ec10c415d19ed7bfc79d016ab466e6c2da482fde not found: ID does not exist" Nov 03 01:04:05.875012 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:04:05.874967 4939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/48186e38-73f5-47c6-a90a-d82afeac4a52-kube-api-access-lwv87\" (UniqueName: \"kubernetes.io/projected/48186e38-73f5-47c6-a90a-d82afeac4a52-kube-api-access-lwv87\") pod \"48186e38-73f5-47c6-a90a-d82afeac4a52\" (UID: \"48186e38-73f5-47c6-a90a-d82afeac4a52\") " Nov 03 01:04:05.875262 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:04:05.875072 4939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/48186e38-73f5-47c6-a90a-d82afeac4a52-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/48186e38-73f5-47c6-a90a-d82afeac4a52-istio-data\") pod \"48186e38-73f5-47c6-a90a-d82afeac4a52\" (UID: \"48186e38-73f5-47c6-a90a-d82afeac4a52\") " Nov 03 01:04:05.875262 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:04:05.875123 4939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/48186e38-73f5-47c6-a90a-d82afeac4a52-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/48186e38-73f5-47c6-a90a-d82afeac4a52-credential-socket\") pod \"48186e38-73f5-47c6-a90a-d82afeac4a52\" (UID: \"48186e38-73f5-47c6-a90a-d82afeac4a52\") " Nov 03 01:04:05.875262 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:04:05.875183 4939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/48186e38-73f5-47c6-a90a-d82afeac4a52-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/48186e38-73f5-47c6-a90a-d82afeac4a52-istiod-ca-cert\") pod \"48186e38-73f5-47c6-a90a-d82afeac4a52\" (UID: \"48186e38-73f5-47c6-a90a-d82afeac4a52\") " Nov 03 01:04:05.875556 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:04:05.875275 4939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/48186e38-73f5-47c6-a90a-d82afeac4a52-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/48186e38-73f5-47c6-a90a-d82afeac4a52-istio-envoy\") pod \"48186e38-73f5-47c6-a90a-d82afeac4a52\" (UID: \"48186e38-73f5-47c6-a90a-d82afeac4a52\") " Nov 03 01:04:05.875556 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:04:05.875312 4939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/48186e38-73f5-47c6-a90a-d82afeac4a52-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/48186e38-73f5-47c6-a90a-d82afeac4a52-istio-podinfo\") pod \"48186e38-73f5-47c6-a90a-d82afeac4a52\" (UID: \"48186e38-73f5-47c6-a90a-d82afeac4a52\") " Nov 03 01:04:05.875556 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:04:05.875375 4939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/48186e38-73f5-47c6-a90a-d82afeac4a52-istio-token\" (UniqueName: \"kubernetes.io/projected/48186e38-73f5-47c6-a90a-d82afeac4a52-istio-token\") pod \"48186e38-73f5-47c6-a90a-d82afeac4a52\" (UID: \"48186e38-73f5-47c6-a90a-d82afeac4a52\") " Nov 03 01:04:05.875556 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:04:05.875454 4939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/48186e38-73f5-47c6-a90a-d82afeac4a52-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/48186e38-73f5-47c6-a90a-d82afeac4a52-workload-certs\") pod \"48186e38-73f5-47c6-a90a-d82afeac4a52\" (UID: \"48186e38-73f5-47c6-a90a-d82afeac4a52\") " Nov 03 01:04:05.875863 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:04:05.875568 4939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/48186e38-73f5-47c6-a90a-d82afeac4a52-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/48186e38-73f5-47c6-a90a-d82afeac4a52-workload-socket\") pod \"48186e38-73f5-47c6-a90a-d82afeac4a52\" (UID: \"48186e38-73f5-47c6-a90a-d82afeac4a52\") " Nov 03 01:04:05.876163 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:04:05.876127 4939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/48186e38-73f5-47c6-a90a-d82afeac4a52-istio-data" pod "48186e38-73f5-47c6-a90a-d82afeac4a52" (UID: "48186e38-73f5-47c6-a90a-d82afeac4a52"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 01:04:05.876306 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:04:05.876276 4939 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/48186e38-73f5-47c6-a90a-d82afeac4a52-istio-data\") on node \"ip-10-0-140-107.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:04:05.876626 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:04:05.876599 4939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/48186e38-73f5-47c6-a90a-d82afeac4a52-credential-socket" pod "48186e38-73f5-47c6-a90a-d82afeac4a52" (UID: "48186e38-73f5-47c6-a90a-d82afeac4a52"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 01:04:05.876999 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:04:05.876966 4939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/48186e38-73f5-47c6-a90a-d82afeac4a52-istiod-ca-cert" pod "48186e38-73f5-47c6-a90a-d82afeac4a52" (UID: "48186e38-73f5-47c6-a90a-d82afeac4a52"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 01:04:05.877215 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:04:05.877190 4939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/48186e38-73f5-47c6-a90a-d82afeac4a52-workload-socket" pod "48186e38-73f5-47c6-a90a-d82afeac4a52" (UID: "48186e38-73f5-47c6-a90a-d82afeac4a52"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 01:04:05.891954 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:04:05.878500 4939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/48186e38-73f5-47c6-a90a-d82afeac4a52-workload-certs" pod "48186e38-73f5-47c6-a90a-d82afeac4a52" (UID: "48186e38-73f5-47c6-a90a-d82afeac4a52"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 01:04:05.891954 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:04:05.890366 4939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/48186e38-73f5-47c6-a90a-d82afeac4a52-istio-envoy" pod "48186e38-73f5-47c6-a90a-d82afeac4a52" (UID: "48186e38-73f5-47c6-a90a-d82afeac4a52"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 01:04:05.891954 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:04:05.890440 4939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/48186e38-73f5-47c6-a90a-d82afeac4a52-istio-podinfo" pod "48186e38-73f5-47c6-a90a-d82afeac4a52" (UID: "48186e38-73f5-47c6-a90a-d82afeac4a52"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 01:04:05.891954 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:04:05.890508 4939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/48186e38-73f5-47c6-a90a-d82afeac4a52-istio-token" pod "48186e38-73f5-47c6-a90a-d82afeac4a52" (UID: "48186e38-73f5-47c6-a90a-d82afeac4a52"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 01:04:05.891954 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:04:05.891745 4939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/48186e38-73f5-47c6-a90a-d82afeac4a52-kube-api-access-lwv87" pod "48186e38-73f5-47c6-a90a-d82afeac4a52" (UID: "48186e38-73f5-47c6-a90a-d82afeac4a52"). InnerVolumeSpecName "kube-api-access-lwv87". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 01:04:05.977901 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:04:05.977857 4939 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/48186e38-73f5-47c6-a90a-d82afeac4a52-workload-socket\") on node \"ip-10-0-140-107.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:04:05.977901 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:04:05.977895 4939 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lwv87\" (UniqueName: \"kubernetes.io/projected/48186e38-73f5-47c6-a90a-d82afeac4a52-kube-api-access-lwv87\") on node \"ip-10-0-140-107.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:04:05.977901 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:04:05.977925 4939 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/48186e38-73f5-47c6-a90a-d82afeac4a52-credential-socket\") on node \"ip-10-0-140-107.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:04:05.977901 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:04:05.977940 4939 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/48186e38-73f5-47c6-a90a-d82afeac4a52-istiod-ca-cert\") on node \"ip-10-0-140-107.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:04:05.978245 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:04:05.977953 4939 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/48186e38-73f5-47c6-a90a-d82afeac4a52-istio-envoy\") on node \"ip-10-0-140-107.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:04:05.978245 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:04:05.977966 4939 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/48186e38-73f5-47c6-a90a-d82afeac4a52-istio-podinfo\") on node \"ip-10-0-140-107.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:04:05.978245 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:04:05.977978 4939 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/48186e38-73f5-47c6-a90a-d82afeac4a52-istio-token\") on node \"ip-10-0-140-107.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:04:05.978245 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:04:05.977991 4939 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/48186e38-73f5-47c6-a90a-d82afeac4a52-workload-certs\") on node \"ip-10-0-140-107.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:04:06.141779 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:04:06.141742 4939 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-pcw82"] Nov 03 01:04:06.147279 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:04:06.147249 4939 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-pcw82"] Nov 03 01:04:06.159846 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:04:06.159820 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:04:06.906758 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:04:06.906725 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:04:07.421810 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:04:07.421776 4939 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-1e352e-585bf55c69-ddbn9"] Nov 03 01:04:07.438626 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:04:07.438597 4939 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-1e352e-6bb6bb4d7f-hd98j"] Nov 03 01:04:07.524412 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:04:07.524371 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:04:07.632044 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:04:07.632013 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:04:07.893685 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:04:07.889508 4939 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="48186e38-73f5-47c6-a90a-d82afeac4a52" path="/var/lib/kubelet/pods/48186e38-73f5-47c6-a90a-d82afeac4a52/volumes" Nov 03 01:04:07.955928 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:04:07.955132 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:04:08.112067 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:04:08.111832 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:04:08.646602 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:04:08.646564 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:04:09.044767 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:04:09.044682 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:04:09.657248 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:04:09.657216 4939 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-88bbf7-openshift-default-574f47fff7-dthhl"] Nov 03 01:04:09.657948 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:04:09.657925 4939 cpu_manager.go:414] "containerMap: removing container" podUID="40f81e1f-3df1-4802-9636-b2ab2263f0ae" containerName="istio-proxy" Nov 03 01:04:09.657948 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:04:09.657947 4939 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="40f81e1f-3df1-4802-9636-b2ab2263f0ae" containerName="istio-proxy" Nov 03 01:04:09.658132 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:04:09.657968 4939 cpu_manager.go:414] "containerMap: removing container" podUID="c629818b-0176-4218-9e31-18bf89542550" containerName="maas-api" Nov 03 01:04:09.658132 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:04:09.657973 4939 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c629818b-0176-4218-9e31-18bf89542550" containerName="maas-api" Nov 03 01:04:09.658132 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:04:09.657998 4939 cpu_manager.go:414] "containerMap: removing container" podUID="48186e38-73f5-47c6-a90a-d82afeac4a52" containerName="istio-proxy" Nov 03 01:04:09.658132 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:04:09.658003 4939 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="48186e38-73f5-47c6-a90a-d82afeac4a52" containerName="istio-proxy" Nov 03 01:04:09.658132 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:04:09.658026 4939 cpu_manager.go:414] "containerMap: removing container" podUID="40f81e1f-3df1-4802-9636-b2ab2263f0ae" containerName="istio-proxy" Nov 03 01:04:09.658132 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:04:09.658031 4939 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="40f81e1f-3df1-4802-9636-b2ab2263f0ae" containerName="istio-proxy" Nov 03 01:04:09.658132 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:04:09.658055 4939 cpu_manager.go:414] "containerMap: removing container" podUID="b645db90-ca61-4e9c-b2d7-1e57ebd22e64" containerName="main" Nov 03 01:04:09.658132 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:04:09.658059 4939 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b645db90-ca61-4e9c-b2d7-1e57ebd22e64" containerName="main" Nov 03 01:04:09.658542 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:04:09.658154 4939 memory_manager.go:365] "RemoveStaleState removing state" podUID="c629818b-0176-4218-9e31-18bf89542550" containerName="maas-api" Nov 03 01:04:09.658542 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:04:09.658169 4939 memory_manager.go:365] "RemoveStaleState removing state" podUID="48186e38-73f5-47c6-a90a-d82afeac4a52" containerName="istio-proxy" Nov 03 01:04:09.658542 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:04:09.658178 4939 memory_manager.go:365] "RemoveStaleState removing state" podUID="40f81e1f-3df1-4802-9636-b2ab2263f0ae" containerName="istio-proxy" Nov 03 01:04:09.658542 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:04:09.658187 4939 memory_manager.go:365] "RemoveStaleState removing state" podUID="b645db90-ca61-4e9c-b2d7-1e57ebd22e64" containerName="main" Nov 03 01:04:09.658542 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:04:09.658192 4939 memory_manager.go:365] "RemoveStaleState removing state" podUID="40f81e1f-3df1-4802-9636-b2ab2263f0ae" containerName="istio-proxy" Nov 03 01:04:09.663345 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:04:09.663326 4939 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-88bbf7-openshift-default-574f47fff7-dthhl" Nov 03 01:04:09.678038 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:04:09.677933 4939 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-88bbf7-openshift-default-574f47fff7-dthhl"] Nov 03 01:04:09.791185 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:04:09.791147 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a1e98e45-9f37-42eb-9ac6-4624f63fe503-istio-data\") pod \"e2e-worker-w1-88bbf7-openshift-default-574f47fff7-dthhl\" (UID: \"a1e98e45-9f37-42eb-9ac6-4624f63fe503\") " pod="openshift-ingress/e2e-worker-w1-88bbf7-openshift-default-574f47fff7-dthhl" Nov 03 01:04:09.791393 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:04:09.791202 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a1e98e45-9f37-42eb-9ac6-4624f63fe503-workload-socket\") pod \"e2e-worker-w1-88bbf7-openshift-default-574f47fff7-dthhl\" (UID: \"a1e98e45-9f37-42eb-9ac6-4624f63fe503\") " pod="openshift-ingress/e2e-worker-w1-88bbf7-openshift-default-574f47fff7-dthhl" Nov 03 01:04:09.791393 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:04:09.791320 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a1e98e45-9f37-42eb-9ac6-4624f63fe503-istio-envoy\") pod \"e2e-worker-w1-88bbf7-openshift-default-574f47fff7-dthhl\" (UID: \"a1e98e45-9f37-42eb-9ac6-4624f63fe503\") " pod="openshift-ingress/e2e-worker-w1-88bbf7-openshift-default-574f47fff7-dthhl" Nov 03 01:04:09.791544 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:04:09.791416 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a1e98e45-9f37-42eb-9ac6-4624f63fe503-istiod-ca-cert\") pod \"e2e-worker-w1-88bbf7-openshift-default-574f47fff7-dthhl\" (UID: \"a1e98e45-9f37-42eb-9ac6-4624f63fe503\") " pod="openshift-ingress/e2e-worker-w1-88bbf7-openshift-default-574f47fff7-dthhl" Nov 03 01:04:09.791544 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:04:09.791491 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-54lt4\" (UniqueName: \"kubernetes.io/projected/a1e98e45-9f37-42eb-9ac6-4624f63fe503-kube-api-access-54lt4\") pod \"e2e-worker-w1-88bbf7-openshift-default-574f47fff7-dthhl\" (UID: \"a1e98e45-9f37-42eb-9ac6-4624f63fe503\") " pod="openshift-ingress/e2e-worker-w1-88bbf7-openshift-default-574f47fff7-dthhl" Nov 03 01:04:09.791654 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:04:09.791579 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a1e98e45-9f37-42eb-9ac6-4624f63fe503-credential-socket\") pod \"e2e-worker-w1-88bbf7-openshift-default-574f47fff7-dthhl\" (UID: \"a1e98e45-9f37-42eb-9ac6-4624f63fe503\") " pod="openshift-ingress/e2e-worker-w1-88bbf7-openshift-default-574f47fff7-dthhl" Nov 03 01:04:09.791739 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:04:09.791715 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a1e98e45-9f37-42eb-9ac6-4624f63fe503-istio-token\") pod \"e2e-worker-w1-88bbf7-openshift-default-574f47fff7-dthhl\" (UID: \"a1e98e45-9f37-42eb-9ac6-4624f63fe503\") " pod="openshift-ingress/e2e-worker-w1-88bbf7-openshift-default-574f47fff7-dthhl" Nov 03 01:04:09.791792 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:04:09.791778 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a1e98e45-9f37-42eb-9ac6-4624f63fe503-istio-podinfo\") pod \"e2e-worker-w1-88bbf7-openshift-default-574f47fff7-dthhl\" (UID: \"a1e98e45-9f37-42eb-9ac6-4624f63fe503\") " pod="openshift-ingress/e2e-worker-w1-88bbf7-openshift-default-574f47fff7-dthhl" Nov 03 01:04:09.791833 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:04:09.791804 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a1e98e45-9f37-42eb-9ac6-4624f63fe503-workload-certs\") pod \"e2e-worker-w1-88bbf7-openshift-default-574f47fff7-dthhl\" (UID: \"a1e98e45-9f37-42eb-9ac6-4624f63fe503\") " pod="openshift-ingress/e2e-worker-w1-88bbf7-openshift-default-574f47fff7-dthhl" Nov 03 01:04:09.893069 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:04:09.893035 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a1e98e45-9f37-42eb-9ac6-4624f63fe503-workload-socket\") pod \"e2e-worker-w1-88bbf7-openshift-default-574f47fff7-dthhl\" (UID: \"a1e98e45-9f37-42eb-9ac6-4624f63fe503\") " pod="openshift-ingress/e2e-worker-w1-88bbf7-openshift-default-574f47fff7-dthhl" Nov 03 01:04:09.893240 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:04:09.893084 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a1e98e45-9f37-42eb-9ac6-4624f63fe503-istio-envoy\") pod \"e2e-worker-w1-88bbf7-openshift-default-574f47fff7-dthhl\" (UID: \"a1e98e45-9f37-42eb-9ac6-4624f63fe503\") " pod="openshift-ingress/e2e-worker-w1-88bbf7-openshift-default-574f47fff7-dthhl" Nov 03 01:04:09.893240 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:04:09.893131 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a1e98e45-9f37-42eb-9ac6-4624f63fe503-istiod-ca-cert\") pod \"e2e-worker-w1-88bbf7-openshift-default-574f47fff7-dthhl\" (UID: \"a1e98e45-9f37-42eb-9ac6-4624f63fe503\") " pod="openshift-ingress/e2e-worker-w1-88bbf7-openshift-default-574f47fff7-dthhl" Nov 03 01:04:09.893240 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:04:09.893173 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-54lt4\" (UniqueName: \"kubernetes.io/projected/a1e98e45-9f37-42eb-9ac6-4624f63fe503-kube-api-access-54lt4\") pod \"e2e-worker-w1-88bbf7-openshift-default-574f47fff7-dthhl\" (UID: \"a1e98e45-9f37-42eb-9ac6-4624f63fe503\") " pod="openshift-ingress/e2e-worker-w1-88bbf7-openshift-default-574f47fff7-dthhl" Nov 03 01:04:09.893240 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:04:09.893220 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a1e98e45-9f37-42eb-9ac6-4624f63fe503-credential-socket\") pod \"e2e-worker-w1-88bbf7-openshift-default-574f47fff7-dthhl\" (UID: \"a1e98e45-9f37-42eb-9ac6-4624f63fe503\") " pod="openshift-ingress/e2e-worker-w1-88bbf7-openshift-default-574f47fff7-dthhl" Nov 03 01:04:09.893476 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:04:09.893338 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a1e98e45-9f37-42eb-9ac6-4624f63fe503-istio-token\") pod \"e2e-worker-w1-88bbf7-openshift-default-574f47fff7-dthhl\" (UID: \"a1e98e45-9f37-42eb-9ac6-4624f63fe503\") " pod="openshift-ingress/e2e-worker-w1-88bbf7-openshift-default-574f47fff7-dthhl" Nov 03 01:04:09.893476 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:04:09.893363 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a1e98e45-9f37-42eb-9ac6-4624f63fe503-istio-podinfo\") pod \"e2e-worker-w1-88bbf7-openshift-default-574f47fff7-dthhl\" (UID: \"a1e98e45-9f37-42eb-9ac6-4624f63fe503\") " pod="openshift-ingress/e2e-worker-w1-88bbf7-openshift-default-574f47fff7-dthhl" Nov 03 01:04:09.893476 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:04:09.893394 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a1e98e45-9f37-42eb-9ac6-4624f63fe503-workload-certs\") pod \"e2e-worker-w1-88bbf7-openshift-default-574f47fff7-dthhl\" (UID: \"a1e98e45-9f37-42eb-9ac6-4624f63fe503\") " pod="openshift-ingress/e2e-worker-w1-88bbf7-openshift-default-574f47fff7-dthhl" Nov 03 01:04:09.893647 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:04:09.893478 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a1e98e45-9f37-42eb-9ac6-4624f63fe503-istio-data\") pod \"e2e-worker-w1-88bbf7-openshift-default-574f47fff7-dthhl\" (UID: \"a1e98e45-9f37-42eb-9ac6-4624f63fe503\") " pod="openshift-ingress/e2e-worker-w1-88bbf7-openshift-default-574f47fff7-dthhl" Nov 03 01:04:09.893647 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:04:09.893476 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a1e98e45-9f37-42eb-9ac6-4624f63fe503-workload-socket\") pod \"e2e-worker-w1-88bbf7-openshift-default-574f47fff7-dthhl\" (UID: \"a1e98e45-9f37-42eb-9ac6-4624f63fe503\") " pod="openshift-ingress/e2e-worker-w1-88bbf7-openshift-default-574f47fff7-dthhl" Nov 03 01:04:09.893647 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:04:09.893636 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a1e98e45-9f37-42eb-9ac6-4624f63fe503-credential-socket\") pod \"e2e-worker-w1-88bbf7-openshift-default-574f47fff7-dthhl\" (UID: \"a1e98e45-9f37-42eb-9ac6-4624f63fe503\") " pod="openshift-ingress/e2e-worker-w1-88bbf7-openshift-default-574f47fff7-dthhl" Nov 03 01:04:09.893894 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:04:09.893872 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a1e98e45-9f37-42eb-9ac6-4624f63fe503-workload-certs\") pod \"e2e-worker-w1-88bbf7-openshift-default-574f47fff7-dthhl\" (UID: \"a1e98e45-9f37-42eb-9ac6-4624f63fe503\") " pod="openshift-ingress/e2e-worker-w1-88bbf7-openshift-default-574f47fff7-dthhl" Nov 03 01:04:09.894004 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:04:09.893900 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a1e98e45-9f37-42eb-9ac6-4624f63fe503-istio-data\") pod \"e2e-worker-w1-88bbf7-openshift-default-574f47fff7-dthhl\" (UID: \"a1e98e45-9f37-42eb-9ac6-4624f63fe503\") " pod="openshift-ingress/e2e-worker-w1-88bbf7-openshift-default-574f47fff7-dthhl" Nov 03 01:04:09.894072 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:04:09.894028 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a1e98e45-9f37-42eb-9ac6-4624f63fe503-istiod-ca-cert\") pod \"e2e-worker-w1-88bbf7-openshift-default-574f47fff7-dthhl\" (UID: \"a1e98e45-9f37-42eb-9ac6-4624f63fe503\") " pod="openshift-ingress/e2e-worker-w1-88bbf7-openshift-default-574f47fff7-dthhl" Nov 03 01:04:09.896587 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:04:09.895793 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a1e98e45-9f37-42eb-9ac6-4624f63fe503-istio-envoy\") pod \"e2e-worker-w1-88bbf7-openshift-default-574f47fff7-dthhl\" (UID: \"a1e98e45-9f37-42eb-9ac6-4624f63fe503\") " pod="openshift-ingress/e2e-worker-w1-88bbf7-openshift-default-574f47fff7-dthhl" Nov 03 01:04:09.896587 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:04:09.896254 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a1e98e45-9f37-42eb-9ac6-4624f63fe503-istio-podinfo\") pod \"e2e-worker-w1-88bbf7-openshift-default-574f47fff7-dthhl\" (UID: \"a1e98e45-9f37-42eb-9ac6-4624f63fe503\") " pod="openshift-ingress/e2e-worker-w1-88bbf7-openshift-default-574f47fff7-dthhl" Nov 03 01:04:09.901277 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:04:09.901251 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a1e98e45-9f37-42eb-9ac6-4624f63fe503-istio-token\") pod \"e2e-worker-w1-88bbf7-openshift-default-574f47fff7-dthhl\" (UID: \"a1e98e45-9f37-42eb-9ac6-4624f63fe503\") " pod="openshift-ingress/e2e-worker-w1-88bbf7-openshift-default-574f47fff7-dthhl" Nov 03 01:04:09.901543 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:04:09.901521 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-54lt4\" (UniqueName: \"kubernetes.io/projected/a1e98e45-9f37-42eb-9ac6-4624f63fe503-kube-api-access-54lt4\") pod \"e2e-worker-w1-88bbf7-openshift-default-574f47fff7-dthhl\" (UID: \"a1e98e45-9f37-42eb-9ac6-4624f63fe503\") " pod="openshift-ingress/e2e-worker-w1-88bbf7-openshift-default-574f47fff7-dthhl" Nov 03 01:04:09.985310 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:04:09.985275 4939 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-88bbf7-openshift-default-574f47fff7-dthhl" Nov 03 01:04:10.016169 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:04:10.016133 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:04:10.146846 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:04:10.146444 4939 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-b-f14bda-openshift-default-7c65bb6969-rdtlg" Nov 03 01:04:10.206501 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:04:10.206452 4939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/959b27c0-16c2-41c2-989c-d86f12cedee9-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/959b27c0-16c2-41c2-989c-d86f12cedee9-istio-data\") pod \"959b27c0-16c2-41c2-989c-d86f12cedee9\" (UID: \"959b27c0-16c2-41c2-989c-d86f12cedee9\") " Nov 03 01:04:10.206688 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:04:10.206566 4939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/959b27c0-16c2-41c2-989c-d86f12cedee9-istio-token\" (UniqueName: \"kubernetes.io/projected/959b27c0-16c2-41c2-989c-d86f12cedee9-istio-token\") pod \"959b27c0-16c2-41c2-989c-d86f12cedee9\" (UID: \"959b27c0-16c2-41c2-989c-d86f12cedee9\") " Nov 03 01:04:10.206688 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:04:10.206607 4939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/959b27c0-16c2-41c2-989c-d86f12cedee9-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/959b27c0-16c2-41c2-989c-d86f12cedee9-istiod-ca-cert\") pod \"959b27c0-16c2-41c2-989c-d86f12cedee9\" (UID: \"959b27c0-16c2-41c2-989c-d86f12cedee9\") " Nov 03 01:04:10.206817 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:04:10.206707 4939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/959b27c0-16c2-41c2-989c-d86f12cedee9-kube-api-access-mck9s\" (UniqueName: \"kubernetes.io/projected/959b27c0-16c2-41c2-989c-d86f12cedee9-kube-api-access-mck9s\") pod \"959b27c0-16c2-41c2-989c-d86f12cedee9\" (UID: \"959b27c0-16c2-41c2-989c-d86f12cedee9\") " Nov 03 01:04:10.206817 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:04:10.206754 4939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/959b27c0-16c2-41c2-989c-d86f12cedee9-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/959b27c0-16c2-41c2-989c-d86f12cedee9-credential-socket\") pod \"959b27c0-16c2-41c2-989c-d86f12cedee9\" (UID: \"959b27c0-16c2-41c2-989c-d86f12cedee9\") " Nov 03 01:04:10.206970 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:04:10.206876 4939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/959b27c0-16c2-41c2-989c-d86f12cedee9-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/959b27c0-16c2-41c2-989c-d86f12cedee9-workload-socket\") pod \"959b27c0-16c2-41c2-989c-d86f12cedee9\" (UID: \"959b27c0-16c2-41c2-989c-d86f12cedee9\") " Nov 03 01:04:10.206970 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:04:10.206951 4939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/959b27c0-16c2-41c2-989c-d86f12cedee9-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/959b27c0-16c2-41c2-989c-d86f12cedee9-workload-certs\") pod \"959b27c0-16c2-41c2-989c-d86f12cedee9\" (UID: \"959b27c0-16c2-41c2-989c-d86f12cedee9\") " Nov 03 01:04:10.207098 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:04:10.207017 4939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/959b27c0-16c2-41c2-989c-d86f12cedee9-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/959b27c0-16c2-41c2-989c-d86f12cedee9-istio-podinfo\") pod \"959b27c0-16c2-41c2-989c-d86f12cedee9\" (UID: \"959b27c0-16c2-41c2-989c-d86f12cedee9\") " Nov 03 01:04:10.207098 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:04:10.207074 4939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/959b27c0-16c2-41c2-989c-d86f12cedee9-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/959b27c0-16c2-41c2-989c-d86f12cedee9-istio-envoy\") pod \"959b27c0-16c2-41c2-989c-d86f12cedee9\" (UID: \"959b27c0-16c2-41c2-989c-d86f12cedee9\") " Nov 03 01:04:10.207716 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:04:10.207312 4939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/959b27c0-16c2-41c2-989c-d86f12cedee9-workload-certs" pod "959b27c0-16c2-41c2-989c-d86f12cedee9" (UID: "959b27c0-16c2-41c2-989c-d86f12cedee9"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 01:04:10.207716 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:04:10.207493 4939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/959b27c0-16c2-41c2-989c-d86f12cedee9-istio-data" pod "959b27c0-16c2-41c2-989c-d86f12cedee9" (UID: "959b27c0-16c2-41c2-989c-d86f12cedee9"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 01:04:10.207716 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:04:10.207514 4939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/959b27c0-16c2-41c2-989c-d86f12cedee9-workload-socket" pod "959b27c0-16c2-41c2-989c-d86f12cedee9" (UID: "959b27c0-16c2-41c2-989c-d86f12cedee9"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 01:04:10.207971 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:04:10.207936 4939 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/959b27c0-16c2-41c2-989c-d86f12cedee9-workload-socket\") on node \"ip-10-0-140-107.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:04:10.207971 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:04:10.207952 4939 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/959b27c0-16c2-41c2-989c-d86f12cedee9-workload-certs\") on node \"ip-10-0-140-107.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:04:10.207971 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:04:10.207961 4939 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/959b27c0-16c2-41c2-989c-d86f12cedee9-istio-data\") on node \"ip-10-0-140-107.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:04:10.208194 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:04:10.208169 4939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/959b27c0-16c2-41c2-989c-d86f12cedee9-credential-socket" pod "959b27c0-16c2-41c2-989c-d86f12cedee9" (UID: "959b27c0-16c2-41c2-989c-d86f12cedee9"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 01:04:10.208391 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:04:10.208373 4939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/959b27c0-16c2-41c2-989c-d86f12cedee9-istiod-ca-cert" pod "959b27c0-16c2-41c2-989c-d86f12cedee9" (UID: "959b27c0-16c2-41c2-989c-d86f12cedee9"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 01:04:10.211112 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:04:10.211085 4939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/959b27c0-16c2-41c2-989c-d86f12cedee9-kube-api-access-mck9s" pod "959b27c0-16c2-41c2-989c-d86f12cedee9" (UID: "959b27c0-16c2-41c2-989c-d86f12cedee9"). InnerVolumeSpecName "kube-api-access-mck9s". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 01:04:10.211215 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:04:10.211176 4939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/959b27c0-16c2-41c2-989c-d86f12cedee9-istio-token" pod "959b27c0-16c2-41c2-989c-d86f12cedee9" (UID: "959b27c0-16c2-41c2-989c-d86f12cedee9"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 01:04:10.211518 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:04:10.211492 4939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/959b27c0-16c2-41c2-989c-d86f12cedee9-istio-podinfo" pod "959b27c0-16c2-41c2-989c-d86f12cedee9" (UID: "959b27c0-16c2-41c2-989c-d86f12cedee9"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 01:04:10.214383 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:04:10.214343 4939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/959b27c0-16c2-41c2-989c-d86f12cedee9-istio-envoy" pod "959b27c0-16c2-41c2-989c-d86f12cedee9" (UID: "959b27c0-16c2-41c2-989c-d86f12cedee9"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 01:04:10.309251 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:04:10.309214 4939 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/959b27c0-16c2-41c2-989c-d86f12cedee9-istio-podinfo\") on node \"ip-10-0-140-107.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:04:10.309251 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:04:10.309248 4939 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/959b27c0-16c2-41c2-989c-d86f12cedee9-istio-envoy\") on node \"ip-10-0-140-107.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:04:10.309251 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:04:10.309257 4939 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/959b27c0-16c2-41c2-989c-d86f12cedee9-istio-token\") on node \"ip-10-0-140-107.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:04:10.309251 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:04:10.309266 4939 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/959b27c0-16c2-41c2-989c-d86f12cedee9-istiod-ca-cert\") on node \"ip-10-0-140-107.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:04:10.309530 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:04:10.309275 4939 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mck9s\" (UniqueName: \"kubernetes.io/projected/959b27c0-16c2-41c2-989c-d86f12cedee9-kube-api-access-mck9s\") on node \"ip-10-0-140-107.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:04:10.309530 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:04:10.309284 4939 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/959b27c0-16c2-41c2-989c-d86f12cedee9-credential-socket\") on node \"ip-10-0-140-107.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:04:10.359930 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:04:10.359889 4939 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-88bbf7-openshift-default-574f47fff7-dthhl"] Nov 03 01:04:10.361053 ip-10-0-140-107 kubenswrapper[4939]: W1103 01:04:10.361025 4939 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda1e98e45_9f37_42eb_9ac6_4624f63fe503.slice/crio-babe89d2a31e97a1cf4f4f7b68c2683d843be742097bc1a69036995c7d058f61 WatchSource:0}: Error finding container babe89d2a31e97a1cf4f4f7b68c2683d843be742097bc1a69036995c7d058f61: Status 404 returned error can't find the container with id babe89d2a31e97a1cf4f4f7b68c2683d843be742097bc1a69036995c7d058f61 Nov 03 01:04:10.363692 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:04:10.363656 4939 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 01:04:10.363779 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:04:10.363729 4939 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 01:04:10.363779 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:04:10.363760 4939 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 01:04:10.899386 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:04:10.899328 4939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-88bbf7-openshift-default-574f47fff7-dthhl" event={"ID":"a1e98e45-9f37-42eb-9ac6-4624f63fe503","Type":"ContainerStarted","Data":"fba2e118254d3ef30d6581dc566cc3df6786088bb12ca7c5f2bdf82e3e01d3f3"} Nov 03 01:04:10.899596 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:04:10.899397 4939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-88bbf7-openshift-default-574f47fff7-dthhl" event={"ID":"a1e98e45-9f37-42eb-9ac6-4624f63fe503","Type":"ContainerStarted","Data":"babe89d2a31e97a1cf4f4f7b68c2683d843be742097bc1a69036995c7d058f61"} Nov 03 01:04:10.902980 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:04:10.902947 4939 generic.go:356] "Generic (PLEG): container finished" podID="959b27c0-16c2-41c2-989c-d86f12cedee9" containerID="7521a7b1aa8deb7826036d0beab664b06c57ed2240764944788e20f3f405dbc1" exitCode=0 Nov 03 01:04:10.903246 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:04:10.903217 4939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-f14bda-openshift-default-7c65bb6969-rdtlg" event={"ID":"959b27c0-16c2-41c2-989c-d86f12cedee9","Type":"ContainerDied","Data":"7521a7b1aa8deb7826036d0beab664b06c57ed2240764944788e20f3f405dbc1"} Nov 03 01:04:10.903406 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:04:10.903278 4939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-f14bda-openshift-default-7c65bb6969-rdtlg" event={"ID":"959b27c0-16c2-41c2-989c-d86f12cedee9","Type":"ContainerDied","Data":"4f69ecc869bdcb8c53ee6824847f79025e2be8b5e5d152b98e0b397a1229006d"} Nov 03 01:04:10.903406 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:04:10.903307 4939 scope.go:122] "RemoveContainer" containerID="7521a7b1aa8deb7826036d0beab664b06c57ed2240764944788e20f3f405dbc1" Nov 03 01:04:10.903537 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:04:10.903472 4939 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-b-f14bda-openshift-default-7c65bb6969-rdtlg" Nov 03 01:04:11.000932 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:04:10.972122 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:04:11.000932 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:04:10.991834 4939 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-88bbf7-openshift-default-574f47fff7-dthhl" Nov 03 01:04:11.012958 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:04:11.010713 4939 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-88bbf7-openshift-default-574f47fff7-dthhl" podStartSLOduration=2.01069485 podStartE2EDuration="2.01069485s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 01:04:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 01:04:10.968206913 +0000 UTC m=+1399.232541860" watchObservedRunningTime="2026-11-03 01:04:11.01069485 +0000 UTC m=+1399.275029794" Nov 03 01:04:11.024274 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:04:11.024178 4939 scope.go:122] "RemoveContainer" containerID="366bf518257472a8114c5031209c632fe88cd3a6bfc6a7cc0175d0edc46192ab" Nov 03 01:04:11.041604 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:04:11.041551 4939 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-praxis-b-f14bda-openshift-default-7c65bb6969-rdtlg"] Nov 03 01:04:11.044579 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:04:11.044305 4939 scope.go:122] "RemoveContainer" containerID="7521a7b1aa8deb7826036d0beab664b06c57ed2240764944788e20f3f405dbc1" Nov 03 01:04:11.045048 ip-10-0-140-107 kubenswrapper[4939]: E1103 01:04:11.044729 4939 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7521a7b1aa8deb7826036d0beab664b06c57ed2240764944788e20f3f405dbc1\": container with ID starting with 7521a7b1aa8deb7826036d0beab664b06c57ed2240764944788e20f3f405dbc1 not found: ID does not exist" containerID="7521a7b1aa8deb7826036d0beab664b06c57ed2240764944788e20f3f405dbc1" Nov 03 01:04:11.045048 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:04:11.044768 4939 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7521a7b1aa8deb7826036d0beab664b06c57ed2240764944788e20f3f405dbc1"} err="failed to get container status \"7521a7b1aa8deb7826036d0beab664b06c57ed2240764944788e20f3f405dbc1\": rpc error: code = NotFound desc = could not find container \"7521a7b1aa8deb7826036d0beab664b06c57ed2240764944788e20f3f405dbc1\": container with ID starting with 7521a7b1aa8deb7826036d0beab664b06c57ed2240764944788e20f3f405dbc1 not found: ID does not exist" Nov 03 01:04:11.045048 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:04:11.044795 4939 scope.go:122] "RemoveContainer" containerID="366bf518257472a8114c5031209c632fe88cd3a6bfc6a7cc0175d0edc46192ab" Nov 03 01:04:11.045507 ip-10-0-140-107 kubenswrapper[4939]: E1103 01:04:11.045419 4939 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"366bf518257472a8114c5031209c632fe88cd3a6bfc6a7cc0175d0edc46192ab\": container with ID starting with 366bf518257472a8114c5031209c632fe88cd3a6bfc6a7cc0175d0edc46192ab not found: ID does not exist" containerID="366bf518257472a8114c5031209c632fe88cd3a6bfc6a7cc0175d0edc46192ab" Nov 03 01:04:11.045507 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:04:11.045454 4939 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"366bf518257472a8114c5031209c632fe88cd3a6bfc6a7cc0175d0edc46192ab"} err="failed to get container status \"366bf518257472a8114c5031209c632fe88cd3a6bfc6a7cc0175d0edc46192ab\": rpc error: code = NotFound desc = could not find container \"366bf518257472a8114c5031209c632fe88cd3a6bfc6a7cc0175d0edc46192ab\": container with ID starting with 366bf518257472a8114c5031209c632fe88cd3a6bfc6a7cc0175d0edc46192ab not found: ID does not exist" Nov 03 01:04:11.049947 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:04:11.049920 4939 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-praxis-b-f14bda-openshift-default-7c65bb6969-rdtlg"] Nov 03 01:04:11.882342 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:04:11.882310 4939 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="959b27c0-16c2-41c2-989c-d86f12cedee9" path="/var/lib/kubelet/pods/959b27c0-16c2-41c2-989c-d86f12cedee9/volumes" Nov 03 01:04:11.890164 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:04:11.890088 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:04:12.001169 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:04:11.999549 4939 patch_prober.go:29] interesting pod/e2e-worker-w1-88bbf7-openshift-default-574f47fff7-dthhl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.161:15021/healthz/ready\": dial tcp 10.132.0.161:15021: connect: connection refused" start-of-body= Nov 03 01:04:12.001169 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:04:11.999612 4939 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-88bbf7-openshift-default-574f47fff7-dthhl" podUID="a1e98e45-9f37-42eb-9ac6-4624f63fe503" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.161:15021/healthz/ready\": dial tcp 10.132.0.161:15021: connect: connection refused" Nov 03 01:04:12.422343 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:04:12.422295 4939 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-praxis-a-1e352e-585bf55c69-ddbn9" podUID="4372e01e-917c-4de9-b2de-3882fcd6d87f" containerName="payload-processing" containerID="cri-o://beacdbc898060f4136c58383b5353930702aae7e938e46b411609394b0d20973" gracePeriod=25 Nov 03 01:04:12.439919 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:04:12.439858 4939 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-1e352e-6bb6bb4d7f-hd98j" podUID="04358d76-4e0d-4b13-9307-4734a8891250" containerName="payload-pre-processing" containerID="cri-o://c9eafbc622358d7ef47203bd8ed798c5814452456c54093c9c5ed4242277be91" gracePeriod=25 Nov 03 01:04:12.601744 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:04:12.601709 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:04:12.985981 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:04:12.985952 4939 patch_prober.go:29] interesting pod/e2e-worker-w1-88bbf7-openshift-default-574f47fff7-dthhl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.161:15021/healthz/ready\": dial tcp 10.132.0.161:15021: connect: connection refused" start-of-body= Nov 03 01:04:12.986501 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:04:12.986009 4939 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-88bbf7-openshift-default-574f47fff7-dthhl" podUID="a1e98e45-9f37-42eb-9ac6-4624f63fe503" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.161:15021/healthz/ready\": dial tcp 10.132.0.161:15021: connect: connection refused" Nov 03 01:04:13.320062 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:04:13.319976 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:04:13.987154 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:04:13.987113 4939 patch_prober.go:29] interesting pod/e2e-worker-w1-88bbf7-openshift-default-574f47fff7-dthhl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.161:15021/healthz/ready\": dial tcp 10.132.0.161:15021: connect: connection refused" start-of-body= Nov 03 01:04:13.987675 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:04:13.987193 4939 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-88bbf7-openshift-default-574f47fff7-dthhl" podUID="a1e98e45-9f37-42eb-9ac6-4624f63fe503" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.161:15021/healthz/ready\": dial tcp 10.132.0.161:15021: connect: connection refused" Nov 03 01:04:14.271571 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:04:14.271479 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:04:14.978265 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:04:14.978235 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:04:14.986803 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:04:14.986773 4939 patch_prober.go:29] interesting pod/e2e-worker-w1-88bbf7-openshift-default-574f47fff7-dthhl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.161:15021/healthz/ready\": dial tcp 10.132.0.161:15021: connect: connection refused" start-of-body= Nov 03 01:04:14.987060 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:04:14.987027 4939 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-88bbf7-openshift-default-574f47fff7-dthhl" podUID="a1e98e45-9f37-42eb-9ac6-4624f63fe503" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.161:15021/healthz/ready\": dial tcp 10.132.0.161:15021: connect: connection refused" Nov 03 01:04:15.127960 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:04:15.125871 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:04:15.916694 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:04:15.916657 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:04:15.986368 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:04:15.986330 4939 patch_prober.go:29] interesting pod/e2e-worker-w1-88bbf7-openshift-default-574f47fff7-dthhl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.161:15021/healthz/ready\": dial tcp 10.132.0.161:15021: connect: connection refused" start-of-body= Nov 03 01:04:15.986648 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:04:15.986621 4939 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-88bbf7-openshift-default-574f47fff7-dthhl" podUID="a1e98e45-9f37-42eb-9ac6-4624f63fe503" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.161:15021/healthz/ready\": dial tcp 10.132.0.161:15021: connect: connection refused" Nov 03 01:04:16.691074 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:04:16.691036 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:04:16.986461 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:04:16.986373 4939 patch_prober.go:29] interesting pod/e2e-worker-w1-88bbf7-openshift-default-574f47fff7-dthhl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.161:15021/healthz/ready\": dial tcp 10.132.0.161:15021: connect: connection refused" start-of-body= Nov 03 01:04:16.986653 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:04:16.986462 4939 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-88bbf7-openshift-default-574f47fff7-dthhl" podUID="a1e98e45-9f37-42eb-9ac6-4624f63fe503" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.161:15021/healthz/ready\": dial tcp 10.132.0.161:15021: connect: connection refused" Nov 03 01:04:17.479578 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:04:17.479541 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:04:17.990819 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:04:17.990784 4939 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-88bbf7-openshift-default-574f47fff7-dthhl" Nov 03 01:04:17.991259 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:04:17.991241 4939 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-88bbf7-openshift-default-574f47fff7-dthhl" Nov 03 01:04:17.991940 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:04:17.991889 4939 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-88bbf7-openshift-default-574f47fff7-dthhl" Nov 03 01:04:18.056804 ip-10-0-140-107 kubenswrapper[4939]: E1103 01:04:18.056764 4939 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-sthjh" containerName="extractor" Nov 03 01:04:18.266054 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:04:18.265967 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:04:18.544753 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:04:18.544654 4939 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-1e352e-6bb6bb4d7f-hd98j" podUID="04358d76-4e0d-4b13-9307-4734a8891250" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 03 01:04:18.544753 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:04:18.544654 4939 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-praxis-a-1e352e-585bf55c69-ddbn9" podUID="4372e01e-917c-4de9-b2de-3882fcd6d87f" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 03 01:04:18.879334 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:04:18.879303 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:04:19.495559 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:04:19.495522 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:04:20.178075 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:04:20.178040 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:04:20.778094 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:04:20.778060 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:04:20.988327 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:04:20.988295 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:04:21.573268 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:04:21.573236 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:04:22.261159 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:04:22.261126 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:04:22.866628 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:04:22.866589 4939 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-4pw7v"] Nov 03 01:04:22.887440 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:04:22.887409 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:04:23.841052 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:04:23.841018 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:04:25.123934 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:04:25.037832 4939 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-f7680d/premium-w2-f7680d-kserve-76565b8cf9-hqbbt" podUID="4712e708-18a2-4a13-a12c-a5669397b2c3" containerName="main" probeResult="failure" output="Get \"https://10.132.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 01:04:25.227217 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:04:25.227160 4939 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-f7680d/distinct-2-w2-f7680d-kserve-7d7c4564bf-cbv2c" podUID="2de18086-761b-4773-b9fc-15ddc4a220fc" containerName="main" probeResult="failure" output="Get \"https://10.132.0.75:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 01:04:25.237326 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:04:25.233491 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:04:25.320051 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:04:25.320005 4939 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-2f9a3a/distinct-w0-2f9a3a-kserve-555565cfcf-jtcr9" podUID="ac0ece06-8718-463f-915f-5315988d9711" containerName="main" probeResult="failure" output="Get \"https://10.132.0.76:8000/ready\": context deadline exceeded" Nov 03 01:04:25.420032 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:04:25.410067 4939 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-88bbf7/distinct-2-w1-88bbf7-kserve-5457cdcdbd-l6tfd" podUID="a397931b-faa9-4c2e-8290-491350546663" containerName="main" probeResult="failure" output="Get \"https://10.132.0.85:8000/health\": context deadline exceeded" Nov 03 01:04:25.478552 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:04:25.478502 4939 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-88bbf7/distinct-2-w1-88bbf7-kserve-5457cdcdbd-l6tfd" podUID="a397931b-faa9-4c2e-8290-491350546663" containerName="main" probeResult="failure" output="Get \"https://10.132.0.85:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 01:04:25.563269 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:04:25.562755 4939 patch_prober.go:29] interesting pod/e2e-worker-w1-88bbf7-openshift-default-574f47fff7-rf5cn container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.59:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 01:04:25.563269 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:04:25.562826 4939 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w1-88bbf7-openshift-default-574f47fff7-rf5cn" podUID="08f62205-b31d-4863-ab9e-3b13b15fbef5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.59:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 01:04:25.607133 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:04:25.607080 4939 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-2f9a3a/simulator-w0-2f9a3a-kserve-9596bbb8f-2trgz" podUID="207e2207-fb6c-4076-95df-72e95671c261" containerName="main" probeResult="failure" output="Get \"https://10.132.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 01:04:25.609390 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:04:25.609356 4939 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-83755a/distinct-2-w3-83755a-kserve-6cf66b995d-2l47b" podUID="c3b05e88-5d5b-4d03-b28d-832f22e3c868" containerName="main" probeResult="failure" output="Get \"https://10.132.0.72:8000/ready\": context deadline exceeded" Nov 03 01:04:25.985551 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:04:25.985505 4939 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-2f9a3a/simulator-w0-2f9a3a-kserve-9596bbb8f-2trgz" podUID="207e2207-fb6c-4076-95df-72e95671c261" containerName="main" probeResult="failure" output="Get \"https://10.132.0.61:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 01:04:26.025089 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:04:26.025043 4939 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-f7680d/distinct-w2-f7680d-kserve-756f874b98-qnpv8" podUID="836753fc-f9ef-4f61-b3fe-e5b92b7ee27b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.70:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 01:04:26.059719 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:04:26.059676 4939 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-88bbf7/simulator-w1-88bbf7-kserve-5fbb669f8f-42hgd" podUID="83b38883-2e7a-444d-aa68-52be4a25ae66" containerName="main" probeResult="failure" output="Get \"https://10.132.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 01:04:26.062991 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:04:26.062532 4939 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-f7680d/premium-w2-f7680d-kserve-76565b8cf9-hqbbt" podUID="4712e708-18a2-4a13-a12c-a5669397b2c3" containerName="main" probeResult="failure" output="Get \"https://10.132.0.63:8000/ready\": context deadline exceeded" Nov 03 01:04:26.112065 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:04:26.112022 4939 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-f7680d/premium-w2-f7680d-kserve-76565b8cf9-hqbbt" Nov 03 01:04:26.112674 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:04:26.089038 4939 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-83755a/distinct-2-w3-83755a-kserve-6cf66b995d-2l47b" podUID="c3b05e88-5d5b-4d03-b28d-832f22e3c868" containerName="main" probeResult="failure" output="Get \"https://10.132.0.72:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 01:04:26.292075 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:04:26.260209 4939 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-f7680d/distinct-2-w2-f7680d-kserve-7d7c4564bf-cbv2c" podUID="2de18086-761b-4773-b9fc-15ddc4a220fc" containerName="main" probeResult="failure" output="Get \"https://10.132.0.75:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 01:04:26.292075 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:04:26.288030 4939 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-f7680d/distinct-2-w2-f7680d-kserve-7d7c4564bf-cbv2c" Nov 03 01:04:26.368933 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:04:26.350886 4939 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-2f9a3a/distinct-w0-2f9a3a-kserve-555565cfcf-jtcr9" podUID="ac0ece06-8718-463f-915f-5315988d9711" containerName="main" probeResult="failure" output="Get \"https://10.132.0.76:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 01:04:26.368933 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:04:26.351312 4939 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-2f9a3a/distinct-w0-2f9a3a-kserve-555565cfcf-jtcr9" Nov 03 01:04:26.421991 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:04:26.402050 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:04:26.643971 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:04:26.636072 4939 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-83755a/distinct-2-w3-83755a-kserve-6cf66b995d-2l47b" podUID="c3b05e88-5d5b-4d03-b28d-832f22e3c868" containerName="main" probeResult="failure" output="Get \"https://10.132.0.72:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 01:04:26.643971 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:04:26.636662 4939 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-2f9a3a/simulator-w0-2f9a3a-kserve-9596bbb8f-2trgz" podUID="207e2207-fb6c-4076-95df-72e95671c261" containerName="main" probeResult="failure" output="Get \"https://10.132.0.61:8000/ready\": context deadline exceeded" Nov 03 01:04:26.651000 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:04:26.646551 4939 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-83755a/distinct-2-w3-83755a-kserve-6cf66b995d-2l47b" Nov 03 01:04:26.651000 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:04:26.646995 4939 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-2f9a3a/simulator-w0-2f9a3a-kserve-9596bbb8f-2trgz" Nov 03 01:04:26.938986 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:04:26.938459 4939 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-88bbf7/distinct-2-w1-88bbf7-kserve-5457cdcdbd-l6tfd" podUID="a397931b-faa9-4c2e-8290-491350546663" containerName="main" probeResult="failure" output="Get \"https://10.132.0.85:8000/ready\": context deadline exceeded" Nov 03 01:04:27.022945 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:04:27.022314 4939 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-88bbf7/distinct-2-w1-88bbf7-kserve-5457cdcdbd-l6tfd" Nov 03 01:04:27.102723 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:04:27.102451 4939 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-f7680d/distinct-w2-f7680d-kserve-756f874b98-qnpv8" podUID="836753fc-f9ef-4f61-b3fe-e5b92b7ee27b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.70:8000/ready\": context deadline exceeded" Nov 03 01:04:27.188647 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:04:27.188600 4939 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-f7680d/distinct-w2-f7680d-kserve-756f874b98-qnpv8" Nov 03 01:04:27.225306 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:04:27.225200 4939 patch_prober.go:29] interesting pod/e2e-worker-w3-83755a-openshift-default-c86f9565c-qlgb7 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.71:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 01:04:27.402941 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:04:27.379496 4939 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w3-83755a-openshift-default-c86f9565c-qlgb7" podUID="ba28e265-fab5-43ba-94b2-dc5b7dd5029f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.71:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 01:04:27.402941 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:04:27.330334 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:04:27.402941 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:04:27.317437 4939 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-f7680d/distinct-2-w2-f7680d-kserve-7d7c4564bf-cbv2c" podUID="2de18086-761b-4773-b9fc-15ddc4a220fc" containerName="main" probeResult="failure" output="Get \"https://10.132.0.75:8000/ready\": context deadline exceeded" Nov 03 01:04:27.402941 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:04:27.353472 4939 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-2f9a3a/distinct-w0-2f9a3a-kserve-555565cfcf-jtcr9" podUID="ac0ece06-8718-463f-915f-5315988d9711" containerName="main" probeResult="failure" output="Get \"https://10.132.0.76:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 01:04:27.402941 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:04:27.371316 4939 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-f7680d/premium-w2-f7680d-kserve-76565b8cf9-hqbbt" podUID="4712e708-18a2-4a13-a12c-a5669397b2c3" containerName="main" probeResult="failure" output="Get \"https://10.132.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 01:04:27.636966 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:04:27.636883 4939 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-2f9a3a/simulator-w0-2f9a3a-kserve-9596bbb8f-2trgz" podUID="207e2207-fb6c-4076-95df-72e95671c261" containerName="main" probeResult="failure" output="Get \"https://10.132.0.61:8000/ready\": context deadline exceeded" Nov 03 01:04:27.690082 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:04:27.648612 4939 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-83755a/distinct-2-w3-83755a-kserve-6cf66b995d-2l47b" podUID="c3b05e88-5d5b-4d03-b28d-832f22e3c868" containerName="main" probeResult="failure" output="Get \"https://10.132.0.72:8000/ready\": context deadline exceeded" Nov 03 01:04:27.891999 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:04:27.853863 4939 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-f7680d/premium-w2-f7680d-kserve-76565b8cf9-hqbbt" Nov 03 01:04:27.896996 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:04:27.896957 4939 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1f2827bd-cxnb5"] Nov 03 01:04:27.898570 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:04:27.898148 4939 cpu_manager.go:414] "containerMap: removing container" podUID="959b27c0-16c2-41c2-989c-d86f12cedee9" containerName="istio-proxy" Nov 03 01:04:27.898767 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:04:27.898745 4939 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="959b27c0-16c2-41c2-989c-d86f12cedee9" containerName="istio-proxy" Nov 03 01:04:27.898889 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:04:27.898876 4939 cpu_manager.go:414] "containerMap: removing container" podUID="959b27c0-16c2-41c2-989c-d86f12cedee9" containerName="istio-proxy" Nov 03 01:04:27.899010 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:04:27.898996 4939 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="959b27c0-16c2-41c2-989c-d86f12cedee9" containerName="istio-proxy" Nov 03 01:04:27.911105 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:04:27.911072 4939 memory_manager.go:365] "RemoveStaleState removing state" podUID="959b27c0-16c2-41c2-989c-d86f12cedee9" containerName="istio-proxy" Nov 03 01:04:27.914182 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:04:27.914124 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:04:27.925934 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:04:27.917538 4939 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1f2827bd-cxnb5" Nov 03 01:04:27.942608 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:04:27.942561 4939 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-6q4xp\"" Nov 03 01:04:27.942990 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:04:27.942958 4939 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1f2827bd-cxnb5"] Nov 03 01:04:27.943628 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:04:27.943608 4939 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 01:04:27.948390 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:04:27.945980 4939 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-88bbf7/distinct-2-w1-88bbf7-kserve-5457cdcdbd-l6tfd" podUID="a397931b-faa9-4c2e-8290-491350546663" containerName="main" probeResult="failure" output="Get \"https://10.132.0.85:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 01:04:28.161013 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:04:28.128728 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8mmw7\" (UniqueName: \"kubernetes.io/projected/344f17d1-1f4c-4b07-8ddd-ea417772c6f0-kube-api-access-8mmw7\") pod \"maas-api-revoke-sub-1f2827bd-cxnb5\" (UID: \"344f17d1-1f4c-4b07-8ddd-ea417772c6f0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1f2827bd-cxnb5" Nov 03 01:04:28.161013 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:04:28.128859 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/344f17d1-1f4c-4b07-8ddd-ea417772c6f0-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1f2827bd-cxnb5\" (UID: \"344f17d1-1f4c-4b07-8ddd-ea417772c6f0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1f2827bd-cxnb5" Nov 03 01:04:28.161013 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:04:28.136136 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/344f17d1-1f4c-4b07-8ddd-ea417772c6f0-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1f2827bd-cxnb5\" (UID: \"344f17d1-1f4c-4b07-8ddd-ea417772c6f0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1f2827bd-cxnb5" Nov 03 01:04:28.161013 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:04:28.137132 4939 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-f7680d/distinct-w2-f7680d-kserve-756f874b98-qnpv8" podUID="836753fc-f9ef-4f61-b3fe-e5b92b7ee27b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.70:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 01:04:28.161013 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:04:28.155666 4939 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-f7680d/distinct-w2-f7680d-kserve-756f874b98-qnpv8" podUID="836753fc-f9ef-4f61-b3fe-e5b92b7ee27b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.70:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 01:04:28.197931 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:04:28.194505 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:04:28.248624 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:04:28.248587 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/344f17d1-1f4c-4b07-8ddd-ea417772c6f0-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1f2827bd-cxnb5\" (UID: \"344f17d1-1f4c-4b07-8ddd-ea417772c6f0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1f2827bd-cxnb5" Nov 03 01:04:28.250397 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:04:28.250348 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8mmw7\" (UniqueName: \"kubernetes.io/projected/344f17d1-1f4c-4b07-8ddd-ea417772c6f0-kube-api-access-8mmw7\") pod \"maas-api-revoke-sub-1f2827bd-cxnb5\" (UID: \"344f17d1-1f4c-4b07-8ddd-ea417772c6f0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1f2827bd-cxnb5" Nov 03 01:04:28.250779 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:04:28.250743 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/344f17d1-1f4c-4b07-8ddd-ea417772c6f0-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1f2827bd-cxnb5\" (UID: \"344f17d1-1f4c-4b07-8ddd-ea417772c6f0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1f2827bd-cxnb5" Nov 03 01:04:28.294085 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:04:28.262252 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/344f17d1-1f4c-4b07-8ddd-ea417772c6f0-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1f2827bd-cxnb5\" (UID: \"344f17d1-1f4c-4b07-8ddd-ea417772c6f0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1f2827bd-cxnb5" Nov 03 01:04:28.294085 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:04:28.276584 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/344f17d1-1f4c-4b07-8ddd-ea417772c6f0-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1f2827bd-cxnb5\" (UID: \"344f17d1-1f4c-4b07-8ddd-ea417772c6f0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1f2827bd-cxnb5" Nov 03 01:04:28.294085 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:04:28.290101 4939 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-f7680d/distinct-w2-f7680d-kserve-756f874b98-qnpv8" Nov 03 01:04:28.294085 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:04:28.290259 4939 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-f7680d/distinct-2-w2-f7680d-kserve-7d7c4564bf-cbv2c" Nov 03 01:04:28.338958 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:04:28.338428 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8mmw7\" (UniqueName: \"kubernetes.io/projected/344f17d1-1f4c-4b07-8ddd-ea417772c6f0-kube-api-access-8mmw7\") pod \"maas-api-revoke-sub-1f2827bd-cxnb5\" (UID: \"344f17d1-1f4c-4b07-8ddd-ea417772c6f0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1f2827bd-cxnb5" Nov 03 01:04:28.441558 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:04:28.391797 4939 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-2f9a3a/distinct-w0-2f9a3a-kserve-555565cfcf-jtcr9" podUID="ac0ece06-8718-463f-915f-5315988d9711" containerName="main" probeResult="failure" output="Get \"https://10.132.0.76:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 01:04:28.470889 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:04:28.459588 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:04:28.495173 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:04:28.495143 4939 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1f2827bd-cxnb5" Nov 03 01:04:28.544465 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:04:28.498739 4939 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-praxis-a-1e352e-openshift-default-75f8bbdbfc-gphv5"] Nov 03 01:04:28.545024 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:04:28.544979 4939 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-praxis-a-1e352e-openshift-default-75f8bbdbfc-gphv5" podUID="1d33734b-7202-4184-9fc0-a58898f68802" containerName="istio-proxy" containerID="cri-o://7eca4f7934c25f38045b05e2902a777f3bc239975a80c5c8412eaab6b3c068ea" gracePeriod=30 Nov 03 01:04:28.677876 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:04:28.676119 4939 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-praxis-a-1e352e-585bf55c69-ddbn9" podUID="4372e01e-917c-4de9-b2de-3882fcd6d87f" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 03 01:04:28.677876 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:04:28.676614 4939 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-2f9a3a/simulator-w0-2f9a3a-kserve-9596bbb8f-2trgz" Nov 03 01:04:28.677876 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:04:28.677525 4939 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-2f9a3a/distinct-w0-2f9a3a-kserve-555565cfcf-jtcr9" Nov 03 01:04:28.678401 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:04:28.678371 4939 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-1e352e-6bb6bb4d7f-hd98j" podUID="04358d76-4e0d-4b13-9307-4734a8891250" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 03 01:04:28.707861 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:04:28.707767 4939 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-88bbf7/distinct-2-w1-88bbf7-kserve-5457cdcdbd-l6tfd" Nov 03 01:04:28.711198 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:04:28.709897 4939 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-83755a/distinct-2-w3-83755a-kserve-6cf66b995d-2l47b" Nov 03 01:04:29.051534 ip-10-0-140-107 kubenswrapper[4939]: W1103 01:04:29.051496 4939 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod344f17d1_1f4c_4b07_8ddd_ea417772c6f0.slice/crio-4ab0ab9c27f074059714a996f97827da750a3f7271df60c8e5eedce42b19891a WatchSource:0}: Error finding container 4ab0ab9c27f074059714a996f97827da750a3f7271df60c8e5eedce42b19891a: Status 404 returned error can't find the container with id 4ab0ab9c27f074059714a996f97827da750a3f7271df60c8e5eedce42b19891a Nov 03 01:04:29.052130 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:04:29.052102 4939 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1f2827bd-cxnb5"] Nov 03 01:04:29.301287 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:04:29.301187 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:04:29.809554 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:04:29.809467 4939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1f2827bd-cxnb5" event={"ID":"344f17d1-1f4c-4b07-8ddd-ea417772c6f0","Type":"ContainerStarted","Data":"7ca713fcdcb798bc67f734a39de4adaec4a34d450df68e0935562db4526d44f3"} Nov 03 01:04:29.809554 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:04:29.809550 4939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1f2827bd-cxnb5" event={"ID":"344f17d1-1f4c-4b07-8ddd-ea417772c6f0","Type":"ContainerStarted","Data":"4ab0ab9c27f074059714a996f97827da750a3f7271df60c8e5eedce42b19891a"} Nov 03 01:04:29.828430 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:04:29.828370 4939 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1f2827bd-cxnb5" podStartSLOduration=2.828356641 podStartE2EDuration="2.828356641s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 01:04:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 01:04:29.826536816 +0000 UTC m=+1418.090871764" watchObservedRunningTime="2026-11-03 01:04:29.828356641 +0000 UTC m=+1418.092691586" Nov 03 01:04:30.080258 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:04:30.080154 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:04:30.817879 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:04:30.817840 4939 generic.go:356] "Generic (PLEG): container finished" podID="344f17d1-1f4c-4b07-8ddd-ea417772c6f0" containerID="7ca713fcdcb798bc67f734a39de4adaec4a34d450df68e0935562db4526d44f3" exitCode=0 Nov 03 01:04:30.818328 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:04:30.817887 4939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1f2827bd-cxnb5" event={"ID":"344f17d1-1f4c-4b07-8ddd-ea417772c6f0","Type":"ContainerDied","Data":"7ca713fcdcb798bc67f734a39de4adaec4a34d450df68e0935562db4526d44f3"} Nov 03 01:04:30.825207 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:04:30.825183 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:04:31.508562 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:04:31.508507 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:04:31.991948 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:04:31.991919 4939 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1f2827bd-cxnb5" Nov 03 01:04:32.102604 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:04:32.102575 4939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/344f17d1-1f4c-4b07-8ddd-ea417772c6f0-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/344f17d1-1f4c-4b07-8ddd-ea417772c6f0-maas-api-xks-ca\") pod \"344f17d1-1f4c-4b07-8ddd-ea417772c6f0\" (UID: \"344f17d1-1f4c-4b07-8ddd-ea417772c6f0\") " Nov 03 01:04:32.102803 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:04:32.102784 4939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/344f17d1-1f4c-4b07-8ddd-ea417772c6f0-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/344f17d1-1f4c-4b07-8ddd-ea417772c6f0-maas-api-openshift-ca\") pod \"344f17d1-1f4c-4b07-8ddd-ea417772c6f0\" (UID: \"344f17d1-1f4c-4b07-8ddd-ea417772c6f0\") " Nov 03 01:04:32.102853 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:04:32.102836 4939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/344f17d1-1f4c-4b07-8ddd-ea417772c6f0-kube-api-access-8mmw7\" (UniqueName: \"kubernetes.io/projected/344f17d1-1f4c-4b07-8ddd-ea417772c6f0-kube-api-access-8mmw7\") pod \"344f17d1-1f4c-4b07-8ddd-ea417772c6f0\" (UID: \"344f17d1-1f4c-4b07-8ddd-ea417772c6f0\") " Nov 03 01:04:32.103175 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:04:32.103146 4939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/344f17d1-1f4c-4b07-8ddd-ea417772c6f0-maas-api-openshift-ca" pod "344f17d1-1f4c-4b07-8ddd-ea417772c6f0" (UID: "344f17d1-1f4c-4b07-8ddd-ea417772c6f0"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 01:04:32.103706 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:04:32.103678 4939 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/344f17d1-1f4c-4b07-8ddd-ea417772c6f0-maas-api-openshift-ca\") on node \"ip-10-0-140-107.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:04:32.104804 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:04:32.104780 4939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/344f17d1-1f4c-4b07-8ddd-ea417772c6f0-maas-api-xks-ca" pod "344f17d1-1f4c-4b07-8ddd-ea417772c6f0" (UID: "344f17d1-1f4c-4b07-8ddd-ea417772c6f0"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 01:04:32.104891 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:04:32.104865 4939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/344f17d1-1f4c-4b07-8ddd-ea417772c6f0-kube-api-access-8mmw7" pod "344f17d1-1f4c-4b07-8ddd-ea417772c6f0" (UID: "344f17d1-1f4c-4b07-8ddd-ea417772c6f0"). InnerVolumeSpecName "kube-api-access-8mmw7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 01:04:32.205421 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:04:32.205383 4939 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8mmw7\" (UniqueName: \"kubernetes.io/projected/344f17d1-1f4c-4b07-8ddd-ea417772c6f0-kube-api-access-8mmw7\") on node \"ip-10-0-140-107.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:04:32.205421 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:04:32.205421 4939 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/344f17d1-1f4c-4b07-8ddd-ea417772c6f0-maas-api-xks-ca\") on node \"ip-10-0-140-107.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:04:32.283419 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:04:32.283381 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:04:32.701788 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:04:32.701752 4939 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-a-1e352e-585bf55c69-ddbn9" Nov 03 01:04:32.750741 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:04:32.750712 4939 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-1e352e-6bb6bb4d7f-hd98j" Nov 03 01:04:32.833293 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:04:32.833256 4939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/04358d76-4e0d-4b13-9307-4734a8891250-kube-api-access-jxd9d\" (UniqueName: \"kubernetes.io/projected/04358d76-4e0d-4b13-9307-4734a8891250-kube-api-access-jxd9d\") pod \"04358d76-4e0d-4b13-9307-4734a8891250\" (UID: \"04358d76-4e0d-4b13-9307-4734a8891250\") " Nov 03 01:04:32.833523 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:04:32.833500 4939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4372e01e-917c-4de9-b2de-3882fcd6d87f-kube-api-access-w2cc6\" (UniqueName: \"kubernetes.io/projected/4372e01e-917c-4de9-b2de-3882fcd6d87f-kube-api-access-w2cc6\") pod \"4372e01e-917c-4de9-b2de-3882fcd6d87f\" (UID: \"4372e01e-917c-4de9-b2de-3882fcd6d87f\") " Nov 03 01:04:32.833714 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:04:32.833568 4939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/04358d76-4e0d-4b13-9307-4734a8891250-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/04358d76-4e0d-4b13-9307-4734a8891250-plugins-config-volume\") pod \"04358d76-4e0d-4b13-9307-4734a8891250\" (UID: \"04358d76-4e0d-4b13-9307-4734a8891250\") " Nov 03 01:04:32.833714 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:04:32.833632 4939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4372e01e-917c-4de9-b2de-3882fcd6d87f-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4372e01e-917c-4de9-b2de-3882fcd6d87f-plugins-config-volume\") pod \"4372e01e-917c-4de9-b2de-3882fcd6d87f\" (UID: \"4372e01e-917c-4de9-b2de-3882fcd6d87f\") " Nov 03 01:04:32.834258 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:04:32.834232 4939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/04358d76-4e0d-4b13-9307-4734a8891250-plugins-config-volume" pod "04358d76-4e0d-4b13-9307-4734a8891250" (UID: "04358d76-4e0d-4b13-9307-4734a8891250"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 01:04:32.834364 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:04:32.834247 4939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4372e01e-917c-4de9-b2de-3882fcd6d87f-plugins-config-volume" pod "4372e01e-917c-4de9-b2de-3882fcd6d87f" (UID: "4372e01e-917c-4de9-b2de-3882fcd6d87f"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 01:04:32.834969 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:04:32.834942 4939 generic.go:356] "Generic (PLEG): container finished" podID="04358d76-4e0d-4b13-9307-4734a8891250" containerID="c9eafbc622358d7ef47203bd8ed798c5814452456c54093c9c5ed4242277be91" exitCode=0 Nov 03 01:04:32.835158 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:04:32.834974 4939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-1e352e-6bb6bb4d7f-hd98j" event={"ID":"04358d76-4e0d-4b13-9307-4734a8891250","Type":"ContainerDied","Data":"c9eafbc622358d7ef47203bd8ed798c5814452456c54093c9c5ed4242277be91"} Nov 03 01:04:32.835158 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:04:32.835004 4939 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-1e352e-6bb6bb4d7f-hd98j" Nov 03 01:04:32.835158 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:04:32.835072 4939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-1e352e-6bb6bb4d7f-hd98j" event={"ID":"04358d76-4e0d-4b13-9307-4734a8891250","Type":"ContainerDied","Data":"65fadd941576d539b543577554ebc34a5016011f8d388fb8a5804c510487d14c"} Nov 03 01:04:32.835158 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:04:32.835102 4939 scope.go:122] "RemoveContainer" containerID="c9eafbc622358d7ef47203bd8ed798c5814452456c54093c9c5ed4242277be91" Nov 03 01:04:32.836106 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:04:32.836083 4939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/04358d76-4e0d-4b13-9307-4734a8891250-kube-api-access-jxd9d" pod "04358d76-4e0d-4b13-9307-4734a8891250" (UID: "04358d76-4e0d-4b13-9307-4734a8891250"). InnerVolumeSpecName "kube-api-access-jxd9d". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 01:04:32.836219 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:04:32.836116 4939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4372e01e-917c-4de9-b2de-3882fcd6d87f-kube-api-access-w2cc6" pod "4372e01e-917c-4de9-b2de-3882fcd6d87f" (UID: "4372e01e-917c-4de9-b2de-3882fcd6d87f"). InnerVolumeSpecName "kube-api-access-w2cc6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 01:04:32.836855 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:04:32.836839 4939 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1f2827bd-cxnb5" Nov 03 01:04:32.837062 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:04:32.836856 4939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1f2827bd-cxnb5" event={"ID":"344f17d1-1f4c-4b07-8ddd-ea417772c6f0","Type":"ContainerDied","Data":"4ab0ab9c27f074059714a996f97827da750a3f7271df60c8e5eedce42b19891a"} Nov 03 01:04:32.837062 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:04:32.836896 4939 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="4ab0ab9c27f074059714a996f97827da750a3f7271df60c8e5eedce42b19891a" Nov 03 01:04:32.838514 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:04:32.838451 4939 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-a-1e352e-585bf55c69-ddbn9" Nov 03 01:04:32.838514 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:04:32.838454 4939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-a-1e352e-585bf55c69-ddbn9" event={"ID":"4372e01e-917c-4de9-b2de-3882fcd6d87f","Type":"ContainerDied","Data":"beacdbc898060f4136c58383b5353930702aae7e938e46b411609394b0d20973"} Nov 03 01:04:32.838690 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:04:32.838444 4939 generic.go:356] "Generic (PLEG): container finished" podID="4372e01e-917c-4de9-b2de-3882fcd6d87f" containerID="beacdbc898060f4136c58383b5353930702aae7e938e46b411609394b0d20973" exitCode=0 Nov 03 01:04:32.838690 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:04:32.838578 4939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-a-1e352e-585bf55c69-ddbn9" event={"ID":"4372e01e-917c-4de9-b2de-3882fcd6d87f","Type":"ContainerDied","Data":"45ba2f60218e0605a5a18080ebf964248e7363e351db248486d49b633afa0e3f"} Nov 03 01:04:32.850663 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:04:32.850644 4939 scope.go:122] "RemoveContainer" containerID="c9eafbc622358d7ef47203bd8ed798c5814452456c54093c9c5ed4242277be91" Nov 03 01:04:32.850982 ip-10-0-140-107 kubenswrapper[4939]: E1103 01:04:32.850962 4939 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c9eafbc622358d7ef47203bd8ed798c5814452456c54093c9c5ed4242277be91\": container with ID starting with c9eafbc622358d7ef47203bd8ed798c5814452456c54093c9c5ed4242277be91 not found: ID does not exist" containerID="c9eafbc622358d7ef47203bd8ed798c5814452456c54093c9c5ed4242277be91" Nov 03 01:04:32.851064 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:04:32.850990 4939 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c9eafbc622358d7ef47203bd8ed798c5814452456c54093c9c5ed4242277be91"} err="failed to get container status \"c9eafbc622358d7ef47203bd8ed798c5814452456c54093c9c5ed4242277be91\": rpc error: code = NotFound desc = could not find container \"c9eafbc622358d7ef47203bd8ed798c5814452456c54093c9c5ed4242277be91\": container with ID starting with c9eafbc622358d7ef47203bd8ed798c5814452456c54093c9c5ed4242277be91 not found: ID does not exist" Nov 03 01:04:32.851064 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:04:32.851007 4939 scope.go:122] "RemoveContainer" containerID="beacdbc898060f4136c58383b5353930702aae7e938e46b411609394b0d20973" Nov 03 01:04:32.860432 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:04:32.860408 4939 scope.go:122] "RemoveContainer" containerID="beacdbc898060f4136c58383b5353930702aae7e938e46b411609394b0d20973" Nov 03 01:04:32.860776 ip-10-0-140-107 kubenswrapper[4939]: E1103 01:04:32.860747 4939 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"beacdbc898060f4136c58383b5353930702aae7e938e46b411609394b0d20973\": container with ID starting with beacdbc898060f4136c58383b5353930702aae7e938e46b411609394b0d20973 not found: ID does not exist" containerID="beacdbc898060f4136c58383b5353930702aae7e938e46b411609394b0d20973" Nov 03 01:04:32.860866 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:04:32.860788 4939 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"beacdbc898060f4136c58383b5353930702aae7e938e46b411609394b0d20973"} err="failed to get container status \"beacdbc898060f4136c58383b5353930702aae7e938e46b411609394b0d20973\": rpc error: code = NotFound desc = could not find container \"beacdbc898060f4136c58383b5353930702aae7e938e46b411609394b0d20973\": container with ID starting with beacdbc898060f4136c58383b5353930702aae7e938e46b411609394b0d20973 not found: ID does not exist" Nov 03 01:04:32.864727 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:04:32.864704 4939 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-1e352e-585bf55c69-ddbn9"] Nov 03 01:04:32.870334 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:04:32.870316 4939 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-1e352e-585bf55c69-ddbn9"] Nov 03 01:04:32.935624 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:04:32.935590 4939 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-w2cc6\" (UniqueName: \"kubernetes.io/projected/4372e01e-917c-4de9-b2de-3882fcd6d87f-kube-api-access-w2cc6\") on node \"ip-10-0-140-107.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:04:32.935624 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:04:32.935620 4939 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/04358d76-4e0d-4b13-9307-4734a8891250-plugins-config-volume\") on node \"ip-10-0-140-107.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:04:32.935624 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:04:32.935630 4939 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4372e01e-917c-4de9-b2de-3882fcd6d87f-plugins-config-volume\") on node \"ip-10-0-140-107.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:04:32.935984 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:04:32.935641 4939 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jxd9d\" (UniqueName: \"kubernetes.io/projected/04358d76-4e0d-4b13-9307-4734a8891250-kube-api-access-jxd9d\") on node \"ip-10-0-140-107.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:04:33.162564 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:04:33.162529 4939 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-1e352e-6bb6bb4d7f-hd98j"] Nov 03 01:04:33.165936 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:04:33.165894 4939 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-1e352e-6bb6bb4d7f-hd98j"] Nov 03 01:04:33.190777 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:04:33.190750 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:04:33.848524 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:04:33.848490 4939 generic.go:356] "Generic (PLEG): container finished" podID="1d33734b-7202-4184-9fc0-a58898f68802" containerID="7eca4f7934c25f38045b05e2902a777f3bc239975a80c5c8412eaab6b3c068ea" exitCode=0 Nov 03 01:04:33.848721 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:04:33.848564 4939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-1e352e-openshift-default-75f8bbdbfc-gphv5" event={"ID":"1d33734b-7202-4184-9fc0-a58898f68802","Type":"ContainerDied","Data":"7eca4f7934c25f38045b05e2902a777f3bc239975a80c5c8412eaab6b3c068ea"} Nov 03 01:04:33.848721 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:04:33.848634 4939 scope.go:122] "RemoveContainer" containerID="e7bc838eb15997b3a1363ee712b7e1769fc4c35e00cdc7ec50b7801791a30994" Nov 03 01:04:33.868777 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:04:33.868745 4939 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="04358d76-4e0d-4b13-9307-4734a8891250" path="/var/lib/kubelet/pods/04358d76-4e0d-4b13-9307-4734a8891250/volumes" Nov 03 01:04:33.869299 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:04:33.869277 4939 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4372e01e-917c-4de9-b2de-3882fcd6d87f" path="/var/lib/kubelet/pods/4372e01e-917c-4de9-b2de-3882fcd6d87f/volumes" Nov 03 01:04:34.072879 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:04:34.072770 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:04:34.189428 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:04:34.189393 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:04:34.364987 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:04:34.364965 4939 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-a-1e352e-openshift-default-75f8bbdbfc-gphv5" Nov 03 01:04:34.466782 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:04:34.466751 4939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1d33734b-7202-4184-9fc0-a58898f68802-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1d33734b-7202-4184-9fc0-a58898f68802-workload-socket\") pod \"1d33734b-7202-4184-9fc0-a58898f68802\" (UID: \"1d33734b-7202-4184-9fc0-a58898f68802\") " Nov 03 01:04:34.466975 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:04:34.466962 4939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1d33734b-7202-4184-9fc0-a58898f68802-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1d33734b-7202-4184-9fc0-a58898f68802-credential-socket\") pod \"1d33734b-7202-4184-9fc0-a58898f68802\" (UID: \"1d33734b-7202-4184-9fc0-a58898f68802\") " Nov 03 01:04:34.467047 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:04:34.466987 4939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1d33734b-7202-4184-9fc0-a58898f68802-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1d33734b-7202-4184-9fc0-a58898f68802-istiod-ca-cert\") pod \"1d33734b-7202-4184-9fc0-a58898f68802\" (UID: \"1d33734b-7202-4184-9fc0-a58898f68802\") " Nov 03 01:04:34.467047 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:04:34.467007 4939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1d33734b-7202-4184-9fc0-a58898f68802-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1d33734b-7202-4184-9fc0-a58898f68802-istio-envoy\") pod \"1d33734b-7202-4184-9fc0-a58898f68802\" (UID: \"1d33734b-7202-4184-9fc0-a58898f68802\") " Nov 03 01:04:34.467047 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:04:34.467028 4939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1d33734b-7202-4184-9fc0-a58898f68802-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1d33734b-7202-4184-9fc0-a58898f68802-istio-data\") pod \"1d33734b-7202-4184-9fc0-a58898f68802\" (UID: \"1d33734b-7202-4184-9fc0-a58898f68802\") " Nov 03 01:04:34.467197 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:04:34.467056 4939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1d33734b-7202-4184-9fc0-a58898f68802-workload-socket" pod "1d33734b-7202-4184-9fc0-a58898f68802" (UID: "1d33734b-7202-4184-9fc0-a58898f68802"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 01:04:34.467197 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:04:34.467074 4939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1d33734b-7202-4184-9fc0-a58898f68802-kube-api-access-55f8w\" (UniqueName: \"kubernetes.io/projected/1d33734b-7202-4184-9fc0-a58898f68802-kube-api-access-55f8w\") pod \"1d33734b-7202-4184-9fc0-a58898f68802\" (UID: \"1d33734b-7202-4184-9fc0-a58898f68802\") " Nov 03 01:04:34.467197 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:04:34.467174 4939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/1d33734b-7202-4184-9fc0-a58898f68802-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1d33734b-7202-4184-9fc0-a58898f68802-istio-podinfo\") pod \"1d33734b-7202-4184-9fc0-a58898f68802\" (UID: \"1d33734b-7202-4184-9fc0-a58898f68802\") " Nov 03 01:04:34.467365 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:04:34.467222 4939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1d33734b-7202-4184-9fc0-a58898f68802-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1d33734b-7202-4184-9fc0-a58898f68802-workload-certs\") pod \"1d33734b-7202-4184-9fc0-a58898f68802\" (UID: \"1d33734b-7202-4184-9fc0-a58898f68802\") " Nov 03 01:04:34.467365 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:04:34.467258 4939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1d33734b-7202-4184-9fc0-a58898f68802-istio-token\" (UniqueName: \"kubernetes.io/projected/1d33734b-7202-4184-9fc0-a58898f68802-istio-token\") pod \"1d33734b-7202-4184-9fc0-a58898f68802\" (UID: \"1d33734b-7202-4184-9fc0-a58898f68802\") " Nov 03 01:04:34.467365 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:04:34.467312 4939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1d33734b-7202-4184-9fc0-a58898f68802-istio-data" pod "1d33734b-7202-4184-9fc0-a58898f68802" (UID: "1d33734b-7202-4184-9fc0-a58898f68802"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 01:04:34.467365 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:04:34.467326 4939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1d33734b-7202-4184-9fc0-a58898f68802-credential-socket" pod "1d33734b-7202-4184-9fc0-a58898f68802" (UID: "1d33734b-7202-4184-9fc0-a58898f68802"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 01:04:34.467591 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:04:34.467413 4939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1d33734b-7202-4184-9fc0-a58898f68802-istiod-ca-cert" pod "1d33734b-7202-4184-9fc0-a58898f68802" (UID: "1d33734b-7202-4184-9fc0-a58898f68802"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 01:04:34.467591 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:04:34.467520 4939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1d33734b-7202-4184-9fc0-a58898f68802-workload-certs" pod "1d33734b-7202-4184-9fc0-a58898f68802" (UID: "1d33734b-7202-4184-9fc0-a58898f68802"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 01:04:34.467891 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:04:34.467867 4939 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1d33734b-7202-4184-9fc0-a58898f68802-credential-socket\") on node \"ip-10-0-140-107.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:04:34.468011 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:04:34.467898 4939 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1d33734b-7202-4184-9fc0-a58898f68802-istiod-ca-cert\") on node \"ip-10-0-140-107.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:04:34.468011 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:04:34.467947 4939 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1d33734b-7202-4184-9fc0-a58898f68802-istio-data\") on node \"ip-10-0-140-107.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:04:34.468011 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:04:34.467958 4939 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1d33734b-7202-4184-9fc0-a58898f68802-workload-certs\") on node \"ip-10-0-140-107.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:04:34.468011 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:04:34.467967 4939 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1d33734b-7202-4184-9fc0-a58898f68802-workload-socket\") on node \"ip-10-0-140-107.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:04:34.469945 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:04:34.469893 4939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1d33734b-7202-4184-9fc0-a58898f68802-kube-api-access-55f8w" pod "1d33734b-7202-4184-9fc0-a58898f68802" (UID: "1d33734b-7202-4184-9fc0-a58898f68802"). InnerVolumeSpecName "kube-api-access-55f8w". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 01:04:34.470045 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:04:34.469920 4939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1d33734b-7202-4184-9fc0-a58898f68802-istio-token" pod "1d33734b-7202-4184-9fc0-a58898f68802" (UID: "1d33734b-7202-4184-9fc0-a58898f68802"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 01:04:34.470237 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:04:34.470212 4939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/1d33734b-7202-4184-9fc0-a58898f68802-istio-podinfo" pod "1d33734b-7202-4184-9fc0-a58898f68802" (UID: "1d33734b-7202-4184-9fc0-a58898f68802"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 01:04:34.470362 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:04:34.470303 4939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1d33734b-7202-4184-9fc0-a58898f68802-istio-envoy" pod "1d33734b-7202-4184-9fc0-a58898f68802" (UID: "1d33734b-7202-4184-9fc0-a58898f68802"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 01:04:34.569701 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:04:34.569654 4939 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1d33734b-7202-4184-9fc0-a58898f68802-istio-envoy\") on node \"ip-10-0-140-107.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:04:34.569701 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:04:34.569699 4939 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-55f8w\" (UniqueName: \"kubernetes.io/projected/1d33734b-7202-4184-9fc0-a58898f68802-kube-api-access-55f8w\") on node \"ip-10-0-140-107.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:04:34.569701 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:04:34.569710 4939 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1d33734b-7202-4184-9fc0-a58898f68802-istio-podinfo\") on node \"ip-10-0-140-107.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:04:34.569965 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:04:34.569733 4939 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1d33734b-7202-4184-9fc0-a58898f68802-istio-token\") on node \"ip-10-0-140-107.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:04:34.787553 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:04:34.787463 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:04:34.868197 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:04:34.865704 4939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-1e352e-openshift-default-75f8bbdbfc-gphv5" event={"ID":"1d33734b-7202-4184-9fc0-a58898f68802","Type":"ContainerDied","Data":"fc0d634dcb8f8ae0378b0a6e2de6430f238fab05046ae21362ecba8a3e4d9229"} Nov 03 01:04:34.868197 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:04:34.865781 4939 scope.go:122] "RemoveContainer" containerID="7eca4f7934c25f38045b05e2902a777f3bc239975a80c5c8412eaab6b3c068ea" Nov 03 01:04:34.868197 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:04:34.865972 4939 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-a-1e352e-openshift-default-75f8bbdbfc-gphv5" Nov 03 01:04:34.936774 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:04:34.936725 4939 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-praxis-a-1e352e-openshift-default-75f8bbdbfc-gphv5"] Nov 03 01:04:34.940097 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:04:34.940072 4939 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-praxis-a-1e352e-openshift-default-75f8bbdbfc-gphv5"] Nov 03 01:04:35.394120 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:04:35.394089 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:04:35.870506 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:04:35.870473 4939 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1d33734b-7202-4184-9fc0-a58898f68802" path="/var/lib/kubelet/pods/1d33734b-7202-4184-9fc0-a58898f68802/volumes" Nov 03 01:04:36.116808 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:04:36.116769 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:04:36.775603 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:04:36.775568 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:04:37.375153 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:04:37.375118 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:04:38.264280 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:04:38.264249 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:04:39.007067 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:04:39.007031 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:04:39.527781 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:04:39.527736 4939 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-208a0d-6f7c665d67-pc68m"] Nov 03 01:04:39.538309 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:04:39.538282 4939 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-208a0d-f9cc48655-wfc6q"] Nov 03 01:04:39.634010 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:04:39.633975 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:04:39.747966 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:04:39.747883 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:04:39.950831 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:04:39.950797 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:04:40.604632 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:04:40.604596 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:04:40.708536 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:04:40.707930 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:04:41.146571 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:04:41.146541 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:04:41.529094 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:04:41.529013 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:04:42.184778 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:04:42.184739 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:04:43.152931 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:04:43.146383 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:04:43.482334 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:04:43.482242 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:04:44.535207 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:04:44.535172 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:04:44.535878 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:04:44.535275 4939 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-praxis-b-208a0d-6f7c665d67-pc68m" podUID="e1d5b015-e51e-4f86-ac94-34b05d8db2d3" containerName="payload-processing" containerID="cri-o://294acea636bb9e10e64a9e3ebf26fe4257515f5bc8a787135e5ab2de4de46f16" gracePeriod=25 Nov 03 01:04:44.554388 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:04:44.554308 4939 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-208a0d-f9cc48655-wfc6q" podUID="19d3d3db-71fc-41f2-9976-fa8bcd10bfec" containerName="payload-pre-processing" containerID="cri-o://f6e857800bf7fcec22a2bf93eebc4b9237f4871fe54a63775de09f4823584346" gracePeriod=25 Nov 03 01:04:45.179985 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:04:45.139343 4939 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-88bbf7/distinct-2-w1-88bbf7-kserve-5457cdcdbd-l6tfd" podUID="a397931b-faa9-4c2e-8290-491350546663" containerName="main" probeResult="failure" output="Get \"https://10.132.0.85:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 01:04:45.179985 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:04:45.154059 4939 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-f7680d/distinct-w2-f7680d-kserve-756f874b98-qnpv8" podUID="836753fc-f9ef-4f61-b3fe-e5b92b7ee27b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.70:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 01:04:45.244945 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:04:45.241190 4939 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w1-2729f2-openshift-default-9fb97b4fc-5ww48"] Nov 03 01:04:45.244945 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:04:45.242111 4939 cpu_manager.go:414] "containerMap: removing container" podUID="1d33734b-7202-4184-9fc0-a58898f68802" containerName="istio-proxy" Nov 03 01:04:45.244945 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:04:45.242134 4939 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1d33734b-7202-4184-9fc0-a58898f68802" containerName="istio-proxy" Nov 03 01:04:45.244945 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:04:45.242162 4939 cpu_manager.go:414] "containerMap: removing container" podUID="344f17d1-1f4c-4b07-8ddd-ea417772c6f0" containerName="revoke-keys" Nov 03 01:04:45.244945 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:04:45.242171 4939 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="344f17d1-1f4c-4b07-8ddd-ea417772c6f0" containerName="revoke-keys" Nov 03 01:04:45.244945 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:04:45.242193 4939 cpu_manager.go:414] "containerMap: removing container" podUID="1d33734b-7202-4184-9fc0-a58898f68802" containerName="istio-proxy" Nov 03 01:04:45.244945 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:04:45.242200 4939 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1d33734b-7202-4184-9fc0-a58898f68802" containerName="istio-proxy" Nov 03 01:04:45.244945 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:04:45.242216 4939 cpu_manager.go:414] "containerMap: removing container" podUID="04358d76-4e0d-4b13-9307-4734a8891250" containerName="payload-pre-processing" Nov 03 01:04:45.244945 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:04:45.242224 4939 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="04358d76-4e0d-4b13-9307-4734a8891250" containerName="payload-pre-processing" Nov 03 01:04:45.244945 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:04:45.242244 4939 cpu_manager.go:414] "containerMap: removing container" podUID="4372e01e-917c-4de9-b2de-3882fcd6d87f" containerName="payload-processing" Nov 03 01:04:45.244945 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:04:45.242252 4939 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4372e01e-917c-4de9-b2de-3882fcd6d87f" containerName="payload-processing" Nov 03 01:04:45.244945 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:04:45.242390 4939 memory_manager.go:365] "RemoveStaleState removing state" podUID="1d33734b-7202-4184-9fc0-a58898f68802" containerName="istio-proxy" Nov 03 01:04:45.244945 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:04:45.242404 4939 memory_manager.go:365] "RemoveStaleState removing state" podUID="344f17d1-1f4c-4b07-8ddd-ea417772c6f0" containerName="revoke-keys" Nov 03 01:04:45.244945 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:04:45.242418 4939 memory_manager.go:365] "RemoveStaleState removing state" podUID="04358d76-4e0d-4b13-9307-4734a8891250" containerName="payload-pre-processing" Nov 03 01:04:45.244945 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:04:45.242430 4939 memory_manager.go:365] "RemoveStaleState removing state" podUID="959b27c0-16c2-41c2-989c-d86f12cedee9" containerName="istio-proxy" Nov 03 01:04:45.244945 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:04:45.242441 4939 memory_manager.go:365] "RemoveStaleState removing state" podUID="4372e01e-917c-4de9-b2de-3882fcd6d87f" containerName="payload-processing" Nov 03 01:04:45.279944 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:04:45.277173 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:04:45.382764 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:04:45.369497 4939 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w1-2729f2-openshift-default-9fb97b4fc-5ww48" Nov 03 01:04:45.406933 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:04:45.401423 4939 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-f7680d/distinct-2-w2-f7680d-kserve-7d7c4564bf-cbv2c" podUID="2de18086-761b-4773-b9fc-15ddc4a220fc" containerName="main" probeResult="failure" output="Get \"https://10.132.0.75:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 01:04:45.406933 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:04:45.402059 4939 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-88bbf7/distinct-2-w1-88bbf7-kserve-5457cdcdbd-l6tfd" podUID="a397931b-faa9-4c2e-8290-491350546663" containerName="main" probeResult="failure" output="Get \"https://10.132.0.85:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 01:04:45.406933 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:04:45.402608 4939 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-f7680d/premium-w2-f7680d-kserve-76565b8cf9-hqbbt" podUID="4712e708-18a2-4a13-a12c-a5669397b2c3" containerName="main" probeResult="failure" output="Get \"https://10.132.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 01:04:45.468978 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:04:45.455331 4939 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-2f9a3a/distinct-w0-2f9a3a-kserve-555565cfcf-jtcr9" podUID="ac0ece06-8718-463f-915f-5315988d9711" containerName="main" probeResult="failure" output="Get \"https://10.132.0.76:8000/ready\": context deadline exceeded" Nov 03 01:04:45.486937 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:04:45.475529 4939 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w1-2729f2-openshift-default-9fb97b4fc-5ww48"] Nov 03 01:04:45.547931 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:04:45.541196 4939 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-w1-2729f2-openshift-default-dockercfg-9nk2m\"" Nov 03 01:04:45.570975 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:04:45.550224 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6wqxc\" (UniqueName: \"kubernetes.io/projected/cba006d1-08cd-4fac-8bdd-86931f742838-kube-api-access-6wqxc\") pod \"e2e-shared-a-w1-2729f2-openshift-default-9fb97b4fc-5ww48\" (UID: \"cba006d1-08cd-4fac-8bdd-86931f742838\") " pod="openshift-ingress/e2e-shared-a-w1-2729f2-openshift-default-9fb97b4fc-5ww48" Nov 03 01:04:45.646499 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:04:45.646462 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cba006d1-08cd-4fac-8bdd-86931f742838-istiod-ca-cert\") pod \"e2e-shared-a-w1-2729f2-openshift-default-9fb97b4fc-5ww48\" (UID: \"cba006d1-08cd-4fac-8bdd-86931f742838\") " pod="openshift-ingress/e2e-shared-a-w1-2729f2-openshift-default-9fb97b4fc-5ww48" Nov 03 01:04:45.646799 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:04:45.646781 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cba006d1-08cd-4fac-8bdd-86931f742838-credential-socket\") pod \"e2e-shared-a-w1-2729f2-openshift-default-9fb97b4fc-5ww48\" (UID: \"cba006d1-08cd-4fac-8bdd-86931f742838\") " pod="openshift-ingress/e2e-shared-a-w1-2729f2-openshift-default-9fb97b4fc-5ww48" Nov 03 01:04:45.646995 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:04:45.646973 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cba006d1-08cd-4fac-8bdd-86931f742838-workload-certs\") pod \"e2e-shared-a-w1-2729f2-openshift-default-9fb97b4fc-5ww48\" (UID: \"cba006d1-08cd-4fac-8bdd-86931f742838\") " pod="openshift-ingress/e2e-shared-a-w1-2729f2-openshift-default-9fb97b4fc-5ww48" Nov 03 01:04:45.647256 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:04:45.647236 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cba006d1-08cd-4fac-8bdd-86931f742838-istio-podinfo\") pod \"e2e-shared-a-w1-2729f2-openshift-default-9fb97b4fc-5ww48\" (UID: \"cba006d1-08cd-4fac-8bdd-86931f742838\") " pod="openshift-ingress/e2e-shared-a-w1-2729f2-openshift-default-9fb97b4fc-5ww48" Nov 03 01:04:45.647479 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:04:45.647461 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cba006d1-08cd-4fac-8bdd-86931f742838-istio-data\") pod \"e2e-shared-a-w1-2729f2-openshift-default-9fb97b4fc-5ww48\" (UID: \"cba006d1-08cd-4fac-8bdd-86931f742838\") " pod="openshift-ingress/e2e-shared-a-w1-2729f2-openshift-default-9fb97b4fc-5ww48" Nov 03 01:04:45.647722 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:04:45.647705 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cba006d1-08cd-4fac-8bdd-86931f742838-workload-socket\") pod \"e2e-shared-a-w1-2729f2-openshift-default-9fb97b4fc-5ww48\" (UID: \"cba006d1-08cd-4fac-8bdd-86931f742838\") " pod="openshift-ingress/e2e-shared-a-w1-2729f2-openshift-default-9fb97b4fc-5ww48" Nov 03 01:04:45.647879 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:04:45.647864 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cba006d1-08cd-4fac-8bdd-86931f742838-istio-envoy\") pod \"e2e-shared-a-w1-2729f2-openshift-default-9fb97b4fc-5ww48\" (UID: \"cba006d1-08cd-4fac-8bdd-86931f742838\") " pod="openshift-ingress/e2e-shared-a-w1-2729f2-openshift-default-9fb97b4fc-5ww48" Nov 03 01:04:45.676191 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:04:45.676149 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cba006d1-08cd-4fac-8bdd-86931f742838-istio-token\") pod \"e2e-shared-a-w1-2729f2-openshift-default-9fb97b4fc-5ww48\" (UID: \"cba006d1-08cd-4fac-8bdd-86931f742838\") " pod="openshift-ingress/e2e-shared-a-w1-2729f2-openshift-default-9fb97b4fc-5ww48" Nov 03 01:04:45.676669 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:04:45.647168 4939 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-2f9a3a/simulator-w0-2f9a3a-kserve-9596bbb8f-2trgz" podUID="207e2207-fb6c-4076-95df-72e95671c261" containerName="main" probeResult="failure" output="Get \"https://10.132.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 01:04:45.678172 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:04:45.634299 4939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-208a0d-f9cc48655-wfc6q" event={"ID":"19d3d3db-71fc-41f2-9976-fa8bcd10bfec","Type":"ContainerDied","Data":"f6e857800bf7fcec22a2bf93eebc4b9237f4871fe54a63775de09f4823584346"} Nov 03 01:04:45.678377 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:04:45.634271 4939 generic.go:356] "Generic (PLEG): container finished" podID="19d3d3db-71fc-41f2-9976-fa8bcd10bfec" containerID="f6e857800bf7fcec22a2bf93eebc4b9237f4871fe54a63775de09f4823584346" exitCode=0 Nov 03 01:04:45.733048 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:04:45.714610 4939 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-83755a/distinct-2-w3-83755a-kserve-6cf66b995d-2l47b" podUID="c3b05e88-5d5b-4d03-b28d-832f22e3c868" containerName="main" probeResult="failure" output="Get \"https://10.132.0.72:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 01:04:45.741973 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:04:45.727233 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:04:45.752321 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:04:45.733396 4939 generic.go:356] "Generic (PLEG): container finished" podID="e1d5b015-e51e-4f86-ac94-34b05d8db2d3" containerID="294acea636bb9e10e64a9e3ebf26fe4257515f5bc8a787135e5ab2de4de46f16" exitCode=0 Nov 03 01:04:45.752321 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:04:45.733429 4939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-b-208a0d-6f7c665d67-pc68m" event={"ID":"e1d5b015-e51e-4f86-ac94-34b05d8db2d3","Type":"ContainerDied","Data":"294acea636bb9e10e64a9e3ebf26fe4257515f5bc8a787135e5ab2de4de46f16"} Nov 03 01:04:45.791943 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:04:45.791891 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cba006d1-08cd-4fac-8bdd-86931f742838-istio-token\") pod \"e2e-shared-a-w1-2729f2-openshift-default-9fb97b4fc-5ww48\" (UID: \"cba006d1-08cd-4fac-8bdd-86931f742838\") " pod="openshift-ingress/e2e-shared-a-w1-2729f2-openshift-default-9fb97b4fc-5ww48" Nov 03 01:04:45.792372 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:04:45.792280 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6wqxc\" (UniqueName: \"kubernetes.io/projected/cba006d1-08cd-4fac-8bdd-86931f742838-kube-api-access-6wqxc\") pod \"e2e-shared-a-w1-2729f2-openshift-default-9fb97b4fc-5ww48\" (UID: \"cba006d1-08cd-4fac-8bdd-86931f742838\") " pod="openshift-ingress/e2e-shared-a-w1-2729f2-openshift-default-9fb97b4fc-5ww48" Nov 03 01:04:45.861964 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:04:45.831153 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cba006d1-08cd-4fac-8bdd-86931f742838-istiod-ca-cert\") pod \"e2e-shared-a-w1-2729f2-openshift-default-9fb97b4fc-5ww48\" (UID: \"cba006d1-08cd-4fac-8bdd-86931f742838\") " pod="openshift-ingress/e2e-shared-a-w1-2729f2-openshift-default-9fb97b4fc-5ww48" Nov 03 01:04:45.877032 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:04:45.876991 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cba006d1-08cd-4fac-8bdd-86931f742838-credential-socket\") pod \"e2e-shared-a-w1-2729f2-openshift-default-9fb97b4fc-5ww48\" (UID: \"cba006d1-08cd-4fac-8bdd-86931f742838\") " pod="openshift-ingress/e2e-shared-a-w1-2729f2-openshift-default-9fb97b4fc-5ww48" Nov 03 01:04:45.877316 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:04:45.877290 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cba006d1-08cd-4fac-8bdd-86931f742838-workload-certs\") pod \"e2e-shared-a-w1-2729f2-openshift-default-9fb97b4fc-5ww48\" (UID: \"cba006d1-08cd-4fac-8bdd-86931f742838\") " pod="openshift-ingress/e2e-shared-a-w1-2729f2-openshift-default-9fb97b4fc-5ww48" Nov 03 01:04:45.877478 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:04:45.877459 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cba006d1-08cd-4fac-8bdd-86931f742838-istio-podinfo\") pod \"e2e-shared-a-w1-2729f2-openshift-default-9fb97b4fc-5ww48\" (UID: \"cba006d1-08cd-4fac-8bdd-86931f742838\") " pod="openshift-ingress/e2e-shared-a-w1-2729f2-openshift-default-9fb97b4fc-5ww48" Nov 03 01:04:45.877647 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:04:45.877629 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cba006d1-08cd-4fac-8bdd-86931f742838-istio-data\") pod \"e2e-shared-a-w1-2729f2-openshift-default-9fb97b4fc-5ww48\" (UID: \"cba006d1-08cd-4fac-8bdd-86931f742838\") " pod="openshift-ingress/e2e-shared-a-w1-2729f2-openshift-default-9fb97b4fc-5ww48" Nov 03 01:04:45.917925 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:04:45.917820 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cba006d1-08cd-4fac-8bdd-86931f742838-workload-socket\") pod \"e2e-shared-a-w1-2729f2-openshift-default-9fb97b4fc-5ww48\" (UID: \"cba006d1-08cd-4fac-8bdd-86931f742838\") " pod="openshift-ingress/e2e-shared-a-w1-2729f2-openshift-default-9fb97b4fc-5ww48" Nov 03 01:04:45.918174 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:04:45.918154 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cba006d1-08cd-4fac-8bdd-86931f742838-istio-envoy\") pod \"e2e-shared-a-w1-2729f2-openshift-default-9fb97b4fc-5ww48\" (UID: \"cba006d1-08cd-4fac-8bdd-86931f742838\") " pod="openshift-ingress/e2e-shared-a-w1-2729f2-openshift-default-9fb97b4fc-5ww48" Nov 03 01:04:45.919432 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:04:45.879588 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cba006d1-08cd-4fac-8bdd-86931f742838-istio-data\") pod \"e2e-shared-a-w1-2729f2-openshift-default-9fb97b4fc-5ww48\" (UID: \"cba006d1-08cd-4fac-8bdd-86931f742838\") " pod="openshift-ingress/e2e-shared-a-w1-2729f2-openshift-default-9fb97b4fc-5ww48" Nov 03 01:04:45.919770 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:04:45.915457 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cba006d1-08cd-4fac-8bdd-86931f742838-istio-podinfo\") pod \"e2e-shared-a-w1-2729f2-openshift-default-9fb97b4fc-5ww48\" (UID: \"cba006d1-08cd-4fac-8bdd-86931f742838\") " pod="openshift-ingress/e2e-shared-a-w1-2729f2-openshift-default-9fb97b4fc-5ww48" Nov 03 01:04:45.919927 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:04:45.878500 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cba006d1-08cd-4fac-8bdd-86931f742838-workload-certs\") pod \"e2e-shared-a-w1-2729f2-openshift-default-9fb97b4fc-5ww48\" (UID: \"cba006d1-08cd-4fac-8bdd-86931f742838\") " pod="openshift-ingress/e2e-shared-a-w1-2729f2-openshift-default-9fb97b4fc-5ww48" Nov 03 01:04:45.920514 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:04:45.920492 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cba006d1-08cd-4fac-8bdd-86931f742838-workload-socket\") pod \"e2e-shared-a-w1-2729f2-openshift-default-9fb97b4fc-5ww48\" (UID: \"cba006d1-08cd-4fac-8bdd-86931f742838\") " pod="openshift-ingress/e2e-shared-a-w1-2729f2-openshift-default-9fb97b4fc-5ww48" Nov 03 01:04:45.920633 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:04:45.879439 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cba006d1-08cd-4fac-8bdd-86931f742838-credential-socket\") pod \"e2e-shared-a-w1-2729f2-openshift-default-9fb97b4fc-5ww48\" (UID: \"cba006d1-08cd-4fac-8bdd-86931f742838\") " pod="openshift-ingress/e2e-shared-a-w1-2729f2-openshift-default-9fb97b4fc-5ww48" Nov 03 01:04:45.922453 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:04:45.922423 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cba006d1-08cd-4fac-8bdd-86931f742838-istio-token\") pod \"e2e-shared-a-w1-2729f2-openshift-default-9fb97b4fc-5ww48\" (UID: \"cba006d1-08cd-4fac-8bdd-86931f742838\") " pod="openshift-ingress/e2e-shared-a-w1-2729f2-openshift-default-9fb97b4fc-5ww48" Nov 03 01:04:45.925861 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:04:45.923337 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cba006d1-08cd-4fac-8bdd-86931f742838-istio-envoy\") pod \"e2e-shared-a-w1-2729f2-openshift-default-9fb97b4fc-5ww48\" (UID: \"cba006d1-08cd-4fac-8bdd-86931f742838\") " pod="openshift-ingress/e2e-shared-a-w1-2729f2-openshift-default-9fb97b4fc-5ww48" Nov 03 01:04:45.949392 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:04:45.949355 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6wqxc\" (UniqueName: \"kubernetes.io/projected/cba006d1-08cd-4fac-8bdd-86931f742838-kube-api-access-6wqxc\") pod \"e2e-shared-a-w1-2729f2-openshift-default-9fb97b4fc-5ww48\" (UID: \"cba006d1-08cd-4fac-8bdd-86931f742838\") " pod="openshift-ingress/e2e-shared-a-w1-2729f2-openshift-default-9fb97b4fc-5ww48" Nov 03 01:04:45.956462 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:04:45.956422 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cba006d1-08cd-4fac-8bdd-86931f742838-istiod-ca-cert\") pod \"e2e-shared-a-w1-2729f2-openshift-default-9fb97b4fc-5ww48\" (UID: \"cba006d1-08cd-4fac-8bdd-86931f742838\") " pod="openshift-ingress/e2e-shared-a-w1-2729f2-openshift-default-9fb97b4fc-5ww48" Nov 03 01:04:46.002103 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:04:46.002063 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:04:46.034378 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:04:46.034341 4939 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w1-2729f2-openshift-default-9fb97b4fc-5ww48" Nov 03 01:04:46.154790 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:04:46.154710 4939 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-208a0d-f9cc48655-wfc6q" Nov 03 01:04:46.157729 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:04:46.157630 4939 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-b-208a0d-6f7c665d67-pc68m" Nov 03 01:04:46.237900 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:04:46.237861 4939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e1d5b015-e51e-4f86-ac94-34b05d8db2d3-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e1d5b015-e51e-4f86-ac94-34b05d8db2d3-plugins-config-volume\") pod \"e1d5b015-e51e-4f86-ac94-34b05d8db2d3\" (UID: \"e1d5b015-e51e-4f86-ac94-34b05d8db2d3\") " Nov 03 01:04:46.238117 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:04:46.238037 4939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/19d3d3db-71fc-41f2-9976-fa8bcd10bfec-kube-api-access-2626m\" (UniqueName: \"kubernetes.io/projected/19d3d3db-71fc-41f2-9976-fa8bcd10bfec-kube-api-access-2626m\") pod \"19d3d3db-71fc-41f2-9976-fa8bcd10bfec\" (UID: \"19d3d3db-71fc-41f2-9976-fa8bcd10bfec\") " Nov 03 01:04:46.238202 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:04:46.238122 4939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e1d5b015-e51e-4f86-ac94-34b05d8db2d3-kube-api-access-2784x\" (UniqueName: \"kubernetes.io/projected/e1d5b015-e51e-4f86-ac94-34b05d8db2d3-kube-api-access-2784x\") pod \"e1d5b015-e51e-4f86-ac94-34b05d8db2d3\" (UID: \"e1d5b015-e51e-4f86-ac94-34b05d8db2d3\") " Nov 03 01:04:46.238202 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:04:46.238152 4939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/19d3d3db-71fc-41f2-9976-fa8bcd10bfec-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/19d3d3db-71fc-41f2-9976-fa8bcd10bfec-plugins-config-volume\") pod \"19d3d3db-71fc-41f2-9976-fa8bcd10bfec\" (UID: \"19d3d3db-71fc-41f2-9976-fa8bcd10bfec\") " Nov 03 01:04:46.238734 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:04:46.238430 4939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e1d5b015-e51e-4f86-ac94-34b05d8db2d3-plugins-config-volume" pod "e1d5b015-e51e-4f86-ac94-34b05d8db2d3" (UID: "e1d5b015-e51e-4f86-ac94-34b05d8db2d3"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 01:04:46.238860 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:04:46.238755 4939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/19d3d3db-71fc-41f2-9976-fa8bcd10bfec-plugins-config-volume" pod "19d3d3db-71fc-41f2-9976-fa8bcd10bfec" (UID: "19d3d3db-71fc-41f2-9976-fa8bcd10bfec"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 01:04:46.239317 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:04:46.239261 4939 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/19d3d3db-71fc-41f2-9976-fa8bcd10bfec-plugins-config-volume\") on node \"ip-10-0-140-107.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:04:46.239317 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:04:46.239293 4939 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e1d5b015-e51e-4f86-ac94-34b05d8db2d3-plugins-config-volume\") on node \"ip-10-0-140-107.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:04:46.240992 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:04:46.240957 4939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e1d5b015-e51e-4f86-ac94-34b05d8db2d3-kube-api-access-2784x" pod "e1d5b015-e51e-4f86-ac94-34b05d8db2d3" (UID: "e1d5b015-e51e-4f86-ac94-34b05d8db2d3"). InnerVolumeSpecName "kube-api-access-2784x". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 01:04:46.243086 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:04:46.243031 4939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/19d3d3db-71fc-41f2-9976-fa8bcd10bfec-kube-api-access-2626m" pod "19d3d3db-71fc-41f2-9976-fa8bcd10bfec" (UID: "19d3d3db-71fc-41f2-9976-fa8bcd10bfec"). InnerVolumeSpecName "kube-api-access-2626m". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 01:04:46.341208 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:04:46.341168 4939 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2626m\" (UniqueName: \"kubernetes.io/projected/19d3d3db-71fc-41f2-9976-fa8bcd10bfec-kube-api-access-2626m\") on node \"ip-10-0-140-107.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:04:46.341208 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:04:46.341207 4939 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2784x\" (UniqueName: \"kubernetes.io/projected/e1d5b015-e51e-4f86-ac94-34b05d8db2d3-kube-api-access-2784x\") on node \"ip-10-0-140-107.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:04:46.413995 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:04:46.413963 4939 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w1-2729f2-openshift-default-9fb97b4fc-5ww48"] Nov 03 01:04:46.414900 ip-10-0-140-107 kubenswrapper[4939]: W1103 01:04:46.414858 4939 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcba006d1_08cd_4fac_8bdd_86931f742838.slice/crio-d66f4b4231c9bd1a18ea78d5a88c9121ffb031e3122fdd7dd4f418d7101c144a WatchSource:0}: Error finding container d66f4b4231c9bd1a18ea78d5a88c9121ffb031e3122fdd7dd4f418d7101c144a: Status 404 returned error can't find the container with id d66f4b4231c9bd1a18ea78d5a88c9121ffb031e3122fdd7dd4f418d7101c144a Nov 03 01:04:46.417432 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:04:46.417398 4939 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 01:04:46.417493 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:04:46.417468 4939 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 01:04:46.417534 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:04:46.417498 4939 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 01:04:46.450619 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:04:46.450588 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:04:46.486129 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:04:46.486104 4939 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-f7680d-openshift-default-c4488d5c9-mlblk"] Nov 03 01:04:46.486762 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:04:46.486743 4939 cpu_manager.go:414] "containerMap: removing container" podUID="19d3d3db-71fc-41f2-9976-fa8bcd10bfec" containerName="payload-pre-processing" Nov 03 01:04:46.486829 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:04:46.486786 4939 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="19d3d3db-71fc-41f2-9976-fa8bcd10bfec" containerName="payload-pre-processing" Nov 03 01:04:46.486916 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:04:46.486857 4939 cpu_manager.go:414] "containerMap: removing container" podUID="e1d5b015-e51e-4f86-ac94-34b05d8db2d3" containerName="payload-processing" Nov 03 01:04:46.486984 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:04:46.486924 4939 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e1d5b015-e51e-4f86-ac94-34b05d8db2d3" containerName="payload-processing" Nov 03 01:04:46.487216 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:04:46.487203 4939 memory_manager.go:365] "RemoveStaleState removing state" podUID="e1d5b015-e51e-4f86-ac94-34b05d8db2d3" containerName="payload-processing" Nov 03 01:04:46.487260 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:04:46.487230 4939 memory_manager.go:365] "RemoveStaleState removing state" podUID="1d33734b-7202-4184-9fc0-a58898f68802" containerName="istio-proxy" Nov 03 01:04:46.487260 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:04:46.487245 4939 memory_manager.go:365] "RemoveStaleState removing state" podUID="19d3d3db-71fc-41f2-9976-fa8bcd10bfec" containerName="payload-pre-processing" Nov 03 01:04:46.490946 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:04:46.490918 4939 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-f7680d-openshift-default-c4488d5c9-mlblk" Nov 03 01:04:46.514571 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:04:46.514538 4939 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-f7680d-openshift-default-c4488d5c9-mlblk"] Nov 03 01:04:46.657954 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:04:46.657890 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e08d7d14-b1fa-4fb2-b2d5-8c7fc073fc89-istiod-ca-cert\") pod \"e2e-worker-w2-f7680d-openshift-default-c4488d5c9-mlblk\" (UID: \"e08d7d14-b1fa-4fb2-b2d5-8c7fc073fc89\") " pod="openshift-ingress/e2e-worker-w2-f7680d-openshift-default-c4488d5c9-mlblk" Nov 03 01:04:46.658545 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:04:46.657975 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e08d7d14-b1fa-4fb2-b2d5-8c7fc073fc89-istio-podinfo\") pod \"e2e-worker-w2-f7680d-openshift-default-c4488d5c9-mlblk\" (UID: \"e08d7d14-b1fa-4fb2-b2d5-8c7fc073fc89\") " pod="openshift-ingress/e2e-worker-w2-f7680d-openshift-default-c4488d5c9-mlblk" Nov 03 01:04:46.658545 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:04:46.658037 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e08d7d14-b1fa-4fb2-b2d5-8c7fc073fc89-workload-socket\") pod \"e2e-worker-w2-f7680d-openshift-default-c4488d5c9-mlblk\" (UID: \"e08d7d14-b1fa-4fb2-b2d5-8c7fc073fc89\") " pod="openshift-ingress/e2e-worker-w2-f7680d-openshift-default-c4488d5c9-mlblk" Nov 03 01:04:46.658545 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:04:46.658131 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e08d7d14-b1fa-4fb2-b2d5-8c7fc073fc89-istio-data\") pod \"e2e-worker-w2-f7680d-openshift-default-c4488d5c9-mlblk\" (UID: \"e08d7d14-b1fa-4fb2-b2d5-8c7fc073fc89\") " pod="openshift-ingress/e2e-worker-w2-f7680d-openshift-default-c4488d5c9-mlblk" Nov 03 01:04:46.658545 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:04:46.658188 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e08d7d14-b1fa-4fb2-b2d5-8c7fc073fc89-istio-token\") pod \"e2e-worker-w2-f7680d-openshift-default-c4488d5c9-mlblk\" (UID: \"e08d7d14-b1fa-4fb2-b2d5-8c7fc073fc89\") " pod="openshift-ingress/e2e-worker-w2-f7680d-openshift-default-c4488d5c9-mlblk" Nov 03 01:04:46.658545 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:04:46.658217 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e08d7d14-b1fa-4fb2-b2d5-8c7fc073fc89-istio-envoy\") pod \"e2e-worker-w2-f7680d-openshift-default-c4488d5c9-mlblk\" (UID: \"e08d7d14-b1fa-4fb2-b2d5-8c7fc073fc89\") " pod="openshift-ingress/e2e-worker-w2-f7680d-openshift-default-c4488d5c9-mlblk" Nov 03 01:04:46.658545 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:04:46.658261 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e08d7d14-b1fa-4fb2-b2d5-8c7fc073fc89-workload-certs\") pod \"e2e-worker-w2-f7680d-openshift-default-c4488d5c9-mlblk\" (UID: \"e08d7d14-b1fa-4fb2-b2d5-8c7fc073fc89\") " pod="openshift-ingress/e2e-worker-w2-f7680d-openshift-default-c4488d5c9-mlblk" Nov 03 01:04:46.658545 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:04:46.658350 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-htxhs\" (UniqueName: \"kubernetes.io/projected/e08d7d14-b1fa-4fb2-b2d5-8c7fc073fc89-kube-api-access-htxhs\") pod \"e2e-worker-w2-f7680d-openshift-default-c4488d5c9-mlblk\" (UID: \"e08d7d14-b1fa-4fb2-b2d5-8c7fc073fc89\") " pod="openshift-ingress/e2e-worker-w2-f7680d-openshift-default-c4488d5c9-mlblk" Nov 03 01:04:46.658545 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:04:46.658414 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e08d7d14-b1fa-4fb2-b2d5-8c7fc073fc89-credential-socket\") pod \"e2e-worker-w2-f7680d-openshift-default-c4488d5c9-mlblk\" (UID: \"e08d7d14-b1fa-4fb2-b2d5-8c7fc073fc89\") " pod="openshift-ingress/e2e-worker-w2-f7680d-openshift-default-c4488d5c9-mlblk" Nov 03 01:04:46.752598 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:04:46.752505 4939 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-208a0d-f9cc48655-wfc6q" Nov 03 01:04:46.752598 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:04:46.752537 4939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-208a0d-f9cc48655-wfc6q" event={"ID":"19d3d3db-71fc-41f2-9976-fa8bcd10bfec","Type":"ContainerDied","Data":"117451719c7dd4f9dc063a2c959d57244c6357b967191382d3e9857ce56e319c"} Nov 03 01:04:46.752839 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:04:46.752613 4939 scope.go:122] "RemoveContainer" containerID="f6e857800bf7fcec22a2bf93eebc4b9237f4871fe54a63775de09f4823584346" Nov 03 01:04:46.754843 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:04:46.754820 4939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-b-208a0d-6f7c665d67-pc68m" event={"ID":"e1d5b015-e51e-4f86-ac94-34b05d8db2d3","Type":"ContainerDied","Data":"d060e6ec87a45a8b3e8b62b4e645ca0bd618b90b84213b9ab12cb587f4d162c6"} Nov 03 01:04:46.754999 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:04:46.754903 4939 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-b-208a0d-6f7c665d67-pc68m" Nov 03 01:04:46.760109 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:04:46.759157 4939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w1-2729f2-openshift-default-9fb97b4fc-5ww48" event={"ID":"cba006d1-08cd-4fac-8bdd-86931f742838","Type":"ContainerStarted","Data":"c7884d3f229a4cf4c41b83c524ce2da09be6b780b239aa5ea0bdd93e1ac60024"} Nov 03 01:04:46.760109 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:04:46.759199 4939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w1-2729f2-openshift-default-9fb97b4fc-5ww48" event={"ID":"cba006d1-08cd-4fac-8bdd-86931f742838","Type":"ContainerStarted","Data":"d66f4b4231c9bd1a18ea78d5a88c9121ffb031e3122fdd7dd4f418d7101c144a"} Nov 03 01:04:46.760109 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:04:46.759529 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e08d7d14-b1fa-4fb2-b2d5-8c7fc073fc89-istio-data\") pod \"e2e-worker-w2-f7680d-openshift-default-c4488d5c9-mlblk\" (UID: \"e08d7d14-b1fa-4fb2-b2d5-8c7fc073fc89\") " pod="openshift-ingress/e2e-worker-w2-f7680d-openshift-default-c4488d5c9-mlblk" Nov 03 01:04:46.760109 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:04:46.759589 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e08d7d14-b1fa-4fb2-b2d5-8c7fc073fc89-istio-token\") pod \"e2e-worker-w2-f7680d-openshift-default-c4488d5c9-mlblk\" (UID: \"e08d7d14-b1fa-4fb2-b2d5-8c7fc073fc89\") " pod="openshift-ingress/e2e-worker-w2-f7680d-openshift-default-c4488d5c9-mlblk" Nov 03 01:04:46.760109 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:04:46.759615 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e08d7d14-b1fa-4fb2-b2d5-8c7fc073fc89-istio-envoy\") pod \"e2e-worker-w2-f7680d-openshift-default-c4488d5c9-mlblk\" (UID: \"e08d7d14-b1fa-4fb2-b2d5-8c7fc073fc89\") " pod="openshift-ingress/e2e-worker-w2-f7680d-openshift-default-c4488d5c9-mlblk" Nov 03 01:04:46.760109 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:04:46.759652 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e08d7d14-b1fa-4fb2-b2d5-8c7fc073fc89-workload-certs\") pod \"e2e-worker-w2-f7680d-openshift-default-c4488d5c9-mlblk\" (UID: \"e08d7d14-b1fa-4fb2-b2d5-8c7fc073fc89\") " pod="openshift-ingress/e2e-worker-w2-f7680d-openshift-default-c4488d5c9-mlblk" Nov 03 01:04:46.760109 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:04:46.759720 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-htxhs\" (UniqueName: \"kubernetes.io/projected/e08d7d14-b1fa-4fb2-b2d5-8c7fc073fc89-kube-api-access-htxhs\") pod \"e2e-worker-w2-f7680d-openshift-default-c4488d5c9-mlblk\" (UID: \"e08d7d14-b1fa-4fb2-b2d5-8c7fc073fc89\") " pod="openshift-ingress/e2e-worker-w2-f7680d-openshift-default-c4488d5c9-mlblk" Nov 03 01:04:46.760109 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:04:46.759772 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e08d7d14-b1fa-4fb2-b2d5-8c7fc073fc89-credential-socket\") pod \"e2e-worker-w2-f7680d-openshift-default-c4488d5c9-mlblk\" (UID: \"e08d7d14-b1fa-4fb2-b2d5-8c7fc073fc89\") " pod="openshift-ingress/e2e-worker-w2-f7680d-openshift-default-c4488d5c9-mlblk" Nov 03 01:04:46.760109 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:04:46.759874 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e08d7d14-b1fa-4fb2-b2d5-8c7fc073fc89-istiod-ca-cert\") pod \"e2e-worker-w2-f7680d-openshift-default-c4488d5c9-mlblk\" (UID: \"e08d7d14-b1fa-4fb2-b2d5-8c7fc073fc89\") " pod="openshift-ingress/e2e-worker-w2-f7680d-openshift-default-c4488d5c9-mlblk" Nov 03 01:04:46.760109 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:04:46.759924 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e08d7d14-b1fa-4fb2-b2d5-8c7fc073fc89-istio-podinfo\") pod \"e2e-worker-w2-f7680d-openshift-default-c4488d5c9-mlblk\" (UID: \"e08d7d14-b1fa-4fb2-b2d5-8c7fc073fc89\") " pod="openshift-ingress/e2e-worker-w2-f7680d-openshift-default-c4488d5c9-mlblk" Nov 03 01:04:46.760109 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:04:46.759963 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e08d7d14-b1fa-4fb2-b2d5-8c7fc073fc89-istio-data\") pod \"e2e-worker-w2-f7680d-openshift-default-c4488d5c9-mlblk\" (UID: \"e08d7d14-b1fa-4fb2-b2d5-8c7fc073fc89\") " pod="openshift-ingress/e2e-worker-w2-f7680d-openshift-default-c4488d5c9-mlblk" Nov 03 01:04:46.760109 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:04:46.759981 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e08d7d14-b1fa-4fb2-b2d5-8c7fc073fc89-workload-socket\") pod \"e2e-worker-w2-f7680d-openshift-default-c4488d5c9-mlblk\" (UID: \"e08d7d14-b1fa-4fb2-b2d5-8c7fc073fc89\") " pod="openshift-ingress/e2e-worker-w2-f7680d-openshift-default-c4488d5c9-mlblk" Nov 03 01:04:46.760939 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:04:46.760270 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e08d7d14-b1fa-4fb2-b2d5-8c7fc073fc89-workload-certs\") pod \"e2e-worker-w2-f7680d-openshift-default-c4488d5c9-mlblk\" (UID: \"e08d7d14-b1fa-4fb2-b2d5-8c7fc073fc89\") " pod="openshift-ingress/e2e-worker-w2-f7680d-openshift-default-c4488d5c9-mlblk" Nov 03 01:04:46.760939 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:04:46.760351 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e08d7d14-b1fa-4fb2-b2d5-8c7fc073fc89-workload-socket\") pod \"e2e-worker-w2-f7680d-openshift-default-c4488d5c9-mlblk\" (UID: \"e08d7d14-b1fa-4fb2-b2d5-8c7fc073fc89\") " pod="openshift-ingress/e2e-worker-w2-f7680d-openshift-default-c4488d5c9-mlblk" Nov 03 01:04:46.761072 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:04:46.761054 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e08d7d14-b1fa-4fb2-b2d5-8c7fc073fc89-istiod-ca-cert\") pod \"e2e-worker-w2-f7680d-openshift-default-c4488d5c9-mlblk\" (UID: \"e08d7d14-b1fa-4fb2-b2d5-8c7fc073fc89\") " pod="openshift-ingress/e2e-worker-w2-f7680d-openshift-default-c4488d5c9-mlblk" Nov 03 01:04:46.761374 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:04:46.761349 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e08d7d14-b1fa-4fb2-b2d5-8c7fc073fc89-credential-socket\") pod \"e2e-worker-w2-f7680d-openshift-default-c4488d5c9-mlblk\" (UID: \"e08d7d14-b1fa-4fb2-b2d5-8c7fc073fc89\") " pod="openshift-ingress/e2e-worker-w2-f7680d-openshift-default-c4488d5c9-mlblk" Nov 03 01:04:46.762876 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:04:46.762841 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e08d7d14-b1fa-4fb2-b2d5-8c7fc073fc89-istio-envoy\") pod \"e2e-worker-w2-f7680d-openshift-default-c4488d5c9-mlblk\" (UID: \"e08d7d14-b1fa-4fb2-b2d5-8c7fc073fc89\") " pod="openshift-ingress/e2e-worker-w2-f7680d-openshift-default-c4488d5c9-mlblk" Nov 03 01:04:46.763105 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:04:46.763085 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e08d7d14-b1fa-4fb2-b2d5-8c7fc073fc89-istio-podinfo\") pod \"e2e-worker-w2-f7680d-openshift-default-c4488d5c9-mlblk\" (UID: \"e08d7d14-b1fa-4fb2-b2d5-8c7fc073fc89\") " pod="openshift-ingress/e2e-worker-w2-f7680d-openshift-default-c4488d5c9-mlblk" Nov 03 01:04:46.774085 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:04:46.774057 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-htxhs\" (UniqueName: \"kubernetes.io/projected/e08d7d14-b1fa-4fb2-b2d5-8c7fc073fc89-kube-api-access-htxhs\") pod \"e2e-worker-w2-f7680d-openshift-default-c4488d5c9-mlblk\" (UID: \"e08d7d14-b1fa-4fb2-b2d5-8c7fc073fc89\") " pod="openshift-ingress/e2e-worker-w2-f7680d-openshift-default-c4488d5c9-mlblk" Nov 03 01:04:46.777699 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:04:46.777648 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e08d7d14-b1fa-4fb2-b2d5-8c7fc073fc89-istio-token\") pod \"e2e-worker-w2-f7680d-openshift-default-c4488d5c9-mlblk\" (UID: \"e08d7d14-b1fa-4fb2-b2d5-8c7fc073fc89\") " pod="openshift-ingress/e2e-worker-w2-f7680d-openshift-default-c4488d5c9-mlblk" Nov 03 01:04:46.778727 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:04:46.778711 4939 scope.go:122] "RemoveContainer" containerID="294acea636bb9e10e64a9e3ebf26fe4257515f5bc8a787135e5ab2de4de46f16" Nov 03 01:04:46.792436 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:04:46.792410 4939 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w4-6c0ec0-openshift-default-79c7584bfc-8hz7h"] Nov 03 01:04:46.798544 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:04:46.798519 4939 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w4-6c0ec0-openshift-default-79c7584bfc-8hz7h" Nov 03 01:04:46.803591 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:04:46.803534 4939 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w1-2729f2-openshift-default-9fb97b4fc-5ww48" podStartSLOduration=2.803516121 podStartE2EDuration="2.803516121s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 01:04:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 01:04:46.801773676 +0000 UTC m=+1435.066108623" watchObservedRunningTime="2026-11-03 01:04:46.803516121 +0000 UTC m=+1435.067851066" Nov 03 01:04:46.806250 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:04:46.806217 4939 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-w4-6c0ec0-openshift-default-dockercfg-dgkf8\"" Nov 03 01:04:46.809460 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:04:46.809282 4939 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-f7680d-openshift-default-c4488d5c9-mlblk" Nov 03 01:04:46.817268 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:04:46.817155 4939 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w4-6c0ec0-openshift-default-79c7584bfc-8hz7h"] Nov 03 01:04:46.851957 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:04:46.850946 4939 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-208a0d-f9cc48655-wfc6q"] Nov 03 01:04:46.856700 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:04:46.856668 4939 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-208a0d-f9cc48655-wfc6q"] Nov 03 01:04:46.871313 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:04:46.871282 4939 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-208a0d-6f7c665d67-pc68m"] Nov 03 01:04:46.884391 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:04:46.884360 4939 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-83755a-openshift-default-c86f9565c-z7flk"] Nov 03 01:04:46.906352 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:04:46.904751 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:04:46.913427 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:04:46.913399 4939 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-83755a-openshift-default-c86f9565c-z7flk" Nov 03 01:04:46.914234 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:04:46.913298 4939 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-208a0d-6f7c665d67-pc68m"] Nov 03 01:04:46.914339 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:04:46.914313 4939 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-83755a-openshift-default-c86f9565c-zj5jg"] Nov 03 01:04:46.922710 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:04:46.922688 4939 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-83755a-openshift-default-c86f9565c-z7flk"] Nov 03 01:04:46.922848 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:04:46.922837 4939 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-83755a-openshift-default-c86f9565c-zj5jg"] Nov 03 01:04:46.922896 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:04:46.922787 4939 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-83755a-openshift-default-c86f9565c-zj5jg" Nov 03 01:04:46.989945 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:04:46.981877 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0a1fe883-48d5-4d06-ab8f-41698965dbbb-istio-token\") pod \"e2e-shared-a-w4-6c0ec0-openshift-default-79c7584bfc-8hz7h\" (UID: \"0a1fe883-48d5-4d06-ab8f-41698965dbbb\") " pod="openshift-ingress/e2e-shared-a-w4-6c0ec0-openshift-default-79c7584bfc-8hz7h" Nov 03 01:04:46.989945 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:04:46.981963 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0a1fe883-48d5-4d06-ab8f-41698965dbbb-istiod-ca-cert\") pod \"e2e-shared-a-w4-6c0ec0-openshift-default-79c7584bfc-8hz7h\" (UID: \"0a1fe883-48d5-4d06-ab8f-41698965dbbb\") " pod="openshift-ingress/e2e-shared-a-w4-6c0ec0-openshift-default-79c7584bfc-8hz7h" Nov 03 01:04:46.989945 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:04:46.981995 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2d64v\" (UniqueName: \"kubernetes.io/projected/0a1fe883-48d5-4d06-ab8f-41698965dbbb-kube-api-access-2d64v\") pod \"e2e-shared-a-w4-6c0ec0-openshift-default-79c7584bfc-8hz7h\" (UID: \"0a1fe883-48d5-4d06-ab8f-41698965dbbb\") " pod="openshift-ingress/e2e-shared-a-w4-6c0ec0-openshift-default-79c7584bfc-8hz7h" Nov 03 01:04:46.989945 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:04:46.982051 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0a1fe883-48d5-4d06-ab8f-41698965dbbb-istio-envoy\") pod \"e2e-shared-a-w4-6c0ec0-openshift-default-79c7584bfc-8hz7h\" (UID: \"0a1fe883-48d5-4d06-ab8f-41698965dbbb\") " pod="openshift-ingress/e2e-shared-a-w4-6c0ec0-openshift-default-79c7584bfc-8hz7h" Nov 03 01:04:46.989945 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:04:46.982133 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0a1fe883-48d5-4d06-ab8f-41698965dbbb-istio-podinfo\") pod \"e2e-shared-a-w4-6c0ec0-openshift-default-79c7584bfc-8hz7h\" (UID: \"0a1fe883-48d5-4d06-ab8f-41698965dbbb\") " pod="openshift-ingress/e2e-shared-a-w4-6c0ec0-openshift-default-79c7584bfc-8hz7h" Nov 03 01:04:46.989945 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:04:46.982255 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0a1fe883-48d5-4d06-ab8f-41698965dbbb-istio-data\") pod \"e2e-shared-a-w4-6c0ec0-openshift-default-79c7584bfc-8hz7h\" (UID: \"0a1fe883-48d5-4d06-ab8f-41698965dbbb\") " pod="openshift-ingress/e2e-shared-a-w4-6c0ec0-openshift-default-79c7584bfc-8hz7h" Nov 03 01:04:46.989945 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:04:46.982343 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0a1fe883-48d5-4d06-ab8f-41698965dbbb-workload-socket\") pod \"e2e-shared-a-w4-6c0ec0-openshift-default-79c7584bfc-8hz7h\" (UID: \"0a1fe883-48d5-4d06-ab8f-41698965dbbb\") " pod="openshift-ingress/e2e-shared-a-w4-6c0ec0-openshift-default-79c7584bfc-8hz7h" Nov 03 01:04:46.989945 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:04:46.982384 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0a1fe883-48d5-4d06-ab8f-41698965dbbb-workload-certs\") pod \"e2e-shared-a-w4-6c0ec0-openshift-default-79c7584bfc-8hz7h\" (UID: \"0a1fe883-48d5-4d06-ab8f-41698965dbbb\") " pod="openshift-ingress/e2e-shared-a-w4-6c0ec0-openshift-default-79c7584bfc-8hz7h" Nov 03 01:04:46.989945 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:04:46.982654 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0a1fe883-48d5-4d06-ab8f-41698965dbbb-credential-socket\") pod \"e2e-shared-a-w4-6c0ec0-openshift-default-79c7584bfc-8hz7h\" (UID: \"0a1fe883-48d5-4d06-ab8f-41698965dbbb\") " pod="openshift-ingress/e2e-shared-a-w4-6c0ec0-openshift-default-79c7584bfc-8hz7h" Nov 03 01:04:47.051984 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:04:47.051060 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:04:47.064934 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:04:47.063031 4939 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w1-2729f2-openshift-default-9fb97b4fc-5ww48" Nov 03 01:04:47.097928 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:04:47.088792 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/569c2920-1523-4da8-8644-f789aebcdc7a-istio-envoy\") pod \"e2e-worker-w3-83755a-openshift-default-c86f9565c-z7flk\" (UID: \"569c2920-1523-4da8-8644-f789aebcdc7a\") " pod="openshift-ingress/e2e-worker-w3-83755a-openshift-default-c86f9565c-z7flk" Nov 03 01:04:47.097928 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:04:47.088868 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0a1fe883-48d5-4d06-ab8f-41698965dbbb-workload-socket\") pod \"e2e-shared-a-w4-6c0ec0-openshift-default-79c7584bfc-8hz7h\" (UID: \"0a1fe883-48d5-4d06-ab8f-41698965dbbb\") " pod="openshift-ingress/e2e-shared-a-w4-6c0ec0-openshift-default-79c7584bfc-8hz7h" Nov 03 01:04:47.097928 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:04:47.088932 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0a1fe883-48d5-4d06-ab8f-41698965dbbb-workload-certs\") pod \"e2e-shared-a-w4-6c0ec0-openshift-default-79c7584bfc-8hz7h\" (UID: \"0a1fe883-48d5-4d06-ab8f-41698965dbbb\") " pod="openshift-ingress/e2e-shared-a-w4-6c0ec0-openshift-default-79c7584bfc-8hz7h" Nov 03 01:04:47.097928 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:04:47.089027 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0dbecc93-5a61-4100-a319-d3f673f7a5fd-credential-socket\") pod \"e2e-worker-w3-83755a-openshift-default-c86f9565c-zj5jg\" (UID: \"0dbecc93-5a61-4100-a319-d3f673f7a5fd\") " pod="openshift-ingress/e2e-worker-w3-83755a-openshift-default-c86f9565c-zj5jg" Nov 03 01:04:47.097928 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:04:47.089063 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0a1fe883-48d5-4d06-ab8f-41698965dbbb-credential-socket\") pod \"e2e-shared-a-w4-6c0ec0-openshift-default-79c7584bfc-8hz7h\" (UID: \"0a1fe883-48d5-4d06-ab8f-41698965dbbb\") " pod="openshift-ingress/e2e-shared-a-w4-6c0ec0-openshift-default-79c7584bfc-8hz7h" Nov 03 01:04:47.097928 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:04:47.089122 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0dbecc93-5a61-4100-a319-d3f673f7a5fd-istio-podinfo\") pod \"e2e-worker-w3-83755a-openshift-default-c86f9565c-zj5jg\" (UID: \"0dbecc93-5a61-4100-a319-d3f673f7a5fd\") " pod="openshift-ingress/e2e-worker-w3-83755a-openshift-default-c86f9565c-zj5jg" Nov 03 01:04:47.097928 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:04:47.089155 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/569c2920-1523-4da8-8644-f789aebcdc7a-istiod-ca-cert\") pod \"e2e-worker-w3-83755a-openshift-default-c86f9565c-z7flk\" (UID: \"569c2920-1523-4da8-8644-f789aebcdc7a\") " pod="openshift-ingress/e2e-worker-w3-83755a-openshift-default-c86f9565c-z7flk" Nov 03 01:04:47.097928 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:04:47.089183 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f4rxn\" (UniqueName: \"kubernetes.io/projected/0dbecc93-5a61-4100-a319-d3f673f7a5fd-kube-api-access-f4rxn\") pod \"e2e-worker-w3-83755a-openshift-default-c86f9565c-zj5jg\" (UID: \"0dbecc93-5a61-4100-a319-d3f673f7a5fd\") " pod="openshift-ingress/e2e-worker-w3-83755a-openshift-default-c86f9565c-zj5jg" Nov 03 01:04:47.097928 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:04:47.089216 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0dbecc93-5a61-4100-a319-d3f673f7a5fd-istiod-ca-cert\") pod \"e2e-worker-w3-83755a-openshift-default-c86f9565c-zj5jg\" (UID: \"0dbecc93-5a61-4100-a319-d3f673f7a5fd\") " pod="openshift-ingress/e2e-worker-w3-83755a-openshift-default-c86f9565c-zj5jg" Nov 03 01:04:47.097928 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:04:47.089264 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0a1fe883-48d5-4d06-ab8f-41698965dbbb-istio-token\") pod \"e2e-shared-a-w4-6c0ec0-openshift-default-79c7584bfc-8hz7h\" (UID: \"0a1fe883-48d5-4d06-ab8f-41698965dbbb\") " pod="openshift-ingress/e2e-shared-a-w4-6c0ec0-openshift-default-79c7584bfc-8hz7h" Nov 03 01:04:47.097928 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:04:47.089349 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0a1fe883-48d5-4d06-ab8f-41698965dbbb-workload-certs\") pod \"e2e-shared-a-w4-6c0ec0-openshift-default-79c7584bfc-8hz7h\" (UID: \"0a1fe883-48d5-4d06-ab8f-41698965dbbb\") " pod="openshift-ingress/e2e-shared-a-w4-6c0ec0-openshift-default-79c7584bfc-8hz7h" Nov 03 01:04:47.097928 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:04:47.089398 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/569c2920-1523-4da8-8644-f789aebcdc7a-istio-podinfo\") pod \"e2e-worker-w3-83755a-openshift-default-c86f9565c-z7flk\" (UID: \"569c2920-1523-4da8-8644-f789aebcdc7a\") " pod="openshift-ingress/e2e-worker-w3-83755a-openshift-default-c86f9565c-z7flk" Nov 03 01:04:47.097928 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:04:47.089452 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0a1fe883-48d5-4d06-ab8f-41698965dbbb-istiod-ca-cert\") pod \"e2e-shared-a-w4-6c0ec0-openshift-default-79c7584bfc-8hz7h\" (UID: \"0a1fe883-48d5-4d06-ab8f-41698965dbbb\") " pod="openshift-ingress/e2e-shared-a-w4-6c0ec0-openshift-default-79c7584bfc-8hz7h" Nov 03 01:04:47.097928 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:04:47.089487 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2d64v\" (UniqueName: \"kubernetes.io/projected/0a1fe883-48d5-4d06-ab8f-41698965dbbb-kube-api-access-2d64v\") pod \"e2e-shared-a-w4-6c0ec0-openshift-default-79c7584bfc-8hz7h\" (UID: \"0a1fe883-48d5-4d06-ab8f-41698965dbbb\") " pod="openshift-ingress/e2e-shared-a-w4-6c0ec0-openshift-default-79c7584bfc-8hz7h" Nov 03 01:04:47.097928 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:04:47.089514 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0dbecc93-5a61-4100-a319-d3f673f7a5fd-istio-token\") pod \"e2e-worker-w3-83755a-openshift-default-c86f9565c-zj5jg\" (UID: \"0dbecc93-5a61-4100-a319-d3f673f7a5fd\") " pod="openshift-ingress/e2e-worker-w3-83755a-openshift-default-c86f9565c-zj5jg" Nov 03 01:04:47.097928 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:04:47.089570 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0dbecc93-5a61-4100-a319-d3f673f7a5fd-istio-envoy\") pod \"e2e-worker-w3-83755a-openshift-default-c86f9565c-zj5jg\" (UID: \"0dbecc93-5a61-4100-a319-d3f673f7a5fd\") " pod="openshift-ingress/e2e-worker-w3-83755a-openshift-default-c86f9565c-zj5jg" Nov 03 01:04:47.097928 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:04:47.089602 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0a1fe883-48d5-4d06-ab8f-41698965dbbb-istio-envoy\") pod \"e2e-shared-a-w4-6c0ec0-openshift-default-79c7584bfc-8hz7h\" (UID: \"0a1fe883-48d5-4d06-ab8f-41698965dbbb\") " pod="openshift-ingress/e2e-shared-a-w4-6c0ec0-openshift-default-79c7584bfc-8hz7h" Nov 03 01:04:47.097928 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:04:47.089645 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0a1fe883-48d5-4d06-ab8f-41698965dbbb-credential-socket\") pod \"e2e-shared-a-w4-6c0ec0-openshift-default-79c7584bfc-8hz7h\" (UID: \"0a1fe883-48d5-4d06-ab8f-41698965dbbb\") " pod="openshift-ingress/e2e-shared-a-w4-6c0ec0-openshift-default-79c7584bfc-8hz7h" Nov 03 01:04:47.097928 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:04:47.089651 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0dbecc93-5a61-4100-a319-d3f673f7a5fd-istio-data\") pod \"e2e-worker-w3-83755a-openshift-default-c86f9565c-zj5jg\" (UID: \"0dbecc93-5a61-4100-a319-d3f673f7a5fd\") " pod="openshift-ingress/e2e-worker-w3-83755a-openshift-default-c86f9565c-zj5jg" Nov 03 01:04:47.097928 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:04:47.090313 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0a1fe883-48d5-4d06-ab8f-41698965dbbb-istiod-ca-cert\") pod \"e2e-shared-a-w4-6c0ec0-openshift-default-79c7584bfc-8hz7h\" (UID: \"0a1fe883-48d5-4d06-ab8f-41698965dbbb\") " pod="openshift-ingress/e2e-shared-a-w4-6c0ec0-openshift-default-79c7584bfc-8hz7h" Nov 03 01:04:47.097928 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:04:47.090751 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/569c2920-1523-4da8-8644-f789aebcdc7a-workload-socket\") pod \"e2e-worker-w3-83755a-openshift-default-c86f9565c-z7flk\" (UID: \"569c2920-1523-4da8-8644-f789aebcdc7a\") " pod="openshift-ingress/e2e-worker-w3-83755a-openshift-default-c86f9565c-z7flk" Nov 03 01:04:47.097928 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:04:47.090837 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0dbecc93-5a61-4100-a319-d3f673f7a5fd-workload-certs\") pod \"e2e-worker-w3-83755a-openshift-default-c86f9565c-zj5jg\" (UID: \"0dbecc93-5a61-4100-a319-d3f673f7a5fd\") " pod="openshift-ingress/e2e-worker-w3-83755a-openshift-default-c86f9565c-zj5jg" Nov 03 01:04:47.097928 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:04:47.090894 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/569c2920-1523-4da8-8644-f789aebcdc7a-istio-token\") pod \"e2e-worker-w3-83755a-openshift-default-c86f9565c-z7flk\" (UID: \"569c2920-1523-4da8-8644-f789aebcdc7a\") " pod="openshift-ingress/e2e-worker-w3-83755a-openshift-default-c86f9565c-z7flk" Nov 03 01:04:47.097928 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:04:47.090935 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0dbecc93-5a61-4100-a319-d3f673f7a5fd-workload-socket\") pod \"e2e-worker-w3-83755a-openshift-default-c86f9565c-zj5jg\" (UID: \"0dbecc93-5a61-4100-a319-d3f673f7a5fd\") " pod="openshift-ingress/e2e-worker-w3-83755a-openshift-default-c86f9565c-zj5jg" Nov 03 01:04:47.097928 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:04:47.090964 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zgqkb\" (UniqueName: \"kubernetes.io/projected/569c2920-1523-4da8-8644-f789aebcdc7a-kube-api-access-zgqkb\") pod \"e2e-worker-w3-83755a-openshift-default-c86f9565c-z7flk\" (UID: \"569c2920-1523-4da8-8644-f789aebcdc7a\") " pod="openshift-ingress/e2e-worker-w3-83755a-openshift-default-c86f9565c-z7flk" Nov 03 01:04:47.097928 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:04:47.091012 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0a1fe883-48d5-4d06-ab8f-41698965dbbb-istio-podinfo\") pod \"e2e-shared-a-w4-6c0ec0-openshift-default-79c7584bfc-8hz7h\" (UID: \"0a1fe883-48d5-4d06-ab8f-41698965dbbb\") " pod="openshift-ingress/e2e-shared-a-w4-6c0ec0-openshift-default-79c7584bfc-8hz7h" Nov 03 01:04:47.097928 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:04:47.093081 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/569c2920-1523-4da8-8644-f789aebcdc7a-workload-certs\") pod \"e2e-worker-w3-83755a-openshift-default-c86f9565c-z7flk\" (UID: \"569c2920-1523-4da8-8644-f789aebcdc7a\") " pod="openshift-ingress/e2e-worker-w3-83755a-openshift-default-c86f9565c-z7flk" Nov 03 01:04:47.097928 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:04:47.093151 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/569c2920-1523-4da8-8644-f789aebcdc7a-credential-socket\") pod \"e2e-worker-w3-83755a-openshift-default-c86f9565c-z7flk\" (UID: \"569c2920-1523-4da8-8644-f789aebcdc7a\") " pod="openshift-ingress/e2e-worker-w3-83755a-openshift-default-c86f9565c-z7flk" Nov 03 01:04:47.097928 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:04:47.093285 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0a1fe883-48d5-4d06-ab8f-41698965dbbb-istio-data\") pod \"e2e-shared-a-w4-6c0ec0-openshift-default-79c7584bfc-8hz7h\" (UID: \"0a1fe883-48d5-4d06-ab8f-41698965dbbb\") " pod="openshift-ingress/e2e-shared-a-w4-6c0ec0-openshift-default-79c7584bfc-8hz7h" Nov 03 01:04:47.097928 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:04:47.093323 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/569c2920-1523-4da8-8644-f789aebcdc7a-istio-data\") pod \"e2e-worker-w3-83755a-openshift-default-c86f9565c-z7flk\" (UID: \"569c2920-1523-4da8-8644-f789aebcdc7a\") " pod="openshift-ingress/e2e-worker-w3-83755a-openshift-default-c86f9565c-z7flk" Nov 03 01:04:47.097928 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:04:47.093683 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0a1fe883-48d5-4d06-ab8f-41698965dbbb-istio-data\") pod \"e2e-shared-a-w4-6c0ec0-openshift-default-79c7584bfc-8hz7h\" (UID: \"0a1fe883-48d5-4d06-ab8f-41698965dbbb\") " pod="openshift-ingress/e2e-shared-a-w4-6c0ec0-openshift-default-79c7584bfc-8hz7h" Nov 03 01:04:47.097928 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:04:47.093896 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0a1fe883-48d5-4d06-ab8f-41698965dbbb-workload-socket\") pod \"e2e-shared-a-w4-6c0ec0-openshift-default-79c7584bfc-8hz7h\" (UID: \"0a1fe883-48d5-4d06-ab8f-41698965dbbb\") " pod="openshift-ingress/e2e-shared-a-w4-6c0ec0-openshift-default-79c7584bfc-8hz7h" Nov 03 01:04:47.117801 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:04:47.117733 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2d64v\" (UniqueName: \"kubernetes.io/projected/0a1fe883-48d5-4d06-ab8f-41698965dbbb-kube-api-access-2d64v\") pod \"e2e-shared-a-w4-6c0ec0-openshift-default-79c7584bfc-8hz7h\" (UID: \"0a1fe883-48d5-4d06-ab8f-41698965dbbb\") " pod="openshift-ingress/e2e-shared-a-w4-6c0ec0-openshift-default-79c7584bfc-8hz7h" Nov 03 01:04:47.130541 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:04:47.130470 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0a1fe883-48d5-4d06-ab8f-41698965dbbb-istio-envoy\") pod \"e2e-shared-a-w4-6c0ec0-openshift-default-79c7584bfc-8hz7h\" (UID: \"0a1fe883-48d5-4d06-ab8f-41698965dbbb\") " pod="openshift-ingress/e2e-shared-a-w4-6c0ec0-openshift-default-79c7584bfc-8hz7h" Nov 03 01:04:47.130945 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:04:47.130866 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0a1fe883-48d5-4d06-ab8f-41698965dbbb-istio-podinfo\") pod \"e2e-shared-a-w4-6c0ec0-openshift-default-79c7584bfc-8hz7h\" (UID: \"0a1fe883-48d5-4d06-ab8f-41698965dbbb\") " pod="openshift-ingress/e2e-shared-a-w4-6c0ec0-openshift-default-79c7584bfc-8hz7h" Nov 03 01:04:47.143544 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:04:47.143474 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0a1fe883-48d5-4d06-ab8f-41698965dbbb-istio-token\") pod \"e2e-shared-a-w4-6c0ec0-openshift-default-79c7584bfc-8hz7h\" (UID: \"0a1fe883-48d5-4d06-ab8f-41698965dbbb\") " pod="openshift-ingress/e2e-shared-a-w4-6c0ec0-openshift-default-79c7584bfc-8hz7h" Nov 03 01:04:47.168304 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:04:47.166484 4939 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w1-2729f2-openshift-default-9fb97b4fc-5ww48" Nov 03 01:04:47.188509 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:04:47.188474 4939 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-f7680d-openshift-default-c4488d5c9-mlblk"] Nov 03 01:04:47.213949 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:04:47.208417 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:04:47.213949 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:04:47.209050 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/569c2920-1523-4da8-8644-f789aebcdc7a-workload-certs\") pod \"e2e-worker-w3-83755a-openshift-default-c86f9565c-z7flk\" (UID: \"569c2920-1523-4da8-8644-f789aebcdc7a\") " pod="openshift-ingress/e2e-worker-w3-83755a-openshift-default-c86f9565c-z7flk" Nov 03 01:04:47.213949 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:04:47.209744 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/569c2920-1523-4da8-8644-f789aebcdc7a-credential-socket\") pod \"e2e-worker-w3-83755a-openshift-default-c86f9565c-z7flk\" (UID: \"569c2920-1523-4da8-8644-f789aebcdc7a\") " pod="openshift-ingress/e2e-worker-w3-83755a-openshift-default-c86f9565c-z7flk" Nov 03 01:04:47.213949 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:04:47.209927 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/569c2920-1523-4da8-8644-f789aebcdc7a-istio-data\") pod \"e2e-worker-w3-83755a-openshift-default-c86f9565c-z7flk\" (UID: \"569c2920-1523-4da8-8644-f789aebcdc7a\") " pod="openshift-ingress/e2e-worker-w3-83755a-openshift-default-c86f9565c-z7flk" Nov 03 01:04:47.213949 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:04:47.209979 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/569c2920-1523-4da8-8644-f789aebcdc7a-istio-envoy\") pod \"e2e-worker-w3-83755a-openshift-default-c86f9565c-z7flk\" (UID: \"569c2920-1523-4da8-8644-f789aebcdc7a\") " pod="openshift-ingress/e2e-worker-w3-83755a-openshift-default-c86f9565c-z7flk" Nov 03 01:04:47.213949 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:04:47.210062 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0dbecc93-5a61-4100-a319-d3f673f7a5fd-credential-socket\") pod \"e2e-worker-w3-83755a-openshift-default-c86f9565c-zj5jg\" (UID: \"0dbecc93-5a61-4100-a319-d3f673f7a5fd\") " pod="openshift-ingress/e2e-worker-w3-83755a-openshift-default-c86f9565c-zj5jg" Nov 03 01:04:47.213949 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:04:47.210115 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0dbecc93-5a61-4100-a319-d3f673f7a5fd-istio-podinfo\") pod \"e2e-worker-w3-83755a-openshift-default-c86f9565c-zj5jg\" (UID: \"0dbecc93-5a61-4100-a319-d3f673f7a5fd\") " pod="openshift-ingress/e2e-worker-w3-83755a-openshift-default-c86f9565c-zj5jg" Nov 03 01:04:47.213949 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:04:47.210144 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/569c2920-1523-4da8-8644-f789aebcdc7a-istiod-ca-cert\") pod \"e2e-worker-w3-83755a-openshift-default-c86f9565c-z7flk\" (UID: \"569c2920-1523-4da8-8644-f789aebcdc7a\") " pod="openshift-ingress/e2e-worker-w3-83755a-openshift-default-c86f9565c-z7flk" Nov 03 01:04:47.213949 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:04:47.210170 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f4rxn\" (UniqueName: \"kubernetes.io/projected/0dbecc93-5a61-4100-a319-d3f673f7a5fd-kube-api-access-f4rxn\") pod \"e2e-worker-w3-83755a-openshift-default-c86f9565c-zj5jg\" (UID: \"0dbecc93-5a61-4100-a319-d3f673f7a5fd\") " pod="openshift-ingress/e2e-worker-w3-83755a-openshift-default-c86f9565c-zj5jg" Nov 03 01:04:47.213949 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:04:47.210199 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0dbecc93-5a61-4100-a319-d3f673f7a5fd-istiod-ca-cert\") pod \"e2e-worker-w3-83755a-openshift-default-c86f9565c-zj5jg\" (UID: \"0dbecc93-5a61-4100-a319-d3f673f7a5fd\") " pod="openshift-ingress/e2e-worker-w3-83755a-openshift-default-c86f9565c-zj5jg" Nov 03 01:04:47.213949 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:04:47.210278 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/569c2920-1523-4da8-8644-f789aebcdc7a-istio-podinfo\") pod \"e2e-worker-w3-83755a-openshift-default-c86f9565c-z7flk\" (UID: \"569c2920-1523-4da8-8644-f789aebcdc7a\") " pod="openshift-ingress/e2e-worker-w3-83755a-openshift-default-c86f9565c-z7flk" Nov 03 01:04:47.213949 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:04:47.210314 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0dbecc93-5a61-4100-a319-d3f673f7a5fd-istio-token\") pod \"e2e-worker-w3-83755a-openshift-default-c86f9565c-zj5jg\" (UID: \"0dbecc93-5a61-4100-a319-d3f673f7a5fd\") " pod="openshift-ingress/e2e-worker-w3-83755a-openshift-default-c86f9565c-zj5jg" Nov 03 01:04:47.213949 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:04:47.210369 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0dbecc93-5a61-4100-a319-d3f673f7a5fd-istio-envoy\") pod \"e2e-worker-w3-83755a-openshift-default-c86f9565c-zj5jg\" (UID: \"0dbecc93-5a61-4100-a319-d3f673f7a5fd\") " pod="openshift-ingress/e2e-worker-w3-83755a-openshift-default-c86f9565c-zj5jg" Nov 03 01:04:47.213949 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:04:47.210420 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0dbecc93-5a61-4100-a319-d3f673f7a5fd-istio-data\") pod \"e2e-worker-w3-83755a-openshift-default-c86f9565c-zj5jg\" (UID: \"0dbecc93-5a61-4100-a319-d3f673f7a5fd\") " pod="openshift-ingress/e2e-worker-w3-83755a-openshift-default-c86f9565c-zj5jg" Nov 03 01:04:47.213949 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:04:47.210456 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/569c2920-1523-4da8-8644-f789aebcdc7a-workload-socket\") pod \"e2e-worker-w3-83755a-openshift-default-c86f9565c-z7flk\" (UID: \"569c2920-1523-4da8-8644-f789aebcdc7a\") " pod="openshift-ingress/e2e-worker-w3-83755a-openshift-default-c86f9565c-z7flk" Nov 03 01:04:47.213949 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:04:47.210500 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0dbecc93-5a61-4100-a319-d3f673f7a5fd-workload-certs\") pod \"e2e-worker-w3-83755a-openshift-default-c86f9565c-zj5jg\" (UID: \"0dbecc93-5a61-4100-a319-d3f673f7a5fd\") " pod="openshift-ingress/e2e-worker-w3-83755a-openshift-default-c86f9565c-zj5jg" Nov 03 01:04:47.213949 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:04:47.210634 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/569c2920-1523-4da8-8644-f789aebcdc7a-istio-token\") pod \"e2e-worker-w3-83755a-openshift-default-c86f9565c-z7flk\" (UID: \"569c2920-1523-4da8-8644-f789aebcdc7a\") " pod="openshift-ingress/e2e-worker-w3-83755a-openshift-default-c86f9565c-z7flk" Nov 03 01:04:47.213949 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:04:47.210664 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0dbecc93-5a61-4100-a319-d3f673f7a5fd-workload-socket\") pod \"e2e-worker-w3-83755a-openshift-default-c86f9565c-zj5jg\" (UID: \"0dbecc93-5a61-4100-a319-d3f673f7a5fd\") " pod="openshift-ingress/e2e-worker-w3-83755a-openshift-default-c86f9565c-zj5jg" Nov 03 01:04:47.213949 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:04:47.210687 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zgqkb\" (UniqueName: \"kubernetes.io/projected/569c2920-1523-4da8-8644-f789aebcdc7a-kube-api-access-zgqkb\") pod \"e2e-worker-w3-83755a-openshift-default-c86f9565c-z7flk\" (UID: \"569c2920-1523-4da8-8644-f789aebcdc7a\") " pod="openshift-ingress/e2e-worker-w3-83755a-openshift-default-c86f9565c-z7flk" Nov 03 01:04:47.213949 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:04:47.209473 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/569c2920-1523-4da8-8644-f789aebcdc7a-workload-certs\") pod \"e2e-worker-w3-83755a-openshift-default-c86f9565c-z7flk\" (UID: \"569c2920-1523-4da8-8644-f789aebcdc7a\") " pod="openshift-ingress/e2e-worker-w3-83755a-openshift-default-c86f9565c-z7flk" Nov 03 01:04:47.213949 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:04:47.211401 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/569c2920-1523-4da8-8644-f789aebcdc7a-credential-socket\") pod \"e2e-worker-w3-83755a-openshift-default-c86f9565c-z7flk\" (UID: \"569c2920-1523-4da8-8644-f789aebcdc7a\") " pod="openshift-ingress/e2e-worker-w3-83755a-openshift-default-c86f9565c-z7flk" Nov 03 01:04:47.213949 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:04:47.211685 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/569c2920-1523-4da8-8644-f789aebcdc7a-istio-data\") pod \"e2e-worker-w3-83755a-openshift-default-c86f9565c-z7flk\" (UID: \"569c2920-1523-4da8-8644-f789aebcdc7a\") " pod="openshift-ingress/e2e-worker-w3-83755a-openshift-default-c86f9565c-z7flk" Nov 03 01:04:47.213949 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:04:47.212686 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0dbecc93-5a61-4100-a319-d3f673f7a5fd-credential-socket\") pod \"e2e-worker-w3-83755a-openshift-default-c86f9565c-zj5jg\" (UID: \"0dbecc93-5a61-4100-a319-d3f673f7a5fd\") " pod="openshift-ingress/e2e-worker-w3-83755a-openshift-default-c86f9565c-zj5jg" Nov 03 01:04:47.220897 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:04:47.215937 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0dbecc93-5a61-4100-a319-d3f673f7a5fd-istio-podinfo\") pod \"e2e-worker-w3-83755a-openshift-default-c86f9565c-zj5jg\" (UID: \"0dbecc93-5a61-4100-a319-d3f673f7a5fd\") " pod="openshift-ingress/e2e-worker-w3-83755a-openshift-default-c86f9565c-zj5jg" Nov 03 01:04:47.220897 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:04:47.216580 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/569c2920-1523-4da8-8644-f789aebcdc7a-istiod-ca-cert\") pod \"e2e-worker-w3-83755a-openshift-default-c86f9565c-z7flk\" (UID: \"569c2920-1523-4da8-8644-f789aebcdc7a\") " pod="openshift-ingress/e2e-worker-w3-83755a-openshift-default-c86f9565c-z7flk" Nov 03 01:04:47.220897 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:04:47.217551 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0dbecc93-5a61-4100-a319-d3f673f7a5fd-istiod-ca-cert\") pod \"e2e-worker-w3-83755a-openshift-default-c86f9565c-zj5jg\" (UID: \"0dbecc93-5a61-4100-a319-d3f673f7a5fd\") " pod="openshift-ingress/e2e-worker-w3-83755a-openshift-default-c86f9565c-zj5jg" Nov 03 01:04:47.220897 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:04:47.219506 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/569c2920-1523-4da8-8644-f789aebcdc7a-istio-envoy\") pod \"e2e-worker-w3-83755a-openshift-default-c86f9565c-z7flk\" (UID: \"569c2920-1523-4da8-8644-f789aebcdc7a\") " pod="openshift-ingress/e2e-worker-w3-83755a-openshift-default-c86f9565c-z7flk" Nov 03 01:04:47.235691 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:04:47.226425 4939 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 01:04:47.235691 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:04:47.228980 4939 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 01:04:47.235691 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:04:47.229051 4939 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 01:04:47.235691 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:04:47.231233 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0dbecc93-5a61-4100-a319-d3f673f7a5fd-istio-data\") pod \"e2e-worker-w3-83755a-openshift-default-c86f9565c-zj5jg\" (UID: \"0dbecc93-5a61-4100-a319-d3f673f7a5fd\") " pod="openshift-ingress/e2e-worker-w3-83755a-openshift-default-c86f9565c-zj5jg" Nov 03 01:04:47.235691 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:04:47.235143 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/569c2920-1523-4da8-8644-f789aebcdc7a-istio-podinfo\") pod \"e2e-worker-w3-83755a-openshift-default-c86f9565c-z7flk\" (UID: \"569c2920-1523-4da8-8644-f789aebcdc7a\") " pod="openshift-ingress/e2e-worker-w3-83755a-openshift-default-c86f9565c-z7flk" Nov 03 01:04:47.236214 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:04:47.236190 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/569c2920-1523-4da8-8644-f789aebcdc7a-workload-socket\") pod \"e2e-worker-w3-83755a-openshift-default-c86f9565c-z7flk\" (UID: \"569c2920-1523-4da8-8644-f789aebcdc7a\") " pod="openshift-ingress/e2e-worker-w3-83755a-openshift-default-c86f9565c-z7flk" Nov 03 01:04:47.236795 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:04:47.236661 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0dbecc93-5a61-4100-a319-d3f673f7a5fd-workload-certs\") pod \"e2e-worker-w3-83755a-openshift-default-c86f9565c-zj5jg\" (UID: \"0dbecc93-5a61-4100-a319-d3f673f7a5fd\") " pod="openshift-ingress/e2e-worker-w3-83755a-openshift-default-c86f9565c-zj5jg" Nov 03 01:04:47.236795 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:04:47.236715 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0dbecc93-5a61-4100-a319-d3f673f7a5fd-istio-envoy\") pod \"e2e-worker-w3-83755a-openshift-default-c86f9565c-zj5jg\" (UID: \"0dbecc93-5a61-4100-a319-d3f673f7a5fd\") " pod="openshift-ingress/e2e-worker-w3-83755a-openshift-default-c86f9565c-zj5jg" Nov 03 01:04:47.237063 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:04:47.237041 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0dbecc93-5a61-4100-a319-d3f673f7a5fd-workload-socket\") pod \"e2e-worker-w3-83755a-openshift-default-c86f9565c-zj5jg\" (UID: \"0dbecc93-5a61-4100-a319-d3f673f7a5fd\") " pod="openshift-ingress/e2e-worker-w3-83755a-openshift-default-c86f9565c-zj5jg" Nov 03 01:04:47.239384 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:04:47.238860 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zgqkb\" (UniqueName: \"kubernetes.io/projected/569c2920-1523-4da8-8644-f789aebcdc7a-kube-api-access-zgqkb\") pod \"e2e-worker-w3-83755a-openshift-default-c86f9565c-z7flk\" (UID: \"569c2920-1523-4da8-8644-f789aebcdc7a\") " pod="openshift-ingress/e2e-worker-w3-83755a-openshift-default-c86f9565c-z7flk" Nov 03 01:04:47.239496 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:04:47.239458 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0dbecc93-5a61-4100-a319-d3f673f7a5fd-istio-token\") pod \"e2e-worker-w3-83755a-openshift-default-c86f9565c-zj5jg\" (UID: \"0dbecc93-5a61-4100-a319-d3f673f7a5fd\") " pod="openshift-ingress/e2e-worker-w3-83755a-openshift-default-c86f9565c-zj5jg" Nov 03 01:04:47.242957 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:04:47.241522 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-f4rxn\" (UniqueName: \"kubernetes.io/projected/0dbecc93-5a61-4100-a319-d3f673f7a5fd-kube-api-access-f4rxn\") pod \"e2e-worker-w3-83755a-openshift-default-c86f9565c-zj5jg\" (UID: \"0dbecc93-5a61-4100-a319-d3f673f7a5fd\") " pod="openshift-ingress/e2e-worker-w3-83755a-openshift-default-c86f9565c-zj5jg" Nov 03 01:04:47.245896 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:04:47.245873 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/569c2920-1523-4da8-8644-f789aebcdc7a-istio-token\") pod \"e2e-worker-w3-83755a-openshift-default-c86f9565c-z7flk\" (UID: \"569c2920-1523-4da8-8644-f789aebcdc7a\") " pod="openshift-ingress/e2e-worker-w3-83755a-openshift-default-c86f9565c-z7flk" Nov 03 01:04:47.248242 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:04:47.248225 4939 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-83755a-openshift-default-c86f9565c-z7flk" Nov 03 01:04:47.360068 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:04:47.353119 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:04:47.378929 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:04:47.362555 4939 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-83755a-openshift-default-c86f9565c-zj5jg" Nov 03 01:04:47.429612 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:04:47.429571 4939 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w4-6c0ec0-openshift-default-79c7584bfc-8hz7h" Nov 03 01:04:47.545471 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:04:47.545442 4939 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-83755a-openshift-default-c86f9565c-z7flk"] Nov 03 01:04:47.546832 ip-10-0-140-107 kubenswrapper[4939]: W1103 01:04:47.546775 4939 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod569c2920_1523_4da8_8644_f789aebcdc7a.slice/crio-fbfa0495231f53bb344cfa2473664f8623d1d79f98eb3e186e8976f59d872304 WatchSource:0}: Error finding container fbfa0495231f53bb344cfa2473664f8623d1d79f98eb3e186e8976f59d872304: Status 404 returned error can't find the container with id fbfa0495231f53bb344cfa2473664f8623d1d79f98eb3e186e8976f59d872304 Nov 03 01:04:47.550880 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:04:47.549714 4939 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 01:04:47.550880 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:04:47.549802 4939 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 01:04:47.550880 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:04:47.549858 4939 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 01:04:47.708589 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:04:47.708487 4939 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-83755a-openshift-default-c86f9565c-zj5jg"] Nov 03 01:04:47.710163 ip-10-0-140-107 kubenswrapper[4939]: W1103 01:04:47.710134 4939 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0dbecc93_5a61_4100_a319_d3f673f7a5fd.slice/crio-7dde1552f382d034e7d5958988008679c7b32c8cd65082ec637886618283baf7 WatchSource:0}: Error finding container 7dde1552f382d034e7d5958988008679c7b32c8cd65082ec637886618283baf7: Status 404 returned error can't find the container with id 7dde1552f382d034e7d5958988008679c7b32c8cd65082ec637886618283baf7 Nov 03 01:04:47.712758 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:04:47.712723 4939 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 01:04:47.712882 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:04:47.712799 4939 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 01:04:47.712882 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:04:47.712855 4939 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 01:04:47.749109 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:04:47.746723 4939 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w4-6c0ec0-openshift-default-79c7584bfc-8hz7h"] Nov 03 01:04:47.761836 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:04:47.761410 4939 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 01:04:47.761836 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:04:47.761494 4939 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 01:04:47.771974 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:04:47.762558 4939 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 01:04:47.800656 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:04:47.798564 4939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-83755a-openshift-default-c86f9565c-z7flk" event={"ID":"569c2920-1523-4da8-8644-f789aebcdc7a","Type":"ContainerStarted","Data":"d19447d0abfd950faea5858874c8ad314badd753800f42c9173c681a1eea80d6"} Nov 03 01:04:47.800656 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:04:47.798669 4939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-83755a-openshift-default-c86f9565c-z7flk" event={"ID":"569c2920-1523-4da8-8644-f789aebcdc7a","Type":"ContainerStarted","Data":"fbfa0495231f53bb344cfa2473664f8623d1d79f98eb3e186e8976f59d872304"} Nov 03 01:04:47.801242 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:04:47.800972 4939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w4-6c0ec0-openshift-default-79c7584bfc-8hz7h" event={"ID":"0a1fe883-48d5-4d06-ab8f-41698965dbbb","Type":"ContainerStarted","Data":"e0422b63d263392686cc9694e1ea6b5235b7da82aa00256ae8ec5792a898739f"} Nov 03 01:04:47.802583 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:04:47.802553 4939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-83755a-openshift-default-c86f9565c-zj5jg" event={"ID":"0dbecc93-5a61-4100-a319-d3f673f7a5fd","Type":"ContainerStarted","Data":"7dde1552f382d034e7d5958988008679c7b32c8cd65082ec637886618283baf7"} Nov 03 01:04:47.806850 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:04:47.806784 4939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-f7680d-openshift-default-c4488d5c9-mlblk" event={"ID":"e08d7d14-b1fa-4fb2-b2d5-8c7fc073fc89","Type":"ContainerStarted","Data":"8fd039107c781ac33f7786416fb1447c41ebda9a3ab07311a652f359dd592f23"} Nov 03 01:04:47.807342 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:04:47.807212 4939 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w1-2729f2-openshift-default-9fb97b4fc-5ww48" Nov 03 01:04:47.807342 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:04:47.807285 4939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-f7680d-openshift-default-c4488d5c9-mlblk" event={"ID":"e08d7d14-b1fa-4fb2-b2d5-8c7fc073fc89","Type":"ContainerStarted","Data":"e3183b5a85d53a1066123f57f142dcddfba055e478569449c87ed804566809c1"} Nov 03 01:04:47.808967 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:04:47.808938 4939 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w1-2729f2-openshift-default-9fb97b4fc-5ww48" Nov 03 01:04:47.810061 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:04:47.810041 4939 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-f7680d-openshift-default-c4488d5c9-mlblk" Nov 03 01:04:47.822514 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:04:47.822470 4939 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-83755a-openshift-default-c86f9565c-z7flk" podStartSLOduration=1.8224528229999999 podStartE2EDuration="1.822452823s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 01:04:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 01:04:47.819296288 +0000 UTC m=+1436.083631232" watchObservedRunningTime="2026-11-03 01:04:47.822452823 +0000 UTC m=+1436.086787768" Nov 03 01:04:47.919983 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:04:47.903665 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:04:47.925232 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:04:47.924279 4939 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-f7680d-openshift-default-c4488d5c9-mlblk" podStartSLOduration=1.924258762 podStartE2EDuration="1.924258762s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 01:04:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 01:04:47.909720816 +0000 UTC m=+1436.174055763" watchObservedRunningTime="2026-11-03 01:04:47.924258762 +0000 UTC m=+1436.188593708" Nov 03 01:04:47.944477 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:04:47.944431 4939 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="19d3d3db-71fc-41f2-9976-fa8bcd10bfec" path="/var/lib/kubelet/pods/19d3d3db-71fc-41f2-9976-fa8bcd10bfec/volumes" Nov 03 01:04:47.945175 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:04:47.945097 4939 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e1d5b015-e51e-4f86-ac94-34b05d8db2d3" path="/var/lib/kubelet/pods/e1d5b015-e51e-4f86-ac94-34b05d8db2d3/volumes" Nov 03 01:04:48.071664 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:04:48.070585 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:04:48.254071 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:04:48.253548 4939 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-83755a-openshift-default-c86f9565c-z7flk" Nov 03 01:04:48.254071 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:04:48.253689 4939 patch_prober.go:29] interesting pod/e2e-worker-w3-83755a-openshift-default-c86f9565c-z7flk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.166:15021/healthz/ready\": dial tcp 10.132.0.166:15021: connect: connection refused" start-of-body= Nov 03 01:04:48.254071 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:04:48.253746 4939 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-83755a-openshift-default-c86f9565c-z7flk" podUID="569c2920-1523-4da8-8644-f789aebcdc7a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.166:15021/healthz/ready\": dial tcp 10.132.0.166:15021: connect: connection refused" Nov 03 01:04:48.300614 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:04:48.300576 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:04:48.328014 ip-10-0-140-107 kubenswrapper[4939]: E1103 01:04:48.327474 4939 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-sthjh" containerName="extractor" Nov 03 01:04:48.819572 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:04:48.819531 4939 patch_prober.go:29] interesting pod/e2e-worker-w2-f7680d-openshift-default-c4488d5c9-mlblk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.164:15021/healthz/ready\": dial tcp 10.132.0.164:15021: connect: connection refused" start-of-body= Nov 03 01:04:48.820206 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:04:48.820173 4939 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-f7680d-openshift-default-c4488d5c9-mlblk" podUID="e08d7d14-b1fa-4fb2-b2d5-8c7fc073fc89" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.164:15021/healthz/ready\": dial tcp 10.132.0.164:15021: connect: connection refused" Nov 03 01:04:48.831142 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:04:48.831112 4939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w4-6c0ec0-openshift-default-79c7584bfc-8hz7h" event={"ID":"0a1fe883-48d5-4d06-ab8f-41698965dbbb","Type":"ContainerStarted","Data":"3a69478699977ff097dfe6b487c6a8b6fe1eb21904e33269e787e9a5b7b2dea6"} Nov 03 01:04:48.835287 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:04:48.835255 4939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-83755a-openshift-default-c86f9565c-zj5jg" event={"ID":"0dbecc93-5a61-4100-a319-d3f673f7a5fd","Type":"ContainerStarted","Data":"988c3b00cfe1d42a496c5073f93a1ea2f634e32400f4bb974be56cfdede7ffa0"} Nov 03 01:04:48.911192 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:04:48.911110 4939 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w4-6c0ec0-openshift-default-79c7584bfc-8hz7h" podStartSLOduration=2.911086789 podStartE2EDuration="2.911086789s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 01:04:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 01:04:48.907711472 +0000 UTC m=+1437.172046418" watchObservedRunningTime="2026-11-03 01:04:48.911086789 +0000 UTC m=+1437.175421736" Nov 03 01:04:49.249420 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:04:49.249382 4939 patch_prober.go:29] interesting pod/e2e-worker-w3-83755a-openshift-default-c86f9565c-z7flk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.166:15021/healthz/ready\": dial tcp 10.132.0.166:15021: connect: connection refused" start-of-body= Nov 03 01:04:49.249717 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:04:49.249672 4939 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-83755a-openshift-default-c86f9565c-z7flk" podUID="569c2920-1523-4da8-8644-f789aebcdc7a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.166:15021/healthz/ready\": dial tcp 10.132.0.166:15021: connect: connection refused" Nov 03 01:04:49.335616 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:04:49.335580 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:04:49.405945 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:04:49.386417 4939 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-83755a-openshift-default-c86f9565c-zj5jg" Nov 03 01:04:49.405945 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:04:49.399273 4939 patch_prober.go:29] interesting pod/e2e-worker-w3-83755a-openshift-default-c86f9565c-zj5jg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.167:15021/healthz/ready\": dial tcp 10.132.0.167:15021: connect: connection refused" start-of-body= Nov 03 01:04:49.405945 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:04:49.399335 4939 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-83755a-openshift-default-c86f9565c-zj5jg" podUID="0dbecc93-5a61-4100-a319-d3f673f7a5fd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.167:15021/healthz/ready\": dial tcp 10.132.0.167:15021: connect: connection refused" Nov 03 01:04:49.435111 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:04:49.435076 4939 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w4-6c0ec0-openshift-default-79c7584bfc-8hz7h" Nov 03 01:04:49.440701 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:04:49.440668 4939 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w4-6c0ec0-openshift-default-79c7584bfc-8hz7h" Nov 03 01:04:49.465941 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:04:49.450330 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:04:49.503990 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:04:49.502565 4939 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-83755a-openshift-default-c86f9565c-zj5jg" podStartSLOduration=3.5025419490000003 podStartE2EDuration="3.502541949s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 01:04:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 01:04:49.004466088 +0000 UTC m=+1437.268801035" watchObservedRunningTime="2026-11-03 01:04:49.502541949 +0000 UTC m=+1437.766876897" Nov 03 01:04:49.816434 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:04:49.816338 4939 patch_prober.go:29] interesting pod/e2e-worker-w2-f7680d-openshift-default-c4488d5c9-mlblk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.164:15021/healthz/ready\": dial tcp 10.132.0.164:15021: connect: connection refused" start-of-body= Nov 03 01:04:49.816740 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:04:49.816707 4939 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-f7680d-openshift-default-c4488d5c9-mlblk" podUID="e08d7d14-b1fa-4fb2-b2d5-8c7fc073fc89" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.164:15021/healthz/ready\": dial tcp 10.132.0.164:15021: connect: connection refused" Nov 03 01:04:49.842797 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:04:49.842763 4939 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w4-6c0ec0-openshift-default-79c7584bfc-8hz7h" Nov 03 01:04:49.844087 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:04:49.843968 4939 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w4-6c0ec0-openshift-default-79c7584bfc-8hz7h" Nov 03 01:04:50.228956 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:04:50.197495 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:04:50.296516 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:04:50.249359 4939 patch_prober.go:29] interesting pod/e2e-worker-w3-83755a-openshift-default-c86f9565c-z7flk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.166:15021/healthz/ready\": dial tcp 10.132.0.166:15021: connect: connection refused" start-of-body= Nov 03 01:04:50.296516 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:04:50.249421 4939 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-83755a-openshift-default-c86f9565c-z7flk" podUID="569c2920-1523-4da8-8644-f789aebcdc7a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.166:15021/healthz/ready\": dial tcp 10.132.0.166:15021: connect: connection refused" Nov 03 01:04:50.392935 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:04:50.392367 4939 patch_prober.go:29] interesting pod/e2e-worker-w3-83755a-openshift-default-c86f9565c-zj5jg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.167:15021/healthz/ready\": dial tcp 10.132.0.167:15021: connect: connection refused" start-of-body= Nov 03 01:04:50.392935 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:04:50.392434 4939 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-83755a-openshift-default-c86f9565c-zj5jg" podUID="0dbecc93-5a61-4100-a319-d3f673f7a5fd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.167:15021/healthz/ready\": dial tcp 10.132.0.167:15021: connect: connection refused" Nov 03 01:04:50.814641 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:04:50.814583 4939 patch_prober.go:29] interesting pod/e2e-worker-w2-f7680d-openshift-default-c4488d5c9-mlblk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.164:15021/healthz/ready\": dial tcp 10.132.0.164:15021: connect: connection refused" start-of-body= Nov 03 01:04:50.814981 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:04:50.814946 4939 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-f7680d-openshift-default-c4488d5c9-mlblk" podUID="e08d7d14-b1fa-4fb2-b2d5-8c7fc073fc89" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.164:15021/healthz/ready\": dial tcp 10.132.0.164:15021: connect: connection refused" Nov 03 01:04:50.929970 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:04:50.892503 4939 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-88bbf7/simulator-w1-88bbf7-kserve-5fbb669f8f-42hgd" podUID="83b38883-2e7a-444d-aa68-52be4a25ae66" containerName="main" probeResult="failure" output="Get \"https://10.132.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 01:04:51.021929 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:04:51.015065 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:04:51.145947 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:04:51.143224 4939 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w1-2729f2-558f777dc-9gclz"] Nov 03 01:04:51.369614 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:04:51.369265 4939 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-2729f2-558f777dc-9gclz" Nov 03 01:04:51.465024 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:04:51.452785 4939 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-a-w1-2729f2-dockercfg-f8pq8\"" Nov 03 01:04:51.465024 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:04:51.453448 4939 patch_prober.go:29] interesting pod/e2e-worker-w3-83755a-openshift-default-c86f9565c-z7flk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.166:15021/healthz/ready\": dial tcp 10.132.0.166:15021: connect: connection refused" start-of-body= Nov 03 01:04:51.465024 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:04:51.453507 4939 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-83755a-openshift-default-c86f9565c-z7flk" podUID="569c2920-1523-4da8-8644-f789aebcdc7a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.166:15021/healthz/ready\": dial tcp 10.132.0.166:15021: connect: connection refused" Nov 03 01:04:51.465024 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:04:51.453965 4939 patch_prober.go:29] interesting pod/e2e-worker-w3-83755a-openshift-default-c86f9565c-zj5jg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.167:15021/healthz/ready\": dial tcp 10.132.0.167:15021: connect: connection refused" start-of-body= Nov 03 01:04:51.465024 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:04:51.454008 4939 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-83755a-openshift-default-c86f9565c-zj5jg" podUID="0dbecc93-5a61-4100-a319-d3f673f7a5fd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.167:15021/healthz/ready\": dial tcp 10.132.0.167:15021: connect: connection refused" Nov 03 01:04:51.465024 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:04:51.455565 4939 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-a-w1-2729f2\"" Nov 03 01:04:51.469264 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:04:51.468981 4939 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-2f9a3a/distinct-w0-2f9a3a-kserve-555565cfcf-jtcr9" podUID="ac0ece06-8718-463f-915f-5315988d9711" containerName="main" probeResult="failure" output="Get \"https://10.132.0.76:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 01:04:51.511939 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:04:51.508703 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-npp6d\" (UniqueName: \"kubernetes.io/projected/c1e683ec-d1fc-4273-9384-1ffcc06a832a-kube-api-access-npp6d\") pod \"payload-pre-processing-e2e-shared-a-w1-2729f2-558f777dc-9gclz\" (UID: \"c1e683ec-d1fc-4273-9384-1ffcc06a832a\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-2729f2-558f777dc-9gclz" Nov 03 01:04:51.511939 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:04:51.508946 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c1e683ec-d1fc-4273-9384-1ffcc06a832a-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w1-2729f2-558f777dc-9gclz\" (UID: \"c1e683ec-d1fc-4273-9384-1ffcc06a832a\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-2729f2-558f777dc-9gclz" Nov 03 01:04:51.538816 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:04:51.538771 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:04:51.617532 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:04:51.617479 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c1e683ec-d1fc-4273-9384-1ffcc06a832a-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w1-2729f2-558f777dc-9gclz\" (UID: \"c1e683ec-d1fc-4273-9384-1ffcc06a832a\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-2729f2-558f777dc-9gclz" Nov 03 01:04:51.623956 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:04:51.621111 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c1e683ec-d1fc-4273-9384-1ffcc06a832a-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w1-2729f2-558f777dc-9gclz\" (UID: \"c1e683ec-d1fc-4273-9384-1ffcc06a832a\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-2729f2-558f777dc-9gclz" Nov 03 01:04:51.624361 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:04:51.624335 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-npp6d\" (UniqueName: \"kubernetes.io/projected/c1e683ec-d1fc-4273-9384-1ffcc06a832a-kube-api-access-npp6d\") pod \"payload-pre-processing-e2e-shared-a-w1-2729f2-558f777dc-9gclz\" (UID: \"c1e683ec-d1fc-4273-9384-1ffcc06a832a\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-2729f2-558f777dc-9gclz" Nov 03 01:04:51.677218 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:04:51.668249 4939 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w1-2729f2-565847c555-wqvj7"] Nov 03 01:04:51.696297 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:04:51.692723 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-npp6d\" (UniqueName: \"kubernetes.io/projected/c1e683ec-d1fc-4273-9384-1ffcc06a832a-kube-api-access-npp6d\") pod \"payload-pre-processing-e2e-shared-a-w1-2729f2-558f777dc-9gclz\" (UID: \"c1e683ec-d1fc-4273-9384-1ffcc06a832a\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-2729f2-558f777dc-9gclz" Nov 03 01:04:51.729990 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:04:51.724755 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:04:51.769027 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:04:51.765646 4939 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w1-2729f2-558f777dc-9gclz"] Nov 03 01:04:51.769027 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:04:51.765708 4939 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w1-2729f2-565847c555-wqvj7"] Nov 03 01:04:51.769027 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:04:51.765853 4939 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-2729f2-565847c555-wqvj7" Nov 03 01:04:51.830979 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:04:51.830937 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tkkxj\" (UniqueName: \"kubernetes.io/projected/89acf0f1-ef07-4c99-afd3-de3d06335488-kube-api-access-tkkxj\") pod \"payload-processing-e2e-shared-a-w1-2729f2-565847c555-wqvj7\" (UID: \"89acf0f1-ef07-4c99-afd3-de3d06335488\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w1-2729f2-565847c555-wqvj7" Nov 03 01:04:51.836236 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:04:51.836208 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/89acf0f1-ef07-4c99-afd3-de3d06335488-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w1-2729f2-565847c555-wqvj7\" (UID: \"89acf0f1-ef07-4c99-afd3-de3d06335488\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w1-2729f2-565847c555-wqvj7" Nov 03 01:04:51.837023 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:04:51.834027 4939 patch_prober.go:29] interesting pod/e2e-worker-w2-f7680d-openshift-default-c4488d5c9-mlblk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.164:15021/healthz/ready\": dial tcp 10.132.0.164:15021: connect: connection refused" start-of-body= Nov 03 01:04:51.837421 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:04:51.837395 4939 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-f7680d-openshift-default-c4488d5c9-mlblk" podUID="e08d7d14-b1fa-4fb2-b2d5-8c7fc073fc89" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.164:15021/healthz/ready\": dial tcp 10.132.0.164:15021: connect: connection refused" Nov 03 01:04:51.917753 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:04:51.916192 4939 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-88bbf7/simulator-w1-88bbf7-kserve-5fbb669f8f-42hgd" podUID="83b38883-2e7a-444d-aa68-52be4a25ae66" containerName="main" probeResult="failure" output="Get \"https://10.132.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 01:04:51.963974 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:04:51.960254 4939 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-a-w1-2729f2-dockercfg-f8pq8\"" Nov 03 01:04:51.965973 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:04:51.964523 4939 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-2729f2-558f777dc-9gclz" Nov 03 01:04:52.036545 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:04:52.036453 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tkkxj\" (UniqueName: \"kubernetes.io/projected/89acf0f1-ef07-4c99-afd3-de3d06335488-kube-api-access-tkkxj\") pod \"payload-processing-e2e-shared-a-w1-2729f2-565847c555-wqvj7\" (UID: \"89acf0f1-ef07-4c99-afd3-de3d06335488\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w1-2729f2-565847c555-wqvj7" Nov 03 01:04:52.036823 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:04:52.036800 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/89acf0f1-ef07-4c99-afd3-de3d06335488-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w1-2729f2-565847c555-wqvj7\" (UID: \"89acf0f1-ef07-4c99-afd3-de3d06335488\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w1-2729f2-565847c555-wqvj7" Nov 03 01:04:52.093932 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:04:52.039185 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/89acf0f1-ef07-4c99-afd3-de3d06335488-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w1-2729f2-565847c555-wqvj7\" (UID: \"89acf0f1-ef07-4c99-afd3-de3d06335488\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w1-2729f2-565847c555-wqvj7" Nov 03 01:04:52.136271 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:04:52.134841 4939 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-88bbf7/simulator-w1-88bbf7-kserve-5fbb669f8f-42hgd" Nov 03 01:04:52.136966 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:04:52.136939 4939 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-88bbf7/simulator-w1-88bbf7-kserve-5fbb669f8f-42hgd" Nov 03 01:04:52.139521 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:04:52.138760 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tkkxj\" (UniqueName: \"kubernetes.io/projected/89acf0f1-ef07-4c99-afd3-de3d06335488-kube-api-access-tkkxj\") pod \"payload-processing-e2e-shared-a-w1-2729f2-565847c555-wqvj7\" (UID: \"89acf0f1-ef07-4c99-afd3-de3d06335488\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w1-2729f2-565847c555-wqvj7" Nov 03 01:04:52.176690 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:04:52.170609 4939 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-2729f2-565847c555-wqvj7" Nov 03 01:04:52.263964 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:04:52.259520 4939 patch_prober.go:29] interesting pod/e2e-worker-w3-83755a-openshift-default-c86f9565c-z7flk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.166:15021/healthz/ready\": dial tcp 10.132.0.166:15021: connect: connection refused" start-of-body= Nov 03 01:04:52.263964 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:04:52.259590 4939 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-83755a-openshift-default-c86f9565c-z7flk" podUID="569c2920-1523-4da8-8644-f789aebcdc7a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.166:15021/healthz/ready\": dial tcp 10.132.0.166:15021: connect: connection refused" Nov 03 01:04:52.445136 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:04:52.432198 4939 patch_prober.go:29] interesting pod/e2e-worker-w3-83755a-openshift-default-c86f9565c-zj5jg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.167:15021/healthz/ready\": dial tcp 10.132.0.167:15021: connect: connection refused" start-of-body= Nov 03 01:04:52.445136 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:04:52.444385 4939 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-83755a-openshift-default-c86f9565c-zj5jg" podUID="0dbecc93-5a61-4100-a319-d3f673f7a5fd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.167:15021/healthz/ready\": dial tcp 10.132.0.167:15021: connect: connection refused" Nov 03 01:04:52.684999 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:04:52.684883 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:04:52.817135 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:04:52.813672 4939 patch_prober.go:29] interesting pod/e2e-worker-w2-f7680d-openshift-default-c4488d5c9-mlblk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.164:15021/healthz/ready\": dial tcp 10.132.0.164:15021: connect: connection refused" start-of-body= Nov 03 01:04:52.817135 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:04:52.813725 4939 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-f7680d-openshift-default-c4488d5c9-mlblk" podUID="e08d7d14-b1fa-4fb2-b2d5-8c7fc073fc89" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.164:15021/healthz/ready\": dial tcp 10.132.0.164:15021: connect: connection refused" Nov 03 01:04:52.830932 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:04:52.825626 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:04:52.985264 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:04:52.985160 4939 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w4-6c0ec0-65c5d46767-hsxz4"] Nov 03 01:04:53.154636 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:04:53.152300 4939 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w4-6c0ec0-65c5d46767-hsxz4"] Nov 03 01:04:53.154636 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:04:53.152434 4939 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-6c0ec0-65c5d46767-hsxz4" Nov 03 01:04:53.192795 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:04:53.167108 4939 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-a-w4-6c0ec0-dockercfg-vm52m\"" Nov 03 01:04:53.192795 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:04:53.168393 4939 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-a-w4-6c0ec0\"" Nov 03 01:04:53.192795 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:04:53.189968 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-52w9v\" (UniqueName: \"kubernetes.io/projected/61b34e6e-2690-4080-bced-3a577c665f12-kube-api-access-52w9v\") pod \"payload-pre-processing-e2e-shared-a-w4-6c0ec0-65c5d46767-hsxz4\" (UID: \"61b34e6e-2690-4080-bced-3a577c665f12\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-6c0ec0-65c5d46767-hsxz4" Nov 03 01:04:53.192795 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:04:53.190187 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/61b34e6e-2690-4080-bced-3a577c665f12-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w4-6c0ec0-65c5d46767-hsxz4\" (UID: \"61b34e6e-2690-4080-bced-3a577c665f12\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-6c0ec0-65c5d46767-hsxz4" Nov 03 01:04:53.272044 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:04:53.271092 4939 patch_prober.go:29] interesting pod/e2e-worker-w3-83755a-openshift-default-c86f9565c-z7flk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.166:15021/healthz/ready\": dial tcp 10.132.0.166:15021: connect: connection refused" start-of-body= Nov 03 01:04:53.272044 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:04:53.271175 4939 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-83755a-openshift-default-c86f9565c-z7flk" podUID="569c2920-1523-4da8-8644-f789aebcdc7a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.166:15021/healthz/ready\": dial tcp 10.132.0.166:15021: connect: connection refused" Nov 03 01:04:53.318302 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:04:53.309948 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:04:53.347942 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:04:53.331774 4939 scope.go:122] "RemoveContainer" containerID="12d0dc8d226d8b6cc9f19c1e2ef05e2721b39285a2986fa00aa2acf7ecabb7e8" Nov 03 01:04:53.396830 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:04:53.396788 4939 patch_prober.go:29] interesting pod/e2e-worker-w3-83755a-openshift-default-c86f9565c-zj5jg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.167:15021/healthz/ready\": dial tcp 10.132.0.167:15021: connect: connection refused" start-of-body= Nov 03 01:04:53.438319 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:04:53.428000 4939 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-83755a-openshift-default-c86f9565c-zj5jg" podUID="0dbecc93-5a61-4100-a319-d3f673f7a5fd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.167:15021/healthz/ready\": dial tcp 10.132.0.167:15021: connect: connection refused" Nov 03 01:04:53.438319 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:04:53.428851 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-52w9v\" (UniqueName: \"kubernetes.io/projected/61b34e6e-2690-4080-bced-3a577c665f12-kube-api-access-52w9v\") pod \"payload-pre-processing-e2e-shared-a-w4-6c0ec0-65c5d46767-hsxz4\" (UID: \"61b34e6e-2690-4080-bced-3a577c665f12\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-6c0ec0-65c5d46767-hsxz4" Nov 03 01:04:53.438319 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:04:53.429132 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/61b34e6e-2690-4080-bced-3a577c665f12-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w4-6c0ec0-65c5d46767-hsxz4\" (UID: \"61b34e6e-2690-4080-bced-3a577c665f12\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-6c0ec0-65c5d46767-hsxz4" Nov 03 01:04:53.469770 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:04:53.469735 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/61b34e6e-2690-4080-bced-3a577c665f12-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w4-6c0ec0-65c5d46767-hsxz4\" (UID: \"61b34e6e-2690-4080-bced-3a577c665f12\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-6c0ec0-65c5d46767-hsxz4" Nov 03 01:04:53.470050 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:04:53.469977 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-52w9v\" (UniqueName: \"kubernetes.io/projected/61b34e6e-2690-4080-bced-3a577c665f12-kube-api-access-52w9v\") pod \"payload-pre-processing-e2e-shared-a-w4-6c0ec0-65c5d46767-hsxz4\" (UID: \"61b34e6e-2690-4080-bced-3a577c665f12\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-6c0ec0-65c5d46767-hsxz4" Nov 03 01:04:53.578514 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:04:53.578294 4939 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-6c0ec0-65c5d46767-hsxz4" Nov 03 01:04:53.815947 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:04:53.787246 4939 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w4-6c0ec0-7474bf5fc5-dhqxc"] Nov 03 01:04:53.841987 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:04:53.817573 4939 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-6c0ec0-7474bf5fc5-dhqxc" Nov 03 01:04:53.843230 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:04:53.842102 4939 patch_prober.go:29] interesting pod/e2e-worker-w2-f7680d-openshift-default-c4488d5c9-mlblk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.164:15021/healthz/ready\": dial tcp 10.132.0.164:15021: connect: connection refused" start-of-body= Nov 03 01:04:53.843458 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:04:53.843423 4939 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-f7680d-openshift-default-c4488d5c9-mlblk" podUID="e08d7d14-b1fa-4fb2-b2d5-8c7fc073fc89" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.164:15021/healthz/ready\": dial tcp 10.132.0.164:15021: connect: connection refused" Nov 03 01:04:53.853945 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:04:53.845632 4939 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w4-6c0ec0-7474bf5fc5-dhqxc"] Nov 03 01:04:54.251650 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:04:54.251602 4939 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-f7680d/distinct-w2-f7680d-kserve-756f874b98-qnpv8" podUID="836753fc-f9ef-4f61-b3fe-e5b92b7ee27b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.70:8000/ready\": context deadline exceeded" Nov 03 01:04:54.448972 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:04:54.444165 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:04:54.539760 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:04:54.539726 4939 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w1-2729f2-558f777dc-9gclz"] Nov 03 01:04:54.543462 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:04:54.543428 4939 patch_prober.go:29] interesting pod/e2e-worker-w3-83755a-openshift-default-c86f9565c-z7flk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.166:15021/healthz/ready\": dial tcp 10.132.0.166:15021: connect: connection refused" start-of-body= Nov 03 01:04:54.543628 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:04:54.543502 4939 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-83755a-openshift-default-c86f9565c-z7flk" podUID="569c2920-1523-4da8-8644-f789aebcdc7a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.166:15021/healthz/ready\": dial tcp 10.132.0.166:15021: connect: connection refused" Nov 03 01:04:54.544134 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:04:54.544108 4939 patch_prober.go:29] interesting pod/e2e-worker-w3-83755a-openshift-default-c86f9565c-zj5jg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.167:15021/healthz/ready\": dial tcp 10.132.0.167:15021: connect: connection refused" start-of-body= Nov 03 01:04:54.544334 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:04:54.544313 4939 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-83755a-openshift-default-c86f9565c-zj5jg" podUID="0dbecc93-5a61-4100-a319-d3f673f7a5fd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.167:15021/healthz/ready\": dial tcp 10.132.0.167:15021: connect: connection refused" Nov 03 01:04:54.598983 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:04:54.546247 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vzz2v\" (UniqueName: \"kubernetes.io/projected/c6de1910-0365-4f48-99f4-b32aebd747bb-kube-api-access-vzz2v\") pod \"payload-processing-e2e-shared-a-w4-6c0ec0-7474bf5fc5-dhqxc\" (UID: \"c6de1910-0365-4f48-99f4-b32aebd747bb\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w4-6c0ec0-7474bf5fc5-dhqxc" Nov 03 01:04:54.632545 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:04:54.632503 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c6de1910-0365-4f48-99f4-b32aebd747bb-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w4-6c0ec0-7474bf5fc5-dhqxc\" (UID: \"c6de1910-0365-4f48-99f4-b32aebd747bb\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w4-6c0ec0-7474bf5fc5-dhqxc" Nov 03 01:04:54.743309 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:04:54.743242 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c6de1910-0365-4f48-99f4-b32aebd747bb-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w4-6c0ec0-7474bf5fc5-dhqxc\" (UID: \"c6de1910-0365-4f48-99f4-b32aebd747bb\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w4-6c0ec0-7474bf5fc5-dhqxc" Nov 03 01:04:54.744109 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:04:54.744064 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vzz2v\" (UniqueName: \"kubernetes.io/projected/c6de1910-0365-4f48-99f4-b32aebd747bb-kube-api-access-vzz2v\") pod \"payload-processing-e2e-shared-a-w4-6c0ec0-7474bf5fc5-dhqxc\" (UID: \"c6de1910-0365-4f48-99f4-b32aebd747bb\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w4-6c0ec0-7474bf5fc5-dhqxc" Nov 03 01:04:54.753979 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:04:54.753943 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c6de1910-0365-4f48-99f4-b32aebd747bb-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w4-6c0ec0-7474bf5fc5-dhqxc\" (UID: \"c6de1910-0365-4f48-99f4-b32aebd747bb\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w4-6c0ec0-7474bf5fc5-dhqxc" Nov 03 01:04:54.895867 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:04:54.895813 4939 patch_prober.go:29] interesting pod/e2e-worker-w2-f7680d-openshift-default-c4488d5c9-mlblk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.164:15021/healthz/ready\": dial tcp 10.132.0.164:15021: connect: connection refused" start-of-body= Nov 03 01:04:54.920432 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:04:54.920388 4939 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-f7680d-openshift-default-c4488d5c9-mlblk" podUID="e08d7d14-b1fa-4fb2-b2d5-8c7fc073fc89" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.164:15021/healthz/ready\": dial tcp 10.132.0.164:15021: connect: connection refused" Nov 03 01:04:55.075017 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:04:55.061206 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vzz2v\" (UniqueName: \"kubernetes.io/projected/c6de1910-0365-4f48-99f4-b32aebd747bb-kube-api-access-vzz2v\") pod \"payload-processing-e2e-shared-a-w4-6c0ec0-7474bf5fc5-dhqxc\" (UID: \"c6de1910-0365-4f48-99f4-b32aebd747bb\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w4-6c0ec0-7474bf5fc5-dhqxc" Nov 03 01:04:55.128093 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:04:55.128001 4939 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w4-6c0ec0-65c5d46767-hsxz4"] Nov 03 01:04:55.140864 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:04:55.140829 4939 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-6c0ec0-7474bf5fc5-dhqxc" Nov 03 01:04:55.237867 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:04:55.237835 4939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-2729f2-558f777dc-9gclz" event={"ID":"c1e683ec-d1fc-4273-9384-1ffcc06a832a","Type":"ContainerStarted","Data":"ba1e8e787ac227c00d92b2e496fa6c7eef28d21491e470fca67d71fb7682a460"} Nov 03 01:04:55.238339 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:04:55.238313 4939 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-2729f2-558f777dc-9gclz" Nov 03 01:04:55.295941 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:04:55.287159 4939 patch_prober.go:29] interesting pod/e2e-worker-w3-83755a-openshift-default-c86f9565c-z7flk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.166:15021/healthz/ready\": dial tcp 10.132.0.166:15021: connect: connection refused" start-of-body= Nov 03 01:04:55.295941 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:04:55.287230 4939 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-83755a-openshift-default-c86f9565c-z7flk" podUID="569c2920-1523-4da8-8644-f789aebcdc7a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.166:15021/healthz/ready\": dial tcp 10.132.0.166:15021: connect: connection refused" Nov 03 01:04:55.338054 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:04:55.338025 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:04:55.376411 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:04:55.376309 4939 patch_prober.go:29] interesting pod/e2e-worker-w3-83755a-openshift-default-c86f9565c-zj5jg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.167:15021/healthz/ready\": dial tcp 10.132.0.167:15021: connect: connection refused" start-of-body= Nov 03 01:04:55.376411 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:04:55.376382 4939 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-83755a-openshift-default-c86f9565c-zj5jg" podUID="0dbecc93-5a61-4100-a319-d3f673f7a5fd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.167:15021/healthz/ready\": dial tcp 10.132.0.167:15021: connect: connection refused" Nov 03 01:04:55.621448 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:04:55.621391 4939 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-2729f2-558f777dc-9gclz" podStartSLOduration=4.621370829 podStartE2EDuration="4.621370829s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 01:04:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 01:04:55.306249007 +0000 UTC m=+1443.570583953" watchObservedRunningTime="2026-11-03 01:04:55.621370829 +0000 UTC m=+1443.885705777" Nov 03 01:04:55.640954 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:04:55.640597 4939 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w1-2729f2-565847c555-wqvj7"] Nov 03 01:04:55.779917 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:04:55.779805 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:04:55.810323 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:04:55.810280 4939 patch_prober.go:29] interesting pod/e2e-worker-w2-f7680d-openshift-default-c4488d5c9-mlblk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.164:15021/healthz/ready\": dial tcp 10.132.0.164:15021: connect: connection refused" start-of-body= Nov 03 01:04:55.810522 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:04:55.810348 4939 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-f7680d-openshift-default-c4488d5c9-mlblk" podUID="e08d7d14-b1fa-4fb2-b2d5-8c7fc073fc89" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.164:15021/healthz/ready\": dial tcp 10.132.0.164:15021: connect: connection refused" Nov 03 01:04:55.928293 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:04:55.928261 4939 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w4-6c0ec0-7474bf5fc5-dhqxc"] Nov 03 01:04:55.931937 ip-10-0-140-107 kubenswrapper[4939]: W1103 01:04:55.931877 4939 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc6de1910_0365_4f48_99f4_b32aebd747bb.slice/crio-3b50a4be96bcd5030b24947c77fb5fdde836f9081bdf85694bd4d4b3b5cbb742 WatchSource:0}: Error finding container 3b50a4be96bcd5030b24947c77fb5fdde836f9081bdf85694bd4d4b3b5cbb742: Status 404 returned error can't find the container with id 3b50a4be96bcd5030b24947c77fb5fdde836f9081bdf85694bd4d4b3b5cbb742 Nov 03 01:04:56.248987 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:04:56.248944 4939 patch_prober.go:29] interesting pod/e2e-worker-w3-83755a-openshift-default-c86f9565c-z7flk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.166:15021/healthz/ready\": dial tcp 10.132.0.166:15021: connect: connection refused" start-of-body= Nov 03 01:04:56.249207 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:04:56.249011 4939 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-83755a-openshift-default-c86f9565c-z7flk" podUID="569c2920-1523-4da8-8644-f789aebcdc7a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.166:15021/healthz/ready\": dial tcp 10.132.0.166:15021: connect: connection refused" Nov 03 01:04:56.250872 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:04:56.250839 4939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-6c0ec0-65c5d46767-hsxz4" event={"ID":"61b34e6e-2690-4080-bced-3a577c665f12","Type":"ContainerStarted","Data":"4953379acfa82f023c872f91888a0f003a2e7fc293fc9ccc0020117a5946d212"} Nov 03 01:04:56.251039 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:04:56.250903 4939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-6c0ec0-65c5d46767-hsxz4" event={"ID":"61b34e6e-2690-4080-bced-3a577c665f12","Type":"ContainerStarted","Data":"ca7d9f5a7656de2ff40d1feab185e4bc13acc8e369a86c0f77888afa7ddd40a3"} Nov 03 01:04:56.252137 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:04:56.252101 4939 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-6c0ec0-65c5d46767-hsxz4" Nov 03 01:04:56.253302 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:04:56.253274 4939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-6c0ec0-7474bf5fc5-dhqxc" event={"ID":"c6de1910-0365-4f48-99f4-b32aebd747bb","Type":"ContainerStarted","Data":"94803b27fbf936d8d29f1d086acc91666547bca0730a0e5ec7c0362f7887771a"} Nov 03 01:04:56.253418 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:04:56.253326 4939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-6c0ec0-7474bf5fc5-dhqxc" event={"ID":"c6de1910-0365-4f48-99f4-b32aebd747bb","Type":"ContainerStarted","Data":"3b50a4be96bcd5030b24947c77fb5fdde836f9081bdf85694bd4d4b3b5cbb742"} Nov 03 01:04:56.253418 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:04:56.253359 4939 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-6c0ec0-7474bf5fc5-dhqxc" Nov 03 01:04:56.255431 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:04:56.255400 4939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-2729f2-558f777dc-9gclz" event={"ID":"c1e683ec-d1fc-4273-9384-1ffcc06a832a","Type":"ContainerStarted","Data":"076f1302b3c0a56f35ca8f05d46e5d588e15f1312c7e774aefbb2da2db6c2c2c"} Nov 03 01:04:56.260689 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:04:56.260659 4939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-2729f2-565847c555-wqvj7" event={"ID":"89acf0f1-ef07-4c99-afd3-de3d06335488","Type":"ContainerStarted","Data":"a29f4ada5ae4f3bf33d23981cc60dd8a9fb34c3ed134e3c6cad41f2cfea2c7cd"} Nov 03 01:04:56.260807 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:04:56.260711 4939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-2729f2-565847c555-wqvj7" event={"ID":"89acf0f1-ef07-4c99-afd3-de3d06335488","Type":"ContainerStarted","Data":"a236efb3b54615897784863c0defdd5c318fa8c499abd6632fcde6f948fdda95"} Nov 03 01:04:56.260807 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:04:56.260762 4939 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-2729f2-565847c555-wqvj7" Nov 03 01:04:56.297602 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:04:56.297538 4939 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-6c0ec0-65c5d46767-hsxz4" podStartSLOduration=4.297516534 podStartE2EDuration="4.297516534s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 01:04:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 01:04:56.27481883 +0000 UTC m=+1444.539153777" watchObservedRunningTime="2026-11-03 01:04:56.297516534 +0000 UTC m=+1444.561851480" Nov 03 01:04:56.302812 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:04:56.302760 4939 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-2729f2-565847c555-wqvj7" podStartSLOduration=5.302742774 podStartE2EDuration="5.302742774s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 01:04:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 01:04:56.298232727 +0000 UTC m=+1444.562567656" watchObservedRunningTime="2026-11-03 01:04:56.302742774 +0000 UTC m=+1444.567077720" Nov 03 01:04:56.321284 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:04:56.321221 4939 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-6c0ec0-7474bf5fc5-dhqxc" podStartSLOduration=4.321202321 podStartE2EDuration="4.321202321s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 01:04:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 01:04:56.318432313 +0000 UTC m=+1444.582767259" watchObservedRunningTime="2026-11-03 01:04:56.321202321 +0000 UTC m=+1444.585537267" Nov 03 01:04:56.327013 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:04:56.326933 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:04:56.373652 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:04:56.373611 4939 patch_prober.go:29] interesting pod/e2e-worker-w3-83755a-openshift-default-c86f9565c-zj5jg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.167:15021/healthz/ready\": dial tcp 10.132.0.167:15021: connect: connection refused" start-of-body= Nov 03 01:04:56.373874 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:04:56.373680 4939 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-83755a-openshift-default-c86f9565c-zj5jg" podUID="0dbecc93-5a61-4100-a319-d3f673f7a5fd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.167:15021/healthz/ready\": dial tcp 10.132.0.167:15021: connect: connection refused" Nov 03 01:04:56.886638 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:04:56.886610 4939 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-f7680d-openshift-default-c4488d5c9-mlblk" Nov 03 01:04:56.887705 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:04:56.887676 4939 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-f7680d-openshift-default-c4488d5c9-mlblk" Nov 03 01:04:56.892354 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:04:56.889593 4939 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-f7680d-openshift-default-c4488d5c9-mlblk" Nov 03 01:04:57.161103 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:04:57.161016 4939 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-2729f2-6f5b74c6cf-h9g6h"] Nov 03 01:04:57.166370 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:04:57.166332 4939 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-2729f2-6f5b74c6cf-h9g6h" Nov 03 01:04:57.182706 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:04:57.182675 4939 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-a-w1-2729f2-serving-cert\"" Nov 03 01:04:57.198158 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:04:57.198127 4939 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-2729f2-6f5b74c6cf-h9g6h"] Nov 03 01:04:57.221671 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:04:57.221634 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hfbfw\" (UniqueName: \"kubernetes.io/projected/6ec3abd1-a454-4860-8c1b-d70c26b14b3a-kube-api-access-hfbfw\") pod \"maas-api-e2e-shared-a-w1-2729f2-6f5b74c6cf-h9g6h\" (UID: \"6ec3abd1-a454-4860-8c1b-d70c26b14b3a\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-2729f2-6f5b74c6cf-h9g6h" Nov 03 01:04:57.221851 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:04:57.221770 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6ec3abd1-a454-4860-8c1b-d70c26b14b3a-maas-api-tls\") pod \"maas-api-e2e-shared-a-w1-2729f2-6f5b74c6cf-h9g6h\" (UID: \"6ec3abd1-a454-4860-8c1b-d70c26b14b3a\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-2729f2-6f5b74c6cf-h9g6h" Nov 03 01:04:57.221986 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:04:57.221968 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6ec3abd1-a454-4860-8c1b-d70c26b14b3a-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w1-2729f2-6f5b74c6cf-h9g6h\" (UID: \"6ec3abd1-a454-4860-8c1b-d70c26b14b3a\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-2729f2-6f5b74c6cf-h9g6h" Nov 03 01:04:57.222086 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:04:57.222012 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6ec3abd1-a454-4860-8c1b-d70c26b14b3a-metrics-certs\") pod \"maas-api-e2e-shared-a-w1-2729f2-6f5b74c6cf-h9g6h\" (UID: \"6ec3abd1-a454-4860-8c1b-d70c26b14b3a\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-2729f2-6f5b74c6cf-h9g6h" Nov 03 01:04:57.251580 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:04:57.251544 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:04:57.269067 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:04:57.269030 4939 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-83755a-openshift-default-c86f9565c-z7flk" Nov 03 01:04:57.270451 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:04:57.269826 4939 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-83755a-openshift-default-c86f9565c-z7flk" Nov 03 01:04:57.271590 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:04:57.271551 4939 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-83755a-openshift-default-c86f9565c-z7flk" Nov 03 01:04:57.323133 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:04:57.323102 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6ec3abd1-a454-4860-8c1b-d70c26b14b3a-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w1-2729f2-6f5b74c6cf-h9g6h\" (UID: \"6ec3abd1-a454-4860-8c1b-d70c26b14b3a\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-2729f2-6f5b74c6cf-h9g6h" Nov 03 01:04:57.323740 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:04:57.323203 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6ec3abd1-a454-4860-8c1b-d70c26b14b3a-metrics-certs\") pod \"maas-api-e2e-shared-a-w1-2729f2-6f5b74c6cf-h9g6h\" (UID: \"6ec3abd1-a454-4860-8c1b-d70c26b14b3a\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-2729f2-6f5b74c6cf-h9g6h" Nov 03 01:04:57.323740 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:04:57.323461 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hfbfw\" (UniqueName: \"kubernetes.io/projected/6ec3abd1-a454-4860-8c1b-d70c26b14b3a-kube-api-access-hfbfw\") pod \"maas-api-e2e-shared-a-w1-2729f2-6f5b74c6cf-h9g6h\" (UID: \"6ec3abd1-a454-4860-8c1b-d70c26b14b3a\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-2729f2-6f5b74c6cf-h9g6h" Nov 03 01:04:57.323740 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:04:57.323653 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6ec3abd1-a454-4860-8c1b-d70c26b14b3a-maas-api-tls\") pod \"maas-api-e2e-shared-a-w1-2729f2-6f5b74c6cf-h9g6h\" (UID: \"6ec3abd1-a454-4860-8c1b-d70c26b14b3a\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-2729f2-6f5b74c6cf-h9g6h" Nov 03 01:04:57.327044 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:04:57.327009 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6ec3abd1-a454-4860-8c1b-d70c26b14b3a-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w1-2729f2-6f5b74c6cf-h9g6h\" (UID: \"6ec3abd1-a454-4860-8c1b-d70c26b14b3a\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-2729f2-6f5b74c6cf-h9g6h" Nov 03 01:04:57.330558 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:04:57.330511 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6ec3abd1-a454-4860-8c1b-d70c26b14b3a-maas-api-tls\") pod \"maas-api-e2e-shared-a-w1-2729f2-6f5b74c6cf-h9g6h\" (UID: \"6ec3abd1-a454-4860-8c1b-d70c26b14b3a\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-2729f2-6f5b74c6cf-h9g6h" Nov 03 01:04:57.381246 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:04:57.380734 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hfbfw\" (UniqueName: \"kubernetes.io/projected/6ec3abd1-a454-4860-8c1b-d70c26b14b3a-kube-api-access-hfbfw\") pod \"maas-api-e2e-shared-a-w1-2729f2-6f5b74c6cf-h9g6h\" (UID: \"6ec3abd1-a454-4860-8c1b-d70c26b14b3a\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-2729f2-6f5b74c6cf-h9g6h" Nov 03 01:04:57.384294 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:04:57.382401 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6ec3abd1-a454-4860-8c1b-d70c26b14b3a-metrics-certs\") pod \"maas-api-e2e-shared-a-w1-2729f2-6f5b74c6cf-h9g6h\" (UID: \"6ec3abd1-a454-4860-8c1b-d70c26b14b3a\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-2729f2-6f5b74c6cf-h9g6h" Nov 03 01:04:57.396822 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:04:57.396778 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:04:57.413314 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:04:57.413230 4939 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-83755a-openshift-default-c86f9565c-zj5jg" Nov 03 01:04:57.423353 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:04:57.415991 4939 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-83755a-openshift-default-c86f9565c-zj5jg" Nov 03 01:04:57.423353 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:04:57.422239 4939 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-83755a-openshift-default-c86f9565c-zj5jg" Nov 03 01:04:57.531117 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:04:57.531065 4939 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-2729f2-6f5b74c6cf-h9g6h" Nov 03 01:04:57.853635 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:04:57.853609 4939 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-2729f2-6f5b74c6cf-h9g6h"] Nov 03 01:04:57.854082 ip-10-0-140-107 kubenswrapper[4939]: W1103 01:04:57.854060 4939 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6ec3abd1_a454_4860_8c1b_d70c26b14b3a.slice/crio-06bfe3e3f1661f92afcdbe649013a7c13afee3cf46ce83ac5a921a2cf129c5e4 WatchSource:0}: Error finding container 06bfe3e3f1661f92afcdbe649013a7c13afee3cf46ce83ac5a921a2cf129c5e4: Status 404 returned error can't find the container with id 06bfe3e3f1661f92afcdbe649013a7c13afee3cf46ce83ac5a921a2cf129c5e4 Nov 03 01:04:58.276338 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:04:58.276249 4939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-2729f2-6f5b74c6cf-h9g6h" event={"ID":"6ec3abd1-a454-4860-8c1b-d70c26b14b3a","Type":"ContainerStarted","Data":"06bfe3e3f1661f92afcdbe649013a7c13afee3cf46ce83ac5a921a2cf129c5e4"} Nov 03 01:04:58.364932 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:04:58.360529 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:04:58.921834 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:04:58.921799 4939 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-6c0ec0-fbd7fd95-chmfz"] Nov 03 01:04:58.926182 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:04:58.926147 4939 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-6c0ec0-fbd7fd95-chmfz" Nov 03 01:04:58.937867 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:04:58.937840 4939 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-a-w4-6c0ec0-serving-cert\"" Nov 03 01:04:58.942149 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:04:58.942121 4939 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-6c0ec0-fbd7fd95-chmfz"] Nov 03 01:04:58.970190 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:04:58.970157 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4bcdc6fd-f0da-46e3-bd48-14b1179bb3f1-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w4-6c0ec0-fbd7fd95-chmfz\" (UID: \"4bcdc6fd-f0da-46e3-bd48-14b1179bb3f1\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-6c0ec0-fbd7fd95-chmfz" Nov 03 01:04:58.970544 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:04:58.970516 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4bcdc6fd-f0da-46e3-bd48-14b1179bb3f1-metrics-certs\") pod \"maas-api-e2e-shared-a-w4-6c0ec0-fbd7fd95-chmfz\" (UID: \"4bcdc6fd-f0da-46e3-bd48-14b1179bb3f1\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-6c0ec0-fbd7fd95-chmfz" Nov 03 01:04:58.970660 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:04:58.970643 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4bcdc6fd-f0da-46e3-bd48-14b1179bb3f1-maas-api-tls\") pod \"maas-api-e2e-shared-a-w4-6c0ec0-fbd7fd95-chmfz\" (UID: \"4bcdc6fd-f0da-46e3-bd48-14b1179bb3f1\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-6c0ec0-fbd7fd95-chmfz" Nov 03 01:04:58.970743 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:04:58.970729 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k22mq\" (UniqueName: \"kubernetes.io/projected/4bcdc6fd-f0da-46e3-bd48-14b1179bb3f1-kube-api-access-k22mq\") pod \"maas-api-e2e-shared-a-w4-6c0ec0-fbd7fd95-chmfz\" (UID: \"4bcdc6fd-f0da-46e3-bd48-14b1179bb3f1\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-6c0ec0-fbd7fd95-chmfz" Nov 03 01:04:59.054552 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:04:59.054518 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:04:59.072377 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:04:59.072337 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k22mq\" (UniqueName: \"kubernetes.io/projected/4bcdc6fd-f0da-46e3-bd48-14b1179bb3f1-kube-api-access-k22mq\") pod \"maas-api-e2e-shared-a-w4-6c0ec0-fbd7fd95-chmfz\" (UID: \"4bcdc6fd-f0da-46e3-bd48-14b1179bb3f1\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-6c0ec0-fbd7fd95-chmfz" Nov 03 01:04:59.072586 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:04:59.072525 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4bcdc6fd-f0da-46e3-bd48-14b1179bb3f1-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w4-6c0ec0-fbd7fd95-chmfz\" (UID: \"4bcdc6fd-f0da-46e3-bd48-14b1179bb3f1\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-6c0ec0-fbd7fd95-chmfz" Nov 03 01:04:59.072665 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:04:59.072630 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4bcdc6fd-f0da-46e3-bd48-14b1179bb3f1-metrics-certs\") pod \"maas-api-e2e-shared-a-w4-6c0ec0-fbd7fd95-chmfz\" (UID: \"4bcdc6fd-f0da-46e3-bd48-14b1179bb3f1\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-6c0ec0-fbd7fd95-chmfz" Nov 03 01:04:59.072754 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:04:59.072739 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4bcdc6fd-f0da-46e3-bd48-14b1179bb3f1-maas-api-tls\") pod \"maas-api-e2e-shared-a-w4-6c0ec0-fbd7fd95-chmfz\" (UID: \"4bcdc6fd-f0da-46e3-bd48-14b1179bb3f1\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-6c0ec0-fbd7fd95-chmfz" Nov 03 01:04:59.073313 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:04:59.073286 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4bcdc6fd-f0da-46e3-bd48-14b1179bb3f1-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w4-6c0ec0-fbd7fd95-chmfz\" (UID: \"4bcdc6fd-f0da-46e3-bd48-14b1179bb3f1\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-6c0ec0-fbd7fd95-chmfz" Nov 03 01:04:59.075057 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:04:59.075036 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4bcdc6fd-f0da-46e3-bd48-14b1179bb3f1-metrics-certs\") pod \"maas-api-e2e-shared-a-w4-6c0ec0-fbd7fd95-chmfz\" (UID: \"4bcdc6fd-f0da-46e3-bd48-14b1179bb3f1\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-6c0ec0-fbd7fd95-chmfz" Nov 03 01:04:59.075193 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:04:59.075180 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4bcdc6fd-f0da-46e3-bd48-14b1179bb3f1-maas-api-tls\") pod \"maas-api-e2e-shared-a-w4-6c0ec0-fbd7fd95-chmfz\" (UID: \"4bcdc6fd-f0da-46e3-bd48-14b1179bb3f1\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-6c0ec0-fbd7fd95-chmfz" Nov 03 01:04:59.081210 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:04:59.081185 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-k22mq\" (UniqueName: \"kubernetes.io/projected/4bcdc6fd-f0da-46e3-bd48-14b1179bb3f1-kube-api-access-k22mq\") pod \"maas-api-e2e-shared-a-w4-6c0ec0-fbd7fd95-chmfz\" (UID: \"4bcdc6fd-f0da-46e3-bd48-14b1179bb3f1\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-6c0ec0-fbd7fd95-chmfz" Nov 03 01:04:59.178920 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:04:59.178824 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:04:59.277590 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:04:59.276224 4939 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-6c0ec0-fbd7fd95-chmfz" Nov 03 01:04:59.309967 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:04:59.309932 4939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-2729f2-6f5b74c6cf-h9g6h" event={"ID":"6ec3abd1-a454-4860-8c1b-d70c26b14b3a","Type":"ContainerStarted","Data":"eff241f8fefc5e2ba6484d635be2a30a46e1089a49c2b897e7b35f8fa90a0b04"} Nov 03 01:04:59.311185 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:04:59.311159 4939 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-2729f2-6f5b74c6cf-h9g6h" Nov 03 01:04:59.356981 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:04:59.354571 4939 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-2729f2-6f5b74c6cf-h9g6h" podStartSLOduration=1.7655806950000001 podStartE2EDuration="2.354545276s" totalImagesPullingTime="588.964581ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 01:04:57 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 01:04:59.338875938 +0000 UTC m=+1447.603210885" watchObservedRunningTime="2026-11-03 01:04:59.354545276 +0000 UTC m=+1447.618880219" Nov 03 01:04:59.623949 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:04:59.623894 4939 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-6c0ec0-fbd7fd95-chmfz"] Nov 03 01:04:59.625039 ip-10-0-140-107 kubenswrapper[4939]: W1103 01:04:59.625007 4939 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4bcdc6fd_f0da_46e3_bd48_14b1179bb3f1.slice/crio-e27acbe07b46892600b9c3515e3df96927a20de1f12ff69da30c657f8a3fcec8 WatchSource:0}: Error finding container e27acbe07b46892600b9c3515e3df96927a20de1f12ff69da30c657f8a3fcec8: Status 404 returned error can't find the container with id e27acbe07b46892600b9c3515e3df96927a20de1f12ff69da30c657f8a3fcec8 Nov 03 01:04:59.715938 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:04:59.715403 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:04:59.982805 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:04:59.982712 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:05:00.119864 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:05:00.119834 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:05:00.257059 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:05:00.256901 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:05:00.354586 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:05:00.354541 4939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-6c0ec0-fbd7fd95-chmfz" event={"ID":"4bcdc6fd-f0da-46e3-bd48-14b1179bb3f1","Type":"ContainerStarted","Data":"50bf3e38ce34a7c755e5054daa66036ac27246669c747a22296823c86da51b3e"} Nov 03 01:05:00.357232 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:05:00.357204 4939 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-6c0ec0-fbd7fd95-chmfz" Nov 03 01:05:00.409409 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:05:00.409365 4939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-6c0ec0-fbd7fd95-chmfz" event={"ID":"4bcdc6fd-f0da-46e3-bd48-14b1179bb3f1","Type":"ContainerStarted","Data":"e27acbe07b46892600b9c3515e3df96927a20de1f12ff69da30c657f8a3fcec8"} Nov 03 01:05:00.893559 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:05:00.893524 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:05:01.642930 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:05:01.640485 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:05:02.019126 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:05:02.019015 4939 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-6c0ec0-fbd7fd95-chmfz" podStartSLOduration=3.570175043 podStartE2EDuration="4.01899951s" totalImagesPullingTime="448.824467ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 01:04:58 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 01:05:00.409104506 +0000 UTC m=+1448.673439452" watchObservedRunningTime="2026-11-03 01:05:02.01899951 +0000 UTC m=+1450.283334454" Nov 03 01:05:02.019659 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:05:02.019641 4939 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-4pw7v"] Nov 03 01:05:02.394834 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:05:02.394781 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:05:02.612927 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:05:02.612887 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:05:03.466878 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:05:03.466838 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:05:03.917792 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:05:03.870566 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:05:04.446429 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:05:04.446393 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:05:04.968935 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:05:04.963361 4939 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-88bbf7/distinct-2-w1-88bbf7-kserve-5457cdcdbd-l6tfd" podUID="a397931b-faa9-4c2e-8290-491350546663" containerName="main" probeResult="failure" output="Get \"https://10.132.0.85:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 01:05:05.159071 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:05:05.159017 4939 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-88bbf7/simulator-w1-88bbf7-kserve-5fbb669f8f-42hgd" podUID="83b38883-2e7a-444d-aa68-52be4a25ae66" containerName="main" probeResult="failure" output="Get \"https://10.132.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 01:05:05.159779 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:05:05.159332 4939 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-f7680d/distinct-w2-f7680d-kserve-756f874b98-qnpv8" podUID="836753fc-f9ef-4f61-b3fe-e5b92b7ee27b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.70:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 01:05:05.383853 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:05:05.383808 4939 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-f7680d/premium-w2-f7680d-kserve-76565b8cf9-hqbbt" podUID="4712e708-18a2-4a13-a12c-a5669397b2c3" containerName="main" probeResult="failure" output="Get \"https://10.132.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 01:05:05.384656 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:05:05.384624 4939 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-f7680d/distinct-2-w2-f7680d-kserve-7d7c4564bf-cbv2c" podUID="2de18086-761b-4773-b9fc-15ddc4a220fc" containerName="main" probeResult="failure" output="Get \"https://10.132.0.75:8000/ready\": context deadline exceeded" Nov 03 01:05:05.410933 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:05:05.407582 4939 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-88bbf7/distinct-2-w1-88bbf7-kserve-5457cdcdbd-l6tfd" podUID="a397931b-faa9-4c2e-8290-491350546663" containerName="main" probeResult="failure" output="Get \"https://10.132.0.85:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 01:05:05.487059 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:05:05.487005 4939 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-2f9a3a/distinct-w0-2f9a3a-kserve-555565cfcf-jtcr9" podUID="ac0ece06-8718-463f-915f-5315988d9711" containerName="main" probeResult="failure" output="Get \"https://10.132.0.76:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 01:05:05.668142 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:05:05.668039 4939 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-2f9a3a/simulator-w0-2f9a3a-kserve-9596bbb8f-2trgz" podUID="207e2207-fb6c-4076-95df-72e95671c261" containerName="main" probeResult="failure" output="Get \"https://10.132.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 01:05:05.730991 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:05:05.730937 4939 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-83755a/distinct-2-w3-83755a-kserve-6cf66b995d-2l47b" podUID="c3b05e88-5d5b-4d03-b28d-832f22e3c868" containerName="main" probeResult="failure" output="Get \"https://10.132.0.72:8000/ready\": context deadline exceeded" Nov 03 01:05:05.945996 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:05:05.942547 4939 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-2f9a3a/simulator-w0-2f9a3a-kserve-9596bbb8f-2trgz" podUID="207e2207-fb6c-4076-95df-72e95671c261" containerName="main" probeResult="failure" output="Get \"https://10.132.0.61:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 01:05:05.979455 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:05:05.979409 4939 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-88bbf7/distinct-2-w1-88bbf7-kserve-5457cdcdbd-l6tfd" podUID="a397931b-faa9-4c2e-8290-491350546663" containerName="main" probeResult="failure" output="Get \"https://10.132.0.85:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 01:05:06.115095 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:05:06.115042 4939 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-83755a/distinct-2-w3-83755a-kserve-6cf66b995d-2l47b" podUID="c3b05e88-5d5b-4d03-b28d-832f22e3c868" containerName="main" probeResult="failure" output="Get \"https://10.132.0.72:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 01:05:06.164934 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:05:06.161046 4939 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-88bbf7/simulator-w1-88bbf7-kserve-5fbb669f8f-42hgd" podUID="83b38883-2e7a-444d-aa68-52be4a25ae66" containerName="main" probeResult="failure" output="Get \"https://10.132.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 01:05:06.246004 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:05:06.224544 4939 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-f7680d/distinct-w2-f7680d-kserve-756f874b98-qnpv8" podUID="836753fc-f9ef-4f61-b3fe-e5b92b7ee27b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.70:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 01:05:06.422934 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:05:06.400778 4939 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-88bbf7/distinct-2-w1-88bbf7-kserve-5457cdcdbd-l6tfd" Nov 03 01:05:06.424278 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:05:06.424250 4939 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-88bbf7/simulator-w1-88bbf7-kserve-5fbb669f8f-42hgd" Nov 03 01:05:06.425797 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:05:06.425776 4939 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-f7680d/distinct-w2-f7680d-kserve-756f874b98-qnpv8" Nov 03 01:05:06.426744 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:05:06.421284 4939 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-f7680d/distinct-2-w2-f7680d-kserve-7d7c4564bf-cbv2c" podUID="2de18086-761b-4773-b9fc-15ddc4a220fc" containerName="main" probeResult="failure" output="Get \"https://10.132.0.75:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 01:05:06.577991 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:05:06.533095 4939 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-f7680d/distinct-2-w2-f7680d-kserve-7d7c4564bf-cbv2c" Nov 03 01:05:06.577991 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:05:06.534210 4939 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-2729f2-6f5b74c6cf-h9g6h" Nov 03 01:05:06.581930 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:05:06.578391 4939 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-2f9a3a/distinct-w0-2f9a3a-kserve-555565cfcf-jtcr9" podUID="ac0ece06-8718-463f-915f-5315988d9711" containerName="main" probeResult="failure" output="Get \"https://10.132.0.76:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 01:05:06.581930 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:05:06.578865 4939 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-2f9a3a/distinct-w0-2f9a3a-kserve-555565cfcf-jtcr9" Nov 03 01:05:06.764932 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:05:06.714011 4939 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-2f9a3a/simulator-w0-2f9a3a-kserve-9596bbb8f-2trgz" podUID="207e2207-fb6c-4076-95df-72e95671c261" containerName="main" probeResult="failure" output="Get \"https://10.132.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 01:05:06.764932 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:05:06.714494 4939 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-2f9a3a/simulator-w0-2f9a3a-kserve-9596bbb8f-2trgz" Nov 03 01:05:06.764932 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:05:06.714879 4939 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-f7680d/premium-w2-f7680d-kserve-76565b8cf9-hqbbt" podUID="4712e708-18a2-4a13-a12c-a5669397b2c3" containerName="main" probeResult="failure" output="Get \"https://10.132.0.63:8000/ready\": context deadline exceeded" Nov 03 01:05:06.764932 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:05:06.715220 4939 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-f7680d/premium-w2-f7680d-kserve-76565b8cf9-hqbbt" Nov 03 01:05:06.764932 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:05:06.756054 4939 patch_prober.go:29] interesting pod/e2e-worker-w0-2f9a3a-openshift-default-7f6f74cf97-dxffl container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.57:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 01:05:06.764932 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:05:06.756126 4939 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w0-2f9a3a-openshift-default-7f6f74cf97-dxffl" podUID="80e0f915-1a1a-4214-92f0-5eeb806be96e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.57:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 01:05:06.764932 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:05:06.756254 4939 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-83755a/distinct-2-w3-83755a-kserve-6cf66b995d-2l47b" podUID="c3b05e88-5d5b-4d03-b28d-832f22e3c868" containerName="main" probeResult="failure" output="Get \"https://10.132.0.72:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 01:05:06.764932 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:05:06.757672 4939 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-83755a/distinct-2-w3-83755a-kserve-6cf66b995d-2l47b" Nov 03 01:05:06.946495 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:05:06.946459 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:05:06.998933 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:05:06.985084 4939 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-88bbf7/distinct-2-w1-88bbf7-kserve-5457cdcdbd-l6tfd" podUID="a397931b-faa9-4c2e-8290-491350546663" containerName="main" probeResult="failure" output="Get \"https://10.132.0.85:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 01:05:07.131876 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:05:07.129466 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:05:07.171225 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:05:07.167118 4939 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-88bbf7/simulator-w1-88bbf7-kserve-5fbb669f8f-42hgd" podUID="83b38883-2e7a-444d-aa68-52be4a25ae66" containerName="main" probeResult="failure" output="Get \"https://10.132.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 01:05:07.351788 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:05:07.340961 4939 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-2729f2-565847c555-wqvj7" Nov 03 01:05:07.392364 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:05:07.392319 4939 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-f7680d/distinct-w2-f7680d-kserve-756f874b98-qnpv8" podUID="836753fc-f9ef-4f61-b3fe-e5b92b7ee27b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.70:8000/ready\": context deadline exceeded" Nov 03 01:05:07.409977 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:05:07.409939 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:05:07.428151 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:05:07.428091 4939 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-f7680d/distinct-2-w2-f7680d-kserve-7d7c4564bf-cbv2c" podUID="2de18086-761b-4773-b9fc-15ddc4a220fc" containerName="main" probeResult="failure" output="Get \"https://10.132.0.75:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 01:05:07.439939 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:05:07.434536 4939 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-6c0ec0-7474bf5fc5-dhqxc" Nov 03 01:05:07.451960 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:05:07.450119 4939 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-2729f2-558f777dc-9gclz" Nov 03 01:05:07.494098 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:05:07.494059 4939 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-6c0ec0-fbd7fd95-chmfz" Nov 03 01:05:07.662988 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:05:07.654941 4939 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-2f9a3a/distinct-w0-2f9a3a-kserve-555565cfcf-jtcr9" podUID="ac0ece06-8718-463f-915f-5315988d9711" containerName="main" probeResult="failure" output="Get \"https://10.132.0.76:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 01:05:07.729932 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:05:07.720095 4939 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-2f9a3a/simulator-w0-2f9a3a-kserve-9596bbb8f-2trgz" podUID="207e2207-fb6c-4076-95df-72e95671c261" containerName="main" probeResult="failure" output="Get \"https://10.132.0.61:8000/ready\": context deadline exceeded" Nov 03 01:05:07.742937 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:05:07.734023 4939 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-f7680d/premium-w2-f7680d-kserve-76565b8cf9-hqbbt" podUID="4712e708-18a2-4a13-a12c-a5669397b2c3" containerName="main" probeResult="failure" output="Get \"https://10.132.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 01:05:07.769987 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:05:07.759623 4939 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-83755a/distinct-2-w3-83755a-kserve-6cf66b995d-2l47b" podUID="c3b05e88-5d5b-4d03-b28d-832f22e3c868" containerName="main" probeResult="failure" output="Get \"https://10.132.0.72:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 01:05:08.005103 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:05:08.004989 4939 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-88bbf7/distinct-2-w1-88bbf7-kserve-5457cdcdbd-l6tfd" podUID="a397931b-faa9-4c2e-8290-491350546663" containerName="main" probeResult="failure" output="Get \"https://10.132.0.85:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 01:05:08.035047 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:05:08.034996 4939 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-f7680d/distinct-w2-f7680d-kserve-756f874b98-qnpv8" podUID="836753fc-f9ef-4f61-b3fe-e5b92b7ee27b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.70:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 01:05:08.224865 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:05:08.224809 4939 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-88bbf7/simulator-w1-88bbf7-kserve-5fbb669f8f-42hgd" podUID="83b38883-2e7a-444d-aa68-52be4a25ae66" containerName="main" probeResult="failure" output="Get \"https://10.132.0.62:8000/health\": context deadline exceeded" Nov 03 01:05:08.236991 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:05:08.236899 4939 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-88bbf7/simulator-w1-88bbf7-kserve-5fbb669f8f-42hgd" podUID="83b38883-2e7a-444d-aa68-52be4a25ae66" containerName="main" probeResult="failure" output="Get \"https://10.132.0.62:8000/ready\": context deadline exceeded" Nov 03 01:05:08.396940 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:05:08.393098 4939 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-f7680d/distinct-w2-f7680d-kserve-756f874b98-qnpv8" podUID="836753fc-f9ef-4f61-b3fe-e5b92b7ee27b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.70:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 01:05:08.443564 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:05:08.443506 4939 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-f7680d/distinct-2-w2-f7680d-kserve-7d7c4564bf-cbv2c" podUID="2de18086-761b-4773-b9fc-15ddc4a220fc" containerName="main" probeResult="failure" output="Get \"https://10.132.0.75:8000/ready\": context deadline exceeded" Nov 03 01:05:08.652985 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:05:08.644491 4939 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-2f9a3a/simulator-w0-2f9a3a-kserve-9596bbb8f-2trgz" Nov 03 01:05:08.678183 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:05:08.678143 4939 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-6c0ec0-65c5d46767-hsxz4" Nov 03 01:05:08.736932 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:05:08.667876 4939 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-2f9a3a/distinct-w0-2f9a3a-kserve-555565cfcf-jtcr9" podUID="ac0ece06-8718-463f-915f-5315988d9711" containerName="main" probeResult="failure" output="Get \"https://10.132.0.76:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 01:05:08.901076 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:05:08.755355 4939 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-f7680d/premium-w2-f7680d-kserve-76565b8cf9-hqbbt" podUID="4712e708-18a2-4a13-a12c-a5669397b2c3" containerName="main" probeResult="failure" output="Get \"https://10.132.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 01:05:08.929030 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:05:08.846481 4939 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-83755a/distinct-2-w3-83755a-kserve-6cf66b995d-2l47b" podUID="c3b05e88-5d5b-4d03-b28d-832f22e3c868" containerName="main" probeResult="failure" output="Get \"https://10.132.0.72:8000/ready\": context deadline exceeded" Nov 03 01:05:09.151863 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:05:09.151827 4939 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-2f9a3a/distinct-w0-2f9a3a-kserve-555565cfcf-jtcr9" Nov 03 01:05:09.220424 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:05:09.205894 4939 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-88bbf7/distinct-2-w1-88bbf7-kserve-5457cdcdbd-l6tfd" Nov 03 01:05:09.231815 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:05:09.231783 4939 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-f7680d/distinct-2-w2-f7680d-kserve-7d7c4564bf-cbv2c" Nov 03 01:05:09.285408 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:05:09.285368 4939 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-88bbf7/simulator-w1-88bbf7-kserve-5fbb669f8f-42hgd" podUID="83b38883-2e7a-444d-aa68-52be4a25ae66" containerName="main" probeResult="failure" output="Get \"https://10.132.0.62:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Nov 03 01:05:09.295667 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:05:09.295636 4939 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-f7680d/distinct-w2-f7680d-kserve-756f874b98-qnpv8" Nov 03 01:05:09.296863 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:05:09.296839 4939 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-f7680d/premium-w2-f7680d-kserve-76565b8cf9-hqbbt" Nov 03 01:05:09.298025 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:05:09.297993 4939 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-83755a/distinct-2-w3-83755a-kserve-6cf66b995d-2l47b" Nov 03 01:05:09.343620 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:05:09.343584 4939 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-88bbf7/simulator-w1-88bbf7-kserve-5fbb669f8f-42hgd" Nov 03 01:05:09.890641 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:05:09.890603 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:05:11.096818 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:05:11.096776 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:05:11.256728 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:05:11.256690 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:05:12.395583 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:05:12.395550 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:05:12.575388 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:05:12.575339 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:05:12.738980 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:05:12.738217 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:05:13.987726 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:05:13.987692 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:05:14.391840 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:05:14.391805 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:05:14.803689 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:05:14.803599 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:05:14.963432 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:05:14.963399 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:05:15.279742 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:05:15.279709 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:05:15.416653 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:05:15.416600 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:05:15.786241 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:05:15.786200 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:05:16.329514 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:05:16.329385 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:05:16.556872 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:05:16.556828 4939 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-717217-54c569b974-sxbg4"] Nov 03 01:05:16.558871 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:05:16.558841 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:05:16.561813 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:05:16.561793 4939 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-717217-54c569b974-sxbg4" Nov 03 01:05:16.576620 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:05:16.576595 4939 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-a-w3-717217-serving-cert\"" Nov 03 01:05:16.580215 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:05:16.580152 4939 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-717217-54c569b974-sxbg4"] Nov 03 01:05:16.735925 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:05:16.735866 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/01da212a-c344-4e25-9f26-ef2224e32b46-metrics-certs\") pod \"maas-api-e2e-shared-a-w3-717217-54c569b974-sxbg4\" (UID: \"01da212a-c344-4e25-9f26-ef2224e32b46\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-717217-54c569b974-sxbg4" Nov 03 01:05:16.736352 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:05:16.736326 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7rwmk\" (UniqueName: \"kubernetes.io/projected/01da212a-c344-4e25-9f26-ef2224e32b46-kube-api-access-7rwmk\") pod \"maas-api-e2e-shared-a-w3-717217-54c569b974-sxbg4\" (UID: \"01da212a-c344-4e25-9f26-ef2224e32b46\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-717217-54c569b974-sxbg4" Nov 03 01:05:16.736691 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:05:16.736669 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/01da212a-c344-4e25-9f26-ef2224e32b46-maas-api-tls\") pod \"maas-api-e2e-shared-a-w3-717217-54c569b974-sxbg4\" (UID: \"01da212a-c344-4e25-9f26-ef2224e32b46\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-717217-54c569b974-sxbg4" Nov 03 01:05:16.736808 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:05:16.736793 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/01da212a-c344-4e25-9f26-ef2224e32b46-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w3-717217-54c569b974-sxbg4\" (UID: \"01da212a-c344-4e25-9f26-ef2224e32b46\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-717217-54c569b974-sxbg4" Nov 03 01:05:16.737624 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:05:16.737597 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:05:16.838160 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:05:16.838126 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/01da212a-c344-4e25-9f26-ef2224e32b46-maas-api-tls\") pod \"maas-api-e2e-shared-a-w3-717217-54c569b974-sxbg4\" (UID: \"01da212a-c344-4e25-9f26-ef2224e32b46\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-717217-54c569b974-sxbg4" Nov 03 01:05:16.838322 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:05:16.838169 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/01da212a-c344-4e25-9f26-ef2224e32b46-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w3-717217-54c569b974-sxbg4\" (UID: \"01da212a-c344-4e25-9f26-ef2224e32b46\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-717217-54c569b974-sxbg4" Nov 03 01:05:16.838322 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:05:16.838228 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/01da212a-c344-4e25-9f26-ef2224e32b46-metrics-certs\") pod \"maas-api-e2e-shared-a-w3-717217-54c569b974-sxbg4\" (UID: \"01da212a-c344-4e25-9f26-ef2224e32b46\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-717217-54c569b974-sxbg4" Nov 03 01:05:16.838416 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:05:16.838402 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7rwmk\" (UniqueName: \"kubernetes.io/projected/01da212a-c344-4e25-9f26-ef2224e32b46-kube-api-access-7rwmk\") pod \"maas-api-e2e-shared-a-w3-717217-54c569b974-sxbg4\" (UID: \"01da212a-c344-4e25-9f26-ef2224e32b46\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-717217-54c569b974-sxbg4" Nov 03 01:05:16.839044 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:05:16.839013 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/01da212a-c344-4e25-9f26-ef2224e32b46-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w3-717217-54c569b974-sxbg4\" (UID: \"01da212a-c344-4e25-9f26-ef2224e32b46\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-717217-54c569b974-sxbg4" Nov 03 01:05:16.840742 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:05:16.840714 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/01da212a-c344-4e25-9f26-ef2224e32b46-metrics-certs\") pod \"maas-api-e2e-shared-a-w3-717217-54c569b974-sxbg4\" (UID: \"01da212a-c344-4e25-9f26-ef2224e32b46\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-717217-54c569b974-sxbg4" Nov 03 01:05:16.840875 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:05:16.840818 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/01da212a-c344-4e25-9f26-ef2224e32b46-maas-api-tls\") pod \"maas-api-e2e-shared-a-w3-717217-54c569b974-sxbg4\" (UID: \"01da212a-c344-4e25-9f26-ef2224e32b46\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-717217-54c569b974-sxbg4" Nov 03 01:05:16.856789 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:05:16.856761 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7rwmk\" (UniqueName: \"kubernetes.io/projected/01da212a-c344-4e25-9f26-ef2224e32b46-kube-api-access-7rwmk\") pod \"maas-api-e2e-shared-a-w3-717217-54c569b974-sxbg4\" (UID: \"01da212a-c344-4e25-9f26-ef2224e32b46\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-717217-54c569b974-sxbg4" Nov 03 01:05:16.876930 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:05:16.876895 4939 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-717217-54c569b974-sxbg4" Nov 03 01:05:16.901622 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:05:16.901592 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:05:17.120022 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:05:17.119942 4939 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w4-4a697f-openshift-default-666db9f8f5-qn8tj"] Nov 03 01:05:17.125467 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:05:17.125437 4939 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w4-4a697f-openshift-default-666db9f8f5-qn8tj" Nov 03 01:05:17.131203 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:05:17.131174 4939 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-w4-4a697f-openshift-default-dockercfg-wr6rf\"" Nov 03 01:05:17.157637 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:05:17.157609 4939 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w4-4a697f-openshift-default-666db9f8f5-qn8tj"] Nov 03 01:05:17.172727 ip-10-0-140-107 kubenswrapper[4939]: W1103 01:05:17.172699 4939 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod01da212a_c344_4e25_9f26_ef2224e32b46.slice/crio-18b46cbc88d7a3e05c2e104e437ac2b344c6b6d5845e221a4cc6247657ca2937 WatchSource:0}: Error finding container 18b46cbc88d7a3e05c2e104e437ac2b344c6b6d5845e221a4cc6247657ca2937: Status 404 returned error can't find the container with id 18b46cbc88d7a3e05c2e104e437ac2b344c6b6d5845e221a4cc6247657ca2937 Nov 03 01:05:17.172884 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:05:17.172868 4939 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-717217-54c569b974-sxbg4"] Nov 03 01:05:17.244456 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:05:17.244412 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5b0cabc6-4960-4dc1-8317-3faebf25b30c-istiod-ca-cert\") pod \"e2e-shared-b-w4-4a697f-openshift-default-666db9f8f5-qn8tj\" (UID: \"5b0cabc6-4960-4dc1-8317-3faebf25b30c\") " pod="openshift-ingress/e2e-shared-b-w4-4a697f-openshift-default-666db9f8f5-qn8tj" Nov 03 01:05:17.244623 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:05:17.244467 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5b0cabc6-4960-4dc1-8317-3faebf25b30c-istio-envoy\") pod \"e2e-shared-b-w4-4a697f-openshift-default-666db9f8f5-qn8tj\" (UID: \"5b0cabc6-4960-4dc1-8317-3faebf25b30c\") " pod="openshift-ingress/e2e-shared-b-w4-4a697f-openshift-default-666db9f8f5-qn8tj" Nov 03 01:05:17.244623 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:05:17.244541 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-29zgn\" (UniqueName: \"kubernetes.io/projected/5b0cabc6-4960-4dc1-8317-3faebf25b30c-kube-api-access-29zgn\") pod \"e2e-shared-b-w4-4a697f-openshift-default-666db9f8f5-qn8tj\" (UID: \"5b0cabc6-4960-4dc1-8317-3faebf25b30c\") " pod="openshift-ingress/e2e-shared-b-w4-4a697f-openshift-default-666db9f8f5-qn8tj" Nov 03 01:05:17.244623 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:05:17.244597 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5b0cabc6-4960-4dc1-8317-3faebf25b30c-credential-socket\") pod \"e2e-shared-b-w4-4a697f-openshift-default-666db9f8f5-qn8tj\" (UID: \"5b0cabc6-4960-4dc1-8317-3faebf25b30c\") " pod="openshift-ingress/e2e-shared-b-w4-4a697f-openshift-default-666db9f8f5-qn8tj" Nov 03 01:05:17.244828 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:05:17.244806 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5b0cabc6-4960-4dc1-8317-3faebf25b30c-workload-socket\") pod \"e2e-shared-b-w4-4a697f-openshift-default-666db9f8f5-qn8tj\" (UID: \"5b0cabc6-4960-4dc1-8317-3faebf25b30c\") " pod="openshift-ingress/e2e-shared-b-w4-4a697f-openshift-default-666db9f8f5-qn8tj" Nov 03 01:05:17.244957 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:05:17.244904 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5b0cabc6-4960-4dc1-8317-3faebf25b30c-workload-certs\") pod \"e2e-shared-b-w4-4a697f-openshift-default-666db9f8f5-qn8tj\" (UID: \"5b0cabc6-4960-4dc1-8317-3faebf25b30c\") " pod="openshift-ingress/e2e-shared-b-w4-4a697f-openshift-default-666db9f8f5-qn8tj" Nov 03 01:05:17.245040 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:05:17.244978 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5b0cabc6-4960-4dc1-8317-3faebf25b30c-istio-data\") pod \"e2e-shared-b-w4-4a697f-openshift-default-666db9f8f5-qn8tj\" (UID: \"5b0cabc6-4960-4dc1-8317-3faebf25b30c\") " pod="openshift-ingress/e2e-shared-b-w4-4a697f-openshift-default-666db9f8f5-qn8tj" Nov 03 01:05:17.245103 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:05:17.245042 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5b0cabc6-4960-4dc1-8317-3faebf25b30c-istio-token\") pod \"e2e-shared-b-w4-4a697f-openshift-default-666db9f8f5-qn8tj\" (UID: \"5b0cabc6-4960-4dc1-8317-3faebf25b30c\") " pod="openshift-ingress/e2e-shared-b-w4-4a697f-openshift-default-666db9f8f5-qn8tj" Nov 03 01:05:17.245103 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:05:17.245095 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5b0cabc6-4960-4dc1-8317-3faebf25b30c-istio-podinfo\") pod \"e2e-shared-b-w4-4a697f-openshift-default-666db9f8f5-qn8tj\" (UID: \"5b0cabc6-4960-4dc1-8317-3faebf25b30c\") " pod="openshift-ingress/e2e-shared-b-w4-4a697f-openshift-default-666db9f8f5-qn8tj" Nov 03 01:05:17.284086 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:05:17.284044 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:05:17.346039 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:05:17.345998 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5b0cabc6-4960-4dc1-8317-3faebf25b30c-istiod-ca-cert\") pod \"e2e-shared-b-w4-4a697f-openshift-default-666db9f8f5-qn8tj\" (UID: \"5b0cabc6-4960-4dc1-8317-3faebf25b30c\") " pod="openshift-ingress/e2e-shared-b-w4-4a697f-openshift-default-666db9f8f5-qn8tj" Nov 03 01:05:17.346531 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:05:17.346056 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5b0cabc6-4960-4dc1-8317-3faebf25b30c-istio-envoy\") pod \"e2e-shared-b-w4-4a697f-openshift-default-666db9f8f5-qn8tj\" (UID: \"5b0cabc6-4960-4dc1-8317-3faebf25b30c\") " pod="openshift-ingress/e2e-shared-b-w4-4a697f-openshift-default-666db9f8f5-qn8tj" Nov 03 01:05:17.346531 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:05:17.346088 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-29zgn\" (UniqueName: \"kubernetes.io/projected/5b0cabc6-4960-4dc1-8317-3faebf25b30c-kube-api-access-29zgn\") pod \"e2e-shared-b-w4-4a697f-openshift-default-666db9f8f5-qn8tj\" (UID: \"5b0cabc6-4960-4dc1-8317-3faebf25b30c\") " pod="openshift-ingress/e2e-shared-b-w4-4a697f-openshift-default-666db9f8f5-qn8tj" Nov 03 01:05:17.346531 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:05:17.346152 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5b0cabc6-4960-4dc1-8317-3faebf25b30c-credential-socket\") pod \"e2e-shared-b-w4-4a697f-openshift-default-666db9f8f5-qn8tj\" (UID: \"5b0cabc6-4960-4dc1-8317-3faebf25b30c\") " pod="openshift-ingress/e2e-shared-b-w4-4a697f-openshift-default-666db9f8f5-qn8tj" Nov 03 01:05:17.346531 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:05:17.346279 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5b0cabc6-4960-4dc1-8317-3faebf25b30c-workload-socket\") pod \"e2e-shared-b-w4-4a697f-openshift-default-666db9f8f5-qn8tj\" (UID: \"5b0cabc6-4960-4dc1-8317-3faebf25b30c\") " pod="openshift-ingress/e2e-shared-b-w4-4a697f-openshift-default-666db9f8f5-qn8tj" Nov 03 01:05:17.346531 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:05:17.346346 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5b0cabc6-4960-4dc1-8317-3faebf25b30c-workload-certs\") pod \"e2e-shared-b-w4-4a697f-openshift-default-666db9f8f5-qn8tj\" (UID: \"5b0cabc6-4960-4dc1-8317-3faebf25b30c\") " pod="openshift-ingress/e2e-shared-b-w4-4a697f-openshift-default-666db9f8f5-qn8tj" Nov 03 01:05:17.346531 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:05:17.346370 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5b0cabc6-4960-4dc1-8317-3faebf25b30c-istio-data\") pod \"e2e-shared-b-w4-4a697f-openshift-default-666db9f8f5-qn8tj\" (UID: \"5b0cabc6-4960-4dc1-8317-3faebf25b30c\") " pod="openshift-ingress/e2e-shared-b-w4-4a697f-openshift-default-666db9f8f5-qn8tj" Nov 03 01:05:17.346531 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:05:17.346442 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5b0cabc6-4960-4dc1-8317-3faebf25b30c-istio-token\") pod \"e2e-shared-b-w4-4a697f-openshift-default-666db9f8f5-qn8tj\" (UID: \"5b0cabc6-4960-4dc1-8317-3faebf25b30c\") " pod="openshift-ingress/e2e-shared-b-w4-4a697f-openshift-default-666db9f8f5-qn8tj" Nov 03 01:05:17.346531 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:05:17.346496 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5b0cabc6-4960-4dc1-8317-3faebf25b30c-istio-podinfo\") pod \"e2e-shared-b-w4-4a697f-openshift-default-666db9f8f5-qn8tj\" (UID: \"5b0cabc6-4960-4dc1-8317-3faebf25b30c\") " pod="openshift-ingress/e2e-shared-b-w4-4a697f-openshift-default-666db9f8f5-qn8tj" Nov 03 01:05:17.347059 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:05:17.346633 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5b0cabc6-4960-4dc1-8317-3faebf25b30c-credential-socket\") pod \"e2e-shared-b-w4-4a697f-openshift-default-666db9f8f5-qn8tj\" (UID: \"5b0cabc6-4960-4dc1-8317-3faebf25b30c\") " pod="openshift-ingress/e2e-shared-b-w4-4a697f-openshift-default-666db9f8f5-qn8tj" Nov 03 01:05:17.347059 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:05:17.346734 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5b0cabc6-4960-4dc1-8317-3faebf25b30c-workload-socket\") pod \"e2e-shared-b-w4-4a697f-openshift-default-666db9f8f5-qn8tj\" (UID: \"5b0cabc6-4960-4dc1-8317-3faebf25b30c\") " pod="openshift-ingress/e2e-shared-b-w4-4a697f-openshift-default-666db9f8f5-qn8tj" Nov 03 01:05:17.347059 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:05:17.346856 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5b0cabc6-4960-4dc1-8317-3faebf25b30c-workload-certs\") pod \"e2e-shared-b-w4-4a697f-openshift-default-666db9f8f5-qn8tj\" (UID: \"5b0cabc6-4960-4dc1-8317-3faebf25b30c\") " pod="openshift-ingress/e2e-shared-b-w4-4a697f-openshift-default-666db9f8f5-qn8tj" Nov 03 01:05:17.347059 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:05:17.346974 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5b0cabc6-4960-4dc1-8317-3faebf25b30c-istiod-ca-cert\") pod \"e2e-shared-b-w4-4a697f-openshift-default-666db9f8f5-qn8tj\" (UID: \"5b0cabc6-4960-4dc1-8317-3faebf25b30c\") " pod="openshift-ingress/e2e-shared-b-w4-4a697f-openshift-default-666db9f8f5-qn8tj" Nov 03 01:05:17.347059 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:05:17.347015 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5b0cabc6-4960-4dc1-8317-3faebf25b30c-istio-data\") pod \"e2e-shared-b-w4-4a697f-openshift-default-666db9f8f5-qn8tj\" (UID: \"5b0cabc6-4960-4dc1-8317-3faebf25b30c\") " pod="openshift-ingress/e2e-shared-b-w4-4a697f-openshift-default-666db9f8f5-qn8tj" Nov 03 01:05:17.348681 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:05:17.348655 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5b0cabc6-4960-4dc1-8317-3faebf25b30c-istio-envoy\") pod \"e2e-shared-b-w4-4a697f-openshift-default-666db9f8f5-qn8tj\" (UID: \"5b0cabc6-4960-4dc1-8317-3faebf25b30c\") " pod="openshift-ingress/e2e-shared-b-w4-4a697f-openshift-default-666db9f8f5-qn8tj" Nov 03 01:05:17.349025 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:05:17.349008 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5b0cabc6-4960-4dc1-8317-3faebf25b30c-istio-podinfo\") pod \"e2e-shared-b-w4-4a697f-openshift-default-666db9f8f5-qn8tj\" (UID: \"5b0cabc6-4960-4dc1-8317-3faebf25b30c\") " pod="openshift-ingress/e2e-shared-b-w4-4a697f-openshift-default-666db9f8f5-qn8tj" Nov 03 01:05:17.360235 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:05:17.360205 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5b0cabc6-4960-4dc1-8317-3faebf25b30c-istio-token\") pod \"e2e-shared-b-w4-4a697f-openshift-default-666db9f8f5-qn8tj\" (UID: \"5b0cabc6-4960-4dc1-8317-3faebf25b30c\") " pod="openshift-ingress/e2e-shared-b-w4-4a697f-openshift-default-666db9f8f5-qn8tj" Nov 03 01:05:17.360363 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:05:17.360349 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-29zgn\" (UniqueName: \"kubernetes.io/projected/5b0cabc6-4960-4dc1-8317-3faebf25b30c-kube-api-access-29zgn\") pod \"e2e-shared-b-w4-4a697f-openshift-default-666db9f8f5-qn8tj\" (UID: \"5b0cabc6-4960-4dc1-8317-3faebf25b30c\") " pod="openshift-ingress/e2e-shared-b-w4-4a697f-openshift-default-666db9f8f5-qn8tj" Nov 03 01:05:17.441772 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:05:17.441690 4939 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w4-4a697f-openshift-default-666db9f8f5-qn8tj" Nov 03 01:05:17.446538 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:05:17.446500 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:05:17.824062 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:05:17.823924 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:05:17.977069 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:05:17.977017 4939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-717217-54c569b974-sxbg4" event={"ID":"01da212a-c344-4e25-9f26-ef2224e32b46","Type":"ContainerStarted","Data":"18b46cbc88d7a3e05c2e104e437ac2b344c6b6d5845e221a4cc6247657ca2937"} Nov 03 01:05:17.985863 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:05:17.985331 4939 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w4-4a697f-openshift-default-666db9f8f5-qn8tj"] Nov 03 01:05:17.991696 ip-10-0-140-107 kubenswrapper[4939]: W1103 01:05:17.991473 4939 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5b0cabc6_4960_4dc1_8317_3faebf25b30c.slice/crio-137d8b8a9ddc5a288b3c4eb85967de7675b8cc390343656575bf6dd655888030 WatchSource:0}: Error finding container 137d8b8a9ddc5a288b3c4eb85967de7675b8cc390343656575bf6dd655888030: Status 404 returned error can't find the container with id 137d8b8a9ddc5a288b3c4eb85967de7675b8cc390343656575bf6dd655888030 Nov 03 01:05:17.995425 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:05:17.995193 4939 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 01:05:17.995425 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:05:17.995287 4939 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 01:05:17.995425 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:05:17.995336 4939 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 01:05:18.030966 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:05:18.030896 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:05:18.034580 ip-10-0-140-107 kubenswrapper[4939]: E1103 01:05:18.034499 4939 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-sthjh" containerName="extractor" Nov 03 01:05:18.436823 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:05:18.436759 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:05:18.583142 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:05:18.583102 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:05:18.961463 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:05:18.961429 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:05:18.990080 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:05:18.990039 4939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-717217-54c569b974-sxbg4" event={"ID":"01da212a-c344-4e25-9f26-ef2224e32b46","Type":"ContainerStarted","Data":"3b873459106f64fde040b9ea0ae6399541cb945be8d76a3cc4103114a497ee9c"} Nov 03 01:05:18.990368 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:05:18.990340 4939 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-717217-54c569b974-sxbg4" Nov 03 01:05:18.992896 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:05:18.992862 4939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w4-4a697f-openshift-default-666db9f8f5-qn8tj" event={"ID":"5b0cabc6-4960-4dc1-8317-3faebf25b30c","Type":"ContainerStarted","Data":"2f9dce65eb234b7acf638877b722bc4efa6182ce79db38c29344f4c05eee7784"} Nov 03 01:05:18.993065 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:05:18.992957 4939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w4-4a697f-openshift-default-666db9f8f5-qn8tj" event={"ID":"5b0cabc6-4960-4dc1-8317-3faebf25b30c","Type":"ContainerStarted","Data":"137d8b8a9ddc5a288b3c4eb85967de7675b8cc390343656575bf6dd655888030"} Nov 03 01:05:19.026171 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:05:19.026123 4939 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-717217-54c569b974-sxbg4" podStartSLOduration=2.211249172 podStartE2EDuration="3.026109804s" totalImagesPullingTime="814.860632ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 01:05:16 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 01:05:19.022705715 +0000 UTC m=+1467.287040660" watchObservedRunningTime="2026-11-03 01:05:19.026109804 +0000 UTC m=+1467.290444748" Nov 03 01:05:19.090824 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:05:19.090774 4939 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w4-4a697f-openshift-default-666db9f8f5-qn8tj" podStartSLOduration=2.090760698 podStartE2EDuration="2.090760698s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 01:05:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 01:05:19.090556391 +0000 UTC m=+1467.354891336" watchObservedRunningTime="2026-11-03 01:05:19.090760698 +0000 UTC m=+1467.355095643" Nov 03 01:05:19.255978 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:05:19.255301 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:05:19.478930 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:05:19.462261 4939 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w4-4a697f-openshift-default-666db9f8f5-qn8tj" Nov 03 01:05:19.485219 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:05:19.479716 4939 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w4-4a697f-openshift-default-666db9f8f5-qn8tj" Nov 03 01:05:19.893845 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:05:19.893804 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:05:20.029930 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:05:20.024283 4939 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w4-4a697f-openshift-default-666db9f8f5-qn8tj" Nov 03 01:05:20.039954 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:05:20.033664 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:05:20.039954 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:05:20.037702 4939 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w4-4a697f-openshift-default-666db9f8f5-qn8tj" Nov 03 01:05:20.152002 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:05:20.151886 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:05:20.343180 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:05:20.343145 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:05:20.462834 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:05:20.462745 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:05:21.211081 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:05:21.211040 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:05:21.427897 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:05:21.427836 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:05:21.557671 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:05:21.557573 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:05:22.315522 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:05:22.315487 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:05:22.496770 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:05:22.496735 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:05:22.767672 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:05:22.767641 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:05:22.808342 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:05:22.808307 4939 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w4-4a697f-94d8d8fc9-kzr29"] Nov 03 01:05:22.814548 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:05:22.814523 4939 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-4a697f-94d8d8fc9-kzr29" Nov 03 01:05:22.819605 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:05:22.819580 4939 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w4-4a697f-94d8d8fc9-kzr29"] Nov 03 01:05:22.822472 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:05:22.822445 4939 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-b-w4-4a697f\"" Nov 03 01:05:22.825393 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:05:22.824608 4939 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-b-w4-4a697f-dockercfg-bltnb\"" Nov 03 01:05:22.864584 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:05:22.864543 4939 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w4-4a697f-79fb8d7d9-gr8cv"] Nov 03 01:05:22.870071 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:05:22.870049 4939 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-4a697f-79fb8d7d9-gr8cv" Nov 03 01:05:22.908929 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:05:22.883601 4939 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w4-4a697f-79fb8d7d9-gr8cv"] Nov 03 01:05:22.908929 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:05:22.892424 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:05:22.942935 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:05:22.940213 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t8s9l\" (UniqueName: \"kubernetes.io/projected/ef71e635-99c1-4f1c-99b0-014934ba2f76-kube-api-access-t8s9l\") pod \"payload-pre-processing-e2e-shared-b-w4-4a697f-94d8d8fc9-kzr29\" (UID: \"ef71e635-99c1-4f1c-99b0-014934ba2f76\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-4a697f-94d8d8fc9-kzr29" Nov 03 01:05:22.942935 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:05:22.940311 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ef71e635-99c1-4f1c-99b0-014934ba2f76-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w4-4a697f-94d8d8fc9-kzr29\" (UID: \"ef71e635-99c1-4f1c-99b0-014934ba2f76\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-4a697f-94d8d8fc9-kzr29" Nov 03 01:05:23.047989 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:05:23.044410 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ef6478d3-bd62-45a7-8acc-1055c24dc53b-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w4-4a697f-79fb8d7d9-gr8cv\" (UID: \"ef6478d3-bd62-45a7-8acc-1055c24dc53b\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w4-4a697f-79fb8d7d9-gr8cv" Nov 03 01:05:23.047989 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:05:23.044502 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t8s9l\" (UniqueName: \"kubernetes.io/projected/ef71e635-99c1-4f1c-99b0-014934ba2f76-kube-api-access-t8s9l\") pod \"payload-pre-processing-e2e-shared-b-w4-4a697f-94d8d8fc9-kzr29\" (UID: \"ef71e635-99c1-4f1c-99b0-014934ba2f76\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-4a697f-94d8d8fc9-kzr29" Nov 03 01:05:23.047989 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:05:23.044534 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hrtqs\" (UniqueName: \"kubernetes.io/projected/ef6478d3-bd62-45a7-8acc-1055c24dc53b-kube-api-access-hrtqs\") pod \"payload-processing-e2e-shared-b-w4-4a697f-79fb8d7d9-gr8cv\" (UID: \"ef6478d3-bd62-45a7-8acc-1055c24dc53b\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w4-4a697f-79fb8d7d9-gr8cv" Nov 03 01:05:23.047989 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:05:23.044602 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ef71e635-99c1-4f1c-99b0-014934ba2f76-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w4-4a697f-94d8d8fc9-kzr29\" (UID: \"ef71e635-99c1-4f1c-99b0-014934ba2f76\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-4a697f-94d8d8fc9-kzr29" Nov 03 01:05:23.047989 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:05:23.045806 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ef71e635-99c1-4f1c-99b0-014934ba2f76-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w4-4a697f-94d8d8fc9-kzr29\" (UID: \"ef71e635-99c1-4f1c-99b0-014934ba2f76\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-4a697f-94d8d8fc9-kzr29" Nov 03 01:05:23.171849 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:05:23.171802 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ef6478d3-bd62-45a7-8acc-1055c24dc53b-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w4-4a697f-79fb8d7d9-gr8cv\" (UID: \"ef6478d3-bd62-45a7-8acc-1055c24dc53b\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w4-4a697f-79fb8d7d9-gr8cv" Nov 03 01:05:23.193947 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:05:23.191414 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hrtqs\" (UniqueName: \"kubernetes.io/projected/ef6478d3-bd62-45a7-8acc-1055c24dc53b-kube-api-access-hrtqs\") pod \"payload-processing-e2e-shared-b-w4-4a697f-79fb8d7d9-gr8cv\" (UID: \"ef6478d3-bd62-45a7-8acc-1055c24dc53b\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w4-4a697f-79fb8d7d9-gr8cv" Nov 03 01:05:23.193947 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:05:23.175179 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-t8s9l\" (UniqueName: \"kubernetes.io/projected/ef71e635-99c1-4f1c-99b0-014934ba2f76-kube-api-access-t8s9l\") pod \"payload-pre-processing-e2e-shared-b-w4-4a697f-94d8d8fc9-kzr29\" (UID: \"ef71e635-99c1-4f1c-99b0-014934ba2f76\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-4a697f-94d8d8fc9-kzr29" Nov 03 01:05:23.199942 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:05:23.196748 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ef6478d3-bd62-45a7-8acc-1055c24dc53b-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w4-4a697f-79fb8d7d9-gr8cv\" (UID: \"ef6478d3-bd62-45a7-8acc-1055c24dc53b\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w4-4a697f-79fb8d7d9-gr8cv" Nov 03 01:05:23.210651 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:05:23.210626 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hrtqs\" (UniqueName: \"kubernetes.io/projected/ef6478d3-bd62-45a7-8acc-1055c24dc53b-kube-api-access-hrtqs\") pod \"payload-processing-e2e-shared-b-w4-4a697f-79fb8d7d9-gr8cv\" (UID: \"ef6478d3-bd62-45a7-8acc-1055c24dc53b\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w4-4a697f-79fb8d7d9-gr8cv" Nov 03 01:05:23.434286 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:05:23.434253 4939 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-4a697f-94d8d8fc9-kzr29" Nov 03 01:05:23.491165 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:05:23.490580 4939 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-4a697f-79fb8d7d9-gr8cv" Nov 03 01:05:23.538560 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:05:23.538483 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:05:23.717903 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:05:23.717493 4939 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w4-4a697f-94d8d8fc9-kzr29"] Nov 03 01:05:23.820154 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:05:23.820122 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:05:23.856364 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:05:23.856338 4939 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w0-50e623-746d8fcf47-tjbwb"] Nov 03 01:05:23.871095 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:05:23.866883 4939 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-50e623-746d8fcf47-tjbwb" Nov 03 01:05:23.891891 ip-10-0-140-107 kubenswrapper[4939]: W1103 01:05:23.891838 4939 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podef6478d3_bd62_45a7_8acc_1055c24dc53b.slice/crio-a7dd1eeb8839df04e75fa07242029ec7dfca0cfaa3efaaaf034314d3b1980c80 WatchSource:0}: Error finding container a7dd1eeb8839df04e75fa07242029ec7dfca0cfaa3efaaaf034314d3b1980c80: Status 404 returned error can't find the container with id a7dd1eeb8839df04e75fa07242029ec7dfca0cfaa3efaaaf034314d3b1980c80 Nov 03 01:05:23.897488 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:05:23.897464 4939 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w0-50e623-746d8fcf47-tjbwb"] Nov 03 01:05:23.897610 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:05:23.897509 4939 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w4-4a697f-79fb8d7d9-gr8cv"] Nov 03 01:05:23.923043 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:05:23.923013 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b7d0152f-07a6-429b-8c20-e29139cf1d23-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w0-50e623-746d8fcf47-tjbwb\" (UID: \"b7d0152f-07a6-429b-8c20-e29139cf1d23\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-50e623-746d8fcf47-tjbwb" Nov 03 01:05:23.923175 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:05:23.923071 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s66zj\" (UniqueName: \"kubernetes.io/projected/b7d0152f-07a6-429b-8c20-e29139cf1d23-kube-api-access-s66zj\") pod \"payload-pre-processing-e2e-shared-a-w0-50e623-746d8fcf47-tjbwb\" (UID: \"b7d0152f-07a6-429b-8c20-e29139cf1d23\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-50e623-746d8fcf47-tjbwb" Nov 03 01:05:23.949354 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:05:23.943297 4939 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-a-w0-50e623-dockercfg-hwxhx\"" Nov 03 01:05:23.954210 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:05:23.954160 4939 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-a-w0-50e623\"" Nov 03 01:05:24.031949 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:05:24.029704 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b7d0152f-07a6-429b-8c20-e29139cf1d23-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w0-50e623-746d8fcf47-tjbwb\" (UID: \"b7d0152f-07a6-429b-8c20-e29139cf1d23\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-50e623-746d8fcf47-tjbwb" Nov 03 01:05:24.031949 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:05:24.029771 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s66zj\" (UniqueName: \"kubernetes.io/projected/b7d0152f-07a6-429b-8c20-e29139cf1d23-kube-api-access-s66zj\") pod \"payload-pre-processing-e2e-shared-a-w0-50e623-746d8fcf47-tjbwb\" (UID: \"b7d0152f-07a6-429b-8c20-e29139cf1d23\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-50e623-746d8fcf47-tjbwb" Nov 03 01:05:24.031949 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:05:24.031871 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b7d0152f-07a6-429b-8c20-e29139cf1d23-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w0-50e623-746d8fcf47-tjbwb\" (UID: \"b7d0152f-07a6-429b-8c20-e29139cf1d23\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-50e623-746d8fcf47-tjbwb" Nov 03 01:05:24.033782 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:05:24.033749 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:05:24.059414 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:05:24.059367 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-s66zj\" (UniqueName: \"kubernetes.io/projected/b7d0152f-07a6-429b-8c20-e29139cf1d23-kube-api-access-s66zj\") pod \"payload-pre-processing-e2e-shared-a-w0-50e623-746d8fcf47-tjbwb\" (UID: \"b7d0152f-07a6-429b-8c20-e29139cf1d23\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-50e623-746d8fcf47-tjbwb" Nov 03 01:05:24.200934 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:05:24.200664 4939 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-50e623-746d8fcf47-tjbwb" Nov 03 01:05:24.285014 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:05:24.284975 4939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-4a697f-79fb8d7d9-gr8cv" event={"ID":"ef6478d3-bd62-45a7-8acc-1055c24dc53b","Type":"ContainerStarted","Data":"4fcbd3d36db1cb376909b018910f367575ae43d65ae89c447ec2b35bc985516d"} Nov 03 01:05:24.285185 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:05:24.285043 4939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-4a697f-79fb8d7d9-gr8cv" event={"ID":"ef6478d3-bd62-45a7-8acc-1055c24dc53b","Type":"ContainerStarted","Data":"a7dd1eeb8839df04e75fa07242029ec7dfca0cfaa3efaaaf034314d3b1980c80"} Nov 03 01:05:24.286206 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:05:24.286182 4939 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-4a697f-79fb8d7d9-gr8cv" Nov 03 01:05:24.304678 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:05:24.304638 4939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-4a697f-94d8d8fc9-kzr29" event={"ID":"ef71e635-99c1-4f1c-99b0-014934ba2f76","Type":"ContainerStarted","Data":"b6bb749d739c14292a547641ed6d373e8323f531c716126beccd5591b304334d"} Nov 03 01:05:24.305943 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:05:24.305901 4939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-4a697f-94d8d8fc9-kzr29" event={"ID":"ef71e635-99c1-4f1c-99b0-014934ba2f76","Type":"ContainerStarted","Data":"c6e3b07ee331461205ce146f0224539ed704b267c2ab2c2e28efdce3bb775e82"} Nov 03 01:05:24.306798 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:05:24.306763 4939 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-4a697f-94d8d8fc9-kzr29" Nov 03 01:05:24.311958 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:05:24.311887 4939 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-4a697f-79fb8d7d9-gr8cv" podStartSLOduration=2.311868914 podStartE2EDuration="2.311868914s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 01:05:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 01:05:24.305360834 +0000 UTC m=+1472.569695781" watchObservedRunningTime="2026-11-03 01:05:24.311868914 +0000 UTC m=+1472.576203863" Nov 03 01:05:24.332824 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:05:24.332069 4939 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-4a697f-94d8d8fc9-kzr29" podStartSLOduration=2.33205251 podStartE2EDuration="2.33205251s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 01:05:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 01:05:24.327440542 +0000 UTC m=+1472.591775485" watchObservedRunningTime="2026-11-03 01:05:24.33205251 +0000 UTC m=+1472.596387456" Nov 03 01:05:24.481496 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:05:24.481458 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:05:24.587142 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:05:24.587108 4939 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w0-50e623-746d8fcf47-tjbwb"] Nov 03 01:05:24.587561 ip-10-0-140-107 kubenswrapper[4939]: W1103 01:05:24.587538 4939 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb7d0152f_07a6_429b_8c20_e29139cf1d23.slice/crio-5efdc889a686291f036f3ddc9dcc691a8ac1e44651aec47c511f775687ef1b7a WatchSource:0}: Error finding container 5efdc889a686291f036f3ddc9dcc691a8ac1e44651aec47c511f775687ef1b7a: Status 404 returned error can't find the container with id 5efdc889a686291f036f3ddc9dcc691a8ac1e44651aec47c511f775687ef1b7a Nov 03 01:05:25.066985 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:05:25.066148 4939 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-717217-54c569b974-sxbg4" Nov 03 01:05:25.520480 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:05:25.520436 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:05:25.557412 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:05:25.557353 4939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-50e623-746d8fcf47-tjbwb" event={"ID":"b7d0152f-07a6-429b-8c20-e29139cf1d23","Type":"ContainerStarted","Data":"16d8bdb1497555572a0251253d1caf45892687b5a545b6f7024a88b2c49ddaf0"} Nov 03 01:05:25.557590 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:05:25.557470 4939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-50e623-746d8fcf47-tjbwb" event={"ID":"b7d0152f-07a6-429b-8c20-e29139cf1d23","Type":"ContainerStarted","Data":"5efdc889a686291f036f3ddc9dcc691a8ac1e44651aec47c511f775687ef1b7a"} Nov 03 01:05:25.557590 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:05:25.557501 4939 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-50e623-746d8fcf47-tjbwb" Nov 03 01:05:25.577894 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:05:25.577825 4939 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-50e623-746d8fcf47-tjbwb" podStartSLOduration=2.577807537 podStartE2EDuration="2.577807537s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 01:05:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 01:05:25.574459845 +0000 UTC m=+1473.838794785" watchObservedRunningTime="2026-11-03 01:05:25.577807537 +0000 UTC m=+1473.842142484" Nov 03 01:05:27.046069 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:05:27.046013 4939 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-88bbf7/distinct-2-w1-88bbf7-kserve-5457cdcdbd-l6tfd" podUID="a397931b-faa9-4c2e-8290-491350546663" containerName="main" probeResult="failure" output="Get \"https://10.132.0.85:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 01:05:27.432190 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:05:27.432140 4939 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-f7680d/distinct-2-w2-f7680d-kserve-7d7c4564bf-cbv2c" podUID="2de18086-761b-4773-b9fc-15ddc4a220fc" containerName="main" probeResult="failure" output="Get \"https://10.132.0.75:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 01:05:27.677939 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:05:27.674496 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:05:27.738002 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:05:27.735007 4939 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-2f9a3a/simulator-w0-2f9a3a-kserve-9596bbb8f-2trgz" podUID="207e2207-fb6c-4076-95df-72e95671c261" containerName="main" probeResult="failure" output="Get \"https://10.132.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 01:05:27.762934 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:05:27.760199 4939 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-f7680d/premium-w2-f7680d-kserve-76565b8cf9-hqbbt" podUID="4712e708-18a2-4a13-a12c-a5669397b2c3" containerName="main" probeResult="failure" output="Get \"https://10.132.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 01:05:27.909146 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:05:27.909091 4939 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-2f9a3a/distinct-w0-2f9a3a-kserve-555565cfcf-jtcr9" podUID="ac0ece06-8718-463f-915f-5315988d9711" containerName="main" probeResult="failure" output="Get \"https://10.132.0.76:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 01:05:28.042993 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:05:28.024138 4939 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-f7680d/distinct-w2-f7680d-kserve-756f874b98-qnpv8" podUID="836753fc-f9ef-4f61-b3fe-e5b92b7ee27b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.70:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 01:05:28.061932 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:05:28.048581 4939 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-88bbf7/distinct-2-w1-88bbf7-kserve-5457cdcdbd-l6tfd" podUID="a397931b-faa9-4c2e-8290-491350546663" containerName="main" probeResult="failure" output="Get \"https://10.132.0.85:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 01:05:28.213937 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:05:28.213395 4939 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-88bbf7/simulator-w1-88bbf7-kserve-5fbb669f8f-42hgd" podUID="83b38883-2e7a-444d-aa68-52be4a25ae66" containerName="main" probeResult="failure" output="Get \"https://10.132.0.62:8000/health\": context deadline exceeded" Nov 03 01:05:28.280998 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:05:28.280140 4939 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-88bbf7/distinct-2-w1-88bbf7-kserve-5457cdcdbd-l6tfd" Nov 03 01:05:28.312043 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:05:28.306160 4939 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-88bbf7/simulator-w1-88bbf7-kserve-5fbb669f8f-42hgd" podUID="83b38883-2e7a-444d-aa68-52be4a25ae66" containerName="main" probeResult="failure" output="Get \"https://10.132.0.62:8000/ready\": context deadline exceeded" Nov 03 01:05:28.406943 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:05:28.402075 4939 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-83755a/distinct-2-w3-83755a-kserve-6cf66b995d-2l47b" podUID="c3b05e88-5d5b-4d03-b28d-832f22e3c868" containerName="main" probeResult="failure" output="Get \"https://10.132.0.72:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 01:05:28.416943 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:05:28.413999 4939 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-f7680d/distinct-w2-f7680d-kserve-756f874b98-qnpv8" podUID="836753fc-f9ef-4f61-b3fe-e5b92b7ee27b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.70:8000/ready\": context deadline exceeded" Nov 03 01:05:28.468966 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:05:28.436212 4939 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-f7680d/distinct-2-w2-f7680d-kserve-7d7c4564bf-cbv2c" podUID="2de18086-761b-4773-b9fc-15ddc4a220fc" containerName="main" probeResult="failure" output="Get \"https://10.132.0.75:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 01:05:28.471090 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:05:28.471056 4939 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-f7680d/distinct-2-w2-f7680d-kserve-7d7c4564bf-cbv2c" Nov 03 01:05:28.488576 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:05:28.484114 4939 patch_prober.go:29] interesting pod/e2e-worker-w3-83755a-openshift-default-c86f9565c-z7flk container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.166:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 01:05:28.504954 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:05:28.495827 4939 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w3-83755a-openshift-default-c86f9565c-z7flk" podUID="569c2920-1523-4da8-8644-f789aebcdc7a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.166:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 01:05:28.572710 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:05:28.511380 4939 patch_prober.go:29] interesting pod/e2e-worker-w3-83755a-openshift-default-c86f9565c-zj5jg container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.167:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 01:05:28.572710 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:05:28.511446 4939 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w3-83755a-openshift-default-c86f9565c-zj5jg" podUID="0dbecc93-5a61-4100-a319-d3f673f7a5fd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.167:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 01:05:28.770938 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:05:28.769064 4939 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-2f9a3a/simulator-w0-2f9a3a-kserve-9596bbb8f-2trgz" podUID="207e2207-fb6c-4076-95df-72e95671c261" containerName="main" probeResult="failure" output="Get \"https://10.132.0.61:8000/ready\": context deadline exceeded" Nov 03 01:05:28.770938 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:05:28.770670 4939 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-2f9a3a/simulator-w0-2f9a3a-kserve-9596bbb8f-2trgz" Nov 03 01:05:28.859989 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:05:28.823241 4939 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-f7680d/premium-w2-f7680d-kserve-76565b8cf9-hqbbt" podUID="4712e708-18a2-4a13-a12c-a5669397b2c3" containerName="main" probeResult="failure" output="Get \"https://10.132.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 01:05:28.859989 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:05:28.824206 4939 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-f7680d/premium-w2-f7680d-kserve-76565b8cf9-hqbbt" Nov 03 01:05:28.928932 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:05:28.918365 4939 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-2f9a3a/distinct-w0-2f9a3a-kserve-555565cfcf-jtcr9" podUID="ac0ece06-8718-463f-915f-5315988d9711" containerName="main" probeResult="failure" output="Get \"https://10.132.0.76:8000/ready\": context deadline exceeded" Nov 03 01:05:28.928932 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:05:28.919891 4939 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-2f9a3a/distinct-w0-2f9a3a-kserve-555565cfcf-jtcr9" Nov 03 01:05:29.076935 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:05:29.071985 4939 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-88bbf7/distinct-2-w1-88bbf7-kserve-5457cdcdbd-l6tfd" podUID="a397931b-faa9-4c2e-8290-491350546663" containerName="main" probeResult="failure" output="Get \"https://10.132.0.85:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 01:05:29.394079 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:05:29.360043 4939 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-88bbf7/simulator-w1-88bbf7-kserve-5fbb669f8f-42hgd" podUID="83b38883-2e7a-444d-aa68-52be4a25ae66" containerName="main" probeResult="failure" output="Get \"https://10.132.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 01:05:29.394079 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:05:29.361356 4939 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-88bbf7/simulator-w1-88bbf7-kserve-5fbb669f8f-42hgd" Nov 03 01:05:29.400962 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:05:29.395721 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:05:29.445067 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:05:29.427455 4939 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-f7680d/distinct-w2-f7680d-kserve-756f874b98-qnpv8" podUID="836753fc-f9ef-4f61-b3fe-e5b92b7ee27b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.70:8000/ready\": context deadline exceeded" Nov 03 01:05:29.445067 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:05:29.429533 4939 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-f7680d/distinct-w2-f7680d-kserve-756f874b98-qnpv8" Nov 03 01:05:29.460818 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:05:29.435272 4939 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-83755a/distinct-2-w3-83755a-kserve-6cf66b995d-2l47b" podUID="c3b05e88-5d5b-4d03-b28d-832f22e3c868" containerName="main" probeResult="failure" output="Get \"https://10.132.0.72:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 01:05:29.461859 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:05:29.461726 4939 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-83755a/distinct-2-w3-83755a-kserve-6cf66b995d-2l47b" Nov 03 01:05:29.472093 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:05:29.471166 4939 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-f7680d/distinct-2-w2-f7680d-kserve-7d7c4564bf-cbv2c" podUID="2de18086-761b-4773-b9fc-15ddc4a220fc" containerName="main" probeResult="failure" output="Get \"https://10.132.0.75:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 01:05:29.484532 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:05:29.484504 4939 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-83755a/distinct-2-w3-83755a-kserve-6cf66b995d-2l47b" Nov 03 01:05:29.793146 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:05:29.793017 4939 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-2f9a3a/simulator-w0-2f9a3a-kserve-9596bbb8f-2trgz" podUID="207e2207-fb6c-4076-95df-72e95671c261" containerName="main" probeResult="failure" output="Get \"https://10.132.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 01:05:29.826968 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:05:29.826891 4939 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-f7680d/premium-w2-f7680d-kserve-76565b8cf9-hqbbt" podUID="4712e708-18a2-4a13-a12c-a5669397b2c3" containerName="main" probeResult="failure" output="Get \"https://10.132.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 01:05:29.840932 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:05:29.838131 4939 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-f7680d/distinct-2-w2-f7680d-kserve-7d7c4564bf-cbv2c" Nov 03 01:05:29.891010 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:05:29.859454 4939 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-2f9a3a/distinct-w0-2f9a3a-kserve-555565cfcf-jtcr9" Nov 03 01:05:30.084296 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:05:30.079621 4939 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-88bbf7/distinct-2-w1-88bbf7-kserve-5457cdcdbd-l6tfd" podUID="a397931b-faa9-4c2e-8290-491350546663" containerName="main" probeResult="failure" output="Get \"https://10.132.0.85:8000/ready\": context deadline exceeded" Nov 03 01:05:30.123574 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:05:30.123537 4939 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-f7680d/premium-w2-f7680d-kserve-76565b8cf9-hqbbt" Nov 03 01:05:30.123844 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:05:30.123830 4939 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-2f9a3a/simulator-w0-2f9a3a-kserve-9596bbb8f-2trgz" Nov 03 01:05:30.124003 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:05:30.123991 4939 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-f7680d/distinct-w2-f7680d-kserve-756f874b98-qnpv8" Nov 03 01:05:30.124119 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:05:30.124110 4939 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-88bbf7/simulator-w1-88bbf7-kserve-5fbb669f8f-42hgd" Nov 03 01:05:30.217640 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:05:30.217470 4939 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-88bbf7/distinct-2-w1-88bbf7-kserve-5457cdcdbd-l6tfd" Nov 03 01:05:31.104945 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:05:31.099610 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:05:31.222585 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:05:31.222520 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:05:32.873706 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:05:32.873614 4939 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-2f9a3a/distinct-w0-2f9a3a-kserve-555565cfcf-jtcr9" podUID="ac0ece06-8718-463f-915f-5315988d9711" containerName="main" probeResult="failure" output="Get \"https://10.132.0.76:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 01:05:33.383391 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:05:33.371440 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:05:33.565935 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:05:33.548857 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:05:33.637243 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:05:33.637160 4939 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-2f9a3a-openshift-default-7f6f74cf97-cv2rj"] Nov 03 01:05:33.643531 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:05:33.643506 4939 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-2f9a3a-openshift-default-7f6f74cf97-cv2rj" Nov 03 01:05:33.743762 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:05:33.743727 4939 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-2f9a3a-openshift-default-7f6f74cf97-cv2rj"] Nov 03 01:05:33.808825 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:05:33.808767 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f4f0fa4a-d49c-46d2-a1af-9195565e8bc2-istio-podinfo\") pod \"e2e-worker-w0-2f9a3a-openshift-default-7f6f74cf97-cv2rj\" (UID: \"f4f0fa4a-d49c-46d2-a1af-9195565e8bc2\") " pod="openshift-ingress/e2e-worker-w0-2f9a3a-openshift-default-7f6f74cf97-cv2rj" Nov 03 01:05:33.809212 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:05:33.809178 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f4f0fa4a-d49c-46d2-a1af-9195565e8bc2-workload-certs\") pod \"e2e-worker-w0-2f9a3a-openshift-default-7f6f74cf97-cv2rj\" (UID: \"f4f0fa4a-d49c-46d2-a1af-9195565e8bc2\") " pod="openshift-ingress/e2e-worker-w0-2f9a3a-openshift-default-7f6f74cf97-cv2rj" Nov 03 01:05:33.809326 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:05:33.809239 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f4f0fa4a-d49c-46d2-a1af-9195565e8bc2-istiod-ca-cert\") pod \"e2e-worker-w0-2f9a3a-openshift-default-7f6f74cf97-cv2rj\" (UID: \"f4f0fa4a-d49c-46d2-a1af-9195565e8bc2\") " pod="openshift-ingress/e2e-worker-w0-2f9a3a-openshift-default-7f6f74cf97-cv2rj" Nov 03 01:05:33.809326 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:05:33.809272 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f4f0fa4a-d49c-46d2-a1af-9195565e8bc2-istio-data\") pod \"e2e-worker-w0-2f9a3a-openshift-default-7f6f74cf97-cv2rj\" (UID: \"f4f0fa4a-d49c-46d2-a1af-9195565e8bc2\") " pod="openshift-ingress/e2e-worker-w0-2f9a3a-openshift-default-7f6f74cf97-cv2rj" Nov 03 01:05:33.809326 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:05:33.809319 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f4f0fa4a-d49c-46d2-a1af-9195565e8bc2-istio-token\") pod \"e2e-worker-w0-2f9a3a-openshift-default-7f6f74cf97-cv2rj\" (UID: \"f4f0fa4a-d49c-46d2-a1af-9195565e8bc2\") " pod="openshift-ingress/e2e-worker-w0-2f9a3a-openshift-default-7f6f74cf97-cv2rj" Nov 03 01:05:33.809493 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:05:33.809391 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f4f0fa4a-d49c-46d2-a1af-9195565e8bc2-istio-envoy\") pod \"e2e-worker-w0-2f9a3a-openshift-default-7f6f74cf97-cv2rj\" (UID: \"f4f0fa4a-d49c-46d2-a1af-9195565e8bc2\") " pod="openshift-ingress/e2e-worker-w0-2f9a3a-openshift-default-7f6f74cf97-cv2rj" Nov 03 01:05:33.809493 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:05:33.809474 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f4f0fa4a-d49c-46d2-a1af-9195565e8bc2-credential-socket\") pod \"e2e-worker-w0-2f9a3a-openshift-default-7f6f74cf97-cv2rj\" (UID: \"f4f0fa4a-d49c-46d2-a1af-9195565e8bc2\") " pod="openshift-ingress/e2e-worker-w0-2f9a3a-openshift-default-7f6f74cf97-cv2rj" Nov 03 01:05:33.809607 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:05:33.809552 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f4f0fa4a-d49c-46d2-a1af-9195565e8bc2-workload-socket\") pod \"e2e-worker-w0-2f9a3a-openshift-default-7f6f74cf97-cv2rj\" (UID: \"f4f0fa4a-d49c-46d2-a1af-9195565e8bc2\") " pod="openshift-ingress/e2e-worker-w0-2f9a3a-openshift-default-7f6f74cf97-cv2rj" Nov 03 01:05:33.809711 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:05:33.809692 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-str2z\" (UniqueName: \"kubernetes.io/projected/f4f0fa4a-d49c-46d2-a1af-9195565e8bc2-kube-api-access-str2z\") pod \"e2e-worker-w0-2f9a3a-openshift-default-7f6f74cf97-cv2rj\" (UID: \"f4f0fa4a-d49c-46d2-a1af-9195565e8bc2\") " pod="openshift-ingress/e2e-worker-w0-2f9a3a-openshift-default-7f6f74cf97-cv2rj" Nov 03 01:05:33.827085 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:05:33.827046 4939 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-2f9a3a-openshift-default-7f6f74cf97-5rvtq"] Nov 03 01:05:33.837992 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:05:33.837959 4939 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-2f9a3a-openshift-default-7f6f74cf97-5rvtq" Nov 03 01:05:33.840003 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:05:33.839978 4939 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-2f9a3a-openshift-default-7f6f74cf97-5rvtq"] Nov 03 01:05:33.912323 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:05:33.911238 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:05:33.916151 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:05:33.916116 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f4f0fa4a-d49c-46d2-a1af-9195565e8bc2-istio-token\") pod \"e2e-worker-w0-2f9a3a-openshift-default-7f6f74cf97-cv2rj\" (UID: \"f4f0fa4a-d49c-46d2-a1af-9195565e8bc2\") " pod="openshift-ingress/e2e-worker-w0-2f9a3a-openshift-default-7f6f74cf97-cv2rj" Nov 03 01:05:33.916304 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:05:33.916193 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9c96c382-a625-4b0c-b72f-82807daf6f3e-workload-socket\") pod \"e2e-worker-w0-2f9a3a-openshift-default-7f6f74cf97-5rvtq\" (UID: \"9c96c382-a625-4b0c-b72f-82807daf6f3e\") " pod="openshift-ingress/e2e-worker-w0-2f9a3a-openshift-default-7f6f74cf97-5rvtq" Nov 03 01:05:33.916304 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:05:33.916239 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f4f0fa4a-d49c-46d2-a1af-9195565e8bc2-istio-envoy\") pod \"e2e-worker-w0-2f9a3a-openshift-default-7f6f74cf97-cv2rj\" (UID: \"f4f0fa4a-d49c-46d2-a1af-9195565e8bc2\") " pod="openshift-ingress/e2e-worker-w0-2f9a3a-openshift-default-7f6f74cf97-cv2rj" Nov 03 01:05:33.916304 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:05:33.916283 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9c96c382-a625-4b0c-b72f-82807daf6f3e-workload-certs\") pod \"e2e-worker-w0-2f9a3a-openshift-default-7f6f74cf97-5rvtq\" (UID: \"9c96c382-a625-4b0c-b72f-82807daf6f3e\") " pod="openshift-ingress/e2e-worker-w0-2f9a3a-openshift-default-7f6f74cf97-5rvtq" Nov 03 01:05:33.916556 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:05:33.916520 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f4f0fa4a-d49c-46d2-a1af-9195565e8bc2-credential-socket\") pod \"e2e-worker-w0-2f9a3a-openshift-default-7f6f74cf97-cv2rj\" (UID: \"f4f0fa4a-d49c-46d2-a1af-9195565e8bc2\") " pod="openshift-ingress/e2e-worker-w0-2f9a3a-openshift-default-7f6f74cf97-cv2rj" Nov 03 01:05:33.916612 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:05:33.916555 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9c96c382-a625-4b0c-b72f-82807daf6f3e-istio-token\") pod \"e2e-worker-w0-2f9a3a-openshift-default-7f6f74cf97-5rvtq\" (UID: \"9c96c382-a625-4b0c-b72f-82807daf6f3e\") " pod="openshift-ingress/e2e-worker-w0-2f9a3a-openshift-default-7f6f74cf97-5rvtq" Nov 03 01:05:33.916612 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:05:33.916603 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f4f0fa4a-d49c-46d2-a1af-9195565e8bc2-workload-socket\") pod \"e2e-worker-w0-2f9a3a-openshift-default-7f6f74cf97-cv2rj\" (UID: \"f4f0fa4a-d49c-46d2-a1af-9195565e8bc2\") " pod="openshift-ingress/e2e-worker-w0-2f9a3a-openshift-default-7f6f74cf97-cv2rj" Nov 03 01:05:33.916722 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:05:33.916620 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9c96c382-a625-4b0c-b72f-82807daf6f3e-credential-socket\") pod \"e2e-worker-w0-2f9a3a-openshift-default-7f6f74cf97-5rvtq\" (UID: \"9c96c382-a625-4b0c-b72f-82807daf6f3e\") " pod="openshift-ingress/e2e-worker-w0-2f9a3a-openshift-default-7f6f74cf97-5rvtq" Nov 03 01:05:33.916722 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:05:33.916650 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9c96c382-a625-4b0c-b72f-82807daf6f3e-istio-podinfo\") pod \"e2e-worker-w0-2f9a3a-openshift-default-7f6f74cf97-5rvtq\" (UID: \"9c96c382-a625-4b0c-b72f-82807daf6f3e\") " pod="openshift-ingress/e2e-worker-w0-2f9a3a-openshift-default-7f6f74cf97-5rvtq" Nov 03 01:05:33.916722 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:05:33.916666 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9c96c382-a625-4b0c-b72f-82807daf6f3e-istiod-ca-cert\") pod \"e2e-worker-w0-2f9a3a-openshift-default-7f6f74cf97-5rvtq\" (UID: \"9c96c382-a625-4b0c-b72f-82807daf6f3e\") " pod="openshift-ingress/e2e-worker-w0-2f9a3a-openshift-default-7f6f74cf97-5rvtq" Nov 03 01:05:33.916883 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:05:33.916747 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9c96c382-a625-4b0c-b72f-82807daf6f3e-istio-envoy\") pod \"e2e-worker-w0-2f9a3a-openshift-default-7f6f74cf97-5rvtq\" (UID: \"9c96c382-a625-4b0c-b72f-82807daf6f3e\") " pod="openshift-ingress/e2e-worker-w0-2f9a3a-openshift-default-7f6f74cf97-5rvtq" Nov 03 01:05:33.916883 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:05:33.916859 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-str2z\" (UniqueName: \"kubernetes.io/projected/f4f0fa4a-d49c-46d2-a1af-9195565e8bc2-kube-api-access-str2z\") pod \"e2e-worker-w0-2f9a3a-openshift-default-7f6f74cf97-cv2rj\" (UID: \"f4f0fa4a-d49c-46d2-a1af-9195565e8bc2\") " pod="openshift-ingress/e2e-worker-w0-2f9a3a-openshift-default-7f6f74cf97-cv2rj" Nov 03 01:05:33.917029 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:05:33.916882 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9c96c382-a625-4b0c-b72f-82807daf6f3e-istio-data\") pod \"e2e-worker-w0-2f9a3a-openshift-default-7f6f74cf97-5rvtq\" (UID: \"9c96c382-a625-4b0c-b72f-82807daf6f3e\") " pod="openshift-ingress/e2e-worker-w0-2f9a3a-openshift-default-7f6f74cf97-5rvtq" Nov 03 01:05:33.917029 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:05:33.916940 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f4f0fa4a-d49c-46d2-a1af-9195565e8bc2-istio-podinfo\") pod \"e2e-worker-w0-2f9a3a-openshift-default-7f6f74cf97-cv2rj\" (UID: \"f4f0fa4a-d49c-46d2-a1af-9195565e8bc2\") " pod="openshift-ingress/e2e-worker-w0-2f9a3a-openshift-default-7f6f74cf97-cv2rj" Nov 03 01:05:33.917029 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:05:33.916962 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f4f0fa4a-d49c-46d2-a1af-9195565e8bc2-credential-socket\") pod \"e2e-worker-w0-2f9a3a-openshift-default-7f6f74cf97-cv2rj\" (UID: \"f4f0fa4a-d49c-46d2-a1af-9195565e8bc2\") " pod="openshift-ingress/e2e-worker-w0-2f9a3a-openshift-default-7f6f74cf97-cv2rj" Nov 03 01:05:33.917029 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:05:33.916971 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f4f0fa4a-d49c-46d2-a1af-9195565e8bc2-workload-certs\") pod \"e2e-worker-w0-2f9a3a-openshift-default-7f6f74cf97-cv2rj\" (UID: \"f4f0fa4a-d49c-46d2-a1af-9195565e8bc2\") " pod="openshift-ingress/e2e-worker-w0-2f9a3a-openshift-default-7f6f74cf97-cv2rj" Nov 03 01:05:33.917238 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:05:33.917032 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f4f0fa4a-d49c-46d2-a1af-9195565e8bc2-istiod-ca-cert\") pod \"e2e-worker-w0-2f9a3a-openshift-default-7f6f74cf97-cv2rj\" (UID: \"f4f0fa4a-d49c-46d2-a1af-9195565e8bc2\") " pod="openshift-ingress/e2e-worker-w0-2f9a3a-openshift-default-7f6f74cf97-cv2rj" Nov 03 01:05:33.917238 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:05:33.917060 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f4f0fa4a-d49c-46d2-a1af-9195565e8bc2-workload-socket\") pod \"e2e-worker-w0-2f9a3a-openshift-default-7f6f74cf97-cv2rj\" (UID: \"f4f0fa4a-d49c-46d2-a1af-9195565e8bc2\") " pod="openshift-ingress/e2e-worker-w0-2f9a3a-openshift-default-7f6f74cf97-cv2rj" Nov 03 01:05:33.917238 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:05:33.917071 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f4f0fa4a-d49c-46d2-a1af-9195565e8bc2-istio-data\") pod \"e2e-worker-w0-2f9a3a-openshift-default-7f6f74cf97-cv2rj\" (UID: \"f4f0fa4a-d49c-46d2-a1af-9195565e8bc2\") " pod="openshift-ingress/e2e-worker-w0-2f9a3a-openshift-default-7f6f74cf97-cv2rj" Nov 03 01:05:33.917238 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:05:33.917106 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9g45v\" (UniqueName: \"kubernetes.io/projected/9c96c382-a625-4b0c-b72f-82807daf6f3e-kube-api-access-9g45v\") pod \"e2e-worker-w0-2f9a3a-openshift-default-7f6f74cf97-5rvtq\" (UID: \"9c96c382-a625-4b0c-b72f-82807daf6f3e\") " pod="openshift-ingress/e2e-worker-w0-2f9a3a-openshift-default-7f6f74cf97-5rvtq" Nov 03 01:05:33.917444 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:05:33.917202 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f4f0fa4a-d49c-46d2-a1af-9195565e8bc2-workload-certs\") pod \"e2e-worker-w0-2f9a3a-openshift-default-7f6f74cf97-cv2rj\" (UID: \"f4f0fa4a-d49c-46d2-a1af-9195565e8bc2\") " pod="openshift-ingress/e2e-worker-w0-2f9a3a-openshift-default-7f6f74cf97-cv2rj" Nov 03 01:05:33.917444 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:05:33.917399 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f4f0fa4a-d49c-46d2-a1af-9195565e8bc2-istio-data\") pod \"e2e-worker-w0-2f9a3a-openshift-default-7f6f74cf97-cv2rj\" (UID: \"f4f0fa4a-d49c-46d2-a1af-9195565e8bc2\") " pod="openshift-ingress/e2e-worker-w0-2f9a3a-openshift-default-7f6f74cf97-cv2rj" Nov 03 01:05:33.917958 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:05:33.917933 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f4f0fa4a-d49c-46d2-a1af-9195565e8bc2-istiod-ca-cert\") pod \"e2e-worker-w0-2f9a3a-openshift-default-7f6f74cf97-cv2rj\" (UID: \"f4f0fa4a-d49c-46d2-a1af-9195565e8bc2\") " pod="openshift-ingress/e2e-worker-w0-2f9a3a-openshift-default-7f6f74cf97-cv2rj" Nov 03 01:05:33.919915 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:05:33.919111 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f4f0fa4a-d49c-46d2-a1af-9195565e8bc2-istio-envoy\") pod \"e2e-worker-w0-2f9a3a-openshift-default-7f6f74cf97-cv2rj\" (UID: \"f4f0fa4a-d49c-46d2-a1af-9195565e8bc2\") " pod="openshift-ingress/e2e-worker-w0-2f9a3a-openshift-default-7f6f74cf97-cv2rj" Nov 03 01:05:33.924580 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:05:33.920659 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f4f0fa4a-d49c-46d2-a1af-9195565e8bc2-istio-podinfo\") pod \"e2e-worker-w0-2f9a3a-openshift-default-7f6f74cf97-cv2rj\" (UID: \"f4f0fa4a-d49c-46d2-a1af-9195565e8bc2\") " pod="openshift-ingress/e2e-worker-w0-2f9a3a-openshift-default-7f6f74cf97-cv2rj" Nov 03 01:05:33.925186 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:05:33.925135 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f4f0fa4a-d49c-46d2-a1af-9195565e8bc2-istio-token\") pod \"e2e-worker-w0-2f9a3a-openshift-default-7f6f74cf97-cv2rj\" (UID: \"f4f0fa4a-d49c-46d2-a1af-9195565e8bc2\") " pod="openshift-ingress/e2e-worker-w0-2f9a3a-openshift-default-7f6f74cf97-cv2rj" Nov 03 01:05:33.926192 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:05:33.926171 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-str2z\" (UniqueName: \"kubernetes.io/projected/f4f0fa4a-d49c-46d2-a1af-9195565e8bc2-kube-api-access-str2z\") pod \"e2e-worker-w0-2f9a3a-openshift-default-7f6f74cf97-cv2rj\" (UID: \"f4f0fa4a-d49c-46d2-a1af-9195565e8bc2\") " pod="openshift-ingress/e2e-worker-w0-2f9a3a-openshift-default-7f6f74cf97-cv2rj" Nov 03 01:05:34.017840 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:05:34.017797 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9c96c382-a625-4b0c-b72f-82807daf6f3e-workload-socket\") pod \"e2e-worker-w0-2f9a3a-openshift-default-7f6f74cf97-5rvtq\" (UID: \"9c96c382-a625-4b0c-b72f-82807daf6f3e\") " pod="openshift-ingress/e2e-worker-w0-2f9a3a-openshift-default-7f6f74cf97-5rvtq" Nov 03 01:05:34.018044 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:05:34.017860 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9c96c382-a625-4b0c-b72f-82807daf6f3e-workload-certs\") pod \"e2e-worker-w0-2f9a3a-openshift-default-7f6f74cf97-5rvtq\" (UID: \"9c96c382-a625-4b0c-b72f-82807daf6f3e\") " pod="openshift-ingress/e2e-worker-w0-2f9a3a-openshift-default-7f6f74cf97-5rvtq" Nov 03 01:05:34.018044 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:05:34.017949 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9c96c382-a625-4b0c-b72f-82807daf6f3e-istio-token\") pod \"e2e-worker-w0-2f9a3a-openshift-default-7f6f74cf97-5rvtq\" (UID: \"9c96c382-a625-4b0c-b72f-82807daf6f3e\") " pod="openshift-ingress/e2e-worker-w0-2f9a3a-openshift-default-7f6f74cf97-5rvtq" Nov 03 01:05:34.018044 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:05:34.018012 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9c96c382-a625-4b0c-b72f-82807daf6f3e-credential-socket\") pod \"e2e-worker-w0-2f9a3a-openshift-default-7f6f74cf97-5rvtq\" (UID: \"9c96c382-a625-4b0c-b72f-82807daf6f3e\") " pod="openshift-ingress/e2e-worker-w0-2f9a3a-openshift-default-7f6f74cf97-5rvtq" Nov 03 01:05:34.018044 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:05:34.018043 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9c96c382-a625-4b0c-b72f-82807daf6f3e-istio-podinfo\") pod \"e2e-worker-w0-2f9a3a-openshift-default-7f6f74cf97-5rvtq\" (UID: \"9c96c382-a625-4b0c-b72f-82807daf6f3e\") " pod="openshift-ingress/e2e-worker-w0-2f9a3a-openshift-default-7f6f74cf97-5rvtq" Nov 03 01:05:34.018201 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:05:34.018058 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9c96c382-a625-4b0c-b72f-82807daf6f3e-istiod-ca-cert\") pod \"e2e-worker-w0-2f9a3a-openshift-default-7f6f74cf97-5rvtq\" (UID: \"9c96c382-a625-4b0c-b72f-82807daf6f3e\") " pod="openshift-ingress/e2e-worker-w0-2f9a3a-openshift-default-7f6f74cf97-5rvtq" Nov 03 01:05:34.018201 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:05:34.018122 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9c96c382-a625-4b0c-b72f-82807daf6f3e-istio-envoy\") pod \"e2e-worker-w0-2f9a3a-openshift-default-7f6f74cf97-5rvtq\" (UID: \"9c96c382-a625-4b0c-b72f-82807daf6f3e\") " pod="openshift-ingress/e2e-worker-w0-2f9a3a-openshift-default-7f6f74cf97-5rvtq" Nov 03 01:05:34.018201 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:05:34.018153 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9c96c382-a625-4b0c-b72f-82807daf6f3e-istio-data\") pod \"e2e-worker-w0-2f9a3a-openshift-default-7f6f74cf97-5rvtq\" (UID: \"9c96c382-a625-4b0c-b72f-82807daf6f3e\") " pod="openshift-ingress/e2e-worker-w0-2f9a3a-openshift-default-7f6f74cf97-5rvtq" Nov 03 01:05:34.018357 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:05:34.018203 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9g45v\" (UniqueName: \"kubernetes.io/projected/9c96c382-a625-4b0c-b72f-82807daf6f3e-kube-api-access-9g45v\") pod \"e2e-worker-w0-2f9a3a-openshift-default-7f6f74cf97-5rvtq\" (UID: \"9c96c382-a625-4b0c-b72f-82807daf6f3e\") " pod="openshift-ingress/e2e-worker-w0-2f9a3a-openshift-default-7f6f74cf97-5rvtq" Nov 03 01:05:34.019206 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:05:34.018994 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9c96c382-a625-4b0c-b72f-82807daf6f3e-istio-data\") pod \"e2e-worker-w0-2f9a3a-openshift-default-7f6f74cf97-5rvtq\" (UID: \"9c96c382-a625-4b0c-b72f-82807daf6f3e\") " pod="openshift-ingress/e2e-worker-w0-2f9a3a-openshift-default-7f6f74cf97-5rvtq" Nov 03 01:05:34.019206 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:05:34.019128 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9c96c382-a625-4b0c-b72f-82807daf6f3e-istiod-ca-cert\") pod \"e2e-worker-w0-2f9a3a-openshift-default-7f6f74cf97-5rvtq\" (UID: \"9c96c382-a625-4b0c-b72f-82807daf6f3e\") " pod="openshift-ingress/e2e-worker-w0-2f9a3a-openshift-default-7f6f74cf97-5rvtq" Nov 03 01:05:34.019206 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:05:34.019155 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9c96c382-a625-4b0c-b72f-82807daf6f3e-workload-certs\") pod \"e2e-worker-w0-2f9a3a-openshift-default-7f6f74cf97-5rvtq\" (UID: \"9c96c382-a625-4b0c-b72f-82807daf6f3e\") " pod="openshift-ingress/e2e-worker-w0-2f9a3a-openshift-default-7f6f74cf97-5rvtq" Nov 03 01:05:34.019482 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:05:34.019311 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9c96c382-a625-4b0c-b72f-82807daf6f3e-workload-socket\") pod \"e2e-worker-w0-2f9a3a-openshift-default-7f6f74cf97-5rvtq\" (UID: \"9c96c382-a625-4b0c-b72f-82807daf6f3e\") " pod="openshift-ingress/e2e-worker-w0-2f9a3a-openshift-default-7f6f74cf97-5rvtq" Nov 03 01:05:34.019482 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:05:34.019474 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9c96c382-a625-4b0c-b72f-82807daf6f3e-credential-socket\") pod \"e2e-worker-w0-2f9a3a-openshift-default-7f6f74cf97-5rvtq\" (UID: \"9c96c382-a625-4b0c-b72f-82807daf6f3e\") " pod="openshift-ingress/e2e-worker-w0-2f9a3a-openshift-default-7f6f74cf97-5rvtq" Nov 03 01:05:34.021325 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:05:34.021300 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9c96c382-a625-4b0c-b72f-82807daf6f3e-istio-envoy\") pod \"e2e-worker-w0-2f9a3a-openshift-default-7f6f74cf97-5rvtq\" (UID: \"9c96c382-a625-4b0c-b72f-82807daf6f3e\") " pod="openshift-ingress/e2e-worker-w0-2f9a3a-openshift-default-7f6f74cf97-5rvtq" Nov 03 01:05:34.021605 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:05:34.021583 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9c96c382-a625-4b0c-b72f-82807daf6f3e-istio-podinfo\") pod \"e2e-worker-w0-2f9a3a-openshift-default-7f6f74cf97-5rvtq\" (UID: \"9c96c382-a625-4b0c-b72f-82807daf6f3e\") " pod="openshift-ingress/e2e-worker-w0-2f9a3a-openshift-default-7f6f74cf97-5rvtq" Nov 03 01:05:34.028231 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:05:34.028209 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9g45v\" (UniqueName: \"kubernetes.io/projected/9c96c382-a625-4b0c-b72f-82807daf6f3e-kube-api-access-9g45v\") pod \"e2e-worker-w0-2f9a3a-openshift-default-7f6f74cf97-5rvtq\" (UID: \"9c96c382-a625-4b0c-b72f-82807daf6f3e\") " pod="openshift-ingress/e2e-worker-w0-2f9a3a-openshift-default-7f6f74cf97-5rvtq" Nov 03 01:05:34.028231 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:05:34.028223 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9c96c382-a625-4b0c-b72f-82807daf6f3e-istio-token\") pod \"e2e-worker-w0-2f9a3a-openshift-default-7f6f74cf97-5rvtq\" (UID: \"9c96c382-a625-4b0c-b72f-82807daf6f3e\") " pod="openshift-ingress/e2e-worker-w0-2f9a3a-openshift-default-7f6f74cf97-5rvtq" Nov 03 01:05:34.114243 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:05:34.114210 4939 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-2f9a3a-openshift-default-7f6f74cf97-cv2rj" Nov 03 01:05:34.219199 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:05:34.219154 4939 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-2f9a3a-openshift-default-7f6f74cf97-5rvtq" Nov 03 01:05:34.271297 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:05:34.271187 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:05:34.570790 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:05:34.564876 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:05:34.710346 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:05:34.710310 4939 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-2f9a3a-openshift-default-7f6f74cf97-cv2rj"] Nov 03 01:05:34.726800 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:05:34.725804 4939 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 01:05:34.726800 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:05:34.725897 4939 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 01:05:34.726800 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:05:34.725960 4939 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 01:05:34.907006 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:05:34.902093 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:05:34.954794 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:05:34.954766 4939 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-2f9a3a-openshift-default-7f6f74cf97-5rvtq"] Nov 03 01:05:34.961589 ip-10-0-140-107 kubenswrapper[4939]: W1103 01:05:34.961557 4939 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9c96c382_a625_4b0c_b72f_82807daf6f3e.slice/crio-aa85a535f359802bfb247435ed55eefe1b43e955a5ea01fa0d4230ece3f0ed92 WatchSource:0}: Error finding container aa85a535f359802bfb247435ed55eefe1b43e955a5ea01fa0d4230ece3f0ed92: Status 404 returned error can't find the container with id aa85a535f359802bfb247435ed55eefe1b43e955a5ea01fa0d4230ece3f0ed92 Nov 03 01:05:34.964573 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:05:34.964530 4939 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 01:05:34.964685 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:05:34.964622 4939 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 01:05:34.964685 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:05:34.964668 4939 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 01:05:35.047897 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:05:35.047866 4939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-2f9a3a-openshift-default-7f6f74cf97-5rvtq" event={"ID":"9c96c382-a625-4b0c-b72f-82807daf6f3e","Type":"ContainerStarted","Data":"aa85a535f359802bfb247435ed55eefe1b43e955a5ea01fa0d4230ece3f0ed92"} Nov 03 01:05:35.053286 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:05:35.053250 4939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-2f9a3a-openshift-default-7f6f74cf97-cv2rj" event={"ID":"f4f0fa4a-d49c-46d2-a1af-9195565e8bc2","Type":"ContainerStarted","Data":"60f729127ba803a14855cf8bdd33d7fffb8a91e0a8513cc7d87e84487d7002c3"} Nov 03 01:05:35.053421 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:05:35.053308 4939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-2f9a3a-openshift-default-7f6f74cf97-cv2rj" event={"ID":"f4f0fa4a-d49c-46d2-a1af-9195565e8bc2","Type":"ContainerStarted","Data":"68f4f7208255e89da8580497f5f008e3eb5bbf5ad16fd210b6b7d1d0b7e8374b"} Nov 03 01:05:35.092666 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:05:35.092601 4939 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-2f9a3a-openshift-default-7f6f74cf97-cv2rj" podStartSLOduration=2.092580046 podStartE2EDuration="2.092580046s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 01:05:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 01:05:35.077155102 +0000 UTC m=+1483.341490050" watchObservedRunningTime="2026-11-03 01:05:35.092580046 +0000 UTC m=+1483.356914992" Nov 03 01:05:35.115651 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:05:35.114929 4939 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-2f9a3a-openshift-default-7f6f74cf97-cv2rj" Nov 03 01:05:35.116156 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:05:35.116122 4939 patch_prober.go:29] interesting pod/e2e-worker-w0-2f9a3a-openshift-default-7f6f74cf97-cv2rj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.179:15021/healthz/ready\": dial tcp 10.132.0.179:15021: connect: connection refused" start-of-body= Nov 03 01:05:35.116301 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:05:35.116221 4939 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-2f9a3a-openshift-default-7f6f74cf97-cv2rj" podUID="f4f0fa4a-d49c-46d2-a1af-9195565e8bc2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.179:15021/healthz/ready\": dial tcp 10.132.0.179:15021: connect: connection refused" Nov 03 01:05:35.644549 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:05:35.644509 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:05:36.010030 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:05:35.969153 4939 patch_prober.go:29] interesting pod/e2e-shared-a-w4-6c0ec0-openshift-default-79c7584bfc-8hz7h container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.165:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 01:05:36.010030 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:05:35.969227 4939 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-a-w4-6c0ec0-openshift-default-79c7584bfc-8hz7h" podUID="0a1fe883-48d5-4d06-ab8f-41698965dbbb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.165:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 01:05:36.147995 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:05:36.143173 4939 patch_prober.go:29] interesting pod/e2e-worker-w0-2f9a3a-openshift-default-7f6f74cf97-cv2rj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.179:15021/healthz/ready\": dial tcp 10.132.0.179:15021: connect: connection refused" start-of-body= Nov 03 01:05:36.147995 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:05:36.143236 4939 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-2f9a3a-openshift-default-7f6f74cf97-cv2rj" podUID="f4f0fa4a-d49c-46d2-a1af-9195565e8bc2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.179:15021/healthz/ready\": dial tcp 10.132.0.179:15021: connect: connection refused" Nov 03 01:05:36.194930 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:05:36.186885 4939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-2f9a3a-openshift-default-7f6f74cf97-5rvtq" event={"ID":"9c96c382-a625-4b0c-b72f-82807daf6f3e","Type":"ContainerStarted","Data":"6e90b367fe9525a005eaec42d07c9414aa9906b5a68d215c39929f17aa671616"} Nov 03 01:05:36.222840 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:05:36.222626 4939 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-2f9a3a-openshift-default-7f6f74cf97-5rvtq" Nov 03 01:05:36.225231 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:05:36.224224 4939 patch_prober.go:29] interesting pod/e2e-worker-w0-2f9a3a-openshift-default-7f6f74cf97-5rvtq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.180:15021/healthz/ready\": dial tcp 10.132.0.180:15021: connect: connection refused" start-of-body= Nov 03 01:05:36.225231 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:05:36.224289 4939 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-2f9a3a-openshift-default-7f6f74cf97-5rvtq" podUID="9c96c382-a625-4b0c-b72f-82807daf6f3e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.180:15021/healthz/ready\": dial tcp 10.132.0.180:15021: connect: connection refused" Nov 03 01:05:36.344279 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:05:36.344211 4939 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-2f9a3a-openshift-default-7f6f74cf97-5rvtq" podStartSLOduration=3.344190228 podStartE2EDuration="3.344190228s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 01:05:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 01:05:36.280896109 +0000 UTC m=+1484.545231109" watchObservedRunningTime="2026-11-03 01:05:36.344190228 +0000 UTC m=+1484.608525173" Nov 03 01:05:36.606201 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:05:36.606113 4939 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-50e623-746d8fcf47-tjbwb" Nov 03 01:05:36.613043 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:05:36.613014 4939 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-4a697f-94d8d8fc9-kzr29" Nov 03 01:05:36.614410 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:05:36.614383 4939 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-4a697f-79fb8d7d9-gr8cv" Nov 03 01:05:37.012964 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:05:37.012811 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:05:37.118992 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:05:37.118941 4939 patch_prober.go:29] interesting pod/e2e-worker-w0-2f9a3a-openshift-default-7f6f74cf97-cv2rj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.179:15021/healthz/ready\": dial tcp 10.132.0.179:15021: connect: connection refused" start-of-body= Nov 03 01:05:37.119168 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:05:37.119024 4939 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-2f9a3a-openshift-default-7f6f74cf97-cv2rj" podUID="f4f0fa4a-d49c-46d2-a1af-9195565e8bc2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.179:15021/healthz/ready\": dial tcp 10.132.0.179:15021: connect: connection refused" Nov 03 01:05:37.131317 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:05:37.131271 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:05:37.242928 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:05:37.242082 4939 patch_prober.go:29] interesting pod/e2e-worker-w0-2f9a3a-openshift-default-7f6f74cf97-5rvtq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.180:15021/healthz/ready\": dial tcp 10.132.0.180:15021: connect: connection refused" start-of-body= Nov 03 01:05:37.242928 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:05:37.242155 4939 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-2f9a3a-openshift-default-7f6f74cf97-5rvtq" podUID="9c96c382-a625-4b0c-b72f-82807daf6f3e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.180:15021/healthz/ready\": dial tcp 10.132.0.180:15021: connect: connection refused" Nov 03 01:05:37.431677 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:05:37.317375 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:05:37.614932 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:05:37.612901 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:05:37.747984 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:05:37.744417 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:05:38.135292 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:05:38.123426 4939 patch_prober.go:29] interesting pod/e2e-worker-w0-2f9a3a-openshift-default-7f6f74cf97-cv2rj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.179:15021/healthz/ready\": dial tcp 10.132.0.179:15021: connect: connection refused" start-of-body= Nov 03 01:05:38.135292 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:05:38.123487 4939 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-2f9a3a-openshift-default-7f6f74cf97-cv2rj" podUID="f4f0fa4a-d49c-46d2-a1af-9195565e8bc2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.179:15021/healthz/ready\": dial tcp 10.132.0.179:15021: connect: connection refused" Nov 03 01:05:38.271108 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:05:38.245697 4939 patch_prober.go:29] interesting pod/e2e-worker-w0-2f9a3a-openshift-default-7f6f74cf97-5rvtq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.180:15021/healthz/ready\": dial tcp 10.132.0.180:15021: connect: connection refused" start-of-body= Nov 03 01:05:38.271108 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:05:38.245773 4939 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-2f9a3a-openshift-default-7f6f74cf97-5rvtq" podUID="9c96c382-a625-4b0c-b72f-82807daf6f3e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.180:15021/healthz/ready\": dial tcp 10.132.0.180:15021: connect: connection refused" Nov 03 01:05:38.419844 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:05:38.419706 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:05:38.596936 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:05:38.581468 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:05:38.826483 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:05:38.820416 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:05:38.953990 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:05:38.953890 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:05:39.115687 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:05:39.115234 4939 patch_prober.go:29] interesting pod/e2e-worker-w0-2f9a3a-openshift-default-7f6f74cf97-cv2rj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.179:15021/healthz/ready\": dial tcp 10.132.0.179:15021: connect: connection refused" start-of-body= Nov 03 01:05:39.115687 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:05:39.115308 4939 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-2f9a3a-openshift-default-7f6f74cf97-cv2rj" podUID="f4f0fa4a-d49c-46d2-a1af-9195565e8bc2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.179:15021/healthz/ready\": dial tcp 10.132.0.179:15021: connect: connection refused" Nov 03 01:05:39.220413 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:05:39.220069 4939 patch_prober.go:29] interesting pod/e2e-worker-w0-2f9a3a-openshift-default-7f6f74cf97-5rvtq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.180:15021/healthz/ready\": dial tcp 10.132.0.180:15021: connect: connection refused" start-of-body= Nov 03 01:05:39.220413 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:05:39.220135 4939 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-2f9a3a-openshift-default-7f6f74cf97-5rvtq" podUID="9c96c382-a625-4b0c-b72f-82807daf6f3e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.180:15021/healthz/ready\": dial tcp 10.132.0.180:15021: connect: connection refused" Nov 03 01:05:39.283928 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:05:39.279818 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:05:39.770933 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:05:39.745121 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:05:39.818010 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:05:39.817967 4939 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w3-556c14-669d5fb59f-cjtpf"] Nov 03 01:05:39.831810 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:05:39.831777 4939 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w3-556c14-669d5fb59f-cjtpf"] Nov 03 01:05:39.832072 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:05:39.831918 4939 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-556c14-669d5fb59f-cjtpf" Nov 03 01:05:39.867934 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:05:39.857993 4939 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-b-w3-556c14\"" Nov 03 01:05:39.867934 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:05:39.858131 4939 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-b-w3-556c14-dockercfg-nzmfx\"" Nov 03 01:05:40.033043 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:05:40.001770 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:05:40.359930 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:05:40.006868 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ps7qn\" (UniqueName: \"kubernetes.io/projected/192c8a1d-60d2-4928-9579-0231138280bb-kube-api-access-ps7qn\") pod \"payload-pre-processing-e2e-shared-b-w3-556c14-669d5fb59f-cjtpf\" (UID: \"192c8a1d-60d2-4928-9579-0231138280bb\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-556c14-669d5fb59f-cjtpf" Nov 03 01:05:40.359930 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:05:40.351974 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/192c8a1d-60d2-4928-9579-0231138280bb-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w3-556c14-669d5fb59f-cjtpf\" (UID: \"192c8a1d-60d2-4928-9579-0231138280bb\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-556c14-669d5fb59f-cjtpf" Nov 03 01:05:40.438962 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:05:40.434057 4939 patch_prober.go:29] interesting pod/e2e-worker-w0-2f9a3a-openshift-default-7f6f74cf97-cv2rj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.179:15021/healthz/ready\": dial tcp 10.132.0.179:15021: connect: connection refused" start-of-body= Nov 03 01:05:40.438962 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:05:40.434133 4939 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-2f9a3a-openshift-default-7f6f74cf97-cv2rj" podUID="f4f0fa4a-d49c-46d2-a1af-9195565e8bc2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.179:15021/healthz/ready\": dial tcp 10.132.0.179:15021: connect: connection refused" Nov 03 01:05:40.438962 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:05:40.436606 4939 patch_prober.go:29] interesting pod/e2e-worker-w0-2f9a3a-openshift-default-7f6f74cf97-5rvtq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.180:15021/healthz/ready\": dial tcp 10.132.0.180:15021: connect: connection refused" start-of-body= Nov 03 01:05:40.438962 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:05:40.436662 4939 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-2f9a3a-openshift-default-7f6f74cf97-5rvtq" podUID="9c96c382-a625-4b0c-b72f-82807daf6f3e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.180:15021/healthz/ready\": dial tcp 10.132.0.180:15021: connect: connection refused" Nov 03 01:05:40.513937 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:05:40.480964 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ps7qn\" (UniqueName: \"kubernetes.io/projected/192c8a1d-60d2-4928-9579-0231138280bb-kube-api-access-ps7qn\") pod \"payload-pre-processing-e2e-shared-b-w3-556c14-669d5fb59f-cjtpf\" (UID: \"192c8a1d-60d2-4928-9579-0231138280bb\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-556c14-669d5fb59f-cjtpf" Nov 03 01:05:40.517982 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:05:40.517945 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/192c8a1d-60d2-4928-9579-0231138280bb-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w3-556c14-669d5fb59f-cjtpf\" (UID: \"192c8a1d-60d2-4928-9579-0231138280bb\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-556c14-669d5fb59f-cjtpf" Nov 03 01:05:40.538936 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:05:40.533014 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/192c8a1d-60d2-4928-9579-0231138280bb-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w3-556c14-669d5fb59f-cjtpf\" (UID: \"192c8a1d-60d2-4928-9579-0231138280bb\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-556c14-669d5fb59f-cjtpf" Nov 03 01:05:40.538936 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:05:40.536416 4939 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w3-556c14-566c7cc65c-lcbl4"] Nov 03 01:05:40.541764 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:05:40.541735 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ps7qn\" (UniqueName: \"kubernetes.io/projected/192c8a1d-60d2-4928-9579-0231138280bb-kube-api-access-ps7qn\") pod \"payload-pre-processing-e2e-shared-b-w3-556c14-669d5fb59f-cjtpf\" (UID: \"192c8a1d-60d2-4928-9579-0231138280bb\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-556c14-669d5fb59f-cjtpf" Nov 03 01:05:40.555824 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:05:40.555795 4939 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w3-556c14-566c7cc65c-lcbl4"] Nov 03 01:05:40.556167 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:05:40.556147 4939 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-556c14-566c7cc65c-lcbl4" Nov 03 01:05:40.634768 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:05:40.634724 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v8np4\" (UniqueName: \"kubernetes.io/projected/23f02c28-7f9e-40d8-990c-aea9e85f1bf6-kube-api-access-v8np4\") pod \"payload-processing-e2e-shared-b-w3-556c14-566c7cc65c-lcbl4\" (UID: \"23f02c28-7f9e-40d8-990c-aea9e85f1bf6\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w3-556c14-566c7cc65c-lcbl4" Nov 03 01:05:40.634934 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:05:40.634797 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/23f02c28-7f9e-40d8-990c-aea9e85f1bf6-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w3-556c14-566c7cc65c-lcbl4\" (UID: \"23f02c28-7f9e-40d8-990c-aea9e85f1bf6\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w3-556c14-566c7cc65c-lcbl4" Nov 03 01:05:40.673944 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:05:40.671842 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:05:40.717938 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:05:40.714041 4939 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-556c14-669d5fb59f-cjtpf" Nov 03 01:05:40.738548 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:05:40.737027 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v8np4\" (UniqueName: \"kubernetes.io/projected/23f02c28-7f9e-40d8-990c-aea9e85f1bf6-kube-api-access-v8np4\") pod \"payload-processing-e2e-shared-b-w3-556c14-566c7cc65c-lcbl4\" (UID: \"23f02c28-7f9e-40d8-990c-aea9e85f1bf6\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w3-556c14-566c7cc65c-lcbl4" Nov 03 01:05:40.738548 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:05:40.737131 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/23f02c28-7f9e-40d8-990c-aea9e85f1bf6-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w3-556c14-566c7cc65c-lcbl4\" (UID: \"23f02c28-7f9e-40d8-990c-aea9e85f1bf6\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w3-556c14-566c7cc65c-lcbl4" Nov 03 01:05:40.738548 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:05:40.738297 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/23f02c28-7f9e-40d8-990c-aea9e85f1bf6-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w3-556c14-566c7cc65c-lcbl4\" (UID: \"23f02c28-7f9e-40d8-990c-aea9e85f1bf6\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w3-556c14-566c7cc65c-lcbl4" Nov 03 01:05:40.800950 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:05:40.797615 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-v8np4\" (UniqueName: \"kubernetes.io/projected/23f02c28-7f9e-40d8-990c-aea9e85f1bf6-kube-api-access-v8np4\") pod \"payload-processing-e2e-shared-b-w3-556c14-566c7cc65c-lcbl4\" (UID: \"23f02c28-7f9e-40d8-990c-aea9e85f1bf6\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w3-556c14-566c7cc65c-lcbl4" Nov 03 01:05:40.919282 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:05:40.918973 4939 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-556c14-566c7cc65c-lcbl4" Nov 03 01:05:41.090873 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:05:41.090790 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:05:41.130652 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:05:41.115806 4939 patch_prober.go:29] interesting pod/e2e-worker-w0-2f9a3a-openshift-default-7f6f74cf97-cv2rj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.179:15021/healthz/ready\": dial tcp 10.132.0.179:15021: connect: connection refused" start-of-body= Nov 03 01:05:41.130652 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:05:41.115866 4939 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-2f9a3a-openshift-default-7f6f74cf97-cv2rj" podUID="f4f0fa4a-d49c-46d2-a1af-9195565e8bc2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.179:15021/healthz/ready\": dial tcp 10.132.0.179:15021: connect: connection refused" Nov 03 01:05:41.224487 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:05:41.224031 4939 patch_prober.go:29] interesting pod/e2e-worker-w0-2f9a3a-openshift-default-7f6f74cf97-5rvtq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.180:15021/healthz/ready\": dial tcp 10.132.0.180:15021: connect: connection refused" start-of-body= Nov 03 01:05:41.224487 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:05:41.224091 4939 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-2f9a3a-openshift-default-7f6f74cf97-5rvtq" podUID="9c96c382-a625-4b0c-b72f-82807daf6f3e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.180:15021/healthz/ready\": dial tcp 10.132.0.180:15021: connect: connection refused" Nov 03 01:05:41.432495 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:05:41.432455 4939 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w3-556c14-669d5fb59f-cjtpf"] Nov 03 01:05:41.442423 ip-10-0-140-107 kubenswrapper[4939]: W1103 01:05:41.442376 4939 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod192c8a1d_60d2_4928_9579_0231138280bb.slice/crio-e0ab39b6a66829e96c05ad51f999acdb2212ad90364da3e376ea8f9d381d3b34 WatchSource:0}: Error finding container e0ab39b6a66829e96c05ad51f999acdb2212ad90364da3e376ea8f9d381d3b34: Status 404 returned error can't find the container with id e0ab39b6a66829e96c05ad51f999acdb2212ad90364da3e376ea8f9d381d3b34 Nov 03 01:05:41.597975 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:05:41.597510 4939 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w3-556c14-566c7cc65c-lcbl4"] Nov 03 01:05:41.615549 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:05:41.615242 4939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-556c14-669d5fb59f-cjtpf" event={"ID":"192c8a1d-60d2-4928-9579-0231138280bb","Type":"ContainerStarted","Data":"e0ab39b6a66829e96c05ad51f999acdb2212ad90364da3e376ea8f9d381d3b34"} Nov 03 01:05:41.616675 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:05:41.616652 4939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-556c14-566c7cc65c-lcbl4" event={"ID":"23f02c28-7f9e-40d8-990c-aea9e85f1bf6","Type":"ContainerStarted","Data":"44e45593589bba64cda93b2acbc4214851055f7fa19585c418bd10786111836b"} Nov 03 01:05:41.824893 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:05:41.824844 4939 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-autores-4248ae/autores-model-4248ae-kserve-5754c66dc7-whjqx"] Nov 03 01:05:41.845438 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:05:41.845390 4939 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-autores-4248ae/autores-model-4248ae-kserve-5754c66dc7-whjqx"] Nov 03 01:05:41.845624 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:05:41.845527 4939 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-autores-4248ae/autores-model-4248ae-kserve-5754c66dc7-whjqx" Nov 03 01:05:41.879966 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:05:41.857933 4939 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-autores-4248ae\"/\"default-dockercfg-grzmr\"" Nov 03 01:05:41.879966 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:05:41.858059 4939 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-autores-4248ae\"/\"autores-model-4248ae-kserve-self-signed-certs\"" Nov 03 01:05:41.879966 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:05:41.859682 4939 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-autores-4248ae\"/\"kube-root-ca.crt\"" Nov 03 01:05:41.879966 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:05:41.869091 4939 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-autores-4248ae\"/\"openshift-service-ca.crt\"" Nov 03 01:05:41.921559 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:05:41.921518 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:05:41.983971 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:05:41.963304 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e7a0e0a1-ab97-4664-b3f6-20896bb6b9ff-tls-certs\") pod \"autores-model-4248ae-kserve-5754c66dc7-whjqx\" (UID: \"e7a0e0a1-ab97-4664-b3f6-20896bb6b9ff\") " pod="ai-tenant-e2e-autores-4248ae/autores-model-4248ae-kserve-5754c66dc7-whjqx" Nov 03 01:05:41.984781 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:05:41.984747 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e7a0e0a1-ab97-4664-b3f6-20896bb6b9ff-tmp-dir\") pod \"autores-model-4248ae-kserve-5754c66dc7-whjqx\" (UID: \"e7a0e0a1-ab97-4664-b3f6-20896bb6b9ff\") " pod="ai-tenant-e2e-autores-4248ae/autores-model-4248ae-kserve-5754c66dc7-whjqx" Nov 03 01:05:41.985047 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:05:41.985023 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e7a0e0a1-ab97-4664-b3f6-20896bb6b9ff-model-cache\") pod \"autores-model-4248ae-kserve-5754c66dc7-whjqx\" (UID: \"e7a0e0a1-ab97-4664-b3f6-20896bb6b9ff\") " pod="ai-tenant-e2e-autores-4248ae/autores-model-4248ae-kserve-5754c66dc7-whjqx" Nov 03 01:05:41.985202 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:05:41.985184 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e7a0e0a1-ab97-4664-b3f6-20896bb6b9ff-dshm\") pod \"autores-model-4248ae-kserve-5754c66dc7-whjqx\" (UID: \"e7a0e0a1-ab97-4664-b3f6-20896bb6b9ff\") " pod="ai-tenant-e2e-autores-4248ae/autores-model-4248ae-kserve-5754c66dc7-whjqx" Nov 03 01:05:41.985496 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:05:41.985478 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f298d\" (UniqueName: \"kubernetes.io/projected/e7a0e0a1-ab97-4664-b3f6-20896bb6b9ff-kube-api-access-f298d\") pod \"autores-model-4248ae-kserve-5754c66dc7-whjqx\" (UID: \"e7a0e0a1-ab97-4664-b3f6-20896bb6b9ff\") " pod="ai-tenant-e2e-autores-4248ae/autores-model-4248ae-kserve-5754c66dc7-whjqx" Nov 03 01:05:41.985769 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:05:41.985751 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e7a0e0a1-ab97-4664-b3f6-20896bb6b9ff-home\") pod \"autores-model-4248ae-kserve-5754c66dc7-whjqx\" (UID: \"e7a0e0a1-ab97-4664-b3f6-20896bb6b9ff\") " pod="ai-tenant-e2e-autores-4248ae/autores-model-4248ae-kserve-5754c66dc7-whjqx" Nov 03 01:05:42.039647 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:05:42.039467 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:05:42.098931 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:05:42.088099 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e7a0e0a1-ab97-4664-b3f6-20896bb6b9ff-home\") pod \"autores-model-4248ae-kserve-5754c66dc7-whjqx\" (UID: \"e7a0e0a1-ab97-4664-b3f6-20896bb6b9ff\") " pod="ai-tenant-e2e-autores-4248ae/autores-model-4248ae-kserve-5754c66dc7-whjqx" Nov 03 01:05:42.098931 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:05:42.088266 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e7a0e0a1-ab97-4664-b3f6-20896bb6b9ff-tls-certs\") pod \"autores-model-4248ae-kserve-5754c66dc7-whjqx\" (UID: \"e7a0e0a1-ab97-4664-b3f6-20896bb6b9ff\") " pod="ai-tenant-e2e-autores-4248ae/autores-model-4248ae-kserve-5754c66dc7-whjqx" Nov 03 01:05:42.098931 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:05:42.088343 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e7a0e0a1-ab97-4664-b3f6-20896bb6b9ff-tmp-dir\") pod \"autores-model-4248ae-kserve-5754c66dc7-whjqx\" (UID: \"e7a0e0a1-ab97-4664-b3f6-20896bb6b9ff\") " pod="ai-tenant-e2e-autores-4248ae/autores-model-4248ae-kserve-5754c66dc7-whjqx" Nov 03 01:05:42.098931 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:05:42.088394 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e7a0e0a1-ab97-4664-b3f6-20896bb6b9ff-model-cache\") pod \"autores-model-4248ae-kserve-5754c66dc7-whjqx\" (UID: \"e7a0e0a1-ab97-4664-b3f6-20896bb6b9ff\") " pod="ai-tenant-e2e-autores-4248ae/autores-model-4248ae-kserve-5754c66dc7-whjqx" Nov 03 01:05:42.098931 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:05:42.088436 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e7a0e0a1-ab97-4664-b3f6-20896bb6b9ff-dshm\") pod \"autores-model-4248ae-kserve-5754c66dc7-whjqx\" (UID: \"e7a0e0a1-ab97-4664-b3f6-20896bb6b9ff\") " pod="ai-tenant-e2e-autores-4248ae/autores-model-4248ae-kserve-5754c66dc7-whjqx" Nov 03 01:05:42.098931 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:05:42.088596 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f298d\" (UniqueName: \"kubernetes.io/projected/e7a0e0a1-ab97-4664-b3f6-20896bb6b9ff-kube-api-access-f298d\") pod \"autores-model-4248ae-kserve-5754c66dc7-whjqx\" (UID: \"e7a0e0a1-ab97-4664-b3f6-20896bb6b9ff\") " pod="ai-tenant-e2e-autores-4248ae/autores-model-4248ae-kserve-5754c66dc7-whjqx" Nov 03 01:05:42.098931 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:05:42.089539 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e7a0e0a1-ab97-4664-b3f6-20896bb6b9ff-home\") pod \"autores-model-4248ae-kserve-5754c66dc7-whjqx\" (UID: \"e7a0e0a1-ab97-4664-b3f6-20896bb6b9ff\") " pod="ai-tenant-e2e-autores-4248ae/autores-model-4248ae-kserve-5754c66dc7-whjqx" Nov 03 01:05:42.108565 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:05:42.101964 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e7a0e0a1-ab97-4664-b3f6-20896bb6b9ff-model-cache\") pod \"autores-model-4248ae-kserve-5754c66dc7-whjqx\" (UID: \"e7a0e0a1-ab97-4664-b3f6-20896bb6b9ff\") " pod="ai-tenant-e2e-autores-4248ae/autores-model-4248ae-kserve-5754c66dc7-whjqx" Nov 03 01:05:42.108565 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:05:42.102240 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e7a0e0a1-ab97-4664-b3f6-20896bb6b9ff-tmp-dir\") pod \"autores-model-4248ae-kserve-5754c66dc7-whjqx\" (UID: \"e7a0e0a1-ab97-4664-b3f6-20896bb6b9ff\") " pod="ai-tenant-e2e-autores-4248ae/autores-model-4248ae-kserve-5754c66dc7-whjqx" Nov 03 01:05:42.120714 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:05:42.118963 4939 patch_prober.go:29] interesting pod/e2e-worker-w0-2f9a3a-openshift-default-7f6f74cf97-cv2rj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.179:15021/healthz/ready\": dial tcp 10.132.0.179:15021: connect: connection refused" start-of-body= Nov 03 01:05:42.120714 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:05:42.119047 4939 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-2f9a3a-openshift-default-7f6f74cf97-cv2rj" podUID="f4f0fa4a-d49c-46d2-a1af-9195565e8bc2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.179:15021/healthz/ready\": dial tcp 10.132.0.179:15021: connect: connection refused" Nov 03 01:05:42.142954 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:05:42.142839 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e7a0e0a1-ab97-4664-b3f6-20896bb6b9ff-tls-certs\") pod \"autores-model-4248ae-kserve-5754c66dc7-whjqx\" (UID: \"e7a0e0a1-ab97-4664-b3f6-20896bb6b9ff\") " pod="ai-tenant-e2e-autores-4248ae/autores-model-4248ae-kserve-5754c66dc7-whjqx" Nov 03 01:05:42.157470 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:05:42.153680 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-f298d\" (UniqueName: \"kubernetes.io/projected/e7a0e0a1-ab97-4664-b3f6-20896bb6b9ff-kube-api-access-f298d\") pod \"autores-model-4248ae-kserve-5754c66dc7-whjqx\" (UID: \"e7a0e0a1-ab97-4664-b3f6-20896bb6b9ff\") " pod="ai-tenant-e2e-autores-4248ae/autores-model-4248ae-kserve-5754c66dc7-whjqx" Nov 03 01:05:42.158667 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:05:42.158091 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e7a0e0a1-ab97-4664-b3f6-20896bb6b9ff-dshm\") pod \"autores-model-4248ae-kserve-5754c66dc7-whjqx\" (UID: \"e7a0e0a1-ab97-4664-b3f6-20896bb6b9ff\") " pod="ai-tenant-e2e-autores-4248ae/autores-model-4248ae-kserve-5754c66dc7-whjqx" Nov 03 01:05:42.227998 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:05:42.227891 4939 patch_prober.go:29] interesting pod/e2e-worker-w0-2f9a3a-openshift-default-7f6f74cf97-5rvtq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.180:15021/healthz/ready\": dial tcp 10.132.0.180:15021: connect: connection refused" start-of-body= Nov 03 01:05:42.228352 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:05:42.228254 4939 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-2f9a3a-openshift-default-7f6f74cf97-5rvtq" podUID="9c96c382-a625-4b0c-b72f-82807daf6f3e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.180:15021/healthz/ready\": dial tcp 10.132.0.180:15021: connect: connection refused" Nov 03 01:05:42.289896 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:05:42.289622 4939 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-autores-4248ae/autores-model-4248ae-kserve-5754c66dc7-whjqx" Nov 03 01:05:42.600890 ip-10-0-140-107 kubenswrapper[4939]: W1103 01:05:42.600832 4939 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pode7a0e0a1_ab97_4664_b3f6_20896bb6b9ff.slice/crio-de8b83a975b9ea69b71a7809bb401a8b96dd20b17547b6cefea2ddc82d4cfb94 WatchSource:0}: Error finding container de8b83a975b9ea69b71a7809bb401a8b96dd20b17547b6cefea2ddc82d4cfb94: Status 404 returned error can't find the container with id de8b83a975b9ea69b71a7809bb401a8b96dd20b17547b6cefea2ddc82d4cfb94 Nov 03 01:05:42.603329 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:05:42.603294 4939 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-autores-4248ae/autores-model-4248ae-kserve-5754c66dc7-whjqx"] Nov 03 01:05:42.631069 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:05:42.630671 4939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-556c14-669d5fb59f-cjtpf" event={"ID":"192c8a1d-60d2-4928-9579-0231138280bb","Type":"ContainerStarted","Data":"f0face4ca56ac8b0d09a7aa6ea1c56ac5622db1ed5a8238106e45cc3a42061c2"} Nov 03 01:05:42.631069 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:05:42.630752 4939 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-556c14-669d5fb59f-cjtpf" Nov 03 01:05:42.635634 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:05:42.633238 4939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-556c14-566c7cc65c-lcbl4" event={"ID":"23f02c28-7f9e-40d8-990c-aea9e85f1bf6","Type":"ContainerStarted","Data":"82722e009994f48eae235ec601132b70f0dee007f538c43936ef0e5e48c7ff34"} Nov 03 01:05:42.635634 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:05:42.634060 4939 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-556c14-566c7cc65c-lcbl4" Nov 03 01:05:42.636262 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:05:42.636236 4939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-autores-4248ae/autores-model-4248ae-kserve-5754c66dc7-whjqx" event={"ID":"e7a0e0a1-ab97-4664-b3f6-20896bb6b9ff","Type":"ContainerStarted","Data":"de8b83a975b9ea69b71a7809bb401a8b96dd20b17547b6cefea2ddc82d4cfb94"} Nov 03 01:05:42.672166 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:05:42.672106 4939 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-556c14-566c7cc65c-lcbl4" podStartSLOduration=3.672085766 podStartE2EDuration="3.672085766s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 01:05:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 01:05:42.67075874 +0000 UTC m=+1490.935093707" watchObservedRunningTime="2026-11-03 01:05:42.672085766 +0000 UTC m=+1490.936420713" Nov 03 01:05:42.674493 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:05:42.673834 4939 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-556c14-669d5fb59f-cjtpf" podStartSLOduration=3.673818487 podStartE2EDuration="3.673818487s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 01:05:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 01:05:42.653415117 +0000 UTC m=+1490.917750063" watchObservedRunningTime="2026-11-03 01:05:42.673818487 +0000 UTC m=+1490.938153435" Nov 03 01:05:43.016143 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:05:43.016022 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:05:43.139969 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:05:43.139926 4939 patch_prober.go:29] interesting pod/e2e-worker-w0-2f9a3a-openshift-default-7f6f74cf97-cv2rj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.179:15021/healthz/ready\": dial tcp 10.132.0.179:15021: connect: connection refused" start-of-body= Nov 03 01:05:43.140638 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:05:43.140228 4939 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-2f9a3a-openshift-default-7f6f74cf97-cv2rj" podUID="f4f0fa4a-d49c-46d2-a1af-9195565e8bc2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.179:15021/healthz/ready\": dial tcp 10.132.0.179:15021: connect: connection refused" Nov 03 01:05:43.220449 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:05:43.220397 4939 patch_prober.go:29] interesting pod/e2e-worker-w0-2f9a3a-openshift-default-7f6f74cf97-5rvtq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.180:15021/healthz/ready\": dial tcp 10.132.0.180:15021: connect: connection refused" start-of-body= Nov 03 01:05:43.220662 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:05:43.220474 4939 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-2f9a3a-openshift-default-7f6f74cf97-5rvtq" podUID="9c96c382-a625-4b0c-b72f-82807daf6f3e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.180:15021/healthz/ready\": dial tcp 10.132.0.180:15021: connect: connection refused" Nov 03 01:05:43.291011 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:05:43.290886 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:05:43.654630 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:05:43.653937 4939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-autores-4248ae/autores-model-4248ae-kserve-5754c66dc7-whjqx" event={"ID":"e7a0e0a1-ab97-4664-b3f6-20896bb6b9ff","Type":"ContainerStarted","Data":"a3c997ce891ef9a1d2d834cb36cd3bfca9bb5f76a229e3a3393143dd55460762"} Nov 03 01:05:44.114929 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:05:44.114413 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:05:44.210934 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:05:44.207002 4939 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-2f9a3a-openshift-default-7f6f74cf97-cv2rj" Nov 03 01:05:44.210934 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:05:44.208166 4939 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-2f9a3a-openshift-default-7f6f74cf97-cv2rj" Nov 03 01:05:44.233932 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:05:44.230580 4939 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-2f9a3a-openshift-default-7f6f74cf97-cv2rj" Nov 03 01:05:44.262206 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:05:44.262140 4939 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-autores-4248ae/autores-model-4248ae-kserve-5754c66dc7-whjqx" podStartSLOduration=3.262117217 podStartE2EDuration="3.262117217s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 01:05:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 01:05:43.674748496 +0000 UTC m=+1491.939083470" watchObservedRunningTime="2026-11-03 01:05:44.262117217 +0000 UTC m=+1492.526452162" Nov 03 01:05:44.278938 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:05:44.274206 4939 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-2f9a3a-openshift-default-7f6f74cf97-5rvtq" Nov 03 01:05:44.278938 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:05:44.277618 4939 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-2f9a3a-openshift-default-7f6f74cf97-5rvtq" Nov 03 01:05:44.281232 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:05:44.281205 4939 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-2f9a3a-openshift-default-7f6f74cf97-5rvtq" Nov 03 01:05:44.293426 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:05:44.293388 4939 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-autores-4248ae/autores-model-4248ae-kserve-5754c66dc7-whjqx" Nov 03 01:05:45.021419 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:05:45.021369 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:05:45.986804 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:05:45.670981 4939 generic.go:356] "Generic (PLEG): container finished" podID="cba006d1-08cd-4fac-8bdd-86931f742838" containerID="c7884d3f229a4cf4c41b83c524ce2da09be6b780b239aa5ea0bdd93e1ac60024" exitCode=0 Nov 03 01:05:45.986804 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:05:45.671054 4939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w1-2729f2-openshift-default-9fb97b4fc-5ww48" event={"ID":"cba006d1-08cd-4fac-8bdd-86931f742838","Type":"ContainerDied","Data":"c7884d3f229a4cf4c41b83c524ce2da09be6b780b239aa5ea0bdd93e1ac60024"} Nov 03 01:05:45.986804 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:05:45.671768 4939 scope.go:122] "RemoveContainer" containerID="c7884d3f229a4cf4c41b83c524ce2da09be6b780b239aa5ea0bdd93e1ac60024" Nov 03 01:05:45.986804 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:05:45.673306 4939 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 01:05:45.986804 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:05:45.673367 4939 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 01:05:45.986804 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:05:45.673403 4939 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 01:05:45.986804 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:05:45.775170 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:05:46.688529 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:05:46.688489 4939 generic.go:356] "Generic (PLEG): container finished" podID="0a1fe883-48d5-4d06-ab8f-41698965dbbb" containerID="3a69478699977ff097dfe6b487c6a8b6fe1eb21904e33269e787e9a5b7b2dea6" exitCode=0 Nov 03 01:05:46.689102 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:05:46.688566 4939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w4-6c0ec0-openshift-default-79c7584bfc-8hz7h" event={"ID":"0a1fe883-48d5-4d06-ab8f-41698965dbbb","Type":"ContainerDied","Data":"3a69478699977ff097dfe6b487c6a8b6fe1eb21904e33269e787e9a5b7b2dea6"} Nov 03 01:05:46.689187 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:05:46.689127 4939 scope.go:122] "RemoveContainer" containerID="3a69478699977ff097dfe6b487c6a8b6fe1eb21904e33269e787e9a5b7b2dea6" Nov 03 01:05:46.691335 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:05:46.691297 4939 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 01:05:46.691446 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:05:46.691340 4939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w1-2729f2-openshift-default-9fb97b4fc-5ww48" event={"ID":"cba006d1-08cd-4fac-8bdd-86931f742838","Type":"ContainerStarted","Data":"ff61f9b0d6ca9947338dd8007679942bf635e5f5b02c19025e6d3a9789bce121"} Nov 03 01:05:46.691446 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:05:46.691395 4939 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 01:05:46.691531 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:05:46.691448 4939 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 01:05:46.720315 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:05:46.720289 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:05:47.044484 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:05:47.044384 4939 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w1-2729f2-openshift-default-9fb97b4fc-5ww48" Nov 03 01:05:47.044856 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:05:47.044826 4939 patch_prober.go:29] interesting pod/e2e-shared-a-w1-2729f2-openshift-default-9fb97b4fc-5ww48 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.163:15021/healthz/ready\": dial tcp 10.132.0.163:15021: connect: connection refused" start-of-body= Nov 03 01:05:47.044957 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:05:47.044898 4939 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w1-2729f2-openshift-default-9fb97b4fc-5ww48" podUID="cba006d1-08cd-4fac-8bdd-86931f742838" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.163:15021/healthz/ready\": dial tcp 10.132.0.163:15021: connect: connection refused" Nov 03 01:05:47.599150 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:05:47.599115 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:05:47.819935 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:05:47.815681 4939 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w1-2729f2-openshift-default-9fb97b4fc-5ww48" Nov 03 01:05:47.832404 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:05:47.823003 4939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w4-6c0ec0-openshift-default-79c7584bfc-8hz7h" event={"ID":"0a1fe883-48d5-4d06-ab8f-41698965dbbb","Type":"ContainerStarted","Data":"2bcb1634a74979a9c04d145a53f1fc437c24da2608a17bfee61d92aed28c52e0"} Nov 03 01:05:48.033793 ip-10-0-140-107 kubenswrapper[4939]: E1103 01:05:48.033711 4939 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-sthjh" containerName="extractor" Nov 03 01:05:48.036220 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:05:48.036197 4939 patch_prober.go:29] interesting pod/e2e-shared-a-w1-2729f2-openshift-default-9fb97b4fc-5ww48 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.163:15021/healthz/ready\": dial tcp 10.132.0.163:15021: connect: connection refused" start-of-body= Nov 03 01:05:48.036307 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:05:48.036242 4939 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w1-2729f2-openshift-default-9fb97b4fc-5ww48" podUID="cba006d1-08cd-4fac-8bdd-86931f742838" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.163:15021/healthz/ready\": dial tcp 10.132.0.163:15021: connect: connection refused" Nov 03 01:05:48.437084 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:05:48.437051 4939 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w4-6c0ec0-openshift-default-79c7584bfc-8hz7h" Nov 03 01:05:48.437460 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:05:48.437436 4939 patch_prober.go:29] interesting pod/e2e-shared-a-w4-6c0ec0-openshift-default-79c7584bfc-8hz7h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.165:15021/healthz/ready\": dial tcp 10.132.0.165:15021: connect: connection refused" start-of-body= Nov 03 01:05:48.437616 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:05:48.437590 4939 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-6c0ec0-openshift-default-79c7584bfc-8hz7h" podUID="0a1fe883-48d5-4d06-ab8f-41698965dbbb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.165:15021/healthz/ready\": dial tcp 10.132.0.165:15021: connect: connection refused" Nov 03 01:05:48.768987 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:05:48.768548 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:05:49.036462 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:05:49.036342 4939 patch_prober.go:29] interesting pod/e2e-shared-a-w1-2729f2-openshift-default-9fb97b4fc-5ww48 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.163:15021/healthz/ready\": dial tcp 10.132.0.163:15021: connect: connection refused" start-of-body= Nov 03 01:05:49.036841 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:05:49.036454 4939 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w1-2729f2-openshift-default-9fb97b4fc-5ww48" podUID="cba006d1-08cd-4fac-8bdd-86931f742838" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.163:15021/healthz/ready\": dial tcp 10.132.0.163:15021: connect: connection refused" Nov 03 01:05:49.239608 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:05:49.239579 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:05:49.433933 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:05:49.430757 4939 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-4pw7v"] Nov 03 01:05:49.433933 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:05:49.432268 4939 patch_prober.go:29] interesting pod/e2e-shared-a-w4-6c0ec0-openshift-default-79c7584bfc-8hz7h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.165:15021/healthz/ready\": dial tcp 10.132.0.165:15021: connect: connection refused" start-of-body= Nov 03 01:05:49.433933 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:05:49.432321 4939 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-6c0ec0-openshift-default-79c7584bfc-8hz7h" podUID="0a1fe883-48d5-4d06-ab8f-41698965dbbb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.165:15021/healthz/ready\": dial tcp 10.132.0.165:15021: connect: connection refused" Nov 03 01:05:49.845048 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:05:49.845013 4939 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w4-6c0ec0-openshift-default-79c7584bfc-8hz7h" Nov 03 01:05:50.036222 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:05:50.036190 4939 patch_prober.go:29] interesting pod/e2e-shared-a-w1-2729f2-openshift-default-9fb97b4fc-5ww48 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.163:15021/healthz/ready\": dial tcp 10.132.0.163:15021: connect: connection refused" start-of-body= Nov 03 01:05:50.036398 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:05:50.036246 4939 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w1-2729f2-openshift-default-9fb97b4fc-5ww48" podUID="cba006d1-08cd-4fac-8bdd-86931f742838" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.163:15021/healthz/ready\": dial tcp 10.132.0.163:15021: connect: connection refused" Nov 03 01:05:50.131460 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:05:50.131378 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:05:50.441409 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:05:50.441311 4939 patch_prober.go:29] interesting pod/e2e-shared-a-w4-6c0ec0-openshift-default-79c7584bfc-8hz7h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.165:15021/healthz/ready\": dial tcp 10.132.0.165:15021: connect: connection refused" start-of-body= Nov 03 01:05:50.441675 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:05:50.441647 4939 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-6c0ec0-openshift-default-79c7584bfc-8hz7h" podUID="0a1fe883-48d5-4d06-ab8f-41698965dbbb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.165:15021/healthz/ready\": dial tcp 10.132.0.165:15021: connect: connection refused" Nov 03 01:05:51.045447 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:05:51.041850 4939 patch_prober.go:29] interesting pod/e2e-shared-a-w1-2729f2-openshift-default-9fb97b4fc-5ww48 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.163:15021/healthz/ready\": dial tcp 10.132.0.163:15021: connect: connection refused" start-of-body= Nov 03 01:05:51.045780 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:05:51.045731 4939 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w1-2729f2-openshift-default-9fb97b4fc-5ww48" podUID="cba006d1-08cd-4fac-8bdd-86931f742838" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.163:15021/healthz/ready\": dial tcp 10.132.0.163:15021: connect: connection refused" Nov 03 01:05:51.415932 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:05:51.413594 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:05:51.431225 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:05:51.430819 4939 patch_prober.go:29] interesting pod/e2e-shared-a-w4-6c0ec0-openshift-default-79c7584bfc-8hz7h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.165:15021/healthz/ready\": dial tcp 10.132.0.165:15021: connect: connection refused" start-of-body= Nov 03 01:05:51.431225 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:05:51.430883 4939 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-6c0ec0-openshift-default-79c7584bfc-8hz7h" podUID="0a1fe883-48d5-4d06-ab8f-41698965dbbb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.165:15021/healthz/ready\": dial tcp 10.132.0.165:15021: connect: connection refused" Nov 03 01:05:52.036351 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:05:52.036318 4939 patch_prober.go:29] interesting pod/e2e-shared-a-w1-2729f2-openshift-default-9fb97b4fc-5ww48 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.163:15021/healthz/ready\": dial tcp 10.132.0.163:15021: connect: connection refused" start-of-body= Nov 03 01:05:52.036649 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:05:52.036622 4939 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w1-2729f2-openshift-default-9fb97b4fc-5ww48" podUID="cba006d1-08cd-4fac-8bdd-86931f742838" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.163:15021/healthz/ready\": dial tcp 10.132.0.163:15021: connect: connection refused" Nov 03 01:05:52.303115 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:05:52.303018 4939 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-autores-4248ae/autores-model-4248ae-kserve-5754c66dc7-whjqx" Nov 03 01:05:52.341410 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:05:52.341380 4939 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-autores-4248ae/autores-model-4248ae-kserve-5754c66dc7-whjqx" Nov 03 01:05:52.430938 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:05:52.430592 4939 patch_prober.go:29] interesting pod/e2e-shared-a-w4-6c0ec0-openshift-default-79c7584bfc-8hz7h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.165:15021/healthz/ready\": dial tcp 10.132.0.165:15021: connect: connection refused" start-of-body= Nov 03 01:05:52.430938 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:05:52.430655 4939 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-6c0ec0-openshift-default-79c7584bfc-8hz7h" podUID="0a1fe883-48d5-4d06-ab8f-41698965dbbb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.165:15021/healthz/ready\": dial tcp 10.132.0.165:15021: connect: connection refused" Nov 03 01:05:52.465170 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:05:52.465135 4939 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-n42s6_65d134ac-8a25-4c67-b6c7-cd752a098daf/console-operator/2.log" Nov 03 01:05:52.495511 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:05:52.495480 4939 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-n42s6_65d134ac-8a25-4c67-b6c7-cd752a098daf/console-operator/2.log" Nov 03 01:05:52.541491 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:05:52.541461 4939 kubelet.go:1689] "Image garbage collection succeeded" Nov 03 01:05:52.895712 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:05:52.895675 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:05:53.036759 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:05:53.036702 4939 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-w1-2729f2/auth-test-model-2729f2-kserve-6bb7bb4c97-n9nht"] Nov 03 01:05:53.037399 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:05:53.037341 4939 patch_prober.go:29] interesting pod/e2e-shared-a-w1-2729f2-openshift-default-9fb97b4fc-5ww48 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.163:15021/healthz/ready\": dial tcp 10.132.0.163:15021: connect: connection refused" start-of-body= Nov 03 01:05:53.037622 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:05:53.037593 4939 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w1-2729f2-openshift-default-9fb97b4fc-5ww48" podUID="cba006d1-08cd-4fac-8bdd-86931f742838" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.163:15021/healthz/ready\": dial tcp 10.132.0.163:15021: connect: connection refused" Nov 03 01:05:53.135821 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:05:53.135783 4939 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w1-2729f2/auth-test-model-2729f2-kserve-6bb7bb4c97-n9nht"] Nov 03 01:05:53.136118 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:05:53.136096 4939 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w1-2729f2/auth-test-model-2729f2-kserve-6bb7bb4c97-n9nht" Nov 03 01:05:53.138104 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:05:53.138060 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:05:53.145954 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:05:53.145854 4939 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w1-2729f2\"/\"default-dockercfg-lhn8w\"" Nov 03 01:05:53.150213 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:05:53.150189 4939 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w1-2729f2\"/\"kube-root-ca.crt\"" Nov 03 01:05:53.161708 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:05:53.161678 4939 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w1-2729f2\"/\"auth-test-model-2729f2-kserve-self-signed-certs\"" Nov 03 01:05:53.162009 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:05:53.161989 4939 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w1-2729f2\"/\"openshift-service-ca.crt\"" Nov 03 01:05:53.269935 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:05:53.257816 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f66fa3c5-ab64-4027-b004-399379a414f4-home\") pod \"auth-test-model-2729f2-kserve-6bb7bb4c97-n9nht\" (UID: \"f66fa3c5-ab64-4027-b004-399379a414f4\") " pod="ai-tenant-e2e-shared-a-w1-2729f2/auth-test-model-2729f2-kserve-6bb7bb4c97-n9nht" Nov 03 01:05:53.269935 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:05:53.257965 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f66fa3c5-ab64-4027-b004-399379a414f4-tmp-dir\") pod \"auth-test-model-2729f2-kserve-6bb7bb4c97-n9nht\" (UID: \"f66fa3c5-ab64-4027-b004-399379a414f4\") " pod="ai-tenant-e2e-shared-a-w1-2729f2/auth-test-model-2729f2-kserve-6bb7bb4c97-n9nht" Nov 03 01:05:53.269935 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:05:53.258190 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f66fa3c5-ab64-4027-b004-399379a414f4-model-cache\") pod \"auth-test-model-2729f2-kserve-6bb7bb4c97-n9nht\" (UID: \"f66fa3c5-ab64-4027-b004-399379a414f4\") " pod="ai-tenant-e2e-shared-a-w1-2729f2/auth-test-model-2729f2-kserve-6bb7bb4c97-n9nht" Nov 03 01:05:53.269935 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:05:53.258260 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f66fa3c5-ab64-4027-b004-399379a414f4-tls-certs\") pod \"auth-test-model-2729f2-kserve-6bb7bb4c97-n9nht\" (UID: \"f66fa3c5-ab64-4027-b004-399379a414f4\") " pod="ai-tenant-e2e-shared-a-w1-2729f2/auth-test-model-2729f2-kserve-6bb7bb4c97-n9nht" Nov 03 01:05:53.269935 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:05:53.258284 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zsdxn\" (UniqueName: \"kubernetes.io/projected/f66fa3c5-ab64-4027-b004-399379a414f4-kube-api-access-zsdxn\") pod \"auth-test-model-2729f2-kserve-6bb7bb4c97-n9nht\" (UID: \"f66fa3c5-ab64-4027-b004-399379a414f4\") " pod="ai-tenant-e2e-shared-a-w1-2729f2/auth-test-model-2729f2-kserve-6bb7bb4c97-n9nht" Nov 03 01:05:53.269935 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:05:53.258410 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f66fa3c5-ab64-4027-b004-399379a414f4-dshm\") pod \"auth-test-model-2729f2-kserve-6bb7bb4c97-n9nht\" (UID: \"f66fa3c5-ab64-4027-b004-399379a414f4\") " pod="ai-tenant-e2e-shared-a-w1-2729f2/auth-test-model-2729f2-kserve-6bb7bb4c97-n9nht" Nov 03 01:05:53.408023 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:05:53.395601 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zsdxn\" (UniqueName: \"kubernetes.io/projected/f66fa3c5-ab64-4027-b004-399379a414f4-kube-api-access-zsdxn\") pod \"auth-test-model-2729f2-kserve-6bb7bb4c97-n9nht\" (UID: \"f66fa3c5-ab64-4027-b004-399379a414f4\") " pod="ai-tenant-e2e-shared-a-w1-2729f2/auth-test-model-2729f2-kserve-6bb7bb4c97-n9nht" Nov 03 01:05:53.408023 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:05:53.395759 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f66fa3c5-ab64-4027-b004-399379a414f4-dshm\") pod \"auth-test-model-2729f2-kserve-6bb7bb4c97-n9nht\" (UID: \"f66fa3c5-ab64-4027-b004-399379a414f4\") " pod="ai-tenant-e2e-shared-a-w1-2729f2/auth-test-model-2729f2-kserve-6bb7bb4c97-n9nht" Nov 03 01:05:53.408023 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:05:53.395853 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f66fa3c5-ab64-4027-b004-399379a414f4-home\") pod \"auth-test-model-2729f2-kserve-6bb7bb4c97-n9nht\" (UID: \"f66fa3c5-ab64-4027-b004-399379a414f4\") " pod="ai-tenant-e2e-shared-a-w1-2729f2/auth-test-model-2729f2-kserve-6bb7bb4c97-n9nht" Nov 03 01:05:53.408023 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:05:53.395968 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f66fa3c5-ab64-4027-b004-399379a414f4-tmp-dir\") pod \"auth-test-model-2729f2-kserve-6bb7bb4c97-n9nht\" (UID: \"f66fa3c5-ab64-4027-b004-399379a414f4\") " pod="ai-tenant-e2e-shared-a-w1-2729f2/auth-test-model-2729f2-kserve-6bb7bb4c97-n9nht" Nov 03 01:05:53.408023 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:05:53.396193 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f66fa3c5-ab64-4027-b004-399379a414f4-model-cache\") pod \"auth-test-model-2729f2-kserve-6bb7bb4c97-n9nht\" (UID: \"f66fa3c5-ab64-4027-b004-399379a414f4\") " pod="ai-tenant-e2e-shared-a-w1-2729f2/auth-test-model-2729f2-kserve-6bb7bb4c97-n9nht" Nov 03 01:05:53.408023 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:05:53.396273 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f66fa3c5-ab64-4027-b004-399379a414f4-tls-certs\") pod \"auth-test-model-2729f2-kserve-6bb7bb4c97-n9nht\" (UID: \"f66fa3c5-ab64-4027-b004-399379a414f4\") " pod="ai-tenant-e2e-shared-a-w1-2729f2/auth-test-model-2729f2-kserve-6bb7bb4c97-n9nht" Nov 03 01:05:53.412591 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:05:53.412554 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f66fa3c5-ab64-4027-b004-399379a414f4-home\") pod \"auth-test-model-2729f2-kserve-6bb7bb4c97-n9nht\" (UID: \"f66fa3c5-ab64-4027-b004-399379a414f4\") " pod="ai-tenant-e2e-shared-a-w1-2729f2/auth-test-model-2729f2-kserve-6bb7bb4c97-n9nht" Nov 03 01:05:53.470027 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:05:53.413082 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f66fa3c5-ab64-4027-b004-399379a414f4-tmp-dir\") pod \"auth-test-model-2729f2-kserve-6bb7bb4c97-n9nht\" (UID: \"f66fa3c5-ab64-4027-b004-399379a414f4\") " pod="ai-tenant-e2e-shared-a-w1-2729f2/auth-test-model-2729f2-kserve-6bb7bb4c97-n9nht" Nov 03 01:05:53.503001 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:05:53.444048 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f66fa3c5-ab64-4027-b004-399379a414f4-model-cache\") pod \"auth-test-model-2729f2-kserve-6bb7bb4c97-n9nht\" (UID: \"f66fa3c5-ab64-4027-b004-399379a414f4\") " pod="ai-tenant-e2e-shared-a-w1-2729f2/auth-test-model-2729f2-kserve-6bb7bb4c97-n9nht" Nov 03 01:05:53.516046 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:05:53.455170 4939 patch_prober.go:29] interesting pod/e2e-shared-a-w4-6c0ec0-openshift-default-79c7584bfc-8hz7h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.165:15021/healthz/ready\": dial tcp 10.132.0.165:15021: connect: connection refused" start-of-body= Nov 03 01:05:53.516332 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:05:53.516299 4939 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-6c0ec0-openshift-default-79c7584bfc-8hz7h" podUID="0a1fe883-48d5-4d06-ab8f-41698965dbbb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.165:15021/healthz/ready\": dial tcp 10.132.0.165:15021: connect: connection refused" Nov 03 01:05:53.527959 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:05:53.476073 4939 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-autores-4248ae/autores-model-4248ae-kserve-5754c66dc7-whjqx" Nov 03 01:05:53.545112 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:05:53.530386 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zsdxn\" (UniqueName: \"kubernetes.io/projected/f66fa3c5-ab64-4027-b004-399379a414f4-kube-api-access-zsdxn\") pod \"auth-test-model-2729f2-kserve-6bb7bb4c97-n9nht\" (UID: \"f66fa3c5-ab64-4027-b004-399379a414f4\") " pod="ai-tenant-e2e-shared-a-w1-2729f2/auth-test-model-2729f2-kserve-6bb7bb4c97-n9nht" Nov 03 01:05:53.591020 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:05:53.590009 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f66fa3c5-ab64-4027-b004-399379a414f4-tls-certs\") pod \"auth-test-model-2729f2-kserve-6bb7bb4c97-n9nht\" (UID: \"f66fa3c5-ab64-4027-b004-399379a414f4\") " pod="ai-tenant-e2e-shared-a-w1-2729f2/auth-test-model-2729f2-kserve-6bb7bb4c97-n9nht" Nov 03 01:05:53.625501 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:05:53.622404 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f66fa3c5-ab64-4027-b004-399379a414f4-dshm\") pod \"auth-test-model-2729f2-kserve-6bb7bb4c97-n9nht\" (UID: \"f66fa3c5-ab64-4027-b004-399379a414f4\") " pod="ai-tenant-e2e-shared-a-w1-2729f2/auth-test-model-2729f2-kserve-6bb7bb4c97-n9nht" Nov 03 01:05:53.732549 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:05:53.732152 4939 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-556c14-669d5fb59f-cjtpf" Nov 03 01:05:53.783338 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:05:53.783301 4939 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w1-2729f2/auth-test-model-2729f2-kserve-6bb7bb4c97-n9nht" Nov 03 01:05:53.823185 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:05:53.823151 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:05:53.998989 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:05:53.976322 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:05:54.433901 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:05:54.426217 4939 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w1-2729f2-openshift-default-9fb97b4fc-5ww48" Nov 03 01:05:54.433901 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:05:54.431850 4939 patch_prober.go:29] interesting pod/e2e-shared-a-w4-6c0ec0-openshift-default-79c7584bfc-8hz7h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.165:15021/healthz/ready\": dial tcp 10.132.0.165:15021: connect: connection refused" start-of-body= Nov 03 01:05:54.433901 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:05:54.431929 4939 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-6c0ec0-openshift-default-79c7584bfc-8hz7h" podUID="0a1fe883-48d5-4d06-ab8f-41698965dbbb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.165:15021/healthz/ready\": dial tcp 10.132.0.165:15021: connect: connection refused" Nov 03 01:05:54.434820 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:05:54.434714 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:05:54.436332 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:05:54.436297 4939 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w1-2729f2-openshift-default-9fb97b4fc-5ww48" Nov 03 01:05:54.595937 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:05:54.595887 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:05:54.712947 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:05:54.709179 4939 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-556c14-566c7cc65c-lcbl4" Nov 03 01:05:54.908954 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:05:54.908775 4939 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w1-2729f2/auth-test-model-2729f2-kserve-6bb7bb4c97-n9nht"] Nov 03 01:05:55.196934 ip-10-0-140-107 kubenswrapper[4939]: E1103 01:05:55.185402 4939 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podf66fa3c5_ab64_4027_b004_399379a414f4.slice/crio-conmon-6affc1952e113f0aec89986179dad54379f158fb3effa0c472c7451e44be964c.scope\": RecentStats: unable to find data in memory cache]" Nov 03 01:05:55.312575 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:05:55.312487 4939 scope.go:122] "RemoveContainer" containerID="a70f79cf1ffff54da53b7c55ba0553d17a6b758e96b3bfc7467c7b99fb6c957e" Nov 03 01:05:55.685944 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:05:55.682049 4939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w1-2729f2/auth-test-model-2729f2-kserve-6bb7bb4c97-n9nht" event={"ID":"f66fa3c5-ab64-4027-b004-399379a414f4","Type":"ContainerStarted","Data":"6affc1952e113f0aec89986179dad54379f158fb3effa0c472c7451e44be964c"} Nov 03 01:05:55.685944 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:05:55.682097 4939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w1-2729f2/auth-test-model-2729f2-kserve-6bb7bb4c97-n9nht" event={"ID":"f66fa3c5-ab64-4027-b004-399379a414f4","Type":"ContainerStarted","Data":"bce579271af93f2a57fb0eb988b91219c0f72a999c963756b90ad16c5abb3769"} Nov 03 01:05:55.795973 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:05:55.792243 4939 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-w1-2729f2/auth-test-model-2729f2-kserve-6bb7bb4c97-n9nht" Nov 03 01:05:55.801974 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:05:55.801879 4939 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-w1-2729f2/auth-test-model-2729f2-kserve-6bb7bb4c97-n9nht" podStartSLOduration=2.801856051 podStartE2EDuration="2.801856051s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 01:05:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 01:05:55.79217618 +0000 UTC m=+1504.056511126" watchObservedRunningTime="2026-11-03 01:05:55.801856051 +0000 UTC m=+1504.066190997" Nov 03 01:05:55.968108 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:05:55.964928 4939 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w4-6c0ec0-openshift-default-79c7584bfc-8hz7h" Nov 03 01:05:56.005360 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:05:55.998782 4939 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w4-6c0ec0-openshift-default-79c7584bfc-8hz7h" Nov 03 01:05:56.309115 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:05:56.308986 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:05:56.721923 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:05:56.721883 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:05:57.494569 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:05:57.494529 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:05:58.389847 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:05:58.389796 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:05:59.107883 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:05:59.107847 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:05:59.891156 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:05:59.891120 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:06:00.621396 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:06:00.621313 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:06:01.410579 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:06:01.410532 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:06:02.291290 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:06:02.291190 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:06:02.742818 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:06:02.742784 4939 generic.go:356] "Generic (PLEG): container finished" podID="5b0cabc6-4960-4dc1-8317-3faebf25b30c" containerID="2f9dce65eb234b7acf638877b722bc4efa6182ce79db38c29344f4c05eee7784" exitCode=0 Nov 03 01:06:02.743260 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:06:02.742858 4939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w4-4a697f-openshift-default-666db9f8f5-qn8tj" event={"ID":"5b0cabc6-4960-4dc1-8317-3faebf25b30c","Type":"ContainerDied","Data":"2f9dce65eb234b7acf638877b722bc4efa6182ce79db38c29344f4c05eee7784"} Nov 03 01:06:02.743367 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:06:02.743347 4939 scope.go:122] "RemoveContainer" containerID="2f9dce65eb234b7acf638877b722bc4efa6182ce79db38c29344f4c05eee7784" Nov 03 01:06:02.745057 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:06:02.745017 4939 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 01:06:02.745135 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:06:02.745099 4939 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 01:06:02.745172 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:06:02.745133 4939 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 01:06:02.814549 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:06:02.814509 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:06:03.044865 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:06:03.044784 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:06:03.253700 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:06:03.253666 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:06:03.608373 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:06:03.608336 4939 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w0-020af9-openshift-default-d77df45d5-mpgn5"] Nov 03 01:06:03.622062 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:06:03.622029 4939 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w0-020af9-openshift-default-d77df45d5-mpgn5"] Nov 03 01:06:03.622426 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:06:03.622354 4939 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w0-020af9-openshift-default-d77df45d5-mpgn5" Nov 03 01:06:03.629820 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:06:03.629800 4939 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-w0-020af9-openshift-default-dockercfg-6wsrt\"" Nov 03 01:06:03.686233 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:06:03.686156 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:06:03.696717 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:06:03.696680 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d7701de4-582e-442f-abc1-13e20bb68b48-workload-certs\") pod \"e2e-shared-b-w0-020af9-openshift-default-d77df45d5-mpgn5\" (UID: \"d7701de4-582e-442f-abc1-13e20bb68b48\") " pod="openshift-ingress/e2e-shared-b-w0-020af9-openshift-default-d77df45d5-mpgn5" Nov 03 01:06:03.697022 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:06:03.696994 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d7701de4-582e-442f-abc1-13e20bb68b48-istio-data\") pod \"e2e-shared-b-w0-020af9-openshift-default-d77df45d5-mpgn5\" (UID: \"d7701de4-582e-442f-abc1-13e20bb68b48\") " pod="openshift-ingress/e2e-shared-b-w0-020af9-openshift-default-d77df45d5-mpgn5" Nov 03 01:06:03.697214 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:06:03.697194 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d7701de4-582e-442f-abc1-13e20bb68b48-istio-podinfo\") pod \"e2e-shared-b-w0-020af9-openshift-default-d77df45d5-mpgn5\" (UID: \"d7701de4-582e-442f-abc1-13e20bb68b48\") " pod="openshift-ingress/e2e-shared-b-w0-020af9-openshift-default-d77df45d5-mpgn5" Nov 03 01:06:03.697381 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:06:03.697362 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d7701de4-582e-442f-abc1-13e20bb68b48-workload-socket\") pod \"e2e-shared-b-w0-020af9-openshift-default-d77df45d5-mpgn5\" (UID: \"d7701de4-582e-442f-abc1-13e20bb68b48\") " pod="openshift-ingress/e2e-shared-b-w0-020af9-openshift-default-d77df45d5-mpgn5" Nov 03 01:06:03.697693 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:06:03.697673 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d7701de4-582e-442f-abc1-13e20bb68b48-istio-token\") pod \"e2e-shared-b-w0-020af9-openshift-default-d77df45d5-mpgn5\" (UID: \"d7701de4-582e-442f-abc1-13e20bb68b48\") " pod="openshift-ingress/e2e-shared-b-w0-020af9-openshift-default-d77df45d5-mpgn5" Nov 03 01:06:03.697822 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:06:03.697806 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vnth5\" (UniqueName: \"kubernetes.io/projected/d7701de4-582e-442f-abc1-13e20bb68b48-kube-api-access-vnth5\") pod \"e2e-shared-b-w0-020af9-openshift-default-d77df45d5-mpgn5\" (UID: \"d7701de4-582e-442f-abc1-13e20bb68b48\") " pod="openshift-ingress/e2e-shared-b-w0-020af9-openshift-default-d77df45d5-mpgn5" Nov 03 01:06:03.698096 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:06:03.698077 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d7701de4-582e-442f-abc1-13e20bb68b48-istio-envoy\") pod \"e2e-shared-b-w0-020af9-openshift-default-d77df45d5-mpgn5\" (UID: \"d7701de4-582e-442f-abc1-13e20bb68b48\") " pod="openshift-ingress/e2e-shared-b-w0-020af9-openshift-default-d77df45d5-mpgn5" Nov 03 01:06:03.698241 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:06:03.698224 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d7701de4-582e-442f-abc1-13e20bb68b48-istiod-ca-cert\") pod \"e2e-shared-b-w0-020af9-openshift-default-d77df45d5-mpgn5\" (UID: \"d7701de4-582e-442f-abc1-13e20bb68b48\") " pod="openshift-ingress/e2e-shared-b-w0-020af9-openshift-default-d77df45d5-mpgn5" Nov 03 01:06:03.698465 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:06:03.698447 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d7701de4-582e-442f-abc1-13e20bb68b48-credential-socket\") pod \"e2e-shared-b-w0-020af9-openshift-default-d77df45d5-mpgn5\" (UID: \"d7701de4-582e-442f-abc1-13e20bb68b48\") " pod="openshift-ingress/e2e-shared-b-w0-020af9-openshift-default-d77df45d5-mpgn5" Nov 03 01:06:03.797963 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:06:03.791374 4939 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-w1-2729f2/auth-test-model-2729f2-kserve-6bb7bb4c97-n9nht" Nov 03 01:06:03.805982 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:06:03.801888 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d7701de4-582e-442f-abc1-13e20bb68b48-workload-certs\") pod \"e2e-shared-b-w0-020af9-openshift-default-d77df45d5-mpgn5\" (UID: \"d7701de4-582e-442f-abc1-13e20bb68b48\") " pod="openshift-ingress/e2e-shared-b-w0-020af9-openshift-default-d77df45d5-mpgn5" Nov 03 01:06:03.805982 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:06:03.801997 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d7701de4-582e-442f-abc1-13e20bb68b48-istio-data\") pod \"e2e-shared-b-w0-020af9-openshift-default-d77df45d5-mpgn5\" (UID: \"d7701de4-582e-442f-abc1-13e20bb68b48\") " pod="openshift-ingress/e2e-shared-b-w0-020af9-openshift-default-d77df45d5-mpgn5" Nov 03 01:06:03.805982 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:06:03.802051 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d7701de4-582e-442f-abc1-13e20bb68b48-istio-podinfo\") pod \"e2e-shared-b-w0-020af9-openshift-default-d77df45d5-mpgn5\" (UID: \"d7701de4-582e-442f-abc1-13e20bb68b48\") " pod="openshift-ingress/e2e-shared-b-w0-020af9-openshift-default-d77df45d5-mpgn5" Nov 03 01:06:03.805982 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:06:03.802122 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d7701de4-582e-442f-abc1-13e20bb68b48-workload-socket\") pod \"e2e-shared-b-w0-020af9-openshift-default-d77df45d5-mpgn5\" (UID: \"d7701de4-582e-442f-abc1-13e20bb68b48\") " pod="openshift-ingress/e2e-shared-b-w0-020af9-openshift-default-d77df45d5-mpgn5" Nov 03 01:06:03.805982 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:06:03.802317 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d7701de4-582e-442f-abc1-13e20bb68b48-istio-token\") pod \"e2e-shared-b-w0-020af9-openshift-default-d77df45d5-mpgn5\" (UID: \"d7701de4-582e-442f-abc1-13e20bb68b48\") " pod="openshift-ingress/e2e-shared-b-w0-020af9-openshift-default-d77df45d5-mpgn5" Nov 03 01:06:03.805982 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:06:03.802347 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vnth5\" (UniqueName: \"kubernetes.io/projected/d7701de4-582e-442f-abc1-13e20bb68b48-kube-api-access-vnth5\") pod \"e2e-shared-b-w0-020af9-openshift-default-d77df45d5-mpgn5\" (UID: \"d7701de4-582e-442f-abc1-13e20bb68b48\") " pod="openshift-ingress/e2e-shared-b-w0-020af9-openshift-default-d77df45d5-mpgn5" Nov 03 01:06:03.805982 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:06:03.802373 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d7701de4-582e-442f-abc1-13e20bb68b48-istio-envoy\") pod \"e2e-shared-b-w0-020af9-openshift-default-d77df45d5-mpgn5\" (UID: \"d7701de4-582e-442f-abc1-13e20bb68b48\") " pod="openshift-ingress/e2e-shared-b-w0-020af9-openshift-default-d77df45d5-mpgn5" Nov 03 01:06:03.805982 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:06:03.802404 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d7701de4-582e-442f-abc1-13e20bb68b48-istiod-ca-cert\") pod \"e2e-shared-b-w0-020af9-openshift-default-d77df45d5-mpgn5\" (UID: \"d7701de4-582e-442f-abc1-13e20bb68b48\") " pod="openshift-ingress/e2e-shared-b-w0-020af9-openshift-default-d77df45d5-mpgn5" Nov 03 01:06:03.805982 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:06:03.802517 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d7701de4-582e-442f-abc1-13e20bb68b48-credential-socket\") pod \"e2e-shared-b-w0-020af9-openshift-default-d77df45d5-mpgn5\" (UID: \"d7701de4-582e-442f-abc1-13e20bb68b48\") " pod="openshift-ingress/e2e-shared-b-w0-020af9-openshift-default-d77df45d5-mpgn5" Nov 03 01:06:03.805982 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:06:03.803039 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d7701de4-582e-442f-abc1-13e20bb68b48-credential-socket\") pod \"e2e-shared-b-w0-020af9-openshift-default-d77df45d5-mpgn5\" (UID: \"d7701de4-582e-442f-abc1-13e20bb68b48\") " pod="openshift-ingress/e2e-shared-b-w0-020af9-openshift-default-d77df45d5-mpgn5" Nov 03 01:06:03.805982 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:06:03.803443 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d7701de4-582e-442f-abc1-13e20bb68b48-workload-certs\") pod \"e2e-shared-b-w0-020af9-openshift-default-d77df45d5-mpgn5\" (UID: \"d7701de4-582e-442f-abc1-13e20bb68b48\") " pod="openshift-ingress/e2e-shared-b-w0-020af9-openshift-default-d77df45d5-mpgn5" Nov 03 01:06:03.805982 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:06:03.803685 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d7701de4-582e-442f-abc1-13e20bb68b48-istio-data\") pod \"e2e-shared-b-w0-020af9-openshift-default-d77df45d5-mpgn5\" (UID: \"d7701de4-582e-442f-abc1-13e20bb68b48\") " pod="openshift-ingress/e2e-shared-b-w0-020af9-openshift-default-d77df45d5-mpgn5" Nov 03 01:06:03.848732 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:06:03.838023 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d7701de4-582e-442f-abc1-13e20bb68b48-istiod-ca-cert\") pod \"e2e-shared-b-w0-020af9-openshift-default-d77df45d5-mpgn5\" (UID: \"d7701de4-582e-442f-abc1-13e20bb68b48\") " pod="openshift-ingress/e2e-shared-b-w0-020af9-openshift-default-d77df45d5-mpgn5" Nov 03 01:06:03.848732 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:06:03.848668 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d7701de4-582e-442f-abc1-13e20bb68b48-workload-socket\") pod \"e2e-shared-b-w0-020af9-openshift-default-d77df45d5-mpgn5\" (UID: \"d7701de4-582e-442f-abc1-13e20bb68b48\") " pod="openshift-ingress/e2e-shared-b-w0-020af9-openshift-default-d77df45d5-mpgn5" Nov 03 01:06:03.999114 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:06:03.998990 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d7701de4-582e-442f-abc1-13e20bb68b48-istio-envoy\") pod \"e2e-shared-b-w0-020af9-openshift-default-d77df45d5-mpgn5\" (UID: \"d7701de4-582e-442f-abc1-13e20bb68b48\") " pod="openshift-ingress/e2e-shared-b-w0-020af9-openshift-default-d77df45d5-mpgn5" Nov 03 01:06:04.005955 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:06:04.000618 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d7701de4-582e-442f-abc1-13e20bb68b48-istio-podinfo\") pod \"e2e-shared-b-w0-020af9-openshift-default-d77df45d5-mpgn5\" (UID: \"d7701de4-582e-442f-abc1-13e20bb68b48\") " pod="openshift-ingress/e2e-shared-b-w0-020af9-openshift-default-d77df45d5-mpgn5" Nov 03 01:06:04.034946 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:06:04.023557 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vnth5\" (UniqueName: \"kubernetes.io/projected/d7701de4-582e-442f-abc1-13e20bb68b48-kube-api-access-vnth5\") pod \"e2e-shared-b-w0-020af9-openshift-default-d77df45d5-mpgn5\" (UID: \"d7701de4-582e-442f-abc1-13e20bb68b48\") " pod="openshift-ingress/e2e-shared-b-w0-020af9-openshift-default-d77df45d5-mpgn5" Nov 03 01:06:04.034946 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:06:04.033527 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d7701de4-582e-442f-abc1-13e20bb68b48-istio-token\") pod \"e2e-shared-b-w0-020af9-openshift-default-d77df45d5-mpgn5\" (UID: \"d7701de4-582e-442f-abc1-13e20bb68b48\") " pod="openshift-ingress/e2e-shared-b-w0-020af9-openshift-default-d77df45d5-mpgn5" Nov 03 01:06:04.038939 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:06:04.036717 4939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w4-4a697f-openshift-default-666db9f8f5-qn8tj" event={"ID":"5b0cabc6-4960-4dc1-8317-3faebf25b30c","Type":"ContainerStarted","Data":"58e1616a5833fada05ba97f4eb82d69ef9c4f95be968f1d1f96f1849f94d30cd"} Nov 03 01:06:04.135489 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:06:04.135190 4939 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-w1-2729f2/auth-test-model-2729f2-kserve-6bb7bb4c97-n9nht" Nov 03 01:06:04.146019 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:06:04.145842 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:06:04.252974 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:06:04.252872 4939 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w0-020af9-openshift-default-d77df45d5-mpgn5" Nov 03 01:06:04.270677 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:06:04.267360 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:06:04.443084 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:06:04.443058 4939 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w4-4a697f-openshift-default-666db9f8f5-qn8tj" Nov 03 01:06:04.444058 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:06:04.444029 4939 patch_prober.go:29] interesting pod/e2e-shared-b-w4-4a697f-openshift-default-666db9f8f5-qn8tj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.175:15021/healthz/ready\": dial tcp 10.132.0.175:15021: connect: connection refused" start-of-body= Nov 03 01:06:04.444199 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:06:04.444100 4939 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w4-4a697f-openshift-default-666db9f8f5-qn8tj" podUID="5b0cabc6-4960-4dc1-8317-3faebf25b30c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.175:15021/healthz/ready\": dial tcp 10.132.0.175:15021: connect: connection refused" Nov 03 01:06:04.645748 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:06:04.645721 4939 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w0-020af9-openshift-default-d77df45d5-mpgn5"] Nov 03 01:06:04.675994 ip-10-0-140-107 kubenswrapper[4939]: W1103 01:06:04.647296 4939 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd7701de4_582e_442f_abc1_13e20bb68b48.slice/crio-0bc1bc040fddcd83d0ca53a57f427d7dd98b795ded8669fbf8bfaa73f94b3838 WatchSource:0}: Error finding container 0bc1bc040fddcd83d0ca53a57f427d7dd98b795ded8669fbf8bfaa73f94b3838: Status 404 returned error can't find the container with id 0bc1bc040fddcd83d0ca53a57f427d7dd98b795ded8669fbf8bfaa73f94b3838 Nov 03 01:06:04.675994 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:06:04.651743 4939 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 01:06:04.675994 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:06:04.651838 4939 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 01:06:04.675994 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:06:04.651882 4939 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 01:06:04.806943 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:06:04.804843 4939 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-w1-2729f2/auth-test-model-2729f2-kserve-6bb7bb4c97-n9nht" Nov 03 01:06:05.044277 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:06:05.025756 4939 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w4-4a697f-openshift-default-666db9f8f5-qn8tj" Nov 03 01:06:05.046227 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:06:05.045756 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:06:05.156928 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:06:05.155359 4939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w0-020af9-openshift-default-d77df45d5-mpgn5" event={"ID":"d7701de4-582e-442f-abc1-13e20bb68b48","Type":"ContainerStarted","Data":"fd71a31300e5566a1081fb5c193940d6e11b98826ebca49ff97ee875fc10533b"} Nov 03 01:06:05.156928 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:06:05.155436 4939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w0-020af9-openshift-default-d77df45d5-mpgn5" event={"ID":"d7701de4-582e-442f-abc1-13e20bb68b48","Type":"ContainerStarted","Data":"0bc1bc040fddcd83d0ca53a57f427d7dd98b795ded8669fbf8bfaa73f94b3838"} Nov 03 01:06:05.229743 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:06:05.229681 4939 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w0-020af9-openshift-default-d77df45d5-mpgn5" podStartSLOduration=2.22966002 podStartE2EDuration="2.22966002s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 01:06:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 01:06:05.177461663 +0000 UTC m=+1513.441796609" watchObservedRunningTime="2026-11-03 01:06:05.22966002 +0000 UTC m=+1513.493994966" Nov 03 01:06:05.257180 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:06:05.257147 4939 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w0-020af9-openshift-default-d77df45d5-mpgn5" Nov 03 01:06:05.257480 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:06:05.257457 4939 patch_prober.go:29] interesting pod/e2e-shared-b-w0-020af9-openshift-default-d77df45d5-mpgn5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.185:15021/healthz/ready\": dial tcp 10.132.0.185:15021: connect: connection refused" start-of-body= Nov 03 01:06:05.257663 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:06:05.257634 4939 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w0-020af9-openshift-default-d77df45d5-mpgn5" podUID="d7701de4-582e-442f-abc1-13e20bb68b48" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.185:15021/healthz/ready\": dial tcp 10.132.0.185:15021: connect: connection refused" Nov 03 01:06:05.445984 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:06:05.445423 4939 patch_prober.go:29] interesting pod/e2e-shared-b-w4-4a697f-openshift-default-666db9f8f5-qn8tj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.175:15021/healthz/ready\": dial tcp 10.132.0.175:15021: connect: connection refused" start-of-body= Nov 03 01:06:05.445984 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:06:05.445487 4939 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w4-4a697f-openshift-default-666db9f8f5-qn8tj" podUID="5b0cabc6-4960-4dc1-8317-3faebf25b30c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.175:15021/healthz/ready\": dial tcp 10.132.0.175:15021: connect: connection refused" Nov 03 01:06:05.542722 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:06:05.542672 4939 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-w4-6c0ec0/auth-test-model-6c0ec0-kserve-589c79bd69-vznsb"] Nov 03 01:06:05.583152 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:06:05.583078 4939 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w4-6c0ec0/auth-test-model-6c0ec0-kserve-589c79bd69-vznsb" Nov 03 01:06:05.585128 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:06:05.585103 4939 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w4-6c0ec0/auth-test-model-6c0ec0-kserve-589c79bd69-vznsb"] Nov 03 01:06:05.593351 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:06:05.593328 4939 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w4-6c0ec0\"/\"openshift-service-ca.crt\"" Nov 03 01:06:05.593487 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:06:05.593406 4939 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w4-6c0ec0\"/\"kube-root-ca.crt\"" Nov 03 01:06:05.593673 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:06:05.593623 4939 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w4-6c0ec0\"/\"default-dockercfg-kbnv9\"" Nov 03 01:06:05.593797 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:06:05.593691 4939 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w4-6c0ec0\"/\"auth-test-model-6c0ec0-kserve-self-signed-certs\"" Nov 03 01:06:05.623334 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:06:05.623291 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:06:05.682148 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:06:05.682105 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d21cb247-bacd-4af2-a7b1-d04b975f0993-tmp-dir\") pod \"auth-test-model-6c0ec0-kserve-589c79bd69-vznsb\" (UID: \"d21cb247-bacd-4af2-a7b1-d04b975f0993\") " pod="ai-tenant-e2e-shared-a-w4-6c0ec0/auth-test-model-6c0ec0-kserve-589c79bd69-vznsb" Nov 03 01:06:05.682342 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:06:05.682153 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xjmf4\" (UniqueName: \"kubernetes.io/projected/d21cb247-bacd-4af2-a7b1-d04b975f0993-kube-api-access-xjmf4\") pod \"auth-test-model-6c0ec0-kserve-589c79bd69-vznsb\" (UID: \"d21cb247-bacd-4af2-a7b1-d04b975f0993\") " pod="ai-tenant-e2e-shared-a-w4-6c0ec0/auth-test-model-6c0ec0-kserve-589c79bd69-vznsb" Nov 03 01:06:05.682342 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:06:05.682188 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d21cb247-bacd-4af2-a7b1-d04b975f0993-model-cache\") pod \"auth-test-model-6c0ec0-kserve-589c79bd69-vznsb\" (UID: \"d21cb247-bacd-4af2-a7b1-d04b975f0993\") " pod="ai-tenant-e2e-shared-a-w4-6c0ec0/auth-test-model-6c0ec0-kserve-589c79bd69-vznsb" Nov 03 01:06:05.682432 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:06:05.682411 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d21cb247-bacd-4af2-a7b1-d04b975f0993-home\") pod \"auth-test-model-6c0ec0-kserve-589c79bd69-vznsb\" (UID: \"d21cb247-bacd-4af2-a7b1-d04b975f0993\") " pod="ai-tenant-e2e-shared-a-w4-6c0ec0/auth-test-model-6c0ec0-kserve-589c79bd69-vznsb" Nov 03 01:06:05.682480 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:06:05.682451 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d21cb247-bacd-4af2-a7b1-d04b975f0993-tls-certs\") pod \"auth-test-model-6c0ec0-kserve-589c79bd69-vznsb\" (UID: \"d21cb247-bacd-4af2-a7b1-d04b975f0993\") " pod="ai-tenant-e2e-shared-a-w4-6c0ec0/auth-test-model-6c0ec0-kserve-589c79bd69-vznsb" Nov 03 01:06:05.682480 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:06:05.682473 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d21cb247-bacd-4af2-a7b1-d04b975f0993-dshm\") pod \"auth-test-model-6c0ec0-kserve-589c79bd69-vznsb\" (UID: \"d21cb247-bacd-4af2-a7b1-d04b975f0993\") " pod="ai-tenant-e2e-shared-a-w4-6c0ec0/auth-test-model-6c0ec0-kserve-589c79bd69-vznsb" Nov 03 01:06:05.788272 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:06:05.788172 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d21cb247-bacd-4af2-a7b1-d04b975f0993-tmp-dir\") pod \"auth-test-model-6c0ec0-kserve-589c79bd69-vznsb\" (UID: \"d21cb247-bacd-4af2-a7b1-d04b975f0993\") " pod="ai-tenant-e2e-shared-a-w4-6c0ec0/auth-test-model-6c0ec0-kserve-589c79bd69-vznsb" Nov 03 01:06:05.799930 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:06:05.799881 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xjmf4\" (UniqueName: \"kubernetes.io/projected/d21cb247-bacd-4af2-a7b1-d04b975f0993-kube-api-access-xjmf4\") pod \"auth-test-model-6c0ec0-kserve-589c79bd69-vznsb\" (UID: \"d21cb247-bacd-4af2-a7b1-d04b975f0993\") " pod="ai-tenant-e2e-shared-a-w4-6c0ec0/auth-test-model-6c0ec0-kserve-589c79bd69-vznsb" Nov 03 01:06:05.804034 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:06:05.803990 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d21cb247-bacd-4af2-a7b1-d04b975f0993-model-cache\") pod \"auth-test-model-6c0ec0-kserve-589c79bd69-vznsb\" (UID: \"d21cb247-bacd-4af2-a7b1-d04b975f0993\") " pod="ai-tenant-e2e-shared-a-w4-6c0ec0/auth-test-model-6c0ec0-kserve-589c79bd69-vznsb" Nov 03 01:06:05.808928 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:06:05.804420 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d21cb247-bacd-4af2-a7b1-d04b975f0993-home\") pod \"auth-test-model-6c0ec0-kserve-589c79bd69-vznsb\" (UID: \"d21cb247-bacd-4af2-a7b1-d04b975f0993\") " pod="ai-tenant-e2e-shared-a-w4-6c0ec0/auth-test-model-6c0ec0-kserve-589c79bd69-vznsb" Nov 03 01:06:05.809503 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:06:05.809480 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d21cb247-bacd-4af2-a7b1-d04b975f0993-tls-certs\") pod \"auth-test-model-6c0ec0-kserve-589c79bd69-vznsb\" (UID: \"d21cb247-bacd-4af2-a7b1-d04b975f0993\") " pod="ai-tenant-e2e-shared-a-w4-6c0ec0/auth-test-model-6c0ec0-kserve-589c79bd69-vznsb" Nov 03 01:06:05.814015 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:06:05.813985 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d21cb247-bacd-4af2-a7b1-d04b975f0993-dshm\") pod \"auth-test-model-6c0ec0-kserve-589c79bd69-vznsb\" (UID: \"d21cb247-bacd-4af2-a7b1-d04b975f0993\") " pod="ai-tenant-e2e-shared-a-w4-6c0ec0/auth-test-model-6c0ec0-kserve-589c79bd69-vznsb" Nov 03 01:06:05.837070 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:06:05.807574 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d21cb247-bacd-4af2-a7b1-d04b975f0993-home\") pod \"auth-test-model-6c0ec0-kserve-589c79bd69-vznsb\" (UID: \"d21cb247-bacd-4af2-a7b1-d04b975f0993\") " pod="ai-tenant-e2e-shared-a-w4-6c0ec0/auth-test-model-6c0ec0-kserve-589c79bd69-vznsb" Nov 03 01:06:05.837416 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:06:05.788891 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d21cb247-bacd-4af2-a7b1-d04b975f0993-tmp-dir\") pod \"auth-test-model-6c0ec0-kserve-589c79bd69-vznsb\" (UID: \"d21cb247-bacd-4af2-a7b1-d04b975f0993\") " pod="ai-tenant-e2e-shared-a-w4-6c0ec0/auth-test-model-6c0ec0-kserve-589c79bd69-vznsb" Nov 03 01:06:05.837416 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:06:05.807324 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d21cb247-bacd-4af2-a7b1-d04b975f0993-model-cache\") pod \"auth-test-model-6c0ec0-kserve-589c79bd69-vznsb\" (UID: \"d21cb247-bacd-4af2-a7b1-d04b975f0993\") " pod="ai-tenant-e2e-shared-a-w4-6c0ec0/auth-test-model-6c0ec0-kserve-589c79bd69-vznsb" Nov 03 01:06:05.855932 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:06:05.843896 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d21cb247-bacd-4af2-a7b1-d04b975f0993-tls-certs\") pod \"auth-test-model-6c0ec0-kserve-589c79bd69-vznsb\" (UID: \"d21cb247-bacd-4af2-a7b1-d04b975f0993\") " pod="ai-tenant-e2e-shared-a-w4-6c0ec0/auth-test-model-6c0ec0-kserve-589c79bd69-vznsb" Nov 03 01:06:05.855932 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:06:05.844581 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xjmf4\" (UniqueName: \"kubernetes.io/projected/d21cb247-bacd-4af2-a7b1-d04b975f0993-kube-api-access-xjmf4\") pod \"auth-test-model-6c0ec0-kserve-589c79bd69-vznsb\" (UID: \"d21cb247-bacd-4af2-a7b1-d04b975f0993\") " pod="ai-tenant-e2e-shared-a-w4-6c0ec0/auth-test-model-6c0ec0-kserve-589c79bd69-vznsb" Nov 03 01:06:05.855932 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:06:05.855352 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d21cb247-bacd-4af2-a7b1-d04b975f0993-dshm\") pod \"auth-test-model-6c0ec0-kserve-589c79bd69-vznsb\" (UID: \"d21cb247-bacd-4af2-a7b1-d04b975f0993\") " pod="ai-tenant-e2e-shared-a-w4-6c0ec0/auth-test-model-6c0ec0-kserve-589c79bd69-vznsb" Nov 03 01:06:05.908548 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:06:05.900462 4939 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w4-6c0ec0/auth-test-model-6c0ec0-kserve-589c79bd69-vznsb" Nov 03 01:06:06.217707 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:06:06.217674 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:06:06.296936 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:06:06.294320 4939 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w0-020af9-openshift-default-d77df45d5-mpgn5" Nov 03 01:06:06.402985 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:06:06.402960 4939 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w4-6c0ec0/auth-test-model-6c0ec0-kserve-589c79bd69-vznsb"] Nov 03 01:06:06.404366 ip-10-0-140-107 kubenswrapper[4939]: W1103 01:06:06.404336 4939 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podd21cb247_bacd_4af2_a7b1_d04b975f0993.slice/crio-3d2bdf00c4724290efca7bed06523f1271eafc6169c64acd02828274a3050a25 WatchSource:0}: Error finding container 3d2bdf00c4724290efca7bed06523f1271eafc6169c64acd02828274a3050a25: Status 404 returned error can't find the container with id 3d2bdf00c4724290efca7bed06523f1271eafc6169c64acd02828274a3050a25 Nov 03 01:06:06.443019 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:06:06.442980 4939 patch_prober.go:29] interesting pod/e2e-shared-b-w4-4a697f-openshift-default-666db9f8f5-qn8tj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.175:15021/healthz/ready\": dial tcp 10.132.0.175:15021: connect: connection refused" start-of-body= Nov 03 01:06:06.443128 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:06:06.443040 4939 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w4-4a697f-openshift-default-666db9f8f5-qn8tj" podUID="5b0cabc6-4960-4dc1-8317-3faebf25b30c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.175:15021/healthz/ready\": dial tcp 10.132.0.175:15021: connect: connection refused" Nov 03 01:06:07.182387 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:06:07.182351 4939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w4-6c0ec0/auth-test-model-6c0ec0-kserve-589c79bd69-vznsb" event={"ID":"d21cb247-bacd-4af2-a7b1-d04b975f0993","Type":"ContainerStarted","Data":"5ad7dbb8fc3fc68239c9e1e97fee011fa7a711add9530153f8ead02c8fa7053c"} Nov 03 01:06:07.182870 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:06:07.182422 4939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w4-6c0ec0/auth-test-model-6c0ec0-kserve-589c79bd69-vznsb" event={"ID":"d21cb247-bacd-4af2-a7b1-d04b975f0993","Type":"ContainerStarted","Data":"3d2bdf00c4724290efca7bed06523f1271eafc6169c64acd02828274a3050a25"} Nov 03 01:06:07.182870 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:06:07.182721 4939 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w0-020af9-openshift-default-d77df45d5-mpgn5" Nov 03 01:06:07.184030 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:06:07.184010 4939 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w0-020af9-openshift-default-d77df45d5-mpgn5" Nov 03 01:06:07.203990 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:06:07.203940 4939 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-w4-6c0ec0/auth-test-model-6c0ec0-kserve-589c79bd69-vznsb" podStartSLOduration=2.20392682 podStartE2EDuration="2.20392682s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 01:06:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 01:06:07.201038907 +0000 UTC m=+1515.465373853" watchObservedRunningTime="2026-11-03 01:06:07.20392682 +0000 UTC m=+1515.468261758" Nov 03 01:06:07.323457 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:06:07.323417 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:06:07.442441 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:06:07.442351 4939 patch_prober.go:29] interesting pod/e2e-shared-b-w4-4a697f-openshift-default-666db9f8f5-qn8tj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.175:15021/healthz/ready\": dial tcp 10.132.0.175:15021: connect: connection refused" start-of-body= Nov 03 01:06:07.442728 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:06:07.442694 4939 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w4-4a697f-openshift-default-666db9f8f5-qn8tj" podUID="5b0cabc6-4960-4dc1-8317-3faebf25b30c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.175:15021/healthz/ready\": dial tcp 10.132.0.175:15021: connect: connection refused" Nov 03 01:06:07.475427 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:06:07.475389 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:06:07.627882 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:06:07.627834 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:06:07.915842 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:06:07.915808 4939 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-w4-6c0ec0/auth-test-model-6c0ec0-kserve-589c79bd69-vznsb" Nov 03 01:06:08.442875 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:06:08.442823 4939 patch_prober.go:29] interesting pod/e2e-shared-b-w4-4a697f-openshift-default-666db9f8f5-qn8tj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.175:15021/healthz/ready\": dial tcp 10.132.0.175:15021: connect: connection refused" start-of-body= Nov 03 01:06:08.443400 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:06:08.442894 4939 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w4-4a697f-openshift-default-666db9f8f5-qn8tj" podUID="5b0cabc6-4960-4dc1-8317-3faebf25b30c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.175:15021/healthz/ready\": dial tcp 10.132.0.175:15021: connect: connection refused" Nov 03 01:06:08.515574 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:06:08.515541 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:06:08.779874 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:06:08.779653 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:06:08.934938 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:06:08.930828 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:06:09.442251 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:06:09.442210 4939 patch_prober.go:29] interesting pod/e2e-shared-b-w4-4a697f-openshift-default-666db9f8f5-qn8tj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.175:15021/healthz/ready\": dial tcp 10.132.0.175:15021: connect: connection refused" start-of-body= Nov 03 01:06:09.442464 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:06:09.442289 4939 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w4-4a697f-openshift-default-666db9f8f5-qn8tj" podUID="5b0cabc6-4960-4dc1-8317-3faebf25b30c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.175:15021/healthz/ready\": dial tcp 10.132.0.175:15021: connect: connection refused" Nov 03 01:06:10.171996 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:06:10.171961 4939 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-autores-4248ae/autores-model-4248ae-kserve-5754c66dc7-whjqx"] Nov 03 01:06:10.271932 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:06:10.248236 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:06:10.487374 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:06:10.487283 4939 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w4-4a697f-openshift-default-666db9f8f5-qn8tj" Nov 03 01:06:10.489986 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:06:10.489956 4939 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w4-4a697f-openshift-default-666db9f8f5-qn8tj" Nov 03 01:06:10.583446 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:06:10.583397 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:06:10.734459 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:06:10.734424 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:06:11.600236 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:06:11.600197 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:06:12.418480 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:06:12.418434 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:06:13.036720 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:06:13.036684 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:06:13.755803 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:06:13.755763 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:06:13.915054 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:06:13.915019 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:06:14.610109 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:06:14.610071 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:06:15.174790 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:06:15.174754 4939 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w3-717217-openshift-default-578c69554c-bpgmb"] Nov 03 01:06:15.187674 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:06:15.183153 4939 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w3-717217-openshift-default-578c69554c-bpgmb" Nov 03 01:06:15.204707 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:06:15.200891 4939 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-w3-717217-openshift-default-dockercfg-4kr2v\"" Nov 03 01:06:15.210185 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:06:15.210154 4939 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-717217-openshift-default-578c69554c-bpgmb"] Nov 03 01:06:15.273442 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:06:15.273405 4939 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w1-4af1f7-openshift-default-745ff98565-hbp26"] Nov 03 01:06:15.279370 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:06:15.279342 4939 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w1-4af1f7-openshift-default-745ff98565-hbp26" Nov 03 01:06:15.287366 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:06:15.287335 4939 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w1-4af1f7-openshift-default-745ff98565-hbp26"] Nov 03 01:06:15.291997 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:06:15.291962 4939 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-w1-4af1f7-openshift-default-dockercfg-dmrp9\"" Nov 03 01:06:15.338489 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:06:15.338454 4939 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-4pw7v"] Nov 03 01:06:15.363088 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:06:15.363051 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/26266573-b6f1-495c-bc39-67bd77a5cb79-istio-podinfo\") pod \"e2e-shared-a-w3-717217-openshift-default-578c69554c-bpgmb\" (UID: \"26266573-b6f1-495c-bc39-67bd77a5cb79\") " pod="openshift-ingress/e2e-shared-a-w3-717217-openshift-default-578c69554c-bpgmb" Nov 03 01:06:15.365135 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:06:15.365100 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/26266573-b6f1-495c-bc39-67bd77a5cb79-workload-socket\") pod \"e2e-shared-a-w3-717217-openshift-default-578c69554c-bpgmb\" (UID: \"26266573-b6f1-495c-bc39-67bd77a5cb79\") " pod="openshift-ingress/e2e-shared-a-w3-717217-openshift-default-578c69554c-bpgmb" Nov 03 01:06:15.365625 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:06:15.365601 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/26266573-b6f1-495c-bc39-67bd77a5cb79-istiod-ca-cert\") pod \"e2e-shared-a-w3-717217-openshift-default-578c69554c-bpgmb\" (UID: \"26266573-b6f1-495c-bc39-67bd77a5cb79\") " pod="openshift-ingress/e2e-shared-a-w3-717217-openshift-default-578c69554c-bpgmb" Nov 03 01:06:15.366745 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:06:15.366721 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/26266573-b6f1-495c-bc39-67bd77a5cb79-credential-socket\") pod \"e2e-shared-a-w3-717217-openshift-default-578c69554c-bpgmb\" (UID: \"26266573-b6f1-495c-bc39-67bd77a5cb79\") " pod="openshift-ingress/e2e-shared-a-w3-717217-openshift-default-578c69554c-bpgmb" Nov 03 01:06:15.366866 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:06:15.366788 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/26266573-b6f1-495c-bc39-67bd77a5cb79-workload-certs\") pod \"e2e-shared-a-w3-717217-openshift-default-578c69554c-bpgmb\" (UID: \"26266573-b6f1-495c-bc39-67bd77a5cb79\") " pod="openshift-ingress/e2e-shared-a-w3-717217-openshift-default-578c69554c-bpgmb" Nov 03 01:06:15.366866 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:06:15.366819 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/26266573-b6f1-495c-bc39-67bd77a5cb79-istio-data\") pod \"e2e-shared-a-w3-717217-openshift-default-578c69554c-bpgmb\" (UID: \"26266573-b6f1-495c-bc39-67bd77a5cb79\") " pod="openshift-ingress/e2e-shared-a-w3-717217-openshift-default-578c69554c-bpgmb" Nov 03 01:06:15.367038 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:06:15.366877 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/26266573-b6f1-495c-bc39-67bd77a5cb79-istio-envoy\") pod \"e2e-shared-a-w3-717217-openshift-default-578c69554c-bpgmb\" (UID: \"26266573-b6f1-495c-bc39-67bd77a5cb79\") " pod="openshift-ingress/e2e-shared-a-w3-717217-openshift-default-578c69554c-bpgmb" Nov 03 01:06:15.367038 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:06:15.366980 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/26266573-b6f1-495c-bc39-67bd77a5cb79-istio-token\") pod \"e2e-shared-a-w3-717217-openshift-default-578c69554c-bpgmb\" (UID: \"26266573-b6f1-495c-bc39-67bd77a5cb79\") " pod="openshift-ingress/e2e-shared-a-w3-717217-openshift-default-578c69554c-bpgmb" Nov 03 01:06:15.367143 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:06:15.367050 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-82xm6\" (UniqueName: \"kubernetes.io/projected/26266573-b6f1-495c-bc39-67bd77a5cb79-kube-api-access-82xm6\") pod \"e2e-shared-a-w3-717217-openshift-default-578c69554c-bpgmb\" (UID: \"26266573-b6f1-495c-bc39-67bd77a5cb79\") " pod="openshift-ingress/e2e-shared-a-w3-717217-openshift-default-578c69554c-bpgmb" Nov 03 01:06:15.476014 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:06:15.472033 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3cb597ee-3ac1-4a73-b28b-23d18233ae5a-istio-token\") pod \"e2e-shared-b-w1-4af1f7-openshift-default-745ff98565-hbp26\" (UID: \"3cb597ee-3ac1-4a73-b28b-23d18233ae5a\") " pod="openshift-ingress/e2e-shared-b-w1-4af1f7-openshift-default-745ff98565-hbp26" Nov 03 01:06:15.476014 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:06:15.472105 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3cb597ee-3ac1-4a73-b28b-23d18233ae5a-credential-socket\") pod \"e2e-shared-b-w1-4af1f7-openshift-default-745ff98565-hbp26\" (UID: \"3cb597ee-3ac1-4a73-b28b-23d18233ae5a\") " pod="openshift-ingress/e2e-shared-b-w1-4af1f7-openshift-default-745ff98565-hbp26" Nov 03 01:06:15.476014 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:06:15.472133 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3cb597ee-3ac1-4a73-b28b-23d18233ae5a-istio-envoy\") pod \"e2e-shared-b-w1-4af1f7-openshift-default-745ff98565-hbp26\" (UID: \"3cb597ee-3ac1-4a73-b28b-23d18233ae5a\") " pod="openshift-ingress/e2e-shared-b-w1-4af1f7-openshift-default-745ff98565-hbp26" Nov 03 01:06:15.476014 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:06:15.472206 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/26266573-b6f1-495c-bc39-67bd77a5cb79-istio-podinfo\") pod \"e2e-shared-a-w3-717217-openshift-default-578c69554c-bpgmb\" (UID: \"26266573-b6f1-495c-bc39-67bd77a5cb79\") " pod="openshift-ingress/e2e-shared-a-w3-717217-openshift-default-578c69554c-bpgmb" Nov 03 01:06:15.476014 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:06:15.472253 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/26266573-b6f1-495c-bc39-67bd77a5cb79-workload-socket\") pod \"e2e-shared-a-w3-717217-openshift-default-578c69554c-bpgmb\" (UID: \"26266573-b6f1-495c-bc39-67bd77a5cb79\") " pod="openshift-ingress/e2e-shared-a-w3-717217-openshift-default-578c69554c-bpgmb" Nov 03 01:06:15.476014 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:06:15.472314 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/26266573-b6f1-495c-bc39-67bd77a5cb79-istiod-ca-cert\") pod \"e2e-shared-a-w3-717217-openshift-default-578c69554c-bpgmb\" (UID: \"26266573-b6f1-495c-bc39-67bd77a5cb79\") " pod="openshift-ingress/e2e-shared-a-w3-717217-openshift-default-578c69554c-bpgmb" Nov 03 01:06:15.476014 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:06:15.472346 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6gfqv\" (UniqueName: \"kubernetes.io/projected/3cb597ee-3ac1-4a73-b28b-23d18233ae5a-kube-api-access-6gfqv\") pod \"e2e-shared-b-w1-4af1f7-openshift-default-745ff98565-hbp26\" (UID: \"3cb597ee-3ac1-4a73-b28b-23d18233ae5a\") " pod="openshift-ingress/e2e-shared-b-w1-4af1f7-openshift-default-745ff98565-hbp26" Nov 03 01:06:15.476014 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:06:15.472396 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3cb597ee-3ac1-4a73-b28b-23d18233ae5a-istiod-ca-cert\") pod \"e2e-shared-b-w1-4af1f7-openshift-default-745ff98565-hbp26\" (UID: \"3cb597ee-3ac1-4a73-b28b-23d18233ae5a\") " pod="openshift-ingress/e2e-shared-b-w1-4af1f7-openshift-default-745ff98565-hbp26" Nov 03 01:06:15.476014 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:06:15.472459 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3cb597ee-3ac1-4a73-b28b-23d18233ae5a-istio-data\") pod \"e2e-shared-b-w1-4af1f7-openshift-default-745ff98565-hbp26\" (UID: \"3cb597ee-3ac1-4a73-b28b-23d18233ae5a\") " pod="openshift-ingress/e2e-shared-b-w1-4af1f7-openshift-default-745ff98565-hbp26" Nov 03 01:06:15.476014 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:06:15.472509 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3cb597ee-3ac1-4a73-b28b-23d18233ae5a-workload-socket\") pod \"e2e-shared-b-w1-4af1f7-openshift-default-745ff98565-hbp26\" (UID: \"3cb597ee-3ac1-4a73-b28b-23d18233ae5a\") " pod="openshift-ingress/e2e-shared-b-w1-4af1f7-openshift-default-745ff98565-hbp26" Nov 03 01:06:15.476014 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:06:15.472589 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/26266573-b6f1-495c-bc39-67bd77a5cb79-credential-socket\") pod \"e2e-shared-a-w3-717217-openshift-default-578c69554c-bpgmb\" (UID: \"26266573-b6f1-495c-bc39-67bd77a5cb79\") " pod="openshift-ingress/e2e-shared-a-w3-717217-openshift-default-578c69554c-bpgmb" Nov 03 01:06:15.476014 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:06:15.472656 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/26266573-b6f1-495c-bc39-67bd77a5cb79-workload-certs\") pod \"e2e-shared-a-w3-717217-openshift-default-578c69554c-bpgmb\" (UID: \"26266573-b6f1-495c-bc39-67bd77a5cb79\") " pod="openshift-ingress/e2e-shared-a-w3-717217-openshift-default-578c69554c-bpgmb" Nov 03 01:06:15.476014 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:06:15.472684 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/26266573-b6f1-495c-bc39-67bd77a5cb79-istio-data\") pod \"e2e-shared-a-w3-717217-openshift-default-578c69554c-bpgmb\" (UID: \"26266573-b6f1-495c-bc39-67bd77a5cb79\") " pod="openshift-ingress/e2e-shared-a-w3-717217-openshift-default-578c69554c-bpgmb" Nov 03 01:06:15.476014 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:06:15.472719 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3cb597ee-3ac1-4a73-b28b-23d18233ae5a-istio-podinfo\") pod \"e2e-shared-b-w1-4af1f7-openshift-default-745ff98565-hbp26\" (UID: \"3cb597ee-3ac1-4a73-b28b-23d18233ae5a\") " pod="openshift-ingress/e2e-shared-b-w1-4af1f7-openshift-default-745ff98565-hbp26" Nov 03 01:06:15.476014 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:06:15.472776 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/26266573-b6f1-495c-bc39-67bd77a5cb79-istio-envoy\") pod \"e2e-shared-a-w3-717217-openshift-default-578c69554c-bpgmb\" (UID: \"26266573-b6f1-495c-bc39-67bd77a5cb79\") " pod="openshift-ingress/e2e-shared-a-w3-717217-openshift-default-578c69554c-bpgmb" Nov 03 01:06:15.476014 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:06:15.472878 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/26266573-b6f1-495c-bc39-67bd77a5cb79-istio-token\") pod \"e2e-shared-a-w3-717217-openshift-default-578c69554c-bpgmb\" (UID: \"26266573-b6f1-495c-bc39-67bd77a5cb79\") " pod="openshift-ingress/e2e-shared-a-w3-717217-openshift-default-578c69554c-bpgmb" Nov 03 01:06:15.476014 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:06:15.472976 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-82xm6\" (UniqueName: \"kubernetes.io/projected/26266573-b6f1-495c-bc39-67bd77a5cb79-kube-api-access-82xm6\") pod \"e2e-shared-a-w3-717217-openshift-default-578c69554c-bpgmb\" (UID: \"26266573-b6f1-495c-bc39-67bd77a5cb79\") " pod="openshift-ingress/e2e-shared-a-w3-717217-openshift-default-578c69554c-bpgmb" Nov 03 01:06:15.476014 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:06:15.473098 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3cb597ee-3ac1-4a73-b28b-23d18233ae5a-workload-certs\") pod \"e2e-shared-b-w1-4af1f7-openshift-default-745ff98565-hbp26\" (UID: \"3cb597ee-3ac1-4a73-b28b-23d18233ae5a\") " pod="openshift-ingress/e2e-shared-b-w1-4af1f7-openshift-default-745ff98565-hbp26" Nov 03 01:06:15.590786 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:06:15.514454 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/26266573-b6f1-495c-bc39-67bd77a5cb79-credential-socket\") pod \"e2e-shared-a-w3-717217-openshift-default-578c69554c-bpgmb\" (UID: \"26266573-b6f1-495c-bc39-67bd77a5cb79\") " pod="openshift-ingress/e2e-shared-a-w3-717217-openshift-default-578c69554c-bpgmb" Nov 03 01:06:15.590786 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:06:15.514759 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/26266573-b6f1-495c-bc39-67bd77a5cb79-workload-certs\") pod \"e2e-shared-a-w3-717217-openshift-default-578c69554c-bpgmb\" (UID: \"26266573-b6f1-495c-bc39-67bd77a5cb79\") " pod="openshift-ingress/e2e-shared-a-w3-717217-openshift-default-578c69554c-bpgmb" Nov 03 01:06:15.590786 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:06:15.515038 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/26266573-b6f1-495c-bc39-67bd77a5cb79-istio-data\") pod \"e2e-shared-a-w3-717217-openshift-default-578c69554c-bpgmb\" (UID: \"26266573-b6f1-495c-bc39-67bd77a5cb79\") " pod="openshift-ingress/e2e-shared-a-w3-717217-openshift-default-578c69554c-bpgmb" Nov 03 01:06:15.590786 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:06:15.515692 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/26266573-b6f1-495c-bc39-67bd77a5cb79-istiod-ca-cert\") pod \"e2e-shared-a-w3-717217-openshift-default-578c69554c-bpgmb\" (UID: \"26266573-b6f1-495c-bc39-67bd77a5cb79\") " pod="openshift-ingress/e2e-shared-a-w3-717217-openshift-default-578c69554c-bpgmb" Nov 03 01:06:15.590786 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:06:15.515962 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/26266573-b6f1-495c-bc39-67bd77a5cb79-workload-socket\") pod \"e2e-shared-a-w3-717217-openshift-default-578c69554c-bpgmb\" (UID: \"26266573-b6f1-495c-bc39-67bd77a5cb79\") " pod="openshift-ingress/e2e-shared-a-w3-717217-openshift-default-578c69554c-bpgmb" Nov 03 01:06:15.590786 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:06:15.535705 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/26266573-b6f1-495c-bc39-67bd77a5cb79-istio-podinfo\") pod \"e2e-shared-a-w3-717217-openshift-default-578c69554c-bpgmb\" (UID: \"26266573-b6f1-495c-bc39-67bd77a5cb79\") " pod="openshift-ingress/e2e-shared-a-w3-717217-openshift-default-578c69554c-bpgmb" Nov 03 01:06:15.590786 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:06:15.587003 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/26266573-b6f1-495c-bc39-67bd77a5cb79-istio-token\") pod \"e2e-shared-a-w3-717217-openshift-default-578c69554c-bpgmb\" (UID: \"26266573-b6f1-495c-bc39-67bd77a5cb79\") " pod="openshift-ingress/e2e-shared-a-w3-717217-openshift-default-578c69554c-bpgmb" Nov 03 01:06:15.628937 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:06:15.610082 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3cb597ee-3ac1-4a73-b28b-23d18233ae5a-workload-certs\") pod \"e2e-shared-b-w1-4af1f7-openshift-default-745ff98565-hbp26\" (UID: \"3cb597ee-3ac1-4a73-b28b-23d18233ae5a\") " pod="openshift-ingress/e2e-shared-b-w1-4af1f7-openshift-default-745ff98565-hbp26" Nov 03 01:06:15.628937 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:06:15.610176 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3cb597ee-3ac1-4a73-b28b-23d18233ae5a-istio-token\") pod \"e2e-shared-b-w1-4af1f7-openshift-default-745ff98565-hbp26\" (UID: \"3cb597ee-3ac1-4a73-b28b-23d18233ae5a\") " pod="openshift-ingress/e2e-shared-b-w1-4af1f7-openshift-default-745ff98565-hbp26" Nov 03 01:06:15.628937 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:06:15.610230 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3cb597ee-3ac1-4a73-b28b-23d18233ae5a-credential-socket\") pod \"e2e-shared-b-w1-4af1f7-openshift-default-745ff98565-hbp26\" (UID: \"3cb597ee-3ac1-4a73-b28b-23d18233ae5a\") " pod="openshift-ingress/e2e-shared-b-w1-4af1f7-openshift-default-745ff98565-hbp26" Nov 03 01:06:15.628937 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:06:15.610254 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3cb597ee-3ac1-4a73-b28b-23d18233ae5a-istio-envoy\") pod \"e2e-shared-b-w1-4af1f7-openshift-default-745ff98565-hbp26\" (UID: \"3cb597ee-3ac1-4a73-b28b-23d18233ae5a\") " pod="openshift-ingress/e2e-shared-b-w1-4af1f7-openshift-default-745ff98565-hbp26" Nov 03 01:06:15.628937 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:06:15.610383 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6gfqv\" (UniqueName: \"kubernetes.io/projected/3cb597ee-3ac1-4a73-b28b-23d18233ae5a-kube-api-access-6gfqv\") pod \"e2e-shared-b-w1-4af1f7-openshift-default-745ff98565-hbp26\" (UID: \"3cb597ee-3ac1-4a73-b28b-23d18233ae5a\") " pod="openshift-ingress/e2e-shared-b-w1-4af1f7-openshift-default-745ff98565-hbp26" Nov 03 01:06:15.628937 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:06:15.610431 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3cb597ee-3ac1-4a73-b28b-23d18233ae5a-istiod-ca-cert\") pod \"e2e-shared-b-w1-4af1f7-openshift-default-745ff98565-hbp26\" (UID: \"3cb597ee-3ac1-4a73-b28b-23d18233ae5a\") " pod="openshift-ingress/e2e-shared-b-w1-4af1f7-openshift-default-745ff98565-hbp26" Nov 03 01:06:15.628937 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:06:15.610481 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3cb597ee-3ac1-4a73-b28b-23d18233ae5a-istio-data\") pod \"e2e-shared-b-w1-4af1f7-openshift-default-745ff98565-hbp26\" (UID: \"3cb597ee-3ac1-4a73-b28b-23d18233ae5a\") " pod="openshift-ingress/e2e-shared-b-w1-4af1f7-openshift-default-745ff98565-hbp26" Nov 03 01:06:15.628937 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:06:15.610529 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3cb597ee-3ac1-4a73-b28b-23d18233ae5a-workload-socket\") pod \"e2e-shared-b-w1-4af1f7-openshift-default-745ff98565-hbp26\" (UID: \"3cb597ee-3ac1-4a73-b28b-23d18233ae5a\") " pod="openshift-ingress/e2e-shared-b-w1-4af1f7-openshift-default-745ff98565-hbp26" Nov 03 01:06:15.628937 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:06:15.610713 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3cb597ee-3ac1-4a73-b28b-23d18233ae5a-istio-podinfo\") pod \"e2e-shared-b-w1-4af1f7-openshift-default-745ff98565-hbp26\" (UID: \"3cb597ee-3ac1-4a73-b28b-23d18233ae5a\") " pod="openshift-ingress/e2e-shared-b-w1-4af1f7-openshift-default-745ff98565-hbp26" Nov 03 01:06:15.635956 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:06:15.630680 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3cb597ee-3ac1-4a73-b28b-23d18233ae5a-workload-certs\") pod \"e2e-shared-b-w1-4af1f7-openshift-default-745ff98565-hbp26\" (UID: \"3cb597ee-3ac1-4a73-b28b-23d18233ae5a\") " pod="openshift-ingress/e2e-shared-b-w1-4af1f7-openshift-default-745ff98565-hbp26" Nov 03 01:06:15.635956 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:06:15.631063 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3cb597ee-3ac1-4a73-b28b-23d18233ae5a-credential-socket\") pod \"e2e-shared-b-w1-4af1f7-openshift-default-745ff98565-hbp26\" (UID: \"3cb597ee-3ac1-4a73-b28b-23d18233ae5a\") " pod="openshift-ingress/e2e-shared-b-w1-4af1f7-openshift-default-745ff98565-hbp26" Nov 03 01:06:15.635956 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:06:15.631890 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3cb597ee-3ac1-4a73-b28b-23d18233ae5a-istiod-ca-cert\") pod \"e2e-shared-b-w1-4af1f7-openshift-default-745ff98565-hbp26\" (UID: \"3cb597ee-3ac1-4a73-b28b-23d18233ae5a\") " pod="openshift-ingress/e2e-shared-b-w1-4af1f7-openshift-default-745ff98565-hbp26" Nov 03 01:06:15.635956 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:06:15.634413 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3cb597ee-3ac1-4a73-b28b-23d18233ae5a-istio-data\") pod \"e2e-shared-b-w1-4af1f7-openshift-default-745ff98565-hbp26\" (UID: \"3cb597ee-3ac1-4a73-b28b-23d18233ae5a\") " pod="openshift-ingress/e2e-shared-b-w1-4af1f7-openshift-default-745ff98565-hbp26" Nov 03 01:06:15.635956 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:06:15.634889 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:06:15.635956 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:06:15.635253 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3cb597ee-3ac1-4a73-b28b-23d18233ae5a-workload-socket\") pod \"e2e-shared-b-w1-4af1f7-openshift-default-745ff98565-hbp26\" (UID: \"3cb597ee-3ac1-4a73-b28b-23d18233ae5a\") " pod="openshift-ingress/e2e-shared-b-w1-4af1f7-openshift-default-745ff98565-hbp26" Nov 03 01:06:15.649933 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:06:15.646747 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-82xm6\" (UniqueName: \"kubernetes.io/projected/26266573-b6f1-495c-bc39-67bd77a5cb79-kube-api-access-82xm6\") pod \"e2e-shared-a-w3-717217-openshift-default-578c69554c-bpgmb\" (UID: \"26266573-b6f1-495c-bc39-67bd77a5cb79\") " pod="openshift-ingress/e2e-shared-a-w3-717217-openshift-default-578c69554c-bpgmb" Nov 03 01:06:15.650421 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:06:15.650367 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/26266573-b6f1-495c-bc39-67bd77a5cb79-istio-envoy\") pod \"e2e-shared-a-w3-717217-openshift-default-578c69554c-bpgmb\" (UID: \"26266573-b6f1-495c-bc39-67bd77a5cb79\") " pod="openshift-ingress/e2e-shared-a-w3-717217-openshift-default-578c69554c-bpgmb" Nov 03 01:06:15.656528 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:06:15.656500 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3cb597ee-3ac1-4a73-b28b-23d18233ae5a-istio-token\") pod \"e2e-shared-b-w1-4af1f7-openshift-default-745ff98565-hbp26\" (UID: \"3cb597ee-3ac1-4a73-b28b-23d18233ae5a\") " pod="openshift-ingress/e2e-shared-b-w1-4af1f7-openshift-default-745ff98565-hbp26" Nov 03 01:06:15.662454 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:06:15.658218 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6gfqv\" (UniqueName: \"kubernetes.io/projected/3cb597ee-3ac1-4a73-b28b-23d18233ae5a-kube-api-access-6gfqv\") pod \"e2e-shared-b-w1-4af1f7-openshift-default-745ff98565-hbp26\" (UID: \"3cb597ee-3ac1-4a73-b28b-23d18233ae5a\") " pod="openshift-ingress/e2e-shared-b-w1-4af1f7-openshift-default-745ff98565-hbp26" Nov 03 01:06:15.662454 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:06:15.661679 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3cb597ee-3ac1-4a73-b28b-23d18233ae5a-istio-podinfo\") pod \"e2e-shared-b-w1-4af1f7-openshift-default-745ff98565-hbp26\" (UID: \"3cb597ee-3ac1-4a73-b28b-23d18233ae5a\") " pod="openshift-ingress/e2e-shared-b-w1-4af1f7-openshift-default-745ff98565-hbp26" Nov 03 01:06:15.665430 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:06:15.665373 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3cb597ee-3ac1-4a73-b28b-23d18233ae5a-istio-envoy\") pod \"e2e-shared-b-w1-4af1f7-openshift-default-745ff98565-hbp26\" (UID: \"3cb597ee-3ac1-4a73-b28b-23d18233ae5a\") " pod="openshift-ingress/e2e-shared-b-w1-4af1f7-openshift-default-745ff98565-hbp26" Nov 03 01:06:15.830212 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:06:15.830115 4939 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w3-717217-openshift-default-578c69554c-bpgmb" Nov 03 01:06:15.842515 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:06:15.842461 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:06:15.898354 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:06:15.898325 4939 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w1-4af1f7-openshift-default-745ff98565-hbp26" Nov 03 01:06:15.917956 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:06:15.916023 4939 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-w4-6c0ec0/auth-test-model-6c0ec0-kserve-589c79bd69-vznsb" Nov 03 01:06:15.935312 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:06:15.934839 4939 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-w4-6c0ec0/auth-test-model-6c0ec0-kserve-589c79bd69-vznsb" Nov 03 01:06:16.140810 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:06:16.140776 4939 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-717217-openshift-default-578c69554c-bpgmb"] Nov 03 01:06:16.141200 ip-10-0-140-107 kubenswrapper[4939]: W1103 01:06:16.141177 4939 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod26266573_b6f1_495c_bc39_67bd77a5cb79.slice/crio-aabf348544c06c6cae5a07d4cf857453228ce4a07e201fe13d5feaac04e34c02 WatchSource:0}: Error finding container aabf348544c06c6cae5a07d4cf857453228ce4a07e201fe13d5feaac04e34c02: Status 404 returned error can't find the container with id aabf348544c06c6cae5a07d4cf857453228ce4a07e201fe13d5feaac04e34c02 Nov 03 01:06:16.144568 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:06:16.144518 4939 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 01:06:16.144652 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:06:16.144604 4939 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 01:06:16.144652 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:06:16.144644 4939 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 01:06:16.236636 ip-10-0-140-107 kubenswrapper[4939]: W1103 01:06:16.236608 4939 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3cb597ee_3ac1_4a73_b28b_23d18233ae5a.slice/crio-cd821eb78fb574faa9856f7e3d1d16007a1937aec70baa8c0dc032ec08ee1958 WatchSource:0}: Error finding container cd821eb78fb574faa9856f7e3d1d16007a1937aec70baa8c0dc032ec08ee1958: Status 404 returned error can't find the container with id cd821eb78fb574faa9856f7e3d1d16007a1937aec70baa8c0dc032ec08ee1958 Nov 03 01:06:16.237179 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:06:16.237160 4939 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w1-4af1f7-openshift-default-745ff98565-hbp26"] Nov 03 01:06:16.239521 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:06:16.239482 4939 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 01:06:16.239615 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:06:16.239560 4939 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 01:06:16.239615 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:06:16.239591 4939 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 01:06:16.675850 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:06:16.675785 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:06:16.713479 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:06:16.707637 4939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w1-4af1f7-openshift-default-745ff98565-hbp26" event={"ID":"3cb597ee-3ac1-4a73-b28b-23d18233ae5a","Type":"ContainerStarted","Data":"ae1a7ad10b4adb6b06acf11c99ea11d87478206058f377362d02555f66a45c82"} Nov 03 01:06:16.713479 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:06:16.707710 4939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w1-4af1f7-openshift-default-745ff98565-hbp26" event={"ID":"3cb597ee-3ac1-4a73-b28b-23d18233ae5a","Type":"ContainerStarted","Data":"cd821eb78fb574faa9856f7e3d1d16007a1937aec70baa8c0dc032ec08ee1958"} Nov 03 01:06:16.734934 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:06:16.730389 4939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-717217-openshift-default-578c69554c-bpgmb" event={"ID":"26266573-b6f1-495c-bc39-67bd77a5cb79","Type":"ContainerStarted","Data":"584eabcda720f30f79164c8fd6aa77c704296dd98fa43e2075af7dbbe68ed6da"} Nov 03 01:06:16.734934 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:06:16.730448 4939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-717217-openshift-default-578c69554c-bpgmb" event={"ID":"26266573-b6f1-495c-bc39-67bd77a5cb79","Type":"ContainerStarted","Data":"aabf348544c06c6cae5a07d4cf857453228ce4a07e201fe13d5feaac04e34c02"} Nov 03 01:06:16.800928 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:06:16.799319 4939 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w1-4af1f7-openshift-default-745ff98565-hbp26" podStartSLOduration=1.799300663 podStartE2EDuration="1.799300663s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 01:06:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 01:06:16.739448509 +0000 UTC m=+1525.003783467" watchObservedRunningTime="2026-11-03 01:06:16.799300663 +0000 UTC m=+1525.063635609" Nov 03 01:06:16.819947 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:06:16.819788 4939 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w3-717217-openshift-default-578c69554c-bpgmb" podStartSLOduration=1.8197657 podStartE2EDuration="1.8197657s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 01:06:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 01:06:16.797652784 +0000 UTC m=+1525.061987729" watchObservedRunningTime="2026-11-03 01:06:16.8197657 +0000 UTC m=+1525.084100646" Nov 03 01:06:16.833946 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:06:16.833770 4939 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w3-717217-openshift-default-578c69554c-bpgmb" Nov 03 01:06:16.903947 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:06:16.901975 4939 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w1-4af1f7-openshift-default-745ff98565-hbp26" Nov 03 01:06:17.015984 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:06:17.013862 4939 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-w4-6c0ec0/auth-test-model-6c0ec0-kserve-589c79bd69-vznsb" Nov 03 01:06:17.832828 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:06:17.832792 4939 patch_prober.go:29] interesting pod/e2e-shared-a-w3-717217-openshift-default-578c69554c-bpgmb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.187:15021/healthz/ready\": dial tcp 10.132.0.187:15021: connect: connection refused" start-of-body= Nov 03 01:06:17.833346 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:06:17.832862 4939 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-717217-openshift-default-578c69554c-bpgmb" podUID="26266573-b6f1-495c-bc39-67bd77a5cb79" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.187:15021/healthz/ready\": dial tcp 10.132.0.187:15021: connect: connection refused" Nov 03 01:06:17.837704 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:06:17.837674 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:06:17.909937 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:06:17.906081 4939 patch_prober.go:29] interesting pod/e2e-shared-b-w1-4af1f7-openshift-default-745ff98565-hbp26 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.188:15021/healthz/ready\": dial tcp 10.132.0.188:15021: connect: connection refused" start-of-body= Nov 03 01:06:17.909937 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:06:17.906145 4939 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-4af1f7-openshift-default-745ff98565-hbp26" podUID="3cb597ee-3ac1-4a73-b28b-23d18233ae5a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.188:15021/healthz/ready\": dial tcp 10.132.0.188:15021: connect: connection refused" Nov 03 01:06:18.140010 ip-10-0-140-107 kubenswrapper[4939]: E1103 01:06:18.139288 4939 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-sthjh" containerName="extractor" Nov 03 01:06:18.512980 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:06:18.508995 4939 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w0-50e623-openshift-default-6c86dc794f-bwzqd"] Nov 03 01:06:18.516841 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:06:18.516812 4939 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w0-50e623-openshift-default-6c86dc794f-bwzqd"] Nov 03 01:06:18.517014 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:06:18.516954 4939 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w0-50e623-openshift-default-6c86dc794f-bwzqd" Nov 03 01:06:18.525740 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:06:18.523619 4939 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-w0-50e623-openshift-default-dockercfg-ngqv2\"" Nov 03 01:06:18.594732 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:06:18.594681 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5d516568-4c21-441e-83e4-e3d87132eb96-istiod-ca-cert\") pod \"e2e-shared-a-w0-50e623-openshift-default-6c86dc794f-bwzqd\" (UID: \"5d516568-4c21-441e-83e4-e3d87132eb96\") " pod="openshift-ingress/e2e-shared-a-w0-50e623-openshift-default-6c86dc794f-bwzqd" Nov 03 01:06:18.595282 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:06:18.595258 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5d516568-4c21-441e-83e4-e3d87132eb96-workload-socket\") pod \"e2e-shared-a-w0-50e623-openshift-default-6c86dc794f-bwzqd\" (UID: \"5d516568-4c21-441e-83e4-e3d87132eb96\") " pod="openshift-ingress/e2e-shared-a-w0-50e623-openshift-default-6c86dc794f-bwzqd" Nov 03 01:06:18.595594 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:06:18.595482 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wm588\" (UniqueName: \"kubernetes.io/projected/5d516568-4c21-441e-83e4-e3d87132eb96-kube-api-access-wm588\") pod \"e2e-shared-a-w0-50e623-openshift-default-6c86dc794f-bwzqd\" (UID: \"5d516568-4c21-441e-83e4-e3d87132eb96\") " pod="openshift-ingress/e2e-shared-a-w0-50e623-openshift-default-6c86dc794f-bwzqd" Nov 03 01:06:18.596097 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:06:18.595986 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5d516568-4c21-441e-83e4-e3d87132eb96-istio-podinfo\") pod \"e2e-shared-a-w0-50e623-openshift-default-6c86dc794f-bwzqd\" (UID: \"5d516568-4c21-441e-83e4-e3d87132eb96\") " pod="openshift-ingress/e2e-shared-a-w0-50e623-openshift-default-6c86dc794f-bwzqd" Nov 03 01:06:18.596443 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:06:18.596407 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5d516568-4c21-441e-83e4-e3d87132eb96-istio-envoy\") pod \"e2e-shared-a-w0-50e623-openshift-default-6c86dc794f-bwzqd\" (UID: \"5d516568-4c21-441e-83e4-e3d87132eb96\") " pod="openshift-ingress/e2e-shared-a-w0-50e623-openshift-default-6c86dc794f-bwzqd" Nov 03 01:06:18.596655 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:06:18.596637 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5d516568-4c21-441e-83e4-e3d87132eb96-istio-data\") pod \"e2e-shared-a-w0-50e623-openshift-default-6c86dc794f-bwzqd\" (UID: \"5d516568-4c21-441e-83e4-e3d87132eb96\") " pod="openshift-ingress/e2e-shared-a-w0-50e623-openshift-default-6c86dc794f-bwzqd" Nov 03 01:06:18.596866 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:06:18.596849 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5d516568-4c21-441e-83e4-e3d87132eb96-workload-certs\") pod \"e2e-shared-a-w0-50e623-openshift-default-6c86dc794f-bwzqd\" (UID: \"5d516568-4c21-441e-83e4-e3d87132eb96\") " pod="openshift-ingress/e2e-shared-a-w0-50e623-openshift-default-6c86dc794f-bwzqd" Nov 03 01:06:18.597099 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:06:18.597081 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5d516568-4c21-441e-83e4-e3d87132eb96-credential-socket\") pod \"e2e-shared-a-w0-50e623-openshift-default-6c86dc794f-bwzqd\" (UID: \"5d516568-4c21-441e-83e4-e3d87132eb96\") " pod="openshift-ingress/e2e-shared-a-w0-50e623-openshift-default-6c86dc794f-bwzqd" Nov 03 01:06:18.597422 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:06:18.597402 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5d516568-4c21-441e-83e4-e3d87132eb96-istio-token\") pod \"e2e-shared-a-w0-50e623-openshift-default-6c86dc794f-bwzqd\" (UID: \"5d516568-4c21-441e-83e4-e3d87132eb96\") " pod="openshift-ingress/e2e-shared-a-w0-50e623-openshift-default-6c86dc794f-bwzqd" Nov 03 01:06:18.692870 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:06:18.692715 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:06:18.709947 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:06:18.706216 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wm588\" (UniqueName: \"kubernetes.io/projected/5d516568-4c21-441e-83e4-e3d87132eb96-kube-api-access-wm588\") pod \"e2e-shared-a-w0-50e623-openshift-default-6c86dc794f-bwzqd\" (UID: \"5d516568-4c21-441e-83e4-e3d87132eb96\") " pod="openshift-ingress/e2e-shared-a-w0-50e623-openshift-default-6c86dc794f-bwzqd" Nov 03 01:06:18.709947 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:06:18.706367 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5d516568-4c21-441e-83e4-e3d87132eb96-istio-podinfo\") pod \"e2e-shared-a-w0-50e623-openshift-default-6c86dc794f-bwzqd\" (UID: \"5d516568-4c21-441e-83e4-e3d87132eb96\") " pod="openshift-ingress/e2e-shared-a-w0-50e623-openshift-default-6c86dc794f-bwzqd" Nov 03 01:06:18.709947 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:06:18.706466 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5d516568-4c21-441e-83e4-e3d87132eb96-istio-envoy\") pod \"e2e-shared-a-w0-50e623-openshift-default-6c86dc794f-bwzqd\" (UID: \"5d516568-4c21-441e-83e4-e3d87132eb96\") " pod="openshift-ingress/e2e-shared-a-w0-50e623-openshift-default-6c86dc794f-bwzqd" Nov 03 01:06:18.709947 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:06:18.706503 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5d516568-4c21-441e-83e4-e3d87132eb96-istio-data\") pod \"e2e-shared-a-w0-50e623-openshift-default-6c86dc794f-bwzqd\" (UID: \"5d516568-4c21-441e-83e4-e3d87132eb96\") " pod="openshift-ingress/e2e-shared-a-w0-50e623-openshift-default-6c86dc794f-bwzqd" Nov 03 01:06:18.709947 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:06:18.706780 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5d516568-4c21-441e-83e4-e3d87132eb96-workload-certs\") pod \"e2e-shared-a-w0-50e623-openshift-default-6c86dc794f-bwzqd\" (UID: \"5d516568-4c21-441e-83e4-e3d87132eb96\") " pod="openshift-ingress/e2e-shared-a-w0-50e623-openshift-default-6c86dc794f-bwzqd" Nov 03 01:06:18.709947 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:06:18.706830 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5d516568-4c21-441e-83e4-e3d87132eb96-credential-socket\") pod \"e2e-shared-a-w0-50e623-openshift-default-6c86dc794f-bwzqd\" (UID: \"5d516568-4c21-441e-83e4-e3d87132eb96\") " pod="openshift-ingress/e2e-shared-a-w0-50e623-openshift-default-6c86dc794f-bwzqd" Nov 03 01:06:18.709947 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:06:18.706986 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5d516568-4c21-441e-83e4-e3d87132eb96-istio-token\") pod \"e2e-shared-a-w0-50e623-openshift-default-6c86dc794f-bwzqd\" (UID: \"5d516568-4c21-441e-83e4-e3d87132eb96\") " pod="openshift-ingress/e2e-shared-a-w0-50e623-openshift-default-6c86dc794f-bwzqd" Nov 03 01:06:18.709947 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:06:18.707060 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5d516568-4c21-441e-83e4-e3d87132eb96-istiod-ca-cert\") pod \"e2e-shared-a-w0-50e623-openshift-default-6c86dc794f-bwzqd\" (UID: \"5d516568-4c21-441e-83e4-e3d87132eb96\") " pod="openshift-ingress/e2e-shared-a-w0-50e623-openshift-default-6c86dc794f-bwzqd" Nov 03 01:06:18.709947 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:06:18.707152 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5d516568-4c21-441e-83e4-e3d87132eb96-workload-socket\") pod \"e2e-shared-a-w0-50e623-openshift-default-6c86dc794f-bwzqd\" (UID: \"5d516568-4c21-441e-83e4-e3d87132eb96\") " pod="openshift-ingress/e2e-shared-a-w0-50e623-openshift-default-6c86dc794f-bwzqd" Nov 03 01:06:18.709947 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:06:18.707875 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5d516568-4c21-441e-83e4-e3d87132eb96-workload-socket\") pod \"e2e-shared-a-w0-50e623-openshift-default-6c86dc794f-bwzqd\" (UID: \"5d516568-4c21-441e-83e4-e3d87132eb96\") " pod="openshift-ingress/e2e-shared-a-w0-50e623-openshift-default-6c86dc794f-bwzqd" Nov 03 01:06:18.711183 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:06:18.711151 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5d516568-4c21-441e-83e4-e3d87132eb96-workload-certs\") pod \"e2e-shared-a-w0-50e623-openshift-default-6c86dc794f-bwzqd\" (UID: \"5d516568-4c21-441e-83e4-e3d87132eb96\") " pod="openshift-ingress/e2e-shared-a-w0-50e623-openshift-default-6c86dc794f-bwzqd" Nov 03 01:06:18.737964 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:06:18.737749 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5d516568-4c21-441e-83e4-e3d87132eb96-istio-data\") pod \"e2e-shared-a-w0-50e623-openshift-default-6c86dc794f-bwzqd\" (UID: \"5d516568-4c21-441e-83e4-e3d87132eb96\") " pod="openshift-ingress/e2e-shared-a-w0-50e623-openshift-default-6c86dc794f-bwzqd" Nov 03 01:06:18.738639 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:06:18.738596 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5d516568-4c21-441e-83e4-e3d87132eb96-credential-socket\") pod \"e2e-shared-a-w0-50e623-openshift-default-6c86dc794f-bwzqd\" (UID: \"5d516568-4c21-441e-83e4-e3d87132eb96\") " pod="openshift-ingress/e2e-shared-a-w0-50e623-openshift-default-6c86dc794f-bwzqd" Nov 03 01:06:18.753732 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:06:18.739366 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5d516568-4c21-441e-83e4-e3d87132eb96-istiod-ca-cert\") pod \"e2e-shared-a-w0-50e623-openshift-default-6c86dc794f-bwzqd\" (UID: \"5d516568-4c21-441e-83e4-e3d87132eb96\") " pod="openshift-ingress/e2e-shared-a-w0-50e623-openshift-default-6c86dc794f-bwzqd" Nov 03 01:06:18.830678 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:06:18.801316 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wm588\" (UniqueName: \"kubernetes.io/projected/5d516568-4c21-441e-83e4-e3d87132eb96-kube-api-access-wm588\") pod \"e2e-shared-a-w0-50e623-openshift-default-6c86dc794f-bwzqd\" (UID: \"5d516568-4c21-441e-83e4-e3d87132eb96\") " pod="openshift-ingress/e2e-shared-a-w0-50e623-openshift-default-6c86dc794f-bwzqd" Nov 03 01:06:18.830678 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:06:18.813850 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5d516568-4c21-441e-83e4-e3d87132eb96-istio-token\") pod \"e2e-shared-a-w0-50e623-openshift-default-6c86dc794f-bwzqd\" (UID: \"5d516568-4c21-441e-83e4-e3d87132eb96\") " pod="openshift-ingress/e2e-shared-a-w0-50e623-openshift-default-6c86dc794f-bwzqd" Nov 03 01:06:18.830678 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:06:18.814510 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5d516568-4c21-441e-83e4-e3d87132eb96-istio-envoy\") pod \"e2e-shared-a-w0-50e623-openshift-default-6c86dc794f-bwzqd\" (UID: \"5d516568-4c21-441e-83e4-e3d87132eb96\") " pod="openshift-ingress/e2e-shared-a-w0-50e623-openshift-default-6c86dc794f-bwzqd" Nov 03 01:06:18.831167 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:06:18.831137 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5d516568-4c21-441e-83e4-e3d87132eb96-istio-podinfo\") pod \"e2e-shared-a-w0-50e623-openshift-default-6c86dc794f-bwzqd\" (UID: \"5d516568-4c21-441e-83e4-e3d87132eb96\") " pod="openshift-ingress/e2e-shared-a-w0-50e623-openshift-default-6c86dc794f-bwzqd" Nov 03 01:06:18.831736 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:06:18.831708 4939 patch_prober.go:29] interesting pod/e2e-shared-a-w3-717217-openshift-default-578c69554c-bpgmb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.187:15021/healthz/ready\": dial tcp 10.132.0.187:15021: connect: connection refused" start-of-body= Nov 03 01:06:18.831927 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:06:18.831888 4939 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-717217-openshift-default-578c69554c-bpgmb" podUID="26266573-b6f1-495c-bc39-67bd77a5cb79" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.187:15021/healthz/ready\": dial tcp 10.132.0.187:15021: connect: connection refused" Nov 03 01:06:18.841748 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:06:18.841718 4939 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w0-50e623-openshift-default-6c86dc794f-bwzqd" Nov 03 01:06:18.912943 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:06:18.908192 4939 patch_prober.go:29] interesting pod/e2e-shared-b-w1-4af1f7-openshift-default-745ff98565-hbp26 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.188:15021/healthz/ready\": dial tcp 10.132.0.188:15021: connect: connection refused" start-of-body= Nov 03 01:06:18.912943 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:06:18.908252 4939 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-4af1f7-openshift-default-745ff98565-hbp26" podUID="3cb597ee-3ac1-4a73-b28b-23d18233ae5a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.188:15021/healthz/ready\": dial tcp 10.132.0.188:15021: connect: connection refused" Nov 03 01:06:19.313735 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:06:19.313697 4939 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w0-50e623-openshift-default-6c86dc794f-bwzqd"] Nov 03 01:06:19.324302 ip-10-0-140-107 kubenswrapper[4939]: W1103 01:06:19.324271 4939 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5d516568_4c21_441e_83e4_e3d87132eb96.slice/crio-021d0d7ba09121178b4ee91aaf55a5a3718b2cff7af90ce99e84187ac5b50c3c WatchSource:0}: Error finding container 021d0d7ba09121178b4ee91aaf55a5a3718b2cff7af90ce99e84187ac5b50c3c: Status 404 returned error can't find the container with id 021d0d7ba09121178b4ee91aaf55a5a3718b2cff7af90ce99e84187ac5b50c3c Nov 03 01:06:19.328740 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:06:19.328701 4939 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 01:06:19.328873 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:06:19.328775 4939 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 01:06:19.328873 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:06:19.328804 4939 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 01:06:19.647937 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:06:19.647637 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:06:19.833923 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:06:19.832593 4939 patch_prober.go:29] interesting pod/e2e-shared-a-w3-717217-openshift-default-578c69554c-bpgmb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.187:15021/healthz/ready\": dial tcp 10.132.0.187:15021: connect: connection refused" start-of-body= Nov 03 01:06:19.833923 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:06:19.832644 4939 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-717217-openshift-default-578c69554c-bpgmb" podUID="26266573-b6f1-495c-bc39-67bd77a5cb79" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.187:15021/healthz/ready\": dial tcp 10.132.0.187:15021: connect: connection refused" Nov 03 01:06:19.911991 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:06:19.911394 4939 patch_prober.go:29] interesting pod/e2e-shared-b-w1-4af1f7-openshift-default-745ff98565-hbp26 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.188:15021/healthz/ready\": dial tcp 10.132.0.188:15021: connect: connection refused" start-of-body= Nov 03 01:06:19.911991 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:06:19.911469 4939 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-4af1f7-openshift-default-745ff98565-hbp26" podUID="3cb597ee-3ac1-4a73-b28b-23d18233ae5a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.188:15021/healthz/ready\": dial tcp 10.132.0.188:15021: connect: connection refused" Nov 03 01:06:19.926413 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:06:19.926375 4939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w0-50e623-openshift-default-6c86dc794f-bwzqd" event={"ID":"5d516568-4c21-441e-83e4-e3d87132eb96","Type":"ContainerStarted","Data":"250328b745dd6347bd3994e71869860e5648b6e2981fcaa81c6db4b680c6b239"} Nov 03 01:06:19.926631 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:06:19.926436 4939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w0-50e623-openshift-default-6c86dc794f-bwzqd" event={"ID":"5d516568-4c21-441e-83e4-e3d87132eb96","Type":"ContainerStarted","Data":"021d0d7ba09121178b4ee91aaf55a5a3718b2cff7af90ce99e84187ac5b50c3c"} Nov 03 01:06:19.988933 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:06:19.988624 4939 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w0-50e623-openshift-default-6c86dc794f-bwzqd" podStartSLOduration=1.988604344 podStartE2EDuration="1.988604344s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 01:06:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 01:06:19.957253937 +0000 UTC m=+1528.221588882" watchObservedRunningTime="2026-11-03 01:06:19.988604344 +0000 UTC m=+1528.252939291" Nov 03 01:06:20.829356 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:06:20.829285 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:06:20.831044 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:06:20.830993 4939 patch_prober.go:29] interesting pod/e2e-shared-a-w3-717217-openshift-default-578c69554c-bpgmb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.187:15021/healthz/ready\": dial tcp 10.132.0.187:15021: connect: connection refused" start-of-body= Nov 03 01:06:20.831534 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:06:20.831075 4939 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-717217-openshift-default-578c69554c-bpgmb" podUID="26266573-b6f1-495c-bc39-67bd77a5cb79" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.187:15021/healthz/ready\": dial tcp 10.132.0.187:15021: connect: connection refused" Nov 03 01:06:20.842785 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:06:20.842755 4939 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w0-50e623-openshift-default-6c86dc794f-bwzqd" Nov 03 01:06:20.878139 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:06:20.878104 4939 patch_prober.go:29] interesting pod/e2e-shared-a-w0-50e623-openshift-default-6c86dc794f-bwzqd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.189:15021/healthz/ready\": dial tcp 10.132.0.189:15021: connect: connection refused" start-of-body= Nov 03 01:06:20.878411 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:06:20.878385 4939 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w0-50e623-openshift-default-6c86dc794f-bwzqd" podUID="5d516568-4c21-441e-83e4-e3d87132eb96" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.189:15021/healthz/ready\": dial tcp 10.132.0.189:15021: connect: connection refused" Nov 03 01:06:20.925031 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:06:20.924994 4939 patch_prober.go:29] interesting pod/e2e-shared-b-w1-4af1f7-openshift-default-745ff98565-hbp26 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.188:15021/healthz/ready\": dial tcp 10.132.0.188:15021: connect: connection refused" start-of-body= Nov 03 01:06:20.925584 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:06:20.925056 4939 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-4af1f7-openshift-default-745ff98565-hbp26" podUID="3cb597ee-3ac1-4a73-b28b-23d18233ae5a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.188:15021/healthz/ready\": dial tcp 10.132.0.188:15021: connect: connection refused" Nov 03 01:06:21.645283 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:06:21.645242 4939 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-w1-4af1f7/auth-test-model-4af1f7-kserve-6c5694b445-v7l4q"] Nov 03 01:06:21.742309 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:06:21.742275 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:06:21.858947 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:06:21.846134 4939 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w1-4af1f7/auth-test-model-4af1f7-kserve-6c5694b445-v7l4q"] Nov 03 01:06:21.858947 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:06:21.846609 4939 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w1-4af1f7/auth-test-model-4af1f7-kserve-6c5694b445-v7l4q" Nov 03 01:06:22.153819 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:06:22.153765 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/afd256d4-75f9-4572-8b84-4faad4e5c694-tmp-dir\") pod \"auth-test-model-4af1f7-kserve-6c5694b445-v7l4q\" (UID: \"afd256d4-75f9-4572-8b84-4faad4e5c694\") " pod="ai-tenant-e2e-shared-b-w1-4af1f7/auth-test-model-4af1f7-kserve-6c5694b445-v7l4q" Nov 03 01:06:22.200418 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:06:22.154490 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gtzct\" (UniqueName: \"kubernetes.io/projected/afd256d4-75f9-4572-8b84-4faad4e5c694-kube-api-access-gtzct\") pod \"auth-test-model-4af1f7-kserve-6c5694b445-v7l4q\" (UID: \"afd256d4-75f9-4572-8b84-4faad4e5c694\") " pod="ai-tenant-e2e-shared-b-w1-4af1f7/auth-test-model-4af1f7-kserve-6c5694b445-v7l4q" Nov 03 01:06:22.203434 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:06:22.203400 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/afd256d4-75f9-4572-8b84-4faad4e5c694-dshm\") pod \"auth-test-model-4af1f7-kserve-6c5694b445-v7l4q\" (UID: \"afd256d4-75f9-4572-8b84-4faad4e5c694\") " pod="ai-tenant-e2e-shared-b-w1-4af1f7/auth-test-model-4af1f7-kserve-6c5694b445-v7l4q" Nov 03 01:06:22.203896 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:06:22.203871 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/afd256d4-75f9-4572-8b84-4faad4e5c694-home\") pod \"auth-test-model-4af1f7-kserve-6c5694b445-v7l4q\" (UID: \"afd256d4-75f9-4572-8b84-4faad4e5c694\") " pod="ai-tenant-e2e-shared-b-w1-4af1f7/auth-test-model-4af1f7-kserve-6c5694b445-v7l4q" Nov 03 01:06:22.204248 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:06:22.204231 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/afd256d4-75f9-4572-8b84-4faad4e5c694-model-cache\") pod \"auth-test-model-4af1f7-kserve-6c5694b445-v7l4q\" (UID: \"afd256d4-75f9-4572-8b84-4faad4e5c694\") " pod="ai-tenant-e2e-shared-b-w1-4af1f7/auth-test-model-4af1f7-kserve-6c5694b445-v7l4q" Nov 03 01:06:22.204483 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:06:22.204468 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/afd256d4-75f9-4572-8b84-4faad4e5c694-tls-certs\") pod \"auth-test-model-4af1f7-kserve-6c5694b445-v7l4q\" (UID: \"afd256d4-75f9-4572-8b84-4faad4e5c694\") " pod="ai-tenant-e2e-shared-b-w1-4af1f7/auth-test-model-4af1f7-kserve-6c5694b445-v7l4q" Nov 03 01:06:22.304949 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:06:22.297426 4939 patch_prober.go:29] interesting pod/e2e-shared-a-w3-717217-openshift-default-578c69554c-bpgmb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.187:15021/healthz/ready\": dial tcp 10.132.0.187:15021: connect: connection refused" start-of-body= Nov 03 01:06:22.304949 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:06:22.297493 4939 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-717217-openshift-default-578c69554c-bpgmb" podUID="26266573-b6f1-495c-bc39-67bd77a5cb79" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.187:15021/healthz/ready\": dial tcp 10.132.0.187:15021: connect: connection refused" Nov 03 01:06:22.304949 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:06:22.299087 4939 patch_prober.go:29] interesting pod/e2e-shared-b-w1-4af1f7-openshift-default-745ff98565-hbp26 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.188:15021/healthz/ready\": dial tcp 10.132.0.188:15021: connect: connection refused" start-of-body= Nov 03 01:06:22.304949 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:06:22.299134 4939 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-4af1f7-openshift-default-745ff98565-hbp26" podUID="3cb597ee-3ac1-4a73-b28b-23d18233ae5a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.188:15021/healthz/ready\": dial tcp 10.132.0.188:15021: connect: connection refused" Nov 03 01:06:22.304949 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:06:22.300084 4939 patch_prober.go:29] interesting pod/e2e-shared-a-w0-50e623-openshift-default-6c86dc794f-bwzqd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.189:15021/healthz/ready\": dial tcp 10.132.0.189:15021: connect: connection refused" start-of-body= Nov 03 01:06:22.304949 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:06:22.300141 4939 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w0-50e623-openshift-default-6c86dc794f-bwzqd" podUID="5d516568-4c21-441e-83e4-e3d87132eb96" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.189:15021/healthz/ready\": dial tcp 10.132.0.189:15021: connect: connection refused" Nov 03 01:06:22.330685 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:06:22.309902 4939 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w1-4af1f7\"/\"default-dockercfg-6c6n6\"" Nov 03 01:06:22.330685 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:06:22.310027 4939 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w1-4af1f7\"/\"openshift-service-ca.crt\"" Nov 03 01:06:22.330685 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:06:22.310143 4939 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w1-4af1f7\"/\"kube-root-ca.crt\"" Nov 03 01:06:22.330685 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:06:22.319253 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/afd256d4-75f9-4572-8b84-4faad4e5c694-dshm\") pod \"auth-test-model-4af1f7-kserve-6c5694b445-v7l4q\" (UID: \"afd256d4-75f9-4572-8b84-4faad4e5c694\") " pod="ai-tenant-e2e-shared-b-w1-4af1f7/auth-test-model-4af1f7-kserve-6c5694b445-v7l4q" Nov 03 01:06:22.330685 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:06:22.319406 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/afd256d4-75f9-4572-8b84-4faad4e5c694-home\") pod \"auth-test-model-4af1f7-kserve-6c5694b445-v7l4q\" (UID: \"afd256d4-75f9-4572-8b84-4faad4e5c694\") " pod="ai-tenant-e2e-shared-b-w1-4af1f7/auth-test-model-4af1f7-kserve-6c5694b445-v7l4q" Nov 03 01:06:22.330685 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:06:22.319500 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/afd256d4-75f9-4572-8b84-4faad4e5c694-model-cache\") pod \"auth-test-model-4af1f7-kserve-6c5694b445-v7l4q\" (UID: \"afd256d4-75f9-4572-8b84-4faad4e5c694\") " pod="ai-tenant-e2e-shared-b-w1-4af1f7/auth-test-model-4af1f7-kserve-6c5694b445-v7l4q" Nov 03 01:06:22.330685 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:06:22.319592 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/afd256d4-75f9-4572-8b84-4faad4e5c694-tls-certs\") pod \"auth-test-model-4af1f7-kserve-6c5694b445-v7l4q\" (UID: \"afd256d4-75f9-4572-8b84-4faad4e5c694\") " pod="ai-tenant-e2e-shared-b-w1-4af1f7/auth-test-model-4af1f7-kserve-6c5694b445-v7l4q" Nov 03 01:06:22.330685 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:06:22.319664 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/afd256d4-75f9-4572-8b84-4faad4e5c694-tmp-dir\") pod \"auth-test-model-4af1f7-kserve-6c5694b445-v7l4q\" (UID: \"afd256d4-75f9-4572-8b84-4faad4e5c694\") " pod="ai-tenant-e2e-shared-b-w1-4af1f7/auth-test-model-4af1f7-kserve-6c5694b445-v7l4q" Nov 03 01:06:22.330685 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:06:22.319698 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gtzct\" (UniqueName: \"kubernetes.io/projected/afd256d4-75f9-4572-8b84-4faad4e5c694-kube-api-access-gtzct\") pod \"auth-test-model-4af1f7-kserve-6c5694b445-v7l4q\" (UID: \"afd256d4-75f9-4572-8b84-4faad4e5c694\") " pod="ai-tenant-e2e-shared-b-w1-4af1f7/auth-test-model-4af1f7-kserve-6c5694b445-v7l4q" Nov 03 01:06:22.358094 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:06:22.358054 4939 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w1-4af1f7\"/\"auth-test-model-4af1f7-kserve-self-signed-certs\"" Nov 03 01:06:22.560730 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:06:22.559584 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/afd256d4-75f9-4572-8b84-4faad4e5c694-tmp-dir\") pod \"auth-test-model-4af1f7-kserve-6c5694b445-v7l4q\" (UID: \"afd256d4-75f9-4572-8b84-4faad4e5c694\") " pod="ai-tenant-e2e-shared-b-w1-4af1f7/auth-test-model-4af1f7-kserve-6c5694b445-v7l4q" Nov 03 01:06:22.560730 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:06:22.559944 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/afd256d4-75f9-4572-8b84-4faad4e5c694-home\") pod \"auth-test-model-4af1f7-kserve-6c5694b445-v7l4q\" (UID: \"afd256d4-75f9-4572-8b84-4faad4e5c694\") " pod="ai-tenant-e2e-shared-b-w1-4af1f7/auth-test-model-4af1f7-kserve-6c5694b445-v7l4q" Nov 03 01:06:22.560730 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:06:22.560234 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/afd256d4-75f9-4572-8b84-4faad4e5c694-model-cache\") pod \"auth-test-model-4af1f7-kserve-6c5694b445-v7l4q\" (UID: \"afd256d4-75f9-4572-8b84-4faad4e5c694\") " pod="ai-tenant-e2e-shared-b-w1-4af1f7/auth-test-model-4af1f7-kserve-6c5694b445-v7l4q" Nov 03 01:06:22.586930 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:06:22.583383 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/afd256d4-75f9-4572-8b84-4faad4e5c694-dshm\") pod \"auth-test-model-4af1f7-kserve-6c5694b445-v7l4q\" (UID: \"afd256d4-75f9-4572-8b84-4faad4e5c694\") " pod="ai-tenant-e2e-shared-b-w1-4af1f7/auth-test-model-4af1f7-kserve-6c5694b445-v7l4q" Nov 03 01:06:22.612265 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:06:22.612229 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gtzct\" (UniqueName: \"kubernetes.io/projected/afd256d4-75f9-4572-8b84-4faad4e5c694-kube-api-access-gtzct\") pod \"auth-test-model-4af1f7-kserve-6c5694b445-v7l4q\" (UID: \"afd256d4-75f9-4572-8b84-4faad4e5c694\") " pod="ai-tenant-e2e-shared-b-w1-4af1f7/auth-test-model-4af1f7-kserve-6c5694b445-v7l4q" Nov 03 01:06:22.647392 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:06:22.647350 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:06:22.655195 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:06:22.655161 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/afd256d4-75f9-4572-8b84-4faad4e5c694-tls-certs\") pod \"auth-test-model-4af1f7-kserve-6c5694b445-v7l4q\" (UID: \"afd256d4-75f9-4572-8b84-4faad4e5c694\") " pod="ai-tenant-e2e-shared-b-w1-4af1f7/auth-test-model-4af1f7-kserve-6c5694b445-v7l4q" Nov 03 01:06:22.837948 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:06:22.837884 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:06:22.838787 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:06:22.838739 4939 patch_prober.go:29] interesting pod/e2e-shared-a-w3-717217-openshift-default-578c69554c-bpgmb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.187:15021/healthz/ready\": dial tcp 10.132.0.187:15021: connect: connection refused" start-of-body= Nov 03 01:06:22.838999 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:06:22.838792 4939 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-717217-openshift-default-578c69554c-bpgmb" podUID="26266573-b6f1-495c-bc39-67bd77a5cb79" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.187:15021/healthz/ready\": dial tcp 10.132.0.187:15021: connect: connection refused" Nov 03 01:06:22.843846 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:06:22.843817 4939 patch_prober.go:29] interesting pod/e2e-shared-a-w0-50e623-openshift-default-6c86dc794f-bwzqd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.189:15021/healthz/ready\": dial tcp 10.132.0.189:15021: connect: connection refused" start-of-body= Nov 03 01:06:22.844053 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:06:22.843864 4939 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w0-50e623-openshift-default-6c86dc794f-bwzqd" podUID="5d516568-4c21-441e-83e4-e3d87132eb96" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.189:15021/healthz/ready\": dial tcp 10.132.0.189:15021: connect: connection refused" Nov 03 01:06:22.844154 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:06:22.844133 4939 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w1-4af1f7/auth-test-model-4af1f7-kserve-6c5694b445-v7l4q" Nov 03 01:06:22.913882 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:06:22.913841 4939 patch_prober.go:29] interesting pod/e2e-shared-b-w1-4af1f7-openshift-default-745ff98565-hbp26 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.188:15021/healthz/ready\": dial tcp 10.132.0.188:15021: connect: connection refused" start-of-body= Nov 03 01:06:22.914174 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:06:22.914142 4939 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-4af1f7-openshift-default-745ff98565-hbp26" podUID="3cb597ee-3ac1-4a73-b28b-23d18233ae5a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.188:15021/healthz/ready\": dial tcp 10.132.0.188:15021: connect: connection refused" Nov 03 01:06:23.836569 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:06:23.836258 4939 patch_prober.go:29] interesting pod/e2e-shared-a-w3-717217-openshift-default-578c69554c-bpgmb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.187:15021/healthz/ready\": dial tcp 10.132.0.187:15021: connect: connection refused" start-of-body= Nov 03 01:06:23.836569 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:06:23.836316 4939 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-717217-openshift-default-578c69554c-bpgmb" podUID="26266573-b6f1-495c-bc39-67bd77a5cb79" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.187:15021/healthz/ready\": dial tcp 10.132.0.187:15021: connect: connection refused" Nov 03 01:06:23.845061 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:06:23.845013 4939 patch_prober.go:29] interesting pod/e2e-shared-a-w0-50e623-openshift-default-6c86dc794f-bwzqd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.189:15021/healthz/ready\": dial tcp 10.132.0.189:15021: connect: connection refused" start-of-body= Nov 03 01:06:23.845304 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:06:23.845275 4939 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w0-50e623-openshift-default-6c86dc794f-bwzqd" podUID="5d516568-4c21-441e-83e4-e3d87132eb96" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.189:15021/healthz/ready\": dial tcp 10.132.0.189:15021: connect: connection refused" Nov 03 01:06:23.900166 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:06:23.899794 4939 patch_prober.go:29] interesting pod/e2e-shared-b-w1-4af1f7-openshift-default-745ff98565-hbp26 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.188:15021/healthz/ready\": dial tcp 10.132.0.188:15021: connect: connection refused" start-of-body= Nov 03 01:06:23.900166 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:06:23.899853 4939 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-4af1f7-openshift-default-745ff98565-hbp26" podUID="3cb597ee-3ac1-4a73-b28b-23d18233ae5a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.188:15021/healthz/ready\": dial tcp 10.132.0.188:15021: connect: connection refused" Nov 03 01:06:24.368460 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:06:24.368354 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:06:24.414987 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:06:24.371326 4939 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w1-4af1f7/auth-test-model-4af1f7-kserve-6c5694b445-v7l4q"] Nov 03 01:06:24.835536 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:06:24.835508 4939 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w3-717217-openshift-default-578c69554c-bpgmb" Nov 03 01:06:24.836020 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:06:24.836000 4939 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w3-717217-openshift-default-578c69554c-bpgmb" Nov 03 01:06:24.836847 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:06:24.836819 4939 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w3-717217-openshift-default-578c69554c-bpgmb" Nov 03 01:06:24.842323 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:06:24.842296 4939 patch_prober.go:29] interesting pod/e2e-shared-a-w0-50e623-openshift-default-6c86dc794f-bwzqd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.189:15021/healthz/ready\": dial tcp 10.132.0.189:15021: connect: connection refused" start-of-body= Nov 03 01:06:24.842466 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:06:24.842350 4939 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w0-50e623-openshift-default-6c86dc794f-bwzqd" podUID="5d516568-4c21-441e-83e4-e3d87132eb96" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.189:15021/healthz/ready\": dial tcp 10.132.0.189:15021: connect: connection refused" Nov 03 01:06:24.878731 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:06:24.878695 4939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w1-4af1f7/auth-test-model-4af1f7-kserve-6c5694b445-v7l4q" event={"ID":"afd256d4-75f9-4572-8b84-4faad4e5c694","Type":"ContainerStarted","Data":"87bd5f46fb7486e7bfd4ded6a1da46dc39d7e85cc2aaf663b7a3cc8bf92296c5"} Nov 03 01:06:24.878731 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:06:24.878739 4939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w1-4af1f7/auth-test-model-4af1f7-kserve-6c5694b445-v7l4q" event={"ID":"afd256d4-75f9-4572-8b84-4faad4e5c694","Type":"ContainerStarted","Data":"8380973a55da4fe3b83e3f81f1557a8c57c2fbf7105252d9c9acadc930019efe"} Nov 03 01:06:24.907001 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:06:24.898454 4939 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-w1-4af1f7/auth-test-model-4af1f7-kserve-6c5694b445-v7l4q" podStartSLOduration=3.898436341 podStartE2EDuration="3.898436341s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 01:06:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 01:06:24.896922532 +0000 UTC m=+1533.161257468" watchObservedRunningTime="2026-11-03 01:06:24.898436341 +0000 UTC m=+1533.162771287" Nov 03 01:06:24.907001 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:06:24.904855 4939 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w1-4af1f7-openshift-default-745ff98565-hbp26" Nov 03 01:06:24.907001 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:06:24.905342 4939 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w1-4af1f7-openshift-default-745ff98565-hbp26" Nov 03 01:06:24.907001 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:06:24.906631 4939 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w1-4af1f7-openshift-default-745ff98565-hbp26" Nov 03 01:06:25.487802 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:06:25.487746 4939 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-autores-4248ae/autores-model-4248ae-kserve-5754c66dc7-whjqx" podUID="e7a0e0a1-ab97-4664-b3f6-20896bb6b9ff" containerName="main" containerID="cri-o://a3c997ce891ef9a1d2d834cb36cd3bfca9bb5f76a229e3a3393143dd55460762" gracePeriod=45 Nov 03 01:06:25.637838 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:06:25.637788 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:06:25.852076 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:06:25.852042 4939 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-w1-4af1f7/auth-test-model-4af1f7-kserve-6c5694b445-v7l4q" Nov 03 01:06:25.853594 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:06:25.853517 4939 patch_prober.go:29] interesting pod/e2e-shared-a-w0-50e623-openshift-default-6c86dc794f-bwzqd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.189:15021/healthz/ready\": dial tcp 10.132.0.189:15021: connect: connection refused" start-of-body= Nov 03 01:06:25.853812 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:06:25.853790 4939 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w0-50e623-openshift-default-6c86dc794f-bwzqd" podUID="5d516568-4c21-441e-83e4-e3d87132eb96" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.189:15021/healthz/ready\": dial tcp 10.132.0.189:15021: connect: connection refused" Nov 03 01:06:26.004958 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:06:26.004867 4939 generic.go:356] "Generic (PLEG): container finished" podID="e7a0e0a1-ab97-4664-b3f6-20896bb6b9ff" containerID="a3c997ce891ef9a1d2d834cb36cd3bfca9bb5f76a229e3a3393143dd55460762" exitCode=0 Nov 03 01:06:26.006259 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:06:26.006230 4939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-autores-4248ae/autores-model-4248ae-kserve-5754c66dc7-whjqx" event={"ID":"e7a0e0a1-ab97-4664-b3f6-20896bb6b9ff","Type":"ContainerDied","Data":"a3c997ce891ef9a1d2d834cb36cd3bfca9bb5f76a229e3a3393143dd55460762"} Nov 03 01:06:26.016992 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:06:26.016965 4939 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-autores-4248ae/autores-model-4248ae-kserve-5754c66dc7-whjqx" Nov 03 01:06:26.082619 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:06:26.082589 4939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e7a0e0a1-ab97-4664-b3f6-20896bb6b9ff-kube-api-access-f298d\" (UniqueName: \"kubernetes.io/projected/e7a0e0a1-ab97-4664-b3f6-20896bb6b9ff-kube-api-access-f298d\") pod \"e7a0e0a1-ab97-4664-b3f6-20896bb6b9ff\" (UID: \"e7a0e0a1-ab97-4664-b3f6-20896bb6b9ff\") " Nov 03 01:06:26.082817 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:06:26.082738 4939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e7a0e0a1-ab97-4664-b3f6-20896bb6b9ff-dshm\" (UniqueName: \"kubernetes.io/empty-dir/e7a0e0a1-ab97-4664-b3f6-20896bb6b9ff-dshm\") pod \"e7a0e0a1-ab97-4664-b3f6-20896bb6b9ff\" (UID: \"e7a0e0a1-ab97-4664-b3f6-20896bb6b9ff\") " Nov 03 01:06:26.082817 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:06:26.082783 4939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e7a0e0a1-ab97-4664-b3f6-20896bb6b9ff-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e7a0e0a1-ab97-4664-b3f6-20896bb6b9ff-model-cache\") pod \"e7a0e0a1-ab97-4664-b3f6-20896bb6b9ff\" (UID: \"e7a0e0a1-ab97-4664-b3f6-20896bb6b9ff\") " Nov 03 01:06:26.082953 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:06:26.082871 4939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e7a0e0a1-ab97-4664-b3f6-20896bb6b9ff-tls-certs\" (UniqueName: \"kubernetes.io/secret/e7a0e0a1-ab97-4664-b3f6-20896bb6b9ff-tls-certs\") pod \"e7a0e0a1-ab97-4664-b3f6-20896bb6b9ff\" (UID: \"e7a0e0a1-ab97-4664-b3f6-20896bb6b9ff\") " Nov 03 01:06:26.083063 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:06:26.083036 4939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e7a0e0a1-ab97-4664-b3f6-20896bb6b9ff-home\" (UniqueName: \"kubernetes.io/empty-dir/e7a0e0a1-ab97-4664-b3f6-20896bb6b9ff-home\") pod \"e7a0e0a1-ab97-4664-b3f6-20896bb6b9ff\" (UID: \"e7a0e0a1-ab97-4664-b3f6-20896bb6b9ff\") " Nov 03 01:06:26.084116 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:06:26.084092 4939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e7a0e0a1-ab97-4664-b3f6-20896bb6b9ff-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e7a0e0a1-ab97-4664-b3f6-20896bb6b9ff-tmp-dir\") pod \"e7a0e0a1-ab97-4664-b3f6-20896bb6b9ff\" (UID: \"e7a0e0a1-ab97-4664-b3f6-20896bb6b9ff\") " Nov 03 01:06:26.087446 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:06:26.083406 4939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e7a0e0a1-ab97-4664-b3f6-20896bb6b9ff-home" pod "e7a0e0a1-ab97-4664-b3f6-20896bb6b9ff" (UID: "e7a0e0a1-ab97-4664-b3f6-20896bb6b9ff"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 01:06:26.087593 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:06:26.083882 4939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e7a0e0a1-ab97-4664-b3f6-20896bb6b9ff-model-cache" pod "e7a0e0a1-ab97-4664-b3f6-20896bb6b9ff" (UID: "e7a0e0a1-ab97-4664-b3f6-20896bb6b9ff"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 01:06:26.087697 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:06:26.084448 4939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e7a0e0a1-ab97-4664-b3f6-20896bb6b9ff-tmp-dir" pod "e7a0e0a1-ab97-4664-b3f6-20896bb6b9ff" (UID: "e7a0e0a1-ab97-4664-b3f6-20896bb6b9ff"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 01:06:26.090768 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:06:26.090742 4939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e7a0e0a1-ab97-4664-b3f6-20896bb6b9ff-dshm" pod "e7a0e0a1-ab97-4664-b3f6-20896bb6b9ff" (UID: "e7a0e0a1-ab97-4664-b3f6-20896bb6b9ff"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 01:06:26.092230 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:06:26.092197 4939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e7a0e0a1-ab97-4664-b3f6-20896bb6b9ff-tls-certs" pod "e7a0e0a1-ab97-4664-b3f6-20896bb6b9ff" (UID: "e7a0e0a1-ab97-4664-b3f6-20896bb6b9ff"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 01:06:26.094055 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:06:26.094026 4939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e7a0e0a1-ab97-4664-b3f6-20896bb6b9ff-kube-api-access-f298d" pod "e7a0e0a1-ab97-4664-b3f6-20896bb6b9ff" (UID: "e7a0e0a1-ab97-4664-b3f6-20896bb6b9ff"). InnerVolumeSpecName "kube-api-access-f298d". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 01:06:26.187152 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:06:26.187023 4939 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e7a0e0a1-ab97-4664-b3f6-20896bb6b9ff-home\") on node \"ip-10-0-140-107.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:06:26.187152 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:06:26.187062 4939 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e7a0e0a1-ab97-4664-b3f6-20896bb6b9ff-tmp-dir\") on node \"ip-10-0-140-107.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:06:26.187152 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:06:26.187082 4939 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-f298d\" (UniqueName: \"kubernetes.io/projected/e7a0e0a1-ab97-4664-b3f6-20896bb6b9ff-kube-api-access-f298d\") on node \"ip-10-0-140-107.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:06:26.187152 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:06:26.187097 4939 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e7a0e0a1-ab97-4664-b3f6-20896bb6b9ff-dshm\") on node \"ip-10-0-140-107.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:06:26.187152 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:06:26.187110 4939 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e7a0e0a1-ab97-4664-b3f6-20896bb6b9ff-model-cache\") on node \"ip-10-0-140-107.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:06:26.187152 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:06:26.187123 4939 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e7a0e0a1-ab97-4664-b3f6-20896bb6b9ff-tls-certs\") on node \"ip-10-0-140-107.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:06:26.484999 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:06:26.484363 4939 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-w3-717217/auth-test-model-717217-kserve-5c9cb7dbd9-vl754"] Nov 03 01:06:26.485624 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:06:26.485594 4939 cpu_manager.go:414] "containerMap: removing container" podUID="e7a0e0a1-ab97-4664-b3f6-20896bb6b9ff" containerName="main" Nov 03 01:06:26.485624 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:06:26.485624 4939 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e7a0e0a1-ab97-4664-b3f6-20896bb6b9ff" containerName="main" Nov 03 01:06:26.485829 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:06:26.485813 4939 memory_manager.go:365] "RemoveStaleState removing state" podUID="e7a0e0a1-ab97-4664-b3f6-20896bb6b9ff" containerName="main" Nov 03 01:06:26.491117 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:06:26.491085 4939 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-717217/auth-test-model-717217-kserve-5c9cb7dbd9-vl754" Nov 03 01:06:26.496437 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:06:26.495868 4939 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w3-717217/auth-test-model-717217-kserve-5c9cb7dbd9-vl754"] Nov 03 01:06:26.499497 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:06:26.499466 4939 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w3-717217\"/\"kube-root-ca.crt\"" Nov 03 01:06:26.499620 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:06:26.499609 4939 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w3-717217\"/\"default-dockercfg-4z75p\"" Nov 03 01:06:26.499685 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:06:26.499650 4939 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w3-717217\"/\"openshift-service-ca.crt\"" Nov 03 01:06:26.499837 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:06:26.499810 4939 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w3-717217\"/\"auth-test-model-717217-kserve-self-signed-certs\"" Nov 03 01:06:26.525961 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:06:26.525918 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:06:26.592602 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:06:26.592566 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9634a946-0832-4901-ad01-bbb7a47f4ac7-tmp-dir\") pod \"auth-test-model-717217-kserve-5c9cb7dbd9-vl754\" (UID: \"9634a946-0832-4901-ad01-bbb7a47f4ac7\") " pod="ai-tenant-e2e-shared-a-w3-717217/auth-test-model-717217-kserve-5c9cb7dbd9-vl754" Nov 03 01:06:26.592854 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:06:26.592824 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-47rl4\" (UniqueName: \"kubernetes.io/projected/9634a946-0832-4901-ad01-bbb7a47f4ac7-kube-api-access-47rl4\") pod \"auth-test-model-717217-kserve-5c9cb7dbd9-vl754\" (UID: \"9634a946-0832-4901-ad01-bbb7a47f4ac7\") " pod="ai-tenant-e2e-shared-a-w3-717217/auth-test-model-717217-kserve-5c9cb7dbd9-vl754" Nov 03 01:06:26.593026 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:06:26.592998 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9634a946-0832-4901-ad01-bbb7a47f4ac7-tls-certs\") pod \"auth-test-model-717217-kserve-5c9cb7dbd9-vl754\" (UID: \"9634a946-0832-4901-ad01-bbb7a47f4ac7\") " pod="ai-tenant-e2e-shared-a-w3-717217/auth-test-model-717217-kserve-5c9cb7dbd9-vl754" Nov 03 01:06:26.593108 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:06:26.593091 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9634a946-0832-4901-ad01-bbb7a47f4ac7-dshm\") pod \"auth-test-model-717217-kserve-5c9cb7dbd9-vl754\" (UID: \"9634a946-0832-4901-ad01-bbb7a47f4ac7\") " pod="ai-tenant-e2e-shared-a-w3-717217/auth-test-model-717217-kserve-5c9cb7dbd9-vl754" Nov 03 01:06:26.593173 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:06:26.593119 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9634a946-0832-4901-ad01-bbb7a47f4ac7-model-cache\") pod \"auth-test-model-717217-kserve-5c9cb7dbd9-vl754\" (UID: \"9634a946-0832-4901-ad01-bbb7a47f4ac7\") " pod="ai-tenant-e2e-shared-a-w3-717217/auth-test-model-717217-kserve-5c9cb7dbd9-vl754" Nov 03 01:06:26.593237 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:06:26.593210 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9634a946-0832-4901-ad01-bbb7a47f4ac7-home\") pod \"auth-test-model-717217-kserve-5c9cb7dbd9-vl754\" (UID: \"9634a946-0832-4901-ad01-bbb7a47f4ac7\") " pod="ai-tenant-e2e-shared-a-w3-717217/auth-test-model-717217-kserve-5c9cb7dbd9-vl754" Nov 03 01:06:26.667957 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:06:26.666796 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:06:26.702270 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:06:26.697472 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9634a946-0832-4901-ad01-bbb7a47f4ac7-tls-certs\") pod \"auth-test-model-717217-kserve-5c9cb7dbd9-vl754\" (UID: \"9634a946-0832-4901-ad01-bbb7a47f4ac7\") " pod="ai-tenant-e2e-shared-a-w3-717217/auth-test-model-717217-kserve-5c9cb7dbd9-vl754" Nov 03 01:06:26.702270 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:06:26.697585 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9634a946-0832-4901-ad01-bbb7a47f4ac7-dshm\") pod \"auth-test-model-717217-kserve-5c9cb7dbd9-vl754\" (UID: \"9634a946-0832-4901-ad01-bbb7a47f4ac7\") " pod="ai-tenant-e2e-shared-a-w3-717217/auth-test-model-717217-kserve-5c9cb7dbd9-vl754" Nov 03 01:06:26.702270 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:06:26.697627 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9634a946-0832-4901-ad01-bbb7a47f4ac7-model-cache\") pod \"auth-test-model-717217-kserve-5c9cb7dbd9-vl754\" (UID: \"9634a946-0832-4901-ad01-bbb7a47f4ac7\") " pod="ai-tenant-e2e-shared-a-w3-717217/auth-test-model-717217-kserve-5c9cb7dbd9-vl754" Nov 03 01:06:26.702270 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:06:26.697733 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9634a946-0832-4901-ad01-bbb7a47f4ac7-home\") pod \"auth-test-model-717217-kserve-5c9cb7dbd9-vl754\" (UID: \"9634a946-0832-4901-ad01-bbb7a47f4ac7\") " pod="ai-tenant-e2e-shared-a-w3-717217/auth-test-model-717217-kserve-5c9cb7dbd9-vl754" Nov 03 01:06:26.702270 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:06:26.697770 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9634a946-0832-4901-ad01-bbb7a47f4ac7-tmp-dir\") pod \"auth-test-model-717217-kserve-5c9cb7dbd9-vl754\" (UID: \"9634a946-0832-4901-ad01-bbb7a47f4ac7\") " pod="ai-tenant-e2e-shared-a-w3-717217/auth-test-model-717217-kserve-5c9cb7dbd9-vl754" Nov 03 01:06:26.702270 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:06:26.698188 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-47rl4\" (UniqueName: \"kubernetes.io/projected/9634a946-0832-4901-ad01-bbb7a47f4ac7-kube-api-access-47rl4\") pod \"auth-test-model-717217-kserve-5c9cb7dbd9-vl754\" (UID: \"9634a946-0832-4901-ad01-bbb7a47f4ac7\") " pod="ai-tenant-e2e-shared-a-w3-717217/auth-test-model-717217-kserve-5c9cb7dbd9-vl754" Nov 03 01:06:26.716259 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:06:26.716224 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9634a946-0832-4901-ad01-bbb7a47f4ac7-model-cache\") pod \"auth-test-model-717217-kserve-5c9cb7dbd9-vl754\" (UID: \"9634a946-0832-4901-ad01-bbb7a47f4ac7\") " pod="ai-tenant-e2e-shared-a-w3-717217/auth-test-model-717217-kserve-5c9cb7dbd9-vl754" Nov 03 01:06:26.727942 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:06:26.719698 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9634a946-0832-4901-ad01-bbb7a47f4ac7-tls-certs\") pod \"auth-test-model-717217-kserve-5c9cb7dbd9-vl754\" (UID: \"9634a946-0832-4901-ad01-bbb7a47f4ac7\") " pod="ai-tenant-e2e-shared-a-w3-717217/auth-test-model-717217-kserve-5c9cb7dbd9-vl754" Nov 03 01:06:26.728235 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:06:26.719760 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-47rl4\" (UniqueName: \"kubernetes.io/projected/9634a946-0832-4901-ad01-bbb7a47f4ac7-kube-api-access-47rl4\") pod \"auth-test-model-717217-kserve-5c9cb7dbd9-vl754\" (UID: \"9634a946-0832-4901-ad01-bbb7a47f4ac7\") " pod="ai-tenant-e2e-shared-a-w3-717217/auth-test-model-717217-kserve-5c9cb7dbd9-vl754" Nov 03 01:06:26.728381 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:06:26.720199 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9634a946-0832-4901-ad01-bbb7a47f4ac7-home\") pod \"auth-test-model-717217-kserve-5c9cb7dbd9-vl754\" (UID: \"9634a946-0832-4901-ad01-bbb7a47f4ac7\") " pod="ai-tenant-e2e-shared-a-w3-717217/auth-test-model-717217-kserve-5c9cb7dbd9-vl754" Nov 03 01:06:26.728489 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:06:26.720441 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9634a946-0832-4901-ad01-bbb7a47f4ac7-tmp-dir\") pod \"auth-test-model-717217-kserve-5c9cb7dbd9-vl754\" (UID: \"9634a946-0832-4901-ad01-bbb7a47f4ac7\") " pod="ai-tenant-e2e-shared-a-w3-717217/auth-test-model-717217-kserve-5c9cb7dbd9-vl754" Nov 03 01:06:26.737996 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:06:26.732191 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9634a946-0832-4901-ad01-bbb7a47f4ac7-dshm\") pod \"auth-test-model-717217-kserve-5c9cb7dbd9-vl754\" (UID: \"9634a946-0832-4901-ad01-bbb7a47f4ac7\") " pod="ai-tenant-e2e-shared-a-w3-717217/auth-test-model-717217-kserve-5c9cb7dbd9-vl754" Nov 03 01:06:26.814320 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:06:26.814276 4939 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-717217/auth-test-model-717217-kserve-5c9cb7dbd9-vl754" Nov 03 01:06:26.845186 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:06:26.845154 4939 patch_prober.go:29] interesting pod/e2e-shared-a-w0-50e623-openshift-default-6c86dc794f-bwzqd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.189:15021/healthz/ready\": dial tcp 10.132.0.189:15021: connect: connection refused" start-of-body= Nov 03 01:06:26.846598 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:06:26.846550 4939 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w0-50e623-openshift-default-6c86dc794f-bwzqd" podUID="5d516568-4c21-441e-83e4-e3d87132eb96" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.189:15021/healthz/ready\": dial tcp 10.132.0.189:15021: connect: connection refused" Nov 03 01:06:27.112596 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:06:27.112561 4939 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-autores-4248ae/autores-model-4248ae-kserve-5754c66dc7-whjqx" Nov 03 01:06:27.113134 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:06:27.112955 4939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-autores-4248ae/autores-model-4248ae-kserve-5754c66dc7-whjqx" event={"ID":"e7a0e0a1-ab97-4664-b3f6-20896bb6b9ff","Type":"ContainerDied","Data":"de8b83a975b9ea69b71a7809bb401a8b96dd20b17547b6cefea2ddc82d4cfb94"} Nov 03 01:06:27.113134 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:06:27.113004 4939 scope.go:122] "RemoveContainer" containerID="a3c997ce891ef9a1d2d834cb36cd3bfca9bb5f76a229e3a3393143dd55460762" Nov 03 01:06:27.155093 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:06:27.155057 4939 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-autores-4248ae/autores-model-4248ae-kserve-5754c66dc7-whjqx"] Nov 03 01:06:27.159579 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:06:27.159550 4939 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-autores-4248ae/autores-model-4248ae-kserve-5754c66dc7-whjqx"] Nov 03 01:06:27.366765 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:06:27.366732 4939 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w3-717217/auth-test-model-717217-kserve-5c9cb7dbd9-vl754"] Nov 03 01:06:27.367154 ip-10-0-140-107 kubenswrapper[4939]: W1103 01:06:27.367131 4939 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod9634a946_0832_4901_ad01_bbb7a47f4ac7.slice/crio-1a1011920bfb16a07db4c773ec0d776e31c0bf44ed10948940ae6116641e1eae WatchSource:0}: Error finding container 1a1011920bfb16a07db4c773ec0d776e31c0bf44ed10948940ae6116641e1eae: Status 404 returned error can't find the container with id 1a1011920bfb16a07db4c773ec0d776e31c0bf44ed10948940ae6116641e1eae Nov 03 01:06:27.500763 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:06:27.500726 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:06:27.847776 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:06:27.847743 4939 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w0-50e623-openshift-default-6c86dc794f-bwzqd" Nov 03 01:06:27.848161 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:06:27.848140 4939 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w0-50e623-openshift-default-6c86dc794f-bwzqd" Nov 03 01:06:27.848900 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:06:27.848874 4939 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w0-50e623-openshift-default-6c86dc794f-bwzqd" Nov 03 01:06:27.872765 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:06:27.872733 4939 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e7a0e0a1-ab97-4664-b3f6-20896bb6b9ff" path="/var/lib/kubelet/pods/e7a0e0a1-ab97-4664-b3f6-20896bb6b9ff/volumes" Nov 03 01:06:28.123525 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:06:28.123420 4939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-717217/auth-test-model-717217-kserve-5c9cb7dbd9-vl754" event={"ID":"9634a946-0832-4901-ad01-bbb7a47f4ac7","Type":"ContainerStarted","Data":"755ca59ab9dd6cf520eafa60f6bbd3c635b5b3bcb3c4b16e29111e5b34f4a4e4"} Nov 03 01:06:28.123525 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:06:28.123476 4939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-717217/auth-test-model-717217-kserve-5c9cb7dbd9-vl754" event={"ID":"9634a946-0832-4901-ad01-bbb7a47f4ac7","Type":"ContainerStarted","Data":"1a1011920bfb16a07db4c773ec0d776e31c0bf44ed10948940ae6116641e1eae"} Nov 03 01:06:28.147489 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:06:28.147437 4939 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-w3-717217/auth-test-model-717217-kserve-5c9cb7dbd9-vl754" podStartSLOduration=2.147423897 podStartE2EDuration="2.147423897s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 01:06:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 01:06:28.145230306 +0000 UTC m=+1536.409565251" watchObservedRunningTime="2026-11-03 01:06:28.147423897 +0000 UTC m=+1536.411758841" Nov 03 01:06:28.227730 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:06:28.227687 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:06:28.816657 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:06:28.816621 4939 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-w3-717217/auth-test-model-717217-kserve-5c9cb7dbd9-vl754" Nov 03 01:06:29.160939 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:06:29.151590 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:06:29.889877 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:06:29.889845 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:06:30.114433 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:06:30.114401 4939 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w1-2729f2-openshift-default-9fb97b4fc-dmrf8"] Nov 03 01:06:30.120412 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:06:30.120380 4939 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w1-2729f2-openshift-default-9fb97b4fc-dmrf8" Nov 03 01:06:30.134358 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:06:30.134336 4939 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w1-2729f2-openshift-default-9fb97b4fc-dmrf8"] Nov 03 01:06:30.195861 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:06:30.195771 4939 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w1-2729f2-openshift-default-9fb97b4fc-czgh9"] Nov 03 01:06:30.207195 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:06:30.207166 4939 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w1-2729f2-openshift-default-9fb97b4fc-czgh9" Nov 03 01:06:30.209522 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:06:30.209493 4939 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w1-2729f2-openshift-default-9fb97b4fc-czgh9"] Nov 03 01:06:30.211360 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:06:30.211331 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1240d677-9bc2-427c-a818-ffed78093771-istio-token\") pod \"e2e-shared-a-w1-2729f2-openshift-default-9fb97b4fc-dmrf8\" (UID: \"1240d677-9bc2-427c-a818-ffed78093771\") " pod="openshift-ingress/e2e-shared-a-w1-2729f2-openshift-default-9fb97b4fc-dmrf8" Nov 03 01:06:30.211492 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:06:30.211398 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1240d677-9bc2-427c-a818-ffed78093771-istio-podinfo\") pod \"e2e-shared-a-w1-2729f2-openshift-default-9fb97b4fc-dmrf8\" (UID: \"1240d677-9bc2-427c-a818-ffed78093771\") " pod="openshift-ingress/e2e-shared-a-w1-2729f2-openshift-default-9fb97b4fc-dmrf8" Nov 03 01:06:30.211563 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:06:30.211511 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1240d677-9bc2-427c-a818-ffed78093771-workload-certs\") pod \"e2e-shared-a-w1-2729f2-openshift-default-9fb97b4fc-dmrf8\" (UID: \"1240d677-9bc2-427c-a818-ffed78093771\") " pod="openshift-ingress/e2e-shared-a-w1-2729f2-openshift-default-9fb97b4fc-dmrf8" Nov 03 01:06:30.211627 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:06:30.211563 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1240d677-9bc2-427c-a818-ffed78093771-istio-data\") pod \"e2e-shared-a-w1-2729f2-openshift-default-9fb97b4fc-dmrf8\" (UID: \"1240d677-9bc2-427c-a818-ffed78093771\") " pod="openshift-ingress/e2e-shared-a-w1-2729f2-openshift-default-9fb97b4fc-dmrf8" Nov 03 01:06:30.211844 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:06:30.211810 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1240d677-9bc2-427c-a818-ffed78093771-workload-socket\") pod \"e2e-shared-a-w1-2729f2-openshift-default-9fb97b4fc-dmrf8\" (UID: \"1240d677-9bc2-427c-a818-ffed78093771\") " pod="openshift-ingress/e2e-shared-a-w1-2729f2-openshift-default-9fb97b4fc-dmrf8" Nov 03 01:06:30.212025 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:06:30.212013 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1240d677-9bc2-427c-a818-ffed78093771-istio-envoy\") pod \"e2e-shared-a-w1-2729f2-openshift-default-9fb97b4fc-dmrf8\" (UID: \"1240d677-9bc2-427c-a818-ffed78093771\") " pod="openshift-ingress/e2e-shared-a-w1-2729f2-openshift-default-9fb97b4fc-dmrf8" Nov 03 01:06:30.212648 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:06:30.212065 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1240d677-9bc2-427c-a818-ffed78093771-credential-socket\") pod \"e2e-shared-a-w1-2729f2-openshift-default-9fb97b4fc-dmrf8\" (UID: \"1240d677-9bc2-427c-a818-ffed78093771\") " pod="openshift-ingress/e2e-shared-a-w1-2729f2-openshift-default-9fb97b4fc-dmrf8" Nov 03 01:06:30.212648 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:06:30.212104 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1240d677-9bc2-427c-a818-ffed78093771-istiod-ca-cert\") pod \"e2e-shared-a-w1-2729f2-openshift-default-9fb97b4fc-dmrf8\" (UID: \"1240d677-9bc2-427c-a818-ffed78093771\") " pod="openshift-ingress/e2e-shared-a-w1-2729f2-openshift-default-9fb97b4fc-dmrf8" Nov 03 01:06:30.212648 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:06:30.212141 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bjqz5\" (UniqueName: \"kubernetes.io/projected/1240d677-9bc2-427c-a818-ffed78093771-kube-api-access-bjqz5\") pod \"e2e-shared-a-w1-2729f2-openshift-default-9fb97b4fc-dmrf8\" (UID: \"1240d677-9bc2-427c-a818-ffed78093771\") " pod="openshift-ingress/e2e-shared-a-w1-2729f2-openshift-default-9fb97b4fc-dmrf8" Nov 03 01:06:30.314076 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:06:30.314033 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1240d677-9bc2-427c-a818-ffed78093771-istio-podinfo\") pod \"e2e-shared-a-w1-2729f2-openshift-default-9fb97b4fc-dmrf8\" (UID: \"1240d677-9bc2-427c-a818-ffed78093771\") " pod="openshift-ingress/e2e-shared-a-w1-2729f2-openshift-default-9fb97b4fc-dmrf8" Nov 03 01:06:30.314273 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:06:30.314114 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1240d677-9bc2-427c-a818-ffed78093771-workload-certs\") pod \"e2e-shared-a-w1-2729f2-openshift-default-9fb97b4fc-dmrf8\" (UID: \"1240d677-9bc2-427c-a818-ffed78093771\") " pod="openshift-ingress/e2e-shared-a-w1-2729f2-openshift-default-9fb97b4fc-dmrf8" Nov 03 01:06:30.314273 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:06:30.314146 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9b57f\" (UniqueName: \"kubernetes.io/projected/66c78580-7ad7-49b8-acf7-b23dd33a8af9-kube-api-access-9b57f\") pod \"e2e-shared-a-w1-2729f2-openshift-default-9fb97b4fc-czgh9\" (UID: \"66c78580-7ad7-49b8-acf7-b23dd33a8af9\") " pod="openshift-ingress/e2e-shared-a-w1-2729f2-openshift-default-9fb97b4fc-czgh9" Nov 03 01:06:30.314273 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:06:30.314166 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1240d677-9bc2-427c-a818-ffed78093771-istio-data\") pod \"e2e-shared-a-w1-2729f2-openshift-default-9fb97b4fc-dmrf8\" (UID: \"1240d677-9bc2-427c-a818-ffed78093771\") " pod="openshift-ingress/e2e-shared-a-w1-2729f2-openshift-default-9fb97b4fc-dmrf8" Nov 03 01:06:30.314273 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:06:30.314243 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/66c78580-7ad7-49b8-acf7-b23dd33a8af9-istio-data\") pod \"e2e-shared-a-w1-2729f2-openshift-default-9fb97b4fc-czgh9\" (UID: \"66c78580-7ad7-49b8-acf7-b23dd33a8af9\") " pod="openshift-ingress/e2e-shared-a-w1-2729f2-openshift-default-9fb97b4fc-czgh9" Nov 03 01:06:30.314462 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:06:30.314432 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/66c78580-7ad7-49b8-acf7-b23dd33a8af9-workload-certs\") pod \"e2e-shared-a-w1-2729f2-openshift-default-9fb97b4fc-czgh9\" (UID: \"66c78580-7ad7-49b8-acf7-b23dd33a8af9\") " pod="openshift-ingress/e2e-shared-a-w1-2729f2-openshift-default-9fb97b4fc-czgh9" Nov 03 01:06:30.314521 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:06:30.314489 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/66c78580-7ad7-49b8-acf7-b23dd33a8af9-istio-envoy\") pod \"e2e-shared-a-w1-2729f2-openshift-default-9fb97b4fc-czgh9\" (UID: \"66c78580-7ad7-49b8-acf7-b23dd33a8af9\") " pod="openshift-ingress/e2e-shared-a-w1-2729f2-openshift-default-9fb97b4fc-czgh9" Nov 03 01:06:30.314625 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:06:30.314597 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1240d677-9bc2-427c-a818-ffed78093771-workload-certs\") pod \"e2e-shared-a-w1-2729f2-openshift-default-9fb97b4fc-dmrf8\" (UID: \"1240d677-9bc2-427c-a818-ffed78093771\") " pod="openshift-ingress/e2e-shared-a-w1-2729f2-openshift-default-9fb97b4fc-dmrf8" Nov 03 01:06:30.314739 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:06:30.314620 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1240d677-9bc2-427c-a818-ffed78093771-workload-socket\") pod \"e2e-shared-a-w1-2729f2-openshift-default-9fb97b4fc-dmrf8\" (UID: \"1240d677-9bc2-427c-a818-ffed78093771\") " pod="openshift-ingress/e2e-shared-a-w1-2729f2-openshift-default-9fb97b4fc-dmrf8" Nov 03 01:06:30.314739 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:06:30.314622 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1240d677-9bc2-427c-a818-ffed78093771-istio-data\") pod \"e2e-shared-a-w1-2729f2-openshift-default-9fb97b4fc-dmrf8\" (UID: \"1240d677-9bc2-427c-a818-ffed78093771\") " pod="openshift-ingress/e2e-shared-a-w1-2729f2-openshift-default-9fb97b4fc-dmrf8" Nov 03 01:06:30.314739 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:06:30.314677 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/66c78580-7ad7-49b8-acf7-b23dd33a8af9-istio-token\") pod \"e2e-shared-a-w1-2729f2-openshift-default-9fb97b4fc-czgh9\" (UID: \"66c78580-7ad7-49b8-acf7-b23dd33a8af9\") " pod="openshift-ingress/e2e-shared-a-w1-2729f2-openshift-default-9fb97b4fc-czgh9" Nov 03 01:06:30.314995 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:06:30.314750 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/66c78580-7ad7-49b8-acf7-b23dd33a8af9-istio-podinfo\") pod \"e2e-shared-a-w1-2729f2-openshift-default-9fb97b4fc-czgh9\" (UID: \"66c78580-7ad7-49b8-acf7-b23dd33a8af9\") " pod="openshift-ingress/e2e-shared-a-w1-2729f2-openshift-default-9fb97b4fc-czgh9" Nov 03 01:06:30.314995 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:06:30.314776 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/66c78580-7ad7-49b8-acf7-b23dd33a8af9-workload-socket\") pod \"e2e-shared-a-w1-2729f2-openshift-default-9fb97b4fc-czgh9\" (UID: \"66c78580-7ad7-49b8-acf7-b23dd33a8af9\") " pod="openshift-ingress/e2e-shared-a-w1-2729f2-openshift-default-9fb97b4fc-czgh9" Nov 03 01:06:30.314995 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:06:30.314797 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1240d677-9bc2-427c-a818-ffed78093771-workload-socket\") pod \"e2e-shared-a-w1-2729f2-openshift-default-9fb97b4fc-dmrf8\" (UID: \"1240d677-9bc2-427c-a818-ffed78093771\") " pod="openshift-ingress/e2e-shared-a-w1-2729f2-openshift-default-9fb97b4fc-dmrf8" Nov 03 01:06:30.314995 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:06:30.314806 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/66c78580-7ad7-49b8-acf7-b23dd33a8af9-credential-socket\") pod \"e2e-shared-a-w1-2729f2-openshift-default-9fb97b4fc-czgh9\" (UID: \"66c78580-7ad7-49b8-acf7-b23dd33a8af9\") " pod="openshift-ingress/e2e-shared-a-w1-2729f2-openshift-default-9fb97b4fc-czgh9" Nov 03 01:06:30.314995 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:06:30.314927 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1240d677-9bc2-427c-a818-ffed78093771-istio-envoy\") pod \"e2e-shared-a-w1-2729f2-openshift-default-9fb97b4fc-dmrf8\" (UID: \"1240d677-9bc2-427c-a818-ffed78093771\") " pod="openshift-ingress/e2e-shared-a-w1-2729f2-openshift-default-9fb97b4fc-dmrf8" Nov 03 01:06:30.314995 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:06:30.314991 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1240d677-9bc2-427c-a818-ffed78093771-credential-socket\") pod \"e2e-shared-a-w1-2729f2-openshift-default-9fb97b4fc-dmrf8\" (UID: \"1240d677-9bc2-427c-a818-ffed78093771\") " pod="openshift-ingress/e2e-shared-a-w1-2729f2-openshift-default-9fb97b4fc-dmrf8" Nov 03 01:06:30.315301 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:06:30.315033 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1240d677-9bc2-427c-a818-ffed78093771-istiod-ca-cert\") pod \"e2e-shared-a-w1-2729f2-openshift-default-9fb97b4fc-dmrf8\" (UID: \"1240d677-9bc2-427c-a818-ffed78093771\") " pod="openshift-ingress/e2e-shared-a-w1-2729f2-openshift-default-9fb97b4fc-dmrf8" Nov 03 01:06:30.315301 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:06:30.315062 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/66c78580-7ad7-49b8-acf7-b23dd33a8af9-istiod-ca-cert\") pod \"e2e-shared-a-w1-2729f2-openshift-default-9fb97b4fc-czgh9\" (UID: \"66c78580-7ad7-49b8-acf7-b23dd33a8af9\") " pod="openshift-ingress/e2e-shared-a-w1-2729f2-openshift-default-9fb97b4fc-czgh9" Nov 03 01:06:30.315301 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:06:30.315111 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bjqz5\" (UniqueName: \"kubernetes.io/projected/1240d677-9bc2-427c-a818-ffed78093771-kube-api-access-bjqz5\") pod \"e2e-shared-a-w1-2729f2-openshift-default-9fb97b4fc-dmrf8\" (UID: \"1240d677-9bc2-427c-a818-ffed78093771\") " pod="openshift-ingress/e2e-shared-a-w1-2729f2-openshift-default-9fb97b4fc-dmrf8" Nov 03 01:06:30.315301 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:06:30.315235 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1240d677-9bc2-427c-a818-ffed78093771-istio-token\") pod \"e2e-shared-a-w1-2729f2-openshift-default-9fb97b4fc-dmrf8\" (UID: \"1240d677-9bc2-427c-a818-ffed78093771\") " pod="openshift-ingress/e2e-shared-a-w1-2729f2-openshift-default-9fb97b4fc-dmrf8" Nov 03 01:06:30.315530 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:06:30.315508 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1240d677-9bc2-427c-a818-ffed78093771-credential-socket\") pod \"e2e-shared-a-w1-2729f2-openshift-default-9fb97b4fc-dmrf8\" (UID: \"1240d677-9bc2-427c-a818-ffed78093771\") " pod="openshift-ingress/e2e-shared-a-w1-2729f2-openshift-default-9fb97b4fc-dmrf8" Nov 03 01:06:30.315813 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:06:30.315796 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1240d677-9bc2-427c-a818-ffed78093771-istiod-ca-cert\") pod \"e2e-shared-a-w1-2729f2-openshift-default-9fb97b4fc-dmrf8\" (UID: \"1240d677-9bc2-427c-a818-ffed78093771\") " pod="openshift-ingress/e2e-shared-a-w1-2729f2-openshift-default-9fb97b4fc-dmrf8" Nov 03 01:06:30.316896 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:06:30.316877 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1240d677-9bc2-427c-a818-ffed78093771-istio-podinfo\") pod \"e2e-shared-a-w1-2729f2-openshift-default-9fb97b4fc-dmrf8\" (UID: \"1240d677-9bc2-427c-a818-ffed78093771\") " pod="openshift-ingress/e2e-shared-a-w1-2729f2-openshift-default-9fb97b4fc-dmrf8" Nov 03 01:06:30.317350 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:06:30.317332 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1240d677-9bc2-427c-a818-ffed78093771-istio-envoy\") pod \"e2e-shared-a-w1-2729f2-openshift-default-9fb97b4fc-dmrf8\" (UID: \"1240d677-9bc2-427c-a818-ffed78093771\") " pod="openshift-ingress/e2e-shared-a-w1-2729f2-openshift-default-9fb97b4fc-dmrf8" Nov 03 01:06:30.323622 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:06:30.323597 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1240d677-9bc2-427c-a818-ffed78093771-istio-token\") pod \"e2e-shared-a-w1-2729f2-openshift-default-9fb97b4fc-dmrf8\" (UID: \"1240d677-9bc2-427c-a818-ffed78093771\") " pod="openshift-ingress/e2e-shared-a-w1-2729f2-openshift-default-9fb97b4fc-dmrf8" Nov 03 01:06:30.323798 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:06:30.323783 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bjqz5\" (UniqueName: \"kubernetes.io/projected/1240d677-9bc2-427c-a818-ffed78093771-kube-api-access-bjqz5\") pod \"e2e-shared-a-w1-2729f2-openshift-default-9fb97b4fc-dmrf8\" (UID: \"1240d677-9bc2-427c-a818-ffed78093771\") " pod="openshift-ingress/e2e-shared-a-w1-2729f2-openshift-default-9fb97b4fc-dmrf8" Nov 03 01:06:30.417006 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:06:30.416965 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9b57f\" (UniqueName: \"kubernetes.io/projected/66c78580-7ad7-49b8-acf7-b23dd33a8af9-kube-api-access-9b57f\") pod \"e2e-shared-a-w1-2729f2-openshift-default-9fb97b4fc-czgh9\" (UID: \"66c78580-7ad7-49b8-acf7-b23dd33a8af9\") " pod="openshift-ingress/e2e-shared-a-w1-2729f2-openshift-default-9fb97b4fc-czgh9" Nov 03 01:06:30.417172 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:06:30.417041 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/66c78580-7ad7-49b8-acf7-b23dd33a8af9-istio-data\") pod \"e2e-shared-a-w1-2729f2-openshift-default-9fb97b4fc-czgh9\" (UID: \"66c78580-7ad7-49b8-acf7-b23dd33a8af9\") " pod="openshift-ingress/e2e-shared-a-w1-2729f2-openshift-default-9fb97b4fc-czgh9" Nov 03 01:06:30.417211 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:06:30.417172 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/66c78580-7ad7-49b8-acf7-b23dd33a8af9-workload-certs\") pod \"e2e-shared-a-w1-2729f2-openshift-default-9fb97b4fc-czgh9\" (UID: \"66c78580-7ad7-49b8-acf7-b23dd33a8af9\") " pod="openshift-ingress/e2e-shared-a-w1-2729f2-openshift-default-9fb97b4fc-czgh9" Nov 03 01:06:30.417211 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:06:30.417205 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/66c78580-7ad7-49b8-acf7-b23dd33a8af9-istio-envoy\") pod \"e2e-shared-a-w1-2729f2-openshift-default-9fb97b4fc-czgh9\" (UID: \"66c78580-7ad7-49b8-acf7-b23dd33a8af9\") " pod="openshift-ingress/e2e-shared-a-w1-2729f2-openshift-default-9fb97b4fc-czgh9" Nov 03 01:06:30.417300 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:06:30.417281 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/66c78580-7ad7-49b8-acf7-b23dd33a8af9-istio-token\") pod \"e2e-shared-a-w1-2729f2-openshift-default-9fb97b4fc-czgh9\" (UID: \"66c78580-7ad7-49b8-acf7-b23dd33a8af9\") " pod="openshift-ingress/e2e-shared-a-w1-2729f2-openshift-default-9fb97b4fc-czgh9" Nov 03 01:06:30.417367 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:06:30.417353 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/66c78580-7ad7-49b8-acf7-b23dd33a8af9-istio-podinfo\") pod \"e2e-shared-a-w1-2729f2-openshift-default-9fb97b4fc-czgh9\" (UID: \"66c78580-7ad7-49b8-acf7-b23dd33a8af9\") " pod="openshift-ingress/e2e-shared-a-w1-2729f2-openshift-default-9fb97b4fc-czgh9" Nov 03 01:06:30.417425 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:06:30.417381 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/66c78580-7ad7-49b8-acf7-b23dd33a8af9-workload-socket\") pod \"e2e-shared-a-w1-2729f2-openshift-default-9fb97b4fc-czgh9\" (UID: \"66c78580-7ad7-49b8-acf7-b23dd33a8af9\") " pod="openshift-ingress/e2e-shared-a-w1-2729f2-openshift-default-9fb97b4fc-czgh9" Nov 03 01:06:30.417479 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:06:30.417425 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/66c78580-7ad7-49b8-acf7-b23dd33a8af9-credential-socket\") pod \"e2e-shared-a-w1-2729f2-openshift-default-9fb97b4fc-czgh9\" (UID: \"66c78580-7ad7-49b8-acf7-b23dd33a8af9\") " pod="openshift-ingress/e2e-shared-a-w1-2729f2-openshift-default-9fb97b4fc-czgh9" Nov 03 01:06:30.417539 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:06:30.417492 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/66c78580-7ad7-49b8-acf7-b23dd33a8af9-istiod-ca-cert\") pod \"e2e-shared-a-w1-2729f2-openshift-default-9fb97b4fc-czgh9\" (UID: \"66c78580-7ad7-49b8-acf7-b23dd33a8af9\") " pod="openshift-ingress/e2e-shared-a-w1-2729f2-openshift-default-9fb97b4fc-czgh9" Nov 03 01:06:30.417596 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:06:30.417537 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/66c78580-7ad7-49b8-acf7-b23dd33a8af9-istio-data\") pod \"e2e-shared-a-w1-2729f2-openshift-default-9fb97b4fc-czgh9\" (UID: \"66c78580-7ad7-49b8-acf7-b23dd33a8af9\") " pod="openshift-ingress/e2e-shared-a-w1-2729f2-openshift-default-9fb97b4fc-czgh9" Nov 03 01:06:30.417596 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:06:30.417555 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/66c78580-7ad7-49b8-acf7-b23dd33a8af9-workload-certs\") pod \"e2e-shared-a-w1-2729f2-openshift-default-9fb97b4fc-czgh9\" (UID: \"66c78580-7ad7-49b8-acf7-b23dd33a8af9\") " pod="openshift-ingress/e2e-shared-a-w1-2729f2-openshift-default-9fb97b4fc-czgh9" Nov 03 01:06:30.417802 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:06:30.417771 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/66c78580-7ad7-49b8-acf7-b23dd33a8af9-workload-socket\") pod \"e2e-shared-a-w1-2729f2-openshift-default-9fb97b4fc-czgh9\" (UID: \"66c78580-7ad7-49b8-acf7-b23dd33a8af9\") " pod="openshift-ingress/e2e-shared-a-w1-2729f2-openshift-default-9fb97b4fc-czgh9" Nov 03 01:06:30.418028 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:06:30.418005 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/66c78580-7ad7-49b8-acf7-b23dd33a8af9-credential-socket\") pod \"e2e-shared-a-w1-2729f2-openshift-default-9fb97b4fc-czgh9\" (UID: \"66c78580-7ad7-49b8-acf7-b23dd33a8af9\") " pod="openshift-ingress/e2e-shared-a-w1-2729f2-openshift-default-9fb97b4fc-czgh9" Nov 03 01:06:30.418238 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:06:30.418219 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/66c78580-7ad7-49b8-acf7-b23dd33a8af9-istiod-ca-cert\") pod \"e2e-shared-a-w1-2729f2-openshift-default-9fb97b4fc-czgh9\" (UID: \"66c78580-7ad7-49b8-acf7-b23dd33a8af9\") " pod="openshift-ingress/e2e-shared-a-w1-2729f2-openshift-default-9fb97b4fc-czgh9" Nov 03 01:06:30.419829 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:06:30.419757 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/66c78580-7ad7-49b8-acf7-b23dd33a8af9-istio-envoy\") pod \"e2e-shared-a-w1-2729f2-openshift-default-9fb97b4fc-czgh9\" (UID: \"66c78580-7ad7-49b8-acf7-b23dd33a8af9\") " pod="openshift-ingress/e2e-shared-a-w1-2729f2-openshift-default-9fb97b4fc-czgh9" Nov 03 01:06:30.420376 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:06:30.420353 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/66c78580-7ad7-49b8-acf7-b23dd33a8af9-istio-podinfo\") pod \"e2e-shared-a-w1-2729f2-openshift-default-9fb97b4fc-czgh9\" (UID: \"66c78580-7ad7-49b8-acf7-b23dd33a8af9\") " pod="openshift-ingress/e2e-shared-a-w1-2729f2-openshift-default-9fb97b4fc-czgh9" Nov 03 01:06:30.426793 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:06:30.426773 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/66c78580-7ad7-49b8-acf7-b23dd33a8af9-istio-token\") pod \"e2e-shared-a-w1-2729f2-openshift-default-9fb97b4fc-czgh9\" (UID: \"66c78580-7ad7-49b8-acf7-b23dd33a8af9\") " pod="openshift-ingress/e2e-shared-a-w1-2729f2-openshift-default-9fb97b4fc-czgh9" Nov 03 01:06:30.426869 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:06:30.426810 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9b57f\" (UniqueName: \"kubernetes.io/projected/66c78580-7ad7-49b8-acf7-b23dd33a8af9-kube-api-access-9b57f\") pod \"e2e-shared-a-w1-2729f2-openshift-default-9fb97b4fc-czgh9\" (UID: \"66c78580-7ad7-49b8-acf7-b23dd33a8af9\") " pod="openshift-ingress/e2e-shared-a-w1-2729f2-openshift-default-9fb97b4fc-czgh9" Nov 03 01:06:30.448243 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:06:30.448176 4939 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w1-2729f2-openshift-default-9fb97b4fc-dmrf8" Nov 03 01:06:30.523340 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:06:30.523307 4939 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w1-2729f2-openshift-default-9fb97b4fc-czgh9" Nov 03 01:06:30.742824 ip-10-0-140-107 kubenswrapper[4939]: W1103 01:06:30.742793 4939 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1240d677_9bc2_427c_a818_ffed78093771.slice/crio-03d129f0d6522b8feff3a8fca6acc345bc71be37632894c5c17f27e937b0071d WatchSource:0}: Error finding container 03d129f0d6522b8feff3a8fca6acc345bc71be37632894c5c17f27e937b0071d: Status 404 returned error can't find the container with id 03d129f0d6522b8feff3a8fca6acc345bc71be37632894c5c17f27e937b0071d Nov 03 01:06:30.743036 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:06:30.743013 4939 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w1-2729f2-openshift-default-9fb97b4fc-dmrf8"] Nov 03 01:06:30.745663 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:06:30.745631 4939 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 01:06:30.745737 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:06:30.745696 4939 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 01:06:30.745737 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:06:30.745727 4939 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 01:06:30.769715 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:06:30.769687 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:06:30.913765 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:06:30.913728 4939 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w1-2729f2-openshift-default-9fb97b4fc-czgh9"] Nov 03 01:06:30.917665 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:06:30.917606 4939 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 01:06:30.917762 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:06:30.917715 4939 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 01:06:30.917814 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:06:30.917770 4939 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 01:06:31.152716 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:06:31.152666 4939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w1-2729f2-openshift-default-9fb97b4fc-czgh9" event={"ID":"66c78580-7ad7-49b8-acf7-b23dd33a8af9","Type":"ContainerStarted","Data":"01c6cd08a5a26c9754fd2f2e153053cf9c6cfb98437cb054bcaf8f98f044b406"} Nov 03 01:06:31.152954 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:06:31.152773 4939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w1-2729f2-openshift-default-9fb97b4fc-czgh9" event={"ID":"66c78580-7ad7-49b8-acf7-b23dd33a8af9","Type":"ContainerStarted","Data":"fcc169b213e682547f3a874adcca5bb0f4c4abec0c4480df666cb2f9b116fec1"} Nov 03 01:06:31.155484 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:06:31.155449 4939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w1-2729f2-openshift-default-9fb97b4fc-dmrf8" event={"ID":"1240d677-9bc2-427c-a818-ffed78093771","Type":"ContainerStarted","Data":"faaf9af48ecb5dfb77b38ea46d984d0db4cb5c35af3172caa13a02d92f04ed09"} Nov 03 01:06:31.155633 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:06:31.155511 4939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w1-2729f2-openshift-default-9fb97b4fc-dmrf8" event={"ID":"1240d677-9bc2-427c-a818-ffed78093771","Type":"ContainerStarted","Data":"03d129f0d6522b8feff3a8fca6acc345bc71be37632894c5c17f27e937b0071d"} Nov 03 01:06:31.178739 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:06:31.176387 4939 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w1-2729f2-openshift-default-9fb97b4fc-czgh9" podStartSLOduration=1.176369919 podStartE2EDuration="1.176369919s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 01:06:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 01:06:31.171246559 +0000 UTC m=+1539.435581505" watchObservedRunningTime="2026-11-03 01:06:31.176369919 +0000 UTC m=+1539.440704865" Nov 03 01:06:31.194484 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:06:31.194431 4939 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w1-2729f2-openshift-default-9fb97b4fc-dmrf8" podStartSLOduration=1.19441819 podStartE2EDuration="1.19441819s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 01:06:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 01:06:31.189858847 +0000 UTC m=+1539.454193827" watchObservedRunningTime="2026-11-03 01:06:31.19441819 +0000 UTC m=+1539.458753139" Nov 03 01:06:31.450983 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:06:31.450806 4939 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w1-2729f2-openshift-default-9fb97b4fc-dmrf8" Nov 03 01:06:31.450983 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:06:31.450948 4939 patch_prober.go:29] interesting pod/e2e-shared-a-w1-2729f2-openshift-default-9fb97b4fc-dmrf8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.192:15021/healthz/ready\": dial tcp 10.132.0.192:15021: connect: connection refused" start-of-body= Nov 03 01:06:31.451593 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:06:31.450991 4939 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w1-2729f2-openshift-default-9fb97b4fc-dmrf8" podUID="1240d677-9bc2-427c-a818-ffed78093771" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.192:15021/healthz/ready\": dial tcp 10.132.0.192:15021: connect: connection refused" Nov 03 01:06:31.524207 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:06:31.524165 4939 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w1-2729f2-openshift-default-9fb97b4fc-czgh9" Nov 03 01:06:32.033418 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:06:32.033375 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:06:32.467937 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:06:32.450331 4939 patch_prober.go:29] interesting pod/e2e-shared-a-w1-2729f2-openshift-default-9fb97b4fc-dmrf8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.192:15021/healthz/ready\": dial tcp 10.132.0.192:15021: connect: connection refused" start-of-body= Nov 03 01:06:32.467937 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:06:32.450399 4939 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w1-2729f2-openshift-default-9fb97b4fc-dmrf8" podUID="1240d677-9bc2-427c-a818-ffed78093771" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.192:15021/healthz/ready\": dial tcp 10.132.0.192:15021: connect: connection refused" Nov 03 01:06:32.525130 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:06:32.525093 4939 patch_prober.go:29] interesting pod/e2e-shared-a-w1-2729f2-openshift-default-9fb97b4fc-czgh9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.193:15021/healthz/ready\": dial tcp 10.132.0.193:15021: connect: connection refused" start-of-body= Nov 03 01:06:32.525307 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:06:32.525147 4939 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w1-2729f2-openshift-default-9fb97b4fc-czgh9" podUID="66c78580-7ad7-49b8-acf7-b23dd33a8af9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.193:15021/healthz/ready\": dial tcp 10.132.0.193:15021: connect: connection refused" Nov 03 01:06:32.800493 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:06:32.800404 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:06:32.855936 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:06:32.855608 4939 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-w1-4af1f7/auth-test-model-4af1f7-kserve-6c5694b445-v7l4q" Nov 03 01:06:32.943071 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:06:32.943042 4939 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-w1-4af1f7/auth-test-model-4af1f7-kserve-6c5694b445-v7l4q" Nov 03 01:06:33.237330 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:06:33.237304 4939 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-w1-4af1f7/auth-test-model-4af1f7-kserve-6c5694b445-v7l4q" Nov 03 01:06:33.451936 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:06:33.450355 4939 patch_prober.go:29] interesting pod/e2e-shared-a-w1-2729f2-openshift-default-9fb97b4fc-dmrf8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.192:15021/healthz/ready\": dial tcp 10.132.0.192:15021: connect: connection refused" start-of-body= Nov 03 01:06:33.451936 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:06:33.450512 4939 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w1-2729f2-openshift-default-9fb97b4fc-dmrf8" podUID="1240d677-9bc2-427c-a818-ffed78093771" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.192:15021/healthz/ready\": dial tcp 10.132.0.192:15021: connect: connection refused" Nov 03 01:06:33.524448 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:06:33.524357 4939 patch_prober.go:29] interesting pod/e2e-shared-a-w1-2729f2-openshift-default-9fb97b4fc-czgh9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.193:15021/healthz/ready\": dial tcp 10.132.0.193:15021: connect: connection refused" start-of-body= Nov 03 01:06:33.524448 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:06:33.524420 4939 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w1-2729f2-openshift-default-9fb97b4fc-czgh9" podUID="66c78580-7ad7-49b8-acf7-b23dd33a8af9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.193:15021/healthz/ready\": dial tcp 10.132.0.193:15021: connect: connection refused" Nov 03 01:06:33.529820 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:06:33.529792 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:06:34.429861 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:06:34.429715 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:06:34.465989 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:06:34.462305 4939 patch_prober.go:29] interesting pod/e2e-shared-a-w1-2729f2-openshift-default-9fb97b4fc-dmrf8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.192:15021/healthz/ready\": dial tcp 10.132.0.192:15021: connect: connection refused" start-of-body= Nov 03 01:06:34.465989 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:06:34.462373 4939 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w1-2729f2-openshift-default-9fb97b4fc-dmrf8" podUID="1240d677-9bc2-427c-a818-ffed78093771" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.192:15021/healthz/ready\": dial tcp 10.132.0.192:15021: connect: connection refused" Nov 03 01:06:34.530932 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:06:34.530379 4939 patch_prober.go:29] interesting pod/e2e-shared-a-w1-2729f2-openshift-default-9fb97b4fc-czgh9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.193:15021/healthz/ready\": dial tcp 10.132.0.193:15021: connect: connection refused" start-of-body= Nov 03 01:06:34.530932 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:06:34.530445 4939 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w1-2729f2-openshift-default-9fb97b4fc-czgh9" podUID="66c78580-7ad7-49b8-acf7-b23dd33a8af9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.193:15021/healthz/ready\": dial tcp 10.132.0.193:15021: connect: connection refused" Nov 03 01:06:35.280012 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:06:35.279974 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:06:35.462705 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:06:35.462666 4939 patch_prober.go:29] interesting pod/e2e-shared-a-w1-2729f2-openshift-default-9fb97b4fc-dmrf8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.192:15021/healthz/ready\": dial tcp 10.132.0.192:15021: connect: connection refused" start-of-body= Nov 03 01:06:35.463032 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:06:35.463000 4939 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w1-2729f2-openshift-default-9fb97b4fc-dmrf8" podUID="1240d677-9bc2-427c-a818-ffed78093771" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.192:15021/healthz/ready\": dial tcp 10.132.0.192:15021: connect: connection refused" Nov 03 01:06:35.524181 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:06:35.524152 4939 patch_prober.go:29] interesting pod/e2e-shared-a-w1-2729f2-openshift-default-9fb97b4fc-czgh9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.193:15021/healthz/ready\": dial tcp 10.132.0.193:15021: connect: connection refused" start-of-body= Nov 03 01:06:35.524442 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:06:35.524418 4939 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w1-2729f2-openshift-default-9fb97b4fc-czgh9" podUID="66c78580-7ad7-49b8-acf7-b23dd33a8af9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.193:15021/healthz/ready\": dial tcp 10.132.0.193:15021: connect: connection refused" Nov 03 01:06:36.199004 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:06:36.198963 4939 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-w4-4a697f/auth-test-model-4a697f-kserve-84b7466bd5-48hbr"] Nov 03 01:06:36.218388 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:06:36.218231 4939 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w4-4a697f/auth-test-model-4a697f-kserve-84b7466bd5-48hbr"] Nov 03 01:06:36.218388 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:06:36.218357 4939 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w4-4a697f/auth-test-model-4a697f-kserve-84b7466bd5-48hbr" Nov 03 01:06:36.224371 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:06:36.224342 4939 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w4-4a697f\"/\"default-dockercfg-q8vgn\"" Nov 03 01:06:36.224771 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:06:36.224404 4939 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w4-4a697f\"/\"auth-test-model-4a697f-kserve-self-signed-certs\"" Nov 03 01:06:36.224967 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:06:36.224724 4939 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w4-4a697f\"/\"openshift-service-ca.crt\"" Nov 03 01:06:36.225757 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:06:36.225734 4939 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w4-4a697f\"/\"kube-root-ca.crt\"" Nov 03 01:06:36.243266 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:06:36.243238 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:06:36.243657 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:06:36.243635 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4zxrl\" (UniqueName: \"kubernetes.io/projected/18846667-85a0-4bc2-8995-61bfb2ab1c5d-kube-api-access-4zxrl\") pod \"auth-test-model-4a697f-kserve-84b7466bd5-48hbr\" (UID: \"18846667-85a0-4bc2-8995-61bfb2ab1c5d\") " pod="ai-tenant-e2e-shared-b-w4-4a697f/auth-test-model-4a697f-kserve-84b7466bd5-48hbr" Nov 03 01:06:36.243777 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:06:36.243759 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/18846667-85a0-4bc2-8995-61bfb2ab1c5d-home\") pod \"auth-test-model-4a697f-kserve-84b7466bd5-48hbr\" (UID: \"18846667-85a0-4bc2-8995-61bfb2ab1c5d\") " pod="ai-tenant-e2e-shared-b-w4-4a697f/auth-test-model-4a697f-kserve-84b7466bd5-48hbr" Nov 03 01:06:36.243845 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:06:36.243815 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/18846667-85a0-4bc2-8995-61bfb2ab1c5d-dshm\") pod \"auth-test-model-4a697f-kserve-84b7466bd5-48hbr\" (UID: \"18846667-85a0-4bc2-8995-61bfb2ab1c5d\") " pod="ai-tenant-e2e-shared-b-w4-4a697f/auth-test-model-4a697f-kserve-84b7466bd5-48hbr" Nov 03 01:06:36.243997 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:06:36.243942 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/18846667-85a0-4bc2-8995-61bfb2ab1c5d-tmp-dir\") pod \"auth-test-model-4a697f-kserve-84b7466bd5-48hbr\" (UID: \"18846667-85a0-4bc2-8995-61bfb2ab1c5d\") " pod="ai-tenant-e2e-shared-b-w4-4a697f/auth-test-model-4a697f-kserve-84b7466bd5-48hbr" Nov 03 01:06:36.244084 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:06:36.244044 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/18846667-85a0-4bc2-8995-61bfb2ab1c5d-tls-certs\") pod \"auth-test-model-4a697f-kserve-84b7466bd5-48hbr\" (UID: \"18846667-85a0-4bc2-8995-61bfb2ab1c5d\") " pod="ai-tenant-e2e-shared-b-w4-4a697f/auth-test-model-4a697f-kserve-84b7466bd5-48hbr" Nov 03 01:06:36.244184 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:06:36.244165 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/18846667-85a0-4bc2-8995-61bfb2ab1c5d-model-cache\") pod \"auth-test-model-4a697f-kserve-84b7466bd5-48hbr\" (UID: \"18846667-85a0-4bc2-8995-61bfb2ab1c5d\") " pod="ai-tenant-e2e-shared-b-w4-4a697f/auth-test-model-4a697f-kserve-84b7466bd5-48hbr" Nov 03 01:06:36.345176 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:06:36.345140 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4zxrl\" (UniqueName: \"kubernetes.io/projected/18846667-85a0-4bc2-8995-61bfb2ab1c5d-kube-api-access-4zxrl\") pod \"auth-test-model-4a697f-kserve-84b7466bd5-48hbr\" (UID: \"18846667-85a0-4bc2-8995-61bfb2ab1c5d\") " pod="ai-tenant-e2e-shared-b-w4-4a697f/auth-test-model-4a697f-kserve-84b7466bd5-48hbr" Nov 03 01:06:36.345335 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:06:36.345269 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/18846667-85a0-4bc2-8995-61bfb2ab1c5d-home\") pod \"auth-test-model-4a697f-kserve-84b7466bd5-48hbr\" (UID: \"18846667-85a0-4bc2-8995-61bfb2ab1c5d\") " pod="ai-tenant-e2e-shared-b-w4-4a697f/auth-test-model-4a697f-kserve-84b7466bd5-48hbr" Nov 03 01:06:36.345335 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:06:36.345323 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/18846667-85a0-4bc2-8995-61bfb2ab1c5d-dshm\") pod \"auth-test-model-4a697f-kserve-84b7466bd5-48hbr\" (UID: \"18846667-85a0-4bc2-8995-61bfb2ab1c5d\") " pod="ai-tenant-e2e-shared-b-w4-4a697f/auth-test-model-4a697f-kserve-84b7466bd5-48hbr" Nov 03 01:06:36.345434 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:06:36.345361 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/18846667-85a0-4bc2-8995-61bfb2ab1c5d-tmp-dir\") pod \"auth-test-model-4a697f-kserve-84b7466bd5-48hbr\" (UID: \"18846667-85a0-4bc2-8995-61bfb2ab1c5d\") " pod="ai-tenant-e2e-shared-b-w4-4a697f/auth-test-model-4a697f-kserve-84b7466bd5-48hbr" Nov 03 01:06:36.345434 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:06:36.345400 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/18846667-85a0-4bc2-8995-61bfb2ab1c5d-tls-certs\") pod \"auth-test-model-4a697f-kserve-84b7466bd5-48hbr\" (UID: \"18846667-85a0-4bc2-8995-61bfb2ab1c5d\") " pod="ai-tenant-e2e-shared-b-w4-4a697f/auth-test-model-4a697f-kserve-84b7466bd5-48hbr" Nov 03 01:06:36.345506 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:06:36.345476 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/18846667-85a0-4bc2-8995-61bfb2ab1c5d-model-cache\") pod \"auth-test-model-4a697f-kserve-84b7466bd5-48hbr\" (UID: \"18846667-85a0-4bc2-8995-61bfb2ab1c5d\") " pod="ai-tenant-e2e-shared-b-w4-4a697f/auth-test-model-4a697f-kserve-84b7466bd5-48hbr" Nov 03 01:06:36.345811 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:06:36.345783 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/18846667-85a0-4bc2-8995-61bfb2ab1c5d-home\") pod \"auth-test-model-4a697f-kserve-84b7466bd5-48hbr\" (UID: \"18846667-85a0-4bc2-8995-61bfb2ab1c5d\") " pod="ai-tenant-e2e-shared-b-w4-4a697f/auth-test-model-4a697f-kserve-84b7466bd5-48hbr" Nov 03 01:06:36.345871 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:06:36.345849 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/18846667-85a0-4bc2-8995-61bfb2ab1c5d-tmp-dir\") pod \"auth-test-model-4a697f-kserve-84b7466bd5-48hbr\" (UID: \"18846667-85a0-4bc2-8995-61bfb2ab1c5d\") " pod="ai-tenant-e2e-shared-b-w4-4a697f/auth-test-model-4a697f-kserve-84b7466bd5-48hbr" Nov 03 01:06:36.414933 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:06:36.398611 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/18846667-85a0-4bc2-8995-61bfb2ab1c5d-model-cache\") pod \"auth-test-model-4a697f-kserve-84b7466bd5-48hbr\" (UID: \"18846667-85a0-4bc2-8995-61bfb2ab1c5d\") " pod="ai-tenant-e2e-shared-b-w4-4a697f/auth-test-model-4a697f-kserve-84b7466bd5-48hbr" Nov 03 01:06:36.417133 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:06:36.417101 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:06:36.450596 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:06:36.450504 4939 patch_prober.go:29] interesting pod/e2e-shared-a-w1-2729f2-openshift-default-9fb97b4fc-dmrf8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.192:15021/healthz/ready\": dial tcp 10.132.0.192:15021: connect: connection refused" start-of-body= Nov 03 01:06:36.450951 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:06:36.450864 4939 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w1-2729f2-openshift-default-9fb97b4fc-dmrf8" podUID="1240d677-9bc2-427c-a818-ffed78093771" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.192:15021/healthz/ready\": dial tcp 10.132.0.192:15021: connect: connection refused" Nov 03 01:06:36.490222 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:06:36.490134 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4zxrl\" (UniqueName: \"kubernetes.io/projected/18846667-85a0-4bc2-8995-61bfb2ab1c5d-kube-api-access-4zxrl\") pod \"auth-test-model-4a697f-kserve-84b7466bd5-48hbr\" (UID: \"18846667-85a0-4bc2-8995-61bfb2ab1c5d\") " pod="ai-tenant-e2e-shared-b-w4-4a697f/auth-test-model-4a697f-kserve-84b7466bd5-48hbr" Nov 03 01:06:36.490686 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:06:36.490663 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/18846667-85a0-4bc2-8995-61bfb2ab1c5d-dshm\") pod \"auth-test-model-4a697f-kserve-84b7466bd5-48hbr\" (UID: \"18846667-85a0-4bc2-8995-61bfb2ab1c5d\") " pod="ai-tenant-e2e-shared-b-w4-4a697f/auth-test-model-4a697f-kserve-84b7466bd5-48hbr" Nov 03 01:06:36.506930 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:06:36.501750 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/18846667-85a0-4bc2-8995-61bfb2ab1c5d-tls-certs\") pod \"auth-test-model-4a697f-kserve-84b7466bd5-48hbr\" (UID: \"18846667-85a0-4bc2-8995-61bfb2ab1c5d\") " pod="ai-tenant-e2e-shared-b-w4-4a697f/auth-test-model-4a697f-kserve-84b7466bd5-48hbr" Nov 03 01:06:36.531315 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:06:36.531276 4939 patch_prober.go:29] interesting pod/e2e-shared-a-w1-2729f2-openshift-default-9fb97b4fc-czgh9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.193:15021/healthz/ready\": dial tcp 10.132.0.193:15021: connect: connection refused" start-of-body= Nov 03 01:06:36.531569 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:06:36.531543 4939 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w1-2729f2-openshift-default-9fb97b4fc-czgh9" podUID="66c78580-7ad7-49b8-acf7-b23dd33a8af9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.193:15021/healthz/ready\": dial tcp 10.132.0.193:15021: connect: connection refused" Nov 03 01:06:36.545474 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:06:36.545442 4939 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w4-4a697f/auth-test-model-4a697f-kserve-84b7466bd5-48hbr" Nov 03 01:06:36.816867 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:06:36.816824 4939 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-w3-717217/auth-test-model-717217-kserve-5c9cb7dbd9-vl754" Nov 03 01:06:36.838784 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:06:36.838756 4939 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-w3-717217/auth-test-model-717217-kserve-5c9cb7dbd9-vl754" Nov 03 01:06:37.225986 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:06:37.225957 4939 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w4-4a697f/auth-test-model-4a697f-kserve-84b7466bd5-48hbr"] Nov 03 01:06:37.340604 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:06:37.340559 4939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w4-4a697f/auth-test-model-4a697f-kserve-84b7466bd5-48hbr" event={"ID":"18846667-85a0-4bc2-8995-61bfb2ab1c5d","Type":"ContainerStarted","Data":"771600e7737d0443659f19c9961d3243b84937a107b514a471ab5f80cb0a2ea7"} Nov 03 01:06:37.395606 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:06:37.395568 4939 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-w3-717217/auth-test-model-717217-kserve-5c9cb7dbd9-vl754" Nov 03 01:06:37.455782 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:06:37.455735 4939 patch_prober.go:29] interesting pod/e2e-shared-a-w1-2729f2-openshift-default-9fb97b4fc-dmrf8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.192:15021/healthz/ready\": dial tcp 10.132.0.192:15021: connect: connection refused" start-of-body= Nov 03 01:06:37.456110 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:06:37.456061 4939 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w1-2729f2-openshift-default-9fb97b4fc-dmrf8" podUID="1240d677-9bc2-427c-a818-ffed78093771" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.192:15021/healthz/ready\": dial tcp 10.132.0.192:15021: connect: connection refused" Nov 03 01:06:37.495643 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:06:37.495546 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:06:37.525042 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:06:37.525002 4939 patch_prober.go:29] interesting pod/e2e-shared-a-w1-2729f2-openshift-default-9fb97b4fc-czgh9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.193:15021/healthz/ready\": dial tcp 10.132.0.193:15021: connect: connection refused" start-of-body= Nov 03 01:06:37.525250 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:06:37.525070 4939 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w1-2729f2-openshift-default-9fb97b4fc-czgh9" podUID="66c78580-7ad7-49b8-acf7-b23dd33a8af9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.193:15021/healthz/ready\": dial tcp 10.132.0.193:15021: connect: connection refused" Nov 03 01:06:38.434429 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:06:38.434381 4939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w4-4a697f/auth-test-model-4a697f-kserve-84b7466bd5-48hbr" event={"ID":"18846667-85a0-4bc2-8995-61bfb2ab1c5d","Type":"ContainerStarted","Data":"dce8a0929af2abbec1201ec9933e9052670e290063b7823a598ecebb430fc3dc"} Nov 03 01:06:38.454930 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:06:38.454888 4939 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w1-2729f2-openshift-default-9fb97b4fc-dmrf8" Nov 03 01:06:38.455332 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:06:38.455313 4939 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w1-2729f2-openshift-default-9fb97b4fc-dmrf8" Nov 03 01:06:38.456122 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:06:38.456102 4939 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w1-2729f2-openshift-default-9fb97b4fc-dmrf8" Nov 03 01:06:38.456349 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:06:38.456312 4939 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-w4-4a697f/auth-test-model-4a697f-kserve-84b7466bd5-48hbr" podStartSLOduration=2.456298571 podStartE2EDuration="2.456298571s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 01:06:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 01:06:38.451195383 +0000 UTC m=+1546.715530329" watchObservedRunningTime="2026-11-03 01:06:38.456298571 +0000 UTC m=+1546.720633516" Nov 03 01:06:38.529394 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:06:38.529308 4939 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w1-2729f2-openshift-default-9fb97b4fc-czgh9" Nov 03 01:06:38.530609 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:06:38.530579 4939 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w1-2729f2-openshift-default-9fb97b4fc-czgh9" Nov 03 01:06:38.534931 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:06:38.534028 4939 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w1-2729f2-openshift-default-9fb97b4fc-czgh9" Nov 03 01:06:38.547085 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:06:38.547053 4939 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-w4-4a697f/auth-test-model-4a697f-kserve-84b7466bd5-48hbr" Nov 03 01:06:39.581324 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:06:39.581287 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:06:39.682035 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:06:39.681998 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:06:40.073688 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:06:40.073594 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:06:40.746868 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:06:40.746834 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:06:41.494447 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:06:41.494413 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:06:42.390104 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:06:42.390062 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:06:42.892260 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:06:42.892207 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:06:43.292996 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:06:43.286413 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:06:43.420021 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:06:43.419988 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:06:44.350731 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:06:44.350705 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:06:44.600991 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:06:44.566262 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:06:46.086704 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:06:46.086639 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:06:46.555002 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:06:46.550326 4939 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-w4-4a697f/auth-test-model-4a697f-kserve-84b7466bd5-48hbr" Nov 03 01:06:46.584547 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:06:46.584513 4939 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-w4-4a697f/auth-test-model-4a697f-kserve-84b7466bd5-48hbr" Nov 03 01:06:46.745754 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:06:46.745726 4939 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-w4-4a697f/auth-test-model-4a697f-kserve-84b7466bd5-48hbr" Nov 03 01:06:47.152863 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:06:47.152829 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:06:47.789619 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:06:47.789581 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:06:48.037440 ip-10-0-140-107 kubenswrapper[4939]: E1103 01:06:48.037403 4939 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-sthjh" containerName="extractor" Nov 03 01:06:48.596982 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:06:48.596934 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:06:49.393290 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:06:49.393247 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:06:50.094675 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:06:50.094642 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:06:51.088855 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:06:51.088822 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:06:51.718491 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:06:51.718394 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:06:51.828778 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:06:51.828742 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:06:51.993986 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:06:51.980600 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:06:52.935967 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:06:52.934699 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:06:53.153297 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:06:53.153256 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:06:53.268123 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:06:53.268031 4939 patch_prober.go:29] interesting pod/e2e-shared-b-w0-020af9-openshift-default-d77df45d5-mpgn5 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.185:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 01:06:53.268123 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:06:53.268100 4939 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-b-w0-020af9-openshift-default-d77df45d5-mpgn5" podUID="d7701de4-582e-442f-abc1-13e20bb68b48" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.185:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 01:06:53.865808 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:06:53.865774 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:06:54.585375 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:06:54.585317 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:06:55.476125 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:06:55.476094 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:06:56.108618 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:06:56.108578 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:06:56.398159 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:06:56.384441 4939 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-w3-556c14/auth-test-model-556c14-kserve-8d5cd469-t6tzc"] Nov 03 01:06:56.398159 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:06:56.389550 4939 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-556c14/auth-test-model-556c14-kserve-8d5cd469-t6tzc" Nov 03 01:06:56.402195 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:06:56.398840 4939 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w3-556c14\"/\"auth-test-model-556c14-kserve-self-signed-certs\"" Nov 03 01:06:56.402195 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:06:56.398989 4939 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w3-556c14\"/\"default-dockercfg-mv62m\"" Nov 03 01:06:56.402195 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:06:56.399038 4939 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w3-556c14\"/\"kube-root-ca.crt\"" Nov 03 01:06:56.402195 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:06:56.399130 4939 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w3-556c14\"/\"openshift-service-ca.crt\"" Nov 03 01:06:56.402195 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:06:56.400873 4939 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w3-556c14/auth-test-model-556c14-kserve-8d5cd469-t6tzc"] Nov 03 01:06:56.455353 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:06:56.455319 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:06:56.470097 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:06:56.470070 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0c4e579c-9fe2-4ec9-b8f4-62758e519980-dshm\") pod \"auth-test-model-556c14-kserve-8d5cd469-t6tzc\" (UID: \"0c4e579c-9fe2-4ec9-b8f4-62758e519980\") " pod="ai-tenant-e2e-shared-b-w3-556c14/auth-test-model-556c14-kserve-8d5cd469-t6tzc" Nov 03 01:06:56.470234 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:06:56.470117 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0c4e579c-9fe2-4ec9-b8f4-62758e519980-home\") pod \"auth-test-model-556c14-kserve-8d5cd469-t6tzc\" (UID: \"0c4e579c-9fe2-4ec9-b8f4-62758e519980\") " pod="ai-tenant-e2e-shared-b-w3-556c14/auth-test-model-556c14-kserve-8d5cd469-t6tzc" Nov 03 01:06:56.470234 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:06:56.470184 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0c4e579c-9fe2-4ec9-b8f4-62758e519980-tmp-dir\") pod \"auth-test-model-556c14-kserve-8d5cd469-t6tzc\" (UID: \"0c4e579c-9fe2-4ec9-b8f4-62758e519980\") " pod="ai-tenant-e2e-shared-b-w3-556c14/auth-test-model-556c14-kserve-8d5cd469-t6tzc" Nov 03 01:06:56.470453 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:06:56.470430 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0c4e579c-9fe2-4ec9-b8f4-62758e519980-tls-certs\") pod \"auth-test-model-556c14-kserve-8d5cd469-t6tzc\" (UID: \"0c4e579c-9fe2-4ec9-b8f4-62758e519980\") " pod="ai-tenant-e2e-shared-b-w3-556c14/auth-test-model-556c14-kserve-8d5cd469-t6tzc" Nov 03 01:06:56.470584 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:06:56.470570 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0c4e579c-9fe2-4ec9-b8f4-62758e519980-model-cache\") pod \"auth-test-model-556c14-kserve-8d5cd469-t6tzc\" (UID: \"0c4e579c-9fe2-4ec9-b8f4-62758e519980\") " pod="ai-tenant-e2e-shared-b-w3-556c14/auth-test-model-556c14-kserve-8d5cd469-t6tzc" Nov 03 01:06:56.470640 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:06:56.470597 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tbsmb\" (UniqueName: \"kubernetes.io/projected/0c4e579c-9fe2-4ec9-b8f4-62758e519980-kube-api-access-tbsmb\") pod \"auth-test-model-556c14-kserve-8d5cd469-t6tzc\" (UID: \"0c4e579c-9fe2-4ec9-b8f4-62758e519980\") " pod="ai-tenant-e2e-shared-b-w3-556c14/auth-test-model-556c14-kserve-8d5cd469-t6tzc" Nov 03 01:06:56.604990 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:06:56.573122 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0c4e579c-9fe2-4ec9-b8f4-62758e519980-dshm\") pod \"auth-test-model-556c14-kserve-8d5cd469-t6tzc\" (UID: \"0c4e579c-9fe2-4ec9-b8f4-62758e519980\") " pod="ai-tenant-e2e-shared-b-w3-556c14/auth-test-model-556c14-kserve-8d5cd469-t6tzc" Nov 03 01:06:56.604990 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:06:56.573200 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0c4e579c-9fe2-4ec9-b8f4-62758e519980-home\") pod \"auth-test-model-556c14-kserve-8d5cd469-t6tzc\" (UID: \"0c4e579c-9fe2-4ec9-b8f4-62758e519980\") " pod="ai-tenant-e2e-shared-b-w3-556c14/auth-test-model-556c14-kserve-8d5cd469-t6tzc" Nov 03 01:06:56.604990 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:06:56.573233 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0c4e579c-9fe2-4ec9-b8f4-62758e519980-tmp-dir\") pod \"auth-test-model-556c14-kserve-8d5cd469-t6tzc\" (UID: \"0c4e579c-9fe2-4ec9-b8f4-62758e519980\") " pod="ai-tenant-e2e-shared-b-w3-556c14/auth-test-model-556c14-kserve-8d5cd469-t6tzc" Nov 03 01:06:56.604990 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:06:56.573537 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0c4e579c-9fe2-4ec9-b8f4-62758e519980-tls-certs\") pod \"auth-test-model-556c14-kserve-8d5cd469-t6tzc\" (UID: \"0c4e579c-9fe2-4ec9-b8f4-62758e519980\") " pod="ai-tenant-e2e-shared-b-w3-556c14/auth-test-model-556c14-kserve-8d5cd469-t6tzc" Nov 03 01:06:56.604990 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:06:56.573733 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0c4e579c-9fe2-4ec9-b8f4-62758e519980-model-cache\") pod \"auth-test-model-556c14-kserve-8d5cd469-t6tzc\" (UID: \"0c4e579c-9fe2-4ec9-b8f4-62758e519980\") " pod="ai-tenant-e2e-shared-b-w3-556c14/auth-test-model-556c14-kserve-8d5cd469-t6tzc" Nov 03 01:06:56.604990 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:06:56.573762 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tbsmb\" (UniqueName: \"kubernetes.io/projected/0c4e579c-9fe2-4ec9-b8f4-62758e519980-kube-api-access-tbsmb\") pod \"auth-test-model-556c14-kserve-8d5cd469-t6tzc\" (UID: \"0c4e579c-9fe2-4ec9-b8f4-62758e519980\") " pod="ai-tenant-e2e-shared-b-w3-556c14/auth-test-model-556c14-kserve-8d5cd469-t6tzc" Nov 03 01:06:56.604990 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:06:56.586702 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0c4e579c-9fe2-4ec9-b8f4-62758e519980-dshm\") pod \"auth-test-model-556c14-kserve-8d5cd469-t6tzc\" (UID: \"0c4e579c-9fe2-4ec9-b8f4-62758e519980\") " pod="ai-tenant-e2e-shared-b-w3-556c14/auth-test-model-556c14-kserve-8d5cd469-t6tzc" Nov 03 01:06:56.604990 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:06:56.587006 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0c4e579c-9fe2-4ec9-b8f4-62758e519980-home\") pod \"auth-test-model-556c14-kserve-8d5cd469-t6tzc\" (UID: \"0c4e579c-9fe2-4ec9-b8f4-62758e519980\") " pod="ai-tenant-e2e-shared-b-w3-556c14/auth-test-model-556c14-kserve-8d5cd469-t6tzc" Nov 03 01:06:56.604990 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:06:56.590992 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0c4e579c-9fe2-4ec9-b8f4-62758e519980-tmp-dir\") pod \"auth-test-model-556c14-kserve-8d5cd469-t6tzc\" (UID: \"0c4e579c-9fe2-4ec9-b8f4-62758e519980\") " pod="ai-tenant-e2e-shared-b-w3-556c14/auth-test-model-556c14-kserve-8d5cd469-t6tzc" Nov 03 01:06:56.604990 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:06:56.597461 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0c4e579c-9fe2-4ec9-b8f4-62758e519980-model-cache\") pod \"auth-test-model-556c14-kserve-8d5cd469-t6tzc\" (UID: \"0c4e579c-9fe2-4ec9-b8f4-62758e519980\") " pod="ai-tenant-e2e-shared-b-w3-556c14/auth-test-model-556c14-kserve-8d5cd469-t6tzc" Nov 03 01:06:56.613993 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:06:56.611339 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tbsmb\" (UniqueName: \"kubernetes.io/projected/0c4e579c-9fe2-4ec9-b8f4-62758e519980-kube-api-access-tbsmb\") pod \"auth-test-model-556c14-kserve-8d5cd469-t6tzc\" (UID: \"0c4e579c-9fe2-4ec9-b8f4-62758e519980\") " pod="ai-tenant-e2e-shared-b-w3-556c14/auth-test-model-556c14-kserve-8d5cd469-t6tzc" Nov 03 01:06:56.624088 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:06:56.621022 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0c4e579c-9fe2-4ec9-b8f4-62758e519980-tls-certs\") pod \"auth-test-model-556c14-kserve-8d5cd469-t6tzc\" (UID: \"0c4e579c-9fe2-4ec9-b8f4-62758e519980\") " pod="ai-tenant-e2e-shared-b-w3-556c14/auth-test-model-556c14-kserve-8d5cd469-t6tzc" Nov 03 01:06:56.782469 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:06:56.781216 4939 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-556c14/auth-test-model-556c14-kserve-8d5cd469-t6tzc" Nov 03 01:06:56.865513 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:06:56.865476 4939 generic.go:356] "Generic (PLEG): container finished" podID="d7701de4-582e-442f-abc1-13e20bb68b48" containerID="fd71a31300e5566a1081fb5c193940d6e11b98826ebca49ff97ee875fc10533b" exitCode=0 Nov 03 01:06:56.865999 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:06:56.865861 4939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w0-020af9-openshift-default-d77df45d5-mpgn5" event={"ID":"d7701de4-582e-442f-abc1-13e20bb68b48","Type":"ContainerDied","Data":"fd71a31300e5566a1081fb5c193940d6e11b98826ebca49ff97ee875fc10533b"} Nov 03 01:06:56.866722 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:06:56.866703 4939 scope.go:122] "RemoveContainer" containerID="fd71a31300e5566a1081fb5c193940d6e11b98826ebca49ff97ee875fc10533b" Nov 03 01:06:56.871381 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:06:56.871348 4939 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 01:06:56.871467 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:06:56.871435 4939 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 01:06:56.871506 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:06:56.871483 4939 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 01:06:57.091439 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:06:57.091361 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:06:57.242690 ip-10-0-140-107 kubenswrapper[4939]: W1103 01:06:57.234008 4939 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod0c4e579c_9fe2_4ec9_b8f4_62758e519980.slice/crio-0558234bd46f2ba455472b3f3a8992f0166135a147446b9fd081a49e24ca2012 WatchSource:0}: Error finding container 0558234bd46f2ba455472b3f3a8992f0166135a147446b9fd081a49e24ca2012: Status 404 returned error can't find the container with id 0558234bd46f2ba455472b3f3a8992f0166135a147446b9fd081a49e24ca2012 Nov 03 01:06:57.247989 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:06:57.245070 4939 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w3-556c14/auth-test-model-556c14-kserve-8d5cd469-t6tzc"] Nov 03 01:06:57.801446 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:06:57.801412 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:06:58.005085 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:06:57.994745 4939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-556c14/auth-test-model-556c14-kserve-8d5cd469-t6tzc" event={"ID":"0c4e579c-9fe2-4ec9-b8f4-62758e519980","Type":"ContainerStarted","Data":"7309d80026ae207eb57dd222930c990ca7ea1f761f48e3aefefcd1bab0e710f9"} Nov 03 01:06:58.005085 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:06:57.994809 4939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-556c14/auth-test-model-556c14-kserve-8d5cd469-t6tzc" event={"ID":"0c4e579c-9fe2-4ec9-b8f4-62758e519980","Type":"ContainerStarted","Data":"0558234bd46f2ba455472b3f3a8992f0166135a147446b9fd081a49e24ca2012"} Nov 03 01:06:58.024252 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:06:58.023837 4939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w0-020af9-openshift-default-d77df45d5-mpgn5" event={"ID":"d7701de4-582e-442f-abc1-13e20bb68b48","Type":"ContainerStarted","Data":"cbedf046d78219b00d277e77d21175532544301e6147a9198d528c309dc3e1af"} Nov 03 01:06:58.024252 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:06:58.024138 4939 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-w3-556c14/auth-test-model-556c14-kserve-8d5cd469-t6tzc" podStartSLOduration=2.024119692 podStartE2EDuration="2.024119692s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 01:06:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 01:06:58.022321859 +0000 UTC m=+1566.286656805" watchObservedRunningTime="2026-11-03 01:06:58.024119692 +0000 UTC m=+1566.288454638" Nov 03 01:06:58.258406 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:06:58.258368 4939 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w0-020af9-openshift-default-d77df45d5-mpgn5" Nov 03 01:06:58.259086 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:06:58.259061 4939 patch_prober.go:29] interesting pod/e2e-shared-b-w0-020af9-openshift-default-d77df45d5-mpgn5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.185:15021/healthz/ready\": dial tcp 10.132.0.185:15021: connect: connection refused" start-of-body= Nov 03 01:06:58.259249 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:06:58.259224 4939 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w0-020af9-openshift-default-d77df45d5-mpgn5" podUID="d7701de4-582e-442f-abc1-13e20bb68b48" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.185:15021/healthz/ready\": dial tcp 10.132.0.185:15021: connect: connection refused" Nov 03 01:06:58.587816 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:06:58.587783 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:06:58.787933 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:06:58.784008 4939 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-w3-556c14/auth-test-model-556c14-kserve-8d5cd469-t6tzc" Nov 03 01:06:59.102345 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:06:59.102314 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:06:59.271503 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:06:59.271464 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:06:59.272332 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:06:59.272303 4939 patch_prober.go:29] interesting pod/e2e-shared-b-w0-020af9-openshift-default-d77df45d5-mpgn5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.185:15021/healthz/ready\": dial tcp 10.132.0.185:15021: connect: connection refused" start-of-body= Nov 03 01:06:59.272539 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:06:59.272513 4939 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w0-020af9-openshift-default-d77df45d5-mpgn5" podUID="d7701de4-582e-442f-abc1-13e20bb68b48" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.185:15021/healthz/ready\": dial tcp 10.132.0.185:15021: connect: connection refused" Nov 03 01:06:59.927125 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:06:59.927092 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:07:00.213002 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:07:00.212883 4939 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-4pw7v"] Nov 03 01:07:00.261623 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:07:00.261567 4939 patch_prober.go:29] interesting pod/e2e-shared-b-w0-020af9-openshift-default-d77df45d5-mpgn5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.185:15021/healthz/ready\": dial tcp 10.132.0.185:15021: connect: connection refused" start-of-body= Nov 03 01:07:00.262304 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:07:00.262268 4939 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w0-020af9-openshift-default-d77df45d5-mpgn5" podUID="d7701de4-582e-442f-abc1-13e20bb68b48" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.185:15021/healthz/ready\": dial tcp 10.132.0.185:15021: connect: connection refused" Nov 03 01:07:00.276458 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:07:00.276423 4939 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w3-717217-openshift-default-578c69554c-5xzz5"] Nov 03 01:07:00.283737 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:07:00.283693 4939 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-717217-openshift-default-578c69554c-5xzz5"] Nov 03 01:07:00.284125 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:07:00.284056 4939 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w3-717217-openshift-default-578c69554c-5xzz5" Nov 03 01:07:00.343261 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:07:00.343216 4939 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w1-4af1f7-openshift-default-745ff98565-ktw4m"] Nov 03 01:07:00.385071 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:07:00.371151 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b22388ea-ae77-4ad1-bfff-9aeb20f5bbbd-workload-socket\") pod \"e2e-shared-a-w3-717217-openshift-default-578c69554c-5xzz5\" (UID: \"b22388ea-ae77-4ad1-bfff-9aeb20f5bbbd\") " pod="openshift-ingress/e2e-shared-a-w3-717217-openshift-default-578c69554c-5xzz5" Nov 03 01:07:00.385732 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:07:00.385685 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b22388ea-ae77-4ad1-bfff-9aeb20f5bbbd-credential-socket\") pod \"e2e-shared-a-w3-717217-openshift-default-578c69554c-5xzz5\" (UID: \"b22388ea-ae77-4ad1-bfff-9aeb20f5bbbd\") " pod="openshift-ingress/e2e-shared-a-w3-717217-openshift-default-578c69554c-5xzz5" Nov 03 01:07:00.386103 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:07:00.386078 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b22388ea-ae77-4ad1-bfff-9aeb20f5bbbd-istio-token\") pod \"e2e-shared-a-w3-717217-openshift-default-578c69554c-5xzz5\" (UID: \"b22388ea-ae77-4ad1-bfff-9aeb20f5bbbd\") " pod="openshift-ingress/e2e-shared-a-w3-717217-openshift-default-578c69554c-5xzz5" Nov 03 01:07:00.386432 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:07:00.386407 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7s5lg\" (UniqueName: \"kubernetes.io/projected/b22388ea-ae77-4ad1-bfff-9aeb20f5bbbd-kube-api-access-7s5lg\") pod \"e2e-shared-a-w3-717217-openshift-default-578c69554c-5xzz5\" (UID: \"b22388ea-ae77-4ad1-bfff-9aeb20f5bbbd\") " pod="openshift-ingress/e2e-shared-a-w3-717217-openshift-default-578c69554c-5xzz5" Nov 03 01:07:00.386848 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:07:00.386824 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b22388ea-ae77-4ad1-bfff-9aeb20f5bbbd-istiod-ca-cert\") pod \"e2e-shared-a-w3-717217-openshift-default-578c69554c-5xzz5\" (UID: \"b22388ea-ae77-4ad1-bfff-9aeb20f5bbbd\") " pod="openshift-ingress/e2e-shared-a-w3-717217-openshift-default-578c69554c-5xzz5" Nov 03 01:07:00.387094 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:07:00.387053 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b22388ea-ae77-4ad1-bfff-9aeb20f5bbbd-istio-envoy\") pod \"e2e-shared-a-w3-717217-openshift-default-578c69554c-5xzz5\" (UID: \"b22388ea-ae77-4ad1-bfff-9aeb20f5bbbd\") " pod="openshift-ingress/e2e-shared-a-w3-717217-openshift-default-578c69554c-5xzz5" Nov 03 01:07:00.387303 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:07:00.387285 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b22388ea-ae77-4ad1-bfff-9aeb20f5bbbd-workload-certs\") pod \"e2e-shared-a-w3-717217-openshift-default-578c69554c-5xzz5\" (UID: \"b22388ea-ae77-4ad1-bfff-9aeb20f5bbbd\") " pod="openshift-ingress/e2e-shared-a-w3-717217-openshift-default-578c69554c-5xzz5" Nov 03 01:07:00.387514 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:07:00.387494 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b22388ea-ae77-4ad1-bfff-9aeb20f5bbbd-istio-data\") pod \"e2e-shared-a-w3-717217-openshift-default-578c69554c-5xzz5\" (UID: \"b22388ea-ae77-4ad1-bfff-9aeb20f5bbbd\") " pod="openshift-ingress/e2e-shared-a-w3-717217-openshift-default-578c69554c-5xzz5" Nov 03 01:07:00.387661 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:07:00.387641 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b22388ea-ae77-4ad1-bfff-9aeb20f5bbbd-istio-podinfo\") pod \"e2e-shared-a-w3-717217-openshift-default-578c69554c-5xzz5\" (UID: \"b22388ea-ae77-4ad1-bfff-9aeb20f5bbbd\") " pod="openshift-ingress/e2e-shared-a-w3-717217-openshift-default-578c69554c-5xzz5" Nov 03 01:07:00.400223 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:07:00.400190 4939 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w1-4af1f7-openshift-default-745ff98565-ktw4m"] Nov 03 01:07:00.400362 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:07:00.400305 4939 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w1-4af1f7-openshift-default-745ff98565-ktw4m" Nov 03 01:07:00.489270 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:07:00.489180 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b22388ea-ae77-4ad1-bfff-9aeb20f5bbbd-istio-token\") pod \"e2e-shared-a-w3-717217-openshift-default-578c69554c-5xzz5\" (UID: \"b22388ea-ae77-4ad1-bfff-9aeb20f5bbbd\") " pod="openshift-ingress/e2e-shared-a-w3-717217-openshift-default-578c69554c-5xzz5" Nov 03 01:07:00.489510 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:07:00.489269 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7s5lg\" (UniqueName: \"kubernetes.io/projected/b22388ea-ae77-4ad1-bfff-9aeb20f5bbbd-kube-api-access-7s5lg\") pod \"e2e-shared-a-w3-717217-openshift-default-578c69554c-5xzz5\" (UID: \"b22388ea-ae77-4ad1-bfff-9aeb20f5bbbd\") " pod="openshift-ingress/e2e-shared-a-w3-717217-openshift-default-578c69554c-5xzz5" Nov 03 01:07:00.489510 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:07:00.489418 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b22388ea-ae77-4ad1-bfff-9aeb20f5bbbd-istiod-ca-cert\") pod \"e2e-shared-a-w3-717217-openshift-default-578c69554c-5xzz5\" (UID: \"b22388ea-ae77-4ad1-bfff-9aeb20f5bbbd\") " pod="openshift-ingress/e2e-shared-a-w3-717217-openshift-default-578c69554c-5xzz5" Nov 03 01:07:00.490110 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:07:00.490089 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b22388ea-ae77-4ad1-bfff-9aeb20f5bbbd-istiod-ca-cert\") pod \"e2e-shared-a-w3-717217-openshift-default-578c69554c-5xzz5\" (UID: \"b22388ea-ae77-4ad1-bfff-9aeb20f5bbbd\") " pod="openshift-ingress/e2e-shared-a-w3-717217-openshift-default-578c69554c-5xzz5" Nov 03 01:07:00.490173 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:07:00.490159 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b22388ea-ae77-4ad1-bfff-9aeb20f5bbbd-istio-envoy\") pod \"e2e-shared-a-w3-717217-openshift-default-578c69554c-5xzz5\" (UID: \"b22388ea-ae77-4ad1-bfff-9aeb20f5bbbd\") " pod="openshift-ingress/e2e-shared-a-w3-717217-openshift-default-578c69554c-5xzz5" Nov 03 01:07:00.490214 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:07:00.490205 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b22388ea-ae77-4ad1-bfff-9aeb20f5bbbd-workload-certs\") pod \"e2e-shared-a-w3-717217-openshift-default-578c69554c-5xzz5\" (UID: \"b22388ea-ae77-4ad1-bfff-9aeb20f5bbbd\") " pod="openshift-ingress/e2e-shared-a-w3-717217-openshift-default-578c69554c-5xzz5" Nov 03 01:07:00.490272 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:07:00.490257 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b22388ea-ae77-4ad1-bfff-9aeb20f5bbbd-istio-data\") pod \"e2e-shared-a-w3-717217-openshift-default-578c69554c-5xzz5\" (UID: \"b22388ea-ae77-4ad1-bfff-9aeb20f5bbbd\") " pod="openshift-ingress/e2e-shared-a-w3-717217-openshift-default-578c69554c-5xzz5" Nov 03 01:07:00.490329 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:07:00.490289 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b22388ea-ae77-4ad1-bfff-9aeb20f5bbbd-istio-podinfo\") pod \"e2e-shared-a-w3-717217-openshift-default-578c69554c-5xzz5\" (UID: \"b22388ea-ae77-4ad1-bfff-9aeb20f5bbbd\") " pod="openshift-ingress/e2e-shared-a-w3-717217-openshift-default-578c69554c-5xzz5" Nov 03 01:07:00.490385 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:07:00.490336 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b22388ea-ae77-4ad1-bfff-9aeb20f5bbbd-workload-socket\") pod \"e2e-shared-a-w3-717217-openshift-default-578c69554c-5xzz5\" (UID: \"b22388ea-ae77-4ad1-bfff-9aeb20f5bbbd\") " pod="openshift-ingress/e2e-shared-a-w3-717217-openshift-default-578c69554c-5xzz5" Nov 03 01:07:00.490435 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:07:00.490412 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b22388ea-ae77-4ad1-bfff-9aeb20f5bbbd-credential-socket\") pod \"e2e-shared-a-w3-717217-openshift-default-578c69554c-5xzz5\" (UID: \"b22388ea-ae77-4ad1-bfff-9aeb20f5bbbd\") " pod="openshift-ingress/e2e-shared-a-w3-717217-openshift-default-578c69554c-5xzz5" Nov 03 01:07:00.490646 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:07:00.490629 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b22388ea-ae77-4ad1-bfff-9aeb20f5bbbd-istio-data\") pod \"e2e-shared-a-w3-717217-openshift-default-578c69554c-5xzz5\" (UID: \"b22388ea-ae77-4ad1-bfff-9aeb20f5bbbd\") " pod="openshift-ingress/e2e-shared-a-w3-717217-openshift-default-578c69554c-5xzz5" Nov 03 01:07:00.490752 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:07:00.490731 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b22388ea-ae77-4ad1-bfff-9aeb20f5bbbd-workload-socket\") pod \"e2e-shared-a-w3-717217-openshift-default-578c69554c-5xzz5\" (UID: \"b22388ea-ae77-4ad1-bfff-9aeb20f5bbbd\") " pod="openshift-ingress/e2e-shared-a-w3-717217-openshift-default-578c69554c-5xzz5" Nov 03 01:07:00.490854 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:07:00.490831 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b22388ea-ae77-4ad1-bfff-9aeb20f5bbbd-credential-socket\") pod \"e2e-shared-a-w3-717217-openshift-default-578c69554c-5xzz5\" (UID: \"b22388ea-ae77-4ad1-bfff-9aeb20f5bbbd\") " pod="openshift-ingress/e2e-shared-a-w3-717217-openshift-default-578c69554c-5xzz5" Nov 03 01:07:00.490968 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:07:00.490950 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b22388ea-ae77-4ad1-bfff-9aeb20f5bbbd-workload-certs\") pod \"e2e-shared-a-w3-717217-openshift-default-578c69554c-5xzz5\" (UID: \"b22388ea-ae77-4ad1-bfff-9aeb20f5bbbd\") " pod="openshift-ingress/e2e-shared-a-w3-717217-openshift-default-578c69554c-5xzz5" Nov 03 01:07:00.492766 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:07:00.492740 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b22388ea-ae77-4ad1-bfff-9aeb20f5bbbd-istio-envoy\") pod \"e2e-shared-a-w3-717217-openshift-default-578c69554c-5xzz5\" (UID: \"b22388ea-ae77-4ad1-bfff-9aeb20f5bbbd\") " pod="openshift-ingress/e2e-shared-a-w3-717217-openshift-default-578c69554c-5xzz5" Nov 03 01:07:00.493115 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:07:00.493096 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b22388ea-ae77-4ad1-bfff-9aeb20f5bbbd-istio-podinfo\") pod \"e2e-shared-a-w3-717217-openshift-default-578c69554c-5xzz5\" (UID: \"b22388ea-ae77-4ad1-bfff-9aeb20f5bbbd\") " pod="openshift-ingress/e2e-shared-a-w3-717217-openshift-default-578c69554c-5xzz5" Nov 03 01:07:00.499077 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:07:00.499050 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b22388ea-ae77-4ad1-bfff-9aeb20f5bbbd-istio-token\") pod \"e2e-shared-a-w3-717217-openshift-default-578c69554c-5xzz5\" (UID: \"b22388ea-ae77-4ad1-bfff-9aeb20f5bbbd\") " pod="openshift-ingress/e2e-shared-a-w3-717217-openshift-default-578c69554c-5xzz5" Nov 03 01:07:00.499252 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:07:00.499232 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7s5lg\" (UniqueName: \"kubernetes.io/projected/b22388ea-ae77-4ad1-bfff-9aeb20f5bbbd-kube-api-access-7s5lg\") pod \"e2e-shared-a-w3-717217-openshift-default-578c69554c-5xzz5\" (UID: \"b22388ea-ae77-4ad1-bfff-9aeb20f5bbbd\") " pod="openshift-ingress/e2e-shared-a-w3-717217-openshift-default-578c69554c-5xzz5" Nov 03 01:07:00.592359 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:07:00.592312 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d2965442-bcbd-4959-b60a-08074c5aead5-istio-data\") pod \"e2e-shared-b-w1-4af1f7-openshift-default-745ff98565-ktw4m\" (UID: \"d2965442-bcbd-4959-b60a-08074c5aead5\") " pod="openshift-ingress/e2e-shared-b-w1-4af1f7-openshift-default-745ff98565-ktw4m" Nov 03 01:07:00.592552 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:07:00.592523 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xgds7\" (UniqueName: \"kubernetes.io/projected/d2965442-bcbd-4959-b60a-08074c5aead5-kube-api-access-xgds7\") pod \"e2e-shared-b-w1-4af1f7-openshift-default-745ff98565-ktw4m\" (UID: \"d2965442-bcbd-4959-b60a-08074c5aead5\") " pod="openshift-ingress/e2e-shared-b-w1-4af1f7-openshift-default-745ff98565-ktw4m" Nov 03 01:07:00.592676 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:07:00.592657 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d2965442-bcbd-4959-b60a-08074c5aead5-istio-podinfo\") pod \"e2e-shared-b-w1-4af1f7-openshift-default-745ff98565-ktw4m\" (UID: \"d2965442-bcbd-4959-b60a-08074c5aead5\") " pod="openshift-ingress/e2e-shared-b-w1-4af1f7-openshift-default-745ff98565-ktw4m" Nov 03 01:07:00.592746 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:07:00.592734 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d2965442-bcbd-4959-b60a-08074c5aead5-workload-certs\") pod \"e2e-shared-b-w1-4af1f7-openshift-default-745ff98565-ktw4m\" (UID: \"d2965442-bcbd-4959-b60a-08074c5aead5\") " pod="openshift-ingress/e2e-shared-b-w1-4af1f7-openshift-default-745ff98565-ktw4m" Nov 03 01:07:00.592866 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:07:00.592820 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d2965442-bcbd-4959-b60a-08074c5aead5-credential-socket\") pod \"e2e-shared-b-w1-4af1f7-openshift-default-745ff98565-ktw4m\" (UID: \"d2965442-bcbd-4959-b60a-08074c5aead5\") " pod="openshift-ingress/e2e-shared-b-w1-4af1f7-openshift-default-745ff98565-ktw4m" Nov 03 01:07:00.592981 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:07:00.592872 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d2965442-bcbd-4959-b60a-08074c5aead5-istio-token\") pod \"e2e-shared-b-w1-4af1f7-openshift-default-745ff98565-ktw4m\" (UID: \"d2965442-bcbd-4959-b60a-08074c5aead5\") " pod="openshift-ingress/e2e-shared-b-w1-4af1f7-openshift-default-745ff98565-ktw4m" Nov 03 01:07:00.593053 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:07:00.593028 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d2965442-bcbd-4959-b60a-08074c5aead5-istiod-ca-cert\") pod \"e2e-shared-b-w1-4af1f7-openshift-default-745ff98565-ktw4m\" (UID: \"d2965442-bcbd-4959-b60a-08074c5aead5\") " pod="openshift-ingress/e2e-shared-b-w1-4af1f7-openshift-default-745ff98565-ktw4m" Nov 03 01:07:00.593171 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:07:00.593147 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d2965442-bcbd-4959-b60a-08074c5aead5-workload-socket\") pod \"e2e-shared-b-w1-4af1f7-openshift-default-745ff98565-ktw4m\" (UID: \"d2965442-bcbd-4959-b60a-08074c5aead5\") " pod="openshift-ingress/e2e-shared-b-w1-4af1f7-openshift-default-745ff98565-ktw4m" Nov 03 01:07:00.593239 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:07:00.593191 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d2965442-bcbd-4959-b60a-08074c5aead5-istio-envoy\") pod \"e2e-shared-b-w1-4af1f7-openshift-default-745ff98565-ktw4m\" (UID: \"d2965442-bcbd-4959-b60a-08074c5aead5\") " pod="openshift-ingress/e2e-shared-b-w1-4af1f7-openshift-default-745ff98565-ktw4m" Nov 03 01:07:00.604945 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:07:00.604897 4939 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w3-717217-openshift-default-578c69554c-5xzz5" Nov 03 01:07:00.699652 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:07:00.699591 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d2965442-bcbd-4959-b60a-08074c5aead5-workload-certs\") pod \"e2e-shared-b-w1-4af1f7-openshift-default-745ff98565-ktw4m\" (UID: \"d2965442-bcbd-4959-b60a-08074c5aead5\") " pod="openshift-ingress/e2e-shared-b-w1-4af1f7-openshift-default-745ff98565-ktw4m" Nov 03 01:07:00.699803 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:07:00.699711 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d2965442-bcbd-4959-b60a-08074c5aead5-credential-socket\") pod \"e2e-shared-b-w1-4af1f7-openshift-default-745ff98565-ktw4m\" (UID: \"d2965442-bcbd-4959-b60a-08074c5aead5\") " pod="openshift-ingress/e2e-shared-b-w1-4af1f7-openshift-default-745ff98565-ktw4m" Nov 03 01:07:00.700631 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:07:00.700146 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d2965442-bcbd-4959-b60a-08074c5aead5-workload-certs\") pod \"e2e-shared-b-w1-4af1f7-openshift-default-745ff98565-ktw4m\" (UID: \"d2965442-bcbd-4959-b60a-08074c5aead5\") " pod="openshift-ingress/e2e-shared-b-w1-4af1f7-openshift-default-745ff98565-ktw4m" Nov 03 01:07:00.700861 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:07:00.700839 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d2965442-bcbd-4959-b60a-08074c5aead5-istio-token\") pod \"e2e-shared-b-w1-4af1f7-openshift-default-745ff98565-ktw4m\" (UID: \"d2965442-bcbd-4959-b60a-08074c5aead5\") " pod="openshift-ingress/e2e-shared-b-w1-4af1f7-openshift-default-745ff98565-ktw4m" Nov 03 01:07:00.701141 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:07:00.701120 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d2965442-bcbd-4959-b60a-08074c5aead5-istiod-ca-cert\") pod \"e2e-shared-b-w1-4af1f7-openshift-default-745ff98565-ktw4m\" (UID: \"d2965442-bcbd-4959-b60a-08074c5aead5\") " pod="openshift-ingress/e2e-shared-b-w1-4af1f7-openshift-default-745ff98565-ktw4m" Nov 03 01:07:00.701303 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:07:00.701280 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d2965442-bcbd-4959-b60a-08074c5aead5-workload-socket\") pod \"e2e-shared-b-w1-4af1f7-openshift-default-745ff98565-ktw4m\" (UID: \"d2965442-bcbd-4959-b60a-08074c5aead5\") " pod="openshift-ingress/e2e-shared-b-w1-4af1f7-openshift-default-745ff98565-ktw4m" Nov 03 01:07:00.701374 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:07:00.701343 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d2965442-bcbd-4959-b60a-08074c5aead5-istio-envoy\") pod \"e2e-shared-b-w1-4af1f7-openshift-default-745ff98565-ktw4m\" (UID: \"d2965442-bcbd-4959-b60a-08074c5aead5\") " pod="openshift-ingress/e2e-shared-b-w1-4af1f7-openshift-default-745ff98565-ktw4m" Nov 03 01:07:00.701463 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:07:00.701449 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d2965442-bcbd-4959-b60a-08074c5aead5-istio-data\") pod \"e2e-shared-b-w1-4af1f7-openshift-default-745ff98565-ktw4m\" (UID: \"d2965442-bcbd-4959-b60a-08074c5aead5\") " pod="openshift-ingress/e2e-shared-b-w1-4af1f7-openshift-default-745ff98565-ktw4m" Nov 03 01:07:00.701575 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:07:00.701559 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xgds7\" (UniqueName: \"kubernetes.io/projected/d2965442-bcbd-4959-b60a-08074c5aead5-kube-api-access-xgds7\") pod \"e2e-shared-b-w1-4af1f7-openshift-default-745ff98565-ktw4m\" (UID: \"d2965442-bcbd-4959-b60a-08074c5aead5\") " pod="openshift-ingress/e2e-shared-b-w1-4af1f7-openshift-default-745ff98565-ktw4m" Nov 03 01:07:00.701678 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:07:00.701664 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d2965442-bcbd-4959-b60a-08074c5aead5-istio-podinfo\") pod \"e2e-shared-b-w1-4af1f7-openshift-default-745ff98565-ktw4m\" (UID: \"d2965442-bcbd-4959-b60a-08074c5aead5\") " pod="openshift-ingress/e2e-shared-b-w1-4af1f7-openshift-default-745ff98565-ktw4m" Nov 03 01:07:00.703388 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:07:00.702348 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d2965442-bcbd-4959-b60a-08074c5aead5-workload-socket\") pod \"e2e-shared-b-w1-4af1f7-openshift-default-745ff98565-ktw4m\" (UID: \"d2965442-bcbd-4959-b60a-08074c5aead5\") " pod="openshift-ingress/e2e-shared-b-w1-4af1f7-openshift-default-745ff98565-ktw4m" Nov 03 01:07:00.703388 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:07:00.702541 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d2965442-bcbd-4959-b60a-08074c5aead5-credential-socket\") pod \"e2e-shared-b-w1-4af1f7-openshift-default-745ff98565-ktw4m\" (UID: \"d2965442-bcbd-4959-b60a-08074c5aead5\") " pod="openshift-ingress/e2e-shared-b-w1-4af1f7-openshift-default-745ff98565-ktw4m" Nov 03 01:07:00.703388 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:07:00.703272 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d2965442-bcbd-4959-b60a-08074c5aead5-istiod-ca-cert\") pod \"e2e-shared-b-w1-4af1f7-openshift-default-745ff98565-ktw4m\" (UID: \"d2965442-bcbd-4959-b60a-08074c5aead5\") " pod="openshift-ingress/e2e-shared-b-w1-4af1f7-openshift-default-745ff98565-ktw4m" Nov 03 01:07:00.703388 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:07:00.703313 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d2965442-bcbd-4959-b60a-08074c5aead5-istio-data\") pod \"e2e-shared-b-w1-4af1f7-openshift-default-745ff98565-ktw4m\" (UID: \"d2965442-bcbd-4959-b60a-08074c5aead5\") " pod="openshift-ingress/e2e-shared-b-w1-4af1f7-openshift-default-745ff98565-ktw4m" Nov 03 01:07:00.706337 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:07:00.706302 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d2965442-bcbd-4959-b60a-08074c5aead5-istio-podinfo\") pod \"e2e-shared-b-w1-4af1f7-openshift-default-745ff98565-ktw4m\" (UID: \"d2965442-bcbd-4959-b60a-08074c5aead5\") " pod="openshift-ingress/e2e-shared-b-w1-4af1f7-openshift-default-745ff98565-ktw4m" Nov 03 01:07:00.708688 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:07:00.708509 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d2965442-bcbd-4959-b60a-08074c5aead5-istio-envoy\") pod \"e2e-shared-b-w1-4af1f7-openshift-default-745ff98565-ktw4m\" (UID: \"d2965442-bcbd-4959-b60a-08074c5aead5\") " pod="openshift-ingress/e2e-shared-b-w1-4af1f7-openshift-default-745ff98565-ktw4m" Nov 03 01:07:00.713728 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:07:00.712817 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d2965442-bcbd-4959-b60a-08074c5aead5-istio-token\") pod \"e2e-shared-b-w1-4af1f7-openshift-default-745ff98565-ktw4m\" (UID: \"d2965442-bcbd-4959-b60a-08074c5aead5\") " pod="openshift-ingress/e2e-shared-b-w1-4af1f7-openshift-default-745ff98565-ktw4m" Nov 03 01:07:00.714942 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:07:00.714660 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xgds7\" (UniqueName: \"kubernetes.io/projected/d2965442-bcbd-4959-b60a-08074c5aead5-kube-api-access-xgds7\") pod \"e2e-shared-b-w1-4af1f7-openshift-default-745ff98565-ktw4m\" (UID: \"d2965442-bcbd-4959-b60a-08074c5aead5\") " pod="openshift-ingress/e2e-shared-b-w1-4af1f7-openshift-default-745ff98565-ktw4m" Nov 03 01:07:00.727048 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:07:00.718037 4939 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w1-4af1f7-openshift-default-745ff98565-ktw4m" Nov 03 01:07:00.779519 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:07:00.779433 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:07:01.255825 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:07:01.255787 4939 patch_prober.go:29] interesting pod/e2e-shared-b-w0-020af9-openshift-default-d77df45d5-mpgn5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.185:15021/healthz/ready\": dial tcp 10.132.0.185:15021: connect: connection refused" start-of-body= Nov 03 01:07:01.256120 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:07:01.256077 4939 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w0-020af9-openshift-default-d77df45d5-mpgn5" podUID="d7701de4-582e-442f-abc1-13e20bb68b48" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.185:15021/healthz/ready\": dial tcp 10.132.0.185:15021: connect: connection refused" Nov 03 01:07:01.314965 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:07:01.314592 4939 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-717217-openshift-default-578c69554c-5xzz5"] Nov 03 01:07:01.317195 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:07:01.317152 4939 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 01:07:01.317431 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:07:01.317391 4939 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 01:07:01.317535 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:07:01.317444 4939 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 01:07:01.624118 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:07:01.624076 4939 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w1-4af1f7-openshift-default-745ff98565-ktw4m"] Nov 03 01:07:01.643260 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:07:01.643215 4939 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 01:07:01.643439 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:07:01.643297 4939 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 01:07:01.643439 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:07:01.643324 4939 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 01:07:02.309322 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:07:02.309226 4939 patch_prober.go:29] interesting pod/e2e-shared-b-w0-020af9-openshift-default-d77df45d5-mpgn5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.185:15021/healthz/ready\": dial tcp 10.132.0.185:15021: connect: connection refused" start-of-body= Nov 03 01:07:02.309598 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:07:02.309566 4939 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w0-020af9-openshift-default-d77df45d5-mpgn5" podUID="d7701de4-582e-442f-abc1-13e20bb68b48" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.185:15021/healthz/ready\": dial tcp 10.132.0.185:15021: connect: connection refused" Nov 03 01:07:02.329563 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:07:02.329507 4939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w1-4af1f7-openshift-default-745ff98565-ktw4m" event={"ID":"d2965442-bcbd-4959-b60a-08074c5aead5","Type":"ContainerStarted","Data":"86f0f2a7c74a736a7de851e69d10082bea516054b535c56a2218344ac8767568"} Nov 03 01:07:02.339421 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:07:02.339379 4939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-717217-openshift-default-578c69554c-5xzz5" event={"ID":"b22388ea-ae77-4ad1-bfff-9aeb20f5bbbd","Type":"ContainerStarted","Data":"9d81b88a637be26b8213faa245c11d4ed680ab79f1d5df2f4c6bf91fd13431e4"} Nov 03 01:07:02.339699 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:07:02.339675 4939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-717217-openshift-default-578c69554c-5xzz5" event={"ID":"b22388ea-ae77-4ad1-bfff-9aeb20f5bbbd","Type":"ContainerStarted","Data":"83349dd90858a6925bc45287148fbad42fa1e78be863611d89590a90d6ad42dc"} Nov 03 01:07:02.403075 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:07:02.403008 4939 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w3-717217-openshift-default-578c69554c-5xzz5" podStartSLOduration=2.402986818 podStartE2EDuration="2.402986818s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 01:07:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 01:07:02.373064381 +0000 UTC m=+1570.637399329" watchObservedRunningTime="2026-11-03 01:07:02.402986818 +0000 UTC m=+1570.667321764" Nov 03 01:07:02.515982 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:07:02.515943 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:07:02.613945 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:07:02.613173 4939 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w3-717217-openshift-default-578c69554c-5xzz5" Nov 03 01:07:02.622964 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:07:02.621192 4939 patch_prober.go:29] interesting pod/e2e-shared-a-w3-717217-openshift-default-578c69554c-5xzz5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.196:15021/healthz/ready\": dial tcp 10.132.0.196:15021: connect: connection refused" start-of-body= Nov 03 01:07:02.622964 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:07:02.621262 4939 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-717217-openshift-default-578c69554c-5xzz5" podUID="b22388ea-ae77-4ad1-bfff-9aeb20f5bbbd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.196:15021/healthz/ready\": dial tcp 10.132.0.196:15021: connect: connection refused" Nov 03 01:07:03.276961 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:07:03.259335 4939 patch_prober.go:29] interesting pod/e2e-shared-b-w0-020af9-openshift-default-d77df45d5-mpgn5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.185:15021/healthz/ready\": dial tcp 10.132.0.185:15021: connect: connection refused" start-of-body= Nov 03 01:07:03.276961 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:07:03.259411 4939 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w0-020af9-openshift-default-d77df45d5-mpgn5" podUID="d7701de4-582e-442f-abc1-13e20bb68b48" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.185:15021/healthz/ready\": dial tcp 10.132.0.185:15021: connect: connection refused" Nov 03 01:07:03.379932 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:07:03.379488 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:07:03.445075 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:07:03.445027 4939 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-w0-020af9/auth-test-model-020af9-kserve-844df75d9c-ljr9s"] Nov 03 01:07:03.524499 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:07:03.509316 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:07:03.789967 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:07:03.771155 4939 patch_prober.go:29] interesting pod/e2e-shared-a-w3-717217-openshift-default-578c69554c-5xzz5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.196:15021/healthz/ready\": dial tcp 10.132.0.196:15021: connect: connection refused" start-of-body= Nov 03 01:07:03.789967 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:07:03.771214 4939 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-717217-openshift-default-578c69554c-5xzz5" podUID="b22388ea-ae77-4ad1-bfff-9aeb20f5bbbd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.196:15021/healthz/ready\": dial tcp 10.132.0.196:15021: connect: connection refused" Nov 03 01:07:04.255005 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:07:04.234571 4939 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w1-4af1f7-openshift-default-745ff98565-ktw4m" podStartSLOduration=4.234548543 podStartE2EDuration="4.234548543s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 01:07:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 01:07:04.120072532 +0000 UTC m=+1572.384407527" watchObservedRunningTime="2026-11-03 01:07:04.234548543 +0000 UTC m=+1572.498883489" Nov 03 01:07:04.340100 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:07:04.340059 4939 patch_prober.go:29] interesting pod/e2e-shared-b-w0-020af9-openshift-default-d77df45d5-mpgn5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.185:15021/healthz/ready\": dial tcp 10.132.0.185:15021: connect: connection refused" start-of-body= Nov 03 01:07:04.366142 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:07:04.366094 4939 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w0-020af9-openshift-default-d77df45d5-mpgn5" podUID="d7701de4-582e-442f-abc1-13e20bb68b48" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.185:15021/healthz/ready\": dial tcp 10.132.0.185:15021: connect: connection refused" Nov 03 01:07:04.417207 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:07:04.386088 4939 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-shared-a-w3-717217/auth-test-model-717217-kserve-5c9cb7dbd9-vl754" podUID="9634a946-0832-4901-ad01-bbb7a47f4ac7" containerName="main" probeResult="failure" output="Get \"https://10.132.0.191:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 01:07:04.421786 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:07:04.421714 4939 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w0-020af9/auth-test-model-020af9-kserve-844df75d9c-ljr9s"] Nov 03 01:07:04.426884 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:07:04.422461 4939 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w0-020af9/auth-test-model-020af9-kserve-844df75d9c-ljr9s" Nov 03 01:07:04.431535 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:07:04.426743 4939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w1-4af1f7-openshift-default-745ff98565-ktw4m" event={"ID":"d2965442-bcbd-4959-b60a-08074c5aead5","Type":"ContainerStarted","Data":"5d291651f437ed1fb8d1fa3e61977d39dd0987d4be3ae72a2a5424d02e9e4256"} Nov 03 01:07:04.431535 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:07:04.430168 4939 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w0-50e623-openshift-default-6c86dc794f-5m78z"] Nov 03 01:07:04.489570 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:07:04.489541 4939 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w0-50e623-openshift-default-6c86dc794f-5m78z" Nov 03 01:07:04.506006 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:07:04.490599 4939 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w0-020af9\"/\"kube-root-ca.crt\"" Nov 03 01:07:04.506006 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:07:04.491062 4939 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w0-020af9\"/\"default-dockercfg-lvsmb\"" Nov 03 01:07:04.506006 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:07:04.491180 4939 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w0-020af9\"/\"auth-test-model-020af9-kserve-self-signed-certs\"" Nov 03 01:07:04.506006 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:07:04.491875 4939 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w0-020af9\"/\"openshift-service-ca.crt\"" Nov 03 01:07:04.508211 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:07:04.507459 4939 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-83755a/distinct-2-w3-83755a-kserve-6cf66b995d-2l47b" podUID="c3b05e88-5d5b-4d03-b28d-832f22e3c868" containerName="main" probeResult="failure" output="Get \"https://10.132.0.72:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Nov 03 01:07:04.532886 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:07:04.532745 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9w5qh\" (UniqueName: \"kubernetes.io/projected/24fb3670-ec70-492c-a573-4cbe174f1ccf-kube-api-access-9w5qh\") pod \"auth-test-model-020af9-kserve-844df75d9c-ljr9s\" (UID: \"24fb3670-ec70-492c-a573-4cbe174f1ccf\") " pod="ai-tenant-e2e-shared-b-w0-020af9/auth-test-model-020af9-kserve-844df75d9c-ljr9s" Nov 03 01:07:04.532886 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:07:04.532814 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/24fb3670-ec70-492c-a573-4cbe174f1ccf-tls-certs\") pod \"auth-test-model-020af9-kserve-844df75d9c-ljr9s\" (UID: \"24fb3670-ec70-492c-a573-4cbe174f1ccf\") " pod="ai-tenant-e2e-shared-b-w0-020af9/auth-test-model-020af9-kserve-844df75d9c-ljr9s" Nov 03 01:07:04.532886 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:07:04.532884 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/24fb3670-ec70-492c-a573-4cbe174f1ccf-model-cache\") pod \"auth-test-model-020af9-kserve-844df75d9c-ljr9s\" (UID: \"24fb3670-ec70-492c-a573-4cbe174f1ccf\") " pod="ai-tenant-e2e-shared-b-w0-020af9/auth-test-model-020af9-kserve-844df75d9c-ljr9s" Nov 03 01:07:04.533216 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:07:04.533005 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/24fb3670-ec70-492c-a573-4cbe174f1ccf-tmp-dir\") pod \"auth-test-model-020af9-kserve-844df75d9c-ljr9s\" (UID: \"24fb3670-ec70-492c-a573-4cbe174f1ccf\") " pod="ai-tenant-e2e-shared-b-w0-020af9/auth-test-model-020af9-kserve-844df75d9c-ljr9s" Nov 03 01:07:04.533216 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:07:04.533067 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/24fb3670-ec70-492c-a573-4cbe174f1ccf-home\") pod \"auth-test-model-020af9-kserve-844df75d9c-ljr9s\" (UID: \"24fb3670-ec70-492c-a573-4cbe174f1ccf\") " pod="ai-tenant-e2e-shared-b-w0-020af9/auth-test-model-020af9-kserve-844df75d9c-ljr9s" Nov 03 01:07:04.533216 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:07:04.533106 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/24fb3670-ec70-492c-a573-4cbe174f1ccf-dshm\") pod \"auth-test-model-020af9-kserve-844df75d9c-ljr9s\" (UID: \"24fb3670-ec70-492c-a573-4cbe174f1ccf\") " pod="ai-tenant-e2e-shared-b-w0-020af9/auth-test-model-020af9-kserve-844df75d9c-ljr9s" Nov 03 01:07:04.600889 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:07:04.600855 4939 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w0-50e623-openshift-default-6c86dc794f-5m78z"] Nov 03 01:07:04.601109 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:07:04.601032 4939 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w0-50e623-openshift-default-6c86dc794f-jmfmb"] Nov 03 01:07:04.608936 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:07:04.606555 4939 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w0-50e623-openshift-default-6c86dc794f-jmfmb"] Nov 03 01:07:04.611055 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:07:04.610209 4939 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w0-50e623-openshift-default-6c86dc794f-jmfmb" Nov 03 01:07:04.612087 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:07:04.612057 4939 patch_prober.go:29] interesting pod/e2e-shared-a-w3-717217-openshift-default-578c69554c-5xzz5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.196:15021/healthz/ready\": dial tcp 10.132.0.196:15021: connect: connection refused" start-of-body= Nov 03 01:07:04.612314 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:07:04.612285 4939 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-717217-openshift-default-578c69554c-5xzz5" podUID="b22388ea-ae77-4ad1-bfff-9aeb20f5bbbd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.196:15021/healthz/ready\": dial tcp 10.132.0.196:15021: connect: connection refused" Nov 03 01:07:04.633840 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:07:04.633803 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:07:04.634842 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:07:04.634813 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/aba9a06e-b5d6-491f-b9a3-34ebd927d4aa-istiod-ca-cert\") pod \"e2e-shared-a-w0-50e623-openshift-default-6c86dc794f-jmfmb\" (UID: \"aba9a06e-b5d6-491f-b9a3-34ebd927d4aa\") " pod="openshift-ingress/e2e-shared-a-w0-50e623-openshift-default-6c86dc794f-jmfmb" Nov 03 01:07:04.634988 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:07:04.634886 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-psmpl\" (UniqueName: \"kubernetes.io/projected/64952141-7a68-4e46-8982-3db9402620e3-kube-api-access-psmpl\") pod \"e2e-shared-a-w0-50e623-openshift-default-6c86dc794f-5m78z\" (UID: \"64952141-7a68-4e46-8982-3db9402620e3\") " pod="openshift-ingress/e2e-shared-a-w0-50e623-openshift-default-6c86dc794f-5m78z" Nov 03 01:07:04.635052 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:07:04.635018 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9w5qh\" (UniqueName: \"kubernetes.io/projected/24fb3670-ec70-492c-a573-4cbe174f1ccf-kube-api-access-9w5qh\") pod \"auth-test-model-020af9-kserve-844df75d9c-ljr9s\" (UID: \"24fb3670-ec70-492c-a573-4cbe174f1ccf\") " pod="ai-tenant-e2e-shared-b-w0-020af9/auth-test-model-020af9-kserve-844df75d9c-ljr9s" Nov 03 01:07:04.635099 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:07:04.635057 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/aba9a06e-b5d6-491f-b9a3-34ebd927d4aa-credential-socket\") pod \"e2e-shared-a-w0-50e623-openshift-default-6c86dc794f-jmfmb\" (UID: \"aba9a06e-b5d6-491f-b9a3-34ebd927d4aa\") " pod="openshift-ingress/e2e-shared-a-w0-50e623-openshift-default-6c86dc794f-jmfmb" Nov 03 01:07:04.635099 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:07:04.635092 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/24fb3670-ec70-492c-a573-4cbe174f1ccf-tls-certs\") pod \"auth-test-model-020af9-kserve-844df75d9c-ljr9s\" (UID: \"24fb3670-ec70-492c-a573-4cbe174f1ccf\") " pod="ai-tenant-e2e-shared-b-w0-020af9/auth-test-model-020af9-kserve-844df75d9c-ljr9s" Nov 03 01:07:04.635171 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:07:04.635107 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/aba9a06e-b5d6-491f-b9a3-34ebd927d4aa-workload-socket\") pod \"e2e-shared-a-w0-50e623-openshift-default-6c86dc794f-jmfmb\" (UID: \"aba9a06e-b5d6-491f-b9a3-34ebd927d4aa\") " pod="openshift-ingress/e2e-shared-a-w0-50e623-openshift-default-6c86dc794f-jmfmb" Nov 03 01:07:04.635171 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:07:04.635155 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/aba9a06e-b5d6-491f-b9a3-34ebd927d4aa-istio-token\") pod \"e2e-shared-a-w0-50e623-openshift-default-6c86dc794f-jmfmb\" (UID: \"aba9a06e-b5d6-491f-b9a3-34ebd927d4aa\") " pod="openshift-ingress/e2e-shared-a-w0-50e623-openshift-default-6c86dc794f-jmfmb" Nov 03 01:07:04.635242 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:07:04.635219 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/24fb3670-ec70-492c-a573-4cbe174f1ccf-model-cache\") pod \"auth-test-model-020af9-kserve-844df75d9c-ljr9s\" (UID: \"24fb3670-ec70-492c-a573-4cbe174f1ccf\") " pod="ai-tenant-e2e-shared-b-w0-020af9/auth-test-model-020af9-kserve-844df75d9c-ljr9s" Nov 03 01:07:04.635278 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:07:04.635255 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/64952141-7a68-4e46-8982-3db9402620e3-credential-socket\") pod \"e2e-shared-a-w0-50e623-openshift-default-6c86dc794f-5m78z\" (UID: \"64952141-7a68-4e46-8982-3db9402620e3\") " pod="openshift-ingress/e2e-shared-a-w0-50e623-openshift-default-6c86dc794f-5m78z" Nov 03 01:07:04.635310 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:07:04.635301 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/aba9a06e-b5d6-491f-b9a3-34ebd927d4aa-istio-envoy\") pod \"e2e-shared-a-w0-50e623-openshift-default-6c86dc794f-jmfmb\" (UID: \"aba9a06e-b5d6-491f-b9a3-34ebd927d4aa\") " pod="openshift-ingress/e2e-shared-a-w0-50e623-openshift-default-6c86dc794f-jmfmb" Nov 03 01:07:04.635383 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:07:04.635364 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/24fb3670-ec70-492c-a573-4cbe174f1ccf-tmp-dir\") pod \"auth-test-model-020af9-kserve-844df75d9c-ljr9s\" (UID: \"24fb3670-ec70-492c-a573-4cbe174f1ccf\") " pod="ai-tenant-e2e-shared-b-w0-020af9/auth-test-model-020af9-kserve-844df75d9c-ljr9s" Nov 03 01:07:04.635422 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:07:04.635406 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/24fb3670-ec70-492c-a573-4cbe174f1ccf-home\") pod \"auth-test-model-020af9-kserve-844df75d9c-ljr9s\" (UID: \"24fb3670-ec70-492c-a573-4cbe174f1ccf\") " pod="ai-tenant-e2e-shared-b-w0-020af9/auth-test-model-020af9-kserve-844df75d9c-ljr9s" Nov 03 01:07:04.635517 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:07:04.635428 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/64952141-7a68-4e46-8982-3db9402620e3-istio-podinfo\") pod \"e2e-shared-a-w0-50e623-openshift-default-6c86dc794f-5m78z\" (UID: \"64952141-7a68-4e46-8982-3db9402620e3\") " pod="openshift-ingress/e2e-shared-a-w0-50e623-openshift-default-6c86dc794f-5m78z" Nov 03 01:07:04.635517 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:07:04.635452 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/24fb3670-ec70-492c-a573-4cbe174f1ccf-dshm\") pod \"auth-test-model-020af9-kserve-844df75d9c-ljr9s\" (UID: \"24fb3670-ec70-492c-a573-4cbe174f1ccf\") " pod="ai-tenant-e2e-shared-b-w0-020af9/auth-test-model-020af9-kserve-844df75d9c-ljr9s" Nov 03 01:07:04.635517 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:07:04.635486 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j9s9s\" (UniqueName: \"kubernetes.io/projected/aba9a06e-b5d6-491f-b9a3-34ebd927d4aa-kube-api-access-j9s9s\") pod \"e2e-shared-a-w0-50e623-openshift-default-6c86dc794f-jmfmb\" (UID: \"aba9a06e-b5d6-491f-b9a3-34ebd927d4aa\") " pod="openshift-ingress/e2e-shared-a-w0-50e623-openshift-default-6c86dc794f-jmfmb" Nov 03 01:07:04.635641 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:07:04.635532 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/64952141-7a68-4e46-8982-3db9402620e3-istiod-ca-cert\") pod \"e2e-shared-a-w0-50e623-openshift-default-6c86dc794f-5m78z\" (UID: \"64952141-7a68-4e46-8982-3db9402620e3\") " pod="openshift-ingress/e2e-shared-a-w0-50e623-openshift-default-6c86dc794f-5m78z" Nov 03 01:07:04.635641 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:07:04.635566 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/64952141-7a68-4e46-8982-3db9402620e3-workload-socket\") pod \"e2e-shared-a-w0-50e623-openshift-default-6c86dc794f-5m78z\" (UID: \"64952141-7a68-4e46-8982-3db9402620e3\") " pod="openshift-ingress/e2e-shared-a-w0-50e623-openshift-default-6c86dc794f-5m78z" Nov 03 01:07:04.635641 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:07:04.635582 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/aba9a06e-b5d6-491f-b9a3-34ebd927d4aa-istio-data\") pod \"e2e-shared-a-w0-50e623-openshift-default-6c86dc794f-jmfmb\" (UID: \"aba9a06e-b5d6-491f-b9a3-34ebd927d4aa\") " pod="openshift-ingress/e2e-shared-a-w0-50e623-openshift-default-6c86dc794f-jmfmb" Nov 03 01:07:04.635641 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:07:04.635624 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/64952141-7a68-4e46-8982-3db9402620e3-workload-certs\") pod \"e2e-shared-a-w0-50e623-openshift-default-6c86dc794f-5m78z\" (UID: \"64952141-7a68-4e46-8982-3db9402620e3\") " pod="openshift-ingress/e2e-shared-a-w0-50e623-openshift-default-6c86dc794f-5m78z" Nov 03 01:07:04.635773 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:07:04.635646 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/aba9a06e-b5d6-491f-b9a3-34ebd927d4aa-workload-certs\") pod \"e2e-shared-a-w0-50e623-openshift-default-6c86dc794f-jmfmb\" (UID: \"aba9a06e-b5d6-491f-b9a3-34ebd927d4aa\") " pod="openshift-ingress/e2e-shared-a-w0-50e623-openshift-default-6c86dc794f-jmfmb" Nov 03 01:07:04.635773 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:07:04.635664 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/aba9a06e-b5d6-491f-b9a3-34ebd927d4aa-istio-podinfo\") pod \"e2e-shared-a-w0-50e623-openshift-default-6c86dc794f-jmfmb\" (UID: \"aba9a06e-b5d6-491f-b9a3-34ebd927d4aa\") " pod="openshift-ingress/e2e-shared-a-w0-50e623-openshift-default-6c86dc794f-jmfmb" Nov 03 01:07:04.635773 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:07:04.635706 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/64952141-7a68-4e46-8982-3db9402620e3-istio-envoy\") pod \"e2e-shared-a-w0-50e623-openshift-default-6c86dc794f-5m78z\" (UID: \"64952141-7a68-4e46-8982-3db9402620e3\") " pod="openshift-ingress/e2e-shared-a-w0-50e623-openshift-default-6c86dc794f-5m78z" Nov 03 01:07:04.635773 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:07:04.635724 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/64952141-7a68-4e46-8982-3db9402620e3-istio-token\") pod \"e2e-shared-a-w0-50e623-openshift-default-6c86dc794f-5m78z\" (UID: \"64952141-7a68-4e46-8982-3db9402620e3\") " pod="openshift-ingress/e2e-shared-a-w0-50e623-openshift-default-6c86dc794f-5m78z" Nov 03 01:07:04.635899 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:07:04.635807 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/64952141-7a68-4e46-8982-3db9402620e3-istio-data\") pod \"e2e-shared-a-w0-50e623-openshift-default-6c86dc794f-5m78z\" (UID: \"64952141-7a68-4e46-8982-3db9402620e3\") " pod="openshift-ingress/e2e-shared-a-w0-50e623-openshift-default-6c86dc794f-5m78z" Nov 03 01:07:04.637117 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:07:04.637080 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/24fb3670-ec70-492c-a573-4cbe174f1ccf-model-cache\") pod \"auth-test-model-020af9-kserve-844df75d9c-ljr9s\" (UID: \"24fb3670-ec70-492c-a573-4cbe174f1ccf\") " pod="ai-tenant-e2e-shared-b-w0-020af9/auth-test-model-020af9-kserve-844df75d9c-ljr9s" Nov 03 01:07:04.637548 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:07:04.637500 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/24fb3670-ec70-492c-a573-4cbe174f1ccf-home\") pod \"auth-test-model-020af9-kserve-844df75d9c-ljr9s\" (UID: \"24fb3670-ec70-492c-a573-4cbe174f1ccf\") " pod="ai-tenant-e2e-shared-b-w0-020af9/auth-test-model-020af9-kserve-844df75d9c-ljr9s" Nov 03 01:07:04.639349 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:07:04.639329 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/24fb3670-ec70-492c-a573-4cbe174f1ccf-tls-certs\") pod \"auth-test-model-020af9-kserve-844df75d9c-ljr9s\" (UID: \"24fb3670-ec70-492c-a573-4cbe174f1ccf\") " pod="ai-tenant-e2e-shared-b-w0-020af9/auth-test-model-020af9-kserve-844df75d9c-ljr9s" Nov 03 01:07:04.639742 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:07:04.639726 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/24fb3670-ec70-492c-a573-4cbe174f1ccf-tmp-dir\") pod \"auth-test-model-020af9-kserve-844df75d9c-ljr9s\" (UID: \"24fb3670-ec70-492c-a573-4cbe174f1ccf\") " pod="ai-tenant-e2e-shared-b-w0-020af9/auth-test-model-020af9-kserve-844df75d9c-ljr9s" Nov 03 01:07:04.640661 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:07:04.640647 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/24fb3670-ec70-492c-a573-4cbe174f1ccf-dshm\") pod \"auth-test-model-020af9-kserve-844df75d9c-ljr9s\" (UID: \"24fb3670-ec70-492c-a573-4cbe174f1ccf\") " pod="ai-tenant-e2e-shared-b-w0-020af9/auth-test-model-020af9-kserve-844df75d9c-ljr9s" Nov 03 01:07:04.705932 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:07:04.689237 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9w5qh\" (UniqueName: \"kubernetes.io/projected/24fb3670-ec70-492c-a573-4cbe174f1ccf-kube-api-access-9w5qh\") pod \"auth-test-model-020af9-kserve-844df75d9c-ljr9s\" (UID: \"24fb3670-ec70-492c-a573-4cbe174f1ccf\") " pod="ai-tenant-e2e-shared-b-w0-020af9/auth-test-model-020af9-kserve-844df75d9c-ljr9s" Nov 03 01:07:04.726532 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:07:04.726497 4939 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w1-4af1f7-openshift-default-745ff98565-ktw4m" Nov 03 01:07:04.777270 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:07:04.777167 4939 patch_prober.go:29] interesting pod/e2e-shared-b-w1-4af1f7-openshift-default-745ff98565-ktw4m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.197:15021/healthz/ready\": dial tcp 10.132.0.197:15021: connect: connection refused" start-of-body= Nov 03 01:07:04.777644 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:07:04.777601 4939 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-4af1f7-openshift-default-745ff98565-ktw4m" podUID="d2965442-bcbd-4959-b60a-08074c5aead5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.197:15021/healthz/ready\": dial tcp 10.132.0.197:15021: connect: connection refused" Nov 03 01:07:04.859107 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:07:04.859068 4939 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w0-020af9/auth-test-model-020af9-kserve-844df75d9c-ljr9s" Nov 03 01:07:05.035821 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:07:05.035729 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j9s9s\" (UniqueName: \"kubernetes.io/projected/aba9a06e-b5d6-491f-b9a3-34ebd927d4aa-kube-api-access-j9s9s\") pod \"e2e-shared-a-w0-50e623-openshift-default-6c86dc794f-jmfmb\" (UID: \"aba9a06e-b5d6-491f-b9a3-34ebd927d4aa\") " pod="openshift-ingress/e2e-shared-a-w0-50e623-openshift-default-6c86dc794f-jmfmb" Nov 03 01:07:05.036150 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:07:05.036127 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/64952141-7a68-4e46-8982-3db9402620e3-istiod-ca-cert\") pod \"e2e-shared-a-w0-50e623-openshift-default-6c86dc794f-5m78z\" (UID: \"64952141-7a68-4e46-8982-3db9402620e3\") " pod="openshift-ingress/e2e-shared-a-w0-50e623-openshift-default-6c86dc794f-5m78z" Nov 03 01:07:05.036322 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:07:05.036305 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/64952141-7a68-4e46-8982-3db9402620e3-workload-socket\") pod \"e2e-shared-a-w0-50e623-openshift-default-6c86dc794f-5m78z\" (UID: \"64952141-7a68-4e46-8982-3db9402620e3\") " pod="openshift-ingress/e2e-shared-a-w0-50e623-openshift-default-6c86dc794f-5m78z" Nov 03 01:07:05.036454 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:07:05.036437 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/aba9a06e-b5d6-491f-b9a3-34ebd927d4aa-istio-data\") pod \"e2e-shared-a-w0-50e623-openshift-default-6c86dc794f-jmfmb\" (UID: \"aba9a06e-b5d6-491f-b9a3-34ebd927d4aa\") " pod="openshift-ingress/e2e-shared-a-w0-50e623-openshift-default-6c86dc794f-jmfmb" Nov 03 01:07:05.036751 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:07:05.036731 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/64952141-7a68-4e46-8982-3db9402620e3-workload-certs\") pod \"e2e-shared-a-w0-50e623-openshift-default-6c86dc794f-5m78z\" (UID: \"64952141-7a68-4e46-8982-3db9402620e3\") " pod="openshift-ingress/e2e-shared-a-w0-50e623-openshift-default-6c86dc794f-5m78z" Nov 03 01:07:05.036936 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:07:05.036917 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/aba9a06e-b5d6-491f-b9a3-34ebd927d4aa-workload-certs\") pod \"e2e-shared-a-w0-50e623-openshift-default-6c86dc794f-jmfmb\" (UID: \"aba9a06e-b5d6-491f-b9a3-34ebd927d4aa\") " pod="openshift-ingress/e2e-shared-a-w0-50e623-openshift-default-6c86dc794f-jmfmb" Nov 03 01:07:05.037084 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:07:05.037065 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/aba9a06e-b5d6-491f-b9a3-34ebd927d4aa-istio-podinfo\") pod \"e2e-shared-a-w0-50e623-openshift-default-6c86dc794f-jmfmb\" (UID: \"aba9a06e-b5d6-491f-b9a3-34ebd927d4aa\") " pod="openshift-ingress/e2e-shared-a-w0-50e623-openshift-default-6c86dc794f-jmfmb" Nov 03 01:07:05.037239 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:07:05.037222 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/64952141-7a68-4e46-8982-3db9402620e3-istio-envoy\") pod \"e2e-shared-a-w0-50e623-openshift-default-6c86dc794f-5m78z\" (UID: \"64952141-7a68-4e46-8982-3db9402620e3\") " pod="openshift-ingress/e2e-shared-a-w0-50e623-openshift-default-6c86dc794f-5m78z" Nov 03 01:07:05.037364 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:07:05.037348 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/64952141-7a68-4e46-8982-3db9402620e3-istio-token\") pod \"e2e-shared-a-w0-50e623-openshift-default-6c86dc794f-5m78z\" (UID: \"64952141-7a68-4e46-8982-3db9402620e3\") " pod="openshift-ingress/e2e-shared-a-w0-50e623-openshift-default-6c86dc794f-5m78z" Nov 03 01:07:05.037599 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:07:05.037577 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/64952141-7a68-4e46-8982-3db9402620e3-istio-data\") pod \"e2e-shared-a-w0-50e623-openshift-default-6c86dc794f-5m78z\" (UID: \"64952141-7a68-4e46-8982-3db9402620e3\") " pod="openshift-ingress/e2e-shared-a-w0-50e623-openshift-default-6c86dc794f-5m78z" Nov 03 01:07:05.037834 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:07:05.037807 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/aba9a06e-b5d6-491f-b9a3-34ebd927d4aa-istiod-ca-cert\") pod \"e2e-shared-a-w0-50e623-openshift-default-6c86dc794f-jmfmb\" (UID: \"aba9a06e-b5d6-491f-b9a3-34ebd927d4aa\") " pod="openshift-ingress/e2e-shared-a-w0-50e623-openshift-default-6c86dc794f-jmfmb" Nov 03 01:07:05.038046 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:07:05.038027 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-psmpl\" (UniqueName: \"kubernetes.io/projected/64952141-7a68-4e46-8982-3db9402620e3-kube-api-access-psmpl\") pod \"e2e-shared-a-w0-50e623-openshift-default-6c86dc794f-5m78z\" (UID: \"64952141-7a68-4e46-8982-3db9402620e3\") " pod="openshift-ingress/e2e-shared-a-w0-50e623-openshift-default-6c86dc794f-5m78z" Nov 03 01:07:05.038304 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:07:05.038228 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/aba9a06e-b5d6-491f-b9a3-34ebd927d4aa-credential-socket\") pod \"e2e-shared-a-w0-50e623-openshift-default-6c86dc794f-jmfmb\" (UID: \"aba9a06e-b5d6-491f-b9a3-34ebd927d4aa\") " pod="openshift-ingress/e2e-shared-a-w0-50e623-openshift-default-6c86dc794f-jmfmb" Nov 03 01:07:05.038456 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:07:05.038437 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/aba9a06e-b5d6-491f-b9a3-34ebd927d4aa-workload-socket\") pod \"e2e-shared-a-w0-50e623-openshift-default-6c86dc794f-jmfmb\" (UID: \"aba9a06e-b5d6-491f-b9a3-34ebd927d4aa\") " pod="openshift-ingress/e2e-shared-a-w0-50e623-openshift-default-6c86dc794f-jmfmb" Nov 03 01:07:05.038606 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:07:05.038588 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/aba9a06e-b5d6-491f-b9a3-34ebd927d4aa-istio-token\") pod \"e2e-shared-a-w0-50e623-openshift-default-6c86dc794f-jmfmb\" (UID: \"aba9a06e-b5d6-491f-b9a3-34ebd927d4aa\") " pod="openshift-ingress/e2e-shared-a-w0-50e623-openshift-default-6c86dc794f-jmfmb" Nov 03 01:07:05.038832 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:07:05.038804 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/64952141-7a68-4e46-8982-3db9402620e3-credential-socket\") pod \"e2e-shared-a-w0-50e623-openshift-default-6c86dc794f-5m78z\" (UID: \"64952141-7a68-4e46-8982-3db9402620e3\") " pod="openshift-ingress/e2e-shared-a-w0-50e623-openshift-default-6c86dc794f-5m78z" Nov 03 01:07:05.039057 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:07:05.039037 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/aba9a06e-b5d6-491f-b9a3-34ebd927d4aa-istio-envoy\") pod \"e2e-shared-a-w0-50e623-openshift-default-6c86dc794f-jmfmb\" (UID: \"aba9a06e-b5d6-491f-b9a3-34ebd927d4aa\") " pod="openshift-ingress/e2e-shared-a-w0-50e623-openshift-default-6c86dc794f-jmfmb" Nov 03 01:07:05.039288 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:07:05.039267 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/64952141-7a68-4e46-8982-3db9402620e3-workload-certs\") pod \"e2e-shared-a-w0-50e623-openshift-default-6c86dc794f-5m78z\" (UID: \"64952141-7a68-4e46-8982-3db9402620e3\") " pod="openshift-ingress/e2e-shared-a-w0-50e623-openshift-default-6c86dc794f-5m78z" Nov 03 01:07:05.039529 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:07:05.039508 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/64952141-7a68-4e46-8982-3db9402620e3-istio-podinfo\") pod \"e2e-shared-a-w0-50e623-openshift-default-6c86dc794f-5m78z\" (UID: \"64952141-7a68-4e46-8982-3db9402620e3\") " pod="openshift-ingress/e2e-shared-a-w0-50e623-openshift-default-6c86dc794f-5m78z" Nov 03 01:07:05.095934 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:07:05.060950 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/64952141-7a68-4e46-8982-3db9402620e3-istio-data\") pod \"e2e-shared-a-w0-50e623-openshift-default-6c86dc794f-5m78z\" (UID: \"64952141-7a68-4e46-8982-3db9402620e3\") " pod="openshift-ingress/e2e-shared-a-w0-50e623-openshift-default-6c86dc794f-5m78z" Nov 03 01:07:05.095934 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:07:05.063200 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/aba9a06e-b5d6-491f-b9a3-34ebd927d4aa-istio-data\") pod \"e2e-shared-a-w0-50e623-openshift-default-6c86dc794f-jmfmb\" (UID: \"aba9a06e-b5d6-491f-b9a3-34ebd927d4aa\") " pod="openshift-ingress/e2e-shared-a-w0-50e623-openshift-default-6c86dc794f-jmfmb" Nov 03 01:07:05.095934 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:07:05.063696 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/64952141-7a68-4e46-8982-3db9402620e3-credential-socket\") pod \"e2e-shared-a-w0-50e623-openshift-default-6c86dc794f-5m78z\" (UID: \"64952141-7a68-4e46-8982-3db9402620e3\") " pod="openshift-ingress/e2e-shared-a-w0-50e623-openshift-default-6c86dc794f-5m78z" Nov 03 01:07:05.095934 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:07:05.063970 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/aba9a06e-b5d6-491f-b9a3-34ebd927d4aa-credential-socket\") pod \"e2e-shared-a-w0-50e623-openshift-default-6c86dc794f-jmfmb\" (UID: \"aba9a06e-b5d6-491f-b9a3-34ebd927d4aa\") " pod="openshift-ingress/e2e-shared-a-w0-50e623-openshift-default-6c86dc794f-jmfmb" Nov 03 01:07:05.095934 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:07:05.064137 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/64952141-7a68-4e46-8982-3db9402620e3-istiod-ca-cert\") pod \"e2e-shared-a-w0-50e623-openshift-default-6c86dc794f-5m78z\" (UID: \"64952141-7a68-4e46-8982-3db9402620e3\") " pod="openshift-ingress/e2e-shared-a-w0-50e623-openshift-default-6c86dc794f-5m78z" Nov 03 01:07:05.095934 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:07:05.064235 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/aba9a06e-b5d6-491f-b9a3-34ebd927d4aa-workload-certs\") pod \"e2e-shared-a-w0-50e623-openshift-default-6c86dc794f-jmfmb\" (UID: \"aba9a06e-b5d6-491f-b9a3-34ebd927d4aa\") " pod="openshift-ingress/e2e-shared-a-w0-50e623-openshift-default-6c86dc794f-jmfmb" Nov 03 01:07:05.095934 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:07:05.064407 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/64952141-7a68-4e46-8982-3db9402620e3-workload-socket\") pod \"e2e-shared-a-w0-50e623-openshift-default-6c86dc794f-5m78z\" (UID: \"64952141-7a68-4e46-8982-3db9402620e3\") " pod="openshift-ingress/e2e-shared-a-w0-50e623-openshift-default-6c86dc794f-5m78z" Nov 03 01:07:05.095934 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:07:05.064978 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/aba9a06e-b5d6-491f-b9a3-34ebd927d4aa-istiod-ca-cert\") pod \"e2e-shared-a-w0-50e623-openshift-default-6c86dc794f-jmfmb\" (UID: \"aba9a06e-b5d6-491f-b9a3-34ebd927d4aa\") " pod="openshift-ingress/e2e-shared-a-w0-50e623-openshift-default-6c86dc794f-jmfmb" Nov 03 01:07:05.095934 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:07:05.089134 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/aba9a06e-b5d6-491f-b9a3-34ebd927d4aa-workload-socket\") pod \"e2e-shared-a-w0-50e623-openshift-default-6c86dc794f-jmfmb\" (UID: \"aba9a06e-b5d6-491f-b9a3-34ebd927d4aa\") " pod="openshift-ingress/e2e-shared-a-w0-50e623-openshift-default-6c86dc794f-jmfmb" Nov 03 01:07:05.116116 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:07:05.113027 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/aba9a06e-b5d6-491f-b9a3-34ebd927d4aa-istio-podinfo\") pod \"e2e-shared-a-w0-50e623-openshift-default-6c86dc794f-jmfmb\" (UID: \"aba9a06e-b5d6-491f-b9a3-34ebd927d4aa\") " pod="openshift-ingress/e2e-shared-a-w0-50e623-openshift-default-6c86dc794f-jmfmb" Nov 03 01:07:05.119416 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:07:05.119380 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/aba9a06e-b5d6-491f-b9a3-34ebd927d4aa-istio-envoy\") pod \"e2e-shared-a-w0-50e623-openshift-default-6c86dc794f-jmfmb\" (UID: \"aba9a06e-b5d6-491f-b9a3-34ebd927d4aa\") " pod="openshift-ingress/e2e-shared-a-w0-50e623-openshift-default-6c86dc794f-jmfmb" Nov 03 01:07:05.120320 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:07:05.120216 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/64952141-7a68-4e46-8982-3db9402620e3-istio-podinfo\") pod \"e2e-shared-a-w0-50e623-openshift-default-6c86dc794f-5m78z\" (UID: \"64952141-7a68-4e46-8982-3db9402620e3\") " pod="openshift-ingress/e2e-shared-a-w0-50e623-openshift-default-6c86dc794f-5m78z" Nov 03 01:07:05.121102 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:07:05.121073 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-psmpl\" (UniqueName: \"kubernetes.io/projected/64952141-7a68-4e46-8982-3db9402620e3-kube-api-access-psmpl\") pod \"e2e-shared-a-w0-50e623-openshift-default-6c86dc794f-5m78z\" (UID: \"64952141-7a68-4e46-8982-3db9402620e3\") " pod="openshift-ingress/e2e-shared-a-w0-50e623-openshift-default-6c86dc794f-5m78z" Nov 03 01:07:05.123949 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:07:05.121939 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/64952141-7a68-4e46-8982-3db9402620e3-istio-token\") pod \"e2e-shared-a-w0-50e623-openshift-default-6c86dc794f-5m78z\" (UID: \"64952141-7a68-4e46-8982-3db9402620e3\") " pod="openshift-ingress/e2e-shared-a-w0-50e623-openshift-default-6c86dc794f-5m78z" Nov 03 01:07:05.123949 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:07:05.123357 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/64952141-7a68-4e46-8982-3db9402620e3-istio-envoy\") pod \"e2e-shared-a-w0-50e623-openshift-default-6c86dc794f-5m78z\" (UID: \"64952141-7a68-4e46-8982-3db9402620e3\") " pod="openshift-ingress/e2e-shared-a-w0-50e623-openshift-default-6c86dc794f-5m78z" Nov 03 01:07:05.130956 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:07:05.130890 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/aba9a06e-b5d6-491f-b9a3-34ebd927d4aa-istio-token\") pod \"e2e-shared-a-w0-50e623-openshift-default-6c86dc794f-jmfmb\" (UID: \"aba9a06e-b5d6-491f-b9a3-34ebd927d4aa\") " pod="openshift-ingress/e2e-shared-a-w0-50e623-openshift-default-6c86dc794f-jmfmb" Nov 03 01:07:05.140207 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:07:05.140181 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-j9s9s\" (UniqueName: \"kubernetes.io/projected/aba9a06e-b5d6-491f-b9a3-34ebd927d4aa-kube-api-access-j9s9s\") pod \"e2e-shared-a-w0-50e623-openshift-default-6c86dc794f-jmfmb\" (UID: \"aba9a06e-b5d6-491f-b9a3-34ebd927d4aa\") " pod="openshift-ingress/e2e-shared-a-w0-50e623-openshift-default-6c86dc794f-jmfmb" Nov 03 01:07:05.215800 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:07:05.215764 4939 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w0-50e623-openshift-default-6c86dc794f-5m78z" Nov 03 01:07:05.231365 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:07:05.231332 4939 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w0-50e623-openshift-default-6c86dc794f-jmfmb" Nov 03 01:07:05.316183 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:07:05.301649 4939 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w0-020af9-openshift-default-d77df45d5-mpgn5" Nov 03 01:07:05.316183 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:07:05.303771 4939 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w0-020af9-openshift-default-d77df45d5-mpgn5" Nov 03 01:07:05.316183 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:07:05.305222 4939 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w0-020af9-openshift-default-d77df45d5-mpgn5" Nov 03 01:07:05.618939 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:07:05.617428 4939 patch_prober.go:29] interesting pod/e2e-shared-a-w3-717217-openshift-default-578c69554c-5xzz5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.196:15021/healthz/ready\": dial tcp 10.132.0.196:15021: connect: connection refused" start-of-body= Nov 03 01:07:05.618939 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:07:05.617492 4939 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-717217-openshift-default-578c69554c-5xzz5" podUID="b22388ea-ae77-4ad1-bfff-9aeb20f5bbbd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.196:15021/healthz/ready\": dial tcp 10.132.0.196:15021: connect: connection refused" Nov 03 01:07:05.744021 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:07:05.742900 4939 patch_prober.go:29] interesting pod/e2e-shared-b-w1-4af1f7-openshift-default-745ff98565-ktw4m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.197:15021/healthz/ready\": dial tcp 10.132.0.197:15021: connect: connection refused" start-of-body= Nov 03 01:07:05.744021 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:07:05.742977 4939 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-4af1f7-openshift-default-745ff98565-ktw4m" podUID="d2965442-bcbd-4959-b60a-08074c5aead5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.197:15021/healthz/ready\": dial tcp 10.132.0.197:15021: connect: connection refused" Nov 03 01:07:05.760586 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:07:05.744865 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:07:05.827337 ip-10-0-140-107 kubenswrapper[4939]: W1103 01:07:05.826927 4939 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod24fb3670_ec70_492c_a573_4cbe174f1ccf.slice/crio-a450effc06c776e21f25d5c7d0c5b30fcb4c07f7e9141688d1992124b5ad59b3 WatchSource:0}: Error finding container a450effc06c776e21f25d5c7d0c5b30fcb4c07f7e9141688d1992124b5ad59b3: Status 404 returned error can't find the container with id a450effc06c776e21f25d5c7d0c5b30fcb4c07f7e9141688d1992124b5ad59b3 Nov 03 01:07:05.886289 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:07:05.852836 4939 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w0-020af9/auth-test-model-020af9-kserve-844df75d9c-ljr9s"] Nov 03 01:07:05.945019 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:07:05.944936 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:07:06.110935 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:07:06.074917 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:07:06.351998 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:07:06.342496 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:07:06.593068 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:07:06.593003 4939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w0-020af9/auth-test-model-020af9-kserve-844df75d9c-ljr9s" event={"ID":"24fb3670-ec70-492c-a573-4cbe174f1ccf","Type":"ContainerStarted","Data":"a450effc06c776e21f25d5c7d0c5b30fcb4c07f7e9141688d1992124b5ad59b3"} Nov 03 01:07:06.615727 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:07:06.615633 4939 patch_prober.go:29] interesting pod/e2e-shared-a-w3-717217-openshift-default-578c69554c-5xzz5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.196:15021/healthz/ready\": dial tcp 10.132.0.196:15021: connect: connection refused" start-of-body= Nov 03 01:07:06.615727 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:07:06.615697 4939 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-717217-openshift-default-578c69554c-5xzz5" podUID="b22388ea-ae77-4ad1-bfff-9aeb20f5bbbd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.196:15021/healthz/ready\": dial tcp 10.132.0.196:15021: connect: connection refused" Nov 03 01:07:06.721936 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:07:06.705928 4939 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w0-50e623-openshift-default-6c86dc794f-jmfmb"] Nov 03 01:07:06.777930 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:07:06.741218 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:07:06.786646 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:07:06.778395 4939 patch_prober.go:29] interesting pod/e2e-shared-b-w1-4af1f7-openshift-default-745ff98565-ktw4m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.197:15021/healthz/ready\": dial tcp 10.132.0.197:15021: connect: connection refused" start-of-body= Nov 03 01:07:06.786646 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:07:06.778458 4939 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-4af1f7-openshift-default-745ff98565-ktw4m" podUID="d2965442-bcbd-4959-b60a-08074c5aead5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.197:15021/healthz/ready\": dial tcp 10.132.0.197:15021: connect: connection refused" Nov 03 01:07:06.804034 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:07:06.803247 4939 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-w3-556c14/auth-test-model-556c14-kserve-8d5cd469-t6tzc" Nov 03 01:07:06.841535 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:07:06.835489 4939 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w0-50e623-openshift-default-6c86dc794f-5m78z"] Nov 03 01:07:07.212936 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:07:07.159640 4939 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 01:07:07.212936 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:07:07.159762 4939 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 01:07:07.212936 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:07:07.159821 4939 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 01:07:07.277355 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:07:07.277301 4939 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 01:07:07.307932 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:07:07.298899 4939 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 01:07:07.307932 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:07:07.298998 4939 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 01:07:07.309258 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:07:07.309231 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:07:07.410683 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:07:07.410359 4939 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-w3-556c14/auth-test-model-556c14-kserve-8d5cd469-t6tzc" Nov 03 01:07:07.620473 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:07:07.620197 4939 patch_prober.go:29] interesting pod/e2e-shared-a-w3-717217-openshift-default-578c69554c-5xzz5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.196:15021/healthz/ready\": dial tcp 10.132.0.196:15021: connect: connection refused" start-of-body= Nov 03 01:07:07.620473 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:07:07.620237 4939 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-717217-openshift-default-578c69554c-5xzz5" podUID="b22388ea-ae77-4ad1-bfff-9aeb20f5bbbd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.196:15021/healthz/ready\": dial tcp 10.132.0.196:15021: connect: connection refused" Nov 03 01:07:07.622401 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:07:07.622364 4939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w0-50e623-openshift-default-6c86dc794f-5m78z" event={"ID":"64952141-7a68-4e46-8982-3db9402620e3","Type":"ContainerStarted","Data":"b0d0da6db254ebda89724e2ecad9bee3881c3cfec3d6022971f2e987a069127f"} Nov 03 01:07:07.631269 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:07:07.627324 4939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w0-50e623-openshift-default-6c86dc794f-jmfmb" event={"ID":"aba9a06e-b5d6-491f-b9a3-34ebd927d4aa","Type":"ContainerStarted","Data":"368f1834e8a5c7ca07bdaa894b6b966b769f839a8b95940ef3da8a33098f6079"} Nov 03 01:07:07.631269 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:07:07.627384 4939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w0-50e623-openshift-default-6c86dc794f-jmfmb" event={"ID":"aba9a06e-b5d6-491f-b9a3-34ebd927d4aa","Type":"ContainerStarted","Data":"376548ea338ce5a24fe4ae8fee1614dcce70f74d3d149e1268033a9eea4e0fcb"} Nov 03 01:07:07.631473 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:07:07.631442 4939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w0-020af9/auth-test-model-020af9-kserve-844df75d9c-ljr9s" event={"ID":"24fb3670-ec70-492c-a573-4cbe174f1ccf","Type":"ContainerStarted","Data":"ff3dfadc0f8d85525140106a5b520adcebba7b60f58a62b60ea5fc4464232299"} Nov 03 01:07:07.684005 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:07:07.668976 4939 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w0-50e623-openshift-default-6c86dc794f-jmfmb" podStartSLOduration=4.668957322 podStartE2EDuration="4.668957322s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 01:07:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 01:07:07.66889708 +0000 UTC m=+1575.933232026" watchObservedRunningTime="2026-11-03 01:07:07.668957322 +0000 UTC m=+1575.933292268" Nov 03 01:07:07.727937 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:07:07.727358 4939 patch_prober.go:29] interesting pod/e2e-shared-b-w1-4af1f7-openshift-default-745ff98565-ktw4m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.197:15021/healthz/ready\": dial tcp 10.132.0.197:15021: connect: connection refused" start-of-body= Nov 03 01:07:07.727937 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:07:07.727421 4939 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-4af1f7-openshift-default-745ff98565-ktw4m" podUID="d2965442-bcbd-4959-b60a-08074c5aead5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.197:15021/healthz/ready\": dial tcp 10.132.0.197:15021: connect: connection refused" Nov 03 01:07:07.741966 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:07:07.741866 4939 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-w0-020af9/auth-test-model-020af9-kserve-844df75d9c-ljr9s" podStartSLOduration=4.74184683 podStartE2EDuration="4.74184683s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 01:07:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 01:07:07.722864755 +0000 UTC m=+1575.987199701" watchObservedRunningTime="2026-11-03 01:07:07.74184683 +0000 UTC m=+1576.006181776" Nov 03 01:07:07.787005 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:07:07.769750 4939 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-w3-556c14/auth-test-model-556c14-kserve-8d5cd469-t6tzc" Nov 03 01:07:07.817099 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:07:07.817022 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:07:07.900300 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:07:07.897592 4939 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-w0-020af9/auth-test-model-020af9-kserve-844df75d9c-ljr9s" Nov 03 01:07:08.233526 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:07:08.233487 4939 patch_prober.go:29] interesting pod/e2e-shared-a-w0-50e623-openshift-default-6c86dc794f-jmfmb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.200:15021/healthz/ready\": dial tcp 10.132.0.200:15021: connect: connection refused" start-of-body= Nov 03 01:07:08.233822 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:07:08.233792 4939 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w0-50e623-openshift-default-6c86dc794f-jmfmb" podUID="aba9a06e-b5d6-491f-b9a3-34ebd927d4aa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.200:15021/healthz/ready\": dial tcp 10.132.0.200:15021: connect: connection refused" Nov 03 01:07:08.325770 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:07:08.325722 4939 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w0-50e623-openshift-default-6c86dc794f-jmfmb" Nov 03 01:07:08.544130 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:07:08.544036 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:07:08.693944 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:07:08.677067 4939 patch_prober.go:29] interesting pod/e2e-shared-a-w3-717217-openshift-default-578c69554c-5xzz5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.196:15021/healthz/ready\": dial tcp 10.132.0.196:15021: connect: connection refused" start-of-body= Nov 03 01:07:08.693944 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:07:08.677153 4939 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-717217-openshift-default-578c69554c-5xzz5" podUID="b22388ea-ae77-4ad1-bfff-9aeb20f5bbbd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.196:15021/healthz/ready\": dial tcp 10.132.0.196:15021: connect: connection refused" Nov 03 01:07:08.775016 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:07:08.721496 4939 patch_prober.go:29] interesting pod/e2e-shared-b-w1-4af1f7-openshift-default-745ff98565-ktw4m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.197:15021/healthz/ready\": dial tcp 10.132.0.197:15021: connect: connection refused" start-of-body= Nov 03 01:07:08.775016 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:07:08.721571 4939 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-4af1f7-openshift-default-745ff98565-ktw4m" podUID="d2965442-bcbd-4959-b60a-08074c5aead5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.197:15021/healthz/ready\": dial tcp 10.132.0.197:15021: connect: connection refused" Nov 03 01:07:08.842995 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:07:08.834448 4939 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-f7680d/premium-w2-f7680d-kserve-76565b8cf9-hqbbt" podUID="4712e708-18a2-4a13-a12c-a5669397b2c3" containerName="main" probeResult="failure" output="Get \"https://10.132.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 01:07:08.842995 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:07:08.837707 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:07:08.842995 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:07:08.840298 4939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w0-50e623-openshift-default-6c86dc794f-5m78z" event={"ID":"64952141-7a68-4e46-8982-3db9402620e3","Type":"ContainerStarted","Data":"6eea49382c75191aaa1998e3e23c1dea09f9bdc053a349a06b05b89a82ed2443"} Nov 03 01:07:08.870931 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:07:08.868779 4939 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-f7680d/distinct-2-w2-f7680d-kserve-7d7c4564bf-cbv2c" podUID="2de18086-761b-4773-b9fc-15ddc4a220fc" containerName="main" probeResult="failure" output="Get \"https://10.132.0.75:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 01:07:08.894092 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:07:08.877702 4939 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w0-50e623-openshift-default-6c86dc794f-5m78z" podStartSLOduration=5.8776827449999995 podStartE2EDuration="5.877682745s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 01:07:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 01:07:08.864148438 +0000 UTC m=+1577.128483387" watchObservedRunningTime="2026-11-03 01:07:08.877682745 +0000 UTC m=+1577.142017692" Nov 03 01:07:09.237991 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:07:09.234133 4939 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w0-50e623-openshift-default-6c86dc794f-5m78z" Nov 03 01:07:09.287935 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:07:09.264029 4939 patch_prober.go:29] interesting pod/e2e-shared-a-w0-50e623-openshift-default-6c86dc794f-5m78z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.199:15021/healthz/ready\": dial tcp 10.132.0.199:15021: connect: connection refused" start-of-body= Nov 03 01:07:09.287935 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:07:09.264096 4939 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w0-50e623-openshift-default-6c86dc794f-5m78z" podUID="64952141-7a68-4e46-8982-3db9402620e3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.199:15021/healthz/ready\": dial tcp 10.132.0.199:15021: connect: connection refused" Nov 03 01:07:09.287935 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:07:09.264115 4939 patch_prober.go:29] interesting pod/e2e-shared-a-w0-50e623-openshift-default-6c86dc794f-jmfmb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.200:15021/healthz/ready\": dial tcp 10.132.0.200:15021: connect: connection refused" start-of-body= Nov 03 01:07:09.287935 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:07:09.264166 4939 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w0-50e623-openshift-default-6c86dc794f-jmfmb" podUID="aba9a06e-b5d6-491f-b9a3-34ebd927d4aa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.200:15021/healthz/ready\": dial tcp 10.132.0.200:15021: connect: connection refused" Nov 03 01:07:09.642937 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:07:09.637446 4939 patch_prober.go:29] interesting pod/e2e-shared-a-w3-717217-openshift-default-578c69554c-5xzz5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.196:15021/healthz/ready\": dial tcp 10.132.0.196:15021: connect: connection refused" start-of-body= Nov 03 01:07:09.642937 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:07:09.637512 4939 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-717217-openshift-default-578c69554c-5xzz5" podUID="b22388ea-ae77-4ad1-bfff-9aeb20f5bbbd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.196:15021/healthz/ready\": dial tcp 10.132.0.196:15021: connect: connection refused" Nov 03 01:07:09.751099 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:07:09.751053 4939 patch_prober.go:29] interesting pod/e2e-shared-b-w1-4af1f7-openshift-default-745ff98565-ktw4m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.197:15021/healthz/ready\": dial tcp 10.132.0.197:15021: connect: connection refused" start-of-body= Nov 03 01:07:09.751313 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:07:09.751122 4939 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-4af1f7-openshift-default-745ff98565-ktw4m" podUID="d2965442-bcbd-4959-b60a-08074c5aead5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.197:15021/healthz/ready\": dial tcp 10.132.0.197:15021: connect: connection refused" Nov 03 01:07:10.254002 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:07:10.242762 4939 patch_prober.go:29] interesting pod/e2e-shared-a-w0-50e623-openshift-default-6c86dc794f-5m78z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.199:15021/healthz/ready\": dial tcp 10.132.0.199:15021: connect: connection refused" start-of-body= Nov 03 01:07:10.254002 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:07:10.242866 4939 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w0-50e623-openshift-default-6c86dc794f-5m78z" podUID="64952141-7a68-4e46-8982-3db9402620e3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.199:15021/healthz/ready\": dial tcp 10.132.0.199:15021: connect: connection refused" Nov 03 01:07:10.543987 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:07:10.526045 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:07:10.559199 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:07:10.556034 4939 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-shared-a-w3-717217/auth-test-model-717217-kserve-5c9cb7dbd9-vl754" podUID="9634a946-0832-4901-ad01-bbb7a47f4ac7" containerName="main" probeResult="failure" output="Get \"https://10.132.0.191:8000/ready\": context deadline exceeded" Nov 03 01:07:10.607935 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:07:10.579492 4939 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-83755a/distinct-2-w3-83755a-kserve-6cf66b995d-2l47b" podUID="c3b05e88-5d5b-4d03-b28d-832f22e3c868" containerName="main" probeResult="failure" output="Get \"https://10.132.0.72:8000/ready\": context deadline exceeded" Nov 03 01:07:10.796179 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:07:10.796081 4939 patch_prober.go:29] interesting pod/e2e-shared-a-w0-50e623-openshift-default-6c86dc794f-jmfmb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.200:15021/healthz/ready\": dial tcp 10.132.0.200:15021: connect: connection refused" start-of-body= Nov 03 01:07:10.796443 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:07:10.796412 4939 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w0-50e623-openshift-default-6c86dc794f-jmfmb" podUID="aba9a06e-b5d6-491f-b9a3-34ebd927d4aa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.200:15021/healthz/ready\": dial tcp 10.132.0.200:15021: connect: connection refused" Nov 03 01:07:10.917934 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:07:10.907795 4939 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-shared-a-w1-2729f2/auth-test-model-2729f2-kserve-6bb7bb4c97-n9nht" podUID="f66fa3c5-ab64-4027-b004-399379a414f4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.184:8000/ready\": context deadline exceeded" Nov 03 01:07:10.995560 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:07:10.995503 4939 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-2f9a3a/simulator-w0-2f9a3a-kserve-9596bbb8f-2trgz" podUID="207e2207-fb6c-4076-95df-72e95671c261" containerName="main" probeResult="failure" output="Get \"https://10.132.0.61:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Nov 03 01:07:11.039151 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:07:11.039101 4939 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-shared-b-w3-556c14/auth-test-model-556c14-kserve-8d5cd469-t6tzc" podUID="0c4e579c-9fe2-4ec9-b8f4-62758e519980" containerName="main" probeResult="failure" output="Get \"https://10.132.0.195:8000/ready\": context deadline exceeded" Nov 03 01:07:11.058111 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:07:11.057993 4939 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-shared-b-w4-4a697f/auth-test-model-4a697f-kserve-84b7466bd5-48hbr" podUID="18846667-85a0-4bc2-8995-61bfb2ab1c5d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.194:8000/ready\": context deadline exceeded" Nov 03 01:07:11.424181 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:07:11.159326 4939 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-shared-a-w4-6c0ec0/auth-test-model-6c0ec0-kserve-589c79bd69-vznsb" podUID="d21cb247-bacd-4af2-a7b1-d04b975f0993" containerName="main" probeResult="failure" output="Get \"https://10.132.0.186:8000/ready\": context deadline exceeded" Nov 03 01:07:11.565954 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:07:11.190728 4939 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-88bbf7/distinct-2-w1-88bbf7-kserve-5457cdcdbd-l6tfd" podUID="a397931b-faa9-4c2e-8290-491350546663" containerName="main" probeResult="failure" output="Get \"https://10.132.0.85:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 01:07:11.591103 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:07:11.242320 4939 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-f7680d/premium-w2-f7680d-kserve-76565b8cf9-hqbbt" podUID="4712e708-18a2-4a13-a12c-a5669397b2c3" containerName="main" probeResult="failure" output="Get \"https://10.132.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 01:07:11.591771 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:07:11.243075 4939 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-f7680d/distinct-w2-f7680d-kserve-756f874b98-qnpv8" podUID="836753fc-f9ef-4f61-b3fe-e5b92b7ee27b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.70:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 01:07:11.592265 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:07:11.493271 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:07:11.592538 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:07:11.564124 4939 patch_prober.go:29] interesting pod/e2e-shared-a-w0-50e623-openshift-default-6c86dc794f-5m78z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.199:15021/healthz/ready\": dial tcp 10.132.0.199:15021: connect: connection refused" start-of-body= Nov 03 01:07:11.592777 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:07:11.592748 4939 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w0-50e623-openshift-default-6c86dc794f-5m78z" podUID="64952141-7a68-4e46-8982-3db9402620e3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.199:15021/healthz/ready\": dial tcp 10.132.0.199:15021: connect: connection refused" Nov 03 01:07:11.627966 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:07:11.610723 4939 patch_prober.go:29] interesting pod/e2e-shared-a-w0-50e623-openshift-default-6c86dc794f-jmfmb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.200:15021/healthz/ready\": dial tcp 10.132.0.200:15021: connect: connection refused" start-of-body= Nov 03 01:07:11.627966 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:07:11.610807 4939 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w0-50e623-openshift-default-6c86dc794f-jmfmb" podUID="aba9a06e-b5d6-491f-b9a3-34ebd927d4aa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.200:15021/healthz/ready\": dial tcp 10.132.0.200:15021: connect: connection refused" Nov 03 01:07:11.694003 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:07:11.672132 4939 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-shared-a-w3-717217/auth-test-model-717217-kserve-5c9cb7dbd9-vl754" podUID="9634a946-0832-4901-ad01-bbb7a47f4ac7" containerName="main" probeResult="failure" output="Get \"https://10.132.0.191:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 01:07:11.823272 ip-10-0-140-107 kubenswrapper[4939]: E1103 01:07:11.823237 4939 kubelet.go:2711] "Housekeeping took longer than expected" err="housekeeping took too long" expected="1s" actual="1.486s" Nov 03 01:07:11.823588 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:07:11.823572 4939 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w3-717217-openshift-default-578c69554c-5xzz5" Nov 03 01:07:11.823738 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:07:11.823728 4939 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-w3-717217/auth-test-model-717217-kserve-5c9cb7dbd9-vl754" Nov 03 01:07:11.823868 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:07:11.823858 4939 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w1-4af1f7-openshift-default-745ff98565-ktw4m" Nov 03 01:07:11.827975 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:07:11.827733 4939 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w3-717217-openshift-default-578c69554c-5xzz5" Nov 03 01:07:11.828201 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:07:11.828153 4939 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w3-717217-openshift-default-578c69554c-5xzz5" Nov 03 01:07:11.829207 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:07:11.829188 4939 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w1-4af1f7-openshift-default-745ff98565-ktw4m" Nov 03 01:07:11.832842 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:07:11.832759 4939 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w1-4af1f7-openshift-default-745ff98565-ktw4m" Nov 03 01:07:11.837434 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:07:11.837315 4939 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-w3-717217/auth-test-model-717217-kserve-5c9cb7dbd9-vl754" Nov 03 01:07:12.228958 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:07:12.228896 4939 patch_prober.go:29] interesting pod/e2e-shared-a-w0-50e623-openshift-default-6c86dc794f-5m78z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.199:15021/healthz/ready\": dial tcp 10.132.0.199:15021: connect: connection refused" start-of-body= Nov 03 01:07:12.229240 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:07:12.229213 4939 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w0-50e623-openshift-default-6c86dc794f-5m78z" podUID="64952141-7a68-4e46-8982-3db9402620e3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.199:15021/healthz/ready\": dial tcp 10.132.0.199:15021: connect: connection refused" Nov 03 01:07:12.240041 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:07:12.238684 4939 patch_prober.go:29] interesting pod/e2e-shared-a-w0-50e623-openshift-default-6c86dc794f-jmfmb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.200:15021/healthz/ready\": dial tcp 10.132.0.200:15021: connect: connection refused" start-of-body= Nov 03 01:07:12.240041 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:07:12.238748 4939 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w0-50e623-openshift-default-6c86dc794f-jmfmb" podUID="aba9a06e-b5d6-491f-b9a3-34ebd927d4aa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.200:15021/healthz/ready\": dial tcp 10.132.0.200:15021: connect: connection refused" Nov 03 01:07:12.377934 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:07:12.374512 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:07:12.491992 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:07:12.491558 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:07:12.988932 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:07:12.987891 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:07:13.216422 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:07:13.216377 4939 patch_prober.go:29] interesting pod/e2e-shared-a-w0-50e623-openshift-default-6c86dc794f-5m78z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.199:15021/healthz/ready\": dial tcp 10.132.0.199:15021: connect: connection refused" start-of-body= Nov 03 01:07:13.216657 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:07:13.216446 4939 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w0-50e623-openshift-default-6c86dc794f-5m78z" podUID="64952141-7a68-4e46-8982-3db9402620e3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.199:15021/healthz/ready\": dial tcp 10.132.0.199:15021: connect: connection refused" Nov 03 01:07:13.233963 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:07:13.231770 4939 patch_prober.go:29] interesting pod/e2e-shared-a-w0-50e623-openshift-default-6c86dc794f-jmfmb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.200:15021/healthz/ready\": dial tcp 10.132.0.200:15021: connect: connection refused" start-of-body= Nov 03 01:07:13.233963 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:07:13.231832 4939 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w0-50e623-openshift-default-6c86dc794f-jmfmb" podUID="aba9a06e-b5d6-491f-b9a3-34ebd927d4aa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.200:15021/healthz/ready\": dial tcp 10.132.0.200:15021: connect: connection refused" Nov 03 01:07:13.707935 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:07:13.701552 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:07:13.833715 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:07:13.833678 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:07:14.238014 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:07:14.224014 4939 patch_prober.go:29] interesting pod/e2e-shared-a-w0-50e623-openshift-default-6c86dc794f-5m78z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.199:15021/healthz/ready\": dial tcp 10.132.0.199:15021: connect: connection refused" start-of-body= Nov 03 01:07:14.238014 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:07:14.224080 4939 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w0-50e623-openshift-default-6c86dc794f-5m78z" podUID="64952141-7a68-4e46-8982-3db9402620e3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.199:15021/healthz/ready\": dial tcp 10.132.0.199:15021: connect: connection refused" Nov 03 01:07:14.242054 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:07:14.242017 4939 patch_prober.go:29] interesting pod/e2e-shared-a-w0-50e623-openshift-default-6c86dc794f-jmfmb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.200:15021/healthz/ready\": dial tcp 10.132.0.200:15021: connect: connection refused" start-of-body= Nov 03 01:07:14.242339 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:07:14.242310 4939 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w0-50e623-openshift-default-6c86dc794f-jmfmb" podUID="aba9a06e-b5d6-491f-b9a3-34ebd927d4aa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.200:15021/healthz/ready\": dial tcp 10.132.0.200:15021: connect: connection refused" Nov 03 01:07:14.357102 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:07:14.357062 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:07:14.917793 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:07:14.892078 4939 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-w0-020af9/auth-test-model-020af9-kserve-844df75d9c-ljr9s" Nov 03 01:07:14.920606 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:07:14.919891 4939 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-w0-020af9/auth-test-model-020af9-kserve-844df75d9c-ljr9s" Nov 03 01:07:15.217463 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:07:15.217364 4939 patch_prober.go:29] interesting pod/e2e-shared-a-w0-50e623-openshift-default-6c86dc794f-5m78z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.199:15021/healthz/ready\": dial tcp 10.132.0.199:15021: connect: connection refused" start-of-body= Nov 03 01:07:15.217463 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:07:15.217434 4939 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w0-50e623-openshift-default-6c86dc794f-5m78z" podUID="64952141-7a68-4e46-8982-3db9402620e3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.199:15021/healthz/ready\": dial tcp 10.132.0.199:15021: connect: connection refused" Nov 03 01:07:15.232155 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:07:15.232113 4939 patch_prober.go:29] interesting pod/e2e-shared-a-w0-50e623-openshift-default-6c86dc794f-jmfmb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.200:15021/healthz/ready\": dial tcp 10.132.0.200:15021: connect: connection refused" start-of-body= Nov 03 01:07:15.232315 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:07:15.232170 4939 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w0-50e623-openshift-default-6c86dc794f-jmfmb" podUID="aba9a06e-b5d6-491f-b9a3-34ebd927d4aa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.200:15021/healthz/ready\": dial tcp 10.132.0.200:15021: connect: connection refused" Nov 03 01:07:15.647997 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:07:15.647919 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:07:15.933185 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:07:15.933102 4939 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-w0-020af9/auth-test-model-020af9-kserve-844df75d9c-ljr9s" Nov 03 01:07:16.217275 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:07:16.217182 4939 patch_prober.go:29] interesting pod/e2e-shared-a-w0-50e623-openshift-default-6c86dc794f-5m78z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.199:15021/healthz/ready\": dial tcp 10.132.0.199:15021: connect: connection refused" start-of-body= Nov 03 01:07:16.217275 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:07:16.217242 4939 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w0-50e623-openshift-default-6c86dc794f-5m78z" podUID="64952141-7a68-4e46-8982-3db9402620e3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.199:15021/healthz/ready\": dial tcp 10.132.0.199:15021: connect: connection refused" Nov 03 01:07:16.232337 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:07:16.232293 4939 patch_prober.go:29] interesting pod/e2e-shared-a-w0-50e623-openshift-default-6c86dc794f-jmfmb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.200:15021/healthz/ready\": dial tcp 10.132.0.200:15021: connect: connection refused" start-of-body= Nov 03 01:07:16.232527 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:07:16.232361 4939 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w0-50e623-openshift-default-6c86dc794f-jmfmb" podUID="aba9a06e-b5d6-491f-b9a3-34ebd927d4aa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.200:15021/healthz/ready\": dial tcp 10.132.0.200:15021: connect: connection refused" Nov 03 01:07:16.387308 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:07:16.387267 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:07:17.069080 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:07:17.069045 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:07:17.221930 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:07:17.221884 4939 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w0-50e623-openshift-default-6c86dc794f-5m78z" Nov 03 01:07:17.222332 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:07:17.222309 4939 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w0-50e623-openshift-default-6c86dc794f-5m78z" Nov 03 01:07:17.223223 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:07:17.223201 4939 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w0-50e623-openshift-default-6c86dc794f-5m78z" Nov 03 01:07:17.236615 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:07:17.236590 4939 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w0-50e623-openshift-default-6c86dc794f-jmfmb" Nov 03 01:07:17.236876 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:07:17.236858 4939 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w0-50e623-openshift-default-6c86dc794f-jmfmb" Nov 03 01:07:17.237564 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:07:17.237543 4939 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w0-50e623-openshift-default-6c86dc794f-jmfmb" Nov 03 01:07:17.756277 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:07:17.756218 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:07:18.034426 ip-10-0-140-107 kubenswrapper[4939]: E1103 01:07:18.034333 4939 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-sthjh" containerName="extractor" Nov 03 01:07:18.629517 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:07:18.629477 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:07:19.384609 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:07:19.384574 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:07:20.087477 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:07:20.087420 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:07:20.812608 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:07:20.812565 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:07:21.642724 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:07:21.642656 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:07:22.348722 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:07:22.348686 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:07:23.991234 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:07:23.991199 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:07:24.099976 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:07:24.099896 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:07:24.927991 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:07:24.927955 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:07:25.323635 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:07:25.323542 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:07:25.439355 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:07:25.439320 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:07:25.715328 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:07:25.715293 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:07:26.417318 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:07:26.417278 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:07:27.211078 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:07:27.211034 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:07:27.959954 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:07:27.959887 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:07:28.587359 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:07:28.587320 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:07:29.257235 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:07:29.257202 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:07:29.913578 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:07:29.913540 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:07:30.743972 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:07:30.726198 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:07:31.395851 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:07:31.395820 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:07:32.079271 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:07:32.079226 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:07:32.889930 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:07:32.878714 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:07:34.179331 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:07:34.179296 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:07:35.273461 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:07:35.259550 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:07:36.010875 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:07:36.010812 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:07:37.270133 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:07:37.270094 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:07:37.489668 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:07:37.489610 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:07:38.155063 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:07:38.155023 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:07:38.861837 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:07:38.861796 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:07:39.510950 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:07:39.510901 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:07:39.663934 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:07:39.658645 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:07:40.489509 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:07:40.489473 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:07:41.346214 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:07:41.346178 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:07:42.219941 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:07:42.219885 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:07:42.901456 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:07:42.901419 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:07:43.413840 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:07:43.413779 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:07:43.800719 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:07:43.800613 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:07:44.679041 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:07:44.679008 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:07:45.381375 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:07:45.381338 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:07:45.755232 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:07:45.755148 4939 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-l9ccf"] Nov 03 01:07:45.755636 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:07:45.755357 4939 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-l9ccf" podUID="c10d48f9-d5fc-4c4c-96c4-6531114191b8" containerName="istio-proxy" containerID="cri-o://7e92a0e30c6a10dcdccb126ec9c43ef3c7d1f0fa48b0880ee2428c2c57ea0346" gracePeriod=30 Nov 03 01:07:46.208986 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:07:46.204718 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:07:47.019355 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:07:47.019303 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:07:47.793731 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:07:47.793670 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:07:48.033080 ip-10-0-140-107 kubenswrapper[4939]: E1103 01:07:48.033039 4939 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-sthjh" containerName="extractor" Nov 03 01:07:48.485836 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:07:48.485796 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:07:49.147673 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:07:49.147641 4939 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-4pw7v"] Nov 03 01:07:49.247620 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:07:49.247583 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:07:50.202445 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:07:50.202405 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:07:50.329723 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:07:50.329678 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:07:50.569990 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:07:50.555474 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:07:51.317489 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:07:51.317399 4939 generic.go:356] "Generic (PLEG): container finished" podID="c10d48f9-d5fc-4c4c-96c4-6531114191b8" containerID="7e92a0e30c6a10dcdccb126ec9c43ef3c7d1f0fa48b0880ee2428c2c57ea0346" exitCode=0 Nov 03 01:07:51.318295 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:07:51.318211 4939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-l9ccf" event={"ID":"c10d48f9-d5fc-4c4c-96c4-6531114191b8","Type":"ContainerDied","Data":"7e92a0e30c6a10dcdccb126ec9c43ef3c7d1f0fa48b0880ee2428c2c57ea0346"} Nov 03 01:07:51.634155 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:07:51.634037 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:07:51.944534 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:07:51.944505 4939 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-l9ccf" Nov 03 01:07:52.099981 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:07:52.099943 4939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c10d48f9-d5fc-4c4c-96c4-6531114191b8-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c10d48f9-d5fc-4c4c-96c4-6531114191b8-istio-envoy\") pod \"c10d48f9-d5fc-4c4c-96c4-6531114191b8\" (UID: \"c10d48f9-d5fc-4c4c-96c4-6531114191b8\") " Nov 03 01:07:52.100178 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:07:52.100066 4939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c10d48f9-d5fc-4c4c-96c4-6531114191b8-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c10d48f9-d5fc-4c4c-96c4-6531114191b8-credential-socket\") pod \"c10d48f9-d5fc-4c4c-96c4-6531114191b8\" (UID: \"c10d48f9-d5fc-4c4c-96c4-6531114191b8\") " Nov 03 01:07:52.100178 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:07:52.100140 4939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c10d48f9-d5fc-4c4c-96c4-6531114191b8-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c10d48f9-d5fc-4c4c-96c4-6531114191b8-workload-certs\") pod \"c10d48f9-d5fc-4c4c-96c4-6531114191b8\" (UID: \"c10d48f9-d5fc-4c4c-96c4-6531114191b8\") " Nov 03 01:07:52.100322 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:07:52.100182 4939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c10d48f9-d5fc-4c4c-96c4-6531114191b8-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c10d48f9-d5fc-4c4c-96c4-6531114191b8-istio-data\") pod \"c10d48f9-d5fc-4c4c-96c4-6531114191b8\" (UID: \"c10d48f9-d5fc-4c4c-96c4-6531114191b8\") " Nov 03 01:07:52.100322 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:07:52.100254 4939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c10d48f9-d5fc-4c4c-96c4-6531114191b8-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c10d48f9-d5fc-4c4c-96c4-6531114191b8-istiod-ca-cert\") pod \"c10d48f9-d5fc-4c4c-96c4-6531114191b8\" (UID: \"c10d48f9-d5fc-4c4c-96c4-6531114191b8\") " Nov 03 01:07:52.100322 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:07:52.100311 4939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c10d48f9-d5fc-4c4c-96c4-6531114191b8-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c10d48f9-d5fc-4c4c-96c4-6531114191b8-workload-socket\") pod \"c10d48f9-d5fc-4c4c-96c4-6531114191b8\" (UID: \"c10d48f9-d5fc-4c4c-96c4-6531114191b8\") " Nov 03 01:07:52.100500 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:07:52.100423 4939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/c10d48f9-d5fc-4c4c-96c4-6531114191b8-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c10d48f9-d5fc-4c4c-96c4-6531114191b8-istio-podinfo\") pod \"c10d48f9-d5fc-4c4c-96c4-6531114191b8\" (UID: \"c10d48f9-d5fc-4c4c-96c4-6531114191b8\") " Nov 03 01:07:52.100500 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:07:52.100409 4939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c10d48f9-d5fc-4c4c-96c4-6531114191b8-credential-socket" pod "c10d48f9-d5fc-4c4c-96c4-6531114191b8" (UID: "c10d48f9-d5fc-4c4c-96c4-6531114191b8"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 01:07:52.100500 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:07:52.100453 4939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c10d48f9-d5fc-4c4c-96c4-6531114191b8-istio-token\" (UniqueName: \"kubernetes.io/projected/c10d48f9-d5fc-4c4c-96c4-6531114191b8-istio-token\") pod \"c10d48f9-d5fc-4c4c-96c4-6531114191b8\" (UID: \"c10d48f9-d5fc-4c4c-96c4-6531114191b8\") " Nov 03 01:07:52.100500 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:07:52.100489 4939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c10d48f9-d5fc-4c4c-96c4-6531114191b8-kube-api-access-dv4tg\" (UniqueName: \"kubernetes.io/projected/c10d48f9-d5fc-4c4c-96c4-6531114191b8-kube-api-access-dv4tg\") pod \"c10d48f9-d5fc-4c4c-96c4-6531114191b8\" (UID: \"c10d48f9-d5fc-4c4c-96c4-6531114191b8\") " Nov 03 01:07:52.100780 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:07:52.100736 4939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c10d48f9-d5fc-4c4c-96c4-6531114191b8-istio-data" pod "c10d48f9-d5fc-4c4c-96c4-6531114191b8" (UID: "c10d48f9-d5fc-4c4c-96c4-6531114191b8"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 01:07:52.100964 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:07:52.100802 4939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c10d48f9-d5fc-4c4c-96c4-6531114191b8-workload-socket" pod "c10d48f9-d5fc-4c4c-96c4-6531114191b8" (UID: "c10d48f9-d5fc-4c4c-96c4-6531114191b8"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 01:07:52.100964 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:07:52.100817 4939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c10d48f9-d5fc-4c4c-96c4-6531114191b8-workload-certs" pod "c10d48f9-d5fc-4c4c-96c4-6531114191b8" (UID: "c10d48f9-d5fc-4c4c-96c4-6531114191b8"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 01:07:52.101262 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:07:52.101241 4939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c10d48f9-d5fc-4c4c-96c4-6531114191b8-istiod-ca-cert" pod "c10d48f9-d5fc-4c4c-96c4-6531114191b8" (UID: "c10d48f9-d5fc-4c4c-96c4-6531114191b8"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 01:07:52.101625 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:07:52.101602 4939 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c10d48f9-d5fc-4c4c-96c4-6531114191b8-credential-socket\") on node \"ip-10-0-140-107.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:07:52.101738 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:07:52.101630 4939 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c10d48f9-d5fc-4c4c-96c4-6531114191b8-workload-certs\") on node \"ip-10-0-140-107.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:07:52.101738 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:07:52.101648 4939 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c10d48f9-d5fc-4c4c-96c4-6531114191b8-istio-data\") on node \"ip-10-0-140-107.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:07:52.101738 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:07:52.101662 4939 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c10d48f9-d5fc-4c4c-96c4-6531114191b8-istiod-ca-cert\") on node \"ip-10-0-140-107.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:07:52.101738 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:07:52.101677 4939 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c10d48f9-d5fc-4c4c-96c4-6531114191b8-workload-socket\") on node \"ip-10-0-140-107.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:07:52.103067 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:07:52.103044 4939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c10d48f9-d5fc-4c4c-96c4-6531114191b8-istio-envoy" pod "c10d48f9-d5fc-4c4c-96c4-6531114191b8" (UID: "c10d48f9-d5fc-4c4c-96c4-6531114191b8"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 01:07:52.103322 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:07:52.103297 4939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c10d48f9-d5fc-4c4c-96c4-6531114191b8-istio-token" pod "c10d48f9-d5fc-4c4c-96c4-6531114191b8" (UID: "c10d48f9-d5fc-4c4c-96c4-6531114191b8"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 01:07:52.104109 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:07:52.104070 4939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/c10d48f9-d5fc-4c4c-96c4-6531114191b8-istio-podinfo" pod "c10d48f9-d5fc-4c4c-96c4-6531114191b8" (UID: "c10d48f9-d5fc-4c4c-96c4-6531114191b8"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 01:07:52.104998 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:07:52.104974 4939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c10d48f9-d5fc-4c4c-96c4-6531114191b8-kube-api-access-dv4tg" pod "c10d48f9-d5fc-4c4c-96c4-6531114191b8" (UID: "c10d48f9-d5fc-4c4c-96c4-6531114191b8"). InnerVolumeSpecName "kube-api-access-dv4tg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 01:07:52.203156 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:07:52.203067 4939 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c10d48f9-d5fc-4c4c-96c4-6531114191b8-istio-podinfo\") on node \"ip-10-0-140-107.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:07:52.203156 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:07:52.203098 4939 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c10d48f9-d5fc-4c4c-96c4-6531114191b8-istio-token\") on node \"ip-10-0-140-107.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:07:52.203156 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:07:52.203108 4939 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dv4tg\" (UniqueName: \"kubernetes.io/projected/c10d48f9-d5fc-4c4c-96c4-6531114191b8-kube-api-access-dv4tg\") on node \"ip-10-0-140-107.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:07:52.203156 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:07:52.203119 4939 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c10d48f9-d5fc-4c4c-96c4-6531114191b8-istio-envoy\") on node \"ip-10-0-140-107.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:07:52.327374 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:07:52.327346 4939 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-l9ccf" Nov 03 01:07:52.327833 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:07:52.327385 4939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-l9ccf" event={"ID":"c10d48f9-d5fc-4c4c-96c4-6531114191b8","Type":"ContainerDied","Data":"1d323cbd6f0e8504b37ae689678082cc85dc007f9b2f0c554b82350ed0d2be2e"} Nov 03 01:07:52.327833 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:07:52.327450 4939 scope.go:122] "RemoveContainer" containerID="7e92a0e30c6a10dcdccb126ec9c43ef3c7d1f0fa48b0880ee2428c2c57ea0346" Nov 03 01:07:52.355359 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:07:52.355328 4939 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-l9ccf"] Nov 03 01:07:52.359939 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:07:52.359900 4939 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-l9ccf"] Nov 03 01:07:52.470842 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:07:52.470754 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:07:53.118659 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:07:53.118626 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:07:53.873190 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:07:53.873149 4939 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c10d48f9-d5fc-4c4c-96c4-6531114191b8" path="/var/lib/kubelet/pods/c10d48f9-d5fc-4c4c-96c4-6531114191b8/volumes" Nov 03 01:07:53.874256 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:07:53.874229 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:07:54.701866 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:07:54.701831 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:07:55.450575 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:07:55.450540 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:07:55.623011 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:07:55.622975 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:07:56.083780 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:07:56.083746 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:07:56.795248 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:07:56.795199 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:07:57.599477 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:07:57.599440 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:07:58.288384 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:07:58.288350 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:07:59.063018 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:07:59.062969 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:07:59.774927 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:07:59.774889 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:08:00.411181 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:08:00.411130 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:08:01.110708 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:08:01.110663 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:08:01.779888 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:08:01.779854 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:08:02.469722 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:08:02.469689 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:08:03.286592 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:08:03.286552 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:08:03.905845 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:08:03.905811 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:08:04.547670 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:08:04.547633 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:08:05.291201 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:08:05.291162 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:08:06.084830 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:08:06.084779 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:08:06.777580 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:08:06.777542 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:08:07.593745 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:08:07.593715 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:08:08.201194 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:08:08.201154 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:08:08.985274 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:08:08.985234 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:08:09.100345 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:08:09.100307 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:08:09.683320 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:08:09.683283 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:08:10.401384 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:08:10.401342 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:08:11.084056 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:08:11.084014 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:08:11.775974 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:08:11.775903 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:08:12.398431 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:08:12.398391 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:08:13.117975 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:08:13.117932 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:08:13.861838 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:08:13.861799 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:08:14.470317 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:08:14.470284 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:08:15.125281 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:08:15.125239 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:08:15.775720 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:08:15.775688 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:08:16.502003 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:08:16.501967 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:08:17.100267 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:08:17.100224 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:08:17.698921 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:08:17.698847 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:08:18.034620 ip-10-0-140-107 kubenswrapper[4939]: E1103 01:08:18.034523 4939 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-sthjh" containerName="extractor" Nov 03 01:08:18.360279 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:08:18.360242 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:08:19.000868 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:08:19.000838 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:08:19.681690 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:08:19.681657 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:08:20.321752 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:08:20.321716 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:08:20.709708 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:08:20.709669 4939 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-4pw7v"] Nov 03 01:08:21.029497 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:08:21.029412 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:08:22.266274 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:08:22.266236 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:08:23.105574 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:08:23.105538 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:08:24.004891 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:08:24.004852 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:08:24.802609 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:08:24.802575 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:08:25.599242 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:08:25.599208 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:08:26.380189 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:08:26.380151 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:08:26.634251 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:08:26.634156 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:08:26.753319 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:08:26.753287 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:08:27.088469 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:08:27.088418 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:08:27.820817 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:08:27.820787 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:08:28.472425 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:08:28.472383 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:08:29.110706 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:08:29.110674 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:08:29.803612 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:08:29.803582 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:08:30.495346 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:08:30.495307 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:08:31.168889 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:08:31.168853 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:08:31.779588 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:08:31.779557 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:08:32.517387 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:08:32.517327 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:08:32.625226 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:08:32.625176 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:08:32.839005 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:08:32.836246 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:08:33.576183 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:08:33.576150 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:08:34.200809 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:08:34.200778 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:08:34.813218 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:08:34.813185 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:08:35.416415 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:08:35.416374 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:08:36.113888 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:08:36.113853 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:08:36.708337 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:08:36.708301 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:08:37.417034 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:08:37.417004 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:08:38.183295 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:08:38.183251 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:08:38.647433 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:08:38.647395 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:08:38.781932 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:08:38.778297 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:08:39.499376 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:08:39.499344 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:08:40.844488 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:08:40.844446 4939 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-shared-a-w3-717217/auth-test-model-717217-kserve-5c9cb7dbd9-vl754" podUID="9634a946-0832-4901-ad01-bbb7a47f4ac7" containerName="main" probeResult="failure" output="Get \"https://10.132.0.191:8000/ready\": context deadline exceeded" Nov 03 01:08:40.866350 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:08:40.866294 4939 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-2f9a3a/distinct-w0-2f9a3a-kserve-555565cfcf-jtcr9" podUID="ac0ece06-8718-463f-915f-5315988d9711" containerName="main" probeResult="failure" output="Get \"https://10.132.0.76:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 01:08:41.066934 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:08:41.066442 4939 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-f7680d/distinct-w2-f7680d-kserve-756f874b98-qnpv8" podUID="836753fc-f9ef-4f61-b3fe-e5b92b7ee27b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.70:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 01:08:41.182210 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:08:41.182117 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:08:41.489125 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:08:41.489016 4939 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-83755a/distinct-2-w3-83755a-kserve-6cf66b995d-2l47b" podUID="c3b05e88-5d5b-4d03-b28d-832f22e3c868" containerName="main" probeResult="failure" output="Get \"https://10.132.0.72:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 01:08:42.028621 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:08:42.022478 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:08:42.691848 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:08:42.691814 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:08:43.415467 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:08:43.415417 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:08:43.603274 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:08:43.603219 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:08:44.103568 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:08:44.103527 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:08:44.624506 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:08:44.624471 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:08:45.470493 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:08:45.470458 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:08:46.142459 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:08:46.142424 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:08:46.905046 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:08:46.904997 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:08:47.507858 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:08:47.507822 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:08:48.032279 ip-10-0-140-107 kubenswrapper[4939]: E1103 01:08:48.032243 4939 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-sthjh" containerName="extractor" Nov 03 01:08:48.286437 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:08:48.286354 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:08:48.932126 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:08:48.932095 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:08:49.576813 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:08:49.576776 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:08:50.224934 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:08:50.224889 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:08:50.911720 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:08:50.911690 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:08:51.893567 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:08:51.893530 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:08:52.613300 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:08:52.613255 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:08:53.202180 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:08:53.202144 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:08:53.565022 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:08:53.564900 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:08:53.703897 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:08:53.703862 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:08:53.976738 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:08:53.976701 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:08:54.139241 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:08:54.139210 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:08:54.553817 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:08:54.553774 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:08:55.265311 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:08:55.265276 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:08:56.112093 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:08:56.112031 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:08:56.880604 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:08:56.880538 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:08:57.558025 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:08:57.557990 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:08:58.274924 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:08:58.274881 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:08:59.009108 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:08:59.009072 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:08:59.211722 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:08:59.211690 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:08:59.633923 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:08:59.633885 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:08:59.880832 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:08:59.880789 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:09:00.248635 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:09:00.248590 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:09:00.596133 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:09:00.596098 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:09:01.370180 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:09:01.370150 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:09:02.079322 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:09:02.079286 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:09:02.999788 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:09:02.999760 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:09:03.780052 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:09:03.780016 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:09:03.957284 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:09:03.957246 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:09:04.974974 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:09:04.974889 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:09:05.220420 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:09:05.220378 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:09:05.743226 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:09:05.743191 4939 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w1-2729f2/auth-test-model-2729f2-kserve-6bb7bb4c97-n9nht"] Nov 03 01:09:05.808686 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:09:05.808651 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:09:06.299495 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:09:06.299460 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:09:07.188573 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:09:07.188536 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:09:07.917473 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:09:07.917443 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:09:08.708574 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:09:08.708537 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:09:09.103836 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:09:09.103801 4939 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-4pw7v"] Nov 03 01:09:09.594342 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:09:09.594307 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:09:10.730658 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:09:10.730623 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:09:11.466965 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:09:11.466929 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:09:11.952929 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:09:11.952847 4939 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w1-4af1f7/auth-test-model-4af1f7-kserve-6c5694b445-v7l4q"] Nov 03 01:09:11.999759 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:09:11.999726 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:09:12.463410 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:09:12.463369 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:09:13.042546 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:09:13.042493 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:09:13.808086 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:09:13.808055 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:09:14.511224 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:09:14.511186 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:09:15.282388 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:09:15.282350 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:09:15.400029 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:09:15.399996 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:09:15.586376 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:09:15.586337 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:09:16.541478 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:09:16.541436 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:09:17.341365 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:09:17.341323 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:09:18.034053 ip-10-0-140-107 kubenswrapper[4939]: E1103 01:09:18.034014 4939 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-sthjh" containerName="extractor" Nov 03 01:09:18.092539 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:09:18.092509 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:09:18.681270 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:09:18.681237 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:09:19.338873 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:09:19.338830 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:09:20.104178 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:09:20.104124 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:09:20.707938 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:09:20.707886 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:09:20.785514 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:09:20.785476 4939 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-w1-2729f2/auth-test-model-2729f2-kserve-6bb7bb4c97-n9nht" podUID="f66fa3c5-ab64-4027-b004-399379a414f4" containerName="main" containerID="cri-o://6affc1952e113f0aec89986179dad54379f158fb3effa0c472c7451e44be964c" gracePeriod=45 Nov 03 01:09:20.856201 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:09:20.855764 4939 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-shared-a-w1-2729f2/auth-test-model-2729f2-kserve-6bb7bb4c97-n9nht" podUID="f66fa3c5-ab64-4027-b004-399379a414f4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.184:8000/ready\": EOF" Nov 03 01:09:21.145885 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:09:21.145857 4939 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w1-2729f2/auth-test-model-2729f2-kserve-6bb7bb4c97-n9nht" Nov 03 01:09:21.215574 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:09:21.215537 4939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f66fa3c5-ab64-4027-b004-399379a414f4-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f66fa3c5-ab64-4027-b004-399379a414f4-model-cache\") pod \"f66fa3c5-ab64-4027-b004-399379a414f4\" (UID: \"f66fa3c5-ab64-4027-b004-399379a414f4\") " Nov 03 01:09:21.215772 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:09:21.215620 4939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f66fa3c5-ab64-4027-b004-399379a414f4-kube-api-access-zsdxn\" (UniqueName: \"kubernetes.io/projected/f66fa3c5-ab64-4027-b004-399379a414f4-kube-api-access-zsdxn\") pod \"f66fa3c5-ab64-4027-b004-399379a414f4\" (UID: \"f66fa3c5-ab64-4027-b004-399379a414f4\") " Nov 03 01:09:21.215847 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:09:21.215823 4939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f66fa3c5-ab64-4027-b004-399379a414f4-tls-certs\" (UniqueName: \"kubernetes.io/secret/f66fa3c5-ab64-4027-b004-399379a414f4-tls-certs\") pod \"f66fa3c5-ab64-4027-b004-399379a414f4\" (UID: \"f66fa3c5-ab64-4027-b004-399379a414f4\") " Nov 03 01:09:21.215931 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:09:21.215856 4939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f66fa3c5-ab64-4027-b004-399379a414f4-home\" (UniqueName: \"kubernetes.io/empty-dir/f66fa3c5-ab64-4027-b004-399379a414f4-home\") pod \"f66fa3c5-ab64-4027-b004-399379a414f4\" (UID: \"f66fa3c5-ab64-4027-b004-399379a414f4\") " Nov 03 01:09:21.215931 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:09:21.215868 4939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f66fa3c5-ab64-4027-b004-399379a414f4-model-cache" pod "f66fa3c5-ab64-4027-b004-399379a414f4" (UID: "f66fa3c5-ab64-4027-b004-399379a414f4"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 01:09:21.216104 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:09:21.216066 4939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f66fa3c5-ab64-4027-b004-399379a414f4-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f66fa3c5-ab64-4027-b004-399379a414f4-tmp-dir\") pod \"f66fa3c5-ab64-4027-b004-399379a414f4\" (UID: \"f66fa3c5-ab64-4027-b004-399379a414f4\") " Nov 03 01:09:21.218561 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:09:21.216266 4939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f66fa3c5-ab64-4027-b004-399379a414f4-home" pod "f66fa3c5-ab64-4027-b004-399379a414f4" (UID: "f66fa3c5-ab64-4027-b004-399379a414f4"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 01:09:21.218561 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:09:21.216339 4939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f66fa3c5-ab64-4027-b004-399379a414f4-dshm\" (UniqueName: \"kubernetes.io/empty-dir/f66fa3c5-ab64-4027-b004-399379a414f4-dshm\") pod \"f66fa3c5-ab64-4027-b004-399379a414f4\" (UID: \"f66fa3c5-ab64-4027-b004-399379a414f4\") " Nov 03 01:09:21.218561 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:09:21.216460 4939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f66fa3c5-ab64-4027-b004-399379a414f4-tmp-dir" pod "f66fa3c5-ab64-4027-b004-399379a414f4" (UID: "f66fa3c5-ab64-4027-b004-399379a414f4"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 01:09:21.218561 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:09:21.217686 4939 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f66fa3c5-ab64-4027-b004-399379a414f4-tmp-dir\") on node \"ip-10-0-140-107.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:09:21.218561 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:09:21.217711 4939 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f66fa3c5-ab64-4027-b004-399379a414f4-model-cache\") on node \"ip-10-0-140-107.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:09:21.218561 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:09:21.217726 4939 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f66fa3c5-ab64-4027-b004-399379a414f4-home\") on node \"ip-10-0-140-107.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:09:21.219008 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:09:21.218820 4939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f66fa3c5-ab64-4027-b004-399379a414f4-kube-api-access-zsdxn" pod "f66fa3c5-ab64-4027-b004-399379a414f4" (UID: "f66fa3c5-ab64-4027-b004-399379a414f4"). InnerVolumeSpecName "kube-api-access-zsdxn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 01:09:21.219308 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:09:21.219282 4939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f66fa3c5-ab64-4027-b004-399379a414f4-dshm" pod "f66fa3c5-ab64-4027-b004-399379a414f4" (UID: "f66fa3c5-ab64-4027-b004-399379a414f4"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 01:09:21.224476 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:09:21.224440 4939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f66fa3c5-ab64-4027-b004-399379a414f4-tls-certs" pod "f66fa3c5-ab64-4027-b004-399379a414f4" (UID: "f66fa3c5-ab64-4027-b004-399379a414f4"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 01:09:21.319476 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:09:21.319377 4939 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f66fa3c5-ab64-4027-b004-399379a414f4-dshm\") on node \"ip-10-0-140-107.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:09:21.319476 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:09:21.319431 4939 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zsdxn\" (UniqueName: \"kubernetes.io/projected/f66fa3c5-ab64-4027-b004-399379a414f4-kube-api-access-zsdxn\") on node \"ip-10-0-140-107.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:09:21.319476 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:09:21.319449 4939 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f66fa3c5-ab64-4027-b004-399379a414f4-tls-certs\") on node \"ip-10-0-140-107.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:09:21.480357 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:09:21.480326 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:09:21.616688 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:09:21.615771 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:09:21.849810 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:09:21.849773 4939 generic.go:356] "Generic (PLEG): container finished" podID="f66fa3c5-ab64-4027-b004-399379a414f4" containerID="6affc1952e113f0aec89986179dad54379f158fb3effa0c472c7451e44be964c" exitCode=0 Nov 03 01:09:21.849810 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:09:21.849813 4939 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w1-2729f2/auth-test-model-2729f2-kserve-6bb7bb4c97-n9nht" Nov 03 01:09:21.850119 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:09:21.849853 4939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w1-2729f2/auth-test-model-2729f2-kserve-6bb7bb4c97-n9nht" event={"ID":"f66fa3c5-ab64-4027-b004-399379a414f4","Type":"ContainerDied","Data":"6affc1952e113f0aec89986179dad54379f158fb3effa0c472c7451e44be964c"} Nov 03 01:09:21.850119 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:09:21.849899 4939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w1-2729f2/auth-test-model-2729f2-kserve-6bb7bb4c97-n9nht" event={"ID":"f66fa3c5-ab64-4027-b004-399379a414f4","Type":"ContainerDied","Data":"bce579271af93f2a57fb0eb988b91219c0f72a999c963756b90ad16c5abb3769"} Nov 03 01:09:21.850119 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:09:21.849939 4939 scope.go:122] "RemoveContainer" containerID="6affc1952e113f0aec89986179dad54379f158fb3effa0c472c7451e44be964c" Nov 03 01:09:21.867129 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:09:21.867094 4939 scope.go:122] "RemoveContainer" containerID="6affc1952e113f0aec89986179dad54379f158fb3effa0c472c7451e44be964c" Nov 03 01:09:21.867426 ip-10-0-140-107 kubenswrapper[4939]: E1103 01:09:21.867395 4939 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6affc1952e113f0aec89986179dad54379f158fb3effa0c472c7451e44be964c\": container with ID starting with 6affc1952e113f0aec89986179dad54379f158fb3effa0c472c7451e44be964c not found: ID does not exist" containerID="6affc1952e113f0aec89986179dad54379f158fb3effa0c472c7451e44be964c" Nov 03 01:09:21.867564 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:09:21.867437 4939 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6affc1952e113f0aec89986179dad54379f158fb3effa0c472c7451e44be964c"} err="failed to get container status \"6affc1952e113f0aec89986179dad54379f158fb3effa0c472c7451e44be964c\": rpc error: code = NotFound desc = could not find container \"6affc1952e113f0aec89986179dad54379f158fb3effa0c472c7451e44be964c\": container with ID starting with 6affc1952e113f0aec89986179dad54379f158fb3effa0c472c7451e44be964c not found: ID does not exist" Nov 03 01:09:21.885957 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:09:21.885932 4939 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w1-2729f2/auth-test-model-2729f2-kserve-6bb7bb4c97-n9nht"] Nov 03 01:09:21.891956 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:09:21.891931 4939 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-w1-2729f2/auth-test-model-2729f2-kserve-6bb7bb4c97-n9nht"] Nov 03 01:09:22.019443 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:09:22.019406 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:09:22.666815 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:09:22.666717 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:09:22.811116 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:09:22.811078 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:09:22.999342 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:09:22.999246 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:09:23.406261 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:09:23.406230 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:09:23.880690 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:09:23.880642 4939 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f66fa3c5-ab64-4027-b004-399379a414f4" path="/var/lib/kubelet/pods/f66fa3c5-ab64-4027-b004-399379a414f4/volumes" Nov 03 01:09:24.153816 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:09:24.153738 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:09:24.855057 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:09:24.855020 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:09:25.700493 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:09:25.700443 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:09:26.138849 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:09:26.138807 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:09:26.264539 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:09:26.264507 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:09:26.595418 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:09:26.595383 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:09:26.985439 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:09:26.985333 4939 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-w1-4af1f7/auth-test-model-4af1f7-kserve-6c5694b445-v7l4q" podUID="afd256d4-75f9-4572-8b84-4faad4e5c694" containerName="main" containerID="cri-o://87bd5f46fb7486e7bfd4ded6a1da46dc39d7e85cc2aaf663b7a3cc8bf92296c5" gracePeriod=45 Nov 03 01:09:27.319627 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:09:27.319592 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:09:27.341816 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:09:27.341790 4939 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w1-4af1f7/auth-test-model-4af1f7-kserve-6c5694b445-v7l4q" Nov 03 01:09:27.418048 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:09:27.418014 4939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/afd256d4-75f9-4572-8b84-4faad4e5c694-dshm\" (UniqueName: \"kubernetes.io/empty-dir/afd256d4-75f9-4572-8b84-4faad4e5c694-dshm\") pod \"afd256d4-75f9-4572-8b84-4faad4e5c694\" (UID: \"afd256d4-75f9-4572-8b84-4faad4e5c694\") " Nov 03 01:09:27.418562 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:09:27.418535 4939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/afd256d4-75f9-4572-8b84-4faad4e5c694-home\" (UniqueName: \"kubernetes.io/empty-dir/afd256d4-75f9-4572-8b84-4faad4e5c694-home\") pod \"afd256d4-75f9-4572-8b84-4faad4e5c694\" (UID: \"afd256d4-75f9-4572-8b84-4faad4e5c694\") " Nov 03 01:09:27.418674 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:09:27.418647 4939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/afd256d4-75f9-4572-8b84-4faad4e5c694-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/afd256d4-75f9-4572-8b84-4faad4e5c694-model-cache\") pod \"afd256d4-75f9-4572-8b84-4faad4e5c694\" (UID: \"afd256d4-75f9-4572-8b84-4faad4e5c694\") " Nov 03 01:09:27.418747 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:09:27.418729 4939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/afd256d4-75f9-4572-8b84-4faad4e5c694-kube-api-access-gtzct\" (UniqueName: \"kubernetes.io/projected/afd256d4-75f9-4572-8b84-4faad4e5c694-kube-api-access-gtzct\") pod \"afd256d4-75f9-4572-8b84-4faad4e5c694\" (UID: \"afd256d4-75f9-4572-8b84-4faad4e5c694\") " Nov 03 01:09:27.418951 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:09:27.418766 4939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/afd256d4-75f9-4572-8b84-4faad4e5c694-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/afd256d4-75f9-4572-8b84-4faad4e5c694-tmp-dir\") pod \"afd256d4-75f9-4572-8b84-4faad4e5c694\" (UID: \"afd256d4-75f9-4572-8b84-4faad4e5c694\") " Nov 03 01:09:27.418951 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:09:27.418855 4939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/afd256d4-75f9-4572-8b84-4faad4e5c694-tls-certs\" (UniqueName: \"kubernetes.io/secret/afd256d4-75f9-4572-8b84-4faad4e5c694-tls-certs\") pod \"afd256d4-75f9-4572-8b84-4faad4e5c694\" (UID: \"afd256d4-75f9-4572-8b84-4faad4e5c694\") " Nov 03 01:09:27.419097 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:09:27.418967 4939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/afd256d4-75f9-4572-8b84-4faad4e5c694-home" pod "afd256d4-75f9-4572-8b84-4faad4e5c694" (UID: "afd256d4-75f9-4572-8b84-4faad4e5c694"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 01:09:27.419975 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:09:27.419952 4939 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/afd256d4-75f9-4572-8b84-4faad4e5c694-home\") on node \"ip-10-0-140-107.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:09:27.420261 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:09:27.419994 4939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/afd256d4-75f9-4572-8b84-4faad4e5c694-tmp-dir" pod "afd256d4-75f9-4572-8b84-4faad4e5c694" (UID: "afd256d4-75f9-4572-8b84-4faad4e5c694"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 01:09:27.420389 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:09:27.420366 4939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/afd256d4-75f9-4572-8b84-4faad4e5c694-model-cache" pod "afd256d4-75f9-4572-8b84-4faad4e5c694" (UID: "afd256d4-75f9-4572-8b84-4faad4e5c694"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 01:09:27.421706 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:09:27.421680 4939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/afd256d4-75f9-4572-8b84-4faad4e5c694-kube-api-access-gtzct" pod "afd256d4-75f9-4572-8b84-4faad4e5c694" (UID: "afd256d4-75f9-4572-8b84-4faad4e5c694"). InnerVolumeSpecName "kube-api-access-gtzct". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 01:09:27.422175 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:09:27.422145 4939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/afd256d4-75f9-4572-8b84-4faad4e5c694-tls-certs" pod "afd256d4-75f9-4572-8b84-4faad4e5c694" (UID: "afd256d4-75f9-4572-8b84-4faad4e5c694"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 01:09:27.422718 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:09:27.422695 4939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/afd256d4-75f9-4572-8b84-4faad4e5c694-dshm" pod "afd256d4-75f9-4572-8b84-4faad4e5c694" (UID: "afd256d4-75f9-4572-8b84-4faad4e5c694"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 01:09:27.447011 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:09:27.446984 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:09:27.522154 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:09:27.522028 4939 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/afd256d4-75f9-4572-8b84-4faad4e5c694-model-cache\") on node \"ip-10-0-140-107.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:09:27.522154 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:09:27.522067 4939 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gtzct\" (UniqueName: \"kubernetes.io/projected/afd256d4-75f9-4572-8b84-4faad4e5c694-kube-api-access-gtzct\") on node \"ip-10-0-140-107.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:09:27.522154 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:09:27.522087 4939 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/afd256d4-75f9-4572-8b84-4faad4e5c694-tmp-dir\") on node \"ip-10-0-140-107.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:09:27.522154 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:09:27.522103 4939 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/afd256d4-75f9-4572-8b84-4faad4e5c694-tls-certs\") on node \"ip-10-0-140-107.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:09:27.522154 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:09:27.522117 4939 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/afd256d4-75f9-4572-8b84-4faad4e5c694-dshm\") on node \"ip-10-0-140-107.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:09:27.605933 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:09:27.590378 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:09:27.991310 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:09:27.991212 4939 generic.go:356] "Generic (PLEG): container finished" podID="afd256d4-75f9-4572-8b84-4faad4e5c694" containerID="87bd5f46fb7486e7bfd4ded6a1da46dc39d7e85cc2aaf663b7a3cc8bf92296c5" exitCode=0 Nov 03 01:09:27.991835 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:09:27.991324 4939 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w1-4af1f7/auth-test-model-4af1f7-kserve-6c5694b445-v7l4q" Nov 03 01:09:27.991835 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:09:27.991413 4939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w1-4af1f7/auth-test-model-4af1f7-kserve-6c5694b445-v7l4q" event={"ID":"afd256d4-75f9-4572-8b84-4faad4e5c694","Type":"ContainerDied","Data":"87bd5f46fb7486e7bfd4ded6a1da46dc39d7e85cc2aaf663b7a3cc8bf92296c5"} Nov 03 01:09:27.991835 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:09:27.991446 4939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w1-4af1f7/auth-test-model-4af1f7-kserve-6c5694b445-v7l4q" event={"ID":"afd256d4-75f9-4572-8b84-4faad4e5c694","Type":"ContainerDied","Data":"8380973a55da4fe3b83e3f81f1557a8c57c2fbf7105252d9c9acadc930019efe"} Nov 03 01:09:27.991835 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:09:27.991468 4939 scope.go:122] "RemoveContainer" containerID="87bd5f46fb7486e7bfd4ded6a1da46dc39d7e85cc2aaf663b7a3cc8bf92296c5" Nov 03 01:09:28.014293 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:09:28.014265 4939 scope.go:122] "RemoveContainer" containerID="87bd5f46fb7486e7bfd4ded6a1da46dc39d7e85cc2aaf663b7a3cc8bf92296c5" Nov 03 01:09:28.014613 ip-10-0-140-107 kubenswrapper[4939]: E1103 01:09:28.014588 4939 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"87bd5f46fb7486e7bfd4ded6a1da46dc39d7e85cc2aaf663b7a3cc8bf92296c5\": container with ID starting with 87bd5f46fb7486e7bfd4ded6a1da46dc39d7e85cc2aaf663b7a3cc8bf92296c5 not found: ID does not exist" containerID="87bd5f46fb7486e7bfd4ded6a1da46dc39d7e85cc2aaf663b7a3cc8bf92296c5" Nov 03 01:09:28.014690 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:09:28.014622 4939 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"87bd5f46fb7486e7bfd4ded6a1da46dc39d7e85cc2aaf663b7a3cc8bf92296c5"} err="failed to get container status \"87bd5f46fb7486e7bfd4ded6a1da46dc39d7e85cc2aaf663b7a3cc8bf92296c5\": rpc error: code = NotFound desc = could not find container \"87bd5f46fb7486e7bfd4ded6a1da46dc39d7e85cc2aaf663b7a3cc8bf92296c5\": container with ID starting with 87bd5f46fb7486e7bfd4ded6a1da46dc39d7e85cc2aaf663b7a3cc8bf92296c5 not found: ID does not exist" Nov 03 01:09:28.036637 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:09:28.036612 4939 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w1-4af1f7/auth-test-model-4af1f7-kserve-6c5694b445-v7l4q"] Nov 03 01:09:28.043657 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:09:28.043620 4939 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-w1-4af1f7/auth-test-model-4af1f7-kserve-6c5694b445-v7l4q"] Nov 03 01:09:28.217806 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:09:28.217756 4939 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-shared-b-w1-4af1f7/auth-test-model-4af1f7-kserve-6c5694b445-v7l4q" podUID="afd256d4-75f9-4572-8b84-4faad4e5c694" containerName="main" probeResult="failure" output="Get \"https://10.132.0.190:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 01:09:28.771602 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:09:28.771565 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:09:29.017532 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:09:29.017494 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:09:29.446451 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:09:29.446417 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:09:29.687934 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:09:29.687875 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:09:29.793135 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:09:29.793050 4939 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w2-ef434a-openshift-default-67cd7c96dc-q5p6d"] Nov 03 01:09:29.794452 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:09:29.794411 4939 cpu_manager.go:414] "containerMap: removing container" podUID="f66fa3c5-ab64-4027-b004-399379a414f4" containerName="main" Nov 03 01:09:29.794452 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:09:29.794444 4939 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f66fa3c5-ab64-4027-b004-399379a414f4" containerName="main" Nov 03 01:09:29.794452 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:09:29.794458 4939 cpu_manager.go:414] "containerMap: removing container" podUID="c10d48f9-d5fc-4c4c-96c4-6531114191b8" containerName="istio-proxy" Nov 03 01:09:29.794747 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:09:29.794467 4939 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c10d48f9-d5fc-4c4c-96c4-6531114191b8" containerName="istio-proxy" Nov 03 01:09:29.794747 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:09:29.794498 4939 cpu_manager.go:414] "containerMap: removing container" podUID="afd256d4-75f9-4572-8b84-4faad4e5c694" containerName="main" Nov 03 01:09:29.794747 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:09:29.794506 4939 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="afd256d4-75f9-4572-8b84-4faad4e5c694" containerName="main" Nov 03 01:09:29.794747 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:09:29.794743 4939 memory_manager.go:365] "RemoveStaleState removing state" podUID="f66fa3c5-ab64-4027-b004-399379a414f4" containerName="main" Nov 03 01:09:29.795103 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:09:29.794760 4939 memory_manager.go:365] "RemoveStaleState removing state" podUID="c10d48f9-d5fc-4c4c-96c4-6531114191b8" containerName="istio-proxy" Nov 03 01:09:29.795103 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:09:29.794776 4939 memory_manager.go:365] "RemoveStaleState removing state" podUID="afd256d4-75f9-4572-8b84-4faad4e5c694" containerName="main" Nov 03 01:09:29.801328 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:09:29.801301 4939 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w2-ef434a-openshift-default-67cd7c96dc-q5p6d" Nov 03 01:09:29.810246 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:09:29.810220 4939 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-w2-ef434a-openshift-default-dockercfg-sc8xt\"" Nov 03 01:09:29.817003 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:09:29.815934 4939 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-ef434a-openshift-default-67cd7c96dc-q5p6d"] Nov 03 01:09:29.850406 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:09:29.850371 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:09:29.873517 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:09:29.873476 4939 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="afd256d4-75f9-4572-8b84-4faad4e5c694" path="/var/lib/kubelet/pods/afd256d4-75f9-4572-8b84-4faad4e5c694/volumes" Nov 03 01:09:30.025970 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:09:30.002996 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/285b736b-2a9f-40cd-96c4-3d7193d3eb8d-workload-certs\") pod \"e2e-shared-b-w2-ef434a-openshift-default-67cd7c96dc-q5p6d\" (UID: \"285b736b-2a9f-40cd-96c4-3d7193d3eb8d\") " pod="openshift-ingress/e2e-shared-b-w2-ef434a-openshift-default-67cd7c96dc-q5p6d" Nov 03 01:09:30.025970 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:09:30.003229 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/285b736b-2a9f-40cd-96c4-3d7193d3eb8d-istio-data\") pod \"e2e-shared-b-w2-ef434a-openshift-default-67cd7c96dc-q5p6d\" (UID: \"285b736b-2a9f-40cd-96c4-3d7193d3eb8d\") " pod="openshift-ingress/e2e-shared-b-w2-ef434a-openshift-default-67cd7c96dc-q5p6d" Nov 03 01:09:30.025970 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:09:30.003317 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/285b736b-2a9f-40cd-96c4-3d7193d3eb8d-istiod-ca-cert\") pod \"e2e-shared-b-w2-ef434a-openshift-default-67cd7c96dc-q5p6d\" (UID: \"285b736b-2a9f-40cd-96c4-3d7193d3eb8d\") " pod="openshift-ingress/e2e-shared-b-w2-ef434a-openshift-default-67cd7c96dc-q5p6d" Nov 03 01:09:30.025970 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:09:30.003436 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/285b736b-2a9f-40cd-96c4-3d7193d3eb8d-workload-socket\") pod \"e2e-shared-b-w2-ef434a-openshift-default-67cd7c96dc-q5p6d\" (UID: \"285b736b-2a9f-40cd-96c4-3d7193d3eb8d\") " pod="openshift-ingress/e2e-shared-b-w2-ef434a-openshift-default-67cd7c96dc-q5p6d" Nov 03 01:09:30.025970 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:09:30.003462 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/285b736b-2a9f-40cd-96c4-3d7193d3eb8d-istio-envoy\") pod \"e2e-shared-b-w2-ef434a-openshift-default-67cd7c96dc-q5p6d\" (UID: \"285b736b-2a9f-40cd-96c4-3d7193d3eb8d\") " pod="openshift-ingress/e2e-shared-b-w2-ef434a-openshift-default-67cd7c96dc-q5p6d" Nov 03 01:09:30.179935 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:09:30.178366 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/285b736b-2a9f-40cd-96c4-3d7193d3eb8d-istio-podinfo\") pod \"e2e-shared-b-w2-ef434a-openshift-default-67cd7c96dc-q5p6d\" (UID: \"285b736b-2a9f-40cd-96c4-3d7193d3eb8d\") " pod="openshift-ingress/e2e-shared-b-w2-ef434a-openshift-default-67cd7c96dc-q5p6d" Nov 03 01:09:30.179935 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:09:30.178435 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/285b736b-2a9f-40cd-96c4-3d7193d3eb8d-istio-token\") pod \"e2e-shared-b-w2-ef434a-openshift-default-67cd7c96dc-q5p6d\" (UID: \"285b736b-2a9f-40cd-96c4-3d7193d3eb8d\") " pod="openshift-ingress/e2e-shared-b-w2-ef434a-openshift-default-67cd7c96dc-q5p6d" Nov 03 01:09:30.179935 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:09:30.178573 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/285b736b-2a9f-40cd-96c4-3d7193d3eb8d-credential-socket\") pod \"e2e-shared-b-w2-ef434a-openshift-default-67cd7c96dc-q5p6d\" (UID: \"285b736b-2a9f-40cd-96c4-3d7193d3eb8d\") " pod="openshift-ingress/e2e-shared-b-w2-ef434a-openshift-default-67cd7c96dc-q5p6d" Nov 03 01:09:30.179935 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:09:30.178666 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wgsfl\" (UniqueName: \"kubernetes.io/projected/285b736b-2a9f-40cd-96c4-3d7193d3eb8d-kube-api-access-wgsfl\") pod \"e2e-shared-b-w2-ef434a-openshift-default-67cd7c96dc-q5p6d\" (UID: \"285b736b-2a9f-40cd-96c4-3d7193d3eb8d\") " pod="openshift-ingress/e2e-shared-b-w2-ef434a-openshift-default-67cd7c96dc-q5p6d" Nov 03 01:09:30.221809 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:09:30.221769 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:09:30.285305 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:09:30.285249 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/285b736b-2a9f-40cd-96c4-3d7193d3eb8d-istio-podinfo\") pod \"e2e-shared-b-w2-ef434a-openshift-default-67cd7c96dc-q5p6d\" (UID: \"285b736b-2a9f-40cd-96c4-3d7193d3eb8d\") " pod="openshift-ingress/e2e-shared-b-w2-ef434a-openshift-default-67cd7c96dc-q5p6d" Nov 03 01:09:30.285726 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:09:30.285688 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/285b736b-2a9f-40cd-96c4-3d7193d3eb8d-istio-token\") pod \"e2e-shared-b-w2-ef434a-openshift-default-67cd7c96dc-q5p6d\" (UID: \"285b736b-2a9f-40cd-96c4-3d7193d3eb8d\") " pod="openshift-ingress/e2e-shared-b-w2-ef434a-openshift-default-67cd7c96dc-q5p6d" Nov 03 01:09:30.286019 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:09:30.285998 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/285b736b-2a9f-40cd-96c4-3d7193d3eb8d-credential-socket\") pod \"e2e-shared-b-w2-ef434a-openshift-default-67cd7c96dc-q5p6d\" (UID: \"285b736b-2a9f-40cd-96c4-3d7193d3eb8d\") " pod="openshift-ingress/e2e-shared-b-w2-ef434a-openshift-default-67cd7c96dc-q5p6d" Nov 03 01:09:30.289244 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:09:30.289218 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wgsfl\" (UniqueName: \"kubernetes.io/projected/285b736b-2a9f-40cd-96c4-3d7193d3eb8d-kube-api-access-wgsfl\") pod \"e2e-shared-b-w2-ef434a-openshift-default-67cd7c96dc-q5p6d\" (UID: \"285b736b-2a9f-40cd-96c4-3d7193d3eb8d\") " pod="openshift-ingress/e2e-shared-b-w2-ef434a-openshift-default-67cd7c96dc-q5p6d" Nov 03 01:09:30.289452 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:09:30.289436 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/285b736b-2a9f-40cd-96c4-3d7193d3eb8d-workload-certs\") pod \"e2e-shared-b-w2-ef434a-openshift-default-67cd7c96dc-q5p6d\" (UID: \"285b736b-2a9f-40cd-96c4-3d7193d3eb8d\") " pod="openshift-ingress/e2e-shared-b-w2-ef434a-openshift-default-67cd7c96dc-q5p6d" Nov 03 01:09:30.289675 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:09:30.289656 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/285b736b-2a9f-40cd-96c4-3d7193d3eb8d-istio-data\") pod \"e2e-shared-b-w2-ef434a-openshift-default-67cd7c96dc-q5p6d\" (UID: \"285b736b-2a9f-40cd-96c4-3d7193d3eb8d\") " pod="openshift-ingress/e2e-shared-b-w2-ef434a-openshift-default-67cd7c96dc-q5p6d" Nov 03 01:09:30.289849 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:09:30.289835 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/285b736b-2a9f-40cd-96c4-3d7193d3eb8d-istiod-ca-cert\") pod \"e2e-shared-b-w2-ef434a-openshift-default-67cd7c96dc-q5p6d\" (UID: \"285b736b-2a9f-40cd-96c4-3d7193d3eb8d\") " pod="openshift-ingress/e2e-shared-b-w2-ef434a-openshift-default-67cd7c96dc-q5p6d" Nov 03 01:09:30.290084 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:09:30.290060 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/285b736b-2a9f-40cd-96c4-3d7193d3eb8d-workload-socket\") pod \"e2e-shared-b-w2-ef434a-openshift-default-67cd7c96dc-q5p6d\" (UID: \"285b736b-2a9f-40cd-96c4-3d7193d3eb8d\") " pod="openshift-ingress/e2e-shared-b-w2-ef434a-openshift-default-67cd7c96dc-q5p6d" Nov 03 01:09:30.290209 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:09:30.290193 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/285b736b-2a9f-40cd-96c4-3d7193d3eb8d-istio-envoy\") pod \"e2e-shared-b-w2-ef434a-openshift-default-67cd7c96dc-q5p6d\" (UID: \"285b736b-2a9f-40cd-96c4-3d7193d3eb8d\") " pod="openshift-ingress/e2e-shared-b-w2-ef434a-openshift-default-67cd7c96dc-q5p6d" Nov 03 01:09:30.298754 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:09:30.298714 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/285b736b-2a9f-40cd-96c4-3d7193d3eb8d-workload-certs\") pod \"e2e-shared-b-w2-ef434a-openshift-default-67cd7c96dc-q5p6d\" (UID: \"285b736b-2a9f-40cd-96c4-3d7193d3eb8d\") " pod="openshift-ingress/e2e-shared-b-w2-ef434a-openshift-default-67cd7c96dc-q5p6d" Nov 03 01:09:30.301518 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:09:30.299442 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/285b736b-2a9f-40cd-96c4-3d7193d3eb8d-istiod-ca-cert\") pod \"e2e-shared-b-w2-ef434a-openshift-default-67cd7c96dc-q5p6d\" (UID: \"285b736b-2a9f-40cd-96c4-3d7193d3eb8d\") " pod="openshift-ingress/e2e-shared-b-w2-ef434a-openshift-default-67cd7c96dc-q5p6d" Nov 03 01:09:30.301754 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:09:30.299595 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/285b736b-2a9f-40cd-96c4-3d7193d3eb8d-istio-data\") pod \"e2e-shared-b-w2-ef434a-openshift-default-67cd7c96dc-q5p6d\" (UID: \"285b736b-2a9f-40cd-96c4-3d7193d3eb8d\") " pod="openshift-ingress/e2e-shared-b-w2-ef434a-openshift-default-67cd7c96dc-q5p6d" Nov 03 01:09:30.301870 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:09:30.299740 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/285b736b-2a9f-40cd-96c4-3d7193d3eb8d-credential-socket\") pod \"e2e-shared-b-w2-ef434a-openshift-default-67cd7c96dc-q5p6d\" (UID: \"285b736b-2a9f-40cd-96c4-3d7193d3eb8d\") " pod="openshift-ingress/e2e-shared-b-w2-ef434a-openshift-default-67cd7c96dc-q5p6d" Nov 03 01:09:30.301988 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:09:30.300237 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/285b736b-2a9f-40cd-96c4-3d7193d3eb8d-workload-socket\") pod \"e2e-shared-b-w2-ef434a-openshift-default-67cd7c96dc-q5p6d\" (UID: \"285b736b-2a9f-40cd-96c4-3d7193d3eb8d\") " pod="openshift-ingress/e2e-shared-b-w2-ef434a-openshift-default-67cd7c96dc-q5p6d" Nov 03 01:09:30.327373 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:09:30.327330 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/285b736b-2a9f-40cd-96c4-3d7193d3eb8d-istio-envoy\") pod \"e2e-shared-b-w2-ef434a-openshift-default-67cd7c96dc-q5p6d\" (UID: \"285b736b-2a9f-40cd-96c4-3d7193d3eb8d\") " pod="openshift-ingress/e2e-shared-b-w2-ef434a-openshift-default-67cd7c96dc-q5p6d" Nov 03 01:09:30.342615 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:09:30.342579 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/285b736b-2a9f-40cd-96c4-3d7193d3eb8d-istio-podinfo\") pod \"e2e-shared-b-w2-ef434a-openshift-default-67cd7c96dc-q5p6d\" (UID: \"285b736b-2a9f-40cd-96c4-3d7193d3eb8d\") " pod="openshift-ingress/e2e-shared-b-w2-ef434a-openshift-default-67cd7c96dc-q5p6d" Nov 03 01:09:30.346403 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:09:30.346264 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/285b736b-2a9f-40cd-96c4-3d7193d3eb8d-istio-token\") pod \"e2e-shared-b-w2-ef434a-openshift-default-67cd7c96dc-q5p6d\" (UID: \"285b736b-2a9f-40cd-96c4-3d7193d3eb8d\") " pod="openshift-ingress/e2e-shared-b-w2-ef434a-openshift-default-67cd7c96dc-q5p6d" Nov 03 01:09:30.351100 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:09:30.351014 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wgsfl\" (UniqueName: \"kubernetes.io/projected/285b736b-2a9f-40cd-96c4-3d7193d3eb8d-kube-api-access-wgsfl\") pod \"e2e-shared-b-w2-ef434a-openshift-default-67cd7c96dc-q5p6d\" (UID: \"285b736b-2a9f-40cd-96c4-3d7193d3eb8d\") " pod="openshift-ingress/e2e-shared-b-w2-ef434a-openshift-default-67cd7c96dc-q5p6d" Nov 03 01:09:30.458228 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:09:30.458130 4939 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w2-ef434a-openshift-default-67cd7c96dc-q5p6d" Nov 03 01:09:30.779663 ip-10-0-140-107 kubenswrapper[4939]: W1103 01:09:30.779623 4939 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod285b736b_2a9f_40cd_96c4_3d7193d3eb8d.slice/crio-438cf23a4e7648ba51c3aaa84fe347aeb05d38750e2624151d0cbe607de85850 WatchSource:0}: Error finding container 438cf23a4e7648ba51c3aaa84fe347aeb05d38750e2624151d0cbe607de85850: Status 404 returned error can't find the container with id 438cf23a4e7648ba51c3aaa84fe347aeb05d38750e2624151d0cbe607de85850 Nov 03 01:09:30.779920 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:09:30.779889 4939 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-ef434a-openshift-default-67cd7c96dc-q5p6d"] Nov 03 01:09:30.782633 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:09:30.782587 4939 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 01:09:30.782716 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:09:30.782683 4939 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 01:09:30.782774 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:09:30.782727 4939 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 01:09:30.920045 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:09:30.916787 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:09:31.250884 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:09:31.250839 4939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-ef434a-openshift-default-67cd7c96dc-q5p6d" event={"ID":"285b736b-2a9f-40cd-96c4-3d7193d3eb8d","Type":"ContainerStarted","Data":"a01d3cc1289dcec7066b49be8e8f85d54d5805285a2912e982dc947054666749"} Nov 03 01:09:31.251313 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:09:31.250904 4939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-ef434a-openshift-default-67cd7c96dc-q5p6d" event={"ID":"285b736b-2a9f-40cd-96c4-3d7193d3eb8d","Type":"ContainerStarted","Data":"438cf23a4e7648ba51c3aaa84fe347aeb05d38750e2624151d0cbe607de85850"} Nov 03 01:09:31.281557 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:09:31.281455 4939 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w2-ef434a-openshift-default-67cd7c96dc-q5p6d" podStartSLOduration=2.281440904 podStartE2EDuration="2.281440904s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 01:09:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 01:09:31.277232006 +0000 UTC m=+1719.541566951" watchObservedRunningTime="2026-11-03 01:09:31.281440904 +0000 UTC m=+1719.545775848" Nov 03 01:09:31.468525 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:09:31.466452 4939 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w2-ef434a-openshift-default-67cd7c96dc-q5p6d" Nov 03 01:09:31.490987 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:09:31.488995 4939 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w2-ef434a-openshift-default-67cd7c96dc-q5p6d" Nov 03 01:09:31.653004 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:09:31.652950 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:09:32.022526 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:09:32.022422 4939 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1f2827bd-cxnb5"] Nov 03 01:09:32.027329 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:09:32.027302 4939 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1f2827bd-cxnb5"] Nov 03 01:09:32.260493 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:09:32.260460 4939 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w2-ef434a-openshift-default-67cd7c96dc-q5p6d" Nov 03 01:09:32.261579 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:09:32.261556 4939 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w2-ef434a-openshift-default-67cd7c96dc-q5p6d" Nov 03 01:09:32.283460 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:09:32.283379 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:09:33.173214 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:09:33.173155 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:09:33.526836 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:09:33.526753 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:09:33.659365 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:09:33.659329 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:09:33.872290 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:09:33.872252 4939 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="344f17d1-1f4c-4b07-8ddd-ea417772c6f0" path="/var/lib/kubelet/pods/344f17d1-1f4c-4b07-8ddd-ea417772c6f0/volumes" Nov 03 01:09:34.089046 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:09:34.082977 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:09:34.694474 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:09:34.694437 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:09:34.717284 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:09:34.717248 4939 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w2-ef434a-7b64fdc54f-9zxp5"] Nov 03 01:09:34.723672 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:09:34.723644 4939 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-ef434a-7b64fdc54f-9zxp5" Nov 03 01:09:34.733313 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:09:34.732289 4939 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-b-w2-ef434a\"" Nov 03 01:09:34.733313 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:09:34.732480 4939 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w2-ef434a-7b64fdc54f-9zxp5"] Nov 03 01:09:34.760208 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:09:34.760176 4939 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-ef434a-845d4b6b99-nqmzg"] Nov 03 01:09:34.765169 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:09:34.765143 4939 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-ef434a-845d4b6b99-nqmzg" Nov 03 01:09:34.771331 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:09:34.771300 4939 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-b-w2-ef434a-dockercfg-ggckt\"" Nov 03 01:09:34.779891 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:09:34.777856 4939 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-ef434a-845d4b6b99-nqmzg"] Nov 03 01:09:34.813112 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:09:34.813081 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:09:34.958982 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:09:34.949452 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5zhnm\" (UniqueName: \"kubernetes.io/projected/9a90ca63-d97d-4695-ad39-eaaa58204ecc-kube-api-access-5zhnm\") pod \"payload-pre-processing-e2e-shared-b-w2-ef434a-845d4b6b99-nqmzg\" (UID: \"9a90ca63-d97d-4695-ad39-eaaa58204ecc\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-ef434a-845d4b6b99-nqmzg" Nov 03 01:09:34.958982 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:09:34.949564 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7c2434b1-8471-41fb-b20d-118aa8f1afc4-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w2-ef434a-7b64fdc54f-9zxp5\" (UID: \"7c2434b1-8471-41fb-b20d-118aa8f1afc4\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w2-ef434a-7b64fdc54f-9zxp5" Nov 03 01:09:34.958982 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:09:34.949742 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gtst8\" (UniqueName: \"kubernetes.io/projected/7c2434b1-8471-41fb-b20d-118aa8f1afc4-kube-api-access-gtst8\") pod \"payload-processing-e2e-shared-b-w2-ef434a-7b64fdc54f-9zxp5\" (UID: \"7c2434b1-8471-41fb-b20d-118aa8f1afc4\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w2-ef434a-7b64fdc54f-9zxp5" Nov 03 01:09:34.958982 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:09:34.949794 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9a90ca63-d97d-4695-ad39-eaaa58204ecc-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w2-ef434a-845d4b6b99-nqmzg\" (UID: \"9a90ca63-d97d-4695-ad39-eaaa58204ecc\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-ef434a-845d4b6b99-nqmzg" Nov 03 01:09:35.106005 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:09:35.077606 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gtst8\" (UniqueName: \"kubernetes.io/projected/7c2434b1-8471-41fb-b20d-118aa8f1afc4-kube-api-access-gtst8\") pod \"payload-processing-e2e-shared-b-w2-ef434a-7b64fdc54f-9zxp5\" (UID: \"7c2434b1-8471-41fb-b20d-118aa8f1afc4\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w2-ef434a-7b64fdc54f-9zxp5" Nov 03 01:09:35.106005 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:09:35.077687 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9a90ca63-d97d-4695-ad39-eaaa58204ecc-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w2-ef434a-845d4b6b99-nqmzg\" (UID: \"9a90ca63-d97d-4695-ad39-eaaa58204ecc\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-ef434a-845d4b6b99-nqmzg" Nov 03 01:09:35.106005 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:09:35.077982 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5zhnm\" (UniqueName: \"kubernetes.io/projected/9a90ca63-d97d-4695-ad39-eaaa58204ecc-kube-api-access-5zhnm\") pod \"payload-pre-processing-e2e-shared-b-w2-ef434a-845d4b6b99-nqmzg\" (UID: \"9a90ca63-d97d-4695-ad39-eaaa58204ecc\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-ef434a-845d4b6b99-nqmzg" Nov 03 01:09:35.106005 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:09:35.078060 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7c2434b1-8471-41fb-b20d-118aa8f1afc4-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w2-ef434a-7b64fdc54f-9zxp5\" (UID: \"7c2434b1-8471-41fb-b20d-118aa8f1afc4\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w2-ef434a-7b64fdc54f-9zxp5" Nov 03 01:09:35.106005 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:09:35.099106 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9a90ca63-d97d-4695-ad39-eaaa58204ecc-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w2-ef434a-845d4b6b99-nqmzg\" (UID: \"9a90ca63-d97d-4695-ad39-eaaa58204ecc\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-ef434a-845d4b6b99-nqmzg" Nov 03 01:09:35.190935 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:09:35.180976 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7c2434b1-8471-41fb-b20d-118aa8f1afc4-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w2-ef434a-7b64fdc54f-9zxp5\" (UID: \"7c2434b1-8471-41fb-b20d-118aa8f1afc4\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w2-ef434a-7b64fdc54f-9zxp5" Nov 03 01:09:35.190935 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:09:35.187612 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:09:35.191660 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:09:35.191530 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gtst8\" (UniqueName: \"kubernetes.io/projected/7c2434b1-8471-41fb-b20d-118aa8f1afc4-kube-api-access-gtst8\") pod \"payload-processing-e2e-shared-b-w2-ef434a-7b64fdc54f-9zxp5\" (UID: \"7c2434b1-8471-41fb-b20d-118aa8f1afc4\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w2-ef434a-7b64fdc54f-9zxp5" Nov 03 01:09:35.213435 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:09:35.202877 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5zhnm\" (UniqueName: \"kubernetes.io/projected/9a90ca63-d97d-4695-ad39-eaaa58204ecc-kube-api-access-5zhnm\") pod \"payload-pre-processing-e2e-shared-b-w2-ef434a-845d4b6b99-nqmzg\" (UID: \"9a90ca63-d97d-4695-ad39-eaaa58204ecc\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-ef434a-845d4b6b99-nqmzg" Nov 03 01:09:35.352016 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:09:35.351974 4939 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-ef434a-7b64fdc54f-9zxp5" Nov 03 01:09:35.387550 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:09:35.387288 4939 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-ef434a-845d4b6b99-nqmzg" Nov 03 01:09:35.802519 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:09:35.802484 4939 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w2-ef434a-7b64fdc54f-9zxp5"] Nov 03 01:09:35.807303 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:09:35.807273 4939 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-ef434a-845d4b6b99-nqmzg"] Nov 03 01:09:35.807486 ip-10-0-140-107 kubenswrapper[4939]: W1103 01:09:35.807414 4939 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9a90ca63_d97d_4695_ad39_eaaa58204ecc.slice/crio-dc9a9782097b2362a92d87e984d3044451120d02f86b6117c342441f028c4560 WatchSource:0}: Error finding container dc9a9782097b2362a92d87e984d3044451120d02f86b6117c342441f028c4560: Status 404 returned error can't find the container with id dc9a9782097b2362a92d87e984d3044451120d02f86b6117c342441f028c4560 Nov 03 01:09:35.984807 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:09:35.984781 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:09:36.336980 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:09:36.336943 4939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-ef434a-845d4b6b99-nqmzg" event={"ID":"9a90ca63-d97d-4695-ad39-eaaa58204ecc","Type":"ContainerStarted","Data":"11f7251c363756a37e10e790158301d48898dab3f8155045c17c54b7e3e22883"} Nov 03 01:09:36.337191 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:09:36.337002 4939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-ef434a-845d4b6b99-nqmzg" event={"ID":"9a90ca63-d97d-4695-ad39-eaaa58204ecc","Type":"ContainerStarted","Data":"dc9a9782097b2362a92d87e984d3044451120d02f86b6117c342441f028c4560"} Nov 03 01:09:36.337191 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:09:36.337180 4939 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-ef434a-845d4b6b99-nqmzg" Nov 03 01:09:36.338967 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:09:36.338926 4939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-ef434a-7b64fdc54f-9zxp5" event={"ID":"7c2434b1-8471-41fb-b20d-118aa8f1afc4","Type":"ContainerStarted","Data":"c047ced06b4bc5d1024afd3d2f4221a8d559d5ccc61b216e9f8ed31aba73d6b2"} Nov 03 01:09:36.338967 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:09:36.338969 4939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-ef434a-7b64fdc54f-9zxp5" event={"ID":"7c2434b1-8471-41fb-b20d-118aa8f1afc4","Type":"ContainerStarted","Data":"4b6e30d8b7fb1dd70f5240dce880798681ef77caa960260919612a772f461738"} Nov 03 01:09:36.339186 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:09:36.339060 4939 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-ef434a-7b64fdc54f-9zxp5" Nov 03 01:09:36.355748 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:09:36.355692 4939 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-ef434a-845d4b6b99-nqmzg" podStartSLOduration=2.355677996 podStartE2EDuration="2.355677996s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 01:09:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 01:09:36.354113721 +0000 UTC m=+1724.618448680" watchObservedRunningTime="2026-11-03 01:09:36.355677996 +0000 UTC m=+1724.620012941" Nov 03 01:09:36.373089 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:09:36.373028 4939 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-ef434a-7b64fdc54f-9zxp5" podStartSLOduration=2.373011099 podStartE2EDuration="2.373011099s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 01:09:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 01:09:36.371662805 +0000 UTC m=+1724.635997751" watchObservedRunningTime="2026-11-03 01:09:36.373011099 +0000 UTC m=+1724.637346046" Nov 03 01:09:36.699359 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:09:36.699263 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:09:37.388305 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:09:37.388264 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:09:38.057399 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:09:38.057368 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:09:38.694624 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:09:38.694581 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:09:38.839936 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:09:38.807855 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:09:39.839965 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:09:39.839933 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:09:40.589735 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:09:40.589699 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:09:41.780936 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:09:41.778603 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:09:42.741940 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:09:42.734017 4939 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-shared-a-w4-6c0ec0/auth-test-model-6c0ec0-kserve-589c79bd69-vznsb" podUID="d21cb247-bacd-4af2-a7b1-d04b975f0993" containerName="main" probeResult="failure" output="Get \"https://10.132.0.186:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 01:09:42.741940 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:09:42.737002 4939 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-shared-b-w4-4a697f/auth-test-model-4a697f-kserve-84b7466bd5-48hbr" podUID="18846667-85a0-4bc2-8995-61bfb2ab1c5d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.194:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 01:09:42.801263 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:09:42.801220 4939 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-2f9a3a/simulator-w0-2f9a3a-kserve-9596bbb8f-2trgz" podUID="207e2207-fb6c-4076-95df-72e95671c261" containerName="main" probeResult="failure" output="Get \"https://10.132.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 01:09:42.841934 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:09:42.840995 4939 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-shared-a-w3-717217/auth-test-model-717217-kserve-5c9cb7dbd9-vl754" podUID="9634a946-0832-4901-ad01-bbb7a47f4ac7" containerName="main" probeResult="failure" output="Get \"https://10.132.0.191:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 01:09:42.841934 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:09:42.841417 4939 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-f7680d/premium-w2-f7680d-kserve-76565b8cf9-hqbbt" podUID="4712e708-18a2-4a13-a12c-a5669397b2c3" containerName="main" probeResult="failure" output="Get \"https://10.132.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 01:09:42.842264 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:09:42.842077 4939 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-f7680d/distinct-2-w2-f7680d-kserve-7d7c4564bf-cbv2c" podUID="2de18086-761b-4773-b9fc-15ddc4a220fc" containerName="main" probeResult="failure" output="Get \"https://10.132.0.75:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 01:09:42.871932 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:09:42.861005 4939 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-2f9a3a/distinct-w0-2f9a3a-kserve-555565cfcf-jtcr9" podUID="ac0ece06-8718-463f-915f-5315988d9711" containerName="main" probeResult="failure" output="Get \"https://10.132.0.76:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 01:09:43.052363 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:09:43.052248 4939 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-f7680d/distinct-w2-f7680d-kserve-756f874b98-qnpv8" podUID="836753fc-f9ef-4f61-b3fe-e5b92b7ee27b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.70:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 01:09:43.079106 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:09:43.079052 4939 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-88bbf7/distinct-2-w1-88bbf7-kserve-5457cdcdbd-l6tfd" podUID="a397931b-faa9-4c2e-8290-491350546663" containerName="main" probeResult="failure" output="Get \"https://10.132.0.85:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 01:09:43.126085 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:09:43.126041 4939 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-88bbf7/simulator-w1-88bbf7-kserve-5fbb669f8f-42hgd" podUID="83b38883-2e7a-444d-aa68-52be4a25ae66" containerName="main" probeResult="failure" output="Get \"https://10.132.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 01:09:43.433049 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:09:43.432988 4939 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-shared-b-w0-020af9/auth-test-model-020af9-kserve-844df75d9c-ljr9s" podUID="24fb3670-ec70-492c-a573-4cbe174f1ccf" containerName="main" probeResult="failure" output="Get \"https://10.132.0.198:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 01:09:43.531302 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:09:43.531258 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:09:43.706251 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:09:43.706140 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:09:44.363054 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:09:44.363019 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:09:45.252596 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:09:45.252546 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:09:45.886610 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:09:45.886572 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:09:46.479705 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:09:46.479664 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:09:47.230603 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:09:47.230566 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:09:47.349119 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:09:47.349081 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:09:47.381019 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:09:47.379832 4939 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-ef434a-845d4b6b99-nqmzg" Nov 03 01:09:47.400674 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:09:47.400631 4939 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-ef434a-7b64fdc54f-9zxp5" Nov 03 01:09:48.348530 ip-10-0-140-107 kubenswrapper[4939]: E1103 01:09:48.348486 4939 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-sthjh" containerName="extractor" Nov 03 01:09:48.430859 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:09:48.430827 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:09:49.065211 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:09:49.065121 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:09:49.286732 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:09:49.286698 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:09:49.900972 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:09:49.900938 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:09:50.611219 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:09:50.611180 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:09:51.294459 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:09:51.294425 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:09:52.082005 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:09:52.081971 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:09:52.793501 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:09:52.793461 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:09:53.490472 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:09:53.490435 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:09:54.713229 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:09:54.713193 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:09:55.517131 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:09:55.517098 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:09:56.313494 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:09:56.313457 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:09:56.439520 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:09:56.439419 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:09:57.107141 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:09:57.107107 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:09:57.646306 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:09:57.646256 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:09:57.825847 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:09:57.825810 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:09:58.015802 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:09:58.015430 4939 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-extmod-b-af8244-openshift-default-594d5d565c-4f5nx"] Nov 03 01:09:58.022979 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:09:58.022951 4939 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-af8244-openshift-default-594d5d565c-4f5nx" Nov 03 01:09:58.032600 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:09:58.031387 4939 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-extmod-b-af8244-openshift-default-dockercfg-7hb8r\"" Nov 03 01:09:58.042784 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:09:58.042750 4939 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-b-af8244-openshift-default-594d5d565c-4f5nx"] Nov 03 01:09:58.104163 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:09:58.104123 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:09:58.180934 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:09:58.169283 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/efde608b-c334-4d17-a0aa-cad98a95a999-istio-token\") pod \"e2e-extmod-b-af8244-openshift-default-594d5d565c-4f5nx\" (UID: \"efde608b-c334-4d17-a0aa-cad98a95a999\") " pod="openshift-ingress/e2e-extmod-b-af8244-openshift-default-594d5d565c-4f5nx" Nov 03 01:09:58.190932 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:09:58.181314 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/efde608b-c334-4d17-a0aa-cad98a95a999-istio-podinfo\") pod \"e2e-extmod-b-af8244-openshift-default-594d5d565c-4f5nx\" (UID: \"efde608b-c334-4d17-a0aa-cad98a95a999\") " pod="openshift-ingress/e2e-extmod-b-af8244-openshift-default-594d5d565c-4f5nx" Nov 03 01:09:58.191288 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:09:58.191257 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fn4f9\" (UniqueName: \"kubernetes.io/projected/efde608b-c334-4d17-a0aa-cad98a95a999-kube-api-access-fn4f9\") pod \"e2e-extmod-b-af8244-openshift-default-594d5d565c-4f5nx\" (UID: \"efde608b-c334-4d17-a0aa-cad98a95a999\") " pod="openshift-ingress/e2e-extmod-b-af8244-openshift-default-594d5d565c-4f5nx" Nov 03 01:09:58.191544 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:09:58.191523 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/efde608b-c334-4d17-a0aa-cad98a95a999-workload-certs\") pod \"e2e-extmod-b-af8244-openshift-default-594d5d565c-4f5nx\" (UID: \"efde608b-c334-4d17-a0aa-cad98a95a999\") " pod="openshift-ingress/e2e-extmod-b-af8244-openshift-default-594d5d565c-4f5nx" Nov 03 01:09:58.191727 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:09:58.191709 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/efde608b-c334-4d17-a0aa-cad98a95a999-istio-data\") pod \"e2e-extmod-b-af8244-openshift-default-594d5d565c-4f5nx\" (UID: \"efde608b-c334-4d17-a0aa-cad98a95a999\") " pod="openshift-ingress/e2e-extmod-b-af8244-openshift-default-594d5d565c-4f5nx" Nov 03 01:09:58.198933 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:09:58.191923 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/efde608b-c334-4d17-a0aa-cad98a95a999-credential-socket\") pod \"e2e-extmod-b-af8244-openshift-default-594d5d565c-4f5nx\" (UID: \"efde608b-c334-4d17-a0aa-cad98a95a999\") " pod="openshift-ingress/e2e-extmod-b-af8244-openshift-default-594d5d565c-4f5nx" Nov 03 01:09:58.199257 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:09:58.199177 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/efde608b-c334-4d17-a0aa-cad98a95a999-istio-envoy\") pod \"e2e-extmod-b-af8244-openshift-default-594d5d565c-4f5nx\" (UID: \"efde608b-c334-4d17-a0aa-cad98a95a999\") " pod="openshift-ingress/e2e-extmod-b-af8244-openshift-default-594d5d565c-4f5nx" Nov 03 01:09:58.199640 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:09:58.199618 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/efde608b-c334-4d17-a0aa-cad98a95a999-workload-socket\") pod \"e2e-extmod-b-af8244-openshift-default-594d5d565c-4f5nx\" (UID: \"efde608b-c334-4d17-a0aa-cad98a95a999\") " pod="openshift-ingress/e2e-extmod-b-af8244-openshift-default-594d5d565c-4f5nx" Nov 03 01:09:58.199934 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:09:58.199894 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/efde608b-c334-4d17-a0aa-cad98a95a999-istiod-ca-cert\") pod \"e2e-extmod-b-af8244-openshift-default-594d5d565c-4f5nx\" (UID: \"efde608b-c334-4d17-a0aa-cad98a95a999\") " pod="openshift-ingress/e2e-extmod-b-af8244-openshift-default-594d5d565c-4f5nx" Nov 03 01:09:58.307116 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:09:58.307023 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/efde608b-c334-4d17-a0aa-cad98a95a999-workload-socket\") pod \"e2e-extmod-b-af8244-openshift-default-594d5d565c-4f5nx\" (UID: \"efde608b-c334-4d17-a0aa-cad98a95a999\") " pod="openshift-ingress/e2e-extmod-b-af8244-openshift-default-594d5d565c-4f5nx" Nov 03 01:09:58.307926 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:09:58.307884 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/efde608b-c334-4d17-a0aa-cad98a95a999-istiod-ca-cert\") pod \"e2e-extmod-b-af8244-openshift-default-594d5d565c-4f5nx\" (UID: \"efde608b-c334-4d17-a0aa-cad98a95a999\") " pod="openshift-ingress/e2e-extmod-b-af8244-openshift-default-594d5d565c-4f5nx" Nov 03 01:09:58.309195 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:09:58.309170 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/efde608b-c334-4d17-a0aa-cad98a95a999-istio-token\") pod \"e2e-extmod-b-af8244-openshift-default-594d5d565c-4f5nx\" (UID: \"efde608b-c334-4d17-a0aa-cad98a95a999\") " pod="openshift-ingress/e2e-extmod-b-af8244-openshift-default-594d5d565c-4f5nx" Nov 03 01:09:58.322755 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:09:58.322723 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/efde608b-c334-4d17-a0aa-cad98a95a999-istio-podinfo\") pod \"e2e-extmod-b-af8244-openshift-default-594d5d565c-4f5nx\" (UID: \"efde608b-c334-4d17-a0aa-cad98a95a999\") " pod="openshift-ingress/e2e-extmod-b-af8244-openshift-default-594d5d565c-4f5nx" Nov 03 01:09:58.323014 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:09:58.322990 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fn4f9\" (UniqueName: \"kubernetes.io/projected/efde608b-c334-4d17-a0aa-cad98a95a999-kube-api-access-fn4f9\") pod \"e2e-extmod-b-af8244-openshift-default-594d5d565c-4f5nx\" (UID: \"efde608b-c334-4d17-a0aa-cad98a95a999\") " pod="openshift-ingress/e2e-extmod-b-af8244-openshift-default-594d5d565c-4f5nx" Nov 03 01:09:58.324008 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:09:58.323985 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/efde608b-c334-4d17-a0aa-cad98a95a999-workload-certs\") pod \"e2e-extmod-b-af8244-openshift-default-594d5d565c-4f5nx\" (UID: \"efde608b-c334-4d17-a0aa-cad98a95a999\") " pod="openshift-ingress/e2e-extmod-b-af8244-openshift-default-594d5d565c-4f5nx" Nov 03 01:09:58.324247 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:09:58.324228 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/efde608b-c334-4d17-a0aa-cad98a95a999-istio-data\") pod \"e2e-extmod-b-af8244-openshift-default-594d5d565c-4f5nx\" (UID: \"efde608b-c334-4d17-a0aa-cad98a95a999\") " pod="openshift-ingress/e2e-extmod-b-af8244-openshift-default-594d5d565c-4f5nx" Nov 03 01:09:58.324431 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:09:58.324416 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/efde608b-c334-4d17-a0aa-cad98a95a999-credential-socket\") pod \"e2e-extmod-b-af8244-openshift-default-594d5d565c-4f5nx\" (UID: \"efde608b-c334-4d17-a0aa-cad98a95a999\") " pod="openshift-ingress/e2e-extmod-b-af8244-openshift-default-594d5d565c-4f5nx" Nov 03 01:09:58.324557 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:09:58.324543 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/efde608b-c334-4d17-a0aa-cad98a95a999-istio-envoy\") pod \"e2e-extmod-b-af8244-openshift-default-594d5d565c-4f5nx\" (UID: \"efde608b-c334-4d17-a0aa-cad98a95a999\") " pod="openshift-ingress/e2e-extmod-b-af8244-openshift-default-594d5d565c-4f5nx" Nov 03 01:09:58.325239 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:09:58.307779 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/efde608b-c334-4d17-a0aa-cad98a95a999-workload-socket\") pod \"e2e-extmod-b-af8244-openshift-default-594d5d565c-4f5nx\" (UID: \"efde608b-c334-4d17-a0aa-cad98a95a999\") " pod="openshift-ingress/e2e-extmod-b-af8244-openshift-default-594d5d565c-4f5nx" Nov 03 01:09:58.325380 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:09:58.309034 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/efde608b-c334-4d17-a0aa-cad98a95a999-istiod-ca-cert\") pod \"e2e-extmod-b-af8244-openshift-default-594d5d565c-4f5nx\" (UID: \"efde608b-c334-4d17-a0aa-cad98a95a999\") " pod="openshift-ingress/e2e-extmod-b-af8244-openshift-default-594d5d565c-4f5nx" Nov 03 01:09:58.325758 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:09:58.325738 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/efde608b-c334-4d17-a0aa-cad98a95a999-workload-certs\") pod \"e2e-extmod-b-af8244-openshift-default-594d5d565c-4f5nx\" (UID: \"efde608b-c334-4d17-a0aa-cad98a95a999\") " pod="openshift-ingress/e2e-extmod-b-af8244-openshift-default-594d5d565c-4f5nx" Nov 03 01:09:58.326142 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:09:58.326121 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/efde608b-c334-4d17-a0aa-cad98a95a999-istio-data\") pod \"e2e-extmod-b-af8244-openshift-default-594d5d565c-4f5nx\" (UID: \"efde608b-c334-4d17-a0aa-cad98a95a999\") " pod="openshift-ingress/e2e-extmod-b-af8244-openshift-default-594d5d565c-4f5nx" Nov 03 01:09:58.326550 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:09:58.326526 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/efde608b-c334-4d17-a0aa-cad98a95a999-credential-socket\") pod \"e2e-extmod-b-af8244-openshift-default-594d5d565c-4f5nx\" (UID: \"efde608b-c334-4d17-a0aa-cad98a95a999\") " pod="openshift-ingress/e2e-extmod-b-af8244-openshift-default-594d5d565c-4f5nx" Nov 03 01:09:58.372876 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:09:58.372780 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/efde608b-c334-4d17-a0aa-cad98a95a999-istio-podinfo\") pod \"e2e-extmod-b-af8244-openshift-default-594d5d565c-4f5nx\" (UID: \"efde608b-c334-4d17-a0aa-cad98a95a999\") " pod="openshift-ingress/e2e-extmod-b-af8244-openshift-default-594d5d565c-4f5nx" Nov 03 01:09:58.382287 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:09:58.382252 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/efde608b-c334-4d17-a0aa-cad98a95a999-istio-envoy\") pod \"e2e-extmod-b-af8244-openshift-default-594d5d565c-4f5nx\" (UID: \"efde608b-c334-4d17-a0aa-cad98a95a999\") " pod="openshift-ingress/e2e-extmod-b-af8244-openshift-default-594d5d565c-4f5nx" Nov 03 01:09:58.395598 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:09:58.395564 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/efde608b-c334-4d17-a0aa-cad98a95a999-istio-token\") pod \"e2e-extmod-b-af8244-openshift-default-594d5d565c-4f5nx\" (UID: \"efde608b-c334-4d17-a0aa-cad98a95a999\") " pod="openshift-ingress/e2e-extmod-b-af8244-openshift-default-594d5d565c-4f5nx" Nov 03 01:09:58.408795 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:09:58.408761 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:09:58.419931 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:09:58.409899 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fn4f9\" (UniqueName: \"kubernetes.io/projected/efde608b-c334-4d17-a0aa-cad98a95a999-kube-api-access-fn4f9\") pod \"e2e-extmod-b-af8244-openshift-default-594d5d565c-4f5nx\" (UID: \"efde608b-c334-4d17-a0aa-cad98a95a999\") " pod="openshift-ingress/e2e-extmod-b-af8244-openshift-default-594d5d565c-4f5nx" Nov 03 01:09:58.648327 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:09:58.648284 4939 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-af8244-openshift-default-594d5d565c-4f5nx" Nov 03 01:09:59.022508 ip-10-0-140-107 kubenswrapper[4939]: W1103 01:09:59.022470 4939 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podefde608b_c334_4d17_a0aa_cad98a95a999.slice/crio-2b68b0c8553e06e22edc362642aa3f346dd654dd1a1452e7424336802726c284 WatchSource:0}: Error finding container 2b68b0c8553e06e22edc362642aa3f346dd654dd1a1452e7424336802726c284: Status 404 returned error can't find the container with id 2b68b0c8553e06e22edc362642aa3f346dd654dd1a1452e7424336802726c284 Nov 03 01:09:59.022960 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:09:59.022935 4939 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-b-af8244-openshift-default-594d5d565c-4f5nx"] Nov 03 01:09:59.025828 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:09:59.025717 4939 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 01:09:59.025828 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:09:59.025809 4939 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 01:09:59.025987 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:09:59.025848 4939 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 01:09:59.088521 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:09:59.088482 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:09:59.917208 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:09:59.917094 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:10:00.002109 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:10:00.002057 4939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-af8244-openshift-default-594d5d565c-4f5nx" event={"ID":"efde608b-c334-4d17-a0aa-cad98a95a999","Type":"ContainerStarted","Data":"be600a011fd4e99c6ae1e3f5111b289f859d31f27a7639165b30f43a4bbe3776"} Nov 03 01:10:00.002637 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:10:00.002601 4939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-af8244-openshift-default-594d5d565c-4f5nx" event={"ID":"efde608b-c334-4d17-a0aa-cad98a95a999","Type":"ContainerStarted","Data":"2b68b0c8553e06e22edc362642aa3f346dd654dd1a1452e7424336802726c284"} Nov 03 01:10:00.030175 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:10:00.030122 4939 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-extmod-b-af8244-openshift-default-594d5d565c-4f5nx" podStartSLOduration=3.030107765 podStartE2EDuration="3.030107765s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 01:09:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 01:10:00.026819182 +0000 UTC m=+1748.291154127" watchObservedRunningTime="2026-11-03 01:10:00.030107765 +0000 UTC m=+1748.294442730" Nov 03 01:10:00.626617 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:10:00.626580 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:10:00.677002 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:10:00.676034 4939 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-extmod-b-af8244-openshift-default-594d5d565c-4f5nx" Nov 03 01:10:00.716308 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:10:00.716261 4939 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-extmod-b-af8244-openshift-default-594d5d565c-4f5nx" Nov 03 01:10:01.010274 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:10:01.010176 4939 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-b-af8244-openshift-default-594d5d565c-4f5nx" Nov 03 01:10:01.011216 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:10:01.011194 4939 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-extmod-b-af8244-openshift-default-594d5d565c-4f5nx" Nov 03 01:10:01.659031 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:10:01.658998 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:10:01.875861 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:10:01.875831 4939 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-4pw7v"] Nov 03 01:10:02.361413 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:10:02.361374 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:10:02.501277 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:10:02.501239 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:10:03.638131 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:10:03.638010 4939 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-shared-b-w3-556c14/auth-test-model-556c14-kserve-8d5cd469-t6tzc" podUID="0c4e579c-9fe2-4ec9-b8f4-62758e519980" containerName="main" probeResult="failure" output="Get \"https://10.132.0.195:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 01:10:03.772930 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:10:03.768476 4939 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-shared-b-w4-4a697f/auth-test-model-4a697f-kserve-84b7466bd5-48hbr" podUID="18846667-85a0-4bc2-8995-61bfb2ab1c5d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.194:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 01:10:03.853933 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:10:03.853011 4939 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-f7680d/premium-w2-f7680d-kserve-76565b8cf9-hqbbt" podUID="4712e708-18a2-4a13-a12c-a5669397b2c3" containerName="main" probeResult="failure" output="Get \"https://10.132.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 01:10:03.882929 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:10:03.882386 4939 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-f7680d/distinct-2-w2-f7680d-kserve-7d7c4564bf-cbv2c" podUID="2de18086-761b-4773-b9fc-15ddc4a220fc" containerName="main" probeResult="failure" output="Get \"https://10.132.0.75:8000/ready\": context deadline exceeded" Nov 03 01:10:03.931037 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:10:03.927409 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:10:04.057935 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:10:04.052999 4939 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-f7680d/distinct-w2-f7680d-kserve-756f874b98-qnpv8" podUID="836753fc-f9ef-4f61-b3fe-e5b92b7ee27b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.70:8000/ready\": context deadline exceeded" Nov 03 01:10:04.213989 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:10:04.206018 4939 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-2f9a3a/simulator-w0-2f9a3a-kserve-9596bbb8f-2trgz" podUID="207e2207-fb6c-4076-95df-72e95671c261" containerName="main" probeResult="failure" output="Get \"https://10.132.0.61:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 01:10:04.236936 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:10:04.233028 4939 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-2f9a3a/distinct-w0-2f9a3a-kserve-555565cfcf-jtcr9" podUID="ac0ece06-8718-463f-915f-5315988d9711" containerName="main" probeResult="failure" output="Get \"https://10.132.0.76:8000/health\": context deadline exceeded" Nov 03 01:10:04.416270 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:10:04.416085 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:10:04.645581 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:10:04.645532 4939 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-shared-b-w3-556c14/auth-test-model-556c14-kserve-8d5cd469-t6tzc" podUID="0c4e579c-9fe2-4ec9-b8f4-62758e519980" containerName="main" probeResult="failure" output="Get \"https://10.132.0.195:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Nov 03 01:10:04.646613 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:10:04.646588 4939 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-w3-556c14/auth-test-model-556c14-kserve-8d5cd469-t6tzc" Nov 03 01:10:04.683039 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:10:04.683007 4939 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-w3-556c14/auth-test-model-556c14-kserve-8d5cd469-t6tzc" Nov 03 01:10:04.977381 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:10:04.975975 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:10:04.995857 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:10:04.995822 4939 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-af8244-84f67c568c-m9kgf"] Nov 03 01:10:05.035764 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:10:05.035721 4939 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-af8244-84f67c568c-m9kgf"] Nov 03 01:10:05.036013 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:10:05.035822 4939 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-af8244-84f67c568c-m9kgf" Nov 03 01:10:05.040348 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:10:05.040316 4939 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-extmod-b-af8244-dockercfg-nfd57\"" Nov 03 01:10:05.040492 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:10:05.040395 4939 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-extmod-b-af8244\"" Nov 03 01:10:05.045756 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:10:05.045732 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1f0db3da-dcf7-459b-994e-a7081e6f0958-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-b-af8244-84f67c568c-m9kgf\" (UID: \"1f0db3da-dcf7-459b-994e-a7081e6f0958\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-af8244-84f67c568c-m9kgf" Nov 03 01:10:05.084544 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:10:05.084468 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xb8cj\" (UniqueName: \"kubernetes.io/projected/1f0db3da-dcf7-459b-994e-a7081e6f0958-kube-api-access-xb8cj\") pod \"payload-pre-processing-e2e-extmod-b-af8244-84f67c568c-m9kgf\" (UID: \"1f0db3da-dcf7-459b-994e-a7081e6f0958\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-af8244-84f67c568c-m9kgf" Nov 03 01:10:05.101424 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:10:05.100507 4939 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-b-af8244-75d9474dd7-pr6cd"] Nov 03 01:10:05.112362 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:10:05.107008 4939 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-b-af8244-75d9474dd7-pr6cd" Nov 03 01:10:05.127952 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:10:05.117308 4939 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-b-af8244-75d9474dd7-pr6cd"] Nov 03 01:10:05.207973 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:10:05.203219 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:10:05.246994 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:10:05.232202 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xb8cj\" (UniqueName: \"kubernetes.io/projected/1f0db3da-dcf7-459b-994e-a7081e6f0958-kube-api-access-xb8cj\") pod \"payload-pre-processing-e2e-extmod-b-af8244-84f67c568c-m9kgf\" (UID: \"1f0db3da-dcf7-459b-994e-a7081e6f0958\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-af8244-84f67c568c-m9kgf" Nov 03 01:10:05.246994 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:10:05.232337 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jfltb\" (UniqueName: \"kubernetes.io/projected/d1f73b7c-084a-44b7-a3bd-c5042532a732-kube-api-access-jfltb\") pod \"payload-processing-e2e-extmod-b-af8244-75d9474dd7-pr6cd\" (UID: \"d1f73b7c-084a-44b7-a3bd-c5042532a732\") " pod="openshift-ingress/payload-processing-e2e-extmod-b-af8244-75d9474dd7-pr6cd" Nov 03 01:10:05.246994 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:10:05.232572 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d1f73b7c-084a-44b7-a3bd-c5042532a732-plugins-config-volume\") pod \"payload-processing-e2e-extmod-b-af8244-75d9474dd7-pr6cd\" (UID: \"d1f73b7c-084a-44b7-a3bd-c5042532a732\") " pod="openshift-ingress/payload-processing-e2e-extmod-b-af8244-75d9474dd7-pr6cd" Nov 03 01:10:05.246994 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:10:05.232636 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1f0db3da-dcf7-459b-994e-a7081e6f0958-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-b-af8244-84f67c568c-m9kgf\" (UID: \"1f0db3da-dcf7-459b-994e-a7081e6f0958\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-af8244-84f67c568c-m9kgf" Nov 03 01:10:05.246994 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:10:05.234085 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1f0db3da-dcf7-459b-994e-a7081e6f0958-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-b-af8244-84f67c568c-m9kgf\" (UID: \"1f0db3da-dcf7-459b-994e-a7081e6f0958\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-af8244-84f67c568c-m9kgf" Nov 03 01:10:05.365349 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:10:05.365306 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jfltb\" (UniqueName: \"kubernetes.io/projected/d1f73b7c-084a-44b7-a3bd-c5042532a732-kube-api-access-jfltb\") pod \"payload-processing-e2e-extmod-b-af8244-75d9474dd7-pr6cd\" (UID: \"d1f73b7c-084a-44b7-a3bd-c5042532a732\") " pod="openshift-ingress/payload-processing-e2e-extmod-b-af8244-75d9474dd7-pr6cd" Nov 03 01:10:05.365786 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:10:05.365762 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d1f73b7c-084a-44b7-a3bd-c5042532a732-plugins-config-volume\") pod \"payload-processing-e2e-extmod-b-af8244-75d9474dd7-pr6cd\" (UID: \"d1f73b7c-084a-44b7-a3bd-c5042532a732\") " pod="openshift-ingress/payload-processing-e2e-extmod-b-af8244-75d9474dd7-pr6cd" Nov 03 01:10:05.378933 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:10:05.367145 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d1f73b7c-084a-44b7-a3bd-c5042532a732-plugins-config-volume\") pod \"payload-processing-e2e-extmod-b-af8244-75d9474dd7-pr6cd\" (UID: \"d1f73b7c-084a-44b7-a3bd-c5042532a732\") " pod="openshift-ingress/payload-processing-e2e-extmod-b-af8244-75d9474dd7-pr6cd" Nov 03 01:10:05.446935 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:10:05.440633 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xb8cj\" (UniqueName: \"kubernetes.io/projected/1f0db3da-dcf7-459b-994e-a7081e6f0958-kube-api-access-xb8cj\") pod \"payload-pre-processing-e2e-extmod-b-af8244-84f67c568c-m9kgf\" (UID: \"1f0db3da-dcf7-459b-994e-a7081e6f0958\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-af8244-84f67c568c-m9kgf" Nov 03 01:10:05.456948 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:10:05.456125 4939 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-af8244-84f67c568c-m9kgf" Nov 03 01:10:05.761726 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:10:05.761641 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jfltb\" (UniqueName: \"kubernetes.io/projected/d1f73b7c-084a-44b7-a3bd-c5042532a732-kube-api-access-jfltb\") pod \"payload-processing-e2e-extmod-b-af8244-75d9474dd7-pr6cd\" (UID: \"d1f73b7c-084a-44b7-a3bd-c5042532a732\") " pod="openshift-ingress/payload-processing-e2e-extmod-b-af8244-75d9474dd7-pr6cd" Nov 03 01:10:05.793676 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:10:05.793635 4939 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-b-af8244-75d9474dd7-pr6cd" Nov 03 01:10:06.077695 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:10:06.077660 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:10:06.085182 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:10:06.083605 4939 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-af8244-84f67c568c-m9kgf"] Nov 03 01:10:06.085182 ip-10-0-140-107 kubenswrapper[4939]: W1103 01:10:06.084419 4939 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1f0db3da_dcf7_459b_994e_a7081e6f0958.slice/crio-59ae3b21edab922dd4ad5111e7c776e38e208f20099ea2868c3e243e2e7197bc WatchSource:0}: Error finding container 59ae3b21edab922dd4ad5111e7c776e38e208f20099ea2868c3e243e2e7197bc: Status 404 returned error can't find the container with id 59ae3b21edab922dd4ad5111e7c776e38e208f20099ea2868c3e243e2e7197bc Nov 03 01:10:06.181312 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:10:06.181279 4939 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-b-af8244-75d9474dd7-pr6cd"] Nov 03 01:10:06.183618 ip-10-0-140-107 kubenswrapper[4939]: W1103 01:10:06.183595 4939 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd1f73b7c_084a_44b7_a3bd_c5042532a732.slice/crio-87e3e43af5dcf61e99c104b70b58c3918bb7cad0b95c9257e96b9793b280bc89 WatchSource:0}: Error finding container 87e3e43af5dcf61e99c104b70b58c3918bb7cad0b95c9257e96b9793b280bc89: Status 404 returned error can't find the container with id 87e3e43af5dcf61e99c104b70b58c3918bb7cad0b95c9257e96b9793b280bc89 Nov 03 01:10:06.274955 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:10:06.274648 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:10:06.450935 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:10:06.447813 4939 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-extmod-a-027c19-openshift-default-58d5bc7645-zjnf8"] Nov 03 01:10:06.511235 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:10:06.510744 4939 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-a-027c19-openshift-default-58d5bc7645-zjnf8"] Nov 03 01:10:06.511235 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:10:06.510865 4939 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-027c19-openshift-default-58d5bc7645-zjnf8" Nov 03 01:10:06.518281 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:10:06.518243 4939 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-extmod-a-027c19-openshift-default-dockercfg-6n7ft\"" Nov 03 01:10:06.531774 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:10:06.531740 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:10:06.798021 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:10:06.796176 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f09e7c2c-246e-4c96-a8ca-147e8ca83560-istio-data\") pod \"e2e-extmod-a-027c19-openshift-default-58d5bc7645-zjnf8\" (UID: \"f09e7c2c-246e-4c96-a8ca-147e8ca83560\") " pod="openshift-ingress/e2e-extmod-a-027c19-openshift-default-58d5bc7645-zjnf8" Nov 03 01:10:06.798021 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:10:06.796308 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dtzkd\" (UniqueName: \"kubernetes.io/projected/f09e7c2c-246e-4c96-a8ca-147e8ca83560-kube-api-access-dtzkd\") pod \"e2e-extmod-a-027c19-openshift-default-58d5bc7645-zjnf8\" (UID: \"f09e7c2c-246e-4c96-a8ca-147e8ca83560\") " pod="openshift-ingress/e2e-extmod-a-027c19-openshift-default-58d5bc7645-zjnf8" Nov 03 01:10:06.798021 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:10:06.796389 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f09e7c2c-246e-4c96-a8ca-147e8ca83560-workload-certs\") pod \"e2e-extmod-a-027c19-openshift-default-58d5bc7645-zjnf8\" (UID: \"f09e7c2c-246e-4c96-a8ca-147e8ca83560\") " pod="openshift-ingress/e2e-extmod-a-027c19-openshift-default-58d5bc7645-zjnf8" Nov 03 01:10:06.798021 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:10:06.796479 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f09e7c2c-246e-4c96-a8ca-147e8ca83560-istio-envoy\") pod \"e2e-extmod-a-027c19-openshift-default-58d5bc7645-zjnf8\" (UID: \"f09e7c2c-246e-4c96-a8ca-147e8ca83560\") " pod="openshift-ingress/e2e-extmod-a-027c19-openshift-default-58d5bc7645-zjnf8" Nov 03 01:10:06.798021 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:10:06.796600 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f09e7c2c-246e-4c96-a8ca-147e8ca83560-credential-socket\") pod \"e2e-extmod-a-027c19-openshift-default-58d5bc7645-zjnf8\" (UID: \"f09e7c2c-246e-4c96-a8ca-147e8ca83560\") " pod="openshift-ingress/e2e-extmod-a-027c19-openshift-default-58d5bc7645-zjnf8" Nov 03 01:10:06.798021 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:10:06.796646 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f09e7c2c-246e-4c96-a8ca-147e8ca83560-istio-token\") pod \"e2e-extmod-a-027c19-openshift-default-58d5bc7645-zjnf8\" (UID: \"f09e7c2c-246e-4c96-a8ca-147e8ca83560\") " pod="openshift-ingress/e2e-extmod-a-027c19-openshift-default-58d5bc7645-zjnf8" Nov 03 01:10:06.798021 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:10:06.796778 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f09e7c2c-246e-4c96-a8ca-147e8ca83560-workload-socket\") pod \"e2e-extmod-a-027c19-openshift-default-58d5bc7645-zjnf8\" (UID: \"f09e7c2c-246e-4c96-a8ca-147e8ca83560\") " pod="openshift-ingress/e2e-extmod-a-027c19-openshift-default-58d5bc7645-zjnf8" Nov 03 01:10:06.798021 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:10:06.796809 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f09e7c2c-246e-4c96-a8ca-147e8ca83560-istio-podinfo\") pod \"e2e-extmod-a-027c19-openshift-default-58d5bc7645-zjnf8\" (UID: \"f09e7c2c-246e-4c96-a8ca-147e8ca83560\") " pod="openshift-ingress/e2e-extmod-a-027c19-openshift-default-58d5bc7645-zjnf8" Nov 03 01:10:06.798021 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:10:06.796842 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f09e7c2c-246e-4c96-a8ca-147e8ca83560-istiod-ca-cert\") pod \"e2e-extmod-a-027c19-openshift-default-58d5bc7645-zjnf8\" (UID: \"f09e7c2c-246e-4c96-a8ca-147e8ca83560\") " pod="openshift-ingress/e2e-extmod-a-027c19-openshift-default-58d5bc7645-zjnf8" Nov 03 01:10:06.858936 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:10:06.847724 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:10:07.008760 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:10:07.004673 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f09e7c2c-246e-4c96-a8ca-147e8ca83560-istio-data\") pod \"e2e-extmod-a-027c19-openshift-default-58d5bc7645-zjnf8\" (UID: \"f09e7c2c-246e-4c96-a8ca-147e8ca83560\") " pod="openshift-ingress/e2e-extmod-a-027c19-openshift-default-58d5bc7645-zjnf8" Nov 03 01:10:07.014113 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:10:07.014077 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dtzkd\" (UniqueName: \"kubernetes.io/projected/f09e7c2c-246e-4c96-a8ca-147e8ca83560-kube-api-access-dtzkd\") pod \"e2e-extmod-a-027c19-openshift-default-58d5bc7645-zjnf8\" (UID: \"f09e7c2c-246e-4c96-a8ca-147e8ca83560\") " pod="openshift-ingress/e2e-extmod-a-027c19-openshift-default-58d5bc7645-zjnf8" Nov 03 01:10:07.014401 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:10:07.014386 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f09e7c2c-246e-4c96-a8ca-147e8ca83560-workload-certs\") pod \"e2e-extmod-a-027c19-openshift-default-58d5bc7645-zjnf8\" (UID: \"f09e7c2c-246e-4c96-a8ca-147e8ca83560\") " pod="openshift-ingress/e2e-extmod-a-027c19-openshift-default-58d5bc7645-zjnf8" Nov 03 01:10:07.020935 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:10:07.015135 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f09e7c2c-246e-4c96-a8ca-147e8ca83560-istio-envoy\") pod \"e2e-extmod-a-027c19-openshift-default-58d5bc7645-zjnf8\" (UID: \"f09e7c2c-246e-4c96-a8ca-147e8ca83560\") " pod="openshift-ingress/e2e-extmod-a-027c19-openshift-default-58d5bc7645-zjnf8" Nov 03 01:10:07.021256 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:10:07.021230 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f09e7c2c-246e-4c96-a8ca-147e8ca83560-credential-socket\") pod \"e2e-extmod-a-027c19-openshift-default-58d5bc7645-zjnf8\" (UID: \"f09e7c2c-246e-4c96-a8ca-147e8ca83560\") " pod="openshift-ingress/e2e-extmod-a-027c19-openshift-default-58d5bc7645-zjnf8" Nov 03 01:10:07.021420 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:10:07.021405 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f09e7c2c-246e-4c96-a8ca-147e8ca83560-istio-token\") pod \"e2e-extmod-a-027c19-openshift-default-58d5bc7645-zjnf8\" (UID: \"f09e7c2c-246e-4c96-a8ca-147e8ca83560\") " pod="openshift-ingress/e2e-extmod-a-027c19-openshift-default-58d5bc7645-zjnf8" Nov 03 01:10:07.022885 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:10:07.021676 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f09e7c2c-246e-4c96-a8ca-147e8ca83560-workload-socket\") pod \"e2e-extmod-a-027c19-openshift-default-58d5bc7645-zjnf8\" (UID: \"f09e7c2c-246e-4c96-a8ca-147e8ca83560\") " pod="openshift-ingress/e2e-extmod-a-027c19-openshift-default-58d5bc7645-zjnf8" Nov 03 01:10:07.029012 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:10:07.028979 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f09e7c2c-246e-4c96-a8ca-147e8ca83560-istio-podinfo\") pod \"e2e-extmod-a-027c19-openshift-default-58d5bc7645-zjnf8\" (UID: \"f09e7c2c-246e-4c96-a8ca-147e8ca83560\") " pod="openshift-ingress/e2e-extmod-a-027c19-openshift-default-58d5bc7645-zjnf8" Nov 03 01:10:07.029199 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:10:07.029183 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f09e7c2c-246e-4c96-a8ca-147e8ca83560-istiod-ca-cert\") pod \"e2e-extmod-a-027c19-openshift-default-58d5bc7645-zjnf8\" (UID: \"f09e7c2c-246e-4c96-a8ca-147e8ca83560\") " pod="openshift-ingress/e2e-extmod-a-027c19-openshift-default-58d5bc7645-zjnf8" Nov 03 01:10:07.038933 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:10:07.005390 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f09e7c2c-246e-4c96-a8ca-147e8ca83560-istio-data\") pod \"e2e-extmod-a-027c19-openshift-default-58d5bc7645-zjnf8\" (UID: \"f09e7c2c-246e-4c96-a8ca-147e8ca83560\") " pod="openshift-ingress/e2e-extmod-a-027c19-openshift-default-58d5bc7645-zjnf8" Nov 03 01:10:07.039139 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:10:07.024527 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f09e7c2c-246e-4c96-a8ca-147e8ca83560-workload-socket\") pod \"e2e-extmod-a-027c19-openshift-default-58d5bc7645-zjnf8\" (UID: \"f09e7c2c-246e-4c96-a8ca-147e8ca83560\") " pod="openshift-ingress/e2e-extmod-a-027c19-openshift-default-58d5bc7645-zjnf8" Nov 03 01:10:07.039235 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:10:07.015317 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f09e7c2c-246e-4c96-a8ca-147e8ca83560-workload-certs\") pod \"e2e-extmod-a-027c19-openshift-default-58d5bc7645-zjnf8\" (UID: \"f09e7c2c-246e-4c96-a8ca-147e8ca83560\") " pod="openshift-ingress/e2e-extmod-a-027c19-openshift-default-58d5bc7645-zjnf8" Nov 03 01:10:07.055012 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:10:07.023294 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f09e7c2c-246e-4c96-a8ca-147e8ca83560-credential-socket\") pod \"e2e-extmod-a-027c19-openshift-default-58d5bc7645-zjnf8\" (UID: \"f09e7c2c-246e-4c96-a8ca-147e8ca83560\") " pod="openshift-ingress/e2e-extmod-a-027c19-openshift-default-58d5bc7645-zjnf8" Nov 03 01:10:07.136262 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:10:07.044106 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f09e7c2c-246e-4c96-a8ca-147e8ca83560-istiod-ca-cert\") pod \"e2e-extmod-a-027c19-openshift-default-58d5bc7645-zjnf8\" (UID: \"f09e7c2c-246e-4c96-a8ca-147e8ca83560\") " pod="openshift-ingress/e2e-extmod-a-027c19-openshift-default-58d5bc7645-zjnf8" Nov 03 01:10:07.216279 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:10:07.103720 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f09e7c2c-246e-4c96-a8ca-147e8ca83560-istio-podinfo\") pod \"e2e-extmod-a-027c19-openshift-default-58d5bc7645-zjnf8\" (UID: \"f09e7c2c-246e-4c96-a8ca-147e8ca83560\") " pod="openshift-ingress/e2e-extmod-a-027c19-openshift-default-58d5bc7645-zjnf8" Nov 03 01:10:07.216279 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:10:07.105531 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f09e7c2c-246e-4c96-a8ca-147e8ca83560-istio-envoy\") pod \"e2e-extmod-a-027c19-openshift-default-58d5bc7645-zjnf8\" (UID: \"f09e7c2c-246e-4c96-a8ca-147e8ca83560\") " pod="openshift-ingress/e2e-extmod-a-027c19-openshift-default-58d5bc7645-zjnf8" Nov 03 01:10:07.216279 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:10:07.193228 4939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-af8244-84f67c568c-m9kgf" event={"ID":"1f0db3da-dcf7-459b-994e-a7081e6f0958","Type":"ContainerStarted","Data":"22f578bc64e5a78ccb195588fcf509b901b156969faf20a928fc0d92ff8e06fb"} Nov 03 01:10:07.216279 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:10:07.200412 4939 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-af8244-84f67c568c-m9kgf" Nov 03 01:10:07.216279 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:10:07.200472 4939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-af8244-84f67c568c-m9kgf" event={"ID":"1f0db3da-dcf7-459b-994e-a7081e6f0958","Type":"ContainerStarted","Data":"59ae3b21edab922dd4ad5111e7c776e38e208f20099ea2868c3e243e2e7197bc"} Nov 03 01:10:07.225530 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:10:07.222130 4939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-b-af8244-75d9474dd7-pr6cd" event={"ID":"d1f73b7c-084a-44b7-a3bd-c5042532a732","Type":"ContainerStarted","Data":"4eccef524b9821502c808736a0d11ae8bd8a6add2167c878b11568b82991675f"} Nov 03 01:10:07.225530 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:10:07.222199 4939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-b-af8244-75d9474dd7-pr6cd" event={"ID":"d1f73b7c-084a-44b7-a3bd-c5042532a732","Type":"ContainerStarted","Data":"87e3e43af5dcf61e99c104b70b58c3918bb7cad0b95c9257e96b9793b280bc89"} Nov 03 01:10:07.225530 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:10:07.223344 4939 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-extmod-b-af8244-75d9474dd7-pr6cd" Nov 03 01:10:07.323077 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:10:07.322776 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dtzkd\" (UniqueName: \"kubernetes.io/projected/f09e7c2c-246e-4c96-a8ca-147e8ca83560-kube-api-access-dtzkd\") pod \"e2e-extmod-a-027c19-openshift-default-58d5bc7645-zjnf8\" (UID: \"f09e7c2c-246e-4c96-a8ca-147e8ca83560\") " pod="openshift-ingress/e2e-extmod-a-027c19-openshift-default-58d5bc7645-zjnf8" Nov 03 01:10:07.326949 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:10:07.326640 4939 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-af8244-84f67c568c-m9kgf" podStartSLOduration=3.32662017 podStartE2EDuration="3.32662017s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 01:10:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 01:10:07.244434827 +0000 UTC m=+1755.508769787" watchObservedRunningTime="2026-11-03 01:10:07.32662017 +0000 UTC m=+1755.590955119" Nov 03 01:10:07.332290 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:10:07.332235 4939 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-extmod-b-af8244-75d9474dd7-pr6cd" podStartSLOduration=2.332218648 podStartE2EDuration="2.332218648s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 01:10:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 01:10:07.262807142 +0000 UTC m=+1755.527142088" watchObservedRunningTime="2026-11-03 01:10:07.332218648 +0000 UTC m=+1755.596553596" Nov 03 01:10:07.333405 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:10:07.333374 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f09e7c2c-246e-4c96-a8ca-147e8ca83560-istio-token\") pod \"e2e-extmod-a-027c19-openshift-default-58d5bc7645-zjnf8\" (UID: \"f09e7c2c-246e-4c96-a8ca-147e8ca83560\") " pod="openshift-ingress/e2e-extmod-a-027c19-openshift-default-58d5bc7645-zjnf8" Nov 03 01:10:07.463970 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:10:07.449496 4939 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-027c19-openshift-default-58d5bc7645-zjnf8" Nov 03 01:10:07.630501 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:10:07.630467 4939 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w4-6c0ec0/auth-test-model-6c0ec0-kserve-589c79bd69-vznsb"] Nov 03 01:10:07.695473 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:10:07.695360 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:10:07.988184 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:10:07.845618 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:10:08.499900 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:10:08.499870 4939 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-a-027c19-openshift-default-58d5bc7645-zjnf8"] Nov 03 01:10:08.506582 ip-10-0-140-107 kubenswrapper[4939]: W1103 01:10:08.506548 4939 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf09e7c2c_246e_4c96_a8ca_147e8ca83560.slice/crio-1e215616e471783f85fa757cd1a5a4a3db0d8141baf07cd643ddced3a53ddd2f WatchSource:0}: Error finding container 1e215616e471783f85fa757cd1a5a4a3db0d8141baf07cd643ddced3a53ddd2f: Status 404 returned error can't find the container with id 1e215616e471783f85fa757cd1a5a4a3db0d8141baf07cd643ddced3a53ddd2f Nov 03 01:10:08.510259 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:10:08.510214 4939 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 01:10:08.510348 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:10:08.510314 4939 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 01:10:08.510397 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:10:08.510358 4939 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 01:10:08.989967 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:10:08.989923 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:10:09.378129 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:10:09.377200 4939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-027c19-openshift-default-58d5bc7645-zjnf8" event={"ID":"f09e7c2c-246e-4c96-a8ca-147e8ca83560","Type":"ContainerStarted","Data":"a547b507af135685227fd7790e98b40a7d9718e0f67e158871a6c8acb5162fcc"} Nov 03 01:10:09.378129 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:10:09.377305 4939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-027c19-openshift-default-58d5bc7645-zjnf8" event={"ID":"f09e7c2c-246e-4c96-a8ca-147e8ca83560","Type":"ContainerStarted","Data":"1e215616e471783f85fa757cd1a5a4a3db0d8141baf07cd643ddced3a53ddd2f"} Nov 03 01:10:09.400920 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:10:09.400860 4939 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-extmod-a-027c19-openshift-default-58d5bc7645-zjnf8" podStartSLOduration=3.400845928 podStartE2EDuration="3.400845928s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 01:10:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 01:10:09.397411782 +0000 UTC m=+1757.661746728" watchObservedRunningTime="2026-11-03 01:10:09.400845928 +0000 UTC m=+1757.665180875" Nov 03 01:10:09.472982 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:10:09.472040 4939 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-extmod-a-027c19-openshift-default-58d5bc7645-zjnf8" Nov 03 01:10:09.477754 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:10:09.477720 4939 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-extmod-a-027c19-openshift-default-58d5bc7645-zjnf8" Nov 03 01:10:09.923409 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:10:09.923380 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:10:10.182119 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:10:10.181967 4939 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-88bbf7-openshift-default-574f47fff7-dthhl"] Nov 03 01:10:10.182608 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:10:10.182226 4939 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-88bbf7-openshift-default-574f47fff7-dthhl" podUID="a1e98e45-9f37-42eb-9ac6-4624f63fe503" containerName="istio-proxy" containerID="cri-o://fba2e118254d3ef30d6581dc566cc3df6786088bb12ca7c5f2bdf82e3e01d3f3" gracePeriod=30 Nov 03 01:10:10.386193 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:10:10.386160 4939 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-a-027c19-openshift-default-58d5bc7645-zjnf8" Nov 03 01:10:10.387303 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:10:10.387279 4939 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-extmod-a-027c19-openshift-default-58d5bc7645-zjnf8" Nov 03 01:10:10.878659 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:10:10.878437 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:10:11.307512 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:10:11.307472 4939 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w4-4a697f/auth-test-model-4a697f-kserve-84b7466bd5-48hbr"] Nov 03 01:10:11.357786 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:10:11.357753 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:10:11.866546 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:10:11.866483 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:10:12.046682 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:10:12.046647 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:10:12.193796 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:10:12.193516 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:10:12.916283 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:10:12.916237 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:10:13.014500 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:10:13.014462 4939 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-027c19-596c6c9695-wm55p"] Nov 03 01:10:13.019934 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:10:13.019898 4939 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-a-027c19-596c6c9695-wm55p" Nov 03 01:10:13.027389 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:10:13.027347 4939 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-extmod-a-027c19\"" Nov 03 01:10:13.034233 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:10:13.034202 4939 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-027c19-596c6c9695-wm55p"] Nov 03 01:10:13.043979 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:10:13.043947 4939 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-027c19-67b548955-rltd2"] Nov 03 01:10:13.053163 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:10:13.053130 4939 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-027c19-67b548955-rltd2" Nov 03 01:10:13.067929 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:10:13.060449 4939 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-extmod-a-027c19-dockercfg-9mfp6\"" Nov 03 01:10:13.069122 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:10:13.068884 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:10:13.082591 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:10:13.082553 4939 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-027c19-67b548955-rltd2"] Nov 03 01:10:13.137475 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:10:13.137434 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cckw9\" (UniqueName: \"kubernetes.io/projected/ce37c54e-196b-4861-8b0d-e2d4351eed24-kube-api-access-cckw9\") pod \"payload-processing-e2e-extmod-a-027c19-596c6c9695-wm55p\" (UID: \"ce37c54e-196b-4861-8b0d-e2d4351eed24\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-027c19-596c6c9695-wm55p" Nov 03 01:10:13.137857 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:10:13.137825 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cwwzd\" (UniqueName: \"kubernetes.io/projected/ab5ffea1-5d04-47c2-9d4d-d720550bc30f-kube-api-access-cwwzd\") pod \"payload-pre-processing-e2e-extmod-a-027c19-67b548955-rltd2\" (UID: \"ab5ffea1-5d04-47c2-9d4d-d720550bc30f\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-027c19-67b548955-rltd2" Nov 03 01:10:13.138162 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:10:13.138137 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ce37c54e-196b-4861-8b0d-e2d4351eed24-plugins-config-volume\") pod \"payload-processing-e2e-extmod-a-027c19-596c6c9695-wm55p\" (UID: \"ce37c54e-196b-4861-8b0d-e2d4351eed24\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-027c19-596c6c9695-wm55p" Nov 03 01:10:13.138348 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:10:13.138329 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ab5ffea1-5d04-47c2-9d4d-d720550bc30f-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-a-027c19-67b548955-rltd2\" (UID: \"ab5ffea1-5d04-47c2-9d4d-d720550bc30f\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-027c19-67b548955-rltd2" Nov 03 01:10:13.272776 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:10:13.272681 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ce37c54e-196b-4861-8b0d-e2d4351eed24-plugins-config-volume\") pod \"payload-processing-e2e-extmod-a-027c19-596c6c9695-wm55p\" (UID: \"ce37c54e-196b-4861-8b0d-e2d4351eed24\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-027c19-596c6c9695-wm55p" Nov 03 01:10:13.273083 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:10:13.273059 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ab5ffea1-5d04-47c2-9d4d-d720550bc30f-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-a-027c19-67b548955-rltd2\" (UID: \"ab5ffea1-5d04-47c2-9d4d-d720550bc30f\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-027c19-67b548955-rltd2" Nov 03 01:10:13.273297 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:10:13.273279 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cckw9\" (UniqueName: \"kubernetes.io/projected/ce37c54e-196b-4861-8b0d-e2d4351eed24-kube-api-access-cckw9\") pod \"payload-processing-e2e-extmod-a-027c19-596c6c9695-wm55p\" (UID: \"ce37c54e-196b-4861-8b0d-e2d4351eed24\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-027c19-596c6c9695-wm55p" Nov 03 01:10:13.273541 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:10:13.273525 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cwwzd\" (UniqueName: \"kubernetes.io/projected/ab5ffea1-5d04-47c2-9d4d-d720550bc30f-kube-api-access-cwwzd\") pod \"payload-pre-processing-e2e-extmod-a-027c19-67b548955-rltd2\" (UID: \"ab5ffea1-5d04-47c2-9d4d-d720550bc30f\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-027c19-67b548955-rltd2" Nov 03 01:10:13.294944 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:10:13.284830 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ab5ffea1-5d04-47c2-9d4d-d720550bc30f-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-a-027c19-67b548955-rltd2\" (UID: \"ab5ffea1-5d04-47c2-9d4d-d720550bc30f\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-027c19-67b548955-rltd2" Nov 03 01:10:13.295580 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:10:13.295533 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ce37c54e-196b-4861-8b0d-e2d4351eed24-plugins-config-volume\") pod \"payload-processing-e2e-extmod-a-027c19-596c6c9695-wm55p\" (UID: \"ce37c54e-196b-4861-8b0d-e2d4351eed24\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-027c19-596c6c9695-wm55p" Nov 03 01:10:13.313819 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:10:13.313787 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cckw9\" (UniqueName: \"kubernetes.io/projected/ce37c54e-196b-4861-8b0d-e2d4351eed24-kube-api-access-cckw9\") pod \"payload-processing-e2e-extmod-a-027c19-596c6c9695-wm55p\" (UID: \"ce37c54e-196b-4861-8b0d-e2d4351eed24\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-027c19-596c6c9695-wm55p" Nov 03 01:10:13.314938 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:10:13.314879 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cwwzd\" (UniqueName: \"kubernetes.io/projected/ab5ffea1-5d04-47c2-9d4d-d720550bc30f-kube-api-access-cwwzd\") pod \"payload-pre-processing-e2e-extmod-a-027c19-67b548955-rltd2\" (UID: \"ab5ffea1-5d04-47c2-9d4d-d720550bc30f\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-027c19-67b548955-rltd2" Nov 03 01:10:13.347729 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:10:13.345240 4939 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-a-027c19-596c6c9695-wm55p" Nov 03 01:10:13.394052 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:10:13.393766 4939 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-027c19-67b548955-rltd2" Nov 03 01:10:13.948945 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:10:13.948896 4939 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-027c19-67b548955-rltd2"] Nov 03 01:10:13.963061 ip-10-0-140-107 kubenswrapper[4939]: W1103 01:10:13.963027 4939 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podce37c54e_196b_4861_8b0d_e2d4351eed24.slice/crio-d180dbd0cfea2d54af90aa6744d70bb4c7eb445d847462d21ebebe567b66b143 WatchSource:0}: Error finding container d180dbd0cfea2d54af90aa6744d70bb4c7eb445d847462d21ebebe567b66b143: Status 404 returned error can't find the container with id d180dbd0cfea2d54af90aa6744d70bb4c7eb445d847462d21ebebe567b66b143 Nov 03 01:10:13.964128 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:10:13.963996 4939 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-027c19-596c6c9695-wm55p"] Nov 03 01:10:14.108751 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:10:14.108711 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:10:14.645397 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:10:14.645351 4939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-027c19-67b548955-rltd2" event={"ID":"ab5ffea1-5d04-47c2-9d4d-d720550bc30f","Type":"ContainerStarted","Data":"5f1ab4affd67f9c19aeb9973a274a941d79ca3d8585bd1f61f141a9875346ae5"} Nov 03 01:10:14.645615 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:10:14.645418 4939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-027c19-67b548955-rltd2" event={"ID":"ab5ffea1-5d04-47c2-9d4d-d720550bc30f","Type":"ContainerStarted","Data":"83d0ef44df2da4f704f3b2a785e33921707cf24b178fbfd056f09e13ec1b933e"} Nov 03 01:10:14.645615 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:10:14.645448 4939 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-027c19-67b548955-rltd2" Nov 03 01:10:14.647442 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:10:14.647409 4939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-a-027c19-596c6c9695-wm55p" event={"ID":"ce37c54e-196b-4861-8b0d-e2d4351eed24","Type":"ContainerStarted","Data":"89db74dd8260ac3a9c06c43a81706ad1b6010460e1db156d2a53ccad4650368c"} Nov 03 01:10:14.647566 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:10:14.647462 4939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-a-027c19-596c6c9695-wm55p" event={"ID":"ce37c54e-196b-4861-8b0d-e2d4351eed24","Type":"ContainerStarted","Data":"d180dbd0cfea2d54af90aa6744d70bb4c7eb445d847462d21ebebe567b66b143"} Nov 03 01:10:14.647566 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:10:14.647489 4939 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-extmod-a-027c19-596c6c9695-wm55p" Nov 03 01:10:14.664814 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:10:14.664749 4939 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-027c19-67b548955-rltd2" podStartSLOduration=1.664729229 podStartE2EDuration="1.664729229s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 01:10:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 01:10:14.660976541 +0000 UTC m=+1762.925311488" watchObservedRunningTime="2026-11-03 01:10:14.664729229 +0000 UTC m=+1762.929064177" Nov 03 01:10:14.676428 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:10:14.676359 4939 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-extmod-a-027c19-596c6c9695-wm55p" podStartSLOduration=2.6763415630000003 podStartE2EDuration="2.676341563s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 01:10:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 01:10:14.675690004 +0000 UTC m=+1762.940024951" watchObservedRunningTime="2026-11-03 01:10:14.676341563 +0000 UTC m=+1762.940676509" Nov 03 01:10:14.886940 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:10:14.882895 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:10:15.516849 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:10:15.516816 4939 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-88bbf7-openshift-default-574f47fff7-dthhl" Nov 03 01:10:15.550836 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:10:15.548096 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:10:15.647173 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:10:15.647137 4939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a1e98e45-9f37-42eb-9ac6-4624f63fe503-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a1e98e45-9f37-42eb-9ac6-4624f63fe503-workload-certs\") pod \"a1e98e45-9f37-42eb-9ac6-4624f63fe503\" (UID: \"a1e98e45-9f37-42eb-9ac6-4624f63fe503\") " Nov 03 01:10:15.647390 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:10:15.647258 4939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a1e98e45-9f37-42eb-9ac6-4624f63fe503-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a1e98e45-9f37-42eb-9ac6-4624f63fe503-credential-socket\") pod \"a1e98e45-9f37-42eb-9ac6-4624f63fe503\" (UID: \"a1e98e45-9f37-42eb-9ac6-4624f63fe503\") " Nov 03 01:10:15.647390 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:10:15.647383 4939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a1e98e45-9f37-42eb-9ac6-4624f63fe503-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a1e98e45-9f37-42eb-9ac6-4624f63fe503-istio-data\") pod \"a1e98e45-9f37-42eb-9ac6-4624f63fe503\" (UID: \"a1e98e45-9f37-42eb-9ac6-4624f63fe503\") " Nov 03 01:10:15.647541 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:10:15.647483 4939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a1e98e45-9f37-42eb-9ac6-4624f63fe503-workload-certs" pod "a1e98e45-9f37-42eb-9ac6-4624f63fe503" (UID: "a1e98e45-9f37-42eb-9ac6-4624f63fe503"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 01:10:15.647541 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:10:15.647528 4939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a1e98e45-9f37-42eb-9ac6-4624f63fe503-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a1e98e45-9f37-42eb-9ac6-4624f63fe503-istio-envoy\") pod \"a1e98e45-9f37-42eb-9ac6-4624f63fe503\" (UID: \"a1e98e45-9f37-42eb-9ac6-4624f63fe503\") " Nov 03 01:10:15.647663 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:10:15.647587 4939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a1e98e45-9f37-42eb-9ac6-4624f63fe503-credential-socket" pod "a1e98e45-9f37-42eb-9ac6-4624f63fe503" (UID: "a1e98e45-9f37-42eb-9ac6-4624f63fe503"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 01:10:15.647663 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:10:15.647650 4939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a1e98e45-9f37-42eb-9ac6-4624f63fe503-istio-data" pod "a1e98e45-9f37-42eb-9ac6-4624f63fe503" (UID: "a1e98e45-9f37-42eb-9ac6-4624f63fe503"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 01:10:15.647780 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:10:15.647659 4939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a1e98e45-9f37-42eb-9ac6-4624f63fe503-istio-token\" (UniqueName: \"kubernetes.io/projected/a1e98e45-9f37-42eb-9ac6-4624f63fe503-istio-token\") pod \"a1e98e45-9f37-42eb-9ac6-4624f63fe503\" (UID: \"a1e98e45-9f37-42eb-9ac6-4624f63fe503\") " Nov 03 01:10:15.647780 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:10:15.647712 4939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a1e98e45-9f37-42eb-9ac6-4624f63fe503-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a1e98e45-9f37-42eb-9ac6-4624f63fe503-workload-socket\") pod \"a1e98e45-9f37-42eb-9ac6-4624f63fe503\" (UID: \"a1e98e45-9f37-42eb-9ac6-4624f63fe503\") " Nov 03 01:10:15.647925 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:10:15.647775 4939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a1e98e45-9f37-42eb-9ac6-4624f63fe503-kube-api-access-54lt4\" (UniqueName: \"kubernetes.io/projected/a1e98e45-9f37-42eb-9ac6-4624f63fe503-kube-api-access-54lt4\") pod \"a1e98e45-9f37-42eb-9ac6-4624f63fe503\" (UID: \"a1e98e45-9f37-42eb-9ac6-4624f63fe503\") " Nov 03 01:10:15.647925 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:10:15.647891 4939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a1e98e45-9f37-42eb-9ac6-4624f63fe503-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a1e98e45-9f37-42eb-9ac6-4624f63fe503-istiod-ca-cert\") pod \"a1e98e45-9f37-42eb-9ac6-4624f63fe503\" (UID: \"a1e98e45-9f37-42eb-9ac6-4624f63fe503\") " Nov 03 01:10:15.648063 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:10:15.647959 4939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/a1e98e45-9f37-42eb-9ac6-4624f63fe503-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a1e98e45-9f37-42eb-9ac6-4624f63fe503-istio-podinfo\") pod \"a1e98e45-9f37-42eb-9ac6-4624f63fe503\" (UID: \"a1e98e45-9f37-42eb-9ac6-4624f63fe503\") " Nov 03 01:10:15.648063 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:10:15.647988 4939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a1e98e45-9f37-42eb-9ac6-4624f63fe503-workload-socket" pod "a1e98e45-9f37-42eb-9ac6-4624f63fe503" (UID: "a1e98e45-9f37-42eb-9ac6-4624f63fe503"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 01:10:15.648641 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:10:15.648615 4939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a1e98e45-9f37-42eb-9ac6-4624f63fe503-istiod-ca-cert" pod "a1e98e45-9f37-42eb-9ac6-4624f63fe503" (UID: "a1e98e45-9f37-42eb-9ac6-4624f63fe503"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 01:10:15.648974 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:10:15.648952 4939 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a1e98e45-9f37-42eb-9ac6-4624f63fe503-workload-socket\") on node \"ip-10-0-140-107.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:10:15.649089 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:10:15.648979 4939 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a1e98e45-9f37-42eb-9ac6-4624f63fe503-istiod-ca-cert\") on node \"ip-10-0-140-107.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:10:15.649089 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:10:15.649000 4939 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a1e98e45-9f37-42eb-9ac6-4624f63fe503-workload-certs\") on node \"ip-10-0-140-107.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:10:15.649089 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:10:15.649022 4939 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a1e98e45-9f37-42eb-9ac6-4624f63fe503-credential-socket\") on node \"ip-10-0-140-107.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:10:15.649089 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:10:15.649048 4939 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a1e98e45-9f37-42eb-9ac6-4624f63fe503-istio-data\") on node \"ip-10-0-140-107.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:10:15.650604 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:10:15.650577 4939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/a1e98e45-9f37-42eb-9ac6-4624f63fe503-istio-podinfo" pod "a1e98e45-9f37-42eb-9ac6-4624f63fe503" (UID: "a1e98e45-9f37-42eb-9ac6-4624f63fe503"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 01:10:15.650993 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:10:15.650888 4939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a1e98e45-9f37-42eb-9ac6-4624f63fe503-istio-envoy" pod "a1e98e45-9f37-42eb-9ac6-4624f63fe503" (UID: "a1e98e45-9f37-42eb-9ac6-4624f63fe503"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 01:10:15.651624 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:10:15.651603 4939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a1e98e45-9f37-42eb-9ac6-4624f63fe503-kube-api-access-54lt4" pod "a1e98e45-9f37-42eb-9ac6-4624f63fe503" (UID: "a1e98e45-9f37-42eb-9ac6-4624f63fe503"). InnerVolumeSpecName "kube-api-access-54lt4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 01:10:15.652117 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:10:15.652095 4939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a1e98e45-9f37-42eb-9ac6-4624f63fe503-istio-token" pod "a1e98e45-9f37-42eb-9ac6-4624f63fe503" (UID: "a1e98e45-9f37-42eb-9ac6-4624f63fe503"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 01:10:15.660424 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:10:15.660396 4939 generic.go:356] "Generic (PLEG): container finished" podID="a1e98e45-9f37-42eb-9ac6-4624f63fe503" containerID="fba2e118254d3ef30d6581dc566cc3df6786088bb12ca7c5f2bdf82e3e01d3f3" exitCode=0 Nov 03 01:10:15.660561 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:10:15.660452 4939 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-88bbf7-openshift-default-574f47fff7-dthhl" Nov 03 01:10:15.660561 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:10:15.660515 4939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-88bbf7-openshift-default-574f47fff7-dthhl" event={"ID":"a1e98e45-9f37-42eb-9ac6-4624f63fe503","Type":"ContainerDied","Data":"fba2e118254d3ef30d6581dc566cc3df6786088bb12ca7c5f2bdf82e3e01d3f3"} Nov 03 01:10:15.660903 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:10:15.660572 4939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-88bbf7-openshift-default-574f47fff7-dthhl" event={"ID":"a1e98e45-9f37-42eb-9ac6-4624f63fe503","Type":"ContainerDied","Data":"babe89d2a31e97a1cf4f4f7b68c2683d843be742097bc1a69036995c7d058f61"} Nov 03 01:10:15.660903 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:10:15.660604 4939 scope.go:122] "RemoveContainer" containerID="fba2e118254d3ef30d6581dc566cc3df6786088bb12ca7c5f2bdf82e3e01d3f3" Nov 03 01:10:15.694156 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:10:15.694126 4939 scope.go:122] "RemoveContainer" containerID="fba2e118254d3ef30d6581dc566cc3df6786088bb12ca7c5f2bdf82e3e01d3f3" Nov 03 01:10:15.694555 ip-10-0-140-107 kubenswrapper[4939]: E1103 01:10:15.694519 4939 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fba2e118254d3ef30d6581dc566cc3df6786088bb12ca7c5f2bdf82e3e01d3f3\": container with ID starting with fba2e118254d3ef30d6581dc566cc3df6786088bb12ca7c5f2bdf82e3e01d3f3 not found: ID does not exist" containerID="fba2e118254d3ef30d6581dc566cc3df6786088bb12ca7c5f2bdf82e3e01d3f3" Nov 03 01:10:15.694653 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:10:15.694566 4939 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fba2e118254d3ef30d6581dc566cc3df6786088bb12ca7c5f2bdf82e3e01d3f3"} err="failed to get container status \"fba2e118254d3ef30d6581dc566cc3df6786088bb12ca7c5f2bdf82e3e01d3f3\": rpc error: code = NotFound desc = could not find container \"fba2e118254d3ef30d6581dc566cc3df6786088bb12ca7c5f2bdf82e3e01d3f3\": container with ID starting with fba2e118254d3ef30d6581dc566cc3df6786088bb12ca7c5f2bdf82e3e01d3f3 not found: ID does not exist" Nov 03 01:10:15.706253 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:10:15.706220 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:10:15.707623 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:10:15.707598 4939 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-88bbf7-openshift-default-574f47fff7-dthhl"] Nov 03 01:10:15.713250 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:10:15.713221 4939 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-88bbf7-openshift-default-574f47fff7-dthhl"] Nov 03 01:10:15.765530 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:10:15.757329 4939 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a1e98e45-9f37-42eb-9ac6-4624f63fe503-istio-envoy\") on node \"ip-10-0-140-107.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:10:15.765530 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:10:15.757367 4939 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a1e98e45-9f37-42eb-9ac6-4624f63fe503-istio-token\") on node \"ip-10-0-140-107.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:10:15.765530 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:10:15.757384 4939 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-54lt4\" (UniqueName: \"kubernetes.io/projected/a1e98e45-9f37-42eb-9ac6-4624f63fe503-kube-api-access-54lt4\") on node \"ip-10-0-140-107.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:10:15.765530 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:10:15.757399 4939 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a1e98e45-9f37-42eb-9ac6-4624f63fe503-istio-podinfo\") on node \"ip-10-0-140-107.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:10:15.889651 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:10:15.889595 4939 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a1e98e45-9f37-42eb-9ac6-4624f63fe503" path="/var/lib/kubelet/pods/a1e98e45-9f37-42eb-9ac6-4624f63fe503/volumes" Nov 03 01:10:15.946002 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:10:15.944714 4939 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-extmod-a-7c759a-openshift-default-56576554df-wkcvp"] Nov 03 01:10:15.946002 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:10:15.945966 4939 cpu_manager.go:414] "containerMap: removing container" podUID="a1e98e45-9f37-42eb-9ac6-4624f63fe503" containerName="istio-proxy" Nov 03 01:10:15.946002 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:10:15.945985 4939 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a1e98e45-9f37-42eb-9ac6-4624f63fe503" containerName="istio-proxy" Nov 03 01:10:15.946325 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:10:15.946149 4939 memory_manager.go:365] "RemoveStaleState removing state" podUID="a1e98e45-9f37-42eb-9ac6-4624f63fe503" containerName="istio-proxy" Nov 03 01:10:15.953195 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:10:15.952630 4939 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-7c759a-openshift-default-56576554df-wkcvp" Nov 03 01:10:15.966136 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:10:15.966113 4939 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-extmod-a-7c759a-openshift-default-dockercfg-8qhc6\"" Nov 03 01:10:15.986058 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:10:15.982683 4939 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-a-7c759a-openshift-default-56576554df-wkcvp"] Nov 03 01:10:16.103931 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:10:16.098320 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6ea24c7b-6d13-4661-bdff-cd52792cb6b6-istiod-ca-cert\") pod \"e2e-extmod-a-7c759a-openshift-default-56576554df-wkcvp\" (UID: \"6ea24c7b-6d13-4661-bdff-cd52792cb6b6\") " pod="openshift-ingress/e2e-extmod-a-7c759a-openshift-default-56576554df-wkcvp" Nov 03 01:10:16.103931 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:10:16.098443 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6ea24c7b-6d13-4661-bdff-cd52792cb6b6-workload-certs\") pod \"e2e-extmod-a-7c759a-openshift-default-56576554df-wkcvp\" (UID: \"6ea24c7b-6d13-4661-bdff-cd52792cb6b6\") " pod="openshift-ingress/e2e-extmod-a-7c759a-openshift-default-56576554df-wkcvp" Nov 03 01:10:16.103931 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:10:16.098507 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6ea24c7b-6d13-4661-bdff-cd52792cb6b6-istio-envoy\") pod \"e2e-extmod-a-7c759a-openshift-default-56576554df-wkcvp\" (UID: \"6ea24c7b-6d13-4661-bdff-cd52792cb6b6\") " pod="openshift-ingress/e2e-extmod-a-7c759a-openshift-default-56576554df-wkcvp" Nov 03 01:10:16.103931 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:10:16.098622 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6ea24c7b-6d13-4661-bdff-cd52792cb6b6-istio-podinfo\") pod \"e2e-extmod-a-7c759a-openshift-default-56576554df-wkcvp\" (UID: \"6ea24c7b-6d13-4661-bdff-cd52792cb6b6\") " pod="openshift-ingress/e2e-extmod-a-7c759a-openshift-default-56576554df-wkcvp" Nov 03 01:10:16.103931 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:10:16.098718 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6ea24c7b-6d13-4661-bdff-cd52792cb6b6-istio-data\") pod \"e2e-extmod-a-7c759a-openshift-default-56576554df-wkcvp\" (UID: \"6ea24c7b-6d13-4661-bdff-cd52792cb6b6\") " pod="openshift-ingress/e2e-extmod-a-7c759a-openshift-default-56576554df-wkcvp" Nov 03 01:10:16.103931 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:10:16.098784 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6ea24c7b-6d13-4661-bdff-cd52792cb6b6-credential-socket\") pod \"e2e-extmod-a-7c759a-openshift-default-56576554df-wkcvp\" (UID: \"6ea24c7b-6d13-4661-bdff-cd52792cb6b6\") " pod="openshift-ingress/e2e-extmod-a-7c759a-openshift-default-56576554df-wkcvp" Nov 03 01:10:16.103931 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:10:16.098882 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6ea24c7b-6d13-4661-bdff-cd52792cb6b6-istio-token\") pod \"e2e-extmod-a-7c759a-openshift-default-56576554df-wkcvp\" (UID: \"6ea24c7b-6d13-4661-bdff-cd52792cb6b6\") " pod="openshift-ingress/e2e-extmod-a-7c759a-openshift-default-56576554df-wkcvp" Nov 03 01:10:16.103931 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:10:16.098978 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6ea24c7b-6d13-4661-bdff-cd52792cb6b6-workload-socket\") pod \"e2e-extmod-a-7c759a-openshift-default-56576554df-wkcvp\" (UID: \"6ea24c7b-6d13-4661-bdff-cd52792cb6b6\") " pod="openshift-ingress/e2e-extmod-a-7c759a-openshift-default-56576554df-wkcvp" Nov 03 01:10:16.103931 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:10:16.099016 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4kmml\" (UniqueName: \"kubernetes.io/projected/6ea24c7b-6d13-4661-bdff-cd52792cb6b6-kube-api-access-4kmml\") pod \"e2e-extmod-a-7c759a-openshift-default-56576554df-wkcvp\" (UID: \"6ea24c7b-6d13-4661-bdff-cd52792cb6b6\") " pod="openshift-ingress/e2e-extmod-a-7c759a-openshift-default-56576554df-wkcvp" Nov 03 01:10:16.165944 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:10:16.161944 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:10:16.212047 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:10:16.206237 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6ea24c7b-6d13-4661-bdff-cd52792cb6b6-workload-certs\") pod \"e2e-extmod-a-7c759a-openshift-default-56576554df-wkcvp\" (UID: \"6ea24c7b-6d13-4661-bdff-cd52792cb6b6\") " pod="openshift-ingress/e2e-extmod-a-7c759a-openshift-default-56576554df-wkcvp" Nov 03 01:10:16.212047 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:10:16.206333 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6ea24c7b-6d13-4661-bdff-cd52792cb6b6-istio-envoy\") pod \"e2e-extmod-a-7c759a-openshift-default-56576554df-wkcvp\" (UID: \"6ea24c7b-6d13-4661-bdff-cd52792cb6b6\") " pod="openshift-ingress/e2e-extmod-a-7c759a-openshift-default-56576554df-wkcvp" Nov 03 01:10:16.212047 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:10:16.206451 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6ea24c7b-6d13-4661-bdff-cd52792cb6b6-istio-podinfo\") pod \"e2e-extmod-a-7c759a-openshift-default-56576554df-wkcvp\" (UID: \"6ea24c7b-6d13-4661-bdff-cd52792cb6b6\") " pod="openshift-ingress/e2e-extmod-a-7c759a-openshift-default-56576554df-wkcvp" Nov 03 01:10:16.212047 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:10:16.206544 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6ea24c7b-6d13-4661-bdff-cd52792cb6b6-istio-data\") pod \"e2e-extmod-a-7c759a-openshift-default-56576554df-wkcvp\" (UID: \"6ea24c7b-6d13-4661-bdff-cd52792cb6b6\") " pod="openshift-ingress/e2e-extmod-a-7c759a-openshift-default-56576554df-wkcvp" Nov 03 01:10:16.212047 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:10:16.206598 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6ea24c7b-6d13-4661-bdff-cd52792cb6b6-credential-socket\") pod \"e2e-extmod-a-7c759a-openshift-default-56576554df-wkcvp\" (UID: \"6ea24c7b-6d13-4661-bdff-cd52792cb6b6\") " pod="openshift-ingress/e2e-extmod-a-7c759a-openshift-default-56576554df-wkcvp" Nov 03 01:10:16.212047 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:10:16.206682 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6ea24c7b-6d13-4661-bdff-cd52792cb6b6-istio-token\") pod \"e2e-extmod-a-7c759a-openshift-default-56576554df-wkcvp\" (UID: \"6ea24c7b-6d13-4661-bdff-cd52792cb6b6\") " pod="openshift-ingress/e2e-extmod-a-7c759a-openshift-default-56576554df-wkcvp" Nov 03 01:10:16.212047 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:10:16.206753 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6ea24c7b-6d13-4661-bdff-cd52792cb6b6-workload-socket\") pod \"e2e-extmod-a-7c759a-openshift-default-56576554df-wkcvp\" (UID: \"6ea24c7b-6d13-4661-bdff-cd52792cb6b6\") " pod="openshift-ingress/e2e-extmod-a-7c759a-openshift-default-56576554df-wkcvp" Nov 03 01:10:16.212047 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:10:16.206790 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4kmml\" (UniqueName: \"kubernetes.io/projected/6ea24c7b-6d13-4661-bdff-cd52792cb6b6-kube-api-access-4kmml\") pod \"e2e-extmod-a-7c759a-openshift-default-56576554df-wkcvp\" (UID: \"6ea24c7b-6d13-4661-bdff-cd52792cb6b6\") " pod="openshift-ingress/e2e-extmod-a-7c759a-openshift-default-56576554df-wkcvp" Nov 03 01:10:16.212047 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:10:16.206967 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6ea24c7b-6d13-4661-bdff-cd52792cb6b6-istiod-ca-cert\") pod \"e2e-extmod-a-7c759a-openshift-default-56576554df-wkcvp\" (UID: \"6ea24c7b-6d13-4661-bdff-cd52792cb6b6\") " pod="openshift-ingress/e2e-extmod-a-7c759a-openshift-default-56576554df-wkcvp" Nov 03 01:10:16.212047 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:10:16.209478 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6ea24c7b-6d13-4661-bdff-cd52792cb6b6-istio-data\") pod \"e2e-extmod-a-7c759a-openshift-default-56576554df-wkcvp\" (UID: \"6ea24c7b-6d13-4661-bdff-cd52792cb6b6\") " pod="openshift-ingress/e2e-extmod-a-7c759a-openshift-default-56576554df-wkcvp" Nov 03 01:10:16.212047 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:10:16.209739 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6ea24c7b-6d13-4661-bdff-cd52792cb6b6-workload-certs\") pod \"e2e-extmod-a-7c759a-openshift-default-56576554df-wkcvp\" (UID: \"6ea24c7b-6d13-4661-bdff-cd52792cb6b6\") " pod="openshift-ingress/e2e-extmod-a-7c759a-openshift-default-56576554df-wkcvp" Nov 03 01:10:16.212047 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:10:16.210522 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6ea24c7b-6d13-4661-bdff-cd52792cb6b6-credential-socket\") pod \"e2e-extmod-a-7c759a-openshift-default-56576554df-wkcvp\" (UID: \"6ea24c7b-6d13-4661-bdff-cd52792cb6b6\") " pod="openshift-ingress/e2e-extmod-a-7c759a-openshift-default-56576554df-wkcvp" Nov 03 01:10:16.212047 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:10:16.210799 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6ea24c7b-6d13-4661-bdff-cd52792cb6b6-workload-socket\") pod \"e2e-extmod-a-7c759a-openshift-default-56576554df-wkcvp\" (UID: \"6ea24c7b-6d13-4661-bdff-cd52792cb6b6\") " pod="openshift-ingress/e2e-extmod-a-7c759a-openshift-default-56576554df-wkcvp" Nov 03 01:10:16.243977 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:10:16.208014 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6ea24c7b-6d13-4661-bdff-cd52792cb6b6-istiod-ca-cert\") pod \"e2e-extmod-a-7c759a-openshift-default-56576554df-wkcvp\" (UID: \"6ea24c7b-6d13-4661-bdff-cd52792cb6b6\") " pod="openshift-ingress/e2e-extmod-a-7c759a-openshift-default-56576554df-wkcvp" Nov 03 01:10:16.454979 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:10:16.446817 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6ea24c7b-6d13-4661-bdff-cd52792cb6b6-istio-token\") pod \"e2e-extmod-a-7c759a-openshift-default-56576554df-wkcvp\" (UID: \"6ea24c7b-6d13-4661-bdff-cd52792cb6b6\") " pod="openshift-ingress/e2e-extmod-a-7c759a-openshift-default-56576554df-wkcvp" Nov 03 01:10:16.468585 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:10:16.468536 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4kmml\" (UniqueName: \"kubernetes.io/projected/6ea24c7b-6d13-4661-bdff-cd52792cb6b6-kube-api-access-4kmml\") pod \"e2e-extmod-a-7c759a-openshift-default-56576554df-wkcvp\" (UID: \"6ea24c7b-6d13-4661-bdff-cd52792cb6b6\") " pod="openshift-ingress/e2e-extmod-a-7c759a-openshift-default-56576554df-wkcvp" Nov 03 01:10:16.479219 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:10:16.479180 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:10:16.680277 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:10:16.679210 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6ea24c7b-6d13-4661-bdff-cd52792cb6b6-istio-envoy\") pod \"e2e-extmod-a-7c759a-openshift-default-56576554df-wkcvp\" (UID: \"6ea24c7b-6d13-4661-bdff-cd52792cb6b6\") " pod="openshift-ingress/e2e-extmod-a-7c759a-openshift-default-56576554df-wkcvp" Nov 03 01:10:16.686788 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:10:16.681848 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6ea24c7b-6d13-4661-bdff-cd52792cb6b6-istio-podinfo\") pod \"e2e-extmod-a-7c759a-openshift-default-56576554df-wkcvp\" (UID: \"6ea24c7b-6d13-4661-bdff-cd52792cb6b6\") " pod="openshift-ingress/e2e-extmod-a-7c759a-openshift-default-56576554df-wkcvp" Nov 03 01:10:16.873301 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:10:16.873271 4939 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-7c759a-openshift-default-56576554df-wkcvp" Nov 03 01:10:16.950360 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:10:16.950326 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:10:17.173982 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:10:17.173893 4939 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-a-7c759a-openshift-default-56576554df-wkcvp"] Nov 03 01:10:17.178503 ip-10-0-140-107 kubenswrapper[4939]: W1103 01:10:17.178474 4939 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6ea24c7b_6d13_4661_bdff_cd52792cb6b6.slice/crio-b18df9e0980f634a78d89b2482a2f52611e2e5d34f6b940894f8d2367455c994 WatchSource:0}: Error finding container b18df9e0980f634a78d89b2482a2f52611e2e5d34f6b940894f8d2367455c994: Status 404 returned error can't find the container with id b18df9e0980f634a78d89b2482a2f52611e2e5d34f6b940894f8d2367455c994 Nov 03 01:10:17.181645 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:10:17.181606 4939 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 01:10:17.181739 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:10:17.181699 4939 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 01:10:17.181815 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:10:17.181749 4939 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 01:10:17.453755 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:10:17.452281 4939 patch_prober.go:29] interesting pod/e2e-shared-b-w2-ef434a-openshift-default-67cd7c96dc-q5p6d container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.201:15021/healthz/ready\": read tcp 10.132.0.2:46496->10.132.0.201:15021: read: connection reset by peer" start-of-body= Nov 03 01:10:17.453755 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:10:17.452541 4939 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-b-w2-ef434a-openshift-default-67cd7c96dc-q5p6d" podUID="285b736b-2a9f-40cd-96c4-3d7193d3eb8d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.201:15021/healthz/ready\": read tcp 10.132.0.2:46496->10.132.0.201:15021: read: connection reset by peer" Nov 03 01:10:17.705718 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:10:17.705605 4939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-7c759a-openshift-default-56576554df-wkcvp" event={"ID":"6ea24c7b-6d13-4661-bdff-cd52792cb6b6","Type":"ContainerStarted","Data":"a45f53d3c2fe34da76be639f2a986c7317499e4dc3d560d8ca33ca00bdf94750"} Nov 03 01:10:17.705718 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:10:17.705671 4939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-7c759a-openshift-default-56576554df-wkcvp" event={"ID":"6ea24c7b-6d13-4661-bdff-cd52792cb6b6","Type":"ContainerStarted","Data":"b18df9e0980f634a78d89b2482a2f52611e2e5d34f6b940894f8d2367455c994"} Nov 03 01:10:17.707574 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:10:17.707550 4939 generic.go:356] "Generic (PLEG): container finished" podID="285b736b-2a9f-40cd-96c4-3d7193d3eb8d" containerID="a01d3cc1289dcec7066b49be8e8f85d54d5805285a2912e982dc947054666749" exitCode=0 Nov 03 01:10:17.707701 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:10:17.707592 4939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-ef434a-openshift-default-67cd7c96dc-q5p6d" event={"ID":"285b736b-2a9f-40cd-96c4-3d7193d3eb8d","Type":"ContainerDied","Data":"a01d3cc1289dcec7066b49be8e8f85d54d5805285a2912e982dc947054666749"} Nov 03 01:10:17.707968 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:10:17.707952 4939 scope.go:122] "RemoveContainer" containerID="a01d3cc1289dcec7066b49be8e8f85d54d5805285a2912e982dc947054666749" Nov 03 01:10:17.709934 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:10:17.709886 4939 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 01:10:17.710029 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:10:17.709977 4939 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 01:10:17.710029 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:10:17.710012 4939 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 01:10:17.730713 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:10:17.730660 4939 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-extmod-a-7c759a-openshift-default-56576554df-wkcvp" podStartSLOduration=2.730640502 podStartE2EDuration="2.730640502s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 01:10:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 01:10:17.729731095 +0000 UTC m=+1765.994066067" watchObservedRunningTime="2026-11-03 01:10:17.730640502 +0000 UTC m=+1765.994975450" Nov 03 01:10:17.802872 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:10:17.802835 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:10:17.965980 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:10:17.965127 4939 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-extmod-a-7c759a-openshift-default-56576554df-wkcvp" Nov 03 01:10:18.098414 ip-10-0-140-107 kubenswrapper[4939]: E1103 01:10:18.098374 4939 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-sthjh" containerName="extractor" Nov 03 01:10:18.346361 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:10:18.346333 4939 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-af8244-84f67c568c-m9kgf" Nov 03 01:10:18.702278 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:10:18.702198 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:10:18.752941 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:10:18.748813 4939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-ef434a-openshift-default-67cd7c96dc-q5p6d" event={"ID":"285b736b-2a9f-40cd-96c4-3d7193d3eb8d","Type":"ContainerStarted","Data":"e7842b2bdf2d7feb7daf09483668d416de00149c08408519bd1e496b3a83bd46"} Nov 03 01:10:18.890940 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:10:18.890890 4939 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-extmod-a-7c759a-openshift-default-56576554df-wkcvp" Nov 03 01:10:19.381651 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:10:19.381613 4939 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-extmod-b-af8244-75d9474dd7-pr6cd" Nov 03 01:10:19.469568 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:10:19.469524 4939 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w2-ef434a-openshift-default-67cd7c96dc-q5p6d" Nov 03 01:10:19.477063 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:10:19.474283 4939 patch_prober.go:29] interesting pod/e2e-shared-b-w2-ef434a-openshift-default-67cd7c96dc-q5p6d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.201:15021/healthz/ready\": dial tcp 10.132.0.201:15021: connect: connection refused" start-of-body= Nov 03 01:10:19.477063 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:10:19.474361 4939 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-ef434a-openshift-default-67cd7c96dc-q5p6d" podUID="285b736b-2a9f-40cd-96c4-3d7193d3eb8d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.201:15021/healthz/ready\": dial tcp 10.132.0.201:15021: connect: connection refused" Nov 03 01:10:19.776635 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:10:19.776530 4939 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-a-7c759a-openshift-default-56576554df-wkcvp" Nov 03 01:10:19.777639 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:10:19.777614 4939 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-extmod-a-7c759a-openshift-default-56576554df-wkcvp" Nov 03 01:10:19.803388 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:10:19.803340 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:10:20.028005 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:10:20.023998 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:10:20.153934 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:10:20.153431 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:10:20.464953 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:10:20.460337 4939 patch_prober.go:29] interesting pod/e2e-shared-b-w2-ef434a-openshift-default-67cd7c96dc-q5p6d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.201:15021/healthz/ready\": dial tcp 10.132.0.201:15021: connect: connection refused" start-of-body= Nov 03 01:10:20.464953 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:10:20.460410 4939 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-ef434a-openshift-default-67cd7c96dc-q5p6d" podUID="285b736b-2a9f-40cd-96c4-3d7193d3eb8d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.201:15021/healthz/ready\": dial tcp 10.132.0.201:15021: connect: connection refused" Nov 03 01:10:20.979895 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:10:20.979824 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:10:21.384830 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:10:21.384516 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:10:21.427583 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:10:21.427550 4939 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-7c759a-66695c8d79-7ws2z"] Nov 03 01:10:21.431699 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:10:21.431677 4939 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-7c759a-66695c8d79-7ws2z" Nov 03 01:10:21.443829 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:10:21.443755 4939 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-extmod-a-7c759a-dockercfg-42rsf\"" Nov 03 01:10:21.443829 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:10:21.443804 4939 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-extmod-a-7c759a\"" Nov 03 01:10:21.461271 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:10:21.446074 4939 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-7c759a-66695c8d79-7ws2z"] Nov 03 01:10:21.478958 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:10:21.477896 4939 patch_prober.go:29] interesting pod/e2e-shared-b-w2-ef434a-openshift-default-67cd7c96dc-q5p6d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.201:15021/healthz/ready\": dial tcp 10.132.0.201:15021: connect: connection refused" start-of-body= Nov 03 01:10:21.478958 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:10:21.478029 4939 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-ef434a-openshift-default-67cd7c96dc-q5p6d" podUID="285b736b-2a9f-40cd-96c4-3d7193d3eb8d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.201:15021/healthz/ready\": dial tcp 10.132.0.201:15021: connect: connection refused" Nov 03 01:10:21.494095 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:10:21.493567 4939 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-7c759a-7745bc7499-bmfl4"] Nov 03 01:10:21.563942 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:10:21.560083 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zdnkv\" (UniqueName: \"kubernetes.io/projected/653b839f-b60a-466c-8b49-24aacea6dd75-kube-api-access-zdnkv\") pod \"payload-pre-processing-e2e-extmod-a-7c759a-66695c8d79-7ws2z\" (UID: \"653b839f-b60a-466c-8b49-24aacea6dd75\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-7c759a-66695c8d79-7ws2z" Nov 03 01:10:21.563942 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:10:21.560231 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/653b839f-b60a-466c-8b49-24aacea6dd75-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-a-7c759a-66695c8d79-7ws2z\" (UID: \"653b839f-b60a-466c-8b49-24aacea6dd75\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-7c759a-66695c8d79-7ws2z" Nov 03 01:10:21.594027 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:10:21.592631 4939 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-a-7c759a-7745bc7499-bmfl4" Nov 03 01:10:21.760016 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:10:21.709649 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:10:21.760016 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:10:21.716078 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/653b839f-b60a-466c-8b49-24aacea6dd75-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-a-7c759a-66695c8d79-7ws2z\" (UID: \"653b839f-b60a-466c-8b49-24aacea6dd75\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-7c759a-66695c8d79-7ws2z" Nov 03 01:10:21.760016 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:10:21.716233 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/19d08863-aa23-47f6-98cd-7908001a4458-plugins-config-volume\") pod \"payload-processing-e2e-extmod-a-7c759a-7745bc7499-bmfl4\" (UID: \"19d08863-aa23-47f6-98cd-7908001a4458\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-7c759a-7745bc7499-bmfl4" Nov 03 01:10:21.760016 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:10:21.716481 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zdnkv\" (UniqueName: \"kubernetes.io/projected/653b839f-b60a-466c-8b49-24aacea6dd75-kube-api-access-zdnkv\") pod \"payload-pre-processing-e2e-extmod-a-7c759a-66695c8d79-7ws2z\" (UID: \"653b839f-b60a-466c-8b49-24aacea6dd75\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-7c759a-66695c8d79-7ws2z" Nov 03 01:10:21.760016 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:10:21.716524 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7dfb2\" (UniqueName: \"kubernetes.io/projected/19d08863-aa23-47f6-98cd-7908001a4458-kube-api-access-7dfb2\") pod \"payload-processing-e2e-extmod-a-7c759a-7745bc7499-bmfl4\" (UID: \"19d08863-aa23-47f6-98cd-7908001a4458\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-7c759a-7745bc7499-bmfl4" Nov 03 01:10:21.760016 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:10:21.717704 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/653b839f-b60a-466c-8b49-24aacea6dd75-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-a-7c759a-66695c8d79-7ws2z\" (UID: \"653b839f-b60a-466c-8b49-24aacea6dd75\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-7c759a-66695c8d79-7ws2z" Nov 03 01:10:21.816949 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:10:21.799509 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zdnkv\" (UniqueName: \"kubernetes.io/projected/653b839f-b60a-466c-8b49-24aacea6dd75-kube-api-access-zdnkv\") pod \"payload-pre-processing-e2e-extmod-a-7c759a-66695c8d79-7ws2z\" (UID: \"653b839f-b60a-466c-8b49-24aacea6dd75\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-7c759a-66695c8d79-7ws2z" Nov 03 01:10:21.887931 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:10:21.831855 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7dfb2\" (UniqueName: \"kubernetes.io/projected/19d08863-aa23-47f6-98cd-7908001a4458-kube-api-access-7dfb2\") pod \"payload-processing-e2e-extmod-a-7c759a-7745bc7499-bmfl4\" (UID: \"19d08863-aa23-47f6-98cd-7908001a4458\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-7c759a-7745bc7499-bmfl4" Nov 03 01:10:21.887931 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:10:21.832207 4939 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-7c759a-7745bc7499-bmfl4"] Nov 03 01:10:21.912247 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:10:21.897462 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/19d08863-aa23-47f6-98cd-7908001a4458-plugins-config-volume\") pod \"payload-processing-e2e-extmod-a-7c759a-7745bc7499-bmfl4\" (UID: \"19d08863-aa23-47f6-98cd-7908001a4458\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-7c759a-7745bc7499-bmfl4" Nov 03 01:10:21.912247 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:10:21.911058 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7dfb2\" (UniqueName: \"kubernetes.io/projected/19d08863-aa23-47f6-98cd-7908001a4458-kube-api-access-7dfb2\") pod \"payload-processing-e2e-extmod-a-7c759a-7745bc7499-bmfl4\" (UID: \"19d08863-aa23-47f6-98cd-7908001a4458\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-7c759a-7745bc7499-bmfl4" Nov 03 01:10:21.912247 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:10:21.912036 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/19d08863-aa23-47f6-98cd-7908001a4458-plugins-config-volume\") pod \"payload-processing-e2e-extmod-a-7c759a-7745bc7499-bmfl4\" (UID: \"19d08863-aa23-47f6-98cd-7908001a4458\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-7c759a-7745bc7499-bmfl4" Nov 03 01:10:22.133956 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:10:22.133802 4939 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-7c759a-66695c8d79-7ws2z" Nov 03 01:10:22.134512 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:10:22.134237 4939 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-a-7c759a-7745bc7499-bmfl4" Nov 03 01:10:22.476935 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:10:22.471141 4939 patch_prober.go:29] interesting pod/e2e-shared-b-w2-ef434a-openshift-default-67cd7c96dc-q5p6d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.201:15021/healthz/ready\": dial tcp 10.132.0.201:15021: connect: connection refused" start-of-body= Nov 03 01:10:22.476935 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:10:22.471204 4939 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-ef434a-openshift-default-67cd7c96dc-q5p6d" podUID="285b736b-2a9f-40cd-96c4-3d7193d3eb8d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.201:15021/healthz/ready\": dial tcp 10.132.0.201:15021: connect: connection refused" Nov 03 01:10:22.494968 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:10:22.493664 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:10:22.777230 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:10:22.730454 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:10:22.777230 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:10:22.772088 4939 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-w4-6c0ec0/auth-test-model-6c0ec0-kserve-589c79bd69-vznsb" podUID="d21cb247-bacd-4af2-a7b1-d04b975f0993" containerName="main" containerID="cri-o://5ad7dbb8fc3fc68239c9e1e97fee011fa7a711add9530153f8ead02c8fa7053c" gracePeriod=45 Nov 03 01:10:22.809862 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:10:22.798836 4939 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-7c759a-7745bc7499-bmfl4"] Nov 03 01:10:22.838875 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:10:22.838834 4939 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-shared-a-w4-6c0ec0/auth-test-model-6c0ec0-kserve-589c79bd69-vznsb" podUID="d21cb247-bacd-4af2-a7b1-d04b975f0993" containerName="main" probeResult="failure" output="Get \"https://10.132.0.186:8000/ready\": EOF" Nov 03 01:10:22.841014 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:10:22.840845 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:10:22.841691 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:10:22.841367 4939 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-7c759a-66695c8d79-7ws2z"] Nov 03 01:10:23.206974 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:10:23.196269 4939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-7c759a-66695c8d79-7ws2z" event={"ID":"653b839f-b60a-466c-8b49-24aacea6dd75","Type":"ContainerStarted","Data":"8f175f99fa639f041e070a9428562d3a812c348ca9357f735b0959496103542a"} Nov 03 01:10:23.207735 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:10:23.207673 4939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-7c759a-66695c8d79-7ws2z" event={"ID":"653b839f-b60a-466c-8b49-24aacea6dd75","Type":"ContainerStarted","Data":"28179f533661c0f7690d0da0523c93e4ead2f7c58226dc9901cc26c606e1d504"} Nov 03 01:10:23.222175 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:10:23.222141 4939 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-7c759a-66695c8d79-7ws2z" Nov 03 01:10:23.257010 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:10:23.256620 4939 generic.go:356] "Generic (PLEG): container finished" podID="d21cb247-bacd-4af2-a7b1-d04b975f0993" containerID="5ad7dbb8fc3fc68239c9e1e97fee011fa7a711add9530153f8ead02c8fa7053c" exitCode=0 Nov 03 01:10:23.257010 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:10:23.256719 4939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w4-6c0ec0/auth-test-model-6c0ec0-kserve-589c79bd69-vznsb" event={"ID":"d21cb247-bacd-4af2-a7b1-d04b975f0993","Type":"ContainerDied","Data":"5ad7dbb8fc3fc68239c9e1e97fee011fa7a711add9530153f8ead02c8fa7053c"} Nov 03 01:10:23.315935 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:10:23.310120 4939 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-7c759a-66695c8d79-7ws2z" podStartSLOduration=2.310098217 podStartE2EDuration="2.310098217s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 01:10:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 01:10:23.266813307 +0000 UTC m=+1771.531148252" watchObservedRunningTime="2026-11-03 01:10:23.310098217 +0000 UTC m=+1771.574433162" Nov 03 01:10:23.355946 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:10:23.354084 4939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-a-7c759a-7745bc7499-bmfl4" event={"ID":"19d08863-aa23-47f6-98cd-7908001a4458","Type":"ContainerStarted","Data":"2c1ef432bd467355b686710995c8a2911b651dcbd6ac353b5209684b8930d156"} Nov 03 01:10:23.355946 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:10:23.354167 4939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-a-7c759a-7745bc7499-bmfl4" event={"ID":"19d08863-aa23-47f6-98cd-7908001a4458","Type":"ContainerStarted","Data":"02266c02dd4a303d8ef0f525c10a22b08ac887816e26212d0b702127e8351525"} Nov 03 01:10:23.355946 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:10:23.354197 4939 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-extmod-a-7c759a-7745bc7499-bmfl4" Nov 03 01:10:23.380955 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:10:23.379521 4939 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-extmod-a-7c759a-7745bc7499-bmfl4" podStartSLOduration=2.379498038 podStartE2EDuration="2.379498038s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 01:10:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 01:10:23.366186128 +0000 UTC m=+1771.630521074" watchObservedRunningTime="2026-11-03 01:10:23.379498038 +0000 UTC m=+1771.643832986" Nov 03 01:10:23.593953 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:10:23.591264 4939 patch_prober.go:29] interesting pod/e2e-shared-b-w2-ef434a-openshift-default-67cd7c96dc-q5p6d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.201:15021/healthz/ready\": dial tcp 10.132.0.201:15021: connect: connection refused" start-of-body= Nov 03 01:10:23.593953 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:10:23.591351 4939 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-ef434a-openshift-default-67cd7c96dc-q5p6d" podUID="285b736b-2a9f-40cd-96c4-3d7193d3eb8d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.201:15021/healthz/ready\": dial tcp 10.132.0.201:15021: connect: connection refused" Nov 03 01:10:23.762564 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:10:23.748101 4939 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w4-6c0ec0/auth-test-model-6c0ec0-kserve-589c79bd69-vznsb" Nov 03 01:10:23.816135 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:10:23.815855 4939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d21cb247-bacd-4af2-a7b1-d04b975f0993-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d21cb247-bacd-4af2-a7b1-d04b975f0993-tmp-dir\") pod \"d21cb247-bacd-4af2-a7b1-d04b975f0993\" (UID: \"d21cb247-bacd-4af2-a7b1-d04b975f0993\") " Nov 03 01:10:23.816135 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:10:23.816025 4939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d21cb247-bacd-4af2-a7b1-d04b975f0993-dshm\" (UniqueName: \"kubernetes.io/empty-dir/d21cb247-bacd-4af2-a7b1-d04b975f0993-dshm\") pod \"d21cb247-bacd-4af2-a7b1-d04b975f0993\" (UID: \"d21cb247-bacd-4af2-a7b1-d04b975f0993\") " Nov 03 01:10:23.816399 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:10:23.816204 4939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d21cb247-bacd-4af2-a7b1-d04b975f0993-tmp-dir" pod "d21cb247-bacd-4af2-a7b1-d04b975f0993" (UID: "d21cb247-bacd-4af2-a7b1-d04b975f0993"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 01:10:23.816858 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:10:23.816629 4939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d21cb247-bacd-4af2-a7b1-d04b975f0993-kube-api-access-xjmf4\" (UniqueName: \"kubernetes.io/projected/d21cb247-bacd-4af2-a7b1-d04b975f0993-kube-api-access-xjmf4\") pod \"d21cb247-bacd-4af2-a7b1-d04b975f0993\" (UID: \"d21cb247-bacd-4af2-a7b1-d04b975f0993\") " Nov 03 01:10:23.817183 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:10:23.817057 4939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d21cb247-bacd-4af2-a7b1-d04b975f0993-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d21cb247-bacd-4af2-a7b1-d04b975f0993-model-cache\") pod \"d21cb247-bacd-4af2-a7b1-d04b975f0993\" (UID: \"d21cb247-bacd-4af2-a7b1-d04b975f0993\") " Nov 03 01:10:23.817183 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:10:23.817136 4939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d21cb247-bacd-4af2-a7b1-d04b975f0993-home\" (UniqueName: \"kubernetes.io/empty-dir/d21cb247-bacd-4af2-a7b1-d04b975f0993-home\") pod \"d21cb247-bacd-4af2-a7b1-d04b975f0993\" (UID: \"d21cb247-bacd-4af2-a7b1-d04b975f0993\") " Nov 03 01:10:23.817463 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:10:23.817317 4939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d21cb247-bacd-4af2-a7b1-d04b975f0993-tls-certs\" (UniqueName: \"kubernetes.io/secret/d21cb247-bacd-4af2-a7b1-d04b975f0993-tls-certs\") pod \"d21cb247-bacd-4af2-a7b1-d04b975f0993\" (UID: \"d21cb247-bacd-4af2-a7b1-d04b975f0993\") " Nov 03 01:10:23.817463 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:10:23.817422 4939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d21cb247-bacd-4af2-a7b1-d04b975f0993-model-cache" pod "d21cb247-bacd-4af2-a7b1-d04b975f0993" (UID: "d21cb247-bacd-4af2-a7b1-d04b975f0993"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 01:10:23.817629 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:10:23.817609 4939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d21cb247-bacd-4af2-a7b1-d04b975f0993-home" pod "d21cb247-bacd-4af2-a7b1-d04b975f0993" (UID: "d21cb247-bacd-4af2-a7b1-d04b975f0993"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 01:10:23.818614 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:10:23.818586 4939 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d21cb247-bacd-4af2-a7b1-d04b975f0993-model-cache\") on node \"ip-10-0-140-107.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:10:23.818780 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:10:23.818768 4939 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d21cb247-bacd-4af2-a7b1-d04b975f0993-home\") on node \"ip-10-0-140-107.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:10:23.818882 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:10:23.818872 4939 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d21cb247-bacd-4af2-a7b1-d04b975f0993-tmp-dir\") on node \"ip-10-0-140-107.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:10:23.824932 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:10:23.822488 4939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d21cb247-bacd-4af2-a7b1-d04b975f0993-kube-api-access-xjmf4" pod "d21cb247-bacd-4af2-a7b1-d04b975f0993" (UID: "d21cb247-bacd-4af2-a7b1-d04b975f0993"). InnerVolumeSpecName "kube-api-access-xjmf4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 01:10:23.825218 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:10:23.825175 4939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d21cb247-bacd-4af2-a7b1-d04b975f0993-tls-certs" pod "d21cb247-bacd-4af2-a7b1-d04b975f0993" (UID: "d21cb247-bacd-4af2-a7b1-d04b975f0993"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 01:10:23.825218 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:10:23.825179 4939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d21cb247-bacd-4af2-a7b1-d04b975f0993-dshm" pod "d21cb247-bacd-4af2-a7b1-d04b975f0993" (UID: "d21cb247-bacd-4af2-a7b1-d04b975f0993"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 01:10:23.902756 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:10:23.902724 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:10:24.012929 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:10:23.982722 4939 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xjmf4\" (UniqueName: \"kubernetes.io/projected/d21cb247-bacd-4af2-a7b1-d04b975f0993-kube-api-access-xjmf4\") on node \"ip-10-0-140-107.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:10:24.044617 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:10:24.015173 4939 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d21cb247-bacd-4af2-a7b1-d04b975f0993-tls-certs\") on node \"ip-10-0-140-107.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:10:24.044617 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:10:24.015210 4939 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d21cb247-bacd-4af2-a7b1-d04b975f0993-dshm\") on node \"ip-10-0-140-107.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:10:24.371937 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:10:24.347062 4939 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w4-6c0ec0/auth-test-model-6c0ec0-kserve-589c79bd69-vznsb" Nov 03 01:10:24.373423 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:10:24.365984 4939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w4-6c0ec0/auth-test-model-6c0ec0-kserve-589c79bd69-vznsb" event={"ID":"d21cb247-bacd-4af2-a7b1-d04b975f0993","Type":"ContainerDied","Data":"3d2bdf00c4724290efca7bed06523f1271eafc6169c64acd02828274a3050a25"} Nov 03 01:10:24.373634 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:10:24.373615 4939 scope.go:122] "RemoveContainer" containerID="5ad7dbb8fc3fc68239c9e1e97fee011fa7a711add9530153f8ead02c8fa7053c" Nov 03 01:10:24.397740 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:10:24.397708 4939 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w4-6c0ec0/auth-test-model-6c0ec0-kserve-589c79bd69-vznsb"] Nov 03 01:10:24.402638 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:10:24.402609 4939 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-w4-6c0ec0/auth-test-model-6c0ec0-kserve-589c79bd69-vznsb"] Nov 03 01:10:24.467935 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:10:24.466576 4939 patch_prober.go:29] interesting pod/e2e-shared-b-w2-ef434a-openshift-default-67cd7c96dc-q5p6d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.201:15021/healthz/ready\": dial tcp 10.132.0.201:15021: connect: connection refused" start-of-body= Nov 03 01:10:24.467935 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:10:24.466660 4939 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-ef434a-openshift-default-67cd7c96dc-q5p6d" podUID="285b736b-2a9f-40cd-96c4-3d7193d3eb8d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.201:15021/healthz/ready\": dial tcp 10.132.0.201:15021: connect: connection refused" Nov 03 01:10:24.624025 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:10:24.623939 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:10:24.766990 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:10:24.762812 4939 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-shared-a-w4-6c0ec0/auth-test-model-6c0ec0-kserve-589c79bd69-vznsb" podUID="d21cb247-bacd-4af2-a7b1-d04b975f0993" containerName="main" probeResult="failure" output="Get \"https://10.132.0.186:8000/ready\": context deadline exceeded" Nov 03 01:10:25.463732 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:10:25.463695 4939 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w2-ef434a-openshift-default-67cd7c96dc-q5p6d" Nov 03 01:10:25.464215 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:10:25.464159 4939 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w2-ef434a-openshift-default-67cd7c96dc-q5p6d" Nov 03 01:10:25.464969 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:10:25.464887 4939 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w2-ef434a-openshift-default-67cd7c96dc-q5p6d" Nov 03 01:10:25.488067 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:10:25.488033 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:10:25.708081 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:10:25.706444 4939 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-extmod-a-027c19-596c6c9695-wm55p" Nov 03 01:10:25.721092 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:10:25.711797 4939 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-027c19-67b548955-rltd2" Nov 03 01:10:25.897118 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:10:25.896697 4939 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d21cb247-bacd-4af2-a7b1-d04b975f0993" path="/var/lib/kubelet/pods/d21cb247-bacd-4af2-a7b1-d04b975f0993/volumes" Nov 03 01:10:26.203475 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:10:26.203442 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:10:26.332935 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:10:26.331251 4939 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-w4-4a697f/auth-test-model-4a697f-kserve-84b7466bd5-48hbr" podUID="18846667-85a0-4bc2-8995-61bfb2ab1c5d" containerName="main" containerID="cri-o://dce8a0929af2abbec1201ec9933e9052670e290063b7823a598ecebb430fc3dc" gracePeriod=45 Nov 03 01:10:26.646310 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:10:26.646284 4939 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w4-4a697f/auth-test-model-4a697f-kserve-84b7466bd5-48hbr" Nov 03 01:10:26.760178 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:10:26.760084 4939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/18846667-85a0-4bc2-8995-61bfb2ab1c5d-tls-certs\" (UniqueName: \"kubernetes.io/secret/18846667-85a0-4bc2-8995-61bfb2ab1c5d-tls-certs\") pod \"18846667-85a0-4bc2-8995-61bfb2ab1c5d\" (UID: \"18846667-85a0-4bc2-8995-61bfb2ab1c5d\") " Nov 03 01:10:26.760178 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:10:26.760160 4939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/18846667-85a0-4bc2-8995-61bfb2ab1c5d-home\" (UniqueName: \"kubernetes.io/empty-dir/18846667-85a0-4bc2-8995-61bfb2ab1c5d-home\") pod \"18846667-85a0-4bc2-8995-61bfb2ab1c5d\" (UID: \"18846667-85a0-4bc2-8995-61bfb2ab1c5d\") " Nov 03 01:10:26.760414 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:10:26.760274 4939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/18846667-85a0-4bc2-8995-61bfb2ab1c5d-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/18846667-85a0-4bc2-8995-61bfb2ab1c5d-tmp-dir\") pod \"18846667-85a0-4bc2-8995-61bfb2ab1c5d\" (UID: \"18846667-85a0-4bc2-8995-61bfb2ab1c5d\") " Nov 03 01:10:26.760414 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:10:26.760352 4939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/18846667-85a0-4bc2-8995-61bfb2ab1c5d-dshm\" (UniqueName: \"kubernetes.io/empty-dir/18846667-85a0-4bc2-8995-61bfb2ab1c5d-dshm\") pod \"18846667-85a0-4bc2-8995-61bfb2ab1c5d\" (UID: \"18846667-85a0-4bc2-8995-61bfb2ab1c5d\") " Nov 03 01:10:26.760414 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:10:26.760381 4939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/18846667-85a0-4bc2-8995-61bfb2ab1c5d-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/18846667-85a0-4bc2-8995-61bfb2ab1c5d-model-cache\") pod \"18846667-85a0-4bc2-8995-61bfb2ab1c5d\" (UID: \"18846667-85a0-4bc2-8995-61bfb2ab1c5d\") " Nov 03 01:10:26.760592 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:10:26.760427 4939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/18846667-85a0-4bc2-8995-61bfb2ab1c5d-home" pod "18846667-85a0-4bc2-8995-61bfb2ab1c5d" (UID: "18846667-85a0-4bc2-8995-61bfb2ab1c5d"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 01:10:26.760646 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:10:26.760622 4939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/18846667-85a0-4bc2-8995-61bfb2ab1c5d-tmp-dir" pod "18846667-85a0-4bc2-8995-61bfb2ab1c5d" (UID: "18846667-85a0-4bc2-8995-61bfb2ab1c5d"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 01:10:26.760705 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:10:26.760691 4939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/18846667-85a0-4bc2-8995-61bfb2ab1c5d-kube-api-access-4zxrl\" (UniqueName: \"kubernetes.io/projected/18846667-85a0-4bc2-8995-61bfb2ab1c5d-kube-api-access-4zxrl\") pod \"18846667-85a0-4bc2-8995-61bfb2ab1c5d\" (UID: \"18846667-85a0-4bc2-8995-61bfb2ab1c5d\") " Nov 03 01:10:26.760989 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:10:26.760883 4939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/18846667-85a0-4bc2-8995-61bfb2ab1c5d-model-cache" pod "18846667-85a0-4bc2-8995-61bfb2ab1c5d" (UID: "18846667-85a0-4bc2-8995-61bfb2ab1c5d"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 01:10:26.761904 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:10:26.761873 4939 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/18846667-85a0-4bc2-8995-61bfb2ab1c5d-home\") on node \"ip-10-0-140-107.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:10:26.761904 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:10:26.761924 4939 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/18846667-85a0-4bc2-8995-61bfb2ab1c5d-tmp-dir\") on node \"ip-10-0-140-107.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:10:26.761904 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:10:26.761939 4939 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/18846667-85a0-4bc2-8995-61bfb2ab1c5d-model-cache\") on node \"ip-10-0-140-107.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:10:26.762698 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:10:26.762672 4939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/18846667-85a0-4bc2-8995-61bfb2ab1c5d-tls-certs" pod "18846667-85a0-4bc2-8995-61bfb2ab1c5d" (UID: "18846667-85a0-4bc2-8995-61bfb2ab1c5d"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 01:10:26.762796 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:10:26.762773 4939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/18846667-85a0-4bc2-8995-61bfb2ab1c5d-dshm" pod "18846667-85a0-4bc2-8995-61bfb2ab1c5d" (UID: "18846667-85a0-4bc2-8995-61bfb2ab1c5d"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 01:10:26.763011 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:10:26.762984 4939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/18846667-85a0-4bc2-8995-61bfb2ab1c5d-kube-api-access-4zxrl" pod "18846667-85a0-4bc2-8995-61bfb2ab1c5d" (UID: "18846667-85a0-4bc2-8995-61bfb2ab1c5d"). InnerVolumeSpecName "kube-api-access-4zxrl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 01:10:26.863969 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:10:26.863932 4939 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/18846667-85a0-4bc2-8995-61bfb2ab1c5d-tls-certs\") on node \"ip-10-0-140-107.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:10:26.863969 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:10:26.863965 4939 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/18846667-85a0-4bc2-8995-61bfb2ab1c5d-dshm\") on node \"ip-10-0-140-107.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:10:26.864186 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:10:26.863979 4939 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4zxrl\" (UniqueName: \"kubernetes.io/projected/18846667-85a0-4bc2-8995-61bfb2ab1c5d-kube-api-access-4zxrl\") on node \"ip-10-0-140-107.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:10:26.892026 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:10:26.891991 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:10:27.380332 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:10:27.380300 4939 generic.go:356] "Generic (PLEG): container finished" podID="18846667-85a0-4bc2-8995-61bfb2ab1c5d" containerID="dce8a0929af2abbec1201ec9933e9052670e290063b7823a598ecebb430fc3dc" exitCode=0 Nov 03 01:10:27.380548 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:10:27.380363 4939 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w4-4a697f/auth-test-model-4a697f-kserve-84b7466bd5-48hbr" Nov 03 01:10:27.380548 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:10:27.380393 4939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w4-4a697f/auth-test-model-4a697f-kserve-84b7466bd5-48hbr" event={"ID":"18846667-85a0-4bc2-8995-61bfb2ab1c5d","Type":"ContainerDied","Data":"dce8a0929af2abbec1201ec9933e9052670e290063b7823a598ecebb430fc3dc"} Nov 03 01:10:27.380548 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:10:27.380453 4939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w4-4a697f/auth-test-model-4a697f-kserve-84b7466bd5-48hbr" event={"ID":"18846667-85a0-4bc2-8995-61bfb2ab1c5d","Type":"ContainerDied","Data":"771600e7737d0443659f19c9961d3243b84937a107b514a471ab5f80cb0a2ea7"} Nov 03 01:10:27.380548 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:10:27.380481 4939 scope.go:122] "RemoveContainer" containerID="dce8a0929af2abbec1201ec9933e9052670e290063b7823a598ecebb430fc3dc" Nov 03 01:10:27.394496 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:10:27.394467 4939 scope.go:122] "RemoveContainer" containerID="dce8a0929af2abbec1201ec9933e9052670e290063b7823a598ecebb430fc3dc" Nov 03 01:10:27.394848 ip-10-0-140-107 kubenswrapper[4939]: E1103 01:10:27.394827 4939 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"dce8a0929af2abbec1201ec9933e9052670e290063b7823a598ecebb430fc3dc\": container with ID starting with dce8a0929af2abbec1201ec9933e9052670e290063b7823a598ecebb430fc3dc not found: ID does not exist" containerID="dce8a0929af2abbec1201ec9933e9052670e290063b7823a598ecebb430fc3dc" Nov 03 01:10:27.394971 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:10:27.394855 4939 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"dce8a0929af2abbec1201ec9933e9052670e290063b7823a598ecebb430fc3dc"} err="failed to get container status \"dce8a0929af2abbec1201ec9933e9052670e290063b7823a598ecebb430fc3dc\": rpc error: code = NotFound desc = could not find container \"dce8a0929af2abbec1201ec9933e9052670e290063b7823a598ecebb430fc3dc\": container with ID starting with dce8a0929af2abbec1201ec9933e9052670e290063b7823a598ecebb430fc3dc not found: ID does not exist" Nov 03 01:10:27.410284 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:10:27.410253 4939 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w4-4a697f/auth-test-model-4a697f-kserve-84b7466bd5-48hbr"] Nov 03 01:10:27.414828 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:10:27.414804 4939 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-w4-4a697f/auth-test-model-4a697f-kserve-84b7466bd5-48hbr"] Nov 03 01:10:27.573320 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:10:27.573279 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:10:27.872458 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:10:27.872424 4939 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="18846667-85a0-4bc2-8995-61bfb2ab1c5d" path="/var/lib/kubelet/pods/18846667-85a0-4bc2-8995-61bfb2ab1c5d/volumes" Nov 03 01:10:28.291099 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:10:28.290993 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:10:28.428933 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:10:28.427834 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:10:29.531845 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:10:29.531802 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:10:29.947758 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:10:29.947711 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:10:30.483182 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:10:30.483149 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:10:31.192843 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:10:31.192798 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:10:31.288707 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:10:31.288665 4939 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w0-020af9/auth-test-model-020af9-kserve-844df75d9c-ljr9s"] Nov 03 01:10:31.345131 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:10:31.345079 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:10:32.000826 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:10:32.000794 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:10:32.149361 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:10:32.149321 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:10:32.371511 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:10:32.371477 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:10:32.487607 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:10:32.487574 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:10:33.297297 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:10:33.297265 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:10:33.993850 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:10:33.993808 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:10:34.350548 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:10:34.350512 4939 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-7c759a-66695c8d79-7ws2z" Nov 03 01:10:34.371069 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:10:34.371036 4939 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-extmod-a-7c759a-7745bc7499-bmfl4" Nov 03 01:10:34.451223 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:10:34.451186 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:10:34.596497 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:10:34.596450 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:10:34.731989 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:10:34.730494 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:10:35.138139 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:10:35.138107 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:10:35.905327 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:10:35.905290 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:10:36.801103 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:10:36.801071 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:10:37.500051 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:10:37.499992 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:10:37.631808 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:10:37.631743 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:10:37.821041 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:10:37.820962 4939 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-4pw7v"] Nov 03 01:10:39.853616 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:10:39.853006 4939 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-f7680d/distinct-2-w2-f7680d-kserve-7d7c4564bf-cbv2c" podUID="2de18086-761b-4773-b9fc-15ddc4a220fc" containerName="main" probeResult="failure" output="Get \"https://10.132.0.75:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 01:10:39.879932 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:10:39.854752 4939 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-shared-a-w3-717217/auth-test-model-717217-kserve-5c9cb7dbd9-vl754" podUID="9634a946-0832-4901-ad01-bbb7a47f4ac7" containerName="main" probeResult="failure" output="Get \"https://10.132.0.191:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 01:10:40.083359 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:10:40.083288 4939 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-88bbf7/distinct-2-w1-88bbf7-kserve-5457cdcdbd-l6tfd" podUID="a397931b-faa9-4c2e-8290-491350546663" containerName="main" probeResult="failure" output="Get \"https://10.132.0.85:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 01:10:40.123102 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:10:40.122022 4939 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-88bbf7/simulator-w1-88bbf7-kserve-5fbb669f8f-42hgd" podUID="83b38883-2e7a-444d-aa68-52be4a25ae66" containerName="main" probeResult="failure" output="Get \"https://10.132.0.62:8000/ready\": context deadline exceeded" Nov 03 01:10:40.433458 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:10:40.430145 4939 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-shared-b-w0-020af9/auth-test-model-020af9-kserve-844df75d9c-ljr9s" podUID="24fb3670-ec70-492c-a573-4cbe174f1ccf" containerName="main" probeResult="failure" output="Get \"https://10.132.0.198:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 01:10:40.493031 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:10:40.492969 4939 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-83755a/distinct-2-w3-83755a-kserve-6cf66b995d-2l47b" podUID="c3b05e88-5d5b-4d03-b28d-832f22e3c868" containerName="main" probeResult="failure" output="Get \"https://10.132.0.72:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 01:10:40.697976 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:10:40.696758 4939 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-shared-b-w3-556c14/auth-test-model-556c14-kserve-8d5cd469-t6tzc" podUID="0c4e579c-9fe2-4ec9-b8f4-62758e519980" containerName="main" probeResult="failure" output="Get \"https://10.132.0.195:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 01:10:40.797160 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:10:40.797106 4939 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-2f9a3a/simulator-w0-2f9a3a-kserve-9596bbb8f-2trgz" podUID="207e2207-fb6c-4076-95df-72e95671c261" containerName="main" probeResult="failure" output="Get \"https://10.132.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 01:10:40.890934 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:10:40.857430 4939 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-f7680d/premium-w2-f7680d-kserve-76565b8cf9-hqbbt" podUID="4712e708-18a2-4a13-a12c-a5669397b2c3" containerName="main" probeResult="failure" output="Get \"https://10.132.0.63:8000/ready\": context deadline exceeded" Nov 03 01:10:40.919486 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:10:40.919441 4939 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-2f9a3a/distinct-w0-2f9a3a-kserve-555565cfcf-jtcr9" podUID="ac0ece06-8718-463f-915f-5315988d9711" containerName="main" probeResult="failure" output="Get \"https://10.132.0.76:8000/ready\": context deadline exceeded" Nov 03 01:10:41.167620 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:10:40.954062 4939 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-f7680d/distinct-2-w2-f7680d-kserve-7d7c4564bf-cbv2c" podUID="2de18086-761b-4773-b9fc-15ddc4a220fc" containerName="main" probeResult="failure" output="Get \"https://10.132.0.75:8000/ready\": context deadline exceeded" Nov 03 01:10:41.167620 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:10:41.021604 4939 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-shared-a-w3-717217/auth-test-model-717217-kserve-5c9cb7dbd9-vl754" podUID="9634a946-0832-4901-ad01-bbb7a47f4ac7" containerName="main" probeResult="failure" output="Get \"https://10.132.0.191:8000/ready\": context deadline exceeded" Nov 03 01:10:41.167620 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:10:41.145449 4939 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-f7680d/distinct-2-w2-f7680d-kserve-7d7c4564bf-cbv2c" Nov 03 01:10:41.167620 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:10:41.145506 4939 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-w3-717217/auth-test-model-717217-kserve-5c9cb7dbd9-vl754" Nov 03 01:10:41.167620 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:10:41.095142 4939 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-88bbf7/distinct-2-w1-88bbf7-kserve-5457cdcdbd-l6tfd" podUID="a397931b-faa9-4c2e-8290-491350546663" containerName="main" probeResult="failure" output="Get \"https://10.132.0.85:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 01:10:41.167620 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:10:41.150781 4939 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-88bbf7/distinct-2-w1-88bbf7-kserve-5457cdcdbd-l6tfd" Nov 03 01:10:41.279962 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:10:41.279926 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:10:41.341335 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:10:41.341025 4939 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-88bbf7/distinct-2-w1-88bbf7-kserve-5457cdcdbd-l6tfd" Nov 03 01:10:41.612538 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:10:41.612416 4939 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-f7680d/distinct-2-w2-f7680d-kserve-7d7c4564bf-cbv2c" Nov 03 01:10:41.612871 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:10:41.612801 4939 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-w3-717217/auth-test-model-717217-kserve-5c9cb7dbd9-vl754" Nov 03 01:10:42.122029 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:10:42.121992 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:10:42.733960 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:10:42.733868 4939 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-noten-a23ed0/noten-model-a23ed0-kserve-5dcdf8b6fc-fgrrv"] Nov 03 01:10:42.734959 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:10:42.734933 4939 cpu_manager.go:414] "containerMap: removing container" podUID="d21cb247-bacd-4af2-a7b1-d04b975f0993" containerName="main" Nov 03 01:10:42.734959 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:10:42.734955 4939 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d21cb247-bacd-4af2-a7b1-d04b975f0993" containerName="main" Nov 03 01:10:42.735155 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:10:42.735001 4939 cpu_manager.go:414] "containerMap: removing container" podUID="18846667-85a0-4bc2-8995-61bfb2ab1c5d" containerName="main" Nov 03 01:10:42.735155 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:10:42.735006 4939 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="18846667-85a0-4bc2-8995-61bfb2ab1c5d" containerName="main" Nov 03 01:10:42.735155 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:10:42.735151 4939 memory_manager.go:365] "RemoveStaleState removing state" podUID="d21cb247-bacd-4af2-a7b1-d04b975f0993" containerName="main" Nov 03 01:10:42.735325 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:10:42.735165 4939 memory_manager.go:365] "RemoveStaleState removing state" podUID="18846667-85a0-4bc2-8995-61bfb2ab1c5d" containerName="main" Nov 03 01:10:42.741214 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:10:42.740986 4939 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-noten-a23ed0/noten-model-a23ed0-kserve-5dcdf8b6fc-fgrrv" Nov 03 01:10:42.770932 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:10:42.767349 4939 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-noten-a23ed0\"/\"default-dockercfg-lkltw\"" Nov 03 01:10:42.770932 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:10:42.767626 4939 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-noten-a23ed0\"/\"openshift-service-ca.crt\"" Nov 03 01:10:42.770932 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:10:42.767689 4939 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-noten-a23ed0\"/\"kube-root-ca.crt\"" Nov 03 01:10:42.770932 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:10:42.768009 4939 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-noten-a23ed0\"/\"noten-model-a23ed0-kserve-self-signed-certs\"" Nov 03 01:10:42.781973 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:10:42.779048 4939 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-noten-a23ed0/noten-model-a23ed0-kserve-5dcdf8b6fc-fgrrv"] Nov 03 01:10:42.800900 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:10:42.800866 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:10:42.831252 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:10:42.831222 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/00fccd6d-ef84-4b47-9531-090a635045e9-home\") pod \"noten-model-a23ed0-kserve-5dcdf8b6fc-fgrrv\" (UID: \"00fccd6d-ef84-4b47-9531-090a635045e9\") " pod="ai-tenant-e2e-noten-a23ed0/noten-model-a23ed0-kserve-5dcdf8b6fc-fgrrv" Nov 03 01:10:42.831413 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:10:42.831370 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4z6x9\" (UniqueName: \"kubernetes.io/projected/00fccd6d-ef84-4b47-9531-090a635045e9-kube-api-access-4z6x9\") pod \"noten-model-a23ed0-kserve-5dcdf8b6fc-fgrrv\" (UID: \"00fccd6d-ef84-4b47-9531-090a635045e9\") " pod="ai-tenant-e2e-noten-a23ed0/noten-model-a23ed0-kserve-5dcdf8b6fc-fgrrv" Nov 03 01:10:42.831544 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:10:42.831523 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/00fccd6d-ef84-4b47-9531-090a635045e9-tls-certs\") pod \"noten-model-a23ed0-kserve-5dcdf8b6fc-fgrrv\" (UID: \"00fccd6d-ef84-4b47-9531-090a635045e9\") " pod="ai-tenant-e2e-noten-a23ed0/noten-model-a23ed0-kserve-5dcdf8b6fc-fgrrv" Nov 03 01:10:42.831611 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:10:42.831589 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/00fccd6d-ef84-4b47-9531-090a635045e9-tmp-dir\") pod \"noten-model-a23ed0-kserve-5dcdf8b6fc-fgrrv\" (UID: \"00fccd6d-ef84-4b47-9531-090a635045e9\") " pod="ai-tenant-e2e-noten-a23ed0/noten-model-a23ed0-kserve-5dcdf8b6fc-fgrrv" Nov 03 01:10:42.831727 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:10:42.831714 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/00fccd6d-ef84-4b47-9531-090a635045e9-model-cache\") pod \"noten-model-a23ed0-kserve-5dcdf8b6fc-fgrrv\" (UID: \"00fccd6d-ef84-4b47-9531-090a635045e9\") " pod="ai-tenant-e2e-noten-a23ed0/noten-model-a23ed0-kserve-5dcdf8b6fc-fgrrv" Nov 03 01:10:42.831786 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:10:42.831774 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/00fccd6d-ef84-4b47-9531-090a635045e9-dshm\") pod \"noten-model-a23ed0-kserve-5dcdf8b6fc-fgrrv\" (UID: \"00fccd6d-ef84-4b47-9531-090a635045e9\") " pod="ai-tenant-e2e-noten-a23ed0/noten-model-a23ed0-kserve-5dcdf8b6fc-fgrrv" Nov 03 01:10:42.944932 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:10:42.944542 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/00fccd6d-ef84-4b47-9531-090a635045e9-dshm\") pod \"noten-model-a23ed0-kserve-5dcdf8b6fc-fgrrv\" (UID: \"00fccd6d-ef84-4b47-9531-090a635045e9\") " pod="ai-tenant-e2e-noten-a23ed0/noten-model-a23ed0-kserve-5dcdf8b6fc-fgrrv" Nov 03 01:10:42.944932 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:10:42.944681 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/00fccd6d-ef84-4b47-9531-090a635045e9-home\") pod \"noten-model-a23ed0-kserve-5dcdf8b6fc-fgrrv\" (UID: \"00fccd6d-ef84-4b47-9531-090a635045e9\") " pod="ai-tenant-e2e-noten-a23ed0/noten-model-a23ed0-kserve-5dcdf8b6fc-fgrrv" Nov 03 01:10:42.944932 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:10:42.944860 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4z6x9\" (UniqueName: \"kubernetes.io/projected/00fccd6d-ef84-4b47-9531-090a635045e9-kube-api-access-4z6x9\") pod \"noten-model-a23ed0-kserve-5dcdf8b6fc-fgrrv\" (UID: \"00fccd6d-ef84-4b47-9531-090a635045e9\") " pod="ai-tenant-e2e-noten-a23ed0/noten-model-a23ed0-kserve-5dcdf8b6fc-fgrrv" Nov 03 01:10:42.946936 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:10:42.945951 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/00fccd6d-ef84-4b47-9531-090a635045e9-home\") pod \"noten-model-a23ed0-kserve-5dcdf8b6fc-fgrrv\" (UID: \"00fccd6d-ef84-4b47-9531-090a635045e9\") " pod="ai-tenant-e2e-noten-a23ed0/noten-model-a23ed0-kserve-5dcdf8b6fc-fgrrv" Nov 03 01:10:42.947096 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:10:42.947007 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/00fccd6d-ef84-4b47-9531-090a635045e9-tls-certs\") pod \"noten-model-a23ed0-kserve-5dcdf8b6fc-fgrrv\" (UID: \"00fccd6d-ef84-4b47-9531-090a635045e9\") " pod="ai-tenant-e2e-noten-a23ed0/noten-model-a23ed0-kserve-5dcdf8b6fc-fgrrv" Nov 03 01:10:42.947165 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:10:42.947098 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/00fccd6d-ef84-4b47-9531-090a635045e9-tmp-dir\") pod \"noten-model-a23ed0-kserve-5dcdf8b6fc-fgrrv\" (UID: \"00fccd6d-ef84-4b47-9531-090a635045e9\") " pod="ai-tenant-e2e-noten-a23ed0/noten-model-a23ed0-kserve-5dcdf8b6fc-fgrrv" Nov 03 01:10:42.950927 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:10:42.947245 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/00fccd6d-ef84-4b47-9531-090a635045e9-model-cache\") pod \"noten-model-a23ed0-kserve-5dcdf8b6fc-fgrrv\" (UID: \"00fccd6d-ef84-4b47-9531-090a635045e9\") " pod="ai-tenant-e2e-noten-a23ed0/noten-model-a23ed0-kserve-5dcdf8b6fc-fgrrv" Nov 03 01:10:42.950927 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:10:42.947603 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/00fccd6d-ef84-4b47-9531-090a635045e9-model-cache\") pod \"noten-model-a23ed0-kserve-5dcdf8b6fc-fgrrv\" (UID: \"00fccd6d-ef84-4b47-9531-090a635045e9\") " pod="ai-tenant-e2e-noten-a23ed0/noten-model-a23ed0-kserve-5dcdf8b6fc-fgrrv" Nov 03 01:10:42.963063 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:10:42.955418 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/00fccd6d-ef84-4b47-9531-090a635045e9-tmp-dir\") pod \"noten-model-a23ed0-kserve-5dcdf8b6fc-fgrrv\" (UID: \"00fccd6d-ef84-4b47-9531-090a635045e9\") " pod="ai-tenant-e2e-noten-a23ed0/noten-model-a23ed0-kserve-5dcdf8b6fc-fgrrv" Nov 03 01:10:42.993976 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:10:42.990419 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/00fccd6d-ef84-4b47-9531-090a635045e9-tls-certs\") pod \"noten-model-a23ed0-kserve-5dcdf8b6fc-fgrrv\" (UID: \"00fccd6d-ef84-4b47-9531-090a635045e9\") " pod="ai-tenant-e2e-noten-a23ed0/noten-model-a23ed0-kserve-5dcdf8b6fc-fgrrv" Nov 03 01:10:42.993976 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:10:42.990764 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4z6x9\" (UniqueName: \"kubernetes.io/projected/00fccd6d-ef84-4b47-9531-090a635045e9-kube-api-access-4z6x9\") pod \"noten-model-a23ed0-kserve-5dcdf8b6fc-fgrrv\" (UID: \"00fccd6d-ef84-4b47-9531-090a635045e9\") " pod="ai-tenant-e2e-noten-a23ed0/noten-model-a23ed0-kserve-5dcdf8b6fc-fgrrv" Nov 03 01:10:43.040379 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:10:43.035242 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/00fccd6d-ef84-4b47-9531-090a635045e9-dshm\") pod \"noten-model-a23ed0-kserve-5dcdf8b6fc-fgrrv\" (UID: \"00fccd6d-ef84-4b47-9531-090a635045e9\") " pod="ai-tenant-e2e-noten-a23ed0/noten-model-a23ed0-kserve-5dcdf8b6fc-fgrrv" Nov 03 01:10:43.097978 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:10:43.091222 4939 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-noten-a23ed0/noten-model-a23ed0-kserve-5dcdf8b6fc-fgrrv" Nov 03 01:10:43.326006 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:10:43.275292 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:10:43.632121 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:10:43.632089 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:10:43.634026 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:10:43.634002 4939 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-650be0-openshift-default-5c7c5fb7d9-flb5n"] Nov 03 01:10:43.635784 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:10:43.634301 4939 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-650be0-openshift-default-5c7c5fb7d9-flb5n" podUID="9146740f-e0d2-46cb-a063-b5942f02b275" containerName="istio-proxy" containerID="cri-o://45291eab55af92c5381548b238ac110b1f64a1980acbeeca8868720be4b7caf7" gracePeriod=30 Nov 03 01:10:43.823170 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:10:43.823113 4939 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-noten-a23ed0/noten-model-a23ed0-kserve-5dcdf8b6fc-fgrrv"] Nov 03 01:10:44.626780 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:10:44.626740 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:10:44.788019 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:10:44.787382 4939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-noten-a23ed0/noten-model-a23ed0-kserve-5dcdf8b6fc-fgrrv" event={"ID":"00fccd6d-ef84-4b47-9531-090a635045e9","Type":"ContainerStarted","Data":"1f517a1b2cdd36419ee083f592ddffd3640659eb91a3a4aefb4232e5cc05a3b5"} Nov 03 01:10:44.788324 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:10:44.788289 4939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-noten-a23ed0/noten-model-a23ed0-kserve-5dcdf8b6fc-fgrrv" event={"ID":"00fccd6d-ef84-4b47-9531-090a635045e9","Type":"ContainerStarted","Data":"013b52a9accacafc49e3cf3c85cb334218db6cc5f869d22e59ab66d61a98d481"} Nov 03 01:10:44.824004 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:10:44.819939 4939 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-noten-a23ed0/noten-model-a23ed0-kserve-5dcdf8b6fc-fgrrv" podStartSLOduration=2.81990289 podStartE2EDuration="2.81990289s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 01:10:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 01:10:44.818385515 +0000 UTC m=+1793.082720461" watchObservedRunningTime="2026-11-03 01:10:44.81990289 +0000 UTC m=+1793.084237836" Nov 03 01:10:45.125931 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:10:45.106650 4939 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-noten-a23ed0/noten-model-a23ed0-kserve-5dcdf8b6fc-fgrrv" Nov 03 01:10:45.857681 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:10:45.857637 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:10:46.316745 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:10:46.316660 4939 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-w0-020af9/auth-test-model-020af9-kserve-844df75d9c-ljr9s" podUID="24fb3670-ec70-492c-a573-4cbe174f1ccf" containerName="main" containerID="cri-o://ff3dfadc0f8d85525140106a5b520adcebba7b60f58a62b60ea5fc4464232299" gracePeriod=45 Nov 03 01:10:46.394062 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:10:46.394024 4939 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-shared-b-w0-020af9/auth-test-model-020af9-kserve-844df75d9c-ljr9s" podUID="24fb3670-ec70-492c-a573-4cbe174f1ccf" containerName="main" probeResult="failure" output="Get \"https://10.132.0.198:8000/ready\": dial tcp 10.132.0.198:8000: connect: connection refused" Nov 03 01:10:46.726152 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:10:46.726118 4939 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w0-020af9/auth-test-model-020af9-kserve-844df75d9c-ljr9s" Nov 03 01:10:46.788736 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:10:46.788691 4939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/24fb3670-ec70-492c-a573-4cbe174f1ccf-tls-certs\" (UniqueName: \"kubernetes.io/secret/24fb3670-ec70-492c-a573-4cbe174f1ccf-tls-certs\") pod \"24fb3670-ec70-492c-a573-4cbe174f1ccf\" (UID: \"24fb3670-ec70-492c-a573-4cbe174f1ccf\") " Nov 03 01:10:46.789113 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:10:46.789084 4939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/24fb3670-ec70-492c-a573-4cbe174f1ccf-kube-api-access-9w5qh\" (UniqueName: \"kubernetes.io/projected/24fb3670-ec70-492c-a573-4cbe174f1ccf-kube-api-access-9w5qh\") pod \"24fb3670-ec70-492c-a573-4cbe174f1ccf\" (UID: \"24fb3670-ec70-492c-a573-4cbe174f1ccf\") " Nov 03 01:10:46.789264 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:10:46.789245 4939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/24fb3670-ec70-492c-a573-4cbe174f1ccf-dshm\" (UniqueName: \"kubernetes.io/empty-dir/24fb3670-ec70-492c-a573-4cbe174f1ccf-dshm\") pod \"24fb3670-ec70-492c-a573-4cbe174f1ccf\" (UID: \"24fb3670-ec70-492c-a573-4cbe174f1ccf\") " Nov 03 01:10:46.789339 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:10:46.789287 4939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/24fb3670-ec70-492c-a573-4cbe174f1ccf-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/24fb3670-ec70-492c-a573-4cbe174f1ccf-tmp-dir\") pod \"24fb3670-ec70-492c-a573-4cbe174f1ccf\" (UID: \"24fb3670-ec70-492c-a573-4cbe174f1ccf\") " Nov 03 01:10:46.789339 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:10:46.789322 4939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/24fb3670-ec70-492c-a573-4cbe174f1ccf-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/24fb3670-ec70-492c-a573-4cbe174f1ccf-model-cache\") pod \"24fb3670-ec70-492c-a573-4cbe174f1ccf\" (UID: \"24fb3670-ec70-492c-a573-4cbe174f1ccf\") " Nov 03 01:10:46.789466 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:10:46.789399 4939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/24fb3670-ec70-492c-a573-4cbe174f1ccf-home\" (UniqueName: \"kubernetes.io/empty-dir/24fb3670-ec70-492c-a573-4cbe174f1ccf-home\") pod \"24fb3670-ec70-492c-a573-4cbe174f1ccf\" (UID: \"24fb3670-ec70-492c-a573-4cbe174f1ccf\") " Nov 03 01:10:46.790233 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:10:46.790194 4939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/24fb3670-ec70-492c-a573-4cbe174f1ccf-model-cache" pod "24fb3670-ec70-492c-a573-4cbe174f1ccf" (UID: "24fb3670-ec70-492c-a573-4cbe174f1ccf"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 01:10:46.790545 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:10:46.790469 4939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/24fb3670-ec70-492c-a573-4cbe174f1ccf-home" pod "24fb3670-ec70-492c-a573-4cbe174f1ccf" (UID: "24fb3670-ec70-492c-a573-4cbe174f1ccf"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 01:10:46.791003 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:10:46.790978 4939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/24fb3670-ec70-492c-a573-4cbe174f1ccf-tmp-dir" pod "24fb3670-ec70-492c-a573-4cbe174f1ccf" (UID: "24fb3670-ec70-492c-a573-4cbe174f1ccf"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 01:10:46.796368 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:10:46.796341 4939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/24fb3670-ec70-492c-a573-4cbe174f1ccf-kube-api-access-9w5qh" pod "24fb3670-ec70-492c-a573-4cbe174f1ccf" (UID: "24fb3670-ec70-492c-a573-4cbe174f1ccf"). InnerVolumeSpecName "kube-api-access-9w5qh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 01:10:46.807932 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:10:46.799012 4939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/24fb3670-ec70-492c-a573-4cbe174f1ccf-dshm" pod "24fb3670-ec70-492c-a573-4cbe174f1ccf" (UID: "24fb3670-ec70-492c-a573-4cbe174f1ccf"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 01:10:46.812528 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:10:46.812506 4939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/24fb3670-ec70-492c-a573-4cbe174f1ccf-tls-certs" pod "24fb3670-ec70-492c-a573-4cbe174f1ccf" (UID: "24fb3670-ec70-492c-a573-4cbe174f1ccf"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 01:10:46.813295 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:10:46.813268 4939 generic.go:356] "Generic (PLEG): container finished" podID="24fb3670-ec70-492c-a573-4cbe174f1ccf" containerID="ff3dfadc0f8d85525140106a5b520adcebba7b60f58a62b60ea5fc4464232299" exitCode=0 Nov 03 01:10:46.813404 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:10:46.813355 4939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w0-020af9/auth-test-model-020af9-kserve-844df75d9c-ljr9s" event={"ID":"24fb3670-ec70-492c-a573-4cbe174f1ccf","Type":"ContainerDied","Data":"ff3dfadc0f8d85525140106a5b520adcebba7b60f58a62b60ea5fc4464232299"} Nov 03 01:10:46.813468 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:10:46.813410 4939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w0-020af9/auth-test-model-020af9-kserve-844df75d9c-ljr9s" event={"ID":"24fb3670-ec70-492c-a573-4cbe174f1ccf","Type":"ContainerDied","Data":"a450effc06c776e21f25d5c7d0c5b30fcb4c07f7e9141688d1992124b5ad59b3"} Nov 03 01:10:46.813468 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:10:46.813437 4939 scope.go:122] "RemoveContainer" containerID="ff3dfadc0f8d85525140106a5b520adcebba7b60f58a62b60ea5fc4464232299" Nov 03 01:10:46.813630 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:10:46.813613 4939 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w0-020af9/auth-test-model-020af9-kserve-844df75d9c-ljr9s" Nov 03 01:10:46.839174 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:10:46.835511 4939 scope.go:122] "RemoveContainer" containerID="ff3dfadc0f8d85525140106a5b520adcebba7b60f58a62b60ea5fc4464232299" Nov 03 01:10:46.839174 ip-10-0-140-107 kubenswrapper[4939]: E1103 01:10:46.835830 4939 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ff3dfadc0f8d85525140106a5b520adcebba7b60f58a62b60ea5fc4464232299\": container with ID starting with ff3dfadc0f8d85525140106a5b520adcebba7b60f58a62b60ea5fc4464232299 not found: ID does not exist" containerID="ff3dfadc0f8d85525140106a5b520adcebba7b60f58a62b60ea5fc4464232299" Nov 03 01:10:46.839174 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:10:46.835860 4939 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ff3dfadc0f8d85525140106a5b520adcebba7b60f58a62b60ea5fc4464232299"} err="failed to get container status \"ff3dfadc0f8d85525140106a5b520adcebba7b60f58a62b60ea5fc4464232299\": rpc error: code = NotFound desc = could not find container \"ff3dfadc0f8d85525140106a5b520adcebba7b60f58a62b60ea5fc4464232299\": container with ID starting with ff3dfadc0f8d85525140106a5b520adcebba7b60f58a62b60ea5fc4464232299 not found: ID does not exist" Nov 03 01:10:46.884258 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:10:46.884043 4939 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w0-020af9/auth-test-model-020af9-kserve-844df75d9c-ljr9s"] Nov 03 01:10:46.889559 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:10:46.889529 4939 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-w0-020af9/auth-test-model-020af9-kserve-844df75d9c-ljr9s"] Nov 03 01:10:46.892872 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:10:46.892846 4939 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/24fb3670-ec70-492c-a573-4cbe174f1ccf-dshm\") on node \"ip-10-0-140-107.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:10:46.893066 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:10:46.892876 4939 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/24fb3670-ec70-492c-a573-4cbe174f1ccf-tmp-dir\") on node \"ip-10-0-140-107.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:10:46.893066 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:10:46.892892 4939 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/24fb3670-ec70-492c-a573-4cbe174f1ccf-model-cache\") on node \"ip-10-0-140-107.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:10:46.893066 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:10:46.892919 4939 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/24fb3670-ec70-492c-a573-4cbe174f1ccf-home\") on node \"ip-10-0-140-107.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:10:46.893066 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:10:46.892934 4939 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/24fb3670-ec70-492c-a573-4cbe174f1ccf-tls-certs\") on node \"ip-10-0-140-107.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:10:46.893066 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:10:46.892947 4939 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9w5qh\" (UniqueName: \"kubernetes.io/projected/24fb3670-ec70-492c-a573-4cbe174f1ccf-kube-api-access-9w5qh\") on node \"ip-10-0-140-107.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:10:47.086517 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:10:47.086485 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:10:47.504153 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:10:47.504072 4939 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-noten-a23ed0/noten-model-a23ed0-kserve-5dcdf8b6fc-fgrrv"] Nov 03 01:10:47.578131 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:10:47.578098 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:10:47.763992 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:10:47.761391 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:10:48.439811 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:10:48.439733 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:10:48.471820 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:10:48.471782 4939 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="24fb3670-ec70-492c-a573-4cbe174f1ccf" path="/var/lib/kubelet/pods/24fb3670-ec70-492c-a573-4cbe174f1ccf/volumes" Nov 03 01:10:48.525692 ip-10-0-140-107 kubenswrapper[4939]: E1103 01:10:48.525649 4939 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-sthjh" containerName="extractor" Nov 03 01:10:48.591921 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:10:48.591871 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:10:48.740977 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:10:48.732822 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:10:49.034199 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:10:49.034173 4939 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-650be0-openshift-default-5c7c5fb7d9-flb5n" Nov 03 01:10:49.073697 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:10:49.073663 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:10:49.211048 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:10:49.211014 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:10:49.218523 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:10:49.218492 4939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9146740f-e0d2-46cb-a063-b5942f02b275-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9146740f-e0d2-46cb-a063-b5942f02b275-credential-socket\") pod \"9146740f-e0d2-46cb-a063-b5942f02b275\" (UID: \"9146740f-e0d2-46cb-a063-b5942f02b275\") " Nov 03 01:10:49.218661 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:10:49.218613 4939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/9146740f-e0d2-46cb-a063-b5942f02b275-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9146740f-e0d2-46cb-a063-b5942f02b275-istio-podinfo\") pod \"9146740f-e0d2-46cb-a063-b5942f02b275\" (UID: \"9146740f-e0d2-46cb-a063-b5942f02b275\") " Nov 03 01:10:49.218704 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:10:49.218666 4939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9146740f-e0d2-46cb-a063-b5942f02b275-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9146740f-e0d2-46cb-a063-b5942f02b275-istio-envoy\") pod \"9146740f-e0d2-46cb-a063-b5942f02b275\" (UID: \"9146740f-e0d2-46cb-a063-b5942f02b275\") " Nov 03 01:10:49.218777 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:10:49.218751 4939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9146740f-e0d2-46cb-a063-b5942f02b275-credential-socket" pod "9146740f-e0d2-46cb-a063-b5942f02b275" (UID: "9146740f-e0d2-46cb-a063-b5942f02b275"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 01:10:49.218821 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:10:49.218762 4939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9146740f-e0d2-46cb-a063-b5942f02b275-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9146740f-e0d2-46cb-a063-b5942f02b275-workload-socket\") pod \"9146740f-e0d2-46cb-a063-b5942f02b275\" (UID: \"9146740f-e0d2-46cb-a063-b5942f02b275\") " Nov 03 01:10:49.219075 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:10:49.219050 4939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9146740f-e0d2-46cb-a063-b5942f02b275-workload-socket" pod "9146740f-e0d2-46cb-a063-b5942f02b275" (UID: "9146740f-e0d2-46cb-a063-b5942f02b275"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 01:10:49.219075 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:10:49.219055 4939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9146740f-e0d2-46cb-a063-b5942f02b275-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9146740f-e0d2-46cb-a063-b5942f02b275-istiod-ca-cert\") pod \"9146740f-e0d2-46cb-a063-b5942f02b275\" (UID: \"9146740f-e0d2-46cb-a063-b5942f02b275\") " Nov 03 01:10:49.219270 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:10:49.219241 4939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9146740f-e0d2-46cb-a063-b5942f02b275-istio-token\" (UniqueName: \"kubernetes.io/projected/9146740f-e0d2-46cb-a063-b5942f02b275-istio-token\") pod \"9146740f-e0d2-46cb-a063-b5942f02b275\" (UID: \"9146740f-e0d2-46cb-a063-b5942f02b275\") " Nov 03 01:10:49.219336 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:10:49.219282 4939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9146740f-e0d2-46cb-a063-b5942f02b275-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9146740f-e0d2-46cb-a063-b5942f02b275-workload-certs\") pod \"9146740f-e0d2-46cb-a063-b5942f02b275\" (UID: \"9146740f-e0d2-46cb-a063-b5942f02b275\") " Nov 03 01:10:49.219431 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:10:49.219326 4939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9146740f-e0d2-46cb-a063-b5942f02b275-istiod-ca-cert" pod "9146740f-e0d2-46cb-a063-b5942f02b275" (UID: "9146740f-e0d2-46cb-a063-b5942f02b275"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 01:10:49.219494 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:10:49.219451 4939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9146740f-e0d2-46cb-a063-b5942f02b275-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9146740f-e0d2-46cb-a063-b5942f02b275-istio-data\") pod \"9146740f-e0d2-46cb-a063-b5942f02b275\" (UID: \"9146740f-e0d2-46cb-a063-b5942f02b275\") " Nov 03 01:10:49.219795 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:10:49.219768 4939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9146740f-e0d2-46cb-a063-b5942f02b275-workload-certs" pod "9146740f-e0d2-46cb-a063-b5942f02b275" (UID: "9146740f-e0d2-46cb-a063-b5942f02b275"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 01:10:49.219795 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:10:49.219774 4939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9146740f-e0d2-46cb-a063-b5942f02b275-kube-api-access-clt5s\" (UniqueName: \"kubernetes.io/projected/9146740f-e0d2-46cb-a063-b5942f02b275-kube-api-access-clt5s\") pod \"9146740f-e0d2-46cb-a063-b5942f02b275\" (UID: \"9146740f-e0d2-46cb-a063-b5942f02b275\") " Nov 03 01:10:49.220107 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:10:49.219887 4939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9146740f-e0d2-46cb-a063-b5942f02b275-istio-data" pod "9146740f-e0d2-46cb-a063-b5942f02b275" (UID: "9146740f-e0d2-46cb-a063-b5942f02b275"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 01:10:49.220768 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:10:49.220741 4939 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9146740f-e0d2-46cb-a063-b5942f02b275-workload-socket\") on node \"ip-10-0-140-107.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:10:49.220871 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:10:49.220773 4939 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9146740f-e0d2-46cb-a063-b5942f02b275-istiod-ca-cert\") on node \"ip-10-0-140-107.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:10:49.220871 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:10:49.220788 4939 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9146740f-e0d2-46cb-a063-b5942f02b275-workload-certs\") on node \"ip-10-0-140-107.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:10:49.220871 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:10:49.220802 4939 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9146740f-e0d2-46cb-a063-b5942f02b275-istio-data\") on node \"ip-10-0-140-107.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:10:49.220871 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:10:49.220818 4939 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9146740f-e0d2-46cb-a063-b5942f02b275-credential-socket\") on node \"ip-10-0-140-107.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:10:49.221547 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:10:49.221522 4939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9146740f-e0d2-46cb-a063-b5942f02b275-istio-envoy" pod "9146740f-e0d2-46cb-a063-b5942f02b275" (UID: "9146740f-e0d2-46cb-a063-b5942f02b275"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 01:10:49.221657 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:10:49.221546 4939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/9146740f-e0d2-46cb-a063-b5942f02b275-istio-podinfo" pod "9146740f-e0d2-46cb-a063-b5942f02b275" (UID: "9146740f-e0d2-46cb-a063-b5942f02b275"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 01:10:49.222102 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:10:49.222077 4939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9146740f-e0d2-46cb-a063-b5942f02b275-kube-api-access-clt5s" pod "9146740f-e0d2-46cb-a063-b5942f02b275" (UID: "9146740f-e0d2-46cb-a063-b5942f02b275"). InnerVolumeSpecName "kube-api-access-clt5s". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 01:10:49.222869 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:10:49.222850 4939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9146740f-e0d2-46cb-a063-b5942f02b275-istio-token" pod "9146740f-e0d2-46cb-a063-b5942f02b275" (UID: "9146740f-e0d2-46cb-a063-b5942f02b275"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 01:10:49.290845 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:10:49.290718 4939 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-2f9a3a-openshift-default-7f6f74cf97-5rvtq"] Nov 03 01:10:49.291115 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:10:49.291051 4939 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-2f9a3a-openshift-default-7f6f74cf97-5rvtq" podUID="9c96c382-a625-4b0c-b72f-82807daf6f3e" containerName="istio-proxy" containerID="cri-o://6e90b367fe9525a005eaec42d07c9414aa9906b5a68d215c39929f17aa671616" gracePeriod=30 Nov 03 01:10:49.297511 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:10:49.297360 4939 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-2f9a3a-openshift-default-7f6f74cf97-cv2rj"] Nov 03 01:10:49.297952 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:10:49.297880 4939 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-2f9a3a-openshift-default-7f6f74cf97-cv2rj" podUID="f4f0fa4a-d49c-46d2-a1af-9195565e8bc2" containerName="istio-proxy" containerID="cri-o://60f729127ba803a14855cf8bdd33d7fffb8a91e0a8513cc7d87e84487d7002c3" gracePeriod=30 Nov 03 01:10:49.302987 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:10:49.302960 4939 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-2f9a3a-openshift-default-7f6f74cf97-ckb7r"] Nov 03 01:10:49.303214 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:10:49.303189 4939 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-2f9a3a-openshift-default-7f6f74cf97-ckb7r" podUID="ee090f53-4805-4a1a-a79f-450b8bc77079" containerName="istio-proxy" containerID="cri-o://9ce5ab3f82c59a07fcab8e231933c24e8c586c6194d2db303bcb6d65033bf2f8" gracePeriod=30 Nov 03 01:10:49.328377 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:10:49.328336 4939 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-clt5s\" (UniqueName: \"kubernetes.io/projected/9146740f-e0d2-46cb-a063-b5942f02b275-kube-api-access-clt5s\") on node \"ip-10-0-140-107.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:10:49.328377 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:10:49.328378 4939 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9146740f-e0d2-46cb-a063-b5942f02b275-istio-podinfo\") on node \"ip-10-0-140-107.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:10:49.328605 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:10:49.328394 4939 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9146740f-e0d2-46cb-a063-b5942f02b275-istio-envoy\") on node \"ip-10-0-140-107.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:10:49.328605 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:10:49.328409 4939 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9146740f-e0d2-46cb-a063-b5942f02b275-istio-token\") on node \"ip-10-0-140-107.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:10:49.374514 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:10:49.374477 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:10:49.577174 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:10:49.574810 4939 generic.go:356] "Generic (PLEG): container finished" podID="9146740f-e0d2-46cb-a063-b5942f02b275" containerID="45291eab55af92c5381548b238ac110b1f64a1980acbeeca8868720be4b7caf7" exitCode=0 Nov 03 01:10:49.577174 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:10:49.575121 4939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-650be0-openshift-default-5c7c5fb7d9-flb5n" event={"ID":"9146740f-e0d2-46cb-a063-b5942f02b275","Type":"ContainerDied","Data":"45291eab55af92c5381548b238ac110b1f64a1980acbeeca8868720be4b7caf7"} Nov 03 01:10:49.577174 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:10:49.575185 4939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-650be0-openshift-default-5c7c5fb7d9-flb5n" event={"ID":"9146740f-e0d2-46cb-a063-b5942f02b275","Type":"ContainerDied","Data":"46950e63eca92b4fc2960c90e6f76b7348913958ba02e6e3d776783b8a3fecf7"} Nov 03 01:10:49.577174 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:10:49.575216 4939 scope.go:122] "RemoveContainer" containerID="45291eab55af92c5381548b238ac110b1f64a1980acbeeca8868720be4b7caf7" Nov 03 01:10:49.577174 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:10:49.575384 4939 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-650be0-openshift-default-5c7c5fb7d9-flb5n" Nov 03 01:10:49.606078 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:10:49.606053 4939 scope.go:122] "RemoveContainer" containerID="45291eab55af92c5381548b238ac110b1f64a1980acbeeca8868720be4b7caf7" Nov 03 01:10:49.606471 ip-10-0-140-107 kubenswrapper[4939]: E1103 01:10:49.606435 4939 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"45291eab55af92c5381548b238ac110b1f64a1980acbeeca8868720be4b7caf7\": container with ID starting with 45291eab55af92c5381548b238ac110b1f64a1980acbeeca8868720be4b7caf7 not found: ID does not exist" containerID="45291eab55af92c5381548b238ac110b1f64a1980acbeeca8868720be4b7caf7" Nov 03 01:10:49.606589 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:10:49.606470 4939 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"45291eab55af92c5381548b238ac110b1f64a1980acbeeca8868720be4b7caf7"} err="failed to get container status \"45291eab55af92c5381548b238ac110b1f64a1980acbeeca8868720be4b7caf7\": rpc error: code = NotFound desc = could not find container \"45291eab55af92c5381548b238ac110b1f64a1980acbeeca8868720be4b7caf7\": container with ID starting with 45291eab55af92c5381548b238ac110b1f64a1980acbeeca8868720be4b7caf7 not found: ID does not exist" Nov 03 01:10:49.621187 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:10:49.621160 4939 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-650be0-openshift-default-5c7c5fb7d9-flb5n"] Nov 03 01:10:49.625951 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:10:49.625922 4939 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-650be0-openshift-default-5c7c5fb7d9-flb5n"] Nov 03 01:10:49.636645 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:10:49.636593 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:10:49.758402 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:10:49.758322 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:10:50.026653 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:10:50.021016 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:10:50.035600 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:10:50.035562 4939 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9146740f-e0d2-46cb-a063-b5942f02b275" path="/var/lib/kubelet/pods/9146740f-e0d2-46cb-a063-b5942f02b275/volumes" Nov 03 01:10:50.419147 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:10:50.419098 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:10:50.539270 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:10:50.539232 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:10:51.390606 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:10:51.390553 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:10:51.510667 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:10:51.510624 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:10:52.398140 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:10:52.398103 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:10:52.601187 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:10:52.601156 4939 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-n42s6_65d134ac-8a25-4c67-b6c7-cd752a098daf/console-operator/2.log" Nov 03 01:10:52.659852 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:10:52.659786 4939 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-n42s6_65d134ac-8a25-4c67-b6c7-cd752a098daf/console-operator/2.log" Nov 03 01:10:53.210934 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:10:53.210296 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:10:53.453403 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:10:53.453357 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:10:54.291856 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:10:54.291817 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:10:54.700022 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:10:54.699987 4939 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-2f9a3a-openshift-default-7f6f74cf97-ckb7r" Nov 03 01:10:54.721353 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:10:54.721329 4939 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-2f9a3a-openshift-default-7f6f74cf97-cv2rj" Nov 03 01:10:54.777433 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:10:54.777404 4939 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-2f9a3a-openshift-default-7f6f74cf97-5rvtq" Nov 03 01:10:54.887041 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:10:54.887005 4939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ee090f53-4805-4a1a-a79f-450b8bc77079-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ee090f53-4805-4a1a-a79f-450b8bc77079-workload-socket\") pod \"ee090f53-4805-4a1a-a79f-450b8bc77079\" (UID: \"ee090f53-4805-4a1a-a79f-450b8bc77079\") " Nov 03 01:10:54.887256 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:10:54.887066 4939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f4f0fa4a-d49c-46d2-a1af-9195565e8bc2-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f4f0fa4a-d49c-46d2-a1af-9195565e8bc2-istiod-ca-cert\") pod \"f4f0fa4a-d49c-46d2-a1af-9195565e8bc2\" (UID: \"f4f0fa4a-d49c-46d2-a1af-9195565e8bc2\") " Nov 03 01:10:54.887256 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:10:54.887104 4939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ee090f53-4805-4a1a-a79f-450b8bc77079-istio-token\" (UniqueName: \"kubernetes.io/projected/ee090f53-4805-4a1a-a79f-450b8bc77079-istio-token\") pod \"ee090f53-4805-4a1a-a79f-450b8bc77079\" (UID: \"ee090f53-4805-4a1a-a79f-450b8bc77079\") " Nov 03 01:10:54.887256 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:10:54.887154 4939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ee090f53-4805-4a1a-a79f-450b8bc77079-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ee090f53-4805-4a1a-a79f-450b8bc77079-istio-data\") pod \"ee090f53-4805-4a1a-a79f-450b8bc77079\" (UID: \"ee090f53-4805-4a1a-a79f-450b8bc77079\") " Nov 03 01:10:54.887459 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:10:54.887274 4939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f4f0fa4a-d49c-46d2-a1af-9195565e8bc2-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f4f0fa4a-d49c-46d2-a1af-9195565e8bc2-credential-socket\") pod \"f4f0fa4a-d49c-46d2-a1af-9195565e8bc2\" (UID: \"f4f0fa4a-d49c-46d2-a1af-9195565e8bc2\") " Nov 03 01:10:54.887459 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:10:54.887352 4939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/ee090f53-4805-4a1a-a79f-450b8bc77079-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ee090f53-4805-4a1a-a79f-450b8bc77079-istio-podinfo\") pod \"ee090f53-4805-4a1a-a79f-450b8bc77079\" (UID: \"ee090f53-4805-4a1a-a79f-450b8bc77079\") " Nov 03 01:10:54.887459 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:10:54.887357 4939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ee090f53-4805-4a1a-a79f-450b8bc77079-workload-socket" pod "ee090f53-4805-4a1a-a79f-450b8bc77079" (UID: "ee090f53-4805-4a1a-a79f-450b8bc77079"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 01:10:54.887631 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:10:54.887463 4939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f4f0fa4a-d49c-46d2-a1af-9195565e8bc2-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f4f0fa4a-d49c-46d2-a1af-9195565e8bc2-workload-certs\") pod \"f4f0fa4a-d49c-46d2-a1af-9195565e8bc2\" (UID: \"f4f0fa4a-d49c-46d2-a1af-9195565e8bc2\") " Nov 03 01:10:54.887631 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:10:54.887503 4939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9c96c382-a625-4b0c-b72f-82807daf6f3e-istio-token\" (UniqueName: \"kubernetes.io/projected/9c96c382-a625-4b0c-b72f-82807daf6f3e-istio-token\") pod \"9c96c382-a625-4b0c-b72f-82807daf6f3e\" (UID: \"9c96c382-a625-4b0c-b72f-82807daf6f3e\") " Nov 03 01:10:54.887631 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:10:54.887571 4939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ee090f53-4805-4a1a-a79f-450b8bc77079-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ee090f53-4805-4a1a-a79f-450b8bc77079-istio-envoy\") pod \"ee090f53-4805-4a1a-a79f-450b8bc77079\" (UID: \"ee090f53-4805-4a1a-a79f-450b8bc77079\") " Nov 03 01:10:54.887631 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:10:54.887505 4939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f4f0fa4a-d49c-46d2-a1af-9195565e8bc2-istiod-ca-cert" pod "f4f0fa4a-d49c-46d2-a1af-9195565e8bc2" (UID: "f4f0fa4a-d49c-46d2-a1af-9195565e8bc2"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 01:10:54.887631 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:10:54.887599 4939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f4f0fa4a-d49c-46d2-a1af-9195565e8bc2-credential-socket" pod "f4f0fa4a-d49c-46d2-a1af-9195565e8bc2" (UID: "f4f0fa4a-d49c-46d2-a1af-9195565e8bc2"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 01:10:54.887899 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:10:54.887638 4939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ee090f53-4805-4a1a-a79f-450b8bc77079-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ee090f53-4805-4a1a-a79f-450b8bc77079-istiod-ca-cert\") pod \"ee090f53-4805-4a1a-a79f-450b8bc77079\" (UID: \"ee090f53-4805-4a1a-a79f-450b8bc77079\") " Nov 03 01:10:54.887899 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:10:54.887694 4939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f4f0fa4a-d49c-46d2-a1af-9195565e8bc2-istio-token\" (UniqueName: \"kubernetes.io/projected/f4f0fa4a-d49c-46d2-a1af-9195565e8bc2-istio-token\") pod \"f4f0fa4a-d49c-46d2-a1af-9195565e8bc2\" (UID: \"f4f0fa4a-d49c-46d2-a1af-9195565e8bc2\") " Nov 03 01:10:54.887899 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:10:54.887734 4939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f4f0fa4a-d49c-46d2-a1af-9195565e8bc2-workload-certs" pod "f4f0fa4a-d49c-46d2-a1af-9195565e8bc2" (UID: "f4f0fa4a-d49c-46d2-a1af-9195565e8bc2"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 01:10:54.887899 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:10:54.887749 4939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9c96c382-a625-4b0c-b72f-82807daf6f3e-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9c96c382-a625-4b0c-b72f-82807daf6f3e-credential-socket\") pod \"9c96c382-a625-4b0c-b72f-82807daf6f3e\" (UID: \"9c96c382-a625-4b0c-b72f-82807daf6f3e\") " Nov 03 01:10:54.887899 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:10:54.887789 4939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9c96c382-a625-4b0c-b72f-82807daf6f3e-kube-api-access-9g45v\" (UniqueName: \"kubernetes.io/projected/9c96c382-a625-4b0c-b72f-82807daf6f3e-kube-api-access-9g45v\") pod \"9c96c382-a625-4b0c-b72f-82807daf6f3e\" (UID: \"9c96c382-a625-4b0c-b72f-82807daf6f3e\") " Nov 03 01:10:54.887899 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:10:54.887822 4939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/9c96c382-a625-4b0c-b72f-82807daf6f3e-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9c96c382-a625-4b0c-b72f-82807daf6f3e-istio-podinfo\") pod \"9c96c382-a625-4b0c-b72f-82807daf6f3e\" (UID: \"9c96c382-a625-4b0c-b72f-82807daf6f3e\") " Nov 03 01:10:54.887899 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:10:54.887862 4939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f4f0fa4a-d49c-46d2-a1af-9195565e8bc2-kube-api-access-str2z\" (UniqueName: \"kubernetes.io/projected/f4f0fa4a-d49c-46d2-a1af-9195565e8bc2-kube-api-access-str2z\") pod \"f4f0fa4a-d49c-46d2-a1af-9195565e8bc2\" (UID: \"f4f0fa4a-d49c-46d2-a1af-9195565e8bc2\") " Nov 03 01:10:54.888291 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:10:54.887903 4939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ee090f53-4805-4a1a-a79f-450b8bc77079-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ee090f53-4805-4a1a-a79f-450b8bc77079-workload-certs\") pod \"ee090f53-4805-4a1a-a79f-450b8bc77079\" (UID: \"ee090f53-4805-4a1a-a79f-450b8bc77079\") " Nov 03 01:10:54.888291 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:10:54.887970 4939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9c96c382-a625-4b0c-b72f-82807daf6f3e-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9c96c382-a625-4b0c-b72f-82807daf6f3e-istio-envoy\") pod \"9c96c382-a625-4b0c-b72f-82807daf6f3e\" (UID: \"9c96c382-a625-4b0c-b72f-82807daf6f3e\") " Nov 03 01:10:54.888291 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:10:54.888067 4939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9c96c382-a625-4b0c-b72f-82807daf6f3e-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9c96c382-a625-4b0c-b72f-82807daf6f3e-istio-data\") pod \"9c96c382-a625-4b0c-b72f-82807daf6f3e\" (UID: \"9c96c382-a625-4b0c-b72f-82807daf6f3e\") " Nov 03 01:10:54.888291 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:10:54.888091 4939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ee090f53-4805-4a1a-a79f-450b8bc77079-istiod-ca-cert" pod "ee090f53-4805-4a1a-a79f-450b8bc77079" (UID: "ee090f53-4805-4a1a-a79f-450b8bc77079"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 01:10:54.888291 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:10:54.888100 4939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9c96c382-a625-4b0c-b72f-82807daf6f3e-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9c96c382-a625-4b0c-b72f-82807daf6f3e-istiod-ca-cert\") pod \"9c96c382-a625-4b0c-b72f-82807daf6f3e\" (UID: \"9c96c382-a625-4b0c-b72f-82807daf6f3e\") " Nov 03 01:10:54.888291 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:10:54.888181 4939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f4f0fa4a-d49c-46d2-a1af-9195565e8bc2-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f4f0fa4a-d49c-46d2-a1af-9195565e8bc2-istio-data\") pod \"f4f0fa4a-d49c-46d2-a1af-9195565e8bc2\" (UID: \"f4f0fa4a-d49c-46d2-a1af-9195565e8bc2\") " Nov 03 01:10:54.888291 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:10:54.888212 4939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9c96c382-a625-4b0c-b72f-82807daf6f3e-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9c96c382-a625-4b0c-b72f-82807daf6f3e-workload-socket\") pod \"9c96c382-a625-4b0c-b72f-82807daf6f3e\" (UID: \"9c96c382-a625-4b0c-b72f-82807daf6f3e\") " Nov 03 01:10:54.888680 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:10:54.888435 4939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9c96c382-a625-4b0c-b72f-82807daf6f3e-istiod-ca-cert" pod "9c96c382-a625-4b0c-b72f-82807daf6f3e" (UID: "9c96c382-a625-4b0c-b72f-82807daf6f3e"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 01:10:54.888803 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:10:54.888775 4939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f4f0fa4a-d49c-46d2-a1af-9195565e8bc2-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f4f0fa4a-d49c-46d2-a1af-9195565e8bc2-workload-socket\") pod \"f4f0fa4a-d49c-46d2-a1af-9195565e8bc2\" (UID: \"f4f0fa4a-d49c-46d2-a1af-9195565e8bc2\") " Nov 03 01:10:54.888901 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:10:54.888881 4939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ee090f53-4805-4a1a-a79f-450b8bc77079-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ee090f53-4805-4a1a-a79f-450b8bc77079-credential-socket\") pod \"ee090f53-4805-4a1a-a79f-450b8bc77079\" (UID: \"ee090f53-4805-4a1a-a79f-450b8bc77079\") " Nov 03 01:10:54.888901 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:10:54.888933 4939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/f4f0fa4a-d49c-46d2-a1af-9195565e8bc2-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f4f0fa4a-d49c-46d2-a1af-9195565e8bc2-istio-podinfo\") pod \"f4f0fa4a-d49c-46d2-a1af-9195565e8bc2\" (UID: \"f4f0fa4a-d49c-46d2-a1af-9195565e8bc2\") " Nov 03 01:10:54.889113 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:10:54.889091 4939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ee090f53-4805-4a1a-a79f-450b8bc77079-kube-api-access-jkrsh\" (UniqueName: \"kubernetes.io/projected/ee090f53-4805-4a1a-a79f-450b8bc77079-kube-api-access-jkrsh\") pod \"ee090f53-4805-4a1a-a79f-450b8bc77079\" (UID: \"ee090f53-4805-4a1a-a79f-450b8bc77079\") " Nov 03 01:10:54.889182 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:10:54.889132 4939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9c96c382-a625-4b0c-b72f-82807daf6f3e-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9c96c382-a625-4b0c-b72f-82807daf6f3e-workload-certs\") pod \"9c96c382-a625-4b0c-b72f-82807daf6f3e\" (UID: \"9c96c382-a625-4b0c-b72f-82807daf6f3e\") " Nov 03 01:10:54.889182 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:10:54.889170 4939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f4f0fa4a-d49c-46d2-a1af-9195565e8bc2-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f4f0fa4a-d49c-46d2-a1af-9195565e8bc2-istio-envoy\") pod \"f4f0fa4a-d49c-46d2-a1af-9195565e8bc2\" (UID: \"f4f0fa4a-d49c-46d2-a1af-9195565e8bc2\") " Nov 03 01:10:54.890515 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:10:54.890481 4939 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ee090f53-4805-4a1a-a79f-450b8bc77079-workload-socket\") on node \"ip-10-0-140-107.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:10:54.890515 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:10:54.890514 4939 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f4f0fa4a-d49c-46d2-a1af-9195565e8bc2-istiod-ca-cert\") on node \"ip-10-0-140-107.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:10:54.890708 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:10:54.890530 4939 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f4f0fa4a-d49c-46d2-a1af-9195565e8bc2-credential-socket\") on node \"ip-10-0-140-107.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:10:54.890708 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:10:54.890545 4939 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f4f0fa4a-d49c-46d2-a1af-9195565e8bc2-workload-certs\") on node \"ip-10-0-140-107.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:10:54.890708 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:10:54.890559 4939 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ee090f53-4805-4a1a-a79f-450b8bc77079-istiod-ca-cert\") on node \"ip-10-0-140-107.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:10:54.890708 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:10:54.890572 4939 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9c96c382-a625-4b0c-b72f-82807daf6f3e-istiod-ca-cert\") on node \"ip-10-0-140-107.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:10:54.890708 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:10:54.890569 4939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ee090f53-4805-4a1a-a79f-450b8bc77079-credential-socket" pod "ee090f53-4805-4a1a-a79f-450b8bc77079" (UID: "ee090f53-4805-4a1a-a79f-450b8bc77079"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 01:10:54.890893 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:10:54.890850 4939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f4f0fa4a-d49c-46d2-a1af-9195565e8bc2-istio-data" pod "f4f0fa4a-d49c-46d2-a1af-9195565e8bc2" (UID: "f4f0fa4a-d49c-46d2-a1af-9195565e8bc2"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 01:10:54.891121 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:10:54.891074 4939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ee090f53-4805-4a1a-a79f-450b8bc77079-istio-data" pod "ee090f53-4805-4a1a-a79f-450b8bc77079" (UID: "ee090f53-4805-4a1a-a79f-450b8bc77079"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 01:10:54.891207 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:10:54.891132 4939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9c96c382-a625-4b0c-b72f-82807daf6f3e-workload-socket" pod "9c96c382-a625-4b0c-b72f-82807daf6f3e" (UID: "9c96c382-a625-4b0c-b72f-82807daf6f3e"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 01:10:54.891313 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:10:54.891292 4939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f4f0fa4a-d49c-46d2-a1af-9195565e8bc2-workload-socket" pod "f4f0fa4a-d49c-46d2-a1af-9195565e8bc2" (UID: "f4f0fa4a-d49c-46d2-a1af-9195565e8bc2"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 01:10:54.891951 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:10:54.891925 4939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9c96c382-a625-4b0c-b72f-82807daf6f3e-kube-api-access-9g45v" pod "9c96c382-a625-4b0c-b72f-82807daf6f3e" (UID: "9c96c382-a625-4b0c-b72f-82807daf6f3e"). InnerVolumeSpecName "kube-api-access-9g45v". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 01:10:54.892069 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:10:54.892025 4939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/9c96c382-a625-4b0c-b72f-82807daf6f3e-istio-podinfo" pod "9c96c382-a625-4b0c-b72f-82807daf6f3e" (UID: "9c96c382-a625-4b0c-b72f-82807daf6f3e"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 01:10:54.892178 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:10:54.892124 4939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ee090f53-4805-4a1a-a79f-450b8bc77079-istio-token" pod "ee090f53-4805-4a1a-a79f-450b8bc77079" (UID: "ee090f53-4805-4a1a-a79f-450b8bc77079"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 01:10:54.892247 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:10:54.892191 4939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9c96c382-a625-4b0c-b72f-82807daf6f3e-istio-token" pod "9c96c382-a625-4b0c-b72f-82807daf6f3e" (UID: "9c96c382-a625-4b0c-b72f-82807daf6f3e"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 01:10:54.892247 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:10:54.892202 4939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9c96c382-a625-4b0c-b72f-82807daf6f3e-credential-socket" pod "9c96c382-a625-4b0c-b72f-82807daf6f3e" (UID: "9c96c382-a625-4b0c-b72f-82807daf6f3e"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 01:10:54.892445 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:10:54.892410 4939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9c96c382-a625-4b0c-b72f-82807daf6f3e-workload-certs" pod "9c96c382-a625-4b0c-b72f-82807daf6f3e" (UID: "9c96c382-a625-4b0c-b72f-82807daf6f3e"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 01:10:54.893257 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:10:54.893080 4939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ee090f53-4805-4a1a-a79f-450b8bc77079-workload-certs" pod "ee090f53-4805-4a1a-a79f-450b8bc77079" (UID: "ee090f53-4805-4a1a-a79f-450b8bc77079"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 01:10:54.893364 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:10:54.893263 4939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f4f0fa4a-d49c-46d2-a1af-9195565e8bc2-istio-envoy" pod "f4f0fa4a-d49c-46d2-a1af-9195565e8bc2" (UID: "f4f0fa4a-d49c-46d2-a1af-9195565e8bc2"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 01:10:54.893517 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:10:54.893488 4939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9c96c382-a625-4b0c-b72f-82807daf6f3e-istio-data" pod "9c96c382-a625-4b0c-b72f-82807daf6f3e" (UID: "9c96c382-a625-4b0c-b72f-82807daf6f3e"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 01:10:54.894749 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:10:54.894727 4939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ee090f53-4805-4a1a-a79f-450b8bc77079-istio-envoy" pod "ee090f53-4805-4a1a-a79f-450b8bc77079" (UID: "ee090f53-4805-4a1a-a79f-450b8bc77079"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 01:10:54.894859 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:10:54.894755 4939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ee090f53-4805-4a1a-a79f-450b8bc77079-kube-api-access-jkrsh" pod "ee090f53-4805-4a1a-a79f-450b8bc77079" (UID: "ee090f53-4805-4a1a-a79f-450b8bc77079"). InnerVolumeSpecName "kube-api-access-jkrsh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 01:10:54.894963 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:10:54.894859 4939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/ee090f53-4805-4a1a-a79f-450b8bc77079-istio-podinfo" pod "ee090f53-4805-4a1a-a79f-450b8bc77079" (UID: "ee090f53-4805-4a1a-a79f-450b8bc77079"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 01:10:54.894963 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:10:54.894947 4939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9c96c382-a625-4b0c-b72f-82807daf6f3e-istio-envoy" pod "9c96c382-a625-4b0c-b72f-82807daf6f3e" (UID: "9c96c382-a625-4b0c-b72f-82807daf6f3e"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 01:10:54.895066 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:10:54.895049 4939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/f4f0fa4a-d49c-46d2-a1af-9195565e8bc2-istio-podinfo" pod "f4f0fa4a-d49c-46d2-a1af-9195565e8bc2" (UID: "f4f0fa4a-d49c-46d2-a1af-9195565e8bc2"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 01:10:54.895225 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:10:54.895208 4939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f4f0fa4a-d49c-46d2-a1af-9195565e8bc2-kube-api-access-str2z" pod "f4f0fa4a-d49c-46d2-a1af-9195565e8bc2" (UID: "f4f0fa4a-d49c-46d2-a1af-9195565e8bc2"). InnerVolumeSpecName "kube-api-access-str2z". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 01:10:54.896720 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:10:54.896693 4939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f4f0fa4a-d49c-46d2-a1af-9195565e8bc2-istio-token" pod "f4f0fa4a-d49c-46d2-a1af-9195565e8bc2" (UID: "f4f0fa4a-d49c-46d2-a1af-9195565e8bc2"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 01:10:54.964079 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:10:54.964038 4939 generic.go:356] "Generic (PLEG): container finished" podID="efde608b-c334-4d17-a0aa-cad98a95a999" containerID="be600a011fd4e99c6ae1e3f5111b289f859d31f27a7639165b30f43a4bbe3776" exitCode=0 Nov 03 01:10:54.964321 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:10:54.964187 4939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-af8244-openshift-default-594d5d565c-4f5nx" event={"ID":"efde608b-c334-4d17-a0aa-cad98a95a999","Type":"ContainerDied","Data":"be600a011fd4e99c6ae1e3f5111b289f859d31f27a7639165b30f43a4bbe3776"} Nov 03 01:10:54.964959 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:10:54.964925 4939 scope.go:122] "RemoveContainer" containerID="be600a011fd4e99c6ae1e3f5111b289f859d31f27a7639165b30f43a4bbe3776" Nov 03 01:10:54.966149 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:10:54.966124 4939 generic.go:356] "Generic (PLEG): container finished" podID="ee090f53-4805-4a1a-a79f-450b8bc77079" containerID="9ce5ab3f82c59a07fcab8e231933c24e8c586c6194d2db303bcb6d65033bf2f8" exitCode=0 Nov 03 01:10:54.966318 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:10:54.966224 4939 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-2f9a3a-openshift-default-7f6f74cf97-ckb7r" Nov 03 01:10:54.966318 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:10:54.966237 4939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-2f9a3a-openshift-default-7f6f74cf97-ckb7r" event={"ID":"ee090f53-4805-4a1a-a79f-450b8bc77079","Type":"ContainerDied","Data":"9ce5ab3f82c59a07fcab8e231933c24e8c586c6194d2db303bcb6d65033bf2f8"} Nov 03 01:10:54.966318 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:10:54.966289 4939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-2f9a3a-openshift-default-7f6f74cf97-ckb7r" event={"ID":"ee090f53-4805-4a1a-a79f-450b8bc77079","Type":"ContainerDied","Data":"1cd68f6d15017a3b45ce57f376a034757dede003cd515b06b803622d33645b88"} Nov 03 01:10:54.966495 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:10:54.966319 4939 scope.go:122] "RemoveContainer" containerID="9ce5ab3f82c59a07fcab8e231933c24e8c586c6194d2db303bcb6d65033bf2f8" Nov 03 01:10:54.967218 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:10:54.967177 4939 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 01:10:54.967329 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:10:54.967266 4939 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 01:10:54.967329 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:10:54.967309 4939 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 01:10:54.968165 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:10:54.968142 4939 generic.go:356] "Generic (PLEG): container finished" podID="9c96c382-a625-4b0c-b72f-82807daf6f3e" containerID="6e90b367fe9525a005eaec42d07c9414aa9906b5a68d215c39929f17aa671616" exitCode=0 Nov 03 01:10:54.968360 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:10:54.968199 4939 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-2f9a3a-openshift-default-7f6f74cf97-5rvtq" Nov 03 01:10:54.968575 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:10:54.968191 4939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-2f9a3a-openshift-default-7f6f74cf97-5rvtq" event={"ID":"9c96c382-a625-4b0c-b72f-82807daf6f3e","Type":"ContainerDied","Data":"6e90b367fe9525a005eaec42d07c9414aa9906b5a68d215c39929f17aa671616"} Nov 03 01:10:54.968674 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:10:54.968585 4939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-2f9a3a-openshift-default-7f6f74cf97-5rvtq" event={"ID":"9c96c382-a625-4b0c-b72f-82807daf6f3e","Type":"ContainerDied","Data":"aa85a535f359802bfb247435ed55eefe1b43e955a5ea01fa0d4230ece3f0ed92"} Nov 03 01:10:54.996093 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:10:54.995857 4939 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9c96c382-a625-4b0c-b72f-82807daf6f3e-istio-data\") on node \"ip-10-0-140-107.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:10:54.996093 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:10:54.995884 4939 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f4f0fa4a-d49c-46d2-a1af-9195565e8bc2-istio-data\") on node \"ip-10-0-140-107.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:10:54.996093 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:10:54.995901 4939 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9c96c382-a625-4b0c-b72f-82807daf6f3e-workload-socket\") on node \"ip-10-0-140-107.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:10:54.996093 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:10:54.995953 4939 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f4f0fa4a-d49c-46d2-a1af-9195565e8bc2-workload-socket\") on node \"ip-10-0-140-107.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:10:54.996093 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:10:54.995967 4939 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ee090f53-4805-4a1a-a79f-450b8bc77079-credential-socket\") on node \"ip-10-0-140-107.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:10:54.996093 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:10:54.995982 4939 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f4f0fa4a-d49c-46d2-a1af-9195565e8bc2-istio-podinfo\") on node \"ip-10-0-140-107.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:10:54.996093 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:10:54.995997 4939 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jkrsh\" (UniqueName: \"kubernetes.io/projected/ee090f53-4805-4a1a-a79f-450b8bc77079-kube-api-access-jkrsh\") on node \"ip-10-0-140-107.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:10:54.996093 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:10:54.996014 4939 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9c96c382-a625-4b0c-b72f-82807daf6f3e-workload-certs\") on node \"ip-10-0-140-107.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:10:54.996093 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:10:54.996028 4939 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f4f0fa4a-d49c-46d2-a1af-9195565e8bc2-istio-envoy\") on node \"ip-10-0-140-107.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:10:54.996093 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:10:54.996042 4939 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ee090f53-4805-4a1a-a79f-450b8bc77079-istio-token\") on node \"ip-10-0-140-107.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:10:54.996093 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:10:54.996055 4939 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ee090f53-4805-4a1a-a79f-450b8bc77079-istio-data\") on node \"ip-10-0-140-107.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:10:54.996093 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:10:54.996068 4939 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ee090f53-4805-4a1a-a79f-450b8bc77079-istio-podinfo\") on node \"ip-10-0-140-107.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:10:54.996093 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:10:54.996080 4939 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9c96c382-a625-4b0c-b72f-82807daf6f3e-istio-token\") on node \"ip-10-0-140-107.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:10:54.996093 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:10:54.996093 4939 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ee090f53-4805-4a1a-a79f-450b8bc77079-istio-envoy\") on node \"ip-10-0-140-107.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:10:54.996899 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:10:54.996107 4939 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f4f0fa4a-d49c-46d2-a1af-9195565e8bc2-istio-token\") on node \"ip-10-0-140-107.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:10:54.996899 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:10:54.996121 4939 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9c96c382-a625-4b0c-b72f-82807daf6f3e-credential-socket\") on node \"ip-10-0-140-107.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:10:54.996899 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:10:54.996133 4939 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9g45v\" (UniqueName: \"kubernetes.io/projected/9c96c382-a625-4b0c-b72f-82807daf6f3e-kube-api-access-9g45v\") on node \"ip-10-0-140-107.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:10:54.996899 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:10:54.996146 4939 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9c96c382-a625-4b0c-b72f-82807daf6f3e-istio-podinfo\") on node \"ip-10-0-140-107.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:10:54.996899 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:10:54.996159 4939 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-str2z\" (UniqueName: \"kubernetes.io/projected/f4f0fa4a-d49c-46d2-a1af-9195565e8bc2-kube-api-access-str2z\") on node \"ip-10-0-140-107.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:10:54.996899 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:10:54.996173 4939 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ee090f53-4805-4a1a-a79f-450b8bc77079-workload-certs\") on node \"ip-10-0-140-107.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:10:54.996899 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:10:54.996186 4939 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9c96c382-a625-4b0c-b72f-82807daf6f3e-istio-envoy\") on node \"ip-10-0-140-107.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:10:54.998388 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:10:54.998321 4939 generic.go:356] "Generic (PLEG): container finished" podID="f4f0fa4a-d49c-46d2-a1af-9195565e8bc2" containerID="60f729127ba803a14855cf8bdd33d7fffb8a91e0a8513cc7d87e84487d7002c3" exitCode=0 Nov 03 01:10:54.998532 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:10:54.998429 4939 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-2f9a3a-openshift-default-7f6f74cf97-cv2rj" Nov 03 01:10:54.998532 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:10:54.998465 4939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-2f9a3a-openshift-default-7f6f74cf97-cv2rj" event={"ID":"f4f0fa4a-d49c-46d2-a1af-9195565e8bc2","Type":"ContainerDied","Data":"60f729127ba803a14855cf8bdd33d7fffb8a91e0a8513cc7d87e84487d7002c3"} Nov 03 01:10:54.998532 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:10:54.998514 4939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-2f9a3a-openshift-default-7f6f74cf97-cv2rj" event={"ID":"f4f0fa4a-d49c-46d2-a1af-9195565e8bc2","Type":"ContainerDied","Data":"68f4f7208255e89da8580497f5f008e3eb5bbf5ad16fd210b6b7d1d0b7e8374b"} Nov 03 01:10:55.011087 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:10:55.011061 4939 scope.go:122] "RemoveContainer" containerID="9ce5ab3f82c59a07fcab8e231933c24e8c586c6194d2db303bcb6d65033bf2f8" Nov 03 01:10:55.011514 ip-10-0-140-107 kubenswrapper[4939]: E1103 01:10:55.011433 4939 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9ce5ab3f82c59a07fcab8e231933c24e8c586c6194d2db303bcb6d65033bf2f8\": container with ID starting with 9ce5ab3f82c59a07fcab8e231933c24e8c586c6194d2db303bcb6d65033bf2f8 not found: ID does not exist" containerID="9ce5ab3f82c59a07fcab8e231933c24e8c586c6194d2db303bcb6d65033bf2f8" Nov 03 01:10:55.011514 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:10:55.011475 4939 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9ce5ab3f82c59a07fcab8e231933c24e8c586c6194d2db303bcb6d65033bf2f8"} err="failed to get container status \"9ce5ab3f82c59a07fcab8e231933c24e8c586c6194d2db303bcb6d65033bf2f8\": rpc error: code = NotFound desc = could not find container \"9ce5ab3f82c59a07fcab8e231933c24e8c586c6194d2db303bcb6d65033bf2f8\": container with ID starting with 9ce5ab3f82c59a07fcab8e231933c24e8c586c6194d2db303bcb6d65033bf2f8 not found: ID does not exist" Nov 03 01:10:55.011514 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:10:55.011500 4939 scope.go:122] "RemoveContainer" containerID="6e90b367fe9525a005eaec42d07c9414aa9906b5a68d215c39929f17aa671616" Nov 03 01:10:55.035457 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:10:55.035102 4939 scope.go:122] "RemoveContainer" containerID="6e90b367fe9525a005eaec42d07c9414aa9906b5a68d215c39929f17aa671616" Nov 03 01:10:55.036701 ip-10-0-140-107 kubenswrapper[4939]: E1103 01:10:55.035698 4939 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6e90b367fe9525a005eaec42d07c9414aa9906b5a68d215c39929f17aa671616\": container with ID starting with 6e90b367fe9525a005eaec42d07c9414aa9906b5a68d215c39929f17aa671616 not found: ID does not exist" containerID="6e90b367fe9525a005eaec42d07c9414aa9906b5a68d215c39929f17aa671616" Nov 03 01:10:55.036701 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:10:55.035730 4939 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6e90b367fe9525a005eaec42d07c9414aa9906b5a68d215c39929f17aa671616"} err="failed to get container status \"6e90b367fe9525a005eaec42d07c9414aa9906b5a68d215c39929f17aa671616\": rpc error: code = NotFound desc = could not find container \"6e90b367fe9525a005eaec42d07c9414aa9906b5a68d215c39929f17aa671616\": container with ID starting with 6e90b367fe9525a005eaec42d07c9414aa9906b5a68d215c39929f17aa671616 not found: ID does not exist" Nov 03 01:10:55.036701 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:10:55.035750 4939 scope.go:122] "RemoveContainer" containerID="60f729127ba803a14855cf8bdd33d7fffb8a91e0a8513cc7d87e84487d7002c3" Nov 03 01:10:55.041558 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:10:55.041517 4939 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-2f9a3a-openshift-default-7f6f74cf97-ckb7r"] Nov 03 01:10:55.063781 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:10:55.063749 4939 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-2f9a3a-openshift-default-7f6f74cf97-ckb7r"] Nov 03 01:10:55.088785 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:10:55.088356 4939 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-2f9a3a-openshift-default-7f6f74cf97-5rvtq"] Nov 03 01:10:55.100271 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:10:55.100248 4939 scope.go:122] "RemoveContainer" containerID="60f729127ba803a14855cf8bdd33d7fffb8a91e0a8513cc7d87e84487d7002c3" Nov 03 01:10:55.100833 ip-10-0-140-107 kubenswrapper[4939]: E1103 01:10:55.100649 4939 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"60f729127ba803a14855cf8bdd33d7fffb8a91e0a8513cc7d87e84487d7002c3\": container with ID starting with 60f729127ba803a14855cf8bdd33d7fffb8a91e0a8513cc7d87e84487d7002c3 not found: ID does not exist" containerID="60f729127ba803a14855cf8bdd33d7fffb8a91e0a8513cc7d87e84487d7002c3" Nov 03 01:10:55.100833 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:10:55.100694 4939 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"60f729127ba803a14855cf8bdd33d7fffb8a91e0a8513cc7d87e84487d7002c3"} err="failed to get container status \"60f729127ba803a14855cf8bdd33d7fffb8a91e0a8513cc7d87e84487d7002c3\": rpc error: code = NotFound desc = could not find container \"60f729127ba803a14855cf8bdd33d7fffb8a91e0a8513cc7d87e84487d7002c3\": container with ID starting with 60f729127ba803a14855cf8bdd33d7fffb8a91e0a8513cc7d87e84487d7002c3 not found: ID does not exist" Nov 03 01:10:55.101478 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:10:55.101452 4939 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-2f9a3a-openshift-default-7f6f74cf97-5rvtq"] Nov 03 01:10:55.114128 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:10:55.114103 4939 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-2f9a3a-openshift-default-7f6f74cf97-cv2rj"] Nov 03 01:10:55.119010 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:10:55.118991 4939 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-2f9a3a-openshift-default-7f6f74cf97-cv2rj"] Nov 03 01:10:55.311259 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:10:55.311180 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:10:55.832967 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:10:55.832927 4939 scope.go:122] "RemoveContainer" containerID="7ca713fcdcb798bc67f734a39de4adaec4a34d450df68e0935562db4526d44f3" Nov 03 01:10:55.888829 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:10:55.888802 4939 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9c96c382-a625-4b0c-b72f-82807daf6f3e" path="/var/lib/kubelet/pods/9c96c382-a625-4b0c-b72f-82807daf6f3e/volumes" Nov 03 01:10:55.889595 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:10:55.889571 4939 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ee090f53-4805-4a1a-a79f-450b8bc77079" path="/var/lib/kubelet/pods/ee090f53-4805-4a1a-a79f-450b8bc77079/volumes" Nov 03 01:10:55.890146 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:10:55.890129 4939 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f4f0fa4a-d49c-46d2-a1af-9195565e8bc2" path="/var/lib/kubelet/pods/f4f0fa4a-d49c-46d2-a1af-9195565e8bc2/volumes" Nov 03 01:10:56.009993 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:10:56.009956 4939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-af8244-openshift-default-594d5d565c-4f5nx" event={"ID":"efde608b-c334-4d17-a0aa-cad98a95a999","Type":"ContainerStarted","Data":"e0519bcdbf5cb34499c8b3aff5536f3c55998f633bc17fcd25b7cb0f97269e8c"} Nov 03 01:10:56.018944 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:10:56.018919 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:10:56.673215 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:10:56.673178 4939 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-extmod-b-af8244-openshift-default-594d5d565c-4f5nx" Nov 03 01:10:56.674010 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:10:56.673974 4939 patch_prober.go:29] interesting pod/e2e-extmod-b-af8244-openshift-default-594d5d565c-4f5nx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.204:15021/healthz/ready\": dial tcp 10.132.0.204:15021: connect: connection refused" start-of-body= Nov 03 01:10:56.674122 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:10:56.674061 4939 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-af8244-openshift-default-594d5d565c-4f5nx" podUID="efde608b-c334-4d17-a0aa-cad98a95a999" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.204:15021/healthz/ready\": dial tcp 10.132.0.204:15021: connect: connection refused" Nov 03 01:10:56.845030 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:10:56.840375 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:10:57.389660 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:10:57.389626 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:10:57.624850 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:10:57.624798 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:10:57.673993 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:10:57.673434 4939 patch_prober.go:29] interesting pod/e2e-extmod-b-af8244-openshift-default-594d5d565c-4f5nx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.204:15021/healthz/ready\": dial tcp 10.132.0.204:15021: connect: connection refused" start-of-body= Nov 03 01:10:57.673993 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:10:57.673513 4939 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-af8244-openshift-default-594d5d565c-4f5nx" podUID="efde608b-c334-4d17-a0aa-cad98a95a999" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.204:15021/healthz/ready\": dial tcp 10.132.0.204:15021: connect: connection refused" Nov 03 01:10:58.328021 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:10:58.327985 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:10:58.669344 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:10:58.669304 4939 patch_prober.go:29] interesting pod/e2e-extmod-b-af8244-openshift-default-594d5d565c-4f5nx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.204:15021/healthz/ready\": dial tcp 10.132.0.204:15021: connect: connection refused" start-of-body= Nov 03 01:10:58.669626 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:10:58.669594 4939 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-af8244-openshift-default-594d5d565c-4f5nx" podUID="efde608b-c334-4d17-a0aa-cad98a95a999" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.204:15021/healthz/ready\": dial tcp 10.132.0.204:15021: connect: connection refused" Nov 03 01:10:58.676474 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:10:58.676416 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:10:59.043011 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:10:59.042922 4939 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-extmod-b-3c5983-openshift-default-86c5bc855c-2hx8m"] Nov 03 01:10:59.044344 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:10:59.044316 4939 cpu_manager.go:414] "containerMap: removing container" podUID="24fb3670-ec70-492c-a573-4cbe174f1ccf" containerName="main" Nov 03 01:10:59.044344 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:10:59.044343 4939 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="24fb3670-ec70-492c-a573-4cbe174f1ccf" containerName="main" Nov 03 01:10:59.044556 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:10:59.044378 4939 cpu_manager.go:414] "containerMap: removing container" podUID="f4f0fa4a-d49c-46d2-a1af-9195565e8bc2" containerName="istio-proxy" Nov 03 01:10:59.044556 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:10:59.044388 4939 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f4f0fa4a-d49c-46d2-a1af-9195565e8bc2" containerName="istio-proxy" Nov 03 01:10:59.044556 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:10:59.044406 4939 cpu_manager.go:414] "containerMap: removing container" podUID="9146740f-e0d2-46cb-a063-b5942f02b275" containerName="istio-proxy" Nov 03 01:10:59.044556 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:10:59.044454 4939 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9146740f-e0d2-46cb-a063-b5942f02b275" containerName="istio-proxy" Nov 03 01:10:59.044556 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:10:59.044473 4939 cpu_manager.go:414] "containerMap: removing container" podUID="9c96c382-a625-4b0c-b72f-82807daf6f3e" containerName="istio-proxy" Nov 03 01:10:59.044556 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:10:59.044481 4939 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9c96c382-a625-4b0c-b72f-82807daf6f3e" containerName="istio-proxy" Nov 03 01:10:59.044556 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:10:59.044501 4939 cpu_manager.go:414] "containerMap: removing container" podUID="ee090f53-4805-4a1a-a79f-450b8bc77079" containerName="istio-proxy" Nov 03 01:10:59.044556 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:10:59.044509 4939 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ee090f53-4805-4a1a-a79f-450b8bc77079" containerName="istio-proxy" Nov 03 01:10:59.045005 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:10:59.044792 4939 memory_manager.go:365] "RemoveStaleState removing state" podUID="24fb3670-ec70-492c-a573-4cbe174f1ccf" containerName="main" Nov 03 01:10:59.045005 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:10:59.044819 4939 memory_manager.go:365] "RemoveStaleState removing state" podUID="f4f0fa4a-d49c-46d2-a1af-9195565e8bc2" containerName="istio-proxy" Nov 03 01:10:59.045005 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:10:59.044833 4939 memory_manager.go:365] "RemoveStaleState removing state" podUID="9146740f-e0d2-46cb-a063-b5942f02b275" containerName="istio-proxy" Nov 03 01:10:59.045005 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:10:59.044844 4939 memory_manager.go:365] "RemoveStaleState removing state" podUID="9c96c382-a625-4b0c-b72f-82807daf6f3e" containerName="istio-proxy" Nov 03 01:10:59.045005 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:10:59.044854 4939 memory_manager.go:365] "RemoveStaleState removing state" podUID="ee090f53-4805-4a1a-a79f-450b8bc77079" containerName="istio-proxy" Nov 03 01:10:59.084246 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:10:59.084208 4939 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-3c5983-openshift-default-86c5bc855c-2hx8m" Nov 03 01:10:59.091064 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:10:59.091032 4939 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-extmod-b-3c5983-openshift-default-dockercfg-jz9jw\"" Nov 03 01:10:59.101590 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:10:59.101552 4939 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-b-3c5983-openshift-default-86c5bc855c-2hx8m"] Nov 03 01:10:59.115349 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:10:59.115307 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:10:59.193930 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:10:59.170392 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/78a38f10-6762-489e-813a-ac1cb91f5844-istio-envoy\") pod \"e2e-extmod-b-3c5983-openshift-default-86c5bc855c-2hx8m\" (UID: \"78a38f10-6762-489e-813a-ac1cb91f5844\") " pod="openshift-ingress/e2e-extmod-b-3c5983-openshift-default-86c5bc855c-2hx8m" Nov 03 01:10:59.193930 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:10:59.170447 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/78a38f10-6762-489e-813a-ac1cb91f5844-istiod-ca-cert\") pod \"e2e-extmod-b-3c5983-openshift-default-86c5bc855c-2hx8m\" (UID: \"78a38f10-6762-489e-813a-ac1cb91f5844\") " pod="openshift-ingress/e2e-extmod-b-3c5983-openshift-default-86c5bc855c-2hx8m" Nov 03 01:10:59.193930 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:10:59.170672 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/78a38f10-6762-489e-813a-ac1cb91f5844-workload-socket\") pod \"e2e-extmod-b-3c5983-openshift-default-86c5bc855c-2hx8m\" (UID: \"78a38f10-6762-489e-813a-ac1cb91f5844\") " pod="openshift-ingress/e2e-extmod-b-3c5983-openshift-default-86c5bc855c-2hx8m" Nov 03 01:10:59.193930 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:10:59.170709 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zmx8s\" (UniqueName: \"kubernetes.io/projected/78a38f10-6762-489e-813a-ac1cb91f5844-kube-api-access-zmx8s\") pod \"e2e-extmod-b-3c5983-openshift-default-86c5bc855c-2hx8m\" (UID: \"78a38f10-6762-489e-813a-ac1cb91f5844\") " pod="openshift-ingress/e2e-extmod-b-3c5983-openshift-default-86c5bc855c-2hx8m" Nov 03 01:10:59.193930 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:10:59.170841 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/78a38f10-6762-489e-813a-ac1cb91f5844-istio-podinfo\") pod \"e2e-extmod-b-3c5983-openshift-default-86c5bc855c-2hx8m\" (UID: \"78a38f10-6762-489e-813a-ac1cb91f5844\") " pod="openshift-ingress/e2e-extmod-b-3c5983-openshift-default-86c5bc855c-2hx8m" Nov 03 01:10:59.193930 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:10:59.170884 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/78a38f10-6762-489e-813a-ac1cb91f5844-istio-token\") pod \"e2e-extmod-b-3c5983-openshift-default-86c5bc855c-2hx8m\" (UID: \"78a38f10-6762-489e-813a-ac1cb91f5844\") " pod="openshift-ingress/e2e-extmod-b-3c5983-openshift-default-86c5bc855c-2hx8m" Nov 03 01:10:59.193930 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:10:59.170974 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/78a38f10-6762-489e-813a-ac1cb91f5844-credential-socket\") pod \"e2e-extmod-b-3c5983-openshift-default-86c5bc855c-2hx8m\" (UID: \"78a38f10-6762-489e-813a-ac1cb91f5844\") " pod="openshift-ingress/e2e-extmod-b-3c5983-openshift-default-86c5bc855c-2hx8m" Nov 03 01:10:59.193930 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:10:59.171094 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/78a38f10-6762-489e-813a-ac1cb91f5844-workload-certs\") pod \"e2e-extmod-b-3c5983-openshift-default-86c5bc855c-2hx8m\" (UID: \"78a38f10-6762-489e-813a-ac1cb91f5844\") " pod="openshift-ingress/e2e-extmod-b-3c5983-openshift-default-86c5bc855c-2hx8m" Nov 03 01:10:59.193930 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:10:59.171142 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/78a38f10-6762-489e-813a-ac1cb91f5844-istio-data\") pod \"e2e-extmod-b-3c5983-openshift-default-86c5bc855c-2hx8m\" (UID: \"78a38f10-6762-489e-813a-ac1cb91f5844\") " pod="openshift-ingress/e2e-extmod-b-3c5983-openshift-default-86c5bc855c-2hx8m" Nov 03 01:10:59.287772 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:10:59.287734 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/78a38f10-6762-489e-813a-ac1cb91f5844-istio-data\") pod \"e2e-extmod-b-3c5983-openshift-default-86c5bc855c-2hx8m\" (UID: \"78a38f10-6762-489e-813a-ac1cb91f5844\") " pod="openshift-ingress/e2e-extmod-b-3c5983-openshift-default-86c5bc855c-2hx8m" Nov 03 01:10:59.314056 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:10:59.305964 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/78a38f10-6762-489e-813a-ac1cb91f5844-istio-envoy\") pod \"e2e-extmod-b-3c5983-openshift-default-86c5bc855c-2hx8m\" (UID: \"78a38f10-6762-489e-813a-ac1cb91f5844\") " pod="openshift-ingress/e2e-extmod-b-3c5983-openshift-default-86c5bc855c-2hx8m" Nov 03 01:10:59.330050 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:10:59.330013 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/78a38f10-6762-489e-813a-ac1cb91f5844-istiod-ca-cert\") pod \"e2e-extmod-b-3c5983-openshift-default-86c5bc855c-2hx8m\" (UID: \"78a38f10-6762-489e-813a-ac1cb91f5844\") " pod="openshift-ingress/e2e-extmod-b-3c5983-openshift-default-86c5bc855c-2hx8m" Nov 03 01:10:59.330892 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:10:59.330860 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/78a38f10-6762-489e-813a-ac1cb91f5844-workload-socket\") pod \"e2e-extmod-b-3c5983-openshift-default-86c5bc855c-2hx8m\" (UID: \"78a38f10-6762-489e-813a-ac1cb91f5844\") " pod="openshift-ingress/e2e-extmod-b-3c5983-openshift-default-86c5bc855c-2hx8m" Nov 03 01:10:59.331132 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:10:59.331108 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zmx8s\" (UniqueName: \"kubernetes.io/projected/78a38f10-6762-489e-813a-ac1cb91f5844-kube-api-access-zmx8s\") pod \"e2e-extmod-b-3c5983-openshift-default-86c5bc855c-2hx8m\" (UID: \"78a38f10-6762-489e-813a-ac1cb91f5844\") " pod="openshift-ingress/e2e-extmod-b-3c5983-openshift-default-86c5bc855c-2hx8m" Nov 03 01:10:59.331426 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:10:59.331401 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/78a38f10-6762-489e-813a-ac1cb91f5844-istio-podinfo\") pod \"e2e-extmod-b-3c5983-openshift-default-86c5bc855c-2hx8m\" (UID: \"78a38f10-6762-489e-813a-ac1cb91f5844\") " pod="openshift-ingress/e2e-extmod-b-3c5983-openshift-default-86c5bc855c-2hx8m" Nov 03 01:10:59.331599 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:10:59.331582 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/78a38f10-6762-489e-813a-ac1cb91f5844-istio-token\") pod \"e2e-extmod-b-3c5983-openshift-default-86c5bc855c-2hx8m\" (UID: \"78a38f10-6762-489e-813a-ac1cb91f5844\") " pod="openshift-ingress/e2e-extmod-b-3c5983-openshift-default-86c5bc855c-2hx8m" Nov 03 01:10:59.331787 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:10:59.331770 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/78a38f10-6762-489e-813a-ac1cb91f5844-credential-socket\") pod \"e2e-extmod-b-3c5983-openshift-default-86c5bc855c-2hx8m\" (UID: \"78a38f10-6762-489e-813a-ac1cb91f5844\") " pod="openshift-ingress/e2e-extmod-b-3c5983-openshift-default-86c5bc855c-2hx8m" Nov 03 01:10:59.332063 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:10:59.332043 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/78a38f10-6762-489e-813a-ac1cb91f5844-workload-certs\") pod \"e2e-extmod-b-3c5983-openshift-default-86c5bc855c-2hx8m\" (UID: \"78a38f10-6762-489e-813a-ac1cb91f5844\") " pod="openshift-ingress/e2e-extmod-b-3c5983-openshift-default-86c5bc855c-2hx8m" Nov 03 01:10:59.348879 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:10:59.338291 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/78a38f10-6762-489e-813a-ac1cb91f5844-workload-certs\") pod \"e2e-extmod-b-3c5983-openshift-default-86c5bc855c-2hx8m\" (UID: \"78a38f10-6762-489e-813a-ac1cb91f5844\") " pod="openshift-ingress/e2e-extmod-b-3c5983-openshift-default-86c5bc855c-2hx8m" Nov 03 01:10:59.349135 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:10:59.296392 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:10:59.349316 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:10:59.338943 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/78a38f10-6762-489e-813a-ac1cb91f5844-istiod-ca-cert\") pod \"e2e-extmod-b-3c5983-openshift-default-86c5bc855c-2hx8m\" (UID: \"78a38f10-6762-489e-813a-ac1cb91f5844\") " pod="openshift-ingress/e2e-extmod-b-3c5983-openshift-default-86c5bc855c-2hx8m" Nov 03 01:10:59.349430 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:10:59.289700 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/78a38f10-6762-489e-813a-ac1cb91f5844-istio-data\") pod \"e2e-extmod-b-3c5983-openshift-default-86c5bc855c-2hx8m\" (UID: \"78a38f10-6762-489e-813a-ac1cb91f5844\") " pod="openshift-ingress/e2e-extmod-b-3c5983-openshift-default-86c5bc855c-2hx8m" Nov 03 01:10:59.349522 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:10:59.339181 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/78a38f10-6762-489e-813a-ac1cb91f5844-workload-socket\") pod \"e2e-extmod-b-3c5983-openshift-default-86c5bc855c-2hx8m\" (UID: \"78a38f10-6762-489e-813a-ac1cb91f5844\") " pod="openshift-ingress/e2e-extmod-b-3c5983-openshift-default-86c5bc855c-2hx8m" Nov 03 01:10:59.349615 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:10:59.344631 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/78a38f10-6762-489e-813a-ac1cb91f5844-credential-socket\") pod \"e2e-extmod-b-3c5983-openshift-default-86c5bc855c-2hx8m\" (UID: \"78a38f10-6762-489e-813a-ac1cb91f5844\") " pod="openshift-ingress/e2e-extmod-b-3c5983-openshift-default-86c5bc855c-2hx8m" Nov 03 01:10:59.384931 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:10:59.380755 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/78a38f10-6762-489e-813a-ac1cb91f5844-istio-envoy\") pod \"e2e-extmod-b-3c5983-openshift-default-86c5bc855c-2hx8m\" (UID: \"78a38f10-6762-489e-813a-ac1cb91f5844\") " pod="openshift-ingress/e2e-extmod-b-3c5983-openshift-default-86c5bc855c-2hx8m" Nov 03 01:10:59.411934 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:10:59.406364 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zmx8s\" (UniqueName: \"kubernetes.io/projected/78a38f10-6762-489e-813a-ac1cb91f5844-kube-api-access-zmx8s\") pod \"e2e-extmod-b-3c5983-openshift-default-86c5bc855c-2hx8m\" (UID: \"78a38f10-6762-489e-813a-ac1cb91f5844\") " pod="openshift-ingress/e2e-extmod-b-3c5983-openshift-default-86c5bc855c-2hx8m" Nov 03 01:10:59.411934 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:10:59.407958 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/78a38f10-6762-489e-813a-ac1cb91f5844-istio-token\") pod \"e2e-extmod-b-3c5983-openshift-default-86c5bc855c-2hx8m\" (UID: \"78a38f10-6762-489e-813a-ac1cb91f5844\") " pod="openshift-ingress/e2e-extmod-b-3c5983-openshift-default-86c5bc855c-2hx8m" Nov 03 01:10:59.421954 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:10:59.420844 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/78a38f10-6762-489e-813a-ac1cb91f5844-istio-podinfo\") pod \"e2e-extmod-b-3c5983-openshift-default-86c5bc855c-2hx8m\" (UID: \"78a38f10-6762-489e-813a-ac1cb91f5844\") " pod="openshift-ingress/e2e-extmod-b-3c5983-openshift-default-86c5bc855c-2hx8m" Nov 03 01:10:59.499446 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:10:59.499357 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:10:59.689935 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:10:59.677478 4939 patch_prober.go:29] interesting pod/e2e-extmod-b-af8244-openshift-default-594d5d565c-4f5nx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.204:15021/healthz/ready\": dial tcp 10.132.0.204:15021: connect: connection refused" start-of-body= Nov 03 01:10:59.689935 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:10:59.677542 4939 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-af8244-openshift-default-594d5d565c-4f5nx" podUID="efde608b-c334-4d17-a0aa-cad98a95a999" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.204:15021/healthz/ready\": dial tcp 10.132.0.204:15021: connect: connection refused" Nov 03 01:10:59.705294 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:10:59.705171 4939 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-3c5983-openshift-default-86c5bc855c-2hx8m" Nov 03 01:11:00.222437 ip-10-0-140-107 kubenswrapper[4939]: W1103 01:11:00.222387 4939 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod78a38f10_6762_489e_813a_ac1cb91f5844.slice/crio-1ffede5791fcbca7c79ae803e5dea9dedf6b5f5bf9989978b6f7fcdfdd1a95e7 WatchSource:0}: Error finding container 1ffede5791fcbca7c79ae803e5dea9dedf6b5f5bf9989978b6f7fcdfdd1a95e7: Status 404 returned error can't find the container with id 1ffede5791fcbca7c79ae803e5dea9dedf6b5f5bf9989978b6f7fcdfdd1a95e7 Nov 03 01:11:00.225666 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:11:00.225623 4939 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 01:11:00.225771 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:11:00.225702 4939 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 01:11:00.225771 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:11:00.225737 4939 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 01:11:00.227621 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:11:00.227592 4939 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-b-3c5983-openshift-default-86c5bc855c-2hx8m"] Nov 03 01:11:00.502005 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:11:00.501814 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:11:00.699931 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:11:00.694526 4939 patch_prober.go:29] interesting pod/e2e-extmod-b-af8244-openshift-default-594d5d565c-4f5nx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.204:15021/healthz/ready\": dial tcp 10.132.0.204:15021: connect: connection refused" start-of-body= Nov 03 01:11:00.699931 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:11:00.694601 4939 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-af8244-openshift-default-594d5d565c-4f5nx" podUID="efde608b-c334-4d17-a0aa-cad98a95a999" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.204:15021/healthz/ready\": dial tcp 10.132.0.204:15021: connect: connection refused" Nov 03 01:11:01.010973 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:11:01.010897 4939 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-b-af8244-openshift-default-594d5d565c-4f5nx" Nov 03 01:11:01.106995 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:11:01.106952 4939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-3c5983-openshift-default-86c5bc855c-2hx8m" event={"ID":"78a38f10-6762-489e-813a-ac1cb91f5844","Type":"ContainerStarted","Data":"2379ea793c99a3d43b7adbe002e932cb7452b8a0597b970f11c7f367a61f7b14"} Nov 03 01:11:01.107217 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:11:01.107020 4939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-3c5983-openshift-default-86c5bc855c-2hx8m" event={"ID":"78a38f10-6762-489e-813a-ac1cb91f5844","Type":"ContainerStarted","Data":"1ffede5791fcbca7c79ae803e5dea9dedf6b5f5bf9989978b6f7fcdfdd1a95e7"} Nov 03 01:11:01.128170 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:11:01.128118 4939 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-extmod-b-3c5983-openshift-default-86c5bc855c-2hx8m" podStartSLOduration=2.128104928 podStartE2EDuration="2.128104928s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 01:10:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 01:11:01.126120481 +0000 UTC m=+1809.390455429" watchObservedRunningTime="2026-11-03 01:11:01.128104928 +0000 UTC m=+1809.392439872" Nov 03 01:11:01.345051 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:11:01.332329 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:11:01.671829 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:11:01.671725 4939 patch_prober.go:29] interesting pod/e2e-extmod-b-af8244-openshift-default-594d5d565c-4f5nx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.204:15021/healthz/ready\": dial tcp 10.132.0.204:15021: connect: connection refused" start-of-body= Nov 03 01:11:01.671829 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:11:01.671795 4939 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-af8244-openshift-default-594d5d565c-4f5nx" podUID="efde608b-c334-4d17-a0aa-cad98a95a999" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.204:15021/healthz/ready\": dial tcp 10.132.0.204:15021: connect: connection refused" Nov 03 01:11:01.705943 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:11:01.705878 4939 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-extmod-b-3c5983-openshift-default-86c5bc855c-2hx8m" Nov 03 01:11:01.712045 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:11:01.712020 4939 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-extmod-b-3c5983-openshift-default-86c5bc855c-2hx8m" Nov 03 01:11:02.003800 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:11:02.003695 4939 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-f7680d-openshift-default-c4488d5c9-ldr7q"] Nov 03 01:11:02.004437 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:11:02.004005 4939 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-f7680d-openshift-default-c4488d5c9-ldr7q" podUID="9b5bb111-d57c-478b-b990-db75ac2df998" containerName="istio-proxy" containerID="cri-o://e5ff86a1fadda7447aaa59b5b3ece9e64d7f369acf0fe17c71c037b90c523ffd" gracePeriod=30 Nov 03 01:11:02.014735 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:11:02.014705 4939 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-f7680d-openshift-default-c4488d5c9-mlblk"] Nov 03 01:11:02.014973 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:11:02.014941 4939 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-f7680d-openshift-default-c4488d5c9-mlblk" podUID="e08d7d14-b1fa-4fb2-b2d5-8c7fc073fc89" containerName="istio-proxy" containerID="cri-o://8fd039107c781ac33f7786416fb1447c41ebda9a3ab07311a652f359dd592f23" gracePeriod=30 Nov 03 01:11:02.121970 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:11:02.117436 4939 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-b-3c5983-openshift-default-86c5bc855c-2hx8m" Nov 03 01:11:02.121970 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:11:02.119043 4939 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-extmod-b-3c5983-openshift-default-86c5bc855c-2hx8m" Nov 03 01:11:02.617343 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:11:02.617300 4939 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-83755a-openshift-default-c86f9565c-z7flk"] Nov 03 01:11:02.617667 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:11:02.617633 4939 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-83755a-openshift-default-c86f9565c-z7flk" podUID="569c2920-1523-4da8-8644-f789aebcdc7a" containerName="istio-proxy" containerID="cri-o://d19447d0abfd950faea5858874c8ad314badd753800f42c9173c681a1eea80d6" gracePeriod=30 Nov 03 01:11:02.626045 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:11:02.626016 4939 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-83755a-openshift-default-c86f9565c-zj5jg"] Nov 03 01:11:02.626519 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:11:02.626464 4939 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-83755a-openshift-default-c86f9565c-zj5jg" podUID="0dbecc93-5a61-4100-a319-d3f673f7a5fd" containerName="istio-proxy" containerID="cri-o://988c3b00cfe1d42a496c5073f93a1ea2f634e32400f4bb974be56cfdede7ffa0" gracePeriod=30 Nov 03 01:11:02.677622 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:11:02.677590 4939 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-extmod-b-af8244-openshift-default-594d5d565c-4f5nx" Nov 03 01:11:02.682759 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:11:02.681356 4939 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-extmod-b-af8244-openshift-default-594d5d565c-4f5nx" Nov 03 01:11:02.832584 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:11:02.832541 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:11:03.476990 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:11:03.476952 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:11:03.550141 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:11:03.550100 4939 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-noten-a23ed0/noten-model-a23ed0-kserve-5dcdf8b6fc-fgrrv" podUID="00fccd6d-ef84-4b47-9531-090a635045e9" containerName="main" containerID="cri-o://1f517a1b2cdd36419ee083f592ddffd3640659eb91a3a4aefb4232e5cc05a3b5" gracePeriod=45 Nov 03 01:11:03.731396 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:11:03.731311 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:11:03.857706 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:11:03.857664 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:11:03.910746 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:11:03.910716 4939 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-noten-a23ed0/noten-model-a23ed0-kserve-5dcdf8b6fc-fgrrv" Nov 03 01:11:03.934286 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:11:03.934253 4939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/00fccd6d-ef84-4b47-9531-090a635045e9-kube-api-access-4z6x9\" (UniqueName: \"kubernetes.io/projected/00fccd6d-ef84-4b47-9531-090a635045e9-kube-api-access-4z6x9\") pod \"00fccd6d-ef84-4b47-9531-090a635045e9\" (UID: \"00fccd6d-ef84-4b47-9531-090a635045e9\") " Nov 03 01:11:03.934444 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:11:03.934317 4939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/00fccd6d-ef84-4b47-9531-090a635045e9-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/00fccd6d-ef84-4b47-9531-090a635045e9-model-cache\") pod \"00fccd6d-ef84-4b47-9531-090a635045e9\" (UID: \"00fccd6d-ef84-4b47-9531-090a635045e9\") " Nov 03 01:11:03.934529 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:11:03.934477 4939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/00fccd6d-ef84-4b47-9531-090a635045e9-home\" (UniqueName: \"kubernetes.io/empty-dir/00fccd6d-ef84-4b47-9531-090a635045e9-home\") pod \"00fccd6d-ef84-4b47-9531-090a635045e9\" (UID: \"00fccd6d-ef84-4b47-9531-090a635045e9\") " Nov 03 01:11:03.934529 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:11:03.934504 4939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/00fccd6d-ef84-4b47-9531-090a635045e9-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/00fccd6d-ef84-4b47-9531-090a635045e9-tmp-dir\") pod \"00fccd6d-ef84-4b47-9531-090a635045e9\" (UID: \"00fccd6d-ef84-4b47-9531-090a635045e9\") " Nov 03 01:11:03.934638 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:11:03.934529 4939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/00fccd6d-ef84-4b47-9531-090a635045e9-model-cache" pod "00fccd6d-ef84-4b47-9531-090a635045e9" (UID: "00fccd6d-ef84-4b47-9531-090a635045e9"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 01:11:03.934807 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:11:03.934726 4939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/00fccd6d-ef84-4b47-9531-090a635045e9-home" pod "00fccd6d-ef84-4b47-9531-090a635045e9" (UID: "00fccd6d-ef84-4b47-9531-090a635045e9"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 01:11:03.934807 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:11:03.934744 4939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/00fccd6d-ef84-4b47-9531-090a635045e9-tls-certs\" (UniqueName: \"kubernetes.io/secret/00fccd6d-ef84-4b47-9531-090a635045e9-tls-certs\") pod \"00fccd6d-ef84-4b47-9531-090a635045e9\" (UID: \"00fccd6d-ef84-4b47-9531-090a635045e9\") " Nov 03 01:11:03.934807 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:11:03.934795 4939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/00fccd6d-ef84-4b47-9531-090a635045e9-tmp-dir" pod "00fccd6d-ef84-4b47-9531-090a635045e9" (UID: "00fccd6d-ef84-4b47-9531-090a635045e9"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 01:11:03.935015 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:11:03.934864 4939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/00fccd6d-ef84-4b47-9531-090a635045e9-dshm\" (UniqueName: \"kubernetes.io/empty-dir/00fccd6d-ef84-4b47-9531-090a635045e9-dshm\") pod \"00fccd6d-ef84-4b47-9531-090a635045e9\" (UID: \"00fccd6d-ef84-4b47-9531-090a635045e9\") " Nov 03 01:11:03.935748 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:11:03.935721 4939 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/00fccd6d-ef84-4b47-9531-090a635045e9-model-cache\") on node \"ip-10-0-140-107.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:11:03.935748 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:11:03.935748 4939 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/00fccd6d-ef84-4b47-9531-090a635045e9-home\") on node \"ip-10-0-140-107.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:11:03.936032 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:11:03.935760 4939 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/00fccd6d-ef84-4b47-9531-090a635045e9-tmp-dir\") on node \"ip-10-0-140-107.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:11:03.937296 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:11:03.937268 4939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/00fccd6d-ef84-4b47-9531-090a635045e9-dshm" pod "00fccd6d-ef84-4b47-9531-090a635045e9" (UID: "00fccd6d-ef84-4b47-9531-090a635045e9"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 01:11:03.937396 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:11:03.937300 4939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/00fccd6d-ef84-4b47-9531-090a635045e9-kube-api-access-4z6x9" pod "00fccd6d-ef84-4b47-9531-090a635045e9" (UID: "00fccd6d-ef84-4b47-9531-090a635045e9"). InnerVolumeSpecName "kube-api-access-4z6x9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 01:11:03.937396 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:11:03.937346 4939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/00fccd6d-ef84-4b47-9531-090a635045e9-tls-certs" pod "00fccd6d-ef84-4b47-9531-090a635045e9" (UID: "00fccd6d-ef84-4b47-9531-090a635045e9"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 01:11:04.038072 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:11:04.037991 4939 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/00fccd6d-ef84-4b47-9531-090a635045e9-tls-certs\") on node \"ip-10-0-140-107.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:11:04.038072 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:11:04.038023 4939 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/00fccd6d-ef84-4b47-9531-090a635045e9-dshm\") on node \"ip-10-0-140-107.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:11:04.038072 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:11:04.038033 4939 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4z6x9\" (UniqueName: \"kubernetes.io/projected/00fccd6d-ef84-4b47-9531-090a635045e9-kube-api-access-4z6x9\") on node \"ip-10-0-140-107.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:11:04.134767 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:11:04.134727 4939 generic.go:356] "Generic (PLEG): container finished" podID="00fccd6d-ef84-4b47-9531-090a635045e9" containerID="1f517a1b2cdd36419ee083f592ddffd3640659eb91a3a4aefb4232e5cc05a3b5" exitCode=0 Nov 03 01:11:04.134970 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:11:04.134828 4939 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-noten-a23ed0/noten-model-a23ed0-kserve-5dcdf8b6fc-fgrrv" Nov 03 01:11:04.134970 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:11:04.134846 4939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-noten-a23ed0/noten-model-a23ed0-kserve-5dcdf8b6fc-fgrrv" event={"ID":"00fccd6d-ef84-4b47-9531-090a635045e9","Type":"ContainerDied","Data":"1f517a1b2cdd36419ee083f592ddffd3640659eb91a3a4aefb4232e5cc05a3b5"} Nov 03 01:11:04.134970 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:11:04.134916 4939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-noten-a23ed0/noten-model-a23ed0-kserve-5dcdf8b6fc-fgrrv" event={"ID":"00fccd6d-ef84-4b47-9531-090a635045e9","Type":"ContainerDied","Data":"013b52a9accacafc49e3cf3c85cb334218db6cc5f869d22e59ab66d61a98d481"} Nov 03 01:11:04.134970 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:11:04.134938 4939 scope.go:122] "RemoveContainer" containerID="1f517a1b2cdd36419ee083f592ddffd3640659eb91a3a4aefb4232e5cc05a3b5" Nov 03 01:11:04.146959 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:11:04.146824 4939 scope.go:122] "RemoveContainer" containerID="1f517a1b2cdd36419ee083f592ddffd3640659eb91a3a4aefb4232e5cc05a3b5" Nov 03 01:11:04.147201 ip-10-0-140-107 kubenswrapper[4939]: E1103 01:11:04.147163 4939 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1f517a1b2cdd36419ee083f592ddffd3640659eb91a3a4aefb4232e5cc05a3b5\": container with ID starting with 1f517a1b2cdd36419ee083f592ddffd3640659eb91a3a4aefb4232e5cc05a3b5 not found: ID does not exist" containerID="1f517a1b2cdd36419ee083f592ddffd3640659eb91a3a4aefb4232e5cc05a3b5" Nov 03 01:11:04.147306 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:11:04.147210 4939 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1f517a1b2cdd36419ee083f592ddffd3640659eb91a3a4aefb4232e5cc05a3b5"} err="failed to get container status \"1f517a1b2cdd36419ee083f592ddffd3640659eb91a3a4aefb4232e5cc05a3b5\": rpc error: code = NotFound desc = could not find container \"1f517a1b2cdd36419ee083f592ddffd3640659eb91a3a4aefb4232e5cc05a3b5\": container with ID starting with 1f517a1b2cdd36419ee083f592ddffd3640659eb91a3a4aefb4232e5cc05a3b5 not found: ID does not exist" Nov 03 01:11:04.162716 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:11:04.162688 4939 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-noten-a23ed0/noten-model-a23ed0-kserve-5dcdf8b6fc-fgrrv"] Nov 03 01:11:04.167115 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:11:04.167090 4939 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-noten-a23ed0/noten-model-a23ed0-kserve-5dcdf8b6fc-fgrrv"] Nov 03 01:11:04.311109 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:11:04.311021 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:11:04.752953 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:11:04.752898 4939 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w3-556c14-openshift-default-754b5bc799-dt7h4"] Nov 03 01:11:04.754305 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:11:04.754281 4939 cpu_manager.go:414] "containerMap: removing container" podUID="00fccd6d-ef84-4b47-9531-090a635045e9" containerName="main" Nov 03 01:11:04.754409 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:11:04.754306 4939 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="00fccd6d-ef84-4b47-9531-090a635045e9" containerName="main" Nov 03 01:11:04.754562 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:11:04.754548 4939 memory_manager.go:365] "RemoveStaleState removing state" podUID="00fccd6d-ef84-4b47-9531-090a635045e9" containerName="main" Nov 03 01:11:04.757985 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:11:04.757965 4939 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w3-556c14-openshift-default-754b5bc799-dt7h4" Nov 03 01:11:04.771705 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:11:04.771676 4939 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-556c14-openshift-default-754b5bc799-dt7h4"] Nov 03 01:11:04.818212 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:11:04.818177 4939 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-w3-556c14-openshift-default-dockercfg-bpvll\"" Nov 03 01:11:04.856403 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:11:04.856315 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fdeb6f74-c52a-4497-be8e-516b4244aefe-istiod-ca-cert\") pod \"e2e-shared-b-w3-556c14-openshift-default-754b5bc799-dt7h4\" (UID: \"fdeb6f74-c52a-4497-be8e-516b4244aefe\") " pod="openshift-ingress/e2e-shared-b-w3-556c14-openshift-default-754b5bc799-dt7h4" Nov 03 01:11:04.856403 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:11:04.856380 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l9l2j\" (UniqueName: \"kubernetes.io/projected/fdeb6f74-c52a-4497-be8e-516b4244aefe-kube-api-access-l9l2j\") pod \"e2e-shared-b-w3-556c14-openshift-default-754b5bc799-dt7h4\" (UID: \"fdeb6f74-c52a-4497-be8e-516b4244aefe\") " pod="openshift-ingress/e2e-shared-b-w3-556c14-openshift-default-754b5bc799-dt7h4" Nov 03 01:11:04.857155 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:11:04.857128 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fdeb6f74-c52a-4497-be8e-516b4244aefe-workload-certs\") pod \"e2e-shared-b-w3-556c14-openshift-default-754b5bc799-dt7h4\" (UID: \"fdeb6f74-c52a-4497-be8e-516b4244aefe\") " pod="openshift-ingress/e2e-shared-b-w3-556c14-openshift-default-754b5bc799-dt7h4" Nov 03 01:11:04.857289 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:11:04.857244 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fdeb6f74-c52a-4497-be8e-516b4244aefe-istio-envoy\") pod \"e2e-shared-b-w3-556c14-openshift-default-754b5bc799-dt7h4\" (UID: \"fdeb6f74-c52a-4497-be8e-516b4244aefe\") " pod="openshift-ingress/e2e-shared-b-w3-556c14-openshift-default-754b5bc799-dt7h4" Nov 03 01:11:04.857289 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:11:04.857286 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fdeb6f74-c52a-4497-be8e-516b4244aefe-istio-token\") pod \"e2e-shared-b-w3-556c14-openshift-default-754b5bc799-dt7h4\" (UID: \"fdeb6f74-c52a-4497-be8e-516b4244aefe\") " pod="openshift-ingress/e2e-shared-b-w3-556c14-openshift-default-754b5bc799-dt7h4" Nov 03 01:11:04.857418 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:11:04.857385 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fdeb6f74-c52a-4497-be8e-516b4244aefe-credential-socket\") pod \"e2e-shared-b-w3-556c14-openshift-default-754b5bc799-dt7h4\" (UID: \"fdeb6f74-c52a-4497-be8e-516b4244aefe\") " pod="openshift-ingress/e2e-shared-b-w3-556c14-openshift-default-754b5bc799-dt7h4" Nov 03 01:11:04.857465 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:11:04.857445 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fdeb6f74-c52a-4497-be8e-516b4244aefe-istio-podinfo\") pod \"e2e-shared-b-w3-556c14-openshift-default-754b5bc799-dt7h4\" (UID: \"fdeb6f74-c52a-4497-be8e-516b4244aefe\") " pod="openshift-ingress/e2e-shared-b-w3-556c14-openshift-default-754b5bc799-dt7h4" Nov 03 01:11:04.857539 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:11:04.857525 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fdeb6f74-c52a-4497-be8e-516b4244aefe-workload-socket\") pod \"e2e-shared-b-w3-556c14-openshift-default-754b5bc799-dt7h4\" (UID: \"fdeb6f74-c52a-4497-be8e-516b4244aefe\") " pod="openshift-ingress/e2e-shared-b-w3-556c14-openshift-default-754b5bc799-dt7h4" Nov 03 01:11:04.857655 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:11:04.857587 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fdeb6f74-c52a-4497-be8e-516b4244aefe-istio-data\") pod \"e2e-shared-b-w3-556c14-openshift-default-754b5bc799-dt7h4\" (UID: \"fdeb6f74-c52a-4497-be8e-516b4244aefe\") " pod="openshift-ingress/e2e-shared-b-w3-556c14-openshift-default-754b5bc799-dt7h4" Nov 03 01:11:04.941134 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:11:04.941092 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:11:04.959710 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:11:04.959676 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fdeb6f74-c52a-4497-be8e-516b4244aefe-istiod-ca-cert\") pod \"e2e-shared-b-w3-556c14-openshift-default-754b5bc799-dt7h4\" (UID: \"fdeb6f74-c52a-4497-be8e-516b4244aefe\") " pod="openshift-ingress/e2e-shared-b-w3-556c14-openshift-default-754b5bc799-dt7h4" Nov 03 01:11:04.959987 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:11:04.959728 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l9l2j\" (UniqueName: \"kubernetes.io/projected/fdeb6f74-c52a-4497-be8e-516b4244aefe-kube-api-access-l9l2j\") pod \"e2e-shared-b-w3-556c14-openshift-default-754b5bc799-dt7h4\" (UID: \"fdeb6f74-c52a-4497-be8e-516b4244aefe\") " pod="openshift-ingress/e2e-shared-b-w3-556c14-openshift-default-754b5bc799-dt7h4" Nov 03 01:11:04.959987 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:11:04.959781 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fdeb6f74-c52a-4497-be8e-516b4244aefe-workload-certs\") pod \"e2e-shared-b-w3-556c14-openshift-default-754b5bc799-dt7h4\" (UID: \"fdeb6f74-c52a-4497-be8e-516b4244aefe\") " pod="openshift-ingress/e2e-shared-b-w3-556c14-openshift-default-754b5bc799-dt7h4" Nov 03 01:11:04.959987 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:11:04.959898 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fdeb6f74-c52a-4497-be8e-516b4244aefe-istio-envoy\") pod \"e2e-shared-b-w3-556c14-openshift-default-754b5bc799-dt7h4\" (UID: \"fdeb6f74-c52a-4497-be8e-516b4244aefe\") " pod="openshift-ingress/e2e-shared-b-w3-556c14-openshift-default-754b5bc799-dt7h4" Nov 03 01:11:04.960253 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:11:04.960012 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fdeb6f74-c52a-4497-be8e-516b4244aefe-istio-token\") pod \"e2e-shared-b-w3-556c14-openshift-default-754b5bc799-dt7h4\" (UID: \"fdeb6f74-c52a-4497-be8e-516b4244aefe\") " pod="openshift-ingress/e2e-shared-b-w3-556c14-openshift-default-754b5bc799-dt7h4" Nov 03 01:11:04.960253 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:11:04.960054 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fdeb6f74-c52a-4497-be8e-516b4244aefe-credential-socket\") pod \"e2e-shared-b-w3-556c14-openshift-default-754b5bc799-dt7h4\" (UID: \"fdeb6f74-c52a-4497-be8e-516b4244aefe\") " pod="openshift-ingress/e2e-shared-b-w3-556c14-openshift-default-754b5bc799-dt7h4" Nov 03 01:11:04.960253 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:11:04.960124 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fdeb6f74-c52a-4497-be8e-516b4244aefe-istio-podinfo\") pod \"e2e-shared-b-w3-556c14-openshift-default-754b5bc799-dt7h4\" (UID: \"fdeb6f74-c52a-4497-be8e-516b4244aefe\") " pod="openshift-ingress/e2e-shared-b-w3-556c14-openshift-default-754b5bc799-dt7h4" Nov 03 01:11:04.960253 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:11:04.960236 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fdeb6f74-c52a-4497-be8e-516b4244aefe-workload-socket\") pod \"e2e-shared-b-w3-556c14-openshift-default-754b5bc799-dt7h4\" (UID: \"fdeb6f74-c52a-4497-be8e-516b4244aefe\") " pod="openshift-ingress/e2e-shared-b-w3-556c14-openshift-default-754b5bc799-dt7h4" Nov 03 01:11:04.960959 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:11:04.960890 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fdeb6f74-c52a-4497-be8e-516b4244aefe-istio-data\") pod \"e2e-shared-b-w3-556c14-openshift-default-754b5bc799-dt7h4\" (UID: \"fdeb6f74-c52a-4497-be8e-516b4244aefe\") " pod="openshift-ingress/e2e-shared-b-w3-556c14-openshift-default-754b5bc799-dt7h4" Nov 03 01:11:04.961935 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:11:04.961881 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fdeb6f74-c52a-4497-be8e-516b4244aefe-workload-certs\") pod \"e2e-shared-b-w3-556c14-openshift-default-754b5bc799-dt7h4\" (UID: \"fdeb6f74-c52a-4497-be8e-516b4244aefe\") " pod="openshift-ingress/e2e-shared-b-w3-556c14-openshift-default-754b5bc799-dt7h4" Nov 03 01:11:04.962038 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:11:04.961951 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fdeb6f74-c52a-4497-be8e-516b4244aefe-istio-data\") pod \"e2e-shared-b-w3-556c14-openshift-default-754b5bc799-dt7h4\" (UID: \"fdeb6f74-c52a-4497-be8e-516b4244aefe\") " pod="openshift-ingress/e2e-shared-b-w3-556c14-openshift-default-754b5bc799-dt7h4" Nov 03 01:11:04.962038 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:11:04.961957 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fdeb6f74-c52a-4497-be8e-516b4244aefe-workload-socket\") pod \"e2e-shared-b-w3-556c14-openshift-default-754b5bc799-dt7h4\" (UID: \"fdeb6f74-c52a-4497-be8e-516b4244aefe\") " pod="openshift-ingress/e2e-shared-b-w3-556c14-openshift-default-754b5bc799-dt7h4" Nov 03 01:11:04.962131 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:11:04.962062 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fdeb6f74-c52a-4497-be8e-516b4244aefe-credential-socket\") pod \"e2e-shared-b-w3-556c14-openshift-default-754b5bc799-dt7h4\" (UID: \"fdeb6f74-c52a-4497-be8e-516b4244aefe\") " pod="openshift-ingress/e2e-shared-b-w3-556c14-openshift-default-754b5bc799-dt7h4" Nov 03 01:11:04.962377 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:11:04.962355 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fdeb6f74-c52a-4497-be8e-516b4244aefe-istiod-ca-cert\") pod \"e2e-shared-b-w3-556c14-openshift-default-754b5bc799-dt7h4\" (UID: \"fdeb6f74-c52a-4497-be8e-516b4244aefe\") " pod="openshift-ingress/e2e-shared-b-w3-556c14-openshift-default-754b5bc799-dt7h4" Nov 03 01:11:04.964103 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:11:04.964073 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fdeb6f74-c52a-4497-be8e-516b4244aefe-istio-envoy\") pod \"e2e-shared-b-w3-556c14-openshift-default-754b5bc799-dt7h4\" (UID: \"fdeb6f74-c52a-4497-be8e-516b4244aefe\") " pod="openshift-ingress/e2e-shared-b-w3-556c14-openshift-default-754b5bc799-dt7h4" Nov 03 01:11:04.964212 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:11:04.964179 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fdeb6f74-c52a-4497-be8e-516b4244aefe-istio-podinfo\") pod \"e2e-shared-b-w3-556c14-openshift-default-754b5bc799-dt7h4\" (UID: \"fdeb6f74-c52a-4497-be8e-516b4244aefe\") " pod="openshift-ingress/e2e-shared-b-w3-556c14-openshift-default-754b5bc799-dt7h4" Nov 03 01:11:04.969249 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:11:04.969228 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fdeb6f74-c52a-4497-be8e-516b4244aefe-istio-token\") pod \"e2e-shared-b-w3-556c14-openshift-default-754b5bc799-dt7h4\" (UID: \"fdeb6f74-c52a-4497-be8e-516b4244aefe\") " pod="openshift-ingress/e2e-shared-b-w3-556c14-openshift-default-754b5bc799-dt7h4" Nov 03 01:11:04.969672 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:11:04.969654 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-l9l2j\" (UniqueName: \"kubernetes.io/projected/fdeb6f74-c52a-4497-be8e-516b4244aefe-kube-api-access-l9l2j\") pod \"e2e-shared-b-w3-556c14-openshift-default-754b5bc799-dt7h4\" (UID: \"fdeb6f74-c52a-4497-be8e-516b4244aefe\") " pod="openshift-ingress/e2e-shared-b-w3-556c14-openshift-default-754b5bc799-dt7h4" Nov 03 01:11:04.989943 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:11:04.989898 4939 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-3c5983-564977cd99-qs5fk"] Nov 03 01:11:04.994239 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:11:04.994220 4939 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-3c5983-564977cd99-qs5fk" Nov 03 01:11:04.998891 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:11:04.998868 4939 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-extmod-b-3c5983\"" Nov 03 01:11:04.999071 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:11:04.998985 4939 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-extmod-b-3c5983-dockercfg-6gfld\"" Nov 03 01:11:05.002076 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:11:05.002049 4939 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-3c5983-564977cd99-qs5fk"] Nov 03 01:11:05.030899 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:11:05.030863 4939 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-b-3c5983-7fbf586cf5-92rtm"] Nov 03 01:11:05.040192 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:11:05.040165 4939 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-b-3c5983-7fbf586cf5-92rtm" Nov 03 01:11:05.042810 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:11:05.042779 4939 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-b-3c5983-7fbf586cf5-92rtm"] Nov 03 01:11:05.063024 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:11:05.062991 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3137d8da-32e8-47fd-87a6-f08aebe2a384-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-b-3c5983-564977cd99-qs5fk\" (UID: \"3137d8da-32e8-47fd-87a6-f08aebe2a384\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-3c5983-564977cd99-qs5fk" Nov 03 01:11:05.063438 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:11:05.063413 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w9kx9\" (UniqueName: \"kubernetes.io/projected/3137d8da-32e8-47fd-87a6-f08aebe2a384-kube-api-access-w9kx9\") pod \"payload-pre-processing-e2e-extmod-b-3c5983-564977cd99-qs5fk\" (UID: \"3137d8da-32e8-47fd-87a6-f08aebe2a384\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-3c5983-564977cd99-qs5fk" Nov 03 01:11:05.074875 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:11:05.071896 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:11:05.086575 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:11:05.086542 4939 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w3-556c14-openshift-default-754b5bc799-dt7h4" Nov 03 01:11:05.204645 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:11:05.203002 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/16b64414-6144-4743-bc96-91dffcd4a6b7-plugins-config-volume\") pod \"payload-processing-e2e-extmod-b-3c5983-7fbf586cf5-92rtm\" (UID: \"16b64414-6144-4743-bc96-91dffcd4a6b7\") " pod="openshift-ingress/payload-processing-e2e-extmod-b-3c5983-7fbf586cf5-92rtm" Nov 03 01:11:05.204645 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:11:05.203078 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w9kx9\" (UniqueName: \"kubernetes.io/projected/3137d8da-32e8-47fd-87a6-f08aebe2a384-kube-api-access-w9kx9\") pod \"payload-pre-processing-e2e-extmod-b-3c5983-564977cd99-qs5fk\" (UID: \"3137d8da-32e8-47fd-87a6-f08aebe2a384\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-3c5983-564977cd99-qs5fk" Nov 03 01:11:05.204645 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:11:05.203176 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r2qwl\" (UniqueName: \"kubernetes.io/projected/16b64414-6144-4743-bc96-91dffcd4a6b7-kube-api-access-r2qwl\") pod \"payload-processing-e2e-extmod-b-3c5983-7fbf586cf5-92rtm\" (UID: \"16b64414-6144-4743-bc96-91dffcd4a6b7\") " pod="openshift-ingress/payload-processing-e2e-extmod-b-3c5983-7fbf586cf5-92rtm" Nov 03 01:11:05.204645 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:11:05.203248 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3137d8da-32e8-47fd-87a6-f08aebe2a384-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-b-3c5983-564977cd99-qs5fk\" (UID: \"3137d8da-32e8-47fd-87a6-f08aebe2a384\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-3c5983-564977cd99-qs5fk" Nov 03 01:11:05.204645 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:11:05.204259 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3137d8da-32e8-47fd-87a6-f08aebe2a384-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-b-3c5983-564977cd99-qs5fk\" (UID: \"3137d8da-32e8-47fd-87a6-f08aebe2a384\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-3c5983-564977cd99-qs5fk" Nov 03 01:11:05.214933 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:11:05.212872 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:11:05.259984 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:11:05.250689 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-w9kx9\" (UniqueName: \"kubernetes.io/projected/3137d8da-32e8-47fd-87a6-f08aebe2a384-kube-api-access-w9kx9\") pod \"payload-pre-processing-e2e-extmod-b-3c5983-564977cd99-qs5fk\" (UID: \"3137d8da-32e8-47fd-87a6-f08aebe2a384\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-3c5983-564977cd99-qs5fk" Nov 03 01:11:05.319941 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:11:05.318308 4939 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-3c5983-564977cd99-qs5fk" Nov 03 01:11:05.329942 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:11:05.324518 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/16b64414-6144-4743-bc96-91dffcd4a6b7-plugins-config-volume\") pod \"payload-processing-e2e-extmod-b-3c5983-7fbf586cf5-92rtm\" (UID: \"16b64414-6144-4743-bc96-91dffcd4a6b7\") " pod="openshift-ingress/payload-processing-e2e-extmod-b-3c5983-7fbf586cf5-92rtm" Nov 03 01:11:05.329942 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:11:05.325342 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/16b64414-6144-4743-bc96-91dffcd4a6b7-plugins-config-volume\") pod \"payload-processing-e2e-extmod-b-3c5983-7fbf586cf5-92rtm\" (UID: \"16b64414-6144-4743-bc96-91dffcd4a6b7\") " pod="openshift-ingress/payload-processing-e2e-extmod-b-3c5983-7fbf586cf5-92rtm" Nov 03 01:11:05.333946 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:11:05.333625 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r2qwl\" (UniqueName: \"kubernetes.io/projected/16b64414-6144-4743-bc96-91dffcd4a6b7-kube-api-access-r2qwl\") pod \"payload-processing-e2e-extmod-b-3c5983-7fbf586cf5-92rtm\" (UID: \"16b64414-6144-4743-bc96-91dffcd4a6b7\") " pod="openshift-ingress/payload-processing-e2e-extmod-b-3c5983-7fbf586cf5-92rtm" Nov 03 01:11:05.443828 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:11:05.443791 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-r2qwl\" (UniqueName: \"kubernetes.io/projected/16b64414-6144-4743-bc96-91dffcd4a6b7-kube-api-access-r2qwl\") pod \"payload-processing-e2e-extmod-b-3c5983-7fbf586cf5-92rtm\" (UID: \"16b64414-6144-4743-bc96-91dffcd4a6b7\") " pod="openshift-ingress/payload-processing-e2e-extmod-b-3c5983-7fbf586cf5-92rtm" Nov 03 01:11:05.658530 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:11:05.658275 4939 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-b-3c5983-7fbf586cf5-92rtm" Nov 03 01:11:05.764794 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:11:05.764722 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:11:05.875288 ip-10-0-140-107 kubenswrapper[4939]: W1103 01:11:05.875236 4939 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfdeb6f74_c52a_4497_be8e_516b4244aefe.slice/crio-a9786ae06ff4ee4af5655728e22c71b793b96e1077ba038e387f54d43822d8e4 WatchSource:0}: Error finding container a9786ae06ff4ee4af5655728e22c71b793b96e1077ba038e387f54d43822d8e4: Status 404 returned error can't find the container with id a9786ae06ff4ee4af5655728e22c71b793b96e1077ba038e387f54d43822d8e4 Nov 03 01:11:05.881730 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:11:05.881699 4939 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="00fccd6d-ef84-4b47-9531-090a635045e9" path="/var/lib/kubelet/pods/00fccd6d-ef84-4b47-9531-090a635045e9/volumes" Nov 03 01:11:05.882417 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:11:05.882393 4939 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-556c14-openshift-default-754b5bc799-dt7h4"] Nov 03 01:11:05.883450 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:11:05.883407 4939 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 01:11:05.885151 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:11:05.883497 4939 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 01:11:05.885151 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:11:05.883576 4939 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 01:11:06.006599 ip-10-0-140-107 kubenswrapper[4939]: W1103 01:11:06.006566 4939 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod16b64414_6144_4743_bc96_91dffcd4a6b7.slice/crio-e671955e66bd811cf8e13de3486a2c54213b910412a59dc7b8507d4cc7ff39d8 WatchSource:0}: Error finding container e671955e66bd811cf8e13de3486a2c54213b910412a59dc7b8507d4cc7ff39d8: Status 404 returned error can't find the container with id e671955e66bd811cf8e13de3486a2c54213b910412a59dc7b8507d4cc7ff39d8 Nov 03 01:11:06.008154 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:11:06.008033 4939 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-b-3c5983-7fbf586cf5-92rtm"] Nov 03 01:11:06.116406 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:11:06.116376 4939 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-3c5983-564977cd99-qs5fk"] Nov 03 01:11:06.127334 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:11:06.127275 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:11:06.311487 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:11:06.311454 4939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-b-3c5983-7fbf586cf5-92rtm" event={"ID":"16b64414-6144-4743-bc96-91dffcd4a6b7","Type":"ContainerStarted","Data":"177c06e36a95a1b726d3dc4d6ca587944b9f13bdb65004220d6bc5f91c9c44d6"} Nov 03 01:11:06.311645 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:11:06.311515 4939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-b-3c5983-7fbf586cf5-92rtm" event={"ID":"16b64414-6144-4743-bc96-91dffcd4a6b7","Type":"ContainerStarted","Data":"e671955e66bd811cf8e13de3486a2c54213b910412a59dc7b8507d4cc7ff39d8"} Nov 03 01:11:06.312085 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:11:06.311791 4939 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-extmod-b-3c5983-7fbf586cf5-92rtm" Nov 03 01:11:06.313408 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:11:06.313380 4939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-3c5983-564977cd99-qs5fk" event={"ID":"3137d8da-32e8-47fd-87a6-f08aebe2a384","Type":"ContainerStarted","Data":"e001b6889046d2fb5bbc3068b42115b7e1d69afe8c8cbe137943670683419f24"} Nov 03 01:11:06.315303 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:11:06.315278 4939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-556c14-openshift-default-754b5bc799-dt7h4" event={"ID":"fdeb6f74-c52a-4497-be8e-516b4244aefe","Type":"ContainerStarted","Data":"fa857baf4fd7a0bb80fc8c8b74ea6d8fb9998f9c7498de24814258378cbf5930"} Nov 03 01:11:06.315402 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:11:06.315326 4939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-556c14-openshift-default-754b5bc799-dt7h4" event={"ID":"fdeb6f74-c52a-4497-be8e-516b4244aefe","Type":"ContainerStarted","Data":"a9786ae06ff4ee4af5655728e22c71b793b96e1077ba038e387f54d43822d8e4"} Nov 03 01:11:06.334280 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:11:06.334195 4939 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-extmod-b-3c5983-7fbf586cf5-92rtm" podStartSLOduration=1.334183383 podStartE2EDuration="1.334183383s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 01:11:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 01:11:06.330075343 +0000 UTC m=+1814.594410287" watchObservedRunningTime="2026-11-03 01:11:06.334183383 +0000 UTC m=+1814.598518327" Nov 03 01:11:06.353509 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:11:06.353329 4939 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w3-556c14-openshift-default-754b5bc799-dt7h4" podStartSLOduration=2.353313536 podStartE2EDuration="2.353313536s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 01:11:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 01:11:06.350789775 +0000 UTC m=+1814.615124732" watchObservedRunningTime="2026-11-03 01:11:06.353313536 +0000 UTC m=+1814.617648481" Nov 03 01:11:06.970382 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:11:06.970347 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:11:07.090445 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:11:07.090418 4939 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w3-556c14-openshift-default-754b5bc799-dt7h4" Nov 03 01:11:07.096946 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:11:07.091384 4939 patch_prober.go:29] interesting pod/e2e-shared-b-w3-556c14-openshift-default-754b5bc799-dt7h4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.215:15021/healthz/ready\": dial tcp 10.132.0.215:15021: connect: connection refused" start-of-body= Nov 03 01:11:07.098928 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:11:07.098871 4939 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-556c14-openshift-default-754b5bc799-dt7h4" podUID="fdeb6f74-c52a-4497-be8e-516b4244aefe" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.215:15021/healthz/ready\": dial tcp 10.132.0.215:15021: connect: connection refused" Nov 03 01:11:07.129492 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:11:07.129459 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:11:07.316501 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:11:07.316468 4939 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-extmod-b-57079b-openshift-default-5c84d8878f-drtdm"] Nov 03 01:11:07.321968 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:11:07.321883 4939 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-57079b-openshift-default-5c84d8878f-drtdm" Nov 03 01:11:07.382395 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:11:07.382140 4939 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-extmod-b-57079b-openshift-default-dockercfg-2dtlx\"" Nov 03 01:11:07.384954 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:11:07.383457 4939 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-b-57079b-openshift-default-5c84d8878f-drtdm"] Nov 03 01:11:07.401235 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:11:07.401118 4939 generic.go:356] "Generic (PLEG): container finished" podID="e08d7d14-b1fa-4fb2-b2d5-8c7fc073fc89" containerID="8fd039107c781ac33f7786416fb1447c41ebda9a3ab07311a652f359dd592f23" exitCode=0 Nov 03 01:11:07.401235 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:11:07.401208 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:11:07.401404 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:11:07.401292 4939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-f7680d-openshift-default-c4488d5c9-mlblk" event={"ID":"e08d7d14-b1fa-4fb2-b2d5-8c7fc073fc89","Type":"ContainerDied","Data":"8fd039107c781ac33f7786416fb1447c41ebda9a3ab07311a652f359dd592f23"} Nov 03 01:11:07.404810 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:11:07.403334 4939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-3c5983-564977cd99-qs5fk" event={"ID":"3137d8da-32e8-47fd-87a6-f08aebe2a384","Type":"ContainerStarted","Data":"aae7b6322990462714bb68ba10cf5baf2dc655cea6d5819f6d0a0a81bad6fed4"} Nov 03 01:11:07.404810 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:11:07.403679 4939 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-3c5983-564977cd99-qs5fk" Nov 03 01:11:07.406746 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:11:07.406407 4939 generic.go:356] "Generic (PLEG): container finished" podID="9b5bb111-d57c-478b-b990-db75ac2df998" containerID="e5ff86a1fadda7447aaa59b5b3ece9e64d7f369acf0fe17c71c037b90c523ffd" exitCode=0 Nov 03 01:11:07.406967 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:11:07.406940 4939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-f7680d-openshift-default-c4488d5c9-ldr7q" event={"ID":"9b5bb111-d57c-478b-b990-db75ac2df998","Type":"ContainerDied","Data":"e5ff86a1fadda7447aaa59b5b3ece9e64d7f369acf0fe17c71c037b90c523ffd"} Nov 03 01:11:07.433902 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:11:07.433832 4939 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-3c5983-564977cd99-qs5fk" podStartSLOduration=3.4338125760000002 podStartE2EDuration="3.433812576s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 01:11:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 01:11:07.429250513 +0000 UTC m=+1815.693585457" watchObservedRunningTime="2026-11-03 01:11:07.433812576 +0000 UTC m=+1815.698147522" Nov 03 01:11:07.483204 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:11:07.483103 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a68659b3-912d-4df9-a162-fa6d5f949d2a-workload-certs\") pod \"e2e-extmod-b-57079b-openshift-default-5c84d8878f-drtdm\" (UID: \"a68659b3-912d-4df9-a162-fa6d5f949d2a\") " pod="openshift-ingress/e2e-extmod-b-57079b-openshift-default-5c84d8878f-drtdm" Nov 03 01:11:07.483404 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:11:07.483344 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dwwc2\" (UniqueName: \"kubernetes.io/projected/a68659b3-912d-4df9-a162-fa6d5f949d2a-kube-api-access-dwwc2\") pod \"e2e-extmod-b-57079b-openshift-default-5c84d8878f-drtdm\" (UID: \"a68659b3-912d-4df9-a162-fa6d5f949d2a\") " pod="openshift-ingress/e2e-extmod-b-57079b-openshift-default-5c84d8878f-drtdm" Nov 03 01:11:07.483404 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:11:07.483380 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a68659b3-912d-4df9-a162-fa6d5f949d2a-istio-data\") pod \"e2e-extmod-b-57079b-openshift-default-5c84d8878f-drtdm\" (UID: \"a68659b3-912d-4df9-a162-fa6d5f949d2a\") " pod="openshift-ingress/e2e-extmod-b-57079b-openshift-default-5c84d8878f-drtdm" Nov 03 01:11:07.483550 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:11:07.483504 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a68659b3-912d-4df9-a162-fa6d5f949d2a-istio-envoy\") pod \"e2e-extmod-b-57079b-openshift-default-5c84d8878f-drtdm\" (UID: \"a68659b3-912d-4df9-a162-fa6d5f949d2a\") " pod="openshift-ingress/e2e-extmod-b-57079b-openshift-default-5c84d8878f-drtdm" Nov 03 01:11:07.483612 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:11:07.483583 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a68659b3-912d-4df9-a162-fa6d5f949d2a-istiod-ca-cert\") pod \"e2e-extmod-b-57079b-openshift-default-5c84d8878f-drtdm\" (UID: \"a68659b3-912d-4df9-a162-fa6d5f949d2a\") " pod="openshift-ingress/e2e-extmod-b-57079b-openshift-default-5c84d8878f-drtdm" Nov 03 01:11:07.483681 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:11:07.483662 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a68659b3-912d-4df9-a162-fa6d5f949d2a-workload-socket\") pod \"e2e-extmod-b-57079b-openshift-default-5c84d8878f-drtdm\" (UID: \"a68659b3-912d-4df9-a162-fa6d5f949d2a\") " pod="openshift-ingress/e2e-extmod-b-57079b-openshift-default-5c84d8878f-drtdm" Nov 03 01:11:07.483738 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:11:07.483704 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a68659b3-912d-4df9-a162-fa6d5f949d2a-credential-socket\") pod \"e2e-extmod-b-57079b-openshift-default-5c84d8878f-drtdm\" (UID: \"a68659b3-912d-4df9-a162-fa6d5f949d2a\") " pod="openshift-ingress/e2e-extmod-b-57079b-openshift-default-5c84d8878f-drtdm" Nov 03 01:11:07.483921 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:11:07.483890 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a68659b3-912d-4df9-a162-fa6d5f949d2a-istio-token\") pod \"e2e-extmod-b-57079b-openshift-default-5c84d8878f-drtdm\" (UID: \"a68659b3-912d-4df9-a162-fa6d5f949d2a\") " pod="openshift-ingress/e2e-extmod-b-57079b-openshift-default-5c84d8878f-drtdm" Nov 03 01:11:07.484035 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:11:07.483989 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a68659b3-912d-4df9-a162-fa6d5f949d2a-istio-podinfo\") pod \"e2e-extmod-b-57079b-openshift-default-5c84d8878f-drtdm\" (UID: \"a68659b3-912d-4df9-a162-fa6d5f949d2a\") " pod="openshift-ingress/e2e-extmod-b-57079b-openshift-default-5c84d8878f-drtdm" Nov 03 01:11:07.531174 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:11:07.531139 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:11:07.598933 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:11:07.592704 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a68659b3-912d-4df9-a162-fa6d5f949d2a-istio-podinfo\") pod \"e2e-extmod-b-57079b-openshift-default-5c84d8878f-drtdm\" (UID: \"a68659b3-912d-4df9-a162-fa6d5f949d2a\") " pod="openshift-ingress/e2e-extmod-b-57079b-openshift-default-5c84d8878f-drtdm" Nov 03 01:11:07.598933 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:11:07.592870 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a68659b3-912d-4df9-a162-fa6d5f949d2a-workload-certs\") pod \"e2e-extmod-b-57079b-openshift-default-5c84d8878f-drtdm\" (UID: \"a68659b3-912d-4df9-a162-fa6d5f949d2a\") " pod="openshift-ingress/e2e-extmod-b-57079b-openshift-default-5c84d8878f-drtdm" Nov 03 01:11:07.598933 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:11:07.593126 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dwwc2\" (UniqueName: \"kubernetes.io/projected/a68659b3-912d-4df9-a162-fa6d5f949d2a-kube-api-access-dwwc2\") pod \"e2e-extmod-b-57079b-openshift-default-5c84d8878f-drtdm\" (UID: \"a68659b3-912d-4df9-a162-fa6d5f949d2a\") " pod="openshift-ingress/e2e-extmod-b-57079b-openshift-default-5c84d8878f-drtdm" Nov 03 01:11:07.598933 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:11:07.593160 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a68659b3-912d-4df9-a162-fa6d5f949d2a-istio-data\") pod \"e2e-extmod-b-57079b-openshift-default-5c84d8878f-drtdm\" (UID: \"a68659b3-912d-4df9-a162-fa6d5f949d2a\") " pod="openshift-ingress/e2e-extmod-b-57079b-openshift-default-5c84d8878f-drtdm" Nov 03 01:11:07.598933 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:11:07.593214 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a68659b3-912d-4df9-a162-fa6d5f949d2a-istio-envoy\") pod \"e2e-extmod-b-57079b-openshift-default-5c84d8878f-drtdm\" (UID: \"a68659b3-912d-4df9-a162-fa6d5f949d2a\") " pod="openshift-ingress/e2e-extmod-b-57079b-openshift-default-5c84d8878f-drtdm" Nov 03 01:11:07.598933 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:11:07.593309 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a68659b3-912d-4df9-a162-fa6d5f949d2a-istiod-ca-cert\") pod \"e2e-extmod-b-57079b-openshift-default-5c84d8878f-drtdm\" (UID: \"a68659b3-912d-4df9-a162-fa6d5f949d2a\") " pod="openshift-ingress/e2e-extmod-b-57079b-openshift-default-5c84d8878f-drtdm" Nov 03 01:11:07.598933 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:11:07.593429 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a68659b3-912d-4df9-a162-fa6d5f949d2a-workload-socket\") pod \"e2e-extmod-b-57079b-openshift-default-5c84d8878f-drtdm\" (UID: \"a68659b3-912d-4df9-a162-fa6d5f949d2a\") " pod="openshift-ingress/e2e-extmod-b-57079b-openshift-default-5c84d8878f-drtdm" Nov 03 01:11:07.598933 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:11:07.593481 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a68659b3-912d-4df9-a162-fa6d5f949d2a-credential-socket\") pod \"e2e-extmod-b-57079b-openshift-default-5c84d8878f-drtdm\" (UID: \"a68659b3-912d-4df9-a162-fa6d5f949d2a\") " pod="openshift-ingress/e2e-extmod-b-57079b-openshift-default-5c84d8878f-drtdm" Nov 03 01:11:07.598933 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:11:07.593654 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a68659b3-912d-4df9-a162-fa6d5f949d2a-istio-token\") pod \"e2e-extmod-b-57079b-openshift-default-5c84d8878f-drtdm\" (UID: \"a68659b3-912d-4df9-a162-fa6d5f949d2a\") " pod="openshift-ingress/e2e-extmod-b-57079b-openshift-default-5c84d8878f-drtdm" Nov 03 01:11:07.635003 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:11:07.621827 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a68659b3-912d-4df9-a162-fa6d5f949d2a-workload-certs\") pod \"e2e-extmod-b-57079b-openshift-default-5c84d8878f-drtdm\" (UID: \"a68659b3-912d-4df9-a162-fa6d5f949d2a\") " pod="openshift-ingress/e2e-extmod-b-57079b-openshift-default-5c84d8878f-drtdm" Nov 03 01:11:07.635003 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:11:07.622447 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a68659b3-912d-4df9-a162-fa6d5f949d2a-workload-socket\") pod \"e2e-extmod-b-57079b-openshift-default-5c84d8878f-drtdm\" (UID: \"a68659b3-912d-4df9-a162-fa6d5f949d2a\") " pod="openshift-ingress/e2e-extmod-b-57079b-openshift-default-5c84d8878f-drtdm" Nov 03 01:11:07.635003 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:11:07.623461 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a68659b3-912d-4df9-a162-fa6d5f949d2a-istio-data\") pod \"e2e-extmod-b-57079b-openshift-default-5c84d8878f-drtdm\" (UID: \"a68659b3-912d-4df9-a162-fa6d5f949d2a\") " pod="openshift-ingress/e2e-extmod-b-57079b-openshift-default-5c84d8878f-drtdm" Nov 03 01:11:07.635003 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:11:07.623854 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a68659b3-912d-4df9-a162-fa6d5f949d2a-credential-socket\") pod \"e2e-extmod-b-57079b-openshift-default-5c84d8878f-drtdm\" (UID: \"a68659b3-912d-4df9-a162-fa6d5f949d2a\") " pod="openshift-ingress/e2e-extmod-b-57079b-openshift-default-5c84d8878f-drtdm" Nov 03 01:11:07.635800 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:11:07.635768 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a68659b3-912d-4df9-a162-fa6d5f949d2a-istiod-ca-cert\") pod \"e2e-extmod-b-57079b-openshift-default-5c84d8878f-drtdm\" (UID: \"a68659b3-912d-4df9-a162-fa6d5f949d2a\") " pod="openshift-ingress/e2e-extmod-b-57079b-openshift-default-5c84d8878f-drtdm" Nov 03 01:11:07.682346 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:11:07.682314 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a68659b3-912d-4df9-a162-fa6d5f949d2a-istio-envoy\") pod \"e2e-extmod-b-57079b-openshift-default-5c84d8878f-drtdm\" (UID: \"a68659b3-912d-4df9-a162-fa6d5f949d2a\") " pod="openshift-ingress/e2e-extmod-b-57079b-openshift-default-5c84d8878f-drtdm" Nov 03 01:11:07.713242 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:11:07.711729 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a68659b3-912d-4df9-a162-fa6d5f949d2a-istio-podinfo\") pod \"e2e-extmod-b-57079b-openshift-default-5c84d8878f-drtdm\" (UID: \"a68659b3-912d-4df9-a162-fa6d5f949d2a\") " pod="openshift-ingress/e2e-extmod-b-57079b-openshift-default-5c84d8878f-drtdm" Nov 03 01:11:07.713242 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:11:07.712583 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dwwc2\" (UniqueName: \"kubernetes.io/projected/a68659b3-912d-4df9-a162-fa6d5f949d2a-kube-api-access-dwwc2\") pod \"e2e-extmod-b-57079b-openshift-default-5c84d8878f-drtdm\" (UID: \"a68659b3-912d-4df9-a162-fa6d5f949d2a\") " pod="openshift-ingress/e2e-extmod-b-57079b-openshift-default-5c84d8878f-drtdm" Nov 03 01:11:07.726082 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:11:07.715382 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a68659b3-912d-4df9-a162-fa6d5f949d2a-istio-token\") pod \"e2e-extmod-b-57079b-openshift-default-5c84d8878f-drtdm\" (UID: \"a68659b3-912d-4df9-a162-fa6d5f949d2a\") " pod="openshift-ingress/e2e-extmod-b-57079b-openshift-default-5c84d8878f-drtdm" Nov 03 01:11:07.732130 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:11:07.732087 4939 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-f7680d-openshift-default-c4488d5c9-mlblk" Nov 03 01:11:08.000466 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:11:08.000340 4939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e08d7d14-b1fa-4fb2-b2d5-8c7fc073fc89-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e08d7d14-b1fa-4fb2-b2d5-8c7fc073fc89-credential-socket\") pod \"e08d7d14-b1fa-4fb2-b2d5-8c7fc073fc89\" (UID: \"e08d7d14-b1fa-4fb2-b2d5-8c7fc073fc89\") " Nov 03 01:11:08.007066 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:11:08.007025 4939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e08d7d14-b1fa-4fb2-b2d5-8c7fc073fc89-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e08d7d14-b1fa-4fb2-b2d5-8c7fc073fc89-workload-certs\") pod \"e08d7d14-b1fa-4fb2-b2d5-8c7fc073fc89\" (UID: \"e08d7d14-b1fa-4fb2-b2d5-8c7fc073fc89\") " Nov 03 01:11:08.007832 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:11:08.007808 4939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e08d7d14-b1fa-4fb2-b2d5-8c7fc073fc89-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e08d7d14-b1fa-4fb2-b2d5-8c7fc073fc89-istio-envoy\") pod \"e08d7d14-b1fa-4fb2-b2d5-8c7fc073fc89\" (UID: \"e08d7d14-b1fa-4fb2-b2d5-8c7fc073fc89\") " Nov 03 01:11:08.028596 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:11:08.028563 4939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e08d7d14-b1fa-4fb2-b2d5-8c7fc073fc89-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e08d7d14-b1fa-4fb2-b2d5-8c7fc073fc89-istio-data\") pod \"e08d7d14-b1fa-4fb2-b2d5-8c7fc073fc89\" (UID: \"e08d7d14-b1fa-4fb2-b2d5-8c7fc073fc89\") " Nov 03 01:11:08.029243 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:11:08.029202 4939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e08d7d14-b1fa-4fb2-b2d5-8c7fc073fc89-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e08d7d14-b1fa-4fb2-b2d5-8c7fc073fc89-workload-socket\") pod \"e08d7d14-b1fa-4fb2-b2d5-8c7fc073fc89\" (UID: \"e08d7d14-b1fa-4fb2-b2d5-8c7fc073fc89\") " Nov 03 01:11:08.029497 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:11:08.029481 4939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/e08d7d14-b1fa-4fb2-b2d5-8c7fc073fc89-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e08d7d14-b1fa-4fb2-b2d5-8c7fc073fc89-istio-podinfo\") pod \"e08d7d14-b1fa-4fb2-b2d5-8c7fc073fc89\" (UID: \"e08d7d14-b1fa-4fb2-b2d5-8c7fc073fc89\") " Nov 03 01:11:08.029928 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:11:08.029889 4939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e08d7d14-b1fa-4fb2-b2d5-8c7fc073fc89-istio-token\" (UniqueName: \"kubernetes.io/projected/e08d7d14-b1fa-4fb2-b2d5-8c7fc073fc89-istio-token\") pod \"e08d7d14-b1fa-4fb2-b2d5-8c7fc073fc89\" (UID: \"e08d7d14-b1fa-4fb2-b2d5-8c7fc073fc89\") " Nov 03 01:11:08.030248 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:11:08.030217 4939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e08d7d14-b1fa-4fb2-b2d5-8c7fc073fc89-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e08d7d14-b1fa-4fb2-b2d5-8c7fc073fc89-istiod-ca-cert\") pod \"e08d7d14-b1fa-4fb2-b2d5-8c7fc073fc89\" (UID: \"e08d7d14-b1fa-4fb2-b2d5-8c7fc073fc89\") " Nov 03 01:11:08.030548 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:11:08.030529 4939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e08d7d14-b1fa-4fb2-b2d5-8c7fc073fc89-kube-api-access-htxhs\" (UniqueName: \"kubernetes.io/projected/e08d7d14-b1fa-4fb2-b2d5-8c7fc073fc89-kube-api-access-htxhs\") pod \"e08d7d14-b1fa-4fb2-b2d5-8c7fc073fc89\" (UID: \"e08d7d14-b1fa-4fb2-b2d5-8c7fc073fc89\") " Nov 03 01:11:08.038593 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:11:08.026754 4939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e08d7d14-b1fa-4fb2-b2d5-8c7fc073fc89-credential-socket" pod "e08d7d14-b1fa-4fb2-b2d5-8c7fc073fc89" (UID: "e08d7d14-b1fa-4fb2-b2d5-8c7fc073fc89"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 01:11:08.038748 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:11:08.032372 4939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e08d7d14-b1fa-4fb2-b2d5-8c7fc073fc89-workload-certs" pod "e08d7d14-b1fa-4fb2-b2d5-8c7fc073fc89" (UID: "e08d7d14-b1fa-4fb2-b2d5-8c7fc073fc89"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 01:11:08.038826 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:11:08.038565 4939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e08d7d14-b1fa-4fb2-b2d5-8c7fc073fc89-workload-socket" pod "e08d7d14-b1fa-4fb2-b2d5-8c7fc073fc89" (UID: "e08d7d14-b1fa-4fb2-b2d5-8c7fc073fc89"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 01:11:08.039277 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:11:08.039244 4939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e08d7d14-b1fa-4fb2-b2d5-8c7fc073fc89-istio-data" pod "e08d7d14-b1fa-4fb2-b2d5-8c7fc073fc89" (UID: "e08d7d14-b1fa-4fb2-b2d5-8c7fc073fc89"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 01:11:08.061353 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:11:08.060283 4939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e08d7d14-b1fa-4fb2-b2d5-8c7fc073fc89-istiod-ca-cert" pod "e08d7d14-b1fa-4fb2-b2d5-8c7fc073fc89" (UID: "e08d7d14-b1fa-4fb2-b2d5-8c7fc073fc89"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 01:11:08.061353 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:11:08.060511 4939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e08d7d14-b1fa-4fb2-b2d5-8c7fc073fc89-kube-api-access-htxhs" pod "e08d7d14-b1fa-4fb2-b2d5-8c7fc073fc89" (UID: "e08d7d14-b1fa-4fb2-b2d5-8c7fc073fc89"). InnerVolumeSpecName "kube-api-access-htxhs". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 01:11:08.115491 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:11:08.115455 4939 patch_prober.go:29] interesting pod/e2e-shared-b-w3-556c14-openshift-default-754b5bc799-dt7h4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.215:15021/healthz/ready\": dial tcp 10.132.0.215:15021: connect: connection refused" start-of-body= Nov 03 01:11:08.115633 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:11:08.115524 4939 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-556c14-openshift-default-754b5bc799-dt7h4" podUID="fdeb6f74-c52a-4497-be8e-516b4244aefe" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.215:15021/healthz/ready\": dial tcp 10.132.0.215:15021: connect: connection refused" Nov 03 01:11:08.132161 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:11:08.132125 4939 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-57079b-openshift-default-5c84d8878f-drtdm" Nov 03 01:11:08.177946 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:11:08.137601 4939 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e08d7d14-b1fa-4fb2-b2d5-8c7fc073fc89-credential-socket\") on node \"ip-10-0-140-107.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:11:08.177946 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:11:08.166334 4939 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e08d7d14-b1fa-4fb2-b2d5-8c7fc073fc89-workload-certs\") on node \"ip-10-0-140-107.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:11:08.177946 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:11:08.166361 4939 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e08d7d14-b1fa-4fb2-b2d5-8c7fc073fc89-istio-data\") on node \"ip-10-0-140-107.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:11:08.177946 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:11:08.166380 4939 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e08d7d14-b1fa-4fb2-b2d5-8c7fc073fc89-workload-socket\") on node \"ip-10-0-140-107.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:11:08.196455 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:11:08.196411 4939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e08d7d14-b1fa-4fb2-b2d5-8c7fc073fc89-istio-token" pod "e08d7d14-b1fa-4fb2-b2d5-8c7fc073fc89" (UID: "e08d7d14-b1fa-4fb2-b2d5-8c7fc073fc89"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 01:11:08.202754 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:11:08.202717 4939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e08d7d14-b1fa-4fb2-b2d5-8c7fc073fc89-istio-envoy" pod "e08d7d14-b1fa-4fb2-b2d5-8c7fc073fc89" (UID: "e08d7d14-b1fa-4fb2-b2d5-8c7fc073fc89"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 01:11:08.214951 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:11:08.214086 4939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/e08d7d14-b1fa-4fb2-b2d5-8c7fc073fc89-istio-podinfo" pod "e08d7d14-b1fa-4fb2-b2d5-8c7fc073fc89" (UID: "e08d7d14-b1fa-4fb2-b2d5-8c7fc073fc89"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 01:11:08.295605 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:11:08.295526 4939 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e08d7d14-b1fa-4fb2-b2d5-8c7fc073fc89-istio-token\") on node \"ip-10-0-140-107.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:11:08.295605 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:11:08.295560 4939 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e08d7d14-b1fa-4fb2-b2d5-8c7fc073fc89-istiod-ca-cert\") on node \"ip-10-0-140-107.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:11:08.295605 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:11:08.295575 4939 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-htxhs\" (UniqueName: \"kubernetes.io/projected/e08d7d14-b1fa-4fb2-b2d5-8c7fc073fc89-kube-api-access-htxhs\") on node \"ip-10-0-140-107.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:11:08.295605 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:11:08.295588 4939 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e08d7d14-b1fa-4fb2-b2d5-8c7fc073fc89-istio-envoy\") on node \"ip-10-0-140-107.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:11:08.295605 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:11:08.295601 4939 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e08d7d14-b1fa-4fb2-b2d5-8c7fc073fc89-istio-podinfo\") on node \"ip-10-0-140-107.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:11:08.421929 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:11:08.421849 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:11:08.429534 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:11:08.429465 4939 generic.go:356] "Generic (PLEG): container finished" podID="569c2920-1523-4da8-8644-f789aebcdc7a" containerID="d19447d0abfd950faea5858874c8ad314badd753800f42c9173c681a1eea80d6" exitCode=0 Nov 03 01:11:08.429841 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:11:08.429804 4939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-83755a-openshift-default-c86f9565c-z7flk" event={"ID":"569c2920-1523-4da8-8644-f789aebcdc7a","Type":"ContainerDied","Data":"d19447d0abfd950faea5858874c8ad314badd753800f42c9173c681a1eea80d6"} Nov 03 01:11:08.430803 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:11:08.430724 4939 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w3-717217/auth-test-model-717217-kserve-5c9cb7dbd9-vl754"] Nov 03 01:11:08.432560 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:11:08.432532 4939 generic.go:356] "Generic (PLEG): container finished" podID="0dbecc93-5a61-4100-a319-d3f673f7a5fd" containerID="988c3b00cfe1d42a496c5073f93a1ea2f634e32400f4bb974be56cfdede7ffa0" exitCode=0 Nov 03 01:11:08.432655 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:11:08.432583 4939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-83755a-openshift-default-c86f9565c-zj5jg" event={"ID":"0dbecc93-5a61-4100-a319-d3f673f7a5fd","Type":"ContainerDied","Data":"988c3b00cfe1d42a496c5073f93a1ea2f634e32400f4bb974be56cfdede7ffa0"} Nov 03 01:11:08.437018 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:11:08.436940 4939 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-f7680d-openshift-default-c4488d5c9-mlblk" Nov 03 01:11:08.437258 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:11:08.437233 4939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-f7680d-openshift-default-c4488d5c9-mlblk" event={"ID":"e08d7d14-b1fa-4fb2-b2d5-8c7fc073fc89","Type":"ContainerDied","Data":"e3183b5a85d53a1066123f57f142dcddfba055e478569449c87ed804566809c1"} Nov 03 01:11:08.437338 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:11:08.437289 4939 scope.go:122] "RemoveContainer" containerID="8fd039107c781ac33f7786416fb1447c41ebda9a3ab07311a652f359dd592f23" Nov 03 01:11:08.508718 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:11:08.508686 4939 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-f7680d-openshift-default-c4488d5c9-mlblk"] Nov 03 01:11:08.515797 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:11:08.515418 4939 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-f7680d-openshift-default-c4488d5c9-ldr7q" Nov 03 01:11:08.521477 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:11:08.521439 4939 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-f7680d-openshift-default-c4488d5c9-mlblk"] Nov 03 01:11:08.585929 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:11:08.584595 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:11:08.641935 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:11:08.639047 4939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9b5bb111-d57c-478b-b990-db75ac2df998-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9b5bb111-d57c-478b-b990-db75ac2df998-workload-certs\") pod \"9b5bb111-d57c-478b-b990-db75ac2df998\" (UID: \"9b5bb111-d57c-478b-b990-db75ac2df998\") " Nov 03 01:11:08.641935 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:11:08.639098 4939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9b5bb111-d57c-478b-b990-db75ac2df998-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9b5bb111-d57c-478b-b990-db75ac2df998-workload-socket\") pod \"9b5bb111-d57c-478b-b990-db75ac2df998\" (UID: \"9b5bb111-d57c-478b-b990-db75ac2df998\") " Nov 03 01:11:08.641935 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:11:08.639170 4939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/9b5bb111-d57c-478b-b990-db75ac2df998-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9b5bb111-d57c-478b-b990-db75ac2df998-istio-podinfo\") pod \"9b5bb111-d57c-478b-b990-db75ac2df998\" (UID: \"9b5bb111-d57c-478b-b990-db75ac2df998\") " Nov 03 01:11:08.641935 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:11:08.639202 4939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9b5bb111-d57c-478b-b990-db75ac2df998-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9b5bb111-d57c-478b-b990-db75ac2df998-istio-envoy\") pod \"9b5bb111-d57c-478b-b990-db75ac2df998\" (UID: \"9b5bb111-d57c-478b-b990-db75ac2df998\") " Nov 03 01:11:08.641935 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:11:08.639294 4939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9b5bb111-d57c-478b-b990-db75ac2df998-kube-api-access-wlbpj\" (UniqueName: \"kubernetes.io/projected/9b5bb111-d57c-478b-b990-db75ac2df998-kube-api-access-wlbpj\") pod \"9b5bb111-d57c-478b-b990-db75ac2df998\" (UID: \"9b5bb111-d57c-478b-b990-db75ac2df998\") " Nov 03 01:11:08.641935 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:11:08.639572 4939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9b5bb111-d57c-478b-b990-db75ac2df998-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9b5bb111-d57c-478b-b990-db75ac2df998-istio-data\") pod \"9b5bb111-d57c-478b-b990-db75ac2df998\" (UID: \"9b5bb111-d57c-478b-b990-db75ac2df998\") " Nov 03 01:11:08.641935 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:11:08.639692 4939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9b5bb111-d57c-478b-b990-db75ac2df998-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9b5bb111-d57c-478b-b990-db75ac2df998-istiod-ca-cert\") pod \"9b5bb111-d57c-478b-b990-db75ac2df998\" (UID: \"9b5bb111-d57c-478b-b990-db75ac2df998\") " Nov 03 01:11:08.641935 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:11:08.639719 4939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9b5bb111-d57c-478b-b990-db75ac2df998-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9b5bb111-d57c-478b-b990-db75ac2df998-credential-socket\") pod \"9b5bb111-d57c-478b-b990-db75ac2df998\" (UID: \"9b5bb111-d57c-478b-b990-db75ac2df998\") " Nov 03 01:11:08.641935 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:11:08.639766 4939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9b5bb111-d57c-478b-b990-db75ac2df998-istio-token\" (UniqueName: \"kubernetes.io/projected/9b5bb111-d57c-478b-b990-db75ac2df998-istio-token\") pod \"9b5bb111-d57c-478b-b990-db75ac2df998\" (UID: \"9b5bb111-d57c-478b-b990-db75ac2df998\") " Nov 03 01:11:08.688936 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:11:08.668062 4939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9b5bb111-d57c-478b-b990-db75ac2df998-istio-data" pod "9b5bb111-d57c-478b-b990-db75ac2df998" (UID: "9b5bb111-d57c-478b-b990-db75ac2df998"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 01:11:08.713935 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:11:08.669396 4939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9b5bb111-d57c-478b-b990-db75ac2df998-istiod-ca-cert" pod "9b5bb111-d57c-478b-b990-db75ac2df998" (UID: "9b5bb111-d57c-478b-b990-db75ac2df998"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 01:11:08.713935 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:11:08.670224 4939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9b5bb111-d57c-478b-b990-db75ac2df998-credential-socket" pod "9b5bb111-d57c-478b-b990-db75ac2df998" (UID: "9b5bb111-d57c-478b-b990-db75ac2df998"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 01:11:08.713935 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:11:08.670405 4939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9b5bb111-d57c-478b-b990-db75ac2df998-workload-socket" pod "9b5bb111-d57c-478b-b990-db75ac2df998" (UID: "9b5bb111-d57c-478b-b990-db75ac2df998"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 01:11:08.713935 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:11:08.670569 4939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9b5bb111-d57c-478b-b990-db75ac2df998-workload-certs" pod "9b5bb111-d57c-478b-b990-db75ac2df998" (UID: "9b5bb111-d57c-478b-b990-db75ac2df998"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 01:11:08.732349 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:11:08.724053 4939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/9b5bb111-d57c-478b-b990-db75ac2df998-istio-podinfo" pod "9b5bb111-d57c-478b-b990-db75ac2df998" (UID: "9b5bb111-d57c-478b-b990-db75ac2df998"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 01:11:08.762528 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:11:08.743492 4939 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9b5bb111-d57c-478b-b990-db75ac2df998-istiod-ca-cert\") on node \"ip-10-0-140-107.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:11:08.762528 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:11:08.743531 4939 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9b5bb111-d57c-478b-b990-db75ac2df998-credential-socket\") on node \"ip-10-0-140-107.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:11:08.762528 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:11:08.743549 4939 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9b5bb111-d57c-478b-b990-db75ac2df998-workload-certs\") on node \"ip-10-0-140-107.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:11:08.762528 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:11:08.743567 4939 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9b5bb111-d57c-478b-b990-db75ac2df998-workload-socket\") on node \"ip-10-0-140-107.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:11:08.762528 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:11:08.743583 4939 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9b5bb111-d57c-478b-b990-db75ac2df998-istio-podinfo\") on node \"ip-10-0-140-107.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:11:08.762528 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:11:08.743602 4939 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9b5bb111-d57c-478b-b990-db75ac2df998-istio-data\") on node \"ip-10-0-140-107.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:11:08.765653 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:11:08.765590 4939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9b5bb111-d57c-478b-b990-db75ac2df998-istio-envoy" pod "9b5bb111-d57c-478b-b990-db75ac2df998" (UID: "9b5bb111-d57c-478b-b990-db75ac2df998"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 01:11:08.766221 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:11:08.766168 4939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9b5bb111-d57c-478b-b990-db75ac2df998-istio-token" pod "9b5bb111-d57c-478b-b990-db75ac2df998" (UID: "9b5bb111-d57c-478b-b990-db75ac2df998"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 01:11:08.789146 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:11:08.786502 4939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9b5bb111-d57c-478b-b990-db75ac2df998-kube-api-access-wlbpj" pod "9b5bb111-d57c-478b-b990-db75ac2df998" (UID: "9b5bb111-d57c-478b-b990-db75ac2df998"). InnerVolumeSpecName "kube-api-access-wlbpj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 01:11:08.884035 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:11:08.846720 4939 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wlbpj\" (UniqueName: \"kubernetes.io/projected/9b5bb111-d57c-478b-b990-db75ac2df998-kube-api-access-wlbpj\") on node \"ip-10-0-140-107.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:11:08.884336 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:11:08.884296 4939 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9b5bb111-d57c-478b-b990-db75ac2df998-istio-token\") on node \"ip-10-0-140-107.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:11:08.884504 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:11:08.884487 4939 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9b5bb111-d57c-478b-b990-db75ac2df998-istio-envoy\") on node \"ip-10-0-140-107.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:11:08.925628 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:11:08.925024 4939 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-83755a-openshift-default-c86f9565c-z7flk" Nov 03 01:11:09.000933 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:11:08.985217 4939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/569c2920-1523-4da8-8644-f789aebcdc7a-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/569c2920-1523-4da8-8644-f789aebcdc7a-credential-socket\") pod \"569c2920-1523-4da8-8644-f789aebcdc7a\" (UID: \"569c2920-1523-4da8-8644-f789aebcdc7a\") " Nov 03 01:11:09.000933 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:11:08.985310 4939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/569c2920-1523-4da8-8644-f789aebcdc7a-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/569c2920-1523-4da8-8644-f789aebcdc7a-istiod-ca-cert\") pod \"569c2920-1523-4da8-8644-f789aebcdc7a\" (UID: \"569c2920-1523-4da8-8644-f789aebcdc7a\") " Nov 03 01:11:09.000933 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:11:08.985390 4939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/569c2920-1523-4da8-8644-f789aebcdc7a-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/569c2920-1523-4da8-8644-f789aebcdc7a-istio-envoy\") pod \"569c2920-1523-4da8-8644-f789aebcdc7a\" (UID: \"569c2920-1523-4da8-8644-f789aebcdc7a\") " Nov 03 01:11:09.000933 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:11:08.985572 4939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/569c2920-1523-4da8-8644-f789aebcdc7a-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/569c2920-1523-4da8-8644-f789aebcdc7a-istio-data\") pod \"569c2920-1523-4da8-8644-f789aebcdc7a\" (UID: \"569c2920-1523-4da8-8644-f789aebcdc7a\") " Nov 03 01:11:09.000933 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:11:08.985666 4939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/569c2920-1523-4da8-8644-f789aebcdc7a-kube-api-access-zgqkb\" (UniqueName: \"kubernetes.io/projected/569c2920-1523-4da8-8644-f789aebcdc7a-kube-api-access-zgqkb\") pod \"569c2920-1523-4da8-8644-f789aebcdc7a\" (UID: \"569c2920-1523-4da8-8644-f789aebcdc7a\") " Nov 03 01:11:09.000933 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:11:08.985699 4939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/569c2920-1523-4da8-8644-f789aebcdc7a-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/569c2920-1523-4da8-8644-f789aebcdc7a-istio-podinfo\") pod \"569c2920-1523-4da8-8644-f789aebcdc7a\" (UID: \"569c2920-1523-4da8-8644-f789aebcdc7a\") " Nov 03 01:11:09.000933 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:11:08.985843 4939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/569c2920-1523-4da8-8644-f789aebcdc7a-istio-token\" (UniqueName: \"kubernetes.io/projected/569c2920-1523-4da8-8644-f789aebcdc7a-istio-token\") pod \"569c2920-1523-4da8-8644-f789aebcdc7a\" (UID: \"569c2920-1523-4da8-8644-f789aebcdc7a\") " Nov 03 01:11:09.000933 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:11:08.985942 4939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/569c2920-1523-4da8-8644-f789aebcdc7a-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/569c2920-1523-4da8-8644-f789aebcdc7a-workload-socket\") pod \"569c2920-1523-4da8-8644-f789aebcdc7a\" (UID: \"569c2920-1523-4da8-8644-f789aebcdc7a\") " Nov 03 01:11:09.000933 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:11:08.985997 4939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/569c2920-1523-4da8-8644-f789aebcdc7a-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/569c2920-1523-4da8-8644-f789aebcdc7a-workload-certs\") pod \"569c2920-1523-4da8-8644-f789aebcdc7a\" (UID: \"569c2920-1523-4da8-8644-f789aebcdc7a\") " Nov 03 01:11:09.002275 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:11:09.002231 4939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/569c2920-1523-4da8-8644-f789aebcdc7a-workload-certs" pod "569c2920-1523-4da8-8644-f789aebcdc7a" (UID: "569c2920-1523-4da8-8644-f789aebcdc7a"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 01:11:09.004779 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:11:09.004732 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:11:09.005051 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:11:09.005026 4939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/569c2920-1523-4da8-8644-f789aebcdc7a-istio-data" pod "569c2920-1523-4da8-8644-f789aebcdc7a" (UID: "569c2920-1523-4da8-8644-f789aebcdc7a"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 01:11:09.005448 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:11:09.005425 4939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/569c2920-1523-4da8-8644-f789aebcdc7a-credential-socket" pod "569c2920-1523-4da8-8644-f789aebcdc7a" (UID: "569c2920-1523-4da8-8644-f789aebcdc7a"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 01:11:09.005799 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:11:09.005776 4939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/569c2920-1523-4da8-8644-f789aebcdc7a-istiod-ca-cert" pod "569c2920-1523-4da8-8644-f789aebcdc7a" (UID: "569c2920-1523-4da8-8644-f789aebcdc7a"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 01:11:09.011674 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:11:09.011626 4939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/569c2920-1523-4da8-8644-f789aebcdc7a-workload-socket" pod "569c2920-1523-4da8-8644-f789aebcdc7a" (UID: "569c2920-1523-4da8-8644-f789aebcdc7a"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 01:11:09.014065 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:11:09.013982 4939 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/569c2920-1523-4da8-8644-f789aebcdc7a-credential-socket\") on node \"ip-10-0-140-107.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:11:09.014065 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:11:09.014011 4939 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/569c2920-1523-4da8-8644-f789aebcdc7a-istiod-ca-cert\") on node \"ip-10-0-140-107.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:11:09.014065 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:11:09.014026 4939 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/569c2920-1523-4da8-8644-f789aebcdc7a-istio-data\") on node \"ip-10-0-140-107.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:11:09.014065 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:11:09.014039 4939 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/569c2920-1523-4da8-8644-f789aebcdc7a-workload-socket\") on node \"ip-10-0-140-107.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:11:09.014065 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:11:09.014052 4939 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/569c2920-1523-4da8-8644-f789aebcdc7a-workload-certs\") on node \"ip-10-0-140-107.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:11:09.025248 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:11:09.025092 4939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/569c2920-1523-4da8-8644-f789aebcdc7a-istio-podinfo" pod "569c2920-1523-4da8-8644-f789aebcdc7a" (UID: "569c2920-1523-4da8-8644-f789aebcdc7a"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 01:11:09.025248 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:11:09.025105 4939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/569c2920-1523-4da8-8644-f789aebcdc7a-istio-token" pod "569c2920-1523-4da8-8644-f789aebcdc7a" (UID: "569c2920-1523-4da8-8644-f789aebcdc7a"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 01:11:09.025248 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:11:09.025214 4939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/569c2920-1523-4da8-8644-f789aebcdc7a-istio-envoy" pod "569c2920-1523-4da8-8644-f789aebcdc7a" (UID: "569c2920-1523-4da8-8644-f789aebcdc7a"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 01:11:09.027265 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:11:09.027157 4939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/569c2920-1523-4da8-8644-f789aebcdc7a-kube-api-access-zgqkb" pod "569c2920-1523-4da8-8644-f789aebcdc7a" (UID: "569c2920-1523-4da8-8644-f789aebcdc7a"). InnerVolumeSpecName "kube-api-access-zgqkb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 01:11:09.031883 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:11:09.031852 4939 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-83755a-openshift-default-c86f9565c-zj5jg" Nov 03 01:11:09.045465 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:11:09.045415 4939 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-b-57079b-openshift-default-5c84d8878f-drtdm"] Nov 03 01:11:09.127142 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:11:09.126381 4939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0dbecc93-5a61-4100-a319-d3f673f7a5fd-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0dbecc93-5a61-4100-a319-d3f673f7a5fd-workload-socket\") pod \"0dbecc93-5a61-4100-a319-d3f673f7a5fd\" (UID: \"0dbecc93-5a61-4100-a319-d3f673f7a5fd\") " Nov 03 01:11:09.127142 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:11:09.126468 4939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0dbecc93-5a61-4100-a319-d3f673f7a5fd-kube-api-access-f4rxn\" (UniqueName: \"kubernetes.io/projected/0dbecc93-5a61-4100-a319-d3f673f7a5fd-kube-api-access-f4rxn\") pod \"0dbecc93-5a61-4100-a319-d3f673f7a5fd\" (UID: \"0dbecc93-5a61-4100-a319-d3f673f7a5fd\") " Nov 03 01:11:09.127142 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:11:09.126534 4939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0dbecc93-5a61-4100-a319-d3f673f7a5fd-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0dbecc93-5a61-4100-a319-d3f673f7a5fd-istio-envoy\") pod \"0dbecc93-5a61-4100-a319-d3f673f7a5fd\" (UID: \"0dbecc93-5a61-4100-a319-d3f673f7a5fd\") " Nov 03 01:11:09.127142 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:11:09.126586 4939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/0dbecc93-5a61-4100-a319-d3f673f7a5fd-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0dbecc93-5a61-4100-a319-d3f673f7a5fd-istio-podinfo\") pod \"0dbecc93-5a61-4100-a319-d3f673f7a5fd\" (UID: \"0dbecc93-5a61-4100-a319-d3f673f7a5fd\") " Nov 03 01:11:09.127142 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:11:09.126690 4939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0dbecc93-5a61-4100-a319-d3f673f7a5fd-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0dbecc93-5a61-4100-a319-d3f673f7a5fd-istio-data\") pod \"0dbecc93-5a61-4100-a319-d3f673f7a5fd\" (UID: \"0dbecc93-5a61-4100-a319-d3f673f7a5fd\") " Nov 03 01:11:09.127142 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:11:09.126757 4939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0dbecc93-5a61-4100-a319-d3f673f7a5fd-istio-token\" (UniqueName: \"kubernetes.io/projected/0dbecc93-5a61-4100-a319-d3f673f7a5fd-istio-token\") pod \"0dbecc93-5a61-4100-a319-d3f673f7a5fd\" (UID: \"0dbecc93-5a61-4100-a319-d3f673f7a5fd\") " Nov 03 01:11:09.127142 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:11:09.126827 4939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0dbecc93-5a61-4100-a319-d3f673f7a5fd-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0dbecc93-5a61-4100-a319-d3f673f7a5fd-credential-socket\") pod \"0dbecc93-5a61-4100-a319-d3f673f7a5fd\" (UID: \"0dbecc93-5a61-4100-a319-d3f673f7a5fd\") " Nov 03 01:11:09.127142 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:11:09.126934 4939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0dbecc93-5a61-4100-a319-d3f673f7a5fd-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0dbecc93-5a61-4100-a319-d3f673f7a5fd-workload-certs\") pod \"0dbecc93-5a61-4100-a319-d3f673f7a5fd\" (UID: \"0dbecc93-5a61-4100-a319-d3f673f7a5fd\") " Nov 03 01:11:09.127142 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:11:09.127083 4939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0dbecc93-5a61-4100-a319-d3f673f7a5fd-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0dbecc93-5a61-4100-a319-d3f673f7a5fd-istiod-ca-cert\") pod \"0dbecc93-5a61-4100-a319-d3f673f7a5fd\" (UID: \"0dbecc93-5a61-4100-a319-d3f673f7a5fd\") " Nov 03 01:11:09.129236 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:11:09.129193 4939 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zgqkb\" (UniqueName: \"kubernetes.io/projected/569c2920-1523-4da8-8644-f789aebcdc7a-kube-api-access-zgqkb\") on node \"ip-10-0-140-107.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:11:09.129427 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:11:09.129409 4939 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/569c2920-1523-4da8-8644-f789aebcdc7a-istio-podinfo\") on node \"ip-10-0-140-107.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:11:09.129539 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:11:09.129524 4939 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/569c2920-1523-4da8-8644-f789aebcdc7a-istio-token\") on node \"ip-10-0-140-107.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:11:09.129636 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:11:09.129624 4939 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/569c2920-1523-4da8-8644-f789aebcdc7a-istio-envoy\") on node \"ip-10-0-140-107.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:11:09.130139 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:11:09.130111 4939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0dbecc93-5a61-4100-a319-d3f673f7a5fd-istiod-ca-cert" pod "0dbecc93-5a61-4100-a319-d3f673f7a5fd" (UID: "0dbecc93-5a61-4100-a319-d3f673f7a5fd"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 01:11:09.130866 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:11:09.130270 4939 patch_prober.go:29] interesting pod/e2e-shared-b-w3-556c14-openshift-default-754b5bc799-dt7h4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.215:15021/healthz/ready\": dial tcp 10.132.0.215:15021: connect: connection refused" start-of-body= Nov 03 01:11:09.130866 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:11:09.130340 4939 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-556c14-openshift-default-754b5bc799-dt7h4" podUID="fdeb6f74-c52a-4497-be8e-516b4244aefe" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.215:15021/healthz/ready\": dial tcp 10.132.0.215:15021: connect: connection refused" Nov 03 01:11:09.132828 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:11:09.132783 4939 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 01:11:09.132954 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:11:09.132883 4939 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 01:11:09.133027 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:11:09.132949 4939 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 01:11:09.133523 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:11:09.133211 4939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0dbecc93-5a61-4100-a319-d3f673f7a5fd-workload-certs" pod "0dbecc93-5a61-4100-a319-d3f673f7a5fd" (UID: "0dbecc93-5a61-4100-a319-d3f673f7a5fd"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 01:11:09.133523 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:11:09.133284 4939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0dbecc93-5a61-4100-a319-d3f673f7a5fd-credential-socket" pod "0dbecc93-5a61-4100-a319-d3f673f7a5fd" (UID: "0dbecc93-5a61-4100-a319-d3f673f7a5fd"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 01:11:09.135248 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:11:09.134210 4939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0dbecc93-5a61-4100-a319-d3f673f7a5fd-workload-socket" pod "0dbecc93-5a61-4100-a319-d3f673f7a5fd" (UID: "0dbecc93-5a61-4100-a319-d3f673f7a5fd"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 01:11:09.139729 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:11:09.137563 4939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0dbecc93-5a61-4100-a319-d3f673f7a5fd-istio-data" pod "0dbecc93-5a61-4100-a319-d3f673f7a5fd" (UID: "0dbecc93-5a61-4100-a319-d3f673f7a5fd"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 01:11:09.209548 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:11:09.208637 4939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0dbecc93-5a61-4100-a319-d3f673f7a5fd-istio-token" pod "0dbecc93-5a61-4100-a319-d3f673f7a5fd" (UID: "0dbecc93-5a61-4100-a319-d3f673f7a5fd"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 01:11:09.222952 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:11:09.222887 4939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0dbecc93-5a61-4100-a319-d3f673f7a5fd-kube-api-access-f4rxn" pod "0dbecc93-5a61-4100-a319-d3f673f7a5fd" (UID: "0dbecc93-5a61-4100-a319-d3f673f7a5fd"). InnerVolumeSpecName "kube-api-access-f4rxn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 01:11:09.223545 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:11:09.223514 4939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/0dbecc93-5a61-4100-a319-d3f673f7a5fd-istio-podinfo" pod "0dbecc93-5a61-4100-a319-d3f673f7a5fd" (UID: "0dbecc93-5a61-4100-a319-d3f673f7a5fd"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 01:11:09.224694 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:11:09.224659 4939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0dbecc93-5a61-4100-a319-d3f673f7a5fd-istio-envoy" pod "0dbecc93-5a61-4100-a319-d3f673f7a5fd" (UID: "0dbecc93-5a61-4100-a319-d3f673f7a5fd"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 01:11:09.232783 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:11:09.231408 4939 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0dbecc93-5a61-4100-a319-d3f673f7a5fd-workload-socket\") on node \"ip-10-0-140-107.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:11:09.232783 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:11:09.231443 4939 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-f4rxn\" (UniqueName: \"kubernetes.io/projected/0dbecc93-5a61-4100-a319-d3f673f7a5fd-kube-api-access-f4rxn\") on node \"ip-10-0-140-107.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:11:09.232783 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:11:09.231459 4939 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0dbecc93-5a61-4100-a319-d3f673f7a5fd-istio-envoy\") on node \"ip-10-0-140-107.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:11:09.232783 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:11:09.231477 4939 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0dbecc93-5a61-4100-a319-d3f673f7a5fd-istio-podinfo\") on node \"ip-10-0-140-107.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:11:09.232783 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:11:09.231491 4939 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0dbecc93-5a61-4100-a319-d3f673f7a5fd-istio-data\") on node \"ip-10-0-140-107.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:11:09.232783 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:11:09.231507 4939 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0dbecc93-5a61-4100-a319-d3f673f7a5fd-istio-token\") on node \"ip-10-0-140-107.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:11:09.232783 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:11:09.231520 4939 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0dbecc93-5a61-4100-a319-d3f673f7a5fd-credential-socket\") on node \"ip-10-0-140-107.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:11:09.232783 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:11:09.231534 4939 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0dbecc93-5a61-4100-a319-d3f673f7a5fd-workload-certs\") on node \"ip-10-0-140-107.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:11:09.232783 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:11:09.231547 4939 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0dbecc93-5a61-4100-a319-d3f673f7a5fd-istiod-ca-cert\") on node \"ip-10-0-140-107.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:11:09.481342 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:11:09.481229 4939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-f7680d-openshift-default-c4488d5c9-ldr7q" event={"ID":"9b5bb111-d57c-478b-b990-db75ac2df998","Type":"ContainerDied","Data":"141c582ac526d7ce0827c274d1319e13ddaa01ce5e1951198e9778c25110b152"} Nov 03 01:11:09.481342 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:11:09.481277 4939 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-f7680d-openshift-default-c4488d5c9-ldr7q" Nov 03 01:11:09.481342 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:11:09.481329 4939 scope.go:122] "RemoveContainer" containerID="e5ff86a1fadda7447aaa59b5b3ece9e64d7f369acf0fe17c71c037b90c523ffd" Nov 03 01:11:09.483611 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:11:09.483582 4939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-83755a-openshift-default-c86f9565c-z7flk" event={"ID":"569c2920-1523-4da8-8644-f789aebcdc7a","Type":"ContainerDied","Data":"fbfa0495231f53bb344cfa2473664f8623d1d79f98eb3e186e8976f59d872304"} Nov 03 01:11:09.483725 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:11:09.483706 4939 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-83755a-openshift-default-c86f9565c-z7flk" Nov 03 01:11:09.487072 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:11:09.487042 4939 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-83755a-openshift-default-c86f9565c-zj5jg" Nov 03 01:11:09.487361 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:11:09.487120 4939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-83755a-openshift-default-c86f9565c-zj5jg" event={"ID":"0dbecc93-5a61-4100-a319-d3f673f7a5fd","Type":"ContainerDied","Data":"7dde1552f382d034e7d5958988008679c7b32c8cd65082ec637886618283baf7"} Nov 03 01:11:09.490691 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:11:09.489232 4939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-57079b-openshift-default-5c84d8878f-drtdm" event={"ID":"a68659b3-912d-4df9-a162-fa6d5f949d2a","Type":"ContainerStarted","Data":"a097be29f3ff470550b4dbcb5148e73d2e6ca303080ba91af7a7b5ff2045d80d"} Nov 03 01:11:09.490691 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:11:09.489276 4939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-57079b-openshift-default-5c84d8878f-drtdm" event={"ID":"a68659b3-912d-4df9-a162-fa6d5f949d2a","Type":"ContainerStarted","Data":"ec1d9252517606456a366046a57c04a9b47dcc942a716dbd772dfbd9ad502835"} Nov 03 01:11:09.498028 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:11:09.498005 4939 scope.go:122] "RemoveContainer" containerID="d19447d0abfd950faea5858874c8ad314badd753800f42c9173c681a1eea80d6" Nov 03 01:11:09.518190 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:11:09.518142 4939 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-extmod-b-57079b-openshift-default-5c84d8878f-drtdm" podStartSLOduration=2.518126417 podStartE2EDuration="2.518126417s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 01:11:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 01:11:09.516424645 +0000 UTC m=+1817.780759589" watchObservedRunningTime="2026-11-03 01:11:09.518126417 +0000 UTC m=+1817.782461363" Nov 03 01:11:09.520709 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:11:09.520691 4939 scope.go:122] "RemoveContainer" containerID="988c3b00cfe1d42a496c5073f93a1ea2f634e32400f4bb974be56cfdede7ffa0" Nov 03 01:11:09.523835 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:11:09.523783 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:11:09.568400 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:11:09.568362 4939 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-f7680d-openshift-default-c4488d5c9-ldr7q"] Nov 03 01:11:09.578857 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:11:09.578809 4939 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-f7680d-openshift-default-c4488d5c9-ldr7q"] Nov 03 01:11:09.733683 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:11:09.733544 4939 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-83755a-openshift-default-c86f9565c-zj5jg"] Nov 03 01:11:09.855006 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:11:09.854969 4939 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-83755a-openshift-default-c86f9565c-zj5jg"] Nov 03 01:11:09.942407 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:11:09.942371 4939 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0dbecc93-5a61-4100-a319-d3f673f7a5fd" path="/var/lib/kubelet/pods/0dbecc93-5a61-4100-a319-d3f673f7a5fd/volumes" Nov 03 01:11:09.943573 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:11:09.943548 4939 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9b5bb111-d57c-478b-b990-db75ac2df998" path="/var/lib/kubelet/pods/9b5bb111-d57c-478b-b990-db75ac2df998/volumes" Nov 03 01:11:09.944441 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:11:09.944421 4939 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e08d7d14-b1fa-4fb2-b2d5-8c7fc073fc89" path="/var/lib/kubelet/pods/e08d7d14-b1fa-4fb2-b2d5-8c7fc073fc89/volumes" Nov 03 01:11:09.945416 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:11:09.945400 4939 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-83755a-openshift-default-c86f9565c-z7flk"] Nov 03 01:11:09.945520 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:11:09.945511 4939 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-83755a-openshift-default-c86f9565c-z7flk"] Nov 03 01:11:10.087277 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:11:10.087239 4939 patch_prober.go:29] interesting pod/e2e-shared-b-w3-556c14-openshift-default-754b5bc799-dt7h4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.215:15021/healthz/ready\": dial tcp 10.132.0.215:15021: connect: connection refused" start-of-body= Nov 03 01:11:10.087679 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:11:10.087309 4939 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-556c14-openshift-default-754b5bc799-dt7h4" podUID="fdeb6f74-c52a-4497-be8e-516b4244aefe" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.215:15021/healthz/ready\": dial tcp 10.132.0.215:15021: connect: connection refused" Nov 03 01:11:10.139979 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:11:10.139940 4939 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-extmod-b-57079b-openshift-default-5c84d8878f-drtdm" Nov 03 01:11:10.169934 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:11:10.151244 4939 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-extmod-b-57079b-openshift-default-5c84d8878f-drtdm" Nov 03 01:11:10.169934 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:11:10.167314 4939 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-4pw7v"] Nov 03 01:11:10.800146 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:11:10.800116 4939 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-b-57079b-openshift-default-5c84d8878f-drtdm" Nov 03 01:11:10.800456 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:11:10.800429 4939 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w3-556c14/auth-test-model-556c14-kserve-8d5cd469-t6tzc"] Nov 03 01:11:10.800701 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:11:10.800676 4939 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-extmod-b-57079b-openshift-default-5c84d8878f-drtdm" Nov 03 01:11:10.802780 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:11:10.802708 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:11:10.917658 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:11:10.917618 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:11:11.103116 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:11:11.101609 4939 patch_prober.go:29] interesting pod/e2e-shared-b-w3-556c14-openshift-default-754b5bc799-dt7h4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.215:15021/healthz/ready\": dial tcp 10.132.0.215:15021: connect: connection refused" start-of-body= Nov 03 01:11:11.103116 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:11:11.101694 4939 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-556c14-openshift-default-754b5bc799-dt7h4" podUID="fdeb6f74-c52a-4497-be8e-516b4244aefe" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.215:15021/healthz/ready\": dial tcp 10.132.0.215:15021: connect: connection refused" Nov 03 01:11:12.026106 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:11:12.015375 4939 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-f7680d/premium-w2-f7680d-kserve-76565b8cf9-hqbbt" podUID="4712e708-18a2-4a13-a12c-a5669397b2c3" containerName="main" probeResult="failure" output="Get \"https://10.132.0.63:8000/health\": context deadline exceeded" Nov 03 01:11:12.026106 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:11:12.015625 4939 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-f7680d/distinct-2-w2-f7680d-kserve-7d7c4564bf-cbv2c" podUID="2de18086-761b-4773-b9fc-15ddc4a220fc" containerName="main" probeResult="failure" output="Get \"https://10.132.0.75:8000/health\": context deadline exceeded" Nov 03 01:11:12.029765 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:11:12.029724 4939 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="569c2920-1523-4da8-8644-f789aebcdc7a" path="/var/lib/kubelet/pods/569c2920-1523-4da8-8644-f789aebcdc7a/volumes" Nov 03 01:11:12.035928 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:11:12.035878 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:11:12.054439 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:11:12.054303 4939 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-f7680d/distinct-w2-f7680d-kserve-756f874b98-qnpv8" podUID="836753fc-f9ef-4f61-b3fe-e5b92b7ee27b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.70:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 01:11:12.102935 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:11:12.088620 4939 patch_prober.go:29] interesting pod/e2e-shared-b-w3-556c14-openshift-default-754b5bc799-dt7h4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.215:15021/healthz/ready\": dial tcp 10.132.0.215:15021: connect: connection refused" start-of-body= Nov 03 01:11:12.102935 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:11:12.088701 4939 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-556c14-openshift-default-754b5bc799-dt7h4" podUID="fdeb6f74-c52a-4497-be8e-516b4244aefe" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.215:15021/healthz/ready\": dial tcp 10.132.0.215:15021: connect: connection refused" Nov 03 01:11:12.141935 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:11:12.132001 4939 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-88bbf7/simulator-w1-88bbf7-kserve-5fbb669f8f-42hgd" podUID="83b38883-2e7a-444d-aa68-52be4a25ae66" containerName="main" probeResult="failure" output="Get \"https://10.132.0.62:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Nov 03 01:11:12.187947 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:11:12.177454 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:11:12.312003 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:11:12.303360 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:11:12.400951 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:11:12.397023 4939 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-88bbf7/distinct-2-w1-88bbf7-kserve-5457cdcdbd-l6tfd" podUID="a397931b-faa9-4c2e-8290-491350546663" containerName="main" probeResult="failure" output="Get \"https://10.132.0.85:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Nov 03 01:11:12.721078 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:11:12.718965 4939 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-extmod-a-d25759-openshift-default-d47fd9785-jcn5l"] Nov 03 01:11:12.721078 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:11:12.720521 4939 cpu_manager.go:414] "containerMap: removing container" podUID="9b5bb111-d57c-478b-b990-db75ac2df998" containerName="istio-proxy" Nov 03 01:11:12.721078 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:11:12.720544 4939 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9b5bb111-d57c-478b-b990-db75ac2df998" containerName="istio-proxy" Nov 03 01:11:12.721078 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:11:12.720566 4939 cpu_manager.go:414] "containerMap: removing container" podUID="e08d7d14-b1fa-4fb2-b2d5-8c7fc073fc89" containerName="istio-proxy" Nov 03 01:11:12.721078 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:11:12.720574 4939 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e08d7d14-b1fa-4fb2-b2d5-8c7fc073fc89" containerName="istio-proxy" Nov 03 01:11:12.721078 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:11:12.720584 4939 cpu_manager.go:414] "containerMap: removing container" podUID="0dbecc93-5a61-4100-a319-d3f673f7a5fd" containerName="istio-proxy" Nov 03 01:11:12.721078 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:11:12.720593 4939 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0dbecc93-5a61-4100-a319-d3f673f7a5fd" containerName="istio-proxy" Nov 03 01:11:12.721078 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:11:12.720626 4939 cpu_manager.go:414] "containerMap: removing container" podUID="569c2920-1523-4da8-8644-f789aebcdc7a" containerName="istio-proxy" Nov 03 01:11:12.721078 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:11:12.720633 4939 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="569c2920-1523-4da8-8644-f789aebcdc7a" containerName="istio-proxy" Nov 03 01:11:12.721078 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:11:12.720859 4939 memory_manager.go:365] "RemoveStaleState removing state" podUID="9b5bb111-d57c-478b-b990-db75ac2df998" containerName="istio-proxy" Nov 03 01:11:12.721078 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:11:12.720871 4939 memory_manager.go:365] "RemoveStaleState removing state" podUID="e08d7d14-b1fa-4fb2-b2d5-8c7fc073fc89" containerName="istio-proxy" Nov 03 01:11:12.721078 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:11:12.720880 4939 memory_manager.go:365] "RemoveStaleState removing state" podUID="0dbecc93-5a61-4100-a319-d3f673f7a5fd" containerName="istio-proxy" Nov 03 01:11:12.721078 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:11:12.720896 4939 memory_manager.go:365] "RemoveStaleState removing state" podUID="569c2920-1523-4da8-8644-f789aebcdc7a" containerName="istio-proxy" Nov 03 01:11:12.731466 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:11:12.731264 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:11:12.735172 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:11:12.735145 4939 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-a-d25759-openshift-default-d47fd9785-jcn5l"] Nov 03 01:11:12.735336 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:11:12.735261 4939 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-d25759-openshift-default-d47fd9785-jcn5l" Nov 03 01:11:12.739291 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:11:12.739249 4939 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-extmod-a-d25759-openshift-default-dockercfg-8fmhw\"" Nov 03 01:11:12.739504 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:11:12.739478 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/93ab25a5-4bed-4d84-9d62-640cea8d4047-istio-envoy\") pod \"e2e-extmod-a-d25759-openshift-default-d47fd9785-jcn5l\" (UID: \"93ab25a5-4bed-4d84-9d62-640cea8d4047\") " pod="openshift-ingress/e2e-extmod-a-d25759-openshift-default-d47fd9785-jcn5l" Nov 03 01:11:12.739794 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:11:12.739769 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/93ab25a5-4bed-4d84-9d62-640cea8d4047-istio-data\") pod \"e2e-extmod-a-d25759-openshift-default-d47fd9785-jcn5l\" (UID: \"93ab25a5-4bed-4d84-9d62-640cea8d4047\") " pod="openshift-ingress/e2e-extmod-a-d25759-openshift-default-d47fd9785-jcn5l" Nov 03 01:11:12.740064 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:11:12.740041 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/93ab25a5-4bed-4d84-9d62-640cea8d4047-credential-socket\") pod \"e2e-extmod-a-d25759-openshift-default-d47fd9785-jcn5l\" (UID: \"93ab25a5-4bed-4d84-9d62-640cea8d4047\") " pod="openshift-ingress/e2e-extmod-a-d25759-openshift-default-d47fd9785-jcn5l" Nov 03 01:11:12.740225 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:11:12.740208 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/93ab25a5-4bed-4d84-9d62-640cea8d4047-istiod-ca-cert\") pod \"e2e-extmod-a-d25759-openshift-default-d47fd9785-jcn5l\" (UID: \"93ab25a5-4bed-4d84-9d62-640cea8d4047\") " pod="openshift-ingress/e2e-extmod-a-d25759-openshift-default-d47fd9785-jcn5l" Nov 03 01:11:12.740357 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:11:12.740341 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/93ab25a5-4bed-4d84-9d62-640cea8d4047-workload-certs\") pod \"e2e-extmod-a-d25759-openshift-default-d47fd9785-jcn5l\" (UID: \"93ab25a5-4bed-4d84-9d62-640cea8d4047\") " pod="openshift-ingress/e2e-extmod-a-d25759-openshift-default-d47fd9785-jcn5l" Nov 03 01:11:12.740560 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:11:12.740541 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6lbbx\" (UniqueName: \"kubernetes.io/projected/93ab25a5-4bed-4d84-9d62-640cea8d4047-kube-api-access-6lbbx\") pod \"e2e-extmod-a-d25759-openshift-default-d47fd9785-jcn5l\" (UID: \"93ab25a5-4bed-4d84-9d62-640cea8d4047\") " pod="openshift-ingress/e2e-extmod-a-d25759-openshift-default-d47fd9785-jcn5l" Nov 03 01:11:12.740807 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:11:12.740786 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/93ab25a5-4bed-4d84-9d62-640cea8d4047-istio-podinfo\") pod \"e2e-extmod-a-d25759-openshift-default-d47fd9785-jcn5l\" (UID: \"93ab25a5-4bed-4d84-9d62-640cea8d4047\") " pod="openshift-ingress/e2e-extmod-a-d25759-openshift-default-d47fd9785-jcn5l" Nov 03 01:11:12.741021 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:11:12.741001 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/93ab25a5-4bed-4d84-9d62-640cea8d4047-workload-socket\") pod \"e2e-extmod-a-d25759-openshift-default-d47fd9785-jcn5l\" (UID: \"93ab25a5-4bed-4d84-9d62-640cea8d4047\") " pod="openshift-ingress/e2e-extmod-a-d25759-openshift-default-d47fd9785-jcn5l" Nov 03 01:11:12.741169 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:11:12.741152 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/93ab25a5-4bed-4d84-9d62-640cea8d4047-istio-token\") pod \"e2e-extmod-a-d25759-openshift-default-d47fd9785-jcn5l\" (UID: \"93ab25a5-4bed-4d84-9d62-640cea8d4047\") " pod="openshift-ingress/e2e-extmod-a-d25759-openshift-default-d47fd9785-jcn5l" Nov 03 01:11:12.891934 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:11:12.887750 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6lbbx\" (UniqueName: \"kubernetes.io/projected/93ab25a5-4bed-4d84-9d62-640cea8d4047-kube-api-access-6lbbx\") pod \"e2e-extmod-a-d25759-openshift-default-d47fd9785-jcn5l\" (UID: \"93ab25a5-4bed-4d84-9d62-640cea8d4047\") " pod="openshift-ingress/e2e-extmod-a-d25759-openshift-default-d47fd9785-jcn5l" Nov 03 01:11:12.891934 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:11:12.887873 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/93ab25a5-4bed-4d84-9d62-640cea8d4047-istio-podinfo\") pod \"e2e-extmod-a-d25759-openshift-default-d47fd9785-jcn5l\" (UID: \"93ab25a5-4bed-4d84-9d62-640cea8d4047\") " pod="openshift-ingress/e2e-extmod-a-d25759-openshift-default-d47fd9785-jcn5l" Nov 03 01:11:12.891934 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:11:12.887973 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/93ab25a5-4bed-4d84-9d62-640cea8d4047-workload-socket\") pod \"e2e-extmod-a-d25759-openshift-default-d47fd9785-jcn5l\" (UID: \"93ab25a5-4bed-4d84-9d62-640cea8d4047\") " pod="openshift-ingress/e2e-extmod-a-d25759-openshift-default-d47fd9785-jcn5l" Nov 03 01:11:12.891934 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:11:12.888011 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/93ab25a5-4bed-4d84-9d62-640cea8d4047-istio-token\") pod \"e2e-extmod-a-d25759-openshift-default-d47fd9785-jcn5l\" (UID: \"93ab25a5-4bed-4d84-9d62-640cea8d4047\") " pod="openshift-ingress/e2e-extmod-a-d25759-openshift-default-d47fd9785-jcn5l" Nov 03 01:11:12.891934 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:11:12.888047 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/93ab25a5-4bed-4d84-9d62-640cea8d4047-istio-envoy\") pod \"e2e-extmod-a-d25759-openshift-default-d47fd9785-jcn5l\" (UID: \"93ab25a5-4bed-4d84-9d62-640cea8d4047\") " pod="openshift-ingress/e2e-extmod-a-d25759-openshift-default-d47fd9785-jcn5l" Nov 03 01:11:12.891934 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:11:12.888212 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/93ab25a5-4bed-4d84-9d62-640cea8d4047-istio-data\") pod \"e2e-extmod-a-d25759-openshift-default-d47fd9785-jcn5l\" (UID: \"93ab25a5-4bed-4d84-9d62-640cea8d4047\") " pod="openshift-ingress/e2e-extmod-a-d25759-openshift-default-d47fd9785-jcn5l" Nov 03 01:11:12.891934 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:11:12.888346 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/93ab25a5-4bed-4d84-9d62-640cea8d4047-credential-socket\") pod \"e2e-extmod-a-d25759-openshift-default-d47fd9785-jcn5l\" (UID: \"93ab25a5-4bed-4d84-9d62-640cea8d4047\") " pod="openshift-ingress/e2e-extmod-a-d25759-openshift-default-d47fd9785-jcn5l" Nov 03 01:11:12.891934 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:11:12.888398 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/93ab25a5-4bed-4d84-9d62-640cea8d4047-istiod-ca-cert\") pod \"e2e-extmod-a-d25759-openshift-default-d47fd9785-jcn5l\" (UID: \"93ab25a5-4bed-4d84-9d62-640cea8d4047\") " pod="openshift-ingress/e2e-extmod-a-d25759-openshift-default-d47fd9785-jcn5l" Nov 03 01:11:12.891934 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:11:12.888446 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/93ab25a5-4bed-4d84-9d62-640cea8d4047-workload-certs\") pod \"e2e-extmod-a-d25759-openshift-default-d47fd9785-jcn5l\" (UID: \"93ab25a5-4bed-4d84-9d62-640cea8d4047\") " pod="openshift-ingress/e2e-extmod-a-d25759-openshift-default-d47fd9785-jcn5l" Nov 03 01:11:12.891934 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:11:12.889119 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/93ab25a5-4bed-4d84-9d62-640cea8d4047-workload-certs\") pod \"e2e-extmod-a-d25759-openshift-default-d47fd9785-jcn5l\" (UID: \"93ab25a5-4bed-4d84-9d62-640cea8d4047\") " pod="openshift-ingress/e2e-extmod-a-d25759-openshift-default-d47fd9785-jcn5l" Nov 03 01:11:12.947942 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:11:12.938338 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/93ab25a5-4bed-4d84-9d62-640cea8d4047-workload-socket\") pod \"e2e-extmod-a-d25759-openshift-default-d47fd9785-jcn5l\" (UID: \"93ab25a5-4bed-4d84-9d62-640cea8d4047\") " pod="openshift-ingress/e2e-extmod-a-d25759-openshift-default-d47fd9785-jcn5l" Nov 03 01:11:12.947942 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:11:12.938609 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/93ab25a5-4bed-4d84-9d62-640cea8d4047-istio-data\") pod \"e2e-extmod-a-d25759-openshift-default-d47fd9785-jcn5l\" (UID: \"93ab25a5-4bed-4d84-9d62-640cea8d4047\") " pod="openshift-ingress/e2e-extmod-a-d25759-openshift-default-d47fd9785-jcn5l" Nov 03 01:11:12.947942 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:11:12.939218 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/93ab25a5-4bed-4d84-9d62-640cea8d4047-credential-socket\") pod \"e2e-extmod-a-d25759-openshift-default-d47fd9785-jcn5l\" (UID: \"93ab25a5-4bed-4d84-9d62-640cea8d4047\") " pod="openshift-ingress/e2e-extmod-a-d25759-openshift-default-d47fd9785-jcn5l" Nov 03 01:11:13.006014 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:11:13.005806 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/93ab25a5-4bed-4d84-9d62-640cea8d4047-istiod-ca-cert\") pod \"e2e-extmod-a-d25759-openshift-default-d47fd9785-jcn5l\" (UID: \"93ab25a5-4bed-4d84-9d62-640cea8d4047\") " pod="openshift-ingress/e2e-extmod-a-d25759-openshift-default-d47fd9785-jcn5l" Nov 03 01:11:13.161935 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:11:13.122350 4939 patch_prober.go:29] interesting pod/e2e-shared-b-w3-556c14-openshift-default-754b5bc799-dt7h4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.215:15021/healthz/ready\": dial tcp 10.132.0.215:15021: connect: connection refused" start-of-body= Nov 03 01:11:13.161935 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:11:13.122414 4939 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-556c14-openshift-default-754b5bc799-dt7h4" podUID="fdeb6f74-c52a-4497-be8e-516b4244aefe" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.215:15021/healthz/ready\": dial tcp 10.132.0.215:15021: connect: connection refused" Nov 03 01:11:13.263990 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:11:13.218803 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/93ab25a5-4bed-4d84-9d62-640cea8d4047-istio-envoy\") pod \"e2e-extmod-a-d25759-openshift-default-d47fd9785-jcn5l\" (UID: \"93ab25a5-4bed-4d84-9d62-640cea8d4047\") " pod="openshift-ingress/e2e-extmod-a-d25759-openshift-default-d47fd9785-jcn5l" Nov 03 01:11:13.263990 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:11:13.235057 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/93ab25a5-4bed-4d84-9d62-640cea8d4047-istio-podinfo\") pod \"e2e-extmod-a-d25759-openshift-default-d47fd9785-jcn5l\" (UID: \"93ab25a5-4bed-4d84-9d62-640cea8d4047\") " pod="openshift-ingress/e2e-extmod-a-d25759-openshift-default-d47fd9785-jcn5l" Nov 03 01:11:13.263990 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:11:13.254524 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/93ab25a5-4bed-4d84-9d62-640cea8d4047-istio-token\") pod \"e2e-extmod-a-d25759-openshift-default-d47fd9785-jcn5l\" (UID: \"93ab25a5-4bed-4d84-9d62-640cea8d4047\") " pod="openshift-ingress/e2e-extmod-a-d25759-openshift-default-d47fd9785-jcn5l" Nov 03 01:11:13.293946 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:11:13.265044 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6lbbx\" (UniqueName: \"kubernetes.io/projected/93ab25a5-4bed-4d84-9d62-640cea8d4047-kube-api-access-6lbbx\") pod \"e2e-extmod-a-d25759-openshift-default-d47fd9785-jcn5l\" (UID: \"93ab25a5-4bed-4d84-9d62-640cea8d4047\") " pod="openshift-ingress/e2e-extmod-a-d25759-openshift-default-d47fd9785-jcn5l" Nov 03 01:11:13.328320 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:11:13.328285 4939 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-d25759-openshift-default-d47fd9785-jcn5l" Nov 03 01:11:13.384902 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:11:13.384697 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:11:13.938820 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:11:13.934888 4939 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-2f9a3a/simulator-w0-2f9a3a-kserve-9596bbb8f-2trgz" podUID="207e2207-fb6c-4076-95df-72e95671c261" containerName="main" probeResult="failure" output="Get \"https://10.132.0.61:8000/ready\": context deadline exceeded" Nov 03 01:11:14.072938 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:11:14.070814 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:11:14.105660 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:11:14.105620 4939 patch_prober.go:29] interesting pod/e2e-shared-b-w3-556c14-openshift-default-754b5bc799-dt7h4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.215:15021/healthz/ready\": dial tcp 10.132.0.215:15021: connect: connection refused" start-of-body= Nov 03 01:11:14.105971 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:11:14.105935 4939 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-556c14-openshift-default-754b5bc799-dt7h4" podUID="fdeb6f74-c52a-4497-be8e-516b4244aefe" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.215:15021/healthz/ready\": dial tcp 10.132.0.215:15021: connect: connection refused" Nov 03 01:11:14.231991 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:11:14.223864 4939 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-88bbf7/simulator-w1-88bbf7-kserve-5fbb669f8f-42hgd" podUID="83b38883-2e7a-444d-aa68-52be4a25ae66" containerName="main" probeResult="failure" output="Get \"https://10.132.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 01:11:14.318933 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:11:14.316556 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:11:14.508262 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:11:14.496974 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:11:14.643335 ip-10-0-140-107 kubenswrapper[4939]: W1103 01:11:14.643291 4939 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod93ab25a5_4bed_4d84_9d62_640cea8d4047.slice/crio-fb003dd49ca60fac8ee17dceebe96fa5a85be971b8b81e0cbd8bcf49e725914b WatchSource:0}: Error finding container fb003dd49ca60fac8ee17dceebe96fa5a85be971b8b81e0cbd8bcf49e725914b: Status 404 returned error can't find the container with id fb003dd49ca60fac8ee17dceebe96fa5a85be971b8b81e0cbd8bcf49e725914b Nov 03 01:11:14.813173 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:11:14.807581 4939 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 01:11:14.813173 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:11:14.807683 4939 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 01:11:14.813173 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:11:14.807726 4939 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 01:11:14.937077 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:11:14.937026 4939 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-2f9a3a/simulator-w0-2f9a3a-kserve-9596bbb8f-2trgz" podUID="207e2207-fb6c-4076-95df-72e95671c261" containerName="main" probeResult="failure" output="Get \"https://10.132.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 01:11:15.107835 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:11:15.032629 4939 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-f7680d/premium-w2-f7680d-kserve-76565b8cf9-hqbbt" podUID="4712e708-18a2-4a13-a12c-a5669397b2c3" containerName="main" probeResult="failure" output="Get \"https://10.132.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 01:11:15.117324 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:11:15.096993 4939 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-f7680d/distinct-w2-f7680d-kserve-756f874b98-qnpv8" podUID="836753fc-f9ef-4f61-b3fe-e5b92b7ee27b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.70:8000/ready\": context deadline exceeded" Nov 03 01:11:15.117324 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:11:15.105646 4939 patch_prober.go:29] interesting pod/e2e-shared-b-w3-556c14-openshift-default-754b5bc799-dt7h4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.215:15021/healthz/ready\": dial tcp 10.132.0.215:15021: connect: connection refused" start-of-body= Nov 03 01:11:15.117324 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:11:15.114146 4939 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-556c14-openshift-default-754b5bc799-dt7h4" podUID="fdeb6f74-c52a-4497-be8e-516b4244aefe" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.215:15021/healthz/ready\": dial tcp 10.132.0.215:15021: connect: connection refused" Nov 03 01:11:15.301936 ip-10-0-140-107 kubenswrapper[4939]: E1103 01:11:15.241739 4939 kubelet.go:2711] "Housekeeping took longer than expected" err="housekeeping took too long" expected="1s" actual="1.009s" Nov 03 01:11:15.301936 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:11:15.241839 4939 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-2f9a3a/simulator-w0-2f9a3a-kserve-9596bbb8f-2trgz" Nov 03 01:11:15.301936 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:11:15.241885 4939 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-b-57079b-5f4958f857-v8cxd"] Nov 03 01:11:15.320658 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:11:15.306085 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:11:15.347763 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:11:15.347731 4939 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-57079b-666f95c947-dv7mr"] Nov 03 01:11:15.457872 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:11:15.376370 4939 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-b-57079b-5f4958f857-v8cxd" Nov 03 01:11:15.460364 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:11:15.460329 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5d25771d-8931-4840-86be-9ebaf92fafca-plugins-config-volume\") pod \"payload-processing-e2e-extmod-b-57079b-5f4958f857-v8cxd\" (UID: \"5d25771d-8931-4840-86be-9ebaf92fafca\") " pod="openshift-ingress/payload-processing-e2e-extmod-b-57079b-5f4958f857-v8cxd" Nov 03 01:11:15.461469 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:11:15.461442 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qzwpl\" (UniqueName: \"kubernetes.io/projected/5d25771d-8931-4840-86be-9ebaf92fafca-kube-api-access-qzwpl\") pod \"payload-processing-e2e-extmod-b-57079b-5f4958f857-v8cxd\" (UID: \"5d25771d-8931-4840-86be-9ebaf92fafca\") " pod="openshift-ingress/payload-processing-e2e-extmod-b-57079b-5f4958f857-v8cxd" Nov 03 01:11:15.469695 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:11:15.469669 4939 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-extmod-b-57079b-dockercfg-fn2sq\"" Nov 03 01:11:15.502400 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:11:15.502352 4939 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-extmod-b-57079b\"" Nov 03 01:11:15.639942 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:11:15.637553 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qzwpl\" (UniqueName: \"kubernetes.io/projected/5d25771d-8931-4840-86be-9ebaf92fafca-kube-api-access-qzwpl\") pod \"payload-processing-e2e-extmod-b-57079b-5f4958f857-v8cxd\" (UID: \"5d25771d-8931-4840-86be-9ebaf92fafca\") " pod="openshift-ingress/payload-processing-e2e-extmod-b-57079b-5f4958f857-v8cxd" Nov 03 01:11:15.639942 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:11:15.637753 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5d25771d-8931-4840-86be-9ebaf92fafca-plugins-config-volume\") pod \"payload-processing-e2e-extmod-b-57079b-5f4958f857-v8cxd\" (UID: \"5d25771d-8931-4840-86be-9ebaf92fafca\") " pod="openshift-ingress/payload-processing-e2e-extmod-b-57079b-5f4958f857-v8cxd" Nov 03 01:11:15.639942 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:11:15.639144 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5d25771d-8931-4840-86be-9ebaf92fafca-plugins-config-volume\") pod \"payload-processing-e2e-extmod-b-57079b-5f4958f857-v8cxd\" (UID: \"5d25771d-8931-4840-86be-9ebaf92fafca\") " pod="openshift-ingress/payload-processing-e2e-extmod-b-57079b-5f4958f857-v8cxd" Nov 03 01:11:15.692038 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:11:15.682804 4939 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-f7680d/distinct-2-w2-f7680d-kserve-7d7c4564bf-cbv2c" podUID="2de18086-761b-4773-b9fc-15ddc4a220fc" containerName="main" probeResult="failure" output="Get \"https://10.132.0.75:8000/ready\": context deadline exceeded" Nov 03 01:11:15.709900 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:11:15.705043 4939 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-shared-b-w3-556c14/auth-test-model-556c14-kserve-8d5cd469-t6tzc" podUID="0c4e579c-9fe2-4ec9-b8f4-62758e519980" containerName="main" probeResult="failure" output="Get \"https://10.132.0.195:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 01:11:15.711315 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:11:15.711272 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qzwpl\" (UniqueName: \"kubernetes.io/projected/5d25771d-8931-4840-86be-9ebaf92fafca-kube-api-access-qzwpl\") pod \"payload-processing-e2e-extmod-b-57079b-5f4958f857-v8cxd\" (UID: \"5d25771d-8931-4840-86be-9ebaf92fafca\") " pod="openshift-ingress/payload-processing-e2e-extmod-b-57079b-5f4958f857-v8cxd" Nov 03 01:11:15.737944 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:11:15.715540 4939 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-57079b-666f95c947-dv7mr"] Nov 03 01:11:15.738279 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:11:15.738254 4939 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-b-57079b-5f4958f857-v8cxd"] Nov 03 01:11:15.738408 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:11:15.738397 4939 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-a-d25759-openshift-default-d47fd9785-jcn5l"] Nov 03 01:11:15.738958 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:11:15.718167 4939 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-57079b-666f95c947-dv7mr" Nov 03 01:11:15.742303 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:11:15.742278 4939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-d25759-openshift-default-d47fd9785-jcn5l" event={"ID":"93ab25a5-4bed-4d84-9d62-640cea8d4047","Type":"ContainerStarted","Data":"fb003dd49ca60fac8ee17dceebe96fa5a85be971b8b81e0cbd8bcf49e725914b"} Nov 03 01:11:15.817024 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:11:15.816989 4939 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-2f9a3a/simulator-w0-2f9a3a-kserve-9596bbb8f-2trgz" Nov 03 01:11:15.832832 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:11:15.831767 4939 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-b-57079b-5f4958f857-v8cxd" Nov 03 01:11:15.845290 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:11:15.844198 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a79df098-23e4-4971-9263-3f52b7d25b0b-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-b-57079b-666f95c947-dv7mr\" (UID: \"a79df098-23e4-4971-9263-3f52b7d25b0b\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-57079b-666f95c947-dv7mr" Nov 03 01:11:15.845290 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:11:15.844350 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-29ptd\" (UniqueName: \"kubernetes.io/projected/a79df098-23e4-4971-9263-3f52b7d25b0b-kube-api-access-29ptd\") pod \"payload-pre-processing-e2e-extmod-b-57079b-666f95c947-dv7mr\" (UID: \"a79df098-23e4-4971-9263-3f52b7d25b0b\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-57079b-666f95c947-dv7mr" Nov 03 01:11:16.008205 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:11:15.947049 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a79df098-23e4-4971-9263-3f52b7d25b0b-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-b-57079b-666f95c947-dv7mr\" (UID: \"a79df098-23e4-4971-9263-3f52b7d25b0b\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-57079b-666f95c947-dv7mr" Nov 03 01:11:16.008205 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:11:15.947169 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-29ptd\" (UniqueName: \"kubernetes.io/projected/a79df098-23e4-4971-9263-3f52b7d25b0b-kube-api-access-29ptd\") pod \"payload-pre-processing-e2e-extmod-b-57079b-666f95c947-dv7mr\" (UID: \"a79df098-23e4-4971-9263-3f52b7d25b0b\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-57079b-666f95c947-dv7mr" Nov 03 01:11:16.046664 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:11:15.948901 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a79df098-23e4-4971-9263-3f52b7d25b0b-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-b-57079b-666f95c947-dv7mr\" (UID: \"a79df098-23e4-4971-9263-3f52b7d25b0b\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-57079b-666f95c947-dv7mr" Nov 03 01:11:16.185280 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:11:16.185237 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-29ptd\" (UniqueName: \"kubernetes.io/projected/a79df098-23e4-4971-9263-3f52b7d25b0b-kube-api-access-29ptd\") pod \"payload-pre-processing-e2e-extmod-b-57079b-666f95c947-dv7mr\" (UID: \"a79df098-23e4-4971-9263-3f52b7d25b0b\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-57079b-666f95c947-dv7mr" Nov 03 01:11:16.406182 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:11:16.375465 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:11:16.419413 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:11:16.419376 4939 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-57079b-666f95c947-dv7mr" Nov 03 01:11:16.540931 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:11:16.512369 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:11:16.657939 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:11:16.630622 4939 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w3-556c14-openshift-default-754b5bc799-dt7h4" Nov 03 01:11:16.664932 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:11:16.660974 4939 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w3-556c14-openshift-default-754b5bc799-dt7h4" Nov 03 01:11:16.734228 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:11:16.719505 4939 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w3-556c14-openshift-default-754b5bc799-dt7h4" Nov 03 01:11:16.824335 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:11:16.824298 4939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-d25759-openshift-default-d47fd9785-jcn5l" event={"ID":"93ab25a5-4bed-4d84-9d62-640cea8d4047","Type":"ContainerStarted","Data":"e6d4ddc1ac83d9addd06be80c75b25054577015f20147ea1379903004987cb75"} Nov 03 01:11:17.244880 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:11:17.243602 4939 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-extmod-a-d25759-openshift-default-d47fd9785-jcn5l" podStartSLOduration=5.243577728 podStartE2EDuration="5.243577728s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 01:11:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 01:11:16.934541816 +0000 UTC m=+1825.198876763" watchObservedRunningTime="2026-11-03 01:11:17.243577728 +0000 UTC m=+1825.507912675" Nov 03 01:11:17.329673 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:11:17.329555 4939 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-extmod-a-d25759-openshift-default-d47fd9785-jcn5l" Nov 03 01:11:17.331697 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:11:17.331674 4939 patch_prober.go:29] interesting pod/e2e-extmod-a-d25759-openshift-default-d47fd9785-jcn5l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.219:15021/healthz/ready\": dial tcp 10.132.0.219:15021: connect: connection refused" start-of-body= Nov 03 01:11:17.336124 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:11:17.336091 4939 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-d25759-openshift-default-d47fd9785-jcn5l" podUID="93ab25a5-4bed-4d84-9d62-640cea8d4047" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.219:15021/healthz/ready\": dial tcp 10.132.0.219:15021: connect: connection refused" Nov 03 01:11:17.343004 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:11:17.342488 4939 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-b-57079b-5f4958f857-v8cxd"] Nov 03 01:11:17.421842 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:11:17.421809 4939 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-extmod-b-3c5983-7fbf586cf5-92rtm" Nov 03 01:11:17.602761 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:11:17.589167 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:11:17.791158 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:11:17.787279 4939 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-57079b-666f95c947-dv7mr"] Nov 03 01:11:17.842837 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:11:17.842792 4939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-b-57079b-5f4958f857-v8cxd" event={"ID":"5d25771d-8931-4840-86be-9ebaf92fafca","Type":"ContainerStarted","Data":"5101ed9255aef93f6e336f9779a488084a365d3549eec894f950371700ef407b"} Nov 03 01:11:17.906986 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:11:17.906183 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:11:18.045110 ip-10-0-140-107 kubenswrapper[4939]: E1103 01:11:18.045077 4939 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-sthjh" containerName="extractor" Nov 03 01:11:18.240844 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:11:18.240755 4939 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-extmod-a-ddd653-openshift-default-779ff6465-t9cvw"] Nov 03 01:11:18.282934 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:11:18.279235 4939 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-ddd653-openshift-default-779ff6465-t9cvw" Nov 03 01:11:18.309943 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:11:18.307890 4939 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-extmod-a-ddd653-openshift-default-dockercfg-zx7f4\"" Nov 03 01:11:18.309943 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:11:18.309881 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:11:18.337655 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:11:18.334177 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jbgtv\" (UniqueName: \"kubernetes.io/projected/aa64537e-0c5b-49ae-9547-df65d9ea4b2d-kube-api-access-jbgtv\") pod \"e2e-extmod-a-ddd653-openshift-default-779ff6465-t9cvw\" (UID: \"aa64537e-0c5b-49ae-9547-df65d9ea4b2d\") " pod="openshift-ingress/e2e-extmod-a-ddd653-openshift-default-779ff6465-t9cvw" Nov 03 01:11:18.337655 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:11:18.334299 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/aa64537e-0c5b-49ae-9547-df65d9ea4b2d-istiod-ca-cert\") pod \"e2e-extmod-a-ddd653-openshift-default-779ff6465-t9cvw\" (UID: \"aa64537e-0c5b-49ae-9547-df65d9ea4b2d\") " pod="openshift-ingress/e2e-extmod-a-ddd653-openshift-default-779ff6465-t9cvw" Nov 03 01:11:18.337655 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:11:18.334333 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/aa64537e-0c5b-49ae-9547-df65d9ea4b2d-workload-certs\") pod \"e2e-extmod-a-ddd653-openshift-default-779ff6465-t9cvw\" (UID: \"aa64537e-0c5b-49ae-9547-df65d9ea4b2d\") " pod="openshift-ingress/e2e-extmod-a-ddd653-openshift-default-779ff6465-t9cvw" Nov 03 01:11:18.337655 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:11:18.334446 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/aa64537e-0c5b-49ae-9547-df65d9ea4b2d-istio-envoy\") pod \"e2e-extmod-a-ddd653-openshift-default-779ff6465-t9cvw\" (UID: \"aa64537e-0c5b-49ae-9547-df65d9ea4b2d\") " pod="openshift-ingress/e2e-extmod-a-ddd653-openshift-default-779ff6465-t9cvw" Nov 03 01:11:18.337655 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:11:18.334481 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/aa64537e-0c5b-49ae-9547-df65d9ea4b2d-credential-socket\") pod \"e2e-extmod-a-ddd653-openshift-default-779ff6465-t9cvw\" (UID: \"aa64537e-0c5b-49ae-9547-df65d9ea4b2d\") " pod="openshift-ingress/e2e-extmod-a-ddd653-openshift-default-779ff6465-t9cvw" Nov 03 01:11:18.337655 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:11:18.334558 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/aa64537e-0c5b-49ae-9547-df65d9ea4b2d-istio-token\") pod \"e2e-extmod-a-ddd653-openshift-default-779ff6465-t9cvw\" (UID: \"aa64537e-0c5b-49ae-9547-df65d9ea4b2d\") " pod="openshift-ingress/e2e-extmod-a-ddd653-openshift-default-779ff6465-t9cvw" Nov 03 01:11:18.337655 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:11:18.334648 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/aa64537e-0c5b-49ae-9547-df65d9ea4b2d-workload-socket\") pod \"e2e-extmod-a-ddd653-openshift-default-779ff6465-t9cvw\" (UID: \"aa64537e-0c5b-49ae-9547-df65d9ea4b2d\") " pod="openshift-ingress/e2e-extmod-a-ddd653-openshift-default-779ff6465-t9cvw" Nov 03 01:11:18.337655 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:11:18.334737 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/aa64537e-0c5b-49ae-9547-df65d9ea4b2d-istio-podinfo\") pod \"e2e-extmod-a-ddd653-openshift-default-779ff6465-t9cvw\" (UID: \"aa64537e-0c5b-49ae-9547-df65d9ea4b2d\") " pod="openshift-ingress/e2e-extmod-a-ddd653-openshift-default-779ff6465-t9cvw" Nov 03 01:11:18.337655 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:11:18.334860 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/aa64537e-0c5b-49ae-9547-df65d9ea4b2d-istio-data\") pod \"e2e-extmod-a-ddd653-openshift-default-779ff6465-t9cvw\" (UID: \"aa64537e-0c5b-49ae-9547-df65d9ea4b2d\") " pod="openshift-ingress/e2e-extmod-a-ddd653-openshift-default-779ff6465-t9cvw" Nov 03 01:11:18.337655 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:11:18.335155 4939 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-a-ddd653-openshift-default-779ff6465-t9cvw"] Nov 03 01:11:18.342122 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:11:18.341781 4939 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-extmod-a-d25759-openshift-default-d47fd9785-jcn5l" Nov 03 01:11:18.485932 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:11:18.463268 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/aa64537e-0c5b-49ae-9547-df65d9ea4b2d-istio-data\") pod \"e2e-extmod-a-ddd653-openshift-default-779ff6465-t9cvw\" (UID: \"aa64537e-0c5b-49ae-9547-df65d9ea4b2d\") " pod="openshift-ingress/e2e-extmod-a-ddd653-openshift-default-779ff6465-t9cvw" Nov 03 01:11:18.698228 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:11:18.666797 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jbgtv\" (UniqueName: \"kubernetes.io/projected/aa64537e-0c5b-49ae-9547-df65d9ea4b2d-kube-api-access-jbgtv\") pod \"e2e-extmod-a-ddd653-openshift-default-779ff6465-t9cvw\" (UID: \"aa64537e-0c5b-49ae-9547-df65d9ea4b2d\") " pod="openshift-ingress/e2e-extmod-a-ddd653-openshift-default-779ff6465-t9cvw" Nov 03 01:11:18.698228 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:11:18.667680 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/aa64537e-0c5b-49ae-9547-df65d9ea4b2d-istiod-ca-cert\") pod \"e2e-extmod-a-ddd653-openshift-default-779ff6465-t9cvw\" (UID: \"aa64537e-0c5b-49ae-9547-df65d9ea4b2d\") " pod="openshift-ingress/e2e-extmod-a-ddd653-openshift-default-779ff6465-t9cvw" Nov 03 01:11:18.698228 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:11:18.488166 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/aa64537e-0c5b-49ae-9547-df65d9ea4b2d-istio-data\") pod \"e2e-extmod-a-ddd653-openshift-default-779ff6465-t9cvw\" (UID: \"aa64537e-0c5b-49ae-9547-df65d9ea4b2d\") " pod="openshift-ingress/e2e-extmod-a-ddd653-openshift-default-779ff6465-t9cvw" Nov 03 01:11:18.698228 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:11:18.576133 4939 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-3c5983-564977cd99-qs5fk" Nov 03 01:11:18.698228 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:11:18.669122 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/aa64537e-0c5b-49ae-9547-df65d9ea4b2d-istiod-ca-cert\") pod \"e2e-extmod-a-ddd653-openshift-default-779ff6465-t9cvw\" (UID: \"aa64537e-0c5b-49ae-9547-df65d9ea4b2d\") " pod="openshift-ingress/e2e-extmod-a-ddd653-openshift-default-779ff6465-t9cvw" Nov 03 01:11:18.698228 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:11:18.669169 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/aa64537e-0c5b-49ae-9547-df65d9ea4b2d-workload-certs\") pod \"e2e-extmod-a-ddd653-openshift-default-779ff6465-t9cvw\" (UID: \"aa64537e-0c5b-49ae-9547-df65d9ea4b2d\") " pod="openshift-ingress/e2e-extmod-a-ddd653-openshift-default-779ff6465-t9cvw" Nov 03 01:11:18.698228 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:11:18.669267 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/aa64537e-0c5b-49ae-9547-df65d9ea4b2d-istio-envoy\") pod \"e2e-extmod-a-ddd653-openshift-default-779ff6465-t9cvw\" (UID: \"aa64537e-0c5b-49ae-9547-df65d9ea4b2d\") " pod="openshift-ingress/e2e-extmod-a-ddd653-openshift-default-779ff6465-t9cvw" Nov 03 01:11:18.698228 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:11:18.669299 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/aa64537e-0c5b-49ae-9547-df65d9ea4b2d-credential-socket\") pod \"e2e-extmod-a-ddd653-openshift-default-779ff6465-t9cvw\" (UID: \"aa64537e-0c5b-49ae-9547-df65d9ea4b2d\") " pod="openshift-ingress/e2e-extmod-a-ddd653-openshift-default-779ff6465-t9cvw" Nov 03 01:11:18.698228 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:11:18.669362 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/aa64537e-0c5b-49ae-9547-df65d9ea4b2d-istio-token\") pod \"e2e-extmod-a-ddd653-openshift-default-779ff6465-t9cvw\" (UID: \"aa64537e-0c5b-49ae-9547-df65d9ea4b2d\") " pod="openshift-ingress/e2e-extmod-a-ddd653-openshift-default-779ff6465-t9cvw" Nov 03 01:11:18.698228 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:11:18.669439 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/aa64537e-0c5b-49ae-9547-df65d9ea4b2d-workload-socket\") pod \"e2e-extmod-a-ddd653-openshift-default-779ff6465-t9cvw\" (UID: \"aa64537e-0c5b-49ae-9547-df65d9ea4b2d\") " pod="openshift-ingress/e2e-extmod-a-ddd653-openshift-default-779ff6465-t9cvw" Nov 03 01:11:18.698228 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:11:18.669515 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/aa64537e-0c5b-49ae-9547-df65d9ea4b2d-istio-podinfo\") pod \"e2e-extmod-a-ddd653-openshift-default-779ff6465-t9cvw\" (UID: \"aa64537e-0c5b-49ae-9547-df65d9ea4b2d\") " pod="openshift-ingress/e2e-extmod-a-ddd653-openshift-default-779ff6465-t9cvw" Nov 03 01:11:18.698228 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:11:18.671133 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/aa64537e-0c5b-49ae-9547-df65d9ea4b2d-credential-socket\") pod \"e2e-extmod-a-ddd653-openshift-default-779ff6465-t9cvw\" (UID: \"aa64537e-0c5b-49ae-9547-df65d9ea4b2d\") " pod="openshift-ingress/e2e-extmod-a-ddd653-openshift-default-779ff6465-t9cvw" Nov 03 01:11:18.698228 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:11:18.671363 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/aa64537e-0c5b-49ae-9547-df65d9ea4b2d-workload-certs\") pod \"e2e-extmod-a-ddd653-openshift-default-779ff6465-t9cvw\" (UID: \"aa64537e-0c5b-49ae-9547-df65d9ea4b2d\") " pod="openshift-ingress/e2e-extmod-a-ddd653-openshift-default-779ff6465-t9cvw" Nov 03 01:11:18.698228 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:11:18.675777 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/aa64537e-0c5b-49ae-9547-df65d9ea4b2d-workload-socket\") pod \"e2e-extmod-a-ddd653-openshift-default-779ff6465-t9cvw\" (UID: \"aa64537e-0c5b-49ae-9547-df65d9ea4b2d\") " pod="openshift-ingress/e2e-extmod-a-ddd653-openshift-default-779ff6465-t9cvw" Nov 03 01:11:18.996704 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:11:18.996622 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:11:18.997678 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:11:18.997460 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/aa64537e-0c5b-49ae-9547-df65d9ea4b2d-istio-envoy\") pod \"e2e-extmod-a-ddd653-openshift-default-779ff6465-t9cvw\" (UID: \"aa64537e-0c5b-49ae-9547-df65d9ea4b2d\") " pod="openshift-ingress/e2e-extmod-a-ddd653-openshift-default-779ff6465-t9cvw" Nov 03 01:11:18.997678 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:11:18.997632 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/aa64537e-0c5b-49ae-9547-df65d9ea4b2d-istio-token\") pod \"e2e-extmod-a-ddd653-openshift-default-779ff6465-t9cvw\" (UID: \"aa64537e-0c5b-49ae-9547-df65d9ea4b2d\") " pod="openshift-ingress/e2e-extmod-a-ddd653-openshift-default-779ff6465-t9cvw" Nov 03 01:11:18.998527 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:11:18.998477 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/aa64537e-0c5b-49ae-9547-df65d9ea4b2d-istio-podinfo\") pod \"e2e-extmod-a-ddd653-openshift-default-779ff6465-t9cvw\" (UID: \"aa64537e-0c5b-49ae-9547-df65d9ea4b2d\") " pod="openshift-ingress/e2e-extmod-a-ddd653-openshift-default-779ff6465-t9cvw" Nov 03 01:11:19.025778 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:11:19.014829 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jbgtv\" (UniqueName: \"kubernetes.io/projected/aa64537e-0c5b-49ae-9547-df65d9ea4b2d-kube-api-access-jbgtv\") pod \"e2e-extmod-a-ddd653-openshift-default-779ff6465-t9cvw\" (UID: \"aa64537e-0c5b-49ae-9547-df65d9ea4b2d\") " pod="openshift-ingress/e2e-extmod-a-ddd653-openshift-default-779ff6465-t9cvw" Nov 03 01:11:19.045007 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:11:19.043409 4939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-57079b-666f95c947-dv7mr" event={"ID":"a79df098-23e4-4971-9263-3f52b7d25b0b","Type":"ContainerStarted","Data":"7ba5aaa0f3337d62777e32bc996b60a0573018aafc9e436e464208aeb016bc29"} Nov 03 01:11:19.045007 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:11:19.043483 4939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-57079b-666f95c947-dv7mr" event={"ID":"a79df098-23e4-4971-9263-3f52b7d25b0b","Type":"ContainerStarted","Data":"1a5d340aead8d2aad8850271e122f1d1e8ffce58e7b4e21bea5299e64e183d5c"} Nov 03 01:11:19.045007 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:11:19.044944 4939 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-57079b-666f95c947-dv7mr" Nov 03 01:11:19.054462 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:11:19.054425 4939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-b-57079b-5f4958f857-v8cxd" event={"ID":"5d25771d-8931-4840-86be-9ebaf92fafca","Type":"ContainerStarted","Data":"efe834ec176f778ad2632569bbf778eb71e02e0117cf2037200131ba3869f8fc"} Nov 03 01:11:19.054719 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:11:19.054700 4939 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-a-d25759-openshift-default-d47fd9785-jcn5l" Nov 03 01:11:19.054866 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:11:19.054853 4939 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-extmod-b-57079b-5f4958f857-v8cxd" Nov 03 01:11:19.057278 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:11:19.057230 4939 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-extmod-a-d25759-openshift-default-d47fd9785-jcn5l" Nov 03 01:11:19.073925 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:11:19.073843 4939 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-57079b-666f95c947-dv7mr" podStartSLOduration=5.073823043 podStartE2EDuration="5.073823043s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 01:11:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 01:11:19.063254804 +0000 UTC m=+1827.327589750" watchObservedRunningTime="2026-11-03 01:11:19.073823043 +0000 UTC m=+1827.338157989" Nov 03 01:11:19.128863 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:11:19.128795 4939 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-extmod-b-57079b-5f4958f857-v8cxd" podStartSLOduration=5.128770476 podStartE2EDuration="5.128770476s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 01:11:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 01:11:19.095408794 +0000 UTC m=+1827.359743745" watchObservedRunningTime="2026-11-03 01:11:19.128770476 +0000 UTC m=+1827.393105421" Nov 03 01:11:19.245517 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:11:19.244441 4939 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-ddd653-openshift-default-779ff6465-t9cvw" Nov 03 01:11:19.359421 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:11:19.358098 4939 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-88bbf7/distinct-2-w1-88bbf7-kserve-5457cdcdbd-l6tfd" podUID="a397931b-faa9-4c2e-8290-491350546663" containerName="main" probeResult="failure" output="Get \"https://10.132.0.85:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 01:11:19.529461 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:11:19.523380 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:11:19.705964 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:11:19.686517 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:11:19.994998 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:11:19.985629 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:11:20.327782 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:11:20.327729 4939 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 01:11:20.328105 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:11:20.328071 4939 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 01:11:20.328270 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:11:20.328252 4939 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 01:11:20.331506 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:11:20.331481 4939 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-a-ddd653-openshift-default-779ff6465-t9cvw"] Nov 03 01:11:20.904308 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:11:20.904247 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:11:20.924169 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:11:20.924133 4939 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-d25759-5587dc5df9-tl5d6"] Nov 03 01:11:20.930811 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:11:20.930784 4939 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-a-d25759-5587dc5df9-tl5d6" Nov 03 01:11:20.939271 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:11:20.939236 4939 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-d25759-5587dc5df9-tl5d6"] Nov 03 01:11:20.946406 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:11:20.943826 4939 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-extmod-a-d25759\"" Nov 03 01:11:21.077983 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:11:21.068355 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:11:21.216961 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:11:21.211029 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f2d0a2aa-d02a-4153-b8e8-997c8d1120fe-plugins-config-volume\") pod \"payload-processing-e2e-extmod-a-d25759-5587dc5df9-tl5d6\" (UID: \"f2d0a2aa-d02a-4153-b8e8-997c8d1120fe\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-d25759-5587dc5df9-tl5d6" Nov 03 01:11:21.216961 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:11:21.211158 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ttxsg\" (UniqueName: \"kubernetes.io/projected/f2d0a2aa-d02a-4153-b8e8-997c8d1120fe-kube-api-access-ttxsg\") pod \"payload-processing-e2e-extmod-a-d25759-5587dc5df9-tl5d6\" (UID: \"f2d0a2aa-d02a-4153-b8e8-997c8d1120fe\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-d25759-5587dc5df9-tl5d6" Nov 03 01:11:21.354939 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:11:21.353112 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f2d0a2aa-d02a-4153-b8e8-997c8d1120fe-plugins-config-volume\") pod \"payload-processing-e2e-extmod-a-d25759-5587dc5df9-tl5d6\" (UID: \"f2d0a2aa-d02a-4153-b8e8-997c8d1120fe\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-d25759-5587dc5df9-tl5d6" Nov 03 01:11:21.354939 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:11:21.353221 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ttxsg\" (UniqueName: \"kubernetes.io/projected/f2d0a2aa-d02a-4153-b8e8-997c8d1120fe-kube-api-access-ttxsg\") pod \"payload-processing-e2e-extmod-a-d25759-5587dc5df9-tl5d6\" (UID: \"f2d0a2aa-d02a-4153-b8e8-997c8d1120fe\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-d25759-5587dc5df9-tl5d6" Nov 03 01:11:21.354939 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:11:21.354838 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f2d0a2aa-d02a-4153-b8e8-997c8d1120fe-plugins-config-volume\") pod \"payload-processing-e2e-extmod-a-d25759-5587dc5df9-tl5d6\" (UID: \"f2d0a2aa-d02a-4153-b8e8-997c8d1120fe\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-d25759-5587dc5df9-tl5d6" Nov 03 01:11:21.380933 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:11:21.376751 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ttxsg\" (UniqueName: \"kubernetes.io/projected/f2d0a2aa-d02a-4153-b8e8-997c8d1120fe-kube-api-access-ttxsg\") pod \"payload-processing-e2e-extmod-a-d25759-5587dc5df9-tl5d6\" (UID: \"f2d0a2aa-d02a-4153-b8e8-997c8d1120fe\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-d25759-5587dc5df9-tl5d6" Nov 03 01:11:21.380933 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:11:21.377631 4939 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-a-d25759-5587dc5df9-tl5d6" Nov 03 01:11:21.499506 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:11:21.499463 4939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-ddd653-openshift-default-779ff6465-t9cvw" event={"ID":"aa64537e-0c5b-49ae-9547-df65d9ea4b2d","Type":"ContainerStarted","Data":"eb4593fecdfedbac58d15758fcc9bfdb55ac993ebc532214ba94ace99e5439a4"} Nov 03 01:11:21.499757 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:11:21.499526 4939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-ddd653-openshift-default-779ff6465-t9cvw" event={"ID":"aa64537e-0c5b-49ae-9547-df65d9ea4b2d","Type":"ContainerStarted","Data":"59820df908f2993f112845e1c10602c53c526132782951589076e4a4c1873246"} Nov 03 01:11:21.527451 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:11:21.527391 4939 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-extmod-a-ddd653-openshift-default-779ff6465-t9cvw" podStartSLOduration=3.527372611 podStartE2EDuration="3.527372611s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 01:11:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 01:11:21.524362798 +0000 UTC m=+1829.788697750" watchObservedRunningTime="2026-11-03 01:11:21.527372611 +0000 UTC m=+1829.791707557" Nov 03 01:11:21.552969 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:11:21.552874 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:11:21.823778 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:11:21.823745 4939 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-d25759-5587dc5df9-tl5d6"] Nov 03 01:11:21.825512 ip-10-0-140-107 kubenswrapper[4939]: W1103 01:11:21.825483 4939 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf2d0a2aa_d02a_4153_b8e8_997c8d1120fe.slice/crio-b0ec7a02c5d1501dad844dfecaf6d2e2d88d4983fad2ea417e90748ff4ac4a58 WatchSource:0}: Error finding container b0ec7a02c5d1501dad844dfecaf6d2e2d88d4983fad2ea417e90748ff4ac4a58: Status 404 returned error can't find the container with id b0ec7a02c5d1501dad844dfecaf6d2e2d88d4983fad2ea417e90748ff4ac4a58 Nov 03 01:11:22.245249 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:11:22.245159 4939 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-extmod-a-ddd653-openshift-default-779ff6465-t9cvw" Nov 03 01:11:22.253077 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:11:22.253045 4939 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-extmod-a-ddd653-openshift-default-779ff6465-t9cvw" Nov 03 01:11:22.509433 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:11:22.509331 4939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-a-d25759-5587dc5df9-tl5d6" event={"ID":"f2d0a2aa-d02a-4153-b8e8-997c8d1120fe","Type":"ContainerStarted","Data":"e51084d1ae4e0dd5dfdb55fd149c245c6d3f834a9ed6e1ecde4ee57467423824"} Nov 03 01:11:22.509433 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:11:22.509403 4939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-a-d25759-5587dc5df9-tl5d6" event={"ID":"f2d0a2aa-d02a-4153-b8e8-997c8d1120fe","Type":"ContainerStarted","Data":"b0ec7a02c5d1501dad844dfecaf6d2e2d88d4983fad2ea417e90748ff4ac4a58"} Nov 03 01:11:22.509433 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:11:22.509435 4939 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-extmod-a-d25759-5587dc5df9-tl5d6" Nov 03 01:11:22.509880 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:11:22.509855 4939 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-a-ddd653-openshift-default-779ff6465-t9cvw" Nov 03 01:11:22.510871 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:11:22.510850 4939 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-extmod-a-ddd653-openshift-default-779ff6465-t9cvw" Nov 03 01:11:22.531983 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:11:22.531922 4939 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-extmod-a-d25759-5587dc5df9-tl5d6" podStartSLOduration=2.5318879709999997 podStartE2EDuration="2.531887971s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 01:11:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 01:11:22.525593827 +0000 UTC m=+1830.789928772" watchObservedRunningTime="2026-11-03 01:11:22.531887971 +0000 UTC m=+1830.796222917" Nov 03 01:11:22.586101 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:11:22.586063 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:11:23.320635 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:11:23.320568 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:11:23.495172 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:11:23.495130 4939 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-w3-717217/auth-test-model-717217-kserve-5c9cb7dbd9-vl754" podUID="9634a946-0832-4901-ad01-bbb7a47f4ac7" containerName="main" containerID="cri-o://755ca59ab9dd6cf520eafa60f6bbd3c635b5b3bcb3c4b16e29111e5b34f4a4e4" gracePeriod=45 Nov 03 01:11:23.604023 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:11:23.603972 4939 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-shared-a-w3-717217/auth-test-model-717217-kserve-5c9cb7dbd9-vl754" podUID="9634a946-0832-4901-ad01-bbb7a47f4ac7" containerName="main" probeResult="failure" output="Get \"https://10.132.0.191:8000/ready\": dial tcp 10.132.0.191:8000: connect: connection refused" Nov 03 01:11:23.763709 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:11:23.761195 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:11:23.879597 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:11:23.879559 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:11:23.914321 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:11:23.914296 4939 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-717217/auth-test-model-717217-kserve-5c9cb7dbd9-vl754" Nov 03 01:11:24.097840 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:11:24.097796 4939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9634a946-0832-4901-ad01-bbb7a47f4ac7-kube-api-access-47rl4\" (UniqueName: \"kubernetes.io/projected/9634a946-0832-4901-ad01-bbb7a47f4ac7-kube-api-access-47rl4\") pod \"9634a946-0832-4901-ad01-bbb7a47f4ac7\" (UID: \"9634a946-0832-4901-ad01-bbb7a47f4ac7\") " Nov 03 01:11:24.098069 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:11:24.097975 4939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9634a946-0832-4901-ad01-bbb7a47f4ac7-tls-certs\" (UniqueName: \"kubernetes.io/secret/9634a946-0832-4901-ad01-bbb7a47f4ac7-tls-certs\") pod \"9634a946-0832-4901-ad01-bbb7a47f4ac7\" (UID: \"9634a946-0832-4901-ad01-bbb7a47f4ac7\") " Nov 03 01:11:24.098069 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:11:24.098023 4939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9634a946-0832-4901-ad01-bbb7a47f4ac7-dshm\" (UniqueName: \"kubernetes.io/empty-dir/9634a946-0832-4901-ad01-bbb7a47f4ac7-dshm\") pod \"9634a946-0832-4901-ad01-bbb7a47f4ac7\" (UID: \"9634a946-0832-4901-ad01-bbb7a47f4ac7\") " Nov 03 01:11:24.098224 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:11:24.098107 4939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9634a946-0832-4901-ad01-bbb7a47f4ac7-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9634a946-0832-4901-ad01-bbb7a47f4ac7-model-cache\") pod \"9634a946-0832-4901-ad01-bbb7a47f4ac7\" (UID: \"9634a946-0832-4901-ad01-bbb7a47f4ac7\") " Nov 03 01:11:24.098372 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:11:24.098352 4939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9634a946-0832-4901-ad01-bbb7a47f4ac7-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9634a946-0832-4901-ad01-bbb7a47f4ac7-tmp-dir\") pod \"9634a946-0832-4901-ad01-bbb7a47f4ac7\" (UID: \"9634a946-0832-4901-ad01-bbb7a47f4ac7\") " Nov 03 01:11:24.098440 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:11:24.098399 4939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9634a946-0832-4901-ad01-bbb7a47f4ac7-model-cache" pod "9634a946-0832-4901-ad01-bbb7a47f4ac7" (UID: "9634a946-0832-4901-ad01-bbb7a47f4ac7"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 01:11:24.098440 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:11:24.098427 4939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9634a946-0832-4901-ad01-bbb7a47f4ac7-home\" (UniqueName: \"kubernetes.io/empty-dir/9634a946-0832-4901-ad01-bbb7a47f4ac7-home\") pod \"9634a946-0832-4901-ad01-bbb7a47f4ac7\" (UID: \"9634a946-0832-4901-ad01-bbb7a47f4ac7\") " Nov 03 01:11:24.098674 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:11:24.098636 4939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9634a946-0832-4901-ad01-bbb7a47f4ac7-tmp-dir" pod "9634a946-0832-4901-ad01-bbb7a47f4ac7" (UID: "9634a946-0832-4901-ad01-bbb7a47f4ac7"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 01:11:24.098752 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:11:24.098688 4939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9634a946-0832-4901-ad01-bbb7a47f4ac7-home" pod "9634a946-0832-4901-ad01-bbb7a47f4ac7" (UID: "9634a946-0832-4901-ad01-bbb7a47f4ac7"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 01:11:24.099475 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:11:24.099444 4939 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9634a946-0832-4901-ad01-bbb7a47f4ac7-home\") on node \"ip-10-0-140-107.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:11:24.099475 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:11:24.099474 4939 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9634a946-0832-4901-ad01-bbb7a47f4ac7-model-cache\") on node \"ip-10-0-140-107.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:11:24.099657 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:11:24.099491 4939 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9634a946-0832-4901-ad01-bbb7a47f4ac7-tmp-dir\") on node \"ip-10-0-140-107.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:11:24.101016 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:11:24.100980 4939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9634a946-0832-4901-ad01-bbb7a47f4ac7-dshm" pod "9634a946-0832-4901-ad01-bbb7a47f4ac7" (UID: "9634a946-0832-4901-ad01-bbb7a47f4ac7"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 01:11:24.101160 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:11:24.101141 4939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9634a946-0832-4901-ad01-bbb7a47f4ac7-kube-api-access-47rl4" pod "9634a946-0832-4901-ad01-bbb7a47f4ac7" (UID: "9634a946-0832-4901-ad01-bbb7a47f4ac7"). InnerVolumeSpecName "kube-api-access-47rl4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 01:11:24.101292 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:11:24.101273 4939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9634a946-0832-4901-ad01-bbb7a47f4ac7-tls-certs" pod "9634a946-0832-4901-ad01-bbb7a47f4ac7" (UID: "9634a946-0832-4901-ad01-bbb7a47f4ac7"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 01:11:24.202094 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:11:24.202045 4939 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9634a946-0832-4901-ad01-bbb7a47f4ac7-tls-certs\") on node \"ip-10-0-140-107.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:11:24.202094 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:11:24.202088 4939 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9634a946-0832-4901-ad01-bbb7a47f4ac7-dshm\") on node \"ip-10-0-140-107.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:11:24.202354 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:11:24.202108 4939 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-47rl4\" (UniqueName: \"kubernetes.io/projected/9634a946-0832-4901-ad01-bbb7a47f4ac7-kube-api-access-47rl4\") on node \"ip-10-0-140-107.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:11:24.322671 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:11:24.321875 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:11:24.532614 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:11:24.532531 4939 generic.go:356] "Generic (PLEG): container finished" podID="9634a946-0832-4901-ad01-bbb7a47f4ac7" containerID="755ca59ab9dd6cf520eafa60f6bbd3c635b5b3bcb3c4b16e29111e5b34f4a4e4" exitCode=0 Nov 03 01:11:24.533692 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:11:24.533666 4939 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-717217/auth-test-model-717217-kserve-5c9cb7dbd9-vl754" Nov 03 01:11:24.537251 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:11:24.537219 4939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-717217/auth-test-model-717217-kserve-5c9cb7dbd9-vl754" event={"ID":"9634a946-0832-4901-ad01-bbb7a47f4ac7","Type":"ContainerDied","Data":"755ca59ab9dd6cf520eafa60f6bbd3c635b5b3bcb3c4b16e29111e5b34f4a4e4"} Nov 03 01:11:24.537369 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:11:24.537275 4939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-717217/auth-test-model-717217-kserve-5c9cb7dbd9-vl754" event={"ID":"9634a946-0832-4901-ad01-bbb7a47f4ac7","Type":"ContainerDied","Data":"1a1011920bfb16a07db4c773ec0d776e31c0bf44ed10948940ae6116641e1eae"} Nov 03 01:11:24.537369 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:11:24.537302 4939 scope.go:122] "RemoveContainer" containerID="755ca59ab9dd6cf520eafa60f6bbd3c635b5b3bcb3c4b16e29111e5b34f4a4e4" Nov 03 01:11:24.553288 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:11:24.553259 4939 scope.go:122] "RemoveContainer" containerID="755ca59ab9dd6cf520eafa60f6bbd3c635b5b3bcb3c4b16e29111e5b34f4a4e4" Nov 03 01:11:24.553640 ip-10-0-140-107 kubenswrapper[4939]: E1103 01:11:24.553614 4939 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"755ca59ab9dd6cf520eafa60f6bbd3c635b5b3bcb3c4b16e29111e5b34f4a4e4\": container with ID starting with 755ca59ab9dd6cf520eafa60f6bbd3c635b5b3bcb3c4b16e29111e5b34f4a4e4 not found: ID does not exist" containerID="755ca59ab9dd6cf520eafa60f6bbd3c635b5b3bcb3c4b16e29111e5b34f4a4e4" Nov 03 01:11:24.553753 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:11:24.553648 4939 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"755ca59ab9dd6cf520eafa60f6bbd3c635b5b3bcb3c4b16e29111e5b34f4a4e4"} err="failed to get container status \"755ca59ab9dd6cf520eafa60f6bbd3c635b5b3bcb3c4b16e29111e5b34f4a4e4\": rpc error: code = NotFound desc = could not find container \"755ca59ab9dd6cf520eafa60f6bbd3c635b5b3bcb3c4b16e29111e5b34f4a4e4\": container with ID starting with 755ca59ab9dd6cf520eafa60f6bbd3c635b5b3bcb3c4b16e29111e5b34f4a4e4 not found: ID does not exist" Nov 03 01:11:24.572804 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:11:24.572768 4939 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w3-717217/auth-test-model-717217-kserve-5c9cb7dbd9-vl754"] Nov 03 01:11:24.578601 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:11:24.578571 4939 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-w3-717217/auth-test-model-717217-kserve-5c9cb7dbd9-vl754"] Nov 03 01:11:24.975320 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:11:24.975253 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:11:24.994744 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:11:24.994706 4939 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-ddd653-84dfd5bd67-wv882"] Nov 03 01:11:24.995792 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:11:24.995769 4939 cpu_manager.go:414] "containerMap: removing container" podUID="9634a946-0832-4901-ad01-bbb7a47f4ac7" containerName="main" Nov 03 01:11:24.995792 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:11:24.995790 4939 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9634a946-0832-4901-ad01-bbb7a47f4ac7" containerName="main" Nov 03 01:11:24.996064 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:11:24.996049 4939 memory_manager.go:365] "RemoveStaleState removing state" podUID="9634a946-0832-4901-ad01-bbb7a47f4ac7" containerName="main" Nov 03 01:11:25.038522 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:11:25.038471 4939 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-ddd653-84dfd5bd67-wv882"] Nov 03 01:11:25.038715 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:11:25.038536 4939 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-ddd653-5764b8bfbc-fcxrz"] Nov 03 01:11:25.045578 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:11:25.045544 4939 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-a-ddd653-84dfd5bd67-wv882" Nov 03 01:11:25.053043 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:11:25.053011 4939 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-extmod-a-ddd653\"" Nov 03 01:11:25.067871 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:11:25.067831 4939 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-ddd653-5764b8bfbc-fcxrz"] Nov 03 01:11:25.068051 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:11:25.068001 4939 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-ddd653-5764b8bfbc-fcxrz" Nov 03 01:11:25.081959 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:11:25.081000 4939 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-extmod-a-ddd653-dockercfg-994nb\"" Nov 03 01:11:25.209930 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:11:25.207319 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:11:25.600936 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:11:25.599356 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c023b319-6159-4d96-bacb-59ec3bc0d55c-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-a-ddd653-5764b8bfbc-fcxrz\" (UID: \"c023b319-6159-4d96-bacb-59ec3bc0d55c\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-ddd653-5764b8bfbc-fcxrz" Nov 03 01:11:25.600936 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:11:25.599426 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s7sb7\" (UniqueName: \"kubernetes.io/projected/c023b319-6159-4d96-bacb-59ec3bc0d55c-kube-api-access-s7sb7\") pod \"payload-pre-processing-e2e-extmod-a-ddd653-5764b8bfbc-fcxrz\" (UID: \"c023b319-6159-4d96-bacb-59ec3bc0d55c\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-ddd653-5764b8bfbc-fcxrz" Nov 03 01:11:25.600936 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:11:25.599461 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/60e096be-eed5-4cdd-b78b-f7e691f2d880-plugins-config-volume\") pod \"payload-processing-e2e-extmod-a-ddd653-84dfd5bd67-wv882\" (UID: \"60e096be-eed5-4cdd-b78b-f7e691f2d880\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-ddd653-84dfd5bd67-wv882" Nov 03 01:11:25.600936 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:11:25.599811 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4jjlh\" (UniqueName: \"kubernetes.io/projected/60e096be-eed5-4cdd-b78b-f7e691f2d880-kube-api-access-4jjlh\") pod \"payload-processing-e2e-extmod-a-ddd653-84dfd5bd67-wv882\" (UID: \"60e096be-eed5-4cdd-b78b-f7e691f2d880\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-ddd653-84dfd5bd67-wv882" Nov 03 01:11:25.701543 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:11:25.701502 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4jjlh\" (UniqueName: \"kubernetes.io/projected/60e096be-eed5-4cdd-b78b-f7e691f2d880-kube-api-access-4jjlh\") pod \"payload-processing-e2e-extmod-a-ddd653-84dfd5bd67-wv882\" (UID: \"60e096be-eed5-4cdd-b78b-f7e691f2d880\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-ddd653-84dfd5bd67-wv882" Nov 03 01:11:25.702038 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:11:25.702009 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c023b319-6159-4d96-bacb-59ec3bc0d55c-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-a-ddd653-5764b8bfbc-fcxrz\" (UID: \"c023b319-6159-4d96-bacb-59ec3bc0d55c\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-ddd653-5764b8bfbc-fcxrz" Nov 03 01:11:25.702165 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:11:25.702051 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s7sb7\" (UniqueName: \"kubernetes.io/projected/c023b319-6159-4d96-bacb-59ec3bc0d55c-kube-api-access-s7sb7\") pod \"payload-pre-processing-e2e-extmod-a-ddd653-5764b8bfbc-fcxrz\" (UID: \"c023b319-6159-4d96-bacb-59ec3bc0d55c\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-ddd653-5764b8bfbc-fcxrz" Nov 03 01:11:25.702165 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:11:25.702083 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/60e096be-eed5-4cdd-b78b-f7e691f2d880-plugins-config-volume\") pod \"payload-processing-e2e-extmod-a-ddd653-84dfd5bd67-wv882\" (UID: \"60e096be-eed5-4cdd-b78b-f7e691f2d880\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-ddd653-84dfd5bd67-wv882" Nov 03 01:11:25.703231 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:11:25.703203 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c023b319-6159-4d96-bacb-59ec3bc0d55c-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-a-ddd653-5764b8bfbc-fcxrz\" (UID: \"c023b319-6159-4d96-bacb-59ec3bc0d55c\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-ddd653-5764b8bfbc-fcxrz" Nov 03 01:11:25.703325 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:11:25.703252 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/60e096be-eed5-4cdd-b78b-f7e691f2d880-plugins-config-volume\") pod \"payload-processing-e2e-extmod-a-ddd653-84dfd5bd67-wv882\" (UID: \"60e096be-eed5-4cdd-b78b-f7e691f2d880\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-ddd653-84dfd5bd67-wv882" Nov 03 01:11:25.709956 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:11:25.709896 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4jjlh\" (UniqueName: \"kubernetes.io/projected/60e096be-eed5-4cdd-b78b-f7e691f2d880-kube-api-access-4jjlh\") pod \"payload-processing-e2e-extmod-a-ddd653-84dfd5bd67-wv882\" (UID: \"60e096be-eed5-4cdd-b78b-f7e691f2d880\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-ddd653-84dfd5bd67-wv882" Nov 03 01:11:25.710122 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:11:25.710099 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-s7sb7\" (UniqueName: \"kubernetes.io/projected/c023b319-6159-4d96-bacb-59ec3bc0d55c-kube-api-access-s7sb7\") pod \"payload-pre-processing-e2e-extmod-a-ddd653-5764b8bfbc-fcxrz\" (UID: \"c023b319-6159-4d96-bacb-59ec3bc0d55c\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-ddd653-5764b8bfbc-fcxrz" Nov 03 01:11:25.720154 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:11:25.720124 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:11:25.751935 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:11:25.735389 4939 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-ddd653-5764b8bfbc-fcxrz" Nov 03 01:11:25.857432 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:11:25.857344 4939 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-w3-556c14/auth-test-model-556c14-kserve-8d5cd469-t6tzc" podUID="0c4e579c-9fe2-4ec9-b8f4-62758e519980" containerName="main" containerID="cri-o://7309d80026ae207eb57dd222930c990ca7ea1f761f48e3aefefcd1bab0e710f9" gracePeriod=45 Nov 03 01:11:25.878703 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:11:25.878634 4939 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9634a946-0832-4901-ad01-bbb7a47f4ac7" path="/var/lib/kubelet/pods/9634a946-0832-4901-ad01-bbb7a47f4ac7/volumes" Nov 03 01:11:26.007233 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:11:26.007154 4939 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-a-ddd653-84dfd5bd67-wv882" Nov 03 01:11:26.160811 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:11:26.160186 4939 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-ddd653-5764b8bfbc-fcxrz"] Nov 03 01:11:26.331573 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:11:26.331537 4939 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-556c14/auth-test-model-556c14-kserve-8d5cd469-t6tzc" Nov 03 01:11:26.363257 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:11:26.363190 4939 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-ddd653-84dfd5bd67-wv882"] Nov 03 01:11:26.431157 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:11:26.431114 4939 patch_prober.go:29] interesting pod/e2e-extmod-a-027c19-openshift-default-58d5bc7645-zjnf8 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.207:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 01:11:26.431376 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:11:26.431185 4939 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-extmod-a-027c19-openshift-default-58d5bc7645-zjnf8" podUID="f09e7c2c-246e-4c96-a8ca-147e8ca83560" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.207:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 01:11:26.437305 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:11:26.437162 4939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0c4e579c-9fe2-4ec9-b8f4-62758e519980-home\" (UniqueName: \"kubernetes.io/empty-dir/0c4e579c-9fe2-4ec9-b8f4-62758e519980-home\") pod \"0c4e579c-9fe2-4ec9-b8f4-62758e519980\" (UID: \"0c4e579c-9fe2-4ec9-b8f4-62758e519980\") " Nov 03 01:11:26.438820 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:11:26.437480 4939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0c4e579c-9fe2-4ec9-b8f4-62758e519980-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0c4e579c-9fe2-4ec9-b8f4-62758e519980-tmp-dir\") pod \"0c4e579c-9fe2-4ec9-b8f4-62758e519980\" (UID: \"0c4e579c-9fe2-4ec9-b8f4-62758e519980\") " Nov 03 01:11:26.438820 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:11:26.437548 4939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0c4e579c-9fe2-4ec9-b8f4-62758e519980-dshm\" (UniqueName: \"kubernetes.io/empty-dir/0c4e579c-9fe2-4ec9-b8f4-62758e519980-dshm\") pod \"0c4e579c-9fe2-4ec9-b8f4-62758e519980\" (UID: \"0c4e579c-9fe2-4ec9-b8f4-62758e519980\") " Nov 03 01:11:26.438820 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:11:26.437585 4939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0c4e579c-9fe2-4ec9-b8f4-62758e519980-kube-api-access-tbsmb\" (UniqueName: \"kubernetes.io/projected/0c4e579c-9fe2-4ec9-b8f4-62758e519980-kube-api-access-tbsmb\") pod \"0c4e579c-9fe2-4ec9-b8f4-62758e519980\" (UID: \"0c4e579c-9fe2-4ec9-b8f4-62758e519980\") " Nov 03 01:11:26.438820 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:11:26.437636 4939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0c4e579c-9fe2-4ec9-b8f4-62758e519980-tls-certs\" (UniqueName: \"kubernetes.io/secret/0c4e579c-9fe2-4ec9-b8f4-62758e519980-tls-certs\") pod \"0c4e579c-9fe2-4ec9-b8f4-62758e519980\" (UID: \"0c4e579c-9fe2-4ec9-b8f4-62758e519980\") " Nov 03 01:11:26.438820 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:11:26.437900 4939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0c4e579c-9fe2-4ec9-b8f4-62758e519980-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0c4e579c-9fe2-4ec9-b8f4-62758e519980-model-cache\") pod \"0c4e579c-9fe2-4ec9-b8f4-62758e519980\" (UID: \"0c4e579c-9fe2-4ec9-b8f4-62758e519980\") " Nov 03 01:11:26.439976 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:11:26.439765 4939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0c4e579c-9fe2-4ec9-b8f4-62758e519980-model-cache" pod "0c4e579c-9fe2-4ec9-b8f4-62758e519980" (UID: "0c4e579c-9fe2-4ec9-b8f4-62758e519980"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 01:11:26.440293 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:11:26.440117 4939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0c4e579c-9fe2-4ec9-b8f4-62758e519980-home" pod "0c4e579c-9fe2-4ec9-b8f4-62758e519980" (UID: "0c4e579c-9fe2-4ec9-b8f4-62758e519980"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 01:11:26.440773 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:11:26.440444 4939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0c4e579c-9fe2-4ec9-b8f4-62758e519980-tmp-dir" pod "0c4e579c-9fe2-4ec9-b8f4-62758e519980" (UID: "0c4e579c-9fe2-4ec9-b8f4-62758e519980"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 01:11:26.447128 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:11:26.443876 4939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0c4e579c-9fe2-4ec9-b8f4-62758e519980-dshm" pod "0c4e579c-9fe2-4ec9-b8f4-62758e519980" (UID: "0c4e579c-9fe2-4ec9-b8f4-62758e519980"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 01:11:26.447128 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:11:26.445278 4939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0c4e579c-9fe2-4ec9-b8f4-62758e519980-kube-api-access-tbsmb" pod "0c4e579c-9fe2-4ec9-b8f4-62758e519980" (UID: "0c4e579c-9fe2-4ec9-b8f4-62758e519980"). InnerVolumeSpecName "kube-api-access-tbsmb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 01:11:26.447128 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:11:26.446216 4939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0c4e579c-9fe2-4ec9-b8f4-62758e519980-tls-certs" pod "0c4e579c-9fe2-4ec9-b8f4-62758e519980" (UID: "0c4e579c-9fe2-4ec9-b8f4-62758e519980"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 01:11:26.496878 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:11:26.496838 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:11:26.583212 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:11:26.562237 4939 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0c4e579c-9fe2-4ec9-b8f4-62758e519980-model-cache\") on node \"ip-10-0-140-107.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:11:26.583212 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:11:26.562271 4939 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0c4e579c-9fe2-4ec9-b8f4-62758e519980-home\") on node \"ip-10-0-140-107.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:11:26.583212 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:11:26.562289 4939 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0c4e579c-9fe2-4ec9-b8f4-62758e519980-tmp-dir\") on node \"ip-10-0-140-107.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:11:26.583212 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:11:26.562302 4939 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0c4e579c-9fe2-4ec9-b8f4-62758e519980-dshm\") on node \"ip-10-0-140-107.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:11:26.583212 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:11:26.562315 4939 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tbsmb\" (UniqueName: \"kubernetes.io/projected/0c4e579c-9fe2-4ec9-b8f4-62758e519980-kube-api-access-tbsmb\") on node \"ip-10-0-140-107.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:11:26.583212 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:11:26.562345 4939 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0c4e579c-9fe2-4ec9-b8f4-62758e519980-tls-certs\") on node \"ip-10-0-140-107.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:11:26.652061 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:11:26.646143 4939 generic.go:356] "Generic (PLEG): container finished" podID="0c4e579c-9fe2-4ec9-b8f4-62758e519980" containerID="7309d80026ae207eb57dd222930c990ca7ea1f761f48e3aefefcd1bab0e710f9" exitCode=0 Nov 03 01:11:26.652061 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:11:26.646325 4939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-556c14/auth-test-model-556c14-kserve-8d5cd469-t6tzc" event={"ID":"0c4e579c-9fe2-4ec9-b8f4-62758e519980","Type":"ContainerDied","Data":"7309d80026ae207eb57dd222930c990ca7ea1f761f48e3aefefcd1bab0e710f9"} Nov 03 01:11:26.652061 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:11:26.646372 4939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-556c14/auth-test-model-556c14-kserve-8d5cd469-t6tzc" event={"ID":"0c4e579c-9fe2-4ec9-b8f4-62758e519980","Type":"ContainerDied","Data":"0558234bd46f2ba455472b3f3a8992f0166135a147446b9fd081a49e24ca2012"} Nov 03 01:11:26.652061 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:11:26.646398 4939 scope.go:122] "RemoveContainer" containerID="7309d80026ae207eb57dd222930c990ca7ea1f761f48e3aefefcd1bab0e710f9" Nov 03 01:11:26.652061 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:11:26.646556 4939 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-556c14/auth-test-model-556c14-kserve-8d5cd469-t6tzc" Nov 03 01:11:26.672325 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:11:26.670342 4939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-a-ddd653-84dfd5bd67-wv882" event={"ID":"60e096be-eed5-4cdd-b78b-f7e691f2d880","Type":"ContainerStarted","Data":"5138b2d4446603efdae58a03377b1649ddaebbcae2eca8e8a567ac23dd8fb38d"} Nov 03 01:11:26.673788 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:11:26.673475 4939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-ddd653-5764b8bfbc-fcxrz" event={"ID":"c023b319-6159-4d96-bacb-59ec3bc0d55c","Type":"ContainerStarted","Data":"3f122c183feec6b2fd59afce69cdb5d76b7035ed9e1b4e1d8a492b0d6391fff8"} Nov 03 01:11:26.673788 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:11:26.673550 4939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-ddd653-5764b8bfbc-fcxrz" event={"ID":"c023b319-6159-4d96-bacb-59ec3bc0d55c","Type":"ContainerStarted","Data":"46b19f4d45f363a7e582f7d9f5daabe0f40cf85b369987dc194aa789df37ea4b"} Nov 03 01:11:26.685448 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:11:26.674616 4939 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-ddd653-5764b8bfbc-fcxrz" Nov 03 01:11:26.686243 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:11:26.686205 4939 scope.go:122] "RemoveContainer" containerID="7309d80026ae207eb57dd222930c990ca7ea1f761f48e3aefefcd1bab0e710f9" Nov 03 01:11:26.686574 ip-10-0-140-107 kubenswrapper[4939]: E1103 01:11:26.686551 4939 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7309d80026ae207eb57dd222930c990ca7ea1f761f48e3aefefcd1bab0e710f9\": container with ID starting with 7309d80026ae207eb57dd222930c990ca7ea1f761f48e3aefefcd1bab0e710f9 not found: ID does not exist" containerID="7309d80026ae207eb57dd222930c990ca7ea1f761f48e3aefefcd1bab0e710f9" Nov 03 01:11:26.686668 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:11:26.686582 4939 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7309d80026ae207eb57dd222930c990ca7ea1f761f48e3aefefcd1bab0e710f9"} err="failed to get container status \"7309d80026ae207eb57dd222930c990ca7ea1f761f48e3aefefcd1bab0e710f9\": rpc error: code = NotFound desc = could not find container \"7309d80026ae207eb57dd222930c990ca7ea1f761f48e3aefefcd1bab0e710f9\": container with ID starting with 7309d80026ae207eb57dd222930c990ca7ea1f761f48e3aefefcd1bab0e710f9 not found: ID does not exist" Nov 03 01:11:26.696008 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:11:26.695950 4939 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-ddd653-5764b8bfbc-fcxrz" podStartSLOduration=2.695933072 podStartE2EDuration="2.695933072s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 01:11:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 01:11:26.694866063 +0000 UTC m=+1834.959201039" watchObservedRunningTime="2026-11-03 01:11:26.695933072 +0000 UTC m=+1834.960268018" Nov 03 01:11:26.712685 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:11:26.712647 4939 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w3-556c14/auth-test-model-556c14-kserve-8d5cd469-t6tzc"] Nov 03 01:11:26.717153 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:11:26.717118 4939 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-w3-556c14/auth-test-model-556c14-kserve-8d5cd469-t6tzc"] Nov 03 01:11:27.410616 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:11:27.298966 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:11:27.420505 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:11:27.415447 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:11:27.685825 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:11:27.685789 4939 generic.go:356] "Generic (PLEG): container finished" podID="6ea24c7b-6d13-4661-bdff-cd52792cb6b6" containerID="a45f53d3c2fe34da76be639f2a986c7317499e4dc3d560d8ca33ca00bdf94750" exitCode=0 Nov 03 01:11:27.686335 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:11:27.685863 4939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-7c759a-openshift-default-56576554df-wkcvp" event={"ID":"6ea24c7b-6d13-4661-bdff-cd52792cb6b6","Type":"ContainerDied","Data":"a45f53d3c2fe34da76be639f2a986c7317499e4dc3d560d8ca33ca00bdf94750"} Nov 03 01:11:27.686456 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:11:27.686436 4939 scope.go:122] "RemoveContainer" containerID="a45f53d3c2fe34da76be639f2a986c7317499e4dc3d560d8ca33ca00bdf94750" Nov 03 01:11:27.687720 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:11:27.687693 4939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-a-ddd653-84dfd5bd67-wv882" event={"ID":"60e096be-eed5-4cdd-b78b-f7e691f2d880","Type":"ContainerStarted","Data":"01e21828ffe67b88db5dfd9651fa1e679804a8f3661a422523931ce366ab6e25"} Nov 03 01:11:27.687814 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:11:27.687746 4939 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-extmod-a-ddd653-84dfd5bd67-wv882" Nov 03 01:11:27.688887 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:11:27.688857 4939 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 01:11:27.688998 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:11:27.688962 4939 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 01:11:27.689068 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:11:27.689011 4939 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 01:11:27.689577 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:11:27.689525 4939 generic.go:356] "Generic (PLEG): container finished" podID="f09e7c2c-246e-4c96-a8ca-147e8ca83560" containerID="a547b507af135685227fd7790e98b40a7d9718e0f67e158871a6c8acb5162fcc" exitCode=0 Nov 03 01:11:27.689648 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:11:27.689592 4939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-027c19-openshift-default-58d5bc7645-zjnf8" event={"ID":"f09e7c2c-246e-4c96-a8ca-147e8ca83560","Type":"ContainerDied","Data":"a547b507af135685227fd7790e98b40a7d9718e0f67e158871a6c8acb5162fcc"} Nov 03 01:11:27.690194 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:11:27.690178 4939 scope.go:122] "RemoveContainer" containerID="a547b507af135685227fd7790e98b40a7d9718e0f67e158871a6c8acb5162fcc" Nov 03 01:11:27.692099 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:11:27.692071 4939 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 01:11:27.692189 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:11:27.692151 4939 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 01:11:27.692262 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:11:27.692200 4939 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 01:11:27.768648 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:11:27.765745 4939 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-extmod-a-ddd653-84dfd5bd67-wv882" podStartSLOduration=3.765712193 podStartE2EDuration="3.765712193s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 01:11:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 01:11:27.759673199 +0000 UTC m=+1836.024008143" watchObservedRunningTime="2026-11-03 01:11:27.765712193 +0000 UTC m=+1836.030047139" Nov 03 01:11:27.877971 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:11:27.877931 4939 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0c4e579c-9fe2-4ec9-b8f4-62758e519980" path="/var/lib/kubelet/pods/0c4e579c-9fe2-4ec9-b8f4-62758e519980/volumes" Nov 03 01:11:28.608581 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:11:28.608542 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:11:28.754353 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:11:28.752294 4939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-7c759a-openshift-default-56576554df-wkcvp" event={"ID":"6ea24c7b-6d13-4661-bdff-cd52792cb6b6","Type":"ContainerStarted","Data":"225a0fd49c51b5f986149881dd6aa294b9a1e96c6ebda525835578dd224f0a1e"} Nov 03 01:11:28.765986 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:11:28.765899 4939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-027c19-openshift-default-58d5bc7645-zjnf8" event={"ID":"f09e7c2c-246e-4c96-a8ca-147e8ca83560","Type":"ContainerStarted","Data":"42a909aaa61a1c703977779fe62c035ddf2c41e0d466ec8d14c4edffdf0f4ee5"} Nov 03 01:11:28.888495 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:11:28.888385 4939 patch_prober.go:29] interesting pod/e2e-extmod-a-7c759a-openshift-default-56576554df-wkcvp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.210:15021/healthz/ready\": dial tcp 10.132.0.210:15021: connect: connection refused" start-of-body= Nov 03 01:11:28.888825 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:11:28.888794 4939 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-7c759a-openshift-default-56576554df-wkcvp" podUID="6ea24c7b-6d13-4661-bdff-cd52792cb6b6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.210:15021/healthz/ready\": dial tcp 10.132.0.210:15021: connect: connection refused" Nov 03 01:11:28.889043 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:11:28.888727 4939 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-extmod-a-7c759a-openshift-default-56576554df-wkcvp" Nov 03 01:11:29.481931 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:11:29.481203 4939 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-extmod-a-027c19-openshift-default-58d5bc7645-zjnf8" Nov 03 01:11:29.496599 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:11:29.485010 4939 patch_prober.go:29] interesting pod/e2e-extmod-a-027c19-openshift-default-58d5bc7645-zjnf8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.207:15021/healthz/ready\": dial tcp 10.132.0.207:15021: connect: connection refused" start-of-body= Nov 03 01:11:29.496599 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:11:29.485071 4939 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-027c19-openshift-default-58d5bc7645-zjnf8" podUID="f09e7c2c-246e-4c96-a8ca-147e8ca83560" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.207:15021/healthz/ready\": dial tcp 10.132.0.207:15021: connect: connection refused" Nov 03 01:11:29.505395 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:11:29.505346 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:11:29.900360 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:11:29.900322 4939 patch_prober.go:29] interesting pod/e2e-extmod-a-7c759a-openshift-default-56576554df-wkcvp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.210:15021/healthz/ready\": dial tcp 10.132.0.210:15021: connect: connection refused" start-of-body= Nov 03 01:11:29.900962 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:11:29.900379 4939 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-7c759a-openshift-default-56576554df-wkcvp" podUID="6ea24c7b-6d13-4661-bdff-cd52792cb6b6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.210:15021/healthz/ready\": dial tcp 10.132.0.210:15021: connect: connection refused" Nov 03 01:11:30.243447 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:11:30.243347 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:11:30.353653 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:11:30.353621 4939 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-extmod-b-57079b-5f4958f857-v8cxd" Nov 03 01:11:30.473051 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:11:30.473011 4939 patch_prober.go:29] interesting pod/e2e-extmod-a-027c19-openshift-default-58d5bc7645-zjnf8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.207:15021/healthz/ready\": dial tcp 10.132.0.207:15021: connect: connection refused" start-of-body= Nov 03 01:11:30.479078 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:11:30.479032 4939 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-027c19-openshift-default-58d5bc7645-zjnf8" podUID="f09e7c2c-246e-4c96-a8ca-147e8ca83560" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.207:15021/healthz/ready\": dial tcp 10.132.0.207:15021: connect: connection refused" Nov 03 01:11:30.635393 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:11:30.635353 4939 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w3-717217-openshift-default-578c69554c-hs4f9"] Nov 03 01:11:30.636851 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:11:30.636819 4939 cpu_manager.go:414] "containerMap: removing container" podUID="0c4e579c-9fe2-4ec9-b8f4-62758e519980" containerName="main" Nov 03 01:11:30.636851 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:11:30.636850 4939 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0c4e579c-9fe2-4ec9-b8f4-62758e519980" containerName="main" Nov 03 01:11:30.637088 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:11:30.637072 4939 memory_manager.go:365] "RemoveStaleState removing state" podUID="0c4e579c-9fe2-4ec9-b8f4-62758e519980" containerName="main" Nov 03 01:11:30.650254 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:11:30.650216 4939 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w3-717217-openshift-default-578c69554c-hs4f9" Nov 03 01:11:30.716581 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:11:30.716549 4939 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-717217-openshift-default-578c69554c-hs4f9"] Nov 03 01:11:30.745798 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:11:30.745259 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8ba28520-d5f4-4ec9-8b86-f8a698325eea-istio-token\") pod \"e2e-shared-a-w3-717217-openshift-default-578c69554c-hs4f9\" (UID: \"8ba28520-d5f4-4ec9-8b86-f8a698325eea\") " pod="openshift-ingress/e2e-shared-a-w3-717217-openshift-default-578c69554c-hs4f9" Nov 03 01:11:30.745798 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:11:30.745387 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8ba28520-d5f4-4ec9-8b86-f8a698325eea-istio-podinfo\") pod \"e2e-shared-a-w3-717217-openshift-default-578c69554c-hs4f9\" (UID: \"8ba28520-d5f4-4ec9-8b86-f8a698325eea\") " pod="openshift-ingress/e2e-shared-a-w3-717217-openshift-default-578c69554c-hs4f9" Nov 03 01:11:30.745798 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:11:30.745440 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cp7w8\" (UniqueName: \"kubernetes.io/projected/8ba28520-d5f4-4ec9-8b86-f8a698325eea-kube-api-access-cp7w8\") pod \"e2e-shared-a-w3-717217-openshift-default-578c69554c-hs4f9\" (UID: \"8ba28520-d5f4-4ec9-8b86-f8a698325eea\") " pod="openshift-ingress/e2e-shared-a-w3-717217-openshift-default-578c69554c-hs4f9" Nov 03 01:11:30.745798 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:11:30.745606 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8ba28520-d5f4-4ec9-8b86-f8a698325eea-istiod-ca-cert\") pod \"e2e-shared-a-w3-717217-openshift-default-578c69554c-hs4f9\" (UID: \"8ba28520-d5f4-4ec9-8b86-f8a698325eea\") " pod="openshift-ingress/e2e-shared-a-w3-717217-openshift-default-578c69554c-hs4f9" Nov 03 01:11:30.745798 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:11:30.745743 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8ba28520-d5f4-4ec9-8b86-f8a698325eea-credential-socket\") pod \"e2e-shared-a-w3-717217-openshift-default-578c69554c-hs4f9\" (UID: \"8ba28520-d5f4-4ec9-8b86-f8a698325eea\") " pod="openshift-ingress/e2e-shared-a-w3-717217-openshift-default-578c69554c-hs4f9" Nov 03 01:11:30.783934 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:11:30.758961 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8ba28520-d5f4-4ec9-8b86-f8a698325eea-workload-certs\") pod \"e2e-shared-a-w3-717217-openshift-default-578c69554c-hs4f9\" (UID: \"8ba28520-d5f4-4ec9-8b86-f8a698325eea\") " pod="openshift-ingress/e2e-shared-a-w3-717217-openshift-default-578c69554c-hs4f9" Nov 03 01:11:30.783934 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:11:30.783003 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8ba28520-d5f4-4ec9-8b86-f8a698325eea-istio-data\") pod \"e2e-shared-a-w3-717217-openshift-default-578c69554c-hs4f9\" (UID: \"8ba28520-d5f4-4ec9-8b86-f8a698325eea\") " pod="openshift-ingress/e2e-shared-a-w3-717217-openshift-default-578c69554c-hs4f9" Nov 03 01:11:30.783934 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:11:30.783137 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8ba28520-d5f4-4ec9-8b86-f8a698325eea-workload-socket\") pod \"e2e-shared-a-w3-717217-openshift-default-578c69554c-hs4f9\" (UID: \"8ba28520-d5f4-4ec9-8b86-f8a698325eea\") " pod="openshift-ingress/e2e-shared-a-w3-717217-openshift-default-578c69554c-hs4f9" Nov 03 01:11:30.783934 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:11:30.783171 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8ba28520-d5f4-4ec9-8b86-f8a698325eea-istio-envoy\") pod \"e2e-shared-a-w3-717217-openshift-default-578c69554c-hs4f9\" (UID: \"8ba28520-d5f4-4ec9-8b86-f8a698325eea\") " pod="openshift-ingress/e2e-shared-a-w3-717217-openshift-default-578c69554c-hs4f9" Nov 03 01:11:30.903028 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:11:30.902693 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8ba28520-d5f4-4ec9-8b86-f8a698325eea-credential-socket\") pod \"e2e-shared-a-w3-717217-openshift-default-578c69554c-hs4f9\" (UID: \"8ba28520-d5f4-4ec9-8b86-f8a698325eea\") " pod="openshift-ingress/e2e-shared-a-w3-717217-openshift-default-578c69554c-hs4f9" Nov 03 01:11:30.903028 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:11:30.902942 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8ba28520-d5f4-4ec9-8b86-f8a698325eea-workload-certs\") pod \"e2e-shared-a-w3-717217-openshift-default-578c69554c-hs4f9\" (UID: \"8ba28520-d5f4-4ec9-8b86-f8a698325eea\") " pod="openshift-ingress/e2e-shared-a-w3-717217-openshift-default-578c69554c-hs4f9" Nov 03 01:11:30.903028 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:11:30.902968 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8ba28520-d5f4-4ec9-8b86-f8a698325eea-istio-data\") pod \"e2e-shared-a-w3-717217-openshift-default-578c69554c-hs4f9\" (UID: \"8ba28520-d5f4-4ec9-8b86-f8a698325eea\") " pod="openshift-ingress/e2e-shared-a-w3-717217-openshift-default-578c69554c-hs4f9" Nov 03 01:11:30.903647 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:11:30.903042 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8ba28520-d5f4-4ec9-8b86-f8a698325eea-workload-socket\") pod \"e2e-shared-a-w3-717217-openshift-default-578c69554c-hs4f9\" (UID: \"8ba28520-d5f4-4ec9-8b86-f8a698325eea\") " pod="openshift-ingress/e2e-shared-a-w3-717217-openshift-default-578c69554c-hs4f9" Nov 03 01:11:30.903647 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:11:30.903066 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8ba28520-d5f4-4ec9-8b86-f8a698325eea-istio-envoy\") pod \"e2e-shared-a-w3-717217-openshift-default-578c69554c-hs4f9\" (UID: \"8ba28520-d5f4-4ec9-8b86-f8a698325eea\") " pod="openshift-ingress/e2e-shared-a-w3-717217-openshift-default-578c69554c-hs4f9" Nov 03 01:11:30.903647 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:11:30.903094 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8ba28520-d5f4-4ec9-8b86-f8a698325eea-istio-token\") pod \"e2e-shared-a-w3-717217-openshift-default-578c69554c-hs4f9\" (UID: \"8ba28520-d5f4-4ec9-8b86-f8a698325eea\") " pod="openshift-ingress/e2e-shared-a-w3-717217-openshift-default-578c69554c-hs4f9" Nov 03 01:11:30.903647 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:11:30.903148 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8ba28520-d5f4-4ec9-8b86-f8a698325eea-istio-podinfo\") pod \"e2e-shared-a-w3-717217-openshift-default-578c69554c-hs4f9\" (UID: \"8ba28520-d5f4-4ec9-8b86-f8a698325eea\") " pod="openshift-ingress/e2e-shared-a-w3-717217-openshift-default-578c69554c-hs4f9" Nov 03 01:11:30.903647 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:11:30.903200 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cp7w8\" (UniqueName: \"kubernetes.io/projected/8ba28520-d5f4-4ec9-8b86-f8a698325eea-kube-api-access-cp7w8\") pod \"e2e-shared-a-w3-717217-openshift-default-578c69554c-hs4f9\" (UID: \"8ba28520-d5f4-4ec9-8b86-f8a698325eea\") " pod="openshift-ingress/e2e-shared-a-w3-717217-openshift-default-578c69554c-hs4f9" Nov 03 01:11:30.903647 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:11:30.903297 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8ba28520-d5f4-4ec9-8b86-f8a698325eea-istiod-ca-cert\") pod \"e2e-shared-a-w3-717217-openshift-default-578c69554c-hs4f9\" (UID: \"8ba28520-d5f4-4ec9-8b86-f8a698325eea\") " pod="openshift-ingress/e2e-shared-a-w3-717217-openshift-default-578c69554c-hs4f9" Nov 03 01:11:30.904845 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:11:30.904198 4939 patch_prober.go:29] interesting pod/e2e-extmod-a-7c759a-openshift-default-56576554df-wkcvp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.210:15021/healthz/ready\": dial tcp 10.132.0.210:15021: connect: connection refused" start-of-body= Nov 03 01:11:30.904845 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:11:30.904223 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8ba28520-d5f4-4ec9-8b86-f8a698325eea-istiod-ca-cert\") pod \"e2e-shared-a-w3-717217-openshift-default-578c69554c-hs4f9\" (UID: \"8ba28520-d5f4-4ec9-8b86-f8a698325eea\") " pod="openshift-ingress/e2e-shared-a-w3-717217-openshift-default-578c69554c-hs4f9" Nov 03 01:11:30.904845 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:11:30.904253 4939 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-7c759a-openshift-default-56576554df-wkcvp" podUID="6ea24c7b-6d13-4661-bdff-cd52792cb6b6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.210:15021/healthz/ready\": dial tcp 10.132.0.210:15021: connect: connection refused" Nov 03 01:11:30.904845 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:11:30.904533 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8ba28520-d5f4-4ec9-8b86-f8a698325eea-credential-socket\") pod \"e2e-shared-a-w3-717217-openshift-default-578c69554c-hs4f9\" (UID: \"8ba28520-d5f4-4ec9-8b86-f8a698325eea\") " pod="openshift-ingress/e2e-shared-a-w3-717217-openshift-default-578c69554c-hs4f9" Nov 03 01:11:30.904845 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:11:30.904795 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8ba28520-d5f4-4ec9-8b86-f8a698325eea-workload-certs\") pod \"e2e-shared-a-w3-717217-openshift-default-578c69554c-hs4f9\" (UID: \"8ba28520-d5f4-4ec9-8b86-f8a698325eea\") " pod="openshift-ingress/e2e-shared-a-w3-717217-openshift-default-578c69554c-hs4f9" Nov 03 01:11:30.908944 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:11:30.906939 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8ba28520-d5f4-4ec9-8b86-f8a698325eea-workload-socket\") pod \"e2e-shared-a-w3-717217-openshift-default-578c69554c-hs4f9\" (UID: \"8ba28520-d5f4-4ec9-8b86-f8a698325eea\") " pod="openshift-ingress/e2e-shared-a-w3-717217-openshift-default-578c69554c-hs4f9" Nov 03 01:11:30.908944 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:11:30.908274 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8ba28520-d5f4-4ec9-8b86-f8a698325eea-istio-data\") pod \"e2e-shared-a-w3-717217-openshift-default-578c69554c-hs4f9\" (UID: \"8ba28520-d5f4-4ec9-8b86-f8a698325eea\") " pod="openshift-ingress/e2e-shared-a-w3-717217-openshift-default-578c69554c-hs4f9" Nov 03 01:11:30.912060 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:11:30.911999 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8ba28520-d5f4-4ec9-8b86-f8a698325eea-istio-podinfo\") pod \"e2e-shared-a-w3-717217-openshift-default-578c69554c-hs4f9\" (UID: \"8ba28520-d5f4-4ec9-8b86-f8a698325eea\") " pod="openshift-ingress/e2e-shared-a-w3-717217-openshift-default-578c69554c-hs4f9" Nov 03 01:11:30.912705 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:11:30.912681 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8ba28520-d5f4-4ec9-8b86-f8a698325eea-istio-envoy\") pod \"e2e-shared-a-w3-717217-openshift-default-578c69554c-hs4f9\" (UID: \"8ba28520-d5f4-4ec9-8b86-f8a698325eea\") " pod="openshift-ingress/e2e-shared-a-w3-717217-openshift-default-578c69554c-hs4f9" Nov 03 01:11:30.920783 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:11:30.920762 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cp7w8\" (UniqueName: \"kubernetes.io/projected/8ba28520-d5f4-4ec9-8b86-f8a698325eea-kube-api-access-cp7w8\") pod \"e2e-shared-a-w3-717217-openshift-default-578c69554c-hs4f9\" (UID: \"8ba28520-d5f4-4ec9-8b86-f8a698325eea\") " pod="openshift-ingress/e2e-shared-a-w3-717217-openshift-default-578c69554c-hs4f9" Nov 03 01:11:30.922312 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:11:30.922285 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8ba28520-d5f4-4ec9-8b86-f8a698325eea-istio-token\") pod \"e2e-shared-a-w3-717217-openshift-default-578c69554c-hs4f9\" (UID: \"8ba28520-d5f4-4ec9-8b86-f8a698325eea\") " pod="openshift-ingress/e2e-shared-a-w3-717217-openshift-default-578c69554c-hs4f9" Nov 03 01:11:31.033787 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:11:31.033751 4939 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w3-717217-openshift-default-578c69554c-hs4f9" Nov 03 01:11:31.578070 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:11:31.573062 4939 patch_prober.go:29] interesting pod/e2e-extmod-a-027c19-openshift-default-58d5bc7645-zjnf8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.207:15021/healthz/ready\": dial tcp 10.132.0.207:15021: connect: connection refused" start-of-body= Nov 03 01:11:31.578070 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:11:31.573138 4939 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-027c19-openshift-default-58d5bc7645-zjnf8" podUID="f09e7c2c-246e-4c96-a8ca-147e8ca83560" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.207:15021/healthz/ready\": dial tcp 10.132.0.207:15021: connect: connection refused" Nov 03 01:11:31.799056 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:11:31.790285 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:11:31.983900 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:11:31.982075 4939 patch_prober.go:29] interesting pod/e2e-extmod-a-7c759a-openshift-default-56576554df-wkcvp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.210:15021/healthz/ready\": dial tcp 10.132.0.210:15021: connect: connection refused" start-of-body= Nov 03 01:11:31.983900 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:11:31.982143 4939 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-7c759a-openshift-default-56576554df-wkcvp" podUID="6ea24c7b-6d13-4661-bdff-cd52792cb6b6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.210:15021/healthz/ready\": dial tcp 10.132.0.210:15021: connect: connection refused" Nov 03 01:11:32.297122 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:11:32.297035 4939 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-57079b-666f95c947-dv7mr" Nov 03 01:11:32.335027 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:11:32.324085 4939 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 01:11:32.335027 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:11:32.327617 4939 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 01:11:32.335027 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:11:32.327671 4939 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 01:11:32.335027 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:11:32.331469 4939 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-717217-openshift-default-578c69554c-hs4f9"] Nov 03 01:11:32.342587 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:11:32.342533 4939 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-88bbf7/distinct-2-w1-88bbf7-kserve-5457cdcdbd-l6tfd" podUID="a397931b-faa9-4c2e-8290-491350546663" containerName="main" probeResult="failure" output="Get \"https://10.132.0.85:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 01:11:32.528792 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:11:32.528747 4939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-717217-openshift-default-578c69554c-hs4f9" event={"ID":"8ba28520-d5f4-4ec9-8b86-f8a698325eea","Type":"ContainerStarted","Data":"8bec0ae3fed7271233d1f347b589c6fd69d8de79d13b5cf72772ed283b707e62"} Nov 03 01:11:32.541773 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:11:32.541739 4939 patch_prober.go:29] interesting pod/e2e-extmod-a-027c19-openshift-default-58d5bc7645-zjnf8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.207:15021/healthz/ready\": dial tcp 10.132.0.207:15021: connect: connection refused" start-of-body= Nov 03 01:11:32.542036 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:11:32.542005 4939 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-027c19-openshift-default-58d5bc7645-zjnf8" podUID="f09e7c2c-246e-4c96-a8ca-147e8ca83560" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.207:15021/healthz/ready\": dial tcp 10.132.0.207:15021: connect: connection refused" Nov 03 01:11:32.909162 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:11:32.909068 4939 patch_prober.go:29] interesting pod/e2e-extmod-a-7c759a-openshift-default-56576554df-wkcvp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.210:15021/healthz/ready\": dial tcp 10.132.0.210:15021: connect: connection refused" start-of-body= Nov 03 01:11:32.909413 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:11:32.909385 4939 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-7c759a-openshift-default-56576554df-wkcvp" podUID="6ea24c7b-6d13-4661-bdff-cd52792cb6b6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.210:15021/healthz/ready\": dial tcp 10.132.0.210:15021: connect: connection refused" Nov 03 01:11:32.974195 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:11:32.974085 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:11:33.471934 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:11:33.471536 4939 patch_prober.go:29] interesting pod/e2e-extmod-a-027c19-openshift-default-58d5bc7645-zjnf8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.207:15021/healthz/ready\": dial tcp 10.132.0.207:15021: connect: connection refused" start-of-body= Nov 03 01:11:33.471934 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:11:33.471598 4939 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-027c19-openshift-default-58d5bc7645-zjnf8" podUID="f09e7c2c-246e-4c96-a8ca-147e8ca83560" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.207:15021/healthz/ready\": dial tcp 10.132.0.207:15021: connect: connection refused" Nov 03 01:11:33.522322 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:11:33.522293 4939 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-extmod-a-d25759-5587dc5df9-tl5d6" Nov 03 01:11:33.542092 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:11:33.542062 4939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-717217-openshift-default-578c69554c-hs4f9" event={"ID":"8ba28520-d5f4-4ec9-8b86-f8a698325eea","Type":"ContainerStarted","Data":"d9710015fabade148b832b280406c56149499b9bb24448cde19a80f22e1d39bc"} Nov 03 01:11:33.622794 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:11:33.622728 4939 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w3-717217-openshift-default-578c69554c-hs4f9" podStartSLOduration=3.622705633 podStartE2EDuration="3.622705633s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 01:11:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 01:11:33.597305563 +0000 UTC m=+1841.861640531" watchObservedRunningTime="2026-11-03 01:11:33.622705633 +0000 UTC m=+1841.887040580" Nov 03 01:11:33.834919 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:11:33.834882 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:11:33.889928 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:11:33.888358 4939 patch_prober.go:29] interesting pod/e2e-extmod-a-7c759a-openshift-default-56576554df-wkcvp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.210:15021/healthz/ready\": dial tcp 10.132.0.210:15021: connect: connection refused" start-of-body= Nov 03 01:11:33.889928 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:11:33.888433 4939 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-7c759a-openshift-default-56576554df-wkcvp" podUID="6ea24c7b-6d13-4661-bdff-cd52792cb6b6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.210:15021/healthz/ready\": dial tcp 10.132.0.210:15021: connect: connection refused" Nov 03 01:11:34.038362 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:11:34.038322 4939 patch_prober.go:29] interesting pod/e2e-shared-a-w3-717217-openshift-default-578c69554c-hs4f9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.226:15021/healthz/ready\": dial tcp 10.132.0.226:15021: connect: connection refused" start-of-body= Nov 03 01:11:34.038658 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:11:34.038627 4939 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-717217-openshift-default-578c69554c-hs4f9" podUID="8ba28520-d5f4-4ec9-8b86-f8a698325eea" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.226:15021/healthz/ready\": dial tcp 10.132.0.226:15021: connect: connection refused" Nov 03 01:11:34.045239 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:11:34.044986 4939 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w3-717217-openshift-default-578c69554c-hs4f9" Nov 03 01:11:34.475455 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:11:34.475417 4939 patch_prober.go:29] interesting pod/e2e-extmod-a-027c19-openshift-default-58d5bc7645-zjnf8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.207:15021/healthz/ready\": dial tcp 10.132.0.207:15021: connect: connection refused" start-of-body= Nov 03 01:11:34.476085 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:11:34.476047 4939 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-027c19-openshift-default-58d5bc7645-zjnf8" podUID="f09e7c2c-246e-4c96-a8ca-147e8ca83560" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.207:15021/healthz/ready\": dial tcp 10.132.0.207:15021: connect: connection refused" Nov 03 01:11:34.777988 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:11:34.777005 4939 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-a-7c759a-openshift-default-56576554df-wkcvp" Nov 03 01:11:34.843506 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:11:34.843471 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:11:34.885932 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:11:34.875442 4939 patch_prober.go:29] interesting pod/e2e-extmod-a-7c759a-openshift-default-56576554df-wkcvp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.210:15021/healthz/ready\": dial tcp 10.132.0.210:15021: connect: connection refused" start-of-body= Nov 03 01:11:34.885932 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:11:34.875518 4939 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-7c759a-openshift-default-56576554df-wkcvp" podUID="6ea24c7b-6d13-4661-bdff-cd52792cb6b6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.210:15021/healthz/ready\": dial tcp 10.132.0.210:15021: connect: connection refused" Nov 03 01:11:35.035288 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:11:35.035198 4939 patch_prober.go:29] interesting pod/e2e-shared-a-w3-717217-openshift-default-578c69554c-hs4f9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.226:15021/healthz/ready\": dial tcp 10.132.0.226:15021: connect: connection refused" start-of-body= Nov 03 01:11:35.035554 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:11:35.035520 4939 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-717217-openshift-default-578c69554c-hs4f9" podUID="8ba28520-d5f4-4ec9-8b86-f8a698325eea" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.226:15021/healthz/ready\": dial tcp 10.132.0.226:15021: connect: connection refused" Nov 03 01:11:35.484947 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:11:35.483342 4939 patch_prober.go:29] interesting pod/e2e-extmod-a-027c19-openshift-default-58d5bc7645-zjnf8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.207:15021/healthz/ready\": dial tcp 10.132.0.207:15021: connect: connection refused" start-of-body= Nov 03 01:11:35.485825 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:11:35.485700 4939 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-027c19-openshift-default-58d5bc7645-zjnf8" podUID="f09e7c2c-246e-4c96-a8ca-147e8ca83560" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.207:15021/healthz/ready\": dial tcp 10.132.0.207:15021: connect: connection refused" Nov 03 01:11:35.814583 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:11:35.814491 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:11:36.036067 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:11:36.035979 4939 patch_prober.go:29] interesting pod/e2e-shared-a-w3-717217-openshift-default-578c69554c-hs4f9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.226:15021/healthz/ready\": dial tcp 10.132.0.226:15021: connect: connection refused" start-of-body= Nov 03 01:11:36.036067 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:11:36.036047 4939 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-717217-openshift-default-578c69554c-hs4f9" podUID="8ba28520-d5f4-4ec9-8b86-f8a698325eea" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.226:15021/healthz/ready\": dial tcp 10.132.0.226:15021: connect: connection refused" Nov 03 01:11:36.045019 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:11:36.044985 4939 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-extmod-a-7c759a-openshift-default-56576554df-wkcvp" Nov 03 01:11:36.068703 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:11:36.068607 4939 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-extmod-a-7c759a-openshift-default-56576554df-wkcvp" Nov 03 01:11:36.479968 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:11:36.479883 4939 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-extmod-a-027c19-openshift-default-58d5bc7645-zjnf8" Nov 03 01:11:36.480650 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:11:36.480606 4939 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-a-027c19-openshift-default-58d5bc7645-zjnf8" Nov 03 01:11:36.481596 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:11:36.481575 4939 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-extmod-a-027c19-openshift-default-58d5bc7645-zjnf8" Nov 03 01:11:36.738315 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:11:36.738229 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:11:36.890932 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:11:36.887872 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:11:37.039341 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:11:37.035147 4939 patch_prober.go:29] interesting pod/e2e-shared-a-w3-717217-openshift-default-578c69554c-hs4f9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.226:15021/healthz/ready\": dial tcp 10.132.0.226:15021: connect: connection refused" start-of-body= Nov 03 01:11:37.039341 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:11:37.035210 4939 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-717217-openshift-default-578c69554c-hs4f9" podUID="8ba28520-d5f4-4ec9-8b86-f8a698325eea" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.226:15021/healthz/ready\": dial tcp 10.132.0.226:15021: connect: connection refused" Nov 03 01:11:37.894251 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:11:37.894216 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:11:38.036936 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:11:38.036386 4939 patch_prober.go:29] interesting pod/e2e-shared-a-w3-717217-openshift-default-578c69554c-hs4f9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.226:15021/healthz/ready\": dial tcp 10.132.0.226:15021: connect: connection refused" start-of-body= Nov 03 01:11:38.036936 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:11:38.036464 4939 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-717217-openshift-default-578c69554c-hs4f9" podUID="8ba28520-d5f4-4ec9-8b86-f8a698325eea" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.226:15021/healthz/ready\": dial tcp 10.132.0.226:15021: connect: connection refused" Nov 03 01:11:38.625952 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:11:38.625890 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:11:38.807934 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:11:38.807319 4939 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-extmod-a-ddd653-84dfd5bd67-wv882" Nov 03 01:11:38.814980 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:11:38.814947 4939 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-ddd653-5764b8bfbc-fcxrz" Nov 03 01:11:39.034211 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:11:39.034138 4939 patch_prober.go:29] interesting pod/e2e-shared-a-w3-717217-openshift-default-578c69554c-hs4f9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.226:15021/healthz/ready\": dial tcp 10.132.0.226:15021: connect: connection refused" start-of-body= Nov 03 01:11:39.034650 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:11:39.034625 4939 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-717217-openshift-default-578c69554c-hs4f9" podUID="8ba28520-d5f4-4ec9-8b86-f8a698325eea" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.226:15021/healthz/ready\": dial tcp 10.132.0.226:15021: connect: connection refused" Nov 03 01:11:39.421990 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:11:39.421953 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:11:40.035132 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:11:40.035095 4939 patch_prober.go:29] interesting pod/e2e-shared-a-w3-717217-openshift-default-578c69554c-hs4f9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.226:15021/healthz/ready\": dial tcp 10.132.0.226:15021: connect: connection refused" start-of-body= Nov 03 01:11:40.035551 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:11:40.035155 4939 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-717217-openshift-default-578c69554c-hs4f9" podUID="8ba28520-d5f4-4ec9-8b86-f8a698325eea" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.226:15021/healthz/ready\": dial tcp 10.132.0.226:15021: connect: connection refused" Nov 03 01:11:40.194032 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:11:40.193995 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:11:40.789738 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:11:40.789699 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:11:41.039970 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:11:41.039878 4939 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w3-717217-openshift-default-578c69554c-hs4f9" Nov 03 01:11:41.040349 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:11:41.040329 4939 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w3-717217-openshift-default-578c69554c-hs4f9" Nov 03 01:11:41.041201 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:11:41.041183 4939 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w3-717217-openshift-default-578c69554c-hs4f9" Nov 03 01:11:41.534299 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:11:41.534230 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:11:41.672937 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:11:41.671615 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:11:41.784110 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:11:41.784068 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:11:42.709099 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:11:42.709065 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:11:43.386528 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:11:43.386491 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:11:43.661832 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:11:43.661748 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:11:44.095383 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:11:44.095327 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:11:44.727094 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:11:44.727061 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:11:44.884317 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:11:44.884272 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:11:45.866764 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:11:45.866732 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:11:46.591476 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:11:46.591438 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:11:47.223154 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:11:47.223120 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:11:47.884831 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:11:47.884784 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:11:48.043430 ip-10-0-140-107 kubenswrapper[4939]: E1103 01:11:48.043390 4939 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-sthjh" containerName="extractor" Nov 03 01:11:48.508787 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:11:48.508751 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:11:49.594027 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:11:49.593990 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:11:50.216667 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:11:50.216635 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:11:51.015853 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:11:51.015822 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:11:51.808667 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:11:51.808625 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:11:52.423877 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:11:52.423842 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:11:53.110820 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:11:53.110776 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:11:53.799576 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:11:53.799541 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:11:54.465084 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:11:54.465047 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:11:54.595885 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:11:54.595845 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:11:55.103179 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:11:55.103139 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:11:55.628275 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:11:55.628233 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:11:55.750270 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:11:55.750233 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:11:55.939314 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:11:55.939226 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:11:56.192990 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:11:56.192958 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:11:56.353802 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:11:56.353764 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:11:56.478069 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:11:56.475418 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:11:57.137933 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:11:57.126384 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:11:57.314032 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:11:57.312554 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:11:57.649930 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:11:57.645470 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:11:58.027502 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:11:58.027418 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:11:58.056815 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:11:58.056778 4939 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-b-3c5983/payload-processing-external-model-e2e-extmod-b-3c5983-7697p7dkr"] Nov 03 01:11:58.064131 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:11:58.064101 4939 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-3c5983/payload-processing-external-model-e2e-extmod-b-3c5983-7697p7dkr" Nov 03 01:11:58.068017 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:11:58.067987 4939 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-3c5983\"/\"agc-cred-803162a56fb2\"" Nov 03 01:11:58.068418 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:11:58.068393 4939 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-3c5983\"/\"payload-processing-external-model-e2e--ba2bb6eb-dockercfg-zkrj5\"" Nov 03 01:11:58.069447 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:11:58.069423 4939 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-3c5983\"/\"kube-root-ca.crt\"" Nov 03 01:11:58.069557 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:11:58.069443 4939 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-3c5983\"/\"routing-overlay\"" Nov 03 01:11:58.069557 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:11:58.069423 4939 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-3c5983\"/\"payload-processing-external-model-plugins-e2e-extmod-b-3c5983\"" Nov 03 01:11:58.069557 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:11:58.069494 4939 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-3c5983\"/\"openshift-service-ca.crt\"" Nov 03 01:11:58.072222 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:11:58.072198 4939 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-3c5983/payload-processing-external-model-e2e-extmod-b-3c5983-7697p7dkr"] Nov 03 01:11:58.189326 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:11:58.189286 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/c0770733-3323-42a9-8f86-619f71a276b6-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-3c5983-7697p7dkr\" (UID: \"c0770733-3323-42a9-8f86-619f71a276b6\") " pod="ai-tenant-e2e-extmod-b-3c5983/payload-processing-external-model-e2e-extmod-b-3c5983-7697p7dkr" Nov 03 01:11:58.189645 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:11:58.189429 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/c0770733-3323-42a9-8f86-619f71a276b6-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-3c5983-7697p7dkr\" (UID: \"c0770733-3323-42a9-8f86-619f71a276b6\") " pod="ai-tenant-e2e-extmod-b-3c5983/payload-processing-external-model-e2e-extmod-b-3c5983-7697p7dkr" Nov 03 01:11:58.189645 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:11:58.189604 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c0770733-3323-42a9-8f86-619f71a276b6-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-3c5983-7697p7dkr\" (UID: \"c0770733-3323-42a9-8f86-619f71a276b6\") " pod="ai-tenant-e2e-extmod-b-3c5983/payload-processing-external-model-e2e-extmod-b-3c5983-7697p7dkr" Nov 03 01:11:58.189792 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:11:58.189645 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hkhdq\" (UniqueName: \"kubernetes.io/projected/c0770733-3323-42a9-8f86-619f71a276b6-kube-api-access-hkhdq\") pod \"payload-processing-external-model-e2e-extmod-b-3c5983-7697p7dkr\" (UID: \"c0770733-3323-42a9-8f86-619f71a276b6\") " pod="ai-tenant-e2e-extmod-b-3c5983/payload-processing-external-model-e2e-extmod-b-3c5983-7697p7dkr" Nov 03 01:11:58.202271 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:11:58.202244 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:11:58.292977 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:11:58.291456 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c0770733-3323-42a9-8f86-619f71a276b6-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-3c5983-7697p7dkr\" (UID: \"c0770733-3323-42a9-8f86-619f71a276b6\") " pod="ai-tenant-e2e-extmod-b-3c5983/payload-processing-external-model-e2e-extmod-b-3c5983-7697p7dkr" Nov 03 01:11:58.292977 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:11:58.291507 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hkhdq\" (UniqueName: \"kubernetes.io/projected/c0770733-3323-42a9-8f86-619f71a276b6-kube-api-access-hkhdq\") pod \"payload-processing-external-model-e2e-extmod-b-3c5983-7697p7dkr\" (UID: \"c0770733-3323-42a9-8f86-619f71a276b6\") " pod="ai-tenant-e2e-extmod-b-3c5983/payload-processing-external-model-e2e-extmod-b-3c5983-7697p7dkr" Nov 03 01:11:58.292977 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:11:58.291857 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/c0770733-3323-42a9-8f86-619f71a276b6-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-3c5983-7697p7dkr\" (UID: \"c0770733-3323-42a9-8f86-619f71a276b6\") " pod="ai-tenant-e2e-extmod-b-3c5983/payload-processing-external-model-e2e-extmod-b-3c5983-7697p7dkr" Nov 03 01:11:58.292977 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:11:58.292016 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/c0770733-3323-42a9-8f86-619f71a276b6-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-3c5983-7697p7dkr\" (UID: \"c0770733-3323-42a9-8f86-619f71a276b6\") " pod="ai-tenant-e2e-extmod-b-3c5983/payload-processing-external-model-e2e-extmod-b-3c5983-7697p7dkr" Nov 03 01:11:58.292977 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:11:58.292881 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/c0770733-3323-42a9-8f86-619f71a276b6-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-3c5983-7697p7dkr\" (UID: \"c0770733-3323-42a9-8f86-619f71a276b6\") " pod="ai-tenant-e2e-extmod-b-3c5983/payload-processing-external-model-e2e-extmod-b-3c5983-7697p7dkr" Nov 03 01:11:58.306936 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:11:58.305459 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c0770733-3323-42a9-8f86-619f71a276b6-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-3c5983-7697p7dkr\" (UID: \"c0770733-3323-42a9-8f86-619f71a276b6\") " pod="ai-tenant-e2e-extmod-b-3c5983/payload-processing-external-model-e2e-extmod-b-3c5983-7697p7dkr" Nov 03 01:11:58.354606 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:11:58.351715 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hkhdq\" (UniqueName: \"kubernetes.io/projected/c0770733-3323-42a9-8f86-619f71a276b6-kube-api-access-hkhdq\") pod \"payload-processing-external-model-e2e-extmod-b-3c5983-7697p7dkr\" (UID: \"c0770733-3323-42a9-8f86-619f71a276b6\") " pod="ai-tenant-e2e-extmod-b-3c5983/payload-processing-external-model-e2e-extmod-b-3c5983-7697p7dkr" Nov 03 01:11:58.394935 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:11:58.382500 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/c0770733-3323-42a9-8f86-619f71a276b6-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-3c5983-7697p7dkr\" (UID: \"c0770733-3323-42a9-8f86-619f71a276b6\") " pod="ai-tenant-e2e-extmod-b-3c5983/payload-processing-external-model-e2e-extmod-b-3c5983-7697p7dkr" Nov 03 01:11:58.420175 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:11:58.399108 4939 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-3c5983/payload-processing-external-model-e2e-extmod-b-3c5983-7697p7dkr" Nov 03 01:11:58.792146 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:11:58.792021 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:11:58.905544 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:11:58.905493 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:11:58.962285 ip-10-0-140-107 kubenswrapper[4939]: W1103 01:11:58.962254 4939 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc0770733_3323_42a9_8f86_619f71a276b6.slice/crio-1b8e8da97999621e8dec4d712a0fb9544fcc7c2933de191a1d47264198fc2aff WatchSource:0}: Error finding container 1b8e8da97999621e8dec4d712a0fb9544fcc7c2933de191a1d47264198fc2aff: Status 404 returned error can't find the container with id 1b8e8da97999621e8dec4d712a0fb9544fcc7c2933de191a1d47264198fc2aff Nov 03 01:11:58.963847 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:11:58.963818 4939 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-3c5983/payload-processing-external-model-e2e-extmod-b-3c5983-7697p7dkr"] Nov 03 01:11:59.049219 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:11:59.049189 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:11:59.236966 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:11:59.236883 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:11:59.370146 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:11:59.370109 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:11:59.412250 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:11:59.412121 4939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-3c5983/payload-processing-external-model-e2e-extmod-b-3c5983-7697p7dkr" event={"ID":"c0770733-3323-42a9-8f86-619f71a276b6","Type":"ContainerStarted","Data":"bae354d39300ae52bf3b6e6197e34b4904f130eba9a4429852f716e3ab887031"} Nov 03 01:11:59.412250 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:11:59.412190 4939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-3c5983/payload-processing-external-model-e2e-extmod-b-3c5983-7697p7dkr" event={"ID":"c0770733-3323-42a9-8f86-619f71a276b6","Type":"ContainerStarted","Data":"1b8e8da97999621e8dec4d712a0fb9544fcc7c2933de191a1d47264198fc2aff"} Nov 03 01:11:59.412250 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:11:59.412217 4939 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-b-3c5983/payload-processing-external-model-e2e-extmod-b-3c5983-7697p7dkr" Nov 03 01:11:59.446036 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:11:59.445973 4939 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-b-3c5983/payload-processing-external-model-e2e-extmod-b-3c5983-7697p7dkr" podStartSLOduration=1.445897714 podStartE2EDuration="1.445897714s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 01:11:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 01:11:59.444501295 +0000 UTC m=+1867.708836239" watchObservedRunningTime="2026-11-03 01:11:59.445897714 +0000 UTC m=+1867.710232661" Nov 03 01:11:59.507163 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:11:59.507079 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:12:00.085329 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:12:00.085246 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:12:00.162493 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:12:00.162458 4939 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-4pw7v"] Nov 03 01:12:00.206368 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:12:00.206330 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:12:00.796952 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:12:00.796876 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:12:01.959933 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:12:01.956623 4939 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-f7680d/distinct-2-w2-f7680d-kserve-7d7c4564bf-cbv2c" podUID="2de18086-761b-4773-b9fc-15ddc4a220fc" containerName="main" probeResult="failure" output="Get \"https://10.132.0.75:8000/health\": context deadline exceeded" Nov 03 01:12:01.969935 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:12:01.964966 4939 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-f7680d/premium-w2-f7680d-kserve-76565b8cf9-hqbbt" podUID="4712e708-18a2-4a13-a12c-a5669397b2c3" containerName="main" probeResult="failure" output="Get \"https://10.132.0.63:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 01:12:02.053428 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:12:02.053384 4939 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-f7680d/distinct-w2-f7680d-kserve-756f874b98-qnpv8" podUID="836753fc-f9ef-4f61-b3fe-e5b92b7ee27b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.70:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 01:12:02.121097 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:12:02.121029 4939 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-88bbf7/simulator-w1-88bbf7-kserve-5fbb669f8f-42hgd" podUID="83b38883-2e7a-444d-aa68-52be4a25ae66" containerName="main" probeResult="failure" output="Get \"https://10.132.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 01:12:02.352120 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:12:02.352070 4939 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-88bbf7/distinct-2-w1-88bbf7-kserve-5457cdcdbd-l6tfd" podUID="a397931b-faa9-4c2e-8290-491350546663" containerName="main" probeResult="failure" output="Get \"https://10.132.0.85:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 01:12:02.501275 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:12:02.501229 4939 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-83755a/distinct-2-w3-83755a-kserve-6cf66b995d-2l47b" podUID="c3b05e88-5d5b-4d03-b28d-832f22e3c868" containerName="main" probeResult="failure" output="Get \"https://10.132.0.72:8000/ready\": context deadline exceeded" Nov 03 01:12:02.567159 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:12:02.567112 4939 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-f7680d/distinct-2-w2-f7680d-kserve-7d7c4564bf-cbv2c" podUID="2de18086-761b-4773-b9fc-15ddc4a220fc" containerName="main" probeResult="failure" output="Get \"https://10.132.0.75:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 01:12:02.831453 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:12:02.831409 4939 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-f7680d/premium-w2-f7680d-kserve-76565b8cf9-hqbbt" podUID="4712e708-18a2-4a13-a12c-a5669397b2c3" containerName="main" probeResult="failure" output="Get \"https://10.132.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 01:12:02.847717 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:12:02.847671 4939 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-2f9a3a/simulator-w0-2f9a3a-kserve-9596bbb8f-2trgz" podUID="207e2207-fb6c-4076-95df-72e95671c261" containerName="main" probeResult="failure" output="Get \"https://10.132.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 01:12:02.882605 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:12:02.882137 4939 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-2f9a3a/distinct-w0-2f9a3a-kserve-555565cfcf-jtcr9" podUID="ac0ece06-8718-463f-915f-5315988d9711" containerName="main" probeResult="failure" output="Get \"https://10.132.0.76:8000/ready\": context deadline exceeded" Nov 03 01:12:03.163484 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:12:03.163375 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:12:04.070838 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:12:04.070798 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:12:05.109903 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:12:05.109813 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:12:05.489990 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:12:05.489234 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:12:05.710963 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:12:05.710929 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:12:05.901406 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:12:05.901360 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:12:06.044457 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:12:06.044419 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:12:06.509935 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:12:06.508312 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:12:06.703933 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:12:06.703541 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:12:07.462932 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:12:07.462001 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:12:07.674935 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:12:07.671428 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:12:09.036938 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:12:09.029049 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:12:09.909049 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:12:09.909008 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:12:10.061937 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:12:10.061291 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:12:10.449189 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:12:10.449162 4939 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-b-3c5983/payload-processing-external-model-e2e-extmod-b-3c5983-7697p7dkr" Nov 03 01:12:10.786290 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:12:10.786168 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:12:11.309119 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:12:11.309055 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:12:11.774235 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:12:11.774145 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:12:11.847430 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:12:11.847028 4939 patch_prober.go:29] interesting pod/e2e-extmod-b-57079b-openshift-default-5c84d8878f-drtdm container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.218:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 01:12:11.847430 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:12:11.847118 4939 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-extmod-b-57079b-openshift-default-5c84d8878f-drtdm" podUID="a68659b3-912d-4df9-a162-fa6d5f949d2a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.218:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 01:12:11.895359 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:12:11.895322 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:12:12.035556 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:12:12.035440 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:12:13.122641 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:12:13.122600 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:12:13.896047 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:12:13.895995 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:12:14.681996 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:12:14.681962 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:12:14.831763 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:12:14.831727 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:12:15.626741 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:12:15.626702 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:12:15.987180 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:12:15.987074 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:12:16.409215 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:12:16.409176 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:12:16.529845 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:12:16.526676 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:12:17.380789 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:12:17.380685 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:12:18.039106 ip-10-0-140-107 kubenswrapper[4939]: E1103 01:12:18.039065 4939 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-sthjh" containerName="extractor" Nov 03 01:12:18.178923 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:12:18.178866 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:12:18.970094 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:12:18.970061 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:12:19.067933 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:12:19.058301 4939 patch_prober.go:29] interesting pod/e2e-extmod-a-d25759-openshift-default-d47fd9785-jcn5l container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.219:15021/healthz/ready\": dial tcp 10.132.0.219:15021: connect: connection refused" start-of-body= Nov 03 01:12:19.067933 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:12:19.058387 4939 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-extmod-a-d25759-openshift-default-d47fd9785-jcn5l" podUID="93ab25a5-4bed-4d84-9d62-640cea8d4047" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.219:15021/healthz/ready\": dial tcp 10.132.0.219:15021: connect: connection refused" Nov 03 01:12:19.097327 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:12:19.095796 4939 generic.go:356] "Generic (PLEG): container finished" podID="a68659b3-912d-4df9-a162-fa6d5f949d2a" containerID="a097be29f3ff470550b4dbcb5148e73d2e6ca303080ba91af7a7b5ff2045d80d" exitCode=0 Nov 03 01:12:19.097327 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:12:19.095888 4939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-57079b-openshift-default-5c84d8878f-drtdm" event={"ID":"a68659b3-912d-4df9-a162-fa6d5f949d2a","Type":"ContainerDied","Data":"a097be29f3ff470550b4dbcb5148e73d2e6ca303080ba91af7a7b5ff2045d80d"} Nov 03 01:12:19.097327 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:12:19.096356 4939 scope.go:122] "RemoveContainer" containerID="a097be29f3ff470550b4dbcb5148e73d2e6ca303080ba91af7a7b5ff2045d80d" Nov 03 01:12:19.099551 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:12:19.099506 4939 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 01:12:19.099703 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:12:19.099602 4939 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 01:12:19.099703 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:12:19.099655 4939 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 01:12:19.108227 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:12:19.104458 4939 generic.go:356] "Generic (PLEG): container finished" podID="93ab25a5-4bed-4d84-9d62-640cea8d4047" containerID="e6d4ddc1ac83d9addd06be80c75b25054577015f20147ea1379903004987cb75" exitCode=0 Nov 03 01:12:19.108227 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:12:19.104505 4939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-d25759-openshift-default-d47fd9785-jcn5l" event={"ID":"93ab25a5-4bed-4d84-9d62-640cea8d4047","Type":"ContainerDied","Data":"e6d4ddc1ac83d9addd06be80c75b25054577015f20147ea1379903004987cb75"} Nov 03 01:12:19.108227 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:12:19.105333 4939 scope.go:122] "RemoveContainer" containerID="e6d4ddc1ac83d9addd06be80c75b25054577015f20147ea1379903004987cb75" Nov 03 01:12:19.116257 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:12:19.110506 4939 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 01:12:19.116257 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:12:19.110727 4939 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 01:12:19.116257 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:12:19.110846 4939 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 01:12:19.809200 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:12:19.809158 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:12:19.988331 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:12:19.988197 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:12:20.126746 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:12:20.126700 4939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-57079b-openshift-default-5c84d8878f-drtdm" event={"ID":"a68659b3-912d-4df9-a162-fa6d5f949d2a","Type":"ContainerStarted","Data":"339bc7b99b219889b6a9d290a2a36b76ab9c5eb0d5ad09fc64724f2574ac1ff3"} Nov 03 01:12:20.129400 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:12:20.129362 4939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-d25759-openshift-default-d47fd9785-jcn5l" event={"ID":"93ab25a5-4bed-4d84-9d62-640cea8d4047","Type":"ContainerStarted","Data":"70d4bb7ccd830af7d7e57b795e8429cff2c52c2fc6329d04c9360c781027bcba"} Nov 03 01:12:20.139729 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:12:20.139701 4939 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-extmod-b-57079b-openshift-default-5c84d8878f-drtdm" Nov 03 01:12:20.139898 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:12:20.139874 4939 patch_prober.go:29] interesting pod/e2e-extmod-b-57079b-openshift-default-5c84d8878f-drtdm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.218:15021/healthz/ready\": dial tcp 10.132.0.218:15021: connect: connection refused" start-of-body= Nov 03 01:12:20.139974 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:12:20.139960 4939 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-57079b-openshift-default-5c84d8878f-drtdm" podUID="a68659b3-912d-4df9-a162-fa6d5f949d2a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.218:15021/healthz/ready\": dial tcp 10.132.0.218:15021: connect: connection refused" Nov 03 01:12:20.331275 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:12:20.331235 4939 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-extmod-a-d25759-openshift-default-d47fd9785-jcn5l" Nov 03 01:12:20.331450 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:12:20.331361 4939 patch_prober.go:29] interesting pod/e2e-extmod-a-d25759-openshift-default-d47fd9785-jcn5l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.219:15021/healthz/ready\": dial tcp 10.132.0.219:15021: connect: connection refused" start-of-body= Nov 03 01:12:20.331450 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:12:20.331398 4939 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-d25759-openshift-default-d47fd9785-jcn5l" podUID="93ab25a5-4bed-4d84-9d62-640cea8d4047" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.219:15021/healthz/ready\": dial tcp 10.132.0.219:15021: connect: connection refused" Nov 03 01:12:20.636525 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:12:20.636477 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:12:21.139902 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:12:21.139860 4939 patch_prober.go:29] interesting pod/e2e-extmod-b-57079b-openshift-default-5c84d8878f-drtdm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.218:15021/healthz/ready\": dial tcp 10.132.0.218:15021: connect: connection refused" start-of-body= Nov 03 01:12:21.140335 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:12:21.139965 4939 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-57079b-openshift-default-5c84d8878f-drtdm" podUID="a68659b3-912d-4df9-a162-fa6d5f949d2a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.218:15021/healthz/ready\": dial tcp 10.132.0.218:15021: connect: connection refused" Nov 03 01:12:21.329433 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:12:21.329396 4939 patch_prober.go:29] interesting pod/e2e-extmod-a-d25759-openshift-default-d47fd9785-jcn5l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.219:15021/healthz/ready\": dial tcp 10.132.0.219:15021: connect: connection refused" start-of-body= Nov 03 01:12:21.329659 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:12:21.329449 4939 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-d25759-openshift-default-d47fd9785-jcn5l" podUID="93ab25a5-4bed-4d84-9d62-640cea8d4047" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.219:15021/healthz/ready\": dial tcp 10.132.0.219:15021: connect: connection refused" Nov 03 01:12:21.469083 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:12:21.468831 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:12:22.140170 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:12:22.140133 4939 patch_prober.go:29] interesting pod/e2e-extmod-b-57079b-openshift-default-5c84d8878f-drtdm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.218:15021/healthz/ready\": dial tcp 10.132.0.218:15021: connect: connection refused" start-of-body= Nov 03 01:12:22.140725 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:12:22.140439 4939 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-57079b-openshift-default-5c84d8878f-drtdm" podUID="a68659b3-912d-4df9-a162-fa6d5f949d2a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.218:15021/healthz/ready\": dial tcp 10.132.0.218:15021: connect: connection refused" Nov 03 01:12:22.315327 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:12:22.315288 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:12:22.330127 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:12:22.330095 4939 patch_prober.go:29] interesting pod/e2e-extmod-a-d25759-openshift-default-d47fd9785-jcn5l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.219:15021/healthz/ready\": dial tcp 10.132.0.219:15021: connect: connection refused" start-of-body= Nov 03 01:12:22.330295 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:12:22.330163 4939 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-d25759-openshift-default-d47fd9785-jcn5l" podUID="93ab25a5-4bed-4d84-9d62-640cea8d4047" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.219:15021/healthz/ready\": dial tcp 10.132.0.219:15021: connect: connection refused" Nov 03 01:12:23.140328 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:12:23.140298 4939 patch_prober.go:29] interesting pod/e2e-extmod-b-57079b-openshift-default-5c84d8878f-drtdm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.218:15021/healthz/ready\": dial tcp 10.132.0.218:15021: connect: connection refused" start-of-body= Nov 03 01:12:23.140801 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:12:23.140775 4939 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-57079b-openshift-default-5c84d8878f-drtdm" podUID="a68659b3-912d-4df9-a162-fa6d5f949d2a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.218:15021/healthz/ready\": dial tcp 10.132.0.218:15021: connect: connection refused" Nov 03 01:12:23.220586 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:12:23.220550 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:12:23.333934 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:12:23.331295 4939 patch_prober.go:29] interesting pod/e2e-extmod-a-d25759-openshift-default-d47fd9785-jcn5l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.219:15021/healthz/ready\": dial tcp 10.132.0.219:15021: connect: connection refused" start-of-body= Nov 03 01:12:23.333934 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:12:23.331374 4939 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-d25759-openshift-default-d47fd9785-jcn5l" podUID="93ab25a5-4bed-4d84-9d62-640cea8d4047" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.219:15021/healthz/ready\": dial tcp 10.132.0.219:15021: connect: connection refused" Nov 03 01:12:23.562988 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:12:23.561869 4939 patch_prober.go:29] interesting pod/e2e-extmod-a-ddd653-openshift-default-779ff6465-t9cvw container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.222:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 01:12:23.562988 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:12:23.561974 4939 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-extmod-a-ddd653-openshift-default-779ff6465-t9cvw" podUID="aa64537e-0c5b-49ae-9547-df65d9ea4b2d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.222:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 01:12:24.139880 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:12:24.139831 4939 patch_prober.go:29] interesting pod/e2e-extmod-b-57079b-openshift-default-5c84d8878f-drtdm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.218:15021/healthz/ready\": dial tcp 10.132.0.218:15021: connect: connection refused" start-of-body= Nov 03 01:12:24.140069 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:12:24.139957 4939 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-57079b-openshift-default-5c84d8878f-drtdm" podUID="a68659b3-912d-4df9-a162-fa6d5f949d2a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.218:15021/healthz/ready\": dial tcp 10.132.0.218:15021: connect: connection refused" Nov 03 01:12:24.183506 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:12:24.183398 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:12:24.335938 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:12:24.333441 4939 patch_prober.go:29] interesting pod/e2e-extmod-a-d25759-openshift-default-d47fd9785-jcn5l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.219:15021/healthz/ready\": dial tcp 10.132.0.219:15021: connect: connection refused" start-of-body= Nov 03 01:12:24.335938 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:12:24.333536 4939 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-d25759-openshift-default-d47fd9785-jcn5l" podUID="93ab25a5-4bed-4d84-9d62-640cea8d4047" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.219:15021/healthz/ready\": dial tcp 10.132.0.219:15021: connect: connection refused" Nov 03 01:12:25.140928 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:12:25.140564 4939 patch_prober.go:29] interesting pod/e2e-extmod-b-57079b-openshift-default-5c84d8878f-drtdm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.218:15021/healthz/ready\": dial tcp 10.132.0.218:15021: connect: connection refused" start-of-body= Nov 03 01:12:25.140928 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:12:25.140630 4939 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-57079b-openshift-default-5c84d8878f-drtdm" podUID="a68659b3-912d-4df9-a162-fa6d5f949d2a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.218:15021/healthz/ready\": dial tcp 10.132.0.218:15021: connect: connection refused" Nov 03 01:12:25.329868 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:12:25.329830 4939 patch_prober.go:29] interesting pod/e2e-extmod-a-d25759-openshift-default-d47fd9785-jcn5l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.219:15021/healthz/ready\": dial tcp 10.132.0.219:15021: connect: connection refused" start-of-body= Nov 03 01:12:25.330305 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:12:25.329884 4939 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-d25759-openshift-default-d47fd9785-jcn5l" podUID="93ab25a5-4bed-4d84-9d62-640cea8d4047" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.219:15021/healthz/ready\": dial tcp 10.132.0.219:15021: connect: connection refused" Nov 03 01:12:25.484993 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:12:25.483642 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:12:25.798214 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:12:25.798133 4939 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-b-57079b-openshift-default-5c84d8878f-drtdm" Nov 03 01:12:26.044004 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:12:26.006964 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:12:26.144691 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:12:26.144658 4939 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-extmod-b-57079b-openshift-default-5c84d8878f-drtdm" Nov 03 01:12:26.335576 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:12:26.335548 4939 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-extmod-a-d25759-openshift-default-d47fd9785-jcn5l" Nov 03 01:12:26.336126 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:12:26.336023 4939 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-a-d25759-openshift-default-d47fd9785-jcn5l" Nov 03 01:12:26.336862 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:12:26.336840 4939 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-extmod-a-d25759-openshift-default-d47fd9785-jcn5l" Nov 03 01:12:26.419984 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:12:26.415878 4939 generic.go:356] "Generic (PLEG): container finished" podID="aa64537e-0c5b-49ae-9547-df65d9ea4b2d" containerID="eb4593fecdfedbac58d15758fcc9bfdb55ac993ebc532214ba94ace99e5439a4" exitCode=0 Nov 03 01:12:26.419984 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:12:26.417236 4939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-ddd653-openshift-default-779ff6465-t9cvw" event={"ID":"aa64537e-0c5b-49ae-9547-df65d9ea4b2d","Type":"ContainerDied","Data":"eb4593fecdfedbac58d15758fcc9bfdb55ac993ebc532214ba94ace99e5439a4"} Nov 03 01:12:26.419984 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:12:26.417633 4939 scope.go:122] "RemoveContainer" containerID="eb4593fecdfedbac58d15758fcc9bfdb55ac993ebc532214ba94ace99e5439a4" Nov 03 01:12:26.420781 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:12:26.420699 4939 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 01:12:26.420781 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:12:26.420771 4939 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 01:12:26.420934 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:12:26.420806 4939 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 01:12:26.423287 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:12:26.421605 4939 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-extmod-b-57079b-openshift-default-5c84d8878f-drtdm" Nov 03 01:12:26.815988 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:12:26.808393 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:12:27.426708 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:12:27.426667 4939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-ddd653-openshift-default-779ff6465-t9cvw" event={"ID":"aa64537e-0c5b-49ae-9547-df65d9ea4b2d","Type":"ContainerStarted","Data":"688beffd8ffb7850f456be4cccfdcedec5d5b8f6c96c9a094d56e7e2675fc8c8"} Nov 03 01:12:27.522340 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:12:27.522302 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:12:28.196968 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:12:28.196868 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:12:28.251939 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:12:28.246390 4939 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-extmod-a-ddd653-openshift-default-779ff6465-t9cvw" Nov 03 01:12:28.251939 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:12:28.246532 4939 patch_prober.go:29] interesting pod/e2e-extmod-a-ddd653-openshift-default-779ff6465-t9cvw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.222:15021/healthz/ready\": dial tcp 10.132.0.222:15021: connect: connection refused" start-of-body= Nov 03 01:12:28.251939 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:12:28.246597 4939 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-ddd653-openshift-default-779ff6465-t9cvw" podUID="aa64537e-0c5b-49ae-9547-df65d9ea4b2d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.222:15021/healthz/ready\": dial tcp 10.132.0.222:15021: connect: connection refused" Nov 03 01:12:28.995214 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:12:28.995178 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:12:29.244808 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:12:29.244763 4939 patch_prober.go:29] interesting pod/e2e-extmod-a-ddd653-openshift-default-779ff6465-t9cvw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.222:15021/healthz/ready\": dial tcp 10.132.0.222:15021: connect: connection refused" start-of-body= Nov 03 01:12:29.245020 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:12:29.244849 4939 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-ddd653-openshift-default-779ff6465-t9cvw" podUID="aa64537e-0c5b-49ae-9547-df65d9ea4b2d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.222:15021/healthz/ready\": dial tcp 10.132.0.222:15021: connect: connection refused" Nov 03 01:12:29.773119 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:12:29.773053 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:12:30.175108 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:12:30.175062 4939 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-4pw7v"] Nov 03 01:12:30.245792 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:12:30.245752 4939 patch_prober.go:29] interesting pod/e2e-extmod-a-ddd653-openshift-default-779ff6465-t9cvw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.222:15021/healthz/ready\": dial tcp 10.132.0.222:15021: connect: connection refused" start-of-body= Nov 03 01:12:30.246011 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:12:30.245810 4939 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-ddd653-openshift-default-779ff6465-t9cvw" podUID="aa64537e-0c5b-49ae-9547-df65d9ea4b2d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.222:15021/healthz/ready\": dial tcp 10.132.0.222:15021: connect: connection refused" Nov 03 01:12:30.672128 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:12:30.663997 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:12:31.245476 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:12:31.245434 4939 patch_prober.go:29] interesting pod/e2e-extmod-a-ddd653-openshift-default-779ff6465-t9cvw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.222:15021/healthz/ready\": dial tcp 10.132.0.222:15021: connect: connection refused" start-of-body= Nov 03 01:12:31.245951 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:12:31.245512 4939 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-ddd653-openshift-default-779ff6465-t9cvw" podUID="aa64537e-0c5b-49ae-9547-df65d9ea4b2d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.222:15021/healthz/ready\": dial tcp 10.132.0.222:15021: connect: connection refused" Nov 03 01:12:32.244991 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:12:32.244952 4939 patch_prober.go:29] interesting pod/e2e-extmod-a-ddd653-openshift-default-779ff6465-t9cvw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.222:15021/healthz/ready\": dial tcp 10.132.0.222:15021: connect: connection refused" start-of-body= Nov 03 01:12:32.245176 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:12:32.245004 4939 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-ddd653-openshift-default-779ff6465-t9cvw" podUID="aa64537e-0c5b-49ae-9547-df65d9ea4b2d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.222:15021/healthz/ready\": dial tcp 10.132.0.222:15021: connect: connection refused" Nov 03 01:12:32.320080 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:12:32.320043 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:12:33.245011 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:12:33.244974 4939 patch_prober.go:29] interesting pod/e2e-extmod-a-ddd653-openshift-default-779ff6465-t9cvw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.222:15021/healthz/ready\": dial tcp 10.132.0.222:15021: connect: connection refused" start-of-body= Nov 03 01:12:33.245198 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:12:33.245032 4939 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-ddd653-openshift-default-779ff6465-t9cvw" podUID="aa64537e-0c5b-49ae-9547-df65d9ea4b2d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.222:15021/healthz/ready\": dial tcp 10.132.0.222:15021: connect: connection refused" Nov 03 01:12:33.454945 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:12:33.453766 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:12:34.265255 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:12:34.264610 4939 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-extmod-a-ddd653-openshift-default-779ff6465-t9cvw" Nov 03 01:12:34.267078 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:12:34.266900 4939 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-a-ddd653-openshift-default-779ff6465-t9cvw" Nov 03 01:12:34.275720 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:12:34.275129 4939 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-extmod-a-ddd653-openshift-default-779ff6465-t9cvw" Nov 03 01:12:34.305395 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:12:34.305364 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:12:35.103109 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:12:35.103073 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:12:35.926433 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:12:35.926394 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:12:36.608686 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:12:36.608652 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:12:37.305672 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:12:37.305632 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:12:38.036982 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:12:38.036943 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:12:38.776852 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:12:38.776812 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:12:39.461006 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:12:39.460943 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:12:40.201510 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:12:40.201471 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:12:40.887483 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:12:40.887440 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:12:41.600438 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:12:41.600407 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:12:42.362934 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:12:42.356671 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:12:42.795111 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:12:42.795024 4939 generic.go:356] "Generic (PLEG): container finished" podID="78a38f10-6762-489e-813a-ac1cb91f5844" containerID="2379ea793c99a3d43b7adbe002e932cb7452b8a0597b970f11c7f367a61f7b14" exitCode=0 Nov 03 01:12:42.795111 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:12:42.795093 4939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-3c5983-openshift-default-86c5bc855c-2hx8m" event={"ID":"78a38f10-6762-489e-813a-ac1cb91f5844","Type":"ContainerDied","Data":"2379ea793c99a3d43b7adbe002e932cb7452b8a0597b970f11c7f367a61f7b14"} Nov 03 01:12:42.795574 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:12:42.795559 4939 scope.go:122] "RemoveContainer" containerID="2379ea793c99a3d43b7adbe002e932cb7452b8a0597b970f11c7f367a61f7b14" Nov 03 01:12:42.797573 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:12:42.797540 4939 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 01:12:42.797670 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:12:42.797615 4939 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 01:12:42.797670 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:12:42.797650 4939 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 01:12:43.257933 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:12:43.250666 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:12:43.769609 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:12:43.769574 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:12:43.805757 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:12:43.805721 4939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-3c5983-openshift-default-86c5bc855c-2hx8m" event={"ID":"78a38f10-6762-489e-813a-ac1cb91f5844","Type":"ContainerStarted","Data":"d25cdbfedd2c4039d8e796f62d8dd7d44fa46d1f7ff0d34c9badaa9eb7301a3d"} Nov 03 01:12:44.150843 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:12:44.150649 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:12:44.706104 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:12:44.706066 4939 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-extmod-b-3c5983-openshift-default-86c5bc855c-2hx8m" Nov 03 01:12:44.706972 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:12:44.706946 4939 patch_prober.go:29] interesting pod/e2e-extmod-b-3c5983-openshift-default-86c5bc855c-2hx8m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.214:15021/healthz/ready\": dial tcp 10.132.0.214:15021: connect: connection refused" start-of-body= Nov 03 01:12:44.707105 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:12:44.706996 4939 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-3c5983-openshift-default-86c5bc855c-2hx8m" podUID="78a38f10-6762-489e-813a-ac1cb91f5844" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.214:15021/healthz/ready\": dial tcp 10.132.0.214:15021: connect: connection refused" Nov 03 01:12:44.811699 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:12:44.811656 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:12:45.599535 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:12:45.599502 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:12:45.709432 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:12:45.709384 4939 patch_prober.go:29] interesting pod/e2e-extmod-b-3c5983-openshift-default-86c5bc855c-2hx8m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.214:15021/healthz/ready\": dial tcp 10.132.0.214:15021: connect: connection refused" start-of-body= Nov 03 01:12:45.709767 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:12:45.709724 4939 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-3c5983-openshift-default-86c5bc855c-2hx8m" podUID="78a38f10-6762-489e-813a-ac1cb91f5844" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.214:15021/healthz/ready\": dial tcp 10.132.0.214:15021: connect: connection refused" Nov 03 01:12:46.478890 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:12:46.478847 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:12:46.706442 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:12:46.706406 4939 patch_prober.go:29] interesting pod/e2e-extmod-b-3c5983-openshift-default-86c5bc855c-2hx8m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.214:15021/healthz/ready\": dial tcp 10.132.0.214:15021: connect: connection refused" start-of-body= Nov 03 01:12:46.706628 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:12:46.706465 4939 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-3c5983-openshift-default-86c5bc855c-2hx8m" podUID="78a38f10-6762-489e-813a-ac1cb91f5844" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.214:15021/healthz/ready\": dial tcp 10.132.0.214:15021: connect: connection refused" Nov 03 01:12:47.117171 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:12:47.117138 4939 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-b-3c5983-openshift-default-86c5bc855c-2hx8m" Nov 03 01:12:47.145372 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:12:47.145330 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:12:47.705658 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:12:47.705626 4939 patch_prober.go:29] interesting pod/e2e-extmod-b-3c5983-openshift-default-86c5bc855c-2hx8m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.214:15021/healthz/ready\": dial tcp 10.132.0.214:15021: connect: connection refused" start-of-body= Nov 03 01:12:47.706239 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:12:47.705680 4939 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-3c5983-openshift-default-86c5bc855c-2hx8m" podUID="78a38f10-6762-489e-813a-ac1cb91f5844" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.214:15021/healthz/ready\": dial tcp 10.132.0.214:15021: connect: connection refused" Nov 03 01:12:48.035742 ip-10-0-140-107 kubenswrapper[4939]: E1103 01:12:48.035655 4939 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-sthjh" containerName="extractor" Nov 03 01:12:48.167448 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:12:48.167373 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:12:48.705758 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:12:48.705716 4939 patch_prober.go:29] interesting pod/e2e-extmod-b-3c5983-openshift-default-86c5bc855c-2hx8m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.214:15021/healthz/ready\": dial tcp 10.132.0.214:15021: connect: connection refused" start-of-body= Nov 03 01:12:48.706206 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:12:48.705792 4939 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-3c5983-openshift-default-86c5bc855c-2hx8m" podUID="78a38f10-6762-489e-813a-ac1cb91f5844" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.214:15021/healthz/ready\": dial tcp 10.132.0.214:15021: connect: connection refused" Nov 03 01:12:49.010551 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:12:49.010470 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:12:49.673153 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:12:49.673122 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:12:49.751977 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:12:49.750449 4939 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-extmod-b-3c5983-openshift-default-86c5bc855c-2hx8m" Nov 03 01:12:49.765932 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:12:49.762015 4939 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-extmod-b-3c5983-openshift-default-86c5bc855c-2hx8m" Nov 03 01:12:50.419384 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:12:50.419347 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:12:51.097416 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:12:51.097384 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:12:51.791992 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:12:51.791961 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:12:52.591743 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:12:52.591703 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:12:53.366265 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:12:53.366227 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:12:54.011583 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:12:54.011554 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:12:54.705273 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:12:54.705240 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:12:55.389048 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:12:55.389018 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:12:56.062210 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:12:56.062162 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:12:56.826259 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:12:56.826225 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:12:57.493568 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:12:57.493495 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:12:58.143971 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:12:58.143938 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:12:58.852142 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:12:58.852104 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:12:59.572812 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:12:59.572779 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:13:00.307060 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:13:00.307029 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:13:00.985509 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:13:00.985477 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:13:01.607866 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:13:01.607835 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:13:02.299315 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:13:02.299261 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:13:02.989040 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:13:02.988957 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:13:03.689592 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:13:03.689556 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:13:04.390303 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:13:04.390273 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:13:05.181749 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:13:05.181703 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:13:05.778409 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:13:05.778373 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:13:06.493288 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:13:06.493250 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:13:07.294364 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:13:07.294331 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:13:08.095657 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:13:08.095623 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:13:08.703966 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:13:08.703873 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:13:09.405628 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:13:09.405582 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:13:10.028752 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:13:10.028712 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:13:10.376669 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:13:10.376629 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:13:10.494742 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:13:10.494693 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:13:10.695210 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:13:10.695117 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:13:11.321689 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:13:11.321645 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:13:12.008374 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:13:12.008343 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:13:12.797611 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:13:12.794982 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:13:13.385281 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:13:13.385233 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:13:14.092094 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:13:14.092061 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:13:14.901407 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:13:14.901367 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:13:15.671725 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:13:15.671689 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:13:16.411141 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:13:16.411103 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:13:17.102949 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:13:17.102902 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:13:17.771177 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:13:17.771137 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:13:18.033536 ip-10-0-140-107 kubenswrapper[4939]: E1103 01:13:18.033444 4939 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-sthjh" containerName="extractor" Nov 03 01:13:18.384842 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:13:18.384780 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:13:19.006963 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:13:19.006885 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:13:19.761035 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:13:19.761001 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:13:20.375748 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:13:20.375716 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:13:21.085576 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:13:21.085536 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:13:21.783440 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:13:21.783405 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:13:22.385204 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:13:22.385171 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:13:23.168316 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:13:23.168228 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:13:23.872397 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:13:23.872334 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:13:24.473528 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:13:24.473476 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:13:25.283475 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:13:25.283437 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:13:25.902096 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:13:25.902060 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:13:26.405190 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:13:26.405154 4939 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-4pw7v"] Nov 03 01:13:26.613900 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:13:26.613851 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:13:28.163826 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:13:28.163796 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:13:28.806249 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:13:28.806212 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:13:29.587396 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:13:29.587366 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:13:29.733245 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:13:29.733205 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:13:29.863000 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:13:29.862856 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:13:30.122104 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:13:30.122015 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:13:30.417278 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:13:30.417191 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:13:30.844337 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:13:30.844302 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:13:30.985929 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:13:30.982803 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:13:31.605103 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:13:31.605062 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:13:31.730935 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:13:31.727484 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:13:32.002093 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:13:32.002006 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:13:32.127028 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:13:32.126987 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:13:32.345430 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:13:32.345389 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:13:33.351331 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:13:33.351300 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:13:34.024246 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:13:34.024206 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:13:34.722500 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:13:34.722470 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:13:35.364281 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:13:35.364247 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:13:36.019544 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:13:36.019514 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:13:36.882549 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:13:36.882515 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:13:37.690814 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:13:37.690778 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:13:38.307312 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:13:38.307282 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:13:38.942944 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:13:38.942878 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:13:39.613188 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:13:39.613149 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:13:40.257940 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:13:40.257888 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:13:40.994402 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:13:40.994371 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:13:41.687642 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:13:41.687599 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:13:42.389449 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:13:42.389413 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:13:43.185597 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:13:43.185557 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:13:43.770658 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:13:43.770626 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:13:43.928935 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:13:43.926649 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:13:44.487227 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:13:44.487193 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:13:45.207264 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:13:45.207235 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:13:45.892318 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:13:45.892286 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:13:47.212029 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:13:47.211957 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:13:47.904708 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:13:47.904673 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:13:48.069007 ip-10-0-140-107 kubenswrapper[4939]: E1103 01:13:48.068968 4939 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-sthjh" containerName="extractor" Nov 03 01:13:48.688832 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:13:48.688798 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:13:49.289341 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:13:49.289310 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:13:49.893497 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:13:49.893464 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:13:50.598450 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:13:50.598418 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:13:51.354560 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:13:51.354518 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:13:51.947934 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:13:51.947871 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:13:52.609627 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:13:52.609599 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:13:53.288475 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:13:53.288442 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:13:53.930202 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:13:53.930172 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:13:54.800257 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:13:54.800221 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:13:55.685438 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:13:55.685401 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:13:56.348684 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:13:56.348642 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:13:57.007036 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:13:57.007001 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:13:57.652732 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:13:57.652687 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:13:58.269391 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:13:58.269356 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:13:59.012590 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:13:59.012510 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:13:59.875217 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:13:59.875186 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:14:00.479517 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:14:00.479481 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:14:01.293477 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:14:01.293439 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:14:01.886476 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:14:01.886435 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:14:02.583046 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:14:02.583001 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:14:03.279893 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:14:03.279852 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:14:03.980019 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:14:03.979981 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:14:04.670283 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:14:04.670252 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:14:05.424882 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:14:05.424848 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:14:06.079006 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:14:06.078978 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:14:06.861955 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:14:06.854138 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:14:07.493359 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:14:07.493326 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:14:08.098199 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:14:08.098165 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:14:08.906080 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:14:08.906019 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:14:09.897862 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:14:09.897827 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:14:10.802106 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:14:10.802073 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:14:10.934935 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:14:10.933071 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:14:11.750134 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:14:11.747301 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:14:12.045206 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:14:12.045128 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:14:12.158261 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:14:12.158215 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:14:12.909758 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:14:12.909726 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:14:13.439158 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:14:13.439124 4939 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-4pw7v"] Nov 03 01:14:13.744877 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:14:13.744775 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:14:15.292050 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:14:15.292008 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:14:15.950014 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:14:15.949965 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:14:16.576880 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:14:16.576836 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:14:16.740973 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:14:16.740903 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:14:17.587844 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:14:17.587793 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:14:17.707366 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:14:17.707333 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:14:18.230561 ip-10-0-140-107 kubenswrapper[4939]: E1103 01:14:18.230498 4939 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-sthjh" containerName="extractor" Nov 03 01:14:18.314315 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:14:18.314282 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:14:18.424596 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:14:18.424556 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:14:19.581686 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:14:19.581657 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:14:20.269593 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:14:20.269559 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:14:20.927510 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:14:20.927444 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:14:21.609994 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:14:21.609946 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:14:22.223647 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:14:22.223593 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:14:22.867780 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:14:22.867746 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:14:23.552441 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:14:23.552406 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:14:24.209457 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:14:24.209420 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:14:24.708737 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:14:24.708687 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:14:25.499184 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:14:25.499142 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:14:26.299361 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:14:26.299326 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:14:27.053491 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:14:27.053450 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:14:27.682712 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:14:27.682656 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:14:28.387315 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:14:28.387276 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:14:29.070843 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:14:29.070805 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:14:29.190107 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:14:29.190069 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:14:30.678955 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:14:30.678044 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:14:30.844645 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:14:30.844605 4939 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w1-2729f2-openshift-default-9fb97b4fc-czgh9"] Nov 03 01:14:30.844851 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:14:30.844803 4939 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w1-2729f2-openshift-default-9fb97b4fc-czgh9" podUID="66c78580-7ad7-49b8-acf7-b23dd33a8af9" containerName="istio-proxy" containerID="cri-o://01c6cd08a5a26c9754fd2f2e153053cf9c6cfb98437cb054bcaf8f98f044b406" gracePeriod=30 Nov 03 01:14:31.476754 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:14:31.476721 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:14:31.605051 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:14:31.605012 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:14:31.716253 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:14:31.716221 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:14:32.592717 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:14:32.592680 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:14:34.110639 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:14:34.110593 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:14:34.908291 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:14:34.908258 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:14:35.664435 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:14:35.664397 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:14:36.160019 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:14:36.159991 4939 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w1-2729f2-openshift-default-9fb97b4fc-czgh9" Nov 03 01:14:36.195141 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:14:36.195106 4939 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-b-57079b/payload-processing-external-model-e2e-extmod-b-57079b-59f8rtzcd"] Nov 03 01:14:36.196819 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:14:36.196792 4939 cpu_manager.go:414] "containerMap: removing container" podUID="66c78580-7ad7-49b8-acf7-b23dd33a8af9" containerName="istio-proxy" Nov 03 01:14:36.196819 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:14:36.196818 4939 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="66c78580-7ad7-49b8-acf7-b23dd33a8af9" containerName="istio-proxy" Nov 03 01:14:36.197086 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:14:36.197071 4939 memory_manager.go:365] "RemoveStaleState removing state" podUID="66c78580-7ad7-49b8-acf7-b23dd33a8af9" containerName="istio-proxy" Nov 03 01:14:36.230620 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:14:36.230535 4939 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-b-af8244/payload-processing-external-model-e2e-extmod-b-af8244-7b67bdr62"] Nov 03 01:14:36.230904 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:14:36.230652 4939 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-57079b/payload-processing-external-model-e2e-extmod-b-57079b-59f8rtzcd" Nov 03 01:14:36.236722 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:14:36.236370 4939 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-57079b\"/\"agc-cred-e705148022ae\"" Nov 03 01:14:36.236722 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:14:36.236411 4939 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-57079b\"/\"openshift-service-ca.crt\"" Nov 03 01:14:36.236722 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:14:36.236450 4939 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-57079b\"/\"agc-cred-83631bdec3fc\"" Nov 03 01:14:36.236722 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:14:36.236532 4939 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-57079b\"/\"payload-processing-external-model-plugins-e2e-extmod-b-57079b\"" Nov 03 01:14:36.236722 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:14:36.236549 4939 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-57079b\"/\"payload-processing-external-model-e2e--4a3ed774-dockercfg-nwdwk\"" Nov 03 01:14:36.236722 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:14:36.236591 4939 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-57079b\"/\"agc-cred-06b235044840\"" Nov 03 01:14:36.236722 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:14:36.236629 4939 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-57079b\"/\"kube-root-ca.crt\"" Nov 03 01:14:36.236722 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:14:36.236688 4939 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-57079b\"/\"routing-overlay\"" Nov 03 01:14:36.263147 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:14:36.263117 4939 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-57079b/payload-processing-external-model-e2e-extmod-b-57079b-59f8rtzcd"] Nov 03 01:14:36.263299 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:14:36.263181 4939 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-af8244/payload-processing-external-model-e2e-extmod-b-af8244-7b67bdr62"] Nov 03 01:14:36.263299 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:14:36.263279 4939 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-af8244/payload-processing-external-model-e2e-extmod-b-af8244-7b67bdr62" Nov 03 01:14:36.268148 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:14:36.267830 4939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/66c78580-7ad7-49b8-acf7-b23dd33a8af9-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/66c78580-7ad7-49b8-acf7-b23dd33a8af9-workload-socket\") pod \"66c78580-7ad7-49b8-acf7-b23dd33a8af9\" (UID: \"66c78580-7ad7-49b8-acf7-b23dd33a8af9\") " Nov 03 01:14:36.269059 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:14:36.269032 4939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/66c78580-7ad7-49b8-acf7-b23dd33a8af9-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/66c78580-7ad7-49b8-acf7-b23dd33a8af9-istio-podinfo\") pod \"66c78580-7ad7-49b8-acf7-b23dd33a8af9\" (UID: \"66c78580-7ad7-49b8-acf7-b23dd33a8af9\") " Nov 03 01:14:36.269323 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:14:36.269307 4939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/66c78580-7ad7-49b8-acf7-b23dd33a8af9-istio-token\" (UniqueName: \"kubernetes.io/projected/66c78580-7ad7-49b8-acf7-b23dd33a8af9-istio-token\") pod \"66c78580-7ad7-49b8-acf7-b23dd33a8af9\" (UID: \"66c78580-7ad7-49b8-acf7-b23dd33a8af9\") " Nov 03 01:14:36.269607 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:14:36.269586 4939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/66c78580-7ad7-49b8-acf7-b23dd33a8af9-kube-api-access-9b57f\" (UniqueName: \"kubernetes.io/projected/66c78580-7ad7-49b8-acf7-b23dd33a8af9-kube-api-access-9b57f\") pod \"66c78580-7ad7-49b8-acf7-b23dd33a8af9\" (UID: \"66c78580-7ad7-49b8-acf7-b23dd33a8af9\") " Nov 03 01:14:36.270078 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:14:36.270054 4939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/66c78580-7ad7-49b8-acf7-b23dd33a8af9-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/66c78580-7ad7-49b8-acf7-b23dd33a8af9-istio-data\") pod \"66c78580-7ad7-49b8-acf7-b23dd33a8af9\" (UID: \"66c78580-7ad7-49b8-acf7-b23dd33a8af9\") " Nov 03 01:14:36.270277 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:14:36.270258 4939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/66c78580-7ad7-49b8-acf7-b23dd33a8af9-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/66c78580-7ad7-49b8-acf7-b23dd33a8af9-workload-certs\") pod \"66c78580-7ad7-49b8-acf7-b23dd33a8af9\" (UID: \"66c78580-7ad7-49b8-acf7-b23dd33a8af9\") " Nov 03 01:14:36.270816 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:14:36.270793 4939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/66c78580-7ad7-49b8-acf7-b23dd33a8af9-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/66c78580-7ad7-49b8-acf7-b23dd33a8af9-credential-socket\") pod \"66c78580-7ad7-49b8-acf7-b23dd33a8af9\" (UID: \"66c78580-7ad7-49b8-acf7-b23dd33a8af9\") " Nov 03 01:14:36.271270 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:14:36.271249 4939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/66c78580-7ad7-49b8-acf7-b23dd33a8af9-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/66c78580-7ad7-49b8-acf7-b23dd33a8af9-istiod-ca-cert\") pod \"66c78580-7ad7-49b8-acf7-b23dd33a8af9\" (UID: \"66c78580-7ad7-49b8-acf7-b23dd33a8af9\") " Nov 03 01:14:36.275628 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:14:36.275607 4939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/66c78580-7ad7-49b8-acf7-b23dd33a8af9-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/66c78580-7ad7-49b8-acf7-b23dd33a8af9-istio-envoy\") pod \"66c78580-7ad7-49b8-acf7-b23dd33a8af9\" (UID: \"66c78580-7ad7-49b8-acf7-b23dd33a8af9\") " Nov 03 01:14:36.276202 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:14:36.276178 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/c54b89b6-6593-4f97-9cf9-21e7d21880b6-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-57079b-59f8rtzcd\" (UID: \"c54b89b6-6593-4f97-9cf9-21e7d21880b6\") " pod="ai-tenant-e2e-extmod-b-57079b/payload-processing-external-model-e2e-extmod-b-57079b-59f8rtzcd" Nov 03 01:14:36.277141 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:14:36.268120 4939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/66c78580-7ad7-49b8-acf7-b23dd33a8af9-workload-socket" pod "66c78580-7ad7-49b8-acf7-b23dd33a8af9" (UID: "66c78580-7ad7-49b8-acf7-b23dd33a8af9"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 01:14:36.277141 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:14:36.272744 4939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/66c78580-7ad7-49b8-acf7-b23dd33a8af9-istio-podinfo" pod "66c78580-7ad7-49b8-acf7-b23dd33a8af9" (UID: "66c78580-7ad7-49b8-acf7-b23dd33a8af9"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 01:14:36.277313 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:14:36.276664 4939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/66c78580-7ad7-49b8-acf7-b23dd33a8af9-credential-socket" pod "66c78580-7ad7-49b8-acf7-b23dd33a8af9" (UID: "66c78580-7ad7-49b8-acf7-b23dd33a8af9"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 01:14:36.277313 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:14:36.276894 4939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/66c78580-7ad7-49b8-acf7-b23dd33a8af9-istio-data" pod "66c78580-7ad7-49b8-acf7-b23dd33a8af9" (UID: "66c78580-7ad7-49b8-acf7-b23dd33a8af9"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 01:14:36.277313 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:14:36.277114 4939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/66c78580-7ad7-49b8-acf7-b23dd33a8af9-workload-certs" pod "66c78580-7ad7-49b8-acf7-b23dd33a8af9" (UID: "66c78580-7ad7-49b8-acf7-b23dd33a8af9"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 01:14:36.277540 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:14:36.277501 4939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/66c78580-7ad7-49b8-acf7-b23dd33a8af9-istiod-ca-cert" pod "66c78580-7ad7-49b8-acf7-b23dd33a8af9" (UID: "66c78580-7ad7-49b8-acf7-b23dd33a8af9"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 01:14:36.277706 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:14:36.277656 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c54b89b6-6593-4f97-9cf9-21e7d21880b6-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-57079b-59f8rtzcd\" (UID: \"c54b89b6-6593-4f97-9cf9-21e7d21880b6\") " pod="ai-tenant-e2e-extmod-b-57079b/payload-processing-external-model-e2e-extmod-b-57079b-59f8rtzcd" Nov 03 01:14:36.277895 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:14:36.277870 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-87tzv\" (UniqueName: \"kubernetes.io/projected/c54b89b6-6593-4f97-9cf9-21e7d21880b6-kube-api-access-87tzv\") pod \"payload-processing-external-model-e2e-extmod-b-57079b-59f8rtzcd\" (UID: \"c54b89b6-6593-4f97-9cf9-21e7d21880b6\") " pod="ai-tenant-e2e-extmod-b-57079b/payload-processing-external-model-e2e-extmod-b-57079b-59f8rtzcd" Nov 03 01:14:36.278030 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:14:36.277949 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/c54b89b6-6593-4f97-9cf9-21e7d21880b6-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-57079b-59f8rtzcd\" (UID: \"c54b89b6-6593-4f97-9cf9-21e7d21880b6\") " pod="ai-tenant-e2e-extmod-b-57079b/payload-processing-external-model-e2e-extmod-b-57079b-59f8rtzcd" Nov 03 01:14:36.278365 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:14:36.278348 4939 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/66c78580-7ad7-49b8-acf7-b23dd33a8af9-istio-data\") on node \"ip-10-0-140-107.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:14:36.278495 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:14:36.278368 4939 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/66c78580-7ad7-49b8-acf7-b23dd33a8af9-workload-certs\") on node \"ip-10-0-140-107.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:14:36.278495 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:14:36.278383 4939 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/66c78580-7ad7-49b8-acf7-b23dd33a8af9-credential-socket\") on node \"ip-10-0-140-107.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:14:36.278495 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:14:36.278396 4939 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/66c78580-7ad7-49b8-acf7-b23dd33a8af9-istiod-ca-cert\") on node \"ip-10-0-140-107.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:14:36.278495 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:14:36.278409 4939 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/66c78580-7ad7-49b8-acf7-b23dd33a8af9-workload-socket\") on node \"ip-10-0-140-107.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:14:36.278495 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:14:36.278422 4939 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/66c78580-7ad7-49b8-acf7-b23dd33a8af9-istio-podinfo\") on node \"ip-10-0-140-107.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:14:36.286394 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:14:36.286371 4939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/66c78580-7ad7-49b8-acf7-b23dd33a8af9-istio-token" pod "66c78580-7ad7-49b8-acf7-b23dd33a8af9" (UID: "66c78580-7ad7-49b8-acf7-b23dd33a8af9"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 01:14:36.286484 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:14:36.286438 4939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/66c78580-7ad7-49b8-acf7-b23dd33a8af9-istio-envoy" pod "66c78580-7ad7-49b8-acf7-b23dd33a8af9" (UID: "66c78580-7ad7-49b8-acf7-b23dd33a8af9"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 01:14:36.286484 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:14:36.286463 4939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/66c78580-7ad7-49b8-acf7-b23dd33a8af9-kube-api-access-9b57f" pod "66c78580-7ad7-49b8-acf7-b23dd33a8af9" (UID: "66c78580-7ad7-49b8-acf7-b23dd33a8af9"). InnerVolumeSpecName "kube-api-access-9b57f". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 01:14:36.334000 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:14:36.333961 4939 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-af8244\"/\"agc-cred-3b03a14485a2\"" Nov 03 01:14:36.334227 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:14:36.333969 4939 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-af8244\"/\"agc-cred-e8606c64533f\"" Nov 03 01:14:36.334298 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:14:36.333986 4939 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-af8244\"/\"agc-cred-a8289a8ace3b\"" Nov 03 01:14:36.334298 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:14:36.333997 4939 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-af8244\"/\"payload-processing-external-model-e2e--fa7ecc9b-dockercfg-95w8h\"" Nov 03 01:14:36.336122 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:14:36.336098 4939 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-af8244\"/\"payload-processing-external-model-plugins-e2e-extmod-b-af8244\"" Nov 03 01:14:36.336268 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:14:36.336211 4939 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-af8244\"/\"routing-overlay\"" Nov 03 01:14:36.336268 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:14:36.336212 4939 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-af8244\"/\"openshift-service-ca.crt\"" Nov 03 01:14:36.336268 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:14:36.336221 4939 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-af8244\"/\"kube-root-ca.crt\"" Nov 03 01:14:36.380332 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:14:36.380302 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xn26c\" (UniqueName: \"kubernetes.io/projected/b83ac5aa-8a33-487b-b97f-e1d45d2553cb-kube-api-access-xn26c\") pod \"payload-processing-external-model-e2e-extmod-b-af8244-7b67bdr62\" (UID: \"b83ac5aa-8a33-487b-b97f-e1d45d2553cb\") " pod="ai-tenant-e2e-extmod-b-af8244/payload-processing-external-model-e2e-extmod-b-af8244-7b67bdr62" Nov 03 01:14:36.380470 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:14:36.380352 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/b83ac5aa-8a33-487b-b97f-e1d45d2553cb-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-af8244-7b67bdr62\" (UID: \"b83ac5aa-8a33-487b-b97f-e1d45d2553cb\") " pod="ai-tenant-e2e-extmod-b-af8244/payload-processing-external-model-e2e-extmod-b-af8244-7b67bdr62" Nov 03 01:14:36.380470 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:14:36.380439 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/c54b89b6-6593-4f97-9cf9-21e7d21880b6-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-57079b-59f8rtzcd\" (UID: \"c54b89b6-6593-4f97-9cf9-21e7d21880b6\") " pod="ai-tenant-e2e-extmod-b-57079b/payload-processing-external-model-e2e-extmod-b-57079b-59f8rtzcd" Nov 03 01:14:36.380583 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:14:36.380537 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/b83ac5aa-8a33-487b-b97f-e1d45d2553cb-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-af8244-7b67bdr62\" (UID: \"b83ac5aa-8a33-487b-b97f-e1d45d2553cb\") " pod="ai-tenant-e2e-extmod-b-af8244/payload-processing-external-model-e2e-extmod-b-af8244-7b67bdr62" Nov 03 01:14:36.380727 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:14:36.380711 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c54b89b6-6593-4f97-9cf9-21e7d21880b6-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-57079b-59f8rtzcd\" (UID: \"c54b89b6-6593-4f97-9cf9-21e7d21880b6\") " pod="ai-tenant-e2e-extmod-b-57079b/payload-processing-external-model-e2e-extmod-b-57079b-59f8rtzcd" Nov 03 01:14:36.380810 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:14:36.380798 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-87tzv\" (UniqueName: \"kubernetes.io/projected/c54b89b6-6593-4f97-9cf9-21e7d21880b6-kube-api-access-87tzv\") pod \"payload-processing-external-model-e2e-extmod-b-57079b-59f8rtzcd\" (UID: \"c54b89b6-6593-4f97-9cf9-21e7d21880b6\") " pod="ai-tenant-e2e-extmod-b-57079b/payload-processing-external-model-e2e-extmod-b-57079b-59f8rtzcd" Nov 03 01:14:36.380849 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:14:36.380824 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/c54b89b6-6593-4f97-9cf9-21e7d21880b6-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-57079b-59f8rtzcd\" (UID: \"c54b89b6-6593-4f97-9cf9-21e7d21880b6\") " pod="ai-tenant-e2e-extmod-b-57079b/payload-processing-external-model-e2e-extmod-b-57079b-59f8rtzcd" Nov 03 01:14:36.380968 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:14:36.380951 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b83ac5aa-8a33-487b-b97f-e1d45d2553cb-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-af8244-7b67bdr62\" (UID: \"b83ac5aa-8a33-487b-b97f-e1d45d2553cb\") " pod="ai-tenant-e2e-extmod-b-af8244/payload-processing-external-model-e2e-extmod-b-af8244-7b67bdr62" Nov 03 01:14:36.381136 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:14:36.381122 4939 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/66c78580-7ad7-49b8-acf7-b23dd33a8af9-istio-token\") on node \"ip-10-0-140-107.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:14:36.381205 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:14:36.381138 4939 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9b57f\" (UniqueName: \"kubernetes.io/projected/66c78580-7ad7-49b8-acf7-b23dd33a8af9-kube-api-access-9b57f\") on node \"ip-10-0-140-107.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:14:36.381205 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:14:36.381149 4939 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/66c78580-7ad7-49b8-acf7-b23dd33a8af9-istio-envoy\") on node \"ip-10-0-140-107.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:14:36.381205 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:14:36.381178 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/c54b89b6-6593-4f97-9cf9-21e7d21880b6-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-57079b-59f8rtzcd\" (UID: \"c54b89b6-6593-4f97-9cf9-21e7d21880b6\") " pod="ai-tenant-e2e-extmod-b-57079b/payload-processing-external-model-e2e-extmod-b-57079b-59f8rtzcd" Nov 03 01:14:36.381511 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:14:36.381491 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c54b89b6-6593-4f97-9cf9-21e7d21880b6-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-57079b-59f8rtzcd\" (UID: \"c54b89b6-6593-4f97-9cf9-21e7d21880b6\") " pod="ai-tenant-e2e-extmod-b-57079b/payload-processing-external-model-e2e-extmod-b-57079b-59f8rtzcd" Nov 03 01:14:36.383568 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:14:36.383545 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/c54b89b6-6593-4f97-9cf9-21e7d21880b6-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-57079b-59f8rtzcd\" (UID: \"c54b89b6-6593-4f97-9cf9-21e7d21880b6\") " pod="ai-tenant-e2e-extmod-b-57079b/payload-processing-external-model-e2e-extmod-b-57079b-59f8rtzcd" Nov 03 01:14:36.391784 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:14:36.391763 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-87tzv\" (UniqueName: \"kubernetes.io/projected/c54b89b6-6593-4f97-9cf9-21e7d21880b6-kube-api-access-87tzv\") pod \"payload-processing-external-model-e2e-extmod-b-57079b-59f8rtzcd\" (UID: \"c54b89b6-6593-4f97-9cf9-21e7d21880b6\") " pod="ai-tenant-e2e-extmod-b-57079b/payload-processing-external-model-e2e-extmod-b-57079b-59f8rtzcd" Nov 03 01:14:36.482354 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:14:36.482268 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b83ac5aa-8a33-487b-b97f-e1d45d2553cb-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-af8244-7b67bdr62\" (UID: \"b83ac5aa-8a33-487b-b97f-e1d45d2553cb\") " pod="ai-tenant-e2e-extmod-b-af8244/payload-processing-external-model-e2e-extmod-b-af8244-7b67bdr62" Nov 03 01:14:36.482514 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:14:36.482372 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xn26c\" (UniqueName: \"kubernetes.io/projected/b83ac5aa-8a33-487b-b97f-e1d45d2553cb-kube-api-access-xn26c\") pod \"payload-processing-external-model-e2e-extmod-b-af8244-7b67bdr62\" (UID: \"b83ac5aa-8a33-487b-b97f-e1d45d2553cb\") " pod="ai-tenant-e2e-extmod-b-af8244/payload-processing-external-model-e2e-extmod-b-af8244-7b67bdr62" Nov 03 01:14:36.482514 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:14:36.482417 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/b83ac5aa-8a33-487b-b97f-e1d45d2553cb-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-af8244-7b67bdr62\" (UID: \"b83ac5aa-8a33-487b-b97f-e1d45d2553cb\") " pod="ai-tenant-e2e-extmod-b-af8244/payload-processing-external-model-e2e-extmod-b-af8244-7b67bdr62" Nov 03 01:14:36.482514 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:14:36.482482 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/b83ac5aa-8a33-487b-b97f-e1d45d2553cb-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-af8244-7b67bdr62\" (UID: \"b83ac5aa-8a33-487b-b97f-e1d45d2553cb\") " pod="ai-tenant-e2e-extmod-b-af8244/payload-processing-external-model-e2e-extmod-b-af8244-7b67bdr62" Nov 03 01:14:36.483211 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:14:36.483179 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/b83ac5aa-8a33-487b-b97f-e1d45d2553cb-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-af8244-7b67bdr62\" (UID: \"b83ac5aa-8a33-487b-b97f-e1d45d2553cb\") " pod="ai-tenant-e2e-extmod-b-af8244/payload-processing-external-model-e2e-extmod-b-af8244-7b67bdr62" Nov 03 01:14:36.483329 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:14:36.483219 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b83ac5aa-8a33-487b-b97f-e1d45d2553cb-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-af8244-7b67bdr62\" (UID: \"b83ac5aa-8a33-487b-b97f-e1d45d2553cb\") " pod="ai-tenant-e2e-extmod-b-af8244/payload-processing-external-model-e2e-extmod-b-af8244-7b67bdr62" Nov 03 01:14:36.485345 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:14:36.485320 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/b83ac5aa-8a33-487b-b97f-e1d45d2553cb-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-af8244-7b67bdr62\" (UID: \"b83ac5aa-8a33-487b-b97f-e1d45d2553cb\") " pod="ai-tenant-e2e-extmod-b-af8244/payload-processing-external-model-e2e-extmod-b-af8244-7b67bdr62" Nov 03 01:14:36.490978 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:14:36.490951 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xn26c\" (UniqueName: \"kubernetes.io/projected/b83ac5aa-8a33-487b-b97f-e1d45d2553cb-kube-api-access-xn26c\") pod \"payload-processing-external-model-e2e-extmod-b-af8244-7b67bdr62\" (UID: \"b83ac5aa-8a33-487b-b97f-e1d45d2553cb\") " pod="ai-tenant-e2e-extmod-b-af8244/payload-processing-external-model-e2e-extmod-b-af8244-7b67bdr62" Nov 03 01:14:36.502493 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:14:36.502468 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:14:36.586944 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:14:36.580645 4939 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-57079b/payload-processing-external-model-e2e-extmod-b-57079b-59f8rtzcd" Nov 03 01:14:36.643701 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:14:36.643651 4939 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-af8244/payload-processing-external-model-e2e-extmod-b-af8244-7b67bdr62" Nov 03 01:14:36.920008 ip-10-0-140-107 kubenswrapper[4939]: W1103 01:14:36.919972 4939 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc54b89b6_6593_4f97_9cf9_21e7d21880b6.slice/crio-419d680c81ec61196845f5d4823a2ccc86c2ea8d26d7f7ecc0d5ea13b3c3abea WatchSource:0}: Error finding container 419d680c81ec61196845f5d4823a2ccc86c2ea8d26d7f7ecc0d5ea13b3c3abea: Status 404 returned error can't find the container with id 419d680c81ec61196845f5d4823a2ccc86c2ea8d26d7f7ecc0d5ea13b3c3abea Nov 03 01:14:36.920439 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:14:36.920085 4939 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-57079b/payload-processing-external-model-e2e-extmod-b-57079b-59f8rtzcd"] Nov 03 01:14:36.926325 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:14:36.926299 4939 generic.go:356] "Generic (PLEG): container finished" podID="66c78580-7ad7-49b8-acf7-b23dd33a8af9" containerID="01c6cd08a5a26c9754fd2f2e153053cf9c6cfb98437cb054bcaf8f98f044b406" exitCode=0 Nov 03 01:14:36.926489 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:14:36.926362 4939 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w1-2729f2-openshift-default-9fb97b4fc-czgh9" Nov 03 01:14:36.926489 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:14:36.926461 4939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w1-2729f2-openshift-default-9fb97b4fc-czgh9" event={"ID":"66c78580-7ad7-49b8-acf7-b23dd33a8af9","Type":"ContainerDied","Data":"01c6cd08a5a26c9754fd2f2e153053cf9c6cfb98437cb054bcaf8f98f044b406"} Nov 03 01:14:36.926731 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:14:36.926504 4939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w1-2729f2-openshift-default-9fb97b4fc-czgh9" event={"ID":"66c78580-7ad7-49b8-acf7-b23dd33a8af9","Type":"ContainerDied","Data":"fcc169b213e682547f3a874adcca5bb0f4c4abec0c4480df666cb2f9b116fec1"} Nov 03 01:14:36.926731 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:14:36.926533 4939 scope.go:122] "RemoveContainer" containerID="01c6cd08a5a26c9754fd2f2e153053cf9c6cfb98437cb054bcaf8f98f044b406" Nov 03 01:14:36.949212 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:14:36.949186 4939 scope.go:122] "RemoveContainer" containerID="01c6cd08a5a26c9754fd2f2e153053cf9c6cfb98437cb054bcaf8f98f044b406" Nov 03 01:14:36.949571 ip-10-0-140-107 kubenswrapper[4939]: E1103 01:14:36.949547 4939 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"01c6cd08a5a26c9754fd2f2e153053cf9c6cfb98437cb054bcaf8f98f044b406\": container with ID starting with 01c6cd08a5a26c9754fd2f2e153053cf9c6cfb98437cb054bcaf8f98f044b406 not found: ID does not exist" containerID="01c6cd08a5a26c9754fd2f2e153053cf9c6cfb98437cb054bcaf8f98f044b406" Nov 03 01:14:36.949653 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:14:36.949590 4939 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"01c6cd08a5a26c9754fd2f2e153053cf9c6cfb98437cb054bcaf8f98f044b406"} err="failed to get container status \"01c6cd08a5a26c9754fd2f2e153053cf9c6cfb98437cb054bcaf8f98f044b406\": rpc error: code = NotFound desc = could not find container \"01c6cd08a5a26c9754fd2f2e153053cf9c6cfb98437cb054bcaf8f98f044b406\": container with ID starting with 01c6cd08a5a26c9754fd2f2e153053cf9c6cfb98437cb054bcaf8f98f044b406 not found: ID does not exist" Nov 03 01:14:36.957061 ip-10-0-140-107 kubenswrapper[4939]: W1103 01:14:36.957037 4939 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb83ac5aa_8a33_487b_b97f_e1d45d2553cb.slice/crio-9f31ae0d806a716b085b6d81fb02872e1a35584ce33eba17334eb00788c44f41 WatchSource:0}: Error finding container 9f31ae0d806a716b085b6d81fb02872e1a35584ce33eba17334eb00788c44f41: Status 404 returned error can't find the container with id 9f31ae0d806a716b085b6d81fb02872e1a35584ce33eba17334eb00788c44f41 Nov 03 01:14:36.960969 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:14:36.960945 4939 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-af8244/payload-processing-external-model-e2e-extmod-b-af8244-7b67bdr62"] Nov 03 01:14:37.012376 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:14:37.012305 4939 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w1-2729f2-openshift-default-9fb97b4fc-czgh9"] Nov 03 01:14:37.018566 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:14:37.017729 4939 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w1-2729f2-openshift-default-9fb97b4fc-czgh9"] Nov 03 01:14:37.212840 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:14:37.212809 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:14:37.814089 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:14:37.814057 4939 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-3c5983/payload-processing-external-model-e2e-extmod-b-3c5983-7697p7dkr"] Nov 03 01:14:37.878398 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:14:37.878354 4939 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="66c78580-7ad7-49b8-acf7-b23dd33a8af9" path="/var/lib/kubelet/pods/66c78580-7ad7-49b8-acf7-b23dd33a8af9/volumes" Nov 03 01:14:37.936998 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:14:37.936958 4939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-57079b/payload-processing-external-model-e2e-extmod-b-57079b-59f8rtzcd" event={"ID":"c54b89b6-6593-4f97-9cf9-21e7d21880b6","Type":"ContainerStarted","Data":"e1aa07c948a31d2f6dc827ac12c0a06874b892d9fb01c24a786dba7bfdc67110"} Nov 03 01:14:37.937494 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:14:37.937022 4939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-57079b/payload-processing-external-model-e2e-extmod-b-57079b-59f8rtzcd" event={"ID":"c54b89b6-6593-4f97-9cf9-21e7d21880b6","Type":"ContainerStarted","Data":"419d680c81ec61196845f5d4823a2ccc86c2ea8d26d7f7ecc0d5ea13b3c3abea"} Nov 03 01:14:37.937494 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:14:37.937054 4939 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-b-57079b/payload-processing-external-model-e2e-extmod-b-57079b-59f8rtzcd" Nov 03 01:14:37.938780 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:14:37.938746 4939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-af8244/payload-processing-external-model-e2e-extmod-b-af8244-7b67bdr62" event={"ID":"b83ac5aa-8a33-487b-b97f-e1d45d2553cb","Type":"ContainerStarted","Data":"4a7b14c23b33892e69ba0826b2be1d47d0df377780444b03f9a24c37187f9f91"} Nov 03 01:14:37.938945 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:14:37.938796 4939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-af8244/payload-processing-external-model-e2e-extmod-b-af8244-7b67bdr62" event={"ID":"b83ac5aa-8a33-487b-b97f-e1d45d2553cb","Type":"ContainerStarted","Data":"9f31ae0d806a716b085b6d81fb02872e1a35584ce33eba17334eb00788c44f41"} Nov 03 01:14:37.938945 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:14:37.938841 4939 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-b-af8244/payload-processing-external-model-e2e-extmod-b-af8244-7b67bdr62" Nov 03 01:14:37.962275 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:14:37.962222 4939 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-b-57079b/payload-processing-external-model-e2e-extmod-b-57079b-59f8rtzcd" podStartSLOduration=19.962209408 podStartE2EDuration="19.962209408s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 01:14:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 01:14:37.959234585 +0000 UTC m=+2026.223569542" watchObservedRunningTime="2026-11-03 01:14:37.962209408 +0000 UTC m=+2026.226544353" Nov 03 01:14:37.971738 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:14:37.971715 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:14:37.978275 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:14:37.978236 4939 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-b-af8244/payload-processing-external-model-e2e-extmod-b-af8244-7b67bdr62" podStartSLOduration=7.978223483 podStartE2EDuration="7.978223483s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 01:14:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 01:14:37.977748477 +0000 UTC m=+2026.242083421" watchObservedRunningTime="2026-11-03 01:14:37.978223483 +0000 UTC m=+2026.242558428" Nov 03 01:14:38.750422 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:14:38.750387 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:14:39.403612 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:14:39.403561 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:14:40.158931 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:14:40.157892 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:14:40.767332 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:14:40.767302 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:14:41.500116 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:14:41.500079 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:14:42.192500 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:14:42.192466 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:14:42.814541 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:14:42.814497 4939 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-b-3c5983/payload-processing-external-model-e2e-extmod-b-3c5983-7697p7dkr" podUID="c0770733-3323-42a9-8f86-619f71a276b6" containerName="payload-processing" containerID="cri-o://bae354d39300ae52bf3b6e6197e34b4904f130eba9a4429852f716e3ab887031" gracePeriod=25 Nov 03 01:14:42.913134 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:14:42.913103 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:14:43.051232 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:14:43.051199 4939 generic.go:356] "Generic (PLEG): container finished" podID="c0770733-3323-42a9-8f86-619f71a276b6" containerID="bae354d39300ae52bf3b6e6197e34b4904f130eba9a4429852f716e3ab887031" exitCode=0 Nov 03 01:14:43.051492 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:14:43.051477 4939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-3c5983/payload-processing-external-model-e2e-extmod-b-3c5983-7697p7dkr" event={"ID":"c0770733-3323-42a9-8f86-619f71a276b6","Type":"ContainerDied","Data":"bae354d39300ae52bf3b6e6197e34b4904f130eba9a4429852f716e3ab887031"} Nov 03 01:14:43.202196 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:14:43.202155 4939 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-3c5983/payload-processing-external-model-e2e-extmod-b-3c5983-7697p7dkr" Nov 03 01:14:43.304920 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:14:43.304874 4939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c0770733-3323-42a9-8f86-619f71a276b6-kube-api-access-hkhdq\" (UniqueName: \"kubernetes.io/projected/c0770733-3323-42a9-8f86-619f71a276b6-kube-api-access-hkhdq\") pod \"c0770733-3323-42a9-8f86-619f71a276b6\" (UID: \"c0770733-3323-42a9-8f86-619f71a276b6\") " Nov 03 01:14:43.305103 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:14:43.304996 4939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c0770733-3323-42a9-8f86-619f71a276b6-provider-credentials\" (UniqueName: \"kubernetes.io/projected/c0770733-3323-42a9-8f86-619f71a276b6-provider-credentials\") pod \"c0770733-3323-42a9-8f86-619f71a276b6\" (UID: \"c0770733-3323-42a9-8f86-619f71a276b6\") " Nov 03 01:14:43.305103 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:14:43.305078 4939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c0770733-3323-42a9-8f86-619f71a276b6-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/c0770733-3323-42a9-8f86-619f71a276b6-routing-overlay-volume\") pod \"c0770733-3323-42a9-8f86-619f71a276b6\" (UID: \"c0770733-3323-42a9-8f86-619f71a276b6\") " Nov 03 01:14:43.305310 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:14:43.305166 4939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c0770733-3323-42a9-8f86-619f71a276b6-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c0770733-3323-42a9-8f86-619f71a276b6-plugins-config-volume\") pod \"c0770733-3323-42a9-8f86-619f71a276b6\" (UID: \"c0770733-3323-42a9-8f86-619f71a276b6\") " Nov 03 01:14:43.305526 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:14:43.305495 4939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c0770733-3323-42a9-8f86-619f71a276b6-routing-overlay-volume" pod "c0770733-3323-42a9-8f86-619f71a276b6" (UID: "c0770733-3323-42a9-8f86-619f71a276b6"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 01:14:43.305590 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:14:43.305553 4939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c0770733-3323-42a9-8f86-619f71a276b6-plugins-config-volume" pod "c0770733-3323-42a9-8f86-619f71a276b6" (UID: "c0770733-3323-42a9-8f86-619f71a276b6"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 01:14:43.305977 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:14:43.305955 4939 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/c0770733-3323-42a9-8f86-619f71a276b6-routing-overlay-volume\") on node \"ip-10-0-140-107.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:14:43.305977 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:14:43.305978 4939 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c0770733-3323-42a9-8f86-619f71a276b6-plugins-config-volume\") on node \"ip-10-0-140-107.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:14:43.307156 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:14:43.307141 4939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c0770733-3323-42a9-8f86-619f71a276b6-kube-api-access-hkhdq" pod "c0770733-3323-42a9-8f86-619f71a276b6" (UID: "c0770733-3323-42a9-8f86-619f71a276b6"). InnerVolumeSpecName "kube-api-access-hkhdq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 01:14:43.307470 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:14:43.307449 4939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c0770733-3323-42a9-8f86-619f71a276b6-provider-credentials" pod "c0770733-3323-42a9-8f86-619f71a276b6" (UID: "c0770733-3323-42a9-8f86-619f71a276b6"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 01:14:43.408770 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:14:43.408640 4939 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hkhdq\" (UniqueName: \"kubernetes.io/projected/c0770733-3323-42a9-8f86-619f71a276b6-kube-api-access-hkhdq\") on node \"ip-10-0-140-107.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:14:43.408770 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:14:43.408684 4939 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/c0770733-3323-42a9-8f86-619f71a276b6-provider-credentials\") on node \"ip-10-0-140-107.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:14:43.648787 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:14:43.648601 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:14:43.877517 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:14:43.877480 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:14:44.062441 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:14:44.062350 4939 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-3c5983/payload-processing-external-model-e2e-extmod-b-3c5983-7697p7dkr" Nov 03 01:14:44.062618 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:14:44.062349 4939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-3c5983/payload-processing-external-model-e2e-extmod-b-3c5983-7697p7dkr" event={"ID":"c0770733-3323-42a9-8f86-619f71a276b6","Type":"ContainerDied","Data":"1b8e8da97999621e8dec4d712a0fb9544fcc7c2933de191a1d47264198fc2aff"} Nov 03 01:14:44.062618 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:14:44.062489 4939 scope.go:122] "RemoveContainer" containerID="bae354d39300ae52bf3b6e6197e34b4904f130eba9a4429852f716e3ab887031" Nov 03 01:14:44.082486 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:14:44.082460 4939 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-3c5983/payload-processing-external-model-e2e-extmod-b-3c5983-7697p7dkr"] Nov 03 01:14:44.088668 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:14:44.088643 4939 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-b-3c5983/payload-processing-external-model-e2e-extmod-b-3c5983-7697p7dkr"] Nov 03 01:14:44.404135 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:14:44.404096 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:14:45.061864 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:14:45.061834 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:14:45.697509 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:14:45.697464 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:14:45.872113 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:14:45.872079 4939 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c0770733-3323-42a9-8f86-619f71a276b6" path="/var/lib/kubelet/pods/c0770733-3323-42a9-8f86-619f71a276b6/volumes" Nov 03 01:14:46.319385 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:14:46.319351 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:14:47.104447 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:14:47.104408 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:14:47.787255 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:14:47.787214 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:14:48.033046 ip-10-0-140-107 kubenswrapper[4939]: E1103 01:14:48.033004 4939 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-sthjh" containerName="extractor" Nov 03 01:14:48.385477 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:14:48.385419 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:14:48.951616 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:14:48.951573 4939 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-b-af8244/payload-processing-external-model-e2e-extmod-b-af8244-7b67bdr62" Nov 03 01:14:48.951805 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:14:48.951651 4939 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-b-57079b/payload-processing-external-model-e2e-extmod-b-57079b-59f8rtzcd" Nov 03 01:14:49.029655 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:14:49.029621 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:14:49.698461 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:14:49.698422 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:14:50.358973 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:14:50.358942 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:14:51.345724 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:14:51.345681 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:14:52.086601 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:14:52.086565 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:14:52.707693 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:14:52.707657 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:14:53.383240 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:14:53.383197 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:14:54.086624 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:14:54.086576 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:14:54.792284 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:14:54.792251 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:14:55.506759 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:14:55.506720 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:14:56.120091 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:14:56.120056 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:14:56.485280 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:14:56.485164 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:14:56.781019 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:14:56.780941 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:14:57.519701 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:14:57.519662 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:14:57.659476 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:14:57.659437 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:14:57.875599 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:14:57.875567 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:14:58.138838 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:14:58.138750 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:14:58.257198 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:14:58.257163 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:14:58.384009 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:14:58.383971 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:14:58.522577 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:14:58.522497 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:14:58.900257 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:14:58.900221 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:14:59.041028 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:14:59.037100 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:14:59.658017 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:14:59.657973 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:15:00.149428 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:15:00.149382 4939 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-2729f2-29894475-pcj4t"] Nov 03 01:15:00.150807 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:15:00.150777 4939 cpu_manager.go:414] "containerMap: removing container" podUID="c0770733-3323-42a9-8f86-619f71a276b6" containerName="payload-processing" Nov 03 01:15:00.150807 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:15:00.150807 4939 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c0770733-3323-42a9-8f86-619f71a276b6" containerName="payload-processing" Nov 03 01:15:00.151192 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:15:00.151176 4939 memory_manager.go:365] "RemoveStaleState removing state" podUID="c0770733-3323-42a9-8f86-619f71a276b6" containerName="payload-processing" Nov 03 01:15:00.158549 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:15:00.158522 4939 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-2729f2-29894475-pcj4t" Nov 03 01:15:00.161159 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:15:00.161135 4939 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-020af9-29894475-xhbqm"] Nov 03 01:15:00.175177 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:15:00.175146 4939 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-6q4xp\"" Nov 03 01:15:00.177294 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:15:00.177267 4939 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-027c19-29894475-nnm44"] Nov 03 01:15:00.178114 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:15:00.178087 4939 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-020af9-29894475-xhbqm" Nov 03 01:15:00.226117 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:15:00.226082 4939 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-6c0ec0-29894475-8vfnr"] Nov 03 01:15:00.226389 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:15:00.226362 4939 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-027c19-29894475-nnm44" Nov 03 01:15:00.239931 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:15:00.239864 4939 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-027c19-29894475-nnm44"] Nov 03 01:15:00.239931 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:15:00.239932 4939 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-020af9-29894475-xhbqm"] Nov 03 01:15:00.240138 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:15:00.239961 4939 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-2729f2-29894475-pcj4t"] Nov 03 01:15:00.240138 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:15:00.239990 4939 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-6c0ec0-29894475-8vfnr"] Nov 03 01:15:00.240138 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:15:00.240082 4939 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-6c0ec0-29894475-8vfnr" Nov 03 01:15:00.245540 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:15:00.245516 4939 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-83755a-29894475-clqb4"] Nov 03 01:15:00.251866 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:15:00.251829 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s754v\" (UniqueName: \"kubernetes.io/projected/8abddaba-107d-4380-9ae4-5c912e6f4a03-kube-api-access-s754v\") pod \"maas-api-key-cleanup-e2e-shared-b-w0-020af9-29894475-xhbqm\" (UID: \"8abddaba-107d-4380-9ae4-5c912e6f4a03\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-020af9-29894475-xhbqm" Nov 03 01:15:00.252072 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:15:00.251902 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a3c9ab79-8ade-4c81-9e5a-38803c90f7a2-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w1-2729f2-29894475-pcj4t\" (UID: \"a3c9ab79-8ade-4c81-9e5a-38803c90f7a2\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-2729f2-29894475-pcj4t" Nov 03 01:15:00.252072 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:15:00.251960 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8abddaba-107d-4380-9ae4-5c912e6f4a03-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w0-020af9-29894475-xhbqm\" (UID: \"8abddaba-107d-4380-9ae4-5c912e6f4a03\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-020af9-29894475-xhbqm" Nov 03 01:15:00.252849 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:15:00.252809 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g2x4v\" (UniqueName: \"kubernetes.io/projected/a3c9ab79-8ade-4c81-9e5a-38803c90f7a2-kube-api-access-g2x4v\") pod \"maas-api-key-cleanup-e2e-shared-a-w1-2729f2-29894475-pcj4t\" (UID: \"a3c9ab79-8ade-4c81-9e5a-38803c90f7a2\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-2729f2-29894475-pcj4t" Nov 03 01:15:00.258582 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:15:00.258561 4939 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-af8244-29894475-h4vvr"] Nov 03 01:15:00.258851 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:15:00.258825 4939 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-83755a-29894475-clqb4" Nov 03 01:15:00.266970 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:15:00.266893 4939 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-821f5a-29894475-6tzm9"] Nov 03 01:15:00.269309 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:15:00.269235 4939 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-af8244-29894475-h4vvr" Nov 03 01:15:00.273769 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:15:00.273750 4939 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-821f5a-29894475-6tzm9" Nov 03 01:15:00.275246 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:15:00.275221 4939 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-af8244-29894475-h4vvr"] Nov 03 01:15:00.282496 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:15:00.281413 4939 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-821f5a-29894475-6tzm9"] Nov 03 01:15:00.287493 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:15:00.287470 4939 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-83755a-29894475-clqb4"] Nov 03 01:15:00.355385 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:15:00.355347 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s754v\" (UniqueName: \"kubernetes.io/projected/8abddaba-107d-4380-9ae4-5c912e6f4a03-kube-api-access-s754v\") pod \"maas-api-key-cleanup-e2e-shared-b-w0-020af9-29894475-xhbqm\" (UID: \"8abddaba-107d-4380-9ae4-5c912e6f4a03\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-020af9-29894475-xhbqm" Nov 03 01:15:00.355385 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:15:00.355385 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zbl6w\" (UniqueName: \"kubernetes.io/projected/f0a403e0-c375-412d-940d-66ecf89e3270-kube-api-access-zbl6w\") pod \"maas-api-key-cleanup-e2e-worker-w3-83755a-29894475-clqb4\" (UID: \"f0a403e0-c375-412d-940d-66ecf89e3270\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-83755a-29894475-clqb4" Nov 03 01:15:00.355633 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:15:00.355406 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6bc4e95b-2075-4115-a856-975abe775ad6-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-a-821f5a-29894475-6tzm9\" (UID: \"6bc4e95b-2075-4115-a856-975abe775ad6\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-821f5a-29894475-6tzm9" Nov 03 01:15:00.355633 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:15:00.355438 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a3c9ab79-8ade-4c81-9e5a-38803c90f7a2-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w1-2729f2-29894475-pcj4t\" (UID: \"a3c9ab79-8ade-4c81-9e5a-38803c90f7a2\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-2729f2-29894475-pcj4t" Nov 03 01:15:00.355633 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:15:00.355468 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8abddaba-107d-4380-9ae4-5c912e6f4a03-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w0-020af9-29894475-xhbqm\" (UID: \"8abddaba-107d-4380-9ae4-5c912e6f4a03\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-020af9-29894475-xhbqm" Nov 03 01:15:00.355633 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:15:00.355515 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5sp52\" (UniqueName: \"kubernetes.io/projected/7eba01d0-b475-421d-8aba-56cbc67c652d-kube-api-access-5sp52\") pod \"maas-api-key-cleanup-e2e-shared-a-w4-6c0ec0-29894475-8vfnr\" (UID: \"7eba01d0-b475-421d-8aba-56cbc67c652d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-6c0ec0-29894475-8vfnr" Nov 03 01:15:00.355633 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:15:00.355562 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hvpc4\" (UniqueName: \"kubernetes.io/projected/ca1eab39-5984-43f1-bd8c-029cd2131a7a-kube-api-access-hvpc4\") pod \"maas-api-key-cleanup-e2e-extmod-a-027c19-29894475-nnm44\" (UID: \"ca1eab39-5984-43f1-bd8c-029cd2131a7a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-027c19-29894475-nnm44" Nov 03 01:15:00.355977 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:15:00.355709 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ca1eab39-5984-43f1-bd8c-029cd2131a7a-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-a-027c19-29894475-nnm44\" (UID: \"ca1eab39-5984-43f1-bd8c-029cd2131a7a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-027c19-29894475-nnm44" Nov 03 01:15:00.355977 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:15:00.355761 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f0a403e0-c375-412d-940d-66ecf89e3270-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w3-83755a-29894475-clqb4\" (UID: \"f0a403e0-c375-412d-940d-66ecf89e3270\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-83755a-29894475-clqb4" Nov 03 01:15:00.355977 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:15:00.355839 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7eba01d0-b475-421d-8aba-56cbc67c652d-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w4-6c0ec0-29894475-8vfnr\" (UID: \"7eba01d0-b475-421d-8aba-56cbc67c652d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-6c0ec0-29894475-8vfnr" Nov 03 01:15:00.355977 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:15:00.355895 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g2x4v\" (UniqueName: \"kubernetes.io/projected/a3c9ab79-8ade-4c81-9e5a-38803c90f7a2-kube-api-access-g2x4v\") pod \"maas-api-key-cleanup-e2e-shared-a-w1-2729f2-29894475-pcj4t\" (UID: \"a3c9ab79-8ade-4c81-9e5a-38803c90f7a2\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-2729f2-29894475-pcj4t" Nov 03 01:15:00.356234 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:15:00.356060 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/899bb599-2a0d-4670-818b-b0eda152816f-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-b-af8244-29894475-h4vvr\" (UID: \"899bb599-2a0d-4670-818b-b0eda152816f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-af8244-29894475-h4vvr" Nov 03 01:15:00.356234 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:15:00.356119 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jvd6s\" (UniqueName: \"kubernetes.io/projected/899bb599-2a0d-4670-818b-b0eda152816f-kube-api-access-jvd6s\") pod \"maas-api-key-cleanup-e2e-extmod-b-af8244-29894475-h4vvr\" (UID: \"899bb599-2a0d-4670-818b-b0eda152816f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-af8244-29894475-h4vvr" Nov 03 01:15:00.356234 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:15:00.356195 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a3c9ab79-8ade-4c81-9e5a-38803c90f7a2-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w1-2729f2-29894475-pcj4t\" (UID: \"a3c9ab79-8ade-4c81-9e5a-38803c90f7a2\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-2729f2-29894475-pcj4t" Nov 03 01:15:00.356402 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:15:00.356255 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4nwl2\" (UniqueName: \"kubernetes.io/projected/6bc4e95b-2075-4115-a856-975abe775ad6-kube-api-access-4nwl2\") pod \"maas-api-key-cleanup-e2e-extmod-a-821f5a-29894475-6tzm9\" (UID: \"6bc4e95b-2075-4115-a856-975abe775ad6\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-821f5a-29894475-6tzm9" Nov 03 01:15:00.356402 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:15:00.356363 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8abddaba-107d-4380-9ae4-5c912e6f4a03-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w0-020af9-29894475-xhbqm\" (UID: \"8abddaba-107d-4380-9ae4-5c912e6f4a03\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-020af9-29894475-xhbqm" Nov 03 01:15:00.365856 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:15:00.365830 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-g2x4v\" (UniqueName: \"kubernetes.io/projected/a3c9ab79-8ade-4c81-9e5a-38803c90f7a2-kube-api-access-g2x4v\") pod \"maas-api-key-cleanup-e2e-shared-a-w1-2729f2-29894475-pcj4t\" (UID: \"a3c9ab79-8ade-4c81-9e5a-38803c90f7a2\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-2729f2-29894475-pcj4t" Nov 03 01:15:00.366204 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:15:00.366185 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-s754v\" (UniqueName: \"kubernetes.io/projected/8abddaba-107d-4380-9ae4-5c912e6f4a03-kube-api-access-s754v\") pod \"maas-api-key-cleanup-e2e-shared-b-w0-020af9-29894475-xhbqm\" (UID: \"8abddaba-107d-4380-9ae4-5c912e6f4a03\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-020af9-29894475-xhbqm" Nov 03 01:15:00.439204 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:15:00.439112 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:15:00.481873 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:15:00.481611 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/899bb599-2a0d-4670-818b-b0eda152816f-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-b-af8244-29894475-h4vvr\" (UID: \"899bb599-2a0d-4670-818b-b0eda152816f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-af8244-29894475-h4vvr" Nov 03 01:15:00.481873 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:15:00.481726 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jvd6s\" (UniqueName: \"kubernetes.io/projected/899bb599-2a0d-4670-818b-b0eda152816f-kube-api-access-jvd6s\") pod \"maas-api-key-cleanup-e2e-extmod-b-af8244-29894475-h4vvr\" (UID: \"899bb599-2a0d-4670-818b-b0eda152816f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-af8244-29894475-h4vvr" Nov 03 01:15:00.481873 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:15:00.481838 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4nwl2\" (UniqueName: \"kubernetes.io/projected/6bc4e95b-2075-4115-a856-975abe775ad6-kube-api-access-4nwl2\") pod \"maas-api-key-cleanup-e2e-extmod-a-821f5a-29894475-6tzm9\" (UID: \"6bc4e95b-2075-4115-a856-975abe775ad6\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-821f5a-29894475-6tzm9" Nov 03 01:15:00.492174 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:15:00.483331 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/899bb599-2a0d-4670-818b-b0eda152816f-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-b-af8244-29894475-h4vvr\" (UID: \"899bb599-2a0d-4670-818b-b0eda152816f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-af8244-29894475-h4vvr" Nov 03 01:15:00.492174 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:15:00.488972 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zbl6w\" (UniqueName: \"kubernetes.io/projected/f0a403e0-c375-412d-940d-66ecf89e3270-kube-api-access-zbl6w\") pod \"maas-api-key-cleanup-e2e-worker-w3-83755a-29894475-clqb4\" (UID: \"f0a403e0-c375-412d-940d-66ecf89e3270\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-83755a-29894475-clqb4" Nov 03 01:15:00.492174 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:15:00.489029 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6bc4e95b-2075-4115-a856-975abe775ad6-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-a-821f5a-29894475-6tzm9\" (UID: \"6bc4e95b-2075-4115-a856-975abe775ad6\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-821f5a-29894475-6tzm9" Nov 03 01:15:00.492174 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:15:00.489120 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5sp52\" (UniqueName: \"kubernetes.io/projected/7eba01d0-b475-421d-8aba-56cbc67c652d-kube-api-access-5sp52\") pod \"maas-api-key-cleanup-e2e-shared-a-w4-6c0ec0-29894475-8vfnr\" (UID: \"7eba01d0-b475-421d-8aba-56cbc67c652d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-6c0ec0-29894475-8vfnr" Nov 03 01:15:00.492174 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:15:00.489173 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hvpc4\" (UniqueName: \"kubernetes.io/projected/ca1eab39-5984-43f1-bd8c-029cd2131a7a-kube-api-access-hvpc4\") pod \"maas-api-key-cleanup-e2e-extmod-a-027c19-29894475-nnm44\" (UID: \"ca1eab39-5984-43f1-bd8c-029cd2131a7a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-027c19-29894475-nnm44" Nov 03 01:15:00.492174 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:15:00.489269 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ca1eab39-5984-43f1-bd8c-029cd2131a7a-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-a-027c19-29894475-nnm44\" (UID: \"ca1eab39-5984-43f1-bd8c-029cd2131a7a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-027c19-29894475-nnm44" Nov 03 01:15:00.492174 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:15:00.489317 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f0a403e0-c375-412d-940d-66ecf89e3270-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w3-83755a-29894475-clqb4\" (UID: \"f0a403e0-c375-412d-940d-66ecf89e3270\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-83755a-29894475-clqb4" Nov 03 01:15:00.492174 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:15:00.489348 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7eba01d0-b475-421d-8aba-56cbc67c652d-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w4-6c0ec0-29894475-8vfnr\" (UID: \"7eba01d0-b475-421d-8aba-56cbc67c652d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-6c0ec0-29894475-8vfnr" Nov 03 01:15:00.503966 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:15:00.502379 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7eba01d0-b475-421d-8aba-56cbc67c652d-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w4-6c0ec0-29894475-8vfnr\" (UID: \"7eba01d0-b475-421d-8aba-56cbc67c652d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-6c0ec0-29894475-8vfnr" Nov 03 01:15:00.523154 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:15:00.517465 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ca1eab39-5984-43f1-bd8c-029cd2131a7a-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-a-027c19-29894475-nnm44\" (UID: \"ca1eab39-5984-43f1-bd8c-029cd2131a7a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-027c19-29894475-nnm44" Nov 03 01:15:00.523154 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:15:00.518234 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f0a403e0-c375-412d-940d-66ecf89e3270-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w3-83755a-29894475-clqb4\" (UID: \"f0a403e0-c375-412d-940d-66ecf89e3270\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-83755a-29894475-clqb4" Nov 03 01:15:00.527113 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:15:00.525218 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6bc4e95b-2075-4115-a856-975abe775ad6-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-a-821f5a-29894475-6tzm9\" (UID: \"6bc4e95b-2075-4115-a856-975abe775ad6\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-821f5a-29894475-6tzm9" Nov 03 01:15:00.539932 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:15:00.535099 4939 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-2729f2-29894475-pcj4t" Nov 03 01:15:00.550846 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:15:00.550743 4939 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-020af9-29894475-xhbqm" Nov 03 01:15:00.568208 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:15:00.568077 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jvd6s\" (UniqueName: \"kubernetes.io/projected/899bb599-2a0d-4670-818b-b0eda152816f-kube-api-access-jvd6s\") pod \"maas-api-key-cleanup-e2e-extmod-b-af8244-29894475-h4vvr\" (UID: \"899bb599-2a0d-4670-818b-b0eda152816f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-af8244-29894475-h4vvr" Nov 03 01:15:00.568882 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:15:00.568847 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hvpc4\" (UniqueName: \"kubernetes.io/projected/ca1eab39-5984-43f1-bd8c-029cd2131a7a-kube-api-access-hvpc4\") pod \"maas-api-key-cleanup-e2e-extmod-a-027c19-29894475-nnm44\" (UID: \"ca1eab39-5984-43f1-bd8c-029cd2131a7a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-027c19-29894475-nnm44" Nov 03 01:15:00.569031 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:15:00.568847 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5sp52\" (UniqueName: \"kubernetes.io/projected/7eba01d0-b475-421d-8aba-56cbc67c652d-kube-api-access-5sp52\") pod \"maas-api-key-cleanup-e2e-shared-a-w4-6c0ec0-29894475-8vfnr\" (UID: \"7eba01d0-b475-421d-8aba-56cbc67c652d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-6c0ec0-29894475-8vfnr" Nov 03 01:15:00.569231 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:15:00.569201 4939 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-6c0ec0-29894475-8vfnr" Nov 03 01:15:00.570842 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:15:00.570791 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4nwl2\" (UniqueName: \"kubernetes.io/projected/6bc4e95b-2075-4115-a856-975abe775ad6-kube-api-access-4nwl2\") pod \"maas-api-key-cleanup-e2e-extmod-a-821f5a-29894475-6tzm9\" (UID: \"6bc4e95b-2075-4115-a856-975abe775ad6\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-821f5a-29894475-6tzm9" Nov 03 01:15:00.573772 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:15:00.573738 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zbl6w\" (UniqueName: \"kubernetes.io/projected/f0a403e0-c375-412d-940d-66ecf89e3270-kube-api-access-zbl6w\") pod \"maas-api-key-cleanup-e2e-worker-w3-83755a-29894475-clqb4\" (UID: \"f0a403e0-c375-412d-940d-66ecf89e3270\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-83755a-29894475-clqb4" Nov 03 01:15:00.584164 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:15:00.584120 4939 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-83755a-29894475-clqb4" Nov 03 01:15:00.596979 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:15:00.596900 4939 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-af8244-29894475-h4vvr" Nov 03 01:15:00.604000 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:15:00.603972 4939 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-821f5a-29894475-6tzm9" Nov 03 01:15:00.857314 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:15:00.857278 4939 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-027c19-29894475-nnm44" Nov 03 01:15:00.904568 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:15:00.904535 4939 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-020af9-29894475-xhbqm"] Nov 03 01:15:00.910410 ip-10-0-140-107 kubenswrapper[4939]: W1103 01:15:00.909805 4939 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8abddaba_107d_4380_9ae4_5c912e6f4a03.slice/crio-35b67ba7dd1f4006ec81c259865566530c845dd734f9193882d1a1219a214d07 WatchSource:0}: Error finding container 35b67ba7dd1f4006ec81c259865566530c845dd734f9193882d1a1219a214d07: Status 404 returned error can't find the container with id 35b67ba7dd1f4006ec81c259865566530c845dd734f9193882d1a1219a214d07 Nov 03 01:15:01.071701 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:15:01.071667 4939 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-2729f2-29894475-pcj4t"] Nov 03 01:15:01.106176 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:15:01.106145 4939 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-6c0ec0-29894475-8vfnr"] Nov 03 01:15:01.127961 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:15:01.127936 4939 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-83755a-29894475-clqb4"] Nov 03 01:15:01.150172 ip-10-0-140-107 kubenswrapper[4939]: W1103 01:15:01.150132 4939 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda3c9ab79_8ade_4c81_9e5a_38803c90f7a2.slice/crio-f277b586e53b7c2d3d9243f70c8e2ec93f954dde8f37faeb9961b607df3f963f WatchSource:0}: Error finding container f277b586e53b7c2d3d9243f70c8e2ec93f954dde8f37faeb9961b607df3f963f: Status 404 returned error can't find the container with id f277b586e53b7c2d3d9243f70c8e2ec93f954dde8f37faeb9961b607df3f963f Nov 03 01:15:01.151160 ip-10-0-140-107 kubenswrapper[4939]: W1103 01:15:01.151139 4939 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7eba01d0_b475_421d_8aba_56cbc67c652d.slice/crio-c33c34d25ef128df92b0ad8d9bb4004e1cfc917aced3b67e50dda5419419bb3d WatchSource:0}: Error finding container c33c34d25ef128df92b0ad8d9bb4004e1cfc917aced3b67e50dda5419419bb3d: Status 404 returned error can't find the container with id c33c34d25ef128df92b0ad8d9bb4004e1cfc917aced3b67e50dda5419419bb3d Nov 03 01:15:01.152587 ip-10-0-140-107 kubenswrapper[4939]: W1103 01:15:01.152564 4939 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf0a403e0_c375_412d_940d_66ecf89e3270.slice/crio-82a8d8c3b54ab926489932e2f45b23ad5bb483a9a68e0a733c7236258fff7f00 WatchSource:0}: Error finding container 82a8d8c3b54ab926489932e2f45b23ad5bb483a9a68e0a733c7236258fff7f00: Status 404 returned error can't find the container with id 82a8d8c3b54ab926489932e2f45b23ad5bb483a9a68e0a733c7236258fff7f00 Nov 03 01:15:01.172074 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:15:01.172048 4939 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-af8244-29894475-h4vvr"] Nov 03 01:15:01.173892 ip-10-0-140-107 kubenswrapper[4939]: W1103 01:15:01.173870 4939 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod899bb599_2a0d_4670_818b_b0eda152816f.slice/crio-ff03087a82abc50bd736a0195c0d31e8b6994900b6a521727bdf3c158fa82862 WatchSource:0}: Error finding container ff03087a82abc50bd736a0195c0d31e8b6994900b6a521727bdf3c158fa82862: Status 404 returned error can't find the container with id ff03087a82abc50bd736a0195c0d31e8b6994900b6a521727bdf3c158fa82862 Nov 03 01:15:01.183673 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:15:01.183652 4939 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-821f5a-29894475-6tzm9"] Nov 03 01:15:01.190082 ip-10-0-140-107 kubenswrapper[4939]: W1103 01:15:01.190051 4939 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6bc4e95b_2075_4115_a856_975abe775ad6.slice/crio-9e1deaec4ff42bbbf358aceb047fc0919139861b920954e81ef15e12dfeada90 WatchSource:0}: Error finding container 9e1deaec4ff42bbbf358aceb047fc0919139861b920954e81ef15e12dfeada90: Status 404 returned error can't find the container with id 9e1deaec4ff42bbbf358aceb047fc0919139861b920954e81ef15e12dfeada90 Nov 03 01:15:01.190588 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:15:01.190477 4939 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-027c19-29894475-nnm44"] Nov 03 01:15:01.200563 ip-10-0-140-107 kubenswrapper[4939]: W1103 01:15:01.200537 4939 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podca1eab39_5984_43f1_bd8c_029cd2131a7a.slice/crio-77463523acbb67960da6b0eacf1abfdcdc77d9313b6d84e433e23a4caa6abc46 WatchSource:0}: Error finding container 77463523acbb67960da6b0eacf1abfdcdc77d9313b6d84e433e23a4caa6abc46: Status 404 returned error can't find the container with id 77463523acbb67960da6b0eacf1abfdcdc77d9313b6d84e433e23a4caa6abc46 Nov 03 01:15:01.262725 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:15:01.262701 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:15:01.404343 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:15:01.404289 4939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-6c0ec0-29894475-8vfnr" event={"ID":"7eba01d0-b475-421d-8aba-56cbc67c652d","Type":"ContainerStarted","Data":"c33c34d25ef128df92b0ad8d9bb4004e1cfc917aced3b67e50dda5419419bb3d"} Nov 03 01:15:01.408623 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:15:01.408533 4939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-2729f2-29894475-pcj4t" event={"ID":"a3c9ab79-8ade-4c81-9e5a-38803c90f7a2","Type":"ContainerStarted","Data":"f277b586e53b7c2d3d9243f70c8e2ec93f954dde8f37faeb9961b607df3f963f"} Nov 03 01:15:01.415232 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:15:01.415149 4939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-027c19-29894475-nnm44" event={"ID":"ca1eab39-5984-43f1-bd8c-029cd2131a7a","Type":"ContainerStarted","Data":"77463523acbb67960da6b0eacf1abfdcdc77d9313b6d84e433e23a4caa6abc46"} Nov 03 01:15:01.418283 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:15:01.417759 4939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-020af9-29894475-xhbqm" event={"ID":"8abddaba-107d-4380-9ae4-5c912e6f4a03","Type":"ContainerStarted","Data":"35b67ba7dd1f4006ec81c259865566530c845dd734f9193882d1a1219a214d07"} Nov 03 01:15:01.420389 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:15:01.420306 4939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-83755a-29894475-clqb4" event={"ID":"f0a403e0-c375-412d-940d-66ecf89e3270","Type":"ContainerStarted","Data":"82a8d8c3b54ab926489932e2f45b23ad5bb483a9a68e0a733c7236258fff7f00"} Nov 03 01:15:01.424520 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:15:01.424372 4939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-821f5a-29894475-6tzm9" event={"ID":"6bc4e95b-2075-4115-a856-975abe775ad6","Type":"ContainerStarted","Data":"9e1deaec4ff42bbbf358aceb047fc0919139861b920954e81ef15e12dfeada90"} Nov 03 01:15:01.425896 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:15:01.425857 4939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-af8244-29894475-h4vvr" event={"ID":"899bb599-2a0d-4670-818b-b0eda152816f","Type":"ContainerStarted","Data":"ff03087a82abc50bd736a0195c0d31e8b6994900b6a521727bdf3c158fa82862"} Nov 03 01:15:01.885292 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:15:01.885251 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:15:02.438040 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:15:02.437952 4939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-83755a-29894475-clqb4" event={"ID":"f0a403e0-c375-412d-940d-66ecf89e3270","Type":"ContainerStarted","Data":"296ed725c2474a8272876c4d2bf86fb70d4119c00fd9ec367b6e3bf208a13715"} Nov 03 01:15:02.440126 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:15:02.440064 4939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-821f5a-29894475-6tzm9" event={"ID":"6bc4e95b-2075-4115-a856-975abe775ad6","Type":"ContainerStarted","Data":"77953dda41cdfabd8c7e00dc38bcda52baca6b374b1f5cc0a9bfc5e43d44ff78"} Nov 03 01:15:02.443504 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:15:02.443465 4939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-af8244-29894475-h4vvr" event={"ID":"899bb599-2a0d-4670-818b-b0eda152816f","Type":"ContainerStarted","Data":"f52167c34598e744d033a35371d074fa6754687c0b6c447c4f05e7301dc1019b"} Nov 03 01:15:02.445463 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:15:02.445430 4939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-6c0ec0-29894475-8vfnr" event={"ID":"7eba01d0-b475-421d-8aba-56cbc67c652d","Type":"ContainerStarted","Data":"1642ed856e1eb27de2cb06eef10bd7210f37a57c1c0fe1d88ef6537548d3fe43"} Nov 03 01:15:02.450485 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:15:02.450157 4939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-2729f2-29894475-pcj4t" event={"ID":"a3c9ab79-8ade-4c81-9e5a-38803c90f7a2","Type":"ContainerStarted","Data":"93e188a41883bf0c692c096f3122a1319714e61b584a75cb827df96a7c15352b"} Nov 03 01:15:02.461398 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:15:02.461369 4939 generic.go:356] "Generic (PLEG): container finished" podID="ca1eab39-5984-43f1-bd8c-029cd2131a7a" containerID="e2a31619d24fed9a4c812b2ff53a94aaac84edfc786f95626008c18dede62ea8" exitCode=60 Nov 03 01:15:02.461528 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:15:02.461452 4939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-027c19-29894475-nnm44" event={"ID":"ca1eab39-5984-43f1-bd8c-029cd2131a7a","Type":"ContainerDied","Data":"e2a31619d24fed9a4c812b2ff53a94aaac84edfc786f95626008c18dede62ea8"} Nov 03 01:15:02.461756 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:15:02.461734 4939 scope.go:122] "RemoveContainer" containerID="e2a31619d24fed9a4c812b2ff53a94aaac84edfc786f95626008c18dede62ea8" Nov 03 01:15:02.463577 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:15:02.463551 4939 generic.go:356] "Generic (PLEG): container finished" podID="8abddaba-107d-4380-9ae4-5c912e6f4a03" containerID="232bcdfbd69e363b1ea6716e142e3a0bf46fbac4632c16e78f7c751de0455eed" exitCode=60 Nov 03 01:15:02.463748 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:15:02.463722 4939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-020af9-29894475-xhbqm" event={"ID":"8abddaba-107d-4380-9ae4-5c912e6f4a03","Type":"ContainerDied","Data":"232bcdfbd69e363b1ea6716e142e3a0bf46fbac4632c16e78f7c751de0455eed"} Nov 03 01:15:02.464002 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:15:02.463979 4939 scope.go:122] "RemoveContainer" containerID="232bcdfbd69e363b1ea6716e142e3a0bf46fbac4632c16e78f7c751de0455eed" Nov 03 01:15:02.474332 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:15:02.474280 4939 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-83755a-29894475-clqb4" podStartSLOduration=2.474265948 podStartE2EDuration="2.474265948s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 01:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 01:15:02.457466941 +0000 UTC m=+2050.721801899" watchObservedRunningTime="2026-11-03 01:15:02.474265948 +0000 UTC m=+2050.738600895" Nov 03 01:15:02.475129 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:15:02.475085 4939 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-6c0ec0-29894475-8vfnr" podStartSLOduration=2.475072241 podStartE2EDuration="2.475072241s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 01:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 01:15:02.473241562 +0000 UTC m=+2050.737576504" watchObservedRunningTime="2026-11-03 01:15:02.475072241 +0000 UTC m=+2050.739407186" Nov 03 01:15:02.499228 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:15:02.498226 4939 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-2729f2-29894475-pcj4t" podStartSLOduration=2.498207212 podStartE2EDuration="2.498207212s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 01:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 01:15:02.490521393 +0000 UTC m=+2050.754856339" watchObservedRunningTime="2026-11-03 01:15:02.498207212 +0000 UTC m=+2050.762542162" Nov 03 01:15:02.510578 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:15:02.510532 4939 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-af8244-29894475-h4vvr" podStartSLOduration=2.510518553 podStartE2EDuration="2.510518553s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 01:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 01:15:02.506646657 +0000 UTC m=+2050.770981597" watchObservedRunningTime="2026-11-03 01:15:02.510518553 +0000 UTC m=+2050.774853497" Nov 03 01:15:02.524364 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:15:02.524319 4939 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-821f5a-29894475-6tzm9" podStartSLOduration=2.524306084 podStartE2EDuration="2.524306084s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 01:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 01:15:02.523939147 +0000 UTC m=+2050.788274093" watchObservedRunningTime="2026-11-03 01:15:02.524306084 +0000 UTC m=+2050.788641029" Nov 03 01:15:02.569339 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:15:02.569300 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:15:03.293863 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:15:03.293784 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:15:03.476636 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:15:03.476533 4939 generic.go:356] "Generic (PLEG): container finished" podID="f0a403e0-c375-412d-940d-66ecf89e3270" containerID="296ed725c2474a8272876c4d2bf86fb70d4119c00fd9ec367b6e3bf208a13715" exitCode=60 Nov 03 01:15:03.476636 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:15:03.476615 4939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-83755a-29894475-clqb4" event={"ID":"f0a403e0-c375-412d-940d-66ecf89e3270","Type":"ContainerDied","Data":"296ed725c2474a8272876c4d2bf86fb70d4119c00fd9ec367b6e3bf208a13715"} Nov 03 01:15:03.477218 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:15:03.477001 4939 scope.go:122] "RemoveContainer" containerID="296ed725c2474a8272876c4d2bf86fb70d4119c00fd9ec367b6e3bf208a13715" Nov 03 01:15:03.479161 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:15:03.479114 4939 generic.go:356] "Generic (PLEG): container finished" podID="6bc4e95b-2075-4115-a856-975abe775ad6" containerID="77953dda41cdfabd8c7e00dc38bcda52baca6b374b1f5cc0a9bfc5e43d44ff78" exitCode=60 Nov 03 01:15:03.479224 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:15:03.479204 4939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-821f5a-29894475-6tzm9" event={"ID":"6bc4e95b-2075-4115-a856-975abe775ad6","Type":"ContainerDied","Data":"77953dda41cdfabd8c7e00dc38bcda52baca6b374b1f5cc0a9bfc5e43d44ff78"} Nov 03 01:15:03.479461 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:15:03.479439 4939 scope.go:122] "RemoveContainer" containerID="77953dda41cdfabd8c7e00dc38bcda52baca6b374b1f5cc0a9bfc5e43d44ff78" Nov 03 01:15:03.481674 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:15:03.481638 4939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-af8244-29894475-h4vvr" event={"ID":"899bb599-2a0d-4670-818b-b0eda152816f","Type":"ContainerDied","Data":"f52167c34598e744d033a35371d074fa6754687c0b6c447c4f05e7301dc1019b"} Nov 03 01:15:03.481819 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:15:03.481659 4939 scope.go:122] "RemoveContainer" containerID="f52167c34598e744d033a35371d074fa6754687c0b6c447c4f05e7301dc1019b" Nov 03 01:15:03.482216 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:15:03.481360 4939 generic.go:356] "Generic (PLEG): container finished" podID="899bb599-2a0d-4670-818b-b0eda152816f" containerID="f52167c34598e744d033a35371d074fa6754687c0b6c447c4f05e7301dc1019b" exitCode=60 Nov 03 01:15:03.486382 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:15:03.486359 4939 generic.go:356] "Generic (PLEG): container finished" podID="7eba01d0-b475-421d-8aba-56cbc67c652d" containerID="1642ed856e1eb27de2cb06eef10bd7210f37a57c1c0fe1d88ef6537548d3fe43" exitCode=60 Nov 03 01:15:03.486498 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:15:03.486480 4939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-6c0ec0-29894475-8vfnr" event={"ID":"7eba01d0-b475-421d-8aba-56cbc67c652d","Type":"ContainerDied","Data":"1642ed856e1eb27de2cb06eef10bd7210f37a57c1c0fe1d88ef6537548d3fe43"} Nov 03 01:15:03.486681 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:15:03.486661 4939 scope.go:122] "RemoveContainer" containerID="1642ed856e1eb27de2cb06eef10bd7210f37a57c1c0fe1d88ef6537548d3fe43" Nov 03 01:15:03.488271 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:15:03.488250 4939 generic.go:356] "Generic (PLEG): container finished" podID="a3c9ab79-8ade-4c81-9e5a-38803c90f7a2" containerID="93e188a41883bf0c692c096f3122a1319714e61b584a75cb827df96a7c15352b" exitCode=60 Nov 03 01:15:03.488371 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:15:03.488276 4939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-2729f2-29894475-pcj4t" event={"ID":"a3c9ab79-8ade-4c81-9e5a-38803c90f7a2","Type":"ContainerDied","Data":"93e188a41883bf0c692c096f3122a1319714e61b584a75cb827df96a7c15352b"} Nov 03 01:15:03.488608 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:15:03.488579 4939 scope.go:122] "RemoveContainer" containerID="93e188a41883bf0c692c096f3122a1319714e61b584a75cb827df96a7c15352b" Nov 03 01:15:03.490689 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:15:03.490655 4939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-027c19-29894475-nnm44" event={"ID":"ca1eab39-5984-43f1-bd8c-029cd2131a7a","Type":"ContainerStarted","Data":"918af5696142b71dae0eb6e8d67f5366a55c9c73562e0e5514deff47c8ebd0f5"} Nov 03 01:15:03.494842 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:15:03.494209 4939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-020af9-29894475-xhbqm" event={"ID":"8abddaba-107d-4380-9ae4-5c912e6f4a03","Type":"ContainerStarted","Data":"1545582c38793e64535020b67b39bbe67dee5815f61704f7cd60f6495de20584"} Nov 03 01:15:03.553355 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:15:03.553288 4939 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-027c19-29894475-nnm44" podStartSLOduration=3.5532693650000002 podStartE2EDuration="3.553269365s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 01:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 01:15:03.544448539 +0000 UTC m=+2051.808783489" watchObservedRunningTime="2026-11-03 01:15:03.553269365 +0000 UTC m=+2051.817604311" Nov 03 01:15:03.597744 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:15:03.597683 4939 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-020af9-29894475-xhbqm" podStartSLOduration=3.597665526 podStartE2EDuration="3.597665526s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 01:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 01:15:03.592609432 +0000 UTC m=+2051.856944379" watchObservedRunningTime="2026-11-03 01:15:03.597665526 +0000 UTC m=+2051.862000472" Nov 03 01:15:03.989629 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:15:03.989581 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:15:04.224622 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:15:04.224590 4939 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-4pw7v"] Nov 03 01:15:04.608940 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:15:04.607326 4939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-2729f2-29894475-pcj4t" event={"ID":"a3c9ab79-8ade-4c81-9e5a-38803c90f7a2","Type":"ContainerStarted","Data":"fad0f4d85a73faf1e5905dc4a4d8fb3ec9181b14c0fd2c21cd92152ec06bc2bf"} Nov 03 01:15:04.644335 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:15:04.633131 4939 generic.go:356] "Generic (PLEG): container finished" podID="ca1eab39-5984-43f1-bd8c-029cd2131a7a" containerID="918af5696142b71dae0eb6e8d67f5366a55c9c73562e0e5514deff47c8ebd0f5" exitCode=60 Nov 03 01:15:04.644335 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:15:04.633236 4939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-027c19-29894475-nnm44" event={"ID":"ca1eab39-5984-43f1-bd8c-029cd2131a7a","Type":"ContainerDied","Data":"918af5696142b71dae0eb6e8d67f5366a55c9c73562e0e5514deff47c8ebd0f5"} Nov 03 01:15:04.644335 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:15:04.633296 4939 scope.go:122] "RemoveContainer" containerID="e2a31619d24fed9a4c812b2ff53a94aaac84edfc786f95626008c18dede62ea8" Nov 03 01:15:04.678855 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:15:04.669513 4939 scope.go:122] "RemoveContainer" containerID="918af5696142b71dae0eb6e8d67f5366a55c9c73562e0e5514deff47c8ebd0f5" Nov 03 01:15:04.682183 ip-10-0-140-107 kubenswrapper[4939]: E1103 01:15:04.682144 4939 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-a-027c19-29894475-nnm44_odh-ai-gateway-infra(ca1eab39-5984-43f1-bd8c-029cd2131a7a)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-027c19-29894475-nnm44" podUID="ca1eab39-5984-43f1-bd8c-029cd2131a7a" Nov 03 01:15:04.694198 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:15:04.692608 4939 generic.go:356] "Generic (PLEG): container finished" podID="8abddaba-107d-4380-9ae4-5c912e6f4a03" containerID="1545582c38793e64535020b67b39bbe67dee5815f61704f7cd60f6495de20584" exitCode=60 Nov 03 01:15:04.694198 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:15:04.693350 4939 scope.go:122] "RemoveContainer" containerID="1545582c38793e64535020b67b39bbe67dee5815f61704f7cd60f6495de20584" Nov 03 01:15:04.694198 ip-10-0-140-107 kubenswrapper[4939]: E1103 01:15:04.693898 4939 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w0-020af9-29894475-xhbqm_odh-ai-gateway-infra(8abddaba-107d-4380-9ae4-5c912e6f4a03)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-020af9-29894475-xhbqm" podUID="8abddaba-107d-4380-9ae4-5c912e6f4a03" Nov 03 01:15:04.694198 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:15:04.694122 4939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-020af9-29894475-xhbqm" event={"ID":"8abddaba-107d-4380-9ae4-5c912e6f4a03","Type":"ContainerDied","Data":"1545582c38793e64535020b67b39bbe67dee5815f61704f7cd60f6495de20584"} Nov 03 01:15:04.695701 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:15:04.695675 4939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-83755a-29894475-clqb4" event={"ID":"f0a403e0-c375-412d-940d-66ecf89e3270","Type":"ContainerStarted","Data":"9819451ae33994ff4b310008a0660499b68c595317a12394ddb6c7d64a5d97d3"} Nov 03 01:15:04.698595 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:15:04.698572 4939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-821f5a-29894475-6tzm9" event={"ID":"6bc4e95b-2075-4115-a856-975abe775ad6","Type":"ContainerStarted","Data":"e9d88163711c188f01b6d11587e5cade1ed426205d86e7ed039e7db28d4265ea"} Nov 03 01:15:04.701515 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:15:04.701477 4939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-af8244-29894475-h4vvr" event={"ID":"899bb599-2a0d-4670-818b-b0eda152816f","Type":"ContainerStarted","Data":"69cc77f6893fea6207289d842c0f8fcf20cbed3de31cf810475e616d4d557780"} Nov 03 01:15:04.701688 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:15:04.701541 4939 scope.go:122] "RemoveContainer" containerID="69cc77f6893fea6207289d842c0f8fcf20cbed3de31cf810475e616d4d557780" Nov 03 01:15:04.702125 ip-10-0-140-107 kubenswrapper[4939]: E1103 01:15:04.702099 4939 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-b-af8244-29894475-h4vvr_odh-ai-gateway-infra(899bb599-2a0d-4670-818b-b0eda152816f)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-af8244-29894475-h4vvr" podUID="899bb599-2a0d-4670-818b-b0eda152816f" Nov 03 01:15:04.703995 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:15:04.703965 4939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-6c0ec0-29894475-8vfnr" event={"ID":"7eba01d0-b475-421d-8aba-56cbc67c652d","Type":"ContainerStarted","Data":"2cb4dbbe5b61f1903a6a469f34f446ecb484b6d7744df78fc59328cf35d36d91"} Nov 03 01:15:04.738040 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:15:04.738010 4939 scope.go:122] "RemoveContainer" containerID="232bcdfbd69e363b1ea6716e142e3a0bf46fbac4632c16e78f7c751de0455eed" Nov 03 01:15:04.994669 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:15:04.994642 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:15:05.894370 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:15:05.894327 4939 generic.go:356] "Generic (PLEG): container finished" podID="7eba01d0-b475-421d-8aba-56cbc67c652d" containerID="2cb4dbbe5b61f1903a6a469f34f446ecb484b6d7744df78fc59328cf35d36d91" exitCode=60 Nov 03 01:15:05.895791 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:15:05.895767 4939 scope.go:122] "RemoveContainer" containerID="2cb4dbbe5b61f1903a6a469f34f446ecb484b6d7744df78fc59328cf35d36d91" Nov 03 01:15:05.917785 ip-10-0-140-107 kubenswrapper[4939]: E1103 01:15:05.917741 4939 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w4-6c0ec0-29894475-8vfnr_odh-ai-gateway-infra(7eba01d0-b475-421d-8aba-56cbc67c652d)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-6c0ec0-29894475-8vfnr" podUID="7eba01d0-b475-421d-8aba-56cbc67c652d" Nov 03 01:15:06.310513 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:15:06.310203 4939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-6c0ec0-29894475-8vfnr" event={"ID":"7eba01d0-b475-421d-8aba-56cbc67c652d","Type":"ContainerDied","Data":"2cb4dbbe5b61f1903a6a469f34f446ecb484b6d7744df78fc59328cf35d36d91"} Nov 03 01:15:06.311074 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:15:06.311044 4939 scope.go:122] "RemoveContainer" containerID="1642ed856e1eb27de2cb06eef10bd7210f37a57c1c0fe1d88ef6537548d3fe43" Nov 03 01:15:06.312085 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:15:06.312054 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:15:06.672271 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:15:06.672230 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:15:06.707938 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:15:06.688264 4939 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-f7680d/distinct-2-w2-f7680d-kserve-7d7c4564bf-cbv2c" podUID="2de18086-761b-4773-b9fc-15ddc4a220fc" containerName="main" probeResult="failure" output="Get \"https://10.132.0.75:8000/ready\": context deadline exceeded" Nov 03 01:15:06.775933 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:15:06.768071 4939 generic.go:356] "Generic (PLEG): container finished" podID="a3c9ab79-8ade-4c81-9e5a-38803c90f7a2" containerID="fad0f4d85a73faf1e5905dc4a4d8fb3ec9181b14c0fd2c21cd92152ec06bc2bf" exitCode=60 Nov 03 01:15:06.775933 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:15:06.769000 4939 scope.go:122] "RemoveContainer" containerID="fad0f4d85a73faf1e5905dc4a4d8fb3ec9181b14c0fd2c21cd92152ec06bc2bf" Nov 03 01:15:06.775933 ip-10-0-140-107 kubenswrapper[4939]: E1103 01:15:06.769793 4939 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w1-2729f2-29894475-pcj4t_odh-ai-gateway-infra(a3c9ab79-8ade-4c81-9e5a-38803c90f7a2)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-2729f2-29894475-pcj4t" podUID="a3c9ab79-8ade-4c81-9e5a-38803c90f7a2" Nov 03 01:15:06.775933 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:15:06.769868 4939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-2729f2-29894475-pcj4t" event={"ID":"a3c9ab79-8ade-4c81-9e5a-38803c90f7a2","Type":"ContainerDied","Data":"fad0f4d85a73faf1e5905dc4a4d8fb3ec9181b14c0fd2c21cd92152ec06bc2bf"} Nov 03 01:15:06.835581 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:15:06.835546 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:15:06.880071 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:15:06.879250 4939 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-2f9a3a/distinct-w0-2f9a3a-kserve-555565cfcf-jtcr9" podUID="ac0ece06-8718-463f-915f-5315988d9711" containerName="main" probeResult="failure" output="Get \"https://10.132.0.76:8000/ready\": context deadline exceeded" Nov 03 01:15:06.883127 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:15:06.883090 4939 scope.go:122] "RemoveContainer" containerID="918af5696142b71dae0eb6e8d67f5366a55c9c73562e0e5514deff47c8ebd0f5" Nov 03 01:15:06.895084 ip-10-0-140-107 kubenswrapper[4939]: E1103 01:15:06.888239 4939 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-a-027c19-29894475-nnm44_odh-ai-gateway-infra(ca1eab39-5984-43f1-bd8c-029cd2131a7a)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-027c19-29894475-nnm44" podUID="ca1eab39-5984-43f1-bd8c-029cd2131a7a" Nov 03 01:15:06.905502 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:15:06.905462 4939 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-f7680d/premium-w2-f7680d-kserve-76565b8cf9-hqbbt" podUID="4712e708-18a2-4a13-a12c-a5669397b2c3" containerName="main" probeResult="failure" output="Get \"https://10.132.0.63:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 01:15:06.907054 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:15:06.907014 4939 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-2f9a3a/simulator-w0-2f9a3a-kserve-9596bbb8f-2trgz" podUID="207e2207-fb6c-4076-95df-72e95671c261" containerName="main" probeResult="failure" output="Get \"https://10.132.0.61:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Nov 03 01:15:06.920569 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:15:06.920547 4939 scope.go:122] "RemoveContainer" containerID="1545582c38793e64535020b67b39bbe67dee5815f61704f7cd60f6495de20584" Nov 03 01:15:06.932311 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:15:06.932273 4939 scope.go:122] "RemoveContainer" containerID="93e188a41883bf0c692c096f3122a1319714e61b584a75cb827df96a7c15352b" Nov 03 01:15:06.939360 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:15:06.939335 4939 generic.go:356] "Generic (PLEG): container finished" podID="f0a403e0-c375-412d-940d-66ecf89e3270" containerID="9819451ae33994ff4b310008a0660499b68c595317a12394ddb6c7d64a5d97d3" exitCode=60 Nov 03 01:15:06.939643 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:15:06.939619 4939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-83755a-29894475-clqb4" event={"ID":"f0a403e0-c375-412d-940d-66ecf89e3270","Type":"ContainerDied","Data":"9819451ae33994ff4b310008a0660499b68c595317a12394ddb6c7d64a5d97d3"} Nov 03 01:15:06.942529 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:15:06.942498 4939 scope.go:122] "RemoveContainer" containerID="9819451ae33994ff4b310008a0660499b68c595317a12394ddb6c7d64a5d97d3" Nov 03 01:15:06.943493 ip-10-0-140-107 kubenswrapper[4939]: E1103 01:15:06.943451 4939 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w3-83755a-29894475-clqb4_odh-ai-gateway-infra(f0a403e0-c375-412d-940d-66ecf89e3270)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-83755a-29894475-clqb4" podUID="f0a403e0-c375-412d-940d-66ecf89e3270" Nov 03 01:15:06.944618 ip-10-0-140-107 kubenswrapper[4939]: E1103 01:15:06.944587 4939 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w0-020af9-29894475-xhbqm_odh-ai-gateway-infra(8abddaba-107d-4380-9ae4-5c912e6f4a03)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-020af9-29894475-xhbqm" podUID="8abddaba-107d-4380-9ae4-5c912e6f4a03" Nov 03 01:15:06.962578 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:15:06.962546 4939 generic.go:356] "Generic (PLEG): container finished" podID="6bc4e95b-2075-4115-a856-975abe775ad6" containerID="e9d88163711c188f01b6d11587e5cade1ed426205d86e7ed039e7db28d4265ea" exitCode=60 Nov 03 01:15:06.963713 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:15:06.963558 4939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-821f5a-29894475-6tzm9" event={"ID":"6bc4e95b-2075-4115-a856-975abe775ad6","Type":"ContainerDied","Data":"e9d88163711c188f01b6d11587e5cade1ed426205d86e7ed039e7db28d4265ea"} Nov 03 01:15:06.971378 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:15:06.971342 4939 generic.go:356] "Generic (PLEG): container finished" podID="899bb599-2a0d-4670-818b-b0eda152816f" containerID="69cc77f6893fea6207289d842c0f8fcf20cbed3de31cf810475e616d4d557780" exitCode=60 Nov 03 01:15:06.981920 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:15:06.971642 4939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-af8244-29894475-h4vvr" event={"ID":"899bb599-2a0d-4670-818b-b0eda152816f","Type":"ContainerDied","Data":"69cc77f6893fea6207289d842c0f8fcf20cbed3de31cf810475e616d4d557780"} Nov 03 01:15:06.982220 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:15:06.978537 4939 scope.go:122] "RemoveContainer" containerID="e9d88163711c188f01b6d11587e5cade1ed426205d86e7ed039e7db28d4265ea" Nov 03 01:15:06.982453 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:15:06.982437 4939 scope.go:122] "RemoveContainer" containerID="296ed725c2474a8272876c4d2bf86fb70d4119c00fd9ec367b6e3bf208a13715" Nov 03 01:15:06.982649 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:15:06.982632 4939 scope.go:122] "RemoveContainer" containerID="69cc77f6893fea6207289d842c0f8fcf20cbed3de31cf810475e616d4d557780" Nov 03 01:15:06.989200 ip-10-0-140-107 kubenswrapper[4939]: E1103 01:15:06.989142 4939 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-a-821f5a-29894475-6tzm9_odh-ai-gateway-infra(6bc4e95b-2075-4115-a856-975abe775ad6)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-821f5a-29894475-6tzm9" podUID="6bc4e95b-2075-4115-a856-975abe775ad6" Nov 03 01:15:06.989791 ip-10-0-140-107 kubenswrapper[4939]: E1103 01:15:06.989761 4939 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-b-af8244-29894475-h4vvr_odh-ai-gateway-infra(899bb599-2a0d-4670-818b-b0eda152816f)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-af8244-29894475-h4vvr" podUID="899bb599-2a0d-4670-818b-b0eda152816f" Nov 03 01:15:07.030890 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:15:07.030870 4939 scope.go:122] "RemoveContainer" containerID="77953dda41cdfabd8c7e00dc38bcda52baca6b374b1f5cc0a9bfc5e43d44ff78" Nov 03 01:15:07.068218 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:15:07.068192 4939 scope.go:122] "RemoveContainer" containerID="f52167c34598e744d033a35371d074fa6754687c0b6c447c4f05e7301dc1019b" Nov 03 01:15:07.461315 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:15:07.461282 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:15:07.993028 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:15:07.992998 4939 scope.go:122] "RemoveContainer" containerID="9819451ae33994ff4b310008a0660499b68c595317a12394ddb6c7d64a5d97d3" Nov 03 01:15:07.993479 ip-10-0-140-107 kubenswrapper[4939]: E1103 01:15:07.993449 4939 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w3-83755a-29894475-clqb4_odh-ai-gateway-infra(f0a403e0-c375-412d-940d-66ecf89e3270)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-83755a-29894475-clqb4" podUID="f0a403e0-c375-412d-940d-66ecf89e3270" Nov 03 01:15:07.994793 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:15:07.994767 4939 scope.go:122] "RemoveContainer" containerID="e9d88163711c188f01b6d11587e5cade1ed426205d86e7ed039e7db28d4265ea" Nov 03 01:15:07.995178 ip-10-0-140-107 kubenswrapper[4939]: E1103 01:15:07.995157 4939 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-a-821f5a-29894475-6tzm9_odh-ai-gateway-infra(6bc4e95b-2075-4115-a856-975abe775ad6)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-821f5a-29894475-6tzm9" podUID="6bc4e95b-2075-4115-a856-975abe775ad6" Nov 03 01:15:07.996468 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:15:07.996445 4939 scope.go:122] "RemoveContainer" containerID="69cc77f6893fea6207289d842c0f8fcf20cbed3de31cf810475e616d4d557780" Nov 03 01:15:07.996811 ip-10-0-140-107 kubenswrapper[4939]: E1103 01:15:07.996783 4939 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-b-af8244-29894475-h4vvr_odh-ai-gateway-infra(899bb599-2a0d-4670-818b-b0eda152816f)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-af8244-29894475-h4vvr" podUID="899bb599-2a0d-4670-818b-b0eda152816f" Nov 03 01:15:07.998283 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:15:07.998264 4939 scope.go:122] "RemoveContainer" containerID="2cb4dbbe5b61f1903a6a469f34f446ecb484b6d7744df78fc59328cf35d36d91" Nov 03 01:15:07.998598 ip-10-0-140-107 kubenswrapper[4939]: E1103 01:15:07.998577 4939 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w4-6c0ec0-29894475-8vfnr_odh-ai-gateway-infra(7eba01d0-b475-421d-8aba-56cbc67c652d)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-6c0ec0-29894475-8vfnr" podUID="7eba01d0-b475-421d-8aba-56cbc67c652d" Nov 03 01:15:07.999996 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:15:07.999977 4939 scope.go:122] "RemoveContainer" containerID="fad0f4d85a73faf1e5905dc4a4d8fb3ec9181b14c0fd2c21cd92152ec06bc2bf" Nov 03 01:15:08.000229 ip-10-0-140-107 kubenswrapper[4939]: E1103 01:15:08.000212 4939 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w1-2729f2-29894475-pcj4t_odh-ai-gateway-infra(a3c9ab79-8ade-4c81-9e5a-38803c90f7a2)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-2729f2-29894475-pcj4t" podUID="a3c9ab79-8ade-4c81-9e5a-38803c90f7a2" Nov 03 01:15:08.371499 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:15:08.371467 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:15:08.852443 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:15:08.852411 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:15:09.088495 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:15:09.088461 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:15:09.778880 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:15:09.778845 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:15:09.947936 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:15:09.940190 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:15:10.771212 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:15:10.771166 4939 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-88bbf7-openshift-default-574f47fff7-rf5cn"] Nov 03 01:15:10.771692 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:15:10.771439 4939 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-88bbf7-openshift-default-574f47fff7-rf5cn" podUID="08f62205-b31d-4863-ab9e-3b13b15fbef5" containerName="istio-proxy" containerID="cri-o://b0f0b67f3e3bf35f2888c554bb137203035916673d4c5e7d02f47cb5dbb416db" gracePeriod=30 Nov 03 01:15:11.163351 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:15:11.163317 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:15:11.790429 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:15:11.790397 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:15:12.614198 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:15:12.614161 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:15:12.962567 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:15:12.962479 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:15:13.093383 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:15:13.093325 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:15:13.212106 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:15:13.212071 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:15:13.536934 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:15:13.530996 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:15:14.348419 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:15:14.348381 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:15:14.566960 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:15:14.566921 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:15:14.711992 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:15:14.701360 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:15:15.385315 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:15:15.385258 4939 scope.go:122] "RemoveContainer" containerID="1545582c38793e64535020b67b39bbe67dee5815f61704f7cd60f6495de20584" Nov 03 01:15:15.392809 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:15:15.392779 4939 scope.go:122] "RemoveContainer" containerID="918af5696142b71dae0eb6e8d67f5366a55c9c73562e0e5514deff47c8ebd0f5" Nov 03 01:15:15.594122 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:15:15.594075 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:15:15.596051 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:15:15.596018 4939 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-f7680d/distinct-2-w2-f7680d-kserve-7d7c4564bf-cbv2c" podUID="2de18086-761b-4773-b9fc-15ddc4a220fc" containerName="main" probeResult="failure" output="Get \"https://10.132.0.75:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 01:15:15.631813 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:15:15.631766 4939 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-83755a/distinct-2-w3-83755a-kserve-6cf66b995d-2l47b" podUID="c3b05e88-5d5b-4d03-b28d-832f22e3c868" containerName="main" probeResult="failure" output="Get \"https://10.132.0.72:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 01:15:15.861043 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:15:15.861015 4939 scope.go:122] "RemoveContainer" containerID="fad0f4d85a73faf1e5905dc4a4d8fb3ec9181b14c0fd2c21cd92152ec06bc2bf" Nov 03 01:15:15.861819 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:15:15.861267 4939 scope.go:122] "RemoveContainer" containerID="e9d88163711c188f01b6d11587e5cade1ed426205d86e7ed039e7db28d4265ea" Nov 03 01:15:15.968919 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:15:15.968879 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:15:16.091562 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:15:16.091468 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:15:16.227098 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:15:16.227037 4939 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-88bbf7-openshift-default-574f47fff7-rf5cn" Nov 03 01:15:16.273301 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:15:16.273274 4939 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-extmod-a-d25759-openshift-default-d47fd9785-7gzj2"] Nov 03 01:15:16.274428 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:15:16.274404 4939 cpu_manager.go:414] "containerMap: removing container" podUID="08f62205-b31d-4863-ab9e-3b13b15fbef5" containerName="istio-proxy" Nov 03 01:15:16.274428 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:15:16.274425 4939 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="08f62205-b31d-4863-ab9e-3b13b15fbef5" containerName="istio-proxy" Nov 03 01:15:16.274611 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:15:16.274450 4939 cpu_manager.go:414] "containerMap: removing container" podUID="08f62205-b31d-4863-ab9e-3b13b15fbef5" containerName="istio-proxy" Nov 03 01:15:16.274611 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:15:16.274455 4939 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="08f62205-b31d-4863-ab9e-3b13b15fbef5" containerName="istio-proxy" Nov 03 01:15:16.274708 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:15:16.274695 4939 memory_manager.go:365] "RemoveStaleState removing state" podUID="08f62205-b31d-4863-ab9e-3b13b15fbef5" containerName="istio-proxy" Nov 03 01:15:16.274787 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:15:16.274709 4939 memory_manager.go:365] "RemoveStaleState removing state" podUID="08f62205-b31d-4863-ab9e-3b13b15fbef5" containerName="istio-proxy" Nov 03 01:15:16.279079 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:15:16.279054 4939 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-d25759-openshift-default-d47fd9785-7gzj2" Nov 03 01:15:16.302508 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:15:16.300036 4939 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-a-d25759-openshift-default-d47fd9785-7gzj2"] Nov 03 01:15:16.332367 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:15:16.332335 4939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/08f62205-b31d-4863-ab9e-3b13b15fbef5-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/08f62205-b31d-4863-ab9e-3b13b15fbef5-istio-podinfo\") pod \"08f62205-b31d-4863-ab9e-3b13b15fbef5\" (UID: \"08f62205-b31d-4863-ab9e-3b13b15fbef5\") " Nov 03 01:15:16.332579 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:15:16.332493 4939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/08f62205-b31d-4863-ab9e-3b13b15fbef5-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/08f62205-b31d-4863-ab9e-3b13b15fbef5-credential-socket\") pod \"08f62205-b31d-4863-ab9e-3b13b15fbef5\" (UID: \"08f62205-b31d-4863-ab9e-3b13b15fbef5\") " Nov 03 01:15:16.332579 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:15:16.332526 4939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/08f62205-b31d-4863-ab9e-3b13b15fbef5-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/08f62205-b31d-4863-ab9e-3b13b15fbef5-workload-socket\") pod \"08f62205-b31d-4863-ab9e-3b13b15fbef5\" (UID: \"08f62205-b31d-4863-ab9e-3b13b15fbef5\") " Nov 03 01:15:16.332579 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:15:16.332552 4939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/08f62205-b31d-4863-ab9e-3b13b15fbef5-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/08f62205-b31d-4863-ab9e-3b13b15fbef5-istio-envoy\") pod \"08f62205-b31d-4863-ab9e-3b13b15fbef5\" (UID: \"08f62205-b31d-4863-ab9e-3b13b15fbef5\") " Nov 03 01:15:16.332833 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:15:16.332604 4939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/08f62205-b31d-4863-ab9e-3b13b15fbef5-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/08f62205-b31d-4863-ab9e-3b13b15fbef5-workload-certs\") pod \"08f62205-b31d-4863-ab9e-3b13b15fbef5\" (UID: \"08f62205-b31d-4863-ab9e-3b13b15fbef5\") " Nov 03 01:15:16.332833 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:15:16.332808 4939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/08f62205-b31d-4863-ab9e-3b13b15fbef5-credential-socket" pod "08f62205-b31d-4863-ab9e-3b13b15fbef5" (UID: "08f62205-b31d-4863-ab9e-3b13b15fbef5"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 01:15:16.332833 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:15:16.332835 4939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/08f62205-b31d-4863-ab9e-3b13b15fbef5-istio-token\" (UniqueName: \"kubernetes.io/projected/08f62205-b31d-4863-ab9e-3b13b15fbef5-istio-token\") pod \"08f62205-b31d-4863-ab9e-3b13b15fbef5\" (UID: \"08f62205-b31d-4863-ab9e-3b13b15fbef5\") " Nov 03 01:15:16.333133 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:15:16.332877 4939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/08f62205-b31d-4863-ab9e-3b13b15fbef5-kube-api-access-fq5xx\" (UniqueName: \"kubernetes.io/projected/08f62205-b31d-4863-ab9e-3b13b15fbef5-kube-api-access-fq5xx\") pod \"08f62205-b31d-4863-ab9e-3b13b15fbef5\" (UID: \"08f62205-b31d-4863-ab9e-3b13b15fbef5\") " Nov 03 01:15:16.333133 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:15:16.332929 4939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/08f62205-b31d-4863-ab9e-3b13b15fbef5-workload-socket" pod "08f62205-b31d-4863-ab9e-3b13b15fbef5" (UID: "08f62205-b31d-4863-ab9e-3b13b15fbef5"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 01:15:16.333133 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:15:16.332967 4939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/08f62205-b31d-4863-ab9e-3b13b15fbef5-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/08f62205-b31d-4863-ab9e-3b13b15fbef5-istio-data\") pod \"08f62205-b31d-4863-ab9e-3b13b15fbef5\" (UID: \"08f62205-b31d-4863-ab9e-3b13b15fbef5\") " Nov 03 01:15:16.333133 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:15:16.333065 4939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/08f62205-b31d-4863-ab9e-3b13b15fbef5-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/08f62205-b31d-4863-ab9e-3b13b15fbef5-istiod-ca-cert\") pod \"08f62205-b31d-4863-ab9e-3b13b15fbef5\" (UID: \"08f62205-b31d-4863-ab9e-3b13b15fbef5\") " Nov 03 01:15:16.333409 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:15:16.333200 4939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/08f62205-b31d-4863-ab9e-3b13b15fbef5-workload-certs" pod "08f62205-b31d-4863-ab9e-3b13b15fbef5" (UID: "08f62205-b31d-4863-ab9e-3b13b15fbef5"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 01:15:16.333409 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:15:16.333355 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ba585f4b-4916-4d03-bf1e-454c029d46ad-istio-token\") pod \"e2e-extmod-a-d25759-openshift-default-d47fd9785-7gzj2\" (UID: \"ba585f4b-4916-4d03-bf1e-454c029d46ad\") " pod="openshift-ingress/e2e-extmod-a-d25759-openshift-default-d47fd9785-7gzj2" Nov 03 01:15:16.333526 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:15:16.333406 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ba585f4b-4916-4d03-bf1e-454c029d46ad-credential-socket\") pod \"e2e-extmod-a-d25759-openshift-default-d47fd9785-7gzj2\" (UID: \"ba585f4b-4916-4d03-bf1e-454c029d46ad\") " pod="openshift-ingress/e2e-extmod-a-d25759-openshift-default-d47fd9785-7gzj2" Nov 03 01:15:16.333526 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:15:16.333433 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ba585f4b-4916-4d03-bf1e-454c029d46ad-istio-podinfo\") pod \"e2e-extmod-a-d25759-openshift-default-d47fd9785-7gzj2\" (UID: \"ba585f4b-4916-4d03-bf1e-454c029d46ad\") " pod="openshift-ingress/e2e-extmod-a-d25759-openshift-default-d47fd9785-7gzj2" Nov 03 01:15:16.333604 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:15:16.333547 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ba585f4b-4916-4d03-bf1e-454c029d46ad-workload-certs\") pod \"e2e-extmod-a-d25759-openshift-default-d47fd9785-7gzj2\" (UID: \"ba585f4b-4916-4d03-bf1e-454c029d46ad\") " pod="openshift-ingress/e2e-extmod-a-d25759-openshift-default-d47fd9785-7gzj2" Nov 03 01:15:16.333642 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:15:16.333613 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ba585f4b-4916-4d03-bf1e-454c029d46ad-istio-data\") pod \"e2e-extmod-a-d25759-openshift-default-d47fd9785-7gzj2\" (UID: \"ba585f4b-4916-4d03-bf1e-454c029d46ad\") " pod="openshift-ingress/e2e-extmod-a-d25759-openshift-default-d47fd9785-7gzj2" Nov 03 01:15:16.333740 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:15:16.333726 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ba585f4b-4916-4d03-bf1e-454c029d46ad-workload-socket\") pod \"e2e-extmod-a-d25759-openshift-default-d47fd9785-7gzj2\" (UID: \"ba585f4b-4916-4d03-bf1e-454c029d46ad\") " pod="openshift-ingress/e2e-extmod-a-d25759-openshift-default-d47fd9785-7gzj2" Nov 03 01:15:16.333782 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:15:16.333754 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ba585f4b-4916-4d03-bf1e-454c029d46ad-istiod-ca-cert\") pod \"e2e-extmod-a-d25759-openshift-default-d47fd9785-7gzj2\" (UID: \"ba585f4b-4916-4d03-bf1e-454c029d46ad\") " pod="openshift-ingress/e2e-extmod-a-d25759-openshift-default-d47fd9785-7gzj2" Nov 03 01:15:16.333833 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:15:16.333822 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rstkw\" (UniqueName: \"kubernetes.io/projected/ba585f4b-4916-4d03-bf1e-454c029d46ad-kube-api-access-rstkw\") pod \"e2e-extmod-a-d25759-openshift-default-d47fd9785-7gzj2\" (UID: \"ba585f4b-4916-4d03-bf1e-454c029d46ad\") " pod="openshift-ingress/e2e-extmod-a-d25759-openshift-default-d47fd9785-7gzj2" Nov 03 01:15:16.333885 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:15:16.333874 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ba585f4b-4916-4d03-bf1e-454c029d46ad-istio-envoy\") pod \"e2e-extmod-a-d25759-openshift-default-d47fd9785-7gzj2\" (UID: \"ba585f4b-4916-4d03-bf1e-454c029d46ad\") " pod="openshift-ingress/e2e-extmod-a-d25759-openshift-default-d47fd9785-7gzj2" Nov 03 01:15:16.334043 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:15:16.334020 4939 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/08f62205-b31d-4863-ab9e-3b13b15fbef5-credential-socket\") on node \"ip-10-0-140-107.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:15:16.334043 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:15:16.334045 4939 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/08f62205-b31d-4863-ab9e-3b13b15fbef5-workload-socket\") on node \"ip-10-0-140-107.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:15:16.334242 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:15:16.334059 4939 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/08f62205-b31d-4863-ab9e-3b13b15fbef5-workload-certs\") on node \"ip-10-0-140-107.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:15:16.334543 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:15:16.334523 4939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/08f62205-b31d-4863-ab9e-3b13b15fbef5-istiod-ca-cert" pod "08f62205-b31d-4863-ab9e-3b13b15fbef5" (UID: "08f62205-b31d-4863-ab9e-3b13b15fbef5"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 01:15:16.334627 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:15:16.334581 4939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/08f62205-b31d-4863-ab9e-3b13b15fbef5-istio-data" pod "08f62205-b31d-4863-ab9e-3b13b15fbef5" (UID: "08f62205-b31d-4863-ab9e-3b13b15fbef5"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 01:15:16.336216 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:15:16.336192 4939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/08f62205-b31d-4863-ab9e-3b13b15fbef5-istio-token" pod "08f62205-b31d-4863-ab9e-3b13b15fbef5" (UID: "08f62205-b31d-4863-ab9e-3b13b15fbef5"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 01:15:16.349076 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:15:16.349034 4939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-027c19-29894475-nnm44" event={"ID":"ca1eab39-5984-43f1-bd8c-029cd2131a7a","Type":"ContainerStarted","Data":"83dd3ab636da82ef190c0b9ea3df71072c5b0e5d39279abefb4ef1e2308329ee"} Nov 03 01:15:16.353843 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:15:16.353803 4939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-020af9-29894475-xhbqm" event={"ID":"8abddaba-107d-4380-9ae4-5c912e6f4a03","Type":"ContainerStarted","Data":"6052e16e3231660b40e3f603962d835ccad947d23a71d7b2352db238305f039d"} Nov 03 01:15:16.356622 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:15:16.356593 4939 generic.go:356] "Generic (PLEG): container finished" podID="08f62205-b31d-4863-ab9e-3b13b15fbef5" containerID="b0f0b67f3e3bf35f2888c554bb137203035916673d4c5e7d02f47cb5dbb416db" exitCode=0 Nov 03 01:15:16.356793 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:15:16.356665 4939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-88bbf7-openshift-default-574f47fff7-rf5cn" event={"ID":"08f62205-b31d-4863-ab9e-3b13b15fbef5","Type":"ContainerDied","Data":"b0f0b67f3e3bf35f2888c554bb137203035916673d4c5e7d02f47cb5dbb416db"} Nov 03 01:15:16.356793 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:15:16.356707 4939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-88bbf7-openshift-default-574f47fff7-rf5cn" event={"ID":"08f62205-b31d-4863-ab9e-3b13b15fbef5","Type":"ContainerDied","Data":"11fc00cb2213c4a3846cc8e0d4aacd2ab70d7747e57f04e0d64c759393dff938"} Nov 03 01:15:16.356793 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:15:16.356738 4939 scope.go:122] "RemoveContainer" containerID="b0f0b67f3e3bf35f2888c554bb137203035916673d4c5e7d02f47cb5dbb416db" Nov 03 01:15:16.357092 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:15:16.356901 4939 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-88bbf7-openshift-default-574f47fff7-rf5cn" Nov 03 01:15:16.360589 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:15:16.360548 4939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/08f62205-b31d-4863-ab9e-3b13b15fbef5-istio-envoy" pod "08f62205-b31d-4863-ab9e-3b13b15fbef5" (UID: "08f62205-b31d-4863-ab9e-3b13b15fbef5"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 01:15:16.360800 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:15:16.360770 4939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/08f62205-b31d-4863-ab9e-3b13b15fbef5-istio-podinfo" pod "08f62205-b31d-4863-ab9e-3b13b15fbef5" (UID: "08f62205-b31d-4863-ab9e-3b13b15fbef5"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 01:15:16.366737 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:15:16.366698 4939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/08f62205-b31d-4863-ab9e-3b13b15fbef5-kube-api-access-fq5xx" pod "08f62205-b31d-4863-ab9e-3b13b15fbef5" (UID: "08f62205-b31d-4863-ab9e-3b13b15fbef5"). InnerVolumeSpecName "kube-api-access-fq5xx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 01:15:16.372097 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:15:16.372072 4939 scope.go:122] "RemoveContainer" containerID="9d890d4ea180cbf4f646aedeabe5f01cf348d5f8e216255c3d18cca4804707e6" Nov 03 01:15:16.410885 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:15:16.410639 4939 scope.go:122] "RemoveContainer" containerID="b0f0b67f3e3bf35f2888c554bb137203035916673d4c5e7d02f47cb5dbb416db" Nov 03 01:15:16.411307 ip-10-0-140-107 kubenswrapper[4939]: E1103 01:15:16.411140 4939 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b0f0b67f3e3bf35f2888c554bb137203035916673d4c5e7d02f47cb5dbb416db\": container with ID starting with b0f0b67f3e3bf35f2888c554bb137203035916673d4c5e7d02f47cb5dbb416db not found: ID does not exist" containerID="b0f0b67f3e3bf35f2888c554bb137203035916673d4c5e7d02f47cb5dbb416db" Nov 03 01:15:16.411307 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:15:16.411193 4939 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b0f0b67f3e3bf35f2888c554bb137203035916673d4c5e7d02f47cb5dbb416db"} err="failed to get container status \"b0f0b67f3e3bf35f2888c554bb137203035916673d4c5e7d02f47cb5dbb416db\": rpc error: code = NotFound desc = could not find container \"b0f0b67f3e3bf35f2888c554bb137203035916673d4c5e7d02f47cb5dbb416db\": container with ID starting with b0f0b67f3e3bf35f2888c554bb137203035916673d4c5e7d02f47cb5dbb416db not found: ID does not exist" Nov 03 01:15:16.411307 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:15:16.411220 4939 scope.go:122] "RemoveContainer" containerID="9d890d4ea180cbf4f646aedeabe5f01cf348d5f8e216255c3d18cca4804707e6" Nov 03 01:15:16.411495 ip-10-0-140-107 kubenswrapper[4939]: E1103 01:15:16.411469 4939 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9d890d4ea180cbf4f646aedeabe5f01cf348d5f8e216255c3d18cca4804707e6\": container with ID starting with 9d890d4ea180cbf4f646aedeabe5f01cf348d5f8e216255c3d18cca4804707e6 not found: ID does not exist" containerID="9d890d4ea180cbf4f646aedeabe5f01cf348d5f8e216255c3d18cca4804707e6" Nov 03 01:15:16.411609 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:15:16.411498 4939 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9d890d4ea180cbf4f646aedeabe5f01cf348d5f8e216255c3d18cca4804707e6"} err="failed to get container status \"9d890d4ea180cbf4f646aedeabe5f01cf348d5f8e216255c3d18cca4804707e6\": rpc error: code = NotFound desc = could not find container \"9d890d4ea180cbf4f646aedeabe5f01cf348d5f8e216255c3d18cca4804707e6\": container with ID starting with 9d890d4ea180cbf4f646aedeabe5f01cf348d5f8e216255c3d18cca4804707e6 not found: ID does not exist" Nov 03 01:15:16.436272 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:15:16.436228 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ba585f4b-4916-4d03-bf1e-454c029d46ad-workload-socket\") pod \"e2e-extmod-a-d25759-openshift-default-d47fd9785-7gzj2\" (UID: \"ba585f4b-4916-4d03-bf1e-454c029d46ad\") " pod="openshift-ingress/e2e-extmod-a-d25759-openshift-default-d47fd9785-7gzj2" Nov 03 01:15:16.436479 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:15:16.436290 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ba585f4b-4916-4d03-bf1e-454c029d46ad-istiod-ca-cert\") pod \"e2e-extmod-a-d25759-openshift-default-d47fd9785-7gzj2\" (UID: \"ba585f4b-4916-4d03-bf1e-454c029d46ad\") " pod="openshift-ingress/e2e-extmod-a-d25759-openshift-default-d47fd9785-7gzj2" Nov 03 01:15:16.436479 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:15:16.436387 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rstkw\" (UniqueName: \"kubernetes.io/projected/ba585f4b-4916-4d03-bf1e-454c029d46ad-kube-api-access-rstkw\") pod \"e2e-extmod-a-d25759-openshift-default-d47fd9785-7gzj2\" (UID: \"ba585f4b-4916-4d03-bf1e-454c029d46ad\") " pod="openshift-ingress/e2e-extmod-a-d25759-openshift-default-d47fd9785-7gzj2" Nov 03 01:15:16.436479 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:15:16.436455 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ba585f4b-4916-4d03-bf1e-454c029d46ad-istio-envoy\") pod \"e2e-extmod-a-d25759-openshift-default-d47fd9785-7gzj2\" (UID: \"ba585f4b-4916-4d03-bf1e-454c029d46ad\") " pod="openshift-ingress/e2e-extmod-a-d25759-openshift-default-d47fd9785-7gzj2" Nov 03 01:15:16.436672 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:15:16.436585 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ba585f4b-4916-4d03-bf1e-454c029d46ad-istio-token\") pod \"e2e-extmod-a-d25759-openshift-default-d47fd9785-7gzj2\" (UID: \"ba585f4b-4916-4d03-bf1e-454c029d46ad\") " pod="openshift-ingress/e2e-extmod-a-d25759-openshift-default-d47fd9785-7gzj2" Nov 03 01:15:16.436672 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:15:16.436653 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ba585f4b-4916-4d03-bf1e-454c029d46ad-credential-socket\") pod \"e2e-extmod-a-d25759-openshift-default-d47fd9785-7gzj2\" (UID: \"ba585f4b-4916-4d03-bf1e-454c029d46ad\") " pod="openshift-ingress/e2e-extmod-a-d25759-openshift-default-d47fd9785-7gzj2" Nov 03 01:15:16.436789 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:15:16.436669 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ba585f4b-4916-4d03-bf1e-454c029d46ad-workload-socket\") pod \"e2e-extmod-a-d25759-openshift-default-d47fd9785-7gzj2\" (UID: \"ba585f4b-4916-4d03-bf1e-454c029d46ad\") " pod="openshift-ingress/e2e-extmod-a-d25759-openshift-default-d47fd9785-7gzj2" Nov 03 01:15:16.436789 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:15:16.436683 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ba585f4b-4916-4d03-bf1e-454c029d46ad-istio-podinfo\") pod \"e2e-extmod-a-d25759-openshift-default-d47fd9785-7gzj2\" (UID: \"ba585f4b-4916-4d03-bf1e-454c029d46ad\") " pod="openshift-ingress/e2e-extmod-a-d25759-openshift-default-d47fd9785-7gzj2" Nov 03 01:15:16.436896 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:15:16.436831 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ba585f4b-4916-4d03-bf1e-454c029d46ad-workload-certs\") pod \"e2e-extmod-a-d25759-openshift-default-d47fd9785-7gzj2\" (UID: \"ba585f4b-4916-4d03-bf1e-454c029d46ad\") " pod="openshift-ingress/e2e-extmod-a-d25759-openshift-default-d47fd9785-7gzj2" Nov 03 01:15:16.437009 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:15:16.436986 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ba585f4b-4916-4d03-bf1e-454c029d46ad-istio-data\") pod \"e2e-extmod-a-d25759-openshift-default-d47fd9785-7gzj2\" (UID: \"ba585f4b-4916-4d03-bf1e-454c029d46ad\") " pod="openshift-ingress/e2e-extmod-a-d25759-openshift-default-d47fd9785-7gzj2" Nov 03 01:15:16.437077 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:15:16.437022 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ba585f4b-4916-4d03-bf1e-454c029d46ad-credential-socket\") pod \"e2e-extmod-a-d25759-openshift-default-d47fd9785-7gzj2\" (UID: \"ba585f4b-4916-4d03-bf1e-454c029d46ad\") " pod="openshift-ingress/e2e-extmod-a-d25759-openshift-default-d47fd9785-7gzj2" Nov 03 01:15:16.437168 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:15:16.437134 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ba585f4b-4916-4d03-bf1e-454c029d46ad-istiod-ca-cert\") pod \"e2e-extmod-a-d25759-openshift-default-d47fd9785-7gzj2\" (UID: \"ba585f4b-4916-4d03-bf1e-454c029d46ad\") " pod="openshift-ingress/e2e-extmod-a-d25759-openshift-default-d47fd9785-7gzj2" Nov 03 01:15:16.437242 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:15:16.437169 4939 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/08f62205-b31d-4863-ab9e-3b13b15fbef5-istio-envoy\") on node \"ip-10-0-140-107.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:15:16.437242 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:15:16.437192 4939 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/08f62205-b31d-4863-ab9e-3b13b15fbef5-istio-token\") on node \"ip-10-0-140-107.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:15:16.437242 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:15:16.437209 4939 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fq5xx\" (UniqueName: \"kubernetes.io/projected/08f62205-b31d-4863-ab9e-3b13b15fbef5-kube-api-access-fq5xx\") on node \"ip-10-0-140-107.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:15:16.437242 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:15:16.437224 4939 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/08f62205-b31d-4863-ab9e-3b13b15fbef5-istio-data\") on node \"ip-10-0-140-107.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:15:16.437242 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:15:16.437237 4939 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/08f62205-b31d-4863-ab9e-3b13b15fbef5-istiod-ca-cert\") on node \"ip-10-0-140-107.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:15:16.437471 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:15:16.437251 4939 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/08f62205-b31d-4863-ab9e-3b13b15fbef5-istio-podinfo\") on node \"ip-10-0-140-107.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:15:16.437471 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:15:16.437206 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ba585f4b-4916-4d03-bf1e-454c029d46ad-workload-certs\") pod \"e2e-extmod-a-d25759-openshift-default-d47fd9785-7gzj2\" (UID: \"ba585f4b-4916-4d03-bf1e-454c029d46ad\") " pod="openshift-ingress/e2e-extmod-a-d25759-openshift-default-d47fd9785-7gzj2" Nov 03 01:15:16.437471 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:15:16.437429 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ba585f4b-4916-4d03-bf1e-454c029d46ad-istio-data\") pod \"e2e-extmod-a-d25759-openshift-default-d47fd9785-7gzj2\" (UID: \"ba585f4b-4916-4d03-bf1e-454c029d46ad\") " pod="openshift-ingress/e2e-extmod-a-d25759-openshift-default-d47fd9785-7gzj2" Nov 03 01:15:16.438980 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:15:16.438957 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ba585f4b-4916-4d03-bf1e-454c029d46ad-istio-envoy\") pod \"e2e-extmod-a-d25759-openshift-default-d47fd9785-7gzj2\" (UID: \"ba585f4b-4916-4d03-bf1e-454c029d46ad\") " pod="openshift-ingress/e2e-extmod-a-d25759-openshift-default-d47fd9785-7gzj2" Nov 03 01:15:16.439128 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:15:16.439111 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ba585f4b-4916-4d03-bf1e-454c029d46ad-istio-podinfo\") pod \"e2e-extmod-a-d25759-openshift-default-d47fd9785-7gzj2\" (UID: \"ba585f4b-4916-4d03-bf1e-454c029d46ad\") " pod="openshift-ingress/e2e-extmod-a-d25759-openshift-default-d47fd9785-7gzj2" Nov 03 01:15:16.445050 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:15:16.445029 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rstkw\" (UniqueName: \"kubernetes.io/projected/ba585f4b-4916-4d03-bf1e-454c029d46ad-kube-api-access-rstkw\") pod \"e2e-extmod-a-d25759-openshift-default-d47fd9785-7gzj2\" (UID: \"ba585f4b-4916-4d03-bf1e-454c029d46ad\") " pod="openshift-ingress/e2e-extmod-a-d25759-openshift-default-d47fd9785-7gzj2" Nov 03 01:15:16.445229 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:15:16.445211 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ba585f4b-4916-4d03-bf1e-454c029d46ad-istio-token\") pod \"e2e-extmod-a-d25759-openshift-default-d47fd9785-7gzj2\" (UID: \"ba585f4b-4916-4d03-bf1e-454c029d46ad\") " pod="openshift-ingress/e2e-extmod-a-d25759-openshift-default-d47fd9785-7gzj2" Nov 03 01:15:16.615836 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:15:16.615792 4939 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-d25759-openshift-default-d47fd9785-7gzj2" Nov 03 01:15:16.692824 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:15:16.692749 4939 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-88bbf7-openshift-default-574f47fff7-rf5cn"] Nov 03 01:15:16.701921 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:15:16.701864 4939 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-88bbf7-openshift-default-574f47fff7-rf5cn"] Nov 03 01:15:16.805796 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:15:16.805707 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:15:16.925757 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:15:16.925722 4939 scope.go:122] "RemoveContainer" containerID="69cc77f6893fea6207289d842c0f8fcf20cbed3de31cf810475e616d4d557780" Nov 03 01:15:16.926564 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:15:16.926542 4939 scope.go:122] "RemoveContainer" containerID="2cb4dbbe5b61f1903a6a469f34f446ecb484b6d7744df78fc59328cf35d36d91" Nov 03 01:15:16.960362 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:15:16.960329 4939 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-a-d25759-openshift-default-d47fd9785-7gzj2"] Nov 03 01:15:17.004383 ip-10-0-140-107 kubenswrapper[4939]: W1103 01:15:17.004351 4939 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podba585f4b_4916_4d03_bf1e_454c029d46ad.slice/crio-c746ff69fb39792575d80fb88e665bfa7bf1e612cdfb668ce9b2670dbc84a992 WatchSource:0}: Error finding container c746ff69fb39792575d80fb88e665bfa7bf1e612cdfb668ce9b2670dbc84a992: Status 404 returned error can't find the container with id c746ff69fb39792575d80fb88e665bfa7bf1e612cdfb668ce9b2670dbc84a992 Nov 03 01:15:17.007819 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:15:17.007779 4939 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 01:15:17.007964 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:15:17.007871 4939 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 01:15:17.007964 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:15:17.007937 4939 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 01:15:17.372696 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:15:17.372660 4939 generic.go:356] "Generic (PLEG): container finished" podID="a3c9ab79-8ade-4c81-9e5a-38803c90f7a2" containerID="fdb68ec87c7b03c2b24eb1f1c6760d77fa2896c3cccfd8f0c8cf1df7baf5643d" exitCode=60 Nov 03 01:15:17.372925 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:15:17.372704 4939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-2729f2-29894475-pcj4t" event={"ID":"a3c9ab79-8ade-4c81-9e5a-38803c90f7a2","Type":"ContainerDied","Data":"fdb68ec87c7b03c2b24eb1f1c6760d77fa2896c3cccfd8f0c8cf1df7baf5643d"} Nov 03 01:15:17.372925 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:15:17.372777 4939 scope.go:122] "RemoveContainer" containerID="fad0f4d85a73faf1e5905dc4a4d8fb3ec9181b14c0fd2c21cd92152ec06bc2bf" Nov 03 01:15:17.373351 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:15:17.373335 4939 scope.go:122] "RemoveContainer" containerID="fdb68ec87c7b03c2b24eb1f1c6760d77fa2896c3cccfd8f0c8cf1df7baf5643d" Nov 03 01:15:17.373781 ip-10-0-140-107 kubenswrapper[4939]: E1103 01:15:17.373754 4939 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w1-2729f2-29894475-pcj4t_odh-ai-gateway-infra(a3c9ab79-8ade-4c81-9e5a-38803c90f7a2)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-2729f2-29894475-pcj4t" podUID="a3c9ab79-8ade-4c81-9e5a-38803c90f7a2" Nov 03 01:15:17.375452 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:15:17.375427 4939 generic.go:356] "Generic (PLEG): container finished" podID="ca1eab39-5984-43f1-bd8c-029cd2131a7a" containerID="83dd3ab636da82ef190c0b9ea3df71072c5b0e5d39279abefb4ef1e2308329ee" exitCode=60 Nov 03 01:15:17.375561 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:15:17.375500 4939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-027c19-29894475-nnm44" event={"ID":"ca1eab39-5984-43f1-bd8c-029cd2131a7a","Type":"ContainerDied","Data":"83dd3ab636da82ef190c0b9ea3df71072c5b0e5d39279abefb4ef1e2308329ee"} Nov 03 01:15:17.376068 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:15:17.376047 4939 scope.go:122] "RemoveContainer" containerID="83dd3ab636da82ef190c0b9ea3df71072c5b0e5d39279abefb4ef1e2308329ee" Nov 03 01:15:17.376603 ip-10-0-140-107 kubenswrapper[4939]: E1103 01:15:17.376574 4939 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-a-027c19-29894475-nnm44_odh-ai-gateway-infra(ca1eab39-5984-43f1-bd8c-029cd2131a7a)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-027c19-29894475-nnm44" podUID="ca1eab39-5984-43f1-bd8c-029cd2131a7a" Nov 03 01:15:17.379244 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:15:17.379223 4939 generic.go:356] "Generic (PLEG): container finished" podID="8abddaba-107d-4380-9ae4-5c912e6f4a03" containerID="6052e16e3231660b40e3f603962d835ccad947d23a71d7b2352db238305f039d" exitCode=60 Nov 03 01:15:17.379350 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:15:17.379304 4939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-020af9-29894475-xhbqm" event={"ID":"8abddaba-107d-4380-9ae4-5c912e6f4a03","Type":"ContainerDied","Data":"6052e16e3231660b40e3f603962d835ccad947d23a71d7b2352db238305f039d"} Nov 03 01:15:17.380254 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:15:17.380230 4939 scope.go:122] "RemoveContainer" containerID="6052e16e3231660b40e3f603962d835ccad947d23a71d7b2352db238305f039d" Nov 03 01:15:17.380705 ip-10-0-140-107 kubenswrapper[4939]: E1103 01:15:17.380679 4939 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w0-020af9-29894475-xhbqm_odh-ai-gateway-infra(8abddaba-107d-4380-9ae4-5c912e6f4a03)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-020af9-29894475-xhbqm" podUID="8abddaba-107d-4380-9ae4-5c912e6f4a03" Nov 03 01:15:17.381588 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:15:17.381566 4939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-d25759-openshift-default-d47fd9785-7gzj2" event={"ID":"ba585f4b-4916-4d03-bf1e-454c029d46ad","Type":"ContainerStarted","Data":"4dff96a063f74b8b786efbb9331372ae589a636a6bf3dadbf90aa7da47d8e532"} Nov 03 01:15:17.381682 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:15:17.381612 4939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-d25759-openshift-default-d47fd9785-7gzj2" event={"ID":"ba585f4b-4916-4d03-bf1e-454c029d46ad","Type":"ContainerStarted","Data":"c746ff69fb39792575d80fb88e665bfa7bf1e612cdfb668ce9b2670dbc84a992"} Nov 03 01:15:17.383992 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:15:17.383968 4939 generic.go:356] "Generic (PLEG): container finished" podID="6bc4e95b-2075-4115-a856-975abe775ad6" containerID="d2ea491757f2582ce132b9a16c53df7157388847c81a70fdc4d8f1341605dcc5" exitCode=60 Nov 03 01:15:17.384079 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:15:17.384015 4939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-821f5a-29894475-6tzm9" event={"ID":"6bc4e95b-2075-4115-a856-975abe775ad6","Type":"ContainerDied","Data":"d2ea491757f2582ce132b9a16c53df7157388847c81a70fdc4d8f1341605dcc5"} Nov 03 01:15:17.384079 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:15:17.384050 4939 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-027c19-29894475-nnm44"] Nov 03 01:15:17.384381 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:15:17.384367 4939 scope.go:122] "RemoveContainer" containerID="d2ea491757f2582ce132b9a16c53df7157388847c81a70fdc4d8f1341605dcc5" Nov 03 01:15:17.384648 ip-10-0-140-107 kubenswrapper[4939]: E1103 01:15:17.384628 4939 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-a-821f5a-29894475-6tzm9_odh-ai-gateway-infra(6bc4e95b-2075-4115-a856-975abe775ad6)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-821f5a-29894475-6tzm9" podUID="6bc4e95b-2075-4115-a856-975abe775ad6" Nov 03 01:15:17.389760 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:15:17.389706 4939 scope.go:122] "RemoveContainer" containerID="918af5696142b71dae0eb6e8d67f5366a55c9c73562e0e5514deff47c8ebd0f5" Nov 03 01:15:17.406795 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:15:17.406760 4939 scope.go:122] "RemoveContainer" containerID="1545582c38793e64535020b67b39bbe67dee5815f61704f7cd60f6495de20584" Nov 03 01:15:17.424321 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:15:17.424130 4939 scope.go:122] "RemoveContainer" containerID="e9d88163711c188f01b6d11587e5cade1ed426205d86e7ed039e7db28d4265ea" Nov 03 01:15:17.425118 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:15:17.425094 4939 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-020af9-29894475-xhbqm"] Nov 03 01:15:17.448940 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:15:17.448871 4939 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-extmod-a-d25759-openshift-default-d47fd9785-7gzj2" podStartSLOduration=185.448855023 podStartE2EDuration="3m5.448855023s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 01:12:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 01:15:17.441465557 +0000 UTC m=+2065.705800506" watchObservedRunningTime="2026-11-03 01:15:17.448855023 +0000 UTC m=+2065.713189968" Nov 03 01:15:17.603454 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:15:17.603420 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:15:17.616055 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:15:17.616022 4939 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-extmod-a-d25759-openshift-default-d47fd9785-7gzj2" Nov 03 01:15:17.861742 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:15:17.861713 4939 scope.go:122] "RemoveContainer" containerID="9819451ae33994ff4b310008a0660499b68c595317a12394ddb6c7d64a5d97d3" Nov 03 01:15:17.877204 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:15:17.877169 4939 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="08f62205-b31d-4863-ab9e-3b13b15fbef5" path="/var/lib/kubelet/pods/08f62205-b31d-4863-ab9e-3b13b15fbef5/volumes" Nov 03 01:15:18.034192 ip-10-0-140-107 kubenswrapper[4939]: E1103 01:15:18.034153 4939 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-sthjh" containerName="extractor" Nov 03 01:15:18.397950 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:15:18.397263 4939 generic.go:356] "Generic (PLEG): container finished" podID="899bb599-2a0d-4670-818b-b0eda152816f" containerID="161ebebe6568108bba787ec5ad637f88794a68747dc850ec96830ea92b1d6265" exitCode=60 Nov 03 01:15:18.397950 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:15:18.397377 4939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-af8244-29894475-h4vvr" event={"ID":"899bb599-2a0d-4670-818b-b0eda152816f","Type":"ContainerDied","Data":"161ebebe6568108bba787ec5ad637f88794a68747dc850ec96830ea92b1d6265"} Nov 03 01:15:18.397950 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:15:18.397464 4939 scope.go:122] "RemoveContainer" containerID="69cc77f6893fea6207289d842c0f8fcf20cbed3de31cf810475e616d4d557780" Nov 03 01:15:18.399511 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:15:18.399489 4939 scope.go:122] "RemoveContainer" containerID="161ebebe6568108bba787ec5ad637f88794a68747dc850ec96830ea92b1d6265" Nov 03 01:15:18.399700 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:15:18.399674 4939 generic.go:356] "Generic (PLEG): container finished" podID="7eba01d0-b475-421d-8aba-56cbc67c652d" containerID="2ea63d250680ae771b6f051a37d7774005f41964879c47684ad6dc5d274b4dad" exitCode=60 Nov 03 01:15:18.399824 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:15:18.399806 4939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-6c0ec0-29894475-8vfnr" event={"ID":"7eba01d0-b475-421d-8aba-56cbc67c652d","Type":"ContainerDied","Data":"2ea63d250680ae771b6f051a37d7774005f41964879c47684ad6dc5d274b4dad"} Nov 03 01:15:18.399946 ip-10-0-140-107 kubenswrapper[4939]: E1103 01:15:18.399899 4939 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-b-af8244-29894475-h4vvr_odh-ai-gateway-infra(899bb599-2a0d-4670-818b-b0eda152816f)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-af8244-29894475-h4vvr" podUID="899bb599-2a0d-4670-818b-b0eda152816f" Nov 03 01:15:18.400329 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:15:18.400308 4939 scope.go:122] "RemoveContainer" containerID="2ea63d250680ae771b6f051a37d7774005f41964879c47684ad6dc5d274b4dad" Nov 03 01:15:18.400784 ip-10-0-140-107 kubenswrapper[4939]: E1103 01:15:18.400754 4939 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w4-6c0ec0-29894475-8vfnr_odh-ai-gateway-infra(7eba01d0-b475-421d-8aba-56cbc67c652d)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-6c0ec0-29894475-8vfnr" podUID="7eba01d0-b475-421d-8aba-56cbc67c652d" Nov 03 01:15:18.405216 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:15:18.405189 4939 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-2729f2-29894475-pcj4t"] Nov 03 01:15:18.419310 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:15:18.419281 4939 scope.go:122] "RemoveContainer" containerID="2cb4dbbe5b61f1903a6a469f34f446ecb484b6d7744df78fc59328cf35d36d91" Nov 03 01:15:18.501344 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:15:18.500957 4939 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-821f5a-29894475-6tzm9"] Nov 03 01:15:18.712186 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:15:18.531710 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:15:18.712186 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:15:18.691064 4939 patch_prober.go:29] interesting pod/e2e-extmod-a-d25759-openshift-default-d47fd9785-7gzj2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.237:15021/healthz/ready\": dial tcp 10.132.0.237:15021: connect: connection refused" start-of-body= Nov 03 01:15:18.712186 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:15:18.691125 4939 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-d25759-openshift-default-d47fd9785-7gzj2" podUID="ba585f4b-4916-4d03-bf1e-454c029d46ad" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.237:15021/healthz/ready\": dial tcp 10.132.0.237:15021: connect: connection refused" Nov 03 01:15:18.875948 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:15:18.875376 4939 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-020af9-29894475-xhbqm" Nov 03 01:15:18.915353 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:15:18.915316 4939 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-556c14-29894475-xxwnm"] Nov 03 01:15:18.916885 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:15:18.916861 4939 cpu_manager.go:414] "containerMap: removing container" podUID="8abddaba-107d-4380-9ae4-5c912e6f4a03" containerName="cleanup" Nov 03 01:15:18.916999 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:15:18.916888 4939 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8abddaba-107d-4380-9ae4-5c912e6f4a03" containerName="cleanup" Nov 03 01:15:18.916999 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:15:18.916964 4939 cpu_manager.go:414] "containerMap: removing container" podUID="8abddaba-107d-4380-9ae4-5c912e6f4a03" containerName="cleanup" Nov 03 01:15:18.916999 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:15:18.916972 4939 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8abddaba-107d-4380-9ae4-5c912e6f4a03" containerName="cleanup" Nov 03 01:15:18.917159 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:15:18.917020 4939 cpu_manager.go:414] "containerMap: removing container" podUID="8abddaba-107d-4380-9ae4-5c912e6f4a03" containerName="cleanup" Nov 03 01:15:18.917159 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:15:18.917028 4939 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8abddaba-107d-4380-9ae4-5c912e6f4a03" containerName="cleanup" Nov 03 01:15:18.917324 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:15:18.917270 4939 memory_manager.go:365] "RemoveStaleState removing state" podUID="8abddaba-107d-4380-9ae4-5c912e6f4a03" containerName="cleanup" Nov 03 01:15:18.917324 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:15:18.917309 4939 memory_manager.go:365] "RemoveStaleState removing state" podUID="8abddaba-107d-4380-9ae4-5c912e6f4a03" containerName="cleanup" Nov 03 01:15:18.934594 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:15:18.934559 4939 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-556c14-29894475-xxwnm" Nov 03 01:15:18.944983 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:15:18.942298 4939 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-556c14-29894475-xxwnm"] Nov 03 01:15:19.015583 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:15:19.015554 4939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8abddaba-107d-4380-9ae4-5c912e6f4a03-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8abddaba-107d-4380-9ae4-5c912e6f4a03-maas-api-service-ca\") pod \"8abddaba-107d-4380-9ae4-5c912e6f4a03\" (UID: \"8abddaba-107d-4380-9ae4-5c912e6f4a03\") " Nov 03 01:15:19.015713 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:15:19.015641 4939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8abddaba-107d-4380-9ae4-5c912e6f4a03-kube-api-access-s754v\" (UniqueName: \"kubernetes.io/projected/8abddaba-107d-4380-9ae4-5c912e6f4a03-kube-api-access-s754v\") pod \"8abddaba-107d-4380-9ae4-5c912e6f4a03\" (UID: \"8abddaba-107d-4380-9ae4-5c912e6f4a03\") " Nov 03 01:15:19.015939 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:15:19.015898 4939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8abddaba-107d-4380-9ae4-5c912e6f4a03-maas-api-service-ca" pod "8abddaba-107d-4380-9ae4-5c912e6f4a03" (UID: "8abddaba-107d-4380-9ae4-5c912e6f4a03"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 01:15:19.016793 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:15:19.016770 4939 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8abddaba-107d-4380-9ae4-5c912e6f4a03-maas-api-service-ca\") on node \"ip-10-0-140-107.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:15:19.019175 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:15:19.019148 4939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8abddaba-107d-4380-9ae4-5c912e6f4a03-kube-api-access-s754v" pod "8abddaba-107d-4380-9ae4-5c912e6f4a03" (UID: "8abddaba-107d-4380-9ae4-5c912e6f4a03"). InnerVolumeSpecName "kube-api-access-s754v". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 01:15:19.092935 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:15:19.092816 4939 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-027c19-29894475-nnm44" Nov 03 01:15:19.096651 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:15:19.096621 4939 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-2729f2-29894475-pcj4t" Nov 03 01:15:19.118657 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:15:19.118620 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-db5x9\" (UniqueName: \"kubernetes.io/projected/8a294b17-dd47-4c08-afb6-4c7236938c01-kube-api-access-db5x9\") pod \"maas-api-key-cleanup-e2e-shared-b-w3-556c14-29894475-xxwnm\" (UID: \"8a294b17-dd47-4c08-afb6-4c7236938c01\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-556c14-29894475-xxwnm" Nov 03 01:15:19.119009 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:15:19.118984 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8a294b17-dd47-4c08-afb6-4c7236938c01-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w3-556c14-29894475-xxwnm\" (UID: \"8a294b17-dd47-4c08-afb6-4c7236938c01\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-556c14-29894475-xxwnm" Nov 03 01:15:19.119391 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:15:19.119365 4939 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-s754v\" (UniqueName: \"kubernetes.io/projected/8abddaba-107d-4380-9ae4-5c912e6f4a03-kube-api-access-s754v\") on node \"ip-10-0-140-107.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:15:19.135333 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:15:19.135305 4939 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-e1f08d-29894475-tkmgm"] Nov 03 01:15:19.136295 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:15:19.136272 4939 cpu_manager.go:414] "containerMap: removing container" podUID="ca1eab39-5984-43f1-bd8c-029cd2131a7a" containerName="cleanup" Nov 03 01:15:19.136295 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:15:19.136290 4939 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ca1eab39-5984-43f1-bd8c-029cd2131a7a" containerName="cleanup" Nov 03 01:15:19.136518 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:15:19.136300 4939 cpu_manager.go:414] "containerMap: removing container" podUID="ca1eab39-5984-43f1-bd8c-029cd2131a7a" containerName="cleanup" Nov 03 01:15:19.136518 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:15:19.136306 4939 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ca1eab39-5984-43f1-bd8c-029cd2131a7a" containerName="cleanup" Nov 03 01:15:19.136518 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:15:19.136326 4939 cpu_manager.go:414] "containerMap: removing container" podUID="a3c9ab79-8ade-4c81-9e5a-38803c90f7a2" containerName="cleanup" Nov 03 01:15:19.136518 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:15:19.136331 4939 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a3c9ab79-8ade-4c81-9e5a-38803c90f7a2" containerName="cleanup" Nov 03 01:15:19.136518 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:15:19.136352 4939 cpu_manager.go:414] "containerMap: removing container" podUID="a3c9ab79-8ade-4c81-9e5a-38803c90f7a2" containerName="cleanup" Nov 03 01:15:19.136518 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:15:19.136357 4939 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a3c9ab79-8ade-4c81-9e5a-38803c90f7a2" containerName="cleanup" Nov 03 01:15:19.136872 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:15:19.136532 4939 memory_manager.go:365] "RemoveStaleState removing state" podUID="a3c9ab79-8ade-4c81-9e5a-38803c90f7a2" containerName="cleanup" Nov 03 01:15:19.136872 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:15:19.136542 4939 memory_manager.go:365] "RemoveStaleState removing state" podUID="ca1eab39-5984-43f1-bd8c-029cd2131a7a" containerName="cleanup" Nov 03 01:15:19.136872 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:15:19.136548 4939 memory_manager.go:365] "RemoveStaleState removing state" podUID="ca1eab39-5984-43f1-bd8c-029cd2131a7a" containerName="cleanup" Nov 03 01:15:19.136872 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:15:19.136556 4939 memory_manager.go:365] "RemoveStaleState removing state" podUID="8abddaba-107d-4380-9ae4-5c912e6f4a03" containerName="cleanup" Nov 03 01:15:19.136872 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:15:19.136566 4939 memory_manager.go:365] "RemoveStaleState removing state" podUID="a3c9ab79-8ade-4c81-9e5a-38803c90f7a2" containerName="cleanup" Nov 03 01:15:19.136872 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:15:19.136575 4939 memory_manager.go:365] "RemoveStaleState removing state" podUID="a3c9ab79-8ade-4c81-9e5a-38803c90f7a2" containerName="cleanup" Nov 03 01:15:19.141355 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:15:19.140706 4939 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-e1f08d-29894475-tkmgm" Nov 03 01:15:19.189833 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:15:19.189804 4939 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-e1f08d-29894475-tkmgm"] Nov 03 01:15:19.198539 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:15:19.198510 4939 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-821f5a-29894475-6tzm9" Nov 03 01:15:19.215849 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:15:19.215640 4939 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-4af1f7-29894475-l9sxr"] Nov 03 01:15:19.217623 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:15:19.217591 4939 cpu_manager.go:414] "containerMap: removing container" podUID="6bc4e95b-2075-4115-a856-975abe775ad6" containerName="cleanup" Nov 03 01:15:19.217623 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:15:19.217621 4939 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6bc4e95b-2075-4115-a856-975abe775ad6" containerName="cleanup" Nov 03 01:15:19.217767 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:15:19.217639 4939 cpu_manager.go:414] "containerMap: removing container" podUID="6bc4e95b-2075-4115-a856-975abe775ad6" containerName="cleanup" Nov 03 01:15:19.217767 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:15:19.217646 4939 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6bc4e95b-2075-4115-a856-975abe775ad6" containerName="cleanup" Nov 03 01:15:19.217767 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:15:19.217666 4939 cpu_manager.go:414] "containerMap: removing container" podUID="6bc4e95b-2075-4115-a856-975abe775ad6" containerName="cleanup" Nov 03 01:15:19.217767 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:15:19.217673 4939 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6bc4e95b-2075-4115-a856-975abe775ad6" containerName="cleanup" Nov 03 01:15:19.217767 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:15:19.217687 4939 cpu_manager.go:414] "containerMap: removing container" podUID="ca1eab39-5984-43f1-bd8c-029cd2131a7a" containerName="cleanup" Nov 03 01:15:19.217767 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:15:19.217694 4939 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ca1eab39-5984-43f1-bd8c-029cd2131a7a" containerName="cleanup" Nov 03 01:15:19.218075 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:15:19.217791 4939 cpu_manager.go:414] "containerMap: removing container" podUID="a3c9ab79-8ade-4c81-9e5a-38803c90f7a2" containerName="cleanup" Nov 03 01:15:19.218075 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:15:19.217800 4939 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a3c9ab79-8ade-4c81-9e5a-38803c90f7a2" containerName="cleanup" Nov 03 01:15:19.218075 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:15:19.218065 4939 memory_manager.go:365] "RemoveStaleState removing state" podUID="6bc4e95b-2075-4115-a856-975abe775ad6" containerName="cleanup" Nov 03 01:15:19.218224 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:15:19.218081 4939 memory_manager.go:365] "RemoveStaleState removing state" podUID="6bc4e95b-2075-4115-a856-975abe775ad6" containerName="cleanup" Nov 03 01:15:19.218224 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:15:19.218096 4939 memory_manager.go:365] "RemoveStaleState removing state" podUID="ca1eab39-5984-43f1-bd8c-029cd2131a7a" containerName="cleanup" Nov 03 01:15:19.220311 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:15:19.220283 4939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a3c9ab79-8ade-4c81-9e5a-38803c90f7a2-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a3c9ab79-8ade-4c81-9e5a-38803c90f7a2-maas-api-service-ca\") pod \"a3c9ab79-8ade-4c81-9e5a-38803c90f7a2\" (UID: \"a3c9ab79-8ade-4c81-9e5a-38803c90f7a2\") " Nov 03 01:15:19.220785 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:15:19.220544 4939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ca1eab39-5984-43f1-bd8c-029cd2131a7a-kube-api-access-hvpc4\" (UniqueName: \"kubernetes.io/projected/ca1eab39-5984-43f1-bd8c-029cd2131a7a-kube-api-access-hvpc4\") pod \"ca1eab39-5984-43f1-bd8c-029cd2131a7a\" (UID: \"ca1eab39-5984-43f1-bd8c-029cd2131a7a\") " Nov 03 01:15:19.220785 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:15:19.220698 4939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ca1eab39-5984-43f1-bd8c-029cd2131a7a-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ca1eab39-5984-43f1-bd8c-029cd2131a7a-maas-api-service-ca\") pod \"ca1eab39-5984-43f1-bd8c-029cd2131a7a\" (UID: \"ca1eab39-5984-43f1-bd8c-029cd2131a7a\") " Nov 03 01:15:19.220785 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:15:19.220746 4939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a3c9ab79-8ade-4c81-9e5a-38803c90f7a2-maas-api-service-ca" pod "a3c9ab79-8ade-4c81-9e5a-38803c90f7a2" (UID: "a3c9ab79-8ade-4c81-9e5a-38803c90f7a2"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 01:15:19.221182 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:15:19.221119 4939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ca1eab39-5984-43f1-bd8c-029cd2131a7a-maas-api-service-ca" pod "ca1eab39-5984-43f1-bd8c-029cd2131a7a" (UID: "ca1eab39-5984-43f1-bd8c-029cd2131a7a"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 01:15:19.221303 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:15:19.221275 4939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a3c9ab79-8ade-4c81-9e5a-38803c90f7a2-kube-api-access-g2x4v\" (UniqueName: \"kubernetes.io/projected/a3c9ab79-8ade-4c81-9e5a-38803c90f7a2-kube-api-access-g2x4v\") pod \"a3c9ab79-8ade-4c81-9e5a-38803c90f7a2\" (UID: \"a3c9ab79-8ade-4c81-9e5a-38803c90f7a2\") " Nov 03 01:15:19.221968 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:15:19.221941 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-db5x9\" (UniqueName: \"kubernetes.io/projected/8a294b17-dd47-4c08-afb6-4c7236938c01-kube-api-access-db5x9\") pod \"maas-api-key-cleanup-e2e-shared-b-w3-556c14-29894475-xxwnm\" (UID: \"8a294b17-dd47-4c08-afb6-4c7236938c01\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-556c14-29894475-xxwnm" Nov 03 01:15:19.222091 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:15:19.222023 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7f838def-06ed-4c8a-90d6-43c5a192354c-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-b-e1f08d-29894475-tkmgm\" (UID: \"7f838def-06ed-4c8a-90d6-43c5a192354c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-e1f08d-29894475-tkmgm" Nov 03 01:15:19.222156 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:15:19.222132 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zk5xc\" (UniqueName: \"kubernetes.io/projected/7f838def-06ed-4c8a-90d6-43c5a192354c-kube-api-access-zk5xc\") pod \"maas-api-key-cleanup-e2e-extmod-b-e1f08d-29894475-tkmgm\" (UID: \"7f838def-06ed-4c8a-90d6-43c5a192354c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-e1f08d-29894475-tkmgm" Nov 03 01:15:19.222427 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:15:19.222407 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8a294b17-dd47-4c08-afb6-4c7236938c01-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w3-556c14-29894475-xxwnm\" (UID: \"8a294b17-dd47-4c08-afb6-4c7236938c01\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-556c14-29894475-xxwnm" Nov 03 01:15:19.222737 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:15:19.222707 4939 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ca1eab39-5984-43f1-bd8c-029cd2131a7a-maas-api-service-ca\") on node \"ip-10-0-140-107.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:15:19.222737 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:15:19.222734 4939 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a3c9ab79-8ade-4c81-9e5a-38803c90f7a2-maas-api-service-ca\") on node \"ip-10-0-140-107.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:15:19.223504 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:15:19.223480 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8a294b17-dd47-4c08-afb6-4c7236938c01-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w3-556c14-29894475-xxwnm\" (UID: \"8a294b17-dd47-4c08-afb6-4c7236938c01\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-556c14-29894475-xxwnm" Nov 03 01:15:19.227634 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:15:19.227612 4939 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-4af1f7-29894475-l9sxr"] Nov 03 01:15:19.227995 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:15:19.227725 4939 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-4af1f7-29894475-l9sxr" Nov 03 01:15:19.233320 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:15:19.233126 4939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a3c9ab79-8ade-4c81-9e5a-38803c90f7a2-kube-api-access-g2x4v" pod "a3c9ab79-8ade-4c81-9e5a-38803c90f7a2" (UID: "a3c9ab79-8ade-4c81-9e5a-38803c90f7a2"). InnerVolumeSpecName "kube-api-access-g2x4v". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 01:15:19.242118 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:15:19.239834 4939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ca1eab39-5984-43f1-bd8c-029cd2131a7a-kube-api-access-hvpc4" pod "ca1eab39-5984-43f1-bd8c-029cd2131a7a" (UID: "ca1eab39-5984-43f1-bd8c-029cd2131a7a"). InnerVolumeSpecName "kube-api-access-hvpc4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 01:15:19.244040 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:15:19.244010 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-db5x9\" (UniqueName: \"kubernetes.io/projected/8a294b17-dd47-4c08-afb6-4c7236938c01-kube-api-access-db5x9\") pod \"maas-api-key-cleanup-e2e-shared-b-w3-556c14-29894475-xxwnm\" (UID: \"8a294b17-dd47-4c08-afb6-4c7236938c01\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-556c14-29894475-xxwnm" Nov 03 01:15:19.293095 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:15:19.293060 4939 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-4a697f-29894475-zcmxf"] Nov 03 01:15:19.294999 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:15:19.294970 4939 memory_manager.go:365] "RemoveStaleState removing state" podUID="6bc4e95b-2075-4115-a856-975abe775ad6" containerName="cleanup" Nov 03 01:15:19.304301 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:15:19.304273 4939 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-4a697f-29894475-zcmxf" Nov 03 01:15:19.306994 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:15:19.305261 4939 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-4a697f-29894475-zcmxf"] Nov 03 01:15:19.323686 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:15:19.323654 4939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6bc4e95b-2075-4115-a856-975abe775ad6-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6bc4e95b-2075-4115-a856-975abe775ad6-maas-api-service-ca\") pod \"6bc4e95b-2075-4115-a856-975abe775ad6\" (UID: \"6bc4e95b-2075-4115-a856-975abe775ad6\") " Nov 03 01:15:19.323865 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:15:19.323755 4939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6bc4e95b-2075-4115-a856-975abe775ad6-kube-api-access-4nwl2\" (UniqueName: \"kubernetes.io/projected/6bc4e95b-2075-4115-a856-975abe775ad6-kube-api-access-4nwl2\") pod \"6bc4e95b-2075-4115-a856-975abe775ad6\" (UID: \"6bc4e95b-2075-4115-a856-975abe775ad6\") " Nov 03 01:15:19.324828 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:15:19.324798 4939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6bc4e95b-2075-4115-a856-975abe775ad6-maas-api-service-ca" pod "6bc4e95b-2075-4115-a856-975abe775ad6" (UID: "6bc4e95b-2075-4115-a856-975abe775ad6"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 01:15:19.327265 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:15:19.326830 4939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6bc4e95b-2075-4115-a856-975abe775ad6-kube-api-access-4nwl2" pod "6bc4e95b-2075-4115-a856-975abe775ad6" (UID: "6bc4e95b-2075-4115-a856-975abe775ad6"). InnerVolumeSpecName "kube-api-access-4nwl2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 01:15:19.327372 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:15:19.327336 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5rgfc\" (UniqueName: \"kubernetes.io/projected/4197807b-0461-4733-ac86-01e34bf5a8ec-kube-api-access-5rgfc\") pod \"maas-api-key-cleanup-e2e-shared-b-w1-4af1f7-29894475-l9sxr\" (UID: \"4197807b-0461-4733-ac86-01e34bf5a8ec\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-4af1f7-29894475-l9sxr" Nov 03 01:15:19.327446 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:15:19.327430 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7f838def-06ed-4c8a-90d6-43c5a192354c-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-b-e1f08d-29894475-tkmgm\" (UID: \"7f838def-06ed-4c8a-90d6-43c5a192354c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-e1f08d-29894475-tkmgm" Nov 03 01:15:19.327680 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:15:19.327542 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zk5xc\" (UniqueName: \"kubernetes.io/projected/7f838def-06ed-4c8a-90d6-43c5a192354c-kube-api-access-zk5xc\") pod \"maas-api-key-cleanup-e2e-extmod-b-e1f08d-29894475-tkmgm\" (UID: \"7f838def-06ed-4c8a-90d6-43c5a192354c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-e1f08d-29894475-tkmgm" Nov 03 01:15:19.328777 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:15:19.328132 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4197807b-0461-4733-ac86-01e34bf5a8ec-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w1-4af1f7-29894475-l9sxr\" (UID: \"4197807b-0461-4733-ac86-01e34bf5a8ec\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-4af1f7-29894475-l9sxr" Nov 03 01:15:19.328777 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:15:19.328468 4939 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6bc4e95b-2075-4115-a856-975abe775ad6-maas-api-service-ca\") on node \"ip-10-0-140-107.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:15:19.330146 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:15:19.330126 4939 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4nwl2\" (UniqueName: \"kubernetes.io/projected/6bc4e95b-2075-4115-a856-975abe775ad6-kube-api-access-4nwl2\") on node \"ip-10-0-140-107.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:15:19.330450 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:15:19.330432 4939 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hvpc4\" (UniqueName: \"kubernetes.io/projected/ca1eab39-5984-43f1-bd8c-029cd2131a7a-kube-api-access-hvpc4\") on node \"ip-10-0-140-107.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:15:19.330574 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:15:19.330557 4939 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-g2x4v\" (UniqueName: \"kubernetes.io/projected/a3c9ab79-8ade-4c81-9e5a-38803c90f7a2-kube-api-access-g2x4v\") on node \"ip-10-0-140-107.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:15:19.331388 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:15:19.331366 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7f838def-06ed-4c8a-90d6-43c5a192354c-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-b-e1f08d-29894475-tkmgm\" (UID: \"7f838def-06ed-4c8a-90d6-43c5a192354c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-e1f08d-29894475-tkmgm" Nov 03 01:15:19.405706 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:15:19.352603 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zk5xc\" (UniqueName: \"kubernetes.io/projected/7f838def-06ed-4c8a-90d6-43c5a192354c-kube-api-access-zk5xc\") pod \"maas-api-key-cleanup-e2e-extmod-b-e1f08d-29894475-tkmgm\" (UID: \"7f838def-06ed-4c8a-90d6-43c5a192354c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-e1f08d-29894475-tkmgm" Nov 03 01:15:19.405706 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:15:19.391495 4939 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-556c14-29894475-xxwnm" Nov 03 01:15:19.426680 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:15:19.426650 4939 generic.go:356] "Generic (PLEG): container finished" podID="f0a403e0-c375-412d-940d-66ecf89e3270" containerID="197bf1e71bd85227bb0cd64c8f56f0018d9bb02fcd71c0767bf56037a0516523" exitCode=60 Nov 03 01:15:19.426819 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:15:19.426736 4939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-83755a-29894475-clqb4" event={"ID":"f0a403e0-c375-412d-940d-66ecf89e3270","Type":"ContainerDied","Data":"197bf1e71bd85227bb0cd64c8f56f0018d9bb02fcd71c0767bf56037a0516523"} Nov 03 01:15:19.426819 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:15:19.426815 4939 scope.go:122] "RemoveContainer" containerID="9819451ae33994ff4b310008a0660499b68c595317a12394ddb6c7d64a5d97d3" Nov 03 01:15:19.427400 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:15:19.427384 4939 scope.go:122] "RemoveContainer" containerID="197bf1e71bd85227bb0cd64c8f56f0018d9bb02fcd71c0767bf56037a0516523" Nov 03 01:15:19.427987 ip-10-0-140-107 kubenswrapper[4939]: E1103 01:15:19.427964 4939 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w3-83755a-29894475-clqb4_odh-ai-gateway-infra(f0a403e0-c375-412d-940d-66ecf89e3270)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-83755a-29894475-clqb4" podUID="f0a403e0-c375-412d-940d-66ecf89e3270" Nov 03 01:15:19.428441 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:15:19.428392 4939 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-821f5a-29894475-6tzm9" Nov 03 01:15:19.428504 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:15:19.428460 4939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-821f5a-29894475-6tzm9" event={"ID":"6bc4e95b-2075-4115-a856-975abe775ad6","Type":"ContainerDied","Data":"9e1deaec4ff42bbbf358aceb047fc0919139861b920954e81ef15e12dfeada90"} Nov 03 01:15:19.431781 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:15:19.431754 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4197807b-0461-4733-ac86-01e34bf5a8ec-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w1-4af1f7-29894475-l9sxr\" (UID: \"4197807b-0461-4733-ac86-01e34bf5a8ec\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-4af1f7-29894475-l9sxr" Nov 03 01:15:19.431867 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:15:19.431830 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-429zj\" (UniqueName: \"kubernetes.io/projected/1ab583fa-19f7-43e2-acf9-17da1e65d1e0-kube-api-access-429zj\") pod \"maas-api-key-cleanup-e2e-shared-b-w4-4a697f-29894475-zcmxf\" (UID: \"1ab583fa-19f7-43e2-acf9-17da1e65d1e0\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-4a697f-29894475-zcmxf" Nov 03 01:15:19.432006 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:15:19.431992 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1ab583fa-19f7-43e2-acf9-17da1e65d1e0-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w4-4a697f-29894475-zcmxf\" (UID: \"1ab583fa-19f7-43e2-acf9-17da1e65d1e0\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-4a697f-29894475-zcmxf" Nov 03 01:15:19.432053 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:15:19.432042 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5rgfc\" (UniqueName: \"kubernetes.io/projected/4197807b-0461-4733-ac86-01e34bf5a8ec-kube-api-access-5rgfc\") pod \"maas-api-key-cleanup-e2e-shared-b-w1-4af1f7-29894475-l9sxr\" (UID: \"4197807b-0461-4733-ac86-01e34bf5a8ec\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-4af1f7-29894475-l9sxr" Nov 03 01:15:19.433360 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:15:19.433326 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4197807b-0461-4733-ac86-01e34bf5a8ec-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w1-4af1f7-29894475-l9sxr\" (UID: \"4197807b-0461-4733-ac86-01e34bf5a8ec\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-4af1f7-29894475-l9sxr" Nov 03 01:15:19.437107 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:15:19.437084 4939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-2729f2-29894475-pcj4t" event={"ID":"a3c9ab79-8ade-4c81-9e5a-38803c90f7a2","Type":"ContainerDied","Data":"f277b586e53b7c2d3d9243f70c8e2ec93f954dde8f37faeb9961b607df3f963f"} Nov 03 01:15:19.437751 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:15:19.437308 4939 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-2729f2-29894475-pcj4t" Nov 03 01:15:19.439168 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:15:19.439143 4939 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-af8244-29894475-h4vvr"] Nov 03 01:15:19.509505 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:15:19.509477 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5rgfc\" (UniqueName: \"kubernetes.io/projected/4197807b-0461-4733-ac86-01e34bf5a8ec-kube-api-access-5rgfc\") pod \"maas-api-key-cleanup-e2e-shared-b-w1-4af1f7-29894475-l9sxr\" (UID: \"4197807b-0461-4733-ac86-01e34bf5a8ec\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-4af1f7-29894475-l9sxr" Nov 03 01:15:19.513372 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:15:19.513130 4939 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-e1f08d-29894475-tkmgm" Nov 03 01:15:19.519167 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:15:19.514768 4939 scope.go:122] "RemoveContainer" containerID="d2ea491757f2582ce132b9a16c53df7157388847c81a70fdc4d8f1341605dcc5" Nov 03 01:15:19.520239 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:15:19.520205 4939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-027c19-29894475-nnm44" event={"ID":"ca1eab39-5984-43f1-bd8c-029cd2131a7a","Type":"ContainerDied","Data":"77463523acbb67960da6b0eacf1abfdcdc77d9313b6d84e433e23a4caa6abc46"} Nov 03 01:15:19.520375 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:15:19.520265 4939 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-027c19-29894475-nnm44" Nov 03 01:15:19.538371 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:15:19.538347 4939 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-020af9-29894475-xhbqm" Nov 03 01:15:19.538715 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:15:19.538693 4939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-020af9-29894475-xhbqm" event={"ID":"8abddaba-107d-4380-9ae4-5c912e6f4a03","Type":"ContainerDied","Data":"35b67ba7dd1f4006ec81c259865566530c845dd734f9193882d1a1219a214d07"} Nov 03 01:15:19.545504 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:15:19.545475 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-429zj\" (UniqueName: \"kubernetes.io/projected/1ab583fa-19f7-43e2-acf9-17da1e65d1e0-kube-api-access-429zj\") pod \"maas-api-key-cleanup-e2e-shared-b-w4-4a697f-29894475-zcmxf\" (UID: \"1ab583fa-19f7-43e2-acf9-17da1e65d1e0\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-4a697f-29894475-zcmxf" Nov 03 01:15:19.545880 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:15:19.545834 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1ab583fa-19f7-43e2-acf9-17da1e65d1e0-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w4-4a697f-29894475-zcmxf\" (UID: \"1ab583fa-19f7-43e2-acf9-17da1e65d1e0\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-4a697f-29894475-zcmxf" Nov 03 01:15:19.609268 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:15:19.593563 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1ab583fa-19f7-43e2-acf9-17da1e65d1e0-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w4-4a697f-29894475-zcmxf\" (UID: \"1ab583fa-19f7-43e2-acf9-17da1e65d1e0\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-4a697f-29894475-zcmxf" Nov 03 01:15:19.611123 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:15:19.604698 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-429zj\" (UniqueName: \"kubernetes.io/projected/1ab583fa-19f7-43e2-acf9-17da1e65d1e0-kube-api-access-429zj\") pod \"maas-api-key-cleanup-e2e-shared-b-w4-4a697f-29894475-zcmxf\" (UID: \"1ab583fa-19f7-43e2-acf9-17da1e65d1e0\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-4a697f-29894475-zcmxf" Nov 03 01:15:19.617684 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:15:19.617653 4939 patch_prober.go:29] interesting pod/e2e-extmod-a-d25759-openshift-default-d47fd9785-7gzj2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.237:15021/healthz/ready\": dial tcp 10.132.0.237:15021: connect: connection refused" start-of-body= Nov 03 01:15:19.617814 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:15:19.617723 4939 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-d25759-openshift-default-d47fd9785-7gzj2" podUID="ba585f4b-4916-4d03-bf1e-454c029d46ad" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.237:15021/healthz/ready\": dial tcp 10.132.0.237:15021: connect: connection refused" Nov 03 01:15:19.632398 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:15:19.632365 4939 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-4af1f7-29894475-l9sxr" Nov 03 01:15:19.640505 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:15:19.640458 4939 scope.go:122] "RemoveContainer" containerID="fdb68ec87c7b03c2b24eb1f1c6760d77fa2896c3cccfd8f0c8cf1df7baf5643d" Nov 03 01:15:19.642308 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:15:19.642104 4939 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-6c0ec0-29894475-8vfnr"] Nov 03 01:15:19.683521 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:15:19.683394 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:15:19.706015 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:15:19.705978 4939 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-4a697f-29894475-zcmxf" Nov 03 01:15:19.713054 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:15:19.712838 4939 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-821f5a-29894475-6tzm9"] Nov 03 01:15:19.781773 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:15:19.718070 4939 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-821f5a-29894475-6tzm9"] Nov 03 01:15:19.935646 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:15:19.935613 4939 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6bc4e95b-2075-4115-a856-975abe775ad6" path="/var/lib/kubelet/pods/6bc4e95b-2075-4115-a856-975abe775ad6/volumes" Nov 03 01:15:19.936416 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:15:19.936396 4939 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-2729f2-29894475-pcj4t"] Nov 03 01:15:19.936568 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:15:19.936556 4939 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-2729f2-29894475-pcj4t"] Nov 03 01:15:19.954729 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:15:19.954662 4939 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-027c19-29894475-nnm44"] Nov 03 01:15:19.976933 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:15:19.976860 4939 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-027c19-29894475-nnm44"] Nov 03 01:15:19.995385 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:15:19.995346 4939 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-020af9-29894475-xhbqm"] Nov 03 01:15:20.002216 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:15:20.002183 4939 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-020af9-29894475-xhbqm"] Nov 03 01:15:20.006281 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:15:20.006186 4939 scope.go:122] "RemoveContainer" containerID="83dd3ab636da82ef190c0b9ea3df71072c5b0e5d39279abefb4ef1e2308329ee" Nov 03 01:15:20.020754 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:15:20.020274 4939 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-556c14-29894475-xxwnm"] Nov 03 01:15:20.112659 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:15:20.112621 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:15:20.122463 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:15:20.122442 4939 scope.go:122] "RemoveContainer" containerID="6052e16e3231660b40e3f603962d835ccad947d23a71d7b2352db238305f039d" Nov 03 01:15:20.189217 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:15:20.189148 4939 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-af8244-29894475-h4vvr" Nov 03 01:15:20.230314 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:15:20.230286 4939 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-88bbf7-29894475-zhljf"] Nov 03 01:15:20.230612 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:15:20.230583 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:15:20.232065 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:15:20.231637 4939 cpu_manager.go:414] "containerMap: removing container" podUID="899bb599-2a0d-4670-818b-b0eda152816f" containerName="cleanup" Nov 03 01:15:20.232065 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:15:20.231655 4939 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="899bb599-2a0d-4670-818b-b0eda152816f" containerName="cleanup" Nov 03 01:15:20.232065 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:15:20.231672 4939 cpu_manager.go:414] "containerMap: removing container" podUID="899bb599-2a0d-4670-818b-b0eda152816f" containerName="cleanup" Nov 03 01:15:20.232065 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:15:20.231677 4939 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="899bb599-2a0d-4670-818b-b0eda152816f" containerName="cleanup" Nov 03 01:15:20.232065 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:15:20.231843 4939 memory_manager.go:365] "RemoveStaleState removing state" podUID="899bb599-2a0d-4670-818b-b0eda152816f" containerName="cleanup" Nov 03 01:15:20.232065 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:15:20.231852 4939 memory_manager.go:365] "RemoveStaleState removing state" podUID="899bb599-2a0d-4670-818b-b0eda152816f" containerName="cleanup" Nov 03 01:15:20.236228 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:15:20.236208 4939 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-88bbf7-29894475-zhljf" Nov 03 01:15:20.247175 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:15:20.246165 4939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/899bb599-2a0d-4670-818b-b0eda152816f-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/899bb599-2a0d-4670-818b-b0eda152816f-maas-api-service-ca\") pod \"899bb599-2a0d-4670-818b-b0eda152816f\" (UID: \"899bb599-2a0d-4670-818b-b0eda152816f\") " Nov 03 01:15:20.247175 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:15:20.246293 4939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/899bb599-2a0d-4670-818b-b0eda152816f-kube-api-access-jvd6s\" (UniqueName: \"kubernetes.io/projected/899bb599-2a0d-4670-818b-b0eda152816f-kube-api-access-jvd6s\") pod \"899bb599-2a0d-4670-818b-b0eda152816f\" (UID: \"899bb599-2a0d-4670-818b-b0eda152816f\") " Nov 03 01:15:20.248270 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:15:20.248248 4939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/899bb599-2a0d-4670-818b-b0eda152816f-maas-api-service-ca" pod "899bb599-2a0d-4670-818b-b0eda152816f" (UID: "899bb599-2a0d-4670-818b-b0eda152816f"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 01:15:20.254798 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:15:20.254757 4939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/899bb599-2a0d-4670-818b-b0eda152816f-kube-api-access-jvd6s" pod "899bb599-2a0d-4670-818b-b0eda152816f" (UID: "899bb599-2a0d-4670-818b-b0eda152816f"). InnerVolumeSpecName "kube-api-access-jvd6s". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 01:15:20.283065 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:15:20.283028 4939 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-88bbf7-29894475-zhljf"] Nov 03 01:15:20.311942 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:15:20.311917 4939 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-6c0ec0-29894475-8vfnr" Nov 03 01:15:20.385499 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:15:20.351385 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pswzc\" (UniqueName: \"kubernetes.io/projected/f4d972cb-44ea-4f7d-a548-91751c607176-kube-api-access-pswzc\") pod \"maas-api-key-cleanup-e2e-worker-w1-88bbf7-29894475-zhljf\" (UID: \"f4d972cb-44ea-4f7d-a548-91751c607176\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-88bbf7-29894475-zhljf" Nov 03 01:15:20.385499 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:15:20.351593 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f4d972cb-44ea-4f7d-a548-91751c607176-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w1-88bbf7-29894475-zhljf\" (UID: \"f4d972cb-44ea-4f7d-a548-91751c607176\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-88bbf7-29894475-zhljf" Nov 03 01:15:20.385499 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:15:20.351811 4939 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/899bb599-2a0d-4670-818b-b0eda152816f-maas-api-service-ca\") on node \"ip-10-0-140-107.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:15:20.385499 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:15:20.351829 4939 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jvd6s\" (UniqueName: \"kubernetes.io/projected/899bb599-2a0d-4670-818b-b0eda152816f-kube-api-access-jvd6s\") on node \"ip-10-0-140-107.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:15:20.387256 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:15:20.387222 4939 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-2f9a3a-29894475-9nh4j"] Nov 03 01:15:20.388712 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:15:20.388688 4939 cpu_manager.go:414] "containerMap: removing container" podUID="7eba01d0-b475-421d-8aba-56cbc67c652d" containerName="cleanup" Nov 03 01:15:20.388780 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:15:20.388715 4939 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7eba01d0-b475-421d-8aba-56cbc67c652d" containerName="cleanup" Nov 03 01:15:20.388780 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:15:20.388733 4939 cpu_manager.go:414] "containerMap: removing container" podUID="7eba01d0-b475-421d-8aba-56cbc67c652d" containerName="cleanup" Nov 03 01:15:20.388780 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:15:20.388741 4939 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7eba01d0-b475-421d-8aba-56cbc67c652d" containerName="cleanup" Nov 03 01:15:20.388928 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:15:20.388779 4939 cpu_manager.go:414] "containerMap: removing container" podUID="7eba01d0-b475-421d-8aba-56cbc67c652d" containerName="cleanup" Nov 03 01:15:20.388928 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:15:20.388787 4939 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7eba01d0-b475-421d-8aba-56cbc67c652d" containerName="cleanup" Nov 03 01:15:20.388928 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:15:20.388828 4939 cpu_manager.go:414] "containerMap: removing container" podUID="899bb599-2a0d-4670-818b-b0eda152816f" containerName="cleanup" Nov 03 01:15:20.388928 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:15:20.388835 4939 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="899bb599-2a0d-4670-818b-b0eda152816f" containerName="cleanup" Nov 03 01:15:20.389205 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:15:20.389180 4939 memory_manager.go:365] "RemoveStaleState removing state" podUID="7eba01d0-b475-421d-8aba-56cbc67c652d" containerName="cleanup" Nov 03 01:15:20.389205 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:15:20.389199 4939 memory_manager.go:365] "RemoveStaleState removing state" podUID="7eba01d0-b475-421d-8aba-56cbc67c652d" containerName="cleanup" Nov 03 01:15:20.389297 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:15:20.389218 4939 memory_manager.go:365] "RemoveStaleState removing state" podUID="7eba01d0-b475-421d-8aba-56cbc67c652d" containerName="cleanup" Nov 03 01:15:20.389297 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:15:20.389234 4939 memory_manager.go:365] "RemoveStaleState removing state" podUID="899bb599-2a0d-4670-818b-b0eda152816f" containerName="cleanup" Nov 03 01:15:20.395194 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:15:20.395172 4939 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-2f9a3a-29894475-9nh4j" Nov 03 01:15:20.399794 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:15:20.398985 4939 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-2f9a3a-29894475-9nh4j"] Nov 03 01:15:20.404393 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:15:20.404257 4939 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-e1f08d-29894475-tkmgm"] Nov 03 01:15:20.428986 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:15:20.428952 4939 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-4af1f7-29894475-l9sxr"] Nov 03 01:15:20.429868 ip-10-0-140-107 kubenswrapper[4939]: W1103 01:15:20.429776 4939 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1ab583fa_19f7_43e2_acf9_17da1e65d1e0.slice/crio-5975bc80f953f4c15554528bb10e05dae09ce88ba822f8821836651f0c30db75 WatchSource:0}: Error finding container 5975bc80f953f4c15554528bb10e05dae09ce88ba822f8821836651f0c30db75: Status 404 returned error can't find the container with id 5975bc80f953f4c15554528bb10e05dae09ce88ba822f8821836651f0c30db75 Nov 03 01:15:20.431603 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:15:20.431079 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:15:20.433793 ip-10-0-140-107 kubenswrapper[4939]: W1103 01:15:20.431844 4939 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4197807b_0461_4733_ac86_01e34bf5a8ec.slice/crio-61d14992bc956d57d746f9f20896cbe5b2a51d693438a33308e6443e9d25eb5c WatchSource:0}: Error finding container 61d14992bc956d57d746f9f20896cbe5b2a51d693438a33308e6443e9d25eb5c: Status 404 returned error can't find the container with id 61d14992bc956d57d746f9f20896cbe5b2a51d693438a33308e6443e9d25eb5c Nov 03 01:15:20.435004 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:15:20.434978 4939 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-4a697f-29894475-zcmxf"] Nov 03 01:15:20.465929 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:15:20.458851 4939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7eba01d0-b475-421d-8aba-56cbc67c652d-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7eba01d0-b475-421d-8aba-56cbc67c652d-maas-api-service-ca\") pod \"7eba01d0-b475-421d-8aba-56cbc67c652d\" (UID: \"7eba01d0-b475-421d-8aba-56cbc67c652d\") " Nov 03 01:15:20.465929 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:15:20.459029 4939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7eba01d0-b475-421d-8aba-56cbc67c652d-kube-api-access-5sp52\" (UniqueName: \"kubernetes.io/projected/7eba01d0-b475-421d-8aba-56cbc67c652d-kube-api-access-5sp52\") pod \"7eba01d0-b475-421d-8aba-56cbc67c652d\" (UID: \"7eba01d0-b475-421d-8aba-56cbc67c652d\") " Nov 03 01:15:20.465929 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:15:20.459863 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pswzc\" (UniqueName: \"kubernetes.io/projected/f4d972cb-44ea-4f7d-a548-91751c607176-kube-api-access-pswzc\") pod \"maas-api-key-cleanup-e2e-worker-w1-88bbf7-29894475-zhljf\" (UID: \"f4d972cb-44ea-4f7d-a548-91751c607176\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-88bbf7-29894475-zhljf" Nov 03 01:15:20.478867 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:15:20.478824 4939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7eba01d0-b475-421d-8aba-56cbc67c652d-maas-api-service-ca" pod "7eba01d0-b475-421d-8aba-56cbc67c652d" (UID: "7eba01d0-b475-421d-8aba-56cbc67c652d"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 01:15:20.524944 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:15:20.489855 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f4d972cb-44ea-4f7d-a548-91751c607176-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w1-88bbf7-29894475-zhljf\" (UID: \"f4d972cb-44ea-4f7d-a548-91751c607176\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-88bbf7-29894475-zhljf" Nov 03 01:15:20.556000 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:15:20.554977 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f4d972cb-44ea-4f7d-a548-91751c607176-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w1-88bbf7-29894475-zhljf\" (UID: \"f4d972cb-44ea-4f7d-a548-91751c607176\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-88bbf7-29894475-zhljf" Nov 03 01:15:20.556000 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:15:20.555095 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/05835a80-2f35-4b67-b578-42e64bfafb9f-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w0-2f9a3a-29894475-9nh4j\" (UID: \"05835a80-2f35-4b67-b578-42e64bfafb9f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-2f9a3a-29894475-9nh4j" Nov 03 01:15:20.556000 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:15:20.555197 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xwdf4\" (UniqueName: \"kubernetes.io/projected/05835a80-2f35-4b67-b578-42e64bfafb9f-kube-api-access-xwdf4\") pod \"maas-api-key-cleanup-e2e-worker-w0-2f9a3a-29894475-9nh4j\" (UID: \"05835a80-2f35-4b67-b578-42e64bfafb9f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-2f9a3a-29894475-9nh4j" Nov 03 01:15:20.556000 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:15:20.555363 4939 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7eba01d0-b475-421d-8aba-56cbc67c652d-maas-api-service-ca\") on node \"ip-10-0-140-107.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:15:20.599271 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:15:20.599226 4939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7eba01d0-b475-421d-8aba-56cbc67c652d-kube-api-access-5sp52" pod "7eba01d0-b475-421d-8aba-56cbc67c652d" (UID: "7eba01d0-b475-421d-8aba-56cbc67c652d"). InnerVolumeSpecName "kube-api-access-5sp52". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 01:15:20.608727 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:15:20.604932 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pswzc\" (UniqueName: \"kubernetes.io/projected/f4d972cb-44ea-4f7d-a548-91751c607176-kube-api-access-pswzc\") pod \"maas-api-key-cleanup-e2e-worker-w1-88bbf7-29894475-zhljf\" (UID: \"f4d972cb-44ea-4f7d-a548-91751c607176\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-88bbf7-29894475-zhljf" Nov 03 01:15:20.610218 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:15:20.610183 4939 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-88bbf7-29894475-zhljf" Nov 03 01:15:20.611881 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:15:20.611124 4939 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-83755a-29894475-clqb4"] Nov 03 01:15:20.617648 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:15:20.617225 4939 patch_prober.go:29] interesting pod/e2e-extmod-a-d25759-openshift-default-d47fd9785-7gzj2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.237:15021/healthz/ready\": dial tcp 10.132.0.237:15021: connect: connection refused" start-of-body= Nov 03 01:15:20.617648 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:15:20.617274 4939 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-d25759-openshift-default-d47fd9785-7gzj2" podUID="ba585f4b-4916-4d03-bf1e-454c029d46ad" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.237:15021/healthz/ready\": dial tcp 10.132.0.237:15021: connect: connection refused" Nov 03 01:15:20.623252 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:15:20.623232 4939 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-af8244-29894475-h4vvr" Nov 03 01:15:20.623381 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:15:20.623246 4939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-af8244-29894475-h4vvr" event={"ID":"899bb599-2a0d-4670-818b-b0eda152816f","Type":"ContainerDied","Data":"ff03087a82abc50bd736a0195c0d31e8b6994900b6a521727bdf3c158fa82862"} Nov 03 01:15:20.623513 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:15:20.623500 4939 scope.go:122] "RemoveContainer" containerID="161ebebe6568108bba787ec5ad637f88794a68747dc850ec96830ea92b1d6265" Nov 03 01:15:20.626798 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:15:20.626766 4939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-4af1f7-29894475-l9sxr" event={"ID":"4197807b-0461-4733-ac86-01e34bf5a8ec","Type":"ContainerStarted","Data":"61d14992bc956d57d746f9f20896cbe5b2a51d693438a33308e6443e9d25eb5c"} Nov 03 01:15:20.630587 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:15:20.630558 4939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-6c0ec0-29894475-8vfnr" event={"ID":"7eba01d0-b475-421d-8aba-56cbc67c652d","Type":"ContainerDied","Data":"c33c34d25ef128df92b0ad8d9bb4004e1cfc917aced3b67e50dda5419419bb3d"} Nov 03 01:15:20.630690 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:15:20.630667 4939 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-6c0ec0-29894475-8vfnr" Nov 03 01:15:20.637072 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:15:20.637052 4939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-e1f08d-29894475-tkmgm" event={"ID":"7f838def-06ed-4c8a-90d6-43c5a192354c","Type":"ContainerStarted","Data":"21b639b38b4467f1458245a443a8469c32aee9f701545f682b64962fbf3383f9"} Nov 03 01:15:20.639884 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:15:20.639859 4939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-556c14-29894475-xxwnm" event={"ID":"8a294b17-dd47-4c08-afb6-4c7236938c01","Type":"ContainerStarted","Data":"a0dcc6e123662c255d6c271b62b70eca7cc92f9bf03076a902d0166ef87b020e"} Nov 03 01:15:20.644398 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:15:20.644372 4939 scope.go:122] "RemoveContainer" containerID="2ea63d250680ae771b6f051a37d7774005f41964879c47684ad6dc5d274b4dad" Nov 03 01:15:20.645375 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:15:20.645250 4939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-4a697f-29894475-zcmxf" event={"ID":"1ab583fa-19f7-43e2-acf9-17da1e65d1e0","Type":"ContainerStarted","Data":"5975bc80f953f4c15554528bb10e05dae09ce88ba822f8821836651f0c30db75"} Nov 03 01:15:20.691353 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:15:20.691092 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/05835a80-2f35-4b67-b578-42e64bfafb9f-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w0-2f9a3a-29894475-9nh4j\" (UID: \"05835a80-2f35-4b67-b578-42e64bfafb9f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-2f9a3a-29894475-9nh4j" Nov 03 01:15:20.691353 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:15:20.691166 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xwdf4\" (UniqueName: \"kubernetes.io/projected/05835a80-2f35-4b67-b578-42e64bfafb9f-kube-api-access-xwdf4\") pod \"maas-api-key-cleanup-e2e-worker-w0-2f9a3a-29894475-9nh4j\" (UID: \"05835a80-2f35-4b67-b578-42e64bfafb9f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-2f9a3a-29894475-9nh4j" Nov 03 01:15:20.691507 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:15:20.691408 4939 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5sp52\" (UniqueName: \"kubernetes.io/projected/7eba01d0-b475-421d-8aba-56cbc67c652d-kube-api-access-5sp52\") on node \"ip-10-0-140-107.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:15:20.693549 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:15:20.692232 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/05835a80-2f35-4b67-b578-42e64bfafb9f-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w0-2f9a3a-29894475-9nh4j\" (UID: \"05835a80-2f35-4b67-b578-42e64bfafb9f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-2f9a3a-29894475-9nh4j" Nov 03 01:15:20.693549 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:15:20.693034 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:15:20.712497 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:15:20.712442 4939 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-af8244-29894475-h4vvr"] Nov 03 01:15:20.718319 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:15:20.718292 4939 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-af8244-29894475-h4vvr"] Nov 03 01:15:20.733216 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:15:20.733184 4939 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-6c0ec0-29894475-8vfnr"] Nov 03 01:15:20.739186 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:15:20.739155 4939 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-6c0ec0-29894475-8vfnr"] Nov 03 01:15:20.788330 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:15:20.786956 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xwdf4\" (UniqueName: \"kubernetes.io/projected/05835a80-2f35-4b67-b578-42e64bfafb9f-kube-api-access-xwdf4\") pod \"maas-api-key-cleanup-e2e-worker-w0-2f9a3a-29894475-9nh4j\" (UID: \"05835a80-2f35-4b67-b578-42e64bfafb9f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-2f9a3a-29894475-9nh4j" Nov 03 01:15:20.838416 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:15:20.838322 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:15:21.028235 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:15:21.028199 4939 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-2f9a3a-29894475-9nh4j" Nov 03 01:15:21.053842 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:15:21.053776 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:15:21.197679 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:15:21.197647 4939 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-83755a-29894475-clqb4" Nov 03 01:15:21.235002 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:15:21.234964 4939 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-f7680d-29894475-qw25k"] Nov 03 01:15:21.236574 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:15:21.236546 4939 cpu_manager.go:414] "containerMap: removing container" podUID="f0a403e0-c375-412d-940d-66ecf89e3270" containerName="cleanup" Nov 03 01:15:21.236574 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:15:21.236572 4939 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f0a403e0-c375-412d-940d-66ecf89e3270" containerName="cleanup" Nov 03 01:15:21.236749 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:15:21.236605 4939 cpu_manager.go:414] "containerMap: removing container" podUID="f0a403e0-c375-412d-940d-66ecf89e3270" containerName="cleanup" Nov 03 01:15:21.236749 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:15:21.236613 4939 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f0a403e0-c375-412d-940d-66ecf89e3270" containerName="cleanup" Nov 03 01:15:21.236749 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:15:21.236671 4939 cpu_manager.go:414] "containerMap: removing container" podUID="f0a403e0-c375-412d-940d-66ecf89e3270" containerName="cleanup" Nov 03 01:15:21.236749 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:15:21.236679 4939 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f0a403e0-c375-412d-940d-66ecf89e3270" containerName="cleanup" Nov 03 01:15:21.236983 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:15:21.236896 4939 memory_manager.go:365] "RemoveStaleState removing state" podUID="f0a403e0-c375-412d-940d-66ecf89e3270" containerName="cleanup" Nov 03 01:15:21.236983 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:15:21.236974 4939 memory_manager.go:365] "RemoveStaleState removing state" podUID="f0a403e0-c375-412d-940d-66ecf89e3270" containerName="cleanup" Nov 03 01:15:21.237175 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:15:21.236989 4939 memory_manager.go:365] "RemoveStaleState removing state" podUID="f0a403e0-c375-412d-940d-66ecf89e3270" containerName="cleanup" Nov 03 01:15:21.243923 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:15:21.242149 4939 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-f7680d-29894475-qw25k" Nov 03 01:15:21.306573 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:15:21.306510 4939 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-f7680d-29894475-qw25k"] Nov 03 01:15:21.334711 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:15:21.334675 4939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f0a403e0-c375-412d-940d-66ecf89e3270-kube-api-access-zbl6w\" (UniqueName: \"kubernetes.io/projected/f0a403e0-c375-412d-940d-66ecf89e3270-kube-api-access-zbl6w\") pod \"f0a403e0-c375-412d-940d-66ecf89e3270\" (UID: \"f0a403e0-c375-412d-940d-66ecf89e3270\") " Nov 03 01:15:21.335209 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:15:21.334946 4939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f0a403e0-c375-412d-940d-66ecf89e3270-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f0a403e0-c375-412d-940d-66ecf89e3270-maas-api-service-ca\") pod \"f0a403e0-c375-412d-940d-66ecf89e3270\" (UID: \"f0a403e0-c375-412d-940d-66ecf89e3270\") " Nov 03 01:15:21.335443 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:15:21.335416 4939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f0a403e0-c375-412d-940d-66ecf89e3270-maas-api-service-ca" pod "f0a403e0-c375-412d-940d-66ecf89e3270" (UID: "f0a403e0-c375-412d-940d-66ecf89e3270"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 01:15:21.335964 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:15:21.335935 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5ba08137-cee0-4cf7-a27a-4c750f9308c4-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w2-f7680d-29894475-qw25k\" (UID: \"5ba08137-cee0-4cf7-a27a-4c750f9308c4\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-f7680d-29894475-qw25k" Nov 03 01:15:21.336052 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:15:21.335992 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-94rt4\" (UniqueName: \"kubernetes.io/projected/5ba08137-cee0-4cf7-a27a-4c750f9308c4-kube-api-access-94rt4\") pod \"maas-api-key-cleanup-e2e-worker-w2-f7680d-29894475-qw25k\" (UID: \"5ba08137-cee0-4cf7-a27a-4c750f9308c4\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-f7680d-29894475-qw25k" Nov 03 01:15:21.336154 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:15:21.336134 4939 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f0a403e0-c375-412d-940d-66ecf89e3270-maas-api-service-ca\") on node \"ip-10-0-140-107.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:15:21.338759 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:15:21.338627 4939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f0a403e0-c375-412d-940d-66ecf89e3270-kube-api-access-zbl6w" pod "f0a403e0-c375-412d-940d-66ecf89e3270" (UID: "f0a403e0-c375-412d-940d-66ecf89e3270"). InnerVolumeSpecName "kube-api-access-zbl6w". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 01:15:21.359577 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:15:21.359550 4939 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-88bbf7-29894475-zhljf"] Nov 03 01:15:21.415625 ip-10-0-140-107 kubenswrapper[4939]: W1103 01:15:21.415557 4939 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf4d972cb_44ea_4f7d_a548_91751c607176.slice/crio-d7a63ef4b4523e2280edd064ce129a07f9db6398ea4d6408913624cab8e62165 WatchSource:0}: Error finding container d7a63ef4b4523e2280edd064ce129a07f9db6398ea4d6408913624cab8e62165: Status 404 returned error can't find the container with id d7a63ef4b4523e2280edd064ce129a07f9db6398ea4d6408913624cab8e62165 Nov 03 01:15:21.438997 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:15:21.438624 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5ba08137-cee0-4cf7-a27a-4c750f9308c4-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w2-f7680d-29894475-qw25k\" (UID: \"5ba08137-cee0-4cf7-a27a-4c750f9308c4\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-f7680d-29894475-qw25k" Nov 03 01:15:21.438997 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:15:21.438691 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-94rt4\" (UniqueName: \"kubernetes.io/projected/5ba08137-cee0-4cf7-a27a-4c750f9308c4-kube-api-access-94rt4\") pod \"maas-api-key-cleanup-e2e-worker-w2-f7680d-29894475-qw25k\" (UID: \"5ba08137-cee0-4cf7-a27a-4c750f9308c4\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-f7680d-29894475-qw25k" Nov 03 01:15:21.438997 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:15:21.438964 4939 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zbl6w\" (UniqueName: \"kubernetes.io/projected/f0a403e0-c375-412d-940d-66ecf89e3270-kube-api-access-zbl6w\") on node \"ip-10-0-140-107.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:15:21.446377 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:15:21.446249 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5ba08137-cee0-4cf7-a27a-4c750f9308c4-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w2-f7680d-29894475-qw25k\" (UID: \"5ba08137-cee0-4cf7-a27a-4c750f9308c4\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-f7680d-29894475-qw25k" Nov 03 01:15:21.481984 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:15:21.481000 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-94rt4\" (UniqueName: \"kubernetes.io/projected/5ba08137-cee0-4cf7-a27a-4c750f9308c4-kube-api-access-94rt4\") pod \"maas-api-key-cleanup-e2e-worker-w2-f7680d-29894475-qw25k\" (UID: \"5ba08137-cee0-4cf7-a27a-4c750f9308c4\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-f7680d-29894475-qw25k" Nov 03 01:15:21.510938 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:15:21.510893 4939 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-2f9a3a-29894475-9nh4j"] Nov 03 01:15:21.511971 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:15:21.511840 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:15:21.515991 ip-10-0-140-107 kubenswrapper[4939]: W1103 01:15:21.515965 4939 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod05835a80_2f35_4b67_b578_42e64bfafb9f.slice/crio-75570ed0b4da36896744ad943d5466dfadd04efff27a498fe52b46af7ae2df25 WatchSource:0}: Error finding container 75570ed0b4da36896744ad943d5466dfadd04efff27a498fe52b46af7ae2df25: Status 404 returned error can't find the container with id 75570ed0b4da36896744ad943d5466dfadd04efff27a498fe52b46af7ae2df25 Nov 03 01:15:21.607830 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:15:21.607797 4939 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-f7680d-29894475-qw25k" Nov 03 01:15:21.616659 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:15:21.616626 4939 patch_prober.go:29] interesting pod/e2e-extmod-a-d25759-openshift-default-d47fd9785-7gzj2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.237:15021/healthz/ready\": dial tcp 10.132.0.237:15021: connect: connection refused" start-of-body= Nov 03 01:15:21.616863 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:15:21.616835 4939 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-d25759-openshift-default-d47fd9785-7gzj2" podUID="ba585f4b-4916-4d03-bf1e-454c029d46ad" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.237:15021/healthz/ready\": dial tcp 10.132.0.237:15021: connect: connection refused" Nov 03 01:15:21.622606 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:15:21.622582 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:15:21.900344 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:15:21.892509 4939 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-83755a-29894475-clqb4" Nov 03 01:15:22.148034 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:15:22.144814 4939 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7eba01d0-b475-421d-8aba-56cbc67c652d" path="/var/lib/kubelet/pods/7eba01d0-b475-421d-8aba-56cbc67c652d/volumes" Nov 03 01:15:22.148034 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:15:22.145502 4939 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="899bb599-2a0d-4670-818b-b0eda152816f" path="/var/lib/kubelet/pods/899bb599-2a0d-4670-818b-b0eda152816f/volumes" Nov 03 01:15:22.148034 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:15:22.146147 4939 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8abddaba-107d-4380-9ae4-5c912e6f4a03" path="/var/lib/kubelet/pods/8abddaba-107d-4380-9ae4-5c912e6f4a03/volumes" Nov 03 01:15:22.407039 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:15:22.406294 4939 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a3c9ab79-8ade-4c81-9e5a-38803c90f7a2" path="/var/lib/kubelet/pods/a3c9ab79-8ade-4c81-9e5a-38803c90f7a2/volumes" Nov 03 01:15:22.407358 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:15:22.407118 4939 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ca1eab39-5984-43f1-bd8c-029cd2131a7a" path="/var/lib/kubelet/pods/ca1eab39-5984-43f1-bd8c-029cd2131a7a/volumes" Nov 03 01:15:22.418275 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:15:22.407992 4939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-83755a-29894475-clqb4" event={"ID":"f0a403e0-c375-412d-940d-66ecf89e3270","Type":"ContainerDied","Data":"82a8d8c3b54ab926489932e2f45b23ad5bb483a9a68e0a733c7236258fff7f00"} Nov 03 01:15:22.418527 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:15:22.418496 4939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-88bbf7-29894475-zhljf" event={"ID":"f4d972cb-44ea-4f7d-a548-91751c607176","Type":"ContainerStarted","Data":"d7a63ef4b4523e2280edd064ce129a07f9db6398ea4d6408913624cab8e62165"} Nov 03 01:15:22.426744 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:15:22.426158 4939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-4af1f7-29894475-l9sxr" event={"ID":"4197807b-0461-4733-ac86-01e34bf5a8ec","Type":"ContainerStarted","Data":"d8d8427bd95d642c90b688db96bd0e8b0d7213a0c3bbef6a112168d742853549"} Nov 03 01:15:22.426744 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:15:22.426600 4939 scope.go:122] "RemoveContainer" containerID="197bf1e71bd85227bb0cd64c8f56f0018d9bb02fcd71c0767bf56037a0516523" Nov 03 01:15:22.516966 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:15:22.463936 4939 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-4af1f7-29894475-l9sxr" podStartSLOduration=22.463900114 podStartE2EDuration="22.463900114s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 01:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 01:15:22.404194815 +0000 UTC m=+2070.668529761" watchObservedRunningTime="2026-11-03 01:15:22.463900114 +0000 UTC m=+2070.728235060" Nov 03 01:15:22.516966 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:15:22.478759 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:15:22.516966 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:15:22.503861 4939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-e1f08d-29894475-tkmgm" event={"ID":"7f838def-06ed-4c8a-90d6-43c5a192354c","Type":"ContainerStarted","Data":"ce7c4d6135fbfe8e00e9ff87bb342e6f2e6984ce7c52258f3db42b65f63e17d6"} Nov 03 01:15:22.516966 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:15:22.508089 4939 generic.go:356] "Generic (PLEG): container finished" podID="8a294b17-dd47-4c08-afb6-4c7236938c01" containerID="26aa38eb96dae44a1d24348df840e331e0416f83d41c9f4411b9112962b85c93" exitCode=60 Nov 03 01:15:22.516966 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:15:22.508180 4939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-556c14-29894475-xxwnm" event={"ID":"8a294b17-dd47-4c08-afb6-4c7236938c01","Type":"ContainerDied","Data":"26aa38eb96dae44a1d24348df840e331e0416f83d41c9f4411b9112962b85c93"} Nov 03 01:15:22.516966 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:15:22.508570 4939 scope.go:122] "RemoveContainer" containerID="26aa38eb96dae44a1d24348df840e331e0416f83d41c9f4411b9112962b85c93" Nov 03 01:15:22.529584 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:15:22.526219 4939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-4a697f-29894475-zcmxf" event={"ID":"1ab583fa-19f7-43e2-acf9-17da1e65d1e0","Type":"ContainerStarted","Data":"0b37db64e1baf008ba3cb275c7d95b9c7f90051c1714958bf212339e01cd26a9"} Nov 03 01:15:22.529584 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:15:22.529172 4939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-2f9a3a-29894475-9nh4j" event={"ID":"05835a80-2f35-4b67-b578-42e64bfafb9f","Type":"ContainerStarted","Data":"75570ed0b4da36896744ad943d5466dfadd04efff27a498fe52b46af7ae2df25"} Nov 03 01:15:22.536008 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:15:22.534210 4939 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-83755a-29894475-clqb4"] Nov 03 01:15:22.594353 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:15:22.594252 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:15:22.594652 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:15:22.594575 4939 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-83755a-29894475-clqb4"] Nov 03 01:15:22.598487 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:15:22.598424 4939 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-e1f08d-29894475-tkmgm" podStartSLOduration=22.598406442 podStartE2EDuration="22.598406442s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 01:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 01:15:22.533736955 +0000 UTC m=+2070.798071899" watchObservedRunningTime="2026-11-03 01:15:22.598406442 +0000 UTC m=+2070.862741389" Nov 03 01:15:22.616235 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:15:22.616136 4939 patch_prober.go:29] interesting pod/e2e-extmod-a-d25759-openshift-default-d47fd9785-7gzj2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.237:15021/healthz/ready\": dial tcp 10.132.0.237:15021: connect: connection refused" start-of-body= Nov 03 01:15:22.616235 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:15:22.616182 4939 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-d25759-openshift-default-d47fd9785-7gzj2" podUID="ba585f4b-4916-4d03-bf1e-454c029d46ad" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.237:15021/healthz/ready\": dial tcp 10.132.0.237:15021: connect: connection refused" Nov 03 01:15:22.618859 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:15:22.618812 4939 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-4a697f-29894475-zcmxf" podStartSLOduration=22.61879513 podStartE2EDuration="22.61879513s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 01:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 01:15:22.605382046 +0000 UTC m=+2070.869717004" watchObservedRunningTime="2026-11-03 01:15:22.61879513 +0000 UTC m=+2070.883130077" Nov 03 01:15:22.723727 ip-10-0-140-107 kubenswrapper[4939]: W1103 01:15:22.721685 4939 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5ba08137_cee0_4cf7_a27a_4c750f9308c4.slice/crio-9af27b717f58d4bed8335a64807408af24f4e45698f57ec7a03d4f45d19da5bf WatchSource:0}: Error finding container 9af27b717f58d4bed8335a64807408af24f4e45698f57ec7a03d4f45d19da5bf: Status 404 returned error can't find the container with id 9af27b717f58d4bed8335a64807408af24f4e45698f57ec7a03d4f45d19da5bf Nov 03 01:15:22.729620 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:15:22.729434 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:15:22.739456 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:15:22.735824 4939 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-f7680d-29894475-qw25k"] Nov 03 01:15:22.880969 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:15:22.880164 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:15:23.514856 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:15:23.514789 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:15:23.555080 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:15:23.554939 4939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-f7680d-29894475-qw25k" event={"ID":"5ba08137-cee0-4cf7-a27a-4c750f9308c4","Type":"ContainerStarted","Data":"0eb8b65c26621edc93ff62c5a1cf775b756af5d6d51af8a3ba611e16bdacc62c"} Nov 03 01:15:23.556272 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:15:23.556242 4939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-f7680d-29894475-qw25k" event={"ID":"5ba08137-cee0-4cf7-a27a-4c750f9308c4","Type":"ContainerStarted","Data":"9af27b717f58d4bed8335a64807408af24f4e45698f57ec7a03d4f45d19da5bf"} Nov 03 01:15:23.558676 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:15:23.558648 4939 generic.go:356] "Generic (PLEG): container finished" podID="f4d972cb-44ea-4f7d-a548-91751c607176" containerID="d2c0edd7c228c2907beb3e7d6d979ce17e1310c155ab7921b3f1092cf7e7ea75" exitCode=60 Nov 03 01:15:23.560322 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:15:23.560185 4939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-88bbf7-29894475-zhljf" event={"ID":"f4d972cb-44ea-4f7d-a548-91751c607176","Type":"ContainerDied","Data":"d2c0edd7c228c2907beb3e7d6d979ce17e1310c155ab7921b3f1092cf7e7ea75"} Nov 03 01:15:23.561462 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:15:23.561441 4939 scope.go:122] "RemoveContainer" containerID="d2c0edd7c228c2907beb3e7d6d979ce17e1310c155ab7921b3f1092cf7e7ea75" Nov 03 01:15:23.628933 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:15:23.605844 4939 generic.go:356] "Generic (PLEG): container finished" podID="4197807b-0461-4733-ac86-01e34bf5a8ec" containerID="d8d8427bd95d642c90b688db96bd0e8b0d7213a0c3bbef6a112168d742853549" exitCode=60 Nov 03 01:15:23.628933 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:15:23.606019 4939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-4af1f7-29894475-l9sxr" event={"ID":"4197807b-0461-4733-ac86-01e34bf5a8ec","Type":"ContainerDied","Data":"d8d8427bd95d642c90b688db96bd0e8b0d7213a0c3bbef6a112168d742853549"} Nov 03 01:15:23.628933 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:15:23.606317 4939 scope.go:122] "RemoveContainer" containerID="d8d8427bd95d642c90b688db96bd0e8b0d7213a0c3bbef6a112168d742853549" Nov 03 01:15:23.628933 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:15:23.611203 4939 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-f7680d-29894475-qw25k" podStartSLOduration=23.611186481 podStartE2EDuration="23.611186481s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 01:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 01:15:23.608577785 +0000 UTC m=+2071.872912731" watchObservedRunningTime="2026-11-03 01:15:23.611186481 +0000 UTC m=+2071.875521426" Nov 03 01:15:23.628933 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:15:23.615818 4939 generic.go:356] "Generic (PLEG): container finished" podID="7f838def-06ed-4c8a-90d6-43c5a192354c" containerID="ce7c4d6135fbfe8e00e9ff87bb342e6f2e6984ce7c52258f3db42b65f63e17d6" exitCode=60 Nov 03 01:15:23.628933 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:15:23.615984 4939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-e1f08d-29894475-tkmgm" event={"ID":"7f838def-06ed-4c8a-90d6-43c5a192354c","Type":"ContainerDied","Data":"ce7c4d6135fbfe8e00e9ff87bb342e6f2e6984ce7c52258f3db42b65f63e17d6"} Nov 03 01:15:23.628933 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:15:23.616287 4939 patch_prober.go:29] interesting pod/e2e-extmod-a-d25759-openshift-default-d47fd9785-7gzj2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.237:15021/healthz/ready\": dial tcp 10.132.0.237:15021: connect: connection refused" start-of-body= Nov 03 01:15:23.628933 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:15:23.616327 4939 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-d25759-openshift-default-d47fd9785-7gzj2" podUID="ba585f4b-4916-4d03-bf1e-454c029d46ad" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.237:15021/healthz/ready\": dial tcp 10.132.0.237:15021: connect: connection refused" Nov 03 01:15:23.628933 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:15:23.616528 4939 scope.go:122] "RemoveContainer" containerID="ce7c4d6135fbfe8e00e9ff87bb342e6f2e6984ce7c52258f3db42b65f63e17d6" Nov 03 01:15:23.628933 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:15:23.626468 4939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-556c14-29894475-xxwnm" event={"ID":"8a294b17-dd47-4c08-afb6-4c7236938c01","Type":"ContainerStarted","Data":"4d4a2bea39af0028c30f7114e2b8c88df52b22c67326277947e6421703591997"} Nov 03 01:15:23.636627 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:15:23.636599 4939 generic.go:356] "Generic (PLEG): container finished" podID="1ab583fa-19f7-43e2-acf9-17da1e65d1e0" containerID="0b37db64e1baf008ba3cb275c7d95b9c7f90051c1714958bf212339e01cd26a9" exitCode=60 Nov 03 01:15:23.636782 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:15:23.636647 4939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-4a697f-29894475-zcmxf" event={"ID":"1ab583fa-19f7-43e2-acf9-17da1e65d1e0","Type":"ContainerDied","Data":"0b37db64e1baf008ba3cb275c7d95b9c7f90051c1714958bf212339e01cd26a9"} Nov 03 01:15:23.637740 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:15:23.637567 4939 scope.go:122] "RemoveContainer" containerID="0b37db64e1baf008ba3cb275c7d95b9c7f90051c1714958bf212339e01cd26a9" Nov 03 01:15:23.641453 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:15:23.641423 4939 generic.go:356] "Generic (PLEG): container finished" podID="05835a80-2f35-4b67-b578-42e64bfafb9f" containerID="86546d683ad9ad1a6f88af9a9e4bdce18f927e19f6f0060791cc2ed68055dd80" exitCode=60 Nov 03 01:15:23.641544 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:15:23.641517 4939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-2f9a3a-29894475-9nh4j" event={"ID":"05835a80-2f35-4b67-b578-42e64bfafb9f","Type":"ContainerDied","Data":"86546d683ad9ad1a6f88af9a9e4bdce18f927e19f6f0060791cc2ed68055dd80"} Nov 03 01:15:23.641869 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:15:23.641848 4939 scope.go:122] "RemoveContainer" containerID="86546d683ad9ad1a6f88af9a9e4bdce18f927e19f6f0060791cc2ed68055dd80" Nov 03 01:15:23.732394 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:15:23.732339 4939 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-556c14-29894475-xxwnm" podStartSLOduration=23.732319905 podStartE2EDuration="23.732319905s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 01:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 01:15:23.724672224 +0000 UTC m=+2071.989007169" watchObservedRunningTime="2026-11-03 01:15:23.732319905 +0000 UTC m=+2071.996654852" Nov 03 01:15:23.908671 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:15:23.908645 4939 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f0a403e0-c375-412d-940d-66ecf89e3270" path="/var/lib/kubelet/pods/f0a403e0-c375-412d-940d-66ecf89e3270/volumes" Nov 03 01:15:24.004371 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:15:24.004342 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:15:24.409329 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:15:24.406664 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:15:24.625398 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:15:24.624560 4939 patch_prober.go:29] interesting pod/e2e-extmod-a-d25759-openshift-default-d47fd9785-7gzj2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.237:15021/healthz/ready\": dial tcp 10.132.0.237:15021: connect: connection refused" start-of-body= Nov 03 01:15:24.625398 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:15:24.624625 4939 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-d25759-openshift-default-d47fd9785-7gzj2" podUID="ba585f4b-4916-4d03-bf1e-454c029d46ad" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.237:15021/healthz/ready\": dial tcp 10.132.0.237:15021: connect: connection refused" Nov 03 01:15:24.691941 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:15:24.677964 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:15:24.719931 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:15:24.715052 4939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-2f9a3a-29894475-9nh4j" event={"ID":"05835a80-2f35-4b67-b578-42e64bfafb9f","Type":"ContainerStarted","Data":"30f5912a0306a6e76808c35970542d5bf4bbd7312491212e22beadd3ceaa3f3f"} Nov 03 01:15:24.777079 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:15:24.777040 4939 generic.go:356] "Generic (PLEG): container finished" podID="5ba08137-cee0-4cf7-a27a-4c750f9308c4" containerID="0eb8b65c26621edc93ff62c5a1cf775b756af5d6d51af8a3ba611e16bdacc62c" exitCode=60 Nov 03 01:15:24.777428 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:15:24.777398 4939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-f7680d-29894475-qw25k" event={"ID":"5ba08137-cee0-4cf7-a27a-4c750f9308c4","Type":"ContainerDied","Data":"0eb8b65c26621edc93ff62c5a1cf775b756af5d6d51af8a3ba611e16bdacc62c"} Nov 03 01:15:24.885010 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:15:24.883278 4939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-88bbf7-29894475-zhljf" event={"ID":"f4d972cb-44ea-4f7d-a548-91751c607176","Type":"ContainerStarted","Data":"bac008af913fe15f1d96270394131b6a1b6a9238f47954b8f065e25f216f5474"} Nov 03 01:15:24.885010 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:15:24.883643 4939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-4af1f7-29894475-l9sxr" event={"ID":"4197807b-0461-4733-ac86-01e34bf5a8ec","Type":"ContainerStarted","Data":"9d1cc8e7a1f24b2c193df66f65f79c36385f6d0df5406ac8f40c3b4d9ed07803"} Nov 03 01:15:24.885010 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:15:24.799123 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:15:24.885010 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:15:24.778124 4939 scope.go:122] "RemoveContainer" containerID="0eb8b65c26621edc93ff62c5a1cf775b756af5d6d51af8a3ba611e16bdacc62c" Nov 03 01:15:24.973434 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:15:24.973380 4939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-e1f08d-29894475-tkmgm" event={"ID":"7f838def-06ed-4c8a-90d6-43c5a192354c","Type":"ContainerStarted","Data":"c3969099dfe6418def321997dc01727896812a93928d3d08f09199b26a8cc36a"} Nov 03 01:15:25.098946 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:15:25.064306 4939 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-2f9a3a-29894475-9nh4j" podStartSLOduration=25.064286193 podStartE2EDuration="25.064286193s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 01:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 01:15:24.755889385 +0000 UTC m=+2073.020224330" watchObservedRunningTime="2026-11-03 01:15:25.064286193 +0000 UTC m=+2073.328621139" Nov 03 01:15:25.111250 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:15:25.110664 4939 generic.go:356] "Generic (PLEG): container finished" podID="8a294b17-dd47-4c08-afb6-4c7236938c01" containerID="4d4a2bea39af0028c30f7114e2b8c88df52b22c67326277947e6421703591997" exitCode=60 Nov 03 01:15:25.111250 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:15:25.110789 4939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-556c14-29894475-xxwnm" event={"ID":"8a294b17-dd47-4c08-afb6-4c7236938c01","Type":"ContainerDied","Data":"4d4a2bea39af0028c30f7114e2b8c88df52b22c67326277947e6421703591997"} Nov 03 01:15:25.111250 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:15:25.110846 4939 scope.go:122] "RemoveContainer" containerID="26aa38eb96dae44a1d24348df840e331e0416f83d41c9f4411b9112962b85c93" Nov 03 01:15:25.113558 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:15:25.113426 4939 scope.go:122] "RemoveContainer" containerID="4d4a2bea39af0028c30f7114e2b8c88df52b22c67326277947e6421703591997" Nov 03 01:15:25.117792 ip-10-0-140-107 kubenswrapper[4939]: E1103 01:15:25.116605 4939 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w3-556c14-29894475-xxwnm_odh-ai-gateway-infra(8a294b17-dd47-4c08-afb6-4c7236938c01)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-556c14-29894475-xxwnm" podUID="8a294b17-dd47-4c08-afb6-4c7236938c01" Nov 03 01:15:25.147961 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:15:25.147920 4939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-4a697f-29894475-zcmxf" event={"ID":"1ab583fa-19f7-43e2-acf9-17da1e65d1e0","Type":"ContainerStarted","Data":"eb49945fee81be86cd487cd233985a664ad09460c7018e37360263a1abe5ff96"} Nov 03 01:15:25.166585 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:15:25.159167 4939 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-88bbf7-29894475-zhljf" podStartSLOduration=25.159146492 podStartE2EDuration="25.159146492s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 01:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 01:15:24.868999367 +0000 UTC m=+2073.133334313" watchObservedRunningTime="2026-11-03 01:15:25.159146492 +0000 UTC m=+2073.423481440" Nov 03 01:15:25.707236 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:15:25.707194 4939 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-extmod-a-d25759-openshift-default-d47fd9785-7gzj2" Nov 03 01:15:25.708302 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:15:25.708230 4939 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-a-d25759-openshift-default-d47fd9785-7gzj2" Nov 03 01:15:25.709351 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:15:25.709328 4939 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-extmod-a-d25759-openshift-default-d47fd9785-7gzj2" Nov 03 01:15:25.911867 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:15:25.911706 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:15:26.158362 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:15:26.158332 4939 scope.go:122] "RemoveContainer" containerID="4d4a2bea39af0028c30f7114e2b8c88df52b22c67326277947e6421703591997" Nov 03 01:15:26.158790 ip-10-0-140-107 kubenswrapper[4939]: E1103 01:15:26.158753 4939 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w3-556c14-29894475-xxwnm_odh-ai-gateway-infra(8a294b17-dd47-4c08-afb6-4c7236938c01)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-556c14-29894475-xxwnm" podUID="8a294b17-dd47-4c08-afb6-4c7236938c01" Nov 03 01:15:26.160020 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:15:26.159996 4939 generic.go:356] "Generic (PLEG): container finished" podID="1ab583fa-19f7-43e2-acf9-17da1e65d1e0" containerID="eb49945fee81be86cd487cd233985a664ad09460c7018e37360263a1abe5ff96" exitCode=60 Nov 03 01:15:26.160125 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:15:26.160084 4939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-4a697f-29894475-zcmxf" event={"ID":"1ab583fa-19f7-43e2-acf9-17da1e65d1e0","Type":"ContainerDied","Data":"eb49945fee81be86cd487cd233985a664ad09460c7018e37360263a1abe5ff96"} Nov 03 01:15:26.160181 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:15:26.160142 4939 scope.go:122] "RemoveContainer" containerID="0b37db64e1baf008ba3cb275c7d95b9c7f90051c1714958bf212339e01cd26a9" Nov 03 01:15:26.160400 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:15:26.160377 4939 scope.go:122] "RemoveContainer" containerID="eb49945fee81be86cd487cd233985a664ad09460c7018e37360263a1abe5ff96" Nov 03 01:15:26.160861 ip-10-0-140-107 kubenswrapper[4939]: E1103 01:15:26.160833 4939 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w4-4a697f-29894475-zcmxf_odh-ai-gateway-infra(1ab583fa-19f7-43e2-acf9-17da1e65d1e0)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-4a697f-29894475-zcmxf" podUID="1ab583fa-19f7-43e2-acf9-17da1e65d1e0" Nov 03 01:15:26.162349 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:15:26.162292 4939 generic.go:356] "Generic (PLEG): container finished" podID="05835a80-2f35-4b67-b578-42e64bfafb9f" containerID="30f5912a0306a6e76808c35970542d5bf4bbd7312491212e22beadd3ceaa3f3f" exitCode=60 Nov 03 01:15:26.162349 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:15:26.162323 4939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-2f9a3a-29894475-9nh4j" event={"ID":"05835a80-2f35-4b67-b578-42e64bfafb9f","Type":"ContainerDied","Data":"30f5912a0306a6e76808c35970542d5bf4bbd7312491212e22beadd3ceaa3f3f"} Nov 03 01:15:26.162641 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:15:26.162627 4939 scope.go:122] "RemoveContainer" containerID="30f5912a0306a6e76808c35970542d5bf4bbd7312491212e22beadd3ceaa3f3f" Nov 03 01:15:26.163043 ip-10-0-140-107 kubenswrapper[4939]: E1103 01:15:26.163020 4939 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w0-2f9a3a-29894475-9nh4j_odh-ai-gateway-infra(05835a80-2f35-4b67-b578-42e64bfafb9f)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-2f9a3a-29894475-9nh4j" podUID="05835a80-2f35-4b67-b578-42e64bfafb9f" Nov 03 01:15:26.164450 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:15:26.164425 4939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-f7680d-29894475-qw25k" event={"ID":"5ba08137-cee0-4cf7-a27a-4c750f9308c4","Type":"ContainerStarted","Data":"7622a14bf17dea92d76843b6dd11d73c6fa9e915ef8fa34b3dd8ebf83b07675f"} Nov 03 01:15:26.166398 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:15:26.166369 4939 generic.go:356] "Generic (PLEG): container finished" podID="f4d972cb-44ea-4f7d-a548-91751c607176" containerID="bac008af913fe15f1d96270394131b6a1b6a9238f47954b8f065e25f216f5474" exitCode=60 Nov 03 01:15:26.166624 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:15:26.166441 4939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-88bbf7-29894475-zhljf" event={"ID":"f4d972cb-44ea-4f7d-a548-91751c607176","Type":"ContainerDied","Data":"bac008af913fe15f1d96270394131b6a1b6a9238f47954b8f065e25f216f5474"} Nov 03 01:15:26.166702 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:15:26.166682 4939 scope.go:122] "RemoveContainer" containerID="bac008af913fe15f1d96270394131b6a1b6a9238f47954b8f065e25f216f5474" Nov 03 01:15:26.167019 ip-10-0-140-107 kubenswrapper[4939]: E1103 01:15:26.166995 4939 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w1-88bbf7-29894475-zhljf_odh-ai-gateway-infra(f4d972cb-44ea-4f7d-a548-91751c607176)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-88bbf7-29894475-zhljf" podUID="f4d972cb-44ea-4f7d-a548-91751c607176" Nov 03 01:15:26.168689 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:15:26.168669 4939 generic.go:356] "Generic (PLEG): container finished" podID="4197807b-0461-4733-ac86-01e34bf5a8ec" containerID="9d1cc8e7a1f24b2c193df66f65f79c36385f6d0df5406ac8f40c3b4d9ed07803" exitCode=60 Nov 03 01:15:26.168794 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:15:26.168752 4939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-4af1f7-29894475-l9sxr" event={"ID":"4197807b-0461-4733-ac86-01e34bf5a8ec","Type":"ContainerDied","Data":"9d1cc8e7a1f24b2c193df66f65f79c36385f6d0df5406ac8f40c3b4d9ed07803"} Nov 03 01:15:26.168988 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:15:26.168972 4939 scope.go:122] "RemoveContainer" containerID="9d1cc8e7a1f24b2c193df66f65f79c36385f6d0df5406ac8f40c3b4d9ed07803" Nov 03 01:15:26.169271 ip-10-0-140-107 kubenswrapper[4939]: E1103 01:15:26.169244 4939 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w1-4af1f7-29894475-l9sxr_odh-ai-gateway-infra(4197807b-0461-4733-ac86-01e34bf5a8ec)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-4af1f7-29894475-l9sxr" podUID="4197807b-0461-4733-ac86-01e34bf5a8ec" Nov 03 01:15:26.170745 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:15:26.170724 4939 generic.go:356] "Generic (PLEG): container finished" podID="7f838def-06ed-4c8a-90d6-43c5a192354c" containerID="c3969099dfe6418def321997dc01727896812a93928d3d08f09199b26a8cc36a" exitCode=60 Nov 03 01:15:26.170822 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:15:26.170806 4939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-e1f08d-29894475-tkmgm" event={"ID":"7f838def-06ed-4c8a-90d6-43c5a192354c","Type":"ContainerDied","Data":"c3969099dfe6418def321997dc01727896812a93928d3d08f09199b26a8cc36a"} Nov 03 01:15:26.171036 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:15:26.171019 4939 scope.go:122] "RemoveContainer" containerID="c3969099dfe6418def321997dc01727896812a93928d3d08f09199b26a8cc36a" Nov 03 01:15:26.171268 ip-10-0-140-107 kubenswrapper[4939]: E1103 01:15:26.171250 4939 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-b-e1f08d-29894475-tkmgm_odh-ai-gateway-infra(7f838def-06ed-4c8a-90d6-43c5a192354c)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-e1f08d-29894475-tkmgm" podUID="7f838def-06ed-4c8a-90d6-43c5a192354c" Nov 03 01:15:26.175707 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:15:26.175692 4939 scope.go:122] "RemoveContainer" containerID="86546d683ad9ad1a6f88af9a9e4bdce18f927e19f6f0060791cc2ed68055dd80" Nov 03 01:15:26.194423 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:15:26.194405 4939 scope.go:122] "RemoveContainer" containerID="d2c0edd7c228c2907beb3e7d6d979ce17e1310c155ab7921b3f1092cf7e7ea75" Nov 03 01:15:26.210096 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:15:26.210077 4939 scope.go:122] "RemoveContainer" containerID="d8d8427bd95d642c90b688db96bd0e8b0d7213a0c3bbef6a112168d742853549" Nov 03 01:15:26.224457 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:15:26.224433 4939 scope.go:122] "RemoveContainer" containerID="ce7c4d6135fbfe8e00e9ff87bb342e6f2e6984ce7c52258f3db42b65f63e17d6" Nov 03 01:15:26.697332 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:15:26.697301 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:15:27.182313 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:15:27.182278 4939 scope.go:122] "RemoveContainer" containerID="bac008af913fe15f1d96270394131b6a1b6a9238f47954b8f065e25f216f5474" Nov 03 01:15:27.182746 ip-10-0-140-107 kubenswrapper[4939]: E1103 01:15:27.182662 4939 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w1-88bbf7-29894475-zhljf_odh-ai-gateway-infra(f4d972cb-44ea-4f7d-a548-91751c607176)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-88bbf7-29894475-zhljf" podUID="f4d972cb-44ea-4f7d-a548-91751c607176" Nov 03 01:15:27.184076 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:15:27.184052 4939 scope.go:122] "RemoveContainer" containerID="9d1cc8e7a1f24b2c193df66f65f79c36385f6d0df5406ac8f40c3b4d9ed07803" Nov 03 01:15:27.184406 ip-10-0-140-107 kubenswrapper[4939]: E1103 01:15:27.184381 4939 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w1-4af1f7-29894475-l9sxr_odh-ai-gateway-infra(4197807b-0461-4733-ac86-01e34bf5a8ec)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-4af1f7-29894475-l9sxr" podUID="4197807b-0461-4733-ac86-01e34bf5a8ec" Nov 03 01:15:27.185817 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:15:27.185798 4939 scope.go:122] "RemoveContainer" containerID="c3969099dfe6418def321997dc01727896812a93928d3d08f09199b26a8cc36a" Nov 03 01:15:27.186154 ip-10-0-140-107 kubenswrapper[4939]: E1103 01:15:27.186126 4939 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-b-e1f08d-29894475-tkmgm_odh-ai-gateway-infra(7f838def-06ed-4c8a-90d6-43c5a192354c)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-e1f08d-29894475-tkmgm" podUID="7f838def-06ed-4c8a-90d6-43c5a192354c" Nov 03 01:15:27.187761 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:15:27.187740 4939 scope.go:122] "RemoveContainer" containerID="eb49945fee81be86cd487cd233985a664ad09460c7018e37360263a1abe5ff96" Nov 03 01:15:27.188154 ip-10-0-140-107 kubenswrapper[4939]: E1103 01:15:27.188126 4939 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w4-4a697f-29894475-zcmxf_odh-ai-gateway-infra(1ab583fa-19f7-43e2-acf9-17da1e65d1e0)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-4a697f-29894475-zcmxf" podUID="1ab583fa-19f7-43e2-acf9-17da1e65d1e0" Nov 03 01:15:27.189460 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:15:27.189441 4939 scope.go:122] "RemoveContainer" containerID="30f5912a0306a6e76808c35970542d5bf4bbd7312491212e22beadd3ceaa3f3f" Nov 03 01:15:27.189744 ip-10-0-140-107 kubenswrapper[4939]: E1103 01:15:27.189723 4939 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w0-2f9a3a-29894475-9nh4j_odh-ai-gateway-infra(05835a80-2f35-4b67-b578-42e64bfafb9f)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-2f9a3a-29894475-9nh4j" podUID="05835a80-2f35-4b67-b578-42e64bfafb9f" Nov 03 01:15:27.191312 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:15:27.191292 4939 generic.go:356] "Generic (PLEG): container finished" podID="5ba08137-cee0-4cf7-a27a-4c750f9308c4" containerID="7622a14bf17dea92d76843b6dd11d73c6fa9e915ef8fa34b3dd8ebf83b07675f" exitCode=60 Nov 03 01:15:27.191422 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:15:27.191344 4939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-f7680d-29894475-qw25k" event={"ID":"5ba08137-cee0-4cf7-a27a-4c750f9308c4","Type":"ContainerDied","Data":"7622a14bf17dea92d76843b6dd11d73c6fa9e915ef8fa34b3dd8ebf83b07675f"} Nov 03 01:15:27.191422 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:15:27.191398 4939 scope.go:122] "RemoveContainer" containerID="0eb8b65c26621edc93ff62c5a1cf775b756af5d6d51af8a3ba611e16bdacc62c" Nov 03 01:15:27.191580 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:15:27.191557 4939 scope.go:122] "RemoveContainer" containerID="7622a14bf17dea92d76843b6dd11d73c6fa9e915ef8fa34b3dd8ebf83b07675f" Nov 03 01:15:27.191864 ip-10-0-140-107 kubenswrapper[4939]: E1103 01:15:27.191840 4939 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w2-f7680d-29894475-qw25k_odh-ai-gateway-infra(5ba08137-cee0-4cf7-a27a-4c750f9308c4)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-f7680d-29894475-qw25k" podUID="5ba08137-cee0-4cf7-a27a-4c750f9308c4" Nov 03 01:15:27.677392 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:15:27.677352 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:15:28.079406 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:15:28.079314 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:15:28.242423 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:15:28.242383 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:15:28.250219 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:15:28.250192 4939 scope.go:122] "RemoveContainer" containerID="7622a14bf17dea92d76843b6dd11d73c6fa9e915ef8fa34b3dd8ebf83b07675f" Nov 03 01:15:28.259967 ip-10-0-140-107 kubenswrapper[4939]: E1103 01:15:28.251338 4939 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w2-f7680d-29894475-qw25k_odh-ai-gateway-infra(5ba08137-cee0-4cf7-a27a-4c750f9308c4)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-f7680d-29894475-qw25k" podUID="5ba08137-cee0-4cf7-a27a-4c750f9308c4" Nov 03 01:15:29.260711 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:15:29.260671 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:15:29.420437 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:15:29.420397 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:15:29.768704 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:15:29.768670 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:15:29.891101 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:15:29.891070 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:15:30.051528 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:15:30.051424 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:15:30.286147 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:15:30.286105 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:15:31.037000 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:15:31.036899 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:15:31.681206 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:15:31.681128 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:15:32.486902 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:15:32.486868 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:15:33.175054 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:15:33.175014 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:15:33.884685 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:15:33.884652 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:15:34.672949 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:15:34.672884 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:15:35.023835 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:15:35.023754 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:15:35.187945 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:15:35.178484 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:15:35.863419 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:15:35.861237 4939 scope.go:122] "RemoveContainer" containerID="4d4a2bea39af0028c30f7114e2b8c88df52b22c67326277947e6421703591997" Nov 03 01:15:35.973973 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:15:35.973942 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:15:36.365802 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:15:36.365761 4939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-556c14-29894475-xxwnm" event={"ID":"8a294b17-dd47-4c08-afb6-4c7236938c01","Type":"ContainerStarted","Data":"f1abc18401f3a3b327bfc017f52af7e98ce6708a5201b13961cf70c1922310d8"} Nov 03 01:15:36.474687 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:15:36.474653 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:15:36.604026 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:15:36.603981 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:15:36.726581 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:15:36.726547 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:15:36.996899 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:15:36.996820 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:15:37.377027 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:15:37.376996 4939 generic.go:356] "Generic (PLEG): container finished" podID="8a294b17-dd47-4c08-afb6-4c7236938c01" containerID="f1abc18401f3a3b327bfc017f52af7e98ce6708a5201b13961cf70c1922310d8" exitCode=60 Nov 03 01:15:37.377194 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:15:37.377063 4939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-556c14-29894475-xxwnm" event={"ID":"8a294b17-dd47-4c08-afb6-4c7236938c01","Type":"ContainerDied","Data":"f1abc18401f3a3b327bfc017f52af7e98ce6708a5201b13961cf70c1922310d8"} Nov 03 01:15:37.377194 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:15:37.377126 4939 scope.go:122] "RemoveContainer" containerID="4d4a2bea39af0028c30f7114e2b8c88df52b22c67326277947e6421703591997" Nov 03 01:15:37.377504 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:15:37.377482 4939 scope.go:122] "RemoveContainer" containerID="f1abc18401f3a3b327bfc017f52af7e98ce6708a5201b13961cf70c1922310d8" Nov 03 01:15:37.377867 ip-10-0-140-107 kubenswrapper[4939]: E1103 01:15:37.377844 4939 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w3-556c14-29894475-xxwnm_odh-ai-gateway-infra(8a294b17-dd47-4c08-afb6-4c7236938c01)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-556c14-29894475-xxwnm" podUID="8a294b17-dd47-4c08-afb6-4c7236938c01" Nov 03 01:15:37.399607 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:15:37.399583 4939 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-556c14-29894475-xxwnm"] Nov 03 01:15:37.745851 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:15:37.745775 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:15:37.866948 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:15:37.863314 4939 scope.go:122] "RemoveContainer" containerID="eb49945fee81be86cd487cd233985a664ad09460c7018e37360263a1abe5ff96" Nov 03 01:15:38.390342 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:15:38.390308 4939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-4a697f-29894475-zcmxf" event={"ID":"1ab583fa-19f7-43e2-acf9-17da1e65d1e0","Type":"ContainerStarted","Data":"d1e8310116ccfd18a1a8500745e81688bc68e90e15d6b44754effeadd6a43956"} Nov 03 01:15:38.399282 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:15:38.399257 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:15:38.665985 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:15:38.665960 4939 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-556c14-29894475-xxwnm" Nov 03 01:15:38.713832 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:15:38.713696 4939 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-50e623-29894475-zkk6f"] Nov 03 01:15:38.715574 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:15:38.715540 4939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8a294b17-dd47-4c08-afb6-4c7236938c01-kube-api-access-db5x9\" (UniqueName: \"kubernetes.io/projected/8a294b17-dd47-4c08-afb6-4c7236938c01-kube-api-access-db5x9\") pod \"8a294b17-dd47-4c08-afb6-4c7236938c01\" (UID: \"8a294b17-dd47-4c08-afb6-4c7236938c01\") " Nov 03 01:15:38.716631 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:15:38.716599 4939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8a294b17-dd47-4c08-afb6-4c7236938c01-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8a294b17-dd47-4c08-afb6-4c7236938c01-maas-api-service-ca\") pod \"8a294b17-dd47-4c08-afb6-4c7236938c01\" (UID: \"8a294b17-dd47-4c08-afb6-4c7236938c01\") " Nov 03 01:15:38.717665 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:15:38.717042 4939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8a294b17-dd47-4c08-afb6-4c7236938c01-maas-api-service-ca" pod "8a294b17-dd47-4c08-afb6-4c7236938c01" (UID: "8a294b17-dd47-4c08-afb6-4c7236938c01"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 01:15:38.717665 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:15:38.717231 4939 cpu_manager.go:414] "containerMap: removing container" podUID="8a294b17-dd47-4c08-afb6-4c7236938c01" containerName="cleanup" Nov 03 01:15:38.717665 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:15:38.717251 4939 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8a294b17-dd47-4c08-afb6-4c7236938c01" containerName="cleanup" Nov 03 01:15:38.717665 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:15:38.717363 4939 cpu_manager.go:414] "containerMap: removing container" podUID="8a294b17-dd47-4c08-afb6-4c7236938c01" containerName="cleanup" Nov 03 01:15:38.717665 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:15:38.717371 4939 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8a294b17-dd47-4c08-afb6-4c7236938c01" containerName="cleanup" Nov 03 01:15:38.717665 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:15:38.717633 4939 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8a294b17-dd47-4c08-afb6-4c7236938c01-maas-api-service-ca\") on node \"ip-10-0-140-107.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:15:38.718072 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:15:38.717676 4939 memory_manager.go:365] "RemoveStaleState removing state" podUID="8a294b17-dd47-4c08-afb6-4c7236938c01" containerName="cleanup" Nov 03 01:15:38.718072 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:15:38.717694 4939 memory_manager.go:365] "RemoveStaleState removing state" podUID="8a294b17-dd47-4c08-afb6-4c7236938c01" containerName="cleanup" Nov 03 01:15:38.718072 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:15:38.717743 4939 memory_manager.go:365] "RemoveStaleState removing state" podUID="8a294b17-dd47-4c08-afb6-4c7236938c01" containerName="cleanup" Nov 03 01:15:38.725187 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:15:38.725164 4939 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-50e623-29894475-zkk6f"] Nov 03 01:15:38.725361 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:15:38.725276 4939 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-50e623-29894475-zkk6f" Nov 03 01:15:38.727162 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:15:38.727121 4939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8a294b17-dd47-4c08-afb6-4c7236938c01-kube-api-access-db5x9" pod "8a294b17-dd47-4c08-afb6-4c7236938c01" (UID: "8a294b17-dd47-4c08-afb6-4c7236938c01"). InnerVolumeSpecName "kube-api-access-db5x9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 01:15:38.819367 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:15:38.819329 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b24379ee-39df-43ad-b624-556142c07324-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w0-50e623-29894475-zkk6f\" (UID: \"b24379ee-39df-43ad-b624-556142c07324\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-50e623-29894475-zkk6f" Nov 03 01:15:38.819581 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:15:38.819560 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r7hrf\" (UniqueName: \"kubernetes.io/projected/b24379ee-39df-43ad-b624-556142c07324-kube-api-access-r7hrf\") pod \"maas-api-key-cleanup-e2e-shared-a-w0-50e623-29894475-zkk6f\" (UID: \"b24379ee-39df-43ad-b624-556142c07324\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-50e623-29894475-zkk6f" Nov 03 01:15:38.819737 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:15:38.819724 4939 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-db5x9\" (UniqueName: \"kubernetes.io/projected/8a294b17-dd47-4c08-afb6-4c7236938c01-kube-api-access-db5x9\") on node \"ip-10-0-140-107.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:15:38.860828 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:15:38.860801 4939 scope.go:122] "RemoveContainer" containerID="c3969099dfe6418def321997dc01727896812a93928d3d08f09199b26a8cc36a" Nov 03 01:15:38.861040 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:15:38.861017 4939 scope.go:122] "RemoveContainer" containerID="7622a14bf17dea92d76843b6dd11d73c6fa9e915ef8fa34b3dd8ebf83b07675f" Nov 03 01:15:38.861808 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:15:38.861789 4939 scope.go:122] "RemoveContainer" containerID="bac008af913fe15f1d96270394131b6a1b6a9238f47954b8f065e25f216f5474" Nov 03 01:15:38.921151 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:15:38.921121 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b24379ee-39df-43ad-b624-556142c07324-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w0-50e623-29894475-zkk6f\" (UID: \"b24379ee-39df-43ad-b624-556142c07324\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-50e623-29894475-zkk6f" Nov 03 01:15:38.921870 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:15:38.921488 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r7hrf\" (UniqueName: \"kubernetes.io/projected/b24379ee-39df-43ad-b624-556142c07324-kube-api-access-r7hrf\") pod \"maas-api-key-cleanup-e2e-shared-a-w0-50e623-29894475-zkk6f\" (UID: \"b24379ee-39df-43ad-b624-556142c07324\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-50e623-29894475-zkk6f" Nov 03 01:15:38.921870 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:15:38.921855 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b24379ee-39df-43ad-b624-556142c07324-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w0-50e623-29894475-zkk6f\" (UID: \"b24379ee-39df-43ad-b624-556142c07324\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-50e623-29894475-zkk6f" Nov 03 01:15:38.930453 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:15:38.930432 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-r7hrf\" (UniqueName: \"kubernetes.io/projected/b24379ee-39df-43ad-b624-556142c07324-kube-api-access-r7hrf\") pod \"maas-api-key-cleanup-e2e-shared-a-w0-50e623-29894475-zkk6f\" (UID: \"b24379ee-39df-43ad-b624-556142c07324\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-50e623-29894475-zkk6f" Nov 03 01:15:39.064547 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:15:39.064511 4939 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-50e623-29894475-zkk6f" Nov 03 01:15:39.166474 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:15:39.166435 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:15:39.402092 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:15:39.402064 4939 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-556c14-29894475-xxwnm" Nov 03 01:15:39.404032 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:15:39.402072 4939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-556c14-29894475-xxwnm" event={"ID":"8a294b17-dd47-4c08-afb6-4c7236938c01","Type":"ContainerDied","Data":"a0dcc6e123662c255d6c271b62b70eca7cc92f9bf03076a902d0166ef87b020e"} Nov 03 01:15:39.404032 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:15:39.402734 4939 scope.go:122] "RemoveContainer" containerID="f1abc18401f3a3b327bfc017f52af7e98ce6708a5201b13961cf70c1922310d8" Nov 03 01:15:39.406190 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:15:39.406163 4939 generic.go:356] "Generic (PLEG): container finished" podID="1ab583fa-19f7-43e2-acf9-17da1e65d1e0" containerID="d1e8310116ccfd18a1a8500745e81688bc68e90e15d6b44754effeadd6a43956" exitCode=60 Nov 03 01:15:39.407281 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:15:39.407260 4939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-4a697f-29894475-zcmxf" event={"ID":"1ab583fa-19f7-43e2-acf9-17da1e65d1e0","Type":"ContainerDied","Data":"d1e8310116ccfd18a1a8500745e81688bc68e90e15d6b44754effeadd6a43956"} Nov 03 01:15:39.408138 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:15:39.408118 4939 scope.go:122] "RemoveContainer" containerID="d1e8310116ccfd18a1a8500745e81688bc68e90e15d6b44754effeadd6a43956" Nov 03 01:15:39.408822 ip-10-0-140-107 kubenswrapper[4939]: E1103 01:15:39.408797 4939 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w4-4a697f-29894475-zcmxf_odh-ai-gateway-infra(1ab583fa-19f7-43e2-acf9-17da1e65d1e0)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-4a697f-29894475-zcmxf" podUID="1ab583fa-19f7-43e2-acf9-17da1e65d1e0" Nov 03 01:15:39.416158 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:15:39.416117 4939 scope.go:122] "RemoveContainer" containerID="eb49945fee81be86cd487cd233985a664ad09460c7018e37360263a1abe5ff96" Nov 03 01:15:39.443025 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:15:39.443001 4939 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-4a697f-29894475-zcmxf"] Nov 03 01:15:39.455033 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:15:39.455008 4939 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-556c14-29894475-xxwnm"] Nov 03 01:15:39.468507 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:15:39.468476 4939 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-556c14-29894475-xxwnm"] Nov 03 01:15:39.475150 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:15:39.475125 4939 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-50e623-29894475-zkk6f"] Nov 03 01:15:39.475288 ip-10-0-140-107 kubenswrapper[4939]: W1103 01:15:39.475262 4939 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb24379ee_39df_43ad_b624_556142c07324.slice/crio-a120a9d160fc916b71b5f4cd50338c56f5703686ccbb748c6bdedfc02738c62c WatchSource:0}: Error finding container a120a9d160fc916b71b5f4cd50338c56f5703686ccbb748c6bdedfc02738c62c: Status 404 returned error can't find the container with id a120a9d160fc916b71b5f4cd50338c56f5703686ccbb748c6bdedfc02738c62c Nov 03 01:15:39.864968 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:15:39.864384 4939 scope.go:122] "RemoveContainer" containerID="9d1cc8e7a1f24b2c193df66f65f79c36385f6d0df5406ac8f40c3b4d9ed07803" Nov 03 01:15:39.869384 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:15:39.868451 4939 scope.go:122] "RemoveContainer" containerID="30f5912a0306a6e76808c35970542d5bf4bbd7312491212e22beadd3ceaa3f3f" Nov 03 01:15:39.887106 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:15:39.887082 4939 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8a294b17-dd47-4c08-afb6-4c7236938c01" path="/var/lib/kubelet/pods/8a294b17-dd47-4c08-afb6-4c7236938c01/volumes" Nov 03 01:15:39.898487 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:15:39.898460 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:15:40.166942 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:15:40.132436 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:15:40.392636 ip-10-0-140-107 kubenswrapper[4939]: E1103 01:15:40.392436 4939 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf4d972cb_44ea_4f7d_a548_91751c607176.slice/crio-f379dbcb8aadbf02d069724828fe383807cb7415aeb7509f4ccf6cd426ed2818.scope\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf4d972cb_44ea_4f7d_a548_91751c607176.slice/crio-conmon-f379dbcb8aadbf02d069724828fe383807cb7415aeb7509f4ccf6cd426ed2818.scope\": RecentStats: unable to find data in memory cache]" Nov 03 01:15:40.418278 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:15:40.418238 4939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-50e623-29894475-zkk6f" event={"ID":"b24379ee-39df-43ad-b624-556142c07324","Type":"ContainerStarted","Data":"57b40bdbe70de0ed33741ab2458be0f063b9d383c5922cf5815c133638319fbd"} Nov 03 01:15:40.418712 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:15:40.418653 4939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-50e623-29894475-zkk6f" event={"ID":"b24379ee-39df-43ad-b624-556142c07324","Type":"ContainerStarted","Data":"a120a9d160fc916b71b5f4cd50338c56f5703686ccbb748c6bdedfc02738c62c"} Nov 03 01:15:40.420186 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:15:40.420162 4939 generic.go:356] "Generic (PLEG): container finished" podID="5ba08137-cee0-4cf7-a27a-4c750f9308c4" containerID="9c2f388457a4268162d02975d15bb613deaad62eaf82177422670bd121158c28" exitCode=60 Nov 03 01:15:40.420307 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:15:40.420232 4939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-f7680d-29894475-qw25k" event={"ID":"5ba08137-cee0-4cf7-a27a-4c750f9308c4","Type":"ContainerDied","Data":"9c2f388457a4268162d02975d15bb613deaad62eaf82177422670bd121158c28"} Nov 03 01:15:40.420307 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:15:40.420281 4939 scope.go:122] "RemoveContainer" containerID="7622a14bf17dea92d76843b6dd11d73c6fa9e915ef8fa34b3dd8ebf83b07675f" Nov 03 01:15:40.420767 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:15:40.420748 4939 scope.go:122] "RemoveContainer" containerID="9c2f388457a4268162d02975d15bb613deaad62eaf82177422670bd121158c28" Nov 03 01:15:40.421320 ip-10-0-140-107 kubenswrapper[4939]: E1103 01:15:40.421293 4939 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w2-f7680d-29894475-qw25k_odh-ai-gateway-infra(5ba08137-cee0-4cf7-a27a-4c750f9308c4)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-f7680d-29894475-qw25k" podUID="5ba08137-cee0-4cf7-a27a-4c750f9308c4" Nov 03 01:15:40.423439 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:15:40.423420 4939 generic.go:356] "Generic (PLEG): container finished" podID="f4d972cb-44ea-4f7d-a548-91751c607176" containerID="f379dbcb8aadbf02d069724828fe383807cb7415aeb7509f4ccf6cd426ed2818" exitCode=60 Nov 03 01:15:40.423550 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:15:40.423488 4939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-88bbf7-29894475-zhljf" event={"ID":"f4d972cb-44ea-4f7d-a548-91751c607176","Type":"ContainerDied","Data":"f379dbcb8aadbf02d069724828fe383807cb7415aeb7509f4ccf6cd426ed2818"} Nov 03 01:15:40.424007 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:15:40.423985 4939 scope.go:122] "RemoveContainer" containerID="f379dbcb8aadbf02d069724828fe383807cb7415aeb7509f4ccf6cd426ed2818" Nov 03 01:15:40.424389 ip-10-0-140-107 kubenswrapper[4939]: E1103 01:15:40.424369 4939 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w1-88bbf7-29894475-zhljf_odh-ai-gateway-infra(f4d972cb-44ea-4f7d-a548-91751c607176)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-88bbf7-29894475-zhljf" podUID="f4d972cb-44ea-4f7d-a548-91751c607176" Nov 03 01:15:40.425599 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:15:40.425568 4939 generic.go:356] "Generic (PLEG): container finished" podID="7f838def-06ed-4c8a-90d6-43c5a192354c" containerID="221ec8655567508bb02701ca412ab200f8f933d81363e517bf3af42cbba9c740" exitCode=60 Nov 03 01:15:40.425669 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:15:40.425637 4939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-e1f08d-29894475-tkmgm" event={"ID":"7f838def-06ed-4c8a-90d6-43c5a192354c","Type":"ContainerDied","Data":"221ec8655567508bb02701ca412ab200f8f933d81363e517bf3af42cbba9c740"} Nov 03 01:15:40.426002 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:15:40.425983 4939 scope.go:122] "RemoveContainer" containerID="221ec8655567508bb02701ca412ab200f8f933d81363e517bf3af42cbba9c740" Nov 03 01:15:40.426332 ip-10-0-140-107 kubenswrapper[4939]: E1103 01:15:40.426293 4939 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-b-e1f08d-29894475-tkmgm_odh-ai-gateway-infra(7f838def-06ed-4c8a-90d6-43c5a192354c)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-e1f08d-29894475-tkmgm" podUID="7f838def-06ed-4c8a-90d6-43c5a192354c" Nov 03 01:15:40.435941 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:15:40.435924 4939 scope.go:122] "RemoveContainer" containerID="bac008af913fe15f1d96270394131b6a1b6a9238f47954b8f065e25f216f5474" Nov 03 01:15:40.444732 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:15:40.444689 4939 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-50e623-29894475-zkk6f" podStartSLOduration=40.44467609 podStartE2EDuration="40.44467609s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 01:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 01:15:40.442742035 +0000 UTC m=+2088.707076983" watchObservedRunningTime="2026-11-03 01:15:40.44467609 +0000 UTC m=+2088.709011037" Nov 03 01:15:40.453896 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:15:40.453867 4939 scope.go:122] "RemoveContainer" containerID="c3969099dfe6418def321997dc01727896812a93928d3d08f09199b26a8cc36a" Nov 03 01:15:41.040340 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:15:41.040314 4939 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-4a697f-29894475-zcmxf" Nov 03 01:15:41.105170 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:15:41.105132 4939 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-41953fef-726x2"] Nov 03 01:15:41.106731 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:15:41.106699 4939 cpu_manager.go:414] "containerMap: removing container" podUID="8a294b17-dd47-4c08-afb6-4c7236938c01" containerName="cleanup" Nov 03 01:15:41.106731 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:15:41.106732 4939 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8a294b17-dd47-4c08-afb6-4c7236938c01" containerName="cleanup" Nov 03 01:15:41.106903 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:15:41.106815 4939 cpu_manager.go:414] "containerMap: removing container" podUID="1ab583fa-19f7-43e2-acf9-17da1e65d1e0" containerName="cleanup" Nov 03 01:15:41.106903 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:15:41.106824 4939 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1ab583fa-19f7-43e2-acf9-17da1e65d1e0" containerName="cleanup" Nov 03 01:15:41.106903 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:15:41.106859 4939 cpu_manager.go:414] "containerMap: removing container" podUID="1ab583fa-19f7-43e2-acf9-17da1e65d1e0" containerName="cleanup" Nov 03 01:15:41.106903 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:15:41.106867 4939 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1ab583fa-19f7-43e2-acf9-17da1e65d1e0" containerName="cleanup" Nov 03 01:15:41.106903 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:15:41.106894 4939 cpu_manager.go:414] "containerMap: removing container" podUID="1ab583fa-19f7-43e2-acf9-17da1e65d1e0" containerName="cleanup" Nov 03 01:15:41.106903 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:15:41.106901 4939 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1ab583fa-19f7-43e2-acf9-17da1e65d1e0" containerName="cleanup" Nov 03 01:15:41.107213 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:15:41.107132 4939 memory_manager.go:365] "RemoveStaleState removing state" podUID="1ab583fa-19f7-43e2-acf9-17da1e65d1e0" containerName="cleanup" Nov 03 01:15:41.107213 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:15:41.107165 4939 memory_manager.go:365] "RemoveStaleState removing state" podUID="1ab583fa-19f7-43e2-acf9-17da1e65d1e0" containerName="cleanup" Nov 03 01:15:41.107213 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:15:41.107179 4939 memory_manager.go:365] "RemoveStaleState removing state" podUID="1ab583fa-19f7-43e2-acf9-17da1e65d1e0" containerName="cleanup" Nov 03 01:15:41.112027 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:15:41.112006 4939 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-41953fef-726x2" Nov 03 01:15:41.112236 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:15:41.112208 4939 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-41953fef-726x2"] Nov 03 01:15:41.116031 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:15:41.116007 4939 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 01:15:41.127594 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:15:41.127573 4939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1ab583fa-19f7-43e2-acf9-17da1e65d1e0-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1ab583fa-19f7-43e2-acf9-17da1e65d1e0-maas-api-service-ca\") pod \"1ab583fa-19f7-43e2-acf9-17da1e65d1e0\" (UID: \"1ab583fa-19f7-43e2-acf9-17da1e65d1e0\") " Nov 03 01:15:41.127721 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:15:41.127670 4939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1ab583fa-19f7-43e2-acf9-17da1e65d1e0-kube-api-access-429zj\" (UniqueName: \"kubernetes.io/projected/1ab583fa-19f7-43e2-acf9-17da1e65d1e0-kube-api-access-429zj\") pod \"1ab583fa-19f7-43e2-acf9-17da1e65d1e0\" (UID: \"1ab583fa-19f7-43e2-acf9-17da1e65d1e0\") " Nov 03 01:15:41.128108 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:15:41.128082 4939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1ab583fa-19f7-43e2-acf9-17da1e65d1e0-maas-api-service-ca" pod "1ab583fa-19f7-43e2-acf9-17da1e65d1e0" (UID: "1ab583fa-19f7-43e2-acf9-17da1e65d1e0"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 01:15:41.128736 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:15:41.128703 4939 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1ab583fa-19f7-43e2-acf9-17da1e65d1e0-maas-api-service-ca\") on node \"ip-10-0-140-107.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:15:41.131141 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:15:41.131114 4939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1ab583fa-19f7-43e2-acf9-17da1e65d1e0-kube-api-access-429zj" pod "1ab583fa-19f7-43e2-acf9-17da1e65d1e0" (UID: "1ab583fa-19f7-43e2-acf9-17da1e65d1e0"). InnerVolumeSpecName "kube-api-access-429zj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 01:15:41.230810 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:15:41.230737 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cvx2f\" (UniqueName: \"kubernetes.io/projected/d59e2793-c378-4558-9b72-25e596ddc473-kube-api-access-cvx2f\") pod \"maas-api-revoke-sub-41953fef-726x2\" (UID: \"d59e2793-c378-4558-9b72-25e596ddc473\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-41953fef-726x2" Nov 03 01:15:41.231060 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:15:41.231041 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d59e2793-c378-4558-9b72-25e596ddc473-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-41953fef-726x2\" (UID: \"d59e2793-c378-4558-9b72-25e596ddc473\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-41953fef-726x2" Nov 03 01:15:41.231131 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:15:41.231110 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d59e2793-c378-4558-9b72-25e596ddc473-maas-api-xks-ca\") pod \"maas-api-revoke-sub-41953fef-726x2\" (UID: \"d59e2793-c378-4558-9b72-25e596ddc473\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-41953fef-726x2" Nov 03 01:15:41.231305 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:15:41.231290 4939 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-429zj\" (UniqueName: \"kubernetes.io/projected/1ab583fa-19f7-43e2-acf9-17da1e65d1e0-kube-api-access-429zj\") on node \"ip-10-0-140-107.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:15:41.332352 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:15:41.332315 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d59e2793-c378-4558-9b72-25e596ddc473-maas-api-xks-ca\") pod \"maas-api-revoke-sub-41953fef-726x2\" (UID: \"d59e2793-c378-4558-9b72-25e596ddc473\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-41953fef-726x2" Nov 03 01:15:41.332510 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:15:41.332489 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cvx2f\" (UniqueName: \"kubernetes.io/projected/d59e2793-c378-4558-9b72-25e596ddc473-kube-api-access-cvx2f\") pod \"maas-api-revoke-sub-41953fef-726x2\" (UID: \"d59e2793-c378-4558-9b72-25e596ddc473\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-41953fef-726x2" Nov 03 01:15:41.332776 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:15:41.332757 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d59e2793-c378-4558-9b72-25e596ddc473-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-41953fef-726x2\" (UID: \"d59e2793-c378-4558-9b72-25e596ddc473\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-41953fef-726x2" Nov 03 01:15:41.333412 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:15:41.333391 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d59e2793-c378-4558-9b72-25e596ddc473-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-41953fef-726x2\" (UID: \"d59e2793-c378-4558-9b72-25e596ddc473\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-41953fef-726x2" Nov 03 01:15:41.334781 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:15:41.334761 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d59e2793-c378-4558-9b72-25e596ddc473-maas-api-xks-ca\") pod \"maas-api-revoke-sub-41953fef-726x2\" (UID: \"d59e2793-c378-4558-9b72-25e596ddc473\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-41953fef-726x2" Nov 03 01:15:41.340747 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:15:41.340721 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cvx2f\" (UniqueName: \"kubernetes.io/projected/d59e2793-c378-4558-9b72-25e596ddc473-kube-api-access-cvx2f\") pod \"maas-api-revoke-sub-41953fef-726x2\" (UID: \"d59e2793-c378-4558-9b72-25e596ddc473\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-41953fef-726x2" Nov 03 01:15:41.433505 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:15:41.433471 4939 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-41953fef-726x2" Nov 03 01:15:41.441406 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:15:41.441379 4939 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-4a697f-29894475-zcmxf" Nov 03 01:15:41.441557 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:15:41.441377 4939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-4a697f-29894475-zcmxf" event={"ID":"1ab583fa-19f7-43e2-acf9-17da1e65d1e0","Type":"ContainerDied","Data":"5975bc80f953f4c15554528bb10e05dae09ce88ba822f8821836651f0c30db75"} Nov 03 01:15:41.441557 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:15:41.441549 4939 scope.go:122] "RemoveContainer" containerID="d1e8310116ccfd18a1a8500745e81688bc68e90e15d6b44754effeadd6a43956" Nov 03 01:15:41.443634 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:15:41.443611 4939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-2f9a3a-29894475-9nh4j" event={"ID":"05835a80-2f35-4b67-b578-42e64bfafb9f","Type":"ContainerStarted","Data":"aa8f751ec2626121aa3bf197b8bf1c0ff0d026398ebff0e70b8adf6a2c6cd733"} Nov 03 01:15:41.446535 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:15:41.446512 4939 generic.go:356] "Generic (PLEG): container finished" podID="b24379ee-39df-43ad-b624-556142c07324" containerID="57b40bdbe70de0ed33741ab2458be0f063b9d383c5922cf5815c133638319fbd" exitCode=60 Nov 03 01:15:41.446658 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:15:41.446631 4939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-50e623-29894475-zkk6f" event={"ID":"b24379ee-39df-43ad-b624-556142c07324","Type":"ContainerDied","Data":"57b40bdbe70de0ed33741ab2458be0f063b9d383c5922cf5815c133638319fbd"} Nov 03 01:15:41.447053 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:15:41.447029 4939 scope.go:122] "RemoveContainer" containerID="57b40bdbe70de0ed33741ab2458be0f063b9d383c5922cf5815c133638319fbd" Nov 03 01:15:41.455189 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:15:41.455166 4939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-4af1f7-29894475-l9sxr" event={"ID":"4197807b-0461-4733-ac86-01e34bf5a8ec","Type":"ContainerStarted","Data":"295352015f26fdca56fef6c40f6e3555ab5216e3eb4fc3eac3fbb11abdab60be"} Nov 03 01:15:41.482577 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:15:41.482497 4939 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-88bbf7-29894475-zhljf"] Nov 03 01:15:41.625809 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:15:41.625675 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:15:41.629592 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:15:41.629126 4939 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-f7680d-29894475-qw25k"] Nov 03 01:15:41.657016 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:15:41.656984 4939 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-e1f08d-29894475-tkmgm"] Nov 03 01:15:41.709082 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:15:41.709044 4939 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-4a697f-29894475-zcmxf"] Nov 03 01:15:41.714282 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:15:41.714247 4939 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-4a697f-29894475-zcmxf"] Nov 03 01:15:41.742076 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:15:41.742045 4939 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-41953fef-726x2"] Nov 03 01:15:41.751240 ip-10-0-140-107 kubenswrapper[4939]: W1103 01:15:41.751213 4939 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd59e2793_c378_4558_9b72_25e596ddc473.slice/crio-3baa032e04c238b15c7adf3422d186853abb0a5397704340a3f6763c63765fa9 WatchSource:0}: Error finding container 3baa032e04c238b15c7adf3422d186853abb0a5397704340a3f6763c63765fa9: Status 404 returned error can't find the container with id 3baa032e04c238b15c7adf3422d186853abb0a5397704340a3f6763c63765fa9 Nov 03 01:15:41.835727 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:15:41.835647 4939 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-88bbf7-29894475-zhljf" Nov 03 01:15:41.870711 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:15:41.867705 4939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f4d972cb-44ea-4f7d-a548-91751c607176-kube-api-access-pswzc\" (UniqueName: \"kubernetes.io/projected/f4d972cb-44ea-4f7d-a548-91751c607176-kube-api-access-pswzc\") pod \"f4d972cb-44ea-4f7d-a548-91751c607176\" (UID: \"f4d972cb-44ea-4f7d-a548-91751c607176\") " Nov 03 01:15:41.870711 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:15:41.868009 4939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f4d972cb-44ea-4f7d-a548-91751c607176-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f4d972cb-44ea-4f7d-a548-91751c607176-maas-api-service-ca\") pod \"f4d972cb-44ea-4f7d-a548-91751c607176\" (UID: \"f4d972cb-44ea-4f7d-a548-91751c607176\") " Nov 03 01:15:41.870711 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:15:41.868463 4939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f4d972cb-44ea-4f7d-a548-91751c607176-maas-api-service-ca" pod "f4d972cb-44ea-4f7d-a548-91751c607176" (UID: "f4d972cb-44ea-4f7d-a548-91751c607176"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 01:15:41.870711 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:15:41.868657 4939 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f4d972cb-44ea-4f7d-a548-91751c607176-maas-api-service-ca\") on node \"ip-10-0-140-107.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:15:41.884315 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:15:41.884286 4939 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1ab583fa-19f7-43e2-acf9-17da1e65d1e0" path="/var/lib/kubelet/pods/1ab583fa-19f7-43e2-acf9-17da1e65d1e0/volumes" Nov 03 01:15:41.949005 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:15:41.948952 4939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f4d972cb-44ea-4f7d-a548-91751c607176-kube-api-access-pswzc" pod "f4d972cb-44ea-4f7d-a548-91751c607176" (UID: "f4d972cb-44ea-4f7d-a548-91751c607176"). InnerVolumeSpecName "kube-api-access-pswzc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 01:15:41.979667 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:15:41.979636 4939 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pswzc\" (UniqueName: \"kubernetes.io/projected/f4d972cb-44ea-4f7d-a548-91751c607176-kube-api-access-pswzc\") on node \"ip-10-0-140-107.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:15:42.047576 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:15:42.047548 4939 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-e1f08d-29894475-tkmgm" Nov 03 01:15:42.050075 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:15:42.050044 4939 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-f7680d-29894475-qw25k" Nov 03 01:15:42.082044 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:15:42.082009 4939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5ba08137-cee0-4cf7-a27a-4c750f9308c4-kube-api-access-94rt4\" (UniqueName: \"kubernetes.io/projected/5ba08137-cee0-4cf7-a27a-4c750f9308c4-kube-api-access-94rt4\") pod \"5ba08137-cee0-4cf7-a27a-4c750f9308c4\" (UID: \"5ba08137-cee0-4cf7-a27a-4c750f9308c4\") " Nov 03 01:15:42.082555 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:15:42.082530 4939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5ba08137-cee0-4cf7-a27a-4c750f9308c4-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5ba08137-cee0-4cf7-a27a-4c750f9308c4-maas-api-service-ca\") pod \"5ba08137-cee0-4cf7-a27a-4c750f9308c4\" (UID: \"5ba08137-cee0-4cf7-a27a-4c750f9308c4\") " Nov 03 01:15:42.082813 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:15:42.082795 4939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7f838def-06ed-4c8a-90d6-43c5a192354c-kube-api-access-zk5xc\" (UniqueName: \"kubernetes.io/projected/7f838def-06ed-4c8a-90d6-43c5a192354c-kube-api-access-zk5xc\") pod \"7f838def-06ed-4c8a-90d6-43c5a192354c\" (UID: \"7f838def-06ed-4c8a-90d6-43c5a192354c\") " Nov 03 01:15:42.083032 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:15:42.083014 4939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7f838def-06ed-4c8a-90d6-43c5a192354c-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7f838def-06ed-4c8a-90d6-43c5a192354c-maas-api-service-ca\") pod \"7f838def-06ed-4c8a-90d6-43c5a192354c\" (UID: \"7f838def-06ed-4c8a-90d6-43c5a192354c\") " Nov 03 01:15:42.083458 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:15:42.083290 4939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5ba08137-cee0-4cf7-a27a-4c750f9308c4-maas-api-service-ca" pod "5ba08137-cee0-4cf7-a27a-4c750f9308c4" (UID: "5ba08137-cee0-4cf7-a27a-4c750f9308c4"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 01:15:42.083961 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:15:42.083940 4939 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5ba08137-cee0-4cf7-a27a-4c750f9308c4-maas-api-service-ca\") on node \"ip-10-0-140-107.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:15:42.084468 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:15:42.084427 4939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7f838def-06ed-4c8a-90d6-43c5a192354c-maas-api-service-ca" pod "7f838def-06ed-4c8a-90d6-43c5a192354c" (UID: "7f838def-06ed-4c8a-90d6-43c5a192354c"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 01:15:42.088623 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:15:42.088596 4939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5ba08137-cee0-4cf7-a27a-4c750f9308c4-kube-api-access-94rt4" pod "5ba08137-cee0-4cf7-a27a-4c750f9308c4" (UID: "5ba08137-cee0-4cf7-a27a-4c750f9308c4"). InnerVolumeSpecName "kube-api-access-94rt4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 01:15:42.089601 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:15:42.089574 4939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7f838def-06ed-4c8a-90d6-43c5a192354c-kube-api-access-zk5xc" pod "7f838def-06ed-4c8a-90d6-43c5a192354c" (UID: "7f838def-06ed-4c8a-90d6-43c5a192354c"). InnerVolumeSpecName "kube-api-access-zk5xc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 01:15:42.185708 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:15:42.185677 4939 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zk5xc\" (UniqueName: \"kubernetes.io/projected/7f838def-06ed-4c8a-90d6-43c5a192354c-kube-api-access-zk5xc\") on node \"ip-10-0-140-107.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:15:42.185708 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:15:42.185711 4939 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7f838def-06ed-4c8a-90d6-43c5a192354c-maas-api-service-ca\") on node \"ip-10-0-140-107.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:15:42.185892 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:15:42.185738 4939 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-94rt4\" (UniqueName: \"kubernetes.io/projected/5ba08137-cee0-4cf7-a27a-4c750f9308c4-kube-api-access-94rt4\") on node \"ip-10-0-140-107.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:15:42.377850 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:15:42.377801 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:15:42.504314 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:15:42.503535 4939 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-2f9a3a-29894475-9nh4j"] Nov 03 01:15:42.507017 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:15:42.506931 4939 generic.go:356] "Generic (PLEG): container finished" podID="05835a80-2f35-4b67-b578-42e64bfafb9f" containerID="aa8f751ec2626121aa3bf197b8bf1c0ff0d026398ebff0e70b8adf6a2c6cd733" exitCode=60 Nov 03 01:15:42.507173 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:15:42.507133 4939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-2f9a3a-29894475-9nh4j" event={"ID":"05835a80-2f35-4b67-b578-42e64bfafb9f","Type":"ContainerDied","Data":"aa8f751ec2626121aa3bf197b8bf1c0ff0d026398ebff0e70b8adf6a2c6cd733"} Nov 03 01:15:42.507227 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:15:42.507187 4939 scope.go:122] "RemoveContainer" containerID="30f5912a0306a6e76808c35970542d5bf4bbd7312491212e22beadd3ceaa3f3f" Nov 03 01:15:42.516037 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:15:42.516008 4939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-50e623-29894475-zkk6f" event={"ID":"b24379ee-39df-43ad-b624-556142c07324","Type":"ContainerStarted","Data":"783c1c257f6a4ed790bd159acf7f3dcfd20ce7e8d098c115225cd820258cba49"} Nov 03 01:15:42.517642 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:15:42.517611 4939 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-4af1f7-29894475-l9sxr"] Nov 03 01:15:42.518809 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:15:42.518785 4939 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-f7680d-29894475-qw25k" Nov 03 01:15:42.519002 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:15:42.518793 4939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-f7680d-29894475-qw25k" event={"ID":"5ba08137-cee0-4cf7-a27a-4c750f9308c4","Type":"ContainerDied","Data":"9af27b717f58d4bed8335a64807408af24f4e45698f57ec7a03d4f45d19da5bf"} Nov 03 01:15:42.521553 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:15:42.521518 4939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-88bbf7-29894475-zhljf" event={"ID":"f4d972cb-44ea-4f7d-a548-91751c607176","Type":"ContainerDied","Data":"d7a63ef4b4523e2280edd064ce129a07f9db6398ea4d6408913624cab8e62165"} Nov 03 01:15:42.522708 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:15:42.522682 4939 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-88bbf7-29894475-zhljf" Nov 03 01:15:42.537410 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:15:42.535041 4939 scope.go:122] "RemoveContainer" containerID="9c2f388457a4268162d02975d15bb613deaad62eaf82177422670bd121158c28" Nov 03 01:15:42.557101 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:15:42.557053 4939 generic.go:356] "Generic (PLEG): container finished" podID="4197807b-0461-4733-ac86-01e34bf5a8ec" containerID="295352015f26fdca56fef6c40f6e3555ab5216e3eb4fc3eac3fbb11abdab60be" exitCode=60 Nov 03 01:15:42.557245 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:15:42.557219 4939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-4af1f7-29894475-l9sxr" event={"ID":"4197807b-0461-4733-ac86-01e34bf5a8ec","Type":"ContainerDied","Data":"295352015f26fdca56fef6c40f6e3555ab5216e3eb4fc3eac3fbb11abdab60be"} Nov 03 01:15:42.564918 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:15:42.564338 4939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-e1f08d-29894475-tkmgm" event={"ID":"7f838def-06ed-4c8a-90d6-43c5a192354c","Type":"ContainerDied","Data":"21b639b38b4467f1458245a443a8469c32aee9f701545f682b64962fbf3383f9"} Nov 03 01:15:42.564918 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:15:42.564486 4939 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-e1f08d-29894475-tkmgm" Nov 03 01:15:42.571512 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:15:42.571478 4939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-41953fef-726x2" event={"ID":"d59e2793-c378-4558-9b72-25e596ddc473","Type":"ContainerStarted","Data":"25c87621c0bf3163a94ba3aa7a555843387cdc3a24dc93778bf38f2975019840"} Nov 03 01:15:42.571673 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:15:42.571657 4939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-41953fef-726x2" event={"ID":"d59e2793-c378-4558-9b72-25e596ddc473","Type":"ContainerStarted","Data":"3baa032e04c238b15c7adf3422d186853abb0a5397704340a3f6763c63765fa9"} Nov 03 01:15:42.576500 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:15:42.576432 4939 scope.go:122] "RemoveContainer" containerID="f379dbcb8aadbf02d069724828fe383807cb7415aeb7509f4ccf6cd426ed2818" Nov 03 01:15:42.584940 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:15:42.583435 4939 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-88bbf7-29894475-zhljf"] Nov 03 01:15:42.596970 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:15:42.595438 4939 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-88bbf7-29894475-zhljf"] Nov 03 01:15:42.600579 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:15:42.598080 4939 scope.go:122] "RemoveContainer" containerID="9d1cc8e7a1f24b2c193df66f65f79c36385f6d0df5406ac8f40c3b4d9ed07803" Nov 03 01:15:42.614374 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:15:42.614347 4939 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-f7680d-29894475-qw25k"] Nov 03 01:15:42.617390 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:15:42.617064 4939 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-f7680d-29894475-qw25k"] Nov 03 01:15:42.642640 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:15:42.639589 4939 scope.go:122] "RemoveContainer" containerID="221ec8655567508bb02701ca412ab200f8f933d81363e517bf3af42cbba9c740" Nov 03 01:15:42.646977 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:15:42.646928 4939 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-e1f08d-29894475-tkmgm"] Nov 03 01:15:42.651927 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:15:42.651886 4939 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-e1f08d-29894475-tkmgm"] Nov 03 01:15:42.665683 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:15:42.665631 4939 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-41953fef-726x2" podStartSLOduration=2.665598121 podStartE2EDuration="2.665598121s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 01:15:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 01:15:42.663510948 +0000 UTC m=+2090.927845900" watchObservedRunningTime="2026-11-03 01:15:42.665598121 +0000 UTC m=+2090.929933067" Nov 03 01:15:42.711096 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:15:42.711068 4939 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-2f9a3a-29894475-9nh4j" Nov 03 01:15:42.814116 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:15:42.814083 4939 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-4af1f7-29894475-l9sxr" Nov 03 01:15:42.845766 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:15:42.845732 4939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/05835a80-2f35-4b67-b578-42e64bfafb9f-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/05835a80-2f35-4b67-b578-42e64bfafb9f-maas-api-service-ca\") pod \"05835a80-2f35-4b67-b578-42e64bfafb9f\" (UID: \"05835a80-2f35-4b67-b578-42e64bfafb9f\") " Nov 03 01:15:42.846018 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:15:42.845997 4939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/05835a80-2f35-4b67-b578-42e64bfafb9f-kube-api-access-xwdf4\" (UniqueName: \"kubernetes.io/projected/05835a80-2f35-4b67-b578-42e64bfafb9f-kube-api-access-xwdf4\") pod \"05835a80-2f35-4b67-b578-42e64bfafb9f\" (UID: \"05835a80-2f35-4b67-b578-42e64bfafb9f\") " Nov 03 01:15:42.846808 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:15:42.846780 4939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/05835a80-2f35-4b67-b578-42e64bfafb9f-maas-api-service-ca" pod "05835a80-2f35-4b67-b578-42e64bfafb9f" (UID: "05835a80-2f35-4b67-b578-42e64bfafb9f"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 01:15:42.852109 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:15:42.852079 4939 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-b-af8244/payload-processing-external-model-e2e-extmod-b-af8244-55d6pxhs9"] Nov 03 01:15:42.854194 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:15:42.853582 4939 cpu_manager.go:414] "containerMap: removing container" podUID="f4d972cb-44ea-4f7d-a548-91751c607176" containerName="cleanup" Nov 03 01:15:42.854382 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:15:42.854362 4939 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f4d972cb-44ea-4f7d-a548-91751c607176" containerName="cleanup" Nov 03 01:15:42.854534 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:15:42.854522 4939 cpu_manager.go:414] "containerMap: removing container" podUID="f4d972cb-44ea-4f7d-a548-91751c607176" containerName="cleanup" Nov 03 01:15:42.854617 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:15:42.854606 4939 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f4d972cb-44ea-4f7d-a548-91751c607176" containerName="cleanup" Nov 03 01:15:42.854710 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:15:42.854700 4939 cpu_manager.go:414] "containerMap: removing container" podUID="05835a80-2f35-4b67-b578-42e64bfafb9f" containerName="cleanup" Nov 03 01:15:42.854785 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:15:42.854775 4939 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="05835a80-2f35-4b67-b578-42e64bfafb9f" containerName="cleanup" Nov 03 01:15:42.854862 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:15:42.854853 4939 cpu_manager.go:414] "containerMap: removing container" podUID="05835a80-2f35-4b67-b578-42e64bfafb9f" containerName="cleanup" Nov 03 01:15:42.854950 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:15:42.854939 4939 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="05835a80-2f35-4b67-b578-42e64bfafb9f" containerName="cleanup" Nov 03 01:15:42.855042 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:15:42.855032 4939 cpu_manager.go:414] "containerMap: removing container" podUID="4197807b-0461-4733-ac86-01e34bf5a8ec" containerName="cleanup" Nov 03 01:15:42.855135 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:15:42.855103 4939 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4197807b-0461-4733-ac86-01e34bf5a8ec" containerName="cleanup" Nov 03 01:15:42.855258 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:15:42.855247 4939 cpu_manager.go:414] "containerMap: removing container" podUID="05835a80-2f35-4b67-b578-42e64bfafb9f" containerName="cleanup" Nov 03 01:15:42.855373 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:15:42.855358 4939 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="05835a80-2f35-4b67-b578-42e64bfafb9f" containerName="cleanup" Nov 03 01:15:42.855472 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:15:42.855460 4939 cpu_manager.go:414] "containerMap: removing container" podUID="f4d972cb-44ea-4f7d-a548-91751c607176" containerName="cleanup" Nov 03 01:15:42.855565 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:15:42.855552 4939 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f4d972cb-44ea-4f7d-a548-91751c607176" containerName="cleanup" Nov 03 01:15:42.855665 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:15:42.855653 4939 cpu_manager.go:414] "containerMap: removing container" podUID="5ba08137-cee0-4cf7-a27a-4c750f9308c4" containerName="cleanup" Nov 03 01:15:42.855771 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:15:42.855758 4939 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5ba08137-cee0-4cf7-a27a-4c750f9308c4" containerName="cleanup" Nov 03 01:15:42.855868 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:15:42.855858 4939 cpu_manager.go:414] "containerMap: removing container" podUID="5ba08137-cee0-4cf7-a27a-4c750f9308c4" containerName="cleanup" Nov 03 01:15:42.855990 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:15:42.855977 4939 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5ba08137-cee0-4cf7-a27a-4c750f9308c4" containerName="cleanup" Nov 03 01:15:42.856103 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:15:42.856091 4939 cpu_manager.go:414] "containerMap: removing container" podUID="7f838def-06ed-4c8a-90d6-43c5a192354c" containerName="cleanup" Nov 03 01:15:42.856184 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:15:42.856173 4939 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7f838def-06ed-4c8a-90d6-43c5a192354c" containerName="cleanup" Nov 03 01:15:42.856308 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:15:42.856296 4939 cpu_manager.go:414] "containerMap: removing container" podUID="7f838def-06ed-4c8a-90d6-43c5a192354c" containerName="cleanup" Nov 03 01:15:42.856447 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:15:42.856434 4939 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7f838def-06ed-4c8a-90d6-43c5a192354c" containerName="cleanup" Nov 03 01:15:42.857301 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:15:42.857281 4939 cpu_manager.go:414] "containerMap: removing container" podUID="4197807b-0461-4733-ac86-01e34bf5a8ec" containerName="cleanup" Nov 03 01:15:42.860735 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:15:42.860715 4939 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4197807b-0461-4733-ac86-01e34bf5a8ec" containerName="cleanup" Nov 03 01:15:42.860897 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:15:42.860885 4939 cpu_manager.go:414] "containerMap: removing container" podUID="7f838def-06ed-4c8a-90d6-43c5a192354c" containerName="cleanup" Nov 03 01:15:42.861010 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:15:42.860998 4939 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7f838def-06ed-4c8a-90d6-43c5a192354c" containerName="cleanup" Nov 03 01:15:42.861388 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:15:42.861373 4939 memory_manager.go:365] "RemoveStaleState removing state" podUID="f4d972cb-44ea-4f7d-a548-91751c607176" containerName="cleanup" Nov 03 01:15:42.861494 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:15:42.861485 4939 memory_manager.go:365] "RemoveStaleState removing state" podUID="f4d972cb-44ea-4f7d-a548-91751c607176" containerName="cleanup" Nov 03 01:15:42.861575 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:15:42.861567 4939 memory_manager.go:365] "RemoveStaleState removing state" podUID="05835a80-2f35-4b67-b578-42e64bfafb9f" containerName="cleanup" Nov 03 01:15:42.861656 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:15:42.861647 4939 memory_manager.go:365] "RemoveStaleState removing state" podUID="05835a80-2f35-4b67-b578-42e64bfafb9f" containerName="cleanup" Nov 03 01:15:42.861732 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:15:42.861723 4939 memory_manager.go:365] "RemoveStaleState removing state" podUID="4197807b-0461-4733-ac86-01e34bf5a8ec" containerName="cleanup" Nov 03 01:15:42.861803 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:15:42.861795 4939 memory_manager.go:365] "RemoveStaleState removing state" podUID="05835a80-2f35-4b67-b578-42e64bfafb9f" containerName="cleanup" Nov 03 01:15:42.861876 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:15:42.861867 4939 memory_manager.go:365] "RemoveStaleState removing state" podUID="5ba08137-cee0-4cf7-a27a-4c750f9308c4" containerName="cleanup" Nov 03 01:15:42.861973 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:15:42.861963 4939 memory_manager.go:365] "RemoveStaleState removing state" podUID="5ba08137-cee0-4cf7-a27a-4c750f9308c4" containerName="cleanup" Nov 03 01:15:42.862050 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:15:42.862042 4939 memory_manager.go:365] "RemoveStaleState removing state" podUID="7f838def-06ed-4c8a-90d6-43c5a192354c" containerName="cleanup" Nov 03 01:15:42.862134 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:15:42.862126 4939 memory_manager.go:365] "RemoveStaleState removing state" podUID="7f838def-06ed-4c8a-90d6-43c5a192354c" containerName="cleanup" Nov 03 01:15:42.862217 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:15:42.862208 4939 memory_manager.go:365] "RemoveStaleState removing state" podUID="4197807b-0461-4733-ac86-01e34bf5a8ec" containerName="cleanup" Nov 03 01:15:42.862304 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:15:42.862295 4939 memory_manager.go:365] "RemoveStaleState removing state" podUID="4197807b-0461-4733-ac86-01e34bf5a8ec" containerName="cleanup" Nov 03 01:15:42.862386 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:15:42.862377 4939 memory_manager.go:365] "RemoveStaleState removing state" podUID="5ba08137-cee0-4cf7-a27a-4c750f9308c4" containerName="cleanup" Nov 03 01:15:42.870666 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:15:42.870642 4939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/05835a80-2f35-4b67-b578-42e64bfafb9f-kube-api-access-xwdf4" pod "05835a80-2f35-4b67-b578-42e64bfafb9f" (UID: "05835a80-2f35-4b67-b578-42e64bfafb9f"). InnerVolumeSpecName "kube-api-access-xwdf4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 01:15:42.873708 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:15:42.873679 4939 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-af8244/payload-processing-external-model-e2e-extmod-b-af8244-55d6pxhs9"] Nov 03 01:15:42.876976 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:15:42.873833 4939 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-af8244/payload-processing-external-model-e2e-extmod-b-af8244-55d6pxhs9" Nov 03 01:15:42.948352 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:15:42.948297 4939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4197807b-0461-4733-ac86-01e34bf5a8ec-kube-api-access-5rgfc\" (UniqueName: \"kubernetes.io/projected/4197807b-0461-4733-ac86-01e34bf5a8ec-kube-api-access-5rgfc\") pod \"4197807b-0461-4733-ac86-01e34bf5a8ec\" (UID: \"4197807b-0461-4733-ac86-01e34bf5a8ec\") " Nov 03 01:15:42.948518 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:15:42.948496 4939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4197807b-0461-4733-ac86-01e34bf5a8ec-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4197807b-0461-4733-ac86-01e34bf5a8ec-maas-api-service-ca\") pod \"4197807b-0461-4733-ac86-01e34bf5a8ec\" (UID: \"4197807b-0461-4733-ac86-01e34bf5a8ec\") " Nov 03 01:15:42.948922 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:15:42.948888 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cff06885-ae10-4a06-ae3d-d5db99244d99-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-af8244-55d6pxhs9\" (UID: \"cff06885-ae10-4a06-ae3d-d5db99244d99\") " pod="ai-tenant-e2e-extmod-b-af8244/payload-processing-external-model-e2e-extmod-b-af8244-55d6pxhs9" Nov 03 01:15:42.949080 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:15:42.949057 4939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4197807b-0461-4733-ac86-01e34bf5a8ec-maas-api-service-ca" pod "4197807b-0461-4733-ac86-01e34bf5a8ec" (UID: "4197807b-0461-4733-ac86-01e34bf5a8ec"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 01:15:42.949080 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:15:42.949063 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/cff06885-ae10-4a06-ae3d-d5db99244d99-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-af8244-55d6pxhs9\" (UID: \"cff06885-ae10-4a06-ae3d-d5db99244d99\") " pod="ai-tenant-e2e-extmod-b-af8244/payload-processing-external-model-e2e-extmod-b-af8244-55d6pxhs9" Nov 03 01:15:42.949327 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:15:42.949308 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/cff06885-ae10-4a06-ae3d-d5db99244d99-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-af8244-55d6pxhs9\" (UID: \"cff06885-ae10-4a06-ae3d-d5db99244d99\") " pod="ai-tenant-e2e-extmod-b-af8244/payload-processing-external-model-e2e-extmod-b-af8244-55d6pxhs9" Nov 03 01:15:42.949924 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:15:42.949886 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-574qq\" (UniqueName: \"kubernetes.io/projected/cff06885-ae10-4a06-ae3d-d5db99244d99-kube-api-access-574qq\") pod \"payload-processing-external-model-e2e-extmod-b-af8244-55d6pxhs9\" (UID: \"cff06885-ae10-4a06-ae3d-d5db99244d99\") " pod="ai-tenant-e2e-extmod-b-af8244/payload-processing-external-model-e2e-extmod-b-af8244-55d6pxhs9" Nov 03 01:15:42.950347 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:15:42.950319 4939 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/05835a80-2f35-4b67-b578-42e64bfafb9f-maas-api-service-ca\") on node \"ip-10-0-140-107.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:15:42.950482 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:15:42.950354 4939 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4197807b-0461-4733-ac86-01e34bf5a8ec-maas-api-service-ca\") on node \"ip-10-0-140-107.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:15:42.950482 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:15:42.950372 4939 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xwdf4\" (UniqueName: \"kubernetes.io/projected/05835a80-2f35-4b67-b578-42e64bfafb9f-kube-api-access-xwdf4\") on node \"ip-10-0-140-107.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:15:42.951066 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:15:42.951045 4939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4197807b-0461-4733-ac86-01e34bf5a8ec-kube-api-access-5rgfc" pod "4197807b-0461-4733-ac86-01e34bf5a8ec" (UID: "4197807b-0461-4733-ac86-01e34bf5a8ec"). InnerVolumeSpecName "kube-api-access-5rgfc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 01:15:43.052409 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:15:43.052332 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/cff06885-ae10-4a06-ae3d-d5db99244d99-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-af8244-55d6pxhs9\" (UID: \"cff06885-ae10-4a06-ae3d-d5db99244d99\") " pod="ai-tenant-e2e-extmod-b-af8244/payload-processing-external-model-e2e-extmod-b-af8244-55d6pxhs9" Nov 03 01:15:43.052560 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:15:43.052431 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-574qq\" (UniqueName: \"kubernetes.io/projected/cff06885-ae10-4a06-ae3d-d5db99244d99-kube-api-access-574qq\") pod \"payload-processing-external-model-e2e-extmod-b-af8244-55d6pxhs9\" (UID: \"cff06885-ae10-4a06-ae3d-d5db99244d99\") " pod="ai-tenant-e2e-extmod-b-af8244/payload-processing-external-model-e2e-extmod-b-af8244-55d6pxhs9" Nov 03 01:15:43.052674 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:15:43.052655 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cff06885-ae10-4a06-ae3d-d5db99244d99-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-af8244-55d6pxhs9\" (UID: \"cff06885-ae10-4a06-ae3d-d5db99244d99\") " pod="ai-tenant-e2e-extmod-b-af8244/payload-processing-external-model-e2e-extmod-b-af8244-55d6pxhs9" Nov 03 01:15:43.052747 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:15:43.052727 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/cff06885-ae10-4a06-ae3d-d5db99244d99-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-af8244-55d6pxhs9\" (UID: \"cff06885-ae10-4a06-ae3d-d5db99244d99\") " pod="ai-tenant-e2e-extmod-b-af8244/payload-processing-external-model-e2e-extmod-b-af8244-55d6pxhs9" Nov 03 01:15:43.052952 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:15:43.052901 4939 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5rgfc\" (UniqueName: \"kubernetes.io/projected/4197807b-0461-4733-ac86-01e34bf5a8ec-kube-api-access-5rgfc\") on node \"ip-10-0-140-107.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:15:43.053263 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:15:43.053232 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/cff06885-ae10-4a06-ae3d-d5db99244d99-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-af8244-55d6pxhs9\" (UID: \"cff06885-ae10-4a06-ae3d-d5db99244d99\") " pod="ai-tenant-e2e-extmod-b-af8244/payload-processing-external-model-e2e-extmod-b-af8244-55d6pxhs9" Nov 03 01:15:43.053516 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:15:43.053491 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cff06885-ae10-4a06-ae3d-d5db99244d99-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-af8244-55d6pxhs9\" (UID: \"cff06885-ae10-4a06-ae3d-d5db99244d99\") " pod="ai-tenant-e2e-extmod-b-af8244/payload-processing-external-model-e2e-extmod-b-af8244-55d6pxhs9" Nov 03 01:15:43.055422 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:15:43.055402 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/cff06885-ae10-4a06-ae3d-d5db99244d99-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-af8244-55d6pxhs9\" (UID: \"cff06885-ae10-4a06-ae3d-d5db99244d99\") " pod="ai-tenant-e2e-extmod-b-af8244/payload-processing-external-model-e2e-extmod-b-af8244-55d6pxhs9" Nov 03 01:15:43.062768 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:15:43.062745 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-574qq\" (UniqueName: \"kubernetes.io/projected/cff06885-ae10-4a06-ae3d-d5db99244d99-kube-api-access-574qq\") pod \"payload-processing-external-model-e2e-extmod-b-af8244-55d6pxhs9\" (UID: \"cff06885-ae10-4a06-ae3d-d5db99244d99\") " pod="ai-tenant-e2e-extmod-b-af8244/payload-processing-external-model-e2e-extmod-b-af8244-55d6pxhs9" Nov 03 01:15:43.086277 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:15:43.086256 4939 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-af8244/payload-processing-external-model-e2e-extmod-b-af8244-55d6pxhs9" Nov 03 01:15:43.100037 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:15:43.100012 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:15:43.261315 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:15:43.261282 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:15:43.452932 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:15:43.451444 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:15:43.643317 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:15:43.641802 4939 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-af8244/payload-processing-external-model-e2e-extmod-b-af8244-55d6pxhs9"] Nov 03 01:15:43.706996 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:15:43.689967 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:15:43.953139 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:15:43.953109 4939 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-4af1f7-29894475-l9sxr" Nov 03 01:15:43.957517 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:15:43.957460 4939 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5ba08137-cee0-4cf7-a27a-4c750f9308c4" path="/var/lib/kubelet/pods/5ba08137-cee0-4cf7-a27a-4c750f9308c4/volumes" Nov 03 01:15:43.958171 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:15:43.958151 4939 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7f838def-06ed-4c8a-90d6-43c5a192354c" path="/var/lib/kubelet/pods/7f838def-06ed-4c8a-90d6-43c5a192354c/volumes" Nov 03 01:15:43.958784 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:15:43.958762 4939 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f4d972cb-44ea-4f7d-a548-91751c607176" path="/var/lib/kubelet/pods/f4d972cb-44ea-4f7d-a548-91751c607176/volumes" Nov 03 01:15:43.960143 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:15:43.960118 4939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-4af1f7-29894475-l9sxr" event={"ID":"4197807b-0461-4733-ac86-01e34bf5a8ec","Type":"ContainerDied","Data":"61d14992bc956d57d746f9f20896cbe5b2a51d693438a33308e6443e9d25eb5c"} Nov 03 01:15:43.960212 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:15:43.960181 4939 scope.go:122] "RemoveContainer" containerID="295352015f26fdca56fef6c40f6e3555ab5216e3eb4fc3eac3fbb11abdab60be" Nov 03 01:15:43.962583 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:15:43.962557 4939 generic.go:356] "Generic (PLEG): container finished" podID="d59e2793-c378-4558-9b72-25e596ddc473" containerID="25c87621c0bf3163a94ba3aa7a555843387cdc3a24dc93778bf38f2975019840" exitCode=0 Nov 03 01:15:43.962693 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:15:43.962660 4939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-41953fef-726x2" event={"ID":"d59e2793-c378-4558-9b72-25e596ddc473","Type":"ContainerDied","Data":"25c87621c0bf3163a94ba3aa7a555843387cdc3a24dc93778bf38f2975019840"} Nov 03 01:15:43.964943 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:15:43.964893 4939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-2f9a3a-29894475-9nh4j" event={"ID":"05835a80-2f35-4b67-b578-42e64bfafb9f","Type":"ContainerDied","Data":"75570ed0b4da36896744ad943d5466dfadd04efff27a498fe52b46af7ae2df25"} Nov 03 01:15:43.965255 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:15:43.965240 4939 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-2f9a3a-29894475-9nh4j" Nov 03 01:15:43.968651 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:15:43.968629 4939 generic.go:356] "Generic (PLEG): container finished" podID="b24379ee-39df-43ad-b624-556142c07324" containerID="783c1c257f6a4ed790bd159acf7f3dcfd20ce7e8d098c115225cd820258cba49" exitCode=60 Nov 03 01:15:43.968733 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:15:43.968669 4939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-50e623-29894475-zkk6f" event={"ID":"b24379ee-39df-43ad-b624-556142c07324","Type":"ContainerDied","Data":"783c1c257f6a4ed790bd159acf7f3dcfd20ce7e8d098c115225cd820258cba49"} Nov 03 01:15:43.969107 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:15:43.969085 4939 scope.go:122] "RemoveContainer" containerID="783c1c257f6a4ed790bd159acf7f3dcfd20ce7e8d098c115225cd820258cba49" Nov 03 01:15:43.969428 ip-10-0-140-107 kubenswrapper[4939]: E1103 01:15:43.969405 4939 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w0-50e623-29894475-zkk6f_odh-ai-gateway-infra(b24379ee-39df-43ad-b624-556142c07324)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-50e623-29894475-zkk6f" podUID="b24379ee-39df-43ad-b624-556142c07324" Nov 03 01:15:43.993652 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:15:43.993630 4939 scope.go:122] "RemoveContainer" containerID="aa8f751ec2626121aa3bf197b8bf1c0ff0d026398ebff0e70b8adf6a2c6cd733" Nov 03 01:15:44.010030 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:15:44.010008 4939 scope.go:122] "RemoveContainer" containerID="57b40bdbe70de0ed33741ab2458be0f063b9d383c5922cf5815c133638319fbd" Nov 03 01:15:44.031492 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:15:44.031455 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:15:44.054401 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:15:44.054364 4939 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-4af1f7-29894475-l9sxr"] Nov 03 01:15:44.071173 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:15:44.071127 4939 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-4af1f7-29894475-l9sxr"] Nov 03 01:15:44.079792 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:15:44.079765 4939 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-2f9a3a-29894475-9nh4j"] Nov 03 01:15:44.090580 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:15:44.090550 4939 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-2f9a3a-29894475-9nh4j"] Nov 03 01:15:44.289637 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:15:44.289554 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:15:44.515524 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:15:44.515487 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:15:44.775960 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:15:44.768202 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:15:44.981770 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:15:44.981730 4939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-af8244/payload-processing-external-model-e2e-extmod-b-af8244-55d6pxhs9" event={"ID":"cff06885-ae10-4a06-ae3d-d5db99244d99","Type":"ContainerStarted","Data":"9b151bd3619d13efa5ebce64287bb1d1ee2ec9b1b63ea670fe2f6d231275175e"} Nov 03 01:15:44.981975 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:15:44.981797 4939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-af8244/payload-processing-external-model-e2e-extmod-b-af8244-55d6pxhs9" event={"ID":"cff06885-ae10-4a06-ae3d-d5db99244d99","Type":"ContainerStarted","Data":"061591ddb0dda47fd787323f2149bbe99892b0dac9d4fa249118e34bbcf3b651"} Nov 03 01:15:44.981975 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:15:44.981904 4939 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-b-af8244/payload-processing-external-model-e2e-extmod-b-af8244-55d6pxhs9" Nov 03 01:15:44.983886 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:15:44.983862 4939 scope.go:122] "RemoveContainer" containerID="783c1c257f6a4ed790bd159acf7f3dcfd20ce7e8d098c115225cd820258cba49" Nov 03 01:15:44.984331 ip-10-0-140-107 kubenswrapper[4939]: E1103 01:15:44.984304 4939 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w0-50e623-29894475-zkk6f_odh-ai-gateway-infra(b24379ee-39df-43ad-b624-556142c07324)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-50e623-29894475-zkk6f" podUID="b24379ee-39df-43ad-b624-556142c07324" Nov 03 01:15:45.012816 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:15:45.012764 4939 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-b-af8244/payload-processing-external-model-e2e-extmod-b-af8244-55d6pxhs9" podStartSLOduration=17.012750577 podStartE2EDuration="17.012750577s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 01:15:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 01:15:45.007601177 +0000 UTC m=+2093.271936123" watchObservedRunningTime="2026-11-03 01:15:45.012750577 +0000 UTC m=+2093.277085541" Nov 03 01:15:45.183979 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:15:45.183952 4939 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-41953fef-726x2" Nov 03 01:15:45.228235 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:15:45.228200 4939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d59e2793-c378-4558-9b72-25e596ddc473-kube-api-access-cvx2f\" (UniqueName: \"kubernetes.io/projected/d59e2793-c378-4558-9b72-25e596ddc473-kube-api-access-cvx2f\") pod \"d59e2793-c378-4558-9b72-25e596ddc473\" (UID: \"d59e2793-c378-4558-9b72-25e596ddc473\") " Nov 03 01:15:45.228436 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:15:45.228348 4939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d59e2793-c378-4558-9b72-25e596ddc473-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d59e2793-c378-4558-9b72-25e596ddc473-maas-api-openshift-ca\") pod \"d59e2793-c378-4558-9b72-25e596ddc473\" (UID: \"d59e2793-c378-4558-9b72-25e596ddc473\") " Nov 03 01:15:45.228802 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:15:45.228773 4939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d59e2793-c378-4558-9b72-25e596ddc473-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d59e2793-c378-4558-9b72-25e596ddc473-maas-api-xks-ca\") pod \"d59e2793-c378-4558-9b72-25e596ddc473\" (UID: \"d59e2793-c378-4558-9b72-25e596ddc473\") " Nov 03 01:15:45.228969 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:15:45.228898 4939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d59e2793-c378-4558-9b72-25e596ddc473-maas-api-openshift-ca" pod "d59e2793-c378-4558-9b72-25e596ddc473" (UID: "d59e2793-c378-4558-9b72-25e596ddc473"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 01:15:45.229732 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:15:45.229703 4939 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d59e2793-c378-4558-9b72-25e596ddc473-maas-api-openshift-ca\") on node \"ip-10-0-140-107.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:15:45.231597 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:15:45.231573 4939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d59e2793-c378-4558-9b72-25e596ddc473-maas-api-xks-ca" pod "d59e2793-c378-4558-9b72-25e596ddc473" (UID: "d59e2793-c378-4558-9b72-25e596ddc473"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 01:15:45.231744 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:15:45.231683 4939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d59e2793-c378-4558-9b72-25e596ddc473-kube-api-access-cvx2f" pod "d59e2793-c378-4558-9b72-25e596ddc473" (UID: "d59e2793-c378-4558-9b72-25e596ddc473"). InnerVolumeSpecName "kube-api-access-cvx2f". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 01:15:45.331955 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:15:45.331930 4939 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d59e2793-c378-4558-9b72-25e596ddc473-maas-api-xks-ca\") on node \"ip-10-0-140-107.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:15:45.332115 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:15:45.331964 4939 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cvx2f\" (UniqueName: \"kubernetes.io/projected/d59e2793-c378-4558-9b72-25e596ddc473-kube-api-access-cvx2f\") on node \"ip-10-0-140-107.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:15:45.706828 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:15:45.706689 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:15:45.872719 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:15:45.872685 4939 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="05835a80-2f35-4b67-b578-42e64bfafb9f" path="/var/lib/kubelet/pods/05835a80-2f35-4b67-b578-42e64bfafb9f/volumes" Nov 03 01:15:45.873351 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:15:45.873331 4939 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4197807b-0461-4733-ac86-01e34bf5a8ec" path="/var/lib/kubelet/pods/4197807b-0461-4733-ac86-01e34bf5a8ec/volumes" Nov 03 01:15:45.996161 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:15:45.996088 4939 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-41953fef-726x2" Nov 03 01:15:45.996161 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:15:45.996126 4939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-41953fef-726x2" event={"ID":"d59e2793-c378-4558-9b72-25e596ddc473","Type":"ContainerDied","Data":"3baa032e04c238b15c7adf3422d186853abb0a5397704340a3f6763c63765fa9"} Nov 03 01:15:45.996336 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:15:45.996182 4939 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="3baa032e04c238b15c7adf3422d186853abb0a5397704340a3f6763c63765fa9" Nov 03 01:15:46.488732 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:15:46.488694 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:15:47.124990 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:15:47.124959 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:15:47.795072 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:15:47.795039 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:15:48.035158 ip-10-0-140-107 kubenswrapper[4939]: E1103 01:15:48.035121 4939 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-sthjh" containerName="extractor" Nov 03 01:15:48.670292 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:15:48.664622 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:15:49.397623 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:15:49.397589 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:15:50.098593 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:15:50.098560 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:15:50.977193 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:15:50.977158 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:15:51.518013 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:15:51.517977 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:15:51.637933 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:15:51.637736 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:15:52.014930 ip-10-0-140-107 kubenswrapper[4939]: E1103 01:15:52.009582 4939 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-b-57079b/agc-cred-83631bdec3fc: secret "agc-cred-83631bdec3fc" not found Nov 03 01:15:52.014930 ip-10-0-140-107 kubenswrapper[4939]: E1103 01:15:52.009614 4939 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-b-57079b/agc-cred-e705148022ae: secret "agc-cred-e705148022ae" not found Nov 03 01:15:52.014930 ip-10-0-140-107 kubenswrapper[4939]: E1103 01:15:52.009629 4939 projected.go:196] Error preparing data for projected volume provider-credentials for pod ai-tenant-e2e-extmod-b-57079b/payload-processing-external-model-e2e-extmod-b-57079b-59f8rtzcd: [secret "agc-cred-83631bdec3fc" not found, secret "agc-cred-e705148022ae" not found] Nov 03 01:15:52.014930 ip-10-0-140-107 kubenswrapper[4939]: E1103 01:15:52.009699 4939 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/c54b89b6-6593-4f97-9cf9-21e7d21880b6-provider-credentials podName:c54b89b6-6593-4f97-9cf9-21e7d21880b6 nodeName:}" failed. No retries permitted until 2026-11-03 01:15:52.509679987 +0000 UTC m=+2100.774014923 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/c54b89b6-6593-4f97-9cf9-21e7d21880b6-provider-credentials") pod "payload-processing-external-model-e2e-extmod-b-57079b-59f8rtzcd" (UID: "c54b89b6-6593-4f97-9cf9-21e7d21880b6") : [secret "agc-cred-83631bdec3fc" not found, secret "agc-cred-e705148022ae" not found] Nov 03 01:15:52.463946 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:15:52.463915 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:15:52.553937 ip-10-0-140-107 kubenswrapper[4939]: E1103 01:15:52.553878 4939 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-b-57079b/agc-cred-83631bdec3fc: secret "agc-cred-83631bdec3fc" not found Nov 03 01:15:52.554523 ip-10-0-140-107 kubenswrapper[4939]: E1103 01:15:52.554504 4939 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-b-57079b/agc-cred-e705148022ae: secret "agc-cred-e705148022ae" not found Nov 03 01:15:52.554649 ip-10-0-140-107 kubenswrapper[4939]: E1103 01:15:52.554636 4939 projected.go:196] Error preparing data for projected volume provider-credentials for pod ai-tenant-e2e-extmod-b-57079b/payload-processing-external-model-e2e-extmod-b-57079b-59f8rtzcd: [secret "agc-cred-83631bdec3fc" not found, secret "agc-cred-e705148022ae" not found] Nov 03 01:15:52.554820 ip-10-0-140-107 kubenswrapper[4939]: E1103 01:15:52.554808 4939 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/c54b89b6-6593-4f97-9cf9-21e7d21880b6-provider-credentials podName:c54b89b6-6593-4f97-9cf9-21e7d21880b6 nodeName:}" failed. No retries permitted until 2026-11-03 01:15:53.55478476 +0000 UTC m=+2101.819119702 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/c54b89b6-6593-4f97-9cf9-21e7d21880b6-provider-credentials") pod "payload-processing-external-model-e2e-extmod-b-57079b-59f8rtzcd" (UID: "c54b89b6-6593-4f97-9cf9-21e7d21880b6") : [secret "agc-cred-83631bdec3fc" not found, secret "agc-cred-e705148022ae" not found] Nov 03 01:15:52.692978 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:15:52.692947 4939 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-n42s6_65d134ac-8a25-4c67-b6c7-cd752a098daf/console-operator/2.log" Nov 03 01:15:52.742204 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:15:52.742131 4939 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-n42s6_65d134ac-8a25-4c67-b6c7-cd752a098daf/console-operator/2.log" Nov 03 01:15:53.311509 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:15:53.311473 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:15:53.576580 ip-10-0-140-107 kubenswrapper[4939]: E1103 01:15:53.576501 4939 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-b-57079b/agc-cred-83631bdec3fc: secret "agc-cred-83631bdec3fc" not found Nov 03 01:15:53.576580 ip-10-0-140-107 kubenswrapper[4939]: E1103 01:15:53.576527 4939 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-b-57079b/agc-cred-e705148022ae: secret "agc-cred-e705148022ae" not found Nov 03 01:15:53.576580 ip-10-0-140-107 kubenswrapper[4939]: E1103 01:15:53.576538 4939 projected.go:196] Error preparing data for projected volume provider-credentials for pod ai-tenant-e2e-extmod-b-57079b/payload-processing-external-model-e2e-extmod-b-57079b-59f8rtzcd: [secret "agc-cred-83631bdec3fc" not found, secret "agc-cred-e705148022ae" not found] Nov 03 01:15:53.577071 ip-10-0-140-107 kubenswrapper[4939]: E1103 01:15:53.576595 4939 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/c54b89b6-6593-4f97-9cf9-21e7d21880b6-provider-credentials podName:c54b89b6-6593-4f97-9cf9-21e7d21880b6 nodeName:}" failed. No retries permitted until 2026-11-03 01:15:55.576581332 +0000 UTC m=+2103.840916256 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/c54b89b6-6593-4f97-9cf9-21e7d21880b6-provider-credentials") pod "payload-processing-external-model-e2e-extmod-b-57079b-59f8rtzcd" (UID: "c54b89b6-6593-4f97-9cf9-21e7d21880b6") : [secret "agc-cred-83631bdec3fc" not found, secret "agc-cred-e705148022ae" not found] Nov 03 01:15:54.028864 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:15:54.028825 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:15:54.712300 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:15:54.712261 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:15:55.400181 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:15:55.400145 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:15:55.641030 ip-10-0-140-107 kubenswrapper[4939]: E1103 01:15:55.640995 4939 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-b-57079b/agc-cred-83631bdec3fc: secret "agc-cred-83631bdec3fc" not found Nov 03 01:15:55.641030 ip-10-0-140-107 kubenswrapper[4939]: E1103 01:15:55.641027 4939 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-b-57079b/agc-cred-e705148022ae: secret "agc-cred-e705148022ae" not found Nov 03 01:15:55.641030 ip-10-0-140-107 kubenswrapper[4939]: E1103 01:15:55.641039 4939 projected.go:196] Error preparing data for projected volume provider-credentials for pod ai-tenant-e2e-extmod-b-57079b/payload-processing-external-model-e2e-extmod-b-57079b-59f8rtzcd: [secret "agc-cred-83631bdec3fc" not found, secret "agc-cred-e705148022ae" not found] Nov 03 01:15:55.641280 ip-10-0-140-107 kubenswrapper[4939]: E1103 01:15:55.641108 4939 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/c54b89b6-6593-4f97-9cf9-21e7d21880b6-provider-credentials podName:c54b89b6-6593-4f97-9cf9-21e7d21880b6 nodeName:}" failed. No retries permitted until 2026-11-03 01:15:59.641088283 +0000 UTC m=+2107.905423226 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/c54b89b6-6593-4f97-9cf9-21e7d21880b6-provider-credentials") pod "payload-processing-external-model-e2e-extmod-b-57079b-59f8rtzcd" (UID: "c54b89b6-6593-4f97-9cf9-21e7d21880b6") : [secret "agc-cred-83631bdec3fc" not found, secret "agc-cred-e705148022ae" not found] Nov 03 01:15:55.861443 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:15:55.861407 4939 scope.go:122] "RemoveContainer" containerID="783c1c257f6a4ed790bd159acf7f3dcfd20ce7e8d098c115225cd820258cba49" Nov 03 01:15:56.000584 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:15:56.000550 4939 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-b-af8244/payload-processing-external-model-e2e-extmod-b-af8244-55d6pxhs9" Nov 03 01:15:56.054783 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:15:56.054747 4939 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-af8244/payload-processing-external-model-e2e-extmod-b-af8244-7b67bdr62"] Nov 03 01:15:56.245129 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:15:56.225101 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:15:56.809086 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:15:56.809055 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:15:57.279135 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:15:57.279104 4939 generic.go:356] "Generic (PLEG): container finished" podID="b24379ee-39df-43ad-b624-556142c07324" containerID="5a0b17ce5c48ec030c5101b4dc5061f94ec5a2c2638b7cbb7c626427a5a8d5d3" exitCode=60 Nov 03 01:15:57.279594 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:15:57.279175 4939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-50e623-29894475-zkk6f" event={"ID":"b24379ee-39df-43ad-b624-556142c07324","Type":"ContainerDied","Data":"5a0b17ce5c48ec030c5101b4dc5061f94ec5a2c2638b7cbb7c626427a5a8d5d3"} Nov 03 01:15:57.279594 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:15:57.279259 4939 scope.go:122] "RemoveContainer" containerID="783c1c257f6a4ed790bd159acf7f3dcfd20ce7e8d098c115225cd820258cba49" Nov 03 01:15:57.279733 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:15:57.279675 4939 scope.go:122] "RemoveContainer" containerID="5a0b17ce5c48ec030c5101b4dc5061f94ec5a2c2638b7cbb7c626427a5a8d5d3" Nov 03 01:15:57.280108 ip-10-0-140-107 kubenswrapper[4939]: E1103 01:15:57.280084 4939 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w0-50e623-29894475-zkk6f_odh-ai-gateway-infra(b24379ee-39df-43ad-b624-556142c07324)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-50e623-29894475-zkk6f" podUID="b24379ee-39df-43ad-b624-556142c07324" Nov 03 01:15:57.490779 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:15:57.490746 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:15:58.191523 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:15:58.191492 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:15:58.324979 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:15:58.323694 4939 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-50e623-29894475-zkk6f"] Nov 03 01:15:58.534763 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:15:58.534736 4939 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-50e623-29894475-zkk6f" Nov 03 01:15:58.714648 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:15:58.714612 4939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b24379ee-39df-43ad-b624-556142c07324-kube-api-access-r7hrf\" (UniqueName: \"kubernetes.io/projected/b24379ee-39df-43ad-b624-556142c07324-kube-api-access-r7hrf\") pod \"b24379ee-39df-43ad-b624-556142c07324\" (UID: \"b24379ee-39df-43ad-b624-556142c07324\") " Nov 03 01:15:58.714844 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:15:58.714830 4939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b24379ee-39df-43ad-b624-556142c07324-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b24379ee-39df-43ad-b624-556142c07324-maas-api-service-ca\") pod \"b24379ee-39df-43ad-b624-556142c07324\" (UID: \"b24379ee-39df-43ad-b624-556142c07324\") " Nov 03 01:15:58.715238 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:15:58.715204 4939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b24379ee-39df-43ad-b624-556142c07324-maas-api-service-ca" pod "b24379ee-39df-43ad-b624-556142c07324" (UID: "b24379ee-39df-43ad-b624-556142c07324"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 01:15:58.715640 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:15:58.715624 4939 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b24379ee-39df-43ad-b624-556142c07324-maas-api-service-ca\") on node \"ip-10-0-140-107.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:15:58.716858 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:15:58.716839 4939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b24379ee-39df-43ad-b624-556142c07324-kube-api-access-r7hrf" pod "b24379ee-39df-43ad-b624-556142c07324" (UID: "b24379ee-39df-43ad-b624-556142c07324"). InnerVolumeSpecName "kube-api-access-r7hrf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 01:15:58.818277 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:15:58.818194 4939 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-r7hrf\" (UniqueName: \"kubernetes.io/projected/b24379ee-39df-43ad-b624-556142c07324-kube-api-access-r7hrf\") on node \"ip-10-0-140-107.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:15:58.995074 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:15:58.995041 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:15:59.341783 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:15:59.341748 4939 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-50e623-29894475-zkk6f" Nov 03 01:15:59.342322 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:15:59.341791 4939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-50e623-29894475-zkk6f" event={"ID":"b24379ee-39df-43ad-b624-556142c07324","Type":"ContainerDied","Data":"a120a9d160fc916b71b5f4cd50338c56f5703686ccbb748c6bdedfc02738c62c"} Nov 03 01:15:59.342322 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:15:59.341883 4939 scope.go:122] "RemoveContainer" containerID="5a0b17ce5c48ec030c5101b4dc5061f94ec5a2c2638b7cbb7c626427a5a8d5d3" Nov 03 01:15:59.367895 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:15:59.367870 4939 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-50e623-29894475-zkk6f"] Nov 03 01:15:59.378626 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:15:59.378579 4939 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-50e623-29894475-zkk6f"] Nov 03 01:15:59.644224 ip-10-0-140-107 kubenswrapper[4939]: E1103 01:15:59.644132 4939 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-b-57079b/agc-cred-83631bdec3fc: secret "agc-cred-83631bdec3fc" not found Nov 03 01:15:59.644224 ip-10-0-140-107 kubenswrapper[4939]: E1103 01:15:59.644165 4939 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-b-57079b/agc-cred-e705148022ae: secret "agc-cred-e705148022ae" not found Nov 03 01:15:59.644224 ip-10-0-140-107 kubenswrapper[4939]: E1103 01:15:59.644177 4939 projected.go:196] Error preparing data for projected volume provider-credentials for pod ai-tenant-e2e-extmod-b-57079b/payload-processing-external-model-e2e-extmod-b-57079b-59f8rtzcd: [secret "agc-cred-83631bdec3fc" not found, secret "agc-cred-e705148022ae" not found] Nov 03 01:15:59.644541 ip-10-0-140-107 kubenswrapper[4939]: E1103 01:15:59.644247 4939 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/c54b89b6-6593-4f97-9cf9-21e7d21880b6-provider-credentials podName:c54b89b6-6593-4f97-9cf9-21e7d21880b6 nodeName:}" failed. No retries permitted until 2026-11-03 01:16:07.644228223 +0000 UTC m=+2115.908563153 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/c54b89b6-6593-4f97-9cf9-21e7d21880b6-provider-credentials") pod "payload-processing-external-model-e2e-extmod-b-57079b-59f8rtzcd" (UID: "c54b89b6-6593-4f97-9cf9-21e7d21880b6") : [secret "agc-cred-83631bdec3fc" not found, secret "agc-cred-e705148022ae" not found] Nov 03 01:15:59.676208 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:15:59.676173 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:15:59.872396 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:15:59.872358 4939 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b24379ee-39df-43ad-b624-556142c07324" path="/var/lib/kubelet/pods/b24379ee-39df-43ad-b624-556142c07324/volumes" Nov 03 01:16:00.289369 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:16:00.289337 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:16:00.636078 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:16:00.636041 4939 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-4pw7v"] Nov 03 01:16:01.055592 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:16:01.055452 4939 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-b-af8244/payload-processing-external-model-e2e-extmod-b-af8244-7b67bdr62" podUID="b83ac5aa-8a33-487b-b97f-e1d45d2553cb" containerName="payload-processing" containerID="cri-o://4a7b14c23b33892e69ba0826b2be1d47d0df377780444b03f9a24c37187f9f91" gracePeriod=25 Nov 03 01:16:01.221124 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:16:01.221088 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:16:01.423036 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:16:01.423005 4939 generic.go:356] "Generic (PLEG): container finished" podID="b83ac5aa-8a33-487b-b97f-e1d45d2553cb" containerID="4a7b14c23b33892e69ba0826b2be1d47d0df377780444b03f9a24c37187f9f91" exitCode=0 Nov 03 01:16:01.423202 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:16:01.423087 4939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-af8244/payload-processing-external-model-e2e-extmod-b-af8244-7b67bdr62" event={"ID":"b83ac5aa-8a33-487b-b97f-e1d45d2553cb","Type":"ContainerDied","Data":"4a7b14c23b33892e69ba0826b2be1d47d0df377780444b03f9a24c37187f9f91"} Nov 03 01:16:01.467052 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:16:01.467025 4939 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-af8244/payload-processing-external-model-e2e-extmod-b-af8244-7b67bdr62" Nov 03 01:16:01.531816 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:16:01.531774 4939 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-a-d25759/payload-processing-external-model-e2e-extmod-a-d25759-5d4fvt4mj"] Nov 03 01:16:01.532746 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:16:01.532724 4939 cpu_manager.go:414] "containerMap: removing container" podUID="b24379ee-39df-43ad-b624-556142c07324" containerName="cleanup" Nov 03 01:16:01.532746 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:16:01.532742 4939 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b24379ee-39df-43ad-b624-556142c07324" containerName="cleanup" Nov 03 01:16:01.532977 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:16:01.532771 4939 cpu_manager.go:414] "containerMap: removing container" podUID="d59e2793-c378-4558-9b72-25e596ddc473" containerName="revoke-keys" Nov 03 01:16:01.532977 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:16:01.532776 4939 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d59e2793-c378-4558-9b72-25e596ddc473" containerName="revoke-keys" Nov 03 01:16:01.532977 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:16:01.532791 4939 cpu_manager.go:414] "containerMap: removing container" podUID="b24379ee-39df-43ad-b624-556142c07324" containerName="cleanup" Nov 03 01:16:01.532977 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:16:01.532796 4939 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b24379ee-39df-43ad-b624-556142c07324" containerName="cleanup" Nov 03 01:16:01.532977 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:16:01.532811 4939 cpu_manager.go:414] "containerMap: removing container" podUID="b83ac5aa-8a33-487b-b97f-e1d45d2553cb" containerName="payload-processing" Nov 03 01:16:01.532977 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:16:01.532816 4939 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b83ac5aa-8a33-487b-b97f-e1d45d2553cb" containerName="payload-processing" Nov 03 01:16:01.532977 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:16:01.532829 4939 cpu_manager.go:414] "containerMap: removing container" podUID="5ba08137-cee0-4cf7-a27a-4c750f9308c4" containerName="cleanup" Nov 03 01:16:01.532977 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:16:01.532834 4939 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5ba08137-cee0-4cf7-a27a-4c750f9308c4" containerName="cleanup" Nov 03 01:16:01.532977 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:16:01.532852 4939 cpu_manager.go:414] "containerMap: removing container" podUID="4197807b-0461-4733-ac86-01e34bf5a8ec" containerName="cleanup" Nov 03 01:16:01.532977 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:16:01.532856 4939 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4197807b-0461-4733-ac86-01e34bf5a8ec" containerName="cleanup" Nov 03 01:16:01.533505 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:16:01.533001 4939 memory_manager.go:365] "RemoveStaleState removing state" podUID="d59e2793-c378-4558-9b72-25e596ddc473" containerName="revoke-keys" Nov 03 01:16:01.533505 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:16:01.533014 4939 memory_manager.go:365] "RemoveStaleState removing state" podUID="f4d972cb-44ea-4f7d-a548-91751c607176" containerName="cleanup" Nov 03 01:16:01.533505 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:16:01.533027 4939 memory_manager.go:365] "RemoveStaleState removing state" podUID="b24379ee-39df-43ad-b624-556142c07324" containerName="cleanup" Nov 03 01:16:01.533505 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:16:01.533035 4939 memory_manager.go:365] "RemoveStaleState removing state" podUID="b83ac5aa-8a33-487b-b97f-e1d45d2553cb" containerName="payload-processing" Nov 03 01:16:01.533505 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:16:01.533042 4939 memory_manager.go:365] "RemoveStaleState removing state" podUID="7f838def-06ed-4c8a-90d6-43c5a192354c" containerName="cleanup" Nov 03 01:16:01.533505 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:16:01.533049 4939 memory_manager.go:365] "RemoveStaleState removing state" podUID="b24379ee-39df-43ad-b624-556142c07324" containerName="cleanup" Nov 03 01:16:01.535955 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:16:01.535891 4939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b83ac5aa-8a33-487b-b97f-e1d45d2553cb-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/b83ac5aa-8a33-487b-b97f-e1d45d2553cb-routing-overlay-volume\") pod \"b83ac5aa-8a33-487b-b97f-e1d45d2553cb\" (UID: \"b83ac5aa-8a33-487b-b97f-e1d45d2553cb\") " Nov 03 01:16:01.536164 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:16:01.536148 4939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b83ac5aa-8a33-487b-b97f-e1d45d2553cb-provider-credentials\" (UniqueName: \"kubernetes.io/projected/b83ac5aa-8a33-487b-b97f-e1d45d2553cb-provider-credentials\") pod \"b83ac5aa-8a33-487b-b97f-e1d45d2553cb\" (UID: \"b83ac5aa-8a33-487b-b97f-e1d45d2553cb\") " Nov 03 01:16:01.536298 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:16:01.536158 4939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b83ac5aa-8a33-487b-b97f-e1d45d2553cb-routing-overlay-volume" pod "b83ac5aa-8a33-487b-b97f-e1d45d2553cb" (UID: "b83ac5aa-8a33-487b-b97f-e1d45d2553cb"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 01:16:01.536298 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:16:01.536236 4939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b83ac5aa-8a33-487b-b97f-e1d45d2553cb-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b83ac5aa-8a33-487b-b97f-e1d45d2553cb-plugins-config-volume\") pod \"b83ac5aa-8a33-487b-b97f-e1d45d2553cb\" (UID: \"b83ac5aa-8a33-487b-b97f-e1d45d2553cb\") " Nov 03 01:16:01.536417 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:16:01.536331 4939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b83ac5aa-8a33-487b-b97f-e1d45d2553cb-kube-api-access-xn26c\" (UniqueName: \"kubernetes.io/projected/b83ac5aa-8a33-487b-b97f-e1d45d2553cb-kube-api-access-xn26c\") pod \"b83ac5aa-8a33-487b-b97f-e1d45d2553cb\" (UID: \"b83ac5aa-8a33-487b-b97f-e1d45d2553cb\") " Nov 03 01:16:01.536758 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:16:01.536731 4939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b83ac5aa-8a33-487b-b97f-e1d45d2553cb-plugins-config-volume" pod "b83ac5aa-8a33-487b-b97f-e1d45d2553cb" (UID: "b83ac5aa-8a33-487b-b97f-e1d45d2553cb"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 01:16:01.537251 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:16:01.537226 4939 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b83ac5aa-8a33-487b-b97f-e1d45d2553cb-plugins-config-volume\") on node \"ip-10-0-140-107.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:16:01.537343 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:16:01.537258 4939 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/b83ac5aa-8a33-487b-b97f-e1d45d2553cb-routing-overlay-volume\") on node \"ip-10-0-140-107.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:16:01.537453 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:16:01.537434 4939 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-d25759/payload-processing-external-model-e2e-extmod-a-d25759-5d4fvt4mj" Nov 03 01:16:01.538942 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:16:01.538877 4939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b83ac5aa-8a33-487b-b97f-e1d45d2553cb-kube-api-access-xn26c" pod "b83ac5aa-8a33-487b-b97f-e1d45d2553cb" (UID: "b83ac5aa-8a33-487b-b97f-e1d45d2553cb"). InnerVolumeSpecName "kube-api-access-xn26c". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 01:16:01.539054 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:16:01.539036 4939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b83ac5aa-8a33-487b-b97f-e1d45d2553cb-provider-credentials" pod "b83ac5aa-8a33-487b-b97f-e1d45d2553cb" (UID: "b83ac5aa-8a33-487b-b97f-e1d45d2553cb"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 01:16:01.576585 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:16:01.573194 4939 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-d25759\"/\"e2e-cfl-e257d5-provider-cred\"" Nov 03 01:16:01.576585 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:16:01.573231 4939 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-d25759\"/\"openshift-service-ca.crt\"" Nov 03 01:16:01.576585 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:16:01.573310 4939 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-d25759\"/\"routing-overlay\"" Nov 03 01:16:01.576585 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:16:01.573322 4939 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-d25759\"/\"payload-processing-external-model-e2e--4d491765-dockercfg-btrfs\"" Nov 03 01:16:01.576585 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:16:01.573408 4939 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-d25759\"/\"agc-cred-229a2b395ead\"" Nov 03 01:16:01.578143 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:16:01.578114 4939 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-d25759\"/\"kube-root-ca.crt\"" Nov 03 01:16:01.578250 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:16:01.578198 4939 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-d25759\"/\"payload-processing-external-model-plugins-e2e-extmod-a-d25759\"" Nov 03 01:16:01.578957 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:16:01.578930 4939 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-d25759/payload-processing-external-model-e2e-extmod-a-d25759-5d4fvt4mj"] Nov 03 01:16:01.639258 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:16:01.639216 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/4568389e-a543-4ff4-a76c-0142ac5b4053-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-d25759-5d4fvt4mj\" (UID: \"4568389e-a543-4ff4-a76c-0142ac5b4053\") " pod="ai-tenant-e2e-extmod-a-d25759/payload-processing-external-model-e2e-extmod-a-d25759-5d4fvt4mj" Nov 03 01:16:01.639674 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:16:01.639376 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/4568389e-a543-4ff4-a76c-0142ac5b4053-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-d25759-5d4fvt4mj\" (UID: \"4568389e-a543-4ff4-a76c-0142ac5b4053\") " pod="ai-tenant-e2e-extmod-a-d25759/payload-processing-external-model-e2e-extmod-a-d25759-5d4fvt4mj" Nov 03 01:16:01.639674 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:16:01.639540 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5hshg\" (UniqueName: \"kubernetes.io/projected/4568389e-a543-4ff4-a76c-0142ac5b4053-kube-api-access-5hshg\") pod \"payload-processing-external-model-e2e-extmod-a-d25759-5d4fvt4mj\" (UID: \"4568389e-a543-4ff4-a76c-0142ac5b4053\") " pod="ai-tenant-e2e-extmod-a-d25759/payload-processing-external-model-e2e-extmod-a-d25759-5d4fvt4mj" Nov 03 01:16:01.639847 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:16:01.639653 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4568389e-a543-4ff4-a76c-0142ac5b4053-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-d25759-5d4fvt4mj\" (UID: \"4568389e-a543-4ff4-a76c-0142ac5b4053\") " pod="ai-tenant-e2e-extmod-a-d25759/payload-processing-external-model-e2e-extmod-a-d25759-5d4fvt4mj" Nov 03 01:16:01.643940 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:16:01.640525 4939 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/b83ac5aa-8a33-487b-b97f-e1d45d2553cb-provider-credentials\") on node \"ip-10-0-140-107.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:16:01.643940 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:16:01.640560 4939 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xn26c\" (UniqueName: \"kubernetes.io/projected/b83ac5aa-8a33-487b-b97f-e1d45d2553cb-kube-api-access-xn26c\") on node \"ip-10-0-140-107.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:16:01.830120 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:16:01.829635 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/4568389e-a543-4ff4-a76c-0142ac5b4053-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-d25759-5d4fvt4mj\" (UID: \"4568389e-a543-4ff4-a76c-0142ac5b4053\") " pod="ai-tenant-e2e-extmod-a-d25759/payload-processing-external-model-e2e-extmod-a-d25759-5d4fvt4mj" Nov 03 01:16:01.830120 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:16:01.829728 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/4568389e-a543-4ff4-a76c-0142ac5b4053-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-d25759-5d4fvt4mj\" (UID: \"4568389e-a543-4ff4-a76c-0142ac5b4053\") " pod="ai-tenant-e2e-extmod-a-d25759/payload-processing-external-model-e2e-extmod-a-d25759-5d4fvt4mj" Nov 03 01:16:01.830120 ip-10-0-140-107 kubenswrapper[4939]: E1103 01:16:01.829854 4939 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-a-d25759/agc-cred-229a2b395ead: secret "agc-cred-229a2b395ead" not found Nov 03 01:16:01.830120 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:16:01.829876 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5hshg\" (UniqueName: \"kubernetes.io/projected/4568389e-a543-4ff4-a76c-0142ac5b4053-kube-api-access-5hshg\") pod \"payload-processing-external-model-e2e-extmod-a-d25759-5d4fvt4mj\" (UID: \"4568389e-a543-4ff4-a76c-0142ac5b4053\") " pod="ai-tenant-e2e-extmod-a-d25759/payload-processing-external-model-e2e-extmod-a-d25759-5d4fvt4mj" Nov 03 01:16:01.830120 ip-10-0-140-107 kubenswrapper[4939]: E1103 01:16:01.829884 4939 projected.go:196] Error preparing data for projected volume provider-credentials for pod ai-tenant-e2e-extmod-a-d25759/payload-processing-external-model-e2e-extmod-a-d25759-5d4fvt4mj: secret "agc-cred-229a2b395ead" not found Nov 03 01:16:01.830120 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:16:01.829922 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4568389e-a543-4ff4-a76c-0142ac5b4053-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-d25759-5d4fvt4mj\" (UID: \"4568389e-a543-4ff4-a76c-0142ac5b4053\") " pod="ai-tenant-e2e-extmod-a-d25759/payload-processing-external-model-e2e-extmod-a-d25759-5d4fvt4mj" Nov 03 01:16:01.884932 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:16:01.859833 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/4568389e-a543-4ff4-a76c-0142ac5b4053-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-d25759-5d4fvt4mj\" (UID: \"4568389e-a543-4ff4-a76c-0142ac5b4053\") " pod="ai-tenant-e2e-extmod-a-d25759/payload-processing-external-model-e2e-extmod-a-d25759-5d4fvt4mj" Nov 03 01:16:01.925933 ip-10-0-140-107 kubenswrapper[4939]: E1103 01:16:01.898673 4939 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/4568389e-a543-4ff4-a76c-0142ac5b4053-provider-credentials podName:4568389e-a543-4ff4-a76c-0142ac5b4053 nodeName:}" failed. No retries permitted until 2026-11-03 01:16:02.398625975 +0000 UTC m=+2110.662960921 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/4568389e-a543-4ff4-a76c-0142ac5b4053-provider-credentials") pod "payload-processing-external-model-e2e-extmod-a-d25759-5d4fvt4mj" (UID: "4568389e-a543-4ff4-a76c-0142ac5b4053") : secret "agc-cred-229a2b395ead" not found Nov 03 01:16:01.967936 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:16:01.967727 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4568389e-a543-4ff4-a76c-0142ac5b4053-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-d25759-5d4fvt4mj\" (UID: \"4568389e-a543-4ff4-a76c-0142ac5b4053\") " pod="ai-tenant-e2e-extmod-a-d25759/payload-processing-external-model-e2e-extmod-a-d25759-5d4fvt4mj" Nov 03 01:16:02.261802 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:16:02.230620 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5hshg\" (UniqueName: \"kubernetes.io/projected/4568389e-a543-4ff4-a76c-0142ac5b4053-kube-api-access-5hshg\") pod \"payload-processing-external-model-e2e-extmod-a-d25759-5d4fvt4mj\" (UID: \"4568389e-a543-4ff4-a76c-0142ac5b4053\") " pod="ai-tenant-e2e-extmod-a-d25759/payload-processing-external-model-e2e-extmod-a-d25759-5d4fvt4mj" Nov 03 01:16:02.490622 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:16:02.490555 4939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-af8244/payload-processing-external-model-e2e-extmod-b-af8244-7b67bdr62" event={"ID":"b83ac5aa-8a33-487b-b97f-e1d45d2553cb","Type":"ContainerDied","Data":"9f31ae0d806a716b085b6d81fb02872e1a35584ce33eba17334eb00788c44f41"} Nov 03 01:16:02.490947 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:16:02.490876 4939 scope.go:122] "RemoveContainer" containerID="4a7b14c23b33892e69ba0826b2be1d47d0df377780444b03f9a24c37187f9f91" Nov 03 01:16:02.491357 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:16:02.491337 4939 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-af8244/payload-processing-external-model-e2e-extmod-b-af8244-7b67bdr62" Nov 03 01:16:02.507200 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:16:02.507167 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/4568389e-a543-4ff4-a76c-0142ac5b4053-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-d25759-5d4fvt4mj\" (UID: \"4568389e-a543-4ff4-a76c-0142ac5b4053\") " pod="ai-tenant-e2e-extmod-a-d25759/payload-processing-external-model-e2e-extmod-a-d25759-5d4fvt4mj" Nov 03 01:16:02.507926 ip-10-0-140-107 kubenswrapper[4939]: E1103 01:16:02.507886 4939 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-a-d25759/agc-cred-229a2b395ead: secret "agc-cred-229a2b395ead" not found Nov 03 01:16:02.515073 ip-10-0-140-107 kubenswrapper[4939]: E1103 01:16:02.514995 4939 projected.go:196] Error preparing data for projected volume provider-credentials for pod ai-tenant-e2e-extmod-a-d25759/payload-processing-external-model-e2e-extmod-a-d25759-5d4fvt4mj: secret "agc-cred-229a2b395ead" not found Nov 03 01:16:02.515293 ip-10-0-140-107 kubenswrapper[4939]: E1103 01:16:02.515278 4939 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/4568389e-a543-4ff4-a76c-0142ac5b4053-provider-credentials podName:4568389e-a543-4ff4-a76c-0142ac5b4053 nodeName:}" failed. No retries permitted until 2026-11-03 01:16:03.51523909 +0000 UTC m=+2111.779574017 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/4568389e-a543-4ff4-a76c-0142ac5b4053-provider-credentials") pod "payload-processing-external-model-e2e-extmod-a-d25759-5d4fvt4mj" (UID: "4568389e-a543-4ff4-a76c-0142ac5b4053") : secret "agc-cred-229a2b395ead" not found Nov 03 01:16:02.829529 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:16:02.821536 4939 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-af8244/payload-processing-external-model-e2e-extmod-b-af8244-7b67bdr62"] Nov 03 01:16:02.829529 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:16:02.821599 4939 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-b-af8244/payload-processing-external-model-e2e-extmod-b-af8244-7b67bdr62"] Nov 03 01:16:02.885633 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:16:02.885593 4939 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-f7680d/premium-w2-f7680d-kserve-76565b8cf9-hqbbt" podUID="4712e708-18a2-4a13-a12c-a5669397b2c3" containerName="main" probeResult="failure" output="Get \"https://10.132.0.63:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Nov 03 01:16:02.887742 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:16:02.887253 4939 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-2f9a3a/simulator-w0-2f9a3a-kserve-9596bbb8f-2trgz" podUID="207e2207-fb6c-4076-95df-72e95671c261" containerName="main" probeResult="failure" output="Get \"https://10.132.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 01:16:02.982045 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:16:02.979446 4939 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-2f9a3a/distinct-w0-2f9a3a-kserve-555565cfcf-jtcr9" podUID="ac0ece06-8718-463f-915f-5315988d9711" containerName="main" probeResult="failure" output="Get \"https://10.132.0.76:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Nov 03 01:16:03.390258 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:16:03.390223 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:16:03.600685 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:16:03.600643 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/4568389e-a543-4ff4-a76c-0142ac5b4053-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-d25759-5d4fvt4mj\" (UID: \"4568389e-a543-4ff4-a76c-0142ac5b4053\") " pod="ai-tenant-e2e-extmod-a-d25759/payload-processing-external-model-e2e-extmod-a-d25759-5d4fvt4mj" Nov 03 01:16:03.600889 ip-10-0-140-107 kubenswrapper[4939]: E1103 01:16:03.600788 4939 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-a-d25759/agc-cred-229a2b395ead: secret "agc-cred-229a2b395ead" not found Nov 03 01:16:03.600889 ip-10-0-140-107 kubenswrapper[4939]: E1103 01:16:03.600812 4939 projected.go:196] Error preparing data for projected volume provider-credentials for pod ai-tenant-e2e-extmod-a-d25759/payload-processing-external-model-e2e-extmod-a-d25759-5d4fvt4mj: secret "agc-cred-229a2b395ead" not found Nov 03 01:16:03.600889 ip-10-0-140-107 kubenswrapper[4939]: E1103 01:16:03.600869 4939 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/4568389e-a543-4ff4-a76c-0142ac5b4053-provider-credentials podName:4568389e-a543-4ff4-a76c-0142ac5b4053 nodeName:}" failed. No retries permitted until 2026-11-03 01:16:05.600855671 +0000 UTC m=+2113.865190598 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/4568389e-a543-4ff4-a76c-0142ac5b4053-provider-credentials") pod "payload-processing-external-model-e2e-extmod-a-d25759-5d4fvt4mj" (UID: "4568389e-a543-4ff4-a76c-0142ac5b4053") : secret "agc-cred-229a2b395ead" not found Nov 03 01:16:03.881026 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:16:03.880984 4939 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b83ac5aa-8a33-487b-b97f-e1d45d2553cb" path="/var/lib/kubelet/pods/b83ac5aa-8a33-487b-b97f-e1d45d2553cb/volumes" Nov 03 01:16:03.990219 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:16:03.990186 4939 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-57079b/payload-processing-external-model-e2e-extmod-b-57079b-59f8rtzcd"] Nov 03 01:16:04.019742 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:16:04.019697 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:16:04.172802 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:16:04.172656 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:16:04.302204 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:16:04.302171 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:16:04.609480 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:16:04.609401 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:16:05.376288 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:16:05.376249 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:16:05.618976 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:16:05.618941 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/4568389e-a543-4ff4-a76c-0142ac5b4053-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-d25759-5d4fvt4mj\" (UID: \"4568389e-a543-4ff4-a76c-0142ac5b4053\") " pod="ai-tenant-e2e-extmod-a-d25759/payload-processing-external-model-e2e-extmod-a-d25759-5d4fvt4mj" Nov 03 01:16:05.619197 ip-10-0-140-107 kubenswrapper[4939]: E1103 01:16:05.619098 4939 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-a-d25759/agc-cred-229a2b395ead: secret "agc-cred-229a2b395ead" not found Nov 03 01:16:05.619197 ip-10-0-140-107 kubenswrapper[4939]: E1103 01:16:05.619124 4939 projected.go:196] Error preparing data for projected volume provider-credentials for pod ai-tenant-e2e-extmod-a-d25759/payload-processing-external-model-e2e-extmod-a-d25759-5d4fvt4mj: secret "agc-cred-229a2b395ead" not found Nov 03 01:16:05.619197 ip-10-0-140-107 kubenswrapper[4939]: E1103 01:16:05.619191 4939 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/4568389e-a543-4ff4-a76c-0142ac5b4053-provider-credentials podName:4568389e-a543-4ff4-a76c-0142ac5b4053 nodeName:}" failed. No retries permitted until 2026-11-03 01:16:09.619176766 +0000 UTC m=+2117.883511702 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/4568389e-a543-4ff4-a76c-0142ac5b4053-provider-credentials") pod "payload-processing-external-model-e2e-extmod-a-d25759-5d4fvt4mj" (UID: "4568389e-a543-4ff4-a76c-0142ac5b4053") : secret "agc-cred-229a2b395ead" not found Nov 03 01:16:06.110230 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:16:06.110181 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:16:06.772364 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:16:06.772336 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:16:07.403002 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:16:07.402965 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:16:07.728363 ip-10-0-140-107 kubenswrapper[4939]: E1103 01:16:07.728253 4939 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-b-57079b/agc-cred-06b235044840: secret "agc-cred-06b235044840" not found Nov 03 01:16:07.728363 ip-10-0-140-107 kubenswrapper[4939]: E1103 01:16:07.728291 4939 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-b-57079b/agc-cred-83631bdec3fc: secret "agc-cred-83631bdec3fc" not found Nov 03 01:16:07.728363 ip-10-0-140-107 kubenswrapper[4939]: E1103 01:16:07.728303 4939 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-b-57079b/agc-cred-e705148022ae: secret "agc-cred-e705148022ae" not found Nov 03 01:16:07.728363 ip-10-0-140-107 kubenswrapper[4939]: E1103 01:16:07.728318 4939 projected.go:196] Error preparing data for projected volume provider-credentials for pod ai-tenant-e2e-extmod-b-57079b/payload-processing-external-model-e2e-extmod-b-57079b-59f8rtzcd: [secret "agc-cred-06b235044840" not found, secret "agc-cred-83631bdec3fc" not found, secret "agc-cred-e705148022ae" not found] Nov 03 01:16:07.728666 ip-10-0-140-107 kubenswrapper[4939]: E1103 01:16:07.728398 4939 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/c54b89b6-6593-4f97-9cf9-21e7d21880b6-provider-credentials podName:c54b89b6-6593-4f97-9cf9-21e7d21880b6 nodeName:}" failed. No retries permitted until 2026-11-03 01:16:23.728375151 +0000 UTC m=+2131.992710084 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/c54b89b6-6593-4f97-9cf9-21e7d21880b6-provider-credentials") pod "payload-processing-external-model-e2e-extmod-b-57079b-59f8rtzcd" (UID: "c54b89b6-6593-4f97-9cf9-21e7d21880b6") : [secret "agc-cred-06b235044840" not found, secret "agc-cred-83631bdec3fc" not found, secret "agc-cred-e705148022ae" not found] Nov 03 01:16:08.079132 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:16:08.079054 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:16:08.775815 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:16:08.775773 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:16:08.991045 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:16:08.990989 4939 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-b-57079b/payload-processing-external-model-e2e-extmod-b-57079b-59f8rtzcd" podUID="c54b89b6-6593-4f97-9cf9-21e7d21880b6" containerName="payload-processing" containerID="cri-o://e1aa07c948a31d2f6dc827ac12c0a06874b892d9fb01c24a786dba7bfdc67110" gracePeriod=25 Nov 03 01:16:09.349296 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:16:09.349262 4939 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-57079b/payload-processing-external-model-e2e-extmod-b-57079b-59f8rtzcd" Nov 03 01:16:09.361831 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:16:09.361804 4939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c54b89b6-6593-4f97-9cf9-21e7d21880b6-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c54b89b6-6593-4f97-9cf9-21e7d21880b6-plugins-config-volume\") pod \"c54b89b6-6593-4f97-9cf9-21e7d21880b6\" (UID: \"c54b89b6-6593-4f97-9cf9-21e7d21880b6\") " Nov 03 01:16:09.362037 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:16:09.361893 4939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c54b89b6-6593-4f97-9cf9-21e7d21880b6-provider-credentials\" (UniqueName: \"kubernetes.io/projected/c54b89b6-6593-4f97-9cf9-21e7d21880b6-provider-credentials\") pod \"c54b89b6-6593-4f97-9cf9-21e7d21880b6\" (UID: \"c54b89b6-6593-4f97-9cf9-21e7d21880b6\") " Nov 03 01:16:09.362037 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:16:09.361980 4939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c54b89b6-6593-4f97-9cf9-21e7d21880b6-kube-api-access-87tzv\" (UniqueName: \"kubernetes.io/projected/c54b89b6-6593-4f97-9cf9-21e7d21880b6-kube-api-access-87tzv\") pod \"c54b89b6-6593-4f97-9cf9-21e7d21880b6\" (UID: \"c54b89b6-6593-4f97-9cf9-21e7d21880b6\") " Nov 03 01:16:09.362037 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:16:09.362004 4939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c54b89b6-6593-4f97-9cf9-21e7d21880b6-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/c54b89b6-6593-4f97-9cf9-21e7d21880b6-routing-overlay-volume\") pod \"c54b89b6-6593-4f97-9cf9-21e7d21880b6\" (UID: \"c54b89b6-6593-4f97-9cf9-21e7d21880b6\") " Nov 03 01:16:09.362434 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:16:09.362404 4939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c54b89b6-6593-4f97-9cf9-21e7d21880b6-routing-overlay-volume" pod "c54b89b6-6593-4f97-9cf9-21e7d21880b6" (UID: "c54b89b6-6593-4f97-9cf9-21e7d21880b6"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 01:16:09.362857 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:16:09.362816 4939 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/c54b89b6-6593-4f97-9cf9-21e7d21880b6-routing-overlay-volume\") on node \"ip-10-0-140-107.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:16:09.367187 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:16:09.367160 4939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c54b89b6-6593-4f97-9cf9-21e7d21880b6-plugins-config-volume" pod "c54b89b6-6593-4f97-9cf9-21e7d21880b6" (UID: "c54b89b6-6593-4f97-9cf9-21e7d21880b6"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 01:16:09.367700 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:16:09.367659 4939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c54b89b6-6593-4f97-9cf9-21e7d21880b6-kube-api-access-87tzv" pod "c54b89b6-6593-4f97-9cf9-21e7d21880b6" (UID: "c54b89b6-6593-4f97-9cf9-21e7d21880b6"). InnerVolumeSpecName "kube-api-access-87tzv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 01:16:09.368075 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:16:09.368047 4939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c54b89b6-6593-4f97-9cf9-21e7d21880b6-provider-credentials" pod "c54b89b6-6593-4f97-9cf9-21e7d21880b6" (UID: "c54b89b6-6593-4f97-9cf9-21e7d21880b6"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 01:16:09.464650 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:16:09.464618 4939 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/c54b89b6-6593-4f97-9cf9-21e7d21880b6-provider-credentials\") on node \"ip-10-0-140-107.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:16:09.464650 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:16:09.464648 4939 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-87tzv\" (UniqueName: \"kubernetes.io/projected/c54b89b6-6593-4f97-9cf9-21e7d21880b6-kube-api-access-87tzv\") on node \"ip-10-0-140-107.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:16:09.464850 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:16:09.464662 4939 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c54b89b6-6593-4f97-9cf9-21e7d21880b6-plugins-config-volume\") on node \"ip-10-0-140-107.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:16:09.676787 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:16:09.676742 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/4568389e-a543-4ff4-a76c-0142ac5b4053-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-d25759-5d4fvt4mj\" (UID: \"4568389e-a543-4ff4-a76c-0142ac5b4053\") " pod="ai-tenant-e2e-extmod-a-d25759/payload-processing-external-model-e2e-extmod-a-d25759-5d4fvt4mj" Nov 03 01:16:09.677292 ip-10-0-140-107 kubenswrapper[4939]: E1103 01:16:09.677261 4939 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-a-d25759/agc-cred-229a2b395ead: secret "agc-cred-229a2b395ead" not found Nov 03 01:16:09.677510 ip-10-0-140-107 kubenswrapper[4939]: E1103 01:16:09.677493 4939 projected.go:196] Error preparing data for projected volume provider-credentials for pod ai-tenant-e2e-extmod-a-d25759/payload-processing-external-model-e2e-extmod-a-d25759-5d4fvt4mj: secret "agc-cred-229a2b395ead" not found Nov 03 01:16:09.678282 ip-10-0-140-107 kubenswrapper[4939]: E1103 01:16:09.678266 4939 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/4568389e-a543-4ff4-a76c-0142ac5b4053-provider-credentials podName:4568389e-a543-4ff4-a76c-0142ac5b4053 nodeName:}" failed. No retries permitted until 2026-11-03 01:16:17.678243816 +0000 UTC m=+2125.942578760 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/4568389e-a543-4ff4-a76c-0142ac5b4053-provider-credentials") pod "payload-processing-external-model-e2e-extmod-a-d25759-5d4fvt4mj" (UID: "4568389e-a543-4ff4-a76c-0142ac5b4053") : secret "agc-cred-229a2b395ead" not found Nov 03 01:16:09.678849 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:16:09.678759 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:16:09.681560 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:16:09.681534 4939 generic.go:356] "Generic (PLEG): container finished" podID="c54b89b6-6593-4f97-9cf9-21e7d21880b6" containerID="e1aa07c948a31d2f6dc827ac12c0a06874b892d9fb01c24a786dba7bfdc67110" exitCode=0 Nov 03 01:16:09.681835 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:16:09.681740 4939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-57079b/payload-processing-external-model-e2e-extmod-b-57079b-59f8rtzcd" event={"ID":"c54b89b6-6593-4f97-9cf9-21e7d21880b6","Type":"ContainerDied","Data":"e1aa07c948a31d2f6dc827ac12c0a06874b892d9fb01c24a786dba7bfdc67110"} Nov 03 01:16:09.682109 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:16:09.682012 4939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-57079b/payload-processing-external-model-e2e-extmod-b-57079b-59f8rtzcd" event={"ID":"c54b89b6-6593-4f97-9cf9-21e7d21880b6","Type":"ContainerDied","Data":"419d680c81ec61196845f5d4823a2ccc86c2ea8d26d7f7ecc0d5ea13b3c3abea"} Nov 03 01:16:09.682303 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:16:09.682099 4939 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-57079b/payload-processing-external-model-e2e-extmod-b-57079b-59f8rtzcd" Nov 03 01:16:09.683132 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:16:09.682133 4939 scope.go:122] "RemoveContainer" containerID="e1aa07c948a31d2f6dc827ac12c0a06874b892d9fb01c24a786dba7bfdc67110" Nov 03 01:16:09.701696 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:16:09.701570 4939 scope.go:122] "RemoveContainer" containerID="e1aa07c948a31d2f6dc827ac12c0a06874b892d9fb01c24a786dba7bfdc67110" Nov 03 01:16:09.702037 ip-10-0-140-107 kubenswrapper[4939]: E1103 01:16:09.702009 4939 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e1aa07c948a31d2f6dc827ac12c0a06874b892d9fb01c24a786dba7bfdc67110\": container with ID starting with e1aa07c948a31d2f6dc827ac12c0a06874b892d9fb01c24a786dba7bfdc67110 not found: ID does not exist" containerID="e1aa07c948a31d2f6dc827ac12c0a06874b892d9fb01c24a786dba7bfdc67110" Nov 03 01:16:09.702130 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:16:09.702055 4939 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e1aa07c948a31d2f6dc827ac12c0a06874b892d9fb01c24a786dba7bfdc67110"} err="failed to get container status \"e1aa07c948a31d2f6dc827ac12c0a06874b892d9fb01c24a786dba7bfdc67110\": rpc error: code = NotFound desc = could not find container \"e1aa07c948a31d2f6dc827ac12c0a06874b892d9fb01c24a786dba7bfdc67110\": container with ID starting with e1aa07c948a31d2f6dc827ac12c0a06874b892d9fb01c24a786dba7bfdc67110 not found: ID does not exist" Nov 03 01:16:09.711859 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:16:09.711835 4939 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-57079b/payload-processing-external-model-e2e-extmod-b-57079b-59f8rtzcd"] Nov 03 01:16:09.716700 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:16:09.716672 4939 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-b-57079b/payload-processing-external-model-e2e-extmod-b-57079b-59f8rtzcd"] Nov 03 01:16:09.873365 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:16:09.873332 4939 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c54b89b6-6593-4f97-9cf9-21e7d21880b6" path="/var/lib/kubelet/pods/c54b89b6-6593-4f97-9cf9-21e7d21880b6/volumes" Nov 03 01:16:10.186427 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:16:10.186143 4939 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-af8244/payload-processing-external-model-e2e-extmod-b-af8244-55d6pxhs9"] Nov 03 01:16:10.194189 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:16:10.191669 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:16:10.303949 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:16:10.303869 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:16:10.777933 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:16:10.774449 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:16:11.051720 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:16:11.049586 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:16:11.177931 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:16:11.177697 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:16:11.309980 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:16:11.309642 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:16:11.509537 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:16:11.509496 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:16:12.183898 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:16:12.183868 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:16:12.902074 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:16:12.902039 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:16:13.579863 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:16:13.579831 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:16:13.988212 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:16:13.988156 4939 pod_container_manager_linux.go:218] "Failed to delete cgroup paths" cgroupName=["kubepods","burstable","pod05835a80-2f35-4b67-b578-42e64bfafb9f"] err="unable to destroy cgroup paths for cgroup [kubepods burstable pod05835a80-2f35-4b67-b578-42e64bfafb9f] : Timed out while waiting for systemd to remove kubepods-burstable-pod05835a80_2f35_4b67_b578_42e64bfafb9f.slice" Nov 03 01:16:13.988212 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:16:13.988208 4939 kubelet_pods.go:2797] "Failed to destroy orphaned pod cgroup" podUID="05835a80-2f35-4b67-b578-42e64bfafb9f" err="failed to delete cgroup paths for [kubepods burstable pod05835a80-2f35-4b67-b578-42e64bfafb9f] : unable to destroy cgroup paths for cgroup [kubepods burstable pod05835a80-2f35-4b67-b578-42e64bfafb9f] : Timed out while waiting for systemd to remove kubepods-burstable-pod05835a80_2f35_4b67_b578_42e64bfafb9f.slice" Nov 03 01:16:14.284112 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:16:14.284025 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:16:14.918169 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:16:14.918132 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:16:15.189218 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:16:15.189133 4939 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-b-af8244/payload-processing-external-model-e2e-extmod-b-af8244-55d6pxhs9" podUID="cff06885-ae10-4a06-ae3d-d5db99244d99" containerName="payload-processing" containerID="cri-o://9b151bd3619d13efa5ebce64287bb1d1ee2ec9b1b63ea670fe2f6d231275175e" gracePeriod=25 Nov 03 01:16:15.498759 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:16:15.498727 4939 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-af8244/payload-processing-external-model-e2e-extmod-b-af8244-55d6pxhs9" Nov 03 01:16:15.536283 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:16:15.536250 4939 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-extmod-b-be59df-openshift-default-6764f766d9-qhdnv"] Nov 03 01:16:15.537773 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:16:15.537746 4939 cpu_manager.go:414] "containerMap: removing container" podUID="c54b89b6-6593-4f97-9cf9-21e7d21880b6" containerName="payload-processing" Nov 03 01:16:15.537843 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:16:15.537777 4939 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c54b89b6-6593-4f97-9cf9-21e7d21880b6" containerName="payload-processing" Nov 03 01:16:15.537843 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:16:15.537815 4939 cpu_manager.go:414] "containerMap: removing container" podUID="b24379ee-39df-43ad-b624-556142c07324" containerName="cleanup" Nov 03 01:16:15.537843 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:16:15.537823 4939 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b24379ee-39df-43ad-b624-556142c07324" containerName="cleanup" Nov 03 01:16:15.537978 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:16:15.537879 4939 cpu_manager.go:414] "containerMap: removing container" podUID="cff06885-ae10-4a06-ae3d-d5db99244d99" containerName="payload-processing" Nov 03 01:16:15.537978 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:16:15.537888 4939 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cff06885-ae10-4a06-ae3d-d5db99244d99" containerName="payload-processing" Nov 03 01:16:15.538133 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:16:15.538117 4939 memory_manager.go:365] "RemoveStaleState removing state" podUID="b24379ee-39df-43ad-b624-556142c07324" containerName="cleanup" Nov 03 01:16:15.538176 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:16:15.538147 4939 memory_manager.go:365] "RemoveStaleState removing state" podUID="cff06885-ae10-4a06-ae3d-d5db99244d99" containerName="payload-processing" Nov 03 01:16:15.538176 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:16:15.538173 4939 memory_manager.go:365] "RemoveStaleState removing state" podUID="c54b89b6-6593-4f97-9cf9-21e7d21880b6" containerName="payload-processing" Nov 03 01:16:15.553173 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:16:15.553116 4939 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-b-be59df-openshift-default-6764f766d9-qhdnv"] Nov 03 01:16:15.553333 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:16:15.553226 4939 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-be59df-openshift-default-6764f766d9-qhdnv" Nov 03 01:16:15.557114 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:16:15.557082 4939 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-extmod-b-be59df-openshift-default-dockercfg-mk8kf\"" Nov 03 01:16:15.573698 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:16:15.573670 4939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cff06885-ae10-4a06-ae3d-d5db99244d99-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cff06885-ae10-4a06-ae3d-d5db99244d99-plugins-config-volume\") pod \"cff06885-ae10-4a06-ae3d-d5db99244d99\" (UID: \"cff06885-ae10-4a06-ae3d-d5db99244d99\") " Nov 03 01:16:15.573870 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:16:15.573850 4939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cff06885-ae10-4a06-ae3d-d5db99244d99-kube-api-access-574qq\" (UniqueName: \"kubernetes.io/projected/cff06885-ae10-4a06-ae3d-d5db99244d99-kube-api-access-574qq\") pod \"cff06885-ae10-4a06-ae3d-d5db99244d99\" (UID: \"cff06885-ae10-4a06-ae3d-d5db99244d99\") " Nov 03 01:16:15.575024 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:16:15.574202 4939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cff06885-ae10-4a06-ae3d-d5db99244d99-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/cff06885-ae10-4a06-ae3d-d5db99244d99-routing-overlay-volume\") pod \"cff06885-ae10-4a06-ae3d-d5db99244d99\" (UID: \"cff06885-ae10-4a06-ae3d-d5db99244d99\") " Nov 03 01:16:15.575024 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:16:15.574220 4939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cff06885-ae10-4a06-ae3d-d5db99244d99-plugins-config-volume" pod "cff06885-ae10-4a06-ae3d-d5db99244d99" (UID: "cff06885-ae10-4a06-ae3d-d5db99244d99"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 01:16:15.575024 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:16:15.574419 4939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cff06885-ae10-4a06-ae3d-d5db99244d99-provider-credentials\" (UniqueName: \"kubernetes.io/projected/cff06885-ae10-4a06-ae3d-d5db99244d99-provider-credentials\") pod \"cff06885-ae10-4a06-ae3d-d5db99244d99\" (UID: \"cff06885-ae10-4a06-ae3d-d5db99244d99\") " Nov 03 01:16:15.575024 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:16:15.574659 4939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cff06885-ae10-4a06-ae3d-d5db99244d99-routing-overlay-volume" pod "cff06885-ae10-4a06-ae3d-d5db99244d99" (UID: "cff06885-ae10-4a06-ae3d-d5db99244d99"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 01:16:15.575024 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:16:15.574858 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d28c9737-e9a3-4497-8b53-a47062c1a2f1-istio-token\") pod \"e2e-extmod-b-be59df-openshift-default-6764f766d9-qhdnv\" (UID: \"d28c9737-e9a3-4497-8b53-a47062c1a2f1\") " pod="openshift-ingress/e2e-extmod-b-be59df-openshift-default-6764f766d9-qhdnv" Nov 03 01:16:15.575024 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:16:15.574959 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d28c9737-e9a3-4497-8b53-a47062c1a2f1-credential-socket\") pod \"e2e-extmod-b-be59df-openshift-default-6764f766d9-qhdnv\" (UID: \"d28c9737-e9a3-4497-8b53-a47062c1a2f1\") " pod="openshift-ingress/e2e-extmod-b-be59df-openshift-default-6764f766d9-qhdnv" Nov 03 01:16:15.575504 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:16:15.575060 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d28c9737-e9a3-4497-8b53-a47062c1a2f1-istio-data\") pod \"e2e-extmod-b-be59df-openshift-default-6764f766d9-qhdnv\" (UID: \"d28c9737-e9a3-4497-8b53-a47062c1a2f1\") " pod="openshift-ingress/e2e-extmod-b-be59df-openshift-default-6764f766d9-qhdnv" Nov 03 01:16:15.575504 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:16:15.575203 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ntph5\" (UniqueName: \"kubernetes.io/projected/d28c9737-e9a3-4497-8b53-a47062c1a2f1-kube-api-access-ntph5\") pod \"e2e-extmod-b-be59df-openshift-default-6764f766d9-qhdnv\" (UID: \"d28c9737-e9a3-4497-8b53-a47062c1a2f1\") " pod="openshift-ingress/e2e-extmod-b-be59df-openshift-default-6764f766d9-qhdnv" Nov 03 01:16:15.575504 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:16:15.575283 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d28c9737-e9a3-4497-8b53-a47062c1a2f1-istio-podinfo\") pod \"e2e-extmod-b-be59df-openshift-default-6764f766d9-qhdnv\" (UID: \"d28c9737-e9a3-4497-8b53-a47062c1a2f1\") " pod="openshift-ingress/e2e-extmod-b-be59df-openshift-default-6764f766d9-qhdnv" Nov 03 01:16:15.575504 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:16:15.575499 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d28c9737-e9a3-4497-8b53-a47062c1a2f1-workload-certs\") pod \"e2e-extmod-b-be59df-openshift-default-6764f766d9-qhdnv\" (UID: \"d28c9737-e9a3-4497-8b53-a47062c1a2f1\") " pod="openshift-ingress/e2e-extmod-b-be59df-openshift-default-6764f766d9-qhdnv" Nov 03 01:16:15.575732 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:16:15.575610 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d28c9737-e9a3-4497-8b53-a47062c1a2f1-workload-socket\") pod \"e2e-extmod-b-be59df-openshift-default-6764f766d9-qhdnv\" (UID: \"d28c9737-e9a3-4497-8b53-a47062c1a2f1\") " pod="openshift-ingress/e2e-extmod-b-be59df-openshift-default-6764f766d9-qhdnv" Nov 03 01:16:15.575732 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:16:15.575672 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d28c9737-e9a3-4497-8b53-a47062c1a2f1-istio-envoy\") pod \"e2e-extmod-b-be59df-openshift-default-6764f766d9-qhdnv\" (UID: \"d28c9737-e9a3-4497-8b53-a47062c1a2f1\") " pod="openshift-ingress/e2e-extmod-b-be59df-openshift-default-6764f766d9-qhdnv" Nov 03 01:16:15.576052 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:16:15.576024 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d28c9737-e9a3-4497-8b53-a47062c1a2f1-istiod-ca-cert\") pod \"e2e-extmod-b-be59df-openshift-default-6764f766d9-qhdnv\" (UID: \"d28c9737-e9a3-4497-8b53-a47062c1a2f1\") " pod="openshift-ingress/e2e-extmod-b-be59df-openshift-default-6764f766d9-qhdnv" Nov 03 01:16:15.576237 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:16:15.576218 4939 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/cff06885-ae10-4a06-ae3d-d5db99244d99-routing-overlay-volume\") on node \"ip-10-0-140-107.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:16:15.576315 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:16:15.576244 4939 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cff06885-ae10-4a06-ae3d-d5db99244d99-plugins-config-volume\") on node \"ip-10-0-140-107.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:16:15.577362 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:16:15.577337 4939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cff06885-ae10-4a06-ae3d-d5db99244d99-provider-credentials" pod "cff06885-ae10-4a06-ae3d-d5db99244d99" (UID: "cff06885-ae10-4a06-ae3d-d5db99244d99"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 01:16:15.577468 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:16:15.577376 4939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cff06885-ae10-4a06-ae3d-d5db99244d99-kube-api-access-574qq" pod "cff06885-ae10-4a06-ae3d-d5db99244d99" (UID: "cff06885-ae10-4a06-ae3d-d5db99244d99"). InnerVolumeSpecName "kube-api-access-574qq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 01:16:15.678055 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:16:15.678022 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d28c9737-e9a3-4497-8b53-a47062c1a2f1-istio-token\") pod \"e2e-extmod-b-be59df-openshift-default-6764f766d9-qhdnv\" (UID: \"d28c9737-e9a3-4497-8b53-a47062c1a2f1\") " pod="openshift-ingress/e2e-extmod-b-be59df-openshift-default-6764f766d9-qhdnv" Nov 03 01:16:15.678223 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:16:15.678080 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d28c9737-e9a3-4497-8b53-a47062c1a2f1-credential-socket\") pod \"e2e-extmod-b-be59df-openshift-default-6764f766d9-qhdnv\" (UID: \"d28c9737-e9a3-4497-8b53-a47062c1a2f1\") " pod="openshift-ingress/e2e-extmod-b-be59df-openshift-default-6764f766d9-qhdnv" Nov 03 01:16:15.678223 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:16:15.678103 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d28c9737-e9a3-4497-8b53-a47062c1a2f1-istio-data\") pod \"e2e-extmod-b-be59df-openshift-default-6764f766d9-qhdnv\" (UID: \"d28c9737-e9a3-4497-8b53-a47062c1a2f1\") " pod="openshift-ingress/e2e-extmod-b-be59df-openshift-default-6764f766d9-qhdnv" Nov 03 01:16:15.678223 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:16:15.678129 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ntph5\" (UniqueName: \"kubernetes.io/projected/d28c9737-e9a3-4497-8b53-a47062c1a2f1-kube-api-access-ntph5\") pod \"e2e-extmod-b-be59df-openshift-default-6764f766d9-qhdnv\" (UID: \"d28c9737-e9a3-4497-8b53-a47062c1a2f1\") " pod="openshift-ingress/e2e-extmod-b-be59df-openshift-default-6764f766d9-qhdnv" Nov 03 01:16:15.678223 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:16:15.678162 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d28c9737-e9a3-4497-8b53-a47062c1a2f1-istio-podinfo\") pod \"e2e-extmod-b-be59df-openshift-default-6764f766d9-qhdnv\" (UID: \"d28c9737-e9a3-4497-8b53-a47062c1a2f1\") " pod="openshift-ingress/e2e-extmod-b-be59df-openshift-default-6764f766d9-qhdnv" Nov 03 01:16:15.678418 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:16:15.678267 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d28c9737-e9a3-4497-8b53-a47062c1a2f1-workload-certs\") pod \"e2e-extmod-b-be59df-openshift-default-6764f766d9-qhdnv\" (UID: \"d28c9737-e9a3-4497-8b53-a47062c1a2f1\") " pod="openshift-ingress/e2e-extmod-b-be59df-openshift-default-6764f766d9-qhdnv" Nov 03 01:16:15.678418 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:16:15.678344 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d28c9737-e9a3-4497-8b53-a47062c1a2f1-workload-socket\") pod \"e2e-extmod-b-be59df-openshift-default-6764f766d9-qhdnv\" (UID: \"d28c9737-e9a3-4497-8b53-a47062c1a2f1\") " pod="openshift-ingress/e2e-extmod-b-be59df-openshift-default-6764f766d9-qhdnv" Nov 03 01:16:15.678418 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:16:15.678391 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d28c9737-e9a3-4497-8b53-a47062c1a2f1-istio-envoy\") pod \"e2e-extmod-b-be59df-openshift-default-6764f766d9-qhdnv\" (UID: \"d28c9737-e9a3-4497-8b53-a47062c1a2f1\") " pod="openshift-ingress/e2e-extmod-b-be59df-openshift-default-6764f766d9-qhdnv" Nov 03 01:16:15.678580 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:16:15.678528 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d28c9737-e9a3-4497-8b53-a47062c1a2f1-istiod-ca-cert\") pod \"e2e-extmod-b-be59df-openshift-default-6764f766d9-qhdnv\" (UID: \"d28c9737-e9a3-4497-8b53-a47062c1a2f1\") " pod="openshift-ingress/e2e-extmod-b-be59df-openshift-default-6764f766d9-qhdnv" Nov 03 01:16:15.678580 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:16:15.678568 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d28c9737-e9a3-4497-8b53-a47062c1a2f1-credential-socket\") pod \"e2e-extmod-b-be59df-openshift-default-6764f766d9-qhdnv\" (UID: \"d28c9737-e9a3-4497-8b53-a47062c1a2f1\") " pod="openshift-ingress/e2e-extmod-b-be59df-openshift-default-6764f766d9-qhdnv" Nov 03 01:16:15.678701 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:16:15.678589 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d28c9737-e9a3-4497-8b53-a47062c1a2f1-istio-data\") pod \"e2e-extmod-b-be59df-openshift-default-6764f766d9-qhdnv\" (UID: \"d28c9737-e9a3-4497-8b53-a47062c1a2f1\") " pod="openshift-ingress/e2e-extmod-b-be59df-openshift-default-6764f766d9-qhdnv" Nov 03 01:16:15.678701 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:16:15.678639 4939 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/cff06885-ae10-4a06-ae3d-d5db99244d99-provider-credentials\") on node \"ip-10-0-140-107.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:16:15.678701 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:16:15.678638 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d28c9737-e9a3-4497-8b53-a47062c1a2f1-workload-socket\") pod \"e2e-extmod-b-be59df-openshift-default-6764f766d9-qhdnv\" (UID: \"d28c9737-e9a3-4497-8b53-a47062c1a2f1\") " pod="openshift-ingress/e2e-extmod-b-be59df-openshift-default-6764f766d9-qhdnv" Nov 03 01:16:15.678701 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:16:15.678659 4939 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-574qq\" (UniqueName: \"kubernetes.io/projected/cff06885-ae10-4a06-ae3d-d5db99244d99-kube-api-access-574qq\") on node \"ip-10-0-140-107.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:16:15.678701 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:16:15.678692 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d28c9737-e9a3-4497-8b53-a47062c1a2f1-workload-certs\") pod \"e2e-extmod-b-be59df-openshift-default-6764f766d9-qhdnv\" (UID: \"d28c9737-e9a3-4497-8b53-a47062c1a2f1\") " pod="openshift-ingress/e2e-extmod-b-be59df-openshift-default-6764f766d9-qhdnv" Nov 03 01:16:15.679184 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:16:15.679161 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d28c9737-e9a3-4497-8b53-a47062c1a2f1-istiod-ca-cert\") pod \"e2e-extmod-b-be59df-openshift-default-6764f766d9-qhdnv\" (UID: \"d28c9737-e9a3-4497-8b53-a47062c1a2f1\") " pod="openshift-ingress/e2e-extmod-b-be59df-openshift-default-6764f766d9-qhdnv" Nov 03 01:16:15.681244 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:16:15.681218 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d28c9737-e9a3-4497-8b53-a47062c1a2f1-istio-podinfo\") pod \"e2e-extmod-b-be59df-openshift-default-6764f766d9-qhdnv\" (UID: \"d28c9737-e9a3-4497-8b53-a47062c1a2f1\") " pod="openshift-ingress/e2e-extmod-b-be59df-openshift-default-6764f766d9-qhdnv" Nov 03 01:16:15.681509 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:16:15.681489 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d28c9737-e9a3-4497-8b53-a47062c1a2f1-istio-envoy\") pod \"e2e-extmod-b-be59df-openshift-default-6764f766d9-qhdnv\" (UID: \"d28c9737-e9a3-4497-8b53-a47062c1a2f1\") " pod="openshift-ingress/e2e-extmod-b-be59df-openshift-default-6764f766d9-qhdnv" Nov 03 01:16:15.687181 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:16:15.687160 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d28c9737-e9a3-4497-8b53-a47062c1a2f1-istio-token\") pod \"e2e-extmod-b-be59df-openshift-default-6764f766d9-qhdnv\" (UID: \"d28c9737-e9a3-4497-8b53-a47062c1a2f1\") " pod="openshift-ingress/e2e-extmod-b-be59df-openshift-default-6764f766d9-qhdnv" Nov 03 01:16:15.687277 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:16:15.687178 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ntph5\" (UniqueName: \"kubernetes.io/projected/d28c9737-e9a3-4497-8b53-a47062c1a2f1-kube-api-access-ntph5\") pod \"e2e-extmod-b-be59df-openshift-default-6764f766d9-qhdnv\" (UID: \"d28c9737-e9a3-4497-8b53-a47062c1a2f1\") " pod="openshift-ingress/e2e-extmod-b-be59df-openshift-default-6764f766d9-qhdnv" Nov 03 01:16:15.700826 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:16:15.700797 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:16:15.826350 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:16:15.826250 4939 generic.go:356] "Generic (PLEG): container finished" podID="cff06885-ae10-4a06-ae3d-d5db99244d99" containerID="9b151bd3619d13efa5ebce64287bb1d1ee2ec9b1b63ea670fe2f6d231275175e" exitCode=0 Nov 03 01:16:15.826350 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:16:15.826292 4939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-af8244/payload-processing-external-model-e2e-extmod-b-af8244-55d6pxhs9" event={"ID":"cff06885-ae10-4a06-ae3d-d5db99244d99","Type":"ContainerDied","Data":"9b151bd3619d13efa5ebce64287bb1d1ee2ec9b1b63ea670fe2f6d231275175e"} Nov 03 01:16:15.826350 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:16:15.826314 4939 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-af8244/payload-processing-external-model-e2e-extmod-b-af8244-55d6pxhs9" Nov 03 01:16:15.826350 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:16:15.826337 4939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-af8244/payload-processing-external-model-e2e-extmod-b-af8244-55d6pxhs9" event={"ID":"cff06885-ae10-4a06-ae3d-d5db99244d99","Type":"ContainerDied","Data":"061591ddb0dda47fd787323f2149bbe99892b0dac9d4fa249118e34bbcf3b651"} Nov 03 01:16:15.826682 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:16:15.826360 4939 scope.go:122] "RemoveContainer" containerID="9b151bd3619d13efa5ebce64287bb1d1ee2ec9b1b63ea670fe2f6d231275175e" Nov 03 01:16:15.841839 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:16:15.841815 4939 scope.go:122] "RemoveContainer" containerID="9b151bd3619d13efa5ebce64287bb1d1ee2ec9b1b63ea670fe2f6d231275175e" Nov 03 01:16:15.842180 ip-10-0-140-107 kubenswrapper[4939]: E1103 01:16:15.842113 4939 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9b151bd3619d13efa5ebce64287bb1d1ee2ec9b1b63ea670fe2f6d231275175e\": container with ID starting with 9b151bd3619d13efa5ebce64287bb1d1ee2ec9b1b63ea670fe2f6d231275175e not found: ID does not exist" containerID="9b151bd3619d13efa5ebce64287bb1d1ee2ec9b1b63ea670fe2f6d231275175e" Nov 03 01:16:15.842180 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:16:15.842152 4939 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9b151bd3619d13efa5ebce64287bb1d1ee2ec9b1b63ea670fe2f6d231275175e"} err="failed to get container status \"9b151bd3619d13efa5ebce64287bb1d1ee2ec9b1b63ea670fe2f6d231275175e\": rpc error: code = NotFound desc = could not find container \"9b151bd3619d13efa5ebce64287bb1d1ee2ec9b1b63ea670fe2f6d231275175e\": container with ID starting with 9b151bd3619d13efa5ebce64287bb1d1ee2ec9b1b63ea670fe2f6d231275175e not found: ID does not exist" Nov 03 01:16:15.880144 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:16:15.880114 4939 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-be59df-openshift-default-6764f766d9-qhdnv" Nov 03 01:16:15.881650 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:16:15.881623 4939 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-af8244/payload-processing-external-model-e2e-extmod-b-af8244-55d6pxhs9"] Nov 03 01:16:15.881836 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:16:15.881821 4939 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-b-af8244/payload-processing-external-model-e2e-extmod-b-af8244-55d6pxhs9"] Nov 03 01:16:16.201898 ip-10-0-140-107 kubenswrapper[4939]: W1103 01:16:16.201862 4939 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd28c9737_e9a3_4497_8b53_a47062c1a2f1.slice/crio-b12cd2b74d7f00348b08f7f784747ac1b1f6f5f448e82bccd88b6627054e43cc WatchSource:0}: Error finding container b12cd2b74d7f00348b08f7f784747ac1b1f6f5f448e82bccd88b6627054e43cc: Status 404 returned error can't find the container with id b12cd2b74d7f00348b08f7f784747ac1b1f6f5f448e82bccd88b6627054e43cc Nov 03 01:16:16.203466 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:16:16.203431 4939 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-b-be59df-openshift-default-6764f766d9-qhdnv"] Nov 03 01:16:16.205074 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:16:16.205037 4939 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 01:16:16.205197 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:16:16.205120 4939 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 01:16:16.205197 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:16:16.205164 4939 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 01:16:16.555014 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:16:16.554920 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:16:17.183126 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:16:17.182823 4939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-be59df-openshift-default-6764f766d9-qhdnv" event={"ID":"d28c9737-e9a3-4497-8b53-a47062c1a2f1","Type":"ContainerStarted","Data":"4495666af2d7394c1f0f7974210ab5e4251abecaa13e8034a9f93b892b2a19d1"} Nov 03 01:16:17.183126 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:16:17.182887 4939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-be59df-openshift-default-6764f766d9-qhdnv" event={"ID":"d28c9737-e9a3-4497-8b53-a47062c1a2f1","Type":"ContainerStarted","Data":"b12cd2b74d7f00348b08f7f784747ac1b1f6f5f448e82bccd88b6627054e43cc"} Nov 03 01:16:17.210671 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:16:17.210609 4939 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-extmod-b-be59df-openshift-default-6764f766d9-qhdnv" podStartSLOduration=241.210589981 podStartE2EDuration="4m1.210589981s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 01:12:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 01:16:17.207054424 +0000 UTC m=+2125.471389370" watchObservedRunningTime="2026-11-03 01:16:17.210589981 +0000 UTC m=+2125.474924928" Nov 03 01:16:17.423934 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:16:17.420621 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:16:17.566078 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:16:17.565997 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:16:17.688089 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:16:17.685816 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:16:17.760817 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:16:17.760778 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/4568389e-a543-4ff4-a76c-0142ac5b4053-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-d25759-5d4fvt4mj\" (UID: \"4568389e-a543-4ff4-a76c-0142ac5b4053\") " pod="ai-tenant-e2e-extmod-a-d25759/payload-processing-external-model-e2e-extmod-a-d25759-5d4fvt4mj" Nov 03 01:16:17.761045 ip-10-0-140-107 kubenswrapper[4939]: E1103 01:16:17.761012 4939 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-a-d25759/agc-cred-229a2b395ead: secret "agc-cred-229a2b395ead" not found Nov 03 01:16:17.761045 ip-10-0-140-107 kubenswrapper[4939]: E1103 01:16:17.761038 4939 projected.go:196] Error preparing data for projected volume provider-credentials for pod ai-tenant-e2e-extmod-a-d25759/payload-processing-external-model-e2e-extmod-a-d25759-5d4fvt4mj: secret "agc-cred-229a2b395ead" not found Nov 03 01:16:17.761199 ip-10-0-140-107 kubenswrapper[4939]: E1103 01:16:17.761119 4939 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/4568389e-a543-4ff4-a76c-0142ac5b4053-provider-credentials podName:4568389e-a543-4ff4-a76c-0142ac5b4053 nodeName:}" failed. No retries permitted until 2026-11-03 01:16:33.761098615 +0000 UTC m=+2142.025433551 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/4568389e-a543-4ff4-a76c-0142ac5b4053-provider-credentials") pod "payload-processing-external-model-e2e-extmod-a-d25759-5d4fvt4mj" (UID: "4568389e-a543-4ff4-a76c-0142ac5b4053") : secret "agc-cred-229a2b395ead" not found Nov 03 01:16:17.873366 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:16:17.873334 4939 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cff06885-ae10-4a06-ae3d-d5db99244d99" path="/var/lib/kubelet/pods/cff06885-ae10-4a06-ae3d-d5db99244d99/volumes" Nov 03 01:16:17.880795 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:16:17.880770 4939 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-extmod-b-be59df-openshift-default-6764f766d9-qhdnv" Nov 03 01:16:17.887430 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:16:17.887404 4939 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-extmod-b-be59df-openshift-default-6764f766d9-qhdnv" Nov 03 01:16:18.032538 ip-10-0-140-107 kubenswrapper[4939]: E1103 01:16:18.032498 4939 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-sthjh" containerName="extractor" Nov 03 01:16:18.133680 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:16:18.133603 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:16:18.204573 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:16:18.204537 4939 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-b-be59df-openshift-default-6764f766d9-qhdnv" Nov 03 01:16:18.205568 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:16:18.205547 4939 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-extmod-b-be59df-openshift-default-6764f766d9-qhdnv" Nov 03 01:16:18.276772 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:16:18.276737 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:16:18.437737 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:16:18.437658 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:16:18.571651 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:16:18.571616 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:16:18.770866 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:16:18.764414 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:16:19.568146 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:16:19.568116 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:16:19.715933 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:16:19.714170 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:16:20.607690 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:16:20.607649 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:16:20.924536 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:16:20.924440 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:16:21.035514 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:16:21.035475 4939 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-d25759/payload-processing-external-model-e2e-extmod-a-d25759-5d4fvt4mj"] Nov 03 01:16:21.036187 ip-10-0-140-107 kubenswrapper[4939]: E1103 01:16:21.036149 4939 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[provider-credentials], unattached volumes=[], failed to process volumes=[]: context canceled" pod="ai-tenant-e2e-extmod-a-d25759/payload-processing-external-model-e2e-extmod-a-d25759-5d4fvt4mj" podUID="4568389e-a543-4ff4-a76c-0142ac5b4053" Nov 03 01:16:21.071244 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:16:21.071207 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:16:21.505456 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:16:21.505396 4939 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-d25759/payload-processing-external-model-e2e-extmod-a-d25759-5d4fvt4mj" Nov 03 01:16:21.702504 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:16:21.702477 4939 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-d25759/payload-processing-external-model-e2e-extmod-a-d25759-5d4fvt4mj" Nov 03 01:16:21.887600 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:16:21.887573 4939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4568389e-a543-4ff4-a76c-0142ac5b4053-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4568389e-a543-4ff4-a76c-0142ac5b4053-plugins-config-volume\") pod \"4568389e-a543-4ff4-a76c-0142ac5b4053\" (UID: \"4568389e-a543-4ff4-a76c-0142ac5b4053\") " Nov 03 01:16:21.887600 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:16:21.887609 4939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4568389e-a543-4ff4-a76c-0142ac5b4053-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/4568389e-a543-4ff4-a76c-0142ac5b4053-routing-overlay-volume\") pod \"4568389e-a543-4ff4-a76c-0142ac5b4053\" (UID: \"4568389e-a543-4ff4-a76c-0142ac5b4053\") " Nov 03 01:16:21.887843 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:16:21.887701 4939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4568389e-a543-4ff4-a76c-0142ac5b4053-kube-api-access-5hshg\" (UniqueName: \"kubernetes.io/projected/4568389e-a543-4ff4-a76c-0142ac5b4053-kube-api-access-5hshg\") pod \"4568389e-a543-4ff4-a76c-0142ac5b4053\" (UID: \"4568389e-a543-4ff4-a76c-0142ac5b4053\") " Nov 03 01:16:21.888026 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:16:21.888001 4939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4568389e-a543-4ff4-a76c-0142ac5b4053-routing-overlay-volume" pod "4568389e-a543-4ff4-a76c-0142ac5b4053" (UID: "4568389e-a543-4ff4-a76c-0142ac5b4053"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 01:16:21.888123 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:16:21.888007 4939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4568389e-a543-4ff4-a76c-0142ac5b4053-plugins-config-volume" pod "4568389e-a543-4ff4-a76c-0142ac5b4053" (UID: "4568389e-a543-4ff4-a76c-0142ac5b4053"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 01:16:21.888494 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:16:21.888470 4939 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4568389e-a543-4ff4-a76c-0142ac5b4053-plugins-config-volume\") on node \"ip-10-0-140-107.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:16:21.888555 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:16:21.888522 4939 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/4568389e-a543-4ff4-a76c-0142ac5b4053-routing-overlay-volume\") on node \"ip-10-0-140-107.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:16:21.890030 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:16:21.890003 4939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4568389e-a543-4ff4-a76c-0142ac5b4053-kube-api-access-5hshg" pod "4568389e-a543-4ff4-a76c-0142ac5b4053" (UID: "4568389e-a543-4ff4-a76c-0142ac5b4053"). InnerVolumeSpecName "kube-api-access-5hshg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 01:16:21.956167 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:16:21.955668 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:16:21.990684 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:16:21.990640 4939 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5hshg\" (UniqueName: \"kubernetes.io/projected/4568389e-a543-4ff4-a76c-0142ac5b4053-kube-api-access-5hshg\") on node \"ip-10-0-140-107.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:16:22.513615 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:16:22.513585 4939 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-d25759/payload-processing-external-model-e2e-extmod-a-d25759-5d4fvt4mj" Nov 03 01:16:22.551823 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:16:22.551792 4939 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-b-be59df-c7b8b85f5-vzcvs"] Nov 03 01:16:22.576966 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:16:22.576933 4939 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-b-be59df-c7b8b85f5-vzcvs" Nov 03 01:16:22.588087 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:16:22.588056 4939 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-extmod-b-be59df\"" Nov 03 01:16:22.588500 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:16:22.588471 4939 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-extmod-b-be59df-dockercfg-l2wrh\"" Nov 03 01:16:22.590109 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:16:22.589502 4939 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-d25759/payload-processing-external-model-e2e-extmod-a-d25759-5d4fvt4mj"] Nov 03 01:16:22.600559 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:16:22.600529 4939 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-a-d25759/payload-processing-external-model-e2e-extmod-a-d25759-5d4fvt4mj"] Nov 03 01:16:22.601025 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:16:22.601001 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mwngw\" (UniqueName: \"kubernetes.io/projected/949acf43-3c7b-4074-b2f1-c323e596c786-kube-api-access-mwngw\") pod \"payload-processing-e2e-extmod-b-be59df-c7b8b85f5-vzcvs\" (UID: \"949acf43-3c7b-4074-b2f1-c323e596c786\") " pod="openshift-ingress/payload-processing-e2e-extmod-b-be59df-c7b8b85f5-vzcvs" Nov 03 01:16:22.601279 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:16:22.601248 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/949acf43-3c7b-4074-b2f1-c323e596c786-plugins-config-volume\") pod \"payload-processing-e2e-extmod-b-be59df-c7b8b85f5-vzcvs\" (UID: \"949acf43-3c7b-4074-b2f1-c323e596c786\") " pod="openshift-ingress/payload-processing-e2e-extmod-b-be59df-c7b8b85f5-vzcvs" Nov 03 01:16:22.605923 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:16:22.605877 4939 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-b-be59df-c7b8b85f5-vzcvs"] Nov 03 01:16:22.615335 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:16:22.615314 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:16:22.716984 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:16:22.714423 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mwngw\" (UniqueName: \"kubernetes.io/projected/949acf43-3c7b-4074-b2f1-c323e596c786-kube-api-access-mwngw\") pod \"payload-processing-e2e-extmod-b-be59df-c7b8b85f5-vzcvs\" (UID: \"949acf43-3c7b-4074-b2f1-c323e596c786\") " pod="openshift-ingress/payload-processing-e2e-extmod-b-be59df-c7b8b85f5-vzcvs" Nov 03 01:16:22.716984 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:16:22.714685 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/949acf43-3c7b-4074-b2f1-c323e596c786-plugins-config-volume\") pod \"payload-processing-e2e-extmod-b-be59df-c7b8b85f5-vzcvs\" (UID: \"949acf43-3c7b-4074-b2f1-c323e596c786\") " pod="openshift-ingress/payload-processing-e2e-extmod-b-be59df-c7b8b85f5-vzcvs" Nov 03 01:16:22.716984 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:16:22.714837 4939 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/4568389e-a543-4ff4-a76c-0142ac5b4053-provider-credentials\") on node \"ip-10-0-140-107.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:16:22.716984 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:16:22.715577 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/949acf43-3c7b-4074-b2f1-c323e596c786-plugins-config-volume\") pod \"payload-processing-e2e-extmod-b-be59df-c7b8b85f5-vzcvs\" (UID: \"949acf43-3c7b-4074-b2f1-c323e596c786\") " pod="openshift-ingress/payload-processing-e2e-extmod-b-be59df-c7b8b85f5-vzcvs" Nov 03 01:16:22.726027 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:16:22.725999 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mwngw\" (UniqueName: \"kubernetes.io/projected/949acf43-3c7b-4074-b2f1-c323e596c786-kube-api-access-mwngw\") pod \"payload-processing-e2e-extmod-b-be59df-c7b8b85f5-vzcvs\" (UID: \"949acf43-3c7b-4074-b2f1-c323e596c786\") " pod="openshift-ingress/payload-processing-e2e-extmod-b-be59df-c7b8b85f5-vzcvs" Nov 03 01:16:22.903222 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:16:22.903187 4939 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-b-be59df-c7b8b85f5-vzcvs" Nov 03 01:16:23.189374 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:16:23.189342 4939 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-b-be59df-c7b8b85f5-vzcvs"] Nov 03 01:16:23.189813 ip-10-0-140-107 kubenswrapper[4939]: W1103 01:16:23.189788 4939 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod949acf43_3c7b_4074_b2f1_c323e596c786.slice/crio-cefbec58f26871fab60825b2a33e004c9bae578eeb844a80ef343cb289d2c6b9 WatchSource:0}: Error finding container cefbec58f26871fab60825b2a33e004c9bae578eeb844a80ef343cb289d2c6b9: Status 404 returned error can't find the container with id cefbec58f26871fab60825b2a33e004c9bae578eeb844a80ef343cb289d2c6b9 Nov 03 01:16:23.418251 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:16:23.418218 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:16:23.545991 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:16:23.545284 4939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-b-be59df-c7b8b85f5-vzcvs" event={"ID":"949acf43-3c7b-4074-b2f1-c323e596c786","Type":"ContainerStarted","Data":"9a1dd6727bef01526e92714e2e8e033a4fbfa3f5582039418d39b033ae2c52f9"} Nov 03 01:16:23.545991 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:16:23.545400 4939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-b-be59df-c7b8b85f5-vzcvs" event={"ID":"949acf43-3c7b-4074-b2f1-c323e596c786","Type":"ContainerStarted","Data":"cefbec58f26871fab60825b2a33e004c9bae578eeb844a80ef343cb289d2c6b9"} Nov 03 01:16:23.546965 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:16:23.546619 4939 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-extmod-b-be59df-c7b8b85f5-vzcvs" Nov 03 01:16:23.567017 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:16:23.566969 4939 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-extmod-b-be59df-c7b8b85f5-vzcvs" podStartSLOduration=3.566956482 podStartE2EDuration="3.566956482s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 01:16:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 01:16:23.563314703 +0000 UTC m=+2131.827649651" watchObservedRunningTime="2026-11-03 01:16:23.566956482 +0000 UTC m=+2131.831291417" Nov 03 01:16:23.871623 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:16:23.871587 4939 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4568389e-a543-4ff4-a76c-0142ac5b4053" path="/var/lib/kubelet/pods/4568389e-a543-4ff4-a76c-0142ac5b4053/volumes" Nov 03 01:16:23.930505 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:16:23.930465 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:16:24.061929 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:16:24.061416 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:16:24.495459 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:16:24.495425 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:16:24.612630 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:16:24.612599 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:16:25.208236 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:16:25.208204 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:16:25.335702 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:16:25.335674 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:16:26.512175 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:16:26.512135 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:16:27.279720 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:16:27.279688 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:16:27.968245 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:16:27.968211 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:16:28.697980 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:16:28.697933 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:16:29.480405 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:16:29.480370 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:16:30.227642 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:16:30.227597 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:16:30.910671 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:16:30.910638 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:16:31.051259 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:16:31.051207 4939 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-717217-openshift-default-578c69554c-hs4f9"] Nov 03 01:16:31.052324 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:16:31.052268 4939 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w3-717217-openshift-default-578c69554c-hs4f9" podUID="8ba28520-d5f4-4ec9-8b86-f8a698325eea" containerName="istio-proxy" containerID="cri-o://d9710015fabade148b832b280406c56149499b9bb24448cde19a80f22e1d39bc" gracePeriod=30 Nov 03 01:16:31.588599 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:16:31.588566 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:16:32.194866 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:16:32.194828 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:16:32.899391 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:16:32.899357 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:16:33.584626 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:16:33.584596 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:16:34.209431 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:16:34.209401 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:16:34.948090 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:16:34.948027 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:16:35.591188 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:16:35.591150 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:16:35.667208 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:16:35.667177 4939 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-extmod-b-be59df-c7b8b85f5-vzcvs" Nov 03 01:16:36.321640 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:16:36.321540 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:16:36.484508 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:16:36.484474 4939 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w3-717217-openshift-default-578c69554c-hs4f9" Nov 03 01:16:36.522661 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:16:36.522626 4939 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-extmod-a-d25759-openshift-default-d47fd9785-ggf9t"] Nov 03 01:16:36.523638 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:16:36.523614 4939 cpu_manager.go:414] "containerMap: removing container" podUID="8ba28520-d5f4-4ec9-8b86-f8a698325eea" containerName="istio-proxy" Nov 03 01:16:36.523638 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:16:36.523638 4939 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8ba28520-d5f4-4ec9-8b86-f8a698325eea" containerName="istio-proxy" Nov 03 01:16:36.523941 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:16:36.523925 4939 memory_manager.go:365] "RemoveStaleState removing state" podUID="8ba28520-d5f4-4ec9-8b86-f8a698325eea" containerName="istio-proxy" Nov 03 01:16:36.541135 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:16:36.541105 4939 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-a-d25759-openshift-default-d47fd9785-ggf9t"] Nov 03 01:16:36.541497 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:16:36.541467 4939 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-d25759-openshift-default-d47fd9785-ggf9t" Nov 03 01:16:36.584679 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:16:36.584640 4939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/8ba28520-d5f4-4ec9-8b86-f8a698325eea-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8ba28520-d5f4-4ec9-8b86-f8a698325eea-istio-podinfo\") pod \"8ba28520-d5f4-4ec9-8b86-f8a698325eea\" (UID: \"8ba28520-d5f4-4ec9-8b86-f8a698325eea\") " Nov 03 01:16:36.584883 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:16:36.584698 4939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8ba28520-d5f4-4ec9-8b86-f8a698325eea-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8ba28520-d5f4-4ec9-8b86-f8a698325eea-istio-data\") pod \"8ba28520-d5f4-4ec9-8b86-f8a698325eea\" (UID: \"8ba28520-d5f4-4ec9-8b86-f8a698325eea\") " Nov 03 01:16:36.584883 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:16:36.584763 4939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8ba28520-d5f4-4ec9-8b86-f8a698325eea-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8ba28520-d5f4-4ec9-8b86-f8a698325eea-istiod-ca-cert\") pod \"8ba28520-d5f4-4ec9-8b86-f8a698325eea\" (UID: \"8ba28520-d5f4-4ec9-8b86-f8a698325eea\") " Nov 03 01:16:36.585108 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:16:36.584899 4939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8ba28520-d5f4-4ec9-8b86-f8a698325eea-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8ba28520-d5f4-4ec9-8b86-f8a698325eea-workload-socket\") pod \"8ba28520-d5f4-4ec9-8b86-f8a698325eea\" (UID: \"8ba28520-d5f4-4ec9-8b86-f8a698325eea\") " Nov 03 01:16:36.585108 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:16:36.584988 4939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8ba28520-d5f4-4ec9-8b86-f8a698325eea-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8ba28520-d5f4-4ec9-8b86-f8a698325eea-workload-certs\") pod \"8ba28520-d5f4-4ec9-8b86-f8a698325eea\" (UID: \"8ba28520-d5f4-4ec9-8b86-f8a698325eea\") " Nov 03 01:16:36.585108 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:16:36.585045 4939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8ba28520-d5f4-4ec9-8b86-f8a698325eea-istio-token\" (UniqueName: \"kubernetes.io/projected/8ba28520-d5f4-4ec9-8b86-f8a698325eea-istio-token\") pod \"8ba28520-d5f4-4ec9-8b86-f8a698325eea\" (UID: \"8ba28520-d5f4-4ec9-8b86-f8a698325eea\") " Nov 03 01:16:36.585108 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:16:36.585101 4939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8ba28520-d5f4-4ec9-8b86-f8a698325eea-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8ba28520-d5f4-4ec9-8b86-f8a698325eea-credential-socket\") pod \"8ba28520-d5f4-4ec9-8b86-f8a698325eea\" (UID: \"8ba28520-d5f4-4ec9-8b86-f8a698325eea\") " Nov 03 01:16:36.585343 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:16:36.585138 4939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8ba28520-d5f4-4ec9-8b86-f8a698325eea-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8ba28520-d5f4-4ec9-8b86-f8a698325eea-istio-envoy\") pod \"8ba28520-d5f4-4ec9-8b86-f8a698325eea\" (UID: \"8ba28520-d5f4-4ec9-8b86-f8a698325eea\") " Nov 03 01:16:36.585343 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:16:36.585266 4939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8ba28520-d5f4-4ec9-8b86-f8a698325eea-kube-api-access-cp7w8\" (UniqueName: \"kubernetes.io/projected/8ba28520-d5f4-4ec9-8b86-f8a698325eea-kube-api-access-cp7w8\") pod \"8ba28520-d5f4-4ec9-8b86-f8a698325eea\" (UID: \"8ba28520-d5f4-4ec9-8b86-f8a698325eea\") " Nov 03 01:16:36.585835 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:16:36.585744 4939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8ba28520-d5f4-4ec9-8b86-f8a698325eea-istio-data" pod "8ba28520-d5f4-4ec9-8b86-f8a698325eea" (UID: "8ba28520-d5f4-4ec9-8b86-f8a698325eea"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 01:16:36.585835 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:16:36.585756 4939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8ba28520-d5f4-4ec9-8b86-f8a698325eea-workload-socket" pod "8ba28520-d5f4-4ec9-8b86-f8a698325eea" (UID: "8ba28520-d5f4-4ec9-8b86-f8a698325eea"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 01:16:36.586028 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:16:36.585973 4939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8ba28520-d5f4-4ec9-8b86-f8a698325eea-credential-socket" pod "8ba28520-d5f4-4ec9-8b86-f8a698325eea" (UID: "8ba28520-d5f4-4ec9-8b86-f8a698325eea"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 01:16:36.586459 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:16:36.586432 4939 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8ba28520-d5f4-4ec9-8b86-f8a698325eea-istio-data\") on node \"ip-10-0-140-107.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:16:36.586622 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:16:36.586466 4939 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8ba28520-d5f4-4ec9-8b86-f8a698325eea-workload-socket\") on node \"ip-10-0-140-107.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:16:36.586622 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:16:36.586482 4939 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8ba28520-d5f4-4ec9-8b86-f8a698325eea-credential-socket\") on node \"ip-10-0-140-107.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:16:36.586952 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:16:36.586879 4939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8ba28520-d5f4-4ec9-8b86-f8a698325eea-workload-certs" pod "8ba28520-d5f4-4ec9-8b86-f8a698325eea" (UID: "8ba28520-d5f4-4ec9-8b86-f8a698325eea"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 01:16:36.588423 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:16:36.588397 4939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8ba28520-d5f4-4ec9-8b86-f8a698325eea-istiod-ca-cert" pod "8ba28520-d5f4-4ec9-8b86-f8a698325eea" (UID: "8ba28520-d5f4-4ec9-8b86-f8a698325eea"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 01:16:36.588877 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:16:36.588855 4939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8ba28520-d5f4-4ec9-8b86-f8a698325eea-istio-envoy" pod "8ba28520-d5f4-4ec9-8b86-f8a698325eea" (UID: "8ba28520-d5f4-4ec9-8b86-f8a698325eea"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 01:16:36.589033 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:16:36.588941 4939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8ba28520-d5f4-4ec9-8b86-f8a698325eea-istio-token" pod "8ba28520-d5f4-4ec9-8b86-f8a698325eea" (UID: "8ba28520-d5f4-4ec9-8b86-f8a698325eea"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 01:16:36.591029 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:16:36.591007 4939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/8ba28520-d5f4-4ec9-8b86-f8a698325eea-istio-podinfo" pod "8ba28520-d5f4-4ec9-8b86-f8a698325eea" (UID: "8ba28520-d5f4-4ec9-8b86-f8a698325eea"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 01:16:36.591743 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:16:36.591717 4939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8ba28520-d5f4-4ec9-8b86-f8a698325eea-kube-api-access-cp7w8" pod "8ba28520-d5f4-4ec9-8b86-f8a698325eea" (UID: "8ba28520-d5f4-4ec9-8b86-f8a698325eea"). InnerVolumeSpecName "kube-api-access-cp7w8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 01:16:36.688406 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:16:36.688372 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r4zrb\" (UniqueName: \"kubernetes.io/projected/0cd1d1a4-4446-46a4-9b40-dfc3f83ac8a8-kube-api-access-r4zrb\") pod \"e2e-extmod-a-d25759-openshift-default-d47fd9785-ggf9t\" (UID: \"0cd1d1a4-4446-46a4-9b40-dfc3f83ac8a8\") " pod="openshift-ingress/e2e-extmod-a-d25759-openshift-default-d47fd9785-ggf9t" Nov 03 01:16:36.688628 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:16:36.688442 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0cd1d1a4-4446-46a4-9b40-dfc3f83ac8a8-workload-socket\") pod \"e2e-extmod-a-d25759-openshift-default-d47fd9785-ggf9t\" (UID: \"0cd1d1a4-4446-46a4-9b40-dfc3f83ac8a8\") " pod="openshift-ingress/e2e-extmod-a-d25759-openshift-default-d47fd9785-ggf9t" Nov 03 01:16:36.688705 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:16:36.688656 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0cd1d1a4-4446-46a4-9b40-dfc3f83ac8a8-credential-socket\") pod \"e2e-extmod-a-d25759-openshift-default-d47fd9785-ggf9t\" (UID: \"0cd1d1a4-4446-46a4-9b40-dfc3f83ac8a8\") " pod="openshift-ingress/e2e-extmod-a-d25759-openshift-default-d47fd9785-ggf9t" Nov 03 01:16:36.688769 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:16:36.688701 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0cd1d1a4-4446-46a4-9b40-dfc3f83ac8a8-istiod-ca-cert\") pod \"e2e-extmod-a-d25759-openshift-default-d47fd9785-ggf9t\" (UID: \"0cd1d1a4-4446-46a4-9b40-dfc3f83ac8a8\") " pod="openshift-ingress/e2e-extmod-a-d25759-openshift-default-d47fd9785-ggf9t" Nov 03 01:16:36.688827 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:16:36.688780 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0cd1d1a4-4446-46a4-9b40-dfc3f83ac8a8-istio-envoy\") pod \"e2e-extmod-a-d25759-openshift-default-d47fd9785-ggf9t\" (UID: \"0cd1d1a4-4446-46a4-9b40-dfc3f83ac8a8\") " pod="openshift-ingress/e2e-extmod-a-d25759-openshift-default-d47fd9785-ggf9t" Nov 03 01:16:36.688958 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:16:36.688932 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0cd1d1a4-4446-46a4-9b40-dfc3f83ac8a8-istio-token\") pod \"e2e-extmod-a-d25759-openshift-default-d47fd9785-ggf9t\" (UID: \"0cd1d1a4-4446-46a4-9b40-dfc3f83ac8a8\") " pod="openshift-ingress/e2e-extmod-a-d25759-openshift-default-d47fd9785-ggf9t" Nov 03 01:16:36.689129 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:16:36.689112 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0cd1d1a4-4446-46a4-9b40-dfc3f83ac8a8-istio-data\") pod \"e2e-extmod-a-d25759-openshift-default-d47fd9785-ggf9t\" (UID: \"0cd1d1a4-4446-46a4-9b40-dfc3f83ac8a8\") " pod="openshift-ingress/e2e-extmod-a-d25759-openshift-default-d47fd9785-ggf9t" Nov 03 01:16:36.689213 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:16:36.689172 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0cd1d1a4-4446-46a4-9b40-dfc3f83ac8a8-workload-certs\") pod \"e2e-extmod-a-d25759-openshift-default-d47fd9785-ggf9t\" (UID: \"0cd1d1a4-4446-46a4-9b40-dfc3f83ac8a8\") " pod="openshift-ingress/e2e-extmod-a-d25759-openshift-default-d47fd9785-ggf9t" Nov 03 01:16:36.689274 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:16:36.689245 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0cd1d1a4-4446-46a4-9b40-dfc3f83ac8a8-istio-podinfo\") pod \"e2e-extmod-a-d25759-openshift-default-d47fd9785-ggf9t\" (UID: \"0cd1d1a4-4446-46a4-9b40-dfc3f83ac8a8\") " pod="openshift-ingress/e2e-extmod-a-d25759-openshift-default-d47fd9785-ggf9t" Nov 03 01:16:36.689377 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:16:36.689362 4939 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8ba28520-d5f4-4ec9-8b86-f8a698325eea-istio-podinfo\") on node \"ip-10-0-140-107.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:16:36.689448 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:16:36.689385 4939 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8ba28520-d5f4-4ec9-8b86-f8a698325eea-istiod-ca-cert\") on node \"ip-10-0-140-107.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:16:36.689448 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:16:36.689402 4939 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8ba28520-d5f4-4ec9-8b86-f8a698325eea-workload-certs\") on node \"ip-10-0-140-107.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:16:36.689448 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:16:36.689419 4939 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8ba28520-d5f4-4ec9-8b86-f8a698325eea-istio-token\") on node \"ip-10-0-140-107.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:16:36.689448 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:16:36.689434 4939 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8ba28520-d5f4-4ec9-8b86-f8a698325eea-istio-envoy\") on node \"ip-10-0-140-107.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:16:36.689583 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:16:36.689451 4939 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cp7w8\" (UniqueName: \"kubernetes.io/projected/8ba28520-d5f4-4ec9-8b86-f8a698325eea-kube-api-access-cp7w8\") on node \"ip-10-0-140-107.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:16:36.752091 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:16:36.752058 4939 generic.go:356] "Generic (PLEG): container finished" podID="8ba28520-d5f4-4ec9-8b86-f8a698325eea" containerID="d9710015fabade148b832b280406c56149499b9bb24448cde19a80f22e1d39bc" exitCode=0 Nov 03 01:16:36.752218 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:16:36.752112 4939 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w3-717217-openshift-default-578c69554c-hs4f9" Nov 03 01:16:36.752218 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:16:36.752136 4939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-717217-openshift-default-578c69554c-hs4f9" event={"ID":"8ba28520-d5f4-4ec9-8b86-f8a698325eea","Type":"ContainerDied","Data":"d9710015fabade148b832b280406c56149499b9bb24448cde19a80f22e1d39bc"} Nov 03 01:16:36.752339 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:16:36.752216 4939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-717217-openshift-default-578c69554c-hs4f9" event={"ID":"8ba28520-d5f4-4ec9-8b86-f8a698325eea","Type":"ContainerDied","Data":"8bec0ae3fed7271233d1f347b589c6fd69d8de79d13b5cf72772ed283b707e62"} Nov 03 01:16:36.752339 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:16:36.752262 4939 scope.go:122] "RemoveContainer" containerID="d9710015fabade148b832b280406c56149499b9bb24448cde19a80f22e1d39bc" Nov 03 01:16:36.765180 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:16:36.765150 4939 scope.go:122] "RemoveContainer" containerID="d9710015fabade148b832b280406c56149499b9bb24448cde19a80f22e1d39bc" Nov 03 01:16:36.765484 ip-10-0-140-107 kubenswrapper[4939]: E1103 01:16:36.765459 4939 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d9710015fabade148b832b280406c56149499b9bb24448cde19a80f22e1d39bc\": container with ID starting with d9710015fabade148b832b280406c56149499b9bb24448cde19a80f22e1d39bc not found: ID does not exist" containerID="d9710015fabade148b832b280406c56149499b9bb24448cde19a80f22e1d39bc" Nov 03 01:16:36.765571 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:16:36.765493 4939 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d9710015fabade148b832b280406c56149499b9bb24448cde19a80f22e1d39bc"} err="failed to get container status \"d9710015fabade148b832b280406c56149499b9bb24448cde19a80f22e1d39bc\": rpc error: code = NotFound desc = could not find container \"d9710015fabade148b832b280406c56149499b9bb24448cde19a80f22e1d39bc\": container with ID starting with d9710015fabade148b832b280406c56149499b9bb24448cde19a80f22e1d39bc not found: ID does not exist" Nov 03 01:16:36.784705 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:16:36.784672 4939 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-717217-openshift-default-578c69554c-hs4f9"] Nov 03 01:16:36.791055 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:16:36.791027 4939 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-717217-openshift-default-578c69554c-hs4f9"] Nov 03 01:16:36.791648 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:16:36.791589 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0cd1d1a4-4446-46a4-9b40-dfc3f83ac8a8-workload-certs\") pod \"e2e-extmod-a-d25759-openshift-default-d47fd9785-ggf9t\" (UID: \"0cd1d1a4-4446-46a4-9b40-dfc3f83ac8a8\") " pod="openshift-ingress/e2e-extmod-a-d25759-openshift-default-d47fd9785-ggf9t" Nov 03 01:16:36.791892 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:16:36.791663 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0cd1d1a4-4446-46a4-9b40-dfc3f83ac8a8-istio-podinfo\") pod \"e2e-extmod-a-d25759-openshift-default-d47fd9785-ggf9t\" (UID: \"0cd1d1a4-4446-46a4-9b40-dfc3f83ac8a8\") " pod="openshift-ingress/e2e-extmod-a-d25759-openshift-default-d47fd9785-ggf9t" Nov 03 01:16:36.791892 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:16:36.791705 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r4zrb\" (UniqueName: \"kubernetes.io/projected/0cd1d1a4-4446-46a4-9b40-dfc3f83ac8a8-kube-api-access-r4zrb\") pod \"e2e-extmod-a-d25759-openshift-default-d47fd9785-ggf9t\" (UID: \"0cd1d1a4-4446-46a4-9b40-dfc3f83ac8a8\") " pod="openshift-ingress/e2e-extmod-a-d25759-openshift-default-d47fd9785-ggf9t" Nov 03 01:16:36.791892 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:16:36.791779 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0cd1d1a4-4446-46a4-9b40-dfc3f83ac8a8-workload-socket\") pod \"e2e-extmod-a-d25759-openshift-default-d47fd9785-ggf9t\" (UID: \"0cd1d1a4-4446-46a4-9b40-dfc3f83ac8a8\") " pod="openshift-ingress/e2e-extmod-a-d25759-openshift-default-d47fd9785-ggf9t" Nov 03 01:16:36.792276 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:16:36.791985 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0cd1d1a4-4446-46a4-9b40-dfc3f83ac8a8-credential-socket\") pod \"e2e-extmod-a-d25759-openshift-default-d47fd9785-ggf9t\" (UID: \"0cd1d1a4-4446-46a4-9b40-dfc3f83ac8a8\") " pod="openshift-ingress/e2e-extmod-a-d25759-openshift-default-d47fd9785-ggf9t" Nov 03 01:16:36.792276 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:16:36.792015 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0cd1d1a4-4446-46a4-9b40-dfc3f83ac8a8-istiod-ca-cert\") pod \"e2e-extmod-a-d25759-openshift-default-d47fd9785-ggf9t\" (UID: \"0cd1d1a4-4446-46a4-9b40-dfc3f83ac8a8\") " pod="openshift-ingress/e2e-extmod-a-d25759-openshift-default-d47fd9785-ggf9t" Nov 03 01:16:36.792276 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:16:36.792047 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0cd1d1a4-4446-46a4-9b40-dfc3f83ac8a8-istio-envoy\") pod \"e2e-extmod-a-d25759-openshift-default-d47fd9785-ggf9t\" (UID: \"0cd1d1a4-4446-46a4-9b40-dfc3f83ac8a8\") " pod="openshift-ingress/e2e-extmod-a-d25759-openshift-default-d47fd9785-ggf9t" Nov 03 01:16:36.792276 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:16:36.792144 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0cd1d1a4-4446-46a4-9b40-dfc3f83ac8a8-istio-token\") pod \"e2e-extmod-a-d25759-openshift-default-d47fd9785-ggf9t\" (UID: \"0cd1d1a4-4446-46a4-9b40-dfc3f83ac8a8\") " pod="openshift-ingress/e2e-extmod-a-d25759-openshift-default-d47fd9785-ggf9t" Nov 03 01:16:36.792611 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:16:36.792347 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0cd1d1a4-4446-46a4-9b40-dfc3f83ac8a8-istio-data\") pod \"e2e-extmod-a-d25759-openshift-default-d47fd9785-ggf9t\" (UID: \"0cd1d1a4-4446-46a4-9b40-dfc3f83ac8a8\") " pod="openshift-ingress/e2e-extmod-a-d25759-openshift-default-d47fd9785-ggf9t" Nov 03 01:16:36.793379 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:16:36.792789 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0cd1d1a4-4446-46a4-9b40-dfc3f83ac8a8-istio-data\") pod \"e2e-extmod-a-d25759-openshift-default-d47fd9785-ggf9t\" (UID: \"0cd1d1a4-4446-46a4-9b40-dfc3f83ac8a8\") " pod="openshift-ingress/e2e-extmod-a-d25759-openshift-default-d47fd9785-ggf9t" Nov 03 01:16:36.793379 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:16:36.793171 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0cd1d1a4-4446-46a4-9b40-dfc3f83ac8a8-credential-socket\") pod \"e2e-extmod-a-d25759-openshift-default-d47fd9785-ggf9t\" (UID: \"0cd1d1a4-4446-46a4-9b40-dfc3f83ac8a8\") " pod="openshift-ingress/e2e-extmod-a-d25759-openshift-default-d47fd9785-ggf9t" Nov 03 01:16:36.794068 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:16:36.793745 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0cd1d1a4-4446-46a4-9b40-dfc3f83ac8a8-istiod-ca-cert\") pod \"e2e-extmod-a-d25759-openshift-default-d47fd9785-ggf9t\" (UID: \"0cd1d1a4-4446-46a4-9b40-dfc3f83ac8a8\") " pod="openshift-ingress/e2e-extmod-a-d25759-openshift-default-d47fd9785-ggf9t" Nov 03 01:16:36.794068 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:16:36.793888 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0cd1d1a4-4446-46a4-9b40-dfc3f83ac8a8-workload-socket\") pod \"e2e-extmod-a-d25759-openshift-default-d47fd9785-ggf9t\" (UID: \"0cd1d1a4-4446-46a4-9b40-dfc3f83ac8a8\") " pod="openshift-ingress/e2e-extmod-a-d25759-openshift-default-d47fd9785-ggf9t" Nov 03 01:16:36.794379 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:16:36.794353 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0cd1d1a4-4446-46a4-9b40-dfc3f83ac8a8-workload-certs\") pod \"e2e-extmod-a-d25759-openshift-default-d47fd9785-ggf9t\" (UID: \"0cd1d1a4-4446-46a4-9b40-dfc3f83ac8a8\") " pod="openshift-ingress/e2e-extmod-a-d25759-openshift-default-d47fd9785-ggf9t" Nov 03 01:16:36.795278 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:16:36.795252 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0cd1d1a4-4446-46a4-9b40-dfc3f83ac8a8-istio-podinfo\") pod \"e2e-extmod-a-d25759-openshift-default-d47fd9785-ggf9t\" (UID: \"0cd1d1a4-4446-46a4-9b40-dfc3f83ac8a8\") " pod="openshift-ingress/e2e-extmod-a-d25759-openshift-default-d47fd9785-ggf9t" Nov 03 01:16:36.796788 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:16:36.796702 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0cd1d1a4-4446-46a4-9b40-dfc3f83ac8a8-istio-envoy\") pod \"e2e-extmod-a-d25759-openshift-default-d47fd9785-ggf9t\" (UID: \"0cd1d1a4-4446-46a4-9b40-dfc3f83ac8a8\") " pod="openshift-ingress/e2e-extmod-a-d25759-openshift-default-d47fd9785-ggf9t" Nov 03 01:16:36.801633 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:16:36.801605 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-r4zrb\" (UniqueName: \"kubernetes.io/projected/0cd1d1a4-4446-46a4-9b40-dfc3f83ac8a8-kube-api-access-r4zrb\") pod \"e2e-extmod-a-d25759-openshift-default-d47fd9785-ggf9t\" (UID: \"0cd1d1a4-4446-46a4-9b40-dfc3f83ac8a8\") " pod="openshift-ingress/e2e-extmod-a-d25759-openshift-default-d47fd9785-ggf9t" Nov 03 01:16:36.802628 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:16:36.802607 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0cd1d1a4-4446-46a4-9b40-dfc3f83ac8a8-istio-token\") pod \"e2e-extmod-a-d25759-openshift-default-d47fd9785-ggf9t\" (UID: \"0cd1d1a4-4446-46a4-9b40-dfc3f83ac8a8\") " pod="openshift-ingress/e2e-extmod-a-d25759-openshift-default-d47fd9785-ggf9t" Nov 03 01:16:36.868562 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:16:36.868481 4939 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-d25759-openshift-default-d47fd9785-ggf9t" Nov 03 01:16:37.084627 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:16:37.084594 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:16:37.221329 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:16:37.221293 4939 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-a-d25759-openshift-default-d47fd9785-ggf9t"] Nov 03 01:16:37.236111 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:16:37.236067 4939 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 01:16:37.236226 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:16:37.236150 4939 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 01:16:37.236226 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:16:37.236191 4939 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 01:16:37.765342 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:16:37.765298 4939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-d25759-openshift-default-d47fd9785-ggf9t" event={"ID":"0cd1d1a4-4446-46a4-9b40-dfc3f83ac8a8","Type":"ContainerStarted","Data":"6a7f8bce5c0afd0e6312af359b0d245e494f28b66e3c3f222f821145ebececb1"} Nov 03 01:16:37.766320 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:16:37.765374 4939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-d25759-openshift-default-d47fd9785-ggf9t" event={"ID":"0cd1d1a4-4446-46a4-9b40-dfc3f83ac8a8","Type":"ContainerStarted","Data":"75c5ac2f311d4656d8c389229d7fc64a6f8b01c4761f14d1bf3634e944aaa2e7"} Nov 03 01:16:37.789681 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:16:37.789615 4939 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-extmod-a-d25759-openshift-default-d47fd9785-ggf9t" podStartSLOduration=220.789597509 podStartE2EDuration="3m40.789597509s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 01:12:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 01:16:37.786331203 +0000 UTC m=+2146.050666149" watchObservedRunningTime="2026-11-03 01:16:37.789597509 +0000 UTC m=+2146.053932455" Nov 03 01:16:37.884767 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:16:37.884735 4939 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8ba28520-d5f4-4ec9-8b86-f8a698325eea" path="/var/lib/kubelet/pods/8ba28520-d5f4-4ec9-8b86-f8a698325eea/volumes" Nov 03 01:16:37.885178 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:16:37.885162 4939 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-extmod-a-d25759-openshift-default-d47fd9785-ggf9t" Nov 03 01:16:37.916110 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:16:37.916082 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:16:38.703682 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:16:38.703628 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:16:38.880929 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:16:38.880406 4939 patch_prober.go:29] interesting pod/e2e-extmod-a-d25759-openshift-default-d47fd9785-ggf9t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.251:15021/healthz/ready\": dial tcp 10.132.0.251:15021: connect: connection refused" start-of-body= Nov 03 01:16:38.880929 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:16:38.880462 4939 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-d25759-openshift-default-d47fd9785-ggf9t" podUID="0cd1d1a4-4446-46a4-9b40-dfc3f83ac8a8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.251:15021/healthz/ready\": dial tcp 10.132.0.251:15021: connect: connection refused" Nov 03 01:16:39.392551 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:16:39.392512 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:16:39.877077 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:16:39.876606 4939 patch_prober.go:29] interesting pod/e2e-extmod-a-d25759-openshift-default-d47fd9785-ggf9t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.251:15021/healthz/ready\": dial tcp 10.132.0.251:15021: connect: connection refused" start-of-body= Nov 03 01:16:39.877077 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:16:39.876668 4939 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-d25759-openshift-default-d47fd9785-ggf9t" podUID="0cd1d1a4-4446-46a4-9b40-dfc3f83ac8a8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.251:15021/healthz/ready\": dial tcp 10.132.0.251:15021: connect: connection refused" Nov 03 01:16:40.240477 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:16:40.240404 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:16:40.378931 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:16:40.378870 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:16:40.871936 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:16:40.870332 4939 patch_prober.go:29] interesting pod/e2e-extmod-a-d25759-openshift-default-d47fd9785-ggf9t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.251:15021/healthz/ready\": dial tcp 10.132.0.251:15021: connect: connection refused" start-of-body= Nov 03 01:16:40.871936 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:16:40.870412 4939 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-d25759-openshift-default-d47fd9785-ggf9t" podUID="0cd1d1a4-4446-46a4-9b40-dfc3f83ac8a8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.251:15021/healthz/ready\": dial tcp 10.132.0.251:15021: connect: connection refused" Nov 03 01:16:41.803767 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:16:41.803732 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:16:41.879056 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:16:41.879012 4939 patch_prober.go:29] interesting pod/e2e-extmod-a-d25759-openshift-default-d47fd9785-ggf9t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.251:15021/healthz/ready\": dial tcp 10.132.0.251:15021: connect: connection refused" start-of-body= Nov 03 01:16:41.879269 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:16:41.879094 4939 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-d25759-openshift-default-d47fd9785-ggf9t" podUID="0cd1d1a4-4446-46a4-9b40-dfc3f83ac8a8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.251:15021/healthz/ready\": dial tcp 10.132.0.251:15021: connect: connection refused" Nov 03 01:16:42.605078 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:16:42.605049 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:16:42.875016 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:16:42.873303 4939 patch_prober.go:29] interesting pod/e2e-extmod-a-d25759-openshift-default-d47fd9785-ggf9t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.251:15021/healthz/ready\": dial tcp 10.132.0.251:15021: connect: connection refused" start-of-body= Nov 03 01:16:42.875016 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:16:42.873377 4939 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-d25759-openshift-default-d47fd9785-ggf9t" podUID="0cd1d1a4-4446-46a4-9b40-dfc3f83ac8a8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.251:15021/healthz/ready\": dial tcp 10.132.0.251:15021: connect: connection refused" Nov 03 01:16:43.413952 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:16:43.413919 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:16:43.869129 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:16:43.869093 4939 patch_prober.go:29] interesting pod/e2e-extmod-a-d25759-openshift-default-d47fd9785-ggf9t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.251:15021/healthz/ready\": dial tcp 10.132.0.251:15021: connect: connection refused" start-of-body= Nov 03 01:16:43.869311 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:16:43.869156 4939 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-d25759-openshift-default-d47fd9785-ggf9t" podUID="0cd1d1a4-4446-46a4-9b40-dfc3f83ac8a8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.251:15021/healthz/ready\": dial tcp 10.132.0.251:15021: connect: connection refused" Nov 03 01:16:44.022953 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:16:44.022894 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:16:44.273351 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:16:44.273314 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:16:44.874074 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:16:44.874040 4939 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-extmod-a-d25759-openshift-default-d47fd9785-ggf9t" Nov 03 01:16:44.874431 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:16:44.874406 4939 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-a-d25759-openshift-default-d47fd9785-ggf9t" Nov 03 01:16:44.875227 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:16:44.875202 4939 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-extmod-a-d25759-openshift-default-d47fd9785-ggf9t" Nov 03 01:16:44.887163 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:16:44.887134 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:16:45.579012 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:16:45.578959 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:16:46.476374 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:16:46.476323 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:16:47.291300 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:16:47.291264 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:16:47.922610 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:16:47.922562 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:16:48.060190 ip-10-0-140-107 kubenswrapper[4939]: E1103 01:16:48.060151 4939 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-sthjh" containerName="extractor" Nov 03 01:16:48.573663 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:16:48.573624 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:16:49.208489 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:16:49.208448 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:16:49.929783 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:16:49.929737 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:16:50.598269 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:16:50.598234 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:16:51.572547 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:16:51.572313 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:16:52.091656 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:16:52.091623 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:16:52.809127 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:16:52.809093 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:16:53.596109 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:16:53.596073 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:16:54.352349 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:16:54.352318 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:16:54.904530 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:16:54.904488 4939 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-4pw7v"] Nov 03 01:16:55.399887 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:16:55.399854 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:16:56.191741 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:16:56.191704 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:16:57.292919 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:16:57.292881 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:16:58.086959 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:16:58.086880 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:16:58.775391 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:16:58.775352 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:16:59.310880 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:16:59.310816 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:16:59.443894 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:16:59.443856 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:17:00.400780 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:17:00.400745 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:17:01.311793 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:17:01.311761 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:17:02.491560 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:17:02.491519 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:17:03.299573 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:17:03.299505 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:17:04.123648 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:17:04.123611 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:17:04.247066 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:17:04.247028 4939 patch_prober.go:29] interesting pod/e2e-extmod-b-be59df-openshift-default-6764f766d9-qhdnv container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.249:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 01:17:04.247257 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:17:04.247115 4939 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-extmod-b-be59df-openshift-default-6764f766d9-qhdnv" podUID="d28c9737-e9a3-4497-8b53-a47062c1a2f1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.249:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 01:17:04.910674 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:17:04.910632 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:17:05.597724 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:17:05.597684 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:17:06.270989 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:17:06.266850 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:17:06.909315 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:17:06.909280 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:17:07.606389 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:17:07.606364 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:17:08.403194 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:17:08.403159 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:17:08.529430 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:17:08.529398 4939 generic.go:356] "Generic (PLEG): container finished" podID="d28c9737-e9a3-4497-8b53-a47062c1a2f1" containerID="4495666af2d7394c1f0f7974210ab5e4251abecaa13e8034a9f93b892b2a19d1" exitCode=0 Nov 03 01:17:08.529609 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:17:08.529468 4939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-be59df-openshift-default-6764f766d9-qhdnv" event={"ID":"d28c9737-e9a3-4497-8b53-a47062c1a2f1","Type":"ContainerDied","Data":"4495666af2d7394c1f0f7974210ab5e4251abecaa13e8034a9f93b892b2a19d1"} Nov 03 01:17:08.530015 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:17:08.529997 4939 scope.go:122] "RemoveContainer" containerID="4495666af2d7394c1f0f7974210ab5e4251abecaa13e8034a9f93b892b2a19d1" Nov 03 01:17:08.532103 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:17:08.532071 4939 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 01:17:08.532223 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:17:08.532146 4939 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 01:17:08.532223 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:17:08.532179 4939 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 01:17:09.099286 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:17:09.099251 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:17:09.540997 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:17:09.540872 4939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-be59df-openshift-default-6764f766d9-qhdnv" event={"ID":"d28c9737-e9a3-4497-8b53-a47062c1a2f1","Type":"ContainerStarted","Data":"1c44a6fcf3a764891ac19aee38afdf17cae5a3c6ae6eb3ee2bb9bfa23cad31ca"} Nov 03 01:17:09.788980 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:17:09.788945 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:17:09.898946 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:17:09.883732 4939 patch_prober.go:29] interesting pod/e2e-extmod-b-be59df-openshift-default-6764f766d9-qhdnv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.249:15021/healthz/ready\": dial tcp 10.132.0.249:15021: connect: connection refused" start-of-body= Nov 03 01:17:09.899260 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:17:09.899219 4939 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-be59df-openshift-default-6764f766d9-qhdnv" podUID="d28c9737-e9a3-4497-8b53-a47062c1a2f1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.249:15021/healthz/ready\": dial tcp 10.132.0.249:15021: connect: connection refused" Nov 03 01:17:10.007591 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:17:10.007553 4939 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-extmod-b-be59df-openshift-default-6764f766d9-qhdnv" Nov 03 01:17:10.683867 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:17:10.683837 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:17:10.881780 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:17:10.881005 4939 patch_prober.go:29] interesting pod/e2e-extmod-b-be59df-openshift-default-6764f766d9-qhdnv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.249:15021/healthz/ready\": dial tcp 10.132.0.249:15021: connect: connection refused" start-of-body= Nov 03 01:17:10.881780 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:17:10.881074 4939 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-be59df-openshift-default-6764f766d9-qhdnv" podUID="d28c9737-e9a3-4497-8b53-a47062c1a2f1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.249:15021/healthz/ready\": dial tcp 10.132.0.249:15021: connect: connection refused" Nov 03 01:17:11.375863 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:17:11.375826 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:17:11.881390 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:17:11.881354 4939 patch_prober.go:29] interesting pod/e2e-extmod-b-be59df-openshift-default-6764f766d9-qhdnv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.249:15021/healthz/ready\": dial tcp 10.132.0.249:15021: connect: connection refused" start-of-body= Nov 03 01:17:11.881881 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:17:11.881849 4939 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-be59df-openshift-default-6764f766d9-qhdnv" podUID="d28c9737-e9a3-4497-8b53-a47062c1a2f1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.249:15021/healthz/ready\": dial tcp 10.132.0.249:15021: connect: connection refused" Nov 03 01:17:12.031991 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:17:12.031958 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:17:12.142305 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:17:12.142214 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:17:12.806132 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:17:12.806097 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:17:12.883936 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:17:12.883321 4939 patch_prober.go:29] interesting pod/e2e-extmod-b-be59df-openshift-default-6764f766d9-qhdnv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.249:15021/healthz/ready\": dial tcp 10.132.0.249:15021: connect: connection refused" start-of-body= Nov 03 01:17:12.883936 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:17:12.883388 4939 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-be59df-openshift-default-6764f766d9-qhdnv" podUID="d28c9737-e9a3-4497-8b53-a47062c1a2f1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.249:15021/healthz/ready\": dial tcp 10.132.0.249:15021: connect: connection refused" Nov 03 01:17:13.584542 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:17:13.584511 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:17:13.883591 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:17:13.883506 4939 patch_prober.go:29] interesting pod/e2e-extmod-b-be59df-openshift-default-6764f766d9-qhdnv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.249:15021/healthz/ready\": dial tcp 10.132.0.249:15021: connect: connection refused" start-of-body= Nov 03 01:17:13.883757 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:17:13.883589 4939 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-be59df-openshift-default-6764f766d9-qhdnv" podUID="d28c9737-e9a3-4497-8b53-a47062c1a2f1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.249:15021/healthz/ready\": dial tcp 10.132.0.249:15021: connect: connection refused" Nov 03 01:17:14.267441 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:17:14.267353 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:17:14.881258 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:17:14.881217 4939 patch_prober.go:29] interesting pod/e2e-extmod-b-be59df-openshift-default-6764f766d9-qhdnv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.249:15021/healthz/ready\": dial tcp 10.132.0.249:15021: connect: connection refused" start-of-body= Nov 03 01:17:14.881440 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:17:14.881283 4939 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-be59df-openshift-default-6764f766d9-qhdnv" podUID="d28c9737-e9a3-4497-8b53-a47062c1a2f1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.249:15021/healthz/ready\": dial tcp 10.132.0.249:15021: connect: connection refused" Nov 03 01:17:15.182971 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:17:15.182844 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:17:15.885846 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:17:15.885815 4939 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-extmod-b-be59df-openshift-default-6764f766d9-qhdnv" Nov 03 01:17:15.886370 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:17:15.886290 4939 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-b-be59df-openshift-default-6764f766d9-qhdnv" Nov 03 01:17:15.886967 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:17:15.886941 4939 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-extmod-b-be59df-openshift-default-6764f766d9-qhdnv" Nov 03 01:17:15.893184 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:17:15.893158 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:17:16.056364 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:17:16.056329 4939 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w1-2729f2-openshift-default-9fb97b4fc-dmrf8"] Nov 03 01:17:16.056623 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:17:16.056576 4939 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w1-2729f2-openshift-default-9fb97b4fc-dmrf8" podUID="1240d677-9bc2-427c-a818-ffed78093771" containerName="istio-proxy" containerID="cri-o://faaf9af48ecb5dfb77b38ea46d984d0db4cb5c35af3172caa13a02d92f04ed09" gracePeriod=30 Nov 03 01:17:16.112489 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:17:16.111585 4939 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-717217-openshift-default-578c69554c-bpgmb"] Nov 03 01:17:16.112489 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:17:16.111843 4939 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w3-717217-openshift-default-578c69554c-bpgmb" podUID="26266573-b6f1-495c-bc39-67bd77a5cb79" containerName="istio-proxy" containerID="cri-o://584eabcda720f30f79164c8fd6aa77c704296dd98fa43e2075af7dbbe68ed6da" gracePeriod=30 Nov 03 01:17:16.636078 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:17:16.636041 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:17:17.520935 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:17:17.520410 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:17:18.035487 ip-10-0-140-107 kubenswrapper[4939]: E1103 01:17:18.035449 4939 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-sthjh" containerName="extractor" Nov 03 01:17:18.352812 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:17:18.352774 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:17:18.901875 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:17:18.901843 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:17:19.610257 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:17:19.610222 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:17:20.301315 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:17:20.301278 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:17:20.926923 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:17:20.926872 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:17:21.441140 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:17:21.441110 4939 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w1-2729f2-openshift-default-9fb97b4fc-dmrf8" Nov 03 01:17:21.489356 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:17:21.489329 4939 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w3-717217-openshift-default-578c69554c-bpgmb" Nov 03 01:17:21.521693 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:17:21.521657 4939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1240d677-9bc2-427c-a818-ffed78093771-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1240d677-9bc2-427c-a818-ffed78093771-workload-certs\") pod \"1240d677-9bc2-427c-a818-ffed78093771\" (UID: \"1240d677-9bc2-427c-a818-ffed78093771\") " Nov 03 01:17:21.521881 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:17:21.521761 4939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/1240d677-9bc2-427c-a818-ffed78093771-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1240d677-9bc2-427c-a818-ffed78093771-istio-podinfo\") pod \"1240d677-9bc2-427c-a818-ffed78093771\" (UID: \"1240d677-9bc2-427c-a818-ffed78093771\") " Nov 03 01:17:21.522105 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:17:21.521897 4939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1240d677-9bc2-427c-a818-ffed78093771-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1240d677-9bc2-427c-a818-ffed78093771-istio-data\") pod \"1240d677-9bc2-427c-a818-ffed78093771\" (UID: \"1240d677-9bc2-427c-a818-ffed78093771\") " Nov 03 01:17:21.522105 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:17:21.522015 4939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1240d677-9bc2-427c-a818-ffed78093771-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1240d677-9bc2-427c-a818-ffed78093771-credential-socket\") pod \"1240d677-9bc2-427c-a818-ffed78093771\" (UID: \"1240d677-9bc2-427c-a818-ffed78093771\") " Nov 03 01:17:21.522105 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:17:21.522053 4939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1240d677-9bc2-427c-a818-ffed78093771-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1240d677-9bc2-427c-a818-ffed78093771-workload-socket\") pod \"1240d677-9bc2-427c-a818-ffed78093771\" (UID: \"1240d677-9bc2-427c-a818-ffed78093771\") " Nov 03 01:17:21.522105 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:17:21.522053 4939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1240d677-9bc2-427c-a818-ffed78093771-workload-certs" pod "1240d677-9bc2-427c-a818-ffed78093771" (UID: "1240d677-9bc2-427c-a818-ffed78093771"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 01:17:21.522330 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:17:21.522135 4939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1240d677-9bc2-427c-a818-ffed78093771-istio-token\" (UniqueName: \"kubernetes.io/projected/1240d677-9bc2-427c-a818-ffed78093771-istio-token\") pod \"1240d677-9bc2-427c-a818-ffed78093771\" (UID: \"1240d677-9bc2-427c-a818-ffed78093771\") " Nov 03 01:17:21.522330 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:17:21.522216 4939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1240d677-9bc2-427c-a818-ffed78093771-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1240d677-9bc2-427c-a818-ffed78093771-istio-envoy\") pod \"1240d677-9bc2-427c-a818-ffed78093771\" (UID: \"1240d677-9bc2-427c-a818-ffed78093771\") " Nov 03 01:17:21.522330 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:17:21.522283 4939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1240d677-9bc2-427c-a818-ffed78093771-kube-api-access-bjqz5\" (UniqueName: \"kubernetes.io/projected/1240d677-9bc2-427c-a818-ffed78093771-kube-api-access-bjqz5\") pod \"1240d677-9bc2-427c-a818-ffed78093771\" (UID: \"1240d677-9bc2-427c-a818-ffed78093771\") " Nov 03 01:17:21.522571 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:17:21.522325 4939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1240d677-9bc2-427c-a818-ffed78093771-istio-data" pod "1240d677-9bc2-427c-a818-ffed78093771" (UID: "1240d677-9bc2-427c-a818-ffed78093771"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 01:17:21.522571 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:17:21.522441 4939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1240d677-9bc2-427c-a818-ffed78093771-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1240d677-9bc2-427c-a818-ffed78093771-istiod-ca-cert\") pod \"1240d677-9bc2-427c-a818-ffed78093771\" (UID: \"1240d677-9bc2-427c-a818-ffed78093771\") " Nov 03 01:17:21.522571 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:17:21.522529 4939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1240d677-9bc2-427c-a818-ffed78093771-workload-socket" pod "1240d677-9bc2-427c-a818-ffed78093771" (UID: "1240d677-9bc2-427c-a818-ffed78093771"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 01:17:21.522758 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:17:21.522565 4939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1240d677-9bc2-427c-a818-ffed78093771-credential-socket" pod "1240d677-9bc2-427c-a818-ffed78093771" (UID: "1240d677-9bc2-427c-a818-ffed78093771"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 01:17:21.523374 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:17:21.523354 4939 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1240d677-9bc2-427c-a818-ffed78093771-workload-certs\") on node \"ip-10-0-140-107.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:17:21.523522 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:17:21.523498 4939 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1240d677-9bc2-427c-a818-ffed78093771-istio-data\") on node \"ip-10-0-140-107.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:17:21.523522 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:17:21.523522 4939 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1240d677-9bc2-427c-a818-ffed78093771-credential-socket\") on node \"ip-10-0-140-107.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:17:21.523712 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:17:21.523534 4939 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1240d677-9bc2-427c-a818-ffed78093771-workload-socket\") on node \"ip-10-0-140-107.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:17:21.523800 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:17:21.523780 4939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1240d677-9bc2-427c-a818-ffed78093771-istiod-ca-cert" pod "1240d677-9bc2-427c-a818-ffed78093771" (UID: "1240d677-9bc2-427c-a818-ffed78093771"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 01:17:21.525441 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:17:21.525403 4939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1240d677-9bc2-427c-a818-ffed78093771-istio-envoy" pod "1240d677-9bc2-427c-a818-ffed78093771" (UID: "1240d677-9bc2-427c-a818-ffed78093771"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 01:17:21.527132 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:17:21.527104 4939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/1240d677-9bc2-427c-a818-ffed78093771-istio-podinfo" pod "1240d677-9bc2-427c-a818-ffed78093771" (UID: "1240d677-9bc2-427c-a818-ffed78093771"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 01:17:21.528318 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:17:21.528296 4939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1240d677-9bc2-427c-a818-ffed78093771-kube-api-access-bjqz5" pod "1240d677-9bc2-427c-a818-ffed78093771" (UID: "1240d677-9bc2-427c-a818-ffed78093771"). InnerVolumeSpecName "kube-api-access-bjqz5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 01:17:21.528458 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:17:21.528437 4939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1240d677-9bc2-427c-a818-ffed78093771-istio-token" pod "1240d677-9bc2-427c-a818-ffed78093771" (UID: "1240d677-9bc2-427c-a818-ffed78093771"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 01:17:21.624371 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:17:21.624338 4939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/26266573-b6f1-495c-bc39-67bd77a5cb79-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/26266573-b6f1-495c-bc39-67bd77a5cb79-workload-socket\") pod \"26266573-b6f1-495c-bc39-67bd77a5cb79\" (UID: \"26266573-b6f1-495c-bc39-67bd77a5cb79\") " Nov 03 01:17:21.624572 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:17:21.624413 4939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/26266573-b6f1-495c-bc39-67bd77a5cb79-istio-token\" (UniqueName: \"kubernetes.io/projected/26266573-b6f1-495c-bc39-67bd77a5cb79-istio-token\") pod \"26266573-b6f1-495c-bc39-67bd77a5cb79\" (UID: \"26266573-b6f1-495c-bc39-67bd77a5cb79\") " Nov 03 01:17:21.624572 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:17:21.624489 4939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/26266573-b6f1-495c-bc39-67bd77a5cb79-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/26266573-b6f1-495c-bc39-67bd77a5cb79-credential-socket\") pod \"26266573-b6f1-495c-bc39-67bd77a5cb79\" (UID: \"26266573-b6f1-495c-bc39-67bd77a5cb79\") " Nov 03 01:17:21.624572 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:17:21.624550 4939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/26266573-b6f1-495c-bc39-67bd77a5cb79-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/26266573-b6f1-495c-bc39-67bd77a5cb79-workload-certs\") pod \"26266573-b6f1-495c-bc39-67bd77a5cb79\" (UID: \"26266573-b6f1-495c-bc39-67bd77a5cb79\") " Nov 03 01:17:21.624763 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:17:21.624600 4939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/26266573-b6f1-495c-bc39-67bd77a5cb79-workload-socket" pod "26266573-b6f1-495c-bc39-67bd77a5cb79" (UID: "26266573-b6f1-495c-bc39-67bd77a5cb79"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 01:17:21.624763 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:17:21.624633 4939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/26266573-b6f1-495c-bc39-67bd77a5cb79-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/26266573-b6f1-495c-bc39-67bd77a5cb79-istio-data\") pod \"26266573-b6f1-495c-bc39-67bd77a5cb79\" (UID: \"26266573-b6f1-495c-bc39-67bd77a5cb79\") " Nov 03 01:17:21.624763 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:17:21.624681 4939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/26266573-b6f1-495c-bc39-67bd77a5cb79-kube-api-access-82xm6\" (UniqueName: \"kubernetes.io/projected/26266573-b6f1-495c-bc39-67bd77a5cb79-kube-api-access-82xm6\") pod \"26266573-b6f1-495c-bc39-67bd77a5cb79\" (UID: \"26266573-b6f1-495c-bc39-67bd77a5cb79\") " Nov 03 01:17:21.624763 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:17:21.624718 4939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/26266573-b6f1-495c-bc39-67bd77a5cb79-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/26266573-b6f1-495c-bc39-67bd77a5cb79-istio-podinfo\") pod \"26266573-b6f1-495c-bc39-67bd77a5cb79\" (UID: \"26266573-b6f1-495c-bc39-67bd77a5cb79\") " Nov 03 01:17:21.624991 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:17:21.624791 4939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/26266573-b6f1-495c-bc39-67bd77a5cb79-credential-socket" pod "26266573-b6f1-495c-bc39-67bd77a5cb79" (UID: "26266573-b6f1-495c-bc39-67bd77a5cb79"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 01:17:21.624991 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:17:21.624803 4939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/26266573-b6f1-495c-bc39-67bd77a5cb79-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/26266573-b6f1-495c-bc39-67bd77a5cb79-istio-envoy\") pod \"26266573-b6f1-495c-bc39-67bd77a5cb79\" (UID: \"26266573-b6f1-495c-bc39-67bd77a5cb79\") " Nov 03 01:17:21.624991 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:17:21.624830 4939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/26266573-b6f1-495c-bc39-67bd77a5cb79-workload-certs" pod "26266573-b6f1-495c-bc39-67bd77a5cb79" (UID: "26266573-b6f1-495c-bc39-67bd77a5cb79"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 01:17:21.624991 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:17:21.624973 4939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/26266573-b6f1-495c-bc39-67bd77a5cb79-istio-data" pod "26266573-b6f1-495c-bc39-67bd77a5cb79" (UID: "26266573-b6f1-495c-bc39-67bd77a5cb79"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 01:17:21.625235 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:17:21.625162 4939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/26266573-b6f1-495c-bc39-67bd77a5cb79-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/26266573-b6f1-495c-bc39-67bd77a5cb79-istiod-ca-cert\") pod \"26266573-b6f1-495c-bc39-67bd77a5cb79\" (UID: \"26266573-b6f1-495c-bc39-67bd77a5cb79\") " Nov 03 01:17:21.625655 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:17:21.625625 4939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/26266573-b6f1-495c-bc39-67bd77a5cb79-istiod-ca-cert" pod "26266573-b6f1-495c-bc39-67bd77a5cb79" (UID: "26266573-b6f1-495c-bc39-67bd77a5cb79"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 01:17:21.626255 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:17:21.626229 4939 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/26266573-b6f1-495c-bc39-67bd77a5cb79-workload-certs\") on node \"ip-10-0-140-107.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:17:21.626378 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:17:21.626260 4939 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1240d677-9bc2-427c-a818-ffed78093771-istiod-ca-cert\") on node \"ip-10-0-140-107.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:17:21.626378 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:17:21.626276 4939 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/26266573-b6f1-495c-bc39-67bd77a5cb79-istio-data\") on node \"ip-10-0-140-107.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:17:21.626378 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:17:21.626291 4939 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1240d677-9bc2-427c-a818-ffed78093771-istio-podinfo\") on node \"ip-10-0-140-107.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:17:21.626378 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:17:21.626305 4939 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/26266573-b6f1-495c-bc39-67bd77a5cb79-istiod-ca-cert\") on node \"ip-10-0-140-107.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:17:21.626378 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:17:21.626319 4939 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/26266573-b6f1-495c-bc39-67bd77a5cb79-workload-socket\") on node \"ip-10-0-140-107.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:17:21.626378 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:17:21.626334 4939 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1240d677-9bc2-427c-a818-ffed78093771-istio-token\") on node \"ip-10-0-140-107.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:17:21.626378 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:17:21.626349 4939 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1240d677-9bc2-427c-a818-ffed78093771-istio-envoy\") on node \"ip-10-0-140-107.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:17:21.626378 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:17:21.626364 4939 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bjqz5\" (UniqueName: \"kubernetes.io/projected/1240d677-9bc2-427c-a818-ffed78093771-kube-api-access-bjqz5\") on node \"ip-10-0-140-107.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:17:21.626378 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:17:21.626378 4939 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/26266573-b6f1-495c-bc39-67bd77a5cb79-credential-socket\") on node \"ip-10-0-140-107.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:17:21.627105 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:17:21.627080 4939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/26266573-b6f1-495c-bc39-67bd77a5cb79-istio-podinfo" pod "26266573-b6f1-495c-bc39-67bd77a5cb79" (UID: "26266573-b6f1-495c-bc39-67bd77a5cb79"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 01:17:21.627208 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:17:21.627135 4939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/26266573-b6f1-495c-bc39-67bd77a5cb79-kube-api-access-82xm6" pod "26266573-b6f1-495c-bc39-67bd77a5cb79" (UID: "26266573-b6f1-495c-bc39-67bd77a5cb79"). InnerVolumeSpecName "kube-api-access-82xm6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 01:17:21.627208 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:17:21.627166 4939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/26266573-b6f1-495c-bc39-67bd77a5cb79-istio-token" pod "26266573-b6f1-495c-bc39-67bd77a5cb79" (UID: "26266573-b6f1-495c-bc39-67bd77a5cb79"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 01:17:21.627327 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:17:21.627219 4939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/26266573-b6f1-495c-bc39-67bd77a5cb79-istio-envoy" pod "26266573-b6f1-495c-bc39-67bd77a5cb79" (UID: "26266573-b6f1-495c-bc39-67bd77a5cb79"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 01:17:21.737230 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:17:21.731965 4939 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/26266573-b6f1-495c-bc39-67bd77a5cb79-istio-token\") on node \"ip-10-0-140-107.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:17:21.737230 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:17:21.732002 4939 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-82xm6\" (UniqueName: \"kubernetes.io/projected/26266573-b6f1-495c-bc39-67bd77a5cb79-kube-api-access-82xm6\") on node \"ip-10-0-140-107.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:17:21.737230 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:17:21.732019 4939 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/26266573-b6f1-495c-bc39-67bd77a5cb79-istio-podinfo\") on node \"ip-10-0-140-107.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:17:21.737230 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:17:21.732033 4939 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/26266573-b6f1-495c-bc39-67bd77a5cb79-istio-envoy\") on node \"ip-10-0-140-107.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:17:21.742970 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:17:21.742931 4939 generic.go:356] "Generic (PLEG): container finished" podID="1240d677-9bc2-427c-a818-ffed78093771" containerID="faaf9af48ecb5dfb77b38ea46d984d0db4cb5c35af3172caa13a02d92f04ed09" exitCode=0 Nov 03 01:17:21.743101 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:17:21.742995 4939 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w1-2729f2-openshift-default-9fb97b4fc-dmrf8" Nov 03 01:17:21.743347 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:17:21.743317 4939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w1-2729f2-openshift-default-9fb97b4fc-dmrf8" event={"ID":"1240d677-9bc2-427c-a818-ffed78093771","Type":"ContainerDied","Data":"faaf9af48ecb5dfb77b38ea46d984d0db4cb5c35af3172caa13a02d92f04ed09"} Nov 03 01:17:21.743422 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:17:21.743402 4939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w1-2729f2-openshift-default-9fb97b4fc-dmrf8" event={"ID":"1240d677-9bc2-427c-a818-ffed78093771","Type":"ContainerDied","Data":"03d129f0d6522b8feff3a8fca6acc345bc71be37632894c5c17f27e937b0071d"} Nov 03 01:17:21.743478 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:17:21.743441 4939 scope.go:122] "RemoveContainer" containerID="faaf9af48ecb5dfb77b38ea46d984d0db4cb5c35af3172caa13a02d92f04ed09" Nov 03 01:17:21.744406 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:17:21.744352 4939 generic.go:356] "Generic (PLEG): container finished" podID="26266573-b6f1-495c-bc39-67bd77a5cb79" containerID="584eabcda720f30f79164c8fd6aa77c704296dd98fa43e2075af7dbbe68ed6da" exitCode=0 Nov 03 01:17:21.744522 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:17:21.744496 4939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-717217-openshift-default-578c69554c-bpgmb" event={"ID":"26266573-b6f1-495c-bc39-67bd77a5cb79","Type":"ContainerDied","Data":"584eabcda720f30f79164c8fd6aa77c704296dd98fa43e2075af7dbbe68ed6da"} Nov 03 01:17:21.744591 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:17:21.744549 4939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-717217-openshift-default-578c69554c-bpgmb" event={"ID":"26266573-b6f1-495c-bc39-67bd77a5cb79","Type":"ContainerDied","Data":"aabf348544c06c6cae5a07d4cf857453228ce4a07e201fe13d5feaac04e34c02"} Nov 03 01:17:21.744661 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:17:21.744623 4939 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w3-717217-openshift-default-578c69554c-bpgmb" Nov 03 01:17:21.756520 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:17:21.756485 4939 scope.go:122] "RemoveContainer" containerID="faaf9af48ecb5dfb77b38ea46d984d0db4cb5c35af3172caa13a02d92f04ed09" Nov 03 01:17:21.756855 ip-10-0-140-107 kubenswrapper[4939]: E1103 01:17:21.756833 4939 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"faaf9af48ecb5dfb77b38ea46d984d0db4cb5c35af3172caa13a02d92f04ed09\": container with ID starting with faaf9af48ecb5dfb77b38ea46d984d0db4cb5c35af3172caa13a02d92f04ed09 not found: ID does not exist" containerID="faaf9af48ecb5dfb77b38ea46d984d0db4cb5c35af3172caa13a02d92f04ed09" Nov 03 01:17:21.756924 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:17:21.756866 4939 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"faaf9af48ecb5dfb77b38ea46d984d0db4cb5c35af3172caa13a02d92f04ed09"} err="failed to get container status \"faaf9af48ecb5dfb77b38ea46d984d0db4cb5c35af3172caa13a02d92f04ed09\": rpc error: code = NotFound desc = could not find container \"faaf9af48ecb5dfb77b38ea46d984d0db4cb5c35af3172caa13a02d92f04ed09\": container with ID starting with faaf9af48ecb5dfb77b38ea46d984d0db4cb5c35af3172caa13a02d92f04ed09 not found: ID does not exist" Nov 03 01:17:21.756924 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:17:21.756887 4939 scope.go:122] "RemoveContainer" containerID="584eabcda720f30f79164c8fd6aa77c704296dd98fa43e2075af7dbbe68ed6da" Nov 03 01:17:21.767882 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:17:21.767860 4939 scope.go:122] "RemoveContainer" containerID="584eabcda720f30f79164c8fd6aa77c704296dd98fa43e2075af7dbbe68ed6da" Nov 03 01:17:21.768197 ip-10-0-140-107 kubenswrapper[4939]: E1103 01:17:21.768177 4939 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"584eabcda720f30f79164c8fd6aa77c704296dd98fa43e2075af7dbbe68ed6da\": container with ID starting with 584eabcda720f30f79164c8fd6aa77c704296dd98fa43e2075af7dbbe68ed6da not found: ID does not exist" containerID="584eabcda720f30f79164c8fd6aa77c704296dd98fa43e2075af7dbbe68ed6da" Nov 03 01:17:21.768267 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:17:21.768207 4939 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"584eabcda720f30f79164c8fd6aa77c704296dd98fa43e2075af7dbbe68ed6da"} err="failed to get container status \"584eabcda720f30f79164c8fd6aa77c704296dd98fa43e2075af7dbbe68ed6da\": rpc error: code = NotFound desc = could not find container \"584eabcda720f30f79164c8fd6aa77c704296dd98fa43e2075af7dbbe68ed6da\": container with ID starting with 584eabcda720f30f79164c8fd6aa77c704296dd98fa43e2075af7dbbe68ed6da not found: ID does not exist" Nov 03 01:17:21.775955 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:17:21.775591 4939 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w1-2729f2-openshift-default-9fb97b4fc-dmrf8"] Nov 03 01:17:21.780922 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:17:21.780882 4939 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w1-2729f2-openshift-default-9fb97b4fc-dmrf8"] Nov 03 01:17:21.792303 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:17:21.792279 4939 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-717217-openshift-default-578c69554c-bpgmb"] Nov 03 01:17:21.796705 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:17:21.796681 4939 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-717217-openshift-default-578c69554c-bpgmb"] Nov 03 01:17:21.874873 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:17:21.874841 4939 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1240d677-9bc2-427c-a818-ffed78093771" path="/var/lib/kubelet/pods/1240d677-9bc2-427c-a818-ffed78093771/volumes" Nov 03 01:17:21.875578 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:17:21.875561 4939 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="26266573-b6f1-495c-bc39-67bd77a5cb79" path="/var/lib/kubelet/pods/26266573-b6f1-495c-bc39-67bd77a5cb79/volumes" Nov 03 01:17:21.902366 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:17:21.902339 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:17:22.567189 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:17:22.567109 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:17:23.216460 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:17:23.216421 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:17:23.917296 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:17:23.917264 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:17:24.591866 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:17:24.591833 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:17:25.258042 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:17:25.258006 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:17:25.918490 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:17:25.918457 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:17:26.091934 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:17:26.091891 4939 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-88bbf7-openshift-default-574f47fff7-b229j"] Nov 03 01:17:26.092140 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:17:26.092116 4939 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-88bbf7-openshift-default-574f47fff7-b229j" podUID="c7753242-f913-42a4-89d5-bc0ded362646" containerName="istio-proxy" containerID="cri-o://077d31059183a7267fe7e4c9d7ad94c6ae65ddb25bb63b933d33c400dd67bfa7" gracePeriod=30 Nov 03 01:17:27.291831 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:17:27.291797 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:17:27.610448 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:17:27.610408 4939 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-4pw7v"] Nov 03 01:17:28.120724 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:17:28.120681 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:17:29.283307 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:17:29.283275 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:17:29.881110 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:17:29.881080 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:17:30.641227 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:17:30.641198 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:17:31.179610 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:17:31.179568 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:17:31.202165 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:17:31.202131 4939 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w1-4af1f7-openshift-default-745ff98565-hbp26"] Nov 03 01:17:31.202432 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:17:31.202381 4939 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w1-4af1f7-openshift-default-745ff98565-hbp26" podUID="3cb597ee-3ac1-4a73-b28b-23d18233ae5a" containerName="istio-proxy" containerID="cri-o://ae1a7ad10b4adb6b06acf11c99ea11d87478206058f377362d02555f66a45c82" gracePeriod=30 Nov 03 01:17:31.298660 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:17:31.298581 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:17:31.854695 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:17:31.854072 4939 generic.go:356] "Generic (PLEG): container finished" podID="c7753242-f913-42a4-89d5-bc0ded362646" containerID="077d31059183a7267fe7e4c9d7ad94c6ae65ddb25bb63b933d33c400dd67bfa7" exitCode=0 Nov 03 01:17:31.854695 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:17:31.854306 4939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-88bbf7-openshift-default-574f47fff7-b229j" event={"ID":"c7753242-f913-42a4-89d5-bc0ded362646","Type":"ContainerDied","Data":"077d31059183a7267fe7e4c9d7ad94c6ae65ddb25bb63b933d33c400dd67bfa7"} Nov 03 01:17:31.854695 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:17:31.854358 4939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-88bbf7-openshift-default-574f47fff7-b229j" event={"ID":"c7753242-f913-42a4-89d5-bc0ded362646","Type":"ContainerDied","Data":"00d10162a6027715228d61dc08fbde06cd4a8e6ee8def426d6c70e804388444e"} Nov 03 01:17:31.854695 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:17:31.854381 4939 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="00d10162a6027715228d61dc08fbde06cd4a8e6ee8def426d6c70e804388444e" Nov 03 01:17:31.860366 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:17:31.860338 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:17:31.969930 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:17:31.963229 4939 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-88bbf7-openshift-default-574f47fff7-b229j" Nov 03 01:17:32.041461 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:17:32.041416 4939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/c7753242-f913-42a4-89d5-bc0ded362646-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c7753242-f913-42a4-89d5-bc0ded362646-istio-podinfo\") pod \"c7753242-f913-42a4-89d5-bc0ded362646\" (UID: \"c7753242-f913-42a4-89d5-bc0ded362646\") " Nov 03 01:17:32.041633 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:17:32.041511 4939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c7753242-f913-42a4-89d5-bc0ded362646-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c7753242-f913-42a4-89d5-bc0ded362646-istiod-ca-cert\") pod \"c7753242-f913-42a4-89d5-bc0ded362646\" (UID: \"c7753242-f913-42a4-89d5-bc0ded362646\") " Nov 03 01:17:32.041633 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:17:32.041587 4939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c7753242-f913-42a4-89d5-bc0ded362646-kube-api-access-6f9zg\" (UniqueName: \"kubernetes.io/projected/c7753242-f913-42a4-89d5-bc0ded362646-kube-api-access-6f9zg\") pod \"c7753242-f913-42a4-89d5-bc0ded362646\" (UID: \"c7753242-f913-42a4-89d5-bc0ded362646\") " Nov 03 01:17:32.041633 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:17:32.041624 4939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c7753242-f913-42a4-89d5-bc0ded362646-istio-token\" (UniqueName: \"kubernetes.io/projected/c7753242-f913-42a4-89d5-bc0ded362646-istio-token\") pod \"c7753242-f913-42a4-89d5-bc0ded362646\" (UID: \"c7753242-f913-42a4-89d5-bc0ded362646\") " Nov 03 01:17:32.041832 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:17:32.041673 4939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c7753242-f913-42a4-89d5-bc0ded362646-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c7753242-f913-42a4-89d5-bc0ded362646-istio-envoy\") pod \"c7753242-f913-42a4-89d5-bc0ded362646\" (UID: \"c7753242-f913-42a4-89d5-bc0ded362646\") " Nov 03 01:17:32.041832 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:17:32.041738 4939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c7753242-f913-42a4-89d5-bc0ded362646-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c7753242-f913-42a4-89d5-bc0ded362646-workload-socket\") pod \"c7753242-f913-42a4-89d5-bc0ded362646\" (UID: \"c7753242-f913-42a4-89d5-bc0ded362646\") " Nov 03 01:17:32.041985 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:17:32.041846 4939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c7753242-f913-42a4-89d5-bc0ded362646-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c7753242-f913-42a4-89d5-bc0ded362646-workload-certs\") pod \"c7753242-f913-42a4-89d5-bc0ded362646\" (UID: \"c7753242-f913-42a4-89d5-bc0ded362646\") " Nov 03 01:17:32.042049 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:17:32.042009 4939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c7753242-f913-42a4-89d5-bc0ded362646-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c7753242-f913-42a4-89d5-bc0ded362646-istio-data\") pod \"c7753242-f913-42a4-89d5-bc0ded362646\" (UID: \"c7753242-f913-42a4-89d5-bc0ded362646\") " Nov 03 01:17:32.042049 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:17:32.042030 4939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c7753242-f913-42a4-89d5-bc0ded362646-istiod-ca-cert" pod "c7753242-f913-42a4-89d5-bc0ded362646" (UID: "c7753242-f913-42a4-89d5-bc0ded362646"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 01:17:32.042159 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:17:32.042072 4939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c7753242-f913-42a4-89d5-bc0ded362646-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c7753242-f913-42a4-89d5-bc0ded362646-credential-socket\") pod \"c7753242-f913-42a4-89d5-bc0ded362646\" (UID: \"c7753242-f913-42a4-89d5-bc0ded362646\") " Nov 03 01:17:32.042159 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:17:32.042141 4939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c7753242-f913-42a4-89d5-bc0ded362646-workload-socket" pod "c7753242-f913-42a4-89d5-bc0ded362646" (UID: "c7753242-f913-42a4-89d5-bc0ded362646"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 01:17:32.042404 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:17:32.042375 4939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c7753242-f913-42a4-89d5-bc0ded362646-istio-data" pod "c7753242-f913-42a4-89d5-bc0ded362646" (UID: "c7753242-f913-42a4-89d5-bc0ded362646"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 01:17:32.042528 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:17:32.042411 4939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c7753242-f913-42a4-89d5-bc0ded362646-credential-socket" pod "c7753242-f913-42a4-89d5-bc0ded362646" (UID: "c7753242-f913-42a4-89d5-bc0ded362646"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 01:17:32.042528 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:17:32.042381 4939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c7753242-f913-42a4-89d5-bc0ded362646-workload-certs" pod "c7753242-f913-42a4-89d5-bc0ded362646" (UID: "c7753242-f913-42a4-89d5-bc0ded362646"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 01:17:32.043105 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:17:32.043079 4939 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c7753242-f913-42a4-89d5-bc0ded362646-workload-socket\") on node \"ip-10-0-140-107.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:17:32.043219 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:17:32.043108 4939 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c7753242-f913-42a4-89d5-bc0ded362646-workload-certs\") on node \"ip-10-0-140-107.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:17:32.043219 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:17:32.043124 4939 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c7753242-f913-42a4-89d5-bc0ded362646-istio-data\") on node \"ip-10-0-140-107.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:17:32.043219 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:17:32.043138 4939 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c7753242-f913-42a4-89d5-bc0ded362646-credential-socket\") on node \"ip-10-0-140-107.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:17:32.043219 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:17:32.043152 4939 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c7753242-f913-42a4-89d5-bc0ded362646-istiod-ca-cert\") on node \"ip-10-0-140-107.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:17:32.045186 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:17:32.045156 4939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/c7753242-f913-42a4-89d5-bc0ded362646-istio-podinfo" pod "c7753242-f913-42a4-89d5-bc0ded362646" (UID: "c7753242-f913-42a4-89d5-bc0ded362646"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 01:17:32.045651 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:17:32.045613 4939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c7753242-f913-42a4-89d5-bc0ded362646-istio-token" pod "c7753242-f913-42a4-89d5-bc0ded362646" (UID: "c7753242-f913-42a4-89d5-bc0ded362646"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 01:17:32.045743 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:17:32.045647 4939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c7753242-f913-42a4-89d5-bc0ded362646-istio-envoy" pod "c7753242-f913-42a4-89d5-bc0ded362646" (UID: "c7753242-f913-42a4-89d5-bc0ded362646"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 01:17:32.045743 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:17:32.045692 4939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c7753242-f913-42a4-89d5-bc0ded362646-kube-api-access-6f9zg" pod "c7753242-f913-42a4-89d5-bc0ded362646" (UID: "c7753242-f913-42a4-89d5-bc0ded362646"). InnerVolumeSpecName "kube-api-access-6f9zg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 01:17:32.061144 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:17:32.061122 4939 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-a-d25759/payload-processing-external-model-e2e-extmod-a-d25759-786cq4j8x"] Nov 03 01:17:32.062691 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:17:32.062663 4939 cpu_manager.go:414] "containerMap: removing container" podUID="26266573-b6f1-495c-bc39-67bd77a5cb79" containerName="istio-proxy" Nov 03 01:17:32.062691 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:17:32.062684 4939 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="26266573-b6f1-495c-bc39-67bd77a5cb79" containerName="istio-proxy" Nov 03 01:17:32.062866 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:17:32.062730 4939 cpu_manager.go:414] "containerMap: removing container" podUID="1240d677-9bc2-427c-a818-ffed78093771" containerName="istio-proxy" Nov 03 01:17:32.062866 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:17:32.062739 4939 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1240d677-9bc2-427c-a818-ffed78093771" containerName="istio-proxy" Nov 03 01:17:32.062866 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:17:32.062761 4939 cpu_manager.go:414] "containerMap: removing container" podUID="c7753242-f913-42a4-89d5-bc0ded362646" containerName="istio-proxy" Nov 03 01:17:32.062866 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:17:32.062768 4939 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c7753242-f913-42a4-89d5-bc0ded362646" containerName="istio-proxy" Nov 03 01:17:32.063139 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:17:32.063046 4939 memory_manager.go:365] "RemoveStaleState removing state" podUID="26266573-b6f1-495c-bc39-67bd77a5cb79" containerName="istio-proxy" Nov 03 01:17:32.063139 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:17:32.063068 4939 memory_manager.go:365] "RemoveStaleState removing state" podUID="1240d677-9bc2-427c-a818-ffed78093771" containerName="istio-proxy" Nov 03 01:17:32.063139 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:17:32.063080 4939 memory_manager.go:365] "RemoveStaleState removing state" podUID="c7753242-f913-42a4-89d5-bc0ded362646" containerName="istio-proxy" Nov 03 01:17:32.068145 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:17:32.068125 4939 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-d25759/payload-processing-external-model-e2e-extmod-a-d25759-786cq4j8x" Nov 03 01:17:32.073419 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:17:32.073397 4939 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-d25759\"/\"payload-processing-external-model-plugins-e2e-extmod-a-d25759\"" Nov 03 01:17:32.073688 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:17:32.073670 4939 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-d25759\"/\"routing-overlay\"" Nov 03 01:17:32.073756 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:17:32.073738 4939 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-d25759\"/\"agc-cred-213e03dcdac3\"" Nov 03 01:17:32.073819 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:17:32.073807 4939 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-d25759\"/\"openshift-service-ca.crt\"" Nov 03 01:17:32.073893 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:17:32.073844 4939 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-d25759\"/\"agc-cred-c8b171fd8537\"" Nov 03 01:17:32.073975 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:17:32.073939 4939 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-d25759\"/\"kube-root-ca.crt\"" Nov 03 01:17:32.074036 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:17:32.074013 4939 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-d25759\"/\"payload-processing-external-model-e2e--4d491765-dockercfg-btrfs\"" Nov 03 01:17:32.074096 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:17:32.074087 4939 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-d25759\"/\"e2e-cfl-e257d5-provider-cred\"" Nov 03 01:17:32.077215 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:17:32.077193 4939 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-d25759/payload-processing-external-model-e2e-extmod-a-d25759-786cq4j8x"] Nov 03 01:17:32.146370 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:17:32.146327 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2aa0d355-deff-4103-a850-ae4195eb3444-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-d25759-786cq4j8x\" (UID: \"2aa0d355-deff-4103-a850-ae4195eb3444\") " pod="ai-tenant-e2e-extmod-a-d25759/payload-processing-external-model-e2e-extmod-a-d25759-786cq4j8x" Nov 03 01:17:32.146778 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:17:32.146752 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rt7bx\" (UniqueName: \"kubernetes.io/projected/2aa0d355-deff-4103-a850-ae4195eb3444-kube-api-access-rt7bx\") pod \"payload-processing-external-model-e2e-extmod-a-d25759-786cq4j8x\" (UID: \"2aa0d355-deff-4103-a850-ae4195eb3444\") " pod="ai-tenant-e2e-extmod-a-d25759/payload-processing-external-model-e2e-extmod-a-d25759-786cq4j8x" Nov 03 01:17:32.146936 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:17:32.146791 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/2aa0d355-deff-4103-a850-ae4195eb3444-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-d25759-786cq4j8x\" (UID: \"2aa0d355-deff-4103-a850-ae4195eb3444\") " pod="ai-tenant-e2e-extmod-a-d25759/payload-processing-external-model-e2e-extmod-a-d25759-786cq4j8x" Nov 03 01:17:32.146936 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:17:32.146815 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/2aa0d355-deff-4103-a850-ae4195eb3444-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-d25759-786cq4j8x\" (UID: \"2aa0d355-deff-4103-a850-ae4195eb3444\") " pod="ai-tenant-e2e-extmod-a-d25759/payload-processing-external-model-e2e-extmod-a-d25759-786cq4j8x" Nov 03 01:17:32.147090 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:17:32.146998 4939 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c7753242-f913-42a4-89d5-bc0ded362646-istio-podinfo\") on node \"ip-10-0-140-107.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:17:32.147090 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:17:32.147019 4939 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6f9zg\" (UniqueName: \"kubernetes.io/projected/c7753242-f913-42a4-89d5-bc0ded362646-kube-api-access-6f9zg\") on node \"ip-10-0-140-107.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:17:32.147090 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:17:32.147035 4939 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c7753242-f913-42a4-89d5-bc0ded362646-istio-token\") on node \"ip-10-0-140-107.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:17:32.147090 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:17:32.147050 4939 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c7753242-f913-42a4-89d5-bc0ded362646-istio-envoy\") on node \"ip-10-0-140-107.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:17:32.249299 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:17:32.249255 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rt7bx\" (UniqueName: \"kubernetes.io/projected/2aa0d355-deff-4103-a850-ae4195eb3444-kube-api-access-rt7bx\") pod \"payload-processing-external-model-e2e-extmod-a-d25759-786cq4j8x\" (UID: \"2aa0d355-deff-4103-a850-ae4195eb3444\") " pod="ai-tenant-e2e-extmod-a-d25759/payload-processing-external-model-e2e-extmod-a-d25759-786cq4j8x" Nov 03 01:17:32.249527 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:17:32.249307 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/2aa0d355-deff-4103-a850-ae4195eb3444-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-d25759-786cq4j8x\" (UID: \"2aa0d355-deff-4103-a850-ae4195eb3444\") " pod="ai-tenant-e2e-extmod-a-d25759/payload-processing-external-model-e2e-extmod-a-d25759-786cq4j8x" Nov 03 01:17:32.249527 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:17:32.249331 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/2aa0d355-deff-4103-a850-ae4195eb3444-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-d25759-786cq4j8x\" (UID: \"2aa0d355-deff-4103-a850-ae4195eb3444\") " pod="ai-tenant-e2e-extmod-a-d25759/payload-processing-external-model-e2e-extmod-a-d25759-786cq4j8x" Nov 03 01:17:32.249527 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:17:32.249509 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2aa0d355-deff-4103-a850-ae4195eb3444-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-d25759-786cq4j8x\" (UID: \"2aa0d355-deff-4103-a850-ae4195eb3444\") " pod="ai-tenant-e2e-extmod-a-d25759/payload-processing-external-model-e2e-extmod-a-d25759-786cq4j8x" Nov 03 01:17:32.250691 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:17:32.250655 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/2aa0d355-deff-4103-a850-ae4195eb3444-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-d25759-786cq4j8x\" (UID: \"2aa0d355-deff-4103-a850-ae4195eb3444\") " pod="ai-tenant-e2e-extmod-a-d25759/payload-processing-external-model-e2e-extmod-a-d25759-786cq4j8x" Nov 03 01:17:32.250848 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:17:32.250730 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2aa0d355-deff-4103-a850-ae4195eb3444-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-d25759-786cq4j8x\" (UID: \"2aa0d355-deff-4103-a850-ae4195eb3444\") " pod="ai-tenant-e2e-extmod-a-d25759/payload-processing-external-model-e2e-extmod-a-d25759-786cq4j8x" Nov 03 01:17:32.254429 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:17:32.252519 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/2aa0d355-deff-4103-a850-ae4195eb3444-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-d25759-786cq4j8x\" (UID: \"2aa0d355-deff-4103-a850-ae4195eb3444\") " pod="ai-tenant-e2e-extmod-a-d25759/payload-processing-external-model-e2e-extmod-a-d25759-786cq4j8x" Nov 03 01:17:32.259545 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:17:32.259522 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rt7bx\" (UniqueName: \"kubernetes.io/projected/2aa0d355-deff-4103-a850-ae4195eb3444-kube-api-access-rt7bx\") pod \"payload-processing-external-model-e2e-extmod-a-d25759-786cq4j8x\" (UID: \"2aa0d355-deff-4103-a850-ae4195eb3444\") " pod="ai-tenant-e2e-extmod-a-d25759/payload-processing-external-model-e2e-extmod-a-d25759-786cq4j8x" Nov 03 01:17:32.411815 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:17:32.411729 4939 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-d25759/payload-processing-external-model-e2e-extmod-a-d25759-786cq4j8x" Nov 03 01:17:32.710980 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:17:32.710949 4939 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-d25759/payload-processing-external-model-e2e-extmod-a-d25759-786cq4j8x"] Nov 03 01:17:32.711123 ip-10-0-140-107 kubenswrapper[4939]: W1103 01:17:32.711079 4939 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2aa0d355_deff_4103_a850_ae4195eb3444.slice/crio-240da72a326b72e9ef10cf663a09a5b58fcb34fb66498bbff1b08970761e3fe0 WatchSource:0}: Error finding container 240da72a326b72e9ef10cf663a09a5b58fcb34fb66498bbff1b08970761e3fe0: Status 404 returned error can't find the container with id 240da72a326b72e9ef10cf663a09a5b58fcb34fb66498bbff1b08970761e3fe0 Nov 03 01:17:32.803537 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:17:32.803504 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:17:32.931514 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:17:32.931404 4939 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-88bbf7-openshift-default-574f47fff7-b229j" Nov 03 01:17:32.934085 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:17:32.933961 4939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-d25759/payload-processing-external-model-e2e-extmod-a-d25759-786cq4j8x" event={"ID":"2aa0d355-deff-4103-a850-ae4195eb3444","Type":"ContainerStarted","Data":"1cd9bff193390a7e9fa8e16c52fbef77abde734bd82ebae4f042e9086ba8d5b4"} Nov 03 01:17:32.934085 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:17:32.934037 4939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-d25759/payload-processing-external-model-e2e-extmod-a-d25759-786cq4j8x" event={"ID":"2aa0d355-deff-4103-a850-ae4195eb3444","Type":"ContainerStarted","Data":"240da72a326b72e9ef10cf663a09a5b58fcb34fb66498bbff1b08970761e3fe0"} Nov 03 01:17:32.934392 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:17:32.934354 4939 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-a-d25759/payload-processing-external-model-e2e-extmod-a-d25759-786cq4j8x" Nov 03 01:17:32.962830 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:17:32.962740 4939 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-a-d25759/payload-processing-external-model-e2e-extmod-a-d25759-786cq4j8x" podStartSLOduration=0.962727546 podStartE2EDuration="962.727546ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 01:17:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 01:17:32.957276471 +0000 UTC m=+2201.221611416" watchObservedRunningTime="2026-11-03 01:17:32.962727546 +0000 UTC m=+2201.227062492" Nov 03 01:17:32.975205 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:17:32.975146 4939 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-88bbf7-openshift-default-574f47fff7-b229j"] Nov 03 01:17:32.980043 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:17:32.980015 4939 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-88bbf7-openshift-default-574f47fff7-b229j"] Nov 03 01:17:33.476215 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:17:33.476183 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:17:33.874613 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:17:33.874578 4939 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c7753242-f913-42a4-89d5-bc0ded362646" path="/var/lib/kubelet/pods/c7753242-f913-42a4-89d5-bc0ded362646/volumes" Nov 03 01:17:34.111085 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:17:34.111055 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:17:34.446422 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:17:34.446385 4939 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w0-50e623-openshift-default-6c86dc794f-bwzqd"] Nov 03 01:17:34.446637 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:17:34.446573 4939 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w0-50e623-openshift-default-6c86dc794f-bwzqd" podUID="5d516568-4c21-441e-83e4-e3d87132eb96" containerName="istio-proxy" containerID="cri-o://250328b745dd6347bd3994e71869860e5648b6e2981fcaa81c6db4b680c6b239" gracePeriod=30 Nov 03 01:17:34.804342 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:17:34.804253 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:17:35.560144 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:17:35.560105 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:17:35.666031 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:17:35.665992 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:17:36.208039 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:17:36.208006 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:17:36.528513 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:17:36.528471 4939 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w1-4af1f7-openshift-default-745ff98565-hbp26" Nov 03 01:17:36.588535 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:17:36.588488 4939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/3cb597ee-3ac1-4a73-b28b-23d18233ae5a-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3cb597ee-3ac1-4a73-b28b-23d18233ae5a-istio-podinfo\") pod \"3cb597ee-3ac1-4a73-b28b-23d18233ae5a\" (UID: \"3cb597ee-3ac1-4a73-b28b-23d18233ae5a\") " Nov 03 01:17:36.589148 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:17:36.589119 4939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3cb597ee-3ac1-4a73-b28b-23d18233ae5a-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3cb597ee-3ac1-4a73-b28b-23d18233ae5a-workload-certs\") pod \"3cb597ee-3ac1-4a73-b28b-23d18233ae5a\" (UID: \"3cb597ee-3ac1-4a73-b28b-23d18233ae5a\") " Nov 03 01:17:36.589344 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:17:36.589324 4939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3cb597ee-3ac1-4a73-b28b-23d18233ae5a-istio-token\" (UniqueName: \"kubernetes.io/projected/3cb597ee-3ac1-4a73-b28b-23d18233ae5a-istio-token\") pod \"3cb597ee-3ac1-4a73-b28b-23d18233ae5a\" (UID: \"3cb597ee-3ac1-4a73-b28b-23d18233ae5a\") " Nov 03 01:17:36.589482 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:17:36.589466 4939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3cb597ee-3ac1-4a73-b28b-23d18233ae5a-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3cb597ee-3ac1-4a73-b28b-23d18233ae5a-istiod-ca-cert\") pod \"3cb597ee-3ac1-4a73-b28b-23d18233ae5a\" (UID: \"3cb597ee-3ac1-4a73-b28b-23d18233ae5a\") " Nov 03 01:17:36.589709 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:17:36.589689 4939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3cb597ee-3ac1-4a73-b28b-23d18233ae5a-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3cb597ee-3ac1-4a73-b28b-23d18233ae5a-workload-socket\") pod \"3cb597ee-3ac1-4a73-b28b-23d18233ae5a\" (UID: \"3cb597ee-3ac1-4a73-b28b-23d18233ae5a\") " Nov 03 01:17:36.589939 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:17:36.589898 4939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3cb597ee-3ac1-4a73-b28b-23d18233ae5a-kube-api-access-6gfqv\" (UniqueName: \"kubernetes.io/projected/3cb597ee-3ac1-4a73-b28b-23d18233ae5a-kube-api-access-6gfqv\") pod \"3cb597ee-3ac1-4a73-b28b-23d18233ae5a\" (UID: \"3cb597ee-3ac1-4a73-b28b-23d18233ae5a\") " Nov 03 01:17:36.590252 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:17:36.590230 4939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3cb597ee-3ac1-4a73-b28b-23d18233ae5a-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3cb597ee-3ac1-4a73-b28b-23d18233ae5a-credential-socket\") pod \"3cb597ee-3ac1-4a73-b28b-23d18233ae5a\" (UID: \"3cb597ee-3ac1-4a73-b28b-23d18233ae5a\") " Nov 03 01:17:36.592604 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:17:36.589402 4939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3cb597ee-3ac1-4a73-b28b-23d18233ae5a-workload-certs" pod "3cb597ee-3ac1-4a73-b28b-23d18233ae5a" (UID: "3cb597ee-3ac1-4a73-b28b-23d18233ae5a"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 01:17:36.592604 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:17:36.590550 4939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3cb597ee-3ac1-4a73-b28b-23d18233ae5a-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3cb597ee-3ac1-4a73-b28b-23d18233ae5a-istio-envoy\") pod \"3cb597ee-3ac1-4a73-b28b-23d18233ae5a\" (UID: \"3cb597ee-3ac1-4a73-b28b-23d18233ae5a\") " Nov 03 01:17:36.592604 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:17:36.590627 4939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3cb597ee-3ac1-4a73-b28b-23d18233ae5a-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3cb597ee-3ac1-4a73-b28b-23d18233ae5a-istio-data\") pod \"3cb597ee-3ac1-4a73-b28b-23d18233ae5a\" (UID: \"3cb597ee-3ac1-4a73-b28b-23d18233ae5a\") " Nov 03 01:17:36.592604 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:17:36.591710 4939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3cb597ee-3ac1-4a73-b28b-23d18233ae5a-istio-data" pod "3cb597ee-3ac1-4a73-b28b-23d18233ae5a" (UID: "3cb597ee-3ac1-4a73-b28b-23d18233ae5a"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 01:17:36.592604 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:17:36.591975 4939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3cb597ee-3ac1-4a73-b28b-23d18233ae5a-workload-socket" pod "3cb597ee-3ac1-4a73-b28b-23d18233ae5a" (UID: "3cb597ee-3ac1-4a73-b28b-23d18233ae5a"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 01:17:36.592604 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:17:36.591986 4939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3cb597ee-3ac1-4a73-b28b-23d18233ae5a-credential-socket" pod "3cb597ee-3ac1-4a73-b28b-23d18233ae5a" (UID: "3cb597ee-3ac1-4a73-b28b-23d18233ae5a"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 01:17:36.592604 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:17:36.592550 4939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3cb597ee-3ac1-4a73-b28b-23d18233ae5a-istiod-ca-cert" pod "3cb597ee-3ac1-4a73-b28b-23d18233ae5a" (UID: "3cb597ee-3ac1-4a73-b28b-23d18233ae5a"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 01:17:36.593223 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:17:36.593203 4939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/3cb597ee-3ac1-4a73-b28b-23d18233ae5a-istio-podinfo" pod "3cb597ee-3ac1-4a73-b28b-23d18233ae5a" (UID: "3cb597ee-3ac1-4a73-b28b-23d18233ae5a"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 01:17:36.598341 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:17:36.598314 4939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3cb597ee-3ac1-4a73-b28b-23d18233ae5a-istio-token" pod "3cb597ee-3ac1-4a73-b28b-23d18233ae5a" (UID: "3cb597ee-3ac1-4a73-b28b-23d18233ae5a"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 01:17:36.599255 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:17:36.599231 4939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3cb597ee-3ac1-4a73-b28b-23d18233ae5a-istio-envoy" pod "3cb597ee-3ac1-4a73-b28b-23d18233ae5a" (UID: "3cb597ee-3ac1-4a73-b28b-23d18233ae5a"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 01:17:36.600602 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:17:36.600576 4939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3cb597ee-3ac1-4a73-b28b-23d18233ae5a-kube-api-access-6gfqv" pod "3cb597ee-3ac1-4a73-b28b-23d18233ae5a" (UID: "3cb597ee-3ac1-4a73-b28b-23d18233ae5a"). InnerVolumeSpecName "kube-api-access-6gfqv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 01:17:36.693475 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:17:36.693436 4939 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3cb597ee-3ac1-4a73-b28b-23d18233ae5a-istio-podinfo\") on node \"ip-10-0-140-107.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:17:36.693475 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:17:36.693469 4939 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3cb597ee-3ac1-4a73-b28b-23d18233ae5a-workload-certs\") on node \"ip-10-0-140-107.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:17:36.693475 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:17:36.693478 4939 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3cb597ee-3ac1-4a73-b28b-23d18233ae5a-istio-token\") on node \"ip-10-0-140-107.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:17:36.693810 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:17:36.693487 4939 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3cb597ee-3ac1-4a73-b28b-23d18233ae5a-istiod-ca-cert\") on node \"ip-10-0-140-107.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:17:36.693810 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:17:36.693496 4939 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3cb597ee-3ac1-4a73-b28b-23d18233ae5a-workload-socket\") on node \"ip-10-0-140-107.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:17:36.693810 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:17:36.693504 4939 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6gfqv\" (UniqueName: \"kubernetes.io/projected/3cb597ee-3ac1-4a73-b28b-23d18233ae5a-kube-api-access-6gfqv\") on node \"ip-10-0-140-107.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:17:36.693810 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:17:36.693516 4939 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3cb597ee-3ac1-4a73-b28b-23d18233ae5a-credential-socket\") on node \"ip-10-0-140-107.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:17:36.693810 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:17:36.693525 4939 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3cb597ee-3ac1-4a73-b28b-23d18233ae5a-istio-envoy\") on node \"ip-10-0-140-107.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:17:36.693810 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:17:36.693533 4939 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3cb597ee-3ac1-4a73-b28b-23d18233ae5a-istio-data\") on node \"ip-10-0-140-107.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:17:36.909400 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:17:36.909370 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:17:36.986999 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:17:36.984983 4939 generic.go:356] "Generic (PLEG): container finished" podID="3cb597ee-3ac1-4a73-b28b-23d18233ae5a" containerID="ae1a7ad10b4adb6b06acf11c99ea11d87478206058f377362d02555f66a45c82" exitCode=0 Nov 03 01:17:36.986999 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:17:36.985347 4939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w1-4af1f7-openshift-default-745ff98565-hbp26" event={"ID":"3cb597ee-3ac1-4a73-b28b-23d18233ae5a","Type":"ContainerDied","Data":"ae1a7ad10b4adb6b06acf11c99ea11d87478206058f377362d02555f66a45c82"} Nov 03 01:17:36.986999 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:17:36.985421 4939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w1-4af1f7-openshift-default-745ff98565-hbp26" event={"ID":"3cb597ee-3ac1-4a73-b28b-23d18233ae5a","Type":"ContainerDied","Data":"cd821eb78fb574faa9856f7e3d1d16007a1937aec70baa8c0dc032ec08ee1958"} Nov 03 01:17:36.986999 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:17:36.985458 4939 scope.go:122] "RemoveContainer" containerID="ae1a7ad10b4adb6b06acf11c99ea11d87478206058f377362d02555f66a45c82" Nov 03 01:17:36.986999 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:17:36.985624 4939 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w1-4af1f7-openshift-default-745ff98565-hbp26" Nov 03 01:17:37.030749 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:17:37.030721 4939 scope.go:122] "RemoveContainer" containerID="ae1a7ad10b4adb6b06acf11c99ea11d87478206058f377362d02555f66a45c82" Nov 03 01:17:37.031180 ip-10-0-140-107 kubenswrapper[4939]: E1103 01:17:37.031149 4939 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ae1a7ad10b4adb6b06acf11c99ea11d87478206058f377362d02555f66a45c82\": container with ID starting with ae1a7ad10b4adb6b06acf11c99ea11d87478206058f377362d02555f66a45c82 not found: ID does not exist" containerID="ae1a7ad10b4adb6b06acf11c99ea11d87478206058f377362d02555f66a45c82" Nov 03 01:17:37.031304 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:17:37.031210 4939 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ae1a7ad10b4adb6b06acf11c99ea11d87478206058f377362d02555f66a45c82"} err="failed to get container status \"ae1a7ad10b4adb6b06acf11c99ea11d87478206058f377362d02555f66a45c82\": rpc error: code = NotFound desc = could not find container \"ae1a7ad10b4adb6b06acf11c99ea11d87478206058f377362d02555f66a45c82\": container with ID starting with ae1a7ad10b4adb6b06acf11c99ea11d87478206058f377362d02555f66a45c82 not found: ID does not exist" Nov 03 01:17:37.036013 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:17:37.035958 4939 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w1-4af1f7-openshift-default-745ff98565-hbp26"] Nov 03 01:17:37.040456 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:17:37.040436 4939 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w1-4af1f7-openshift-default-745ff98565-hbp26"] Nov 03 01:17:37.571476 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:17:37.571434 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:17:37.873533 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:17:37.873492 4939 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3cb597ee-3ac1-4a73-b28b-23d18233ae5a" path="/var/lib/kubelet/pods/3cb597ee-3ac1-4a73-b28b-23d18233ae5a/volumes" Nov 03 01:17:38.194141 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:17:38.194037 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:17:38.871794 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:17:38.871762 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:17:39.580487 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:17:39.580457 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:17:39.825388 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:17:39.825360 4939 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w0-50e623-openshift-default-6c86dc794f-bwzqd" Nov 03 01:17:39.890595 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:17:39.890558 4939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5d516568-4c21-441e-83e4-e3d87132eb96-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5d516568-4c21-441e-83e4-e3d87132eb96-istio-envoy\") pod \"5d516568-4c21-441e-83e4-e3d87132eb96\" (UID: \"5d516568-4c21-441e-83e4-e3d87132eb96\") " Nov 03 01:17:39.890770 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:17:39.890669 4939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5d516568-4c21-441e-83e4-e3d87132eb96-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5d516568-4c21-441e-83e4-e3d87132eb96-istiod-ca-cert\") pod \"5d516568-4c21-441e-83e4-e3d87132eb96\" (UID: \"5d516568-4c21-441e-83e4-e3d87132eb96\") " Nov 03 01:17:39.890770 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:17:39.890701 4939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5d516568-4c21-441e-83e4-e3d87132eb96-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5d516568-4c21-441e-83e4-e3d87132eb96-istio-data\") pod \"5d516568-4c21-441e-83e4-e3d87132eb96\" (UID: \"5d516568-4c21-441e-83e4-e3d87132eb96\") " Nov 03 01:17:39.890849 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:17:39.890772 4939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/5d516568-4c21-441e-83e4-e3d87132eb96-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5d516568-4c21-441e-83e4-e3d87132eb96-istio-podinfo\") pod \"5d516568-4c21-441e-83e4-e3d87132eb96\" (UID: \"5d516568-4c21-441e-83e4-e3d87132eb96\") " Nov 03 01:17:39.890849 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:17:39.890818 4939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5d516568-4c21-441e-83e4-e3d87132eb96-kube-api-access-wm588\" (UniqueName: \"kubernetes.io/projected/5d516568-4c21-441e-83e4-e3d87132eb96-kube-api-access-wm588\") pod \"5d516568-4c21-441e-83e4-e3d87132eb96\" (UID: \"5d516568-4c21-441e-83e4-e3d87132eb96\") " Nov 03 01:17:39.890970 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:17:39.890897 4939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5d516568-4c21-441e-83e4-e3d87132eb96-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5d516568-4c21-441e-83e4-e3d87132eb96-credential-socket\") pod \"5d516568-4c21-441e-83e4-e3d87132eb96\" (UID: \"5d516568-4c21-441e-83e4-e3d87132eb96\") " Nov 03 01:17:39.890970 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:17:39.890953 4939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5d516568-4c21-441e-83e4-e3d87132eb96-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5d516568-4c21-441e-83e4-e3d87132eb96-workload-certs\") pod \"5d516568-4c21-441e-83e4-e3d87132eb96\" (UID: \"5d516568-4c21-441e-83e4-e3d87132eb96\") " Nov 03 01:17:39.891107 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:17:39.890998 4939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5d516568-4c21-441e-83e4-e3d87132eb96-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5d516568-4c21-441e-83e4-e3d87132eb96-workload-socket\") pod \"5d516568-4c21-441e-83e4-e3d87132eb96\" (UID: \"5d516568-4c21-441e-83e4-e3d87132eb96\") " Nov 03 01:17:39.891107 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:17:39.891070 4939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5d516568-4c21-441e-83e4-e3d87132eb96-istio-token\" (UniqueName: \"kubernetes.io/projected/5d516568-4c21-441e-83e4-e3d87132eb96-istio-token\") pod \"5d516568-4c21-441e-83e4-e3d87132eb96\" (UID: \"5d516568-4c21-441e-83e4-e3d87132eb96\") " Nov 03 01:17:39.892862 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:17:39.892699 4939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5d516568-4c21-441e-83e4-e3d87132eb96-istiod-ca-cert" pod "5d516568-4c21-441e-83e4-e3d87132eb96" (UID: "5d516568-4c21-441e-83e4-e3d87132eb96"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 01:17:39.893184 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:17:39.893139 4939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5d516568-4c21-441e-83e4-e3d87132eb96-workload-certs" pod "5d516568-4c21-441e-83e4-e3d87132eb96" (UID: "5d516568-4c21-441e-83e4-e3d87132eb96"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 01:17:39.893288 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:17:39.893169 4939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5d516568-4c21-441e-83e4-e3d87132eb96-workload-socket" pod "5d516568-4c21-441e-83e4-e3d87132eb96" (UID: "5d516568-4c21-441e-83e4-e3d87132eb96"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 01:17:39.893443 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:17:39.893410 4939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5d516568-4c21-441e-83e4-e3d87132eb96-credential-socket" pod "5d516568-4c21-441e-83e4-e3d87132eb96" (UID: "5d516568-4c21-441e-83e4-e3d87132eb96"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 01:17:39.894644 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:17:39.894616 4939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/5d516568-4c21-441e-83e4-e3d87132eb96-istio-podinfo" pod "5d516568-4c21-441e-83e4-e3d87132eb96" (UID: "5d516568-4c21-441e-83e4-e3d87132eb96"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 01:17:39.895395 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:17:39.895364 4939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5d516568-4c21-441e-83e4-e3d87132eb96-istio-token" pod "5d516568-4c21-441e-83e4-e3d87132eb96" (UID: "5d516568-4c21-441e-83e4-e3d87132eb96"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 01:17:39.895514 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:17:39.895399 4939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5d516568-4c21-441e-83e4-e3d87132eb96-istio-envoy" pod "5d516568-4c21-441e-83e4-e3d87132eb96" (UID: "5d516568-4c21-441e-83e4-e3d87132eb96"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 01:17:39.895688 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:17:39.895643 4939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5d516568-4c21-441e-83e4-e3d87132eb96-kube-api-access-wm588" pod "5d516568-4c21-441e-83e4-e3d87132eb96" (UID: "5d516568-4c21-441e-83e4-e3d87132eb96"). InnerVolumeSpecName "kube-api-access-wm588". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 01:17:39.896116 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:17:39.896091 4939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5d516568-4c21-441e-83e4-e3d87132eb96-istio-data" pod "5d516568-4c21-441e-83e4-e3d87132eb96" (UID: "5d516568-4c21-441e-83e4-e3d87132eb96"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 01:17:39.993656 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:17:39.993614 4939 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5d516568-4c21-441e-83e4-e3d87132eb96-istio-podinfo\") on node \"ip-10-0-140-107.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:17:39.993656 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:17:39.993649 4939 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wm588\" (UniqueName: \"kubernetes.io/projected/5d516568-4c21-441e-83e4-e3d87132eb96-kube-api-access-wm588\") on node \"ip-10-0-140-107.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:17:39.993656 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:17:39.993661 4939 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5d516568-4c21-441e-83e4-e3d87132eb96-credential-socket\") on node \"ip-10-0-140-107.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:17:39.993928 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:17:39.993670 4939 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5d516568-4c21-441e-83e4-e3d87132eb96-workload-certs\") on node \"ip-10-0-140-107.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:17:39.993928 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:17:39.993679 4939 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5d516568-4c21-441e-83e4-e3d87132eb96-workload-socket\") on node \"ip-10-0-140-107.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:17:39.993928 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:17:39.993691 4939 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5d516568-4c21-441e-83e4-e3d87132eb96-istio-token\") on node \"ip-10-0-140-107.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:17:39.993928 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:17:39.993705 4939 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5d516568-4c21-441e-83e4-e3d87132eb96-istio-envoy\") on node \"ip-10-0-140-107.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:17:39.993928 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:17:39.993718 4939 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5d516568-4c21-441e-83e4-e3d87132eb96-istiod-ca-cert\") on node \"ip-10-0-140-107.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:17:39.993928 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:17:39.993732 4939 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5d516568-4c21-441e-83e4-e3d87132eb96-istio-data\") on node \"ip-10-0-140-107.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:17:40.016637 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:17:40.016604 4939 generic.go:356] "Generic (PLEG): container finished" podID="5d516568-4c21-441e-83e4-e3d87132eb96" containerID="250328b745dd6347bd3994e71869860e5648b6e2981fcaa81c6db4b680c6b239" exitCode=0 Nov 03 01:17:40.016824 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:17:40.016659 4939 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w0-50e623-openshift-default-6c86dc794f-bwzqd" Nov 03 01:17:40.016824 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:17:40.016665 4939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w0-50e623-openshift-default-6c86dc794f-bwzqd" event={"ID":"5d516568-4c21-441e-83e4-e3d87132eb96","Type":"ContainerDied","Data":"250328b745dd6347bd3994e71869860e5648b6e2981fcaa81c6db4b680c6b239"} Nov 03 01:17:40.016824 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:17:40.016785 4939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w0-50e623-openshift-default-6c86dc794f-bwzqd" event={"ID":"5d516568-4c21-441e-83e4-e3d87132eb96","Type":"ContainerDied","Data":"021d0d7ba09121178b4ee91aaf55a5a3718b2cff7af90ce99e84187ac5b50c3c"} Nov 03 01:17:40.016824 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:17:40.016805 4939 scope.go:122] "RemoveContainer" containerID="250328b745dd6347bd3994e71869860e5648b6e2981fcaa81c6db4b680c6b239" Nov 03 01:17:40.031846 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:17:40.031822 4939 scope.go:122] "RemoveContainer" containerID="250328b745dd6347bd3994e71869860e5648b6e2981fcaa81c6db4b680c6b239" Nov 03 01:17:40.032245 ip-10-0-140-107 kubenswrapper[4939]: E1103 01:17:40.032208 4939 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"250328b745dd6347bd3994e71869860e5648b6e2981fcaa81c6db4b680c6b239\": container with ID starting with 250328b745dd6347bd3994e71869860e5648b6e2981fcaa81c6db4b680c6b239 not found: ID does not exist" containerID="250328b745dd6347bd3994e71869860e5648b6e2981fcaa81c6db4b680c6b239" Nov 03 01:17:40.032357 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:17:40.032258 4939 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"250328b745dd6347bd3994e71869860e5648b6e2981fcaa81c6db4b680c6b239"} err="failed to get container status \"250328b745dd6347bd3994e71869860e5648b6e2981fcaa81c6db4b680c6b239\": rpc error: code = NotFound desc = could not find container \"250328b745dd6347bd3994e71869860e5648b6e2981fcaa81c6db4b680c6b239\": container with ID starting with 250328b745dd6347bd3994e71869860e5648b6e2981fcaa81c6db4b680c6b239 not found: ID does not exist" Nov 03 01:17:40.056242 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:17:40.056154 4939 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w0-50e623-openshift-default-6c86dc794f-bwzqd"] Nov 03 01:17:40.064118 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:17:40.064043 4939 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w0-50e623-openshift-default-6c86dc794f-bwzqd"] Nov 03 01:17:40.274769 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:17:40.274721 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:17:40.798490 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:17:40.798444 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:17:40.926944 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:17:40.917421 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:17:40.958032 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:17:40.957987 4939 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-b-3c5983/payload-processing-external-model-e2e-extmod-b-3c5983-7697gtj7m"] Nov 03 01:17:40.960342 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:17:40.960303 4939 cpu_manager.go:414] "containerMap: removing container" podUID="5d516568-4c21-441e-83e4-e3d87132eb96" containerName="istio-proxy" Nov 03 01:17:40.960522 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:17:40.960499 4939 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5d516568-4c21-441e-83e4-e3d87132eb96" containerName="istio-proxy" Nov 03 01:17:40.960698 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:17:40.960683 4939 cpu_manager.go:414] "containerMap: removing container" podUID="3cb597ee-3ac1-4a73-b28b-23d18233ae5a" containerName="istio-proxy" Nov 03 01:17:40.960809 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:17:40.960796 4939 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3cb597ee-3ac1-4a73-b28b-23d18233ae5a" containerName="istio-proxy" Nov 03 01:17:40.961235 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:17:40.961219 4939 memory_manager.go:365] "RemoveStaleState removing state" podUID="5d516568-4c21-441e-83e4-e3d87132eb96" containerName="istio-proxy" Nov 03 01:17:40.961383 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:17:40.961371 4939 memory_manager.go:365] "RemoveStaleState removing state" podUID="3cb597ee-3ac1-4a73-b28b-23d18233ae5a" containerName="istio-proxy" Nov 03 01:17:40.981501 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:17:40.979191 4939 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-3c5983/payload-processing-external-model-e2e-extmod-b-3c5983-7697gtj7m" Nov 03 01:17:40.998036 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:17:40.997997 4939 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-3c5983/payload-processing-external-model-e2e-extmod-b-3c5983-7697gtj7m"] Nov 03 01:17:41.009967 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:17:41.004337 4939 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-3c5983\"/\"openshift-service-ca.crt\"" Nov 03 01:17:41.009967 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:17:41.004630 4939 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-3c5983\"/\"agc-cred-803162a56fb2\"" Nov 03 01:17:41.009967 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:17:41.004701 4939 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-3c5983\"/\"kube-root-ca.crt\"" Nov 03 01:17:41.009967 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:17:41.004743 4939 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-3c5983\"/\"routing-overlay\"" Nov 03 01:17:41.009967 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:17:41.004799 4939 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-3c5983\"/\"payload-processing-external-model-e2e--ba2bb6eb-dockercfg-zkrj5\"" Nov 03 01:17:41.009967 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:17:41.004821 4939 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-3c5983\"/\"payload-processing-external-model-plugins-e2e-extmod-b-3c5983\"" Nov 03 01:17:41.074381 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:17:41.073399 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/dbf522b9-d27e-40eb-9e43-5dbfafa5ed18-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-3c5983-7697gtj7m\" (UID: \"dbf522b9-d27e-40eb-9e43-5dbfafa5ed18\") " pod="ai-tenant-e2e-extmod-b-3c5983/payload-processing-external-model-e2e-extmod-b-3c5983-7697gtj7m" Nov 03 01:17:41.074381 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:17:41.073466 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s4ht8\" (UniqueName: \"kubernetes.io/projected/dbf522b9-d27e-40eb-9e43-5dbfafa5ed18-kube-api-access-s4ht8\") pod \"payload-processing-external-model-e2e-extmod-b-3c5983-7697gtj7m\" (UID: \"dbf522b9-d27e-40eb-9e43-5dbfafa5ed18\") " pod="ai-tenant-e2e-extmod-b-3c5983/payload-processing-external-model-e2e-extmod-b-3c5983-7697gtj7m" Nov 03 01:17:41.074381 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:17:41.073689 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/dbf522b9-d27e-40eb-9e43-5dbfafa5ed18-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-3c5983-7697gtj7m\" (UID: \"dbf522b9-d27e-40eb-9e43-5dbfafa5ed18\") " pod="ai-tenant-e2e-extmod-b-3c5983/payload-processing-external-model-e2e-extmod-b-3c5983-7697gtj7m" Nov 03 01:17:41.074381 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:17:41.073750 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dbf522b9-d27e-40eb-9e43-5dbfafa5ed18-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-3c5983-7697gtj7m\" (UID: \"dbf522b9-d27e-40eb-9e43-5dbfafa5ed18\") " pod="ai-tenant-e2e-extmod-b-3c5983/payload-processing-external-model-e2e-extmod-b-3c5983-7697gtj7m" Nov 03 01:17:41.175167 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:17:41.175133 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/dbf522b9-d27e-40eb-9e43-5dbfafa5ed18-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-3c5983-7697gtj7m\" (UID: \"dbf522b9-d27e-40eb-9e43-5dbfafa5ed18\") " pod="ai-tenant-e2e-extmod-b-3c5983/payload-processing-external-model-e2e-extmod-b-3c5983-7697gtj7m" Nov 03 01:17:41.175359 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:17:41.175183 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s4ht8\" (UniqueName: \"kubernetes.io/projected/dbf522b9-d27e-40eb-9e43-5dbfafa5ed18-kube-api-access-s4ht8\") pod \"payload-processing-external-model-e2e-extmod-b-3c5983-7697gtj7m\" (UID: \"dbf522b9-d27e-40eb-9e43-5dbfafa5ed18\") " pod="ai-tenant-e2e-extmod-b-3c5983/payload-processing-external-model-e2e-extmod-b-3c5983-7697gtj7m" Nov 03 01:17:41.175359 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:17:41.175308 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/dbf522b9-d27e-40eb-9e43-5dbfafa5ed18-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-3c5983-7697gtj7m\" (UID: \"dbf522b9-d27e-40eb-9e43-5dbfafa5ed18\") " pod="ai-tenant-e2e-extmod-b-3c5983/payload-processing-external-model-e2e-extmod-b-3c5983-7697gtj7m" Nov 03 01:17:41.175359 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:17:41.175352 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dbf522b9-d27e-40eb-9e43-5dbfafa5ed18-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-3c5983-7697gtj7m\" (UID: \"dbf522b9-d27e-40eb-9e43-5dbfafa5ed18\") " pod="ai-tenant-e2e-extmod-b-3c5983/payload-processing-external-model-e2e-extmod-b-3c5983-7697gtj7m" Nov 03 01:17:41.176334 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:17:41.176304 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dbf522b9-d27e-40eb-9e43-5dbfafa5ed18-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-3c5983-7697gtj7m\" (UID: \"dbf522b9-d27e-40eb-9e43-5dbfafa5ed18\") " pod="ai-tenant-e2e-extmod-b-3c5983/payload-processing-external-model-e2e-extmod-b-3c5983-7697gtj7m" Nov 03 01:17:41.176670 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:17:41.176647 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/dbf522b9-d27e-40eb-9e43-5dbfafa5ed18-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-3c5983-7697gtj7m\" (UID: \"dbf522b9-d27e-40eb-9e43-5dbfafa5ed18\") " pod="ai-tenant-e2e-extmod-b-3c5983/payload-processing-external-model-e2e-extmod-b-3c5983-7697gtj7m" Nov 03 01:17:41.178152 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:17:41.178130 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/dbf522b9-d27e-40eb-9e43-5dbfafa5ed18-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-3c5983-7697gtj7m\" (UID: \"dbf522b9-d27e-40eb-9e43-5dbfafa5ed18\") " pod="ai-tenant-e2e-extmod-b-3c5983/payload-processing-external-model-e2e-extmod-b-3c5983-7697gtj7m" Nov 03 01:17:41.186208 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:17:41.186186 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-s4ht8\" (UniqueName: \"kubernetes.io/projected/dbf522b9-d27e-40eb-9e43-5dbfafa5ed18-kube-api-access-s4ht8\") pod \"payload-processing-external-model-e2e-extmod-b-3c5983-7697gtj7m\" (UID: \"dbf522b9-d27e-40eb-9e43-5dbfafa5ed18\") " pod="ai-tenant-e2e-extmod-b-3c5983/payload-processing-external-model-e2e-extmod-b-3c5983-7697gtj7m" Nov 03 01:17:41.371779 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:17:41.371747 4939 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-3c5983/payload-processing-external-model-e2e-extmod-b-3c5983-7697gtj7m" Nov 03 01:17:41.665814 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:17:41.665783 4939 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-3c5983/payload-processing-external-model-e2e-extmod-b-3c5983-7697gtj7m"] Nov 03 01:17:41.675139 ip-10-0-140-107 kubenswrapper[4939]: W1103 01:17:41.675107 4939 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddbf522b9_d27e_40eb_9e43_5dbfafa5ed18.slice/crio-4c4480feed69a615a4fa652aa60965448a5a88d7686512e157a98e3bf3eb0ad8 WatchSource:0}: Error finding container 4c4480feed69a615a4fa652aa60965448a5a88d7686512e157a98e3bf3eb0ad8: Status 404 returned error can't find the container with id 4c4480feed69a615a4fa652aa60965448a5a88d7686512e157a98e3bf3eb0ad8 Nov 03 01:17:41.759249 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:17:41.759222 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:17:41.914428 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:17:41.913854 4939 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5d516568-4c21-441e-83e4-e3d87132eb96" path="/var/lib/kubelet/pods/5d516568-4c21-441e-83e4-e3d87132eb96/volumes" Nov 03 01:17:42.073802 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:17:42.073767 4939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-3c5983/payload-processing-external-model-e2e-extmod-b-3c5983-7697gtj7m" event={"ID":"dbf522b9-d27e-40eb-9e43-5dbfafa5ed18","Type":"ContainerStarted","Data":"ddce77c88fd8799ae6005554fc1e620ee81fcd48d69ee45e12fb264c241cb437"} Nov 03 01:17:42.074011 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:17:42.073825 4939 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-b-3c5983/payload-processing-external-model-e2e-extmod-b-3c5983-7697gtj7m" Nov 03 01:17:42.074011 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:17:42.073842 4939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-3c5983/payload-processing-external-model-e2e-extmod-b-3c5983-7697gtj7m" event={"ID":"dbf522b9-d27e-40eb-9e43-5dbfafa5ed18","Type":"ContainerStarted","Data":"4c4480feed69a615a4fa652aa60965448a5a88d7686512e157a98e3bf3eb0ad8"} Nov 03 01:17:42.102050 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:17:42.101995 4939 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-b-3c5983/payload-processing-external-model-e2e-extmod-b-3c5983-7697gtj7m" podStartSLOduration=2.101981283 podStartE2EDuration="2.101981283s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 01:17:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 01:17:42.097665714 +0000 UTC m=+2210.362000659" watchObservedRunningTime="2026-11-03 01:17:42.101981283 +0000 UTC m=+2210.366316228" Nov 03 01:17:42.706931 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:17:42.700554 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:17:42.849598 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:17:42.849561 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:17:43.695418 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:17:43.695382 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:17:43.944559 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:17:43.944524 4939 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-a-d25759/payload-processing-external-model-e2e-extmod-a-d25759-786cq4j8x" Nov 03 01:17:44.166840 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:17:44.166808 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:17:44.283329 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:17:44.283293 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:17:44.513536 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:17:44.513457 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:17:45.206645 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:17:45.206610 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:17:45.912517 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:17:45.912485 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:17:46.495936 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:17:46.492396 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:17:47.390254 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:17:47.390176 4939 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-88bbf7/distinct-2-w1-88bbf7-kserve-5457cdcdbd-l6tfd" podUID="a397931b-faa9-4c2e-8290-491350546663" containerName="main" probeResult="failure" output="Get \"https://10.132.0.85:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Nov 03 01:17:47.610783 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:17:47.610751 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:17:47.726819 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:17:47.726735 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:17:47.923987 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:17:47.923939 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:17:48.158491 ip-10-0-140-107 kubenswrapper[4939]: E1103 01:17:48.158162 4939 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-sthjh" containerName="extractor" Nov 03 01:17:48.210652 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:17:48.210606 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:17:48.345167 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:17:48.345135 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:17:48.469076 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:17:48.468971 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:17:48.921260 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:17:48.921229 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:17:49.044338 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:17:49.044310 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:17:49.328690 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:17:49.328601 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:17:49.457705 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:17:49.457670 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:17:49.512538 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:17:49.512492 4939 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-016f9241-fngh8"] Nov 03 01:17:49.524598 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:17:49.524566 4939 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-016f9241-fngh8"] Nov 03 01:17:49.524792 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:17:49.524671 4939 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-016f9241-fngh8" Nov 03 01:17:49.582851 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:17:49.582817 4939 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 01:17:49.583099 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:17:49.582985 4939 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-6q4xp\"" Nov 03 01:17:49.610977 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:17:49.610800 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bxvd2\" (UniqueName: \"kubernetes.io/projected/6905cd74-619b-4184-a706-08d4722436c9-kube-api-access-bxvd2\") pod \"maas-api-revoke-sub-016f9241-fngh8\" (UID: \"6905cd74-619b-4184-a706-08d4722436c9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-016f9241-fngh8" Nov 03 01:17:49.611161 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:17:49.611029 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6905cd74-619b-4184-a706-08d4722436c9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-016f9241-fngh8\" (UID: \"6905cd74-619b-4184-a706-08d4722436c9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-016f9241-fngh8" Nov 03 01:17:49.611323 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:17:49.611302 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6905cd74-619b-4184-a706-08d4722436c9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-016f9241-fngh8\" (UID: \"6905cd74-619b-4184-a706-08d4722436c9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-016f9241-fngh8" Nov 03 01:17:49.712451 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:17:49.712416 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6905cd74-619b-4184-a706-08d4722436c9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-016f9241-fngh8\" (UID: \"6905cd74-619b-4184-a706-08d4722436c9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-016f9241-fngh8" Nov 03 01:17:49.712623 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:17:49.712565 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bxvd2\" (UniqueName: \"kubernetes.io/projected/6905cd74-619b-4184-a706-08d4722436c9-kube-api-access-bxvd2\") pod \"maas-api-revoke-sub-016f9241-fngh8\" (UID: \"6905cd74-619b-4184-a706-08d4722436c9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-016f9241-fngh8" Nov 03 01:17:49.712706 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:17:49.712625 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6905cd74-619b-4184-a706-08d4722436c9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-016f9241-fngh8\" (UID: \"6905cd74-619b-4184-a706-08d4722436c9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-016f9241-fngh8" Nov 03 01:17:49.713238 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:17:49.713216 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6905cd74-619b-4184-a706-08d4722436c9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-016f9241-fngh8\" (UID: \"6905cd74-619b-4184-a706-08d4722436c9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-016f9241-fngh8" Nov 03 01:17:49.715223 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:17:49.715201 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6905cd74-619b-4184-a706-08d4722436c9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-016f9241-fngh8\" (UID: \"6905cd74-619b-4184-a706-08d4722436c9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-016f9241-fngh8" Nov 03 01:17:49.721006 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:17:49.720975 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bxvd2\" (UniqueName: \"kubernetes.io/projected/6905cd74-619b-4184-a706-08d4722436c9-kube-api-access-bxvd2\") pod \"maas-api-revoke-sub-016f9241-fngh8\" (UID: \"6905cd74-619b-4184-a706-08d4722436c9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-016f9241-fngh8" Nov 03 01:17:49.902898 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:17:49.902820 4939 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-016f9241-fngh8" Nov 03 01:17:50.074610 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:17:50.074559 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:17:50.094314 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:17:50.094276 4939 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-b-be59df/payload-processing-external-model-e2e-extmod-b-be59df-5885fq5qq"] Nov 03 01:17:50.100362 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:17:50.100340 4939 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-be59df/payload-processing-external-model-e2e-extmod-b-be59df-5885fq5qq" Nov 03 01:17:50.126018 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:17:50.125761 4939 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-be59df/payload-processing-external-model-e2e-extmod-b-be59df-5885fq5qq"] Nov 03 01:17:50.126126 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:17:50.126065 4939 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-be59df\"/\"agc-cred-b25395d0face\"" Nov 03 01:17:50.126505 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:17:50.126322 4939 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-be59df\"/\"kube-root-ca.crt\"" Nov 03 01:17:50.126505 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:17:50.126355 4939 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-be59df\"/\"routing-overlay\"" Nov 03 01:17:50.126505 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:17:50.126460 4939 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-be59df\"/\"openshift-service-ca.crt\"" Nov 03 01:17:50.126505 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:17:50.126467 4939 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-be59df\"/\"payload-processing-external-model-plugins-e2e-extmod-b-be59df\"" Nov 03 01:17:50.126795 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:17:50.126637 4939 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-be59df\"/\"payload-processing-external-model-e2e--f68ff7cb-dockercfg-c9kdm\"" Nov 03 01:17:50.198131 ip-10-0-140-107 kubenswrapper[4939]: W1103 01:17:50.198103 4939 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6905cd74_619b_4184_a706_08d4722436c9.slice/crio-7ee3329289255460e7bd8fbc307893128af5c88be39e996105d97a0bf2ff8002 WatchSource:0}: Error finding container 7ee3329289255460e7bd8fbc307893128af5c88be39e996105d97a0bf2ff8002: Status 404 returned error can't find the container with id 7ee3329289255460e7bd8fbc307893128af5c88be39e996105d97a0bf2ff8002 Nov 03 01:17:50.201309 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:17:50.201286 4939 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-016f9241-fngh8"] Nov 03 01:17:50.204434 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:17:50.204410 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:17:50.301944 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:17:50.281817 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1e022297-33c3-41ab-a32f-ab20ef6676c8-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-be59df-5885fq5qq\" (UID: \"1e022297-33c3-41ab-a32f-ab20ef6676c8\") " pod="ai-tenant-e2e-extmod-b-be59df/payload-processing-external-model-e2e-extmod-b-be59df-5885fq5qq" Nov 03 01:17:50.301944 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:17:50.281945 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j9cwv\" (UniqueName: \"kubernetes.io/projected/1e022297-33c3-41ab-a32f-ab20ef6676c8-kube-api-access-j9cwv\") pod \"payload-processing-external-model-e2e-extmod-b-be59df-5885fq5qq\" (UID: \"1e022297-33c3-41ab-a32f-ab20ef6676c8\") " pod="ai-tenant-e2e-extmod-b-be59df/payload-processing-external-model-e2e-extmod-b-be59df-5885fq5qq" Nov 03 01:17:50.301944 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:17:50.282263 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/1e022297-33c3-41ab-a32f-ab20ef6676c8-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-be59df-5885fq5qq\" (UID: \"1e022297-33c3-41ab-a32f-ab20ef6676c8\") " pod="ai-tenant-e2e-extmod-b-be59df/payload-processing-external-model-e2e-extmod-b-be59df-5885fq5qq" Nov 03 01:17:50.301944 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:17:50.285942 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/1e022297-33c3-41ab-a32f-ab20ef6676c8-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-be59df-5885fq5qq\" (UID: \"1e022297-33c3-41ab-a32f-ab20ef6676c8\") " pod="ai-tenant-e2e-extmod-b-be59df/payload-processing-external-model-e2e-extmod-b-be59df-5885fq5qq" Nov 03 01:17:50.405951 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:17:50.397631 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/1e022297-33c3-41ab-a32f-ab20ef6676c8-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-be59df-5885fq5qq\" (UID: \"1e022297-33c3-41ab-a32f-ab20ef6676c8\") " pod="ai-tenant-e2e-extmod-b-be59df/payload-processing-external-model-e2e-extmod-b-be59df-5885fq5qq" Nov 03 01:17:50.405951 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:17:50.397695 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/1e022297-33c3-41ab-a32f-ab20ef6676c8-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-be59df-5885fq5qq\" (UID: \"1e022297-33c3-41ab-a32f-ab20ef6676c8\") " pod="ai-tenant-e2e-extmod-b-be59df/payload-processing-external-model-e2e-extmod-b-be59df-5885fq5qq" Nov 03 01:17:50.405951 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:17:50.397969 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1e022297-33c3-41ab-a32f-ab20ef6676c8-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-be59df-5885fq5qq\" (UID: \"1e022297-33c3-41ab-a32f-ab20ef6676c8\") " pod="ai-tenant-e2e-extmod-b-be59df/payload-processing-external-model-e2e-extmod-b-be59df-5885fq5qq" Nov 03 01:17:50.405951 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:17:50.398039 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j9cwv\" (UniqueName: \"kubernetes.io/projected/1e022297-33c3-41ab-a32f-ab20ef6676c8-kube-api-access-j9cwv\") pod \"payload-processing-external-model-e2e-extmod-b-be59df-5885fq5qq\" (UID: \"1e022297-33c3-41ab-a32f-ab20ef6676c8\") " pod="ai-tenant-e2e-extmod-b-be59df/payload-processing-external-model-e2e-extmod-b-be59df-5885fq5qq" Nov 03 01:17:50.409938 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:17:50.408674 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/1e022297-33c3-41ab-a32f-ab20ef6676c8-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-be59df-5885fq5qq\" (UID: \"1e022297-33c3-41ab-a32f-ab20ef6676c8\") " pod="ai-tenant-e2e-extmod-b-be59df/payload-processing-external-model-e2e-extmod-b-be59df-5885fq5qq" Nov 03 01:17:50.426968 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:17:50.420675 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1e022297-33c3-41ab-a32f-ab20ef6676c8-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-be59df-5885fq5qq\" (UID: \"1e022297-33c3-41ab-a32f-ab20ef6676c8\") " pod="ai-tenant-e2e-extmod-b-be59df/payload-processing-external-model-e2e-extmod-b-be59df-5885fq5qq" Nov 03 01:17:50.458007 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:17:50.445588 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/1e022297-33c3-41ab-a32f-ab20ef6676c8-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-be59df-5885fq5qq\" (UID: \"1e022297-33c3-41ab-a32f-ab20ef6676c8\") " pod="ai-tenant-e2e-extmod-b-be59df/payload-processing-external-model-e2e-extmod-b-be59df-5885fq5qq" Nov 03 01:17:50.476444 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:17:50.471873 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-j9cwv\" (UniqueName: \"kubernetes.io/projected/1e022297-33c3-41ab-a32f-ab20ef6676c8-kube-api-access-j9cwv\") pod \"payload-processing-external-model-e2e-extmod-b-be59df-5885fq5qq\" (UID: \"1e022297-33c3-41ab-a32f-ab20ef6676c8\") " pod="ai-tenant-e2e-extmod-b-be59df/payload-processing-external-model-e2e-extmod-b-be59df-5885fq5qq" Nov 03 01:17:50.560848 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:17:50.560801 4939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-016f9241-fngh8" event={"ID":"6905cd74-619b-4184-a706-08d4722436c9","Type":"ContainerStarted","Data":"7ee3329289255460e7bd8fbc307893128af5c88be39e996105d97a0bf2ff8002"} Nov 03 01:17:50.739556 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:17:50.739466 4939 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-be59df/payload-processing-external-model-e2e-extmod-b-be59df-5885fq5qq" Nov 03 01:17:51.051742 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:17:51.051654 4939 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-be59df/payload-processing-external-model-e2e-extmod-b-be59df-5885fq5qq"] Nov 03 01:17:51.058275 ip-10-0-140-107 kubenswrapper[4939]: W1103 01:17:51.058244 4939 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1e022297_33c3_41ab_a32f_ab20ef6676c8.slice/crio-d20309aec5bb7cdd757d769082e9fa9dcd708ff4b2e4a88bbb7b5f18a1100e99 WatchSource:0}: Error finding container d20309aec5bb7cdd757d769082e9fa9dcd708ff4b2e4a88bbb7b5f18a1100e99: Status 404 returned error can't find the container with id d20309aec5bb7cdd757d769082e9fa9dcd708ff4b2e4a88bbb7b5f18a1100e99 Nov 03 01:17:51.099165 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:17:51.099138 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:17:51.572277 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:17:51.572236 4939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-be59df/payload-processing-external-model-e2e-extmod-b-be59df-5885fq5qq" event={"ID":"1e022297-33c3-41ab-a32f-ab20ef6676c8","Type":"ContainerStarted","Data":"3a372aaea621d5acf2cc1c5655a8034526c6b483feea35604ee9d711dda91c6e"} Nov 03 01:17:51.572440 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:17:51.572311 4939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-be59df/payload-processing-external-model-e2e-extmod-b-be59df-5885fq5qq" event={"ID":"1e022297-33c3-41ab-a32f-ab20ef6676c8","Type":"ContainerStarted","Data":"d20309aec5bb7cdd757d769082e9fa9dcd708ff4b2e4a88bbb7b5f18a1100e99"} Nov 03 01:17:51.572440 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:17:51.572343 4939 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-b-be59df/payload-processing-external-model-e2e-extmod-b-be59df-5885fq5qq" Nov 03 01:17:51.574281 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:17:51.574254 4939 generic.go:356] "Generic (PLEG): container finished" podID="6905cd74-619b-4184-a706-08d4722436c9" containerID="a970713b526550134675994ce29fd24b548b80a5d39b73954a26f0228139d60a" exitCode=0 Nov 03 01:17:51.574397 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:17:51.574334 4939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-016f9241-fngh8" event={"ID":"6905cd74-619b-4184-a706-08d4722436c9","Type":"ContainerDied","Data":"a970713b526550134675994ce29fd24b548b80a5d39b73954a26f0228139d60a"} Nov 03 01:17:51.591620 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:17:51.591576 4939 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-b-be59df/payload-processing-external-model-e2e-extmod-b-be59df-5885fq5qq" podStartSLOduration=1.5915635049999999 podStartE2EDuration="1.591563505s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 01:17:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 01:17:51.588104028 +0000 UTC m=+2219.852438973" watchObservedRunningTime="2026-11-03 01:17:51.591563505 +0000 UTC m=+2219.855898449" Nov 03 01:17:51.925310 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:17:51.925274 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:17:52.574821 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:17:52.574785 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:17:52.843798 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:17:52.843768 4939 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-016f9241-fngh8" Nov 03 01:17:52.928370 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:17:52.928341 4939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6905cd74-619b-4184-a706-08d4722436c9-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6905cd74-619b-4184-a706-08d4722436c9-maas-api-openshift-ca\") pod \"6905cd74-619b-4184-a706-08d4722436c9\" (UID: \"6905cd74-619b-4184-a706-08d4722436c9\") " Nov 03 01:17:52.928600 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:17:52.928584 4939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6905cd74-619b-4184-a706-08d4722436c9-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6905cd74-619b-4184-a706-08d4722436c9-maas-api-xks-ca\") pod \"6905cd74-619b-4184-a706-08d4722436c9\" (UID: \"6905cd74-619b-4184-a706-08d4722436c9\") " Nov 03 01:17:52.928659 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:17:52.928623 4939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6905cd74-619b-4184-a706-08d4722436c9-kube-api-access-bxvd2\" (UniqueName: \"kubernetes.io/projected/6905cd74-619b-4184-a706-08d4722436c9-kube-api-access-bxvd2\") pod \"6905cd74-619b-4184-a706-08d4722436c9\" (UID: \"6905cd74-619b-4184-a706-08d4722436c9\") " Nov 03 01:17:52.928724 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:17:52.928705 4939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6905cd74-619b-4184-a706-08d4722436c9-maas-api-openshift-ca" pod "6905cd74-619b-4184-a706-08d4722436c9" (UID: "6905cd74-619b-4184-a706-08d4722436c9"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 01:17:52.929570 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:17:52.929542 4939 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6905cd74-619b-4184-a706-08d4722436c9-maas-api-openshift-ca\") on node \"ip-10-0-140-107.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:17:52.931143 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:17:52.931117 4939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6905cd74-619b-4184-a706-08d4722436c9-kube-api-access-bxvd2" pod "6905cd74-619b-4184-a706-08d4722436c9" (UID: "6905cd74-619b-4184-a706-08d4722436c9"). InnerVolumeSpecName "kube-api-access-bxvd2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 01:17:52.931228 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:17:52.931139 4939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6905cd74-619b-4184-a706-08d4722436c9-maas-api-xks-ca" pod "6905cd74-619b-4184-a706-08d4722436c9" (UID: "6905cd74-619b-4184-a706-08d4722436c9"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 01:17:53.031483 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:17:53.031453 4939 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6905cd74-619b-4184-a706-08d4722436c9-maas-api-xks-ca\") on node \"ip-10-0-140-107.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:17:53.031483 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:17:53.031483 4939 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bxvd2\" (UniqueName: \"kubernetes.io/projected/6905cd74-619b-4184-a706-08d4722436c9-kube-api-access-bxvd2\") on node \"ip-10-0-140-107.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:17:53.086478 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:17:53.086448 4939 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-b-3c5983/payload-processing-external-model-e2e-extmod-b-3c5983-7697gtj7m" Nov 03 01:17:53.179076 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:17:53.178984 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:17:53.591637 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:17:53.591607 4939 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-016f9241-fngh8" Nov 03 01:17:53.592191 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:17:53.591641 4939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-016f9241-fngh8" event={"ID":"6905cd74-619b-4184-a706-08d4722436c9","Type":"ContainerDied","Data":"7ee3329289255460e7bd8fbc307893128af5c88be39e996105d97a0bf2ff8002"} Nov 03 01:17:53.592191 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:17:53.591696 4939 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="7ee3329289255460e7bd8fbc307893128af5c88be39e996105d97a0bf2ff8002" Nov 03 01:17:53.793039 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:17:53.793005 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:17:54.377917 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:17:54.377882 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:17:54.991353 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:17:54.991321 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:17:55.714519 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:17:55.714481 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:17:56.369797 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:17:56.369765 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:17:56.714307 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:17:56.714226 4939 scope.go:122] "RemoveContainer" containerID="077d31059183a7267fe7e4c9d7ad94c6ae65ddb25bb63b933d33c400dd67bfa7" Nov 03 01:17:57.083777 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:17:57.083741 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:17:57.699211 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:17:57.699170 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:17:58.372561 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:17:58.372531 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:17:59.019122 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:17:59.019088 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:17:59.680805 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:17:59.680776 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:17:59.895379 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:17:59.895307 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:18:00.002400 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:18:00.002319 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:18:00.150708 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:18:00.150640 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:18:00.612964 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:18:00.612931 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:18:00.736665 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:18:00.736635 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:18:00.900722 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:18:00.900653 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:18:00.995196 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:18:00.995150 4939 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b20dc57a-hc8mf"] Nov 03 01:18:00.996710 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:18:00.996669 4939 cpu_manager.go:414] "containerMap: removing container" podUID="6905cd74-619b-4184-a706-08d4722436c9" containerName="revoke-keys" Nov 03 01:18:00.996933 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:18:00.996875 4939 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6905cd74-619b-4184-a706-08d4722436c9" containerName="revoke-keys" Nov 03 01:18:00.997180 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:18:00.997160 4939 memory_manager.go:365] "RemoveStaleState removing state" podUID="6905cd74-619b-4184-a706-08d4722436c9" containerName="revoke-keys" Nov 03 01:18:01.012034 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:18:01.012007 4939 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b20dc57a-hc8mf"] Nov 03 01:18:01.012312 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:18:01.012290 4939 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b20dc57a-hc8mf" Nov 03 01:18:01.024524 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:18:01.024487 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:18:01.087839 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:18:01.087805 4939 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 01:18:01.088027 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:18:01.087842 4939 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-6q4xp\"" Nov 03 01:18:01.133889 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:18:01.133856 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:18:01.161864 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:18:01.161786 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4zsrs\" (UniqueName: \"kubernetes.io/projected/9a9131c6-08db-48a1-a356-e4a19c077c22-kube-api-access-4zsrs\") pod \"maas-api-revoke-sub-b20dc57a-hc8mf\" (UID: \"9a9131c6-08db-48a1-a356-e4a19c077c22\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b20dc57a-hc8mf" Nov 03 01:18:01.162251 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:18:01.162022 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9a9131c6-08db-48a1-a356-e4a19c077c22-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b20dc57a-hc8mf\" (UID: \"9a9131c6-08db-48a1-a356-e4a19c077c22\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b20dc57a-hc8mf" Nov 03 01:18:01.162251 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:18:01.162051 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9a9131c6-08db-48a1-a356-e4a19c077c22-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b20dc57a-hc8mf\" (UID: \"9a9131c6-08db-48a1-a356-e4a19c077c22\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b20dc57a-hc8mf" Nov 03 01:18:01.263579 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:18:01.263547 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4zsrs\" (UniqueName: \"kubernetes.io/projected/9a9131c6-08db-48a1-a356-e4a19c077c22-kube-api-access-4zsrs\") pod \"maas-api-revoke-sub-b20dc57a-hc8mf\" (UID: \"9a9131c6-08db-48a1-a356-e4a19c077c22\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b20dc57a-hc8mf" Nov 03 01:18:01.263800 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:18:01.263778 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9a9131c6-08db-48a1-a356-e4a19c077c22-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b20dc57a-hc8mf\" (UID: \"9a9131c6-08db-48a1-a356-e4a19c077c22\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b20dc57a-hc8mf" Nov 03 01:18:01.263880 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:18:01.263824 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9a9131c6-08db-48a1-a356-e4a19c077c22-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b20dc57a-hc8mf\" (UID: \"9a9131c6-08db-48a1-a356-e4a19c077c22\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b20dc57a-hc8mf" Nov 03 01:18:01.264493 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:18:01.264473 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9a9131c6-08db-48a1-a356-e4a19c077c22-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b20dc57a-hc8mf\" (UID: \"9a9131c6-08db-48a1-a356-e4a19c077c22\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b20dc57a-hc8mf" Nov 03 01:18:01.266267 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:18:01.266242 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9a9131c6-08db-48a1-a356-e4a19c077c22-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b20dc57a-hc8mf\" (UID: \"9a9131c6-08db-48a1-a356-e4a19c077c22\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b20dc57a-hc8mf" Nov 03 01:18:01.272767 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:18:01.272746 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4zsrs\" (UniqueName: \"kubernetes.io/projected/9a9131c6-08db-48a1-a356-e4a19c077c22-kube-api-access-4zsrs\") pod \"maas-api-revoke-sub-b20dc57a-hc8mf\" (UID: \"9a9131c6-08db-48a1-a356-e4a19c077c22\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b20dc57a-hc8mf" Nov 03 01:18:01.306082 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:18:01.306058 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:18:01.362440 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:18:01.362406 4939 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b20dc57a-hc8mf" Nov 03 01:18:01.497262 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:18:01.497218 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:18:01.670796 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:18:01.670760 4939 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b20dc57a-hc8mf"] Nov 03 01:18:01.694186 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:18:01.694159 4939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b20dc57a-hc8mf" event={"ID":"9a9131c6-08db-48a1-a356-e4a19c077c22","Type":"ContainerStarted","Data":"41e111463eae28b32448111da09917fcfa8fb9d647514019d5120561eafac7ce"} Nov 03 01:18:01.922245 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:18:01.922218 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:18:02.112994 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:18:02.112073 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:18:02.584972 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:18:02.584924 4939 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-b-be59df/payload-processing-external-model-e2e-extmod-b-be59df-5885fq5qq" Nov 03 01:18:02.703006 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:18:02.702924 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:18:02.708455 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:18:02.708418 4939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b20dc57a-hc8mf" event={"ID":"9a9131c6-08db-48a1-a356-e4a19c077c22","Type":"ContainerStarted","Data":"1192511cd77038a4ec725b9a5e3362d4ff11ede6418eeb10fc882c79aba91ff3"} Nov 03 01:18:02.731860 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:18:02.731814 4939 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b20dc57a-hc8mf" podStartSLOduration=2.731800691 podStartE2EDuration="2.731800691s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 01:18:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 01:18:02.72839102 +0000 UTC m=+2230.992725964" watchObservedRunningTime="2026-11-03 01:18:02.731800691 +0000 UTC m=+2230.996135635" Nov 03 01:18:02.790358 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:18:02.790324 4939 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-a-7c759a/payload-processing-external-model-e2e-extmod-a-7c759a-54dbtcnb5"] Nov 03 01:18:02.795482 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:18:02.795463 4939 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-7c759a/payload-processing-external-model-e2e-extmod-a-7c759a-54dbtcnb5" Nov 03 01:18:02.801498 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:18:02.801472 4939 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-7c759a\"/\"agc-cred-ce0492ae50e9\"" Nov 03 01:18:02.801498 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:18:02.801472 4939 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-7c759a\"/\"openshift-service-ca.crt\"" Nov 03 01:18:02.803071 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:18:02.801484 4939 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-7c759a\"/\"routing-overlay\"" Nov 03 01:18:02.803071 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:18:02.801539 4939 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-7c759a\"/\"payload-processing-external-model-plugins-e2e-extmod-a-7c759a\"" Nov 03 01:18:02.803071 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:18:02.801579 4939 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-7c759a\"/\"kube-root-ca.crt\"" Nov 03 01:18:02.803071 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:18:02.801639 4939 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-7c759a\"/\"payload-processing-external-model-e2e--d95606a1-dockercfg-v5djz\"" Nov 03 01:18:02.820366 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:18:02.819833 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:18:02.827051 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:18:02.827025 4939 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-7c759a/payload-processing-external-model-e2e-extmod-a-7c759a-54dbtcnb5"] Nov 03 01:18:02.830063 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:18:02.830029 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3057b25c-0d3c-441d-ad3c-00b4166051c0-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-7c759a-54dbtcnb5\" (UID: \"3057b25c-0d3c-441d-ad3c-00b4166051c0\") " pod="ai-tenant-e2e-extmod-a-7c759a/payload-processing-external-model-e2e-extmod-a-7c759a-54dbtcnb5" Nov 03 01:18:02.830203 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:18:02.830155 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6nqpl\" (UniqueName: \"kubernetes.io/projected/3057b25c-0d3c-441d-ad3c-00b4166051c0-kube-api-access-6nqpl\") pod \"payload-processing-external-model-e2e-extmod-a-7c759a-54dbtcnb5\" (UID: \"3057b25c-0d3c-441d-ad3c-00b4166051c0\") " pod="ai-tenant-e2e-extmod-a-7c759a/payload-processing-external-model-e2e-extmod-a-7c759a-54dbtcnb5" Nov 03 01:18:02.830287 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:18:02.830269 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/3057b25c-0d3c-441d-ad3c-00b4166051c0-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-7c759a-54dbtcnb5\" (UID: \"3057b25c-0d3c-441d-ad3c-00b4166051c0\") " pod="ai-tenant-e2e-extmod-a-7c759a/payload-processing-external-model-e2e-extmod-a-7c759a-54dbtcnb5" Nov 03 01:18:02.830368 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:18:02.830352 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/3057b25c-0d3c-441d-ad3c-00b4166051c0-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-7c759a-54dbtcnb5\" (UID: \"3057b25c-0d3c-441d-ad3c-00b4166051c0\") " pod="ai-tenant-e2e-extmod-a-7c759a/payload-processing-external-model-e2e-extmod-a-7c759a-54dbtcnb5" Nov 03 01:18:02.997982 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:18:02.958718 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6nqpl\" (UniqueName: \"kubernetes.io/projected/3057b25c-0d3c-441d-ad3c-00b4166051c0-kube-api-access-6nqpl\") pod \"payload-processing-external-model-e2e-extmod-a-7c759a-54dbtcnb5\" (UID: \"3057b25c-0d3c-441d-ad3c-00b4166051c0\") " pod="ai-tenant-e2e-extmod-a-7c759a/payload-processing-external-model-e2e-extmod-a-7c759a-54dbtcnb5" Nov 03 01:18:02.997982 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:18:02.958888 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/3057b25c-0d3c-441d-ad3c-00b4166051c0-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-7c759a-54dbtcnb5\" (UID: \"3057b25c-0d3c-441d-ad3c-00b4166051c0\") " pod="ai-tenant-e2e-extmod-a-7c759a/payload-processing-external-model-e2e-extmod-a-7c759a-54dbtcnb5" Nov 03 01:18:02.997982 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:18:02.958984 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/3057b25c-0d3c-441d-ad3c-00b4166051c0-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-7c759a-54dbtcnb5\" (UID: \"3057b25c-0d3c-441d-ad3c-00b4166051c0\") " pod="ai-tenant-e2e-extmod-a-7c759a/payload-processing-external-model-e2e-extmod-a-7c759a-54dbtcnb5" Nov 03 01:18:02.997982 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:18:02.959309 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3057b25c-0d3c-441d-ad3c-00b4166051c0-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-7c759a-54dbtcnb5\" (UID: \"3057b25c-0d3c-441d-ad3c-00b4166051c0\") " pod="ai-tenant-e2e-extmod-a-7c759a/payload-processing-external-model-e2e-extmod-a-7c759a-54dbtcnb5" Nov 03 01:18:02.997982 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:18:02.972800 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3057b25c-0d3c-441d-ad3c-00b4166051c0-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-7c759a-54dbtcnb5\" (UID: \"3057b25c-0d3c-441d-ad3c-00b4166051c0\") " pod="ai-tenant-e2e-extmod-a-7c759a/payload-processing-external-model-e2e-extmod-a-7c759a-54dbtcnb5" Nov 03 01:18:02.997982 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:18:02.973680 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/3057b25c-0d3c-441d-ad3c-00b4166051c0-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-7c759a-54dbtcnb5\" (UID: \"3057b25c-0d3c-441d-ad3c-00b4166051c0\") " pod="ai-tenant-e2e-extmod-a-7c759a/payload-processing-external-model-e2e-extmod-a-7c759a-54dbtcnb5" Nov 03 01:18:03.081931 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:18:03.080986 4939 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-a-7c759a/payload-processing-external-model-e2e-extmod-a-7c759a-5bf4hhjsl"] Nov 03 01:18:03.333658 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:18:03.310222 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6nqpl\" (UniqueName: \"kubernetes.io/projected/3057b25c-0d3c-441d-ad3c-00b4166051c0-kube-api-access-6nqpl\") pod \"payload-processing-external-model-e2e-extmod-a-7c759a-54dbtcnb5\" (UID: \"3057b25c-0d3c-441d-ad3c-00b4166051c0\") " pod="ai-tenant-e2e-extmod-a-7c759a/payload-processing-external-model-e2e-extmod-a-7c759a-54dbtcnb5" Nov 03 01:18:03.333658 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:18:03.329508 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/3057b25c-0d3c-441d-ad3c-00b4166051c0-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-7c759a-54dbtcnb5\" (UID: \"3057b25c-0d3c-441d-ad3c-00b4166051c0\") " pod="ai-tenant-e2e-extmod-a-7c759a/payload-processing-external-model-e2e-extmod-a-7c759a-54dbtcnb5" Nov 03 01:18:03.335846 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:18:03.335817 4939 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-7c759a/payload-processing-external-model-e2e-extmod-a-7c759a-5bf4hhjsl"] Nov 03 01:18:03.336164 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:18:03.336142 4939 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-7c759a/payload-processing-external-model-e2e-extmod-a-7c759a-5bf4hhjsl" Nov 03 01:18:03.346336 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:18:03.346293 4939 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-7c759a\"/\"agc-cred-8cd07c22a743\"" Nov 03 01:18:03.377480 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:18:03.377449 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/486543c2-6226-478d-91c3-bcc7112c4940-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-7c759a-5bf4hhjsl\" (UID: \"486543c2-6226-478d-91c3-bcc7112c4940\") " pod="ai-tenant-e2e-extmod-a-7c759a/payload-processing-external-model-e2e-extmod-a-7c759a-5bf4hhjsl" Nov 03 01:18:03.377604 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:18:03.377566 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/486543c2-6226-478d-91c3-bcc7112c4940-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-7c759a-5bf4hhjsl\" (UID: \"486543c2-6226-478d-91c3-bcc7112c4940\") " pod="ai-tenant-e2e-extmod-a-7c759a/payload-processing-external-model-e2e-extmod-a-7c759a-5bf4hhjsl" Nov 03 01:18:03.377654 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:18:03.377626 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/486543c2-6226-478d-91c3-bcc7112c4940-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-7c759a-5bf4hhjsl\" (UID: \"486543c2-6226-478d-91c3-bcc7112c4940\") " pod="ai-tenant-e2e-extmod-a-7c759a/payload-processing-external-model-e2e-extmod-a-7c759a-5bf4hhjsl" Nov 03 01:18:03.377751 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:18:03.377739 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k5mgh\" (UniqueName: \"kubernetes.io/projected/486543c2-6226-478d-91c3-bcc7112c4940-kube-api-access-k5mgh\") pod \"payload-processing-external-model-e2e-extmod-a-7c759a-5bf4hhjsl\" (UID: \"486543c2-6226-478d-91c3-bcc7112c4940\") " pod="ai-tenant-e2e-extmod-a-7c759a/payload-processing-external-model-e2e-extmod-a-7c759a-5bf4hhjsl" Nov 03 01:18:03.417571 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:18:03.417536 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:18:03.429607 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:18:03.429582 4939 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-7c759a/payload-processing-external-model-e2e-extmod-a-7c759a-54dbtcnb5" Nov 03 01:18:03.485100 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:18:03.483291 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/486543c2-6226-478d-91c3-bcc7112c4940-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-7c759a-5bf4hhjsl\" (UID: \"486543c2-6226-478d-91c3-bcc7112c4940\") " pod="ai-tenant-e2e-extmod-a-7c759a/payload-processing-external-model-e2e-extmod-a-7c759a-5bf4hhjsl" Nov 03 01:18:03.485100 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:18:03.483392 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/486543c2-6226-478d-91c3-bcc7112c4940-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-7c759a-5bf4hhjsl\" (UID: \"486543c2-6226-478d-91c3-bcc7112c4940\") " pod="ai-tenant-e2e-extmod-a-7c759a/payload-processing-external-model-e2e-extmod-a-7c759a-5bf4hhjsl" Nov 03 01:18:03.485100 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:18:03.483599 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k5mgh\" (UniqueName: \"kubernetes.io/projected/486543c2-6226-478d-91c3-bcc7112c4940-kube-api-access-k5mgh\") pod \"payload-processing-external-model-e2e-extmod-a-7c759a-5bf4hhjsl\" (UID: \"486543c2-6226-478d-91c3-bcc7112c4940\") " pod="ai-tenant-e2e-extmod-a-7c759a/payload-processing-external-model-e2e-extmod-a-7c759a-5bf4hhjsl" Nov 03 01:18:03.485100 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:18:03.483872 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/486543c2-6226-478d-91c3-bcc7112c4940-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-7c759a-5bf4hhjsl\" (UID: \"486543c2-6226-478d-91c3-bcc7112c4940\") " pod="ai-tenant-e2e-extmod-a-7c759a/payload-processing-external-model-e2e-extmod-a-7c759a-5bf4hhjsl" Nov 03 01:18:03.510929 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:18:03.505965 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/486543c2-6226-478d-91c3-bcc7112c4940-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-7c759a-5bf4hhjsl\" (UID: \"486543c2-6226-478d-91c3-bcc7112c4940\") " pod="ai-tenant-e2e-extmod-a-7c759a/payload-processing-external-model-e2e-extmod-a-7c759a-5bf4hhjsl" Nov 03 01:18:03.520944 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:18:03.518622 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/486543c2-6226-478d-91c3-bcc7112c4940-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-7c759a-5bf4hhjsl\" (UID: \"486543c2-6226-478d-91c3-bcc7112c4940\") " pod="ai-tenant-e2e-extmod-a-7c759a/payload-processing-external-model-e2e-extmod-a-7c759a-5bf4hhjsl" Nov 03 01:18:03.540045 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:18:03.540010 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/486543c2-6226-478d-91c3-bcc7112c4940-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-7c759a-5bf4hhjsl\" (UID: \"486543c2-6226-478d-91c3-bcc7112c4940\") " pod="ai-tenant-e2e-extmod-a-7c759a/payload-processing-external-model-e2e-extmod-a-7c759a-5bf4hhjsl" Nov 03 01:18:03.543415 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:18:03.543358 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-k5mgh\" (UniqueName: \"kubernetes.io/projected/486543c2-6226-478d-91c3-bcc7112c4940-kube-api-access-k5mgh\") pod \"payload-processing-external-model-e2e-extmod-a-7c759a-5bf4hhjsl\" (UID: \"486543c2-6226-478d-91c3-bcc7112c4940\") " pod="ai-tenant-e2e-extmod-a-7c759a/payload-processing-external-model-e2e-extmod-a-7c759a-5bf4hhjsl" Nov 03 01:18:03.676902 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:18:03.676866 4939 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-7c759a/payload-processing-external-model-e2e-extmod-a-7c759a-5bf4hhjsl" Nov 03 01:18:03.719734 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:18:03.719699 4939 generic.go:356] "Generic (PLEG): container finished" podID="9a9131c6-08db-48a1-a356-e4a19c077c22" containerID="1192511cd77038a4ec725b9a5e3362d4ff11ede6418eeb10fc882c79aba91ff3" exitCode=0 Nov 03 01:18:03.719919 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:18:03.719824 4939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b20dc57a-hc8mf" event={"ID":"9a9131c6-08db-48a1-a356-e4a19c077c22","Type":"ContainerDied","Data":"1192511cd77038a4ec725b9a5e3362d4ff11ede6418eeb10fc882c79aba91ff3"} Nov 03 01:18:03.826694 ip-10-0-140-107 kubenswrapper[4939]: W1103 01:18:03.826657 4939 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3057b25c_0d3c_441d_ad3c_00b4166051c0.slice/crio-5d6e5e8e970465b95208adbb17250499fd8121ea0e1581c1f2b974500fcf312c WatchSource:0}: Error finding container 5d6e5e8e970465b95208adbb17250499fd8121ea0e1581c1f2b974500fcf312c: Status 404 returned error can't find the container with id 5d6e5e8e970465b95208adbb17250499fd8121ea0e1581c1f2b974500fcf312c Nov 03 01:18:03.828655 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:18:03.828626 4939 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-7c759a/payload-processing-external-model-e2e-extmod-a-7c759a-54dbtcnb5"] Nov 03 01:18:03.967778 ip-10-0-140-107 kubenswrapper[4939]: W1103 01:18:03.967742 4939 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod486543c2_6226_478d_91c3_bcc7112c4940.slice/crio-b4383b26c954eaba7ab167baf4bc9dbc5f87dce1910a7b29e55e828f44554e0c WatchSource:0}: Error finding container b4383b26c954eaba7ab167baf4bc9dbc5f87dce1910a7b29e55e828f44554e0c: Status 404 returned error can't find the container with id b4383b26c954eaba7ab167baf4bc9dbc5f87dce1910a7b29e55e828f44554e0c Nov 03 01:18:03.968267 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:18:03.968244 4939 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-7c759a/payload-processing-external-model-e2e-extmod-a-7c759a-5bf4hhjsl"] Nov 03 01:18:04.174864 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:18:04.174786 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:18:04.730613 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:18:04.730567 4939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-7c759a/payload-processing-external-model-e2e-extmod-a-7c759a-5bf4hhjsl" event={"ID":"486543c2-6226-478d-91c3-bcc7112c4940","Type":"ContainerStarted","Data":"731adb6a2eeedb9c7aa8c5fd9ee03bf5162dbf13cb5021f5798fc249ad4a6c24"} Nov 03 01:18:04.731175 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:18:04.730630 4939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-7c759a/payload-processing-external-model-e2e-extmod-a-7c759a-5bf4hhjsl" event={"ID":"486543c2-6226-478d-91c3-bcc7112c4940","Type":"ContainerStarted","Data":"b4383b26c954eaba7ab167baf4bc9dbc5f87dce1910a7b29e55e828f44554e0c"} Nov 03 01:18:04.731175 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:18:04.730659 4939 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-a-7c759a/payload-processing-external-model-e2e-extmod-a-7c759a-5bf4hhjsl" Nov 03 01:18:04.732434 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:18:04.732406 4939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-7c759a/payload-processing-external-model-e2e-extmod-a-7c759a-54dbtcnb5" event={"ID":"3057b25c-0d3c-441d-ad3c-00b4166051c0","Type":"ContainerStarted","Data":"b88f9c714f6fc0d1a3a799f816b24aeda78b42d4d361a1d1b9eeb9db50c177f7"} Nov 03 01:18:04.732586 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:18:04.732450 4939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-7c759a/payload-processing-external-model-e2e-extmod-a-7c759a-54dbtcnb5" event={"ID":"3057b25c-0d3c-441d-ad3c-00b4166051c0","Type":"ContainerStarted","Data":"5d6e5e8e970465b95208adbb17250499fd8121ea0e1581c1f2b974500fcf312c"} Nov 03 01:18:04.732586 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:18:04.732480 4939 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-a-7c759a/payload-processing-external-model-e2e-extmod-a-7c759a-54dbtcnb5" Nov 03 01:18:04.753207 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:18:04.753160 4939 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-a-7c759a/payload-processing-external-model-e2e-extmod-a-7c759a-5bf4hhjsl" podStartSLOduration=2.753146804 podStartE2EDuration="2.753146804s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 01:18:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 01:18:04.749667238 +0000 UTC m=+2233.014002184" watchObservedRunningTime="2026-11-03 01:18:04.753146804 +0000 UTC m=+2233.017481750" Nov 03 01:18:04.771059 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:18:04.771014 4939 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-a-7c759a/payload-processing-external-model-e2e-extmod-a-7c759a-54dbtcnb5" podStartSLOduration=2.771000697 podStartE2EDuration="2.771000697s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 01:18:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 01:18:04.767025351 +0000 UTC m=+2233.031360297" watchObservedRunningTime="2026-11-03 01:18:04.771000697 +0000 UTC m=+2233.035335641" Nov 03 01:18:04.809267 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:18:04.809231 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:18:04.998520 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:18:04.998460 4939 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b20dc57a-hc8mf" Nov 03 01:18:05.023405 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:18:05.023367 4939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9a9131c6-08db-48a1-a356-e4a19c077c22-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9a9131c6-08db-48a1-a356-e4a19c077c22-maas-api-openshift-ca\") pod \"9a9131c6-08db-48a1-a356-e4a19c077c22\" (UID: \"9a9131c6-08db-48a1-a356-e4a19c077c22\") " Nov 03 01:18:05.023566 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:18:05.023420 4939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9a9131c6-08db-48a1-a356-e4a19c077c22-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9a9131c6-08db-48a1-a356-e4a19c077c22-maas-api-xks-ca\") pod \"9a9131c6-08db-48a1-a356-e4a19c077c22\" (UID: \"9a9131c6-08db-48a1-a356-e4a19c077c22\") " Nov 03 01:18:05.023566 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:18:05.023488 4939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9a9131c6-08db-48a1-a356-e4a19c077c22-kube-api-access-4zsrs\" (UniqueName: \"kubernetes.io/projected/9a9131c6-08db-48a1-a356-e4a19c077c22-kube-api-access-4zsrs\") pod \"9a9131c6-08db-48a1-a356-e4a19c077c22\" (UID: \"9a9131c6-08db-48a1-a356-e4a19c077c22\") " Nov 03 01:18:05.023743 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:18:05.023719 4939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9a9131c6-08db-48a1-a356-e4a19c077c22-maas-api-openshift-ca" pod "9a9131c6-08db-48a1-a356-e4a19c077c22" (UID: "9a9131c6-08db-48a1-a356-e4a19c077c22"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 01:18:05.024747 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:18:05.024635 4939 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9a9131c6-08db-48a1-a356-e4a19c077c22-maas-api-openshift-ca\") on node \"ip-10-0-140-107.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:18:05.027052 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:18:05.027021 4939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9a9131c6-08db-48a1-a356-e4a19c077c22-kube-api-access-4zsrs" pod "9a9131c6-08db-48a1-a356-e4a19c077c22" (UID: "9a9131c6-08db-48a1-a356-e4a19c077c22"). InnerVolumeSpecName "kube-api-access-4zsrs". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 01:18:05.027163 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:18:05.027118 4939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9a9131c6-08db-48a1-a356-e4a19c077c22-maas-api-xks-ca" pod "9a9131c6-08db-48a1-a356-e4a19c077c22" (UID: "9a9131c6-08db-48a1-a356-e4a19c077c22"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 01:18:05.125838 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:18:05.125805 4939 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9a9131c6-08db-48a1-a356-e4a19c077c22-maas-api-xks-ca\") on node \"ip-10-0-140-107.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:18:05.125838 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:18:05.125834 4939 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4zsrs\" (UniqueName: \"kubernetes.io/projected/9a9131c6-08db-48a1-a356-e4a19c077c22-kube-api-access-4zsrs\") on node \"ip-10-0-140-107.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:18:05.611371 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:18:05.611340 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:18:05.751997 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:18:05.751944 4939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b20dc57a-hc8mf" event={"ID":"9a9131c6-08db-48a1-a356-e4a19c077c22","Type":"ContainerDied","Data":"41e111463eae28b32448111da09917fcfa8fb9d647514019d5120561eafac7ce"} Nov 03 01:18:05.752393 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:18:05.752014 4939 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="41e111463eae28b32448111da09917fcfa8fb9d647514019d5120561eafac7ce" Nov 03 01:18:05.752393 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:18:05.751974 4939 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b20dc57a-hc8mf" Nov 03 01:18:06.225376 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:18:06.225344 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:18:06.900265 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:18:06.900230 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:18:07.571784 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:18:07.571749 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:18:08.180660 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:18:08.180622 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:18:08.875782 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:18:08.875746 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:18:09.493312 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:18:09.493276 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:18:10.119173 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:18:10.119139 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:18:10.809191 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:18:10.809155 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:18:11.491416 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:18:11.491382 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:18:11.884766 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:18:11.884725 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:18:12.058931 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:18:12.056469 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:18:12.243998 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:18:12.239872 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:18:12.597037 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:18:12.596985 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:18:12.711273 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:18:12.711223 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:18:12.865318 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:18:12.865226 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:18:13.001750 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:18:13.001717 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:18:13.136396 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:18:13.136315 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:18:13.300383 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:18:13.300350 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:18:13.802606 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:18:13.802562 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:18:13.909979 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:18:13.909941 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:18:14.427506 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:18:14.427463 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:18:14.584632 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:18:14.584594 4939 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-a-821f5a/payload-processing-external-model-e2e-extmod-a-821f5a-94cbf4lx9"] Nov 03 01:18:14.584969 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:18:14.584933 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:18:14.585567 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:18:14.585550 4939 cpu_manager.go:414] "containerMap: removing container" podUID="9a9131c6-08db-48a1-a356-e4a19c077c22" containerName="revoke-keys" Nov 03 01:18:14.585629 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:18:14.585568 4939 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9a9131c6-08db-48a1-a356-e4a19c077c22" containerName="revoke-keys" Nov 03 01:18:14.585735 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:18:14.585726 4939 memory_manager.go:365] "RemoveStaleState removing state" podUID="9a9131c6-08db-48a1-a356-e4a19c077c22" containerName="revoke-keys" Nov 03 01:18:14.591422 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:18:14.591402 4939 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-821f5a/payload-processing-external-model-e2e-extmod-a-821f5a-94cbf4lx9" Nov 03 01:18:14.595304 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:18:14.595283 4939 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-821f5a\"/\"kube-root-ca.crt\"" Nov 03 01:18:14.595470 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:18:14.595293 4939 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-821f5a\"/\"agc-cred-6031577574a1\"" Nov 03 01:18:14.595470 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:18:14.595334 4939 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-821f5a\"/\"payload-processing-external-model-plugins-e2e-extmod-a-821f5a\"" Nov 03 01:18:14.595733 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:18:14.595449 4939 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-821f5a\"/\"openshift-service-ca.crt\"" Nov 03 01:18:14.596644 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:18:14.596622 4939 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-821f5a\"/\"routing-overlay\"" Nov 03 01:18:14.600818 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:18:14.600788 4939 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-821f5a/payload-processing-external-model-e2e-extmod-a-821f5a-94cbf4lx9"] Nov 03 01:18:14.692336 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:18:14.692230 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/97ca98ed-903e-422a-bffd-ad208fc2a3aa-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-821f5a-94cbf4lx9\" (UID: \"97ca98ed-903e-422a-bffd-ad208fc2a3aa\") " pod="ai-tenant-e2e-extmod-a-821f5a/payload-processing-external-model-e2e-extmod-a-821f5a-94cbf4lx9" Nov 03 01:18:14.692522 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:18:14.692387 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/97ca98ed-903e-422a-bffd-ad208fc2a3aa-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-821f5a-94cbf4lx9\" (UID: \"97ca98ed-903e-422a-bffd-ad208fc2a3aa\") " pod="ai-tenant-e2e-extmod-a-821f5a/payload-processing-external-model-e2e-extmod-a-821f5a-94cbf4lx9" Nov 03 01:18:14.692596 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:18:14.692524 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/97ca98ed-903e-422a-bffd-ad208fc2a3aa-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-821f5a-94cbf4lx9\" (UID: \"97ca98ed-903e-422a-bffd-ad208fc2a3aa\") " pod="ai-tenant-e2e-extmod-a-821f5a/payload-processing-external-model-e2e-extmod-a-821f5a-94cbf4lx9" Nov 03 01:18:14.692596 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:18:14.692578 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hxpv4\" (UniqueName: \"kubernetes.io/projected/97ca98ed-903e-422a-bffd-ad208fc2a3aa-kube-api-access-hxpv4\") pod \"payload-processing-external-model-e2e-extmod-a-821f5a-94cbf4lx9\" (UID: \"97ca98ed-903e-422a-bffd-ad208fc2a3aa\") " pod="ai-tenant-e2e-extmod-a-821f5a/payload-processing-external-model-e2e-extmod-a-821f5a-94cbf4lx9" Nov 03 01:18:14.807945 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:18:14.800269 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/97ca98ed-903e-422a-bffd-ad208fc2a3aa-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-821f5a-94cbf4lx9\" (UID: \"97ca98ed-903e-422a-bffd-ad208fc2a3aa\") " pod="ai-tenant-e2e-extmod-a-821f5a/payload-processing-external-model-e2e-extmod-a-821f5a-94cbf4lx9" Nov 03 01:18:14.807945 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:18:14.800465 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/97ca98ed-903e-422a-bffd-ad208fc2a3aa-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-821f5a-94cbf4lx9\" (UID: \"97ca98ed-903e-422a-bffd-ad208fc2a3aa\") " pod="ai-tenant-e2e-extmod-a-821f5a/payload-processing-external-model-e2e-extmod-a-821f5a-94cbf4lx9" Nov 03 01:18:14.807945 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:18:14.800583 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/97ca98ed-903e-422a-bffd-ad208fc2a3aa-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-821f5a-94cbf4lx9\" (UID: \"97ca98ed-903e-422a-bffd-ad208fc2a3aa\") " pod="ai-tenant-e2e-extmod-a-821f5a/payload-processing-external-model-e2e-extmod-a-821f5a-94cbf4lx9" Nov 03 01:18:14.807945 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:18:14.800629 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hxpv4\" (UniqueName: \"kubernetes.io/projected/97ca98ed-903e-422a-bffd-ad208fc2a3aa-kube-api-access-hxpv4\") pod \"payload-processing-external-model-e2e-extmod-a-821f5a-94cbf4lx9\" (UID: \"97ca98ed-903e-422a-bffd-ad208fc2a3aa\") " pod="ai-tenant-e2e-extmod-a-821f5a/payload-processing-external-model-e2e-extmod-a-821f5a-94cbf4lx9" Nov 03 01:18:14.828943 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:18:14.822679 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/97ca98ed-903e-422a-bffd-ad208fc2a3aa-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-821f5a-94cbf4lx9\" (UID: \"97ca98ed-903e-422a-bffd-ad208fc2a3aa\") " pod="ai-tenant-e2e-extmod-a-821f5a/payload-processing-external-model-e2e-extmod-a-821f5a-94cbf4lx9" Nov 03 01:18:14.828943 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:18:14.823595 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/97ca98ed-903e-422a-bffd-ad208fc2a3aa-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-821f5a-94cbf4lx9\" (UID: \"97ca98ed-903e-422a-bffd-ad208fc2a3aa\") " pod="ai-tenant-e2e-extmod-a-821f5a/payload-processing-external-model-e2e-extmod-a-821f5a-94cbf4lx9" Nov 03 01:18:14.838345 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:18:14.834514 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/97ca98ed-903e-422a-bffd-ad208fc2a3aa-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-821f5a-94cbf4lx9\" (UID: \"97ca98ed-903e-422a-bffd-ad208fc2a3aa\") " pod="ai-tenant-e2e-extmod-a-821f5a/payload-processing-external-model-e2e-extmod-a-821f5a-94cbf4lx9" Nov 03 01:18:14.936933 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:18:14.913969 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hxpv4\" (UniqueName: \"kubernetes.io/projected/97ca98ed-903e-422a-bffd-ad208fc2a3aa-kube-api-access-hxpv4\") pod \"payload-processing-external-model-e2e-extmod-a-821f5a-94cbf4lx9\" (UID: \"97ca98ed-903e-422a-bffd-ad208fc2a3aa\") " pod="ai-tenant-e2e-extmod-a-821f5a/payload-processing-external-model-e2e-extmod-a-821f5a-94cbf4lx9" Nov 03 01:18:14.936933 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:18:14.931556 4939 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-821f5a/payload-processing-external-model-e2e-extmod-a-821f5a-94cbf4lx9" Nov 03 01:18:15.430743 ip-10-0-140-107 kubenswrapper[4939]: W1103 01:18:15.430702 4939 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod97ca98ed_903e_422a_bffd_ad208fc2a3aa.slice/crio-8236af36cdb94c322738badf5d75c436f2b12bf93200d0a410be1a89027bd9f4 WatchSource:0}: Error finding container 8236af36cdb94c322738badf5d75c436f2b12bf93200d0a410be1a89027bd9f4: Status 404 returned error can't find the container with id 8236af36cdb94c322738badf5d75c436f2b12bf93200d0a410be1a89027bd9f4 Nov 03 01:18:15.431447 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:18:15.431419 4939 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-821f5a/payload-processing-external-model-e2e-extmod-a-821f5a-94cbf4lx9"] Nov 03 01:18:15.587535 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:18:15.587480 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:18:15.755747 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:18:15.755662 4939 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-a-7c759a/payload-processing-external-model-e2e-extmod-a-7c759a-5bf4hhjsl" Nov 03 01:18:15.755747 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:18:15.755729 4939 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-a-7c759a/payload-processing-external-model-e2e-extmod-a-7c759a-54dbtcnb5" Nov 03 01:18:15.823223 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:18:15.823174 4939 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-7c759a/payload-processing-external-model-e2e-extmod-a-7c759a-54dbtcnb5"] Nov 03 01:18:15.950962 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:18:15.950922 4939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-821f5a/payload-processing-external-model-e2e-extmod-a-821f5a-94cbf4lx9" event={"ID":"97ca98ed-903e-422a-bffd-ad208fc2a3aa","Type":"ContainerStarted","Data":"aa31d2f76160892a6d8d3557762986d6a3387098f57d33cbacd418afcc516487"} Nov 03 01:18:15.951240 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:18:15.950990 4939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-821f5a/payload-processing-external-model-e2e-extmod-a-821f5a-94cbf4lx9" event={"ID":"97ca98ed-903e-422a-bffd-ad208fc2a3aa","Type":"ContainerStarted","Data":"8236af36cdb94c322738badf5d75c436f2b12bf93200d0a410be1a89027bd9f4"} Nov 03 01:18:15.951240 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:18:15.951020 4939 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-a-821f5a/payload-processing-external-model-e2e-extmod-a-821f5a-94cbf4lx9" Nov 03 01:18:15.969316 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:18:15.969267 4939 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-a-821f5a/payload-processing-external-model-e2e-extmod-a-821f5a-94cbf4lx9" podStartSLOduration=1.969252913 podStartE2EDuration="1.969252913s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 01:18:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 01:18:15.968069167 +0000 UTC m=+2244.232404106" watchObservedRunningTime="2026-11-03 01:18:15.969252913 +0000 UTC m=+2244.233587858" Nov 03 01:18:16.315710 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:18:16.315680 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:18:17.101230 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:18:17.101193 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:18:17.775729 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:18:17.775699 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:18:18.033882 ip-10-0-140-107 kubenswrapper[4939]: E1103 01:18:18.033798 4939 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-sthjh" containerName="extractor" Nov 03 01:18:18.493092 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:18:18.493030 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:18:18.508426 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:18:18.508375 4939 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-4pw7v"] Nov 03 01:18:19.252794 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:18:19.252763 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:18:20.798338 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:18:20.798300 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:18:20.951828 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:18:20.951791 4939 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-a-7c759a/payload-processing-external-model-e2e-extmod-a-7c759a-54dbtcnb5" podUID="3057b25c-0d3c-441d-ad3c-00b4166051c0" containerName="payload-processing" containerID="cri-o://b88f9c714f6fc0d1a3a799f816b24aeda78b42d4d361a1d1b9eeb9db50c177f7" gracePeriod=25 Nov 03 01:18:21.075407 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:18:21.075373 4939 generic.go:356] "Generic (PLEG): container finished" podID="3057b25c-0d3c-441d-ad3c-00b4166051c0" containerID="b88f9c714f6fc0d1a3a799f816b24aeda78b42d4d361a1d1b9eeb9db50c177f7" exitCode=0 Nov 03 01:18:21.075570 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:18:21.075443 4939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-7c759a/payload-processing-external-model-e2e-extmod-a-7c759a-54dbtcnb5" event={"ID":"3057b25c-0d3c-441d-ad3c-00b4166051c0","Type":"ContainerDied","Data":"b88f9c714f6fc0d1a3a799f816b24aeda78b42d4d361a1d1b9eeb9db50c177f7"} Nov 03 01:18:21.263123 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:18:21.263095 4939 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-7c759a/payload-processing-external-model-e2e-extmod-a-7c759a-54dbtcnb5" Nov 03 01:18:21.319741 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:18:21.319649 4939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3057b25c-0d3c-441d-ad3c-00b4166051c0-provider-credentials\" (UniqueName: \"kubernetes.io/projected/3057b25c-0d3c-441d-ad3c-00b4166051c0-provider-credentials\") pod \"3057b25c-0d3c-441d-ad3c-00b4166051c0\" (UID: \"3057b25c-0d3c-441d-ad3c-00b4166051c0\") " Nov 03 01:18:21.319741 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:18:21.319715 4939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3057b25c-0d3c-441d-ad3c-00b4166051c0-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3057b25c-0d3c-441d-ad3c-00b4166051c0-plugins-config-volume\") pod \"3057b25c-0d3c-441d-ad3c-00b4166051c0\" (UID: \"3057b25c-0d3c-441d-ad3c-00b4166051c0\") " Nov 03 01:18:21.319741 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:18:21.319742 4939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3057b25c-0d3c-441d-ad3c-00b4166051c0-kube-api-access-6nqpl\" (UniqueName: \"kubernetes.io/projected/3057b25c-0d3c-441d-ad3c-00b4166051c0-kube-api-access-6nqpl\") pod \"3057b25c-0d3c-441d-ad3c-00b4166051c0\" (UID: \"3057b25c-0d3c-441d-ad3c-00b4166051c0\") " Nov 03 01:18:21.320072 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:18:21.319791 4939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3057b25c-0d3c-441d-ad3c-00b4166051c0-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/3057b25c-0d3c-441d-ad3c-00b4166051c0-routing-overlay-volume\") pod \"3057b25c-0d3c-441d-ad3c-00b4166051c0\" (UID: \"3057b25c-0d3c-441d-ad3c-00b4166051c0\") " Nov 03 01:18:21.320314 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:18:21.320280 4939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3057b25c-0d3c-441d-ad3c-00b4166051c0-plugins-config-volume" pod "3057b25c-0d3c-441d-ad3c-00b4166051c0" (UID: "3057b25c-0d3c-441d-ad3c-00b4166051c0"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 01:18:21.320450 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:18:21.320303 4939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3057b25c-0d3c-441d-ad3c-00b4166051c0-routing-overlay-volume" pod "3057b25c-0d3c-441d-ad3c-00b4166051c0" (UID: "3057b25c-0d3c-441d-ad3c-00b4166051c0"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 01:18:21.321016 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:18:21.320995 4939 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3057b25c-0d3c-441d-ad3c-00b4166051c0-plugins-config-volume\") on node \"ip-10-0-140-107.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:18:21.321098 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:18:21.321020 4939 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/3057b25c-0d3c-441d-ad3c-00b4166051c0-routing-overlay-volume\") on node \"ip-10-0-140-107.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:18:21.322788 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:18:21.322767 4939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3057b25c-0d3c-441d-ad3c-00b4166051c0-provider-credentials" pod "3057b25c-0d3c-441d-ad3c-00b4166051c0" (UID: "3057b25c-0d3c-441d-ad3c-00b4166051c0"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 01:18:21.322870 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:18:21.322805 4939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3057b25c-0d3c-441d-ad3c-00b4166051c0-kube-api-access-6nqpl" pod "3057b25c-0d3c-441d-ad3c-00b4166051c0" (UID: "3057b25c-0d3c-441d-ad3c-00b4166051c0"). InnerVolumeSpecName "kube-api-access-6nqpl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 01:18:21.423652 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:18:21.423616 4939 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/3057b25c-0d3c-441d-ad3c-00b4166051c0-provider-credentials\") on node \"ip-10-0-140-107.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:18:21.423652 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:18:21.423653 4939 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6nqpl\" (UniqueName: \"kubernetes.io/projected/3057b25c-0d3c-441d-ad3c-00b4166051c0-kube-api-access-6nqpl\") on node \"ip-10-0-140-107.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:18:21.508145 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:18:21.508109 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:18:22.087140 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:18:22.087100 4939 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-7c759a/payload-processing-external-model-e2e-extmod-a-7c759a-54dbtcnb5" Nov 03 01:18:22.087140 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:18:22.087125 4939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-7c759a/payload-processing-external-model-e2e-extmod-a-7c759a-54dbtcnb5" event={"ID":"3057b25c-0d3c-441d-ad3c-00b4166051c0","Type":"ContainerDied","Data":"5d6e5e8e970465b95208adbb17250499fd8121ea0e1581c1f2b974500fcf312c"} Nov 03 01:18:22.087671 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:18:22.087206 4939 scope.go:122] "RemoveContainer" containerID="b88f9c714f6fc0d1a3a799f816b24aeda78b42d4d361a1d1b9eeb9db50c177f7" Nov 03 01:18:22.108647 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:18:22.108618 4939 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-7c759a/payload-processing-external-model-e2e-extmod-a-7c759a-54dbtcnb5"] Nov 03 01:18:22.113734 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:18:22.113707 4939 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-a-7c759a/payload-processing-external-model-e2e-extmod-a-7c759a-54dbtcnb5"] Nov 03 01:18:22.221846 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:18:22.221766 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:18:22.977936 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:18:22.977880 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:18:23.832221 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:18:23.832191 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:18:23.959375 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:18:23.959327 4939 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3057b25c-0d3c-441d-ad3c-00b4166051c0" path="/var/lib/kubelet/pods/3057b25c-0d3c-441d-ad3c-00b4166051c0/volumes" Nov 03 01:18:24.483286 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:18:24.483249 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:18:25.123044 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:18:25.123008 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:18:25.805746 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:18:25.805710 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:18:26.466393 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:18:26.466361 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:18:26.964678 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:18:26.964640 4939 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-a-821f5a/payload-processing-external-model-e2e-extmod-a-821f5a-94cbf4lx9" Nov 03 01:18:27.098322 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:18:27.098288 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:18:27.941871 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:18:27.941834 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:18:28.595718 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:18:28.595686 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:18:29.281473 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:18:29.281438 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:18:29.983102 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:18:29.983065 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:18:30.761792 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:18:30.761756 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:18:31.385530 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:18:31.385498 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:18:32.049146 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:18:32.049111 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:18:32.686680 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:18:32.686643 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:18:33.281787 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:18:33.281750 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:18:33.942153 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:18:33.942114 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:18:34.533202 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:18:34.533165 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:18:34.589784 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:18:34.589748 4939 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-3c5983/payload-processing-external-model-e2e-extmod-b-3c5983-7697gtj7m"] Nov 03 01:18:34.639815 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:18:34.639787 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:18:34.760811 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:18:34.760763 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:18:35.160953 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:18:35.152212 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:18:35.444181 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:18:35.444092 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:18:36.099666 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:18:36.099636 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:18:36.690242 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:18:36.690203 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:18:37.389622 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:18:37.389589 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:18:38.062934 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:18:38.045897 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:18:38.688789 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:18:38.688757 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:18:39.371607 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:18:39.371561 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:18:39.590957 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:18:39.590895 4939 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-b-3c5983/payload-processing-external-model-e2e-extmod-b-3c5983-7697gtj7m" podUID="dbf522b9-d27e-40eb-9e43-5dbfafa5ed18" containerName="payload-processing" containerID="cri-o://ddce77c88fd8799ae6005554fc1e620ee81fcd48d69ee45e12fb264c241cb437" gracePeriod=25 Nov 03 01:18:39.907343 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:18:39.907269 4939 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-3c5983/payload-processing-external-model-e2e-extmod-b-3c5983-7697gtj7m" Nov 03 01:18:39.964841 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:18:39.964808 4939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/dbf522b9-d27e-40eb-9e43-5dbfafa5ed18-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dbf522b9-d27e-40eb-9e43-5dbfafa5ed18-plugins-config-volume\") pod \"dbf522b9-d27e-40eb-9e43-5dbfafa5ed18\" (UID: \"dbf522b9-d27e-40eb-9e43-5dbfafa5ed18\") " Nov 03 01:18:39.965064 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:18:39.965050 4939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dbf522b9-d27e-40eb-9e43-5dbfafa5ed18-kube-api-access-s4ht8\" (UniqueName: \"kubernetes.io/projected/dbf522b9-d27e-40eb-9e43-5dbfafa5ed18-kube-api-access-s4ht8\") pod \"dbf522b9-d27e-40eb-9e43-5dbfafa5ed18\" (UID: \"dbf522b9-d27e-40eb-9e43-5dbfafa5ed18\") " Nov 03 01:18:39.965173 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:18:39.965158 4939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dbf522b9-d27e-40eb-9e43-5dbfafa5ed18-provider-credentials\" (UniqueName: \"kubernetes.io/projected/dbf522b9-d27e-40eb-9e43-5dbfafa5ed18-provider-credentials\") pod \"dbf522b9-d27e-40eb-9e43-5dbfafa5ed18\" (UID: \"dbf522b9-d27e-40eb-9e43-5dbfafa5ed18\") " Nov 03 01:18:39.965223 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:18:39.965185 4939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/dbf522b9-d27e-40eb-9e43-5dbfafa5ed18-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/dbf522b9-d27e-40eb-9e43-5dbfafa5ed18-routing-overlay-volume\") pod \"dbf522b9-d27e-40eb-9e43-5dbfafa5ed18\" (UID: \"dbf522b9-d27e-40eb-9e43-5dbfafa5ed18\") " Nov 03 01:18:39.965362 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:18:39.965332 4939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dbf522b9-d27e-40eb-9e43-5dbfafa5ed18-plugins-config-volume" pod "dbf522b9-d27e-40eb-9e43-5dbfafa5ed18" (UID: "dbf522b9-d27e-40eb-9e43-5dbfafa5ed18"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 01:18:39.965601 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:18:39.965574 4939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dbf522b9-d27e-40eb-9e43-5dbfafa5ed18-routing-overlay-volume" pod "dbf522b9-d27e-40eb-9e43-5dbfafa5ed18" (UID: "dbf522b9-d27e-40eb-9e43-5dbfafa5ed18"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 01:18:39.966243 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:18:39.966218 4939 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dbf522b9-d27e-40eb-9e43-5dbfafa5ed18-plugins-config-volume\") on node \"ip-10-0-140-107.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:18:39.966366 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:18:39.966247 4939 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/dbf522b9-d27e-40eb-9e43-5dbfafa5ed18-routing-overlay-volume\") on node \"ip-10-0-140-107.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:18:39.967364 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:18:39.967343 4939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dbf522b9-d27e-40eb-9e43-5dbfafa5ed18-kube-api-access-s4ht8" pod "dbf522b9-d27e-40eb-9e43-5dbfafa5ed18" (UID: "dbf522b9-d27e-40eb-9e43-5dbfafa5ed18"). InnerVolumeSpecName "kube-api-access-s4ht8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 01:18:39.967777 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:18:39.967748 4939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dbf522b9-d27e-40eb-9e43-5dbfafa5ed18-provider-credentials" pod "dbf522b9-d27e-40eb-9e43-5dbfafa5ed18" (UID: "dbf522b9-d27e-40eb-9e43-5dbfafa5ed18"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 01:18:39.998483 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:18:39.998459 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:18:40.071746 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:18:40.069379 4939 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-s4ht8\" (UniqueName: \"kubernetes.io/projected/dbf522b9-d27e-40eb-9e43-5dbfafa5ed18-kube-api-access-s4ht8\") on node \"ip-10-0-140-107.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:18:40.071746 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:18:40.069418 4939 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/dbf522b9-d27e-40eb-9e43-5dbfafa5ed18-provider-credentials\") on node \"ip-10-0-140-107.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:18:40.332020 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:18:40.331991 4939 generic.go:356] "Generic (PLEG): container finished" podID="dbf522b9-d27e-40eb-9e43-5dbfafa5ed18" containerID="ddce77c88fd8799ae6005554fc1e620ee81fcd48d69ee45e12fb264c241cb437" exitCode=0 Nov 03 01:18:40.332202 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:18:40.332033 4939 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-3c5983/payload-processing-external-model-e2e-extmod-b-3c5983-7697gtj7m" Nov 03 01:18:40.332202 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:18:40.332058 4939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-3c5983/payload-processing-external-model-e2e-extmod-b-3c5983-7697gtj7m" event={"ID":"dbf522b9-d27e-40eb-9e43-5dbfafa5ed18","Type":"ContainerDied","Data":"ddce77c88fd8799ae6005554fc1e620ee81fcd48d69ee45e12fb264c241cb437"} Nov 03 01:18:40.332202 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:18:40.332112 4939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-3c5983/payload-processing-external-model-e2e-extmod-b-3c5983-7697gtj7m" event={"ID":"dbf522b9-d27e-40eb-9e43-5dbfafa5ed18","Type":"ContainerDied","Data":"4c4480feed69a615a4fa652aa60965448a5a88d7686512e157a98e3bf3eb0ad8"} Nov 03 01:18:40.332202 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:18:40.332134 4939 scope.go:122] "RemoveContainer" containerID="ddce77c88fd8799ae6005554fc1e620ee81fcd48d69ee45e12fb264c241cb437" Nov 03 01:18:40.345425 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:18:40.345395 4939 scope.go:122] "RemoveContainer" containerID="ddce77c88fd8799ae6005554fc1e620ee81fcd48d69ee45e12fb264c241cb437" Nov 03 01:18:40.345724 ip-10-0-140-107 kubenswrapper[4939]: E1103 01:18:40.345695 4939 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ddce77c88fd8799ae6005554fc1e620ee81fcd48d69ee45e12fb264c241cb437\": container with ID starting with ddce77c88fd8799ae6005554fc1e620ee81fcd48d69ee45e12fb264c241cb437 not found: ID does not exist" containerID="ddce77c88fd8799ae6005554fc1e620ee81fcd48d69ee45e12fb264c241cb437" Nov 03 01:18:40.345819 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:18:40.345735 4939 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ddce77c88fd8799ae6005554fc1e620ee81fcd48d69ee45e12fb264c241cb437"} err="failed to get container status \"ddce77c88fd8799ae6005554fc1e620ee81fcd48d69ee45e12fb264c241cb437\": rpc error: code = NotFound desc = could not find container \"ddce77c88fd8799ae6005554fc1e620ee81fcd48d69ee45e12fb264c241cb437\": container with ID starting with ddce77c88fd8799ae6005554fc1e620ee81fcd48d69ee45e12fb264c241cb437 not found: ID does not exist" Nov 03 01:18:40.361051 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:18:40.361029 4939 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-3c5983/payload-processing-external-model-e2e-extmod-b-3c5983-7697gtj7m"] Nov 03 01:18:40.365747 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:18:40.365716 4939 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-b-3c5983/payload-processing-external-model-e2e-extmod-b-3c5983-7697gtj7m"] Nov 03 01:18:40.675385 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:18:40.675301 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:18:41.207145 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:18:41.207092 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:18:41.324959 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:18:41.324532 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:18:41.465980 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:18:41.464902 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:18:41.871513 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:18:41.871476 4939 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="dbf522b9-d27e-40eb-9e43-5dbfafa5ed18" path="/var/lib/kubelet/pods/dbf522b9-d27e-40eb-9e43-5dbfafa5ed18/volumes" Nov 03 01:18:42.012336 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:18:42.012304 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:18:42.250355 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:18:42.250275 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:18:42.350719 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:18:42.350684 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:18:42.694994 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:18:42.694946 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:18:43.369311 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:18:43.369279 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:18:43.987046 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:18:43.987013 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:18:44.203899 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:18:44.203856 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:18:44.605265 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:18:44.605228 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:18:45.277686 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:18:45.277652 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:18:45.882100 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:18:45.882068 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:18:46.562406 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:18:46.562373 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:18:47.366401 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:18:47.366371 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:18:47.855727 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:18:47.855686 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:18:47.987754 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:18:47.987723 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:18:48.032965 ip-10-0-140-107 kubenswrapper[4939]: E1103 01:18:48.032887 4939 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-sthjh" containerName="extractor" Nov 03 01:18:48.122985 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:18:48.120387 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:18:48.284476 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:18:48.284439 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:18:48.425953 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:18:48.425837 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:18:48.560267 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:18:48.560228 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:18:48.732592 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:18:48.732479 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:18:49.016036 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:18:49.015958 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:18:49.169762 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:18:49.169712 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:18:49.283020 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:18:49.282942 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:18:49.977257 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:18:49.977217 4939 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-a-027c19/payload-processing-external-model-e2e-extmod-a-027c19-7974m9f5c"] Nov 03 01:18:49.978633 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:18:49.978607 4939 cpu_manager.go:414] "containerMap: removing container" podUID="dbf522b9-d27e-40eb-9e43-5dbfafa5ed18" containerName="payload-processing" Nov 03 01:18:49.978706 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:18:49.978636 4939 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dbf522b9-d27e-40eb-9e43-5dbfafa5ed18" containerName="payload-processing" Nov 03 01:18:49.978706 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:18:49.978655 4939 cpu_manager.go:414] "containerMap: removing container" podUID="3057b25c-0d3c-441d-ad3c-00b4166051c0" containerName="payload-processing" Nov 03 01:18:49.978706 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:18:49.978663 4939 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3057b25c-0d3c-441d-ad3c-00b4166051c0" containerName="payload-processing" Nov 03 01:18:49.978988 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:18:49.978972 4939 memory_manager.go:365] "RemoveStaleState removing state" podUID="dbf522b9-d27e-40eb-9e43-5dbfafa5ed18" containerName="payload-processing" Nov 03 01:18:49.979036 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:18:49.978992 4939 memory_manager.go:365] "RemoveStaleState removing state" podUID="3057b25c-0d3c-441d-ad3c-00b4166051c0" containerName="payload-processing" Nov 03 01:18:49.983920 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:18:49.983882 4939 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-027c19/payload-processing-external-model-e2e-extmod-a-027c19-7974m9f5c" Nov 03 01:18:49.987597 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:18:49.987573 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:18:49.988636 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:18:49.988616 4939 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-027c19\"/\"kube-root-ca.crt\"" Nov 03 01:18:49.989171 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:18:49.989120 4939 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-027c19\"/\"agc-cred-902f2ea03c53\"" Nov 03 01:18:49.989281 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:18:49.989178 4939 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-027c19\"/\"payload-processing-external-model-plugins-e2e-extmod-a-027c19\"" Nov 03 01:18:49.989281 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:18:49.989210 4939 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-027c19\"/\"routing-overlay\"" Nov 03 01:18:49.989459 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:18:49.989433 4939 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-027c19\"/\"openshift-service-ca.crt\"" Nov 03 01:18:49.990658 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:18:49.990635 4939 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-027c19/payload-processing-external-model-e2e-extmod-a-027c19-7974m9f5c"] Nov 03 01:18:50.071948 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:18:50.071887 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/13b4c050-74c4-4b51-8d41-816257c2aa51-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-027c19-7974m9f5c\" (UID: \"13b4c050-74c4-4b51-8d41-816257c2aa51\") " pod="ai-tenant-e2e-extmod-a-027c19/payload-processing-external-model-e2e-extmod-a-027c19-7974m9f5c" Nov 03 01:18:50.072156 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:18:50.072066 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/13b4c050-74c4-4b51-8d41-816257c2aa51-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-027c19-7974m9f5c\" (UID: \"13b4c050-74c4-4b51-8d41-816257c2aa51\") " pod="ai-tenant-e2e-extmod-a-027c19/payload-processing-external-model-e2e-extmod-a-027c19-7974m9f5c" Nov 03 01:18:50.072233 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:18:50.072168 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/13b4c050-74c4-4b51-8d41-816257c2aa51-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-027c19-7974m9f5c\" (UID: \"13b4c050-74c4-4b51-8d41-816257c2aa51\") " pod="ai-tenant-e2e-extmod-a-027c19/payload-processing-external-model-e2e-extmod-a-027c19-7974m9f5c" Nov 03 01:18:50.072444 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:18:50.072427 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6dd4s\" (UniqueName: \"kubernetes.io/projected/13b4c050-74c4-4b51-8d41-816257c2aa51-kube-api-access-6dd4s\") pod \"payload-processing-external-model-e2e-extmod-a-027c19-7974m9f5c\" (UID: \"13b4c050-74c4-4b51-8d41-816257c2aa51\") " pod="ai-tenant-e2e-extmod-a-027c19/payload-processing-external-model-e2e-extmod-a-027c19-7974m9f5c" Nov 03 01:18:50.104159 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:18:50.104127 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:18:50.187598 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:18:50.187548 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6dd4s\" (UniqueName: \"kubernetes.io/projected/13b4c050-74c4-4b51-8d41-816257c2aa51-kube-api-access-6dd4s\") pod \"payload-processing-external-model-e2e-extmod-a-027c19-7974m9f5c\" (UID: \"13b4c050-74c4-4b51-8d41-816257c2aa51\") " pod="ai-tenant-e2e-extmod-a-027c19/payload-processing-external-model-e2e-extmod-a-027c19-7974m9f5c" Nov 03 01:18:50.188311 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:18:50.188280 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/13b4c050-74c4-4b51-8d41-816257c2aa51-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-027c19-7974m9f5c\" (UID: \"13b4c050-74c4-4b51-8d41-816257c2aa51\") " pod="ai-tenant-e2e-extmod-a-027c19/payload-processing-external-model-e2e-extmod-a-027c19-7974m9f5c" Nov 03 01:18:50.189185 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:18:50.189116 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/13b4c050-74c4-4b51-8d41-816257c2aa51-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-027c19-7974m9f5c\" (UID: \"13b4c050-74c4-4b51-8d41-816257c2aa51\") " pod="ai-tenant-e2e-extmod-a-027c19/payload-processing-external-model-e2e-extmod-a-027c19-7974m9f5c" Nov 03 01:18:50.189659 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:18:50.189621 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/13b4c050-74c4-4b51-8d41-816257c2aa51-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-027c19-7974m9f5c\" (UID: \"13b4c050-74c4-4b51-8d41-816257c2aa51\") " pod="ai-tenant-e2e-extmod-a-027c19/payload-processing-external-model-e2e-extmod-a-027c19-7974m9f5c" Nov 03 01:18:50.199930 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:18:50.190963 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/13b4c050-74c4-4b51-8d41-816257c2aa51-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-027c19-7974m9f5c\" (UID: \"13b4c050-74c4-4b51-8d41-816257c2aa51\") " pod="ai-tenant-e2e-extmod-a-027c19/payload-processing-external-model-e2e-extmod-a-027c19-7974m9f5c" Nov 03 01:18:50.200807 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:18:50.193757 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/13b4c050-74c4-4b51-8d41-816257c2aa51-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-027c19-7974m9f5c\" (UID: \"13b4c050-74c4-4b51-8d41-816257c2aa51\") " pod="ai-tenant-e2e-extmod-a-027c19/payload-processing-external-model-e2e-extmod-a-027c19-7974m9f5c" Nov 03 01:18:50.221932 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:18:50.218519 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/13b4c050-74c4-4b51-8d41-816257c2aa51-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-027c19-7974m9f5c\" (UID: \"13b4c050-74c4-4b51-8d41-816257c2aa51\") " pod="ai-tenant-e2e-extmod-a-027c19/payload-processing-external-model-e2e-extmod-a-027c19-7974m9f5c" Nov 03 01:18:50.234986 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:18:50.232678 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6dd4s\" (UniqueName: \"kubernetes.io/projected/13b4c050-74c4-4b51-8d41-816257c2aa51-kube-api-access-6dd4s\") pod \"payload-processing-external-model-e2e-extmod-a-027c19-7974m9f5c\" (UID: \"13b4c050-74c4-4b51-8d41-816257c2aa51\") " pod="ai-tenant-e2e-extmod-a-027c19/payload-processing-external-model-e2e-extmod-a-027c19-7974m9f5c" Nov 03 01:18:50.234986 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:18:50.234583 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:18:50.325996 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:18:50.319191 4939 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-027c19/payload-processing-external-model-e2e-extmod-a-027c19-7974m9f5c" Nov 03 01:18:50.760903 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:18:50.760862 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:18:51.044017 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:18:51.037972 4939 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-027c19/payload-processing-external-model-e2e-extmod-a-027c19-7974m9f5c"] Nov 03 01:18:51.047927 ip-10-0-140-107 kubenswrapper[4939]: W1103 01:18:51.046056 4939 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod13b4c050_74c4_4b51_8d41_816257c2aa51.slice/crio-9f21edb9dadf3f14d51e482bacaa02cfef2c967b41b6386b3090cd2677189772 WatchSource:0}: Error finding container 9f21edb9dadf3f14d51e482bacaa02cfef2c967b41b6386b3090cd2677189772: Status 404 returned error can't find the container with id 9f21edb9dadf3f14d51e482bacaa02cfef2c967b41b6386b3090cd2677189772 Nov 03 01:18:51.489564 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:18:51.466963 4939 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-a-027c19/payload-processing-external-model-e2e-extmod-a-027c19-55f89tmjf"] Nov 03 01:18:51.651936 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:18:51.638504 4939 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-027c19/payload-processing-external-model-e2e-extmod-a-027c19-55f89tmjf" Nov 03 01:18:51.661352 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:18:51.661312 4939 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-027c19\"/\"payload-processing-external-model-e2e--4c667f43-dockercfg-x82ws\"" Nov 03 01:18:51.662106 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:18:51.662080 4939 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-027c19\"/\"agc-cred-98635240b877\"" Nov 03 01:18:51.725561 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:18:51.725223 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/8096d833-87a8-4153-b55b-1ce355bfd4f4-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-027c19-55f89tmjf\" (UID: \"8096d833-87a8-4153-b55b-1ce355bfd4f4\") " pod="ai-tenant-e2e-extmod-a-027c19/payload-processing-external-model-e2e-extmod-a-027c19-55f89tmjf" Nov 03 01:18:51.725990 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:18:51.725873 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k48gm\" (UniqueName: \"kubernetes.io/projected/8096d833-87a8-4153-b55b-1ce355bfd4f4-kube-api-access-k48gm\") pod \"payload-processing-external-model-e2e-extmod-a-027c19-55f89tmjf\" (UID: \"8096d833-87a8-4153-b55b-1ce355bfd4f4\") " pod="ai-tenant-e2e-extmod-a-027c19/payload-processing-external-model-e2e-extmod-a-027c19-55f89tmjf" Nov 03 01:18:51.725990 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:18:51.725965 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8096d833-87a8-4153-b55b-1ce355bfd4f4-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-027c19-55f89tmjf\" (UID: \"8096d833-87a8-4153-b55b-1ce355bfd4f4\") " pod="ai-tenant-e2e-extmod-a-027c19/payload-processing-external-model-e2e-extmod-a-027c19-55f89tmjf" Nov 03 01:18:51.726783 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:18:51.726280 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/8096d833-87a8-4153-b55b-1ce355bfd4f4-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-027c19-55f89tmjf\" (UID: \"8096d833-87a8-4153-b55b-1ce355bfd4f4\") " pod="ai-tenant-e2e-extmod-a-027c19/payload-processing-external-model-e2e-extmod-a-027c19-55f89tmjf" Nov 03 01:18:51.842653 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:18:51.841597 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/8096d833-87a8-4153-b55b-1ce355bfd4f4-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-027c19-55f89tmjf\" (UID: \"8096d833-87a8-4153-b55b-1ce355bfd4f4\") " pod="ai-tenant-e2e-extmod-a-027c19/payload-processing-external-model-e2e-extmod-a-027c19-55f89tmjf" Nov 03 01:18:51.842653 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:18:51.841780 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/8096d833-87a8-4153-b55b-1ce355bfd4f4-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-027c19-55f89tmjf\" (UID: \"8096d833-87a8-4153-b55b-1ce355bfd4f4\") " pod="ai-tenant-e2e-extmod-a-027c19/payload-processing-external-model-e2e-extmod-a-027c19-55f89tmjf" Nov 03 01:18:51.842653 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:18:51.841986 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k48gm\" (UniqueName: \"kubernetes.io/projected/8096d833-87a8-4153-b55b-1ce355bfd4f4-kube-api-access-k48gm\") pod \"payload-processing-external-model-e2e-extmod-a-027c19-55f89tmjf\" (UID: \"8096d833-87a8-4153-b55b-1ce355bfd4f4\") " pod="ai-tenant-e2e-extmod-a-027c19/payload-processing-external-model-e2e-extmod-a-027c19-55f89tmjf" Nov 03 01:18:51.842653 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:18:51.842060 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8096d833-87a8-4153-b55b-1ce355bfd4f4-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-027c19-55f89tmjf\" (UID: \"8096d833-87a8-4153-b55b-1ce355bfd4f4\") " pod="ai-tenant-e2e-extmod-a-027c19/payload-processing-external-model-e2e-extmod-a-027c19-55f89tmjf" Nov 03 01:18:51.844010 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:18:51.843213 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8096d833-87a8-4153-b55b-1ce355bfd4f4-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-027c19-55f89tmjf\" (UID: \"8096d833-87a8-4153-b55b-1ce355bfd4f4\") " pod="ai-tenant-e2e-extmod-a-027c19/payload-processing-external-model-e2e-extmod-a-027c19-55f89tmjf" Nov 03 01:18:51.844010 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:18:51.843688 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/8096d833-87a8-4153-b55b-1ce355bfd4f4-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-027c19-55f89tmjf\" (UID: \"8096d833-87a8-4153-b55b-1ce355bfd4f4\") " pod="ai-tenant-e2e-extmod-a-027c19/payload-processing-external-model-e2e-extmod-a-027c19-55f89tmjf" Nov 03 01:18:51.847016 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:18:51.846889 4939 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-027c19/payload-processing-external-model-e2e-extmod-a-027c19-55f89tmjf"] Nov 03 01:18:51.892463 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:18:51.880506 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/8096d833-87a8-4153-b55b-1ce355bfd4f4-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-027c19-55f89tmjf\" (UID: \"8096d833-87a8-4153-b55b-1ce355bfd4f4\") " pod="ai-tenant-e2e-extmod-a-027c19/payload-processing-external-model-e2e-extmod-a-027c19-55f89tmjf" Nov 03 01:18:51.937963 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:18:51.937536 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-k48gm\" (UniqueName: \"kubernetes.io/projected/8096d833-87a8-4153-b55b-1ce355bfd4f4-kube-api-access-k48gm\") pod \"payload-processing-external-model-e2e-extmod-a-027c19-55f89tmjf\" (UID: \"8096d833-87a8-4153-b55b-1ce355bfd4f4\") " pod="ai-tenant-e2e-extmod-a-027c19/payload-processing-external-model-e2e-extmod-a-027c19-55f89tmjf" Nov 03 01:18:51.989964 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:18:51.989040 4939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-027c19/payload-processing-external-model-e2e-extmod-a-027c19-7974m9f5c" event={"ID":"13b4c050-74c4-4b51-8d41-816257c2aa51","Type":"ContainerStarted","Data":"9f21edb9dadf3f14d51e482bacaa02cfef2c967b41b6386b3090cd2677189772"} Nov 03 01:18:52.058326 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:18:52.058290 4939 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-027c19\"/\"payload-processing-external-model-e2e--4c667f43-dockercfg-x82ws\"" Nov 03 01:18:52.065813 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:18:52.065789 4939 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-027c19/payload-processing-external-model-e2e-extmod-a-027c19-55f89tmjf" Nov 03 01:18:52.364824 ip-10-0-140-107 kubenswrapper[4939]: W1103 01:18:52.364791 4939 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8096d833_87a8_4153_b55b_1ce355bfd4f4.slice/crio-83115c760888df70a72fa67845cf5481ce9bc8dd1dee29e1384b7d34b9d58d9d WatchSource:0}: Error finding container 83115c760888df70a72fa67845cf5481ce9bc8dd1dee29e1384b7d34b9d58d9d: Status 404 returned error can't find the container with id 83115c760888df70a72fa67845cf5481ce9bc8dd1dee29e1384b7d34b9d58d9d Nov 03 01:18:52.365894 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:18:52.365866 4939 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-027c19/payload-processing-external-model-e2e-extmod-a-027c19-55f89tmjf"] Nov 03 01:18:52.575166 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:18:52.575124 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:18:52.953867 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:18:52.953827 4939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-027c19/payload-processing-external-model-e2e-extmod-a-027c19-7974m9f5c" event={"ID":"13b4c050-74c4-4b51-8d41-816257c2aa51","Type":"ContainerStarted","Data":"b4b4efe1844579d4bee10470a387960a80ffa647b0ec70c81b95a003102452bd"} Nov 03 01:18:52.954089 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:18:52.953940 4939 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-a-027c19/payload-processing-external-model-e2e-extmod-a-027c19-7974m9f5c" Nov 03 01:18:52.955651 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:18:52.955623 4939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-027c19/payload-processing-external-model-e2e-extmod-a-027c19-55f89tmjf" event={"ID":"8096d833-87a8-4153-b55b-1ce355bfd4f4","Type":"ContainerStarted","Data":"92790343e5cffe5f3283ce8241f6db2bba12461ecfb3eb0a3d0534a9fe8b0ad8"} Nov 03 01:18:52.955793 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:18:52.955671 4939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-027c19/payload-processing-external-model-e2e-extmod-a-027c19-55f89tmjf" event={"ID":"8096d833-87a8-4153-b55b-1ce355bfd4f4","Type":"ContainerStarted","Data":"83115c760888df70a72fa67845cf5481ce9bc8dd1dee29e1384b7d34b9d58d9d"} Nov 03 01:18:52.955793 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:18:52.955695 4939 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-a-027c19/payload-processing-external-model-e2e-extmod-a-027c19-55f89tmjf" Nov 03 01:18:52.971515 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:18:52.971469 4939 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-a-027c19/payload-processing-external-model-e2e-extmod-a-027c19-7974m9f5c" podStartSLOduration=3.9714567819999997 podStartE2EDuration="3.971456782s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 01:18:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 01:18:52.969357 +0000 UTC m=+2281.233691970" watchObservedRunningTime="2026-11-03 01:18:52.971456782 +0000 UTC m=+2281.235791774" Nov 03 01:18:52.989211 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:18:52.989170 4939 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-a-027c19/payload-processing-external-model-e2e-extmod-a-027c19-55f89tmjf" podStartSLOduration=2.989157086 podStartE2EDuration="2.989157086s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 01:18:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 01:18:52.984310454 +0000 UTC m=+2281.248645401" watchObservedRunningTime="2026-11-03 01:18:52.989157086 +0000 UTC m=+2281.253492030" Nov 03 01:18:53.218262 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:18:53.218176 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:18:53.891007 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:18:53.890955 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:18:54.806021 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:18:54.805985 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:18:55.588615 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:18:55.588585 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:18:56.386601 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:18:56.386559 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:18:57.087833 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:18:57.087792 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:18:57.687432 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:18:57.687393 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:18:58.384430 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:18:58.384395 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:18:59.098751 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:18:59.098716 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:18:59.779715 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:18:59.779679 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:19:00.377498 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:19:00.377462 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:19:01.007288 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:19:01.007249 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:19:01.654528 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:19:01.654490 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:19:02.365000 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:19:02.364962 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:19:03.072799 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:19:03.072704 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:19:03.677854 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:19:03.677810 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:19:04.005064 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:19:04.004989 4939 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-a-027c19/payload-processing-external-model-e2e-extmod-a-027c19-55f89tmjf" Nov 03 01:19:04.005221 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:19:04.005069 4939 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-a-027c19/payload-processing-external-model-e2e-extmod-a-027c19-7974m9f5c" Nov 03 01:19:04.084448 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:19:04.084412 4939 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-027c19/payload-processing-external-model-e2e-extmod-a-027c19-7974m9f5c"] Nov 03 01:19:04.419134 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:19:04.419081 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:19:05.105375 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:19:05.105307 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:19:05.851231 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:19:05.851198 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:19:06.495536 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:19:06.495496 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:19:06.671361 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:19:06.671318 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:19:06.811889 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:19:06.811795 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:19:06.878932 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:19:06.876972 4939 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-a-d25759/payload-processing-external-model-e2e-extmod-a-d25759-57647wtkj"] Nov 03 01:19:06.928947 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:19:06.909981 4939 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-d25759/payload-processing-external-model-e2e-extmod-a-d25759-57647wtkj" Nov 03 01:19:07.016450 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:19:07.016418 4939 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-d25759/payload-processing-external-model-e2e-extmod-a-d25759-57647wtkj"] Nov 03 01:19:07.046973 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:19:07.046892 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/ccebcb67-8345-4bf4-8ca4-b5da0fce50fd-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-d25759-57647wtkj\" (UID: \"ccebcb67-8345-4bf4-8ca4-b5da0fce50fd\") " pod="ai-tenant-e2e-extmod-a-d25759/payload-processing-external-model-e2e-extmod-a-d25759-57647wtkj" Nov 03 01:19:07.047228 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:19:07.047201 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kw8zl\" (UniqueName: \"kubernetes.io/projected/ccebcb67-8345-4bf4-8ca4-b5da0fce50fd-kube-api-access-kw8zl\") pod \"payload-processing-external-model-e2e-extmod-a-d25759-57647wtkj\" (UID: \"ccebcb67-8345-4bf4-8ca4-b5da0fce50fd\") " pod="ai-tenant-e2e-extmod-a-d25759/payload-processing-external-model-e2e-extmod-a-d25759-57647wtkj" Nov 03 01:19:07.047352 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:19:07.047313 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/ccebcb67-8345-4bf4-8ca4-b5da0fce50fd-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-d25759-57647wtkj\" (UID: \"ccebcb67-8345-4bf4-8ca4-b5da0fce50fd\") " pod="ai-tenant-e2e-extmod-a-d25759/payload-processing-external-model-e2e-extmod-a-d25759-57647wtkj" Nov 03 01:19:07.047597 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:19:07.047576 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ccebcb67-8345-4bf4-8ca4-b5da0fce50fd-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-d25759-57647wtkj\" (UID: \"ccebcb67-8345-4bf4-8ca4-b5da0fce50fd\") " pod="ai-tenant-e2e-extmod-a-d25759/payload-processing-external-model-e2e-extmod-a-d25759-57647wtkj" Nov 03 01:19:07.149372 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:19:07.149336 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/ccebcb67-8345-4bf4-8ca4-b5da0fce50fd-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-d25759-57647wtkj\" (UID: \"ccebcb67-8345-4bf4-8ca4-b5da0fce50fd\") " pod="ai-tenant-e2e-extmod-a-d25759/payload-processing-external-model-e2e-extmod-a-d25759-57647wtkj" Nov 03 01:19:07.149595 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:19:07.149413 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ccebcb67-8345-4bf4-8ca4-b5da0fce50fd-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-d25759-57647wtkj\" (UID: \"ccebcb67-8345-4bf4-8ca4-b5da0fce50fd\") " pod="ai-tenant-e2e-extmod-a-d25759/payload-processing-external-model-e2e-extmod-a-d25759-57647wtkj" Nov 03 01:19:07.149798 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:19:07.149779 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/ccebcb67-8345-4bf4-8ca4-b5da0fce50fd-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-d25759-57647wtkj\" (UID: \"ccebcb67-8345-4bf4-8ca4-b5da0fce50fd\") " pod="ai-tenant-e2e-extmod-a-d25759/payload-processing-external-model-e2e-extmod-a-d25759-57647wtkj" Nov 03 01:19:07.149876 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:19:07.149839 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kw8zl\" (UniqueName: \"kubernetes.io/projected/ccebcb67-8345-4bf4-8ca4-b5da0fce50fd-kube-api-access-kw8zl\") pod \"payload-processing-external-model-e2e-extmod-a-d25759-57647wtkj\" (UID: \"ccebcb67-8345-4bf4-8ca4-b5da0fce50fd\") " pod="ai-tenant-e2e-extmod-a-d25759/payload-processing-external-model-e2e-extmod-a-d25759-57647wtkj" Nov 03 01:19:07.150265 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:19:07.150227 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/ccebcb67-8345-4bf4-8ca4-b5da0fce50fd-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-d25759-57647wtkj\" (UID: \"ccebcb67-8345-4bf4-8ca4-b5da0fce50fd\") " pod="ai-tenant-e2e-extmod-a-d25759/payload-processing-external-model-e2e-extmod-a-d25759-57647wtkj" Nov 03 01:19:07.150408 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:19:07.150374 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ccebcb67-8345-4bf4-8ca4-b5da0fce50fd-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-d25759-57647wtkj\" (UID: \"ccebcb67-8345-4bf4-8ca4-b5da0fce50fd\") " pod="ai-tenant-e2e-extmod-a-d25759/payload-processing-external-model-e2e-extmod-a-d25759-57647wtkj" Nov 03 01:19:07.152883 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:19:07.152857 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/ccebcb67-8345-4bf4-8ca4-b5da0fce50fd-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-d25759-57647wtkj\" (UID: \"ccebcb67-8345-4bf4-8ca4-b5da0fce50fd\") " pod="ai-tenant-e2e-extmod-a-d25759/payload-processing-external-model-e2e-extmod-a-d25759-57647wtkj" Nov 03 01:19:07.158673 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:19:07.158652 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kw8zl\" (UniqueName: \"kubernetes.io/projected/ccebcb67-8345-4bf4-8ca4-b5da0fce50fd-kube-api-access-kw8zl\") pod \"payload-processing-external-model-e2e-extmod-a-d25759-57647wtkj\" (UID: \"ccebcb67-8345-4bf4-8ca4-b5da0fce50fd\") " pod="ai-tenant-e2e-extmod-a-d25759/payload-processing-external-model-e2e-extmod-a-d25759-57647wtkj" Nov 03 01:19:07.314774 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:19:07.314512 4939 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-d25759/payload-processing-external-model-e2e-extmod-a-d25759-57647wtkj" Nov 03 01:19:07.849156 ip-10-0-140-107 kubenswrapper[4939]: W1103 01:19:07.849120 4939 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podccebcb67_8345_4bf4_8ca4_b5da0fce50fd.slice/crio-3f54b7af11b9ea739f8626f430546ad3062cd8b75d10be32dd09d7e07b7a8d50 WatchSource:0}: Error finding container 3f54b7af11b9ea739f8626f430546ad3062cd8b75d10be32dd09d7e07b7a8d50: Status 404 returned error can't find the container with id 3f54b7af11b9ea739f8626f430546ad3062cd8b75d10be32dd09d7e07b7a8d50 Nov 03 01:19:07.851175 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:19:07.851141 4939 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-d25759/payload-processing-external-model-e2e-extmod-a-d25759-57647wtkj"] Nov 03 01:19:08.093776 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:19:08.093739 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:19:08.276779 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:19:08.276673 4939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-d25759/payload-processing-external-model-e2e-extmod-a-d25759-57647wtkj" event={"ID":"ccebcb67-8345-4bf4-8ca4-b5da0fce50fd","Type":"ContainerStarted","Data":"f898ff8b21257764bac02ec020302a341c0c6f3dc2c708ec25739e7ed87e0503"} Nov 03 01:19:08.276779 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:19:08.276745 4939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-d25759/payload-processing-external-model-e2e-extmod-a-d25759-57647wtkj" event={"ID":"ccebcb67-8345-4bf4-8ca4-b5da0fce50fd","Type":"ContainerStarted","Data":"3f54b7af11b9ea739f8626f430546ad3062cd8b75d10be32dd09d7e07b7a8d50"} Nov 03 01:19:08.276779 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:19:08.276780 4939 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-a-d25759/payload-processing-external-model-e2e-extmod-a-d25759-57647wtkj" Nov 03 01:19:08.298217 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:19:08.298164 4939 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-a-d25759/payload-processing-external-model-e2e-extmod-a-d25759-57647wtkj" podStartSLOduration=2.29814992 podStartE2EDuration="2.29814992s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 01:19:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 01:19:08.293505185 +0000 UTC m=+2296.557840176" watchObservedRunningTime="2026-11-03 01:19:08.29814992 +0000 UTC m=+2296.562484865" Nov 03 01:19:08.788484 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:19:08.788444 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:19:09.170432 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:19:09.169978 4939 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-a-027c19/payload-processing-external-model-e2e-extmod-a-027c19-7974m9f5c" podUID="13b4c050-74c4-4b51-8d41-816257c2aa51" containerName="payload-processing" containerID="cri-o://b4b4efe1844579d4bee10470a387960a80ffa647b0ec70c81b95a003102452bd" gracePeriod=25 Nov 03 01:19:09.290733 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:19:09.290701 4939 generic.go:356] "Generic (PLEG): container finished" podID="13b4c050-74c4-4b51-8d41-816257c2aa51" containerID="b4b4efe1844579d4bee10470a387960a80ffa647b0ec70c81b95a003102452bd" exitCode=0 Nov 03 01:19:09.290856 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:19:09.290757 4939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-027c19/payload-processing-external-model-e2e-extmod-a-027c19-7974m9f5c" event={"ID":"13b4c050-74c4-4b51-8d41-816257c2aa51","Type":"ContainerDied","Data":"b4b4efe1844579d4bee10470a387960a80ffa647b0ec70c81b95a003102452bd"} Nov 03 01:19:09.482516 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:19:09.482486 4939 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-027c19/payload-processing-external-model-e2e-extmod-a-027c19-7974m9f5c" Nov 03 01:19:09.561518 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:19:09.561477 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:19:09.585316 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:19:09.563986 4939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/13b4c050-74c4-4b51-8d41-816257c2aa51-provider-credentials\" (UniqueName: \"kubernetes.io/projected/13b4c050-74c4-4b51-8d41-816257c2aa51-provider-credentials\") pod \"13b4c050-74c4-4b51-8d41-816257c2aa51\" (UID: \"13b4c050-74c4-4b51-8d41-816257c2aa51\") " Nov 03 01:19:09.591782 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:19:09.585683 4939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/13b4c050-74c4-4b51-8d41-816257c2aa51-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/13b4c050-74c4-4b51-8d41-816257c2aa51-routing-overlay-volume\") pod \"13b4c050-74c4-4b51-8d41-816257c2aa51\" (UID: \"13b4c050-74c4-4b51-8d41-816257c2aa51\") " Nov 03 01:19:09.592285 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:19:09.592255 4939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/13b4c050-74c4-4b51-8d41-816257c2aa51-kube-api-access-6dd4s\" (UniqueName: \"kubernetes.io/projected/13b4c050-74c4-4b51-8d41-816257c2aa51-kube-api-access-6dd4s\") pod \"13b4c050-74c4-4b51-8d41-816257c2aa51\" (UID: \"13b4c050-74c4-4b51-8d41-816257c2aa51\") " Nov 03 01:19:09.592529 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:19:09.592501 4939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/13b4c050-74c4-4b51-8d41-816257c2aa51-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/13b4c050-74c4-4b51-8d41-816257c2aa51-plugins-config-volume\") pod \"13b4c050-74c4-4b51-8d41-816257c2aa51\" (UID: \"13b4c050-74c4-4b51-8d41-816257c2aa51\") " Nov 03 01:19:09.594148 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:19:09.569097 4939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/13b4c050-74c4-4b51-8d41-816257c2aa51-provider-credentials" pod "13b4c050-74c4-4b51-8d41-816257c2aa51" (UID: "13b4c050-74c4-4b51-8d41-816257c2aa51"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 01:19:09.594309 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:19:09.594115 4939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/13b4c050-74c4-4b51-8d41-816257c2aa51-plugins-config-volume" pod "13b4c050-74c4-4b51-8d41-816257c2aa51" (UID: "13b4c050-74c4-4b51-8d41-816257c2aa51"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 01:19:09.597034 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:19:09.597003 4939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/13b4c050-74c4-4b51-8d41-816257c2aa51-routing-overlay-volume" pod "13b4c050-74c4-4b51-8d41-816257c2aa51" (UID: "13b4c050-74c4-4b51-8d41-816257c2aa51"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 01:19:09.601138 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:19:09.601107 4939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/13b4c050-74c4-4b51-8d41-816257c2aa51-kube-api-access-6dd4s" pod "13b4c050-74c4-4b51-8d41-816257c2aa51" (UID: "13b4c050-74c4-4b51-8d41-816257c2aa51"). InnerVolumeSpecName "kube-api-access-6dd4s". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 01:19:09.696070 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:19:09.695973 4939 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/13b4c050-74c4-4b51-8d41-816257c2aa51-provider-credentials\") on node \"ip-10-0-140-107.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:19:09.696070 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:19:09.696019 4939 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/13b4c050-74c4-4b51-8d41-816257c2aa51-routing-overlay-volume\") on node \"ip-10-0-140-107.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:19:09.696070 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:19:09.696035 4939 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6dd4s\" (UniqueName: \"kubernetes.io/projected/13b4c050-74c4-4b51-8d41-816257c2aa51-kube-api-access-6dd4s\") on node \"ip-10-0-140-107.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:19:09.696070 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:19:09.696049 4939 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/13b4c050-74c4-4b51-8d41-816257c2aa51-plugins-config-volume\") on node \"ip-10-0-140-107.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:19:10.136720 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:19:10.136678 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:19:10.302444 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:19:10.302413 4939 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-027c19/payload-processing-external-model-e2e-extmod-a-027c19-7974m9f5c" Nov 03 01:19:10.302444 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:19:10.302436 4939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-027c19/payload-processing-external-model-e2e-extmod-a-027c19-7974m9f5c" event={"ID":"13b4c050-74c4-4b51-8d41-816257c2aa51","Type":"ContainerDied","Data":"9f21edb9dadf3f14d51e482bacaa02cfef2c967b41b6386b3090cd2677189772"} Nov 03 01:19:10.302954 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:19:10.302503 4939 scope.go:122] "RemoveContainer" containerID="b4b4efe1844579d4bee10470a387960a80ffa647b0ec70c81b95a003102452bd" Nov 03 01:19:10.325322 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:19:10.325293 4939 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-027c19/payload-processing-external-model-e2e-extmod-a-027c19-7974m9f5c"] Nov 03 01:19:10.330080 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:19:10.330052 4939 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-a-027c19/payload-processing-external-model-e2e-extmod-a-027c19-7974m9f5c"] Nov 03 01:19:10.892858 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:19:10.892823 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:19:11.605516 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:19:11.605484 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:19:11.873105 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:19:11.873026 4939 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="13b4c050-74c4-4b51-8d41-816257c2aa51" path="/var/lib/kubelet/pods/13b4c050-74c4-4b51-8d41-816257c2aa51/volumes" Nov 03 01:19:12.205673 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:19:12.205583 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:19:12.844580 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:19:12.844548 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:19:13.496792 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:19:13.496762 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:19:14.084072 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:19:14.084039 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:19:14.675428 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:19:14.675381 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:19:15.304942 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:19:15.304895 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:19:16.008193 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:19:16.008138 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:19:16.603689 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:19:16.603644 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:19:17.275828 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:19:17.275787 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:19:17.986528 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:19:17.986493 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:19:18.130119 ip-10-0-140-107 kubenswrapper[4939]: E1103 01:19:18.130071 4939 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-sthjh" containerName="extractor" Nov 03 01:19:18.759813 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:19:18.759772 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:19:19.294530 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:19:19.294503 4939 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-a-d25759/payload-processing-external-model-e2e-extmod-a-d25759-57647wtkj" Nov 03 01:19:19.368157 ip-10-0-140-107 kubenswrapper[4939]: E1103 01:19:19.368128 4939 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-a-d25759/agc-cred-213e03dcdac3: secret "agc-cred-213e03dcdac3" not found Nov 03 01:19:19.368387 ip-10-0-140-107 kubenswrapper[4939]: E1103 01:19:19.368371 4939 projected.go:196] Error preparing data for projected volume provider-credentials for pod ai-tenant-e2e-extmod-a-d25759/payload-processing-external-model-e2e-extmod-a-d25759-57647wtkj: secret "agc-cred-213e03dcdac3" not found Nov 03 01:19:19.368560 ip-10-0-140-107 kubenswrapper[4939]: E1103 01:19:19.368547 4939 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/ccebcb67-8345-4bf4-8ca4-b5da0fce50fd-provider-credentials podName:ccebcb67-8345-4bf4-8ca4-b5da0fce50fd nodeName:}" failed. No retries permitted until 2026-11-03 01:19:19.868524985 +0000 UTC m=+2308.132859921 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/ccebcb67-8345-4bf4-8ca4-b5da0fce50fd-provider-credentials") pod "payload-processing-external-model-e2e-extmod-a-d25759-57647wtkj" (UID: "ccebcb67-8345-4bf4-8ca4-b5da0fce50fd") : secret "agc-cred-213e03dcdac3" not found Nov 03 01:19:19.369305 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:19:19.369282 4939 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-d25759/payload-processing-external-model-e2e-extmod-a-d25759-786cq4j8x"] Nov 03 01:19:19.420170 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:19:19.420123 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:19:19.501825 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:19:19.501793 4939 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-a-d25759/payload-processing-external-model-e2e-extmod-a-d25759-b8942fskd"] Nov 03 01:19:19.502801 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:19:19.502785 4939 cpu_manager.go:414] "containerMap: removing container" podUID="13b4c050-74c4-4b51-8d41-816257c2aa51" containerName="payload-processing" Nov 03 01:19:19.502851 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:19:19.502802 4939 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="13b4c050-74c4-4b51-8d41-816257c2aa51" containerName="payload-processing" Nov 03 01:19:19.502959 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:19:19.502949 4939 memory_manager.go:365] "RemoveStaleState removing state" podUID="13b4c050-74c4-4b51-8d41-816257c2aa51" containerName="payload-processing" Nov 03 01:19:19.507042 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:19:19.507015 4939 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-d25759/payload-processing-external-model-e2e-extmod-a-d25759-b8942fskd" Nov 03 01:19:19.516243 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:19:19.516205 4939 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-d25759/payload-processing-external-model-e2e-extmod-a-d25759-b8942fskd"] Nov 03 01:19:19.538858 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:19:19.538825 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:19:19.570533 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:19:19.570457 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/dfc3d56f-062c-4e44-98b0-281dcfc3290c-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-d25759-b8942fskd\" (UID: \"dfc3d56f-062c-4e44-98b0-281dcfc3290c\") " pod="ai-tenant-e2e-extmod-a-d25759/payload-processing-external-model-e2e-extmod-a-d25759-b8942fskd" Nov 03 01:19:19.570702 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:19:19.570581 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/dfc3d56f-062c-4e44-98b0-281dcfc3290c-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-d25759-b8942fskd\" (UID: \"dfc3d56f-062c-4e44-98b0-281dcfc3290c\") " pod="ai-tenant-e2e-extmod-a-d25759/payload-processing-external-model-e2e-extmod-a-d25759-b8942fskd" Nov 03 01:19:19.570877 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:19:19.570853 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s7sxt\" (UniqueName: \"kubernetes.io/projected/dfc3d56f-062c-4e44-98b0-281dcfc3290c-kube-api-access-s7sxt\") pod \"payload-processing-external-model-e2e-extmod-a-d25759-b8942fskd\" (UID: \"dfc3d56f-062c-4e44-98b0-281dcfc3290c\") " pod="ai-tenant-e2e-extmod-a-d25759/payload-processing-external-model-e2e-extmod-a-d25759-b8942fskd" Nov 03 01:19:19.571031 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:19:19.571013 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dfc3d56f-062c-4e44-98b0-281dcfc3290c-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-d25759-b8942fskd\" (UID: \"dfc3d56f-062c-4e44-98b0-281dcfc3290c\") " pod="ai-tenant-e2e-extmod-a-d25759/payload-processing-external-model-e2e-extmod-a-d25759-b8942fskd" Nov 03 01:19:19.679932 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:19:19.675219 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/dfc3d56f-062c-4e44-98b0-281dcfc3290c-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-d25759-b8942fskd\" (UID: \"dfc3d56f-062c-4e44-98b0-281dcfc3290c\") " pod="ai-tenant-e2e-extmod-a-d25759/payload-processing-external-model-e2e-extmod-a-d25759-b8942fskd" Nov 03 01:19:19.679932 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:19:19.675342 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/dfc3d56f-062c-4e44-98b0-281dcfc3290c-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-d25759-b8942fskd\" (UID: \"dfc3d56f-062c-4e44-98b0-281dcfc3290c\") " pod="ai-tenant-e2e-extmod-a-d25759/payload-processing-external-model-e2e-extmod-a-d25759-b8942fskd" Nov 03 01:19:19.679932 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:19:19.675599 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s7sxt\" (UniqueName: \"kubernetes.io/projected/dfc3d56f-062c-4e44-98b0-281dcfc3290c-kube-api-access-s7sxt\") pod \"payload-processing-external-model-e2e-extmod-a-d25759-b8942fskd\" (UID: \"dfc3d56f-062c-4e44-98b0-281dcfc3290c\") " pod="ai-tenant-e2e-extmod-a-d25759/payload-processing-external-model-e2e-extmod-a-d25759-b8942fskd" Nov 03 01:19:19.679932 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:19:19.675689 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dfc3d56f-062c-4e44-98b0-281dcfc3290c-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-d25759-b8942fskd\" (UID: \"dfc3d56f-062c-4e44-98b0-281dcfc3290c\") " pod="ai-tenant-e2e-extmod-a-d25759/payload-processing-external-model-e2e-extmod-a-d25759-b8942fskd" Nov 03 01:19:19.679932 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:19:19.676806 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dfc3d56f-062c-4e44-98b0-281dcfc3290c-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-d25759-b8942fskd\" (UID: \"dfc3d56f-062c-4e44-98b0-281dcfc3290c\") " pod="ai-tenant-e2e-extmod-a-d25759/payload-processing-external-model-e2e-extmod-a-d25759-b8942fskd" Nov 03 01:19:19.679932 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:19:19.677846 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/dfc3d56f-062c-4e44-98b0-281dcfc3290c-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-d25759-b8942fskd\" (UID: \"dfc3d56f-062c-4e44-98b0-281dcfc3290c\") " pod="ai-tenant-e2e-extmod-a-d25759/payload-processing-external-model-e2e-extmod-a-d25759-b8942fskd" Nov 03 01:19:19.706030 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:19:19.705993 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/dfc3d56f-062c-4e44-98b0-281dcfc3290c-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-d25759-b8942fskd\" (UID: \"dfc3d56f-062c-4e44-98b0-281dcfc3290c\") " pod="ai-tenant-e2e-extmod-a-d25759/payload-processing-external-model-e2e-extmod-a-d25759-b8942fskd" Nov 03 01:19:19.751749 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:19:19.751709 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-s7sxt\" (UniqueName: \"kubernetes.io/projected/dfc3d56f-062c-4e44-98b0-281dcfc3290c-kube-api-access-s7sxt\") pod \"payload-processing-external-model-e2e-extmod-a-d25759-b8942fskd\" (UID: \"dfc3d56f-062c-4e44-98b0-281dcfc3290c\") " pod="ai-tenant-e2e-extmod-a-d25759/payload-processing-external-model-e2e-extmod-a-d25759-b8942fskd" Nov 03 01:19:19.846993 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:19:19.836237 4939 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-d25759/payload-processing-external-model-e2e-extmod-a-d25759-b8942fskd" Nov 03 01:19:19.912009 ip-10-0-140-107 kubenswrapper[4939]: E1103 01:19:19.893942 4939 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-a-d25759/agc-cred-213e03dcdac3: secret "agc-cred-213e03dcdac3" not found Nov 03 01:19:19.912009 ip-10-0-140-107 kubenswrapper[4939]: E1103 01:19:19.893992 4939 projected.go:196] Error preparing data for projected volume provider-credentials for pod ai-tenant-e2e-extmod-a-d25759/payload-processing-external-model-e2e-extmod-a-d25759-57647wtkj: secret "agc-cred-213e03dcdac3" not found Nov 03 01:19:19.912009 ip-10-0-140-107 kubenswrapper[4939]: E1103 01:19:19.894071 4939 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/ccebcb67-8345-4bf4-8ca4-b5da0fce50fd-provider-credentials podName:ccebcb67-8345-4bf4-8ca4-b5da0fce50fd nodeName:}" failed. No retries permitted until 2026-11-03 01:19:20.894051584 +0000 UTC m=+2309.158386519 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/ccebcb67-8345-4bf4-8ca4-b5da0fce50fd-provider-credentials") pod "payload-processing-external-model-e2e-extmod-a-d25759-57647wtkj" (UID: "ccebcb67-8345-4bf4-8ca4-b5da0fce50fd") : secret "agc-cred-213e03dcdac3" not found Nov 03 01:19:20.296218 ip-10-0-140-107 kubenswrapper[4939]: W1103 01:19:20.296185 4939 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddfc3d56f_062c_4e44_98b0_281dcfc3290c.slice/crio-c2b84c3abc99c2e83470d1ff63db1e4cdcbf82c1b8ec19ce08e31df8a48c2f4d WatchSource:0}: Error finding container c2b84c3abc99c2e83470d1ff63db1e4cdcbf82c1b8ec19ce08e31df8a48c2f4d: Status 404 returned error can't find the container with id c2b84c3abc99c2e83470d1ff63db1e4cdcbf82c1b8ec19ce08e31df8a48c2f4d Nov 03 01:19:20.296879 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:19:20.296851 4939 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-d25759/payload-processing-external-model-e2e-extmod-a-d25759-b8942fskd"] Nov 03 01:19:20.440179 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:19:20.440143 4939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-d25759/payload-processing-external-model-e2e-extmod-a-d25759-b8942fskd" event={"ID":"dfc3d56f-062c-4e44-98b0-281dcfc3290c","Type":"ContainerStarted","Data":"3b32d03eddb11665d9f06307cffab2dc7e3e1fd21c14ed42bac53c1c0a6cb184"} Nov 03 01:19:20.440382 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:19:20.440195 4939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-d25759/payload-processing-external-model-e2e-extmod-a-d25759-b8942fskd" event={"ID":"dfc3d56f-062c-4e44-98b0-281dcfc3290c","Type":"ContainerStarted","Data":"c2b84c3abc99c2e83470d1ff63db1e4cdcbf82c1b8ec19ce08e31df8a48c2f4d"} Nov 03 01:19:20.440382 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:19:20.440261 4939 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-a-d25759/payload-processing-external-model-e2e-extmod-a-d25759-b8942fskd" Nov 03 01:19:20.461546 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:19:20.461496 4939 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-a-d25759/payload-processing-external-model-e2e-extmod-a-d25759-b8942fskd" podStartSLOduration=1.461485037 podStartE2EDuration="1.461485037s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 01:19:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 01:19:20.457238775 +0000 UTC m=+2308.721573744" watchObservedRunningTime="2026-11-03 01:19:20.461485037 +0000 UTC m=+2308.725819983" Nov 03 01:19:20.559186 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:19:20.559075 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:19:20.692140 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:19:20.692104 4939 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-4pw7v"] Nov 03 01:19:20.916064 ip-10-0-140-107 kubenswrapper[4939]: E1103 01:19:20.916026 4939 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-a-d25759/agc-cred-213e03dcdac3: secret "agc-cred-213e03dcdac3" not found Nov 03 01:19:20.916064 ip-10-0-140-107 kubenswrapper[4939]: E1103 01:19:20.916065 4939 projected.go:196] Error preparing data for projected volume provider-credentials for pod ai-tenant-e2e-extmod-a-d25759/payload-processing-external-model-e2e-extmod-a-d25759-57647wtkj: secret "agc-cred-213e03dcdac3" not found Nov 03 01:19:20.916346 ip-10-0-140-107 kubenswrapper[4939]: E1103 01:19:20.916127 4939 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/ccebcb67-8345-4bf4-8ca4-b5da0fce50fd-provider-credentials podName:ccebcb67-8345-4bf4-8ca4-b5da0fce50fd nodeName:}" failed. No retries permitted until 2026-11-03 01:19:22.916106535 +0000 UTC m=+2311.180441478 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/ccebcb67-8345-4bf4-8ca4-b5da0fce50fd-provider-credentials") pod "payload-processing-external-model-e2e-extmod-a-d25759-57647wtkj" (UID: "ccebcb67-8345-4bf4-8ca4-b5da0fce50fd") : secret "agc-cred-213e03dcdac3" not found Nov 03 01:19:21.374968 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:19:21.374932 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:19:22.843936 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:19:22.843861 4939 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-f7680d/premium-w2-f7680d-kserve-76565b8cf9-hqbbt" podUID="4712e708-18a2-4a13-a12c-a5669397b2c3" containerName="main" probeResult="failure" output="Get \"https://10.132.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 01:19:22.863377 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:19:22.863327 4939 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-2f9a3a/simulator-w0-2f9a3a-kserve-9596bbb8f-2trgz" podUID="207e2207-fb6c-4076-95df-72e95671c261" containerName="main" probeResult="failure" output="Get \"https://10.132.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 01:19:22.937667 ip-10-0-140-107 kubenswrapper[4939]: E1103 01:19:22.937552 4939 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-a-d25759/agc-cred-213e03dcdac3: secret "agc-cred-213e03dcdac3" not found Nov 03 01:19:22.937667 ip-10-0-140-107 kubenswrapper[4939]: E1103 01:19:22.937592 4939 projected.go:196] Error preparing data for projected volume provider-credentials for pod ai-tenant-e2e-extmod-a-d25759/payload-processing-external-model-e2e-extmod-a-d25759-57647wtkj: secret "agc-cred-213e03dcdac3" not found Nov 03 01:19:22.937981 ip-10-0-140-107 kubenswrapper[4939]: E1103 01:19:22.937676 4939 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/ccebcb67-8345-4bf4-8ca4-b5da0fce50fd-provider-credentials podName:ccebcb67-8345-4bf4-8ca4-b5da0fce50fd nodeName:}" failed. No retries permitted until 2026-11-03 01:19:26.937655722 +0000 UTC m=+2315.201990660 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/ccebcb67-8345-4bf4-8ca4-b5da0fce50fd-provider-credentials") pod "payload-processing-external-model-e2e-extmod-a-d25759-57647wtkj" (UID: "ccebcb67-8345-4bf4-8ca4-b5da0fce50fd") : secret "agc-cred-213e03dcdac3" not found Nov 03 01:19:23.386148 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:19:23.386111 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:19:24.197919 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:19:24.197876 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:19:24.370053 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:19:24.370014 4939 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-a-d25759/payload-processing-external-model-e2e-extmod-a-d25759-786cq4j8x" podUID="2aa0d355-deff-4103-a850-ae4195eb3444" containerName="payload-processing" containerID="cri-o://1cd9bff193390a7e9fa8e16c52fbef77abde734bd82ebae4f042e9086ba8d5b4" gracePeriod=25 Nov 03 01:19:24.681227 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:19:24.681198 4939 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-d25759/payload-processing-external-model-e2e-extmod-a-d25759-786cq4j8x" Nov 03 01:19:24.788793 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:19:24.788684 4939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2aa0d355-deff-4103-a850-ae4195eb3444-provider-credentials\" (UniqueName: \"kubernetes.io/projected/2aa0d355-deff-4103-a850-ae4195eb3444-provider-credentials\") pod \"2aa0d355-deff-4103-a850-ae4195eb3444\" (UID: \"2aa0d355-deff-4103-a850-ae4195eb3444\") " Nov 03 01:19:24.789025 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:19:24.788805 4939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2aa0d355-deff-4103-a850-ae4195eb3444-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/2aa0d355-deff-4103-a850-ae4195eb3444-routing-overlay-volume\") pod \"2aa0d355-deff-4103-a850-ae4195eb3444\" (UID: \"2aa0d355-deff-4103-a850-ae4195eb3444\") " Nov 03 01:19:24.789215 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:19:24.789190 4939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2aa0d355-deff-4103-a850-ae4195eb3444-kube-api-access-rt7bx\" (UniqueName: \"kubernetes.io/projected/2aa0d355-deff-4103-a850-ae4195eb3444-kube-api-access-rt7bx\") pod \"2aa0d355-deff-4103-a850-ae4195eb3444\" (UID: \"2aa0d355-deff-4103-a850-ae4195eb3444\") " Nov 03 01:19:24.789297 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:19:24.789254 4939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2aa0d355-deff-4103-a850-ae4195eb3444-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2aa0d355-deff-4103-a850-ae4195eb3444-plugins-config-volume\") pod \"2aa0d355-deff-4103-a850-ae4195eb3444\" (UID: \"2aa0d355-deff-4103-a850-ae4195eb3444\") " Nov 03 01:19:24.789367 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:19:24.789339 4939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2aa0d355-deff-4103-a850-ae4195eb3444-routing-overlay-volume" pod "2aa0d355-deff-4103-a850-ae4195eb3444" (UID: "2aa0d355-deff-4103-a850-ae4195eb3444"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 01:19:24.789711 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:19:24.789682 4939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2aa0d355-deff-4103-a850-ae4195eb3444-plugins-config-volume" pod "2aa0d355-deff-4103-a850-ae4195eb3444" (UID: "2aa0d355-deff-4103-a850-ae4195eb3444"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 01:19:24.790189 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:19:24.790166 4939 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2aa0d355-deff-4103-a850-ae4195eb3444-plugins-config-volume\") on node \"ip-10-0-140-107.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:19:24.790258 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:19:24.790197 4939 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/2aa0d355-deff-4103-a850-ae4195eb3444-routing-overlay-volume\") on node \"ip-10-0-140-107.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:19:24.791320 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:19:24.791294 4939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2aa0d355-deff-4103-a850-ae4195eb3444-provider-credentials" pod "2aa0d355-deff-4103-a850-ae4195eb3444" (UID: "2aa0d355-deff-4103-a850-ae4195eb3444"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 01:19:24.791579 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:19:24.791556 4939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2aa0d355-deff-4103-a850-ae4195eb3444-kube-api-access-rt7bx" pod "2aa0d355-deff-4103-a850-ae4195eb3444" (UID: "2aa0d355-deff-4103-a850-ae4195eb3444"). InnerVolumeSpecName "kube-api-access-rt7bx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 01:19:24.891944 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:19:24.891861 4939 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/2aa0d355-deff-4103-a850-ae4195eb3444-provider-credentials\") on node \"ip-10-0-140-107.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:19:24.891944 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:19:24.891897 4939 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rt7bx\" (UniqueName: \"kubernetes.io/projected/2aa0d355-deff-4103-a850-ae4195eb3444-kube-api-access-rt7bx\") on node \"ip-10-0-140-107.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:19:24.958596 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:19:24.958530 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:19:24.961209 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:19:24.961181 4939 generic.go:356] "Generic (PLEG): container finished" podID="2aa0d355-deff-4103-a850-ae4195eb3444" containerID="1cd9bff193390a7e9fa8e16c52fbef77abde734bd82ebae4f042e9086ba8d5b4" exitCode=0 Nov 03 01:19:24.961341 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:19:24.961226 4939 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-d25759/payload-processing-external-model-e2e-extmod-a-d25759-786cq4j8x" Nov 03 01:19:24.961341 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:19:24.961259 4939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-d25759/payload-processing-external-model-e2e-extmod-a-d25759-786cq4j8x" event={"ID":"2aa0d355-deff-4103-a850-ae4195eb3444","Type":"ContainerDied","Data":"1cd9bff193390a7e9fa8e16c52fbef77abde734bd82ebae4f042e9086ba8d5b4"} Nov 03 01:19:24.961341 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:19:24.961311 4939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-d25759/payload-processing-external-model-e2e-extmod-a-d25759-786cq4j8x" event={"ID":"2aa0d355-deff-4103-a850-ae4195eb3444","Type":"ContainerDied","Data":"240da72a326b72e9ef10cf663a09a5b58fcb34fb66498bbff1b08970761e3fe0"} Nov 03 01:19:24.961341 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:19:24.961335 4939 scope.go:122] "RemoveContainer" containerID="1cd9bff193390a7e9fa8e16c52fbef77abde734bd82ebae4f042e9086ba8d5b4" Nov 03 01:19:24.975257 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:19:24.974219 4939 scope.go:122] "RemoveContainer" containerID="1cd9bff193390a7e9fa8e16c52fbef77abde734bd82ebae4f042e9086ba8d5b4" Nov 03 01:19:24.976164 ip-10-0-140-107 kubenswrapper[4939]: E1103 01:19:24.976064 4939 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1cd9bff193390a7e9fa8e16c52fbef77abde734bd82ebae4f042e9086ba8d5b4\": container with ID starting with 1cd9bff193390a7e9fa8e16c52fbef77abde734bd82ebae4f042e9086ba8d5b4 not found: ID does not exist" containerID="1cd9bff193390a7e9fa8e16c52fbef77abde734bd82ebae4f042e9086ba8d5b4" Nov 03 01:19:24.976164 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:19:24.976101 4939 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1cd9bff193390a7e9fa8e16c52fbef77abde734bd82ebae4f042e9086ba8d5b4"} err="failed to get container status \"1cd9bff193390a7e9fa8e16c52fbef77abde734bd82ebae4f042e9086ba8d5b4\": rpc error: code = NotFound desc = could not find container \"1cd9bff193390a7e9fa8e16c52fbef77abde734bd82ebae4f042e9086ba8d5b4\": container with ID starting with 1cd9bff193390a7e9fa8e16c52fbef77abde734bd82ebae4f042e9086ba8d5b4 not found: ID does not exist" Nov 03 01:19:25.064676 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:19:25.059197 4939 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-d25759/payload-processing-external-model-e2e-extmod-a-d25759-786cq4j8x"] Nov 03 01:19:25.067966 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:19:25.067930 4939 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-a-d25759/payload-processing-external-model-e2e-extmod-a-d25759-786cq4j8x"] Nov 03 01:19:25.597157 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:19:25.597119 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:19:25.715052 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:19:25.715020 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:19:25.823810 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:19:25.823761 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:19:25.877981 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:19:25.877424 4939 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2aa0d355-deff-4103-a850-ae4195eb3444" path="/var/lib/kubelet/pods/2aa0d355-deff-4103-a850-ae4195eb3444/volumes" Nov 03 01:19:26.945228 ip-10-0-140-107 kubenswrapper[4939]: E1103 01:19:26.938602 4939 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-a-d25759/agc-cred-213e03dcdac3: secret "agc-cred-213e03dcdac3" not found Nov 03 01:19:26.945228 ip-10-0-140-107 kubenswrapper[4939]: E1103 01:19:26.938628 4939 projected.go:196] Error preparing data for projected volume provider-credentials for pod ai-tenant-e2e-extmod-a-d25759/payload-processing-external-model-e2e-extmod-a-d25759-57647wtkj: secret "agc-cred-213e03dcdac3" not found Nov 03 01:19:26.945228 ip-10-0-140-107 kubenswrapper[4939]: E1103 01:19:26.938679 4939 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/ccebcb67-8345-4bf4-8ca4-b5da0fce50fd-provider-credentials podName:ccebcb67-8345-4bf4-8ca4-b5da0fce50fd nodeName:}" failed. No retries permitted until 2026-11-03 01:19:34.938665101 +0000 UTC m=+2323.203000040 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/ccebcb67-8345-4bf4-8ca4-b5da0fce50fd-provider-credentials") pod "payload-processing-external-model-e2e-extmod-a-d25759-57647wtkj" (UID: "ccebcb67-8345-4bf4-8ca4-b5da0fce50fd") : secret "agc-cred-213e03dcdac3" not found Nov 03 01:19:27.002921 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:19:27.002837 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:19:27.017997 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:19:27.017959 4939 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-a-ddd653/payload-processing-external-model-e2e-extmod-a-ddd653-84f4dxgcm"] Nov 03 01:19:27.018970 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:19:27.018952 4939 cpu_manager.go:414] "containerMap: removing container" podUID="2aa0d355-deff-4103-a850-ae4195eb3444" containerName="payload-processing" Nov 03 01:19:27.018970 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:19:27.018971 4939 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2aa0d355-deff-4103-a850-ae4195eb3444" containerName="payload-processing" Nov 03 01:19:27.019174 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:19:27.019163 4939 memory_manager.go:365] "RemoveStaleState removing state" podUID="2aa0d355-deff-4103-a850-ae4195eb3444" containerName="payload-processing" Nov 03 01:19:27.025008 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:19:27.024983 4939 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-ddd653/payload-processing-external-model-e2e-extmod-a-ddd653-84f4dxgcm" Nov 03 01:19:27.029417 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:19:27.029390 4939 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-ddd653\"/\"agc-cred-3caad7842c13\"" Nov 03 01:19:27.030750 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:19:27.030724 4939 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-ddd653\"/\"routing-overlay\"" Nov 03 01:19:27.030888 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:19:27.030745 4939 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-ddd653\"/\"payload-processing-external-model-e2e--21c45087-dockercfg-6g7g8\"" Nov 03 01:19:27.030888 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:19:27.030724 4939 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-ddd653\"/\"payload-processing-external-model-plugins-e2e-extmod-a-ddd653\"" Nov 03 01:19:27.030888 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:19:27.030787 4939 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-ddd653\"/\"kube-root-ca.crt\"" Nov 03 01:19:27.030888 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:19:27.030790 4939 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-ddd653\"/\"openshift-service-ca.crt\"" Nov 03 01:19:27.032644 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:19:27.032617 4939 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-ddd653/payload-processing-external-model-e2e-extmod-a-ddd653-84f4dxgcm"] Nov 03 01:19:27.150533 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:19:27.147778 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/56da757c-2a4a-4328-bf6a-bbef5a5e6def-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-ddd653-84f4dxgcm\" (UID: \"56da757c-2a4a-4328-bf6a-bbef5a5e6def\") " pod="ai-tenant-e2e-extmod-a-ddd653/payload-processing-external-model-e2e-extmod-a-ddd653-84f4dxgcm" Nov 03 01:19:27.150533 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:19:27.147874 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/56da757c-2a4a-4328-bf6a-bbef5a5e6def-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-ddd653-84f4dxgcm\" (UID: \"56da757c-2a4a-4328-bf6a-bbef5a5e6def\") " pod="ai-tenant-e2e-extmod-a-ddd653/payload-processing-external-model-e2e-extmod-a-ddd653-84f4dxgcm" Nov 03 01:19:27.150533 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:19:27.148109 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/56da757c-2a4a-4328-bf6a-bbef5a5e6def-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-ddd653-84f4dxgcm\" (UID: \"56da757c-2a4a-4328-bf6a-bbef5a5e6def\") " pod="ai-tenant-e2e-extmod-a-ddd653/payload-processing-external-model-e2e-extmod-a-ddd653-84f4dxgcm" Nov 03 01:19:27.150533 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:19:27.148328 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2thwd\" (UniqueName: \"kubernetes.io/projected/56da757c-2a4a-4328-bf6a-bbef5a5e6def-kube-api-access-2thwd\") pod \"payload-processing-external-model-e2e-extmod-a-ddd653-84f4dxgcm\" (UID: \"56da757c-2a4a-4328-bf6a-bbef5a5e6def\") " pod="ai-tenant-e2e-extmod-a-ddd653/payload-processing-external-model-e2e-extmod-a-ddd653-84f4dxgcm" Nov 03 01:19:27.300980 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:19:27.296271 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/56da757c-2a4a-4328-bf6a-bbef5a5e6def-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-ddd653-84f4dxgcm\" (UID: \"56da757c-2a4a-4328-bf6a-bbef5a5e6def\") " pod="ai-tenant-e2e-extmod-a-ddd653/payload-processing-external-model-e2e-extmod-a-ddd653-84f4dxgcm" Nov 03 01:19:27.300980 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:19:27.296363 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/56da757c-2a4a-4328-bf6a-bbef5a5e6def-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-ddd653-84f4dxgcm\" (UID: \"56da757c-2a4a-4328-bf6a-bbef5a5e6def\") " pod="ai-tenant-e2e-extmod-a-ddd653/payload-processing-external-model-e2e-extmod-a-ddd653-84f4dxgcm" Nov 03 01:19:27.300980 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:19:27.296576 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/56da757c-2a4a-4328-bf6a-bbef5a5e6def-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-ddd653-84f4dxgcm\" (UID: \"56da757c-2a4a-4328-bf6a-bbef5a5e6def\") " pod="ai-tenant-e2e-extmod-a-ddd653/payload-processing-external-model-e2e-extmod-a-ddd653-84f4dxgcm" Nov 03 01:19:27.300980 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:19:27.296790 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2thwd\" (UniqueName: \"kubernetes.io/projected/56da757c-2a4a-4328-bf6a-bbef5a5e6def-kube-api-access-2thwd\") pod \"payload-processing-external-model-e2e-extmod-a-ddd653-84f4dxgcm\" (UID: \"56da757c-2a4a-4328-bf6a-bbef5a5e6def\") " pod="ai-tenant-e2e-extmod-a-ddd653/payload-processing-external-model-e2e-extmod-a-ddd653-84f4dxgcm" Nov 03 01:19:27.300980 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:19:27.298198 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/56da757c-2a4a-4328-bf6a-bbef5a5e6def-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-ddd653-84f4dxgcm\" (UID: \"56da757c-2a4a-4328-bf6a-bbef5a5e6def\") " pod="ai-tenant-e2e-extmod-a-ddd653/payload-processing-external-model-e2e-extmod-a-ddd653-84f4dxgcm" Nov 03 01:19:27.300980 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:19:27.298457 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/56da757c-2a4a-4328-bf6a-bbef5a5e6def-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-ddd653-84f4dxgcm\" (UID: \"56da757c-2a4a-4328-bf6a-bbef5a5e6def\") " pod="ai-tenant-e2e-extmod-a-ddd653/payload-processing-external-model-e2e-extmod-a-ddd653-84f4dxgcm" Nov 03 01:19:27.422980 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:19:27.422586 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/56da757c-2a4a-4328-bf6a-bbef5a5e6def-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-ddd653-84f4dxgcm\" (UID: \"56da757c-2a4a-4328-bf6a-bbef5a5e6def\") " pod="ai-tenant-e2e-extmod-a-ddd653/payload-processing-external-model-e2e-extmod-a-ddd653-84f4dxgcm" Nov 03 01:19:27.495944 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:19:27.495852 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2thwd\" (UniqueName: \"kubernetes.io/projected/56da757c-2a4a-4328-bf6a-bbef5a5e6def-kube-api-access-2thwd\") pod \"payload-processing-external-model-e2e-extmod-a-ddd653-84f4dxgcm\" (UID: \"56da757c-2a4a-4328-bf6a-bbef5a5e6def\") " pod="ai-tenant-e2e-extmod-a-ddd653/payload-processing-external-model-e2e-extmod-a-ddd653-84f4dxgcm" Nov 03 01:19:27.496919 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:19:27.496853 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:19:27.644672 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:19:27.644631 4939 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-ddd653/payload-processing-external-model-e2e-extmod-a-ddd653-84f4dxgcm" Nov 03 01:19:27.675391 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:19:27.675351 4939 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-b-be59df/payload-processing-external-model-e2e-extmod-b-be59df-7c485tstt"] Nov 03 01:19:27.682407 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:19:27.682378 4939 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-be59df/payload-processing-external-model-e2e-extmod-b-be59df-7c485tstt" Nov 03 01:19:27.686639 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:19:27.686610 4939 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-be59df\"/\"agc-cred-7bc4ad9ac349\"" Nov 03 01:19:27.690615 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:19:27.690589 4939 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-be59df/payload-processing-external-model-e2e-extmod-b-be59df-7c485tstt"] Nov 03 01:19:27.758098 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:19:27.758059 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/445d854e-55d7-4a58-8053-7ac0ee779a5b-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-be59df-7c485tstt\" (UID: \"445d854e-55d7-4a58-8053-7ac0ee779a5b\") " pod="ai-tenant-e2e-extmod-b-be59df/payload-processing-external-model-e2e-extmod-b-be59df-7c485tstt" Nov 03 01:19:27.758329 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:19:27.758227 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/445d854e-55d7-4a58-8053-7ac0ee779a5b-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-be59df-7c485tstt\" (UID: \"445d854e-55d7-4a58-8053-7ac0ee779a5b\") " pod="ai-tenant-e2e-extmod-b-be59df/payload-processing-external-model-e2e-extmod-b-be59df-7c485tstt" Nov 03 01:19:27.758329 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:19:27.758281 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/445d854e-55d7-4a58-8053-7ac0ee779a5b-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-be59df-7c485tstt\" (UID: \"445d854e-55d7-4a58-8053-7ac0ee779a5b\") " pod="ai-tenant-e2e-extmod-b-be59df/payload-processing-external-model-e2e-extmod-b-be59df-7c485tstt" Nov 03 01:19:27.758486 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:19:27.758331 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hblzd\" (UniqueName: \"kubernetes.io/projected/445d854e-55d7-4a58-8053-7ac0ee779a5b-kube-api-access-hblzd\") pod \"payload-processing-external-model-e2e-extmod-b-be59df-7c485tstt\" (UID: \"445d854e-55d7-4a58-8053-7ac0ee779a5b\") " pod="ai-tenant-e2e-extmod-b-be59df/payload-processing-external-model-e2e-extmod-b-be59df-7c485tstt" Nov 03 01:19:27.868898 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:19:27.868861 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/445d854e-55d7-4a58-8053-7ac0ee779a5b-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-be59df-7c485tstt\" (UID: \"445d854e-55d7-4a58-8053-7ac0ee779a5b\") " pod="ai-tenant-e2e-extmod-b-be59df/payload-processing-external-model-e2e-extmod-b-be59df-7c485tstt" Nov 03 01:19:27.869183 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:19:27.869054 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/445d854e-55d7-4a58-8053-7ac0ee779a5b-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-be59df-7c485tstt\" (UID: \"445d854e-55d7-4a58-8053-7ac0ee779a5b\") " pod="ai-tenant-e2e-extmod-b-be59df/payload-processing-external-model-e2e-extmod-b-be59df-7c485tstt" Nov 03 01:19:27.869183 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:19:27.869097 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/445d854e-55d7-4a58-8053-7ac0ee779a5b-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-be59df-7c485tstt\" (UID: \"445d854e-55d7-4a58-8053-7ac0ee779a5b\") " pod="ai-tenant-e2e-extmod-b-be59df/payload-processing-external-model-e2e-extmod-b-be59df-7c485tstt" Nov 03 01:19:27.869183 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:19:27.869138 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hblzd\" (UniqueName: \"kubernetes.io/projected/445d854e-55d7-4a58-8053-7ac0ee779a5b-kube-api-access-hblzd\") pod \"payload-processing-external-model-e2e-extmod-b-be59df-7c485tstt\" (UID: \"445d854e-55d7-4a58-8053-7ac0ee779a5b\") " pod="ai-tenant-e2e-extmod-b-be59df/payload-processing-external-model-e2e-extmod-b-be59df-7c485tstt" Nov 03 01:19:27.869822 ip-10-0-140-107 kubenswrapper[4939]: E1103 01:19:27.869795 4939 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-b-be59df/agc-cred-7bc4ad9ac349: secret "agc-cred-7bc4ad9ac349" not found Nov 03 01:19:27.870817 ip-10-0-140-107 kubenswrapper[4939]: E1103 01:19:27.869826 4939 projected.go:196] Error preparing data for projected volume provider-credentials for pod ai-tenant-e2e-extmod-b-be59df/payload-processing-external-model-e2e-extmod-b-be59df-7c485tstt: secret "agc-cred-7bc4ad9ac349" not found Nov 03 01:19:27.870817 ip-10-0-140-107 kubenswrapper[4939]: E1103 01:19:27.870231 4939 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/445d854e-55d7-4a58-8053-7ac0ee779a5b-provider-credentials podName:445d854e-55d7-4a58-8053-7ac0ee779a5b nodeName:}" failed. No retries permitted until 2026-11-03 01:19:28.370208507 +0000 UTC m=+2316.634543444 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/445d854e-55d7-4a58-8053-7ac0ee779a5b-provider-credentials") pod "payload-processing-external-model-e2e-extmod-b-be59df-7c485tstt" (UID: "445d854e-55d7-4a58-8053-7ac0ee779a5b") : secret "agc-cred-7bc4ad9ac349" not found Nov 03 01:19:27.870817 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:19:27.870417 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/445d854e-55d7-4a58-8053-7ac0ee779a5b-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-be59df-7c485tstt\" (UID: \"445d854e-55d7-4a58-8053-7ac0ee779a5b\") " pod="ai-tenant-e2e-extmod-b-be59df/payload-processing-external-model-e2e-extmod-b-be59df-7c485tstt" Nov 03 01:19:27.870817 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:19:27.870470 4939 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-ddd653/payload-processing-external-model-e2e-extmod-a-ddd653-84f4dxgcm"] Nov 03 01:19:27.871363 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:19:27.871342 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/445d854e-55d7-4a58-8053-7ac0ee779a5b-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-be59df-7c485tstt\" (UID: \"445d854e-55d7-4a58-8053-7ac0ee779a5b\") " pod="ai-tenant-e2e-extmod-b-be59df/payload-processing-external-model-e2e-extmod-b-be59df-7c485tstt" Nov 03 01:19:27.877959 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:19:27.877853 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:19:27.880762 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:19:27.880699 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hblzd\" (UniqueName: \"kubernetes.io/projected/445d854e-55d7-4a58-8053-7ac0ee779a5b-kube-api-access-hblzd\") pod \"payload-processing-external-model-e2e-extmod-b-be59df-7c485tstt\" (UID: \"445d854e-55d7-4a58-8053-7ac0ee779a5b\") " pod="ai-tenant-e2e-extmod-b-be59df/payload-processing-external-model-e2e-extmod-b-be59df-7c485tstt" Nov 03 01:19:27.978500 ip-10-0-140-107 kubenswrapper[4939]: W1103 01:19:27.978457 4939 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod56da757c_2a4a_4328_bf6a_bbef5a5e6def.slice/crio-63c97860df371929e1635931c7fcd3f8ebea1214a694cfb5558a7c67c5da0e8f WatchSource:0}: Error finding container 63c97860df371929e1635931c7fcd3f8ebea1214a694cfb5558a7c67c5da0e8f: Status 404 returned error can't find the container with id 63c97860df371929e1635931c7fcd3f8ebea1214a694cfb5558a7c67c5da0e8f Nov 03 01:19:27.979473 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:19:27.979443 4939 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-ddd653/payload-processing-external-model-e2e-extmod-a-ddd653-84f4dxgcm"] Nov 03 01:19:27.981195 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:19:27.981160 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:19:28.327931 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:19:28.302163 4939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-ddd653/payload-processing-external-model-e2e-extmod-a-ddd653-84f4dxgcm" event={"ID":"56da757c-2a4a-4328-bf6a-bbef5a5e6def","Type":"ContainerStarted","Data":"63c97860df371929e1635931c7fcd3f8ebea1214a694cfb5558a7c67c5da0e8f"} Nov 03 01:19:28.393942 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:19:28.393694 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/445d854e-55d7-4a58-8053-7ac0ee779a5b-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-be59df-7c485tstt\" (UID: \"445d854e-55d7-4a58-8053-7ac0ee779a5b\") " pod="ai-tenant-e2e-extmod-b-be59df/payload-processing-external-model-e2e-extmod-b-be59df-7c485tstt" Nov 03 01:19:28.394149 ip-10-0-140-107 kubenswrapper[4939]: E1103 01:19:28.394055 4939 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-b-be59df/agc-cred-7bc4ad9ac349: secret "agc-cred-7bc4ad9ac349" not found Nov 03 01:19:28.394149 ip-10-0-140-107 kubenswrapper[4939]: E1103 01:19:28.394085 4939 projected.go:196] Error preparing data for projected volume provider-credentials for pod ai-tenant-e2e-extmod-b-be59df/payload-processing-external-model-e2e-extmod-b-be59df-7c485tstt: secret "agc-cred-7bc4ad9ac349" not found Nov 03 01:19:28.394294 ip-10-0-140-107 kubenswrapper[4939]: E1103 01:19:28.394152 4939 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/445d854e-55d7-4a58-8053-7ac0ee779a5b-provider-credentials podName:445d854e-55d7-4a58-8053-7ac0ee779a5b nodeName:}" failed. No retries permitted until 2026-11-03 01:19:29.394132989 +0000 UTC m=+2317.658467913 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/445d854e-55d7-4a58-8053-7ac0ee779a5b-provider-credentials") pod "payload-processing-external-model-e2e-extmod-b-be59df-7c485tstt" (UID: "445d854e-55d7-4a58-8053-7ac0ee779a5b") : secret "agc-cred-7bc4ad9ac349" not found Nov 03 01:19:28.955917 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:19:28.955877 4939 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-be59df/payload-processing-external-model-e2e-extmod-b-be59df-7c485tstt"] Nov 03 01:19:28.956532 ip-10-0-140-107 kubenswrapper[4939]: E1103 01:19:28.956489 4939 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[provider-credentials], unattached volumes=[], failed to process volumes=[]: context canceled" pod="ai-tenant-e2e-extmod-b-be59df/payload-processing-external-model-e2e-extmod-b-be59df-7c485tstt" podUID="445d854e-55d7-4a58-8053-7ac0ee779a5b" Nov 03 01:19:29.292713 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:19:29.292610 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:19:29.360071 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:19:29.360034 4939 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-be59df/payload-processing-external-model-e2e-extmod-b-be59df-7c485tstt" Nov 03 01:19:29.361957 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:19:29.361898 4939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-ddd653/payload-processing-external-model-e2e-extmod-a-ddd653-84f4dxgcm" event={"ID":"56da757c-2a4a-4328-bf6a-bbef5a5e6def","Type":"ContainerStarted","Data":"be3cab55636f21798accf42a2d8c8b23108ae7bd1b151bf745fe529d94c529b0"} Nov 03 01:19:29.362562 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:19:29.362540 4939 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-a-ddd653/payload-processing-external-model-e2e-extmod-a-ddd653-84f4dxgcm" Nov 03 01:19:29.393932 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:19:29.386471 4939 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-be59df/payload-processing-external-model-e2e-extmod-b-be59df-7c485tstt" Nov 03 01:19:29.403901 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:19:29.403848 4939 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-a-ddd653/payload-processing-external-model-e2e-extmod-a-ddd653-84f4dxgcm" podStartSLOduration=3.4038340959999998 podStartE2EDuration="3.403834096s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 01:19:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 01:19:29.401464333 +0000 UTC m=+2317.665799302" watchObservedRunningTime="2026-11-03 01:19:29.403834096 +0000 UTC m=+2317.668169042" Nov 03 01:19:29.453202 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:19:29.453166 4939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/445d854e-55d7-4a58-8053-7ac0ee779a5b-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/445d854e-55d7-4a58-8053-7ac0ee779a5b-routing-overlay-volume\") pod \"445d854e-55d7-4a58-8053-7ac0ee779a5b\" (UID: \"445d854e-55d7-4a58-8053-7ac0ee779a5b\") " Nov 03 01:19:29.453378 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:19:29.453283 4939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/445d854e-55d7-4a58-8053-7ac0ee779a5b-kube-api-access-hblzd\" (UniqueName: \"kubernetes.io/projected/445d854e-55d7-4a58-8053-7ac0ee779a5b-kube-api-access-hblzd\") pod \"445d854e-55d7-4a58-8053-7ac0ee779a5b\" (UID: \"445d854e-55d7-4a58-8053-7ac0ee779a5b\") " Nov 03 01:19:29.453474 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:19:29.453462 4939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/445d854e-55d7-4a58-8053-7ac0ee779a5b-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/445d854e-55d7-4a58-8053-7ac0ee779a5b-plugins-config-volume\") pod \"445d854e-55d7-4a58-8053-7ac0ee779a5b\" (UID: \"445d854e-55d7-4a58-8053-7ac0ee779a5b\") " Nov 03 01:19:29.453545 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:19:29.453510 4939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/445d854e-55d7-4a58-8053-7ac0ee779a5b-routing-overlay-volume" pod "445d854e-55d7-4a58-8053-7ac0ee779a5b" (UID: "445d854e-55d7-4a58-8053-7ac0ee779a5b"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 01:19:29.453870 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:19:29.453843 4939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/445d854e-55d7-4a58-8053-7ac0ee779a5b-plugins-config-volume" pod "445d854e-55d7-4a58-8053-7ac0ee779a5b" (UID: "445d854e-55d7-4a58-8053-7ac0ee779a5b"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 01:19:29.454087 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:19:29.454069 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/445d854e-55d7-4a58-8053-7ac0ee779a5b-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-be59df-7c485tstt\" (UID: \"445d854e-55d7-4a58-8053-7ac0ee779a5b\") " pod="ai-tenant-e2e-extmod-b-be59df/payload-processing-external-model-e2e-extmod-b-be59df-7c485tstt" Nov 03 01:19:29.454232 ip-10-0-140-107 kubenswrapper[4939]: E1103 01:19:29.454214 4939 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-b-be59df/agc-cred-7bc4ad9ac349: secret "agc-cred-7bc4ad9ac349" not found Nov 03 01:19:29.454306 ip-10-0-140-107 kubenswrapper[4939]: E1103 01:19:29.454242 4939 projected.go:196] Error preparing data for projected volume provider-credentials for pod ai-tenant-e2e-extmod-b-be59df/payload-processing-external-model-e2e-extmod-b-be59df-7c485tstt: secret "agc-cred-7bc4ad9ac349" not found Nov 03 01:19:29.454366 ip-10-0-140-107 kubenswrapper[4939]: E1103 01:19:29.454320 4939 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/445d854e-55d7-4a58-8053-7ac0ee779a5b-provider-credentials podName:445d854e-55d7-4a58-8053-7ac0ee779a5b nodeName:}" failed. No retries permitted until 2026-11-03 01:19:31.454299891 +0000 UTC m=+2319.718634832 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/445d854e-55d7-4a58-8053-7ac0ee779a5b-provider-credentials") pod "payload-processing-external-model-e2e-extmod-b-be59df-7c485tstt" (UID: "445d854e-55d7-4a58-8053-7ac0ee779a5b") : secret "agc-cred-7bc4ad9ac349" not found Nov 03 01:19:29.454366 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:19:29.454344 4939 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/445d854e-55d7-4a58-8053-7ac0ee779a5b-plugins-config-volume\") on node \"ip-10-0-140-107.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:19:29.454366 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:19:29.454361 4939 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/445d854e-55d7-4a58-8053-7ac0ee779a5b-routing-overlay-volume\") on node \"ip-10-0-140-107.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:19:29.455584 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:19:29.455561 4939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/445d854e-55d7-4a58-8053-7ac0ee779a5b-kube-api-access-hblzd" pod "445d854e-55d7-4a58-8053-7ac0ee779a5b" (UID: "445d854e-55d7-4a58-8053-7ac0ee779a5b"). InnerVolumeSpecName "kube-api-access-hblzd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 01:19:29.556584 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:19:29.556490 4939 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hblzd\" (UniqueName: \"kubernetes.io/projected/445d854e-55d7-4a58-8053-7ac0ee779a5b-kube-api-access-hblzd\") on node \"ip-10-0-140-107.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:19:30.063971 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:19:30.057971 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:19:30.177986 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:19:30.176394 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:19:30.369285 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:19:30.369252 4939 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-be59df/payload-processing-external-model-e2e-extmod-b-be59df-7c485tstt" Nov 03 01:19:30.406023 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:19:30.405985 4939 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-be59df/payload-processing-external-model-e2e-extmod-b-be59df-7c485tstt"] Nov 03 01:19:30.410625 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:19:30.410591 4939 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-b-be59df/payload-processing-external-model-e2e-extmod-b-be59df-7c485tstt"] Nov 03 01:19:30.479447 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:19:30.479359 4939 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/445d854e-55d7-4a58-8053-7ac0ee779a5b-provider-credentials\") on node \"ip-10-0-140-107.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:19:30.978876 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:19:30.978839 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:19:31.120276 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:19:31.120242 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:19:31.134991 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:19:31.134957 4939 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-d25759/payload-processing-external-model-e2e-extmod-a-d25759-b8942fskd"] Nov 03 01:19:31.157027 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:19:31.154861 4939 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-d25759/payload-processing-external-model-e2e-extmod-a-d25759-57647wtkj"] Nov 03 01:19:31.181044 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:19:31.178677 4939 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-a-d25759/payload-processing-external-model-e2e-extmod-a-d25759-b8942fskd" Nov 03 01:19:31.240206 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:19:31.240092 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:19:31.368539 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:19:31.368483 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:19:31.492802 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:19:31.492724 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:19:31.584225 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:19:31.584187 4939 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-a-d25759/payload-processing-external-model-e2e-extmod-a-d25759-7c7b65hzq"] Nov 03 01:19:31.589778 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:19:31.589748 4939 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-d25759/payload-processing-external-model-e2e-extmod-a-d25759-7c7b65hzq" Nov 03 01:19:31.594455 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:19:31.594423 4939 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-d25759\"/\"agc-cred-229a2b395ead\"" Nov 03 01:19:31.594595 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:19:31.594509 4939 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-d25759\"/\"payload-processing-external-model-e2e--4d491765-dockercfg-2c6gz\"" Nov 03 01:19:31.595799 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:19:31.595772 4939 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-d25759/payload-processing-external-model-e2e-extmod-a-d25759-7c7b65hzq"] Nov 03 01:19:31.610663 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:19:31.610601 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:19:31.712955 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:19:31.712895 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/7563b3d3-09b8-41d7-af57-c6f6a8570f2f-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-d25759-7c7b65hzq\" (UID: \"7563b3d3-09b8-41d7-af57-c6f6a8570f2f\") " pod="ai-tenant-e2e-extmod-a-d25759/payload-processing-external-model-e2e-extmod-a-d25759-7c7b65hzq" Nov 03 01:19:31.713133 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:19:31.713045 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7563b3d3-09b8-41d7-af57-c6f6a8570f2f-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-d25759-7c7b65hzq\" (UID: \"7563b3d3-09b8-41d7-af57-c6f6a8570f2f\") " pod="ai-tenant-e2e-extmod-a-d25759/payload-processing-external-model-e2e-extmod-a-d25759-7c7b65hzq" Nov 03 01:19:31.713224 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:19:31.713206 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4kv57\" (UniqueName: \"kubernetes.io/projected/7563b3d3-09b8-41d7-af57-c6f6a8570f2f-kube-api-access-4kv57\") pod \"payload-processing-external-model-e2e-extmod-a-d25759-7c7b65hzq\" (UID: \"7563b3d3-09b8-41d7-af57-c6f6a8570f2f\") " pod="ai-tenant-e2e-extmod-a-d25759/payload-processing-external-model-e2e-extmod-a-d25759-7c7b65hzq" Nov 03 01:19:31.713337 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:19:31.713325 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/7563b3d3-09b8-41d7-af57-c6f6a8570f2f-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-d25759-7c7b65hzq\" (UID: \"7563b3d3-09b8-41d7-af57-c6f6a8570f2f\") " pod="ai-tenant-e2e-extmod-a-d25759/payload-processing-external-model-e2e-extmod-a-d25759-7c7b65hzq" Nov 03 01:19:31.831981 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:19:31.819082 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4kv57\" (UniqueName: \"kubernetes.io/projected/7563b3d3-09b8-41d7-af57-c6f6a8570f2f-kube-api-access-4kv57\") pod \"payload-processing-external-model-e2e-extmod-a-d25759-7c7b65hzq\" (UID: \"7563b3d3-09b8-41d7-af57-c6f6a8570f2f\") " pod="ai-tenant-e2e-extmod-a-d25759/payload-processing-external-model-e2e-extmod-a-d25759-7c7b65hzq" Nov 03 01:19:31.831981 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:19:31.819259 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/7563b3d3-09b8-41d7-af57-c6f6a8570f2f-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-d25759-7c7b65hzq\" (UID: \"7563b3d3-09b8-41d7-af57-c6f6a8570f2f\") " pod="ai-tenant-e2e-extmod-a-d25759/payload-processing-external-model-e2e-extmod-a-d25759-7c7b65hzq" Nov 03 01:19:31.831981 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:19:31.819469 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/7563b3d3-09b8-41d7-af57-c6f6a8570f2f-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-d25759-7c7b65hzq\" (UID: \"7563b3d3-09b8-41d7-af57-c6f6a8570f2f\") " pod="ai-tenant-e2e-extmod-a-d25759/payload-processing-external-model-e2e-extmod-a-d25759-7c7b65hzq" Nov 03 01:19:31.831981 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:19:31.819548 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7563b3d3-09b8-41d7-af57-c6f6a8570f2f-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-d25759-7c7b65hzq\" (UID: \"7563b3d3-09b8-41d7-af57-c6f6a8570f2f\") " pod="ai-tenant-e2e-extmod-a-d25759/payload-processing-external-model-e2e-extmod-a-d25759-7c7b65hzq" Nov 03 01:19:31.831981 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:19:31.829517 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7563b3d3-09b8-41d7-af57-c6f6a8570f2f-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-d25759-7c7b65hzq\" (UID: \"7563b3d3-09b8-41d7-af57-c6f6a8570f2f\") " pod="ai-tenant-e2e-extmod-a-d25759/payload-processing-external-model-e2e-extmod-a-d25759-7c7b65hzq" Nov 03 01:19:31.832836 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:19:31.832335 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/7563b3d3-09b8-41d7-af57-c6f6a8570f2f-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-d25759-7c7b65hzq\" (UID: \"7563b3d3-09b8-41d7-af57-c6f6a8570f2f\") " pod="ai-tenant-e2e-extmod-a-d25759/payload-processing-external-model-e2e-extmod-a-d25759-7c7b65hzq" Nov 03 01:19:31.851820 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:19:31.844648 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4kv57\" (UniqueName: \"kubernetes.io/projected/7563b3d3-09b8-41d7-af57-c6f6a8570f2f-kube-api-access-4kv57\") pod \"payload-processing-external-model-e2e-extmod-a-d25759-7c7b65hzq\" (UID: \"7563b3d3-09b8-41d7-af57-c6f6a8570f2f\") " pod="ai-tenant-e2e-extmod-a-d25759/payload-processing-external-model-e2e-extmod-a-d25759-7c7b65hzq" Nov 03 01:19:31.872562 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:19:31.866234 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/7563b3d3-09b8-41d7-af57-c6f6a8570f2f-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-d25759-7c7b65hzq\" (UID: \"7563b3d3-09b8-41d7-af57-c6f6a8570f2f\") " pod="ai-tenant-e2e-extmod-a-d25759/payload-processing-external-model-e2e-extmod-a-d25759-7c7b65hzq" Nov 03 01:19:31.908930 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:19:31.908784 4939 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-d25759/payload-processing-external-model-e2e-extmod-a-d25759-7c7b65hzq" Nov 03 01:19:32.081708 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:19:32.081662 4939 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="445d854e-55d7-4a58-8053-7ac0ee779a5b" path="/var/lib/kubelet/pods/445d854e-55d7-4a58-8053-7ac0ee779a5b/volumes" Nov 03 01:19:32.447972 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:19:32.447794 4939 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-d25759/payload-processing-external-model-e2e-extmod-a-d25759-7c7b65hzq"] Nov 03 01:19:32.472985 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:19:32.472950 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:19:32.583825 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:19:32.583788 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:19:32.617997 ip-10-0-140-107 kubenswrapper[4939]: W1103 01:19:32.617967 4939 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7563b3d3_09b8_41d7_af57_c6f6a8570f2f.slice/crio-a157448982ed7e928213191371bc2c7b48ad97ee387efc40cf93b3a474d1fd39 WatchSource:0}: Error finding container a157448982ed7e928213191371bc2c7b48ad97ee387efc40cf93b3a474d1fd39: Status 404 returned error can't find the container with id a157448982ed7e928213191371bc2c7b48ad97ee387efc40cf93b3a474d1fd39 Nov 03 01:19:32.618582 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:19:32.618558 4939 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-d25759/payload-processing-external-model-e2e-extmod-a-d25759-7c7b65hzq"] Nov 03 01:19:33.075014 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:19:33.074986 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:19:33.402264 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:19:33.402230 4939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-d25759/payload-processing-external-model-e2e-extmod-a-d25759-7c7b65hzq" event={"ID":"7563b3d3-09b8-41d7-af57-c6f6a8570f2f","Type":"ContainerStarted","Data":"04e346d2c7bbaa11a622674927f8e3dd3334ec5ecdfadf88b160273641376ceb"} Nov 03 01:19:33.402488 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:19:33.402289 4939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-d25759/payload-processing-external-model-e2e-extmod-a-d25759-7c7b65hzq" event={"ID":"7563b3d3-09b8-41d7-af57-c6f6a8570f2f","Type":"ContainerStarted","Data":"a157448982ed7e928213191371bc2c7b48ad97ee387efc40cf93b3a474d1fd39"} Nov 03 01:19:33.402488 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:19:33.402348 4939 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-a-d25759/payload-processing-external-model-e2e-extmod-a-d25759-7c7b65hzq" Nov 03 01:19:33.420471 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:19:33.420413 4939 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-a-d25759/payload-processing-external-model-e2e-extmod-a-d25759-7c7b65hzq" podStartSLOduration=2.420398414 podStartE2EDuration="2.420398414s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 01:19:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 01:19:33.418228971 +0000 UTC m=+2321.682563928" watchObservedRunningTime="2026-11-03 01:19:33.420398414 +0000 UTC m=+2321.684733358" Nov 03 01:19:33.672572 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:19:33.672482 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:19:34.275463 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:19:34.275432 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:19:34.365307 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:19:34.365251 4939 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-a-ddd653/payload-processing-external-model-e2e-extmod-a-ddd653-84f4dxgcm" podUID="56da757c-2a4a-4328-bf6a-bbef5a5e6def" containerName="payload-processing" containerID="cri-o://be3cab55636f21798accf42a2d8c8b23108ae7bd1b151bf745fe529d94c529b0" gracePeriod=25 Nov 03 01:19:34.677638 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:19:34.677608 4939 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-ddd653/payload-processing-external-model-e2e-extmod-a-ddd653-84f4dxgcm" Nov 03 01:19:34.743857 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:19:34.743819 4939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/56da757c-2a4a-4328-bf6a-bbef5a5e6def-provider-credentials\" (UniqueName: \"kubernetes.io/projected/56da757c-2a4a-4328-bf6a-bbef5a5e6def-provider-credentials\") pod \"56da757c-2a4a-4328-bf6a-bbef5a5e6def\" (UID: \"56da757c-2a4a-4328-bf6a-bbef5a5e6def\") " Nov 03 01:19:34.744111 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:19:34.743892 4939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/56da757c-2a4a-4328-bf6a-bbef5a5e6def-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/56da757c-2a4a-4328-bf6a-bbef5a5e6def-plugins-config-volume\") pod \"56da757c-2a4a-4328-bf6a-bbef5a5e6def\" (UID: \"56da757c-2a4a-4328-bf6a-bbef5a5e6def\") " Nov 03 01:19:34.744214 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:19:34.744180 4939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/56da757c-2a4a-4328-bf6a-bbef5a5e6def-kube-api-access-2thwd\" (UniqueName: \"kubernetes.io/projected/56da757c-2a4a-4328-bf6a-bbef5a5e6def-kube-api-access-2thwd\") pod \"56da757c-2a4a-4328-bf6a-bbef5a5e6def\" (UID: \"56da757c-2a4a-4328-bf6a-bbef5a5e6def\") " Nov 03 01:19:34.744364 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:19:34.744343 4939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/56da757c-2a4a-4328-bf6a-bbef5a5e6def-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/56da757c-2a4a-4328-bf6a-bbef5a5e6def-routing-overlay-volume\") pod \"56da757c-2a4a-4328-bf6a-bbef5a5e6def\" (UID: \"56da757c-2a4a-4328-bf6a-bbef5a5e6def\") " Nov 03 01:19:34.744492 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:19:34.744382 4939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/56da757c-2a4a-4328-bf6a-bbef5a5e6def-plugins-config-volume" pod "56da757c-2a4a-4328-bf6a-bbef5a5e6def" (UID: "56da757c-2a4a-4328-bf6a-bbef5a5e6def"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 01:19:34.744896 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:19:34.744867 4939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/56da757c-2a4a-4328-bf6a-bbef5a5e6def-routing-overlay-volume" pod "56da757c-2a4a-4328-bf6a-bbef5a5e6def" (UID: "56da757c-2a4a-4328-bf6a-bbef5a5e6def"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 01:19:34.745357 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:19:34.745331 4939 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/56da757c-2a4a-4328-bf6a-bbef5a5e6def-plugins-config-volume\") on node \"ip-10-0-140-107.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:19:34.745465 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:19:34.745362 4939 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/56da757c-2a4a-4328-bf6a-bbef5a5e6def-routing-overlay-volume\") on node \"ip-10-0-140-107.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:19:34.746832 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:19:34.746808 4939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/56da757c-2a4a-4328-bf6a-bbef5a5e6def-kube-api-access-2thwd" pod "56da757c-2a4a-4328-bf6a-bbef5a5e6def" (UID: "56da757c-2a4a-4328-bf6a-bbef5a5e6def"). InnerVolumeSpecName "kube-api-access-2thwd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 01:19:34.751288 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:19:34.751268 4939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/56da757c-2a4a-4328-bf6a-bbef5a5e6def-provider-credentials" pod "56da757c-2a4a-4328-bf6a-bbef5a5e6def" (UID: "56da757c-2a4a-4328-bf6a-bbef5a5e6def"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 01:19:34.847414 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:19:34.847379 4939 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2thwd\" (UniqueName: \"kubernetes.io/projected/56da757c-2a4a-4328-bf6a-bbef5a5e6def-kube-api-access-2thwd\") on node \"ip-10-0-140-107.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:19:34.847414 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:19:34.847409 4939 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/56da757c-2a4a-4328-bf6a-bbef5a5e6def-provider-credentials\") on node \"ip-10-0-140-107.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:19:34.949526 ip-10-0-140-107 kubenswrapper[4939]: E1103 01:19:34.949283 4939 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-a-d25759/agc-cred-213e03dcdac3: secret "agc-cred-213e03dcdac3" not found Nov 03 01:19:34.949526 ip-10-0-140-107 kubenswrapper[4939]: E1103 01:19:34.949315 4939 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-a-d25759/e2e-cfl-e257d5-provider-cred: secret "e2e-cfl-e257d5-provider-cred" not found Nov 03 01:19:34.949526 ip-10-0-140-107 kubenswrapper[4939]: E1103 01:19:34.949327 4939 projected.go:196] Error preparing data for projected volume provider-credentials for pod ai-tenant-e2e-extmod-a-d25759/payload-processing-external-model-e2e-extmod-a-d25759-57647wtkj: [secret "agc-cred-213e03dcdac3" not found, secret "e2e-cfl-e257d5-provider-cred" not found] Nov 03 01:19:34.949526 ip-10-0-140-107 kubenswrapper[4939]: E1103 01:19:34.949393 4939 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/ccebcb67-8345-4bf4-8ca4-b5da0fce50fd-provider-credentials podName:ccebcb67-8345-4bf4-8ca4-b5da0fce50fd nodeName:}" failed. No retries permitted until 2026-11-03 01:19:50.949373069 +0000 UTC m=+2339.213708012 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/ccebcb67-8345-4bf4-8ca4-b5da0fce50fd-provider-credentials") pod "payload-processing-external-model-e2e-extmod-a-d25759-57647wtkj" (UID: "ccebcb67-8345-4bf4-8ca4-b5da0fce50fd") : [secret "agc-cred-213e03dcdac3" not found, secret "e2e-cfl-e257d5-provider-cred" not found] Nov 03 01:19:34.991521 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:19:34.991481 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:19:35.422670 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:19:35.422634 4939 generic.go:356] "Generic (PLEG): container finished" podID="56da757c-2a4a-4328-bf6a-bbef5a5e6def" containerID="be3cab55636f21798accf42a2d8c8b23108ae7bd1b151bf745fe529d94c529b0" exitCode=0 Nov 03 01:19:35.422953 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:19:35.422702 4939 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-ddd653/payload-processing-external-model-e2e-extmod-a-ddd653-84f4dxgcm" Nov 03 01:19:35.422953 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:19:35.422720 4939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-ddd653/payload-processing-external-model-e2e-extmod-a-ddd653-84f4dxgcm" event={"ID":"56da757c-2a4a-4328-bf6a-bbef5a5e6def","Type":"ContainerDied","Data":"be3cab55636f21798accf42a2d8c8b23108ae7bd1b151bf745fe529d94c529b0"} Nov 03 01:19:35.422953 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:19:35.422794 4939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-ddd653/payload-processing-external-model-e2e-extmod-a-ddd653-84f4dxgcm" event={"ID":"56da757c-2a4a-4328-bf6a-bbef5a5e6def","Type":"ContainerDied","Data":"63c97860df371929e1635931c7fcd3f8ebea1214a694cfb5558a7c67c5da0e8f"} Nov 03 01:19:35.422953 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:19:35.422817 4939 scope.go:122] "RemoveContainer" containerID="be3cab55636f21798accf42a2d8c8b23108ae7bd1b151bf745fe529d94c529b0" Nov 03 01:19:35.435530 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:19:35.435509 4939 scope.go:122] "RemoveContainer" containerID="be3cab55636f21798accf42a2d8c8b23108ae7bd1b151bf745fe529d94c529b0" Nov 03 01:19:35.435814 ip-10-0-140-107 kubenswrapper[4939]: E1103 01:19:35.435791 4939 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"be3cab55636f21798accf42a2d8c8b23108ae7bd1b151bf745fe529d94c529b0\": container with ID starting with be3cab55636f21798accf42a2d8c8b23108ae7bd1b151bf745fe529d94c529b0 not found: ID does not exist" containerID="be3cab55636f21798accf42a2d8c8b23108ae7bd1b151bf745fe529d94c529b0" Nov 03 01:19:35.435899 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:19:35.435819 4939 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"be3cab55636f21798accf42a2d8c8b23108ae7bd1b151bf745fe529d94c529b0"} err="failed to get container status \"be3cab55636f21798accf42a2d8c8b23108ae7bd1b151bf745fe529d94c529b0\": rpc error: code = NotFound desc = could not find container \"be3cab55636f21798accf42a2d8c8b23108ae7bd1b151bf745fe529d94c529b0\": container with ID starting with be3cab55636f21798accf42a2d8c8b23108ae7bd1b151bf745fe529d94c529b0 not found: ID does not exist" Nov 03 01:19:35.452625 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:19:35.452588 4939 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-ddd653/payload-processing-external-model-e2e-extmod-a-ddd653-84f4dxgcm"] Nov 03 01:19:35.456989 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:19:35.456962 4939 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-a-ddd653/payload-processing-external-model-e2e-extmod-a-ddd653-84f4dxgcm"] Nov 03 01:19:35.668428 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:19:35.668388 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:19:35.872426 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:19:35.872393 4939 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="56da757c-2a4a-4328-bf6a-bbef5a5e6def" path="/var/lib/kubelet/pods/56da757c-2a4a-4328-bf6a-bbef5a5e6def/volumes" Nov 03 01:19:36.136022 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:19:36.135937 4939 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-a-d25759/payload-processing-external-model-e2e-extmod-a-d25759-b8942fskd" podUID="dfc3d56f-062c-4e44-98b0-281dcfc3290c" containerName="payload-processing" containerID="cri-o://3b32d03eddb11665d9f06307cffab2dc7e3e1fd21c14ed42bac53c1c0a6cb184" gracePeriod=25 Nov 03 01:19:36.156345 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:19:36.156306 4939 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-a-d25759/payload-processing-external-model-e2e-extmod-a-d25759-57647wtkj" podUID="ccebcb67-8345-4bf4-8ca4-b5da0fce50fd" containerName="payload-processing" containerID="cri-o://f898ff8b21257764bac02ec020302a341c0c6f3dc2c708ec25739e7ed87e0503" gracePeriod=25 Nov 03 01:19:36.295007 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:19:36.294945 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:19:36.466598 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:19:36.460596 4939 generic.go:356] "Generic (PLEG): container finished" podID="dfc3d56f-062c-4e44-98b0-281dcfc3290c" containerID="3b32d03eddb11665d9f06307cffab2dc7e3e1fd21c14ed42bac53c1c0a6cb184" exitCode=0 Nov 03 01:19:36.466598 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:19:36.460736 4939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-d25759/payload-processing-external-model-e2e-extmod-a-d25759-b8942fskd" event={"ID":"dfc3d56f-062c-4e44-98b0-281dcfc3290c","Type":"ContainerDied","Data":"3b32d03eddb11665d9f06307cffab2dc7e3e1fd21c14ed42bac53c1c0a6cb184"} Nov 03 01:19:36.466598 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:19:36.462457 4939 generic.go:356] "Generic (PLEG): container finished" podID="ccebcb67-8345-4bf4-8ca4-b5da0fce50fd" containerID="f898ff8b21257764bac02ec020302a341c0c6f3dc2c708ec25739e7ed87e0503" exitCode=0 Nov 03 01:19:36.466598 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:19:36.462565 4939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-d25759/payload-processing-external-model-e2e-extmod-a-d25759-57647wtkj" event={"ID":"ccebcb67-8345-4bf4-8ca4-b5da0fce50fd","Type":"ContainerDied","Data":"f898ff8b21257764bac02ec020302a341c0c6f3dc2c708ec25739e7ed87e0503"} Nov 03 01:19:36.593620 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:19:36.593592 4939 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-d25759/payload-processing-external-model-e2e-extmod-a-d25759-b8942fskd" Nov 03 01:19:36.649835 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:19:36.649807 4939 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-d25759/payload-processing-external-model-e2e-extmod-a-d25759-57647wtkj" Nov 03 01:19:36.722091 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:19:36.722000 4939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/dfc3d56f-062c-4e44-98b0-281dcfc3290c-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dfc3d56f-062c-4e44-98b0-281dcfc3290c-plugins-config-volume\") pod \"dfc3d56f-062c-4e44-98b0-281dcfc3290c\" (UID: \"dfc3d56f-062c-4e44-98b0-281dcfc3290c\") " Nov 03 01:19:36.722091 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:19:36.722079 4939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ccebcb67-8345-4bf4-8ca4-b5da0fce50fd-kube-api-access-kw8zl\" (UniqueName: \"kubernetes.io/projected/ccebcb67-8345-4bf4-8ca4-b5da0fce50fd-kube-api-access-kw8zl\") pod \"ccebcb67-8345-4bf4-8ca4-b5da0fce50fd\" (UID: \"ccebcb67-8345-4bf4-8ca4-b5da0fce50fd\") " Nov 03 01:19:36.722336 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:19:36.722147 4939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ccebcb67-8345-4bf4-8ca4-b5da0fce50fd-provider-credentials\" (UniqueName: \"kubernetes.io/projected/ccebcb67-8345-4bf4-8ca4-b5da0fce50fd-provider-credentials\") pod \"ccebcb67-8345-4bf4-8ca4-b5da0fce50fd\" (UID: \"ccebcb67-8345-4bf4-8ca4-b5da0fce50fd\") " Nov 03 01:19:36.722336 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:19:36.722317 4939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ccebcb67-8345-4bf4-8ca4-b5da0fce50fd-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ccebcb67-8345-4bf4-8ca4-b5da0fce50fd-plugins-config-volume\") pod \"ccebcb67-8345-4bf4-8ca4-b5da0fce50fd\" (UID: \"ccebcb67-8345-4bf4-8ca4-b5da0fce50fd\") " Nov 03 01:19:36.722466 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:19:36.722353 4939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dfc3d56f-062c-4e44-98b0-281dcfc3290c-kube-api-access-s7sxt\" (UniqueName: \"kubernetes.io/projected/dfc3d56f-062c-4e44-98b0-281dcfc3290c-kube-api-access-s7sxt\") pod \"dfc3d56f-062c-4e44-98b0-281dcfc3290c\" (UID: \"dfc3d56f-062c-4e44-98b0-281dcfc3290c\") " Nov 03 01:19:36.722528 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:19:36.722474 4939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ccebcb67-8345-4bf4-8ca4-b5da0fce50fd-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/ccebcb67-8345-4bf4-8ca4-b5da0fce50fd-routing-overlay-volume\") pod \"ccebcb67-8345-4bf4-8ca4-b5da0fce50fd\" (UID: \"ccebcb67-8345-4bf4-8ca4-b5da0fce50fd\") " Nov 03 01:19:36.722528 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:19:36.722473 4939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dfc3d56f-062c-4e44-98b0-281dcfc3290c-plugins-config-volume" pod "dfc3d56f-062c-4e44-98b0-281dcfc3290c" (UID: "dfc3d56f-062c-4e44-98b0-281dcfc3290c"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 01:19:36.722650 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:19:36.722583 4939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dfc3d56f-062c-4e44-98b0-281dcfc3290c-provider-credentials\" (UniqueName: \"kubernetes.io/projected/dfc3d56f-062c-4e44-98b0-281dcfc3290c-provider-credentials\") pod \"dfc3d56f-062c-4e44-98b0-281dcfc3290c\" (UID: \"dfc3d56f-062c-4e44-98b0-281dcfc3290c\") " Nov 03 01:19:36.722650 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:19:36.722625 4939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/dfc3d56f-062c-4e44-98b0-281dcfc3290c-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/dfc3d56f-062c-4e44-98b0-281dcfc3290c-routing-overlay-volume\") pod \"dfc3d56f-062c-4e44-98b0-281dcfc3290c\" (UID: \"dfc3d56f-062c-4e44-98b0-281dcfc3290c\") " Nov 03 01:19:36.723079 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:19:36.722969 4939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ccebcb67-8345-4bf4-8ca4-b5da0fce50fd-plugins-config-volume" pod "ccebcb67-8345-4bf4-8ca4-b5da0fce50fd" (UID: "ccebcb67-8345-4bf4-8ca4-b5da0fce50fd"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 01:19:36.723079 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:19:36.722997 4939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ccebcb67-8345-4bf4-8ca4-b5da0fce50fd-routing-overlay-volume" pod "ccebcb67-8345-4bf4-8ca4-b5da0fce50fd" (UID: "ccebcb67-8345-4bf4-8ca4-b5da0fce50fd"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 01:19:36.723455 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:19:36.723413 4939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dfc3d56f-062c-4e44-98b0-281dcfc3290c-routing-overlay-volume" pod "dfc3d56f-062c-4e44-98b0-281dcfc3290c" (UID: "dfc3d56f-062c-4e44-98b0-281dcfc3290c"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 01:19:36.724035 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:19:36.724006 4939 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ccebcb67-8345-4bf4-8ca4-b5da0fce50fd-plugins-config-volume\") on node \"ip-10-0-140-107.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:19:36.724035 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:19:36.724038 4939 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/ccebcb67-8345-4bf4-8ca4-b5da0fce50fd-routing-overlay-volume\") on node \"ip-10-0-140-107.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:19:36.724281 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:19:36.724054 4939 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/dfc3d56f-062c-4e44-98b0-281dcfc3290c-routing-overlay-volume\") on node \"ip-10-0-140-107.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:19:36.724281 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:19:36.724070 4939 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dfc3d56f-062c-4e44-98b0-281dcfc3290c-plugins-config-volume\") on node \"ip-10-0-140-107.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:19:36.725350 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:19:36.725322 4939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ccebcb67-8345-4bf4-8ca4-b5da0fce50fd-kube-api-access-kw8zl" pod "ccebcb67-8345-4bf4-8ca4-b5da0fce50fd" (UID: "ccebcb67-8345-4bf4-8ca4-b5da0fce50fd"). InnerVolumeSpecName "kube-api-access-kw8zl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 01:19:36.725449 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:19:36.725359 4939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ccebcb67-8345-4bf4-8ca4-b5da0fce50fd-provider-credentials" pod "ccebcb67-8345-4bf4-8ca4-b5da0fce50fd" (UID: "ccebcb67-8345-4bf4-8ca4-b5da0fce50fd"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 01:19:36.725508 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:19:36.725480 4939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dfc3d56f-062c-4e44-98b0-281dcfc3290c-provider-credentials" pod "dfc3d56f-062c-4e44-98b0-281dcfc3290c" (UID: "dfc3d56f-062c-4e44-98b0-281dcfc3290c"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 01:19:36.726764 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:19:36.726738 4939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dfc3d56f-062c-4e44-98b0-281dcfc3290c-kube-api-access-s7sxt" pod "dfc3d56f-062c-4e44-98b0-281dcfc3290c" (UID: "dfc3d56f-062c-4e44-98b0-281dcfc3290c"). InnerVolumeSpecName "kube-api-access-s7sxt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 01:19:36.825981 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:19:36.825945 4939 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-s7sxt\" (UniqueName: \"kubernetes.io/projected/dfc3d56f-062c-4e44-98b0-281dcfc3290c-kube-api-access-s7sxt\") on node \"ip-10-0-140-107.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:19:36.825981 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:19:36.825978 4939 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/dfc3d56f-062c-4e44-98b0-281dcfc3290c-provider-credentials\") on node \"ip-10-0-140-107.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:19:36.825981 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:19:36.825989 4939 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kw8zl\" (UniqueName: \"kubernetes.io/projected/ccebcb67-8345-4bf4-8ca4-b5da0fce50fd-kube-api-access-kw8zl\") on node \"ip-10-0-140-107.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:19:36.826224 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:19:36.825998 4939 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/ccebcb67-8345-4bf4-8ca4-b5da0fce50fd-provider-credentials\") on node \"ip-10-0-140-107.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:19:37.005586 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:19:37.005503 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:19:37.475277 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:19:37.475242 4939 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-d25759/payload-processing-external-model-e2e-extmod-a-d25759-57647wtkj" Nov 03 01:19:37.475277 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:19:37.475259 4939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-d25759/payload-processing-external-model-e2e-extmod-a-d25759-57647wtkj" event={"ID":"ccebcb67-8345-4bf4-8ca4-b5da0fce50fd","Type":"ContainerDied","Data":"3f54b7af11b9ea739f8626f430546ad3062cd8b75d10be32dd09d7e07b7a8d50"} Nov 03 01:19:37.475612 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:19:37.475353 4939 scope.go:122] "RemoveContainer" containerID="f898ff8b21257764bac02ec020302a341c0c6f3dc2c708ec25739e7ed87e0503" Nov 03 01:19:37.477227 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:19:37.477195 4939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-d25759/payload-processing-external-model-e2e-extmod-a-d25759-b8942fskd" event={"ID":"dfc3d56f-062c-4e44-98b0-281dcfc3290c","Type":"ContainerDied","Data":"c2b84c3abc99c2e83470d1ff63db1e4cdcbf82c1b8ec19ce08e31df8a48c2f4d"} Nov 03 01:19:37.477373 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:19:37.477262 4939 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-d25759/payload-processing-external-model-e2e-extmod-a-d25759-b8942fskd" Nov 03 01:19:37.491252 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:19:37.491219 4939 scope.go:122] "RemoveContainer" containerID="3b32d03eddb11665d9f06307cffab2dc7e3e1fd21c14ed42bac53c1c0a6cb184" Nov 03 01:19:37.508212 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:19:37.508187 4939 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-d25759/payload-processing-external-model-e2e-extmod-a-d25759-b8942fskd"] Nov 03 01:19:37.512901 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:19:37.512874 4939 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-a-d25759/payload-processing-external-model-e2e-extmod-a-d25759-b8942fskd"] Nov 03 01:19:37.521439 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:19:37.521416 4939 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-d25759/payload-processing-external-model-e2e-extmod-a-d25759-57647wtkj"] Nov 03 01:19:37.525377 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:19:37.525357 4939 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-a-d25759/payload-processing-external-model-e2e-extmod-a-d25759-57647wtkj"] Nov 03 01:19:37.705478 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:19:37.705440 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:19:37.872432 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:19:37.872400 4939 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ccebcb67-8345-4bf4-8ca4-b5da0fce50fd" path="/var/lib/kubelet/pods/ccebcb67-8345-4bf4-8ca4-b5da0fce50fd/volumes" Nov 03 01:19:37.872859 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:19:37.872845 4939 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="dfc3d56f-062c-4e44-98b0-281dcfc3290c" path="/var/lib/kubelet/pods/dfc3d56f-062c-4e44-98b0-281dcfc3290c/volumes" Nov 03 01:19:38.403107 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:19:38.403038 4939 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-a-d25759/payload-processing-external-model-e2e-extmod-a-d25759-7c7b65hzq" podUID="7563b3d3-09b8-41d7-af57-c6f6a8570f2f" containerName="payload-processing" containerID="cri-o://04e346d2c7bbaa11a622674927f8e3dd3334ec5ecdfadf88b160273641376ceb" gracePeriod=25 Nov 03 01:19:38.419622 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:19:38.419587 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:19:38.742891 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:19:38.742863 4939 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-d25759/payload-processing-external-model-e2e-extmod-a-d25759-7c7b65hzq" Nov 03 01:19:38.888304 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:19:38.888271 4939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7563b3d3-09b8-41d7-af57-c6f6a8570f2f-kube-api-access-4kv57\" (UniqueName: \"kubernetes.io/projected/7563b3d3-09b8-41d7-af57-c6f6a8570f2f-kube-api-access-4kv57\") pod \"7563b3d3-09b8-41d7-af57-c6f6a8570f2f\" (UID: \"7563b3d3-09b8-41d7-af57-c6f6a8570f2f\") " Nov 03 01:19:38.888515 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:19:38.888459 4939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7563b3d3-09b8-41d7-af57-c6f6a8570f2f-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/7563b3d3-09b8-41d7-af57-c6f6a8570f2f-routing-overlay-volume\") pod \"7563b3d3-09b8-41d7-af57-c6f6a8570f2f\" (UID: \"7563b3d3-09b8-41d7-af57-c6f6a8570f2f\") " Nov 03 01:19:38.888632 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:19:38.888617 4939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7563b3d3-09b8-41d7-af57-c6f6a8570f2f-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7563b3d3-09b8-41d7-af57-c6f6a8570f2f-plugins-config-volume\") pod \"7563b3d3-09b8-41d7-af57-c6f6a8570f2f\" (UID: \"7563b3d3-09b8-41d7-af57-c6f6a8570f2f\") " Nov 03 01:19:38.888699 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:19:38.888661 4939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7563b3d3-09b8-41d7-af57-c6f6a8570f2f-provider-credentials\" (UniqueName: \"kubernetes.io/projected/7563b3d3-09b8-41d7-af57-c6f6a8570f2f-provider-credentials\") pod \"7563b3d3-09b8-41d7-af57-c6f6a8570f2f\" (UID: \"7563b3d3-09b8-41d7-af57-c6f6a8570f2f\") " Nov 03 01:19:38.888829 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:19:38.888803 4939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7563b3d3-09b8-41d7-af57-c6f6a8570f2f-routing-overlay-volume" pod "7563b3d3-09b8-41d7-af57-c6f6a8570f2f" (UID: "7563b3d3-09b8-41d7-af57-c6f6a8570f2f"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 01:19:38.889084 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:19:38.889060 4939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7563b3d3-09b8-41d7-af57-c6f6a8570f2f-plugins-config-volume" pod "7563b3d3-09b8-41d7-af57-c6f6a8570f2f" (UID: "7563b3d3-09b8-41d7-af57-c6f6a8570f2f"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 01:19:38.889358 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:19:38.889334 4939 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7563b3d3-09b8-41d7-af57-c6f6a8570f2f-plugins-config-volume\") on node \"ip-10-0-140-107.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:19:38.889358 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:19:38.889358 4939 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/7563b3d3-09b8-41d7-af57-c6f6a8570f2f-routing-overlay-volume\") on node \"ip-10-0-140-107.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:19:38.891049 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:19:38.891019 4939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7563b3d3-09b8-41d7-af57-c6f6a8570f2f-provider-credentials" pod "7563b3d3-09b8-41d7-af57-c6f6a8570f2f" (UID: "7563b3d3-09b8-41d7-af57-c6f6a8570f2f"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 01:19:38.891143 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:19:38.891075 4939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7563b3d3-09b8-41d7-af57-c6f6a8570f2f-kube-api-access-4kv57" pod "7563b3d3-09b8-41d7-af57-c6f6a8570f2f" (UID: "7563b3d3-09b8-41d7-af57-c6f6a8570f2f"). InnerVolumeSpecName "kube-api-access-4kv57". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 01:19:38.991405 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:19:38.991324 4939 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/7563b3d3-09b8-41d7-af57-c6f6a8570f2f-provider-credentials\") on node \"ip-10-0-140-107.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:19:38.991405 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:19:38.991362 4939 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4kv57\" (UniqueName: \"kubernetes.io/projected/7563b3d3-09b8-41d7-af57-c6f6a8570f2f-kube-api-access-4kv57\") on node \"ip-10-0-140-107.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:19:39.131342 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:19:39.131306 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:19:39.438342 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:19:39.438303 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:19:39.521657 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:19:39.521623 4939 generic.go:356] "Generic (PLEG): container finished" podID="7563b3d3-09b8-41d7-af57-c6f6a8570f2f" containerID="04e346d2c7bbaa11a622674927f8e3dd3334ec5ecdfadf88b160273641376ceb" exitCode=0 Nov 03 01:19:39.521837 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:19:39.521667 4939 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-d25759/payload-processing-external-model-e2e-extmod-a-d25759-7c7b65hzq" Nov 03 01:19:39.521837 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:19:39.521707 4939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-d25759/payload-processing-external-model-e2e-extmod-a-d25759-7c7b65hzq" event={"ID":"7563b3d3-09b8-41d7-af57-c6f6a8570f2f","Type":"ContainerDied","Data":"04e346d2c7bbaa11a622674927f8e3dd3334ec5ecdfadf88b160273641376ceb"} Nov 03 01:19:39.521837 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:19:39.521765 4939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-d25759/payload-processing-external-model-e2e-extmod-a-d25759-7c7b65hzq" event={"ID":"7563b3d3-09b8-41d7-af57-c6f6a8570f2f","Type":"ContainerDied","Data":"a157448982ed7e928213191371bc2c7b48ad97ee387efc40cf93b3a474d1fd39"} Nov 03 01:19:39.521837 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:19:39.521792 4939 scope.go:122] "RemoveContainer" containerID="04e346d2c7bbaa11a622674927f8e3dd3334ec5ecdfadf88b160273641376ceb" Nov 03 01:19:39.535109 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:19:39.535083 4939 scope.go:122] "RemoveContainer" containerID="04e346d2c7bbaa11a622674927f8e3dd3334ec5ecdfadf88b160273641376ceb" Nov 03 01:19:39.535426 ip-10-0-140-107 kubenswrapper[4939]: E1103 01:19:39.535405 4939 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"04e346d2c7bbaa11a622674927f8e3dd3334ec5ecdfadf88b160273641376ceb\": container with ID starting with 04e346d2c7bbaa11a622674927f8e3dd3334ec5ecdfadf88b160273641376ceb not found: ID does not exist" containerID="04e346d2c7bbaa11a622674927f8e3dd3334ec5ecdfadf88b160273641376ceb" Nov 03 01:19:39.535488 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:19:39.535434 4939 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"04e346d2c7bbaa11a622674927f8e3dd3334ec5ecdfadf88b160273641376ceb"} err="failed to get container status \"04e346d2c7bbaa11a622674927f8e3dd3334ec5ecdfadf88b160273641376ceb\": rpc error: code = NotFound desc = could not find container \"04e346d2c7bbaa11a622674927f8e3dd3334ec5ecdfadf88b160273641376ceb\": container with ID starting with 04e346d2c7bbaa11a622674927f8e3dd3334ec5ecdfadf88b160273641376ceb not found: ID does not exist" Nov 03 01:19:39.547674 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:19:39.547647 4939 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-d25759/payload-processing-external-model-e2e-extmod-a-d25759-7c7b65hzq"] Nov 03 01:19:39.555330 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:19:39.555305 4939 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-a-d25759/payload-processing-external-model-e2e-extmod-a-d25759-7c7b65hzq"] Nov 03 01:19:39.557213 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:19:39.557192 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:19:39.681498 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:19:39.681460 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:19:39.871687 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:19:39.871654 4939 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7563b3d3-09b8-41d7-af57-c6f6a8570f2f" path="/var/lib/kubelet/pods/7563b3d3-09b8-41d7-af57-c6f6a8570f2f/volumes" Nov 03 01:19:39.905307 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:19:39.905276 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:19:40.518363 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:19:40.518329 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:19:41.291152 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:19:41.291117 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:19:41.842923 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:19:41.842880 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:19:41.995263 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:19:41.995211 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:19:42.147701 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:19:42.147601 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:19:42.790738 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:19:42.790704 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:19:43.367722 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:19:43.367678 4939 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-be59df/payload-processing-external-model-e2e-extmod-b-be59df-5885fq5qq"] Nov 03 01:19:43.403403 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:19:43.403352 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:19:43.519387 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:19:43.519352 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:19:43.970126 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:19:43.967479 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:19:44.099116 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:19:44.098462 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:19:44.334749 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:19:44.334712 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:19:44.904745 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:19:44.904714 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:19:45.516474 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:19:45.516432 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:19:45.811964 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:19:45.811840 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:19:45.918166 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:19:45.918129 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:19:46.272156 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:19:46.272120 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:19:46.645547 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:19:46.645497 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:19:46.774930 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:19:46.774774 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:19:47.342252 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:19:47.342221 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:19:47.356517 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:19:47.356487 4939 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["models-as-a-service/payload-processing-external-model-858d96bddf-mjhfr"] Nov 03 01:19:47.357442 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:19:47.357424 4939 cpu_manager.go:414] "containerMap: removing container" podUID="ccebcb67-8345-4bf4-8ca4-b5da0fce50fd" containerName="payload-processing" Nov 03 01:19:47.357513 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:19:47.357444 4939 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ccebcb67-8345-4bf4-8ca4-b5da0fce50fd" containerName="payload-processing" Nov 03 01:19:47.357513 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:19:47.357455 4939 cpu_manager.go:414] "containerMap: removing container" podUID="dfc3d56f-062c-4e44-98b0-281dcfc3290c" containerName="payload-processing" Nov 03 01:19:47.357513 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:19:47.357461 4939 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dfc3d56f-062c-4e44-98b0-281dcfc3290c" containerName="payload-processing" Nov 03 01:19:47.357513 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:19:47.357490 4939 cpu_manager.go:414] "containerMap: removing container" podUID="7563b3d3-09b8-41d7-af57-c6f6a8570f2f" containerName="payload-processing" Nov 03 01:19:47.357513 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:19:47.357495 4939 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7563b3d3-09b8-41d7-af57-c6f6a8570f2f" containerName="payload-processing" Nov 03 01:19:47.357685 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:19:47.357530 4939 cpu_manager.go:414] "containerMap: removing container" podUID="56da757c-2a4a-4328-bf6a-bbef5a5e6def" containerName="payload-processing" Nov 03 01:19:47.357685 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:19:47.357535 4939 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="56da757c-2a4a-4328-bf6a-bbef5a5e6def" containerName="payload-processing" Nov 03 01:19:47.357685 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:19:47.357679 4939 memory_manager.go:365] "RemoveStaleState removing state" podUID="56da757c-2a4a-4328-bf6a-bbef5a5e6def" containerName="payload-processing" Nov 03 01:19:47.357784 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:19:47.357692 4939 memory_manager.go:365] "RemoveStaleState removing state" podUID="ccebcb67-8345-4bf4-8ca4-b5da0fce50fd" containerName="payload-processing" Nov 03 01:19:47.357784 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:19:47.357703 4939 memory_manager.go:365] "RemoveStaleState removing state" podUID="dfc3d56f-062c-4e44-98b0-281dcfc3290c" containerName="payload-processing" Nov 03 01:19:47.357784 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:19:47.357714 4939 memory_manager.go:365] "RemoveStaleState removing state" podUID="7563b3d3-09b8-41d7-af57-c6f6a8570f2f" containerName="payload-processing" Nov 03 01:19:47.361753 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:19:47.361718 4939 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-858d96bddf-mjhfr" Nov 03 01:19:47.365283 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:19:47.365255 4939 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"models-as-a-service\"/\"agc-cred-2b77e15a3e7d\"" Nov 03 01:19:47.366078 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:19:47.366052 4939 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"models-as-a-service\"/\"kube-root-ca.crt\"" Nov 03 01:19:47.366229 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:19:47.366162 4939 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"models-as-a-service\"/\"routing-overlay\"" Nov 03 01:19:47.366229 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:19:47.366226 4939 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"models-as-a-service\"/\"payload-processing-external-model-plugins\"" Nov 03 01:19:47.366370 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:19:47.366332 4939 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"models-as-a-service\"/\"payload-processing-external-model-dockercfg-k4l22\"" Nov 03 01:19:47.366370 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:19:47.366363 4939 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"models-as-a-service\"/\"openshift-service-ca.crt\"" Nov 03 01:19:47.370425 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:19:47.370402 4939 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["models-as-a-service/payload-processing-external-model-858d96bddf-mjhfr"] Nov 03 01:19:47.582933 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:19:47.559042 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/e4404321-b58e-4ddf-9264-cf8790848ac0-routing-overlay-volume\") pod \"payload-processing-external-model-858d96bddf-mjhfr\" (UID: \"e4404321-b58e-4ddf-9264-cf8790848ac0\") " pod="models-as-a-service/payload-processing-external-model-858d96bddf-mjhfr" Nov 03 01:19:47.582933 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:19:47.559281 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/e4404321-b58e-4ddf-9264-cf8790848ac0-provider-credentials\") pod \"payload-processing-external-model-858d96bddf-mjhfr\" (UID: \"e4404321-b58e-4ddf-9264-cf8790848ac0\") " pod="models-as-a-service/payload-processing-external-model-858d96bddf-mjhfr" Nov 03 01:19:47.582933 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:19:47.559349 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e4404321-b58e-4ddf-9264-cf8790848ac0-plugins-config-volume\") pod \"payload-processing-external-model-858d96bddf-mjhfr\" (UID: \"e4404321-b58e-4ddf-9264-cf8790848ac0\") " pod="models-as-a-service/payload-processing-external-model-858d96bddf-mjhfr" Nov 03 01:19:47.582933 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:19:47.559431 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k5skd\" (UniqueName: \"kubernetes.io/projected/e4404321-b58e-4ddf-9264-cf8790848ac0-kube-api-access-k5skd\") pod \"payload-processing-external-model-858d96bddf-mjhfr\" (UID: \"e4404321-b58e-4ddf-9264-cf8790848ac0\") " pod="models-as-a-service/payload-processing-external-model-858d96bddf-mjhfr" Nov 03 01:19:47.758680 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:19:47.747985 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/e4404321-b58e-4ddf-9264-cf8790848ac0-provider-credentials\") pod \"payload-processing-external-model-858d96bddf-mjhfr\" (UID: \"e4404321-b58e-4ddf-9264-cf8790848ac0\") " pod="models-as-a-service/payload-processing-external-model-858d96bddf-mjhfr" Nov 03 01:19:47.758680 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:19:47.748143 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e4404321-b58e-4ddf-9264-cf8790848ac0-plugins-config-volume\") pod \"payload-processing-external-model-858d96bddf-mjhfr\" (UID: \"e4404321-b58e-4ddf-9264-cf8790848ac0\") " pod="models-as-a-service/payload-processing-external-model-858d96bddf-mjhfr" Nov 03 01:19:47.758680 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:19:47.748287 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k5skd\" (UniqueName: \"kubernetes.io/projected/e4404321-b58e-4ddf-9264-cf8790848ac0-kube-api-access-k5skd\") pod \"payload-processing-external-model-858d96bddf-mjhfr\" (UID: \"e4404321-b58e-4ddf-9264-cf8790848ac0\") " pod="models-as-a-service/payload-processing-external-model-858d96bddf-mjhfr" Nov 03 01:19:47.758680 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:19:47.748616 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/e4404321-b58e-4ddf-9264-cf8790848ac0-routing-overlay-volume\") pod \"payload-processing-external-model-858d96bddf-mjhfr\" (UID: \"e4404321-b58e-4ddf-9264-cf8790848ac0\") " pod="models-as-a-service/payload-processing-external-model-858d96bddf-mjhfr" Nov 03 01:19:47.758680 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:19:47.749849 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/e4404321-b58e-4ddf-9264-cf8790848ac0-routing-overlay-volume\") pod \"payload-processing-external-model-858d96bddf-mjhfr\" (UID: \"e4404321-b58e-4ddf-9264-cf8790848ac0\") " pod="models-as-a-service/payload-processing-external-model-858d96bddf-mjhfr" Nov 03 01:19:47.758680 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:19:47.756777 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e4404321-b58e-4ddf-9264-cf8790848ac0-plugins-config-volume\") pod \"payload-processing-external-model-858d96bddf-mjhfr\" (UID: \"e4404321-b58e-4ddf-9264-cf8790848ac0\") " pod="models-as-a-service/payload-processing-external-model-858d96bddf-mjhfr" Nov 03 01:19:47.831648 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:19:47.831614 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/e4404321-b58e-4ddf-9264-cf8790848ac0-provider-credentials\") pod \"payload-processing-external-model-858d96bddf-mjhfr\" (UID: \"e4404321-b58e-4ddf-9264-cf8790848ac0\") " pod="models-as-a-service/payload-processing-external-model-858d96bddf-mjhfr" Nov 03 01:19:47.842438 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:19:47.842407 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-k5skd\" (UniqueName: \"kubernetes.io/projected/e4404321-b58e-4ddf-9264-cf8790848ac0-kube-api-access-k5skd\") pod \"payload-processing-external-model-858d96bddf-mjhfr\" (UID: \"e4404321-b58e-4ddf-9264-cf8790848ac0\") " pod="models-as-a-service/payload-processing-external-model-858d96bddf-mjhfr" Nov 03 01:19:47.982402 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:19:47.982365 4939 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-858d96bddf-mjhfr" Nov 03 01:19:47.984034 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:19:47.984003 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:19:48.123273 ip-10-0-140-107 kubenswrapper[4939]: E1103 01:19:48.123229 4939 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-sthjh" containerName="extractor" Nov 03 01:19:48.368819 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:19:48.368725 4939 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-b-be59df/payload-processing-external-model-e2e-extmod-b-be59df-5885fq5qq" podUID="1e022297-33c3-41ab-a32f-ab20ef6676c8" containerName="payload-processing" containerID="cri-o://3a372aaea621d5acf2cc1c5655a8034526c6b483feea35604ee9d711dda91c6e" gracePeriod=25 Nov 03 01:19:48.398712 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:19:48.398683 4939 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["models-as-a-service/payload-processing-external-model-858d96bddf-mjhfr"] Nov 03 01:19:48.456086 ip-10-0-140-107 kubenswrapper[4939]: W1103 01:19:48.456044 4939 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode4404321_b58e_4ddf_9264_cf8790848ac0.slice/crio-d550a37d7717159eec487ac896f9d79ec8003aff397735e4f7f1eb21d5f3fe3c WatchSource:0}: Error finding container d550a37d7717159eec487ac896f9d79ec8003aff397735e4f7f1eb21d5f3fe3c: Status 404 returned error can't find the container with id d550a37d7717159eec487ac896f9d79ec8003aff397735e4f7f1eb21d5f3fe3c Nov 03 01:19:48.664142 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:19:48.664111 4939 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-be59df/payload-processing-external-model-e2e-extmod-b-be59df-5885fq5qq" Nov 03 01:19:48.775468 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:19:48.775440 4939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1e022297-33c3-41ab-a32f-ab20ef6676c8-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1e022297-33c3-41ab-a32f-ab20ef6676c8-plugins-config-volume\") pod \"1e022297-33c3-41ab-a32f-ab20ef6676c8\" (UID: \"1e022297-33c3-41ab-a32f-ab20ef6676c8\") " Nov 03 01:19:48.775602 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:19:48.775564 4939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1e022297-33c3-41ab-a32f-ab20ef6676c8-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/1e022297-33c3-41ab-a32f-ab20ef6676c8-routing-overlay-volume\") pod \"1e022297-33c3-41ab-a32f-ab20ef6676c8\" (UID: \"1e022297-33c3-41ab-a32f-ab20ef6676c8\") " Nov 03 01:19:48.775674 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:19:48.775659 4939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1e022297-33c3-41ab-a32f-ab20ef6676c8-kube-api-access-j9cwv\" (UniqueName: \"kubernetes.io/projected/1e022297-33c3-41ab-a32f-ab20ef6676c8-kube-api-access-j9cwv\") pod \"1e022297-33c3-41ab-a32f-ab20ef6676c8\" (UID: \"1e022297-33c3-41ab-a32f-ab20ef6676c8\") " Nov 03 01:19:48.775952 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:19:48.775927 4939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1e022297-33c3-41ab-a32f-ab20ef6676c8-plugins-config-volume" pod "1e022297-33c3-41ab-a32f-ab20ef6676c8" (UID: "1e022297-33c3-41ab-a32f-ab20ef6676c8"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 01:19:48.775952 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:19:48.775938 4939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1e022297-33c3-41ab-a32f-ab20ef6676c8-routing-overlay-volume" pod "1e022297-33c3-41ab-a32f-ab20ef6676c8" (UID: "1e022297-33c3-41ab-a32f-ab20ef6676c8"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 01:19:48.776180 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:19:48.775960 4939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1e022297-33c3-41ab-a32f-ab20ef6676c8-provider-credentials\" (UniqueName: \"kubernetes.io/projected/1e022297-33c3-41ab-a32f-ab20ef6676c8-provider-credentials\") pod \"1e022297-33c3-41ab-a32f-ab20ef6676c8\" (UID: \"1e022297-33c3-41ab-a32f-ab20ef6676c8\") " Nov 03 01:19:48.776868 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:19:48.776845 4939 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1e022297-33c3-41ab-a32f-ab20ef6676c8-plugins-config-volume\") on node \"ip-10-0-140-107.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:19:48.777069 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:19:48.776870 4939 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/1e022297-33c3-41ab-a32f-ab20ef6676c8-routing-overlay-volume\") on node \"ip-10-0-140-107.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:19:48.778529 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:19:48.778507 4939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1e022297-33c3-41ab-a32f-ab20ef6676c8-provider-credentials" pod "1e022297-33c3-41ab-a32f-ab20ef6676c8" (UID: "1e022297-33c3-41ab-a32f-ab20ef6676c8"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 01:19:48.778610 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:19:48.778546 4939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1e022297-33c3-41ab-a32f-ab20ef6676c8-kube-api-access-j9cwv" pod "1e022297-33c3-41ab-a32f-ab20ef6676c8" (UID: "1e022297-33c3-41ab-a32f-ab20ef6676c8"). InnerVolumeSpecName "kube-api-access-j9cwv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 01:19:48.807285 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:19:48.807256 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:19:48.890962 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:19:48.886399 4939 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/1e022297-33c3-41ab-a32f-ab20ef6676c8-provider-credentials\") on node \"ip-10-0-140-107.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:19:48.890962 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:19:48.886439 4939 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-j9cwv\" (UniqueName: \"kubernetes.io/projected/1e022297-33c3-41ab-a32f-ab20ef6676c8-kube-api-access-j9cwv\") on node \"ip-10-0-140-107.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:19:48.905262 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:19:48.905160 4939 generic.go:356] "Generic (PLEG): container finished" podID="1e022297-33c3-41ab-a32f-ab20ef6676c8" containerID="3a372aaea621d5acf2cc1c5655a8034526c6b483feea35604ee9d711dda91c6e" exitCode=0 Nov 03 01:19:48.905451 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:19:48.905328 4939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-be59df/payload-processing-external-model-e2e-extmod-b-be59df-5885fq5qq" event={"ID":"1e022297-33c3-41ab-a32f-ab20ef6676c8","Type":"ContainerDied","Data":"3a372aaea621d5acf2cc1c5655a8034526c6b483feea35604ee9d711dda91c6e"} Nov 03 01:19:48.905451 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:19:48.905376 4939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-be59df/payload-processing-external-model-e2e-extmod-b-be59df-5885fq5qq" event={"ID":"1e022297-33c3-41ab-a32f-ab20ef6676c8","Type":"ContainerDied","Data":"d20309aec5bb7cdd757d769082e9fa9dcd708ff4b2e4a88bbb7b5f18a1100e99"} Nov 03 01:19:48.905451 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:19:48.905397 4939 scope.go:122] "RemoveContainer" containerID="3a372aaea621d5acf2cc1c5655a8034526c6b483feea35604ee9d711dda91c6e" Nov 03 01:19:48.905679 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:19:48.905529 4939 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-be59df/payload-processing-external-model-e2e-extmod-b-be59df-5885fq5qq" Nov 03 01:19:48.910674 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:19:48.910644 4939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-858d96bddf-mjhfr" event={"ID":"e4404321-b58e-4ddf-9264-cf8790848ac0","Type":"ContainerStarted","Data":"ad49c67948ed9bce54849928fa911fe4c880873a7dfd014016f9c99883e7ad62"} Nov 03 01:19:48.910819 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:19:48.910700 4939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-858d96bddf-mjhfr" event={"ID":"e4404321-b58e-4ddf-9264-cf8790848ac0","Type":"ContainerStarted","Data":"d550a37d7717159eec487ac896f9d79ec8003aff397735e4f7f1eb21d5f3fe3c"} Nov 03 01:19:48.911006 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:19:48.910981 4939 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="models-as-a-service/payload-processing-external-model-858d96bddf-mjhfr" Nov 03 01:19:48.920983 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:19:48.920961 4939 scope.go:122] "RemoveContainer" containerID="3a372aaea621d5acf2cc1c5655a8034526c6b483feea35604ee9d711dda91c6e" Nov 03 01:19:48.921284 ip-10-0-140-107 kubenswrapper[4939]: E1103 01:19:48.921263 4939 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3a372aaea621d5acf2cc1c5655a8034526c6b483feea35604ee9d711dda91c6e\": container with ID starting with 3a372aaea621d5acf2cc1c5655a8034526c6b483feea35604ee9d711dda91c6e not found: ID does not exist" containerID="3a372aaea621d5acf2cc1c5655a8034526c6b483feea35604ee9d711dda91c6e" Nov 03 01:19:48.921369 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:19:48.921295 4939 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3a372aaea621d5acf2cc1c5655a8034526c6b483feea35604ee9d711dda91c6e"} err="failed to get container status \"3a372aaea621d5acf2cc1c5655a8034526c6b483feea35604ee9d711dda91c6e\": rpc error: code = NotFound desc = could not find container \"3a372aaea621d5acf2cc1c5655a8034526c6b483feea35604ee9d711dda91c6e\": container with ID starting with 3a372aaea621d5acf2cc1c5655a8034526c6b483feea35604ee9d711dda91c6e not found: ID does not exist" Nov 03 01:19:48.947532 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:19:48.947481 4939 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="models-as-a-service/payload-processing-external-model-858d96bddf-mjhfr" podStartSLOduration=1.947468936 podStartE2EDuration="1.947468936s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 01:19:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 01:19:48.944721901 +0000 UTC m=+2337.209056847" watchObservedRunningTime="2026-11-03 01:19:48.947468936 +0000 UTC m=+2337.211803916" Nov 03 01:19:48.962777 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:19:48.962744 4939 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-be59df/payload-processing-external-model-e2e-extmod-b-be59df-5885fq5qq"] Nov 03 01:19:48.970396 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:19:48.970368 4939 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-b-be59df/payload-processing-external-model-e2e-extmod-b-be59df-5885fq5qq"] Nov 03 01:19:49.703887 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:19:49.703846 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:19:50.056989 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:19:50.054228 4939 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1e022297-33c3-41ab-a32f-ab20ef6676c8" path="/var/lib/kubelet/pods/1e022297-33c3-41ab-a32f-ab20ef6676c8/volumes" Nov 03 01:19:50.174811 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:19:50.174765 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:19:50.296577 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:19:50.296544 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:19:50.405193 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:19:50.405152 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:19:50.581580 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:19:50.581535 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:19:51.324850 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:19:51.324817 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:19:52.206204 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:19:52.206174 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:19:52.816707 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:19:52.816670 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:19:53.289673 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:19:53.289636 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:19:53.471112 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:19:53.471069 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:19:54.286782 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:19:54.286740 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:19:55.485991 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:19:55.485951 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:19:56.106859 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:19:56.106817 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:19:57.019839 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:19:57.019807 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:19:57.128797 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:19:57.128762 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:19:57.265752 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:19:57.265717 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:19:57.677805 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:19:57.677763 4939 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-08057cfd-vh5dj"] Nov 03 01:19:57.702261 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:19:57.682483 4939 cpu_manager.go:414] "containerMap: removing container" podUID="1e022297-33c3-41ab-a32f-ab20ef6676c8" containerName="payload-processing" Nov 03 01:19:57.702463 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:19:57.702274 4939 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1e022297-33c3-41ab-a32f-ab20ef6676c8" containerName="payload-processing" Nov 03 01:19:57.702874 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:19:57.702841 4939 memory_manager.go:365] "RemoveStaleState removing state" podUID="1e022297-33c3-41ab-a32f-ab20ef6676c8" containerName="payload-processing" Nov 03 01:19:57.710856 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:19:57.710090 4939 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-08057cfd-vh5dj"] Nov 03 01:19:57.710856 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:19:57.710223 4939 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-08057cfd-vh5dj" Nov 03 01:19:57.769404 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:19:57.769358 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t2pjt\" (UniqueName: \"kubernetes.io/projected/e0d5a8a0-e2bd-4339-be4f-8a059b229f52-kube-api-access-t2pjt\") pod \"maas-api-revoke-sub-08057cfd-vh5dj\" (UID: \"e0d5a8a0-e2bd-4339-be4f-8a059b229f52\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-08057cfd-vh5dj" Nov 03 01:19:57.769605 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:19:57.769552 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e0d5a8a0-e2bd-4339-be4f-8a059b229f52-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-08057cfd-vh5dj\" (UID: \"e0d5a8a0-e2bd-4339-be4f-8a059b229f52\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-08057cfd-vh5dj" Nov 03 01:19:57.769930 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:19:57.769882 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e0d5a8a0-e2bd-4339-be4f-8a059b229f52-maas-api-xks-ca\") pod \"maas-api-revoke-sub-08057cfd-vh5dj\" (UID: \"e0d5a8a0-e2bd-4339-be4f-8a059b229f52\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-08057cfd-vh5dj" Nov 03 01:19:57.792595 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:19:57.792541 4939 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 01:19:57.792941 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:19:57.792896 4939 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-6q4xp\"" Nov 03 01:19:57.871631 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:19:57.871592 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e0d5a8a0-e2bd-4339-be4f-8a059b229f52-maas-api-xks-ca\") pod \"maas-api-revoke-sub-08057cfd-vh5dj\" (UID: \"e0d5a8a0-e2bd-4339-be4f-8a059b229f52\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-08057cfd-vh5dj" Nov 03 01:19:57.871835 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:19:57.871797 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t2pjt\" (UniqueName: \"kubernetes.io/projected/e0d5a8a0-e2bd-4339-be4f-8a059b229f52-kube-api-access-t2pjt\") pod \"maas-api-revoke-sub-08057cfd-vh5dj\" (UID: \"e0d5a8a0-e2bd-4339-be4f-8a059b229f52\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-08057cfd-vh5dj" Nov 03 01:19:57.871931 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:19:57.871883 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e0d5a8a0-e2bd-4339-be4f-8a059b229f52-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-08057cfd-vh5dj\" (UID: \"e0d5a8a0-e2bd-4339-be4f-8a059b229f52\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-08057cfd-vh5dj" Nov 03 01:19:57.872638 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:19:57.872603 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e0d5a8a0-e2bd-4339-be4f-8a059b229f52-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-08057cfd-vh5dj\" (UID: \"e0d5a8a0-e2bd-4339-be4f-8a059b229f52\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-08057cfd-vh5dj" Nov 03 01:19:57.876174 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:19:57.875598 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e0d5a8a0-e2bd-4339-be4f-8a059b229f52-maas-api-xks-ca\") pod \"maas-api-revoke-sub-08057cfd-vh5dj\" (UID: \"e0d5a8a0-e2bd-4339-be4f-8a059b229f52\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-08057cfd-vh5dj" Nov 03 01:19:57.876853 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:19:57.876827 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:19:57.880408 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:19:57.880385 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-t2pjt\" (UniqueName: \"kubernetes.io/projected/e0d5a8a0-e2bd-4339-be4f-8a059b229f52-kube-api-access-t2pjt\") pod \"maas-api-revoke-sub-08057cfd-vh5dj\" (UID: \"e0d5a8a0-e2bd-4339-be4f-8a059b229f52\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-08057cfd-vh5dj" Nov 03 01:19:57.998986 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:19:57.998213 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:19:58.032940 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:19:58.032232 4939 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-08057cfd-vh5dj" Nov 03 01:19:58.618385 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:19:58.618351 4939 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["models-as-a-service/payload-processing-external-model-67757c7bbc-7lndz"] Nov 03 01:19:58.624693 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:19:58.624668 4939 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-67757c7bbc-7lndz" Nov 03 01:19:58.628605 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:19:58.628584 4939 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"models-as-a-service\"/\"agc-cred-efc5db69d681\"" Nov 03 01:19:58.631132 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:19:58.631107 4939 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["models-as-a-service/payload-processing-external-model-67757c7bbc-7lndz"] Nov 03 01:19:58.699799 ip-10-0-140-107 kubenswrapper[4939]: W1103 01:19:58.699762 4939 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode0d5a8a0_e2bd_4339_be4f_8a059b229f52.slice/crio-72eb43ad374c9f5b2938efa7853a069666cc9bab26274086159342dbee123f38 WatchSource:0}: Error finding container 72eb43ad374c9f5b2938efa7853a069666cc9bab26274086159342dbee123f38: Status 404 returned error can't find the container with id 72eb43ad374c9f5b2938efa7853a069666cc9bab26274086159342dbee123f38 Nov 03 01:19:58.700220 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:19:58.700201 4939 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-08057cfd-vh5dj"] Nov 03 01:19:58.804821 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:19:58.804775 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/12cc6461-f4d9-4878-b247-2e6ccf21512c-routing-overlay-volume\") pod \"payload-processing-external-model-67757c7bbc-7lndz\" (UID: \"12cc6461-f4d9-4878-b247-2e6ccf21512c\") " pod="models-as-a-service/payload-processing-external-model-67757c7bbc-7lndz" Nov 03 01:19:58.805086 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:19:58.804969 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/12cc6461-f4d9-4878-b247-2e6ccf21512c-plugins-config-volume\") pod \"payload-processing-external-model-67757c7bbc-7lndz\" (UID: \"12cc6461-f4d9-4878-b247-2e6ccf21512c\") " pod="models-as-a-service/payload-processing-external-model-67757c7bbc-7lndz" Nov 03 01:19:58.805154 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:19:58.805102 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dv2fl\" (UniqueName: \"kubernetes.io/projected/12cc6461-f4d9-4878-b247-2e6ccf21512c-kube-api-access-dv2fl\") pod \"payload-processing-external-model-67757c7bbc-7lndz\" (UID: \"12cc6461-f4d9-4878-b247-2e6ccf21512c\") " pod="models-as-a-service/payload-processing-external-model-67757c7bbc-7lndz" Nov 03 01:19:58.805212 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:19:58.805169 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/12cc6461-f4d9-4878-b247-2e6ccf21512c-provider-credentials\") pod \"payload-processing-external-model-67757c7bbc-7lndz\" (UID: \"12cc6461-f4d9-4878-b247-2e6ccf21512c\") " pod="models-as-a-service/payload-processing-external-model-67757c7bbc-7lndz" Nov 03 01:19:58.873452 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:19:58.873370 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:19:58.915966 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:19:58.910940 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/12cc6461-f4d9-4878-b247-2e6ccf21512c-routing-overlay-volume\") pod \"payload-processing-external-model-67757c7bbc-7lndz\" (UID: \"12cc6461-f4d9-4878-b247-2e6ccf21512c\") " pod="models-as-a-service/payload-processing-external-model-67757c7bbc-7lndz" Nov 03 01:19:58.915966 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:19:58.911074 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/12cc6461-f4d9-4878-b247-2e6ccf21512c-plugins-config-volume\") pod \"payload-processing-external-model-67757c7bbc-7lndz\" (UID: \"12cc6461-f4d9-4878-b247-2e6ccf21512c\") " pod="models-as-a-service/payload-processing-external-model-67757c7bbc-7lndz" Nov 03 01:19:58.915966 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:19:58.911189 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dv2fl\" (UniqueName: \"kubernetes.io/projected/12cc6461-f4d9-4878-b247-2e6ccf21512c-kube-api-access-dv2fl\") pod \"payload-processing-external-model-67757c7bbc-7lndz\" (UID: \"12cc6461-f4d9-4878-b247-2e6ccf21512c\") " pod="models-as-a-service/payload-processing-external-model-67757c7bbc-7lndz" Nov 03 01:19:58.915966 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:19:58.911252 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/12cc6461-f4d9-4878-b247-2e6ccf21512c-provider-credentials\") pod \"payload-processing-external-model-67757c7bbc-7lndz\" (UID: \"12cc6461-f4d9-4878-b247-2e6ccf21512c\") " pod="models-as-a-service/payload-processing-external-model-67757c7bbc-7lndz" Nov 03 01:19:58.916342 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:19:58.916030 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/12cc6461-f4d9-4878-b247-2e6ccf21512c-plugins-config-volume\") pod \"payload-processing-external-model-67757c7bbc-7lndz\" (UID: \"12cc6461-f4d9-4878-b247-2e6ccf21512c\") " pod="models-as-a-service/payload-processing-external-model-67757c7bbc-7lndz" Nov 03 01:19:58.950049 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:19:58.932509 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/12cc6461-f4d9-4878-b247-2e6ccf21512c-routing-overlay-volume\") pod \"payload-processing-external-model-67757c7bbc-7lndz\" (UID: \"12cc6461-f4d9-4878-b247-2e6ccf21512c\") " pod="models-as-a-service/payload-processing-external-model-67757c7bbc-7lndz" Nov 03 01:19:58.978335 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:19:58.978299 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/12cc6461-f4d9-4878-b247-2e6ccf21512c-provider-credentials\") pod \"payload-processing-external-model-67757c7bbc-7lndz\" (UID: \"12cc6461-f4d9-4878-b247-2e6ccf21512c\") " pod="models-as-a-service/payload-processing-external-model-67757c7bbc-7lndz" Nov 03 01:19:58.979165 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:19:58.979135 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dv2fl\" (UniqueName: \"kubernetes.io/projected/12cc6461-f4d9-4878-b247-2e6ccf21512c-kube-api-access-dv2fl\") pod \"payload-processing-external-model-67757c7bbc-7lndz\" (UID: \"12cc6461-f4d9-4878-b247-2e6ccf21512c\") " pod="models-as-a-service/payload-processing-external-model-67757c7bbc-7lndz" Nov 03 01:19:59.249650 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:19:59.249620 4939 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-67757c7bbc-7lndz" Nov 03 01:19:59.296377 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:19:59.296341 4939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-08057cfd-vh5dj" event={"ID":"e0d5a8a0-e2bd-4339-be4f-8a059b229f52","Type":"ContainerStarted","Data":"1c9d456f802cb91ef9c5559b77cb367b73208196c3e7eac377754b747aa15619"} Nov 03 01:19:59.296549 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:19:59.296400 4939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-08057cfd-vh5dj" event={"ID":"e0d5a8a0-e2bd-4339-be4f-8a059b229f52","Type":"ContainerStarted","Data":"72eb43ad374c9f5b2938efa7853a069666cc9bab26274086159342dbee123f38"} Nov 03 01:19:59.323398 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:19:59.323346 4939 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-08057cfd-vh5dj" podStartSLOduration=2.323331601 podStartE2EDuration="2.323331601s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 01:19:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 01:19:59.318495781 +0000 UTC m=+2347.582830725" watchObservedRunningTime="2026-11-03 01:19:59.323331601 +0000 UTC m=+2347.587666578" Nov 03 01:19:59.536693 ip-10-0-140-107 kubenswrapper[4939]: W1103 01:19:59.536659 4939 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod12cc6461_f4d9_4878_b247_2e6ccf21512c.slice/crio-514f09eea528894019b72f9c95237cd7bfa467bb7f1f47d641b5eb46343cf837 WatchSource:0}: Error finding container 514f09eea528894019b72f9c95237cd7bfa467bb7f1f47d641b5eb46343cf837: Status 404 returned error can't find the container with id 514f09eea528894019b72f9c95237cd7bfa467bb7f1f47d641b5eb46343cf837 Nov 03 01:19:59.537036 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:19:59.537012 4939 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["models-as-a-service/payload-processing-external-model-67757c7bbc-7lndz"] Nov 03 01:19:59.606103 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:19:59.606079 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:20:00.081382 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:20:00.081354 4939 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="models-as-a-service/payload-processing-external-model-858d96bddf-mjhfr" Nov 03 01:20:00.306127 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:20:00.306025 4939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-67757c7bbc-7lndz" event={"ID":"12cc6461-f4d9-4878-b247-2e6ccf21512c","Type":"ContainerStarted","Data":"e6de5b172d56fcec7d3fb67314719b9df19bef6a14a8712b4139422406fa4f89"} Nov 03 01:20:00.306127 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:20:00.306100 4939 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="models-as-a-service/payload-processing-external-model-67757c7bbc-7lndz" Nov 03 01:20:00.306127 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:20:00.306122 4939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-67757c7bbc-7lndz" event={"ID":"12cc6461-f4d9-4878-b247-2e6ccf21512c","Type":"ContainerStarted","Data":"514f09eea528894019b72f9c95237cd7bfa467bb7f1f47d641b5eb46343cf837"} Nov 03 01:20:00.307865 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:20:00.307832 4939 generic.go:356] "Generic (PLEG): container finished" podID="e0d5a8a0-e2bd-4339-be4f-8a059b229f52" containerID="1c9d456f802cb91ef9c5559b77cb367b73208196c3e7eac377754b747aa15619" exitCode=0 Nov 03 01:20:00.308014 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:20:00.307924 4939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-08057cfd-vh5dj" event={"ID":"e0d5a8a0-e2bd-4339-be4f-8a059b229f52","Type":"ContainerDied","Data":"1c9d456f802cb91ef9c5559b77cb367b73208196c3e7eac377754b747aa15619"} Nov 03 01:20:00.338359 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:20:00.338307 4939 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="models-as-a-service/payload-processing-external-model-67757c7bbc-7lndz" podStartSLOduration=2.338293376 podStartE2EDuration="2.338293376s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 01:19:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 01:20:00.332170725 +0000 UTC m=+2348.596505669" watchObservedRunningTime="2026-11-03 01:20:00.338293376 +0000 UTC m=+2348.602628344" Nov 03 01:20:00.359243 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:20:00.359213 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:20:01.186248 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:20:01.186210 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:20:01.517378 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:20:01.517349 4939 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-08057cfd-vh5dj" Nov 03 01:20:01.569197 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:20:01.569166 4939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e0d5a8a0-e2bd-4339-be4f-8a059b229f52-kube-api-access-t2pjt\" (UniqueName: \"kubernetes.io/projected/e0d5a8a0-e2bd-4339-be4f-8a059b229f52-kube-api-access-t2pjt\") pod \"e0d5a8a0-e2bd-4339-be4f-8a059b229f52\" (UID: \"e0d5a8a0-e2bd-4339-be4f-8a059b229f52\") " Nov 03 01:20:01.569394 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:20:01.569231 4939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e0d5a8a0-e2bd-4339-be4f-8a059b229f52-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e0d5a8a0-e2bd-4339-be4f-8a059b229f52-maas-api-xks-ca\") pod \"e0d5a8a0-e2bd-4339-be4f-8a059b229f52\" (UID: \"e0d5a8a0-e2bd-4339-be4f-8a059b229f52\") " Nov 03 01:20:01.569394 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:20:01.569290 4939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e0d5a8a0-e2bd-4339-be4f-8a059b229f52-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e0d5a8a0-e2bd-4339-be4f-8a059b229f52-maas-api-openshift-ca\") pod \"e0d5a8a0-e2bd-4339-be4f-8a059b229f52\" (UID: \"e0d5a8a0-e2bd-4339-be4f-8a059b229f52\") " Nov 03 01:20:01.569787 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:20:01.569751 4939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e0d5a8a0-e2bd-4339-be4f-8a059b229f52-maas-api-openshift-ca" pod "e0d5a8a0-e2bd-4339-be4f-8a059b229f52" (UID: "e0d5a8a0-e2bd-4339-be4f-8a059b229f52"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 01:20:01.570451 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:20:01.570420 4939 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e0d5a8a0-e2bd-4339-be4f-8a059b229f52-maas-api-openshift-ca\") on node \"ip-10-0-140-107.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:20:01.571849 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:20:01.571826 4939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e0d5a8a0-e2bd-4339-be4f-8a059b229f52-kube-api-access-t2pjt" pod "e0d5a8a0-e2bd-4339-be4f-8a059b229f52" (UID: "e0d5a8a0-e2bd-4339-be4f-8a059b229f52"). InnerVolumeSpecName "kube-api-access-t2pjt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 01:20:01.572060 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:20:01.572034 4939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e0d5a8a0-e2bd-4339-be4f-8a059b229f52-maas-api-xks-ca" pod "e0d5a8a0-e2bd-4339-be4f-8a059b229f52" (UID: "e0d5a8a0-e2bd-4339-be4f-8a059b229f52"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 01:20:01.672394 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:20:01.672355 4939 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-t2pjt\" (UniqueName: \"kubernetes.io/projected/e0d5a8a0-e2bd-4339-be4f-8a059b229f52-kube-api-access-t2pjt\") on node \"ip-10-0-140-107.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:20:01.672394 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:20:01.672390 4939 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e0d5a8a0-e2bd-4339-be4f-8a059b229f52-maas-api-xks-ca\") on node \"ip-10-0-140-107.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:20:01.962079 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:20:01.962046 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:20:02.327360 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:20:02.327279 4939 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-08057cfd-vh5dj" Nov 03 01:20:02.327360 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:20:02.327316 4939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-08057cfd-vh5dj" event={"ID":"e0d5a8a0-e2bd-4339-be4f-8a059b229f52","Type":"ContainerDied","Data":"72eb43ad374c9f5b2938efa7853a069666cc9bab26274086159342dbee123f38"} Nov 03 01:20:02.327548 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:20:02.327374 4939 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="72eb43ad374c9f5b2938efa7853a069666cc9bab26274086159342dbee123f38" Nov 03 01:20:02.811900 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:20:02.810763 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:20:03.580636 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:20:03.580598 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:20:04.266099 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:20:04.266049 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:20:04.892971 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:20:04.892928 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:20:05.598831 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:20:05.598775 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:20:06.375608 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:20:06.375576 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:20:06.998650 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:20:06.998601 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:20:07.857704 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:20:07.857672 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:20:08.793795 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:20:08.792764 4939 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-821f5a/payload-processing-external-model-e2e-extmod-a-821f5a-94cbf4lx9"] Nov 03 01:20:08.796589 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:20:08.796562 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:20:08.909888 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:20:08.909856 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:20:09.397923 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:20:09.397878 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:20:09.888937 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:20:09.888892 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:20:10.504759 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:20:10.504725 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:20:11.187661 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:20:11.187624 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:20:11.321773 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:20:11.321742 4939 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="models-as-a-service/payload-processing-external-model-67757c7bbc-7lndz" Nov 03 01:20:11.372733 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:20:11.372699 4939 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["models-as-a-service/payload-processing-external-model-858d96bddf-mjhfr"] Nov 03 01:20:11.833617 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:20:11.833587 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:20:12.061627 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:20:12.061593 4939 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-4pw7v"] Nov 03 01:20:12.584402 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:20:12.584354 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:20:13.799082 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:20:13.799012 4939 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-a-821f5a/payload-processing-external-model-e2e-extmod-a-821f5a-94cbf4lx9" podUID="97ca98ed-903e-422a-bffd-ad208fc2a3aa" containerName="payload-processing" containerID="cri-o://aa31d2f76160892a6d8d3557762986d6a3387098f57d33cbacd418afcc516487" gracePeriod=25 Nov 03 01:20:14.108811 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:20:14.101753 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:20:14.383770 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:20:14.383698 4939 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-821f5a/payload-processing-external-model-e2e-extmod-a-821f5a-94cbf4lx9" Nov 03 01:20:14.580826 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:20:14.580781 4939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/97ca98ed-903e-422a-bffd-ad208fc2a3aa-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/97ca98ed-903e-422a-bffd-ad208fc2a3aa-plugins-config-volume\") pod \"97ca98ed-903e-422a-bffd-ad208fc2a3aa\" (UID: \"97ca98ed-903e-422a-bffd-ad208fc2a3aa\") " Nov 03 01:20:14.581056 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:20:14.580900 4939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/97ca98ed-903e-422a-bffd-ad208fc2a3aa-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/97ca98ed-903e-422a-bffd-ad208fc2a3aa-routing-overlay-volume\") pod \"97ca98ed-903e-422a-bffd-ad208fc2a3aa\" (UID: \"97ca98ed-903e-422a-bffd-ad208fc2a3aa\") " Nov 03 01:20:14.581056 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:20:14.581002 4939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/97ca98ed-903e-422a-bffd-ad208fc2a3aa-provider-credentials\" (UniqueName: \"kubernetes.io/projected/97ca98ed-903e-422a-bffd-ad208fc2a3aa-provider-credentials\") pod \"97ca98ed-903e-422a-bffd-ad208fc2a3aa\" (UID: \"97ca98ed-903e-422a-bffd-ad208fc2a3aa\") " Nov 03 01:20:14.581293 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:20:14.581268 4939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/97ca98ed-903e-422a-bffd-ad208fc2a3aa-routing-overlay-volume" pod "97ca98ed-903e-422a-bffd-ad208fc2a3aa" (UID: "97ca98ed-903e-422a-bffd-ad208fc2a3aa"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 01:20:14.581369 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:20:14.581286 4939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/97ca98ed-903e-422a-bffd-ad208fc2a3aa-kube-api-access-hxpv4\" (UniqueName: \"kubernetes.io/projected/97ca98ed-903e-422a-bffd-ad208fc2a3aa-kube-api-access-hxpv4\") pod \"97ca98ed-903e-422a-bffd-ad208fc2a3aa\" (UID: \"97ca98ed-903e-422a-bffd-ad208fc2a3aa\") " Nov 03 01:20:14.581369 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:20:14.581310 4939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/97ca98ed-903e-422a-bffd-ad208fc2a3aa-plugins-config-volume" pod "97ca98ed-903e-422a-bffd-ad208fc2a3aa" (UID: "97ca98ed-903e-422a-bffd-ad208fc2a3aa"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 01:20:14.582429 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:20:14.582397 4939 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/97ca98ed-903e-422a-bffd-ad208fc2a3aa-plugins-config-volume\") on node \"ip-10-0-140-107.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:20:14.582559 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:20:14.582440 4939 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/97ca98ed-903e-422a-bffd-ad208fc2a3aa-routing-overlay-volume\") on node \"ip-10-0-140-107.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:20:14.587797 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:20:14.587773 4939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/97ca98ed-903e-422a-bffd-ad208fc2a3aa-kube-api-access-hxpv4" pod "97ca98ed-903e-422a-bffd-ad208fc2a3aa" (UID: "97ca98ed-903e-422a-bffd-ad208fc2a3aa"). InnerVolumeSpecName "kube-api-access-hxpv4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 01:20:14.587893 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:20:14.587815 4939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/97ca98ed-903e-422a-bffd-ad208fc2a3aa-provider-credentials" pod "97ca98ed-903e-422a-bffd-ad208fc2a3aa" (UID: "97ca98ed-903e-422a-bffd-ad208fc2a3aa"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 01:20:14.650646 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:20:14.650552 4939 generic.go:356] "Generic (PLEG): container finished" podID="97ca98ed-903e-422a-bffd-ad208fc2a3aa" containerID="aa31d2f76160892a6d8d3557762986d6a3387098f57d33cbacd418afcc516487" exitCode=0 Nov 03 01:20:14.650646 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:20:14.650619 4939 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-821f5a/payload-processing-external-model-e2e-extmod-a-821f5a-94cbf4lx9" Nov 03 01:20:14.650887 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:20:14.650643 4939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-821f5a/payload-processing-external-model-e2e-extmod-a-821f5a-94cbf4lx9" event={"ID":"97ca98ed-903e-422a-bffd-ad208fc2a3aa","Type":"ContainerDied","Data":"aa31d2f76160892a6d8d3557762986d6a3387098f57d33cbacd418afcc516487"} Nov 03 01:20:14.650887 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:20:14.650725 4939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-821f5a/payload-processing-external-model-e2e-extmod-a-821f5a-94cbf4lx9" event={"ID":"97ca98ed-903e-422a-bffd-ad208fc2a3aa","Type":"ContainerDied","Data":"8236af36cdb94c322738badf5d75c436f2b12bf93200d0a410be1a89027bd9f4"} Nov 03 01:20:14.650887 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:20:14.650759 4939 scope.go:122] "RemoveContainer" containerID="aa31d2f76160892a6d8d3557762986d6a3387098f57d33cbacd418afcc516487" Nov 03 01:20:14.680805 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:20:14.679084 4939 scope.go:122] "RemoveContainer" containerID="aa31d2f76160892a6d8d3557762986d6a3387098f57d33cbacd418afcc516487" Nov 03 01:20:14.680805 ip-10-0-140-107 kubenswrapper[4939]: E1103 01:20:14.680602 4939 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"aa31d2f76160892a6d8d3557762986d6a3387098f57d33cbacd418afcc516487\": container with ID starting with aa31d2f76160892a6d8d3557762986d6a3387098f57d33cbacd418afcc516487 not found: ID does not exist" containerID="aa31d2f76160892a6d8d3557762986d6a3387098f57d33cbacd418afcc516487" Nov 03 01:20:14.680805 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:20:14.680755 4939 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"aa31d2f76160892a6d8d3557762986d6a3387098f57d33cbacd418afcc516487"} err="failed to get container status \"aa31d2f76160892a6d8d3557762986d6a3387098f57d33cbacd418afcc516487\": rpc error: code = NotFound desc = could not find container \"aa31d2f76160892a6d8d3557762986d6a3387098f57d33cbacd418afcc516487\": container with ID starting with aa31d2f76160892a6d8d3557762986d6a3387098f57d33cbacd418afcc516487 not found: ID does not exist" Nov 03 01:20:14.685777 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:20:14.685742 4939 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/97ca98ed-903e-422a-bffd-ad208fc2a3aa-provider-credentials\") on node \"ip-10-0-140-107.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:20:14.685976 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:20:14.685779 4939 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hxpv4\" (UniqueName: \"kubernetes.io/projected/97ca98ed-903e-422a-bffd-ad208fc2a3aa-kube-api-access-hxpv4\") on node \"ip-10-0-140-107.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:20:14.688112 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:20:14.688081 4939 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-821f5a/payload-processing-external-model-e2e-extmod-a-821f5a-94cbf4lx9"] Nov 03 01:20:14.692886 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:20:14.692858 4939 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-a-821f5a/payload-processing-external-model-e2e-extmod-a-821f5a-94cbf4lx9"] Nov 03 01:20:14.994865 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:20:14.994777 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:20:15.910931 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:20:15.905476 4939 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="97ca98ed-903e-422a-bffd-ad208fc2a3aa" path="/var/lib/kubelet/pods/97ca98ed-903e-422a-bffd-ad208fc2a3aa/volumes" Nov 03 01:20:16.006937 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:20:16.001385 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:20:16.373932 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:20:16.373636 4939 kuberuntime_container.go:898] "Killing container with a grace period" pod="models-as-a-service/payload-processing-external-model-858d96bddf-mjhfr" podUID="e4404321-b58e-4ddf-9264-cf8790848ac0" containerName="payload-processing" containerID="cri-o://ad49c67948ed9bce54849928fa911fe4c880873a7dfd014016f9c99883e7ad62" gracePeriod=25 Nov 03 01:20:16.682945 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:20:16.680421 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:20:16.841761 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:20:16.841704 4939 generic.go:356] "Generic (PLEG): container finished" podID="e4404321-b58e-4ddf-9264-cf8790848ac0" containerID="ad49c67948ed9bce54849928fa911fe4c880873a7dfd014016f9c99883e7ad62" exitCode=0 Nov 03 01:20:16.842115 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:20:16.842072 4939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-858d96bddf-mjhfr" event={"ID":"e4404321-b58e-4ddf-9264-cf8790848ac0","Type":"ContainerDied","Data":"ad49c67948ed9bce54849928fa911fe4c880873a7dfd014016f9c99883e7ad62"} Nov 03 01:20:16.870671 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:20:16.870594 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:20:16.922970 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:20:16.922945 4939 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-858d96bddf-mjhfr" Nov 03 01:20:16.995268 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:20:16.995212 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:20:17.009313 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:20:17.009281 4939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e4404321-b58e-4ddf-9264-cf8790848ac0-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/e4404321-b58e-4ddf-9264-cf8790848ac0-routing-overlay-volume\") pod \"e4404321-b58e-4ddf-9264-cf8790848ac0\" (UID: \"e4404321-b58e-4ddf-9264-cf8790848ac0\") " Nov 03 01:20:17.009831 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:20:17.009517 4939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e4404321-b58e-4ddf-9264-cf8790848ac0-kube-api-access-k5skd\" (UniqueName: \"kubernetes.io/projected/e4404321-b58e-4ddf-9264-cf8790848ac0-kube-api-access-k5skd\") pod \"e4404321-b58e-4ddf-9264-cf8790848ac0\" (UID: \"e4404321-b58e-4ddf-9264-cf8790848ac0\") " Nov 03 01:20:17.009831 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:20:17.009638 4939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e4404321-b58e-4ddf-9264-cf8790848ac0-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e4404321-b58e-4ddf-9264-cf8790848ac0-plugins-config-volume\") pod \"e4404321-b58e-4ddf-9264-cf8790848ac0\" (UID: \"e4404321-b58e-4ddf-9264-cf8790848ac0\") " Nov 03 01:20:17.009831 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:20:17.009727 4939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e4404321-b58e-4ddf-9264-cf8790848ac0-routing-overlay-volume" pod "e4404321-b58e-4ddf-9264-cf8790848ac0" (UID: "e4404321-b58e-4ddf-9264-cf8790848ac0"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 01:20:17.009831 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:20:17.009803 4939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e4404321-b58e-4ddf-9264-cf8790848ac0-provider-credentials\" (UniqueName: \"kubernetes.io/projected/e4404321-b58e-4ddf-9264-cf8790848ac0-provider-credentials\") pod \"e4404321-b58e-4ddf-9264-cf8790848ac0\" (UID: \"e4404321-b58e-4ddf-9264-cf8790848ac0\") " Nov 03 01:20:17.010430 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:20:17.010399 4939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e4404321-b58e-4ddf-9264-cf8790848ac0-plugins-config-volume" pod "e4404321-b58e-4ddf-9264-cf8790848ac0" (UID: "e4404321-b58e-4ddf-9264-cf8790848ac0"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 01:20:17.010715 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:20:17.010693 4939 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/e4404321-b58e-4ddf-9264-cf8790848ac0-routing-overlay-volume\") on node \"ip-10-0-140-107.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:20:17.010840 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:20:17.010723 4939 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e4404321-b58e-4ddf-9264-cf8790848ac0-plugins-config-volume\") on node \"ip-10-0-140-107.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:20:17.012408 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:20:17.012377 4939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e4404321-b58e-4ddf-9264-cf8790848ac0-provider-credentials" pod "e4404321-b58e-4ddf-9264-cf8790848ac0" (UID: "e4404321-b58e-4ddf-9264-cf8790848ac0"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 01:20:17.012567 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:20:17.012547 4939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e4404321-b58e-4ddf-9264-cf8790848ac0-kube-api-access-k5skd" pod "e4404321-b58e-4ddf-9264-cf8790848ac0" (UID: "e4404321-b58e-4ddf-9264-cf8790848ac0"). InnerVolumeSpecName "kube-api-access-k5skd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 01:20:17.123950 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:20:17.122563 4939 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/e4404321-b58e-4ddf-9264-cf8790848ac0-provider-credentials\") on node \"ip-10-0-140-107.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:20:17.123950 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:20:17.122598 4939 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k5skd\" (UniqueName: \"kubernetes.io/projected/e4404321-b58e-4ddf-9264-cf8790848ac0-kube-api-access-k5skd\") on node \"ip-10-0-140-107.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:20:17.725136 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:20:17.725104 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:20:17.853043 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:20:17.853014 4939 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-858d96bddf-mjhfr" Nov 03 01:20:17.853233 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:20:17.853018 4939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-858d96bddf-mjhfr" event={"ID":"e4404321-b58e-4ddf-9264-cf8790848ac0","Type":"ContainerDied","Data":"d550a37d7717159eec487ac896f9d79ec8003aff397735e4f7f1eb21d5f3fe3c"} Nov 03 01:20:17.853233 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:20:17.853149 4939 scope.go:122] "RemoveContainer" containerID="ad49c67948ed9bce54849928fa911fe4c880873a7dfd014016f9c99883e7ad62" Nov 03 01:20:17.889993 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:20:17.889958 4939 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["models-as-a-service/payload-processing-external-model-858d96bddf-mjhfr"] Nov 03 01:20:17.894618 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:20:17.894592 4939 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["models-as-a-service/payload-processing-external-model-858d96bddf-mjhfr"] Nov 03 01:20:18.033447 ip-10-0-140-107 kubenswrapper[4939]: E1103 01:20:18.033367 4939 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-sthjh" containerName="extractor" Nov 03 01:20:18.387761 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:20:18.387728 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:20:19.183169 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:20:19.183133 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:20:19.389405 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:20:19.389370 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:20:19.538988 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:20:19.533276 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:20:19.556938 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:20:19.555291 4939 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-a-7c759a/payload-processing-external-model-e2e-extmod-a-7c759a-54dbrs2dj"] Nov 03 01:20:19.556938 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:20:19.556702 4939 cpu_manager.go:414] "containerMap: removing container" podUID="97ca98ed-903e-422a-bffd-ad208fc2a3aa" containerName="payload-processing" Nov 03 01:20:19.556938 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:20:19.556725 4939 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="97ca98ed-903e-422a-bffd-ad208fc2a3aa" containerName="payload-processing" Nov 03 01:20:19.556938 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:20:19.556772 4939 cpu_manager.go:414] "containerMap: removing container" podUID="e4404321-b58e-4ddf-9264-cf8790848ac0" containerName="payload-processing" Nov 03 01:20:19.556938 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:20:19.556781 4939 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e4404321-b58e-4ddf-9264-cf8790848ac0" containerName="payload-processing" Nov 03 01:20:19.556938 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:20:19.556820 4939 cpu_manager.go:414] "containerMap: removing container" podUID="e0d5a8a0-e2bd-4339-be4f-8a059b229f52" containerName="revoke-keys" Nov 03 01:20:19.556938 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:20:19.556828 4939 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e0d5a8a0-e2bd-4339-be4f-8a059b229f52" containerName="revoke-keys" Nov 03 01:20:19.557455 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:20:19.557058 4939 memory_manager.go:365] "RemoveStaleState removing state" podUID="e4404321-b58e-4ddf-9264-cf8790848ac0" containerName="payload-processing" Nov 03 01:20:19.557455 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:20:19.557083 4939 memory_manager.go:365] "RemoveStaleState removing state" podUID="e0d5a8a0-e2bd-4339-be4f-8a059b229f52" containerName="revoke-keys" Nov 03 01:20:19.557455 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:20:19.557104 4939 memory_manager.go:365] "RemoveStaleState removing state" podUID="97ca98ed-903e-422a-bffd-ad208fc2a3aa" containerName="payload-processing" Nov 03 01:20:19.616998 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:20:19.614386 4939 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-7c759a/payload-processing-external-model-e2e-extmod-a-7c759a-54dbrs2dj"] Nov 03 01:20:19.616998 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:20:19.614517 4939 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-7c759a/payload-processing-external-model-e2e-extmod-a-7c759a-54dbrs2dj" Nov 03 01:20:19.626017 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:20:19.625941 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dzwlz\" (UniqueName: \"kubernetes.io/projected/0fae2685-234b-43e0-aae4-47f912d3bb5b-kube-api-access-dzwlz\") pod \"payload-processing-external-model-e2e-extmod-a-7c759a-54dbrs2dj\" (UID: \"0fae2685-234b-43e0-aae4-47f912d3bb5b\") " pod="ai-tenant-e2e-extmod-a-7c759a/payload-processing-external-model-e2e-extmod-a-7c759a-54dbrs2dj" Nov 03 01:20:19.626287 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:20:19.626252 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/0fae2685-234b-43e0-aae4-47f912d3bb5b-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-7c759a-54dbrs2dj\" (UID: \"0fae2685-234b-43e0-aae4-47f912d3bb5b\") " pod="ai-tenant-e2e-extmod-a-7c759a/payload-processing-external-model-e2e-extmod-a-7c759a-54dbrs2dj" Nov 03 01:20:19.626438 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:20:19.626346 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0fae2685-234b-43e0-aae4-47f912d3bb5b-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-7c759a-54dbrs2dj\" (UID: \"0fae2685-234b-43e0-aae4-47f912d3bb5b\") " pod="ai-tenant-e2e-extmod-a-7c759a/payload-processing-external-model-e2e-extmod-a-7c759a-54dbrs2dj" Nov 03 01:20:19.626438 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:20:19.626410 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/0fae2685-234b-43e0-aae4-47f912d3bb5b-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-7c759a-54dbrs2dj\" (UID: \"0fae2685-234b-43e0-aae4-47f912d3bb5b\") " pod="ai-tenant-e2e-extmod-a-7c759a/payload-processing-external-model-e2e-extmod-a-7c759a-54dbrs2dj" Nov 03 01:20:19.728516 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:20:19.728477 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dzwlz\" (UniqueName: \"kubernetes.io/projected/0fae2685-234b-43e0-aae4-47f912d3bb5b-kube-api-access-dzwlz\") pod \"payload-processing-external-model-e2e-extmod-a-7c759a-54dbrs2dj\" (UID: \"0fae2685-234b-43e0-aae4-47f912d3bb5b\") " pod="ai-tenant-e2e-extmod-a-7c759a/payload-processing-external-model-e2e-extmod-a-7c759a-54dbrs2dj" Nov 03 01:20:19.728717 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:20:19.728680 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/0fae2685-234b-43e0-aae4-47f912d3bb5b-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-7c759a-54dbrs2dj\" (UID: \"0fae2685-234b-43e0-aae4-47f912d3bb5b\") " pod="ai-tenant-e2e-extmod-a-7c759a/payload-processing-external-model-e2e-extmod-a-7c759a-54dbrs2dj" Nov 03 01:20:19.728783 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:20:19.728730 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0fae2685-234b-43e0-aae4-47f912d3bb5b-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-7c759a-54dbrs2dj\" (UID: \"0fae2685-234b-43e0-aae4-47f912d3bb5b\") " pod="ai-tenant-e2e-extmod-a-7c759a/payload-processing-external-model-e2e-extmod-a-7c759a-54dbrs2dj" Nov 03 01:20:19.728895 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:20:19.728869 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/0fae2685-234b-43e0-aae4-47f912d3bb5b-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-7c759a-54dbrs2dj\" (UID: \"0fae2685-234b-43e0-aae4-47f912d3bb5b\") " pod="ai-tenant-e2e-extmod-a-7c759a/payload-processing-external-model-e2e-extmod-a-7c759a-54dbrs2dj" Nov 03 01:20:19.729450 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:20:19.729425 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/0fae2685-234b-43e0-aae4-47f912d3bb5b-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-7c759a-54dbrs2dj\" (UID: \"0fae2685-234b-43e0-aae4-47f912d3bb5b\") " pod="ai-tenant-e2e-extmod-a-7c759a/payload-processing-external-model-e2e-extmod-a-7c759a-54dbrs2dj" Nov 03 01:20:19.729550 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:20:19.729472 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0fae2685-234b-43e0-aae4-47f912d3bb5b-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-7c759a-54dbrs2dj\" (UID: \"0fae2685-234b-43e0-aae4-47f912d3bb5b\") " pod="ai-tenant-e2e-extmod-a-7c759a/payload-processing-external-model-e2e-extmod-a-7c759a-54dbrs2dj" Nov 03 01:20:19.731304 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:20:19.731276 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/0fae2685-234b-43e0-aae4-47f912d3bb5b-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-7c759a-54dbrs2dj\" (UID: \"0fae2685-234b-43e0-aae4-47f912d3bb5b\") " pod="ai-tenant-e2e-extmod-a-7c759a/payload-processing-external-model-e2e-extmod-a-7c759a-54dbrs2dj" Nov 03 01:20:19.736618 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:20:19.736597 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dzwlz\" (UniqueName: \"kubernetes.io/projected/0fae2685-234b-43e0-aae4-47f912d3bb5b-kube-api-access-dzwlz\") pod \"payload-processing-external-model-e2e-extmod-a-7c759a-54dbrs2dj\" (UID: \"0fae2685-234b-43e0-aae4-47f912d3bb5b\") " pod="ai-tenant-e2e-extmod-a-7c759a/payload-processing-external-model-e2e-extmod-a-7c759a-54dbrs2dj" Nov 03 01:20:19.873352 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:20:19.873315 4939 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e4404321-b58e-4ddf-9264-cf8790848ac0" path="/var/lib/kubelet/pods/e4404321-b58e-4ddf-9264-cf8790848ac0/volumes" Nov 03 01:20:19.995738 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:20:19.995701 4939 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-7c759a/payload-processing-external-model-e2e-extmod-a-7c759a-54dbrs2dj" Nov 03 01:20:20.187984 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:20:20.187882 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:20:20.325250 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:20:20.323729 4939 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-7c759a/payload-processing-external-model-e2e-extmod-a-7c759a-54dbrs2dj"] Nov 03 01:20:20.419384 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:20:20.416576 4939 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-7c759a/payload-processing-external-model-e2e-extmod-a-7c759a-54dbrs2dj"] Nov 03 01:20:20.421216 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:20:20.421190 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:20:20.427799 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:20:20.427023 4939 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-7c759a/payload-processing-external-model-e2e-extmod-a-7c759a-5bf4hhjsl"] Nov 03 01:20:20.535325 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:20:20.535282 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:20:21.011932 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:20:21.006096 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:20:21.016351 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:20:21.014745 4939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-7c759a/payload-processing-external-model-e2e-extmod-a-7c759a-54dbrs2dj" event={"ID":"0fae2685-234b-43e0-aae4-47f912d3bb5b","Type":"ContainerStarted","Data":"43435815ad147ad5b94bfea549aeac051ccc2effefa6fe25792e12f48a091daa"} Nov 03 01:20:21.016351 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:20:21.014815 4939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-7c759a/payload-processing-external-model-e2e-extmod-a-7c759a-54dbrs2dj" event={"ID":"0fae2685-234b-43e0-aae4-47f912d3bb5b","Type":"ContainerStarted","Data":"8742cce92043eecd61a8be191f0e92be419c01d99ccaf300dc4f86d24a8a8b46"} Nov 03 01:20:21.016351 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:20:21.015021 4939 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-a-7c759a/payload-processing-external-model-e2e-extmod-a-7c759a-54dbrs2dj" Nov 03 01:20:21.138466 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:20:21.138396 4939 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-a-7c759a/payload-processing-external-model-e2e-extmod-a-7c759a-54dbrs2dj" podStartSLOduration=2.138371678 podStartE2EDuration="2.138371678s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 01:20:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 01:20:21.104325376 +0000 UTC m=+2369.368660358" watchObservedRunningTime="2026-11-03 01:20:21.138371678 +0000 UTC m=+2369.402706623" Nov 03 01:20:21.508547 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:20:21.508518 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:20:22.317271 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:20:22.317231 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:20:22.818931 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:20:22.818668 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:20:22.989522 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:20:22.989488 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:20:23.170104 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:20:23.170069 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:20:23.557202 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:20:23.557116 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:20:23.666532 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:20:23.666497 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:20:24.295991 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:20:24.295952 4939 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["models-as-a-service/payload-processing-external-model-5774fbf4b9-pl5dw"] Nov 03 01:20:24.302858 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:20:24.302832 4939 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-5774fbf4b9-pl5dw" Nov 03 01:20:24.306307 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:20:24.306278 4939 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"models-as-a-service\"/\"agc-cred-9c667d6233e1\"" Nov 03 01:20:24.309092 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:20:24.309064 4939 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["models-as-a-service/payload-processing-external-model-5774fbf4b9-pl5dw"] Nov 03 01:20:24.346390 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:20:24.346353 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:20:24.530222 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:20:24.530182 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m2ks5\" (UniqueName: \"kubernetes.io/projected/5035296e-642f-420d-ade7-d9c36030cca3-kube-api-access-m2ks5\") pod \"payload-processing-external-model-5774fbf4b9-pl5dw\" (UID: \"5035296e-642f-420d-ade7-d9c36030cca3\") " pod="models-as-a-service/payload-processing-external-model-5774fbf4b9-pl5dw" Nov 03 01:20:24.530410 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:20:24.530242 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/5035296e-642f-420d-ade7-d9c36030cca3-routing-overlay-volume\") pod \"payload-processing-external-model-5774fbf4b9-pl5dw\" (UID: \"5035296e-642f-420d-ade7-d9c36030cca3\") " pod="models-as-a-service/payload-processing-external-model-5774fbf4b9-pl5dw" Nov 03 01:20:24.530548 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:20:24.530520 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/5035296e-642f-420d-ade7-d9c36030cca3-provider-credentials\") pod \"payload-processing-external-model-5774fbf4b9-pl5dw\" (UID: \"5035296e-642f-420d-ade7-d9c36030cca3\") " pod="models-as-a-service/payload-processing-external-model-5774fbf4b9-pl5dw" Nov 03 01:20:24.530603 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:20:24.530564 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5035296e-642f-420d-ade7-d9c36030cca3-plugins-config-volume\") pod \"payload-processing-external-model-5774fbf4b9-pl5dw\" (UID: \"5035296e-642f-420d-ade7-d9c36030cca3\") " pod="models-as-a-service/payload-processing-external-model-5774fbf4b9-pl5dw" Nov 03 01:20:24.632155 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:20:24.632116 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m2ks5\" (UniqueName: \"kubernetes.io/projected/5035296e-642f-420d-ade7-d9c36030cca3-kube-api-access-m2ks5\") pod \"payload-processing-external-model-5774fbf4b9-pl5dw\" (UID: \"5035296e-642f-420d-ade7-d9c36030cca3\") " pod="models-as-a-service/payload-processing-external-model-5774fbf4b9-pl5dw" Nov 03 01:20:24.632347 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:20:24.632168 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/5035296e-642f-420d-ade7-d9c36030cca3-routing-overlay-volume\") pod \"payload-processing-external-model-5774fbf4b9-pl5dw\" (UID: \"5035296e-642f-420d-ade7-d9c36030cca3\") " pod="models-as-a-service/payload-processing-external-model-5774fbf4b9-pl5dw" Nov 03 01:20:24.632394 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:20:24.632359 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/5035296e-642f-420d-ade7-d9c36030cca3-provider-credentials\") pod \"payload-processing-external-model-5774fbf4b9-pl5dw\" (UID: \"5035296e-642f-420d-ade7-d9c36030cca3\") " pod="models-as-a-service/payload-processing-external-model-5774fbf4b9-pl5dw" Nov 03 01:20:24.632394 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:20:24.632385 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5035296e-642f-420d-ade7-d9c36030cca3-plugins-config-volume\") pod \"payload-processing-external-model-5774fbf4b9-pl5dw\" (UID: \"5035296e-642f-420d-ade7-d9c36030cca3\") " pod="models-as-a-service/payload-processing-external-model-5774fbf4b9-pl5dw" Nov 03 01:20:24.632968 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:20:24.632942 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/5035296e-642f-420d-ade7-d9c36030cca3-routing-overlay-volume\") pod \"payload-processing-external-model-5774fbf4b9-pl5dw\" (UID: \"5035296e-642f-420d-ade7-d9c36030cca3\") " pod="models-as-a-service/payload-processing-external-model-5774fbf4b9-pl5dw" Nov 03 01:20:24.633137 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:20:24.633116 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5035296e-642f-420d-ade7-d9c36030cca3-plugins-config-volume\") pod \"payload-processing-external-model-5774fbf4b9-pl5dw\" (UID: \"5035296e-642f-420d-ade7-d9c36030cca3\") " pod="models-as-a-service/payload-processing-external-model-5774fbf4b9-pl5dw" Nov 03 01:20:24.635072 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:20:24.635048 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/5035296e-642f-420d-ade7-d9c36030cca3-provider-credentials\") pod \"payload-processing-external-model-5774fbf4b9-pl5dw\" (UID: \"5035296e-642f-420d-ade7-d9c36030cca3\") " pod="models-as-a-service/payload-processing-external-model-5774fbf4b9-pl5dw" Nov 03 01:20:24.640969 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:20:24.640947 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-m2ks5\" (UniqueName: \"kubernetes.io/projected/5035296e-642f-420d-ade7-d9c36030cca3-kube-api-access-m2ks5\") pod \"payload-processing-external-model-5774fbf4b9-pl5dw\" (UID: \"5035296e-642f-420d-ade7-d9c36030cca3\") " pod="models-as-a-service/payload-processing-external-model-5774fbf4b9-pl5dw" Nov 03 01:20:24.766733 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:20:24.766686 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:20:24.892614 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:20:24.892523 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:20:24.923805 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:20:24.923771 4939 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-5774fbf4b9-pl5dw" Nov 03 01:20:25.111397 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:20:25.111342 4939 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["models-as-a-service/payload-processing-external-model-5774fbf4b9-pl5dw"] Nov 03 01:20:25.443934 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:20:25.434057 4939 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-a-7c759a/payload-processing-external-model-e2e-extmod-a-7c759a-5bf4hhjsl" podUID="486543c2-6226-478d-91c3-bcc7112c4940" containerName="payload-processing" containerID="cri-o://731adb6a2eeedb9c7aa8c5fd9ee03bf5162dbf13cb5021f5798fc249ad4a6c24" gracePeriod=25 Nov 03 01:20:25.568118 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:20:25.567957 4939 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["models-as-a-service/payload-processing-external-model-5774fbf4b9-pl5dw"] Nov 03 01:20:25.758143 ip-10-0-140-107 kubenswrapper[4939]: W1103 01:20:25.756708 4939 logging.go:55] [core] [Channel #8398 SubChannel #8399]grpc: addrConn.createTransport failed to connect to {Addr: "10.132.1.2:50052", ServerName: "10.132.1.2:50052", BalancerAttributes: {"<%!p(pickfirstleaf.managedByPickfirstKeyType={})>": "<%!p(bool=true)>" }}. Err: connection error: desc = "transport: Error while dialing: dial tcp 10.132.1.2:50052: connect: connection refused" Nov 03 01:20:25.787697 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:20:25.787655 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:20:26.006086 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:20:25.965780 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:20:26.069659 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:20:26.034365 4939 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-a-7c759a/payload-processing-external-model-e2e-extmod-a-7c759a-54dbrs2dj" podUID="0fae2685-234b-43e0-aae4-47f912d3bb5b" containerName="payload-processing" containerID="cri-o://43435815ad147ad5b94bfea549aeac051ccc2effefa6fe25792e12f48a091daa" gracePeriod=25 Nov 03 01:20:26.144400 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:20:26.141257 4939 generic.go:356] "Generic (PLEG): container finished" podID="486543c2-6226-478d-91c3-bcc7112c4940" containerID="731adb6a2eeedb9c7aa8c5fd9ee03bf5162dbf13cb5021f5798fc249ad4a6c24" exitCode=0 Nov 03 01:20:26.150777 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:20:26.150292 4939 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="models-as-a-service/payload-processing-external-model-5774fbf4b9-pl5dw" Nov 03 01:20:26.150777 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:20:26.150368 4939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-7c759a/payload-processing-external-model-e2e-extmod-a-7c759a-5bf4hhjsl" event={"ID":"486543c2-6226-478d-91c3-bcc7112c4940","Type":"ContainerDied","Data":"731adb6a2eeedb9c7aa8c5fd9ee03bf5162dbf13cb5021f5798fc249ad4a6c24"} Nov 03 01:20:26.150777 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:20:26.150404 4939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-5774fbf4b9-pl5dw" event={"ID":"5035296e-642f-420d-ade7-d9c36030cca3","Type":"ContainerStarted","Data":"46fca909cdf142df420f0590df23aaac6daf57809cab88af38cdee004834f4c8"} Nov 03 01:20:26.150777 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:20:26.150428 4939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-5774fbf4b9-pl5dw" event={"ID":"5035296e-642f-420d-ade7-d9c36030cca3","Type":"ContainerStarted","Data":"d45ff5cc7d88fc14a4bc9da0d10505f8c36db50d3c1df7cf48ecb46875707020"} Nov 03 01:20:26.183198 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:20:26.183137 4939 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="models-as-a-service/payload-processing-external-model-5774fbf4b9-pl5dw" podStartSLOduration=2.183116719 podStartE2EDuration="2.183116719s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 01:20:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 01:20:26.170411075 +0000 UTC m=+2374.434746022" watchObservedRunningTime="2026-11-03 01:20:26.183116719 +0000 UTC m=+2374.447451664" Nov 03 01:20:26.223162 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:20:26.223130 4939 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-7c759a/payload-processing-external-model-e2e-extmod-a-7c759a-5bf4hhjsl" Nov 03 01:20:26.247037 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:20:26.246993 4939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/486543c2-6226-478d-91c3-bcc7112c4940-provider-credentials\" (UniqueName: \"kubernetes.io/projected/486543c2-6226-478d-91c3-bcc7112c4940-provider-credentials\") pod \"486543c2-6226-478d-91c3-bcc7112c4940\" (UID: \"486543c2-6226-478d-91c3-bcc7112c4940\") " Nov 03 01:20:26.252392 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:20:26.247253 4939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/486543c2-6226-478d-91c3-bcc7112c4940-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/486543c2-6226-478d-91c3-bcc7112c4940-routing-overlay-volume\") pod \"486543c2-6226-478d-91c3-bcc7112c4940\" (UID: \"486543c2-6226-478d-91c3-bcc7112c4940\") " Nov 03 01:20:26.252392 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:20:26.247393 4939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/486543c2-6226-478d-91c3-bcc7112c4940-kube-api-access-k5mgh\" (UniqueName: \"kubernetes.io/projected/486543c2-6226-478d-91c3-bcc7112c4940-kube-api-access-k5mgh\") pod \"486543c2-6226-478d-91c3-bcc7112c4940\" (UID: \"486543c2-6226-478d-91c3-bcc7112c4940\") " Nov 03 01:20:26.252392 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:20:26.247458 4939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/486543c2-6226-478d-91c3-bcc7112c4940-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/486543c2-6226-478d-91c3-bcc7112c4940-plugins-config-volume\") pod \"486543c2-6226-478d-91c3-bcc7112c4940\" (UID: \"486543c2-6226-478d-91c3-bcc7112c4940\") " Nov 03 01:20:26.252392 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:20:26.248630 4939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/486543c2-6226-478d-91c3-bcc7112c4940-plugins-config-volume" pod "486543c2-6226-478d-91c3-bcc7112c4940" (UID: "486543c2-6226-478d-91c3-bcc7112c4940"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 01:20:26.252392 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:20:26.252121 4939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/486543c2-6226-478d-91c3-bcc7112c4940-provider-credentials" pod "486543c2-6226-478d-91c3-bcc7112c4940" (UID: "486543c2-6226-478d-91c3-bcc7112c4940"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 01:20:26.253140 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:20:26.253113 4939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/486543c2-6226-478d-91c3-bcc7112c4940-routing-overlay-volume" pod "486543c2-6226-478d-91c3-bcc7112c4940" (UID: "486543c2-6226-478d-91c3-bcc7112c4940"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 01:20:26.261199 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:20:26.261119 4939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/486543c2-6226-478d-91c3-bcc7112c4940-kube-api-access-k5mgh" pod "486543c2-6226-478d-91c3-bcc7112c4940" (UID: "486543c2-6226-478d-91c3-bcc7112c4940"). InnerVolumeSpecName "kube-api-access-k5mgh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 01:20:26.270488 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:20:26.270455 4939 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["models-as-a-service/payload-processing-external-model-76c77f5454-kqvq2"] Nov 03 01:20:26.271691 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:20:26.271666 4939 cpu_manager.go:414] "containerMap: removing container" podUID="486543c2-6226-478d-91c3-bcc7112c4940" containerName="payload-processing" Nov 03 01:20:26.271691 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:20:26.271690 4939 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="486543c2-6226-478d-91c3-bcc7112c4940" containerName="payload-processing" Nov 03 01:20:26.271960 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:20:26.271942 4939 memory_manager.go:365] "RemoveStaleState removing state" podUID="486543c2-6226-478d-91c3-bcc7112c4940" containerName="payload-processing" Nov 03 01:20:26.276593 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:20:26.276573 4939 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-76c77f5454-kqvq2" Nov 03 01:20:26.282734 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:20:26.282665 4939 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["models-as-a-service/payload-processing-external-model-76c77f5454-kqvq2"] Nov 03 01:20:26.304599 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:20:26.304552 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:20:26.377952 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:20:26.373758 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mkx6q\" (UniqueName: \"kubernetes.io/projected/84a1e602-05ea-44fa-8a3c-db2d82dea9d8-kube-api-access-mkx6q\") pod \"payload-processing-external-model-76c77f5454-kqvq2\" (UID: \"84a1e602-05ea-44fa-8a3c-db2d82dea9d8\") " pod="models-as-a-service/payload-processing-external-model-76c77f5454-kqvq2" Nov 03 01:20:26.377952 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:20:26.373863 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/84a1e602-05ea-44fa-8a3c-db2d82dea9d8-provider-credentials\") pod \"payload-processing-external-model-76c77f5454-kqvq2\" (UID: \"84a1e602-05ea-44fa-8a3c-db2d82dea9d8\") " pod="models-as-a-service/payload-processing-external-model-76c77f5454-kqvq2" Nov 03 01:20:26.377952 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:20:26.373983 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/84a1e602-05ea-44fa-8a3c-db2d82dea9d8-plugins-config-volume\") pod \"payload-processing-external-model-76c77f5454-kqvq2\" (UID: \"84a1e602-05ea-44fa-8a3c-db2d82dea9d8\") " pod="models-as-a-service/payload-processing-external-model-76c77f5454-kqvq2" Nov 03 01:20:26.377952 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:20:26.374103 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/84a1e602-05ea-44fa-8a3c-db2d82dea9d8-routing-overlay-volume\") pod \"payload-processing-external-model-76c77f5454-kqvq2\" (UID: \"84a1e602-05ea-44fa-8a3c-db2d82dea9d8\") " pod="models-as-a-service/payload-processing-external-model-76c77f5454-kqvq2" Nov 03 01:20:26.377952 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:20:26.374473 4939 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/486543c2-6226-478d-91c3-bcc7112c4940-routing-overlay-volume\") on node \"ip-10-0-140-107.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:20:26.377952 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:20:26.374496 4939 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k5mgh\" (UniqueName: \"kubernetes.io/projected/486543c2-6226-478d-91c3-bcc7112c4940-kube-api-access-k5mgh\") on node \"ip-10-0-140-107.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:20:26.377952 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:20:26.374513 4939 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/486543c2-6226-478d-91c3-bcc7112c4940-plugins-config-volume\") on node \"ip-10-0-140-107.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:20:26.377952 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:20:26.374528 4939 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/486543c2-6226-478d-91c3-bcc7112c4940-provider-credentials\") on node \"ip-10-0-140-107.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:20:26.505941 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:20:26.482322 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mkx6q\" (UniqueName: \"kubernetes.io/projected/84a1e602-05ea-44fa-8a3c-db2d82dea9d8-kube-api-access-mkx6q\") pod \"payload-processing-external-model-76c77f5454-kqvq2\" (UID: \"84a1e602-05ea-44fa-8a3c-db2d82dea9d8\") " pod="models-as-a-service/payload-processing-external-model-76c77f5454-kqvq2" Nov 03 01:20:26.505941 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:20:26.482451 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/84a1e602-05ea-44fa-8a3c-db2d82dea9d8-provider-credentials\") pod \"payload-processing-external-model-76c77f5454-kqvq2\" (UID: \"84a1e602-05ea-44fa-8a3c-db2d82dea9d8\") " pod="models-as-a-service/payload-processing-external-model-76c77f5454-kqvq2" Nov 03 01:20:26.505941 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:20:26.482579 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/84a1e602-05ea-44fa-8a3c-db2d82dea9d8-plugins-config-volume\") pod \"payload-processing-external-model-76c77f5454-kqvq2\" (UID: \"84a1e602-05ea-44fa-8a3c-db2d82dea9d8\") " pod="models-as-a-service/payload-processing-external-model-76c77f5454-kqvq2" Nov 03 01:20:26.505941 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:20:26.482705 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/84a1e602-05ea-44fa-8a3c-db2d82dea9d8-routing-overlay-volume\") pod \"payload-processing-external-model-76c77f5454-kqvq2\" (UID: \"84a1e602-05ea-44fa-8a3c-db2d82dea9d8\") " pod="models-as-a-service/payload-processing-external-model-76c77f5454-kqvq2" Nov 03 01:20:26.505941 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:20:26.484111 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/84a1e602-05ea-44fa-8a3c-db2d82dea9d8-routing-overlay-volume\") pod \"payload-processing-external-model-76c77f5454-kqvq2\" (UID: \"84a1e602-05ea-44fa-8a3c-db2d82dea9d8\") " pod="models-as-a-service/payload-processing-external-model-76c77f5454-kqvq2" Nov 03 01:20:26.505941 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:20:26.499877 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/84a1e602-05ea-44fa-8a3c-db2d82dea9d8-plugins-config-volume\") pod \"payload-processing-external-model-76c77f5454-kqvq2\" (UID: \"84a1e602-05ea-44fa-8a3c-db2d82dea9d8\") " pod="models-as-a-service/payload-processing-external-model-76c77f5454-kqvq2" Nov 03 01:20:26.589983 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:20:26.564457 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/84a1e602-05ea-44fa-8a3c-db2d82dea9d8-provider-credentials\") pod \"payload-processing-external-model-76c77f5454-kqvq2\" (UID: \"84a1e602-05ea-44fa-8a3c-db2d82dea9d8\") " pod="models-as-a-service/payload-processing-external-model-76c77f5454-kqvq2" Nov 03 01:20:26.624019 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:20:26.620414 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mkx6q\" (UniqueName: \"kubernetes.io/projected/84a1e602-05ea-44fa-8a3c-db2d82dea9d8-kube-api-access-mkx6q\") pod \"payload-processing-external-model-76c77f5454-kqvq2\" (UID: \"84a1e602-05ea-44fa-8a3c-db2d82dea9d8\") " pod="models-as-a-service/payload-processing-external-model-76c77f5454-kqvq2" Nov 03 01:20:26.633572 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:20:26.627209 4939 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-76c77f5454-kqvq2" Nov 03 01:20:26.756967 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:20:26.756645 4939 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-extmod-a-7c759a/payload-processing-external-model-e2e-extmod-a-7c759a-5bf4hhjsl" podUID="486543c2-6226-478d-91c3-bcc7112c4940" containerName="payload-processing" probeResult="failure" output="timeout: failed to connect service \"10.132.1.2:50052\" within 1s: context deadline exceeded" Nov 03 01:20:26.979168 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:20:26.979130 4939 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-7c759a/payload-processing-external-model-e2e-extmod-a-7c759a-54dbrs2dj" Nov 03 01:20:27.012540 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:20:27.012504 4939 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["models-as-a-service/payload-processing-external-model-76c77f5454-kqvq2"] Nov 03 01:20:27.142544 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:20:27.142451 4939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0fae2685-234b-43e0-aae4-47f912d3bb5b-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/0fae2685-234b-43e0-aae4-47f912d3bb5b-routing-overlay-volume\") pod \"0fae2685-234b-43e0-aae4-47f912d3bb5b\" (UID: \"0fae2685-234b-43e0-aae4-47f912d3bb5b\") " Nov 03 01:20:27.142798 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:20:27.142777 4939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0fae2685-234b-43e0-aae4-47f912d3bb5b-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0fae2685-234b-43e0-aae4-47f912d3bb5b-plugins-config-volume\") pod \"0fae2685-234b-43e0-aae4-47f912d3bb5b\" (UID: \"0fae2685-234b-43e0-aae4-47f912d3bb5b\") " Nov 03 01:20:27.142872 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:20:27.142853 4939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0fae2685-234b-43e0-aae4-47f912d3bb5b-routing-overlay-volume" pod "0fae2685-234b-43e0-aae4-47f912d3bb5b" (UID: "0fae2685-234b-43e0-aae4-47f912d3bb5b"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 01:20:27.142952 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:20:27.142904 4939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0fae2685-234b-43e0-aae4-47f912d3bb5b-kube-api-access-dzwlz\" (UniqueName: \"kubernetes.io/projected/0fae2685-234b-43e0-aae4-47f912d3bb5b-kube-api-access-dzwlz\") pod \"0fae2685-234b-43e0-aae4-47f912d3bb5b\" (UID: \"0fae2685-234b-43e0-aae4-47f912d3bb5b\") " Nov 03 01:20:27.142989 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:20:27.142979 4939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0fae2685-234b-43e0-aae4-47f912d3bb5b-provider-credentials\" (UniqueName: \"kubernetes.io/projected/0fae2685-234b-43e0-aae4-47f912d3bb5b-provider-credentials\") pod \"0fae2685-234b-43e0-aae4-47f912d3bb5b\" (UID: \"0fae2685-234b-43e0-aae4-47f912d3bb5b\") " Nov 03 01:20:27.143194 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:20:27.143171 4939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0fae2685-234b-43e0-aae4-47f912d3bb5b-plugins-config-volume" pod "0fae2685-234b-43e0-aae4-47f912d3bb5b" (UID: "0fae2685-234b-43e0-aae4-47f912d3bb5b"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 01:20:27.143676 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:20:27.143659 4939 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0fae2685-234b-43e0-aae4-47f912d3bb5b-plugins-config-volume\") on node \"ip-10-0-140-107.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:20:27.143751 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:20:27.143678 4939 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/0fae2685-234b-43e0-aae4-47f912d3bb5b-routing-overlay-volume\") on node \"ip-10-0-140-107.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:20:27.145163 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:20:27.145142 4939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0fae2685-234b-43e0-aae4-47f912d3bb5b-provider-credentials" pod "0fae2685-234b-43e0-aae4-47f912d3bb5b" (UID: "0fae2685-234b-43e0-aae4-47f912d3bb5b"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 01:20:27.145256 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:20:27.145182 4939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0fae2685-234b-43e0-aae4-47f912d3bb5b-kube-api-access-dzwlz" pod "0fae2685-234b-43e0-aae4-47f912d3bb5b" (UID: "0fae2685-234b-43e0-aae4-47f912d3bb5b"). InnerVolumeSpecName "kube-api-access-dzwlz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 01:20:27.158549 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:20:27.158519 4939 generic.go:356] "Generic (PLEG): container finished" podID="0fae2685-234b-43e0-aae4-47f912d3bb5b" containerID="43435815ad147ad5b94bfea549aeac051ccc2effefa6fe25792e12f48a091daa" exitCode=0 Nov 03 01:20:27.158662 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:20:27.158573 4939 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-7c759a/payload-processing-external-model-e2e-extmod-a-7c759a-54dbrs2dj" Nov 03 01:20:27.158662 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:20:27.158608 4939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-7c759a/payload-processing-external-model-e2e-extmod-a-7c759a-54dbrs2dj" event={"ID":"0fae2685-234b-43e0-aae4-47f912d3bb5b","Type":"ContainerDied","Data":"43435815ad147ad5b94bfea549aeac051ccc2effefa6fe25792e12f48a091daa"} Nov 03 01:20:27.158662 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:20:27.158657 4939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-7c759a/payload-processing-external-model-e2e-extmod-a-7c759a-54dbrs2dj" event={"ID":"0fae2685-234b-43e0-aae4-47f912d3bb5b","Type":"ContainerDied","Data":"8742cce92043eecd61a8be191f0e92be419c01d99ccaf300dc4f86d24a8a8b46"} Nov 03 01:20:27.158774 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:20:27.158689 4939 scope.go:122] "RemoveContainer" containerID="43435815ad147ad5b94bfea549aeac051ccc2effefa6fe25792e12f48a091daa" Nov 03 01:20:27.160624 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:20:27.160598 4939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-76c77f5454-kqvq2" event={"ID":"84a1e602-05ea-44fa-8a3c-db2d82dea9d8","Type":"ContainerStarted","Data":"5ce76f64ded4525e90504b9c6a2c5ec11b20abfd3a1fb153b538b427d5d32ba7"} Nov 03 01:20:27.160740 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:20:27.160650 4939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-76c77f5454-kqvq2" event={"ID":"84a1e602-05ea-44fa-8a3c-db2d82dea9d8","Type":"ContainerStarted","Data":"8195c852eedf010b64ca654706205cc0342c00b0b614236c1a036499e43edc5d"} Nov 03 01:20:27.160740 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:20:27.160683 4939 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="models-as-a-service/payload-processing-external-model-76c77f5454-kqvq2" Nov 03 01:20:27.162325 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:20:27.162300 4939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-7c759a/payload-processing-external-model-e2e-extmod-a-7c759a-5bf4hhjsl" event={"ID":"486543c2-6226-478d-91c3-bcc7112c4940","Type":"ContainerDied","Data":"b4383b26c954eaba7ab167baf4bc9dbc5f87dce1910a7b29e55e828f44554e0c"} Nov 03 01:20:27.162435 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:20:27.162327 4939 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-7c759a/payload-processing-external-model-e2e-extmod-a-7c759a-5bf4hhjsl" Nov 03 01:20:27.170840 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:20:27.170820 4939 scope.go:122] "RemoveContainer" containerID="43435815ad147ad5b94bfea549aeac051ccc2effefa6fe25792e12f48a091daa" Nov 03 01:20:27.171145 ip-10-0-140-107 kubenswrapper[4939]: E1103 01:20:27.171125 4939 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"43435815ad147ad5b94bfea549aeac051ccc2effefa6fe25792e12f48a091daa\": container with ID starting with 43435815ad147ad5b94bfea549aeac051ccc2effefa6fe25792e12f48a091daa not found: ID does not exist" containerID="43435815ad147ad5b94bfea549aeac051ccc2effefa6fe25792e12f48a091daa" Nov 03 01:20:27.171217 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:20:27.171154 4939 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"43435815ad147ad5b94bfea549aeac051ccc2effefa6fe25792e12f48a091daa"} err="failed to get container status \"43435815ad147ad5b94bfea549aeac051ccc2effefa6fe25792e12f48a091daa\": rpc error: code = NotFound desc = could not find container \"43435815ad147ad5b94bfea549aeac051ccc2effefa6fe25792e12f48a091daa\": container with ID starting with 43435815ad147ad5b94bfea549aeac051ccc2effefa6fe25792e12f48a091daa not found: ID does not exist" Nov 03 01:20:27.171217 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:20:27.171172 4939 scope.go:122] "RemoveContainer" containerID="731adb6a2eeedb9c7aa8c5fd9ee03bf5162dbf13cb5021f5798fc249ad4a6c24" Nov 03 01:20:27.180712 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:20:27.180662 4939 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="models-as-a-service/payload-processing-external-model-76c77f5454-kqvq2" podStartSLOduration=1.180649253 podStartE2EDuration="1.180649253s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 01:20:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 01:20:27.178888912 +0000 UTC m=+2375.443223858" watchObservedRunningTime="2026-11-03 01:20:27.180649253 +0000 UTC m=+2375.444984209" Nov 03 01:20:27.201217 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:20:27.201192 4939 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-7c759a/payload-processing-external-model-e2e-extmod-a-7c759a-5bf4hhjsl"] Nov 03 01:20:27.209358 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:20:27.209334 4939 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-a-7c759a/payload-processing-external-model-e2e-extmod-a-7c759a-5bf4hhjsl"] Nov 03 01:20:27.219616 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:20:27.219592 4939 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-7c759a/payload-processing-external-model-e2e-extmod-a-7c759a-54dbrs2dj"] Nov 03 01:20:27.224691 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:20:27.224664 4939 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-a-7c759a/payload-processing-external-model-e2e-extmod-a-7c759a-54dbrs2dj"] Nov 03 01:20:27.245397 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:20:27.245374 4939 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dzwlz\" (UniqueName: \"kubernetes.io/projected/0fae2685-234b-43e0-aae4-47f912d3bb5b-kube-api-access-dzwlz\") on node \"ip-10-0-140-107.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:20:27.245496 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:20:27.245398 4939 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/0fae2685-234b-43e0-aae4-47f912d3bb5b-provider-credentials\") on node \"ip-10-0-140-107.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:20:27.273487 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:20:27.273460 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:20:27.452985 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:20:27.452313 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:20:27.624935 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:20:27.618635 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:20:27.816686 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:20:27.816600 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:20:27.873592 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:20:27.873540 4939 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0fae2685-234b-43e0-aae4-47f912d3bb5b" path="/var/lib/kubelet/pods/0fae2685-234b-43e0-aae4-47f912d3bb5b/volumes" Nov 03 01:20:27.874234 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:20:27.874208 4939 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="486543c2-6226-478d-91c3-bcc7112c4940" path="/var/lib/kubelet/pods/486543c2-6226-478d-91c3-bcc7112c4940/volumes" Nov 03 01:20:27.977053 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:20:27.977020 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:20:28.109933 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:20:28.108489 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:20:28.759929 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:20:28.754844 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:20:28.892310 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:20:28.892273 4939 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["models-as-a-service/payload-processing-external-model-76c77f5454-kqvq2"] Nov 03 01:20:28.919605 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:20:28.919571 4939 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["models-as-a-service/payload-processing-external-model-59dbb77fb8-5h2hn"] Nov 03 01:20:28.920512 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:20:28.920495 4939 cpu_manager.go:414] "containerMap: removing container" podUID="0fae2685-234b-43e0-aae4-47f912d3bb5b" containerName="payload-processing" Nov 03 01:20:28.920512 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:20:28.920513 4939 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0fae2685-234b-43e0-aae4-47f912d3bb5b" containerName="payload-processing" Nov 03 01:20:28.920688 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:20:28.920677 4939 memory_manager.go:365] "RemoveStaleState removing state" podUID="0fae2685-234b-43e0-aae4-47f912d3bb5b" containerName="payload-processing" Nov 03 01:20:28.926418 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:20:28.926394 4939 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-59dbb77fb8-5h2hn" Nov 03 01:20:28.937966 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:20:28.937940 4939 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"models-as-a-service\"/\"agc-cred-7379d3751ff2\"" Nov 03 01:20:28.946737 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:20:28.946479 4939 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["models-as-a-service/payload-processing-external-model-59dbb77fb8-5h2hn"] Nov 03 01:20:28.985712 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:20:28.985678 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:20:29.025078 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:20:29.024962 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c0272368-d591-4cae-ace9-be49c71471f3-plugins-config-volume\") pod \"payload-processing-external-model-59dbb77fb8-5h2hn\" (UID: \"c0272368-d591-4cae-ace9-be49c71471f3\") " pod="models-as-a-service/payload-processing-external-model-59dbb77fb8-5h2hn" Nov 03 01:20:29.025078 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:20:29.025014 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qs7cw\" (UniqueName: \"kubernetes.io/projected/c0272368-d591-4cae-ace9-be49c71471f3-kube-api-access-qs7cw\") pod \"payload-processing-external-model-59dbb77fb8-5h2hn\" (UID: \"c0272368-d591-4cae-ace9-be49c71471f3\") " pod="models-as-a-service/payload-processing-external-model-59dbb77fb8-5h2hn" Nov 03 01:20:29.025384 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:20:29.025351 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/c0272368-d591-4cae-ace9-be49c71471f3-routing-overlay-volume\") pod \"payload-processing-external-model-59dbb77fb8-5h2hn\" (UID: \"c0272368-d591-4cae-ace9-be49c71471f3\") " pod="models-as-a-service/payload-processing-external-model-59dbb77fb8-5h2hn" Nov 03 01:20:29.027149 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:20:29.027115 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/c0272368-d591-4cae-ace9-be49c71471f3-provider-credentials\") pod \"payload-processing-external-model-59dbb77fb8-5h2hn\" (UID: \"c0272368-d591-4cae-ace9-be49c71471f3\") " pod="models-as-a-service/payload-processing-external-model-59dbb77fb8-5h2hn" Nov 03 01:20:29.111690 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:20:29.111653 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:20:29.129133 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:20:29.129098 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/c0272368-d591-4cae-ace9-be49c71471f3-routing-overlay-volume\") pod \"payload-processing-external-model-59dbb77fb8-5h2hn\" (UID: \"c0272368-d591-4cae-ace9-be49c71471f3\") " pod="models-as-a-service/payload-processing-external-model-59dbb77fb8-5h2hn" Nov 03 01:20:29.129324 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:20:29.129186 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/c0272368-d591-4cae-ace9-be49c71471f3-provider-credentials\") pod \"payload-processing-external-model-59dbb77fb8-5h2hn\" (UID: \"c0272368-d591-4cae-ace9-be49c71471f3\") " pod="models-as-a-service/payload-processing-external-model-59dbb77fb8-5h2hn" Nov 03 01:20:29.129375 ip-10-0-140-107 kubenswrapper[4939]: E1103 01:20:29.129337 4939 projected.go:266] Couldn't get secret models-as-a-service/agc-cred-7379d3751ff2: secret "agc-cred-7379d3751ff2" not found Nov 03 01:20:29.129375 ip-10-0-140-107 kubenswrapper[4939]: E1103 01:20:29.129366 4939 projected.go:196] Error preparing data for projected volume provider-credentials for pod models-as-a-service/payload-processing-external-model-59dbb77fb8-5h2hn: secret "agc-cred-7379d3751ff2" not found Nov 03 01:20:29.129458 ip-10-0-140-107 kubenswrapper[4939]: E1103 01:20:29.129433 4939 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/c0272368-d591-4cae-ace9-be49c71471f3-provider-credentials podName:c0272368-d591-4cae-ace9-be49c71471f3 nodeName:}" failed. No retries permitted until 2026-11-03 01:20:29.629410452 +0000 UTC m=+2377.893745393 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/c0272368-d591-4cae-ace9-be49c71471f3-provider-credentials") pod "payload-processing-external-model-59dbb77fb8-5h2hn" (UID: "c0272368-d591-4cae-ace9-be49c71471f3") : secret "agc-cred-7379d3751ff2" not found Nov 03 01:20:29.129513 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:20:29.129343 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c0272368-d591-4cae-ace9-be49c71471f3-plugins-config-volume\") pod \"payload-processing-external-model-59dbb77fb8-5h2hn\" (UID: \"c0272368-d591-4cae-ace9-be49c71471f3\") " pod="models-as-a-service/payload-processing-external-model-59dbb77fb8-5h2hn" Nov 03 01:20:29.129555 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:20:29.129508 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qs7cw\" (UniqueName: \"kubernetes.io/projected/c0272368-d591-4cae-ace9-be49c71471f3-kube-api-access-qs7cw\") pod \"payload-processing-external-model-59dbb77fb8-5h2hn\" (UID: \"c0272368-d591-4cae-ace9-be49c71471f3\") " pod="models-as-a-service/payload-processing-external-model-59dbb77fb8-5h2hn" Nov 03 01:20:29.129960 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:20:29.129937 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/c0272368-d591-4cae-ace9-be49c71471f3-routing-overlay-volume\") pod \"payload-processing-external-model-59dbb77fb8-5h2hn\" (UID: \"c0272368-d591-4cae-ace9-be49c71471f3\") " pod="models-as-a-service/payload-processing-external-model-59dbb77fb8-5h2hn" Nov 03 01:20:29.130027 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:20:29.130010 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c0272368-d591-4cae-ace9-be49c71471f3-plugins-config-volume\") pod \"payload-processing-external-model-59dbb77fb8-5h2hn\" (UID: \"c0272368-d591-4cae-ace9-be49c71471f3\") " pod="models-as-a-service/payload-processing-external-model-59dbb77fb8-5h2hn" Nov 03 01:20:29.139464 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:20:29.139438 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qs7cw\" (UniqueName: \"kubernetes.io/projected/c0272368-d591-4cae-ace9-be49c71471f3-kube-api-access-qs7cw\") pod \"payload-processing-external-model-59dbb77fb8-5h2hn\" (UID: \"c0272368-d591-4cae-ace9-be49c71471f3\") " pod="models-as-a-service/payload-processing-external-model-59dbb77fb8-5h2hn" Nov 03 01:20:29.233384 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:20:29.233352 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:20:29.254983 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:20:29.254948 4939 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-a-821f5a/payload-processing-external-model-e2e-extmod-a-821f5a-5df8cj84n"] Nov 03 01:20:29.260650 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:20:29.260624 4939 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-821f5a/payload-processing-external-model-e2e-extmod-a-821f5a-5df8cj84n" Nov 03 01:20:29.268429 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:20:29.268396 4939 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-821f5a\"/\"agc-cred-f4bdc094c20f\"" Nov 03 01:20:29.269095 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:20:29.268731 4939 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-821f5a\"/\"payload-processing-external-model-plugins-e2e-extmod-a-821f5a\"" Nov 03 01:20:29.269095 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:20:29.268825 4939 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-821f5a\"/\"routing-overlay\"" Nov 03 01:20:29.269095 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:20:29.268879 4939 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-821f5a\"/\"openshift-service-ca.crt\"" Nov 03 01:20:29.269095 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:20:29.268938 4939 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-821f5a\"/\"kube-root-ca.crt\"" Nov 03 01:20:29.269413 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:20:29.269107 4939 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-821f5a\"/\"payload-processing-external-model-e2e--1074f2f6-dockercfg-f5qf7\"" Nov 03 01:20:29.273412 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:20:29.273383 4939 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-821f5a/payload-processing-external-model-e2e-extmod-a-821f5a-5df8cj84n"] Nov 03 01:20:29.332924 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:20:29.332883 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/5806cd37-5cdf-4afe-aa0b-c16b09db3106-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-821f5a-5df8cj84n\" (UID: \"5806cd37-5cdf-4afe-aa0b-c16b09db3106\") " pod="ai-tenant-e2e-extmod-a-821f5a/payload-processing-external-model-e2e-extmod-a-821f5a-5df8cj84n" Nov 03 01:20:29.333122 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:20:29.332984 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5806cd37-5cdf-4afe-aa0b-c16b09db3106-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-821f5a-5df8cj84n\" (UID: \"5806cd37-5cdf-4afe-aa0b-c16b09db3106\") " pod="ai-tenant-e2e-extmod-a-821f5a/payload-processing-external-model-e2e-extmod-a-821f5a-5df8cj84n" Nov 03 01:20:29.333185 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:20:29.333119 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/5806cd37-5cdf-4afe-aa0b-c16b09db3106-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-821f5a-5df8cj84n\" (UID: \"5806cd37-5cdf-4afe-aa0b-c16b09db3106\") " pod="ai-tenant-e2e-extmod-a-821f5a/payload-processing-external-model-e2e-extmod-a-821f5a-5df8cj84n" Nov 03 01:20:29.333185 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:20:29.333166 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rgrzx\" (UniqueName: \"kubernetes.io/projected/5806cd37-5cdf-4afe-aa0b-c16b09db3106-kube-api-access-rgrzx\") pod \"payload-processing-external-model-e2e-extmod-a-821f5a-5df8cj84n\" (UID: \"5806cd37-5cdf-4afe-aa0b-c16b09db3106\") " pod="ai-tenant-e2e-extmod-a-821f5a/payload-processing-external-model-e2e-extmod-a-821f5a-5df8cj84n" Nov 03 01:20:29.437932 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:20:29.435587 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/5806cd37-5cdf-4afe-aa0b-c16b09db3106-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-821f5a-5df8cj84n\" (UID: \"5806cd37-5cdf-4afe-aa0b-c16b09db3106\") " pod="ai-tenant-e2e-extmod-a-821f5a/payload-processing-external-model-e2e-extmod-a-821f5a-5df8cj84n" Nov 03 01:20:29.437932 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:20:29.435684 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rgrzx\" (UniqueName: \"kubernetes.io/projected/5806cd37-5cdf-4afe-aa0b-c16b09db3106-kube-api-access-rgrzx\") pod \"payload-processing-external-model-e2e-extmod-a-821f5a-5df8cj84n\" (UID: \"5806cd37-5cdf-4afe-aa0b-c16b09db3106\") " pod="ai-tenant-e2e-extmod-a-821f5a/payload-processing-external-model-e2e-extmod-a-821f5a-5df8cj84n" Nov 03 01:20:29.437932 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:20:29.436046 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/5806cd37-5cdf-4afe-aa0b-c16b09db3106-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-821f5a-5df8cj84n\" (UID: \"5806cd37-5cdf-4afe-aa0b-c16b09db3106\") " pod="ai-tenant-e2e-extmod-a-821f5a/payload-processing-external-model-e2e-extmod-a-821f5a-5df8cj84n" Nov 03 01:20:29.437932 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:20:29.436116 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5806cd37-5cdf-4afe-aa0b-c16b09db3106-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-821f5a-5df8cj84n\" (UID: \"5806cd37-5cdf-4afe-aa0b-c16b09db3106\") " pod="ai-tenant-e2e-extmod-a-821f5a/payload-processing-external-model-e2e-extmod-a-821f5a-5df8cj84n" Nov 03 01:20:29.437932 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:20:29.437179 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5806cd37-5cdf-4afe-aa0b-c16b09db3106-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-821f5a-5df8cj84n\" (UID: \"5806cd37-5cdf-4afe-aa0b-c16b09db3106\") " pod="ai-tenant-e2e-extmod-a-821f5a/payload-processing-external-model-e2e-extmod-a-821f5a-5df8cj84n" Nov 03 01:20:29.470942 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:20:29.437901 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/5806cd37-5cdf-4afe-aa0b-c16b09db3106-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-821f5a-5df8cj84n\" (UID: \"5806cd37-5cdf-4afe-aa0b-c16b09db3106\") " pod="ai-tenant-e2e-extmod-a-821f5a/payload-processing-external-model-e2e-extmod-a-821f5a-5df8cj84n" Nov 03 01:20:29.510961 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:20:29.497048 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rgrzx\" (UniqueName: \"kubernetes.io/projected/5806cd37-5cdf-4afe-aa0b-c16b09db3106-kube-api-access-rgrzx\") pod \"payload-processing-external-model-e2e-extmod-a-821f5a-5df8cj84n\" (UID: \"5806cd37-5cdf-4afe-aa0b-c16b09db3106\") " pod="ai-tenant-e2e-extmod-a-821f5a/payload-processing-external-model-e2e-extmod-a-821f5a-5df8cj84n" Nov 03 01:20:29.510961 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:20:29.508325 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/5806cd37-5cdf-4afe-aa0b-c16b09db3106-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-821f5a-5df8cj84n\" (UID: \"5806cd37-5cdf-4afe-aa0b-c16b09db3106\") " pod="ai-tenant-e2e-extmod-a-821f5a/payload-processing-external-model-e2e-extmod-a-821f5a-5df8cj84n" Nov 03 01:20:29.520475 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:20:29.520425 4939 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["models-as-a-service/payload-processing-external-model-59dbb77fb8-5h2hn"] Nov 03 01:20:29.528089 ip-10-0-140-107 kubenswrapper[4939]: E1103 01:20:29.527971 4939 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[provider-credentials], unattached volumes=[], failed to process volumes=[]: context canceled" pod="models-as-a-service/payload-processing-external-model-59dbb77fb8-5h2hn" podUID="c0272368-d591-4cae-ace9-be49c71471f3" Nov 03 01:20:29.586222 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:20:29.586133 4939 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-821f5a/payload-processing-external-model-e2e-extmod-a-821f5a-5df8cj84n" Nov 03 01:20:29.673016 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:20:29.672973 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/c0272368-d591-4cae-ace9-be49c71471f3-provider-credentials\") pod \"payload-processing-external-model-59dbb77fb8-5h2hn\" (UID: \"c0272368-d591-4cae-ace9-be49c71471f3\") " pod="models-as-a-service/payload-processing-external-model-59dbb77fb8-5h2hn" Nov 03 01:20:29.673935 ip-10-0-140-107 kubenswrapper[4939]: E1103 01:20:29.673888 4939 projected.go:266] Couldn't get secret models-as-a-service/agc-cred-7379d3751ff2: secret "agc-cred-7379d3751ff2" not found Nov 03 01:20:29.674095 ip-10-0-140-107 kubenswrapper[4939]: E1103 01:20:29.674081 4939 projected.go:196] Error preparing data for projected volume provider-credentials for pod models-as-a-service/payload-processing-external-model-59dbb77fb8-5h2hn: secret "agc-cred-7379d3751ff2" not found Nov 03 01:20:29.674240 ip-10-0-140-107 kubenswrapper[4939]: E1103 01:20:29.674229 4939 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/c0272368-d591-4cae-ace9-be49c71471f3-provider-credentials podName:c0272368-d591-4cae-ace9-be49c71471f3 nodeName:}" failed. No retries permitted until 2026-11-03 01:20:30.67420834 +0000 UTC m=+2378.938543266 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/c0272368-d591-4cae-ace9-be49c71471f3-provider-credentials") pod "payload-processing-external-model-59dbb77fb8-5h2hn" (UID: "c0272368-d591-4cae-ace9-be49c71471f3") : secret "agc-cred-7379d3751ff2" not found Nov 03 01:20:29.730974 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:20:29.729079 4939 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["models-as-a-service/payload-processing-external-model-76c77f5454-5cg5j"] Nov 03 01:20:29.838593 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:20:29.838481 4939 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["models-as-a-service/payload-processing-external-model-76c77f5454-5cg5j"] Nov 03 01:20:29.839098 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:20:29.838609 4939 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-76c77f5454-5cg5j" Nov 03 01:20:29.893930 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:20:29.893311 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/e9d0a336-6111-4224-86d9-c95fe94b9545-provider-credentials\") pod \"payload-processing-external-model-76c77f5454-5cg5j\" (UID: \"e9d0a336-6111-4224-86d9-c95fe94b9545\") " pod="models-as-a-service/payload-processing-external-model-76c77f5454-5cg5j" Nov 03 01:20:29.893930 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:20:29.893405 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qfg9p\" (UniqueName: \"kubernetes.io/projected/e9d0a336-6111-4224-86d9-c95fe94b9545-kube-api-access-qfg9p\") pod \"payload-processing-external-model-76c77f5454-5cg5j\" (UID: \"e9d0a336-6111-4224-86d9-c95fe94b9545\") " pod="models-as-a-service/payload-processing-external-model-76c77f5454-5cg5j" Nov 03 01:20:29.893930 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:20:29.893551 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e9d0a336-6111-4224-86d9-c95fe94b9545-plugins-config-volume\") pod \"payload-processing-external-model-76c77f5454-5cg5j\" (UID: \"e9d0a336-6111-4224-86d9-c95fe94b9545\") " pod="models-as-a-service/payload-processing-external-model-76c77f5454-5cg5j" Nov 03 01:20:29.893930 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:20:29.893899 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/e9d0a336-6111-4224-86d9-c95fe94b9545-routing-overlay-volume\") pod \"payload-processing-external-model-76c77f5454-5cg5j\" (UID: \"e9d0a336-6111-4224-86d9-c95fe94b9545\") " pod="models-as-a-service/payload-processing-external-model-76c77f5454-5cg5j" Nov 03 01:20:29.995540 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:20:29.995495 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/e9d0a336-6111-4224-86d9-c95fe94b9545-routing-overlay-volume\") pod \"payload-processing-external-model-76c77f5454-5cg5j\" (UID: \"e9d0a336-6111-4224-86d9-c95fe94b9545\") " pod="models-as-a-service/payload-processing-external-model-76c77f5454-5cg5j" Nov 03 01:20:29.995800 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:20:29.995652 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/e9d0a336-6111-4224-86d9-c95fe94b9545-provider-credentials\") pod \"payload-processing-external-model-76c77f5454-5cg5j\" (UID: \"e9d0a336-6111-4224-86d9-c95fe94b9545\") " pod="models-as-a-service/payload-processing-external-model-76c77f5454-5cg5j" Nov 03 01:20:29.996226 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:20:29.995766 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qfg9p\" (UniqueName: \"kubernetes.io/projected/e9d0a336-6111-4224-86d9-c95fe94b9545-kube-api-access-qfg9p\") pod \"payload-processing-external-model-76c77f5454-5cg5j\" (UID: \"e9d0a336-6111-4224-86d9-c95fe94b9545\") " pod="models-as-a-service/payload-processing-external-model-76c77f5454-5cg5j" Nov 03 01:20:29.996333 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:20:29.996315 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/e9d0a336-6111-4224-86d9-c95fe94b9545-routing-overlay-volume\") pod \"payload-processing-external-model-76c77f5454-5cg5j\" (UID: \"e9d0a336-6111-4224-86d9-c95fe94b9545\") " pod="models-as-a-service/payload-processing-external-model-76c77f5454-5cg5j" Nov 03 01:20:29.996407 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:20:29.996389 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e9d0a336-6111-4224-86d9-c95fe94b9545-plugins-config-volume\") pod \"payload-processing-external-model-76c77f5454-5cg5j\" (UID: \"e9d0a336-6111-4224-86d9-c95fe94b9545\") " pod="models-as-a-service/payload-processing-external-model-76c77f5454-5cg5j" Nov 03 01:20:29.997127 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:20:29.997101 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e9d0a336-6111-4224-86d9-c95fe94b9545-plugins-config-volume\") pod \"payload-processing-external-model-76c77f5454-5cg5j\" (UID: \"e9d0a336-6111-4224-86d9-c95fe94b9545\") " pod="models-as-a-service/payload-processing-external-model-76c77f5454-5cg5j" Nov 03 01:20:29.998554 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:20:29.998530 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/e9d0a336-6111-4224-86d9-c95fe94b9545-provider-credentials\") pod \"payload-processing-external-model-76c77f5454-5cg5j\" (UID: \"e9d0a336-6111-4224-86d9-c95fe94b9545\") " pod="models-as-a-service/payload-processing-external-model-76c77f5454-5cg5j" Nov 03 01:20:30.006341 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:20:30.006312 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qfg9p\" (UniqueName: \"kubernetes.io/projected/e9d0a336-6111-4224-86d9-c95fe94b9545-kube-api-access-qfg9p\") pod \"payload-processing-external-model-76c77f5454-5cg5j\" (UID: \"e9d0a336-6111-4224-86d9-c95fe94b9545\") " pod="models-as-a-service/payload-processing-external-model-76c77f5454-5cg5j" Nov 03 01:20:30.104096 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:20:30.103889 4939 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-821f5a/payload-processing-external-model-e2e-extmod-a-821f5a-5df8cj84n"] Nov 03 01:20:30.104096 ip-10-0-140-107 kubenswrapper[4939]: W1103 01:20:30.104022 4939 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5806cd37_5cdf_4afe_aa0b_c16b09db3106.slice/crio-2e7d75f0f17b35e331da5703bde1edff059e4e766607cb62d0c70b0afe8e6464 WatchSource:0}: Error finding container 2e7d75f0f17b35e331da5703bde1edff059e4e766607cb62d0c70b0afe8e6464: Status 404 returned error can't find the container with id 2e7d75f0f17b35e331da5703bde1edff059e4e766607cb62d0c70b0afe8e6464 Nov 03 01:20:30.179237 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:20:30.179193 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:20:30.182356 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:20:30.182329 4939 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-76c77f5454-5cg5j" Nov 03 01:20:30.310283 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:20:30.310244 4939 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-59dbb77fb8-5h2hn" Nov 03 01:20:30.315153 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:20:30.312735 4939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-821f5a/payload-processing-external-model-e2e-extmod-a-821f5a-5df8cj84n" event={"ID":"5806cd37-5cdf-4afe-aa0b-c16b09db3106","Type":"ContainerStarted","Data":"a72d9e520390deaf772ad83182d18514d7c6bf327f1f12e4176ad76fcc1732bc"} Nov 03 01:20:30.315153 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:20:30.313605 4939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-821f5a/payload-processing-external-model-e2e-extmod-a-821f5a-5df8cj84n" event={"ID":"5806cd37-5cdf-4afe-aa0b-c16b09db3106","Type":"ContainerStarted","Data":"2e7d75f0f17b35e331da5703bde1edff059e4e766607cb62d0c70b0afe8e6464"} Nov 03 01:20:30.315153 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:20:30.313655 4939 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-a-821f5a/payload-processing-external-model-e2e-extmod-a-821f5a-5df8cj84n" Nov 03 01:20:30.326245 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:20:30.326216 4939 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-59dbb77fb8-5h2hn" Nov 03 01:20:30.388205 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:20:30.388085 4939 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-a-821f5a/payload-processing-external-model-e2e-extmod-a-821f5a-5df8cj84n" podStartSLOduration=1.388065558 podStartE2EDuration="1.388065558s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 01:20:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 01:20:30.386262016 +0000 UTC m=+2378.650596962" watchObservedRunningTime="2026-11-03 01:20:30.388065558 +0000 UTC m=+2378.652400506" Nov 03 01:20:30.420232 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:20:30.420194 4939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c0272368-d591-4cae-ace9-be49c71471f3-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c0272368-d591-4cae-ace9-be49c71471f3-plugins-config-volume\") pod \"c0272368-d591-4cae-ace9-be49c71471f3\" (UID: \"c0272368-d591-4cae-ace9-be49c71471f3\") " Nov 03 01:20:30.420627 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:20:30.420582 4939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c0272368-d591-4cae-ace9-be49c71471f3-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/c0272368-d591-4cae-ace9-be49c71471f3-routing-overlay-volume\") pod \"c0272368-d591-4cae-ace9-be49c71471f3\" (UID: \"c0272368-d591-4cae-ace9-be49c71471f3\") " Nov 03 01:20:30.420774 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:20:30.420641 4939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c0272368-d591-4cae-ace9-be49c71471f3-kube-api-access-qs7cw\" (UniqueName: \"kubernetes.io/projected/c0272368-d591-4cae-ace9-be49c71471f3-kube-api-access-qs7cw\") pod \"c0272368-d591-4cae-ace9-be49c71471f3\" (UID: \"c0272368-d591-4cae-ace9-be49c71471f3\") " Nov 03 01:20:30.420774 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:20:30.420714 4939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c0272368-d591-4cae-ace9-be49c71471f3-plugins-config-volume" pod "c0272368-d591-4cae-ace9-be49c71471f3" (UID: "c0272368-d591-4cae-ace9-be49c71471f3"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 01:20:30.421105 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:20:30.421075 4939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c0272368-d591-4cae-ace9-be49c71471f3-routing-overlay-volume" pod "c0272368-d591-4cae-ace9-be49c71471f3" (UID: "c0272368-d591-4cae-ace9-be49c71471f3"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 01:20:30.422271 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:20:30.422240 4939 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c0272368-d591-4cae-ace9-be49c71471f3-plugins-config-volume\") on node \"ip-10-0-140-107.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:20:30.422396 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:20:30.422273 4939 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/c0272368-d591-4cae-ace9-be49c71471f3-routing-overlay-volume\") on node \"ip-10-0-140-107.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:20:30.423281 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:20:30.423256 4939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c0272368-d591-4cae-ace9-be49c71471f3-kube-api-access-qs7cw" pod "c0272368-d591-4cae-ace9-be49c71471f3" (UID: "c0272368-d591-4cae-ace9-be49c71471f3"). InnerVolumeSpecName "kube-api-access-qs7cw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 01:20:30.527408 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:20:30.527316 4939 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qs7cw\" (UniqueName: \"kubernetes.io/projected/c0272368-d591-4cae-ace9-be49c71471f3-kube-api-access-qs7cw\") on node \"ip-10-0-140-107.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:20:30.616460 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:20:30.616418 4939 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["models-as-a-service/payload-processing-external-model-76c77f5454-5cg5j"] Nov 03 01:20:30.621192 ip-10-0-140-107 kubenswrapper[4939]: W1103 01:20:30.621156 4939 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode9d0a336_6111_4224_86d9_c95fe94b9545.slice/crio-8967e0771fec295539a7ab8f856f673f9791d9709e981aec880a4d863a9ba19b WatchSource:0}: Error finding container 8967e0771fec295539a7ab8f856f673f9791d9709e981aec880a4d863a9ba19b: Status 404 returned error can't find the container with id 8967e0771fec295539a7ab8f856f673f9791d9709e981aec880a4d863a9ba19b Nov 03 01:20:30.730344 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:20:30.730305 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/c0272368-d591-4cae-ace9-be49c71471f3-provider-credentials\") pod \"payload-processing-external-model-59dbb77fb8-5h2hn\" (UID: \"c0272368-d591-4cae-ace9-be49c71471f3\") " pod="models-as-a-service/payload-processing-external-model-59dbb77fb8-5h2hn" Nov 03 01:20:30.730507 ip-10-0-140-107 kubenswrapper[4939]: E1103 01:20:30.730455 4939 projected.go:266] Couldn't get secret models-as-a-service/agc-cred-7379d3751ff2: secret "agc-cred-7379d3751ff2" not found Nov 03 01:20:30.730507 ip-10-0-140-107 kubenswrapper[4939]: E1103 01:20:30.730484 4939 projected.go:196] Error preparing data for projected volume provider-credentials for pod models-as-a-service/payload-processing-external-model-59dbb77fb8-5h2hn: secret "agc-cred-7379d3751ff2" not found Nov 03 01:20:30.730579 ip-10-0-140-107 kubenswrapper[4939]: E1103 01:20:30.730540 4939 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/c0272368-d591-4cae-ace9-be49c71471f3-provider-credentials podName:c0272368-d591-4cae-ace9-be49c71471f3 nodeName:}" failed. No retries permitted until 2026-11-03 01:20:32.730525999 +0000 UTC m=+2380.994860922 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/c0272368-d591-4cae-ace9-be49c71471f3-provider-credentials") pod "payload-processing-external-model-59dbb77fb8-5h2hn" (UID: "c0272368-d591-4cae-ace9-be49c71471f3") : secret "agc-cred-7379d3751ff2" not found Nov 03 01:20:31.089445 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:20:31.089386 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:20:31.168866 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:20:31.160679 4939 kuberuntime_container.go:898] "Killing container with a grace period" pod="models-as-a-service/payload-processing-external-model-5774fbf4b9-pl5dw" podUID="5035296e-642f-420d-ade7-d9c36030cca3" containerName="payload-processing" containerID="cri-o://46fca909cdf142df420f0590df23aaac6daf57809cab88af38cdee004834f4c8" gracePeriod=25 Nov 03 01:20:31.323052 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:20:31.323017 4939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-76c77f5454-5cg5j" event={"ID":"e9d0a336-6111-4224-86d9-c95fe94b9545","Type":"ContainerStarted","Data":"635c7c99a38e1d77efed13c4dcc77b0a0b6eaa8589018df5632d637efeaed3e1"} Nov 03 01:20:31.323249 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:20:31.323073 4939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-76c77f5454-5cg5j" event={"ID":"e9d0a336-6111-4224-86d9-c95fe94b9545","Type":"ContainerStarted","Data":"8967e0771fec295539a7ab8f856f673f9791d9709e981aec880a4d863a9ba19b"} Nov 03 01:20:31.323249 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:20:31.323116 4939 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="models-as-a-service/payload-processing-external-model-76c77f5454-5cg5j" Nov 03 01:20:31.324810 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:20:31.324778 4939 generic.go:356] "Generic (PLEG): container finished" podID="5035296e-642f-420d-ade7-d9c36030cca3" containerID="46fca909cdf142df420f0590df23aaac6daf57809cab88af38cdee004834f4c8" exitCode=0 Nov 03 01:20:31.324978 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:20:31.324854 4939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-5774fbf4b9-pl5dw" event={"ID":"5035296e-642f-420d-ade7-d9c36030cca3","Type":"ContainerDied","Data":"46fca909cdf142df420f0590df23aaac6daf57809cab88af38cdee004834f4c8"} Nov 03 01:20:31.325143 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:20:31.325124 4939 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-59dbb77fb8-5h2hn" Nov 03 01:20:31.350366 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:20:31.348178 4939 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="models-as-a-service/payload-processing-external-model-76c77f5454-5cg5j" podStartSLOduration=2.348157268 podStartE2EDuration="2.348157268s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 01:20:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 01:20:31.340741118 +0000 UTC m=+2379.605076075" watchObservedRunningTime="2026-11-03 01:20:31.348157268 +0000 UTC m=+2379.612492214" Nov 03 01:20:31.378966 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:20:31.378186 4939 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["models-as-a-service/payload-processing-external-model-59dbb77fb8-5h2hn"] Nov 03 01:20:31.404365 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:20:31.404325 4939 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["models-as-a-service/payload-processing-external-model-59dbb77fb8-5h2hn"] Nov 03 01:20:31.549121 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:20:31.549081 4939 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/c0272368-d591-4cae-ace9-be49c71471f3-provider-credentials\") on node \"ip-10-0-140-107.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:20:31.561753 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:20:31.561727 4939 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-5774fbf4b9-pl5dw" Nov 03 01:20:31.649950 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:20:31.649833 4939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5035296e-642f-420d-ade7-d9c36030cca3-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/5035296e-642f-420d-ade7-d9c36030cca3-routing-overlay-volume\") pod \"5035296e-642f-420d-ade7-d9c36030cca3\" (UID: \"5035296e-642f-420d-ade7-d9c36030cca3\") " Nov 03 01:20:31.650282 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:20:31.650259 4939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5035296e-642f-420d-ade7-d9c36030cca3-provider-credentials\" (UniqueName: \"kubernetes.io/projected/5035296e-642f-420d-ade7-d9c36030cca3-provider-credentials\") pod \"5035296e-642f-420d-ade7-d9c36030cca3\" (UID: \"5035296e-642f-420d-ade7-d9c36030cca3\") " Nov 03 01:20:31.650346 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:20:31.650277 4939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5035296e-642f-420d-ade7-d9c36030cca3-routing-overlay-volume" pod "5035296e-642f-420d-ade7-d9c36030cca3" (UID: "5035296e-642f-420d-ade7-d9c36030cca3"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 01:20:31.650413 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:20:31.650363 4939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5035296e-642f-420d-ade7-d9c36030cca3-kube-api-access-m2ks5\" (UniqueName: \"kubernetes.io/projected/5035296e-642f-420d-ade7-d9c36030cca3-kube-api-access-m2ks5\") pod \"5035296e-642f-420d-ade7-d9c36030cca3\" (UID: \"5035296e-642f-420d-ade7-d9c36030cca3\") " Nov 03 01:20:31.650488 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:20:31.650472 4939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5035296e-642f-420d-ade7-d9c36030cca3-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5035296e-642f-420d-ade7-d9c36030cca3-plugins-config-volume\") pod \"5035296e-642f-420d-ade7-d9c36030cca3\" (UID: \"5035296e-642f-420d-ade7-d9c36030cca3\") " Nov 03 01:20:31.650975 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:20:31.650945 4939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5035296e-642f-420d-ade7-d9c36030cca3-plugins-config-volume" pod "5035296e-642f-420d-ade7-d9c36030cca3" (UID: "5035296e-642f-420d-ade7-d9c36030cca3"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 01:20:31.651517 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:20:31.651490 4939 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5035296e-642f-420d-ade7-d9c36030cca3-plugins-config-volume\") on node \"ip-10-0-140-107.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:20:31.651633 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:20:31.651523 4939 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/5035296e-642f-420d-ade7-d9c36030cca3-routing-overlay-volume\") on node \"ip-10-0-140-107.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:20:31.653402 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:20:31.653376 4939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5035296e-642f-420d-ade7-d9c36030cca3-provider-credentials" pod "5035296e-642f-420d-ade7-d9c36030cca3" (UID: "5035296e-642f-420d-ade7-d9c36030cca3"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 01:20:31.653543 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:20:31.653418 4939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5035296e-642f-420d-ade7-d9c36030cca3-kube-api-access-m2ks5" pod "5035296e-642f-420d-ade7-d9c36030cca3" (UID: "5035296e-642f-420d-ade7-d9c36030cca3"). InnerVolumeSpecName "kube-api-access-m2ks5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 01:20:31.753554 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:20:31.753515 4939 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/5035296e-642f-420d-ade7-d9c36030cca3-provider-credentials\") on node \"ip-10-0-140-107.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:20:31.753554 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:20:31.753550 4939 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m2ks5\" (UniqueName: \"kubernetes.io/projected/5035296e-642f-420d-ade7-d9c36030cca3-kube-api-access-m2ks5\") on node \"ip-10-0-140-107.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:20:31.873484 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:20:31.873449 4939 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c0272368-d591-4cae-ace9-be49c71471f3" path="/var/lib/kubelet/pods/c0272368-d591-4cae-ace9-be49c71471f3/volumes" Nov 03 01:20:31.894686 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:20:31.894646 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:20:32.015145 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:20:32.012564 4939 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-vm5w5"] Nov 03 01:20:32.018463 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:20:32.017881 4939 cpu_manager.go:414] "containerMap: removing container" podUID="5035296e-642f-420d-ade7-d9c36030cca3" containerName="payload-processing" Nov 03 01:20:32.018463 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:20:32.017927 4939 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5035296e-642f-420d-ade7-d9c36030cca3" containerName="payload-processing" Nov 03 01:20:32.018463 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:20:32.018266 4939 memory_manager.go:365] "RemoveStaleState removing state" podUID="5035296e-642f-420d-ade7-d9c36030cca3" containerName="payload-processing" Nov 03 01:20:32.027959 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:20:32.027929 4939 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-vm5w5" Nov 03 01:20:32.034447 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:20:32.034421 4939 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-r2cgw"] Nov 03 01:20:32.040869 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:20:32.040831 4939 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-vm5w5"] Nov 03 01:20:32.041183 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:20:32.041170 4939 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-r2cgw" Nov 03 01:20:32.048983 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:20:32.048801 4939 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-r2cgw"] Nov 03 01:20:32.162128 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:20:32.162093 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9a335557-68f1-44ab-b673-f9e0a18be917-istio-podinfo\") pod \"maas-default-gateway-openshift-default-596dc9545f-vm5w5\" (UID: \"9a335557-68f1-44ab-b673-f9e0a18be917\") " pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-vm5w5" Nov 03 01:20:32.162128 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:20:32.162130 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/20419baf-ec7d-47c2-ad77-5f32d6d93992-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-596dc9545f-r2cgw\" (UID: \"20419baf-ec7d-47c2-ad77-5f32d6d93992\") " pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-r2cgw" Nov 03 01:20:32.162675 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:20:32.162156 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9a335557-68f1-44ab-b673-f9e0a18be917-istio-token\") pod \"maas-default-gateway-openshift-default-596dc9545f-vm5w5\" (UID: \"9a335557-68f1-44ab-b673-f9e0a18be917\") " pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-vm5w5" Nov 03 01:20:32.162675 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:20:32.162171 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/20419baf-ec7d-47c2-ad77-5f32d6d93992-istio-token\") pod \"maas-default-gateway-openshift-default-596dc9545f-r2cgw\" (UID: \"20419baf-ec7d-47c2-ad77-5f32d6d93992\") " pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-r2cgw" Nov 03 01:20:32.162675 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:20:32.162260 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9a335557-68f1-44ab-b673-f9e0a18be917-workload-certs\") pod \"maas-default-gateway-openshift-default-596dc9545f-vm5w5\" (UID: \"9a335557-68f1-44ab-b673-f9e0a18be917\") " pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-vm5w5" Nov 03 01:20:32.162675 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:20:32.162463 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9a335557-68f1-44ab-b673-f9e0a18be917-istio-data\") pod \"maas-default-gateway-openshift-default-596dc9545f-vm5w5\" (UID: \"9a335557-68f1-44ab-b673-f9e0a18be917\") " pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-vm5w5" Nov 03 01:20:32.162675 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:20:32.162521 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9a335557-68f1-44ab-b673-f9e0a18be917-istio-envoy\") pod \"maas-default-gateway-openshift-default-596dc9545f-vm5w5\" (UID: \"9a335557-68f1-44ab-b673-f9e0a18be917\") " pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-vm5w5" Nov 03 01:20:32.162675 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:20:32.162578 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/20419baf-ec7d-47c2-ad77-5f32d6d93992-istio-envoy\") pod \"maas-default-gateway-openshift-default-596dc9545f-r2cgw\" (UID: \"20419baf-ec7d-47c2-ad77-5f32d6d93992\") " pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-r2cgw" Nov 03 01:20:32.163061 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:20:32.162768 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/20419baf-ec7d-47c2-ad77-5f32d6d93992-istio-data\") pod \"maas-default-gateway-openshift-default-596dc9545f-r2cgw\" (UID: \"20419baf-ec7d-47c2-ad77-5f32d6d93992\") " pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-r2cgw" Nov 03 01:20:32.163127 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:20:32.163008 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/20419baf-ec7d-47c2-ad77-5f32d6d93992-workload-socket\") pod \"maas-default-gateway-openshift-default-596dc9545f-r2cgw\" (UID: \"20419baf-ec7d-47c2-ad77-5f32d6d93992\") " pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-r2cgw" Nov 03 01:20:32.163183 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:20:32.163148 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9a335557-68f1-44ab-b673-f9e0a18be917-credential-socket\") pod \"maas-default-gateway-openshift-default-596dc9545f-vm5w5\" (UID: \"9a335557-68f1-44ab-b673-f9e0a18be917\") " pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-vm5w5" Nov 03 01:20:32.163322 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:20:32.163293 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/20419baf-ec7d-47c2-ad77-5f32d6d93992-credential-socket\") pod \"maas-default-gateway-openshift-default-596dc9545f-r2cgw\" (UID: \"20419baf-ec7d-47c2-ad77-5f32d6d93992\") " pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-r2cgw" Nov 03 01:20:32.163396 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:20:32.163350 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-chnk5\" (UniqueName: \"kubernetes.io/projected/9a335557-68f1-44ab-b673-f9e0a18be917-kube-api-access-chnk5\") pod \"maas-default-gateway-openshift-default-596dc9545f-vm5w5\" (UID: \"9a335557-68f1-44ab-b673-f9e0a18be917\") " pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-vm5w5" Nov 03 01:20:32.163447 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:20:32.163394 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/20419baf-ec7d-47c2-ad77-5f32d6d93992-workload-certs\") pod \"maas-default-gateway-openshift-default-596dc9545f-r2cgw\" (UID: \"20419baf-ec7d-47c2-ad77-5f32d6d93992\") " pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-r2cgw" Nov 03 01:20:32.163514 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:20:32.163489 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/20419baf-ec7d-47c2-ad77-5f32d6d93992-istio-podinfo\") pod \"maas-default-gateway-openshift-default-596dc9545f-r2cgw\" (UID: \"20419baf-ec7d-47c2-ad77-5f32d6d93992\") " pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-r2cgw" Nov 03 01:20:32.163622 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:20:32.163575 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9a335557-68f1-44ab-b673-f9e0a18be917-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-596dc9545f-vm5w5\" (UID: \"9a335557-68f1-44ab-b673-f9e0a18be917\") " pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-vm5w5" Nov 03 01:20:32.163706 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:20:32.163691 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g6sk2\" (UniqueName: \"kubernetes.io/projected/20419baf-ec7d-47c2-ad77-5f32d6d93992-kube-api-access-g6sk2\") pod \"maas-default-gateway-openshift-default-596dc9545f-r2cgw\" (UID: \"20419baf-ec7d-47c2-ad77-5f32d6d93992\") " pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-r2cgw" Nov 03 01:20:32.163743 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:20:32.163719 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9a335557-68f1-44ab-b673-f9e0a18be917-workload-socket\") pod \"maas-default-gateway-openshift-default-596dc9545f-vm5w5\" (UID: \"9a335557-68f1-44ab-b673-f9e0a18be917\") " pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-vm5w5" Nov 03 01:20:32.265534 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:20:32.265440 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/20419baf-ec7d-47c2-ad77-5f32d6d93992-istio-data\") pod \"maas-default-gateway-openshift-default-596dc9545f-r2cgw\" (UID: \"20419baf-ec7d-47c2-ad77-5f32d6d93992\") " pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-r2cgw" Nov 03 01:20:32.265698 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:20:32.265566 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/20419baf-ec7d-47c2-ad77-5f32d6d93992-workload-socket\") pod \"maas-default-gateway-openshift-default-596dc9545f-r2cgw\" (UID: \"20419baf-ec7d-47c2-ad77-5f32d6d93992\") " pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-r2cgw" Nov 03 01:20:32.265698 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:20:32.265623 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9a335557-68f1-44ab-b673-f9e0a18be917-credential-socket\") pod \"maas-default-gateway-openshift-default-596dc9545f-vm5w5\" (UID: \"9a335557-68f1-44ab-b673-f9e0a18be917\") " pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-vm5w5" Nov 03 01:20:32.265698 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:20:32.265657 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/20419baf-ec7d-47c2-ad77-5f32d6d93992-credential-socket\") pod \"maas-default-gateway-openshift-default-596dc9545f-r2cgw\" (UID: \"20419baf-ec7d-47c2-ad77-5f32d6d93992\") " pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-r2cgw" Nov 03 01:20:32.265698 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:20:32.265690 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-chnk5\" (UniqueName: \"kubernetes.io/projected/9a335557-68f1-44ab-b673-f9e0a18be917-kube-api-access-chnk5\") pod \"maas-default-gateway-openshift-default-596dc9545f-vm5w5\" (UID: \"9a335557-68f1-44ab-b673-f9e0a18be917\") " pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-vm5w5" Nov 03 01:20:32.265953 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:20:32.265722 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/20419baf-ec7d-47c2-ad77-5f32d6d93992-workload-certs\") pod \"maas-default-gateway-openshift-default-596dc9545f-r2cgw\" (UID: \"20419baf-ec7d-47c2-ad77-5f32d6d93992\") " pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-r2cgw" Nov 03 01:20:32.265953 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:20:32.265782 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/20419baf-ec7d-47c2-ad77-5f32d6d93992-istio-podinfo\") pod \"maas-default-gateway-openshift-default-596dc9545f-r2cgw\" (UID: \"20419baf-ec7d-47c2-ad77-5f32d6d93992\") " pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-r2cgw" Nov 03 01:20:32.265953 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:20:32.265817 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9a335557-68f1-44ab-b673-f9e0a18be917-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-596dc9545f-vm5w5\" (UID: \"9a335557-68f1-44ab-b673-f9e0a18be917\") " pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-vm5w5" Nov 03 01:20:32.265953 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:20:32.265921 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g6sk2\" (UniqueName: \"kubernetes.io/projected/20419baf-ec7d-47c2-ad77-5f32d6d93992-kube-api-access-g6sk2\") pod \"maas-default-gateway-openshift-default-596dc9545f-r2cgw\" (UID: \"20419baf-ec7d-47c2-ad77-5f32d6d93992\") " pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-r2cgw" Nov 03 01:20:32.266210 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:20:32.265958 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9a335557-68f1-44ab-b673-f9e0a18be917-workload-socket\") pod \"maas-default-gateway-openshift-default-596dc9545f-vm5w5\" (UID: \"9a335557-68f1-44ab-b673-f9e0a18be917\") " pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-vm5w5" Nov 03 01:20:32.266210 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:20:32.265974 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/20419baf-ec7d-47c2-ad77-5f32d6d93992-workload-socket\") pod \"maas-default-gateway-openshift-default-596dc9545f-r2cgw\" (UID: \"20419baf-ec7d-47c2-ad77-5f32d6d93992\") " pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-r2cgw" Nov 03 01:20:32.266210 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:20:32.265956 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/20419baf-ec7d-47c2-ad77-5f32d6d93992-istio-data\") pod \"maas-default-gateway-openshift-default-596dc9545f-r2cgw\" (UID: \"20419baf-ec7d-47c2-ad77-5f32d6d93992\") " pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-r2cgw" Nov 03 01:20:32.266210 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:20:32.266050 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9a335557-68f1-44ab-b673-f9e0a18be917-istio-podinfo\") pod \"maas-default-gateway-openshift-default-596dc9545f-vm5w5\" (UID: \"9a335557-68f1-44ab-b673-f9e0a18be917\") " pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-vm5w5" Nov 03 01:20:32.266210 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:20:32.266079 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/20419baf-ec7d-47c2-ad77-5f32d6d93992-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-596dc9545f-r2cgw\" (UID: \"20419baf-ec7d-47c2-ad77-5f32d6d93992\") " pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-r2cgw" Nov 03 01:20:32.266210 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:20:32.266138 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9a335557-68f1-44ab-b673-f9e0a18be917-credential-socket\") pod \"maas-default-gateway-openshift-default-596dc9545f-vm5w5\" (UID: \"9a335557-68f1-44ab-b673-f9e0a18be917\") " pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-vm5w5" Nov 03 01:20:32.266210 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:20:32.266171 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9a335557-68f1-44ab-b673-f9e0a18be917-istio-token\") pod \"maas-default-gateway-openshift-default-596dc9545f-vm5w5\" (UID: \"9a335557-68f1-44ab-b673-f9e0a18be917\") " pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-vm5w5" Nov 03 01:20:32.266210 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:20:32.266202 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/20419baf-ec7d-47c2-ad77-5f32d6d93992-istio-token\") pod \"maas-default-gateway-openshift-default-596dc9545f-r2cgw\" (UID: \"20419baf-ec7d-47c2-ad77-5f32d6d93992\") " pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-r2cgw" Nov 03 01:20:32.266636 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:20:32.266241 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/20419baf-ec7d-47c2-ad77-5f32d6d93992-workload-certs\") pod \"maas-default-gateway-openshift-default-596dc9545f-r2cgw\" (UID: \"20419baf-ec7d-47c2-ad77-5f32d6d93992\") " pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-r2cgw" Nov 03 01:20:32.266636 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:20:32.266271 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9a335557-68f1-44ab-b673-f9e0a18be917-workload-certs\") pod \"maas-default-gateway-openshift-default-596dc9545f-vm5w5\" (UID: \"9a335557-68f1-44ab-b673-f9e0a18be917\") " pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-vm5w5" Nov 03 01:20:32.266636 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:20:32.266318 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9a335557-68f1-44ab-b673-f9e0a18be917-istio-data\") pod \"maas-default-gateway-openshift-default-596dc9545f-vm5w5\" (UID: \"9a335557-68f1-44ab-b673-f9e0a18be917\") " pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-vm5w5" Nov 03 01:20:32.266636 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:20:32.266356 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9a335557-68f1-44ab-b673-f9e0a18be917-istio-envoy\") pod \"maas-default-gateway-openshift-default-596dc9545f-vm5w5\" (UID: \"9a335557-68f1-44ab-b673-f9e0a18be917\") " pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-vm5w5" Nov 03 01:20:32.266636 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:20:32.266388 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/20419baf-ec7d-47c2-ad77-5f32d6d93992-istio-envoy\") pod \"maas-default-gateway-openshift-default-596dc9545f-r2cgw\" (UID: \"20419baf-ec7d-47c2-ad77-5f32d6d93992\") " pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-r2cgw" Nov 03 01:20:32.266636 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:20:32.266458 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9a335557-68f1-44ab-b673-f9e0a18be917-workload-socket\") pod \"maas-default-gateway-openshift-default-596dc9545f-vm5w5\" (UID: \"9a335557-68f1-44ab-b673-f9e0a18be917\") " pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-vm5w5" Nov 03 01:20:32.266636 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:20:32.266512 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/20419baf-ec7d-47c2-ad77-5f32d6d93992-credential-socket\") pod \"maas-default-gateway-openshift-default-596dc9545f-r2cgw\" (UID: \"20419baf-ec7d-47c2-ad77-5f32d6d93992\") " pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-r2cgw" Nov 03 01:20:32.267042 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:20:32.266704 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9a335557-68f1-44ab-b673-f9e0a18be917-istio-data\") pod \"maas-default-gateway-openshift-default-596dc9545f-vm5w5\" (UID: \"9a335557-68f1-44ab-b673-f9e0a18be917\") " pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-vm5w5" Nov 03 01:20:32.267042 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:20:32.266761 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9a335557-68f1-44ab-b673-f9e0a18be917-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-596dc9545f-vm5w5\" (UID: \"9a335557-68f1-44ab-b673-f9e0a18be917\") " pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-vm5w5" Nov 03 01:20:32.267042 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:20:32.266904 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/20419baf-ec7d-47c2-ad77-5f32d6d93992-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-596dc9545f-r2cgw\" (UID: \"20419baf-ec7d-47c2-ad77-5f32d6d93992\") " pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-r2cgw" Nov 03 01:20:32.267042 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:20:32.266960 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9a335557-68f1-44ab-b673-f9e0a18be917-workload-certs\") pod \"maas-default-gateway-openshift-default-596dc9545f-vm5w5\" (UID: \"9a335557-68f1-44ab-b673-f9e0a18be917\") " pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-vm5w5" Nov 03 01:20:32.268867 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:20:32.268834 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/20419baf-ec7d-47c2-ad77-5f32d6d93992-istio-envoy\") pod \"maas-default-gateway-openshift-default-596dc9545f-r2cgw\" (UID: \"20419baf-ec7d-47c2-ad77-5f32d6d93992\") " pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-r2cgw" Nov 03 01:20:32.269103 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:20:32.269081 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9a335557-68f1-44ab-b673-f9e0a18be917-istio-envoy\") pod \"maas-default-gateway-openshift-default-596dc9545f-vm5w5\" (UID: \"9a335557-68f1-44ab-b673-f9e0a18be917\") " pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-vm5w5" Nov 03 01:20:32.269183 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:20:32.269148 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9a335557-68f1-44ab-b673-f9e0a18be917-istio-podinfo\") pod \"maas-default-gateway-openshift-default-596dc9545f-vm5w5\" (UID: \"9a335557-68f1-44ab-b673-f9e0a18be917\") " pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-vm5w5" Nov 03 01:20:32.269355 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:20:32.269333 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/20419baf-ec7d-47c2-ad77-5f32d6d93992-istio-podinfo\") pod \"maas-default-gateway-openshift-default-596dc9545f-r2cgw\" (UID: \"20419baf-ec7d-47c2-ad77-5f32d6d93992\") " pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-r2cgw" Nov 03 01:20:32.275089 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:20:32.275059 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/20419baf-ec7d-47c2-ad77-5f32d6d93992-istio-token\") pod \"maas-default-gateway-openshift-default-596dc9545f-r2cgw\" (UID: \"20419baf-ec7d-47c2-ad77-5f32d6d93992\") " pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-r2cgw" Nov 03 01:20:32.275355 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:20:32.275331 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-g6sk2\" (UniqueName: \"kubernetes.io/projected/20419baf-ec7d-47c2-ad77-5f32d6d93992-kube-api-access-g6sk2\") pod \"maas-default-gateway-openshift-default-596dc9545f-r2cgw\" (UID: \"20419baf-ec7d-47c2-ad77-5f32d6d93992\") " pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-r2cgw" Nov 03 01:20:32.275552 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:20:32.275532 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9a335557-68f1-44ab-b673-f9e0a18be917-istio-token\") pod \"maas-default-gateway-openshift-default-596dc9545f-vm5w5\" (UID: \"9a335557-68f1-44ab-b673-f9e0a18be917\") " pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-vm5w5" Nov 03 01:20:32.275864 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:20:32.275847 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-chnk5\" (UniqueName: \"kubernetes.io/projected/9a335557-68f1-44ab-b673-f9e0a18be917-kube-api-access-chnk5\") pod \"maas-default-gateway-openshift-default-596dc9545f-vm5w5\" (UID: \"9a335557-68f1-44ab-b673-f9e0a18be917\") " pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-vm5w5" Nov 03 01:20:32.334617 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:20:32.334578 4939 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-5774fbf4b9-pl5dw" Nov 03 01:20:32.334805 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:20:32.334610 4939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-5774fbf4b9-pl5dw" event={"ID":"5035296e-642f-420d-ade7-d9c36030cca3","Type":"ContainerDied","Data":"d45ff5cc7d88fc14a4bc9da0d10505f8c36db50d3c1df7cf48ecb46875707020"} Nov 03 01:20:32.334805 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:20:32.334674 4939 scope.go:122] "RemoveContainer" containerID="46fca909cdf142df420f0590df23aaac6daf57809cab88af38cdee004834f4c8" Nov 03 01:20:32.357843 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:20:32.357809 4939 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["models-as-a-service/payload-processing-external-model-5774fbf4b9-pl5dw"] Nov 03 01:20:32.364166 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:20:32.364138 4939 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["models-as-a-service/payload-processing-external-model-5774fbf4b9-pl5dw"] Nov 03 01:20:32.375898 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:20:32.375870 4939 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-vm5w5" Nov 03 01:20:32.388781 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:20:32.388754 4939 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-r2cgw" Nov 03 01:20:32.679651 ip-10-0-140-107 kubenswrapper[4939]: W1103 01:20:32.679605 4939 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9a335557_68f1_44ab_b673_f9e0a18be917.slice/crio-af754d8e3c37eb853492a4feb2595f32fa479e072e416e7675c6e68ff777efa3 WatchSource:0}: Error finding container af754d8e3c37eb853492a4feb2595f32fa479e072e416e7675c6e68ff777efa3: Status 404 returned error can't find the container with id af754d8e3c37eb853492a4feb2595f32fa479e072e416e7675c6e68ff777efa3 Nov 03 01:20:32.681169 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:20:32.681137 4939 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-vm5w5"] Nov 03 01:20:32.683530 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:20:32.683486 4939 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 01:20:32.683622 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:20:32.683580 4939 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 01:20:32.683661 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:20:32.683629 4939 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 01:20:32.698931 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:20:32.698889 4939 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-r2cgw"] Nov 03 01:20:32.699457 ip-10-0-140-107 kubenswrapper[4939]: W1103 01:20:32.699433 4939 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod20419baf_ec7d_47c2_ad77_5f32d6d93992.slice/crio-80d92fd9a26879d7e74a7f1a6eae0622d9254f3372066ad5a006655484f95879 WatchSource:0}: Error finding container 80d92fd9a26879d7e74a7f1a6eae0622d9254f3372066ad5a006655484f95879: Status 404 returned error can't find the container with id 80d92fd9a26879d7e74a7f1a6eae0622d9254f3372066ad5a006655484f95879 Nov 03 01:20:32.703092 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:20:32.703054 4939 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 01:20:32.703193 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:20:32.703127 4939 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 01:20:32.703193 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:20:32.703164 4939 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 01:20:32.742934 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:20:32.742086 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:20:32.937304 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:20:32.937109 4939 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-f7680d-openshift-default-c4488d5c9-h5j8d"] Nov 03 01:20:32.937793 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:20:32.937758 4939 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-f7680d-openshift-default-c4488d5c9-h5j8d" podUID="4fe351d2-d66c-4cd4-8bf7-264a40d2551c" containerName="istio-proxy" containerID="cri-o://40fc2a8fbd662b472eaf98c301d6481da7424cd5121b415342c40e5637d2665b" gracePeriod=30 Nov 03 01:20:33.199407 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:20:33.199363 4939 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-83755a-openshift-default-c86f9565c-shrff"] Nov 03 01:20:33.199953 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:20:33.199594 4939 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-83755a-openshift-default-c86f9565c-shrff" podUID="30435a8a-8666-4929-b02e-02a6bb02d878" containerName="istio-proxy" containerID="cri-o://93d16e09d35069a5b5b048f130ba15a456d72265fe9958895510c5111f8a4eda" gracePeriod=30 Nov 03 01:20:33.375931 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:20:33.375868 4939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-r2cgw" event={"ID":"20419baf-ec7d-47c2-ad77-5f32d6d93992","Type":"ContainerStarted","Data":"f2912e656092ab81c03075e5fd025cb575c1c444b6d8f2ea7339e32028af741e"} Nov 03 01:20:33.376703 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:20:33.376676 4939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-r2cgw" event={"ID":"20419baf-ec7d-47c2-ad77-5f32d6d93992","Type":"ContainerStarted","Data":"80d92fd9a26879d7e74a7f1a6eae0622d9254f3372066ad5a006655484f95879"} Nov 03 01:20:33.381001 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:20:33.380964 4939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-vm5w5" event={"ID":"9a335557-68f1-44ab-b673-f9e0a18be917","Type":"ContainerStarted","Data":"826a8578ff5dfb1d46a50b4e807e3f51541d5c3eee320b16fcf1f4a58bd0e66b"} Nov 03 01:20:33.381152 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:20:33.381062 4939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-vm5w5" event={"ID":"9a335557-68f1-44ab-b673-f9e0a18be917","Type":"ContainerStarted","Data":"af754d8e3c37eb853492a4feb2595f32fa479e072e416e7675c6e68ff777efa3"} Nov 03 01:20:33.389402 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:20:33.389373 4939 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-r2cgw" Nov 03 01:20:33.391218 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:20:33.391187 4939 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-596dc9545f-r2cgw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.21:15021/healthz/ready\": dial tcp 10.132.1.21:15021: connect: connection refused" start-of-body= Nov 03 01:20:33.391340 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:20:33.391247 4939 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-r2cgw" podUID="20419baf-ec7d-47c2-ad77-5f32d6d93992" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.21:15021/healthz/ready\": dial tcp 10.132.1.21:15021: connect: connection refused" Nov 03 01:20:33.401689 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:20:33.401628 4939 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-r2cgw" podStartSLOduration=2.401609353 podStartE2EDuration="2.401609353s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 01:20:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 01:20:33.39465352 +0000 UTC m=+2381.658988466" watchObservedRunningTime="2026-11-03 01:20:33.401609353 +0000 UTC m=+2381.665944301" Nov 03 01:20:33.417190 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:20:33.417126 4939 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-vm5w5" podStartSLOduration=2.417108599 podStartE2EDuration="2.417108599s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 01:20:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 01:20:33.414226666 +0000 UTC m=+2381.678561613" watchObservedRunningTime="2026-11-03 01:20:33.417108599 +0000 UTC m=+2381.681443545" Nov 03 01:20:33.652930 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:20:33.641001 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:20:33.911617 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:20:33.911478 4939 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5035296e-642f-420d-ade7-d9c36030cca3" path="/var/lib/kubelet/pods/5035296e-642f-420d-ade7-d9c36030cca3/volumes" Nov 03 01:20:34.273996 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:20:34.270089 4939 kuberuntime_container.go:898] "Killing container with a grace period" pod="models-as-a-service/payload-processing-external-model-76c77f5454-kqvq2" podUID="84a1e602-05ea-44fa-8a3c-db2d82dea9d8" containerName="payload-processing" containerID="cri-o://5ce76f64ded4525e90504b9c6a2c5ec11b20abfd3a1fb153b538b427d5d32ba7" gracePeriod=25 Nov 03 01:20:34.321796 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:20:34.321766 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:20:34.380461 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:20:34.380407 4939 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-vm5w5" Nov 03 01:20:34.388079 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:20:34.388042 4939 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-596dc9545f-vm5w5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.20:15021/healthz/ready\": dial tcp 10.132.1.20:15021: connect: connection refused" start-of-body= Nov 03 01:20:34.388372 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:20:34.388340 4939 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-vm5w5" podUID="9a335557-68f1-44ab-b673-f9e0a18be917" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.20:15021/healthz/ready\": dial tcp 10.132.1.20:15021: connect: connection refused" Nov 03 01:20:34.391541 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:20:34.391510 4939 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-596dc9545f-r2cgw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.21:15021/healthz/ready\": dial tcp 10.132.1.21:15021: connect: connection refused" start-of-body= Nov 03 01:20:34.391744 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:20:34.391712 4939 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-r2cgw" podUID="20419baf-ec7d-47c2-ad77-5f32d6d93992" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.21:15021/healthz/ready\": dial tcp 10.132.1.21:15021: connect: connection refused" Nov 03 01:20:34.468936 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:20:34.468453 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:20:34.608499 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:20:34.595580 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:20:34.852947 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:20:34.799710 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:20:35.231133 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:20:35.214534 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:20:35.377797 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:20:35.377754 4939 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-596dc9545f-vm5w5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.20:15021/healthz/ready\": dial tcp 10.132.1.20:15021: connect: connection refused" start-of-body= Nov 03 01:20:35.378406 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:20:35.378373 4939 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-vm5w5" podUID="9a335557-68f1-44ab-b673-f9e0a18be917" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.20:15021/healthz/ready\": dial tcp 10.132.1.20:15021: connect: connection refused" Nov 03 01:20:35.401929 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:20:35.399331 4939 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-596dc9545f-r2cgw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.21:15021/healthz/ready\": dial tcp 10.132.1.21:15021: connect: connection refused" start-of-body= Nov 03 01:20:35.401929 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:20:35.399393 4939 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-r2cgw" podUID="20419baf-ec7d-47c2-ad77-5f32d6d93992" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.21:15021/healthz/ready\": dial tcp 10.132.1.21:15021: connect: connection refused" Nov 03 01:20:35.443960 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:20:35.433880 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:20:35.720938 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:20:35.720884 4939 generic.go:356] "Generic (PLEG): container finished" podID="84a1e602-05ea-44fa-8a3c-db2d82dea9d8" containerID="5ce76f64ded4525e90504b9c6a2c5ec11b20abfd3a1fb153b538b427d5d32ba7" exitCode=0 Nov 03 01:20:35.746954 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:20:35.722241 4939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-76c77f5454-kqvq2" event={"ID":"84a1e602-05ea-44fa-8a3c-db2d82dea9d8","Type":"ContainerDied","Data":"5ce76f64ded4525e90504b9c6a2c5ec11b20abfd3a1fb153b538b427d5d32ba7"} Nov 03 01:20:35.746954 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:20:35.722306 4939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-76c77f5454-kqvq2" event={"ID":"84a1e602-05ea-44fa-8a3c-db2d82dea9d8","Type":"ContainerDied","Data":"8195c852eedf010b64ca654706205cc0342c00b0b614236c1a036499e43edc5d"} Nov 03 01:20:35.746954 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:20:35.722326 4939 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="8195c852eedf010b64ca654706205cc0342c00b0b614236c1a036499e43edc5d" Nov 03 01:20:35.746954 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:20:35.746241 4939 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-76c77f5454-kqvq2" Nov 03 01:20:36.070300 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:20:36.065556 4939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/84a1e602-05ea-44fa-8a3c-db2d82dea9d8-kube-api-access-mkx6q\" (UniqueName: \"kubernetes.io/projected/84a1e602-05ea-44fa-8a3c-db2d82dea9d8-kube-api-access-mkx6q\") pod \"84a1e602-05ea-44fa-8a3c-db2d82dea9d8\" (UID: \"84a1e602-05ea-44fa-8a3c-db2d82dea9d8\") " Nov 03 01:20:36.070300 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:20:36.065845 4939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/84a1e602-05ea-44fa-8a3c-db2d82dea9d8-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/84a1e602-05ea-44fa-8a3c-db2d82dea9d8-plugins-config-volume\") pod \"84a1e602-05ea-44fa-8a3c-db2d82dea9d8\" (UID: \"84a1e602-05ea-44fa-8a3c-db2d82dea9d8\") " Nov 03 01:20:36.070300 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:20:36.066035 4939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/84a1e602-05ea-44fa-8a3c-db2d82dea9d8-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/84a1e602-05ea-44fa-8a3c-db2d82dea9d8-routing-overlay-volume\") pod \"84a1e602-05ea-44fa-8a3c-db2d82dea9d8\" (UID: \"84a1e602-05ea-44fa-8a3c-db2d82dea9d8\") " Nov 03 01:20:36.070300 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:20:36.066086 4939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/84a1e602-05ea-44fa-8a3c-db2d82dea9d8-provider-credentials\" (UniqueName: \"kubernetes.io/projected/84a1e602-05ea-44fa-8a3c-db2d82dea9d8-provider-credentials\") pod \"84a1e602-05ea-44fa-8a3c-db2d82dea9d8\" (UID: \"84a1e602-05ea-44fa-8a3c-db2d82dea9d8\") " Nov 03 01:20:36.083062 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:20:36.073488 4939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/84a1e602-05ea-44fa-8a3c-db2d82dea9d8-plugins-config-volume" pod "84a1e602-05ea-44fa-8a3c-db2d82dea9d8" (UID: "84a1e602-05ea-44fa-8a3c-db2d82dea9d8"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 01:20:36.086097 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:20:36.086066 4939 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/84a1e602-05ea-44fa-8a3c-db2d82dea9d8-plugins-config-volume\") on node \"ip-10-0-140-107.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:20:36.137984 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:20:36.135190 4939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/84a1e602-05ea-44fa-8a3c-db2d82dea9d8-kube-api-access-mkx6q" pod "84a1e602-05ea-44fa-8a3c-db2d82dea9d8" (UID: "84a1e602-05ea-44fa-8a3c-db2d82dea9d8"). InnerVolumeSpecName "kube-api-access-mkx6q". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 01:20:36.141306 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:20:36.141268 4939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/84a1e602-05ea-44fa-8a3c-db2d82dea9d8-provider-credentials" pod "84a1e602-05ea-44fa-8a3c-db2d82dea9d8" (UID: "84a1e602-05ea-44fa-8a3c-db2d82dea9d8"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 01:20:36.183938 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:20:36.178152 4939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/84a1e602-05ea-44fa-8a3c-db2d82dea9d8-routing-overlay-volume" pod "84a1e602-05ea-44fa-8a3c-db2d82dea9d8" (UID: "84a1e602-05ea-44fa-8a3c-db2d82dea9d8"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 01:20:36.214098 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:20:36.202553 4939 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/84a1e602-05ea-44fa-8a3c-db2d82dea9d8-routing-overlay-volume\") on node \"ip-10-0-140-107.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:20:36.214098 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:20:36.202594 4939 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/84a1e602-05ea-44fa-8a3c-db2d82dea9d8-provider-credentials\") on node \"ip-10-0-140-107.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:20:36.214098 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:20:36.208394 4939 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mkx6q\" (UniqueName: \"kubernetes.io/projected/84a1e602-05ea-44fa-8a3c-db2d82dea9d8-kube-api-access-mkx6q\") on node \"ip-10-0-140-107.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:20:36.388935 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:20:36.386347 4939 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-596dc9545f-vm5w5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.20:15021/healthz/ready\": dial tcp 10.132.1.20:15021: connect: connection refused" start-of-body= Nov 03 01:20:36.388935 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:20:36.386431 4939 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-vm5w5" podUID="9a335557-68f1-44ab-b673-f9e0a18be917" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.20:15021/healthz/ready\": dial tcp 10.132.1.20:15021: connect: connection refused" Nov 03 01:20:36.403128 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:20:36.403080 4939 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-596dc9545f-r2cgw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.21:15021/healthz/ready\": dial tcp 10.132.1.21:15021: connect: connection refused" start-of-body= Nov 03 01:20:36.403834 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:20:36.403784 4939 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-r2cgw" podUID="20419baf-ec7d-47c2-ad77-5f32d6d93992" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.21:15021/healthz/ready\": dial tcp 10.132.1.21:15021: connect: connection refused" Nov 03 01:20:36.510991 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:20:36.510936 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:20:36.616270 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:20:36.614451 4939 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["models-as-a-service/payload-processing-external-model-76c77f5454-5cg5j"] Nov 03 01:20:36.619453 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:20:36.619426 4939 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="models-as-a-service/payload-processing-external-model-76c77f5454-5cg5j" Nov 03 01:20:36.622196 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:20:36.622123 4939 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["models-as-a-service/payload-processing-external-model-59dbb77fb8-grz4z"] Nov 03 01:20:36.623682 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:20:36.623645 4939 cpu_manager.go:414] "containerMap: removing container" podUID="84a1e602-05ea-44fa-8a3c-db2d82dea9d8" containerName="payload-processing" Nov 03 01:20:36.623682 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:20:36.623675 4939 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="84a1e602-05ea-44fa-8a3c-db2d82dea9d8" containerName="payload-processing" Nov 03 01:20:36.623891 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:20:36.623880 4939 memory_manager.go:365] "RemoveStaleState removing state" podUID="84a1e602-05ea-44fa-8a3c-db2d82dea9d8" containerName="payload-processing" Nov 03 01:20:36.628486 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:20:36.628459 4939 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-59dbb77fb8-grz4z" Nov 03 01:20:36.636594 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:20:36.636230 4939 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"models-as-a-service\"/\"agc-cred-7379d3751ff2\"" Nov 03 01:20:36.640634 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:20:36.640601 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:20:36.640760 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:20:36.640694 4939 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["models-as-a-service/payload-processing-external-model-59dbb77fb8-grz4z"] Nov 03 01:20:36.725626 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:20:36.725585 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/8327ba11-84f8-4660-89a4-51433d2c3a0e-provider-credentials\") pod \"payload-processing-external-model-59dbb77fb8-grz4z\" (UID: \"8327ba11-84f8-4660-89a4-51433d2c3a0e\") " pod="models-as-a-service/payload-processing-external-model-59dbb77fb8-grz4z" Nov 03 01:20:36.725812 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:20:36.725734 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/8327ba11-84f8-4660-89a4-51433d2c3a0e-routing-overlay-volume\") pod \"payload-processing-external-model-59dbb77fb8-grz4z\" (UID: \"8327ba11-84f8-4660-89a4-51433d2c3a0e\") " pod="models-as-a-service/payload-processing-external-model-59dbb77fb8-grz4z" Nov 03 01:20:36.726218 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:20:36.726191 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fswng\" (UniqueName: \"kubernetes.io/projected/8327ba11-84f8-4660-89a4-51433d2c3a0e-kube-api-access-fswng\") pod \"payload-processing-external-model-59dbb77fb8-grz4z\" (UID: \"8327ba11-84f8-4660-89a4-51433d2c3a0e\") " pod="models-as-a-service/payload-processing-external-model-59dbb77fb8-grz4z" Nov 03 01:20:36.726397 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:20:36.726380 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8327ba11-84f8-4660-89a4-51433d2c3a0e-plugins-config-volume\") pod \"payload-processing-external-model-59dbb77fb8-grz4z\" (UID: \"8327ba11-84f8-4660-89a4-51433d2c3a0e\") " pod="models-as-a-service/payload-processing-external-model-59dbb77fb8-grz4z" Nov 03 01:20:36.735018 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:20:36.732389 4939 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-76c77f5454-kqvq2" Nov 03 01:20:36.743798 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:20:36.743765 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:20:36.976987 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:20:36.958036 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8327ba11-84f8-4660-89a4-51433d2c3a0e-plugins-config-volume\") pod \"payload-processing-external-model-59dbb77fb8-grz4z\" (UID: \"8327ba11-84f8-4660-89a4-51433d2c3a0e\") " pod="models-as-a-service/payload-processing-external-model-59dbb77fb8-grz4z" Nov 03 01:20:36.976987 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:20:36.958186 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/8327ba11-84f8-4660-89a4-51433d2c3a0e-provider-credentials\") pod \"payload-processing-external-model-59dbb77fb8-grz4z\" (UID: \"8327ba11-84f8-4660-89a4-51433d2c3a0e\") " pod="models-as-a-service/payload-processing-external-model-59dbb77fb8-grz4z" Nov 03 01:20:36.976987 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:20:36.958292 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/8327ba11-84f8-4660-89a4-51433d2c3a0e-routing-overlay-volume\") pod \"payload-processing-external-model-59dbb77fb8-grz4z\" (UID: \"8327ba11-84f8-4660-89a4-51433d2c3a0e\") " pod="models-as-a-service/payload-processing-external-model-59dbb77fb8-grz4z" Nov 03 01:20:36.976987 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:20:36.958664 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fswng\" (UniqueName: \"kubernetes.io/projected/8327ba11-84f8-4660-89a4-51433d2c3a0e-kube-api-access-fswng\") pod \"payload-processing-external-model-59dbb77fb8-grz4z\" (UID: \"8327ba11-84f8-4660-89a4-51433d2c3a0e\") " pod="models-as-a-service/payload-processing-external-model-59dbb77fb8-grz4z" Nov 03 01:20:36.976987 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:20:36.960240 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8327ba11-84f8-4660-89a4-51433d2c3a0e-plugins-config-volume\") pod \"payload-processing-external-model-59dbb77fb8-grz4z\" (UID: \"8327ba11-84f8-4660-89a4-51433d2c3a0e\") " pod="models-as-a-service/payload-processing-external-model-59dbb77fb8-grz4z" Nov 03 01:20:36.976987 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:20:36.964523 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/8327ba11-84f8-4660-89a4-51433d2c3a0e-routing-overlay-volume\") pod \"payload-processing-external-model-59dbb77fb8-grz4z\" (UID: \"8327ba11-84f8-4660-89a4-51433d2c3a0e\") " pod="models-as-a-service/payload-processing-external-model-59dbb77fb8-grz4z" Nov 03 01:20:37.007942 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:20:36.990272 4939 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["models-as-a-service/payload-processing-external-model-76c77f5454-kqvq2"] Nov 03 01:20:37.091969 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:20:37.068622 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/8327ba11-84f8-4660-89a4-51433d2c3a0e-provider-credentials\") pod \"payload-processing-external-model-59dbb77fb8-grz4z\" (UID: \"8327ba11-84f8-4660-89a4-51433d2c3a0e\") " pod="models-as-a-service/payload-processing-external-model-59dbb77fb8-grz4z" Nov 03 01:20:37.092271 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:20:37.069727 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fswng\" (UniqueName: \"kubernetes.io/projected/8327ba11-84f8-4660-89a4-51433d2c3a0e-kube-api-access-fswng\") pod \"payload-processing-external-model-59dbb77fb8-grz4z\" (UID: \"8327ba11-84f8-4660-89a4-51433d2c3a0e\") " pod="models-as-a-service/payload-processing-external-model-59dbb77fb8-grz4z" Nov 03 01:20:37.209948 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:20:37.208316 4939 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["models-as-a-service/payload-processing-external-model-76c77f5454-kqvq2"] Nov 03 01:20:37.305991 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:20:37.257879 4939 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-59dbb77fb8-grz4z" Nov 03 01:20:37.378332 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:20:37.378288 4939 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-596dc9545f-vm5w5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.20:15021/healthz/ready\": dial tcp 10.132.1.20:15021: connect: connection refused" start-of-body= Nov 03 01:20:37.378582 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:20:37.378551 4939 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-vm5w5" podUID="9a335557-68f1-44ab-b673-f9e0a18be917" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.20:15021/healthz/ready\": dial tcp 10.132.1.20:15021: connect: connection refused" Nov 03 01:20:37.397921 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:20:37.396396 4939 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-596dc9545f-r2cgw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.21:15021/healthz/ready\": dial tcp 10.132.1.21:15021: connect: connection refused" start-of-body= Nov 03 01:20:37.397921 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:20:37.396473 4939 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-r2cgw" podUID="20419baf-ec7d-47c2-ad77-5f32d6d93992" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.21:15021/healthz/ready\": dial tcp 10.132.1.21:15021: connect: connection refused" Nov 03 01:20:38.041894 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:20:38.041857 4939 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="84a1e602-05ea-44fa-8a3c-db2d82dea9d8" path="/var/lib/kubelet/pods/84a1e602-05ea-44fa-8a3c-db2d82dea9d8/volumes" Nov 03 01:20:38.314435 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:20:38.314403 4939 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["models-as-a-service/payload-processing-external-model-59dbb77fb8-grz4z"] Nov 03 01:20:38.321053 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:20:38.320787 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:20:38.418592 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:20:38.414728 4939 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-596dc9545f-vm5w5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.20:15021/healthz/ready\": dial tcp 10.132.1.20:15021: connect: connection refused" start-of-body= Nov 03 01:20:38.418592 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:20:38.414788 4939 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-vm5w5" podUID="9a335557-68f1-44ab-b673-f9e0a18be917" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.20:15021/healthz/ready\": dial tcp 10.132.1.20:15021: connect: connection refused" Nov 03 01:20:38.418592 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:20:38.415183 4939 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-596dc9545f-r2cgw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.21:15021/healthz/ready\": dial tcp 10.132.1.21:15021: connect: connection refused" start-of-body= Nov 03 01:20:38.418592 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:20:38.415221 4939 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-r2cgw" podUID="20419baf-ec7d-47c2-ad77-5f32d6d93992" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.21:15021/healthz/ready\": dial tcp 10.132.1.21:15021: connect: connection refused" Nov 03 01:20:38.429152 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:20:38.427821 4939 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-a-7c759a/payload-processing-external-model-e2e-extmod-a-7c759a-799fk2rjw"] Nov 03 01:20:38.458942 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:20:38.438860 4939 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-7c759a/payload-processing-external-model-e2e-extmod-a-7c759a-799fk2rjw"] Nov 03 01:20:38.458942 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:20:38.438995 4939 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-7c759a/payload-processing-external-model-e2e-extmod-a-7c759a-799fk2rjw" Nov 03 01:20:38.558727 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:20:38.459972 4939 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-7c759a\"/\"payload-processing-external-model-e2e--d95606a1-dockercfg-dl4kq\"" Nov 03 01:20:38.558727 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:20:38.460447 4939 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-7c759a\"/\"openshift-service-ca.crt\"" Nov 03 01:20:38.558727 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:20:38.460524 4939 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-7c759a\"/\"routing-overlay\"" Nov 03 01:20:38.558727 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:20:38.460603 4939 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-7c759a\"/\"payload-processing-external-model-plugins-e2e-extmod-a-7c759a\"" Nov 03 01:20:38.558727 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:20:38.460665 4939 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-7c759a\"/\"kube-root-ca.crt\"" Nov 03 01:20:38.558727 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:20:38.460741 4939 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-7c759a\"/\"agc-cred-be1d6e8d29d8\"" Nov 03 01:20:38.593951 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:20:38.586090 4939 patch_prober.go:29] interesting pod/e2e-worker-w2-f7680d-openshift-default-c4488d5c9-h5j8d container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.65:15021/healthz/ready\": dial tcp 10.132.0.65:15021: connect: connection refused" start-of-body= Nov 03 01:20:38.593951 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:20:38.586158 4939 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w2-f7680d-openshift-default-c4488d5c9-h5j8d" podUID="4fe351d2-d66c-4cd4-8bf7-264a40d2551c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.65:15021/healthz/ready\": dial tcp 10.132.0.65:15021: connect: connection refused" Nov 03 01:20:38.924980 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:20:38.919635 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/d239ed01-45c1-4f47-b35d-50d2b437e179-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-7c759a-799fk2rjw\" (UID: \"d239ed01-45c1-4f47-b35d-50d2b437e179\") " pod="ai-tenant-e2e-extmod-a-7c759a/payload-processing-external-model-e2e-extmod-a-7c759a-799fk2rjw" Nov 03 01:20:38.924980 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:20:38.919881 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cclzn\" (UniqueName: \"kubernetes.io/projected/d239ed01-45c1-4f47-b35d-50d2b437e179-kube-api-access-cclzn\") pod \"payload-processing-external-model-e2e-extmod-a-7c759a-799fk2rjw\" (UID: \"d239ed01-45c1-4f47-b35d-50d2b437e179\") " pod="ai-tenant-e2e-extmod-a-7c759a/payload-processing-external-model-e2e-extmod-a-7c759a-799fk2rjw" Nov 03 01:20:38.924980 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:20:38.920090 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d239ed01-45c1-4f47-b35d-50d2b437e179-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-7c759a-799fk2rjw\" (UID: \"d239ed01-45c1-4f47-b35d-50d2b437e179\") " pod="ai-tenant-e2e-extmod-a-7c759a/payload-processing-external-model-e2e-extmod-a-7c759a-799fk2rjw" Nov 03 01:20:38.924980 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:20:38.920316 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/d239ed01-45c1-4f47-b35d-50d2b437e179-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-7c759a-799fk2rjw\" (UID: \"d239ed01-45c1-4f47-b35d-50d2b437e179\") " pod="ai-tenant-e2e-extmod-a-7c759a/payload-processing-external-model-e2e-extmod-a-7c759a-799fk2rjw" Nov 03 01:20:39.102508 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:20:39.102440 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/d239ed01-45c1-4f47-b35d-50d2b437e179-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-7c759a-799fk2rjw\" (UID: \"d239ed01-45c1-4f47-b35d-50d2b437e179\") " pod="ai-tenant-e2e-extmod-a-7c759a/payload-processing-external-model-e2e-extmod-a-7c759a-799fk2rjw" Nov 03 01:20:39.103223 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:20:39.103175 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cclzn\" (UniqueName: \"kubernetes.io/projected/d239ed01-45c1-4f47-b35d-50d2b437e179-kube-api-access-cclzn\") pod \"payload-processing-external-model-e2e-extmod-a-7c759a-799fk2rjw\" (UID: \"d239ed01-45c1-4f47-b35d-50d2b437e179\") " pod="ai-tenant-e2e-extmod-a-7c759a/payload-processing-external-model-e2e-extmod-a-7c759a-799fk2rjw" Nov 03 01:20:39.103550 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:20:39.103515 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d239ed01-45c1-4f47-b35d-50d2b437e179-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-7c759a-799fk2rjw\" (UID: \"d239ed01-45c1-4f47-b35d-50d2b437e179\") " pod="ai-tenant-e2e-extmod-a-7c759a/payload-processing-external-model-e2e-extmod-a-7c759a-799fk2rjw" Nov 03 01:20:39.105032 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:20:39.104996 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d239ed01-45c1-4f47-b35d-50d2b437e179-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-7c759a-799fk2rjw\" (UID: \"d239ed01-45c1-4f47-b35d-50d2b437e179\") " pod="ai-tenant-e2e-extmod-a-7c759a/payload-processing-external-model-e2e-extmod-a-7c759a-799fk2rjw" Nov 03 01:20:39.128893 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:20:39.117816 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cclzn\" (UniqueName: \"kubernetes.io/projected/d239ed01-45c1-4f47-b35d-50d2b437e179-kube-api-access-cclzn\") pod \"payload-processing-external-model-e2e-extmod-a-7c759a-799fk2rjw\" (UID: \"d239ed01-45c1-4f47-b35d-50d2b437e179\") " pod="ai-tenant-e2e-extmod-a-7c759a/payload-processing-external-model-e2e-extmod-a-7c759a-799fk2rjw" Nov 03 01:20:39.132576 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:20:39.132546 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/d239ed01-45c1-4f47-b35d-50d2b437e179-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-7c759a-799fk2rjw\" (UID: \"d239ed01-45c1-4f47-b35d-50d2b437e179\") " pod="ai-tenant-e2e-extmod-a-7c759a/payload-processing-external-model-e2e-extmod-a-7c759a-799fk2rjw" Nov 03 01:20:39.136728 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:20:39.136700 4939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-59dbb77fb8-grz4z" event={"ID":"8327ba11-84f8-4660-89a4-51433d2c3a0e","Type":"ContainerStarted","Data":"357b34948b382451ac5bb3da7f8b4e7ce67310d9f6d87d502016c9ba915dca5f"} Nov 03 01:20:39.145693 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:20:39.145662 4939 generic.go:356] "Generic (PLEG): container finished" podID="4fe351d2-d66c-4cd4-8bf7-264a40d2551c" containerID="40fc2a8fbd662b472eaf98c301d6481da7424cd5121b415342c40e5637d2665b" exitCode=0 Nov 03 01:20:39.151896 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:20:39.145716 4939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-f7680d-openshift-default-c4488d5c9-h5j8d" event={"ID":"4fe351d2-d66c-4cd4-8bf7-264a40d2551c","Type":"ContainerDied","Data":"40fc2a8fbd662b472eaf98c301d6481da7424cd5121b415342c40e5637d2665b"} Nov 03 01:20:39.221312 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:20:39.221283 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/d239ed01-45c1-4f47-b35d-50d2b437e179-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-7c759a-799fk2rjw\" (UID: \"d239ed01-45c1-4f47-b35d-50d2b437e179\") " pod="ai-tenant-e2e-extmod-a-7c759a/payload-processing-external-model-e2e-extmod-a-7c759a-799fk2rjw" Nov 03 01:20:39.224875 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:20:39.221333 4939 generic.go:356] "Generic (PLEG): container finished" podID="30435a8a-8666-4929-b02e-02a6bb02d878" containerID="93d16e09d35069a5b5b048f130ba15a456d72265fe9958895510c5111f8a4eda" exitCode=0 Nov 03 01:20:39.225304 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:20:39.221365 4939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-83755a-openshift-default-c86f9565c-shrff" event={"ID":"30435a8a-8666-4929-b02e-02a6bb02d878","Type":"ContainerDied","Data":"93d16e09d35069a5b5b048f130ba15a456d72265fe9958895510c5111f8a4eda"} Nov 03 01:20:39.225456 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:20:39.225443 4939 scope.go:122] "RemoveContainer" containerID="84a6e26ec7fe983ad1671971374185c5be4b64c87d96563859ff05d313517926" Nov 03 01:20:39.225646 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:20:39.222100 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/d239ed01-45c1-4f47-b35d-50d2b437e179-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-7c759a-799fk2rjw\" (UID: \"d239ed01-45c1-4f47-b35d-50d2b437e179\") " pod="ai-tenant-e2e-extmod-a-7c759a/payload-processing-external-model-e2e-extmod-a-7c759a-799fk2rjw" Nov 03 01:20:39.240594 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:20:39.240530 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:20:39.319968 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:20:39.319672 4939 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-7c759a/payload-processing-external-model-e2e-extmod-a-7c759a-799fk2rjw" Nov 03 01:20:39.382379 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:20:39.382264 4939 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-596dc9545f-vm5w5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.20:15021/healthz/ready\": dial tcp 10.132.1.20:15021: connect: connection refused" start-of-body= Nov 03 01:20:39.382681 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:20:39.382633 4939 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-vm5w5" podUID="9a335557-68f1-44ab-b673-f9e0a18be917" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.20:15021/healthz/ready\": dial tcp 10.132.1.20:15021: connect: connection refused" Nov 03 01:20:39.403625 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:20:39.403590 4939 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-596dc9545f-r2cgw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.21:15021/healthz/ready\": dial tcp 10.132.1.21:15021: connect: connection refused" start-of-body= Nov 03 01:20:39.403894 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:20:39.403864 4939 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-r2cgw" podUID="20419baf-ec7d-47c2-ad77-5f32d6d93992" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.21:15021/healthz/ready\": dial tcp 10.132.1.21:15021: connect: connection refused" Nov 03 01:20:39.432822 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:20:39.431089 4939 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-f7680d-openshift-default-c4488d5c9-h5j8d" Nov 03 01:20:39.491978 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:20:39.488490 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:20:39.582178 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:20:39.582141 4939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4fe351d2-d66c-4cd4-8bf7-264a40d2551c-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4fe351d2-d66c-4cd4-8bf7-264a40d2551c-istio-data\") pod \"4fe351d2-d66c-4cd4-8bf7-264a40d2551c\" (UID: \"4fe351d2-d66c-4cd4-8bf7-264a40d2551c\") " Nov 03 01:20:39.582513 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:20:39.582488 4939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4fe351d2-d66c-4cd4-8bf7-264a40d2551c-kube-api-access-s7cqg\" (UniqueName: \"kubernetes.io/projected/4fe351d2-d66c-4cd4-8bf7-264a40d2551c-kube-api-access-s7cqg\") pod \"4fe351d2-d66c-4cd4-8bf7-264a40d2551c\" (UID: \"4fe351d2-d66c-4cd4-8bf7-264a40d2551c\") " Nov 03 01:20:39.582650 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:20:39.582636 4939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4fe351d2-d66c-4cd4-8bf7-264a40d2551c-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4fe351d2-d66c-4cd4-8bf7-264a40d2551c-workload-certs\") pod \"4fe351d2-d66c-4cd4-8bf7-264a40d2551c\" (UID: \"4fe351d2-d66c-4cd4-8bf7-264a40d2551c\") " Nov 03 01:20:39.582777 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:20:39.582764 4939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4fe351d2-d66c-4cd4-8bf7-264a40d2551c-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4fe351d2-d66c-4cd4-8bf7-264a40d2551c-istiod-ca-cert\") pod \"4fe351d2-d66c-4cd4-8bf7-264a40d2551c\" (UID: \"4fe351d2-d66c-4cd4-8bf7-264a40d2551c\") " Nov 03 01:20:39.582938 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:20:39.582925 4939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/4fe351d2-d66c-4cd4-8bf7-264a40d2551c-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4fe351d2-d66c-4cd4-8bf7-264a40d2551c-istio-podinfo\") pod \"4fe351d2-d66c-4cd4-8bf7-264a40d2551c\" (UID: \"4fe351d2-d66c-4cd4-8bf7-264a40d2551c\") " Nov 03 01:20:39.583161 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:20:39.583149 4939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4fe351d2-d66c-4cd4-8bf7-264a40d2551c-istio-token\" (UniqueName: \"kubernetes.io/projected/4fe351d2-d66c-4cd4-8bf7-264a40d2551c-istio-token\") pod \"4fe351d2-d66c-4cd4-8bf7-264a40d2551c\" (UID: \"4fe351d2-d66c-4cd4-8bf7-264a40d2551c\") " Nov 03 01:20:39.583263 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:20:39.583251 4939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4fe351d2-d66c-4cd4-8bf7-264a40d2551c-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4fe351d2-d66c-4cd4-8bf7-264a40d2551c-istio-envoy\") pod \"4fe351d2-d66c-4cd4-8bf7-264a40d2551c\" (UID: \"4fe351d2-d66c-4cd4-8bf7-264a40d2551c\") " Nov 03 01:20:39.583365 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:20:39.583351 4939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4fe351d2-d66c-4cd4-8bf7-264a40d2551c-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4fe351d2-d66c-4cd4-8bf7-264a40d2551c-credential-socket\") pod \"4fe351d2-d66c-4cd4-8bf7-264a40d2551c\" (UID: \"4fe351d2-d66c-4cd4-8bf7-264a40d2551c\") " Nov 03 01:20:39.583582 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:20:39.583568 4939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4fe351d2-d66c-4cd4-8bf7-264a40d2551c-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4fe351d2-d66c-4cd4-8bf7-264a40d2551c-workload-socket\") pod \"4fe351d2-d66c-4cd4-8bf7-264a40d2551c\" (UID: \"4fe351d2-d66c-4cd4-8bf7-264a40d2551c\") " Nov 03 01:20:39.593298 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:20:39.593260 4939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4fe351d2-d66c-4cd4-8bf7-264a40d2551c-workload-socket" pod "4fe351d2-d66c-4cd4-8bf7-264a40d2551c" (UID: "4fe351d2-d66c-4cd4-8bf7-264a40d2551c"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 01:20:39.595003 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:20:39.594975 4939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4fe351d2-d66c-4cd4-8bf7-264a40d2551c-istio-data" pod "4fe351d2-d66c-4cd4-8bf7-264a40d2551c" (UID: "4fe351d2-d66c-4cd4-8bf7-264a40d2551c"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 01:20:39.646627 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:20:39.646257 4939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4fe351d2-d66c-4cd4-8bf7-264a40d2551c-credential-socket" pod "4fe351d2-d66c-4cd4-8bf7-264a40d2551c" (UID: "4fe351d2-d66c-4cd4-8bf7-264a40d2551c"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 01:20:39.652940 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:20:39.646978 4939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4fe351d2-d66c-4cd4-8bf7-264a40d2551c-workload-certs" pod "4fe351d2-d66c-4cd4-8bf7-264a40d2551c" (UID: "4fe351d2-d66c-4cd4-8bf7-264a40d2551c"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 01:20:39.679674 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:20:39.679512 4939 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4fe351d2-d66c-4cd4-8bf7-264a40d2551c-workload-socket\") on node \"ip-10-0-140-107.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:20:39.679674 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:20:39.679553 4939 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4fe351d2-d66c-4cd4-8bf7-264a40d2551c-istio-data\") on node \"ip-10-0-140-107.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:20:39.679674 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:20:39.679570 4939 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4fe351d2-d66c-4cd4-8bf7-264a40d2551c-workload-certs\") on node \"ip-10-0-140-107.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:20:39.679674 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:20:39.679586 4939 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4fe351d2-d66c-4cd4-8bf7-264a40d2551c-credential-socket\") on node \"ip-10-0-140-107.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:20:39.726066 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:20:39.726025 4939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4fe351d2-d66c-4cd4-8bf7-264a40d2551c-istiod-ca-cert" pod "4fe351d2-d66c-4cd4-8bf7-264a40d2551c" (UID: "4fe351d2-d66c-4cd4-8bf7-264a40d2551c"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 01:20:39.801933 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:20:39.782107 4939 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4fe351d2-d66c-4cd4-8bf7-264a40d2551c-istiod-ca-cert\") on node \"ip-10-0-140-107.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:20:39.848762 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:20:39.846045 4939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4fe351d2-d66c-4cd4-8bf7-264a40d2551c-kube-api-access-s7cqg" pod "4fe351d2-d66c-4cd4-8bf7-264a40d2551c" (UID: "4fe351d2-d66c-4cd4-8bf7-264a40d2551c"). InnerVolumeSpecName "kube-api-access-s7cqg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 01:20:39.935249 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:20:39.906045 4939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4fe351d2-d66c-4cd4-8bf7-264a40d2551c-istio-envoy" pod "4fe351d2-d66c-4cd4-8bf7-264a40d2551c" (UID: "4fe351d2-d66c-4cd4-8bf7-264a40d2551c"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 01:20:39.935249 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:20:39.910523 4939 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-s7cqg\" (UniqueName: \"kubernetes.io/projected/4fe351d2-d66c-4cd4-8bf7-264a40d2551c-kube-api-access-s7cqg\") on node \"ip-10-0-140-107.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:20:39.935249 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:20:39.910556 4939 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4fe351d2-d66c-4cd4-8bf7-264a40d2551c-istio-envoy\") on node \"ip-10-0-140-107.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:20:39.935809 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:20:39.935730 4939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/4fe351d2-d66c-4cd4-8bf7-264a40d2551c-istio-podinfo" pod "4fe351d2-d66c-4cd4-8bf7-264a40d2551c" (UID: "4fe351d2-d66c-4cd4-8bf7-264a40d2551c"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 01:20:39.969045 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:20:39.968535 4939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4fe351d2-d66c-4cd4-8bf7-264a40d2551c-istio-token" pod "4fe351d2-d66c-4cd4-8bf7-264a40d2551c" (UID: "4fe351d2-d66c-4cd4-8bf7-264a40d2551c"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 01:20:40.080963 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:20:40.058350 4939 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4fe351d2-d66c-4cd4-8bf7-264a40d2551c-istio-podinfo\") on node \"ip-10-0-140-107.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:20:40.080963 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:20:40.058392 4939 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4fe351d2-d66c-4cd4-8bf7-264a40d2551c-istio-token\") on node \"ip-10-0-140-107.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:20:40.209617 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:20:40.144804 4939 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-83755a-openshift-default-c86f9565c-shrff" Nov 03 01:20:40.234971 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:20:40.221466 4939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/30435a8a-8666-4929-b02e-02a6bb02d878-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/30435a8a-8666-4929-b02e-02a6bb02d878-istio-data\") pod \"30435a8a-8666-4929-b02e-02a6bb02d878\" (UID: \"30435a8a-8666-4929-b02e-02a6bb02d878\") " Nov 03 01:20:40.234971 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:20:40.221558 4939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/30435a8a-8666-4929-b02e-02a6bb02d878-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/30435a8a-8666-4929-b02e-02a6bb02d878-istiod-ca-cert\") pod \"30435a8a-8666-4929-b02e-02a6bb02d878\" (UID: \"30435a8a-8666-4929-b02e-02a6bb02d878\") " Nov 03 01:20:40.234971 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:20:40.221616 4939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/30435a8a-8666-4929-b02e-02a6bb02d878-kube-api-access-4nvwn\" (UniqueName: \"kubernetes.io/projected/30435a8a-8666-4929-b02e-02a6bb02d878-kube-api-access-4nvwn\") pod \"30435a8a-8666-4929-b02e-02a6bb02d878\" (UID: \"30435a8a-8666-4929-b02e-02a6bb02d878\") " Nov 03 01:20:40.234971 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:20:40.222255 4939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/30435a8a-8666-4929-b02e-02a6bb02d878-istio-data" pod "30435a8a-8666-4929-b02e-02a6bb02d878" (UID: "30435a8a-8666-4929-b02e-02a6bb02d878"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 01:20:40.234971 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:20:40.224271 4939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/30435a8a-8666-4929-b02e-02a6bb02d878-istiod-ca-cert" pod "30435a8a-8666-4929-b02e-02a6bb02d878" (UID: "30435a8a-8666-4929-b02e-02a6bb02d878"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 01:20:40.234971 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:20:40.225138 4939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/30435a8a-8666-4929-b02e-02a6bb02d878-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/30435a8a-8666-4929-b02e-02a6bb02d878-workload-socket\") pod \"30435a8a-8666-4929-b02e-02a6bb02d878\" (UID: \"30435a8a-8666-4929-b02e-02a6bb02d878\") " Nov 03 01:20:40.234971 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:20:40.225471 4939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/30435a8a-8666-4929-b02e-02a6bb02d878-workload-socket" pod "30435a8a-8666-4929-b02e-02a6bb02d878" (UID: "30435a8a-8666-4929-b02e-02a6bb02d878"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 01:20:40.234971 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:20:40.230781 4939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/30435a8a-8666-4929-b02e-02a6bb02d878-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/30435a8a-8666-4929-b02e-02a6bb02d878-workload-certs\") pod \"30435a8a-8666-4929-b02e-02a6bb02d878\" (UID: \"30435a8a-8666-4929-b02e-02a6bb02d878\") " Nov 03 01:20:40.234971 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:20:40.230876 4939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/30435a8a-8666-4929-b02e-02a6bb02d878-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/30435a8a-8666-4929-b02e-02a6bb02d878-istio-podinfo\") pod \"30435a8a-8666-4929-b02e-02a6bb02d878\" (UID: \"30435a8a-8666-4929-b02e-02a6bb02d878\") " Nov 03 01:20:40.234971 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:20:40.231209 4939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/30435a8a-8666-4929-b02e-02a6bb02d878-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/30435a8a-8666-4929-b02e-02a6bb02d878-istio-envoy\") pod \"30435a8a-8666-4929-b02e-02a6bb02d878\" (UID: \"30435a8a-8666-4929-b02e-02a6bb02d878\") " Nov 03 01:20:40.234971 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:20:40.231350 4939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/30435a8a-8666-4929-b02e-02a6bb02d878-istio-token\" (UniqueName: \"kubernetes.io/projected/30435a8a-8666-4929-b02e-02a6bb02d878-istio-token\") pod \"30435a8a-8666-4929-b02e-02a6bb02d878\" (UID: \"30435a8a-8666-4929-b02e-02a6bb02d878\") " Nov 03 01:20:40.234971 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:20:40.231406 4939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/30435a8a-8666-4929-b02e-02a6bb02d878-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/30435a8a-8666-4929-b02e-02a6bb02d878-credential-socket\") pod \"30435a8a-8666-4929-b02e-02a6bb02d878\" (UID: \"30435a8a-8666-4929-b02e-02a6bb02d878\") " Nov 03 01:20:40.234971 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:20:40.232409 4939 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/30435a8a-8666-4929-b02e-02a6bb02d878-workload-socket\") on node \"ip-10-0-140-107.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:20:40.234971 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:20:40.232440 4939 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/30435a8a-8666-4929-b02e-02a6bb02d878-istio-data\") on node \"ip-10-0-140-107.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:20:40.234971 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:20:40.232456 4939 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/30435a8a-8666-4929-b02e-02a6bb02d878-istiod-ca-cert\") on node \"ip-10-0-140-107.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:20:40.234971 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:20:40.232685 4939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/30435a8a-8666-4929-b02e-02a6bb02d878-credential-socket" pod "30435a8a-8666-4929-b02e-02a6bb02d878" (UID: "30435a8a-8666-4929-b02e-02a6bb02d878"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 01:20:40.234971 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:20:40.232857 4939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/30435a8a-8666-4929-b02e-02a6bb02d878-workload-certs" pod "30435a8a-8666-4929-b02e-02a6bb02d878" (UID: "30435a8a-8666-4929-b02e-02a6bb02d878"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 01:20:40.285152 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:20:40.278334 4939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/30435a8a-8666-4929-b02e-02a6bb02d878-kube-api-access-4nvwn" pod "30435a8a-8666-4929-b02e-02a6bb02d878" (UID: "30435a8a-8666-4929-b02e-02a6bb02d878"). InnerVolumeSpecName "kube-api-access-4nvwn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 01:20:40.286665 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:20:40.286605 4939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/30435a8a-8666-4929-b02e-02a6bb02d878-istio-podinfo" pod "30435a8a-8666-4929-b02e-02a6bb02d878" (UID: "30435a8a-8666-4929-b02e-02a6bb02d878"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 01:20:40.320551 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:20:40.320484 4939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/30435a8a-8666-4929-b02e-02a6bb02d878-istio-envoy" pod "30435a8a-8666-4929-b02e-02a6bb02d878" (UID: "30435a8a-8666-4929-b02e-02a6bb02d878"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 01:20:40.325368 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:20:40.323067 4939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/30435a8a-8666-4929-b02e-02a6bb02d878-istio-token" pod "30435a8a-8666-4929-b02e-02a6bb02d878" (UID: "30435a8a-8666-4929-b02e-02a6bb02d878"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 01:20:40.328685 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:20:40.327981 4939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-59dbb77fb8-grz4z" event={"ID":"8327ba11-84f8-4660-89a4-51433d2c3a0e","Type":"ContainerStarted","Data":"ea7609644d99ab7751f2da5bf272ad43f1764f8807473eab1bac8d2ea5c8b09e"} Nov 03 01:20:40.329131 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:20:40.329042 4939 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="models-as-a-service/payload-processing-external-model-59dbb77fb8-grz4z" Nov 03 01:20:40.331394 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:20:40.331252 4939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-f7680d-openshift-default-c4488d5c9-h5j8d" event={"ID":"4fe351d2-d66c-4cd4-8bf7-264a40d2551c","Type":"ContainerDied","Data":"b3d49abb8aca5b0ba9793afa1d699942589b243091048f2c663a526f5b22f305"} Nov 03 01:20:40.331394 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:20:40.331309 4939 scope.go:122] "RemoveContainer" containerID="40fc2a8fbd662b472eaf98c301d6481da7424cd5121b415342c40e5637d2665b" Nov 03 01:20:40.331586 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:20:40.331420 4939 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-f7680d-openshift-default-c4488d5c9-h5j8d" Nov 03 01:20:40.334982 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:20:40.334902 4939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-83755a-openshift-default-c86f9565c-shrff" event={"ID":"30435a8a-8666-4929-b02e-02a6bb02d878","Type":"ContainerDied","Data":"c52fd75c0012a45c1cfc0e40f217fbf71fa6ca5ad8f0fbed96dce072f1f6014a"} Nov 03 01:20:40.335147 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:20:40.335084 4939 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-83755a-openshift-default-c86f9565c-shrff" Nov 03 01:20:40.335997 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:20:40.335805 4939 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/30435a8a-8666-4929-b02e-02a6bb02d878-istio-envoy\") on node \"ip-10-0-140-107.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:20:40.335997 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:20:40.335827 4939 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/30435a8a-8666-4929-b02e-02a6bb02d878-istio-token\") on node \"ip-10-0-140-107.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:20:40.335997 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:20:40.335838 4939 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/30435a8a-8666-4929-b02e-02a6bb02d878-credential-socket\") on node \"ip-10-0-140-107.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:20:40.335997 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:20:40.335849 4939 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4nvwn\" (UniqueName: \"kubernetes.io/projected/30435a8a-8666-4929-b02e-02a6bb02d878-kube-api-access-4nvwn\") on node \"ip-10-0-140-107.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:20:40.335997 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:20:40.335863 4939 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/30435a8a-8666-4929-b02e-02a6bb02d878-workload-certs\") on node \"ip-10-0-140-107.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:20:40.335997 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:20:40.335873 4939 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/30435a8a-8666-4929-b02e-02a6bb02d878-istio-podinfo\") on node \"ip-10-0-140-107.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:20:40.380495 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:20:40.380026 4939 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-596dc9545f-vm5w5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.20:15021/healthz/ready\": dial tcp 10.132.1.20:15021: connect: connection refused" start-of-body= Nov 03 01:20:40.380495 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:20:40.380089 4939 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-vm5w5" podUID="9a335557-68f1-44ab-b673-f9e0a18be917" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.20:15021/healthz/ready\": dial tcp 10.132.1.20:15021: connect: connection refused" Nov 03 01:20:40.405190 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:20:40.405118 4939 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="models-as-a-service/payload-processing-external-model-59dbb77fb8-grz4z" podStartSLOduration=4.405095146 podStartE2EDuration="4.405095146s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 01:20:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 01:20:40.377542254 +0000 UTC m=+2388.641877205" watchObservedRunningTime="2026-11-03 01:20:40.405095146 +0000 UTC m=+2388.669430093" Nov 03 01:20:40.409960 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:20:40.409684 4939 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-596dc9545f-r2cgw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.21:15021/healthz/ready\": dial tcp 10.132.1.21:15021: connect: connection refused" start-of-body= Nov 03 01:20:40.409960 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:20:40.409739 4939 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-r2cgw" podUID="20419baf-ec7d-47c2-ad77-5f32d6d93992" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.21:15021/healthz/ready\": dial tcp 10.132.1.21:15021: connect: connection refused" Nov 03 01:20:40.414452 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:20:40.414310 4939 scope.go:122] "RemoveContainer" containerID="93d16e09d35069a5b5b048f130ba15a456d72265fe9958895510c5111f8a4eda" Nov 03 01:20:40.431681 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:20:40.431601 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:20:40.631889 ip-10-0-140-107 kubenswrapper[4939]: W1103 01:20:40.612227 4939 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd239ed01_45c1_4f47_b35d_50d2b437e179.slice/crio-93252a13de37ac6f307a84717fecb4cd956e5ab6a97904441d7c1adcd15bc644 WatchSource:0}: Error finding container 93252a13de37ac6f307a84717fecb4cd956e5ab6a97904441d7c1adcd15bc644: Status 404 returned error can't find the container with id 93252a13de37ac6f307a84717fecb4cd956e5ab6a97904441d7c1adcd15bc644 Nov 03 01:20:40.632278 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:20:40.632019 4939 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-83755a-openshift-default-c86f9565c-shrff"] Nov 03 01:20:40.640860 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:20:40.640813 4939 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-83755a-openshift-default-c86f9565c-shrff"] Nov 03 01:20:40.653015 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:20:40.652981 4939 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-f7680d-openshift-default-c4488d5c9-h5j8d"] Nov 03 01:20:40.703062 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:20:40.703018 4939 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-f7680d-openshift-default-c4488d5c9-h5j8d"] Nov 03 01:20:40.705796 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:20:40.705762 4939 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-7c759a/payload-processing-external-model-e2e-extmod-a-7c759a-799fk2rjw"] Nov 03 01:20:41.327480 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:20:41.327448 4939 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-a-821f5a/payload-processing-external-model-e2e-extmod-a-821f5a-5df8cj84n" Nov 03 01:20:41.407816 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:20:41.406687 4939 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-596dc9545f-r2cgw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.21:15021/healthz/ready\": dial tcp 10.132.1.21:15021: connect: connection refused" start-of-body= Nov 03 01:20:41.407816 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:20:41.406746 4939 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-r2cgw" podUID="20419baf-ec7d-47c2-ad77-5f32d6d93992" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.21:15021/healthz/ready\": dial tcp 10.132.1.21:15021: connect: connection refused" Nov 03 01:20:41.407816 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:20:41.406778 4939 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-596dc9545f-vm5w5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.20:15021/healthz/ready\": dial tcp 10.132.1.20:15021: connect: connection refused" start-of-body= Nov 03 01:20:41.407816 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:20:41.406804 4939 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-vm5w5" podUID="9a335557-68f1-44ab-b673-f9e0a18be917" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.20:15021/healthz/ready\": dial tcp 10.132.1.20:15021: connect: connection refused" Nov 03 01:20:41.416567 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:20:41.415390 4939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-7c759a/payload-processing-external-model-e2e-extmod-a-7c759a-799fk2rjw" event={"ID":"d239ed01-45c1-4f47-b35d-50d2b437e179","Type":"ContainerStarted","Data":"d6c4261bd8ff736e5159aac1a56b5a5b08845f5bc34688fde2ef31fdf9b37939"} Nov 03 01:20:41.416567 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:20:41.415460 4939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-7c759a/payload-processing-external-model-e2e-extmod-a-7c759a-799fk2rjw" event={"ID":"d239ed01-45c1-4f47-b35d-50d2b437e179","Type":"ContainerStarted","Data":"93252a13de37ac6f307a84717fecb4cd956e5ab6a97904441d7c1adcd15bc644"} Nov 03 01:20:41.416567 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:20:41.416271 4939 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-a-7c759a/payload-processing-external-model-e2e-extmod-a-7c759a-799fk2rjw" Nov 03 01:20:41.435255 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:20:41.435201 4939 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-a-7c759a/payload-processing-external-model-e2e-extmod-a-7c759a-799fk2rjw" podStartSLOduration=3.435182935 podStartE2EDuration="3.435182935s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 01:20:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 01:20:41.433256524 +0000 UTC m=+2389.697591470" watchObservedRunningTime="2026-11-03 01:20:41.435182935 +0000 UTC m=+2389.699517884" Nov 03 01:20:41.435565 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:20:41.435534 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:20:41.637100 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:20:41.637056 4939 kuberuntime_container.go:898] "Killing container with a grace period" pod="models-as-a-service/payload-processing-external-model-76c77f5454-5cg5j" podUID="e9d0a336-6111-4224-86d9-c95fe94b9545" containerName="payload-processing" containerID="cri-o://635c7c99a38e1d77efed13c4dcc77b0a0b6eaa8589018df5632d637efeaed3e1" gracePeriod=25 Nov 03 01:20:41.750791 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:20:41.750753 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:20:42.093715 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:20:42.093659 4939 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="30435a8a-8666-4929-b02e-02a6bb02d878" path="/var/lib/kubelet/pods/30435a8a-8666-4929-b02e-02a6bb02d878/volumes" Nov 03 01:20:42.094763 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:20:42.094737 4939 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4fe351d2-d66c-4cd4-8bf7-264a40d2551c" path="/var/lib/kubelet/pods/4fe351d2-d66c-4cd4-8bf7-264a40d2551c/volumes" Nov 03 01:20:42.096307 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:20:42.096270 4939 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["models-as-a-service/payload-processing-external-model-59dbb77fb8-grz4z"] Nov 03 01:20:42.195941 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:20:42.192653 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:20:42.402729 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:20:42.402615 4939 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-596dc9545f-r2cgw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.21:15021/healthz/ready\": dial tcp 10.132.1.21:15021: connect: connection refused" start-of-body= Nov 03 01:20:42.429366 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:20:42.402991 4939 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-r2cgw" podUID="20419baf-ec7d-47c2-ad77-5f32d6d93992" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.21:15021/healthz/ready\": dial tcp 10.132.1.21:15021: connect: connection refused" Nov 03 01:20:42.430427 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:20:42.430395 4939 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-596dc9545f-vm5w5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.20:15021/healthz/ready\": dial tcp 10.132.1.20:15021: connect: connection refused" start-of-body= Nov 03 01:20:42.430671 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:20:42.430636 4939 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-vm5w5" podUID="9a335557-68f1-44ab-b673-f9e0a18be917" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.20:15021/healthz/ready\": dial tcp 10.132.1.20:15021: connect: connection refused" Nov 03 01:20:42.553617 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:20:42.553548 4939 generic.go:356] "Generic (PLEG): container finished" podID="e9d0a336-6111-4224-86d9-c95fe94b9545" containerID="635c7c99a38e1d77efed13c4dcc77b0a0b6eaa8589018df5632d637efeaed3e1" exitCode=0 Nov 03 01:20:42.555871 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:20:42.554104 4939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-76c77f5454-5cg5j" event={"ID":"e9d0a336-6111-4224-86d9-c95fe94b9545","Type":"ContainerDied","Data":"635c7c99a38e1d77efed13c4dcc77b0a0b6eaa8589018df5632d637efeaed3e1"} Nov 03 01:20:42.571426 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:20:42.565218 4939 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["models-as-a-service/payload-processing-external-model-76c77f5454-842p6"] Nov 03 01:20:42.571426 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:20:42.566633 4939 cpu_manager.go:414] "containerMap: removing container" podUID="4fe351d2-d66c-4cd4-8bf7-264a40d2551c" containerName="istio-proxy" Nov 03 01:20:42.571426 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:20:42.566655 4939 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4fe351d2-d66c-4cd4-8bf7-264a40d2551c" containerName="istio-proxy" Nov 03 01:20:42.571426 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:20:42.566818 4939 cpu_manager.go:414] "containerMap: removing container" podUID="30435a8a-8666-4929-b02e-02a6bb02d878" containerName="istio-proxy" Nov 03 01:20:42.571426 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:20:42.566832 4939 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="30435a8a-8666-4929-b02e-02a6bb02d878" containerName="istio-proxy" Nov 03 01:20:42.571426 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:20:42.566886 4939 cpu_manager.go:414] "containerMap: removing container" podUID="30435a8a-8666-4929-b02e-02a6bb02d878" containerName="istio-proxy" Nov 03 01:20:42.571426 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:20:42.566897 4939 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="30435a8a-8666-4929-b02e-02a6bb02d878" containerName="istio-proxy" Nov 03 01:20:42.571426 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:20:42.567179 4939 memory_manager.go:365] "RemoveStaleState removing state" podUID="30435a8a-8666-4929-b02e-02a6bb02d878" containerName="istio-proxy" Nov 03 01:20:42.571426 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:20:42.567204 4939 memory_manager.go:365] "RemoveStaleState removing state" podUID="30435a8a-8666-4929-b02e-02a6bb02d878" containerName="istio-proxy" Nov 03 01:20:42.571426 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:20:42.567222 4939 memory_manager.go:365] "RemoveStaleState removing state" podUID="4fe351d2-d66c-4cd4-8bf7-264a40d2551c" containerName="istio-proxy" Nov 03 01:20:42.577142 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:20:42.577115 4939 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["models-as-a-service/payload-processing-external-model-76c77f5454-842p6"] Nov 03 01:20:42.577257 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:20:42.577242 4939 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-76c77f5454-842p6" Nov 03 01:20:42.599943 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:20:42.599895 4939 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-76c77f5454-5cg5j" Nov 03 01:20:42.640477 ip-10-0-140-107 kubenswrapper[4939]: E1103 01:20:42.640448 4939 projected.go:266] Couldn't get secret models-as-a-service/agc-cred-7379d3751ff2: secret "agc-cred-7379d3751ff2" not found Nov 03 01:20:42.640477 ip-10-0-140-107 kubenswrapper[4939]: E1103 01:20:42.640480 4939 projected.go:196] Error preparing data for projected volume provider-credentials for pod models-as-a-service/payload-processing-external-model-59dbb77fb8-grz4z: secret "agc-cred-7379d3751ff2" not found Nov 03 01:20:42.641093 ip-10-0-140-107 kubenswrapper[4939]: E1103 01:20:42.640532 4939 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/8327ba11-84f8-4660-89a4-51433d2c3a0e-provider-credentials podName:8327ba11-84f8-4660-89a4-51433d2c3a0e nodeName:}" failed. No retries permitted until 2026-11-03 01:20:43.140514297 +0000 UTC m=+2391.404849226 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/8327ba11-84f8-4660-89a4-51433d2c3a0e-provider-credentials") pod "payload-processing-external-model-59dbb77fb8-grz4z" (UID: "8327ba11-84f8-4660-89a4-51433d2c3a0e") : secret "agc-cred-7379d3751ff2" not found Nov 03 01:20:42.741898 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:20:42.741788 4939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e9d0a336-6111-4224-86d9-c95fe94b9545-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/e9d0a336-6111-4224-86d9-c95fe94b9545-routing-overlay-volume\") pod \"e9d0a336-6111-4224-86d9-c95fe94b9545\" (UID: \"e9d0a336-6111-4224-86d9-c95fe94b9545\") " Nov 03 01:20:42.742158 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:20:42.741952 4939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e9d0a336-6111-4224-86d9-c95fe94b9545-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e9d0a336-6111-4224-86d9-c95fe94b9545-plugins-config-volume\") pod \"e9d0a336-6111-4224-86d9-c95fe94b9545\" (UID: \"e9d0a336-6111-4224-86d9-c95fe94b9545\") " Nov 03 01:20:42.742316 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:20:42.742293 4939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e9d0a336-6111-4224-86d9-c95fe94b9545-kube-api-access-qfg9p\" (UniqueName: \"kubernetes.io/projected/e9d0a336-6111-4224-86d9-c95fe94b9545-kube-api-access-qfg9p\") pod \"e9d0a336-6111-4224-86d9-c95fe94b9545\" (UID: \"e9d0a336-6111-4224-86d9-c95fe94b9545\") " Nov 03 01:20:42.742395 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:20:42.742355 4939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e9d0a336-6111-4224-86d9-c95fe94b9545-provider-credentials\" (UniqueName: \"kubernetes.io/projected/e9d0a336-6111-4224-86d9-c95fe94b9545-provider-credentials\") pod \"e9d0a336-6111-4224-86d9-c95fe94b9545\" (UID: \"e9d0a336-6111-4224-86d9-c95fe94b9545\") " Nov 03 01:20:42.742820 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:20:42.742787 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7eeeafad-d236-43d4-9661-1e3011e274fc-plugins-config-volume\") pod \"payload-processing-external-model-76c77f5454-842p6\" (UID: \"7eeeafad-d236-43d4-9661-1e3011e274fc\") " pod="models-as-a-service/payload-processing-external-model-76c77f5454-842p6" Nov 03 01:20:42.742985 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:20:42.742897 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/7eeeafad-d236-43d4-9661-1e3011e274fc-provider-credentials\") pod \"payload-processing-external-model-76c77f5454-842p6\" (UID: \"7eeeafad-d236-43d4-9661-1e3011e274fc\") " pod="models-as-a-service/payload-processing-external-model-76c77f5454-842p6" Nov 03 01:20:42.742985 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:20:42.742949 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/7eeeafad-d236-43d4-9661-1e3011e274fc-routing-overlay-volume\") pod \"payload-processing-external-model-76c77f5454-842p6\" (UID: \"7eeeafad-d236-43d4-9661-1e3011e274fc\") " pod="models-as-a-service/payload-processing-external-model-76c77f5454-842p6" Nov 03 01:20:42.744228 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:20:42.743301 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gcs2q\" (UniqueName: \"kubernetes.io/projected/7eeeafad-d236-43d4-9661-1e3011e274fc-kube-api-access-gcs2q\") pod \"payload-processing-external-model-76c77f5454-842p6\" (UID: \"7eeeafad-d236-43d4-9661-1e3011e274fc\") " pod="models-as-a-service/payload-processing-external-model-76c77f5454-842p6" Nov 03 01:20:42.744228 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:20:42.743387 4939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e9d0a336-6111-4224-86d9-c95fe94b9545-routing-overlay-volume" pod "e9d0a336-6111-4224-86d9-c95fe94b9545" (UID: "e9d0a336-6111-4224-86d9-c95fe94b9545"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 01:20:42.744228 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:20:42.743405 4939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e9d0a336-6111-4224-86d9-c95fe94b9545-plugins-config-volume" pod "e9d0a336-6111-4224-86d9-c95fe94b9545" (UID: "e9d0a336-6111-4224-86d9-c95fe94b9545"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 01:20:42.744228 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:20:42.743618 4939 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/e9d0a336-6111-4224-86d9-c95fe94b9545-routing-overlay-volume\") on node \"ip-10-0-140-107.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:20:42.744228 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:20:42.743634 4939 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e9d0a336-6111-4224-86d9-c95fe94b9545-plugins-config-volume\") on node \"ip-10-0-140-107.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:20:42.745451 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:20:42.745418 4939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e9d0a336-6111-4224-86d9-c95fe94b9545-kube-api-access-qfg9p" pod "e9d0a336-6111-4224-86d9-c95fe94b9545" (UID: "e9d0a336-6111-4224-86d9-c95fe94b9545"). InnerVolumeSpecName "kube-api-access-qfg9p". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 01:20:42.745802 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:20:42.745777 4939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e9d0a336-6111-4224-86d9-c95fe94b9545-provider-credentials" pod "e9d0a336-6111-4224-86d9-c95fe94b9545" (UID: "e9d0a336-6111-4224-86d9-c95fe94b9545"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 01:20:42.844691 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:20:42.844657 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/7eeeafad-d236-43d4-9661-1e3011e274fc-provider-credentials\") pod \"payload-processing-external-model-76c77f5454-842p6\" (UID: \"7eeeafad-d236-43d4-9661-1e3011e274fc\") " pod="models-as-a-service/payload-processing-external-model-76c77f5454-842p6" Nov 03 01:20:42.844691 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:20:42.844703 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/7eeeafad-d236-43d4-9661-1e3011e274fc-routing-overlay-volume\") pod \"payload-processing-external-model-76c77f5454-842p6\" (UID: \"7eeeafad-d236-43d4-9661-1e3011e274fc\") " pod="models-as-a-service/payload-processing-external-model-76c77f5454-842p6" Nov 03 01:20:42.845063 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:20:42.845026 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gcs2q\" (UniqueName: \"kubernetes.io/projected/7eeeafad-d236-43d4-9661-1e3011e274fc-kube-api-access-gcs2q\") pod \"payload-processing-external-model-76c77f5454-842p6\" (UID: \"7eeeafad-d236-43d4-9661-1e3011e274fc\") " pod="models-as-a-service/payload-processing-external-model-76c77f5454-842p6" Nov 03 01:20:42.845396 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:20:42.845369 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7eeeafad-d236-43d4-9661-1e3011e274fc-plugins-config-volume\") pod \"payload-processing-external-model-76c77f5454-842p6\" (UID: \"7eeeafad-d236-43d4-9661-1e3011e274fc\") " pod="models-as-a-service/payload-processing-external-model-76c77f5454-842p6" Nov 03 01:20:42.845512 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:20:42.845485 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/7eeeafad-d236-43d4-9661-1e3011e274fc-routing-overlay-volume\") pod \"payload-processing-external-model-76c77f5454-842p6\" (UID: \"7eeeafad-d236-43d4-9661-1e3011e274fc\") " pod="models-as-a-service/payload-processing-external-model-76c77f5454-842p6" Nov 03 01:20:42.845595 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:20:42.845521 4939 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qfg9p\" (UniqueName: \"kubernetes.io/projected/e9d0a336-6111-4224-86d9-c95fe94b9545-kube-api-access-qfg9p\") on node \"ip-10-0-140-107.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:20:42.845595 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:20:42.845543 4939 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/e9d0a336-6111-4224-86d9-c95fe94b9545-provider-credentials\") on node \"ip-10-0-140-107.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:20:42.846361 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:20:42.846325 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7eeeafad-d236-43d4-9661-1e3011e274fc-plugins-config-volume\") pod \"payload-processing-external-model-76c77f5454-842p6\" (UID: \"7eeeafad-d236-43d4-9661-1e3011e274fc\") " pod="models-as-a-service/payload-processing-external-model-76c77f5454-842p6" Nov 03 01:20:42.848085 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:20:42.848060 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/7eeeafad-d236-43d4-9661-1e3011e274fc-provider-credentials\") pod \"payload-processing-external-model-76c77f5454-842p6\" (UID: \"7eeeafad-d236-43d4-9661-1e3011e274fc\") " pod="models-as-a-service/payload-processing-external-model-76c77f5454-842p6" Nov 03 01:20:42.854413 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:20:42.854387 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gcs2q\" (UniqueName: \"kubernetes.io/projected/7eeeafad-d236-43d4-9661-1e3011e274fc-kube-api-access-gcs2q\") pod \"payload-processing-external-model-76c77f5454-842p6\" (UID: \"7eeeafad-d236-43d4-9661-1e3011e274fc\") " pod="models-as-a-service/payload-processing-external-model-76c77f5454-842p6" Nov 03 01:20:42.911330 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:20:42.911293 4939 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-76c77f5454-842p6" Nov 03 01:20:43.097680 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:20:43.097640 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:20:43.151596 ip-10-0-140-107 kubenswrapper[4939]: E1103 01:20:43.151563 4939 projected.go:266] Couldn't get secret models-as-a-service/agc-cred-7379d3751ff2: secret "agc-cred-7379d3751ff2" not found Nov 03 01:20:43.151596 ip-10-0-140-107 kubenswrapper[4939]: E1103 01:20:43.151598 4939 projected.go:196] Error preparing data for projected volume provider-credentials for pod models-as-a-service/payload-processing-external-model-59dbb77fb8-grz4z: secret "agc-cred-7379d3751ff2" not found Nov 03 01:20:43.151874 ip-10-0-140-107 kubenswrapper[4939]: E1103 01:20:43.151667 4939 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/8327ba11-84f8-4660-89a4-51433d2c3a0e-provider-credentials podName:8327ba11-84f8-4660-89a4-51433d2c3a0e nodeName:}" failed. No retries permitted until 2026-11-03 01:20:44.151647244 +0000 UTC m=+2392.415982168 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/8327ba11-84f8-4660-89a4-51433d2c3a0e-provider-credentials") pod "payload-processing-external-model-59dbb77fb8-grz4z" (UID: "8327ba11-84f8-4660-89a4-51433d2c3a0e") : secret "agc-cred-7379d3751ff2" not found Nov 03 01:20:43.247154 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:20:43.242250 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:20:43.274968 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:20:43.274304 4939 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["models-as-a-service/payload-processing-external-model-76c77f5454-842p6"] Nov 03 01:20:43.328951 ip-10-0-140-107 kubenswrapper[4939]: W1103 01:20:43.318024 4939 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7eeeafad_d236_43d4_9661_1e3011e274fc.slice/crio-98b5a8321a5a537319086920856d555db8c696c5c420e1085cbe2788224c160b WatchSource:0}: Error finding container 98b5a8321a5a537319086920856d555db8c696c5c420e1085cbe2788224c160b: Status 404 returned error can't find the container with id 98b5a8321a5a537319086920856d555db8c696c5c420e1085cbe2788224c160b Nov 03 01:20:43.390931 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:20:43.384618 4939 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-596dc9545f-vm5w5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.20:15021/healthz/ready\": dial tcp 10.132.1.20:15021: connect: connection refused" start-of-body= Nov 03 01:20:43.390931 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:20:43.384676 4939 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-vm5w5" podUID="9a335557-68f1-44ab-b673-f9e0a18be917" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.20:15021/healthz/ready\": dial tcp 10.132.1.20:15021: connect: connection refused" Nov 03 01:20:43.416945 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:20:43.403530 4939 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-596dc9545f-r2cgw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.21:15021/healthz/ready\": dial tcp 10.132.1.21:15021: connect: connection refused" start-of-body= Nov 03 01:20:43.416945 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:20:43.403845 4939 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-r2cgw" podUID="20419baf-ec7d-47c2-ad77-5f32d6d93992" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.21:15021/healthz/ready\": dial tcp 10.132.1.21:15021: connect: connection refused" Nov 03 01:20:43.476931 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:20:43.476665 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:20:43.664965 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:20:43.664476 4939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-76c77f5454-5cg5j" event={"ID":"e9d0a336-6111-4224-86d9-c95fe94b9545","Type":"ContainerDied","Data":"8967e0771fec295539a7ab8f856f673f9791d9709e981aec880a4d863a9ba19b"} Nov 03 01:20:43.664965 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:20:43.664550 4939 scope.go:122] "RemoveContainer" containerID="635c7c99a38e1d77efed13c4dcc77b0a0b6eaa8589018df5632d637efeaed3e1" Nov 03 01:20:43.664965 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:20:43.664722 4939 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-76c77f5454-5cg5j" Nov 03 01:20:43.671874 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:20:43.671831 4939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-76c77f5454-842p6" event={"ID":"7eeeafad-d236-43d4-9661-1e3011e274fc","Type":"ContainerStarted","Data":"98b5a8321a5a537319086920856d555db8c696c5c420e1085cbe2788224c160b"} Nov 03 01:20:43.695531 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:20:43.695500 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:20:43.711690 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:20:43.711639 4939 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["models-as-a-service/payload-processing-external-model-76c77f5454-5cg5j"] Nov 03 01:20:43.718029 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:20:43.717999 4939 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["models-as-a-service/payload-processing-external-model-76c77f5454-5cg5j"] Nov 03 01:20:43.882933 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:20:43.882528 4939 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e9d0a336-6111-4224-86d9-c95fe94b9545" path="/var/lib/kubelet/pods/e9d0a336-6111-4224-86d9-c95fe94b9545/volumes" Nov 03 01:20:44.173423 ip-10-0-140-107 kubenswrapper[4939]: E1103 01:20:44.173383 4939 projected.go:266] Couldn't get secret models-as-a-service/agc-cred-7379d3751ff2: secret "agc-cred-7379d3751ff2" not found Nov 03 01:20:44.173423 ip-10-0-140-107 kubenswrapper[4939]: E1103 01:20:44.173427 4939 projected.go:196] Error preparing data for projected volume provider-credentials for pod models-as-a-service/payload-processing-external-model-59dbb77fb8-grz4z: secret "agc-cred-7379d3751ff2" not found Nov 03 01:20:44.173709 ip-10-0-140-107 kubenswrapper[4939]: E1103 01:20:44.173496 4939 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/8327ba11-84f8-4660-89a4-51433d2c3a0e-provider-credentials podName:8327ba11-84f8-4660-89a4-51433d2c3a0e nodeName:}" failed. No retries permitted until 2026-11-03 01:20:46.173477228 +0000 UTC m=+2394.437812174 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/8327ba11-84f8-4660-89a4-51433d2c3a0e-provider-credentials") pod "payload-processing-external-model-59dbb77fb8-grz4z" (UID: "8327ba11-84f8-4660-89a4-51433d2c3a0e") : secret "agc-cred-7379d3751ff2" not found Nov 03 01:20:44.181603 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:20:44.181564 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:20:44.318656 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:20:44.318579 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:20:44.377280 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:20:44.377240 4939 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-596dc9545f-vm5w5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.20:15021/healthz/ready\": dial tcp 10.132.1.20:15021: connect: connection refused" start-of-body= Nov 03 01:20:44.377497 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:20:44.377321 4939 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-vm5w5" podUID="9a335557-68f1-44ab-b673-f9e0a18be917" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.20:15021/healthz/ready\": dial tcp 10.132.1.20:15021: connect: connection refused" Nov 03 01:20:44.389695 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:20:44.389659 4939 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-596dc9545f-r2cgw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.21:15021/healthz/ready\": dial tcp 10.132.1.21:15021: connect: connection refused" start-of-body= Nov 03 01:20:44.389826 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:20:44.389728 4939 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-r2cgw" podUID="20419baf-ec7d-47c2-ad77-5f32d6d93992" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.21:15021/healthz/ready\": dial tcp 10.132.1.21:15021: connect: connection refused" Nov 03 01:20:44.430718 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:20:44.430588 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:20:44.578922 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:20:44.578875 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:20:44.609527 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:20:44.609486 4939 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-a-d25759/payload-processing-external-model-e2e-extmod-a-d25759-579bb87bp"] Nov 03 01:20:44.611200 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:20:44.611169 4939 cpu_manager.go:414] "containerMap: removing container" podUID="e9d0a336-6111-4224-86d9-c95fe94b9545" containerName="payload-processing" Nov 03 01:20:44.611200 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:20:44.611198 4939 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e9d0a336-6111-4224-86d9-c95fe94b9545" containerName="payload-processing" Nov 03 01:20:44.611438 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:20:44.611419 4939 memory_manager.go:365] "RemoveStaleState removing state" podUID="e9d0a336-6111-4224-86d9-c95fe94b9545" containerName="payload-processing" Nov 03 01:20:44.621166 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:20:44.619882 4939 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-d25759/payload-processing-external-model-e2e-extmod-a-d25759-579bb87bp" Nov 03 01:20:44.626856 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:20:44.626737 4939 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-d25759\"/\"routing-overlay\"" Nov 03 01:20:44.626856 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:20:44.626776 4939 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-d25759\"/\"agc-cred-22a51f875964\"" Nov 03 01:20:44.626856 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:20:44.626820 4939 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-d25759\"/\"payload-processing-external-model-plugins-e2e-extmod-a-d25759\"" Nov 03 01:20:44.661490 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:20:44.637009 4939 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-d25759/payload-processing-external-model-e2e-extmod-a-d25759-579bb87bp"] Nov 03 01:20:44.670285 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:20:44.669340 4939 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-d25759\"/\"payload-processing-external-model-e2e--4d491765-dockercfg-56tgf\"" Nov 03 01:20:44.713440 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:20:44.713313 4939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-76c77f5454-842p6" event={"ID":"7eeeafad-d236-43d4-9661-1e3011e274fc","Type":"ContainerStarted","Data":"ba502782bf1fe19c765b490fe04393e5c8545856b28165534c1007174c09c894"} Nov 03 01:20:44.714956 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:20:44.714930 4939 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="models-as-a-service/payload-processing-external-model-76c77f5454-842p6" Nov 03 01:20:44.718623 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:20:44.717560 4939 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-d25759\"/\"openshift-service-ca.crt\"" Nov 03 01:20:44.718623 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:20:44.717696 4939 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-d25759\"/\"kube-root-ca.crt\"" Nov 03 01:20:44.744752 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:20:44.744671 4939 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="models-as-a-service/payload-processing-external-model-76c77f5454-842p6" podStartSLOduration=3.744652434 podStartE2EDuration="3.744652434s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 01:20:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 01:20:44.731550764 +0000 UTC m=+2392.995885710" watchObservedRunningTime="2026-11-03 01:20:44.744652434 +0000 UTC m=+2393.008987380" Nov 03 01:20:44.759928 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:20:44.759731 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/2151f14f-ab60-4499-aa61-cc42f8404e86-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-d25759-579bb87bp\" (UID: \"2151f14f-ab60-4499-aa61-cc42f8404e86\") " pod="ai-tenant-e2e-extmod-a-d25759/payload-processing-external-model-e2e-extmod-a-d25759-579bb87bp" Nov 03 01:20:44.761286 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:20:44.760156 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/2151f14f-ab60-4499-aa61-cc42f8404e86-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-d25759-579bb87bp\" (UID: \"2151f14f-ab60-4499-aa61-cc42f8404e86\") " pod="ai-tenant-e2e-extmod-a-d25759/payload-processing-external-model-e2e-extmod-a-d25759-579bb87bp" Nov 03 01:20:44.761286 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:20:44.760281 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zzjvj\" (UniqueName: \"kubernetes.io/projected/2151f14f-ab60-4499-aa61-cc42f8404e86-kube-api-access-zzjvj\") pod \"payload-processing-external-model-e2e-extmod-a-d25759-579bb87bp\" (UID: \"2151f14f-ab60-4499-aa61-cc42f8404e86\") " pod="ai-tenant-e2e-extmod-a-d25759/payload-processing-external-model-e2e-extmod-a-d25759-579bb87bp" Nov 03 01:20:44.761286 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:20:44.760339 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2151f14f-ab60-4499-aa61-cc42f8404e86-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-d25759-579bb87bp\" (UID: \"2151f14f-ab60-4499-aa61-cc42f8404e86\") " pod="ai-tenant-e2e-extmod-a-d25759/payload-processing-external-model-e2e-extmod-a-d25759-579bb87bp" Nov 03 01:20:44.866930 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:20:44.864416 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/2151f14f-ab60-4499-aa61-cc42f8404e86-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-d25759-579bb87bp\" (UID: \"2151f14f-ab60-4499-aa61-cc42f8404e86\") " pod="ai-tenant-e2e-extmod-a-d25759/payload-processing-external-model-e2e-extmod-a-d25759-579bb87bp" Nov 03 01:20:44.866930 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:20:44.864591 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zzjvj\" (UniqueName: \"kubernetes.io/projected/2151f14f-ab60-4499-aa61-cc42f8404e86-kube-api-access-zzjvj\") pod \"payload-processing-external-model-e2e-extmod-a-d25759-579bb87bp\" (UID: \"2151f14f-ab60-4499-aa61-cc42f8404e86\") " pod="ai-tenant-e2e-extmod-a-d25759/payload-processing-external-model-e2e-extmod-a-d25759-579bb87bp" Nov 03 01:20:44.866930 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:20:44.864678 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2151f14f-ab60-4499-aa61-cc42f8404e86-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-d25759-579bb87bp\" (UID: \"2151f14f-ab60-4499-aa61-cc42f8404e86\") " pod="ai-tenant-e2e-extmod-a-d25759/payload-processing-external-model-e2e-extmod-a-d25759-579bb87bp" Nov 03 01:20:44.866930 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:20:44.866407 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/2151f14f-ab60-4499-aa61-cc42f8404e86-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-d25759-579bb87bp\" (UID: \"2151f14f-ab60-4499-aa61-cc42f8404e86\") " pod="ai-tenant-e2e-extmod-a-d25759/payload-processing-external-model-e2e-extmod-a-d25759-579bb87bp" Nov 03 01:20:44.901125 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:20:44.900028 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2151f14f-ab60-4499-aa61-cc42f8404e86-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-d25759-579bb87bp\" (UID: \"2151f14f-ab60-4499-aa61-cc42f8404e86\") " pod="ai-tenant-e2e-extmod-a-d25759/payload-processing-external-model-e2e-extmod-a-d25759-579bb87bp" Nov 03 01:20:44.901125 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:20:44.900737 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/2151f14f-ab60-4499-aa61-cc42f8404e86-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-d25759-579bb87bp\" (UID: \"2151f14f-ab60-4499-aa61-cc42f8404e86\") " pod="ai-tenant-e2e-extmod-a-d25759/payload-processing-external-model-e2e-extmod-a-d25759-579bb87bp" Nov 03 01:20:44.936203 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:20:44.935282 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zzjvj\" (UniqueName: \"kubernetes.io/projected/2151f14f-ab60-4499-aa61-cc42f8404e86-kube-api-access-zzjvj\") pod \"payload-processing-external-model-e2e-extmod-a-d25759-579bb87bp\" (UID: \"2151f14f-ab60-4499-aa61-cc42f8404e86\") " pod="ai-tenant-e2e-extmod-a-d25759/payload-processing-external-model-e2e-extmod-a-d25759-579bb87bp" Nov 03 01:20:44.969050 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:20:44.956825 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/2151f14f-ab60-4499-aa61-cc42f8404e86-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-d25759-579bb87bp\" (UID: \"2151f14f-ab60-4499-aa61-cc42f8404e86\") " pod="ai-tenant-e2e-extmod-a-d25759/payload-processing-external-model-e2e-extmod-a-d25759-579bb87bp" Nov 03 01:20:45.275979 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:20:45.269306 4939 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-d25759/payload-processing-external-model-e2e-extmod-a-d25759-579bb87bp" Nov 03 01:20:45.470648 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:20:45.470500 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:20:45.612421 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:20:45.612382 4939 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-b-e1f08d/payload-processing-external-model-e2e-extmod-b-e1f08d-6cf6k74l8"] Nov 03 01:20:45.622791 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:20:45.622759 4939 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-e1f08d/payload-processing-external-model-e2e-extmod-b-e1f08d-6cf6k74l8" Nov 03 01:20:45.631958 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:20:45.631932 4939 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-e1f08d\"/\"agc-cred-4646824b7cb1\"" Nov 03 01:20:45.632223 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:20:45.632203 4939 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-e1f08d\"/\"payload-processing-external-model-plugins-e2e-extmod-b-e1f08d\"" Nov 03 01:20:45.632467 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:20:45.632448 4939 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-e1f08d\"/\"kube-root-ca.crt\"" Nov 03 01:20:45.632939 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:20:45.632757 4939 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-e1f08d\"/\"openshift-service-ca.crt\"" Nov 03 01:20:45.632939 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:20:45.632762 4939 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-e1f08d\"/\"routing-overlay\"" Nov 03 01:20:45.632939 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:20:45.632878 4939 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-e1f08d\"/\"payload-processing-external-model-e2e--b70757fa-dockercfg-s4rrb\"" Nov 03 01:20:45.635657 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:20:45.635633 4939 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-e1f08d/payload-processing-external-model-e2e-extmod-b-e1f08d-6cf6k74l8"] Nov 03 01:20:45.652350 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:20:45.652322 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wfrzg\" (UniqueName: \"kubernetes.io/projected/72e20aa3-0a6c-4e0d-b51b-891b86a7607e-kube-api-access-wfrzg\") pod \"payload-processing-external-model-e2e-extmod-b-e1f08d-6cf6k74l8\" (UID: \"72e20aa3-0a6c-4e0d-b51b-891b86a7607e\") " pod="ai-tenant-e2e-extmod-b-e1f08d/payload-processing-external-model-e2e-extmod-b-e1f08d-6cf6k74l8" Nov 03 01:20:45.652708 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:20:45.652682 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/72e20aa3-0a6c-4e0d-b51b-891b86a7607e-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-e1f08d-6cf6k74l8\" (UID: \"72e20aa3-0a6c-4e0d-b51b-891b86a7607e\") " pod="ai-tenant-e2e-extmod-b-e1f08d/payload-processing-external-model-e2e-extmod-b-e1f08d-6cf6k74l8" Nov 03 01:20:45.652869 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:20:45.652850 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/72e20aa3-0a6c-4e0d-b51b-891b86a7607e-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-e1f08d-6cf6k74l8\" (UID: \"72e20aa3-0a6c-4e0d-b51b-891b86a7607e\") " pod="ai-tenant-e2e-extmod-b-e1f08d/payload-processing-external-model-e2e-extmod-b-e1f08d-6cf6k74l8" Nov 03 01:20:45.653188 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:20:45.653164 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/72e20aa3-0a6c-4e0d-b51b-891b86a7607e-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-e1f08d-6cf6k74l8\" (UID: \"72e20aa3-0a6c-4e0d-b51b-891b86a7607e\") " pod="ai-tenant-e2e-extmod-b-e1f08d/payload-processing-external-model-e2e-extmod-b-e1f08d-6cf6k74l8" Nov 03 01:20:45.931310 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:20:45.758308 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wfrzg\" (UniqueName: \"kubernetes.io/projected/72e20aa3-0a6c-4e0d-b51b-891b86a7607e-kube-api-access-wfrzg\") pod \"payload-processing-external-model-e2e-extmod-b-e1f08d-6cf6k74l8\" (UID: \"72e20aa3-0a6c-4e0d-b51b-891b86a7607e\") " pod="ai-tenant-e2e-extmod-b-e1f08d/payload-processing-external-model-e2e-extmod-b-e1f08d-6cf6k74l8" Nov 03 01:20:45.931310 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:20:45.758446 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/72e20aa3-0a6c-4e0d-b51b-891b86a7607e-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-e1f08d-6cf6k74l8\" (UID: \"72e20aa3-0a6c-4e0d-b51b-891b86a7607e\") " pod="ai-tenant-e2e-extmod-b-e1f08d/payload-processing-external-model-e2e-extmod-b-e1f08d-6cf6k74l8" Nov 03 01:20:45.931310 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:20:45.758484 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/72e20aa3-0a6c-4e0d-b51b-891b86a7607e-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-e1f08d-6cf6k74l8\" (UID: \"72e20aa3-0a6c-4e0d-b51b-891b86a7607e\") " pod="ai-tenant-e2e-extmod-b-e1f08d/payload-processing-external-model-e2e-extmod-b-e1f08d-6cf6k74l8" Nov 03 01:20:45.931310 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:20:45.758660 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/72e20aa3-0a6c-4e0d-b51b-891b86a7607e-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-e1f08d-6cf6k74l8\" (UID: \"72e20aa3-0a6c-4e0d-b51b-891b86a7607e\") " pod="ai-tenant-e2e-extmod-b-e1f08d/payload-processing-external-model-e2e-extmod-b-e1f08d-6cf6k74l8" Nov 03 01:20:45.931310 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:20:45.759764 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/72e20aa3-0a6c-4e0d-b51b-891b86a7607e-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-e1f08d-6cf6k74l8\" (UID: \"72e20aa3-0a6c-4e0d-b51b-891b86a7607e\") " pod="ai-tenant-e2e-extmod-b-e1f08d/payload-processing-external-model-e2e-extmod-b-e1f08d-6cf6k74l8" Nov 03 01:20:45.931310 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:20:45.794183 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/72e20aa3-0a6c-4e0d-b51b-891b86a7607e-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-e1f08d-6cf6k74l8\" (UID: \"72e20aa3-0a6c-4e0d-b51b-891b86a7607e\") " pod="ai-tenant-e2e-extmod-b-e1f08d/payload-processing-external-model-e2e-extmod-b-e1f08d-6cf6k74l8" Nov 03 01:20:46.138664 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:20:46.138629 4939 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-41953fef-726x2"] Nov 03 01:20:46.155796 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:20:46.152902 4939 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-41953fef-726x2"] Nov 03 01:20:46.155796 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:20:46.147690 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wfrzg\" (UniqueName: \"kubernetes.io/projected/72e20aa3-0a6c-4e0d-b51b-891b86a7607e-kube-api-access-wfrzg\") pod \"payload-processing-external-model-e2e-extmod-b-e1f08d-6cf6k74l8\" (UID: \"72e20aa3-0a6c-4e0d-b51b-891b86a7607e\") " pod="ai-tenant-e2e-extmod-b-e1f08d/payload-processing-external-model-e2e-extmod-b-e1f08d-6cf6k74l8" Nov 03 01:20:46.157867 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:20:46.157834 4939 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-d25759/payload-processing-external-model-e2e-extmod-a-d25759-579bb87bp"] Nov 03 01:20:46.159745 ip-10-0-140-107 kubenswrapper[4939]: W1103 01:20:46.159615 4939 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2151f14f_ab60_4499_aa61_cc42f8404e86.slice/crio-8f91265768cdccbd96de9e506904ddb3f1226f47b976198ff0b418e15cf0af81 WatchSource:0}: Error finding container 8f91265768cdccbd96de9e506904ddb3f1226f47b976198ff0b418e15cf0af81: Status 404 returned error can't find the container with id 8f91265768cdccbd96de9e506904ddb3f1226f47b976198ff0b418e15cf0af81 Nov 03 01:20:46.180396 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:20:46.175088 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/72e20aa3-0a6c-4e0d-b51b-891b86a7607e-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-e1f08d-6cf6k74l8\" (UID: \"72e20aa3-0a6c-4e0d-b51b-891b86a7607e\") " pod="ai-tenant-e2e-extmod-b-e1f08d/payload-processing-external-model-e2e-extmod-b-e1f08d-6cf6k74l8" Nov 03 01:20:46.208204 ip-10-0-140-107 kubenswrapper[4939]: E1103 01:20:46.208076 4939 projected.go:266] Couldn't get secret models-as-a-service/agc-cred-7379d3751ff2: secret "agc-cred-7379d3751ff2" not found Nov 03 01:20:46.209044 ip-10-0-140-107 kubenswrapper[4939]: E1103 01:20:46.209027 4939 projected.go:196] Error preparing data for projected volume provider-credentials for pod models-as-a-service/payload-processing-external-model-59dbb77fb8-grz4z: secret "agc-cred-7379d3751ff2" not found Nov 03 01:20:46.209253 ip-10-0-140-107 kubenswrapper[4939]: E1103 01:20:46.209238 4939 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/8327ba11-84f8-4660-89a4-51433d2c3a0e-provider-credentials podName:8327ba11-84f8-4660-89a4-51433d2c3a0e nodeName:}" failed. No retries permitted until 2026-11-03 01:20:50.209215641 +0000 UTC m=+2398.473550578 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/8327ba11-84f8-4660-89a4-51433d2c3a0e-provider-credentials") pod "payload-processing-external-model-59dbb77fb8-grz4z" (UID: "8327ba11-84f8-4660-89a4-51433d2c3a0e") : secret "agc-cred-7379d3751ff2" not found Nov 03 01:20:46.326604 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:20:46.326567 4939 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-e1f08d/payload-processing-external-model-e2e-extmod-b-e1f08d-6cf6k74l8" Nov 03 01:20:46.424721 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:20:46.424150 4939 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-596dc9545f-vm5w5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.20:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 01:20:46.424721 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:20:46.424214 4939 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-vm5w5" podUID="9a335557-68f1-44ab-b673-f9e0a18be917" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.20:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 01:20:46.424721 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:20:46.424228 4939 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-596dc9545f-r2cgw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.21:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 01:20:46.424721 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:20:46.424290 4939 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-r2cgw" podUID="20419baf-ec7d-47c2-ad77-5f32d6d93992" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.21:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 01:20:46.482603 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:20:46.480662 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:20:46.525676 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:20:46.525246 4939 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-r2cgw" Nov 03 01:20:46.526712 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:20:46.526143 4939 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-r2cgw" Nov 03 01:20:46.527304 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:20:46.527237 4939 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-r2cgw" Nov 03 01:20:46.610460 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:20:46.607836 4939 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-vm5w5" Nov 03 01:20:46.611438 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:20:46.611281 4939 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-vm5w5" Nov 03 01:20:46.611585 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:20:46.611512 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:20:46.617971 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:20:46.613896 4939 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-vm5w5" Nov 03 01:20:46.699007 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:20:46.698542 4939 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-e1f08d/payload-processing-external-model-e2e-extmod-b-e1f08d-6cf6k74l8"] Nov 03 01:20:46.757125 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:20:46.757099 4939 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-a-d25759/payload-processing-external-model-e2e-extmod-a-d25759-56bbc6hhk"] Nov 03 01:20:46.764842 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:20:46.764820 4939 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-d25759/payload-processing-external-model-e2e-extmod-a-d25759-56bbc6hhk" Nov 03 01:20:46.771969 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:20:46.769989 4939 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-d25759\"/\"agc-cred-8aa58f91c6d5\"" Nov 03 01:20:46.771969 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:20:46.770424 4939 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-d25759/payload-processing-external-model-e2e-extmod-a-d25759-56bbc6hhk"] Nov 03 01:20:46.857938 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:20:46.836359 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/8c6f8abe-dd67-45b9-b043-1c1e13ee96de-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-d25759-56bbc6hhk\" (UID: \"8c6f8abe-dd67-45b9-b043-1c1e13ee96de\") " pod="ai-tenant-e2e-extmod-a-d25759/payload-processing-external-model-e2e-extmod-a-d25759-56bbc6hhk" Nov 03 01:20:46.941948 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:20:46.941895 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9t2sl\" (UniqueName: \"kubernetes.io/projected/8c6f8abe-dd67-45b9-b043-1c1e13ee96de-kube-api-access-9t2sl\") pod \"payload-processing-external-model-e2e-extmod-a-d25759-56bbc6hhk\" (UID: \"8c6f8abe-dd67-45b9-b043-1c1e13ee96de\") " pod="ai-tenant-e2e-extmod-a-d25759/payload-processing-external-model-e2e-extmod-a-d25759-56bbc6hhk" Nov 03 01:20:47.082936 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:20:47.081839 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8c6f8abe-dd67-45b9-b043-1c1e13ee96de-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-d25759-56bbc6hhk\" (UID: \"8c6f8abe-dd67-45b9-b043-1c1e13ee96de\") " pod="ai-tenant-e2e-extmod-a-d25759/payload-processing-external-model-e2e-extmod-a-d25759-56bbc6hhk" Nov 03 01:20:47.082936 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:20:47.082077 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/8c6f8abe-dd67-45b9-b043-1c1e13ee96de-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-d25759-56bbc6hhk\" (UID: \"8c6f8abe-dd67-45b9-b043-1c1e13ee96de\") " pod="ai-tenant-e2e-extmod-a-d25759/payload-processing-external-model-e2e-extmod-a-d25759-56bbc6hhk" Nov 03 01:20:47.198489 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:20:47.198432 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9t2sl\" (UniqueName: \"kubernetes.io/projected/8c6f8abe-dd67-45b9-b043-1c1e13ee96de-kube-api-access-9t2sl\") pod \"payload-processing-external-model-e2e-extmod-a-d25759-56bbc6hhk\" (UID: \"8c6f8abe-dd67-45b9-b043-1c1e13ee96de\") " pod="ai-tenant-e2e-extmod-a-d25759/payload-processing-external-model-e2e-extmod-a-d25759-56bbc6hhk" Nov 03 01:20:47.199095 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:20:47.199051 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8c6f8abe-dd67-45b9-b043-1c1e13ee96de-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-d25759-56bbc6hhk\" (UID: \"8c6f8abe-dd67-45b9-b043-1c1e13ee96de\") " pod="ai-tenant-e2e-extmod-a-d25759/payload-processing-external-model-e2e-extmod-a-d25759-56bbc6hhk" Nov 03 01:20:47.199566 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:20:47.199529 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/8c6f8abe-dd67-45b9-b043-1c1e13ee96de-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-d25759-56bbc6hhk\" (UID: \"8c6f8abe-dd67-45b9-b043-1c1e13ee96de\") " pod="ai-tenant-e2e-extmod-a-d25759/payload-processing-external-model-e2e-extmod-a-d25759-56bbc6hhk" Nov 03 01:20:47.199985 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:20:47.199964 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/8c6f8abe-dd67-45b9-b043-1c1e13ee96de-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-d25759-56bbc6hhk\" (UID: \"8c6f8abe-dd67-45b9-b043-1c1e13ee96de\") " pod="ai-tenant-e2e-extmod-a-d25759/payload-processing-external-model-e2e-extmod-a-d25759-56bbc6hhk" Nov 03 01:20:47.212833 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:20:47.212550 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/8c6f8abe-dd67-45b9-b043-1c1e13ee96de-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-d25759-56bbc6hhk\" (UID: \"8c6f8abe-dd67-45b9-b043-1c1e13ee96de\") " pod="ai-tenant-e2e-extmod-a-d25759/payload-processing-external-model-e2e-extmod-a-d25759-56bbc6hhk" Nov 03 01:20:47.218853 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:20:47.218825 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8c6f8abe-dd67-45b9-b043-1c1e13ee96de-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-d25759-56bbc6hhk\" (UID: \"8c6f8abe-dd67-45b9-b043-1c1e13ee96de\") " pod="ai-tenant-e2e-extmod-a-d25759/payload-processing-external-model-e2e-extmod-a-d25759-56bbc6hhk" Nov 03 01:20:47.219301 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:20:47.219141 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/8c6f8abe-dd67-45b9-b043-1c1e13ee96de-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-d25759-56bbc6hhk\" (UID: \"8c6f8abe-dd67-45b9-b043-1c1e13ee96de\") " pod="ai-tenant-e2e-extmod-a-d25759/payload-processing-external-model-e2e-extmod-a-d25759-56bbc6hhk" Nov 03 01:20:47.229093 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:20:47.229060 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9t2sl\" (UniqueName: \"kubernetes.io/projected/8c6f8abe-dd67-45b9-b043-1c1e13ee96de-kube-api-access-9t2sl\") pod \"payload-processing-external-model-e2e-extmod-a-d25759-56bbc6hhk\" (UID: \"8c6f8abe-dd67-45b9-b043-1c1e13ee96de\") " pod="ai-tenant-e2e-extmod-a-d25759/payload-processing-external-model-e2e-extmod-a-d25759-56bbc6hhk" Nov 03 01:20:47.258482 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:20:47.258442 4939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-d25759/payload-processing-external-model-e2e-extmod-a-d25759-579bb87bp" event={"ID":"2151f14f-ab60-4499-aa61-cc42f8404e86","Type":"ContainerStarted","Data":"4e8babc8b437e5a8125135de09dceda45731e5317170b77a97979446f32f62c0"} Nov 03 01:20:47.258626 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:20:47.258543 4939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-d25759/payload-processing-external-model-e2e-extmod-a-d25759-579bb87bp" event={"ID":"2151f14f-ab60-4499-aa61-cc42f8404e86","Type":"ContainerStarted","Data":"8f91265768cdccbd96de9e506904ddb3f1226f47b976198ff0b418e15cf0af81"} Nov 03 01:20:47.258626 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:20:47.258574 4939 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-a-d25759/payload-processing-external-model-e2e-extmod-a-d25759-579bb87bp" Nov 03 01:20:47.262389 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:20:47.262360 4939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-e1f08d/payload-processing-external-model-e2e-extmod-b-e1f08d-6cf6k74l8" event={"ID":"72e20aa3-0a6c-4e0d-b51b-891b86a7607e","Type":"ContainerStarted","Data":"700559a30f523fae6ed4b034bbed1a4124deb73b8948fc9ae767ade3ecc02a98"} Nov 03 01:20:47.262520 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:20:47.262408 4939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-e1f08d/payload-processing-external-model-e2e-extmod-b-e1f08d-6cf6k74l8" event={"ID":"72e20aa3-0a6c-4e0d-b51b-891b86a7607e","Type":"ContainerStarted","Data":"3c72b88b03591d15388392b826f5811c40f03883386e8760ce6f773067f814af"} Nov 03 01:20:47.263060 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:20:47.263040 4939 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-b-e1f08d/payload-processing-external-model-e2e-extmod-b-e1f08d-6cf6k74l8" Nov 03 01:20:47.281977 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:20:47.281732 4939 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-a-d25759/payload-processing-external-model-e2e-extmod-a-d25759-579bb87bp" podStartSLOduration=3.281719768 podStartE2EDuration="3.281719768s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 01:20:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 01:20:47.276320164 +0000 UTC m=+2395.540655111" watchObservedRunningTime="2026-11-03 01:20:47.281719768 +0000 UTC m=+2395.546054736" Nov 03 01:20:47.293946 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:20:47.293840 4939 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-b-e1f08d/payload-processing-external-model-e2e-extmod-b-e1f08d-6cf6k74l8" podStartSLOduration=2.293828563 podStartE2EDuration="2.293828563s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 01:20:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 01:20:47.291683599 +0000 UTC m=+2395.556018544" watchObservedRunningTime="2026-11-03 01:20:47.293828563 +0000 UTC m=+2395.558163508" Nov 03 01:20:47.343368 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:20:47.343285 4939 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-b-e1f08d/payload-processing-external-model-e2e-extmod-b-e1f08d-9f5cms4gl"] Nov 03 01:20:47.355312 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:20:47.355284 4939 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-e1f08d/payload-processing-external-model-e2e-extmod-b-e1f08d-9f5cms4gl" Nov 03 01:20:47.358846 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:20:47.358816 4939 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-e1f08d/payload-processing-external-model-e2e-extmod-b-e1f08d-9f5cms4gl"] Nov 03 01:20:47.359548 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:20:47.359525 4939 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-e1f08d\"/\"agc-cred-fbc8c941c1bc\"" Nov 03 01:20:47.425383 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:20:47.424363 4939 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-d25759/payload-processing-external-model-e2e-extmod-a-d25759-56bbc6hhk" Nov 03 01:20:47.540303 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:20:47.540270 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/0a9abcbc-bbe0-4f43-ae83-7e2ec3bc9d13-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-e1f08d-9f5cms4gl\" (UID: \"0a9abcbc-bbe0-4f43-ae83-7e2ec3bc9d13\") " pod="ai-tenant-e2e-extmod-b-e1f08d/payload-processing-external-model-e2e-extmod-b-e1f08d-9f5cms4gl" Nov 03 01:20:47.541074 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:20:47.541052 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qh966\" (UniqueName: \"kubernetes.io/projected/0a9abcbc-bbe0-4f43-ae83-7e2ec3bc9d13-kube-api-access-qh966\") pod \"payload-processing-external-model-e2e-extmod-b-e1f08d-9f5cms4gl\" (UID: \"0a9abcbc-bbe0-4f43-ae83-7e2ec3bc9d13\") " pod="ai-tenant-e2e-extmod-b-e1f08d/payload-processing-external-model-e2e-extmod-b-e1f08d-9f5cms4gl" Nov 03 01:20:47.541242 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:20:47.541230 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/0a9abcbc-bbe0-4f43-ae83-7e2ec3bc9d13-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-e1f08d-9f5cms4gl\" (UID: \"0a9abcbc-bbe0-4f43-ae83-7e2ec3bc9d13\") " pod="ai-tenant-e2e-extmod-b-e1f08d/payload-processing-external-model-e2e-extmod-b-e1f08d-9f5cms4gl" Nov 03 01:20:47.541403 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:20:47.541392 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0a9abcbc-bbe0-4f43-ae83-7e2ec3bc9d13-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-e1f08d-9f5cms4gl\" (UID: \"0a9abcbc-bbe0-4f43-ae83-7e2ec3bc9d13\") " pod="ai-tenant-e2e-extmod-b-e1f08d/payload-processing-external-model-e2e-extmod-b-e1f08d-9f5cms4gl" Nov 03 01:20:47.643261 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:20:47.643222 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/0a9abcbc-bbe0-4f43-ae83-7e2ec3bc9d13-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-e1f08d-9f5cms4gl\" (UID: \"0a9abcbc-bbe0-4f43-ae83-7e2ec3bc9d13\") " pod="ai-tenant-e2e-extmod-b-e1f08d/payload-processing-external-model-e2e-extmod-b-e1f08d-9f5cms4gl" Nov 03 01:20:47.643501 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:20:47.643476 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qh966\" (UniqueName: \"kubernetes.io/projected/0a9abcbc-bbe0-4f43-ae83-7e2ec3bc9d13-kube-api-access-qh966\") pod \"payload-processing-external-model-e2e-extmod-b-e1f08d-9f5cms4gl\" (UID: \"0a9abcbc-bbe0-4f43-ae83-7e2ec3bc9d13\") " pod="ai-tenant-e2e-extmod-b-e1f08d/payload-processing-external-model-e2e-extmod-b-e1f08d-9f5cms4gl" Nov 03 01:20:47.643984 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:20:47.643957 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/0a9abcbc-bbe0-4f43-ae83-7e2ec3bc9d13-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-e1f08d-9f5cms4gl\" (UID: \"0a9abcbc-bbe0-4f43-ae83-7e2ec3bc9d13\") " pod="ai-tenant-e2e-extmod-b-e1f08d/payload-processing-external-model-e2e-extmod-b-e1f08d-9f5cms4gl" Nov 03 01:20:47.644123 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:20:47.644061 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0a9abcbc-bbe0-4f43-ae83-7e2ec3bc9d13-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-e1f08d-9f5cms4gl\" (UID: \"0a9abcbc-bbe0-4f43-ae83-7e2ec3bc9d13\") " pod="ai-tenant-e2e-extmod-b-e1f08d/payload-processing-external-model-e2e-extmod-b-e1f08d-9f5cms4gl" Nov 03 01:20:47.644452 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:20:47.644419 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/0a9abcbc-bbe0-4f43-ae83-7e2ec3bc9d13-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-e1f08d-9f5cms4gl\" (UID: \"0a9abcbc-bbe0-4f43-ae83-7e2ec3bc9d13\") " pod="ai-tenant-e2e-extmod-b-e1f08d/payload-processing-external-model-e2e-extmod-b-e1f08d-9f5cms4gl" Nov 03 01:20:47.644892 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:20:47.644870 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0a9abcbc-bbe0-4f43-ae83-7e2ec3bc9d13-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-e1f08d-9f5cms4gl\" (UID: \"0a9abcbc-bbe0-4f43-ae83-7e2ec3bc9d13\") " pod="ai-tenant-e2e-extmod-b-e1f08d/payload-processing-external-model-e2e-extmod-b-e1f08d-9f5cms4gl" Nov 03 01:20:47.647286 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:20:47.647264 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/0a9abcbc-bbe0-4f43-ae83-7e2ec3bc9d13-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-e1f08d-9f5cms4gl\" (UID: \"0a9abcbc-bbe0-4f43-ae83-7e2ec3bc9d13\") " pod="ai-tenant-e2e-extmod-b-e1f08d/payload-processing-external-model-e2e-extmod-b-e1f08d-9f5cms4gl" Nov 03 01:20:47.653976 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:20:47.653954 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qh966\" (UniqueName: \"kubernetes.io/projected/0a9abcbc-bbe0-4f43-ae83-7e2ec3bc9d13-kube-api-access-qh966\") pod \"payload-processing-external-model-e2e-extmod-b-e1f08d-9f5cms4gl\" (UID: \"0a9abcbc-bbe0-4f43-ae83-7e2ec3bc9d13\") " pod="ai-tenant-e2e-extmod-b-e1f08d/payload-processing-external-model-e2e-extmod-b-e1f08d-9f5cms4gl" Nov 03 01:20:47.684958 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:20:47.684898 4939 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-e1f08d/payload-processing-external-model-e2e-extmod-b-e1f08d-9f5cms4gl" Nov 03 01:20:47.812839 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:20:47.812800 4939 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-d25759/payload-processing-external-model-e2e-extmod-a-d25759-56bbc6hhk"] Nov 03 01:20:47.817787 ip-10-0-140-107 kubenswrapper[4939]: W1103 01:20:47.817755 4939 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8c6f8abe_dd67_45b9_b043_1c1e13ee96de.slice/crio-3860b5f7441b5b2a38486e230a41f5cdd66f47ec738f6aac81ab22a68435402f WatchSource:0}: Error finding container 3860b5f7441b5b2a38486e230a41f5cdd66f47ec738f6aac81ab22a68435402f: Status 404 returned error can't find the container with id 3860b5f7441b5b2a38486e230a41f5cdd66f47ec738f6aac81ab22a68435402f Nov 03 01:20:47.874633 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:20:47.874600 4939 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d59e2793-c378-4558-9b72-25e596ddc473" path="/var/lib/kubelet/pods/d59e2793-c378-4558-9b72-25e596ddc473/volumes" Nov 03 01:20:47.973958 ip-10-0-140-107 kubenswrapper[4939]: W1103 01:20:47.973922 4939 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0a9abcbc_bbe0_4f43_ae83_7e2ec3bc9d13.slice/crio-a54e70125aaa3e541a3ffa324cd658a10dc5df2256c3be61b921362c7f86f70c WatchSource:0}: Error finding container a54e70125aaa3e541a3ffa324cd658a10dc5df2256c3be61b921362c7f86f70c: Status 404 returned error can't find the container with id a54e70125aaa3e541a3ffa324cd658a10dc5df2256c3be61b921362c7f86f70c Nov 03 01:20:47.975093 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:20:47.975069 4939 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-e1f08d/payload-processing-external-model-e2e-extmod-b-e1f08d-9f5cms4gl"] Nov 03 01:20:48.040832 ip-10-0-140-107 kubenswrapper[4939]: E1103 01:20:48.040795 4939 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-sthjh" containerName="extractor" Nov 03 01:20:48.089448 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:20:48.089420 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:20:48.271767 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:20:48.271674 4939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-d25759/payload-processing-external-model-e2e-extmod-a-d25759-56bbc6hhk" event={"ID":"8c6f8abe-dd67-45b9-b043-1c1e13ee96de","Type":"ContainerStarted","Data":"3853ec3202f62b4818fdf238ceb134be23c50838315b4ad8bdef74edad15dcc4"} Nov 03 01:20:48.271767 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:20:48.271743 4939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-d25759/payload-processing-external-model-e2e-extmod-a-d25759-56bbc6hhk" event={"ID":"8c6f8abe-dd67-45b9-b043-1c1e13ee96de","Type":"ContainerStarted","Data":"3860b5f7441b5b2a38486e230a41f5cdd66f47ec738f6aac81ab22a68435402f"} Nov 03 01:20:48.272068 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:20:48.271902 4939 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-a-d25759/payload-processing-external-model-e2e-extmod-a-d25759-56bbc6hhk" Nov 03 01:20:48.273614 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:20:48.273581 4939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-e1f08d/payload-processing-external-model-e2e-extmod-b-e1f08d-9f5cms4gl" event={"ID":"0a9abcbc-bbe0-4f43-ae83-7e2ec3bc9d13","Type":"ContainerStarted","Data":"4359c6b272ecb521ff8a09fef0cb1539a23d09da77aec9c0c81efba9718511f7"} Nov 03 01:20:48.274212 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:20:48.274191 4939 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-b-e1f08d/payload-processing-external-model-e2e-extmod-b-e1f08d-9f5cms4gl" Nov 03 01:20:48.274339 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:20:48.274229 4939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-e1f08d/payload-processing-external-model-e2e-extmod-b-e1f08d-9f5cms4gl" event={"ID":"0a9abcbc-bbe0-4f43-ae83-7e2ec3bc9d13","Type":"ContainerStarted","Data":"a54e70125aaa3e541a3ffa324cd658a10dc5df2256c3be61b921362c7f86f70c"} Nov 03 01:20:48.293606 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:20:48.293551 4939 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-a-d25759/payload-processing-external-model-e2e-extmod-a-d25759-56bbc6hhk" podStartSLOduration=2.293536519 podStartE2EDuration="2.293536519s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 01:20:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 01:20:48.288128575 +0000 UTC m=+2396.552463548" watchObservedRunningTime="2026-11-03 01:20:48.293536519 +0000 UTC m=+2396.557871464" Nov 03 01:20:48.310670 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:20:48.310630 4939 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-b-e1f08d/payload-processing-external-model-e2e-extmod-b-e1f08d-9f5cms4gl" podStartSLOduration=1.310617458 podStartE2EDuration="1.310617458s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 01:20:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 01:20:48.306509802 +0000 UTC m=+2396.570844859" watchObservedRunningTime="2026-11-03 01:20:48.310617458 +0000 UTC m=+2396.574952425" Nov 03 01:20:48.672637 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:20:48.672563 4939 kuberuntime_container.go:898] "Killing container with a grace period" pod="models-as-a-service/payload-processing-external-model-59dbb77fb8-grz4z" podUID="8327ba11-84f8-4660-89a4-51433d2c3a0e" containerName="payload-processing" containerID="cri-o://ea7609644d99ab7751f2da5bf272ad43f1764f8807473eab1bac8d2ea5c8b09e" gracePeriod=25 Nov 03 01:20:48.793803 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:20:48.793774 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:20:49.069727 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:20:49.069697 4939 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-59dbb77fb8-grz4z" Nov 03 01:20:49.188924 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:20:49.188886 4939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8327ba11-84f8-4660-89a4-51433d2c3a0e-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/8327ba11-84f8-4660-89a4-51433d2c3a0e-routing-overlay-volume\") pod \"8327ba11-84f8-4660-89a4-51433d2c3a0e\" (UID: \"8327ba11-84f8-4660-89a4-51433d2c3a0e\") " Nov 03 01:20:49.189101 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:20:49.188962 4939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8327ba11-84f8-4660-89a4-51433d2c3a0e-provider-credentials\" (UniqueName: \"kubernetes.io/projected/8327ba11-84f8-4660-89a4-51433d2c3a0e-provider-credentials\") pod \"8327ba11-84f8-4660-89a4-51433d2c3a0e\" (UID: \"8327ba11-84f8-4660-89a4-51433d2c3a0e\") " Nov 03 01:20:49.189101 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:20:49.189072 4939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8327ba11-84f8-4660-89a4-51433d2c3a0e-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8327ba11-84f8-4660-89a4-51433d2c3a0e-plugins-config-volume\") pod \"8327ba11-84f8-4660-89a4-51433d2c3a0e\" (UID: \"8327ba11-84f8-4660-89a4-51433d2c3a0e\") " Nov 03 01:20:49.189177 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:20:49.189119 4939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8327ba11-84f8-4660-89a4-51433d2c3a0e-kube-api-access-fswng\" (UniqueName: \"kubernetes.io/projected/8327ba11-84f8-4660-89a4-51433d2c3a0e-kube-api-access-fswng\") pod \"8327ba11-84f8-4660-89a4-51433d2c3a0e\" (UID: \"8327ba11-84f8-4660-89a4-51433d2c3a0e\") " Nov 03 01:20:49.189369 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:20:49.189338 4939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8327ba11-84f8-4660-89a4-51433d2c3a0e-routing-overlay-volume" pod "8327ba11-84f8-4660-89a4-51433d2c3a0e" (UID: "8327ba11-84f8-4660-89a4-51433d2c3a0e"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 01:20:49.189533 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:20:49.189510 4939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8327ba11-84f8-4660-89a4-51433d2c3a0e-plugins-config-volume" pod "8327ba11-84f8-4660-89a4-51433d2c3a0e" (UID: "8327ba11-84f8-4660-89a4-51433d2c3a0e"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 01:20:49.190248 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:20:49.190224 4939 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/8327ba11-84f8-4660-89a4-51433d2c3a0e-routing-overlay-volume\") on node \"ip-10-0-140-107.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:20:49.190248 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:20:49.190248 4939 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8327ba11-84f8-4660-89a4-51433d2c3a0e-plugins-config-volume\") on node \"ip-10-0-140-107.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:20:49.191342 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:20:49.191316 4939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8327ba11-84f8-4660-89a4-51433d2c3a0e-provider-credentials" pod "8327ba11-84f8-4660-89a4-51433d2c3a0e" (UID: "8327ba11-84f8-4660-89a4-51433d2c3a0e"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 01:20:49.191429 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:20:49.191342 4939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8327ba11-84f8-4660-89a4-51433d2c3a0e-kube-api-access-fswng" pod "8327ba11-84f8-4660-89a4-51433d2c3a0e" (UID: "8327ba11-84f8-4660-89a4-51433d2c3a0e"). InnerVolumeSpecName "kube-api-access-fswng". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 01:20:49.285018 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:20:49.284976 4939 generic.go:356] "Generic (PLEG): container finished" podID="8327ba11-84f8-4660-89a4-51433d2c3a0e" containerID="ea7609644d99ab7751f2da5bf272ad43f1764f8807473eab1bac8d2ea5c8b09e" exitCode=0 Nov 03 01:20:49.285190 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:20:49.285023 4939 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-59dbb77fb8-grz4z" Nov 03 01:20:49.285190 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:20:49.285061 4939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-59dbb77fb8-grz4z" event={"ID":"8327ba11-84f8-4660-89a4-51433d2c3a0e","Type":"ContainerDied","Data":"ea7609644d99ab7751f2da5bf272ad43f1764f8807473eab1bac8d2ea5c8b09e"} Nov 03 01:20:49.285190 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:20:49.285122 4939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-59dbb77fb8-grz4z" event={"ID":"8327ba11-84f8-4660-89a4-51433d2c3a0e","Type":"ContainerDied","Data":"357b34948b382451ac5bb3da7f8b4e7ce67310d9f6d87d502016c9ba915dca5f"} Nov 03 01:20:49.285190 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:20:49.285151 4939 scope.go:122] "RemoveContainer" containerID="ea7609644d99ab7751f2da5bf272ad43f1764f8807473eab1bac8d2ea5c8b09e" Nov 03 01:20:49.291340 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:20:49.291313 4939 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/8327ba11-84f8-4660-89a4-51433d2c3a0e-provider-credentials\") on node \"ip-10-0-140-107.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:20:49.291456 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:20:49.291345 4939 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fswng\" (UniqueName: \"kubernetes.io/projected/8327ba11-84f8-4660-89a4-51433d2c3a0e-kube-api-access-fswng\") on node \"ip-10-0-140-107.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:20:49.300006 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:20:49.299990 4939 scope.go:122] "RemoveContainer" containerID="ea7609644d99ab7751f2da5bf272ad43f1764f8807473eab1bac8d2ea5c8b09e" Nov 03 01:20:49.300296 ip-10-0-140-107 kubenswrapper[4939]: E1103 01:20:49.300278 4939 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ea7609644d99ab7751f2da5bf272ad43f1764f8807473eab1bac8d2ea5c8b09e\": container with ID starting with ea7609644d99ab7751f2da5bf272ad43f1764f8807473eab1bac8d2ea5c8b09e not found: ID does not exist" containerID="ea7609644d99ab7751f2da5bf272ad43f1764f8807473eab1bac8d2ea5c8b09e" Nov 03 01:20:49.300355 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:20:49.300304 4939 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ea7609644d99ab7751f2da5bf272ad43f1764f8807473eab1bac8d2ea5c8b09e"} err="failed to get container status \"ea7609644d99ab7751f2da5bf272ad43f1764f8807473eab1bac8d2ea5c8b09e\": rpc error: code = NotFound desc = could not find container \"ea7609644d99ab7751f2da5bf272ad43f1764f8807473eab1bac8d2ea5c8b09e\": container with ID starting with ea7609644d99ab7751f2da5bf272ad43f1764f8807473eab1bac8d2ea5c8b09e not found: ID does not exist" Nov 03 01:20:49.312175 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:20:49.312149 4939 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["models-as-a-service/payload-processing-external-model-59dbb77fb8-grz4z"] Nov 03 01:20:49.317159 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:20:49.317138 4939 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["models-as-a-service/payload-processing-external-model-59dbb77fb8-grz4z"] Nov 03 01:20:49.497839 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:20:49.497758 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:20:49.872111 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:20:49.872076 4939 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8327ba11-84f8-4660-89a4-51433d2c3a0e" path="/var/lib/kubelet/pods/8327ba11-84f8-4660-89a4-51433d2c3a0e/volumes" Nov 03 01:20:50.194887 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:20:50.194808 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:20:50.795661 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:20:50.795623 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:20:51.505056 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:20:51.505029 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:20:52.166502 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:20:52.166470 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:20:52.560576 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:20:52.560503 4939 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-a-7c759a/payload-processing-external-model-e2e-extmod-a-7c759a-799fk2rjw" Nov 03 01:20:52.759438 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:20:52.759407 4939 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-n42s6_65d134ac-8a25-4c67-b6c7-cd752a098daf/console-operator/2.log" Nov 03 01:20:52.826194 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:20:52.826113 4939 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-n42s6_65d134ac-8a25-4c67-b6c7-cd752a098daf/console-operator/2.log" Nov 03 01:20:52.878846 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:20:52.878811 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:20:53.579669 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:20:53.579634 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:20:54.219546 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:20:54.219509 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:20:54.877672 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:20:54.877633 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:20:55.722565 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:20:55.722532 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:20:57.191766 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:20:57.177716 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:20:57.282245 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:20:57.282096 4939 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="models-as-a-service/payload-processing-external-model-76c77f5454-842p6" Nov 03 01:20:57.384231 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:20:57.384193 4939 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["models-as-a-service/payload-processing-external-model-67757c7bbc-7lndz"] Nov 03 01:20:58.125742 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:20:58.125705 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:20:58.284231 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:20:58.284191 4939 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-a-d25759/payload-processing-external-model-e2e-extmod-a-d25759-579bb87bp" Nov 03 01:20:59.345936 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:20:59.340095 4939 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-b-e1f08d/payload-processing-external-model-e2e-extmod-b-e1f08d-6cf6k74l8" Nov 03 01:20:59.345936 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:20:59.341640 4939 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-a-d25759/payload-processing-external-model-e2e-extmod-a-d25759-56bbc6hhk" Nov 03 01:20:59.484117 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:20:59.484063 4939 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-88bbf7/distinct-2-w1-88bbf7-kserve-5457cdcdbd-l6tfd" podUID="a397931b-faa9-4c2e-8290-491350546663" containerName="main" probeResult="failure" output="Get \"https://10.132.0.85:8000/ready\": context deadline exceeded" Nov 03 01:20:59.574163 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:20:59.568041 4939 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-83755a/distinct-2-w3-83755a-kserve-6cf66b995d-2l47b" podUID="c3b05e88-5d5b-4d03-b28d-832f22e3c868" containerName="main" probeResult="failure" output="Get \"https://10.132.0.72:8000/ready\": context deadline exceeded" Nov 03 01:20:59.597987 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:20:59.579717 4939 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-f7680d/distinct-2-w2-f7680d-kserve-7d7c4564bf-cbv2c" podUID="2de18086-761b-4773-b9fc-15ddc4a220fc" containerName="main" probeResult="failure" output="Get \"https://10.132.0.75:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 01:20:59.597987 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:20:59.581874 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:20:59.834933 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:20:59.834051 4939 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-f7680d/premium-w2-f7680d-kserve-76565b8cf9-hqbbt" podUID="4712e708-18a2-4a13-a12c-a5669397b2c3" containerName="main" probeResult="failure" output="Get \"https://10.132.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 01:20:59.954992 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:20:59.939059 4939 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-2f9a3a/distinct-w0-2f9a3a-kserve-555565cfcf-jtcr9" podUID="ac0ece06-8718-463f-915f-5315988d9711" containerName="main" probeResult="failure" output="Get \"https://10.132.0.76:8000/ready\": context deadline exceeded" Nov 03 01:20:59.958385 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:20:59.958327 4939 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-2f9a3a/simulator-w0-2f9a3a-kserve-9596bbb8f-2trgz" podUID="207e2207-fb6c-4076-95df-72e95671c261" containerName="main" probeResult="failure" output="Get \"https://10.132.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 01:21:00.055243 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:21:00.055196 4939 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-f7680d/distinct-w2-f7680d-kserve-756f874b98-qnpv8" podUID="836753fc-f9ef-4f61-b3fe-e5b92b7ee27b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.70:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 01:21:00.224985 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:21:00.224287 4939 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-d25759/payload-processing-external-model-e2e-extmod-a-d25759-579bb87bp"] Nov 03 01:21:00.336935 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:21:00.334975 4939 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-b-e1f08d/payload-processing-external-model-e2e-extmod-b-e1f08d-9f5cms4gl" Nov 03 01:21:00.481957 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:21:00.481846 4939 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-e1f08d/payload-processing-external-model-e2e-extmod-b-e1f08d-6cf6k74l8"] Nov 03 01:21:00.907866 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:21:00.907830 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:21:01.631929 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:21:01.630859 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:21:02.385525 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:21:02.385467 4939 kuberuntime_container.go:898] "Killing container with a grace period" pod="models-as-a-service/payload-processing-external-model-67757c7bbc-7lndz" podUID="12cc6461-f4d9-4878-b247-2e6ccf21512c" containerName="payload-processing" containerID="cri-o://e6de5b172d56fcec7d3fb67314719b9df19bef6a14a8712b4139422406fa4f89" gracePeriod=25 Nov 03 01:21:02.502271 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:21:02.502245 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:21:02.761992 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:21:02.761963 4939 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-67757c7bbc-7lndz" Nov 03 01:21:02.785693 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:21:02.785667 4939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/12cc6461-f4d9-4878-b247-2e6ccf21512c-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/12cc6461-f4d9-4878-b247-2e6ccf21512c-plugins-config-volume\") pod \"12cc6461-f4d9-4878-b247-2e6ccf21512c\" (UID: \"12cc6461-f4d9-4878-b247-2e6ccf21512c\") " Nov 03 01:21:02.785900 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:21:02.785815 4939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/12cc6461-f4d9-4878-b247-2e6ccf21512c-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/12cc6461-f4d9-4878-b247-2e6ccf21512c-routing-overlay-volume\") pod \"12cc6461-f4d9-4878-b247-2e6ccf21512c\" (UID: \"12cc6461-f4d9-4878-b247-2e6ccf21512c\") " Nov 03 01:21:02.785978 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:21:02.785965 4939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/12cc6461-f4d9-4878-b247-2e6ccf21512c-provider-credentials\" (UniqueName: \"kubernetes.io/projected/12cc6461-f4d9-4878-b247-2e6ccf21512c-provider-credentials\") pod \"12cc6461-f4d9-4878-b247-2e6ccf21512c\" (UID: \"12cc6461-f4d9-4878-b247-2e6ccf21512c\") " Nov 03 01:21:02.786073 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:21:02.786057 4939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/12cc6461-f4d9-4878-b247-2e6ccf21512c-kube-api-access-dv2fl\" (UniqueName: \"kubernetes.io/projected/12cc6461-f4d9-4878-b247-2e6ccf21512c-kube-api-access-dv2fl\") pod \"12cc6461-f4d9-4878-b247-2e6ccf21512c\" (UID: \"12cc6461-f4d9-4878-b247-2e6ccf21512c\") " Nov 03 01:21:02.786261 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:21:02.786233 4939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/12cc6461-f4d9-4878-b247-2e6ccf21512c-plugins-config-volume" pod "12cc6461-f4d9-4878-b247-2e6ccf21512c" (UID: "12cc6461-f4d9-4878-b247-2e6ccf21512c"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 01:21:02.786351 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:21:02.786258 4939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/12cc6461-f4d9-4878-b247-2e6ccf21512c-routing-overlay-volume" pod "12cc6461-f4d9-4878-b247-2e6ccf21512c" (UID: "12cc6461-f4d9-4878-b247-2e6ccf21512c"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 01:21:02.786956 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:21:02.786931 4939 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/12cc6461-f4d9-4878-b247-2e6ccf21512c-routing-overlay-volume\") on node \"ip-10-0-140-107.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:21:02.787064 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:21:02.786958 4939 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/12cc6461-f4d9-4878-b247-2e6ccf21512c-plugins-config-volume\") on node \"ip-10-0-140-107.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:21:02.788440 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:21:02.788387 4939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/12cc6461-f4d9-4878-b247-2e6ccf21512c-provider-credentials" pod "12cc6461-f4d9-4878-b247-2e6ccf21512c" (UID: "12cc6461-f4d9-4878-b247-2e6ccf21512c"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 01:21:02.788682 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:21:02.788656 4939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/12cc6461-f4d9-4878-b247-2e6ccf21512c-kube-api-access-dv2fl" pod "12cc6461-f4d9-4878-b247-2e6ccf21512c" (UID: "12cc6461-f4d9-4878-b247-2e6ccf21512c"). InnerVolumeSpecName "kube-api-access-dv2fl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 01:21:02.888333 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:21:02.888299 4939 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/12cc6461-f4d9-4878-b247-2e6ccf21512c-provider-credentials\") on node \"ip-10-0-140-107.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:21:02.888561 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:21:02.888536 4939 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dv2fl\" (UniqueName: \"kubernetes.io/projected/12cc6461-f4d9-4878-b247-2e6ccf21512c-kube-api-access-dv2fl\") on node \"ip-10-0-140-107.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:21:02.989374 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:21:02.989249 4939 generic.go:356] "Generic (PLEG): container finished" podID="12cc6461-f4d9-4878-b247-2e6ccf21512c" containerID="e6de5b172d56fcec7d3fb67314719b9df19bef6a14a8712b4139422406fa4f89" exitCode=0 Nov 03 01:21:02.989374 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:21:02.989309 4939 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-67757c7bbc-7lndz" Nov 03 01:21:02.989612 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:21:02.989531 4939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-67757c7bbc-7lndz" event={"ID":"12cc6461-f4d9-4878-b247-2e6ccf21512c","Type":"ContainerDied","Data":"e6de5b172d56fcec7d3fb67314719b9df19bef6a14a8712b4139422406fa4f89"} Nov 03 01:21:02.989612 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:21:02.989577 4939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-67757c7bbc-7lndz" event={"ID":"12cc6461-f4d9-4878-b247-2e6ccf21512c","Type":"ContainerDied","Data":"514f09eea528894019b72f9c95237cd7bfa467bb7f1f47d641b5eb46343cf837"} Nov 03 01:21:02.989612 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:21:02.989598 4939 scope.go:122] "RemoveContainer" containerID="e6de5b172d56fcec7d3fb67314719b9df19bef6a14a8712b4139422406fa4f89" Nov 03 01:21:03.002744 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:21:03.002718 4939 scope.go:122] "RemoveContainer" containerID="e6de5b172d56fcec7d3fb67314719b9df19bef6a14a8712b4139422406fa4f89" Nov 03 01:21:03.003080 ip-10-0-140-107 kubenswrapper[4939]: E1103 01:21:03.003059 4939 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e6de5b172d56fcec7d3fb67314719b9df19bef6a14a8712b4139422406fa4f89\": container with ID starting with e6de5b172d56fcec7d3fb67314719b9df19bef6a14a8712b4139422406fa4f89 not found: ID does not exist" containerID="e6de5b172d56fcec7d3fb67314719b9df19bef6a14a8712b4139422406fa4f89" Nov 03 01:21:03.003189 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:21:03.003090 4939 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e6de5b172d56fcec7d3fb67314719b9df19bef6a14a8712b4139422406fa4f89"} err="failed to get container status \"e6de5b172d56fcec7d3fb67314719b9df19bef6a14a8712b4139422406fa4f89\": rpc error: code = NotFound desc = could not find container \"e6de5b172d56fcec7d3fb67314719b9df19bef6a14a8712b4139422406fa4f89\": container with ID starting with e6de5b172d56fcec7d3fb67314719b9df19bef6a14a8712b4139422406fa4f89 not found: ID does not exist" Nov 03 01:21:03.019531 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:21:03.019509 4939 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["models-as-a-service/payload-processing-external-model-67757c7bbc-7lndz"] Nov 03 01:21:03.024579 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:21:03.024555 4939 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["models-as-a-service/payload-processing-external-model-67757c7bbc-7lndz"] Nov 03 01:21:03.195521 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:21:03.195489 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:21:03.799814 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:21:03.799782 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:21:03.906902 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:21:03.906864 4939 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="12cc6461-f4d9-4878-b247-2e6ccf21512c" path="/var/lib/kubelet/pods/12cc6461-f4d9-4878-b247-2e6ccf21512c/volumes" Nov 03 01:21:04.417128 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:21:04.417091 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:21:05.099425 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:21:05.099390 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:21:05.226072 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:21:05.226014 4939 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-a-d25759/payload-processing-external-model-e2e-extmod-a-d25759-579bb87bp" podUID="2151f14f-ab60-4499-aa61-cc42f8404e86" containerName="payload-processing" containerID="cri-o://4e8babc8b437e5a8125135de09dceda45731e5317170b77a97979446f32f62c0" gracePeriod=25 Nov 03 01:21:05.483522 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:21:05.483475 4939 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-b-e1f08d/payload-processing-external-model-e2e-extmod-b-e1f08d-6cf6k74l8" podUID="72e20aa3-0a6c-4e0d-b51b-891b86a7607e" containerName="payload-processing" containerID="cri-o://700559a30f523fae6ed4b034bbed1a4124deb73b8948fc9ae767ade3ecc02a98" gracePeriod=25 Nov 03 01:21:05.630241 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:21:05.630206 4939 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-d25759/payload-processing-external-model-e2e-extmod-a-d25759-579bb87bp" Nov 03 01:21:05.737730 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:21:05.737684 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:21:05.753313 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:21:05.753283 4939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2151f14f-ab60-4499-aa61-cc42f8404e86-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2151f14f-ab60-4499-aa61-cc42f8404e86-plugins-config-volume\") pod \"2151f14f-ab60-4499-aa61-cc42f8404e86\" (UID: \"2151f14f-ab60-4499-aa61-cc42f8404e86\") " Nov 03 01:21:05.753504 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:21:05.753489 4939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2151f14f-ab60-4499-aa61-cc42f8404e86-provider-credentials\" (UniqueName: \"kubernetes.io/projected/2151f14f-ab60-4499-aa61-cc42f8404e86-provider-credentials\") pod \"2151f14f-ab60-4499-aa61-cc42f8404e86\" (UID: \"2151f14f-ab60-4499-aa61-cc42f8404e86\") " Nov 03 01:21:05.753608 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:21:05.753589 4939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2151f14f-ab60-4499-aa61-cc42f8404e86-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/2151f14f-ab60-4499-aa61-cc42f8404e86-routing-overlay-volume\") pod \"2151f14f-ab60-4499-aa61-cc42f8404e86\" (UID: \"2151f14f-ab60-4499-aa61-cc42f8404e86\") " Nov 03 01:21:05.753678 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:21:05.753663 4939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2151f14f-ab60-4499-aa61-cc42f8404e86-kube-api-access-zzjvj\" (UniqueName: \"kubernetes.io/projected/2151f14f-ab60-4499-aa61-cc42f8404e86-kube-api-access-zzjvj\") pod \"2151f14f-ab60-4499-aa61-cc42f8404e86\" (UID: \"2151f14f-ab60-4499-aa61-cc42f8404e86\") " Nov 03 01:21:05.753771 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:21:05.753745 4939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2151f14f-ab60-4499-aa61-cc42f8404e86-plugins-config-volume" pod "2151f14f-ab60-4499-aa61-cc42f8404e86" (UID: "2151f14f-ab60-4499-aa61-cc42f8404e86"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 01:21:05.754044 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:21:05.754016 4939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2151f14f-ab60-4499-aa61-cc42f8404e86-routing-overlay-volume" pod "2151f14f-ab60-4499-aa61-cc42f8404e86" (UID: "2151f14f-ab60-4499-aa61-cc42f8404e86"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 01:21:05.754837 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:21:05.754807 4939 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2151f14f-ab60-4499-aa61-cc42f8404e86-plugins-config-volume\") on node \"ip-10-0-140-107.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:21:05.754997 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:21:05.754843 4939 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/2151f14f-ab60-4499-aa61-cc42f8404e86-routing-overlay-volume\") on node \"ip-10-0-140-107.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:21:05.756193 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:21:05.756169 4939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2151f14f-ab60-4499-aa61-cc42f8404e86-provider-credentials" pod "2151f14f-ab60-4499-aa61-cc42f8404e86" (UID: "2151f14f-ab60-4499-aa61-cc42f8404e86"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 01:21:05.756516 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:21:05.756490 4939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2151f14f-ab60-4499-aa61-cc42f8404e86-kube-api-access-zzjvj" pod "2151f14f-ab60-4499-aa61-cc42f8404e86" (UID: "2151f14f-ab60-4499-aa61-cc42f8404e86"). InnerVolumeSpecName "kube-api-access-zzjvj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 01:21:05.808959 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:21:05.804126 4939 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-e1f08d/payload-processing-external-model-e2e-extmod-b-e1f08d-6cf6k74l8" Nov 03 01:21:05.874681 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:21:05.874636 4939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/72e20aa3-0a6c-4e0d-b51b-891b86a7607e-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/72e20aa3-0a6c-4e0d-b51b-891b86a7607e-plugins-config-volume\") pod \"72e20aa3-0a6c-4e0d-b51b-891b86a7607e\" (UID: \"72e20aa3-0a6c-4e0d-b51b-891b86a7607e\") " Nov 03 01:21:05.874899 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:21:05.874867 4939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/72e20aa3-0a6c-4e0d-b51b-891b86a7607e-provider-credentials\" (UniqueName: \"kubernetes.io/projected/72e20aa3-0a6c-4e0d-b51b-891b86a7607e-provider-credentials\") pod \"72e20aa3-0a6c-4e0d-b51b-891b86a7607e\" (UID: \"72e20aa3-0a6c-4e0d-b51b-891b86a7607e\") " Nov 03 01:21:05.876606 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:21:05.874956 4939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/72e20aa3-0a6c-4e0d-b51b-891b86a7607e-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/72e20aa3-0a6c-4e0d-b51b-891b86a7607e-routing-overlay-volume\") pod \"72e20aa3-0a6c-4e0d-b51b-891b86a7607e\" (UID: \"72e20aa3-0a6c-4e0d-b51b-891b86a7607e\") " Nov 03 01:21:05.876606 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:21:05.875039 4939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/72e20aa3-0a6c-4e0d-b51b-891b86a7607e-kube-api-access-wfrzg\" (UniqueName: \"kubernetes.io/projected/72e20aa3-0a6c-4e0d-b51b-891b86a7607e-kube-api-access-wfrzg\") pod \"72e20aa3-0a6c-4e0d-b51b-891b86a7607e\" (UID: \"72e20aa3-0a6c-4e0d-b51b-891b86a7607e\") " Nov 03 01:21:05.876817 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:21:05.876764 4939 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/2151f14f-ab60-4499-aa61-cc42f8404e86-provider-credentials\") on node \"ip-10-0-140-107.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:21:05.876817 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:21:05.876791 4939 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zzjvj\" (UniqueName: \"kubernetes.io/projected/2151f14f-ab60-4499-aa61-cc42f8404e86-kube-api-access-zzjvj\") on node \"ip-10-0-140-107.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:21:05.884767 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:21:05.884719 4939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/72e20aa3-0a6c-4e0d-b51b-891b86a7607e-provider-credentials" pod "72e20aa3-0a6c-4e0d-b51b-891b86a7607e" (UID: "72e20aa3-0a6c-4e0d-b51b-891b86a7607e"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 01:21:05.885836 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:21:05.885806 4939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/72e20aa3-0a6c-4e0d-b51b-891b86a7607e-routing-overlay-volume" pod "72e20aa3-0a6c-4e0d-b51b-891b86a7607e" (UID: "72e20aa3-0a6c-4e0d-b51b-891b86a7607e"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 01:21:05.899872 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:21:05.895145 4939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/72e20aa3-0a6c-4e0d-b51b-891b86a7607e-plugins-config-volume" pod "72e20aa3-0a6c-4e0d-b51b-891b86a7607e" (UID: "72e20aa3-0a6c-4e0d-b51b-891b86a7607e"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 01:21:05.907923 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:21:05.906405 4939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/72e20aa3-0a6c-4e0d-b51b-891b86a7607e-kube-api-access-wfrzg" pod "72e20aa3-0a6c-4e0d-b51b-891b86a7607e" (UID: "72e20aa3-0a6c-4e0d-b51b-891b86a7607e"). InnerVolumeSpecName "kube-api-access-wfrzg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 01:21:05.979411 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:21:05.979378 4939 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/72e20aa3-0a6c-4e0d-b51b-891b86a7607e-plugins-config-volume\") on node \"ip-10-0-140-107.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:21:05.979411 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:21:05.979415 4939 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/72e20aa3-0a6c-4e0d-b51b-891b86a7607e-provider-credentials\") on node \"ip-10-0-140-107.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:21:05.979607 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:21:05.979430 4939 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/72e20aa3-0a6c-4e0d-b51b-891b86a7607e-routing-overlay-volume\") on node \"ip-10-0-140-107.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:21:05.979607 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:21:05.979444 4939 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wfrzg\" (UniqueName: \"kubernetes.io/projected/72e20aa3-0a6c-4e0d-b51b-891b86a7607e-kube-api-access-wfrzg\") on node \"ip-10-0-140-107.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:21:06.020128 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:21:06.020071 4939 generic.go:356] "Generic (PLEG): container finished" podID="72e20aa3-0a6c-4e0d-b51b-891b86a7607e" containerID="700559a30f523fae6ed4b034bbed1a4124deb73b8948fc9ae767ade3ecc02a98" exitCode=0 Nov 03 01:21:06.020342 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:21:06.020132 4939 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-e1f08d/payload-processing-external-model-e2e-extmod-b-e1f08d-6cf6k74l8" Nov 03 01:21:06.020342 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:21:06.020160 4939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-e1f08d/payload-processing-external-model-e2e-extmod-b-e1f08d-6cf6k74l8" event={"ID":"72e20aa3-0a6c-4e0d-b51b-891b86a7607e","Type":"ContainerDied","Data":"700559a30f523fae6ed4b034bbed1a4124deb73b8948fc9ae767ade3ecc02a98"} Nov 03 01:21:06.020342 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:21:06.020227 4939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-e1f08d/payload-processing-external-model-e2e-extmod-b-e1f08d-6cf6k74l8" event={"ID":"72e20aa3-0a6c-4e0d-b51b-891b86a7607e","Type":"ContainerDied","Data":"3c72b88b03591d15388392b826f5811c40f03883386e8760ce6f773067f814af"} Nov 03 01:21:06.020342 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:21:06.020259 4939 scope.go:122] "RemoveContainer" containerID="700559a30f523fae6ed4b034bbed1a4124deb73b8948fc9ae767ade3ecc02a98" Nov 03 01:21:06.022204 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:21:06.022181 4939 generic.go:356] "Generic (PLEG): container finished" podID="2151f14f-ab60-4499-aa61-cc42f8404e86" containerID="4e8babc8b437e5a8125135de09dceda45731e5317170b77a97979446f32f62c0" exitCode=0 Nov 03 01:21:06.022325 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:21:06.022246 4939 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-d25759/payload-processing-external-model-e2e-extmod-a-d25759-579bb87bp" Nov 03 01:21:06.022325 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:21:06.022270 4939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-d25759/payload-processing-external-model-e2e-extmod-a-d25759-579bb87bp" event={"ID":"2151f14f-ab60-4499-aa61-cc42f8404e86","Type":"ContainerDied","Data":"4e8babc8b437e5a8125135de09dceda45731e5317170b77a97979446f32f62c0"} Nov 03 01:21:06.022507 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:21:06.022325 4939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-d25759/payload-processing-external-model-e2e-extmod-a-d25759-579bb87bp" event={"ID":"2151f14f-ab60-4499-aa61-cc42f8404e86","Type":"ContainerDied","Data":"8f91265768cdccbd96de9e506904ddb3f1226f47b976198ff0b418e15cf0af81"} Nov 03 01:21:06.032250 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:21:06.032230 4939 scope.go:122] "RemoveContainer" containerID="700559a30f523fae6ed4b034bbed1a4124deb73b8948fc9ae767ade3ecc02a98" Nov 03 01:21:06.032545 ip-10-0-140-107 kubenswrapper[4939]: E1103 01:21:06.032527 4939 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"700559a30f523fae6ed4b034bbed1a4124deb73b8948fc9ae767ade3ecc02a98\": container with ID starting with 700559a30f523fae6ed4b034bbed1a4124deb73b8948fc9ae767ade3ecc02a98 not found: ID does not exist" containerID="700559a30f523fae6ed4b034bbed1a4124deb73b8948fc9ae767ade3ecc02a98" Nov 03 01:21:06.032605 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:21:06.032552 4939 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"700559a30f523fae6ed4b034bbed1a4124deb73b8948fc9ae767ade3ecc02a98"} err="failed to get container status \"700559a30f523fae6ed4b034bbed1a4124deb73b8948fc9ae767ade3ecc02a98\": rpc error: code = NotFound desc = could not find container \"700559a30f523fae6ed4b034bbed1a4124deb73b8948fc9ae767ade3ecc02a98\": container with ID starting with 700559a30f523fae6ed4b034bbed1a4124deb73b8948fc9ae767ade3ecc02a98 not found: ID does not exist" Nov 03 01:21:06.032605 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:21:06.032567 4939 scope.go:122] "RemoveContainer" containerID="4e8babc8b437e5a8125135de09dceda45731e5317170b77a97979446f32f62c0" Nov 03 01:21:06.046658 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:21:06.046640 4939 scope.go:122] "RemoveContainer" containerID="4e8babc8b437e5a8125135de09dceda45731e5317170b77a97979446f32f62c0" Nov 03 01:21:06.047002 ip-10-0-140-107 kubenswrapper[4939]: E1103 01:21:06.046976 4939 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4e8babc8b437e5a8125135de09dceda45731e5317170b77a97979446f32f62c0\": container with ID starting with 4e8babc8b437e5a8125135de09dceda45731e5317170b77a97979446f32f62c0 not found: ID does not exist" containerID="4e8babc8b437e5a8125135de09dceda45731e5317170b77a97979446f32f62c0" Nov 03 01:21:06.047091 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:21:06.047012 4939 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4e8babc8b437e5a8125135de09dceda45731e5317170b77a97979446f32f62c0"} err="failed to get container status \"4e8babc8b437e5a8125135de09dceda45731e5317170b77a97979446f32f62c0\": rpc error: code = NotFound desc = could not find container \"4e8babc8b437e5a8125135de09dceda45731e5317170b77a97979446f32f62c0\": container with ID starting with 4e8babc8b437e5a8125135de09dceda45731e5317170b77a97979446f32f62c0 not found: ID does not exist" Nov 03 01:21:06.047091 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:21:06.047054 4939 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-e1f08d/payload-processing-external-model-e2e-extmod-b-e1f08d-6cf6k74l8"] Nov 03 01:21:06.051803 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:21:06.051782 4939 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-b-e1f08d/payload-processing-external-model-e2e-extmod-b-e1f08d-6cf6k74l8"] Nov 03 01:21:06.057143 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:21:06.057124 4939 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-d25759/payload-processing-external-model-e2e-extmod-a-d25759-579bb87bp"] Nov 03 01:21:06.062684 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:21:06.062664 4939 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-a-d25759/payload-processing-external-model-e2e-extmod-a-d25759-579bb87bp"] Nov 03 01:21:06.374634 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:21:06.374604 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:21:07.085026 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:21:07.084991 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:21:07.757726 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:21:07.757694 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:21:07.878242 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:21:07.878210 4939 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2151f14f-ab60-4499-aa61-cc42f8404e86" path="/var/lib/kubelet/pods/2151f14f-ab60-4499-aa61-cc42f8404e86/volumes" Nov 03 01:21:07.878755 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:21:07.878736 4939 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="72e20aa3-0a6c-4e0d-b51b-891b86a7607e" path="/var/lib/kubelet/pods/72e20aa3-0a6c-4e0d-b51b-891b86a7607e/volumes" Nov 03 01:21:08.512268 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:21:08.512219 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:21:09.101632 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:21:09.101587 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:21:09.723749 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:21:09.723716 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:21:09.841563 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:21:09.841529 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:21:10.374180 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:21:10.374146 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:21:11.002019 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:21:11.001981 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:21:11.673667 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:21:11.673630 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:21:12.447752 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:21:12.447719 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:21:13.077155 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:21:13.077121 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:21:13.775707 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:21:13.775624 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:21:14.504767 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:21:14.504735 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:21:15.163990 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:21:15.163952 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:21:15.832508 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:21:15.832455 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:21:16.492257 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:21:16.492191 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:21:17.171711 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:21:17.171677 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:21:17.775725 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:21:17.775688 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:21:18.033698 ip-10-0-140-107 kubenswrapper[4939]: E1103 01:21:18.033607 4939 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-sthjh" containerName="extractor" Nov 03 01:21:18.492150 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:21:18.492118 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:21:19.105952 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:21:19.105918 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:21:19.773380 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:21:19.773339 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:21:20.377085 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:21:20.377048 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:21:20.974464 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:21:20.974431 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:21:21.691241 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:21:21.691204 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:21:22.392057 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:21:22.392024 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:21:23.116265 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:21:23.116211 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:21:23.375539 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:21:23.375396 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:21:23.610398 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:21:23.588664 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:21:23.822924 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:21:23.822813 4939 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-a-027c19/payload-processing-external-model-e2e-extmod-a-027c19-7974gqxtk"] Nov 03 01:21:23.825038 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:21:23.825008 4939 cpu_manager.go:414] "containerMap: removing container" podUID="8327ba11-84f8-4660-89a4-51433d2c3a0e" containerName="payload-processing" Nov 03 01:21:23.825038 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:21:23.825038 4939 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8327ba11-84f8-4660-89a4-51433d2c3a0e" containerName="payload-processing" Nov 03 01:21:23.825252 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:21:23.825235 4939 cpu_manager.go:414] "containerMap: removing container" podUID="2151f14f-ab60-4499-aa61-cc42f8404e86" containerName="payload-processing" Nov 03 01:21:23.825318 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:21:23.825250 4939 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2151f14f-ab60-4499-aa61-cc42f8404e86" containerName="payload-processing" Nov 03 01:21:23.825381 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:21:23.825344 4939 cpu_manager.go:414] "containerMap: removing container" podUID="72e20aa3-0a6c-4e0d-b51b-891b86a7607e" containerName="payload-processing" Nov 03 01:21:23.825381 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:21:23.825355 4939 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="72e20aa3-0a6c-4e0d-b51b-891b86a7607e" containerName="payload-processing" Nov 03 01:21:23.825501 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:21:23.825412 4939 cpu_manager.go:414] "containerMap: removing container" podUID="12cc6461-f4d9-4878-b247-2e6ccf21512c" containerName="payload-processing" Nov 03 01:21:23.825501 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:21:23.825420 4939 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="12cc6461-f4d9-4878-b247-2e6ccf21512c" containerName="payload-processing" Nov 03 01:21:23.825864 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:21:23.825842 4939 memory_manager.go:365] "RemoveStaleState removing state" podUID="72e20aa3-0a6c-4e0d-b51b-891b86a7607e" containerName="payload-processing" Nov 03 01:21:23.826031 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:21:23.826013 4939 memory_manager.go:365] "RemoveStaleState removing state" podUID="12cc6461-f4d9-4878-b247-2e6ccf21512c" containerName="payload-processing" Nov 03 01:21:23.826104 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:21:23.826050 4939 memory_manager.go:365] "RemoveStaleState removing state" podUID="8327ba11-84f8-4660-89a4-51433d2c3a0e" containerName="payload-processing" Nov 03 01:21:23.826168 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:21:23.826105 4939 memory_manager.go:365] "RemoveStaleState removing state" podUID="2151f14f-ab60-4499-aa61-cc42f8404e86" containerName="payload-processing" Nov 03 01:21:23.833080 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:21:23.833053 4939 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-027c19/payload-processing-external-model-e2e-extmod-a-027c19-7974gqxtk" Nov 03 01:21:23.834373 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:21:23.834345 4939 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-027c19/payload-processing-external-model-e2e-extmod-a-027c19-7974gqxtk"] Nov 03 01:21:23.924641 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:21:23.924600 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a33ae0fa-7699-4c6c-8628-af11765d22df-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-027c19-7974gqxtk\" (UID: \"a33ae0fa-7699-4c6c-8628-af11765d22df\") " pod="ai-tenant-e2e-extmod-a-027c19/payload-processing-external-model-e2e-extmod-a-027c19-7974gqxtk" Nov 03 01:21:23.924842 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:21:23.924693 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5h6rz\" (UniqueName: \"kubernetes.io/projected/a33ae0fa-7699-4c6c-8628-af11765d22df-kube-api-access-5h6rz\") pod \"payload-processing-external-model-e2e-extmod-a-027c19-7974gqxtk\" (UID: \"a33ae0fa-7699-4c6c-8628-af11765d22df\") " pod="ai-tenant-e2e-extmod-a-027c19/payload-processing-external-model-e2e-extmod-a-027c19-7974gqxtk" Nov 03 01:21:23.924842 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:21:23.924830 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/a33ae0fa-7699-4c6c-8628-af11765d22df-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-027c19-7974gqxtk\" (UID: \"a33ae0fa-7699-4c6c-8628-af11765d22df\") " pod="ai-tenant-e2e-extmod-a-027c19/payload-processing-external-model-e2e-extmod-a-027c19-7974gqxtk" Nov 03 01:21:23.925095 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:21:23.925070 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/a33ae0fa-7699-4c6c-8628-af11765d22df-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-027c19-7974gqxtk\" (UID: \"a33ae0fa-7699-4c6c-8628-af11765d22df\") " pod="ai-tenant-e2e-extmod-a-027c19/payload-processing-external-model-e2e-extmod-a-027c19-7974gqxtk" Nov 03 01:21:24.027280 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:21:24.027230 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/a33ae0fa-7699-4c6c-8628-af11765d22df-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-027c19-7974gqxtk\" (UID: \"a33ae0fa-7699-4c6c-8628-af11765d22df\") " pod="ai-tenant-e2e-extmod-a-027c19/payload-processing-external-model-e2e-extmod-a-027c19-7974gqxtk" Nov 03 01:21:24.027527 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:21:24.027503 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a33ae0fa-7699-4c6c-8628-af11765d22df-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-027c19-7974gqxtk\" (UID: \"a33ae0fa-7699-4c6c-8628-af11765d22df\") " pod="ai-tenant-e2e-extmod-a-027c19/payload-processing-external-model-e2e-extmod-a-027c19-7974gqxtk" Nov 03 01:21:24.027674 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:21:24.027570 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5h6rz\" (UniqueName: \"kubernetes.io/projected/a33ae0fa-7699-4c6c-8628-af11765d22df-kube-api-access-5h6rz\") pod \"payload-processing-external-model-e2e-extmod-a-027c19-7974gqxtk\" (UID: \"a33ae0fa-7699-4c6c-8628-af11765d22df\") " pod="ai-tenant-e2e-extmod-a-027c19/payload-processing-external-model-e2e-extmod-a-027c19-7974gqxtk" Nov 03 01:21:24.027674 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:21:24.027636 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/a33ae0fa-7699-4c6c-8628-af11765d22df-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-027c19-7974gqxtk\" (UID: \"a33ae0fa-7699-4c6c-8628-af11765d22df\") " pod="ai-tenant-e2e-extmod-a-027c19/payload-processing-external-model-e2e-extmod-a-027c19-7974gqxtk" Nov 03 01:21:24.028275 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:21:24.028248 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/a33ae0fa-7699-4c6c-8628-af11765d22df-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-027c19-7974gqxtk\" (UID: \"a33ae0fa-7699-4c6c-8628-af11765d22df\") " pod="ai-tenant-e2e-extmod-a-027c19/payload-processing-external-model-e2e-extmod-a-027c19-7974gqxtk" Nov 03 01:21:24.028361 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:21:24.028335 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a33ae0fa-7699-4c6c-8628-af11765d22df-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-027c19-7974gqxtk\" (UID: \"a33ae0fa-7699-4c6c-8628-af11765d22df\") " pod="ai-tenant-e2e-extmod-a-027c19/payload-processing-external-model-e2e-extmod-a-027c19-7974gqxtk" Nov 03 01:21:24.030104 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:21:24.030081 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/a33ae0fa-7699-4c6c-8628-af11765d22df-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-027c19-7974gqxtk\" (UID: \"a33ae0fa-7699-4c6c-8628-af11765d22df\") " pod="ai-tenant-e2e-extmod-a-027c19/payload-processing-external-model-e2e-extmod-a-027c19-7974gqxtk" Nov 03 01:21:24.036062 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:21:24.036036 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5h6rz\" (UniqueName: \"kubernetes.io/projected/a33ae0fa-7699-4c6c-8628-af11765d22df-kube-api-access-5h6rz\") pod \"payload-processing-external-model-e2e-extmod-a-027c19-7974gqxtk\" (UID: \"a33ae0fa-7699-4c6c-8628-af11765d22df\") " pod="ai-tenant-e2e-extmod-a-027c19/payload-processing-external-model-e2e-extmod-a-027c19-7974gqxtk" Nov 03 01:21:24.169707 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:21:24.169676 4939 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-027c19/payload-processing-external-model-e2e-extmod-a-027c19-7974gqxtk" Nov 03 01:21:24.311719 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:21:24.311676 4939 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-027c19/payload-processing-external-model-e2e-extmod-a-027c19-55f89tmjf"] Nov 03 01:21:24.317793 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:21:24.317763 4939 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-027c19/payload-processing-external-model-e2e-extmod-a-027c19-7974gqxtk"] Nov 03 01:21:24.323353 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:21:24.323322 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:21:24.432421 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:21:24.432337 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:21:24.463097 ip-10-0-140-107 kubenswrapper[4939]: W1103 01:21:24.463067 4939 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda33ae0fa_7699_4c6c_8628_af11765d22df.slice/crio-004deaa2588ce378a100d5c8ad6f798204d1e9aeb8263563cb86be84fb3e0a84 WatchSource:0}: Error finding container 004deaa2588ce378a100d5c8ad6f798204d1e9aeb8263563cb86be84fb3e0a84: Status 404 returned error can't find the container with id 004deaa2588ce378a100d5c8ad6f798204d1e9aeb8263563cb86be84fb3e0a84 Nov 03 01:21:24.464222 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:21:24.464196 4939 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-027c19/payload-processing-external-model-e2e-extmod-a-027c19-7974gqxtk"] Nov 03 01:21:24.961302 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:21:24.961156 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:21:25.399903 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:21:25.399860 4939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-027c19/payload-processing-external-model-e2e-extmod-a-027c19-7974gqxtk" event={"ID":"a33ae0fa-7699-4c6c-8628-af11765d22df","Type":"ContainerStarted","Data":"af3b3b295f335187ed43c6016b8538388ddd5959b182210989ad04a362b7199c"} Nov 03 01:21:25.400128 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:21:25.399961 4939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-027c19/payload-processing-external-model-e2e-extmod-a-027c19-7974gqxtk" event={"ID":"a33ae0fa-7699-4c6c-8628-af11765d22df","Type":"ContainerStarted","Data":"004deaa2588ce378a100d5c8ad6f798204d1e9aeb8263563cb86be84fb3e0a84"} Nov 03 01:21:25.400128 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:21:25.399997 4939 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-a-027c19/payload-processing-external-model-e2e-extmod-a-027c19-7974gqxtk" Nov 03 01:21:25.422366 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:21:25.422312 4939 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-a-027c19/payload-processing-external-model-e2e-extmod-a-027c19-7974gqxtk" podStartSLOduration=2.4222967730000002 podStartE2EDuration="2.422296773s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 01:21:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 01:21:25.419549382 +0000 UTC m=+2433.683884328" watchObservedRunningTime="2026-11-03 01:21:25.422296773 +0000 UTC m=+2433.686631717" Nov 03 01:21:25.555928 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:21:25.547894 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:21:26.080034 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:21:26.080000 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:21:26.775280 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:21:26.775242 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:21:27.378187 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:21:27.378146 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:21:28.080845 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:21:28.080809 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:21:28.801390 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:21:28.801348 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:21:29.312575 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:21:29.312531 4939 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-a-027c19/payload-processing-external-model-e2e-extmod-a-027c19-55f89tmjf" podUID="8096d833-87a8-4153-b55b-1ce355bfd4f4" containerName="payload-processing" containerID="cri-o://92790343e5cffe5f3283ce8241f6db2bba12461ecfb3eb0a3d0534a9fe8b0ad8" gracePeriod=25 Nov 03 01:21:29.469269 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:21:29.469234 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:21:29.557590 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:21:29.557528 4939 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-4pw7v"] Nov 03 01:21:29.610020 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:21:29.609875 4939 generic.go:356] "Generic (PLEG): container finished" podID="8096d833-87a8-4153-b55b-1ce355bfd4f4" containerID="92790343e5cffe5f3283ce8241f6db2bba12461ecfb3eb0a3d0534a9fe8b0ad8" exitCode=0 Nov 03 01:21:29.610303 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:21:29.610266 4939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-027c19/payload-processing-external-model-e2e-extmod-a-027c19-55f89tmjf" event={"ID":"8096d833-87a8-4153-b55b-1ce355bfd4f4","Type":"ContainerDied","Data":"92790343e5cffe5f3283ce8241f6db2bba12461ecfb3eb0a3d0534a9fe8b0ad8"} Nov 03 01:21:29.713460 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:21:29.713426 4939 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-027c19/payload-processing-external-model-e2e-extmod-a-027c19-55f89tmjf" Nov 03 01:21:29.797604 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:21:29.797572 4939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8096d833-87a8-4153-b55b-1ce355bfd4f4-provider-credentials\" (UniqueName: \"kubernetes.io/projected/8096d833-87a8-4153-b55b-1ce355bfd4f4-provider-credentials\") pod \"8096d833-87a8-4153-b55b-1ce355bfd4f4\" (UID: \"8096d833-87a8-4153-b55b-1ce355bfd4f4\") " Nov 03 01:21:29.797822 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:21:29.797688 4939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8096d833-87a8-4153-b55b-1ce355bfd4f4-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/8096d833-87a8-4153-b55b-1ce355bfd4f4-routing-overlay-volume\") pod \"8096d833-87a8-4153-b55b-1ce355bfd4f4\" (UID: \"8096d833-87a8-4153-b55b-1ce355bfd4f4\") " Nov 03 01:21:29.797920 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:21:29.797887 4939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8096d833-87a8-4153-b55b-1ce355bfd4f4-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8096d833-87a8-4153-b55b-1ce355bfd4f4-plugins-config-volume\") pod \"8096d833-87a8-4153-b55b-1ce355bfd4f4\" (UID: \"8096d833-87a8-4153-b55b-1ce355bfd4f4\") " Nov 03 01:21:29.798052 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:21:29.797969 4939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8096d833-87a8-4153-b55b-1ce355bfd4f4-kube-api-access-k48gm\" (UniqueName: \"kubernetes.io/projected/8096d833-87a8-4153-b55b-1ce355bfd4f4-kube-api-access-k48gm\") pod \"8096d833-87a8-4153-b55b-1ce355bfd4f4\" (UID: \"8096d833-87a8-4153-b55b-1ce355bfd4f4\") " Nov 03 01:21:29.798168 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:21:29.798059 4939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8096d833-87a8-4153-b55b-1ce355bfd4f4-routing-overlay-volume" pod "8096d833-87a8-4153-b55b-1ce355bfd4f4" (UID: "8096d833-87a8-4153-b55b-1ce355bfd4f4"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 01:21:29.798319 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:21:29.798278 4939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8096d833-87a8-4153-b55b-1ce355bfd4f4-plugins-config-volume" pod "8096d833-87a8-4153-b55b-1ce355bfd4f4" (UID: "8096d833-87a8-4153-b55b-1ce355bfd4f4"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 01:21:29.798863 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:21:29.798841 4939 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/8096d833-87a8-4153-b55b-1ce355bfd4f4-routing-overlay-volume\") on node \"ip-10-0-140-107.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:21:29.798863 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:21:29.798863 4939 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8096d833-87a8-4153-b55b-1ce355bfd4f4-plugins-config-volume\") on node \"ip-10-0-140-107.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:21:29.800021 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:21:29.799995 4939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8096d833-87a8-4153-b55b-1ce355bfd4f4-provider-credentials" pod "8096d833-87a8-4153-b55b-1ce355bfd4f4" (UID: "8096d833-87a8-4153-b55b-1ce355bfd4f4"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 01:21:29.800304 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:21:29.800286 4939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8096d833-87a8-4153-b55b-1ce355bfd4f4-kube-api-access-k48gm" pod "8096d833-87a8-4153-b55b-1ce355bfd4f4" (UID: "8096d833-87a8-4153-b55b-1ce355bfd4f4"). InnerVolumeSpecName "kube-api-access-k48gm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 01:21:29.901076 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:21:29.901040 4939 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k48gm\" (UniqueName: \"kubernetes.io/projected/8096d833-87a8-4153-b55b-1ce355bfd4f4-kube-api-access-k48gm\") on node \"ip-10-0-140-107.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:21:29.901076 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:21:29.901077 4939 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/8096d833-87a8-4153-b55b-1ce355bfd4f4-provider-credentials\") on node \"ip-10-0-140-107.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:21:30.152240 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:21:30.152146 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:21:30.413988 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:21:30.410974 4939 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-a-027c19/payload-processing-external-model-e2e-extmod-a-027c19-7974gqxtk" podUID="a33ae0fa-7699-4c6c-8628-af11765d22df" containerName="payload-processing" containerID="cri-o://af3b3b295f335187ed43c6016b8538388ddd5959b182210989ad04a362b7199c" gracePeriod=25 Nov 03 01:21:30.745661 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:21:30.745152 4939 generic.go:356] "Generic (PLEG): container finished" podID="a33ae0fa-7699-4c6c-8628-af11765d22df" containerID="af3b3b295f335187ed43c6016b8538388ddd5959b182210989ad04a362b7199c" exitCode=0 Nov 03 01:21:30.745661 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:21:30.745476 4939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-027c19/payload-processing-external-model-e2e-extmod-a-027c19-7974gqxtk" event={"ID":"a33ae0fa-7699-4c6c-8628-af11765d22df","Type":"ContainerDied","Data":"af3b3b295f335187ed43c6016b8538388ddd5959b182210989ad04a362b7199c"} Nov 03 01:21:30.749567 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:21:30.747955 4939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-027c19/payload-processing-external-model-e2e-extmod-a-027c19-55f89tmjf" event={"ID":"8096d833-87a8-4153-b55b-1ce355bfd4f4","Type":"ContainerDied","Data":"83115c760888df70a72fa67845cf5481ce9bc8dd1dee29e1384b7d34b9d58d9d"} Nov 03 01:21:30.749567 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:21:30.748020 4939 scope.go:122] "RemoveContainer" containerID="92790343e5cffe5f3283ce8241f6db2bba12461ecfb3eb0a3d0534a9fe8b0ad8" Nov 03 01:21:30.749567 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:21:30.748219 4939 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-027c19/payload-processing-external-model-e2e-extmod-a-027c19-55f89tmjf" Nov 03 01:21:30.890348 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:21:30.890310 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:21:30.913927 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:21:30.897728 4939 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-027c19/payload-processing-external-model-e2e-extmod-a-027c19-55f89tmjf"] Nov 03 01:21:30.913927 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:21:30.897788 4939 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-a-027c19/payload-processing-external-model-e2e-extmod-a-027c19-55f89tmjf"] Nov 03 01:21:31.021070 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:21:31.020042 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:21:31.145557 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:21:31.145106 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:21:31.839335 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:21:31.809968 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:21:32.122170 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:21:32.117761 4939 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-027c19/payload-processing-external-model-e2e-extmod-a-027c19-7974gqxtk" Nov 03 01:21:32.130022 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:21:32.129984 4939 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8096d833-87a8-4153-b55b-1ce355bfd4f4" path="/var/lib/kubelet/pods/8096d833-87a8-4153-b55b-1ce355bfd4f4/volumes" Nov 03 01:21:32.211586 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:21:32.211545 4939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a33ae0fa-7699-4c6c-8628-af11765d22df-kube-api-access-5h6rz\" (UniqueName: \"kubernetes.io/projected/a33ae0fa-7699-4c6c-8628-af11765d22df-kube-api-access-5h6rz\") pod \"a33ae0fa-7699-4c6c-8628-af11765d22df\" (UID: \"a33ae0fa-7699-4c6c-8628-af11765d22df\") " Nov 03 01:21:32.211942 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:21:32.211893 4939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a33ae0fa-7699-4c6c-8628-af11765d22df-provider-credentials\" (UniqueName: \"kubernetes.io/projected/a33ae0fa-7699-4c6c-8628-af11765d22df-provider-credentials\") pod \"a33ae0fa-7699-4c6c-8628-af11765d22df\" (UID: \"a33ae0fa-7699-4c6c-8628-af11765d22df\") " Nov 03 01:21:32.212838 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:21:32.212797 4939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a33ae0fa-7699-4c6c-8628-af11765d22df-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a33ae0fa-7699-4c6c-8628-af11765d22df-plugins-config-volume\") pod \"a33ae0fa-7699-4c6c-8628-af11765d22df\" (UID: \"a33ae0fa-7699-4c6c-8628-af11765d22df\") " Nov 03 01:21:32.214660 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:21:32.214621 4939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a33ae0fa-7699-4c6c-8628-af11765d22df-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/a33ae0fa-7699-4c6c-8628-af11765d22df-routing-overlay-volume\") pod \"a33ae0fa-7699-4c6c-8628-af11765d22df\" (UID: \"a33ae0fa-7699-4c6c-8628-af11765d22df\") " Nov 03 01:21:32.216842 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:21:32.216819 4939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a33ae0fa-7699-4c6c-8628-af11765d22df-plugins-config-volume" pod "a33ae0fa-7699-4c6c-8628-af11765d22df" (UID: "a33ae0fa-7699-4c6c-8628-af11765d22df"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 01:21:32.217583 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:21:32.217561 4939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a33ae0fa-7699-4c6c-8628-af11765d22df-routing-overlay-volume" pod "a33ae0fa-7699-4c6c-8628-af11765d22df" (UID: "a33ae0fa-7699-4c6c-8628-af11765d22df"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 01:21:32.221181 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:21:32.221095 4939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a33ae0fa-7699-4c6c-8628-af11765d22df-kube-api-access-5h6rz" pod "a33ae0fa-7699-4c6c-8628-af11765d22df" (UID: "a33ae0fa-7699-4c6c-8628-af11765d22df"). InnerVolumeSpecName "kube-api-access-5h6rz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 01:21:32.224262 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:21:32.223679 4939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a33ae0fa-7699-4c6c-8628-af11765d22df-provider-credentials" pod "a33ae0fa-7699-4c6c-8628-af11765d22df" (UID: "a33ae0fa-7699-4c6c-8628-af11765d22df"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 01:21:32.318303 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:21:32.318252 4939 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5h6rz\" (UniqueName: \"kubernetes.io/projected/a33ae0fa-7699-4c6c-8628-af11765d22df-kube-api-access-5h6rz\") on node \"ip-10-0-140-107.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:21:32.318303 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:21:32.318299 4939 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/a33ae0fa-7699-4c6c-8628-af11765d22df-provider-credentials\") on node \"ip-10-0-140-107.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:21:32.318582 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:21:32.318318 4939 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a33ae0fa-7699-4c6c-8628-af11765d22df-plugins-config-volume\") on node \"ip-10-0-140-107.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:21:32.318582 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:21:32.318334 4939 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/a33ae0fa-7699-4c6c-8628-af11765d22df-routing-overlay-volume\") on node \"ip-10-0-140-107.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:21:32.797592 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:21:32.797556 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:21:33.143040 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:21:33.143004 4939 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-027c19/payload-processing-external-model-e2e-extmod-a-027c19-7974gqxtk" Nov 03 01:21:33.145146 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:21:33.143132 4939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-027c19/payload-processing-external-model-e2e-extmod-a-027c19-7974gqxtk" event={"ID":"a33ae0fa-7699-4c6c-8628-af11765d22df","Type":"ContainerDied","Data":"004deaa2588ce378a100d5c8ad6f798204d1e9aeb8263563cb86be84fb3e0a84"} Nov 03 01:21:33.145281 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:21:33.145190 4939 scope.go:122] "RemoveContainer" containerID="af3b3b295f335187ed43c6016b8538388ddd5959b182210989ad04a362b7199c" Nov 03 01:21:33.862932 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:21:33.862055 4939 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-027c19/payload-processing-external-model-e2e-extmod-a-027c19-7974gqxtk"] Nov 03 01:21:33.916932 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:21:33.916233 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:21:34.070930 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:21:34.070024 4939 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-a-027c19/payload-processing-external-model-e2e-extmod-a-027c19-7974gqxtk"] Nov 03 01:21:34.290992 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:21:34.290094 4939 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-2f9a3a/distinct-w0-2f9a3a-kserve-555565cfcf-jtcr9" podUID="ac0ece06-8718-463f-915f-5315988d9711" containerName="main" probeResult="failure" output="Get \"https://10.132.0.76:8000/health\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Nov 03 01:21:34.488931 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:21:34.487996 4939 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-88bbf7/distinct-2-w1-88bbf7-kserve-5457cdcdbd-l6tfd" podUID="a397931b-faa9-4c2e-8290-491350546663" containerName="main" probeResult="failure" output="Get \"https://10.132.0.85:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 01:21:34.498930 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:21:34.492970 4939 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-83755a/distinct-2-w3-83755a-kserve-6cf66b995d-2l47b" podUID="c3b05e88-5d5b-4d03-b28d-832f22e3c868" containerName="main" probeResult="failure" output="Get \"https://10.132.0.72:8000/ready\": context deadline exceeded" Nov 03 01:21:34.982934 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:21:34.854866 4939 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-2f9a3a/simulator-w0-2f9a3a-kserve-9596bbb8f-2trgz" podUID="207e2207-fb6c-4076-95df-72e95671c261" containerName="main" probeResult="failure" output="Get \"https://10.132.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 01:21:35.021932 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:21:35.017801 4939 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-2f9a3a/distinct-w0-2f9a3a-kserve-555565cfcf-jtcr9" podUID="ac0ece06-8718-463f-915f-5315988d9711" containerName="main" probeResult="failure" output="Get \"https://10.132.0.76:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Nov 03 01:21:35.106966 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:21:35.106852 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:21:35.872205 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:21:35.872168 4939 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a33ae0fa-7699-4c6c-8628-af11765d22df" path="/var/lib/kubelet/pods/a33ae0fa-7699-4c6c-8628-af11765d22df/volumes" Nov 03 01:21:35.874764 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:21:35.874740 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:21:36.661681 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:21:36.661643 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:21:37.286011 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:21:37.285975 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:21:37.775011 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:21:37.774975 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:21:37.900208 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:21:37.900161 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:21:38.026264 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:21:38.026174 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:21:38.165932 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:21:38.165338 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:21:38.282663 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:21:38.282580 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:21:38.488753 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:21:38.488718 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:21:38.811399 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:21:38.811363 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:21:39.061887 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:21:39.061782 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:21:39.312985 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:21:39.307883 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:21:40.427473 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:21:40.427419 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:21:40.515089 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:21:40.515056 4939 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-a-027c19/payload-processing-external-model-e2e-extmod-a-027c19-6f8dxgc6v"] Nov 03 01:21:40.516039 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:21:40.516019 4939 cpu_manager.go:414] "containerMap: removing container" podUID="a33ae0fa-7699-4c6c-8628-af11765d22df" containerName="payload-processing" Nov 03 01:21:40.516039 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:21:40.516036 4939 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a33ae0fa-7699-4c6c-8628-af11765d22df" containerName="payload-processing" Nov 03 01:21:40.516232 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:21:40.516047 4939 cpu_manager.go:414] "containerMap: removing container" podUID="8096d833-87a8-4153-b55b-1ce355bfd4f4" containerName="payload-processing" Nov 03 01:21:40.516232 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:21:40.516052 4939 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8096d833-87a8-4153-b55b-1ce355bfd4f4" containerName="payload-processing" Nov 03 01:21:40.516232 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:21:40.516202 4939 memory_manager.go:365] "RemoveStaleState removing state" podUID="a33ae0fa-7699-4c6c-8628-af11765d22df" containerName="payload-processing" Nov 03 01:21:40.516232 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:21:40.516213 4939 memory_manager.go:365] "RemoveStaleState removing state" podUID="8096d833-87a8-4153-b55b-1ce355bfd4f4" containerName="payload-processing" Nov 03 01:21:40.522392 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:21:40.522367 4939 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-027c19/payload-processing-external-model-e2e-extmod-a-027c19-6f8dxgc6v" Nov 03 01:21:40.527130 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:21:40.527102 4939 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-027c19/payload-processing-external-model-e2e-extmod-a-027c19-6f8dxgc6v"] Nov 03 01:21:40.528775 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:21:40.528751 4939 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-027c19\"/\"routing-overlay\"" Nov 03 01:21:40.529583 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:21:40.529559 4939 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-027c19\"/\"agc-cred-1eaa70db813c\"" Nov 03 01:21:40.529583 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:21:40.529572 4939 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-027c19\"/\"payload-processing-external-model-plugins-e2e-extmod-a-027c19\"" Nov 03 01:21:40.529750 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:21:40.529594 4939 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-027c19\"/\"kube-root-ca.crt\"" Nov 03 01:21:40.529750 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:21:40.529604 4939 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-027c19\"/\"payload-processing-external-model-e2e--4c667f43-dockercfg-6mcnf\"" Nov 03 01:21:40.529750 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:21:40.529638 4939 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-027c19\"/\"openshift-service-ca.crt\"" Nov 03 01:21:40.561145 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:21:40.561115 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:21:40.656940 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:21:40.654497 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/007181d2-dac6-436f-83b6-e5bbeaac0a30-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-027c19-6f8dxgc6v\" (UID: \"007181d2-dac6-436f-83b6-e5bbeaac0a30\") " pod="ai-tenant-e2e-extmod-a-027c19/payload-processing-external-model-e2e-extmod-a-027c19-6f8dxgc6v" Nov 03 01:21:40.700987 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:21:40.691532 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/007181d2-dac6-436f-83b6-e5bbeaac0a30-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-027c19-6f8dxgc6v\" (UID: \"007181d2-dac6-436f-83b6-e5bbeaac0a30\") " pod="ai-tenant-e2e-extmod-a-027c19/payload-processing-external-model-e2e-extmod-a-027c19-6f8dxgc6v" Nov 03 01:21:40.702784 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:21:40.702737 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:21:41.038529 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:21:41.035498 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gp96t\" (UniqueName: \"kubernetes.io/projected/007181d2-dac6-436f-83b6-e5bbeaac0a30-kube-api-access-gp96t\") pod \"payload-processing-external-model-e2e-extmod-a-027c19-6f8dxgc6v\" (UID: \"007181d2-dac6-436f-83b6-e5bbeaac0a30\") " pod="ai-tenant-e2e-extmod-a-027c19/payload-processing-external-model-e2e-extmod-a-027c19-6f8dxgc6v" Nov 03 01:21:41.038529 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:21:41.035768 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/007181d2-dac6-436f-83b6-e5bbeaac0a30-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-027c19-6f8dxgc6v\" (UID: \"007181d2-dac6-436f-83b6-e5bbeaac0a30\") " pod="ai-tenant-e2e-extmod-a-027c19/payload-processing-external-model-e2e-extmod-a-027c19-6f8dxgc6v" Nov 03 01:21:41.038529 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:21:41.035864 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/007181d2-dac6-436f-83b6-e5bbeaac0a30-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-027c19-6f8dxgc6v\" (UID: \"007181d2-dac6-436f-83b6-e5bbeaac0a30\") " pod="ai-tenant-e2e-extmod-a-027c19/payload-processing-external-model-e2e-extmod-a-027c19-6f8dxgc6v" Nov 03 01:21:41.038529 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:21:41.036566 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:21:41.123647 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:21:41.087326 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/007181d2-dac6-436f-83b6-e5bbeaac0a30-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-027c19-6f8dxgc6v\" (UID: \"007181d2-dac6-436f-83b6-e5bbeaac0a30\") " pod="ai-tenant-e2e-extmod-a-027c19/payload-processing-external-model-e2e-extmod-a-027c19-6f8dxgc6v" Nov 03 01:21:41.123647 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:21:41.088237 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/007181d2-dac6-436f-83b6-e5bbeaac0a30-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-027c19-6f8dxgc6v\" (UID: \"007181d2-dac6-436f-83b6-e5bbeaac0a30\") " pod="ai-tenant-e2e-extmod-a-027c19/payload-processing-external-model-e2e-extmod-a-027c19-6f8dxgc6v" Nov 03 01:21:41.135010 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:21:41.128253 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/007181d2-dac6-436f-83b6-e5bbeaac0a30-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-027c19-6f8dxgc6v\" (UID: \"007181d2-dac6-436f-83b6-e5bbeaac0a30\") " pod="ai-tenant-e2e-extmod-a-027c19/payload-processing-external-model-e2e-extmod-a-027c19-6f8dxgc6v" Nov 03 01:21:41.171932 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:21:41.157030 4939 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-b-3c5983/payload-processing-external-model-e2e-extmod-b-3c5983-5dc6r55b2"] Nov 03 01:21:41.244959 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:21:41.207849 4939 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-3c5983/payload-processing-external-model-e2e-extmod-b-3c5983-5dc6r55b2" Nov 03 01:21:41.244959 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:21:41.218724 4939 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-3c5983\"/\"agc-cred-daaef83b4fbf\"" Nov 03 01:21:41.244959 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:21:41.218838 4939 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-3c5983\"/\"routing-overlay\"" Nov 03 01:21:41.244959 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:21:41.220452 4939 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-3c5983\"/\"payload-processing-external-model-plugins-e2e-extmod-b-3c5983\"" Nov 03 01:21:41.244959 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:21:41.221735 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zp7bk\" (UniqueName: \"kubernetes.io/projected/dc81bbe2-2c01-4962-84a8-46b954bad81a-kube-api-access-zp7bk\") pod \"payload-processing-external-model-e2e-extmod-b-3c5983-5dc6r55b2\" (UID: \"dc81bbe2-2c01-4962-84a8-46b954bad81a\") " pod="ai-tenant-e2e-extmod-b-3c5983/payload-processing-external-model-e2e-extmod-b-3c5983-5dc6r55b2" Nov 03 01:21:41.244959 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:21:41.221883 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gp96t\" (UniqueName: \"kubernetes.io/projected/007181d2-dac6-436f-83b6-e5bbeaac0a30-kube-api-access-gp96t\") pod \"payload-processing-external-model-e2e-extmod-a-027c19-6f8dxgc6v\" (UID: \"007181d2-dac6-436f-83b6-e5bbeaac0a30\") " pod="ai-tenant-e2e-extmod-a-027c19/payload-processing-external-model-e2e-extmod-a-027c19-6f8dxgc6v" Nov 03 01:21:41.385870 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:21:41.385817 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dc81bbe2-2c01-4962-84a8-46b954bad81a-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-3c5983-5dc6r55b2\" (UID: \"dc81bbe2-2c01-4962-84a8-46b954bad81a\") " pod="ai-tenant-e2e-extmod-b-3c5983/payload-processing-external-model-e2e-extmod-b-3c5983-5dc6r55b2" Nov 03 01:21:41.387569 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:21:41.387525 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/dc81bbe2-2c01-4962-84a8-46b954bad81a-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-3c5983-5dc6r55b2\" (UID: \"dc81bbe2-2c01-4962-84a8-46b954bad81a\") " pod="ai-tenant-e2e-extmod-b-3c5983/payload-processing-external-model-e2e-extmod-b-3c5983-5dc6r55b2" Nov 03 01:21:41.387791 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:21:41.387762 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/dc81bbe2-2c01-4962-84a8-46b954bad81a-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-3c5983-5dc6r55b2\" (UID: \"dc81bbe2-2c01-4962-84a8-46b954bad81a\") " pod="ai-tenant-e2e-extmod-b-3c5983/payload-processing-external-model-e2e-extmod-b-3c5983-5dc6r55b2" Nov 03 01:21:41.502932 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:21:41.498827 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/007181d2-dac6-436f-83b6-e5bbeaac0a30-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-027c19-6f8dxgc6v\" (UID: \"007181d2-dac6-436f-83b6-e5bbeaac0a30\") " pod="ai-tenant-e2e-extmod-a-027c19/payload-processing-external-model-e2e-extmod-a-027c19-6f8dxgc6v" Nov 03 01:21:41.558967 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:21:41.556864 4939 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-3c5983\"/\"payload-processing-external-model-e2e--ba2bb6eb-dockercfg-r88kk\"" Nov 03 01:21:41.559502 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:21:41.559478 4939 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-3c5983\"/\"openshift-service-ca.crt\"" Nov 03 01:21:41.559875 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:21:41.559858 4939 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-3c5983\"/\"kube-root-ca.crt\"" Nov 03 01:21:41.607232 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:21:41.604611 4939 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-3c5983/payload-processing-external-model-e2e-extmod-b-3c5983-5dc6r55b2"] Nov 03 01:21:41.613187 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:21:41.613152 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/007181d2-dac6-436f-83b6-e5bbeaac0a30-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-027c19-6f8dxgc6v\" (UID: \"007181d2-dac6-436f-83b6-e5bbeaac0a30\") " pod="ai-tenant-e2e-extmod-a-027c19/payload-processing-external-model-e2e-extmod-a-027c19-6f8dxgc6v" Nov 03 01:21:41.614962 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:21:41.614827 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dc81bbe2-2c01-4962-84a8-46b954bad81a-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-3c5983-5dc6r55b2\" (UID: \"dc81bbe2-2c01-4962-84a8-46b954bad81a\") " pod="ai-tenant-e2e-extmod-b-3c5983/payload-processing-external-model-e2e-extmod-b-3c5983-5dc6r55b2" Nov 03 01:21:41.615122 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:21:41.614987 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/dc81bbe2-2c01-4962-84a8-46b954bad81a-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-3c5983-5dc6r55b2\" (UID: \"dc81bbe2-2c01-4962-84a8-46b954bad81a\") " pod="ai-tenant-e2e-extmod-b-3c5983/payload-processing-external-model-e2e-extmod-b-3c5983-5dc6r55b2" Nov 03 01:21:41.615122 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:21:41.615026 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/dc81bbe2-2c01-4962-84a8-46b954bad81a-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-3c5983-5dc6r55b2\" (UID: \"dc81bbe2-2c01-4962-84a8-46b954bad81a\") " pod="ai-tenant-e2e-extmod-b-3c5983/payload-processing-external-model-e2e-extmod-b-3c5983-5dc6r55b2" Nov 03 01:21:41.615326 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:21:41.615306 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zp7bk\" (UniqueName: \"kubernetes.io/projected/dc81bbe2-2c01-4962-84a8-46b954bad81a-kube-api-access-zp7bk\") pod \"payload-processing-external-model-e2e-extmod-b-3c5983-5dc6r55b2\" (UID: \"dc81bbe2-2c01-4962-84a8-46b954bad81a\") " pod="ai-tenant-e2e-extmod-b-3c5983/payload-processing-external-model-e2e-extmod-b-3c5983-5dc6r55b2" Nov 03 01:21:41.616129 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:21:41.616093 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dc81bbe2-2c01-4962-84a8-46b954bad81a-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-3c5983-5dc6r55b2\" (UID: \"dc81bbe2-2c01-4962-84a8-46b954bad81a\") " pod="ai-tenant-e2e-extmod-b-3c5983/payload-processing-external-model-e2e-extmod-b-3c5983-5dc6r55b2" Nov 03 01:21:41.616920 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:21:41.616873 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/dc81bbe2-2c01-4962-84a8-46b954bad81a-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-3c5983-5dc6r55b2\" (UID: \"dc81bbe2-2c01-4962-84a8-46b954bad81a\") " pod="ai-tenant-e2e-extmod-b-3c5983/payload-processing-external-model-e2e-extmod-b-3c5983-5dc6r55b2" Nov 03 01:21:41.617712 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:21:41.617685 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:21:41.618157 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:21:41.618132 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gp96t\" (UniqueName: \"kubernetes.io/projected/007181d2-dac6-436f-83b6-e5bbeaac0a30-kube-api-access-gp96t\") pod \"payload-processing-external-model-e2e-extmod-a-027c19-6f8dxgc6v\" (UID: \"007181d2-dac6-436f-83b6-e5bbeaac0a30\") " pod="ai-tenant-e2e-extmod-a-027c19/payload-processing-external-model-e2e-extmod-a-027c19-6f8dxgc6v" Nov 03 01:21:41.623161 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:21:41.623133 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/dc81bbe2-2c01-4962-84a8-46b954bad81a-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-3c5983-5dc6r55b2\" (UID: \"dc81bbe2-2c01-4962-84a8-46b954bad81a\") " pod="ai-tenant-e2e-extmod-b-3c5983/payload-processing-external-model-e2e-extmod-b-3c5983-5dc6r55b2" Nov 03 01:21:41.626443 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:21:41.626421 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zp7bk\" (UniqueName: \"kubernetes.io/projected/dc81bbe2-2c01-4962-84a8-46b954bad81a-kube-api-access-zp7bk\") pod \"payload-processing-external-model-e2e-extmod-b-3c5983-5dc6r55b2\" (UID: \"dc81bbe2-2c01-4962-84a8-46b954bad81a\") " pod="ai-tenant-e2e-extmod-b-3c5983/payload-processing-external-model-e2e-extmod-b-3c5983-5dc6r55b2" Nov 03 01:21:41.650398 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:21:41.650332 4939 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-3c5983/payload-processing-external-model-e2e-extmod-b-3c5983-5dc6r55b2" Nov 03 01:21:41.753261 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:21:41.753225 4939 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-027c19/payload-processing-external-model-e2e-extmod-a-027c19-6f8dxgc6v" Nov 03 01:21:41.874134 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:21:41.874098 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:21:42.024939 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:21:42.024836 4939 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-3c5983/payload-processing-external-model-e2e-extmod-b-3c5983-5dc6r55b2"] Nov 03 01:21:42.044654 ip-10-0-140-107 kubenswrapper[4939]: W1103 01:21:42.042374 4939 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddc81bbe2_2c01_4962_84a8_46b954bad81a.slice/crio-d7d3c9c5fdd71297ea7683caf147b2add4f613d6d8b95b2725e2c20a895045da WatchSource:0}: Error finding container d7d3c9c5fdd71297ea7683caf147b2add4f613d6d8b95b2725e2c20a895045da: Status 404 returned error can't find the container with id d7d3c9c5fdd71297ea7683caf147b2add4f613d6d8b95b2725e2c20a895045da Nov 03 01:21:42.061323 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:21:42.060980 4939 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-027c19/payload-processing-external-model-e2e-extmod-a-027c19-6f8dxgc6v"] Nov 03 01:21:42.071217 ip-10-0-140-107 kubenswrapper[4939]: W1103 01:21:42.071186 4939 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod007181d2_dac6_436f_83b6_e5bbeaac0a30.slice/crio-a8882a8f512df387e55c2711ecf0113ea20cffb2dbe65c6883fe498b9c10d263 WatchSource:0}: Error finding container a8882a8f512df387e55c2711ecf0113ea20cffb2dbe65c6883fe498b9c10d263: Status 404 returned error can't find the container with id a8882a8f512df387e55c2711ecf0113ea20cffb2dbe65c6883fe498b9c10d263 Nov 03 01:21:42.491185 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:21:42.491146 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:21:42.635278 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:21:42.635237 4939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-027c19/payload-processing-external-model-e2e-extmod-a-027c19-6f8dxgc6v" event={"ID":"007181d2-dac6-436f-83b6-e5bbeaac0a30","Type":"ContainerStarted","Data":"af567c408fbfdfa5ac82e0f78d7216ef6817c8dae9d67799bbd372094218a492"} Nov 03 01:21:42.635728 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:21:42.635314 4939 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-a-027c19/payload-processing-external-model-e2e-extmod-a-027c19-6f8dxgc6v" Nov 03 01:21:42.635728 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:21:42.635340 4939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-027c19/payload-processing-external-model-e2e-extmod-a-027c19-6f8dxgc6v" event={"ID":"007181d2-dac6-436f-83b6-e5bbeaac0a30","Type":"ContainerStarted","Data":"a8882a8f512df387e55c2711ecf0113ea20cffb2dbe65c6883fe498b9c10d263"} Nov 03 01:21:42.637051 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:21:42.637023 4939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-3c5983/payload-processing-external-model-e2e-extmod-b-3c5983-5dc6r55b2" event={"ID":"dc81bbe2-2c01-4962-84a8-46b954bad81a","Type":"ContainerStarted","Data":"454a982fde66e3348d0a9456530abf0b4face7082889677ce6096958558af7e7"} Nov 03 01:21:42.637191 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:21:42.637073 4939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-3c5983/payload-processing-external-model-e2e-extmod-b-3c5983-5dc6r55b2" event={"ID":"dc81bbe2-2c01-4962-84a8-46b954bad81a","Type":"ContainerStarted","Data":"d7d3c9c5fdd71297ea7683caf147b2add4f613d6d8b95b2725e2c20a895045da"} Nov 03 01:21:42.637191 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:21:42.637099 4939 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-b-3c5983/payload-processing-external-model-e2e-extmod-b-3c5983-5dc6r55b2" Nov 03 01:21:42.654936 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:21:42.654861 4939 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-a-027c19/payload-processing-external-model-e2e-extmod-a-027c19-6f8dxgc6v" podStartSLOduration=2.654847668 podStartE2EDuration="2.654847668s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 01:21:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 01:21:42.65320528 +0000 UTC m=+2450.917540229" watchObservedRunningTime="2026-11-03 01:21:42.654847668 +0000 UTC m=+2450.919182613" Nov 03 01:21:42.672319 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:21:42.672273 4939 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-b-3c5983/payload-processing-external-model-e2e-extmod-b-3c5983-5dc6r55b2" podStartSLOduration=2.672260655 podStartE2EDuration="2.672260655s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 01:21:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 01:21:42.668742089 +0000 UTC m=+2450.933077035" watchObservedRunningTime="2026-11-03 01:21:42.672260655 +0000 UTC m=+2450.936595600" Nov 03 01:21:43.175175 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:21:43.175140 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:21:43.808923 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:21:43.808887 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:21:44.462113 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:21:44.462082 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:21:44.576884 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:21:44.576346 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:21:45.070715 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:21:45.070628 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:21:45.708812 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:21:45.708777 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:21:46.339779 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:21:46.339747 4939 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-extmod-shr-user-08aa9e/e2e-usage-sim-92e6a0-5b4db7d754-db6qq"] Nov 03 01:21:46.346021 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:21:46.345990 4939 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-extmod-shr-user-08aa9e/e2e-usage-sim-92e6a0-5b4db7d754-db6qq" Nov 03 01:21:46.349970 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:21:46.349944 4939 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-extmod-shr-user-08aa9e/e2e-usage-sim-92e6a0-5b4db7d754-db6qq"] Nov 03 01:21:46.351810 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:21:46.351775 4939 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-extmod-shr-user-08aa9e\"/\"e2e-usage-sim-92e6a0-cert\"" Nov 03 01:21:46.351810 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:21:46.351790 4939 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-extmod-shr-user-08aa9e\"/\"kube-root-ca.crt\"" Nov 03 01:21:46.352043 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:21:46.351892 4939 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-extmod-shr-user-08aa9e\"/\"default-dockercfg-flsn2\"" Nov 03 01:21:46.352043 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:21:46.351976 4939 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-extmod-shr-user-08aa9e\"/\"openshift-service-ca.crt\"" Nov 03 01:21:46.380252 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:21:46.380221 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:21:46.450934 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:21:46.446837 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls\" (UniqueName: \"kubernetes.io/secret/000f2bd4-56ac-4df8-8379-f4947877a0af-tls\") pod \"e2e-usage-sim-92e6a0-5b4db7d754-db6qq\" (UID: \"000f2bd4-56ac-4df8-8379-f4947877a0af\") " pod="e2e-extmod-shr-user-08aa9e/e2e-usage-sim-92e6a0-5b4db7d754-db6qq" Nov 03 01:21:46.450934 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:21:46.447281 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pknng\" (UniqueName: \"kubernetes.io/projected/000f2bd4-56ac-4df8-8379-f4947877a0af-kube-api-access-pknng\") pod \"e2e-usage-sim-92e6a0-5b4db7d754-db6qq\" (UID: \"000f2bd4-56ac-4df8-8379-f4947877a0af\") " pod="e2e-extmod-shr-user-08aa9e/e2e-usage-sim-92e6a0-5b4db7d754-db6qq" Nov 03 01:21:46.548960 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:21:46.548902 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pknng\" (UniqueName: \"kubernetes.io/projected/000f2bd4-56ac-4df8-8379-f4947877a0af-kube-api-access-pknng\") pod \"e2e-usage-sim-92e6a0-5b4db7d754-db6qq\" (UID: \"000f2bd4-56ac-4df8-8379-f4947877a0af\") " pod="e2e-extmod-shr-user-08aa9e/e2e-usage-sim-92e6a0-5b4db7d754-db6qq" Nov 03 01:21:46.549153 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:21:46.549126 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls\" (UniqueName: \"kubernetes.io/secret/000f2bd4-56ac-4df8-8379-f4947877a0af-tls\") pod \"e2e-usage-sim-92e6a0-5b4db7d754-db6qq\" (UID: \"000f2bd4-56ac-4df8-8379-f4947877a0af\") " pod="e2e-extmod-shr-user-08aa9e/e2e-usage-sim-92e6a0-5b4db7d754-db6qq" Nov 03 01:21:46.551967 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:21:46.551942 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls\" (UniqueName: \"kubernetes.io/secret/000f2bd4-56ac-4df8-8379-f4947877a0af-tls\") pod \"e2e-usage-sim-92e6a0-5b4db7d754-db6qq\" (UID: \"000f2bd4-56ac-4df8-8379-f4947877a0af\") " pod="e2e-extmod-shr-user-08aa9e/e2e-usage-sim-92e6a0-5b4db7d754-db6qq" Nov 03 01:21:46.556748 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:21:46.556724 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pknng\" (UniqueName: \"kubernetes.io/projected/000f2bd4-56ac-4df8-8379-f4947877a0af-kube-api-access-pknng\") pod \"e2e-usage-sim-92e6a0-5b4db7d754-db6qq\" (UID: \"000f2bd4-56ac-4df8-8379-f4947877a0af\") " pod="e2e-extmod-shr-user-08aa9e/e2e-usage-sim-92e6a0-5b4db7d754-db6qq" Nov 03 01:21:46.663508 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:21:46.663426 4939 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-extmod-shr-user-08aa9e/e2e-usage-sim-92e6a0-5b4db7d754-db6qq" Nov 03 01:21:46.955862 ip-10-0-140-107 kubenswrapper[4939]: W1103 01:21:46.955827 4939 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod000f2bd4_56ac_4df8_8379_f4947877a0af.slice/crio-30e2d0595348f3f55b2ebb11a9597518200ee082c7cde69c9804bca0d5c19f43 WatchSource:0}: Error finding container 30e2d0595348f3f55b2ebb11a9597518200ee082c7cde69c9804bca0d5c19f43: Status 404 returned error can't find the container with id 30e2d0595348f3f55b2ebb11a9597518200ee082c7cde69c9804bca0d5c19f43 Nov 03 01:21:46.956282 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:21:46.956258 4939 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-extmod-shr-user-08aa9e/e2e-usage-sim-92e6a0-5b4db7d754-db6qq"] Nov 03 01:21:46.982441 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:21:46.982417 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:21:47.687452 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:21:47.687418 4939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-extmod-shr-user-08aa9e/e2e-usage-sim-92e6a0-5b4db7d754-db6qq" event={"ID":"000f2bd4-56ac-4df8-8379-f4947877a0af","Type":"ContainerStarted","Data":"1671913d48fdf59fbdf6e36613f1cfb94407716fe0e869d94a3559f9d0a04ecd"} Nov 03 01:21:47.687884 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:21:47.687467 4939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-extmod-shr-user-08aa9e/e2e-usage-sim-92e6a0-5b4db7d754-db6qq" event={"ID":"000f2bd4-56ac-4df8-8379-f4947877a0af","Type":"ContainerStarted","Data":"30e2d0595348f3f55b2ebb11a9597518200ee082c7cde69c9804bca0d5c19f43"} Nov 03 01:21:47.687884 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:21:47.687688 4939 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-extmod-shr-user-08aa9e/e2e-usage-sim-92e6a0-5b4db7d754-db6qq" Nov 03 01:21:47.692175 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:21:47.692149 4939 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-extmod-shr-user-08aa9e/e2e-usage-sim-92e6a0-5b4db7d754-db6qq" Nov 03 01:21:47.703969 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:21:47.703925 4939 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-extmod-shr-user-08aa9e/e2e-usage-sim-92e6a0-5b4db7d754-db6qq" podStartSLOduration=1.703898751 podStartE2EDuration="1.703898751s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 01:21:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 01:21:47.702052565 +0000 UTC m=+2455.966387514" watchObservedRunningTime="2026-11-03 01:21:47.703898751 +0000 UTC m=+2455.968233696" Nov 03 01:21:47.784867 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:21:47.784830 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:21:48.033464 ip-10-0-140-107 kubenswrapper[4939]: E1103 01:21:48.033371 4939 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-sthjh" containerName="extractor" Nov 03 01:21:48.463852 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:21:48.463798 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:21:49.057170 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:21:49.057136 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:21:49.699977 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:21:49.699941 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:21:50.589950 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:21:50.589433 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:21:51.179679 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:21:51.179644 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:21:51.784880 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:21:51.784850 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:21:52.239733 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:21:52.239698 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:21:52.354102 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:21:52.354068 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:21:53.512683 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:21:53.512645 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:21:53.615870 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:21:53.615827 4939 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-a-027c19/payload-processing-external-model-e2e-extmod-a-027c19-65568h8hw"] Nov 03 01:21:53.621713 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:21:53.621666 4939 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-027c19/payload-processing-external-model-e2e-extmod-a-027c19-65568h8hw"] Nov 03 01:21:53.621891 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:21:53.621774 4939 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-027c19/payload-processing-external-model-e2e-extmod-a-027c19-65568h8hw" Nov 03 01:21:53.627183 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:21:53.626473 4939 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-027c19\"/\"agc-cred-d91f96a4ea6f\"" Nov 03 01:21:53.648979 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:21:53.648928 4939 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-a-027c19/payload-processing-external-model-e2e-extmod-a-027c19-6f8dxgc6v" Nov 03 01:21:53.649335 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:21:53.649316 4939 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-b-3c5983/payload-processing-external-model-e2e-extmod-b-3c5983-5dc6r55b2" Nov 03 01:21:53.739994 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:21:53.739953 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wfqrs\" (UniqueName: \"kubernetes.io/projected/07a22731-ed88-4d35-be31-46351a8f5ce9-kube-api-access-wfqrs\") pod \"payload-processing-external-model-e2e-extmod-a-027c19-65568h8hw\" (UID: \"07a22731-ed88-4d35-be31-46351a8f5ce9\") " pod="ai-tenant-e2e-extmod-a-027c19/payload-processing-external-model-e2e-extmod-a-027c19-65568h8hw" Nov 03 01:21:53.740272 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:21:53.740073 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/07a22731-ed88-4d35-be31-46351a8f5ce9-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-027c19-65568h8hw\" (UID: \"07a22731-ed88-4d35-be31-46351a8f5ce9\") " pod="ai-tenant-e2e-extmod-a-027c19/payload-processing-external-model-e2e-extmod-a-027c19-65568h8hw" Nov 03 01:21:53.740272 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:21:53.740194 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/07a22731-ed88-4d35-be31-46351a8f5ce9-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-027c19-65568h8hw\" (UID: \"07a22731-ed88-4d35-be31-46351a8f5ce9\") " pod="ai-tenant-e2e-extmod-a-027c19/payload-processing-external-model-e2e-extmod-a-027c19-65568h8hw" Nov 03 01:21:53.740477 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:21:53.740425 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/07a22731-ed88-4d35-be31-46351a8f5ce9-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-027c19-65568h8hw\" (UID: \"07a22731-ed88-4d35-be31-46351a8f5ce9\") " pod="ai-tenant-e2e-extmod-a-027c19/payload-processing-external-model-e2e-extmod-a-027c19-65568h8hw" Nov 03 01:21:53.842028 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:21:53.841986 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wfqrs\" (UniqueName: \"kubernetes.io/projected/07a22731-ed88-4d35-be31-46351a8f5ce9-kube-api-access-wfqrs\") pod \"payload-processing-external-model-e2e-extmod-a-027c19-65568h8hw\" (UID: \"07a22731-ed88-4d35-be31-46351a8f5ce9\") " pod="ai-tenant-e2e-extmod-a-027c19/payload-processing-external-model-e2e-extmod-a-027c19-65568h8hw" Nov 03 01:21:53.842204 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:21:53.842067 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/07a22731-ed88-4d35-be31-46351a8f5ce9-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-027c19-65568h8hw\" (UID: \"07a22731-ed88-4d35-be31-46351a8f5ce9\") " pod="ai-tenant-e2e-extmod-a-027c19/payload-processing-external-model-e2e-extmod-a-027c19-65568h8hw" Nov 03 01:21:53.842204 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:21:53.842139 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/07a22731-ed88-4d35-be31-46351a8f5ce9-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-027c19-65568h8hw\" (UID: \"07a22731-ed88-4d35-be31-46351a8f5ce9\") " pod="ai-tenant-e2e-extmod-a-027c19/payload-processing-external-model-e2e-extmod-a-027c19-65568h8hw" Nov 03 01:21:53.842355 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:21:53.842263 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/07a22731-ed88-4d35-be31-46351a8f5ce9-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-027c19-65568h8hw\" (UID: \"07a22731-ed88-4d35-be31-46351a8f5ce9\") " pod="ai-tenant-e2e-extmod-a-027c19/payload-processing-external-model-e2e-extmod-a-027c19-65568h8hw" Nov 03 01:21:53.842886 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:21:53.842859 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/07a22731-ed88-4d35-be31-46351a8f5ce9-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-027c19-65568h8hw\" (UID: \"07a22731-ed88-4d35-be31-46351a8f5ce9\") " pod="ai-tenant-e2e-extmod-a-027c19/payload-processing-external-model-e2e-extmod-a-027c19-65568h8hw" Nov 03 01:21:53.843028 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:21:53.842956 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/07a22731-ed88-4d35-be31-46351a8f5ce9-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-027c19-65568h8hw\" (UID: \"07a22731-ed88-4d35-be31-46351a8f5ce9\") " pod="ai-tenant-e2e-extmod-a-027c19/payload-processing-external-model-e2e-extmod-a-027c19-65568h8hw" Nov 03 01:21:53.845056 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:21:53.845030 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/07a22731-ed88-4d35-be31-46351a8f5ce9-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-027c19-65568h8hw\" (UID: \"07a22731-ed88-4d35-be31-46351a8f5ce9\") " pod="ai-tenant-e2e-extmod-a-027c19/payload-processing-external-model-e2e-extmod-a-027c19-65568h8hw" Nov 03 01:21:53.850713 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:21:53.850692 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wfqrs\" (UniqueName: \"kubernetes.io/projected/07a22731-ed88-4d35-be31-46351a8f5ce9-kube-api-access-wfqrs\") pod \"payload-processing-external-model-e2e-extmod-a-027c19-65568h8hw\" (UID: \"07a22731-ed88-4d35-be31-46351a8f5ce9\") " pod="ai-tenant-e2e-extmod-a-027c19/payload-processing-external-model-e2e-extmod-a-027c19-65568h8hw" Nov 03 01:21:53.923496 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:21:53.923454 4939 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-b-3c5983/payload-processing-external-model-e2e-extmod-b-3c5983-9fbdwjsfd"] Nov 03 01:21:53.930195 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:21:53.930174 4939 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-3c5983/payload-processing-external-model-e2e-extmod-b-3c5983-9fbdwjsfd" Nov 03 01:21:53.938953 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:21:53.938884 4939 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-3c5983\"/\"agc-cred-8fae420a3d6c\"" Nov 03 01:21:53.939175 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:21:53.939005 4939 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-3c5983/payload-processing-external-model-e2e-extmod-b-3c5983-9fbdwjsfd"] Nov 03 01:21:53.951562 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:21:53.951537 4939 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-027c19/payload-processing-external-model-e2e-extmod-a-027c19-65568h8hw" Nov 03 01:21:54.047649 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:21:54.047594 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fb2md\" (UniqueName: \"kubernetes.io/projected/88ecb698-2cb1-4a6b-987c-3bf5c797f763-kube-api-access-fb2md\") pod \"payload-processing-external-model-e2e-extmod-b-3c5983-9fbdwjsfd\" (UID: \"88ecb698-2cb1-4a6b-987c-3bf5c797f763\") " pod="ai-tenant-e2e-extmod-b-3c5983/payload-processing-external-model-e2e-extmod-b-3c5983-9fbdwjsfd" Nov 03 01:21:54.047844 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:21:54.047660 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/88ecb698-2cb1-4a6b-987c-3bf5c797f763-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-3c5983-9fbdwjsfd\" (UID: \"88ecb698-2cb1-4a6b-987c-3bf5c797f763\") " pod="ai-tenant-e2e-extmod-b-3c5983/payload-processing-external-model-e2e-extmod-b-3c5983-9fbdwjsfd" Nov 03 01:21:54.048128 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:21:54.048092 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/88ecb698-2cb1-4a6b-987c-3bf5c797f763-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-3c5983-9fbdwjsfd\" (UID: \"88ecb698-2cb1-4a6b-987c-3bf5c797f763\") " pod="ai-tenant-e2e-extmod-b-3c5983/payload-processing-external-model-e2e-extmod-b-3c5983-9fbdwjsfd" Nov 03 01:21:54.048386 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:21:54.048358 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/88ecb698-2cb1-4a6b-987c-3bf5c797f763-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-3c5983-9fbdwjsfd\" (UID: \"88ecb698-2cb1-4a6b-987c-3bf5c797f763\") " pod="ai-tenant-e2e-extmod-b-3c5983/payload-processing-external-model-e2e-extmod-b-3c5983-9fbdwjsfd" Nov 03 01:21:54.150387 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:21:54.150217 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/88ecb698-2cb1-4a6b-987c-3bf5c797f763-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-3c5983-9fbdwjsfd\" (UID: \"88ecb698-2cb1-4a6b-987c-3bf5c797f763\") " pod="ai-tenant-e2e-extmod-b-3c5983/payload-processing-external-model-e2e-extmod-b-3c5983-9fbdwjsfd" Nov 03 01:21:54.150873 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:21:54.150728 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/88ecb698-2cb1-4a6b-987c-3bf5c797f763-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-3c5983-9fbdwjsfd\" (UID: \"88ecb698-2cb1-4a6b-987c-3bf5c797f763\") " pod="ai-tenant-e2e-extmod-b-3c5983/payload-processing-external-model-e2e-extmod-b-3c5983-9fbdwjsfd" Nov 03 01:21:54.150873 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:21:54.150842 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fb2md\" (UniqueName: \"kubernetes.io/projected/88ecb698-2cb1-4a6b-987c-3bf5c797f763-kube-api-access-fb2md\") pod \"payload-processing-external-model-e2e-extmod-b-3c5983-9fbdwjsfd\" (UID: \"88ecb698-2cb1-4a6b-987c-3bf5c797f763\") " pod="ai-tenant-e2e-extmod-b-3c5983/payload-processing-external-model-e2e-extmod-b-3c5983-9fbdwjsfd" Nov 03 01:21:54.151089 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:21:54.150873 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/88ecb698-2cb1-4a6b-987c-3bf5c797f763-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-3c5983-9fbdwjsfd\" (UID: \"88ecb698-2cb1-4a6b-987c-3bf5c797f763\") " pod="ai-tenant-e2e-extmod-b-3c5983/payload-processing-external-model-e2e-extmod-b-3c5983-9fbdwjsfd" Nov 03 01:21:54.151675 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:21:54.151645 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/88ecb698-2cb1-4a6b-987c-3bf5c797f763-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-3c5983-9fbdwjsfd\" (UID: \"88ecb698-2cb1-4a6b-987c-3bf5c797f763\") " pod="ai-tenant-e2e-extmod-b-3c5983/payload-processing-external-model-e2e-extmod-b-3c5983-9fbdwjsfd" Nov 03 01:21:54.152427 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:21:54.152400 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/88ecb698-2cb1-4a6b-987c-3bf5c797f763-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-3c5983-9fbdwjsfd\" (UID: \"88ecb698-2cb1-4a6b-987c-3bf5c797f763\") " pod="ai-tenant-e2e-extmod-b-3c5983/payload-processing-external-model-e2e-extmod-b-3c5983-9fbdwjsfd" Nov 03 01:21:54.153631 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:21:54.153600 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/88ecb698-2cb1-4a6b-987c-3bf5c797f763-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-3c5983-9fbdwjsfd\" (UID: \"88ecb698-2cb1-4a6b-987c-3bf5c797f763\") " pod="ai-tenant-e2e-extmod-b-3c5983/payload-processing-external-model-e2e-extmod-b-3c5983-9fbdwjsfd" Nov 03 01:21:54.159392 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:21:54.159367 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fb2md\" (UniqueName: \"kubernetes.io/projected/88ecb698-2cb1-4a6b-987c-3bf5c797f763-kube-api-access-fb2md\") pod \"payload-processing-external-model-e2e-extmod-b-3c5983-9fbdwjsfd\" (UID: \"88ecb698-2cb1-4a6b-987c-3bf5c797f763\") " pod="ai-tenant-e2e-extmod-b-3c5983/payload-processing-external-model-e2e-extmod-b-3c5983-9fbdwjsfd" Nov 03 01:21:54.246445 ip-10-0-140-107 kubenswrapper[4939]: W1103 01:21:54.246399 4939 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod07a22731_ed88_4d35_be31_46351a8f5ce9.slice/crio-eb16674552fbc05db6289197b19287c6cca3aa5fd4d15ba9ae087cee597606f4 WatchSource:0}: Error finding container eb16674552fbc05db6289197b19287c6cca3aa5fd4d15ba9ae087cee597606f4: Status 404 returned error can't find the container with id eb16674552fbc05db6289197b19287c6cca3aa5fd4d15ba9ae087cee597606f4 Nov 03 01:21:54.247345 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:21:54.247220 4939 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-027c19/payload-processing-external-model-e2e-extmod-a-027c19-65568h8hw"] Nov 03 01:21:54.251281 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:21:54.251257 4939 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-3c5983/payload-processing-external-model-e2e-extmod-b-3c5983-9fbdwjsfd" Nov 03 01:21:54.306503 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:21:54.305520 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:21:54.765981 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:21:54.765936 4939 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-3c5983/payload-processing-external-model-e2e-extmod-b-3c5983-9fbdwjsfd"] Nov 03 01:21:54.803219 ip-10-0-140-107 kubenswrapper[4939]: W1103 01:21:54.803179 4939 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod88ecb698_2cb1_4a6b_987c_3bf5c797f763.slice/crio-38926cbf984d3376b84566564df65859290eae3f0d1a4d981d0001183203bd16 WatchSource:0}: Error finding container 38926cbf984d3376b84566564df65859290eae3f0d1a4d981d0001183203bd16: Status 404 returned error can't find the container with id 38926cbf984d3376b84566564df65859290eae3f0d1a4d981d0001183203bd16 Nov 03 01:21:54.815856 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:21:54.815826 4939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-027c19/payload-processing-external-model-e2e-extmod-a-027c19-65568h8hw" event={"ID":"07a22731-ed88-4d35-be31-46351a8f5ce9","Type":"ContainerStarted","Data":"5106c2789578a525217990f7bffe265d8c0aaf6ece11c31890a51ddeace0e43f"} Nov 03 01:21:54.816007 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:21:54.815881 4939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-027c19/payload-processing-external-model-e2e-extmod-a-027c19-65568h8hw" event={"ID":"07a22731-ed88-4d35-be31-46351a8f5ce9","Type":"ContainerStarted","Data":"eb16674552fbc05db6289197b19287c6cca3aa5fd4d15ba9ae087cee597606f4"} Nov 03 01:21:54.816774 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:21:54.816757 4939 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-a-027c19/payload-processing-external-model-e2e-extmod-a-027c19-65568h8hw" Nov 03 01:21:54.847777 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:21:54.847730 4939 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-a-027c19/payload-processing-external-model-e2e-extmod-a-027c19-65568h8hw" podStartSLOduration=1.847716106 podStartE2EDuration="1.847716106s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 01:21:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 01:21:54.834165305 +0000 UTC m=+2463.098500250" watchObservedRunningTime="2026-11-03 01:21:54.847716106 +0000 UTC m=+2463.112051051" Nov 03 01:21:55.658931 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:21:55.636893 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:21:55.947125 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:21:55.942112 4939 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-b-3c5983/payload-processing-external-model-e2e-extmod-b-3c5983-9fbdwjsfd" podStartSLOduration=2.942094288 podStartE2EDuration="2.942094288s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 01:21:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 01:21:55.940708066 +0000 UTC m=+2464.205043014" watchObservedRunningTime="2026-11-03 01:21:55.942094288 +0000 UTC m=+2464.206429237" Nov 03 01:21:55.976613 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:21:55.976574 4939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-3c5983/payload-processing-external-model-e2e-extmod-b-3c5983-9fbdwjsfd" event={"ID":"88ecb698-2cb1-4a6b-987c-3bf5c797f763","Type":"ContainerStarted","Data":"ab2729093eecd07df2e4db4a2e3ee7b5c5ad02e6ba77c14d6fc3b745150703eb"} Nov 03 01:21:55.976888 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:21:55.976864 4939 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-b-3c5983/payload-processing-external-model-e2e-extmod-b-3c5983-9fbdwjsfd" Nov 03 01:21:55.977057 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:21:55.977040 4939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-3c5983/payload-processing-external-model-e2e-extmod-b-3c5983-9fbdwjsfd" event={"ID":"88ecb698-2cb1-4a6b-987c-3bf5c797f763","Type":"ContainerStarted","Data":"38926cbf984d3376b84566564df65859290eae3f0d1a4d981d0001183203bd16"} Nov 03 01:21:57.094566 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:21:57.094527 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:21:57.222964 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:21:57.222927 4939 scope.go:122] "RemoveContainer" containerID="25c87621c0bf3163a94ba3aa7a555843387cdc3a24dc93778bf38f2975019840" Nov 03 01:21:58.147932 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:21:58.146895 4939 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-88bbf7/simulator-w1-88bbf7-kserve-5fbb669f8f-42hgd" podUID="83b38883-2e7a-444d-aa68-52be4a25ae66" containerName="main" probeResult="failure" output="Get \"https://10.132.0.62:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 01:21:58.201072 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:21:58.201016 4939 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-88bbf7/simulator-w1-88bbf7-kserve-5fbb669f8f-42hgd" podUID="83b38883-2e7a-444d-aa68-52be4a25ae66" containerName="main" probeResult="failure" output="Get \"https://10.132.0.62:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 01:21:58.824424 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:21:58.824373 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:21:58.858927 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:21:58.849990 4939 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-2f9a3a/simulator-w0-2f9a3a-kserve-9596bbb8f-2trgz" podUID="207e2207-fb6c-4076-95df-72e95671c261" containerName="main" probeResult="failure" output="Get \"https://10.132.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 01:21:58.897943 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:21:58.878422 4939 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-f7680d/premium-w2-f7680d-kserve-76565b8cf9-hqbbt" podUID="4712e708-18a2-4a13-a12c-a5669397b2c3" containerName="main" probeResult="failure" output="Get \"https://10.132.0.63:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Nov 03 01:21:58.897943 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:21:58.878964 4939 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-2f9a3a/distinct-w0-2f9a3a-kserve-555565cfcf-jtcr9" podUID="ac0ece06-8718-463f-915f-5315988d9711" containerName="main" probeResult="failure" output="Get \"https://10.132.0.76:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 01:21:59.468294 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:21:59.468239 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:21:59.591935 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:21:59.590509 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:22:00.081125 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:22:00.081089 4939 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-b-af8244/payload-processing-external-model-e2e-extmod-b-af8244-6cbfbtf5b"] Nov 03 01:22:00.083309 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:22:00.083244 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:22:00.088063 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:22:00.088041 4939 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-af8244/payload-processing-external-model-e2e-extmod-b-af8244-6cbfbtf5b" Nov 03 01:22:00.092372 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:22:00.092346 4939 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-af8244\"/\"agc-cred-2327466017af\"" Nov 03 01:22:00.093713 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:22:00.093692 4939 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-af8244\"/\"openshift-service-ca.crt\"" Nov 03 01:22:00.093847 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:22:00.093757 4939 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-af8244\"/\"routing-overlay\"" Nov 03 01:22:00.093979 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:22:00.093797 4939 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-af8244\"/\"kube-root-ca.crt\"" Nov 03 01:22:00.093979 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:22:00.093901 4939 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-af8244\"/\"payload-processing-external-model-plugins-e2e-extmod-b-af8244\"" Nov 03 01:22:00.097960 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:22:00.097934 4939 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-af8244/payload-processing-external-model-e2e-extmod-b-af8244-6cbfbtf5b"] Nov 03 01:22:00.196489 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:22:00.196448 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/99099fe7-73ed-4732-9a33-2df9bed89665-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-af8244-6cbfbtf5b\" (UID: \"99099fe7-73ed-4732-9a33-2df9bed89665\") " pod="ai-tenant-e2e-extmod-b-af8244/payload-processing-external-model-e2e-extmod-b-af8244-6cbfbtf5b" Nov 03 01:22:00.196974 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:22:00.196584 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/99099fe7-73ed-4732-9a33-2df9bed89665-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-af8244-6cbfbtf5b\" (UID: \"99099fe7-73ed-4732-9a33-2df9bed89665\") " pod="ai-tenant-e2e-extmod-b-af8244/payload-processing-external-model-e2e-extmod-b-af8244-6cbfbtf5b" Nov 03 01:22:00.196974 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:22:00.196741 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/99099fe7-73ed-4732-9a33-2df9bed89665-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-af8244-6cbfbtf5b\" (UID: \"99099fe7-73ed-4732-9a33-2df9bed89665\") " pod="ai-tenant-e2e-extmod-b-af8244/payload-processing-external-model-e2e-extmod-b-af8244-6cbfbtf5b" Nov 03 01:22:00.196974 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:22:00.196897 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8jxcc\" (UniqueName: \"kubernetes.io/projected/99099fe7-73ed-4732-9a33-2df9bed89665-kube-api-access-8jxcc\") pod \"payload-processing-external-model-e2e-extmod-b-af8244-6cbfbtf5b\" (UID: \"99099fe7-73ed-4732-9a33-2df9bed89665\") " pod="ai-tenant-e2e-extmod-b-af8244/payload-processing-external-model-e2e-extmod-b-af8244-6cbfbtf5b" Nov 03 01:22:00.352990 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:22:00.342725 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/99099fe7-73ed-4732-9a33-2df9bed89665-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-af8244-6cbfbtf5b\" (UID: \"99099fe7-73ed-4732-9a33-2df9bed89665\") " pod="ai-tenant-e2e-extmod-b-af8244/payload-processing-external-model-e2e-extmod-b-af8244-6cbfbtf5b" Nov 03 01:22:00.352990 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:22:00.342843 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/99099fe7-73ed-4732-9a33-2df9bed89665-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-af8244-6cbfbtf5b\" (UID: \"99099fe7-73ed-4732-9a33-2df9bed89665\") " pod="ai-tenant-e2e-extmod-b-af8244/payload-processing-external-model-e2e-extmod-b-af8244-6cbfbtf5b" Nov 03 01:22:00.352990 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:22:00.343037 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/99099fe7-73ed-4732-9a33-2df9bed89665-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-af8244-6cbfbtf5b\" (UID: \"99099fe7-73ed-4732-9a33-2df9bed89665\") " pod="ai-tenant-e2e-extmod-b-af8244/payload-processing-external-model-e2e-extmod-b-af8244-6cbfbtf5b" Nov 03 01:22:00.352990 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:22:00.343298 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8jxcc\" (UniqueName: \"kubernetes.io/projected/99099fe7-73ed-4732-9a33-2df9bed89665-kube-api-access-8jxcc\") pod \"payload-processing-external-model-e2e-extmod-b-af8244-6cbfbtf5b\" (UID: \"99099fe7-73ed-4732-9a33-2df9bed89665\") " pod="ai-tenant-e2e-extmod-b-af8244/payload-processing-external-model-e2e-extmod-b-af8244-6cbfbtf5b" Nov 03 01:22:00.352990 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:22:00.347958 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/99099fe7-73ed-4732-9a33-2df9bed89665-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-af8244-6cbfbtf5b\" (UID: \"99099fe7-73ed-4732-9a33-2df9bed89665\") " pod="ai-tenant-e2e-extmod-b-af8244/payload-processing-external-model-e2e-extmod-b-af8244-6cbfbtf5b" Nov 03 01:22:00.352990 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:22:00.348699 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/99099fe7-73ed-4732-9a33-2df9bed89665-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-af8244-6cbfbtf5b\" (UID: \"99099fe7-73ed-4732-9a33-2df9bed89665\") " pod="ai-tenant-e2e-extmod-b-af8244/payload-processing-external-model-e2e-extmod-b-af8244-6cbfbtf5b" Nov 03 01:22:00.395951 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:22:00.395858 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/99099fe7-73ed-4732-9a33-2df9bed89665-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-af8244-6cbfbtf5b\" (UID: \"99099fe7-73ed-4732-9a33-2df9bed89665\") " pod="ai-tenant-e2e-extmod-b-af8244/payload-processing-external-model-e2e-extmod-b-af8244-6cbfbtf5b" Nov 03 01:22:00.456583 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:22:00.456537 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:22:00.473439 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:22:00.473399 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8jxcc\" (UniqueName: \"kubernetes.io/projected/99099fe7-73ed-4732-9a33-2df9bed89665-kube-api-access-8jxcc\") pod \"payload-processing-external-model-e2e-extmod-b-af8244-6cbfbtf5b\" (UID: \"99099fe7-73ed-4732-9a33-2df9bed89665\") " pod="ai-tenant-e2e-extmod-b-af8244/payload-processing-external-model-e2e-extmod-b-af8244-6cbfbtf5b" Nov 03 01:22:00.706823 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:22:00.706728 4939 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-af8244/payload-processing-external-model-e2e-extmod-b-af8244-6cbfbtf5b" Nov 03 01:22:01.017576 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:22:01.017546 4939 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-af8244/payload-processing-external-model-e2e-extmod-b-af8244-6cbfbtf5b"] Nov 03 01:22:01.017920 ip-10-0-140-107 kubenswrapper[4939]: W1103 01:22:01.017879 4939 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod99099fe7_73ed_4732_9a33_2df9bed89665.slice/crio-e114aaf2fedefbfefe5c1c99dfe6b6e7965edfdbcefc2ac23dced39afaaaac11 WatchSource:0}: Error finding container e114aaf2fedefbfefe5c1c99dfe6b6e7965edfdbcefc2ac23dced39afaaaac11: Status 404 returned error can't find the container with id e114aaf2fedefbfefe5c1c99dfe6b6e7965edfdbcefc2ac23dced39afaaaac11 Nov 03 01:22:01.160674 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:22:01.160636 4939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-af8244/payload-processing-external-model-e2e-extmod-b-af8244-6cbfbtf5b" event={"ID":"99099fe7-73ed-4732-9a33-2df9bed89665","Type":"ContainerStarted","Data":"6da8e13a4665c438db1ed016e08ce569161e7e6a734916ac931e20cf895b3ba1"} Nov 03 01:22:01.160874 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:22:01.160702 4939 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-b-af8244/payload-processing-external-model-e2e-extmod-b-af8244-6cbfbtf5b" Nov 03 01:22:01.160874 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:22:01.160729 4939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-af8244/payload-processing-external-model-e2e-extmod-b-af8244-6cbfbtf5b" event={"ID":"99099fe7-73ed-4732-9a33-2df9bed89665","Type":"ContainerStarted","Data":"e114aaf2fedefbfefe5c1c99dfe6b6e7965edfdbcefc2ac23dced39afaaaac11"} Nov 03 01:22:01.182666 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:22:01.182601 4939 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-b-af8244/payload-processing-external-model-e2e-extmod-b-af8244-6cbfbtf5b" podStartSLOduration=1.182582816 podStartE2EDuration="1.182582816s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 01:22:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 01:22:01.18082846 +0000 UTC m=+2469.445163408" watchObservedRunningTime="2026-11-03 01:22:01.182582816 +0000 UTC m=+2469.446917761" Nov 03 01:22:01.454501 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:22:01.454456 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:22:02.093817 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:22:02.093773 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:22:02.820283 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:22:02.820243 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:22:03.409188 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:22:03.409157 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:22:04.072166 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:22:04.072125 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:22:04.796333 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:22:04.796297 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:22:05.498495 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:22:05.498454 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:22:06.105265 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:22:06.105202 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:22:06.789756 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:22:06.789711 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:22:06.978487 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:22:06.978453 4939 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-b-3c5983/payload-processing-external-model-e2e-extmod-b-3c5983-9fbdwjsfd" Nov 03 01:22:06.978945 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:22:06.978921 4939 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-a-027c19/payload-processing-external-model-e2e-extmod-a-027c19-65568h8hw" Nov 03 01:22:07.033178 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:22:07.033144 4939 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-3c5983/payload-processing-external-model-e2e-extmod-b-3c5983-5dc6r55b2"] Nov 03 01:22:07.050443 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:22:07.050314 4939 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-027c19/payload-processing-external-model-e2e-extmod-a-027c19-6f8dxgc6v"] Nov 03 01:22:07.555793 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:22:07.555754 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:22:08.157653 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:22:08.157618 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:22:08.774392 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:22:08.774353 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:22:09.518604 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:22:09.518564 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:22:10.099086 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:22:10.099052 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:22:10.797348 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:22:10.797318 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:22:11.507036 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:22:11.507007 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:22:12.034274 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:22:12.034204 4939 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-b-3c5983/payload-processing-external-model-e2e-extmod-b-3c5983-5dc6r55b2" podUID="dc81bbe2-2c01-4962-84a8-46b954bad81a" containerName="payload-processing" containerID="cri-o://454a982fde66e3348d0a9456530abf0b4face7082889677ce6096958558af7e7" gracePeriod=25 Nov 03 01:22:12.051070 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:22:12.051008 4939 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-a-027c19/payload-processing-external-model-e2e-extmod-a-027c19-6f8dxgc6v" podUID="007181d2-dac6-436f-83b6-e5bbeaac0a30" containerName="payload-processing" containerID="cri-o://af567c408fbfdfa5ac82e0f78d7216ef6817c8dae9d67799bbd372094218a492" gracePeriod=25 Nov 03 01:22:12.175811 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:22:12.175750 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:22:12.246645 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:22:12.245369 4939 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-b-af8244/payload-processing-external-model-e2e-extmod-b-af8244-6cbfbtf5b" Nov 03 01:22:12.329175 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:22:12.329060 4939 generic.go:356] "Generic (PLEG): container finished" podID="007181d2-dac6-436f-83b6-e5bbeaac0a30" containerID="af567c408fbfdfa5ac82e0f78d7216ef6817c8dae9d67799bbd372094218a492" exitCode=0 Nov 03 01:22:12.329349 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:22:12.329216 4939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-027c19/payload-processing-external-model-e2e-extmod-a-027c19-6f8dxgc6v" event={"ID":"007181d2-dac6-436f-83b6-e5bbeaac0a30","Type":"ContainerDied","Data":"af567c408fbfdfa5ac82e0f78d7216ef6817c8dae9d67799bbd372094218a492"} Nov 03 01:22:12.333501 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:22:12.333470 4939 generic.go:356] "Generic (PLEG): container finished" podID="dc81bbe2-2c01-4962-84a8-46b954bad81a" containerID="454a982fde66e3348d0a9456530abf0b4face7082889677ce6096958558af7e7" exitCode=0 Nov 03 01:22:12.333670 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:22:12.333648 4939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-3c5983/payload-processing-external-model-e2e-extmod-b-3c5983-5dc6r55b2" event={"ID":"dc81bbe2-2c01-4962-84a8-46b954bad81a","Type":"ContainerDied","Data":"454a982fde66e3348d0a9456530abf0b4face7082889677ce6096958558af7e7"} Nov 03 01:22:12.551648 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:22:12.551614 4939 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-3c5983/payload-processing-external-model-e2e-extmod-b-3c5983-5dc6r55b2" Nov 03 01:22:12.556415 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:22:12.556383 4939 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-027c19/payload-processing-external-model-e2e-extmod-a-027c19-6f8dxgc6v" Nov 03 01:22:12.665729 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:22:12.665687 4939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/007181d2-dac6-436f-83b6-e5bbeaac0a30-provider-credentials\" (UniqueName: \"kubernetes.io/projected/007181d2-dac6-436f-83b6-e5bbeaac0a30-provider-credentials\") pod \"007181d2-dac6-436f-83b6-e5bbeaac0a30\" (UID: \"007181d2-dac6-436f-83b6-e5bbeaac0a30\") " Nov 03 01:22:12.665988 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:22:12.665789 4939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/007181d2-dac6-436f-83b6-e5bbeaac0a30-kube-api-access-gp96t\" (UniqueName: \"kubernetes.io/projected/007181d2-dac6-436f-83b6-e5bbeaac0a30-kube-api-access-gp96t\") pod \"007181d2-dac6-436f-83b6-e5bbeaac0a30\" (UID: \"007181d2-dac6-436f-83b6-e5bbeaac0a30\") " Nov 03 01:22:12.665988 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:22:12.665948 4939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dc81bbe2-2c01-4962-84a8-46b954bad81a-kube-api-access-zp7bk\" (UniqueName: \"kubernetes.io/projected/dc81bbe2-2c01-4962-84a8-46b954bad81a-kube-api-access-zp7bk\") pod \"dc81bbe2-2c01-4962-84a8-46b954bad81a\" (UID: \"dc81bbe2-2c01-4962-84a8-46b954bad81a\") " Nov 03 01:22:12.666144 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:22:12.666130 4939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/dc81bbe2-2c01-4962-84a8-46b954bad81a-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/dc81bbe2-2c01-4962-84a8-46b954bad81a-routing-overlay-volume\") pod \"dc81bbe2-2c01-4962-84a8-46b954bad81a\" (UID: \"dc81bbe2-2c01-4962-84a8-46b954bad81a\") " Nov 03 01:22:12.666299 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:22:12.666283 4939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/dc81bbe2-2c01-4962-84a8-46b954bad81a-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dc81bbe2-2c01-4962-84a8-46b954bad81a-plugins-config-volume\") pod \"dc81bbe2-2c01-4962-84a8-46b954bad81a\" (UID: \"dc81bbe2-2c01-4962-84a8-46b954bad81a\") " Nov 03 01:22:12.666394 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:22:12.666376 4939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/007181d2-dac6-436f-83b6-e5bbeaac0a30-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/007181d2-dac6-436f-83b6-e5bbeaac0a30-plugins-config-volume\") pod \"007181d2-dac6-436f-83b6-e5bbeaac0a30\" (UID: \"007181d2-dac6-436f-83b6-e5bbeaac0a30\") " Nov 03 01:22:12.666461 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:22:12.666451 4939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/007181d2-dac6-436f-83b6-e5bbeaac0a30-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/007181d2-dac6-436f-83b6-e5bbeaac0a30-routing-overlay-volume\") pod \"007181d2-dac6-436f-83b6-e5bbeaac0a30\" (UID: \"007181d2-dac6-436f-83b6-e5bbeaac0a30\") " Nov 03 01:22:12.666524 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:22:12.666511 4939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dc81bbe2-2c01-4962-84a8-46b954bad81a-provider-credentials\" (UniqueName: \"kubernetes.io/projected/dc81bbe2-2c01-4962-84a8-46b954bad81a-provider-credentials\") pod \"dc81bbe2-2c01-4962-84a8-46b954bad81a\" (UID: \"dc81bbe2-2c01-4962-84a8-46b954bad81a\") " Nov 03 01:22:12.666646 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:22:12.666599 4939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dc81bbe2-2c01-4962-84a8-46b954bad81a-routing-overlay-volume" pod "dc81bbe2-2c01-4962-84a8-46b954bad81a" (UID: "dc81bbe2-2c01-4962-84a8-46b954bad81a"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 01:22:12.667208 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:22:12.667040 4939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/007181d2-dac6-436f-83b6-e5bbeaac0a30-routing-overlay-volume" pod "007181d2-dac6-436f-83b6-e5bbeaac0a30" (UID: "007181d2-dac6-436f-83b6-e5bbeaac0a30"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 01:22:12.667317 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:22:12.667207 4939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dc81bbe2-2c01-4962-84a8-46b954bad81a-plugins-config-volume" pod "dc81bbe2-2c01-4962-84a8-46b954bad81a" (UID: "dc81bbe2-2c01-4962-84a8-46b954bad81a"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 01:22:12.667827 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:22:12.667802 4939 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/dc81bbe2-2c01-4962-84a8-46b954bad81a-routing-overlay-volume\") on node \"ip-10-0-140-107.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:22:12.667945 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:22:12.667832 4939 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dc81bbe2-2c01-4962-84a8-46b954bad81a-plugins-config-volume\") on node \"ip-10-0-140-107.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:22:12.667945 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:22:12.667850 4939 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/007181d2-dac6-436f-83b6-e5bbeaac0a30-routing-overlay-volume\") on node \"ip-10-0-140-107.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:22:12.668218 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:22:12.668194 4939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/007181d2-dac6-436f-83b6-e5bbeaac0a30-plugins-config-volume" pod "007181d2-dac6-436f-83b6-e5bbeaac0a30" (UID: "007181d2-dac6-436f-83b6-e5bbeaac0a30"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 01:22:12.668399 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:22:12.668373 4939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/007181d2-dac6-436f-83b6-e5bbeaac0a30-provider-credentials" pod "007181d2-dac6-436f-83b6-e5bbeaac0a30" (UID: "007181d2-dac6-436f-83b6-e5bbeaac0a30"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 01:22:12.668493 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:22:12.668438 4939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/007181d2-dac6-436f-83b6-e5bbeaac0a30-kube-api-access-gp96t" pod "007181d2-dac6-436f-83b6-e5bbeaac0a30" (UID: "007181d2-dac6-436f-83b6-e5bbeaac0a30"). InnerVolumeSpecName "kube-api-access-gp96t". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 01:22:12.668591 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:22:12.668570 4939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dc81bbe2-2c01-4962-84a8-46b954bad81a-kube-api-access-zp7bk" pod "dc81bbe2-2c01-4962-84a8-46b954bad81a" (UID: "dc81bbe2-2c01-4962-84a8-46b954bad81a"). InnerVolumeSpecName "kube-api-access-zp7bk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 01:22:12.669131 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:22:12.669107 4939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dc81bbe2-2c01-4962-84a8-46b954bad81a-provider-credentials" pod "dc81bbe2-2c01-4962-84a8-46b954bad81a" (UID: "dc81bbe2-2c01-4962-84a8-46b954bad81a"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 01:22:12.769799 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:22:12.769760 4939 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/007181d2-dac6-436f-83b6-e5bbeaac0a30-plugins-config-volume\") on node \"ip-10-0-140-107.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:22:12.769799 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:22:12.769793 4939 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/dc81bbe2-2c01-4962-84a8-46b954bad81a-provider-credentials\") on node \"ip-10-0-140-107.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:22:12.769799 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:22:12.769803 4939 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/007181d2-dac6-436f-83b6-e5bbeaac0a30-provider-credentials\") on node \"ip-10-0-140-107.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:22:12.770159 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:22:12.769812 4939 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gp96t\" (UniqueName: \"kubernetes.io/projected/007181d2-dac6-436f-83b6-e5bbeaac0a30-kube-api-access-gp96t\") on node \"ip-10-0-140-107.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:22:12.770159 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:22:12.769821 4939 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zp7bk\" (UniqueName: \"kubernetes.io/projected/dc81bbe2-2c01-4962-84a8-46b954bad81a-kube-api-access-zp7bk\") on node \"ip-10-0-140-107.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:22:12.804443 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:22:12.804406 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:22:13.345620 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:22:13.345585 4939 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-027c19/payload-processing-external-model-e2e-extmod-a-027c19-6f8dxgc6v" Nov 03 01:22:13.345827 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:22:13.345594 4939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-027c19/payload-processing-external-model-e2e-extmod-a-027c19-6f8dxgc6v" event={"ID":"007181d2-dac6-436f-83b6-e5bbeaac0a30","Type":"ContainerDied","Data":"a8882a8f512df387e55c2711ecf0113ea20cffb2dbe65c6883fe498b9c10d263"} Nov 03 01:22:13.345827 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:22:13.345736 4939 scope.go:122] "RemoveContainer" containerID="af567c408fbfdfa5ac82e0f78d7216ef6817c8dae9d67799bbd372094218a492" Nov 03 01:22:13.352958 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:22:13.348062 4939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-3c5983/payload-processing-external-model-e2e-extmod-b-3c5983-5dc6r55b2" event={"ID":"dc81bbe2-2c01-4962-84a8-46b954bad81a","Type":"ContainerDied","Data":"d7d3c9c5fdd71297ea7683caf147b2add4f613d6d8b95b2725e2c20a895045da"} Nov 03 01:22:13.352958 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:22:13.348130 4939 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-3c5983/payload-processing-external-model-e2e-extmod-b-3c5983-5dc6r55b2" Nov 03 01:22:13.361295 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:22:13.361273 4939 scope.go:122] "RemoveContainer" containerID="454a982fde66e3348d0a9456530abf0b4face7082889677ce6096958558af7e7" Nov 03 01:22:13.389583 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:22:13.389542 4939 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-3c5983/payload-processing-external-model-e2e-extmod-b-3c5983-5dc6r55b2"] Nov 03 01:22:13.397352 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:22:13.397319 4939 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-b-3c5983/payload-processing-external-model-e2e-extmod-b-3c5983-5dc6r55b2"] Nov 03 01:22:13.402306 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:22:13.402278 4939 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-027c19/payload-processing-external-model-e2e-extmod-a-027c19-6f8dxgc6v"] Nov 03 01:22:13.406764 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:22:13.406741 4939 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-a-027c19/payload-processing-external-model-e2e-extmod-a-027c19-6f8dxgc6v"] Nov 03 01:22:13.588546 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:22:13.588480 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:22:13.874153 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:22:13.874116 4939 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="007181d2-dac6-436f-83b6-e5bbeaac0a30" path="/var/lib/kubelet/pods/007181d2-dac6-436f-83b6-e5bbeaac0a30/volumes" Nov 03 01:22:13.874718 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:22:13.874701 4939 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="dc81bbe2-2c01-4962-84a8-46b954bad81a" path="/var/lib/kubelet/pods/dc81bbe2-2c01-4962-84a8-46b954bad81a/volumes" Nov 03 01:22:14.190003 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:22:14.189857 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:22:14.797989 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:22:14.797947 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:22:15.444398 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:22:15.444368 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:22:16.087345 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:22:16.087311 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:22:16.770867 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:22:16.770835 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:22:17.536692 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:22:17.536652 4939 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["models-as-a-service/payload-processing-external-model-76c77f5454-842p6"] Nov 03 01:22:17.579993 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:22:17.579856 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:22:17.699054 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:22:17.699019 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:22:18.247198 ip-10-0-140-107 kubenswrapper[4939]: E1103 01:22:18.247162 4939 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-sthjh" containerName="extractor" Nov 03 01:22:18.969305 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:22:18.969266 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:22:19.622534 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:22:19.622500 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:22:19.730385 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:22:19.730353 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:22:19.980040 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:22:19.977888 4939 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-a-a88862-openshift-default-7b98d598f-jbgzc"] Nov 03 01:22:19.980040 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:22:19.978841 4939 cpu_manager.go:414] "containerMap: removing container" podUID="dc81bbe2-2c01-4962-84a8-46b954bad81a" containerName="payload-processing" Nov 03 01:22:19.980040 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:22:19.978858 4939 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dc81bbe2-2c01-4962-84a8-46b954bad81a" containerName="payload-processing" Nov 03 01:22:19.980040 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:22:19.978955 4939 cpu_manager.go:414] "containerMap: removing container" podUID="007181d2-dac6-436f-83b6-e5bbeaac0a30" containerName="payload-processing" Nov 03 01:22:19.980040 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:22:19.978963 4939 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="007181d2-dac6-436f-83b6-e5bbeaac0a30" containerName="payload-processing" Nov 03 01:22:19.980040 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:22:19.979149 4939 memory_manager.go:365] "RemoveStaleState removing state" podUID="007181d2-dac6-436f-83b6-e5bbeaac0a30" containerName="payload-processing" Nov 03 01:22:19.980040 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:22:19.979161 4939 memory_manager.go:365] "RemoveStaleState removing state" podUID="dc81bbe2-2c01-4962-84a8-46b954bad81a" containerName="payload-processing" Nov 03 01:22:19.983702 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:22:19.983679 4939 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-a88862-openshift-default-7b98d598f-jbgzc" Nov 03 01:22:19.990355 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:22:19.990327 4939 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-a-a88862-openshift-default-dockercfg-k95sf\"" Nov 03 01:22:19.997806 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:22:19.997778 4939 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-a88862-openshift-default-7b98d598f-jbgzc"] Nov 03 01:22:20.007978 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:22:20.007944 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0de902db-4a72-49b2-8f60-8b63663cc98d-workload-certs\") pod \"e2e-inf-a-a88862-openshift-default-7b98d598f-jbgzc\" (UID: \"0de902db-4a72-49b2-8f60-8b63663cc98d\") " pod="openshift-ingress/e2e-inf-a-a88862-openshift-default-7b98d598f-jbgzc" Nov 03 01:22:20.009202 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:22:20.008066 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0de902db-4a72-49b2-8f60-8b63663cc98d-istiod-ca-cert\") pod \"e2e-inf-a-a88862-openshift-default-7b98d598f-jbgzc\" (UID: \"0de902db-4a72-49b2-8f60-8b63663cc98d\") " pod="openshift-ingress/e2e-inf-a-a88862-openshift-default-7b98d598f-jbgzc" Nov 03 01:22:20.009202 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:22:20.008134 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0de902db-4a72-49b2-8f60-8b63663cc98d-credential-socket\") pod \"e2e-inf-a-a88862-openshift-default-7b98d598f-jbgzc\" (UID: \"0de902db-4a72-49b2-8f60-8b63663cc98d\") " pod="openshift-ingress/e2e-inf-a-a88862-openshift-default-7b98d598f-jbgzc" Nov 03 01:22:20.009202 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:22:20.008190 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0de902db-4a72-49b2-8f60-8b63663cc98d-istio-envoy\") pod \"e2e-inf-a-a88862-openshift-default-7b98d598f-jbgzc\" (UID: \"0de902db-4a72-49b2-8f60-8b63663cc98d\") " pod="openshift-ingress/e2e-inf-a-a88862-openshift-default-7b98d598f-jbgzc" Nov 03 01:22:20.009202 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:22:20.008330 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0de902db-4a72-49b2-8f60-8b63663cc98d-istio-token\") pod \"e2e-inf-a-a88862-openshift-default-7b98d598f-jbgzc\" (UID: \"0de902db-4a72-49b2-8f60-8b63663cc98d\") " pod="openshift-ingress/e2e-inf-a-a88862-openshift-default-7b98d598f-jbgzc" Nov 03 01:22:20.009202 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:22:20.008408 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0de902db-4a72-49b2-8f60-8b63663cc98d-workload-socket\") pod \"e2e-inf-a-a88862-openshift-default-7b98d598f-jbgzc\" (UID: \"0de902db-4a72-49b2-8f60-8b63663cc98d\") " pod="openshift-ingress/e2e-inf-a-a88862-openshift-default-7b98d598f-jbgzc" Nov 03 01:22:20.009202 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:22:20.008450 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0de902db-4a72-49b2-8f60-8b63663cc98d-istio-podinfo\") pod \"e2e-inf-a-a88862-openshift-default-7b98d598f-jbgzc\" (UID: \"0de902db-4a72-49b2-8f60-8b63663cc98d\") " pod="openshift-ingress/e2e-inf-a-a88862-openshift-default-7b98d598f-jbgzc" Nov 03 01:22:20.009202 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:22:20.008548 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4g5mt\" (UniqueName: \"kubernetes.io/projected/0de902db-4a72-49b2-8f60-8b63663cc98d-kube-api-access-4g5mt\") pod \"e2e-inf-a-a88862-openshift-default-7b98d598f-jbgzc\" (UID: \"0de902db-4a72-49b2-8f60-8b63663cc98d\") " pod="openshift-ingress/e2e-inf-a-a88862-openshift-default-7b98d598f-jbgzc" Nov 03 01:22:20.009202 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:22:20.008697 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0de902db-4a72-49b2-8f60-8b63663cc98d-istio-data\") pod \"e2e-inf-a-a88862-openshift-default-7b98d598f-jbgzc\" (UID: \"0de902db-4a72-49b2-8f60-8b63663cc98d\") " pod="openshift-ingress/e2e-inf-a-a88862-openshift-default-7b98d598f-jbgzc" Nov 03 01:22:20.042573 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:22:20.042533 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:22:20.110353 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:22:20.110303 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0de902db-4a72-49b2-8f60-8b63663cc98d-workload-socket\") pod \"e2e-inf-a-a88862-openshift-default-7b98d598f-jbgzc\" (UID: \"0de902db-4a72-49b2-8f60-8b63663cc98d\") " pod="openshift-ingress/e2e-inf-a-a88862-openshift-default-7b98d598f-jbgzc" Nov 03 01:22:20.110562 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:22:20.110363 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0de902db-4a72-49b2-8f60-8b63663cc98d-istio-podinfo\") pod \"e2e-inf-a-a88862-openshift-default-7b98d598f-jbgzc\" (UID: \"0de902db-4a72-49b2-8f60-8b63663cc98d\") " pod="openshift-ingress/e2e-inf-a-a88862-openshift-default-7b98d598f-jbgzc" Nov 03 01:22:20.110562 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:22:20.110450 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4g5mt\" (UniqueName: \"kubernetes.io/projected/0de902db-4a72-49b2-8f60-8b63663cc98d-kube-api-access-4g5mt\") pod \"e2e-inf-a-a88862-openshift-default-7b98d598f-jbgzc\" (UID: \"0de902db-4a72-49b2-8f60-8b63663cc98d\") " pod="openshift-ingress/e2e-inf-a-a88862-openshift-default-7b98d598f-jbgzc" Nov 03 01:22:20.110692 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:22:20.110614 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0de902db-4a72-49b2-8f60-8b63663cc98d-istio-data\") pod \"e2e-inf-a-a88862-openshift-default-7b98d598f-jbgzc\" (UID: \"0de902db-4a72-49b2-8f60-8b63663cc98d\") " pod="openshift-ingress/e2e-inf-a-a88862-openshift-default-7b98d598f-jbgzc" Nov 03 01:22:20.110750 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:22:20.110702 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0de902db-4a72-49b2-8f60-8b63663cc98d-workload-certs\") pod \"e2e-inf-a-a88862-openshift-default-7b98d598f-jbgzc\" (UID: \"0de902db-4a72-49b2-8f60-8b63663cc98d\") " pod="openshift-ingress/e2e-inf-a-a88862-openshift-default-7b98d598f-jbgzc" Nov 03 01:22:20.110750 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:22:20.110709 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0de902db-4a72-49b2-8f60-8b63663cc98d-workload-socket\") pod \"e2e-inf-a-a88862-openshift-default-7b98d598f-jbgzc\" (UID: \"0de902db-4a72-49b2-8f60-8b63663cc98d\") " pod="openshift-ingress/e2e-inf-a-a88862-openshift-default-7b98d598f-jbgzc" Nov 03 01:22:20.110872 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:22:20.110816 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0de902db-4a72-49b2-8f60-8b63663cc98d-istiod-ca-cert\") pod \"e2e-inf-a-a88862-openshift-default-7b98d598f-jbgzc\" (UID: \"0de902db-4a72-49b2-8f60-8b63663cc98d\") " pod="openshift-ingress/e2e-inf-a-a88862-openshift-default-7b98d598f-jbgzc" Nov 03 01:22:20.110872 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:22:20.110864 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0de902db-4a72-49b2-8f60-8b63663cc98d-credential-socket\") pod \"e2e-inf-a-a88862-openshift-default-7b98d598f-jbgzc\" (UID: \"0de902db-4a72-49b2-8f60-8b63663cc98d\") " pod="openshift-ingress/e2e-inf-a-a88862-openshift-default-7b98d598f-jbgzc" Nov 03 01:22:20.111051 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:22:20.110946 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0de902db-4a72-49b2-8f60-8b63663cc98d-istio-envoy\") pod \"e2e-inf-a-a88862-openshift-default-7b98d598f-jbgzc\" (UID: \"0de902db-4a72-49b2-8f60-8b63663cc98d\") " pod="openshift-ingress/e2e-inf-a-a88862-openshift-default-7b98d598f-jbgzc" Nov 03 01:22:20.111051 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:22:20.110990 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0de902db-4a72-49b2-8f60-8b63663cc98d-istio-data\") pod \"e2e-inf-a-a88862-openshift-default-7b98d598f-jbgzc\" (UID: \"0de902db-4a72-49b2-8f60-8b63663cc98d\") " pod="openshift-ingress/e2e-inf-a-a88862-openshift-default-7b98d598f-jbgzc" Nov 03 01:22:20.111177 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:22:20.111067 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0de902db-4a72-49b2-8f60-8b63663cc98d-workload-certs\") pod \"e2e-inf-a-a88862-openshift-default-7b98d598f-jbgzc\" (UID: \"0de902db-4a72-49b2-8f60-8b63663cc98d\") " pod="openshift-ingress/e2e-inf-a-a88862-openshift-default-7b98d598f-jbgzc" Nov 03 01:22:20.111177 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:22:20.111115 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0de902db-4a72-49b2-8f60-8b63663cc98d-istio-token\") pod \"e2e-inf-a-a88862-openshift-default-7b98d598f-jbgzc\" (UID: \"0de902db-4a72-49b2-8f60-8b63663cc98d\") " pod="openshift-ingress/e2e-inf-a-a88862-openshift-default-7b98d598f-jbgzc" Nov 03 01:22:20.111292 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:22:20.111253 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0de902db-4a72-49b2-8f60-8b63663cc98d-credential-socket\") pod \"e2e-inf-a-a88862-openshift-default-7b98d598f-jbgzc\" (UID: \"0de902db-4a72-49b2-8f60-8b63663cc98d\") " pod="openshift-ingress/e2e-inf-a-a88862-openshift-default-7b98d598f-jbgzc" Nov 03 01:22:20.111680 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:22:20.111654 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0de902db-4a72-49b2-8f60-8b63663cc98d-istiod-ca-cert\") pod \"e2e-inf-a-a88862-openshift-default-7b98d598f-jbgzc\" (UID: \"0de902db-4a72-49b2-8f60-8b63663cc98d\") " pod="openshift-ingress/e2e-inf-a-a88862-openshift-default-7b98d598f-jbgzc" Nov 03 01:22:20.113237 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:22:20.113215 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0de902db-4a72-49b2-8f60-8b63663cc98d-istio-envoy\") pod \"e2e-inf-a-a88862-openshift-default-7b98d598f-jbgzc\" (UID: \"0de902db-4a72-49b2-8f60-8b63663cc98d\") " pod="openshift-ingress/e2e-inf-a-a88862-openshift-default-7b98d598f-jbgzc" Nov 03 01:22:20.113357 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:22:20.113306 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0de902db-4a72-49b2-8f60-8b63663cc98d-istio-podinfo\") pod \"e2e-inf-a-a88862-openshift-default-7b98d598f-jbgzc\" (UID: \"0de902db-4a72-49b2-8f60-8b63663cc98d\") " pod="openshift-ingress/e2e-inf-a-a88862-openshift-default-7b98d598f-jbgzc" Nov 03 01:22:20.139964 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:22:20.122022 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0de902db-4a72-49b2-8f60-8b63663cc98d-istio-token\") pod \"e2e-inf-a-a88862-openshift-default-7b98d598f-jbgzc\" (UID: \"0de902db-4a72-49b2-8f60-8b63663cc98d\") " pod="openshift-ingress/e2e-inf-a-a88862-openshift-default-7b98d598f-jbgzc" Nov 03 01:22:20.139964 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:22:20.122581 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4g5mt\" (UniqueName: \"kubernetes.io/projected/0de902db-4a72-49b2-8f60-8b63663cc98d-kube-api-access-4g5mt\") pod \"e2e-inf-a-a88862-openshift-default-7b98d598f-jbgzc\" (UID: \"0de902db-4a72-49b2-8f60-8b63663cc98d\") " pod="openshift-ingress/e2e-inf-a-a88862-openshift-default-7b98d598f-jbgzc" Nov 03 01:22:20.254982 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:22:20.253497 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:22:20.313957 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:22:20.308402 4939 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-a88862-openshift-default-7b98d598f-jbgzc" Nov 03 01:22:20.719617 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:22:20.719576 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:22:20.832921 ip-10-0-140-107 kubenswrapper[4939]: W1103 01:22:20.832872 4939 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0de902db_4a72_49b2_8f60_8b63663cc98d.slice/crio-87f49ec8ca1466f9c260605200e31f0cd67864278b2dcc6f1256df7f234ae819 WatchSource:0}: Error finding container 87f49ec8ca1466f9c260605200e31f0cd67864278b2dcc6f1256df7f234ae819: Status 404 returned error can't find the container with id 87f49ec8ca1466f9c260605200e31f0cd67864278b2dcc6f1256df7f234ae819 Nov 03 01:22:20.834661 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:22:20.834635 4939 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-a88862-openshift-default-7b98d598f-jbgzc"] Nov 03 01:22:20.836892 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:22:20.836822 4939 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 01:22:20.836982 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:22:20.836904 4939 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 01:22:20.836982 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:22:20.836964 4939 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 01:22:21.456251 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:22:21.456215 4939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-a88862-openshift-default-7b98d598f-jbgzc" event={"ID":"0de902db-4a72-49b2-8f60-8b63663cc98d","Type":"ContainerStarted","Data":"1c190ea68c6e2b48ce74b7d1e60eb45265c6485d155a6e2319418f0a058075b2"} Nov 03 01:22:21.456733 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:22:21.456273 4939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-a88862-openshift-default-7b98d598f-jbgzc" event={"ID":"0de902db-4a72-49b2-8f60-8b63663cc98d","Type":"ContainerStarted","Data":"87f49ec8ca1466f9c260605200e31f0cd67864278b2dcc6f1256df7f234ae819"} Nov 03 01:22:21.481387 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:22:21.481340 4939 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-a-a88862-openshift-default-7b98d598f-jbgzc" podStartSLOduration=2.481327206 podStartE2EDuration="2.481327206s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 01:22:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 01:22:21.477782025 +0000 UTC m=+2489.742116970" watchObservedRunningTime="2026-11-03 01:22:21.481327206 +0000 UTC m=+2489.745662152" Nov 03 01:22:21.491812 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:22:21.491780 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:22:22.094988 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:22:22.094952 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:22:22.310332 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:22:22.310299 4939 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-a88862-openshift-default-7b98d598f-jbgzc" Nov 03 01:22:22.318464 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:22:22.318432 4939 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-a88862-openshift-default-7b98d598f-jbgzc" Nov 03 01:22:22.465528 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:22:22.465442 4939 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-a88862-openshift-default-7b98d598f-jbgzc" Nov 03 01:22:22.466459 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:22:22.466438 4939 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-a88862-openshift-default-7b98d598f-jbgzc" Nov 03 01:22:22.537741 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:22:22.537692 4939 kuberuntime_container.go:898] "Killing container with a grace period" pod="models-as-a-service/payload-processing-external-model-76c77f5454-842p6" podUID="7eeeafad-d236-43d4-9661-1e3011e274fc" containerName="payload-processing" containerID="cri-o://ba502782bf1fe19c765b490fe04393e5c8545856b28165534c1007174c09c894" gracePeriod=25 Nov 03 01:22:22.844762 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:22:22.844736 4939 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-76c77f5454-842p6" Nov 03 01:22:22.917991 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:22:22.917959 4939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7eeeafad-d236-43d4-9661-1e3011e274fc-kube-api-access-gcs2q\" (UniqueName: \"kubernetes.io/projected/7eeeafad-d236-43d4-9661-1e3011e274fc-kube-api-access-gcs2q\") pod \"7eeeafad-d236-43d4-9661-1e3011e274fc\" (UID: \"7eeeafad-d236-43d4-9661-1e3011e274fc\") " Nov 03 01:22:22.918416 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:22:22.918389 4939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7eeeafad-d236-43d4-9661-1e3011e274fc-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7eeeafad-d236-43d4-9661-1e3011e274fc-plugins-config-volume\") pod \"7eeeafad-d236-43d4-9661-1e3011e274fc\" (UID: \"7eeeafad-d236-43d4-9661-1e3011e274fc\") " Nov 03 01:22:22.918564 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:22:22.918463 4939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7eeeafad-d236-43d4-9661-1e3011e274fc-provider-credentials\" (UniqueName: \"kubernetes.io/projected/7eeeafad-d236-43d4-9661-1e3011e274fc-provider-credentials\") pod \"7eeeafad-d236-43d4-9661-1e3011e274fc\" (UID: \"7eeeafad-d236-43d4-9661-1e3011e274fc\") " Nov 03 01:22:22.918642 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:22:22.918561 4939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7eeeafad-d236-43d4-9661-1e3011e274fc-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/7eeeafad-d236-43d4-9661-1e3011e274fc-routing-overlay-volume\") pod \"7eeeafad-d236-43d4-9661-1e3011e274fc\" (UID: \"7eeeafad-d236-43d4-9661-1e3011e274fc\") " Nov 03 01:22:22.919236 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:22:22.918873 4939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7eeeafad-d236-43d4-9661-1e3011e274fc-plugins-config-volume" pod "7eeeafad-d236-43d4-9661-1e3011e274fc" (UID: "7eeeafad-d236-43d4-9661-1e3011e274fc"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 01:22:22.919236 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:22:22.919195 4939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7eeeafad-d236-43d4-9661-1e3011e274fc-routing-overlay-volume" pod "7eeeafad-d236-43d4-9661-1e3011e274fc" (UID: "7eeeafad-d236-43d4-9661-1e3011e274fc"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 01:22:22.919583 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:22:22.919557 4939 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7eeeafad-d236-43d4-9661-1e3011e274fc-plugins-config-volume\") on node \"ip-10-0-140-107.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:22:22.919583 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:22:22.919581 4939 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/7eeeafad-d236-43d4-9661-1e3011e274fc-routing-overlay-volume\") on node \"ip-10-0-140-107.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:22:22.920622 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:22:22.920580 4939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7eeeafad-d236-43d4-9661-1e3011e274fc-kube-api-access-gcs2q" pod "7eeeafad-d236-43d4-9661-1e3011e274fc" (UID: "7eeeafad-d236-43d4-9661-1e3011e274fc"). InnerVolumeSpecName "kube-api-access-gcs2q". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 01:22:22.921700 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:22:22.921674 4939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7eeeafad-d236-43d4-9661-1e3011e274fc-provider-credentials" pod "7eeeafad-d236-43d4-9661-1e3011e274fc" (UID: "7eeeafad-d236-43d4-9661-1e3011e274fc"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 01:22:23.021457 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:22:23.021049 4939 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gcs2q\" (UniqueName: \"kubernetes.io/projected/7eeeafad-d236-43d4-9661-1e3011e274fc-kube-api-access-gcs2q\") on node \"ip-10-0-140-107.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:22:23.021457 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:22:23.021088 4939 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/7eeeafad-d236-43d4-9661-1e3011e274fc-provider-credentials\") on node \"ip-10-0-140-107.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:22:23.057770 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:22:23.057730 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:22:23.475846 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:22:23.475812 4939 generic.go:356] "Generic (PLEG): container finished" podID="7eeeafad-d236-43d4-9661-1e3011e274fc" containerID="ba502782bf1fe19c765b490fe04393e5c8545856b28165534c1007174c09c894" exitCode=0 Nov 03 01:22:23.476301 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:22:23.475854 4939 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-76c77f5454-842p6" Nov 03 01:22:23.476301 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:22:23.475893 4939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-76c77f5454-842p6" event={"ID":"7eeeafad-d236-43d4-9661-1e3011e274fc","Type":"ContainerDied","Data":"ba502782bf1fe19c765b490fe04393e5c8545856b28165534c1007174c09c894"} Nov 03 01:22:23.476301 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:22:23.475955 4939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-76c77f5454-842p6" event={"ID":"7eeeafad-d236-43d4-9661-1e3011e274fc","Type":"ContainerDied","Data":"98b5a8321a5a537319086920856d555db8c696c5c420e1085cbe2788224c160b"} Nov 03 01:22:23.476301 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:22:23.475980 4939 scope.go:122] "RemoveContainer" containerID="ba502782bf1fe19c765b490fe04393e5c8545856b28165534c1007174c09c894" Nov 03 01:22:23.489481 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:22:23.489461 4939 scope.go:122] "RemoveContainer" containerID="ba502782bf1fe19c765b490fe04393e5c8545856b28165534c1007174c09c894" Nov 03 01:22:23.489771 ip-10-0-140-107 kubenswrapper[4939]: E1103 01:22:23.489750 4939 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ba502782bf1fe19c765b490fe04393e5c8545856b28165534c1007174c09c894\": container with ID starting with ba502782bf1fe19c765b490fe04393e5c8545856b28165534c1007174c09c894 not found: ID does not exist" containerID="ba502782bf1fe19c765b490fe04393e5c8545856b28165534c1007174c09c894" Nov 03 01:22:23.489849 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:22:23.489779 4939 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ba502782bf1fe19c765b490fe04393e5c8545856b28165534c1007174c09c894"} err="failed to get container status \"ba502782bf1fe19c765b490fe04393e5c8545856b28165534c1007174c09c894\": rpc error: code = NotFound desc = could not find container \"ba502782bf1fe19c765b490fe04393e5c8545856b28165534c1007174c09c894\": container with ID starting with ba502782bf1fe19c765b490fe04393e5c8545856b28165534c1007174c09c894 not found: ID does not exist" Nov 03 01:22:23.504860 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:22:23.504840 4939 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["models-as-a-service/payload-processing-external-model-76c77f5454-842p6"] Nov 03 01:22:23.509723 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:22:23.509700 4939 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["models-as-a-service/payload-processing-external-model-76c77f5454-842p6"] Nov 03 01:22:23.708226 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:22:23.708188 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:22:23.825680 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:22:23.825596 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:22:23.993975 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:22:23.989749 4939 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7eeeafad-d236-43d4-9661-1e3011e274fc" path="/var/lib/kubelet/pods/7eeeafad-d236-43d4-9661-1e3011e274fc/volumes" Nov 03 01:22:24.595617 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:22:24.595587 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:22:24.926306 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:22:24.926155 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:22:24.954334 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:22:24.954301 4939 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-a88862-6bccb6967f-46486"] Nov 03 01:22:24.955410 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:22:24.955388 4939 cpu_manager.go:414] "containerMap: removing container" podUID="7eeeafad-d236-43d4-9661-1e3011e274fc" containerName="payload-processing" Nov 03 01:22:24.955480 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:22:24.955413 4939 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7eeeafad-d236-43d4-9661-1e3011e274fc" containerName="payload-processing" Nov 03 01:22:24.955694 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:22:24.955680 4939 memory_manager.go:365] "RemoveStaleState removing state" podUID="7eeeafad-d236-43d4-9661-1e3011e274fc" containerName="payload-processing" Nov 03 01:22:24.961591 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:22:24.961567 4939 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-a88862-6bccb6967f-46486" Nov 03 01:22:24.966646 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:22:24.966622 4939 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-a-a88862\"" Nov 03 01:22:24.967534 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:22:24.967513 4939 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-a88862-6bccb6967f-46486"] Nov 03 01:22:25.041126 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:22:25.041097 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:22:25.041446 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:22:25.041412 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6q5bz\" (UniqueName: \"kubernetes.io/projected/dd50d5d4-bbf2-439b-a851-2c2ae58da605-kube-api-access-6q5bz\") pod \"payload-processing-e2e-inf-a-a88862-6bccb6967f-46486\" (UID: \"dd50d5d4-bbf2-439b-a851-2c2ae58da605\") " pod="openshift-ingress/payload-processing-e2e-inf-a-a88862-6bccb6967f-46486" Nov 03 01:22:25.041640 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:22:25.041620 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dd50d5d4-bbf2-439b-a851-2c2ae58da605-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-a88862-6bccb6967f-46486\" (UID: \"dd50d5d4-bbf2-439b-a851-2c2ae58da605\") " pod="openshift-ingress/payload-processing-e2e-inf-a-a88862-6bccb6967f-46486" Nov 03 01:22:25.149944 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:22:25.149284 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6q5bz\" (UniqueName: \"kubernetes.io/projected/dd50d5d4-bbf2-439b-a851-2c2ae58da605-kube-api-access-6q5bz\") pod \"payload-processing-e2e-inf-a-a88862-6bccb6967f-46486\" (UID: \"dd50d5d4-bbf2-439b-a851-2c2ae58da605\") " pod="openshift-ingress/payload-processing-e2e-inf-a-a88862-6bccb6967f-46486" Nov 03 01:22:25.149944 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:22:25.149545 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dd50d5d4-bbf2-439b-a851-2c2ae58da605-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-a88862-6bccb6967f-46486\" (UID: \"dd50d5d4-bbf2-439b-a851-2c2ae58da605\") " pod="openshift-ingress/payload-processing-e2e-inf-a-a88862-6bccb6967f-46486" Nov 03 01:22:25.153943 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:22:25.150864 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dd50d5d4-bbf2-439b-a851-2c2ae58da605-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-a88862-6bccb6967f-46486\" (UID: \"dd50d5d4-bbf2-439b-a851-2c2ae58da605\") " pod="openshift-ingress/payload-processing-e2e-inf-a-a88862-6bccb6967f-46486" Nov 03 01:22:25.244984 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:22:25.238617 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6q5bz\" (UniqueName: \"kubernetes.io/projected/dd50d5d4-bbf2-439b-a851-2c2ae58da605-kube-api-access-6q5bz\") pod \"payload-processing-e2e-inf-a-a88862-6bccb6967f-46486\" (UID: \"dd50d5d4-bbf2-439b-a851-2c2ae58da605\") " pod="openshift-ingress/payload-processing-e2e-inf-a-a88862-6bccb6967f-46486" Nov 03 01:22:25.288976 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:22:25.286214 4939 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-a88862-6bccb6967f-46486" Nov 03 01:22:25.743633 ip-10-0-140-107 kubenswrapper[4939]: W1103 01:22:25.743595 4939 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddd50d5d4_bbf2_439b_a851_2c2ae58da605.slice/crio-a1f6041cba2b4072fbd2bc9dfdf304dc1886f5bc5f01ea8a85d863607edae3d3 WatchSource:0}: Error finding container a1f6041cba2b4072fbd2bc9dfdf304dc1886f5bc5f01ea8a85d863607edae3d3: Status 404 returned error can't find the container with id a1f6041cba2b4072fbd2bc9dfdf304dc1886f5bc5f01ea8a85d863607edae3d3 Nov 03 01:22:25.744178 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:22:25.744116 4939 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-a88862-6bccb6967f-46486"] Nov 03 01:22:25.796804 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:22:25.796776 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:22:26.401213 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:22:26.401174 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:22:26.524284 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:22:26.524242 4939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-a88862-6bccb6967f-46486" event={"ID":"dd50d5d4-bbf2-439b-a851-2c2ae58da605","Type":"ContainerStarted","Data":"d022249c6b7f0ad709448febf97a28b05d4540b81fe89ae5ce58e2542e963980"} Nov 03 01:22:26.524471 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:22:26.524312 4939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-a88862-6bccb6967f-46486" event={"ID":"dd50d5d4-bbf2-439b-a851-2c2ae58da605","Type":"ContainerStarted","Data":"a1f6041cba2b4072fbd2bc9dfdf304dc1886f5bc5f01ea8a85d863607edae3d3"} Nov 03 01:22:26.524471 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:22:26.524385 4939 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-inf-a-a88862-6bccb6967f-46486" Nov 03 01:22:26.542635 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:22:26.542577 4939 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-inf-a-a88862-6bccb6967f-46486" podStartSLOduration=2.542563549 podStartE2EDuration="2.542563549s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 01:22:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 01:22:26.53959777 +0000 UTC m=+2494.803932717" watchObservedRunningTime="2026-11-03 01:22:26.542563549 +0000 UTC m=+2494.806898494" Nov 03 01:22:27.177583 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:22:27.177514 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:22:27.970166 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:22:27.970130 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:22:29.243496 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:22:29.243438 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:22:29.906731 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:22:29.906682 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:22:30.584988 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:22:30.584952 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:22:31.195138 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:22:31.195102 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:22:32.006142 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:22:32.006108 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:22:32.695421 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:22:32.695388 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:22:33.280003 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:22:33.279964 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:22:33.912130 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:22:33.912092 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:22:34.007850 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:22:34.007787 4939 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-4pw7v"] Nov 03 01:22:34.694112 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:22:34.694075 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:22:36.130933 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:22:36.126001 4939 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-88bbf7/simulator-w1-88bbf7-kserve-5fbb669f8f-42hgd" podUID="83b38883-2e7a-444d-aa68-52be4a25ae66" containerName="main" probeResult="failure" output="Get \"https://10.132.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 01:22:36.345942 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:22:36.344985 4939 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-88bbf7/distinct-2-w1-88bbf7-kserve-5457cdcdbd-l6tfd" podUID="a397931b-faa9-4c2e-8290-491350546663" containerName="main" probeResult="failure" output="Get \"https://10.132.0.85:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 01:22:36.487558 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:22:36.486001 4939 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-83755a/distinct-2-w3-83755a-kserve-6cf66b995d-2l47b" podUID="c3b05e88-5d5b-4d03-b28d-832f22e3c868" containerName="main" probeResult="failure" output="Get \"https://10.132.0.72:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 01:22:36.603206 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:22:36.603151 4939 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-f7680d/distinct-2-w2-f7680d-kserve-7d7c4564bf-cbv2c" podUID="2de18086-761b-4773-b9fc-15ddc4a220fc" containerName="main" probeResult="failure" output="Get \"https://10.132.0.75:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Nov 03 01:22:36.833927 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:22:36.833864 4939 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-f7680d/premium-w2-f7680d-kserve-76565b8cf9-hqbbt" podUID="4712e708-18a2-4a13-a12c-a5669397b2c3" containerName="main" probeResult="failure" output="Get \"https://10.132.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 01:22:36.859410 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:22:36.859372 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:22:36.877423 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:22:36.877382 4939 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-2f9a3a/simulator-w0-2f9a3a-kserve-9596bbb8f-2trgz" podUID="207e2207-fb6c-4076-95df-72e95671c261" containerName="main" probeResult="failure" output="Get \"https://10.132.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 01:22:37.496021 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:22:37.491364 4939 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-88bbf7/distinct-2-w1-88bbf7-kserve-5457cdcdbd-l6tfd" podUID="a397931b-faa9-4c2e-8290-491350546663" containerName="main" probeResult="failure" output="Get \"https://10.132.0.85:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 01:22:37.496021 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:22:37.491718 4939 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-83755a/distinct-2-w3-83755a-kserve-6cf66b995d-2l47b" podUID="c3b05e88-5d5b-4d03-b28d-832f22e3c868" containerName="main" probeResult="failure" output="Get \"https://10.132.0.72:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 01:22:37.496021 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:22:37.492287 4939 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-88bbf7/distinct-2-w1-88bbf7-kserve-5457cdcdbd-l6tfd" Nov 03 01:22:37.496021 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:22:37.492348 4939 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-83755a/distinct-2-w3-83755a-kserve-6cf66b995d-2l47b" Nov 03 01:22:37.566472 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:22:37.566433 4939 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-88bbf7/distinct-2-w1-88bbf7-kserve-5457cdcdbd-l6tfd" Nov 03 01:22:37.610085 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:22:37.610005 4939 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-f7680d/distinct-2-w2-f7680d-kserve-7d7c4564bf-cbv2c" podUID="2de18086-761b-4773-b9fc-15ddc4a220fc" containerName="main" probeResult="failure" output="Get \"https://10.132.0.75:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 01:22:37.611255 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:22:37.611152 4939 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-f7680d/distinct-2-w2-f7680d-kserve-7d7c4564bf-cbv2c" Nov 03 01:22:37.642736 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:22:37.642704 4939 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-inf-a-a88862-6bccb6967f-46486" Nov 03 01:22:37.844004 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:22:37.839213 4939 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-f7680d/premium-w2-f7680d-kserve-76565b8cf9-hqbbt" podUID="4712e708-18a2-4a13-a12c-a5669397b2c3" containerName="main" probeResult="failure" output="Get \"https://10.132.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 01:22:37.844004 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:22:37.840649 4939 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-f7680d/premium-w2-f7680d-kserve-76565b8cf9-hqbbt" Nov 03 01:22:37.870248 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:22:37.870207 4939 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-2f9a3a/distinct-w0-2f9a3a-kserve-555565cfcf-jtcr9" podUID="ac0ece06-8718-463f-915f-5315988d9711" containerName="main" probeResult="failure" output="Get \"https://10.132.0.76:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Nov 03 01:22:37.893037 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:22:37.892993 4939 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-2f9a3a/simulator-w0-2f9a3a-kserve-9596bbb8f-2trgz" podUID="207e2207-fb6c-4076-95df-72e95671c261" containerName="main" probeResult="failure" output="Get \"https://10.132.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 01:22:38.097985 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:22:38.052240 4939 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-2f9a3a/simulator-w0-2f9a3a-kserve-9596bbb8f-2trgz" Nov 03 01:22:38.103059 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:22:38.103030 4939 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-f7680d/distinct-2-w2-f7680d-kserve-7d7c4564bf-cbv2c" Nov 03 01:22:38.105450 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:22:38.105427 4939 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-83755a/distinct-2-w3-83755a-kserve-6cf66b995d-2l47b" Nov 03 01:22:38.839407 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:22:38.839361 4939 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-f7680d/premium-w2-f7680d-kserve-76565b8cf9-hqbbt" podUID="4712e708-18a2-4a13-a12c-a5669397b2c3" containerName="main" probeResult="failure" output="Get \"https://10.132.0.63:8000/ready\": context deadline exceeded" Nov 03 01:22:38.895362 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:22:38.895316 4939 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-2f9a3a/simulator-w0-2f9a3a-kserve-9596bbb8f-2trgz" podUID="207e2207-fb6c-4076-95df-72e95671c261" containerName="main" probeResult="failure" output="Get \"https://10.132.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 01:22:39.032930 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:22:39.018943 4939 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-f7680d/premium-w2-f7680d-kserve-76565b8cf9-hqbbt" Nov 03 01:22:39.336303 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:22:39.336276 4939 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-2f9a3a/simulator-w0-2f9a3a-kserve-9596bbb8f-2trgz" Nov 03 01:22:39.370786 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:22:39.370751 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:22:40.454190 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:22:40.454156 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:22:41.217296 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:22:41.217259 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:22:41.905242 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:22:41.905204 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:22:42.687055 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:22:42.687021 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:22:43.516475 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:22:43.516435 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:22:44.295505 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:22:44.295469 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:22:44.547954 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:22:44.547834 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:22:44.900531 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:22:44.900491 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:22:45.682742 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:22:45.682701 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:22:45.814051 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:22:45.814008 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:22:45.929461 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:22:45.929432 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:22:46.654832 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:22:46.654797 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:22:47.281174 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:22:47.281135 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:22:47.972245 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:22:47.972211 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:22:48.115606 ip-10-0-140-107 kubenswrapper[4939]: E1103 01:22:48.115569 4939 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-sthjh" containerName="extractor" Nov 03 01:22:48.840640 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:22:48.840603 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:22:49.493889 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:22:49.493845 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:22:50.101358 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:22:50.101322 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:22:50.707887 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:22:50.707838 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:22:51.409437 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:22:51.409389 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:22:52.055319 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:22:52.055257 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:22:52.681692 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:22:52.681596 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:22:53.029139 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:22:53.029040 4939 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-016f9241-fngh8"] Nov 03 01:22:53.035882 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:22:53.035851 4939 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-016f9241-fngh8"] Nov 03 01:22:53.378896 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:22:53.378860 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:22:53.879942 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:22:53.879883 4939 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6905cd74-619b-4184-a706-08d4722436c9" path="/var/lib/kubelet/pods/6905cd74-619b-4184-a706-08d4722436c9/volumes" Nov 03 01:22:54.094867 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:22:54.094834 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:22:54.710301 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:22:54.710262 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:22:55.377153 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:22:55.377123 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:22:56.061342 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:22:56.061311 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:22:56.780734 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:22:56.780699 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:22:57.398956 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:22:57.398889 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:22:57.984983 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:22:57.984942 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:22:58.601782 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:22:58.601734 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:22:59.318027 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:22:59.317946 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:22:59.983428 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:22:59.983395 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:23:00.658358 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:23:00.658320 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:23:01.282725 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:23:01.282686 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:23:01.999744 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:23:01.999710 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:23:02.663358 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:23:02.663324 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:23:03.278132 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:23:03.278095 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:23:04.002857 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:23:04.002812 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:23:05.030840 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:23:05.030788 4939 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b20dc57a-hc8mf"] Nov 03 01:23:05.036986 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:23:05.036958 4939 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b20dc57a-hc8mf"] Nov 03 01:23:05.040641 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:23:05.040611 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:23:05.750743 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:23:05.750699 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:23:05.877421 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:23:05.877385 4939 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9a9131c6-08db-48a1-a356-e4a19c077c22" path="/var/lib/kubelet/pods/9a9131c6-08db-48a1-a356-e4a19c077c22/volumes" Nov 03 01:23:06.380453 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:23:06.380412 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:23:07.067755 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:23:07.067722 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:23:07.676415 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:23:07.676379 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:23:08.378315 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:23:08.378277 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:23:09.071748 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:23:09.071711 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:23:09.582147 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:23:09.582116 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:23:09.884279 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:23:09.884191 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:23:10.090311 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:23:10.090274 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:23:10.521428 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:23:10.521396 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:23:10.741030 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:23:10.740975 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:23:10.907930 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:23:10.898746 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:23:11.901530 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:23:11.901495 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:23:12.651081 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:23:12.651046 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:23:13.694397 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:23:13.694360 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:23:14.524725 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:23:14.524675 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:23:14.771620 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:23:14.771568 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:23:14.903633 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:23:14.903593 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:23:15.032575 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:23:15.032535 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:23:15.822500 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:23:15.822466 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:23:15.981469 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:23:15.981416 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:23:16.096948 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:23:16.096891 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:23:16.787017 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:23:16.786982 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:23:17.533578 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:23:17.533544 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:23:18.035881 ip-10-0-140-107 kubenswrapper[4939]: E1103 01:23:18.035839 4939 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-sthjh" containerName="extractor" Nov 03 01:23:18.374445 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:23:18.374378 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:23:19.020256 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:23:19.020204 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:23:19.798619 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:23:19.798580 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:23:20.789784 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:23:20.789687 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:23:21.391693 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:23:21.391659 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:23:22.046537 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:23:22.046480 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:23:22.680000 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:23:22.679963 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:23:23.487334 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:23:23.487298 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:23:23.506237 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:23:23.506083 4939 patch_prober.go:29] interesting pod/e2e-inf-a-a88862-openshift-default-7b98d598f-jbgzc container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.1.36:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 01:23:23.506237 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:23:23.506147 4939 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-a-a88862-openshift-default-7b98d598f-jbgzc" podUID="0de902db-4a72-49b2-8f60-8b63663cc98d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.36:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 01:23:24.380835 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:23:24.380796 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:23:24.985465 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:23:24.985384 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:23:25.094518 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:23:25.094486 4939 generic.go:356] "Generic (PLEG): container finished" podID="0de902db-4a72-49b2-8f60-8b63663cc98d" containerID="1c190ea68c6e2b48ce74b7d1e60eb45265c6485d155a6e2319418f0a058075b2" exitCode=0 Nov 03 01:23:25.094738 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:23:25.094639 4939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-a88862-openshift-default-7b98d598f-jbgzc" event={"ID":"0de902db-4a72-49b2-8f60-8b63663cc98d","Type":"ContainerDied","Data":"1c190ea68c6e2b48ce74b7d1e60eb45265c6485d155a6e2319418f0a058075b2"} Nov 03 01:23:25.095292 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:23:25.095248 4939 scope.go:122] "RemoveContainer" containerID="1c190ea68c6e2b48ce74b7d1e60eb45265c6485d155a6e2319418f0a058075b2" Nov 03 01:23:25.097712 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:23:25.097677 4939 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 01:23:25.097841 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:23:25.097773 4939 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 01:23:25.097841 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:23:25.097824 4939 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 01:23:25.768205 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:23:25.768160 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:23:26.105306 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:23:26.105266 4939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-a88862-openshift-default-7b98d598f-jbgzc" event={"ID":"0de902db-4a72-49b2-8f60-8b63663cc98d","Type":"ContainerStarted","Data":"b27f1483f0bc66d6493c8775c31ddae558650ba330be12a2401c08afbcffa1da"} Nov 03 01:23:26.310246 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:23:26.310208 4939 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-a88862-openshift-default-7b98d598f-jbgzc" Nov 03 01:23:26.310427 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:23:26.310379 4939 patch_prober.go:29] interesting pod/e2e-inf-a-a88862-openshift-default-7b98d598f-jbgzc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.36:15021/healthz/ready\": dial tcp 10.132.1.36:15021: connect: connection refused" start-of-body= Nov 03 01:23:26.310476 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:23:26.310428 4939 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-a88862-openshift-default-7b98d598f-jbgzc" podUID="0de902db-4a72-49b2-8f60-8b63663cc98d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.36:15021/healthz/ready\": dial tcp 10.132.1.36:15021: connect: connection refused" Nov 03 01:23:26.486859 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:23:26.486769 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:23:27.236486 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:23:27.236450 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:23:27.315409 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:23:27.315368 4939 patch_prober.go:29] interesting pod/e2e-inf-a-a88862-openshift-default-7b98d598f-jbgzc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.36:15021/healthz/ready\": dial tcp 10.132.1.36:15021: connect: connection refused" start-of-body= Nov 03 01:23:27.315701 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:23:27.315670 4939 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-a88862-openshift-default-7b98d598f-jbgzc" podUID="0de902db-4a72-49b2-8f60-8b63663cc98d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.36:15021/healthz/ready\": dial tcp 10.132.1.36:15021: connect: connection refused" Nov 03 01:23:27.992114 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:23:27.992080 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:23:28.310658 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:23:28.310579 4939 patch_prober.go:29] interesting pod/e2e-inf-a-a88862-openshift-default-7b98d598f-jbgzc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.36:15021/healthz/ready\": dial tcp 10.132.1.36:15021: connect: connection refused" start-of-body= Nov 03 01:23:28.310658 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:23:28.310632 4939 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-a88862-openshift-default-7b98d598f-jbgzc" podUID="0de902db-4a72-49b2-8f60-8b63663cc98d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.36:15021/healthz/ready\": dial tcp 10.132.1.36:15021: connect: connection refused" Nov 03 01:23:28.615624 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:23:28.615592 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:23:29.310058 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:23:29.310020 4939 patch_prober.go:29] interesting pod/e2e-inf-a-a88862-openshift-default-7b98d598f-jbgzc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.36:15021/healthz/ready\": dial tcp 10.132.1.36:15021: connect: connection refused" start-of-body= Nov 03 01:23:29.310263 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:23:29.310084 4939 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-a88862-openshift-default-7b98d598f-jbgzc" podUID="0de902db-4a72-49b2-8f60-8b63663cc98d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.36:15021/healthz/ready\": dial tcp 10.132.1.36:15021: connect: connection refused" Nov 03 01:23:29.458561 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:23:29.458519 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:23:30.172491 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:23:30.172455 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:23:30.310793 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:23:30.310753 4939 patch_prober.go:29] interesting pod/e2e-inf-a-a88862-openshift-default-7b98d598f-jbgzc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.36:15021/healthz/ready\": dial tcp 10.132.1.36:15021: connect: connection refused" start-of-body= Nov 03 01:23:30.311064 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:23:30.310817 4939 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-a88862-openshift-default-7b98d598f-jbgzc" podUID="0de902db-4a72-49b2-8f60-8b63663cc98d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.36:15021/healthz/ready\": dial tcp 10.132.1.36:15021: connect: connection refused" Nov 03 01:23:30.798291 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:23:30.798255 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:23:31.310522 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:23:31.310484 4939 patch_prober.go:29] interesting pod/e2e-inf-a-a88862-openshift-default-7b98d598f-jbgzc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.36:15021/healthz/ready\": dial tcp 10.132.1.36:15021: connect: connection refused" start-of-body= Nov 03 01:23:31.310725 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:23:31.310536 4939 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-a88862-openshift-default-7b98d598f-jbgzc" podUID="0de902db-4a72-49b2-8f60-8b63663cc98d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.36:15021/healthz/ready\": dial tcp 10.132.1.36:15021: connect: connection refused" Nov 03 01:23:31.693934 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:23:31.673618 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:23:31.802665 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:23:31.802630 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:23:32.267988 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:23:32.267953 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:23:32.353236 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:23:32.353202 4939 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-a88862-openshift-default-7b98d598f-jbgzc" Nov 03 01:23:32.354852 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:23:32.354683 4939 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-a88862-openshift-default-7b98d598f-jbgzc" Nov 03 01:23:32.356396 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:23:32.356343 4939 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-a88862-openshift-default-7b98d598f-jbgzc" Nov 03 01:23:32.887719 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:23:32.887682 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:23:33.584737 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:23:33.584702 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:23:34.295252 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:23:34.295214 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:23:34.428670 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:23:34.428613 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:23:34.573380 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:23:34.573276 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:23:35.273108 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:23:35.273056 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:23:35.924035 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:23:35.923994 4939 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-4pw7v"] Nov 03 01:23:36.107585 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:23:36.107525 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:23:36.774415 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:23:36.774375 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:23:37.654054 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:23:37.654021 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:23:38.469617 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:23:38.469583 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:23:39.082206 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:23:39.082171 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:23:39.707678 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:23:39.707645 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:23:40.445854 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:23:40.445820 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:23:41.186590 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:23:41.186543 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:23:41.827809 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:23:41.827780 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:23:42.484011 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:23:42.483975 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:23:42.596463 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:23:42.596381 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:23:43.183267 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:23:43.183231 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:23:43.801191 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:23:43.801150 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:23:44.471296 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:23:44.471261 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:23:44.612423 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:23:44.612384 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:23:45.081885 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:23:45.081848 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:23:45.807919 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:23:45.807856 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:23:46.490881 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:23:46.490844 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:23:46.604396 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:23:46.604362 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:23:47.112315 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:23:47.112282 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:23:47.811173 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:23:47.811133 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:23:48.039110 ip-10-0-140-107 kubenswrapper[4939]: E1103 01:23:48.039073 4939 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-sthjh" containerName="extractor" Nov 03 01:23:48.540779 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:23:48.540744 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:23:49.146761 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:23:49.146703 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:23:49.864447 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:23:49.864411 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:23:50.481606 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:23:50.481577 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:23:51.187671 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:23:51.187635 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:23:51.800412 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:23:51.800373 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:23:52.425141 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:23:52.425101 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:23:53.042279 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:23:53.042243 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:23:53.788762 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:23:53.788710 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:23:54.809664 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:23:54.809628 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:23:55.126034 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:23:55.125988 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:23:55.554879 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:23:55.554789 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:23:56.241803 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:23:56.241772 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:23:56.915700 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:23:56.915662 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:23:56.916399 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:23:56.916373 4939 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-a-a88862/test-model-a88862-kserve-554fc8cc49-zl7l2"] Nov 03 01:23:56.927338 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:23:56.927309 4939 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-a88862/test-model-a88862-kserve-554fc8cc49-zl7l2" Nov 03 01:23:56.931683 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:23:56.931654 4939 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-a88862\"/\"openshift-service-ca.crt\"" Nov 03 01:23:56.933765 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:23:56.933717 4939 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-a88862\"/\"default-dockercfg-6mmcz\"" Nov 03 01:23:56.933981 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:23:56.933862 4939 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-a88862\"/\"kube-root-ca.crt\"" Nov 03 01:23:56.933981 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:23:56.933937 4939 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-a88862\"/\"test-model-a88862-kserve-self-signed-certs\"" Nov 03 01:23:56.936810 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:23:56.936783 4939 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-a88862/test-model-a88862-kserve-554fc8cc49-zl7l2"] Nov 03 01:23:57.003827 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:23:57.003794 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7c49ba9e-e354-4e34-be2e-2269aac5e015-tmp-dir\") pod \"test-model-a88862-kserve-554fc8cc49-zl7l2\" (UID: \"7c49ba9e-e354-4e34-be2e-2269aac5e015\") " pod="ai-tenant-e2e-inf-a-a88862/test-model-a88862-kserve-554fc8cc49-zl7l2" Nov 03 01:23:57.004052 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:23:57.003990 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7c49ba9e-e354-4e34-be2e-2269aac5e015-home\") pod \"test-model-a88862-kserve-554fc8cc49-zl7l2\" (UID: \"7c49ba9e-e354-4e34-be2e-2269aac5e015\") " pod="ai-tenant-e2e-inf-a-a88862/test-model-a88862-kserve-554fc8cc49-zl7l2" Nov 03 01:23:57.004117 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:23:57.004084 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7c49ba9e-e354-4e34-be2e-2269aac5e015-model-cache\") pod \"test-model-a88862-kserve-554fc8cc49-zl7l2\" (UID: \"7c49ba9e-e354-4e34-be2e-2269aac5e015\") " pod="ai-tenant-e2e-inf-a-a88862/test-model-a88862-kserve-554fc8cc49-zl7l2" Nov 03 01:23:57.004170 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:23:57.004122 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7c49ba9e-e354-4e34-be2e-2269aac5e015-dshm\") pod \"test-model-a88862-kserve-554fc8cc49-zl7l2\" (UID: \"7c49ba9e-e354-4e34-be2e-2269aac5e015\") " pod="ai-tenant-e2e-inf-a-a88862/test-model-a88862-kserve-554fc8cc49-zl7l2" Nov 03 01:23:57.004224 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:23:57.004206 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-55h2j\" (UniqueName: \"kubernetes.io/projected/7c49ba9e-e354-4e34-be2e-2269aac5e015-kube-api-access-55h2j\") pod \"test-model-a88862-kserve-554fc8cc49-zl7l2\" (UID: \"7c49ba9e-e354-4e34-be2e-2269aac5e015\") " pod="ai-tenant-e2e-inf-a-a88862/test-model-a88862-kserve-554fc8cc49-zl7l2" Nov 03 01:23:57.004271 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:23:57.004248 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7c49ba9e-e354-4e34-be2e-2269aac5e015-tls-certs\") pod \"test-model-a88862-kserve-554fc8cc49-zl7l2\" (UID: \"7c49ba9e-e354-4e34-be2e-2269aac5e015\") " pod="ai-tenant-e2e-inf-a-a88862/test-model-a88862-kserve-554fc8cc49-zl7l2" Nov 03 01:23:57.023012 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:23:57.022974 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:23:57.134582 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:23:57.134536 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7c49ba9e-e354-4e34-be2e-2269aac5e015-tmp-dir\") pod \"test-model-a88862-kserve-554fc8cc49-zl7l2\" (UID: \"7c49ba9e-e354-4e34-be2e-2269aac5e015\") " pod="ai-tenant-e2e-inf-a-a88862/test-model-a88862-kserve-554fc8cc49-zl7l2" Nov 03 01:23:57.137658 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:23:57.137622 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7c49ba9e-e354-4e34-be2e-2269aac5e015-home\") pod \"test-model-a88862-kserve-554fc8cc49-zl7l2\" (UID: \"7c49ba9e-e354-4e34-be2e-2269aac5e015\") " pod="ai-tenant-e2e-inf-a-a88862/test-model-a88862-kserve-554fc8cc49-zl7l2" Nov 03 01:23:57.138251 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:23:57.138232 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7c49ba9e-e354-4e34-be2e-2269aac5e015-model-cache\") pod \"test-model-a88862-kserve-554fc8cc49-zl7l2\" (UID: \"7c49ba9e-e354-4e34-be2e-2269aac5e015\") " pod="ai-tenant-e2e-inf-a-a88862/test-model-a88862-kserve-554fc8cc49-zl7l2" Nov 03 01:23:57.138801 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:23:57.138783 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7c49ba9e-e354-4e34-be2e-2269aac5e015-dshm\") pod \"test-model-a88862-kserve-554fc8cc49-zl7l2\" (UID: \"7c49ba9e-e354-4e34-be2e-2269aac5e015\") " pod="ai-tenant-e2e-inf-a-a88862/test-model-a88862-kserve-554fc8cc49-zl7l2" Nov 03 01:23:57.139447 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:23:57.139417 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-55h2j\" (UniqueName: \"kubernetes.io/projected/7c49ba9e-e354-4e34-be2e-2269aac5e015-kube-api-access-55h2j\") pod \"test-model-a88862-kserve-554fc8cc49-zl7l2\" (UID: \"7c49ba9e-e354-4e34-be2e-2269aac5e015\") " pod="ai-tenant-e2e-inf-a-a88862/test-model-a88862-kserve-554fc8cc49-zl7l2" Nov 03 01:23:57.148061 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:23:57.138699 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7c49ba9e-e354-4e34-be2e-2269aac5e015-model-cache\") pod \"test-model-a88862-kserve-554fc8cc49-zl7l2\" (UID: \"7c49ba9e-e354-4e34-be2e-2269aac5e015\") " pod="ai-tenant-e2e-inf-a-a88862/test-model-a88862-kserve-554fc8cc49-zl7l2" Nov 03 01:23:57.148244 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:23:57.138141 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7c49ba9e-e354-4e34-be2e-2269aac5e015-home\") pod \"test-model-a88862-kserve-554fc8cc49-zl7l2\" (UID: \"7c49ba9e-e354-4e34-be2e-2269aac5e015\") " pod="ai-tenant-e2e-inf-a-a88862/test-model-a88862-kserve-554fc8cc49-zl7l2" Nov 03 01:23:57.148349 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:23:57.136675 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7c49ba9e-e354-4e34-be2e-2269aac5e015-tmp-dir\") pod \"test-model-a88862-kserve-554fc8cc49-zl7l2\" (UID: \"7c49ba9e-e354-4e34-be2e-2269aac5e015\") " pod="ai-tenant-e2e-inf-a-a88862/test-model-a88862-kserve-554fc8cc49-zl7l2" Nov 03 01:23:57.148495 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:23:57.148478 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7c49ba9e-e354-4e34-be2e-2269aac5e015-tls-certs\") pod \"test-model-a88862-kserve-554fc8cc49-zl7l2\" (UID: \"7c49ba9e-e354-4e34-be2e-2269aac5e015\") " pod="ai-tenant-e2e-inf-a-a88862/test-model-a88862-kserve-554fc8cc49-zl7l2" Nov 03 01:23:57.183101 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:23:57.167221 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7c49ba9e-e354-4e34-be2e-2269aac5e015-dshm\") pod \"test-model-a88862-kserve-554fc8cc49-zl7l2\" (UID: \"7c49ba9e-e354-4e34-be2e-2269aac5e015\") " pod="ai-tenant-e2e-inf-a-a88862/test-model-a88862-kserve-554fc8cc49-zl7l2" Nov 03 01:23:57.184539 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:23:57.184495 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7c49ba9e-e354-4e34-be2e-2269aac5e015-tls-certs\") pod \"test-model-a88862-kserve-554fc8cc49-zl7l2\" (UID: \"7c49ba9e-e354-4e34-be2e-2269aac5e015\") " pod="ai-tenant-e2e-inf-a-a88862/test-model-a88862-kserve-554fc8cc49-zl7l2" Nov 03 01:23:57.204930 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:23:57.193225 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-55h2j\" (UniqueName: \"kubernetes.io/projected/7c49ba9e-e354-4e34-be2e-2269aac5e015-kube-api-access-55h2j\") pod \"test-model-a88862-kserve-554fc8cc49-zl7l2\" (UID: \"7c49ba9e-e354-4e34-be2e-2269aac5e015\") " pod="ai-tenant-e2e-inf-a-a88862/test-model-a88862-kserve-554fc8cc49-zl7l2" Nov 03 01:23:57.277964 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:23:57.273343 4939 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-a88862/test-model-a88862-kserve-554fc8cc49-zl7l2" Nov 03 01:23:57.606024 ip-10-0-140-107 kubenswrapper[4939]: W1103 01:23:57.605990 4939 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod7c49ba9e_e354_4e34_be2e_2269aac5e015.slice/crio-99abea7443f59a25158ede0b1eeb6003c70909c110caa8be8bf85d1a5e3bf763 WatchSource:0}: Error finding container 99abea7443f59a25158ede0b1eeb6003c70909c110caa8be8bf85d1a5e3bf763: Status 404 returned error can't find the container with id 99abea7443f59a25158ede0b1eeb6003c70909c110caa8be8bf85d1a5e3bf763 Nov 03 01:23:57.606771 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:23:57.606746 4939 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-a88862/test-model-a88862-kserve-554fc8cc49-zl7l2"] Nov 03 01:23:57.688924 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:23:57.688875 4939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-a88862/test-model-a88862-kserve-554fc8cc49-zl7l2" event={"ID":"7c49ba9e-e354-4e34-be2e-2269aac5e015","Type":"ContainerStarted","Data":"db36987c67ecb311528b67948d63d4b3d62512a1c2a6e98cb1326e16ba240b78"} Nov 03 01:23:57.689057 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:23:57.688941 4939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-a88862/test-model-a88862-kserve-554fc8cc49-zl7l2" event={"ID":"7c49ba9e-e354-4e34-be2e-2269aac5e015","Type":"ContainerStarted","Data":"99abea7443f59a25158ede0b1eeb6003c70909c110caa8be8bf85d1a5e3bf763"} Nov 03 01:23:57.706669 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:23:57.706619 4939 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-inf-a-a88862/test-model-a88862-kserve-554fc8cc49-zl7l2" podStartSLOduration=1.706605164 podStartE2EDuration="1.706605164s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 01:23:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 01:23:57.704871472 +0000 UTC m=+2585.969206417" watchObservedRunningTime="2026-11-03 01:23:57.706605164 +0000 UTC m=+2585.970940108" Nov 03 01:23:57.791236 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:23:57.791189 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:23:58.145704 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:23:58.145674 4939 scope.go:122] "RemoveContainer" containerID="a970713b526550134675994ce29fd24b548b80a5d39b73954a26f0228139d60a" Nov 03 01:23:58.274656 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:23:58.274620 4939 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-a-a88862/test-model-a88862-kserve-554fc8cc49-zl7l2" Nov 03 01:23:58.387472 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:23:58.387439 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:23:59.083695 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:23:59.083657 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:23:59.985672 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:23:59.985638 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:24:00.861521 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:24:00.861476 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:24:01.577318 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:24:01.577284 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:24:02.310904 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:24:02.310869 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:24:02.996423 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:24:02.996389 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:24:03.684202 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:24:03.684167 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:24:04.393414 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:24:04.393372 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:24:05.061044 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:24:05.061016 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:24:05.676797 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:24:05.676759 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:24:06.307088 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:24:06.307054 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:24:06.990649 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:24:06.990601 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:24:07.274480 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:24:07.274403 4939 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-a-a88862/test-model-a88862-kserve-554fc8cc49-zl7l2" Nov 03 01:24:07.287067 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:24:07.287037 4939 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-a-a88862/test-model-a88862-kserve-554fc8cc49-zl7l2" Nov 03 01:24:07.580202 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:24:07.580122 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:24:07.810119 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:24:07.810088 4939 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-a-a88862/test-model-a88862-kserve-554fc8cc49-zl7l2" Nov 03 01:24:08.369946 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:24:08.369868 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:24:08.921734 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:24:08.921696 4939 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-b-ce04af/test-model-ce04af-kserve-84bf7f6d7c-qd56c"] Nov 03 01:24:08.930420 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:24:08.930390 4939 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-ce04af/test-model-ce04af-kserve-84bf7f6d7c-qd56c" Nov 03 01:24:08.934372 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:24:08.934328 4939 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-ce04af\"/\"kube-root-ca.crt\"" Nov 03 01:24:08.948396 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:24:08.948370 4939 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-ce04af\"/\"openshift-service-ca.crt\"" Nov 03 01:24:08.948615 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:24:08.948418 4939 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-ce04af\"/\"test-model-ce04af-kserve-self-signed-certs\"" Nov 03 01:24:08.948615 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:24:08.948475 4939 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-ce04af\"/\"default-dockercfg-zgkn4\"" Nov 03 01:24:08.957297 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:24:08.957274 4939 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-ce04af/test-model-ce04af-kserve-84bf7f6d7c-qd56c"] Nov 03 01:24:08.979108 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:24:08.979074 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:24:09.037538 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:24:09.037506 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/72db0238-20cc-40cb-affb-158746adb87b-dshm\") pod \"test-model-ce04af-kserve-84bf7f6d7c-qd56c\" (UID: \"72db0238-20cc-40cb-affb-158746adb87b\") " pod="ai-tenant-e2e-inf-b-ce04af/test-model-ce04af-kserve-84bf7f6d7c-qd56c" Nov 03 01:24:09.037703 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:24:09.037570 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/72db0238-20cc-40cb-affb-158746adb87b-tmp-dir\") pod \"test-model-ce04af-kserve-84bf7f6d7c-qd56c\" (UID: \"72db0238-20cc-40cb-affb-158746adb87b\") " pod="ai-tenant-e2e-inf-b-ce04af/test-model-ce04af-kserve-84bf7f6d7c-qd56c" Nov 03 01:24:09.037703 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:24:09.037649 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dh5j7\" (UniqueName: \"kubernetes.io/projected/72db0238-20cc-40cb-affb-158746adb87b-kube-api-access-dh5j7\") pod \"test-model-ce04af-kserve-84bf7f6d7c-qd56c\" (UID: \"72db0238-20cc-40cb-affb-158746adb87b\") " pod="ai-tenant-e2e-inf-b-ce04af/test-model-ce04af-kserve-84bf7f6d7c-qd56c" Nov 03 01:24:09.038005 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:24:09.037979 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/72db0238-20cc-40cb-affb-158746adb87b-home\") pod \"test-model-ce04af-kserve-84bf7f6d7c-qd56c\" (UID: \"72db0238-20cc-40cb-affb-158746adb87b\") " pod="ai-tenant-e2e-inf-b-ce04af/test-model-ce04af-kserve-84bf7f6d7c-qd56c" Nov 03 01:24:09.038070 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:24:09.038042 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/72db0238-20cc-40cb-affb-158746adb87b-tls-certs\") pod \"test-model-ce04af-kserve-84bf7f6d7c-qd56c\" (UID: \"72db0238-20cc-40cb-affb-158746adb87b\") " pod="ai-tenant-e2e-inf-b-ce04af/test-model-ce04af-kserve-84bf7f6d7c-qd56c" Nov 03 01:24:09.038187 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:24:09.038167 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/72db0238-20cc-40cb-affb-158746adb87b-model-cache\") pod \"test-model-ce04af-kserve-84bf7f6d7c-qd56c\" (UID: \"72db0238-20cc-40cb-affb-158746adb87b\") " pod="ai-tenant-e2e-inf-b-ce04af/test-model-ce04af-kserve-84bf7f6d7c-qd56c" Nov 03 01:24:09.093705 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:24:09.093674 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:24:09.155760 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:24:09.141117 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/72db0238-20cc-40cb-affb-158746adb87b-dshm\") pod \"test-model-ce04af-kserve-84bf7f6d7c-qd56c\" (UID: \"72db0238-20cc-40cb-affb-158746adb87b\") " pod="ai-tenant-e2e-inf-b-ce04af/test-model-ce04af-kserve-84bf7f6d7c-qd56c" Nov 03 01:24:09.162284 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:24:09.162247 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/72db0238-20cc-40cb-affb-158746adb87b-tmp-dir\") pod \"test-model-ce04af-kserve-84bf7f6d7c-qd56c\" (UID: \"72db0238-20cc-40cb-affb-158746adb87b\") " pod="ai-tenant-e2e-inf-b-ce04af/test-model-ce04af-kserve-84bf7f6d7c-qd56c" Nov 03 01:24:09.172478 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:24:09.172397 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dh5j7\" (UniqueName: \"kubernetes.io/projected/72db0238-20cc-40cb-affb-158746adb87b-kube-api-access-dh5j7\") pod \"test-model-ce04af-kserve-84bf7f6d7c-qd56c\" (UID: \"72db0238-20cc-40cb-affb-158746adb87b\") " pod="ai-tenant-e2e-inf-b-ce04af/test-model-ce04af-kserve-84bf7f6d7c-qd56c" Nov 03 01:24:09.173086 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:24:09.173058 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/72db0238-20cc-40cb-affb-158746adb87b-home\") pod \"test-model-ce04af-kserve-84bf7f6d7c-qd56c\" (UID: \"72db0238-20cc-40cb-affb-158746adb87b\") " pod="ai-tenant-e2e-inf-b-ce04af/test-model-ce04af-kserve-84bf7f6d7c-qd56c" Nov 03 01:24:09.184767 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:24:09.184734 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/72db0238-20cc-40cb-affb-158746adb87b-tls-certs\") pod \"test-model-ce04af-kserve-84bf7f6d7c-qd56c\" (UID: \"72db0238-20cc-40cb-affb-158746adb87b\") " pod="ai-tenant-e2e-inf-b-ce04af/test-model-ce04af-kserve-84bf7f6d7c-qd56c" Nov 03 01:24:09.185448 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:24:09.162837 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/72db0238-20cc-40cb-affb-158746adb87b-tmp-dir\") pod \"test-model-ce04af-kserve-84bf7f6d7c-qd56c\" (UID: \"72db0238-20cc-40cb-affb-158746adb87b\") " pod="ai-tenant-e2e-inf-b-ce04af/test-model-ce04af-kserve-84bf7f6d7c-qd56c" Nov 03 01:24:09.185448 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:24:09.173861 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/72db0238-20cc-40cb-affb-158746adb87b-home\") pod \"test-model-ce04af-kserve-84bf7f6d7c-qd56c\" (UID: \"72db0238-20cc-40cb-affb-158746adb87b\") " pod="ai-tenant-e2e-inf-b-ce04af/test-model-ce04af-kserve-84bf7f6d7c-qd56c" Nov 03 01:24:09.185448 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:24:09.185148 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/72db0238-20cc-40cb-affb-158746adb87b-model-cache\") pod \"test-model-ce04af-kserve-84bf7f6d7c-qd56c\" (UID: \"72db0238-20cc-40cb-affb-158746adb87b\") " pod="ai-tenant-e2e-inf-b-ce04af/test-model-ce04af-kserve-84bf7f6d7c-qd56c" Nov 03 01:24:09.185693 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:24:09.185679 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/72db0238-20cc-40cb-affb-158746adb87b-model-cache\") pod \"test-model-ce04af-kserve-84bf7f6d7c-qd56c\" (UID: \"72db0238-20cc-40cb-affb-158746adb87b\") " pod="ai-tenant-e2e-inf-b-ce04af/test-model-ce04af-kserve-84bf7f6d7c-qd56c" Nov 03 01:24:09.186967 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:24:09.186943 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/72db0238-20cc-40cb-affb-158746adb87b-dshm\") pod \"test-model-ce04af-kserve-84bf7f6d7c-qd56c\" (UID: \"72db0238-20cc-40cb-affb-158746adb87b\") " pod="ai-tenant-e2e-inf-b-ce04af/test-model-ce04af-kserve-84bf7f6d7c-qd56c" Nov 03 01:24:09.203933 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:24:09.196688 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dh5j7\" (UniqueName: \"kubernetes.io/projected/72db0238-20cc-40cb-affb-158746adb87b-kube-api-access-dh5j7\") pod \"test-model-ce04af-kserve-84bf7f6d7c-qd56c\" (UID: \"72db0238-20cc-40cb-affb-158746adb87b\") " pod="ai-tenant-e2e-inf-b-ce04af/test-model-ce04af-kserve-84bf7f6d7c-qd56c" Nov 03 01:24:09.203933 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:24:09.197318 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/72db0238-20cc-40cb-affb-158746adb87b-tls-certs\") pod \"test-model-ce04af-kserve-84bf7f6d7c-qd56c\" (UID: \"72db0238-20cc-40cb-affb-158746adb87b\") " pod="ai-tenant-e2e-inf-b-ce04af/test-model-ce04af-kserve-84bf7f6d7c-qd56c" Nov 03 01:24:09.267962 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:24:09.265395 4939 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-ce04af/test-model-ce04af-kserve-84bf7f6d7c-qd56c" Nov 03 01:24:09.700128 ip-10-0-140-107 kubenswrapper[4939]: W1103 01:24:09.700094 4939 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod72db0238_20cc_40cb_affb_158746adb87b.slice/crio-0101c2b2d2abf22b2524f341f9ce383306cf633723782184e5c02e0af90f4f29 WatchSource:0}: Error finding container 0101c2b2d2abf22b2524f341f9ce383306cf633723782184e5c02e0af90f4f29: Status 404 returned error can't find the container with id 0101c2b2d2abf22b2524f341f9ce383306cf633723782184e5c02e0af90f4f29 Nov 03 01:24:09.700485 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:24:09.700273 4939 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-ce04af/test-model-ce04af-kserve-84bf7f6d7c-qd56c"] Nov 03 01:24:09.819210 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:24:09.819171 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:24:09.822031 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:24:09.822005 4939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-ce04af/test-model-ce04af-kserve-84bf7f6d7c-qd56c" event={"ID":"72db0238-20cc-40cb-affb-158746adb87b","Type":"ContainerStarted","Data":"8c5b07d70b0371fa9339b585cf59fbbf46d5c4332a79766c422536bbdcfd9dc1"} Nov 03 01:24:09.822149 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:24:09.822046 4939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-ce04af/test-model-ce04af-kserve-84bf7f6d7c-qd56c" event={"ID":"72db0238-20cc-40cb-affb-158746adb87b","Type":"ContainerStarted","Data":"0101c2b2d2abf22b2524f341f9ce383306cf633723782184e5c02e0af90f4f29"} Nov 03 01:24:09.890935 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:24:09.884884 4939 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-inf-b-ce04af/test-model-ce04af-kserve-84bf7f6d7c-qd56c" podStartSLOduration=1.884864249 podStartE2EDuration="1.884864249s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 01:24:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 01:24:09.8634697 +0000 UTC m=+2598.127804657" watchObservedRunningTime="2026-11-03 01:24:09.884864249 +0000 UTC m=+2598.149199215" Nov 03 01:24:10.266433 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:24:10.266399 4939 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-b-ce04af/test-model-ce04af-kserve-84bf7f6d7c-qd56c" Nov 03 01:24:10.496782 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:24:10.496748 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:24:11.300081 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:24:11.300048 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:24:12.412693 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:24:12.412631 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:24:13.131870 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:24:13.131837 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:24:13.808126 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:24:13.808092 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:24:13.970928 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:24:13.970861 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:24:14.090791 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:24:14.090743 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:24:15.264291 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:24:15.264231 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:24:15.948982 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:24:15.948898 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:24:16.054277 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:24:16.054238 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:24:16.569694 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:24:16.569663 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:24:17.818198 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:24:17.818163 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:24:18.006897 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:24:18.006854 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:24:18.059774 ip-10-0-140-107 kubenswrapper[4939]: E1103 01:24:18.059736 4939 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-sthjh" containerName="extractor" Nov 03 01:24:18.298961 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:24:18.298924 4939 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f1a0eff5-kpnxp"] Nov 03 01:24:18.307221 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:24:18.307194 4939 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f1a0eff5-kpnxp" Nov 03 01:24:18.336945 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:24:18.329430 4939 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 01:24:18.336945 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:24:18.329871 4939 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-6q4xp\"" Nov 03 01:24:18.468971 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:24:18.457583 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/752af256-3554-41a6-a9bb-e7548419c6b2-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f1a0eff5-kpnxp\" (UID: \"752af256-3554-41a6-a9bb-e7548419c6b2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f1a0eff5-kpnxp" Nov 03 01:24:18.468971 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:24:18.457748 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/752af256-3554-41a6-a9bb-e7548419c6b2-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f1a0eff5-kpnxp\" (UID: \"752af256-3554-41a6-a9bb-e7548419c6b2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f1a0eff5-kpnxp" Nov 03 01:24:18.468971 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:24:18.458032 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mxc9w\" (UniqueName: \"kubernetes.io/projected/752af256-3554-41a6-a9bb-e7548419c6b2-kube-api-access-mxc9w\") pod \"maas-api-revoke-sub-f1a0eff5-kpnxp\" (UID: \"752af256-3554-41a6-a9bb-e7548419c6b2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f1a0eff5-kpnxp" Nov 03 01:24:18.492071 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:24:18.492040 4939 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f1a0eff5-kpnxp"] Nov 03 01:24:18.608089 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:24:18.566752 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/752af256-3554-41a6-a9bb-e7548419c6b2-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f1a0eff5-kpnxp\" (UID: \"752af256-3554-41a6-a9bb-e7548419c6b2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f1a0eff5-kpnxp" Nov 03 01:24:18.652084 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:24:18.652039 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mxc9w\" (UniqueName: \"kubernetes.io/projected/752af256-3554-41a6-a9bb-e7548419c6b2-kube-api-access-mxc9w\") pod \"maas-api-revoke-sub-f1a0eff5-kpnxp\" (UID: \"752af256-3554-41a6-a9bb-e7548419c6b2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f1a0eff5-kpnxp" Nov 03 01:24:18.652628 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:24:18.652596 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/752af256-3554-41a6-a9bb-e7548419c6b2-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f1a0eff5-kpnxp\" (UID: \"752af256-3554-41a6-a9bb-e7548419c6b2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f1a0eff5-kpnxp" Nov 03 01:24:18.654248 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:24:18.572561 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/752af256-3554-41a6-a9bb-e7548419c6b2-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f1a0eff5-kpnxp\" (UID: \"752af256-3554-41a6-a9bb-e7548419c6b2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f1a0eff5-kpnxp" Nov 03 01:24:18.698229 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:24:18.695859 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mxc9w\" (UniqueName: \"kubernetes.io/projected/752af256-3554-41a6-a9bb-e7548419c6b2-kube-api-access-mxc9w\") pod \"maas-api-revoke-sub-f1a0eff5-kpnxp\" (UID: \"752af256-3554-41a6-a9bb-e7548419c6b2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f1a0eff5-kpnxp" Nov 03 01:24:18.699220 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:24:18.699189 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/752af256-3554-41a6-a9bb-e7548419c6b2-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f1a0eff5-kpnxp\" (UID: \"752af256-3554-41a6-a9bb-e7548419c6b2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f1a0eff5-kpnxp" Nov 03 01:24:18.721652 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:24:18.721594 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:24:18.743003 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:24:18.741358 4939 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f1a0eff5-kpnxp" Nov 03 01:24:19.148123 ip-10-0-140-107 kubenswrapper[4939]: W1103 01:24:19.148054 4939 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod752af256_3554_41a6_a9bb_e7548419c6b2.slice/crio-2e8cd15d814475db0d2a9a694591880c8d102cbfba587d21bbcaf678dbf4c6e9 WatchSource:0}: Error finding container 2e8cd15d814475db0d2a9a694591880c8d102cbfba587d21bbcaf678dbf4c6e9: Status 404 returned error can't find the container with id 2e8cd15d814475db0d2a9a694591880c8d102cbfba587d21bbcaf678dbf4c6e9 Nov 03 01:24:19.148574 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:24:19.148412 4939 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f1a0eff5-kpnxp"] Nov 03 01:24:19.266398 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:24:19.266370 4939 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-b-ce04af/test-model-ce04af-kserve-84bf7f6d7c-qd56c" Nov 03 01:24:19.280839 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:24:19.280814 4939 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-b-ce04af/test-model-ce04af-kserve-84bf7f6d7c-qd56c" Nov 03 01:24:19.491698 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:24:19.491617 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:24:20.060275 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:24:20.060235 4939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f1a0eff5-kpnxp" event={"ID":"752af256-3554-41a6-a9bb-e7548419c6b2","Type":"ContainerStarted","Data":"e0f50c3298f7133373574a098c05e9f525bd661943401f2552fd9bd72a95a234"} Nov 03 01:24:20.060462 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:24:20.060304 4939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f1a0eff5-kpnxp" event={"ID":"752af256-3554-41a6-a9bb-e7548419c6b2","Type":"ContainerStarted","Data":"2e8cd15d814475db0d2a9a694591880c8d102cbfba587d21bbcaf678dbf4c6e9"} Nov 03 01:24:20.072115 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:24:20.072089 4939 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-b-ce04af/test-model-ce04af-kserve-84bf7f6d7c-qd56c" Nov 03 01:24:20.078568 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:24:20.078524 4939 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f1a0eff5-kpnxp" podStartSLOduration=2.078510945 podStartE2EDuration="2.078510945s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 01:24:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 01:24:20.075630571 +0000 UTC m=+2608.339965515" watchObservedRunningTime="2026-11-03 01:24:20.078510945 +0000 UTC m=+2608.342845889" Nov 03 01:24:20.120426 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:24:20.120396 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:24:20.793855 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:24:20.793821 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:24:21.071218 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:24:21.071130 4939 generic.go:356] "Generic (PLEG): container finished" podID="752af256-3554-41a6-a9bb-e7548419c6b2" containerID="e0f50c3298f7133373574a098c05e9f525bd661943401f2552fd9bd72a95a234" exitCode=0 Nov 03 01:24:21.071380 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:24:21.071215 4939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f1a0eff5-kpnxp" event={"ID":"752af256-3554-41a6-a9bb-e7548419c6b2","Type":"ContainerDied","Data":"e0f50c3298f7133373574a098c05e9f525bd661943401f2552fd9bd72a95a234"} Nov 03 01:24:21.384129 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:24:21.384098 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:24:22.086091 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:24:22.086058 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:24:22.343646 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:24:22.343623 4939 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f1a0eff5-kpnxp" Nov 03 01:24:22.423616 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:24:22.423578 4939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/752af256-3554-41a6-a9bb-e7548419c6b2-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/752af256-3554-41a6-a9bb-e7548419c6b2-maas-api-xks-ca\") pod \"752af256-3554-41a6-a9bb-e7548419c6b2\" (UID: \"752af256-3554-41a6-a9bb-e7548419c6b2\") " Nov 03 01:24:22.423870 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:24:22.423709 4939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/752af256-3554-41a6-a9bb-e7548419c6b2-kube-api-access-mxc9w\" (UniqueName: \"kubernetes.io/projected/752af256-3554-41a6-a9bb-e7548419c6b2-kube-api-access-mxc9w\") pod \"752af256-3554-41a6-a9bb-e7548419c6b2\" (UID: \"752af256-3554-41a6-a9bb-e7548419c6b2\") " Nov 03 01:24:22.424063 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:24:22.424034 4939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/752af256-3554-41a6-a9bb-e7548419c6b2-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/752af256-3554-41a6-a9bb-e7548419c6b2-maas-api-openshift-ca\") pod \"752af256-3554-41a6-a9bb-e7548419c6b2\" (UID: \"752af256-3554-41a6-a9bb-e7548419c6b2\") " Nov 03 01:24:22.424421 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:24:22.424396 4939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/752af256-3554-41a6-a9bb-e7548419c6b2-maas-api-openshift-ca" pod "752af256-3554-41a6-a9bb-e7548419c6b2" (UID: "752af256-3554-41a6-a9bb-e7548419c6b2"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 01:24:22.424895 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:24:22.424877 4939 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/752af256-3554-41a6-a9bb-e7548419c6b2-maas-api-openshift-ca\") on node \"ip-10-0-140-107.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:24:22.425945 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:24:22.425894 4939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/752af256-3554-41a6-a9bb-e7548419c6b2-maas-api-xks-ca" pod "752af256-3554-41a6-a9bb-e7548419c6b2" (UID: "752af256-3554-41a6-a9bb-e7548419c6b2"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 01:24:22.426133 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:24:22.426110 4939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/752af256-3554-41a6-a9bb-e7548419c6b2-kube-api-access-mxc9w" pod "752af256-3554-41a6-a9bb-e7548419c6b2" (UID: "752af256-3554-41a6-a9bb-e7548419c6b2"). InnerVolumeSpecName "kube-api-access-mxc9w". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 01:24:22.527289 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:24:22.527251 4939 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/752af256-3554-41a6-a9bb-e7548419c6b2-maas-api-xks-ca\") on node \"ip-10-0-140-107.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:24:22.527289 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:24:22.527287 4939 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mxc9w\" (UniqueName: \"kubernetes.io/projected/752af256-3554-41a6-a9bb-e7548419c6b2-kube-api-access-mxc9w\") on node \"ip-10-0-140-107.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:24:22.799685 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:24:22.799541 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:24:23.090455 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:24:23.090417 4939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f1a0eff5-kpnxp" event={"ID":"752af256-3554-41a6-a9bb-e7548419c6b2","Type":"ContainerDied","Data":"2e8cd15d814475db0d2a9a694591880c8d102cbfba587d21bbcaf678dbf4c6e9"} Nov 03 01:24:23.090964 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:24:23.090481 4939 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="2e8cd15d814475db0d2a9a694591880c8d102cbfba587d21bbcaf678dbf4c6e9" Nov 03 01:24:23.090964 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:24:23.090440 4939 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f1a0eff5-kpnxp" Nov 03 01:24:23.611258 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:24:23.611224 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:24:24.283243 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:24:24.283208 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:24:24.992135 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:24:24.992102 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:24:25.172956 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:24:25.172898 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:24:25.198749 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:24:25.198711 4939 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-af8244/payload-processing-external-model-e2e-extmod-b-af8244-6cbfbtf5b"] Nov 03 01:24:25.292318 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:24:25.292232 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:24:26.399529 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:24:26.399498 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:24:27.071231 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:24:27.071178 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:24:27.685472 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:24:27.685439 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:24:28.312272 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:24:28.312220 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:24:28.888653 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:24:28.888619 4939 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7df33a0c-mdc5q"] Nov 03 01:24:28.889667 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:24:28.889649 4939 cpu_manager.go:414] "containerMap: removing container" podUID="752af256-3554-41a6-a9bb-e7548419c6b2" containerName="revoke-keys" Nov 03 01:24:28.889667 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:24:28.889668 4939 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="752af256-3554-41a6-a9bb-e7548419c6b2" containerName="revoke-keys" Nov 03 01:24:28.889838 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:24:28.889828 4939 memory_manager.go:365] "RemoveStaleState removing state" podUID="752af256-3554-41a6-a9bb-e7548419c6b2" containerName="revoke-keys" Nov 03 01:24:28.894979 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:24:28.894956 4939 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7df33a0c-mdc5q" Nov 03 01:24:28.899207 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:24:28.898926 4939 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 01:24:28.899832 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:24:28.899809 4939 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-6q4xp\"" Nov 03 01:24:28.900501 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:24:28.900482 4939 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7df33a0c-mdc5q"] Nov 03 01:24:29.021281 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:24:29.021239 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z8gjb\" (UniqueName: \"kubernetes.io/projected/59795f2c-592e-4820-892a-bfdd35951594-kube-api-access-z8gjb\") pod \"maas-api-revoke-sub-7df33a0c-mdc5q\" (UID: \"59795f2c-592e-4820-892a-bfdd35951594\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7df33a0c-mdc5q" Nov 03 01:24:29.021468 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:24:29.021408 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/59795f2c-592e-4820-892a-bfdd35951594-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7df33a0c-mdc5q\" (UID: \"59795f2c-592e-4820-892a-bfdd35951594\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7df33a0c-mdc5q" Nov 03 01:24:29.021878 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:24:29.021854 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/59795f2c-592e-4820-892a-bfdd35951594-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7df33a0c-mdc5q\" (UID: \"59795f2c-592e-4820-892a-bfdd35951594\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7df33a0c-mdc5q" Nov 03 01:24:29.066992 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:24:29.066953 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:24:29.127931 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:24:29.126638 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/59795f2c-592e-4820-892a-bfdd35951594-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7df33a0c-mdc5q\" (UID: \"59795f2c-592e-4820-892a-bfdd35951594\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7df33a0c-mdc5q" Nov 03 01:24:29.127931 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:24:29.126707 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z8gjb\" (UniqueName: \"kubernetes.io/projected/59795f2c-592e-4820-892a-bfdd35951594-kube-api-access-z8gjb\") pod \"maas-api-revoke-sub-7df33a0c-mdc5q\" (UID: \"59795f2c-592e-4820-892a-bfdd35951594\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7df33a0c-mdc5q" Nov 03 01:24:29.163135 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:24:29.163025 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/59795f2c-592e-4820-892a-bfdd35951594-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7df33a0c-mdc5q\" (UID: \"59795f2c-592e-4820-892a-bfdd35951594\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7df33a0c-mdc5q" Nov 03 01:24:29.172039 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:24:29.164483 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/59795f2c-592e-4820-892a-bfdd35951594-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7df33a0c-mdc5q\" (UID: \"59795f2c-592e-4820-892a-bfdd35951594\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7df33a0c-mdc5q" Nov 03 01:24:29.186294 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:24:29.186260 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/59795f2c-592e-4820-892a-bfdd35951594-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7df33a0c-mdc5q\" (UID: \"59795f2c-592e-4820-892a-bfdd35951594\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7df33a0c-mdc5q" Nov 03 01:24:29.192392 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:24:29.192355 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-z8gjb\" (UniqueName: \"kubernetes.io/projected/59795f2c-592e-4820-892a-bfdd35951594-kube-api-access-z8gjb\") pod \"maas-api-revoke-sub-7df33a0c-mdc5q\" (UID: \"59795f2c-592e-4820-892a-bfdd35951594\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7df33a0c-mdc5q" Nov 03 01:24:29.238328 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:24:29.238302 4939 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7df33a0c-mdc5q" Nov 03 01:24:29.329492 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:24:29.329404 4939 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-7c759a/payload-processing-external-model-e2e-extmod-a-7c759a-799fk2rjw"] Nov 03 01:24:29.551948 ip-10-0-140-107 kubenswrapper[4939]: W1103 01:24:29.551894 4939 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod59795f2c_592e_4820_892a_bfdd35951594.slice/crio-01ec8a4513efebc24ef2a75044ea9f11e2705e68cbaff1e9f398e4eb60078cbd WatchSource:0}: Error finding container 01ec8a4513efebc24ef2a75044ea9f11e2705e68cbaff1e9f398e4eb60078cbd: Status 404 returned error can't find the container with id 01ec8a4513efebc24ef2a75044ea9f11e2705e68cbaff1e9f398e4eb60078cbd Nov 03 01:24:29.552088 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:24:29.552061 4939 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7df33a0c-mdc5q"] Nov 03 01:24:29.801637 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:24:29.801547 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:24:30.200131 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:24:30.200082 4939 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-b-af8244/payload-processing-external-model-e2e-extmod-b-af8244-6cbfbtf5b" podUID="99099fe7-73ed-4732-9a33-2df9bed89665" containerName="payload-processing" containerID="cri-o://6da8e13a4665c438db1ed016e08ce569161e7e6a734916ac931e20cf895b3ba1" gracePeriod=25 Nov 03 01:24:30.228177 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:24:30.228138 4939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7df33a0c-mdc5q" event={"ID":"59795f2c-592e-4820-892a-bfdd35951594","Type":"ContainerStarted","Data":"e4cd9b1a7a872bc58bdabe9a9f2cbed304f5735100060d680085955560752c69"} Nov 03 01:24:30.228322 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:24:30.228202 4939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7df33a0c-mdc5q" event={"ID":"59795f2c-592e-4820-892a-bfdd35951594","Type":"ContainerStarted","Data":"01ec8a4513efebc24ef2a75044ea9f11e2705e68cbaff1e9f398e4eb60078cbd"} Nov 03 01:24:30.245955 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:24:30.245882 4939 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7df33a0c-mdc5q" podStartSLOduration=2.245867965 podStartE2EDuration="2.245867965s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 01:24:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 01:24:30.244552802 +0000 UTC m=+2618.508887747" watchObservedRunningTime="2026-11-03 01:24:30.245867965 +0000 UTC m=+2618.510202909" Nov 03 01:24:30.408160 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:24:30.408080 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:24:30.609134 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:24:30.609107 4939 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-af8244/payload-processing-external-model-e2e-extmod-b-af8244-6cbfbtf5b" Nov 03 01:24:30.617231 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:24:30.617203 4939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/99099fe7-73ed-4732-9a33-2df9bed89665-kube-api-access-8jxcc\" (UniqueName: \"kubernetes.io/projected/99099fe7-73ed-4732-9a33-2df9bed89665-kube-api-access-8jxcc\") pod \"99099fe7-73ed-4732-9a33-2df9bed89665\" (UID: \"99099fe7-73ed-4732-9a33-2df9bed89665\") " Nov 03 01:24:30.617376 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:24:30.617291 4939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/99099fe7-73ed-4732-9a33-2df9bed89665-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/99099fe7-73ed-4732-9a33-2df9bed89665-plugins-config-volume\") pod \"99099fe7-73ed-4732-9a33-2df9bed89665\" (UID: \"99099fe7-73ed-4732-9a33-2df9bed89665\") " Nov 03 01:24:30.617376 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:24:30.617370 4939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/99099fe7-73ed-4732-9a33-2df9bed89665-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/99099fe7-73ed-4732-9a33-2df9bed89665-routing-overlay-volume\") pod \"99099fe7-73ed-4732-9a33-2df9bed89665\" (UID: \"99099fe7-73ed-4732-9a33-2df9bed89665\") " Nov 03 01:24:30.617507 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:24:30.617493 4939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/99099fe7-73ed-4732-9a33-2df9bed89665-provider-credentials\" (UniqueName: \"kubernetes.io/projected/99099fe7-73ed-4732-9a33-2df9bed89665-provider-credentials\") pod \"99099fe7-73ed-4732-9a33-2df9bed89665\" (UID: \"99099fe7-73ed-4732-9a33-2df9bed89665\") " Nov 03 01:24:30.617722 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:24:30.617696 4939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/99099fe7-73ed-4732-9a33-2df9bed89665-plugins-config-volume" pod "99099fe7-73ed-4732-9a33-2df9bed89665" (UID: "99099fe7-73ed-4732-9a33-2df9bed89665"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 01:24:30.617849 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:24:30.617716 4939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/99099fe7-73ed-4732-9a33-2df9bed89665-routing-overlay-volume" pod "99099fe7-73ed-4732-9a33-2df9bed89665" (UID: "99099fe7-73ed-4732-9a33-2df9bed89665"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 01:24:30.618303 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:24:30.618284 4939 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/99099fe7-73ed-4732-9a33-2df9bed89665-plugins-config-volume\") on node \"ip-10-0-140-107.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:24:30.618372 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:24:30.618306 4939 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/99099fe7-73ed-4732-9a33-2df9bed89665-routing-overlay-volume\") on node \"ip-10-0-140-107.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:24:30.619694 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:24:30.619667 4939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/99099fe7-73ed-4732-9a33-2df9bed89665-kube-api-access-8jxcc" pod "99099fe7-73ed-4732-9a33-2df9bed89665" (UID: "99099fe7-73ed-4732-9a33-2df9bed89665"). InnerVolumeSpecName "kube-api-access-8jxcc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 01:24:30.619787 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:24:30.619715 4939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/99099fe7-73ed-4732-9a33-2df9bed89665-provider-credentials" pod "99099fe7-73ed-4732-9a33-2df9bed89665" (UID: "99099fe7-73ed-4732-9a33-2df9bed89665"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 01:24:30.720888 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:24:30.720849 4939 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8jxcc\" (UniqueName: \"kubernetes.io/projected/99099fe7-73ed-4732-9a33-2df9bed89665-kube-api-access-8jxcc\") on node \"ip-10-0-140-107.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:24:30.720888 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:24:30.720892 4939 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/99099fe7-73ed-4732-9a33-2df9bed89665-provider-credentials\") on node \"ip-10-0-140-107.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:24:31.097035 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:24:31.097001 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:24:31.239625 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:24:31.239591 4939 generic.go:356] "Generic (PLEG): container finished" podID="99099fe7-73ed-4732-9a33-2df9bed89665" containerID="6da8e13a4665c438db1ed016e08ce569161e7e6a734916ac931e20cf895b3ba1" exitCode=0 Nov 03 01:24:31.240198 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:24:31.239637 4939 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-af8244/payload-processing-external-model-e2e-extmod-b-af8244-6cbfbtf5b" Nov 03 01:24:31.240198 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:24:31.239671 4939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-af8244/payload-processing-external-model-e2e-extmod-b-af8244-6cbfbtf5b" event={"ID":"99099fe7-73ed-4732-9a33-2df9bed89665","Type":"ContainerDied","Data":"6da8e13a4665c438db1ed016e08ce569161e7e6a734916ac931e20cf895b3ba1"} Nov 03 01:24:31.240198 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:24:31.239732 4939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-af8244/payload-processing-external-model-e2e-extmod-b-af8244-6cbfbtf5b" event={"ID":"99099fe7-73ed-4732-9a33-2df9bed89665","Type":"ContainerDied","Data":"e114aaf2fedefbfefe5c1c99dfe6b6e7965edfdbcefc2ac23dced39afaaaac11"} Nov 03 01:24:31.240198 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:24:31.239764 4939 scope.go:122] "RemoveContainer" containerID="6da8e13a4665c438db1ed016e08ce569161e7e6a734916ac931e20cf895b3ba1" Nov 03 01:24:31.251990 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:24:31.251965 4939 scope.go:122] "RemoveContainer" containerID="6da8e13a4665c438db1ed016e08ce569161e7e6a734916ac931e20cf895b3ba1" Nov 03 01:24:31.252519 ip-10-0-140-107 kubenswrapper[4939]: E1103 01:24:31.252497 4939 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6da8e13a4665c438db1ed016e08ce569161e7e6a734916ac931e20cf895b3ba1\": container with ID starting with 6da8e13a4665c438db1ed016e08ce569161e7e6a734916ac931e20cf895b3ba1 not found: ID does not exist" containerID="6da8e13a4665c438db1ed016e08ce569161e7e6a734916ac931e20cf895b3ba1" Nov 03 01:24:31.252585 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:24:31.252535 4939 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6da8e13a4665c438db1ed016e08ce569161e7e6a734916ac931e20cf895b3ba1"} err="failed to get container status \"6da8e13a4665c438db1ed016e08ce569161e7e6a734916ac931e20cf895b3ba1\": rpc error: code = NotFound desc = could not find container \"6da8e13a4665c438db1ed016e08ce569161e7e6a734916ac931e20cf895b3ba1\": container with ID starting with 6da8e13a4665c438db1ed016e08ce569161e7e6a734916ac931e20cf895b3ba1 not found: ID does not exist" Nov 03 01:24:31.266711 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:24:31.266686 4939 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-af8244/payload-processing-external-model-e2e-extmod-b-af8244-6cbfbtf5b"] Nov 03 01:24:31.271618 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:24:31.271596 4939 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-b-af8244/payload-processing-external-model-e2e-extmod-b-af8244-6cbfbtf5b"] Nov 03 01:24:31.716207 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:24:31.716176 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:24:31.896298 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:24:31.896261 4939 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="99099fe7-73ed-4732-9a33-2df9bed89665" path="/var/lib/kubelet/pods/99099fe7-73ed-4732-9a33-2df9bed89665/volumes" Nov 03 01:24:32.371984 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:24:32.371951 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:24:33.091692 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:24:33.091655 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:24:33.758940 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:24:33.758830 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:24:34.269202 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:24:34.269167 4939 generic.go:356] "Generic (PLEG): container finished" podID="59795f2c-592e-4820-892a-bfdd35951594" containerID="e4cd9b1a7a872bc58bdabe9a9f2cbed304f5735100060d680085955560752c69" exitCode=0 Nov 03 01:24:34.269374 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:24:34.269217 4939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7df33a0c-mdc5q" event={"ID":"59795f2c-592e-4820-892a-bfdd35951594","Type":"ContainerDied","Data":"e4cd9b1a7a872bc58bdabe9a9f2cbed304f5735100060d680085955560752c69"} Nov 03 01:24:34.330237 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:24:34.330191 4939 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-a-7c759a/payload-processing-external-model-e2e-extmod-a-7c759a-799fk2rjw" podUID="d239ed01-45c1-4f47-b35d-50d2b437e179" containerName="payload-processing" containerID="cri-o://d6c4261bd8ff736e5159aac1a56b5a5b08845f5bc34688fde2ef31fdf9b37939" gracePeriod=25 Nov 03 01:24:34.375567 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:24:34.375537 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:24:34.693522 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:24:34.693486 4939 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-7c759a/payload-processing-external-model-e2e-extmod-a-7c759a-799fk2rjw" Nov 03 01:24:34.725347 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:24:34.725311 4939 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-b-ce04af-openshift-default-7dccd45f47-vxtn5"] Nov 03 01:24:34.726428 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:24:34.726400 4939 cpu_manager.go:414] "containerMap: removing container" podUID="99099fe7-73ed-4732-9a33-2df9bed89665" containerName="payload-processing" Nov 03 01:24:34.726428 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:24:34.726428 4939 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="99099fe7-73ed-4732-9a33-2df9bed89665" containerName="payload-processing" Nov 03 01:24:34.726638 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:24:34.726511 4939 cpu_manager.go:414] "containerMap: removing container" podUID="d239ed01-45c1-4f47-b35d-50d2b437e179" containerName="payload-processing" Nov 03 01:24:34.726638 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:24:34.726520 4939 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d239ed01-45c1-4f47-b35d-50d2b437e179" containerName="payload-processing" Nov 03 01:24:34.726806 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:24:34.726789 4939 memory_manager.go:365] "RemoveStaleState removing state" podUID="d239ed01-45c1-4f47-b35d-50d2b437e179" containerName="payload-processing" Nov 03 01:24:34.726869 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:24:34.726818 4939 memory_manager.go:365] "RemoveStaleState removing state" podUID="99099fe7-73ed-4732-9a33-2df9bed89665" containerName="payload-processing" Nov 03 01:24:34.730976 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:24:34.730954 4939 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-ce04af-openshift-default-7dccd45f47-vxtn5" Nov 03 01:24:34.743847 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:24:34.740108 4939 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-b-ce04af-openshift-default-dockercfg-rdq4s\"" Nov 03 01:24:34.743847 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:24:34.740538 4939 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-ce04af-openshift-default-7dccd45f47-vxtn5"] Nov 03 01:24:34.844604 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:24:34.844570 4939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d239ed01-45c1-4f47-b35d-50d2b437e179-provider-credentials\" (UniqueName: \"kubernetes.io/projected/d239ed01-45c1-4f47-b35d-50d2b437e179-provider-credentials\") pod \"d239ed01-45c1-4f47-b35d-50d2b437e179\" (UID: \"d239ed01-45c1-4f47-b35d-50d2b437e179\") " Nov 03 01:24:34.845196 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:24:34.844664 4939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d239ed01-45c1-4f47-b35d-50d2b437e179-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d239ed01-45c1-4f47-b35d-50d2b437e179-plugins-config-volume\") pod \"d239ed01-45c1-4f47-b35d-50d2b437e179\" (UID: \"d239ed01-45c1-4f47-b35d-50d2b437e179\") " Nov 03 01:24:34.845196 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:24:34.844697 4939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d239ed01-45c1-4f47-b35d-50d2b437e179-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/d239ed01-45c1-4f47-b35d-50d2b437e179-routing-overlay-volume\") pod \"d239ed01-45c1-4f47-b35d-50d2b437e179\" (UID: \"d239ed01-45c1-4f47-b35d-50d2b437e179\") " Nov 03 01:24:34.845196 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:24:34.845081 4939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d239ed01-45c1-4f47-b35d-50d2b437e179-kube-api-access-cclzn\" (UniqueName: \"kubernetes.io/projected/d239ed01-45c1-4f47-b35d-50d2b437e179-kube-api-access-cclzn\") pod \"d239ed01-45c1-4f47-b35d-50d2b437e179\" (UID: \"d239ed01-45c1-4f47-b35d-50d2b437e179\") " Nov 03 01:24:34.845403 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:24:34.845210 4939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d239ed01-45c1-4f47-b35d-50d2b437e179-routing-overlay-volume" pod "d239ed01-45c1-4f47-b35d-50d2b437e179" (UID: "d239ed01-45c1-4f47-b35d-50d2b437e179"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 01:24:34.845403 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:24:34.845219 4939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d239ed01-45c1-4f47-b35d-50d2b437e179-plugins-config-volume" pod "d239ed01-45c1-4f47-b35d-50d2b437e179" (UID: "d239ed01-45c1-4f47-b35d-50d2b437e179"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 01:24:34.845968 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:24:34.845934 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2c2192ff-91c4-4488-8229-629018fd881d-workload-socket\") pod \"e2e-inf-b-ce04af-openshift-default-7dccd45f47-vxtn5\" (UID: \"2c2192ff-91c4-4488-8229-629018fd881d\") " pod="openshift-ingress/e2e-inf-b-ce04af-openshift-default-7dccd45f47-vxtn5" Nov 03 01:24:34.846120 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:24:34.846036 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2c2192ff-91c4-4488-8229-629018fd881d-istio-token\") pod \"e2e-inf-b-ce04af-openshift-default-7dccd45f47-vxtn5\" (UID: \"2c2192ff-91c4-4488-8229-629018fd881d\") " pod="openshift-ingress/e2e-inf-b-ce04af-openshift-default-7dccd45f47-vxtn5" Nov 03 01:24:34.846269 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:24:34.846245 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2c2192ff-91c4-4488-8229-629018fd881d-workload-certs\") pod \"e2e-inf-b-ce04af-openshift-default-7dccd45f47-vxtn5\" (UID: \"2c2192ff-91c4-4488-8229-629018fd881d\") " pod="openshift-ingress/e2e-inf-b-ce04af-openshift-default-7dccd45f47-vxtn5" Nov 03 01:24:34.846395 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:24:34.846354 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2c2192ff-91c4-4488-8229-629018fd881d-istio-data\") pod \"e2e-inf-b-ce04af-openshift-default-7dccd45f47-vxtn5\" (UID: \"2c2192ff-91c4-4488-8229-629018fd881d\") " pod="openshift-ingress/e2e-inf-b-ce04af-openshift-default-7dccd45f47-vxtn5" Nov 03 01:24:34.846468 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:24:34.846408 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2c2192ff-91c4-4488-8229-629018fd881d-istiod-ca-cert\") pod \"e2e-inf-b-ce04af-openshift-default-7dccd45f47-vxtn5\" (UID: \"2c2192ff-91c4-4488-8229-629018fd881d\") " pod="openshift-ingress/e2e-inf-b-ce04af-openshift-default-7dccd45f47-vxtn5" Nov 03 01:24:34.846537 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:24:34.846482 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2c2192ff-91c4-4488-8229-629018fd881d-istio-podinfo\") pod \"e2e-inf-b-ce04af-openshift-default-7dccd45f47-vxtn5\" (UID: \"2c2192ff-91c4-4488-8229-629018fd881d\") " pod="openshift-ingress/e2e-inf-b-ce04af-openshift-default-7dccd45f47-vxtn5" Nov 03 01:24:34.846537 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:24:34.846526 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jh5rp\" (UniqueName: \"kubernetes.io/projected/2c2192ff-91c4-4488-8229-629018fd881d-kube-api-access-jh5rp\") pod \"e2e-inf-b-ce04af-openshift-default-7dccd45f47-vxtn5\" (UID: \"2c2192ff-91c4-4488-8229-629018fd881d\") " pod="openshift-ingress/e2e-inf-b-ce04af-openshift-default-7dccd45f47-vxtn5" Nov 03 01:24:34.846656 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:24:34.846612 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2c2192ff-91c4-4488-8229-629018fd881d-istio-envoy\") pod \"e2e-inf-b-ce04af-openshift-default-7dccd45f47-vxtn5\" (UID: \"2c2192ff-91c4-4488-8229-629018fd881d\") " pod="openshift-ingress/e2e-inf-b-ce04af-openshift-default-7dccd45f47-vxtn5" Nov 03 01:24:34.846715 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:24:34.846667 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2c2192ff-91c4-4488-8229-629018fd881d-credential-socket\") pod \"e2e-inf-b-ce04af-openshift-default-7dccd45f47-vxtn5\" (UID: \"2c2192ff-91c4-4488-8229-629018fd881d\") " pod="openshift-ingress/e2e-inf-b-ce04af-openshift-default-7dccd45f47-vxtn5" Nov 03 01:24:34.846832 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:24:34.846817 4939 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d239ed01-45c1-4f47-b35d-50d2b437e179-plugins-config-volume\") on node \"ip-10-0-140-107.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:24:34.847060 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:24:34.846832 4939 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/d239ed01-45c1-4f47-b35d-50d2b437e179-routing-overlay-volume\") on node \"ip-10-0-140-107.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:24:34.848428 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:24:34.848400 4939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d239ed01-45c1-4f47-b35d-50d2b437e179-provider-credentials" pod "d239ed01-45c1-4f47-b35d-50d2b437e179" (UID: "d239ed01-45c1-4f47-b35d-50d2b437e179"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 01:24:34.848542 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:24:34.848442 4939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d239ed01-45c1-4f47-b35d-50d2b437e179-kube-api-access-cclzn" pod "d239ed01-45c1-4f47-b35d-50d2b437e179" (UID: "d239ed01-45c1-4f47-b35d-50d2b437e179"). InnerVolumeSpecName "kube-api-access-cclzn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 01:24:34.948085 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:24:34.948045 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2c2192ff-91c4-4488-8229-629018fd881d-workload-certs\") pod \"e2e-inf-b-ce04af-openshift-default-7dccd45f47-vxtn5\" (UID: \"2c2192ff-91c4-4488-8229-629018fd881d\") " pod="openshift-ingress/e2e-inf-b-ce04af-openshift-default-7dccd45f47-vxtn5" Nov 03 01:24:34.948290 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:24:34.948222 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2c2192ff-91c4-4488-8229-629018fd881d-istio-data\") pod \"e2e-inf-b-ce04af-openshift-default-7dccd45f47-vxtn5\" (UID: \"2c2192ff-91c4-4488-8229-629018fd881d\") " pod="openshift-ingress/e2e-inf-b-ce04af-openshift-default-7dccd45f47-vxtn5" Nov 03 01:24:34.948372 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:24:34.948297 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2c2192ff-91c4-4488-8229-629018fd881d-istiod-ca-cert\") pod \"e2e-inf-b-ce04af-openshift-default-7dccd45f47-vxtn5\" (UID: \"2c2192ff-91c4-4488-8229-629018fd881d\") " pod="openshift-ingress/e2e-inf-b-ce04af-openshift-default-7dccd45f47-vxtn5" Nov 03 01:24:34.948372 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:24:34.948364 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2c2192ff-91c4-4488-8229-629018fd881d-istio-podinfo\") pod \"e2e-inf-b-ce04af-openshift-default-7dccd45f47-vxtn5\" (UID: \"2c2192ff-91c4-4488-8229-629018fd881d\") " pod="openshift-ingress/e2e-inf-b-ce04af-openshift-default-7dccd45f47-vxtn5" Nov 03 01:24:34.948507 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:24:34.948394 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jh5rp\" (UniqueName: \"kubernetes.io/projected/2c2192ff-91c4-4488-8229-629018fd881d-kube-api-access-jh5rp\") pod \"e2e-inf-b-ce04af-openshift-default-7dccd45f47-vxtn5\" (UID: \"2c2192ff-91c4-4488-8229-629018fd881d\") " pod="openshift-ingress/e2e-inf-b-ce04af-openshift-default-7dccd45f47-vxtn5" Nov 03 01:24:34.948577 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:24:34.948530 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2c2192ff-91c4-4488-8229-629018fd881d-istio-envoy\") pod \"e2e-inf-b-ce04af-openshift-default-7dccd45f47-vxtn5\" (UID: \"2c2192ff-91c4-4488-8229-629018fd881d\") " pod="openshift-ingress/e2e-inf-b-ce04af-openshift-default-7dccd45f47-vxtn5" Nov 03 01:24:34.948633 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:24:34.948580 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2c2192ff-91c4-4488-8229-629018fd881d-credential-socket\") pod \"e2e-inf-b-ce04af-openshift-default-7dccd45f47-vxtn5\" (UID: \"2c2192ff-91c4-4488-8229-629018fd881d\") " pod="openshift-ingress/e2e-inf-b-ce04af-openshift-default-7dccd45f47-vxtn5" Nov 03 01:24:34.948633 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:24:34.948586 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2c2192ff-91c4-4488-8229-629018fd881d-workload-certs\") pod \"e2e-inf-b-ce04af-openshift-default-7dccd45f47-vxtn5\" (UID: \"2c2192ff-91c4-4488-8229-629018fd881d\") " pod="openshift-ingress/e2e-inf-b-ce04af-openshift-default-7dccd45f47-vxtn5" Nov 03 01:24:34.948737 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:24:34.948645 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2c2192ff-91c4-4488-8229-629018fd881d-istio-data\") pod \"e2e-inf-b-ce04af-openshift-default-7dccd45f47-vxtn5\" (UID: \"2c2192ff-91c4-4488-8229-629018fd881d\") " pod="openshift-ingress/e2e-inf-b-ce04af-openshift-default-7dccd45f47-vxtn5" Nov 03 01:24:34.948737 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:24:34.948710 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2c2192ff-91c4-4488-8229-629018fd881d-workload-socket\") pod \"e2e-inf-b-ce04af-openshift-default-7dccd45f47-vxtn5\" (UID: \"2c2192ff-91c4-4488-8229-629018fd881d\") " pod="openshift-ingress/e2e-inf-b-ce04af-openshift-default-7dccd45f47-vxtn5" Nov 03 01:24:34.948850 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:24:34.948761 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2c2192ff-91c4-4488-8229-629018fd881d-istio-token\") pod \"e2e-inf-b-ce04af-openshift-default-7dccd45f47-vxtn5\" (UID: \"2c2192ff-91c4-4488-8229-629018fd881d\") " pod="openshift-ingress/e2e-inf-b-ce04af-openshift-default-7dccd45f47-vxtn5" Nov 03 01:24:34.949265 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:24:34.948895 4939 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/d239ed01-45c1-4f47-b35d-50d2b437e179-provider-credentials\") on node \"ip-10-0-140-107.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:24:34.949265 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:24:34.948932 4939 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cclzn\" (UniqueName: \"kubernetes.io/projected/d239ed01-45c1-4f47-b35d-50d2b437e179-kube-api-access-cclzn\") on node \"ip-10-0-140-107.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:24:34.949265 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:24:34.949128 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2c2192ff-91c4-4488-8229-629018fd881d-credential-socket\") pod \"e2e-inf-b-ce04af-openshift-default-7dccd45f47-vxtn5\" (UID: \"2c2192ff-91c4-4488-8229-629018fd881d\") " pod="openshift-ingress/e2e-inf-b-ce04af-openshift-default-7dccd45f47-vxtn5" Nov 03 01:24:34.949265 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:24:34.949195 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2c2192ff-91c4-4488-8229-629018fd881d-workload-socket\") pod \"e2e-inf-b-ce04af-openshift-default-7dccd45f47-vxtn5\" (UID: \"2c2192ff-91c4-4488-8229-629018fd881d\") " pod="openshift-ingress/e2e-inf-b-ce04af-openshift-default-7dccd45f47-vxtn5" Nov 03 01:24:34.949558 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:24:34.949381 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2c2192ff-91c4-4488-8229-629018fd881d-istiod-ca-cert\") pod \"e2e-inf-b-ce04af-openshift-default-7dccd45f47-vxtn5\" (UID: \"2c2192ff-91c4-4488-8229-629018fd881d\") " pod="openshift-ingress/e2e-inf-b-ce04af-openshift-default-7dccd45f47-vxtn5" Nov 03 01:24:34.951333 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:24:34.951310 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2c2192ff-91c4-4488-8229-629018fd881d-istio-podinfo\") pod \"e2e-inf-b-ce04af-openshift-default-7dccd45f47-vxtn5\" (UID: \"2c2192ff-91c4-4488-8229-629018fd881d\") " pod="openshift-ingress/e2e-inf-b-ce04af-openshift-default-7dccd45f47-vxtn5" Nov 03 01:24:34.951402 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:24:34.951390 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2c2192ff-91c4-4488-8229-629018fd881d-istio-envoy\") pod \"e2e-inf-b-ce04af-openshift-default-7dccd45f47-vxtn5\" (UID: \"2c2192ff-91c4-4488-8229-629018fd881d\") " pod="openshift-ingress/e2e-inf-b-ce04af-openshift-default-7dccd45f47-vxtn5" Nov 03 01:24:34.956756 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:24:34.956730 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2c2192ff-91c4-4488-8229-629018fd881d-istio-token\") pod \"e2e-inf-b-ce04af-openshift-default-7dccd45f47-vxtn5\" (UID: \"2c2192ff-91c4-4488-8229-629018fd881d\") " pod="openshift-ingress/e2e-inf-b-ce04af-openshift-default-7dccd45f47-vxtn5" Nov 03 01:24:34.956988 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:24:34.956969 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jh5rp\" (UniqueName: \"kubernetes.io/projected/2c2192ff-91c4-4488-8229-629018fd881d-kube-api-access-jh5rp\") pod \"e2e-inf-b-ce04af-openshift-default-7dccd45f47-vxtn5\" (UID: \"2c2192ff-91c4-4488-8229-629018fd881d\") " pod="openshift-ingress/e2e-inf-b-ce04af-openshift-default-7dccd45f47-vxtn5" Nov 03 01:24:35.054262 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:24:35.054229 4939 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-ce04af-openshift-default-7dccd45f47-vxtn5" Nov 03 01:24:35.088743 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:24:35.088715 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:24:35.169983 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:24:35.169578 4939 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-4pw7v"] Nov 03 01:24:35.281347 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:24:35.281307 4939 generic.go:356] "Generic (PLEG): container finished" podID="d239ed01-45c1-4f47-b35d-50d2b437e179" containerID="d6c4261bd8ff736e5159aac1a56b5a5b08845f5bc34688fde2ef31fdf9b37939" exitCode=0 Nov 03 01:24:35.281524 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:24:35.281406 4939 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-7c759a/payload-processing-external-model-e2e-extmod-a-7c759a-799fk2rjw" Nov 03 01:24:35.281524 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:24:35.281451 4939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-7c759a/payload-processing-external-model-e2e-extmod-a-7c759a-799fk2rjw" event={"ID":"d239ed01-45c1-4f47-b35d-50d2b437e179","Type":"ContainerDied","Data":"d6c4261bd8ff736e5159aac1a56b5a5b08845f5bc34688fde2ef31fdf9b37939"} Nov 03 01:24:35.281524 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:24:35.281515 4939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-7c759a/payload-processing-external-model-e2e-extmod-a-7c759a-799fk2rjw" event={"ID":"d239ed01-45c1-4f47-b35d-50d2b437e179","Type":"ContainerDied","Data":"93252a13de37ac6f307a84717fecb4cd956e5ab6a97904441d7c1adcd15bc644"} Nov 03 01:24:35.281675 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:24:35.281551 4939 scope.go:122] "RemoveContainer" containerID="d6c4261bd8ff736e5159aac1a56b5a5b08845f5bc34688fde2ef31fdf9b37939" Nov 03 01:24:35.309463 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:24:35.309429 4939 scope.go:122] "RemoveContainer" containerID="d6c4261bd8ff736e5159aac1a56b5a5b08845f5bc34688fde2ef31fdf9b37939" Nov 03 01:24:35.323724 ip-10-0-140-107 kubenswrapper[4939]: E1103 01:24:35.322159 4939 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d6c4261bd8ff736e5159aac1a56b5a5b08845f5bc34688fde2ef31fdf9b37939\": container with ID starting with d6c4261bd8ff736e5159aac1a56b5a5b08845f5bc34688fde2ef31fdf9b37939 not found: ID does not exist" containerID="d6c4261bd8ff736e5159aac1a56b5a5b08845f5bc34688fde2ef31fdf9b37939" Nov 03 01:24:35.323724 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:24:35.322210 4939 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d6c4261bd8ff736e5159aac1a56b5a5b08845f5bc34688fde2ef31fdf9b37939"} err="failed to get container status \"d6c4261bd8ff736e5159aac1a56b5a5b08845f5bc34688fde2ef31fdf9b37939\": rpc error: code = NotFound desc = could not find container \"d6c4261bd8ff736e5159aac1a56b5a5b08845f5bc34688fde2ef31fdf9b37939\": container with ID starting with d6c4261bd8ff736e5159aac1a56b5a5b08845f5bc34688fde2ef31fdf9b37939 not found: ID does not exist" Nov 03 01:24:35.343901 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:24:35.343860 4939 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-7c759a/payload-processing-external-model-e2e-extmod-a-7c759a-799fk2rjw"] Nov 03 01:24:35.347714 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:24:35.347688 4939 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-a-7c759a/payload-processing-external-model-e2e-extmod-a-7c759a-799fk2rjw"] Nov 03 01:24:35.427219 ip-10-0-140-107 kubenswrapper[4939]: W1103 01:24:35.427185 4939 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2c2192ff_91c4_4488_8229_629018fd881d.slice/crio-c18027769179fc51c278647d7e4bbd823fb3c51faf9ab929569b3eff5f5d49a7 WatchSource:0}: Error finding container c18027769179fc51c278647d7e4bbd823fb3c51faf9ab929569b3eff5f5d49a7: Status 404 returned error can't find the container with id c18027769179fc51c278647d7e4bbd823fb3c51faf9ab929569b3eff5f5d49a7 Nov 03 01:24:35.427987 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:24:35.427958 4939 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-ce04af-openshift-default-7dccd45f47-vxtn5"] Nov 03 01:24:35.430703 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:24:35.430663 4939 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 01:24:35.430824 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:24:35.430740 4939 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 01:24:35.430824 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:24:35.430771 4939 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 01:24:35.482188 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:24:35.482165 4939 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7df33a0c-mdc5q" Nov 03 01:24:35.667111 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:24:35.667081 4939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/59795f2c-592e-4820-892a-bfdd35951594-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/59795f2c-592e-4820-892a-bfdd35951594-maas-api-xks-ca\") pod \"59795f2c-592e-4820-892a-bfdd35951594\" (UID: \"59795f2c-592e-4820-892a-bfdd35951594\") " Nov 03 01:24:35.667218 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:24:35.667199 4939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/59795f2c-592e-4820-892a-bfdd35951594-kube-api-access-z8gjb\" (UniqueName: \"kubernetes.io/projected/59795f2c-592e-4820-892a-bfdd35951594-kube-api-access-z8gjb\") pod \"59795f2c-592e-4820-892a-bfdd35951594\" (UID: \"59795f2c-592e-4820-892a-bfdd35951594\") " Nov 03 01:24:35.668402 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:24:35.667485 4939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/59795f2c-592e-4820-892a-bfdd35951594-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/59795f2c-592e-4820-892a-bfdd35951594-maas-api-openshift-ca\") pod \"59795f2c-592e-4820-892a-bfdd35951594\" (UID: \"59795f2c-592e-4820-892a-bfdd35951594\") " Nov 03 01:24:35.668402 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:24:35.667902 4939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/59795f2c-592e-4820-892a-bfdd35951594-maas-api-openshift-ca" pod "59795f2c-592e-4820-892a-bfdd35951594" (UID: "59795f2c-592e-4820-892a-bfdd35951594"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 01:24:35.668402 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:24:35.668367 4939 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/59795f2c-592e-4820-892a-bfdd35951594-maas-api-openshift-ca\") on node \"ip-10-0-140-107.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:24:35.669882 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:24:35.669853 4939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/59795f2c-592e-4820-892a-bfdd35951594-maas-api-xks-ca" pod "59795f2c-592e-4820-892a-bfdd35951594" (UID: "59795f2c-592e-4820-892a-bfdd35951594"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 01:24:35.670082 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:24:35.670062 4939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/59795f2c-592e-4820-892a-bfdd35951594-kube-api-access-z8gjb" pod "59795f2c-592e-4820-892a-bfdd35951594" (UID: "59795f2c-592e-4820-892a-bfdd35951594"). InnerVolumeSpecName "kube-api-access-z8gjb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 01:24:35.770051 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:24:35.770009 4939 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/59795f2c-592e-4820-892a-bfdd35951594-maas-api-xks-ca\") on node \"ip-10-0-140-107.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:24:35.770051 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:24:35.770045 4939 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-z8gjb\" (UniqueName: \"kubernetes.io/projected/59795f2c-592e-4820-892a-bfdd35951594-kube-api-access-z8gjb\") on node \"ip-10-0-140-107.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:24:35.873002 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:24:35.872892 4939 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d239ed01-45c1-4f47-b35d-50d2b437e179" path="/var/lib/kubelet/pods/d239ed01-45c1-4f47-b35d-50d2b437e179/volumes" Nov 03 01:24:35.915687 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:24:35.915652 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:24:36.294741 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:24:36.294700 4939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-ce04af-openshift-default-7dccd45f47-vxtn5" event={"ID":"2c2192ff-91c4-4488-8229-629018fd881d","Type":"ContainerStarted","Data":"f82319615907c985e605951dc5dc72d7600b1fbb6431ed69d94645c064d8dc84"} Nov 03 01:24:36.294963 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:24:36.294768 4939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-ce04af-openshift-default-7dccd45f47-vxtn5" event={"ID":"2c2192ff-91c4-4488-8229-629018fd881d","Type":"ContainerStarted","Data":"c18027769179fc51c278647d7e4bbd823fb3c51faf9ab929569b3eff5f5d49a7"} Nov 03 01:24:36.296379 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:24:36.296353 4939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7df33a0c-mdc5q" event={"ID":"59795f2c-592e-4820-892a-bfdd35951594","Type":"ContainerDied","Data":"01ec8a4513efebc24ef2a75044ea9f11e2705e68cbaff1e9f398e4eb60078cbd"} Nov 03 01:24:36.296487 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:24:36.296382 4939 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7df33a0c-mdc5q" Nov 03 01:24:36.296487 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:24:36.296393 4939 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="01ec8a4513efebc24ef2a75044ea9f11e2705e68cbaff1e9f398e4eb60078cbd" Nov 03 01:24:36.321497 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:24:36.321443 4939 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-b-ce04af-openshift-default-7dccd45f47-vxtn5" podStartSLOduration=11.321429843 podStartE2EDuration="11.321429843s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 01:24:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 01:24:36.318680955 +0000 UTC m=+2624.583015900" watchObservedRunningTime="2026-11-03 01:24:36.321429843 +0000 UTC m=+2624.585764787" Nov 03 01:24:37.057940 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:24:37.057722 4939 patch_prober.go:29] interesting pod/e2e-inf-b-ce04af-openshift-default-7dccd45f47-vxtn5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.42:15021/healthz/ready\": dial tcp 10.132.1.42:15021: connect: connection refused" start-of-body= Nov 03 01:24:37.057940 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:24:37.057814 4939 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-ce04af-openshift-default-7dccd45f47-vxtn5" podUID="2c2192ff-91c4-4488-8229-629018fd881d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.42:15021/healthz/ready\": dial tcp 10.132.1.42:15021: connect: connection refused" Nov 03 01:24:37.058506 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:24:37.058164 4939 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-ce04af-openshift-default-7dccd45f47-vxtn5" Nov 03 01:24:37.589929 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:24:37.582040 4939 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-88bbf7/distinct-2-w1-88bbf7-kserve-5457cdcdbd-l6tfd" podUID="a397931b-faa9-4c2e-8290-491350546663" containerName="main" probeResult="failure" output="Get \"https://10.132.0.85:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 01:24:37.838050 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:24:37.838013 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:24:38.058414 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:24:38.058316 4939 patch_prober.go:29] interesting pod/e2e-inf-b-ce04af-openshift-default-7dccd45f47-vxtn5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.42:15021/healthz/ready\": dial tcp 10.132.1.42:15021: connect: connection refused" start-of-body= Nov 03 01:24:38.059036 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:24:38.058999 4939 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-ce04af-openshift-default-7dccd45f47-vxtn5" podUID="2c2192ff-91c4-4488-8229-629018fd881d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.42:15021/healthz/ready\": dial tcp 10.132.1.42:15021: connect: connection refused" Nov 03 01:24:38.715787 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:24:38.715751 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:24:39.072128 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:24:39.057350 4939 patch_prober.go:29] interesting pod/e2e-inf-b-ce04af-openshift-default-7dccd45f47-vxtn5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.42:15021/healthz/ready\": dial tcp 10.132.1.42:15021: connect: connection refused" start-of-body= Nov 03 01:24:39.072128 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:24:39.057415 4939 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-ce04af-openshift-default-7dccd45f47-vxtn5" podUID="2c2192ff-91c4-4488-8229-629018fd881d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.42:15021/healthz/ready\": dial tcp 10.132.1.42:15021: connect: connection refused" Nov 03 01:24:39.575931 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:24:39.573282 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:24:40.055371 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:24:40.055334 4939 patch_prober.go:29] interesting pod/e2e-inf-b-ce04af-openshift-default-7dccd45f47-vxtn5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.42:15021/healthz/ready\": dial tcp 10.132.1.42:15021: connect: connection refused" start-of-body= Nov 03 01:24:40.055647 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:24:40.055619 4939 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-ce04af-openshift-default-7dccd45f47-vxtn5" podUID="2c2192ff-91c4-4488-8229-629018fd881d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.42:15021/healthz/ready\": dial tcp 10.132.1.42:15021: connect: connection refused" Nov 03 01:24:40.301305 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:24:40.301227 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:24:40.573930 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:24:40.572470 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:24:41.092311 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:24:41.092021 4939 patch_prober.go:29] interesting pod/e2e-inf-b-ce04af-openshift-default-7dccd45f47-vxtn5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.42:15021/healthz/ready\": dial tcp 10.132.1.42:15021: connect: connection refused" start-of-body= Nov 03 01:24:41.092311 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:24:41.092087 4939 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-ce04af-openshift-default-7dccd45f47-vxtn5" podUID="2c2192ff-91c4-4488-8229-629018fd881d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.42:15021/healthz/ready\": dial tcp 10.132.1.42:15021: connect: connection refused" Nov 03 01:24:42.060052 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:24:42.060015 4939 patch_prober.go:29] interesting pod/e2e-inf-b-ce04af-openshift-default-7dccd45f47-vxtn5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.42:15021/healthz/ready\": dial tcp 10.132.1.42:15021: connect: connection refused" start-of-body= Nov 03 01:24:42.060631 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:24:42.060596 4939 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-ce04af-openshift-default-7dccd45f47-vxtn5" podUID="2c2192ff-91c4-4488-8229-629018fd881d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.42:15021/healthz/ready\": dial tcp 10.132.1.42:15021: connect: connection refused" Nov 03 01:24:42.123175 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:24:42.123136 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:24:42.242932 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:24:42.235735 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:24:42.425280 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:24:42.421515 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:24:42.619146 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:24:42.612197 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:24:42.637928 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:24:42.637127 4939 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-d25759/payload-processing-external-model-e2e-extmod-a-d25759-56bbc6hhk"] Nov 03 01:24:42.724133 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:24:42.724053 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:24:43.114101 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:24:43.114065 4939 patch_prober.go:29] interesting pod/e2e-inf-b-ce04af-openshift-default-7dccd45f47-vxtn5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.42:15021/healthz/ready\": dial tcp 10.132.1.42:15021: connect: connection refused" start-of-body= Nov 03 01:24:43.115175 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:24:43.115138 4939 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-ce04af-openshift-default-7dccd45f47-vxtn5" podUID="2c2192ff-91c4-4488-8229-629018fd881d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.42:15021/healthz/ready\": dial tcp 10.132.1.42:15021: connect: connection refused" Nov 03 01:24:43.306875 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:24:43.306828 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:24:43.313167 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:24:43.313135 4939 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-e1f08d/payload-processing-external-model-e2e-extmod-b-e1f08d-9f5cms4gl"] Nov 03 01:24:44.055272 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:24:44.055224 4939 patch_prober.go:29] interesting pod/e2e-inf-b-ce04af-openshift-default-7dccd45f47-vxtn5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.42:15021/healthz/ready\": dial tcp 10.132.1.42:15021: connect: connection refused" start-of-body= Nov 03 01:24:44.055490 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:24:44.055294 4939 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-ce04af-openshift-default-7dccd45f47-vxtn5" podUID="2c2192ff-91c4-4488-8229-629018fd881d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.42:15021/healthz/ready\": dial tcp 10.132.1.42:15021: connect: connection refused" Nov 03 01:24:44.195699 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:24:44.195645 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:24:44.654266 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:24:44.654225 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:24:44.912791 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:24:44.912704 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:24:45.060867 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:24:45.060839 4939 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-ce04af-openshift-default-7dccd45f47-vxtn5" Nov 03 01:24:45.061367 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:24:45.061347 4939 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-ce04af-openshift-default-7dccd45f47-vxtn5" Nov 03 01:24:45.062217 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:24:45.062198 4939 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-ce04af-openshift-default-7dccd45f47-vxtn5" Nov 03 01:24:45.579897 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:24:45.579857 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:24:46.352216 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:24:46.352154 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:24:46.787872 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:24:46.787783 4939 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-821f5a/payload-processing-external-model-e2e-extmod-a-821f5a-5df8cj84n"] Nov 03 01:24:46.811896 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:24:46.811868 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:24:46.917659 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:24:46.917628 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:24:47.384931 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:24:47.384591 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:24:47.637902 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:24:47.637810 4939 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-a-d25759/payload-processing-external-model-e2e-extmod-a-d25759-56bbc6hhk" podUID="8c6f8abe-dd67-45b9-b043-1c1e13ee96de" containerName="payload-processing" containerID="cri-o://3853ec3202f62b4818fdf238ceb134be23c50838315b4ad8bdef74edad15dcc4" gracePeriod=25 Nov 03 01:24:47.763989 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:24:47.760966 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:24:47.974235 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:24:47.974204 4939 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-d25759/payload-processing-external-model-e2e-extmod-a-d25759-56bbc6hhk" Nov 03 01:24:48.040440 ip-10-0-140-107 kubenswrapper[4939]: E1103 01:24:48.040409 4939 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-sthjh" containerName="extractor" Nov 03 01:24:48.090652 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:24:48.090622 4939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8c6f8abe-dd67-45b9-b043-1c1e13ee96de-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/8c6f8abe-dd67-45b9-b043-1c1e13ee96de-routing-overlay-volume\") pod \"8c6f8abe-dd67-45b9-b043-1c1e13ee96de\" (UID: \"8c6f8abe-dd67-45b9-b043-1c1e13ee96de\") " Nov 03 01:24:48.090842 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:24:48.090823 4939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8c6f8abe-dd67-45b9-b043-1c1e13ee96de-provider-credentials\" (UniqueName: \"kubernetes.io/projected/8c6f8abe-dd67-45b9-b043-1c1e13ee96de-provider-credentials\") pod \"8c6f8abe-dd67-45b9-b043-1c1e13ee96de\" (UID: \"8c6f8abe-dd67-45b9-b043-1c1e13ee96de\") " Nov 03 01:24:48.091028 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:24:48.091007 4939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8c6f8abe-dd67-45b9-b043-1c1e13ee96de-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8c6f8abe-dd67-45b9-b043-1c1e13ee96de-plugins-config-volume\") pod \"8c6f8abe-dd67-45b9-b043-1c1e13ee96de\" (UID: \"8c6f8abe-dd67-45b9-b043-1c1e13ee96de\") " Nov 03 01:24:48.091120 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:24:48.091069 4939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8c6f8abe-dd67-45b9-b043-1c1e13ee96de-routing-overlay-volume" pod "8c6f8abe-dd67-45b9-b043-1c1e13ee96de" (UID: "8c6f8abe-dd67-45b9-b043-1c1e13ee96de"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 01:24:48.091204 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:24:48.091188 4939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8c6f8abe-dd67-45b9-b043-1c1e13ee96de-kube-api-access-9t2sl\" (UniqueName: \"kubernetes.io/projected/8c6f8abe-dd67-45b9-b043-1c1e13ee96de-kube-api-access-9t2sl\") pod \"8c6f8abe-dd67-45b9-b043-1c1e13ee96de\" (UID: \"8c6f8abe-dd67-45b9-b043-1c1e13ee96de\") " Nov 03 01:24:48.091565 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:24:48.091538 4939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8c6f8abe-dd67-45b9-b043-1c1e13ee96de-plugins-config-volume" pod "8c6f8abe-dd67-45b9-b043-1c1e13ee96de" (UID: "8c6f8abe-dd67-45b9-b043-1c1e13ee96de"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 01:24:48.092200 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:24:48.092176 4939 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/8c6f8abe-dd67-45b9-b043-1c1e13ee96de-routing-overlay-volume\") on node \"ip-10-0-140-107.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:24:48.092200 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:24:48.092201 4939 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8c6f8abe-dd67-45b9-b043-1c1e13ee96de-plugins-config-volume\") on node \"ip-10-0-140-107.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:24:48.093089 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:24:48.093068 4939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8c6f8abe-dd67-45b9-b043-1c1e13ee96de-provider-credentials" pod "8c6f8abe-dd67-45b9-b043-1c1e13ee96de" (UID: "8c6f8abe-dd67-45b9-b043-1c1e13ee96de"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 01:24:48.093465 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:24:48.093444 4939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8c6f8abe-dd67-45b9-b043-1c1e13ee96de-kube-api-access-9t2sl" pod "8c6f8abe-dd67-45b9-b043-1c1e13ee96de" (UID: "8c6f8abe-dd67-45b9-b043-1c1e13ee96de"). InnerVolumeSpecName "kube-api-access-9t2sl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 01:24:48.194188 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:24:48.194098 4939 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/8c6f8abe-dd67-45b9-b043-1c1e13ee96de-provider-credentials\") on node \"ip-10-0-140-107.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:24:48.194188 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:24:48.194131 4939 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9t2sl\" (UniqueName: \"kubernetes.io/projected/8c6f8abe-dd67-45b9-b043-1c1e13ee96de-kube-api-access-9t2sl\") on node \"ip-10-0-140-107.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:24:48.313989 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:24:48.313940 4939 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-b-e1f08d/payload-processing-external-model-e2e-extmod-b-e1f08d-9f5cms4gl" podUID="0a9abcbc-bbe0-4f43-ae83-7e2ec3bc9d13" containerName="payload-processing" containerID="cri-o://4359c6b272ecb521ff8a09fef0cb1539a23d09da77aec9c0c81efba9718511f7" gracePeriod=25 Nov 03 01:24:48.410779 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:24:48.410749 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:24:48.700971 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:24:48.700948 4939 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-e1f08d/payload-processing-external-model-e2e-extmod-b-e1f08d-9f5cms4gl" Nov 03 01:24:48.752638 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:24:48.752214 4939 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-b-ce04af-openshift-default-7dccd45f47-75xwt"] Nov 03 01:24:48.754759 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:24:48.754725 4939 cpu_manager.go:414] "containerMap: removing container" podUID="59795f2c-592e-4820-892a-bfdd35951594" containerName="revoke-keys" Nov 03 01:24:48.754759 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:24:48.754759 4939 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="59795f2c-592e-4820-892a-bfdd35951594" containerName="revoke-keys" Nov 03 01:24:48.755559 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:24:48.754899 4939 cpu_manager.go:414] "containerMap: removing container" podUID="8c6f8abe-dd67-45b9-b043-1c1e13ee96de" containerName="payload-processing" Nov 03 01:24:48.755559 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:24:48.754927 4939 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8c6f8abe-dd67-45b9-b043-1c1e13ee96de" containerName="payload-processing" Nov 03 01:24:48.755559 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:24:48.754973 4939 cpu_manager.go:414] "containerMap: removing container" podUID="0a9abcbc-bbe0-4f43-ae83-7e2ec3bc9d13" containerName="payload-processing" Nov 03 01:24:48.755559 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:24:48.754982 4939 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0a9abcbc-bbe0-4f43-ae83-7e2ec3bc9d13" containerName="payload-processing" Nov 03 01:24:48.755559 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:24:48.755450 4939 memory_manager.go:365] "RemoveStaleState removing state" podUID="59795f2c-592e-4820-892a-bfdd35951594" containerName="revoke-keys" Nov 03 01:24:48.755559 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:24:48.755504 4939 memory_manager.go:365] "RemoveStaleState removing state" podUID="8c6f8abe-dd67-45b9-b043-1c1e13ee96de" containerName="payload-processing" Nov 03 01:24:48.755559 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:24:48.755519 4939 memory_manager.go:365] "RemoveStaleState removing state" podUID="0a9abcbc-bbe0-4f43-ae83-7e2ec3bc9d13" containerName="payload-processing" Nov 03 01:24:48.771138 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:24:48.771106 4939 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-ce04af-openshift-default-7dccd45f47-75xwt"] Nov 03 01:24:48.771268 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:24:48.771221 4939 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-ce04af-openshift-default-7dccd45f47-75xwt" Nov 03 01:24:48.786066 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:24:48.786039 4939 generic.go:356] "Generic (PLEG): container finished" podID="0a9abcbc-bbe0-4f43-ae83-7e2ec3bc9d13" containerID="4359c6b272ecb521ff8a09fef0cb1539a23d09da77aec9c0c81efba9718511f7" exitCode=0 Nov 03 01:24:48.786197 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:24:48.786097 4939 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-e1f08d/payload-processing-external-model-e2e-extmod-b-e1f08d-9f5cms4gl" Nov 03 01:24:48.786959 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:24:48.786355 4939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-e1f08d/payload-processing-external-model-e2e-extmod-b-e1f08d-9f5cms4gl" event={"ID":"0a9abcbc-bbe0-4f43-ae83-7e2ec3bc9d13","Type":"ContainerDied","Data":"4359c6b272ecb521ff8a09fef0cb1539a23d09da77aec9c0c81efba9718511f7"} Nov 03 01:24:48.786959 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:24:48.786403 4939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-e1f08d/payload-processing-external-model-e2e-extmod-b-e1f08d-9f5cms4gl" event={"ID":"0a9abcbc-bbe0-4f43-ae83-7e2ec3bc9d13","Type":"ContainerDied","Data":"a54e70125aaa3e541a3ffa324cd658a10dc5df2256c3be61b921362c7f86f70c"} Nov 03 01:24:48.786959 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:24:48.786435 4939 scope.go:122] "RemoveContainer" containerID="4359c6b272ecb521ff8a09fef0cb1539a23d09da77aec9c0c81efba9718511f7" Nov 03 01:24:48.788865 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:24:48.788833 4939 generic.go:356] "Generic (PLEG): container finished" podID="8c6f8abe-dd67-45b9-b043-1c1e13ee96de" containerID="3853ec3202f62b4818fdf238ceb134be23c50838315b4ad8bdef74edad15dcc4" exitCode=0 Nov 03 01:24:48.789026 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:24:48.788875 4939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-d25759/payload-processing-external-model-e2e-extmod-a-d25759-56bbc6hhk" event={"ID":"8c6f8abe-dd67-45b9-b043-1c1e13ee96de","Type":"ContainerDied","Data":"3853ec3202f62b4818fdf238ceb134be23c50838315b4ad8bdef74edad15dcc4"} Nov 03 01:24:48.789125 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:24:48.789108 4939 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-d25759/payload-processing-external-model-e2e-extmod-a-d25759-56bbc6hhk" Nov 03 01:24:48.790328 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:24:48.790287 4939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-d25759/payload-processing-external-model-e2e-extmod-a-d25759-56bbc6hhk" event={"ID":"8c6f8abe-dd67-45b9-b043-1c1e13ee96de","Type":"ContainerDied","Data":"3860b5f7441b5b2a38486e230a41f5cdd66f47ec738f6aac81ab22a68435402f"} Nov 03 01:24:48.804298 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:24:48.804274 4939 scope.go:122] "RemoveContainer" containerID="4359c6b272ecb521ff8a09fef0cb1539a23d09da77aec9c0c81efba9718511f7" Nov 03 01:24:48.804710 ip-10-0-140-107 kubenswrapper[4939]: E1103 01:24:48.804673 4939 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4359c6b272ecb521ff8a09fef0cb1539a23d09da77aec9c0c81efba9718511f7\": container with ID starting with 4359c6b272ecb521ff8a09fef0cb1539a23d09da77aec9c0c81efba9718511f7 not found: ID does not exist" containerID="4359c6b272ecb521ff8a09fef0cb1539a23d09da77aec9c0c81efba9718511f7" Nov 03 01:24:48.804858 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:24:48.804712 4939 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4359c6b272ecb521ff8a09fef0cb1539a23d09da77aec9c0c81efba9718511f7"} err="failed to get container status \"4359c6b272ecb521ff8a09fef0cb1539a23d09da77aec9c0c81efba9718511f7\": rpc error: code = NotFound desc = could not find container \"4359c6b272ecb521ff8a09fef0cb1539a23d09da77aec9c0c81efba9718511f7\": container with ID starting with 4359c6b272ecb521ff8a09fef0cb1539a23d09da77aec9c0c81efba9718511f7 not found: ID does not exist" Nov 03 01:24:48.804858 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:24:48.804737 4939 scope.go:122] "RemoveContainer" containerID="3853ec3202f62b4818fdf238ceb134be23c50838315b4ad8bdef74edad15dcc4" Nov 03 01:24:48.805395 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:24:48.805362 4939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0a9abcbc-bbe0-4f43-ae83-7e2ec3bc9d13-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0a9abcbc-bbe0-4f43-ae83-7e2ec3bc9d13-plugins-config-volume\") pod \"0a9abcbc-bbe0-4f43-ae83-7e2ec3bc9d13\" (UID: \"0a9abcbc-bbe0-4f43-ae83-7e2ec3bc9d13\") " Nov 03 01:24:48.805631 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:24:48.805613 4939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0a9abcbc-bbe0-4f43-ae83-7e2ec3bc9d13-kube-api-access-qh966\" (UniqueName: \"kubernetes.io/projected/0a9abcbc-bbe0-4f43-ae83-7e2ec3bc9d13-kube-api-access-qh966\") pod \"0a9abcbc-bbe0-4f43-ae83-7e2ec3bc9d13\" (UID: \"0a9abcbc-bbe0-4f43-ae83-7e2ec3bc9d13\") " Nov 03 01:24:48.805716 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:24:48.805664 4939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0a9abcbc-bbe0-4f43-ae83-7e2ec3bc9d13-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/0a9abcbc-bbe0-4f43-ae83-7e2ec3bc9d13-routing-overlay-volume\") pod \"0a9abcbc-bbe0-4f43-ae83-7e2ec3bc9d13\" (UID: \"0a9abcbc-bbe0-4f43-ae83-7e2ec3bc9d13\") " Nov 03 01:24:48.805778 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:24:48.805743 4939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0a9abcbc-bbe0-4f43-ae83-7e2ec3bc9d13-provider-credentials\" (UniqueName: \"kubernetes.io/projected/0a9abcbc-bbe0-4f43-ae83-7e2ec3bc9d13-provider-credentials\") pod \"0a9abcbc-bbe0-4f43-ae83-7e2ec3bc9d13\" (UID: \"0a9abcbc-bbe0-4f43-ae83-7e2ec3bc9d13\") " Nov 03 01:24:48.808737 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:24:48.807880 4939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0a9abcbc-bbe0-4f43-ae83-7e2ec3bc9d13-plugins-config-volume" pod "0a9abcbc-bbe0-4f43-ae83-7e2ec3bc9d13" (UID: "0a9abcbc-bbe0-4f43-ae83-7e2ec3bc9d13"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 01:24:48.809346 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:24:48.809324 4939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0a9abcbc-bbe0-4f43-ae83-7e2ec3bc9d13-routing-overlay-volume" pod "0a9abcbc-bbe0-4f43-ae83-7e2ec3bc9d13" (UID: "0a9abcbc-bbe0-4f43-ae83-7e2ec3bc9d13"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 01:24:48.810035 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:24:48.810012 4939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0a9abcbc-bbe0-4f43-ae83-7e2ec3bc9d13-provider-credentials" pod "0a9abcbc-bbe0-4f43-ae83-7e2ec3bc9d13" (UID: "0a9abcbc-bbe0-4f43-ae83-7e2ec3bc9d13"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 01:24:48.811525 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:24:48.811323 4939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0a9abcbc-bbe0-4f43-ae83-7e2ec3bc9d13-kube-api-access-qh966" pod "0a9abcbc-bbe0-4f43-ae83-7e2ec3bc9d13" (UID: "0a9abcbc-bbe0-4f43-ae83-7e2ec3bc9d13"). InnerVolumeSpecName "kube-api-access-qh966". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 01:24:48.825176 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:24:48.824954 4939 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-d25759/payload-processing-external-model-e2e-extmod-a-d25759-56bbc6hhk"] Nov 03 01:24:48.830945 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:24:48.830884 4939 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-a-d25759/payload-processing-external-model-e2e-extmod-a-d25759-56bbc6hhk"] Nov 03 01:24:48.850070 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:24:48.850052 4939 scope.go:122] "RemoveContainer" containerID="3853ec3202f62b4818fdf238ceb134be23c50838315b4ad8bdef74edad15dcc4" Nov 03 01:24:48.850400 ip-10-0-140-107 kubenswrapper[4939]: E1103 01:24:48.850377 4939 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3853ec3202f62b4818fdf238ceb134be23c50838315b4ad8bdef74edad15dcc4\": container with ID starting with 3853ec3202f62b4818fdf238ceb134be23c50838315b4ad8bdef74edad15dcc4 not found: ID does not exist" containerID="3853ec3202f62b4818fdf238ceb134be23c50838315b4ad8bdef74edad15dcc4" Nov 03 01:24:48.850474 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:24:48.850410 4939 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3853ec3202f62b4818fdf238ceb134be23c50838315b4ad8bdef74edad15dcc4"} err="failed to get container status \"3853ec3202f62b4818fdf238ceb134be23c50838315b4ad8bdef74edad15dcc4\": rpc error: code = NotFound desc = could not find container \"3853ec3202f62b4818fdf238ceb134be23c50838315b4ad8bdef74edad15dcc4\": container with ID starting with 3853ec3202f62b4818fdf238ceb134be23c50838315b4ad8bdef74edad15dcc4 not found: ID does not exist" Nov 03 01:24:48.870346 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:24:48.870315 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:24:48.921941 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:24:48.920164 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8ea9e0d9-99b8-4bbb-b98d-b459f1560c89-istio-podinfo\") pod \"e2e-inf-b-ce04af-openshift-default-7dccd45f47-75xwt\" (UID: \"8ea9e0d9-99b8-4bbb-b98d-b459f1560c89\") " pod="openshift-ingress/e2e-inf-b-ce04af-openshift-default-7dccd45f47-75xwt" Nov 03 01:24:48.921941 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:24:48.920241 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8ea9e0d9-99b8-4bbb-b98d-b459f1560c89-credential-socket\") pod \"e2e-inf-b-ce04af-openshift-default-7dccd45f47-75xwt\" (UID: \"8ea9e0d9-99b8-4bbb-b98d-b459f1560c89\") " pod="openshift-ingress/e2e-inf-b-ce04af-openshift-default-7dccd45f47-75xwt" Nov 03 01:24:48.921941 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:24:48.920321 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8ea9e0d9-99b8-4bbb-b98d-b459f1560c89-istio-envoy\") pod \"e2e-inf-b-ce04af-openshift-default-7dccd45f47-75xwt\" (UID: \"8ea9e0d9-99b8-4bbb-b98d-b459f1560c89\") " pod="openshift-ingress/e2e-inf-b-ce04af-openshift-default-7dccd45f47-75xwt" Nov 03 01:24:48.921941 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:24:48.920552 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8ea9e0d9-99b8-4bbb-b98d-b459f1560c89-workload-socket\") pod \"e2e-inf-b-ce04af-openshift-default-7dccd45f47-75xwt\" (UID: \"8ea9e0d9-99b8-4bbb-b98d-b459f1560c89\") " pod="openshift-ingress/e2e-inf-b-ce04af-openshift-default-7dccd45f47-75xwt" Nov 03 01:24:48.921941 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:24:48.920792 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8ea9e0d9-99b8-4bbb-b98d-b459f1560c89-workload-certs\") pod \"e2e-inf-b-ce04af-openshift-default-7dccd45f47-75xwt\" (UID: \"8ea9e0d9-99b8-4bbb-b98d-b459f1560c89\") " pod="openshift-ingress/e2e-inf-b-ce04af-openshift-default-7dccd45f47-75xwt" Nov 03 01:24:48.921941 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:24:48.920872 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8ea9e0d9-99b8-4bbb-b98d-b459f1560c89-istiod-ca-cert\") pod \"e2e-inf-b-ce04af-openshift-default-7dccd45f47-75xwt\" (UID: \"8ea9e0d9-99b8-4bbb-b98d-b459f1560c89\") " pod="openshift-ingress/e2e-inf-b-ce04af-openshift-default-7dccd45f47-75xwt" Nov 03 01:24:48.921941 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:24:48.920967 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zdrp9\" (UniqueName: \"kubernetes.io/projected/8ea9e0d9-99b8-4bbb-b98d-b459f1560c89-kube-api-access-zdrp9\") pod \"e2e-inf-b-ce04af-openshift-default-7dccd45f47-75xwt\" (UID: \"8ea9e0d9-99b8-4bbb-b98d-b459f1560c89\") " pod="openshift-ingress/e2e-inf-b-ce04af-openshift-default-7dccd45f47-75xwt" Nov 03 01:24:48.921941 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:24:48.921011 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8ea9e0d9-99b8-4bbb-b98d-b459f1560c89-istio-data\") pod \"e2e-inf-b-ce04af-openshift-default-7dccd45f47-75xwt\" (UID: \"8ea9e0d9-99b8-4bbb-b98d-b459f1560c89\") " pod="openshift-ingress/e2e-inf-b-ce04af-openshift-default-7dccd45f47-75xwt" Nov 03 01:24:48.921941 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:24:48.921069 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8ea9e0d9-99b8-4bbb-b98d-b459f1560c89-istio-token\") pod \"e2e-inf-b-ce04af-openshift-default-7dccd45f47-75xwt\" (UID: \"8ea9e0d9-99b8-4bbb-b98d-b459f1560c89\") " pod="openshift-ingress/e2e-inf-b-ce04af-openshift-default-7dccd45f47-75xwt" Nov 03 01:24:48.922940 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:24:48.921241 4939 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0a9abcbc-bbe0-4f43-ae83-7e2ec3bc9d13-plugins-config-volume\") on node \"ip-10-0-140-107.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:24:48.922940 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:24:48.922744 4939 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qh966\" (UniqueName: \"kubernetes.io/projected/0a9abcbc-bbe0-4f43-ae83-7e2ec3bc9d13-kube-api-access-qh966\") on node \"ip-10-0-140-107.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:24:48.922940 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:24:48.922779 4939 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/0a9abcbc-bbe0-4f43-ae83-7e2ec3bc9d13-routing-overlay-volume\") on node \"ip-10-0-140-107.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:24:48.922940 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:24:48.922798 4939 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/0a9abcbc-bbe0-4f43-ae83-7e2ec3bc9d13-provider-credentials\") on node \"ip-10-0-140-107.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:24:48.988713 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:24:48.988681 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:24:49.024791 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:24:49.024719 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8ea9e0d9-99b8-4bbb-b98d-b459f1560c89-workload-certs\") pod \"e2e-inf-b-ce04af-openshift-default-7dccd45f47-75xwt\" (UID: \"8ea9e0d9-99b8-4bbb-b98d-b459f1560c89\") " pod="openshift-ingress/e2e-inf-b-ce04af-openshift-default-7dccd45f47-75xwt" Nov 03 01:24:49.024791 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:24:49.024776 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8ea9e0d9-99b8-4bbb-b98d-b459f1560c89-istiod-ca-cert\") pod \"e2e-inf-b-ce04af-openshift-default-7dccd45f47-75xwt\" (UID: \"8ea9e0d9-99b8-4bbb-b98d-b459f1560c89\") " pod="openshift-ingress/e2e-inf-b-ce04af-openshift-default-7dccd45f47-75xwt" Nov 03 01:24:49.025036 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:24:49.024833 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zdrp9\" (UniqueName: \"kubernetes.io/projected/8ea9e0d9-99b8-4bbb-b98d-b459f1560c89-kube-api-access-zdrp9\") pod \"e2e-inf-b-ce04af-openshift-default-7dccd45f47-75xwt\" (UID: \"8ea9e0d9-99b8-4bbb-b98d-b459f1560c89\") " pod="openshift-ingress/e2e-inf-b-ce04af-openshift-default-7dccd45f47-75xwt" Nov 03 01:24:49.025036 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:24:49.024861 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8ea9e0d9-99b8-4bbb-b98d-b459f1560c89-istio-data\") pod \"e2e-inf-b-ce04af-openshift-default-7dccd45f47-75xwt\" (UID: \"8ea9e0d9-99b8-4bbb-b98d-b459f1560c89\") " pod="openshift-ingress/e2e-inf-b-ce04af-openshift-default-7dccd45f47-75xwt" Nov 03 01:24:49.025036 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:24:49.024926 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8ea9e0d9-99b8-4bbb-b98d-b459f1560c89-istio-token\") pod \"e2e-inf-b-ce04af-openshift-default-7dccd45f47-75xwt\" (UID: \"8ea9e0d9-99b8-4bbb-b98d-b459f1560c89\") " pod="openshift-ingress/e2e-inf-b-ce04af-openshift-default-7dccd45f47-75xwt" Nov 03 01:24:49.025036 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:24:49.024994 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8ea9e0d9-99b8-4bbb-b98d-b459f1560c89-istio-podinfo\") pod \"e2e-inf-b-ce04af-openshift-default-7dccd45f47-75xwt\" (UID: \"8ea9e0d9-99b8-4bbb-b98d-b459f1560c89\") " pod="openshift-ingress/e2e-inf-b-ce04af-openshift-default-7dccd45f47-75xwt" Nov 03 01:24:49.025036 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:24:49.025033 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8ea9e0d9-99b8-4bbb-b98d-b459f1560c89-credential-socket\") pod \"e2e-inf-b-ce04af-openshift-default-7dccd45f47-75xwt\" (UID: \"8ea9e0d9-99b8-4bbb-b98d-b459f1560c89\") " pod="openshift-ingress/e2e-inf-b-ce04af-openshift-default-7dccd45f47-75xwt" Nov 03 01:24:49.025320 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:24:49.025108 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8ea9e0d9-99b8-4bbb-b98d-b459f1560c89-istio-envoy\") pod \"e2e-inf-b-ce04af-openshift-default-7dccd45f47-75xwt\" (UID: \"8ea9e0d9-99b8-4bbb-b98d-b459f1560c89\") " pod="openshift-ingress/e2e-inf-b-ce04af-openshift-default-7dccd45f47-75xwt" Nov 03 01:24:49.025320 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:24:49.025195 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8ea9e0d9-99b8-4bbb-b98d-b459f1560c89-workload-certs\") pod \"e2e-inf-b-ce04af-openshift-default-7dccd45f47-75xwt\" (UID: \"8ea9e0d9-99b8-4bbb-b98d-b459f1560c89\") " pod="openshift-ingress/e2e-inf-b-ce04af-openshift-default-7dccd45f47-75xwt" Nov 03 01:24:49.025441 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:24:49.025319 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8ea9e0d9-99b8-4bbb-b98d-b459f1560c89-workload-socket\") pod \"e2e-inf-b-ce04af-openshift-default-7dccd45f47-75xwt\" (UID: \"8ea9e0d9-99b8-4bbb-b98d-b459f1560c89\") " pod="openshift-ingress/e2e-inf-b-ce04af-openshift-default-7dccd45f47-75xwt" Nov 03 01:24:49.025567 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:24:49.025541 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8ea9e0d9-99b8-4bbb-b98d-b459f1560c89-istiod-ca-cert\") pod \"e2e-inf-b-ce04af-openshift-default-7dccd45f47-75xwt\" (UID: \"8ea9e0d9-99b8-4bbb-b98d-b459f1560c89\") " pod="openshift-ingress/e2e-inf-b-ce04af-openshift-default-7dccd45f47-75xwt" Nov 03 01:24:49.025813 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:24:49.025782 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8ea9e0d9-99b8-4bbb-b98d-b459f1560c89-istio-data\") pod \"e2e-inf-b-ce04af-openshift-default-7dccd45f47-75xwt\" (UID: \"8ea9e0d9-99b8-4bbb-b98d-b459f1560c89\") " pod="openshift-ingress/e2e-inf-b-ce04af-openshift-default-7dccd45f47-75xwt" Nov 03 01:24:49.025959 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:24:49.025824 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8ea9e0d9-99b8-4bbb-b98d-b459f1560c89-workload-socket\") pod \"e2e-inf-b-ce04af-openshift-default-7dccd45f47-75xwt\" (UID: \"8ea9e0d9-99b8-4bbb-b98d-b459f1560c89\") " pod="openshift-ingress/e2e-inf-b-ce04af-openshift-default-7dccd45f47-75xwt" Nov 03 01:24:49.025959 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:24:49.025902 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8ea9e0d9-99b8-4bbb-b98d-b459f1560c89-credential-socket\") pod \"e2e-inf-b-ce04af-openshift-default-7dccd45f47-75xwt\" (UID: \"8ea9e0d9-99b8-4bbb-b98d-b459f1560c89\") " pod="openshift-ingress/e2e-inf-b-ce04af-openshift-default-7dccd45f47-75xwt" Nov 03 01:24:49.027993 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:24:49.027964 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8ea9e0d9-99b8-4bbb-b98d-b459f1560c89-istio-envoy\") pod \"e2e-inf-b-ce04af-openshift-default-7dccd45f47-75xwt\" (UID: \"8ea9e0d9-99b8-4bbb-b98d-b459f1560c89\") " pod="openshift-ingress/e2e-inf-b-ce04af-openshift-default-7dccd45f47-75xwt" Nov 03 01:24:49.028099 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:24:49.028035 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8ea9e0d9-99b8-4bbb-b98d-b459f1560c89-istio-podinfo\") pod \"e2e-inf-b-ce04af-openshift-default-7dccd45f47-75xwt\" (UID: \"8ea9e0d9-99b8-4bbb-b98d-b459f1560c89\") " pod="openshift-ingress/e2e-inf-b-ce04af-openshift-default-7dccd45f47-75xwt" Nov 03 01:24:49.034371 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:24:49.034344 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8ea9e0d9-99b8-4bbb-b98d-b459f1560c89-istio-token\") pod \"e2e-inf-b-ce04af-openshift-default-7dccd45f47-75xwt\" (UID: \"8ea9e0d9-99b8-4bbb-b98d-b459f1560c89\") " pod="openshift-ingress/e2e-inf-b-ce04af-openshift-default-7dccd45f47-75xwt" Nov 03 01:24:49.034472 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:24:49.034456 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zdrp9\" (UniqueName: \"kubernetes.io/projected/8ea9e0d9-99b8-4bbb-b98d-b459f1560c89-kube-api-access-zdrp9\") pod \"e2e-inf-b-ce04af-openshift-default-7dccd45f47-75xwt\" (UID: \"8ea9e0d9-99b8-4bbb-b98d-b459f1560c89\") " pod="openshift-ingress/e2e-inf-b-ce04af-openshift-default-7dccd45f47-75xwt" Nov 03 01:24:49.097524 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:24:49.097499 4939 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-ce04af-openshift-default-7dccd45f47-75xwt" Nov 03 01:24:49.119994 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:24:49.119963 4939 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-e1f08d/payload-processing-external-model-e2e-extmod-b-e1f08d-9f5cms4gl"] Nov 03 01:24:49.125542 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:24:49.125515 4939 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-b-e1f08d/payload-processing-external-model-e2e-extmod-b-e1f08d-9f5cms4gl"] Nov 03 01:24:49.180287 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:24:49.178150 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:24:49.332150 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:24:49.332120 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:24:49.399211 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:24:49.399175 4939 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-ce04af-openshift-default-7dccd45f47-75xwt"] Nov 03 01:24:49.399346 ip-10-0-140-107 kubenswrapper[4939]: W1103 01:24:49.399253 4939 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8ea9e0d9_99b8_4bbb_b98d_b459f1560c89.slice/crio-59283d2f8f702348becd13724b6a7f6ebc5a39c3491493e19cadb9b5981eea90 WatchSource:0}: Error finding container 59283d2f8f702348becd13724b6a7f6ebc5a39c3491493e19cadb9b5981eea90: Status 404 returned error can't find the container with id 59283d2f8f702348becd13724b6a7f6ebc5a39c3491493e19cadb9b5981eea90 Nov 03 01:24:49.402576 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:24:49.402541 4939 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 01:24:49.402670 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:24:49.402609 4939 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 01:24:49.402670 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:24:49.402640 4939 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 01:24:49.736031 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:24:49.735902 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:24:49.801010 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:24:49.800955 4939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-ce04af-openshift-default-7dccd45f47-75xwt" event={"ID":"8ea9e0d9-99b8-4bbb-b98d-b459f1560c89","Type":"ContainerStarted","Data":"a24f21dac1517c1ad57ee59169b7a7401d3a8349983a274d17f83f89201f147c"} Nov 03 01:24:49.801211 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:24:49.801029 4939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-ce04af-openshift-default-7dccd45f47-75xwt" event={"ID":"8ea9e0d9-99b8-4bbb-b98d-b459f1560c89","Type":"ContainerStarted","Data":"59283d2f8f702348becd13724b6a7f6ebc5a39c3491493e19cadb9b5981eea90"} Nov 03 01:24:49.829845 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:24:49.829795 4939 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-b-ce04af-openshift-default-7dccd45f47-75xwt" podStartSLOduration=9.829780297 podStartE2EDuration="9.829780297s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 01:24:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 01:24:49.826035493 +0000 UTC m=+2638.090370462" watchObservedRunningTime="2026-11-03 01:24:49.829780297 +0000 UTC m=+2638.094115241" Nov 03 01:24:49.852760 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:24:49.852727 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:24:49.872098 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:24:49.872067 4939 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0a9abcbc-bbe0-4f43-ae83-7e2ec3bc9d13" path="/var/lib/kubelet/pods/0a9abcbc-bbe0-4f43-ae83-7e2ec3bc9d13/volumes" Nov 03 01:24:49.872700 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:24:49.872684 4939 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8c6f8abe-dd67-45b9-b043-1c1e13ee96de" path="/var/lib/kubelet/pods/8c6f8abe-dd67-45b9-b043-1c1e13ee96de/volumes" Nov 03 01:24:49.983854 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:24:49.983775 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:24:50.102929 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:24:50.099944 4939 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-ce04af-openshift-default-7dccd45f47-75xwt" Nov 03 01:24:50.102929 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:24:50.100117 4939 patch_prober.go:29] interesting pod/e2e-inf-b-ce04af-openshift-default-7dccd45f47-75xwt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.43:15021/healthz/ready\": dial tcp 10.132.1.43:15021: connect: connection refused" start-of-body= Nov 03 01:24:50.102929 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:24:50.100167 4939 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-ce04af-openshift-default-7dccd45f47-75xwt" podUID="8ea9e0d9-99b8-4bbb-b98d-b459f1560c89" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.43:15021/healthz/ready\": dial tcp 10.132.1.43:15021: connect: connection refused" Nov 03 01:24:50.816966 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:24:50.816929 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:24:51.098272 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:24:51.098242 4939 patch_prober.go:29] interesting pod/e2e-inf-b-ce04af-openshift-default-7dccd45f47-75xwt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.43:15021/healthz/ready\": dial tcp 10.132.1.43:15021: connect: connection refused" start-of-body= Nov 03 01:24:51.098461 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:24:51.098293 4939 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-ce04af-openshift-default-7dccd45f47-75xwt" podUID="8ea9e0d9-99b8-4bbb-b98d-b459f1560c89" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.43:15021/healthz/ready\": dial tcp 10.132.1.43:15021: connect: connection refused" Nov 03 01:24:51.580394 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:24:51.580294 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:24:51.788319 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:24:51.788270 4939 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-a-821f5a/payload-processing-external-model-e2e-extmod-a-821f5a-5df8cj84n" podUID="5806cd37-5cdf-4afe-aa0b-c16b09db3106" containerName="payload-processing" containerID="cri-o://a72d9e520390deaf772ad83182d18514d7c6bf327f1f12e4176ad76fcc1732bc" gracePeriod=25 Nov 03 01:24:52.097973 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:24:52.097895 4939 patch_prober.go:29] interesting pod/e2e-inf-b-ce04af-openshift-default-7dccd45f47-75xwt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.43:15021/healthz/ready\": dial tcp 10.132.1.43:15021: connect: connection refused" start-of-body= Nov 03 01:24:52.098248 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:24:52.098220 4939 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-ce04af-openshift-default-7dccd45f47-75xwt" podUID="8ea9e0d9-99b8-4bbb-b98d-b459f1560c89" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.43:15021/healthz/ready\": dial tcp 10.132.1.43:15021: connect: connection refused" Nov 03 01:24:52.211111 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:24:52.211031 4939 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-821f5a/payload-processing-external-model-e2e-extmod-a-821f5a-5df8cj84n" Nov 03 01:24:52.354941 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:24:52.348567 4939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5806cd37-5cdf-4afe-aa0b-c16b09db3106-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/5806cd37-5cdf-4afe-aa0b-c16b09db3106-routing-overlay-volume\") pod \"5806cd37-5cdf-4afe-aa0b-c16b09db3106\" (UID: \"5806cd37-5cdf-4afe-aa0b-c16b09db3106\") " Nov 03 01:24:52.354941 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:24:52.349044 4939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5806cd37-5cdf-4afe-aa0b-c16b09db3106-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5806cd37-5cdf-4afe-aa0b-c16b09db3106-plugins-config-volume\") pod \"5806cd37-5cdf-4afe-aa0b-c16b09db3106\" (UID: \"5806cd37-5cdf-4afe-aa0b-c16b09db3106\") " Nov 03 01:24:52.354941 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:24:52.349146 4939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5806cd37-5cdf-4afe-aa0b-c16b09db3106-kube-api-access-rgrzx\" (UniqueName: \"kubernetes.io/projected/5806cd37-5cdf-4afe-aa0b-c16b09db3106-kube-api-access-rgrzx\") pod \"5806cd37-5cdf-4afe-aa0b-c16b09db3106\" (UID: \"5806cd37-5cdf-4afe-aa0b-c16b09db3106\") " Nov 03 01:24:52.354941 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:24:52.349217 4939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5806cd37-5cdf-4afe-aa0b-c16b09db3106-provider-credentials\" (UniqueName: \"kubernetes.io/projected/5806cd37-5cdf-4afe-aa0b-c16b09db3106-provider-credentials\") pod \"5806cd37-5cdf-4afe-aa0b-c16b09db3106\" (UID: \"5806cd37-5cdf-4afe-aa0b-c16b09db3106\") " Nov 03 01:24:52.371929 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:24:52.371192 4939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5806cd37-5cdf-4afe-aa0b-c16b09db3106-plugins-config-volume" pod "5806cd37-5cdf-4afe-aa0b-c16b09db3106" (UID: "5806cd37-5cdf-4afe-aa0b-c16b09db3106"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 01:24:52.379943 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:24:52.376375 4939 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5806cd37-5cdf-4afe-aa0b-c16b09db3106-plugins-config-volume\") on node \"ip-10-0-140-107.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:24:52.406932 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:24:52.388194 4939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5806cd37-5cdf-4afe-aa0b-c16b09db3106-routing-overlay-volume" pod "5806cd37-5cdf-4afe-aa0b-c16b09db3106" (UID: "5806cd37-5cdf-4afe-aa0b-c16b09db3106"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 01:24:52.412258 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:24:52.412209 4939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5806cd37-5cdf-4afe-aa0b-c16b09db3106-kube-api-access-rgrzx" pod "5806cd37-5cdf-4afe-aa0b-c16b09db3106" (UID: "5806cd37-5cdf-4afe-aa0b-c16b09db3106"). InnerVolumeSpecName "kube-api-access-rgrzx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 01:24:52.414999 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:24:52.414089 4939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5806cd37-5cdf-4afe-aa0b-c16b09db3106-provider-credentials" pod "5806cd37-5cdf-4afe-aa0b-c16b09db3106" (UID: "5806cd37-5cdf-4afe-aa0b-c16b09db3106"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 01:24:52.423978 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:24:52.420577 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:24:52.479511 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:24:52.479418 4939 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/5806cd37-5cdf-4afe-aa0b-c16b09db3106-provider-credentials\") on node \"ip-10-0-140-107.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:24:52.479511 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:24:52.479453 4939 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/5806cd37-5cdf-4afe-aa0b-c16b09db3106-routing-overlay-volume\") on node \"ip-10-0-140-107.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:24:52.479511 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:24:52.479471 4939 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rgrzx\" (UniqueName: \"kubernetes.io/projected/5806cd37-5cdf-4afe-aa0b-c16b09db3106-kube-api-access-rgrzx\") on node \"ip-10-0-140-107.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:24:52.836365 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:24:52.836329 4939 generic.go:356] "Generic (PLEG): container finished" podID="5806cd37-5cdf-4afe-aa0b-c16b09db3106" containerID="a72d9e520390deaf772ad83182d18514d7c6bf327f1f12e4176ad76fcc1732bc" exitCode=0 Nov 03 01:24:52.836813 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:24:52.836373 4939 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-821f5a/payload-processing-external-model-e2e-extmod-a-821f5a-5df8cj84n" Nov 03 01:24:52.836813 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:24:52.836411 4939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-821f5a/payload-processing-external-model-e2e-extmod-a-821f5a-5df8cj84n" event={"ID":"5806cd37-5cdf-4afe-aa0b-c16b09db3106","Type":"ContainerDied","Data":"a72d9e520390deaf772ad83182d18514d7c6bf327f1f12e4176ad76fcc1732bc"} Nov 03 01:24:52.836813 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:24:52.836490 4939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-821f5a/payload-processing-external-model-e2e-extmod-a-821f5a-5df8cj84n" event={"ID":"5806cd37-5cdf-4afe-aa0b-c16b09db3106","Type":"ContainerDied","Data":"2e7d75f0f17b35e331da5703bde1edff059e4e766607cb62d0c70b0afe8e6464"} Nov 03 01:24:52.836813 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:24:52.836582 4939 scope.go:122] "RemoveContainer" containerID="a72d9e520390deaf772ad83182d18514d7c6bf327f1f12e4176ad76fcc1732bc" Nov 03 01:24:52.872957 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:24:52.872467 4939 scope.go:122] "RemoveContainer" containerID="a72d9e520390deaf772ad83182d18514d7c6bf327f1f12e4176ad76fcc1732bc" Nov 03 01:24:52.873543 ip-10-0-140-107 kubenswrapper[4939]: E1103 01:24:52.873474 4939 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a72d9e520390deaf772ad83182d18514d7c6bf327f1f12e4176ad76fcc1732bc\": container with ID starting with a72d9e520390deaf772ad83182d18514d7c6bf327f1f12e4176ad76fcc1732bc not found: ID does not exist" containerID="a72d9e520390deaf772ad83182d18514d7c6bf327f1f12e4176ad76fcc1732bc" Nov 03 01:24:52.873653 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:24:52.873560 4939 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a72d9e520390deaf772ad83182d18514d7c6bf327f1f12e4176ad76fcc1732bc"} err="failed to get container status \"a72d9e520390deaf772ad83182d18514d7c6bf327f1f12e4176ad76fcc1732bc\": rpc error: code = NotFound desc = could not find container \"a72d9e520390deaf772ad83182d18514d7c6bf327f1f12e4176ad76fcc1732bc\": container with ID starting with a72d9e520390deaf772ad83182d18514d7c6bf327f1f12e4176ad76fcc1732bc not found: ID does not exist" Nov 03 01:24:52.889989 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:24:52.889961 4939 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-821f5a/payload-processing-external-model-e2e-extmod-a-821f5a-5df8cj84n"] Nov 03 01:24:52.895004 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:24:52.894981 4939 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-a-821f5a/payload-processing-external-model-e2e-extmod-a-821f5a-5df8cj84n"] Nov 03 01:24:53.113986 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:24:53.107017 4939 patch_prober.go:29] interesting pod/e2e-inf-b-ce04af-openshift-default-7dccd45f47-75xwt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.43:15021/healthz/ready\": dial tcp 10.132.1.43:15021: connect: connection refused" start-of-body= Nov 03 01:24:53.113986 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:24:53.107080 4939 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-ce04af-openshift-default-7dccd45f47-75xwt" podUID="8ea9e0d9-99b8-4bbb-b98d-b459f1560c89" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.43:15021/healthz/ready\": dial tcp 10.132.1.43:15021: connect: connection refused" Nov 03 01:24:53.220944 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:24:53.214537 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:24:53.414893 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:24:53.414807 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:24:53.533559 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:24:53.533526 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:24:53.660936 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:24:53.658773 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:24:53.846594 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:24:53.846546 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:24:54.002950 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:24:54.002875 4939 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5806cd37-5cdf-4afe-aa0b-c16b09db3106" path="/var/lib/kubelet/pods/5806cd37-5cdf-4afe-aa0b-c16b09db3106/volumes" Nov 03 01:24:54.098575 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:24:54.098493 4939 patch_prober.go:29] interesting pod/e2e-inf-b-ce04af-openshift-default-7dccd45f47-75xwt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.43:15021/healthz/ready\": dial tcp 10.132.1.43:15021: connect: connection refused" start-of-body= Nov 03 01:24:54.098575 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:24:54.098551 4939 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-ce04af-openshift-default-7dccd45f47-75xwt" podUID="8ea9e0d9-99b8-4bbb-b98d-b459f1560c89" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.43:15021/healthz/ready\": dial tcp 10.132.1.43:15021: connect: connection refused" Nov 03 01:24:54.136296 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:24:54.136266 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:24:54.375353 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:24:54.372456 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:24:54.680838 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:24:54.680704 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:24:54.836004 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:24:54.828077 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:24:54.945192 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:24:54.945103 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:24:55.098284 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:24:55.098252 4939 patch_prober.go:29] interesting pod/e2e-inf-b-ce04af-openshift-default-7dccd45f47-75xwt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.43:15021/healthz/ready\": dial tcp 10.132.1.43:15021: connect: connection refused" start-of-body= Nov 03 01:24:55.098469 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:24:55.098329 4939 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-ce04af-openshift-default-7dccd45f47-75xwt" podUID="8ea9e0d9-99b8-4bbb-b98d-b459f1560c89" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.43:15021/healthz/ready\": dial tcp 10.132.1.43:15021: connect: connection refused" Nov 03 01:24:55.685902 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:24:55.685868 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:24:55.818179 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:24:55.818145 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:24:55.915762 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:24:55.915724 4939 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-a-821f5a/payload-processing-external-model-e2e-extmod-a-821f5a-b486v68hp"] Nov 03 01:24:55.917129 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:24:55.917105 4939 cpu_manager.go:414] "containerMap: removing container" podUID="5806cd37-5cdf-4afe-aa0b-c16b09db3106" containerName="payload-processing" Nov 03 01:24:55.917129 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:24:55.917125 4939 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5806cd37-5cdf-4afe-aa0b-c16b09db3106" containerName="payload-processing" Nov 03 01:24:55.917303 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:24:55.917271 4939 memory_manager.go:365] "RemoveStaleState removing state" podUID="5806cd37-5cdf-4afe-aa0b-c16b09db3106" containerName="payload-processing" Nov 03 01:24:55.922972 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:24:55.922951 4939 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-821f5a/payload-processing-external-model-e2e-extmod-a-821f5a-b486v68hp" Nov 03 01:24:55.930442 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:24:55.930415 4939 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-821f5a\"/\"payload-processing-external-model-e2e--1074f2f6-dockercfg-m2rb7\"" Nov 03 01:24:55.930442 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:24:55.930427 4939 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-821f5a\"/\"agc-cred-066bb86b720d\"" Nov 03 01:24:55.931433 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:24:55.931411 4939 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-821f5a/payload-processing-external-model-e2e-extmod-a-821f5a-b486v68hp"] Nov 03 01:24:55.932986 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:24:55.932963 4939 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-821f5a\"/\"kube-root-ca.crt\"" Nov 03 01:24:55.933100 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:24:55.933014 4939 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-821f5a\"/\"routing-overlay\"" Nov 03 01:24:55.933100 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:24:55.933063 4939 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-821f5a\"/\"openshift-service-ca.crt\"" Nov 03 01:24:55.933100 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:24:55.933091 4939 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-821f5a\"/\"payload-processing-external-model-plugins-e2e-extmod-a-821f5a\"" Nov 03 01:24:55.982816 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:24:55.982725 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:24:56.031195 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:24:56.031161 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cba7757f-1340-4cf1-be98-b588dbb29fa0-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-821f5a-b486v68hp\" (UID: \"cba7757f-1340-4cf1-be98-b588dbb29fa0\") " pod="ai-tenant-e2e-extmod-a-821f5a/payload-processing-external-model-e2e-extmod-a-821f5a-b486v68hp" Nov 03 01:24:56.031394 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:24:56.031373 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jgg25\" (UniqueName: \"kubernetes.io/projected/cba7757f-1340-4cf1-be98-b588dbb29fa0-kube-api-access-jgg25\") pod \"payload-processing-external-model-e2e-extmod-a-821f5a-b486v68hp\" (UID: \"cba7757f-1340-4cf1-be98-b588dbb29fa0\") " pod="ai-tenant-e2e-extmod-a-821f5a/payload-processing-external-model-e2e-extmod-a-821f5a-b486v68hp" Nov 03 01:24:56.031475 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:24:56.031461 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/cba7757f-1340-4cf1-be98-b588dbb29fa0-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-821f5a-b486v68hp\" (UID: \"cba7757f-1340-4cf1-be98-b588dbb29fa0\") " pod="ai-tenant-e2e-extmod-a-821f5a/payload-processing-external-model-e2e-extmod-a-821f5a-b486v68hp" Nov 03 01:24:56.031568 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:24:56.031554 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/cba7757f-1340-4cf1-be98-b588dbb29fa0-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-821f5a-b486v68hp\" (UID: \"cba7757f-1340-4cf1-be98-b588dbb29fa0\") " pod="ai-tenant-e2e-extmod-a-821f5a/payload-processing-external-model-e2e-extmod-a-821f5a-b486v68hp" Nov 03 01:24:56.099021 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:24:56.098983 4939 patch_prober.go:29] interesting pod/e2e-inf-b-ce04af-openshift-default-7dccd45f47-75xwt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.43:15021/healthz/ready\": dial tcp 10.132.1.43:15021: connect: connection refused" start-of-body= Nov 03 01:24:56.099202 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:24:56.099039 4939 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-ce04af-openshift-default-7dccd45f47-75xwt" podUID="8ea9e0d9-99b8-4bbb-b98d-b459f1560c89" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.43:15021/healthz/ready\": dial tcp 10.132.1.43:15021: connect: connection refused" Nov 03 01:24:56.172840 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:24:56.172787 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jgg25\" (UniqueName: \"kubernetes.io/projected/cba7757f-1340-4cf1-be98-b588dbb29fa0-kube-api-access-jgg25\") pod \"payload-processing-external-model-e2e-extmod-a-821f5a-b486v68hp\" (UID: \"cba7757f-1340-4cf1-be98-b588dbb29fa0\") " pod="ai-tenant-e2e-extmod-a-821f5a/payload-processing-external-model-e2e-extmod-a-821f5a-b486v68hp" Nov 03 01:24:56.173610 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:24:56.173582 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/cba7757f-1340-4cf1-be98-b588dbb29fa0-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-821f5a-b486v68hp\" (UID: \"cba7757f-1340-4cf1-be98-b588dbb29fa0\") " pod="ai-tenant-e2e-extmod-a-821f5a/payload-processing-external-model-e2e-extmod-a-821f5a-b486v68hp" Nov 03 01:24:56.174363 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:24:56.174340 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/cba7757f-1340-4cf1-be98-b588dbb29fa0-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-821f5a-b486v68hp\" (UID: \"cba7757f-1340-4cf1-be98-b588dbb29fa0\") " pod="ai-tenant-e2e-extmod-a-821f5a/payload-processing-external-model-e2e-extmod-a-821f5a-b486v68hp" Nov 03 01:24:56.189080 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:24:56.189045 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cba7757f-1340-4cf1-be98-b588dbb29fa0-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-821f5a-b486v68hp\" (UID: \"cba7757f-1340-4cf1-be98-b588dbb29fa0\") " pod="ai-tenant-e2e-extmod-a-821f5a/payload-processing-external-model-e2e-extmod-a-821f5a-b486v68hp" Nov 03 01:24:56.190473 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:24:56.190076 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/cba7757f-1340-4cf1-be98-b588dbb29fa0-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-821f5a-b486v68hp\" (UID: \"cba7757f-1340-4cf1-be98-b588dbb29fa0\") " pod="ai-tenant-e2e-extmod-a-821f5a/payload-processing-external-model-e2e-extmod-a-821f5a-b486v68hp" Nov 03 01:24:56.196932 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:24:56.190816 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cba7757f-1340-4cf1-be98-b588dbb29fa0-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-821f5a-b486v68hp\" (UID: \"cba7757f-1340-4cf1-be98-b588dbb29fa0\") " pod="ai-tenant-e2e-extmod-a-821f5a/payload-processing-external-model-e2e-extmod-a-821f5a-b486v68hp" Nov 03 01:24:56.196932 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:24:56.191620 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jgg25\" (UniqueName: \"kubernetes.io/projected/cba7757f-1340-4cf1-be98-b588dbb29fa0-kube-api-access-jgg25\") pod \"payload-processing-external-model-e2e-extmod-a-821f5a-b486v68hp\" (UID: \"cba7757f-1340-4cf1-be98-b588dbb29fa0\") " pod="ai-tenant-e2e-extmod-a-821f5a/payload-processing-external-model-e2e-extmod-a-821f5a-b486v68hp" Nov 03 01:24:56.231984 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:24:56.231945 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/cba7757f-1340-4cf1-be98-b588dbb29fa0-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-821f5a-b486v68hp\" (UID: \"cba7757f-1340-4cf1-be98-b588dbb29fa0\") " pod="ai-tenant-e2e-extmod-a-821f5a/payload-processing-external-model-e2e-extmod-a-821f5a-b486v68hp" Nov 03 01:24:56.253005 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:24:56.245139 4939 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-821f5a/payload-processing-external-model-e2e-extmod-a-821f5a-b486v68hp" Nov 03 01:24:56.693205 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:24:56.692990 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:24:56.815018 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:24:56.814975 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:24:57.125964 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:24:57.106661 4939 patch_prober.go:29] interesting pod/e2e-inf-b-ce04af-openshift-default-7dccd45f47-75xwt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.43:15021/healthz/ready\": dial tcp 10.132.1.43:15021: connect: connection refused" start-of-body= Nov 03 01:24:57.125964 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:24:57.106737 4939 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-ce04af-openshift-default-7dccd45f47-75xwt" podUID="8ea9e0d9-99b8-4bbb-b98d-b459f1560c89" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.43:15021/healthz/ready\": dial tcp 10.132.1.43:15021: connect: connection refused" Nov 03 01:24:57.129472 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:24:57.129439 4939 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-821f5a/payload-processing-external-model-e2e-extmod-a-821f5a-b486v68hp"] Nov 03 01:24:57.239935 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:24:57.219596 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:24:57.454720 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:24:57.454593 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:24:57.938371 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:24:57.899251 4939 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-2f9a3a/simulator-w0-2f9a3a-kserve-9596bbb8f-2trgz" podUID="207e2207-fb6c-4076-95df-72e95671c261" containerName="main" probeResult="failure" output="Get \"https://10.132.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 01:24:57.998121 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:24:57.917997 4939 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-2f9a3a/distinct-w0-2f9a3a-kserve-555565cfcf-jtcr9" podUID="ac0ece06-8718-463f-915f-5315988d9711" containerName="main" probeResult="failure" output="Get \"https://10.132.0.76:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 01:24:57.998483 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:24:57.966804 4939 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-f7680d/distinct-2-w2-f7680d-kserve-7d7c4564bf-cbv2c" podUID="2de18086-761b-4773-b9fc-15ddc4a220fc" containerName="main" probeResult="failure" output="Get \"https://10.132.0.75:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 01:24:58.026445 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:24:58.026401 4939 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-f7680d/distinct-w2-f7680d-kserve-756f874b98-qnpv8" podUID="836753fc-f9ef-4f61-b3fe-e5b92b7ee27b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.70:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 01:24:58.049119 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:24:58.047830 4939 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-83755a/distinct-2-w3-83755a-kserve-6cf66b995d-2l47b" podUID="c3b05e88-5d5b-4d03-b28d-832f22e3c868" containerName="main" probeResult="failure" output="Get \"https://10.132.0.72:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 01:24:58.049119 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:24:58.048249 4939 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-f7680d/premium-w2-f7680d-kserve-76565b8cf9-hqbbt" podUID="4712e708-18a2-4a13-a12c-a5669397b2c3" containerName="main" probeResult="failure" output="Get \"https://10.132.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 01:24:58.051184 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:24:58.051137 4939 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-f7680d/distinct-w2-f7680d-kserve-756f874b98-qnpv8" podUID="836753fc-f9ef-4f61-b3fe-e5b92b7ee27b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.70:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 01:24:58.070034 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:24:58.069982 4939 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-b-ce04af/test-model-ce04af-kserve-84bf7f6d7c-qd56c" podUID="72db0238-20cc-40cb-affb-158746adb87b" containerName="main" probeResult="failure" output="Get \"https://10.132.1.39:8000/ready\": context deadline exceeded" Nov 03 01:24:58.126011 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:24:58.125884 4939 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-88bbf7/simulator-w1-88bbf7-kserve-5fbb669f8f-42hgd" podUID="83b38883-2e7a-444d-aa68-52be4a25ae66" containerName="main" probeResult="failure" output="Get \"https://10.132.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 01:24:58.168649 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:24:58.168605 4939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-821f5a/payload-processing-external-model-e2e-extmod-a-821f5a-b486v68hp" event={"ID":"cba7757f-1340-4cf1-be98-b588dbb29fa0","Type":"ContainerStarted","Data":"d4697be95f5691e9f7c9dcb9c20e94ad717df0fabd94623b53db37022836deb7"} Nov 03 01:24:58.191432 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:24:58.191406 4939 scope.go:122] "RemoveContainer" containerID="1192511cd77038a4ec725b9a5e3362d4ff11ede6418eeb10fc882c79aba91ff3" Nov 03 01:24:58.319880 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:24:58.319846 4939 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-ce04af-openshift-default-7dccd45f47-75xwt" Nov 03 01:24:58.320476 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:24:58.320451 4939 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-ce04af-openshift-default-7dccd45f47-75xwt" Nov 03 01:24:58.321280 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:24:58.321257 4939 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-ce04af-openshift-default-7dccd45f47-75xwt" Nov 03 01:24:58.325467 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:24:58.325442 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:24:58.982250 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:24:58.982214 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:24:59.179999 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:24:59.179961 4939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-821f5a/payload-processing-external-model-e2e-extmod-a-821f5a-b486v68hp" event={"ID":"cba7757f-1340-4cf1-be98-b588dbb29fa0","Type":"ContainerStarted","Data":"f1e4c50251bf52c46ec3d2391cec2a644a63074d1231bb7eff3dcc3c45378156"} Nov 03 01:24:59.180405 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:24:59.180057 4939 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-a-821f5a/payload-processing-external-model-e2e-extmod-a-821f5a-b486v68hp" Nov 03 01:24:59.200761 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:24:59.200707 4939 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-a-821f5a/payload-processing-external-model-e2e-extmod-a-821f5a-b486v68hp" podStartSLOduration=4.200691953 podStartE2EDuration="4.200691953s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 01:24:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 01:24:59.195995922 +0000 UTC m=+2647.460330864" watchObservedRunningTime="2026-11-03 01:24:59.200691953 +0000 UTC m=+2647.465026899" Nov 03 01:24:59.684279 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:24:59.684244 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:25:00.405404 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:25:00.405369 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:25:01.026795 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:25:01.026757 4939 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-08057cfd-vh5dj"] Nov 03 01:25:01.033992 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:25:01.033965 4939 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-08057cfd-vh5dj"] Nov 03 01:25:01.184896 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:25:01.184828 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:25:01.877705 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:25:01.877662 4939 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e0d5a8a0-e2bd-4339-be4f-8a059b229f52" path="/var/lib/kubelet/pods/e0d5a8a0-e2bd-4339-be4f-8a059b229f52/volumes" Nov 03 01:25:01.893546 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:25:01.893495 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:25:01.919833 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:25:01.919801 4939 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-821f5a/payload-processing-external-model-e2e-extmod-a-821f5a-b486v68hp"] Nov 03 01:25:02.143462 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:25:02.143378 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:25:02.294058 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:25:02.293974 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:25:02.739586 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:25:02.739553 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:25:02.849078 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:25:02.849047 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:25:03.479138 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:25:03.479107 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:25:04.236972 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:25:04.236940 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:25:04.879365 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:25:04.879333 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:25:05.601183 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:25:05.601148 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:25:06.171031 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:25:06.170999 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:25:06.774743 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:25:06.774710 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:25:06.920670 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:25:06.920629 4939 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-a-821f5a/payload-processing-external-model-e2e-extmod-a-821f5a-b486v68hp" podUID="cba7757f-1340-4cf1-be98-b588dbb29fa0" containerName="payload-processing" containerID="cri-o://f1e4c50251bf52c46ec3d2391cec2a644a63074d1231bb7eff3dcc3c45378156" gracePeriod=25 Nov 03 01:25:07.220280 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:25:07.220251 4939 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-821f5a/payload-processing-external-model-e2e-extmod-a-821f5a-b486v68hp" Nov 03 01:25:07.256508 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:25:07.256469 4939 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-ee618c-8bc685f5f-l65d9"] Nov 03 01:25:07.258011 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:25:07.257982 4939 cpu_manager.go:414] "containerMap: removing container" podUID="cba7757f-1340-4cf1-be98-b588dbb29fa0" containerName="payload-processing" Nov 03 01:25:07.258011 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:25:07.258008 4939 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cba7757f-1340-4cf1-be98-b588dbb29fa0" containerName="payload-processing" Nov 03 01:25:07.262995 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:25:07.258372 4939 memory_manager.go:365] "RemoveStaleState removing state" podUID="cba7757f-1340-4cf1-be98-b588dbb29fa0" containerName="payload-processing" Nov 03 01:25:07.272586 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:25:07.267965 4939 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-ee618c-8bc685f5f-l65d9"] Nov 03 01:25:07.272586 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:25:07.268059 4939 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-ee618c-8bc685f5f-l65d9" Nov 03 01:25:07.272586 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:25:07.272328 4939 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-a-ee618c\"" Nov 03 01:25:07.272586 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:25:07.272498 4939 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-a-ee618c-dockercfg-nmqxx\"" Nov 03 01:25:07.295694 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:25:07.295670 4939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cba7757f-1340-4cf1-be98-b588dbb29fa0-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cba7757f-1340-4cf1-be98-b588dbb29fa0-plugins-config-volume\") pod \"cba7757f-1340-4cf1-be98-b588dbb29fa0\" (UID: \"cba7757f-1340-4cf1-be98-b588dbb29fa0\") " Nov 03 01:25:07.295816 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:25:07.295716 4939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cba7757f-1340-4cf1-be98-b588dbb29fa0-kube-api-access-jgg25\" (UniqueName: \"kubernetes.io/projected/cba7757f-1340-4cf1-be98-b588dbb29fa0-kube-api-access-jgg25\") pod \"cba7757f-1340-4cf1-be98-b588dbb29fa0\" (UID: \"cba7757f-1340-4cf1-be98-b588dbb29fa0\") " Nov 03 01:25:07.296033 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:25:07.296020 4939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cba7757f-1340-4cf1-be98-b588dbb29fa0-provider-credentials\" (UniqueName: \"kubernetes.io/projected/cba7757f-1340-4cf1-be98-b588dbb29fa0-provider-credentials\") pod \"cba7757f-1340-4cf1-be98-b588dbb29fa0\" (UID: \"cba7757f-1340-4cf1-be98-b588dbb29fa0\") " Nov 03 01:25:07.296147 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:25:07.296130 4939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cba7757f-1340-4cf1-be98-b588dbb29fa0-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/cba7757f-1340-4cf1-be98-b588dbb29fa0-routing-overlay-volume\") pod \"cba7757f-1340-4cf1-be98-b588dbb29fa0\" (UID: \"cba7757f-1340-4cf1-be98-b588dbb29fa0\") " Nov 03 01:25:07.296237 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:25:07.296164 4939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cba7757f-1340-4cf1-be98-b588dbb29fa0-plugins-config-volume" pod "cba7757f-1340-4cf1-be98-b588dbb29fa0" (UID: "cba7757f-1340-4cf1-be98-b588dbb29fa0"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 01:25:07.296520 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:25:07.296494 4939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cba7757f-1340-4cf1-be98-b588dbb29fa0-routing-overlay-volume" pod "cba7757f-1340-4cf1-be98-b588dbb29fa0" (UID: "cba7757f-1340-4cf1-be98-b588dbb29fa0"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 01:25:07.297256 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:25:07.297227 4939 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/cba7757f-1340-4cf1-be98-b588dbb29fa0-routing-overlay-volume\") on node \"ip-10-0-140-107.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:25:07.297366 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:25:07.297260 4939 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cba7757f-1340-4cf1-be98-b588dbb29fa0-plugins-config-volume\") on node \"ip-10-0-140-107.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:25:07.298722 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:25:07.298699 4939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cba7757f-1340-4cf1-be98-b588dbb29fa0-kube-api-access-jgg25" pod "cba7757f-1340-4cf1-be98-b588dbb29fa0" (UID: "cba7757f-1340-4cf1-be98-b588dbb29fa0"). InnerVolumeSpecName "kube-api-access-jgg25". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 01:25:07.298816 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:25:07.298794 4939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cba7757f-1340-4cf1-be98-b588dbb29fa0-provider-credentials" pod "cba7757f-1340-4cf1-be98-b588dbb29fa0" (UID: "cba7757f-1340-4cf1-be98-b588dbb29fa0"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 01:25:07.382891 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:25:07.382847 4939 generic.go:356] "Generic (PLEG): container finished" podID="cba7757f-1340-4cf1-be98-b588dbb29fa0" containerID="f1e4c50251bf52c46ec3d2391cec2a644a63074d1231bb7eff3dcc3c45378156" exitCode=0 Nov 03 01:25:07.383130 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:25:07.382944 4939 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-821f5a/payload-processing-external-model-e2e-extmod-a-821f5a-b486v68hp" Nov 03 01:25:07.383130 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:25:07.382961 4939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-821f5a/payload-processing-external-model-e2e-extmod-a-821f5a-b486v68hp" event={"ID":"cba7757f-1340-4cf1-be98-b588dbb29fa0","Type":"ContainerDied","Data":"f1e4c50251bf52c46ec3d2391cec2a644a63074d1231bb7eff3dcc3c45378156"} Nov 03 01:25:07.383130 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:25:07.383051 4939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-821f5a/payload-processing-external-model-e2e-extmod-a-821f5a-b486v68hp" event={"ID":"cba7757f-1340-4cf1-be98-b588dbb29fa0","Type":"ContainerDied","Data":"d4697be95f5691e9f7c9dcb9c20e94ad717df0fabd94623b53db37022836deb7"} Nov 03 01:25:07.383130 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:25:07.383077 4939 scope.go:122] "RemoveContainer" containerID="f1e4c50251bf52c46ec3d2391cec2a644a63074d1231bb7eff3dcc3c45378156" Nov 03 01:25:07.397194 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:25:07.396704 4939 scope.go:122] "RemoveContainer" containerID="f1e4c50251bf52c46ec3d2391cec2a644a63074d1231bb7eff3dcc3c45378156" Nov 03 01:25:07.397194 ip-10-0-140-107 kubenswrapper[4939]: E1103 01:25:07.397045 4939 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f1e4c50251bf52c46ec3d2391cec2a644a63074d1231bb7eff3dcc3c45378156\": container with ID starting with f1e4c50251bf52c46ec3d2391cec2a644a63074d1231bb7eff3dcc3c45378156 not found: ID does not exist" containerID="f1e4c50251bf52c46ec3d2391cec2a644a63074d1231bb7eff3dcc3c45378156" Nov 03 01:25:07.397194 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:25:07.397070 4939 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f1e4c50251bf52c46ec3d2391cec2a644a63074d1231bb7eff3dcc3c45378156"} err="failed to get container status \"f1e4c50251bf52c46ec3d2391cec2a644a63074d1231bb7eff3dcc3c45378156\": rpc error: code = NotFound desc = could not find container \"f1e4c50251bf52c46ec3d2391cec2a644a63074d1231bb7eff3dcc3c45378156\": container with ID starting with f1e4c50251bf52c46ec3d2391cec2a644a63074d1231bb7eff3dcc3c45378156 not found: ID does not exist" Nov 03 01:25:07.400092 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:25:07.400069 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vjf69\" (UniqueName: \"kubernetes.io/projected/3d52a5d5-9404-4bdf-b6cb-9f134f145c6f-kube-api-access-vjf69\") pod \"payload-pre-processing-e2e-inf-a-ee618c-8bc685f5f-l65d9\" (UID: \"3d52a5d5-9404-4bdf-b6cb-9f134f145c6f\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-ee618c-8bc685f5f-l65d9" Nov 03 01:25:07.400229 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:25:07.400216 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3d52a5d5-9404-4bdf-b6cb-9f134f145c6f-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-ee618c-8bc685f5f-l65d9\" (UID: \"3d52a5d5-9404-4bdf-b6cb-9f134f145c6f\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-ee618c-8bc685f5f-l65d9" Nov 03 01:25:07.400408 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:25:07.400389 4939 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/cba7757f-1340-4cf1-be98-b588dbb29fa0-provider-credentials\") on node \"ip-10-0-140-107.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:25:07.400446 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:25:07.400412 4939 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jgg25\" (UniqueName: \"kubernetes.io/projected/cba7757f-1340-4cf1-be98-b588dbb29fa0-kube-api-access-jgg25\") on node \"ip-10-0-140-107.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:25:07.415749 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:25:07.415724 4939 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-821f5a/payload-processing-external-model-e2e-extmod-a-821f5a-b486v68hp"] Nov 03 01:25:07.420712 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:25:07.420680 4939 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-a-821f5a/payload-processing-external-model-e2e-extmod-a-821f5a-b486v68hp"] Nov 03 01:25:07.516932 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:25:07.502830 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vjf69\" (UniqueName: \"kubernetes.io/projected/3d52a5d5-9404-4bdf-b6cb-9f134f145c6f-kube-api-access-vjf69\") pod \"payload-pre-processing-e2e-inf-a-ee618c-8bc685f5f-l65d9\" (UID: \"3d52a5d5-9404-4bdf-b6cb-9f134f145c6f\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-ee618c-8bc685f5f-l65d9" Nov 03 01:25:07.516932 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:25:07.503445 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3d52a5d5-9404-4bdf-b6cb-9f134f145c6f-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-ee618c-8bc685f5f-l65d9\" (UID: \"3d52a5d5-9404-4bdf-b6cb-9f134f145c6f\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-ee618c-8bc685f5f-l65d9" Nov 03 01:25:07.516932 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:25:07.504862 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3d52a5d5-9404-4bdf-b6cb-9f134f145c6f-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-ee618c-8bc685f5f-l65d9\" (UID: \"3d52a5d5-9404-4bdf-b6cb-9f134f145c6f\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-ee618c-8bc685f5f-l65d9" Nov 03 01:25:07.551979 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:25:07.547713 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vjf69\" (UniqueName: \"kubernetes.io/projected/3d52a5d5-9404-4bdf-b6cb-9f134f145c6f-kube-api-access-vjf69\") pod \"payload-pre-processing-e2e-inf-a-ee618c-8bc685f5f-l65d9\" (UID: \"3d52a5d5-9404-4bdf-b6cb-9f134f145c6f\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-ee618c-8bc685f5f-l65d9" Nov 03 01:25:07.569968 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:25:07.567250 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:25:07.597057 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:25:07.597025 4939 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-ee618c-8bc685f5f-l65d9" Nov 03 01:25:07.873888 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:25:07.873852 4939 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cba7757f-1340-4cf1-be98-b588dbb29fa0" path="/var/lib/kubelet/pods/cba7757f-1340-4cf1-be98-b588dbb29fa0/volumes" Nov 03 01:25:07.881331 ip-10-0-140-107 kubenswrapper[4939]: W1103 01:25:07.881302 4939 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3d52a5d5_9404_4bdf_b6cb_9f134f145c6f.slice/crio-c2e55a208f55e41b02ea24625063b2fbb6289a409b1f725c24106d374acb7615 WatchSource:0}: Error finding container c2e55a208f55e41b02ea24625063b2fbb6289a409b1f725c24106d374acb7615: Status 404 returned error can't find the container with id c2e55a208f55e41b02ea24625063b2fbb6289a409b1f725c24106d374acb7615 Nov 03 01:25:07.882295 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:25:07.882271 4939 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-ee618c-8bc685f5f-l65d9"] Nov 03 01:25:08.093368 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:25:08.093329 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:25:08.395474 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:25:08.395440 4939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-ee618c-8bc685f5f-l65d9" event={"ID":"3d52a5d5-9404-4bdf-b6cb-9f134f145c6f","Type":"ContainerStarted","Data":"308485434bd7c443355f49edf1854b03d0c5193f0af258a6abd6cb492780c05a"} Nov 03 01:25:08.395880 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:25:08.395493 4939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-ee618c-8bc685f5f-l65d9" event={"ID":"3d52a5d5-9404-4bdf-b6cb-9f134f145c6f","Type":"ContainerStarted","Data":"c2e55a208f55e41b02ea24625063b2fbb6289a409b1f725c24106d374acb7615"} Nov 03 01:25:08.395880 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:25:08.395517 4939 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-ee618c-8bc685f5f-l65d9" Nov 03 01:25:08.415898 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:25:08.415843 4939 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-ee618c-8bc685f5f-l65d9" podStartSLOduration=11.41582954 podStartE2EDuration="11.41582954s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 01:24:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 01:25:08.411041164 +0000 UTC m=+2656.675376112" watchObservedRunningTime="2026-11-03 01:25:08.41582954 +0000 UTC m=+2656.680164485" Nov 03 01:25:08.886428 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:25:08.886384 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:25:09.022952 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:25:09.021151 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:25:09.359176 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:25:09.359143 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:25:09.726816 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:25:09.726733 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:25:09.851368 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:25:09.851335 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:25:10.042006 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:25:10.039464 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:25:10.663438 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:25:10.663403 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:25:11.306787 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:25:11.306355 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:25:11.915501 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:25:11.915464 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:25:12.657830 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:25:12.657794 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:25:13.369787 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:25:13.369750 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:25:14.076201 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:25:14.076168 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:25:14.772709 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:25:14.772672 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:25:15.397003 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:25:15.396963 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:25:16.056078 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:25:16.056047 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:25:16.695224 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:25:16.695187 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:25:17.477016 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:25:17.476982 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:25:17.630410 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:25:17.630373 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:25:18.035463 ip-10-0-140-107 kubenswrapper[4939]: E1103 01:25:18.035428 4939 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-sthjh" containerName="extractor" Nov 03 01:25:18.410384 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:25:18.410347 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:25:19.379276 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:25:19.379244 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:25:19.445752 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:25:19.445711 4939 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-ee618c-8bc685f5f-l65d9" Nov 03 01:25:20.072790 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:25:20.072750 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:25:20.785613 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:25:20.785577 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:25:21.389232 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:25:21.389196 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:25:22.066734 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:25:22.066677 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:25:22.683326 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:25:22.683295 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:25:23.290093 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:25:23.290062 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:25:23.973862 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:25:23.973828 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:25:24.582728 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:25:24.582691 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:25:25.186361 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:25:25.186323 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:25:25.793599 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:25:25.793560 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:25:26.386468 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:25:26.386427 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:25:27.063273 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:25:27.063240 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:25:27.691233 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:25:27.691193 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:25:28.372763 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:25:28.372732 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:25:29.000673 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:25:29.000636 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:25:29.585445 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:25:29.585407 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:25:30.239060 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:25:30.239026 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:25:30.974822 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:25:30.974789 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:25:31.770502 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:25:31.770460 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:25:32.375277 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:25:32.375242 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:25:33.142391 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:25:33.142358 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:25:33.778243 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:25:33.778205 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:25:34.541768 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:25:34.541733 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:25:35.148321 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:25:35.148291 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:25:35.898039 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:25:35.898006 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:25:36.556798 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:25:36.556755 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:25:37.178540 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:25:37.178501 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:25:37.902353 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:25:37.902316 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:25:38.572289 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:25:38.572251 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:25:39.180838 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:25:39.180799 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:25:39.882965 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:25:39.882930 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:25:40.717242 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:25:40.717202 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:25:41.306778 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:25:41.306742 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:25:41.998393 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:25:41.998359 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:25:42.658930 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:25:42.656071 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:25:43.267979 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:25:43.267940 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:25:43.908478 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:25:43.908445 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:25:44.689459 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:25:44.689425 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:25:44.805930 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:25:44.804732 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:25:45.307799 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:25:45.307765 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:25:45.720903 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:25:45.720865 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:25:45.846449 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:25:45.846412 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:25:46.256573 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:25:46.256539 4939 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-extmod-shr-user-08aa9e/e2e-usage-sim-92e6a0-5b4db7d754-db6qq"] Nov 03 01:25:46.256748 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:25:46.256719 4939 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-extmod-shr-user-08aa9e/e2e-usage-sim-92e6a0-5b4db7d754-db6qq" podUID="000f2bd4-56ac-4df8-8379-f4947877a0af" containerName="sim" containerID="cri-o://1671913d48fdf59fbdf6e36613f1cfb94407716fe0e869d94a3559f9d0a04ecd" gracePeriod=30 Nov 03 01:25:46.505934 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:25:46.499407 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:25:46.736949 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:25:46.708835 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:25:47.111205 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:25:47.111177 4939 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-extmod-shr-user-08aa9e/e2e-usage-sim-92e6a0-5b4db7d754-db6qq" Nov 03 01:25:47.153397 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:25:47.153358 4939 generic.go:356] "Generic (PLEG): container finished" podID="000f2bd4-56ac-4df8-8379-f4947877a0af" containerID="1671913d48fdf59fbdf6e36613f1cfb94407716fe0e869d94a3559f9d0a04ecd" exitCode=0 Nov 03 01:25:47.153583 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:25:47.153411 4939 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-extmod-shr-user-08aa9e/e2e-usage-sim-92e6a0-5b4db7d754-db6qq" Nov 03 01:25:47.153670 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:25:47.153639 4939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-extmod-shr-user-08aa9e/e2e-usage-sim-92e6a0-5b4db7d754-db6qq" event={"ID":"000f2bd4-56ac-4df8-8379-f4947877a0af","Type":"ContainerDied","Data":"1671913d48fdf59fbdf6e36613f1cfb94407716fe0e869d94a3559f9d0a04ecd"} Nov 03 01:25:47.153733 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:25:47.153672 4939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-extmod-shr-user-08aa9e/e2e-usage-sim-92e6a0-5b4db7d754-db6qq" event={"ID":"000f2bd4-56ac-4df8-8379-f4947877a0af","Type":"ContainerDied","Data":"30e2d0595348f3f55b2ebb11a9597518200ee082c7cde69c9804bca0d5c19f43"} Nov 03 01:25:47.153733 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:25:47.153688 4939 scope.go:122] "RemoveContainer" containerID="1671913d48fdf59fbdf6e36613f1cfb94407716fe0e869d94a3559f9d0a04ecd" Nov 03 01:25:47.166800 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:25:47.166771 4939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/000f2bd4-56ac-4df8-8379-f4947877a0af-kube-api-access-pknng\" (UniqueName: \"kubernetes.io/projected/000f2bd4-56ac-4df8-8379-f4947877a0af-kube-api-access-pknng\") pod \"000f2bd4-56ac-4df8-8379-f4947877a0af\" (UID: \"000f2bd4-56ac-4df8-8379-f4947877a0af\") " Nov 03 01:25:47.167125 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:25:47.167097 4939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/000f2bd4-56ac-4df8-8379-f4947877a0af-tls\" (UniqueName: \"kubernetes.io/secret/000f2bd4-56ac-4df8-8379-f4947877a0af-tls\") pod \"000f2bd4-56ac-4df8-8379-f4947877a0af\" (UID: \"000f2bd4-56ac-4df8-8379-f4947877a0af\") " Nov 03 01:25:47.167783 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:25:47.167760 4939 scope.go:122] "RemoveContainer" containerID="1671913d48fdf59fbdf6e36613f1cfb94407716fe0e869d94a3559f9d0a04ecd" Nov 03 01:25:47.168193 ip-10-0-140-107 kubenswrapper[4939]: E1103 01:25:47.168157 4939 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1671913d48fdf59fbdf6e36613f1cfb94407716fe0e869d94a3559f9d0a04ecd\": container with ID starting with 1671913d48fdf59fbdf6e36613f1cfb94407716fe0e869d94a3559f9d0a04ecd not found: ID does not exist" containerID="1671913d48fdf59fbdf6e36613f1cfb94407716fe0e869d94a3559f9d0a04ecd" Nov 03 01:25:47.168296 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:25:47.168197 4939 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1671913d48fdf59fbdf6e36613f1cfb94407716fe0e869d94a3559f9d0a04ecd"} err="failed to get container status \"1671913d48fdf59fbdf6e36613f1cfb94407716fe0e869d94a3559f9d0a04ecd\": rpc error: code = NotFound desc = could not find container \"1671913d48fdf59fbdf6e36613f1cfb94407716fe0e869d94a3559f9d0a04ecd\": container with ID starting with 1671913d48fdf59fbdf6e36613f1cfb94407716fe0e869d94a3559f9d0a04ecd not found: ID does not exist" Nov 03 01:25:47.169408 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:25:47.169381 4939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/000f2bd4-56ac-4df8-8379-f4947877a0af-kube-api-access-pknng" pod "000f2bd4-56ac-4df8-8379-f4947877a0af" (UID: "000f2bd4-56ac-4df8-8379-f4947877a0af"). InnerVolumeSpecName "kube-api-access-pknng". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 01:25:47.169603 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:25:47.169584 4939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/000f2bd4-56ac-4df8-8379-f4947877a0af-tls" pod "000f2bd4-56ac-4df8-8379-f4947877a0af" (UID: "000f2bd4-56ac-4df8-8379-f4947877a0af"). InnerVolumeSpecName "tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 01:25:47.269509 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:25:47.269475 4939 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pknng\" (UniqueName: \"kubernetes.io/projected/000f2bd4-56ac-4df8-8379-f4947877a0af-kube-api-access-pknng\") on node \"ip-10-0-140-107.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:25:47.269509 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:25:47.269504 4939 reconciler_common.go:299] "Volume detached for volume \"tls\" (UniqueName: \"kubernetes.io/secret/000f2bd4-56ac-4df8-8379-f4947877a0af-tls\") on node \"ip-10-0-140-107.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:25:47.485553 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:25:47.485514 4939 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-extmod-shr-user-08aa9e/e2e-usage-sim-92e6a0-5b4db7d754-db6qq"] Nov 03 01:25:47.490597 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:25:47.490566 4939 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-extmod-shr-user-08aa9e/e2e-usage-sim-92e6a0-5b4db7d754-db6qq"] Nov 03 01:25:47.597175 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:25:47.597143 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:25:47.773334 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:25:47.773261 4939 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-027c19/payload-processing-external-model-e2e-extmod-a-027c19-65568h8hw"] Nov 03 01:25:47.859043 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:25:47.859010 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:25:47.892707 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:25:47.892670 4939 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="000f2bd4-56ac-4df8-8379-f4947877a0af" path="/var/lib/kubelet/pods/000f2bd4-56ac-4df8-8379-f4947877a0af/volumes" Nov 03 01:25:48.004943 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:25:48.001680 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:25:48.323207 ip-10-0-140-107 kubenswrapper[4939]: E1103 01:25:48.323151 4939 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-sthjh" containerName="extractor" Nov 03 01:25:48.445722 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:25:48.445686 4939 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-3c5983/payload-processing-external-model-e2e-extmod-b-3c5983-9fbdwjsfd"] Nov 03 01:25:48.499261 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:25:48.499225 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:25:48.670934 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:25:48.667208 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:25:49.252604 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:25:49.252567 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:25:49.852780 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:25:49.852728 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:25:50.500474 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:25:50.500442 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:25:51.087439 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:25:51.087403 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:25:51.871743 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:25:51.871712 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:25:52.568637 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:25:52.568603 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:25:52.774401 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:25:52.774350 4939 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-a-027c19/payload-processing-external-model-e2e-extmod-a-027c19-65568h8hw" podUID="07a22731-ed88-4d35-be31-46351a8f5ce9" containerName="payload-processing" containerID="cri-o://5106c2789578a525217990f7bffe265d8c0aaf6ece11c31890a51ddeace0e43f" gracePeriod=25 Nov 03 01:25:52.829948 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:25:52.829853 4939 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-n42s6_65d134ac-8a25-4c67-b6c7-cd752a098daf/console-operator/2.log" Nov 03 01:25:52.890157 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:25:52.890129 4939 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-n42s6_65d134ac-8a25-4c67-b6c7-cd752a098daf/console-operator/2.log" Nov 03 01:25:53.095514 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:25:53.095491 4939 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-027c19/payload-processing-external-model-e2e-extmod-a-027c19-65568h8hw" Nov 03 01:25:53.214877 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:25:53.214848 4939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/07a22731-ed88-4d35-be31-46351a8f5ce9-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/07a22731-ed88-4d35-be31-46351a8f5ce9-plugins-config-volume\") pod \"07a22731-ed88-4d35-be31-46351a8f5ce9\" (UID: \"07a22731-ed88-4d35-be31-46351a8f5ce9\") " Nov 03 01:25:53.215067 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:25:53.215013 4939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/07a22731-ed88-4d35-be31-46351a8f5ce9-provider-credentials\" (UniqueName: \"kubernetes.io/projected/07a22731-ed88-4d35-be31-46351a8f5ce9-provider-credentials\") pod \"07a22731-ed88-4d35-be31-46351a8f5ce9\" (UID: \"07a22731-ed88-4d35-be31-46351a8f5ce9\") " Nov 03 01:25:53.215133 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:25:53.215077 4939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/07a22731-ed88-4d35-be31-46351a8f5ce9-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/07a22731-ed88-4d35-be31-46351a8f5ce9-routing-overlay-volume\") pod \"07a22731-ed88-4d35-be31-46351a8f5ce9\" (UID: \"07a22731-ed88-4d35-be31-46351a8f5ce9\") " Nov 03 01:25:53.215254 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:25:53.215229 4939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/07a22731-ed88-4d35-be31-46351a8f5ce9-plugins-config-volume" pod "07a22731-ed88-4d35-be31-46351a8f5ce9" (UID: "07a22731-ed88-4d35-be31-46351a8f5ce9"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 01:25:53.215405 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:25:53.215383 4939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/07a22731-ed88-4d35-be31-46351a8f5ce9-kube-api-access-wfqrs\" (UniqueName: \"kubernetes.io/projected/07a22731-ed88-4d35-be31-46351a8f5ce9-kube-api-access-wfqrs\") pod \"07a22731-ed88-4d35-be31-46351a8f5ce9\" (UID: \"07a22731-ed88-4d35-be31-46351a8f5ce9\") " Nov 03 01:25:53.215506 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:25:53.215483 4939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/07a22731-ed88-4d35-be31-46351a8f5ce9-routing-overlay-volume" pod "07a22731-ed88-4d35-be31-46351a8f5ce9" (UID: "07a22731-ed88-4d35-be31-46351a8f5ce9"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 01:25:53.216125 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:25:53.216101 4939 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/07a22731-ed88-4d35-be31-46351a8f5ce9-plugins-config-volume\") on node \"ip-10-0-140-107.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:25:53.216125 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:25:53.216122 4939 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/07a22731-ed88-4d35-be31-46351a8f5ce9-routing-overlay-volume\") on node \"ip-10-0-140-107.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:25:53.217700 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:25:53.217675 4939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/07a22731-ed88-4d35-be31-46351a8f5ce9-provider-credentials" pod "07a22731-ed88-4d35-be31-46351a8f5ce9" (UID: "07a22731-ed88-4d35-be31-46351a8f5ce9"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 01:25:53.217798 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:25:53.217781 4939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/07a22731-ed88-4d35-be31-46351a8f5ce9-kube-api-access-wfqrs" pod "07a22731-ed88-4d35-be31-46351a8f5ce9" (UID: "07a22731-ed88-4d35-be31-46351a8f5ce9"). InnerVolumeSpecName "kube-api-access-wfqrs". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 01:25:53.291129 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:25:53.291093 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:25:53.339936 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:25:53.329926 4939 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/07a22731-ed88-4d35-be31-46351a8f5ce9-provider-credentials\") on node \"ip-10-0-140-107.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:25:53.339936 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:25:53.329974 4939 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wfqrs\" (UniqueName: \"kubernetes.io/projected/07a22731-ed88-4d35-be31-46351a8f5ce9-kube-api-access-wfqrs\") on node \"ip-10-0-140-107.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:25:53.429684 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:25:53.429587 4939 generic.go:356] "Generic (PLEG): container finished" podID="07a22731-ed88-4d35-be31-46351a8f5ce9" containerID="5106c2789578a525217990f7bffe265d8c0aaf6ece11c31890a51ddeace0e43f" exitCode=0 Nov 03 01:25:53.429684 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:25:53.429637 4939 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-027c19/payload-processing-external-model-e2e-extmod-a-027c19-65568h8hw" Nov 03 01:25:53.429684 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:25:53.429651 4939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-027c19/payload-processing-external-model-e2e-extmod-a-027c19-65568h8hw" event={"ID":"07a22731-ed88-4d35-be31-46351a8f5ce9","Type":"ContainerDied","Data":"5106c2789578a525217990f7bffe265d8c0aaf6ece11c31890a51ddeace0e43f"} Nov 03 01:25:53.430027 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:25:53.429724 4939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-027c19/payload-processing-external-model-e2e-extmod-a-027c19-65568h8hw" event={"ID":"07a22731-ed88-4d35-be31-46351a8f5ce9","Type":"ContainerDied","Data":"eb16674552fbc05db6289197b19287c6cca3aa5fd4d15ba9ae087cee597606f4"} Nov 03 01:25:53.430027 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:25:53.429747 4939 scope.go:122] "RemoveContainer" containerID="5106c2789578a525217990f7bffe265d8c0aaf6ece11c31890a51ddeace0e43f" Nov 03 01:25:53.446232 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:25:53.445787 4939 scope.go:122] "RemoveContainer" containerID="5106c2789578a525217990f7bffe265d8c0aaf6ece11c31890a51ddeace0e43f" Nov 03 01:25:53.451119 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:25:53.446360 4939 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-b-3c5983/payload-processing-external-model-e2e-extmod-b-3c5983-9fbdwjsfd" podUID="88ecb698-2cb1-4a6b-987c-3bf5c797f763" containerName="payload-processing" containerID="cri-o://ab2729093eecd07df2e4db4a2e3ee7b5c5ad02e6ba77c14d6fc3b745150703eb" gracePeriod=25 Nov 03 01:25:53.451859 ip-10-0-140-107 kubenswrapper[4939]: E1103 01:25:53.451441 4939 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5106c2789578a525217990f7bffe265d8c0aaf6ece11c31890a51ddeace0e43f\": container with ID starting with 5106c2789578a525217990f7bffe265d8c0aaf6ece11c31890a51ddeace0e43f not found: ID does not exist" containerID="5106c2789578a525217990f7bffe265d8c0aaf6ece11c31890a51ddeace0e43f" Nov 03 01:25:53.451859 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:25:53.451483 4939 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5106c2789578a525217990f7bffe265d8c0aaf6ece11c31890a51ddeace0e43f"} err="failed to get container status \"5106c2789578a525217990f7bffe265d8c0aaf6ece11c31890a51ddeace0e43f\": rpc error: code = NotFound desc = could not find container \"5106c2789578a525217990f7bffe265d8c0aaf6ece11c31890a51ddeace0e43f\": container with ID starting with 5106c2789578a525217990f7bffe265d8c0aaf6ece11c31890a51ddeace0e43f not found: ID does not exist" Nov 03 01:25:53.469495 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:25:53.469373 4939 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-027c19/payload-processing-external-model-e2e-extmod-a-027c19-65568h8hw"] Nov 03 01:25:53.474183 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:25:53.474159 4939 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-a-027c19/payload-processing-external-model-e2e-extmod-a-027c19-65568h8hw"] Nov 03 01:25:53.748237 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:25:53.748209 4939 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-3c5983/payload-processing-external-model-e2e-extmod-b-3c5983-9fbdwjsfd" Nov 03 01:25:53.781814 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:25:53.781776 4939 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-a-4b17ac-openshift-default-6b75f999cf-k5vtr"] Nov 03 01:25:53.782860 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:25:53.782834 4939 cpu_manager.go:414] "containerMap: removing container" podUID="07a22731-ed88-4d35-be31-46351a8f5ce9" containerName="payload-processing" Nov 03 01:25:53.782860 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:25:53.782859 4939 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="07a22731-ed88-4d35-be31-46351a8f5ce9" containerName="payload-processing" Nov 03 01:25:53.783093 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:25:53.782956 4939 cpu_manager.go:414] "containerMap: removing container" podUID="88ecb698-2cb1-4a6b-987c-3bf5c797f763" containerName="payload-processing" Nov 03 01:25:53.783093 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:25:53.782997 4939 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="88ecb698-2cb1-4a6b-987c-3bf5c797f763" containerName="payload-processing" Nov 03 01:25:53.783093 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:25:53.783029 4939 cpu_manager.go:414] "containerMap: removing container" podUID="000f2bd4-56ac-4df8-8379-f4947877a0af" containerName="sim" Nov 03 01:25:53.783093 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:25:53.783037 4939 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="000f2bd4-56ac-4df8-8379-f4947877a0af" containerName="sim" Nov 03 01:25:53.783306 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:25:53.783285 4939 memory_manager.go:365] "RemoveStaleState removing state" podUID="000f2bd4-56ac-4df8-8379-f4947877a0af" containerName="sim" Nov 03 01:25:53.783359 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:25:53.783309 4939 memory_manager.go:365] "RemoveStaleState removing state" podUID="07a22731-ed88-4d35-be31-46351a8f5ce9" containerName="payload-processing" Nov 03 01:25:53.783359 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:25:53.783334 4939 memory_manager.go:365] "RemoveStaleState removing state" podUID="88ecb698-2cb1-4a6b-987c-3bf5c797f763" containerName="payload-processing" Nov 03 01:25:53.787381 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:25:53.787356 4939 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-4b17ac-openshift-default-6b75f999cf-k5vtr" Nov 03 01:25:53.801651 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:25:53.791594 4939 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-a-4b17ac-openshift-default-dockercfg-w2h8c\"" Nov 03 01:25:53.801651 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:25:53.797609 4939 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-4b17ac-openshift-default-6b75f999cf-k5vtr"] Nov 03 01:25:53.839518 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:25:53.839493 4939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/88ecb698-2cb1-4a6b-987c-3bf5c797f763-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/88ecb698-2cb1-4a6b-987c-3bf5c797f763-plugins-config-volume\") pod \"88ecb698-2cb1-4a6b-987c-3bf5c797f763\" (UID: \"88ecb698-2cb1-4a6b-987c-3bf5c797f763\") " Nov 03 01:25:53.839664 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:25:53.839603 4939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/88ecb698-2cb1-4a6b-987c-3bf5c797f763-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/88ecb698-2cb1-4a6b-987c-3bf5c797f763-routing-overlay-volume\") pod \"88ecb698-2cb1-4a6b-987c-3bf5c797f763\" (UID: \"88ecb698-2cb1-4a6b-987c-3bf5c797f763\") " Nov 03 01:25:53.839664 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:25:53.839657 4939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/88ecb698-2cb1-4a6b-987c-3bf5c797f763-provider-credentials\" (UniqueName: \"kubernetes.io/projected/88ecb698-2cb1-4a6b-987c-3bf5c797f763-provider-credentials\") pod \"88ecb698-2cb1-4a6b-987c-3bf5c797f763\" (UID: \"88ecb698-2cb1-4a6b-987c-3bf5c797f763\") " Nov 03 01:25:53.839840 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:25:53.839825 4939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/88ecb698-2cb1-4a6b-987c-3bf5c797f763-kube-api-access-fb2md\" (UniqueName: \"kubernetes.io/projected/88ecb698-2cb1-4a6b-987c-3bf5c797f763-kube-api-access-fb2md\") pod \"88ecb698-2cb1-4a6b-987c-3bf5c797f763\" (UID: \"88ecb698-2cb1-4a6b-987c-3bf5c797f763\") " Nov 03 01:25:53.840228 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:25:53.839945 4939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/88ecb698-2cb1-4a6b-987c-3bf5c797f763-plugins-config-volume" pod "88ecb698-2cb1-4a6b-987c-3bf5c797f763" (UID: "88ecb698-2cb1-4a6b-987c-3bf5c797f763"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 01:25:53.840228 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:25:53.839998 4939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/88ecb698-2cb1-4a6b-987c-3bf5c797f763-routing-overlay-volume" pod "88ecb698-2cb1-4a6b-987c-3bf5c797f763" (UID: "88ecb698-2cb1-4a6b-987c-3bf5c797f763"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 01:25:53.840228 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:25:53.840177 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5e9f1346-a093-44e2-8f84-712cafcdffdf-istio-token\") pod \"e2e-inf-a-4b17ac-openshift-default-6b75f999cf-k5vtr\" (UID: \"5e9f1346-a093-44e2-8f84-712cafcdffdf\") " pod="openshift-ingress/e2e-inf-a-4b17ac-openshift-default-6b75f999cf-k5vtr" Nov 03 01:25:53.840419 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:25:53.840333 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5e9f1346-a093-44e2-8f84-712cafcdffdf-istiod-ca-cert\") pod \"e2e-inf-a-4b17ac-openshift-default-6b75f999cf-k5vtr\" (UID: \"5e9f1346-a093-44e2-8f84-712cafcdffdf\") " pod="openshift-ingress/e2e-inf-a-4b17ac-openshift-default-6b75f999cf-k5vtr" Nov 03 01:25:53.840462 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:25:53.840443 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5e9f1346-a093-44e2-8f84-712cafcdffdf-istio-podinfo\") pod \"e2e-inf-a-4b17ac-openshift-default-6b75f999cf-k5vtr\" (UID: \"5e9f1346-a093-44e2-8f84-712cafcdffdf\") " pod="openshift-ingress/e2e-inf-a-4b17ac-openshift-default-6b75f999cf-k5vtr" Nov 03 01:25:53.840695 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:25:53.840673 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5e9f1346-a093-44e2-8f84-712cafcdffdf-istio-data\") pod \"e2e-inf-a-4b17ac-openshift-default-6b75f999cf-k5vtr\" (UID: \"5e9f1346-a093-44e2-8f84-712cafcdffdf\") " pod="openshift-ingress/e2e-inf-a-4b17ac-openshift-default-6b75f999cf-k5vtr" Nov 03 01:25:53.840754 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:25:53.840734 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mg664\" (UniqueName: \"kubernetes.io/projected/5e9f1346-a093-44e2-8f84-712cafcdffdf-kube-api-access-mg664\") pod \"e2e-inf-a-4b17ac-openshift-default-6b75f999cf-k5vtr\" (UID: \"5e9f1346-a093-44e2-8f84-712cafcdffdf\") " pod="openshift-ingress/e2e-inf-a-4b17ac-openshift-default-6b75f999cf-k5vtr" Nov 03 01:25:53.840817 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:25:53.840775 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5e9f1346-a093-44e2-8f84-712cafcdffdf-workload-certs\") pod \"e2e-inf-a-4b17ac-openshift-default-6b75f999cf-k5vtr\" (UID: \"5e9f1346-a093-44e2-8f84-712cafcdffdf\") " pod="openshift-ingress/e2e-inf-a-4b17ac-openshift-default-6b75f999cf-k5vtr" Nov 03 01:25:53.840817 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:25:53.840808 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5e9f1346-a093-44e2-8f84-712cafcdffdf-istio-envoy\") pod \"e2e-inf-a-4b17ac-openshift-default-6b75f999cf-k5vtr\" (UID: \"5e9f1346-a093-44e2-8f84-712cafcdffdf\") " pod="openshift-ingress/e2e-inf-a-4b17ac-openshift-default-6b75f999cf-k5vtr" Nov 03 01:25:53.840997 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:25:53.840894 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5e9f1346-a093-44e2-8f84-712cafcdffdf-credential-socket\") pod \"e2e-inf-a-4b17ac-openshift-default-6b75f999cf-k5vtr\" (UID: \"5e9f1346-a093-44e2-8f84-712cafcdffdf\") " pod="openshift-ingress/e2e-inf-a-4b17ac-openshift-default-6b75f999cf-k5vtr" Nov 03 01:25:53.841099 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:25:53.841077 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5e9f1346-a093-44e2-8f84-712cafcdffdf-workload-socket\") pod \"e2e-inf-a-4b17ac-openshift-default-6b75f999cf-k5vtr\" (UID: \"5e9f1346-a093-44e2-8f84-712cafcdffdf\") " pod="openshift-ingress/e2e-inf-a-4b17ac-openshift-default-6b75f999cf-k5vtr" Nov 03 01:25:53.841266 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:25:53.841247 4939 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/88ecb698-2cb1-4a6b-987c-3bf5c797f763-plugins-config-volume\") on node \"ip-10-0-140-107.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:25:53.841333 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:25:53.841272 4939 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/88ecb698-2cb1-4a6b-987c-3bf5c797f763-routing-overlay-volume\") on node \"ip-10-0-140-107.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:25:53.842204 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:25:53.842154 4939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/88ecb698-2cb1-4a6b-987c-3bf5c797f763-provider-credentials" pod "88ecb698-2cb1-4a6b-987c-3bf5c797f763" (UID: "88ecb698-2cb1-4a6b-987c-3bf5c797f763"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 01:25:53.842392 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:25:53.842370 4939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/88ecb698-2cb1-4a6b-987c-3bf5c797f763-kube-api-access-fb2md" pod "88ecb698-2cb1-4a6b-987c-3bf5c797f763" (UID: "88ecb698-2cb1-4a6b-987c-3bf5c797f763"). InnerVolumeSpecName "kube-api-access-fb2md". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 01:25:53.875048 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:25:53.875019 4939 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="07a22731-ed88-4d35-be31-46351a8f5ce9" path="/var/lib/kubelet/pods/07a22731-ed88-4d35-be31-46351a8f5ce9/volumes" Nov 03 01:25:53.942095 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:25:53.942061 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5e9f1346-a093-44e2-8f84-712cafcdffdf-workload-socket\") pod \"e2e-inf-a-4b17ac-openshift-default-6b75f999cf-k5vtr\" (UID: \"5e9f1346-a093-44e2-8f84-712cafcdffdf\") " pod="openshift-ingress/e2e-inf-a-4b17ac-openshift-default-6b75f999cf-k5vtr" Nov 03 01:25:53.942459 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:25:53.942129 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5e9f1346-a093-44e2-8f84-712cafcdffdf-istio-token\") pod \"e2e-inf-a-4b17ac-openshift-default-6b75f999cf-k5vtr\" (UID: \"5e9f1346-a093-44e2-8f84-712cafcdffdf\") " pod="openshift-ingress/e2e-inf-a-4b17ac-openshift-default-6b75f999cf-k5vtr" Nov 03 01:25:53.942459 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:25:53.942220 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5e9f1346-a093-44e2-8f84-712cafcdffdf-istiod-ca-cert\") pod \"e2e-inf-a-4b17ac-openshift-default-6b75f999cf-k5vtr\" (UID: \"5e9f1346-a093-44e2-8f84-712cafcdffdf\") " pod="openshift-ingress/e2e-inf-a-4b17ac-openshift-default-6b75f999cf-k5vtr" Nov 03 01:25:53.942459 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:25:53.942300 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5e9f1346-a093-44e2-8f84-712cafcdffdf-istio-podinfo\") pod \"e2e-inf-a-4b17ac-openshift-default-6b75f999cf-k5vtr\" (UID: \"5e9f1346-a093-44e2-8f84-712cafcdffdf\") " pod="openshift-ingress/e2e-inf-a-4b17ac-openshift-default-6b75f999cf-k5vtr" Nov 03 01:25:53.942675 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:25:53.942493 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5e9f1346-a093-44e2-8f84-712cafcdffdf-workload-socket\") pod \"e2e-inf-a-4b17ac-openshift-default-6b75f999cf-k5vtr\" (UID: \"5e9f1346-a093-44e2-8f84-712cafcdffdf\") " pod="openshift-ingress/e2e-inf-a-4b17ac-openshift-default-6b75f999cf-k5vtr" Nov 03 01:25:53.942675 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:25:53.942546 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5e9f1346-a093-44e2-8f84-712cafcdffdf-istio-data\") pod \"e2e-inf-a-4b17ac-openshift-default-6b75f999cf-k5vtr\" (UID: \"5e9f1346-a093-44e2-8f84-712cafcdffdf\") " pod="openshift-ingress/e2e-inf-a-4b17ac-openshift-default-6b75f999cf-k5vtr" Nov 03 01:25:53.942675 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:25:53.942611 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mg664\" (UniqueName: \"kubernetes.io/projected/5e9f1346-a093-44e2-8f84-712cafcdffdf-kube-api-access-mg664\") pod \"e2e-inf-a-4b17ac-openshift-default-6b75f999cf-k5vtr\" (UID: \"5e9f1346-a093-44e2-8f84-712cafcdffdf\") " pod="openshift-ingress/e2e-inf-a-4b17ac-openshift-default-6b75f999cf-k5vtr" Nov 03 01:25:53.942675 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:25:53.942664 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5e9f1346-a093-44e2-8f84-712cafcdffdf-workload-certs\") pod \"e2e-inf-a-4b17ac-openshift-default-6b75f999cf-k5vtr\" (UID: \"5e9f1346-a093-44e2-8f84-712cafcdffdf\") " pod="openshift-ingress/e2e-inf-a-4b17ac-openshift-default-6b75f999cf-k5vtr" Nov 03 01:25:53.942976 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:25:53.942930 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5e9f1346-a093-44e2-8f84-712cafcdffdf-istio-envoy\") pod \"e2e-inf-a-4b17ac-openshift-default-6b75f999cf-k5vtr\" (UID: \"5e9f1346-a093-44e2-8f84-712cafcdffdf\") " pod="openshift-ingress/e2e-inf-a-4b17ac-openshift-default-6b75f999cf-k5vtr" Nov 03 01:25:53.942976 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:25:53.942966 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5e9f1346-a093-44e2-8f84-712cafcdffdf-workload-certs\") pod \"e2e-inf-a-4b17ac-openshift-default-6b75f999cf-k5vtr\" (UID: \"5e9f1346-a093-44e2-8f84-712cafcdffdf\") " pod="openshift-ingress/e2e-inf-a-4b17ac-openshift-default-6b75f999cf-k5vtr" Nov 03 01:25:53.943187 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:25:53.943035 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5e9f1346-a093-44e2-8f84-712cafcdffdf-credential-socket\") pod \"e2e-inf-a-4b17ac-openshift-default-6b75f999cf-k5vtr\" (UID: \"5e9f1346-a093-44e2-8f84-712cafcdffdf\") " pod="openshift-ingress/e2e-inf-a-4b17ac-openshift-default-6b75f999cf-k5vtr" Nov 03 01:25:53.943187 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:25:53.943061 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5e9f1346-a093-44e2-8f84-712cafcdffdf-istiod-ca-cert\") pod \"e2e-inf-a-4b17ac-openshift-default-6b75f999cf-k5vtr\" (UID: \"5e9f1346-a093-44e2-8f84-712cafcdffdf\") " pod="openshift-ingress/e2e-inf-a-4b17ac-openshift-default-6b75f999cf-k5vtr" Nov 03 01:25:53.943313 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:25:53.943228 4939 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/88ecb698-2cb1-4a6b-987c-3bf5c797f763-provider-credentials\") on node \"ip-10-0-140-107.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:25:53.943313 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:25:53.943247 4939 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fb2md\" (UniqueName: \"kubernetes.io/projected/88ecb698-2cb1-4a6b-987c-3bf5c797f763-kube-api-access-fb2md\") on node \"ip-10-0-140-107.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:25:53.943436 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:25:53.943361 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5e9f1346-a093-44e2-8f84-712cafcdffdf-credential-socket\") pod \"e2e-inf-a-4b17ac-openshift-default-6b75f999cf-k5vtr\" (UID: \"5e9f1346-a093-44e2-8f84-712cafcdffdf\") " pod="openshift-ingress/e2e-inf-a-4b17ac-openshift-default-6b75f999cf-k5vtr" Nov 03 01:25:53.943436 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:25:53.943404 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5e9f1346-a093-44e2-8f84-712cafcdffdf-istio-data\") pod \"e2e-inf-a-4b17ac-openshift-default-6b75f999cf-k5vtr\" (UID: \"5e9f1346-a093-44e2-8f84-712cafcdffdf\") " pod="openshift-ingress/e2e-inf-a-4b17ac-openshift-default-6b75f999cf-k5vtr" Nov 03 01:25:53.945136 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:25:53.945111 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5e9f1346-a093-44e2-8f84-712cafcdffdf-istio-podinfo\") pod \"e2e-inf-a-4b17ac-openshift-default-6b75f999cf-k5vtr\" (UID: \"5e9f1346-a093-44e2-8f84-712cafcdffdf\") " pod="openshift-ingress/e2e-inf-a-4b17ac-openshift-default-6b75f999cf-k5vtr" Nov 03 01:25:53.945346 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:25:53.945321 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5e9f1346-a093-44e2-8f84-712cafcdffdf-istio-envoy\") pod \"e2e-inf-a-4b17ac-openshift-default-6b75f999cf-k5vtr\" (UID: \"5e9f1346-a093-44e2-8f84-712cafcdffdf\") " pod="openshift-ingress/e2e-inf-a-4b17ac-openshift-default-6b75f999cf-k5vtr" Nov 03 01:25:53.950597 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:25:53.950574 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5e9f1346-a093-44e2-8f84-712cafcdffdf-istio-token\") pod \"e2e-inf-a-4b17ac-openshift-default-6b75f999cf-k5vtr\" (UID: \"5e9f1346-a093-44e2-8f84-712cafcdffdf\") " pod="openshift-ingress/e2e-inf-a-4b17ac-openshift-default-6b75f999cf-k5vtr" Nov 03 01:25:53.951356 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:25:53.951333 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mg664\" (UniqueName: \"kubernetes.io/projected/5e9f1346-a093-44e2-8f84-712cafcdffdf-kube-api-access-mg664\") pod \"e2e-inf-a-4b17ac-openshift-default-6b75f999cf-k5vtr\" (UID: \"5e9f1346-a093-44e2-8f84-712cafcdffdf\") " pod="openshift-ingress/e2e-inf-a-4b17ac-openshift-default-6b75f999cf-k5vtr" Nov 03 01:25:54.078969 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:25:54.078856 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:25:54.161935 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:25:54.147269 4939 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-4b17ac-openshift-default-6b75f999cf-k5vtr" Nov 03 01:25:54.267773 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:25:54.267729 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:25:54.405934 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:25:54.404507 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:25:54.461028 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:25:54.456033 4939 generic.go:356] "Generic (PLEG): container finished" podID="88ecb698-2cb1-4a6b-987c-3bf5c797f763" containerID="ab2729093eecd07df2e4db4a2e3ee7b5c5ad02e6ba77c14d6fc3b745150703eb" exitCode=0 Nov 03 01:25:54.461028 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:25:54.456219 4939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-3c5983/payload-processing-external-model-e2e-extmod-b-3c5983-9fbdwjsfd" event={"ID":"88ecb698-2cb1-4a6b-987c-3bf5c797f763","Type":"ContainerDied","Data":"ab2729093eecd07df2e4db4a2e3ee7b5c5ad02e6ba77c14d6fc3b745150703eb"} Nov 03 01:25:54.461028 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:25:54.456276 4939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-3c5983/payload-processing-external-model-e2e-extmod-b-3c5983-9fbdwjsfd" event={"ID":"88ecb698-2cb1-4a6b-987c-3bf5c797f763","Type":"ContainerDied","Data":"38926cbf984d3376b84566564df65859290eae3f0d1a4d981d0001183203bd16"} Nov 03 01:25:54.461028 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:25:54.456305 4939 scope.go:122] "RemoveContainer" containerID="ab2729093eecd07df2e4db4a2e3ee7b5c5ad02e6ba77c14d6fc3b745150703eb" Nov 03 01:25:54.461028 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:25:54.456450 4939 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-3c5983/payload-processing-external-model-e2e-extmod-b-3c5983-9fbdwjsfd" Nov 03 01:25:54.503338 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:25:54.503299 4939 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-3c5983/payload-processing-external-model-e2e-extmod-b-3c5983-9fbdwjsfd"] Nov 03 01:25:54.507518 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:25:54.507487 4939 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-b-3c5983/payload-processing-external-model-e2e-extmod-b-3c5983-9fbdwjsfd"] Nov 03 01:25:54.520284 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:25:54.520255 4939 scope.go:122] "RemoveContainer" containerID="ab2729093eecd07df2e4db4a2e3ee7b5c5ad02e6ba77c14d6fc3b745150703eb" Nov 03 01:25:54.520687 ip-10-0-140-107 kubenswrapper[4939]: E1103 01:25:54.520661 4939 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ab2729093eecd07df2e4db4a2e3ee7b5c5ad02e6ba77c14d6fc3b745150703eb\": container with ID starting with ab2729093eecd07df2e4db4a2e3ee7b5c5ad02e6ba77c14d6fc3b745150703eb not found: ID does not exist" containerID="ab2729093eecd07df2e4db4a2e3ee7b5c5ad02e6ba77c14d6fc3b745150703eb" Nov 03 01:25:54.520813 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:25:54.520697 4939 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ab2729093eecd07df2e4db4a2e3ee7b5c5ad02e6ba77c14d6fc3b745150703eb"} err="failed to get container status \"ab2729093eecd07df2e4db4a2e3ee7b5c5ad02e6ba77c14d6fc3b745150703eb\": rpc error: code = NotFound desc = could not find container \"ab2729093eecd07df2e4db4a2e3ee7b5c5ad02e6ba77c14d6fc3b745150703eb\": container with ID starting with ab2729093eecd07df2e4db4a2e3ee7b5c5ad02e6ba77c14d6fc3b745150703eb not found: ID does not exist" Nov 03 01:25:54.541058 ip-10-0-140-107 kubenswrapper[4939]: W1103 01:25:54.541031 4939 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5e9f1346_a093_44e2_8f84_712cafcdffdf.slice/crio-44a4568a3ab0b5804ad38a3c32806288d733e3ab0fbab838d45b550ffd3b8166 WatchSource:0}: Error finding container 44a4568a3ab0b5804ad38a3c32806288d733e3ab0fbab838d45b550ffd3b8166: Status 404 returned error can't find the container with id 44a4568a3ab0b5804ad38a3c32806288d733e3ab0fbab838d45b550ffd3b8166 Nov 03 01:25:54.541700 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:25:54.541673 4939 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-4b17ac-openshift-default-6b75f999cf-k5vtr"] Nov 03 01:25:54.544219 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:25:54.544184 4939 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 01:25:54.544293 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:25:54.544253 4939 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 01:25:54.544293 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:25:54.544281 4939 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 01:25:54.692442 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:25:54.692352 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:25:54.808727 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:25:54.808692 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:25:55.001418 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:25:55.001332 4939 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-4pw7v"] Nov 03 01:25:55.125133 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:25:55.125098 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:25:55.383940 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:25:55.373811 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:25:55.534934 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:25:55.534184 4939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-4b17ac-openshift-default-6b75f999cf-k5vtr" event={"ID":"5e9f1346-a093-44e2-8f84-712cafcdffdf","Type":"ContainerStarted","Data":"ec5a9665db5301394447fd6076ac3a6dbc1e2fb1aeebd87a07d86f21d2873ddb"} Nov 03 01:25:55.534934 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:25:55.534251 4939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-4b17ac-openshift-default-6b75f999cf-k5vtr" event={"ID":"5e9f1346-a093-44e2-8f84-712cafcdffdf","Type":"ContainerStarted","Data":"44a4568a3ab0b5804ad38a3c32806288d733e3ab0fbab838d45b550ffd3b8166"} Nov 03 01:25:55.604949 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:25:55.604664 4939 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-a-4b17ac-openshift-default-6b75f999cf-k5vtr" podStartSLOduration=46.604640991 podStartE2EDuration="46.604640991s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 01:25:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 01:25:55.570471703 +0000 UTC m=+2703.834806649" watchObservedRunningTime="2026-11-03 01:25:55.604640991 +0000 UTC m=+2703.868975937" Nov 03 01:25:55.876127 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:25:55.876089 4939 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="88ecb698-2cb1-4a6b-987c-3bf5c797f763" path="/var/lib/kubelet/pods/88ecb698-2cb1-4a6b-987c-3bf5c797f763/volumes" Nov 03 01:25:56.026405 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:25:56.026370 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:25:56.154985 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:25:56.154269 4939 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-4b17ac-openshift-default-6b75f999cf-k5vtr" Nov 03 01:25:56.757803 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:25:56.757668 4939 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-4b17ac-openshift-default-6b75f999cf-k5vtr" Nov 03 01:25:57.146938 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:25:57.146276 4939 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-b-ce04af/test-model-ce04af-kserve-84bf7f6d7c-qd56c" podUID="72db0238-20cc-40cb-affb-158746adb87b" containerName="main" probeResult="failure" output="Get \"https://10.132.1.39:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 01:25:57.581005 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:25:57.576309 4939 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-88bbf7/distinct-2-w1-88bbf7-kserve-5457cdcdbd-l6tfd" podUID="a397931b-faa9-4c2e-8290-491350546663" containerName="main" probeResult="failure" output="Get \"https://10.132.0.85:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 01:25:57.692301 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:25:57.692270 4939 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-4b17ac-openshift-default-6b75f999cf-k5vtr" Nov 03 01:25:57.694455 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:25:57.694435 4939 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-4b17ac-openshift-default-6b75f999cf-k5vtr" Nov 03 01:25:57.944038 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:25:57.933834 4939 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-f7680d/distinct-2-w2-f7680d-kserve-7d7c4564bf-cbv2c" podUID="2de18086-761b-4773-b9fc-15ddc4a220fc" containerName="main" probeResult="failure" output="Get \"https://10.132.0.75:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 01:25:58.170713 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:25:58.170672 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:25:58.300732 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:25:58.300699 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:25:58.428940 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:25:58.428094 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:25:59.379631 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:25:59.379596 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:25:59.562449 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:25:59.562414 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:25:59.620865 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:25:59.620827 4939 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-4b17ac-84f44c7d99-8286l"] Nov 03 01:25:59.627849 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:25:59.627820 4939 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-4b17ac-84f44c7d99-8286l" Nov 03 01:25:59.631271 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:25:59.631206 4939 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-4b17ac-84f44c7d99-8286l"] Nov 03 01:25:59.631600 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:25:59.631573 4939 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-a-4b17ac\"" Nov 03 01:25:59.632768 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:25:59.632742 4939 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-a-4b17ac-dockercfg-ckrnv\"" Nov 03 01:25:59.678811 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:25:59.678752 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:25:59.682613 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:25:59.682585 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m5v8q\" (UniqueName: \"kubernetes.io/projected/abd497dc-d8a1-42f5-8f41-c8340af41dd9-kube-api-access-m5v8q\") pod \"payload-pre-processing-e2e-inf-a-4b17ac-84f44c7d99-8286l\" (UID: \"abd497dc-d8a1-42f5-8f41-c8340af41dd9\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-4b17ac-84f44c7d99-8286l" Nov 03 01:25:59.682826 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:25:59.682806 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/abd497dc-d8a1-42f5-8f41-c8340af41dd9-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-4b17ac-84f44c7d99-8286l\" (UID: \"abd497dc-d8a1-42f5-8f41-c8340af41dd9\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-4b17ac-84f44c7d99-8286l" Nov 03 01:25:59.817604 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:25:59.796338 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m5v8q\" (UniqueName: \"kubernetes.io/projected/abd497dc-d8a1-42f5-8f41-c8340af41dd9-kube-api-access-m5v8q\") pod \"payload-pre-processing-e2e-inf-a-4b17ac-84f44c7d99-8286l\" (UID: \"abd497dc-d8a1-42f5-8f41-c8340af41dd9\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-4b17ac-84f44c7d99-8286l" Nov 03 01:25:59.817604 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:25:59.796574 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/abd497dc-d8a1-42f5-8f41-c8340af41dd9-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-4b17ac-84f44c7d99-8286l\" (UID: \"abd497dc-d8a1-42f5-8f41-c8340af41dd9\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-4b17ac-84f44c7d99-8286l" Nov 03 01:25:59.818941 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:25:59.818830 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/abd497dc-d8a1-42f5-8f41-c8340af41dd9-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-4b17ac-84f44c7d99-8286l\" (UID: \"abd497dc-d8a1-42f5-8f41-c8340af41dd9\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-4b17ac-84f44c7d99-8286l" Nov 03 01:25:59.884016 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:25:59.853229 4939 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-650be0-openshift-default-5c7c5fb7d9-tkzdb"] Nov 03 01:25:59.884016 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:25:59.853489 4939 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-650be0-openshift-default-5c7c5fb7d9-tkzdb" podUID="4da03a65-9706-40b4-955f-6e628901b6be" containerName="istio-proxy" containerID="cri-o://30bc7f034396870ec020bebe35ea11831db41e782506d32f1b37ace2348c0a85" gracePeriod=30 Nov 03 01:26:00.232842 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:26:00.229437 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-m5v8q\" (UniqueName: \"kubernetes.io/projected/abd497dc-d8a1-42f5-8f41-c8340af41dd9-kube-api-access-m5v8q\") pod \"payload-pre-processing-e2e-inf-a-4b17ac-84f44c7d99-8286l\" (UID: \"abd497dc-d8a1-42f5-8f41-c8340af41dd9\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-4b17ac-84f44c7d99-8286l" Nov 03 01:26:00.256254 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:26:00.256217 4939 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-4b17ac-84f44c7d99-8286l" Nov 03 01:26:00.488723 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:26:00.488631 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:26:00.600819 ip-10-0-140-107 kubenswrapper[4939]: W1103 01:26:00.600762 4939 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podabd497dc_d8a1_42f5_8f41_c8340af41dd9.slice/crio-70468bbc72743a3470e9a93a3693dc439cdcdd3802c23fd77acefcbd49293e5a WatchSource:0}: Error finding container 70468bbc72743a3470e9a93a3693dc439cdcdd3802c23fd77acefcbd49293e5a: Status 404 returned error can't find the container with id 70468bbc72743a3470e9a93a3693dc439cdcdd3802c23fd77acefcbd49293e5a Nov 03 01:26:00.601606 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:26:00.601575 4939 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-4b17ac-84f44c7d99-8286l"] Nov 03 01:26:01.094778 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:26:01.094739 4939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-4b17ac-84f44c7d99-8286l" event={"ID":"abd497dc-d8a1-42f5-8f41-c8340af41dd9","Type":"ContainerStarted","Data":"3764e33077507458a7b7e4cba350f19dacd8cbf57e5d14875fe70652e31a137f"} Nov 03 01:26:01.095039 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:26:01.094804 4939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-4b17ac-84f44c7d99-8286l" event={"ID":"abd497dc-d8a1-42f5-8f41-c8340af41dd9","Type":"ContainerStarted","Data":"70468bbc72743a3470e9a93a3693dc439cdcdd3802c23fd77acefcbd49293e5a"} Nov 03 01:26:01.095039 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:26:01.094837 4939 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-4b17ac-84f44c7d99-8286l" Nov 03 01:26:01.096622 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:26:01.096595 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:26:01.115434 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:26:01.115390 4939 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-4b17ac-84f44c7d99-8286l" podStartSLOduration=2.115374224 podStartE2EDuration="2.115374224s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 01:25:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 01:26:01.11109199 +0000 UTC m=+2709.375426940" watchObservedRunningTime="2026-11-03 01:26:01.115374224 +0000 UTC m=+2709.379709170" Nov 03 01:26:01.932981 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:26:01.932863 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:26:02.584206 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:26:02.584170 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:26:03.202106 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:26:03.202042 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:26:03.873745 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:26:03.873712 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:26:04.518983 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:26:04.518942 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:26:05.139189 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:26:05.139151 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:26:05.242555 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:26:05.242514 4939 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w0-50e623-openshift-default-6c86dc794f-5m78z"] Nov 03 01:26:05.243199 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:26:05.243149 4939 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w0-50e623-openshift-default-6c86dc794f-5m78z" podUID="64952141-7a68-4e46-8982-3db9402620e3" containerName="istio-proxy" containerID="cri-o://6eea49382c75191aaa1998e3e23c1dea09f9bdc053a349a06b05b89a82ed2443" gracePeriod=30 Nov 03 01:26:05.522132 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:26:05.522103 4939 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-650be0-openshift-default-5c7c5fb7d9-tkzdb" Nov 03 01:26:05.564181 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:26:05.564109 4939 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-b-ce04af-openshift-default-7dccd45f47-n5ndk"] Nov 03 01:26:05.568322 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:26:05.568278 4939 cpu_manager.go:414] "containerMap: removing container" podUID="4da03a65-9706-40b4-955f-6e628901b6be" containerName="istio-proxy" Nov 03 01:26:05.568322 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:26:05.568310 4939 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4da03a65-9706-40b4-955f-6e628901b6be" containerName="istio-proxy" Nov 03 01:26:05.568638 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:26:05.568380 4939 cpu_manager.go:414] "containerMap: removing container" podUID="4da03a65-9706-40b4-955f-6e628901b6be" containerName="istio-proxy" Nov 03 01:26:05.568638 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:26:05.568389 4939 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4da03a65-9706-40b4-955f-6e628901b6be" containerName="istio-proxy" Nov 03 01:26:05.569383 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:26:05.568882 4939 memory_manager.go:365] "RemoveStaleState removing state" podUID="4da03a65-9706-40b4-955f-6e628901b6be" containerName="istio-proxy" Nov 03 01:26:05.569383 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:26:05.568952 4939 memory_manager.go:365] "RemoveStaleState removing state" podUID="4da03a65-9706-40b4-955f-6e628901b6be" containerName="istio-proxy" Nov 03 01:26:05.598433 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:26:05.597933 4939 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-ce04af-openshift-default-7dccd45f47-n5ndk"] Nov 03 01:26:05.598433 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:26:05.598068 4939 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-ce04af-openshift-default-7dccd45f47-n5ndk" Nov 03 01:26:05.632821 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:26:05.632787 4939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4da03a65-9706-40b4-955f-6e628901b6be-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4da03a65-9706-40b4-955f-6e628901b6be-istio-envoy\") pod \"4da03a65-9706-40b4-955f-6e628901b6be\" (UID: \"4da03a65-9706-40b4-955f-6e628901b6be\") " Nov 03 01:26:05.633037 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:26:05.632928 4939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4da03a65-9706-40b4-955f-6e628901b6be-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4da03a65-9706-40b4-955f-6e628901b6be-credential-socket\") pod \"4da03a65-9706-40b4-955f-6e628901b6be\" (UID: \"4da03a65-9706-40b4-955f-6e628901b6be\") " Nov 03 01:26:05.633127 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:26:05.633073 4939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4da03a65-9706-40b4-955f-6e628901b6be-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4da03a65-9706-40b4-955f-6e628901b6be-workload-certs\") pod \"4da03a65-9706-40b4-955f-6e628901b6be\" (UID: \"4da03a65-9706-40b4-955f-6e628901b6be\") " Nov 03 01:26:05.633240 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:26:05.633165 4939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4da03a65-9706-40b4-955f-6e628901b6be-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4da03a65-9706-40b4-955f-6e628901b6be-istio-data\") pod \"4da03a65-9706-40b4-955f-6e628901b6be\" (UID: \"4da03a65-9706-40b4-955f-6e628901b6be\") " Nov 03 01:26:05.633304 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:26:05.633271 4939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4da03a65-9706-40b4-955f-6e628901b6be-credential-socket" pod "4da03a65-9706-40b4-955f-6e628901b6be" (UID: "4da03a65-9706-40b4-955f-6e628901b6be"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 01:26:05.633364 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:26:05.633307 4939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4da03a65-9706-40b4-955f-6e628901b6be-istio-token\" (UniqueName: \"kubernetes.io/projected/4da03a65-9706-40b4-955f-6e628901b6be-istio-token\") pod \"4da03a65-9706-40b4-955f-6e628901b6be\" (UID: \"4da03a65-9706-40b4-955f-6e628901b6be\") " Nov 03 01:26:05.633364 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:26:05.633348 4939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/4da03a65-9706-40b4-955f-6e628901b6be-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4da03a65-9706-40b4-955f-6e628901b6be-istio-podinfo\") pod \"4da03a65-9706-40b4-955f-6e628901b6be\" (UID: \"4da03a65-9706-40b4-955f-6e628901b6be\") " Nov 03 01:26:05.633472 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:26:05.633399 4939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4da03a65-9706-40b4-955f-6e628901b6be-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4da03a65-9706-40b4-955f-6e628901b6be-workload-socket\") pod \"4da03a65-9706-40b4-955f-6e628901b6be\" (UID: \"4da03a65-9706-40b4-955f-6e628901b6be\") " Nov 03 01:26:05.633534 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:26:05.633518 4939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4da03a65-9706-40b4-955f-6e628901b6be-kube-api-access-7lbj6\" (UniqueName: \"kubernetes.io/projected/4da03a65-9706-40b4-955f-6e628901b6be-kube-api-access-7lbj6\") pod \"4da03a65-9706-40b4-955f-6e628901b6be\" (UID: \"4da03a65-9706-40b4-955f-6e628901b6be\") " Nov 03 01:26:05.633702 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:26:05.633666 4939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4da03a65-9706-40b4-955f-6e628901b6be-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4da03a65-9706-40b4-955f-6e628901b6be-istiod-ca-cert\") pod \"4da03a65-9706-40b4-955f-6e628901b6be\" (UID: \"4da03a65-9706-40b4-955f-6e628901b6be\") " Nov 03 01:26:05.633702 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:26:05.633694 4939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4da03a65-9706-40b4-955f-6e628901b6be-workload-socket" pod "4da03a65-9706-40b4-955f-6e628901b6be" (UID: "4da03a65-9706-40b4-955f-6e628901b6be"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 01:26:05.633894 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:26:05.633782 4939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4da03a65-9706-40b4-955f-6e628901b6be-istio-data" pod "4da03a65-9706-40b4-955f-6e628901b6be" (UID: "4da03a65-9706-40b4-955f-6e628901b6be"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 01:26:05.634227 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:26:05.634203 4939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4da03a65-9706-40b4-955f-6e628901b6be-istiod-ca-cert" pod "4da03a65-9706-40b4-955f-6e628901b6be" (UID: "4da03a65-9706-40b4-955f-6e628901b6be"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 01:26:05.634664 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:26:05.634640 4939 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4da03a65-9706-40b4-955f-6e628901b6be-istiod-ca-cert\") on node \"ip-10-0-140-107.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:26:05.634773 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:26:05.634671 4939 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4da03a65-9706-40b4-955f-6e628901b6be-credential-socket\") on node \"ip-10-0-140-107.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:26:05.634773 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:26:05.634687 4939 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4da03a65-9706-40b4-955f-6e628901b6be-istio-data\") on node \"ip-10-0-140-107.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:26:05.634773 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:26:05.634702 4939 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4da03a65-9706-40b4-955f-6e628901b6be-workload-socket\") on node \"ip-10-0-140-107.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:26:05.635000 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:26:05.634895 4939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4da03a65-9706-40b4-955f-6e628901b6be-workload-certs" pod "4da03a65-9706-40b4-955f-6e628901b6be" (UID: "4da03a65-9706-40b4-955f-6e628901b6be"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 01:26:05.636434 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:26:05.636405 4939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4da03a65-9706-40b4-955f-6e628901b6be-kube-api-access-7lbj6" pod "4da03a65-9706-40b4-955f-6e628901b6be" (UID: "4da03a65-9706-40b4-955f-6e628901b6be"). InnerVolumeSpecName "kube-api-access-7lbj6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 01:26:05.637065 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:26:05.637037 4939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4da03a65-9706-40b4-955f-6e628901b6be-istio-token" pod "4da03a65-9706-40b4-955f-6e628901b6be" (UID: "4da03a65-9706-40b4-955f-6e628901b6be"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 01:26:05.638081 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:26:05.638059 4939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/4da03a65-9706-40b4-955f-6e628901b6be-istio-podinfo" pod "4da03a65-9706-40b4-955f-6e628901b6be" (UID: "4da03a65-9706-40b4-955f-6e628901b6be"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 01:26:05.639054 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:26:05.639032 4939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4da03a65-9706-40b4-955f-6e628901b6be-istio-envoy" pod "4da03a65-9706-40b4-955f-6e628901b6be" (UID: "4da03a65-9706-40b4-955f-6e628901b6be"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 01:26:05.736288 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:26:05.736248 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5188b487-c967-42f8-a5c8-43f118b3e50d-istiod-ca-cert\") pod \"e2e-inf-b-ce04af-openshift-default-7dccd45f47-n5ndk\" (UID: \"5188b487-c967-42f8-a5c8-43f118b3e50d\") " pod="openshift-ingress/e2e-inf-b-ce04af-openshift-default-7dccd45f47-n5ndk" Nov 03 01:26:05.736476 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:26:05.736433 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5188b487-c967-42f8-a5c8-43f118b3e50d-workload-certs\") pod \"e2e-inf-b-ce04af-openshift-default-7dccd45f47-n5ndk\" (UID: \"5188b487-c967-42f8-a5c8-43f118b3e50d\") " pod="openshift-ingress/e2e-inf-b-ce04af-openshift-default-7dccd45f47-n5ndk" Nov 03 01:26:05.736582 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:26:05.736520 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5188b487-c967-42f8-a5c8-43f118b3e50d-istio-podinfo\") pod \"e2e-inf-b-ce04af-openshift-default-7dccd45f47-n5ndk\" (UID: \"5188b487-c967-42f8-a5c8-43f118b3e50d\") " pod="openshift-ingress/e2e-inf-b-ce04af-openshift-default-7dccd45f47-n5ndk" Nov 03 01:26:05.736582 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:26:05.736540 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5188b487-c967-42f8-a5c8-43f118b3e50d-workload-socket\") pod \"e2e-inf-b-ce04af-openshift-default-7dccd45f47-n5ndk\" (UID: \"5188b487-c967-42f8-a5c8-43f118b3e50d\") " pod="openshift-ingress/e2e-inf-b-ce04af-openshift-default-7dccd45f47-n5ndk" Nov 03 01:26:05.736788 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:26:05.736756 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5188b487-c967-42f8-a5c8-43f118b3e50d-istio-token\") pod \"e2e-inf-b-ce04af-openshift-default-7dccd45f47-n5ndk\" (UID: \"5188b487-c967-42f8-a5c8-43f118b3e50d\") " pod="openshift-ingress/e2e-inf-b-ce04af-openshift-default-7dccd45f47-n5ndk" Nov 03 01:26:05.736891 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:26:05.736869 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5188b487-c967-42f8-a5c8-43f118b3e50d-credential-socket\") pod \"e2e-inf-b-ce04af-openshift-default-7dccd45f47-n5ndk\" (UID: \"5188b487-c967-42f8-a5c8-43f118b3e50d\") " pod="openshift-ingress/e2e-inf-b-ce04af-openshift-default-7dccd45f47-n5ndk" Nov 03 01:26:05.736984 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:26:05.736974 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5188b487-c967-42f8-a5c8-43f118b3e50d-istio-data\") pod \"e2e-inf-b-ce04af-openshift-default-7dccd45f47-n5ndk\" (UID: \"5188b487-c967-42f8-a5c8-43f118b3e50d\") " pod="openshift-ingress/e2e-inf-b-ce04af-openshift-default-7dccd45f47-n5ndk" Nov 03 01:26:05.737045 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:26:05.737037 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dsrt8\" (UniqueName: \"kubernetes.io/projected/5188b487-c967-42f8-a5c8-43f118b3e50d-kube-api-access-dsrt8\") pod \"e2e-inf-b-ce04af-openshift-default-7dccd45f47-n5ndk\" (UID: \"5188b487-c967-42f8-a5c8-43f118b3e50d\") " pod="openshift-ingress/e2e-inf-b-ce04af-openshift-default-7dccd45f47-n5ndk" Nov 03 01:26:05.737214 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:26:05.737191 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5188b487-c967-42f8-a5c8-43f118b3e50d-istio-envoy\") pod \"e2e-inf-b-ce04af-openshift-default-7dccd45f47-n5ndk\" (UID: \"5188b487-c967-42f8-a5c8-43f118b3e50d\") " pod="openshift-ingress/e2e-inf-b-ce04af-openshift-default-7dccd45f47-n5ndk" Nov 03 01:26:05.737393 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:26:05.737377 4939 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4da03a65-9706-40b4-955f-6e628901b6be-istio-envoy\") on node \"ip-10-0-140-107.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:26:05.737436 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:26:05.737405 4939 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4da03a65-9706-40b4-955f-6e628901b6be-workload-certs\") on node \"ip-10-0-140-107.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:26:05.737436 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:26:05.737422 4939 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4da03a65-9706-40b4-955f-6e628901b6be-istio-token\") on node \"ip-10-0-140-107.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:26:05.737510 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:26:05.737438 4939 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4da03a65-9706-40b4-955f-6e628901b6be-istio-podinfo\") on node \"ip-10-0-140-107.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:26:05.737510 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:26:05.737453 4939 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7lbj6\" (UniqueName: \"kubernetes.io/projected/4da03a65-9706-40b4-955f-6e628901b6be-kube-api-access-7lbj6\") on node \"ip-10-0-140-107.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:26:05.839438 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:26:05.839400 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5188b487-c967-42f8-a5c8-43f118b3e50d-istio-podinfo\") pod \"e2e-inf-b-ce04af-openshift-default-7dccd45f47-n5ndk\" (UID: \"5188b487-c967-42f8-a5c8-43f118b3e50d\") " pod="openshift-ingress/e2e-inf-b-ce04af-openshift-default-7dccd45f47-n5ndk" Nov 03 01:26:05.839611 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:26:05.839453 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5188b487-c967-42f8-a5c8-43f118b3e50d-workload-socket\") pod \"e2e-inf-b-ce04af-openshift-default-7dccd45f47-n5ndk\" (UID: \"5188b487-c967-42f8-a5c8-43f118b3e50d\") " pod="openshift-ingress/e2e-inf-b-ce04af-openshift-default-7dccd45f47-n5ndk" Nov 03 01:26:05.839698 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:26:05.839624 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5188b487-c967-42f8-a5c8-43f118b3e50d-istio-token\") pod \"e2e-inf-b-ce04af-openshift-default-7dccd45f47-n5ndk\" (UID: \"5188b487-c967-42f8-a5c8-43f118b3e50d\") " pod="openshift-ingress/e2e-inf-b-ce04af-openshift-default-7dccd45f47-n5ndk" Nov 03 01:26:05.839698 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:26:05.839687 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5188b487-c967-42f8-a5c8-43f118b3e50d-credential-socket\") pod \"e2e-inf-b-ce04af-openshift-default-7dccd45f47-n5ndk\" (UID: \"5188b487-c967-42f8-a5c8-43f118b3e50d\") " pod="openshift-ingress/e2e-inf-b-ce04af-openshift-default-7dccd45f47-n5ndk" Nov 03 01:26:05.839834 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:26:05.839717 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5188b487-c967-42f8-a5c8-43f118b3e50d-istio-data\") pod \"e2e-inf-b-ce04af-openshift-default-7dccd45f47-n5ndk\" (UID: \"5188b487-c967-42f8-a5c8-43f118b3e50d\") " pod="openshift-ingress/e2e-inf-b-ce04af-openshift-default-7dccd45f47-n5ndk" Nov 03 01:26:05.839834 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:26:05.839767 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dsrt8\" (UniqueName: \"kubernetes.io/projected/5188b487-c967-42f8-a5c8-43f118b3e50d-kube-api-access-dsrt8\") pod \"e2e-inf-b-ce04af-openshift-default-7dccd45f47-n5ndk\" (UID: \"5188b487-c967-42f8-a5c8-43f118b3e50d\") " pod="openshift-ingress/e2e-inf-b-ce04af-openshift-default-7dccd45f47-n5ndk" Nov 03 01:26:05.840040 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:26:05.839842 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5188b487-c967-42f8-a5c8-43f118b3e50d-istio-envoy\") pod \"e2e-inf-b-ce04af-openshift-default-7dccd45f47-n5ndk\" (UID: \"5188b487-c967-42f8-a5c8-43f118b3e50d\") " pod="openshift-ingress/e2e-inf-b-ce04af-openshift-default-7dccd45f47-n5ndk" Nov 03 01:26:05.840040 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:26:05.839876 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5188b487-c967-42f8-a5c8-43f118b3e50d-istiod-ca-cert\") pod \"e2e-inf-b-ce04af-openshift-default-7dccd45f47-n5ndk\" (UID: \"5188b487-c967-42f8-a5c8-43f118b3e50d\") " pod="openshift-ingress/e2e-inf-b-ce04af-openshift-default-7dccd45f47-n5ndk" Nov 03 01:26:05.840040 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:26:05.839981 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5188b487-c967-42f8-a5c8-43f118b3e50d-workload-socket\") pod \"e2e-inf-b-ce04af-openshift-default-7dccd45f47-n5ndk\" (UID: \"5188b487-c967-42f8-a5c8-43f118b3e50d\") " pod="openshift-ingress/e2e-inf-b-ce04af-openshift-default-7dccd45f47-n5ndk" Nov 03 01:26:05.840040 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:26:05.840029 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5188b487-c967-42f8-a5c8-43f118b3e50d-workload-certs\") pod \"e2e-inf-b-ce04af-openshift-default-7dccd45f47-n5ndk\" (UID: \"5188b487-c967-42f8-a5c8-43f118b3e50d\") " pod="openshift-ingress/e2e-inf-b-ce04af-openshift-default-7dccd45f47-n5ndk" Nov 03 01:26:05.840283 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:26:05.840138 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5188b487-c967-42f8-a5c8-43f118b3e50d-istio-data\") pod \"e2e-inf-b-ce04af-openshift-default-7dccd45f47-n5ndk\" (UID: \"5188b487-c967-42f8-a5c8-43f118b3e50d\") " pod="openshift-ingress/e2e-inf-b-ce04af-openshift-default-7dccd45f47-n5ndk" Nov 03 01:26:05.840283 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:26:05.840194 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5188b487-c967-42f8-a5c8-43f118b3e50d-credential-socket\") pod \"e2e-inf-b-ce04af-openshift-default-7dccd45f47-n5ndk\" (UID: \"5188b487-c967-42f8-a5c8-43f118b3e50d\") " pod="openshift-ingress/e2e-inf-b-ce04af-openshift-default-7dccd45f47-n5ndk" Nov 03 01:26:05.840398 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:26:05.840376 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5188b487-c967-42f8-a5c8-43f118b3e50d-workload-certs\") pod \"e2e-inf-b-ce04af-openshift-default-7dccd45f47-n5ndk\" (UID: \"5188b487-c967-42f8-a5c8-43f118b3e50d\") " pod="openshift-ingress/e2e-inf-b-ce04af-openshift-default-7dccd45f47-n5ndk" Nov 03 01:26:05.840706 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:26:05.840688 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5188b487-c967-42f8-a5c8-43f118b3e50d-istiod-ca-cert\") pod \"e2e-inf-b-ce04af-openshift-default-7dccd45f47-n5ndk\" (UID: \"5188b487-c967-42f8-a5c8-43f118b3e50d\") " pod="openshift-ingress/e2e-inf-b-ce04af-openshift-default-7dccd45f47-n5ndk" Nov 03 01:26:05.842040 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:26:05.842022 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5188b487-c967-42f8-a5c8-43f118b3e50d-istio-envoy\") pod \"e2e-inf-b-ce04af-openshift-default-7dccd45f47-n5ndk\" (UID: \"5188b487-c967-42f8-a5c8-43f118b3e50d\") " pod="openshift-ingress/e2e-inf-b-ce04af-openshift-default-7dccd45f47-n5ndk" Nov 03 01:26:05.842451 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:26:05.842427 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5188b487-c967-42f8-a5c8-43f118b3e50d-istio-podinfo\") pod \"e2e-inf-b-ce04af-openshift-default-7dccd45f47-n5ndk\" (UID: \"5188b487-c967-42f8-a5c8-43f118b3e50d\") " pod="openshift-ingress/e2e-inf-b-ce04af-openshift-default-7dccd45f47-n5ndk" Nov 03 01:26:05.847157 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:26:05.847133 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5188b487-c967-42f8-a5c8-43f118b3e50d-istio-token\") pod \"e2e-inf-b-ce04af-openshift-default-7dccd45f47-n5ndk\" (UID: \"5188b487-c967-42f8-a5c8-43f118b3e50d\") " pod="openshift-ingress/e2e-inf-b-ce04af-openshift-default-7dccd45f47-n5ndk" Nov 03 01:26:05.848190 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:26:05.848167 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dsrt8\" (UniqueName: \"kubernetes.io/projected/5188b487-c967-42f8-a5c8-43f118b3e50d-kube-api-access-dsrt8\") pod \"e2e-inf-b-ce04af-openshift-default-7dccd45f47-n5ndk\" (UID: \"5188b487-c967-42f8-a5c8-43f118b3e50d\") " pod="openshift-ingress/e2e-inf-b-ce04af-openshift-default-7dccd45f47-n5ndk" Nov 03 01:26:05.875123 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:26:05.875085 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:26:05.924708 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:26:05.922008 4939 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-ce04af-openshift-default-7dccd45f47-n5ndk" Nov 03 01:26:06.150949 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:26:06.150880 4939 generic.go:356] "Generic (PLEG): container finished" podID="4da03a65-9706-40b4-955f-6e628901b6be" containerID="30bc7f034396870ec020bebe35ea11831db41e782506d32f1b37ace2348c0a85" exitCode=0 Nov 03 01:26:06.151211 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:26:06.151188 4939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-650be0-openshift-default-5c7c5fb7d9-tkzdb" event={"ID":"4da03a65-9706-40b4-955f-6e628901b6be","Type":"ContainerDied","Data":"30bc7f034396870ec020bebe35ea11831db41e782506d32f1b37ace2348c0a85"} Nov 03 01:26:06.151306 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:26:06.151254 4939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-650be0-openshift-default-5c7c5fb7d9-tkzdb" event={"ID":"4da03a65-9706-40b4-955f-6e628901b6be","Type":"ContainerDied","Data":"5818e364320fa9d73d59c4aabc9bef4eb863b9bee7304a6ef449cdcf823f2d40"} Nov 03 01:26:06.151306 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:26:06.151289 4939 scope.go:122] "RemoveContainer" containerID="30bc7f034396870ec020bebe35ea11831db41e782506d32f1b37ace2348c0a85" Nov 03 01:26:06.151489 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:26:06.151472 4939 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-650be0-openshift-default-5c7c5fb7d9-tkzdb" Nov 03 01:26:06.168303 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:26:06.168243 4939 scope.go:122] "RemoveContainer" containerID="b60c499c7f01628148ff6755b3ae78bf771351dcc2d9253dfbd6c45015e55519" Nov 03 01:26:06.183633 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:26:06.183599 4939 scope.go:122] "RemoveContainer" containerID="30bc7f034396870ec020bebe35ea11831db41e782506d32f1b37ace2348c0a85" Nov 03 01:26:06.183975 ip-10-0-140-107 kubenswrapper[4939]: E1103 01:26:06.183946 4939 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"30bc7f034396870ec020bebe35ea11831db41e782506d32f1b37ace2348c0a85\": container with ID starting with 30bc7f034396870ec020bebe35ea11831db41e782506d32f1b37ace2348c0a85 not found: ID does not exist" containerID="30bc7f034396870ec020bebe35ea11831db41e782506d32f1b37ace2348c0a85" Nov 03 01:26:06.184047 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:26:06.183990 4939 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"30bc7f034396870ec020bebe35ea11831db41e782506d32f1b37ace2348c0a85"} err="failed to get container status \"30bc7f034396870ec020bebe35ea11831db41e782506d32f1b37ace2348c0a85\": rpc error: code = NotFound desc = could not find container \"30bc7f034396870ec020bebe35ea11831db41e782506d32f1b37ace2348c0a85\": container with ID starting with 30bc7f034396870ec020bebe35ea11831db41e782506d32f1b37ace2348c0a85 not found: ID does not exist" Nov 03 01:26:06.184047 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:26:06.184016 4939 scope.go:122] "RemoveContainer" containerID="b60c499c7f01628148ff6755b3ae78bf771351dcc2d9253dfbd6c45015e55519" Nov 03 01:26:06.184504 ip-10-0-140-107 kubenswrapper[4939]: E1103 01:26:06.184400 4939 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b60c499c7f01628148ff6755b3ae78bf771351dcc2d9253dfbd6c45015e55519\": container with ID starting with b60c499c7f01628148ff6755b3ae78bf771351dcc2d9253dfbd6c45015e55519 not found: ID does not exist" containerID="b60c499c7f01628148ff6755b3ae78bf771351dcc2d9253dfbd6c45015e55519" Nov 03 01:26:06.184504 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:26:06.184440 4939 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b60c499c7f01628148ff6755b3ae78bf771351dcc2d9253dfbd6c45015e55519"} err="failed to get container status \"b60c499c7f01628148ff6755b3ae78bf771351dcc2d9253dfbd6c45015e55519\": rpc error: code = NotFound desc = could not find container \"b60c499c7f01628148ff6755b3ae78bf771351dcc2d9253dfbd6c45015e55519\": container with ID starting with b60c499c7f01628148ff6755b3ae78bf771351dcc2d9253dfbd6c45015e55519 not found: ID does not exist" Nov 03 01:26:06.189055 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:26:06.189025 4939 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-650be0-openshift-default-5c7c5fb7d9-tkzdb"] Nov 03 01:26:06.195549 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:26:06.195524 4939 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-650be0-openshift-default-5c7c5fb7d9-tkzdb"] Nov 03 01:26:06.271739 ip-10-0-140-107 kubenswrapper[4939]: W1103 01:26:06.271706 4939 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5188b487_c967_42f8_a5c8_43f118b3e50d.slice/crio-3b6f2d9b2771e6b7fa84fa8faf5208a4a19af848e06b0db54a7f420f5e0491bf WatchSource:0}: Error finding container 3b6f2d9b2771e6b7fa84fa8faf5208a4a19af848e06b0db54a7f420f5e0491bf: Status 404 returned error can't find the container with id 3b6f2d9b2771e6b7fa84fa8faf5208a4a19af848e06b0db54a7f420f5e0491bf Nov 03 01:26:06.272450 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:26:06.272357 4939 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-ce04af-openshift-default-7dccd45f47-n5ndk"] Nov 03 01:26:06.275264 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:26:06.275226 4939 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 01:26:06.275369 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:26:06.275299 4939 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 01:26:06.275369 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:26:06.275331 4939 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 01:26:06.582417 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:26:06.582380 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:26:07.183580 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:26:07.183537 4939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-ce04af-openshift-default-7dccd45f47-n5ndk" event={"ID":"5188b487-c967-42f8-a5c8-43f118b3e50d","Type":"ContainerStarted","Data":"9c4f71736d18633fc600dc773a00b3ec7b3814be88b7cebaf2e86d5fac01463c"} Nov 03 01:26:07.183812 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:26:07.183601 4939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-ce04af-openshift-default-7dccd45f47-n5ndk" event={"ID":"5188b487-c967-42f8-a5c8-43f118b3e50d","Type":"ContainerStarted","Data":"3b6f2d9b2771e6b7fa84fa8faf5208a4a19af848e06b0db54a7f420f5e0491bf"} Nov 03 01:26:07.208642 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:26:07.208588 4939 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-b-ce04af-openshift-default-7dccd45f47-n5ndk" podStartSLOduration=72.208575648 podStartE2EDuration="1m12.208575648s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 01:24:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 01:26:07.202537932 +0000 UTC m=+2715.466872887" watchObservedRunningTime="2026-11-03 01:26:07.208575648 +0000 UTC m=+2715.472910592" Nov 03 01:26:07.296961 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:26:07.296871 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:26:07.885200 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:26:07.885164 4939 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4da03a65-9706-40b4-955f-6e628901b6be" path="/var/lib/kubelet/pods/4da03a65-9706-40b4-955f-6e628901b6be/volumes" Nov 03 01:26:07.923178 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:26:07.923140 4939 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-ce04af-openshift-default-7dccd45f47-n5ndk" Nov 03 01:26:07.924645 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:26:07.924617 4939 patch_prober.go:29] interesting pod/e2e-inf-b-ce04af-openshift-default-7dccd45f47-n5ndk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.48:15021/healthz/ready\": dial tcp 10.132.1.48:15021: connect: connection refused" start-of-body= Nov 03 01:26:07.924774 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:26:07.924668 4939 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-ce04af-openshift-default-7dccd45f47-n5ndk" podUID="5188b487-c967-42f8-a5c8-43f118b3e50d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.48:15021/healthz/ready\": dial tcp 10.132.1.48:15021: connect: connection refused" Nov 03 01:26:07.997379 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:26:07.997339 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:26:08.799885 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:26:08.799844 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:26:08.939396 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:26:08.931615 4939 patch_prober.go:29] interesting pod/e2e-inf-b-ce04af-openshift-default-7dccd45f47-n5ndk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.48:15021/healthz/ready\": dial tcp 10.132.1.48:15021: connect: connection refused" start-of-body= Nov 03 01:26:08.942538 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:26:08.942491 4939 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-ce04af-openshift-default-7dccd45f47-n5ndk" podUID="5188b487-c967-42f8-a5c8-43f118b3e50d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.48:15021/healthz/ready\": dial tcp 10.132.1.48:15021: connect: connection refused" Nov 03 01:26:09.789000 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:26:09.788960 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:26:09.924956 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:26:09.923339 4939 patch_prober.go:29] interesting pod/e2e-inf-b-ce04af-openshift-default-7dccd45f47-n5ndk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.48:15021/healthz/ready\": dial tcp 10.132.1.48:15021: connect: connection refused" start-of-body= Nov 03 01:26:09.924956 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:26:09.923438 4939 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-ce04af-openshift-default-7dccd45f47-n5ndk" podUID="5188b487-c967-42f8-a5c8-43f118b3e50d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.48:15021/healthz/ready\": dial tcp 10.132.1.48:15021: connect: connection refused" Nov 03 01:26:10.724562 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:26:10.724536 4939 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w0-50e623-openshift-default-6c86dc794f-5m78z" Nov 03 01:26:10.798207 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:26:10.798115 4939 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-b-ce04af-openshift-default-7dccd45f47-wc5pz"] Nov 03 01:26:10.799741 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:26:10.799705 4939 cpu_manager.go:414] "containerMap: removing container" podUID="64952141-7a68-4e46-8982-3db9402620e3" containerName="istio-proxy" Nov 03 01:26:10.799741 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:26:10.799740 4939 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="64952141-7a68-4e46-8982-3db9402620e3" containerName="istio-proxy" Nov 03 01:26:10.800076 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:26:10.800055 4939 memory_manager.go:365] "RemoveStaleState removing state" podUID="64952141-7a68-4e46-8982-3db9402620e3" containerName="istio-proxy" Nov 03 01:26:10.805519 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:26:10.805490 4939 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-ce04af-openshift-default-7dccd45f47-wc5pz"] Nov 03 01:26:10.805637 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:26:10.805609 4939 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-ce04af-openshift-default-7dccd45f47-wc5pz" Nov 03 01:26:10.810628 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:26:10.810595 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:26:10.825367 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:26:10.825342 4939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/64952141-7a68-4e46-8982-3db9402620e3-kube-api-access-psmpl\" (UniqueName: \"kubernetes.io/projected/64952141-7a68-4e46-8982-3db9402620e3-kube-api-access-psmpl\") pod \"64952141-7a68-4e46-8982-3db9402620e3\" (UID: \"64952141-7a68-4e46-8982-3db9402620e3\") " Nov 03 01:26:10.825509 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:26:10.825381 4939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/64952141-7a68-4e46-8982-3db9402620e3-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/64952141-7a68-4e46-8982-3db9402620e3-credential-socket\") pod \"64952141-7a68-4e46-8982-3db9402620e3\" (UID: \"64952141-7a68-4e46-8982-3db9402620e3\") " Nov 03 01:26:10.825509 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:26:10.825481 4939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/64952141-7a68-4e46-8982-3db9402620e3-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/64952141-7a68-4e46-8982-3db9402620e3-workload-certs\") pod \"64952141-7a68-4e46-8982-3db9402620e3\" (UID: \"64952141-7a68-4e46-8982-3db9402620e3\") " Nov 03 01:26:10.825663 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:26:10.825533 4939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/64952141-7a68-4e46-8982-3db9402620e3-istio-token\" (UniqueName: \"kubernetes.io/projected/64952141-7a68-4e46-8982-3db9402620e3-istio-token\") pod \"64952141-7a68-4e46-8982-3db9402620e3\" (UID: \"64952141-7a68-4e46-8982-3db9402620e3\") " Nov 03 01:26:10.825663 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:26:10.825593 4939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/64952141-7a68-4e46-8982-3db9402620e3-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/64952141-7a68-4e46-8982-3db9402620e3-istio-envoy\") pod \"64952141-7a68-4e46-8982-3db9402620e3\" (UID: \"64952141-7a68-4e46-8982-3db9402620e3\") " Nov 03 01:26:10.825663 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:26:10.825640 4939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/64952141-7a68-4e46-8982-3db9402620e3-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/64952141-7a68-4e46-8982-3db9402620e3-workload-socket\") pod \"64952141-7a68-4e46-8982-3db9402620e3\" (UID: \"64952141-7a68-4e46-8982-3db9402620e3\") " Nov 03 01:26:10.825820 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:26:10.825733 4939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/64952141-7a68-4e46-8982-3db9402620e3-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/64952141-7a68-4e46-8982-3db9402620e3-istio-podinfo\") pod \"64952141-7a68-4e46-8982-3db9402620e3\" (UID: \"64952141-7a68-4e46-8982-3db9402620e3\") " Nov 03 01:26:10.825882 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:26:10.825863 4939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/64952141-7a68-4e46-8982-3db9402620e3-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/64952141-7a68-4e46-8982-3db9402620e3-istiod-ca-cert\") pod \"64952141-7a68-4e46-8982-3db9402620e3\" (UID: \"64952141-7a68-4e46-8982-3db9402620e3\") " Nov 03 01:26:10.825969 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:26:10.825891 4939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/64952141-7a68-4e46-8982-3db9402620e3-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/64952141-7a68-4e46-8982-3db9402620e3-istio-data\") pod \"64952141-7a68-4e46-8982-3db9402620e3\" (UID: \"64952141-7a68-4e46-8982-3db9402620e3\") " Nov 03 01:26:10.826425 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:26:10.826395 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0412ca91-563b-4ee7-8a02-4f4e768e51f7-workload-certs\") pod \"e2e-inf-b-ce04af-openshift-default-7dccd45f47-wc5pz\" (UID: \"0412ca91-563b-4ee7-8a02-4f4e768e51f7\") " pod="openshift-ingress/e2e-inf-b-ce04af-openshift-default-7dccd45f47-wc5pz" Nov 03 01:26:10.826523 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:26:10.826475 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0412ca91-563b-4ee7-8a02-4f4e768e51f7-istio-token\") pod \"e2e-inf-b-ce04af-openshift-default-7dccd45f47-wc5pz\" (UID: \"0412ca91-563b-4ee7-8a02-4f4e768e51f7\") " pod="openshift-ingress/e2e-inf-b-ce04af-openshift-default-7dccd45f47-wc5pz" Nov 03 01:26:10.826523 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:26:10.826503 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0412ca91-563b-4ee7-8a02-4f4e768e51f7-istiod-ca-cert\") pod \"e2e-inf-b-ce04af-openshift-default-7dccd45f47-wc5pz\" (UID: \"0412ca91-563b-4ee7-8a02-4f4e768e51f7\") " pod="openshift-ingress/e2e-inf-b-ce04af-openshift-default-7dccd45f47-wc5pz" Nov 03 01:26:10.826679 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:26:10.826659 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8lpdh\" (UniqueName: \"kubernetes.io/projected/0412ca91-563b-4ee7-8a02-4f4e768e51f7-kube-api-access-8lpdh\") pod \"e2e-inf-b-ce04af-openshift-default-7dccd45f47-wc5pz\" (UID: \"0412ca91-563b-4ee7-8a02-4f4e768e51f7\") " pod="openshift-ingress/e2e-inf-b-ce04af-openshift-default-7dccd45f47-wc5pz" Nov 03 01:26:10.826801 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:26:10.826781 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0412ca91-563b-4ee7-8a02-4f4e768e51f7-istio-podinfo\") pod \"e2e-inf-b-ce04af-openshift-default-7dccd45f47-wc5pz\" (UID: \"0412ca91-563b-4ee7-8a02-4f4e768e51f7\") " pod="openshift-ingress/e2e-inf-b-ce04af-openshift-default-7dccd45f47-wc5pz" Nov 03 01:26:10.826966 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:26:10.826944 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0412ca91-563b-4ee7-8a02-4f4e768e51f7-credential-socket\") pod \"e2e-inf-b-ce04af-openshift-default-7dccd45f47-wc5pz\" (UID: \"0412ca91-563b-4ee7-8a02-4f4e768e51f7\") " pod="openshift-ingress/e2e-inf-b-ce04af-openshift-default-7dccd45f47-wc5pz" Nov 03 01:26:10.827038 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:26:10.826992 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0412ca91-563b-4ee7-8a02-4f4e768e51f7-istio-envoy\") pod \"e2e-inf-b-ce04af-openshift-default-7dccd45f47-wc5pz\" (UID: \"0412ca91-563b-4ee7-8a02-4f4e768e51f7\") " pod="openshift-ingress/e2e-inf-b-ce04af-openshift-default-7dccd45f47-wc5pz" Nov 03 01:26:10.827097 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:26:10.827083 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0412ca91-563b-4ee7-8a02-4f4e768e51f7-istio-data\") pod \"e2e-inf-b-ce04af-openshift-default-7dccd45f47-wc5pz\" (UID: \"0412ca91-563b-4ee7-8a02-4f4e768e51f7\") " pod="openshift-ingress/e2e-inf-b-ce04af-openshift-default-7dccd45f47-wc5pz" Nov 03 01:26:10.827265 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:26:10.827243 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0412ca91-563b-4ee7-8a02-4f4e768e51f7-workload-socket\") pod \"e2e-inf-b-ce04af-openshift-default-7dccd45f47-wc5pz\" (UID: \"0412ca91-563b-4ee7-8a02-4f4e768e51f7\") " pod="openshift-ingress/e2e-inf-b-ce04af-openshift-default-7dccd45f47-wc5pz" Nov 03 01:26:10.827987 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:26:10.826394 4939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/64952141-7a68-4e46-8982-3db9402620e3-workload-socket" pod "64952141-7a68-4e46-8982-3db9402620e3" (UID: "64952141-7a68-4e46-8982-3db9402620e3"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 01:26:10.827987 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:26:10.826576 4939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/64952141-7a68-4e46-8982-3db9402620e3-credential-socket" pod "64952141-7a68-4e46-8982-3db9402620e3" (UID: "64952141-7a68-4e46-8982-3db9402620e3"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 01:26:10.828166 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:26:10.827679 4939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/64952141-7a68-4e46-8982-3db9402620e3-istiod-ca-cert" pod "64952141-7a68-4e46-8982-3db9402620e3" (UID: "64952141-7a68-4e46-8982-3db9402620e3"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 01:26:10.828166 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:26:10.827927 4939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/64952141-7a68-4e46-8982-3db9402620e3-istio-data" pod "64952141-7a68-4e46-8982-3db9402620e3" (UID: "64952141-7a68-4e46-8982-3db9402620e3"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 01:26:10.828340 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:26:10.828283 4939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/64952141-7a68-4e46-8982-3db9402620e3-workload-certs" pod "64952141-7a68-4e46-8982-3db9402620e3" (UID: "64952141-7a68-4e46-8982-3db9402620e3"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 01:26:10.830021 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:26:10.829993 4939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/64952141-7a68-4e46-8982-3db9402620e3-istio-envoy" pod "64952141-7a68-4e46-8982-3db9402620e3" (UID: "64952141-7a68-4e46-8982-3db9402620e3"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 01:26:10.831346 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:26:10.831299 4939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/64952141-7a68-4e46-8982-3db9402620e3-istio-token" pod "64952141-7a68-4e46-8982-3db9402620e3" (UID: "64952141-7a68-4e46-8982-3db9402620e3"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 01:26:10.831620 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:26:10.831561 4939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/64952141-7a68-4e46-8982-3db9402620e3-kube-api-access-psmpl" pod "64952141-7a68-4e46-8982-3db9402620e3" (UID: "64952141-7a68-4e46-8982-3db9402620e3"). InnerVolumeSpecName "kube-api-access-psmpl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 01:26:10.831620 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:26:10.831574 4939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/64952141-7a68-4e46-8982-3db9402620e3-istio-podinfo" pod "64952141-7a68-4e46-8982-3db9402620e3" (UID: "64952141-7a68-4e46-8982-3db9402620e3"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 01:26:11.018936 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:26:10.944241 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0412ca91-563b-4ee7-8a02-4f4e768e51f7-istio-token\") pod \"e2e-inf-b-ce04af-openshift-default-7dccd45f47-wc5pz\" (UID: \"0412ca91-563b-4ee7-8a02-4f4e768e51f7\") " pod="openshift-ingress/e2e-inf-b-ce04af-openshift-default-7dccd45f47-wc5pz" Nov 03 01:26:11.026936 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:26:11.021565 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0412ca91-563b-4ee7-8a02-4f4e768e51f7-istiod-ca-cert\") pod \"e2e-inf-b-ce04af-openshift-default-7dccd45f47-wc5pz\" (UID: \"0412ca91-563b-4ee7-8a02-4f4e768e51f7\") " pod="openshift-ingress/e2e-inf-b-ce04af-openshift-default-7dccd45f47-wc5pz" Nov 03 01:26:11.026936 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:26:11.021790 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8lpdh\" (UniqueName: \"kubernetes.io/projected/0412ca91-563b-4ee7-8a02-4f4e768e51f7-kube-api-access-8lpdh\") pod \"e2e-inf-b-ce04af-openshift-default-7dccd45f47-wc5pz\" (UID: \"0412ca91-563b-4ee7-8a02-4f4e768e51f7\") " pod="openshift-ingress/e2e-inf-b-ce04af-openshift-default-7dccd45f47-wc5pz" Nov 03 01:26:11.026936 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:26:11.021931 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0412ca91-563b-4ee7-8a02-4f4e768e51f7-istio-podinfo\") pod \"e2e-inf-b-ce04af-openshift-default-7dccd45f47-wc5pz\" (UID: \"0412ca91-563b-4ee7-8a02-4f4e768e51f7\") " pod="openshift-ingress/e2e-inf-b-ce04af-openshift-default-7dccd45f47-wc5pz" Nov 03 01:26:11.026936 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:26:11.022086 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0412ca91-563b-4ee7-8a02-4f4e768e51f7-credential-socket\") pod \"e2e-inf-b-ce04af-openshift-default-7dccd45f47-wc5pz\" (UID: \"0412ca91-563b-4ee7-8a02-4f4e768e51f7\") " pod="openshift-ingress/e2e-inf-b-ce04af-openshift-default-7dccd45f47-wc5pz" Nov 03 01:26:11.026936 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:26:11.022124 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0412ca91-563b-4ee7-8a02-4f4e768e51f7-istio-envoy\") pod \"e2e-inf-b-ce04af-openshift-default-7dccd45f47-wc5pz\" (UID: \"0412ca91-563b-4ee7-8a02-4f4e768e51f7\") " pod="openshift-ingress/e2e-inf-b-ce04af-openshift-default-7dccd45f47-wc5pz" Nov 03 01:26:11.026936 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:26:11.022213 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0412ca91-563b-4ee7-8a02-4f4e768e51f7-istio-data\") pod \"e2e-inf-b-ce04af-openshift-default-7dccd45f47-wc5pz\" (UID: \"0412ca91-563b-4ee7-8a02-4f4e768e51f7\") " pod="openshift-ingress/e2e-inf-b-ce04af-openshift-default-7dccd45f47-wc5pz" Nov 03 01:26:11.026936 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:26:11.022392 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0412ca91-563b-4ee7-8a02-4f4e768e51f7-workload-socket\") pod \"e2e-inf-b-ce04af-openshift-default-7dccd45f47-wc5pz\" (UID: \"0412ca91-563b-4ee7-8a02-4f4e768e51f7\") " pod="openshift-ingress/e2e-inf-b-ce04af-openshift-default-7dccd45f47-wc5pz" Nov 03 01:26:11.026936 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:26:11.022545 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0412ca91-563b-4ee7-8a02-4f4e768e51f7-workload-certs\") pod \"e2e-inf-b-ce04af-openshift-default-7dccd45f47-wc5pz\" (UID: \"0412ca91-563b-4ee7-8a02-4f4e768e51f7\") " pod="openshift-ingress/e2e-inf-b-ce04af-openshift-default-7dccd45f47-wc5pz" Nov 03 01:26:11.026936 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:26:11.022686 4939 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/64952141-7a68-4e46-8982-3db9402620e3-istiod-ca-cert\") on node \"ip-10-0-140-107.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:26:11.026936 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:26:11.022704 4939 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/64952141-7a68-4e46-8982-3db9402620e3-istio-data\") on node \"ip-10-0-140-107.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:26:11.026936 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:26:11.022719 4939 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-psmpl\" (UniqueName: \"kubernetes.io/projected/64952141-7a68-4e46-8982-3db9402620e3-kube-api-access-psmpl\") on node \"ip-10-0-140-107.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:26:11.026936 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:26:11.022734 4939 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/64952141-7a68-4e46-8982-3db9402620e3-credential-socket\") on node \"ip-10-0-140-107.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:26:11.026936 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:26:11.022751 4939 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/64952141-7a68-4e46-8982-3db9402620e3-workload-certs\") on node \"ip-10-0-140-107.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:26:11.026936 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:26:11.022766 4939 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/64952141-7a68-4e46-8982-3db9402620e3-istio-token\") on node \"ip-10-0-140-107.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:26:11.026936 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:26:11.022779 4939 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/64952141-7a68-4e46-8982-3db9402620e3-istio-envoy\") on node \"ip-10-0-140-107.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:26:11.026936 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:26:11.022793 4939 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/64952141-7a68-4e46-8982-3db9402620e3-workload-socket\") on node \"ip-10-0-140-107.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:26:11.026936 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:26:11.022806 4939 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/64952141-7a68-4e46-8982-3db9402620e3-istio-podinfo\") on node \"ip-10-0-140-107.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:26:11.026936 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:26:11.023157 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0412ca91-563b-4ee7-8a02-4f4e768e51f7-workload-certs\") pod \"e2e-inf-b-ce04af-openshift-default-7dccd45f47-wc5pz\" (UID: \"0412ca91-563b-4ee7-8a02-4f4e768e51f7\") " pod="openshift-ingress/e2e-inf-b-ce04af-openshift-default-7dccd45f47-wc5pz" Nov 03 01:26:11.026936 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:26:10.974372 4939 patch_prober.go:29] interesting pod/e2e-inf-b-ce04af-openshift-default-7dccd45f47-n5ndk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.48:15021/healthz/ready\": dial tcp 10.132.1.48:15021: connect: connection refused" start-of-body= Nov 03 01:26:11.026936 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:26:11.023244 4939 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-ce04af-openshift-default-7dccd45f47-n5ndk" podUID="5188b487-c967-42f8-a5c8-43f118b3e50d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.48:15021/healthz/ready\": dial tcp 10.132.1.48:15021: connect: connection refused" Nov 03 01:26:11.026936 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:26:11.024775 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0412ca91-563b-4ee7-8a02-4f4e768e51f7-credential-socket\") pod \"e2e-inf-b-ce04af-openshift-default-7dccd45f47-wc5pz\" (UID: \"0412ca91-563b-4ee7-8a02-4f4e768e51f7\") " pod="openshift-ingress/e2e-inf-b-ce04af-openshift-default-7dccd45f47-wc5pz" Nov 03 01:26:11.026936 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:26:11.019871 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0412ca91-563b-4ee7-8a02-4f4e768e51f7-istio-token\") pod \"e2e-inf-b-ce04af-openshift-default-7dccd45f47-wc5pz\" (UID: \"0412ca91-563b-4ee7-8a02-4f4e768e51f7\") " pod="openshift-ingress/e2e-inf-b-ce04af-openshift-default-7dccd45f47-wc5pz" Nov 03 01:26:11.027792 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:26:11.027224 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0412ca91-563b-4ee7-8a02-4f4e768e51f7-istio-data\") pod \"e2e-inf-b-ce04af-openshift-default-7dccd45f47-wc5pz\" (UID: \"0412ca91-563b-4ee7-8a02-4f4e768e51f7\") " pod="openshift-ingress/e2e-inf-b-ce04af-openshift-default-7dccd45f47-wc5pz" Nov 03 01:26:11.027792 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:26:11.027454 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0412ca91-563b-4ee7-8a02-4f4e768e51f7-workload-socket\") pod \"e2e-inf-b-ce04af-openshift-default-7dccd45f47-wc5pz\" (UID: \"0412ca91-563b-4ee7-8a02-4f4e768e51f7\") " pod="openshift-ingress/e2e-inf-b-ce04af-openshift-default-7dccd45f47-wc5pz" Nov 03 01:26:11.028761 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:26:11.028732 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0412ca91-563b-4ee7-8a02-4f4e768e51f7-istiod-ca-cert\") pod \"e2e-inf-b-ce04af-openshift-default-7dccd45f47-wc5pz\" (UID: \"0412ca91-563b-4ee7-8a02-4f4e768e51f7\") " pod="openshift-ingress/e2e-inf-b-ce04af-openshift-default-7dccd45f47-wc5pz" Nov 03 01:26:11.040855 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:26:11.040216 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0412ca91-563b-4ee7-8a02-4f4e768e51f7-istio-envoy\") pod \"e2e-inf-b-ce04af-openshift-default-7dccd45f47-wc5pz\" (UID: \"0412ca91-563b-4ee7-8a02-4f4e768e51f7\") " pod="openshift-ingress/e2e-inf-b-ce04af-openshift-default-7dccd45f47-wc5pz" Nov 03 01:26:11.040855 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:26:11.040567 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0412ca91-563b-4ee7-8a02-4f4e768e51f7-istio-podinfo\") pod \"e2e-inf-b-ce04af-openshift-default-7dccd45f47-wc5pz\" (UID: \"0412ca91-563b-4ee7-8a02-4f4e768e51f7\") " pod="openshift-ingress/e2e-inf-b-ce04af-openshift-default-7dccd45f47-wc5pz" Nov 03 01:26:11.041151 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:26:11.041078 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8lpdh\" (UniqueName: \"kubernetes.io/projected/0412ca91-563b-4ee7-8a02-4f4e768e51f7-kube-api-access-8lpdh\") pod \"e2e-inf-b-ce04af-openshift-default-7dccd45f47-wc5pz\" (UID: \"0412ca91-563b-4ee7-8a02-4f4e768e51f7\") " pod="openshift-ingress/e2e-inf-b-ce04af-openshift-default-7dccd45f47-wc5pz" Nov 03 01:26:11.130600 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:26:11.130566 4939 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-ce04af-openshift-default-7dccd45f47-wc5pz" Nov 03 01:26:11.230419 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:26:11.230388 4939 generic.go:356] "Generic (PLEG): container finished" podID="64952141-7a68-4e46-8982-3db9402620e3" containerID="6eea49382c75191aaa1998e3e23c1dea09f9bdc053a349a06b05b89a82ed2443" exitCode=0 Nov 03 01:26:11.230692 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:26:11.230668 4939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w0-50e623-openshift-default-6c86dc794f-5m78z" event={"ID":"64952141-7a68-4e46-8982-3db9402620e3","Type":"ContainerDied","Data":"6eea49382c75191aaa1998e3e23c1dea09f9bdc053a349a06b05b89a82ed2443"} Nov 03 01:26:11.230794 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:26:11.230737 4939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w0-50e623-openshift-default-6c86dc794f-5m78z" event={"ID":"64952141-7a68-4e46-8982-3db9402620e3","Type":"ContainerDied","Data":"b0d0da6db254ebda89724e2ecad9bee3881c3cfec3d6022971f2e987a069127f"} Nov 03 01:26:11.230794 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:26:11.230771 4939 scope.go:122] "RemoveContainer" containerID="6eea49382c75191aaa1998e3e23c1dea09f9bdc053a349a06b05b89a82ed2443" Nov 03 01:26:11.230976 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:26:11.230959 4939 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w0-50e623-openshift-default-6c86dc794f-5m78z" Nov 03 01:26:11.302202 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:26:11.300377 4939 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w0-50e623-openshift-default-6c86dc794f-5m78z"] Nov 03 01:26:11.303689 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:26:11.303662 4939 scope.go:122] "RemoveContainer" containerID="6eea49382c75191aaa1998e3e23c1dea09f9bdc053a349a06b05b89a82ed2443" Nov 03 01:26:11.304101 ip-10-0-140-107 kubenswrapper[4939]: E1103 01:26:11.304067 4939 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6eea49382c75191aaa1998e3e23c1dea09f9bdc053a349a06b05b89a82ed2443\": container with ID starting with 6eea49382c75191aaa1998e3e23c1dea09f9bdc053a349a06b05b89a82ed2443 not found: ID does not exist" containerID="6eea49382c75191aaa1998e3e23c1dea09f9bdc053a349a06b05b89a82ed2443" Nov 03 01:26:11.304202 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:26:11.304112 4939 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6eea49382c75191aaa1998e3e23c1dea09f9bdc053a349a06b05b89a82ed2443"} err="failed to get container status \"6eea49382c75191aaa1998e3e23c1dea09f9bdc053a349a06b05b89a82ed2443\": rpc error: code = NotFound desc = could not find container \"6eea49382c75191aaa1998e3e23c1dea09f9bdc053a349a06b05b89a82ed2443\": container with ID starting with 6eea49382c75191aaa1998e3e23c1dea09f9bdc053a349a06b05b89a82ed2443 not found: ID does not exist" Nov 03 01:26:11.307256 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:26:11.307214 4939 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w0-50e623-openshift-default-6c86dc794f-5m78z"] Nov 03 01:26:11.483091 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:26:11.483059 4939 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-ce04af-openshift-default-7dccd45f47-wc5pz"] Nov 03 01:26:11.488325 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:26:11.488291 4939 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 01:26:11.488436 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:26:11.488364 4939 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 01:26:11.488436 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:26:11.488396 4939 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 01:26:11.600600 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:26:11.600564 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:26:11.721688 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:26:11.721648 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:26:11.909426 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:26:11.909393 4939 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="64952141-7a68-4e46-8982-3db9402620e3" path="/var/lib/kubelet/pods/64952141-7a68-4e46-8982-3db9402620e3/volumes" Nov 03 01:26:11.923623 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:26:11.923593 4939 patch_prober.go:29] interesting pod/e2e-inf-b-ce04af-openshift-default-7dccd45f47-n5ndk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.48:15021/healthz/ready\": dial tcp 10.132.1.48:15021: connect: connection refused" start-of-body= Nov 03 01:26:11.923812 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:26:11.923644 4939 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-ce04af-openshift-default-7dccd45f47-n5ndk" podUID="5188b487-c967-42f8-a5c8-43f118b3e50d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.48:15021/healthz/ready\": dial tcp 10.132.1.48:15021: connect: connection refused" Nov 03 01:26:12.106572 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:26:12.106535 4939 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-4b17ac-84f44c7d99-8286l" Nov 03 01:26:12.171798 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:26:12.171699 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:26:12.328227 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:26:12.327257 4939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-ce04af-openshift-default-7dccd45f47-wc5pz" event={"ID":"0412ca91-563b-4ee7-8a02-4f4e768e51f7","Type":"ContainerStarted","Data":"a1f057279f28d75acdb60e495251a558ad6fdfa1ff112c36890e28722ac1cf25"} Nov 03 01:26:12.328227 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:26:12.327327 4939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-ce04af-openshift-default-7dccd45f47-wc5pz" event={"ID":"0412ca91-563b-4ee7-8a02-4f4e768e51f7","Type":"ContainerStarted","Data":"0656c94b54b221a37dc490a8655c19830f40173e1fc028892fdfbc3a2b6576c8"} Nov 03 01:26:12.924277 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:26:12.924245 4939 patch_prober.go:29] interesting pod/e2e-inf-b-ce04af-openshift-default-7dccd45f47-n5ndk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.48:15021/healthz/ready\": dial tcp 10.132.1.48:15021: connect: connection refused" start-of-body= Nov 03 01:26:12.924823 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:26:12.924325 4939 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-ce04af-openshift-default-7dccd45f47-n5ndk" podUID="5188b487-c967-42f8-a5c8-43f118b3e50d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.48:15021/healthz/ready\": dial tcp 10.132.1.48:15021: connect: connection refused" Nov 03 01:26:13.000875 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:26:13.000832 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:26:13.141196 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:26:13.138214 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:26:13.141196 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:26:13.138650 4939 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-ce04af-openshift-default-7dccd45f47-wc5pz" Nov 03 01:26:13.143191 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:26:13.143143 4939 patch_prober.go:29] interesting pod/e2e-inf-b-ce04af-openshift-default-7dccd45f47-wc5pz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.49:15021/healthz/ready\": dial tcp 10.132.1.49:15021: connect: connection refused" start-of-body= Nov 03 01:26:13.143430 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:26:13.143387 4939 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-ce04af-openshift-default-7dccd45f47-wc5pz" podUID="0412ca91-563b-4ee7-8a02-4f4e768e51f7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.49:15021/healthz/ready\": dial tcp 10.132.1.49:15021: connect: connection refused" Nov 03 01:26:13.541934 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:26:13.534510 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:26:13.933376 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:26:13.932369 4939 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-ce04af-openshift-default-7dccd45f47-n5ndk" Nov 03 01:26:13.933961 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:26:13.933940 4939 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-ce04af-openshift-default-7dccd45f47-n5ndk" Nov 03 01:26:13.935304 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:26:13.935283 4939 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-ce04af-openshift-default-7dccd45f47-n5ndk" Nov 03 01:26:13.986094 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:26:13.986038 4939 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-b-ce04af-openshift-default-7dccd45f47-wc5pz" podStartSLOduration=78.986019123 podStartE2EDuration="1m18.986019123s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 01:24:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 01:26:12.386561553 +0000 UTC m=+2720.650896499" watchObservedRunningTime="2026-11-03 01:26:13.986019123 +0000 UTC m=+2722.250354068" Nov 03 01:26:14.132978 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:26:14.132940 4939 patch_prober.go:29] interesting pod/e2e-inf-b-ce04af-openshift-default-7dccd45f47-wc5pz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.49:15021/healthz/ready\": dial tcp 10.132.1.49:15021: connect: connection refused" start-of-body= Nov 03 01:26:14.133210 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:26:14.133007 4939 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-ce04af-openshift-default-7dccd45f47-wc5pz" podUID="0412ca91-563b-4ee7-8a02-4f4e768e51f7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.49:15021/healthz/ready\": dial tcp 10.132.1.49:15021: connect: connection refused" Nov 03 01:26:14.497549 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:26:14.497509 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:26:15.131439 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:26:15.131405 4939 patch_prober.go:29] interesting pod/e2e-inf-b-ce04af-openshift-default-7dccd45f47-wc5pz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.49:15021/healthz/ready\": dial tcp 10.132.1.49:15021: connect: connection refused" start-of-body= Nov 03 01:26:15.131981 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:26:15.131460 4939 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-ce04af-openshift-default-7dccd45f47-wc5pz" podUID="0412ca91-563b-4ee7-8a02-4f4e768e51f7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.49:15021/healthz/ready\": dial tcp 10.132.1.49:15021: connect: connection refused" Nov 03 01:26:15.201941 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:26:15.201867 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:26:16.077361 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:26:16.077297 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:26:16.136951 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:26:16.135036 4939 patch_prober.go:29] interesting pod/e2e-inf-b-ce04af-openshift-default-7dccd45f47-wc5pz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.49:15021/healthz/ready\": dial tcp 10.132.1.49:15021: connect: connection refused" start-of-body= Nov 03 01:26:16.136951 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:26:16.135101 4939 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-ce04af-openshift-default-7dccd45f47-wc5pz" podUID="0412ca91-563b-4ee7-8a02-4f4e768e51f7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.49:15021/healthz/ready\": dial tcp 10.132.1.49:15021: connect: connection refused" Nov 03 01:26:16.908446 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:26:16.908417 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:26:17.133182 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:26:17.133137 4939 patch_prober.go:29] interesting pod/e2e-inf-b-ce04af-openshift-default-7dccd45f47-wc5pz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.49:15021/healthz/ready\": dial tcp 10.132.1.49:15021: connect: connection refused" start-of-body= Nov 03 01:26:17.133397 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:26:17.133208 4939 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-ce04af-openshift-default-7dccd45f47-wc5pz" podUID="0412ca91-563b-4ee7-8a02-4f4e768e51f7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.49:15021/healthz/ready\": dial tcp 10.132.1.49:15021: connect: connection refused" Nov 03 01:26:17.690651 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:26:17.690608 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:26:18.134688 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:26:18.134648 4939 patch_prober.go:29] interesting pod/e2e-inf-b-ce04af-openshift-default-7dccd45f47-wc5pz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.49:15021/healthz/ready\": dial tcp 10.132.1.49:15021: connect: connection refused" start-of-body= Nov 03 01:26:18.134933 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:26:18.134758 4939 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-ce04af-openshift-default-7dccd45f47-wc5pz" podUID="0412ca91-563b-4ee7-8a02-4f4e768e51f7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.49:15021/healthz/ready\": dial tcp 10.132.1.49:15021: connect: connection refused" Nov 03 01:26:18.209175 ip-10-0-140-107 kubenswrapper[4939]: E1103 01:26:18.209134 4939 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-sthjh" containerName="extractor" Nov 03 01:26:18.970807 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:26:18.970771 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:26:19.131425 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:26:19.131384 4939 patch_prober.go:29] interesting pod/e2e-inf-b-ce04af-openshift-default-7dccd45f47-wc5pz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.49:15021/healthz/ready\": dial tcp 10.132.1.49:15021: connect: connection refused" start-of-body= Nov 03 01:26:19.131599 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:26:19.131455 4939 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-ce04af-openshift-default-7dccd45f47-wc5pz" podUID="0412ca91-563b-4ee7-8a02-4f4e768e51f7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.49:15021/healthz/ready\": dial tcp 10.132.1.49:15021: connect: connection refused" Nov 03 01:26:19.772899 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:26:19.772861 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:26:20.136797 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:26:20.136762 4939 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-ce04af-openshift-default-7dccd45f47-wc5pz" Nov 03 01:26:20.137289 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:26:20.137263 4939 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-ce04af-openshift-default-7dccd45f47-wc5pz" Nov 03 01:26:20.138113 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:26:20.138086 4939 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-ce04af-openshift-default-7dccd45f47-wc5pz" Nov 03 01:26:20.492998 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:26:20.492866 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:26:21.168884 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:26:21.168840 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:26:21.911594 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:26:21.911558 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:26:22.610757 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:26:22.610719 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:26:22.715747 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:26:22.715712 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:26:22.823815 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:26:22.823772 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:26:23.373364 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:26:23.373329 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:26:23.483617 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:26:23.483525 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:26:24.215662 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:26:24.215626 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:26:25.101144 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:26:25.101108 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:26:25.783283 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:26:25.783252 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:26:26.427170 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:26:26.427135 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:26:27.215637 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:26:27.215604 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:26:28.389424 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:26:28.389387 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:26:29.002109 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:26:29.002064 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:26:29.639375 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:26:29.638931 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:26:30.290357 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:26:30.290318 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:26:31.026943 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:26:31.024984 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:26:31.661934 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:26:31.657701 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:26:32.284414 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:26:32.284384 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:26:32.599752 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:26:32.599719 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:26:32.991005 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:26:32.990878 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:26:33.103140 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:26:33.103108 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:26:33.467027 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:26:33.466991 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:26:33.633142 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:26:33.633106 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:26:34.206371 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:26:34.206328 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:26:35.056438 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:26:35.056408 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:26:35.803402 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:26:35.803363 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:26:36.585052 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:26:36.585016 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:26:36.710940 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:26:36.710887 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:26:37.638891 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:26:37.638853 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:26:38.277676 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:26:38.277611 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:26:38.972276 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:26:38.972227 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:26:39.604959 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:26:39.604924 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:26:40.181622 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:26:40.181559 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:26:40.811764 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:26:40.811734 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:26:40.985038 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:26:40.984997 4939 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-4pw7v"] Nov 03 01:26:41.752698 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:26:41.752657 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:26:42.817931 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:26:42.817032 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:26:43.412245 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:26:43.412202 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:26:44.009940 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:26:44.009893 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:26:44.724044 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:26:44.721335 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:26:44.843283 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:26:44.843250 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:26:45.489883 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:26:45.489813 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:26:46.131114 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:26:46.131079 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:26:46.290436 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:26:46.290398 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:26:46.649677 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:26:46.649640 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:26:46.767499 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:26:46.767464 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:26:46.893929 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:26:46.892949 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:26:47.683063 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:26:47.683020 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:26:48.034676 ip-10-0-140-107 kubenswrapper[4939]: E1103 01:26:48.034586 4939 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-sthjh" containerName="extractor" Nov 03 01:26:48.336525 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:26:48.336458 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:26:48.991259 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:26:48.991224 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:26:49.594063 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:26:49.594027 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:26:50.281122 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:26:50.281089 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:26:50.985167 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:26:50.985128 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:26:51.659745 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:26:51.659702 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:26:52.323249 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:26:52.323212 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:26:52.983488 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:26:52.983457 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:26:53.585923 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:26:53.585887 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:26:53.725062 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:26:53.725032 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:26:54.041704 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:26:54.041618 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:26:54.463659 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:26:54.463623 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:26:54.599932 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:26:54.599400 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:26:54.846932 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:26:54.844468 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:26:55.390112 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:26:55.390067 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:26:55.991331 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:26:55.991298 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:26:56.675683 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:26:56.675649 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:26:57.283398 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:26:57.283357 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:26:57.884500 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:26:57.884464 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:26:58.330842 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:26:58.330764 4939 scope.go:122] "RemoveContainer" containerID="1c9d456f802cb91ef9c5559b77cb367b73208196c3e7eac377754b747aa15619" Nov 03 01:26:58.342526 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:26:58.342500 4939 scope.go:122] "RemoveContainer" containerID="5ce76f64ded4525e90504b9c6a2c5ec11b20abfd3a1fb153b538b427d5d32ba7" Nov 03 01:26:58.571765 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:26:58.571733 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:26:59.186396 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:26:59.186034 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:26:59.904181 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:26:59.904145 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:27:00.494040 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:27:00.494001 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:27:01.095577 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:27:01.095542 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:27:01.774047 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:27:01.774000 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:27:02.392806 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:27:02.392749 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:27:03.109111 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:27:03.109070 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:27:03.776953 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:27:03.776883 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:27:04.602411 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:27:04.602372 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:27:05.158347 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:27:05.158311 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:27:05.892134 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:27:05.892093 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:27:06.667979 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:27:06.667945 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:27:07.585213 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:27:07.585162 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:27:07.801944 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:27:07.801900 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:27:07.927843 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:27:07.927743 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:27:08.568682 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:27:08.568590 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:27:08.987105 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:27:08.987066 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:27:09.105459 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:27:09.105410 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:27:10.057541 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:27:10.057482 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:27:10.999376 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:27:10.999340 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:27:11.610279 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:27:11.610247 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:27:12.258509 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:27:12.258478 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:27:13.072033 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:27:13.071995 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:27:13.763469 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:27:13.763434 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:27:14.380506 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:27:14.380466 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:27:15.011176 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:27:15.011139 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:27:15.374320 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:27:15.374263 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:27:15.649094 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:27:15.648999 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:27:16.354502 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:27:16.354462 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:27:17.075740 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:27:17.075700 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:27:17.679561 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:27:17.679527 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:27:18.036785 ip-10-0-140-107 kubenswrapper[4939]: E1103 01:27:18.036691 4939 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-sthjh" containerName="extractor" Nov 03 01:27:18.305504 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:27:18.305412 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:27:18.896958 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:27:18.896904 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:27:19.487379 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:27:19.487329 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:27:20.196418 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:27:20.196386 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:27:20.801148 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:27:20.801112 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:27:21.396380 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:27:21.396344 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:27:22.147517 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:27:22.147478 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:27:22.496769 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:27:22.496673 4939 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-4pw7v"] Nov 03 01:27:23.187382 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:27:23.187349 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:27:24.919565 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:27:24.919522 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:27:25.886510 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:27:25.886476 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:27:26.481263 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:27:26.481222 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:27:27.211212 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:27:27.211152 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:27:27.804953 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:27:27.804887 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:27:28.412235 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:27:28.412202 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:27:29.013899 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:27:29.013860 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:27:29.670429 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:27:29.670395 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:27:31.520620 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:27:31.520578 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:27:32.094598 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:27:32.094551 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:27:32.775161 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:27:32.775121 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:27:33.380038 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:27:33.379999 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:27:34.098244 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:27:34.098181 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:27:34.793722 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:27:34.793683 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:27:35.391009 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:27:35.390972 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:27:36.087042 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:27:36.087005 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:27:36.701097 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:27:36.701060 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:27:37.502246 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:27:37.502205 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:27:38.200234 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:27:38.200197 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:27:38.787989 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:27:38.787958 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:27:39.415161 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:27:39.415125 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:27:40.097341 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:27:40.097310 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:27:40.779001 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:27:40.778967 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:27:41.504050 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:27:41.504018 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:27:42.165826 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:27:42.165789 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:27:42.811659 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:27:42.811621 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:27:43.557262 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:27:43.557225 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:27:44.185552 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:27:44.185512 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:27:44.791435 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:27:44.791396 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:27:44.920164 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:27:44.911925 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:27:45.393892 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:27:45.393859 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:27:46.048534 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:27:46.048489 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:27:46.665091 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:27:46.665060 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:27:47.299154 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:27:47.299118 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:27:47.969736 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:27:47.969698 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:27:48.113146 ip-10-0-140-107 kubenswrapper[4939]: E1103 01:27:48.113102 4939 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-sthjh" containerName="extractor" Nov 03 01:27:48.574077 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:27:48.574044 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:27:49.252710 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:27:49.252674 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:27:49.880669 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:27:49.880629 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:27:50.474525 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:27:50.474492 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:27:51.285364 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:27:51.285326 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:27:52.091430 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:27:52.091397 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:27:52.767186 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:27:52.767150 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:27:53.404568 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:27:53.404535 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:27:53.984725 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:27:53.984689 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:27:54.589479 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:27:54.589450 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:27:55.345368 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:27:55.345323 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:27:55.877397 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:27:55.877368 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:27:56.568140 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:27:56.568105 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:27:57.208025 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:27:57.207990 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:27:57.870711 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:27:57.870673 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:27:58.483613 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:27:58.483575 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:27:59.082176 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:27:59.082133 4939 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-a-d25759-openshift-default-d47fd9785-ggf9t"] Nov 03 01:27:59.082724 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:27:59.082402 4939 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-extmod-a-d25759-openshift-default-d47fd9785-ggf9t" podUID="0cd1d1a4-4446-46a4-9b40-dfc3f83ac8a8" containerName="istio-proxy" containerID="cri-o://6a7f8bce5c0afd0e6312af359b0d245e494f28b66e3c3f222f821145ebececb1" gracePeriod=30 Nov 03 01:27:59.320874 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:27:59.320837 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:27:59.963329 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:27:59.959582 4939 patch_prober.go:29] interesting pod/e2e-extmod-a-d25759-openshift-default-d47fd9785-ggf9t container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 03 01:27:59.963329 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:27:59.959661 4939 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-extmod-a-d25759-openshift-default-d47fd9785-ggf9t" podUID="0cd1d1a4-4446-46a4-9b40-dfc3f83ac8a8" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 03 01:28:00.285638 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:28:00.285551 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:28:00.875733 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:28:00.875698 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:28:01.530025 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:28:01.529963 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:28:01.951943 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:28:01.951849 4939 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-a88862/test-model-a88862-kserve-554fc8cc49-zl7l2"] Nov 03 01:28:02.006195 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:28:02.006160 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:28:02.399220 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:28:02.399164 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:28:03.082177 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:28:03.082068 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:28:03.805473 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:28:03.805438 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:28:04.249869 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:28:04.249827 4939 generic.go:356] "Generic (PLEG): container finished" podID="0cd1d1a4-4446-46a4-9b40-dfc3f83ac8a8" containerID="6a7f8bce5c0afd0e6312af359b0d245e494f28b66e3c3f222f821145ebececb1" exitCode=0 Nov 03 01:28:04.250611 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:28:04.250579 4939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-d25759-openshift-default-d47fd9785-ggf9t" event={"ID":"0cd1d1a4-4446-46a4-9b40-dfc3f83ac8a8","Type":"ContainerDied","Data":"6a7f8bce5c0afd0e6312af359b0d245e494f28b66e3c3f222f821145ebececb1"} Nov 03 01:28:04.402304 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:28:04.402275 4939 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-d25759-openshift-default-d47fd9785-ggf9t" Nov 03 01:28:04.438477 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:28:04.438437 4939 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-a-6fec8a-openshift-default-5cc5f9b957-p9mtp"] Nov 03 01:28:04.439975 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:28:04.439952 4939 cpu_manager.go:414] "containerMap: removing container" podUID="0cd1d1a4-4446-46a4-9b40-dfc3f83ac8a8" containerName="istio-proxy" Nov 03 01:28:04.440039 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:28:04.439978 4939 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0cd1d1a4-4446-46a4-9b40-dfc3f83ac8a8" containerName="istio-proxy" Nov 03 01:28:04.440302 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:28:04.440285 4939 memory_manager.go:365] "RemoveStaleState removing state" podUID="0cd1d1a4-4446-46a4-9b40-dfc3f83ac8a8" containerName="istio-proxy" Nov 03 01:28:04.445184 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:28:04.445161 4939 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-6fec8a-openshift-default-5cc5f9b957-p9mtp" Nov 03 01:28:04.450607 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:28:04.450582 4939 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-a-6fec8a-openshift-default-dockercfg-7vz6w\"" Nov 03 01:28:04.462035 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:28:04.462009 4939 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-6fec8a-openshift-default-5cc5f9b957-p9mtp"] Nov 03 01:28:04.474642 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:28:04.474615 4939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0cd1d1a4-4446-46a4-9b40-dfc3f83ac8a8-istio-token\" (UniqueName: \"kubernetes.io/projected/0cd1d1a4-4446-46a4-9b40-dfc3f83ac8a8-istio-token\") pod \"0cd1d1a4-4446-46a4-9b40-dfc3f83ac8a8\" (UID: \"0cd1d1a4-4446-46a4-9b40-dfc3f83ac8a8\") " Nov 03 01:28:04.474842 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:28:04.474825 4939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0cd1d1a4-4446-46a4-9b40-dfc3f83ac8a8-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0cd1d1a4-4446-46a4-9b40-dfc3f83ac8a8-istio-data\") pod \"0cd1d1a4-4446-46a4-9b40-dfc3f83ac8a8\" (UID: \"0cd1d1a4-4446-46a4-9b40-dfc3f83ac8a8\") " Nov 03 01:28:04.475015 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:28:04.474864 4939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0cd1d1a4-4446-46a4-9b40-dfc3f83ac8a8-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0cd1d1a4-4446-46a4-9b40-dfc3f83ac8a8-workload-socket\") pod \"0cd1d1a4-4446-46a4-9b40-dfc3f83ac8a8\" (UID: \"0cd1d1a4-4446-46a4-9b40-dfc3f83ac8a8\") " Nov 03 01:28:04.475206 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:28:04.475186 4939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0cd1d1a4-4446-46a4-9b40-dfc3f83ac8a8-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0cd1d1a4-4446-46a4-9b40-dfc3f83ac8a8-credential-socket\") pod \"0cd1d1a4-4446-46a4-9b40-dfc3f83ac8a8\" (UID: \"0cd1d1a4-4446-46a4-9b40-dfc3f83ac8a8\") " Nov 03 01:28:04.475326 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:28:04.475312 4939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0cd1d1a4-4446-46a4-9b40-dfc3f83ac8a8-kube-api-access-r4zrb\" (UniqueName: \"kubernetes.io/projected/0cd1d1a4-4446-46a4-9b40-dfc3f83ac8a8-kube-api-access-r4zrb\") pod \"0cd1d1a4-4446-46a4-9b40-dfc3f83ac8a8\" (UID: \"0cd1d1a4-4446-46a4-9b40-dfc3f83ac8a8\") " Nov 03 01:28:04.475390 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:28:04.475341 4939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0cd1d1a4-4446-46a4-9b40-dfc3f83ac8a8-workload-socket" pod "0cd1d1a4-4446-46a4-9b40-dfc3f83ac8a8" (UID: "0cd1d1a4-4446-46a4-9b40-dfc3f83ac8a8"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 01:28:04.475390 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:28:04.475357 4939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0cd1d1a4-4446-46a4-9b40-dfc3f83ac8a8-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0cd1d1a4-4446-46a4-9b40-dfc3f83ac8a8-workload-certs\") pod \"0cd1d1a4-4446-46a4-9b40-dfc3f83ac8a8\" (UID: \"0cd1d1a4-4446-46a4-9b40-dfc3f83ac8a8\") " Nov 03 01:28:04.475507 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:28:04.475409 4939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0cd1d1a4-4446-46a4-9b40-dfc3f83ac8a8-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0cd1d1a4-4446-46a4-9b40-dfc3f83ac8a8-istio-envoy\") pod \"0cd1d1a4-4446-46a4-9b40-dfc3f83ac8a8\" (UID: \"0cd1d1a4-4446-46a4-9b40-dfc3f83ac8a8\") " Nov 03 01:28:04.475507 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:28:04.475465 4939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0cd1d1a4-4446-46a4-9b40-dfc3f83ac8a8-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0cd1d1a4-4446-46a4-9b40-dfc3f83ac8a8-istiod-ca-cert\") pod \"0cd1d1a4-4446-46a4-9b40-dfc3f83ac8a8\" (UID: \"0cd1d1a4-4446-46a4-9b40-dfc3f83ac8a8\") " Nov 03 01:28:04.475612 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:28:04.475511 4939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/0cd1d1a4-4446-46a4-9b40-dfc3f83ac8a8-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0cd1d1a4-4446-46a4-9b40-dfc3f83ac8a8-istio-podinfo\") pod \"0cd1d1a4-4446-46a4-9b40-dfc3f83ac8a8\" (UID: \"0cd1d1a4-4446-46a4-9b40-dfc3f83ac8a8\") " Nov 03 01:28:04.476061 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:28:04.476037 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7064128b-8764-4a6a-a4bc-e42b083f944f-istio-envoy\") pod \"e2e-inf-a-6fec8a-openshift-default-5cc5f9b957-p9mtp\" (UID: \"7064128b-8764-4a6a-a4bc-e42b083f944f\") " pod="openshift-ingress/e2e-inf-a-6fec8a-openshift-default-5cc5f9b957-p9mtp" Nov 03 01:28:04.476400 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:28:04.476377 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f47tc\" (UniqueName: \"kubernetes.io/projected/7064128b-8764-4a6a-a4bc-e42b083f944f-kube-api-access-f47tc\") pod \"e2e-inf-a-6fec8a-openshift-default-5cc5f9b957-p9mtp\" (UID: \"7064128b-8764-4a6a-a4bc-e42b083f944f\") " pod="openshift-ingress/e2e-inf-a-6fec8a-openshift-default-5cc5f9b957-p9mtp" Nov 03 01:28:04.476594 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:28:04.476214 4939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0cd1d1a4-4446-46a4-9b40-dfc3f83ac8a8-workload-certs" pod "0cd1d1a4-4446-46a4-9b40-dfc3f83ac8a8" (UID: "0cd1d1a4-4446-46a4-9b40-dfc3f83ac8a8"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 01:28:04.476594 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:28:04.476529 4939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0cd1d1a4-4446-46a4-9b40-dfc3f83ac8a8-credential-socket" pod "0cd1d1a4-4446-46a4-9b40-dfc3f83ac8a8" (UID: "0cd1d1a4-4446-46a4-9b40-dfc3f83ac8a8"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 01:28:04.476871 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:28:04.476846 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7064128b-8764-4a6a-a4bc-e42b083f944f-istio-podinfo\") pod \"e2e-inf-a-6fec8a-openshift-default-5cc5f9b957-p9mtp\" (UID: \"7064128b-8764-4a6a-a4bc-e42b083f944f\") " pod="openshift-ingress/e2e-inf-a-6fec8a-openshift-default-5cc5f9b957-p9mtp" Nov 03 01:28:04.476977 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:28:04.476943 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7064128b-8764-4a6a-a4bc-e42b083f944f-istiod-ca-cert\") pod \"e2e-inf-a-6fec8a-openshift-default-5cc5f9b957-p9mtp\" (UID: \"7064128b-8764-4a6a-a4bc-e42b083f944f\") " pod="openshift-ingress/e2e-inf-a-6fec8a-openshift-default-5cc5f9b957-p9mtp" Nov 03 01:28:04.477047 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:28:04.476987 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7064128b-8764-4a6a-a4bc-e42b083f944f-credential-socket\") pod \"e2e-inf-a-6fec8a-openshift-default-5cc5f9b957-p9mtp\" (UID: \"7064128b-8764-4a6a-a4bc-e42b083f944f\") " pod="openshift-ingress/e2e-inf-a-6fec8a-openshift-default-5cc5f9b957-p9mtp" Nov 03 01:28:04.477047 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:28:04.477031 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7064128b-8764-4a6a-a4bc-e42b083f944f-workload-certs\") pod \"e2e-inf-a-6fec8a-openshift-default-5cc5f9b957-p9mtp\" (UID: \"7064128b-8764-4a6a-a4bc-e42b083f944f\") " pod="openshift-ingress/e2e-inf-a-6fec8a-openshift-default-5cc5f9b957-p9mtp" Nov 03 01:28:04.477168 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:28:04.477115 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7064128b-8764-4a6a-a4bc-e42b083f944f-istio-token\") pod \"e2e-inf-a-6fec8a-openshift-default-5cc5f9b957-p9mtp\" (UID: \"7064128b-8764-4a6a-a4bc-e42b083f944f\") " pod="openshift-ingress/e2e-inf-a-6fec8a-openshift-default-5cc5f9b957-p9mtp" Nov 03 01:28:04.477168 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:28:04.477148 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7064128b-8764-4a6a-a4bc-e42b083f944f-workload-socket\") pod \"e2e-inf-a-6fec8a-openshift-default-5cc5f9b957-p9mtp\" (UID: \"7064128b-8764-4a6a-a4bc-e42b083f944f\") " pod="openshift-ingress/e2e-inf-a-6fec8a-openshift-default-5cc5f9b957-p9mtp" Nov 03 01:28:04.477280 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:28:04.477223 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7064128b-8764-4a6a-a4bc-e42b083f944f-istio-data\") pod \"e2e-inf-a-6fec8a-openshift-default-5cc5f9b957-p9mtp\" (UID: \"7064128b-8764-4a6a-a4bc-e42b083f944f\") " pod="openshift-ingress/e2e-inf-a-6fec8a-openshift-default-5cc5f9b957-p9mtp" Nov 03 01:28:04.477423 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:28:04.477404 4939 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0cd1d1a4-4446-46a4-9b40-dfc3f83ac8a8-workload-socket\") on node \"ip-10-0-140-107.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:28:04.477551 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:28:04.477537 4939 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0cd1d1a4-4446-46a4-9b40-dfc3f83ac8a8-credential-socket\") on node \"ip-10-0-140-107.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:28:04.477653 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:28:04.477634 4939 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0cd1d1a4-4446-46a4-9b40-dfc3f83ac8a8-workload-certs\") on node \"ip-10-0-140-107.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:28:04.479717 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:28:04.479106 4939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0cd1d1a4-4446-46a4-9b40-dfc3f83ac8a8-istio-token" pod "0cd1d1a4-4446-46a4-9b40-dfc3f83ac8a8" (UID: "0cd1d1a4-4446-46a4-9b40-dfc3f83ac8a8"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 01:28:04.479717 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:28:04.479672 4939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0cd1d1a4-4446-46a4-9b40-dfc3f83ac8a8-istiod-ca-cert" pod "0cd1d1a4-4446-46a4-9b40-dfc3f83ac8a8" (UID: "0cd1d1a4-4446-46a4-9b40-dfc3f83ac8a8"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 01:28:04.480088 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:28:04.480065 4939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0cd1d1a4-4446-46a4-9b40-dfc3f83ac8a8-istio-data" pod "0cd1d1a4-4446-46a4-9b40-dfc3f83ac8a8" (UID: "0cd1d1a4-4446-46a4-9b40-dfc3f83ac8a8"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 01:28:04.482262 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:28:04.482242 4939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0cd1d1a4-4446-46a4-9b40-dfc3f83ac8a8-istio-envoy" pod "0cd1d1a4-4446-46a4-9b40-dfc3f83ac8a8" (UID: "0cd1d1a4-4446-46a4-9b40-dfc3f83ac8a8"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 01:28:04.482373 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:28:04.482349 4939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0cd1d1a4-4446-46a4-9b40-dfc3f83ac8a8-kube-api-access-r4zrb" pod "0cd1d1a4-4446-46a4-9b40-dfc3f83ac8a8" (UID: "0cd1d1a4-4446-46a4-9b40-dfc3f83ac8a8"). InnerVolumeSpecName "kube-api-access-r4zrb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 01:28:04.483520 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:28:04.483499 4939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/0cd1d1a4-4446-46a4-9b40-dfc3f83ac8a8-istio-podinfo" pod "0cd1d1a4-4446-46a4-9b40-dfc3f83ac8a8" (UID: "0cd1d1a4-4446-46a4-9b40-dfc3f83ac8a8"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 01:28:04.579201 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:28:04.579167 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7064128b-8764-4a6a-a4bc-e42b083f944f-istio-envoy\") pod \"e2e-inf-a-6fec8a-openshift-default-5cc5f9b957-p9mtp\" (UID: \"7064128b-8764-4a6a-a4bc-e42b083f944f\") " pod="openshift-ingress/e2e-inf-a-6fec8a-openshift-default-5cc5f9b957-p9mtp" Nov 03 01:28:04.579409 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:28:04.579322 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f47tc\" (UniqueName: \"kubernetes.io/projected/7064128b-8764-4a6a-a4bc-e42b083f944f-kube-api-access-f47tc\") pod \"e2e-inf-a-6fec8a-openshift-default-5cc5f9b957-p9mtp\" (UID: \"7064128b-8764-4a6a-a4bc-e42b083f944f\") " pod="openshift-ingress/e2e-inf-a-6fec8a-openshift-default-5cc5f9b957-p9mtp" Nov 03 01:28:04.579488 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:28:04.579420 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7064128b-8764-4a6a-a4bc-e42b083f944f-istio-podinfo\") pod \"e2e-inf-a-6fec8a-openshift-default-5cc5f9b957-p9mtp\" (UID: \"7064128b-8764-4a6a-a4bc-e42b083f944f\") " pod="openshift-ingress/e2e-inf-a-6fec8a-openshift-default-5cc5f9b957-p9mtp" Nov 03 01:28:04.579488 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:28:04.579455 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7064128b-8764-4a6a-a4bc-e42b083f944f-istiod-ca-cert\") pod \"e2e-inf-a-6fec8a-openshift-default-5cc5f9b957-p9mtp\" (UID: \"7064128b-8764-4a6a-a4bc-e42b083f944f\") " pod="openshift-ingress/e2e-inf-a-6fec8a-openshift-default-5cc5f9b957-p9mtp" Nov 03 01:28:04.579488 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:28:04.579477 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7064128b-8764-4a6a-a4bc-e42b083f944f-credential-socket\") pod \"e2e-inf-a-6fec8a-openshift-default-5cc5f9b957-p9mtp\" (UID: \"7064128b-8764-4a6a-a4bc-e42b083f944f\") " pod="openshift-ingress/e2e-inf-a-6fec8a-openshift-default-5cc5f9b957-p9mtp" Nov 03 01:28:04.579684 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:28:04.579510 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7064128b-8764-4a6a-a4bc-e42b083f944f-workload-certs\") pod \"e2e-inf-a-6fec8a-openshift-default-5cc5f9b957-p9mtp\" (UID: \"7064128b-8764-4a6a-a4bc-e42b083f944f\") " pod="openshift-ingress/e2e-inf-a-6fec8a-openshift-default-5cc5f9b957-p9mtp" Nov 03 01:28:04.579684 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:28:04.579556 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7064128b-8764-4a6a-a4bc-e42b083f944f-istio-token\") pod \"e2e-inf-a-6fec8a-openshift-default-5cc5f9b957-p9mtp\" (UID: \"7064128b-8764-4a6a-a4bc-e42b083f944f\") " pod="openshift-ingress/e2e-inf-a-6fec8a-openshift-default-5cc5f9b957-p9mtp" Nov 03 01:28:04.579684 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:28:04.579580 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7064128b-8764-4a6a-a4bc-e42b083f944f-workload-socket\") pod \"e2e-inf-a-6fec8a-openshift-default-5cc5f9b957-p9mtp\" (UID: \"7064128b-8764-4a6a-a4bc-e42b083f944f\") " pod="openshift-ingress/e2e-inf-a-6fec8a-openshift-default-5cc5f9b957-p9mtp" Nov 03 01:28:04.579684 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:28:04.579641 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7064128b-8764-4a6a-a4bc-e42b083f944f-istio-data\") pod \"e2e-inf-a-6fec8a-openshift-default-5cc5f9b957-p9mtp\" (UID: \"7064128b-8764-4a6a-a4bc-e42b083f944f\") " pod="openshift-ingress/e2e-inf-a-6fec8a-openshift-default-5cc5f9b957-p9mtp" Nov 03 01:28:04.579902 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:28:04.579874 4939 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0cd1d1a4-4446-46a4-9b40-dfc3f83ac8a8-istio-token\") on node \"ip-10-0-140-107.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:28:04.579902 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:28:04.579894 4939 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0cd1d1a4-4446-46a4-9b40-dfc3f83ac8a8-istio-data\") on node \"ip-10-0-140-107.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:28:04.580044 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:28:04.579927 4939 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-r4zrb\" (UniqueName: \"kubernetes.io/projected/0cd1d1a4-4446-46a4-9b40-dfc3f83ac8a8-kube-api-access-r4zrb\") on node \"ip-10-0-140-107.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:28:04.580044 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:28:04.579942 4939 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0cd1d1a4-4446-46a4-9b40-dfc3f83ac8a8-istio-envoy\") on node \"ip-10-0-140-107.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:28:04.580044 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:28:04.579953 4939 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0cd1d1a4-4446-46a4-9b40-dfc3f83ac8a8-istiod-ca-cert\") on node \"ip-10-0-140-107.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:28:04.580044 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:28:04.579961 4939 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0cd1d1a4-4446-46a4-9b40-dfc3f83ac8a8-istio-podinfo\") on node \"ip-10-0-140-107.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:28:04.580044 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:28:04.580010 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7064128b-8764-4a6a-a4bc-e42b083f944f-credential-socket\") pod \"e2e-inf-a-6fec8a-openshift-default-5cc5f9b957-p9mtp\" (UID: \"7064128b-8764-4a6a-a4bc-e42b083f944f\") " pod="openshift-ingress/e2e-inf-a-6fec8a-openshift-default-5cc5f9b957-p9mtp" Nov 03 01:28:04.580341 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:28:04.580077 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7064128b-8764-4a6a-a4bc-e42b083f944f-workload-certs\") pod \"e2e-inf-a-6fec8a-openshift-default-5cc5f9b957-p9mtp\" (UID: \"7064128b-8764-4a6a-a4bc-e42b083f944f\") " pod="openshift-ingress/e2e-inf-a-6fec8a-openshift-default-5cc5f9b957-p9mtp" Nov 03 01:28:04.580341 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:28:04.580126 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7064128b-8764-4a6a-a4bc-e42b083f944f-workload-socket\") pod \"e2e-inf-a-6fec8a-openshift-default-5cc5f9b957-p9mtp\" (UID: \"7064128b-8764-4a6a-a4bc-e42b083f944f\") " pod="openshift-ingress/e2e-inf-a-6fec8a-openshift-default-5cc5f9b957-p9mtp" Nov 03 01:28:04.580341 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:28:04.580211 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7064128b-8764-4a6a-a4bc-e42b083f944f-istio-data\") pod \"e2e-inf-a-6fec8a-openshift-default-5cc5f9b957-p9mtp\" (UID: \"7064128b-8764-4a6a-a4bc-e42b083f944f\") " pod="openshift-ingress/e2e-inf-a-6fec8a-openshift-default-5cc5f9b957-p9mtp" Nov 03 01:28:04.580531 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:28:04.580394 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7064128b-8764-4a6a-a4bc-e42b083f944f-istiod-ca-cert\") pod \"e2e-inf-a-6fec8a-openshift-default-5cc5f9b957-p9mtp\" (UID: \"7064128b-8764-4a6a-a4bc-e42b083f944f\") " pod="openshift-ingress/e2e-inf-a-6fec8a-openshift-default-5cc5f9b957-p9mtp" Nov 03 01:28:04.581735 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:28:04.581709 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7064128b-8764-4a6a-a4bc-e42b083f944f-istio-envoy\") pod \"e2e-inf-a-6fec8a-openshift-default-5cc5f9b957-p9mtp\" (UID: \"7064128b-8764-4a6a-a4bc-e42b083f944f\") " pod="openshift-ingress/e2e-inf-a-6fec8a-openshift-default-5cc5f9b957-p9mtp" Nov 03 01:28:04.582131 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:28:04.582109 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7064128b-8764-4a6a-a4bc-e42b083f944f-istio-podinfo\") pod \"e2e-inf-a-6fec8a-openshift-default-5cc5f9b957-p9mtp\" (UID: \"7064128b-8764-4a6a-a4bc-e42b083f944f\") " pod="openshift-ingress/e2e-inf-a-6fec8a-openshift-default-5cc5f9b957-p9mtp" Nov 03 01:28:04.582246 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:28:04.582226 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:28:04.587264 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:28:04.587240 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7064128b-8764-4a6a-a4bc-e42b083f944f-istio-token\") pod \"e2e-inf-a-6fec8a-openshift-default-5cc5f9b957-p9mtp\" (UID: \"7064128b-8764-4a6a-a4bc-e42b083f944f\") " pod="openshift-ingress/e2e-inf-a-6fec8a-openshift-default-5cc5f9b957-p9mtp" Nov 03 01:28:04.587484 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:28:04.587463 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-f47tc\" (UniqueName: \"kubernetes.io/projected/7064128b-8764-4a6a-a4bc-e42b083f944f-kube-api-access-f47tc\") pod \"e2e-inf-a-6fec8a-openshift-default-5cc5f9b957-p9mtp\" (UID: \"7064128b-8764-4a6a-a4bc-e42b083f944f\") " pod="openshift-ingress/e2e-inf-a-6fec8a-openshift-default-5cc5f9b957-p9mtp" Nov 03 01:28:04.770700 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:28:04.770615 4939 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-6fec8a-openshift-default-5cc5f9b957-p9mtp" Nov 03 01:28:05.068829 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:28:05.065563 4939 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-6fec8a-openshift-default-5cc5f9b957-p9mtp"] Nov 03 01:28:05.071873 ip-10-0-140-107 kubenswrapper[4939]: W1103 01:28:05.071843 4939 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7064128b_8764_4a6a_a4bc_e42b083f944f.slice/crio-38e3f4d1923ab0ea63036772ea798e0c253eaa986e52c13ffb3e7c9afaea0347 WatchSource:0}: Error finding container 38e3f4d1923ab0ea63036772ea798e0c253eaa986e52c13ffb3e7c9afaea0347: Status 404 returned error can't find the container with id 38e3f4d1923ab0ea63036772ea798e0c253eaa986e52c13ffb3e7c9afaea0347 Nov 03 01:28:05.076347 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:28:05.076301 4939 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 01:28:05.076460 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:28:05.076397 4939 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 01:28:05.076460 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:28:05.076442 4939 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 01:28:05.262983 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:28:05.262900 4939 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-d25759-openshift-default-d47fd9785-ggf9t" Nov 03 01:28:05.262983 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:28:05.262954 4939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-d25759-openshift-default-d47fd9785-ggf9t" event={"ID":"0cd1d1a4-4446-46a4-9b40-dfc3f83ac8a8","Type":"ContainerDied","Data":"75c5ac2f311d4656d8c389229d7fc64a6f8b01c4761f14d1bf3634e944aaa2e7"} Nov 03 01:28:05.263527 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:28:05.263028 4939 scope.go:122] "RemoveContainer" containerID="6a7f8bce5c0afd0e6312af359b0d245e494f28b66e3c3f222f821145ebececb1" Nov 03 01:28:05.265574 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:28:05.265546 4939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-6fec8a-openshift-default-5cc5f9b957-p9mtp" event={"ID":"7064128b-8764-4a6a-a4bc-e42b083f944f","Type":"ContainerStarted","Data":"8717adbb605eabe2ae8a56d66659a3a6a308f1d9ae5fa997071b413addebbc6c"} Nov 03 01:28:05.265723 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:28:05.265596 4939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-6fec8a-openshift-default-5cc5f9b957-p9mtp" event={"ID":"7064128b-8764-4a6a-a4bc-e42b083f944f","Type":"ContainerStarted","Data":"38e3f4d1923ab0ea63036772ea798e0c253eaa986e52c13ffb3e7c9afaea0347"} Nov 03 01:28:05.290775 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:28:05.290729 4939 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-a-6fec8a-openshift-default-5cc5f9b957-p9mtp" podStartSLOduration=71.290711492 podStartE2EDuration="1m11.290711492s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 01:26:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 01:28:05.288315356 +0000 UTC m=+2833.552650313" watchObservedRunningTime="2026-11-03 01:28:05.290711492 +0000 UTC m=+2833.555046436" Nov 03 01:28:05.299098 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:28:05.299065 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:28:05.311377 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:28:05.311351 4939 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-a-d25759-openshift-default-d47fd9785-ggf9t"] Nov 03 01:28:05.319528 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:28:05.319473 4939 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-extmod-a-d25759-openshift-default-d47fd9785-ggf9t"] Nov 03 01:28:05.770784 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:28:05.770681 4939 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-6fec8a-openshift-default-5cc5f9b957-p9mtp" Nov 03 01:28:05.873495 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:28:05.873456 4939 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0cd1d1a4-4446-46a4-9b40-dfc3f83ac8a8" path="/var/lib/kubelet/pods/0cd1d1a4-4446-46a4-9b40-dfc3f83ac8a8/volumes" Nov 03 01:28:05.996642 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:28:05.996608 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:28:06.777440 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:28:06.777409 4939 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-6fec8a-openshift-default-5cc5f9b957-p9mtp" Nov 03 01:28:06.992643 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:28:06.992613 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:28:07.287880 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:28:07.287847 4939 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-6fec8a-openshift-default-5cc5f9b957-p9mtp" Nov 03 01:28:07.288856 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:28:07.288836 4939 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-6fec8a-openshift-default-5cc5f9b957-p9mtp" Nov 03 01:28:07.575282 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:28:07.575185 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:28:07.743808 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:28:07.743764 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:28:07.871929 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:28:07.871572 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:28:08.582275 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:28:08.582243 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:28:08.790613 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:28:08.790569 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:28:08.969987 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:28:08.950835 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:28:09.608404 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:28:09.608368 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:28:10.301651 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:28:10.301620 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:28:10.908973 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:28:10.908939 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:28:11.587934 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:28:11.576543 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:28:12.187835 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:28:12.187797 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:28:12.868255 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:28:12.868224 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:28:13.109647 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:28:13.109604 4939 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-ce04af/test-model-ce04af-kserve-84bf7f6d7c-qd56c"] Nov 03 01:28:13.149356 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:28:13.149277 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:28:13.679256 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:28:13.679216 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:28:13.794006 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:28:13.793899 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:28:14.399712 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:28:14.399674 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:28:15.167500 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:28:15.167463 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:28:15.887445 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:28:15.887404 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:28:16.586900 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:28:16.586865 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:28:16.995981 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:28:16.995849 4939 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-inf-a-a88862/test-model-a88862-kserve-554fc8cc49-zl7l2" podUID="7c49ba9e-e354-4e34-be2e-2269aac5e015" containerName="main" containerID="cri-o://db36987c67ecb311528b67948d63d4b3d62512a1c2a6e98cb1326e16ba240b78" gracePeriod=45 Nov 03 01:28:17.258721 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:28:17.258677 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:28:17.339982 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:28:17.328827 4939 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-a88862/test-model-a88862-kserve-554fc8cc49-zl7l2" Nov 03 01:28:17.427403 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:28:17.427361 4939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7c49ba9e-e354-4e34-be2e-2269aac5e015-tls-certs\" (UniqueName: \"kubernetes.io/secret/7c49ba9e-e354-4e34-be2e-2269aac5e015-tls-certs\") pod \"7c49ba9e-e354-4e34-be2e-2269aac5e015\" (UID: \"7c49ba9e-e354-4e34-be2e-2269aac5e015\") " Nov 03 01:28:17.427616 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:28:17.427468 4939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7c49ba9e-e354-4e34-be2e-2269aac5e015-home\" (UniqueName: \"kubernetes.io/empty-dir/7c49ba9e-e354-4e34-be2e-2269aac5e015-home\") pod \"7c49ba9e-e354-4e34-be2e-2269aac5e015\" (UID: \"7c49ba9e-e354-4e34-be2e-2269aac5e015\") " Nov 03 01:28:17.427736 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:28:17.427712 4939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7c49ba9e-e354-4e34-be2e-2269aac5e015-kube-api-access-55h2j\" (UniqueName: \"kubernetes.io/projected/7c49ba9e-e354-4e34-be2e-2269aac5e015-kube-api-access-55h2j\") pod \"7c49ba9e-e354-4e34-be2e-2269aac5e015\" (UID: \"7c49ba9e-e354-4e34-be2e-2269aac5e015\") " Nov 03 01:28:17.427807 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:28:17.427763 4939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7c49ba9e-e354-4e34-be2e-2269aac5e015-home" pod "7c49ba9e-e354-4e34-be2e-2269aac5e015" (UID: "7c49ba9e-e354-4e34-be2e-2269aac5e015"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 01:28:17.427880 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:28:17.427864 4939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7c49ba9e-e354-4e34-be2e-2269aac5e015-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7c49ba9e-e354-4e34-be2e-2269aac5e015-tmp-dir\") pod \"7c49ba9e-e354-4e34-be2e-2269aac5e015\" (UID: \"7c49ba9e-e354-4e34-be2e-2269aac5e015\") " Nov 03 01:28:17.428063 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:28:17.428042 4939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7c49ba9e-e354-4e34-be2e-2269aac5e015-dshm\" (UniqueName: \"kubernetes.io/empty-dir/7c49ba9e-e354-4e34-be2e-2269aac5e015-dshm\") pod \"7c49ba9e-e354-4e34-be2e-2269aac5e015\" (UID: \"7c49ba9e-e354-4e34-be2e-2269aac5e015\") " Nov 03 01:28:17.428192 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:28:17.428167 4939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7c49ba9e-e354-4e34-be2e-2269aac5e015-tmp-dir" pod "7c49ba9e-e354-4e34-be2e-2269aac5e015" (UID: "7c49ba9e-e354-4e34-be2e-2269aac5e015"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 01:28:17.428422 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:28:17.428399 4939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7c49ba9e-e354-4e34-be2e-2269aac5e015-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7c49ba9e-e354-4e34-be2e-2269aac5e015-model-cache\") pod \"7c49ba9e-e354-4e34-be2e-2269aac5e015\" (UID: \"7c49ba9e-e354-4e34-be2e-2269aac5e015\") " Nov 03 01:28:17.428765 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:28:17.428739 4939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7c49ba9e-e354-4e34-be2e-2269aac5e015-model-cache" pod "7c49ba9e-e354-4e34-be2e-2269aac5e015" (UID: "7c49ba9e-e354-4e34-be2e-2269aac5e015"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 01:28:17.429360 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:28:17.429334 4939 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7c49ba9e-e354-4e34-be2e-2269aac5e015-model-cache\") on node \"ip-10-0-140-107.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:28:17.429464 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:28:17.429363 4939 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7c49ba9e-e354-4e34-be2e-2269aac5e015-home\") on node \"ip-10-0-140-107.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:28:17.429464 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:28:17.429378 4939 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7c49ba9e-e354-4e34-be2e-2269aac5e015-tmp-dir\") on node \"ip-10-0-140-107.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:28:17.430155 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:28:17.430126 4939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7c49ba9e-e354-4e34-be2e-2269aac5e015-tls-certs" pod "7c49ba9e-e354-4e34-be2e-2269aac5e015" (UID: "7c49ba9e-e354-4e34-be2e-2269aac5e015"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 01:28:17.430369 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:28:17.430339 4939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7c49ba9e-e354-4e34-be2e-2269aac5e015-kube-api-access-55h2j" pod "7c49ba9e-e354-4e34-be2e-2269aac5e015" (UID: "7c49ba9e-e354-4e34-be2e-2269aac5e015"). InnerVolumeSpecName "kube-api-access-55h2j". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 01:28:17.430456 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:28:17.430406 4939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7c49ba9e-e354-4e34-be2e-2269aac5e015-dshm" pod "7c49ba9e-e354-4e34-be2e-2269aac5e015" (UID: "7c49ba9e-e354-4e34-be2e-2269aac5e015"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 01:28:17.510038 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:28:17.509949 4939 generic.go:356] "Generic (PLEG): container finished" podID="7c49ba9e-e354-4e34-be2e-2269aac5e015" containerID="db36987c67ecb311528b67948d63d4b3d62512a1c2a6e98cb1326e16ba240b78" exitCode=0 Nov 03 01:28:17.510038 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:28:17.509993 4939 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-a88862/test-model-a88862-kserve-554fc8cc49-zl7l2" Nov 03 01:28:17.510260 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:28:17.510040 4939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-a88862/test-model-a88862-kserve-554fc8cc49-zl7l2" event={"ID":"7c49ba9e-e354-4e34-be2e-2269aac5e015","Type":"ContainerDied","Data":"db36987c67ecb311528b67948d63d4b3d62512a1c2a6e98cb1326e16ba240b78"} Nov 03 01:28:17.510260 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:28:17.510081 4939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-a88862/test-model-a88862-kserve-554fc8cc49-zl7l2" event={"ID":"7c49ba9e-e354-4e34-be2e-2269aac5e015","Type":"ContainerDied","Data":"99abea7443f59a25158ede0b1eeb6003c70909c110caa8be8bf85d1a5e3bf763"} Nov 03 01:28:17.510260 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:28:17.510103 4939 scope.go:122] "RemoveContainer" containerID="db36987c67ecb311528b67948d63d4b3d62512a1c2a6e98cb1326e16ba240b78" Nov 03 01:28:17.525937 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:28:17.525916 4939 scope.go:122] "RemoveContainer" containerID="db36987c67ecb311528b67948d63d4b3d62512a1c2a6e98cb1326e16ba240b78" Nov 03 01:28:17.526222 ip-10-0-140-107 kubenswrapper[4939]: E1103 01:28:17.526202 4939 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"db36987c67ecb311528b67948d63d4b3d62512a1c2a6e98cb1326e16ba240b78\": container with ID starting with db36987c67ecb311528b67948d63d4b3d62512a1c2a6e98cb1326e16ba240b78 not found: ID does not exist" containerID="db36987c67ecb311528b67948d63d4b3d62512a1c2a6e98cb1326e16ba240b78" Nov 03 01:28:17.526287 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:28:17.526232 4939 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"db36987c67ecb311528b67948d63d4b3d62512a1c2a6e98cb1326e16ba240b78"} err="failed to get container status \"db36987c67ecb311528b67948d63d4b3d62512a1c2a6e98cb1326e16ba240b78\": rpc error: code = NotFound desc = could not find container \"db36987c67ecb311528b67948d63d4b3d62512a1c2a6e98cb1326e16ba240b78\": container with ID starting with db36987c67ecb311528b67948d63d4b3d62512a1c2a6e98cb1326e16ba240b78 not found: ID does not exist" Nov 03 01:28:17.531629 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:28:17.531606 4939 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-55h2j\" (UniqueName: \"kubernetes.io/projected/7c49ba9e-e354-4e34-be2e-2269aac5e015-kube-api-access-55h2j\") on node \"ip-10-0-140-107.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:28:17.531727 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:28:17.531633 4939 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7c49ba9e-e354-4e34-be2e-2269aac5e015-dshm\") on node \"ip-10-0-140-107.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:28:17.531727 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:28:17.531642 4939 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7c49ba9e-e354-4e34-be2e-2269aac5e015-tls-certs\") on node \"ip-10-0-140-107.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:28:17.540254 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:28:17.540228 4939 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-a88862/test-model-a88862-kserve-554fc8cc49-zl7l2"] Nov 03 01:28:17.545156 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:28:17.545135 4939 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-a-a88862/test-model-a88862-kserve-554fc8cc49-zl7l2"] Nov 03 01:28:17.727013 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:28:17.726979 4939 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-r2cgw"] Nov 03 01:28:17.727233 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:28:17.727195 4939 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-r2cgw" podUID="20419baf-ec7d-47c2-ad77-5f32d6d93992" containerName="istio-proxy" containerID="cri-o://f2912e656092ab81c03075e5fd025cb575c1c444b6d8f2ea7339e32028af741e" gracePeriod=30 Nov 03 01:28:17.873060 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:28:17.873026 4939 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7c49ba9e-e354-4e34-be2e-2269aac5e015" path="/var/lib/kubelet/pods/7c49ba9e-e354-4e34-be2e-2269aac5e015/volumes" Nov 03 01:28:17.994276 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:28:17.994242 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:28:18.102574 ip-10-0-140-107 kubenswrapper[4939]: E1103 01:28:18.102493 4939 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-sthjh" containerName="extractor" Nov 03 01:28:18.392828 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:28:18.392793 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:28:18.500035 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:28:18.499986 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:28:18.706485 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:28:18.706086 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:28:18.734106 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:28:18.734075 4939 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-a88862-6bccb6967f-46486"] Nov 03 01:28:18.837538 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:28:18.837503 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:28:18.937635 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:28:18.937593 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:28:19.603104 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:28:19.603066 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:28:19.971982 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:28:19.967305 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:28:20.691020 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:28:20.690989 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:28:21.399183 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:28:21.399147 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:28:22.022399 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:28:22.022360 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:28:22.124750 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:28:22.124702 4939 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-4pw7v"] Nov 03 01:28:22.674717 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:28:22.674680 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:28:22.921931 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:28:22.920320 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:28:23.499171 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:28:23.499144 4939 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-r2cgw" Nov 03 01:28:23.534718 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:28:23.534686 4939 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-a-ee618c-openshift-default-5ff9c74dc9-ssz95"] Nov 03 01:28:23.535715 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:28:23.535696 4939 cpu_manager.go:414] "containerMap: removing container" podUID="20419baf-ec7d-47c2-ad77-5f32d6d93992" containerName="istio-proxy" Nov 03 01:28:23.535715 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:28:23.535715 4939 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="20419baf-ec7d-47c2-ad77-5f32d6d93992" containerName="istio-proxy" Nov 03 01:28:23.535824 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:28:23.535731 4939 cpu_manager.go:414] "containerMap: removing container" podUID="7c49ba9e-e354-4e34-be2e-2269aac5e015" containerName="main" Nov 03 01:28:23.535824 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:28:23.535737 4939 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7c49ba9e-e354-4e34-be2e-2269aac5e015" containerName="main" Nov 03 01:28:23.535901 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:28:23.535894 4939 memory_manager.go:365] "RemoveStaleState removing state" podUID="20419baf-ec7d-47c2-ad77-5f32d6d93992" containerName="istio-proxy" Nov 03 01:28:23.535974 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:28:23.535916 4939 memory_manager.go:365] "RemoveStaleState removing state" podUID="7c49ba9e-e354-4e34-be2e-2269aac5e015" containerName="main" Nov 03 01:28:23.540205 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:28:23.540179 4939 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-ee618c-openshift-default-5ff9c74dc9-ssz95" Nov 03 01:28:23.545471 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:28:23.545433 4939 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-a-ee618c-openshift-default-dockercfg-d6ps5\"" Nov 03 01:28:23.556572 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:28:23.556524 4939 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-ee618c-openshift-default-5ff9c74dc9-ssz95"] Nov 03 01:28:23.587327 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:28:23.587288 4939 generic.go:356] "Generic (PLEG): container finished" podID="20419baf-ec7d-47c2-ad77-5f32d6d93992" containerID="f2912e656092ab81c03075e5fd025cb575c1c444b6d8f2ea7339e32028af741e" exitCode=0 Nov 03 01:28:23.587560 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:28:23.587533 4939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-r2cgw" event={"ID":"20419baf-ec7d-47c2-ad77-5f32d6d93992","Type":"ContainerDied","Data":"f2912e656092ab81c03075e5fd025cb575c1c444b6d8f2ea7339e32028af741e"} Nov 03 01:28:23.587647 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:28:23.587589 4939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-r2cgw" event={"ID":"20419baf-ec7d-47c2-ad77-5f32d6d93992","Type":"ContainerDied","Data":"80d92fd9a26879d7e74a7f1a6eae0622d9254f3372066ad5a006655484f95879"} Nov 03 01:28:23.587647 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:28:23.587621 4939 scope.go:122] "RemoveContainer" containerID="f2912e656092ab81c03075e5fd025cb575c1c444b6d8f2ea7339e32028af741e" Nov 03 01:28:23.587851 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:28:23.587833 4939 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-r2cgw" Nov 03 01:28:23.607002 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:28:23.606976 4939 scope.go:122] "RemoveContainer" containerID="f2912e656092ab81c03075e5fd025cb575c1c444b6d8f2ea7339e32028af741e" Nov 03 01:28:23.607443 ip-10-0-140-107 kubenswrapper[4939]: E1103 01:28:23.607419 4939 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f2912e656092ab81c03075e5fd025cb575c1c444b6d8f2ea7339e32028af741e\": container with ID starting with f2912e656092ab81c03075e5fd025cb575c1c444b6d8f2ea7339e32028af741e not found: ID does not exist" containerID="f2912e656092ab81c03075e5fd025cb575c1c444b6d8f2ea7339e32028af741e" Nov 03 01:28:23.607532 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:28:23.607454 4939 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f2912e656092ab81c03075e5fd025cb575c1c444b6d8f2ea7339e32028af741e"} err="failed to get container status \"f2912e656092ab81c03075e5fd025cb575c1c444b6d8f2ea7339e32028af741e\": rpc error: code = NotFound desc = could not find container \"f2912e656092ab81c03075e5fd025cb575c1c444b6d8f2ea7339e32028af741e\": container with ID starting with f2912e656092ab81c03075e5fd025cb575c1c444b6d8f2ea7339e32028af741e not found: ID does not exist" Nov 03 01:28:23.707673 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:28:23.707219 4939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/20419baf-ec7d-47c2-ad77-5f32d6d93992-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/20419baf-ec7d-47c2-ad77-5f32d6d93992-workload-socket\") pod \"20419baf-ec7d-47c2-ad77-5f32d6d93992\" (UID: \"20419baf-ec7d-47c2-ad77-5f32d6d93992\") " Nov 03 01:28:23.709533 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:28:23.707976 4939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/20419baf-ec7d-47c2-ad77-5f32d6d93992-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/20419baf-ec7d-47c2-ad77-5f32d6d93992-istio-envoy\") pod \"20419baf-ec7d-47c2-ad77-5f32d6d93992\" (UID: \"20419baf-ec7d-47c2-ad77-5f32d6d93992\") " Nov 03 01:28:23.709533 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:28:23.708092 4939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/20419baf-ec7d-47c2-ad77-5f32d6d93992-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/20419baf-ec7d-47c2-ad77-5f32d6d93992-credential-socket\") pod \"20419baf-ec7d-47c2-ad77-5f32d6d93992\" (UID: \"20419baf-ec7d-47c2-ad77-5f32d6d93992\") " Nov 03 01:28:23.709533 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:28:23.708224 4939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/20419baf-ec7d-47c2-ad77-5f32d6d93992-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/20419baf-ec7d-47c2-ad77-5f32d6d93992-workload-certs\") pod \"20419baf-ec7d-47c2-ad77-5f32d6d93992\" (UID: \"20419baf-ec7d-47c2-ad77-5f32d6d93992\") " Nov 03 01:28:23.709533 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:28:23.708365 4939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/20419baf-ec7d-47c2-ad77-5f32d6d93992-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/20419baf-ec7d-47c2-ad77-5f32d6d93992-istio-data\") pod \"20419baf-ec7d-47c2-ad77-5f32d6d93992\" (UID: \"20419baf-ec7d-47c2-ad77-5f32d6d93992\") " Nov 03 01:28:23.709533 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:28:23.708406 4939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/20419baf-ec7d-47c2-ad77-5f32d6d93992-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/20419baf-ec7d-47c2-ad77-5f32d6d93992-istiod-ca-cert\") pod \"20419baf-ec7d-47c2-ad77-5f32d6d93992\" (UID: \"20419baf-ec7d-47c2-ad77-5f32d6d93992\") " Nov 03 01:28:23.709533 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:28:23.708466 4939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/20419baf-ec7d-47c2-ad77-5f32d6d93992-kube-api-access-g6sk2\" (UniqueName: \"kubernetes.io/projected/20419baf-ec7d-47c2-ad77-5f32d6d93992-kube-api-access-g6sk2\") pod \"20419baf-ec7d-47c2-ad77-5f32d6d93992\" (UID: \"20419baf-ec7d-47c2-ad77-5f32d6d93992\") " Nov 03 01:28:23.709533 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:28:23.708550 4939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/20419baf-ec7d-47c2-ad77-5f32d6d93992-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/20419baf-ec7d-47c2-ad77-5f32d6d93992-istio-podinfo\") pod \"20419baf-ec7d-47c2-ad77-5f32d6d93992\" (UID: \"20419baf-ec7d-47c2-ad77-5f32d6d93992\") " Nov 03 01:28:23.709533 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:28:23.708833 4939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/20419baf-ec7d-47c2-ad77-5f32d6d93992-istio-token\" (UniqueName: \"kubernetes.io/projected/20419baf-ec7d-47c2-ad77-5f32d6d93992-istio-token\") pod \"20419baf-ec7d-47c2-ad77-5f32d6d93992\" (UID: \"20419baf-ec7d-47c2-ad77-5f32d6d93992\") " Nov 03 01:28:23.710770 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:28:23.710093 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fef89d35-3c5f-483f-b896-7b47733ab2df-istio-podinfo\") pod \"e2e-inf-a-ee618c-openshift-default-5ff9c74dc9-ssz95\" (UID: \"fef89d35-3c5f-483f-b896-7b47733ab2df\") " pod="openshift-ingress/e2e-inf-a-ee618c-openshift-default-5ff9c74dc9-ssz95" Nov 03 01:28:23.710770 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:28:23.710289 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fef89d35-3c5f-483f-b896-7b47733ab2df-istio-token\") pod \"e2e-inf-a-ee618c-openshift-default-5ff9c74dc9-ssz95\" (UID: \"fef89d35-3c5f-483f-b896-7b47733ab2df\") " pod="openshift-ingress/e2e-inf-a-ee618c-openshift-default-5ff9c74dc9-ssz95" Nov 03 01:28:23.712046 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:28:23.711040 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fef89d35-3c5f-483f-b896-7b47733ab2df-istio-data\") pod \"e2e-inf-a-ee618c-openshift-default-5ff9c74dc9-ssz95\" (UID: \"fef89d35-3c5f-483f-b896-7b47733ab2df\") " pod="openshift-ingress/e2e-inf-a-ee618c-openshift-default-5ff9c74dc9-ssz95" Nov 03 01:28:23.712046 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:28:23.711197 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fef89d35-3c5f-483f-b896-7b47733ab2df-credential-socket\") pod \"e2e-inf-a-ee618c-openshift-default-5ff9c74dc9-ssz95\" (UID: \"fef89d35-3c5f-483f-b896-7b47733ab2df\") " pod="openshift-ingress/e2e-inf-a-ee618c-openshift-default-5ff9c74dc9-ssz95" Nov 03 01:28:23.712046 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:28:23.711289 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fef89d35-3c5f-483f-b896-7b47733ab2df-workload-certs\") pod \"e2e-inf-a-ee618c-openshift-default-5ff9c74dc9-ssz95\" (UID: \"fef89d35-3c5f-483f-b896-7b47733ab2df\") " pod="openshift-ingress/e2e-inf-a-ee618c-openshift-default-5ff9c74dc9-ssz95" Nov 03 01:28:23.712046 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:28:23.711343 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fef89d35-3c5f-483f-b896-7b47733ab2df-workload-socket\") pod \"e2e-inf-a-ee618c-openshift-default-5ff9c74dc9-ssz95\" (UID: \"fef89d35-3c5f-483f-b896-7b47733ab2df\") " pod="openshift-ingress/e2e-inf-a-ee618c-openshift-default-5ff9c74dc9-ssz95" Nov 03 01:28:23.712046 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:28:23.711527 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xd2xk\" (UniqueName: \"kubernetes.io/projected/fef89d35-3c5f-483f-b896-7b47733ab2df-kube-api-access-xd2xk\") pod \"e2e-inf-a-ee618c-openshift-default-5ff9c74dc9-ssz95\" (UID: \"fef89d35-3c5f-483f-b896-7b47733ab2df\") " pod="openshift-ingress/e2e-inf-a-ee618c-openshift-default-5ff9c74dc9-ssz95" Nov 03 01:28:23.712046 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:28:23.711568 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fef89d35-3c5f-483f-b896-7b47733ab2df-istiod-ca-cert\") pod \"e2e-inf-a-ee618c-openshift-default-5ff9c74dc9-ssz95\" (UID: \"fef89d35-3c5f-483f-b896-7b47733ab2df\") " pod="openshift-ingress/e2e-inf-a-ee618c-openshift-default-5ff9c74dc9-ssz95" Nov 03 01:28:23.712046 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:28:23.711612 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fef89d35-3c5f-483f-b896-7b47733ab2df-istio-envoy\") pod \"e2e-inf-a-ee618c-openshift-default-5ff9c74dc9-ssz95\" (UID: \"fef89d35-3c5f-483f-b896-7b47733ab2df\") " pod="openshift-ingress/e2e-inf-a-ee618c-openshift-default-5ff9c74dc9-ssz95" Nov 03 01:28:23.713112 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:28:23.712742 4939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/20419baf-ec7d-47c2-ad77-5f32d6d93992-istiod-ca-cert" pod "20419baf-ec7d-47c2-ad77-5f32d6d93992" (UID: "20419baf-ec7d-47c2-ad77-5f32d6d93992"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 01:28:23.738938 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:28:23.734834 4939 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-inf-a-a88862-6bccb6967f-46486" podUID="dd50d5d4-bbf2-439b-a851-2c2ae58da605" containerName="payload-processing" containerID="cri-o://d022249c6b7f0ad709448febf97a28b05d4540b81fe89ae5ce58e2542e963980" gracePeriod=25 Nov 03 01:28:23.739300 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:28:23.739273 4939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/20419baf-ec7d-47c2-ad77-5f32d6d93992-credential-socket" pod "20419baf-ec7d-47c2-ad77-5f32d6d93992" (UID: "20419baf-ec7d-47c2-ad77-5f32d6d93992"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 01:28:23.750285 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:28:23.749175 4939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/20419baf-ec7d-47c2-ad77-5f32d6d93992-kube-api-access-g6sk2" pod "20419baf-ec7d-47c2-ad77-5f32d6d93992" (UID: "20419baf-ec7d-47c2-ad77-5f32d6d93992"). InnerVolumeSpecName "kube-api-access-g6sk2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 01:28:23.750285 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:28:23.749543 4939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/20419baf-ec7d-47c2-ad77-5f32d6d93992-istio-data" pod "20419baf-ec7d-47c2-ad77-5f32d6d93992" (UID: "20419baf-ec7d-47c2-ad77-5f32d6d93992"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 01:28:23.776014 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:28:23.755982 4939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/20419baf-ec7d-47c2-ad77-5f32d6d93992-workload-socket" pod "20419baf-ec7d-47c2-ad77-5f32d6d93992" (UID: "20419baf-ec7d-47c2-ad77-5f32d6d93992"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 01:28:23.776014 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:28:23.756063 4939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/20419baf-ec7d-47c2-ad77-5f32d6d93992-workload-certs" pod "20419baf-ec7d-47c2-ad77-5f32d6d93992" (UID: "20419baf-ec7d-47c2-ad77-5f32d6d93992"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 01:28:23.847952 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:28:23.830388 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fef89d35-3c5f-483f-b896-7b47733ab2df-credential-socket\") pod \"e2e-inf-a-ee618c-openshift-default-5ff9c74dc9-ssz95\" (UID: \"fef89d35-3c5f-483f-b896-7b47733ab2df\") " pod="openshift-ingress/e2e-inf-a-ee618c-openshift-default-5ff9c74dc9-ssz95" Nov 03 01:28:23.847952 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:28:23.830494 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fef89d35-3c5f-483f-b896-7b47733ab2df-workload-certs\") pod \"e2e-inf-a-ee618c-openshift-default-5ff9c74dc9-ssz95\" (UID: \"fef89d35-3c5f-483f-b896-7b47733ab2df\") " pod="openshift-ingress/e2e-inf-a-ee618c-openshift-default-5ff9c74dc9-ssz95" Nov 03 01:28:23.847952 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:28:23.830546 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fef89d35-3c5f-483f-b896-7b47733ab2df-workload-socket\") pod \"e2e-inf-a-ee618c-openshift-default-5ff9c74dc9-ssz95\" (UID: \"fef89d35-3c5f-483f-b896-7b47733ab2df\") " pod="openshift-ingress/e2e-inf-a-ee618c-openshift-default-5ff9c74dc9-ssz95" Nov 03 01:28:23.847952 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:28:23.830705 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xd2xk\" (UniqueName: \"kubernetes.io/projected/fef89d35-3c5f-483f-b896-7b47733ab2df-kube-api-access-xd2xk\") pod \"e2e-inf-a-ee618c-openshift-default-5ff9c74dc9-ssz95\" (UID: \"fef89d35-3c5f-483f-b896-7b47733ab2df\") " pod="openshift-ingress/e2e-inf-a-ee618c-openshift-default-5ff9c74dc9-ssz95" Nov 03 01:28:23.847952 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:28:23.830742 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fef89d35-3c5f-483f-b896-7b47733ab2df-istiod-ca-cert\") pod \"e2e-inf-a-ee618c-openshift-default-5ff9c74dc9-ssz95\" (UID: \"fef89d35-3c5f-483f-b896-7b47733ab2df\") " pod="openshift-ingress/e2e-inf-a-ee618c-openshift-default-5ff9c74dc9-ssz95" Nov 03 01:28:23.847952 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:28:23.830784 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fef89d35-3c5f-483f-b896-7b47733ab2df-istio-envoy\") pod \"e2e-inf-a-ee618c-openshift-default-5ff9c74dc9-ssz95\" (UID: \"fef89d35-3c5f-483f-b896-7b47733ab2df\") " pod="openshift-ingress/e2e-inf-a-ee618c-openshift-default-5ff9c74dc9-ssz95" Nov 03 01:28:23.847952 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:28:23.830948 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fef89d35-3c5f-483f-b896-7b47733ab2df-istio-podinfo\") pod \"e2e-inf-a-ee618c-openshift-default-5ff9c74dc9-ssz95\" (UID: \"fef89d35-3c5f-483f-b896-7b47733ab2df\") " pod="openshift-ingress/e2e-inf-a-ee618c-openshift-default-5ff9c74dc9-ssz95" Nov 03 01:28:23.847952 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:28:23.831104 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fef89d35-3c5f-483f-b896-7b47733ab2df-istio-token\") pod \"e2e-inf-a-ee618c-openshift-default-5ff9c74dc9-ssz95\" (UID: \"fef89d35-3c5f-483f-b896-7b47733ab2df\") " pod="openshift-ingress/e2e-inf-a-ee618c-openshift-default-5ff9c74dc9-ssz95" Nov 03 01:28:23.847952 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:28:23.831179 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fef89d35-3c5f-483f-b896-7b47733ab2df-istio-data\") pod \"e2e-inf-a-ee618c-openshift-default-5ff9c74dc9-ssz95\" (UID: \"fef89d35-3c5f-483f-b896-7b47733ab2df\") " pod="openshift-ingress/e2e-inf-a-ee618c-openshift-default-5ff9c74dc9-ssz95" Nov 03 01:28:23.847952 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:28:23.831418 4939 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/20419baf-ec7d-47c2-ad77-5f32d6d93992-istio-data\") on node \"ip-10-0-140-107.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:28:23.847952 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:28:23.831434 4939 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/20419baf-ec7d-47c2-ad77-5f32d6d93992-istiod-ca-cert\") on node \"ip-10-0-140-107.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:28:23.847952 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:28:23.831452 4939 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-g6sk2\" (UniqueName: \"kubernetes.io/projected/20419baf-ec7d-47c2-ad77-5f32d6d93992-kube-api-access-g6sk2\") on node \"ip-10-0-140-107.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:28:23.847952 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:28:23.831470 4939 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/20419baf-ec7d-47c2-ad77-5f32d6d93992-workload-socket\") on node \"ip-10-0-140-107.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:28:23.847952 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:28:23.831485 4939 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/20419baf-ec7d-47c2-ad77-5f32d6d93992-credential-socket\") on node \"ip-10-0-140-107.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:28:23.847952 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:28:23.831500 4939 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/20419baf-ec7d-47c2-ad77-5f32d6d93992-workload-certs\") on node \"ip-10-0-140-107.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:28:23.878981 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:28:23.850928 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fef89d35-3c5f-483f-b896-7b47733ab2df-credential-socket\") pod \"e2e-inf-a-ee618c-openshift-default-5ff9c74dc9-ssz95\" (UID: \"fef89d35-3c5f-483f-b896-7b47733ab2df\") " pod="openshift-ingress/e2e-inf-a-ee618c-openshift-default-5ff9c74dc9-ssz95" Nov 03 01:28:23.878981 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:28:23.851215 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fef89d35-3c5f-483f-b896-7b47733ab2df-workload-certs\") pod \"e2e-inf-a-ee618c-openshift-default-5ff9c74dc9-ssz95\" (UID: \"fef89d35-3c5f-483f-b896-7b47733ab2df\") " pod="openshift-ingress/e2e-inf-a-ee618c-openshift-default-5ff9c74dc9-ssz95" Nov 03 01:28:23.878981 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:28:23.851467 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fef89d35-3c5f-483f-b896-7b47733ab2df-workload-socket\") pod \"e2e-inf-a-ee618c-openshift-default-5ff9c74dc9-ssz95\" (UID: \"fef89d35-3c5f-483f-b896-7b47733ab2df\") " pod="openshift-ingress/e2e-inf-a-ee618c-openshift-default-5ff9c74dc9-ssz95" Nov 03 01:28:23.878981 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:28:23.855046 4939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/20419baf-ec7d-47c2-ad77-5f32d6d93992-istio-envoy" pod "20419baf-ec7d-47c2-ad77-5f32d6d93992" (UID: "20419baf-ec7d-47c2-ad77-5f32d6d93992"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 01:28:23.878981 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:28:23.855870 4939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/20419baf-ec7d-47c2-ad77-5f32d6d93992-istio-podinfo" pod "20419baf-ec7d-47c2-ad77-5f32d6d93992" (UID: "20419baf-ec7d-47c2-ad77-5f32d6d93992"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 01:28:23.878981 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:28:23.856052 4939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/20419baf-ec7d-47c2-ad77-5f32d6d93992-istio-token" pod "20419baf-ec7d-47c2-ad77-5f32d6d93992" (UID: "20419baf-ec7d-47c2-ad77-5f32d6d93992"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 01:28:23.878981 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:28:23.856586 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fef89d35-3c5f-483f-b896-7b47733ab2df-istiod-ca-cert\") pod \"e2e-inf-a-ee618c-openshift-default-5ff9c74dc9-ssz95\" (UID: \"fef89d35-3c5f-483f-b896-7b47733ab2df\") " pod="openshift-ingress/e2e-inf-a-ee618c-openshift-default-5ff9c74dc9-ssz95" Nov 03 01:28:23.878981 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:28:23.857163 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fef89d35-3c5f-483f-b896-7b47733ab2df-istio-data\") pod \"e2e-inf-a-ee618c-openshift-default-5ff9c74dc9-ssz95\" (UID: \"fef89d35-3c5f-483f-b896-7b47733ab2df\") " pod="openshift-ingress/e2e-inf-a-ee618c-openshift-default-5ff9c74dc9-ssz95" Nov 03 01:28:23.917154 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:28:23.905436 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fef89d35-3c5f-483f-b896-7b47733ab2df-istio-envoy\") pod \"e2e-inf-a-ee618c-openshift-default-5ff9c74dc9-ssz95\" (UID: \"fef89d35-3c5f-483f-b896-7b47733ab2df\") " pod="openshift-ingress/e2e-inf-a-ee618c-openshift-default-5ff9c74dc9-ssz95" Nov 03 01:28:23.944723 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:28:23.944445 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fef89d35-3c5f-483f-b896-7b47733ab2df-istio-token\") pod \"e2e-inf-a-ee618c-openshift-default-5ff9c74dc9-ssz95\" (UID: \"fef89d35-3c5f-483f-b896-7b47733ab2df\") " pod="openshift-ingress/e2e-inf-a-ee618c-openshift-default-5ff9c74dc9-ssz95" Nov 03 01:28:23.944723 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:28:23.944567 4939 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/20419baf-ec7d-47c2-ad77-5f32d6d93992-istio-token\") on node \"ip-10-0-140-107.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:28:23.944723 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:28:23.944592 4939 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/20419baf-ec7d-47c2-ad77-5f32d6d93992-istio-envoy\") on node \"ip-10-0-140-107.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:28:23.944723 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:28:23.944607 4939 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/20419baf-ec7d-47c2-ad77-5f32d6d93992-istio-podinfo\") on node \"ip-10-0-140-107.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:28:23.989805 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:28:23.985542 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fef89d35-3c5f-483f-b896-7b47733ab2df-istio-podinfo\") pod \"e2e-inf-a-ee618c-openshift-default-5ff9c74dc9-ssz95\" (UID: \"fef89d35-3c5f-483f-b896-7b47733ab2df\") " pod="openshift-ingress/e2e-inf-a-ee618c-openshift-default-5ff9c74dc9-ssz95" Nov 03 01:28:24.029620 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:28:24.029524 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xd2xk\" (UniqueName: \"kubernetes.io/projected/fef89d35-3c5f-483f-b896-7b47733ab2df-kube-api-access-xd2xk\") pod \"e2e-inf-a-ee618c-openshift-default-5ff9c74dc9-ssz95\" (UID: \"fef89d35-3c5f-483f-b896-7b47733ab2df\") " pod="openshift-ingress/e2e-inf-a-ee618c-openshift-default-5ff9c74dc9-ssz95" Nov 03 01:28:24.044081 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:28:24.044043 4939 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-r2cgw"] Nov 03 01:28:24.054180 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:28:24.054148 4939 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-r2cgw"] Nov 03 01:28:24.178665 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:28:24.178567 4939 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-ee618c-openshift-default-5ff9c74dc9-ssz95" Nov 03 01:28:24.211329 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:28:24.211301 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:28:24.486410 ip-10-0-140-107 kubenswrapper[4939]: W1103 01:28:24.486374 4939 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfef89d35_3c5f_483f_b896_7b47733ab2df.slice/crio-7a3b86f98204dba32db6a4f066856a10eba1a3d2beb0bd3df40119620943af08 WatchSource:0}: Error finding container 7a3b86f98204dba32db6a4f066856a10eba1a3d2beb0bd3df40119620943af08: Status 404 returned error can't find the container with id 7a3b86f98204dba32db6a4f066856a10eba1a3d2beb0bd3df40119620943af08 Nov 03 01:28:24.487412 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:28:24.487386 4939 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-ee618c-openshift-default-5ff9c74dc9-ssz95"] Nov 03 01:28:24.489810 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:28:24.489773 4939 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 01:28:24.489880 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:28:24.489847 4939 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 01:28:24.489940 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:28:24.489878 4939 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 01:28:24.602870 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:28:24.602833 4939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-ee618c-openshift-default-5ff9c74dc9-ssz95" event={"ID":"fef89d35-3c5f-483f-b896-7b47733ab2df","Type":"ContainerStarted","Data":"7a3b86f98204dba32db6a4f066856a10eba1a3d2beb0bd3df40119620943af08"} Nov 03 01:28:24.626372 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:28:24.626286 4939 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-a-ee618c-openshift-default-5ff9c74dc9-ssz95" podStartSLOduration=80.626250322 podStartE2EDuration="1m20.626250322s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 01:27:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 01:28:24.626063755 +0000 UTC m=+2852.890398713" watchObservedRunningTime="2026-11-03 01:28:24.626250322 +0000 UTC m=+2852.890585271" Nov 03 01:28:24.963247 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:28:24.963211 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:28:25.179226 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:28:25.179027 4939 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-ee618c-openshift-default-5ff9c74dc9-ssz95" Nov 03 01:28:25.181104 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:28:25.180686 4939 patch_prober.go:29] interesting pod/e2e-inf-a-ee618c-openshift-default-5ff9c74dc9-ssz95 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.51:15021/healthz/ready\": dial tcp 10.132.1.51:15021: connect: connection refused" start-of-body= Nov 03 01:28:25.181104 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:28:25.180746 4939 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-ee618c-openshift-default-5ff9c74dc9-ssz95" podUID="fef89d35-3c5f-483f-b896-7b47733ab2df" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.51:15021/healthz/ready\": dial tcp 10.132.1.51:15021: connect: connection refused" Nov 03 01:28:25.527346 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:28:25.527315 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:28:25.613999 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:28:25.613959 4939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-ee618c-openshift-default-5ff9c74dc9-ssz95" event={"ID":"fef89d35-3c5f-483f-b896-7b47733ab2df","Type":"ContainerStarted","Data":"8eed9477dcb60efc399a950decb550840ff9a5096a9ecfe2dca04f714f38dab6"} Nov 03 01:28:25.662410 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:28:25.657693 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:28:25.888583 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:28:25.888543 4939 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="20419baf-ec7d-47c2-ad77-5f32d6d93992" path="/var/lib/kubelet/pods/20419baf-ec7d-47c2-ad77-5f32d6d93992/volumes" Nov 03 01:28:26.143533 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:28:26.143447 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:28:26.190937 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:28:26.186366 4939 patch_prober.go:29] interesting pod/e2e-inf-a-ee618c-openshift-default-5ff9c74dc9-ssz95 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.51:15021/healthz/ready\": dial tcp 10.132.1.51:15021: connect: connection refused" start-of-body= Nov 03 01:28:26.190937 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:28:26.186431 4939 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-ee618c-openshift-default-5ff9c74dc9-ssz95" podUID="fef89d35-3c5f-483f-b896-7b47733ab2df" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.51:15021/healthz/ready\": dial tcp 10.132.1.51:15021: connect: connection refused" Nov 03 01:28:26.433417 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:28:26.431800 4939 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-a88862-openshift-default-7b98d598f-jbgzc"] Nov 03 01:28:26.433417 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:28:26.432074 4939 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-a-a88862-openshift-default-7b98d598f-jbgzc" podUID="0de902db-4a72-49b2-8f60-8b63663cc98d" containerName="istio-proxy" containerID="cri-o://b27f1483f0bc66d6493c8775c31ddae558650ba330be12a2401c08afbcffa1da" gracePeriod=30 Nov 03 01:28:26.498656 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:28:26.498615 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:28:27.101690 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:28:27.101651 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:28:27.190932 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:28:27.186359 4939 patch_prober.go:29] interesting pod/e2e-inf-a-ee618c-openshift-default-5ff9c74dc9-ssz95 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.51:15021/healthz/ready\": dial tcp 10.132.1.51:15021: connect: connection refused" start-of-body= Nov 03 01:28:27.190932 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:28:27.186429 4939 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-ee618c-openshift-default-5ff9c74dc9-ssz95" podUID="fef89d35-3c5f-483f-b896-7b47733ab2df" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.51:15021/healthz/ready\": dial tcp 10.132.1.51:15021: connect: connection refused" Nov 03 01:28:27.278810 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:28:27.278771 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:28:27.540657 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:28:27.540558 4939 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-inf-a-a88862-6bccb6967f-46486" podUID="dd50d5d4-bbf2-439b-a851-2c2ae58da605" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 03 01:28:27.800249 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:28:27.800161 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:28:28.134179 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:28:28.134145 4939 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-inf-b-ce04af/test-model-ce04af-kserve-84bf7f6d7c-qd56c" podUID="72db0238-20cc-40cb-affb-158746adb87b" containerName="main" containerID="cri-o://8c5b07d70b0371fa9339b585cf59fbbf46d5c4332a79766c422536bbdcfd9dc1" gracePeriod=45 Nov 03 01:28:28.179537 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:28:28.179502 4939 patch_prober.go:29] interesting pod/e2e-inf-a-ee618c-openshift-default-5ff9c74dc9-ssz95 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.51:15021/healthz/ready\": dial tcp 10.132.1.51:15021: connect: connection refused" start-of-body= Nov 03 01:28:28.179666 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:28:28.179571 4939 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-ee618c-openshift-default-5ff9c74dc9-ssz95" podUID="fef89d35-3c5f-483f-b896-7b47733ab2df" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.51:15021/healthz/ready\": dial tcp 10.132.1.51:15021: connect: connection refused" Nov 03 01:28:28.488266 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:28:28.488159 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:28:28.630760 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:28:28.630727 4939 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-ce04af/test-model-ce04af-kserve-84bf7f6d7c-qd56c" Nov 03 01:28:28.708893 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:28:28.708852 4939 generic.go:356] "Generic (PLEG): container finished" podID="72db0238-20cc-40cb-affb-158746adb87b" containerID="8c5b07d70b0371fa9339b585cf59fbbf46d5c4332a79766c422536bbdcfd9dc1" exitCode=0 Nov 03 01:28:28.709133 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:28:28.708972 4939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-ce04af/test-model-ce04af-kserve-84bf7f6d7c-qd56c" event={"ID":"72db0238-20cc-40cb-affb-158746adb87b","Type":"ContainerDied","Data":"8c5b07d70b0371fa9339b585cf59fbbf46d5c4332a79766c422536bbdcfd9dc1"} Nov 03 01:28:28.709133 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:28:28.709037 4939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-ce04af/test-model-ce04af-kserve-84bf7f6d7c-qd56c" event={"ID":"72db0238-20cc-40cb-affb-158746adb87b","Type":"ContainerDied","Data":"0101c2b2d2abf22b2524f341f9ce383306cf633723782184e5c02e0af90f4f29"} Nov 03 01:28:28.709133 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:28:28.709052 4939 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-ce04af/test-model-ce04af-kserve-84bf7f6d7c-qd56c" Nov 03 01:28:28.709133 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:28:28.709064 4939 scope.go:122] "RemoveContainer" containerID="8c5b07d70b0371fa9339b585cf59fbbf46d5c4332a79766c422536bbdcfd9dc1" Nov 03 01:28:28.723810 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:28:28.723656 4939 scope.go:122] "RemoveContainer" containerID="8c5b07d70b0371fa9339b585cf59fbbf46d5c4332a79766c422536bbdcfd9dc1" Nov 03 01:28:28.724013 ip-10-0-140-107 kubenswrapper[4939]: E1103 01:28:28.723985 4939 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8c5b07d70b0371fa9339b585cf59fbbf46d5c4332a79766c422536bbdcfd9dc1\": container with ID starting with 8c5b07d70b0371fa9339b585cf59fbbf46d5c4332a79766c422536bbdcfd9dc1 not found: ID does not exist" containerID="8c5b07d70b0371fa9339b585cf59fbbf46d5c4332a79766c422536bbdcfd9dc1" Nov 03 01:28:28.724136 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:28:28.724023 4939 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8c5b07d70b0371fa9339b585cf59fbbf46d5c4332a79766c422536bbdcfd9dc1"} err="failed to get container status \"8c5b07d70b0371fa9339b585cf59fbbf46d5c4332a79766c422536bbdcfd9dc1\": rpc error: code = NotFound desc = could not find container \"8c5b07d70b0371fa9339b585cf59fbbf46d5c4332a79766c422536bbdcfd9dc1\": container with ID starting with 8c5b07d70b0371fa9339b585cf59fbbf46d5c4332a79766c422536bbdcfd9dc1 not found: ID does not exist" Nov 03 01:28:28.779062 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:28:28.769498 4939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/72db0238-20cc-40cb-affb-158746adb87b-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/72db0238-20cc-40cb-affb-158746adb87b-tmp-dir\") pod \"72db0238-20cc-40cb-affb-158746adb87b\" (UID: \"72db0238-20cc-40cb-affb-158746adb87b\") " Nov 03 01:28:28.779062 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:28:28.769699 4939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/72db0238-20cc-40cb-affb-158746adb87b-home\" (UniqueName: \"kubernetes.io/empty-dir/72db0238-20cc-40cb-affb-158746adb87b-home\") pod \"72db0238-20cc-40cb-affb-158746adb87b\" (UID: \"72db0238-20cc-40cb-affb-158746adb87b\") " Nov 03 01:28:28.779062 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:28:28.769793 4939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/72db0238-20cc-40cb-affb-158746adb87b-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/72db0238-20cc-40cb-affb-158746adb87b-model-cache\") pod \"72db0238-20cc-40cb-affb-158746adb87b\" (UID: \"72db0238-20cc-40cb-affb-158746adb87b\") " Nov 03 01:28:28.779062 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:28:28.770483 4939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/72db0238-20cc-40cb-affb-158746adb87b-tls-certs\" (UniqueName: \"kubernetes.io/secret/72db0238-20cc-40cb-affb-158746adb87b-tls-certs\") pod \"72db0238-20cc-40cb-affb-158746adb87b\" (UID: \"72db0238-20cc-40cb-affb-158746adb87b\") " Nov 03 01:28:28.779062 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:28:28.770652 4939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/72db0238-20cc-40cb-affb-158746adb87b-kube-api-access-dh5j7\" (UniqueName: \"kubernetes.io/projected/72db0238-20cc-40cb-affb-158746adb87b-kube-api-access-dh5j7\") pod \"72db0238-20cc-40cb-affb-158746adb87b\" (UID: \"72db0238-20cc-40cb-affb-158746adb87b\") " Nov 03 01:28:28.779062 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:28:28.770734 4939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/72db0238-20cc-40cb-affb-158746adb87b-dshm\" (UniqueName: \"kubernetes.io/empty-dir/72db0238-20cc-40cb-affb-158746adb87b-dshm\") pod \"72db0238-20cc-40cb-affb-158746adb87b\" (UID: \"72db0238-20cc-40cb-affb-158746adb87b\") " Nov 03 01:28:28.779835 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:28:28.779799 4939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/72db0238-20cc-40cb-affb-158746adb87b-tmp-dir" pod "72db0238-20cc-40cb-affb-158746adb87b" (UID: "72db0238-20cc-40cb-affb-158746adb87b"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 01:28:28.780574 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:28:28.780547 4939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/72db0238-20cc-40cb-affb-158746adb87b-home" pod "72db0238-20cc-40cb-affb-158746adb87b" (UID: "72db0238-20cc-40cb-affb-158746adb87b"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 01:28:28.781116 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:28:28.781083 4939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/72db0238-20cc-40cb-affb-158746adb87b-model-cache" pod "72db0238-20cc-40cb-affb-158746adb87b" (UID: "72db0238-20cc-40cb-affb-158746adb87b"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 01:28:28.781861 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:28:28.781836 4939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/72db0238-20cc-40cb-affb-158746adb87b-tls-certs" pod "72db0238-20cc-40cb-affb-158746adb87b" (UID: "72db0238-20cc-40cb-affb-158746adb87b"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 01:28:28.782797 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:28:28.782768 4939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/72db0238-20cc-40cb-affb-158746adb87b-kube-api-access-dh5j7" pod "72db0238-20cc-40cb-affb-158746adb87b" (UID: "72db0238-20cc-40cb-affb-158746adb87b"). InnerVolumeSpecName "kube-api-access-dh5j7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 01:28:28.783843 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:28:28.783820 4939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/72db0238-20cc-40cb-affb-158746adb87b-dshm" pod "72db0238-20cc-40cb-affb-158746adb87b" (UID: "72db0238-20cc-40cb-affb-158746adb87b"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 01:28:28.873778 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:28:28.873728 4939 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/72db0238-20cc-40cb-affb-158746adb87b-home\") on node \"ip-10-0-140-107.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:28:28.873778 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:28:28.873770 4939 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/72db0238-20cc-40cb-affb-158746adb87b-model-cache\") on node \"ip-10-0-140-107.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:28:28.873778 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:28:28.873786 4939 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/72db0238-20cc-40cb-affb-158746adb87b-tls-certs\") on node \"ip-10-0-140-107.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:28:28.874155 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:28:28.873803 4939 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dh5j7\" (UniqueName: \"kubernetes.io/projected/72db0238-20cc-40cb-affb-158746adb87b-kube-api-access-dh5j7\") on node \"ip-10-0-140-107.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:28:28.874155 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:28:28.873818 4939 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/72db0238-20cc-40cb-affb-158746adb87b-dshm\") on node \"ip-10-0-140-107.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:28:28.874155 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:28:28.873961 4939 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/72db0238-20cc-40cb-affb-158746adb87b-tmp-dir\") on node \"ip-10-0-140-107.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:28:29.036396 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:28:29.036308 4939 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-ce04af/test-model-ce04af-kserve-84bf7f6d7c-qd56c"] Nov 03 01:28:29.041208 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:28:29.041181 4939 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-b-ce04af/test-model-ce04af-kserve-84bf7f6d7c-qd56c"] Nov 03 01:28:29.179810 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:28:29.179771 4939 patch_prober.go:29] interesting pod/e2e-inf-a-ee618c-openshift-default-5ff9c74dc9-ssz95 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.51:15021/healthz/ready\": dial tcp 10.132.1.51:15021: connect: connection refused" start-of-body= Nov 03 01:28:29.180275 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:28:29.179826 4939 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-ee618c-openshift-default-5ff9c74dc9-ssz95" podUID="fef89d35-3c5f-483f-b896-7b47733ab2df" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.51:15021/healthz/ready\": dial tcp 10.132.1.51:15021: connect: connection refused" Nov 03 01:28:29.282962 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:28:29.282930 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:28:29.884740 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:28:29.884702 4939 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="72db0238-20cc-40cb-affb-158746adb87b" path="/var/lib/kubelet/pods/72db0238-20cc-40cb-affb-158746adb87b/volumes" Nov 03 01:28:29.914333 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:28:29.914283 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:28:30.180097 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:28:30.180003 4939 patch_prober.go:29] interesting pod/e2e-inf-a-ee618c-openshift-default-5ff9c74dc9-ssz95 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.51:15021/healthz/ready\": dial tcp 10.132.1.51:15021: connect: connection refused" start-of-body= Nov 03 01:28:30.180097 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:28:30.180070 4939 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-ee618c-openshift-default-5ff9c74dc9-ssz95" podUID="fef89d35-3c5f-483f-b896-7b47733ab2df" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.51:15021/healthz/ready\": dial tcp 10.132.1.51:15021: connect: connection refused" Nov 03 01:28:30.632471 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:28:30.632439 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:28:31.219489 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:28:31.219455 4939 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-ee618c-openshift-default-5ff9c74dc9-ssz95" Nov 03 01:28:31.220705 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:28:31.220673 4939 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-ee618c-openshift-default-5ff9c74dc9-ssz95" Nov 03 01:28:31.222658 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:28:31.222633 4939 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-ee618c-openshift-default-5ff9c74dc9-ssz95" Nov 03 01:28:31.477067 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:28:31.477009 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:28:31.597619 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:28:31.597579 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:28:31.705207 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:28:31.705078 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:28:31.751313 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:28:31.751279 4939 generic.go:356] "Generic (PLEG): container finished" podID="0de902db-4a72-49b2-8f60-8b63663cc98d" containerID="b27f1483f0bc66d6493c8775c31ddae558650ba330be12a2401c08afbcffa1da" exitCode=0 Nov 03 01:28:31.751502 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:28:31.751331 4939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-a88862-openshift-default-7b98d598f-jbgzc" event={"ID":"0de902db-4a72-49b2-8f60-8b63663cc98d","Type":"ContainerDied","Data":"b27f1483f0bc66d6493c8775c31ddae558650ba330be12a2401c08afbcffa1da"} Nov 03 01:28:31.751502 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:28:31.751400 4939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-a88862-openshift-default-7b98d598f-jbgzc" event={"ID":"0de902db-4a72-49b2-8f60-8b63663cc98d","Type":"ContainerDied","Data":"87f49ec8ca1466f9c260605200e31f0cd67864278b2dcc6f1256df7f234ae819"} Nov 03 01:28:31.751502 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:28:31.751460 4939 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="87f49ec8ca1466f9c260605200e31f0cd67864278b2dcc6f1256df7f234ae819" Nov 03 01:28:31.751502 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:28:31.751480 4939 scope.go:122] "RemoveContainer" containerID="1c190ea68c6e2b48ce74b7d1e60eb45265c6485d155a6e2319418f0a058075b2" Nov 03 01:28:31.785267 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:28:31.785245 4939 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-a88862-openshift-default-7b98d598f-jbgzc" Nov 03 01:28:31.817679 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:28:31.817652 4939 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-6fec8a-6cdc9b4ff5-xjq85"] Nov 03 01:28:31.818866 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:28:31.818841 4939 cpu_manager.go:414] "containerMap: removing container" podUID="0de902db-4a72-49b2-8f60-8b63663cc98d" containerName="istio-proxy" Nov 03 01:28:31.818975 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:28:31.818866 4939 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0de902db-4a72-49b2-8f60-8b63663cc98d" containerName="istio-proxy" Nov 03 01:28:31.818975 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:28:31.818943 4939 cpu_manager.go:414] "containerMap: removing container" podUID="72db0238-20cc-40cb-affb-158746adb87b" containerName="main" Nov 03 01:28:31.818975 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:28:31.818950 4939 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="72db0238-20cc-40cb-affb-158746adb87b" containerName="main" Nov 03 01:28:31.819169 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:28:31.819157 4939 memory_manager.go:365] "RemoveStaleState removing state" podUID="0de902db-4a72-49b2-8f60-8b63663cc98d" containerName="istio-proxy" Nov 03 01:28:31.819210 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:28:31.819171 4939 memory_manager.go:365] "RemoveStaleState removing state" podUID="0de902db-4a72-49b2-8f60-8b63663cc98d" containerName="istio-proxy" Nov 03 01:28:31.819210 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:28:31.819184 4939 memory_manager.go:365] "RemoveStaleState removing state" podUID="72db0238-20cc-40cb-affb-158746adb87b" containerName="main" Nov 03 01:28:31.823507 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:28:31.823486 4939 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-6fec8a-6cdc9b4ff5-xjq85" Nov 03 01:28:31.829409 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:28:31.829381 4939 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-a-6fec8a-serving-cert\"" Nov 03 01:28:31.832678 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:28:31.832653 4939 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-6fec8a-6cdc9b4ff5-xjq85"] Nov 03 01:28:31.866652 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:28:31.866626 4939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0de902db-4a72-49b2-8f60-8b63663cc98d-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0de902db-4a72-49b2-8f60-8b63663cc98d-workload-socket\") pod \"0de902db-4a72-49b2-8f60-8b63663cc98d\" (UID: \"0de902db-4a72-49b2-8f60-8b63663cc98d\") " Nov 03 01:28:31.866823 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:28:31.866755 4939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0de902db-4a72-49b2-8f60-8b63663cc98d-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0de902db-4a72-49b2-8f60-8b63663cc98d-istio-envoy\") pod \"0de902db-4a72-49b2-8f60-8b63663cc98d\" (UID: \"0de902db-4a72-49b2-8f60-8b63663cc98d\") " Nov 03 01:28:31.867011 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:28:31.866980 4939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0de902db-4a72-49b2-8f60-8b63663cc98d-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0de902db-4a72-49b2-8f60-8b63663cc98d-istiod-ca-cert\") pod \"0de902db-4a72-49b2-8f60-8b63663cc98d\" (UID: \"0de902db-4a72-49b2-8f60-8b63663cc98d\") " Nov 03 01:28:31.867118 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:28:31.867056 4939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/0de902db-4a72-49b2-8f60-8b63663cc98d-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0de902db-4a72-49b2-8f60-8b63663cc98d-istio-podinfo\") pod \"0de902db-4a72-49b2-8f60-8b63663cc98d\" (UID: \"0de902db-4a72-49b2-8f60-8b63663cc98d\") " Nov 03 01:28:31.867193 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:28:31.867178 4939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0de902db-4a72-49b2-8f60-8b63663cc98d-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0de902db-4a72-49b2-8f60-8b63663cc98d-istio-data\") pod \"0de902db-4a72-49b2-8f60-8b63663cc98d\" (UID: \"0de902db-4a72-49b2-8f60-8b63663cc98d\") " Nov 03 01:28:31.867522 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:28:31.867497 4939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0de902db-4a72-49b2-8f60-8b63663cc98d-istio-data" pod "0de902db-4a72-49b2-8f60-8b63663cc98d" (UID: "0de902db-4a72-49b2-8f60-8b63663cc98d"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 01:28:31.867660 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:28:31.867636 4939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0de902db-4a72-49b2-8f60-8b63663cc98d-istio-token\" (UniqueName: \"kubernetes.io/projected/0de902db-4a72-49b2-8f60-8b63663cc98d-istio-token\") pod \"0de902db-4a72-49b2-8f60-8b63663cc98d\" (UID: \"0de902db-4a72-49b2-8f60-8b63663cc98d\") " Nov 03 01:28:31.867742 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:28:31.867726 4939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0de902db-4a72-49b2-8f60-8b63663cc98d-kube-api-access-4g5mt\" (UniqueName: \"kubernetes.io/projected/0de902db-4a72-49b2-8f60-8b63663cc98d-kube-api-access-4g5mt\") pod \"0de902db-4a72-49b2-8f60-8b63663cc98d\" (UID: \"0de902db-4a72-49b2-8f60-8b63663cc98d\") " Nov 03 01:28:31.867820 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:28:31.867799 4939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0de902db-4a72-49b2-8f60-8b63663cc98d-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0de902db-4a72-49b2-8f60-8b63663cc98d-workload-certs\") pod \"0de902db-4a72-49b2-8f60-8b63663cc98d\" (UID: \"0de902db-4a72-49b2-8f60-8b63663cc98d\") " Nov 03 01:28:31.867882 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:28:31.867822 4939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0de902db-4a72-49b2-8f60-8b63663cc98d-istiod-ca-cert" pod "0de902db-4a72-49b2-8f60-8b63663cc98d" (UID: "0de902db-4a72-49b2-8f60-8b63663cc98d"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 01:28:31.867963 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:28:31.867884 4939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0de902db-4a72-49b2-8f60-8b63663cc98d-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0de902db-4a72-49b2-8f60-8b63663cc98d-credential-socket\") pod \"0de902db-4a72-49b2-8f60-8b63663cc98d\" (UID: \"0de902db-4a72-49b2-8f60-8b63663cc98d\") " Nov 03 01:28:31.868622 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:28:31.868596 4939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0de902db-4a72-49b2-8f60-8b63663cc98d-workload-socket" pod "0de902db-4a72-49b2-8f60-8b63663cc98d" (UID: "0de902db-4a72-49b2-8f60-8b63663cc98d"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 01:28:31.868977 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:28:31.868951 4939 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0de902db-4a72-49b2-8f60-8b63663cc98d-istiod-ca-cert\") on node \"ip-10-0-140-107.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:28:31.869084 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:28:31.868986 4939 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0de902db-4a72-49b2-8f60-8b63663cc98d-istio-data\") on node \"ip-10-0-140-107.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:28:31.869084 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:28:31.869004 4939 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0de902db-4a72-49b2-8f60-8b63663cc98d-workload-socket\") on node \"ip-10-0-140-107.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:28:31.869084 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:28:31.869031 4939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0de902db-4a72-49b2-8f60-8b63663cc98d-workload-certs" pod "0de902db-4a72-49b2-8f60-8b63663cc98d" (UID: "0de902db-4a72-49b2-8f60-8b63663cc98d"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 01:28:31.869386 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:28:31.869363 4939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0de902db-4a72-49b2-8f60-8b63663cc98d-credential-socket" pod "0de902db-4a72-49b2-8f60-8b63663cc98d" (UID: "0de902db-4a72-49b2-8f60-8b63663cc98d"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 01:28:31.869628 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:28:31.869593 4939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0de902db-4a72-49b2-8f60-8b63663cc98d-istio-envoy" pod "0de902db-4a72-49b2-8f60-8b63663cc98d" (UID: "0de902db-4a72-49b2-8f60-8b63663cc98d"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 01:28:31.871406 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:28:31.871375 4939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/0de902db-4a72-49b2-8f60-8b63663cc98d-istio-podinfo" pod "0de902db-4a72-49b2-8f60-8b63663cc98d" (UID: "0de902db-4a72-49b2-8f60-8b63663cc98d"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 01:28:31.871406 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:28:31.871389 4939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0de902db-4a72-49b2-8f60-8b63663cc98d-istio-token" pod "0de902db-4a72-49b2-8f60-8b63663cc98d" (UID: "0de902db-4a72-49b2-8f60-8b63663cc98d"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 01:28:31.874366 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:28:31.874344 4939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0de902db-4a72-49b2-8f60-8b63663cc98d-kube-api-access-4g5mt" pod "0de902db-4a72-49b2-8f60-8b63663cc98d" (UID: "0de902db-4a72-49b2-8f60-8b63663cc98d"). InnerVolumeSpecName "kube-api-access-4g5mt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 01:28:31.971375 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:28:31.971340 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4vt9b\" (UniqueName: \"kubernetes.io/projected/05a50b7a-195c-4ec9-9269-5d58faee868b-kube-api-access-4vt9b\") pod \"maas-api-e2e-inf-a-6fec8a-6cdc9b4ff5-xjq85\" (UID: \"05a50b7a-195c-4ec9-9269-5d58faee868b\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-6fec8a-6cdc9b4ff5-xjq85" Nov 03 01:28:31.971554 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:28:31.971386 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/05a50b7a-195c-4ec9-9269-5d58faee868b-maas-api-tls\") pod \"maas-api-e2e-inf-a-6fec8a-6cdc9b4ff5-xjq85\" (UID: \"05a50b7a-195c-4ec9-9269-5d58faee868b\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-6fec8a-6cdc9b4ff5-xjq85" Nov 03 01:28:31.972094 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:28:31.972064 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/05a50b7a-195c-4ec9-9269-5d58faee868b-openshift-service-ca\") pod \"maas-api-e2e-inf-a-6fec8a-6cdc9b4ff5-xjq85\" (UID: \"05a50b7a-195c-4ec9-9269-5d58faee868b\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-6fec8a-6cdc9b4ff5-xjq85" Nov 03 01:28:31.972238 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:28:31.972115 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/05a50b7a-195c-4ec9-9269-5d58faee868b-metrics-certs\") pod \"maas-api-e2e-inf-a-6fec8a-6cdc9b4ff5-xjq85\" (UID: \"05a50b7a-195c-4ec9-9269-5d58faee868b\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-6fec8a-6cdc9b4ff5-xjq85" Nov 03 01:28:31.972335 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:28:31.972313 4939 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0de902db-4a72-49b2-8f60-8b63663cc98d-istio-envoy\") on node \"ip-10-0-140-107.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:28:31.972389 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:28:31.972342 4939 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0de902db-4a72-49b2-8f60-8b63663cc98d-istio-podinfo\") on node \"ip-10-0-140-107.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:28:31.972389 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:28:31.972356 4939 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0de902db-4a72-49b2-8f60-8b63663cc98d-istio-token\") on node \"ip-10-0-140-107.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:28:31.972389 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:28:31.972366 4939 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4g5mt\" (UniqueName: \"kubernetes.io/projected/0de902db-4a72-49b2-8f60-8b63663cc98d-kube-api-access-4g5mt\") on node \"ip-10-0-140-107.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:28:31.972389 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:28:31.972375 4939 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0de902db-4a72-49b2-8f60-8b63663cc98d-workload-certs\") on node \"ip-10-0-140-107.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:28:31.972389 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:28:31.972384 4939 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0de902db-4a72-49b2-8f60-8b63663cc98d-credential-socket\") on node \"ip-10-0-140-107.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:28:32.080449 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:28:32.080342 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4vt9b\" (UniqueName: \"kubernetes.io/projected/05a50b7a-195c-4ec9-9269-5d58faee868b-kube-api-access-4vt9b\") pod \"maas-api-e2e-inf-a-6fec8a-6cdc9b4ff5-xjq85\" (UID: \"05a50b7a-195c-4ec9-9269-5d58faee868b\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-6fec8a-6cdc9b4ff5-xjq85" Nov 03 01:28:32.081149 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:28:32.081120 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/05a50b7a-195c-4ec9-9269-5d58faee868b-maas-api-tls\") pod \"maas-api-e2e-inf-a-6fec8a-6cdc9b4ff5-xjq85\" (UID: \"05a50b7a-195c-4ec9-9269-5d58faee868b\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-6fec8a-6cdc9b4ff5-xjq85" Nov 03 01:28:32.091997 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:28:32.091960 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/05a50b7a-195c-4ec9-9269-5d58faee868b-maas-api-tls\") pod \"maas-api-e2e-inf-a-6fec8a-6cdc9b4ff5-xjq85\" (UID: \"05a50b7a-195c-4ec9-9269-5d58faee868b\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-6fec8a-6cdc9b4ff5-xjq85" Nov 03 01:28:32.094660 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:28:32.094628 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/05a50b7a-195c-4ec9-9269-5d58faee868b-openshift-service-ca\") pod \"maas-api-e2e-inf-a-6fec8a-6cdc9b4ff5-xjq85\" (UID: \"05a50b7a-195c-4ec9-9269-5d58faee868b\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-6fec8a-6cdc9b4ff5-xjq85" Nov 03 01:28:32.102935 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:28:32.101039 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/05a50b7a-195c-4ec9-9269-5d58faee868b-openshift-service-ca\") pod \"maas-api-e2e-inf-a-6fec8a-6cdc9b4ff5-xjq85\" (UID: \"05a50b7a-195c-4ec9-9269-5d58faee868b\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-6fec8a-6cdc9b4ff5-xjq85" Nov 03 01:28:32.102935 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:28:32.101118 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/05a50b7a-195c-4ec9-9269-5d58faee868b-metrics-certs\") pod \"maas-api-e2e-inf-a-6fec8a-6cdc9b4ff5-xjq85\" (UID: \"05a50b7a-195c-4ec9-9269-5d58faee868b\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-6fec8a-6cdc9b4ff5-xjq85" Nov 03 01:28:32.127931 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:28:32.114929 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4vt9b\" (UniqueName: \"kubernetes.io/projected/05a50b7a-195c-4ec9-9269-5d58faee868b-kube-api-access-4vt9b\") pod \"maas-api-e2e-inf-a-6fec8a-6cdc9b4ff5-xjq85\" (UID: \"05a50b7a-195c-4ec9-9269-5d58faee868b\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-6fec8a-6cdc9b4ff5-xjq85" Nov 03 01:28:32.128728 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:28:32.128701 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/05a50b7a-195c-4ec9-9269-5d58faee868b-metrics-certs\") pod \"maas-api-e2e-inf-a-6fec8a-6cdc9b4ff5-xjq85\" (UID: \"05a50b7a-195c-4ec9-9269-5d58faee868b\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-6fec8a-6cdc9b4ff5-xjq85" Nov 03 01:28:32.167931 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:28:32.161325 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:28:32.169586 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:28:32.169560 4939 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-6fec8a-6cdc9b4ff5-xjq85" Nov 03 01:28:32.726883 ip-10-0-140-107 kubenswrapper[4939]: W1103 01:28:32.726852 4939 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod05a50b7a_195c_4ec9_9269_5d58faee868b.slice/crio-7c7b01bd58665cbfd8d8ce6f0521be9e6e79f30f6ec8b9a3c14a7555995442f8 WatchSource:0}: Error finding container 7c7b01bd58665cbfd8d8ce6f0521be9e6e79f30f6ec8b9a3c14a7555995442f8: Status 404 returned error can't find the container with id 7c7b01bd58665cbfd8d8ce6f0521be9e6e79f30f6ec8b9a3c14a7555995442f8 Nov 03 01:28:32.728681 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:28:32.728658 4939 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Nov 03 01:28:32.728789 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:28:32.728671 4939 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-6fec8a-6cdc9b4ff5-xjq85"] Nov 03 01:28:32.762765 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:28:32.762701 4939 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-a88862-openshift-default-7b98d598f-jbgzc" Nov 03 01:28:32.764365 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:28:32.764335 4939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-6fec8a-6cdc9b4ff5-xjq85" event={"ID":"05a50b7a-195c-4ec9-9269-5d58faee868b","Type":"ContainerStarted","Data":"7c7b01bd58665cbfd8d8ce6f0521be9e6e79f30f6ec8b9a3c14a7555995442f8"} Nov 03 01:28:32.799590 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:28:32.799566 4939 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-a88862-openshift-default-7b98d598f-jbgzc"] Nov 03 01:28:32.804763 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:28:32.804738 4939 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-a-a88862-openshift-default-7b98d598f-jbgzc"] Nov 03 01:28:33.004617 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:28:33.004568 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:28:33.634474 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:28:33.634350 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:28:33.747743 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:28:33.747708 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:28:33.779010 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:28:33.778977 4939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-6fec8a-6cdc9b4ff5-xjq85" event={"ID":"05a50b7a-195c-4ec9-9269-5d58faee868b","Type":"ContainerStarted","Data":"8d4b58e052a8f6f6d7c66b430f1e71b6f4a161f8f4de727bacc5e5b79b5a0a55"} Nov 03 01:28:33.779181 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:28:33.779107 4939 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-6fec8a-6cdc9b4ff5-xjq85" Nov 03 01:28:33.799617 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:28:33.799571 4939 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-6fec8a-6cdc9b4ff5-xjq85" podStartSLOduration=2.280185927 podStartE2EDuration="2.799558861s" totalImagesPullingTime="519.372934ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 01:28:31 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 01:28:33.798585378 +0000 UTC m=+2862.062920331" watchObservedRunningTime="2026-11-03 01:28:33.799558861 +0000 UTC m=+2862.063893805" Nov 03 01:28:33.859772 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:28:33.859735 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:28:33.876255 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:28:33.876226 4939 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0de902db-4a72-49b2-8f60-8b63663cc98d" path="/var/lib/kubelet/pods/0de902db-4a72-49b2-8f60-8b63663cc98d/volumes" Nov 03 01:28:34.374011 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:28:34.373951 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:28:35.083582 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:28:35.083550 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:28:35.678289 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:28:35.678235 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:28:36.303737 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:28:36.303698 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:28:37.096268 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:28:37.096234 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:28:37.539813 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:28:37.539727 4939 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-inf-a-a88862-6bccb6967f-46486" podUID="dd50d5d4-bbf2-439b-a851-2c2ae58da605" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 03 01:28:37.776106 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:28:37.776073 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:28:38.538945 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:28:38.537194 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:28:38.880928 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:28:38.880379 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:28:39.469393 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:28:39.469363 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:28:39.664635 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:28:39.664601 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:28:39.794789 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:28:39.794719 4939 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-6fec8a-6cdc9b4ff5-xjq85" Nov 03 01:28:39.911717 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:28:39.911679 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:28:40.200867 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:28:40.200823 4939 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-ce04af-openshift-default-7dccd45f47-75xwt"] Nov 03 01:28:40.201250 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:28:40.201113 4939 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-b-ce04af-openshift-default-7dccd45f47-75xwt" podUID="8ea9e0d9-99b8-4bbb-b98d-b459f1560c89" containerName="istio-proxy" containerID="cri-o://a24f21dac1517c1ad57ee59169b7a7401d3a8349983a274d17f83f89201f147c" gracePeriod=30 Nov 03 01:28:40.217976 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:28:40.217552 4939 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-ce04af-openshift-default-7dccd45f47-vxtn5"] Nov 03 01:28:40.217976 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:28:40.217769 4939 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-b-ce04af-openshift-default-7dccd45f47-vxtn5" podUID="2c2192ff-91c4-4488-8229-629018fd881d" containerName="istio-proxy" containerID="cri-o://f82319615907c985e605951dc5dc72d7600b1fbb6431ed69d94645c064d8dc84" gracePeriod=30 Nov 03 01:28:40.257823 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:28:40.249966 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:28:40.257823 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:28:40.250095 4939 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-ce04af-openshift-default-7dccd45f47-n5ndk"] Nov 03 01:28:40.257823 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:28:40.250317 4939 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-b-ce04af-openshift-default-7dccd45f47-n5ndk" podUID="5188b487-c967-42f8-a5c8-43f118b3e50d" containerName="istio-proxy" containerID="cri-o://9c4f71736d18633fc600dc773a00b3ec7b3814be88b7cebaf2e86d5fac01463c" gracePeriod=30 Nov 03 01:28:40.276526 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:28:40.276480 4939 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-ce04af-openshift-default-7dccd45f47-wc5pz"] Nov 03 01:28:40.276769 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:28:40.276742 4939 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-b-ce04af-openshift-default-7dccd45f47-wc5pz" podUID="0412ca91-563b-4ee7-8a02-4f4e768e51f7" containerName="istio-proxy" containerID="cri-o://a1f057279f28d75acdb60e495251a558ad6fdfa1ff112c36890e28722ac1cf25" gracePeriod=30 Nov 03 01:28:40.753458 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:28:40.753422 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:28:41.155509 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:28:41.155456 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:28:41.259240 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:28:41.259210 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:28:41.369700 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:28:41.369661 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:28:41.656124 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:28:41.656083 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:28:42.448681 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:28:42.448642 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:28:43.170028 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:28:43.169993 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:28:43.322429 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:28:43.322392 4939 patch_prober.go:29] interesting pod/e2e-inf-b-ce04af-openshift-default-7dccd45f47-75xwt container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 03 01:28:43.322611 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:28:43.322462 4939 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-b-ce04af-openshift-default-7dccd45f47-75xwt" podUID="8ea9e0d9-99b8-4bbb-b98d-b459f1560c89" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 03 01:28:43.842523 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:28:43.842493 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:28:43.958714 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:28:43.955373 4939 patch_prober.go:29] interesting pod/e2e-inf-b-ce04af-openshift-default-7dccd45f47-n5ndk container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 03 01:28:43.958714 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:28:43.955443 4939 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-b-ce04af-openshift-default-7dccd45f47-n5ndk" podUID="5188b487-c967-42f8-a5c8-43f118b3e50d" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 03 01:28:44.060961 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:28:44.060871 4939 generic.go:356] "Generic (PLEG): container finished" podID="dd50d5d4-bbf2-439b-a851-2c2ae58da605" containerID="d022249c6b7f0ad709448febf97a28b05d4540b81fe89ae5ce58e2542e963980" exitCode=0 Nov 03 01:28:44.061195 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:28:44.060960 4939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-a88862-6bccb6967f-46486" event={"ID":"dd50d5d4-bbf2-439b-a851-2c2ae58da605","Type":"ContainerDied","Data":"d022249c6b7f0ad709448febf97a28b05d4540b81fe89ae5ce58e2542e963980"} Nov 03 01:28:44.061282 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:28:44.061230 4939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-a88862-6bccb6967f-46486" event={"ID":"dd50d5d4-bbf2-439b-a851-2c2ae58da605","Type":"ContainerDied","Data":"a1f6041cba2b4072fbd2bc9dfdf304dc1886f5bc5f01ea8a85d863607edae3d3"} Nov 03 01:28:44.061282 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:28:44.061260 4939 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a1f6041cba2b4072fbd2bc9dfdf304dc1886f5bc5f01ea8a85d863607edae3d3" Nov 03 01:28:44.084776 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:28:44.084749 4939 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-a88862-6bccb6967f-46486" Nov 03 01:28:44.158552 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:28:44.158520 4939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dd50d5d4-bbf2-439b-a851-2c2ae58da605-kube-api-access-6q5bz\" (UniqueName: \"kubernetes.io/projected/dd50d5d4-bbf2-439b-a851-2c2ae58da605-kube-api-access-6q5bz\") pod \"dd50d5d4-bbf2-439b-a851-2c2ae58da605\" (UID: \"dd50d5d4-bbf2-439b-a851-2c2ae58da605\") " Nov 03 01:28:44.158942 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:28:44.158920 4939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/dd50d5d4-bbf2-439b-a851-2c2ae58da605-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dd50d5d4-bbf2-439b-a851-2c2ae58da605-plugins-config-volume\") pod \"dd50d5d4-bbf2-439b-a851-2c2ae58da605\" (UID: \"dd50d5d4-bbf2-439b-a851-2c2ae58da605\") " Nov 03 01:28:44.159376 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:28:44.159346 4939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dd50d5d4-bbf2-439b-a851-2c2ae58da605-plugins-config-volume" pod "dd50d5d4-bbf2-439b-a851-2c2ae58da605" (UID: "dd50d5d4-bbf2-439b-a851-2c2ae58da605"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 01:28:44.159977 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:28:44.159953 4939 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dd50d5d4-bbf2-439b-a851-2c2ae58da605-plugins-config-volume\") on node \"ip-10-0-140-107.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:28:44.163166 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:28:44.163108 4939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dd50d5d4-bbf2-439b-a851-2c2ae58da605-kube-api-access-6q5bz" pod "dd50d5d4-bbf2-439b-a851-2c2ae58da605" (UID: "dd50d5d4-bbf2-439b-a851-2c2ae58da605"). InnerVolumeSpecName "kube-api-access-6q5bz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 01:28:44.261757 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:28:44.261723 4939 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6q5bz\" (UniqueName: \"kubernetes.io/projected/dd50d5d4-bbf2-439b-a851-2c2ae58da605-kube-api-access-6q5bz\") on node \"ip-10-0-140-107.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:28:44.444165 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:28:44.444119 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:28:44.837898 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:28:44.837837 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:28:45.063712 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:28:45.063668 4939 patch_prober.go:29] interesting pod/e2e-inf-b-ce04af-openshift-default-7dccd45f47-vxtn5 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 03 01:28:45.064187 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:28:45.063734 4939 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-b-ce04af-openshift-default-7dccd45f47-vxtn5" podUID="2c2192ff-91c4-4488-8229-629018fd881d" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 03 01:28:45.069963 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:28:45.069902 4939 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-a88862-6bccb6967f-46486" Nov 03 01:28:45.096055 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:28:45.095999 4939 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-a88862-6bccb6967f-46486"] Nov 03 01:28:45.103032 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:28:45.103006 4939 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-a88862-6bccb6967f-46486"] Nov 03 01:28:45.105731 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:28:45.105692 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:28:45.614222 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:28:45.611628 4939 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-ce04af-openshift-default-7dccd45f47-n5ndk" Nov 03 01:28:45.653540 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:28:45.653484 4939 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-a-ee618c-openshift-default-5ff9c74dc9-z4qxq"] Nov 03 01:28:45.654948 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:28:45.654896 4939 cpu_manager.go:414] "containerMap: removing container" podUID="0de902db-4a72-49b2-8f60-8b63663cc98d" containerName="istio-proxy" Nov 03 01:28:45.654948 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:28:45.654935 4939 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0de902db-4a72-49b2-8f60-8b63663cc98d" containerName="istio-proxy" Nov 03 01:28:45.655165 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:28:45.654959 4939 cpu_manager.go:414] "containerMap: removing container" podUID="dd50d5d4-bbf2-439b-a851-2c2ae58da605" containerName="payload-processing" Nov 03 01:28:45.655165 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:28:45.654977 4939 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dd50d5d4-bbf2-439b-a851-2c2ae58da605" containerName="payload-processing" Nov 03 01:28:45.655165 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:28:45.654989 4939 cpu_manager.go:414] "containerMap: removing container" podUID="5188b487-c967-42f8-a5c8-43f118b3e50d" containerName="istio-proxy" Nov 03 01:28:45.655165 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:28:45.654996 4939 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5188b487-c967-42f8-a5c8-43f118b3e50d" containerName="istio-proxy" Nov 03 01:28:45.655446 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:28:45.655308 4939 memory_manager.go:365] "RemoveStaleState removing state" podUID="dd50d5d4-bbf2-439b-a851-2c2ae58da605" containerName="payload-processing" Nov 03 01:28:45.655446 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:28:45.655324 4939 memory_manager.go:365] "RemoveStaleState removing state" podUID="5188b487-c967-42f8-a5c8-43f118b3e50d" containerName="istio-proxy" Nov 03 01:28:45.683932 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:28:45.678621 4939 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-ee618c-openshift-default-5ff9c74dc9-z4qxq" Nov 03 01:28:45.683932 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:28:45.682745 4939 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-ee618c-openshift-default-5ff9c74dc9-z4qxq"] Nov 03 01:28:45.702929 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:28:45.693403 4939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/5188b487-c967-42f8-a5c8-43f118b3e50d-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5188b487-c967-42f8-a5c8-43f118b3e50d-istio-podinfo\") pod \"5188b487-c967-42f8-a5c8-43f118b3e50d\" (UID: \"5188b487-c967-42f8-a5c8-43f118b3e50d\") " Nov 03 01:28:45.702929 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:28:45.693507 4939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5188b487-c967-42f8-a5c8-43f118b3e50d-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5188b487-c967-42f8-a5c8-43f118b3e50d-credential-socket\") pod \"5188b487-c967-42f8-a5c8-43f118b3e50d\" (UID: \"5188b487-c967-42f8-a5c8-43f118b3e50d\") " Nov 03 01:28:45.702929 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:28:45.693562 4939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5188b487-c967-42f8-a5c8-43f118b3e50d-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5188b487-c967-42f8-a5c8-43f118b3e50d-istiod-ca-cert\") pod \"5188b487-c967-42f8-a5c8-43f118b3e50d\" (UID: \"5188b487-c967-42f8-a5c8-43f118b3e50d\") " Nov 03 01:28:45.702929 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:28:45.693633 4939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5188b487-c967-42f8-a5c8-43f118b3e50d-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5188b487-c967-42f8-a5c8-43f118b3e50d-istio-envoy\") pod \"5188b487-c967-42f8-a5c8-43f118b3e50d\" (UID: \"5188b487-c967-42f8-a5c8-43f118b3e50d\") " Nov 03 01:28:45.702929 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:28:45.693774 4939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5188b487-c967-42f8-a5c8-43f118b3e50d-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5188b487-c967-42f8-a5c8-43f118b3e50d-istio-data\") pod \"5188b487-c967-42f8-a5c8-43f118b3e50d\" (UID: \"5188b487-c967-42f8-a5c8-43f118b3e50d\") " Nov 03 01:28:45.702929 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:28:45.693972 4939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5188b487-c967-42f8-a5c8-43f118b3e50d-kube-api-access-dsrt8\" (UniqueName: \"kubernetes.io/projected/5188b487-c967-42f8-a5c8-43f118b3e50d-kube-api-access-dsrt8\") pod \"5188b487-c967-42f8-a5c8-43f118b3e50d\" (UID: \"5188b487-c967-42f8-a5c8-43f118b3e50d\") " Nov 03 01:28:45.702929 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:28:45.694032 4939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5188b487-c967-42f8-a5c8-43f118b3e50d-istio-token\" (UniqueName: \"kubernetes.io/projected/5188b487-c967-42f8-a5c8-43f118b3e50d-istio-token\") pod \"5188b487-c967-42f8-a5c8-43f118b3e50d\" (UID: \"5188b487-c967-42f8-a5c8-43f118b3e50d\") " Nov 03 01:28:45.702929 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:28:45.694166 4939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5188b487-c967-42f8-a5c8-43f118b3e50d-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5188b487-c967-42f8-a5c8-43f118b3e50d-workload-socket\") pod \"5188b487-c967-42f8-a5c8-43f118b3e50d\" (UID: \"5188b487-c967-42f8-a5c8-43f118b3e50d\") " Nov 03 01:28:45.702929 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:28:45.694261 4939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5188b487-c967-42f8-a5c8-43f118b3e50d-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5188b487-c967-42f8-a5c8-43f118b3e50d-workload-certs\") pod \"5188b487-c967-42f8-a5c8-43f118b3e50d\" (UID: \"5188b487-c967-42f8-a5c8-43f118b3e50d\") " Nov 03 01:28:45.702929 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:28:45.695373 4939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5188b487-c967-42f8-a5c8-43f118b3e50d-credential-socket" pod "5188b487-c967-42f8-a5c8-43f118b3e50d" (UID: "5188b487-c967-42f8-a5c8-43f118b3e50d"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 01:28:45.702929 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:28:45.695789 4939 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5188b487-c967-42f8-a5c8-43f118b3e50d-credential-socket\") on node \"ip-10-0-140-107.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:28:45.702929 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:28:45.696372 4939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5188b487-c967-42f8-a5c8-43f118b3e50d-workload-certs" pod "5188b487-c967-42f8-a5c8-43f118b3e50d" (UID: "5188b487-c967-42f8-a5c8-43f118b3e50d"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 01:28:45.702929 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:28:45.698308 4939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5188b487-c967-42f8-a5c8-43f118b3e50d-istio-data" pod "5188b487-c967-42f8-a5c8-43f118b3e50d" (UID: "5188b487-c967-42f8-a5c8-43f118b3e50d"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 01:28:45.707216 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:28:45.707172 4939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5188b487-c967-42f8-a5c8-43f118b3e50d-workload-socket" pod "5188b487-c967-42f8-a5c8-43f118b3e50d" (UID: "5188b487-c967-42f8-a5c8-43f118b3e50d"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 01:28:45.713352 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:28:45.713324 4939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/5188b487-c967-42f8-a5c8-43f118b3e50d-istio-podinfo" pod "5188b487-c967-42f8-a5c8-43f118b3e50d" (UID: "5188b487-c967-42f8-a5c8-43f118b3e50d"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 01:28:45.713485 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:28:45.713424 4939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5188b487-c967-42f8-a5c8-43f118b3e50d-kube-api-access-dsrt8" pod "5188b487-c967-42f8-a5c8-43f118b3e50d" (UID: "5188b487-c967-42f8-a5c8-43f118b3e50d"). InnerVolumeSpecName "kube-api-access-dsrt8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 01:28:45.713485 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:28:45.713456 4939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5188b487-c967-42f8-a5c8-43f118b3e50d-istio-envoy" pod "5188b487-c967-42f8-a5c8-43f118b3e50d" (UID: "5188b487-c967-42f8-a5c8-43f118b3e50d"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 01:28:45.713748 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:28:45.713726 4939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5188b487-c967-42f8-a5c8-43f118b3e50d-istiod-ca-cert" pod "5188b487-c967-42f8-a5c8-43f118b3e50d" (UID: "5188b487-c967-42f8-a5c8-43f118b3e50d"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 01:28:45.715290 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:28:45.715260 4939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5188b487-c967-42f8-a5c8-43f118b3e50d-istio-token" pod "5188b487-c967-42f8-a5c8-43f118b3e50d" (UID: "5188b487-c967-42f8-a5c8-43f118b3e50d"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 01:28:45.728134 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:28:45.728107 4939 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-ce04af-openshift-default-7dccd45f47-75xwt" Nov 03 01:28:45.744671 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:28:45.744647 4939 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-ce04af-openshift-default-7dccd45f47-vxtn5" Nov 03 01:28:45.750669 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:28:45.750608 4939 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-ce04af-openshift-default-7dccd45f47-wc5pz" Nov 03 01:28:45.764869 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:28:45.764846 4939 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-a-ee618c-openshift-default-5ff9c74dc9-c4jpz"] Nov 03 01:28:45.766110 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:28:45.766090 4939 cpu_manager.go:414] "containerMap: removing container" podUID="8ea9e0d9-99b8-4bbb-b98d-b459f1560c89" containerName="istio-proxy" Nov 03 01:28:45.766221 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:28:45.766212 4939 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8ea9e0d9-99b8-4bbb-b98d-b459f1560c89" containerName="istio-proxy" Nov 03 01:28:45.766348 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:28:45.766340 4939 cpu_manager.go:414] "containerMap: removing container" podUID="2c2192ff-91c4-4488-8229-629018fd881d" containerName="istio-proxy" Nov 03 01:28:45.766404 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:28:45.766397 4939 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2c2192ff-91c4-4488-8229-629018fd881d" containerName="istio-proxy" Nov 03 01:28:45.766461 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:28:45.766454 4939 cpu_manager.go:414] "containerMap: removing container" podUID="0412ca91-563b-4ee7-8a02-4f4e768e51f7" containerName="istio-proxy" Nov 03 01:28:45.766508 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:28:45.766501 4939 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0412ca91-563b-4ee7-8a02-4f4e768e51f7" containerName="istio-proxy" Nov 03 01:28:45.766695 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:28:45.766685 4939 memory_manager.go:365] "RemoveStaleState removing state" podUID="8ea9e0d9-99b8-4bbb-b98d-b459f1560c89" containerName="istio-proxy" Nov 03 01:28:45.766779 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:28:45.766772 4939 memory_manager.go:365] "RemoveStaleState removing state" podUID="2c2192ff-91c4-4488-8229-629018fd881d" containerName="istio-proxy" Nov 03 01:28:45.766833 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:28:45.766827 4939 memory_manager.go:365] "RemoveStaleState removing state" podUID="0412ca91-563b-4ee7-8a02-4f4e768e51f7" containerName="istio-proxy" Nov 03 01:28:45.772179 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:28:45.772158 4939 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-ee618c-openshift-default-5ff9c74dc9-c4jpz" Nov 03 01:28:45.790122 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:28:45.790093 4939 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-ee618c-openshift-default-5ff9c74dc9-c4jpz"] Nov 03 01:28:45.796538 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:28:45.796504 4939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2c2192ff-91c4-4488-8229-629018fd881d-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2c2192ff-91c4-4488-8229-629018fd881d-istio-envoy\") pod \"2c2192ff-91c4-4488-8229-629018fd881d\" (UID: \"2c2192ff-91c4-4488-8229-629018fd881d\") " Nov 03 01:28:45.796687 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:28:45.796587 4939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0412ca91-563b-4ee7-8a02-4f4e768e51f7-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0412ca91-563b-4ee7-8a02-4f4e768e51f7-istio-envoy\") pod \"0412ca91-563b-4ee7-8a02-4f4e768e51f7\" (UID: \"0412ca91-563b-4ee7-8a02-4f4e768e51f7\") " Nov 03 01:28:45.796687 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:28:45.796635 4939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0412ca91-563b-4ee7-8a02-4f4e768e51f7-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0412ca91-563b-4ee7-8a02-4f4e768e51f7-workload-certs\") pod \"0412ca91-563b-4ee7-8a02-4f4e768e51f7\" (UID: \"0412ca91-563b-4ee7-8a02-4f4e768e51f7\") " Nov 03 01:28:45.796798 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:28:45.796700 4939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2c2192ff-91c4-4488-8229-629018fd881d-istio-token\" (UniqueName: \"kubernetes.io/projected/2c2192ff-91c4-4488-8229-629018fd881d-istio-token\") pod \"2c2192ff-91c4-4488-8229-629018fd881d\" (UID: \"2c2192ff-91c4-4488-8229-629018fd881d\") " Nov 03 01:28:45.796798 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:28:45.796746 4939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0412ca91-563b-4ee7-8a02-4f4e768e51f7-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0412ca91-563b-4ee7-8a02-4f4e768e51f7-credential-socket\") pod \"0412ca91-563b-4ee7-8a02-4f4e768e51f7\" (UID: \"0412ca91-563b-4ee7-8a02-4f4e768e51f7\") " Nov 03 01:28:45.796798 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:28:45.796790 4939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2c2192ff-91c4-4488-8229-629018fd881d-kube-api-access-jh5rp\" (UniqueName: \"kubernetes.io/projected/2c2192ff-91c4-4488-8229-629018fd881d-kube-api-access-jh5rp\") pod \"2c2192ff-91c4-4488-8229-629018fd881d\" (UID: \"2c2192ff-91c4-4488-8229-629018fd881d\") " Nov 03 01:28:45.796982 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:28:45.796826 4939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8ea9e0d9-99b8-4bbb-b98d-b459f1560c89-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8ea9e0d9-99b8-4bbb-b98d-b459f1560c89-workload-socket\") pod \"8ea9e0d9-99b8-4bbb-b98d-b459f1560c89\" (UID: \"8ea9e0d9-99b8-4bbb-b98d-b459f1560c89\") " Nov 03 01:28:45.797047 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:28:45.796997 4939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8ea9e0d9-99b8-4bbb-b98d-b459f1560c89-istio-token\" (UniqueName: \"kubernetes.io/projected/8ea9e0d9-99b8-4bbb-b98d-b459f1560c89-istio-token\") pod \"8ea9e0d9-99b8-4bbb-b98d-b459f1560c89\" (UID: \"8ea9e0d9-99b8-4bbb-b98d-b459f1560c89\") " Nov 03 01:28:45.797047 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:28:45.797037 4939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/0412ca91-563b-4ee7-8a02-4f4e768e51f7-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0412ca91-563b-4ee7-8a02-4f4e768e51f7-istio-podinfo\") pod \"0412ca91-563b-4ee7-8a02-4f4e768e51f7\" (UID: \"0412ca91-563b-4ee7-8a02-4f4e768e51f7\") " Nov 03 01:28:45.797219 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:28:45.797075 4939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8ea9e0d9-99b8-4bbb-b98d-b459f1560c89-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8ea9e0d9-99b8-4bbb-b98d-b459f1560c89-istio-data\") pod \"8ea9e0d9-99b8-4bbb-b98d-b459f1560c89\" (UID: \"8ea9e0d9-99b8-4bbb-b98d-b459f1560c89\") " Nov 03 01:28:45.797219 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:28:45.797109 4939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/2c2192ff-91c4-4488-8229-629018fd881d-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2c2192ff-91c4-4488-8229-629018fd881d-istio-podinfo\") pod \"2c2192ff-91c4-4488-8229-629018fd881d\" (UID: \"2c2192ff-91c4-4488-8229-629018fd881d\") " Nov 03 01:28:45.797219 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:28:45.797142 4939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0412ca91-563b-4ee7-8a02-4f4e768e51f7-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0412ca91-563b-4ee7-8a02-4f4e768e51f7-istiod-ca-cert\") pod \"0412ca91-563b-4ee7-8a02-4f4e768e51f7\" (UID: \"0412ca91-563b-4ee7-8a02-4f4e768e51f7\") " Nov 03 01:28:45.797390 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:28:45.797248 4939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2c2192ff-91c4-4488-8229-629018fd881d-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2c2192ff-91c4-4488-8229-629018fd881d-workload-certs\") pod \"2c2192ff-91c4-4488-8229-629018fd881d\" (UID: \"2c2192ff-91c4-4488-8229-629018fd881d\") " Nov 03 01:28:45.797390 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:28:45.797365 4939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8ea9e0d9-99b8-4bbb-b98d-b459f1560c89-kube-api-access-zdrp9\" (UniqueName: \"kubernetes.io/projected/8ea9e0d9-99b8-4bbb-b98d-b459f1560c89-kube-api-access-zdrp9\") pod \"8ea9e0d9-99b8-4bbb-b98d-b459f1560c89\" (UID: \"8ea9e0d9-99b8-4bbb-b98d-b459f1560c89\") " Nov 03 01:28:45.797504 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:28:45.797400 4939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2c2192ff-91c4-4488-8229-629018fd881d-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2c2192ff-91c4-4488-8229-629018fd881d-workload-socket\") pod \"2c2192ff-91c4-4488-8229-629018fd881d\" (UID: \"2c2192ff-91c4-4488-8229-629018fd881d\") " Nov 03 01:28:45.797504 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:28:45.797432 4939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0412ca91-563b-4ee7-8a02-4f4e768e51f7-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0412ca91-563b-4ee7-8a02-4f4e768e51f7-istio-data\") pod \"0412ca91-563b-4ee7-8a02-4f4e768e51f7\" (UID: \"0412ca91-563b-4ee7-8a02-4f4e768e51f7\") " Nov 03 01:28:45.797614 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:28:45.797547 4939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8ea9e0d9-99b8-4bbb-b98d-b459f1560c89-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8ea9e0d9-99b8-4bbb-b98d-b459f1560c89-istio-envoy\") pod \"8ea9e0d9-99b8-4bbb-b98d-b459f1560c89\" (UID: \"8ea9e0d9-99b8-4bbb-b98d-b459f1560c89\") " Nov 03 01:28:45.797692 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:28:45.797610 4939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8ea9e0d9-99b8-4bbb-b98d-b459f1560c89-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8ea9e0d9-99b8-4bbb-b98d-b459f1560c89-istiod-ca-cert\") pod \"8ea9e0d9-99b8-4bbb-b98d-b459f1560c89\" (UID: \"8ea9e0d9-99b8-4bbb-b98d-b459f1560c89\") " Nov 03 01:28:45.797692 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:28:45.797650 4939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8ea9e0d9-99b8-4bbb-b98d-b459f1560c89-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8ea9e0d9-99b8-4bbb-b98d-b459f1560c89-workload-certs\") pod \"8ea9e0d9-99b8-4bbb-b98d-b459f1560c89\" (UID: \"8ea9e0d9-99b8-4bbb-b98d-b459f1560c89\") " Nov 03 01:28:45.797692 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:28:45.797686 4939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/8ea9e0d9-99b8-4bbb-b98d-b459f1560c89-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8ea9e0d9-99b8-4bbb-b98d-b459f1560c89-istio-podinfo\") pod \"8ea9e0d9-99b8-4bbb-b98d-b459f1560c89\" (UID: \"8ea9e0d9-99b8-4bbb-b98d-b459f1560c89\") " Nov 03 01:28:45.797852 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:28:45.797740 4939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0412ca91-563b-4ee7-8a02-4f4e768e51f7-istio-token\" (UniqueName: \"kubernetes.io/projected/0412ca91-563b-4ee7-8a02-4f4e768e51f7-istio-token\") pod \"0412ca91-563b-4ee7-8a02-4f4e768e51f7\" (UID: \"0412ca91-563b-4ee7-8a02-4f4e768e51f7\") " Nov 03 01:28:45.797852 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:28:45.797778 4939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0412ca91-563b-4ee7-8a02-4f4e768e51f7-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0412ca91-563b-4ee7-8a02-4f4e768e51f7-workload-socket\") pod \"0412ca91-563b-4ee7-8a02-4f4e768e51f7\" (UID: \"0412ca91-563b-4ee7-8a02-4f4e768e51f7\") " Nov 03 01:28:45.797852 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:28:45.797833 4939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0412ca91-563b-4ee7-8a02-4f4e768e51f7-kube-api-access-8lpdh\" (UniqueName: \"kubernetes.io/projected/0412ca91-563b-4ee7-8a02-4f4e768e51f7-kube-api-access-8lpdh\") pod \"0412ca91-563b-4ee7-8a02-4f4e768e51f7\" (UID: \"0412ca91-563b-4ee7-8a02-4f4e768e51f7\") " Nov 03 01:28:45.798041 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:28:45.797948 4939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2c2192ff-91c4-4488-8229-629018fd881d-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2c2192ff-91c4-4488-8229-629018fd881d-istio-data\") pod \"2c2192ff-91c4-4488-8229-629018fd881d\" (UID: \"2c2192ff-91c4-4488-8229-629018fd881d\") " Nov 03 01:28:45.798041 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:28:45.797994 4939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8ea9e0d9-99b8-4bbb-b98d-b459f1560c89-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8ea9e0d9-99b8-4bbb-b98d-b459f1560c89-credential-socket\") pod \"8ea9e0d9-99b8-4bbb-b98d-b459f1560c89\" (UID: \"8ea9e0d9-99b8-4bbb-b98d-b459f1560c89\") " Nov 03 01:28:45.798147 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:28:45.798061 4939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2c2192ff-91c4-4488-8229-629018fd881d-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2c2192ff-91c4-4488-8229-629018fd881d-istiod-ca-cert\") pod \"2c2192ff-91c4-4488-8229-629018fd881d\" (UID: \"2c2192ff-91c4-4488-8229-629018fd881d\") " Nov 03 01:28:45.798147 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:28:45.798112 4939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2c2192ff-91c4-4488-8229-629018fd881d-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2c2192ff-91c4-4488-8229-629018fd881d-credential-socket\") pod \"2c2192ff-91c4-4488-8229-629018fd881d\" (UID: \"2c2192ff-91c4-4488-8229-629018fd881d\") " Nov 03 01:28:45.798602 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:28:45.798574 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6165abef-0df3-4007-ae59-5536a2a77e45-istio-token\") pod \"e2e-inf-a-ee618c-openshift-default-5ff9c74dc9-z4qxq\" (UID: \"6165abef-0df3-4007-ae59-5536a2a77e45\") " pod="openshift-ingress/e2e-inf-a-ee618c-openshift-default-5ff9c74dc9-z4qxq" Nov 03 01:28:45.798683 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:28:45.798659 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6165abef-0df3-4007-ae59-5536a2a77e45-workload-socket\") pod \"e2e-inf-a-ee618c-openshift-default-5ff9c74dc9-z4qxq\" (UID: \"6165abef-0df3-4007-ae59-5536a2a77e45\") " pod="openshift-ingress/e2e-inf-a-ee618c-openshift-default-5ff9c74dc9-z4qxq" Nov 03 01:28:45.799026 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:28:45.799001 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6165abef-0df3-4007-ae59-5536a2a77e45-istio-podinfo\") pod \"e2e-inf-a-ee618c-openshift-default-5ff9c74dc9-z4qxq\" (UID: \"6165abef-0df3-4007-ae59-5536a2a77e45\") " pod="openshift-ingress/e2e-inf-a-ee618c-openshift-default-5ff9c74dc9-z4qxq" Nov 03 01:28:45.799168 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:28:45.799150 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6165abef-0df3-4007-ae59-5536a2a77e45-credential-socket\") pod \"e2e-inf-a-ee618c-openshift-default-5ff9c74dc9-z4qxq\" (UID: \"6165abef-0df3-4007-ae59-5536a2a77e45\") " pod="openshift-ingress/e2e-inf-a-ee618c-openshift-default-5ff9c74dc9-z4qxq" Nov 03 01:28:45.799239 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:28:45.799193 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6165abef-0df3-4007-ae59-5536a2a77e45-workload-certs\") pod \"e2e-inf-a-ee618c-openshift-default-5ff9c74dc9-z4qxq\" (UID: \"6165abef-0df3-4007-ae59-5536a2a77e45\") " pod="openshift-ingress/e2e-inf-a-ee618c-openshift-default-5ff9c74dc9-z4qxq" Nov 03 01:28:45.799239 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:28:45.799222 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5lqkc\" (UniqueName: \"kubernetes.io/projected/6165abef-0df3-4007-ae59-5536a2a77e45-kube-api-access-5lqkc\") pod \"e2e-inf-a-ee618c-openshift-default-5ff9c74dc9-z4qxq\" (UID: \"6165abef-0df3-4007-ae59-5536a2a77e45\") " pod="openshift-ingress/e2e-inf-a-ee618c-openshift-default-5ff9c74dc9-z4qxq" Nov 03 01:28:45.799499 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:28:45.799478 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6165abef-0df3-4007-ae59-5536a2a77e45-istio-data\") pod \"e2e-inf-a-ee618c-openshift-default-5ff9c74dc9-z4qxq\" (UID: \"6165abef-0df3-4007-ae59-5536a2a77e45\") " pod="openshift-ingress/e2e-inf-a-ee618c-openshift-default-5ff9c74dc9-z4qxq" Nov 03 01:28:45.799591 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:28:45.799574 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6165abef-0df3-4007-ae59-5536a2a77e45-istio-envoy\") pod \"e2e-inf-a-ee618c-openshift-default-5ff9c74dc9-z4qxq\" (UID: \"6165abef-0df3-4007-ae59-5536a2a77e45\") " pod="openshift-ingress/e2e-inf-a-ee618c-openshift-default-5ff9c74dc9-z4qxq" Nov 03 01:28:45.799672 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:28:45.799656 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6165abef-0df3-4007-ae59-5536a2a77e45-istiod-ca-cert\") pod \"e2e-inf-a-ee618c-openshift-default-5ff9c74dc9-z4qxq\" (UID: \"6165abef-0df3-4007-ae59-5536a2a77e45\") " pod="openshift-ingress/e2e-inf-a-ee618c-openshift-default-5ff9c74dc9-z4qxq" Nov 03 01:28:45.799896 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:28:45.799878 4939 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5188b487-c967-42f8-a5c8-43f118b3e50d-istio-envoy\") on node \"ip-10-0-140-107.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:28:45.800002 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:28:45.799902 4939 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5188b487-c967-42f8-a5c8-43f118b3e50d-istio-data\") on node \"ip-10-0-140-107.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:28:45.800002 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:28:45.799945 4939 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dsrt8\" (UniqueName: \"kubernetes.io/projected/5188b487-c967-42f8-a5c8-43f118b3e50d-kube-api-access-dsrt8\") on node \"ip-10-0-140-107.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:28:45.800002 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:28:45.799960 4939 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5188b487-c967-42f8-a5c8-43f118b3e50d-istio-token\") on node \"ip-10-0-140-107.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:28:45.800002 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:28:45.799976 4939 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5188b487-c967-42f8-a5c8-43f118b3e50d-workload-socket\") on node \"ip-10-0-140-107.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:28:45.800002 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:28:45.799992 4939 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5188b487-c967-42f8-a5c8-43f118b3e50d-workload-certs\") on node \"ip-10-0-140-107.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:28:45.800251 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:28:45.800007 4939 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5188b487-c967-42f8-a5c8-43f118b3e50d-istio-podinfo\") on node \"ip-10-0-140-107.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:28:45.800251 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:28:45.800022 4939 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5188b487-c967-42f8-a5c8-43f118b3e50d-istiod-ca-cert\") on node \"ip-10-0-140-107.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:28:45.800890 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:28:45.800848 4939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2c2192ff-91c4-4488-8229-629018fd881d-credential-socket" pod "2c2192ff-91c4-4488-8229-629018fd881d" (UID: "2c2192ff-91c4-4488-8229-629018fd881d"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 01:28:45.801170 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:28:45.801144 4939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8ea9e0d9-99b8-4bbb-b98d-b459f1560c89-workload-certs" pod "8ea9e0d9-99b8-4bbb-b98d-b459f1560c89" (UID: "8ea9e0d9-99b8-4bbb-b98d-b459f1560c89"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 01:28:45.801962 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:28:45.801931 4939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8ea9e0d9-99b8-4bbb-b98d-b459f1560c89-istiod-ca-cert" pod "8ea9e0d9-99b8-4bbb-b98d-b459f1560c89" (UID: "8ea9e0d9-99b8-4bbb-b98d-b459f1560c89"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 01:28:45.802178 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:28:45.802035 4939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0412ca91-563b-4ee7-8a02-4f4e768e51f7-credential-socket" pod "0412ca91-563b-4ee7-8a02-4f4e768e51f7" (UID: "0412ca91-563b-4ee7-8a02-4f4e768e51f7"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 01:28:45.802442 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:28:45.802427 4939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8ea9e0d9-99b8-4bbb-b98d-b459f1560c89-istio-data" pod "8ea9e0d9-99b8-4bbb-b98d-b459f1560c89" (UID: "8ea9e0d9-99b8-4bbb-b98d-b459f1560c89"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 01:28:45.803085 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:28:45.803062 4939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2c2192ff-91c4-4488-8229-629018fd881d-istio-envoy" pod "2c2192ff-91c4-4488-8229-629018fd881d" (UID: "2c2192ff-91c4-4488-8229-629018fd881d"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 01:28:45.806693 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:28:45.806670 4939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/2c2192ff-91c4-4488-8229-629018fd881d-istio-podinfo" pod "2c2192ff-91c4-4488-8229-629018fd881d" (UID: "2c2192ff-91c4-4488-8229-629018fd881d"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 01:28:45.808509 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:28:45.808469 4939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2c2192ff-91c4-4488-8229-629018fd881d-workload-socket" pod "2c2192ff-91c4-4488-8229-629018fd881d" (UID: "2c2192ff-91c4-4488-8229-629018fd881d"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 01:28:45.808695 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:28:45.808666 4939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2c2192ff-91c4-4488-8229-629018fd881d-workload-certs" pod "2c2192ff-91c4-4488-8229-629018fd881d" (UID: "2c2192ff-91c4-4488-8229-629018fd881d"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 01:28:45.808845 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:28:45.808821 4939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/0412ca91-563b-4ee7-8a02-4f4e768e51f7-istio-podinfo" pod "0412ca91-563b-4ee7-8a02-4f4e768e51f7" (UID: "0412ca91-563b-4ee7-8a02-4f4e768e51f7"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 01:28:45.809365 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:28:45.809342 4939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0412ca91-563b-4ee7-8a02-4f4e768e51f7-istiod-ca-cert" pod "0412ca91-563b-4ee7-8a02-4f4e768e51f7" (UID: "0412ca91-563b-4ee7-8a02-4f4e768e51f7"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 01:28:45.809800 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:28:45.809780 4939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0412ca91-563b-4ee7-8a02-4f4e768e51f7-istio-data" pod "0412ca91-563b-4ee7-8a02-4f4e768e51f7" (UID: "0412ca91-563b-4ee7-8a02-4f4e768e51f7"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 01:28:45.811201 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:28:45.811177 4939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0412ca91-563b-4ee7-8a02-4f4e768e51f7-kube-api-access-8lpdh" pod "0412ca91-563b-4ee7-8a02-4f4e768e51f7" (UID: "0412ca91-563b-4ee7-8a02-4f4e768e51f7"). InnerVolumeSpecName "kube-api-access-8lpdh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 01:28:45.811398 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:28:45.811372 4939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8ea9e0d9-99b8-4bbb-b98d-b459f1560c89-kube-api-access-zdrp9" pod "8ea9e0d9-99b8-4bbb-b98d-b459f1560c89" (UID: "8ea9e0d9-99b8-4bbb-b98d-b459f1560c89"). InnerVolumeSpecName "kube-api-access-zdrp9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 01:28:45.812317 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:28:45.812291 4939 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-5bfc51-6c9fd54845-wq8nz"] Nov 03 01:28:45.812896 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:28:45.812868 4939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0412ca91-563b-4ee7-8a02-4f4e768e51f7-istio-envoy" pod "0412ca91-563b-4ee7-8a02-4f4e768e51f7" (UID: "0412ca91-563b-4ee7-8a02-4f4e768e51f7"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 01:28:45.813200 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:28:45.813109 4939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0412ca91-563b-4ee7-8a02-4f4e768e51f7-workload-certs" pod "0412ca91-563b-4ee7-8a02-4f4e768e51f7" (UID: "0412ca91-563b-4ee7-8a02-4f4e768e51f7"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 01:28:45.814392 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:28:45.814217 4939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0412ca91-563b-4ee7-8a02-4f4e768e51f7-workload-socket" pod "0412ca91-563b-4ee7-8a02-4f4e768e51f7" (UID: "0412ca91-563b-4ee7-8a02-4f4e768e51f7"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 01:28:45.814990 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:28:45.814658 4939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2c2192ff-91c4-4488-8229-629018fd881d-istiod-ca-cert" pod "2c2192ff-91c4-4488-8229-629018fd881d" (UID: "2c2192ff-91c4-4488-8229-629018fd881d"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 01:28:45.814990 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:28:45.814731 4939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8ea9e0d9-99b8-4bbb-b98d-b459f1560c89-workload-socket" pod "8ea9e0d9-99b8-4bbb-b98d-b459f1560c89" (UID: "8ea9e0d9-99b8-4bbb-b98d-b459f1560c89"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 01:28:45.814990 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:28:45.814844 4939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2c2192ff-91c4-4488-8229-629018fd881d-istio-data" pod "2c2192ff-91c4-4488-8229-629018fd881d" (UID: "2c2192ff-91c4-4488-8229-629018fd881d"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 01:28:45.815353 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:28:45.815259 4939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8ea9e0d9-99b8-4bbb-b98d-b459f1560c89-credential-socket" pod "8ea9e0d9-99b8-4bbb-b98d-b459f1560c89" (UID: "8ea9e0d9-99b8-4bbb-b98d-b459f1560c89"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 01:28:45.819516 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:28:45.819137 4939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0412ca91-563b-4ee7-8a02-4f4e768e51f7-istio-token" pod "0412ca91-563b-4ee7-8a02-4f4e768e51f7" (UID: "0412ca91-563b-4ee7-8a02-4f4e768e51f7"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 01:28:45.819516 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:28:45.819280 4939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8ea9e0d9-99b8-4bbb-b98d-b459f1560c89-istio-envoy" pod "8ea9e0d9-99b8-4bbb-b98d-b459f1560c89" (UID: "8ea9e0d9-99b8-4bbb-b98d-b459f1560c89"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 01:28:45.819516 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:28:45.819386 4939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8ea9e0d9-99b8-4bbb-b98d-b459f1560c89-istio-token" pod "8ea9e0d9-99b8-4bbb-b98d-b459f1560c89" (UID: "8ea9e0d9-99b8-4bbb-b98d-b459f1560c89"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 01:28:45.819516 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:28:45.819469 4939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2c2192ff-91c4-4488-8229-629018fd881d-kube-api-access-jh5rp" pod "2c2192ff-91c4-4488-8229-629018fd881d" (UID: "2c2192ff-91c4-4488-8229-629018fd881d"). InnerVolumeSpecName "kube-api-access-jh5rp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 01:28:45.824067 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:28:45.816073 4939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2c2192ff-91c4-4488-8229-629018fd881d-istio-token" pod "2c2192ff-91c4-4488-8229-629018fd881d" (UID: "2c2192ff-91c4-4488-8229-629018fd881d"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 01:28:45.829052 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:28:45.827497 4939 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-5bfc51-6c9fd54845-wq8nz"] Nov 03 01:28:45.829052 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:28:45.827635 4939 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-5bfc51-6c9fd54845-wq8nz" Nov 03 01:28:45.842755 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:28:45.839627 4939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/8ea9e0d9-99b8-4bbb-b98d-b459f1560c89-istio-podinfo" pod "8ea9e0d9-99b8-4bbb-b98d-b459f1560c89" (UID: "8ea9e0d9-99b8-4bbb-b98d-b459f1560c89"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 01:28:45.850560 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:28:45.849589 4939 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-b-5bfc51-serving-cert\"" Nov 03 01:28:45.874639 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:28:45.874577 4939 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="dd50d5d4-bbf2-439b-a851-2c2ae58da605" path="/var/lib/kubelet/pods/dd50d5d4-bbf2-439b-a851-2c2ae58da605/volumes" Nov 03 01:28:45.877282 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:28:45.877260 4939 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-b6afd6-54f7b98c7c-lzq84"] Nov 03 01:28:45.892863 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:28:45.892794 4939 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-b6afd6-54f7b98c7c-lzq84"] Nov 03 01:28:45.893045 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:28:45.892946 4939 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-b6afd6-54f7b98c7c-lzq84" Nov 03 01:28:45.909428 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:28:45.909402 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:28:45.910860 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:28:45.910834 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7fc8db38-3791-40c0-bcfb-716313223a85-istiod-ca-cert\") pod \"e2e-inf-a-ee618c-openshift-default-5ff9c74dc9-c4jpz\" (UID: \"7fc8db38-3791-40c0-bcfb-716313223a85\") " pod="openshift-ingress/e2e-inf-a-ee618c-openshift-default-5ff9c74dc9-c4jpz" Nov 03 01:28:45.911069 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:28:45.911049 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7fc8db38-3791-40c0-bcfb-716313223a85-istio-podinfo\") pod \"e2e-inf-a-ee618c-openshift-default-5ff9c74dc9-c4jpz\" (UID: \"7fc8db38-3791-40c0-bcfb-716313223a85\") " pod="openshift-ingress/e2e-inf-a-ee618c-openshift-default-5ff9c74dc9-c4jpz" Nov 03 01:28:45.911197 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:28:45.911184 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6165abef-0df3-4007-ae59-5536a2a77e45-istio-podinfo\") pod \"e2e-inf-a-ee618c-openshift-default-5ff9c74dc9-z4qxq\" (UID: \"6165abef-0df3-4007-ae59-5536a2a77e45\") " pod="openshift-ingress/e2e-inf-a-ee618c-openshift-default-5ff9c74dc9-z4qxq" Nov 03 01:28:45.911299 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:28:45.911285 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7fc8db38-3791-40c0-bcfb-716313223a85-credential-socket\") pod \"e2e-inf-a-ee618c-openshift-default-5ff9c74dc9-c4jpz\" (UID: \"7fc8db38-3791-40c0-bcfb-716313223a85\") " pod="openshift-ingress/e2e-inf-a-ee618c-openshift-default-5ff9c74dc9-c4jpz" Nov 03 01:28:45.911441 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:28:45.911428 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6165abef-0df3-4007-ae59-5536a2a77e45-credential-socket\") pod \"e2e-inf-a-ee618c-openshift-default-5ff9c74dc9-z4qxq\" (UID: \"6165abef-0df3-4007-ae59-5536a2a77e45\") " pod="openshift-ingress/e2e-inf-a-ee618c-openshift-default-5ff9c74dc9-z4qxq" Nov 03 01:28:45.911533 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:28:45.911520 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6165abef-0df3-4007-ae59-5536a2a77e45-workload-certs\") pod \"e2e-inf-a-ee618c-openshift-default-5ff9c74dc9-z4qxq\" (UID: \"6165abef-0df3-4007-ae59-5536a2a77e45\") " pod="openshift-ingress/e2e-inf-a-ee618c-openshift-default-5ff9c74dc9-z4qxq" Nov 03 01:28:45.911619 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:28:45.911608 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5lqkc\" (UniqueName: \"kubernetes.io/projected/6165abef-0df3-4007-ae59-5536a2a77e45-kube-api-access-5lqkc\") pod \"e2e-inf-a-ee618c-openshift-default-5ff9c74dc9-z4qxq\" (UID: \"6165abef-0df3-4007-ae59-5536a2a77e45\") " pod="openshift-ingress/e2e-inf-a-ee618c-openshift-default-5ff9c74dc9-z4qxq" Nov 03 01:28:45.911716 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:28:45.911705 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7fc8db38-3791-40c0-bcfb-716313223a85-istio-data\") pod \"e2e-inf-a-ee618c-openshift-default-5ff9c74dc9-c4jpz\" (UID: \"7fc8db38-3791-40c0-bcfb-716313223a85\") " pod="openshift-ingress/e2e-inf-a-ee618c-openshift-default-5ff9c74dc9-c4jpz" Nov 03 01:28:45.911830 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:28:45.911817 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3ba1b5d8-378c-4014-ba58-0fdae6897601-maas-api-tls\") pod \"maas-api-e2e-inf-b-5bfc51-6c9fd54845-wq8nz\" (UID: \"3ba1b5d8-378c-4014-ba58-0fdae6897601\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-5bfc51-6c9fd54845-wq8nz" Nov 03 01:28:45.912046 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:28:45.912023 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6165abef-0df3-4007-ae59-5536a2a77e45-istio-data\") pod \"e2e-inf-a-ee618c-openshift-default-5ff9c74dc9-z4qxq\" (UID: \"6165abef-0df3-4007-ae59-5536a2a77e45\") " pod="openshift-ingress/e2e-inf-a-ee618c-openshift-default-5ff9c74dc9-z4qxq" Nov 03 01:28:45.912142 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:28:45.912096 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6165abef-0df3-4007-ae59-5536a2a77e45-istio-envoy\") pod \"e2e-inf-a-ee618c-openshift-default-5ff9c74dc9-z4qxq\" (UID: \"6165abef-0df3-4007-ae59-5536a2a77e45\") " pod="openshift-ingress/e2e-inf-a-ee618c-openshift-default-5ff9c74dc9-z4qxq" Nov 03 01:28:45.912209 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:28:45.912138 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7fc8db38-3791-40c0-bcfb-716313223a85-istio-token\") pod \"e2e-inf-a-ee618c-openshift-default-5ff9c74dc9-c4jpz\" (UID: \"7fc8db38-3791-40c0-bcfb-716313223a85\") " pod="openshift-ingress/e2e-inf-a-ee618c-openshift-default-5ff9c74dc9-c4jpz" Nov 03 01:28:45.912209 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:28:45.912176 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5knp8\" (UniqueName: \"kubernetes.io/projected/7fc8db38-3791-40c0-bcfb-716313223a85-kube-api-access-5knp8\") pod \"e2e-inf-a-ee618c-openshift-default-5ff9c74dc9-c4jpz\" (UID: \"7fc8db38-3791-40c0-bcfb-716313223a85\") " pod="openshift-ingress/e2e-inf-a-ee618c-openshift-default-5ff9c74dc9-c4jpz" Nov 03 01:28:45.912325 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:28:45.912211 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6165abef-0df3-4007-ae59-5536a2a77e45-istiod-ca-cert\") pod \"e2e-inf-a-ee618c-openshift-default-5ff9c74dc9-z4qxq\" (UID: \"6165abef-0df3-4007-ae59-5536a2a77e45\") " pod="openshift-ingress/e2e-inf-a-ee618c-openshift-default-5ff9c74dc9-z4qxq" Nov 03 01:28:45.912325 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:28:45.912258 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3ba1b5d8-378c-4014-ba58-0fdae6897601-metrics-certs\") pod \"maas-api-e2e-inf-b-5bfc51-6c9fd54845-wq8nz\" (UID: \"3ba1b5d8-378c-4014-ba58-0fdae6897601\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-5bfc51-6c9fd54845-wq8nz" Nov 03 01:28:45.912325 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:28:45.912297 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7fc8db38-3791-40c0-bcfb-716313223a85-workload-socket\") pod \"e2e-inf-a-ee618c-openshift-default-5ff9c74dc9-c4jpz\" (UID: \"7fc8db38-3791-40c0-bcfb-716313223a85\") " pod="openshift-ingress/e2e-inf-a-ee618c-openshift-default-5ff9c74dc9-c4jpz" Nov 03 01:28:45.912508 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:28:45.912357 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6165abef-0df3-4007-ae59-5536a2a77e45-istio-token\") pod \"e2e-inf-a-ee618c-openshift-default-5ff9c74dc9-z4qxq\" (UID: \"6165abef-0df3-4007-ae59-5536a2a77e45\") " pod="openshift-ingress/e2e-inf-a-ee618c-openshift-default-5ff9c74dc9-z4qxq" Nov 03 01:28:45.912508 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:28:45.912404 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7fc8db38-3791-40c0-bcfb-716313223a85-workload-certs\") pod \"e2e-inf-a-ee618c-openshift-default-5ff9c74dc9-c4jpz\" (UID: \"7fc8db38-3791-40c0-bcfb-716313223a85\") " pod="openshift-ingress/e2e-inf-a-ee618c-openshift-default-5ff9c74dc9-c4jpz" Nov 03 01:28:45.912508 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:28:45.912441 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6165abef-0df3-4007-ae59-5536a2a77e45-workload-socket\") pod \"e2e-inf-a-ee618c-openshift-default-5ff9c74dc9-z4qxq\" (UID: \"6165abef-0df3-4007-ae59-5536a2a77e45\") " pod="openshift-ingress/e2e-inf-a-ee618c-openshift-default-5ff9c74dc9-z4qxq" Nov 03 01:28:45.912508 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:28:45.912477 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3ba1b5d8-378c-4014-ba58-0fdae6897601-openshift-service-ca\") pod \"maas-api-e2e-inf-b-5bfc51-6c9fd54845-wq8nz\" (UID: \"3ba1b5d8-378c-4014-ba58-0fdae6897601\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-5bfc51-6c9fd54845-wq8nz" Nov 03 01:28:45.912731 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:28:45.912527 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z5d87\" (UniqueName: \"kubernetes.io/projected/3ba1b5d8-378c-4014-ba58-0fdae6897601-kube-api-access-z5d87\") pod \"maas-api-e2e-inf-b-5bfc51-6c9fd54845-wq8nz\" (UID: \"3ba1b5d8-378c-4014-ba58-0fdae6897601\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-5bfc51-6c9fd54845-wq8nz" Nov 03 01:28:45.912731 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:28:45.912562 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7fc8db38-3791-40c0-bcfb-716313223a85-istio-envoy\") pod \"e2e-inf-a-ee618c-openshift-default-5ff9c74dc9-c4jpz\" (UID: \"7fc8db38-3791-40c0-bcfb-716313223a85\") " pod="openshift-ingress/e2e-inf-a-ee618c-openshift-default-5ff9c74dc9-c4jpz" Nov 03 01:28:45.912731 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:28:45.912669 4939 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8ea9e0d9-99b8-4bbb-b98d-b459f1560c89-istio-envoy\") on node \"ip-10-0-140-107.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:28:45.912731 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:28:45.912687 4939 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8ea9e0d9-99b8-4bbb-b98d-b459f1560c89-istiod-ca-cert\") on node \"ip-10-0-140-107.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:28:45.912731 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:28:45.912703 4939 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8ea9e0d9-99b8-4bbb-b98d-b459f1560c89-workload-certs\") on node \"ip-10-0-140-107.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:28:45.912731 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:28:45.912718 4939 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8ea9e0d9-99b8-4bbb-b98d-b459f1560c89-istio-podinfo\") on node \"ip-10-0-140-107.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:28:45.913145 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:28:45.912732 4939 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0412ca91-563b-4ee7-8a02-4f4e768e51f7-istio-token\") on node \"ip-10-0-140-107.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:28:45.913145 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:28:45.912746 4939 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0412ca91-563b-4ee7-8a02-4f4e768e51f7-workload-socket\") on node \"ip-10-0-140-107.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:28:45.913145 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:28:45.912761 4939 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8lpdh\" (UniqueName: \"kubernetes.io/projected/0412ca91-563b-4ee7-8a02-4f4e768e51f7-kube-api-access-8lpdh\") on node \"ip-10-0-140-107.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:28:45.913145 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:28:45.912777 4939 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2c2192ff-91c4-4488-8229-629018fd881d-istio-data\") on node \"ip-10-0-140-107.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:28:45.913145 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:28:45.912792 4939 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8ea9e0d9-99b8-4bbb-b98d-b459f1560c89-credential-socket\") on node \"ip-10-0-140-107.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:28:45.913145 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:28:45.912806 4939 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2c2192ff-91c4-4488-8229-629018fd881d-istiod-ca-cert\") on node \"ip-10-0-140-107.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:28:45.913145 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:28:45.912822 4939 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2c2192ff-91c4-4488-8229-629018fd881d-credential-socket\") on node \"ip-10-0-140-107.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:28:45.913145 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:28:45.912836 4939 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2c2192ff-91c4-4488-8229-629018fd881d-istio-envoy\") on node \"ip-10-0-140-107.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:28:45.913145 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:28:45.912851 4939 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0412ca91-563b-4ee7-8a02-4f4e768e51f7-istio-envoy\") on node \"ip-10-0-140-107.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:28:45.913145 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:28:45.912866 4939 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0412ca91-563b-4ee7-8a02-4f4e768e51f7-workload-certs\") on node \"ip-10-0-140-107.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:28:45.913145 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:28:45.912882 4939 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2c2192ff-91c4-4488-8229-629018fd881d-istio-token\") on node \"ip-10-0-140-107.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:28:45.913145 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:28:45.912897 4939 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0412ca91-563b-4ee7-8a02-4f4e768e51f7-credential-socket\") on node \"ip-10-0-140-107.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:28:45.913145 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:28:45.912979 4939 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jh5rp\" (UniqueName: \"kubernetes.io/projected/2c2192ff-91c4-4488-8229-629018fd881d-kube-api-access-jh5rp\") on node \"ip-10-0-140-107.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:28:45.913145 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:28:45.912996 4939 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8ea9e0d9-99b8-4bbb-b98d-b459f1560c89-workload-socket\") on node \"ip-10-0-140-107.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:28:45.913145 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:28:45.913012 4939 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8ea9e0d9-99b8-4bbb-b98d-b459f1560c89-istio-token\") on node \"ip-10-0-140-107.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:28:45.913145 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:28:45.913027 4939 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0412ca91-563b-4ee7-8a02-4f4e768e51f7-istio-podinfo\") on node \"ip-10-0-140-107.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:28:45.913145 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:28:45.913042 4939 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8ea9e0d9-99b8-4bbb-b98d-b459f1560c89-istio-data\") on node \"ip-10-0-140-107.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:28:45.913145 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:28:45.913058 4939 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2c2192ff-91c4-4488-8229-629018fd881d-istio-podinfo\") on node \"ip-10-0-140-107.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:28:45.913145 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:28:45.913075 4939 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0412ca91-563b-4ee7-8a02-4f4e768e51f7-istiod-ca-cert\") on node \"ip-10-0-140-107.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:28:45.913145 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:28:45.913089 4939 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2c2192ff-91c4-4488-8229-629018fd881d-workload-certs\") on node \"ip-10-0-140-107.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:28:45.913145 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:28:45.913103 4939 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zdrp9\" (UniqueName: \"kubernetes.io/projected/8ea9e0d9-99b8-4bbb-b98d-b459f1560c89-kube-api-access-zdrp9\") on node \"ip-10-0-140-107.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:28:45.913145 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:28:45.913119 4939 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2c2192ff-91c4-4488-8229-629018fd881d-workload-socket\") on node \"ip-10-0-140-107.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:28:45.913145 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:28:45.913134 4939 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0412ca91-563b-4ee7-8a02-4f4e768e51f7-istio-data\") on node \"ip-10-0-140-107.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:28:45.914391 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:28:45.913518 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6165abef-0df3-4007-ae59-5536a2a77e45-credential-socket\") pod \"e2e-inf-a-ee618c-openshift-default-5ff9c74dc9-z4qxq\" (UID: \"6165abef-0df3-4007-ae59-5536a2a77e45\") " pod="openshift-ingress/e2e-inf-a-ee618c-openshift-default-5ff9c74dc9-z4qxq" Nov 03 01:28:45.914391 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:28:45.913859 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6165abef-0df3-4007-ae59-5536a2a77e45-workload-certs\") pod \"e2e-inf-a-ee618c-openshift-default-5ff9c74dc9-z4qxq\" (UID: \"6165abef-0df3-4007-ae59-5536a2a77e45\") " pod="openshift-ingress/e2e-inf-a-ee618c-openshift-default-5ff9c74dc9-z4qxq" Nov 03 01:28:45.914512 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:28:45.914408 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6165abef-0df3-4007-ae59-5536a2a77e45-istio-data\") pod \"e2e-inf-a-ee618c-openshift-default-5ff9c74dc9-z4qxq\" (UID: \"6165abef-0df3-4007-ae59-5536a2a77e45\") " pod="openshift-ingress/e2e-inf-a-ee618c-openshift-default-5ff9c74dc9-z4qxq" Nov 03 01:28:45.914703 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:28:45.914619 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6165abef-0df3-4007-ae59-5536a2a77e45-istio-podinfo\") pod \"e2e-inf-a-ee618c-openshift-default-5ff9c74dc9-z4qxq\" (UID: \"6165abef-0df3-4007-ae59-5536a2a77e45\") " pod="openshift-ingress/e2e-inf-a-ee618c-openshift-default-5ff9c74dc9-z4qxq" Nov 03 01:28:45.915276 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:28:45.915166 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6165abef-0df3-4007-ae59-5536a2a77e45-workload-socket\") pod \"e2e-inf-a-ee618c-openshift-default-5ff9c74dc9-z4qxq\" (UID: \"6165abef-0df3-4007-ae59-5536a2a77e45\") " pod="openshift-ingress/e2e-inf-a-ee618c-openshift-default-5ff9c74dc9-z4qxq" Nov 03 01:28:45.915500 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:28:45.915475 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6165abef-0df3-4007-ae59-5536a2a77e45-istiod-ca-cert\") pod \"e2e-inf-a-ee618c-openshift-default-5ff9c74dc9-z4qxq\" (UID: \"6165abef-0df3-4007-ae59-5536a2a77e45\") " pod="openshift-ingress/e2e-inf-a-ee618c-openshift-default-5ff9c74dc9-z4qxq" Nov 03 01:28:45.918307 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:28:45.918271 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6165abef-0df3-4007-ae59-5536a2a77e45-istio-envoy\") pod \"e2e-inf-a-ee618c-openshift-default-5ff9c74dc9-z4qxq\" (UID: \"6165abef-0df3-4007-ae59-5536a2a77e45\") " pod="openshift-ingress/e2e-inf-a-ee618c-openshift-default-5ff9c74dc9-z4qxq" Nov 03 01:28:45.926208 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:28:45.926186 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5lqkc\" (UniqueName: \"kubernetes.io/projected/6165abef-0df3-4007-ae59-5536a2a77e45-kube-api-access-5lqkc\") pod \"e2e-inf-a-ee618c-openshift-default-5ff9c74dc9-z4qxq\" (UID: \"6165abef-0df3-4007-ae59-5536a2a77e45\") " pod="openshift-ingress/e2e-inf-a-ee618c-openshift-default-5ff9c74dc9-z4qxq" Nov 03 01:28:45.938741 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:28:45.938712 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6165abef-0df3-4007-ae59-5536a2a77e45-istio-token\") pod \"e2e-inf-a-ee618c-openshift-default-5ff9c74dc9-z4qxq\" (UID: \"6165abef-0df3-4007-ae59-5536a2a77e45\") " pod="openshift-ingress/e2e-inf-a-ee618c-openshift-default-5ff9c74dc9-z4qxq" Nov 03 01:28:45.956933 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:28:45.956881 4939 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-a-b6afd6-serving-cert\"" Nov 03 01:28:46.029395 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:28:46.017327 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7fc8db38-3791-40c0-bcfb-716313223a85-workload-certs\") pod \"e2e-inf-a-ee618c-openshift-default-5ff9c74dc9-c4jpz\" (UID: \"7fc8db38-3791-40c0-bcfb-716313223a85\") " pod="openshift-ingress/e2e-inf-a-ee618c-openshift-default-5ff9c74dc9-c4jpz" Nov 03 01:28:46.029395 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:28:46.017407 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3ba1b5d8-378c-4014-ba58-0fdae6897601-openshift-service-ca\") pod \"maas-api-e2e-inf-b-5bfc51-6c9fd54845-wq8nz\" (UID: \"3ba1b5d8-378c-4014-ba58-0fdae6897601\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-5bfc51-6c9fd54845-wq8nz" Nov 03 01:28:46.029395 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:28:46.017472 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z5d87\" (UniqueName: \"kubernetes.io/projected/3ba1b5d8-378c-4014-ba58-0fdae6897601-kube-api-access-z5d87\") pod \"maas-api-e2e-inf-b-5bfc51-6c9fd54845-wq8nz\" (UID: \"3ba1b5d8-378c-4014-ba58-0fdae6897601\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-5bfc51-6c9fd54845-wq8nz" Nov 03 01:28:46.029395 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:28:46.017508 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7fc8db38-3791-40c0-bcfb-716313223a85-istio-envoy\") pod \"e2e-inf-a-ee618c-openshift-default-5ff9c74dc9-c4jpz\" (UID: \"7fc8db38-3791-40c0-bcfb-716313223a85\") " pod="openshift-ingress/e2e-inf-a-ee618c-openshift-default-5ff9c74dc9-c4jpz" Nov 03 01:28:46.029395 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:28:46.019183 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7fc8db38-3791-40c0-bcfb-716313223a85-istiod-ca-cert\") pod \"e2e-inf-a-ee618c-openshift-default-5ff9c74dc9-c4jpz\" (UID: \"7fc8db38-3791-40c0-bcfb-716313223a85\") " pod="openshift-ingress/e2e-inf-a-ee618c-openshift-default-5ff9c74dc9-c4jpz" Nov 03 01:28:46.029395 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:28:46.019311 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7fc8db38-3791-40c0-bcfb-716313223a85-istio-podinfo\") pod \"e2e-inf-a-ee618c-openshift-default-5ff9c74dc9-c4jpz\" (UID: \"7fc8db38-3791-40c0-bcfb-716313223a85\") " pod="openshift-ingress/e2e-inf-a-ee618c-openshift-default-5ff9c74dc9-c4jpz" Nov 03 01:28:46.029395 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:28:46.019370 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7fc8db38-3791-40c0-bcfb-716313223a85-credential-socket\") pod \"e2e-inf-a-ee618c-openshift-default-5ff9c74dc9-c4jpz\" (UID: \"7fc8db38-3791-40c0-bcfb-716313223a85\") " pod="openshift-ingress/e2e-inf-a-ee618c-openshift-default-5ff9c74dc9-c4jpz" Nov 03 01:28:46.029395 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:28:46.019463 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/32cc7c64-ec44-4a5d-815a-a378d840e3ff-openshift-service-ca\") pod \"maas-api-e2e-inf-a-b6afd6-54f7b98c7c-lzq84\" (UID: \"32cc7c64-ec44-4a5d-815a-a378d840e3ff\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-b6afd6-54f7b98c7c-lzq84" Nov 03 01:28:46.029395 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:28:46.019519 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/32cc7c64-ec44-4a5d-815a-a378d840e3ff-metrics-certs\") pod \"maas-api-e2e-inf-a-b6afd6-54f7b98c7c-lzq84\" (UID: \"32cc7c64-ec44-4a5d-815a-a378d840e3ff\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-b6afd6-54f7b98c7c-lzq84" Nov 03 01:28:46.029395 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:28:46.019592 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7fc8db38-3791-40c0-bcfb-716313223a85-istio-data\") pod \"e2e-inf-a-ee618c-openshift-default-5ff9c74dc9-c4jpz\" (UID: \"7fc8db38-3791-40c0-bcfb-716313223a85\") " pod="openshift-ingress/e2e-inf-a-ee618c-openshift-default-5ff9c74dc9-c4jpz" Nov 03 01:28:46.029395 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:28:46.019631 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3ba1b5d8-378c-4014-ba58-0fdae6897601-maas-api-tls\") pod \"maas-api-e2e-inf-b-5bfc51-6c9fd54845-wq8nz\" (UID: \"3ba1b5d8-378c-4014-ba58-0fdae6897601\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-5bfc51-6c9fd54845-wq8nz" Nov 03 01:28:46.029395 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:28:46.019705 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lq8k6\" (UniqueName: \"kubernetes.io/projected/32cc7c64-ec44-4a5d-815a-a378d840e3ff-kube-api-access-lq8k6\") pod \"maas-api-e2e-inf-a-b6afd6-54f7b98c7c-lzq84\" (UID: \"32cc7c64-ec44-4a5d-815a-a378d840e3ff\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-b6afd6-54f7b98c7c-lzq84" Nov 03 01:28:46.029395 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:28:46.019800 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/32cc7c64-ec44-4a5d-815a-a378d840e3ff-maas-api-tls\") pod \"maas-api-e2e-inf-a-b6afd6-54f7b98c7c-lzq84\" (UID: \"32cc7c64-ec44-4a5d-815a-a378d840e3ff\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-b6afd6-54f7b98c7c-lzq84" Nov 03 01:28:46.029395 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:28:46.020055 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7fc8db38-3791-40c0-bcfb-716313223a85-istio-token\") pod \"e2e-inf-a-ee618c-openshift-default-5ff9c74dc9-c4jpz\" (UID: \"7fc8db38-3791-40c0-bcfb-716313223a85\") " pod="openshift-ingress/e2e-inf-a-ee618c-openshift-default-5ff9c74dc9-c4jpz" Nov 03 01:28:46.029395 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:28:46.020107 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5knp8\" (UniqueName: \"kubernetes.io/projected/7fc8db38-3791-40c0-bcfb-716313223a85-kube-api-access-5knp8\") pod \"e2e-inf-a-ee618c-openshift-default-5ff9c74dc9-c4jpz\" (UID: \"7fc8db38-3791-40c0-bcfb-716313223a85\") " pod="openshift-ingress/e2e-inf-a-ee618c-openshift-default-5ff9c74dc9-c4jpz" Nov 03 01:28:46.029395 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:28:46.020173 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3ba1b5d8-378c-4014-ba58-0fdae6897601-metrics-certs\") pod \"maas-api-e2e-inf-b-5bfc51-6c9fd54845-wq8nz\" (UID: \"3ba1b5d8-378c-4014-ba58-0fdae6897601\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-5bfc51-6c9fd54845-wq8nz" Nov 03 01:28:46.029395 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:28:46.020211 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7fc8db38-3791-40c0-bcfb-716313223a85-workload-socket\") pod \"e2e-inf-a-ee618c-openshift-default-5ff9c74dc9-c4jpz\" (UID: \"7fc8db38-3791-40c0-bcfb-716313223a85\") " pod="openshift-ingress/e2e-inf-a-ee618c-openshift-default-5ff9c74dc9-c4jpz" Nov 03 01:28:46.029395 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:28:46.026848 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3ba1b5d8-378c-4014-ba58-0fdae6897601-openshift-service-ca\") pod \"maas-api-e2e-inf-b-5bfc51-6c9fd54845-wq8nz\" (UID: \"3ba1b5d8-378c-4014-ba58-0fdae6897601\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-5bfc51-6c9fd54845-wq8nz" Nov 03 01:28:46.031627 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:28:46.029690 4939 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-ee618c-openshift-default-5ff9c74dc9-z4qxq" Nov 03 01:28:46.031627 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:28:46.030546 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7fc8db38-3791-40c0-bcfb-716313223a85-workload-socket\") pod \"e2e-inf-a-ee618c-openshift-default-5ff9c74dc9-c4jpz\" (UID: \"7fc8db38-3791-40c0-bcfb-716313223a85\") " pod="openshift-ingress/e2e-inf-a-ee618c-openshift-default-5ff9c74dc9-c4jpz" Nov 03 01:28:46.031627 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:28:46.031621 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7fc8db38-3791-40c0-bcfb-716313223a85-credential-socket\") pod \"e2e-inf-a-ee618c-openshift-default-5ff9c74dc9-c4jpz\" (UID: \"7fc8db38-3791-40c0-bcfb-716313223a85\") " pod="openshift-ingress/e2e-inf-a-ee618c-openshift-default-5ff9c74dc9-c4jpz" Nov 03 01:28:46.032373 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:28:46.032352 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7fc8db38-3791-40c0-bcfb-716313223a85-istio-data\") pod \"e2e-inf-a-ee618c-openshift-default-5ff9c74dc9-c4jpz\" (UID: \"7fc8db38-3791-40c0-bcfb-716313223a85\") " pod="openshift-ingress/e2e-inf-a-ee618c-openshift-default-5ff9c74dc9-c4jpz" Nov 03 01:28:46.032727 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:28:46.032701 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7fc8db38-3791-40c0-bcfb-716313223a85-istio-envoy\") pod \"e2e-inf-a-ee618c-openshift-default-5ff9c74dc9-c4jpz\" (UID: \"7fc8db38-3791-40c0-bcfb-716313223a85\") " pod="openshift-ingress/e2e-inf-a-ee618c-openshift-default-5ff9c74dc9-c4jpz" Nov 03 01:28:46.036966 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:28:46.033478 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7fc8db38-3791-40c0-bcfb-716313223a85-workload-certs\") pod \"e2e-inf-a-ee618c-openshift-default-5ff9c74dc9-c4jpz\" (UID: \"7fc8db38-3791-40c0-bcfb-716313223a85\") " pod="openshift-ingress/e2e-inf-a-ee618c-openshift-default-5ff9c74dc9-c4jpz" Nov 03 01:28:46.036966 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:28:46.033898 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7fc8db38-3791-40c0-bcfb-716313223a85-istiod-ca-cert\") pod \"e2e-inf-a-ee618c-openshift-default-5ff9c74dc9-c4jpz\" (UID: \"7fc8db38-3791-40c0-bcfb-716313223a85\") " pod="openshift-ingress/e2e-inf-a-ee618c-openshift-default-5ff9c74dc9-c4jpz" Nov 03 01:28:46.036966 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:28:46.034129 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3ba1b5d8-378c-4014-ba58-0fdae6897601-maas-api-tls\") pod \"maas-api-e2e-inf-b-5bfc51-6c9fd54845-wq8nz\" (UID: \"3ba1b5d8-378c-4014-ba58-0fdae6897601\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-5bfc51-6c9fd54845-wq8nz" Nov 03 01:28:46.036966 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:28:46.036736 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3ba1b5d8-378c-4014-ba58-0fdae6897601-metrics-certs\") pod \"maas-api-e2e-inf-b-5bfc51-6c9fd54845-wq8nz\" (UID: \"3ba1b5d8-378c-4014-ba58-0fdae6897601\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-5bfc51-6c9fd54845-wq8nz" Nov 03 01:28:46.038439 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:28:46.038210 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-z5d87\" (UniqueName: \"kubernetes.io/projected/3ba1b5d8-378c-4014-ba58-0fdae6897601-kube-api-access-z5d87\") pod \"maas-api-e2e-inf-b-5bfc51-6c9fd54845-wq8nz\" (UID: \"3ba1b5d8-378c-4014-ba58-0fdae6897601\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-5bfc51-6c9fd54845-wq8nz" Nov 03 01:28:46.041634 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:28:46.041611 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5knp8\" (UniqueName: \"kubernetes.io/projected/7fc8db38-3791-40c0-bcfb-716313223a85-kube-api-access-5knp8\") pod \"e2e-inf-a-ee618c-openshift-default-5ff9c74dc9-c4jpz\" (UID: \"7fc8db38-3791-40c0-bcfb-716313223a85\") " pod="openshift-ingress/e2e-inf-a-ee618c-openshift-default-5ff9c74dc9-c4jpz" Nov 03 01:28:46.041740 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:28:46.041708 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7fc8db38-3791-40c0-bcfb-716313223a85-istio-token\") pod \"e2e-inf-a-ee618c-openshift-default-5ff9c74dc9-c4jpz\" (UID: \"7fc8db38-3791-40c0-bcfb-716313223a85\") " pod="openshift-ingress/e2e-inf-a-ee618c-openshift-default-5ff9c74dc9-c4jpz" Nov 03 01:28:46.043873 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:28:46.043850 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7fc8db38-3791-40c0-bcfb-716313223a85-istio-podinfo\") pod \"e2e-inf-a-ee618c-openshift-default-5ff9c74dc9-c4jpz\" (UID: \"7fc8db38-3791-40c0-bcfb-716313223a85\") " pod="openshift-ingress/e2e-inf-a-ee618c-openshift-default-5ff9c74dc9-c4jpz" Nov 03 01:28:46.082497 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:28:46.082464 4939 generic.go:356] "Generic (PLEG): container finished" podID="2c2192ff-91c4-4488-8229-629018fd881d" containerID="f82319615907c985e605951dc5dc72d7600b1fbb6431ed69d94645c064d8dc84" exitCode=0 Nov 03 01:28:46.082948 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:28:46.082534 4939 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-ce04af-openshift-default-7dccd45f47-vxtn5" Nov 03 01:28:46.082948 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:28:46.082535 4939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-ce04af-openshift-default-7dccd45f47-vxtn5" event={"ID":"2c2192ff-91c4-4488-8229-629018fd881d","Type":"ContainerDied","Data":"f82319615907c985e605951dc5dc72d7600b1fbb6431ed69d94645c064d8dc84"} Nov 03 01:28:46.082948 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:28:46.082663 4939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-ce04af-openshift-default-7dccd45f47-vxtn5" event={"ID":"2c2192ff-91c4-4488-8229-629018fd881d","Type":"ContainerDied","Data":"c18027769179fc51c278647d7e4bbd823fb3c51faf9ab929569b3eff5f5d49a7"} Nov 03 01:28:46.082948 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:28:46.082691 4939 scope.go:122] "RemoveContainer" containerID="f82319615907c985e605951dc5dc72d7600b1fbb6431ed69d94645c064d8dc84" Nov 03 01:28:46.088566 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:28:46.088539 4939 generic.go:356] "Generic (PLEG): container finished" podID="8ea9e0d9-99b8-4bbb-b98d-b459f1560c89" containerID="a24f21dac1517c1ad57ee59169b7a7401d3a8349983a274d17f83f89201f147c" exitCode=0 Nov 03 01:28:46.088709 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:28:46.088619 4939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-ce04af-openshift-default-7dccd45f47-75xwt" event={"ID":"8ea9e0d9-99b8-4bbb-b98d-b459f1560c89","Type":"ContainerDied","Data":"a24f21dac1517c1ad57ee59169b7a7401d3a8349983a274d17f83f89201f147c"} Nov 03 01:28:46.088709 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:28:46.088666 4939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-ce04af-openshift-default-7dccd45f47-75xwt" event={"ID":"8ea9e0d9-99b8-4bbb-b98d-b459f1560c89","Type":"ContainerDied","Data":"59283d2f8f702348becd13724b6a7f6ebc5a39c3491493e19cadb9b5981eea90"} Nov 03 01:28:46.088881 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:28:46.088861 4939 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-ce04af-openshift-default-7dccd45f47-75xwt" Nov 03 01:28:46.091688 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:28:46.091667 4939 generic.go:356] "Generic (PLEG): container finished" podID="0412ca91-563b-4ee7-8a02-4f4e768e51f7" containerID="a1f057279f28d75acdb60e495251a558ad6fdfa1ff112c36890e28722ac1cf25" exitCode=0 Nov 03 01:28:46.092008 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:28:46.091987 4939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-ce04af-openshift-default-7dccd45f47-wc5pz" event={"ID":"0412ca91-563b-4ee7-8a02-4f4e768e51f7","Type":"ContainerDied","Data":"a1f057279f28d75acdb60e495251a558ad6fdfa1ff112c36890e28722ac1cf25"} Nov 03 01:28:46.092121 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:28:46.092028 4939 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-ce04af-openshift-default-7dccd45f47-wc5pz" Nov 03 01:28:46.092319 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:28:46.092033 4939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-ce04af-openshift-default-7dccd45f47-wc5pz" event={"ID":"0412ca91-563b-4ee7-8a02-4f4e768e51f7","Type":"ContainerDied","Data":"0656c94b54b221a37dc490a8655c19830f40173e1fc028892fdfbc3a2b6576c8"} Nov 03 01:28:46.098085 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:28:46.098063 4939 generic.go:356] "Generic (PLEG): container finished" podID="5188b487-c967-42f8-a5c8-43f118b3e50d" containerID="9c4f71736d18633fc600dc773a00b3ec7b3814be88b7cebaf2e86d5fac01463c" exitCode=0 Nov 03 01:28:46.098262 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:28:46.098247 4939 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-ce04af-openshift-default-7dccd45f47-n5ndk" Nov 03 01:28:46.098417 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:28:46.098396 4939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-ce04af-openshift-default-7dccd45f47-n5ndk" event={"ID":"5188b487-c967-42f8-a5c8-43f118b3e50d","Type":"ContainerDied","Data":"9c4f71736d18633fc600dc773a00b3ec7b3814be88b7cebaf2e86d5fac01463c"} Nov 03 01:28:46.098551 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:28:46.098534 4939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-ce04af-openshift-default-7dccd45f47-n5ndk" event={"ID":"5188b487-c967-42f8-a5c8-43f118b3e50d","Type":"ContainerDied","Data":"3b6f2d9b2771e6b7fa84fa8faf5208a4a19af848e06b0db54a7f420f5e0491bf"} Nov 03 01:28:46.107431 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:28:46.107415 4939 scope.go:122] "RemoveContainer" containerID="f82319615907c985e605951dc5dc72d7600b1fbb6431ed69d94645c064d8dc84" Nov 03 01:28:46.107828 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:28:46.107817 4939 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-ee618c-openshift-default-5ff9c74dc9-c4jpz" Nov 03 01:28:46.109145 ip-10-0-140-107 kubenswrapper[4939]: E1103 01:28:46.109067 4939 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f82319615907c985e605951dc5dc72d7600b1fbb6431ed69d94645c064d8dc84\": container with ID starting with f82319615907c985e605951dc5dc72d7600b1fbb6431ed69d94645c064d8dc84 not found: ID does not exist" containerID="f82319615907c985e605951dc5dc72d7600b1fbb6431ed69d94645c064d8dc84" Nov 03 01:28:46.109145 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:28:46.109106 4939 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f82319615907c985e605951dc5dc72d7600b1fbb6431ed69d94645c064d8dc84"} err="failed to get container status \"f82319615907c985e605951dc5dc72d7600b1fbb6431ed69d94645c064d8dc84\": rpc error: code = NotFound desc = could not find container \"f82319615907c985e605951dc5dc72d7600b1fbb6431ed69d94645c064d8dc84\": container with ID starting with f82319615907c985e605951dc5dc72d7600b1fbb6431ed69d94645c064d8dc84 not found: ID does not exist" Nov 03 01:28:46.109145 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:28:46.109130 4939 scope.go:122] "RemoveContainer" containerID="a24f21dac1517c1ad57ee59169b7a7401d3a8349983a274d17f83f89201f147c" Nov 03 01:28:46.117355 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:28:46.117335 4939 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-ce04af-openshift-default-7dccd45f47-75xwt"] Nov 03 01:28:46.124215 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:28:46.123376 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/32cc7c64-ec44-4a5d-815a-a378d840e3ff-openshift-service-ca\") pod \"maas-api-e2e-inf-a-b6afd6-54f7b98c7c-lzq84\" (UID: \"32cc7c64-ec44-4a5d-815a-a378d840e3ff\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-b6afd6-54f7b98c7c-lzq84" Nov 03 01:28:46.124215 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:28:46.123434 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/32cc7c64-ec44-4a5d-815a-a378d840e3ff-metrics-certs\") pod \"maas-api-e2e-inf-a-b6afd6-54f7b98c7c-lzq84\" (UID: \"32cc7c64-ec44-4a5d-815a-a378d840e3ff\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-b6afd6-54f7b98c7c-lzq84" Nov 03 01:28:46.124215 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:28:46.123521 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lq8k6\" (UniqueName: \"kubernetes.io/projected/32cc7c64-ec44-4a5d-815a-a378d840e3ff-kube-api-access-lq8k6\") pod \"maas-api-e2e-inf-a-b6afd6-54f7b98c7c-lzq84\" (UID: \"32cc7c64-ec44-4a5d-815a-a378d840e3ff\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-b6afd6-54f7b98c7c-lzq84" Nov 03 01:28:46.124215 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:28:46.123588 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/32cc7c64-ec44-4a5d-815a-a378d840e3ff-maas-api-tls\") pod \"maas-api-e2e-inf-a-b6afd6-54f7b98c7c-lzq84\" (UID: \"32cc7c64-ec44-4a5d-815a-a378d840e3ff\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-b6afd6-54f7b98c7c-lzq84" Nov 03 01:28:46.138148 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:28:46.125552 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/32cc7c64-ec44-4a5d-815a-a378d840e3ff-openshift-service-ca\") pod \"maas-api-e2e-inf-a-b6afd6-54f7b98c7c-lzq84\" (UID: \"32cc7c64-ec44-4a5d-815a-a378d840e3ff\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-b6afd6-54f7b98c7c-lzq84" Nov 03 01:28:46.138148 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:28:46.136547 4939 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-b-ce04af-openshift-default-7dccd45f47-75xwt"] Nov 03 01:28:46.138148 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:28:46.136597 4939 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-ce04af-openshift-default-7dccd45f47-vxtn5"] Nov 03 01:28:46.153537 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:28:46.138554 4939 scope.go:122] "RemoveContainer" containerID="a24f21dac1517c1ad57ee59169b7a7401d3a8349983a274d17f83f89201f147c" Nov 03 01:28:46.153537 ip-10-0-140-107 kubenswrapper[4939]: E1103 01:28:46.139351 4939 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a24f21dac1517c1ad57ee59169b7a7401d3a8349983a274d17f83f89201f147c\": container with ID starting with a24f21dac1517c1ad57ee59169b7a7401d3a8349983a274d17f83f89201f147c not found: ID does not exist" containerID="a24f21dac1517c1ad57ee59169b7a7401d3a8349983a274d17f83f89201f147c" Nov 03 01:28:46.153537 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:28:46.139391 4939 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a24f21dac1517c1ad57ee59169b7a7401d3a8349983a274d17f83f89201f147c"} err="failed to get container status \"a24f21dac1517c1ad57ee59169b7a7401d3a8349983a274d17f83f89201f147c\": rpc error: code = NotFound desc = could not find container \"a24f21dac1517c1ad57ee59169b7a7401d3a8349983a274d17f83f89201f147c\": container with ID starting with a24f21dac1517c1ad57ee59169b7a7401d3a8349983a274d17f83f89201f147c not found: ID does not exist" Nov 03 01:28:46.153537 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:28:46.139417 4939 scope.go:122] "RemoveContainer" containerID="a1f057279f28d75acdb60e495251a558ad6fdfa1ff112c36890e28722ac1cf25" Nov 03 01:28:46.153537 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:28:46.147960 4939 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-b-ce04af-openshift-default-7dccd45f47-vxtn5"] Nov 03 01:28:46.153537 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:28:46.150556 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/32cc7c64-ec44-4a5d-815a-a378d840e3ff-maas-api-tls\") pod \"maas-api-e2e-inf-a-b6afd6-54f7b98c7c-lzq84\" (UID: \"32cc7c64-ec44-4a5d-815a-a378d840e3ff\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-b6afd6-54f7b98c7c-lzq84" Nov 03 01:28:46.154001 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:28:46.153883 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/32cc7c64-ec44-4a5d-815a-a378d840e3ff-metrics-certs\") pod \"maas-api-e2e-inf-a-b6afd6-54f7b98c7c-lzq84\" (UID: \"32cc7c64-ec44-4a5d-815a-a378d840e3ff\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-b6afd6-54f7b98c7c-lzq84" Nov 03 01:28:46.158547 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:28:46.158498 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lq8k6\" (UniqueName: \"kubernetes.io/projected/32cc7c64-ec44-4a5d-815a-a378d840e3ff-kube-api-access-lq8k6\") pod \"maas-api-e2e-inf-a-b6afd6-54f7b98c7c-lzq84\" (UID: \"32cc7c64-ec44-4a5d-815a-a378d840e3ff\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-b6afd6-54f7b98c7c-lzq84" Nov 03 01:28:46.163145 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:28:46.162883 4939 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-5bfc51-6c9fd54845-wq8nz" Nov 03 01:28:46.175111 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:28:46.174008 4939 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-ce04af-openshift-default-7dccd45f47-n5ndk"] Nov 03 01:28:46.186542 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:28:46.186479 4939 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-b-ce04af-openshift-default-7dccd45f47-n5ndk"] Nov 03 01:28:46.191155 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:28:46.188125 4939 scope.go:122] "RemoveContainer" containerID="a1f057279f28d75acdb60e495251a558ad6fdfa1ff112c36890e28722ac1cf25" Nov 03 01:28:46.192388 ip-10-0-140-107 kubenswrapper[4939]: E1103 01:28:46.192360 4939 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a1f057279f28d75acdb60e495251a558ad6fdfa1ff112c36890e28722ac1cf25\": container with ID starting with a1f057279f28d75acdb60e495251a558ad6fdfa1ff112c36890e28722ac1cf25 not found: ID does not exist" containerID="a1f057279f28d75acdb60e495251a558ad6fdfa1ff112c36890e28722ac1cf25" Nov 03 01:28:46.192498 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:28:46.192402 4939 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a1f057279f28d75acdb60e495251a558ad6fdfa1ff112c36890e28722ac1cf25"} err="failed to get container status \"a1f057279f28d75acdb60e495251a558ad6fdfa1ff112c36890e28722ac1cf25\": rpc error: code = NotFound desc = could not find container \"a1f057279f28d75acdb60e495251a558ad6fdfa1ff112c36890e28722ac1cf25\": container with ID starting with a1f057279f28d75acdb60e495251a558ad6fdfa1ff112c36890e28722ac1cf25 not found: ID does not exist" Nov 03 01:28:46.192498 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:28:46.192426 4939 scope.go:122] "RemoveContainer" containerID="9c4f71736d18633fc600dc773a00b3ec7b3814be88b7cebaf2e86d5fac01463c" Nov 03 01:28:46.201184 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:28:46.201158 4939 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-ce04af-openshift-default-7dccd45f47-wc5pz"] Nov 03 01:28:46.215205 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:28:46.215181 4939 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-b-ce04af-openshift-default-7dccd45f47-wc5pz"] Nov 03 01:28:46.228076 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:28:46.227165 4939 scope.go:122] "RemoveContainer" containerID="9c4f71736d18633fc600dc773a00b3ec7b3814be88b7cebaf2e86d5fac01463c" Nov 03 01:28:46.228997 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:28:46.228549 4939 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-b6afd6-54f7b98c7c-lzq84" Nov 03 01:28:46.229104 ip-10-0-140-107 kubenswrapper[4939]: E1103 01:28:46.229034 4939 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9c4f71736d18633fc600dc773a00b3ec7b3814be88b7cebaf2e86d5fac01463c\": container with ID starting with 9c4f71736d18633fc600dc773a00b3ec7b3814be88b7cebaf2e86d5fac01463c not found: ID does not exist" containerID="9c4f71736d18633fc600dc773a00b3ec7b3814be88b7cebaf2e86d5fac01463c" Nov 03 01:28:46.229171 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:28:46.229104 4939 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9c4f71736d18633fc600dc773a00b3ec7b3814be88b7cebaf2e86d5fac01463c"} err="failed to get container status \"9c4f71736d18633fc600dc773a00b3ec7b3814be88b7cebaf2e86d5fac01463c\": rpc error: code = NotFound desc = could not find container \"9c4f71736d18633fc600dc773a00b3ec7b3814be88b7cebaf2e86d5fac01463c\": container with ID starting with 9c4f71736d18633fc600dc773a00b3ec7b3814be88b7cebaf2e86d5fac01463c not found: ID does not exist" Nov 03 01:28:46.336603 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:28:46.336571 4939 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-ee618c-openshift-default-5ff9c74dc9-z4qxq"] Nov 03 01:28:46.341521 ip-10-0-140-107 kubenswrapper[4939]: W1103 01:28:46.339642 4939 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6165abef_0df3_4007_ae59_5536a2a77e45.slice/crio-d5f702dc7a363dc956348f001c4b430d36962bb0062bd680533bce95e2bcb825 WatchSource:0}: Error finding container d5f702dc7a363dc956348f001c4b430d36962bb0062bd680533bce95e2bcb825: Status 404 returned error can't find the container with id d5f702dc7a363dc956348f001c4b430d36962bb0062bd680533bce95e2bcb825 Nov 03 01:28:46.344650 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:28:46.344601 4939 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 01:28:46.344759 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:28:46.344703 4939 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 01:28:46.344819 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:28:46.344757 4939 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 01:28:46.444766 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:28:46.444734 4939 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-ee618c-openshift-default-5ff9c74dc9-c4jpz"] Nov 03 01:28:46.450125 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:28:46.449608 4939 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 01:28:46.452136 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:28:46.452110 4939 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 01:28:46.452327 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:28:46.452304 4939 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 01:28:46.535385 ip-10-0-140-107 kubenswrapper[4939]: W1103 01:28:46.535343 4939 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3ba1b5d8_378c_4014_ba58_0fdae6897601.slice/crio-7bb5f46c9d7c2654db2a3ee9597511632b96b46234faaf4df4682faa345f6d20 WatchSource:0}: Error finding container 7bb5f46c9d7c2654db2a3ee9597511632b96b46234faaf4df4682faa345f6d20: Status 404 returned error can't find the container with id 7bb5f46c9d7c2654db2a3ee9597511632b96b46234faaf4df4682faa345f6d20 Nov 03 01:28:46.543521 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:28:46.543486 4939 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-5bfc51-6c9fd54845-wq8nz"] Nov 03 01:28:46.556957 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:28:46.556868 4939 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-b6afd6-54f7b98c7c-lzq84"] Nov 03 01:28:46.756564 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:28:46.756467 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:28:47.123924 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:28:47.123871 4939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-5bfc51-6c9fd54845-wq8nz" event={"ID":"3ba1b5d8-378c-4014-ba58-0fdae6897601","Type":"ContainerStarted","Data":"923c231e27333ad1540fc3d3655559b6740928d450b5a843d4b26580a5885f97"} Nov 03 01:28:47.130876 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:28:47.124030 4939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-5bfc51-6c9fd54845-wq8nz" event={"ID":"3ba1b5d8-378c-4014-ba58-0fdae6897601","Type":"ContainerStarted","Data":"7bb5f46c9d7c2654db2a3ee9597511632b96b46234faaf4df4682faa345f6d20"} Nov 03 01:28:47.130876 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:28:47.124087 4939 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-5bfc51-6c9fd54845-wq8nz" Nov 03 01:28:47.130876 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:28:47.126464 4939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-ee618c-openshift-default-5ff9c74dc9-z4qxq" event={"ID":"6165abef-0df3-4007-ae59-5536a2a77e45","Type":"ContainerStarted","Data":"710675672c4704b956985b778c9969f2de90e04a802951d0744ecd2defb6b0d9"} Nov 03 01:28:47.130876 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:28:47.126517 4939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-ee618c-openshift-default-5ff9c74dc9-z4qxq" event={"ID":"6165abef-0df3-4007-ae59-5536a2a77e45","Type":"ContainerStarted","Data":"d5f702dc7a363dc956348f001c4b430d36962bb0062bd680533bce95e2bcb825"} Nov 03 01:28:47.131645 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:28:47.131614 4939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-b6afd6-54f7b98c7c-lzq84" event={"ID":"32cc7c64-ec44-4a5d-815a-a378d840e3ff","Type":"ContainerStarted","Data":"a8cddbd123dce0327f642516c05761fc4374b6e6e0aceb6ae243fc8335588ae4"} Nov 03 01:28:47.139484 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:28:47.139456 4939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-ee618c-openshift-default-5ff9c74dc9-c4jpz" event={"ID":"7fc8db38-3791-40c0-bcfb-716313223a85","Type":"ContainerStarted","Data":"02aaf5672c7c46eedfc7b87e00d31cdb560aed8c52b7cb1b556bffa5e037bea1"} Nov 03 01:28:47.139612 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:28:47.139500 4939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-ee618c-openshift-default-5ff9c74dc9-c4jpz" event={"ID":"7fc8db38-3791-40c0-bcfb-716313223a85","Type":"ContainerStarted","Data":"aa3f47b0fd2754dcea2c7d2cec69ac0a43bbdd6e1fc782e84fb7e2cf4daebbd3"} Nov 03 01:28:47.144854 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:28:47.144813 4939 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-5bfc51-6c9fd54845-wq8nz" podStartSLOduration=76.749735866 podStartE2EDuration="1m17.144799184s" totalImagesPullingTime="395.063318ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 01:27:30 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 01:28:47.142024866 +0000 UTC m=+2875.406359811" watchObservedRunningTime="2026-11-03 01:28:47.144799184 +0000 UTC m=+2875.409134129" Nov 03 01:28:47.170518 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:28:47.170442 4939 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-a-ee618c-openshift-default-5ff9c74dc9-c4jpz" podStartSLOduration=88.170423559 podStartE2EDuration="1m28.170423559s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 01:27:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 01:28:47.164373121 +0000 UTC m=+2875.428708153" watchObservedRunningTime="2026-11-03 01:28:47.170423559 +0000 UTC m=+2875.434758504" Nov 03 01:28:47.190496 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:28:47.190436 4939 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-a-ee618c-openshift-default-5ff9c74dc9-z4qxq" podStartSLOduration=88.190417964 podStartE2EDuration="1m28.190417964s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 01:27:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 01:28:47.186093439 +0000 UTC m=+2875.450428385" watchObservedRunningTime="2026-11-03 01:28:47.190417964 +0000 UTC m=+2875.454752909" Nov 03 01:28:47.740220 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:28:47.740186 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:28:47.935792 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:28:47.933225 4939 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0412ca91-563b-4ee7-8a02-4f4e768e51f7" path="/var/lib/kubelet/pods/0412ca91-563b-4ee7-8a02-4f4e768e51f7/volumes" Nov 03 01:28:47.935792 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:28:47.933952 4939 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2c2192ff-91c4-4488-8229-629018fd881d" path="/var/lib/kubelet/pods/2c2192ff-91c4-4488-8229-629018fd881d/volumes" Nov 03 01:28:47.935792 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:28:47.934921 4939 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5188b487-c967-42f8-a5c8-43f118b3e50d" path="/var/lib/kubelet/pods/5188b487-c967-42f8-a5c8-43f118b3e50d/volumes" Nov 03 01:28:47.936567 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:28:47.936366 4939 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8ea9e0d9-99b8-4bbb-b98d-b459f1560c89" path="/var/lib/kubelet/pods/8ea9e0d9-99b8-4bbb-b98d-b459f1560c89/volumes" Nov 03 01:28:48.034171 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:28:48.034086 4939 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-ee618c-openshift-default-5ff9c74dc9-z4qxq" Nov 03 01:28:48.035658 ip-10-0-140-107 kubenswrapper[4939]: E1103 01:28:48.035625 4939 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-sthjh" containerName="extractor" Nov 03 01:28:48.035828 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:28:48.035671 4939 patch_prober.go:29] interesting pod/e2e-inf-a-ee618c-openshift-default-5ff9c74dc9-z4qxq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.53:15021/healthz/ready\": dial tcp 10.132.1.53:15021: connect: connection refused" start-of-body= Nov 03 01:28:48.035960 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:28:48.035861 4939 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-ee618c-openshift-default-5ff9c74dc9-z4qxq" podUID="6165abef-0df3-4007-ae59-5536a2a77e45" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.53:15021/healthz/ready\": dial tcp 10.132.1.53:15021: connect: connection refused" Nov 03 01:28:48.111683 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:28:48.111646 4939 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-ee618c-openshift-default-5ff9c74dc9-c4jpz" Nov 03 01:28:48.111854 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:28:48.111776 4939 patch_prober.go:29] interesting pod/e2e-inf-a-ee618c-openshift-default-5ff9c74dc9-c4jpz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.54:15021/healthz/ready\": dial tcp 10.132.1.54:15021: connect: connection refused" start-of-body= Nov 03 01:28:48.111854 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:28:48.111810 4939 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-ee618c-openshift-default-5ff9c74dc9-c4jpz" podUID="7fc8db38-3791-40c0-bcfb-716313223a85" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.54:15021/healthz/ready\": dial tcp 10.132.1.54:15021: connect: connection refused" Nov 03 01:28:48.201604 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:28:48.201570 4939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-b6afd6-54f7b98c7c-lzq84" event={"ID":"32cc7c64-ec44-4a5d-815a-a378d840e3ff","Type":"ContainerStarted","Data":"a11121971cff6d386abd3de73b678887187a59487367bbf28c9802796918e69d"} Nov 03 01:28:48.202298 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:28:48.202267 4939 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-b6afd6-54f7b98c7c-lzq84" Nov 03 01:28:48.220615 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:28:48.220565 4939 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-b6afd6-54f7b98c7c-lzq84" podStartSLOduration=131.739145429 podStartE2EDuration="2m12.220549806s" totalImagesPullingTime="481.404377ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 01:26:36 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 01:28:48.218222743 +0000 UTC m=+2876.482557685" watchObservedRunningTime="2026-11-03 01:28:48.220549806 +0000 UTC m=+2876.484884751" Nov 03 01:28:48.618679 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:28:48.618643 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:28:49.033694 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:28:49.033612 4939 patch_prober.go:29] interesting pod/e2e-inf-a-ee618c-openshift-default-5ff9c74dc9-z4qxq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.53:15021/healthz/ready\": dial tcp 10.132.1.53:15021: connect: connection refused" start-of-body= Nov 03 01:28:49.033694 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:28:49.033664 4939 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-ee618c-openshift-default-5ff9c74dc9-z4qxq" podUID="6165abef-0df3-4007-ae59-5536a2a77e45" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.53:15021/healthz/ready\": dial tcp 10.132.1.53:15021: connect: connection refused" Nov 03 01:28:49.110328 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:28:49.110291 4939 patch_prober.go:29] interesting pod/e2e-inf-a-ee618c-openshift-default-5ff9c74dc9-c4jpz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.54:15021/healthz/ready\": dial tcp 10.132.1.54:15021: connect: connection refused" start-of-body= Nov 03 01:28:49.110478 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:28:49.110358 4939 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-ee618c-openshift-default-5ff9c74dc9-c4jpz" podUID="7fc8db38-3791-40c0-bcfb-716313223a85" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.54:15021/healthz/ready\": dial tcp 10.132.1.54:15021: connect: connection refused" Nov 03 01:28:49.397930 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:28:49.397856 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:28:50.033976 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:28:50.033944 4939 patch_prober.go:29] interesting pod/e2e-inf-a-ee618c-openshift-default-5ff9c74dc9-z4qxq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.53:15021/healthz/ready\": dial tcp 10.132.1.53:15021: connect: connection refused" start-of-body= Nov 03 01:28:50.034138 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:28:50.033994 4939 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-ee618c-openshift-default-5ff9c74dc9-z4qxq" podUID="6165abef-0df3-4007-ae59-5536a2a77e45" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.53:15021/healthz/ready\": dial tcp 10.132.1.53:15021: connect: connection refused" Nov 03 01:28:50.109953 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:28:50.109920 4939 patch_prober.go:29] interesting pod/e2e-inf-a-ee618c-openshift-default-5ff9c74dc9-c4jpz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.54:15021/healthz/ready\": dial tcp 10.132.1.54:15021: connect: connection refused" start-of-body= Nov 03 01:28:50.110179 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:28:50.110151 4939 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-ee618c-openshift-default-5ff9c74dc9-c4jpz" podUID="7fc8db38-3791-40c0-bcfb-716313223a85" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.54:15021/healthz/ready\": dial tcp 10.132.1.54:15021: connect: connection refused" Nov 03 01:28:50.229083 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:28:50.229042 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:28:50.968195 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:28:50.968157 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:28:51.033735 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:28:51.033192 4939 patch_prober.go:29] interesting pod/e2e-inf-a-ee618c-openshift-default-5ff9c74dc9-z4qxq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.53:15021/healthz/ready\": dial tcp 10.132.1.53:15021: connect: connection refused" start-of-body= Nov 03 01:28:51.033735 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:28:51.033255 4939 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-ee618c-openshift-default-5ff9c74dc9-z4qxq" podUID="6165abef-0df3-4007-ae59-5536a2a77e45" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.53:15021/healthz/ready\": dial tcp 10.132.1.53:15021: connect: connection refused" Nov 03 01:28:51.122931 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:28:51.121890 4939 patch_prober.go:29] interesting pod/e2e-inf-a-ee618c-openshift-default-5ff9c74dc9-c4jpz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.54:15021/healthz/ready\": dial tcp 10.132.1.54:15021: connect: connection refused" start-of-body= Nov 03 01:28:51.122931 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:28:51.121970 4939 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-ee618c-openshift-default-5ff9c74dc9-c4jpz" podUID="7fc8db38-3791-40c0-bcfb-716313223a85" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.54:15021/healthz/ready\": dial tcp 10.132.1.54:15021: connect: connection refused" Nov 03 01:28:51.718779 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:28:51.718734 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:28:52.033244 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:28:52.033169 4939 patch_prober.go:29] interesting pod/e2e-inf-a-ee618c-openshift-default-5ff9c74dc9-z4qxq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.53:15021/healthz/ready\": dial tcp 10.132.1.53:15021: connect: connection refused" start-of-body= Nov 03 01:28:52.033244 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:28:52.033212 4939 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-ee618c-openshift-default-5ff9c74dc9-z4qxq" podUID="6165abef-0df3-4007-ae59-5536a2a77e45" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.53:15021/healthz/ready\": dial tcp 10.132.1.53:15021: connect: connection refused" Nov 03 01:28:52.109802 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:28:52.109768 4939 patch_prober.go:29] interesting pod/e2e-inf-a-ee618c-openshift-default-5ff9c74dc9-c4jpz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.54:15021/healthz/ready\": dial tcp 10.132.1.54:15021: connect: connection refused" start-of-body= Nov 03 01:28:52.109994 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:28:52.109820 4939 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-ee618c-openshift-default-5ff9c74dc9-c4jpz" podUID="7fc8db38-3791-40c0-bcfb-716313223a85" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.54:15021/healthz/ready\": dial tcp 10.132.1.54:15021: connect: connection refused" Nov 03 01:28:52.568933 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:28:52.568393 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:28:53.039942 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:28:53.039589 4939 patch_prober.go:29] interesting pod/e2e-inf-a-ee618c-openshift-default-5ff9c74dc9-z4qxq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.53:15021/healthz/ready\": dial tcp 10.132.1.53:15021: connect: connection refused" start-of-body= Nov 03 01:28:53.039942 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:28:53.039657 4939 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-ee618c-openshift-default-5ff9c74dc9-z4qxq" podUID="6165abef-0df3-4007-ae59-5536a2a77e45" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.53:15021/healthz/ready\": dial tcp 10.132.1.53:15021: connect: connection refused" Nov 03 01:28:53.174315 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:28:53.174273 4939 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-ee618c-openshift-default-5ff9c74dc9-c4jpz" Nov 03 01:28:53.176719 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:28:53.175314 4939 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-ee618c-openshift-default-5ff9c74dc9-c4jpz" Nov 03 01:28:53.177524 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:28:53.177502 4939 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-ee618c-openshift-default-5ff9c74dc9-c4jpz" Nov 03 01:28:53.218341 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:28:53.218290 4939 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-5bfc51-6c9fd54845-wq8nz" Nov 03 01:28:53.332467 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:28:53.332418 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:28:54.002493 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:28:54.002460 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:28:54.111934 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:28:54.111247 4939 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-ee618c-openshift-default-5ff9c74dc9-z4qxq" Nov 03 01:28:54.113242 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:28:54.113216 4939 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-ee618c-openshift-default-5ff9c74dc9-z4qxq" Nov 03 01:28:54.114358 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:28:54.114332 4939 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-ee618c-openshift-default-5ff9c74dc9-z4qxq" Nov 03 01:28:54.217861 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:28:54.217833 4939 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-b6afd6-54f7b98c7c-lzq84" Nov 03 01:28:54.675241 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:28:54.675208 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:28:55.301559 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:28:55.301518 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:28:55.994161 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:28:55.994131 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:28:57.052099 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:28:57.052059 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:28:57.260808 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:28:57.260769 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:28:57.972186 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:28:57.972152 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:28:58.417785 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:28:58.417752 4939 scope.go:122] "RemoveContainer" containerID="d022249c6b7f0ad709448febf97a28b05d4540b81fe89ae5ce58e2542e963980" Nov 03 01:28:58.451565 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:28:58.451542 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:28:58.757259 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:28:58.757172 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:28:58.869846 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:28:58.869809 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:28:59.332958 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:28:59.332920 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:28:59.572935 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:28:59.549508 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:28:59.803041 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:28:59.800360 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:29:00.477552 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:29:00.477521 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:29:00.585453 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:29:00.585416 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:29:00.950104 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:29:00.950057 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:29:01.216890 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:29:01.216805 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:29:01.337953 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:29:01.337890 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:29:01.809735 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:29:01.809666 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:29:02.296727 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:29:02.296690 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:29:02.950870 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:29:02.950831 4939 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-4b17ac-84f44c7d99-8286l"] Nov 03 01:29:03.038269 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:29:03.036452 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:29:03.153176 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:29:03.153112 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:29:03.406281 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:29:03.406247 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:29:03.795988 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:29:03.795868 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:29:04.067805 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:29:04.067690 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:29:04.246337 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:29:04.246303 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:29:05.290343 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:29:05.290245 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:29:05.342999 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:29:05.342960 4939 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-5dab6a-57cf987799-9svqc"] Nov 03 01:29:05.426940 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:29:05.426495 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:29:05.611267 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:29:05.611100 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:29:05.839758 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:29:05.839721 4939 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-5dab6a-57cf987799-9svqc" Nov 03 01:29:05.856276 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:29:05.855606 4939 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-b-5dab6a\"" Nov 03 01:29:05.941975 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:29:05.941354 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2vg9b\" (UniqueName: \"kubernetes.io/projected/63a13b02-7c84-47a7-aa2a-73a9ed53481e-kube-api-access-2vg9b\") pod \"payload-pre-processing-e2e-inf-b-5dab6a-57cf987799-9svqc\" (UID: \"63a13b02-7c84-47a7-aa2a-73a9ed53481e\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-5dab6a-57cf987799-9svqc" Nov 03 01:29:05.941975 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:29:05.941420 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/63a13b02-7c84-47a7-aa2a-73a9ed53481e-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-5dab6a-57cf987799-9svqc\" (UID: \"63a13b02-7c84-47a7-aa2a-73a9ed53481e\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-5dab6a-57cf987799-9svqc" Nov 03 01:29:06.106933 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:29:06.043468 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2vg9b\" (UniqueName: \"kubernetes.io/projected/63a13b02-7c84-47a7-aa2a-73a9ed53481e-kube-api-access-2vg9b\") pod \"payload-pre-processing-e2e-inf-b-5dab6a-57cf987799-9svqc\" (UID: \"63a13b02-7c84-47a7-aa2a-73a9ed53481e\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-5dab6a-57cf987799-9svqc" Nov 03 01:29:06.132071 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:29:06.132026 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/63a13b02-7c84-47a7-aa2a-73a9ed53481e-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-5dab6a-57cf987799-9svqc\" (UID: \"63a13b02-7c84-47a7-aa2a-73a9ed53481e\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-5dab6a-57cf987799-9svqc" Nov 03 01:29:06.132981 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:29:06.080809 4939 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-5dab6a-57cf987799-9svqc"] Nov 03 01:29:06.165938 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:29:06.134185 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/63a13b02-7c84-47a7-aa2a-73a9ed53481e-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-5dab6a-57cf987799-9svqc\" (UID: \"63a13b02-7c84-47a7-aa2a-73a9ed53481e\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-5dab6a-57cf987799-9svqc" Nov 03 01:29:06.166258 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:29:06.117708 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2vg9b\" (UniqueName: \"kubernetes.io/projected/63a13b02-7c84-47a7-aa2a-73a9ed53481e-kube-api-access-2vg9b\") pod \"payload-pre-processing-e2e-inf-b-5dab6a-57cf987799-9svqc\" (UID: \"63a13b02-7c84-47a7-aa2a-73a9ed53481e\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-5dab6a-57cf987799-9svqc" Nov 03 01:29:06.355933 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:29:06.354263 4939 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-5dab6a-57cf987799-9svqc" Nov 03 01:29:07.267934 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:29:07.267728 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:29:07.572427 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:29:07.572313 4939 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-88bbf7/distinct-2-w1-88bbf7-kserve-5457cdcdbd-l6tfd" podUID="a397931b-faa9-4c2e-8290-491350546663" containerName="main" probeResult="failure" output="Get \"https://10.132.0.85:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 01:29:07.917257 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:29:07.916033 4939 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-2f9a3a/simulator-w0-2f9a3a-kserve-9596bbb8f-2trgz" podUID="207e2207-fb6c-4076-95df-72e95671c261" containerName="main" probeResult="failure" output="Get \"https://10.132.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 01:29:07.982018 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:29:07.963440 4939 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-f7680d/distinct-2-w2-f7680d-kserve-7d7c4564bf-cbv2c" podUID="2de18086-761b-4773-b9fc-15ddc4a220fc" containerName="main" probeResult="failure" output="Get \"https://10.132.0.75:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 01:29:07.983746 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:29:07.983699 4939 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-4b17ac-84f44c7d99-8286l" podUID="abd497dc-d8a1-42f5-8f41-c8340af41dd9" containerName="payload-pre-processing" containerID="cri-o://3764e33077507458a7b7e4cba350f19dacd8cbf57e5d14875fe70652e31a137f" gracePeriod=25 Nov 03 01:29:07.984199 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:29:07.984174 4939 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-2f9a3a/distinct-w0-2f9a3a-kserve-555565cfcf-jtcr9" podUID="ac0ece06-8718-463f-915f-5315988d9711" containerName="main" probeResult="failure" output="Get \"https://10.132.0.76:8000/ready\": context deadline exceeded" Nov 03 01:29:08.057739 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:29:08.041300 4939 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-f7680d/distinct-w2-f7680d-kserve-756f874b98-qnpv8" podUID="836753fc-f9ef-4f61-b3fe-e5b92b7ee27b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.70:8000/health\": context deadline exceeded" Nov 03 01:29:08.057739 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:29:08.041599 4939 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-f7680d/premium-w2-f7680d-kserve-76565b8cf9-hqbbt" podUID="4712e708-18a2-4a13-a12c-a5669397b2c3" containerName="main" probeResult="failure" output="Get \"https://10.132.0.63:8000/ready\": context deadline exceeded" Nov 03 01:29:08.057739 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:29:08.041695 4939 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-83755a/distinct-2-w3-83755a-kserve-6cf66b995d-2l47b" podUID="c3b05e88-5d5b-4d03-b28d-832f22e3c868" containerName="main" probeResult="failure" output="Get \"https://10.132.0.72:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 01:29:08.135096 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:29:08.134652 4939 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-88bbf7/simulator-w1-88bbf7-kserve-5fbb669f8f-42hgd" podUID="83b38883-2e7a-444d-aa68-52be4a25ae66" containerName="main" probeResult="failure" output="Get \"https://10.132.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 01:29:08.188986 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:29:08.185236 4939 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-f7680d/distinct-w2-f7680d-kserve-756f874b98-qnpv8" podUID="836753fc-f9ef-4f61-b3fe-e5b92b7ee27b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.70:8000/ready\": context deadline exceeded" Nov 03 01:29:09.245471 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:29:09.245416 4939 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-88bbf7/simulator-w1-88bbf7-kserve-5fbb669f8f-42hgd" podUID="83b38883-2e7a-444d-aa68-52be4a25ae66" containerName="main" probeResult="failure" output="Get \"https://10.132.0.62:8000/ready\": context deadline exceeded" Nov 03 01:29:09.266934 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:29:09.246957 4939 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-f7680d/distinct-w2-f7680d-kserve-756f874b98-qnpv8" podUID="836753fc-f9ef-4f61-b3fe-e5b92b7ee27b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.70:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 01:29:09.676931 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:29:09.268640 4939 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8dx487" podUID="33fe945b-76e3-4201-970d-235ff8f719a3" containerName="main" probeResult="failure" output="Get \"https://10.132.0.39:8000/ready\": context deadline exceeded" Nov 03 01:29:09.676931 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:29:09.268831 4939 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-f7680d/premium-w2-f7680d-kserve-76565b8cf9-hqbbt" podUID="4712e708-18a2-4a13-a12c-a5669397b2c3" containerName="main" probeResult="failure" output="Get \"https://10.132.0.63:8000/ready\": context deadline exceeded" Nov 03 01:29:09.677200 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:29:09.579426 4939 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-2f9a3a/distinct-w0-2f9a3a-kserve-555565cfcf-jtcr9" podUID="ac0ece06-8718-463f-915f-5315988d9711" containerName="main" probeResult="failure" output="Get \"https://10.132.0.76:8000/ready\": context deadline exceeded" Nov 03 01:29:09.677272 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:29:09.580337 4939 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-f7680d/distinct-2-w2-f7680d-kserve-7d7c4564bf-cbv2c" podUID="2de18086-761b-4773-b9fc-15ddc4a220fc" containerName="main" probeResult="failure" output="Get \"https://10.132.0.75:8000/ready\": context deadline exceeded" Nov 03 01:29:09.743931 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:29:09.606708 4939 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-88bbf7/simulator-w1-88bbf7-kserve-5fbb669f8f-42hgd" podUID="83b38883-2e7a-444d-aa68-52be4a25ae66" containerName="main" probeResult="failure" output="Get \"https://10.132.0.62:8000/health\": context deadline exceeded" Nov 03 01:29:09.743931 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:29:09.642601 4939 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-88bbf7/distinct-2-w1-88bbf7-kserve-5457cdcdbd-l6tfd" podUID="a397931b-faa9-4c2e-8290-491350546663" containerName="main" probeResult="failure" output="Get \"https://10.132.0.85:8000/ready\": context deadline exceeded" Nov 03 01:29:09.744210 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:29:09.672116 4939 patch_prober.go:29] interesting pod/e2e-inf-a-ee618c-openshift-default-5ff9c74dc9-c4jpz container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.1.54:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 01:29:09.744210 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:29:09.744168 4939 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-a-ee618c-openshift-default-5ff9c74dc9-c4jpz" podUID="7fc8db38-3791-40c0-bcfb-716313223a85" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.54:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 01:29:09.915933 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:29:09.906234 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:29:09.943190 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:29:09.938892 4939 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-2f9a3a/simulator-w0-2f9a3a-kserve-9596bbb8f-2trgz" podUID="207e2207-fb6c-4076-95df-72e95671c261" containerName="main" probeResult="failure" output="Get \"https://10.132.0.61:8000/ready\": context deadline exceeded" Nov 03 01:29:09.943190 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:29:09.940641 4939 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-83755a/distinct-2-w3-83755a-kserve-6cf66b995d-2l47b" podUID="c3b05e88-5d5b-4d03-b28d-832f22e3c868" containerName="main" probeResult="failure" output="Get \"https://10.132.0.72:8000/ready\": context deadline exceeded" Nov 03 01:29:10.023990 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:29:10.023676 4939 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-3c5983-564977cd99-qs5fk" podUID="3137d8da-32e8-47fd-87a6-f08aebe2a384" containerName="payload-pre-processing" probeResult="failure" output="timeout: failed to connect service \"10.132.0.216:50052\" within 1s: context deadline exceeded" Nov 03 01:29:10.065472 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:29:10.064027 4939 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-6c0ec0-65c5d46767-hsxz4" podUID="61b34e6e-2690-4080-bced-3a577c665f12" containerName="payload-pre-processing" probeResult="failure" output="timeout: failed to connect service \"10.132.0.170:50052\" within 1s: context deadline exceeded" Nov 03 01:29:10.215044 ip-10-0-140-107 kubenswrapper[4939]: E1103 01:29:10.198067 4939 kubelet.go:2711] "Housekeeping took longer than expected" err="housekeeping took too long" expected="1s" actual="2.336s" Nov 03 01:29:10.215044 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:29:10.198118 4939 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-5dab6a-57cf987799-9svqc"] Nov 03 01:29:10.376801 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:29:10.320507 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:29:10.440614 ip-10-0-140-107 kubenswrapper[4939]: E1103 01:29:10.440577 4939 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podabd497dc_d8a1_42f5_8f41_c8340af41dd9.slice/crio-3764e33077507458a7b7e4cba350f19dacd8cbf57e5d14875fe70652e31a137f.scope\": RecentStats: unable to find data in memory cache]" Nov 03 01:29:10.497021 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:29:10.459469 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:29:10.826413 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:29:10.815021 4939 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-88bbf7/simulator-w1-88bbf7-kserve-5fbb669f8f-42hgd" Nov 03 01:29:10.826413 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:29:10.815240 4939 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-f7680d/distinct-w2-f7680d-kserve-756f874b98-qnpv8" Nov 03 01:29:10.826413 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:29:10.815279 4939 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-f7680d/premium-w2-f7680d-kserve-76565b8cf9-hqbbt" Nov 03 01:29:10.826413 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:29:10.815317 4939 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-2f9a3a/distinct-w0-2f9a3a-kserve-555565cfcf-jtcr9" Nov 03 01:29:10.826413 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:29:10.815350 4939 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-f7680d/distinct-2-w2-f7680d-kserve-7d7c4564bf-cbv2c" Nov 03 01:29:10.826413 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:29:10.815387 4939 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-88bbf7/distinct-2-w1-88bbf7-kserve-5457cdcdbd-l6tfd" Nov 03 01:29:10.826413 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:29:10.815421 4939 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-2f9a3a/simulator-w0-2f9a3a-kserve-9596bbb8f-2trgz" Nov 03 01:29:10.826413 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:29:10.815454 4939 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-83755a/distinct-2-w3-83755a-kserve-6cf66b995d-2l47b" Nov 03 01:29:10.826413 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:29:10.815486 4939 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-2f9a3a/distinct-w0-2f9a3a-kserve-555565cfcf-jtcr9" Nov 03 01:29:10.826413 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:29:10.815514 4939 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-f7680d/premium-w2-f7680d-kserve-76565b8cf9-hqbbt" Nov 03 01:29:10.826413 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:29:10.815543 4939 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-88bbf7/simulator-w1-88bbf7-kserve-5fbb669f8f-42hgd" Nov 03 01:29:10.826413 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:29:10.815570 4939 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-f7680d/distinct-2-w2-f7680d-kserve-7d7c4564bf-cbv2c" Nov 03 01:29:10.826413 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:29:10.815596 4939 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-2f9a3a/simulator-w0-2f9a3a-kserve-9596bbb8f-2trgz" Nov 03 01:29:10.826413 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:29:10.815625 4939 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-88bbf7/distinct-2-w1-88bbf7-kserve-5457cdcdbd-l6tfd" Nov 03 01:29:10.826413 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:29:10.815654 4939 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-83755a/distinct-2-w3-83755a-kserve-6cf66b995d-2l47b" Nov 03 01:29:10.826413 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:29:10.815682 4939 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-f7680d/distinct-w2-f7680d-kserve-756f874b98-qnpv8" Nov 03 01:29:10.854928 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:29:10.843606 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:29:11.012738 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:29:11.012672 4939 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-4b17ac-84f44c7d99-8286l" Nov 03 01:29:11.136004 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:29:11.135966 4939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/abd497dc-d8a1-42f5-8f41-c8340af41dd9-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/abd497dc-d8a1-42f5-8f41-c8340af41dd9-plugins-config-volume\") pod \"abd497dc-d8a1-42f5-8f41-c8340af41dd9\" (UID: \"abd497dc-d8a1-42f5-8f41-c8340af41dd9\") " Nov 03 01:29:11.136247 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:29:11.136228 4939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/abd497dc-d8a1-42f5-8f41-c8340af41dd9-kube-api-access-m5v8q\" (UniqueName: \"kubernetes.io/projected/abd497dc-d8a1-42f5-8f41-c8340af41dd9-kube-api-access-m5v8q\") pod \"abd497dc-d8a1-42f5-8f41-c8340af41dd9\" (UID: \"abd497dc-d8a1-42f5-8f41-c8340af41dd9\") " Nov 03 01:29:11.136442 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:29:11.136391 4939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/abd497dc-d8a1-42f5-8f41-c8340af41dd9-plugins-config-volume" pod "abd497dc-d8a1-42f5-8f41-c8340af41dd9" (UID: "abd497dc-d8a1-42f5-8f41-c8340af41dd9"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 01:29:11.139120 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:29:11.139093 4939 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/abd497dc-d8a1-42f5-8f41-c8340af41dd9-plugins-config-volume\") on node \"ip-10-0-140-107.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:29:11.141304 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:29:11.141280 4939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/abd497dc-d8a1-42f5-8f41-c8340af41dd9-kube-api-access-m5v8q" pod "abd497dc-d8a1-42f5-8f41-c8340af41dd9" (UID: "abd497dc-d8a1-42f5-8f41-c8340af41dd9"). InnerVolumeSpecName "kube-api-access-m5v8q". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 01:29:11.243254 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:29:11.242662 4939 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m5v8q\" (UniqueName: \"kubernetes.io/projected/abd497dc-d8a1-42f5-8f41-c8340af41dd9-kube-api-access-m5v8q\") on node \"ip-10-0-140-107.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:29:11.244933 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:29:11.244870 4939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-5dab6a-57cf987799-9svqc" event={"ID":"63a13b02-7c84-47a7-aa2a-73a9ed53481e","Type":"ContainerStarted","Data":"5b803b19b851b18fe07ec3b154d6892be07f2408741c7cf0c43bc44ce3404e9e"} Nov 03 01:29:11.245089 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:29:11.244995 4939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-5dab6a-57cf987799-9svqc" event={"ID":"63a13b02-7c84-47a7-aa2a-73a9ed53481e","Type":"ContainerStarted","Data":"9856e654c27c8b06c2641dfcf82f1b68fc8e02f3de2a480f5bbd72a212c72f6a"} Nov 03 01:29:11.245618 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:29:11.245592 4939 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-5dab6a-57cf987799-9svqc" Nov 03 01:29:11.268934 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:29:11.255524 4939 generic.go:356] "Generic (PLEG): container finished" podID="abd497dc-d8a1-42f5-8f41-c8340af41dd9" containerID="3764e33077507458a7b7e4cba350f19dacd8cbf57e5d14875fe70652e31a137f" exitCode=0 Nov 03 01:29:11.268934 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:29:11.255784 4939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-4b17ac-84f44c7d99-8286l" event={"ID":"abd497dc-d8a1-42f5-8f41-c8340af41dd9","Type":"ContainerDied","Data":"3764e33077507458a7b7e4cba350f19dacd8cbf57e5d14875fe70652e31a137f"} Nov 03 01:29:11.268934 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:29:11.255837 4939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-4b17ac-84f44c7d99-8286l" event={"ID":"abd497dc-d8a1-42f5-8f41-c8340af41dd9","Type":"ContainerDied","Data":"70468bbc72743a3470e9a93a3693dc439cdcdd3802c23fd77acefcbd49293e5a"} Nov 03 01:29:11.268934 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:29:11.255886 4939 scope.go:122] "RemoveContainer" containerID="3764e33077507458a7b7e4cba350f19dacd8cbf57e5d14875fe70652e31a137f" Nov 03 01:29:11.268934 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:29:11.256129 4939 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-4b17ac-84f44c7d99-8286l" Nov 03 01:29:11.293934 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:29:11.273151 4939 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-5dab6a-57cf987799-9svqc" podStartSLOduration=6.273130455 podStartE2EDuration="6.273130455s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 01:29:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 01:29:11.269221012 +0000 UTC m=+2899.533555958" watchObservedRunningTime="2026-11-03 01:29:11.273130455 +0000 UTC m=+2899.537465404" Nov 03 01:29:11.328700 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:29:11.328670 4939 scope.go:122] "RemoveContainer" containerID="3764e33077507458a7b7e4cba350f19dacd8cbf57e5d14875fe70652e31a137f" Nov 03 01:29:11.329191 ip-10-0-140-107 kubenswrapper[4939]: E1103 01:29:11.329162 4939 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3764e33077507458a7b7e4cba350f19dacd8cbf57e5d14875fe70652e31a137f\": container with ID starting with 3764e33077507458a7b7e4cba350f19dacd8cbf57e5d14875fe70652e31a137f not found: ID does not exist" containerID="3764e33077507458a7b7e4cba350f19dacd8cbf57e5d14875fe70652e31a137f" Nov 03 01:29:11.329295 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:29:11.329201 4939 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3764e33077507458a7b7e4cba350f19dacd8cbf57e5d14875fe70652e31a137f"} err="failed to get container status \"3764e33077507458a7b7e4cba350f19dacd8cbf57e5d14875fe70652e31a137f\": rpc error: code = NotFound desc = could not find container \"3764e33077507458a7b7e4cba350f19dacd8cbf57e5d14875fe70652e31a137f\": container with ID starting with 3764e33077507458a7b7e4cba350f19dacd8cbf57e5d14875fe70652e31a137f not found: ID does not exist" Nov 03 01:29:11.335602 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:29:11.335572 4939 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-4b17ac-84f44c7d99-8286l"] Nov 03 01:29:11.342476 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:29:11.342452 4939 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-4b17ac-84f44c7d99-8286l"] Nov 03 01:29:11.712378 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:29:11.712319 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:29:11.933955 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:29:11.932951 4939 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="abd497dc-d8a1-42f5-8f41-c8340af41dd9" path="/var/lib/kubelet/pods/abd497dc-d8a1-42f5-8f41-c8340af41dd9/volumes" Nov 03 01:29:11.933955 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:29:11.933473 4939 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-a-ee618c/test-model-ee618c-kserve-844f5b79dc-f6kl7"] Nov 03 01:29:11.936971 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:29:11.935316 4939 cpu_manager.go:414] "containerMap: removing container" podUID="abd497dc-d8a1-42f5-8f41-c8340af41dd9" containerName="payload-pre-processing" Nov 03 01:29:11.936971 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:29:11.935343 4939 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="abd497dc-d8a1-42f5-8f41-c8340af41dd9" containerName="payload-pre-processing" Nov 03 01:29:11.936971 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:29:11.935596 4939 memory_manager.go:365] "RemoveStaleState removing state" podUID="abd497dc-d8a1-42f5-8f41-c8340af41dd9" containerName="payload-pre-processing" Nov 03 01:29:11.944931 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:29:11.941363 4939 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-ee618c/test-model-ee618c-kserve-844f5b79dc-f6kl7"] Nov 03 01:29:11.944931 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:29:11.941466 4939 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-ee618c/test-model-ee618c-kserve-844f5b79dc-f6kl7" Nov 03 01:29:11.980017 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:29:11.978808 4939 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-ee618c\"/\"default-dockercfg-fs4xk\"" Nov 03 01:29:11.980017 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:29:11.979440 4939 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-ee618c\"/\"test-model-ee618c-kserve-self-signed-certs\"" Nov 03 01:29:11.980017 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:29:11.979525 4939 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-ee618c\"/\"kube-root-ca.crt\"" Nov 03 01:29:11.980017 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:29:11.979626 4939 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-ee618c\"/\"openshift-service-ca.crt\"" Nov 03 01:29:11.980017 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:29:11.979847 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:29:11.995620 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:29:11.995589 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/abab7b3b-f57d-4abe-be59-295e4ab8a769-tmp-dir\") pod \"test-model-ee618c-kserve-844f5b79dc-f6kl7\" (UID: \"abab7b3b-f57d-4abe-be59-295e4ab8a769\") " pod="ai-tenant-e2e-inf-a-ee618c/test-model-ee618c-kserve-844f5b79dc-f6kl7" Nov 03 01:29:11.995751 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:29:11.995643 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/abab7b3b-f57d-4abe-be59-295e4ab8a769-dshm\") pod \"test-model-ee618c-kserve-844f5b79dc-f6kl7\" (UID: \"abab7b3b-f57d-4abe-be59-295e4ab8a769\") " pod="ai-tenant-e2e-inf-a-ee618c/test-model-ee618c-kserve-844f5b79dc-f6kl7" Nov 03 01:29:11.995823 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:29:11.995797 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/abab7b3b-f57d-4abe-be59-295e4ab8a769-home\") pod \"test-model-ee618c-kserve-844f5b79dc-f6kl7\" (UID: \"abab7b3b-f57d-4abe-be59-295e4ab8a769\") " pod="ai-tenant-e2e-inf-a-ee618c/test-model-ee618c-kserve-844f5b79dc-f6kl7" Nov 03 01:29:11.995974 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:29:11.995941 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/abab7b3b-f57d-4abe-be59-295e4ab8a769-model-cache\") pod \"test-model-ee618c-kserve-844f5b79dc-f6kl7\" (UID: \"abab7b3b-f57d-4abe-be59-295e4ab8a769\") " pod="ai-tenant-e2e-inf-a-ee618c/test-model-ee618c-kserve-844f5b79dc-f6kl7" Nov 03 01:29:11.996058 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:29:11.996009 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/abab7b3b-f57d-4abe-be59-295e4ab8a769-tls-certs\") pod \"test-model-ee618c-kserve-844f5b79dc-f6kl7\" (UID: \"abab7b3b-f57d-4abe-be59-295e4ab8a769\") " pod="ai-tenant-e2e-inf-a-ee618c/test-model-ee618c-kserve-844f5b79dc-f6kl7" Nov 03 01:29:11.996223 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:29:11.996201 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vrr2k\" (UniqueName: \"kubernetes.io/projected/abab7b3b-f57d-4abe-be59-295e4ab8a769-kube-api-access-vrr2k\") pod \"test-model-ee618c-kserve-844f5b79dc-f6kl7\" (UID: \"abab7b3b-f57d-4abe-be59-295e4ab8a769\") " pod="ai-tenant-e2e-inf-a-ee618c/test-model-ee618c-kserve-844f5b79dc-f6kl7" Nov 03 01:29:12.108938 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:29:12.100783 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vrr2k\" (UniqueName: \"kubernetes.io/projected/abab7b3b-f57d-4abe-be59-295e4ab8a769-kube-api-access-vrr2k\") pod \"test-model-ee618c-kserve-844f5b79dc-f6kl7\" (UID: \"abab7b3b-f57d-4abe-be59-295e4ab8a769\") " pod="ai-tenant-e2e-inf-a-ee618c/test-model-ee618c-kserve-844f5b79dc-f6kl7" Nov 03 01:29:12.109353 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:29:12.109320 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/abab7b3b-f57d-4abe-be59-295e4ab8a769-tmp-dir\") pod \"test-model-ee618c-kserve-844f5b79dc-f6kl7\" (UID: \"abab7b3b-f57d-4abe-be59-295e4ab8a769\") " pod="ai-tenant-e2e-inf-a-ee618c/test-model-ee618c-kserve-844f5b79dc-f6kl7" Nov 03 01:29:12.109541 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:29:12.109524 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/abab7b3b-f57d-4abe-be59-295e4ab8a769-dshm\") pod \"test-model-ee618c-kserve-844f5b79dc-f6kl7\" (UID: \"abab7b3b-f57d-4abe-be59-295e4ab8a769\") " pod="ai-tenant-e2e-inf-a-ee618c/test-model-ee618c-kserve-844f5b79dc-f6kl7" Nov 03 01:29:12.109764 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:29:12.109749 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/abab7b3b-f57d-4abe-be59-295e4ab8a769-home\") pod \"test-model-ee618c-kserve-844f5b79dc-f6kl7\" (UID: \"abab7b3b-f57d-4abe-be59-295e4ab8a769\") " pod="ai-tenant-e2e-inf-a-ee618c/test-model-ee618c-kserve-844f5b79dc-f6kl7" Nov 03 01:29:12.109964 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:29:12.109946 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/abab7b3b-f57d-4abe-be59-295e4ab8a769-model-cache\") pod \"test-model-ee618c-kserve-844f5b79dc-f6kl7\" (UID: \"abab7b3b-f57d-4abe-be59-295e4ab8a769\") " pod="ai-tenant-e2e-inf-a-ee618c/test-model-ee618c-kserve-844f5b79dc-f6kl7" Nov 03 01:29:12.110126 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:29:12.110110 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/abab7b3b-f57d-4abe-be59-295e4ab8a769-tls-certs\") pod \"test-model-ee618c-kserve-844f5b79dc-f6kl7\" (UID: \"abab7b3b-f57d-4abe-be59-295e4ab8a769\") " pod="ai-tenant-e2e-inf-a-ee618c/test-model-ee618c-kserve-844f5b79dc-f6kl7" Nov 03 01:29:12.151443 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:29:12.111449 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/abab7b3b-f57d-4abe-be59-295e4ab8a769-tmp-dir\") pod \"test-model-ee618c-kserve-844f5b79dc-f6kl7\" (UID: \"abab7b3b-f57d-4abe-be59-295e4ab8a769\") " pod="ai-tenant-e2e-inf-a-ee618c/test-model-ee618c-kserve-844f5b79dc-f6kl7" Nov 03 01:29:12.151443 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:29:12.111727 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/abab7b3b-f57d-4abe-be59-295e4ab8a769-home\") pod \"test-model-ee618c-kserve-844f5b79dc-f6kl7\" (UID: \"abab7b3b-f57d-4abe-be59-295e4ab8a769\") " pod="ai-tenant-e2e-inf-a-ee618c/test-model-ee618c-kserve-844f5b79dc-f6kl7" Nov 03 01:29:12.151443 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:29:12.113102 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/abab7b3b-f57d-4abe-be59-295e4ab8a769-model-cache\") pod \"test-model-ee618c-kserve-844f5b79dc-f6kl7\" (UID: \"abab7b3b-f57d-4abe-be59-295e4ab8a769\") " pod="ai-tenant-e2e-inf-a-ee618c/test-model-ee618c-kserve-844f5b79dc-f6kl7" Nov 03 01:29:12.381167 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:29:12.297766 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vrr2k\" (UniqueName: \"kubernetes.io/projected/abab7b3b-f57d-4abe-be59-295e4ab8a769-kube-api-access-vrr2k\") pod \"test-model-ee618c-kserve-844f5b79dc-f6kl7\" (UID: \"abab7b3b-f57d-4abe-be59-295e4ab8a769\") " pod="ai-tenant-e2e-inf-a-ee618c/test-model-ee618c-kserve-844f5b79dc-f6kl7" Nov 03 01:29:12.381167 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:29:12.298564 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/abab7b3b-f57d-4abe-be59-295e4ab8a769-dshm\") pod \"test-model-ee618c-kserve-844f5b79dc-f6kl7\" (UID: \"abab7b3b-f57d-4abe-be59-295e4ab8a769\") " pod="ai-tenant-e2e-inf-a-ee618c/test-model-ee618c-kserve-844f5b79dc-f6kl7" Nov 03 01:29:12.381167 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:29:12.325711 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/abab7b3b-f57d-4abe-be59-295e4ab8a769-tls-certs\") pod \"test-model-ee618c-kserve-844f5b79dc-f6kl7\" (UID: \"abab7b3b-f57d-4abe-be59-295e4ab8a769\") " pod="ai-tenant-e2e-inf-a-ee618c/test-model-ee618c-kserve-844f5b79dc-f6kl7" Nov 03 01:29:12.593442 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:29:12.593401 4939 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-ee618c/test-model-ee618c-kserve-844f5b79dc-f6kl7" Nov 03 01:29:12.798606 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:29:12.796648 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:29:12.929199 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:29:12.927337 4939 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-ee618c/test-model-ee618c-kserve-844f5b79dc-f6kl7"] Nov 03 01:29:13.331830 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:29:13.331780 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:29:13.353588 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:29:13.353537 4939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-ee618c/test-model-ee618c-kserve-844f5b79dc-f6kl7" event={"ID":"abab7b3b-f57d-4abe-be59-295e4ab8a769","Type":"ContainerStarted","Data":"78e5d343d7e82472c3033db211fd34235eaf4ffc481172518a49f59e95136f13"} Nov 03 01:29:13.353777 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:29:13.353596 4939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-ee618c/test-model-ee618c-kserve-844f5b79dc-f6kl7" event={"ID":"abab7b3b-f57d-4abe-be59-295e4ab8a769","Type":"ContainerStarted","Data":"cef7c24d87f62643e1c865788528ddd2ce9fcf49822ba0261c8f7892e9a19e38"} Nov 03 01:29:13.378784 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:29:13.378721 4939 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-inf-a-ee618c/test-model-ee618c-kserve-844f5b79dc-f6kl7" podStartSLOduration=2.378702253 podStartE2EDuration="2.378702253s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 01:29:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 01:29:13.372136786 +0000 UTC m=+2901.636471731" watchObservedRunningTime="2026-11-03 01:29:13.378702253 +0000 UTC m=+2901.643037199" Nov 03 01:29:13.603270 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:29:13.603155 4939 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-a-ee618c/test-model-ee618c-kserve-844f5b79dc-f6kl7" Nov 03 01:29:13.603853 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:29:13.603827 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:29:13.750746 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:29:13.750416 4939 patch_prober.go:29] interesting pod/e2e-inf-a-4b17ac-openshift-default-6b75f999cf-k5vtr container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.1.46:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 01:29:13.750746 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:29:13.750481 4939 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-a-4b17ac-openshift-default-6b75f999cf-k5vtr" podUID="5e9f1346-a093-44e2-8f84-712cafcdffdf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.46:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 01:29:14.479940 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:29:14.477468 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:29:15.189674 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:29:15.189637 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:29:15.380946 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:29:15.380536 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:29:15.496402 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:29:15.496309 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:29:15.855788 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:29:15.855758 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:29:16.165329 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:29:16.165241 4939 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-4b17ac-openshift-default-6b75f999cf-k5vtr"] Nov 03 01:29:16.168791 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:29:16.168752 4939 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-a-4b17ac-openshift-default-6b75f999cf-k5vtr" podUID="5e9f1346-a093-44e2-8f84-712cafcdffdf" containerName="istio-proxy" containerID="cri-o://ec5a9665db5301394447fd6076ac3a6dbc1e2fb1aeebd87a07d86f21d2873ddb" gracePeriod=30 Nov 03 01:29:16.169563 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:29:16.168324 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:29:16.295384 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:29:16.295351 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:29:17.296206 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:29:17.296166 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:29:17.504826 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:29:17.504794 4939 generic.go:356] "Generic (PLEG): container finished" podID="5e9f1346-a093-44e2-8f84-712cafcdffdf" containerID="ec5a9665db5301394447fd6076ac3a6dbc1e2fb1aeebd87a07d86f21d2873ddb" exitCode=0 Nov 03 01:29:17.504999 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:29:17.504845 4939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-4b17ac-openshift-default-6b75f999cf-k5vtr" event={"ID":"5e9f1346-a093-44e2-8f84-712cafcdffdf","Type":"ContainerDied","Data":"ec5a9665db5301394447fd6076ac3a6dbc1e2fb1aeebd87a07d86f21d2873ddb"} Nov 03 01:29:17.580397 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:29:17.580361 4939 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-4b17ac-openshift-default-6b75f999cf-k5vtr" Nov 03 01:29:17.618362 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:29:17.618323 4939 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-b-b4971a-openshift-default-5dc48765-8rn6x"] Nov 03 01:29:17.619795 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:29:17.619761 4939 cpu_manager.go:414] "containerMap: removing container" podUID="5e9f1346-a093-44e2-8f84-712cafcdffdf" containerName="istio-proxy" Nov 03 01:29:17.619795 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:29:17.619790 4939 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5e9f1346-a093-44e2-8f84-712cafcdffdf" containerName="istio-proxy" Nov 03 01:29:17.620089 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:29:17.620073 4939 memory_manager.go:365] "RemoveStaleState removing state" podUID="5e9f1346-a093-44e2-8f84-712cafcdffdf" containerName="istio-proxy" Nov 03 01:29:17.634499 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:29:17.632583 4939 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-b4971a-openshift-default-5dc48765-8rn6x"] Nov 03 01:29:17.634499 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:29:17.632754 4939 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-b4971a-openshift-default-5dc48765-8rn6x" Nov 03 01:29:17.641206 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:29:17.638487 4939 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-b-b4971a-openshift-default-dockercfg-st5b4\"" Nov 03 01:29:17.679246 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:29:17.679194 4939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5e9f1346-a093-44e2-8f84-712cafcdffdf-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5e9f1346-a093-44e2-8f84-712cafcdffdf-istio-envoy\") pod \"5e9f1346-a093-44e2-8f84-712cafcdffdf\" (UID: \"5e9f1346-a093-44e2-8f84-712cafcdffdf\") " Nov 03 01:29:17.679436 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:29:17.679280 4939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5e9f1346-a093-44e2-8f84-712cafcdffdf-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5e9f1346-a093-44e2-8f84-712cafcdffdf-workload-certs\") pod \"5e9f1346-a093-44e2-8f84-712cafcdffdf\" (UID: \"5e9f1346-a093-44e2-8f84-712cafcdffdf\") " Nov 03 01:29:17.679813 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:29:17.679745 4939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5e9f1346-a093-44e2-8f84-712cafcdffdf-kube-api-access-mg664\" (UniqueName: \"kubernetes.io/projected/5e9f1346-a093-44e2-8f84-712cafcdffdf-kube-api-access-mg664\") pod \"5e9f1346-a093-44e2-8f84-712cafcdffdf\" (UID: \"5e9f1346-a093-44e2-8f84-712cafcdffdf\") " Nov 03 01:29:17.679989 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:29:17.679818 4939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5e9f1346-a093-44e2-8f84-712cafcdffdf-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5e9f1346-a093-44e2-8f84-712cafcdffdf-istiod-ca-cert\") pod \"5e9f1346-a093-44e2-8f84-712cafcdffdf\" (UID: \"5e9f1346-a093-44e2-8f84-712cafcdffdf\") " Nov 03 01:29:17.679989 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:29:17.679850 4939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/5e9f1346-a093-44e2-8f84-712cafcdffdf-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5e9f1346-a093-44e2-8f84-712cafcdffdf-istio-podinfo\") pod \"5e9f1346-a093-44e2-8f84-712cafcdffdf\" (UID: \"5e9f1346-a093-44e2-8f84-712cafcdffdf\") " Nov 03 01:29:17.679989 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:29:17.679947 4939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5e9f1346-a093-44e2-8f84-712cafcdffdf-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5e9f1346-a093-44e2-8f84-712cafcdffdf-istio-data\") pod \"5e9f1346-a093-44e2-8f84-712cafcdffdf\" (UID: \"5e9f1346-a093-44e2-8f84-712cafcdffdf\") " Nov 03 01:29:17.680177 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:29:17.679995 4939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5e9f1346-a093-44e2-8f84-712cafcdffdf-istio-token\" (UniqueName: \"kubernetes.io/projected/5e9f1346-a093-44e2-8f84-712cafcdffdf-istio-token\") pod \"5e9f1346-a093-44e2-8f84-712cafcdffdf\" (UID: \"5e9f1346-a093-44e2-8f84-712cafcdffdf\") " Nov 03 01:29:17.680177 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:29:17.680060 4939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5e9f1346-a093-44e2-8f84-712cafcdffdf-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5e9f1346-a093-44e2-8f84-712cafcdffdf-credential-socket\") pod \"5e9f1346-a093-44e2-8f84-712cafcdffdf\" (UID: \"5e9f1346-a093-44e2-8f84-712cafcdffdf\") " Nov 03 01:29:17.680177 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:29:17.680099 4939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5e9f1346-a093-44e2-8f84-712cafcdffdf-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5e9f1346-a093-44e2-8f84-712cafcdffdf-workload-socket\") pod \"5e9f1346-a093-44e2-8f84-712cafcdffdf\" (UID: \"5e9f1346-a093-44e2-8f84-712cafcdffdf\") " Nov 03 01:29:17.680444 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:29:17.680415 4939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5e9f1346-a093-44e2-8f84-712cafcdffdf-workload-certs" pod "5e9f1346-a093-44e2-8f84-712cafcdffdf" (UID: "5e9f1346-a093-44e2-8f84-712cafcdffdf"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 01:29:17.680562 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:29:17.680429 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/febb83d9-3a57-40a5-b972-c45611313469-istio-envoy\") pod \"e2e-inf-b-b4971a-openshift-default-5dc48765-8rn6x\" (UID: \"febb83d9-3a57-40a5-b972-c45611313469\") " pod="openshift-ingress/e2e-inf-b-b4971a-openshift-default-5dc48765-8rn6x" Nov 03 01:29:17.680727 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:29:17.680692 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/febb83d9-3a57-40a5-b972-c45611313469-workload-socket\") pod \"e2e-inf-b-b4971a-openshift-default-5dc48765-8rn6x\" (UID: \"febb83d9-3a57-40a5-b972-c45611313469\") " pod="openshift-ingress/e2e-inf-b-b4971a-openshift-default-5dc48765-8rn6x" Nov 03 01:29:17.680942 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:29:17.680921 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/febb83d9-3a57-40a5-b972-c45611313469-workload-certs\") pod \"e2e-inf-b-b4971a-openshift-default-5dc48765-8rn6x\" (UID: \"febb83d9-3a57-40a5-b972-c45611313469\") " pod="openshift-ingress/e2e-inf-b-b4971a-openshift-default-5dc48765-8rn6x" Nov 03 01:29:17.681346 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:29:17.681322 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/febb83d9-3a57-40a5-b972-c45611313469-istio-token\") pod \"e2e-inf-b-b4971a-openshift-default-5dc48765-8rn6x\" (UID: \"febb83d9-3a57-40a5-b972-c45611313469\") " pod="openshift-ingress/e2e-inf-b-b4971a-openshift-default-5dc48765-8rn6x" Nov 03 01:29:17.681537 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:29:17.681029 4939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5e9f1346-a093-44e2-8f84-712cafcdffdf-credential-socket" pod "5e9f1346-a093-44e2-8f84-712cafcdffdf" (UID: "5e9f1346-a093-44e2-8f84-712cafcdffdf"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 01:29:17.681537 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:29:17.681503 4939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5e9f1346-a093-44e2-8f84-712cafcdffdf-istiod-ca-cert" pod "5e9f1346-a093-44e2-8f84-712cafcdffdf" (UID: "5e9f1346-a093-44e2-8f84-712cafcdffdf"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 01:29:17.681709 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:29:17.681515 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/febb83d9-3a57-40a5-b972-c45611313469-istio-podinfo\") pod \"e2e-inf-b-b4971a-openshift-default-5dc48765-8rn6x\" (UID: \"febb83d9-3a57-40a5-b972-c45611313469\") " pod="openshift-ingress/e2e-inf-b-b4971a-openshift-default-5dc48765-8rn6x" Nov 03 01:29:17.681709 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:29:17.681683 4939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5e9f1346-a093-44e2-8f84-712cafcdffdf-istio-data" pod "5e9f1346-a093-44e2-8f84-712cafcdffdf" (UID: "5e9f1346-a093-44e2-8f84-712cafcdffdf"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 01:29:17.681834 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:29:17.681807 4939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5e9f1346-a093-44e2-8f84-712cafcdffdf-workload-socket" pod "5e9f1346-a093-44e2-8f84-712cafcdffdf" (UID: "5e9f1346-a093-44e2-8f84-712cafcdffdf"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 01:29:17.682156 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:29:17.682132 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/febb83d9-3a57-40a5-b972-c45611313469-credential-socket\") pod \"e2e-inf-b-b4971a-openshift-default-5dc48765-8rn6x\" (UID: \"febb83d9-3a57-40a5-b972-c45611313469\") " pod="openshift-ingress/e2e-inf-b-b4971a-openshift-default-5dc48765-8rn6x" Nov 03 01:29:17.682340 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:29:17.682320 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l8qt8\" (UniqueName: \"kubernetes.io/projected/febb83d9-3a57-40a5-b972-c45611313469-kube-api-access-l8qt8\") pod \"e2e-inf-b-b4971a-openshift-default-5dc48765-8rn6x\" (UID: \"febb83d9-3a57-40a5-b972-c45611313469\") " pod="openshift-ingress/e2e-inf-b-b4971a-openshift-default-5dc48765-8rn6x" Nov 03 01:29:17.682580 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:29:17.682557 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/febb83d9-3a57-40a5-b972-c45611313469-istio-data\") pod \"e2e-inf-b-b4971a-openshift-default-5dc48765-8rn6x\" (UID: \"febb83d9-3a57-40a5-b972-c45611313469\") " pod="openshift-ingress/e2e-inf-b-b4971a-openshift-default-5dc48765-8rn6x" Nov 03 01:29:17.682726 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:29:17.682708 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/febb83d9-3a57-40a5-b972-c45611313469-istiod-ca-cert\") pod \"e2e-inf-b-b4971a-openshift-default-5dc48765-8rn6x\" (UID: \"febb83d9-3a57-40a5-b972-c45611313469\") " pod="openshift-ingress/e2e-inf-b-b4971a-openshift-default-5dc48765-8rn6x" Nov 03 01:29:17.683154 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:29:17.683050 4939 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5e9f1346-a093-44e2-8f84-712cafcdffdf-istiod-ca-cert\") on node \"ip-10-0-140-107.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:29:17.683154 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:29:17.683080 4939 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5e9f1346-a093-44e2-8f84-712cafcdffdf-istio-data\") on node \"ip-10-0-140-107.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:29:17.683154 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:29:17.683097 4939 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5e9f1346-a093-44e2-8f84-712cafcdffdf-credential-socket\") on node \"ip-10-0-140-107.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:29:17.683154 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:29:17.683111 4939 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5e9f1346-a093-44e2-8f84-712cafcdffdf-workload-socket\") on node \"ip-10-0-140-107.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:29:17.683154 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:29:17.683125 4939 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5e9f1346-a093-44e2-8f84-712cafcdffdf-workload-certs\") on node \"ip-10-0-140-107.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:29:17.684813 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:29:17.684762 4939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5e9f1346-a093-44e2-8f84-712cafcdffdf-istio-envoy" pod "5e9f1346-a093-44e2-8f84-712cafcdffdf" (UID: "5e9f1346-a093-44e2-8f84-712cafcdffdf"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 01:29:17.685214 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:29:17.685171 4939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/5e9f1346-a093-44e2-8f84-712cafcdffdf-istio-podinfo" pod "5e9f1346-a093-44e2-8f84-712cafcdffdf" (UID: "5e9f1346-a093-44e2-8f84-712cafcdffdf"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 01:29:17.685313 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:29:17.685288 4939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5e9f1346-a093-44e2-8f84-712cafcdffdf-istio-token" pod "5e9f1346-a093-44e2-8f84-712cafcdffdf" (UID: "5e9f1346-a093-44e2-8f84-712cafcdffdf"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 01:29:17.686578 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:29:17.686556 4939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5e9f1346-a093-44e2-8f84-712cafcdffdf-kube-api-access-mg664" pod "5e9f1346-a093-44e2-8f84-712cafcdffdf" (UID: "5e9f1346-a093-44e2-8f84-712cafcdffdf"). InnerVolumeSpecName "kube-api-access-mg664". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 01:29:17.784332 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:29:17.784299 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/febb83d9-3a57-40a5-b972-c45611313469-istio-data\") pod \"e2e-inf-b-b4971a-openshift-default-5dc48765-8rn6x\" (UID: \"febb83d9-3a57-40a5-b972-c45611313469\") " pod="openshift-ingress/e2e-inf-b-b4971a-openshift-default-5dc48765-8rn6x" Nov 03 01:29:17.785635 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:29:17.784342 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/febb83d9-3a57-40a5-b972-c45611313469-istiod-ca-cert\") pod \"e2e-inf-b-b4971a-openshift-default-5dc48765-8rn6x\" (UID: \"febb83d9-3a57-40a5-b972-c45611313469\") " pod="openshift-ingress/e2e-inf-b-b4971a-openshift-default-5dc48765-8rn6x" Nov 03 01:29:17.785635 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:29:17.784402 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/febb83d9-3a57-40a5-b972-c45611313469-istio-envoy\") pod \"e2e-inf-b-b4971a-openshift-default-5dc48765-8rn6x\" (UID: \"febb83d9-3a57-40a5-b972-c45611313469\") " pod="openshift-ingress/e2e-inf-b-b4971a-openshift-default-5dc48765-8rn6x" Nov 03 01:29:17.785635 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:29:17.784427 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/febb83d9-3a57-40a5-b972-c45611313469-workload-socket\") pod \"e2e-inf-b-b4971a-openshift-default-5dc48765-8rn6x\" (UID: \"febb83d9-3a57-40a5-b972-c45611313469\") " pod="openshift-ingress/e2e-inf-b-b4971a-openshift-default-5dc48765-8rn6x" Nov 03 01:29:17.785635 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:29:17.784474 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/febb83d9-3a57-40a5-b972-c45611313469-workload-certs\") pod \"e2e-inf-b-b4971a-openshift-default-5dc48765-8rn6x\" (UID: \"febb83d9-3a57-40a5-b972-c45611313469\") " pod="openshift-ingress/e2e-inf-b-b4971a-openshift-default-5dc48765-8rn6x" Nov 03 01:29:17.785635 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:29:17.784628 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/febb83d9-3a57-40a5-b972-c45611313469-istio-token\") pod \"e2e-inf-b-b4971a-openshift-default-5dc48765-8rn6x\" (UID: \"febb83d9-3a57-40a5-b972-c45611313469\") " pod="openshift-ingress/e2e-inf-b-b4971a-openshift-default-5dc48765-8rn6x" Nov 03 01:29:17.785635 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:29:17.784690 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/febb83d9-3a57-40a5-b972-c45611313469-istio-podinfo\") pod \"e2e-inf-b-b4971a-openshift-default-5dc48765-8rn6x\" (UID: \"febb83d9-3a57-40a5-b972-c45611313469\") " pod="openshift-ingress/e2e-inf-b-b4971a-openshift-default-5dc48765-8rn6x" Nov 03 01:29:17.785635 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:29:17.784765 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/febb83d9-3a57-40a5-b972-c45611313469-istio-data\") pod \"e2e-inf-b-b4971a-openshift-default-5dc48765-8rn6x\" (UID: \"febb83d9-3a57-40a5-b972-c45611313469\") " pod="openshift-ingress/e2e-inf-b-b4971a-openshift-default-5dc48765-8rn6x" Nov 03 01:29:17.785635 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:29:17.784810 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/febb83d9-3a57-40a5-b972-c45611313469-credential-socket\") pod \"e2e-inf-b-b4971a-openshift-default-5dc48765-8rn6x\" (UID: \"febb83d9-3a57-40a5-b972-c45611313469\") " pod="openshift-ingress/e2e-inf-b-b4971a-openshift-default-5dc48765-8rn6x" Nov 03 01:29:17.785635 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:29:17.784870 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/febb83d9-3a57-40a5-b972-c45611313469-workload-socket\") pod \"e2e-inf-b-b4971a-openshift-default-5dc48765-8rn6x\" (UID: \"febb83d9-3a57-40a5-b972-c45611313469\") " pod="openshift-ingress/e2e-inf-b-b4971a-openshift-default-5dc48765-8rn6x" Nov 03 01:29:17.785635 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:29:17.784871 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l8qt8\" (UniqueName: \"kubernetes.io/projected/febb83d9-3a57-40a5-b972-c45611313469-kube-api-access-l8qt8\") pod \"e2e-inf-b-b4971a-openshift-default-5dc48765-8rn6x\" (UID: \"febb83d9-3a57-40a5-b972-c45611313469\") " pod="openshift-ingress/e2e-inf-b-b4971a-openshift-default-5dc48765-8rn6x" Nov 03 01:29:17.785635 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:29:17.785151 4939 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5e9f1346-a093-44e2-8f84-712cafcdffdf-istio-podinfo\") on node \"ip-10-0-140-107.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:29:17.785635 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:29:17.785175 4939 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5e9f1346-a093-44e2-8f84-712cafcdffdf-istio-token\") on node \"ip-10-0-140-107.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:29:17.785635 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:29:17.785190 4939 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5e9f1346-a093-44e2-8f84-712cafcdffdf-istio-envoy\") on node \"ip-10-0-140-107.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:29:17.785635 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:29:17.785201 4939 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mg664\" (UniqueName: \"kubernetes.io/projected/5e9f1346-a093-44e2-8f84-712cafcdffdf-kube-api-access-mg664\") on node \"ip-10-0-140-107.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:29:17.785635 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:29:17.785254 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/febb83d9-3a57-40a5-b972-c45611313469-istiod-ca-cert\") pod \"e2e-inf-b-b4971a-openshift-default-5dc48765-8rn6x\" (UID: \"febb83d9-3a57-40a5-b972-c45611313469\") " pod="openshift-ingress/e2e-inf-b-b4971a-openshift-default-5dc48765-8rn6x" Nov 03 01:29:17.785635 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:29:17.785315 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/febb83d9-3a57-40a5-b972-c45611313469-credential-socket\") pod \"e2e-inf-b-b4971a-openshift-default-5dc48765-8rn6x\" (UID: \"febb83d9-3a57-40a5-b972-c45611313469\") " pod="openshift-ingress/e2e-inf-b-b4971a-openshift-default-5dc48765-8rn6x" Nov 03 01:29:17.785635 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:29:17.785333 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/febb83d9-3a57-40a5-b972-c45611313469-workload-certs\") pod \"e2e-inf-b-b4971a-openshift-default-5dc48765-8rn6x\" (UID: \"febb83d9-3a57-40a5-b972-c45611313469\") " pod="openshift-ingress/e2e-inf-b-b4971a-openshift-default-5dc48765-8rn6x" Nov 03 01:29:17.787002 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:29:17.786972 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/febb83d9-3a57-40a5-b972-c45611313469-istio-envoy\") pod \"e2e-inf-b-b4971a-openshift-default-5dc48765-8rn6x\" (UID: \"febb83d9-3a57-40a5-b972-c45611313469\") " pod="openshift-ingress/e2e-inf-b-b4971a-openshift-default-5dc48765-8rn6x" Nov 03 01:29:17.787243 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:29:17.787227 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/febb83d9-3a57-40a5-b972-c45611313469-istio-podinfo\") pod \"e2e-inf-b-b4971a-openshift-default-5dc48765-8rn6x\" (UID: \"febb83d9-3a57-40a5-b972-c45611313469\") " pod="openshift-ingress/e2e-inf-b-b4971a-openshift-default-5dc48765-8rn6x" Nov 03 01:29:17.792358 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:29:17.792336 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/febb83d9-3a57-40a5-b972-c45611313469-istio-token\") pod \"e2e-inf-b-b4971a-openshift-default-5dc48765-8rn6x\" (UID: \"febb83d9-3a57-40a5-b972-c45611313469\") " pod="openshift-ingress/e2e-inf-b-b4971a-openshift-default-5dc48765-8rn6x" Nov 03 01:29:17.792688 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:29:17.792670 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-l8qt8\" (UniqueName: \"kubernetes.io/projected/febb83d9-3a57-40a5-b972-c45611313469-kube-api-access-l8qt8\") pod \"e2e-inf-b-b4971a-openshift-default-5dc48765-8rn6x\" (UID: \"febb83d9-3a57-40a5-b972-c45611313469\") " pod="openshift-ingress/e2e-inf-b-b4971a-openshift-default-5dc48765-8rn6x" Nov 03 01:29:17.877342 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:29:17.877310 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:29:17.966532 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:29:17.966343 4939 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-b4971a-openshift-default-5dc48765-8rn6x" Nov 03 01:29:18.039267 ip-10-0-140-107 kubenswrapper[4939]: E1103 01:29:18.039226 4939 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-sthjh" containerName="extractor" Nov 03 01:29:18.278822 ip-10-0-140-107 kubenswrapper[4939]: W1103 01:29:18.278794 4939 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfebb83d9_3a57_40a5_b972_c45611313469.slice/crio-f254e9b7e85741988da4d32451be4ed9c00a5a7fe299b3c47998f7b2bca3df15 WatchSource:0}: Error finding container f254e9b7e85741988da4d32451be4ed9c00a5a7fe299b3c47998f7b2bca3df15: Status 404 returned error can't find the container with id f254e9b7e85741988da4d32451be4ed9c00a5a7fe299b3c47998f7b2bca3df15 Nov 03 01:29:18.279830 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:29:18.279807 4939 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-b4971a-openshift-default-5dc48765-8rn6x"] Nov 03 01:29:18.282108 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:29:18.282073 4939 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 01:29:18.282207 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:29:18.282140 4939 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 01:29:18.282207 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:29:18.282170 4939 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 01:29:18.516573 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:29:18.516531 4939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-b4971a-openshift-default-5dc48765-8rn6x" event={"ID":"febb83d9-3a57-40a5-b972-c45611313469","Type":"ContainerStarted","Data":"0f6ed1cdff446ef3dae0ff58786613404bafe9eef128ca4b36a6bd7611f2630a"} Nov 03 01:29:18.517072 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:29:18.516596 4939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-b4971a-openshift-default-5dc48765-8rn6x" event={"ID":"febb83d9-3a57-40a5-b972-c45611313469","Type":"ContainerStarted","Data":"f254e9b7e85741988da4d32451be4ed9c00a5a7fe299b3c47998f7b2bca3df15"} Nov 03 01:29:18.518353 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:29:18.518327 4939 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-4b17ac-openshift-default-6b75f999cf-k5vtr" Nov 03 01:29:18.518494 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:29:18.518365 4939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-4b17ac-openshift-default-6b75f999cf-k5vtr" event={"ID":"5e9f1346-a093-44e2-8f84-712cafcdffdf","Type":"ContainerDied","Data":"44a4568a3ab0b5804ad38a3c32806288d733e3ab0fbab838d45b550ffd3b8166"} Nov 03 01:29:18.518494 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:29:18.518424 4939 scope.go:122] "RemoveContainer" containerID="ec5a9665db5301394447fd6076ac3a6dbc1e2fb1aeebd87a07d86f21d2873ddb" Nov 03 01:29:18.521042 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:29:18.521018 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:29:18.521895 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:29:18.521679 4939 generic.go:356] "Generic (PLEG): container finished" podID="7064128b-8764-4a6a-a4bc-e42b083f944f" containerID="8717adbb605eabe2ae8a56d66659a3a6a308f1d9ae5fa997071b413addebbc6c" exitCode=0 Nov 03 01:29:18.521895 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:29:18.521781 4939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-6fec8a-openshift-default-5cc5f9b957-p9mtp" event={"ID":"7064128b-8764-4a6a-a4bc-e42b083f944f","Type":"ContainerDied","Data":"8717adbb605eabe2ae8a56d66659a3a6a308f1d9ae5fa997071b413addebbc6c"} Nov 03 01:29:18.522318 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:29:18.522298 4939 scope.go:122] "RemoveContainer" containerID="8717adbb605eabe2ae8a56d66659a3a6a308f1d9ae5fa997071b413addebbc6c" Nov 03 01:29:18.524441 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:29:18.524408 4939 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 01:29:18.524565 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:29:18.524492 4939 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 01:29:18.524565 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:29:18.524538 4939 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 01:29:18.541266 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:29:18.541217 4939 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-b-b4971a-openshift-default-5dc48765-8rn6x" podStartSLOduration=17.541201819 podStartE2EDuration="17.541201819s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 01:29:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 01:29:18.539444982 +0000 UTC m=+2906.803779942" watchObservedRunningTime="2026-11-03 01:29:18.541201819 +0000 UTC m=+2906.805536766" Nov 03 01:29:18.594733 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:29:18.594705 4939 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-4b17ac-openshift-default-6b75f999cf-k5vtr"] Nov 03 01:29:18.610192 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:29:18.610161 4939 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-a-4b17ac-openshift-default-6b75f999cf-k5vtr"] Nov 03 01:29:18.640246 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:29:18.640217 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:29:18.967931 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:29:18.967023 4939 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-b4971a-openshift-default-5dc48765-8rn6x" Nov 03 01:29:19.537705 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:29:19.537621 4939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-6fec8a-openshift-default-5cc5f9b957-p9mtp" event={"ID":"7064128b-8764-4a6a-a4bc-e42b083f944f","Type":"ContainerStarted","Data":"3f3e26e3315a129208f02c6f66b93eb29a3f64905d3c5a0d0ebb33176a2cd2a5"} Nov 03 01:29:19.565839 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:29:19.565807 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:29:19.772653 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:29:19.771981 4939 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-6fec8a-openshift-default-5cc5f9b957-p9mtp" Nov 03 01:29:19.772874 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:29:19.772740 4939 patch_prober.go:29] interesting pod/e2e-inf-a-6fec8a-openshift-default-5cc5f9b957-p9mtp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.50:15021/healthz/ready\": dial tcp 10.132.1.50:15021: connect: connection refused" start-of-body= Nov 03 01:29:19.772874 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:29:19.772808 4939 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-6fec8a-openshift-default-5cc5f9b957-p9mtp" podUID="7064128b-8764-4a6a-a4bc-e42b083f944f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.50:15021/healthz/ready\": dial tcp 10.132.1.50:15021: connect: connection refused" Nov 03 01:29:19.889274 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:29:19.889235 4939 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5e9f1346-a093-44e2-8f84-712cafcdffdf" path="/var/lib/kubelet/pods/5e9f1346-a093-44e2-8f84-712cafcdffdf/volumes" Nov 03 01:29:19.983008 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:29:19.982967 4939 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-b4971a-openshift-default-5dc48765-8rn6x" Nov 03 01:29:20.569007 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:29:20.568971 4939 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-b4971a-openshift-default-5dc48765-8rn6x" Nov 03 01:29:20.575031 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:29:20.575002 4939 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-b4971a-openshift-default-5dc48765-8rn6x" Nov 03 01:29:20.626081 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:29:20.626048 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:29:20.790933 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:29:20.778090 4939 patch_prober.go:29] interesting pod/e2e-inf-a-6fec8a-openshift-default-5cc5f9b957-p9mtp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.50:15021/healthz/ready\": dial tcp 10.132.1.50:15021: connect: connection refused" start-of-body= Nov 03 01:29:20.790933 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:29:20.778158 4939 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-6fec8a-openshift-default-5cc5f9b957-p9mtp" podUID="7064128b-8764-4a6a-a4bc-e42b083f944f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.50:15021/healthz/ready\": dial tcp 10.132.1.50:15021: connect: connection refused" Nov 03 01:29:20.937940 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:29:20.932377 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:29:21.081341 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:29:21.081307 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:29:21.772044 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:29:21.771602 4939 patch_prober.go:29] interesting pod/e2e-inf-a-6fec8a-openshift-default-5cc5f9b957-p9mtp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.50:15021/healthz/ready\": dial tcp 10.132.1.50:15021: connect: connection refused" start-of-body= Nov 03 01:29:21.772044 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:29:21.771678 4939 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-6fec8a-openshift-default-5cc5f9b957-p9mtp" podUID="7064128b-8764-4a6a-a4bc-e42b083f944f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.50:15021/healthz/ready\": dial tcp 10.132.1.50:15021: connect: connection refused" Nov 03 01:29:21.872291 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:29:21.872251 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:29:22.244635 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:29:22.244598 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:29:22.288156 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:29:22.288119 4939 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-6fec8a-openshift-default-5cc5f9b957-p9mtp" Nov 03 01:29:22.301034 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:29:22.301002 4939 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-b4971a-6c58f96447-rjn8w"] Nov 03 01:29:22.305508 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:29:22.305482 4939 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-b4971a-6c58f96447-rjn8w" Nov 03 01:29:22.320878 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:29:22.320855 4939 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-b-b4971a\"" Nov 03 01:29:22.321086 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:29:22.320887 4939 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-b-b4971a-dockercfg-lzk7n\"" Nov 03 01:29:22.327942 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:29:22.327898 4939 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-b4971a-6c58f96447-rjn8w"] Nov 03 01:29:22.343951 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:29:22.342501 4939 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-5dab6a-57cf987799-9svqc" Nov 03 01:29:22.370174 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:29:22.370140 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:29:22.376474 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:29:22.375260 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6712b32f-f3bd-4917-9e5e-101cf604625c-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-b4971a-6c58f96447-rjn8w\" (UID: \"6712b32f-f3bd-4917-9e5e-101cf604625c\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-b4971a-6c58f96447-rjn8w" Nov 03 01:29:22.376474 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:29:22.375304 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g6sbs\" (UniqueName: \"kubernetes.io/projected/6712b32f-f3bd-4917-9e5e-101cf604625c-kube-api-access-g6sbs\") pod \"payload-pre-processing-e2e-inf-b-b4971a-6c58f96447-rjn8w\" (UID: \"6712b32f-f3bd-4917-9e5e-101cf604625c\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-b4971a-6c58f96447-rjn8w" Nov 03 01:29:22.477639 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:29:22.477587 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6712b32f-f3bd-4917-9e5e-101cf604625c-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-b4971a-6c58f96447-rjn8w\" (UID: \"6712b32f-f3bd-4917-9e5e-101cf604625c\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-b4971a-6c58f96447-rjn8w" Nov 03 01:29:22.477842 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:29:22.477665 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g6sbs\" (UniqueName: \"kubernetes.io/projected/6712b32f-f3bd-4917-9e5e-101cf604625c-kube-api-access-g6sbs\") pod \"payload-pre-processing-e2e-inf-b-b4971a-6c58f96447-rjn8w\" (UID: \"6712b32f-f3bd-4917-9e5e-101cf604625c\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-b4971a-6c58f96447-rjn8w" Nov 03 01:29:22.478992 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:29:22.478965 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6712b32f-f3bd-4917-9e5e-101cf604625c-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-b4971a-6c58f96447-rjn8w\" (UID: \"6712b32f-f3bd-4917-9e5e-101cf604625c\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-b4971a-6c58f96447-rjn8w" Nov 03 01:29:22.486132 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:29:22.486110 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-g6sbs\" (UniqueName: \"kubernetes.io/projected/6712b32f-f3bd-4917-9e5e-101cf604625c-kube-api-access-g6sbs\") pod \"payload-pre-processing-e2e-inf-b-b4971a-6c58f96447-rjn8w\" (UID: \"6712b32f-f3bd-4917-9e5e-101cf604625c\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-b4971a-6c58f96447-rjn8w" Nov 03 01:29:22.612993 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:29:22.605141 4939 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-a-ee618c/test-model-ee618c-kserve-844f5b79dc-f6kl7" Nov 03 01:29:22.638934 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:29:22.638558 4939 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-b4971a-6c58f96447-rjn8w" Nov 03 01:29:22.848943 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:29:22.848268 4939 patch_prober.go:29] interesting pod/e2e-inf-a-6fec8a-openshift-default-5cc5f9b957-p9mtp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.50:15021/healthz/ready\": dial tcp 10.132.1.50:15021: connect: connection refused" start-of-body= Nov 03 01:29:22.848943 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:29:22.848346 4939 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-6fec8a-openshift-default-5cc5f9b957-p9mtp" podUID="7064128b-8764-4a6a-a4bc-e42b083f944f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.50:15021/healthz/ready\": dial tcp 10.132.1.50:15021: connect: connection refused" Nov 03 01:29:23.115763 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:29:22.960965 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:29:23.115763 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:29:23.012499 4939 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-b-5bfc51/test-model-5bfc51-kserve-5c5676fcf9-tssvf"] Nov 03 01:29:23.133938 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:29:23.118171 4939 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-5bfc51/test-model-5bfc51-kserve-5c5676fcf9-tssvf" Nov 03 01:29:23.133938 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:29:23.127654 4939 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-5bfc51\"/\"openshift-service-ca.crt\"" Nov 03 01:29:23.133938 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:29:23.128455 4939 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-5bfc51\"/\"test-model-5bfc51-kserve-self-signed-certs\"" Nov 03 01:29:23.133938 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:29:23.128584 4939 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-5bfc51\"/\"default-dockercfg-b6n8g\"" Nov 03 01:29:23.133938 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:29:23.128677 4939 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-5bfc51\"/\"kube-root-ca.crt\"" Nov 03 01:29:23.289932 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:29:23.280677 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d0ed2957-3656-49bd-8aff-27ab5483d367-dshm\") pod \"test-model-5bfc51-kserve-5c5676fcf9-tssvf\" (UID: \"d0ed2957-3656-49bd-8aff-27ab5483d367\") " pod="ai-tenant-e2e-inf-b-5bfc51/test-model-5bfc51-kserve-5c5676fcf9-tssvf" Nov 03 01:29:23.289932 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:29:23.280882 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d0ed2957-3656-49bd-8aff-27ab5483d367-tmp-dir\") pod \"test-model-5bfc51-kserve-5c5676fcf9-tssvf\" (UID: \"d0ed2957-3656-49bd-8aff-27ab5483d367\") " pod="ai-tenant-e2e-inf-b-5bfc51/test-model-5bfc51-kserve-5c5676fcf9-tssvf" Nov 03 01:29:23.289932 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:29:23.282983 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s9smz\" (UniqueName: \"kubernetes.io/projected/d0ed2957-3656-49bd-8aff-27ab5483d367-kube-api-access-s9smz\") pod \"test-model-5bfc51-kserve-5c5676fcf9-tssvf\" (UID: \"d0ed2957-3656-49bd-8aff-27ab5483d367\") " pod="ai-tenant-e2e-inf-b-5bfc51/test-model-5bfc51-kserve-5c5676fcf9-tssvf" Nov 03 01:29:23.289932 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:29:23.283242 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d0ed2957-3656-49bd-8aff-27ab5483d367-tls-certs\") pod \"test-model-5bfc51-kserve-5c5676fcf9-tssvf\" (UID: \"d0ed2957-3656-49bd-8aff-27ab5483d367\") " pod="ai-tenant-e2e-inf-b-5bfc51/test-model-5bfc51-kserve-5c5676fcf9-tssvf" Nov 03 01:29:23.289932 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:29:23.283286 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d0ed2957-3656-49bd-8aff-27ab5483d367-home\") pod \"test-model-5bfc51-kserve-5c5676fcf9-tssvf\" (UID: \"d0ed2957-3656-49bd-8aff-27ab5483d367\") " pod="ai-tenant-e2e-inf-b-5bfc51/test-model-5bfc51-kserve-5c5676fcf9-tssvf" Nov 03 01:29:23.289932 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:29:23.283360 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d0ed2957-3656-49bd-8aff-27ab5483d367-model-cache\") pod \"test-model-5bfc51-kserve-5c5676fcf9-tssvf\" (UID: \"d0ed2957-3656-49bd-8aff-27ab5483d367\") " pod="ai-tenant-e2e-inf-b-5bfc51/test-model-5bfc51-kserve-5c5676fcf9-tssvf" Nov 03 01:29:23.289932 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:29:23.288072 4939 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-5bfc51/test-model-5bfc51-kserve-5c5676fcf9-tssvf"] Nov 03 01:29:23.299934 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:29:23.296970 4939 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f1a0eff5-kpnxp"] Nov 03 01:29:23.305966 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:29:23.304828 4939 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f1a0eff5-kpnxp"] Nov 03 01:29:23.308957 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:29:23.306149 4939 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-a-ee618c/test-model-ee618c-kserve-844f5b79dc-f6kl7" Nov 03 01:29:23.331506 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:29:23.331475 4939 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-a-ee618c/test-model-ee618c-kserve-844f5b79dc-f6kl7" Nov 03 01:29:23.386823 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:29:23.385225 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d0ed2957-3656-49bd-8aff-27ab5483d367-dshm\") pod \"test-model-5bfc51-kserve-5c5676fcf9-tssvf\" (UID: \"d0ed2957-3656-49bd-8aff-27ab5483d367\") " pod="ai-tenant-e2e-inf-b-5bfc51/test-model-5bfc51-kserve-5c5676fcf9-tssvf" Nov 03 01:29:23.386823 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:29:23.385420 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d0ed2957-3656-49bd-8aff-27ab5483d367-tmp-dir\") pod \"test-model-5bfc51-kserve-5c5676fcf9-tssvf\" (UID: \"d0ed2957-3656-49bd-8aff-27ab5483d367\") " pod="ai-tenant-e2e-inf-b-5bfc51/test-model-5bfc51-kserve-5c5676fcf9-tssvf" Nov 03 01:29:23.386823 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:29:23.385543 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s9smz\" (UniqueName: \"kubernetes.io/projected/d0ed2957-3656-49bd-8aff-27ab5483d367-kube-api-access-s9smz\") pod \"test-model-5bfc51-kserve-5c5676fcf9-tssvf\" (UID: \"d0ed2957-3656-49bd-8aff-27ab5483d367\") " pod="ai-tenant-e2e-inf-b-5bfc51/test-model-5bfc51-kserve-5c5676fcf9-tssvf" Nov 03 01:29:23.386823 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:29:23.385631 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d0ed2957-3656-49bd-8aff-27ab5483d367-tls-certs\") pod \"test-model-5bfc51-kserve-5c5676fcf9-tssvf\" (UID: \"d0ed2957-3656-49bd-8aff-27ab5483d367\") " pod="ai-tenant-e2e-inf-b-5bfc51/test-model-5bfc51-kserve-5c5676fcf9-tssvf" Nov 03 01:29:23.386823 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:29:23.385666 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d0ed2957-3656-49bd-8aff-27ab5483d367-home\") pod \"test-model-5bfc51-kserve-5c5676fcf9-tssvf\" (UID: \"d0ed2957-3656-49bd-8aff-27ab5483d367\") " pod="ai-tenant-e2e-inf-b-5bfc51/test-model-5bfc51-kserve-5c5676fcf9-tssvf" Nov 03 01:29:23.386823 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:29:23.385731 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d0ed2957-3656-49bd-8aff-27ab5483d367-model-cache\") pod \"test-model-5bfc51-kserve-5c5676fcf9-tssvf\" (UID: \"d0ed2957-3656-49bd-8aff-27ab5483d367\") " pod="ai-tenant-e2e-inf-b-5bfc51/test-model-5bfc51-kserve-5c5676fcf9-tssvf" Nov 03 01:29:23.386823 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:29:23.386317 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d0ed2957-3656-49bd-8aff-27ab5483d367-model-cache\") pod \"test-model-5bfc51-kserve-5c5676fcf9-tssvf\" (UID: \"d0ed2957-3656-49bd-8aff-27ab5483d367\") " pod="ai-tenant-e2e-inf-b-5bfc51/test-model-5bfc51-kserve-5c5676fcf9-tssvf" Nov 03 01:29:23.386823 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:29:23.386708 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d0ed2957-3656-49bd-8aff-27ab5483d367-tmp-dir\") pod \"test-model-5bfc51-kserve-5c5676fcf9-tssvf\" (UID: \"d0ed2957-3656-49bd-8aff-27ab5483d367\") " pod="ai-tenant-e2e-inf-b-5bfc51/test-model-5bfc51-kserve-5c5676fcf9-tssvf" Nov 03 01:29:23.386823 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:29:23.386743 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d0ed2957-3656-49bd-8aff-27ab5483d367-home\") pod \"test-model-5bfc51-kserve-5c5676fcf9-tssvf\" (UID: \"d0ed2957-3656-49bd-8aff-27ab5483d367\") " pod="ai-tenant-e2e-inf-b-5bfc51/test-model-5bfc51-kserve-5c5676fcf9-tssvf" Nov 03 01:29:23.391199 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:29:23.391145 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d0ed2957-3656-49bd-8aff-27ab5483d367-tls-certs\") pod \"test-model-5bfc51-kserve-5c5676fcf9-tssvf\" (UID: \"d0ed2957-3656-49bd-8aff-27ab5483d367\") " pod="ai-tenant-e2e-inf-b-5bfc51/test-model-5bfc51-kserve-5c5676fcf9-tssvf" Nov 03 01:29:23.391199 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:29:23.391188 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d0ed2957-3656-49bd-8aff-27ab5483d367-dshm\") pod \"test-model-5bfc51-kserve-5c5676fcf9-tssvf\" (UID: \"d0ed2957-3656-49bd-8aff-27ab5483d367\") " pod="ai-tenant-e2e-inf-b-5bfc51/test-model-5bfc51-kserve-5c5676fcf9-tssvf" Nov 03 01:29:23.393544 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:29:23.393514 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-s9smz\" (UniqueName: \"kubernetes.io/projected/d0ed2957-3656-49bd-8aff-27ab5483d367-kube-api-access-s9smz\") pod \"test-model-5bfc51-kserve-5c5676fcf9-tssvf\" (UID: \"d0ed2957-3656-49bd-8aff-27ab5483d367\") " pod="ai-tenant-e2e-inf-b-5bfc51/test-model-5bfc51-kserve-5c5676fcf9-tssvf" Nov 03 01:29:23.495039 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:29:23.494969 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:29:23.601405 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:29:23.601368 4939 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-5bfc51/test-model-5bfc51-kserve-5c5676fcf9-tssvf" Nov 03 01:29:23.638230 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:29:23.637350 4939 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-b4971a-6c58f96447-rjn8w"] Nov 03 01:29:23.776136 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:29:23.776102 4939 patch_prober.go:29] interesting pod/e2e-inf-a-6fec8a-openshift-default-5cc5f9b957-p9mtp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.50:15021/healthz/ready\": dial tcp 10.132.1.50:15021: connect: connection refused" start-of-body= Nov 03 01:29:23.776308 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:29:23.776160 4939 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-6fec8a-openshift-default-5cc5f9b957-p9mtp" podUID="7064128b-8764-4a6a-a4bc-e42b083f944f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.50:15021/healthz/ready\": dial tcp 10.132.1.50:15021: connect: connection refused" Nov 03 01:29:23.881256 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:29:23.881223 4939 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="752af256-3554-41a6-a9bb-e7548419c6b2" path="/var/lib/kubelet/pods/752af256-3554-41a6-a9bb-e7548419c6b2/volumes" Nov 03 01:29:23.989998 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:29:23.989963 4939 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-5bfc51/test-model-5bfc51-kserve-5c5676fcf9-tssvf"] Nov 03 01:29:24.004002 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:29:24.003673 4939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-b4971a-6c58f96447-rjn8w" event={"ID":"6712b32f-f3bd-4917-9e5e-101cf604625c","Type":"ContainerStarted","Data":"384febeac8280063b68f71ab034887b1d84a8e149d013ded2756f3066aeecdf7"} Nov 03 01:29:24.004395 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:29:24.004365 4939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-b4971a-6c58f96447-rjn8w" event={"ID":"6712b32f-f3bd-4917-9e5e-101cf604625c","Type":"ContainerStarted","Data":"c4d3c828c1922f9760a0735709482befc9025e8f4e39367e18532e4200e124b6"} Nov 03 01:29:24.004556 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:29:24.004541 4939 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-b4971a-6c58f96447-rjn8w" Nov 03 01:29:24.006031 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:29:24.006003 4939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-5bfc51/test-model-5bfc51-kserve-5c5676fcf9-tssvf" event={"ID":"d0ed2957-3656-49bd-8aff-27ab5483d367","Type":"ContainerStarted","Data":"fd0b1baa31a6313e319394cb5fa7fe1605ffeec4a070fc1b36c8e8ea92acf639"} Nov 03 01:29:24.207266 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:29:24.207169 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:29:24.771401 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:29:24.771358 4939 patch_prober.go:29] interesting pod/e2e-inf-a-6fec8a-openshift-default-5cc5f9b957-p9mtp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.50:15021/healthz/ready\": dial tcp 10.132.1.50:15021: connect: connection refused" start-of-body= Nov 03 01:29:24.771630 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:29:24.771421 4939 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-6fec8a-openshift-default-5cc5f9b957-p9mtp" podUID="7064128b-8764-4a6a-a4bc-e42b083f944f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.50:15021/healthz/ready\": dial tcp 10.132.1.50:15021: connect: connection refused" Nov 03 01:29:25.028976 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:29:25.027476 4939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-5bfc51/test-model-5bfc51-kserve-5c5676fcf9-tssvf" event={"ID":"d0ed2957-3656-49bd-8aff-27ab5483d367","Type":"ContainerStarted","Data":"5b9557d64e7a642db2cbbdfd10203121fe0ec76b67473542658a9b2472b54879"} Nov 03 01:29:25.071933 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:29:25.066872 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:29:25.072965 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:29:25.072707 4939 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-inf-b-5bfc51/test-model-5bfc51-kserve-5c5676fcf9-tssvf" podStartSLOduration=3.072687669 podStartE2EDuration="3.072687669s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 01:29:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 01:29:25.044367574 +0000 UTC m=+2913.308702513" watchObservedRunningTime="2026-11-03 01:29:25.072687669 +0000 UTC m=+2913.337022615" Nov 03 01:29:25.074591 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:29:25.074549 4939 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-b4971a-6c58f96447-rjn8w" podStartSLOduration=3.074538952 podStartE2EDuration="3.074538952s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 01:29:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 01:29:24.022334335 +0000 UTC m=+2912.286669281" watchObservedRunningTime="2026-11-03 01:29:25.074538952 +0000 UTC m=+2913.338873898" Nov 03 01:29:25.602422 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:29:25.602386 4939 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-b-5bfc51/test-model-5bfc51-kserve-5c5676fcf9-tssvf" Nov 03 01:29:25.715106 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:29:25.715071 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:29:25.779934 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:29:25.777189 4939 patch_prober.go:29] interesting pod/e2e-inf-a-6fec8a-openshift-default-5cc5f9b957-p9mtp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.50:15021/healthz/ready\": dial tcp 10.132.1.50:15021: connect: connection refused" start-of-body= Nov 03 01:29:25.779934 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:29:25.777254 4939 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-6fec8a-openshift-default-5cc5f9b957-p9mtp" podUID="7064128b-8764-4a6a-a4bc-e42b083f944f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.50:15021/healthz/ready\": dial tcp 10.132.1.50:15021: connect: connection refused" Nov 03 01:29:26.462310 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:29:26.462276 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:29:26.776211 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:29:26.776120 4939 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-6fec8a-openshift-default-5cc5f9b957-p9mtp" Nov 03 01:29:26.777382 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:29:26.777358 4939 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-6fec8a-openshift-default-5cc5f9b957-p9mtp" Nov 03 01:29:27.275275 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:29:27.275246 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:29:27.875436 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:29:27.875393 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:29:28.355469 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:29:28.355435 4939 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-4pw7v"] Nov 03 01:29:28.488158 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:29:28.488099 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:29:29.287130 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:29:29.287096 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:29:30.141802 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:29:30.141765 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:29:30.805390 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:29:30.805353 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:29:31.497707 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:29:31.497672 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:29:32.101248 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:29:32.101215 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:29:32.771358 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:29:32.771313 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:29:32.916935 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:29:32.916234 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:29:33.617943 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:29:33.616862 4939 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-b-5bfc51/test-model-5bfc51-kserve-5c5676fcf9-tssvf" Nov 03 01:29:33.758383 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:29:33.758346 4939 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-b-5bfc51/test-model-5bfc51-kserve-5c5676fcf9-tssvf" Nov 03 01:29:33.882594 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:29:33.882510 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:29:34.265628 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:29:34.265539 4939 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-b-5bfc51/test-model-5bfc51-kserve-5c5676fcf9-tssvf" Nov 03 01:29:34.555705 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:29:34.555623 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:29:34.793989 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:29:34.793936 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:29:35.029429 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:29:35.029398 4939 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-b4971a-6c58f96447-rjn8w" Nov 03 01:29:35.179383 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:29:35.179351 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:29:35.878536 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:29:35.878502 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:29:36.032619 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:29:36.032582 4939 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7df33a0c-mdc5q"] Nov 03 01:29:36.037339 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:29:36.037304 4939 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7df33a0c-mdc5q"] Nov 03 01:29:36.482221 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:29:36.482187 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:29:37.167665 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:29:37.167632 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:29:37.808639 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:29:37.808603 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:29:37.915008 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:29:37.914964 4939 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="59795f2c-592e-4820-892a-bfdd35951594" path="/var/lib/kubelet/pods/59795f2c-592e-4820-892a-bfdd35951594/volumes" Nov 03 01:29:38.700084 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:29:38.700041 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:29:39.467368 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:29:39.467335 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:29:40.304092 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:29:40.304060 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:29:41.021797 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:29:41.021759 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:29:41.304215 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:29:41.304130 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:29:41.716029 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:29:41.715999 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:29:42.375614 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:29:42.375583 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:29:42.973151 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:29:42.973100 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:29:43.698440 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:29:43.698409 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:29:44.288678 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:29:44.288630 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:29:44.414560 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:29:44.414428 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:29:44.608353 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:29:44.608317 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:29:44.717480 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:29:44.717423 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:29:44.886792 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:29:44.886696 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:29:45.324417 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:29:45.324280 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:29:46.003029 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:29:46.002989 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:29:46.619333 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:29:46.619298 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:29:47.303080 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:29:47.303045 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:29:48.033534 ip-10-0-140-107 kubenswrapper[4939]: E1103 01:29:48.033495 4939 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-sthjh" containerName="extractor" Nov 03 01:29:48.057957 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:29:48.057886 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:29:48.670782 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:29:48.670745 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:29:49.314375 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:29:49.314339 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:29:50.129850 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:29:50.129812 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:29:50.884189 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:29:50.884145 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:29:51.080736 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:29:51.080696 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:29:51.697755 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:29:51.697720 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:29:52.304966 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:29:52.304935 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:29:52.885072 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:29:52.885034 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:29:53.592356 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:29:53.592321 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:29:54.181639 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:29:54.181600 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:29:54.798701 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:29:54.798667 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:29:55.482976 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:29:55.482943 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:29:56.120110 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:29:56.120072 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:29:56.692799 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:29:56.692767 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:29:57.387788 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:29:57.387756 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:29:57.974692 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:29:57.974659 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:29:58.540498 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:29:58.540468 4939 scope.go:122] "RemoveContainer" containerID="b27f1483f0bc66d6493c8775c31ddae558650ba330be12a2401c08afbcffa1da" Nov 03 01:29:58.606003 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:29:58.605974 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:29:59.290391 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:29:59.290356 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:29:59.879866 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:29:59.879824 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:30:00.141143 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:30:00.141053 4939 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-556c14-29894490-cqqp5"] Nov 03 01:30:00.148726 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:30:00.148694 4939 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-2f9a3a-29894490-58pnm"] Nov 03 01:30:00.148926 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:30:00.148813 4939 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-556c14-29894490-cqqp5" Nov 03 01:30:00.158746 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:30:00.158715 4939 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-6q4xp\"" Nov 03 01:30:00.166318 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:30:00.166291 4939 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-821f5a-29894490-9h7dh"] Nov 03 01:30:00.166709 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:30:00.166683 4939 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-2f9a3a-29894490-58pnm" Nov 03 01:30:00.172587 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:30:00.172563 4939 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-7c759a-29894490-blnms"] Nov 03 01:30:00.172820 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:30:00.172795 4939 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-821f5a-29894490-9h7dh" Nov 03 01:30:00.181726 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:30:00.181698 4939 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-2f9a3a-29894490-58pnm"] Nov 03 01:30:00.181871 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:30:00.181744 4939 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-556c14-29894490-cqqp5"] Nov 03 01:30:00.181871 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:30:00.181773 4939 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-821f5a-29894490-9h7dh"] Nov 03 01:30:00.181871 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:30:00.181861 4939 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-7c759a-29894490-blnms" Nov 03 01:30:00.191679 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:30:00.191656 4939 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-7c759a-29894490-blnms"] Nov 03 01:30:00.222015 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:30:00.221987 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-scpqm\" (UniqueName: \"kubernetes.io/projected/90e0cb11-fc6e-479e-8951-a36552f42532-kube-api-access-scpqm\") pod \"maas-api-key-cleanup-e2e-shared-b-w3-556c14-29894490-cqqp5\" (UID: \"90e0cb11-fc6e-479e-8951-a36552f42532\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-556c14-29894490-cqqp5" Nov 03 01:30:00.222144 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:30:00.222033 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h6kf8\" (UniqueName: \"kubernetes.io/projected/af3de10a-1341-4219-878f-9c73279482f0-kube-api-access-h6kf8\") pod \"maas-api-key-cleanup-e2e-extmod-a-821f5a-29894490-9h7dh\" (UID: \"af3de10a-1341-4219-878f-9c73279482f0\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-821f5a-29894490-9h7dh" Nov 03 01:30:00.222231 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:30:00.222218 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e505287a-7d6d-46f9-9a7c-2de4605546f2-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w0-2f9a3a-29894490-58pnm\" (UID: \"e505287a-7d6d-46f9-9a7c-2de4605546f2\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-2f9a3a-29894490-58pnm" Nov 03 01:30:00.222374 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:30:00.222353 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/af3de10a-1341-4219-878f-9c73279482f0-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-a-821f5a-29894490-9h7dh\" (UID: \"af3de10a-1341-4219-878f-9c73279482f0\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-821f5a-29894490-9h7dh" Nov 03 01:30:00.222460 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:30:00.222443 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-52lmq\" (UniqueName: \"kubernetes.io/projected/e505287a-7d6d-46f9-9a7c-2de4605546f2-kube-api-access-52lmq\") pod \"maas-api-key-cleanup-e2e-worker-w0-2f9a3a-29894490-58pnm\" (UID: \"e505287a-7d6d-46f9-9a7c-2de4605546f2\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-2f9a3a-29894490-58pnm" Nov 03 01:30:00.222564 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:30:00.222551 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/90e0cb11-fc6e-479e-8951-a36552f42532-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w3-556c14-29894490-cqqp5\" (UID: \"90e0cb11-fc6e-479e-8951-a36552f42532\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-556c14-29894490-cqqp5" Nov 03 01:30:00.324624 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:30:00.324582 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e505287a-7d6d-46f9-9a7c-2de4605546f2-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w0-2f9a3a-29894490-58pnm\" (UID: \"e505287a-7d6d-46f9-9a7c-2de4605546f2\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-2f9a3a-29894490-58pnm" Nov 03 01:30:00.325042 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:30:00.324688 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/38ef32d5-3c39-4b9d-a82a-774f3ba09ed2-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-a-7c759a-29894490-blnms\" (UID: \"38ef32d5-3c39-4b9d-a82a-774f3ba09ed2\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-7c759a-29894490-blnms" Nov 03 01:30:00.325042 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:30:00.324733 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tkbgg\" (UniqueName: \"kubernetes.io/projected/38ef32d5-3c39-4b9d-a82a-774f3ba09ed2-kube-api-access-tkbgg\") pod \"maas-api-key-cleanup-e2e-extmod-a-7c759a-29894490-blnms\" (UID: \"38ef32d5-3c39-4b9d-a82a-774f3ba09ed2\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-7c759a-29894490-blnms" Nov 03 01:30:00.325042 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:30:00.324807 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/af3de10a-1341-4219-878f-9c73279482f0-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-a-821f5a-29894490-9h7dh\" (UID: \"af3de10a-1341-4219-878f-9c73279482f0\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-821f5a-29894490-9h7dh" Nov 03 01:30:00.325042 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:30:00.324854 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-52lmq\" (UniqueName: \"kubernetes.io/projected/e505287a-7d6d-46f9-9a7c-2de4605546f2-kube-api-access-52lmq\") pod \"maas-api-key-cleanup-e2e-worker-w0-2f9a3a-29894490-58pnm\" (UID: \"e505287a-7d6d-46f9-9a7c-2de4605546f2\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-2f9a3a-29894490-58pnm" Nov 03 01:30:00.325042 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:30:00.324930 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/90e0cb11-fc6e-479e-8951-a36552f42532-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w3-556c14-29894490-cqqp5\" (UID: \"90e0cb11-fc6e-479e-8951-a36552f42532\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-556c14-29894490-cqqp5" Nov 03 01:30:00.325491 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:30:00.325097 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-scpqm\" (UniqueName: \"kubernetes.io/projected/90e0cb11-fc6e-479e-8951-a36552f42532-kube-api-access-scpqm\") pod \"maas-api-key-cleanup-e2e-shared-b-w3-556c14-29894490-cqqp5\" (UID: \"90e0cb11-fc6e-479e-8951-a36552f42532\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-556c14-29894490-cqqp5" Nov 03 01:30:00.325491 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:30:00.325133 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h6kf8\" (UniqueName: \"kubernetes.io/projected/af3de10a-1341-4219-878f-9c73279482f0-kube-api-access-h6kf8\") pod \"maas-api-key-cleanup-e2e-extmod-a-821f5a-29894490-9h7dh\" (UID: \"af3de10a-1341-4219-878f-9c73279482f0\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-821f5a-29894490-9h7dh" Nov 03 01:30:00.325491 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:30:00.325296 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e505287a-7d6d-46f9-9a7c-2de4605546f2-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w0-2f9a3a-29894490-58pnm\" (UID: \"e505287a-7d6d-46f9-9a7c-2de4605546f2\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-2f9a3a-29894490-58pnm" Nov 03 01:30:00.325662 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:30:00.325554 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/af3de10a-1341-4219-878f-9c73279482f0-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-a-821f5a-29894490-9h7dh\" (UID: \"af3de10a-1341-4219-878f-9c73279482f0\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-821f5a-29894490-9h7dh" Nov 03 01:30:00.325951 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:30:00.325874 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/90e0cb11-fc6e-479e-8951-a36552f42532-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w3-556c14-29894490-cqqp5\" (UID: \"90e0cb11-fc6e-479e-8951-a36552f42532\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-556c14-29894490-cqqp5" Nov 03 01:30:00.339837 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:30:00.339813 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-52lmq\" (UniqueName: \"kubernetes.io/projected/e505287a-7d6d-46f9-9a7c-2de4605546f2-kube-api-access-52lmq\") pod \"maas-api-key-cleanup-e2e-worker-w0-2f9a3a-29894490-58pnm\" (UID: \"e505287a-7d6d-46f9-9a7c-2de4605546f2\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-2f9a3a-29894490-58pnm" Nov 03 01:30:00.340256 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:30:00.340234 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-scpqm\" (UniqueName: \"kubernetes.io/projected/90e0cb11-fc6e-479e-8951-a36552f42532-kube-api-access-scpqm\") pod \"maas-api-key-cleanup-e2e-shared-b-w3-556c14-29894490-cqqp5\" (UID: \"90e0cb11-fc6e-479e-8951-a36552f42532\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-556c14-29894490-cqqp5" Nov 03 01:30:00.341817 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:30:00.341792 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-h6kf8\" (UniqueName: \"kubernetes.io/projected/af3de10a-1341-4219-878f-9c73279482f0-kube-api-access-h6kf8\") pod \"maas-api-key-cleanup-e2e-extmod-a-821f5a-29894490-9h7dh\" (UID: \"af3de10a-1341-4219-878f-9c73279482f0\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-821f5a-29894490-9h7dh" Nov 03 01:30:00.427031 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:30:00.426949 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/38ef32d5-3c39-4b9d-a82a-774f3ba09ed2-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-a-7c759a-29894490-blnms\" (UID: \"38ef32d5-3c39-4b9d-a82a-774f3ba09ed2\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-7c759a-29894490-blnms" Nov 03 01:30:00.427031 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:30:00.426997 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tkbgg\" (UniqueName: \"kubernetes.io/projected/38ef32d5-3c39-4b9d-a82a-774f3ba09ed2-kube-api-access-tkbgg\") pod \"maas-api-key-cleanup-e2e-extmod-a-7c759a-29894490-blnms\" (UID: \"38ef32d5-3c39-4b9d-a82a-774f3ba09ed2\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-7c759a-29894490-blnms" Nov 03 01:30:00.427570 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:30:00.427545 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/38ef32d5-3c39-4b9d-a82a-774f3ba09ed2-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-a-7c759a-29894490-blnms\" (UID: \"38ef32d5-3c39-4b9d-a82a-774f3ba09ed2\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-7c759a-29894490-blnms" Nov 03 01:30:00.435284 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:30:00.435252 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tkbgg\" (UniqueName: \"kubernetes.io/projected/38ef32d5-3c39-4b9d-a82a-774f3ba09ed2-kube-api-access-tkbgg\") pod \"maas-api-key-cleanup-e2e-extmod-a-7c759a-29894490-blnms\" (UID: \"38ef32d5-3c39-4b9d-a82a-774f3ba09ed2\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-7c759a-29894490-blnms" Nov 03 01:30:00.472105 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:30:00.472071 4939 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-556c14-29894490-cqqp5" Nov 03 01:30:00.491839 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:30:00.491811 4939 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-2f9a3a-29894490-58pnm" Nov 03 01:30:00.510825 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:30:00.510794 4939 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-821f5a-29894490-9h7dh" Nov 03 01:30:00.517358 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:30:00.517333 4939 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-7c759a-29894490-blnms" Nov 03 01:30:00.685484 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:30:00.685381 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:30:00.838742 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:30:00.838707 4939 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-556c14-29894490-cqqp5"] Nov 03 01:30:01.005749 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:30:01.005720 4939 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-2f9a3a-29894490-58pnm"] Nov 03 01:30:01.035651 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:30:01.035557 4939 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-821f5a-29894490-9h7dh"] Nov 03 01:30:01.052144 ip-10-0-140-107 kubenswrapper[4939]: W1103 01:30:01.052110 4939 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode505287a_7d6d_46f9_9a7c_2de4605546f2.slice/crio-224d19f6f904de239ad0542385317ab79941c0e5a4242c3befc335b7eb6450fa WatchSource:0}: Error finding container 224d19f6f904de239ad0542385317ab79941c0e5a4242c3befc335b7eb6450fa: Status 404 returned error can't find the container with id 224d19f6f904de239ad0542385317ab79941c0e5a4242c3befc335b7eb6450fa Nov 03 01:30:01.054384 ip-10-0-140-107 kubenswrapper[4939]: W1103 01:30:01.054354 4939 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaf3de10a_1341_4219_878f_9c73279482f0.slice/crio-f5d47e7d7b840899448eb742c0b04ea4eda3b6839d122ba74b2b5fade4cd20e3 WatchSource:0}: Error finding container f5d47e7d7b840899448eb742c0b04ea4eda3b6839d122ba74b2b5fade4cd20e3: Status 404 returned error can't find the container with id f5d47e7d7b840899448eb742c0b04ea4eda3b6839d122ba74b2b5fade4cd20e3 Nov 03 01:30:01.055103 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:30:01.054995 4939 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-7c759a-29894490-blnms"] Nov 03 01:30:01.056282 ip-10-0-140-107 kubenswrapper[4939]: W1103 01:30:01.056097 4939 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod38ef32d5_3c39_4b9d_a82a_774f3ba09ed2.slice/crio-55efaf696ac483ea3b61b726263ad7bca226ac9652f768369146dc8098f42ce8 WatchSource:0}: Error finding container 55efaf696ac483ea3b61b726263ad7bca226ac9652f768369146dc8098f42ce8: Status 404 returned error can't find the container with id 55efaf696ac483ea3b61b726263ad7bca226ac9652f768369146dc8098f42ce8 Nov 03 01:30:01.533679 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:30:01.533638 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:30:01.644936 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:30:01.644832 4939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-7c759a-29894490-blnms" event={"ID":"38ef32d5-3c39-4b9d-a82a-774f3ba09ed2","Type":"ContainerStarted","Data":"55efaf696ac483ea3b61b726263ad7bca226ac9652f768369146dc8098f42ce8"} Nov 03 01:30:01.646751 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:30:01.646710 4939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-556c14-29894490-cqqp5" event={"ID":"90e0cb11-fc6e-479e-8951-a36552f42532","Type":"ContainerStarted","Data":"c40cec6dac5a7110bd066983de7924b3d94e8c47fb5b6f55c6b6fdb32402a7d4"} Nov 03 01:30:01.646901 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:30:01.646764 4939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-556c14-29894490-cqqp5" event={"ID":"90e0cb11-fc6e-479e-8951-a36552f42532","Type":"ContainerStarted","Data":"9d2fdf63e4e1e22dc91f5ea819da0eac6849f43419fd27b057c08ef759cc15c5"} Nov 03 01:30:01.649796 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:30:01.649762 4939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-821f5a-29894490-9h7dh" event={"ID":"af3de10a-1341-4219-878f-9c73279482f0","Type":"ContainerStarted","Data":"f5d47e7d7b840899448eb742c0b04ea4eda3b6839d122ba74b2b5fade4cd20e3"} Nov 03 01:30:01.651410 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:30:01.651383 4939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-2f9a3a-29894490-58pnm" event={"ID":"e505287a-7d6d-46f9-9a7c-2de4605546f2","Type":"ContainerStarted","Data":"224d19f6f904de239ad0542385317ab79941c0e5a4242c3befc335b7eb6450fa"} Nov 03 01:30:01.663433 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:30:01.663381 4939 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-556c14-29894490-cqqp5" podStartSLOduration=1.6633652749999999 podStartE2EDuration="1.663365275s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 01:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 01:30:01.662365819 +0000 UTC m=+2949.926700764" watchObservedRunningTime="2026-11-03 01:30:01.663365275 +0000 UTC m=+2949.927700221" Nov 03 01:30:02.176740 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:30:02.176712 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:30:02.662787 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:30:02.662701 4939 generic.go:356] "Generic (PLEG): container finished" podID="38ef32d5-3c39-4b9d-a82a-774f3ba09ed2" containerID="243797e55ae21326b0c91f2093af1152ab7525c6f612c40060166d35add8e300" exitCode=60 Nov 03 01:30:02.663249 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:30:02.662783 4939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-7c759a-29894490-blnms" event={"ID":"38ef32d5-3c39-4b9d-a82a-774f3ba09ed2","Type":"ContainerDied","Data":"243797e55ae21326b0c91f2093af1152ab7525c6f612c40060166d35add8e300"} Nov 03 01:30:02.663249 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:30:02.663100 4939 scope.go:122] "RemoveContainer" containerID="243797e55ae21326b0c91f2093af1152ab7525c6f612c40060166d35add8e300" Nov 03 01:30:02.664835 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:30:02.664797 4939 generic.go:356] "Generic (PLEG): container finished" podID="90e0cb11-fc6e-479e-8951-a36552f42532" containerID="c40cec6dac5a7110bd066983de7924b3d94e8c47fb5b6f55c6b6fdb32402a7d4" exitCode=60 Nov 03 01:30:02.664975 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:30:02.664847 4939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-556c14-29894490-cqqp5" event={"ID":"90e0cb11-fc6e-479e-8951-a36552f42532","Type":"ContainerDied","Data":"c40cec6dac5a7110bd066983de7924b3d94e8c47fb5b6f55c6b6fdb32402a7d4"} Nov 03 01:30:02.665533 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:30:02.665149 4939 scope.go:122] "RemoveContainer" containerID="c40cec6dac5a7110bd066983de7924b3d94e8c47fb5b6f55c6b6fdb32402a7d4" Nov 03 01:30:02.667171 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:30:02.667148 4939 generic.go:356] "Generic (PLEG): container finished" podID="af3de10a-1341-4219-878f-9c73279482f0" containerID="e0091e676b49052c7b8f00cf360e7f9ec13f830aa0e809b9c28b056e4441b96e" exitCode=60 Nov 03 01:30:02.667311 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:30:02.667176 4939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-821f5a-29894490-9h7dh" event={"ID":"af3de10a-1341-4219-878f-9c73279482f0","Type":"ContainerDied","Data":"e0091e676b49052c7b8f00cf360e7f9ec13f830aa0e809b9c28b056e4441b96e"} Nov 03 01:30:02.667527 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:30:02.667472 4939 scope.go:122] "RemoveContainer" containerID="e0091e676b49052c7b8f00cf360e7f9ec13f830aa0e809b9c28b056e4441b96e" Nov 03 01:30:02.675666 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:30:02.675539 4939 generic.go:356] "Generic (PLEG): container finished" podID="e505287a-7d6d-46f9-9a7c-2de4605546f2" containerID="414c644ecb33ccfbc6f1690fed678c8308be1ffc69c2dd16ef77d5793c25d6e0" exitCode=60 Nov 03 01:30:02.675666 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:30:02.675576 4939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-2f9a3a-29894490-58pnm" event={"ID":"e505287a-7d6d-46f9-9a7c-2de4605546f2","Type":"ContainerDied","Data":"414c644ecb33ccfbc6f1690fed678c8308be1ffc69c2dd16ef77d5793c25d6e0"} Nov 03 01:30:02.675898 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:30:02.675881 4939 scope.go:122] "RemoveContainer" containerID="414c644ecb33ccfbc6f1690fed678c8308be1ffc69c2dd16ef77d5793c25d6e0" Nov 03 01:30:02.889451 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:30:02.889413 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:30:03.456353 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:30:03.456317 4939 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-4pw7v"] Nov 03 01:30:03.501852 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:30:03.501819 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:30:03.742965 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:30:03.740549 4939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-2f9a3a-29894490-58pnm" event={"ID":"e505287a-7d6d-46f9-9a7c-2de4605546f2","Type":"ContainerStarted","Data":"d5dd503093c4b544a97bf71709a93c848823c02ef2ac93580bbaf59d6b4c581b"} Nov 03 01:30:03.774212 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:30:03.761525 4939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-7c759a-29894490-blnms" event={"ID":"38ef32d5-3c39-4b9d-a82a-774f3ba09ed2","Type":"ContainerStarted","Data":"bba0bf1fc174bf1d43516f0f4e80130c3b8116f94631af10f680dbe4b74ed2ef"} Nov 03 01:30:03.786936 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:30:03.780969 4939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-556c14-29894490-cqqp5" event={"ID":"90e0cb11-fc6e-479e-8951-a36552f42532","Type":"ContainerStarted","Data":"be4e5351082de069612028fe73e0c145838bcd5fb6e2a321e51ee29c87186c1c"} Nov 03 01:30:03.795965 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:30:03.793140 4939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-821f5a-29894490-9h7dh" event={"ID":"af3de10a-1341-4219-878f-9c73279482f0","Type":"ContainerStarted","Data":"59dab4ba045c3051aa73a75c78e9507f121a39640510b6f33033dabd84d44a43"} Nov 03 01:30:03.949952 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:30:03.947443 4939 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-821f5a-29894490-9h7dh" podStartSLOduration=3.947424404 podStartE2EDuration="3.947424404s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 01:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 01:30:03.857405585 +0000 UTC m=+2952.121740535" watchObservedRunningTime="2026-11-03 01:30:03.947424404 +0000 UTC m=+2952.211759350" Nov 03 01:30:03.959745 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:30:03.958957 4939 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-2f9a3a-29894490-58pnm" podStartSLOduration=3.958943893 podStartE2EDuration="3.958943893s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 01:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 01:30:03.804169201 +0000 UTC m=+2952.068504147" watchObservedRunningTime="2026-11-03 01:30:03.958943893 +0000 UTC m=+2952.223278839" Nov 03 01:30:04.017274 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:30:04.015302 4939 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-7c759a-29894490-blnms" podStartSLOduration=4.015284277 podStartE2EDuration="4.015284277s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 01:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 01:30:03.971015907 +0000 UTC m=+2952.235350853" watchObservedRunningTime="2026-11-03 01:30:04.015284277 +0000 UTC m=+2952.279619252" Nov 03 01:30:04.392319 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:30:04.392285 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:30:04.842932 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:30:04.840422 4939 generic.go:356] "Generic (PLEG): container finished" podID="38ef32d5-3c39-4b9d-a82a-774f3ba09ed2" containerID="bba0bf1fc174bf1d43516f0f4e80130c3b8116f94631af10f680dbe4b74ed2ef" exitCode=60 Nov 03 01:30:04.842932 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:30:04.840520 4939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-7c759a-29894490-blnms" event={"ID":"38ef32d5-3c39-4b9d-a82a-774f3ba09ed2","Type":"ContainerDied","Data":"bba0bf1fc174bf1d43516f0f4e80130c3b8116f94631af10f680dbe4b74ed2ef"} Nov 03 01:30:04.842932 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:30:04.840606 4939 scope.go:122] "RemoveContainer" containerID="243797e55ae21326b0c91f2093af1152ab7525c6f612c40060166d35add8e300" Nov 03 01:30:04.842932 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:30:04.841181 4939 scope.go:122] "RemoveContainer" containerID="bba0bf1fc174bf1d43516f0f4e80130c3b8116f94631af10f680dbe4b74ed2ef" Nov 03 01:30:04.842932 ip-10-0-140-107 kubenswrapper[4939]: E1103 01:30:04.841678 4939 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-a-7c759a-29894490-blnms_odh-ai-gateway-infra(38ef32d5-3c39-4b9d-a82a-774f3ba09ed2)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-7c759a-29894490-blnms" podUID="38ef32d5-3c39-4b9d-a82a-774f3ba09ed2" Nov 03 01:30:04.871973 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:30:04.868628 4939 generic.go:356] "Generic (PLEG): container finished" podID="90e0cb11-fc6e-479e-8951-a36552f42532" containerID="be4e5351082de069612028fe73e0c145838bcd5fb6e2a321e51ee29c87186c1c" exitCode=60 Nov 03 01:30:04.872289 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:30:04.872262 4939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-556c14-29894490-cqqp5" event={"ID":"90e0cb11-fc6e-479e-8951-a36552f42532","Type":"ContainerDied","Data":"be4e5351082de069612028fe73e0c145838bcd5fb6e2a321e51ee29c87186c1c"} Nov 03 01:30:04.884041 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:30:04.884017 4939 scope.go:122] "RemoveContainer" containerID="be4e5351082de069612028fe73e0c145838bcd5fb6e2a321e51ee29c87186c1c" Nov 03 01:30:04.887275 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:30:04.887248 4939 generic.go:356] "Generic (PLEG): container finished" podID="af3de10a-1341-4219-878f-9c73279482f0" containerID="59dab4ba045c3051aa73a75c78e9507f121a39640510b6f33033dabd84d44a43" exitCode=60 Nov 03 01:30:04.904476 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:30:04.904433 4939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-821f5a-29894490-9h7dh" event={"ID":"af3de10a-1341-4219-878f-9c73279482f0","Type":"ContainerDied","Data":"59dab4ba045c3051aa73a75c78e9507f121a39640510b6f33033dabd84d44a43"} Nov 03 01:30:04.904950 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:30:04.904903 4939 scope.go:122] "RemoveContainer" containerID="59dab4ba045c3051aa73a75c78e9507f121a39640510b6f33033dabd84d44a43" Nov 03 01:30:04.905089 ip-10-0-140-107 kubenswrapper[4939]: E1103 01:30:04.886933 4939 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w3-556c14-29894490-cqqp5_odh-ai-gateway-infra(90e0cb11-fc6e-479e-8951-a36552f42532)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-556c14-29894490-cqqp5" podUID="90e0cb11-fc6e-479e-8951-a36552f42532" Nov 03 01:30:04.905449 ip-10-0-140-107 kubenswrapper[4939]: E1103 01:30:04.905381 4939 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-a-821f5a-29894490-9h7dh_odh-ai-gateway-infra(af3de10a-1341-4219-878f-9c73279482f0)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-821f5a-29894490-9h7dh" podUID="af3de10a-1341-4219-878f-9c73279482f0" Nov 03 01:30:05.027385 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:30:05.007979 4939 scope.go:122] "RemoveContainer" containerID="c40cec6dac5a7110bd066983de7924b3d94e8c47fb5b6f55c6b6fdb32402a7d4" Nov 03 01:30:05.070633 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:30:05.070591 4939 generic.go:356] "Generic (PLEG): container finished" podID="e505287a-7d6d-46f9-9a7c-2de4605546f2" containerID="d5dd503093c4b544a97bf71709a93c848823c02ef2ac93580bbaf59d6b4c581b" exitCode=60 Nov 03 01:30:05.071966 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:30:05.071126 4939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-2f9a3a-29894490-58pnm" event={"ID":"e505287a-7d6d-46f9-9a7c-2de4605546f2","Type":"ContainerDied","Data":"d5dd503093c4b544a97bf71709a93c848823c02ef2ac93580bbaf59d6b4c581b"} Nov 03 01:30:05.133845 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:30:05.071189 4939 scope.go:122] "RemoveContainer" containerID="d5dd503093c4b544a97bf71709a93c848823c02ef2ac93580bbaf59d6b4c581b" Nov 03 01:30:05.164947 ip-10-0-140-107 kubenswrapper[4939]: E1103 01:30:05.149443 4939 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w0-2f9a3a-29894490-58pnm_odh-ai-gateway-infra(e505287a-7d6d-46f9-9a7c-2de4605546f2)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-2f9a3a-29894490-58pnm" podUID="e505287a-7d6d-46f9-9a7c-2de4605546f2" Nov 03 01:30:05.305313 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:30:05.301285 4939 scope.go:122] "RemoveContainer" containerID="e0091e676b49052c7b8f00cf360e7f9ec13f830aa0e809b9c28b056e4441b96e" Nov 03 01:30:05.744947 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:30:05.741367 4939 scope.go:122] "RemoveContainer" containerID="414c644ecb33ccfbc6f1690fed678c8308be1ffc69c2dd16ef77d5793c25d6e0" Nov 03 01:30:05.916084 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:30:05.915992 4939 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-2f9a3a/distinct-w0-2f9a3a-kserve-555565cfcf-jtcr9" podUID="ac0ece06-8718-463f-915f-5315988d9711" containerName="main" probeResult="failure" output="Get \"https://10.132.0.76:8000/ready\": context deadline exceeded" Nov 03 01:30:05.950340 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:30:05.932683 4939 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-2f9a3a/simulator-w0-2f9a3a-kserve-9596bbb8f-2trgz" podUID="207e2207-fb6c-4076-95df-72e95671c261" containerName="main" probeResult="failure" output="Get \"https://10.132.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 01:30:06.054116 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:30:06.054072 4939 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-f7680d/premium-w2-f7680d-kserve-76565b8cf9-hqbbt" podUID="4712e708-18a2-4a13-a12c-a5669397b2c3" containerName="main" probeResult="failure" output="Get \"https://10.132.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 01:30:06.092933 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:30:06.090698 4939 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-83755a/distinct-2-w3-83755a-kserve-6cf66b995d-2l47b" podUID="c3b05e88-5d5b-4d03-b28d-832f22e3c868" containerName="main" probeResult="failure" output="Get \"https://10.132.0.72:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 01:30:06.113987 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:30:06.094301 4939 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-f7680d/distinct-w2-f7680d-kserve-756f874b98-qnpv8" podUID="836753fc-f9ef-4f61-b3fe-e5b92b7ee27b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.70:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 01:30:06.141038 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:30:06.138983 4939 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-88bbf7/simulator-w1-88bbf7-kserve-5fbb669f8f-42hgd" podUID="83b38883-2e7a-444d-aa68-52be4a25ae66" containerName="main" probeResult="failure" output="Get \"https://10.132.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 01:30:06.141557 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:30:06.139039 4939 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-83755a/distinct-2-w3-83755a-kserve-6cf66b995d-2l47b" podUID="c3b05e88-5d5b-4d03-b28d-832f22e3c868" containerName="main" probeResult="failure" output="Get \"https://10.132.0.72:8000/ready\": context deadline exceeded" Nov 03 01:30:06.174363 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:30:06.162369 4939 scope.go:122] "RemoveContainer" containerID="bba0bf1fc174bf1d43516f0f4e80130c3b8116f94631af10f680dbe4b74ed2ef" Nov 03 01:30:06.174363 ip-10-0-140-107 kubenswrapper[4939]: E1103 01:30:06.169026 4939 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-a-7c759a-29894490-blnms_odh-ai-gateway-infra(38ef32d5-3c39-4b9d-a82a-774f3ba09ed2)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-7c759a-29894490-blnms" podUID="38ef32d5-3c39-4b9d-a82a-774f3ba09ed2" Nov 03 01:30:06.202949 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:30:06.197187 4939 scope.go:122] "RemoveContainer" containerID="be4e5351082de069612028fe73e0c145838bcd5fb6e2a321e51ee29c87186c1c" Nov 03 01:30:06.202949 ip-10-0-140-107 kubenswrapper[4939]: E1103 01:30:06.197696 4939 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w3-556c14-29894490-cqqp5_odh-ai-gateway-infra(90e0cb11-fc6e-479e-8951-a36552f42532)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-556c14-29894490-cqqp5" podUID="90e0cb11-fc6e-479e-8951-a36552f42532" Nov 03 01:30:06.219148 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:30:06.219098 4939 scope.go:122] "RemoveContainer" containerID="59dab4ba045c3051aa73a75c78e9507f121a39640510b6f33033dabd84d44a43" Nov 03 01:30:06.224831 ip-10-0-140-107 kubenswrapper[4939]: E1103 01:30:06.219822 4939 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-a-821f5a-29894490-9h7dh_odh-ai-gateway-infra(af3de10a-1341-4219-878f-9c73279482f0)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-821f5a-29894490-9h7dh" podUID="af3de10a-1341-4219-878f-9c73279482f0" Nov 03 01:30:06.247759 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:30:06.247713 4939 scope.go:122] "RemoveContainer" containerID="d5dd503093c4b544a97bf71709a93c848823c02ef2ac93580bbaf59d6b4c581b" Nov 03 01:30:06.248845 ip-10-0-140-107 kubenswrapper[4939]: E1103 01:30:06.248817 4939 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w0-2f9a3a-29894490-58pnm_odh-ai-gateway-infra(e505287a-7d6d-46f9-9a7c-2de4605546f2)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-2f9a3a-29894490-58pnm" podUID="e505287a-7d6d-46f9-9a7c-2de4605546f2" Nov 03 01:30:06.271943 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:30:06.271675 4939 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-b-5bfc51/test-model-5bfc51-kserve-5c5676fcf9-tssvf" podUID="d0ed2957-3656-49bd-8aff-27ab5483d367" containerName="main" probeResult="failure" output="Get \"https://10.132.1.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 01:30:06.328932 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:30:06.311995 4939 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-a-ee618c/test-model-ee618c-kserve-844f5b79dc-f6kl7" podUID="abab7b3b-f57d-4abe-be59-295e4ab8a769" containerName="main" probeResult="failure" output="Get \"https://10.132.1.58:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 01:30:06.591240 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:30:06.590374 4939 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-88bbf7/distinct-2-w1-88bbf7-kserve-5457cdcdbd-l6tfd" podUID="a397931b-faa9-4c2e-8290-491350546663" containerName="main" probeResult="failure" output="Get \"https://10.132.0.85:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 01:30:06.896987 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:30:06.853390 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:30:07.025553 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:30:06.934007 4939 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-f7680d/distinct-2-w2-f7680d-kserve-7d7c4564bf-cbv2c" podUID="2de18086-761b-4773-b9fc-15ddc4a220fc" containerName="main" probeResult="failure" output="Get \"https://10.132.0.75:8000/ready\": context deadline exceeded" Nov 03 01:30:07.025553 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:30:06.965252 4939 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-2f9a3a/simulator-w0-2f9a3a-kserve-9596bbb8f-2trgz" podUID="207e2207-fb6c-4076-95df-72e95671c261" containerName="main" probeResult="failure" output="Get \"https://10.132.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 01:30:07.025553 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:30:06.965540 4939 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-2f9a3a/distinct-w0-2f9a3a-kserve-555565cfcf-jtcr9" podUID="ac0ece06-8718-463f-915f-5315988d9711" containerName="main" probeResult="failure" output="Get \"https://10.132.0.76:8000/ready\": context deadline exceeded" Nov 03 01:30:07.025553 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:30:06.966014 4939 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-2f9a3a/simulator-w0-2f9a3a-kserve-9596bbb8f-2trgz" Nov 03 01:30:07.025553 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:30:06.966074 4939 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-2f9a3a/distinct-w0-2f9a3a-kserve-555565cfcf-jtcr9" Nov 03 01:30:07.078930 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:30:07.055111 4939 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-f7680d/premium-w2-f7680d-kserve-76565b8cf9-hqbbt" podUID="4712e708-18a2-4a13-a12c-a5669397b2c3" containerName="main" probeResult="failure" output="Get \"https://10.132.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 01:30:07.078930 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:30:07.059155 4939 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-f7680d/premium-w2-f7680d-kserve-76565b8cf9-hqbbt" Nov 03 01:30:07.149339 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:30:07.140638 4939 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-f7680d/distinct-w2-f7680d-kserve-756f874b98-qnpv8" podUID="836753fc-f9ef-4f61-b3fe-e5b92b7ee27b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.70:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 01:30:07.178939 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:30:07.174845 4939 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-f7680d/distinct-w2-f7680d-kserve-756f874b98-qnpv8" Nov 03 01:30:07.232314 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:30:07.232266 4939 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-88bbf7/simulator-w1-88bbf7-kserve-5fbb669f8f-42hgd" podUID="83b38883-2e7a-444d-aa68-52be4a25ae66" containerName="main" probeResult="failure" output="Get \"https://10.132.0.62:8000/ready\": context deadline exceeded" Nov 03 01:30:07.233404 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:30:07.233372 4939 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-88bbf7/simulator-w1-88bbf7-kserve-5fbb669f8f-42hgd" Nov 03 01:30:07.291612 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:30:07.291569 4939 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-b-5bfc51/test-model-5bfc51-kserve-5c5676fcf9-tssvf" podUID="d0ed2957-3656-49bd-8aff-27ab5483d367" containerName="main" probeResult="failure" output="Get \"https://10.132.1.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 01:30:07.325934 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:30:07.300145 4939 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-b-5bfc51/test-model-5bfc51-kserve-5c5676fcf9-tssvf" Nov 03 01:30:07.325934 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:30:07.323021 4939 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-a-ee618c/test-model-ee618c-kserve-844f5b79dc-f6kl7" podUID="abab7b3b-f57d-4abe-be59-295e4ab8a769" containerName="main" probeResult="failure" output="Get \"https://10.132.1.58:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 01:30:07.353929 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:30:07.327195 4939 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-a-ee618c/test-model-ee618c-kserve-844f5b79dc-f6kl7" Nov 03 01:30:07.597189 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:30:07.597147 4939 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-88bbf7/distinct-2-w1-88bbf7-kserve-5457cdcdbd-l6tfd" podUID="a397931b-faa9-4c2e-8290-491350546663" containerName="main" probeResult="failure" output="Get \"https://10.132.0.85:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 01:30:07.725517 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:30:07.725481 4939 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-88bbf7/distinct-2-w1-88bbf7-kserve-5457cdcdbd-l6tfd" Nov 03 01:30:07.725731 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:30:07.725551 4939 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-a-ee618c/test-model-ee618c-kserve-844f5b79dc-f6kl7" Nov 03 01:30:07.990987 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:30:07.965418 4939 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-2f9a3a/simulator-w0-2f9a3a-kserve-9596bbb8f-2trgz" podUID="207e2207-fb6c-4076-95df-72e95671c261" containerName="main" probeResult="failure" output="Get \"https://10.132.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 01:30:08.097952 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:30:07.991447 4939 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-f7680d/distinct-2-w2-f7680d-kserve-7d7c4564bf-cbv2c" podUID="2de18086-761b-4773-b9fc-15ddc4a220fc" containerName="main" probeResult="failure" output="Get \"https://10.132.0.75:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 01:30:08.097952 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:30:07.994010 4939 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-2f9a3a/distinct-w0-2f9a3a-kserve-555565cfcf-jtcr9" podUID="ac0ece06-8718-463f-915f-5315988d9711" containerName="main" probeResult="failure" output="Get \"https://10.132.0.76:8000/ready\": context deadline exceeded" Nov 03 01:30:08.097952 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:30:08.021288 4939 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-f7680d/distinct-w2-f7680d-kserve-756f874b98-qnpv8" podUID="836753fc-f9ef-4f61-b3fe-e5b92b7ee27b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.70:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 01:30:08.097952 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:30:08.055295 4939 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-f7680d/premium-w2-f7680d-kserve-76565b8cf9-hqbbt" podUID="4712e708-18a2-4a13-a12c-a5669397b2c3" containerName="main" probeResult="failure" output="Get \"https://10.132.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 01:30:08.154939 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:30:08.154449 4939 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-f7680d/distinct-w2-f7680d-kserve-756f874b98-qnpv8" podUID="836753fc-f9ef-4f61-b3fe-e5b92b7ee27b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.70:8000/ready\": context deadline exceeded" Nov 03 01:30:08.267594 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:30:08.239108 4939 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-88bbf7/simulator-w1-88bbf7-kserve-5fbb669f8f-42hgd" podUID="83b38883-2e7a-444d-aa68-52be4a25ae66" containerName="main" probeResult="failure" output="Get \"https://10.132.0.62:8000/ready\": context deadline exceeded" Nov 03 01:30:08.268572 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:30:08.268523 4939 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-88bbf7/simulator-w1-88bbf7-kserve-5fbb669f8f-42hgd" podUID="83b38883-2e7a-444d-aa68-52be4a25ae66" containerName="main" probeResult="failure" output="Get \"https://10.132.0.62:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 01:30:08.348894 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:30:08.348849 4939 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-b-5bfc51/test-model-5bfc51-kserve-5c5676fcf9-tssvf" podUID="d0ed2957-3656-49bd-8aff-27ab5483d367" containerName="main" probeResult="failure" output="Get \"https://10.132.1.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 01:30:08.793048 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:30:08.787196 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:30:08.793048 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:30:08.792478 4939 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-88bbf7/distinct-2-w1-88bbf7-kserve-5457cdcdbd-l6tfd" podUID="a397931b-faa9-4c2e-8290-491350546663" containerName="main" probeResult="failure" output="Get \"https://10.132.0.85:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 01:30:08.833237 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:30:08.833202 4939 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-f7680d/distinct-2-w2-f7680d-kserve-7d7c4564bf-cbv2c" Nov 03 01:30:08.833526 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:30:08.833507 4939 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-b-5bfc51/test-model-5bfc51-kserve-5c5676fcf9-tssvf" Nov 03 01:30:08.833677 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:30:08.833663 4939 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-f7680d/premium-w2-f7680d-kserve-76565b8cf9-hqbbt" Nov 03 01:30:08.833812 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:30:08.833800 4939 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-88bbf7/simulator-w1-88bbf7-kserve-5fbb669f8f-42hgd" Nov 03 01:30:08.833960 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:30:08.833947 4939 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-f7680d/distinct-2-w2-f7680d-kserve-7d7c4564bf-cbv2c" Nov 03 01:30:08.834102 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:30:08.834089 4939 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-f7680d/distinct-w2-f7680d-kserve-756f874b98-qnpv8" Nov 03 01:30:08.834229 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:30:08.834217 4939 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-2f9a3a/distinct-w0-2f9a3a-kserve-555565cfcf-jtcr9" Nov 03 01:30:08.834352 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:30:08.834342 4939 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-2f9a3a/simulator-w0-2f9a3a-kserve-9596bbb8f-2trgz" Nov 03 01:30:08.849256 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:30:08.849229 4939 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-88bbf7/distinct-2-w1-88bbf7-kserve-5457cdcdbd-l6tfd" Nov 03 01:30:09.400217 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:30:09.400178 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:30:10.122631 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:30:10.122597 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:30:11.004696 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:30:11.004660 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:30:11.874073 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:30:11.874018 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:30:12.492871 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:30:12.492834 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:30:13.166359 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:30:13.166323 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:30:13.979999 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:30:13.979963 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:30:14.765058 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:30:14.765026 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:30:14.866474 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:30:14.866436 4939 scope.go:122] "RemoveContainer" containerID="59dab4ba045c3051aa73a75c78e9507f121a39640510b6f33033dabd84d44a43" Nov 03 01:30:15.606346 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:30:15.606295 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:30:15.963181 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:30:15.963147 4939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-821f5a-29894490-9h7dh" event={"ID":"af3de10a-1341-4219-878f-9c73279482f0","Type":"ContainerStarted","Data":"2636254d1297090734d479deb8d7cde7e4e84115180d33b5a8c550d0a300124d"} Nov 03 01:30:15.963696 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:30:15.963674 4939 scope.go:122] "RemoveContainer" containerID="2636254d1297090734d479deb8d7cde7e4e84115180d33b5a8c550d0a300124d" Nov 03 01:30:15.964120 ip-10-0-140-107 kubenswrapper[4939]: E1103 01:30:15.964096 4939 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-a-821f5a-29894490-9h7dh_odh-ai-gateway-infra(af3de10a-1341-4219-878f-9c73279482f0)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-821f5a-29894490-9h7dh" podUID="af3de10a-1341-4219-878f-9c73279482f0" Nov 03 01:30:16.335396 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:30:16.335367 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:30:16.862801 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:30:16.862771 4939 scope.go:122] "RemoveContainer" containerID="d5dd503093c4b544a97bf71709a93c848823c02ef2ac93580bbaf59d6b4c581b" Nov 03 01:30:16.966279 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:30:16.966246 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:30:16.974225 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:30:16.974195 4939 generic.go:356] "Generic (PLEG): container finished" podID="febb83d9-3a57-40a5-b972-c45611313469" containerID="0f6ed1cdff446ef3dae0ff58786613404bafe9eef128ca4b36a6bd7611f2630a" exitCode=0 Nov 03 01:30:16.974343 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:30:16.974263 4939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-b4971a-openshift-default-5dc48765-8rn6x" event={"ID":"febb83d9-3a57-40a5-b972-c45611313469","Type":"ContainerDied","Data":"0f6ed1cdff446ef3dae0ff58786613404bafe9eef128ca4b36a6bd7611f2630a"} Nov 03 01:30:16.974745 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:30:16.974723 4939 scope.go:122] "RemoveContainer" containerID="0f6ed1cdff446ef3dae0ff58786613404bafe9eef128ca4b36a6bd7611f2630a" Nov 03 01:30:16.976971 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:30:16.976897 4939 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 01:30:16.977062 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:30:16.977018 4939 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 01:30:16.977062 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:30:16.977051 4939 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 01:30:16.977603 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:30:16.977489 4939 generic.go:356] "Generic (PLEG): container finished" podID="af3de10a-1341-4219-878f-9c73279482f0" containerID="2636254d1297090734d479deb8d7cde7e4e84115180d33b5a8c550d0a300124d" exitCode=60 Nov 03 01:30:16.977603 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:30:16.977514 4939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-821f5a-29894490-9h7dh" event={"ID":"af3de10a-1341-4219-878f-9c73279482f0","Type":"ContainerDied","Data":"2636254d1297090734d479deb8d7cde7e4e84115180d33b5a8c550d0a300124d"} Nov 03 01:30:16.977603 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:30:16.977572 4939 scope.go:122] "RemoveContainer" containerID="59dab4ba045c3051aa73a75c78e9507f121a39640510b6f33033dabd84d44a43" Nov 03 01:30:16.977986 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:30:16.977968 4939 scope.go:122] "RemoveContainer" containerID="2636254d1297090734d479deb8d7cde7e4e84115180d33b5a8c550d0a300124d" Nov 03 01:30:16.978271 ip-10-0-140-107 kubenswrapper[4939]: E1103 01:30:16.978246 4939 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-a-821f5a-29894490-9h7dh_odh-ai-gateway-infra(af3de10a-1341-4219-878f-9c73279482f0)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-821f5a-29894490-9h7dh" podUID="af3de10a-1341-4219-878f-9c73279482f0" Nov 03 01:30:17.107399 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:30:17.107305 4939 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-821f5a-29894490-9h7dh"] Nov 03 01:30:17.818988 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:30:17.809655 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:30:17.871679 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:30:17.871641 4939 scope.go:122] "RemoveContainer" containerID="be4e5351082de069612028fe73e0c145838bcd5fb6e2a321e51ee29c87186c1c" Nov 03 01:30:17.873456 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:30:17.873430 4939 scope.go:122] "RemoveContainer" containerID="bba0bf1fc174bf1d43516f0f4e80130c3b8116f94631af10f680dbe4b74ed2ef" Nov 03 01:30:17.988789 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:30:17.988754 4939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-b4971a-openshift-default-5dc48765-8rn6x" event={"ID":"febb83d9-3a57-40a5-b972-c45611313469","Type":"ContainerStarted","Data":"9f8d2d055b7c0d16ceed10a149a9af6f53d1436f29e72f3eb22bc4949a225ea7"} Nov 03 01:30:17.992738 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:30:17.992710 4939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-2f9a3a-29894490-58pnm" event={"ID":"e505287a-7d6d-46f9-9a7c-2de4605546f2","Type":"ContainerStarted","Data":"f8139818738d11f7f7b00f7731d1d52b889ecd858cb30b73b83e6a3d89417bc0"} Nov 03 01:30:18.037436 ip-10-0-140-107 kubenswrapper[4939]: E1103 01:30:18.037396 4939 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-sthjh" containerName="extractor" Nov 03 01:30:18.318096 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:30:18.318067 4939 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-821f5a-29894490-9h7dh" Nov 03 01:30:18.392754 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:30:18.392668 4939 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-027c19-29894490-dwg89"] Nov 03 01:30:18.394225 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:30:18.394201 4939 cpu_manager.go:414] "containerMap: removing container" podUID="af3de10a-1341-4219-878f-9c73279482f0" containerName="cleanup" Nov 03 01:30:18.394328 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:30:18.394229 4939 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="af3de10a-1341-4219-878f-9c73279482f0" containerName="cleanup" Nov 03 01:30:18.394328 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:30:18.394288 4939 cpu_manager.go:414] "containerMap: removing container" podUID="af3de10a-1341-4219-878f-9c73279482f0" containerName="cleanup" Nov 03 01:30:18.394328 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:30:18.394296 4939 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="af3de10a-1341-4219-878f-9c73279482f0" containerName="cleanup" Nov 03 01:30:18.394328 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:30:18.394324 4939 cpu_manager.go:414] "containerMap: removing container" podUID="af3de10a-1341-4219-878f-9c73279482f0" containerName="cleanup" Nov 03 01:30:18.394508 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:30:18.394332 4939 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="af3de10a-1341-4219-878f-9c73279482f0" containerName="cleanup" Nov 03 01:30:18.394646 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:30:18.394627 4939 memory_manager.go:365] "RemoveStaleState removing state" podUID="af3de10a-1341-4219-878f-9c73279482f0" containerName="cleanup" Nov 03 01:30:18.394733 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:30:18.394649 4939 memory_manager.go:365] "RemoveStaleState removing state" podUID="af3de10a-1341-4219-878f-9c73279482f0" containerName="cleanup" Nov 03 01:30:18.394733 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:30:18.394674 4939 memory_manager.go:365] "RemoveStaleState removing state" podUID="af3de10a-1341-4219-878f-9c73279482f0" containerName="cleanup" Nov 03 01:30:18.399716 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:30:18.399694 4939 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-027c19-29894490-dwg89" Nov 03 01:30:18.402047 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:30:18.402023 4939 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-027c19-29894490-dwg89"] Nov 03 01:30:18.436711 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:30:18.436667 4939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/af3de10a-1341-4219-878f-9c73279482f0-kube-api-access-h6kf8\" (UniqueName: \"kubernetes.io/projected/af3de10a-1341-4219-878f-9c73279482f0-kube-api-access-h6kf8\") pod \"af3de10a-1341-4219-878f-9c73279482f0\" (UID: \"af3de10a-1341-4219-878f-9c73279482f0\") " Nov 03 01:30:18.437003 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:30:18.436969 4939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/af3de10a-1341-4219-878f-9c73279482f0-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/af3de10a-1341-4219-878f-9c73279482f0-maas-api-service-ca\") pod \"af3de10a-1341-4219-878f-9c73279482f0\" (UID: \"af3de10a-1341-4219-878f-9c73279482f0\") " Nov 03 01:30:18.437493 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:30:18.437447 4939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/af3de10a-1341-4219-878f-9c73279482f0-maas-api-service-ca" pod "af3de10a-1341-4219-878f-9c73279482f0" (UID: "af3de10a-1341-4219-878f-9c73279482f0"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 01:30:18.437650 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:30:18.437597 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2pbgz\" (UniqueName: \"kubernetes.io/projected/56595211-63c2-491f-9912-b681df440d0a-kube-api-access-2pbgz\") pod \"maas-api-key-cleanup-e2e-extmod-a-027c19-29894490-dwg89\" (UID: \"56595211-63c2-491f-9912-b681df440d0a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-027c19-29894490-dwg89" Nov 03 01:30:18.437776 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:30:18.437704 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/56595211-63c2-491f-9912-b681df440d0a-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-a-027c19-29894490-dwg89\" (UID: \"56595211-63c2-491f-9912-b681df440d0a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-027c19-29894490-dwg89" Nov 03 01:30:18.438769 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:30:18.438742 4939 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/af3de10a-1341-4219-878f-9c73279482f0-maas-api-service-ca\") on node \"ip-10-0-140-107.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:30:18.522292 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:30:18.522261 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:30:18.586962 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:30:18.571120 4939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/af3de10a-1341-4219-878f-9c73279482f0-kube-api-access-h6kf8" pod "af3de10a-1341-4219-878f-9c73279482f0" (UID: "af3de10a-1341-4219-878f-9c73279482f0"). InnerVolumeSpecName "kube-api-access-h6kf8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 01:30:18.587847 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:30:18.587806 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2pbgz\" (UniqueName: \"kubernetes.io/projected/56595211-63c2-491f-9912-b681df440d0a-kube-api-access-2pbgz\") pod \"maas-api-key-cleanup-e2e-extmod-a-027c19-29894490-dwg89\" (UID: \"56595211-63c2-491f-9912-b681df440d0a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-027c19-29894490-dwg89" Nov 03 01:30:18.588280 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:30:18.588243 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/56595211-63c2-491f-9912-b681df440d0a-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-a-027c19-29894490-dwg89\" (UID: \"56595211-63c2-491f-9912-b681df440d0a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-027c19-29894490-dwg89" Nov 03 01:30:18.589349 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:30:18.589317 4939 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h6kf8\" (UniqueName: \"kubernetes.io/projected/af3de10a-1341-4219-878f-9c73279482f0-kube-api-access-h6kf8\") on node \"ip-10-0-140-107.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:30:18.590345 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:30:18.590324 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/56595211-63c2-491f-9912-b681df440d0a-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-a-027c19-29894490-dwg89\" (UID: \"56595211-63c2-491f-9912-b681df440d0a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-027c19-29894490-dwg89" Nov 03 01:30:18.642168 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:30:18.642123 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2pbgz\" (UniqueName: \"kubernetes.io/projected/56595211-63c2-491f-9912-b681df440d0a-kube-api-access-2pbgz\") pod \"maas-api-key-cleanup-e2e-extmod-a-027c19-29894490-dwg89\" (UID: \"56595211-63c2-491f-9912-b681df440d0a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-027c19-29894490-dwg89" Nov 03 01:30:18.719151 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:30:18.719121 4939 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-027c19-29894490-dwg89" Nov 03 01:30:18.977294 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:30:18.977155 4939 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-b4971a-openshift-default-5dc48765-8rn6x" Nov 03 01:30:18.978288 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:30:18.978259 4939 patch_prober.go:29] interesting pod/e2e-inf-b-b4971a-openshift-default-5dc48765-8rn6x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.59:15021/healthz/ready\": dial tcp 10.132.1.59:15021: connect: connection refused" start-of-body= Nov 03 01:30:18.978435 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:30:18.978319 4939 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-b4971a-openshift-default-5dc48765-8rn6x" podUID="febb83d9-3a57-40a5-b972-c45611313469" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.59:15021/healthz/ready\": dial tcp 10.132.1.59:15021: connect: connection refused" Nov 03 01:30:19.008264 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:30:19.008229 4939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-7c759a-29894490-blnms" event={"ID":"38ef32d5-3c39-4b9d-a82a-774f3ba09ed2","Type":"ContainerStarted","Data":"82268704ad8287ecf308618e07e2ff486eadef4346d3b81b996ee3cad6bd52dc"} Nov 03 01:30:19.010463 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:30:19.010439 4939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-556c14-29894490-cqqp5" event={"ID":"90e0cb11-fc6e-479e-8951-a36552f42532","Type":"ContainerStarted","Data":"9d52d3830f93300f1edd12e10d813c583cba6fb3189245ceea58dc7a098a704a"} Nov 03 01:30:19.011955 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:30:19.011931 4939 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-821f5a-29894490-9h7dh" Nov 03 01:30:19.011955 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:30:19.011942 4939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-821f5a-29894490-9h7dh" event={"ID":"af3de10a-1341-4219-878f-9c73279482f0","Type":"ContainerDied","Data":"f5d47e7d7b840899448eb742c0b04ea4eda3b6839d122ba74b2b5fade4cd20e3"} Nov 03 01:30:19.012166 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:30:19.011993 4939 scope.go:122] "RemoveContainer" containerID="2636254d1297090734d479deb8d7cde7e4e84115180d33b5a8c550d0a300124d" Nov 03 01:30:19.014590 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:30:19.014568 4939 generic.go:356] "Generic (PLEG): container finished" podID="e505287a-7d6d-46f9-9a7c-2de4605546f2" containerID="f8139818738d11f7f7b00f7731d1d52b889ecd858cb30b73b83e6a3d89417bc0" exitCode=60 Nov 03 01:30:19.014722 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:30:19.014641 4939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-2f9a3a-29894490-58pnm" event={"ID":"e505287a-7d6d-46f9-9a7c-2de4605546f2","Type":"ContainerDied","Data":"f8139818738d11f7f7b00f7731d1d52b889ecd858cb30b73b83e6a3d89417bc0"} Nov 03 01:30:19.015321 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:30:19.015299 4939 scope.go:122] "RemoveContainer" containerID="f8139818738d11f7f7b00f7731d1d52b889ecd858cb30b73b83e6a3d89417bc0" Nov 03 01:30:19.015725 ip-10-0-140-107 kubenswrapper[4939]: E1103 01:30:19.015661 4939 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w0-2f9a3a-29894490-58pnm_odh-ai-gateway-infra(e505287a-7d6d-46f9-9a7c-2de4605546f2)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-2f9a3a-29894490-58pnm" podUID="e505287a-7d6d-46f9-9a7c-2de4605546f2" Nov 03 01:30:19.028084 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:30:19.028061 4939 scope.go:122] "RemoveContainer" containerID="d5dd503093c4b544a97bf71709a93c848823c02ef2ac93580bbaf59d6b4c581b" Nov 03 01:30:19.042799 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:30:19.042774 4939 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-2f9a3a-29894490-58pnm"] Nov 03 01:30:19.097276 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:30:19.096058 4939 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-027c19-29894490-dwg89"] Nov 03 01:30:19.121997 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:30:19.121969 4939 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-821f5a-29894490-9h7dh"] Nov 03 01:30:19.126840 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:30:19.126822 4939 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-821f5a-29894490-9h7dh"] Nov 03 01:30:19.337179 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:30:19.337143 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:30:19.893779 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:30:19.893743 4939 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="af3de10a-1341-4219-878f-9c73279482f0" path="/var/lib/kubelet/pods/af3de10a-1341-4219-878f-9c73279482f0/volumes" Nov 03 01:30:19.977163 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:30:19.977036 4939 patch_prober.go:29] interesting pod/e2e-inf-b-b4971a-openshift-default-5dc48765-8rn6x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.59:15021/healthz/ready\": dial tcp 10.132.1.59:15021: connect: connection refused" start-of-body= Nov 03 01:30:19.977163 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:30:19.977108 4939 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-b4971a-openshift-default-5dc48765-8rn6x" podUID="febb83d9-3a57-40a5-b972-c45611313469" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.59:15021/healthz/ready\": dial tcp 10.132.1.59:15021: connect: connection refused" Nov 03 01:30:20.027679 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:30:20.027640 4939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-027c19-29894490-dwg89" event={"ID":"56595211-63c2-491f-9912-b681df440d0a","Type":"ContainerStarted","Data":"189ff57c32b501a75aea88a7d5f9ad829f4a0a721b6b3faca1d949f7b2206c50"} Nov 03 01:30:20.027874 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:30:20.027706 4939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-027c19-29894490-dwg89" event={"ID":"56595211-63c2-491f-9912-b681df440d0a","Type":"ContainerStarted","Data":"ea969fb19a3857038913de7ca1287f26424afa3c93d8f61f52a5b24af18b92b2"} Nov 03 01:30:20.029953 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:30:20.029889 4939 generic.go:356] "Generic (PLEG): container finished" podID="38ef32d5-3c39-4b9d-a82a-774f3ba09ed2" containerID="82268704ad8287ecf308618e07e2ff486eadef4346d3b81b996ee3cad6bd52dc" exitCode=60 Nov 03 01:30:20.030098 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:30:20.029959 4939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-7c759a-29894490-blnms" event={"ID":"38ef32d5-3c39-4b9d-a82a-774f3ba09ed2","Type":"ContainerDied","Data":"82268704ad8287ecf308618e07e2ff486eadef4346d3b81b996ee3cad6bd52dc"} Nov 03 01:30:20.030098 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:30:20.030012 4939 scope.go:122] "RemoveContainer" containerID="bba0bf1fc174bf1d43516f0f4e80130c3b8116f94631af10f680dbe4b74ed2ef" Nov 03 01:30:20.030389 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:30:20.030372 4939 scope.go:122] "RemoveContainer" containerID="82268704ad8287ecf308618e07e2ff486eadef4346d3b81b996ee3cad6bd52dc" Nov 03 01:30:20.030677 ip-10-0-140-107 kubenswrapper[4939]: E1103 01:30:20.030660 4939 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-a-7c759a-29894490-blnms_odh-ai-gateway-infra(38ef32d5-3c39-4b9d-a82a-774f3ba09ed2)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-7c759a-29894490-blnms" podUID="38ef32d5-3c39-4b9d-a82a-774f3ba09ed2" Nov 03 01:30:20.033116 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:30:20.033089 4939 generic.go:356] "Generic (PLEG): container finished" podID="90e0cb11-fc6e-479e-8951-a36552f42532" containerID="9d52d3830f93300f1edd12e10d813c583cba6fb3189245ceea58dc7a098a704a" exitCode=60 Nov 03 01:30:20.033304 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:30:20.033188 4939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-556c14-29894490-cqqp5" event={"ID":"90e0cb11-fc6e-479e-8951-a36552f42532","Type":"ContainerDied","Data":"9d52d3830f93300f1edd12e10d813c583cba6fb3189245ceea58dc7a098a704a"} Nov 03 01:30:20.033573 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:30:20.033552 4939 scope.go:122] "RemoveContainer" containerID="9d52d3830f93300f1edd12e10d813c583cba6fb3189245ceea58dc7a098a704a" Nov 03 01:30:20.033866 ip-10-0-140-107 kubenswrapper[4939]: E1103 01:30:20.033846 4939 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w3-556c14-29894490-cqqp5_odh-ai-gateway-infra(90e0cb11-fc6e-479e-8951-a36552f42532)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-556c14-29894490-cqqp5" podUID="90e0cb11-fc6e-479e-8951-a36552f42532" Nov 03 01:30:20.041952 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:30:20.041930 4939 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-7c759a-29894490-blnms"] Nov 03 01:30:20.076000 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:30:20.075333 4939 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-027c19-29894490-dwg89" podStartSLOduration=20.075315001 podStartE2EDuration="20.075315001s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 01:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 01:30:20.074696773 +0000 UTC m=+2968.339031731" watchObservedRunningTime="2026-11-03 01:30:20.075315001 +0000 UTC m=+2968.339649950" Nov 03 01:30:20.080103 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:30:20.080078 4939 scope.go:122] "RemoveContainer" containerID="be4e5351082de069612028fe73e0c145838bcd5fb6e2a321e51ee29c87186c1c" Nov 03 01:30:20.093714 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:30:20.093683 4939 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-556c14-29894490-cqqp5"] Nov 03 01:30:20.097300 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:30:20.097274 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:30:20.430659 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:30:20.430627 4939 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-2f9a3a-29894490-58pnm" Nov 03 01:30:20.490793 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:30:20.490760 4939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e505287a-7d6d-46f9-9a7c-2de4605546f2-kube-api-access-52lmq\" (UniqueName: \"kubernetes.io/projected/e505287a-7d6d-46f9-9a7c-2de4605546f2-kube-api-access-52lmq\") pod \"e505287a-7d6d-46f9-9a7c-2de4605546f2\" (UID: \"e505287a-7d6d-46f9-9a7c-2de4605546f2\") " Nov 03 01:30:20.491065 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:30:20.491043 4939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e505287a-7d6d-46f9-9a7c-2de4605546f2-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e505287a-7d6d-46f9-9a7c-2de4605546f2-maas-api-service-ca\") pod \"e505287a-7d6d-46f9-9a7c-2de4605546f2\" (UID: \"e505287a-7d6d-46f9-9a7c-2de4605546f2\") " Nov 03 01:30:20.491681 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:30:20.491649 4939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e505287a-7d6d-46f9-9a7c-2de4605546f2-maas-api-service-ca" pod "e505287a-7d6d-46f9-9a7c-2de4605546f2" (UID: "e505287a-7d6d-46f9-9a7c-2de4605546f2"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 01:30:20.492553 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:30:20.492523 4939 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e505287a-7d6d-46f9-9a7c-2de4605546f2-maas-api-service-ca\") on node \"ip-10-0-140-107.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:30:20.493701 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:30:20.493674 4939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e505287a-7d6d-46f9-9a7c-2de4605546f2-kube-api-access-52lmq" pod "e505287a-7d6d-46f9-9a7c-2de4605546f2" (UID: "e505287a-7d6d-46f9-9a7c-2de4605546f2"). InnerVolumeSpecName "kube-api-access-52lmq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 01:30:20.583118 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:30:20.582951 4939 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-b4971a-openshift-default-5dc48765-8rn6x" Nov 03 01:30:20.587150 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:30:20.587123 4939 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-2729f2-29894490-d2d29"] Nov 03 01:30:20.597163 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:30:20.588150 4939 cpu_manager.go:414] "containerMap: removing container" podUID="e505287a-7d6d-46f9-9a7c-2de4605546f2" containerName="cleanup" Nov 03 01:30:20.597163 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:30:20.588168 4939 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e505287a-7d6d-46f9-9a7c-2de4605546f2" containerName="cleanup" Nov 03 01:30:20.597163 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:30:20.588180 4939 cpu_manager.go:414] "containerMap: removing container" podUID="e505287a-7d6d-46f9-9a7c-2de4605546f2" containerName="cleanup" Nov 03 01:30:20.597163 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:30:20.588185 4939 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e505287a-7d6d-46f9-9a7c-2de4605546f2" containerName="cleanup" Nov 03 01:30:20.597163 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:30:20.588377 4939 memory_manager.go:365] "RemoveStaleState removing state" podUID="e505287a-7d6d-46f9-9a7c-2de4605546f2" containerName="cleanup" Nov 03 01:30:20.597163 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:30:20.588397 4939 memory_manager.go:365] "RemoveStaleState removing state" podUID="e505287a-7d6d-46f9-9a7c-2de4605546f2" containerName="cleanup" Nov 03 01:30:20.597163 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:30:20.588405 4939 memory_manager.go:365] "RemoveStaleState removing state" podUID="e505287a-7d6d-46f9-9a7c-2de4605546f2" containerName="cleanup" Nov 03 01:30:20.597163 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:30:20.595229 4939 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-52lmq\" (UniqueName: \"kubernetes.io/projected/e505287a-7d6d-46f9-9a7c-2de4605546f2-kube-api-access-52lmq\") on node \"ip-10-0-140-107.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:30:20.601786 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:30:20.601764 4939 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-2729f2-29894490-d2d29"] Nov 03 01:30:20.601961 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:30:20.601855 4939 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-2729f2-29894490-d2d29" Nov 03 01:30:20.703672 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:30:20.703631 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/cee4a201-3e0b-4e98-b8c4-8dcb2acfca18-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w1-2729f2-29894490-d2d29\" (UID: \"cee4a201-3e0b-4e98-b8c4-8dcb2acfca18\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-2729f2-29894490-d2d29" Nov 03 01:30:20.703872 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:30:20.703730 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mjdzs\" (UniqueName: \"kubernetes.io/projected/cee4a201-3e0b-4e98-b8c4-8dcb2acfca18-kube-api-access-mjdzs\") pod \"maas-api-key-cleanup-e2e-shared-a-w1-2729f2-29894490-d2d29\" (UID: \"cee4a201-3e0b-4e98-b8c4-8dcb2acfca18\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-2729f2-29894490-d2d29" Nov 03 01:30:20.806312 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:30:20.806274 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/cee4a201-3e0b-4e98-b8c4-8dcb2acfca18-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w1-2729f2-29894490-d2d29\" (UID: \"cee4a201-3e0b-4e98-b8c4-8dcb2acfca18\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-2729f2-29894490-d2d29" Nov 03 01:30:20.806517 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:30:20.806348 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mjdzs\" (UniqueName: \"kubernetes.io/projected/cee4a201-3e0b-4e98-b8c4-8dcb2acfca18-kube-api-access-mjdzs\") pod \"maas-api-key-cleanup-e2e-shared-a-w1-2729f2-29894490-d2d29\" (UID: \"cee4a201-3e0b-4e98-b8c4-8dcb2acfca18\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-2729f2-29894490-d2d29" Nov 03 01:30:20.807035 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:30:20.807007 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/cee4a201-3e0b-4e98-b8c4-8dcb2acfca18-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w1-2729f2-29894490-d2d29\" (UID: \"cee4a201-3e0b-4e98-b8c4-8dcb2acfca18\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-2729f2-29894490-d2d29" Nov 03 01:30:20.820735 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:30:20.820700 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mjdzs\" (UniqueName: \"kubernetes.io/projected/cee4a201-3e0b-4e98-b8c4-8dcb2acfca18-kube-api-access-mjdzs\") pod \"maas-api-key-cleanup-e2e-shared-a-w1-2729f2-29894490-d2d29\" (UID: \"cee4a201-3e0b-4e98-b8c4-8dcb2acfca18\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-2729f2-29894490-d2d29" Nov 03 01:30:20.922634 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:30:20.922552 4939 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-2729f2-29894490-d2d29" Nov 03 01:30:20.982921 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:30:20.982876 4939 patch_prober.go:29] interesting pod/e2e-inf-b-b4971a-openshift-default-5dc48765-8rn6x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.59:15021/healthz/ready\": dial tcp 10.132.1.59:15021: connect: connection refused" start-of-body= Nov 03 01:30:20.983295 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:30:20.982987 4939 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-b4971a-openshift-default-5dc48765-8rn6x" podUID="febb83d9-3a57-40a5-b972-c45611313469" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.59:15021/healthz/ready\": dial tcp 10.132.1.59:15021: connect: connection refused" Nov 03 01:30:21.035991 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:30:21.035103 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:30:21.189353 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:30:21.189156 4939 generic.go:356] "Generic (PLEG): container finished" podID="56595211-63c2-491f-9912-b681df440d0a" containerID="189ff57c32b501a75aea88a7d5f9ad829f4a0a721b6b3faca1d949f7b2206c50" exitCode=60 Nov 03 01:30:21.189839 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:30:21.189783 4939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-027c19-29894490-dwg89" event={"ID":"56595211-63c2-491f-9912-b681df440d0a","Type":"ContainerDied","Data":"189ff57c32b501a75aea88a7d5f9ad829f4a0a721b6b3faca1d949f7b2206c50"} Nov 03 01:30:21.191559 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:30:21.190620 4939 scope.go:122] "RemoveContainer" containerID="189ff57c32b501a75aea88a7d5f9ad829f4a0a721b6b3faca1d949f7b2206c50" Nov 03 01:30:21.220430 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:30:21.220401 4939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-2f9a3a-29894490-58pnm" event={"ID":"e505287a-7d6d-46f9-9a7c-2de4605546f2","Type":"ContainerDied","Data":"224d19f6f904de239ad0542385317ab79941c0e5a4242c3befc335b7eb6450fa"} Nov 03 01:30:21.220556 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:30:21.220463 4939 scope.go:122] "RemoveContainer" containerID="f8139818738d11f7f7b00f7731d1d52b889ecd858cb30b73b83e6a3d89417bc0" Nov 03 01:30:21.221289 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:30:21.221189 4939 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-2f9a3a-29894490-58pnm" Nov 03 01:30:21.321535 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:30:21.321457 4939 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-2f9a3a-29894490-58pnm"] Nov 03 01:30:21.338809 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:30:21.337547 4939 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-2f9a3a-29894490-58pnm"] Nov 03 01:30:21.499896 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:30:21.498290 4939 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-2729f2-29894490-d2d29"] Nov 03 01:30:21.700264 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:30:21.700085 4939 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-7c759a-29894490-blnms" Nov 03 01:30:21.704899 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:30:21.704834 4939 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-556c14-29894490-cqqp5" Nov 03 01:30:21.728456 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:30:21.728428 4939 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-d25759-29894490-czcmr"] Nov 03 01:30:21.729441 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:30:21.729416 4939 cpu_manager.go:414] "containerMap: removing container" podUID="38ef32d5-3c39-4b9d-a82a-774f3ba09ed2" containerName="cleanup" Nov 03 01:30:21.729441 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:30:21.729439 4939 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="38ef32d5-3c39-4b9d-a82a-774f3ba09ed2" containerName="cleanup" Nov 03 01:30:21.729625 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:30:21.729488 4939 cpu_manager.go:414] "containerMap: removing container" podUID="e505287a-7d6d-46f9-9a7c-2de4605546f2" containerName="cleanup" Nov 03 01:30:21.729625 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:30:21.729496 4939 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e505287a-7d6d-46f9-9a7c-2de4605546f2" containerName="cleanup" Nov 03 01:30:21.729625 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:30:21.729512 4939 cpu_manager.go:414] "containerMap: removing container" podUID="90e0cb11-fc6e-479e-8951-a36552f42532" containerName="cleanup" Nov 03 01:30:21.729625 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:30:21.729519 4939 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="90e0cb11-fc6e-479e-8951-a36552f42532" containerName="cleanup" Nov 03 01:30:21.729625 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:30:21.729530 4939 cpu_manager.go:414] "containerMap: removing container" podUID="38ef32d5-3c39-4b9d-a82a-774f3ba09ed2" containerName="cleanup" Nov 03 01:30:21.729625 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:30:21.729537 4939 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="38ef32d5-3c39-4b9d-a82a-774f3ba09ed2" containerName="cleanup" Nov 03 01:30:21.729625 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:30:21.729552 4939 cpu_manager.go:414] "containerMap: removing container" podUID="90e0cb11-fc6e-479e-8951-a36552f42532" containerName="cleanup" Nov 03 01:30:21.729625 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:30:21.729560 4939 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="90e0cb11-fc6e-479e-8951-a36552f42532" containerName="cleanup" Nov 03 01:30:21.729625 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:30:21.729576 4939 cpu_manager.go:414] "containerMap: removing container" podUID="90e0cb11-fc6e-479e-8951-a36552f42532" containerName="cleanup" Nov 03 01:30:21.729625 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:30:21.729585 4939 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="90e0cb11-fc6e-479e-8951-a36552f42532" containerName="cleanup" Nov 03 01:30:21.729625 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:30:21.729609 4939 cpu_manager.go:414] "containerMap: removing container" podUID="38ef32d5-3c39-4b9d-a82a-774f3ba09ed2" containerName="cleanup" Nov 03 01:30:21.729625 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:30:21.729616 4939 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="38ef32d5-3c39-4b9d-a82a-774f3ba09ed2" containerName="cleanup" Nov 03 01:30:21.730248 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:30:21.729882 4939 memory_manager.go:365] "RemoveStaleState removing state" podUID="90e0cb11-fc6e-479e-8951-a36552f42532" containerName="cleanup" Nov 03 01:30:21.730248 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:30:21.729902 4939 memory_manager.go:365] "RemoveStaleState removing state" podUID="90e0cb11-fc6e-479e-8951-a36552f42532" containerName="cleanup" Nov 03 01:30:21.730248 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:30:21.729934 4939 memory_manager.go:365] "RemoveStaleState removing state" podUID="38ef32d5-3c39-4b9d-a82a-774f3ba09ed2" containerName="cleanup" Nov 03 01:30:21.730248 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:30:21.729955 4939 memory_manager.go:365] "RemoveStaleState removing state" podUID="38ef32d5-3c39-4b9d-a82a-774f3ba09ed2" containerName="cleanup" Nov 03 01:30:21.735044 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:30:21.734975 4939 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-d25759-29894490-czcmr" Nov 03 01:30:21.789534 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:30:21.788680 4939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/38ef32d5-3c39-4b9d-a82a-774f3ba09ed2-kube-api-access-tkbgg\" (UniqueName: \"kubernetes.io/projected/38ef32d5-3c39-4b9d-a82a-774f3ba09ed2-kube-api-access-tkbgg\") pod \"38ef32d5-3c39-4b9d-a82a-774f3ba09ed2\" (UID: \"38ef32d5-3c39-4b9d-a82a-774f3ba09ed2\") " Nov 03 01:30:21.789534 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:30:21.788733 4939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/90e0cb11-fc6e-479e-8951-a36552f42532-kube-api-access-scpqm\" (UniqueName: \"kubernetes.io/projected/90e0cb11-fc6e-479e-8951-a36552f42532-kube-api-access-scpqm\") pod \"90e0cb11-fc6e-479e-8951-a36552f42532\" (UID: \"90e0cb11-fc6e-479e-8951-a36552f42532\") " Nov 03 01:30:21.789534 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:30:21.788764 4939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/38ef32d5-3c39-4b9d-a82a-774f3ba09ed2-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/38ef32d5-3c39-4b9d-a82a-774f3ba09ed2-maas-api-service-ca\") pod \"38ef32d5-3c39-4b9d-a82a-774f3ba09ed2\" (UID: \"38ef32d5-3c39-4b9d-a82a-774f3ba09ed2\") " Nov 03 01:30:21.789534 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:30:21.788890 4939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/90e0cb11-fc6e-479e-8951-a36552f42532-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/90e0cb11-fc6e-479e-8951-a36552f42532-maas-api-service-ca\") pod \"90e0cb11-fc6e-479e-8951-a36552f42532\" (UID: \"90e0cb11-fc6e-479e-8951-a36552f42532\") " Nov 03 01:30:21.790459 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:30:21.790426 4939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/90e0cb11-fc6e-479e-8951-a36552f42532-maas-api-service-ca" pod "90e0cb11-fc6e-479e-8951-a36552f42532" (UID: "90e0cb11-fc6e-479e-8951-a36552f42532"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 01:30:21.790821 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:30:21.790798 4939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/38ef32d5-3c39-4b9d-a82a-774f3ba09ed2-maas-api-service-ca" pod "38ef32d5-3c39-4b9d-a82a-774f3ba09ed2" (UID: "38ef32d5-3c39-4b9d-a82a-774f3ba09ed2"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 01:30:21.792118 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:30:21.792088 4939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/90e0cb11-fc6e-479e-8951-a36552f42532-kube-api-access-scpqm" pod "90e0cb11-fc6e-479e-8951-a36552f42532" (UID: "90e0cb11-fc6e-479e-8951-a36552f42532"). InnerVolumeSpecName "kube-api-access-scpqm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 01:30:21.792234 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:30:21.792140 4939 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-d25759-29894490-czcmr"] Nov 03 01:30:21.794769 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:30:21.794729 4939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/38ef32d5-3c39-4b9d-a82a-774f3ba09ed2-kube-api-access-tkbgg" pod "38ef32d5-3c39-4b9d-a82a-774f3ba09ed2" (UID: "38ef32d5-3c39-4b9d-a82a-774f3ba09ed2"). InnerVolumeSpecName "kube-api-access-tkbgg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 01:30:21.805361 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:30:21.800117 4939 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-ddd653-29894490-qvrbc"] Nov 03 01:30:21.805361 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:30:21.801957 4939 memory_manager.go:365] "RemoveStaleState removing state" podUID="38ef32d5-3c39-4b9d-a82a-774f3ba09ed2" containerName="cleanup" Nov 03 01:30:21.805361 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:30:21.802006 4939 memory_manager.go:365] "RemoveStaleState removing state" podUID="90e0cb11-fc6e-479e-8951-a36552f42532" containerName="cleanup" Nov 03 01:30:21.808584 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:30:21.808522 4939 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-ddd653-29894490-qvrbc" Nov 03 01:30:21.814211 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:30:21.814168 4939 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-ddd653-29894490-qvrbc"] Nov 03 01:30:21.891520 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:30:21.891491 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m8nvl\" (UniqueName: \"kubernetes.io/projected/cd17c69a-8daf-42cb-8791-f2ebbaeaa6c0-kube-api-access-m8nvl\") pod \"maas-api-key-cleanup-e2e-extmod-a-ddd653-29894490-qvrbc\" (UID: \"cd17c69a-8daf-42cb-8791-f2ebbaeaa6c0\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-ddd653-29894490-qvrbc" Nov 03 01:30:21.891641 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:30:21.891593 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-shvxc\" (UniqueName: \"kubernetes.io/projected/e1d8185f-c5b3-4e38-93ea-343306023393-kube-api-access-shvxc\") pod \"maas-api-key-cleanup-e2e-extmod-a-d25759-29894490-czcmr\" (UID: \"e1d8185f-c5b3-4e38-93ea-343306023393\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-d25759-29894490-czcmr" Nov 03 01:30:21.891709 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:30:21.891699 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e1d8185f-c5b3-4e38-93ea-343306023393-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-a-d25759-29894490-czcmr\" (UID: \"e1d8185f-c5b3-4e38-93ea-343306023393\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-d25759-29894490-czcmr" Nov 03 01:30:21.891774 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:30:21.891761 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/cd17c69a-8daf-42cb-8791-f2ebbaeaa6c0-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-a-ddd653-29894490-qvrbc\" (UID: \"cd17c69a-8daf-42cb-8791-f2ebbaeaa6c0\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-ddd653-29894490-qvrbc" Nov 03 01:30:21.891853 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:30:21.891841 4939 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/90e0cb11-fc6e-479e-8951-a36552f42532-maas-api-service-ca\") on node \"ip-10-0-140-107.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:30:21.891928 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:30:21.891856 4939 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tkbgg\" (UniqueName: \"kubernetes.io/projected/38ef32d5-3c39-4b9d-a82a-774f3ba09ed2-kube-api-access-tkbgg\") on node \"ip-10-0-140-107.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:30:21.891928 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:30:21.891873 4939 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-scpqm\" (UniqueName: \"kubernetes.io/projected/90e0cb11-fc6e-479e-8951-a36552f42532-kube-api-access-scpqm\") on node \"ip-10-0-140-107.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:30:21.891928 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:30:21.891884 4939 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/38ef32d5-3c39-4b9d-a82a-774f3ba09ed2-maas-api-service-ca\") on node \"ip-10-0-140-107.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:30:21.897667 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:30:21.896241 4939 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e505287a-7d6d-46f9-9a7c-2de4605546f2" path="/var/lib/kubelet/pods/e505287a-7d6d-46f9-9a7c-2de4605546f2/volumes" Nov 03 01:30:21.973981 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:30:21.971183 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:30:21.976930 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:30:21.976380 4939 patch_prober.go:29] interesting pod/e2e-inf-b-b4971a-openshift-default-5dc48765-8rn6x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.59:15021/healthz/ready\": dial tcp 10.132.1.59:15021: connect: connection refused" start-of-body= Nov 03 01:30:21.976930 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:30:21.976451 4939 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-b4971a-openshift-default-5dc48765-8rn6x" podUID="febb83d9-3a57-40a5-b972-c45611313469" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.59:15021/healthz/ready\": dial tcp 10.132.1.59:15021: connect: connection refused" Nov 03 01:30:21.993891 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:30:21.993863 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/cd17c69a-8daf-42cb-8791-f2ebbaeaa6c0-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-a-ddd653-29894490-qvrbc\" (UID: \"cd17c69a-8daf-42cb-8791-f2ebbaeaa6c0\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-ddd653-29894490-qvrbc" Nov 03 01:30:21.994336 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:30:21.994127 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m8nvl\" (UniqueName: \"kubernetes.io/projected/cd17c69a-8daf-42cb-8791-f2ebbaeaa6c0-kube-api-access-m8nvl\") pod \"maas-api-key-cleanup-e2e-extmod-a-ddd653-29894490-qvrbc\" (UID: \"cd17c69a-8daf-42cb-8791-f2ebbaeaa6c0\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-ddd653-29894490-qvrbc" Nov 03 01:30:21.994336 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:30:21.994279 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-shvxc\" (UniqueName: \"kubernetes.io/projected/e1d8185f-c5b3-4e38-93ea-343306023393-kube-api-access-shvxc\") pod \"maas-api-key-cleanup-e2e-extmod-a-d25759-29894490-czcmr\" (UID: \"e1d8185f-c5b3-4e38-93ea-343306023393\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-d25759-29894490-czcmr" Nov 03 01:30:21.994546 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:30:21.994525 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e1d8185f-c5b3-4e38-93ea-343306023393-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-a-d25759-29894490-czcmr\" (UID: \"e1d8185f-c5b3-4e38-93ea-343306023393\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-d25759-29894490-czcmr" Nov 03 01:30:21.995401 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:30:21.994641 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/cd17c69a-8daf-42cb-8791-f2ebbaeaa6c0-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-a-ddd653-29894490-qvrbc\" (UID: \"cd17c69a-8daf-42cb-8791-f2ebbaeaa6c0\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-ddd653-29894490-qvrbc" Nov 03 01:30:21.996193 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:30:21.996165 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e1d8185f-c5b3-4e38-93ea-343306023393-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-a-d25759-29894490-czcmr\" (UID: \"e1d8185f-c5b3-4e38-93ea-343306023393\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-d25759-29894490-czcmr" Nov 03 01:30:22.059834 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:30:22.059795 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-shvxc\" (UniqueName: \"kubernetes.io/projected/e1d8185f-c5b3-4e38-93ea-343306023393-kube-api-access-shvxc\") pod \"maas-api-key-cleanup-e2e-extmod-a-d25759-29894490-czcmr\" (UID: \"e1d8185f-c5b3-4e38-93ea-343306023393\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-d25759-29894490-czcmr" Nov 03 01:30:22.061198 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:30:22.061171 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-m8nvl\" (UniqueName: \"kubernetes.io/projected/cd17c69a-8daf-42cb-8791-f2ebbaeaa6c0-kube-api-access-m8nvl\") pod \"maas-api-key-cleanup-e2e-extmod-a-ddd653-29894490-qvrbc\" (UID: \"cd17c69a-8daf-42cb-8791-f2ebbaeaa6c0\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-ddd653-29894490-qvrbc" Nov 03 01:30:22.117005 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:30:22.114456 4939 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-d25759-29894490-czcmr" Nov 03 01:30:22.126361 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:30:22.126250 4939 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-ddd653-29894490-qvrbc" Nov 03 01:30:22.240216 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:30:22.239698 4939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-027c19-29894490-dwg89" event={"ID":"56595211-63c2-491f-9912-b681df440d0a","Type":"ContainerStarted","Data":"e0dc38d5a87451e06f7a42069c31cd5eae9a0d2be9333ea5da8125b09099593a"} Nov 03 01:30:22.243216 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:30:22.243191 4939 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-7c759a-29894490-blnms" Nov 03 01:30:22.243369 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:30:22.243207 4939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-7c759a-29894490-blnms" event={"ID":"38ef32d5-3c39-4b9d-a82a-774f3ba09ed2","Type":"ContainerDied","Data":"55efaf696ac483ea3b61b726263ad7bca226ac9652f768369146dc8098f42ce8"} Nov 03 01:30:22.243503 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:30:22.243488 4939 scope.go:122] "RemoveContainer" containerID="82268704ad8287ecf308618e07e2ff486eadef4346d3b81b996ee3cad6bd52dc" Nov 03 01:30:22.265930 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:30:22.260398 4939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-556c14-29894490-cqqp5" event={"ID":"90e0cb11-fc6e-479e-8951-a36552f42532","Type":"ContainerDied","Data":"9d2fdf63e4e1e22dc91f5ea819da0eac6849f43419fd27b057c08ef759cc15c5"} Nov 03 01:30:22.273026 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:30:22.272987 4939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-2729f2-29894490-d2d29" event={"ID":"cee4a201-3e0b-4e98-b8c4-8dcb2acfca18","Type":"ContainerStarted","Data":"2b3b4b4c6a1866d1adfedece8bed85672a963175fc890b415e3ac64cc968a678"} Nov 03 01:30:22.273248 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:30:22.273223 4939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-2729f2-29894490-d2d29" event={"ID":"cee4a201-3e0b-4e98-b8c4-8dcb2acfca18","Type":"ContainerStarted","Data":"40e51da40ad7e901e3182808b640c8ee6e956d01e34f3f57d51dc1b901bb34ee"} Nov 03 01:30:22.273391 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:30:22.264875 4939 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-556c14-29894490-cqqp5" Nov 03 01:30:22.324312 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:30:22.324282 4939 scope.go:122] "RemoveContainer" containerID="9d52d3830f93300f1edd12e10d813c583cba6fb3189245ceea58dc7a098a704a" Nov 03 01:30:22.328237 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:30:22.328192 4939 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-2729f2-29894490-d2d29" podStartSLOduration=22.328178855 podStartE2EDuration="22.328178855s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 01:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 01:30:22.289394946 +0000 UTC m=+2970.553729892" watchObservedRunningTime="2026-11-03 01:30:22.328178855 +0000 UTC m=+2970.592513801" Nov 03 01:30:22.329220 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:30:22.329194 4939 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-7c759a-29894490-blnms"] Nov 03 01:30:22.335029 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:30:22.334987 4939 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-7c759a-29894490-blnms"] Nov 03 01:30:22.379235 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:30:22.379201 4939 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-556c14-29894490-cqqp5"] Nov 03 01:30:22.386790 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:30:22.386754 4939 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-556c14-29894490-cqqp5"] Nov 03 01:30:22.432979 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:30:22.432955 4939 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-d25759-29894490-czcmr"] Nov 03 01:30:22.434858 ip-10-0-140-107 kubenswrapper[4939]: W1103 01:30:22.434835 4939 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode1d8185f_c5b3_4e38_93ea_343306023393.slice/crio-eeaf4ce6e79c5a689ab3ce49c125a69b0209928564593d7488f58f12f505c571 WatchSource:0}: Error finding container eeaf4ce6e79c5a689ab3ce49c125a69b0209928564593d7488f58f12f505c571: Status 404 returned error can't find the container with id eeaf4ce6e79c5a689ab3ce49c125a69b0209928564593d7488f58f12f505c571 Nov 03 01:30:22.443204 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:30:22.443182 4939 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-ddd653-29894490-qvrbc"] Nov 03 01:30:22.886010 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:30:22.885976 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:30:22.967085 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:30:22.967042 4939 patch_prober.go:29] interesting pod/e2e-inf-b-b4971a-openshift-default-5dc48765-8rn6x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.59:15021/healthz/ready\": dial tcp 10.132.1.59:15021: connect: connection refused" start-of-body= Nov 03 01:30:22.967416 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:30:22.967381 4939 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-b4971a-openshift-default-5dc48765-8rn6x" podUID="febb83d9-3a57-40a5-b972-c45611313469" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.59:15021/healthz/ready\": dial tcp 10.132.1.59:15021: connect: connection refused" Nov 03 01:30:23.298128 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:30:23.298092 4939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-d25759-29894490-czcmr" event={"ID":"e1d8185f-c5b3-4e38-93ea-343306023393","Type":"ContainerStarted","Data":"ad3000f113f17caf182f81f0d98ea7889e991ca141b464b4ea79ed43e622da27"} Nov 03 01:30:23.298545 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:30:23.298158 4939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-d25759-29894490-czcmr" event={"ID":"e1d8185f-c5b3-4e38-93ea-343306023393","Type":"ContainerStarted","Data":"eeaf4ce6e79c5a689ab3ce49c125a69b0209928564593d7488f58f12f505c571"} Nov 03 01:30:23.300681 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:30:23.300645 4939 generic.go:356] "Generic (PLEG): container finished" podID="56595211-63c2-491f-9912-b681df440d0a" containerID="e0dc38d5a87451e06f7a42069c31cd5eae9a0d2be9333ea5da8125b09099593a" exitCode=60 Nov 03 01:30:23.300798 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:30:23.300739 4939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-027c19-29894490-dwg89" event={"ID":"56595211-63c2-491f-9912-b681df440d0a","Type":"ContainerDied","Data":"e0dc38d5a87451e06f7a42069c31cd5eae9a0d2be9333ea5da8125b09099593a"} Nov 03 01:30:23.300893 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:30:23.300797 4939 scope.go:122] "RemoveContainer" containerID="189ff57c32b501a75aea88a7d5f9ad829f4a0a721b6b3faca1d949f7b2206c50" Nov 03 01:30:23.301201 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:30:23.301178 4939 scope.go:122] "RemoveContainer" containerID="e0dc38d5a87451e06f7a42069c31cd5eae9a0d2be9333ea5da8125b09099593a" Nov 03 01:30:23.301722 ip-10-0-140-107 kubenswrapper[4939]: E1103 01:30:23.301682 4939 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-a-027c19-29894490-dwg89_odh-ai-gateway-infra(56595211-63c2-491f-9912-b681df440d0a)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-027c19-29894490-dwg89" podUID="56595211-63c2-491f-9912-b681df440d0a" Nov 03 01:30:23.305028 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:30:23.305004 4939 generic.go:356] "Generic (PLEG): container finished" podID="cee4a201-3e0b-4e98-b8c4-8dcb2acfca18" containerID="2b3b4b4c6a1866d1adfedece8bed85672a963175fc890b415e3ac64cc968a678" exitCode=60 Nov 03 01:30:23.305128 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:30:23.305079 4939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-2729f2-29894490-d2d29" event={"ID":"cee4a201-3e0b-4e98-b8c4-8dcb2acfca18","Type":"ContainerDied","Data":"2b3b4b4c6a1866d1adfedece8bed85672a963175fc890b415e3ac64cc968a678"} Nov 03 01:30:23.305448 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:30:23.305425 4939 scope.go:122] "RemoveContainer" containerID="2b3b4b4c6a1866d1adfedece8bed85672a963175fc890b415e3ac64cc968a678" Nov 03 01:30:23.307421 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:30:23.307395 4939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-ddd653-29894490-qvrbc" event={"ID":"cd17c69a-8daf-42cb-8791-f2ebbaeaa6c0","Type":"ContainerStarted","Data":"eb6be6dc6621835d6f94a9b87463908fc1a4d3bb694b6568edf6eb3851b30f3c"} Nov 03 01:30:23.307520 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:30:23.307452 4939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-ddd653-29894490-qvrbc" event={"ID":"cd17c69a-8daf-42cb-8791-f2ebbaeaa6c0","Type":"ContainerStarted","Data":"9bc02cfc8c76bb238cb17586788027c9379a41fccf3043690f9c39195b69e38c"} Nov 03 01:30:23.319868 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:30:23.319828 4939 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-d25759-29894490-czcmr" podStartSLOduration=23.319816682 podStartE2EDuration="23.319816682s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 01:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 01:30:23.314148536 +0000 UTC m=+2971.578483481" watchObservedRunningTime="2026-11-03 01:30:23.319816682 +0000 UTC m=+2971.584151627" Nov 03 01:30:23.366335 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:30:23.366172 4939 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-ddd653-29894490-qvrbc" podStartSLOduration=23.366151341 podStartE2EDuration="23.366151341s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 01:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 01:30:23.346681216 +0000 UTC m=+2971.611016160" watchObservedRunningTime="2026-11-03 01:30:23.366151341 +0000 UTC m=+2971.630486291" Nov 03 01:30:23.682367 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:30:23.682282 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:30:23.873260 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:30:23.873233 4939 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="38ef32d5-3c39-4b9d-a82a-774f3ba09ed2" path="/var/lib/kubelet/pods/38ef32d5-3c39-4b9d-a82a-774f3ba09ed2/volumes" Nov 03 01:30:23.873796 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:30:23.873780 4939 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="90e0cb11-fc6e-479e-8951-a36552f42532" path="/var/lib/kubelet/pods/90e0cb11-fc6e-479e-8951-a36552f42532/volumes" Nov 03 01:30:23.972278 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:30:23.972207 4939 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-b4971a-openshift-default-5dc48765-8rn6x" Nov 03 01:30:23.973353 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:30:23.973332 4939 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-b4971a-openshift-default-5dc48765-8rn6x" Nov 03 01:30:24.318375 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:30:24.318286 4939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-2729f2-29894490-d2d29" event={"ID":"cee4a201-3e0b-4e98-b8c4-8dcb2acfca18","Type":"ContainerStarted","Data":"b46a70b26b17907bf36a4c7920346b70c06b7d52511e8a624c6bf767d105919e"} Nov 03 01:30:24.320206 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:30:24.320178 4939 generic.go:356] "Generic (PLEG): container finished" podID="cd17c69a-8daf-42cb-8791-f2ebbaeaa6c0" containerID="eb6be6dc6621835d6f94a9b87463908fc1a4d3bb694b6568edf6eb3851b30f3c" exitCode=60 Nov 03 01:30:24.320328 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:30:24.320247 4939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-ddd653-29894490-qvrbc" event={"ID":"cd17c69a-8daf-42cb-8791-f2ebbaeaa6c0","Type":"ContainerDied","Data":"eb6be6dc6621835d6f94a9b87463908fc1a4d3bb694b6568edf6eb3851b30f3c"} Nov 03 01:30:24.320558 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:30:24.320537 4939 scope.go:122] "RemoveContainer" containerID="eb6be6dc6621835d6f94a9b87463908fc1a4d3bb694b6568edf6eb3851b30f3c" Nov 03 01:30:24.322461 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:30:24.322438 4939 generic.go:356] "Generic (PLEG): container finished" podID="e1d8185f-c5b3-4e38-93ea-343306023393" containerID="ad3000f113f17caf182f81f0d98ea7889e991ca141b464b4ea79ed43e622da27" exitCode=60 Nov 03 01:30:24.322578 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:30:24.322517 4939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-d25759-29894490-czcmr" event={"ID":"e1d8185f-c5b3-4e38-93ea-343306023393","Type":"ContainerDied","Data":"ad3000f113f17caf182f81f0d98ea7889e991ca141b464b4ea79ed43e622da27"} Nov 03 01:30:24.322804 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:30:24.322787 4939 scope.go:122] "RemoveContainer" containerID="ad3000f113f17caf182f81f0d98ea7889e991ca141b464b4ea79ed43e622da27" Nov 03 01:30:24.325395 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:30:24.325379 4939 scope.go:122] "RemoveContainer" containerID="e0dc38d5a87451e06f7a42069c31cd5eae9a0d2be9333ea5da8125b09099593a" Nov 03 01:30:24.327241 ip-10-0-140-107 kubenswrapper[4939]: E1103 01:30:24.327212 4939 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-a-027c19-29894490-dwg89_odh-ai-gateway-infra(56595211-63c2-491f-9912-b681df440d0a)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-027c19-29894490-dwg89" podUID="56595211-63c2-491f-9912-b681df440d0a" Nov 03 01:30:24.405311 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:30:24.405287 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:30:25.082408 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:30:25.082376 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:30:25.337731 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:30:25.337648 4939 generic.go:356] "Generic (PLEG): container finished" podID="cee4a201-3e0b-4e98-b8c4-8dcb2acfca18" containerID="b46a70b26b17907bf36a4c7920346b70c06b7d52511e8a624c6bf767d105919e" exitCode=60 Nov 03 01:30:25.337731 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:30:25.337719 4939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-2729f2-29894490-d2d29" event={"ID":"cee4a201-3e0b-4e98-b8c4-8dcb2acfca18","Type":"ContainerDied","Data":"b46a70b26b17907bf36a4c7920346b70c06b7d52511e8a624c6bf767d105919e"} Nov 03 01:30:25.338210 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:30:25.337779 4939 scope.go:122] "RemoveContainer" containerID="2b3b4b4c6a1866d1adfedece8bed85672a963175fc890b415e3ac64cc968a678" Nov 03 01:30:25.338210 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:30:25.338163 4939 scope.go:122] "RemoveContainer" containerID="b46a70b26b17907bf36a4c7920346b70c06b7d52511e8a624c6bf767d105919e" Nov 03 01:30:25.338699 ip-10-0-140-107 kubenswrapper[4939]: E1103 01:30:25.338675 4939 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w1-2729f2-29894490-d2d29_odh-ai-gateway-infra(cee4a201-3e0b-4e98-b8c4-8dcb2acfca18)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-2729f2-29894490-d2d29" podUID="cee4a201-3e0b-4e98-b8c4-8dcb2acfca18" Nov 03 01:30:25.339979 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:30:25.339954 4939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-ddd653-29894490-qvrbc" event={"ID":"cd17c69a-8daf-42cb-8791-f2ebbaeaa6c0","Type":"ContainerStarted","Data":"6c08c716f8950e653b359bd684c373123590b527bba559c08fc66e0c83abc4c1"} Nov 03 01:30:25.342091 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:30:25.342065 4939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-d25759-29894490-czcmr" event={"ID":"e1d8185f-c5b3-4e38-93ea-343306023393","Type":"ContainerStarted","Data":"1a240d684b9bcab99b9cf747348cbc2c41036442793868b14f0e5d3c72c54f75"} Nov 03 01:30:25.789697 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:30:25.789614 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:30:26.355258 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:30:26.355225 4939 scope.go:122] "RemoveContainer" containerID="b46a70b26b17907bf36a4c7920346b70c06b7d52511e8a624c6bf767d105919e" Nov 03 01:30:26.355735 ip-10-0-140-107 kubenswrapper[4939]: E1103 01:30:26.355650 4939 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w1-2729f2-29894490-d2d29_odh-ai-gateway-infra(cee4a201-3e0b-4e98-b8c4-8dcb2acfca18)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-2729f2-29894490-d2d29" podUID="cee4a201-3e0b-4e98-b8c4-8dcb2acfca18" Nov 03 01:30:26.358038 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:30:26.358004 4939 generic.go:356] "Generic (PLEG): container finished" podID="cd17c69a-8daf-42cb-8791-f2ebbaeaa6c0" containerID="6c08c716f8950e653b359bd684c373123590b527bba559c08fc66e0c83abc4c1" exitCode=60 Nov 03 01:30:26.358163 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:30:26.358084 4939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-ddd653-29894490-qvrbc" event={"ID":"cd17c69a-8daf-42cb-8791-f2ebbaeaa6c0","Type":"ContainerDied","Data":"6c08c716f8950e653b359bd684c373123590b527bba559c08fc66e0c83abc4c1"} Nov 03 01:30:26.358163 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:30:26.358154 4939 scope.go:122] "RemoveContainer" containerID="eb6be6dc6621835d6f94a9b87463908fc1a4d3bb694b6568edf6eb3851b30f3c" Nov 03 01:30:26.358411 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:30:26.358391 4939 scope.go:122] "RemoveContainer" containerID="6c08c716f8950e653b359bd684c373123590b527bba559c08fc66e0c83abc4c1" Nov 03 01:30:26.358846 ip-10-0-140-107 kubenswrapper[4939]: E1103 01:30:26.358825 4939 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-a-ddd653-29894490-qvrbc_odh-ai-gateway-infra(cd17c69a-8daf-42cb-8791-f2ebbaeaa6c0)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-ddd653-29894490-qvrbc" podUID="cd17c69a-8daf-42cb-8791-f2ebbaeaa6c0" Nov 03 01:30:26.362268 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:30:26.361087 4939 generic.go:356] "Generic (PLEG): container finished" podID="e1d8185f-c5b3-4e38-93ea-343306023393" containerID="1a240d684b9bcab99b9cf747348cbc2c41036442793868b14f0e5d3c72c54f75" exitCode=60 Nov 03 01:30:26.362268 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:30:26.361340 4939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-d25759-29894490-czcmr" event={"ID":"e1d8185f-c5b3-4e38-93ea-343306023393","Type":"ContainerDied","Data":"1a240d684b9bcab99b9cf747348cbc2c41036442793868b14f0e5d3c72c54f75"} Nov 03 01:30:26.362268 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:30:26.361864 4939 scope.go:122] "RemoveContainer" containerID="1a240d684b9bcab99b9cf747348cbc2c41036442793868b14f0e5d3c72c54f75" Nov 03 01:30:26.362470 ip-10-0-140-107 kubenswrapper[4939]: E1103 01:30:26.362283 4939 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-a-d25759-29894490-czcmr_odh-ai-gateway-infra(e1d8185f-c5b3-4e38-93ea-343306023393)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-d25759-29894490-czcmr" podUID="e1d8185f-c5b3-4e38-93ea-343306023393" Nov 03 01:30:26.374968 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:30:26.374922 4939 scope.go:122] "RemoveContainer" containerID="ad3000f113f17caf182f81f0d98ea7889e991ca141b464b4ea79ed43e622da27" Nov 03 01:30:26.609330 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:30:26.609221 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:30:27.269370 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:30:27.269333 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:30:27.371557 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:30:27.371526 4939 scope.go:122] "RemoveContainer" containerID="6c08c716f8950e653b359bd684c373123590b527bba559c08fc66e0c83abc4c1" Nov 03 01:30:27.372039 ip-10-0-140-107 kubenswrapper[4939]: E1103 01:30:27.371946 4939 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-a-ddd653-29894490-qvrbc_odh-ai-gateway-infra(cd17c69a-8daf-42cb-8791-f2ebbaeaa6c0)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-ddd653-29894490-qvrbc" podUID="cd17c69a-8daf-42cb-8791-f2ebbaeaa6c0" Nov 03 01:30:27.373235 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:30:27.373214 4939 scope.go:122] "RemoveContainer" containerID="1a240d684b9bcab99b9cf747348cbc2c41036442793868b14f0e5d3c72c54f75" Nov 03 01:30:27.373539 ip-10-0-140-107 kubenswrapper[4939]: E1103 01:30:27.373521 4939 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-a-d25759-29894490-czcmr_odh-ai-gateway-infra(e1d8185f-c5b3-4e38-93ea-343306023393)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-d25759-29894490-czcmr" podUID="e1d8185f-c5b3-4e38-93ea-343306023393" Nov 03 01:30:27.920633 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:30:27.920598 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:30:28.694052 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:30:28.693994 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:30:29.374441 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:30:29.374407 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:30:30.078109 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:30:30.078066 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:30:30.684984 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:30:30.684940 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:30:31.311225 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:30:31.311196 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:30:32.005136 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:30:32.005103 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:30:32.789745 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:30:32.789713 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:30:33.716060 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:30:33.716023 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:30:34.861549 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:30:34.861026 4939 scope.go:122] "RemoveContainer" containerID="e0dc38d5a87451e06f7a42069c31cd5eae9a0d2be9333ea5da8125b09099593a" Nov 03 01:30:34.861549 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:30:34.861520 4939 scope.go:122] "RemoveContainer" containerID="b46a70b26b17907bf36a4c7920346b70c06b7d52511e8a624c6bf767d105919e" Nov 03 01:30:34.988967 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:30:34.988933 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:30:35.494041 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:30:35.493951 4939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-027c19-29894490-dwg89" event={"ID":"56595211-63c2-491f-9912-b681df440d0a","Type":"ContainerStarted","Data":"774b13e057c349d7573b5847f4b207ae78e709c8f697777a304f08cd74a0f30d"} Nov 03 01:30:35.496063 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:30:35.496033 4939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-2729f2-29894490-d2d29" event={"ID":"cee4a201-3e0b-4e98-b8c4-8dcb2acfca18","Type":"ContainerStarted","Data":"c83cc9182318ecfe97028beb031b8a8bfa315f22b4c38fcb8e755b7f4eb67eed"} Nov 03 01:30:35.668130 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:30:35.668097 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:30:36.475870 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:30:36.475837 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:30:36.613026 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:30:36.600428 4939 generic.go:356] "Generic (PLEG): container finished" podID="56595211-63c2-491f-9912-b681df440d0a" containerID="774b13e057c349d7573b5847f4b207ae78e709c8f697777a304f08cd74a0f30d" exitCode=60 Nov 03 01:30:36.613026 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:30:36.600679 4939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-027c19-29894490-dwg89" event={"ID":"56595211-63c2-491f-9912-b681df440d0a","Type":"ContainerDied","Data":"774b13e057c349d7573b5847f4b207ae78e709c8f697777a304f08cd74a0f30d"} Nov 03 01:30:36.613026 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:30:36.600739 4939 scope.go:122] "RemoveContainer" containerID="e0dc38d5a87451e06f7a42069c31cd5eae9a0d2be9333ea5da8125b09099593a" Nov 03 01:30:36.613026 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:30:36.601455 4939 scope.go:122] "RemoveContainer" containerID="774b13e057c349d7573b5847f4b207ae78e709c8f697777a304f08cd74a0f30d" Nov 03 01:30:36.613026 ip-10-0-140-107 kubenswrapper[4939]: E1103 01:30:36.601964 4939 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-a-027c19-29894490-dwg89_odh-ai-gateway-infra(56595211-63c2-491f-9912-b681df440d0a)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-027c19-29894490-dwg89" podUID="56595211-63c2-491f-9912-b681df440d0a" Nov 03 01:30:36.619010 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:30:36.618974 4939 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-027c19-29894490-dwg89"] Nov 03 01:30:36.634957 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:30:36.627809 4939 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-2729f2-29894490-d2d29"] Nov 03 01:30:36.674928 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:30:36.660106 4939 generic.go:356] "Generic (PLEG): container finished" podID="cee4a201-3e0b-4e98-b8c4-8dcb2acfca18" containerID="c83cc9182318ecfe97028beb031b8a8bfa315f22b4c38fcb8e755b7f4eb67eed" exitCode=60 Nov 03 01:30:36.674928 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:30:36.660185 4939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-2729f2-29894490-d2d29" event={"ID":"cee4a201-3e0b-4e98-b8c4-8dcb2acfca18","Type":"ContainerDied","Data":"c83cc9182318ecfe97028beb031b8a8bfa315f22b4c38fcb8e755b7f4eb67eed"} Nov 03 01:30:36.680324 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:30:36.680293 4939 scope.go:122] "RemoveContainer" containerID="b46a70b26b17907bf36a4c7920346b70c06b7d52511e8a624c6bf767d105919e" Nov 03 01:30:36.873110 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:30:36.873084 4939 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-2729f2-29894490-d2d29" Nov 03 01:30:36.912244 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:30:36.912213 4939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cee4a201-3e0b-4e98-b8c4-8dcb2acfca18-kube-api-access-mjdzs\" (UniqueName: \"kubernetes.io/projected/cee4a201-3e0b-4e98-b8c4-8dcb2acfca18-kube-api-access-mjdzs\") pod \"cee4a201-3e0b-4e98-b8c4-8dcb2acfca18\" (UID: \"cee4a201-3e0b-4e98-b8c4-8dcb2acfca18\") " Nov 03 01:30:36.912428 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:30:36.912249 4939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cee4a201-3e0b-4e98-b8c4-8dcb2acfca18-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/cee4a201-3e0b-4e98-b8c4-8dcb2acfca18-maas-api-service-ca\") pod \"cee4a201-3e0b-4e98-b8c4-8dcb2acfca18\" (UID: \"cee4a201-3e0b-4e98-b8c4-8dcb2acfca18\") " Nov 03 01:30:36.912984 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:30:36.912956 4939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cee4a201-3e0b-4e98-b8c4-8dcb2acfca18-maas-api-service-ca" pod "cee4a201-3e0b-4e98-b8c4-8dcb2acfca18" (UID: "cee4a201-3e0b-4e98-b8c4-8dcb2acfca18"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 01:30:36.913369 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:30:36.913344 4939 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/cee4a201-3e0b-4e98-b8c4-8dcb2acfca18-maas-api-service-ca\") on node \"ip-10-0-140-107.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:30:36.915159 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:30:36.915124 4939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cee4a201-3e0b-4e98-b8c4-8dcb2acfca18-kube-api-access-mjdzs" pod "cee4a201-3e0b-4e98-b8c4-8dcb2acfca18" (UID: "cee4a201-3e0b-4e98-b8c4-8dcb2acfca18"). InnerVolumeSpecName "kube-api-access-mjdzs". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 01:30:37.014937 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:30:37.014840 4939 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mjdzs\" (UniqueName: \"kubernetes.io/projected/cee4a201-3e0b-4e98-b8c4-8dcb2acfca18-kube-api-access-mjdzs\") on node \"ip-10-0-140-107.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:30:37.148542 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:30:37.148496 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:30:37.286044 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:30:37.281093 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:30:37.673319 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:30:37.673280 4939 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-2729f2-29894490-d2d29" Nov 03 01:30:37.673829 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:30:37.673334 4939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-2729f2-29894490-d2d29" event={"ID":"cee4a201-3e0b-4e98-b8c4-8dcb2acfca18","Type":"ContainerDied","Data":"40e51da40ad7e901e3182808b640c8ee6e956d01e34f3f57d51dc1b901bb34ee"} Nov 03 01:30:37.673829 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:30:37.673404 4939 scope.go:122] "RemoveContainer" containerID="c83cc9182318ecfe97028beb031b8a8bfa315f22b4c38fcb8e755b7f4eb67eed" Nov 03 01:30:37.720419 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:30:37.720388 4939 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-2729f2-29894490-d2d29"] Nov 03 01:30:37.725989 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:30:37.725961 4939 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-2729f2-29894490-d2d29"] Nov 03 01:30:37.862782 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:30:37.862754 4939 scope.go:122] "RemoveContainer" containerID="6c08c716f8950e653b359bd684c373123590b527bba559c08fc66e0c83abc4c1" Nov 03 01:30:37.891116 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:30:37.891077 4939 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cee4a201-3e0b-4e98-b8c4-8dcb2acfca18" path="/var/lib/kubelet/pods/cee4a201-3e0b-4e98-b8c4-8dcb2acfca18/volumes" Nov 03 01:30:37.902613 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:30:37.902587 4939 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-027c19-29894490-dwg89" Nov 03 01:30:37.940558 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:30:37.940525 4939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/56595211-63c2-491f-9912-b681df440d0a-kube-api-access-2pbgz\" (UniqueName: \"kubernetes.io/projected/56595211-63c2-491f-9912-b681df440d0a-kube-api-access-2pbgz\") pod \"56595211-63c2-491f-9912-b681df440d0a\" (UID: \"56595211-63c2-491f-9912-b681df440d0a\") " Nov 03 01:30:37.940765 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:30:37.940644 4939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/56595211-63c2-491f-9912-b681df440d0a-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/56595211-63c2-491f-9912-b681df440d0a-maas-api-service-ca\") pod \"56595211-63c2-491f-9912-b681df440d0a\" (UID: \"56595211-63c2-491f-9912-b681df440d0a\") " Nov 03 01:30:37.941246 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:30:37.941221 4939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/56595211-63c2-491f-9912-b681df440d0a-maas-api-service-ca" pod "56595211-63c2-491f-9912-b681df440d0a" (UID: "56595211-63c2-491f-9912-b681df440d0a"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 01:30:37.941572 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:30:37.941552 4939 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/56595211-63c2-491f-9912-b681df440d0a-maas-api-service-ca\") on node \"ip-10-0-140-107.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:30:37.947707 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:30:37.947652 4939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/56595211-63c2-491f-9912-b681df440d0a-kube-api-access-2pbgz" pod "56595211-63c2-491f-9912-b681df440d0a" (UID: "56595211-63c2-491f-9912-b681df440d0a"). InnerVolumeSpecName "kube-api-access-2pbgz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 01:30:38.043150 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:30:38.043115 4939 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2pbgz\" (UniqueName: \"kubernetes.io/projected/56595211-63c2-491f-9912-b681df440d0a-kube-api-access-2pbgz\") on node \"ip-10-0-140-107.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:30:38.159830 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:30:38.154759 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:30:38.252194 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:30:38.252153 4939 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-a-b6afd6/test-model-b6afd6-kserve-86dc6b8b77-pl784"] Nov 03 01:30:38.253494 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:30:38.253467 4939 cpu_manager.go:414] "containerMap: removing container" podUID="56595211-63c2-491f-9912-b681df440d0a" containerName="cleanup" Nov 03 01:30:38.253494 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:30:38.253492 4939 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="56595211-63c2-491f-9912-b681df440d0a" containerName="cleanup" Nov 03 01:30:38.254384 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:30:38.253513 4939 cpu_manager.go:414] "containerMap: removing container" podUID="56595211-63c2-491f-9912-b681df440d0a" containerName="cleanup" Nov 03 01:30:38.254384 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:30:38.253521 4939 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="56595211-63c2-491f-9912-b681df440d0a" containerName="cleanup" Nov 03 01:30:38.254384 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:30:38.253558 4939 cpu_manager.go:414] "containerMap: removing container" podUID="cee4a201-3e0b-4e98-b8c4-8dcb2acfca18" containerName="cleanup" Nov 03 01:30:38.254384 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:30:38.253564 4939 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cee4a201-3e0b-4e98-b8c4-8dcb2acfca18" containerName="cleanup" Nov 03 01:30:38.254384 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:30:38.253589 4939 cpu_manager.go:414] "containerMap: removing container" podUID="cee4a201-3e0b-4e98-b8c4-8dcb2acfca18" containerName="cleanup" Nov 03 01:30:38.254384 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:30:38.253597 4939 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cee4a201-3e0b-4e98-b8c4-8dcb2acfca18" containerName="cleanup" Nov 03 01:30:38.254384 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:30:38.253845 4939 memory_manager.go:365] "RemoveStaleState removing state" podUID="56595211-63c2-491f-9912-b681df440d0a" containerName="cleanup" Nov 03 01:30:38.254384 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:30:38.253864 4939 memory_manager.go:365] "RemoveStaleState removing state" podUID="cee4a201-3e0b-4e98-b8c4-8dcb2acfca18" containerName="cleanup" Nov 03 01:30:38.254384 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:30:38.253879 4939 memory_manager.go:365] "RemoveStaleState removing state" podUID="cee4a201-3e0b-4e98-b8c4-8dcb2acfca18" containerName="cleanup" Nov 03 01:30:38.254384 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:30:38.253892 4939 memory_manager.go:365] "RemoveStaleState removing state" podUID="56595211-63c2-491f-9912-b681df440d0a" containerName="cleanup" Nov 03 01:30:38.254384 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:30:38.253900 4939 memory_manager.go:365] "RemoveStaleState removing state" podUID="cee4a201-3e0b-4e98-b8c4-8dcb2acfca18" containerName="cleanup" Nov 03 01:30:38.260254 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:30:38.259739 4939 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-b6afd6/test-model-b6afd6-kserve-86dc6b8b77-pl784" Nov 03 01:30:38.263595 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:30:38.263564 4939 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-b6afd6/test-model-b6afd6-kserve-86dc6b8b77-pl784"] Nov 03 01:30:38.265230 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:30:38.265204 4939 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-b6afd6\"/\"openshift-service-ca.crt\"" Nov 03 01:30:38.265893 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:30:38.265871 4939 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-b6afd6\"/\"default-dockercfg-swnzr\"" Nov 03 01:30:38.267230 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:30:38.266007 4939 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-b6afd6\"/\"test-model-b6afd6-kserve-self-signed-certs\"" Nov 03 01:30:38.267230 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:30:38.266059 4939 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-b6afd6\"/\"kube-root-ca.crt\"" Nov 03 01:30:38.357082 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:30:38.357051 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sq2p8\" (UniqueName: \"kubernetes.io/projected/04722cf5-df49-495f-a59a-e108d802c8e1-kube-api-access-sq2p8\") pod \"test-model-b6afd6-kserve-86dc6b8b77-pl784\" (UID: \"04722cf5-df49-495f-a59a-e108d802c8e1\") " pod="ai-tenant-e2e-inf-a-b6afd6/test-model-b6afd6-kserve-86dc6b8b77-pl784" Nov 03 01:30:38.357216 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:30:38.357192 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/04722cf5-df49-495f-a59a-e108d802c8e1-tls-certs\") pod \"test-model-b6afd6-kserve-86dc6b8b77-pl784\" (UID: \"04722cf5-df49-495f-a59a-e108d802c8e1\") " pod="ai-tenant-e2e-inf-a-b6afd6/test-model-b6afd6-kserve-86dc6b8b77-pl784" Nov 03 01:30:38.357269 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:30:38.357225 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/04722cf5-df49-495f-a59a-e108d802c8e1-model-cache\") pod \"test-model-b6afd6-kserve-86dc6b8b77-pl784\" (UID: \"04722cf5-df49-495f-a59a-e108d802c8e1\") " pod="ai-tenant-e2e-inf-a-b6afd6/test-model-b6afd6-kserve-86dc6b8b77-pl784" Nov 03 01:30:38.357387 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:30:38.357360 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/04722cf5-df49-495f-a59a-e108d802c8e1-tmp-dir\") pod \"test-model-b6afd6-kserve-86dc6b8b77-pl784\" (UID: \"04722cf5-df49-495f-a59a-e108d802c8e1\") " pod="ai-tenant-e2e-inf-a-b6afd6/test-model-b6afd6-kserve-86dc6b8b77-pl784" Nov 03 01:30:38.357519 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:30:38.357414 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/04722cf5-df49-495f-a59a-e108d802c8e1-dshm\") pod \"test-model-b6afd6-kserve-86dc6b8b77-pl784\" (UID: \"04722cf5-df49-495f-a59a-e108d802c8e1\") " pod="ai-tenant-e2e-inf-a-b6afd6/test-model-b6afd6-kserve-86dc6b8b77-pl784" Nov 03 01:30:38.357655 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:30:38.357567 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/04722cf5-df49-495f-a59a-e108d802c8e1-home\") pod \"test-model-b6afd6-kserve-86dc6b8b77-pl784\" (UID: \"04722cf5-df49-495f-a59a-e108d802c8e1\") " pod="ai-tenant-e2e-inf-a-b6afd6/test-model-b6afd6-kserve-86dc6b8b77-pl784" Nov 03 01:30:38.437079 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:30:38.429608 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:30:38.461399 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:30:38.461295 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sq2p8\" (UniqueName: \"kubernetes.io/projected/04722cf5-df49-495f-a59a-e108d802c8e1-kube-api-access-sq2p8\") pod \"test-model-b6afd6-kserve-86dc6b8b77-pl784\" (UID: \"04722cf5-df49-495f-a59a-e108d802c8e1\") " pod="ai-tenant-e2e-inf-a-b6afd6/test-model-b6afd6-kserve-86dc6b8b77-pl784" Nov 03 01:30:38.468204 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:30:38.468169 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/04722cf5-df49-495f-a59a-e108d802c8e1-tls-certs\") pod \"test-model-b6afd6-kserve-86dc6b8b77-pl784\" (UID: \"04722cf5-df49-495f-a59a-e108d802c8e1\") " pod="ai-tenant-e2e-inf-a-b6afd6/test-model-b6afd6-kserve-86dc6b8b77-pl784" Nov 03 01:30:38.470832 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:30:38.470791 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/04722cf5-df49-495f-a59a-e108d802c8e1-model-cache\") pod \"test-model-b6afd6-kserve-86dc6b8b77-pl784\" (UID: \"04722cf5-df49-495f-a59a-e108d802c8e1\") " pod="ai-tenant-e2e-inf-a-b6afd6/test-model-b6afd6-kserve-86dc6b8b77-pl784" Nov 03 01:30:38.476625 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:30:38.471385 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/04722cf5-df49-495f-a59a-e108d802c8e1-model-cache\") pod \"test-model-b6afd6-kserve-86dc6b8b77-pl784\" (UID: \"04722cf5-df49-495f-a59a-e108d802c8e1\") " pod="ai-tenant-e2e-inf-a-b6afd6/test-model-b6afd6-kserve-86dc6b8b77-pl784" Nov 03 01:30:38.477287 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:30:38.476828 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/04722cf5-df49-495f-a59a-e108d802c8e1-tmp-dir\") pod \"test-model-b6afd6-kserve-86dc6b8b77-pl784\" (UID: \"04722cf5-df49-495f-a59a-e108d802c8e1\") " pod="ai-tenant-e2e-inf-a-b6afd6/test-model-b6afd6-kserve-86dc6b8b77-pl784" Nov 03 01:30:38.477500 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:30:38.477148 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/04722cf5-df49-495f-a59a-e108d802c8e1-tmp-dir\") pod \"test-model-b6afd6-kserve-86dc6b8b77-pl784\" (UID: \"04722cf5-df49-495f-a59a-e108d802c8e1\") " pod="ai-tenant-e2e-inf-a-b6afd6/test-model-b6afd6-kserve-86dc6b8b77-pl784" Nov 03 01:30:38.478034 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:30:38.478012 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/04722cf5-df49-495f-a59a-e108d802c8e1-dshm\") pod \"test-model-b6afd6-kserve-86dc6b8b77-pl784\" (UID: \"04722cf5-df49-495f-a59a-e108d802c8e1\") " pod="ai-tenant-e2e-inf-a-b6afd6/test-model-b6afd6-kserve-86dc6b8b77-pl784" Nov 03 01:30:38.478285 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:30:38.478263 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/04722cf5-df49-495f-a59a-e108d802c8e1-home\") pod \"test-model-b6afd6-kserve-86dc6b8b77-pl784\" (UID: \"04722cf5-df49-495f-a59a-e108d802c8e1\") " pod="ai-tenant-e2e-inf-a-b6afd6/test-model-b6afd6-kserve-86dc6b8b77-pl784" Nov 03 01:30:38.496515 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:30:38.496474 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/04722cf5-df49-495f-a59a-e108d802c8e1-home\") pod \"test-model-b6afd6-kserve-86dc6b8b77-pl784\" (UID: \"04722cf5-df49-495f-a59a-e108d802c8e1\") " pod="ai-tenant-e2e-inf-a-b6afd6/test-model-b6afd6-kserve-86dc6b8b77-pl784" Nov 03 01:30:38.521930 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:30:38.505613 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sq2p8\" (UniqueName: \"kubernetes.io/projected/04722cf5-df49-495f-a59a-e108d802c8e1-kube-api-access-sq2p8\") pod \"test-model-b6afd6-kserve-86dc6b8b77-pl784\" (UID: \"04722cf5-df49-495f-a59a-e108d802c8e1\") " pod="ai-tenant-e2e-inf-a-b6afd6/test-model-b6afd6-kserve-86dc6b8b77-pl784" Nov 03 01:30:38.521930 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:30:38.507769 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/04722cf5-df49-495f-a59a-e108d802c8e1-tls-certs\") pod \"test-model-b6afd6-kserve-86dc6b8b77-pl784\" (UID: \"04722cf5-df49-495f-a59a-e108d802c8e1\") " pod="ai-tenant-e2e-inf-a-b6afd6/test-model-b6afd6-kserve-86dc6b8b77-pl784" Nov 03 01:30:38.545784 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:30:38.532442 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/04722cf5-df49-495f-a59a-e108d802c8e1-dshm\") pod \"test-model-b6afd6-kserve-86dc6b8b77-pl784\" (UID: \"04722cf5-df49-495f-a59a-e108d802c8e1\") " pod="ai-tenant-e2e-inf-a-b6afd6/test-model-b6afd6-kserve-86dc6b8b77-pl784" Nov 03 01:30:38.581601 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:30:38.581564 4939 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-b6afd6/test-model-b6afd6-kserve-86dc6b8b77-pl784" Nov 03 01:30:38.688988 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:30:38.688958 4939 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-027c19-29894490-dwg89" Nov 03 01:30:38.688988 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:30:38.688969 4939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-027c19-29894490-dwg89" event={"ID":"56595211-63c2-491f-9912-b681df440d0a","Type":"ContainerDied","Data":"ea969fb19a3857038913de7ca1287f26424afa3c93d8f61f52a5b24af18b92b2"} Nov 03 01:30:38.689448 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:30:38.689042 4939 scope.go:122] "RemoveContainer" containerID="774b13e057c349d7573b5847f4b207ae78e709c8f697777a304f08cd74a0f30d" Nov 03 01:30:38.694488 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:30:38.694458 4939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-ddd653-29894490-qvrbc" event={"ID":"cd17c69a-8daf-42cb-8791-f2ebbaeaa6c0","Type":"ContainerStarted","Data":"4a4e4f125645b35bcda5ed1d12a4fbf21e921b50860af868ee24bea887937836"} Nov 03 01:30:38.750874 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:30:38.750832 4939 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-027c19-29894490-dwg89"] Nov 03 01:30:38.756828 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:30:38.756800 4939 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-027c19-29894490-dwg89"] Nov 03 01:30:38.945969 ip-10-0-140-107 kubenswrapper[4939]: W1103 01:30:38.945931 4939 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod04722cf5_df49_495f_a59a_e108d802c8e1.slice/crio-3e08fa6c91f664f57f496248033ba7ece7880161c0f2f1a63c2649fdd59936d3 WatchSource:0}: Error finding container 3e08fa6c91f664f57f496248033ba7ece7880161c0f2f1a63c2649fdd59936d3: Status 404 returned error can't find the container with id 3e08fa6c91f664f57f496248033ba7ece7880161c0f2f1a63c2649fdd59936d3 Nov 03 01:30:38.948185 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:30:38.948159 4939 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-b6afd6/test-model-b6afd6-kserve-86dc6b8b77-pl784"] Nov 03 01:30:39.188309 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:30:39.188263 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:30:39.709783 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:30:39.709746 4939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-b6afd6/test-model-b6afd6-kserve-86dc6b8b77-pl784" event={"ID":"04722cf5-df49-495f-a59a-e108d802c8e1","Type":"ContainerStarted","Data":"308fdc41d0ef1aaa117dd922584091976046bc41729824ee503708ae3cb63560"} Nov 03 01:30:39.710303 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:30:39.709800 4939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-b6afd6/test-model-b6afd6-kserve-86dc6b8b77-pl784" event={"ID":"04722cf5-df49-495f-a59a-e108d802c8e1","Type":"ContainerStarted","Data":"3e08fa6c91f664f57f496248033ba7ece7880161c0f2f1a63c2649fdd59936d3"} Nov 03 01:30:39.712043 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:30:39.712021 4939 generic.go:356] "Generic (PLEG): container finished" podID="cd17c69a-8daf-42cb-8791-f2ebbaeaa6c0" containerID="4a4e4f125645b35bcda5ed1d12a4fbf21e921b50860af868ee24bea887937836" exitCode=60 Nov 03 01:30:39.712167 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:30:39.712096 4939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-ddd653-29894490-qvrbc" event={"ID":"cd17c69a-8daf-42cb-8791-f2ebbaeaa6c0","Type":"ContainerDied","Data":"4a4e4f125645b35bcda5ed1d12a4fbf21e921b50860af868ee24bea887937836"} Nov 03 01:30:39.712167 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:30:39.712156 4939 scope.go:122] "RemoveContainer" containerID="6c08c716f8950e653b359bd684c373123590b527bba559c08fc66e0c83abc4c1" Nov 03 01:30:39.712661 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:30:39.712642 4939 scope.go:122] "RemoveContainer" containerID="4a4e4f125645b35bcda5ed1d12a4fbf21e921b50860af868ee24bea887937836" Nov 03 01:30:39.713168 ip-10-0-140-107 kubenswrapper[4939]: E1103 01:30:39.713110 4939 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-a-ddd653-29894490-qvrbc_odh-ai-gateway-infra(cd17c69a-8daf-42cb-8791-f2ebbaeaa6c0)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-ddd653-29894490-qvrbc" podUID="cd17c69a-8daf-42cb-8791-f2ebbaeaa6c0" Nov 03 01:30:39.727696 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:30:39.727667 4939 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-ddd653-29894490-qvrbc"] Nov 03 01:30:39.730809 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:30:39.730754 4939 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-inf-a-b6afd6/test-model-b6afd6-kserve-86dc6b8b77-pl784" podStartSLOduration=1.730741573 podStartE2EDuration="1.730741573s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 01:30:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 01:30:39.726885123 +0000 UTC m=+2987.991220068" watchObservedRunningTime="2026-11-03 01:30:39.730741573 +0000 UTC m=+2987.995076522" Nov 03 01:30:39.861236 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:30:39.861201 4939 scope.go:122] "RemoveContainer" containerID="1a240d684b9bcab99b9cf747348cbc2c41036442793868b14f0e5d3c72c54f75" Nov 03 01:30:39.873392 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:30:39.873360 4939 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="56595211-63c2-491f-9912-b681df440d0a" path="/var/lib/kubelet/pods/56595211-63c2-491f-9912-b681df440d0a/volumes" Nov 03 01:30:39.878684 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:30:39.878640 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:30:40.582655 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:30:40.582616 4939 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-a-b6afd6/test-model-b6afd6-kserve-86dc6b8b77-pl784" Nov 03 01:30:40.698187 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:30:40.698145 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:30:40.818091 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:30:40.818047 4939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-d25759-29894490-czcmr" event={"ID":"e1d8185f-c5b3-4e38-93ea-343306023393","Type":"ContainerStarted","Data":"5cfbdd1fce9b78e7339e4fe4d69bdaf6714daec211b63ee723d26b29c1dd0183"} Nov 03 01:30:41.136248 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:30:41.136214 4939 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-ddd653-29894490-qvrbc" Nov 03 01:30:41.288593 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:30:41.288481 4939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cd17c69a-8daf-42cb-8791-f2ebbaeaa6c0-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/cd17c69a-8daf-42cb-8791-f2ebbaeaa6c0-maas-api-service-ca\") pod \"cd17c69a-8daf-42cb-8791-f2ebbaeaa6c0\" (UID: \"cd17c69a-8daf-42cb-8791-f2ebbaeaa6c0\") " Nov 03 01:30:41.288797 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:30:41.288610 4939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cd17c69a-8daf-42cb-8791-f2ebbaeaa6c0-kube-api-access-m8nvl\" (UniqueName: \"kubernetes.io/projected/cd17c69a-8daf-42cb-8791-f2ebbaeaa6c0-kube-api-access-m8nvl\") pod \"cd17c69a-8daf-42cb-8791-f2ebbaeaa6c0\" (UID: \"cd17c69a-8daf-42cb-8791-f2ebbaeaa6c0\") " Nov 03 01:30:41.288891 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:30:41.288866 4939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cd17c69a-8daf-42cb-8791-f2ebbaeaa6c0-maas-api-service-ca" pod "cd17c69a-8daf-42cb-8791-f2ebbaeaa6c0" (UID: "cd17c69a-8daf-42cb-8791-f2ebbaeaa6c0"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 01:30:41.289710 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:30:41.289686 4939 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/cd17c69a-8daf-42cb-8791-f2ebbaeaa6c0-maas-api-service-ca\") on node \"ip-10-0-140-107.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:30:41.290889 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:30:41.290863 4939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cd17c69a-8daf-42cb-8791-f2ebbaeaa6c0-kube-api-access-m8nvl" pod "cd17c69a-8daf-42cb-8791-f2ebbaeaa6c0" (UID: "cd17c69a-8daf-42cb-8791-f2ebbaeaa6c0"). InnerVolumeSpecName "kube-api-access-m8nvl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 01:30:41.391757 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:30:41.391718 4939 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m8nvl\" (UniqueName: \"kubernetes.io/projected/cd17c69a-8daf-42cb-8791-f2ebbaeaa6c0-kube-api-access-m8nvl\") on node \"ip-10-0-140-107.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:30:41.526772 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:30:41.514955 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:30:41.832302 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:30:41.832275 4939 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-ddd653-29894490-qvrbc" Nov 03 01:30:41.832808 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:30:41.832312 4939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-ddd653-29894490-qvrbc" event={"ID":"cd17c69a-8daf-42cb-8791-f2ebbaeaa6c0","Type":"ContainerDied","Data":"9bc02cfc8c76bb238cb17586788027c9379a41fccf3043690f9c39195b69e38c"} Nov 03 01:30:41.832808 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:30:41.832377 4939 scope.go:122] "RemoveContainer" containerID="4a4e4f125645b35bcda5ed1d12a4fbf21e921b50860af868ee24bea887937836" Nov 03 01:30:41.834792 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:30:41.834767 4939 generic.go:356] "Generic (PLEG): container finished" podID="e1d8185f-c5b3-4e38-93ea-343306023393" containerID="5cfbdd1fce9b78e7339e4fe4d69bdaf6714daec211b63ee723d26b29c1dd0183" exitCode=60 Nov 03 01:30:41.834949 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:30:41.834846 4939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-d25759-29894490-czcmr" event={"ID":"e1d8185f-c5b3-4e38-93ea-343306023393","Type":"ContainerDied","Data":"5cfbdd1fce9b78e7339e4fe4d69bdaf6714daec211b63ee723d26b29c1dd0183"} Nov 03 01:30:41.835358 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:30:41.835341 4939 scope.go:122] "RemoveContainer" containerID="5cfbdd1fce9b78e7339e4fe4d69bdaf6714daec211b63ee723d26b29c1dd0183" Nov 03 01:30:41.835721 ip-10-0-140-107 kubenswrapper[4939]: E1103 01:30:41.835703 4939 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-a-d25759-29894490-czcmr_odh-ai-gateway-infra(e1d8185f-c5b3-4e38-93ea-343306023393)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-d25759-29894490-czcmr" podUID="e1d8185f-c5b3-4e38-93ea-343306023393" Nov 03 01:30:41.845626 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:30:41.845446 4939 scope.go:122] "RemoveContainer" containerID="1a240d684b9bcab99b9cf747348cbc2c41036442793868b14f0e5d3c72c54f75" Nov 03 01:30:41.851727 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:30:41.851704 4939 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-d25759-29894490-czcmr"] Nov 03 01:30:41.885635 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:30:41.885603 4939 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-ddd653-29894490-qvrbc"] Nov 03 01:30:41.891500 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:30:41.891472 4939 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-ddd653-29894490-qvrbc"] Nov 03 01:30:42.369767 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:30:42.369732 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:30:43.059347 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:30:43.059320 4939 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-d25759-29894490-czcmr" Nov 03 01:30:43.133669 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:30:43.133584 4939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e1d8185f-c5b3-4e38-93ea-343306023393-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e1d8185f-c5b3-4e38-93ea-343306023393-maas-api-service-ca\") pod \"e1d8185f-c5b3-4e38-93ea-343306023393\" (UID: \"e1d8185f-c5b3-4e38-93ea-343306023393\") " Nov 03 01:30:43.133833 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:30:43.133754 4939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e1d8185f-c5b3-4e38-93ea-343306023393-kube-api-access-shvxc\" (UniqueName: \"kubernetes.io/projected/e1d8185f-c5b3-4e38-93ea-343306023393-kube-api-access-shvxc\") pod \"e1d8185f-c5b3-4e38-93ea-343306023393\" (UID: \"e1d8185f-c5b3-4e38-93ea-343306023393\") " Nov 03 01:30:43.134059 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:30:43.134027 4939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e1d8185f-c5b3-4e38-93ea-343306023393-maas-api-service-ca" pod "e1d8185f-c5b3-4e38-93ea-343306023393" (UID: "e1d8185f-c5b3-4e38-93ea-343306023393"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 01:30:43.134745 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:30:43.134703 4939 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e1d8185f-c5b3-4e38-93ea-343306023393-maas-api-service-ca\") on node \"ip-10-0-140-107.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:30:43.136514 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:30:43.136482 4939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e1d8185f-c5b3-4e38-93ea-343306023393-kube-api-access-shvxc" pod "e1d8185f-c5b3-4e38-93ea-343306023393" (UID: "e1d8185f-c5b3-4e38-93ea-343306023393"). InnerVolumeSpecName "kube-api-access-shvxc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 01:30:43.253936 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:30:43.253555 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:30:43.259929 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:30:43.258927 4939 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-shvxc\" (UniqueName: \"kubernetes.io/projected/e1d8185f-c5b3-4e38-93ea-343306023393-kube-api-access-shvxc\") on node \"ip-10-0-140-107.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:30:43.859519 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:30:43.859476 4939 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-d25759-29894490-czcmr" Nov 03 01:30:43.859753 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:30:43.859472 4939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-d25759-29894490-czcmr" event={"ID":"e1d8185f-c5b3-4e38-93ea-343306023393","Type":"ContainerDied","Data":"eeaf4ce6e79c5a689ab3ce49c125a69b0209928564593d7488f58f12f505c571"} Nov 03 01:30:43.859753 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:30:43.859641 4939 scope.go:122] "RemoveContainer" containerID="5cfbdd1fce9b78e7339e4fe4d69bdaf6714daec211b63ee723d26b29c1dd0183" Nov 03 01:30:43.872836 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:30:43.872814 4939 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cd17c69a-8daf-42cb-8791-f2ebbaeaa6c0" path="/var/lib/kubelet/pods/cd17c69a-8daf-42cb-8791-f2ebbaeaa6c0/volumes" Nov 03 01:30:43.901063 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:30:43.900884 4939 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-d25759-29894490-czcmr"] Nov 03 01:30:43.910584 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:30:43.910550 4939 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-d25759-29894490-czcmr"] Nov 03 01:30:43.981829 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:30:43.981797 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:30:44.784539 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:30:44.784489 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:30:45.028267 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:30:45.028232 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:30:45.504364 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:30:45.504327 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:30:45.871709 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:30:45.871675 4939 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e1d8185f-c5b3-4e38-93ea-343306023393" path="/var/lib/kubelet/pods/e1d8185f-c5b3-4e38-93ea-343306023393/volumes" Nov 03 01:30:46.387300 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:30:46.387260 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:30:47.085954 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:30:47.085920 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:30:47.823641 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:30:47.823576 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:30:48.027796 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:30:48.027761 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:30:48.034366 ip-10-0-140-107 kubenswrapper[4939]: E1103 01:30:48.034332 4939 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-sthjh" containerName="extractor" Nov 03 01:30:48.154323 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:30:48.154279 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:30:48.439269 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:30:48.439152 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:30:48.582347 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:30:48.582302 4939 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-a-b6afd6/test-model-b6afd6-kserve-86dc6b8b77-pl784" Nov 03 01:30:48.595280 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:30:48.595253 4939 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-a-b6afd6/test-model-b6afd6-kserve-86dc6b8b77-pl784" Nov 03 01:30:48.984963 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:30:48.984923 4939 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-a-b6afd6/test-model-b6afd6-kserve-86dc6b8b77-pl784" Nov 03 01:30:49.183526 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:30:49.183495 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:30:49.806967 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:30:49.801318 4939 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-b-b4971a/test-model-b4971a-kserve-5f54798fc5-hc2bn"] Nov 03 01:30:49.806967 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:30:49.802815 4939 cpu_manager.go:414] "containerMap: removing container" podUID="e1d8185f-c5b3-4e38-93ea-343306023393" containerName="cleanup" Nov 03 01:30:49.806967 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:30:49.802838 4939 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e1d8185f-c5b3-4e38-93ea-343306023393" containerName="cleanup" Nov 03 01:30:49.806967 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:30:49.802883 4939 cpu_manager.go:414] "containerMap: removing container" podUID="cd17c69a-8daf-42cb-8791-f2ebbaeaa6c0" containerName="cleanup" Nov 03 01:30:49.806967 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:30:49.802891 4939 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cd17c69a-8daf-42cb-8791-f2ebbaeaa6c0" containerName="cleanup" Nov 03 01:30:49.806967 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:30:49.802902 4939 cpu_manager.go:414] "containerMap: removing container" podUID="e1d8185f-c5b3-4e38-93ea-343306023393" containerName="cleanup" Nov 03 01:30:49.806967 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:30:49.802933 4939 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e1d8185f-c5b3-4e38-93ea-343306023393" containerName="cleanup" Nov 03 01:30:49.806967 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:30:49.802953 4939 cpu_manager.go:414] "containerMap: removing container" podUID="56595211-63c2-491f-9912-b681df440d0a" containerName="cleanup" Nov 03 01:30:49.806967 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:30:49.802961 4939 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="56595211-63c2-491f-9912-b681df440d0a" containerName="cleanup" Nov 03 01:30:49.806967 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:30:49.803000 4939 cpu_manager.go:414] "containerMap: removing container" podUID="cd17c69a-8daf-42cb-8791-f2ebbaeaa6c0" containerName="cleanup" Nov 03 01:30:49.806967 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:30:49.803008 4939 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cd17c69a-8daf-42cb-8791-f2ebbaeaa6c0" containerName="cleanup" Nov 03 01:30:49.806967 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:30:49.803043 4939 cpu_manager.go:414] "containerMap: removing container" podUID="e1d8185f-c5b3-4e38-93ea-343306023393" containerName="cleanup" Nov 03 01:30:49.806967 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:30:49.803052 4939 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e1d8185f-c5b3-4e38-93ea-343306023393" containerName="cleanup" Nov 03 01:30:49.806967 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:30:49.803082 4939 cpu_manager.go:414] "containerMap: removing container" podUID="cee4a201-3e0b-4e98-b8c4-8dcb2acfca18" containerName="cleanup" Nov 03 01:30:49.806967 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:30:49.803090 4939 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cee4a201-3e0b-4e98-b8c4-8dcb2acfca18" containerName="cleanup" Nov 03 01:30:49.806967 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:30:49.803110 4939 cpu_manager.go:414] "containerMap: removing container" podUID="cd17c69a-8daf-42cb-8791-f2ebbaeaa6c0" containerName="cleanup" Nov 03 01:30:49.806967 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:30:49.803118 4939 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cd17c69a-8daf-42cb-8791-f2ebbaeaa6c0" containerName="cleanup" Nov 03 01:30:49.806967 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:30:49.803349 4939 memory_manager.go:365] "RemoveStaleState removing state" podUID="cd17c69a-8daf-42cb-8791-f2ebbaeaa6c0" containerName="cleanup" Nov 03 01:30:49.806967 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:30:49.803376 4939 memory_manager.go:365] "RemoveStaleState removing state" podUID="e1d8185f-c5b3-4e38-93ea-343306023393" containerName="cleanup" Nov 03 01:30:49.806967 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:30:49.803394 4939 memory_manager.go:365] "RemoveStaleState removing state" podUID="56595211-63c2-491f-9912-b681df440d0a" containerName="cleanup" Nov 03 01:30:49.806967 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:30:49.803413 4939 memory_manager.go:365] "RemoveStaleState removing state" podUID="cd17c69a-8daf-42cb-8791-f2ebbaeaa6c0" containerName="cleanup" Nov 03 01:30:49.806967 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:30:49.803427 4939 memory_manager.go:365] "RemoveStaleState removing state" podUID="e1d8185f-c5b3-4e38-93ea-343306023393" containerName="cleanup" Nov 03 01:30:49.810103 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:30:49.810075 4939 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-b4971a/test-model-b4971a-kserve-5f54798fc5-hc2bn" Nov 03 01:30:49.810978 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:30:49.810952 4939 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-b4971a/test-model-b4971a-kserve-5f54798fc5-hc2bn"] Nov 03 01:30:49.815890 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:30:49.815867 4939 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-b4971a\"/\"openshift-service-ca.crt\"" Nov 03 01:30:49.816748 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:30:49.816059 4939 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-b4971a\"/\"kube-root-ca.crt\"" Nov 03 01:30:49.816748 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:30:49.816113 4939 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-b4971a\"/\"default-dockercfg-v48pv\"" Nov 03 01:30:49.816945 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:30:49.816411 4939 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-b4971a\"/\"test-model-b4971a-kserve-self-signed-certs\"" Nov 03 01:30:49.858025 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:30:49.857980 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:30:49.939466 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:30:49.939434 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1c5e3dbc-2267-4ffa-b4eb-eb25bb89c2a8-tmp-dir\") pod \"test-model-b4971a-kserve-5f54798fc5-hc2bn\" (UID: \"1c5e3dbc-2267-4ffa-b4eb-eb25bb89c2a8\") " pod="ai-tenant-e2e-inf-b-b4971a/test-model-b4971a-kserve-5f54798fc5-hc2bn" Nov 03 01:30:49.939644 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:30:49.939628 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1c5e3dbc-2267-4ffa-b4eb-eb25bb89c2a8-model-cache\") pod \"test-model-b4971a-kserve-5f54798fc5-hc2bn\" (UID: \"1c5e3dbc-2267-4ffa-b4eb-eb25bb89c2a8\") " pod="ai-tenant-e2e-inf-b-b4971a/test-model-b4971a-kserve-5f54798fc5-hc2bn" Nov 03 01:30:49.939685 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:30:49.939666 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1c5e3dbc-2267-4ffa-b4eb-eb25bb89c2a8-home\") pod \"test-model-b4971a-kserve-5f54798fc5-hc2bn\" (UID: \"1c5e3dbc-2267-4ffa-b4eb-eb25bb89c2a8\") " pod="ai-tenant-e2e-inf-b-b4971a/test-model-b4971a-kserve-5f54798fc5-hc2bn" Nov 03 01:30:49.939862 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:30:49.939846 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1c5e3dbc-2267-4ffa-b4eb-eb25bb89c2a8-tls-certs\") pod \"test-model-b4971a-kserve-5f54798fc5-hc2bn\" (UID: \"1c5e3dbc-2267-4ffa-b4eb-eb25bb89c2a8\") " pod="ai-tenant-e2e-inf-b-b4971a/test-model-b4971a-kserve-5f54798fc5-hc2bn" Nov 03 01:30:49.939942 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:30:49.939923 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-99zzj\" (UniqueName: \"kubernetes.io/projected/1c5e3dbc-2267-4ffa-b4eb-eb25bb89c2a8-kube-api-access-99zzj\") pod \"test-model-b4971a-kserve-5f54798fc5-hc2bn\" (UID: \"1c5e3dbc-2267-4ffa-b4eb-eb25bb89c2a8\") " pod="ai-tenant-e2e-inf-b-b4971a/test-model-b4971a-kserve-5f54798fc5-hc2bn" Nov 03 01:30:49.940049 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:30:49.940035 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1c5e3dbc-2267-4ffa-b4eb-eb25bb89c2a8-dshm\") pod \"test-model-b4971a-kserve-5f54798fc5-hc2bn\" (UID: \"1c5e3dbc-2267-4ffa-b4eb-eb25bb89c2a8\") " pod="ai-tenant-e2e-inf-b-b4971a/test-model-b4971a-kserve-5f54798fc5-hc2bn" Nov 03 01:30:49.965003 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:30:49.964974 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:30:50.066777 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:30:50.057451 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-99zzj\" (UniqueName: \"kubernetes.io/projected/1c5e3dbc-2267-4ffa-b4eb-eb25bb89c2a8-kube-api-access-99zzj\") pod \"test-model-b4971a-kserve-5f54798fc5-hc2bn\" (UID: \"1c5e3dbc-2267-4ffa-b4eb-eb25bb89c2a8\") " pod="ai-tenant-e2e-inf-b-b4971a/test-model-b4971a-kserve-5f54798fc5-hc2bn" Nov 03 01:30:50.066777 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:30:50.057547 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1c5e3dbc-2267-4ffa-b4eb-eb25bb89c2a8-dshm\") pod \"test-model-b4971a-kserve-5f54798fc5-hc2bn\" (UID: \"1c5e3dbc-2267-4ffa-b4eb-eb25bb89c2a8\") " pod="ai-tenant-e2e-inf-b-b4971a/test-model-b4971a-kserve-5f54798fc5-hc2bn" Nov 03 01:30:50.066777 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:30:50.057738 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1c5e3dbc-2267-4ffa-b4eb-eb25bb89c2a8-tmp-dir\") pod \"test-model-b4971a-kserve-5f54798fc5-hc2bn\" (UID: \"1c5e3dbc-2267-4ffa-b4eb-eb25bb89c2a8\") " pod="ai-tenant-e2e-inf-b-b4971a/test-model-b4971a-kserve-5f54798fc5-hc2bn" Nov 03 01:30:50.066777 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:30:50.057958 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1c5e3dbc-2267-4ffa-b4eb-eb25bb89c2a8-model-cache\") pod \"test-model-b4971a-kserve-5f54798fc5-hc2bn\" (UID: \"1c5e3dbc-2267-4ffa-b4eb-eb25bb89c2a8\") " pod="ai-tenant-e2e-inf-b-b4971a/test-model-b4971a-kserve-5f54798fc5-hc2bn" Nov 03 01:30:50.066777 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:30:50.058012 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1c5e3dbc-2267-4ffa-b4eb-eb25bb89c2a8-home\") pod \"test-model-b4971a-kserve-5f54798fc5-hc2bn\" (UID: \"1c5e3dbc-2267-4ffa-b4eb-eb25bb89c2a8\") " pod="ai-tenant-e2e-inf-b-b4971a/test-model-b4971a-kserve-5f54798fc5-hc2bn" Nov 03 01:30:50.066777 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:30:50.058214 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1c5e3dbc-2267-4ffa-b4eb-eb25bb89c2a8-tls-certs\") pod \"test-model-b4971a-kserve-5f54798fc5-hc2bn\" (UID: \"1c5e3dbc-2267-4ffa-b4eb-eb25bb89c2a8\") " pod="ai-tenant-e2e-inf-b-b4971a/test-model-b4971a-kserve-5f54798fc5-hc2bn" Nov 03 01:30:50.066777 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:30:50.059139 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1c5e3dbc-2267-4ffa-b4eb-eb25bb89c2a8-tmp-dir\") pod \"test-model-b4971a-kserve-5f54798fc5-hc2bn\" (UID: \"1c5e3dbc-2267-4ffa-b4eb-eb25bb89c2a8\") " pod="ai-tenant-e2e-inf-b-b4971a/test-model-b4971a-kserve-5f54798fc5-hc2bn" Nov 03 01:30:50.066777 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:30:50.060950 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1c5e3dbc-2267-4ffa-b4eb-eb25bb89c2a8-model-cache\") pod \"test-model-b4971a-kserve-5f54798fc5-hc2bn\" (UID: \"1c5e3dbc-2267-4ffa-b4eb-eb25bb89c2a8\") " pod="ai-tenant-e2e-inf-b-b4971a/test-model-b4971a-kserve-5f54798fc5-hc2bn" Nov 03 01:30:50.066777 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:30:50.061260 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1c5e3dbc-2267-4ffa-b4eb-eb25bb89c2a8-home\") pod \"test-model-b4971a-kserve-5f54798fc5-hc2bn\" (UID: \"1c5e3dbc-2267-4ffa-b4eb-eb25bb89c2a8\") " pod="ai-tenant-e2e-inf-b-b4971a/test-model-b4971a-kserve-5f54798fc5-hc2bn" Nov 03 01:30:50.107931 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:30:50.090437 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1c5e3dbc-2267-4ffa-b4eb-eb25bb89c2a8-tls-certs\") pod \"test-model-b4971a-kserve-5f54798fc5-hc2bn\" (UID: \"1c5e3dbc-2267-4ffa-b4eb-eb25bb89c2a8\") " pod="ai-tenant-e2e-inf-b-b4971a/test-model-b4971a-kserve-5f54798fc5-hc2bn" Nov 03 01:30:50.107931 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:30:50.095007 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1c5e3dbc-2267-4ffa-b4eb-eb25bb89c2a8-dshm\") pod \"test-model-b4971a-kserve-5f54798fc5-hc2bn\" (UID: \"1c5e3dbc-2267-4ffa-b4eb-eb25bb89c2a8\") " pod="ai-tenant-e2e-inf-b-b4971a/test-model-b4971a-kserve-5f54798fc5-hc2bn" Nov 03 01:30:50.107931 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:30:50.095721 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-99zzj\" (UniqueName: \"kubernetes.io/projected/1c5e3dbc-2267-4ffa-b4eb-eb25bb89c2a8-kube-api-access-99zzj\") pod \"test-model-b4971a-kserve-5f54798fc5-hc2bn\" (UID: \"1c5e3dbc-2267-4ffa-b4eb-eb25bb89c2a8\") " pod="ai-tenant-e2e-inf-b-b4971a/test-model-b4971a-kserve-5f54798fc5-hc2bn" Nov 03 01:30:50.171692 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:30:50.171652 4939 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-b4971a/test-model-b4971a-kserve-5f54798fc5-hc2bn" Nov 03 01:30:50.560499 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:30:50.560465 4939 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-b4971a/test-model-b4971a-kserve-5f54798fc5-hc2bn"] Nov 03 01:30:50.560951 ip-10-0-140-107 kubenswrapper[4939]: W1103 01:30:50.560627 4939 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod1c5e3dbc_2267_4ffa_b4eb_eb25bb89c2a8.slice/crio-c2d32489c7ccfa8355cf96898a1ce1b66620ab8eb589e0847abeabd9317fdb99 WatchSource:0}: Error finding container c2d32489c7ccfa8355cf96898a1ce1b66620ab8eb589e0847abeabd9317fdb99: Status 404 returned error can't find the container with id c2d32489c7ccfa8355cf96898a1ce1b66620ab8eb589e0847abeabd9317fdb99 Nov 03 01:30:50.629590 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:30:50.628959 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:30:51.001456 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:30:51.001363 4939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-b4971a/test-model-b4971a-kserve-5f54798fc5-hc2bn" event={"ID":"1c5e3dbc-2267-4ffa-b4eb-eb25bb89c2a8","Type":"ContainerStarted","Data":"a4e42ae5cd492a07ddb012238b4a02885354f2001ffb7eb237fbdff3b5bf9014"} Nov 03 01:30:51.001456 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:30:51.001412 4939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-b4971a/test-model-b4971a-kserve-5f54798fc5-hc2bn" event={"ID":"1c5e3dbc-2267-4ffa-b4eb-eb25bb89c2a8","Type":"ContainerStarted","Data":"c2d32489c7ccfa8355cf96898a1ce1b66620ab8eb589e0847abeabd9317fdb99"} Nov 03 01:30:51.020047 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:30:51.019997 4939 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-inf-b-b4971a/test-model-b4971a-kserve-5f54798fc5-hc2bn" podStartSLOduration=2.019983635 podStartE2EDuration="2.019983635s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 01:30:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 01:30:51.017818359 +0000 UTC m=+2999.282153306" watchObservedRunningTime="2026-11-03 01:30:51.019983635 +0000 UTC m=+2999.284318580" Nov 03 01:30:51.172755 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:30:51.172717 4939 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-b-b4971a/test-model-b4971a-kserve-5f54798fc5-hc2bn" Nov 03 01:30:51.478277 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:30:51.478246 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:30:51.973345 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:30:51.973311 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:30:52.697954 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:30:52.697875 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:30:52.894390 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:30:52.894361 4939 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-n42s6_65d134ac-8a25-4c67-b6c7-cd752a098daf/console-operator/2.log" Nov 03 01:30:52.960810 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:30:52.960736 4939 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-n42s6_65d134ac-8a25-4c67-b6c7-cd752a098daf/console-operator/2.log" Nov 03 01:30:53.376039 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:30:53.376011 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:30:54.086656 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:30:54.086622 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:30:54.716419 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:30:54.716379 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:30:55.586150 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:30:55.586101 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:30:56.188109 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:30:56.188071 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:30:56.800126 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:30:56.800084 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:30:57.386353 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:30:57.386317 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:30:57.998191 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:30:57.998157 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:30:58.591283 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:30:58.591239 4939 scope.go:122] "RemoveContainer" containerID="e0f50c3298f7133373574a098c05e9f525bd661943401f2552fd9bd72a95a234" Nov 03 01:30:58.597844 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:30:58.597819 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:30:58.603173 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:30:58.603148 4939 scope.go:122] "RemoveContainer" containerID="e4cd9b1a7a872bc58bdabe9a9f2cbed304f5735100060d680085955560752c69" Nov 03 01:30:59.178812 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:30:59.178775 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:30:59.910192 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:30:59.910161 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:31:00.172949 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:31:00.172834 4939 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-b-b4971a/test-model-b4971a-kserve-5f54798fc5-hc2bn" Nov 03 01:31:00.186793 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:31:00.186765 4939 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-b-b4971a/test-model-b4971a-kserve-5f54798fc5-hc2bn" Nov 03 01:31:00.684443 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:31:00.684413 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:31:01.152770 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:31:01.152738 4939 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-b-b4971a/test-model-b4971a-kserve-5f54798fc5-hc2bn" Nov 03 01:31:01.284957 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:31:01.284883 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:31:01.988101 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:31:01.988066 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:31:02.586743 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:31:02.586700 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:31:03.167852 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:31:03.167820 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:31:03.918266 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:31:03.911262 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:31:04.218344 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:31:04.218248 4939 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-4pw7v"] Nov 03 01:31:04.521772 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:31:04.521681 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:31:04.932966 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:31:04.932930 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:31:05.043848 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:31:05.043820 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:31:05.258986 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:31:05.253295 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:31:06.703773 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:31:06.703730 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:31:07.451762 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:31:07.451718 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:31:08.077545 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:31:08.077513 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:31:08.710160 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:31:08.710122 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:31:09.501325 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:31:09.501292 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:31:10.431775 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:31:10.431736 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:31:11.188846 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:31:11.188778 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:31:11.540381 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:31:11.540291 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:31:11.891185 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:31:11.891141 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:31:12.558159 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:31:12.558124 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:31:13.294663 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:31:13.294601 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:31:13.874979 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:31:13.874872 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:31:14.523974 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:31:14.523934 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:31:15.136171 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:31:15.136136 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:31:15.271491 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:31:15.271286 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:31:15.384419 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:31:15.384381 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:31:15.682503 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:31:15.682467 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:31:15.686598 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:31:15.686561 4939 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-a-6fec8a/test-model-6fec8a-kserve-68c5cd696f-kmgsd"] Nov 03 01:31:15.688262 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:31:15.688236 4939 memory_manager.go:365] "RemoveStaleState removing state" podUID="cd17c69a-8daf-42cb-8791-f2ebbaeaa6c0" containerName="cleanup" Nov 03 01:31:15.688396 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:31:15.688282 4939 memory_manager.go:365] "RemoveStaleState removing state" podUID="e1d8185f-c5b3-4e38-93ea-343306023393" containerName="cleanup" Nov 03 01:31:15.694951 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:31:15.694895 4939 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-6fec8a/test-model-6fec8a-kserve-68c5cd696f-kmgsd"] Nov 03 01:31:15.695056 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:31:15.695004 4939 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-6fec8a/test-model-6fec8a-kserve-68c5cd696f-kmgsd" Nov 03 01:31:15.703749 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:31:15.703705 4939 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-6fec8a\"/\"default-dockercfg-ww49m\"" Nov 03 01:31:15.703898 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:31:15.703748 4939 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-6fec8a\"/\"test-model-6fec8a-kserve-self-signed-certs\"" Nov 03 01:31:15.703898 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:31:15.703759 4939 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-6fec8a\"/\"kube-root-ca.crt\"" Nov 03 01:31:15.703898 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:31:15.703838 4939 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-6fec8a\"/\"openshift-service-ca.crt\"" Nov 03 01:31:15.853935 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:31:15.842763 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:31:15.902187 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:31:15.902154 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cab8e48f-de7c-42aa-93f5-9c7ac27f4cb9-tls-certs\") pod \"test-model-6fec8a-kserve-68c5cd696f-kmgsd\" (UID: \"cab8e48f-de7c-42aa-93f5-9c7ac27f4cb9\") " pod="ai-tenant-e2e-inf-a-6fec8a/test-model-6fec8a-kserve-68c5cd696f-kmgsd" Nov 03 01:31:15.902814 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:31:15.902770 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cab8e48f-de7c-42aa-93f5-9c7ac27f4cb9-tmp-dir\") pod \"test-model-6fec8a-kserve-68c5cd696f-kmgsd\" (UID: \"cab8e48f-de7c-42aa-93f5-9c7ac27f4cb9\") " pod="ai-tenant-e2e-inf-a-6fec8a/test-model-6fec8a-kserve-68c5cd696f-kmgsd" Nov 03 01:31:15.917144 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:31:15.917104 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cab8e48f-de7c-42aa-93f5-9c7ac27f4cb9-model-cache\") pod \"test-model-6fec8a-kserve-68c5cd696f-kmgsd\" (UID: \"cab8e48f-de7c-42aa-93f5-9c7ac27f4cb9\") " pod="ai-tenant-e2e-inf-a-6fec8a/test-model-6fec8a-kserve-68c5cd696f-kmgsd" Nov 03 01:31:15.917527 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:31:15.917508 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/cab8e48f-de7c-42aa-93f5-9c7ac27f4cb9-dshm\") pod \"test-model-6fec8a-kserve-68c5cd696f-kmgsd\" (UID: \"cab8e48f-de7c-42aa-93f5-9c7ac27f4cb9\") " pod="ai-tenant-e2e-inf-a-6fec8a/test-model-6fec8a-kserve-68c5cd696f-kmgsd" Nov 03 01:31:15.917800 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:31:15.917781 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/cab8e48f-de7c-42aa-93f5-9c7ac27f4cb9-home\") pod \"test-model-6fec8a-kserve-68c5cd696f-kmgsd\" (UID: \"cab8e48f-de7c-42aa-93f5-9c7ac27f4cb9\") " pod="ai-tenant-e2e-inf-a-6fec8a/test-model-6fec8a-kserve-68c5cd696f-kmgsd" Nov 03 01:31:15.918174 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:31:15.918151 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7vnb4\" (UniqueName: \"kubernetes.io/projected/cab8e48f-de7c-42aa-93f5-9c7ac27f4cb9-kube-api-access-7vnb4\") pod \"test-model-6fec8a-kserve-68c5cd696f-kmgsd\" (UID: \"cab8e48f-de7c-42aa-93f5-9c7ac27f4cb9\") " pod="ai-tenant-e2e-inf-a-6fec8a/test-model-6fec8a-kserve-68c5cd696f-kmgsd" Nov 03 01:31:16.004532 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:31:16.004413 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:31:16.069163 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:31:16.049654 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7vnb4\" (UniqueName: \"kubernetes.io/projected/cab8e48f-de7c-42aa-93f5-9c7ac27f4cb9-kube-api-access-7vnb4\") pod \"test-model-6fec8a-kserve-68c5cd696f-kmgsd\" (UID: \"cab8e48f-de7c-42aa-93f5-9c7ac27f4cb9\") " pod="ai-tenant-e2e-inf-a-6fec8a/test-model-6fec8a-kserve-68c5cd696f-kmgsd" Nov 03 01:31:16.069163 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:31:16.049714 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cab8e48f-de7c-42aa-93f5-9c7ac27f4cb9-tls-certs\") pod \"test-model-6fec8a-kserve-68c5cd696f-kmgsd\" (UID: \"cab8e48f-de7c-42aa-93f5-9c7ac27f4cb9\") " pod="ai-tenant-e2e-inf-a-6fec8a/test-model-6fec8a-kserve-68c5cd696f-kmgsd" Nov 03 01:31:16.069163 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:31:16.049949 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cab8e48f-de7c-42aa-93f5-9c7ac27f4cb9-tmp-dir\") pod \"test-model-6fec8a-kserve-68c5cd696f-kmgsd\" (UID: \"cab8e48f-de7c-42aa-93f5-9c7ac27f4cb9\") " pod="ai-tenant-e2e-inf-a-6fec8a/test-model-6fec8a-kserve-68c5cd696f-kmgsd" Nov 03 01:31:16.069163 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:31:16.050056 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cab8e48f-de7c-42aa-93f5-9c7ac27f4cb9-model-cache\") pod \"test-model-6fec8a-kserve-68c5cd696f-kmgsd\" (UID: \"cab8e48f-de7c-42aa-93f5-9c7ac27f4cb9\") " pod="ai-tenant-e2e-inf-a-6fec8a/test-model-6fec8a-kserve-68c5cd696f-kmgsd" Nov 03 01:31:16.069163 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:31:16.050151 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/cab8e48f-de7c-42aa-93f5-9c7ac27f4cb9-dshm\") pod \"test-model-6fec8a-kserve-68c5cd696f-kmgsd\" (UID: \"cab8e48f-de7c-42aa-93f5-9c7ac27f4cb9\") " pod="ai-tenant-e2e-inf-a-6fec8a/test-model-6fec8a-kserve-68c5cd696f-kmgsd" Nov 03 01:31:16.069163 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:31:16.050257 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/cab8e48f-de7c-42aa-93f5-9c7ac27f4cb9-home\") pod \"test-model-6fec8a-kserve-68c5cd696f-kmgsd\" (UID: \"cab8e48f-de7c-42aa-93f5-9c7ac27f4cb9\") " pod="ai-tenant-e2e-inf-a-6fec8a/test-model-6fec8a-kserve-68c5cd696f-kmgsd" Nov 03 01:31:16.069163 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:31:16.050878 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/cab8e48f-de7c-42aa-93f5-9c7ac27f4cb9-home\") pod \"test-model-6fec8a-kserve-68c5cd696f-kmgsd\" (UID: \"cab8e48f-de7c-42aa-93f5-9c7ac27f4cb9\") " pod="ai-tenant-e2e-inf-a-6fec8a/test-model-6fec8a-kserve-68c5cd696f-kmgsd" Nov 03 01:31:16.069163 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:31:16.051165 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cab8e48f-de7c-42aa-93f5-9c7ac27f4cb9-model-cache\") pod \"test-model-6fec8a-kserve-68c5cd696f-kmgsd\" (UID: \"cab8e48f-de7c-42aa-93f5-9c7ac27f4cb9\") " pod="ai-tenant-e2e-inf-a-6fec8a/test-model-6fec8a-kserve-68c5cd696f-kmgsd" Nov 03 01:31:16.069163 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:31:16.052502 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cab8e48f-de7c-42aa-93f5-9c7ac27f4cb9-tmp-dir\") pod \"test-model-6fec8a-kserve-68c5cd696f-kmgsd\" (UID: \"cab8e48f-de7c-42aa-93f5-9c7ac27f4cb9\") " pod="ai-tenant-e2e-inf-a-6fec8a/test-model-6fec8a-kserve-68c5cd696f-kmgsd" Nov 03 01:31:16.077127 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:31:16.077052 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/cab8e48f-de7c-42aa-93f5-9c7ac27f4cb9-dshm\") pod \"test-model-6fec8a-kserve-68c5cd696f-kmgsd\" (UID: \"cab8e48f-de7c-42aa-93f5-9c7ac27f4cb9\") " pod="ai-tenant-e2e-inf-a-6fec8a/test-model-6fec8a-kserve-68c5cd696f-kmgsd" Nov 03 01:31:16.077318 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:31:16.077203 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cab8e48f-de7c-42aa-93f5-9c7ac27f4cb9-tls-certs\") pod \"test-model-6fec8a-kserve-68c5cd696f-kmgsd\" (UID: \"cab8e48f-de7c-42aa-93f5-9c7ac27f4cb9\") " pod="ai-tenant-e2e-inf-a-6fec8a/test-model-6fec8a-kserve-68c5cd696f-kmgsd" Nov 03 01:31:16.077318 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:31:16.077291 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7vnb4\" (UniqueName: \"kubernetes.io/projected/cab8e48f-de7c-42aa-93f5-9c7ac27f4cb9-kube-api-access-7vnb4\") pod \"test-model-6fec8a-kserve-68c5cd696f-kmgsd\" (UID: \"cab8e48f-de7c-42aa-93f5-9c7ac27f4cb9\") " pod="ai-tenant-e2e-inf-a-6fec8a/test-model-6fec8a-kserve-68c5cd696f-kmgsd" Nov 03 01:31:16.301054 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:31:16.300966 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:31:16.314745 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:31:16.314718 4939 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-6fec8a/test-model-6fec8a-kserve-68c5cd696f-kmgsd" Nov 03 01:31:16.700815 ip-10-0-140-107 kubenswrapper[4939]: W1103 01:31:16.700776 4939 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podcab8e48f_de7c_42aa_93f5_9c7ac27f4cb9.slice/crio-3f1ebeeb7e2ac927f9f857306aad4069279692eab9d03bdb658da7b32ffad7f3 WatchSource:0}: Error finding container 3f1ebeeb7e2ac927f9f857306aad4069279692eab9d03bdb658da7b32ffad7f3: Status 404 returned error can't find the container with id 3f1ebeeb7e2ac927f9f857306aad4069279692eab9d03bdb658da7b32ffad7f3 Nov 03 01:31:16.701269 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:31:16.701243 4939 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-6fec8a/test-model-6fec8a-kserve-68c5cd696f-kmgsd"] Nov 03 01:31:17.091403 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:31:17.091363 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:31:17.641991 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:31:17.641947 4939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-6fec8a/test-model-6fec8a-kserve-68c5cd696f-kmgsd" event={"ID":"cab8e48f-de7c-42aa-93f5-9c7ac27f4cb9","Type":"ContainerStarted","Data":"812f97ed476361456515aa4af72cc94b9764614d59841af53fcacc924a407789"} Nov 03 01:31:17.642437 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:31:17.642005 4939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-6fec8a/test-model-6fec8a-kserve-68c5cd696f-kmgsd" event={"ID":"cab8e48f-de7c-42aa-93f5-9c7ac27f4cb9","Type":"ContainerStarted","Data":"3f1ebeeb7e2ac927f9f857306aad4069279692eab9d03bdb658da7b32ffad7f3"} Nov 03 01:31:17.658194 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:31:17.658144 4939 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-inf-a-6fec8a/test-model-6fec8a-kserve-68c5cd696f-kmgsd" podStartSLOduration=2.658130985 podStartE2EDuration="2.658130985s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 01:31:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 01:31:17.657414928 +0000 UTC m=+3025.921749876" watchObservedRunningTime="2026-11-03 01:31:17.658130985 +0000 UTC m=+3025.922465929" Nov 03 01:31:17.772128 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:31:17.772083 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:31:18.040440 ip-10-0-140-107 kubenswrapper[4939]: E1103 01:31:18.040342 4939 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-sthjh" containerName="extractor" Nov 03 01:31:18.315081 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:31:18.314981 4939 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-a-6fec8a/test-model-6fec8a-kserve-68c5cd696f-kmgsd" Nov 03 01:31:18.610854 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:31:18.610822 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:31:19.292881 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:31:19.292839 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:31:20.066075 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:31:20.066028 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:31:20.822089 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:31:20.822041 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:31:21.480517 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:31:21.480473 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:31:22.210867 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:31:22.210815 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:31:22.899036 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:31:22.899006 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:31:23.583900 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:31:23.583853 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:31:24.268645 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:31:24.268608 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:31:24.428302 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:31:24.428238 4939 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-b-5dab6a/test-model-5dab6a-kserve-5c4f47747c-z44hb"] Nov 03 01:31:24.443046 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:31:24.443014 4939 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-5dab6a/test-model-5dab6a-kserve-5c4f47747c-z44hb" Nov 03 01:31:24.449357 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:31:24.449317 4939 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-5dab6a/test-model-5dab6a-kserve-5c4f47747c-z44hb"] Nov 03 01:31:24.456174 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:31:24.456128 4939 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-5dab6a\"/\"openshift-service-ca.crt\"" Nov 03 01:31:24.456512 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:31:24.456478 4939 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-5dab6a\"/\"test-model-5dab6a-kserve-self-signed-certs\"" Nov 03 01:31:24.456692 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:31:24.456675 4939 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-5dab6a\"/\"default-dockercfg-xvgg9\"" Nov 03 01:31:24.456922 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:31:24.456879 4939 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-5dab6a\"/\"kube-root-ca.crt\"" Nov 03 01:31:24.505424 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:31:24.505389 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9a691811-c7b4-4da0-892d-13d98fb3a0b0-tmp-dir\") pod \"test-model-5dab6a-kserve-5c4f47747c-z44hb\" (UID: \"9a691811-c7b4-4da0-892d-13d98fb3a0b0\") " pod="ai-tenant-e2e-inf-b-5dab6a/test-model-5dab6a-kserve-5c4f47747c-z44hb" Nov 03 01:31:24.505612 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:31:24.505453 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9a691811-c7b4-4da0-892d-13d98fb3a0b0-model-cache\") pod \"test-model-5dab6a-kserve-5c4f47747c-z44hb\" (UID: \"9a691811-c7b4-4da0-892d-13d98fb3a0b0\") " pod="ai-tenant-e2e-inf-b-5dab6a/test-model-5dab6a-kserve-5c4f47747c-z44hb" Nov 03 01:31:24.505612 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:31:24.505578 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9a691811-c7b4-4da0-892d-13d98fb3a0b0-dshm\") pod \"test-model-5dab6a-kserve-5c4f47747c-z44hb\" (UID: \"9a691811-c7b4-4da0-892d-13d98fb3a0b0\") " pod="ai-tenant-e2e-inf-b-5dab6a/test-model-5dab6a-kserve-5c4f47747c-z44hb" Nov 03 01:31:24.505714 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:31:24.505617 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xdct6\" (UniqueName: \"kubernetes.io/projected/9a691811-c7b4-4da0-892d-13d98fb3a0b0-kube-api-access-xdct6\") pod \"test-model-5dab6a-kserve-5c4f47747c-z44hb\" (UID: \"9a691811-c7b4-4da0-892d-13d98fb3a0b0\") " pod="ai-tenant-e2e-inf-b-5dab6a/test-model-5dab6a-kserve-5c4f47747c-z44hb" Nov 03 01:31:24.505879 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:31:24.505865 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9a691811-c7b4-4da0-892d-13d98fb3a0b0-tls-certs\") pod \"test-model-5dab6a-kserve-5c4f47747c-z44hb\" (UID: \"9a691811-c7b4-4da0-892d-13d98fb3a0b0\") " pod="ai-tenant-e2e-inf-b-5dab6a/test-model-5dab6a-kserve-5c4f47747c-z44hb" Nov 03 01:31:24.505966 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:31:24.505897 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9a691811-c7b4-4da0-892d-13d98fb3a0b0-home\") pod \"test-model-5dab6a-kserve-5c4f47747c-z44hb\" (UID: \"9a691811-c7b4-4da0-892d-13d98fb3a0b0\") " pod="ai-tenant-e2e-inf-b-5dab6a/test-model-5dab6a-kserve-5c4f47747c-z44hb" Nov 03 01:31:24.524048 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:31:24.523969 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:31:24.607486 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:31:24.607450 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9a691811-c7b4-4da0-892d-13d98fb3a0b0-home\") pod \"test-model-5dab6a-kserve-5c4f47747c-z44hb\" (UID: \"9a691811-c7b4-4da0-892d-13d98fb3a0b0\") " pod="ai-tenant-e2e-inf-b-5dab6a/test-model-5dab6a-kserve-5c4f47747c-z44hb" Nov 03 01:31:24.608020 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:31:24.607541 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9a691811-c7b4-4da0-892d-13d98fb3a0b0-tmp-dir\") pod \"test-model-5dab6a-kserve-5c4f47747c-z44hb\" (UID: \"9a691811-c7b4-4da0-892d-13d98fb3a0b0\") " pod="ai-tenant-e2e-inf-b-5dab6a/test-model-5dab6a-kserve-5c4f47747c-z44hb" Nov 03 01:31:24.608020 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:31:24.607594 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9a691811-c7b4-4da0-892d-13d98fb3a0b0-model-cache\") pod \"test-model-5dab6a-kserve-5c4f47747c-z44hb\" (UID: \"9a691811-c7b4-4da0-892d-13d98fb3a0b0\") " pod="ai-tenant-e2e-inf-b-5dab6a/test-model-5dab6a-kserve-5c4f47747c-z44hb" Nov 03 01:31:24.608020 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:31:24.607652 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9a691811-c7b4-4da0-892d-13d98fb3a0b0-dshm\") pod \"test-model-5dab6a-kserve-5c4f47747c-z44hb\" (UID: \"9a691811-c7b4-4da0-892d-13d98fb3a0b0\") " pod="ai-tenant-e2e-inf-b-5dab6a/test-model-5dab6a-kserve-5c4f47747c-z44hb" Nov 03 01:31:24.608020 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:31:24.607678 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xdct6\" (UniqueName: \"kubernetes.io/projected/9a691811-c7b4-4da0-892d-13d98fb3a0b0-kube-api-access-xdct6\") pod \"test-model-5dab6a-kserve-5c4f47747c-z44hb\" (UID: \"9a691811-c7b4-4da0-892d-13d98fb3a0b0\") " pod="ai-tenant-e2e-inf-b-5dab6a/test-model-5dab6a-kserve-5c4f47747c-z44hb" Nov 03 01:31:24.608020 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:31:24.607891 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9a691811-c7b4-4da0-892d-13d98fb3a0b0-home\") pod \"test-model-5dab6a-kserve-5c4f47747c-z44hb\" (UID: \"9a691811-c7b4-4da0-892d-13d98fb3a0b0\") " pod="ai-tenant-e2e-inf-b-5dab6a/test-model-5dab6a-kserve-5c4f47747c-z44hb" Nov 03 01:31:24.608020 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:31:24.607997 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9a691811-c7b4-4da0-892d-13d98fb3a0b0-tmp-dir\") pod \"test-model-5dab6a-kserve-5c4f47747c-z44hb\" (UID: \"9a691811-c7b4-4da0-892d-13d98fb3a0b0\") " pod="ai-tenant-e2e-inf-b-5dab6a/test-model-5dab6a-kserve-5c4f47747c-z44hb" Nov 03 01:31:24.608430 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:31:24.608024 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9a691811-c7b4-4da0-892d-13d98fb3a0b0-tls-certs\") pod \"test-model-5dab6a-kserve-5c4f47747c-z44hb\" (UID: \"9a691811-c7b4-4da0-892d-13d98fb3a0b0\") " pod="ai-tenant-e2e-inf-b-5dab6a/test-model-5dab6a-kserve-5c4f47747c-z44hb" Nov 03 01:31:24.608430 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:31:24.608050 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9a691811-c7b4-4da0-892d-13d98fb3a0b0-model-cache\") pod \"test-model-5dab6a-kserve-5c4f47747c-z44hb\" (UID: \"9a691811-c7b4-4da0-892d-13d98fb3a0b0\") " pod="ai-tenant-e2e-inf-b-5dab6a/test-model-5dab6a-kserve-5c4f47747c-z44hb" Nov 03 01:31:24.610223 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:31:24.610201 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9a691811-c7b4-4da0-892d-13d98fb3a0b0-dshm\") pod \"test-model-5dab6a-kserve-5c4f47747c-z44hb\" (UID: \"9a691811-c7b4-4da0-892d-13d98fb3a0b0\") " pod="ai-tenant-e2e-inf-b-5dab6a/test-model-5dab6a-kserve-5c4f47747c-z44hb" Nov 03 01:31:24.610430 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:31:24.610411 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9a691811-c7b4-4da0-892d-13d98fb3a0b0-tls-certs\") pod \"test-model-5dab6a-kserve-5c4f47747c-z44hb\" (UID: \"9a691811-c7b4-4da0-892d-13d98fb3a0b0\") " pod="ai-tenant-e2e-inf-b-5dab6a/test-model-5dab6a-kserve-5c4f47747c-z44hb" Nov 03 01:31:24.666657 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:31:24.666027 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xdct6\" (UniqueName: \"kubernetes.io/projected/9a691811-c7b4-4da0-892d-13d98fb3a0b0-kube-api-access-xdct6\") pod \"test-model-5dab6a-kserve-5c4f47747c-z44hb\" (UID: \"9a691811-c7b4-4da0-892d-13d98fb3a0b0\") " pod="ai-tenant-e2e-inf-b-5dab6a/test-model-5dab6a-kserve-5c4f47747c-z44hb" Nov 03 01:31:24.788622 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:31:24.788238 4939 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-5dab6a/test-model-5dab6a-kserve-5c4f47747c-z44hb" Nov 03 01:31:25.165445 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:31:25.165407 4939 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-5dab6a/test-model-5dab6a-kserve-5c4f47747c-z44hb"] Nov 03 01:31:25.316778 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:31:25.316733 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:31:25.801864 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:31:25.801823 4939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-5dab6a/test-model-5dab6a-kserve-5c4f47747c-z44hb" event={"ID":"9a691811-c7b4-4da0-892d-13d98fb3a0b0","Type":"ContainerStarted","Data":"19db77487df3cc3019000d70933d954c52519531c1ed657adfadd7cf22dda66e"} Nov 03 01:31:25.801864 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:31:25.801871 4939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-5dab6a/test-model-5dab6a-kserve-5c4f47747c-z44hb" event={"ID":"9a691811-c7b4-4da0-892d-13d98fb3a0b0","Type":"ContainerStarted","Data":"a2d2816d572ec076f2f0ddc17582c0a45a3581903ed9c01be347da9733c00553"} Nov 03 01:31:25.822083 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:31:25.822031 4939 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-inf-b-5dab6a/test-model-5dab6a-kserve-5c4f47747c-z44hb" podStartSLOduration=1.82201743 podStartE2EDuration="1.82201743s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 01:31:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 01:31:25.81848791 +0000 UTC m=+3034.082822860" watchObservedRunningTime="2026-11-03 01:31:25.82201743 +0000 UTC m=+3034.086352374" Nov 03 01:31:25.979129 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:31:25.979095 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:31:26.315033 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:31:26.314994 4939 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-a-6fec8a/test-model-6fec8a-kserve-68c5cd696f-kmgsd" Nov 03 01:31:26.328264 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:31:26.328237 4939 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-a-6fec8a/test-model-6fec8a-kserve-68c5cd696f-kmgsd" Nov 03 01:31:26.610784 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:31:26.610752 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:31:26.789672 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:31:26.789640 4939 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-b-5dab6a/test-model-5dab6a-kserve-5c4f47747c-z44hb" Nov 03 01:31:26.822531 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:31:26.822498 4939 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-a-6fec8a/test-model-6fec8a-kserve-68c5cd696f-kmgsd" Nov 03 01:31:27.317939 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:31:27.317892 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:31:28.395197 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:31:28.395162 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:31:29.106052 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:31:29.106019 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:31:29.897687 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:31:29.897649 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:31:30.483598 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:31:30.483561 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:31:31.098365 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:31:31.098334 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:31:31.888712 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:31:31.888679 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:31:32.538153 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:31:32.538120 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:31:33.275154 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:31:33.275119 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:31:33.878078 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:31:33.878044 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:31:34.680369 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:31:34.680318 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:31:34.789614 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:31:34.789582 4939 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-b-5dab6a/test-model-5dab6a-kserve-5c4f47747c-z44hb" Nov 03 01:31:34.803802 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:31:34.803774 4939 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-b-5dab6a/test-model-5dab6a-kserve-5c4f47747c-z44hb" Nov 03 01:31:34.898652 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:31:34.898620 4939 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-b-5dab6a/test-model-5dab6a-kserve-5c4f47747c-z44hb" Nov 03 01:31:35.476222 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:31:35.476191 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:31:36.100307 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:31:36.100273 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:31:36.783250 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:31:36.783212 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:31:37.380235 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:31:37.380206 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:31:38.120804 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:31:38.120763 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:31:38.782261 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:31:38.782227 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:31:39.485453 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:31:39.485417 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:31:40.110577 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:31:40.110543 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:31:40.793515 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:31:40.793472 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:31:41.377427 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:31:41.377396 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:31:42.030800 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:31:42.030768 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:31:42.166805 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:31:42.166771 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:31:42.774684 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:31:42.774647 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:31:43.383064 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:31:43.383033 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:31:44.019896 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:31:44.019840 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:31:44.660843 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:31:44.660811 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:31:45.081805 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:31:45.081778 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:31:45.315868 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:31:45.315832 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:31:46.056515 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:31:46.056475 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:31:46.689352 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:31:46.689307 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:31:47.288923 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:31:47.288866 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:31:48.035990 ip-10-0-140-107 kubenswrapper[4939]: E1103 01:31:48.035949 4939 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-sthjh" containerName="extractor" Nov 03 01:31:48.063343 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:31:48.063312 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:31:48.581756 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:31:48.581726 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:31:48.777654 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:31:48.777613 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:31:49.473134 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:31:49.473099 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:31:50.174557 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:31:50.174524 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:31:50.783551 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:31:50.783516 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:31:51.379317 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:31:51.379279 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:31:51.999877 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:31:51.999837 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:31:52.665049 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:31:52.662859 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:31:53.200509 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:31:53.200465 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:31:53.995956 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:31:53.995893 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:31:54.588794 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:31:54.588756 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:31:55.252250 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:31:55.252211 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:31:55.872627 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:31:55.872590 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:31:56.479591 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:31:56.479560 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:31:57.198657 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:31:57.198624 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:31:57.887235 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:31:57.887201 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:31:58.599701 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:31:58.599668 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:31:59.185759 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:31:59.185713 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:31:59.813674 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:31:59.813636 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:32:00.507050 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:32:00.507016 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:32:01.180536 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:32:01.180465 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:32:01.793162 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:32:01.793123 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:32:02.580477 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:32:02.580441 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:32:02.815624 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:32:02.815592 4939 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-4pw7v"] Nov 03 01:32:03.244467 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:32:03.244434 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:32:04.543885 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:32:04.543824 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:32:05.314249 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:32:05.314204 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:32:06.107669 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:32:06.107613 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:32:06.702198 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:32:06.702157 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:32:07.478468 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:32:07.478435 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:32:08.180644 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:32:08.180606 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:32:08.798021 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:32:08.797974 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:32:09.592768 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:32:09.592728 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:32:10.184856 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:32:10.184825 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:32:10.787497 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:32:10.787467 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:32:11.474414 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:32:11.474381 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:32:12.092424 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:32:12.092365 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:32:12.781166 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:32:12.781133 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:32:13.474021 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:32:13.473969 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:32:14.125634 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:32:14.125588 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:32:14.739548 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:32:14.739510 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:32:15.365614 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:32:15.365580 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:32:16.098126 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:32:16.098090 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:32:16.743025 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:32:16.742989 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:32:17.461002 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:32:17.460969 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:32:18.035754 ip-10-0-140-107 kubenswrapper[4939]: E1103 01:32:18.035716 4939 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-sthjh" containerName="extractor" Nov 03 01:32:18.135650 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:32:18.135611 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:32:18.795709 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:32:18.795669 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:32:19.377890 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:32:19.377847 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:32:20.000190 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:32:20.000152 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:32:20.783175 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:32:20.783143 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:32:21.475232 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:32:21.475194 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:32:22.182638 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:32:22.182599 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:32:22.886649 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:32:22.886610 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:32:23.608882 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:32:23.608846 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:32:24.265853 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:32:24.265817 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:32:24.872667 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:32:24.872633 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:32:25.587090 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:32:25.587053 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:32:26.181184 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:32:26.181147 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:32:26.889409 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:32:26.889370 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:32:27.492083 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:32:27.492038 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:32:28.083369 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:32:28.083337 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:32:28.700120 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:32:28.700079 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:32:29.315805 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:32:29.315771 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:32:30.268324 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:32:30.268282 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:32:30.910000 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:32:30.909963 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:32:31.538777 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:32:31.538746 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:32:33.359765 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:32:33.359686 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:32:33.660054 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:32:33.659974 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:32:34.281092 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:32:34.281043 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:32:34.971828 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:32:34.971780 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:32:35.570277 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:32:35.570227 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:32:36.204902 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:32:36.204867 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:32:36.879170 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:32:36.879134 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:32:37.474937 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:32:37.474874 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:32:38.107124 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:32:38.107090 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:32:38.776190 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:32:38.776156 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:32:38.892348 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:32:38.892313 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:32:39.380537 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:32:39.380501 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:32:40.119017 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:32:40.118973 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:32:40.797403 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:32:40.797370 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:32:41.421118 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:32:41.421090 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:32:42.092876 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:32:42.092847 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:32:42.797037 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:32:42.797008 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:32:43.588844 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:32:43.588808 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:32:44.198395 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:32:44.198363 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:32:44.807019 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:32:44.806971 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:32:45.139951 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:32:45.122173 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:32:45.460585 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:32:45.460499 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:32:45.606794 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:32:45.606755 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:32:46.274210 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:32:46.274176 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:32:46.978221 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:32:46.978182 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:32:47.576474 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:32:47.576439 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:32:48.034404 ip-10-0-140-107 kubenswrapper[4939]: E1103 01:32:48.034362 4939 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-sthjh" containerName="extractor" Nov 03 01:32:48.284038 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:32:48.284001 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:32:48.903252 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:32:48.903217 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:32:49.482731 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:32:49.482694 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:32:50.163070 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:32:50.163034 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:32:50.783779 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:32:50.783733 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:32:51.396692 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:32:51.396655 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:32:51.975287 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:32:51.975256 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:32:52.663848 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:32:52.663811 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:32:53.272673 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:32:53.272639 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:32:54.090932 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:32:54.088097 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:32:54.709247 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:32:54.709210 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:32:55.482969 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:32:55.482898 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:32:56.078184 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:32:56.078154 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:32:56.710706 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:32:56.710637 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:32:57.455364 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:32:57.455327 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:32:58.079309 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:32:58.079262 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:32:58.759179 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:32:58.759142 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:32:59.364570 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:32:59.364537 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:33:00.076335 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:33:00.076300 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:33:00.770899 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:33:00.770861 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:33:01.387559 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:33:01.387521 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:33:02.186001 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:33:02.185963 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:33:02.869373 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:33:02.869325 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:33:03.478232 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:33:03.478192 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:33:04.148060 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:33:04.148026 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:33:04.262262 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:33:04.262109 4939 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-4pw7v"] Nov 03 01:33:04.991644 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:33:04.991610 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:33:05.797158 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:33:05.797100 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:33:05.992553 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:33:05.992513 4939 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-ee618c/test-model-ee618c-kserve-844f5b79dc-f6kl7"] Nov 03 01:33:06.061160 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:33:06.061068 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:33:06.732933 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:33:06.727043 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:33:07.366654 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:33:07.366619 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:33:08.172140 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:33:08.172106 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:33:08.984121 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:33:08.984091 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:33:09.688812 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:33:09.688781 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:33:10.449250 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:33:10.449208 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:33:11.282080 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:33:11.282036 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:33:11.946735 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:33:11.946697 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:33:12.586542 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:33:12.586508 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:33:12.698204 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:33:12.698162 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:33:13.218904 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:33:13.218864 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:33:13.373944 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:33:13.368402 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:33:13.503984 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:33:13.503351 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:33:14.220649 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:33:14.220575 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:33:14.833158 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:33:14.833123 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:33:15.568716 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:33:15.568676 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:33:16.484824 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:33:16.484784 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:33:16.993334 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:33:16.993305 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:33:17.121141 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:33:17.121104 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:33:17.245949 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:33:17.245852 4939 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-5bfc51/test-model-5bfc51-kserve-5c5676fcf9-tssvf"] Nov 03 01:33:17.310620 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:33:17.310592 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:33:17.442351 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:33:17.442317 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:33:17.781184 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:33:17.781139 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:33:18.035191 ip-10-0-140-107 kubenswrapper[4939]: E1103 01:33:18.035108 4939 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-sthjh" containerName="extractor" Nov 03 01:33:18.161072 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:33:18.161037 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:33:18.291003 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:33:18.290887 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:33:18.913318 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:33:18.913281 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:33:19.043944 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:33:19.043889 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:33:19.697593 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:33:19.697560 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:33:20.396779 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:33:20.396750 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:33:21.019841 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:33:21.019794 4939 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-inf-a-ee618c/test-model-ee618c-kserve-844f5b79dc-f6kl7" podUID="abab7b3b-f57d-4abe-be59-295e4ab8a769" containerName="main" containerID="cri-o://78e5d343d7e82472c3033db211fd34235eaf4ffc481172518a49f59e95136f13" gracePeriod=45 Nov 03 01:33:21.178591 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:33:21.178543 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:33:21.398408 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:33:21.398374 4939 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-ee618c/test-model-ee618c-kserve-844f5b79dc-f6kl7" Nov 03 01:33:21.477157 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:33:21.477128 4939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/abab7b3b-f57d-4abe-be59-295e4ab8a769-dshm\" (UniqueName: \"kubernetes.io/empty-dir/abab7b3b-f57d-4abe-be59-295e4ab8a769-dshm\") pod \"abab7b3b-f57d-4abe-be59-295e4ab8a769\" (UID: \"abab7b3b-f57d-4abe-be59-295e4ab8a769\") " Nov 03 01:33:21.477345 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:33:21.477169 4939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/abab7b3b-f57d-4abe-be59-295e4ab8a769-tls-certs\" (UniqueName: \"kubernetes.io/secret/abab7b3b-f57d-4abe-be59-295e4ab8a769-tls-certs\") pod \"abab7b3b-f57d-4abe-be59-295e4ab8a769\" (UID: \"abab7b3b-f57d-4abe-be59-295e4ab8a769\") " Nov 03 01:33:21.477345 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:33:21.477222 4939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/abab7b3b-f57d-4abe-be59-295e4ab8a769-home\" (UniqueName: \"kubernetes.io/empty-dir/abab7b3b-f57d-4abe-be59-295e4ab8a769-home\") pod \"abab7b3b-f57d-4abe-be59-295e4ab8a769\" (UID: \"abab7b3b-f57d-4abe-be59-295e4ab8a769\") " Nov 03 01:33:21.477345 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:33:21.477267 4939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/abab7b3b-f57d-4abe-be59-295e4ab8a769-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/abab7b3b-f57d-4abe-be59-295e4ab8a769-model-cache\") pod \"abab7b3b-f57d-4abe-be59-295e4ab8a769\" (UID: \"abab7b3b-f57d-4abe-be59-295e4ab8a769\") " Nov 03 01:33:21.477549 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:33:21.477532 4939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/abab7b3b-f57d-4abe-be59-295e4ab8a769-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/abab7b3b-f57d-4abe-be59-295e4ab8a769-tmp-dir\") pod \"abab7b3b-f57d-4abe-be59-295e4ab8a769\" (UID: \"abab7b3b-f57d-4abe-be59-295e4ab8a769\") " Nov 03 01:33:21.477713 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:33:21.477689 4939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/abab7b3b-f57d-4abe-be59-295e4ab8a769-model-cache" pod "abab7b3b-f57d-4abe-be59-295e4ab8a769" (UID: "abab7b3b-f57d-4abe-be59-295e4ab8a769"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 01:33:21.477799 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:33:21.477779 4939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/abab7b3b-f57d-4abe-be59-295e4ab8a769-home" pod "abab7b3b-f57d-4abe-be59-295e4ab8a769" (UID: "abab7b3b-f57d-4abe-be59-295e4ab8a769"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 01:33:21.477872 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:33:21.477793 4939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/abab7b3b-f57d-4abe-be59-295e4ab8a769-kube-api-access-vrr2k\" (UniqueName: \"kubernetes.io/projected/abab7b3b-f57d-4abe-be59-295e4ab8a769-kube-api-access-vrr2k\") pod \"abab7b3b-f57d-4abe-be59-295e4ab8a769\" (UID: \"abab7b3b-f57d-4abe-be59-295e4ab8a769\") " Nov 03 01:33:21.477947 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:33:21.477880 4939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/abab7b3b-f57d-4abe-be59-295e4ab8a769-tmp-dir" pod "abab7b3b-f57d-4abe-be59-295e4ab8a769" (UID: "abab7b3b-f57d-4abe-be59-295e4ab8a769"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 01:33:21.478893 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:33:21.478863 4939 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/abab7b3b-f57d-4abe-be59-295e4ab8a769-model-cache\") on node \"ip-10-0-140-107.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:33:21.479033 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:33:21.478897 4939 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/abab7b3b-f57d-4abe-be59-295e4ab8a769-tmp-dir\") on node \"ip-10-0-140-107.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:33:21.479033 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:33:21.478938 4939 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/abab7b3b-f57d-4abe-be59-295e4ab8a769-home\") on node \"ip-10-0-140-107.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:33:21.480184 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:33:21.480158 4939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/abab7b3b-f57d-4abe-be59-295e4ab8a769-dshm" pod "abab7b3b-f57d-4abe-be59-295e4ab8a769" (UID: "abab7b3b-f57d-4abe-be59-295e4ab8a769"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 01:33:21.480279 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:33:21.480173 4939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/abab7b3b-f57d-4abe-be59-295e4ab8a769-tls-certs" pod "abab7b3b-f57d-4abe-be59-295e4ab8a769" (UID: "abab7b3b-f57d-4abe-be59-295e4ab8a769"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 01:33:21.480415 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:33:21.480393 4939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/abab7b3b-f57d-4abe-be59-295e4ab8a769-kube-api-access-vrr2k" pod "abab7b3b-f57d-4abe-be59-295e4ab8a769" (UID: "abab7b3b-f57d-4abe-be59-295e4ab8a769"). InnerVolumeSpecName "kube-api-access-vrr2k". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 01:33:21.581286 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:33:21.581252 4939 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vrr2k\" (UniqueName: \"kubernetes.io/projected/abab7b3b-f57d-4abe-be59-295e4ab8a769-kube-api-access-vrr2k\") on node \"ip-10-0-140-107.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:33:21.581426 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:33:21.581290 4939 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/abab7b3b-f57d-4abe-be59-295e4ab8a769-dshm\") on node \"ip-10-0-140-107.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:33:21.581426 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:33:21.581307 4939 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/abab7b3b-f57d-4abe-be59-295e4ab8a769-tls-certs\") on node \"ip-10-0-140-107.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:33:21.703188 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:33:21.703104 4939 generic.go:356] "Generic (PLEG): container finished" podID="abab7b3b-f57d-4abe-be59-295e4ab8a769" containerID="78e5d343d7e82472c3033db211fd34235eaf4ffc481172518a49f59e95136f13" exitCode=0 Nov 03 01:33:21.703188 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:33:21.703145 4939 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-ee618c/test-model-ee618c-kserve-844f5b79dc-f6kl7" Nov 03 01:33:21.703406 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:33:21.703180 4939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-ee618c/test-model-ee618c-kserve-844f5b79dc-f6kl7" event={"ID":"abab7b3b-f57d-4abe-be59-295e4ab8a769","Type":"ContainerDied","Data":"78e5d343d7e82472c3033db211fd34235eaf4ffc481172518a49f59e95136f13"} Nov 03 01:33:21.703406 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:33:21.703228 4939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-ee618c/test-model-ee618c-kserve-844f5b79dc-f6kl7" event={"ID":"abab7b3b-f57d-4abe-be59-295e4ab8a769","Type":"ContainerDied","Data":"cef7c24d87f62643e1c865788528ddd2ce9fcf49822ba0261c8f7892e9a19e38"} Nov 03 01:33:21.703406 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:33:21.703250 4939 scope.go:122] "RemoveContainer" containerID="78e5d343d7e82472c3033db211fd34235eaf4ffc481172518a49f59e95136f13" Nov 03 01:33:21.717476 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:33:21.717452 4939 scope.go:122] "RemoveContainer" containerID="78e5d343d7e82472c3033db211fd34235eaf4ffc481172518a49f59e95136f13" Nov 03 01:33:21.717746 ip-10-0-140-107 kubenswrapper[4939]: E1103 01:33:21.717726 4939 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"78e5d343d7e82472c3033db211fd34235eaf4ffc481172518a49f59e95136f13\": container with ID starting with 78e5d343d7e82472c3033db211fd34235eaf4ffc481172518a49f59e95136f13 not found: ID does not exist" containerID="78e5d343d7e82472c3033db211fd34235eaf4ffc481172518a49f59e95136f13" Nov 03 01:33:21.717797 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:33:21.717755 4939 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"78e5d343d7e82472c3033db211fd34235eaf4ffc481172518a49f59e95136f13"} err="failed to get container status \"78e5d343d7e82472c3033db211fd34235eaf4ffc481172518a49f59e95136f13\": rpc error: code = NotFound desc = could not find container \"78e5d343d7e82472c3033db211fd34235eaf4ffc481172518a49f59e95136f13\": container with ID starting with 78e5d343d7e82472c3033db211fd34235eaf4ffc481172518a49f59e95136f13 not found: ID does not exist" Nov 03 01:33:21.736956 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:33:21.736933 4939 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-ee618c/test-model-ee618c-kserve-844f5b79dc-f6kl7"] Nov 03 01:33:21.741775 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:33:21.741754 4939 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-a-ee618c/test-model-ee618c-kserve-844f5b79dc-f6kl7"] Nov 03 01:33:21.799720 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:33:21.799694 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:33:21.923172 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:33:21.923132 4939 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="abab7b3b-f57d-4abe-be59-295e4ab8a769" path="/var/lib/kubelet/pods/abab7b3b-f57d-4abe-be59-295e4ab8a769/volumes" Nov 03 01:33:22.512656 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:33:22.512621 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:33:23.367157 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:33:23.367111 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:33:23.704787 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:33:23.704710 4939 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-ee618c-8bc685f5f-l65d9"] Nov 03 01:33:23.731674 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:33:23.730480 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:33:23.834971 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:33:23.834941 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:33:24.637269 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:33:24.637236 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:33:25.379512 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:33:25.379477 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:33:25.994685 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:33:25.994636 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:33:26.802301 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:33:26.802259 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:33:27.397069 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:33:27.397035 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:33:27.989234 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:33:27.989202 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:33:28.598656 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:33:28.598618 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:33:28.705486 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:33:28.705443 4939 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-ee618c-8bc685f5f-l65d9" podUID="3d52a5d5-9404-4bdf-b6cb-9f134f145c6f" containerName="payload-pre-processing" containerID="cri-o://308485434bd7c443355f49edf1854b03d0c5193f0af258a6abd6cb492780c05a" gracePeriod=25 Nov 03 01:33:29.207814 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:33:29.207774 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:33:29.406559 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:33:29.406523 4939 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-ee618c-8bc685f5f-l65d9" podUID="3d52a5d5-9404-4bdf-b6cb-9f134f145c6f" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 03 01:33:29.886852 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:33:29.886814 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:33:30.053114 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:33:30.053079 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:33:30.577322 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:33:30.577285 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:33:31.225271 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:33:31.225232 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:33:32.273029 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:33:32.272987 4939 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-inf-b-5bfc51/test-model-5bfc51-kserve-5c5676fcf9-tssvf" podUID="d0ed2957-3656-49bd-8aff-27ab5483d367" containerName="main" containerID="cri-o://5b9557d64e7a642db2cbbdfd10203121fe0ec76b67473542658a9b2472b54879" gracePeriod=45 Nov 03 01:33:32.276059 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:33:32.275489 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:33:32.535473 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:33:32.535371 4939 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-b-5bfc51/test-model-5bfc51-kserve-5c5676fcf9-tssvf" podUID="d0ed2957-3656-49bd-8aff-27ab5483d367" containerName="main" probeResult="failure" output="Get \"https://10.132.1.61:8000/ready\": dial tcp 10.132.1.61:8000: connect: connection refused" Nov 03 01:33:32.582817 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:33:32.582778 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:33:32.919374 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:33:32.919338 4939 generic.go:356] "Generic (PLEG): container finished" podID="d0ed2957-3656-49bd-8aff-27ab5483d367" containerID="5b9557d64e7a642db2cbbdfd10203121fe0ec76b67473542658a9b2472b54879" exitCode=0 Nov 03 01:33:32.919641 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:33:32.919624 4939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-5bfc51/test-model-5bfc51-kserve-5c5676fcf9-tssvf" event={"ID":"d0ed2957-3656-49bd-8aff-27ab5483d367","Type":"ContainerDied","Data":"5b9557d64e7a642db2cbbdfd10203121fe0ec76b67473542658a9b2472b54879"} Nov 03 01:33:32.968383 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:33:32.968347 4939 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-5bfc51/test-model-5bfc51-kserve-5c5676fcf9-tssvf" Nov 03 01:33:33.112065 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:33:33.112030 4939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d0ed2957-3656-49bd-8aff-27ab5483d367-kube-api-access-s9smz\" (UniqueName: \"kubernetes.io/projected/d0ed2957-3656-49bd-8aff-27ab5483d367-kube-api-access-s9smz\") pod \"d0ed2957-3656-49bd-8aff-27ab5483d367\" (UID: \"d0ed2957-3656-49bd-8aff-27ab5483d367\") " Nov 03 01:33:33.112277 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:33:33.112121 4939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d0ed2957-3656-49bd-8aff-27ab5483d367-tls-certs\" (UniqueName: \"kubernetes.io/secret/d0ed2957-3656-49bd-8aff-27ab5483d367-tls-certs\") pod \"d0ed2957-3656-49bd-8aff-27ab5483d367\" (UID: \"d0ed2957-3656-49bd-8aff-27ab5483d367\") " Nov 03 01:33:33.112277 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:33:33.112216 4939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d0ed2957-3656-49bd-8aff-27ab5483d367-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d0ed2957-3656-49bd-8aff-27ab5483d367-tmp-dir\") pod \"d0ed2957-3656-49bd-8aff-27ab5483d367\" (UID: \"d0ed2957-3656-49bd-8aff-27ab5483d367\") " Nov 03 01:33:33.112277 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:33:33.112245 4939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d0ed2957-3656-49bd-8aff-27ab5483d367-dshm\" (UniqueName: \"kubernetes.io/empty-dir/d0ed2957-3656-49bd-8aff-27ab5483d367-dshm\") pod \"d0ed2957-3656-49bd-8aff-27ab5483d367\" (UID: \"d0ed2957-3656-49bd-8aff-27ab5483d367\") " Nov 03 01:33:33.112450 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:33:33.112366 4939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d0ed2957-3656-49bd-8aff-27ab5483d367-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d0ed2957-3656-49bd-8aff-27ab5483d367-model-cache\") pod \"d0ed2957-3656-49bd-8aff-27ab5483d367\" (UID: \"d0ed2957-3656-49bd-8aff-27ab5483d367\") " Nov 03 01:33:33.112579 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:33:33.112544 4939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d0ed2957-3656-49bd-8aff-27ab5483d367-tmp-dir" pod "d0ed2957-3656-49bd-8aff-27ab5483d367" (UID: "d0ed2957-3656-49bd-8aff-27ab5483d367"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 01:33:33.112771 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:33:33.112604 4939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d0ed2957-3656-49bd-8aff-27ab5483d367-home\" (UniqueName: \"kubernetes.io/empty-dir/d0ed2957-3656-49bd-8aff-27ab5483d367-home\") pod \"d0ed2957-3656-49bd-8aff-27ab5483d367\" (UID: \"d0ed2957-3656-49bd-8aff-27ab5483d367\") " Nov 03 01:33:33.112771 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:33:33.112655 4939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d0ed2957-3656-49bd-8aff-27ab5483d367-model-cache" pod "d0ed2957-3656-49bd-8aff-27ab5483d367" (UID: "d0ed2957-3656-49bd-8aff-27ab5483d367"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 01:33:33.112990 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:33:33.112948 4939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d0ed2957-3656-49bd-8aff-27ab5483d367-home" pod "d0ed2957-3656-49bd-8aff-27ab5483d367" (UID: "d0ed2957-3656-49bd-8aff-27ab5483d367"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 01:33:33.113645 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:33:33.113619 4939 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d0ed2957-3656-49bd-8aff-27ab5483d367-tmp-dir\") on node \"ip-10-0-140-107.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:33:33.113743 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:33:33.113649 4939 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d0ed2957-3656-49bd-8aff-27ab5483d367-model-cache\") on node \"ip-10-0-140-107.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:33:33.113743 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:33:33.113664 4939 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d0ed2957-3656-49bd-8aff-27ab5483d367-home\") on node \"ip-10-0-140-107.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:33:33.114639 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:33:33.114620 4939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d0ed2957-3656-49bd-8aff-27ab5483d367-tls-certs" pod "d0ed2957-3656-49bd-8aff-27ab5483d367" (UID: "d0ed2957-3656-49bd-8aff-27ab5483d367"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 01:33:33.114717 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:33:33.114657 4939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d0ed2957-3656-49bd-8aff-27ab5483d367-kube-api-access-s9smz" pod "d0ed2957-3656-49bd-8aff-27ab5483d367" (UID: "d0ed2957-3656-49bd-8aff-27ab5483d367"). InnerVolumeSpecName "kube-api-access-s9smz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 01:33:33.114820 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:33:33.114800 4939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d0ed2957-3656-49bd-8aff-27ab5483d367-dshm" pod "d0ed2957-3656-49bd-8aff-27ab5483d367" (UID: "d0ed2957-3656-49bd-8aff-27ab5483d367"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 01:33:33.215933 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:33:33.215832 4939 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d0ed2957-3656-49bd-8aff-27ab5483d367-dshm\") on node \"ip-10-0-140-107.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:33:33.215933 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:33:33.215866 4939 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-s9smz\" (UniqueName: \"kubernetes.io/projected/d0ed2957-3656-49bd-8aff-27ab5483d367-kube-api-access-s9smz\") on node \"ip-10-0-140-107.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:33:33.215933 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:33:33.215882 4939 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d0ed2957-3656-49bd-8aff-27ab5483d367-tls-certs\") on node \"ip-10-0-140-107.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:33:33.411859 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:33:33.411825 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:33:33.930101 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:33:33.930061 4939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-5bfc51/test-model-5bfc51-kserve-5c5676fcf9-tssvf" event={"ID":"d0ed2957-3656-49bd-8aff-27ab5483d367","Type":"ContainerDied","Data":"fd0b1baa31a6313e319394cb5fa7fe1605ffeec4a070fc1b36c8e8ea92acf639"} Nov 03 01:33:33.930298 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:33:33.930122 4939 scope.go:122] "RemoveContainer" containerID="5b9557d64e7a642db2cbbdfd10203121fe0ec76b67473542658a9b2472b54879" Nov 03 01:33:33.930298 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:33:33.930286 4939 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-5bfc51/test-model-5bfc51-kserve-5c5676fcf9-tssvf" Nov 03 01:33:33.955966 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:33:33.955934 4939 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-5bfc51/test-model-5bfc51-kserve-5c5676fcf9-tssvf"] Nov 03 01:33:33.961066 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:33:33.961040 4939 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-b-5bfc51/test-model-5bfc51-kserve-5c5676fcf9-tssvf"] Nov 03 01:33:34.033161 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:33:34.033131 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:33:34.696284 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:33:34.696253 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:33:35.384070 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:33:35.384032 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:33:35.871472 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:33:35.871440 4939 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d0ed2957-3656-49bd-8aff-27ab5483d367" path="/var/lib/kubelet/pods/d0ed2957-3656-49bd-8aff-27ab5483d367/volumes" Nov 03 01:33:35.982278 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:33:35.982244 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:33:36.170730 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:33:36.170649 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:33:36.297801 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:33:36.297766 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:33:36.680775 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:33:36.680736 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:33:36.984443 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:33:36.980831 4939 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-ee618c-openshift-default-5ff9c74dc9-z4qxq"] Nov 03 01:33:36.984443 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:33:36.981645 4939 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-a-ee618c-openshift-default-5ff9c74dc9-z4qxq" podUID="6165abef-0df3-4007-ae59-5536a2a77e45" containerName="istio-proxy" containerID="cri-o://710675672c4704b956985b778c9969f2de90e04a802951d0744ecd2defb6b0d9" gracePeriod=30 Nov 03 01:33:36.990976 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:33:36.990061 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:33:37.007644 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:33:37.006603 4939 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-ee618c-openshift-default-5ff9c74dc9-ssz95"] Nov 03 01:33:37.007644 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:33:37.006828 4939 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-a-ee618c-openshift-default-5ff9c74dc9-ssz95" podUID="fef89d35-3c5f-483f-b896-7b47733ab2df" containerName="istio-proxy" containerID="cri-o://8eed9477dcb60efc399a950decb550840ff9a5096a9ecfe2dca04f714f38dab6" gracePeriod=30 Nov 03 01:33:37.009295 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:33:37.009266 4939 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-ee618c-openshift-default-5ff9c74dc9-c4jpz"] Nov 03 01:33:37.009521 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:33:37.009486 4939 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-a-ee618c-openshift-default-5ff9c74dc9-c4jpz" podUID="7fc8db38-3791-40c0-bcfb-716313223a85" containerName="istio-proxy" containerID="cri-o://02aaf5672c7c46eedfc7b87e00d31cdb560aed8c52b7cb1b556bffa5e037bea1" gracePeriod=30 Nov 03 01:33:37.251891 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:33:37.251726 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:33:37.795721 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:33:37.795688 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:33:37.906141 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:33:37.906101 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:33:38.177696 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:33:38.177645 4939 patch_prober.go:29] interesting pod/e2e-inf-a-ee618c-openshift-default-5ff9c74dc9-c4jpz container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 03 01:33:38.178253 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:33:38.177737 4939 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-a-ee618c-openshift-default-5ff9c74dc9-c4jpz" podUID="7fc8db38-3791-40c0-bcfb-716313223a85" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 03 01:33:38.499121 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:33:38.499034 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:33:39.098441 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:33:39.098410 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:33:39.115748 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:33:39.115485 4939 patch_prober.go:29] interesting pod/e2e-inf-a-ee618c-openshift-default-5ff9c74dc9-z4qxq container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 03 01:33:39.115748 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:33:39.115542 4939 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-a-ee618c-openshift-default-5ff9c74dc9-z4qxq" podUID="6165abef-0df3-4007-ae59-5536a2a77e45" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 03 01:33:39.406643 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:33:39.406554 4939 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-ee618c-8bc685f5f-l65d9" podUID="3d52a5d5-9404-4bdf-b6cb-9f134f145c6f" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 03 01:33:39.731376 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:33:39.731295 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:33:39.841187 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:33:39.841155 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:33:40.728365 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:33:40.728331 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:33:41.401061 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:33:41.401026 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:33:42.079270 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:33:42.079243 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:33:42.407963 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:33:42.407891 4939 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-ee618c-openshift-default-5ff9c74dc9-c4jpz" Nov 03 01:33:42.486328 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:33:42.486277 4939 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-ee618c-openshift-default-5ff9c74dc9-z4qxq" Nov 03 01:33:42.510839 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:33:42.510792 4939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6165abef-0df3-4007-ae59-5536a2a77e45-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6165abef-0df3-4007-ae59-5536a2a77e45-workload-certs\") pod \"6165abef-0df3-4007-ae59-5536a2a77e45\" (UID: \"6165abef-0df3-4007-ae59-5536a2a77e45\") " Nov 03 01:33:42.511087 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:33:42.510859 4939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6165abef-0df3-4007-ae59-5536a2a77e45-istio-token\" (UniqueName: \"kubernetes.io/projected/6165abef-0df3-4007-ae59-5536a2a77e45-istio-token\") pod \"6165abef-0df3-4007-ae59-5536a2a77e45\" (UID: \"6165abef-0df3-4007-ae59-5536a2a77e45\") " Nov 03 01:33:42.511087 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:33:42.510960 4939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7fc8db38-3791-40c0-bcfb-716313223a85-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7fc8db38-3791-40c0-bcfb-716313223a85-istio-data\") pod \"7fc8db38-3791-40c0-bcfb-716313223a85\" (UID: \"7fc8db38-3791-40c0-bcfb-716313223a85\") " Nov 03 01:33:42.511221 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:33:42.511074 4939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7fc8db38-3791-40c0-bcfb-716313223a85-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7fc8db38-3791-40c0-bcfb-716313223a85-workload-socket\") pod \"7fc8db38-3791-40c0-bcfb-716313223a85\" (UID: \"7fc8db38-3791-40c0-bcfb-716313223a85\") " Nov 03 01:33:42.511221 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:33:42.511194 4939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/6165abef-0df3-4007-ae59-5536a2a77e45-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6165abef-0df3-4007-ae59-5536a2a77e45-istio-podinfo\") pod \"6165abef-0df3-4007-ae59-5536a2a77e45\" (UID: \"6165abef-0df3-4007-ae59-5536a2a77e45\") " Nov 03 01:33:42.511342 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:33:42.511234 4939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7fc8db38-3791-40c0-bcfb-716313223a85-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7fc8db38-3791-40c0-bcfb-716313223a85-istio-envoy\") pod \"7fc8db38-3791-40c0-bcfb-716313223a85\" (UID: \"7fc8db38-3791-40c0-bcfb-716313223a85\") " Nov 03 01:33:42.511342 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:33:42.511294 4939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6165abef-0df3-4007-ae59-5536a2a77e45-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6165abef-0df3-4007-ae59-5536a2a77e45-istio-envoy\") pod \"6165abef-0df3-4007-ae59-5536a2a77e45\" (UID: \"6165abef-0df3-4007-ae59-5536a2a77e45\") " Nov 03 01:33:42.511461 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:33:42.511337 4939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/7fc8db38-3791-40c0-bcfb-716313223a85-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7fc8db38-3791-40c0-bcfb-716313223a85-istio-podinfo\") pod \"7fc8db38-3791-40c0-bcfb-716313223a85\" (UID: \"7fc8db38-3791-40c0-bcfb-716313223a85\") " Nov 03 01:33:42.511461 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:33:42.511412 4939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7fc8db38-3791-40c0-bcfb-716313223a85-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7fc8db38-3791-40c0-bcfb-716313223a85-istiod-ca-cert\") pod \"7fc8db38-3791-40c0-bcfb-716313223a85\" (UID: \"7fc8db38-3791-40c0-bcfb-716313223a85\") " Nov 03 01:33:42.511579 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:33:42.511510 4939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6165abef-0df3-4007-ae59-5536a2a77e45-kube-api-access-5lqkc\" (UniqueName: \"kubernetes.io/projected/6165abef-0df3-4007-ae59-5536a2a77e45-kube-api-access-5lqkc\") pod \"6165abef-0df3-4007-ae59-5536a2a77e45\" (UID: \"6165abef-0df3-4007-ae59-5536a2a77e45\") " Nov 03 01:33:42.511680 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:33:42.511662 4939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7fc8db38-3791-40c0-bcfb-716313223a85-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7fc8db38-3791-40c0-bcfb-716313223a85-credential-socket\") pod \"7fc8db38-3791-40c0-bcfb-716313223a85\" (UID: \"7fc8db38-3791-40c0-bcfb-716313223a85\") " Nov 03 01:33:42.511767 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:33:42.511748 4939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7fc8db38-3791-40c0-bcfb-716313223a85-kube-api-access-5knp8\" (UniqueName: \"kubernetes.io/projected/7fc8db38-3791-40c0-bcfb-716313223a85-kube-api-access-5knp8\") pod \"7fc8db38-3791-40c0-bcfb-716313223a85\" (UID: \"7fc8db38-3791-40c0-bcfb-716313223a85\") " Nov 03 01:33:42.511857 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:33:42.511840 4939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6165abef-0df3-4007-ae59-5536a2a77e45-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6165abef-0df3-4007-ae59-5536a2a77e45-credential-socket\") pod \"6165abef-0df3-4007-ae59-5536a2a77e45\" (UID: \"6165abef-0df3-4007-ae59-5536a2a77e45\") " Nov 03 01:33:42.511938 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:33:42.511885 4939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6165abef-0df3-4007-ae59-5536a2a77e45-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6165abef-0df3-4007-ae59-5536a2a77e45-istiod-ca-cert\") pod \"6165abef-0df3-4007-ae59-5536a2a77e45\" (UID: \"6165abef-0df3-4007-ae59-5536a2a77e45\") " Nov 03 01:33:42.512102 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:33:42.512085 4939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7fc8db38-3791-40c0-bcfb-716313223a85-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7fc8db38-3791-40c0-bcfb-716313223a85-workload-certs\") pod \"7fc8db38-3791-40c0-bcfb-716313223a85\" (UID: \"7fc8db38-3791-40c0-bcfb-716313223a85\") " Nov 03 01:33:42.512164 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:33:42.512130 4939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6165abef-0df3-4007-ae59-5536a2a77e45-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6165abef-0df3-4007-ae59-5536a2a77e45-istio-data\") pod \"6165abef-0df3-4007-ae59-5536a2a77e45\" (UID: \"6165abef-0df3-4007-ae59-5536a2a77e45\") " Nov 03 01:33:42.512299 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:33:42.512207 4939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7fc8db38-3791-40c0-bcfb-716313223a85-istio-token\" (UniqueName: \"kubernetes.io/projected/7fc8db38-3791-40c0-bcfb-716313223a85-istio-token\") pod \"7fc8db38-3791-40c0-bcfb-716313223a85\" (UID: \"7fc8db38-3791-40c0-bcfb-716313223a85\") " Nov 03 01:33:42.512405 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:33:42.512372 4939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6165abef-0df3-4007-ae59-5536a2a77e45-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6165abef-0df3-4007-ae59-5536a2a77e45-workload-socket\") pod \"6165abef-0df3-4007-ae59-5536a2a77e45\" (UID: \"6165abef-0df3-4007-ae59-5536a2a77e45\") " Nov 03 01:33:42.513230 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:33:42.512944 4939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7fc8db38-3791-40c0-bcfb-716313223a85-istiod-ca-cert" pod "7fc8db38-3791-40c0-bcfb-716313223a85" (UID: "7fc8db38-3791-40c0-bcfb-716313223a85"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 01:33:42.514537 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:33:42.514511 4939 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7fc8db38-3791-40c0-bcfb-716313223a85-istiod-ca-cert\") on node \"ip-10-0-140-107.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:33:42.514764 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:33:42.514739 4939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6165abef-0df3-4007-ae59-5536a2a77e45-workload-socket" pod "6165abef-0df3-4007-ae59-5536a2a77e45" (UID: "6165abef-0df3-4007-ae59-5536a2a77e45"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 01:33:42.514984 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:33:42.514961 4939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6165abef-0df3-4007-ae59-5536a2a77e45-credential-socket" pod "6165abef-0df3-4007-ae59-5536a2a77e45" (UID: "6165abef-0df3-4007-ae59-5536a2a77e45"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 01:33:42.515111 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:33:42.515096 4939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6165abef-0df3-4007-ae59-5536a2a77e45-workload-certs" pod "6165abef-0df3-4007-ae59-5536a2a77e45" (UID: "6165abef-0df3-4007-ae59-5536a2a77e45"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 01:33:42.515361 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:33:42.515346 4939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6165abef-0df3-4007-ae59-5536a2a77e45-istiod-ca-cert" pod "6165abef-0df3-4007-ae59-5536a2a77e45" (UID: "6165abef-0df3-4007-ae59-5536a2a77e45"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 01:33:42.515559 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:33:42.515539 4939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7fc8db38-3791-40c0-bcfb-716313223a85-workload-certs" pod "7fc8db38-3791-40c0-bcfb-716313223a85" (UID: "7fc8db38-3791-40c0-bcfb-716313223a85"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 01:33:42.515742 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:33:42.515724 4939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6165abef-0df3-4007-ae59-5536a2a77e45-istio-data" pod "6165abef-0df3-4007-ae59-5536a2a77e45" (UID: "6165abef-0df3-4007-ae59-5536a2a77e45"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 01:33:42.516660 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:33:42.516630 4939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7fc8db38-3791-40c0-bcfb-716313223a85-istio-data" pod "7fc8db38-3791-40c0-bcfb-716313223a85" (UID: "7fc8db38-3791-40c0-bcfb-716313223a85"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 01:33:42.517097 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:33:42.517070 4939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7fc8db38-3791-40c0-bcfb-716313223a85-workload-socket" pod "7fc8db38-3791-40c0-bcfb-716313223a85" (UID: "7fc8db38-3791-40c0-bcfb-716313223a85"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 01:33:42.517310 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:33:42.517287 4939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7fc8db38-3791-40c0-bcfb-716313223a85-credential-socket" pod "7fc8db38-3791-40c0-bcfb-716313223a85" (UID: "7fc8db38-3791-40c0-bcfb-716313223a85"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 01:33:42.520092 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:33:42.520061 4939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6165abef-0df3-4007-ae59-5536a2a77e45-kube-api-access-5lqkc" pod "6165abef-0df3-4007-ae59-5536a2a77e45" (UID: "6165abef-0df3-4007-ae59-5536a2a77e45"). InnerVolumeSpecName "kube-api-access-5lqkc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 01:33:42.520197 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:33:42.520106 4939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7fc8db38-3791-40c0-bcfb-716313223a85-istio-token" pod "7fc8db38-3791-40c0-bcfb-716313223a85" (UID: "7fc8db38-3791-40c0-bcfb-716313223a85"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 01:33:42.528536 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:33:42.528491 4939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7fc8db38-3791-40c0-bcfb-716313223a85-istio-envoy" pod "7fc8db38-3791-40c0-bcfb-716313223a85" (UID: "7fc8db38-3791-40c0-bcfb-716313223a85"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 01:33:42.528536 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:33:42.528511 4939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/6165abef-0df3-4007-ae59-5536a2a77e45-istio-podinfo" pod "6165abef-0df3-4007-ae59-5536a2a77e45" (UID: "6165abef-0df3-4007-ae59-5536a2a77e45"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 01:33:42.528705 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:33:42.528581 4939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6165abef-0df3-4007-ae59-5536a2a77e45-istio-envoy" pod "6165abef-0df3-4007-ae59-5536a2a77e45" (UID: "6165abef-0df3-4007-ae59-5536a2a77e45"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 01:33:42.529111 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:33:42.529075 4939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/7fc8db38-3791-40c0-bcfb-716313223a85-istio-podinfo" pod "7fc8db38-3791-40c0-bcfb-716313223a85" (UID: "7fc8db38-3791-40c0-bcfb-716313223a85"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 01:33:42.529453 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:33:42.529426 4939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7fc8db38-3791-40c0-bcfb-716313223a85-kube-api-access-5knp8" pod "7fc8db38-3791-40c0-bcfb-716313223a85" (UID: "7fc8db38-3791-40c0-bcfb-716313223a85"). InnerVolumeSpecName "kube-api-access-5knp8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 01:33:42.530842 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:33:42.530820 4939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6165abef-0df3-4007-ae59-5536a2a77e45-istio-token" pod "6165abef-0df3-4007-ae59-5536a2a77e45" (UID: "6165abef-0df3-4007-ae59-5536a2a77e45"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 01:33:42.541284 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:33:42.541259 4939 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-ee618c-openshift-default-5ff9c74dc9-ssz95" Nov 03 01:33:42.615863 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:33:42.615775 4939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fef89d35-3c5f-483f-b896-7b47733ab2df-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fef89d35-3c5f-483f-b896-7b47733ab2df-workload-certs\") pod \"fef89d35-3c5f-483f-b896-7b47733ab2df\" (UID: \"fef89d35-3c5f-483f-b896-7b47733ab2df\") " Nov 03 01:33:42.616105 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:33:42.615919 4939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/fef89d35-3c5f-483f-b896-7b47733ab2df-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fef89d35-3c5f-483f-b896-7b47733ab2df-istio-podinfo\") pod \"fef89d35-3c5f-483f-b896-7b47733ab2df\" (UID: \"fef89d35-3c5f-483f-b896-7b47733ab2df\") " Nov 03 01:33:42.616183 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:33:42.616107 4939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fef89d35-3c5f-483f-b896-7b47733ab2df-workload-certs" pod "fef89d35-3c5f-483f-b896-7b47733ab2df" (UID: "fef89d35-3c5f-483f-b896-7b47733ab2df"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 01:33:42.616183 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:33:42.616175 4939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fef89d35-3c5f-483f-b896-7b47733ab2df-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fef89d35-3c5f-483f-b896-7b47733ab2df-istio-data\") pod \"fef89d35-3c5f-483f-b896-7b47733ab2df\" (UID: \"fef89d35-3c5f-483f-b896-7b47733ab2df\") " Nov 03 01:33:42.616316 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:33:42.616210 4939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fef89d35-3c5f-483f-b896-7b47733ab2df-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fef89d35-3c5f-483f-b896-7b47733ab2df-istio-envoy\") pod \"fef89d35-3c5f-483f-b896-7b47733ab2df\" (UID: \"fef89d35-3c5f-483f-b896-7b47733ab2df\") " Nov 03 01:33:42.616316 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:33:42.616271 4939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fef89d35-3c5f-483f-b896-7b47733ab2df-istio-token\" (UniqueName: \"kubernetes.io/projected/fef89d35-3c5f-483f-b896-7b47733ab2df-istio-token\") pod \"fef89d35-3c5f-483f-b896-7b47733ab2df\" (UID: \"fef89d35-3c5f-483f-b896-7b47733ab2df\") " Nov 03 01:33:42.616316 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:33:42.616310 4939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fef89d35-3c5f-483f-b896-7b47733ab2df-kube-api-access-xd2xk\" (UniqueName: \"kubernetes.io/projected/fef89d35-3c5f-483f-b896-7b47733ab2df-kube-api-access-xd2xk\") pod \"fef89d35-3c5f-483f-b896-7b47733ab2df\" (UID: \"fef89d35-3c5f-483f-b896-7b47733ab2df\") " Nov 03 01:33:42.616488 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:33:42.616432 4939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fef89d35-3c5f-483f-b896-7b47733ab2df-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fef89d35-3c5f-483f-b896-7b47733ab2df-istiod-ca-cert\") pod \"fef89d35-3c5f-483f-b896-7b47733ab2df\" (UID: \"fef89d35-3c5f-483f-b896-7b47733ab2df\") " Nov 03 01:33:42.616488 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:33:42.616449 4939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fef89d35-3c5f-483f-b896-7b47733ab2df-istio-data" pod "fef89d35-3c5f-483f-b896-7b47733ab2df" (UID: "fef89d35-3c5f-483f-b896-7b47733ab2df"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 01:33:42.616488 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:33:42.616475 4939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fef89d35-3c5f-483f-b896-7b47733ab2df-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fef89d35-3c5f-483f-b896-7b47733ab2df-workload-socket\") pod \"fef89d35-3c5f-483f-b896-7b47733ab2df\" (UID: \"fef89d35-3c5f-483f-b896-7b47733ab2df\") " Nov 03 01:33:42.616664 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:33:42.616584 4939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fef89d35-3c5f-483f-b896-7b47733ab2df-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fef89d35-3c5f-483f-b896-7b47733ab2df-credential-socket\") pod \"fef89d35-3c5f-483f-b896-7b47733ab2df\" (UID: \"fef89d35-3c5f-483f-b896-7b47733ab2df\") " Nov 03 01:33:42.617351 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:33:42.616893 4939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fef89d35-3c5f-483f-b896-7b47733ab2df-istiod-ca-cert" pod "fef89d35-3c5f-483f-b896-7b47733ab2df" (UID: "fef89d35-3c5f-483f-b896-7b47733ab2df"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 01:33:42.617351 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:33:42.617164 4939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fef89d35-3c5f-483f-b896-7b47733ab2df-credential-socket" pod "fef89d35-3c5f-483f-b896-7b47733ab2df" (UID: "fef89d35-3c5f-483f-b896-7b47733ab2df"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 01:33:42.617351 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:33:42.617286 4939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fef89d35-3c5f-483f-b896-7b47733ab2df-workload-socket" pod "fef89d35-3c5f-483f-b896-7b47733ab2df" (UID: "fef89d35-3c5f-483f-b896-7b47733ab2df"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 01:33:42.618067 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:33:42.617649 4939 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fef89d35-3c5f-483f-b896-7b47733ab2df-credential-socket\") on node \"ip-10-0-140-107.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:33:42.618067 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:33:42.617681 4939 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7fc8db38-3791-40c0-bcfb-716313223a85-workload-certs\") on node \"ip-10-0-140-107.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:33:42.618067 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:33:42.617697 4939 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fef89d35-3c5f-483f-b896-7b47733ab2df-workload-certs\") on node \"ip-10-0-140-107.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:33:42.618067 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:33:42.617711 4939 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6165abef-0df3-4007-ae59-5536a2a77e45-istio-data\") on node \"ip-10-0-140-107.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:33:42.618067 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:33:42.617727 4939 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7fc8db38-3791-40c0-bcfb-716313223a85-istio-token\") on node \"ip-10-0-140-107.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:33:42.618067 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:33:42.617741 4939 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6165abef-0df3-4007-ae59-5536a2a77e45-workload-socket\") on node \"ip-10-0-140-107.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:33:42.618067 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:33:42.617755 4939 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6165abef-0df3-4007-ae59-5536a2a77e45-workload-certs\") on node \"ip-10-0-140-107.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:33:42.618067 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:33:42.617768 4939 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6165abef-0df3-4007-ae59-5536a2a77e45-istio-token\") on node \"ip-10-0-140-107.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:33:42.618067 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:33:42.617782 4939 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7fc8db38-3791-40c0-bcfb-716313223a85-istio-data\") on node \"ip-10-0-140-107.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:33:42.618067 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:33:42.617796 4939 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7fc8db38-3791-40c0-bcfb-716313223a85-workload-socket\") on node \"ip-10-0-140-107.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:33:42.618067 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:33:42.617811 4939 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6165abef-0df3-4007-ae59-5536a2a77e45-istio-podinfo\") on node \"ip-10-0-140-107.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:33:42.618067 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:33:42.617824 4939 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7fc8db38-3791-40c0-bcfb-716313223a85-istio-envoy\") on node \"ip-10-0-140-107.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:33:42.618067 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:33:42.617839 4939 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6165abef-0df3-4007-ae59-5536a2a77e45-istio-envoy\") on node \"ip-10-0-140-107.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:33:42.618067 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:33:42.617854 4939 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7fc8db38-3791-40c0-bcfb-716313223a85-istio-podinfo\") on node \"ip-10-0-140-107.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:33:42.618067 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:33:42.617867 4939 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fef89d35-3c5f-483f-b896-7b47733ab2df-istio-data\") on node \"ip-10-0-140-107.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:33:42.618067 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:33:42.617882 4939 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5lqkc\" (UniqueName: \"kubernetes.io/projected/6165abef-0df3-4007-ae59-5536a2a77e45-kube-api-access-5lqkc\") on node \"ip-10-0-140-107.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:33:42.618067 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:33:42.617898 4939 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7fc8db38-3791-40c0-bcfb-716313223a85-credential-socket\") on node \"ip-10-0-140-107.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:33:42.618067 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:33:42.617933 4939 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5knp8\" (UniqueName: \"kubernetes.io/projected/7fc8db38-3791-40c0-bcfb-716313223a85-kube-api-access-5knp8\") on node \"ip-10-0-140-107.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:33:42.618067 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:33:42.617948 4939 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fef89d35-3c5f-483f-b896-7b47733ab2df-istiod-ca-cert\") on node \"ip-10-0-140-107.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:33:42.618067 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:33:42.617962 4939 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fef89d35-3c5f-483f-b896-7b47733ab2df-workload-socket\") on node \"ip-10-0-140-107.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:33:42.618067 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:33:42.617976 4939 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6165abef-0df3-4007-ae59-5536a2a77e45-credential-socket\") on node \"ip-10-0-140-107.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:33:42.618067 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:33:42.617991 4939 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6165abef-0df3-4007-ae59-5536a2a77e45-istiod-ca-cert\") on node \"ip-10-0-140-107.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:33:42.619642 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:33:42.619136 4939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fef89d35-3c5f-483f-b896-7b47733ab2df-kube-api-access-xd2xk" pod "fef89d35-3c5f-483f-b896-7b47733ab2df" (UID: "fef89d35-3c5f-483f-b896-7b47733ab2df"). InnerVolumeSpecName "kube-api-access-xd2xk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 01:33:42.619642 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:33:42.619248 4939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fef89d35-3c5f-483f-b896-7b47733ab2df-istio-envoy" pod "fef89d35-3c5f-483f-b896-7b47733ab2df" (UID: "fef89d35-3c5f-483f-b896-7b47733ab2df"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 01:33:42.619813 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:33:42.619790 4939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/fef89d35-3c5f-483f-b896-7b47733ab2df-istio-podinfo" pod "fef89d35-3c5f-483f-b896-7b47733ab2df" (UID: "fef89d35-3c5f-483f-b896-7b47733ab2df"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 01:33:42.620141 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:33:42.620112 4939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fef89d35-3c5f-483f-b896-7b47733ab2df-istio-token" pod "fef89d35-3c5f-483f-b896-7b47733ab2df" (UID: "fef89d35-3c5f-483f-b896-7b47733ab2df"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 01:33:42.719466 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:33:42.719433 4939 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fef89d35-3c5f-483f-b896-7b47733ab2df-istio-envoy\") on node \"ip-10-0-140-107.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:33:42.719466 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:33:42.719462 4939 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fef89d35-3c5f-483f-b896-7b47733ab2df-istio-token\") on node \"ip-10-0-140-107.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:33:42.719694 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:33:42.719476 4939 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xd2xk\" (UniqueName: \"kubernetes.io/projected/fef89d35-3c5f-483f-b896-7b47733ab2df-kube-api-access-xd2xk\") on node \"ip-10-0-140-107.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:33:42.719694 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:33:42.719486 4939 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fef89d35-3c5f-483f-b896-7b47733ab2df-istio-podinfo\") on node \"ip-10-0-140-107.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:33:42.893881 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:33:42.893795 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:33:43.062839 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:33:43.062800 4939 generic.go:356] "Generic (PLEG): container finished" podID="7fc8db38-3791-40c0-bcfb-716313223a85" containerID="02aaf5672c7c46eedfc7b87e00d31cdb560aed8c52b7cb1b556bffa5e037bea1" exitCode=0 Nov 03 01:33:43.063037 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:33:43.062849 4939 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-ee618c-openshift-default-5ff9c74dc9-c4jpz" Nov 03 01:33:43.063037 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:33:43.062884 4939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-ee618c-openshift-default-5ff9c74dc9-c4jpz" event={"ID":"7fc8db38-3791-40c0-bcfb-716313223a85","Type":"ContainerDied","Data":"02aaf5672c7c46eedfc7b87e00d31cdb560aed8c52b7cb1b556bffa5e037bea1"} Nov 03 01:33:43.063037 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:33:43.062984 4939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-ee618c-openshift-default-5ff9c74dc9-c4jpz" event={"ID":"7fc8db38-3791-40c0-bcfb-716313223a85","Type":"ContainerDied","Data":"aa3f47b0fd2754dcea2c7d2cec69ac0a43bbdd6e1fc782e84fb7e2cf4daebbd3"} Nov 03 01:33:43.063037 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:33:43.063019 4939 scope.go:122] "RemoveContainer" containerID="02aaf5672c7c46eedfc7b87e00d31cdb560aed8c52b7cb1b556bffa5e037bea1" Nov 03 01:33:43.064765 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:33:43.064735 4939 generic.go:356] "Generic (PLEG): container finished" podID="fef89d35-3c5f-483f-b896-7b47733ab2df" containerID="8eed9477dcb60efc399a950decb550840ff9a5096a9ecfe2dca04f714f38dab6" exitCode=0 Nov 03 01:33:43.064918 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:33:43.064799 4939 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-ee618c-openshift-default-5ff9c74dc9-ssz95" Nov 03 01:33:43.064918 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:33:43.064818 4939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-ee618c-openshift-default-5ff9c74dc9-ssz95" event={"ID":"fef89d35-3c5f-483f-b896-7b47733ab2df","Type":"ContainerDied","Data":"8eed9477dcb60efc399a950decb550840ff9a5096a9ecfe2dca04f714f38dab6"} Nov 03 01:33:43.064918 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:33:43.064880 4939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-ee618c-openshift-default-5ff9c74dc9-ssz95" event={"ID":"fef89d35-3c5f-483f-b896-7b47733ab2df","Type":"ContainerDied","Data":"7a3b86f98204dba32db6a4f066856a10eba1a3d2beb0bd3df40119620943af08"} Nov 03 01:33:43.071466 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:33:43.071442 4939 generic.go:356] "Generic (PLEG): container finished" podID="6165abef-0df3-4007-ae59-5536a2a77e45" containerID="710675672c4704b956985b778c9969f2de90e04a802951d0744ecd2defb6b0d9" exitCode=0 Nov 03 01:33:43.071630 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:33:43.071478 4939 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-ee618c-openshift-default-5ff9c74dc9-z4qxq" Nov 03 01:33:43.071630 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:33:43.071591 4939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-ee618c-openshift-default-5ff9c74dc9-z4qxq" event={"ID":"6165abef-0df3-4007-ae59-5536a2a77e45","Type":"ContainerDied","Data":"710675672c4704b956985b778c9969f2de90e04a802951d0744ecd2defb6b0d9"} Nov 03 01:33:43.071735 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:33:43.071637 4939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-ee618c-openshift-default-5ff9c74dc9-z4qxq" event={"ID":"6165abef-0df3-4007-ae59-5536a2a77e45","Type":"ContainerDied","Data":"d5f702dc7a363dc956348f001c4b430d36962bb0062bd680533bce95e2bcb825"} Nov 03 01:33:43.100653 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:33:43.100630 4939 scope.go:122] "RemoveContainer" containerID="02aaf5672c7c46eedfc7b87e00d31cdb560aed8c52b7cb1b556bffa5e037bea1" Nov 03 01:33:43.101014 ip-10-0-140-107 kubenswrapper[4939]: E1103 01:33:43.100993 4939 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"02aaf5672c7c46eedfc7b87e00d31cdb560aed8c52b7cb1b556bffa5e037bea1\": container with ID starting with 02aaf5672c7c46eedfc7b87e00d31cdb560aed8c52b7cb1b556bffa5e037bea1 not found: ID does not exist" containerID="02aaf5672c7c46eedfc7b87e00d31cdb560aed8c52b7cb1b556bffa5e037bea1" Nov 03 01:33:43.101079 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:33:43.101020 4939 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"02aaf5672c7c46eedfc7b87e00d31cdb560aed8c52b7cb1b556bffa5e037bea1"} err="failed to get container status \"02aaf5672c7c46eedfc7b87e00d31cdb560aed8c52b7cb1b556bffa5e037bea1\": rpc error: code = NotFound desc = could not find container \"02aaf5672c7c46eedfc7b87e00d31cdb560aed8c52b7cb1b556bffa5e037bea1\": container with ID starting with 02aaf5672c7c46eedfc7b87e00d31cdb560aed8c52b7cb1b556bffa5e037bea1 not found: ID does not exist" Nov 03 01:33:43.101079 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:33:43.101038 4939 scope.go:122] "RemoveContainer" containerID="8eed9477dcb60efc399a950decb550840ff9a5096a9ecfe2dca04f714f38dab6" Nov 03 01:33:43.116135 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:33:43.116109 4939 scope.go:122] "RemoveContainer" containerID="8eed9477dcb60efc399a950decb550840ff9a5096a9ecfe2dca04f714f38dab6" Nov 03 01:33:43.116700 ip-10-0-140-107 kubenswrapper[4939]: E1103 01:33:43.116669 4939 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8eed9477dcb60efc399a950decb550840ff9a5096a9ecfe2dca04f714f38dab6\": container with ID starting with 8eed9477dcb60efc399a950decb550840ff9a5096a9ecfe2dca04f714f38dab6 not found: ID does not exist" containerID="8eed9477dcb60efc399a950decb550840ff9a5096a9ecfe2dca04f714f38dab6" Nov 03 01:33:43.116801 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:33:43.116711 4939 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8eed9477dcb60efc399a950decb550840ff9a5096a9ecfe2dca04f714f38dab6"} err="failed to get container status \"8eed9477dcb60efc399a950decb550840ff9a5096a9ecfe2dca04f714f38dab6\": rpc error: code = NotFound desc = could not find container \"8eed9477dcb60efc399a950decb550840ff9a5096a9ecfe2dca04f714f38dab6\": container with ID starting with 8eed9477dcb60efc399a950decb550840ff9a5096a9ecfe2dca04f714f38dab6 not found: ID does not exist" Nov 03 01:33:43.116801 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:33:43.116736 4939 scope.go:122] "RemoveContainer" containerID="710675672c4704b956985b778c9969f2de90e04a802951d0744ecd2defb6b0d9" Nov 03 01:33:43.118893 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:33:43.118869 4939 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-ee618c-openshift-default-5ff9c74dc9-c4jpz"] Nov 03 01:33:43.133942 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:33:43.133828 4939 scope.go:122] "RemoveContainer" containerID="710675672c4704b956985b778c9969f2de90e04a802951d0744ecd2defb6b0d9" Nov 03 01:33:43.134266 ip-10-0-140-107 kubenswrapper[4939]: E1103 01:33:43.134237 4939 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"710675672c4704b956985b778c9969f2de90e04a802951d0744ecd2defb6b0d9\": container with ID starting with 710675672c4704b956985b778c9969f2de90e04a802951d0744ecd2defb6b0d9 not found: ID does not exist" containerID="710675672c4704b956985b778c9969f2de90e04a802951d0744ecd2defb6b0d9" Nov 03 01:33:43.134402 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:33:43.134284 4939 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"710675672c4704b956985b778c9969f2de90e04a802951d0744ecd2defb6b0d9"} err="failed to get container status \"710675672c4704b956985b778c9969f2de90e04a802951d0744ecd2defb6b0d9\": rpc error: code = NotFound desc = could not find container \"710675672c4704b956985b778c9969f2de90e04a802951d0744ecd2defb6b0d9\": container with ID starting with 710675672c4704b956985b778c9969f2de90e04a802951d0744ecd2defb6b0d9 not found: ID does not exist" Nov 03 01:33:43.145653 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:33:43.145595 4939 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-a-ee618c-openshift-default-5ff9c74dc9-c4jpz"] Nov 03 01:33:43.153723 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:33:43.153698 4939 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-ee618c-openshift-default-5ff9c74dc9-ssz95"] Nov 03 01:33:43.159232 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:33:43.159208 4939 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-a-ee618c-openshift-default-5ff9c74dc9-ssz95"] Nov 03 01:33:43.163413 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:33:43.163391 4939 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-ee618c-openshift-default-5ff9c74dc9-z4qxq"] Nov 03 01:33:43.167288 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:33:43.167265 4939 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-a-ee618c-openshift-default-5ff9c74dc9-z4qxq"] Nov 03 01:33:43.689123 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:33:43.689084 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:33:43.872181 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:33:43.872137 4939 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6165abef-0df3-4007-ae59-5536a2a77e45" path="/var/lib/kubelet/pods/6165abef-0df3-4007-ae59-5536a2a77e45/volumes" Nov 03 01:33:43.872755 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:33:43.872740 4939 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7fc8db38-3791-40c0-bcfb-716313223a85" path="/var/lib/kubelet/pods/7fc8db38-3791-40c0-bcfb-716313223a85/volumes" Nov 03 01:33:43.873340 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:33:43.873323 4939 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fef89d35-3c5f-483f-b896-7b47733ab2df" path="/var/lib/kubelet/pods/fef89d35-3c5f-483f-b896-7b47733ab2df/volumes" Nov 03 01:33:44.324759 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:33:44.324719 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:33:44.431043 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:33:44.430999 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:33:44.599934 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:33:44.596390 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:33:45.372972 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:33:45.372879 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:33:45.556717 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:33:45.556688 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:33:46.177076 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:33:46.177039 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:33:46.793048 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:33:46.793002 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:33:47.516931 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:33:47.515752 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:33:48.037786 ip-10-0-140-107 kubenswrapper[4939]: E1103 01:33:48.037745 4939 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-sthjh" containerName="extractor" Nov 03 01:33:48.087354 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:33:48.087318 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:33:48.771882 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:33:48.771856 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:33:49.066483 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:33:49.066453 4939 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-ee618c-8bc685f5f-l65d9" Nov 03 01:33:49.138112 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:33:49.138073 4939 generic.go:356] "Generic (PLEG): container finished" podID="3d52a5d5-9404-4bdf-b6cb-9f134f145c6f" containerID="308485434bd7c443355f49edf1854b03d0c5193f0af258a6abd6cb492780c05a" exitCode=0 Nov 03 01:33:49.138318 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:33:49.138131 4939 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-ee618c-8bc685f5f-l65d9" Nov 03 01:33:49.138318 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:33:49.138162 4939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-ee618c-8bc685f5f-l65d9" event={"ID":"3d52a5d5-9404-4bdf-b6cb-9f134f145c6f","Type":"ContainerDied","Data":"308485434bd7c443355f49edf1854b03d0c5193f0af258a6abd6cb492780c05a"} Nov 03 01:33:49.138318 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:33:49.138224 4939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-ee618c-8bc685f5f-l65d9" event={"ID":"3d52a5d5-9404-4bdf-b6cb-9f134f145c6f","Type":"ContainerDied","Data":"c2e55a208f55e41b02ea24625063b2fbb6289a409b1f725c24106d374acb7615"} Nov 03 01:33:49.138318 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:33:49.138255 4939 scope.go:122] "RemoveContainer" containerID="308485434bd7c443355f49edf1854b03d0c5193f0af258a6abd6cb492780c05a" Nov 03 01:33:49.139515 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:33:49.139492 4939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3d52a5d5-9404-4bdf-b6cb-9f134f145c6f-kube-api-access-vjf69\" (UniqueName: \"kubernetes.io/projected/3d52a5d5-9404-4bdf-b6cb-9f134f145c6f-kube-api-access-vjf69\") pod \"3d52a5d5-9404-4bdf-b6cb-9f134f145c6f\" (UID: \"3d52a5d5-9404-4bdf-b6cb-9f134f145c6f\") " Nov 03 01:33:49.139734 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:33:49.139711 4939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3d52a5d5-9404-4bdf-b6cb-9f134f145c6f-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3d52a5d5-9404-4bdf-b6cb-9f134f145c6f-plugins-config-volume\") pod \"3d52a5d5-9404-4bdf-b6cb-9f134f145c6f\" (UID: \"3d52a5d5-9404-4bdf-b6cb-9f134f145c6f\") " Nov 03 01:33:49.140185 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:33:49.140157 4939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3d52a5d5-9404-4bdf-b6cb-9f134f145c6f-plugins-config-volume" pod "3d52a5d5-9404-4bdf-b6cb-9f134f145c6f" (UID: "3d52a5d5-9404-4bdf-b6cb-9f134f145c6f"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 01:33:49.140889 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:33:49.140862 4939 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3d52a5d5-9404-4bdf-b6cb-9f134f145c6f-plugins-config-volume\") on node \"ip-10-0-140-107.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:33:49.142339 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:33:49.142317 4939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3d52a5d5-9404-4bdf-b6cb-9f134f145c6f-kube-api-access-vjf69" pod "3d52a5d5-9404-4bdf-b6cb-9f134f145c6f" (UID: "3d52a5d5-9404-4bdf-b6cb-9f134f145c6f"). InnerVolumeSpecName "kube-api-access-vjf69". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 01:33:49.175902 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:33:49.175877 4939 scope.go:122] "RemoveContainer" containerID="308485434bd7c443355f49edf1854b03d0c5193f0af258a6abd6cb492780c05a" Nov 03 01:33:49.176214 ip-10-0-140-107 kubenswrapper[4939]: E1103 01:33:49.176189 4939 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"308485434bd7c443355f49edf1854b03d0c5193f0af258a6abd6cb492780c05a\": container with ID starting with 308485434bd7c443355f49edf1854b03d0c5193f0af258a6abd6cb492780c05a not found: ID does not exist" containerID="308485434bd7c443355f49edf1854b03d0c5193f0af258a6abd6cb492780c05a" Nov 03 01:33:49.176306 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:33:49.176221 4939 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"308485434bd7c443355f49edf1854b03d0c5193f0af258a6abd6cb492780c05a"} err="failed to get container status \"308485434bd7c443355f49edf1854b03d0c5193f0af258a6abd6cb492780c05a\": rpc error: code = NotFound desc = could not find container \"308485434bd7c443355f49edf1854b03d0c5193f0af258a6abd6cb492780c05a\": container with ID starting with 308485434bd7c443355f49edf1854b03d0c5193f0af258a6abd6cb492780c05a not found: ID does not exist" Nov 03 01:33:49.243027 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:33:49.243002 4939 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vjf69\" (UniqueName: \"kubernetes.io/projected/3d52a5d5-9404-4bdf-b6cb-9f134f145c6f-kube-api-access-vjf69\") on node \"ip-10-0-140-107.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:33:49.465984 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:33:49.465949 4939 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-ee618c-8bc685f5f-l65d9"] Nov 03 01:33:49.470530 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:33:49.470503 4939 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-ee618c-8bc685f5f-l65d9"] Nov 03 01:33:49.507935 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:33:49.504239 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:33:49.872874 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:33:49.872837 4939 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3d52a5d5-9404-4bdf-b6cb-9f134f145c6f" path="/var/lib/kubelet/pods/3d52a5d5-9404-4bdf-b6cb-9f134f145c6f/volumes" Nov 03 01:33:49.947193 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:33:49.947161 4939 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-5bfc51-6c9fd54845-wq8nz"] Nov 03 01:33:49.947464 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:33:49.947417 4939 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-5bfc51-6c9fd54845-wq8nz" podUID="3ba1b5d8-378c-4014-ba58-0fdae6897601" containerName="maas-api" containerID="cri-o://923c231e27333ad1540fc3d3655559b6740928d450b5a843d4b26580a5885f97" gracePeriod=30 Nov 03 01:33:50.015984 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:33:50.015901 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:33:50.274131 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:33:50.270664 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:33:50.692052 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:33:50.692025 4939 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-5bfc51-6c9fd54845-wq8nz" Nov 03 01:33:50.780214 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:33:50.780167 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:33:50.825310 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:33:50.825230 4939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3ba1b5d8-378c-4014-ba58-0fdae6897601-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3ba1b5d8-378c-4014-ba58-0fdae6897601-openshift-service-ca\") pod \"3ba1b5d8-378c-4014-ba58-0fdae6897601\" (UID: \"3ba1b5d8-378c-4014-ba58-0fdae6897601\") " Nov 03 01:33:50.825310 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:33:50.825300 4939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3ba1b5d8-378c-4014-ba58-0fdae6897601-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3ba1b5d8-378c-4014-ba58-0fdae6897601-maas-api-tls\") pod \"3ba1b5d8-378c-4014-ba58-0fdae6897601\" (UID: \"3ba1b5d8-378c-4014-ba58-0fdae6897601\") " Nov 03 01:33:50.825550 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:33:50.825411 4939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3ba1b5d8-378c-4014-ba58-0fdae6897601-kube-api-access-z5d87\" (UniqueName: \"kubernetes.io/projected/3ba1b5d8-378c-4014-ba58-0fdae6897601-kube-api-access-z5d87\") pod \"3ba1b5d8-378c-4014-ba58-0fdae6897601\" (UID: \"3ba1b5d8-378c-4014-ba58-0fdae6897601\") " Nov 03 01:33:50.825550 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:33:50.825444 4939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3ba1b5d8-378c-4014-ba58-0fdae6897601-metrics-certs\" (UniqueName: \"kubernetes.io/secret/3ba1b5d8-378c-4014-ba58-0fdae6897601-metrics-certs\") pod \"3ba1b5d8-378c-4014-ba58-0fdae6897601\" (UID: \"3ba1b5d8-378c-4014-ba58-0fdae6897601\") " Nov 03 01:33:50.825683 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:33:50.825627 4939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3ba1b5d8-378c-4014-ba58-0fdae6897601-openshift-service-ca" pod "3ba1b5d8-378c-4014-ba58-0fdae6897601" (UID: "3ba1b5d8-378c-4014-ba58-0fdae6897601"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 01:33:50.828984 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:33:50.826486 4939 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3ba1b5d8-378c-4014-ba58-0fdae6897601-openshift-service-ca\") on node \"ip-10-0-140-107.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:33:50.828984 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:33:50.828243 4939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3ba1b5d8-378c-4014-ba58-0fdae6897601-maas-api-tls" pod "3ba1b5d8-378c-4014-ba58-0fdae6897601" (UID: "3ba1b5d8-378c-4014-ba58-0fdae6897601"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 01:33:50.828984 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:33:50.828637 4939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3ba1b5d8-378c-4014-ba58-0fdae6897601-kube-api-access-z5d87" pod "3ba1b5d8-378c-4014-ba58-0fdae6897601" (UID: "3ba1b5d8-378c-4014-ba58-0fdae6897601"). InnerVolumeSpecName "kube-api-access-z5d87". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 01:33:50.888428 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:33:50.888388 4939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3ba1b5d8-378c-4014-ba58-0fdae6897601-metrics-certs" pod "3ba1b5d8-378c-4014-ba58-0fdae6897601" (UID: "3ba1b5d8-378c-4014-ba58-0fdae6897601"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 01:33:50.906989 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:33:50.906952 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:33:50.928184 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:33:50.928146 4939 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3ba1b5d8-378c-4014-ba58-0fdae6897601-maas-api-tls\") on node \"ip-10-0-140-107.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:33:50.928184 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:33:50.928179 4939 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-z5d87\" (UniqueName: \"kubernetes.io/projected/3ba1b5d8-378c-4014-ba58-0fdae6897601-kube-api-access-z5d87\") on node \"ip-10-0-140-107.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:33:50.928184 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:33:50.928193 4939 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3ba1b5d8-378c-4014-ba58-0fdae6897601-metrics-certs\") on node \"ip-10-0-140-107.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:33:51.100671 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:33:51.100637 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:33:51.257186 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:33:51.257152 4939 generic.go:356] "Generic (PLEG): container finished" podID="3ba1b5d8-378c-4014-ba58-0fdae6897601" containerID="923c231e27333ad1540fc3d3655559b6740928d450b5a843d4b26580a5885f97" exitCode=0 Nov 03 01:33:51.257347 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:33:51.257230 4939 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-5bfc51-6c9fd54845-wq8nz" Nov 03 01:33:51.257347 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:33:51.257247 4939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-5bfc51-6c9fd54845-wq8nz" event={"ID":"3ba1b5d8-378c-4014-ba58-0fdae6897601","Type":"ContainerDied","Data":"923c231e27333ad1540fc3d3655559b6740928d450b5a843d4b26580a5885f97"} Nov 03 01:33:51.257347 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:33:51.257323 4939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-5bfc51-6c9fd54845-wq8nz" event={"ID":"3ba1b5d8-378c-4014-ba58-0fdae6897601","Type":"ContainerDied","Data":"7bb5f46c9d7c2654db2a3ee9597511632b96b46234faaf4df4682faa345f6d20"} Nov 03 01:33:51.257522 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:33:51.257357 4939 scope.go:122] "RemoveContainer" containerID="923c231e27333ad1540fc3d3655559b6740928d450b5a843d4b26580a5885f97" Nov 03 01:33:51.271324 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:33:51.271306 4939 scope.go:122] "RemoveContainer" containerID="923c231e27333ad1540fc3d3655559b6740928d450b5a843d4b26580a5885f97" Nov 03 01:33:51.271622 ip-10-0-140-107 kubenswrapper[4939]: E1103 01:33:51.271593 4939 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"923c231e27333ad1540fc3d3655559b6740928d450b5a843d4b26580a5885f97\": container with ID starting with 923c231e27333ad1540fc3d3655559b6740928d450b5a843d4b26580a5885f97 not found: ID does not exist" containerID="923c231e27333ad1540fc3d3655559b6740928d450b5a843d4b26580a5885f97" Nov 03 01:33:51.271686 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:33:51.271623 4939 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"923c231e27333ad1540fc3d3655559b6740928d450b5a843d4b26580a5885f97"} err="failed to get container status \"923c231e27333ad1540fc3d3655559b6740928d450b5a843d4b26580a5885f97\": rpc error: code = NotFound desc = could not find container \"923c231e27333ad1540fc3d3655559b6740928d450b5a843d4b26580a5885f97\": container with ID starting with 923c231e27333ad1540fc3d3655559b6740928d450b5a843d4b26580a5885f97 not found: ID does not exist" Nov 03 01:33:51.289021 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:33:51.289000 4939 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-5bfc51-6c9fd54845-wq8nz"] Nov 03 01:33:51.293449 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:33:51.293428 4939 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-5bfc51-6c9fd54845-wq8nz"] Nov 03 01:33:51.310643 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:33:51.310618 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:33:51.674464 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:33:51.674427 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:33:51.921932 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:33:51.918735 4939 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3ba1b5d8-378c-4014-ba58-0fdae6897601" path="/var/lib/kubelet/pods/3ba1b5d8-378c-4014-ba58-0fdae6897601/volumes" Nov 03 01:33:52.381987 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:33:52.381951 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:33:52.599859 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:33:52.599827 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:33:52.796251 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:33:52.796171 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:33:52.899275 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:33:52.899243 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:33:53.179788 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:33:53.179745 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:33:53.902180 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:33:53.902146 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:33:54.509788 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:33:54.509754 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:33:55.184257 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:33:55.184222 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:33:55.876612 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:33:55.876573 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:33:56.497497 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:33:56.497461 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:33:57.084153 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:33:57.084091 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:33:57.696872 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:33:57.696838 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:33:58.565869 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:33:58.565828 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:33:59.141481 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:33:59.141400 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:33:59.261303 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:33:59.261260 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:33:59.939148 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:33:59.939114 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:34:00.585214 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:34:00.585180 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:34:01.184261 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:34:01.184227 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:34:01.845563 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:34:01.845528 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:34:02.451333 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:34:02.451301 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:34:03.372549 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:34:03.372450 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:34:03.986889 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:34:03.986844 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:34:04.623989 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:34:04.623949 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:34:05.308138 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:34:05.308092 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:34:05.965546 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:34:05.965515 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:34:06.598552 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:34:06.598499 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:34:07.249822 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:34:07.249788 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:34:07.762435 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:34:07.762401 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:34:07.926771 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:34:07.926737 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:34:08.610895 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:34:08.610856 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:34:08.830931 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:34:08.807288 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:34:09.367050 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:34:09.366994 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:34:09.483095 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:34:09.483058 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:34:09.592450 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:34:09.592418 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:34:09.733861 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:34:09.733777 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:34:10.529355 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:34:10.529316 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:34:10.675006 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:34:10.673507 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:34:10.856023 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:34:10.855978 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:34:11.413897 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:34:11.413859 4939 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-4pw7v"] Nov 03 01:34:11.700054 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:34:11.699946 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:34:12.988195 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:34:12.988128 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:34:13.605955 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:34:13.605919 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:34:14.307324 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:34:14.307213 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:34:14.893683 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:34:14.893648 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:34:15.684676 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:34:15.684633 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:34:15.846425 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:34:15.846390 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:34:16.282475 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:34:16.282443 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:34:16.867781 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:34:16.867750 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:34:17.497322 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:34:17.497285 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:34:18.034989 ip-10-0-140-107 kubenswrapper[4939]: E1103 01:34:18.034947 4939 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-sthjh" containerName="extractor" Nov 03 01:34:18.088357 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:34:18.088322 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:34:18.686181 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:34:18.686146 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:34:19.360636 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:34:19.360602 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:34:19.993891 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:34:19.993859 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:34:20.641855 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:34:20.641813 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:34:21.348086 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:34:21.348027 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:34:22.053674 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:34:22.053631 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:34:22.713704 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:34:22.713669 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:34:22.844934 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:34:22.832007 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:34:23.379318 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:34:23.379284 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:34:24.174417 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:34:24.174385 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:34:24.785647 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:34:24.785614 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:34:25.469903 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:34:25.469859 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:34:26.145362 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:34:26.145316 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:34:26.780626 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:34:26.780590 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:34:27.412349 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:34:27.412314 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:34:28.103653 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:34:28.103620 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:34:28.209953 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:34:28.207184 4939 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-737bf2d8-openshift-default-56dc96b957-vswqf"] Nov 03 01:34:28.209953 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:34:28.208590 4939 cpu_manager.go:414] "containerMap: removing container" podUID="fef89d35-3c5f-483f-b896-7b47733ab2df" containerName="istio-proxy" Nov 03 01:34:28.209953 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:34:28.208616 4939 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fef89d35-3c5f-483f-b896-7b47733ab2df" containerName="istio-proxy" Nov 03 01:34:28.209953 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:34:28.208655 4939 cpu_manager.go:414] "containerMap: removing container" podUID="3ba1b5d8-378c-4014-ba58-0fdae6897601" containerName="maas-api" Nov 03 01:34:28.209953 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:34:28.208663 4939 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3ba1b5d8-378c-4014-ba58-0fdae6897601" containerName="maas-api" Nov 03 01:34:28.209953 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:34:28.208694 4939 cpu_manager.go:414] "containerMap: removing container" podUID="6165abef-0df3-4007-ae59-5536a2a77e45" containerName="istio-proxy" Nov 03 01:34:28.209953 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:34:28.208703 4939 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6165abef-0df3-4007-ae59-5536a2a77e45" containerName="istio-proxy" Nov 03 01:34:28.209953 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:34:28.208753 4939 cpu_manager.go:414] "containerMap: removing container" podUID="7fc8db38-3791-40c0-bcfb-716313223a85" containerName="istio-proxy" Nov 03 01:34:28.209953 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:34:28.208762 4939 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7fc8db38-3791-40c0-bcfb-716313223a85" containerName="istio-proxy" Nov 03 01:34:28.209953 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:34:28.208775 4939 cpu_manager.go:414] "containerMap: removing container" podUID="abab7b3b-f57d-4abe-be59-295e4ab8a769" containerName="main" Nov 03 01:34:28.209953 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:34:28.208783 4939 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="abab7b3b-f57d-4abe-be59-295e4ab8a769" containerName="main" Nov 03 01:34:28.209953 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:34:28.208809 4939 cpu_manager.go:414] "containerMap: removing container" podUID="3d52a5d5-9404-4bdf-b6cb-9f134f145c6f" containerName="payload-pre-processing" Nov 03 01:34:28.209953 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:34:28.208817 4939 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3d52a5d5-9404-4bdf-b6cb-9f134f145c6f" containerName="payload-pre-processing" Nov 03 01:34:28.209953 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:34:28.208875 4939 cpu_manager.go:414] "containerMap: removing container" podUID="d0ed2957-3656-49bd-8aff-27ab5483d367" containerName="main" Nov 03 01:34:28.209953 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:34:28.208882 4939 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d0ed2957-3656-49bd-8aff-27ab5483d367" containerName="main" Nov 03 01:34:28.209953 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:34:28.209124 4939 memory_manager.go:365] "RemoveStaleState removing state" podUID="7fc8db38-3791-40c0-bcfb-716313223a85" containerName="istio-proxy" Nov 03 01:34:28.209953 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:34:28.209144 4939 memory_manager.go:365] "RemoveStaleState removing state" podUID="abab7b3b-f57d-4abe-be59-295e4ab8a769" containerName="main" Nov 03 01:34:28.209953 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:34:28.209160 4939 memory_manager.go:365] "RemoveStaleState removing state" podUID="3d52a5d5-9404-4bdf-b6cb-9f134f145c6f" containerName="payload-pre-processing" Nov 03 01:34:28.209953 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:34:28.209177 4939 memory_manager.go:365] "RemoveStaleState removing state" podUID="d0ed2957-3656-49bd-8aff-27ab5483d367" containerName="main" Nov 03 01:34:28.209953 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:34:28.209187 4939 memory_manager.go:365] "RemoveStaleState removing state" podUID="fef89d35-3c5f-483f-b896-7b47733ab2df" containerName="istio-proxy" Nov 03 01:34:28.209953 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:34:28.209199 4939 memory_manager.go:365] "RemoveStaleState removing state" podUID="3ba1b5d8-378c-4014-ba58-0fdae6897601" containerName="maas-api" Nov 03 01:34:28.209953 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:34:28.209212 4939 memory_manager.go:365] "RemoveStaleState removing state" podUID="6165abef-0df3-4007-ae59-5536a2a77e45" containerName="istio-proxy" Nov 03 01:34:28.217930 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:34:28.217309 4939 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-737bf2d8-openshift-default-56dc96b957-vswqf" Nov 03 01:34:28.233028 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:34:28.232993 4939 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-mt-737bf2d8-openshift-default-dockercfg-7f9q9\"" Nov 03 01:34:28.237069 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:34:28.237041 4939 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-737bf2d8-openshift-default-56dc96b957-vswqf"] Nov 03 01:34:28.335660 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:34:28.335626 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/80a21f03-7cfa-49f0-b35b-df8edc58932e-istiod-ca-cert\") pod \"e2e-mt-737bf2d8-openshift-default-56dc96b957-vswqf\" (UID: \"80a21f03-7cfa-49f0-b35b-df8edc58932e\") " pod="openshift-ingress/e2e-mt-737bf2d8-openshift-default-56dc96b957-vswqf" Nov 03 01:34:28.335848 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:34:28.335783 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/80a21f03-7cfa-49f0-b35b-df8edc58932e-workload-certs\") pod \"e2e-mt-737bf2d8-openshift-default-56dc96b957-vswqf\" (UID: \"80a21f03-7cfa-49f0-b35b-df8edc58932e\") " pod="openshift-ingress/e2e-mt-737bf2d8-openshift-default-56dc96b957-vswqf" Nov 03 01:34:28.335940 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:34:28.335852 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/80a21f03-7cfa-49f0-b35b-df8edc58932e-istio-podinfo\") pod \"e2e-mt-737bf2d8-openshift-default-56dc96b957-vswqf\" (UID: \"80a21f03-7cfa-49f0-b35b-df8edc58932e\") " pod="openshift-ingress/e2e-mt-737bf2d8-openshift-default-56dc96b957-vswqf" Nov 03 01:34:28.335940 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:34:28.335924 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mfpzb\" (UniqueName: \"kubernetes.io/projected/80a21f03-7cfa-49f0-b35b-df8edc58932e-kube-api-access-mfpzb\") pod \"e2e-mt-737bf2d8-openshift-default-56dc96b957-vswqf\" (UID: \"80a21f03-7cfa-49f0-b35b-df8edc58932e\") " pod="openshift-ingress/e2e-mt-737bf2d8-openshift-default-56dc96b957-vswqf" Nov 03 01:34:28.336039 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:34:28.336020 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/80a21f03-7cfa-49f0-b35b-df8edc58932e-workload-socket\") pod \"e2e-mt-737bf2d8-openshift-default-56dc96b957-vswqf\" (UID: \"80a21f03-7cfa-49f0-b35b-df8edc58932e\") " pod="openshift-ingress/e2e-mt-737bf2d8-openshift-default-56dc96b957-vswqf" Nov 03 01:34:28.336102 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:34:28.336087 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/80a21f03-7cfa-49f0-b35b-df8edc58932e-istio-envoy\") pod \"e2e-mt-737bf2d8-openshift-default-56dc96b957-vswqf\" (UID: \"80a21f03-7cfa-49f0-b35b-df8edc58932e\") " pod="openshift-ingress/e2e-mt-737bf2d8-openshift-default-56dc96b957-vswqf" Nov 03 01:34:28.336216 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:34:28.336202 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/80a21f03-7cfa-49f0-b35b-df8edc58932e-istio-token\") pod \"e2e-mt-737bf2d8-openshift-default-56dc96b957-vswqf\" (UID: \"80a21f03-7cfa-49f0-b35b-df8edc58932e\") " pod="openshift-ingress/e2e-mt-737bf2d8-openshift-default-56dc96b957-vswqf" Nov 03 01:34:28.336257 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:34:28.336238 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/80a21f03-7cfa-49f0-b35b-df8edc58932e-credential-socket\") pod \"e2e-mt-737bf2d8-openshift-default-56dc96b957-vswqf\" (UID: \"80a21f03-7cfa-49f0-b35b-df8edc58932e\") " pod="openshift-ingress/e2e-mt-737bf2d8-openshift-default-56dc96b957-vswqf" Nov 03 01:34:28.336315 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:34:28.336300 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/80a21f03-7cfa-49f0-b35b-df8edc58932e-istio-data\") pod \"e2e-mt-737bf2d8-openshift-default-56dc96b957-vswqf\" (UID: \"80a21f03-7cfa-49f0-b35b-df8edc58932e\") " pod="openshift-ingress/e2e-mt-737bf2d8-openshift-default-56dc96b957-vswqf" Nov 03 01:34:28.438493 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:34:28.438403 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/80a21f03-7cfa-49f0-b35b-df8edc58932e-istio-token\") pod \"e2e-mt-737bf2d8-openshift-default-56dc96b957-vswqf\" (UID: \"80a21f03-7cfa-49f0-b35b-df8edc58932e\") " pod="openshift-ingress/e2e-mt-737bf2d8-openshift-default-56dc96b957-vswqf" Nov 03 01:34:28.438493 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:34:28.438454 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/80a21f03-7cfa-49f0-b35b-df8edc58932e-credential-socket\") pod \"e2e-mt-737bf2d8-openshift-default-56dc96b957-vswqf\" (UID: \"80a21f03-7cfa-49f0-b35b-df8edc58932e\") " pod="openshift-ingress/e2e-mt-737bf2d8-openshift-default-56dc96b957-vswqf" Nov 03 01:34:28.439101 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:34:28.438548 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/80a21f03-7cfa-49f0-b35b-df8edc58932e-istio-data\") pod \"e2e-mt-737bf2d8-openshift-default-56dc96b957-vswqf\" (UID: \"80a21f03-7cfa-49f0-b35b-df8edc58932e\") " pod="openshift-ingress/e2e-mt-737bf2d8-openshift-default-56dc96b957-vswqf" Nov 03 01:34:28.439101 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:34:28.438654 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/80a21f03-7cfa-49f0-b35b-df8edc58932e-istiod-ca-cert\") pod \"e2e-mt-737bf2d8-openshift-default-56dc96b957-vswqf\" (UID: \"80a21f03-7cfa-49f0-b35b-df8edc58932e\") " pod="openshift-ingress/e2e-mt-737bf2d8-openshift-default-56dc96b957-vswqf" Nov 03 01:34:28.439101 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:34:28.438780 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/80a21f03-7cfa-49f0-b35b-df8edc58932e-workload-certs\") pod \"e2e-mt-737bf2d8-openshift-default-56dc96b957-vswqf\" (UID: \"80a21f03-7cfa-49f0-b35b-df8edc58932e\") " pod="openshift-ingress/e2e-mt-737bf2d8-openshift-default-56dc96b957-vswqf" Nov 03 01:34:28.439101 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:34:28.438836 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/80a21f03-7cfa-49f0-b35b-df8edc58932e-istio-podinfo\") pod \"e2e-mt-737bf2d8-openshift-default-56dc96b957-vswqf\" (UID: \"80a21f03-7cfa-49f0-b35b-df8edc58932e\") " pod="openshift-ingress/e2e-mt-737bf2d8-openshift-default-56dc96b957-vswqf" Nov 03 01:34:28.439101 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:34:28.438899 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/80a21f03-7cfa-49f0-b35b-df8edc58932e-credential-socket\") pod \"e2e-mt-737bf2d8-openshift-default-56dc96b957-vswqf\" (UID: \"80a21f03-7cfa-49f0-b35b-df8edc58932e\") " pod="openshift-ingress/e2e-mt-737bf2d8-openshift-default-56dc96b957-vswqf" Nov 03 01:34:28.439101 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:34:28.438936 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mfpzb\" (UniqueName: \"kubernetes.io/projected/80a21f03-7cfa-49f0-b35b-df8edc58932e-kube-api-access-mfpzb\") pod \"e2e-mt-737bf2d8-openshift-default-56dc96b957-vswqf\" (UID: \"80a21f03-7cfa-49f0-b35b-df8edc58932e\") " pod="openshift-ingress/e2e-mt-737bf2d8-openshift-default-56dc96b957-vswqf" Nov 03 01:34:28.439101 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:34:28.438981 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/80a21f03-7cfa-49f0-b35b-df8edc58932e-istio-data\") pod \"e2e-mt-737bf2d8-openshift-default-56dc96b957-vswqf\" (UID: \"80a21f03-7cfa-49f0-b35b-df8edc58932e\") " pod="openshift-ingress/e2e-mt-737bf2d8-openshift-default-56dc96b957-vswqf" Nov 03 01:34:28.439101 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:34:28.439036 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/80a21f03-7cfa-49f0-b35b-df8edc58932e-workload-socket\") pod \"e2e-mt-737bf2d8-openshift-default-56dc96b957-vswqf\" (UID: \"80a21f03-7cfa-49f0-b35b-df8edc58932e\") " pod="openshift-ingress/e2e-mt-737bf2d8-openshift-default-56dc96b957-vswqf" Nov 03 01:34:28.439101 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:34:28.439091 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/80a21f03-7cfa-49f0-b35b-df8edc58932e-istio-envoy\") pod \"e2e-mt-737bf2d8-openshift-default-56dc96b957-vswqf\" (UID: \"80a21f03-7cfa-49f0-b35b-df8edc58932e\") " pod="openshift-ingress/e2e-mt-737bf2d8-openshift-default-56dc96b957-vswqf" Nov 03 01:34:28.439650 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:34:28.439602 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/80a21f03-7cfa-49f0-b35b-df8edc58932e-workload-certs\") pod \"e2e-mt-737bf2d8-openshift-default-56dc96b957-vswqf\" (UID: \"80a21f03-7cfa-49f0-b35b-df8edc58932e\") " pod="openshift-ingress/e2e-mt-737bf2d8-openshift-default-56dc96b957-vswqf" Nov 03 01:34:28.439650 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:34:28.439637 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/80a21f03-7cfa-49f0-b35b-df8edc58932e-workload-socket\") pod \"e2e-mt-737bf2d8-openshift-default-56dc96b957-vswqf\" (UID: \"80a21f03-7cfa-49f0-b35b-df8edc58932e\") " pod="openshift-ingress/e2e-mt-737bf2d8-openshift-default-56dc96b957-vswqf" Nov 03 01:34:28.439814 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:34:28.439791 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/80a21f03-7cfa-49f0-b35b-df8edc58932e-istiod-ca-cert\") pod \"e2e-mt-737bf2d8-openshift-default-56dc96b957-vswqf\" (UID: \"80a21f03-7cfa-49f0-b35b-df8edc58932e\") " pod="openshift-ingress/e2e-mt-737bf2d8-openshift-default-56dc96b957-vswqf" Nov 03 01:34:28.441668 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:34:28.441634 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/80a21f03-7cfa-49f0-b35b-df8edc58932e-istio-envoy\") pod \"e2e-mt-737bf2d8-openshift-default-56dc96b957-vswqf\" (UID: \"80a21f03-7cfa-49f0-b35b-df8edc58932e\") " pod="openshift-ingress/e2e-mt-737bf2d8-openshift-default-56dc96b957-vswqf" Nov 03 01:34:28.441955 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:34:28.441896 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/80a21f03-7cfa-49f0-b35b-df8edc58932e-istio-podinfo\") pod \"e2e-mt-737bf2d8-openshift-default-56dc96b957-vswqf\" (UID: \"80a21f03-7cfa-49f0-b35b-df8edc58932e\") " pod="openshift-ingress/e2e-mt-737bf2d8-openshift-default-56dc96b957-vswqf" Nov 03 01:34:28.446730 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:34:28.446705 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/80a21f03-7cfa-49f0-b35b-df8edc58932e-istio-token\") pod \"e2e-mt-737bf2d8-openshift-default-56dc96b957-vswqf\" (UID: \"80a21f03-7cfa-49f0-b35b-df8edc58932e\") " pod="openshift-ingress/e2e-mt-737bf2d8-openshift-default-56dc96b957-vswqf" Nov 03 01:34:28.446937 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:34:28.446901 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mfpzb\" (UniqueName: \"kubernetes.io/projected/80a21f03-7cfa-49f0-b35b-df8edc58932e-kube-api-access-mfpzb\") pod \"e2e-mt-737bf2d8-openshift-default-56dc96b957-vswqf\" (UID: \"80a21f03-7cfa-49f0-b35b-df8edc58932e\") " pod="openshift-ingress/e2e-mt-737bf2d8-openshift-default-56dc96b957-vswqf" Nov 03 01:34:28.573707 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:34:28.573661 4939 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-737bf2d8-openshift-default-56dc96b957-vswqf" Nov 03 01:34:28.871518 ip-10-0-140-107 kubenswrapper[4939]: W1103 01:34:28.871481 4939 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod80a21f03_7cfa_49f0_b35b_df8edc58932e.slice/crio-e9efcfc60b3f95ba4b3a09c263e53ada2a466d49acf5efaced2cb2390fba09a0 WatchSource:0}: Error finding container e9efcfc60b3f95ba4b3a09c263e53ada2a466d49acf5efaced2cb2390fba09a0: Status 404 returned error can't find the container with id e9efcfc60b3f95ba4b3a09c263e53ada2a466d49acf5efaced2cb2390fba09a0 Nov 03 01:34:28.872497 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:34:28.872466 4939 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-737bf2d8-openshift-default-56dc96b957-vswqf"] Nov 03 01:34:28.874704 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:34:28.874666 4939 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 01:34:28.874808 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:34:28.874742 4939 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 01:34:28.874808 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:34:28.874770 4939 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 01:34:28.889336 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:34:28.889312 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:34:29.398116 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:34:29.398076 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:34:29.586302 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:34:29.586258 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:34:29.782504 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:34:29.782411 4939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-737bf2d8-openshift-default-56dc96b957-vswqf" event={"ID":"80a21f03-7cfa-49f0-b35b-df8edc58932e","Type":"ContainerStarted","Data":"1c960f788db7e960250ae0de477e4f2230f41cd1e4a7cf72bebca036c2eb6971"} Nov 03 01:34:29.783480 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:34:29.783447 4939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-737bf2d8-openshift-default-56dc96b957-vswqf" event={"ID":"80a21f03-7cfa-49f0-b35b-df8edc58932e","Type":"ContainerStarted","Data":"e9efcfc60b3f95ba4b3a09c263e53ada2a466d49acf5efaced2cb2390fba09a0"} Nov 03 01:34:29.809253 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:34:29.809199 4939 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-737bf2d8-openshift-default-56dc96b957-vswqf" podStartSLOduration=1.8091767220000001 podStartE2EDuration="1.809176722s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 01:34:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 01:34:29.80831181 +0000 UTC m=+3218.072646756" watchObservedRunningTime="2026-11-03 01:34:29.809176722 +0000 UTC m=+3218.073511742" Nov 03 01:34:30.306937 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:34:30.306892 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:34:30.574263 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:34:30.574182 4939 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-737bf2d8-openshift-default-56dc96b957-vswqf" Nov 03 01:34:30.575838 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:34:30.575813 4939 patch_prober.go:29] interesting pod/e2e-mt-737bf2d8-openshift-default-56dc96b957-vswqf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.74:15021/healthz/ready\": dial tcp 10.132.1.74:15021: connect: connection refused" start-of-body= Nov 03 01:34:30.576101 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:34:30.576076 4939 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-737bf2d8-openshift-default-56dc96b957-vswqf" podUID="80a21f03-7cfa-49f0-b35b-df8edc58932e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.74:15021/healthz/ready\": dial tcp 10.132.1.74:15021: connect: connection refused" Nov 03 01:34:31.026885 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:34:31.026842 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:34:31.575174 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:34:31.575130 4939 patch_prober.go:29] interesting pod/e2e-mt-737bf2d8-openshift-default-56dc96b957-vswqf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.74:15021/healthz/ready\": dial tcp 10.132.1.74:15021: connect: connection refused" start-of-body= Nov 03 01:34:31.575403 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:34:31.575215 4939 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-737bf2d8-openshift-default-56dc96b957-vswqf" podUID="80a21f03-7cfa-49f0-b35b-df8edc58932e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.74:15021/healthz/ready\": dial tcp 10.132.1.74:15021: connect: connection refused" Nov 03 01:34:31.795529 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:34:31.795492 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:34:32.465932 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:34:32.465427 4939 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w1-2729f2-openshift-default-9fb97b4fc-5ww48"] Nov 03 01:34:32.465932 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:34:32.465676 4939 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w1-2729f2-openshift-default-9fb97b4fc-5ww48" podUID="cba006d1-08cd-4fac-8bdd-86931f742838" containerName="istio-proxy" containerID="cri-o://ff61f9b0d6ca9947338dd8007679942bf635e5f5b02c19025e6d3a9789bce121" gracePeriod=30 Nov 03 01:34:32.484554 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:34:32.484506 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:34:32.587931 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:34:32.587358 4939 patch_prober.go:29] interesting pod/e2e-mt-737bf2d8-openshift-default-56dc96b957-vswqf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.74:15021/healthz/ready\": dial tcp 10.132.1.74:15021: connect: connection refused" start-of-body= Nov 03 01:34:32.587931 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:34:32.587436 4939 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-737bf2d8-openshift-default-56dc96b957-vswqf" podUID="80a21f03-7cfa-49f0-b35b-df8edc58932e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.74:15021/healthz/ready\": dial tcp 10.132.1.74:15021: connect: connection refused" Nov 03 01:34:33.189129 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:34:33.189092 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:34:33.574524 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:34:33.574424 4939 patch_prober.go:29] interesting pod/e2e-mt-737bf2d8-openshift-default-56dc96b957-vswqf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.74:15021/healthz/ready\": dial tcp 10.132.1.74:15021: connect: connection refused" start-of-body= Nov 03 01:34:33.574524 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:34:33.574484 4939 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-737bf2d8-openshift-default-56dc96b957-vswqf" podUID="80a21f03-7cfa-49f0-b35b-df8edc58932e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.74:15021/healthz/ready\": dial tcp 10.132.1.74:15021: connect: connection refused" Nov 03 01:34:34.135437 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:34:34.135392 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:34:34.342988 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:34:34.342951 4939 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w4-6c0ec0-openshift-default-79c7584bfc-8hz7h"] Nov 03 01:34:34.523393 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:34:34.523319 4939 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-83755a-openshift-default-c86f9565c-qlgb7"] Nov 03 01:34:34.523556 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:34:34.523524 4939 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-83755a-openshift-default-c86f9565c-qlgb7" podUID="ba28e265-fab5-43ba-94b2-dc5b7dd5029f" containerName="istio-proxy" containerID="cri-o://417abf9a2d53f92307b997d0dd67ef2989b4a47fbb9f30f80d4fcc23d92ab776" gracePeriod=30 Nov 03 01:34:34.577373 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:34:34.577301 4939 patch_prober.go:29] interesting pod/e2e-mt-737bf2d8-openshift-default-56dc96b957-vswqf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.74:15021/healthz/ready\": dial tcp 10.132.1.74:15021: connect: connection refused" start-of-body= Nov 03 01:34:34.578010 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:34:34.577972 4939 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-737bf2d8-openshift-default-56dc96b957-vswqf" podUID="80a21f03-7cfa-49f0-b35b-df8edc58932e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.74:15021/healthz/ready\": dial tcp 10.132.1.74:15021: connect: connection refused" Nov 03 01:34:34.837869 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:34:34.837834 4939 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w4-6c0ec0-openshift-default-79c7584bfc-8hz7h" podUID="0a1fe883-48d5-4d06-ab8f-41698965dbbb" containerName="istio-proxy" containerID="cri-o://2bcb1634a74979a9c04d145a53f1fc437c24da2608a17bfee61d92aed28c52e0" gracePeriod=30 Nov 03 01:34:34.893895 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:34:34.893865 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:34:35.579316 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:34:35.579282 4939 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-737bf2d8-openshift-default-56dc96b957-vswqf" Nov 03 01:34:35.579767 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:34:35.579743 4939 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-737bf2d8-openshift-default-56dc96b957-vswqf" Nov 03 01:34:35.580548 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:34:35.580527 4939 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-737bf2d8-openshift-default-56dc96b957-vswqf" Nov 03 01:34:35.610281 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:34:35.610253 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:34:35.874087 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:34:35.874035 4939 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w0-50e623-openshift-default-6c86dc794f-jmfmb"] Nov 03 01:34:35.874322 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:34:35.874292 4939 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w0-50e623-openshift-default-6c86dc794f-jmfmb" podUID="aba9a06e-b5d6-491f-b9a3-34ebd927d4aa" containerName="istio-proxy" containerID="cri-o://368f1834e8a5c7ca07bdaa894b6b966b769f839a8b95940ef3da8a33098f6079" gracePeriod=30 Nov 03 01:34:36.379009 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:34:36.378894 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:34:37.098304 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:34:37.098274 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:34:37.776484 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:34:37.776459 4939 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w1-2729f2-openshift-default-9fb97b4fc-5ww48" Nov 03 01:34:37.786248 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:34:37.786223 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:34:37.926373 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:34:37.924680 4939 generic.go:356] "Generic (PLEG): container finished" podID="cba006d1-08cd-4fac-8bdd-86931f742838" containerID="ff61f9b0d6ca9947338dd8007679942bf635e5f5b02c19025e6d3a9789bce121" exitCode=0 Nov 03 01:34:37.926373 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:34:37.925179 4939 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w1-2729f2-openshift-default-9fb97b4fc-5ww48" Nov 03 01:34:37.955601 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:34:37.955571 4939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w1-2729f2-openshift-default-9fb97b4fc-5ww48" event={"ID":"cba006d1-08cd-4fac-8bdd-86931f742838","Type":"ContainerDied","Data":"ff61f9b0d6ca9947338dd8007679942bf635e5f5b02c19025e6d3a9789bce121"} Nov 03 01:34:37.955857 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:34:37.955840 4939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w1-2729f2-openshift-default-9fb97b4fc-5ww48" event={"ID":"cba006d1-08cd-4fac-8bdd-86931f742838","Type":"ContainerDied","Data":"d66f4b4231c9bd1a18ea78d5a88c9121ffb031e3122fdd7dd4f418d7101c144a"} Nov 03 01:34:37.956015 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:34:37.955999 4939 scope.go:122] "RemoveContainer" containerID="ff61f9b0d6ca9947338dd8007679942bf635e5f5b02c19025e6d3a9789bce121" Nov 03 01:34:37.981142 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:34:37.981114 4939 scope.go:122] "RemoveContainer" containerID="c7884d3f229a4cf4c41b83c524ce2da09be6b780b239aa5ea0bdd93e1ac60024" Nov 03 01:34:37.993731 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:34:37.993704 4939 scope.go:122] "RemoveContainer" containerID="ff61f9b0d6ca9947338dd8007679942bf635e5f5b02c19025e6d3a9789bce121" Nov 03 01:34:37.994050 ip-10-0-140-107 kubenswrapper[4939]: E1103 01:34:37.994027 4939 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ff61f9b0d6ca9947338dd8007679942bf635e5f5b02c19025e6d3a9789bce121\": container with ID starting with ff61f9b0d6ca9947338dd8007679942bf635e5f5b02c19025e6d3a9789bce121 not found: ID does not exist" containerID="ff61f9b0d6ca9947338dd8007679942bf635e5f5b02c19025e6d3a9789bce121" Nov 03 01:34:37.994145 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:34:37.994057 4939 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ff61f9b0d6ca9947338dd8007679942bf635e5f5b02c19025e6d3a9789bce121"} err="failed to get container status \"ff61f9b0d6ca9947338dd8007679942bf635e5f5b02c19025e6d3a9789bce121\": rpc error: code = NotFound desc = could not find container \"ff61f9b0d6ca9947338dd8007679942bf635e5f5b02c19025e6d3a9789bce121\": container with ID starting with ff61f9b0d6ca9947338dd8007679942bf635e5f5b02c19025e6d3a9789bce121 not found: ID does not exist" Nov 03 01:34:37.994145 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:34:37.994073 4939 scope.go:122] "RemoveContainer" containerID="c7884d3f229a4cf4c41b83c524ce2da09be6b780b239aa5ea0bdd93e1ac60024" Nov 03 01:34:37.994391 ip-10-0-140-107 kubenswrapper[4939]: E1103 01:34:37.994369 4939 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c7884d3f229a4cf4c41b83c524ce2da09be6b780b239aa5ea0bdd93e1ac60024\": container with ID starting with c7884d3f229a4cf4c41b83c524ce2da09be6b780b239aa5ea0bdd93e1ac60024 not found: ID does not exist" containerID="c7884d3f229a4cf4c41b83c524ce2da09be6b780b239aa5ea0bdd93e1ac60024" Nov 03 01:34:37.994451 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:34:37.994393 4939 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c7884d3f229a4cf4c41b83c524ce2da09be6b780b239aa5ea0bdd93e1ac60024"} err="failed to get container status \"c7884d3f229a4cf4c41b83c524ce2da09be6b780b239aa5ea0bdd93e1ac60024\": rpc error: code = NotFound desc = could not find container \"c7884d3f229a4cf4c41b83c524ce2da09be6b780b239aa5ea0bdd93e1ac60024\": container with ID starting with c7884d3f229a4cf4c41b83c524ce2da09be6b780b239aa5ea0bdd93e1ac60024 not found: ID does not exist" Nov 03 01:34:38.004830 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:34:38.004804 4939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cba006d1-08cd-4fac-8bdd-86931f742838-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cba006d1-08cd-4fac-8bdd-86931f742838-istio-data\") pod \"cba006d1-08cd-4fac-8bdd-86931f742838\" (UID: \"cba006d1-08cd-4fac-8bdd-86931f742838\") " Nov 03 01:34:38.005101 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:34:38.005075 4939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/cba006d1-08cd-4fac-8bdd-86931f742838-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cba006d1-08cd-4fac-8bdd-86931f742838-istio-podinfo\") pod \"cba006d1-08cd-4fac-8bdd-86931f742838\" (UID: \"cba006d1-08cd-4fac-8bdd-86931f742838\") " Nov 03 01:34:38.005194 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:34:38.005125 4939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cba006d1-08cd-4fac-8bdd-86931f742838-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cba006d1-08cd-4fac-8bdd-86931f742838-istio-envoy\") pod \"cba006d1-08cd-4fac-8bdd-86931f742838\" (UID: \"cba006d1-08cd-4fac-8bdd-86931f742838\") " Nov 03 01:34:38.005266 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:34:38.005208 4939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cba006d1-08cd-4fac-8bdd-86931f742838-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cba006d1-08cd-4fac-8bdd-86931f742838-credential-socket\") pod \"cba006d1-08cd-4fac-8bdd-86931f742838\" (UID: \"cba006d1-08cd-4fac-8bdd-86931f742838\") " Nov 03 01:34:38.005358 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:34:38.005342 4939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cba006d1-08cd-4fac-8bdd-86931f742838-kube-api-access-6wqxc\" (UniqueName: \"kubernetes.io/projected/cba006d1-08cd-4fac-8bdd-86931f742838-kube-api-access-6wqxc\") pod \"cba006d1-08cd-4fac-8bdd-86931f742838\" (UID: \"cba006d1-08cd-4fac-8bdd-86931f742838\") " Nov 03 01:34:38.005423 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:34:38.005389 4939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cba006d1-08cd-4fac-8bdd-86931f742838-istio-token\" (UniqueName: \"kubernetes.io/projected/cba006d1-08cd-4fac-8bdd-86931f742838-istio-token\") pod \"cba006d1-08cd-4fac-8bdd-86931f742838\" (UID: \"cba006d1-08cd-4fac-8bdd-86931f742838\") " Nov 03 01:34:38.005423 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:34:38.005411 4939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cba006d1-08cd-4fac-8bdd-86931f742838-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cba006d1-08cd-4fac-8bdd-86931f742838-istiod-ca-cert\") pod \"cba006d1-08cd-4fac-8bdd-86931f742838\" (UID: \"cba006d1-08cd-4fac-8bdd-86931f742838\") " Nov 03 01:34:38.005547 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:34:38.005431 4939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cba006d1-08cd-4fac-8bdd-86931f742838-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cba006d1-08cd-4fac-8bdd-86931f742838-workload-socket\") pod \"cba006d1-08cd-4fac-8bdd-86931f742838\" (UID: \"cba006d1-08cd-4fac-8bdd-86931f742838\") " Nov 03 01:34:38.005547 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:34:38.005458 4939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cba006d1-08cd-4fac-8bdd-86931f742838-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cba006d1-08cd-4fac-8bdd-86931f742838-workload-certs\") pod \"cba006d1-08cd-4fac-8bdd-86931f742838\" (UID: \"cba006d1-08cd-4fac-8bdd-86931f742838\") " Nov 03 01:34:38.005878 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:34:38.005853 4939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cba006d1-08cd-4fac-8bdd-86931f742838-istio-data" pod "cba006d1-08cd-4fac-8bdd-86931f742838" (UID: "cba006d1-08cd-4fac-8bdd-86931f742838"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 01:34:38.006165 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:34:38.006143 4939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cba006d1-08cd-4fac-8bdd-86931f742838-workload-socket" pod "cba006d1-08cd-4fac-8bdd-86931f742838" (UID: "cba006d1-08cd-4fac-8bdd-86931f742838"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 01:34:38.006344 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:34:38.006301 4939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cba006d1-08cd-4fac-8bdd-86931f742838-credential-socket" pod "cba006d1-08cd-4fac-8bdd-86931f742838" (UID: "cba006d1-08cd-4fac-8bdd-86931f742838"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 01:34:38.006422 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:34:38.006371 4939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cba006d1-08cd-4fac-8bdd-86931f742838-workload-certs" pod "cba006d1-08cd-4fac-8bdd-86931f742838" (UID: "cba006d1-08cd-4fac-8bdd-86931f742838"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 01:34:38.006559 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:34:38.006532 4939 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cba006d1-08cd-4fac-8bdd-86931f742838-credential-socket\") on node \"ip-10-0-140-107.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:34:38.006670 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:34:38.006561 4939 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cba006d1-08cd-4fac-8bdd-86931f742838-workload-socket\") on node \"ip-10-0-140-107.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:34:38.006670 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:34:38.006577 4939 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cba006d1-08cd-4fac-8bdd-86931f742838-workload-certs\") on node \"ip-10-0-140-107.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:34:38.006670 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:34:38.006622 4939 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cba006d1-08cd-4fac-8bdd-86931f742838-istio-data\") on node \"ip-10-0-140-107.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:34:38.006850 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:34:38.006740 4939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cba006d1-08cd-4fac-8bdd-86931f742838-istiod-ca-cert" pod "cba006d1-08cd-4fac-8bdd-86931f742838" (UID: "cba006d1-08cd-4fac-8bdd-86931f742838"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 01:34:38.008552 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:34:38.008519 4939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cba006d1-08cd-4fac-8bdd-86931f742838-istio-token" pod "cba006d1-08cd-4fac-8bdd-86931f742838" (UID: "cba006d1-08cd-4fac-8bdd-86931f742838"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 01:34:38.012504 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:34:38.012483 4939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/cba006d1-08cd-4fac-8bdd-86931f742838-istio-podinfo" pod "cba006d1-08cd-4fac-8bdd-86931f742838" (UID: "cba006d1-08cd-4fac-8bdd-86931f742838"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 01:34:38.012613 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:34:38.012499 4939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cba006d1-08cd-4fac-8bdd-86931f742838-istio-envoy" pod "cba006d1-08cd-4fac-8bdd-86931f742838" (UID: "cba006d1-08cd-4fac-8bdd-86931f742838"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 01:34:38.013922 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:34:38.013885 4939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cba006d1-08cd-4fac-8bdd-86931f742838-kube-api-access-6wqxc" pod "cba006d1-08cd-4fac-8bdd-86931f742838" (UID: "cba006d1-08cd-4fac-8bdd-86931f742838"). InnerVolumeSpecName "kube-api-access-6wqxc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 01:34:38.108350 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:34:38.108322 4939 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cba006d1-08cd-4fac-8bdd-86931f742838-istio-podinfo\") on node \"ip-10-0-140-107.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:34:38.108350 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:34:38.108350 4939 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cba006d1-08cd-4fac-8bdd-86931f742838-istio-envoy\") on node \"ip-10-0-140-107.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:34:38.108733 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:34:38.108363 4939 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6wqxc\" (UniqueName: \"kubernetes.io/projected/cba006d1-08cd-4fac-8bdd-86931f742838-kube-api-access-6wqxc\") on node \"ip-10-0-140-107.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:34:38.108733 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:34:38.108378 4939 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cba006d1-08cd-4fac-8bdd-86931f742838-istio-token\") on node \"ip-10-0-140-107.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:34:38.108733 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:34:38.108390 4939 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cba006d1-08cd-4fac-8bdd-86931f742838-istiod-ca-cert\") on node \"ip-10-0-140-107.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:34:38.254692 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:34:38.254614 4939 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w1-2729f2-openshift-default-9fb97b4fc-5ww48"] Nov 03 01:34:38.260264 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:34:38.260236 4939 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w1-2729f2-openshift-default-9fb97b4fc-5ww48"] Nov 03 01:34:38.393021 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:34:38.392989 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:34:38.999058 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:34:38.999018 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:34:39.696371 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:34:39.696330 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:34:39.879586 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:34:39.879554 4939 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cba006d1-08cd-4fac-8bdd-86931f742838" path="/var/lib/kubelet/pods/cba006d1-08cd-4fac-8bdd-86931f742838/volumes" Nov 03 01:34:39.949679 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:34:39.949640 4939 generic.go:356] "Generic (PLEG): container finished" podID="ba28e265-fab5-43ba-94b2-dc5b7dd5029f" containerID="417abf9a2d53f92307b997d0dd67ef2989b4a47fbb9f30f80d4fcc23d92ab776" exitCode=0 Nov 03 01:34:39.949873 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:34:39.949713 4939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-83755a-openshift-default-c86f9565c-qlgb7" event={"ID":"ba28e265-fab5-43ba-94b2-dc5b7dd5029f","Type":"ContainerDied","Data":"417abf9a2d53f92307b997d0dd67ef2989b4a47fbb9f30f80d4fcc23d92ab776"} Nov 03 01:34:39.949873 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:34:39.949775 4939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-83755a-openshift-default-c86f9565c-qlgb7" event={"ID":"ba28e265-fab5-43ba-94b2-dc5b7dd5029f","Type":"ContainerDied","Data":"dcee058b87d0628003e1115acf538db0ef71c6e3326e04ecfee850cccbd04f11"} Nov 03 01:34:39.949873 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:34:39.949799 4939 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="dcee058b87d0628003e1115acf538db0ef71c6e3326e04ecfee850cccbd04f11" Nov 03 01:34:39.965199 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:34:39.965177 4939 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-83755a-openshift-default-c86f9565c-qlgb7" Nov 03 01:34:40.089475 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:34:40.089441 4939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ba28e265-fab5-43ba-94b2-dc5b7dd5029f-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ba28e265-fab5-43ba-94b2-dc5b7dd5029f-istiod-ca-cert\") pod \"ba28e265-fab5-43ba-94b2-dc5b7dd5029f\" (UID: \"ba28e265-fab5-43ba-94b2-dc5b7dd5029f\") " Nov 03 01:34:40.089641 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:34:40.089538 4939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ba28e265-fab5-43ba-94b2-dc5b7dd5029f-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ba28e265-fab5-43ba-94b2-dc5b7dd5029f-credential-socket\") pod \"ba28e265-fab5-43ba-94b2-dc5b7dd5029f\" (UID: \"ba28e265-fab5-43ba-94b2-dc5b7dd5029f\") " Nov 03 01:34:40.089641 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:34:40.089573 4939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ba28e265-fab5-43ba-94b2-dc5b7dd5029f-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ba28e265-fab5-43ba-94b2-dc5b7dd5029f-workload-socket\") pod \"ba28e265-fab5-43ba-94b2-dc5b7dd5029f\" (UID: \"ba28e265-fab5-43ba-94b2-dc5b7dd5029f\") " Nov 03 01:34:40.089771 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:34:40.089679 4939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ba28e265-fab5-43ba-94b2-dc5b7dd5029f-kube-api-access-xg7pd\" (UniqueName: \"kubernetes.io/projected/ba28e265-fab5-43ba-94b2-dc5b7dd5029f-kube-api-access-xg7pd\") pod \"ba28e265-fab5-43ba-94b2-dc5b7dd5029f\" (UID: \"ba28e265-fab5-43ba-94b2-dc5b7dd5029f\") " Nov 03 01:34:40.089829 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:34:40.089820 4939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ba28e265-fab5-43ba-94b2-dc5b7dd5029f-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ba28e265-fab5-43ba-94b2-dc5b7dd5029f-istio-envoy\") pod \"ba28e265-fab5-43ba-94b2-dc5b7dd5029f\" (UID: \"ba28e265-fab5-43ba-94b2-dc5b7dd5029f\") " Nov 03 01:34:40.089962 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:34:40.089945 4939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ba28e265-fab5-43ba-94b2-dc5b7dd5029f-istio-token\" (UniqueName: \"kubernetes.io/projected/ba28e265-fab5-43ba-94b2-dc5b7dd5029f-istio-token\") pod \"ba28e265-fab5-43ba-94b2-dc5b7dd5029f\" (UID: \"ba28e265-fab5-43ba-94b2-dc5b7dd5029f\") " Nov 03 01:34:40.090043 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:34:40.089967 4939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ba28e265-fab5-43ba-94b2-dc5b7dd5029f-istiod-ca-cert" pod "ba28e265-fab5-43ba-94b2-dc5b7dd5029f" (UID: "ba28e265-fab5-43ba-94b2-dc5b7dd5029f"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 01:34:40.090043 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:34:40.090022 4939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ba28e265-fab5-43ba-94b2-dc5b7dd5029f-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ba28e265-fab5-43ba-94b2-dc5b7dd5029f-istio-data\") pod \"ba28e265-fab5-43ba-94b2-dc5b7dd5029f\" (UID: \"ba28e265-fab5-43ba-94b2-dc5b7dd5029f\") " Nov 03 01:34:40.090157 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:34:40.090068 4939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/ba28e265-fab5-43ba-94b2-dc5b7dd5029f-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ba28e265-fab5-43ba-94b2-dc5b7dd5029f-istio-podinfo\") pod \"ba28e265-fab5-43ba-94b2-dc5b7dd5029f\" (UID: \"ba28e265-fab5-43ba-94b2-dc5b7dd5029f\") " Nov 03 01:34:40.090157 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:34:40.090142 4939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ba28e265-fab5-43ba-94b2-dc5b7dd5029f-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ba28e265-fab5-43ba-94b2-dc5b7dd5029f-workload-certs\") pod \"ba28e265-fab5-43ba-94b2-dc5b7dd5029f\" (UID: \"ba28e265-fab5-43ba-94b2-dc5b7dd5029f\") " Nov 03 01:34:40.090856 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:34:40.090497 4939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ba28e265-fab5-43ba-94b2-dc5b7dd5029f-istio-data" pod "ba28e265-fab5-43ba-94b2-dc5b7dd5029f" (UID: "ba28e265-fab5-43ba-94b2-dc5b7dd5029f"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 01:34:40.090856 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:34:40.090576 4939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ba28e265-fab5-43ba-94b2-dc5b7dd5029f-workload-socket" pod "ba28e265-fab5-43ba-94b2-dc5b7dd5029f" (UID: "ba28e265-fab5-43ba-94b2-dc5b7dd5029f"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 01:34:40.090856 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:34:40.090774 4939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ba28e265-fab5-43ba-94b2-dc5b7dd5029f-workload-certs" pod "ba28e265-fab5-43ba-94b2-dc5b7dd5029f" (UID: "ba28e265-fab5-43ba-94b2-dc5b7dd5029f"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 01:34:40.091124 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:34:40.090849 4939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ba28e265-fab5-43ba-94b2-dc5b7dd5029f-credential-socket" pod "ba28e265-fab5-43ba-94b2-dc5b7dd5029f" (UID: "ba28e265-fab5-43ba-94b2-dc5b7dd5029f"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 01:34:40.091384 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:34:40.091359 4939 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ba28e265-fab5-43ba-94b2-dc5b7dd5029f-istiod-ca-cert\") on node \"ip-10-0-140-107.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:34:40.091494 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:34:40.091390 4939 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ba28e265-fab5-43ba-94b2-dc5b7dd5029f-credential-socket\") on node \"ip-10-0-140-107.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:34:40.091494 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:34:40.091407 4939 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ba28e265-fab5-43ba-94b2-dc5b7dd5029f-workload-socket\") on node \"ip-10-0-140-107.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:34:40.091494 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:34:40.091423 4939 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ba28e265-fab5-43ba-94b2-dc5b7dd5029f-istio-data\") on node \"ip-10-0-140-107.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:34:40.091494 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:34:40.091438 4939 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ba28e265-fab5-43ba-94b2-dc5b7dd5029f-workload-certs\") on node \"ip-10-0-140-107.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:34:40.105855 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:34:40.102805 4939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/ba28e265-fab5-43ba-94b2-dc5b7dd5029f-istio-podinfo" pod "ba28e265-fab5-43ba-94b2-dc5b7dd5029f" (UID: "ba28e265-fab5-43ba-94b2-dc5b7dd5029f"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 01:34:40.105855 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:34:40.105356 4939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ba28e265-fab5-43ba-94b2-dc5b7dd5029f-kube-api-access-xg7pd" pod "ba28e265-fab5-43ba-94b2-dc5b7dd5029f" (UID: "ba28e265-fab5-43ba-94b2-dc5b7dd5029f"). InnerVolumeSpecName "kube-api-access-xg7pd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 01:34:40.109654 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:34:40.109609 4939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ba28e265-fab5-43ba-94b2-dc5b7dd5029f-istio-envoy" pod "ba28e265-fab5-43ba-94b2-dc5b7dd5029f" (UID: "ba28e265-fab5-43ba-94b2-dc5b7dd5029f"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 01:34:40.109820 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:34:40.109705 4939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ba28e265-fab5-43ba-94b2-dc5b7dd5029f-istio-token" pod "ba28e265-fab5-43ba-94b2-dc5b7dd5029f" (UID: "ba28e265-fab5-43ba-94b2-dc5b7dd5029f"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 01:34:40.181248 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:34:40.181223 4939 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w4-6c0ec0-openshift-default-79c7584bfc-8hz7h" Nov 03 01:34:40.192722 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:34:40.192691 4939 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ba28e265-fab5-43ba-94b2-dc5b7dd5029f-istio-token\") on node \"ip-10-0-140-107.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:34:40.192872 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:34:40.192719 4939 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ba28e265-fab5-43ba-94b2-dc5b7dd5029f-istio-podinfo\") on node \"ip-10-0-140-107.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:34:40.192872 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:34:40.192738 4939 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xg7pd\" (UniqueName: \"kubernetes.io/projected/ba28e265-fab5-43ba-94b2-dc5b7dd5029f-kube-api-access-xg7pd\") on node \"ip-10-0-140-107.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:34:40.192872 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:34:40.192747 4939 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ba28e265-fab5-43ba-94b2-dc5b7dd5029f-istio-envoy\") on node \"ip-10-0-140-107.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:34:40.293424 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:34:40.293388 4939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0a1fe883-48d5-4d06-ab8f-41698965dbbb-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0a1fe883-48d5-4d06-ab8f-41698965dbbb-istio-data\") pod \"0a1fe883-48d5-4d06-ab8f-41698965dbbb\" (UID: \"0a1fe883-48d5-4d06-ab8f-41698965dbbb\") " Nov 03 01:34:40.293624 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:34:40.293437 4939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/0a1fe883-48d5-4d06-ab8f-41698965dbbb-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0a1fe883-48d5-4d06-ab8f-41698965dbbb-istio-podinfo\") pod \"0a1fe883-48d5-4d06-ab8f-41698965dbbb\" (UID: \"0a1fe883-48d5-4d06-ab8f-41698965dbbb\") " Nov 03 01:34:40.293624 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:34:40.293491 4939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0a1fe883-48d5-4d06-ab8f-41698965dbbb-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0a1fe883-48d5-4d06-ab8f-41698965dbbb-workload-certs\") pod \"0a1fe883-48d5-4d06-ab8f-41698965dbbb\" (UID: \"0a1fe883-48d5-4d06-ab8f-41698965dbbb\") " Nov 03 01:34:40.293757 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:34:40.293650 4939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0a1fe883-48d5-4d06-ab8f-41698965dbbb-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0a1fe883-48d5-4d06-ab8f-41698965dbbb-credential-socket\") pod \"0a1fe883-48d5-4d06-ab8f-41698965dbbb\" (UID: \"0a1fe883-48d5-4d06-ab8f-41698965dbbb\") " Nov 03 01:34:40.293757 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:34:40.293694 4939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0a1fe883-48d5-4d06-ab8f-41698965dbbb-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0a1fe883-48d5-4d06-ab8f-41698965dbbb-workload-socket\") pod \"0a1fe883-48d5-4d06-ab8f-41698965dbbb\" (UID: \"0a1fe883-48d5-4d06-ab8f-41698965dbbb\") " Nov 03 01:34:40.293882 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:34:40.293791 4939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0a1fe883-48d5-4d06-ab8f-41698965dbbb-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0a1fe883-48d5-4d06-ab8f-41698965dbbb-istiod-ca-cert\") pod \"0a1fe883-48d5-4d06-ab8f-41698965dbbb\" (UID: \"0a1fe883-48d5-4d06-ab8f-41698965dbbb\") " Nov 03 01:34:40.293882 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:34:40.293849 4939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0a1fe883-48d5-4d06-ab8f-41698965dbbb-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0a1fe883-48d5-4d06-ab8f-41698965dbbb-istio-envoy\") pod \"0a1fe883-48d5-4d06-ab8f-41698965dbbb\" (UID: \"0a1fe883-48d5-4d06-ab8f-41698965dbbb\") " Nov 03 01:34:40.294085 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:34:40.293986 4939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0a1fe883-48d5-4d06-ab8f-41698965dbbb-workload-certs" pod "0a1fe883-48d5-4d06-ab8f-41698965dbbb" (UID: "0a1fe883-48d5-4d06-ab8f-41698965dbbb"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 01:34:40.294355 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:34:40.294169 4939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0a1fe883-48d5-4d06-ab8f-41698965dbbb-istio-token\" (UniqueName: \"kubernetes.io/projected/0a1fe883-48d5-4d06-ab8f-41698965dbbb-istio-token\") pod \"0a1fe883-48d5-4d06-ab8f-41698965dbbb\" (UID: \"0a1fe883-48d5-4d06-ab8f-41698965dbbb\") " Nov 03 01:34:40.294355 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:34:40.294278 4939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0a1fe883-48d5-4d06-ab8f-41698965dbbb-kube-api-access-2d64v\" (UniqueName: \"kubernetes.io/projected/0a1fe883-48d5-4d06-ab8f-41698965dbbb-kube-api-access-2d64v\") pod \"0a1fe883-48d5-4d06-ab8f-41698965dbbb\" (UID: \"0a1fe883-48d5-4d06-ab8f-41698965dbbb\") " Nov 03 01:34:40.294564 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:34:40.294362 4939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0a1fe883-48d5-4d06-ab8f-41698965dbbb-credential-socket" pod "0a1fe883-48d5-4d06-ab8f-41698965dbbb" (UID: "0a1fe883-48d5-4d06-ab8f-41698965dbbb"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 01:34:40.295232 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:34:40.295202 4939 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0a1fe883-48d5-4d06-ab8f-41698965dbbb-workload-certs\") on node \"ip-10-0-140-107.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:34:40.295404 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:34:40.295235 4939 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0a1fe883-48d5-4d06-ab8f-41698965dbbb-credential-socket\") on node \"ip-10-0-140-107.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:34:40.295537 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:34:40.295439 4939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0a1fe883-48d5-4d06-ab8f-41698965dbbb-istiod-ca-cert" pod "0a1fe883-48d5-4d06-ab8f-41698965dbbb" (UID: "0a1fe883-48d5-4d06-ab8f-41698965dbbb"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 01:34:40.295807 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:34:40.295782 4939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0a1fe883-48d5-4d06-ab8f-41698965dbbb-istio-data" pod "0a1fe883-48d5-4d06-ab8f-41698965dbbb" (UID: "0a1fe883-48d5-4d06-ab8f-41698965dbbb"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 01:34:40.295903 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:34:40.295842 4939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0a1fe883-48d5-4d06-ab8f-41698965dbbb-workload-socket" pod "0a1fe883-48d5-4d06-ab8f-41698965dbbb" (UID: "0a1fe883-48d5-4d06-ab8f-41698965dbbb"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 01:34:40.296332 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:34:40.296307 4939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/0a1fe883-48d5-4d06-ab8f-41698965dbbb-istio-podinfo" pod "0a1fe883-48d5-4d06-ab8f-41698965dbbb" (UID: "0a1fe883-48d5-4d06-ab8f-41698965dbbb"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 01:34:40.296798 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:34:40.296778 4939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0a1fe883-48d5-4d06-ab8f-41698965dbbb-istio-envoy" pod "0a1fe883-48d5-4d06-ab8f-41698965dbbb" (UID: "0a1fe883-48d5-4d06-ab8f-41698965dbbb"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 01:34:40.297710 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:34:40.297690 4939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0a1fe883-48d5-4d06-ab8f-41698965dbbb-kube-api-access-2d64v" pod "0a1fe883-48d5-4d06-ab8f-41698965dbbb" (UID: "0a1fe883-48d5-4d06-ab8f-41698965dbbb"). InnerVolumeSpecName "kube-api-access-2d64v". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 01:34:40.297780 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:34:40.297721 4939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0a1fe883-48d5-4d06-ab8f-41698965dbbb-istio-token" pod "0a1fe883-48d5-4d06-ab8f-41698965dbbb" (UID: "0a1fe883-48d5-4d06-ab8f-41698965dbbb"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 01:34:40.389556 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:34:40.389474 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:34:40.396886 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:34:40.396858 4939 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0a1fe883-48d5-4d06-ab8f-41698965dbbb-workload-socket\") on node \"ip-10-0-140-107.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:34:40.397071 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:34:40.396889 4939 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0a1fe883-48d5-4d06-ab8f-41698965dbbb-istiod-ca-cert\") on node \"ip-10-0-140-107.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:34:40.397071 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:34:40.396918 4939 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0a1fe883-48d5-4d06-ab8f-41698965dbbb-istio-envoy\") on node \"ip-10-0-140-107.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:34:40.397071 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:34:40.396930 4939 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0a1fe883-48d5-4d06-ab8f-41698965dbbb-istio-token\") on node \"ip-10-0-140-107.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:34:40.397071 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:34:40.396940 4939 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2d64v\" (UniqueName: \"kubernetes.io/projected/0a1fe883-48d5-4d06-ab8f-41698965dbbb-kube-api-access-2d64v\") on node \"ip-10-0-140-107.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:34:40.397071 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:34:40.396954 4939 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0a1fe883-48d5-4d06-ab8f-41698965dbbb-istio-data\") on node \"ip-10-0-140-107.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:34:40.397071 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:34:40.396967 4939 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0a1fe883-48d5-4d06-ab8f-41698965dbbb-istio-podinfo\") on node \"ip-10-0-140-107.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:34:40.962278 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:34:40.962245 4939 generic.go:356] "Generic (PLEG): container finished" podID="0a1fe883-48d5-4d06-ab8f-41698965dbbb" containerID="2bcb1634a74979a9c04d145a53f1fc437c24da2608a17bfee61d92aed28c52e0" exitCode=0 Nov 03 01:34:40.962742 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:34:40.962297 4939 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w4-6c0ec0-openshift-default-79c7584bfc-8hz7h" Nov 03 01:34:40.962742 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:34:40.962334 4939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w4-6c0ec0-openshift-default-79c7584bfc-8hz7h" event={"ID":"0a1fe883-48d5-4d06-ab8f-41698965dbbb","Type":"ContainerDied","Data":"2bcb1634a74979a9c04d145a53f1fc437c24da2608a17bfee61d92aed28c52e0"} Nov 03 01:34:40.962742 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:34:40.962386 4939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w4-6c0ec0-openshift-default-79c7584bfc-8hz7h" event={"ID":"0a1fe883-48d5-4d06-ab8f-41698965dbbb","Type":"ContainerDied","Data":"e0422b63d263392686cc9694e1ea6b5235b7da82aa00256ae8ec5792a898739f"} Nov 03 01:34:40.962742 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:34:40.962414 4939 scope.go:122] "RemoveContainer" containerID="2bcb1634a74979a9c04d145a53f1fc437c24da2608a17bfee61d92aed28c52e0" Nov 03 01:34:40.962742 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:34:40.962659 4939 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-83755a-openshift-default-c86f9565c-qlgb7" Nov 03 01:34:41.000532 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:34:40.999602 4939 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w4-6c0ec0-openshift-default-79c7584bfc-8hz7h"] Nov 03 01:34:41.005058 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:34:41.005035 4939 scope.go:122] "RemoveContainer" containerID="3a69478699977ff097dfe6b487c6a8b6fe1eb21904e33269e787e9a5b7b2dea6" Nov 03 01:34:41.005482 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:34:41.005460 4939 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w4-6c0ec0-openshift-default-79c7584bfc-8hz7h"] Nov 03 01:34:41.013550 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:34:41.013522 4939 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-83755a-openshift-default-c86f9565c-qlgb7"] Nov 03 01:34:41.019781 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:34:41.019757 4939 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-83755a-openshift-default-c86f9565c-qlgb7"] Nov 03 01:34:41.027147 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:34:41.027125 4939 scope.go:122] "RemoveContainer" containerID="2bcb1634a74979a9c04d145a53f1fc437c24da2608a17bfee61d92aed28c52e0" Nov 03 01:34:41.027534 ip-10-0-140-107 kubenswrapper[4939]: E1103 01:34:41.027496 4939 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2bcb1634a74979a9c04d145a53f1fc437c24da2608a17bfee61d92aed28c52e0\": container with ID starting with 2bcb1634a74979a9c04d145a53f1fc437c24da2608a17bfee61d92aed28c52e0 not found: ID does not exist" containerID="2bcb1634a74979a9c04d145a53f1fc437c24da2608a17bfee61d92aed28c52e0" Nov 03 01:34:41.027657 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:34:41.027533 4939 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2bcb1634a74979a9c04d145a53f1fc437c24da2608a17bfee61d92aed28c52e0"} err="failed to get container status \"2bcb1634a74979a9c04d145a53f1fc437c24da2608a17bfee61d92aed28c52e0\": rpc error: code = NotFound desc = could not find container \"2bcb1634a74979a9c04d145a53f1fc437c24da2608a17bfee61d92aed28c52e0\": container with ID starting with 2bcb1634a74979a9c04d145a53f1fc437c24da2608a17bfee61d92aed28c52e0 not found: ID does not exist" Nov 03 01:34:41.027657 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:34:41.027561 4939 scope.go:122] "RemoveContainer" containerID="3a69478699977ff097dfe6b487c6a8b6fe1eb21904e33269e787e9a5b7b2dea6" Nov 03 01:34:41.027877 ip-10-0-140-107 kubenswrapper[4939]: E1103 01:34:41.027839 4939 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3a69478699977ff097dfe6b487c6a8b6fe1eb21904e33269e787e9a5b7b2dea6\": container with ID starting with 3a69478699977ff097dfe6b487c6a8b6fe1eb21904e33269e787e9a5b7b2dea6 not found: ID does not exist" containerID="3a69478699977ff097dfe6b487c6a8b6fe1eb21904e33269e787e9a5b7b2dea6" Nov 03 01:34:41.027877 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:34:41.027862 4939 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3a69478699977ff097dfe6b487c6a8b6fe1eb21904e33269e787e9a5b7b2dea6"} err="failed to get container status \"3a69478699977ff097dfe6b487c6a8b6fe1eb21904e33269e787e9a5b7b2dea6\": rpc error: code = NotFound desc = could not find container \"3a69478699977ff097dfe6b487c6a8b6fe1eb21904e33269e787e9a5b7b2dea6\": container with ID starting with 3a69478699977ff097dfe6b487c6a8b6fe1eb21904e33269e787e9a5b7b2dea6 not found: ID does not exist" Nov 03 01:34:41.081704 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:34:41.076041 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:34:41.243880 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:34:41.243856 4939 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w0-50e623-openshift-default-6c86dc794f-jmfmb" Nov 03 01:34:41.319348 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:34:41.319312 4939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/aba9a06e-b5d6-491f-b9a3-34ebd927d4aa-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/aba9a06e-b5d6-491f-b9a3-34ebd927d4aa-credential-socket\") pod \"aba9a06e-b5d6-491f-b9a3-34ebd927d4aa\" (UID: \"aba9a06e-b5d6-491f-b9a3-34ebd927d4aa\") " Nov 03 01:34:41.319556 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:34:41.319375 4939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/aba9a06e-b5d6-491f-b9a3-34ebd927d4aa-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/aba9a06e-b5d6-491f-b9a3-34ebd927d4aa-istio-podinfo\") pod \"aba9a06e-b5d6-491f-b9a3-34ebd927d4aa\" (UID: \"aba9a06e-b5d6-491f-b9a3-34ebd927d4aa\") " Nov 03 01:34:41.319556 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:34:41.319494 4939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/aba9a06e-b5d6-491f-b9a3-34ebd927d4aa-istio-token\" (UniqueName: \"kubernetes.io/projected/aba9a06e-b5d6-491f-b9a3-34ebd927d4aa-istio-token\") pod \"aba9a06e-b5d6-491f-b9a3-34ebd927d4aa\" (UID: \"aba9a06e-b5d6-491f-b9a3-34ebd927d4aa\") " Nov 03 01:34:41.319701 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:34:41.319558 4939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/aba9a06e-b5d6-491f-b9a3-34ebd927d4aa-kube-api-access-j9s9s\" (UniqueName: \"kubernetes.io/projected/aba9a06e-b5d6-491f-b9a3-34ebd927d4aa-kube-api-access-j9s9s\") pod \"aba9a06e-b5d6-491f-b9a3-34ebd927d4aa\" (UID: \"aba9a06e-b5d6-491f-b9a3-34ebd927d4aa\") " Nov 03 01:34:41.319701 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:34:41.319632 4939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/aba9a06e-b5d6-491f-b9a3-34ebd927d4aa-credential-socket" pod "aba9a06e-b5d6-491f-b9a3-34ebd927d4aa" (UID: "aba9a06e-b5d6-491f-b9a3-34ebd927d4aa"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 01:34:41.319821 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:34:41.319695 4939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/aba9a06e-b5d6-491f-b9a3-34ebd927d4aa-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/aba9a06e-b5d6-491f-b9a3-34ebd927d4aa-workload-socket\") pod \"aba9a06e-b5d6-491f-b9a3-34ebd927d4aa\" (UID: \"aba9a06e-b5d6-491f-b9a3-34ebd927d4aa\") " Nov 03 01:34:41.319821 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:34:41.319735 4939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/aba9a06e-b5d6-491f-b9a3-34ebd927d4aa-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/aba9a06e-b5d6-491f-b9a3-34ebd927d4aa-istio-data\") pod \"aba9a06e-b5d6-491f-b9a3-34ebd927d4aa\" (UID: \"aba9a06e-b5d6-491f-b9a3-34ebd927d4aa\") " Nov 03 01:34:41.320057 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:34:41.320029 4939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/aba9a06e-b5d6-491f-b9a3-34ebd927d4aa-workload-socket" pod "aba9a06e-b5d6-491f-b9a3-34ebd927d4aa" (UID: "aba9a06e-b5d6-491f-b9a3-34ebd927d4aa"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 01:34:41.320161 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:34:41.320075 4939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/aba9a06e-b5d6-491f-b9a3-34ebd927d4aa-istio-data" pod "aba9a06e-b5d6-491f-b9a3-34ebd927d4aa" (UID: "aba9a06e-b5d6-491f-b9a3-34ebd927d4aa"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 01:34:41.320161 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:34:41.320113 4939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/aba9a06e-b5d6-491f-b9a3-34ebd927d4aa-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/aba9a06e-b5d6-491f-b9a3-34ebd927d4aa-istiod-ca-cert\") pod \"aba9a06e-b5d6-491f-b9a3-34ebd927d4aa\" (UID: \"aba9a06e-b5d6-491f-b9a3-34ebd927d4aa\") " Nov 03 01:34:41.320285 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:34:41.320191 4939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/aba9a06e-b5d6-491f-b9a3-34ebd927d4aa-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/aba9a06e-b5d6-491f-b9a3-34ebd927d4aa-workload-certs\") pod \"aba9a06e-b5d6-491f-b9a3-34ebd927d4aa\" (UID: \"aba9a06e-b5d6-491f-b9a3-34ebd927d4aa\") " Nov 03 01:34:41.320380 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:34:41.320298 4939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/aba9a06e-b5d6-491f-b9a3-34ebd927d4aa-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/aba9a06e-b5d6-491f-b9a3-34ebd927d4aa-istio-envoy\") pod \"aba9a06e-b5d6-491f-b9a3-34ebd927d4aa\" (UID: \"aba9a06e-b5d6-491f-b9a3-34ebd927d4aa\") " Nov 03 01:34:41.320475 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:34:41.320429 4939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/aba9a06e-b5d6-491f-b9a3-34ebd927d4aa-workload-certs" pod "aba9a06e-b5d6-491f-b9a3-34ebd927d4aa" (UID: "aba9a06e-b5d6-491f-b9a3-34ebd927d4aa"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 01:34:41.320848 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:34:41.320822 4939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/aba9a06e-b5d6-491f-b9a3-34ebd927d4aa-istiod-ca-cert" pod "aba9a06e-b5d6-491f-b9a3-34ebd927d4aa" (UID: "aba9a06e-b5d6-491f-b9a3-34ebd927d4aa"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 01:34:41.321325 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:34:41.321297 4939 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/aba9a06e-b5d6-491f-b9a3-34ebd927d4aa-istiod-ca-cert\") on node \"ip-10-0-140-107.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:34:41.321427 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:34:41.321337 4939 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/aba9a06e-b5d6-491f-b9a3-34ebd927d4aa-workload-certs\") on node \"ip-10-0-140-107.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:34:41.321427 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:34:41.321353 4939 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/aba9a06e-b5d6-491f-b9a3-34ebd927d4aa-credential-socket\") on node \"ip-10-0-140-107.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:34:41.321427 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:34:41.321367 4939 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/aba9a06e-b5d6-491f-b9a3-34ebd927d4aa-workload-socket\") on node \"ip-10-0-140-107.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:34:41.321427 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:34:41.321382 4939 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/aba9a06e-b5d6-491f-b9a3-34ebd927d4aa-istio-data\") on node \"ip-10-0-140-107.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:34:41.322643 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:34:41.322496 4939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/aba9a06e-b5d6-491f-b9a3-34ebd927d4aa-istio-podinfo" pod "aba9a06e-b5d6-491f-b9a3-34ebd927d4aa" (UID: "aba9a06e-b5d6-491f-b9a3-34ebd927d4aa"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 01:34:41.322759 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:34:41.322640 4939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/aba9a06e-b5d6-491f-b9a3-34ebd927d4aa-kube-api-access-j9s9s" pod "aba9a06e-b5d6-491f-b9a3-34ebd927d4aa" (UID: "aba9a06e-b5d6-491f-b9a3-34ebd927d4aa"). InnerVolumeSpecName "kube-api-access-j9s9s". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 01:34:41.322759 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:34:41.322738 4939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/aba9a06e-b5d6-491f-b9a3-34ebd927d4aa-istio-token" pod "aba9a06e-b5d6-491f-b9a3-34ebd927d4aa" (UID: "aba9a06e-b5d6-491f-b9a3-34ebd927d4aa"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 01:34:41.323019 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:34:41.322992 4939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/aba9a06e-b5d6-491f-b9a3-34ebd927d4aa-istio-envoy" pod "aba9a06e-b5d6-491f-b9a3-34ebd927d4aa" (UID: "aba9a06e-b5d6-491f-b9a3-34ebd927d4aa"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 01:34:41.422783 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:34:41.422747 4939 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/aba9a06e-b5d6-491f-b9a3-34ebd927d4aa-istio-podinfo\") on node \"ip-10-0-140-107.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:34:41.422783 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:34:41.422782 4939 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/aba9a06e-b5d6-491f-b9a3-34ebd927d4aa-istio-token\") on node \"ip-10-0-140-107.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:34:41.423037 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:34:41.422793 4939 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-j9s9s\" (UniqueName: \"kubernetes.io/projected/aba9a06e-b5d6-491f-b9a3-34ebd927d4aa-kube-api-access-j9s9s\") on node \"ip-10-0-140-107.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:34:41.423037 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:34:41.422803 4939 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/aba9a06e-b5d6-491f-b9a3-34ebd927d4aa-istio-envoy\") on node \"ip-10-0-140-107.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:34:41.872804 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:34:41.872770 4939 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0a1fe883-48d5-4d06-ab8f-41698965dbbb" path="/var/lib/kubelet/pods/0a1fe883-48d5-4d06-ab8f-41698965dbbb/volumes" Nov 03 01:34:41.873438 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:34:41.873416 4939 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ba28e265-fab5-43ba-94b2-dc5b7dd5029f" path="/var/lib/kubelet/pods/ba28e265-fab5-43ba-94b2-dc5b7dd5029f/volumes" Nov 03 01:34:41.875574 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:34:41.875545 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:34:41.974699 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:34:41.974665 4939 generic.go:356] "Generic (PLEG): container finished" podID="aba9a06e-b5d6-491f-b9a3-34ebd927d4aa" containerID="368f1834e8a5c7ca07bdaa894b6b966b769f839a8b95940ef3da8a33098f6079" exitCode=0 Nov 03 01:34:41.975203 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:34:41.974729 4939 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w0-50e623-openshift-default-6c86dc794f-jmfmb" Nov 03 01:34:41.975203 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:34:41.974745 4939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w0-50e623-openshift-default-6c86dc794f-jmfmb" event={"ID":"aba9a06e-b5d6-491f-b9a3-34ebd927d4aa","Type":"ContainerDied","Data":"368f1834e8a5c7ca07bdaa894b6b966b769f839a8b95940ef3da8a33098f6079"} Nov 03 01:34:41.975203 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:34:41.974791 4939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w0-50e623-openshift-default-6c86dc794f-jmfmb" event={"ID":"aba9a06e-b5d6-491f-b9a3-34ebd927d4aa","Type":"ContainerDied","Data":"376548ea338ce5a24fe4ae8fee1614dcce70f74d3d149e1268033a9eea4e0fcb"} Nov 03 01:34:41.975203 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:34:41.974812 4939 scope.go:122] "RemoveContainer" containerID="368f1834e8a5c7ca07bdaa894b6b966b769f839a8b95940ef3da8a33098f6079" Nov 03 01:34:41.986900 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:34:41.986854 4939 scope.go:122] "RemoveContainer" containerID="368f1834e8a5c7ca07bdaa894b6b966b769f839a8b95940ef3da8a33098f6079" Nov 03 01:34:41.987225 ip-10-0-140-107 kubenswrapper[4939]: E1103 01:34:41.987203 4939 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"368f1834e8a5c7ca07bdaa894b6b966b769f839a8b95940ef3da8a33098f6079\": container with ID starting with 368f1834e8a5c7ca07bdaa894b6b966b769f839a8b95940ef3da8a33098f6079 not found: ID does not exist" containerID="368f1834e8a5c7ca07bdaa894b6b966b769f839a8b95940ef3da8a33098f6079" Nov 03 01:34:41.987338 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:34:41.987233 4939 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"368f1834e8a5c7ca07bdaa894b6b966b769f839a8b95940ef3da8a33098f6079"} err="failed to get container status \"368f1834e8a5c7ca07bdaa894b6b966b769f839a8b95940ef3da8a33098f6079\": rpc error: code = NotFound desc = could not find container \"368f1834e8a5c7ca07bdaa894b6b966b769f839a8b95940ef3da8a33098f6079\": container with ID starting with 368f1834e8a5c7ca07bdaa894b6b966b769f839a8b95940ef3da8a33098f6079 not found: ID does not exist" Nov 03 01:34:41.999664 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:34:41.999639 4939 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w0-50e623-openshift-default-6c86dc794f-jmfmb"] Nov 03 01:34:42.004556 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:34:42.004535 4939 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w0-50e623-openshift-default-6c86dc794f-jmfmb"] Nov 03 01:34:42.503420 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:34:42.503388 4939 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-88bbf7-openshift-default-574f47fff7-k87gb"] Nov 03 01:34:42.503614 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:34:42.503591 4939 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-88bbf7-openshift-default-574f47fff7-k87gb" podUID="c7907eaa-df92-4d28-aa19-ba6f03cb54b1" containerName="istio-proxy" containerID="cri-o://dddb3be79f5aa8b4fa1455c37072e16e4a1b3338a22817f7840fb711772b6885" gracePeriod=30 Nov 03 01:34:42.595001 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:34:42.594964 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:34:43.186362 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:34:43.186323 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:34:43.871924 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:34:43.871885 4939 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="aba9a06e-b5d6-491f-b9a3-34ebd927d4aa" path="/var/lib/kubelet/pods/aba9a06e-b5d6-491f-b9a3-34ebd927d4aa/volumes" Nov 03 01:34:43.898030 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:34:43.888774 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:34:44.694356 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:34:44.694260 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:34:45.301826 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:34:45.301788 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:34:45.471120 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:34:45.471087 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:34:46.093524 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:34:46.093493 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:34:46.723188 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:34:46.723152 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:34:47.334436 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:34:47.334396 4939 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-b6afd6/test-model-b6afd6-kserve-86dc6b8b77-pl784"] Nov 03 01:34:47.391403 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:34:47.391372 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:34:47.505816 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:34:47.503778 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:34:47.662873 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:34:47.662842 4939 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-717217-openshift-default-578c69554c-5xzz5"] Nov 03 01:34:47.663101 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:34:47.663078 4939 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w3-717217-openshift-default-578c69554c-5xzz5" podUID="b22388ea-ae77-4ad1-bfff-9aeb20f5bbbd" containerName="istio-proxy" containerID="cri-o://9d81b88a637be26b8213faa245c11d4ed680ab79f1d5df2f4c6bf91fd13431e4" gracePeriod=30 Nov 03 01:34:47.872340 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:34:47.872269 4939 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-88bbf7-openshift-default-574f47fff7-k87gb" Nov 03 01:34:47.919387 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:34:47.919348 4939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c7907eaa-df92-4d28-aa19-ba6f03cb54b1-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c7907eaa-df92-4d28-aa19-ba6f03cb54b1-istiod-ca-cert\") pod \"c7907eaa-df92-4d28-aa19-ba6f03cb54b1\" (UID: \"c7907eaa-df92-4d28-aa19-ba6f03cb54b1\") " Nov 03 01:34:47.919593 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:34:47.919508 4939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c7907eaa-df92-4d28-aa19-ba6f03cb54b1-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c7907eaa-df92-4d28-aa19-ba6f03cb54b1-istio-data\") pod \"c7907eaa-df92-4d28-aa19-ba6f03cb54b1\" (UID: \"c7907eaa-df92-4d28-aa19-ba6f03cb54b1\") " Nov 03 01:34:47.919866 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:34:47.919810 4939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c7907eaa-df92-4d28-aa19-ba6f03cb54b1-istio-data" pod "c7907eaa-df92-4d28-aa19-ba6f03cb54b1" (UID: "c7907eaa-df92-4d28-aa19-ba6f03cb54b1"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 01:34:47.920059 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:34:47.919944 4939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c7907eaa-df92-4d28-aa19-ba6f03cb54b1-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c7907eaa-df92-4d28-aa19-ba6f03cb54b1-credential-socket\") pod \"c7907eaa-df92-4d28-aa19-ba6f03cb54b1\" (UID: \"c7907eaa-df92-4d28-aa19-ba6f03cb54b1\") " Nov 03 01:34:47.920059 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:34:47.920007 4939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c7907eaa-df92-4d28-aa19-ba6f03cb54b1-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c7907eaa-df92-4d28-aa19-ba6f03cb54b1-workload-socket\") pod \"c7907eaa-df92-4d28-aa19-ba6f03cb54b1\" (UID: \"c7907eaa-df92-4d28-aa19-ba6f03cb54b1\") " Nov 03 01:34:47.920545 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:34:47.920485 4939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c7907eaa-df92-4d28-aa19-ba6f03cb54b1-credential-socket" pod "c7907eaa-df92-4d28-aa19-ba6f03cb54b1" (UID: "c7907eaa-df92-4d28-aa19-ba6f03cb54b1"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 01:34:47.920770 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:34:47.920570 4939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c7907eaa-df92-4d28-aa19-ba6f03cb54b1-workload-socket" pod "c7907eaa-df92-4d28-aa19-ba6f03cb54b1" (UID: "c7907eaa-df92-4d28-aa19-ba6f03cb54b1"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 01:34:47.920770 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:34:47.920601 4939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c7907eaa-df92-4d28-aa19-ba6f03cb54b1-istiod-ca-cert" pod "c7907eaa-df92-4d28-aa19-ba6f03cb54b1" (UID: "c7907eaa-df92-4d28-aa19-ba6f03cb54b1"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 01:34:47.920987 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:34:47.920806 4939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c7907eaa-df92-4d28-aa19-ba6f03cb54b1-kube-api-access-r84h9\" (UniqueName: \"kubernetes.io/projected/c7907eaa-df92-4d28-aa19-ba6f03cb54b1-kube-api-access-r84h9\") pod \"c7907eaa-df92-4d28-aa19-ba6f03cb54b1\" (UID: \"c7907eaa-df92-4d28-aa19-ba6f03cb54b1\") " Nov 03 01:34:47.920987 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:34:47.920934 4939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c7907eaa-df92-4d28-aa19-ba6f03cb54b1-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c7907eaa-df92-4d28-aa19-ba6f03cb54b1-istio-envoy\") pod \"c7907eaa-df92-4d28-aa19-ba6f03cb54b1\" (UID: \"c7907eaa-df92-4d28-aa19-ba6f03cb54b1\") " Nov 03 01:34:47.921414 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:34:47.921378 4939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/c7907eaa-df92-4d28-aa19-ba6f03cb54b1-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c7907eaa-df92-4d28-aa19-ba6f03cb54b1-istio-podinfo\") pod \"c7907eaa-df92-4d28-aa19-ba6f03cb54b1\" (UID: \"c7907eaa-df92-4d28-aa19-ba6f03cb54b1\") " Nov 03 01:34:47.922126 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:34:47.922100 4939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c7907eaa-df92-4d28-aa19-ba6f03cb54b1-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c7907eaa-df92-4d28-aa19-ba6f03cb54b1-workload-certs\") pod \"c7907eaa-df92-4d28-aa19-ba6f03cb54b1\" (UID: \"c7907eaa-df92-4d28-aa19-ba6f03cb54b1\") " Nov 03 01:34:47.922235 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:34:47.922190 4939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c7907eaa-df92-4d28-aa19-ba6f03cb54b1-istio-token\" (UniqueName: \"kubernetes.io/projected/c7907eaa-df92-4d28-aa19-ba6f03cb54b1-istio-token\") pod \"c7907eaa-df92-4d28-aa19-ba6f03cb54b1\" (UID: \"c7907eaa-df92-4d28-aa19-ba6f03cb54b1\") " Nov 03 01:34:47.924050 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:34:47.924027 4939 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c7907eaa-df92-4d28-aa19-ba6f03cb54b1-istiod-ca-cert\") on node \"ip-10-0-140-107.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:34:47.924227 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:34:47.924057 4939 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c7907eaa-df92-4d28-aa19-ba6f03cb54b1-istio-data\") on node \"ip-10-0-140-107.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:34:47.924227 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:34:47.924073 4939 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c7907eaa-df92-4d28-aa19-ba6f03cb54b1-credential-socket\") on node \"ip-10-0-140-107.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:34:47.924227 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:34:47.924086 4939 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c7907eaa-df92-4d28-aa19-ba6f03cb54b1-workload-socket\") on node \"ip-10-0-140-107.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:34:47.924424 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:34:47.924265 4939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c7907eaa-df92-4d28-aa19-ba6f03cb54b1-workload-certs" pod "c7907eaa-df92-4d28-aa19-ba6f03cb54b1" (UID: "c7907eaa-df92-4d28-aa19-ba6f03cb54b1"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 01:34:47.927157 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:34:47.927123 4939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c7907eaa-df92-4d28-aa19-ba6f03cb54b1-kube-api-access-r84h9" pod "c7907eaa-df92-4d28-aa19-ba6f03cb54b1" (UID: "c7907eaa-df92-4d28-aa19-ba6f03cb54b1"). InnerVolumeSpecName "kube-api-access-r84h9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 01:34:47.927272 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:34:47.927242 4939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c7907eaa-df92-4d28-aa19-ba6f03cb54b1-istio-token" pod "c7907eaa-df92-4d28-aa19-ba6f03cb54b1" (UID: "c7907eaa-df92-4d28-aa19-ba6f03cb54b1"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 01:34:47.928175 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:34:47.928141 4939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/c7907eaa-df92-4d28-aa19-ba6f03cb54b1-istio-podinfo" pod "c7907eaa-df92-4d28-aa19-ba6f03cb54b1" (UID: "c7907eaa-df92-4d28-aa19-ba6f03cb54b1"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 01:34:47.928827 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:34:47.928792 4939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c7907eaa-df92-4d28-aa19-ba6f03cb54b1-istio-envoy" pod "c7907eaa-df92-4d28-aa19-ba6f03cb54b1" (UID: "c7907eaa-df92-4d28-aa19-ba6f03cb54b1"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 01:34:48.027089 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:34:48.027048 4939 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-r84h9\" (UniqueName: \"kubernetes.io/projected/c7907eaa-df92-4d28-aa19-ba6f03cb54b1-kube-api-access-r84h9\") on node \"ip-10-0-140-107.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:34:48.027089 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:34:48.027088 4939 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c7907eaa-df92-4d28-aa19-ba6f03cb54b1-istio-envoy\") on node \"ip-10-0-140-107.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:34:48.027399 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:34:48.027105 4939 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c7907eaa-df92-4d28-aa19-ba6f03cb54b1-istio-podinfo\") on node \"ip-10-0-140-107.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:34:48.027399 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:34:48.027120 4939 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c7907eaa-df92-4d28-aa19-ba6f03cb54b1-workload-certs\") on node \"ip-10-0-140-107.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:34:48.027399 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:34:48.027134 4939 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c7907eaa-df92-4d28-aa19-ba6f03cb54b1-istio-token\") on node \"ip-10-0-140-107.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:34:48.043963 ip-10-0-140-107 kubenswrapper[4939]: E1103 01:34:48.043892 4939 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-sthjh" containerName="extractor" Nov 03 01:34:48.089626 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:34:48.089589 4939 generic.go:356] "Generic (PLEG): container finished" podID="c7907eaa-df92-4d28-aa19-ba6f03cb54b1" containerID="dddb3be79f5aa8b4fa1455c37072e16e4a1b3338a22817f7840fb711772b6885" exitCode=0 Nov 03 01:34:48.089786 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:34:48.089653 4939 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-88bbf7-openshift-default-574f47fff7-k87gb" Nov 03 01:34:48.089786 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:34:48.089670 4939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-88bbf7-openshift-default-574f47fff7-k87gb" event={"ID":"c7907eaa-df92-4d28-aa19-ba6f03cb54b1","Type":"ContainerDied","Data":"dddb3be79f5aa8b4fa1455c37072e16e4a1b3338a22817f7840fb711772b6885"} Nov 03 01:34:48.089786 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:34:48.089724 4939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-88bbf7-openshift-default-574f47fff7-k87gb" event={"ID":"c7907eaa-df92-4d28-aa19-ba6f03cb54b1","Type":"ContainerDied","Data":"4f16624bb13bdc4cb203774ad42595d9e1a471f145e687306027ba3a15cb2a80"} Nov 03 01:34:48.089786 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:34:48.089750 4939 scope.go:122] "RemoveContainer" containerID="dddb3be79f5aa8b4fa1455c37072e16e4a1b3338a22817f7840fb711772b6885" Nov 03 01:34:48.102856 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:34:48.102837 4939 scope.go:122] "RemoveContainer" containerID="dddb3be79f5aa8b4fa1455c37072e16e4a1b3338a22817f7840fb711772b6885" Nov 03 01:34:48.103208 ip-10-0-140-107 kubenswrapper[4939]: E1103 01:34:48.103182 4939 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"dddb3be79f5aa8b4fa1455c37072e16e4a1b3338a22817f7840fb711772b6885\": container with ID starting with dddb3be79f5aa8b4fa1455c37072e16e4a1b3338a22817f7840fb711772b6885 not found: ID does not exist" containerID="dddb3be79f5aa8b4fa1455c37072e16e4a1b3338a22817f7840fb711772b6885" Nov 03 01:34:48.103284 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:34:48.103216 4939 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"dddb3be79f5aa8b4fa1455c37072e16e4a1b3338a22817f7840fb711772b6885"} err="failed to get container status \"dddb3be79f5aa8b4fa1455c37072e16e4a1b3338a22817f7840fb711772b6885\": rpc error: code = NotFound desc = could not find container \"dddb3be79f5aa8b4fa1455c37072e16e4a1b3338a22817f7840fb711772b6885\": container with ID starting with dddb3be79f5aa8b4fa1455c37072e16e4a1b3338a22817f7840fb711772b6885 not found: ID does not exist" Nov 03 01:34:48.123044 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:34:48.122978 4939 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-88bbf7-openshift-default-574f47fff7-k87gb"] Nov 03 01:34:48.128942 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:34:48.128900 4939 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-88bbf7-openshift-default-574f47fff7-k87gb"] Nov 03 01:34:48.283408 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:34:48.283372 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:34:49.185301 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:34:49.185270 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:34:49.879932 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:34:49.879674 4939 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c7907eaa-df92-4d28-aa19-ba6f03cb54b1" path="/var/lib/kubelet/pods/c7907eaa-df92-4d28-aa19-ba6f03cb54b1/volumes" Nov 03 01:34:50.015725 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:34:50.015691 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:34:50.993932 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:34:50.992418 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:34:51.333621 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:34:51.333584 4939 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-2f9a3a-openshift-default-7f6f74cf97-dxffl"] Nov 03 01:34:51.333893 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:34:51.333853 4939 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-2f9a3a-openshift-default-7f6f74cf97-dxffl" podUID="80e0f915-1a1a-4214-92f0-5eeb806be96e" containerName="istio-proxy" containerID="cri-o://b9db464c8699b13b1cdd17a9bb0329c298b15fccb73d53481ccb96658a2c707f" gracePeriod=30 Nov 03 01:34:51.936297 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:34:51.936263 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:34:52.781321 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:34:52.781292 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:34:53.033798 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:34:53.033770 4939 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w3-717217-openshift-default-578c69554c-5xzz5" Nov 03 01:34:53.044199 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:34:53.044169 4939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/b22388ea-ae77-4ad1-bfff-9aeb20f5bbbd-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b22388ea-ae77-4ad1-bfff-9aeb20f5bbbd-istio-podinfo\") pod \"b22388ea-ae77-4ad1-bfff-9aeb20f5bbbd\" (UID: \"b22388ea-ae77-4ad1-bfff-9aeb20f5bbbd\") " Nov 03 01:34:53.044351 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:34:53.044261 4939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b22388ea-ae77-4ad1-bfff-9aeb20f5bbbd-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b22388ea-ae77-4ad1-bfff-9aeb20f5bbbd-credential-socket\") pod \"b22388ea-ae77-4ad1-bfff-9aeb20f5bbbd\" (UID: \"b22388ea-ae77-4ad1-bfff-9aeb20f5bbbd\") " Nov 03 01:34:53.044351 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:34:53.044304 4939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b22388ea-ae77-4ad1-bfff-9aeb20f5bbbd-istio-token\" (UniqueName: \"kubernetes.io/projected/b22388ea-ae77-4ad1-bfff-9aeb20f5bbbd-istio-token\") pod \"b22388ea-ae77-4ad1-bfff-9aeb20f5bbbd\" (UID: \"b22388ea-ae77-4ad1-bfff-9aeb20f5bbbd\") " Nov 03 01:34:53.044479 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:34:53.044362 4939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b22388ea-ae77-4ad1-bfff-9aeb20f5bbbd-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b22388ea-ae77-4ad1-bfff-9aeb20f5bbbd-workload-certs\") pod \"b22388ea-ae77-4ad1-bfff-9aeb20f5bbbd\" (UID: \"b22388ea-ae77-4ad1-bfff-9aeb20f5bbbd\") " Nov 03 01:34:53.044545 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:34:53.044514 4939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b22388ea-ae77-4ad1-bfff-9aeb20f5bbbd-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b22388ea-ae77-4ad1-bfff-9aeb20f5bbbd-istio-data\") pod \"b22388ea-ae77-4ad1-bfff-9aeb20f5bbbd\" (UID: \"b22388ea-ae77-4ad1-bfff-9aeb20f5bbbd\") " Nov 03 01:34:53.044610 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:34:53.044590 4939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b22388ea-ae77-4ad1-bfff-9aeb20f5bbbd-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b22388ea-ae77-4ad1-bfff-9aeb20f5bbbd-istiod-ca-cert\") pod \"b22388ea-ae77-4ad1-bfff-9aeb20f5bbbd\" (UID: \"b22388ea-ae77-4ad1-bfff-9aeb20f5bbbd\") " Nov 03 01:34:53.044666 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:34:53.044597 4939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b22388ea-ae77-4ad1-bfff-9aeb20f5bbbd-credential-socket" pod "b22388ea-ae77-4ad1-bfff-9aeb20f5bbbd" (UID: "b22388ea-ae77-4ad1-bfff-9aeb20f5bbbd"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 01:34:53.044666 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:34:53.044648 4939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b22388ea-ae77-4ad1-bfff-9aeb20f5bbbd-kube-api-access-7s5lg\" (UniqueName: \"kubernetes.io/projected/b22388ea-ae77-4ad1-bfff-9aeb20f5bbbd-kube-api-access-7s5lg\") pod \"b22388ea-ae77-4ad1-bfff-9aeb20f5bbbd\" (UID: \"b22388ea-ae77-4ad1-bfff-9aeb20f5bbbd\") " Nov 03 01:34:53.044828 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:34:53.044701 4939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b22388ea-ae77-4ad1-bfff-9aeb20f5bbbd-workload-certs" pod "b22388ea-ae77-4ad1-bfff-9aeb20f5bbbd" (UID: "b22388ea-ae77-4ad1-bfff-9aeb20f5bbbd"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 01:34:53.044828 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:34:53.044725 4939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b22388ea-ae77-4ad1-bfff-9aeb20f5bbbd-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b22388ea-ae77-4ad1-bfff-9aeb20f5bbbd-workload-socket\") pod \"b22388ea-ae77-4ad1-bfff-9aeb20f5bbbd\" (UID: \"b22388ea-ae77-4ad1-bfff-9aeb20f5bbbd\") " Nov 03 01:34:53.044828 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:34:53.044795 4939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b22388ea-ae77-4ad1-bfff-9aeb20f5bbbd-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b22388ea-ae77-4ad1-bfff-9aeb20f5bbbd-istio-envoy\") pod \"b22388ea-ae77-4ad1-bfff-9aeb20f5bbbd\" (UID: \"b22388ea-ae77-4ad1-bfff-9aeb20f5bbbd\") " Nov 03 01:34:53.045054 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:34:53.044836 4939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b22388ea-ae77-4ad1-bfff-9aeb20f5bbbd-istio-data" pod "b22388ea-ae77-4ad1-bfff-9aeb20f5bbbd" (UID: "b22388ea-ae77-4ad1-bfff-9aeb20f5bbbd"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 01:34:53.045113 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:34:53.044988 4939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b22388ea-ae77-4ad1-bfff-9aeb20f5bbbd-istiod-ca-cert" pod "b22388ea-ae77-4ad1-bfff-9aeb20f5bbbd" (UID: "b22388ea-ae77-4ad1-bfff-9aeb20f5bbbd"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 01:34:53.045113 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:34:53.045091 4939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b22388ea-ae77-4ad1-bfff-9aeb20f5bbbd-workload-socket" pod "b22388ea-ae77-4ad1-bfff-9aeb20f5bbbd" (UID: "b22388ea-ae77-4ad1-bfff-9aeb20f5bbbd"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 01:34:53.045569 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:34:53.045547 4939 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b22388ea-ae77-4ad1-bfff-9aeb20f5bbbd-istio-data\") on node \"ip-10-0-140-107.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:34:53.045664 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:34:53.045572 4939 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b22388ea-ae77-4ad1-bfff-9aeb20f5bbbd-istiod-ca-cert\") on node \"ip-10-0-140-107.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:34:53.045664 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:34:53.045588 4939 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b22388ea-ae77-4ad1-bfff-9aeb20f5bbbd-workload-socket\") on node \"ip-10-0-140-107.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:34:53.045664 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:34:53.045600 4939 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b22388ea-ae77-4ad1-bfff-9aeb20f5bbbd-credential-socket\") on node \"ip-10-0-140-107.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:34:53.045664 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:34:53.045614 4939 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b22388ea-ae77-4ad1-bfff-9aeb20f5bbbd-workload-certs\") on node \"ip-10-0-140-107.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:34:53.046727 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:34:53.046700 4939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/b22388ea-ae77-4ad1-bfff-9aeb20f5bbbd-istio-podinfo" pod "b22388ea-ae77-4ad1-bfff-9aeb20f5bbbd" (UID: "b22388ea-ae77-4ad1-bfff-9aeb20f5bbbd"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 01:34:53.046918 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:34:53.046885 4939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b22388ea-ae77-4ad1-bfff-9aeb20f5bbbd-istio-envoy" pod "b22388ea-ae77-4ad1-bfff-9aeb20f5bbbd" (UID: "b22388ea-ae77-4ad1-bfff-9aeb20f5bbbd"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 01:34:53.047009 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:34:53.046959 4939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b22388ea-ae77-4ad1-bfff-9aeb20f5bbbd-kube-api-access-7s5lg" pod "b22388ea-ae77-4ad1-bfff-9aeb20f5bbbd" (UID: "b22388ea-ae77-4ad1-bfff-9aeb20f5bbbd"). InnerVolumeSpecName "kube-api-access-7s5lg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 01:34:53.047218 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:34:53.047201 4939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b22388ea-ae77-4ad1-bfff-9aeb20f5bbbd-istio-token" pod "b22388ea-ae77-4ad1-bfff-9aeb20f5bbbd" (UID: "b22388ea-ae77-4ad1-bfff-9aeb20f5bbbd"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 01:34:53.147696 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:34:53.147661 4939 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b22388ea-ae77-4ad1-bfff-9aeb20f5bbbd-istio-token\") on node \"ip-10-0-140-107.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:34:53.147696 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:34:53.147695 4939 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7s5lg\" (UniqueName: \"kubernetes.io/projected/b22388ea-ae77-4ad1-bfff-9aeb20f5bbbd-kube-api-access-7s5lg\") on node \"ip-10-0-140-107.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:34:53.147896 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:34:53.147707 4939 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b22388ea-ae77-4ad1-bfff-9aeb20f5bbbd-istio-envoy\") on node \"ip-10-0-140-107.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:34:53.147896 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:34:53.147717 4939 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b22388ea-ae77-4ad1-bfff-9aeb20f5bbbd-istio-podinfo\") on node \"ip-10-0-140-107.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:34:53.300645 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:34:53.300563 4939 generic.go:356] "Generic (PLEG): container finished" podID="b22388ea-ae77-4ad1-bfff-9aeb20f5bbbd" containerID="9d81b88a637be26b8213faa245c11d4ed680ab79f1d5df2f4c6bf91fd13431e4" exitCode=0 Nov 03 01:34:53.300645 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:34:53.300613 4939 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w3-717217-openshift-default-578c69554c-5xzz5" Nov 03 01:34:53.300836 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:34:53.300641 4939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-717217-openshift-default-578c69554c-5xzz5" event={"ID":"b22388ea-ae77-4ad1-bfff-9aeb20f5bbbd","Type":"ContainerDied","Data":"9d81b88a637be26b8213faa245c11d4ed680ab79f1d5df2f4c6bf91fd13431e4"} Nov 03 01:34:53.300836 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:34:53.300732 4939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-717217-openshift-default-578c69554c-5xzz5" event={"ID":"b22388ea-ae77-4ad1-bfff-9aeb20f5bbbd","Type":"ContainerDied","Data":"83349dd90858a6925bc45287148fbad42fa1e78be863611d89590a90d6ad42dc"} Nov 03 01:34:53.300836 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:34:53.300764 4939 scope.go:122] "RemoveContainer" containerID="9d81b88a637be26b8213faa245c11d4ed680ab79f1d5df2f4c6bf91fd13431e4" Nov 03 01:34:53.313354 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:34:53.313334 4939 scope.go:122] "RemoveContainer" containerID="9d81b88a637be26b8213faa245c11d4ed680ab79f1d5df2f4c6bf91fd13431e4" Nov 03 01:34:53.313636 ip-10-0-140-107 kubenswrapper[4939]: E1103 01:34:53.313610 4939 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9d81b88a637be26b8213faa245c11d4ed680ab79f1d5df2f4c6bf91fd13431e4\": container with ID starting with 9d81b88a637be26b8213faa245c11d4ed680ab79f1d5df2f4c6bf91fd13431e4 not found: ID does not exist" containerID="9d81b88a637be26b8213faa245c11d4ed680ab79f1d5df2f4c6bf91fd13431e4" Nov 03 01:34:53.313796 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:34:53.313640 4939 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9d81b88a637be26b8213faa245c11d4ed680ab79f1d5df2f4c6bf91fd13431e4"} err="failed to get container status \"9d81b88a637be26b8213faa245c11d4ed680ab79f1d5df2f4c6bf91fd13431e4\": rpc error: code = NotFound desc = could not find container \"9d81b88a637be26b8213faa245c11d4ed680ab79f1d5df2f4c6bf91fd13431e4\": container with ID starting with 9d81b88a637be26b8213faa245c11d4ed680ab79f1d5df2f4c6bf91fd13431e4 not found: ID does not exist" Nov 03 01:34:53.329486 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:34:53.329455 4939 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-717217-openshift-default-578c69554c-5xzz5"] Nov 03 01:34:53.333691 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:34:53.333665 4939 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-717217-openshift-default-578c69554c-5xzz5"] Nov 03 01:34:53.507961 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:34:53.507888 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:34:53.875079 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:34:53.875044 4939 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b22388ea-ae77-4ad1-bfff-9aeb20f5bbbd" path="/var/lib/kubelet/pods/b22388ea-ae77-4ad1-bfff-9aeb20f5bbbd/volumes" Nov 03 01:34:54.193153 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:34:54.193073 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:34:54.900153 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:34:54.900119 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:34:55.599531 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:34:55.599497 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:34:56.202918 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:34:56.202886 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:34:56.650489 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:34:56.650449 4939 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-2f9a3a-openshift-default-7f6f74cf97-dxffl" Nov 03 01:34:56.705803 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:34:56.705720 4939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/80e0f915-1a1a-4214-92f0-5eeb806be96e-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/80e0f915-1a1a-4214-92f0-5eeb806be96e-workload-certs\") pod \"80e0f915-1a1a-4214-92f0-5eeb806be96e\" (UID: \"80e0f915-1a1a-4214-92f0-5eeb806be96e\") " Nov 03 01:34:56.705803 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:34:56.705788 4939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/80e0f915-1a1a-4214-92f0-5eeb806be96e-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/80e0f915-1a1a-4214-92f0-5eeb806be96e-credential-socket\") pod \"80e0f915-1a1a-4214-92f0-5eeb806be96e\" (UID: \"80e0f915-1a1a-4214-92f0-5eeb806be96e\") " Nov 03 01:34:56.706068 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:34:56.705819 4939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/80e0f915-1a1a-4214-92f0-5eeb806be96e-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/80e0f915-1a1a-4214-92f0-5eeb806be96e-workload-socket\") pod \"80e0f915-1a1a-4214-92f0-5eeb806be96e\" (UID: \"80e0f915-1a1a-4214-92f0-5eeb806be96e\") " Nov 03 01:34:56.706068 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:34:56.705869 4939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/80e0f915-1a1a-4214-92f0-5eeb806be96e-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/80e0f915-1a1a-4214-92f0-5eeb806be96e-istiod-ca-cert\") pod \"80e0f915-1a1a-4214-92f0-5eeb806be96e\" (UID: \"80e0f915-1a1a-4214-92f0-5eeb806be96e\") " Nov 03 01:34:56.706068 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:34:56.705984 4939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/80e0f915-1a1a-4214-92f0-5eeb806be96e-workload-certs" pod "80e0f915-1a1a-4214-92f0-5eeb806be96e" (UID: "80e0f915-1a1a-4214-92f0-5eeb806be96e"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 01:34:56.706263 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:34:56.706114 4939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/80e0f915-1a1a-4214-92f0-5eeb806be96e-credential-socket" pod "80e0f915-1a1a-4214-92f0-5eeb806be96e" (UID: "80e0f915-1a1a-4214-92f0-5eeb806be96e"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 01:34:56.706263 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:34:56.706180 4939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/80e0f915-1a1a-4214-92f0-5eeb806be96e-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/80e0f915-1a1a-4214-92f0-5eeb806be96e-istio-data\") pod \"80e0f915-1a1a-4214-92f0-5eeb806be96e\" (UID: \"80e0f915-1a1a-4214-92f0-5eeb806be96e\") " Nov 03 01:34:56.706263 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:34:56.706220 4939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/80e0f915-1a1a-4214-92f0-5eeb806be96e-istio-token\" (UniqueName: \"kubernetes.io/projected/80e0f915-1a1a-4214-92f0-5eeb806be96e-istio-token\") pod \"80e0f915-1a1a-4214-92f0-5eeb806be96e\" (UID: \"80e0f915-1a1a-4214-92f0-5eeb806be96e\") " Nov 03 01:34:56.706263 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:34:56.706242 4939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/80e0f915-1a1a-4214-92f0-5eeb806be96e-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/80e0f915-1a1a-4214-92f0-5eeb806be96e-istio-podinfo\") pod \"80e0f915-1a1a-4214-92f0-5eeb806be96e\" (UID: \"80e0f915-1a1a-4214-92f0-5eeb806be96e\") " Nov 03 01:34:56.706478 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:34:56.706328 4939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/80e0f915-1a1a-4214-92f0-5eeb806be96e-istiod-ca-cert" pod "80e0f915-1a1a-4214-92f0-5eeb806be96e" (UID: "80e0f915-1a1a-4214-92f0-5eeb806be96e"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 01:34:56.706478 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:34:56.706347 4939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/80e0f915-1a1a-4214-92f0-5eeb806be96e-kube-api-access-2qzls\" (UniqueName: \"kubernetes.io/projected/80e0f915-1a1a-4214-92f0-5eeb806be96e-kube-api-access-2qzls\") pod \"80e0f915-1a1a-4214-92f0-5eeb806be96e\" (UID: \"80e0f915-1a1a-4214-92f0-5eeb806be96e\") " Nov 03 01:34:56.706604 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:34:56.706485 4939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/80e0f915-1a1a-4214-92f0-5eeb806be96e-istio-data" pod "80e0f915-1a1a-4214-92f0-5eeb806be96e" (UID: "80e0f915-1a1a-4214-92f0-5eeb806be96e"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 01:34:56.706604 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:34:56.706499 4939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/80e0f915-1a1a-4214-92f0-5eeb806be96e-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/80e0f915-1a1a-4214-92f0-5eeb806be96e-istio-envoy\") pod \"80e0f915-1a1a-4214-92f0-5eeb806be96e\" (UID: \"80e0f915-1a1a-4214-92f0-5eeb806be96e\") " Nov 03 01:34:56.706979 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:34:56.706952 4939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/80e0f915-1a1a-4214-92f0-5eeb806be96e-workload-socket" pod "80e0f915-1a1a-4214-92f0-5eeb806be96e" (UID: "80e0f915-1a1a-4214-92f0-5eeb806be96e"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 01:34:56.707819 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:34:56.707795 4939 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/80e0f915-1a1a-4214-92f0-5eeb806be96e-istio-data\") on node \"ip-10-0-140-107.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:34:56.708002 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:34:56.707982 4939 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/80e0f915-1a1a-4214-92f0-5eeb806be96e-workload-certs\") on node \"ip-10-0-140-107.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:34:56.708132 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:34:56.708109 4939 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/80e0f915-1a1a-4214-92f0-5eeb806be96e-credential-socket\") on node \"ip-10-0-140-107.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:34:56.708252 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:34:56.708134 4939 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/80e0f915-1a1a-4214-92f0-5eeb806be96e-workload-socket\") on node \"ip-10-0-140-107.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:34:56.708252 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:34:56.708150 4939 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/80e0f915-1a1a-4214-92f0-5eeb806be96e-istiod-ca-cert\") on node \"ip-10-0-140-107.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:34:56.708767 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:34:56.708742 4939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/80e0f915-1a1a-4214-92f0-5eeb806be96e-istio-token" pod "80e0f915-1a1a-4214-92f0-5eeb806be96e" (UID: "80e0f915-1a1a-4214-92f0-5eeb806be96e"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 01:34:56.709063 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:34:56.709037 4939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/80e0f915-1a1a-4214-92f0-5eeb806be96e-istio-envoy" pod "80e0f915-1a1a-4214-92f0-5eeb806be96e" (UID: "80e0f915-1a1a-4214-92f0-5eeb806be96e"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 01:34:56.710067 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:34:56.710031 4939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/80e0f915-1a1a-4214-92f0-5eeb806be96e-istio-podinfo" pod "80e0f915-1a1a-4214-92f0-5eeb806be96e" (UID: "80e0f915-1a1a-4214-92f0-5eeb806be96e"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 01:34:56.710486 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:34:56.710458 4939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/80e0f915-1a1a-4214-92f0-5eeb806be96e-kube-api-access-2qzls" pod "80e0f915-1a1a-4214-92f0-5eeb806be96e" (UID: "80e0f915-1a1a-4214-92f0-5eeb806be96e"). InnerVolumeSpecName "kube-api-access-2qzls". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 01:34:56.810128 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:34:56.810087 4939 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/80e0f915-1a1a-4214-92f0-5eeb806be96e-istio-envoy\") on node \"ip-10-0-140-107.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:34:56.810128 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:34:56.810122 4939 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/80e0f915-1a1a-4214-92f0-5eeb806be96e-istio-token\") on node \"ip-10-0-140-107.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:34:56.810128 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:34:56.810132 4939 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/80e0f915-1a1a-4214-92f0-5eeb806be96e-istio-podinfo\") on node \"ip-10-0-140-107.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:34:56.810426 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:34:56.810142 4939 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2qzls\" (UniqueName: \"kubernetes.io/projected/80e0f915-1a1a-4214-92f0-5eeb806be96e-kube-api-access-2qzls\") on node \"ip-10-0-140-107.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:34:57.072807 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:34:57.072711 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:34:57.339087 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:34:57.339052 4939 generic.go:356] "Generic (PLEG): container finished" podID="80e0f915-1a1a-4214-92f0-5eeb806be96e" containerID="b9db464c8699b13b1cdd17a9bb0329c298b15fccb73d53481ccb96658a2c707f" exitCode=0 Nov 03 01:34:57.339523 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:34:57.339101 4939 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-2f9a3a-openshift-default-7f6f74cf97-dxffl" Nov 03 01:34:57.339523 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:34:57.339134 4939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-2f9a3a-openshift-default-7f6f74cf97-dxffl" event={"ID":"80e0f915-1a1a-4214-92f0-5eeb806be96e","Type":"ContainerDied","Data":"b9db464c8699b13b1cdd17a9bb0329c298b15fccb73d53481ccb96658a2c707f"} Nov 03 01:34:57.339523 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:34:57.339187 4939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-2f9a3a-openshift-default-7f6f74cf97-dxffl" event={"ID":"80e0f915-1a1a-4214-92f0-5eeb806be96e","Type":"ContainerDied","Data":"af8a98c9f3db734673618a17fc8ef34b517999f2ed9dff5a0c6fc75ff1503d43"} Nov 03 01:34:57.339523 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:34:57.339209 4939 scope.go:122] "RemoveContainer" containerID="b9db464c8699b13b1cdd17a9bb0329c298b15fccb73d53481ccb96658a2c707f" Nov 03 01:34:57.352621 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:34:57.352606 4939 scope.go:122] "RemoveContainer" containerID="daffb5100933e1f21d8c999ff6a9c75480633c4dfc1f5014ebff8b4f61f3e10a" Nov 03 01:34:57.362964 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:34:57.362945 4939 scope.go:122] "RemoveContainer" containerID="b9db464c8699b13b1cdd17a9bb0329c298b15fccb73d53481ccb96658a2c707f" Nov 03 01:34:57.363322 ip-10-0-140-107 kubenswrapper[4939]: E1103 01:34:57.363301 4939 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b9db464c8699b13b1cdd17a9bb0329c298b15fccb73d53481ccb96658a2c707f\": container with ID starting with b9db464c8699b13b1cdd17a9bb0329c298b15fccb73d53481ccb96658a2c707f not found: ID does not exist" containerID="b9db464c8699b13b1cdd17a9bb0329c298b15fccb73d53481ccb96658a2c707f" Nov 03 01:34:57.363423 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:34:57.363333 4939 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b9db464c8699b13b1cdd17a9bb0329c298b15fccb73d53481ccb96658a2c707f"} err="failed to get container status \"b9db464c8699b13b1cdd17a9bb0329c298b15fccb73d53481ccb96658a2c707f\": rpc error: code = NotFound desc = could not find container \"b9db464c8699b13b1cdd17a9bb0329c298b15fccb73d53481ccb96658a2c707f\": container with ID starting with b9db464c8699b13b1cdd17a9bb0329c298b15fccb73d53481ccb96658a2c707f not found: ID does not exist" Nov 03 01:34:57.363423 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:34:57.363356 4939 scope.go:122] "RemoveContainer" containerID="daffb5100933e1f21d8c999ff6a9c75480633c4dfc1f5014ebff8b4f61f3e10a" Nov 03 01:34:57.363653 ip-10-0-140-107 kubenswrapper[4939]: E1103 01:34:57.363634 4939 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"daffb5100933e1f21d8c999ff6a9c75480633c4dfc1f5014ebff8b4f61f3e10a\": container with ID starting with daffb5100933e1f21d8c999ff6a9c75480633c4dfc1f5014ebff8b4f61f3e10a not found: ID does not exist" containerID="daffb5100933e1f21d8c999ff6a9c75480633c4dfc1f5014ebff8b4f61f3e10a" Nov 03 01:34:57.363712 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:34:57.363660 4939 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"daffb5100933e1f21d8c999ff6a9c75480633c4dfc1f5014ebff8b4f61f3e10a"} err="failed to get container status \"daffb5100933e1f21d8c999ff6a9c75480633c4dfc1f5014ebff8b4f61f3e10a\": rpc error: code = NotFound desc = could not find container \"daffb5100933e1f21d8c999ff6a9c75480633c4dfc1f5014ebff8b4f61f3e10a\": container with ID starting with daffb5100933e1f21d8c999ff6a9c75480633c4dfc1f5014ebff8b4f61f3e10a not found: ID does not exist" Nov 03 01:34:57.377458 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:34:57.377434 4939 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-2f9a3a-openshift-default-7f6f74cf97-dxffl"] Nov 03 01:34:57.409199 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:34:57.408949 4939 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-2f9a3a-openshift-default-7f6f74cf97-dxffl"] Nov 03 01:34:57.840146 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:34:57.840086 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:34:57.893190 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:34:57.893144 4939 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="80e0f915-1a1a-4214-92f0-5eeb806be96e" path="/var/lib/kubelet/pods/80e0f915-1a1a-4214-92f0-5eeb806be96e/volumes" Nov 03 01:34:58.421611 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:34:58.421579 4939 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-b4971a/test-model-b4971a-kserve-5f54798fc5-hc2bn"] Nov 03 01:34:58.473057 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:34:58.473025 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:34:58.602777 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:34:58.602719 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:34:58.918976 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:34:58.918943 4939 scope.go:122] "RemoveContainer" containerID="417abf9a2d53f92307b997d0dd67ef2989b4a47fbb9f30f80d4fcc23d92ab776" Nov 03 01:34:58.943885 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:34:58.943856 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:34:59.317034 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:34:59.316948 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:35:00.002131 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:35:00.002096 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:35:00.765649 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:35:00.765609 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:35:01.481790 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:35:01.481753 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:35:02.243396 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:35:02.243269 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:35:02.363272 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:35:02.363220 4939 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-inf-a-b6afd6/test-model-b6afd6-kserve-86dc6b8b77-pl784" podUID="04722cf5-df49-495f-a59a-e108d802c8e1" containerName="main" containerID="cri-o://308fdc41d0ef1aaa117dd922584091976046bc41729824ee503708ae3cb63560" gracePeriod=45 Nov 03 01:35:02.679799 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:35:02.679764 4939 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-b6afd6/test-model-b6afd6-kserve-86dc6b8b77-pl784" Nov 03 01:35:02.847198 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:35:02.847156 4939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/04722cf5-df49-495f-a59a-e108d802c8e1-home\" (UniqueName: \"kubernetes.io/empty-dir/04722cf5-df49-495f-a59a-e108d802c8e1-home\") pod \"04722cf5-df49-495f-a59a-e108d802c8e1\" (UID: \"04722cf5-df49-495f-a59a-e108d802c8e1\") " Nov 03 01:35:02.847390 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:35:02.847253 4939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/04722cf5-df49-495f-a59a-e108d802c8e1-kube-api-access-sq2p8\" (UniqueName: \"kubernetes.io/projected/04722cf5-df49-495f-a59a-e108d802c8e1-kube-api-access-sq2p8\") pod \"04722cf5-df49-495f-a59a-e108d802c8e1\" (UID: \"04722cf5-df49-495f-a59a-e108d802c8e1\") " Nov 03 01:35:02.847390 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:35:02.847301 4939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/04722cf5-df49-495f-a59a-e108d802c8e1-dshm\" (UniqueName: \"kubernetes.io/empty-dir/04722cf5-df49-495f-a59a-e108d802c8e1-dshm\") pod \"04722cf5-df49-495f-a59a-e108d802c8e1\" (UID: \"04722cf5-df49-495f-a59a-e108d802c8e1\") " Nov 03 01:35:02.847483 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:35:02.847463 4939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/04722cf5-df49-495f-a59a-e108d802c8e1-home" pod "04722cf5-df49-495f-a59a-e108d802c8e1" (UID: "04722cf5-df49-495f-a59a-e108d802c8e1"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 01:35:02.847625 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:35:02.847605 4939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/04722cf5-df49-495f-a59a-e108d802c8e1-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/04722cf5-df49-495f-a59a-e108d802c8e1-model-cache\") pod \"04722cf5-df49-495f-a59a-e108d802c8e1\" (UID: \"04722cf5-df49-495f-a59a-e108d802c8e1\") " Nov 03 01:35:02.847704 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:35:02.847653 4939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/04722cf5-df49-495f-a59a-e108d802c8e1-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/04722cf5-df49-495f-a59a-e108d802c8e1-tmp-dir\") pod \"04722cf5-df49-495f-a59a-e108d802c8e1\" (UID: \"04722cf5-df49-495f-a59a-e108d802c8e1\") " Nov 03 01:35:02.847779 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:35:02.847741 4939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/04722cf5-df49-495f-a59a-e108d802c8e1-tls-certs\" (UniqueName: \"kubernetes.io/secret/04722cf5-df49-495f-a59a-e108d802c8e1-tls-certs\") pod \"04722cf5-df49-495f-a59a-e108d802c8e1\" (UID: \"04722cf5-df49-495f-a59a-e108d802c8e1\") " Nov 03 01:35:02.847899 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:35:02.847879 4939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/04722cf5-df49-495f-a59a-e108d802c8e1-model-cache" pod "04722cf5-df49-495f-a59a-e108d802c8e1" (UID: "04722cf5-df49-495f-a59a-e108d802c8e1"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 01:35:02.848108 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:35:02.848045 4939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/04722cf5-df49-495f-a59a-e108d802c8e1-tmp-dir" pod "04722cf5-df49-495f-a59a-e108d802c8e1" (UID: "04722cf5-df49-495f-a59a-e108d802c8e1"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 01:35:02.848591 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:35:02.848566 4939 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/04722cf5-df49-495f-a59a-e108d802c8e1-model-cache\") on node \"ip-10-0-140-107.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:35:02.848591 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:35:02.848595 4939 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/04722cf5-df49-495f-a59a-e108d802c8e1-tmp-dir\") on node \"ip-10-0-140-107.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:35:02.848769 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:35:02.848610 4939 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/04722cf5-df49-495f-a59a-e108d802c8e1-home\") on node \"ip-10-0-140-107.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:35:02.850036 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:35:02.850011 4939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/04722cf5-df49-495f-a59a-e108d802c8e1-dshm" pod "04722cf5-df49-495f-a59a-e108d802c8e1" (UID: "04722cf5-df49-495f-a59a-e108d802c8e1"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 01:35:02.850296 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:35:02.850269 4939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/04722cf5-df49-495f-a59a-e108d802c8e1-kube-api-access-sq2p8" pod "04722cf5-df49-495f-a59a-e108d802c8e1" (UID: "04722cf5-df49-495f-a59a-e108d802c8e1"). InnerVolumeSpecName "kube-api-access-sq2p8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 01:35:02.851538 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:35:02.851514 4939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/04722cf5-df49-495f-a59a-e108d802c8e1-tls-certs" pod "04722cf5-df49-495f-a59a-e108d802c8e1" (UID: "04722cf5-df49-495f-a59a-e108d802c8e1"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 01:35:02.950618 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:35:02.950180 4939 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/04722cf5-df49-495f-a59a-e108d802c8e1-tls-certs\") on node \"ip-10-0-140-107.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:35:02.950618 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:35:02.950219 4939 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sq2p8\" (UniqueName: \"kubernetes.io/projected/04722cf5-df49-495f-a59a-e108d802c8e1-kube-api-access-sq2p8\") on node \"ip-10-0-140-107.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:35:02.950618 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:35:02.950234 4939 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/04722cf5-df49-495f-a59a-e108d802c8e1-dshm\") on node \"ip-10-0-140-107.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:35:03.068089 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:35:03.068053 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:35:03.421786 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:35:03.421752 4939 generic.go:356] "Generic (PLEG): container finished" podID="04722cf5-df49-495f-a59a-e108d802c8e1" containerID="308fdc41d0ef1aaa117dd922584091976046bc41729824ee503708ae3cb63560" exitCode=0 Nov 03 01:35:03.422039 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:35:03.421806 4939 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-b6afd6/test-model-b6afd6-kserve-86dc6b8b77-pl784" Nov 03 01:35:03.422039 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:35:03.421835 4939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-b6afd6/test-model-b6afd6-kserve-86dc6b8b77-pl784" event={"ID":"04722cf5-df49-495f-a59a-e108d802c8e1","Type":"ContainerDied","Data":"308fdc41d0ef1aaa117dd922584091976046bc41729824ee503708ae3cb63560"} Nov 03 01:35:03.422039 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:35:03.421876 4939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-b6afd6/test-model-b6afd6-kserve-86dc6b8b77-pl784" event={"ID":"04722cf5-df49-495f-a59a-e108d802c8e1","Type":"ContainerDied","Data":"3e08fa6c91f664f57f496248033ba7ece7880161c0f2f1a63c2649fdd59936d3"} Nov 03 01:35:03.422039 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:35:03.421895 4939 scope.go:122] "RemoveContainer" containerID="308fdc41d0ef1aaa117dd922584091976046bc41729824ee503708ae3cb63560" Nov 03 01:35:03.433897 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:35:03.433878 4939 scope.go:122] "RemoveContainer" containerID="308fdc41d0ef1aaa117dd922584091976046bc41729824ee503708ae3cb63560" Nov 03 01:35:03.434264 ip-10-0-140-107 kubenswrapper[4939]: E1103 01:35:03.434244 4939 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"308fdc41d0ef1aaa117dd922584091976046bc41729824ee503708ae3cb63560\": container with ID starting with 308fdc41d0ef1aaa117dd922584091976046bc41729824ee503708ae3cb63560 not found: ID does not exist" containerID="308fdc41d0ef1aaa117dd922584091976046bc41729824ee503708ae3cb63560" Nov 03 01:35:03.434328 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:35:03.434275 4939 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"308fdc41d0ef1aaa117dd922584091976046bc41729824ee503708ae3cb63560"} err="failed to get container status \"308fdc41d0ef1aaa117dd922584091976046bc41729824ee503708ae3cb63560\": rpc error: code = NotFound desc = could not find container \"308fdc41d0ef1aaa117dd922584091976046bc41729824ee503708ae3cb63560\": container with ID starting with 308fdc41d0ef1aaa117dd922584091976046bc41729824ee503708ae3cb63560 not found: ID does not exist" Nov 03 01:35:03.449137 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:35:03.449106 4939 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-b6afd6/test-model-b6afd6-kserve-86dc6b8b77-pl784"] Nov 03 01:35:03.453363 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:35:03.453339 4939 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-a-b6afd6/test-model-b6afd6-kserve-86dc6b8b77-pl784"] Nov 03 01:35:03.778094 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:35:03.778008 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:35:03.871900 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:35:03.871863 4939 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="04722cf5-df49-495f-a59a-e108d802c8e1" path="/var/lib/kubelet/pods/04722cf5-df49-495f-a59a-e108d802c8e1/volumes" Nov 03 01:35:04.587515 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:35:04.587481 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:35:04.959465 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:35:04.959381 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:35:05.071325 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:35:05.071294 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:35:05.395837 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:35:05.395805 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:35:05.576109 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:35:05.576011 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:35:06.193492 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:35:06.193459 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:35:06.903998 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:35:06.903957 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:35:07.638560 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:35:07.638528 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:35:08.366346 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:35:08.366311 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:35:08.901103 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:35:08.901055 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:35:09.602389 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:35:09.602351 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:35:10.259026 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:35:10.258994 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:35:10.877824 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:35:10.877793 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:35:11.546249 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:35:11.546214 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:35:11.651840 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:35:11.651789 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:35:12.047051 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:35:12.047019 4939 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-b6afd6-54f7b98c7c-lzq84"] Nov 03 01:35:12.047296 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:35:12.047268 4939 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-b6afd6-54f7b98c7c-lzq84" podUID="32cc7c64-ec44-4a5d-815a-a378d840e3ff" containerName="maas-api" containerID="cri-o://a11121971cff6d386abd3de73b678887187a59487367bbf28c9802796918e69d" gracePeriod=30 Nov 03 01:35:12.127310 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:35:12.127270 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:35:12.506930 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:35:12.506880 4939 generic.go:356] "Generic (PLEG): container finished" podID="32cc7c64-ec44-4a5d-815a-a378d840e3ff" containerID="a11121971cff6d386abd3de73b678887187a59487367bbf28c9802796918e69d" exitCode=0 Nov 03 01:35:12.507479 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:35:12.507451 4939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-b6afd6-54f7b98c7c-lzq84" event={"ID":"32cc7c64-ec44-4a5d-815a-a378d840e3ff","Type":"ContainerDied","Data":"a11121971cff6d386abd3de73b678887187a59487367bbf28c9802796918e69d"} Nov 03 01:35:12.614023 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:35:12.613934 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:35:12.701864 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:35:12.701834 4939 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-b6afd6-54f7b98c7c-lzq84" Nov 03 01:35:12.793088 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:35:12.793041 4939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/32cc7c64-ec44-4a5d-815a-a378d840e3ff-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/32cc7c64-ec44-4a5d-815a-a378d840e3ff-maas-api-tls\") pod \"32cc7c64-ec44-4a5d-815a-a378d840e3ff\" (UID: \"32cc7c64-ec44-4a5d-815a-a378d840e3ff\") " Nov 03 01:35:12.793299 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:35:12.793277 4939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/32cc7c64-ec44-4a5d-815a-a378d840e3ff-kube-api-access-lq8k6\" (UniqueName: \"kubernetes.io/projected/32cc7c64-ec44-4a5d-815a-a378d840e3ff-kube-api-access-lq8k6\") pod \"32cc7c64-ec44-4a5d-815a-a378d840e3ff\" (UID: \"32cc7c64-ec44-4a5d-815a-a378d840e3ff\") " Nov 03 01:35:12.793428 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:35:12.793318 4939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/32cc7c64-ec44-4a5d-815a-a378d840e3ff-metrics-certs\" (UniqueName: \"kubernetes.io/secret/32cc7c64-ec44-4a5d-815a-a378d840e3ff-metrics-certs\") pod \"32cc7c64-ec44-4a5d-815a-a378d840e3ff\" (UID: \"32cc7c64-ec44-4a5d-815a-a378d840e3ff\") " Nov 03 01:35:12.793428 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:35:12.793352 4939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/32cc7c64-ec44-4a5d-815a-a378d840e3ff-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/32cc7c64-ec44-4a5d-815a-a378d840e3ff-openshift-service-ca\") pod \"32cc7c64-ec44-4a5d-815a-a378d840e3ff\" (UID: \"32cc7c64-ec44-4a5d-815a-a378d840e3ff\") " Nov 03 01:35:12.793849 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:35:12.793800 4939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/32cc7c64-ec44-4a5d-815a-a378d840e3ff-openshift-service-ca" pod "32cc7c64-ec44-4a5d-815a-a378d840e3ff" (UID: "32cc7c64-ec44-4a5d-815a-a378d840e3ff"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 01:35:12.794190 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:35:12.794163 4939 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/32cc7c64-ec44-4a5d-815a-a378d840e3ff-openshift-service-ca\") on node \"ip-10-0-140-107.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:35:12.795589 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:35:12.795562 4939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/32cc7c64-ec44-4a5d-815a-a378d840e3ff-metrics-certs" pod "32cc7c64-ec44-4a5d-815a-a378d840e3ff" (UID: "32cc7c64-ec44-4a5d-815a-a378d840e3ff"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 01:35:12.795771 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:35:12.795742 4939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/32cc7c64-ec44-4a5d-815a-a378d840e3ff-kube-api-access-lq8k6" pod "32cc7c64-ec44-4a5d-815a-a378d840e3ff" (UID: "32cc7c64-ec44-4a5d-815a-a378d840e3ff"). InnerVolumeSpecName "kube-api-access-lq8k6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 01:35:12.795864 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:35:12.795780 4939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/32cc7c64-ec44-4a5d-815a-a378d840e3ff-maas-api-tls" pod "32cc7c64-ec44-4a5d-815a-a378d840e3ff" (UID: "32cc7c64-ec44-4a5d-815a-a378d840e3ff"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 01:35:12.896318 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:35:12.896006 4939 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/32cc7c64-ec44-4a5d-815a-a378d840e3ff-maas-api-tls\") on node \"ip-10-0-140-107.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:35:12.896318 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:35:12.896039 4939 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lq8k6\" (UniqueName: \"kubernetes.io/projected/32cc7c64-ec44-4a5d-815a-a378d840e3ff-kube-api-access-lq8k6\") on node \"ip-10-0-140-107.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:35:12.896318 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:35:12.896149 4939 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/32cc7c64-ec44-4a5d-815a-a378d840e3ff-metrics-certs\") on node \"ip-10-0-140-107.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:35:13.250470 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:35:13.250385 4939 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-4pw7v"] Nov 03 01:35:13.264049 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:35:13.264022 4939 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-737bf2d8-openshift-default-56dc96b957-jhxgw"] Nov 03 01:35:13.264858 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:35:13.264842 4939 cpu_manager.go:414] "containerMap: removing container" podUID="0a1fe883-48d5-4d06-ab8f-41698965dbbb" containerName="istio-proxy" Nov 03 01:35:13.264964 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:35:13.264859 4939 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0a1fe883-48d5-4d06-ab8f-41698965dbbb" containerName="istio-proxy" Nov 03 01:35:13.264964 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:35:13.264880 4939 cpu_manager.go:414] "containerMap: removing container" podUID="aba9a06e-b5d6-491f-b9a3-34ebd927d4aa" containerName="istio-proxy" Nov 03 01:35:13.264964 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:35:13.264886 4939 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="aba9a06e-b5d6-491f-b9a3-34ebd927d4aa" containerName="istio-proxy" Nov 03 01:35:13.264964 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:35:13.264893 4939 cpu_manager.go:414] "containerMap: removing container" podUID="32cc7c64-ec44-4a5d-815a-a378d840e3ff" containerName="maas-api" Nov 03 01:35:13.264964 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:35:13.264900 4939 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="32cc7c64-ec44-4a5d-815a-a378d840e3ff" containerName="maas-api" Nov 03 01:35:13.264964 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:35:13.264930 4939 cpu_manager.go:414] "containerMap: removing container" podUID="80e0f915-1a1a-4214-92f0-5eeb806be96e" containerName="istio-proxy" Nov 03 01:35:13.264964 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:35:13.264936 4939 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="80e0f915-1a1a-4214-92f0-5eeb806be96e" containerName="istio-proxy" Nov 03 01:35:13.264964 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:35:13.264968 4939 cpu_manager.go:414] "containerMap: removing container" podUID="cba006d1-08cd-4fac-8bdd-86931f742838" containerName="istio-proxy" Nov 03 01:35:13.265255 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:35:13.264973 4939 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cba006d1-08cd-4fac-8bdd-86931f742838" containerName="istio-proxy" Nov 03 01:35:13.265255 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:35:13.264987 4939 cpu_manager.go:414] "containerMap: removing container" podUID="04722cf5-df49-495f-a59a-e108d802c8e1" containerName="main" Nov 03 01:35:13.265255 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:35:13.264992 4939 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="04722cf5-df49-495f-a59a-e108d802c8e1" containerName="main" Nov 03 01:35:13.265255 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:35:13.264998 4939 cpu_manager.go:414] "containerMap: removing container" podUID="b22388ea-ae77-4ad1-bfff-9aeb20f5bbbd" containerName="istio-proxy" Nov 03 01:35:13.265255 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:35:13.265003 4939 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b22388ea-ae77-4ad1-bfff-9aeb20f5bbbd" containerName="istio-proxy" Nov 03 01:35:13.265255 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:35:13.265012 4939 cpu_manager.go:414] "containerMap: removing container" podUID="cba006d1-08cd-4fac-8bdd-86931f742838" containerName="istio-proxy" Nov 03 01:35:13.265255 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:35:13.265016 4939 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cba006d1-08cd-4fac-8bdd-86931f742838" containerName="istio-proxy" Nov 03 01:35:13.265255 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:35:13.265026 4939 cpu_manager.go:414] "containerMap: removing container" podUID="80e0f915-1a1a-4214-92f0-5eeb806be96e" containerName="istio-proxy" Nov 03 01:35:13.265255 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:35:13.265031 4939 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="80e0f915-1a1a-4214-92f0-5eeb806be96e" containerName="istio-proxy" Nov 03 01:35:13.265255 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:35:13.265040 4939 cpu_manager.go:414] "containerMap: removing container" podUID="c7907eaa-df92-4d28-aa19-ba6f03cb54b1" containerName="istio-proxy" Nov 03 01:35:13.265255 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:35:13.265044 4939 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c7907eaa-df92-4d28-aa19-ba6f03cb54b1" containerName="istio-proxy" Nov 03 01:35:13.265255 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:35:13.265059 4939 cpu_manager.go:414] "containerMap: removing container" podUID="ba28e265-fab5-43ba-94b2-dc5b7dd5029f" containerName="istio-proxy" Nov 03 01:35:13.265255 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:35:13.265064 4939 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ba28e265-fab5-43ba-94b2-dc5b7dd5029f" containerName="istio-proxy" Nov 03 01:35:13.265255 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:35:13.265198 4939 memory_manager.go:365] "RemoveStaleState removing state" podUID="cba006d1-08cd-4fac-8bdd-86931f742838" containerName="istio-proxy" Nov 03 01:35:13.265255 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:35:13.265208 4939 memory_manager.go:365] "RemoveStaleState removing state" podUID="0a1fe883-48d5-4d06-ab8f-41698965dbbb" containerName="istio-proxy" Nov 03 01:35:13.265255 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:35:13.265215 4939 memory_manager.go:365] "RemoveStaleState removing state" podUID="04722cf5-df49-495f-a59a-e108d802c8e1" containerName="main" Nov 03 01:35:13.265255 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:35:13.265223 4939 memory_manager.go:365] "RemoveStaleState removing state" podUID="b22388ea-ae77-4ad1-bfff-9aeb20f5bbbd" containerName="istio-proxy" Nov 03 01:35:13.265255 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:35:13.265228 4939 memory_manager.go:365] "RemoveStaleState removing state" podUID="cba006d1-08cd-4fac-8bdd-86931f742838" containerName="istio-proxy" Nov 03 01:35:13.265255 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:35:13.265234 4939 memory_manager.go:365] "RemoveStaleState removing state" podUID="80e0f915-1a1a-4214-92f0-5eeb806be96e" containerName="istio-proxy" Nov 03 01:35:13.265255 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:35:13.265240 4939 memory_manager.go:365] "RemoveStaleState removing state" podUID="c7907eaa-df92-4d28-aa19-ba6f03cb54b1" containerName="istio-proxy" Nov 03 01:35:13.265255 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:35:13.265248 4939 memory_manager.go:365] "RemoveStaleState removing state" podUID="ba28e265-fab5-43ba-94b2-dc5b7dd5029f" containerName="istio-proxy" Nov 03 01:35:13.265255 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:35:13.265256 4939 memory_manager.go:365] "RemoveStaleState removing state" podUID="aba9a06e-b5d6-491f-b9a3-34ebd927d4aa" containerName="istio-proxy" Nov 03 01:35:13.265255 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:35:13.265262 4939 memory_manager.go:365] "RemoveStaleState removing state" podUID="32cc7c64-ec44-4a5d-815a-a378d840e3ff" containerName="maas-api" Nov 03 01:35:13.269540 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:35:13.269519 4939 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-737bf2d8-openshift-default-56dc96b957-jhxgw" Nov 03 01:35:13.329002 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:35:13.314143 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cc9b0d3f-c18a-4e8f-ab5b-1c9d9d1b244c-istio-data\") pod \"e2e-mt-737bf2d8-openshift-default-56dc96b957-jhxgw\" (UID: \"cc9b0d3f-c18a-4e8f-ab5b-1c9d9d1b244c\") " pod="openshift-ingress/e2e-mt-737bf2d8-openshift-default-56dc96b957-jhxgw" Nov 03 01:35:13.330564 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:35:13.317970 4939 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-737bf2d8-openshift-default-56dc96b957-jhxgw"] Nov 03 01:35:13.330564 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:35:13.329515 4939 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-737bf2d8-openshift-default-56dc96b957-mwnjn"] Nov 03 01:35:13.330564 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:35:13.329551 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cc9b0d3f-c18a-4e8f-ab5b-1c9d9d1b244c-istio-envoy\") pod \"e2e-mt-737bf2d8-openshift-default-56dc96b957-jhxgw\" (UID: \"cc9b0d3f-c18a-4e8f-ab5b-1c9d9d1b244c\") " pod="openshift-ingress/e2e-mt-737bf2d8-openshift-default-56dc96b957-jhxgw" Nov 03 01:35:13.330564 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:35:13.329585 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cc9b0d3f-c18a-4e8f-ab5b-1c9d9d1b244c-istio-podinfo\") pod \"e2e-mt-737bf2d8-openshift-default-56dc96b957-jhxgw\" (UID: \"cc9b0d3f-c18a-4e8f-ab5b-1c9d9d1b244c\") " pod="openshift-ingress/e2e-mt-737bf2d8-openshift-default-56dc96b957-jhxgw" Nov 03 01:35:13.330564 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:35:13.329676 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cc9b0d3f-c18a-4e8f-ab5b-1c9d9d1b244c-istio-token\") pod \"e2e-mt-737bf2d8-openshift-default-56dc96b957-jhxgw\" (UID: \"cc9b0d3f-c18a-4e8f-ab5b-1c9d9d1b244c\") " pod="openshift-ingress/e2e-mt-737bf2d8-openshift-default-56dc96b957-jhxgw" Nov 03 01:35:13.330564 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:35:13.329794 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6sw2h\" (UniqueName: \"kubernetes.io/projected/cc9b0d3f-c18a-4e8f-ab5b-1c9d9d1b244c-kube-api-access-6sw2h\") pod \"e2e-mt-737bf2d8-openshift-default-56dc96b957-jhxgw\" (UID: \"cc9b0d3f-c18a-4e8f-ab5b-1c9d9d1b244c\") " pod="openshift-ingress/e2e-mt-737bf2d8-openshift-default-56dc96b957-jhxgw" Nov 03 01:35:13.330564 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:35:13.329874 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cc9b0d3f-c18a-4e8f-ab5b-1c9d9d1b244c-credential-socket\") pod \"e2e-mt-737bf2d8-openshift-default-56dc96b957-jhxgw\" (UID: \"cc9b0d3f-c18a-4e8f-ab5b-1c9d9d1b244c\") " pod="openshift-ingress/e2e-mt-737bf2d8-openshift-default-56dc96b957-jhxgw" Nov 03 01:35:13.330564 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:35:13.330033 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cc9b0d3f-c18a-4e8f-ab5b-1c9d9d1b244c-istiod-ca-cert\") pod \"e2e-mt-737bf2d8-openshift-default-56dc96b957-jhxgw\" (UID: \"cc9b0d3f-c18a-4e8f-ab5b-1c9d9d1b244c\") " pod="openshift-ingress/e2e-mt-737bf2d8-openshift-default-56dc96b957-jhxgw" Nov 03 01:35:13.330564 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:35:13.330170 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cc9b0d3f-c18a-4e8f-ab5b-1c9d9d1b244c-workload-certs\") pod \"e2e-mt-737bf2d8-openshift-default-56dc96b957-jhxgw\" (UID: \"cc9b0d3f-c18a-4e8f-ab5b-1c9d9d1b244c\") " pod="openshift-ingress/e2e-mt-737bf2d8-openshift-default-56dc96b957-jhxgw" Nov 03 01:35:13.330564 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:35:13.330202 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cc9b0d3f-c18a-4e8f-ab5b-1c9d9d1b244c-workload-socket\") pod \"e2e-mt-737bf2d8-openshift-default-56dc96b957-jhxgw\" (UID: \"cc9b0d3f-c18a-4e8f-ab5b-1c9d9d1b244c\") " pod="openshift-ingress/e2e-mt-737bf2d8-openshift-default-56dc96b957-jhxgw" Nov 03 01:35:13.338510 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:35:13.322625 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:35:13.338510 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:35:13.334373 4939 cpu_manager.go:414] "containerMap: removing container" podUID="0a1fe883-48d5-4d06-ab8f-41698965dbbb" containerName="istio-proxy" Nov 03 01:35:13.338510 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:35:13.334398 4939 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0a1fe883-48d5-4d06-ab8f-41698965dbbb" containerName="istio-proxy" Nov 03 01:35:13.341363 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:35:13.339093 4939 memory_manager.go:365] "RemoveStaleState removing state" podUID="0a1fe883-48d5-4d06-ab8f-41698965dbbb" containerName="istio-proxy" Nov 03 01:35:13.341363 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:35:13.339126 4939 memory_manager.go:365] "RemoveStaleState removing state" podUID="80e0f915-1a1a-4214-92f0-5eeb806be96e" containerName="istio-proxy" Nov 03 01:35:13.343739 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:35:13.343715 4939 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-737bf2d8-openshift-default-56dc96b957-mwnjn" Nov 03 01:35:13.370072 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:35:13.370045 4939 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-737bf2d8-openshift-default-56dc96b957-mwnjn"] Nov 03 01:35:13.431701 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:35:13.431663 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ae67a584-3143-44d0-a76f-97f60d25dd60-istiod-ca-cert\") pod \"e2e-mt-737bf2d8-openshift-default-56dc96b957-mwnjn\" (UID: \"ae67a584-3143-44d0-a76f-97f60d25dd60\") " pod="openshift-ingress/e2e-mt-737bf2d8-openshift-default-56dc96b957-mwnjn" Nov 03 01:35:13.431894 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:35:13.431713 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ae67a584-3143-44d0-a76f-97f60d25dd60-istio-envoy\") pod \"e2e-mt-737bf2d8-openshift-default-56dc96b957-mwnjn\" (UID: \"ae67a584-3143-44d0-a76f-97f60d25dd60\") " pod="openshift-ingress/e2e-mt-737bf2d8-openshift-default-56dc96b957-mwnjn" Nov 03 01:35:13.431894 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:35:13.431874 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cc9b0d3f-c18a-4e8f-ab5b-1c9d9d1b244c-istiod-ca-cert\") pod \"e2e-mt-737bf2d8-openshift-default-56dc96b957-jhxgw\" (UID: \"cc9b0d3f-c18a-4e8f-ab5b-1c9d9d1b244c\") " pod="openshift-ingress/e2e-mt-737bf2d8-openshift-default-56dc96b957-jhxgw" Nov 03 01:35:13.432086 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:35:13.432066 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cc9b0d3f-c18a-4e8f-ab5b-1c9d9d1b244c-workload-certs\") pod \"e2e-mt-737bf2d8-openshift-default-56dc96b957-jhxgw\" (UID: \"cc9b0d3f-c18a-4e8f-ab5b-1c9d9d1b244c\") " pod="openshift-ingress/e2e-mt-737bf2d8-openshift-default-56dc96b957-jhxgw" Nov 03 01:35:13.432158 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:35:13.432107 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cc9b0d3f-c18a-4e8f-ab5b-1c9d9d1b244c-workload-socket\") pod \"e2e-mt-737bf2d8-openshift-default-56dc96b957-jhxgw\" (UID: \"cc9b0d3f-c18a-4e8f-ab5b-1c9d9d1b244c\") " pod="openshift-ingress/e2e-mt-737bf2d8-openshift-default-56dc96b957-jhxgw" Nov 03 01:35:13.432158 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:35:13.432147 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cc9b0d3f-c18a-4e8f-ab5b-1c9d9d1b244c-istio-data\") pod \"e2e-mt-737bf2d8-openshift-default-56dc96b957-jhxgw\" (UID: \"cc9b0d3f-c18a-4e8f-ab5b-1c9d9d1b244c\") " pod="openshift-ingress/e2e-mt-737bf2d8-openshift-default-56dc96b957-jhxgw" Nov 03 01:35:13.432283 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:35:13.432202 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ae67a584-3143-44d0-a76f-97f60d25dd60-workload-socket\") pod \"e2e-mt-737bf2d8-openshift-default-56dc96b957-mwnjn\" (UID: \"ae67a584-3143-44d0-a76f-97f60d25dd60\") " pod="openshift-ingress/e2e-mt-737bf2d8-openshift-default-56dc96b957-mwnjn" Nov 03 01:35:13.432345 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:35:13.432302 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cc9b0d3f-c18a-4e8f-ab5b-1c9d9d1b244c-istio-envoy\") pod \"e2e-mt-737bf2d8-openshift-default-56dc96b957-jhxgw\" (UID: \"cc9b0d3f-c18a-4e8f-ab5b-1c9d9d1b244c\") " pod="openshift-ingress/e2e-mt-737bf2d8-openshift-default-56dc96b957-jhxgw" Nov 03 01:35:13.432345 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:35:13.432332 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cc9b0d3f-c18a-4e8f-ab5b-1c9d9d1b244c-istio-podinfo\") pod \"e2e-mt-737bf2d8-openshift-default-56dc96b957-jhxgw\" (UID: \"cc9b0d3f-c18a-4e8f-ab5b-1c9d9d1b244c\") " pod="openshift-ingress/e2e-mt-737bf2d8-openshift-default-56dc96b957-jhxgw" Nov 03 01:35:13.432460 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:35:13.432377 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ae67a584-3143-44d0-a76f-97f60d25dd60-istio-data\") pod \"e2e-mt-737bf2d8-openshift-default-56dc96b957-mwnjn\" (UID: \"ae67a584-3143-44d0-a76f-97f60d25dd60\") " pod="openshift-ingress/e2e-mt-737bf2d8-openshift-default-56dc96b957-mwnjn" Nov 03 01:35:13.432460 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:35:13.432427 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cc9b0d3f-c18a-4e8f-ab5b-1c9d9d1b244c-istio-token\") pod \"e2e-mt-737bf2d8-openshift-default-56dc96b957-jhxgw\" (UID: \"cc9b0d3f-c18a-4e8f-ab5b-1c9d9d1b244c\") " pod="openshift-ingress/e2e-mt-737bf2d8-openshift-default-56dc96b957-jhxgw" Nov 03 01:35:13.432594 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:35:13.432457 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cc9b0d3f-c18a-4e8f-ab5b-1c9d9d1b244c-workload-certs\") pod \"e2e-mt-737bf2d8-openshift-default-56dc96b957-jhxgw\" (UID: \"cc9b0d3f-c18a-4e8f-ab5b-1c9d9d1b244c\") " pod="openshift-ingress/e2e-mt-737bf2d8-openshift-default-56dc96b957-jhxgw" Nov 03 01:35:13.432594 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:35:13.432468 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ae67a584-3143-44d0-a76f-97f60d25dd60-workload-certs\") pod \"e2e-mt-737bf2d8-openshift-default-56dc96b957-mwnjn\" (UID: \"ae67a584-3143-44d0-a76f-97f60d25dd60\") " pod="openshift-ingress/e2e-mt-737bf2d8-openshift-default-56dc96b957-mwnjn" Nov 03 01:35:13.432594 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:35:13.432475 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cc9b0d3f-c18a-4e8f-ab5b-1c9d9d1b244c-workload-socket\") pod \"e2e-mt-737bf2d8-openshift-default-56dc96b957-jhxgw\" (UID: \"cc9b0d3f-c18a-4e8f-ab5b-1c9d9d1b244c\") " pod="openshift-ingress/e2e-mt-737bf2d8-openshift-default-56dc96b957-jhxgw" Nov 03 01:35:13.432594 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:35:13.432530 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cc9b0d3f-c18a-4e8f-ab5b-1c9d9d1b244c-istio-data\") pod \"e2e-mt-737bf2d8-openshift-default-56dc96b957-jhxgw\" (UID: \"cc9b0d3f-c18a-4e8f-ab5b-1c9d9d1b244c\") " pod="openshift-ingress/e2e-mt-737bf2d8-openshift-default-56dc96b957-jhxgw" Nov 03 01:35:13.432818 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:35:13.432645 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cc9b0d3f-c18a-4e8f-ab5b-1c9d9d1b244c-istiod-ca-cert\") pod \"e2e-mt-737bf2d8-openshift-default-56dc96b957-jhxgw\" (UID: \"cc9b0d3f-c18a-4e8f-ab5b-1c9d9d1b244c\") " pod="openshift-ingress/e2e-mt-737bf2d8-openshift-default-56dc96b957-jhxgw" Nov 03 01:35:13.432818 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:35:13.432669 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ae67a584-3143-44d0-a76f-97f60d25dd60-credential-socket\") pod \"e2e-mt-737bf2d8-openshift-default-56dc96b957-mwnjn\" (UID: \"ae67a584-3143-44d0-a76f-97f60d25dd60\") " pod="openshift-ingress/e2e-mt-737bf2d8-openshift-default-56dc96b957-mwnjn" Nov 03 01:35:13.432818 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:35:13.432719 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6sw2h\" (UniqueName: \"kubernetes.io/projected/cc9b0d3f-c18a-4e8f-ab5b-1c9d9d1b244c-kube-api-access-6sw2h\") pod \"e2e-mt-737bf2d8-openshift-default-56dc96b957-jhxgw\" (UID: \"cc9b0d3f-c18a-4e8f-ab5b-1c9d9d1b244c\") " pod="openshift-ingress/e2e-mt-737bf2d8-openshift-default-56dc96b957-jhxgw" Nov 03 01:35:13.432818 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:35:13.432748 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9299j\" (UniqueName: \"kubernetes.io/projected/ae67a584-3143-44d0-a76f-97f60d25dd60-kube-api-access-9299j\") pod \"e2e-mt-737bf2d8-openshift-default-56dc96b957-mwnjn\" (UID: \"ae67a584-3143-44d0-a76f-97f60d25dd60\") " pod="openshift-ingress/e2e-mt-737bf2d8-openshift-default-56dc96b957-mwnjn" Nov 03 01:35:13.432818 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:35:13.432800 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ae67a584-3143-44d0-a76f-97f60d25dd60-istio-podinfo\") pod \"e2e-mt-737bf2d8-openshift-default-56dc96b957-mwnjn\" (UID: \"ae67a584-3143-44d0-a76f-97f60d25dd60\") " pod="openshift-ingress/e2e-mt-737bf2d8-openshift-default-56dc96b957-mwnjn" Nov 03 01:35:13.433151 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:35:13.432829 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cc9b0d3f-c18a-4e8f-ab5b-1c9d9d1b244c-credential-socket\") pod \"e2e-mt-737bf2d8-openshift-default-56dc96b957-jhxgw\" (UID: \"cc9b0d3f-c18a-4e8f-ab5b-1c9d9d1b244c\") " pod="openshift-ingress/e2e-mt-737bf2d8-openshift-default-56dc96b957-jhxgw" Nov 03 01:35:13.433151 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:35:13.432899 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ae67a584-3143-44d0-a76f-97f60d25dd60-istio-token\") pod \"e2e-mt-737bf2d8-openshift-default-56dc96b957-mwnjn\" (UID: \"ae67a584-3143-44d0-a76f-97f60d25dd60\") " pod="openshift-ingress/e2e-mt-737bf2d8-openshift-default-56dc96b957-mwnjn" Nov 03 01:35:13.433268 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:35:13.433150 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cc9b0d3f-c18a-4e8f-ab5b-1c9d9d1b244c-credential-socket\") pod \"e2e-mt-737bf2d8-openshift-default-56dc96b957-jhxgw\" (UID: \"cc9b0d3f-c18a-4e8f-ab5b-1c9d9d1b244c\") " pod="openshift-ingress/e2e-mt-737bf2d8-openshift-default-56dc96b957-jhxgw" Nov 03 01:35:13.434968 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:35:13.434944 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cc9b0d3f-c18a-4e8f-ab5b-1c9d9d1b244c-istio-envoy\") pod \"e2e-mt-737bf2d8-openshift-default-56dc96b957-jhxgw\" (UID: \"cc9b0d3f-c18a-4e8f-ab5b-1c9d9d1b244c\") " pod="openshift-ingress/e2e-mt-737bf2d8-openshift-default-56dc96b957-jhxgw" Nov 03 01:35:13.435181 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:35:13.435158 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cc9b0d3f-c18a-4e8f-ab5b-1c9d9d1b244c-istio-podinfo\") pod \"e2e-mt-737bf2d8-openshift-default-56dc96b957-jhxgw\" (UID: \"cc9b0d3f-c18a-4e8f-ab5b-1c9d9d1b244c\") " pod="openshift-ingress/e2e-mt-737bf2d8-openshift-default-56dc96b957-jhxgw" Nov 03 01:35:13.441266 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:35:13.441245 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cc9b0d3f-c18a-4e8f-ab5b-1c9d9d1b244c-istio-token\") pod \"e2e-mt-737bf2d8-openshift-default-56dc96b957-jhxgw\" (UID: \"cc9b0d3f-c18a-4e8f-ab5b-1c9d9d1b244c\") " pod="openshift-ingress/e2e-mt-737bf2d8-openshift-default-56dc96b957-jhxgw" Nov 03 01:35:13.441616 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:35:13.441593 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6sw2h\" (UniqueName: \"kubernetes.io/projected/cc9b0d3f-c18a-4e8f-ab5b-1c9d9d1b244c-kube-api-access-6sw2h\") pod \"e2e-mt-737bf2d8-openshift-default-56dc96b957-jhxgw\" (UID: \"cc9b0d3f-c18a-4e8f-ab5b-1c9d9d1b244c\") " pod="openshift-ingress/e2e-mt-737bf2d8-openshift-default-56dc96b957-jhxgw" Nov 03 01:35:13.444876 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:35:13.444843 4939 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-inf-b-b4971a/test-model-b4971a-kserve-5f54798fc5-hc2bn" podUID="1c5e3dbc-2267-4ffa-b4eb-eb25bb89c2a8" containerName="main" containerID="cri-o://a4e42ae5cd492a07ddb012238b4a02885354f2001ffb7eb237fbdff3b5bf9014" gracePeriod=45 Nov 03 01:35:13.517384 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:35:13.517325 4939 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-b6afd6-54f7b98c7c-lzq84" Nov 03 01:35:13.517502 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:35:13.517326 4939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-b6afd6-54f7b98c7c-lzq84" event={"ID":"32cc7c64-ec44-4a5d-815a-a378d840e3ff","Type":"ContainerDied","Data":"a8cddbd123dce0327f642516c05761fc4374b6e6e0aceb6ae243fc8335588ae4"} Nov 03 01:35:13.517502 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:35:13.517450 4939 scope.go:122] "RemoveContainer" containerID="a11121971cff6d386abd3de73b678887187a59487367bbf28c9802796918e69d" Nov 03 01:35:13.534441 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:35:13.534415 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ae67a584-3143-44d0-a76f-97f60d25dd60-workload-socket\") pod \"e2e-mt-737bf2d8-openshift-default-56dc96b957-mwnjn\" (UID: \"ae67a584-3143-44d0-a76f-97f60d25dd60\") " pod="openshift-ingress/e2e-mt-737bf2d8-openshift-default-56dc96b957-mwnjn" Nov 03 01:35:13.534556 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:35:13.534502 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ae67a584-3143-44d0-a76f-97f60d25dd60-istio-data\") pod \"e2e-mt-737bf2d8-openshift-default-56dc96b957-mwnjn\" (UID: \"ae67a584-3143-44d0-a76f-97f60d25dd60\") " pod="openshift-ingress/e2e-mt-737bf2d8-openshift-default-56dc96b957-mwnjn" Nov 03 01:35:13.534556 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:35:13.534543 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ae67a584-3143-44d0-a76f-97f60d25dd60-workload-certs\") pod \"e2e-mt-737bf2d8-openshift-default-56dc96b957-mwnjn\" (UID: \"ae67a584-3143-44d0-a76f-97f60d25dd60\") " pod="openshift-ingress/e2e-mt-737bf2d8-openshift-default-56dc96b957-mwnjn" Nov 03 01:35:13.534677 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:35:13.534583 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ae67a584-3143-44d0-a76f-97f60d25dd60-credential-socket\") pod \"e2e-mt-737bf2d8-openshift-default-56dc96b957-mwnjn\" (UID: \"ae67a584-3143-44d0-a76f-97f60d25dd60\") " pod="openshift-ingress/e2e-mt-737bf2d8-openshift-default-56dc96b957-mwnjn" Nov 03 01:35:13.534677 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:35:13.534618 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9299j\" (UniqueName: \"kubernetes.io/projected/ae67a584-3143-44d0-a76f-97f60d25dd60-kube-api-access-9299j\") pod \"e2e-mt-737bf2d8-openshift-default-56dc96b957-mwnjn\" (UID: \"ae67a584-3143-44d0-a76f-97f60d25dd60\") " pod="openshift-ingress/e2e-mt-737bf2d8-openshift-default-56dc96b957-mwnjn" Nov 03 01:35:13.534677 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:35:13.534655 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ae67a584-3143-44d0-a76f-97f60d25dd60-istio-podinfo\") pod \"e2e-mt-737bf2d8-openshift-default-56dc96b957-mwnjn\" (UID: \"ae67a584-3143-44d0-a76f-97f60d25dd60\") " pod="openshift-ingress/e2e-mt-737bf2d8-openshift-default-56dc96b957-mwnjn" Nov 03 01:35:13.534864 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:35:13.534688 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ae67a584-3143-44d0-a76f-97f60d25dd60-istio-token\") pod \"e2e-mt-737bf2d8-openshift-default-56dc96b957-mwnjn\" (UID: \"ae67a584-3143-44d0-a76f-97f60d25dd60\") " pod="openshift-ingress/e2e-mt-737bf2d8-openshift-default-56dc96b957-mwnjn" Nov 03 01:35:13.534864 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:35:13.534722 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ae67a584-3143-44d0-a76f-97f60d25dd60-istiod-ca-cert\") pod \"e2e-mt-737bf2d8-openshift-default-56dc96b957-mwnjn\" (UID: \"ae67a584-3143-44d0-a76f-97f60d25dd60\") " pod="openshift-ingress/e2e-mt-737bf2d8-openshift-default-56dc96b957-mwnjn" Nov 03 01:35:13.534864 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:35:13.534768 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ae67a584-3143-44d0-a76f-97f60d25dd60-istio-envoy\") pod \"e2e-mt-737bf2d8-openshift-default-56dc96b957-mwnjn\" (UID: \"ae67a584-3143-44d0-a76f-97f60d25dd60\") " pod="openshift-ingress/e2e-mt-737bf2d8-openshift-default-56dc96b957-mwnjn" Nov 03 01:35:13.534864 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:35:13.534797 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ae67a584-3143-44d0-a76f-97f60d25dd60-workload-socket\") pod \"e2e-mt-737bf2d8-openshift-default-56dc96b957-mwnjn\" (UID: \"ae67a584-3143-44d0-a76f-97f60d25dd60\") " pod="openshift-ingress/e2e-mt-737bf2d8-openshift-default-56dc96b957-mwnjn" Nov 03 01:35:13.535093 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:35:13.534871 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ae67a584-3143-44d0-a76f-97f60d25dd60-istio-data\") pod \"e2e-mt-737bf2d8-openshift-default-56dc96b957-mwnjn\" (UID: \"ae67a584-3143-44d0-a76f-97f60d25dd60\") " pod="openshift-ingress/e2e-mt-737bf2d8-openshift-default-56dc96b957-mwnjn" Nov 03 01:35:13.535093 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:35:13.534981 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ae67a584-3143-44d0-a76f-97f60d25dd60-credential-socket\") pod \"e2e-mt-737bf2d8-openshift-default-56dc96b957-mwnjn\" (UID: \"ae67a584-3143-44d0-a76f-97f60d25dd60\") " pod="openshift-ingress/e2e-mt-737bf2d8-openshift-default-56dc96b957-mwnjn" Nov 03 01:35:13.535093 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:35:13.534997 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ae67a584-3143-44d0-a76f-97f60d25dd60-workload-certs\") pod \"e2e-mt-737bf2d8-openshift-default-56dc96b957-mwnjn\" (UID: \"ae67a584-3143-44d0-a76f-97f60d25dd60\") " pod="openshift-ingress/e2e-mt-737bf2d8-openshift-default-56dc96b957-mwnjn" Nov 03 01:35:13.535486 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:35:13.535466 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ae67a584-3143-44d0-a76f-97f60d25dd60-istiod-ca-cert\") pod \"e2e-mt-737bf2d8-openshift-default-56dc96b957-mwnjn\" (UID: \"ae67a584-3143-44d0-a76f-97f60d25dd60\") " pod="openshift-ingress/e2e-mt-737bf2d8-openshift-default-56dc96b957-mwnjn" Nov 03 01:35:13.537082 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:35:13.537063 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ae67a584-3143-44d0-a76f-97f60d25dd60-istio-envoy\") pod \"e2e-mt-737bf2d8-openshift-default-56dc96b957-mwnjn\" (UID: \"ae67a584-3143-44d0-a76f-97f60d25dd60\") " pod="openshift-ingress/e2e-mt-737bf2d8-openshift-default-56dc96b957-mwnjn" Nov 03 01:35:13.537298 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:35:13.537275 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ae67a584-3143-44d0-a76f-97f60d25dd60-istio-podinfo\") pod \"e2e-mt-737bf2d8-openshift-default-56dc96b957-mwnjn\" (UID: \"ae67a584-3143-44d0-a76f-97f60d25dd60\") " pod="openshift-ingress/e2e-mt-737bf2d8-openshift-default-56dc96b957-mwnjn" Nov 03 01:35:13.547390 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:35:13.547360 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ae67a584-3143-44d0-a76f-97f60d25dd60-istio-token\") pod \"e2e-mt-737bf2d8-openshift-default-56dc96b957-mwnjn\" (UID: \"ae67a584-3143-44d0-a76f-97f60d25dd60\") " pod="openshift-ingress/e2e-mt-737bf2d8-openshift-default-56dc96b957-mwnjn" Nov 03 01:35:13.547713 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:35:13.547692 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9299j\" (UniqueName: \"kubernetes.io/projected/ae67a584-3143-44d0-a76f-97f60d25dd60-kube-api-access-9299j\") pod \"e2e-mt-737bf2d8-openshift-default-56dc96b957-mwnjn\" (UID: \"ae67a584-3143-44d0-a76f-97f60d25dd60\") " pod="openshift-ingress/e2e-mt-737bf2d8-openshift-default-56dc96b957-mwnjn" Nov 03 01:35:13.593128 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:35:13.593079 4939 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-b6afd6-54f7b98c7c-lzq84"] Nov 03 01:35:13.598921 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:35:13.598883 4939 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-b6afd6-54f7b98c7c-lzq84"] Nov 03 01:35:13.640716 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:35:13.640685 4939 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-737bf2d8-openshift-default-56dc96b957-jhxgw" Nov 03 01:35:13.667137 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:35:13.667104 4939 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-737bf2d8-openshift-default-56dc96b957-mwnjn" Nov 03 01:35:13.720866 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:35:13.720841 4939 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-b4971a/test-model-b4971a-kserve-5f54798fc5-hc2bn" Nov 03 01:35:13.844586 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:35:13.844554 4939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1c5e3dbc-2267-4ffa-b4eb-eb25bb89c2a8-dshm\" (UniqueName: \"kubernetes.io/empty-dir/1c5e3dbc-2267-4ffa-b4eb-eb25bb89c2a8-dshm\") pod \"1c5e3dbc-2267-4ffa-b4eb-eb25bb89c2a8\" (UID: \"1c5e3dbc-2267-4ffa-b4eb-eb25bb89c2a8\") " Nov 03 01:35:13.844772 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:35:13.844717 4939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1c5e3dbc-2267-4ffa-b4eb-eb25bb89c2a8-home\" (UniqueName: \"kubernetes.io/empty-dir/1c5e3dbc-2267-4ffa-b4eb-eb25bb89c2a8-home\") pod \"1c5e3dbc-2267-4ffa-b4eb-eb25bb89c2a8\" (UID: \"1c5e3dbc-2267-4ffa-b4eb-eb25bb89c2a8\") " Nov 03 01:35:13.844847 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:35:13.844783 4939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1c5e3dbc-2267-4ffa-b4eb-eb25bb89c2a8-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1c5e3dbc-2267-4ffa-b4eb-eb25bb89c2a8-tmp-dir\") pod \"1c5e3dbc-2267-4ffa-b4eb-eb25bb89c2a8\" (UID: \"1c5e3dbc-2267-4ffa-b4eb-eb25bb89c2a8\") " Nov 03 01:35:13.844847 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:35:13.844812 4939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1c5e3dbc-2267-4ffa-b4eb-eb25bb89c2a8-kube-api-access-99zzj\" (UniqueName: \"kubernetes.io/projected/1c5e3dbc-2267-4ffa-b4eb-eb25bb89c2a8-kube-api-access-99zzj\") pod \"1c5e3dbc-2267-4ffa-b4eb-eb25bb89c2a8\" (UID: \"1c5e3dbc-2267-4ffa-b4eb-eb25bb89c2a8\") " Nov 03 01:35:13.844985 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:35:13.844871 4939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1c5e3dbc-2267-4ffa-b4eb-eb25bb89c2a8-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1c5e3dbc-2267-4ffa-b4eb-eb25bb89c2a8-model-cache\") pod \"1c5e3dbc-2267-4ffa-b4eb-eb25bb89c2a8\" (UID: \"1c5e3dbc-2267-4ffa-b4eb-eb25bb89c2a8\") " Nov 03 01:35:13.845102 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:35:13.845056 4939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1c5e3dbc-2267-4ffa-b4eb-eb25bb89c2a8-tls-certs\" (UniqueName: \"kubernetes.io/secret/1c5e3dbc-2267-4ffa-b4eb-eb25bb89c2a8-tls-certs\") pod \"1c5e3dbc-2267-4ffa-b4eb-eb25bb89c2a8\" (UID: \"1c5e3dbc-2267-4ffa-b4eb-eb25bb89c2a8\") " Nov 03 01:35:13.845102 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:35:13.845051 4939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1c5e3dbc-2267-4ffa-b4eb-eb25bb89c2a8-home" pod "1c5e3dbc-2267-4ffa-b4eb-eb25bb89c2a8" (UID: "1c5e3dbc-2267-4ffa-b4eb-eb25bb89c2a8"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 01:35:13.845227 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:35:13.845103 4939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1c5e3dbc-2267-4ffa-b4eb-eb25bb89c2a8-tmp-dir" pod "1c5e3dbc-2267-4ffa-b4eb-eb25bb89c2a8" (UID: "1c5e3dbc-2267-4ffa-b4eb-eb25bb89c2a8"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 01:35:13.845352 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:35:13.845324 4939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1c5e3dbc-2267-4ffa-b4eb-eb25bb89c2a8-model-cache" pod "1c5e3dbc-2267-4ffa-b4eb-eb25bb89c2a8" (UID: "1c5e3dbc-2267-4ffa-b4eb-eb25bb89c2a8"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 01:35:13.846195 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:35:13.846167 4939 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1c5e3dbc-2267-4ffa-b4eb-eb25bb89c2a8-home\") on node \"ip-10-0-140-107.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:35:13.846319 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:35:13.846197 4939 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1c5e3dbc-2267-4ffa-b4eb-eb25bb89c2a8-tmp-dir\") on node \"ip-10-0-140-107.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:35:13.846319 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:35:13.846212 4939 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1c5e3dbc-2267-4ffa-b4eb-eb25bb89c2a8-model-cache\") on node \"ip-10-0-140-107.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:35:13.847650 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:35:13.847623 4939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1c5e3dbc-2267-4ffa-b4eb-eb25bb89c2a8-dshm" pod "1c5e3dbc-2267-4ffa-b4eb-eb25bb89c2a8" (UID: "1c5e3dbc-2267-4ffa-b4eb-eb25bb89c2a8"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 01:35:13.847786 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:35:13.847725 4939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1c5e3dbc-2267-4ffa-b4eb-eb25bb89c2a8-kube-api-access-99zzj" pod "1c5e3dbc-2267-4ffa-b4eb-eb25bb89c2a8" (UID: "1c5e3dbc-2267-4ffa-b4eb-eb25bb89c2a8"). InnerVolumeSpecName "kube-api-access-99zzj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 01:35:13.848584 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:35:13.848561 4939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1c5e3dbc-2267-4ffa-b4eb-eb25bb89c2a8-tls-certs" pod "1c5e3dbc-2267-4ffa-b4eb-eb25bb89c2a8" (UID: "1c5e3dbc-2267-4ffa-b4eb-eb25bb89c2a8"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 01:35:13.881374 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:35:13.881336 4939 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="32cc7c64-ec44-4a5d-815a-a378d840e3ff" path="/var/lib/kubelet/pods/32cc7c64-ec44-4a5d-815a-a378d840e3ff/volumes" Nov 03 01:35:13.948859 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:35:13.948495 4939 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-737bf2d8-openshift-default-56dc96b957-jhxgw"] Nov 03 01:35:13.950129 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:35:13.950101 4939 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1c5e3dbc-2267-4ffa-b4eb-eb25bb89c2a8-dshm\") on node \"ip-10-0-140-107.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:35:13.950129 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:35:13.950125 4939 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-99zzj\" (UniqueName: \"kubernetes.io/projected/1c5e3dbc-2267-4ffa-b4eb-eb25bb89c2a8-kube-api-access-99zzj\") on node \"ip-10-0-140-107.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:35:13.950282 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:35:13.950135 4939 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1c5e3dbc-2267-4ffa-b4eb-eb25bb89c2a8-tls-certs\") on node \"ip-10-0-140-107.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:35:13.954077 ip-10-0-140-107 kubenswrapper[4939]: W1103 01:35:13.954049 4939 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcc9b0d3f_c18a_4e8f_ab5b_1c9d9d1b244c.slice/crio-05909e1c44fc0a0adb074b952d5c1d20536ec2f60c7514160a053a62b272597e WatchSource:0}: Error finding container 05909e1c44fc0a0adb074b952d5c1d20536ec2f60c7514160a053a62b272597e: Status 404 returned error can't find the container with id 05909e1c44fc0a0adb074b952d5c1d20536ec2f60c7514160a053a62b272597e Nov 03 01:35:13.956501 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:35:13.956465 4939 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 01:35:13.956634 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:35:13.956570 4939 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 01:35:13.956715 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:35:13.956658 4939 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 01:35:13.982553 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:35:13.982311 4939 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-737bf2d8-openshift-default-56dc96b957-mwnjn"] Nov 03 01:35:13.982671 ip-10-0-140-107 kubenswrapper[4939]: W1103 01:35:13.982560 4939 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podae67a584_3143_44d0_a76f_97f60d25dd60.slice/crio-667bef9233812c01fbeb59ddb95cd5c23bc15f4c43fc8d95150a5663fd54ea18 WatchSource:0}: Error finding container 667bef9233812c01fbeb59ddb95cd5c23bc15f4c43fc8d95150a5663fd54ea18: Status 404 returned error can't find the container with id 667bef9233812c01fbeb59ddb95cd5c23bc15f4c43fc8d95150a5663fd54ea18 Nov 03 01:35:13.986189 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:35:13.986128 4939 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 01:35:13.986243 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:35:13.986208 4939 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 01:35:13.986279 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:35:13.986254 4939 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 01:35:14.191064 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:35:14.191014 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:35:14.615872 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:35:14.615829 4939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-737bf2d8-openshift-default-56dc96b957-mwnjn" event={"ID":"ae67a584-3143-44d0-a76f-97f60d25dd60","Type":"ContainerStarted","Data":"2e42335c8e69619af14f901ad8db624926d2018a34d85ff786f42c928145104e"} Nov 03 01:35:14.616246 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:35:14.616219 4939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-737bf2d8-openshift-default-56dc96b957-mwnjn" event={"ID":"ae67a584-3143-44d0-a76f-97f60d25dd60","Type":"ContainerStarted","Data":"667bef9233812c01fbeb59ddb95cd5c23bc15f4c43fc8d95150a5663fd54ea18"} Nov 03 01:35:14.670511 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:35:14.670431 4939 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-737bf2d8-openshift-default-56dc96b957-mwnjn" Nov 03 01:35:14.691000 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:35:14.687944 4939 generic.go:356] "Generic (PLEG): container finished" podID="1c5e3dbc-2267-4ffa-b4eb-eb25bb89c2a8" containerID="a4e42ae5cd492a07ddb012238b4a02885354f2001ffb7eb237fbdff3b5bf9014" exitCode=0 Nov 03 01:35:14.691000 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:35:14.688112 4939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-b4971a/test-model-b4971a-kserve-5f54798fc5-hc2bn" event={"ID":"1c5e3dbc-2267-4ffa-b4eb-eb25bb89c2a8","Type":"ContainerDied","Data":"a4e42ae5cd492a07ddb012238b4a02885354f2001ffb7eb237fbdff3b5bf9014"} Nov 03 01:35:14.691000 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:35:14.688156 4939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-b4971a/test-model-b4971a-kserve-5f54798fc5-hc2bn" event={"ID":"1c5e3dbc-2267-4ffa-b4eb-eb25bb89c2a8","Type":"ContainerDied","Data":"c2d32489c7ccfa8355cf96898a1ce1b66620ab8eb589e0847abeabd9317fdb99"} Nov 03 01:35:14.691000 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:35:14.688181 4939 scope.go:122] "RemoveContainer" containerID="a4e42ae5cd492a07ddb012238b4a02885354f2001ffb7eb237fbdff3b5bf9014" Nov 03 01:35:14.691000 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:35:14.688295 4939 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-b4971a/test-model-b4971a-kserve-5f54798fc5-hc2bn" Nov 03 01:35:14.709978 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:35:14.709450 4939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-737bf2d8-openshift-default-56dc96b957-jhxgw" event={"ID":"cc9b0d3f-c18a-4e8f-ab5b-1c9d9d1b244c","Type":"ContainerStarted","Data":"457336f8e52c97c94cbb3e22be387a007e1af0b7e67300ab70768191782e4685"} Nov 03 01:35:14.709978 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:35:14.709526 4939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-737bf2d8-openshift-default-56dc96b957-jhxgw" event={"ID":"cc9b0d3f-c18a-4e8f-ab5b-1c9d9d1b244c","Type":"ContainerStarted","Data":"05909e1c44fc0a0adb074b952d5c1d20536ec2f60c7514160a053a62b272597e"} Nov 03 01:35:14.774362 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:35:14.770454 4939 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-737bf2d8-openshift-default-56dc96b957-jhxgw" podStartSLOduration=1.7704350679999998 podStartE2EDuration="1.770435068s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 01:35:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 01:35:14.768373485 +0000 UTC m=+3263.032708431" watchObservedRunningTime="2026-11-03 01:35:14.770435068 +0000 UTC m=+3263.034770013" Nov 03 01:35:14.799952 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:35:14.798440 4939 scope.go:122] "RemoveContainer" containerID="a4e42ae5cd492a07ddb012238b4a02885354f2001ffb7eb237fbdff3b5bf9014" Nov 03 01:35:14.800156 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:35:14.800029 4939 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-737bf2d8-openshift-default-56dc96b957-mwnjn" podStartSLOduration=1.800012529 podStartE2EDuration="1.800012529s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 01:35:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 01:35:14.669238331 +0000 UTC m=+3262.933573281" watchObservedRunningTime="2026-11-03 01:35:14.800012529 +0000 UTC m=+3263.064347475" Nov 03 01:35:14.845517 ip-10-0-140-107 kubenswrapper[4939]: E1103 01:35:14.845420 4939 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a4e42ae5cd492a07ddb012238b4a02885354f2001ffb7eb237fbdff3b5bf9014\": container with ID starting with a4e42ae5cd492a07ddb012238b4a02885354f2001ffb7eb237fbdff3b5bf9014 not found: ID does not exist" containerID="a4e42ae5cd492a07ddb012238b4a02885354f2001ffb7eb237fbdff3b5bf9014" Nov 03 01:35:14.845517 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:35:14.845466 4939 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a4e42ae5cd492a07ddb012238b4a02885354f2001ffb7eb237fbdff3b5bf9014"} err="failed to get container status \"a4e42ae5cd492a07ddb012238b4a02885354f2001ffb7eb237fbdff3b5bf9014\": rpc error: code = NotFound desc = could not find container \"a4e42ae5cd492a07ddb012238b4a02885354f2001ffb7eb237fbdff3b5bf9014\": container with ID starting with a4e42ae5cd492a07ddb012238b4a02885354f2001ffb7eb237fbdff3b5bf9014 not found: ID does not exist" Nov 03 01:35:14.866930 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:35:14.866876 4939 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-b4971a/test-model-b4971a-kserve-5f54798fc5-hc2bn"] Nov 03 01:35:14.935936 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:35:14.914442 4939 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-b-b4971a/test-model-b4971a-kserve-5f54798fc5-hc2bn"] Nov 03 01:35:15.342605 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:35:15.342562 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:35:15.641629 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:35:15.641547 4939 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-737bf2d8-openshift-default-56dc96b957-jhxgw" Nov 03 01:35:15.642921 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:35:15.642883 4939 patch_prober.go:29] interesting pod/e2e-mt-737bf2d8-openshift-default-56dc96b957-jhxgw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.75:15021/healthz/ready\": dial tcp 10.132.1.75:15021: connect: connection refused" start-of-body= Nov 03 01:35:15.643057 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:35:15.642956 4939 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-737bf2d8-openshift-default-56dc96b957-jhxgw" podUID="cc9b0d3f-c18a-4e8f-ab5b-1c9d9d1b244c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.75:15021/healthz/ready\": dial tcp 10.132.1.75:15021: connect: connection refused" Nov 03 01:35:15.698107 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:35:15.697271 4939 patch_prober.go:29] interesting pod/e2e-mt-737bf2d8-openshift-default-56dc96b957-mwnjn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.76:15021/healthz/ready\": dial tcp 10.132.1.76:15021: connect: connection refused" start-of-body= Nov 03 01:35:15.698107 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:35:15.697352 4939 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-737bf2d8-openshift-default-56dc96b957-mwnjn" podUID="ae67a584-3143-44d0-a76f-97f60d25dd60" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.76:15021/healthz/ready\": dial tcp 10.132.1.76:15021: connect: connection refused" Nov 03 01:35:15.914551 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:35:15.914464 4939 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1c5e3dbc-2267-4ffa-b4eb-eb25bb89c2a8" path="/var/lib/kubelet/pods/1c5e3dbc-2267-4ffa-b4eb-eb25bb89c2a8/volumes" Nov 03 01:35:16.332688 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:35:16.332654 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:35:16.641772 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:35:16.641683 4939 patch_prober.go:29] interesting pod/e2e-mt-737bf2d8-openshift-default-56dc96b957-jhxgw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.75:15021/healthz/ready\": dial tcp 10.132.1.75:15021: connect: connection refused" start-of-body= Nov 03 01:35:16.641772 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:35:16.641744 4939 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-737bf2d8-openshift-default-56dc96b957-jhxgw" podUID="cc9b0d3f-c18a-4e8f-ab5b-1c9d9d1b244c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.75:15021/healthz/ready\": dial tcp 10.132.1.75:15021: connect: connection refused" Nov 03 01:35:16.681934 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:35:16.679042 4939 patch_prober.go:29] interesting pod/e2e-mt-737bf2d8-openshift-default-56dc96b957-mwnjn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.76:15021/healthz/ready\": dial tcp 10.132.1.76:15021: connect: connection refused" start-of-body= Nov 03 01:35:16.681934 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:35:16.679107 4939 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-737bf2d8-openshift-default-56dc96b957-mwnjn" podUID="ae67a584-3143-44d0-a76f-97f60d25dd60" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.76:15021/healthz/ready\": dial tcp 10.132.1.76:15021: connect: connection refused" Nov 03 01:35:17.373479 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:35:17.373393 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:35:17.642266 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:35:17.642176 4939 patch_prober.go:29] interesting pod/e2e-mt-737bf2d8-openshift-default-56dc96b957-jhxgw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.75:15021/healthz/ready\": dial tcp 10.132.1.75:15021: connect: connection refused" start-of-body= Nov 03 01:35:17.642488 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:35:17.642462 4939 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-737bf2d8-openshift-default-56dc96b957-jhxgw" podUID="cc9b0d3f-c18a-4e8f-ab5b-1c9d9d1b244c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.75:15021/healthz/ready\": dial tcp 10.132.1.75:15021: connect: connection refused" Nov 03 01:35:17.675832 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:35:17.675387 4939 patch_prober.go:29] interesting pod/e2e-mt-737bf2d8-openshift-default-56dc96b957-mwnjn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.76:15021/healthz/ready\": dial tcp 10.132.1.76:15021: connect: connection refused" start-of-body= Nov 03 01:35:17.675832 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:35:17.675449 4939 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-737bf2d8-openshift-default-56dc96b957-mwnjn" podUID="ae67a584-3143-44d0-a76f-97f60d25dd60" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.76:15021/healthz/ready\": dial tcp 10.132.1.76:15021: connect: connection refused" Nov 03 01:35:18.033486 ip-10-0-140-107 kubenswrapper[4939]: E1103 01:35:18.033405 4939 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-sthjh" containerName="extractor" Nov 03 01:35:18.431016 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:35:18.430983 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:35:18.641656 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:35:18.641618 4939 patch_prober.go:29] interesting pod/e2e-mt-737bf2d8-openshift-default-56dc96b957-jhxgw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.75:15021/healthz/ready\": dial tcp 10.132.1.75:15021: connect: connection refused" start-of-body= Nov 03 01:35:18.641864 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:35:18.641678 4939 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-737bf2d8-openshift-default-56dc96b957-jhxgw" podUID="cc9b0d3f-c18a-4e8f-ab5b-1c9d9d1b244c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.75:15021/healthz/ready\": dial tcp 10.132.1.75:15021: connect: connection refused" Nov 03 01:35:18.696218 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:35:18.696106 4939 patch_prober.go:29] interesting pod/e2e-mt-737bf2d8-openshift-default-56dc96b957-mwnjn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.76:15021/healthz/ready\": dial tcp 10.132.1.76:15021: connect: connection refused" start-of-body= Nov 03 01:35:18.696218 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:35:18.696175 4939 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-737bf2d8-openshift-default-56dc96b957-mwnjn" podUID="ae67a584-3143-44d0-a76f-97f60d25dd60" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.76:15021/healthz/ready\": dial tcp 10.132.1.76:15021: connect: connection refused" Nov 03 01:35:19.167931 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:35:19.167488 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:35:19.290197 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:35:19.290121 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:35:19.418633 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:35:19.418509 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:35:19.641281 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:35:19.641244 4939 patch_prober.go:29] interesting pod/e2e-mt-737bf2d8-openshift-default-56dc96b957-jhxgw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.75:15021/healthz/ready\": dial tcp 10.132.1.75:15021: connect: connection refused" start-of-body= Nov 03 01:35:19.641772 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:35:19.641302 4939 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-737bf2d8-openshift-default-56dc96b957-jhxgw" podUID="cc9b0d3f-c18a-4e8f-ab5b-1c9d9d1b244c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.75:15021/healthz/ready\": dial tcp 10.132.1.75:15021: connect: connection refused" Nov 03 01:35:19.668302 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:35:19.668266 4939 patch_prober.go:29] interesting pod/e2e-mt-737bf2d8-openshift-default-56dc96b957-mwnjn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.76:15021/healthz/ready\": dial tcp 10.132.1.76:15021: connect: connection refused" start-of-body= Nov 03 01:35:19.668584 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:35:19.668552 4939 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-737bf2d8-openshift-default-56dc96b957-mwnjn" podUID="ae67a584-3143-44d0-a76f-97f60d25dd60" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.76:15021/healthz/ready\": dial tcp 10.132.1.76:15021: connect: connection refused" Nov 03 01:35:19.797527 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:35:19.797436 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:35:20.233009 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:35:20.209044 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:35:20.641280 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:35:20.641247 4939 patch_prober.go:29] interesting pod/e2e-mt-737bf2d8-openshift-default-56dc96b957-jhxgw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.75:15021/healthz/ready\": dial tcp 10.132.1.75:15021: connect: connection refused" start-of-body= Nov 03 01:35:20.641495 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:35:20.641305 4939 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-737bf2d8-openshift-default-56dc96b957-jhxgw" podUID="cc9b0d3f-c18a-4e8f-ab5b-1c9d9d1b244c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.75:15021/healthz/ready\": dial tcp 10.132.1.75:15021: connect: connection refused" Nov 03 01:35:20.668112 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:35:20.668079 4939 patch_prober.go:29] interesting pod/e2e-mt-737bf2d8-openshift-default-56dc96b957-mwnjn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.76:15021/healthz/ready\": dial tcp 10.132.1.76:15021: connect: connection refused" start-of-body= Nov 03 01:35:20.668613 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:35:20.668580 4939 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-737bf2d8-openshift-default-56dc96b957-mwnjn" podUID="ae67a584-3143-44d0-a76f-97f60d25dd60" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.76:15021/healthz/ready\": dial tcp 10.132.1.76:15021: connect: connection refused" Nov 03 01:35:20.746299 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:35:20.746259 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:35:20.978470 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:35:20.978294 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:35:21.470100 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:35:21.470060 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:35:21.646089 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:35:21.646055 4939 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-737bf2d8-openshift-default-56dc96b957-jhxgw" Nov 03 01:35:21.646546 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:35:21.646527 4939 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-737bf2d8-openshift-default-56dc96b957-jhxgw" Nov 03 01:35:21.647281 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:35:21.647256 4939 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-737bf2d8-openshift-default-56dc96b957-jhxgw" Nov 03 01:35:21.672087 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:35:21.672061 4939 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-737bf2d8-openshift-default-56dc96b957-mwnjn" Nov 03 01:35:21.672357 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:35:21.672333 4939 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-737bf2d8-openshift-default-56dc96b957-mwnjn" Nov 03 01:35:21.673073 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:35:21.673055 4939 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-737bf2d8-openshift-default-56dc96b957-mwnjn" Nov 03 01:35:22.204608 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:35:22.204573 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:35:22.885220 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:35:22.885176 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:35:23.475705 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:35:23.475672 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:35:24.115753 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:35:24.115712 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:35:24.793869 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:35:24.793834 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:35:25.259768 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:35:25.259723 4939 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-6fec8a/test-model-6fec8a-kserve-68c5cd696f-kmgsd"] Nov 03 01:35:25.312841 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:35:25.312802 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:35:25.563460 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:35:25.563364 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:35:26.277402 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:35:26.277369 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:35:26.938806 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:35:26.938765 4939 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-b4971a-6c58f96447-rjn8w"] Nov 03 01:35:26.967758 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:35:26.967723 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:35:27.070633 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:35:27.070594 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:35:27.388743 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:35:27.388709 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:35:28.499547 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:35:28.499513 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:35:29.135959 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:35:29.132996 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:35:29.636201 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:35:29.636167 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:35:29.671685 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:35:29.671650 4939 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-a-d25759-openshift-default-d47fd9785-jcn5l"] Nov 03 01:35:29.671903 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:35:29.671882 4939 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-extmod-a-d25759-openshift-default-d47fd9785-jcn5l" podUID="93ab25a5-4bed-4d84-9d62-640cea8d4047" containerName="istio-proxy" containerID="cri-o://70d4bb7ccd830af7d7e57b795e8429cff2c52c2fc6329d04c9360c781027bcba" gracePeriod=30 Nov 03 01:35:30.111593 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:35:30.111554 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:35:30.894353 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:35:30.894320 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:35:31.525403 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:35:31.525376 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:35:31.939688 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:35:31.939644 4939 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-b4971a-6c58f96447-rjn8w" podUID="6712b32f-f3bd-4917-9e5e-101cf604625c" containerName="payload-pre-processing" containerID="cri-o://384febeac8280063b68f71ab034887b1d84a8e149d013ded2756f3066aeecdf7" gracePeriod=25 Nov 03 01:35:32.263742 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:35:32.263639 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:35:32.819182 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:35:32.819148 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:35:33.089326 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:35:33.089292 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:35:33.217792 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:35:33.217756 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:35:33.344156 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:35:33.344071 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:35:33.620171 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:35:33.620030 4939 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-b4971a-openshift-default-5dc48765-8rn6x"] Nov 03 01:35:33.620359 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:35:33.620313 4939 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-b-b4971a-openshift-default-5dc48765-8rn6x" podUID="febb83d9-3a57-40a5-b972-c45611313469" containerName="istio-proxy" containerID="cri-o://9f8d2d055b7c0d16ceed10a149a9af6f53d1436f29e72f3eb22bc4949a225ea7" gracePeriod=30 Nov 03 01:35:33.674694 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:35:33.674659 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:35:34.121122 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:35:34.121084 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:35:34.904588 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:35:34.904499 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:35:35.029685 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:35:35.029402 4939 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-b4971a-6c58f96447-rjn8w" podUID="6712b32f-f3bd-4917-9e5e-101cf604625c" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 03 01:35:35.044436 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:35:35.044410 4939 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-d25759-openshift-default-d47fd9785-jcn5l" Nov 03 01:35:35.125590 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:35:35.125549 4939 generic.go:356] "Generic (PLEG): container finished" podID="93ab25a5-4bed-4d84-9d62-640cea8d4047" containerID="70d4bb7ccd830af7d7e57b795e8429cff2c52c2fc6329d04c9360c781027bcba" exitCode=0 Nov 03 01:35:35.126039 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:35:35.125637 4939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-d25759-openshift-default-d47fd9785-jcn5l" event={"ID":"93ab25a5-4bed-4d84-9d62-640cea8d4047","Type":"ContainerDied","Data":"70d4bb7ccd830af7d7e57b795e8429cff2c52c2fc6329d04c9360c781027bcba"} Nov 03 01:35:35.126039 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:35:35.125692 4939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-d25759-openshift-default-d47fd9785-jcn5l" event={"ID":"93ab25a5-4bed-4d84-9d62-640cea8d4047","Type":"ContainerDied","Data":"fb003dd49ca60fac8ee17dceebe96fa5a85be971b8b81e0cbd8bcf49e725914b"} Nov 03 01:35:35.126039 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:35:35.125722 4939 scope.go:122] "RemoveContainer" containerID="70d4bb7ccd830af7d7e57b795e8429cff2c52c2fc6329d04c9360c781027bcba" Nov 03 01:35:35.126039 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:35:35.125900 4939 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-d25759-openshift-default-d47fd9785-jcn5l" Nov 03 01:35:35.138618 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:35:35.138596 4939 scope.go:122] "RemoveContainer" containerID="e6d4ddc1ac83d9addd06be80c75b25054577015f20147ea1379903004987cb75" Nov 03 01:35:35.149299 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:35:35.149276 4939 scope.go:122] "RemoveContainer" containerID="70d4bb7ccd830af7d7e57b795e8429cff2c52c2fc6329d04c9360c781027bcba" Nov 03 01:35:35.149575 ip-10-0-140-107 kubenswrapper[4939]: E1103 01:35:35.149550 4939 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"70d4bb7ccd830af7d7e57b795e8429cff2c52c2fc6329d04c9360c781027bcba\": container with ID starting with 70d4bb7ccd830af7d7e57b795e8429cff2c52c2fc6329d04c9360c781027bcba not found: ID does not exist" containerID="70d4bb7ccd830af7d7e57b795e8429cff2c52c2fc6329d04c9360c781027bcba" Nov 03 01:35:35.149652 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:35:35.149583 4939 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"70d4bb7ccd830af7d7e57b795e8429cff2c52c2fc6329d04c9360c781027bcba"} err="failed to get container status \"70d4bb7ccd830af7d7e57b795e8429cff2c52c2fc6329d04c9360c781027bcba\": rpc error: code = NotFound desc = could not find container \"70d4bb7ccd830af7d7e57b795e8429cff2c52c2fc6329d04c9360c781027bcba\": container with ID starting with 70d4bb7ccd830af7d7e57b795e8429cff2c52c2fc6329d04c9360c781027bcba not found: ID does not exist" Nov 03 01:35:35.149652 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:35:35.149599 4939 scope.go:122] "RemoveContainer" containerID="e6d4ddc1ac83d9addd06be80c75b25054577015f20147ea1379903004987cb75" Nov 03 01:35:35.149865 ip-10-0-140-107 kubenswrapper[4939]: E1103 01:35:35.149845 4939 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e6d4ddc1ac83d9addd06be80c75b25054577015f20147ea1379903004987cb75\": container with ID starting with e6d4ddc1ac83d9addd06be80c75b25054577015f20147ea1379903004987cb75 not found: ID does not exist" containerID="e6d4ddc1ac83d9addd06be80c75b25054577015f20147ea1379903004987cb75" Nov 03 01:35:35.149938 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:35:35.149870 4939 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e6d4ddc1ac83d9addd06be80c75b25054577015f20147ea1379903004987cb75"} err="failed to get container status \"e6d4ddc1ac83d9addd06be80c75b25054577015f20147ea1379903004987cb75\": rpc error: code = NotFound desc = could not find container \"e6d4ddc1ac83d9addd06be80c75b25054577015f20147ea1379903004987cb75\": container with ID starting with e6d4ddc1ac83d9addd06be80c75b25054577015f20147ea1379903004987cb75 not found: ID does not exist" Nov 03 01:35:35.194527 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:35:35.194451 4939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/93ab25a5-4bed-4d84-9d62-640cea8d4047-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/93ab25a5-4bed-4d84-9d62-640cea8d4047-istiod-ca-cert\") pod \"93ab25a5-4bed-4d84-9d62-640cea8d4047\" (UID: \"93ab25a5-4bed-4d84-9d62-640cea8d4047\") " Nov 03 01:35:35.194739 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:35:35.194537 4939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/93ab25a5-4bed-4d84-9d62-640cea8d4047-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/93ab25a5-4bed-4d84-9d62-640cea8d4047-workload-certs\") pod \"93ab25a5-4bed-4d84-9d62-640cea8d4047\" (UID: \"93ab25a5-4bed-4d84-9d62-640cea8d4047\") " Nov 03 01:35:35.194739 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:35:35.194615 4939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/93ab25a5-4bed-4d84-9d62-640cea8d4047-kube-api-access-6lbbx\" (UniqueName: \"kubernetes.io/projected/93ab25a5-4bed-4d84-9d62-640cea8d4047-kube-api-access-6lbbx\") pod \"93ab25a5-4bed-4d84-9d62-640cea8d4047\" (UID: \"93ab25a5-4bed-4d84-9d62-640cea8d4047\") " Nov 03 01:35:35.194739 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:35:35.194718 4939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/93ab25a5-4bed-4d84-9d62-640cea8d4047-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/93ab25a5-4bed-4d84-9d62-640cea8d4047-credential-socket\") pod \"93ab25a5-4bed-4d84-9d62-640cea8d4047\" (UID: \"93ab25a5-4bed-4d84-9d62-640cea8d4047\") " Nov 03 01:35:35.194978 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:35:35.194830 4939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/93ab25a5-4bed-4d84-9d62-640cea8d4047-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/93ab25a5-4bed-4d84-9d62-640cea8d4047-istio-data\") pod \"93ab25a5-4bed-4d84-9d62-640cea8d4047\" (UID: \"93ab25a5-4bed-4d84-9d62-640cea8d4047\") " Nov 03 01:35:35.194978 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:35:35.194956 4939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/93ab25a5-4bed-4d84-9d62-640cea8d4047-istiod-ca-cert" pod "93ab25a5-4bed-4d84-9d62-640cea8d4047" (UID: "93ab25a5-4bed-4d84-9d62-640cea8d4047"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 01:35:35.195117 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:35:35.194836 4939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/93ab25a5-4bed-4d84-9d62-640cea8d4047-workload-certs" pod "93ab25a5-4bed-4d84-9d62-640cea8d4047" (UID: "93ab25a5-4bed-4d84-9d62-640cea8d4047"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 01:35:35.195117 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:35:35.195085 4939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/93ab25a5-4bed-4d84-9d62-640cea8d4047-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/93ab25a5-4bed-4d84-9d62-640cea8d4047-istio-podinfo\") pod \"93ab25a5-4bed-4d84-9d62-640cea8d4047\" (UID: \"93ab25a5-4bed-4d84-9d62-640cea8d4047\") " Nov 03 01:35:35.195230 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:35:35.195154 4939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/93ab25a5-4bed-4d84-9d62-640cea8d4047-istio-data" pod "93ab25a5-4bed-4d84-9d62-640cea8d4047" (UID: "93ab25a5-4bed-4d84-9d62-640cea8d4047"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 01:35:35.195230 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:35:35.195168 4939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/93ab25a5-4bed-4d84-9d62-640cea8d4047-credential-socket" pod "93ab25a5-4bed-4d84-9d62-640cea8d4047" (UID: "93ab25a5-4bed-4d84-9d62-640cea8d4047"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 01:35:35.195347 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:35:35.195239 4939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/93ab25a5-4bed-4d84-9d62-640cea8d4047-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/93ab25a5-4bed-4d84-9d62-640cea8d4047-workload-socket\") pod \"93ab25a5-4bed-4d84-9d62-640cea8d4047\" (UID: \"93ab25a5-4bed-4d84-9d62-640cea8d4047\") " Nov 03 01:35:35.195347 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:35:35.195281 4939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/93ab25a5-4bed-4d84-9d62-640cea8d4047-istio-token\" (UniqueName: \"kubernetes.io/projected/93ab25a5-4bed-4d84-9d62-640cea8d4047-istio-token\") pod \"93ab25a5-4bed-4d84-9d62-640cea8d4047\" (UID: \"93ab25a5-4bed-4d84-9d62-640cea8d4047\") " Nov 03 01:35:35.195347 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:35:35.195315 4939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/93ab25a5-4bed-4d84-9d62-640cea8d4047-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/93ab25a5-4bed-4d84-9d62-640cea8d4047-istio-envoy\") pod \"93ab25a5-4bed-4d84-9d62-640cea8d4047\" (UID: \"93ab25a5-4bed-4d84-9d62-640cea8d4047\") " Nov 03 01:35:35.195548 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:35:35.195527 4939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/93ab25a5-4bed-4d84-9d62-640cea8d4047-workload-socket" pod "93ab25a5-4bed-4d84-9d62-640cea8d4047" (UID: "93ab25a5-4bed-4d84-9d62-640cea8d4047"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 01:35:35.196115 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:35:35.196092 4939 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/93ab25a5-4bed-4d84-9d62-640cea8d4047-istio-data\") on node \"ip-10-0-140-107.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:35:35.196206 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:35:35.196117 4939 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/93ab25a5-4bed-4d84-9d62-640cea8d4047-workload-socket\") on node \"ip-10-0-140-107.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:35:35.196206 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:35:35.196133 4939 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/93ab25a5-4bed-4d84-9d62-640cea8d4047-istiod-ca-cert\") on node \"ip-10-0-140-107.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:35:35.196206 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:35:35.196146 4939 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/93ab25a5-4bed-4d84-9d62-640cea8d4047-workload-certs\") on node \"ip-10-0-140-107.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:35:35.196206 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:35:35.196159 4939 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/93ab25a5-4bed-4d84-9d62-640cea8d4047-credential-socket\") on node \"ip-10-0-140-107.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:35:35.197221 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:35:35.197198 4939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/93ab25a5-4bed-4d84-9d62-640cea8d4047-istio-podinfo" pod "93ab25a5-4bed-4d84-9d62-640cea8d4047" (UID: "93ab25a5-4bed-4d84-9d62-640cea8d4047"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 01:35:35.197479 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:35:35.197446 4939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/93ab25a5-4bed-4d84-9d62-640cea8d4047-istio-envoy" pod "93ab25a5-4bed-4d84-9d62-640cea8d4047" (UID: "93ab25a5-4bed-4d84-9d62-640cea8d4047"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 01:35:35.197598 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:35:35.197564 4939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/93ab25a5-4bed-4d84-9d62-640cea8d4047-kube-api-access-6lbbx" pod "93ab25a5-4bed-4d84-9d62-640cea8d4047" (UID: "93ab25a5-4bed-4d84-9d62-640cea8d4047"). InnerVolumeSpecName "kube-api-access-6lbbx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 01:35:35.197863 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:35:35.197831 4939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/93ab25a5-4bed-4d84-9d62-640cea8d4047-istio-token" pod "93ab25a5-4bed-4d84-9d62-640cea8d4047" (UID: "93ab25a5-4bed-4d84-9d62-640cea8d4047"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 01:35:35.298128 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:35:35.298052 4939 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6lbbx\" (UniqueName: \"kubernetes.io/projected/93ab25a5-4bed-4d84-9d62-640cea8d4047-kube-api-access-6lbbx\") on node \"ip-10-0-140-107.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:35:35.298128 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:35:35.298082 4939 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/93ab25a5-4bed-4d84-9d62-640cea8d4047-istio-podinfo\") on node \"ip-10-0-140-107.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:35:35.298128 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:35:35.298093 4939 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/93ab25a5-4bed-4d84-9d62-640cea8d4047-istio-token\") on node \"ip-10-0-140-107.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:35:35.298128 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:35:35.298103 4939 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/93ab25a5-4bed-4d84-9d62-640cea8d4047-istio-envoy\") on node \"ip-10-0-140-107.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:35:35.366529 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:35:35.366494 4939 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-w2-531447/sub-test-model-531447-kserve-65cffbb7c-h6b59"] Nov 03 01:35:35.367516 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:35:35.367490 4939 cpu_manager.go:414] "containerMap: removing container" podUID="93ab25a5-4bed-4d84-9d62-640cea8d4047" containerName="istio-proxy" Nov 03 01:35:35.367516 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:35:35.367514 4939 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="93ab25a5-4bed-4d84-9d62-640cea8d4047" containerName="istio-proxy" Nov 03 01:35:35.367716 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:35:35.367566 4939 cpu_manager.go:414] "containerMap: removing container" podUID="93ab25a5-4bed-4d84-9d62-640cea8d4047" containerName="istio-proxy" Nov 03 01:35:35.367716 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:35:35.367576 4939 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="93ab25a5-4bed-4d84-9d62-640cea8d4047" containerName="istio-proxy" Nov 03 01:35:35.367716 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:35:35.367595 4939 cpu_manager.go:414] "containerMap: removing container" podUID="1c5e3dbc-2267-4ffa-b4eb-eb25bb89c2a8" containerName="main" Nov 03 01:35:35.367716 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:35:35.367603 4939 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1c5e3dbc-2267-4ffa-b4eb-eb25bb89c2a8" containerName="main" Nov 03 01:35:35.367942 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:35:35.367875 4939 memory_manager.go:365] "RemoveStaleState removing state" podUID="93ab25a5-4bed-4d84-9d62-640cea8d4047" containerName="istio-proxy" Nov 03 01:35:35.367942 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:35:35.367900 4939 memory_manager.go:365] "RemoveStaleState removing state" podUID="1c5e3dbc-2267-4ffa-b4eb-eb25bb89c2a8" containerName="main" Nov 03 01:35:35.375778 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:35:35.375752 4939 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w2-531447/sub-test-model-531447-kserve-65cffbb7c-h6b59" Nov 03 01:35:35.381620 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:35:35.381590 4939 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w2-531447/sub-test-model-531447-kserve-65cffbb7c-h6b59"] Nov 03 01:35:35.382570 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:35:35.382520 4939 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w2-531447\"/\"sub-test-model-531447-kserve-self-signed-certs\"" Nov 03 01:35:35.383222 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:35:35.383198 4939 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w2-531447\"/\"kube-root-ca.crt\"" Nov 03 01:35:35.383755 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:35:35.383734 4939 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w2-531447\"/\"openshift-service-ca.crt\"" Nov 03 01:35:35.383849 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:35:35.383797 4939 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w2-531447\"/\"default-dockercfg-wp9tl\"" Nov 03 01:35:35.428429 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:35:35.428401 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:35:35.452452 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:35:35.452423 4939 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-a-d25759-openshift-default-d47fd9785-jcn5l"] Nov 03 01:35:35.456730 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:35:35.456704 4939 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-extmod-a-d25759-openshift-default-d47fd9785-jcn5l"] Nov 03 01:35:35.502353 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:35:35.502326 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/de222447-f690-4b85-bdee-b419f81767eb-dshm\") pod \"sub-test-model-531447-kserve-65cffbb7c-h6b59\" (UID: \"de222447-f690-4b85-bdee-b419f81767eb\") " pod="ai-tenant-e2e-shared-a-w2-531447/sub-test-model-531447-kserve-65cffbb7c-h6b59" Nov 03 01:35:35.502479 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:35:35.502378 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/de222447-f690-4b85-bdee-b419f81767eb-tls-certs\") pod \"sub-test-model-531447-kserve-65cffbb7c-h6b59\" (UID: \"de222447-f690-4b85-bdee-b419f81767eb\") " pod="ai-tenant-e2e-shared-a-w2-531447/sub-test-model-531447-kserve-65cffbb7c-h6b59" Nov 03 01:35:35.502529 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:35:35.502483 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/de222447-f690-4b85-bdee-b419f81767eb-home\") pod \"sub-test-model-531447-kserve-65cffbb7c-h6b59\" (UID: \"de222447-f690-4b85-bdee-b419f81767eb\") " pod="ai-tenant-e2e-shared-a-w2-531447/sub-test-model-531447-kserve-65cffbb7c-h6b59" Nov 03 01:35:35.502572 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:35:35.502560 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ssj4g\" (UniqueName: \"kubernetes.io/projected/de222447-f690-4b85-bdee-b419f81767eb-kube-api-access-ssj4g\") pod \"sub-test-model-531447-kserve-65cffbb7c-h6b59\" (UID: \"de222447-f690-4b85-bdee-b419f81767eb\") " pod="ai-tenant-e2e-shared-a-w2-531447/sub-test-model-531447-kserve-65cffbb7c-h6b59" Nov 03 01:35:35.502768 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:35:35.502752 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/de222447-f690-4b85-bdee-b419f81767eb-tmp-dir\") pod \"sub-test-model-531447-kserve-65cffbb7c-h6b59\" (UID: \"de222447-f690-4b85-bdee-b419f81767eb\") " pod="ai-tenant-e2e-shared-a-w2-531447/sub-test-model-531447-kserve-65cffbb7c-h6b59" Nov 03 01:35:35.502809 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:35:35.502789 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/de222447-f690-4b85-bdee-b419f81767eb-model-cache\") pod \"sub-test-model-531447-kserve-65cffbb7c-h6b59\" (UID: \"de222447-f690-4b85-bdee-b419f81767eb\") " pod="ai-tenant-e2e-shared-a-w2-531447/sub-test-model-531447-kserve-65cffbb7c-h6b59" Nov 03 01:35:35.590891 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:35:35.590863 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:35:35.604515 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:35:35.604471 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/de222447-f690-4b85-bdee-b419f81767eb-dshm\") pod \"sub-test-model-531447-kserve-65cffbb7c-h6b59\" (UID: \"de222447-f690-4b85-bdee-b419f81767eb\") " pod="ai-tenant-e2e-shared-a-w2-531447/sub-test-model-531447-kserve-65cffbb7c-h6b59" Nov 03 01:35:35.604672 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:35:35.604557 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/de222447-f690-4b85-bdee-b419f81767eb-tls-certs\") pod \"sub-test-model-531447-kserve-65cffbb7c-h6b59\" (UID: \"de222447-f690-4b85-bdee-b419f81767eb\") " pod="ai-tenant-e2e-shared-a-w2-531447/sub-test-model-531447-kserve-65cffbb7c-h6b59" Nov 03 01:35:35.604672 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:35:35.604626 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/de222447-f690-4b85-bdee-b419f81767eb-home\") pod \"sub-test-model-531447-kserve-65cffbb7c-h6b59\" (UID: \"de222447-f690-4b85-bdee-b419f81767eb\") " pod="ai-tenant-e2e-shared-a-w2-531447/sub-test-model-531447-kserve-65cffbb7c-h6b59" Nov 03 01:35:35.604792 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:35:35.604686 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ssj4g\" (UniqueName: \"kubernetes.io/projected/de222447-f690-4b85-bdee-b419f81767eb-kube-api-access-ssj4g\") pod \"sub-test-model-531447-kserve-65cffbb7c-h6b59\" (UID: \"de222447-f690-4b85-bdee-b419f81767eb\") " pod="ai-tenant-e2e-shared-a-w2-531447/sub-test-model-531447-kserve-65cffbb7c-h6b59" Nov 03 01:35:35.604962 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:35:35.604935 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/de222447-f690-4b85-bdee-b419f81767eb-tmp-dir\") pod \"sub-test-model-531447-kserve-65cffbb7c-h6b59\" (UID: \"de222447-f690-4b85-bdee-b419f81767eb\") " pod="ai-tenant-e2e-shared-a-w2-531447/sub-test-model-531447-kserve-65cffbb7c-h6b59" Nov 03 01:35:35.605044 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:35:35.604988 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/de222447-f690-4b85-bdee-b419f81767eb-model-cache\") pod \"sub-test-model-531447-kserve-65cffbb7c-h6b59\" (UID: \"de222447-f690-4b85-bdee-b419f81767eb\") " pod="ai-tenant-e2e-shared-a-w2-531447/sub-test-model-531447-kserve-65cffbb7c-h6b59" Nov 03 01:35:35.605105 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:35:35.605064 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/de222447-f690-4b85-bdee-b419f81767eb-home\") pod \"sub-test-model-531447-kserve-65cffbb7c-h6b59\" (UID: \"de222447-f690-4b85-bdee-b419f81767eb\") " pod="ai-tenant-e2e-shared-a-w2-531447/sub-test-model-531447-kserve-65cffbb7c-h6b59" Nov 03 01:35:35.605245 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:35:35.605228 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/de222447-f690-4b85-bdee-b419f81767eb-tmp-dir\") pod \"sub-test-model-531447-kserve-65cffbb7c-h6b59\" (UID: \"de222447-f690-4b85-bdee-b419f81767eb\") " pod="ai-tenant-e2e-shared-a-w2-531447/sub-test-model-531447-kserve-65cffbb7c-h6b59" Nov 03 01:35:35.605317 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:35:35.605301 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/de222447-f690-4b85-bdee-b419f81767eb-model-cache\") pod \"sub-test-model-531447-kserve-65cffbb7c-h6b59\" (UID: \"de222447-f690-4b85-bdee-b419f81767eb\") " pod="ai-tenant-e2e-shared-a-w2-531447/sub-test-model-531447-kserve-65cffbb7c-h6b59" Nov 03 01:35:35.607394 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:35:35.607367 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/de222447-f690-4b85-bdee-b419f81767eb-dshm\") pod \"sub-test-model-531447-kserve-65cffbb7c-h6b59\" (UID: \"de222447-f690-4b85-bdee-b419f81767eb\") " pod="ai-tenant-e2e-shared-a-w2-531447/sub-test-model-531447-kserve-65cffbb7c-h6b59" Nov 03 01:35:35.607786 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:35:35.607766 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/de222447-f690-4b85-bdee-b419f81767eb-tls-certs\") pod \"sub-test-model-531447-kserve-65cffbb7c-h6b59\" (UID: \"de222447-f690-4b85-bdee-b419f81767eb\") " pod="ai-tenant-e2e-shared-a-w2-531447/sub-test-model-531447-kserve-65cffbb7c-h6b59" Nov 03 01:35:35.631938 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:35:35.624141 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ssj4g\" (UniqueName: \"kubernetes.io/projected/de222447-f690-4b85-bdee-b419f81767eb-kube-api-access-ssj4g\") pod \"sub-test-model-531447-kserve-65cffbb7c-h6b59\" (UID: \"de222447-f690-4b85-bdee-b419f81767eb\") " pod="ai-tenant-e2e-shared-a-w2-531447/sub-test-model-531447-kserve-65cffbb7c-h6b59" Nov 03 01:35:35.707930 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:35:35.707814 4939 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w2-531447/sub-test-model-531447-kserve-65cffbb7c-h6b59" Nov 03 01:35:35.879903 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:35:35.879812 4939 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="93ab25a5-4bed-4d84-9d62-640cea8d4047" path="/var/lib/kubelet/pods/93ab25a5-4bed-4d84-9d62-640cea8d4047/volumes" Nov 03 01:35:36.056824 ip-10-0-140-107 kubenswrapper[4939]: W1103 01:35:36.056773 4939 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podde222447_f690_4b85_bdee_b419f81767eb.slice/crio-e2a6aeecc5071b6ef4960bb7c5eee1d850e7e9597b0a1e743bea8135d49833a6 WatchSource:0}: Error finding container e2a6aeecc5071b6ef4960bb7c5eee1d850e7e9597b0a1e743bea8135d49833a6: Status 404 returned error can't find the container with id e2a6aeecc5071b6ef4960bb7c5eee1d850e7e9597b0a1e743bea8135d49833a6 Nov 03 01:35:36.058486 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:35:36.058445 4939 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w2-531447/sub-test-model-531447-kserve-65cffbb7c-h6b59"] Nov 03 01:35:36.137420 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:35:36.137388 4939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w2-531447/sub-test-model-531447-kserve-65cffbb7c-h6b59" event={"ID":"de222447-f690-4b85-bdee-b419f81767eb","Type":"ContainerStarted","Data":"e2a6aeecc5071b6ef4960bb7c5eee1d850e7e9597b0a1e743bea8135d49833a6"} Nov 03 01:35:36.384539 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:35:36.384443 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:35:37.066700 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:35:37.066665 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:35:37.149530 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:35:37.149495 4939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w2-531447/sub-test-model-531447-kserve-65cffbb7c-h6b59" event={"ID":"de222447-f690-4b85-bdee-b419f81767eb","Type":"ContainerStarted","Data":"4e0a1360f81ad320881890154d63a557f4a5c96ef7afc03eb7d8c2141d13d420"} Nov 03 01:35:37.170496 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:35:37.170450 4939 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-w2-531447/sub-test-model-531447-kserve-65cffbb7c-h6b59" podStartSLOduration=2.1704382 podStartE2EDuration="2.1704382s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 01:35:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 01:35:37.165093187 +0000 UTC m=+3285.429428132" watchObservedRunningTime="2026-11-03 01:35:37.1704382 +0000 UTC m=+3285.434773145" Nov 03 01:35:37.708699 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:35:37.708660 4939 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-w2-531447/sub-test-model-531447-kserve-65cffbb7c-h6b59" Nov 03 01:35:37.753262 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:35:37.753230 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:35:38.108022 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:35:38.107985 4939 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-5dab6a/test-model-5dab6a-kserve-5c4f47747c-z44hb"] Nov 03 01:35:38.186014 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:35:38.185980 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:35:38.673247 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:35:38.673214 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:35:38.788857 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:35:38.788805 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:35:38.932316 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:35:38.932257 4939 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-b4971a-openshift-default-5dc48765-8rn6x" Nov 03 01:35:38.975804 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:35:38.975774 4939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/febb83d9-3a57-40a5-b972-c45611313469-kube-api-access-l8qt8\" (UniqueName: \"kubernetes.io/projected/febb83d9-3a57-40a5-b972-c45611313469-kube-api-access-l8qt8\") pod \"febb83d9-3a57-40a5-b972-c45611313469\" (UID: \"febb83d9-3a57-40a5-b972-c45611313469\") " Nov 03 01:35:38.976037 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:35:38.975878 4939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/febb83d9-3a57-40a5-b972-c45611313469-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/febb83d9-3a57-40a5-b972-c45611313469-workload-socket\") pod \"febb83d9-3a57-40a5-b972-c45611313469\" (UID: \"febb83d9-3a57-40a5-b972-c45611313469\") " Nov 03 01:35:38.976037 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:35:38.975996 4939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/febb83d9-3a57-40a5-b972-c45611313469-istio-token\" (UniqueName: \"kubernetes.io/projected/febb83d9-3a57-40a5-b972-c45611313469-istio-token\") pod \"febb83d9-3a57-40a5-b972-c45611313469\" (UID: \"febb83d9-3a57-40a5-b972-c45611313469\") " Nov 03 01:35:38.976195 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:35:38.976112 4939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/febb83d9-3a57-40a5-b972-c45611313469-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/febb83d9-3a57-40a5-b972-c45611313469-workload-certs\") pod \"febb83d9-3a57-40a5-b972-c45611313469\" (UID: \"febb83d9-3a57-40a5-b972-c45611313469\") " Nov 03 01:35:38.976195 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:35:38.976171 4939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/febb83d9-3a57-40a5-b972-c45611313469-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/febb83d9-3a57-40a5-b972-c45611313469-istio-envoy\") pod \"febb83d9-3a57-40a5-b972-c45611313469\" (UID: \"febb83d9-3a57-40a5-b972-c45611313469\") " Nov 03 01:35:38.976309 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:35:38.976252 4939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/febb83d9-3a57-40a5-b972-c45611313469-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/febb83d9-3a57-40a5-b972-c45611313469-istiod-ca-cert\") pod \"febb83d9-3a57-40a5-b972-c45611313469\" (UID: \"febb83d9-3a57-40a5-b972-c45611313469\") " Nov 03 01:35:38.976416 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:35:38.976398 4939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/febb83d9-3a57-40a5-b972-c45611313469-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/febb83d9-3a57-40a5-b972-c45611313469-istio-podinfo\") pod \"febb83d9-3a57-40a5-b972-c45611313469\" (UID: \"febb83d9-3a57-40a5-b972-c45611313469\") " Nov 03 01:35:38.977373 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:35:38.977339 4939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/febb83d9-3a57-40a5-b972-c45611313469-workload-certs" pod "febb83d9-3a57-40a5-b972-c45611313469" (UID: "febb83d9-3a57-40a5-b972-c45611313469"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 01:35:38.977517 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:35:38.977384 4939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/febb83d9-3a57-40a5-b972-c45611313469-istiod-ca-cert" pod "febb83d9-3a57-40a5-b972-c45611313469" (UID: "febb83d9-3a57-40a5-b972-c45611313469"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 01:35:38.977517 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:35:38.977476 4939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/febb83d9-3a57-40a5-b972-c45611313469-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/febb83d9-3a57-40a5-b972-c45611313469-credential-socket\") pod \"febb83d9-3a57-40a5-b972-c45611313469\" (UID: \"febb83d9-3a57-40a5-b972-c45611313469\") " Nov 03 01:35:38.977653 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:35:38.977557 4939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/febb83d9-3a57-40a5-b972-c45611313469-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/febb83d9-3a57-40a5-b972-c45611313469-istio-data\") pod \"febb83d9-3a57-40a5-b972-c45611313469\" (UID: \"febb83d9-3a57-40a5-b972-c45611313469\") " Nov 03 01:35:38.977822 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:35:38.977794 4939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/febb83d9-3a57-40a5-b972-c45611313469-credential-socket" pod "febb83d9-3a57-40a5-b972-c45611313469" (UID: "febb83d9-3a57-40a5-b972-c45611313469"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 01:35:38.978118 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:35:38.978093 4939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/febb83d9-3a57-40a5-b972-c45611313469-istio-data" pod "febb83d9-3a57-40a5-b972-c45611313469" (UID: "febb83d9-3a57-40a5-b972-c45611313469"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 01:35:38.978206 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:35:38.978127 4939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/febb83d9-3a57-40a5-b972-c45611313469-workload-socket" pod "febb83d9-3a57-40a5-b972-c45611313469" (UID: "febb83d9-3a57-40a5-b972-c45611313469"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 01:35:38.978540 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:35:38.978515 4939 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/febb83d9-3a57-40a5-b972-c45611313469-istio-data\") on node \"ip-10-0-140-107.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:35:38.978664 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:35:38.978548 4939 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/febb83d9-3a57-40a5-b972-c45611313469-workload-socket\") on node \"ip-10-0-140-107.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:35:38.978664 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:35:38.978564 4939 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/febb83d9-3a57-40a5-b972-c45611313469-workload-certs\") on node \"ip-10-0-140-107.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:35:38.978664 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:35:38.978578 4939 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/febb83d9-3a57-40a5-b972-c45611313469-istiod-ca-cert\") on node \"ip-10-0-140-107.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:35:38.978664 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:35:38.978600 4939 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/febb83d9-3a57-40a5-b972-c45611313469-credential-socket\") on node \"ip-10-0-140-107.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:35:38.980599 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:35:38.980493 4939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/febb83d9-3a57-40a5-b972-c45611313469-istio-envoy" pod "febb83d9-3a57-40a5-b972-c45611313469" (UID: "febb83d9-3a57-40a5-b972-c45611313469"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 01:35:38.980599 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:35:38.981458 4939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/febb83d9-3a57-40a5-b972-c45611313469-kube-api-access-l8qt8" pod "febb83d9-3a57-40a5-b972-c45611313469" (UID: "febb83d9-3a57-40a5-b972-c45611313469"). InnerVolumeSpecName "kube-api-access-l8qt8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 01:35:38.980599 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:35:38.981497 4939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/febb83d9-3a57-40a5-b972-c45611313469-istio-token" pod "febb83d9-3a57-40a5-b972-c45611313469" (UID: "febb83d9-3a57-40a5-b972-c45611313469"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 01:35:38.983190 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:35:38.983158 4939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/febb83d9-3a57-40a5-b972-c45611313469-istio-podinfo" pod "febb83d9-3a57-40a5-b972-c45611313469" (UID: "febb83d9-3a57-40a5-b972-c45611313469"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 01:35:39.080921 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:35:39.080875 4939 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-l8qt8\" (UniqueName: \"kubernetes.io/projected/febb83d9-3a57-40a5-b972-c45611313469-kube-api-access-l8qt8\") on node \"ip-10-0-140-107.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:35:39.081077 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:35:39.080931 4939 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/febb83d9-3a57-40a5-b972-c45611313469-istio-token\") on node \"ip-10-0-140-107.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:35:39.081077 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:35:39.080948 4939 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/febb83d9-3a57-40a5-b972-c45611313469-istio-envoy\") on node \"ip-10-0-140-107.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:35:39.081077 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:35:39.080965 4939 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/febb83d9-3a57-40a5-b972-c45611313469-istio-podinfo\") on node \"ip-10-0-140-107.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:35:39.170306 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:35:39.170272 4939 generic.go:356] "Generic (PLEG): container finished" podID="febb83d9-3a57-40a5-b972-c45611313469" containerID="9f8d2d055b7c0d16ceed10a149a9af6f53d1436f29e72f3eb22bc4949a225ea7" exitCode=0 Nov 03 01:35:39.170610 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:35:39.170461 4939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-b4971a-openshift-default-5dc48765-8rn6x" event={"ID":"febb83d9-3a57-40a5-b972-c45611313469","Type":"ContainerDied","Data":"9f8d2d055b7c0d16ceed10a149a9af6f53d1436f29e72f3eb22bc4949a225ea7"} Nov 03 01:35:39.170610 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:35:39.170530 4939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-b4971a-openshift-default-5dc48765-8rn6x" event={"ID":"febb83d9-3a57-40a5-b972-c45611313469","Type":"ContainerDied","Data":"f254e9b7e85741988da4d32451be4ed9c00a5a7fe299b3c47998f7b2bca3df15"} Nov 03 01:35:39.170610 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:35:39.170573 4939 scope.go:122] "RemoveContainer" containerID="9f8d2d055b7c0d16ceed10a149a9af6f53d1436f29e72f3eb22bc4949a225ea7" Nov 03 01:35:39.171032 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:35:39.170832 4939 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-b4971a-openshift-default-5dc48765-8rn6x" Nov 03 01:35:39.185227 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:35:39.185206 4939 scope.go:122] "RemoveContainer" containerID="0f6ed1cdff446ef3dae0ff58786613404bafe9eef128ca4b36a6bd7611f2630a" Nov 03 01:35:39.197004 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:35:39.196416 4939 scope.go:122] "RemoveContainer" containerID="9f8d2d055b7c0d16ceed10a149a9af6f53d1436f29e72f3eb22bc4949a225ea7" Nov 03 01:35:39.197342 ip-10-0-140-107 kubenswrapper[4939]: E1103 01:35:39.197045 4939 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9f8d2d055b7c0d16ceed10a149a9af6f53d1436f29e72f3eb22bc4949a225ea7\": container with ID starting with 9f8d2d055b7c0d16ceed10a149a9af6f53d1436f29e72f3eb22bc4949a225ea7 not found: ID does not exist" containerID="9f8d2d055b7c0d16ceed10a149a9af6f53d1436f29e72f3eb22bc4949a225ea7" Nov 03 01:35:39.197342 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:35:39.197074 4939 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9f8d2d055b7c0d16ceed10a149a9af6f53d1436f29e72f3eb22bc4949a225ea7"} err="failed to get container status \"9f8d2d055b7c0d16ceed10a149a9af6f53d1436f29e72f3eb22bc4949a225ea7\": rpc error: code = NotFound desc = could not find container \"9f8d2d055b7c0d16ceed10a149a9af6f53d1436f29e72f3eb22bc4949a225ea7\": container with ID starting with 9f8d2d055b7c0d16ceed10a149a9af6f53d1436f29e72f3eb22bc4949a225ea7 not found: ID does not exist" Nov 03 01:35:39.197342 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:35:39.197095 4939 scope.go:122] "RemoveContainer" containerID="0f6ed1cdff446ef3dae0ff58786613404bafe9eef128ca4b36a6bd7611f2630a" Nov 03 01:35:39.197480 ip-10-0-140-107 kubenswrapper[4939]: E1103 01:35:39.197389 4939 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0f6ed1cdff446ef3dae0ff58786613404bafe9eef128ca4b36a6bd7611f2630a\": container with ID starting with 0f6ed1cdff446ef3dae0ff58786613404bafe9eef128ca4b36a6bd7611f2630a not found: ID does not exist" containerID="0f6ed1cdff446ef3dae0ff58786613404bafe9eef128ca4b36a6bd7611f2630a" Nov 03 01:35:39.197480 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:35:39.197413 4939 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0f6ed1cdff446ef3dae0ff58786613404bafe9eef128ca4b36a6bd7611f2630a"} err="failed to get container status \"0f6ed1cdff446ef3dae0ff58786613404bafe9eef128ca4b36a6bd7611f2630a\": rpc error: code = NotFound desc = could not find container \"0f6ed1cdff446ef3dae0ff58786613404bafe9eef128ca4b36a6bd7611f2630a\": container with ID starting with 0f6ed1cdff446ef3dae0ff58786613404bafe9eef128ca4b36a6bd7611f2630a not found: ID does not exist" Nov 03 01:35:39.198309 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:35:39.198294 4939 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-b4971a-openshift-default-5dc48765-8rn6x"] Nov 03 01:35:39.202719 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:35:39.202698 4939 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-b-b4971a-openshift-default-5dc48765-8rn6x"] Nov 03 01:35:39.311034 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:35:39.311000 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:35:39.872302 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:35:39.872268 4939 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="febb83d9-3a57-40a5-b972-c45611313469" path="/var/lib/kubelet/pods/febb83d9-3a57-40a5-b972-c45611313469/volumes" Nov 03 01:35:39.993022 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:35:39.992990 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:35:40.105293 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:35:40.105259 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:35:40.285560 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:35:40.285468 4939 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-inf-a-6fec8a/test-model-6fec8a-kserve-68c5cd696f-kmgsd" podUID="cab8e48f-de7c-42aa-93f5-9c7ac27f4cb9" containerName="main" containerID="cri-o://812f97ed476361456515aa4af72cc94b9764614d59841af53fcacc924a407789" gracePeriod=45 Nov 03 01:35:40.588168 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:35:40.588140 4939 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-6fec8a/test-model-6fec8a-kserve-68c5cd696f-kmgsd" Nov 03 01:35:40.711943 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:35:40.711892 4939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cab8e48f-de7c-42aa-93f5-9c7ac27f4cb9-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cab8e48f-de7c-42aa-93f5-9c7ac27f4cb9-tmp-dir\") pod \"cab8e48f-de7c-42aa-93f5-9c7ac27f4cb9\" (UID: \"cab8e48f-de7c-42aa-93f5-9c7ac27f4cb9\") " Nov 03 01:35:40.712130 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:35:40.712016 4939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/cab8e48f-de7c-42aa-93f5-9c7ac27f4cb9-tls-certs\" (UniqueName: \"kubernetes.io/secret/cab8e48f-de7c-42aa-93f5-9c7ac27f4cb9-tls-certs\") pod \"cab8e48f-de7c-42aa-93f5-9c7ac27f4cb9\" (UID: \"cab8e48f-de7c-42aa-93f5-9c7ac27f4cb9\") " Nov 03 01:35:40.712130 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:35:40.712080 4939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cab8e48f-de7c-42aa-93f5-9c7ac27f4cb9-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cab8e48f-de7c-42aa-93f5-9c7ac27f4cb9-model-cache\") pod \"cab8e48f-de7c-42aa-93f5-9c7ac27f4cb9\" (UID: \"cab8e48f-de7c-42aa-93f5-9c7ac27f4cb9\") " Nov 03 01:35:40.712234 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:35:40.712131 4939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cab8e48f-de7c-42aa-93f5-9c7ac27f4cb9-tmp-dir" pod "cab8e48f-de7c-42aa-93f5-9c7ac27f4cb9" (UID: "cab8e48f-de7c-42aa-93f5-9c7ac27f4cb9"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 01:35:40.712276 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:35:40.712250 4939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cab8e48f-de7c-42aa-93f5-9c7ac27f4cb9-dshm\" (UniqueName: \"kubernetes.io/empty-dir/cab8e48f-de7c-42aa-93f5-9c7ac27f4cb9-dshm\") pod \"cab8e48f-de7c-42aa-93f5-9c7ac27f4cb9\" (UID: \"cab8e48f-de7c-42aa-93f5-9c7ac27f4cb9\") " Nov 03 01:35:40.712330 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:35:40.712305 4939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cab8e48f-de7c-42aa-93f5-9c7ac27f4cb9-home\" (UniqueName: \"kubernetes.io/empty-dir/cab8e48f-de7c-42aa-93f5-9c7ac27f4cb9-home\") pod \"cab8e48f-de7c-42aa-93f5-9c7ac27f4cb9\" (UID: \"cab8e48f-de7c-42aa-93f5-9c7ac27f4cb9\") " Nov 03 01:35:40.712385 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:35:40.712329 4939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cab8e48f-de7c-42aa-93f5-9c7ac27f4cb9-kube-api-access-7vnb4\" (UniqueName: \"kubernetes.io/projected/cab8e48f-de7c-42aa-93f5-9c7ac27f4cb9-kube-api-access-7vnb4\") pod \"cab8e48f-de7c-42aa-93f5-9c7ac27f4cb9\" (UID: \"cab8e48f-de7c-42aa-93f5-9c7ac27f4cb9\") " Nov 03 01:35:40.712385 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:35:40.712339 4939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cab8e48f-de7c-42aa-93f5-9c7ac27f4cb9-model-cache" pod "cab8e48f-de7c-42aa-93f5-9c7ac27f4cb9" (UID: "cab8e48f-de7c-42aa-93f5-9c7ac27f4cb9"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 01:35:40.712816 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:35:40.712786 4939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cab8e48f-de7c-42aa-93f5-9c7ac27f4cb9-home" pod "cab8e48f-de7c-42aa-93f5-9c7ac27f4cb9" (UID: "cab8e48f-de7c-42aa-93f5-9c7ac27f4cb9"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 01:35:40.713162 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:35:40.713140 4939 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cab8e48f-de7c-42aa-93f5-9c7ac27f4cb9-tmp-dir\") on node \"ip-10-0-140-107.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:35:40.713251 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:35:40.713168 4939 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cab8e48f-de7c-42aa-93f5-9c7ac27f4cb9-model-cache\") on node \"ip-10-0-140-107.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:35:40.713251 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:35:40.713185 4939 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/cab8e48f-de7c-42aa-93f5-9c7ac27f4cb9-home\") on node \"ip-10-0-140-107.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:35:40.714475 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:35:40.714452 4939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cab8e48f-de7c-42aa-93f5-9c7ac27f4cb9-tls-certs" pod "cab8e48f-de7c-42aa-93f5-9c7ac27f4cb9" (UID: "cab8e48f-de7c-42aa-93f5-9c7ac27f4cb9"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 01:35:40.714938 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:35:40.714630 4939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cab8e48f-de7c-42aa-93f5-9c7ac27f4cb9-dshm" pod "cab8e48f-de7c-42aa-93f5-9c7ac27f4cb9" (UID: "cab8e48f-de7c-42aa-93f5-9c7ac27f4cb9"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 01:35:40.714938 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:35:40.714714 4939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cab8e48f-de7c-42aa-93f5-9c7ac27f4cb9-kube-api-access-7vnb4" pod "cab8e48f-de7c-42aa-93f5-9c7ac27f4cb9" (UID: "cab8e48f-de7c-42aa-93f5-9c7ac27f4cb9"). InnerVolumeSpecName "kube-api-access-7vnb4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 01:35:40.718682 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:35:40.718661 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:35:40.814748 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:35:40.814668 4939 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cab8e48f-de7c-42aa-93f5-9c7ac27f4cb9-tls-certs\") on node \"ip-10-0-140-107.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:35:40.814748 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:35:40.814698 4939 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/cab8e48f-de7c-42aa-93f5-9c7ac27f4cb9-dshm\") on node \"ip-10-0-140-107.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:35:40.814748 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:35:40.814710 4939 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7vnb4\" (UniqueName: \"kubernetes.io/projected/cab8e48f-de7c-42aa-93f5-9c7ac27f4cb9-kube-api-access-7vnb4\") on node \"ip-10-0-140-107.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:35:41.189917 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:35:41.189870 4939 generic.go:356] "Generic (PLEG): container finished" podID="cab8e48f-de7c-42aa-93f5-9c7ac27f4cb9" containerID="812f97ed476361456515aa4af72cc94b9764614d59841af53fcacc924a407789" exitCode=0 Nov 03 01:35:41.190251 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:35:41.189934 4939 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-6fec8a/test-model-6fec8a-kserve-68c5cd696f-kmgsd" Nov 03 01:35:41.190251 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:35:41.189942 4939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-6fec8a/test-model-6fec8a-kserve-68c5cd696f-kmgsd" event={"ID":"cab8e48f-de7c-42aa-93f5-9c7ac27f4cb9","Type":"ContainerDied","Data":"812f97ed476361456515aa4af72cc94b9764614d59841af53fcacc924a407789"} Nov 03 01:35:41.190251 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:35:41.189975 4939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-6fec8a/test-model-6fec8a-kserve-68c5cd696f-kmgsd" event={"ID":"cab8e48f-de7c-42aa-93f5-9c7ac27f4cb9","Type":"ContainerDied","Data":"3f1ebeeb7e2ac927f9f857306aad4069279692eab9d03bdb658da7b32ffad7f3"} Nov 03 01:35:41.190251 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:35:41.189993 4939 scope.go:122] "RemoveContainer" containerID="812f97ed476361456515aa4af72cc94b9764614d59841af53fcacc924a407789" Nov 03 01:35:41.202321 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:35:41.202302 4939 scope.go:122] "RemoveContainer" containerID="812f97ed476361456515aa4af72cc94b9764614d59841af53fcacc924a407789" Nov 03 01:35:41.202585 ip-10-0-140-107 kubenswrapper[4939]: E1103 01:35:41.202560 4939 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"812f97ed476361456515aa4af72cc94b9764614d59841af53fcacc924a407789\": container with ID starting with 812f97ed476361456515aa4af72cc94b9764614d59841af53fcacc924a407789 not found: ID does not exist" containerID="812f97ed476361456515aa4af72cc94b9764614d59841af53fcacc924a407789" Nov 03 01:35:41.202635 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:35:41.202599 4939 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"812f97ed476361456515aa4af72cc94b9764614d59841af53fcacc924a407789"} err="failed to get container status \"812f97ed476361456515aa4af72cc94b9764614d59841af53fcacc924a407789\": rpc error: code = NotFound desc = could not find container \"812f97ed476361456515aa4af72cc94b9764614d59841af53fcacc924a407789\": container with ID starting with 812f97ed476361456515aa4af72cc94b9764614d59841af53fcacc924a407789 not found: ID does not exist" Nov 03 01:35:41.216289 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:35:41.216266 4939 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-6fec8a/test-model-6fec8a-kserve-68c5cd696f-kmgsd"] Nov 03 01:35:41.220364 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:35:41.220343 4939 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-a-6fec8a/test-model-6fec8a-kserve-68c5cd696f-kmgsd"] Nov 03 01:35:41.478032 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:35:41.477941 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:35:41.873850 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:35:41.873817 4939 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cab8e48f-de7c-42aa-93f5-9c7ac27f4cb9" path="/var/lib/kubelet/pods/cab8e48f-de7c-42aa-93f5-9c7ac27f4cb9/volumes" Nov 03 01:35:42.299874 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:35:42.299791 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:35:42.929246 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:35:42.929215 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:35:43.576055 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:35:43.576022 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:35:44.349346 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:35:44.349310 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:35:44.518653 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:35:44.518619 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:35:45.028107 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:35:45.028064 4939 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-b4971a-6c58f96447-rjn8w" podUID="6712b32f-f3bd-4917-9e5e-101cf604625c" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 03 01:35:45.088131 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:35:45.088091 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:35:45.308790 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:35:45.308710 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:35:45.516090 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:35:45.516052 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:35:45.627978 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:35:45.627945 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:35:45.708772 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:35:45.708737 4939 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-w2-531447/sub-test-model-531447-kserve-65cffbb7c-h6b59" Nov 03 01:35:45.725836 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:35:45.725795 4939 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-w2-531447/sub-test-model-531447-kserve-65cffbb7c-h6b59" Nov 03 01:35:46.219855 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:35:46.219817 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:35:46.245665 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:35:46.245637 4939 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-w2-531447/sub-test-model-531447-kserve-65cffbb7c-h6b59" Nov 03 01:35:46.381896 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:35:46.381860 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:35:46.558499 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:35:46.558409 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:35:47.167302 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:35:47.167269 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:35:47.788549 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:35:47.788517 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:35:48.038700 ip-10-0-140-107 kubenswrapper[4939]: E1103 01:35:48.038607 4939 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-sthjh" containerName="extractor" Nov 03 01:35:48.449173 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:35:48.449135 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:35:49.106541 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:35:49.106507 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:35:49.814601 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:35:49.814566 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:35:50.453774 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:35:50.453739 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:35:51.297748 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:35:51.297710 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:35:51.409877 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:35:51.409845 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:35:51.859777 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:35:51.859740 4939 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-w2-ef434a/sub-test-model-ef434a-kserve-659ddb8595-zr7c2"] Nov 03 01:35:51.861112 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:35:51.861085 4939 cpu_manager.go:414] "containerMap: removing container" podUID="cab8e48f-de7c-42aa-93f5-9c7ac27f4cb9" containerName="main" Nov 03 01:35:51.861251 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:35:51.861111 4939 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cab8e48f-de7c-42aa-93f5-9c7ac27f4cb9" containerName="main" Nov 03 01:35:51.861251 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:35:51.861194 4939 cpu_manager.go:414] "containerMap: removing container" podUID="febb83d9-3a57-40a5-b972-c45611313469" containerName="istio-proxy" Nov 03 01:35:51.861251 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:35:51.861204 4939 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="febb83d9-3a57-40a5-b972-c45611313469" containerName="istio-proxy" Nov 03 01:35:51.861251 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:35:51.861224 4939 cpu_manager.go:414] "containerMap: removing container" podUID="febb83d9-3a57-40a5-b972-c45611313469" containerName="istio-proxy" Nov 03 01:35:51.861251 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:35:51.861232 4939 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="febb83d9-3a57-40a5-b972-c45611313469" containerName="istio-proxy" Nov 03 01:35:51.861448 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:35:51.861436 4939 memory_manager.go:365] "RemoveStaleState removing state" podUID="febb83d9-3a57-40a5-b972-c45611313469" containerName="istio-proxy" Nov 03 01:35:51.861487 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:35:51.861455 4939 memory_manager.go:365] "RemoveStaleState removing state" podUID="febb83d9-3a57-40a5-b972-c45611313469" containerName="istio-proxy" Nov 03 01:35:51.861487 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:35:51.861475 4939 memory_manager.go:365] "RemoveStaleState removing state" podUID="cab8e48f-de7c-42aa-93f5-9c7ac27f4cb9" containerName="main" Nov 03 01:35:51.861597 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:35:51.861491 4939 memory_manager.go:365] "RemoveStaleState removing state" podUID="93ab25a5-4bed-4d84-9d62-640cea8d4047" containerName="istio-proxy" Nov 03 01:35:51.867814 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:35:51.867769 4939 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w2-ef434a/sub-test-model-ef434a-kserve-659ddb8595-zr7c2" Nov 03 01:35:51.874450 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:35:51.874424 4939 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w2-ef434a\"/\"sub-test-model-ef434a-kserve-self-signed-certs\"" Nov 03 01:35:51.874758 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:35:51.874740 4939 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w2-ef434a\"/\"kube-root-ca.crt\"" Nov 03 01:35:51.875019 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:35:51.875003 4939 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w2-ef434a\"/\"openshift-service-ca.crt\"" Nov 03 01:35:51.875837 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:35:51.875815 4939 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w2-ef434a\"/\"default-dockercfg-ttddl\"" Nov 03 01:35:51.899171 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:35:51.899143 4939 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w2-ef434a/sub-test-model-ef434a-kserve-659ddb8595-zr7c2"] Nov 03 01:35:51.900195 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:35:51.900165 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:35:51.947114 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:35:51.947069 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4s57v\" (UniqueName: \"kubernetes.io/projected/5f7f5188-d311-486b-bb78-583a436145c5-kube-api-access-4s57v\") pod \"sub-test-model-ef434a-kserve-659ddb8595-zr7c2\" (UID: \"5f7f5188-d311-486b-bb78-583a436145c5\") " pod="ai-tenant-e2e-shared-b-w2-ef434a/sub-test-model-ef434a-kserve-659ddb8595-zr7c2" Nov 03 01:35:51.947324 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:35:51.947303 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5f7f5188-d311-486b-bb78-583a436145c5-tls-certs\") pod \"sub-test-model-ef434a-kserve-659ddb8595-zr7c2\" (UID: \"5f7f5188-d311-486b-bb78-583a436145c5\") " pod="ai-tenant-e2e-shared-b-w2-ef434a/sub-test-model-ef434a-kserve-659ddb8595-zr7c2" Nov 03 01:35:51.947497 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:35:51.947474 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5f7f5188-d311-486b-bb78-583a436145c5-dshm\") pod \"sub-test-model-ef434a-kserve-659ddb8595-zr7c2\" (UID: \"5f7f5188-d311-486b-bb78-583a436145c5\") " pod="ai-tenant-e2e-shared-b-w2-ef434a/sub-test-model-ef434a-kserve-659ddb8595-zr7c2" Nov 03 01:35:51.947695 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:35:51.947669 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5f7f5188-d311-486b-bb78-583a436145c5-home\") pod \"sub-test-model-ef434a-kserve-659ddb8595-zr7c2\" (UID: \"5f7f5188-d311-486b-bb78-583a436145c5\") " pod="ai-tenant-e2e-shared-b-w2-ef434a/sub-test-model-ef434a-kserve-659ddb8595-zr7c2" Nov 03 01:35:51.947892 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:35:51.947867 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5f7f5188-d311-486b-bb78-583a436145c5-model-cache\") pod \"sub-test-model-ef434a-kserve-659ddb8595-zr7c2\" (UID: \"5f7f5188-d311-486b-bb78-583a436145c5\") " pod="ai-tenant-e2e-shared-b-w2-ef434a/sub-test-model-ef434a-kserve-659ddb8595-zr7c2" Nov 03 01:35:51.948105 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:35:51.948087 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5f7f5188-d311-486b-bb78-583a436145c5-tmp-dir\") pod \"sub-test-model-ef434a-kserve-659ddb8595-zr7c2\" (UID: \"5f7f5188-d311-486b-bb78-583a436145c5\") " pod="ai-tenant-e2e-shared-b-w2-ef434a/sub-test-model-ef434a-kserve-659ddb8595-zr7c2" Nov 03 01:35:52.015038 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:35:52.015009 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:35:52.049812 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:35:52.049776 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5f7f5188-d311-486b-bb78-583a436145c5-tmp-dir\") pod \"sub-test-model-ef434a-kserve-659ddb8595-zr7c2\" (UID: \"5f7f5188-d311-486b-bb78-583a436145c5\") " pod="ai-tenant-e2e-shared-b-w2-ef434a/sub-test-model-ef434a-kserve-659ddb8595-zr7c2" Nov 03 01:35:52.050044 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:35:52.049880 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4s57v\" (UniqueName: \"kubernetes.io/projected/5f7f5188-d311-486b-bb78-583a436145c5-kube-api-access-4s57v\") pod \"sub-test-model-ef434a-kserve-659ddb8595-zr7c2\" (UID: \"5f7f5188-d311-486b-bb78-583a436145c5\") " pod="ai-tenant-e2e-shared-b-w2-ef434a/sub-test-model-ef434a-kserve-659ddb8595-zr7c2" Nov 03 01:35:52.050167 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:35:52.050077 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5f7f5188-d311-486b-bb78-583a436145c5-tls-certs\") pod \"sub-test-model-ef434a-kserve-659ddb8595-zr7c2\" (UID: \"5f7f5188-d311-486b-bb78-583a436145c5\") " pod="ai-tenant-e2e-shared-b-w2-ef434a/sub-test-model-ef434a-kserve-659ddb8595-zr7c2" Nov 03 01:35:52.050239 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:35:52.050201 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5f7f5188-d311-486b-bb78-583a436145c5-dshm\") pod \"sub-test-model-ef434a-kserve-659ddb8595-zr7c2\" (UID: \"5f7f5188-d311-486b-bb78-583a436145c5\") " pod="ai-tenant-e2e-shared-b-w2-ef434a/sub-test-model-ef434a-kserve-659ddb8595-zr7c2" Nov 03 01:35:52.050239 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:35:52.050234 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5f7f5188-d311-486b-bb78-583a436145c5-home\") pod \"sub-test-model-ef434a-kserve-659ddb8595-zr7c2\" (UID: \"5f7f5188-d311-486b-bb78-583a436145c5\") " pod="ai-tenant-e2e-shared-b-w2-ef434a/sub-test-model-ef434a-kserve-659ddb8595-zr7c2" Nov 03 01:35:52.050345 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:35:52.050269 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5f7f5188-d311-486b-bb78-583a436145c5-tmp-dir\") pod \"sub-test-model-ef434a-kserve-659ddb8595-zr7c2\" (UID: \"5f7f5188-d311-486b-bb78-583a436145c5\") " pod="ai-tenant-e2e-shared-b-w2-ef434a/sub-test-model-ef434a-kserve-659ddb8595-zr7c2" Nov 03 01:35:52.050345 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:35:52.050295 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5f7f5188-d311-486b-bb78-583a436145c5-model-cache\") pod \"sub-test-model-ef434a-kserve-659ddb8595-zr7c2\" (UID: \"5f7f5188-d311-486b-bb78-583a436145c5\") " pod="ai-tenant-e2e-shared-b-w2-ef434a/sub-test-model-ef434a-kserve-659ddb8595-zr7c2" Nov 03 01:35:52.050602 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:35:52.050577 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5f7f5188-d311-486b-bb78-583a436145c5-home\") pod \"sub-test-model-ef434a-kserve-659ddb8595-zr7c2\" (UID: \"5f7f5188-d311-486b-bb78-583a436145c5\") " pod="ai-tenant-e2e-shared-b-w2-ef434a/sub-test-model-ef434a-kserve-659ddb8595-zr7c2" Nov 03 01:35:52.050712 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:35:52.050673 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5f7f5188-d311-486b-bb78-583a436145c5-model-cache\") pod \"sub-test-model-ef434a-kserve-659ddb8595-zr7c2\" (UID: \"5f7f5188-d311-486b-bb78-583a436145c5\") " pod="ai-tenant-e2e-shared-b-w2-ef434a/sub-test-model-ef434a-kserve-659ddb8595-zr7c2" Nov 03 01:35:52.052443 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:35:52.052416 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5f7f5188-d311-486b-bb78-583a436145c5-dshm\") pod \"sub-test-model-ef434a-kserve-659ddb8595-zr7c2\" (UID: \"5f7f5188-d311-486b-bb78-583a436145c5\") " pod="ai-tenant-e2e-shared-b-w2-ef434a/sub-test-model-ef434a-kserve-659ddb8595-zr7c2" Nov 03 01:35:52.052703 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:35:52.052684 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5f7f5188-d311-486b-bb78-583a436145c5-tls-certs\") pod \"sub-test-model-ef434a-kserve-659ddb8595-zr7c2\" (UID: \"5f7f5188-d311-486b-bb78-583a436145c5\") " pod="ai-tenant-e2e-shared-b-w2-ef434a/sub-test-model-ef434a-kserve-659ddb8595-zr7c2" Nov 03 01:35:52.182825 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:35:52.182747 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4s57v\" (UniqueName: \"kubernetes.io/projected/5f7f5188-d311-486b-bb78-583a436145c5-kube-api-access-4s57v\") pod \"sub-test-model-ef434a-kserve-659ddb8595-zr7c2\" (UID: \"5f7f5188-d311-486b-bb78-583a436145c5\") " pod="ai-tenant-e2e-shared-b-w2-ef434a/sub-test-model-ef434a-kserve-659ddb8595-zr7c2" Nov 03 01:35:52.206087 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:35:52.206053 4939 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w2-ef434a\"/\"default-dockercfg-ttddl\"" Nov 03 01:35:52.221793 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:35:52.221754 4939 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w2-ef434a/sub-test-model-ef434a-kserve-659ddb8595-zr7c2" Nov 03 01:35:52.301286 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:35:52.300554 4939 generic.go:356] "Generic (PLEG): container finished" podID="6712b32f-f3bd-4917-9e5e-101cf604625c" containerID="384febeac8280063b68f71ab034887b1d84a8e149d013ded2756f3066aeecdf7" exitCode=0 Nov 03 01:35:52.301286 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:35:52.300835 4939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-b4971a-6c58f96447-rjn8w" event={"ID":"6712b32f-f3bd-4917-9e5e-101cf604625c","Type":"ContainerDied","Data":"384febeac8280063b68f71ab034887b1d84a8e149d013ded2756f3066aeecdf7"} Nov 03 01:35:52.391719 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:35:52.391645 4939 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-b4971a-6c58f96447-rjn8w" Nov 03 01:35:52.447452 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:35:52.447414 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:35:52.463539 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:35:52.463497 4939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6712b32f-f3bd-4917-9e5e-101cf604625c-kube-api-access-g6sbs\" (UniqueName: \"kubernetes.io/projected/6712b32f-f3bd-4917-9e5e-101cf604625c-kube-api-access-g6sbs\") pod \"6712b32f-f3bd-4917-9e5e-101cf604625c\" (UID: \"6712b32f-f3bd-4917-9e5e-101cf604625c\") " Nov 03 01:35:52.463743 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:35:52.463572 4939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6712b32f-f3bd-4917-9e5e-101cf604625c-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6712b32f-f3bd-4917-9e5e-101cf604625c-plugins-config-volume\") pod \"6712b32f-f3bd-4917-9e5e-101cf604625c\" (UID: \"6712b32f-f3bd-4917-9e5e-101cf604625c\") " Nov 03 01:35:52.464041 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:35:52.464013 4939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6712b32f-f3bd-4917-9e5e-101cf604625c-plugins-config-volume" pod "6712b32f-f3bd-4917-9e5e-101cf604625c" (UID: "6712b32f-f3bd-4917-9e5e-101cf604625c"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 01:35:52.464316 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:35:52.464297 4939 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6712b32f-f3bd-4917-9e5e-101cf604625c-plugins-config-volume\") on node \"ip-10-0-140-107.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:35:52.465864 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:35:52.465838 4939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6712b32f-f3bd-4917-9e5e-101cf604625c-kube-api-access-g6sbs" pod "6712b32f-f3bd-4917-9e5e-101cf604625c" (UID: "6712b32f-f3bd-4917-9e5e-101cf604625c"). InnerVolumeSpecName "kube-api-access-g6sbs". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 01:35:52.538182 ip-10-0-140-107 kubenswrapper[4939]: W1103 01:35:52.538144 4939 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod5f7f5188_d311_486b_bb78_583a436145c5.slice/crio-e5d69550164c1b4d6d0fe38a095ad984d71b1813445a98f3b0def53b22052fb0 WatchSource:0}: Error finding container e5d69550164c1b4d6d0fe38a095ad984d71b1813445a98f3b0def53b22052fb0: Status 404 returned error can't find the container with id e5d69550164c1b4d6d0fe38a095ad984d71b1813445a98f3b0def53b22052fb0 Nov 03 01:35:52.539242 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:35:52.539205 4939 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w2-ef434a/sub-test-model-ef434a-kserve-659ddb8595-zr7c2"] Nov 03 01:35:52.566136 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:35:52.566101 4939 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-g6sbs\" (UniqueName: \"kubernetes.io/projected/6712b32f-f3bd-4917-9e5e-101cf604625c-kube-api-access-g6sbs\") on node \"ip-10-0-140-107.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:35:52.757246 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:35:52.757160 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:35:52.955466 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:35:52.955433 4939 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-n42s6_65d134ac-8a25-4c67-b6c7-cd752a098daf/console-operator/2.log" Nov 03 01:35:53.022426 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:35:53.022327 4939 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-n42s6_65d134ac-8a25-4c67-b6c7-cd752a098daf/console-operator/2.log" Nov 03 01:35:53.132595 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:35:53.132550 4939 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-inf-b-5dab6a/test-model-5dab6a-kserve-5c4f47747c-z44hb" podUID="9a691811-c7b4-4da0-892d-13d98fb3a0b0" containerName="main" containerID="cri-o://19db77487df3cc3019000d70933d954c52519531c1ed657adfadd7cf22dda66e" gracePeriod=45 Nov 03 01:35:53.311224 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:35:53.311142 4939 generic.go:356] "Generic (PLEG): container finished" podID="9a691811-c7b4-4da0-892d-13d98fb3a0b0" containerID="19db77487df3cc3019000d70933d954c52519531c1ed657adfadd7cf22dda66e" exitCode=0 Nov 03 01:35:53.311376 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:35:53.311216 4939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-5dab6a/test-model-5dab6a-kserve-5c4f47747c-z44hb" event={"ID":"9a691811-c7b4-4da0-892d-13d98fb3a0b0","Type":"ContainerDied","Data":"19db77487df3cc3019000d70933d954c52519531c1ed657adfadd7cf22dda66e"} Nov 03 01:35:53.313358 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:35:53.313307 4939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w2-ef434a/sub-test-model-ef434a-kserve-659ddb8595-zr7c2" event={"ID":"5f7f5188-d311-486b-bb78-583a436145c5","Type":"ContainerStarted","Data":"bdd2ff5643812c0947899c193a7d22052a6cc38247cf5cbb3acc63b98c0907d1"} Nov 03 01:35:53.313358 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:35:53.313352 4939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w2-ef434a/sub-test-model-ef434a-kserve-659ddb8595-zr7c2" event={"ID":"5f7f5188-d311-486b-bb78-583a436145c5","Type":"ContainerStarted","Data":"e5d69550164c1b4d6d0fe38a095ad984d71b1813445a98f3b0def53b22052fb0"} Nov 03 01:35:53.315098 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:35:53.315073 4939 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-b4971a-6c58f96447-rjn8w" Nov 03 01:35:53.315233 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:35:53.315096 4939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-b4971a-6c58f96447-rjn8w" event={"ID":"6712b32f-f3bd-4917-9e5e-101cf604625c","Type":"ContainerDied","Data":"c4d3c828c1922f9760a0735709482befc9025e8f4e39367e18532e4200e124b6"} Nov 03 01:35:53.315233 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:35:53.315149 4939 scope.go:122] "RemoveContainer" containerID="384febeac8280063b68f71ab034887b1d84a8e149d013ded2756f3066aeecdf7" Nov 03 01:35:53.367373 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:35:53.367305 4939 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-w2-ef434a/sub-test-model-ef434a-kserve-659ddb8595-zr7c2" podStartSLOduration=2.367285312 podStartE2EDuration="2.367285312s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 01:35:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 01:35:53.342900292 +0000 UTC m=+3301.607235242" watchObservedRunningTime="2026-11-03 01:35:53.367285312 +0000 UTC m=+3301.631620257" Nov 03 01:35:53.374799 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:35:53.374763 4939 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-b4971a-6c58f96447-rjn8w"] Nov 03 01:35:53.381221 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:35:53.381186 4939 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-b4971a-6c58f96447-rjn8w"] Nov 03 01:35:53.412562 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:35:53.412483 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:35:53.504579 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:35:53.504549 4939 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-5dab6a/test-model-5dab6a-kserve-5c4f47747c-z44hb" Nov 03 01:35:53.592702 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:35:53.592669 4939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9a691811-c7b4-4da0-892d-13d98fb3a0b0-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9a691811-c7b4-4da0-892d-13d98fb3a0b0-tmp-dir\") pod \"9a691811-c7b4-4da0-892d-13d98fb3a0b0\" (UID: \"9a691811-c7b4-4da0-892d-13d98fb3a0b0\") " Nov 03 01:35:53.593004 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:35:53.592978 4939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9a691811-c7b4-4da0-892d-13d98fb3a0b0-tmp-dir" pod "9a691811-c7b4-4da0-892d-13d98fb3a0b0" (UID: "9a691811-c7b4-4da0-892d-13d98fb3a0b0"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 01:35:53.593115 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:35:53.593004 4939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9a691811-c7b4-4da0-892d-13d98fb3a0b0-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9a691811-c7b4-4da0-892d-13d98fb3a0b0-model-cache\") pod \"9a691811-c7b4-4da0-892d-13d98fb3a0b0\" (UID: \"9a691811-c7b4-4da0-892d-13d98fb3a0b0\") " Nov 03 01:35:53.593188 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:35:53.593110 4939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9a691811-c7b4-4da0-892d-13d98fb3a0b0-kube-api-access-xdct6\" (UniqueName: \"kubernetes.io/projected/9a691811-c7b4-4da0-892d-13d98fb3a0b0-kube-api-access-xdct6\") pod \"9a691811-c7b4-4da0-892d-13d98fb3a0b0\" (UID: \"9a691811-c7b4-4da0-892d-13d98fb3a0b0\") " Nov 03 01:35:53.593188 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:35:53.593149 4939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9a691811-c7b4-4da0-892d-13d98fb3a0b0-dshm\" (UniqueName: \"kubernetes.io/empty-dir/9a691811-c7b4-4da0-892d-13d98fb3a0b0-dshm\") pod \"9a691811-c7b4-4da0-892d-13d98fb3a0b0\" (UID: \"9a691811-c7b4-4da0-892d-13d98fb3a0b0\") " Nov 03 01:35:53.593300 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:35:53.593186 4939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9a691811-c7b4-4da0-892d-13d98fb3a0b0-tls-certs\" (UniqueName: \"kubernetes.io/secret/9a691811-c7b4-4da0-892d-13d98fb3a0b0-tls-certs\") pod \"9a691811-c7b4-4da0-892d-13d98fb3a0b0\" (UID: \"9a691811-c7b4-4da0-892d-13d98fb3a0b0\") " Nov 03 01:35:53.593300 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:35:53.593214 4939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9a691811-c7b4-4da0-892d-13d98fb3a0b0-home\" (UniqueName: \"kubernetes.io/empty-dir/9a691811-c7b4-4da0-892d-13d98fb3a0b0-home\") pod \"9a691811-c7b4-4da0-892d-13d98fb3a0b0\" (UID: \"9a691811-c7b4-4da0-892d-13d98fb3a0b0\") " Nov 03 01:35:53.593300 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:35:53.593245 4939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9a691811-c7b4-4da0-892d-13d98fb3a0b0-model-cache" pod "9a691811-c7b4-4da0-892d-13d98fb3a0b0" (UID: "9a691811-c7b4-4da0-892d-13d98fb3a0b0"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 01:35:53.593566 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:35:53.593539 4939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9a691811-c7b4-4da0-892d-13d98fb3a0b0-home" pod "9a691811-c7b4-4da0-892d-13d98fb3a0b0" (UID: "9a691811-c7b4-4da0-892d-13d98fb3a0b0"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 01:35:53.594231 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:35:53.594205 4939 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9a691811-c7b4-4da0-892d-13d98fb3a0b0-tmp-dir\") on node \"ip-10-0-140-107.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:35:53.594231 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:35:53.594236 4939 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9a691811-c7b4-4da0-892d-13d98fb3a0b0-model-cache\") on node \"ip-10-0-140-107.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:35:53.594414 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:35:53.594252 4939 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9a691811-c7b4-4da0-892d-13d98fb3a0b0-home\") on node \"ip-10-0-140-107.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:35:53.595492 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:35:53.595464 4939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9a691811-c7b4-4da0-892d-13d98fb3a0b0-kube-api-access-xdct6" pod "9a691811-c7b4-4da0-892d-13d98fb3a0b0" (UID: "9a691811-c7b4-4da0-892d-13d98fb3a0b0"). InnerVolumeSpecName "kube-api-access-xdct6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 01:35:53.595492 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:35:53.595488 4939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9a691811-c7b4-4da0-892d-13d98fb3a0b0-dshm" pod "9a691811-c7b4-4da0-892d-13d98fb3a0b0" (UID: "9a691811-c7b4-4da0-892d-13d98fb3a0b0"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 01:35:53.595670 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:35:53.595576 4939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9a691811-c7b4-4da0-892d-13d98fb3a0b0-tls-certs" pod "9a691811-c7b4-4da0-892d-13d98fb3a0b0" (UID: "9a691811-c7b4-4da0-892d-13d98fb3a0b0"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 01:35:53.695414 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:35:53.695377 4939 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xdct6\" (UniqueName: \"kubernetes.io/projected/9a691811-c7b4-4da0-892d-13d98fb3a0b0-kube-api-access-xdct6\") on node \"ip-10-0-140-107.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:35:53.695414 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:35:53.695407 4939 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9a691811-c7b4-4da0-892d-13d98fb3a0b0-dshm\") on node \"ip-10-0-140-107.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:35:53.695414 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:35:53.695419 4939 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9a691811-c7b4-4da0-892d-13d98fb3a0b0-tls-certs\") on node \"ip-10-0-140-107.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:35:53.871797 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:35:53.871706 4939 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6712b32f-f3bd-4917-9e5e-101cf604625c" path="/var/lib/kubelet/pods/6712b32f-f3bd-4917-9e5e-101cf604625c/volumes" Nov 03 01:35:54.170011 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:35:54.169929 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:35:54.232931 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:35:54.226517 4939 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-w2-ef434a/sub-test-model-ef434a-kserve-659ddb8595-zr7c2" Nov 03 01:35:54.326426 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:35:54.326392 4939 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-5dab6a/test-model-5dab6a-kserve-5c4f47747c-z44hb" Nov 03 01:35:54.326426 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:35:54.326402 4939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-5dab6a/test-model-5dab6a-kserve-5c4f47747c-z44hb" event={"ID":"9a691811-c7b4-4da0-892d-13d98fb3a0b0","Type":"ContainerDied","Data":"a2d2816d572ec076f2f0ddc17582c0a45a3581903ed9c01be347da9733c00553"} Nov 03 01:35:54.326714 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:35:54.326460 4939 scope.go:122] "RemoveContainer" containerID="19db77487df3cc3019000d70933d954c52519531c1ed657adfadd7cf22dda66e" Nov 03 01:35:54.345646 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:35:54.345613 4939 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-5dab6a/test-model-5dab6a-kserve-5c4f47747c-z44hb"] Nov 03 01:35:54.349840 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:35:54.349815 4939 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-b-5dab6a/test-model-5dab6a-kserve-5c4f47747c-z44hb"] Nov 03 01:35:54.530310 ip-10-0-140-107 kubenswrapper[4939]: E1103 01:35:54.530203 4939 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6712b32f_f3bd_4917_9e5e_101cf604625c.slice/crio-384febeac8280063b68f71ab034887b1d84a8e149d013ded2756f3066aeecdf7.scope\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6712b32f_f3bd_4917_9e5e_101cf604625c.slice/crio-conmon-384febeac8280063b68f71ab034887b1d84a8e149d013ded2756f3066aeecdf7.scope\": RecentStats: unable to find data in memory cache]" Nov 03 01:35:54.836343 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:35:54.836315 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:35:55.157705 ip-10-0-140-107 kubenswrapper[4939]: E1103 01:35:55.157608 4939 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6712b32f_f3bd_4917_9e5e_101cf604625c.slice/crio-conmon-384febeac8280063b68f71ab034887b1d84a8e149d013ded2756f3066aeecdf7.scope\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6712b32f_f3bd_4917_9e5e_101cf604625c.slice/crio-384febeac8280063b68f71ab034887b1d84a8e149d013ded2756f3066aeecdf7.scope\": RecentStats: unable to find data in memory cache]" Nov 03 01:35:55.515193 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:35:55.515113 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:35:55.871279 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:35:55.871238 4939 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9a691811-c7b4-4da0-892d-13d98fb3a0b0" path="/var/lib/kubelet/pods/9a691811-c7b4-4da0-892d-13d98fb3a0b0/volumes" Nov 03 01:35:56.095294 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:35:56.095264 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:35:56.579686 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:35:56.579654 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:35:57.423928 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:35:57.423510 4939 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-6fec8a-6cdc9b4ff5-xjq85"] Nov 03 01:35:57.423928 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:35:57.423792 4939 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-6fec8a-6cdc9b4ff5-xjq85" podUID="05a50b7a-195c-4ec9-9269-5d58faee868b" containerName="maas-api" containerID="cri-o://8d4b58e052a8f6f6d7c66b430f1e71b6f4a161f8f4de727bacc5e5b79b5a0a55" gracePeriod=30 Nov 03 01:35:57.475554 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:35:57.475494 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:35:57.988098 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:35:57.979240 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:35:58.042573 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:35:58.042545 4939 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-6fec8a-6cdc9b4ff5-xjq85" Nov 03 01:35:58.069531 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:35:58.069495 4939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/05a50b7a-195c-4ec9-9269-5d58faee868b-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/05a50b7a-195c-4ec9-9269-5d58faee868b-maas-api-tls\") pod \"05a50b7a-195c-4ec9-9269-5d58faee868b\" (UID: \"05a50b7a-195c-4ec9-9269-5d58faee868b\") " Nov 03 01:35:58.069824 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:35:58.069809 4939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/05a50b7a-195c-4ec9-9269-5d58faee868b-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/05a50b7a-195c-4ec9-9269-5d58faee868b-openshift-service-ca\") pod \"05a50b7a-195c-4ec9-9269-5d58faee868b\" (UID: \"05a50b7a-195c-4ec9-9269-5d58faee868b\") " Nov 03 01:35:58.069977 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:35:58.069852 4939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/05a50b7a-195c-4ec9-9269-5d58faee868b-metrics-certs\" (UniqueName: \"kubernetes.io/secret/05a50b7a-195c-4ec9-9269-5d58faee868b-metrics-certs\") pod \"05a50b7a-195c-4ec9-9269-5d58faee868b\" (UID: \"05a50b7a-195c-4ec9-9269-5d58faee868b\") " Nov 03 01:35:58.069977 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:35:58.069920 4939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/05a50b7a-195c-4ec9-9269-5d58faee868b-kube-api-access-4vt9b\" (UniqueName: \"kubernetes.io/projected/05a50b7a-195c-4ec9-9269-5d58faee868b-kube-api-access-4vt9b\") pod \"05a50b7a-195c-4ec9-9269-5d58faee868b\" (UID: \"05a50b7a-195c-4ec9-9269-5d58faee868b\") " Nov 03 01:35:58.071226 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:35:58.071197 4939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/05a50b7a-195c-4ec9-9269-5d58faee868b-openshift-service-ca" pod "05a50b7a-195c-4ec9-9269-5d58faee868b" (UID: "05a50b7a-195c-4ec9-9269-5d58faee868b"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 01:35:58.073772 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:35:58.073738 4939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/05a50b7a-195c-4ec9-9269-5d58faee868b-kube-api-access-4vt9b" pod "05a50b7a-195c-4ec9-9269-5d58faee868b" (UID: "05a50b7a-195c-4ec9-9269-5d58faee868b"). InnerVolumeSpecName "kube-api-access-4vt9b". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 01:35:58.075197 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:35:58.075173 4939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/05a50b7a-195c-4ec9-9269-5d58faee868b-maas-api-tls" pod "05a50b7a-195c-4ec9-9269-5d58faee868b" (UID: "05a50b7a-195c-4ec9-9269-5d58faee868b"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 01:35:58.075653 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:35:58.075627 4939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/05a50b7a-195c-4ec9-9269-5d58faee868b-metrics-certs" pod "05a50b7a-195c-4ec9-9269-5d58faee868b" (UID: "05a50b7a-195c-4ec9-9269-5d58faee868b"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 01:35:58.172115 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:35:58.172077 4939 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/05a50b7a-195c-4ec9-9269-5d58faee868b-openshift-service-ca\") on node \"ip-10-0-140-107.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:35:58.172115 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:35:58.172110 4939 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/05a50b7a-195c-4ec9-9269-5d58faee868b-metrics-certs\") on node \"ip-10-0-140-107.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:35:58.172115 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:35:58.172122 4939 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4vt9b\" (UniqueName: \"kubernetes.io/projected/05a50b7a-195c-4ec9-9269-5d58faee868b-kube-api-access-4vt9b\") on node \"ip-10-0-140-107.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:35:58.172363 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:35:58.172131 4939 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/05a50b7a-195c-4ec9-9269-5d58faee868b-maas-api-tls\") on node \"ip-10-0-140-107.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:35:58.494147 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:35:58.494104 4939 generic.go:356] "Generic (PLEG): container finished" podID="05a50b7a-195c-4ec9-9269-5d58faee868b" containerID="8d4b58e052a8f6f6d7c66b430f1e71b6f4a161f8f4de727bacc5e5b79b5a0a55" exitCode=0 Nov 03 01:35:58.494459 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:35:58.494170 4939 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-6fec8a-6cdc9b4ff5-xjq85" Nov 03 01:35:58.494695 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:35:58.494193 4939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-6fec8a-6cdc9b4ff5-xjq85" event={"ID":"05a50b7a-195c-4ec9-9269-5d58faee868b","Type":"ContainerDied","Data":"8d4b58e052a8f6f6d7c66b430f1e71b6f4a161f8f4de727bacc5e5b79b5a0a55"} Nov 03 01:35:58.494830 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:35:58.494814 4939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-6fec8a-6cdc9b4ff5-xjq85" event={"ID":"05a50b7a-195c-4ec9-9269-5d58faee868b","Type":"ContainerDied","Data":"7c7b01bd58665cbfd8d8ce6f0521be9e6e79f30f6ec8b9a3c14a7555995442f8"} Nov 03 01:35:58.494972 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:35:58.494956 4939 scope.go:122] "RemoveContainer" containerID="8d4b58e052a8f6f6d7c66b430f1e71b6f4a161f8f4de727bacc5e5b79b5a0a55" Nov 03 01:35:58.511366 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:35:58.511346 4939 scope.go:122] "RemoveContainer" containerID="8d4b58e052a8f6f6d7c66b430f1e71b6f4a161f8f4de727bacc5e5b79b5a0a55" Nov 03 01:35:58.511759 ip-10-0-140-107 kubenswrapper[4939]: E1103 01:35:58.511690 4939 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8d4b58e052a8f6f6d7c66b430f1e71b6f4a161f8f4de727bacc5e5b79b5a0a55\": container with ID starting with 8d4b58e052a8f6f6d7c66b430f1e71b6f4a161f8f4de727bacc5e5b79b5a0a55 not found: ID does not exist" containerID="8d4b58e052a8f6f6d7c66b430f1e71b6f4a161f8f4de727bacc5e5b79b5a0a55" Nov 03 01:35:58.511759 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:35:58.511728 4939 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8d4b58e052a8f6f6d7c66b430f1e71b6f4a161f8f4de727bacc5e5b79b5a0a55"} err="failed to get container status \"8d4b58e052a8f6f6d7c66b430f1e71b6f4a161f8f4de727bacc5e5b79b5a0a55\": rpc error: code = NotFound desc = could not find container \"8d4b58e052a8f6f6d7c66b430f1e71b6f4a161f8f4de727bacc5e5b79b5a0a55\": container with ID starting with 8d4b58e052a8f6f6d7c66b430f1e71b6f4a161f8f4de727bacc5e5b79b5a0a55 not found: ID does not exist" Nov 03 01:35:58.529033 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:35:58.529002 4939 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-6fec8a-6cdc9b4ff5-xjq85"] Nov 03 01:35:58.533248 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:35:58.533225 4939 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-6fec8a-6cdc9b4ff5-xjq85"] Nov 03 01:35:58.553481 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:35:58.553455 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:35:58.724544 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:35:58.724495 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:35:59.051211 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:35:59.051184 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:35:59.383954 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:35:59.383884 4939 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-6fec8a-openshift-default-5cc5f9b957-p9mtp"] Nov 03 01:35:59.384201 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:35:59.384171 4939 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-a-6fec8a-openshift-default-5cc5f9b957-p9mtp" podUID="7064128b-8764-4a6a-a4bc-e42b083f944f" containerName="istio-proxy" containerID="cri-o://3f3e26e3315a129208f02c6f66b93eb29a3f64905d3c5a0d0ebb33176a2cd2a5" gracePeriod=30 Nov 03 01:35:59.452241 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:35:59.452207 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:35:59.871585 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:35:59.871549 4939 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="05a50b7a-195c-4ec9-9269-5d58faee868b" path="/var/lib/kubelet/pods/05a50b7a-195c-4ec9-9269-5d58faee868b/volumes" Nov 03 01:35:59.993737 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:35:59.993701 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:36:00.249228 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:36:00.249130 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:36:00.732833 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:36:00.727229 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:36:01.096214 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:36:01.096174 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:36:01.217741 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:36:01.217709 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:36:01.381220 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:36:01.381138 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:36:01.588661 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:36:01.588628 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:36:01.913529 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:36:01.913479 4939 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-737bf2d8-openshift-default-56dc96b957-vswqf"] Nov 03 01:36:01.913769 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:36:01.913725 4939 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-737bf2d8-openshift-default-56dc96b957-vswqf" podUID="80a21f03-7cfa-49f0-b35b-df8edc58932e" containerName="istio-proxy" containerID="cri-o://1c960f788db7e960250ae0de477e4f2230f41cd1e4a7cf72bebca036c2eb6971" gracePeriod=30 Nov 03 01:36:01.918675 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:36:01.918646 4939 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-737bf2d8-openshift-default-56dc96b957-mwnjn"] Nov 03 01:36:01.918867 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:36:01.918834 4939 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-737bf2d8-openshift-default-56dc96b957-mwnjn" podUID="ae67a584-3143-44d0-a76f-97f60d25dd60" containerName="istio-proxy" containerID="cri-o://2e42335c8e69619af14f901ad8db624926d2018a34d85ff786f42c928145104e" gracePeriod=30 Nov 03 01:36:01.922826 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:36:01.922802 4939 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-737bf2d8-openshift-default-56dc96b957-jhxgw"] Nov 03 01:36:01.923029 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:36:01.922998 4939 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-737bf2d8-openshift-default-56dc96b957-jhxgw" podUID="cc9b0d3f-c18a-4e8f-ab5b-1c9d9d1b244c" containerName="istio-proxy" containerID="cri-o://457336f8e52c97c94cbb3e22be387a007e1af0b7e67300ab70768191782e4685" gracePeriod=30 Nov 03 01:36:01.974457 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:36:01.974429 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:36:02.223219 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:36:02.223136 4939 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-w2-ef434a/sub-test-model-ef434a-kserve-659ddb8595-zr7c2" Nov 03 01:36:02.236476 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:36:02.236446 4939 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-w2-ef434a/sub-test-model-ef434a-kserve-659ddb8595-zr7c2" Nov 03 01:36:02.437251 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:36:02.437214 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:36:02.568983 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:36:02.568489 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:36:02.701374 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:36:02.701341 4939 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-w2-ef434a/sub-test-model-ef434a-kserve-659ddb8595-zr7c2" Nov 03 01:36:02.801160 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:36:02.801123 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:36:03.186428 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:36:03.186391 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:36:03.648477 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:36:03.648445 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:36:03.862112 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:36:03.862075 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:36:04.509889 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:36:04.509862 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:36:04.659968 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:36:04.659837 4939 generic.go:356] "Generic (PLEG): container finished" podID="7064128b-8764-4a6a-a4bc-e42b083f944f" containerID="3f3e26e3315a129208f02c6f66b93eb29a3f64905d3c5a0d0ebb33176a2cd2a5" exitCode=0 Nov 03 01:36:04.659968 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:36:04.659934 4939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-6fec8a-openshift-default-5cc5f9b957-p9mtp" event={"ID":"7064128b-8764-4a6a-a4bc-e42b083f944f","Type":"ContainerDied","Data":"3f3e26e3315a129208f02c6f66b93eb29a3f64905d3c5a0d0ebb33176a2cd2a5"} Nov 03 01:36:04.660526 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:36:04.660014 4939 scope.go:122] "RemoveContainer" containerID="8717adbb605eabe2ae8a56d66659a3a6a308f1d9ae5fa997071b413addebbc6c" Nov 03 01:36:04.684085 ip-10-0-140-107 kubenswrapper[4939]: E1103 01:36:04.684050 4939 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6712b32f_f3bd_4917_9e5e_101cf604625c.slice/crio-384febeac8280063b68f71ab034887b1d84a8e149d013ded2756f3066aeecdf7.scope\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6712b32f_f3bd_4917_9e5e_101cf604625c.slice/crio-conmon-384febeac8280063b68f71ab034887b1d84a8e149d013ded2756f3066aeecdf7.scope\": RecentStats: unable to find data in memory cache]" Nov 03 01:36:04.730867 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:36:04.730839 4939 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-6fec8a-openshift-default-5cc5f9b957-p9mtp" Nov 03 01:36:04.855400 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:36:04.855360 4939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7064128b-8764-4a6a-a4bc-e42b083f944f-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7064128b-8764-4a6a-a4bc-e42b083f944f-credential-socket\") pod \"7064128b-8764-4a6a-a4bc-e42b083f944f\" (UID: \"7064128b-8764-4a6a-a4bc-e42b083f944f\") " Nov 03 01:36:04.855611 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:36:04.855595 4939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7064128b-8764-4a6a-a4bc-e42b083f944f-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7064128b-8764-4a6a-a4bc-e42b083f944f-workload-socket\") pod \"7064128b-8764-4a6a-a4bc-e42b083f944f\" (UID: \"7064128b-8764-4a6a-a4bc-e42b083f944f\") " Nov 03 01:36:04.855664 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:36:04.855646 4939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7064128b-8764-4a6a-a4bc-e42b083f944f-credential-socket" pod "7064128b-8764-4a6a-a4bc-e42b083f944f" (UID: "7064128b-8764-4a6a-a4bc-e42b083f944f"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 01:36:04.855720 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:36:04.855665 4939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/7064128b-8764-4a6a-a4bc-e42b083f944f-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7064128b-8764-4a6a-a4bc-e42b083f944f-istio-podinfo\") pod \"7064128b-8764-4a6a-a4bc-e42b083f944f\" (UID: \"7064128b-8764-4a6a-a4bc-e42b083f944f\") " Nov 03 01:36:04.855720 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:36:04.855702 4939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7064128b-8764-4a6a-a4bc-e42b083f944f-kube-api-access-f47tc\" (UniqueName: \"kubernetes.io/projected/7064128b-8764-4a6a-a4bc-e42b083f944f-kube-api-access-f47tc\") pod \"7064128b-8764-4a6a-a4bc-e42b083f944f\" (UID: \"7064128b-8764-4a6a-a4bc-e42b083f944f\") " Nov 03 01:36:04.855832 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:36:04.855735 4939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7064128b-8764-4a6a-a4bc-e42b083f944f-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7064128b-8764-4a6a-a4bc-e42b083f944f-istiod-ca-cert\") pod \"7064128b-8764-4a6a-a4bc-e42b083f944f\" (UID: \"7064128b-8764-4a6a-a4bc-e42b083f944f\") " Nov 03 01:36:04.855832 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:36:04.855776 4939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7064128b-8764-4a6a-a4bc-e42b083f944f-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7064128b-8764-4a6a-a4bc-e42b083f944f-workload-certs\") pod \"7064128b-8764-4a6a-a4bc-e42b083f944f\" (UID: \"7064128b-8764-4a6a-a4bc-e42b083f944f\") " Nov 03 01:36:04.855983 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:36:04.855827 4939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7064128b-8764-4a6a-a4bc-e42b083f944f-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7064128b-8764-4a6a-a4bc-e42b083f944f-istio-envoy\") pod \"7064128b-8764-4a6a-a4bc-e42b083f944f\" (UID: \"7064128b-8764-4a6a-a4bc-e42b083f944f\") " Nov 03 01:36:04.855983 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:36:04.855936 4939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7064128b-8764-4a6a-a4bc-e42b083f944f-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7064128b-8764-4a6a-a4bc-e42b083f944f-istio-data\") pod \"7064128b-8764-4a6a-a4bc-e42b083f944f\" (UID: \"7064128b-8764-4a6a-a4bc-e42b083f944f\") " Nov 03 01:36:04.856111 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:36:04.855996 4939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7064128b-8764-4a6a-a4bc-e42b083f944f-istio-token\" (UniqueName: \"kubernetes.io/projected/7064128b-8764-4a6a-a4bc-e42b083f944f-istio-token\") pod \"7064128b-8764-4a6a-a4bc-e42b083f944f\" (UID: \"7064128b-8764-4a6a-a4bc-e42b083f944f\") " Nov 03 01:36:04.856243 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:36:04.855931 4939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7064128b-8764-4a6a-a4bc-e42b083f944f-workload-socket" pod "7064128b-8764-4a6a-a4bc-e42b083f944f" (UID: "7064128b-8764-4a6a-a4bc-e42b083f944f"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 01:36:04.856342 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:36:04.856258 4939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7064128b-8764-4a6a-a4bc-e42b083f944f-istio-data" pod "7064128b-8764-4a6a-a4bc-e42b083f944f" (UID: "7064128b-8764-4a6a-a4bc-e42b083f944f"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 01:36:04.856342 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:36:04.856183 4939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7064128b-8764-4a6a-a4bc-e42b083f944f-istiod-ca-cert" pod "7064128b-8764-4a6a-a4bc-e42b083f944f" (UID: "7064128b-8764-4a6a-a4bc-e42b083f944f"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 01:36:04.856475 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:36:04.856331 4939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7064128b-8764-4a6a-a4bc-e42b083f944f-workload-certs" pod "7064128b-8764-4a6a-a4bc-e42b083f944f" (UID: "7064128b-8764-4a6a-a4bc-e42b083f944f"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 01:36:04.856864 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:36:04.856839 4939 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7064128b-8764-4a6a-a4bc-e42b083f944f-istio-data\") on node \"ip-10-0-140-107.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:36:04.856999 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:36:04.856873 4939 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7064128b-8764-4a6a-a4bc-e42b083f944f-credential-socket\") on node \"ip-10-0-140-107.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:36:04.856999 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:36:04.856890 4939 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7064128b-8764-4a6a-a4bc-e42b083f944f-workload-socket\") on node \"ip-10-0-140-107.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:36:04.856999 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:36:04.856904 4939 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7064128b-8764-4a6a-a4bc-e42b083f944f-istiod-ca-cert\") on node \"ip-10-0-140-107.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:36:04.856999 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:36:04.856979 4939 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7064128b-8764-4a6a-a4bc-e42b083f944f-workload-certs\") on node \"ip-10-0-140-107.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:36:04.858252 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:36:04.858193 4939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7064128b-8764-4a6a-a4bc-e42b083f944f-istio-envoy" pod "7064128b-8764-4a6a-a4bc-e42b083f944f" (UID: "7064128b-8764-4a6a-a4bc-e42b083f944f"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 01:36:04.858373 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:36:04.858320 4939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/7064128b-8764-4a6a-a4bc-e42b083f944f-istio-podinfo" pod "7064128b-8764-4a6a-a4bc-e42b083f944f" (UID: "7064128b-8764-4a6a-a4bc-e42b083f944f"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 01:36:04.858583 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:36:04.858563 4939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7064128b-8764-4a6a-a4bc-e42b083f944f-kube-api-access-f47tc" pod "7064128b-8764-4a6a-a4bc-e42b083f944f" (UID: "7064128b-8764-4a6a-a4bc-e42b083f944f"). InnerVolumeSpecName "kube-api-access-f47tc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 01:36:04.858661 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:36:04.858639 4939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7064128b-8764-4a6a-a4bc-e42b083f944f-istio-token" pod "7064128b-8764-4a6a-a4bc-e42b083f944f" (UID: "7064128b-8764-4a6a-a4bc-e42b083f944f"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 01:36:04.959157 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:36:04.959067 4939 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7064128b-8764-4a6a-a4bc-e42b083f944f-istio-podinfo\") on node \"ip-10-0-140-107.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:36:04.959157 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:36:04.959100 4939 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-f47tc\" (UniqueName: \"kubernetes.io/projected/7064128b-8764-4a6a-a4bc-e42b083f944f-kube-api-access-f47tc\") on node \"ip-10-0-140-107.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:36:04.959157 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:36:04.959114 4939 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7064128b-8764-4a6a-a4bc-e42b083f944f-istio-envoy\") on node \"ip-10-0-140-107.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:36:04.959157 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:36:04.959123 4939 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7064128b-8764-4a6a-a4bc-e42b083f944f-istio-token\") on node \"ip-10-0-140-107.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:36:05.275273 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:36:05.275238 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:36:05.582673 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:36:05.582632 4939 patch_prober.go:29] interesting pod/e2e-mt-737bf2d8-openshift-default-56dc96b957-vswqf container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 03 01:36:05.582873 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:36:05.582701 4939 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-mt-737bf2d8-openshift-default-56dc96b957-vswqf" podUID="80a21f03-7cfa-49f0-b35b-df8edc58932e" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 03 01:36:05.669379 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:36:05.669343 4939 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-6fec8a-openshift-default-5cc5f9b957-p9mtp" Nov 03 01:36:05.669379 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:36:05.669368 4939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-6fec8a-openshift-default-5cc5f9b957-p9mtp" event={"ID":"7064128b-8764-4a6a-a4bc-e42b083f944f","Type":"ContainerDied","Data":"38e3f4d1923ab0ea63036772ea798e0c253eaa986e52c13ffb3e7c9afaea0347"} Nov 03 01:36:05.669956 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:36:05.669457 4939 scope.go:122] "RemoveContainer" containerID="3f3e26e3315a129208f02c6f66b93eb29a3f64905d3c5a0d0ebb33176a2cd2a5" Nov 03 01:36:05.696358 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:36:05.696329 4939 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-6fec8a-openshift-default-5cc5f9b957-p9mtp"] Nov 03 01:36:05.700568 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:36:05.700543 4939 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-a-6fec8a-openshift-default-5cc5f9b957-p9mtp"] Nov 03 01:36:05.780464 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:36:05.780430 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:36:05.871901 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:36:05.871816 4939 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7064128b-8764-4a6a-a4bc-e42b083f944f" path="/var/lib/kubelet/pods/7064128b-8764-4a6a-a4bc-e42b083f944f/volumes" Nov 03 01:36:06.043642 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:36:06.043605 4939 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-5dab6a-57cf987799-9svqc"] Nov 03 01:36:06.122026 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:36:06.121022 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:36:06.236785 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:36:06.236750 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:36:06.648412 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:36:06.648370 4939 patch_prober.go:29] interesting pod/e2e-mt-737bf2d8-openshift-default-56dc96b957-jhxgw container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 03 01:36:06.648604 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:36:06.648432 4939 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-mt-737bf2d8-openshift-default-56dc96b957-jhxgw" podUID="cc9b0d3f-c18a-4e8f-ab5b-1c9d9d1b244c" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 03 01:36:06.668243 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:36:06.668213 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:36:06.675657 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:36:06.675619 4939 patch_prober.go:29] interesting pod/e2e-mt-737bf2d8-openshift-default-56dc96b957-mwnjn container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 03 01:36:06.676225 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:36:06.675745 4939 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-mt-737bf2d8-openshift-default-56dc96b957-mwnjn" podUID="ae67a584-3143-44d0-a76f-97f60d25dd60" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 03 01:36:06.774672 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:36:06.774630 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:36:07.237735 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:36:07.237703 4939 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-737bf2d8-openshift-default-56dc96b957-mwnjn" Nov 03 01:36:07.293519 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:36:07.293484 4939 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-737bf2d8-openshift-default-56dc96b957-vswqf" Nov 03 01:36:07.322260 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:36:07.322222 4939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ae67a584-3143-44d0-a76f-97f60d25dd60-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ae67a584-3143-44d0-a76f-97f60d25dd60-workload-certs\") pod \"ae67a584-3143-44d0-a76f-97f60d25dd60\" (UID: \"ae67a584-3143-44d0-a76f-97f60d25dd60\") " Nov 03 01:36:07.322466 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:36:07.322272 4939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/ae67a584-3143-44d0-a76f-97f60d25dd60-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ae67a584-3143-44d0-a76f-97f60d25dd60-istio-podinfo\") pod \"ae67a584-3143-44d0-a76f-97f60d25dd60\" (UID: \"ae67a584-3143-44d0-a76f-97f60d25dd60\") " Nov 03 01:36:07.322566 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:36:07.322549 4939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ae67a584-3143-44d0-a76f-97f60d25dd60-istio-token\" (UniqueName: \"kubernetes.io/projected/ae67a584-3143-44d0-a76f-97f60d25dd60-istio-token\") pod \"ae67a584-3143-44d0-a76f-97f60d25dd60\" (UID: \"ae67a584-3143-44d0-a76f-97f60d25dd60\") " Nov 03 01:36:07.322636 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:36:07.322613 4939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ae67a584-3143-44d0-a76f-97f60d25dd60-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ae67a584-3143-44d0-a76f-97f60d25dd60-istio-envoy\") pod \"ae67a584-3143-44d0-a76f-97f60d25dd60\" (UID: \"ae67a584-3143-44d0-a76f-97f60d25dd60\") " Nov 03 01:36:07.322711 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:36:07.322697 4939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ae67a584-3143-44d0-a76f-97f60d25dd60-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ae67a584-3143-44d0-a76f-97f60d25dd60-istio-data\") pod \"ae67a584-3143-44d0-a76f-97f60d25dd60\" (UID: \"ae67a584-3143-44d0-a76f-97f60d25dd60\") " Nov 03 01:36:07.322797 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:36:07.322783 4939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ae67a584-3143-44d0-a76f-97f60d25dd60-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ae67a584-3143-44d0-a76f-97f60d25dd60-istiod-ca-cert\") pod \"ae67a584-3143-44d0-a76f-97f60d25dd60\" (UID: \"ae67a584-3143-44d0-a76f-97f60d25dd60\") " Nov 03 01:36:07.322871 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:36:07.322849 4939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ae67a584-3143-44d0-a76f-97f60d25dd60-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ae67a584-3143-44d0-a76f-97f60d25dd60-credential-socket\") pod \"ae67a584-3143-44d0-a76f-97f60d25dd60\" (UID: \"ae67a584-3143-44d0-a76f-97f60d25dd60\") " Nov 03 01:36:07.322991 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:36:07.322880 4939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ae67a584-3143-44d0-a76f-97f60d25dd60-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ae67a584-3143-44d0-a76f-97f60d25dd60-workload-socket\") pod \"ae67a584-3143-44d0-a76f-97f60d25dd60\" (UID: \"ae67a584-3143-44d0-a76f-97f60d25dd60\") " Nov 03 01:36:07.322991 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:36:07.322975 4939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ae67a584-3143-44d0-a76f-97f60d25dd60-kube-api-access-9299j\" (UniqueName: \"kubernetes.io/projected/ae67a584-3143-44d0-a76f-97f60d25dd60-kube-api-access-9299j\") pod \"ae67a584-3143-44d0-a76f-97f60d25dd60\" (UID: \"ae67a584-3143-44d0-a76f-97f60d25dd60\") " Nov 03 01:36:07.324673 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:36:07.324566 4939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ae67a584-3143-44d0-a76f-97f60d25dd60-istio-data" pod "ae67a584-3143-44d0-a76f-97f60d25dd60" (UID: "ae67a584-3143-44d0-a76f-97f60d25dd60"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 01:36:07.325178 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:36:07.325139 4939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ae67a584-3143-44d0-a76f-97f60d25dd60-workload-certs" pod "ae67a584-3143-44d0-a76f-97f60d25dd60" (UID: "ae67a584-3143-44d0-a76f-97f60d25dd60"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 01:36:07.325783 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:36:07.325757 4939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ae67a584-3143-44d0-a76f-97f60d25dd60-workload-socket" pod "ae67a584-3143-44d0-a76f-97f60d25dd60" (UID: "ae67a584-3143-44d0-a76f-97f60d25dd60"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 01:36:07.325887 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:36:07.325820 4939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ae67a584-3143-44d0-a76f-97f60d25dd60-credential-socket" pod "ae67a584-3143-44d0-a76f-97f60d25dd60" (UID: "ae67a584-3143-44d0-a76f-97f60d25dd60"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 01:36:07.326454 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:36:07.326429 4939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ae67a584-3143-44d0-a76f-97f60d25dd60-istiod-ca-cert" pod "ae67a584-3143-44d0-a76f-97f60d25dd60" (UID: "ae67a584-3143-44d0-a76f-97f60d25dd60"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 01:36:07.327621 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:36:07.327580 4939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ae67a584-3143-44d0-a76f-97f60d25dd60-kube-api-access-9299j" pod "ae67a584-3143-44d0-a76f-97f60d25dd60" (UID: "ae67a584-3143-44d0-a76f-97f60d25dd60"). InnerVolumeSpecName "kube-api-access-9299j". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 01:36:07.328657 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:36:07.328631 4939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/ae67a584-3143-44d0-a76f-97f60d25dd60-istio-podinfo" pod "ae67a584-3143-44d0-a76f-97f60d25dd60" (UID: "ae67a584-3143-44d0-a76f-97f60d25dd60"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 01:36:07.329609 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:36:07.329574 4939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ae67a584-3143-44d0-a76f-97f60d25dd60-istio-token" pod "ae67a584-3143-44d0-a76f-97f60d25dd60" (UID: "ae67a584-3143-44d0-a76f-97f60d25dd60"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 01:36:07.330581 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:36:07.330558 4939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ae67a584-3143-44d0-a76f-97f60d25dd60-istio-envoy" pod "ae67a584-3143-44d0-a76f-97f60d25dd60" (UID: "ae67a584-3143-44d0-a76f-97f60d25dd60"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 01:36:07.337101 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:36:07.337077 4939 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-737bf2d8-openshift-default-56dc96b957-jhxgw" Nov 03 01:36:07.424543 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:36:07.424497 4939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cc9b0d3f-c18a-4e8f-ab5b-1c9d9d1b244c-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cc9b0d3f-c18a-4e8f-ab5b-1c9d9d1b244c-workload-certs\") pod \"cc9b0d3f-c18a-4e8f-ab5b-1c9d9d1b244c\" (UID: \"cc9b0d3f-c18a-4e8f-ab5b-1c9d9d1b244c\") " Nov 03 01:36:07.424751 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:36:07.424573 4939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/80a21f03-7cfa-49f0-b35b-df8edc58932e-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/80a21f03-7cfa-49f0-b35b-df8edc58932e-workload-certs\") pod \"80a21f03-7cfa-49f0-b35b-df8edc58932e\" (UID: \"80a21f03-7cfa-49f0-b35b-df8edc58932e\") " Nov 03 01:36:07.424751 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:36:07.424630 4939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cc9b0d3f-c18a-4e8f-ab5b-1c9d9d1b244c-kube-api-access-6sw2h\" (UniqueName: \"kubernetes.io/projected/cc9b0d3f-c18a-4e8f-ab5b-1c9d9d1b244c-kube-api-access-6sw2h\") pod \"cc9b0d3f-c18a-4e8f-ab5b-1c9d9d1b244c\" (UID: \"cc9b0d3f-c18a-4e8f-ab5b-1c9d9d1b244c\") " Nov 03 01:36:07.425208 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:36:07.424769 4939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cc9b0d3f-c18a-4e8f-ab5b-1c9d9d1b244c-istio-token\" (UniqueName: \"kubernetes.io/projected/cc9b0d3f-c18a-4e8f-ab5b-1c9d9d1b244c-istio-token\") pod \"cc9b0d3f-c18a-4e8f-ab5b-1c9d9d1b244c\" (UID: \"cc9b0d3f-c18a-4e8f-ab5b-1c9d9d1b244c\") " Nov 03 01:36:07.425208 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:36:07.424804 4939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/80a21f03-7cfa-49f0-b35b-df8edc58932e-istio-token\" (UniqueName: \"kubernetes.io/projected/80a21f03-7cfa-49f0-b35b-df8edc58932e-istio-token\") pod \"80a21f03-7cfa-49f0-b35b-df8edc58932e\" (UID: \"80a21f03-7cfa-49f0-b35b-df8edc58932e\") " Nov 03 01:36:07.425208 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:36:07.424864 4939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/80a21f03-7cfa-49f0-b35b-df8edc58932e-workload-certs" pod "80a21f03-7cfa-49f0-b35b-df8edc58932e" (UID: "80a21f03-7cfa-49f0-b35b-df8edc58932e"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 01:36:07.425208 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:36:07.424938 4939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cc9b0d3f-c18a-4e8f-ab5b-1c9d9d1b244c-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cc9b0d3f-c18a-4e8f-ab5b-1c9d9d1b244c-istio-data\") pod \"cc9b0d3f-c18a-4e8f-ab5b-1c9d9d1b244c\" (UID: \"cc9b0d3f-c18a-4e8f-ab5b-1c9d9d1b244c\") " Nov 03 01:36:07.425208 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:36:07.425119 4939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cc9b0d3f-c18a-4e8f-ab5b-1c9d9d1b244c-workload-certs" pod "cc9b0d3f-c18a-4e8f-ab5b-1c9d9d1b244c" (UID: "cc9b0d3f-c18a-4e8f-ab5b-1c9d9d1b244c"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 01:36:07.425540 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:36:07.425355 4939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/80a21f03-7cfa-49f0-b35b-df8edc58932e-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/80a21f03-7cfa-49f0-b35b-df8edc58932e-istio-data\") pod \"80a21f03-7cfa-49f0-b35b-df8edc58932e\" (UID: \"80a21f03-7cfa-49f0-b35b-df8edc58932e\") " Nov 03 01:36:07.425540 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:36:07.425395 4939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/80a21f03-7cfa-49f0-b35b-df8edc58932e-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/80a21f03-7cfa-49f0-b35b-df8edc58932e-istio-envoy\") pod \"80a21f03-7cfa-49f0-b35b-df8edc58932e\" (UID: \"80a21f03-7cfa-49f0-b35b-df8edc58932e\") " Nov 03 01:36:07.425540 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:36:07.425481 4939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/80a21f03-7cfa-49f0-b35b-df8edc58932e-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/80a21f03-7cfa-49f0-b35b-df8edc58932e-workload-socket\") pod \"80a21f03-7cfa-49f0-b35b-df8edc58932e\" (UID: \"80a21f03-7cfa-49f0-b35b-df8edc58932e\") " Nov 03 01:36:07.425734 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:36:07.425545 4939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/80a21f03-7cfa-49f0-b35b-df8edc58932e-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/80a21f03-7cfa-49f0-b35b-df8edc58932e-credential-socket\") pod \"80a21f03-7cfa-49f0-b35b-df8edc58932e\" (UID: \"80a21f03-7cfa-49f0-b35b-df8edc58932e\") " Nov 03 01:36:07.425734 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:36:07.425585 4939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/cc9b0d3f-c18a-4e8f-ab5b-1c9d9d1b244c-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cc9b0d3f-c18a-4e8f-ab5b-1c9d9d1b244c-istio-podinfo\") pod \"cc9b0d3f-c18a-4e8f-ab5b-1c9d9d1b244c\" (UID: \"cc9b0d3f-c18a-4e8f-ab5b-1c9d9d1b244c\") " Nov 03 01:36:07.425734 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:36:07.425681 4939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cc9b0d3f-c18a-4e8f-ab5b-1c9d9d1b244c-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cc9b0d3f-c18a-4e8f-ab5b-1c9d9d1b244c-istio-envoy\") pod \"cc9b0d3f-c18a-4e8f-ab5b-1c9d9d1b244c\" (UID: \"cc9b0d3f-c18a-4e8f-ab5b-1c9d9d1b244c\") " Nov 03 01:36:07.425897 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:36:07.425748 4939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/80a21f03-7cfa-49f0-b35b-df8edc58932e-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/80a21f03-7cfa-49f0-b35b-df8edc58932e-istiod-ca-cert\") pod \"80a21f03-7cfa-49f0-b35b-df8edc58932e\" (UID: \"80a21f03-7cfa-49f0-b35b-df8edc58932e\") " Nov 03 01:36:07.425897 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:36:07.425780 4939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cc9b0d3f-c18a-4e8f-ab5b-1c9d9d1b244c-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cc9b0d3f-c18a-4e8f-ab5b-1c9d9d1b244c-credential-socket\") pod \"cc9b0d3f-c18a-4e8f-ab5b-1c9d9d1b244c\" (UID: \"cc9b0d3f-c18a-4e8f-ab5b-1c9d9d1b244c\") " Nov 03 01:36:07.425897 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:36:07.425840 4939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/80a21f03-7cfa-49f0-b35b-df8edc58932e-kube-api-access-mfpzb\" (UniqueName: \"kubernetes.io/projected/80a21f03-7cfa-49f0-b35b-df8edc58932e-kube-api-access-mfpzb\") pod \"80a21f03-7cfa-49f0-b35b-df8edc58932e\" (UID: \"80a21f03-7cfa-49f0-b35b-df8edc58932e\") " Nov 03 01:36:07.425897 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:36:07.425853 4939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/80a21f03-7cfa-49f0-b35b-df8edc58932e-workload-socket" pod "80a21f03-7cfa-49f0-b35b-df8edc58932e" (UID: "80a21f03-7cfa-49f0-b35b-df8edc58932e"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 01:36:07.425897 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:36:07.425889 4939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cc9b0d3f-c18a-4e8f-ab5b-1c9d9d1b244c-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cc9b0d3f-c18a-4e8f-ab5b-1c9d9d1b244c-workload-socket\") pod \"cc9b0d3f-c18a-4e8f-ab5b-1c9d9d1b244c\" (UID: \"cc9b0d3f-c18a-4e8f-ab5b-1c9d9d1b244c\") " Nov 03 01:36:07.426196 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:36:07.425933 4939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cc9b0d3f-c18a-4e8f-ab5b-1c9d9d1b244c-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cc9b0d3f-c18a-4e8f-ab5b-1c9d9d1b244c-istiod-ca-cert\") pod \"cc9b0d3f-c18a-4e8f-ab5b-1c9d9d1b244c\" (UID: \"cc9b0d3f-c18a-4e8f-ab5b-1c9d9d1b244c\") " Nov 03 01:36:07.426359 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:36:07.426334 4939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/80a21f03-7cfa-49f0-b35b-df8edc58932e-istiod-ca-cert" pod "80a21f03-7cfa-49f0-b35b-df8edc58932e" (UID: "80a21f03-7cfa-49f0-b35b-df8edc58932e"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 01:36:07.426452 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:36:07.426391 4939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/80a21f03-7cfa-49f0-b35b-df8edc58932e-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/80a21f03-7cfa-49f0-b35b-df8edc58932e-istio-podinfo\") pod \"80a21f03-7cfa-49f0-b35b-df8edc58932e\" (UID: \"80a21f03-7cfa-49f0-b35b-df8edc58932e\") " Nov 03 01:36:07.426452 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:36:07.426420 4939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cc9b0d3f-c18a-4e8f-ab5b-1c9d9d1b244c-istio-data" pod "cc9b0d3f-c18a-4e8f-ab5b-1c9d9d1b244c" (UID: "cc9b0d3f-c18a-4e8f-ab5b-1c9d9d1b244c"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 01:36:07.426741 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:36:07.426716 4939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cc9b0d3f-c18a-4e8f-ab5b-1c9d9d1b244c-credential-socket" pod "cc9b0d3f-c18a-4e8f-ab5b-1c9d9d1b244c" (UID: "cc9b0d3f-c18a-4e8f-ab5b-1c9d9d1b244c"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 01:36:07.427117 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:36:07.427090 4939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/80a21f03-7cfa-49f0-b35b-df8edc58932e-credential-socket" pod "80a21f03-7cfa-49f0-b35b-df8edc58932e" (UID: "80a21f03-7cfa-49f0-b35b-df8edc58932e"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 01:36:07.428034 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:36:07.427301 4939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cc9b0d3f-c18a-4e8f-ab5b-1c9d9d1b244c-workload-socket" pod "cc9b0d3f-c18a-4e8f-ab5b-1c9d9d1b244c" (UID: "cc9b0d3f-c18a-4e8f-ab5b-1c9d9d1b244c"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 01:36:07.428034 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:36:07.427445 4939 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ae67a584-3143-44d0-a76f-97f60d25dd60-istio-envoy\") on node \"ip-10-0-140-107.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:36:07.428034 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:36:07.427457 4939 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ae67a584-3143-44d0-a76f-97f60d25dd60-istio-data\") on node \"ip-10-0-140-107.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:36:07.428034 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:36:07.427466 4939 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cc9b0d3f-c18a-4e8f-ab5b-1c9d9d1b244c-istio-data\") on node \"ip-10-0-140-107.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:36:07.428034 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:36:07.427476 4939 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ae67a584-3143-44d0-a76f-97f60d25dd60-istiod-ca-cert\") on node \"ip-10-0-140-107.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:36:07.428034 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:36:07.427486 4939 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ae67a584-3143-44d0-a76f-97f60d25dd60-credential-socket\") on node \"ip-10-0-140-107.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:36:07.428034 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:36:07.427496 4939 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ae67a584-3143-44d0-a76f-97f60d25dd60-workload-socket\") on node \"ip-10-0-140-107.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:36:07.428034 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:36:07.427505 4939 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9299j\" (UniqueName: \"kubernetes.io/projected/ae67a584-3143-44d0-a76f-97f60d25dd60-kube-api-access-9299j\") on node \"ip-10-0-140-107.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:36:07.428034 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:36:07.427514 4939 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/80a21f03-7cfa-49f0-b35b-df8edc58932e-workload-socket\") on node \"ip-10-0-140-107.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:36:07.428034 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:36:07.427523 4939 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/80a21f03-7cfa-49f0-b35b-df8edc58932e-credential-socket\") on node \"ip-10-0-140-107.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:36:07.428034 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:36:07.427534 4939 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ae67a584-3143-44d0-a76f-97f60d25dd60-workload-certs\") on node \"ip-10-0-140-107.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:36:07.428034 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:36:07.427547 4939 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ae67a584-3143-44d0-a76f-97f60d25dd60-istio-podinfo\") on node \"ip-10-0-140-107.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:36:07.428034 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:36:07.427563 4939 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/80a21f03-7cfa-49f0-b35b-df8edc58932e-istiod-ca-cert\") on node \"ip-10-0-140-107.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:36:07.428034 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:36:07.427576 4939 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cc9b0d3f-c18a-4e8f-ab5b-1c9d9d1b244c-credential-socket\") on node \"ip-10-0-140-107.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:36:07.428034 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:36:07.427591 4939 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cc9b0d3f-c18a-4e8f-ab5b-1c9d9d1b244c-workload-socket\") on node \"ip-10-0-140-107.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:36:07.428034 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:36:07.427604 4939 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cc9b0d3f-c18a-4e8f-ab5b-1c9d9d1b244c-workload-certs\") on node \"ip-10-0-140-107.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:36:07.428034 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:36:07.427618 4939 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/80a21f03-7cfa-49f0-b35b-df8edc58932e-workload-certs\") on node \"ip-10-0-140-107.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:36:07.428034 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:36:07.427629 4939 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ae67a584-3143-44d0-a76f-97f60d25dd60-istio-token\") on node \"ip-10-0-140-107.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:36:07.428034 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:36:07.427674 4939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/80a21f03-7cfa-49f0-b35b-df8edc58932e-istio-token" pod "80a21f03-7cfa-49f0-b35b-df8edc58932e" (UID: "80a21f03-7cfa-49f0-b35b-df8edc58932e"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 01:36:07.428034 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:36:07.428020 4939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/80a21f03-7cfa-49f0-b35b-df8edc58932e-istio-data" pod "80a21f03-7cfa-49f0-b35b-df8edc58932e" (UID: "80a21f03-7cfa-49f0-b35b-df8edc58932e"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 01:36:07.429217 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:36:07.427946 4939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cc9b0d3f-c18a-4e8f-ab5b-1c9d9d1b244c-istiod-ca-cert" pod "cc9b0d3f-c18a-4e8f-ab5b-1c9d9d1b244c" (UID: "cc9b0d3f-c18a-4e8f-ab5b-1c9d9d1b244c"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 01:36:07.430381 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:36:07.430356 4939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/cc9b0d3f-c18a-4e8f-ab5b-1c9d9d1b244c-istio-podinfo" pod "cc9b0d3f-c18a-4e8f-ab5b-1c9d9d1b244c" (UID: "cc9b0d3f-c18a-4e8f-ab5b-1c9d9d1b244c"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 01:36:07.430629 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:36:07.430601 4939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/80a21f03-7cfa-49f0-b35b-df8edc58932e-istio-envoy" pod "80a21f03-7cfa-49f0-b35b-df8edc58932e" (UID: "80a21f03-7cfa-49f0-b35b-df8edc58932e"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 01:36:07.430838 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:36:07.430815 4939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cc9b0d3f-c18a-4e8f-ab5b-1c9d9d1b244c-istio-token" pod "cc9b0d3f-c18a-4e8f-ab5b-1c9d9d1b244c" (UID: "cc9b0d3f-c18a-4e8f-ab5b-1c9d9d1b244c"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 01:36:07.431642 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:36:07.431615 4939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/80a21f03-7cfa-49f0-b35b-df8edc58932e-istio-podinfo" pod "80a21f03-7cfa-49f0-b35b-df8edc58932e" (UID: "80a21f03-7cfa-49f0-b35b-df8edc58932e"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 01:36:07.432062 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:36:07.432021 4939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/80a21f03-7cfa-49f0-b35b-df8edc58932e-kube-api-access-mfpzb" pod "80a21f03-7cfa-49f0-b35b-df8edc58932e" (UID: "80a21f03-7cfa-49f0-b35b-df8edc58932e"). InnerVolumeSpecName "kube-api-access-mfpzb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 01:36:07.432226 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:36:07.432201 4939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cc9b0d3f-c18a-4e8f-ab5b-1c9d9d1b244c-kube-api-access-6sw2h" pod "cc9b0d3f-c18a-4e8f-ab5b-1c9d9d1b244c" (UID: "cc9b0d3f-c18a-4e8f-ab5b-1c9d9d1b244c"). InnerVolumeSpecName "kube-api-access-6sw2h". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 01:36:07.432366 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:36:07.432343 4939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cc9b0d3f-c18a-4e8f-ab5b-1c9d9d1b244c-istio-envoy" pod "cc9b0d3f-c18a-4e8f-ab5b-1c9d9d1b244c" (UID: "cc9b0d3f-c18a-4e8f-ab5b-1c9d9d1b244c"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 01:36:07.501547 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:36:07.501467 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:36:07.536066 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:36:07.529433 4939 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6sw2h\" (UniqueName: \"kubernetes.io/projected/cc9b0d3f-c18a-4e8f-ab5b-1c9d9d1b244c-kube-api-access-6sw2h\") on node \"ip-10-0-140-107.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:36:07.536066 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:36:07.529477 4939 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cc9b0d3f-c18a-4e8f-ab5b-1c9d9d1b244c-istio-token\") on node \"ip-10-0-140-107.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:36:07.536066 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:36:07.529494 4939 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/80a21f03-7cfa-49f0-b35b-df8edc58932e-istio-token\") on node \"ip-10-0-140-107.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:36:07.536066 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:36:07.529508 4939 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/80a21f03-7cfa-49f0-b35b-df8edc58932e-istio-data\") on node \"ip-10-0-140-107.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:36:07.536066 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:36:07.529524 4939 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/80a21f03-7cfa-49f0-b35b-df8edc58932e-istio-envoy\") on node \"ip-10-0-140-107.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:36:07.536066 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:36:07.529540 4939 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cc9b0d3f-c18a-4e8f-ab5b-1c9d9d1b244c-istio-podinfo\") on node \"ip-10-0-140-107.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:36:07.536066 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:36:07.529554 4939 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cc9b0d3f-c18a-4e8f-ab5b-1c9d9d1b244c-istio-envoy\") on node \"ip-10-0-140-107.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:36:07.536066 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:36:07.529569 4939 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mfpzb\" (UniqueName: \"kubernetes.io/projected/80a21f03-7cfa-49f0-b35b-df8edc58932e-kube-api-access-mfpzb\") on node \"ip-10-0-140-107.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:36:07.536066 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:36:07.529583 4939 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cc9b0d3f-c18a-4e8f-ab5b-1c9d9d1b244c-istiod-ca-cert\") on node \"ip-10-0-140-107.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:36:07.536066 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:36:07.529596 4939 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/80a21f03-7cfa-49f0-b35b-df8edc58932e-istio-podinfo\") on node \"ip-10-0-140-107.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:36:07.693179 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:36:07.693140 4939 generic.go:356] "Generic (PLEG): container finished" podID="ae67a584-3143-44d0-a76f-97f60d25dd60" containerID="2e42335c8e69619af14f901ad8db624926d2018a34d85ff786f42c928145104e" exitCode=0 Nov 03 01:36:07.693649 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:36:07.693222 4939 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-737bf2d8-openshift-default-56dc96b957-mwnjn" Nov 03 01:36:07.693649 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:36:07.693238 4939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-737bf2d8-openshift-default-56dc96b957-mwnjn" event={"ID":"ae67a584-3143-44d0-a76f-97f60d25dd60","Type":"ContainerDied","Data":"2e42335c8e69619af14f901ad8db624926d2018a34d85ff786f42c928145104e"} Nov 03 01:36:07.693649 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:36:07.693305 4939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-737bf2d8-openshift-default-56dc96b957-mwnjn" event={"ID":"ae67a584-3143-44d0-a76f-97f60d25dd60","Type":"ContainerDied","Data":"667bef9233812c01fbeb59ddb95cd5c23bc15f4c43fc8d95150a5663fd54ea18"} Nov 03 01:36:07.693649 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:36:07.693340 4939 scope.go:122] "RemoveContainer" containerID="2e42335c8e69619af14f901ad8db624926d2018a34d85ff786f42c928145104e" Nov 03 01:36:07.695229 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:36:07.695206 4939 generic.go:356] "Generic (PLEG): container finished" podID="cc9b0d3f-c18a-4e8f-ab5b-1c9d9d1b244c" containerID="457336f8e52c97c94cbb3e22be387a007e1af0b7e67300ab70768191782e4685" exitCode=0 Nov 03 01:36:07.695321 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:36:07.695262 4939 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-737bf2d8-openshift-default-56dc96b957-jhxgw" Nov 03 01:36:07.695321 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:36:07.695279 4939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-737bf2d8-openshift-default-56dc96b957-jhxgw" event={"ID":"cc9b0d3f-c18a-4e8f-ab5b-1c9d9d1b244c","Type":"ContainerDied","Data":"457336f8e52c97c94cbb3e22be387a007e1af0b7e67300ab70768191782e4685"} Nov 03 01:36:07.695321 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:36:07.695315 4939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-737bf2d8-openshift-default-56dc96b957-jhxgw" event={"ID":"cc9b0d3f-c18a-4e8f-ab5b-1c9d9d1b244c","Type":"ContainerDied","Data":"05909e1c44fc0a0adb074b952d5c1d20536ec2f60c7514160a053a62b272597e"} Nov 03 01:36:07.697566 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:36:07.697540 4939 generic.go:356] "Generic (PLEG): container finished" podID="80a21f03-7cfa-49f0-b35b-df8edc58932e" containerID="1c960f788db7e960250ae0de477e4f2230f41cd1e4a7cf72bebca036c2eb6971" exitCode=0 Nov 03 01:36:07.697737 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:36:07.697629 4939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-737bf2d8-openshift-default-56dc96b957-vswqf" event={"ID":"80a21f03-7cfa-49f0-b35b-df8edc58932e","Type":"ContainerDied","Data":"1c960f788db7e960250ae0de477e4f2230f41cd1e4a7cf72bebca036c2eb6971"} Nov 03 01:36:07.697737 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:36:07.697670 4939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-737bf2d8-openshift-default-56dc96b957-vswqf" event={"ID":"80a21f03-7cfa-49f0-b35b-df8edc58932e","Type":"ContainerDied","Data":"e9efcfc60b3f95ba4b3a09c263e53ada2a466d49acf5efaced2cb2390fba09a0"} Nov 03 01:36:07.697860 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:36:07.697748 4939 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-737bf2d8-openshift-default-56dc96b957-vswqf" Nov 03 01:36:07.708057 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:36:07.707887 4939 scope.go:122] "RemoveContainer" containerID="2e42335c8e69619af14f901ad8db624926d2018a34d85ff786f42c928145104e" Nov 03 01:36:07.708809 ip-10-0-140-107 kubenswrapper[4939]: E1103 01:36:07.708212 4939 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2e42335c8e69619af14f901ad8db624926d2018a34d85ff786f42c928145104e\": container with ID starting with 2e42335c8e69619af14f901ad8db624926d2018a34d85ff786f42c928145104e not found: ID does not exist" containerID="2e42335c8e69619af14f901ad8db624926d2018a34d85ff786f42c928145104e" Nov 03 01:36:07.708809 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:36:07.708245 4939 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2e42335c8e69619af14f901ad8db624926d2018a34d85ff786f42c928145104e"} err="failed to get container status \"2e42335c8e69619af14f901ad8db624926d2018a34d85ff786f42c928145104e\": rpc error: code = NotFound desc = could not find container \"2e42335c8e69619af14f901ad8db624926d2018a34d85ff786f42c928145104e\": container with ID starting with 2e42335c8e69619af14f901ad8db624926d2018a34d85ff786f42c928145104e not found: ID does not exist" Nov 03 01:36:07.708809 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:36:07.708268 4939 scope.go:122] "RemoveContainer" containerID="457336f8e52c97c94cbb3e22be387a007e1af0b7e67300ab70768191782e4685" Nov 03 01:36:07.720535 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:36:07.720475 4939 scope.go:122] "RemoveContainer" containerID="457336f8e52c97c94cbb3e22be387a007e1af0b7e67300ab70768191782e4685" Nov 03 01:36:07.721006 ip-10-0-140-107 kubenswrapper[4939]: E1103 01:36:07.720885 4939 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"457336f8e52c97c94cbb3e22be387a007e1af0b7e67300ab70768191782e4685\": container with ID starting with 457336f8e52c97c94cbb3e22be387a007e1af0b7e67300ab70768191782e4685 not found: ID does not exist" containerID="457336f8e52c97c94cbb3e22be387a007e1af0b7e67300ab70768191782e4685" Nov 03 01:36:07.721006 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:36:07.720932 4939 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"457336f8e52c97c94cbb3e22be387a007e1af0b7e67300ab70768191782e4685"} err="failed to get container status \"457336f8e52c97c94cbb3e22be387a007e1af0b7e67300ab70768191782e4685\": rpc error: code = NotFound desc = could not find container \"457336f8e52c97c94cbb3e22be387a007e1af0b7e67300ab70768191782e4685\": container with ID starting with 457336f8e52c97c94cbb3e22be387a007e1af0b7e67300ab70768191782e4685 not found: ID does not exist" Nov 03 01:36:07.721006 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:36:07.720953 4939 scope.go:122] "RemoveContainer" containerID="1c960f788db7e960250ae0de477e4f2230f41cd1e4a7cf72bebca036c2eb6971" Nov 03 01:36:07.726212 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:36:07.726190 4939 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-737bf2d8-openshift-default-56dc96b957-jhxgw"] Nov 03 01:36:07.730553 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:36:07.730530 4939 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-737bf2d8-openshift-default-56dc96b957-jhxgw"] Nov 03 01:36:07.733472 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:36:07.733451 4939 scope.go:122] "RemoveContainer" containerID="1c960f788db7e960250ae0de477e4f2230f41cd1e4a7cf72bebca036c2eb6971" Nov 03 01:36:07.733786 ip-10-0-140-107 kubenswrapper[4939]: E1103 01:36:07.733759 4939 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1c960f788db7e960250ae0de477e4f2230f41cd1e4a7cf72bebca036c2eb6971\": container with ID starting with 1c960f788db7e960250ae0de477e4f2230f41cd1e4a7cf72bebca036c2eb6971 not found: ID does not exist" containerID="1c960f788db7e960250ae0de477e4f2230f41cd1e4a7cf72bebca036c2eb6971" Nov 03 01:36:07.733856 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:36:07.733793 4939 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1c960f788db7e960250ae0de477e4f2230f41cd1e4a7cf72bebca036c2eb6971"} err="failed to get container status \"1c960f788db7e960250ae0de477e4f2230f41cd1e4a7cf72bebca036c2eb6971\": rpc error: code = NotFound desc = could not find container \"1c960f788db7e960250ae0de477e4f2230f41cd1e4a7cf72bebca036c2eb6971\": container with ID starting with 1c960f788db7e960250ae0de477e4f2230f41cd1e4a7cf72bebca036c2eb6971 not found: ID does not exist" Nov 03 01:36:07.742083 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:36:07.742057 4939 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-737bf2d8-openshift-default-56dc96b957-mwnjn"] Nov 03 01:36:07.747050 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:36:07.747028 4939 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-737bf2d8-openshift-default-56dc96b957-mwnjn"] Nov 03 01:36:07.758471 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:36:07.758359 4939 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-737bf2d8-openshift-default-56dc96b957-vswqf"] Nov 03 01:36:07.762387 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:36:07.762365 4939 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-737bf2d8-openshift-default-56dc96b957-vswqf"] Nov 03 01:36:07.874486 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:36:07.874451 4939 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="80a21f03-7cfa-49f0-b35b-df8edc58932e" path="/var/lib/kubelet/pods/80a21f03-7cfa-49f0-b35b-df8edc58932e/volumes" Nov 03 01:36:07.875082 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:36:07.875065 4939 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ae67a584-3143-44d0-a76f-97f60d25dd60" path="/var/lib/kubelet/pods/ae67a584-3143-44d0-a76f-97f60d25dd60/volumes" Nov 03 01:36:07.875652 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:36:07.875633 4939 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cc9b0d3f-c18a-4e8f-ab5b-1c9d9d1b244c" path="/var/lib/kubelet/pods/cc9b0d3f-c18a-4e8f-ab5b-1c9d9d1b244c/volumes" Nov 03 01:36:08.181989 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:36:08.181953 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:36:08.909048 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:36:08.909009 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:36:09.216726 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:36:09.216640 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:36:09.417680 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:36:09.417647 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:36:09.655444 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:36:09.655412 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:36:09.782660 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:36:09.782630 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:36:10.159944 ip-10-0-140-107 kubenswrapper[4939]: E1103 01:36:10.159884 4939 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6712b32f_f3bd_4917_9e5e_101cf604625c.slice/crio-384febeac8280063b68f71ab034887b1d84a8e149d013ded2756f3066aeecdf7.scope\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6712b32f_f3bd_4917_9e5e_101cf604625c.slice/crio-conmon-384febeac8280063b68f71ab034887b1d84a8e149d013ded2756f3066aeecdf7.scope\": RecentStats: unable to find data in memory cache]" Nov 03 01:36:10.436001 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:36:10.435875 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:36:10.602987 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:36:10.602950 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:36:11.045108 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:36:11.045065 4939 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-5dab6a-57cf987799-9svqc" podUID="63a13b02-7c84-47a7-aa2a-73a9ed53481e" containerName="payload-pre-processing" containerID="cri-o://5b803b19b851b18fe07ec3b154d6892be07f2408741c7cf0c43bc44ce3404e9e" gracePeriod=25 Nov 03 01:36:11.106491 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:36:11.106461 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:36:11.889199 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:36:11.889161 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:36:12.333413 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:36:12.333363 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:36:12.339261 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:36:12.339235 4939 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-5dab6a-57cf987799-9svqc" podUID="63a13b02-7c84-47a7-aa2a-73a9ed53481e" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 03 01:36:12.488842 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:36:12.488802 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:36:13.276707 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:36:13.276676 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:36:13.896300 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:36:13.896262 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:36:14.516758 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:36:14.516723 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:36:14.798471 ip-10-0-140-107 kubenswrapper[4939]: E1103 01:36:14.798377 4939 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6712b32f_f3bd_4917_9e5e_101cf604625c.slice/crio-conmon-384febeac8280063b68f71ab034887b1d84a8e149d013ded2756f3066aeecdf7.scope\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6712b32f_f3bd_4917_9e5e_101cf604625c.slice/crio-384febeac8280063b68f71ab034887b1d84a8e149d013ded2756f3066aeecdf7.scope\": RecentStats: unable to find data in memory cache]" Nov 03 01:36:15.004656 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:36:15.004620 4939 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-ddd653-84dfd5bd67-wv882"] Nov 03 01:36:15.017405 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:36:15.017375 4939 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-ddd653-5764b8bfbc-fcxrz"] Nov 03 01:36:15.109483 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:36:15.109446 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:36:15.230143 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:36:15.230105 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:36:15.646642 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:36:15.646604 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:36:16.279863 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:36:16.279829 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:36:16.993887 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:36:16.993849 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:36:17.093732 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:36:17.093699 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:36:17.683336 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:36:17.683303 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:36:18.034775 ip-10-0-140-107 kubenswrapper[4939]: E1103 01:36:18.034690 4939 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-sthjh" containerName="extractor" Nov 03 01:36:18.389830 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:36:18.389779 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:36:18.994679 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:36:18.994643 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:36:19.574722 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:36:19.574689 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:36:20.005168 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:36:20.005123 4939 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-extmod-a-ddd653-84dfd5bd67-wv882" podUID="60e096be-eed5-4cdd-b78b-f7e691f2d880" containerName="payload-processing" containerID="cri-o://01e21828ffe67b88db5dfd9651fa1e679804a8f3661a422523931ce366ab6e25" gracePeriod=25 Nov 03 01:36:20.017808 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:36:20.017759 4939 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-ddd653-5764b8bfbc-fcxrz" podUID="c023b319-6159-4d96-bacb-59ec3bc0d55c" containerName="payload-pre-processing" containerID="cri-o://3f122c183feec6b2fd59afce69cdb5d76b7035ed9e1b4e1d8a492b0d6391fff8" gracePeriod=25 Nov 03 01:36:20.312995 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:36:20.312966 4939 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-a-ddd653-84dfd5bd67-wv882" Nov 03 01:36:20.375936 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:36:20.368867 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:36:20.394489 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:36:20.394454 4939 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-ddd653-5764b8bfbc-fcxrz" Nov 03 01:36:20.418748 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:36:20.418711 4939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/60e096be-eed5-4cdd-b78b-f7e691f2d880-kube-api-access-4jjlh\" (UniqueName: \"kubernetes.io/projected/60e096be-eed5-4cdd-b78b-f7e691f2d880-kube-api-access-4jjlh\") pod \"60e096be-eed5-4cdd-b78b-f7e691f2d880\" (UID: \"60e096be-eed5-4cdd-b78b-f7e691f2d880\") " Nov 03 01:36:20.418996 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:36:20.418965 4939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/60e096be-eed5-4cdd-b78b-f7e691f2d880-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/60e096be-eed5-4cdd-b78b-f7e691f2d880-plugins-config-volume\") pod \"60e096be-eed5-4cdd-b78b-f7e691f2d880\" (UID: \"60e096be-eed5-4cdd-b78b-f7e691f2d880\") " Nov 03 01:36:20.419362 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:36:20.419336 4939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/60e096be-eed5-4cdd-b78b-f7e691f2d880-plugins-config-volume" pod "60e096be-eed5-4cdd-b78b-f7e691f2d880" (UID: "60e096be-eed5-4cdd-b78b-f7e691f2d880"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 01:36:20.419855 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:36:20.419831 4939 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/60e096be-eed5-4cdd-b78b-f7e691f2d880-plugins-config-volume\") on node \"ip-10-0-140-107.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:36:20.421780 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:36:20.421752 4939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/60e096be-eed5-4cdd-b78b-f7e691f2d880-kube-api-access-4jjlh" pod "60e096be-eed5-4cdd-b78b-f7e691f2d880" (UID: "60e096be-eed5-4cdd-b78b-f7e691f2d880"). InnerVolumeSpecName "kube-api-access-4jjlh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 01:36:20.520813 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:36:20.520740 4939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c023b319-6159-4d96-bacb-59ec3bc0d55c-kube-api-access-s7sb7\" (UniqueName: \"kubernetes.io/projected/c023b319-6159-4d96-bacb-59ec3bc0d55c-kube-api-access-s7sb7\") pod \"c023b319-6159-4d96-bacb-59ec3bc0d55c\" (UID: \"c023b319-6159-4d96-bacb-59ec3bc0d55c\") " Nov 03 01:36:20.521017 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:36:20.520853 4939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c023b319-6159-4d96-bacb-59ec3bc0d55c-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c023b319-6159-4d96-bacb-59ec3bc0d55c-plugins-config-volume\") pod \"c023b319-6159-4d96-bacb-59ec3bc0d55c\" (UID: \"c023b319-6159-4d96-bacb-59ec3bc0d55c\") " Nov 03 01:36:20.521259 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:36:20.521231 4939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c023b319-6159-4d96-bacb-59ec3bc0d55c-plugins-config-volume" pod "c023b319-6159-4d96-bacb-59ec3bc0d55c" (UID: "c023b319-6159-4d96-bacb-59ec3bc0d55c"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 01:36:20.521435 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:36:20.521414 4939 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4jjlh\" (UniqueName: \"kubernetes.io/projected/60e096be-eed5-4cdd-b78b-f7e691f2d880-kube-api-access-4jjlh\") on node \"ip-10-0-140-107.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:36:20.521495 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:36:20.521441 4939 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c023b319-6159-4d96-bacb-59ec3bc0d55c-plugins-config-volume\") on node \"ip-10-0-140-107.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:36:20.522975 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:36:20.522953 4939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c023b319-6159-4d96-bacb-59ec3bc0d55c-kube-api-access-s7sb7" pod "c023b319-6159-4d96-bacb-59ec3bc0d55c" (UID: "c023b319-6159-4d96-bacb-59ec3bc0d55c"). InnerVolumeSpecName "kube-api-access-s7sb7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 01:36:20.623393 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:36:20.623355 4939 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-s7sb7\" (UniqueName: \"kubernetes.io/projected/c023b319-6159-4d96-bacb-59ec3bc0d55c-kube-api-access-s7sb7\") on node \"ip-10-0-140-107.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:36:20.807964 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:36:20.807840 4939 generic.go:356] "Generic (PLEG): container finished" podID="c023b319-6159-4d96-bacb-59ec3bc0d55c" containerID="3f122c183feec6b2fd59afce69cdb5d76b7035ed9e1b4e1d8a492b0d6391fff8" exitCode=0 Nov 03 01:36:20.807964 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:36:20.807884 4939 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-ddd653-5764b8bfbc-fcxrz" Nov 03 01:36:20.807964 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:36:20.807928 4939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-ddd653-5764b8bfbc-fcxrz" event={"ID":"c023b319-6159-4d96-bacb-59ec3bc0d55c","Type":"ContainerDied","Data":"3f122c183feec6b2fd59afce69cdb5d76b7035ed9e1b4e1d8a492b0d6391fff8"} Nov 03 01:36:20.808250 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:36:20.807986 4939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-ddd653-5764b8bfbc-fcxrz" event={"ID":"c023b319-6159-4d96-bacb-59ec3bc0d55c","Type":"ContainerDied","Data":"46b19f4d45f363a7e582f7d9f5daabe0f40cf85b369987dc194aa789df37ea4b"} Nov 03 01:36:20.808250 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:36:20.808016 4939 scope.go:122] "RemoveContainer" containerID="3f122c183feec6b2fd59afce69cdb5d76b7035ed9e1b4e1d8a492b0d6391fff8" Nov 03 01:36:20.809873 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:36:20.809848 4939 generic.go:356] "Generic (PLEG): container finished" podID="60e096be-eed5-4cdd-b78b-f7e691f2d880" containerID="01e21828ffe67b88db5dfd9651fa1e679804a8f3661a422523931ce366ab6e25" exitCode=0 Nov 03 01:36:20.810022 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:36:20.809900 4939 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-a-ddd653-84dfd5bd67-wv882" Nov 03 01:36:20.810022 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:36:20.809933 4939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-a-ddd653-84dfd5bd67-wv882" event={"ID":"60e096be-eed5-4cdd-b78b-f7e691f2d880","Type":"ContainerDied","Data":"01e21828ffe67b88db5dfd9651fa1e679804a8f3661a422523931ce366ab6e25"} Nov 03 01:36:20.810136 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:36:20.810022 4939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-a-ddd653-84dfd5bd67-wv882" event={"ID":"60e096be-eed5-4cdd-b78b-f7e691f2d880","Type":"ContainerDied","Data":"5138b2d4446603efdae58a03377b1649ddaebbcae2eca8e8a567ac23dd8fb38d"} Nov 03 01:36:20.821635 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:36:20.821618 4939 scope.go:122] "RemoveContainer" containerID="3f122c183feec6b2fd59afce69cdb5d76b7035ed9e1b4e1d8a492b0d6391fff8" Nov 03 01:36:20.821884 ip-10-0-140-107 kubenswrapper[4939]: E1103 01:36:20.821868 4939 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3f122c183feec6b2fd59afce69cdb5d76b7035ed9e1b4e1d8a492b0d6391fff8\": container with ID starting with 3f122c183feec6b2fd59afce69cdb5d76b7035ed9e1b4e1d8a492b0d6391fff8 not found: ID does not exist" containerID="3f122c183feec6b2fd59afce69cdb5d76b7035ed9e1b4e1d8a492b0d6391fff8" Nov 03 01:36:20.821946 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:36:20.821891 4939 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3f122c183feec6b2fd59afce69cdb5d76b7035ed9e1b4e1d8a492b0d6391fff8"} err="failed to get container status \"3f122c183feec6b2fd59afce69cdb5d76b7035ed9e1b4e1d8a492b0d6391fff8\": rpc error: code = NotFound desc = could not find container \"3f122c183feec6b2fd59afce69cdb5d76b7035ed9e1b4e1d8a492b0d6391fff8\": container with ID starting with 3f122c183feec6b2fd59afce69cdb5d76b7035ed9e1b4e1d8a492b0d6391fff8 not found: ID does not exist" Nov 03 01:36:20.821946 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:36:20.821920 4939 scope.go:122] "RemoveContainer" containerID="01e21828ffe67b88db5dfd9651fa1e679804a8f3661a422523931ce366ab6e25" Nov 03 01:36:20.832562 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:36:20.832539 4939 scope.go:122] "RemoveContainer" containerID="01e21828ffe67b88db5dfd9651fa1e679804a8f3661a422523931ce366ab6e25" Nov 03 01:36:20.832842 ip-10-0-140-107 kubenswrapper[4939]: E1103 01:36:20.832821 4939 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"01e21828ffe67b88db5dfd9651fa1e679804a8f3661a422523931ce366ab6e25\": container with ID starting with 01e21828ffe67b88db5dfd9651fa1e679804a8f3661a422523931ce366ab6e25 not found: ID does not exist" containerID="01e21828ffe67b88db5dfd9651fa1e679804a8f3661a422523931ce366ab6e25" Nov 03 01:36:20.832894 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:36:20.832856 4939 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"01e21828ffe67b88db5dfd9651fa1e679804a8f3661a422523931ce366ab6e25"} err="failed to get container status \"01e21828ffe67b88db5dfd9651fa1e679804a8f3661a422523931ce366ab6e25\": rpc error: code = NotFound desc = could not find container \"01e21828ffe67b88db5dfd9651fa1e679804a8f3661a422523931ce366ab6e25\": container with ID starting with 01e21828ffe67b88db5dfd9651fa1e679804a8f3661a422523931ce366ab6e25 not found: ID does not exist" Nov 03 01:36:20.834798 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:36:20.834779 4939 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-ddd653-5764b8bfbc-fcxrz"] Nov 03 01:36:20.840171 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:36:20.840149 4939 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-ddd653-5764b8bfbc-fcxrz"] Nov 03 01:36:20.850502 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:36:20.850480 4939 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-ddd653-84dfd5bd67-wv882"] Nov 03 01:36:20.854649 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:36:20.854624 4939 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-ddd653-84dfd5bd67-wv882"] Nov 03 01:36:20.969877 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:36:20.969841 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:36:21.581839 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:36:21.581810 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:36:21.869825 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:36:21.869747 4939 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="60e096be-eed5-4cdd-b78b-f7e691f2d880" path="/var/lib/kubelet/pods/60e096be-eed5-4cdd-b78b-f7e691f2d880/volumes" Nov 03 01:36:21.870250 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:36:21.870233 4939 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c023b319-6159-4d96-bacb-59ec3bc0d55c" path="/var/lib/kubelet/pods/c023b319-6159-4d96-bacb-59ec3bc0d55c/volumes" Nov 03 01:36:22.128926 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:36:22.128814 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:36:22.298663 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:36:22.298628 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:36:22.360055 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:36:22.360011 4939 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-5dab6a-57cf987799-9svqc" podUID="63a13b02-7c84-47a7-aa2a-73a9ed53481e" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 03 01:36:22.942710 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:36:22.942674 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:36:23.073220 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:36:23.073171 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:36:23.682566 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:36:23.682532 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:36:24.270745 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:36:24.270713 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:36:24.896781 ip-10-0-140-107 kubenswrapper[4939]: E1103 01:36:24.896742 4939 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6712b32f_f3bd_4917_9e5e_101cf604625c.slice/crio-conmon-384febeac8280063b68f71ab034887b1d84a8e149d013ded2756f3066aeecdf7.scope\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6712b32f_f3bd_4917_9e5e_101cf604625c.slice/crio-384febeac8280063b68f71ab034887b1d84a8e149d013ded2756f3066aeecdf7.scope\": RecentStats: unable to find data in memory cache]" Nov 03 01:36:25.006219 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:36:25.006182 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:36:25.159094 ip-10-0-140-107 kubenswrapper[4939]: E1103 01:36:25.159004 4939 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6712b32f_f3bd_4917_9e5e_101cf604625c.slice/crio-384febeac8280063b68f71ab034887b1d84a8e149d013ded2756f3066aeecdf7.scope\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6712b32f_f3bd_4917_9e5e_101cf604625c.slice/crio-conmon-384febeac8280063b68f71ab034887b1d84a8e149d013ded2756f3066aeecdf7.scope\": RecentStats: unable to find data in memory cache]" Nov 03 01:36:25.423029 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:36:25.422889 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:36:25.907954 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:36:25.907878 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:36:26.576894 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:36:26.576848 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:36:27.175764 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:36:27.175718 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:36:27.475191 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:36:27.475082 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:36:27.594311 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:36:27.594277 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:36:27.815295 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:36:27.815189 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:36:28.092943 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:36:28.092873 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:36:28.451100 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:36:28.451004 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:36:28.998122 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:36:28.998077 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:36:29.118917 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:36:29.118857 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:36:29.244228 ip-10-0-140-107 kubenswrapper[4939]: E1103 01:36:29.243713 4939 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6712b32f_f3bd_4917_9e5e_101cf604625c.slice/crio-384febeac8280063b68f71ab034887b1d84a8e149d013ded2756f3066aeecdf7.scope\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6712b32f_f3bd_4917_9e5e_101cf604625c.slice/crio-conmon-384febeac8280063b68f71ab034887b1d84a8e149d013ded2756f3066aeecdf7.scope\": RecentStats: unable to find data in memory cache]" Nov 03 01:36:29.248531 ip-10-0-140-107 kubenswrapper[4939]: E1103 01:36:29.248423 4939 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6712b32f_f3bd_4917_9e5e_101cf604625c.slice/crio-conmon-384febeac8280063b68f71ab034887b1d84a8e149d013ded2756f3066aeecdf7.scope\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6712b32f_f3bd_4917_9e5e_101cf604625c.slice/crio-384febeac8280063b68f71ab034887b1d84a8e149d013ded2756f3066aeecdf7.scope\": RecentStats: unable to find data in memory cache]" Nov 03 01:36:29.646008 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:36:29.645974 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:36:29.779972 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:36:29.779936 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:36:29.960511 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:36:29.960424 4939 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-a-ddd653-openshift-default-779ff6465-t9cvw"] Nov 03 01:36:29.961053 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:36:29.961021 4939 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-extmod-a-ddd653-openshift-default-779ff6465-t9cvw" podUID="aa64537e-0c5b-49ae-9547-df65d9ea4b2d" containerName="istio-proxy" containerID="cri-o://688beffd8ffb7850f456be4cccfdcedec5d5b8f6c96c9a094d56e7e2675fc8c8" gracePeriod=30 Nov 03 01:36:30.010902 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:36:30.010865 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:36:30.137295 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:36:30.137245 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:36:30.791958 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:36:30.791923 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:36:30.904731 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:36:30.904696 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:36:31.325883 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:36:31.325854 4939 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-5dab6a-57cf987799-9svqc" Nov 03 01:36:31.403521 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:36:31.403486 4939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/63a13b02-7c84-47a7-aa2a-73a9ed53481e-kube-api-access-2vg9b\" (UniqueName: \"kubernetes.io/projected/63a13b02-7c84-47a7-aa2a-73a9ed53481e-kube-api-access-2vg9b\") pod \"63a13b02-7c84-47a7-aa2a-73a9ed53481e\" (UID: \"63a13b02-7c84-47a7-aa2a-73a9ed53481e\") " Nov 03 01:36:31.403697 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:36:31.403598 4939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/63a13b02-7c84-47a7-aa2a-73a9ed53481e-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/63a13b02-7c84-47a7-aa2a-73a9ed53481e-plugins-config-volume\") pod \"63a13b02-7c84-47a7-aa2a-73a9ed53481e\" (UID: \"63a13b02-7c84-47a7-aa2a-73a9ed53481e\") " Nov 03 01:36:31.404150 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:36:31.404117 4939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/63a13b02-7c84-47a7-aa2a-73a9ed53481e-plugins-config-volume" pod "63a13b02-7c84-47a7-aa2a-73a9ed53481e" (UID: "63a13b02-7c84-47a7-aa2a-73a9ed53481e"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 01:36:31.404589 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:36:31.404561 4939 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/63a13b02-7c84-47a7-aa2a-73a9ed53481e-plugins-config-volume\") on node \"ip-10-0-140-107.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:36:31.405801 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:36:31.405778 4939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/63a13b02-7c84-47a7-aa2a-73a9ed53481e-kube-api-access-2vg9b" pod "63a13b02-7c84-47a7-aa2a-73a9ed53481e" (UID: "63a13b02-7c84-47a7-aa2a-73a9ed53481e"). InnerVolumeSpecName "kube-api-access-2vg9b". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 01:36:31.505814 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:36:31.505777 4939 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2vg9b\" (UniqueName: \"kubernetes.io/projected/63a13b02-7c84-47a7-aa2a-73a9ed53481e-kube-api-access-2vg9b\") on node \"ip-10-0-140-107.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:36:31.566685 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:36:31.566602 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:36:31.844321 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:36:31.844286 4939 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w2-531447-openshift-default-7995d46767-jcnlj"] Nov 03 01:36:31.845304 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:36:31.845283 4939 cpu_manager.go:414] "containerMap: removing container" podUID="9a691811-c7b4-4da0-892d-13d98fb3a0b0" containerName="main" Nov 03 01:36:31.845304 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:36:31.845304 4939 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9a691811-c7b4-4da0-892d-13d98fb3a0b0" containerName="main" Nov 03 01:36:31.845433 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:36:31.845319 4939 cpu_manager.go:414] "containerMap: removing container" podUID="c023b319-6159-4d96-bacb-59ec3bc0d55c" containerName="payload-pre-processing" Nov 03 01:36:31.845433 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:36:31.845324 4939 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c023b319-6159-4d96-bacb-59ec3bc0d55c" containerName="payload-pre-processing" Nov 03 01:36:31.845433 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:36:31.845340 4939 cpu_manager.go:414] "containerMap: removing container" podUID="ae67a584-3143-44d0-a76f-97f60d25dd60" containerName="istio-proxy" Nov 03 01:36:31.845433 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:36:31.845345 4939 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ae67a584-3143-44d0-a76f-97f60d25dd60" containerName="istio-proxy" Nov 03 01:36:31.845433 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:36:31.845358 4939 cpu_manager.go:414] "containerMap: removing container" podUID="60e096be-eed5-4cdd-b78b-f7e691f2d880" containerName="payload-processing" Nov 03 01:36:31.845433 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:36:31.845363 4939 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="60e096be-eed5-4cdd-b78b-f7e691f2d880" containerName="payload-processing" Nov 03 01:36:31.845433 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:36:31.845374 4939 cpu_manager.go:414] "containerMap: removing container" podUID="cc9b0d3f-c18a-4e8f-ab5b-1c9d9d1b244c" containerName="istio-proxy" Nov 03 01:36:31.845433 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:36:31.845379 4939 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cc9b0d3f-c18a-4e8f-ab5b-1c9d9d1b244c" containerName="istio-proxy" Nov 03 01:36:31.845433 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:36:31.845397 4939 cpu_manager.go:414] "containerMap: removing container" podUID="80a21f03-7cfa-49f0-b35b-df8edc58932e" containerName="istio-proxy" Nov 03 01:36:31.845433 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:36:31.845402 4939 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="80a21f03-7cfa-49f0-b35b-df8edc58932e" containerName="istio-proxy" Nov 03 01:36:31.845433 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:36:31.845420 4939 cpu_manager.go:414] "containerMap: removing container" podUID="7064128b-8764-4a6a-a4bc-e42b083f944f" containerName="istio-proxy" Nov 03 01:36:31.845433 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:36:31.845424 4939 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7064128b-8764-4a6a-a4bc-e42b083f944f" containerName="istio-proxy" Nov 03 01:36:31.845433 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:36:31.845434 4939 cpu_manager.go:414] "containerMap: removing container" podUID="7064128b-8764-4a6a-a4bc-e42b083f944f" containerName="istio-proxy" Nov 03 01:36:31.845433 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:36:31.845439 4939 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7064128b-8764-4a6a-a4bc-e42b083f944f" containerName="istio-proxy" Nov 03 01:36:31.845874 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:36:31.845446 4939 cpu_manager.go:414] "containerMap: removing container" podUID="6712b32f-f3bd-4917-9e5e-101cf604625c" containerName="payload-pre-processing" Nov 03 01:36:31.845874 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:36:31.845451 4939 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6712b32f-f3bd-4917-9e5e-101cf604625c" containerName="payload-pre-processing" Nov 03 01:36:31.845874 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:36:31.845467 4939 cpu_manager.go:414] "containerMap: removing container" podUID="63a13b02-7c84-47a7-aa2a-73a9ed53481e" containerName="payload-pre-processing" Nov 03 01:36:31.845874 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:36:31.845472 4939 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="63a13b02-7c84-47a7-aa2a-73a9ed53481e" containerName="payload-pre-processing" Nov 03 01:36:31.845874 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:36:31.845482 4939 cpu_manager.go:414] "containerMap: removing container" podUID="05a50b7a-195c-4ec9-9269-5d58faee868b" containerName="maas-api" Nov 03 01:36:31.845874 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:36:31.845486 4939 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="05a50b7a-195c-4ec9-9269-5d58faee868b" containerName="maas-api" Nov 03 01:36:31.845874 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:36:31.845589 4939 memory_manager.go:365] "RemoveStaleState removing state" podUID="63a13b02-7c84-47a7-aa2a-73a9ed53481e" containerName="payload-pre-processing" Nov 03 01:36:31.845874 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:36:31.845599 4939 memory_manager.go:365] "RemoveStaleState removing state" podUID="05a50b7a-195c-4ec9-9269-5d58faee868b" containerName="maas-api" Nov 03 01:36:31.845874 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:36:31.845607 4939 memory_manager.go:365] "RemoveStaleState removing state" podUID="9a691811-c7b4-4da0-892d-13d98fb3a0b0" containerName="main" Nov 03 01:36:31.845874 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:36:31.845616 4939 memory_manager.go:365] "RemoveStaleState removing state" podUID="c023b319-6159-4d96-bacb-59ec3bc0d55c" containerName="payload-pre-processing" Nov 03 01:36:31.845874 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:36:31.845624 4939 memory_manager.go:365] "RemoveStaleState removing state" podUID="ae67a584-3143-44d0-a76f-97f60d25dd60" containerName="istio-proxy" Nov 03 01:36:31.845874 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:36:31.845633 4939 memory_manager.go:365] "RemoveStaleState removing state" podUID="60e096be-eed5-4cdd-b78b-f7e691f2d880" containerName="payload-processing" Nov 03 01:36:31.845874 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:36:31.845639 4939 memory_manager.go:365] "RemoveStaleState removing state" podUID="cc9b0d3f-c18a-4e8f-ab5b-1c9d9d1b244c" containerName="istio-proxy" Nov 03 01:36:31.845874 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:36:31.845648 4939 memory_manager.go:365] "RemoveStaleState removing state" podUID="80a21f03-7cfa-49f0-b35b-df8edc58932e" containerName="istio-proxy" Nov 03 01:36:31.845874 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:36:31.845655 4939 memory_manager.go:365] "RemoveStaleState removing state" podUID="7064128b-8764-4a6a-a4bc-e42b083f944f" containerName="istio-proxy" Nov 03 01:36:31.845874 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:36:31.845661 4939 memory_manager.go:365] "RemoveStaleState removing state" podUID="7064128b-8764-4a6a-a4bc-e42b083f944f" containerName="istio-proxy" Nov 03 01:36:31.845874 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:36:31.845666 4939 memory_manager.go:365] "RemoveStaleState removing state" podUID="6712b32f-f3bd-4917-9e5e-101cf604625c" containerName="payload-pre-processing" Nov 03 01:36:31.849518 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:36:31.849496 4939 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w2-531447-openshift-default-7995d46767-jcnlj" Nov 03 01:36:31.869361 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:36:31.869333 4939 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-w2-531447-openshift-default-dockercfg-z7zxz\"" Nov 03 01:36:31.881787 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:36:31.881757 4939 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-531447-openshift-default-7995d46767-jcnlj"] Nov 03 01:36:31.914585 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:36:31.914553 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kp9kt\" (UniqueName: \"kubernetes.io/projected/aa4094a2-6349-4637-9630-40a4ee74ecce-kube-api-access-kp9kt\") pod \"e2e-shared-a-w2-531447-openshift-default-7995d46767-jcnlj\" (UID: \"aa4094a2-6349-4637-9630-40a4ee74ecce\") " pod="openshift-ingress/e2e-shared-a-w2-531447-openshift-default-7995d46767-jcnlj" Nov 03 01:36:31.914795 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:36:31.914610 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/aa4094a2-6349-4637-9630-40a4ee74ecce-istio-token\") pod \"e2e-shared-a-w2-531447-openshift-default-7995d46767-jcnlj\" (UID: \"aa4094a2-6349-4637-9630-40a4ee74ecce\") " pod="openshift-ingress/e2e-shared-a-w2-531447-openshift-default-7995d46767-jcnlj" Nov 03 01:36:31.914795 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:36:31.914734 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/aa4094a2-6349-4637-9630-40a4ee74ecce-istiod-ca-cert\") pod \"e2e-shared-a-w2-531447-openshift-default-7995d46767-jcnlj\" (UID: \"aa4094a2-6349-4637-9630-40a4ee74ecce\") " pod="openshift-ingress/e2e-shared-a-w2-531447-openshift-default-7995d46767-jcnlj" Nov 03 01:36:31.914795 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:36:31.914777 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/aa4094a2-6349-4637-9630-40a4ee74ecce-workload-socket\") pod \"e2e-shared-a-w2-531447-openshift-default-7995d46767-jcnlj\" (UID: \"aa4094a2-6349-4637-9630-40a4ee74ecce\") " pod="openshift-ingress/e2e-shared-a-w2-531447-openshift-default-7995d46767-jcnlj" Nov 03 01:36:31.915073 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:36:31.915050 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/aa4094a2-6349-4637-9630-40a4ee74ecce-credential-socket\") pod \"e2e-shared-a-w2-531447-openshift-default-7995d46767-jcnlj\" (UID: \"aa4094a2-6349-4637-9630-40a4ee74ecce\") " pod="openshift-ingress/e2e-shared-a-w2-531447-openshift-default-7995d46767-jcnlj" Nov 03 01:36:31.915347 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:36:31.915319 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/aa4094a2-6349-4637-9630-40a4ee74ecce-istio-data\") pod \"e2e-shared-a-w2-531447-openshift-default-7995d46767-jcnlj\" (UID: \"aa4094a2-6349-4637-9630-40a4ee74ecce\") " pod="openshift-ingress/e2e-shared-a-w2-531447-openshift-default-7995d46767-jcnlj" Nov 03 01:36:31.915446 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:36:31.915359 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/aa4094a2-6349-4637-9630-40a4ee74ecce-istio-podinfo\") pod \"e2e-shared-a-w2-531447-openshift-default-7995d46767-jcnlj\" (UID: \"aa4094a2-6349-4637-9630-40a4ee74ecce\") " pod="openshift-ingress/e2e-shared-a-w2-531447-openshift-default-7995d46767-jcnlj" Nov 03 01:36:31.915518 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:36:31.915467 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/aa4094a2-6349-4637-9630-40a4ee74ecce-workload-certs\") pod \"e2e-shared-a-w2-531447-openshift-default-7995d46767-jcnlj\" (UID: \"aa4094a2-6349-4637-9630-40a4ee74ecce\") " pod="openshift-ingress/e2e-shared-a-w2-531447-openshift-default-7995d46767-jcnlj" Nov 03 01:36:31.915580 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:36:31.915515 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/aa4094a2-6349-4637-9630-40a4ee74ecce-istio-envoy\") pod \"e2e-shared-a-w2-531447-openshift-default-7995d46767-jcnlj\" (UID: \"aa4094a2-6349-4637-9630-40a4ee74ecce\") " pod="openshift-ingress/e2e-shared-a-w2-531447-openshift-default-7995d46767-jcnlj" Nov 03 01:36:31.937500 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:36:31.937467 4939 generic.go:356] "Generic (PLEG): container finished" podID="63a13b02-7c84-47a7-aa2a-73a9ed53481e" containerID="5b803b19b851b18fe07ec3b154d6892be07f2408741c7cf0c43bc44ce3404e9e" exitCode=0 Nov 03 01:36:31.937641 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:36:31.937564 4939 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-5dab6a-57cf987799-9svqc" Nov 03 01:36:31.937688 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:36:31.937649 4939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-5dab6a-57cf987799-9svqc" event={"ID":"63a13b02-7c84-47a7-aa2a-73a9ed53481e","Type":"ContainerDied","Data":"5b803b19b851b18fe07ec3b154d6892be07f2408741c7cf0c43bc44ce3404e9e"} Nov 03 01:36:31.937733 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:36:31.937697 4939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-5dab6a-57cf987799-9svqc" event={"ID":"63a13b02-7c84-47a7-aa2a-73a9ed53481e","Type":"ContainerDied","Data":"9856e654c27c8b06c2641dfcf82f1b68fc8e02f3de2a480f5bbd72a212c72f6a"} Nov 03 01:36:31.937733 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:36:31.937725 4939 scope.go:122] "RemoveContainer" containerID="5b803b19b851b18fe07ec3b154d6892be07f2408741c7cf0c43bc44ce3404e9e" Nov 03 01:36:31.950816 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:36:31.950797 4939 scope.go:122] "RemoveContainer" containerID="5b803b19b851b18fe07ec3b154d6892be07f2408741c7cf0c43bc44ce3404e9e" Nov 03 01:36:31.951128 ip-10-0-140-107 kubenswrapper[4939]: E1103 01:36:31.951104 4939 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5b803b19b851b18fe07ec3b154d6892be07f2408741c7cf0c43bc44ce3404e9e\": container with ID starting with 5b803b19b851b18fe07ec3b154d6892be07f2408741c7cf0c43bc44ce3404e9e not found: ID does not exist" containerID="5b803b19b851b18fe07ec3b154d6892be07f2408741c7cf0c43bc44ce3404e9e" Nov 03 01:36:31.951222 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:36:31.951140 4939 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5b803b19b851b18fe07ec3b154d6892be07f2408741c7cf0c43bc44ce3404e9e"} err="failed to get container status \"5b803b19b851b18fe07ec3b154d6892be07f2408741c7cf0c43bc44ce3404e9e\": rpc error: code = NotFound desc = could not find container \"5b803b19b851b18fe07ec3b154d6892be07f2408741c7cf0c43bc44ce3404e9e\": container with ID starting with 5b803b19b851b18fe07ec3b154d6892be07f2408741c7cf0c43bc44ce3404e9e not found: ID does not exist" Nov 03 01:36:31.968118 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:36:31.968086 4939 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-5dab6a-57cf987799-9svqc"] Nov 03 01:36:31.995459 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:36:31.995430 4939 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-5dab6a-57cf987799-9svqc"] Nov 03 01:36:32.016513 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:36:32.016487 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/aa4094a2-6349-4637-9630-40a4ee74ecce-credential-socket\") pod \"e2e-shared-a-w2-531447-openshift-default-7995d46767-jcnlj\" (UID: \"aa4094a2-6349-4637-9630-40a4ee74ecce\") " pod="openshift-ingress/e2e-shared-a-w2-531447-openshift-default-7995d46767-jcnlj" Nov 03 01:36:32.016669 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:36:32.016607 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/aa4094a2-6349-4637-9630-40a4ee74ecce-istio-data\") pod \"e2e-shared-a-w2-531447-openshift-default-7995d46767-jcnlj\" (UID: \"aa4094a2-6349-4637-9630-40a4ee74ecce\") " pod="openshift-ingress/e2e-shared-a-w2-531447-openshift-default-7995d46767-jcnlj" Nov 03 01:36:32.016669 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:36:32.016640 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/aa4094a2-6349-4637-9630-40a4ee74ecce-istio-podinfo\") pod \"e2e-shared-a-w2-531447-openshift-default-7995d46767-jcnlj\" (UID: \"aa4094a2-6349-4637-9630-40a4ee74ecce\") " pod="openshift-ingress/e2e-shared-a-w2-531447-openshift-default-7995d46767-jcnlj" Nov 03 01:36:32.016801 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:36:32.016705 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/aa4094a2-6349-4637-9630-40a4ee74ecce-workload-certs\") pod \"e2e-shared-a-w2-531447-openshift-default-7995d46767-jcnlj\" (UID: \"aa4094a2-6349-4637-9630-40a4ee74ecce\") " pod="openshift-ingress/e2e-shared-a-w2-531447-openshift-default-7995d46767-jcnlj" Nov 03 01:36:32.016801 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:36:32.016745 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/aa4094a2-6349-4637-9630-40a4ee74ecce-istio-envoy\") pod \"e2e-shared-a-w2-531447-openshift-default-7995d46767-jcnlj\" (UID: \"aa4094a2-6349-4637-9630-40a4ee74ecce\") " pod="openshift-ingress/e2e-shared-a-w2-531447-openshift-default-7995d46767-jcnlj" Nov 03 01:36:32.016801 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:36:32.016778 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kp9kt\" (UniqueName: \"kubernetes.io/projected/aa4094a2-6349-4637-9630-40a4ee74ecce-kube-api-access-kp9kt\") pod \"e2e-shared-a-w2-531447-openshift-default-7995d46767-jcnlj\" (UID: \"aa4094a2-6349-4637-9630-40a4ee74ecce\") " pod="openshift-ingress/e2e-shared-a-w2-531447-openshift-default-7995d46767-jcnlj" Nov 03 01:36:32.016991 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:36:32.016820 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/aa4094a2-6349-4637-9630-40a4ee74ecce-istio-token\") pod \"e2e-shared-a-w2-531447-openshift-default-7995d46767-jcnlj\" (UID: \"aa4094a2-6349-4637-9630-40a4ee74ecce\") " pod="openshift-ingress/e2e-shared-a-w2-531447-openshift-default-7995d46767-jcnlj" Nov 03 01:36:32.016991 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:36:32.016877 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/aa4094a2-6349-4637-9630-40a4ee74ecce-credential-socket\") pod \"e2e-shared-a-w2-531447-openshift-default-7995d46767-jcnlj\" (UID: \"aa4094a2-6349-4637-9630-40a4ee74ecce\") " pod="openshift-ingress/e2e-shared-a-w2-531447-openshift-default-7995d46767-jcnlj" Nov 03 01:36:32.016991 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:36:32.016885 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/aa4094a2-6349-4637-9630-40a4ee74ecce-istiod-ca-cert\") pod \"e2e-shared-a-w2-531447-openshift-default-7995d46767-jcnlj\" (UID: \"aa4094a2-6349-4637-9630-40a4ee74ecce\") " pod="openshift-ingress/e2e-shared-a-w2-531447-openshift-default-7995d46767-jcnlj" Nov 03 01:36:32.016991 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:36:32.016957 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/aa4094a2-6349-4637-9630-40a4ee74ecce-workload-socket\") pod \"e2e-shared-a-w2-531447-openshift-default-7995d46767-jcnlj\" (UID: \"aa4094a2-6349-4637-9630-40a4ee74ecce\") " pod="openshift-ingress/e2e-shared-a-w2-531447-openshift-default-7995d46767-jcnlj" Nov 03 01:36:32.016991 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:36:32.016979 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/aa4094a2-6349-4637-9630-40a4ee74ecce-istio-data\") pod \"e2e-shared-a-w2-531447-openshift-default-7995d46767-jcnlj\" (UID: \"aa4094a2-6349-4637-9630-40a4ee74ecce\") " pod="openshift-ingress/e2e-shared-a-w2-531447-openshift-default-7995d46767-jcnlj" Nov 03 01:36:32.017292 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:36:32.017068 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/aa4094a2-6349-4637-9630-40a4ee74ecce-workload-certs\") pod \"e2e-shared-a-w2-531447-openshift-default-7995d46767-jcnlj\" (UID: \"aa4094a2-6349-4637-9630-40a4ee74ecce\") " pod="openshift-ingress/e2e-shared-a-w2-531447-openshift-default-7995d46767-jcnlj" Nov 03 01:36:32.017292 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:36:32.017239 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/aa4094a2-6349-4637-9630-40a4ee74ecce-workload-socket\") pod \"e2e-shared-a-w2-531447-openshift-default-7995d46767-jcnlj\" (UID: \"aa4094a2-6349-4637-9630-40a4ee74ecce\") " pod="openshift-ingress/e2e-shared-a-w2-531447-openshift-default-7995d46767-jcnlj" Nov 03 01:36:32.017599 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:36:32.017578 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/aa4094a2-6349-4637-9630-40a4ee74ecce-istiod-ca-cert\") pod \"e2e-shared-a-w2-531447-openshift-default-7995d46767-jcnlj\" (UID: \"aa4094a2-6349-4637-9630-40a4ee74ecce\") " pod="openshift-ingress/e2e-shared-a-w2-531447-openshift-default-7995d46767-jcnlj" Nov 03 01:36:32.019258 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:36:32.019236 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/aa4094a2-6349-4637-9630-40a4ee74ecce-istio-envoy\") pod \"e2e-shared-a-w2-531447-openshift-default-7995d46767-jcnlj\" (UID: \"aa4094a2-6349-4637-9630-40a4ee74ecce\") " pod="openshift-ingress/e2e-shared-a-w2-531447-openshift-default-7995d46767-jcnlj" Nov 03 01:36:32.019406 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:36:32.019382 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/aa4094a2-6349-4637-9630-40a4ee74ecce-istio-podinfo\") pod \"e2e-shared-a-w2-531447-openshift-default-7995d46767-jcnlj\" (UID: \"aa4094a2-6349-4637-9630-40a4ee74ecce\") " pod="openshift-ingress/e2e-shared-a-w2-531447-openshift-default-7995d46767-jcnlj" Nov 03 01:36:32.034306 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:36:32.034282 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/aa4094a2-6349-4637-9630-40a4ee74ecce-istio-token\") pod \"e2e-shared-a-w2-531447-openshift-default-7995d46767-jcnlj\" (UID: \"aa4094a2-6349-4637-9630-40a4ee74ecce\") " pod="openshift-ingress/e2e-shared-a-w2-531447-openshift-default-7995d46767-jcnlj" Nov 03 01:36:32.034401 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:36:32.034333 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kp9kt\" (UniqueName: \"kubernetes.io/projected/aa4094a2-6349-4637-9630-40a4ee74ecce-kube-api-access-kp9kt\") pod \"e2e-shared-a-w2-531447-openshift-default-7995d46767-jcnlj\" (UID: \"aa4094a2-6349-4637-9630-40a4ee74ecce\") " pod="openshift-ingress/e2e-shared-a-w2-531447-openshift-default-7995d46767-jcnlj" Nov 03 01:36:32.173647 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:36:32.173541 4939 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w2-531447-openshift-default-7995d46767-jcnlj" Nov 03 01:36:32.308119 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:36:32.308053 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:36:32.417824 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:36:32.417787 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:36:32.486680 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:36:32.486523 4939 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-531447-openshift-default-7995d46767-jcnlj"] Nov 03 01:36:32.487103 ip-10-0-140-107 kubenswrapper[4939]: W1103 01:36:32.486951 4939 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaa4094a2_6349_4637_9630_40a4ee74ecce.slice/crio-39dcda2883b0b8c244b2046274b295e5db4fb1bfd9d169e480653a5679dfeab2 WatchSource:0}: Error finding container 39dcda2883b0b8c244b2046274b295e5db4fb1bfd9d169e480653a5679dfeab2: Status 404 returned error can't find the container with id 39dcda2883b0b8c244b2046274b295e5db4fb1bfd9d169e480653a5679dfeab2 Nov 03 01:36:32.490170 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:36:32.490135 4939 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 01:36:32.490265 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:36:32.490210 4939 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 01:36:32.490265 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:36:32.490253 4939 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 01:36:32.948096 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:36:32.948060 4939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-531447-openshift-default-7995d46767-jcnlj" event={"ID":"aa4094a2-6349-4637-9630-40a4ee74ecce","Type":"ContainerStarted","Data":"2920ddcaec0d00e9f2de1c4222fe18cead32be1c73ee861d487bb3f6fc8daddb"} Nov 03 01:36:32.948290 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:36:32.948115 4939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-531447-openshift-default-7995d46767-jcnlj" event={"ID":"aa4094a2-6349-4637-9630-40a4ee74ecce","Type":"ContainerStarted","Data":"39dcda2883b0b8c244b2046274b295e5db4fb1bfd9d169e480653a5679dfeab2"} Nov 03 01:36:32.969376 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:36:32.969320 4939 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w2-531447-openshift-default-7995d46767-jcnlj" podStartSLOduration=1.969305575 podStartE2EDuration="1.969305575s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 01:36:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 01:36:32.967277756 +0000 UTC m=+3341.231612713" watchObservedRunningTime="2026-11-03 01:36:32.969305575 +0000 UTC m=+3341.233640520" Nov 03 01:36:33.174930 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:36:33.174088 4939 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w2-531447-openshift-default-7995d46767-jcnlj" Nov 03 01:36:33.177604 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:36:33.177568 4939 patch_prober.go:29] interesting pod/e2e-shared-a-w2-531447-openshift-default-7995d46767-jcnlj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.79:15021/healthz/ready\": dial tcp 10.132.1.79:15021: connect: connection refused" start-of-body= Nov 03 01:36:33.177760 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:36:33.177635 4939 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-531447-openshift-default-7995d46767-jcnlj" podUID="aa4094a2-6349-4637-9630-40a4ee74ecce" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.79:15021/healthz/ready\": dial tcp 10.132.1.79:15021: connect: connection refused" Nov 03 01:36:33.220806 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:36:33.220729 4939 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-4pw7v"] Nov 03 01:36:33.375424 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:36:33.375383 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:36:33.894840 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:36:33.894160 4939 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="63a13b02-7c84-47a7-aa2a-73a9ed53481e" path="/var/lib/kubelet/pods/63a13b02-7c84-47a7-aa2a-73a9ed53481e/volumes" Nov 03 01:36:34.176990 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:36:34.174071 4939 patch_prober.go:29] interesting pod/e2e-shared-a-w2-531447-openshift-default-7995d46767-jcnlj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.79:15021/healthz/ready\": dial tcp 10.132.1.79:15021: connect: connection refused" start-of-body= Nov 03 01:36:34.176990 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:36:34.174132 4939 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-531447-openshift-default-7995d46767-jcnlj" podUID="aa4094a2-6349-4637-9630-40a4ee74ecce" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.79:15021/healthz/ready\": dial tcp 10.132.1.79:15021: connect: connection refused" Nov 03 01:36:34.273354 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:36:34.273308 4939 patch_prober.go:29] interesting pod/e2e-extmod-a-ddd653-openshift-default-779ff6465-t9cvw container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 03 01:36:34.273521 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:36:34.273379 4939 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-extmod-a-ddd653-openshift-default-779ff6465-t9cvw" podUID="aa64537e-0c5b-49ae-9547-df65d9ea4b2d" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 03 01:36:34.338989 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:36:34.338949 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:36:35.179035 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:36:35.178547 4939 patch_prober.go:29] interesting pod/e2e-shared-a-w2-531447-openshift-default-7995d46767-jcnlj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.79:15021/healthz/ready\": dial tcp 10.132.1.79:15021: connect: connection refused" start-of-body= Nov 03 01:36:35.179035 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:36:35.178612 4939 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-531447-openshift-default-7995d46767-jcnlj" podUID="aa4094a2-6349-4637-9630-40a4ee74ecce" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.79:15021/healthz/ready\": dial tcp 10.132.1.79:15021: connect: connection refused" Nov 03 01:36:35.548714 ip-10-0-140-107 kubenswrapper[4939]: E1103 01:36:35.548651 4939 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6712b32f_f3bd_4917_9e5e_101cf604625c.slice/crio-384febeac8280063b68f71ab034887b1d84a8e149d013ded2756f3066aeecdf7.scope\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaa64537e_0c5b_49ae_9547_df65d9ea4b2d.slice/crio-688beffd8ffb7850f456be4cccfdcedec5d5b8f6c96c9a094d56e7e2675fc8c8.scope\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaa64537e_0c5b_49ae_9547_df65d9ea4b2d.slice/crio-conmon-688beffd8ffb7850f456be4cccfdcedec5d5b8f6c96c9a094d56e7e2675fc8c8.scope\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6712b32f_f3bd_4917_9e5e_101cf604625c.slice/crio-conmon-384febeac8280063b68f71ab034887b1d84a8e149d013ded2756f3066aeecdf7.scope\": RecentStats: unable to find data in memory cache]" Nov 03 01:36:35.814023 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:36:35.813974 4939 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-f7680d/distinct-w2-f7680d-kserve-756f874b98-qnpv8" podUID="836753fc-f9ef-4f61-b3fe-e5b92b7ee27b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.70:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 01:36:35.943933 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:36:35.943862 4939 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-88bbf7/distinct-2-w1-88bbf7-kserve-5457cdcdbd-l6tfd" podUID="a397931b-faa9-4c2e-8290-491350546663" containerName="main" probeResult="failure" output="Get \"https://10.132.0.85:8000/ready\": context deadline exceeded" Nov 03 01:36:36.034928 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:36:36.021663 4939 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-2f9a3a/simulator-w0-2f9a3a-kserve-9596bbb8f-2trgz" podUID="207e2207-fb6c-4076-95df-72e95671c261" containerName="main" probeResult="failure" output="Get \"https://10.132.0.61:8000/health\": context deadline exceeded" Nov 03 01:36:36.301259 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:36:36.301213 4939 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-83755a/distinct-2-w3-83755a-kserve-6cf66b995d-2l47b" podUID="c3b05e88-5d5b-4d03-b28d-832f22e3c868" containerName="main" probeResult="failure" output="Get \"https://10.132.0.72:8000/ready\": context deadline exceeded" Nov 03 01:36:36.320944 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:36:36.302412 4939 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-83755a/distinct-2-w3-83755a-kserve-6cf66b995d-2l47b" podUID="c3b05e88-5d5b-4d03-b28d-832f22e3c868" containerName="main" probeResult="failure" output="Get \"https://10.132.0.72:8000/health\": context deadline exceeded" Nov 03 01:36:36.329600 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:36:36.329568 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:36:36.500234 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:36:36.500188 4939 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-2f9a3a/simulator-w0-2f9a3a-kserve-9596bbb8f-2trgz" podUID="207e2207-fb6c-4076-95df-72e95671c261" containerName="main" probeResult="failure" output="Get \"https://10.132.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 01:36:36.550277 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:36:36.550234 4939 patch_prober.go:29] interesting pod/e2e-shared-a-w2-531447-openshift-default-7995d46767-jcnlj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.79:15021/healthz/ready\": dial tcp 10.132.1.79:15021: connect: connection refused" start-of-body= Nov 03 01:36:36.550589 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:36:36.550556 4939 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-531447-openshift-default-7995d46767-jcnlj" podUID="aa4094a2-6349-4637-9630-40a4ee74ecce" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.79:15021/healthz/ready\": dial tcp 10.132.1.79:15021: connect: connection refused" Nov 03 01:36:36.557289 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:36:36.557078 4939 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-shared-a-w2-531447/sub-test-model-531447-kserve-65cffbb7c-h6b59" podUID="de222447-f690-4b85-bdee-b419f81767eb" containerName="main" probeResult="failure" output="Get \"https://10.132.1.77:8000/ready\": context deadline exceeded" Nov 03 01:36:36.636059 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:36:36.636006 4939 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-88bbf7/simulator-w1-88bbf7-kserve-5fbb669f8f-42hgd" podUID="83b38883-2e7a-444d-aa68-52be4a25ae66" containerName="main" probeResult="failure" output="Get \"https://10.132.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 01:36:36.641046 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:36:36.641006 4939 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-shared-b-w2-ef434a/sub-test-model-ef434a-kserve-659ddb8595-zr7c2" podUID="5f7f5188-d311-486b-bb78-583a436145c5" containerName="main" probeResult="failure" output="Get \"https://10.132.1.78:8000/ready\": context deadline exceeded" Nov 03 01:36:36.713064 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:36:36.713000 4939 prober.go:122] "Probe failed" probeType="Liveness" pod="ai-tenant-e2e-shared-a-w2-531447/sub-test-model-531447-kserve-65cffbb7c-h6b59" podUID="de222447-f690-4b85-bdee-b419f81767eb" containerName="main" probeResult="failure" output="Get \"https://10.132.1.77:8000/health\": context deadline exceeded" Nov 03 01:36:36.767071 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:36:36.767022 4939 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-f7680d/premium-w2-f7680d-kserve-76565b8cf9-hqbbt" podUID="4712e708-18a2-4a13-a12c-a5669397b2c3" containerName="main" probeResult="failure" output="Get \"https://10.132.0.63:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 01:36:36.855990 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:36:36.835330 4939 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-ddd653-openshift-default-779ff6465-t9cvw" Nov 03 01:36:36.862159 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:36:36.862101 4939 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-f7680d/distinct-w2-f7680d-kserve-756f874b98-qnpv8" podUID="836753fc-f9ef-4f61-b3fe-e5b92b7ee27b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.70:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Nov 03 01:36:36.863339 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:36:36.863305 4939 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-f7680d/distinct-w2-f7680d-kserve-756f874b98-qnpv8" Nov 03 01:36:36.872032 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:36:36.871999 4939 generic.go:356] "Generic (PLEG): container finished" podID="aa64537e-0c5b-49ae-9547-df65d9ea4b2d" containerID="688beffd8ffb7850f456be4cccfdcedec5d5b8f6c96c9a094d56e7e2675fc8c8" exitCode=0 Nov 03 01:36:36.888458 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:36:36.888422 4939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-ddd653-openshift-default-779ff6465-t9cvw" event={"ID":"aa64537e-0c5b-49ae-9547-df65d9ea4b2d","Type":"ContainerDied","Data":"688beffd8ffb7850f456be4cccfdcedec5d5b8f6c96c9a094d56e7e2675fc8c8"} Nov 03 01:36:36.888717 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:36:36.888691 4939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-ddd653-openshift-default-779ff6465-t9cvw" event={"ID":"aa64537e-0c5b-49ae-9547-df65d9ea4b2d","Type":"ContainerDied","Data":"59820df908f2993f112845e1c10602c53c526132782951589076e4a4c1873246"} Nov 03 01:36:36.888862 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:36:36.888846 4939 scope.go:122] "RemoveContainer" containerID="688beffd8ffb7850f456be4cccfdcedec5d5b8f6c96c9a094d56e7e2675fc8c8" Nov 03 01:36:36.946684 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:36:36.946638 4939 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-88bbf7/distinct-2-w1-88bbf7-kserve-5457cdcdbd-l6tfd" podUID="a397931b-faa9-4c2e-8290-491350546663" containerName="main" probeResult="failure" output="Get \"https://10.132.0.85:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 01:36:37.063402 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:36:37.063373 4939 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-88bbf7/distinct-2-w1-88bbf7-kserve-5457cdcdbd-l6tfd" Nov 03 01:36:37.064225 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:36:37.013219 4939 scope.go:122] "RemoveContainer" containerID="eb4593fecdfedbac58d15758fcc9bfdb55ac993ebc532214ba94ace99e5439a4" Nov 03 01:36:37.064646 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:36:37.060028 4939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/aa64537e-0c5b-49ae-9547-df65d9ea4b2d-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/aa64537e-0c5b-49ae-9547-df65d9ea4b2d-istiod-ca-cert\") pod \"aa64537e-0c5b-49ae-9547-df65d9ea4b2d\" (UID: \"aa64537e-0c5b-49ae-9547-df65d9ea4b2d\") " Nov 03 01:36:37.064768 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:36:37.064682 4939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/aa64537e-0c5b-49ae-9547-df65d9ea4b2d-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/aa64537e-0c5b-49ae-9547-df65d9ea4b2d-workload-socket\") pod \"aa64537e-0c5b-49ae-9547-df65d9ea4b2d\" (UID: \"aa64537e-0c5b-49ae-9547-df65d9ea4b2d\") " Nov 03 01:36:37.064887 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:36:37.064871 4939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/aa64537e-0c5b-49ae-9547-df65d9ea4b2d-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/aa64537e-0c5b-49ae-9547-df65d9ea4b2d-workload-certs\") pod \"aa64537e-0c5b-49ae-9547-df65d9ea4b2d\" (UID: \"aa64537e-0c5b-49ae-9547-df65d9ea4b2d\") " Nov 03 01:36:37.065041 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:36:37.064937 4939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/aa64537e-0c5b-49ae-9547-df65d9ea4b2d-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/aa64537e-0c5b-49ae-9547-df65d9ea4b2d-istio-podinfo\") pod \"aa64537e-0c5b-49ae-9547-df65d9ea4b2d\" (UID: \"aa64537e-0c5b-49ae-9547-df65d9ea4b2d\") " Nov 03 01:36:37.065041 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:36:37.064970 4939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/aa64537e-0c5b-49ae-9547-df65d9ea4b2d-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/aa64537e-0c5b-49ae-9547-df65d9ea4b2d-credential-socket\") pod \"aa64537e-0c5b-49ae-9547-df65d9ea4b2d\" (UID: \"aa64537e-0c5b-49ae-9547-df65d9ea4b2d\") " Nov 03 01:36:37.065041 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:36:37.065021 4939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/aa64537e-0c5b-49ae-9547-df65d9ea4b2d-istio-token\" (UniqueName: \"kubernetes.io/projected/aa64537e-0c5b-49ae-9547-df65d9ea4b2d-istio-token\") pod \"aa64537e-0c5b-49ae-9547-df65d9ea4b2d\" (UID: \"aa64537e-0c5b-49ae-9547-df65d9ea4b2d\") " Nov 03 01:36:37.065211 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:36:37.065075 4939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/aa64537e-0c5b-49ae-9547-df65d9ea4b2d-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/aa64537e-0c5b-49ae-9547-df65d9ea4b2d-istio-data\") pod \"aa64537e-0c5b-49ae-9547-df65d9ea4b2d\" (UID: \"aa64537e-0c5b-49ae-9547-df65d9ea4b2d\") " Nov 03 01:36:37.065211 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:36:37.065102 4939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/aa64537e-0c5b-49ae-9547-df65d9ea4b2d-kube-api-access-jbgtv\" (UniqueName: \"kubernetes.io/projected/aa64537e-0c5b-49ae-9547-df65d9ea4b2d-kube-api-access-jbgtv\") pod \"aa64537e-0c5b-49ae-9547-df65d9ea4b2d\" (UID: \"aa64537e-0c5b-49ae-9547-df65d9ea4b2d\") " Nov 03 01:36:37.065318 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:36:37.065287 4939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/aa64537e-0c5b-49ae-9547-df65d9ea4b2d-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/aa64537e-0c5b-49ae-9547-df65d9ea4b2d-istio-envoy\") pod \"aa64537e-0c5b-49ae-9547-df65d9ea4b2d\" (UID: \"aa64537e-0c5b-49ae-9547-df65d9ea4b2d\") " Nov 03 01:36:37.066739 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:36:37.060323 4939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/aa64537e-0c5b-49ae-9547-df65d9ea4b2d-istiod-ca-cert" pod "aa64537e-0c5b-49ae-9547-df65d9ea4b2d" (UID: "aa64537e-0c5b-49ae-9547-df65d9ea4b2d"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 01:36:37.066739 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:36:37.066341 4939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/aa64537e-0c5b-49ae-9547-df65d9ea4b2d-credential-socket" pod "aa64537e-0c5b-49ae-9547-df65d9ea4b2d" (UID: "aa64537e-0c5b-49ae-9547-df65d9ea4b2d"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 01:36:37.066946 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:36:37.066737 4939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/aa64537e-0c5b-49ae-9547-df65d9ea4b2d-workload-certs" pod "aa64537e-0c5b-49ae-9547-df65d9ea4b2d" (UID: "aa64537e-0c5b-49ae-9547-df65d9ea4b2d"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 01:36:37.067162 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:36:37.067140 4939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/aa64537e-0c5b-49ae-9547-df65d9ea4b2d-workload-socket" pod "aa64537e-0c5b-49ae-9547-df65d9ea4b2d" (UID: "aa64537e-0c5b-49ae-9547-df65d9ea4b2d"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 01:36:37.079253 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:36:37.076187 4939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/aa64537e-0c5b-49ae-9547-df65d9ea4b2d-istio-data" pod "aa64537e-0c5b-49ae-9547-df65d9ea4b2d" (UID: "aa64537e-0c5b-49ae-9547-df65d9ea4b2d"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 01:36:37.079253 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:36:37.076354 4939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/aa64537e-0c5b-49ae-9547-df65d9ea4b2d-kube-api-access-jbgtv" pod "aa64537e-0c5b-49ae-9547-df65d9ea4b2d" (UID: "aa64537e-0c5b-49ae-9547-df65d9ea4b2d"). InnerVolumeSpecName "kube-api-access-jbgtv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 01:36:37.091941 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:36:37.082064 4939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/aa64537e-0c5b-49ae-9547-df65d9ea4b2d-istio-podinfo" pod "aa64537e-0c5b-49ae-9547-df65d9ea4b2d" (UID: "aa64537e-0c5b-49ae-9547-df65d9ea4b2d"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 01:36:37.091941 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:36:37.082091 4939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/aa64537e-0c5b-49ae-9547-df65d9ea4b2d-istio-envoy" pod "aa64537e-0c5b-49ae-9547-df65d9ea4b2d" (UID: "aa64537e-0c5b-49ae-9547-df65d9ea4b2d"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 01:36:37.091941 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:36:37.082192 4939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/aa64537e-0c5b-49ae-9547-df65d9ea4b2d-istio-token" pod "aa64537e-0c5b-49ae-9547-df65d9ea4b2d" (UID: "aa64537e-0c5b-49ae-9547-df65d9ea4b2d"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 01:36:37.111210 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:36:37.110728 4939 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-88bbf7/distinct-2-w1-88bbf7-kserve-5457cdcdbd-l6tfd" Nov 03 01:36:37.129114 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:36:37.123759 4939 scope.go:122] "RemoveContainer" containerID="688beffd8ffb7850f456be4cccfdcedec5d5b8f6c96c9a094d56e7e2675fc8c8" Nov 03 01:36:37.138210 ip-10-0-140-107 kubenswrapper[4939]: E1103 01:36:37.134296 4939 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"688beffd8ffb7850f456be4cccfdcedec5d5b8f6c96c9a094d56e7e2675fc8c8\": container with ID starting with 688beffd8ffb7850f456be4cccfdcedec5d5b8f6c96c9a094d56e7e2675fc8c8 not found: ID does not exist" containerID="688beffd8ffb7850f456be4cccfdcedec5d5b8f6c96c9a094d56e7e2675fc8c8" Nov 03 01:36:37.138210 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:36:37.134342 4939 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"688beffd8ffb7850f456be4cccfdcedec5d5b8f6c96c9a094d56e7e2675fc8c8"} err="failed to get container status \"688beffd8ffb7850f456be4cccfdcedec5d5b8f6c96c9a094d56e7e2675fc8c8\": rpc error: code = NotFound desc = could not find container \"688beffd8ffb7850f456be4cccfdcedec5d5b8f6c96c9a094d56e7e2675fc8c8\": container with ID starting with 688beffd8ffb7850f456be4cccfdcedec5d5b8f6c96c9a094d56e7e2675fc8c8 not found: ID does not exist" Nov 03 01:36:37.138210 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:36:37.134368 4939 scope.go:122] "RemoveContainer" containerID="eb4593fecdfedbac58d15758fcc9bfdb55ac993ebc532214ba94ace99e5439a4" Nov 03 01:36:37.138210 ip-10-0-140-107 kubenswrapper[4939]: E1103 01:36:37.134780 4939 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"eb4593fecdfedbac58d15758fcc9bfdb55ac993ebc532214ba94ace99e5439a4\": container with ID starting with eb4593fecdfedbac58d15758fcc9bfdb55ac993ebc532214ba94ace99e5439a4 not found: ID does not exist" containerID="eb4593fecdfedbac58d15758fcc9bfdb55ac993ebc532214ba94ace99e5439a4" Nov 03 01:36:37.138210 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:36:37.134818 4939 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"eb4593fecdfedbac58d15758fcc9bfdb55ac993ebc532214ba94ace99e5439a4"} err="failed to get container status \"eb4593fecdfedbac58d15758fcc9bfdb55ac993ebc532214ba94ace99e5439a4\": rpc error: code = NotFound desc = could not find container \"eb4593fecdfedbac58d15758fcc9bfdb55ac993ebc532214ba94ace99e5439a4\": container with ID starting with eb4593fecdfedbac58d15758fcc9bfdb55ac993ebc532214ba94ace99e5439a4 not found: ID does not exist" Nov 03 01:36:37.176929 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:36:37.174889 4939 patch_prober.go:29] interesting pod/e2e-shared-a-w2-531447-openshift-default-7995d46767-jcnlj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.79:15021/healthz/ready\": dial tcp 10.132.1.79:15021: connect: connection refused" start-of-body= Nov 03 01:36:37.176929 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:36:37.174970 4939 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-531447-openshift-default-7995d46767-jcnlj" podUID="aa4094a2-6349-4637-9630-40a4ee74ecce" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.79:15021/healthz/ready\": dial tcp 10.132.1.79:15021: connect: connection refused" Nov 03 01:36:37.202231 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:36:37.196404 4939 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/aa64537e-0c5b-49ae-9547-df65d9ea4b2d-istiod-ca-cert\") on node \"ip-10-0-140-107.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:36:37.202231 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:36:37.199936 4939 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/aa64537e-0c5b-49ae-9547-df65d9ea4b2d-workload-socket\") on node \"ip-10-0-140-107.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:36:37.202231 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:36:37.199960 4939 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/aa64537e-0c5b-49ae-9547-df65d9ea4b2d-workload-certs\") on node \"ip-10-0-140-107.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:36:37.202231 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:36:37.199992 4939 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/aa64537e-0c5b-49ae-9547-df65d9ea4b2d-istio-podinfo\") on node \"ip-10-0-140-107.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:36:37.202231 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:36:37.200008 4939 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/aa64537e-0c5b-49ae-9547-df65d9ea4b2d-credential-socket\") on node \"ip-10-0-140-107.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:36:37.202231 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:36:37.200024 4939 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/aa64537e-0c5b-49ae-9547-df65d9ea4b2d-istio-token\") on node \"ip-10-0-140-107.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:36:37.202231 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:36:37.200054 4939 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/aa64537e-0c5b-49ae-9547-df65d9ea4b2d-istio-data\") on node \"ip-10-0-140-107.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:36:37.202231 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:36:37.200070 4939 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jbgtv\" (UniqueName: \"kubernetes.io/projected/aa64537e-0c5b-49ae-9547-df65d9ea4b2d-kube-api-access-jbgtv\") on node \"ip-10-0-140-107.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:36:37.202231 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:36:37.200086 4939 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/aa64537e-0c5b-49ae-9547-df65d9ea4b2d-istio-envoy\") on node \"ip-10-0-140-107.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:36:37.340001 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:36:37.339970 4939 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-f7680d/distinct-w2-f7680d-kserve-756f874b98-qnpv8" Nov 03 01:36:37.522894 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:36:37.522787 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:36:37.704164 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:36:37.704127 4939 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-b-be59df-c7b8b85f5-vzcvs"] Nov 03 01:36:37.815735 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:36:37.815640 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:36:37.898209 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:36:37.898190 4939 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-ddd653-openshift-default-779ff6465-t9cvw" Nov 03 01:36:37.919446 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:36:37.919414 4939 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-a-ddd653-openshift-default-779ff6465-t9cvw"] Nov 03 01:36:37.927340 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:36:37.927303 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:36:37.928214 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:36:37.928188 4939 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-extmod-a-ddd653-openshift-default-779ff6465-t9cvw"] Nov 03 01:36:38.179972 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:36:38.179321 4939 patch_prober.go:29] interesting pod/e2e-shared-a-w2-531447-openshift-default-7995d46767-jcnlj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.79:15021/healthz/ready\": dial tcp 10.132.1.79:15021: connect: connection refused" start-of-body= Nov 03 01:36:38.179972 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:36:38.179399 4939 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-531447-openshift-default-7995d46767-jcnlj" podUID="aa4094a2-6349-4637-9630-40a4ee74ecce" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.79:15021/healthz/ready\": dial tcp 10.132.1.79:15021: connect: connection refused" Nov 03 01:36:39.174931 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:36:39.174387 4939 patch_prober.go:29] interesting pod/e2e-shared-a-w2-531447-openshift-default-7995d46767-jcnlj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.79:15021/healthz/ready\": dial tcp 10.132.1.79:15021: connect: connection refused" start-of-body= Nov 03 01:36:39.174931 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:36:39.174449 4939 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-531447-openshift-default-7995d46767-jcnlj" podUID="aa4094a2-6349-4637-9630-40a4ee74ecce" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.79:15021/healthz/ready\": dial tcp 10.132.1.79:15021: connect: connection refused" Nov 03 01:36:39.872774 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:36:39.872739 4939 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="aa64537e-0c5b-49ae-9547-df65d9ea4b2d" path="/var/lib/kubelet/pods/aa64537e-0c5b-49ae-9547-df65d9ea4b2d/volumes" Nov 03 01:36:40.103873 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:36:40.103832 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:36:40.181407 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:36:40.181324 4939 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w2-531447-openshift-default-7995d46767-jcnlj" Nov 03 01:36:40.196754 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:36:40.196721 4939 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w2-531447-openshift-default-7995d46767-jcnlj" Nov 03 01:36:40.197739 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:36:40.197711 4939 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w2-531447-openshift-default-7995d46767-jcnlj" Nov 03 01:36:40.218861 ip-10-0-140-107 kubenswrapper[4939]: E1103 01:36:40.218796 4939 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6712b32f_f3bd_4917_9e5e_101cf604625c.slice/crio-384febeac8280063b68f71ab034887b1d84a8e149d013ded2756f3066aeecdf7.scope\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6712b32f_f3bd_4917_9e5e_101cf604625c.slice/crio-conmon-384febeac8280063b68f71ab034887b1d84a8e149d013ded2756f3066aeecdf7.scope\": RecentStats: unable to find data in memory cache]" Nov 03 01:36:40.226441 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:36:40.226409 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:36:40.553489 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:36:40.553401 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:36:41.222434 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:36:41.222396 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:36:41.340494 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:36:41.340455 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:36:42.207767 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:36:42.207709 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:36:42.704826 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:36:42.704772 4939 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-extmod-b-be59df-c7b8b85f5-vzcvs" podUID="949acf43-3c7b-4074-b2f1-c323e596c786" containerName="payload-processing" containerID="cri-o://9a1dd6727bef01526e92714e2e8e033a4fbfa3f5582039418d39b033ae2c52f9" gracePeriod=25 Nov 03 01:36:43.033682 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:36:43.033582 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:36:43.698401 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:36:43.698367 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:36:44.370704 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:36:44.370670 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:36:44.808662 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:36:44.808584 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:36:44.981322 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:36:44.981277 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:36:45.626416 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:36:45.626374 4939 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-extmod-b-be59df-c7b8b85f5-vzcvs" podUID="949acf43-3c7b-4074-b2f1-c323e596c786" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 03 01:36:45.629159 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:36:45.629134 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:36:45.826579 ip-10-0-140-107 kubenswrapper[4939]: E1103 01:36:45.826542 4939 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6712b32f_f3bd_4917_9e5e_101cf604625c.slice/crio-conmon-384febeac8280063b68f71ab034887b1d84a8e149d013ded2756f3066aeecdf7.scope\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6712b32f_f3bd_4917_9e5e_101cf604625c.slice/crio-384febeac8280063b68f71ab034887b1d84a8e149d013ded2756f3066aeecdf7.scope\": RecentStats: unable to find data in memory cache]" Nov 03 01:36:45.832392 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:36:45.832361 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:36:46.449352 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:36:46.449295 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:36:46.859650 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:36:46.859607 4939 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w2-531447-openshift-default-7995d46767-gfccz"] Nov 03 01:36:46.860763 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:36:46.860739 4939 cpu_manager.go:414] "containerMap: removing container" podUID="aa64537e-0c5b-49ae-9547-df65d9ea4b2d" containerName="istio-proxy" Nov 03 01:36:46.860821 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:36:46.860765 4939 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="aa64537e-0c5b-49ae-9547-df65d9ea4b2d" containerName="istio-proxy" Nov 03 01:36:46.861052 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:36:46.861025 4939 memory_manager.go:365] "RemoveStaleState removing state" podUID="aa64537e-0c5b-49ae-9547-df65d9ea4b2d" containerName="istio-proxy" Nov 03 01:36:46.866527 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:36:46.866506 4939 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w2-531447-openshift-default-7995d46767-gfccz" Nov 03 01:36:46.877278 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:36:46.877016 4939 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-531447-openshift-default-7995d46767-gfccz"] Nov 03 01:36:46.899266 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:36:46.899210 4939 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w2-531447-openshift-default-7995d46767-jpz2z"] Nov 03 01:36:46.900757 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:36:46.900729 4939 cpu_manager.go:414] "containerMap: removing container" podUID="aa64537e-0c5b-49ae-9547-df65d9ea4b2d" containerName="istio-proxy" Nov 03 01:36:46.904965 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:36:46.904938 4939 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="aa64537e-0c5b-49ae-9547-df65d9ea4b2d" containerName="istio-proxy" Nov 03 01:36:46.905290 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:36:46.905235 4939 memory_manager.go:365] "RemoveStaleState removing state" podUID="aa64537e-0c5b-49ae-9547-df65d9ea4b2d" containerName="istio-proxy" Nov 03 01:36:46.914138 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:36:46.914109 4939 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w2-531447-openshift-default-7995d46767-jpz2z" Nov 03 01:36:46.915569 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:36:46.915540 4939 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-531447-openshift-default-7995d46767-jpz2z"] Nov 03 01:36:46.941984 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:36:46.941953 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fb776e00-2114-4f9c-86b1-bb790c3d64ca-istio-podinfo\") pod \"e2e-shared-a-w2-531447-openshift-default-7995d46767-gfccz\" (UID: \"fb776e00-2114-4f9c-86b1-bb790c3d64ca\") " pod="openshift-ingress/e2e-shared-a-w2-531447-openshift-default-7995d46767-gfccz" Nov 03 01:36:46.942166 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:36:46.941995 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fb776e00-2114-4f9c-86b1-bb790c3d64ca-istio-data\") pod \"e2e-shared-a-w2-531447-openshift-default-7995d46767-gfccz\" (UID: \"fb776e00-2114-4f9c-86b1-bb790c3d64ca\") " pod="openshift-ingress/e2e-shared-a-w2-531447-openshift-default-7995d46767-gfccz" Nov 03 01:36:46.942166 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:36:46.942021 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dqzqz\" (UniqueName: \"kubernetes.io/projected/fb776e00-2114-4f9c-86b1-bb790c3d64ca-kube-api-access-dqzqz\") pod \"e2e-shared-a-w2-531447-openshift-default-7995d46767-gfccz\" (UID: \"fb776e00-2114-4f9c-86b1-bb790c3d64ca\") " pod="openshift-ingress/e2e-shared-a-w2-531447-openshift-default-7995d46767-gfccz" Nov 03 01:36:46.942166 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:36:46.942111 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fb776e00-2114-4f9c-86b1-bb790c3d64ca-credential-socket\") pod \"e2e-shared-a-w2-531447-openshift-default-7995d46767-gfccz\" (UID: \"fb776e00-2114-4f9c-86b1-bb790c3d64ca\") " pod="openshift-ingress/e2e-shared-a-w2-531447-openshift-default-7995d46767-gfccz" Nov 03 01:36:46.942361 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:36:46.942166 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fb776e00-2114-4f9c-86b1-bb790c3d64ca-istiod-ca-cert\") pod \"e2e-shared-a-w2-531447-openshift-default-7995d46767-gfccz\" (UID: \"fb776e00-2114-4f9c-86b1-bb790c3d64ca\") " pod="openshift-ingress/e2e-shared-a-w2-531447-openshift-default-7995d46767-gfccz" Nov 03 01:36:46.942361 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:36:46.942272 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fb776e00-2114-4f9c-86b1-bb790c3d64ca-istio-token\") pod \"e2e-shared-a-w2-531447-openshift-default-7995d46767-gfccz\" (UID: \"fb776e00-2114-4f9c-86b1-bb790c3d64ca\") " pod="openshift-ingress/e2e-shared-a-w2-531447-openshift-default-7995d46767-gfccz" Nov 03 01:36:46.942485 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:36:46.942367 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fb776e00-2114-4f9c-86b1-bb790c3d64ca-workload-socket\") pod \"e2e-shared-a-w2-531447-openshift-default-7995d46767-gfccz\" (UID: \"fb776e00-2114-4f9c-86b1-bb790c3d64ca\") " pod="openshift-ingress/e2e-shared-a-w2-531447-openshift-default-7995d46767-gfccz" Nov 03 01:36:46.942485 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:36:46.942464 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fb776e00-2114-4f9c-86b1-bb790c3d64ca-istio-envoy\") pod \"e2e-shared-a-w2-531447-openshift-default-7995d46767-gfccz\" (UID: \"fb776e00-2114-4f9c-86b1-bb790c3d64ca\") " pod="openshift-ingress/e2e-shared-a-w2-531447-openshift-default-7995d46767-gfccz" Nov 03 01:36:46.942605 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:36:46.942582 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fb776e00-2114-4f9c-86b1-bb790c3d64ca-workload-certs\") pod \"e2e-shared-a-w2-531447-openshift-default-7995d46767-gfccz\" (UID: \"fb776e00-2114-4f9c-86b1-bb790c3d64ca\") " pod="openshift-ingress/e2e-shared-a-w2-531447-openshift-default-7995d46767-gfccz" Nov 03 01:36:47.044240 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:36:47.044199 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b7603676-9654-4c7f-ba1c-8a6a9c7e7382-istio-token\") pod \"e2e-shared-a-w2-531447-openshift-default-7995d46767-jpz2z\" (UID: \"b7603676-9654-4c7f-ba1c-8a6a9c7e7382\") " pod="openshift-ingress/e2e-shared-a-w2-531447-openshift-default-7995d46767-jpz2z" Nov 03 01:36:47.044240 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:36:47.044238 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b7603676-9654-4c7f-ba1c-8a6a9c7e7382-istiod-ca-cert\") pod \"e2e-shared-a-w2-531447-openshift-default-7995d46767-jpz2z\" (UID: \"b7603676-9654-4c7f-ba1c-8a6a9c7e7382\") " pod="openshift-ingress/e2e-shared-a-w2-531447-openshift-default-7995d46767-jpz2z" Nov 03 01:36:47.044520 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:36:47.044264 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fb776e00-2114-4f9c-86b1-bb790c3d64ca-istio-podinfo\") pod \"e2e-shared-a-w2-531447-openshift-default-7995d46767-gfccz\" (UID: \"fb776e00-2114-4f9c-86b1-bb790c3d64ca\") " pod="openshift-ingress/e2e-shared-a-w2-531447-openshift-default-7995d46767-gfccz" Nov 03 01:36:47.044520 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:36:47.044331 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fb776e00-2114-4f9c-86b1-bb790c3d64ca-istio-data\") pod \"e2e-shared-a-w2-531447-openshift-default-7995d46767-gfccz\" (UID: \"fb776e00-2114-4f9c-86b1-bb790c3d64ca\") " pod="openshift-ingress/e2e-shared-a-w2-531447-openshift-default-7995d46767-gfccz" Nov 03 01:36:47.044520 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:36:47.044393 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dqzqz\" (UniqueName: \"kubernetes.io/projected/fb776e00-2114-4f9c-86b1-bb790c3d64ca-kube-api-access-dqzqz\") pod \"e2e-shared-a-w2-531447-openshift-default-7995d46767-gfccz\" (UID: \"fb776e00-2114-4f9c-86b1-bb790c3d64ca\") " pod="openshift-ingress/e2e-shared-a-w2-531447-openshift-default-7995d46767-gfccz" Nov 03 01:36:47.044520 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:36:47.044433 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fb776e00-2114-4f9c-86b1-bb790c3d64ca-credential-socket\") pod \"e2e-shared-a-w2-531447-openshift-default-7995d46767-gfccz\" (UID: \"fb776e00-2114-4f9c-86b1-bb790c3d64ca\") " pod="openshift-ingress/e2e-shared-a-w2-531447-openshift-default-7995d46767-gfccz" Nov 03 01:36:47.044520 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:36:47.044477 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b7603676-9654-4c7f-ba1c-8a6a9c7e7382-istio-data\") pod \"e2e-shared-a-w2-531447-openshift-default-7995d46767-jpz2z\" (UID: \"b7603676-9654-4c7f-ba1c-8a6a9c7e7382\") " pod="openshift-ingress/e2e-shared-a-w2-531447-openshift-default-7995d46767-jpz2z" Nov 03 01:36:47.044863 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:36:47.044532 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fb776e00-2114-4f9c-86b1-bb790c3d64ca-istiod-ca-cert\") pod \"e2e-shared-a-w2-531447-openshift-default-7995d46767-gfccz\" (UID: \"fb776e00-2114-4f9c-86b1-bb790c3d64ca\") " pod="openshift-ingress/e2e-shared-a-w2-531447-openshift-default-7995d46767-gfccz" Nov 03 01:36:47.044863 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:36:47.044579 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fb776e00-2114-4f9c-86b1-bb790c3d64ca-istio-token\") pod \"e2e-shared-a-w2-531447-openshift-default-7995d46767-gfccz\" (UID: \"fb776e00-2114-4f9c-86b1-bb790c3d64ca\") " pod="openshift-ingress/e2e-shared-a-w2-531447-openshift-default-7995d46767-gfccz" Nov 03 01:36:47.044863 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:36:47.044695 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fb776e00-2114-4f9c-86b1-bb790c3d64ca-istio-data\") pod \"e2e-shared-a-w2-531447-openshift-default-7995d46767-gfccz\" (UID: \"fb776e00-2114-4f9c-86b1-bb790c3d64ca\") " pod="openshift-ingress/e2e-shared-a-w2-531447-openshift-default-7995d46767-gfccz" Nov 03 01:36:47.044863 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:36:47.044706 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fb776e00-2114-4f9c-86b1-bb790c3d64ca-workload-socket\") pod \"e2e-shared-a-w2-531447-openshift-default-7995d46767-gfccz\" (UID: \"fb776e00-2114-4f9c-86b1-bb790c3d64ca\") " pod="openshift-ingress/e2e-shared-a-w2-531447-openshift-default-7995d46767-gfccz" Nov 03 01:36:47.044863 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:36:47.044756 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fb776e00-2114-4f9c-86b1-bb790c3d64ca-credential-socket\") pod \"e2e-shared-a-w2-531447-openshift-default-7995d46767-gfccz\" (UID: \"fb776e00-2114-4f9c-86b1-bb790c3d64ca\") " pod="openshift-ingress/e2e-shared-a-w2-531447-openshift-default-7995d46767-gfccz" Nov 03 01:36:47.044863 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:36:47.044837 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fb776e00-2114-4f9c-86b1-bb790c3d64ca-istio-envoy\") pod \"e2e-shared-a-w2-531447-openshift-default-7995d46767-gfccz\" (UID: \"fb776e00-2114-4f9c-86b1-bb790c3d64ca\") " pod="openshift-ingress/e2e-shared-a-w2-531447-openshift-default-7995d46767-gfccz" Nov 03 01:36:47.045262 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:36:47.044965 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fb776e00-2114-4f9c-86b1-bb790c3d64ca-workload-certs\") pod \"e2e-shared-a-w2-531447-openshift-default-7995d46767-gfccz\" (UID: \"fb776e00-2114-4f9c-86b1-bb790c3d64ca\") " pod="openshift-ingress/e2e-shared-a-w2-531447-openshift-default-7995d46767-gfccz" Nov 03 01:36:47.045262 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:36:47.045007 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b7603676-9654-4c7f-ba1c-8a6a9c7e7382-workload-socket\") pod \"e2e-shared-a-w2-531447-openshift-default-7995d46767-jpz2z\" (UID: \"b7603676-9654-4c7f-ba1c-8a6a9c7e7382\") " pod="openshift-ingress/e2e-shared-a-w2-531447-openshift-default-7995d46767-jpz2z" Nov 03 01:36:47.045262 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:36:47.045025 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b7603676-9654-4c7f-ba1c-8a6a9c7e7382-workload-certs\") pod \"e2e-shared-a-w2-531447-openshift-default-7995d46767-jpz2z\" (UID: \"b7603676-9654-4c7f-ba1c-8a6a9c7e7382\") " pod="openshift-ingress/e2e-shared-a-w2-531447-openshift-default-7995d46767-jpz2z" Nov 03 01:36:47.045262 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:36:47.045047 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b7603676-9654-4c7f-ba1c-8a6a9c7e7382-istio-envoy\") pod \"e2e-shared-a-w2-531447-openshift-default-7995d46767-jpz2z\" (UID: \"b7603676-9654-4c7f-ba1c-8a6a9c7e7382\") " pod="openshift-ingress/e2e-shared-a-w2-531447-openshift-default-7995d46767-jpz2z" Nov 03 01:36:47.045262 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:36:47.045074 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fb776e00-2114-4f9c-86b1-bb790c3d64ca-workload-socket\") pod \"e2e-shared-a-w2-531447-openshift-default-7995d46767-gfccz\" (UID: \"fb776e00-2114-4f9c-86b1-bb790c3d64ca\") " pod="openshift-ingress/e2e-shared-a-w2-531447-openshift-default-7995d46767-gfccz" Nov 03 01:36:47.045262 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:36:47.045101 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b7603676-9654-4c7f-ba1c-8a6a9c7e7382-istio-podinfo\") pod \"e2e-shared-a-w2-531447-openshift-default-7995d46767-jpz2z\" (UID: \"b7603676-9654-4c7f-ba1c-8a6a9c7e7382\") " pod="openshift-ingress/e2e-shared-a-w2-531447-openshift-default-7995d46767-jpz2z" Nov 03 01:36:47.045262 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:36:47.045208 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b7603676-9654-4c7f-ba1c-8a6a9c7e7382-credential-socket\") pod \"e2e-shared-a-w2-531447-openshift-default-7995d46767-jpz2z\" (UID: \"b7603676-9654-4c7f-ba1c-8a6a9c7e7382\") " pod="openshift-ingress/e2e-shared-a-w2-531447-openshift-default-7995d46767-jpz2z" Nov 03 01:36:47.045594 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:36:47.045273 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7z8rk\" (UniqueName: \"kubernetes.io/projected/b7603676-9654-4c7f-ba1c-8a6a9c7e7382-kube-api-access-7z8rk\") pod \"e2e-shared-a-w2-531447-openshift-default-7995d46767-jpz2z\" (UID: \"b7603676-9654-4c7f-ba1c-8a6a9c7e7382\") " pod="openshift-ingress/e2e-shared-a-w2-531447-openshift-default-7995d46767-jpz2z" Nov 03 01:36:47.045594 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:36:47.045306 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fb776e00-2114-4f9c-86b1-bb790c3d64ca-istiod-ca-cert\") pod \"e2e-shared-a-w2-531447-openshift-default-7995d46767-gfccz\" (UID: \"fb776e00-2114-4f9c-86b1-bb790c3d64ca\") " pod="openshift-ingress/e2e-shared-a-w2-531447-openshift-default-7995d46767-gfccz" Nov 03 01:36:47.045594 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:36:47.045363 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fb776e00-2114-4f9c-86b1-bb790c3d64ca-workload-certs\") pod \"e2e-shared-a-w2-531447-openshift-default-7995d46767-gfccz\" (UID: \"fb776e00-2114-4f9c-86b1-bb790c3d64ca\") " pod="openshift-ingress/e2e-shared-a-w2-531447-openshift-default-7995d46767-gfccz" Nov 03 01:36:47.047157 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:36:47.047135 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fb776e00-2114-4f9c-86b1-bb790c3d64ca-istio-podinfo\") pod \"e2e-shared-a-w2-531447-openshift-default-7995d46767-gfccz\" (UID: \"fb776e00-2114-4f9c-86b1-bb790c3d64ca\") " pod="openshift-ingress/e2e-shared-a-w2-531447-openshift-default-7995d46767-gfccz" Nov 03 01:36:47.047334 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:36:47.047313 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fb776e00-2114-4f9c-86b1-bb790c3d64ca-istio-envoy\") pod \"e2e-shared-a-w2-531447-openshift-default-7995d46767-gfccz\" (UID: \"fb776e00-2114-4f9c-86b1-bb790c3d64ca\") " pod="openshift-ingress/e2e-shared-a-w2-531447-openshift-default-7995d46767-gfccz" Nov 03 01:36:47.052770 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:36:47.052741 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fb776e00-2114-4f9c-86b1-bb790c3d64ca-istio-token\") pod \"e2e-shared-a-w2-531447-openshift-default-7995d46767-gfccz\" (UID: \"fb776e00-2114-4f9c-86b1-bb790c3d64ca\") " pod="openshift-ingress/e2e-shared-a-w2-531447-openshift-default-7995d46767-gfccz" Nov 03 01:36:47.053428 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:36:47.053409 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dqzqz\" (UniqueName: \"kubernetes.io/projected/fb776e00-2114-4f9c-86b1-bb790c3d64ca-kube-api-access-dqzqz\") pod \"e2e-shared-a-w2-531447-openshift-default-7995d46767-gfccz\" (UID: \"fb776e00-2114-4f9c-86b1-bb790c3d64ca\") " pod="openshift-ingress/e2e-shared-a-w2-531447-openshift-default-7995d46767-gfccz" Nov 03 01:36:47.143962 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:36:47.143843 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:36:47.146615 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:36:47.146577 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b7603676-9654-4c7f-ba1c-8a6a9c7e7382-istio-podinfo\") pod \"e2e-shared-a-w2-531447-openshift-default-7995d46767-jpz2z\" (UID: \"b7603676-9654-4c7f-ba1c-8a6a9c7e7382\") " pod="openshift-ingress/e2e-shared-a-w2-531447-openshift-default-7995d46767-jpz2z" Nov 03 01:36:47.146800 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:36:47.146653 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b7603676-9654-4c7f-ba1c-8a6a9c7e7382-credential-socket\") pod \"e2e-shared-a-w2-531447-openshift-default-7995d46767-jpz2z\" (UID: \"b7603676-9654-4c7f-ba1c-8a6a9c7e7382\") " pod="openshift-ingress/e2e-shared-a-w2-531447-openshift-default-7995d46767-jpz2z" Nov 03 01:36:47.146800 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:36:47.146687 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7z8rk\" (UniqueName: \"kubernetes.io/projected/b7603676-9654-4c7f-ba1c-8a6a9c7e7382-kube-api-access-7z8rk\") pod \"e2e-shared-a-w2-531447-openshift-default-7995d46767-jpz2z\" (UID: \"b7603676-9654-4c7f-ba1c-8a6a9c7e7382\") " pod="openshift-ingress/e2e-shared-a-w2-531447-openshift-default-7995d46767-jpz2z" Nov 03 01:36:47.146989 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:36:47.146964 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b7603676-9654-4c7f-ba1c-8a6a9c7e7382-istio-token\") pod \"e2e-shared-a-w2-531447-openshift-default-7995d46767-jpz2z\" (UID: \"b7603676-9654-4c7f-ba1c-8a6a9c7e7382\") " pod="openshift-ingress/e2e-shared-a-w2-531447-openshift-default-7995d46767-jpz2z" Nov 03 01:36:47.147072 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:36:47.147024 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b7603676-9654-4c7f-ba1c-8a6a9c7e7382-istiod-ca-cert\") pod \"e2e-shared-a-w2-531447-openshift-default-7995d46767-jpz2z\" (UID: \"b7603676-9654-4c7f-ba1c-8a6a9c7e7382\") " pod="openshift-ingress/e2e-shared-a-w2-531447-openshift-default-7995d46767-jpz2z" Nov 03 01:36:47.147141 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:36:47.147085 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b7603676-9654-4c7f-ba1c-8a6a9c7e7382-istio-data\") pod \"e2e-shared-a-w2-531447-openshift-default-7995d46767-jpz2z\" (UID: \"b7603676-9654-4c7f-ba1c-8a6a9c7e7382\") " pod="openshift-ingress/e2e-shared-a-w2-531447-openshift-default-7995d46767-jpz2z" Nov 03 01:36:47.147459 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:36:47.147435 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b7603676-9654-4c7f-ba1c-8a6a9c7e7382-workload-socket\") pod \"e2e-shared-a-w2-531447-openshift-default-7995d46767-jpz2z\" (UID: \"b7603676-9654-4c7f-ba1c-8a6a9c7e7382\") " pod="openshift-ingress/e2e-shared-a-w2-531447-openshift-default-7995d46767-jpz2z" Nov 03 01:36:47.147551 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:36:47.147475 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b7603676-9654-4c7f-ba1c-8a6a9c7e7382-workload-certs\") pod \"e2e-shared-a-w2-531447-openshift-default-7995d46767-jpz2z\" (UID: \"b7603676-9654-4c7f-ba1c-8a6a9c7e7382\") " pod="openshift-ingress/e2e-shared-a-w2-531447-openshift-default-7995d46767-jpz2z" Nov 03 01:36:47.147551 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:36:47.147513 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b7603676-9654-4c7f-ba1c-8a6a9c7e7382-istio-envoy\") pod \"e2e-shared-a-w2-531447-openshift-default-7995d46767-jpz2z\" (UID: \"b7603676-9654-4c7f-ba1c-8a6a9c7e7382\") " pod="openshift-ingress/e2e-shared-a-w2-531447-openshift-default-7995d46767-jpz2z" Nov 03 01:36:47.147673 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:36:47.147642 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b7603676-9654-4c7f-ba1c-8a6a9c7e7382-credential-socket\") pod \"e2e-shared-a-w2-531447-openshift-default-7995d46767-jpz2z\" (UID: \"b7603676-9654-4c7f-ba1c-8a6a9c7e7382\") " pod="openshift-ingress/e2e-shared-a-w2-531447-openshift-default-7995d46767-jpz2z" Nov 03 01:36:47.147763 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:36:47.147739 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b7603676-9654-4c7f-ba1c-8a6a9c7e7382-istio-data\") pod \"e2e-shared-a-w2-531447-openshift-default-7995d46767-jpz2z\" (UID: \"b7603676-9654-4c7f-ba1c-8a6a9c7e7382\") " pod="openshift-ingress/e2e-shared-a-w2-531447-openshift-default-7995d46767-jpz2z" Nov 03 01:36:47.147973 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:36:47.147904 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b7603676-9654-4c7f-ba1c-8a6a9c7e7382-istiod-ca-cert\") pod \"e2e-shared-a-w2-531447-openshift-default-7995d46767-jpz2z\" (UID: \"b7603676-9654-4c7f-ba1c-8a6a9c7e7382\") " pod="openshift-ingress/e2e-shared-a-w2-531447-openshift-default-7995d46767-jpz2z" Nov 03 01:36:47.148085 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:36:47.147999 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b7603676-9654-4c7f-ba1c-8a6a9c7e7382-workload-socket\") pod \"e2e-shared-a-w2-531447-openshift-default-7995d46767-jpz2z\" (UID: \"b7603676-9654-4c7f-ba1c-8a6a9c7e7382\") " pod="openshift-ingress/e2e-shared-a-w2-531447-openshift-default-7995d46767-jpz2z" Nov 03 01:36:47.148085 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:36:47.148068 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b7603676-9654-4c7f-ba1c-8a6a9c7e7382-workload-certs\") pod \"e2e-shared-a-w2-531447-openshift-default-7995d46767-jpz2z\" (UID: \"b7603676-9654-4c7f-ba1c-8a6a9c7e7382\") " pod="openshift-ingress/e2e-shared-a-w2-531447-openshift-default-7995d46767-jpz2z" Nov 03 01:36:47.150234 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:36:47.150210 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b7603676-9654-4c7f-ba1c-8a6a9c7e7382-istio-podinfo\") pod \"e2e-shared-a-w2-531447-openshift-default-7995d46767-jpz2z\" (UID: \"b7603676-9654-4c7f-ba1c-8a6a9c7e7382\") " pod="openshift-ingress/e2e-shared-a-w2-531447-openshift-default-7995d46767-jpz2z" Nov 03 01:36:47.150383 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:36:47.150273 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b7603676-9654-4c7f-ba1c-8a6a9c7e7382-istio-envoy\") pod \"e2e-shared-a-w2-531447-openshift-default-7995d46767-jpz2z\" (UID: \"b7603676-9654-4c7f-ba1c-8a6a9c7e7382\") " pod="openshift-ingress/e2e-shared-a-w2-531447-openshift-default-7995d46767-jpz2z" Nov 03 01:36:47.155100 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:36:47.155074 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b7603676-9654-4c7f-ba1c-8a6a9c7e7382-istio-token\") pod \"e2e-shared-a-w2-531447-openshift-default-7995d46767-jpz2z\" (UID: \"b7603676-9654-4c7f-ba1c-8a6a9c7e7382\") " pod="openshift-ingress/e2e-shared-a-w2-531447-openshift-default-7995d46767-jpz2z" Nov 03 01:36:47.155236 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:36:47.155137 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7z8rk\" (UniqueName: \"kubernetes.io/projected/b7603676-9654-4c7f-ba1c-8a6a9c7e7382-kube-api-access-7z8rk\") pod \"e2e-shared-a-w2-531447-openshift-default-7995d46767-jpz2z\" (UID: \"b7603676-9654-4c7f-ba1c-8a6a9c7e7382\") " pod="openshift-ingress/e2e-shared-a-w2-531447-openshift-default-7995d46767-jpz2z" Nov 03 01:36:47.218360 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:36:47.218328 4939 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w2-531447-openshift-default-7995d46767-gfccz" Nov 03 01:36:47.252091 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:36:47.252058 4939 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w2-531447-openshift-default-7995d46767-jpz2z" Nov 03 01:36:47.478297 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:36:47.478199 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:36:47.507842 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:36:47.507809 4939 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-531447-openshift-default-7995d46767-gfccz"] Nov 03 01:36:47.508595 ip-10-0-140-107 kubenswrapper[4939]: W1103 01:36:47.508568 4939 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfb776e00_2114_4f9c_86b1_bb790c3d64ca.slice/crio-8eabfa514ab17e69492abb6a64c4066d7e6dba422e4376e70981a0e8ea50973f WatchSource:0}: Error finding container 8eabfa514ab17e69492abb6a64c4066d7e6dba422e4376e70981a0e8ea50973f: Status 404 returned error can't find the container with id 8eabfa514ab17e69492abb6a64c4066d7e6dba422e4376e70981a0e8ea50973f Nov 03 01:36:47.511641 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:36:47.511605 4939 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 01:36:47.511750 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:36:47.511703 4939 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 01:36:47.511820 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:36:47.511751 4939 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 01:36:47.550861 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:36:47.550833 4939 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-531447-openshift-default-7995d46767-jpz2z"] Nov 03 01:36:47.555331 ip-10-0-140-107 kubenswrapper[4939]: W1103 01:36:47.553410 4939 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb7603676_9654_4c7f_ba1c_8a6a9c7e7382.slice/crio-7f0397dc14202abf8f2e0962fdab8b7cc159e373b5955c0215adde1dbd895873 WatchSource:0}: Error finding container 7f0397dc14202abf8f2e0962fdab8b7cc159e373b5955c0215adde1dbd895873: Status 404 returned error can't find the container with id 7f0397dc14202abf8f2e0962fdab8b7cc159e373b5955c0215adde1dbd895873 Nov 03 01:36:47.560398 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:36:47.560365 4939 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 01:36:47.560499 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:36:47.560435 4939 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 01:36:47.560499 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:36:47.560464 4939 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 01:36:47.916621 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:36:47.916560 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:36:48.017994 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:36:48.017725 4939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-531447-openshift-default-7995d46767-jpz2z" event={"ID":"b7603676-9654-4c7f-ba1c-8a6a9c7e7382","Type":"ContainerStarted","Data":"663e1f52536f8a8802570c3e93687487dac156a592342f44a73a689467d24874"} Nov 03 01:36:48.017994 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:36:48.017816 4939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-531447-openshift-default-7995d46767-jpz2z" event={"ID":"b7603676-9654-4c7f-ba1c-8a6a9c7e7382","Type":"ContainerStarted","Data":"7f0397dc14202abf8f2e0962fdab8b7cc159e373b5955c0215adde1dbd895873"} Nov 03 01:36:48.019782 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:36:48.019743 4939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-531447-openshift-default-7995d46767-gfccz" event={"ID":"fb776e00-2114-4f9c-86b1-bb790c3d64ca","Type":"ContainerStarted","Data":"c3b5e51c1b87ee07927c892b6fdd6c9e9b924b2402f13910c7bea18ce7e58644"} Nov 03 01:36:48.020018 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:36:48.019811 4939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-531447-openshift-default-7995d46767-gfccz" event={"ID":"fb776e00-2114-4f9c-86b1-bb790c3d64ca","Type":"ContainerStarted","Data":"8eabfa514ab17e69492abb6a64c4066d7e6dba422e4376e70981a0e8ea50973f"} Nov 03 01:36:48.042263 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:36:48.042202 4939 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w2-531447-openshift-default-7995d46767-jpz2z" podStartSLOduration=2.042183507 podStartE2EDuration="2.042183507s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 01:36:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 01:36:48.036469757 +0000 UTC m=+3356.300804704" watchObservedRunningTime="2026-11-03 01:36:48.042183507 +0000 UTC m=+3356.306518453" Nov 03 01:36:48.046790 ip-10-0-140-107 kubenswrapper[4939]: E1103 01:36:48.046750 4939 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-sthjh" containerName="extractor" Nov 03 01:36:48.058571 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:36:48.058510 4939 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w2-531447-openshift-default-7995d46767-gfccz" podStartSLOduration=2.058494068 podStartE2EDuration="2.058494068s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 01:36:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 01:36:48.057037262 +0000 UTC m=+3356.321372209" watchObservedRunningTime="2026-11-03 01:36:48.058494068 +0000 UTC m=+3356.322829014" Nov 03 01:36:48.218684 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:36:48.218651 4939 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w2-531447-openshift-default-7995d46767-gfccz" Nov 03 01:36:48.220309 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:36:48.220279 4939 patch_prober.go:29] interesting pod/e2e-shared-a-w2-531447-openshift-default-7995d46767-gfccz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.80:15021/healthz/ready\": dial tcp 10.132.1.80:15021: connect: connection refused" start-of-body= Nov 03 01:36:48.220462 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:36:48.220332 4939 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-531447-openshift-default-7995d46767-gfccz" podUID="fb776e00-2114-4f9c-86b1-bb790c3d64ca" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.80:15021/healthz/ready\": dial tcp 10.132.1.80:15021: connect: connection refused" Nov 03 01:36:48.254326 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:36:48.254289 4939 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w2-531447-openshift-default-7995d46767-jpz2z" Nov 03 01:36:48.265725 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:36:48.262505 4939 patch_prober.go:29] interesting pod/e2e-shared-a-w2-531447-openshift-default-7995d46767-jpz2z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.81:15021/healthz/ready\": dial tcp 10.132.1.81:15021: connect: connection refused" start-of-body= Nov 03 01:36:48.265725 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:36:48.262590 4939 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-531447-openshift-default-7995d46767-jpz2z" podUID="b7603676-9654-4c7f-ba1c-8a6a9c7e7382" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.81:15021/healthz/ready\": dial tcp 10.132.1.81:15021: connect: connection refused" Nov 03 01:36:48.714802 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:36:48.714763 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:36:49.219297 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:36:49.219261 4939 patch_prober.go:29] interesting pod/e2e-shared-a-w2-531447-openshift-default-7995d46767-gfccz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.80:15021/healthz/ready\": dial tcp 10.132.1.80:15021: connect: connection refused" start-of-body= Nov 03 01:36:49.219709 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:36:49.219321 4939 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-531447-openshift-default-7995d46767-gfccz" podUID="fb776e00-2114-4f9c-86b1-bb790c3d64ca" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.80:15021/healthz/ready\": dial tcp 10.132.1.80:15021: connect: connection refused" Nov 03 01:36:49.269933 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:36:49.267653 4939 patch_prober.go:29] interesting pod/e2e-shared-a-w2-531447-openshift-default-7995d46767-jpz2z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.81:15021/healthz/ready\": dial tcp 10.132.1.81:15021: connect: connection refused" start-of-body= Nov 03 01:36:49.269933 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:36:49.267716 4939 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-531447-openshift-default-7995d46767-jpz2z" podUID="b7603676-9654-4c7f-ba1c-8a6a9c7e7382" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.81:15021/healthz/ready\": dial tcp 10.132.1.81:15021: connect: connection refused" Nov 03 01:36:49.610622 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:36:49.610595 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:36:50.219597 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:36:50.219558 4939 patch_prober.go:29] interesting pod/e2e-shared-a-w2-531447-openshift-default-7995d46767-gfccz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.80:15021/healthz/ready\": dial tcp 10.132.1.80:15021: connect: connection refused" start-of-body= Nov 03 01:36:50.220078 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:36:50.219616 4939 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-531447-openshift-default-7995d46767-gfccz" podUID="fb776e00-2114-4f9c-86b1-bb790c3d64ca" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.80:15021/healthz/ready\": dial tcp 10.132.1.80:15021: connect: connection refused" Nov 03 01:36:50.263933 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:36:50.254199 4939 patch_prober.go:29] interesting pod/e2e-shared-a-w2-531447-openshift-default-7995d46767-jpz2z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.81:15021/healthz/ready\": dial tcp 10.132.1.81:15021: connect: connection refused" start-of-body= Nov 03 01:36:50.263933 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:36:50.254256 4939 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-531447-openshift-default-7995d46767-jpz2z" podUID="b7603676-9654-4c7f-ba1c-8a6a9c7e7382" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.81:15021/healthz/ready\": dial tcp 10.132.1.81:15021: connect: connection refused" Nov 03 01:36:50.365671 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:36:50.365632 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:36:51.219716 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:36:51.219682 4939 patch_prober.go:29] interesting pod/e2e-shared-a-w2-531447-openshift-default-7995d46767-gfccz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.80:15021/healthz/ready\": dial tcp 10.132.1.80:15021: connect: connection refused" start-of-body= Nov 03 01:36:51.220276 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:36:51.219754 4939 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-531447-openshift-default-7995d46767-gfccz" podUID="fb776e00-2114-4f9c-86b1-bb790c3d64ca" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.80:15021/healthz/ready\": dial tcp 10.132.1.80:15021: connect: connection refused" Nov 03 01:36:51.259932 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:36:51.255326 4939 patch_prober.go:29] interesting pod/e2e-shared-a-w2-531447-openshift-default-7995d46767-jpz2z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.81:15021/healthz/ready\": dial tcp 10.132.1.81:15021: connect: connection refused" start-of-body= Nov 03 01:36:51.259932 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:36:51.255386 4939 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-531447-openshift-default-7995d46767-jpz2z" podUID="b7603676-9654-4c7f-ba1c-8a6a9c7e7382" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.81:15021/healthz/ready\": dial tcp 10.132.1.81:15021: connect: connection refused" Nov 03 01:36:51.324113 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:36:51.324061 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:36:51.970931 ip-10-0-140-107 kubenswrapper[4939]: E1103 01:36:51.970438 4939 fsHandler.go:119] failed to collect filesystem stats - rootDiskErr: could not stat "/var/lib/containers/storage/overlay/cb83c5a17ecef465abe535dbb09cb171079fabe8720144b1acb609a666ee7963/diff" to get inode usage: stat /var/lib/containers/storage/overlay/cb83c5a17ecef465abe535dbb09cb171079fabe8720144b1acb609a666ee7963/diff: no such file or directory, extraDiskErr: could not stat "/var/log/pods/openshift-ingress_payload-pre-processing-e2e-inf-b-b4971a-6c58f96447-rjn8w_6712b32f-f3bd-4917-9e5e-101cf604625c/payload-pre-processing/0.log" to get inode usage: stat /var/log/pods/openshift-ingress_payload-pre-processing-e2e-inf-b-b4971a-6c58f96447-rjn8w_6712b32f-f3bd-4917-9e5e-101cf604625c/payload-pre-processing/0.log: no such file or directory Nov 03 01:36:52.219048 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:36:52.219009 4939 patch_prober.go:29] interesting pod/e2e-shared-a-w2-531447-openshift-default-7995d46767-gfccz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.80:15021/healthz/ready\": dial tcp 10.132.1.80:15021: connect: connection refused" start-of-body= Nov 03 01:36:52.219237 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:36:52.219070 4939 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-531447-openshift-default-7995d46767-gfccz" podUID="fb776e00-2114-4f9c-86b1-bb790c3d64ca" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.80:15021/healthz/ready\": dial tcp 10.132.1.80:15021: connect: connection refused" Nov 03 01:36:52.239965 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:36:52.237690 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:36:52.259017 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:36:52.255198 4939 patch_prober.go:29] interesting pod/e2e-shared-a-w2-531447-openshift-default-7995d46767-jpz2z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.81:15021/healthz/ready\": dial tcp 10.132.1.81:15021: connect: connection refused" start-of-body= Nov 03 01:36:52.259017 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:36:52.255256 4939 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-531447-openshift-default-7995d46767-jpz2z" podUID="b7603676-9654-4c7f-ba1c-8a6a9c7e7382" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.81:15021/healthz/ready\": dial tcp 10.132.1.81:15021: connect: connection refused" Nov 03 01:36:53.082548 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:36:53.082445 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:36:53.219136 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:36:53.219102 4939 patch_prober.go:29] interesting pod/e2e-shared-a-w2-531447-openshift-default-7995d46767-gfccz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.80:15021/healthz/ready\": dial tcp 10.132.1.80:15021: connect: connection refused" start-of-body= Nov 03 01:36:53.219351 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:36:53.219330 4939 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-531447-openshift-default-7995d46767-gfccz" podUID="fb776e00-2114-4f9c-86b1-bb790c3d64ca" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.80:15021/healthz/ready\": dial tcp 10.132.1.80:15021: connect: connection refused" Nov 03 01:36:53.267527 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:36:53.267489 4939 patch_prober.go:29] interesting pod/e2e-shared-a-w2-531447-openshift-default-7995d46767-jpz2z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.81:15021/healthz/ready\": dial tcp 10.132.1.81:15021: connect: connection refused" start-of-body= Nov 03 01:36:53.268124 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:36:53.268087 4939 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-531447-openshift-default-7995d46767-jpz2z" podUID="b7603676-9654-4c7f-ba1c-8a6a9c7e7382" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.81:15021/healthz/ready\": dial tcp 10.132.1.81:15021: connect: connection refused" Nov 03 01:36:54.083688 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:36:54.083649 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:36:54.252475 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:36:54.252431 4939 patch_prober.go:29] interesting pod/e2e-shared-a-w2-531447-openshift-default-7995d46767-jpz2z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.81:15021/healthz/ready\": dial tcp 10.132.1.81:15021: connect: connection refused" start-of-body= Nov 03 01:36:54.252687 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:36:54.252486 4939 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-531447-openshift-default-7995d46767-jpz2z" podUID="b7603676-9654-4c7f-ba1c-8a6a9c7e7382" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.81:15021/healthz/ready\": dial tcp 10.132.1.81:15021: connect: connection refused" Nov 03 01:36:54.413783 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:36:54.413680 4939 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w2-531447-openshift-default-7995d46767-gfccz" Nov 03 01:36:54.415347 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:36:54.415310 4939 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w2-531447-openshift-default-7995d46767-gfccz" Nov 03 01:36:54.417088 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:36:54.417068 4939 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w2-531447-openshift-default-7995d46767-gfccz" Nov 03 01:36:54.821721 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:36:54.821647 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:36:55.230108 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:36:55.230073 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:36:55.259076 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:36:55.259042 4939 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w2-531447-openshift-default-7995d46767-jpz2z" Nov 03 01:36:55.259857 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:36:55.259831 4939 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w2-531447-openshift-default-7995d46767-jpz2z" Nov 03 01:36:55.260735 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:36:55.260701 4939 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w2-531447-openshift-default-7995d46767-jpz2z" Nov 03 01:36:55.625221 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:36:55.625181 4939 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-extmod-b-be59df-c7b8b85f5-vzcvs" podUID="949acf43-3c7b-4074-b2f1-c323e596c786" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 03 01:36:55.883181 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:36:55.883096 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:36:56.711294 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:36:56.711254 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:36:57.471738 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:36:57.471700 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:36:58.199017 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:36:58.198983 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:36:59.025349 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:36:59.025311 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:36:59.553400 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:36:59.553352 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:36:59.695475 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:36:59.695437 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:36:59.808866 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:36:59.808785 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:36:59.943873 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:36:59.943833 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:37:00.219185 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:37:00.219144 4939 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-b-be59df-openshift-default-6764f766d9-qhdnv"] Nov 03 01:37:00.219448 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:37:00.219413 4939 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-extmod-b-be59df-openshift-default-6764f766d9-qhdnv" podUID="d28c9737-e9a3-4497-8b53-a47062c1a2f1" containerName="istio-proxy" containerID="cri-o://1c44a6fcf3a764891ac19aee38afdf17cae5a3c6ae6eb3ee2bb9bfa23cad31ca" gracePeriod=30 Nov 03 01:37:00.277672 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:37:00.277635 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:37:00.847950 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:37:00.847894 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:37:00.888566 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:37:00.888529 4939 patch_prober.go:29] interesting pod/e2e-extmod-b-be59df-openshift-default-6764f766d9-qhdnv container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 03 01:37:00.888721 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:37:00.888599 4939 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-extmod-b-be59df-openshift-default-6764f766d9-qhdnv" podUID="d28c9737-e9a3-4497-8b53-a47062c1a2f1" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 03 01:37:01.019554 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:37:01.019522 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:37:01.520823 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:37:01.520790 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:37:01.650935 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:37:01.649897 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:37:02.338859 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:37:02.338828 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:37:02.978923 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:37:02.978843 4939 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-b-be59df-c7b8b85f5-vzcvs" Nov 03 01:37:03.088008 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:37:03.087966 4939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/949acf43-3c7b-4074-b2f1-c323e596c786-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/949acf43-3c7b-4074-b2f1-c323e596c786-plugins-config-volume\") pod \"949acf43-3c7b-4074-b2f1-c323e596c786\" (UID: \"949acf43-3c7b-4074-b2f1-c323e596c786\") " Nov 03 01:37:03.088181 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:37:03.088137 4939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/949acf43-3c7b-4074-b2f1-c323e596c786-kube-api-access-mwngw\" (UniqueName: \"kubernetes.io/projected/949acf43-3c7b-4074-b2f1-c323e596c786-kube-api-access-mwngw\") pod \"949acf43-3c7b-4074-b2f1-c323e596c786\" (UID: \"949acf43-3c7b-4074-b2f1-c323e596c786\") " Nov 03 01:37:03.088359 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:37:03.088337 4939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/949acf43-3c7b-4074-b2f1-c323e596c786-plugins-config-volume" pod "949acf43-3c7b-4074-b2f1-c323e596c786" (UID: "949acf43-3c7b-4074-b2f1-c323e596c786"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 01:37:03.088686 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:37:03.088669 4939 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/949acf43-3c7b-4074-b2f1-c323e596c786-plugins-config-volume\") on node \"ip-10-0-140-107.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:37:03.090308 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:37:03.090287 4939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/949acf43-3c7b-4074-b2f1-c323e596c786-kube-api-access-mwngw" pod "949acf43-3c7b-4074-b2f1-c323e596c786" (UID: "949acf43-3c7b-4074-b2f1-c323e596c786"). InnerVolumeSpecName "kube-api-access-mwngw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 01:37:03.190485 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:37:03.190449 4939 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mwngw\" (UniqueName: \"kubernetes.io/projected/949acf43-3c7b-4074-b2f1-c323e596c786-kube-api-access-mwngw\") on node \"ip-10-0-140-107.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:37:03.220313 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:37:03.220280 4939 generic.go:356] "Generic (PLEG): container finished" podID="949acf43-3c7b-4074-b2f1-c323e596c786" containerID="9a1dd6727bef01526e92714e2e8e033a4fbfa3f5582039418d39b033ae2c52f9" exitCode=0 Nov 03 01:37:03.220527 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:37:03.220328 4939 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-b-be59df-c7b8b85f5-vzcvs" Nov 03 01:37:03.220527 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:37:03.220350 4939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-b-be59df-c7b8b85f5-vzcvs" event={"ID":"949acf43-3c7b-4074-b2f1-c323e596c786","Type":"ContainerDied","Data":"9a1dd6727bef01526e92714e2e8e033a4fbfa3f5582039418d39b033ae2c52f9"} Nov 03 01:37:03.220527 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:37:03.220418 4939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-b-be59df-c7b8b85f5-vzcvs" event={"ID":"949acf43-3c7b-4074-b2f1-c323e596c786","Type":"ContainerDied","Data":"cefbec58f26871fab60825b2a33e004c9bae578eeb844a80ef343cb289d2c6b9"} Nov 03 01:37:03.220527 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:37:03.220449 4939 scope.go:122] "RemoveContainer" containerID="9a1dd6727bef01526e92714e2e8e033a4fbfa3f5582039418d39b033ae2c52f9" Nov 03 01:37:03.233157 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:37:03.233130 4939 scope.go:122] "RemoveContainer" containerID="9a1dd6727bef01526e92714e2e8e033a4fbfa3f5582039418d39b033ae2c52f9" Nov 03 01:37:03.233700 ip-10-0-140-107 kubenswrapper[4939]: E1103 01:37:03.233665 4939 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9a1dd6727bef01526e92714e2e8e033a4fbfa3f5582039418d39b033ae2c52f9\": container with ID starting with 9a1dd6727bef01526e92714e2e8e033a4fbfa3f5582039418d39b033ae2c52f9 not found: ID does not exist" containerID="9a1dd6727bef01526e92714e2e8e033a4fbfa3f5582039418d39b033ae2c52f9" Nov 03 01:37:03.233838 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:37:03.233709 4939 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9a1dd6727bef01526e92714e2e8e033a4fbfa3f5582039418d39b033ae2c52f9"} err="failed to get container status \"9a1dd6727bef01526e92714e2e8e033a4fbfa3f5582039418d39b033ae2c52f9\": rpc error: code = NotFound desc = could not find container \"9a1dd6727bef01526e92714e2e8e033a4fbfa3f5582039418d39b033ae2c52f9\": container with ID starting with 9a1dd6727bef01526e92714e2e8e033a4fbfa3f5582039418d39b033ae2c52f9 not found: ID does not exist" Nov 03 01:37:03.252851 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:37:03.250370 4939 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-b-be59df-c7b8b85f5-vzcvs"] Nov 03 01:37:03.254959 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:37:03.254927 4939 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-b-be59df-c7b8b85f5-vzcvs"] Nov 03 01:37:03.293223 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:37:03.293187 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:37:03.871429 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:37:03.871390 4939 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="949acf43-3c7b-4074-b2f1-c323e596c786" path="/var/lib/kubelet/pods/949acf43-3c7b-4074-b2f1-c323e596c786/volumes" Nov 03 01:37:04.071292 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:37:04.071255 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:37:04.891793 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:37:04.891758 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:37:05.536849 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:37:05.536822 4939 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-be59df-openshift-default-6764f766d9-qhdnv" Nov 03 01:37:05.630317 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:37:05.630275 4939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d28c9737-e9a3-4497-8b53-a47062c1a2f1-istio-token\" (UniqueName: \"kubernetes.io/projected/d28c9737-e9a3-4497-8b53-a47062c1a2f1-istio-token\") pod \"d28c9737-e9a3-4497-8b53-a47062c1a2f1\" (UID: \"d28c9737-e9a3-4497-8b53-a47062c1a2f1\") " Nov 03 01:37:05.630524 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:37:05.630329 4939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d28c9737-e9a3-4497-8b53-a47062c1a2f1-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d28c9737-e9a3-4497-8b53-a47062c1a2f1-workload-socket\") pod \"d28c9737-e9a3-4497-8b53-a47062c1a2f1\" (UID: \"d28c9737-e9a3-4497-8b53-a47062c1a2f1\") " Nov 03 01:37:05.630524 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:37:05.630374 4939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d28c9737-e9a3-4497-8b53-a47062c1a2f1-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d28c9737-e9a3-4497-8b53-a47062c1a2f1-workload-certs\") pod \"d28c9737-e9a3-4497-8b53-a47062c1a2f1\" (UID: \"d28c9737-e9a3-4497-8b53-a47062c1a2f1\") " Nov 03 01:37:05.630524 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:37:05.630448 4939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d28c9737-e9a3-4497-8b53-a47062c1a2f1-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d28c9737-e9a3-4497-8b53-a47062c1a2f1-istio-data\") pod \"d28c9737-e9a3-4497-8b53-a47062c1a2f1\" (UID: \"d28c9737-e9a3-4497-8b53-a47062c1a2f1\") " Nov 03 01:37:05.630524 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:37:05.630518 4939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d28c9737-e9a3-4497-8b53-a47062c1a2f1-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d28c9737-e9a3-4497-8b53-a47062c1a2f1-credential-socket\") pod \"d28c9737-e9a3-4497-8b53-a47062c1a2f1\" (UID: \"d28c9737-e9a3-4497-8b53-a47062c1a2f1\") " Nov 03 01:37:05.630750 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:37:05.630712 4939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d28c9737-e9a3-4497-8b53-a47062c1a2f1-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d28c9737-e9a3-4497-8b53-a47062c1a2f1-istio-envoy\") pod \"d28c9737-e9a3-4497-8b53-a47062c1a2f1\" (UID: \"d28c9737-e9a3-4497-8b53-a47062c1a2f1\") " Nov 03 01:37:05.630750 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:37:05.630698 4939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d28c9737-e9a3-4497-8b53-a47062c1a2f1-workload-socket" pod "d28c9737-e9a3-4497-8b53-a47062c1a2f1" (UID: "d28c9737-e9a3-4497-8b53-a47062c1a2f1"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 01:37:05.630872 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:37:05.630772 4939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d28c9737-e9a3-4497-8b53-a47062c1a2f1-workload-certs" pod "d28c9737-e9a3-4497-8b53-a47062c1a2f1" (UID: "d28c9737-e9a3-4497-8b53-a47062c1a2f1"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 01:37:05.630972 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:37:05.630948 4939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d28c9737-e9a3-4497-8b53-a47062c1a2f1-credential-socket" pod "d28c9737-e9a3-4497-8b53-a47062c1a2f1" (UID: "d28c9737-e9a3-4497-8b53-a47062c1a2f1"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 01:37:05.631044 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:37:05.630956 4939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/d28c9737-e9a3-4497-8b53-a47062c1a2f1-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d28c9737-e9a3-4497-8b53-a47062c1a2f1-istio-podinfo\") pod \"d28c9737-e9a3-4497-8b53-a47062c1a2f1\" (UID: \"d28c9737-e9a3-4497-8b53-a47062c1a2f1\") " Nov 03 01:37:05.631115 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:37:05.631090 4939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d28c9737-e9a3-4497-8b53-a47062c1a2f1-kube-api-access-ntph5\" (UniqueName: \"kubernetes.io/projected/d28c9737-e9a3-4497-8b53-a47062c1a2f1-kube-api-access-ntph5\") pod \"d28c9737-e9a3-4497-8b53-a47062c1a2f1\" (UID: \"d28c9737-e9a3-4497-8b53-a47062c1a2f1\") " Nov 03 01:37:05.631234 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:37:05.631207 4939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d28c9737-e9a3-4497-8b53-a47062c1a2f1-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d28c9737-e9a3-4497-8b53-a47062c1a2f1-istiod-ca-cert\") pod \"d28c9737-e9a3-4497-8b53-a47062c1a2f1\" (UID: \"d28c9737-e9a3-4497-8b53-a47062c1a2f1\") " Nov 03 01:37:05.632500 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:37:05.631965 4939 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d28c9737-e9a3-4497-8b53-a47062c1a2f1-workload-socket\") on node \"ip-10-0-140-107.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:37:05.632500 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:37:05.631994 4939 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d28c9737-e9a3-4497-8b53-a47062c1a2f1-workload-certs\") on node \"ip-10-0-140-107.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:37:05.632500 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:37:05.632008 4939 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d28c9737-e9a3-4497-8b53-a47062c1a2f1-credential-socket\") on node \"ip-10-0-140-107.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:37:05.632500 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:37:05.632277 4939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d28c9737-e9a3-4497-8b53-a47062c1a2f1-istio-data" pod "d28c9737-e9a3-4497-8b53-a47062c1a2f1" (UID: "d28c9737-e9a3-4497-8b53-a47062c1a2f1"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 01:37:05.632500 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:37:05.632472 4939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d28c9737-e9a3-4497-8b53-a47062c1a2f1-istiod-ca-cert" pod "d28c9737-e9a3-4497-8b53-a47062c1a2f1" (UID: "d28c9737-e9a3-4497-8b53-a47062c1a2f1"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 01:37:05.633184 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:37:05.633157 4939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d28c9737-e9a3-4497-8b53-a47062c1a2f1-istio-token" pod "d28c9737-e9a3-4497-8b53-a47062c1a2f1" (UID: "d28c9737-e9a3-4497-8b53-a47062c1a2f1"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 01:37:05.633413 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:37:05.633393 4939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d28c9737-e9a3-4497-8b53-a47062c1a2f1-istio-envoy" pod "d28c9737-e9a3-4497-8b53-a47062c1a2f1" (UID: "d28c9737-e9a3-4497-8b53-a47062c1a2f1"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 01:37:05.633494 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:37:05.633463 4939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/d28c9737-e9a3-4497-8b53-a47062c1a2f1-istio-podinfo" pod "d28c9737-e9a3-4497-8b53-a47062c1a2f1" (UID: "d28c9737-e9a3-4497-8b53-a47062c1a2f1"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 01:37:05.633625 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:37:05.633604 4939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d28c9737-e9a3-4497-8b53-a47062c1a2f1-kube-api-access-ntph5" pod "d28c9737-e9a3-4497-8b53-a47062c1a2f1" (UID: "d28c9737-e9a3-4497-8b53-a47062c1a2f1"). InnerVolumeSpecName "kube-api-access-ntph5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 01:37:05.687201 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:37:05.687161 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:37:05.734506 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:37:05.734360 4939 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d28c9737-e9a3-4497-8b53-a47062c1a2f1-istio-envoy\") on node \"ip-10-0-140-107.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:37:05.734506 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:37:05.734404 4939 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d28c9737-e9a3-4497-8b53-a47062c1a2f1-istio-podinfo\") on node \"ip-10-0-140-107.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:37:05.734506 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:37:05.734422 4939 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ntph5\" (UniqueName: \"kubernetes.io/projected/d28c9737-e9a3-4497-8b53-a47062c1a2f1-kube-api-access-ntph5\") on node \"ip-10-0-140-107.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:37:05.734506 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:37:05.734437 4939 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d28c9737-e9a3-4497-8b53-a47062c1a2f1-istiod-ca-cert\") on node \"ip-10-0-140-107.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:37:05.734506 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:37:05.734451 4939 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d28c9737-e9a3-4497-8b53-a47062c1a2f1-istio-token\") on node \"ip-10-0-140-107.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:37:05.734506 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:37:05.734464 4939 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d28c9737-e9a3-4497-8b53-a47062c1a2f1-istio-data\") on node \"ip-10-0-140-107.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:37:05.942842 ip-10-0-140-107 kubenswrapper[4939]: E1103 01:37:05.942809 4939 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd28c9737_e9a3_4497_8b53_a47062c1a2f1.slice/crio-b12cd2b74d7f00348b08f7f784747ac1b1f6f5f448e82bccd88b6627054e43cc\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd28c9737_e9a3_4497_8b53_a47062c1a2f1.slice\": RecentStats: unable to find data in memory cache]" Nov 03 01:37:06.247789 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:37:06.247704 4939 generic.go:356] "Generic (PLEG): container finished" podID="d28c9737-e9a3-4497-8b53-a47062c1a2f1" containerID="1c44a6fcf3a764891ac19aee38afdf17cae5a3c6ae6eb3ee2bb9bfa23cad31ca" exitCode=0 Nov 03 01:37:06.247789 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:37:06.247758 4939 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-be59df-openshift-default-6764f766d9-qhdnv" Nov 03 01:37:06.248045 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:37:06.247781 4939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-be59df-openshift-default-6764f766d9-qhdnv" event={"ID":"d28c9737-e9a3-4497-8b53-a47062c1a2f1","Type":"ContainerDied","Data":"1c44a6fcf3a764891ac19aee38afdf17cae5a3c6ae6eb3ee2bb9bfa23cad31ca"} Nov 03 01:37:06.248045 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:37:06.247848 4939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-be59df-openshift-default-6764f766d9-qhdnv" event={"ID":"d28c9737-e9a3-4497-8b53-a47062c1a2f1","Type":"ContainerDied","Data":"b12cd2b74d7f00348b08f7f784747ac1b1f6f5f448e82bccd88b6627054e43cc"} Nov 03 01:37:06.248045 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:37:06.247872 4939 scope.go:122] "RemoveContainer" containerID="1c44a6fcf3a764891ac19aee38afdf17cae5a3c6ae6eb3ee2bb9bfa23cad31ca" Nov 03 01:37:06.260221 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:37:06.260202 4939 scope.go:122] "RemoveContainer" containerID="4495666af2d7394c1f0f7974210ab5e4251abecaa13e8034a9f93b892b2a19d1" Nov 03 01:37:06.270683 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:37:06.270663 4939 scope.go:122] "RemoveContainer" containerID="1c44a6fcf3a764891ac19aee38afdf17cae5a3c6ae6eb3ee2bb9bfa23cad31ca" Nov 03 01:37:06.271017 ip-10-0-140-107 kubenswrapper[4939]: E1103 01:37:06.270996 4939 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1c44a6fcf3a764891ac19aee38afdf17cae5a3c6ae6eb3ee2bb9bfa23cad31ca\": container with ID starting with 1c44a6fcf3a764891ac19aee38afdf17cae5a3c6ae6eb3ee2bb9bfa23cad31ca not found: ID does not exist" containerID="1c44a6fcf3a764891ac19aee38afdf17cae5a3c6ae6eb3ee2bb9bfa23cad31ca" Nov 03 01:37:06.271108 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:37:06.271024 4939 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1c44a6fcf3a764891ac19aee38afdf17cae5a3c6ae6eb3ee2bb9bfa23cad31ca"} err="failed to get container status \"1c44a6fcf3a764891ac19aee38afdf17cae5a3c6ae6eb3ee2bb9bfa23cad31ca\": rpc error: code = NotFound desc = could not find container \"1c44a6fcf3a764891ac19aee38afdf17cae5a3c6ae6eb3ee2bb9bfa23cad31ca\": container with ID starting with 1c44a6fcf3a764891ac19aee38afdf17cae5a3c6ae6eb3ee2bb9bfa23cad31ca not found: ID does not exist" Nov 03 01:37:06.271108 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:37:06.271046 4939 scope.go:122] "RemoveContainer" containerID="4495666af2d7394c1f0f7974210ab5e4251abecaa13e8034a9f93b892b2a19d1" Nov 03 01:37:06.271284 ip-10-0-140-107 kubenswrapper[4939]: E1103 01:37:06.271264 4939 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4495666af2d7394c1f0f7974210ab5e4251abecaa13e8034a9f93b892b2a19d1\": container with ID starting with 4495666af2d7394c1f0f7974210ab5e4251abecaa13e8034a9f93b892b2a19d1 not found: ID does not exist" containerID="4495666af2d7394c1f0f7974210ab5e4251abecaa13e8034a9f93b892b2a19d1" Nov 03 01:37:06.271344 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:37:06.271290 4939 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4495666af2d7394c1f0f7974210ab5e4251abecaa13e8034a9f93b892b2a19d1"} err="failed to get container status \"4495666af2d7394c1f0f7974210ab5e4251abecaa13e8034a9f93b892b2a19d1\": rpc error: code = NotFound desc = could not find container \"4495666af2d7394c1f0f7974210ab5e4251abecaa13e8034a9f93b892b2a19d1\": container with ID starting with 4495666af2d7394c1f0f7974210ab5e4251abecaa13e8034a9f93b892b2a19d1 not found: ID does not exist" Nov 03 01:37:06.275260 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:37:06.275233 4939 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-b-be59df-openshift-default-6764f766d9-qhdnv"] Nov 03 01:37:06.279060 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:37:06.279038 4939 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-extmod-b-be59df-openshift-default-6764f766d9-qhdnv"] Nov 03 01:37:06.424634 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:37:06.424598 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:37:07.141947 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:37:07.141884 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:37:07.871404 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:37:07.871369 4939 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d28c9737-e9a3-4497-8b53-a47062c1a2f1" path="/var/lib/kubelet/pods/d28c9737-e9a3-4497-8b53-a47062c1a2f1/volumes" Nov 03 01:37:07.927841 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:37:07.927804 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:37:08.044561 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:37:08.044525 4939 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w0-50e623-746d8fcf47-tjbwb"] Nov 03 01:37:08.117872 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:37:08.117842 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:37:08.231783 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:37:08.231698 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:37:08.448183 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:37:08.448133 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:37:08.795823 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:37:08.795766 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:37:09.414922 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:37:09.414877 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:37:10.169769 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:37:10.169724 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:37:10.808877 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:37:10.808843 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:37:11.408034 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:37:11.407980 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:37:11.609400 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:37:11.609364 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:37:12.396828 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:37:12.396796 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:37:13.045016 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:37:13.044977 4939 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-50e623-746d8fcf47-tjbwb" podUID="b7d0152f-07a6-429b-8c20-e29139cf1d23" containerName="payload-pre-processing" containerID="cri-o://16d8bdb1497555572a0251253d1caf45892687b5a545b6f7024a88b2c49ddaf0" gracePeriod=25 Nov 03 01:37:13.074654 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:37:13.074619 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:37:13.321088 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:37:13.321059 4939 generic.go:356] "Generic (PLEG): container finished" podID="b7d0152f-07a6-429b-8c20-e29139cf1d23" containerID="16d8bdb1497555572a0251253d1caf45892687b5a545b6f7024a88b2c49ddaf0" exitCode=0 Nov 03 01:37:13.321233 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:37:13.321142 4939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-50e623-746d8fcf47-tjbwb" event={"ID":"b7d0152f-07a6-429b-8c20-e29139cf1d23","Type":"ContainerDied","Data":"16d8bdb1497555572a0251253d1caf45892687b5a545b6f7024a88b2c49ddaf0"} Nov 03 01:37:13.321233 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:37:13.321207 4939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-50e623-746d8fcf47-tjbwb" event={"ID":"b7d0152f-07a6-429b-8c20-e29139cf1d23","Type":"ContainerDied","Data":"5efdc889a686291f036f3ddc9dcc691a8ac1e44651aec47c511f775687ef1b7a"} Nov 03 01:37:13.321233 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:37:13.321230 4939 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="5efdc889a686291f036f3ddc9dcc691a8ac1e44651aec47c511f775687ef1b7a" Nov 03 01:37:13.339205 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:37:13.339184 4939 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-50e623-746d8fcf47-tjbwb" Nov 03 01:37:13.390898 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:37:13.390860 4939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b7d0152f-07a6-429b-8c20-e29139cf1d23-kube-api-access-s66zj\" (UniqueName: \"kubernetes.io/projected/b7d0152f-07a6-429b-8c20-e29139cf1d23-kube-api-access-s66zj\") pod \"b7d0152f-07a6-429b-8c20-e29139cf1d23\" (UID: \"b7d0152f-07a6-429b-8c20-e29139cf1d23\") " Nov 03 01:37:13.391128 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:37:13.390974 4939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b7d0152f-07a6-429b-8c20-e29139cf1d23-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b7d0152f-07a6-429b-8c20-e29139cf1d23-plugins-config-volume\") pod \"b7d0152f-07a6-429b-8c20-e29139cf1d23\" (UID: \"b7d0152f-07a6-429b-8c20-e29139cf1d23\") " Nov 03 01:37:13.391395 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:37:13.391361 4939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b7d0152f-07a6-429b-8c20-e29139cf1d23-plugins-config-volume" pod "b7d0152f-07a6-429b-8c20-e29139cf1d23" (UID: "b7d0152f-07a6-429b-8c20-e29139cf1d23"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 01:37:13.392004 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:37:13.391979 4939 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b7d0152f-07a6-429b-8c20-e29139cf1d23-plugins-config-volume\") on node \"ip-10-0-140-107.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:37:13.393323 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:37:13.393297 4939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b7d0152f-07a6-429b-8c20-e29139cf1d23-kube-api-access-s66zj" pod "b7d0152f-07a6-429b-8c20-e29139cf1d23" (UID: "b7d0152f-07a6-429b-8c20-e29139cf1d23"). InnerVolumeSpecName "kube-api-access-s66zj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 01:37:13.493452 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:37:13.493415 4939 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-s66zj\" (UniqueName: \"kubernetes.io/projected/b7d0152f-07a6-429b-8c20-e29139cf1d23-kube-api-access-s66zj\") on node \"ip-10-0-140-107.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:37:13.743839 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:37:13.743808 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:37:14.329660 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:37:14.329625 4939 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-50e623-746d8fcf47-tjbwb" Nov 03 01:37:14.350320 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:37:14.350293 4939 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w0-50e623-746d8fcf47-tjbwb"] Nov 03 01:37:14.354881 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:37:14.354858 4939 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w0-50e623-746d8fcf47-tjbwb"] Nov 03 01:37:14.524949 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:37:14.524867 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:37:14.729348 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:37:14.729317 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:37:14.862447 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:37:14.862369 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:37:15.300589 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:37:15.300536 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:37:15.491056 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:37:15.491020 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:37:15.618921 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:37:15.618876 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:37:15.869993 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:37:15.869898 4939 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b7d0152f-07a6-429b-8c20-e29139cf1d23" path="/var/lib/kubelet/pods/b7d0152f-07a6-429b-8c20-e29139cf1d23/volumes" Nov 03 01:37:16.326476 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:37:16.326358 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:37:16.449030 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:37:16.448985 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:37:17.059376 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:37:17.059333 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:37:17.645661 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:37:17.645628 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:37:17.965933 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:37:17.965832 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:37:18.037493 ip-10-0-140-107 kubenswrapper[4939]: E1103 01:37:18.037457 4939 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-sthjh" containerName="extractor" Nov 03 01:37:18.527420 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:37:18.527379 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:37:19.378278 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:37:19.378241 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:37:19.980724 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:37:19.980675 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:37:20.796306 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:37:20.796270 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:37:21.413063 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:37:21.413032 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:37:21.609114 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:37:21.609080 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:37:21.720271 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:37:21.720195 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:37:22.357623 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:37:22.357593 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:37:22.653964 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:37:22.653851 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:37:23.098061 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:37:23.098029 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:37:23.789210 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:37:23.789173 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:37:24.506076 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:37:24.506043 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:37:25.052854 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:37:25.052818 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:37:25.312020 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:37:25.311874 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:37:26.082536 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:37:26.082506 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:37:26.908779 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:37:26.908739 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:37:27.686523 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:37:27.686494 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:37:28.289389 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:37:28.289353 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:37:28.520277 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:37:28.520243 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:37:28.639070 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:37:28.639040 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:37:29.137208 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:37:29.137173 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:37:29.247007 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:37:29.246971 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:37:29.451590 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:37:29.451489 4939 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w0-020af9-openshift-default-d77df45d5-mpgn5"] Nov 03 01:37:29.451889 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:37:29.451852 4939 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w0-020af9-openshift-default-d77df45d5-mpgn5" podUID="d7701de4-582e-442f-abc1-13e20bb68b48" containerName="istio-proxy" containerID="cri-o://cbedf046d78219b00d277e77d21175532544301e6147a9198d528c309dc3e1af" gracePeriod=30 Nov 03 01:37:29.501605 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:37:29.501571 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:37:29.930979 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:37:29.930947 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:37:30.599356 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:37:30.596494 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:37:31.203864 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:37:31.203818 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:37:31.904680 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:37:31.904644 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:37:32.475113 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:37:32.475079 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:37:33.243403 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:37:33.243367 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:37:33.878973 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:37:33.878934 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:37:34.374465 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:37:34.374432 4939 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-4pw7v"] Nov 03 01:37:34.784374 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:37:34.784349 4939 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w0-020af9-openshift-default-d77df45d5-mpgn5" Nov 03 01:37:34.794806 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:37:34.794774 4939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/d7701de4-582e-442f-abc1-13e20bb68b48-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d7701de4-582e-442f-abc1-13e20bb68b48-istio-podinfo\") pod \"d7701de4-582e-442f-abc1-13e20bb68b48\" (UID: \"d7701de4-582e-442f-abc1-13e20bb68b48\") " Nov 03 01:37:34.794951 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:37:34.794849 4939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d7701de4-582e-442f-abc1-13e20bb68b48-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d7701de4-582e-442f-abc1-13e20bb68b48-workload-socket\") pod \"d7701de4-582e-442f-abc1-13e20bb68b48\" (UID: \"d7701de4-582e-442f-abc1-13e20bb68b48\") " Nov 03 01:37:34.794951 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:37:34.794889 4939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d7701de4-582e-442f-abc1-13e20bb68b48-kube-api-access-vnth5\" (UniqueName: \"kubernetes.io/projected/d7701de4-582e-442f-abc1-13e20bb68b48-kube-api-access-vnth5\") pod \"d7701de4-582e-442f-abc1-13e20bb68b48\" (UID: \"d7701de4-582e-442f-abc1-13e20bb68b48\") " Nov 03 01:37:34.795066 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:37:34.795021 4939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d7701de4-582e-442f-abc1-13e20bb68b48-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d7701de4-582e-442f-abc1-13e20bb68b48-credential-socket\") pod \"d7701de4-582e-442f-abc1-13e20bb68b48\" (UID: \"d7701de4-582e-442f-abc1-13e20bb68b48\") " Nov 03 01:37:34.795066 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:37:34.795057 4939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d7701de4-582e-442f-abc1-13e20bb68b48-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d7701de4-582e-442f-abc1-13e20bb68b48-istiod-ca-cert\") pod \"d7701de4-582e-442f-abc1-13e20bb68b48\" (UID: \"d7701de4-582e-442f-abc1-13e20bb68b48\") " Nov 03 01:37:34.795156 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:37:34.795090 4939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d7701de4-582e-442f-abc1-13e20bb68b48-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d7701de4-582e-442f-abc1-13e20bb68b48-workload-certs\") pod \"d7701de4-582e-442f-abc1-13e20bb68b48\" (UID: \"d7701de4-582e-442f-abc1-13e20bb68b48\") " Nov 03 01:37:34.795156 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:37:34.795122 4939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d7701de4-582e-442f-abc1-13e20bb68b48-istio-token\" (UniqueName: \"kubernetes.io/projected/d7701de4-582e-442f-abc1-13e20bb68b48-istio-token\") pod \"d7701de4-582e-442f-abc1-13e20bb68b48\" (UID: \"d7701de4-582e-442f-abc1-13e20bb68b48\") " Nov 03 01:37:34.795271 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:37:34.795170 4939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d7701de4-582e-442f-abc1-13e20bb68b48-workload-socket" pod "d7701de4-582e-442f-abc1-13e20bb68b48" (UID: "d7701de4-582e-442f-abc1-13e20bb68b48"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 01:37:34.795271 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:37:34.795241 4939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d7701de4-582e-442f-abc1-13e20bb68b48-credential-socket" pod "d7701de4-582e-442f-abc1-13e20bb68b48" (UID: "d7701de4-582e-442f-abc1-13e20bb68b48"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 01:37:34.795452 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:37:34.795422 4939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d7701de4-582e-442f-abc1-13e20bb68b48-workload-certs" pod "d7701de4-582e-442f-abc1-13e20bb68b48" (UID: "d7701de4-582e-442f-abc1-13e20bb68b48"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 01:37:34.795539 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:37:34.795455 4939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d7701de4-582e-442f-abc1-13e20bb68b48-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d7701de4-582e-442f-abc1-13e20bb68b48-istio-data\") pod \"d7701de4-582e-442f-abc1-13e20bb68b48\" (UID: \"d7701de4-582e-442f-abc1-13e20bb68b48\") " Nov 03 01:37:34.795539 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:37:34.795471 4939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d7701de4-582e-442f-abc1-13e20bb68b48-istiod-ca-cert" pod "d7701de4-582e-442f-abc1-13e20bb68b48" (UID: "d7701de4-582e-442f-abc1-13e20bb68b48"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 01:37:34.795539 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:37:34.795503 4939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d7701de4-582e-442f-abc1-13e20bb68b48-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d7701de4-582e-442f-abc1-13e20bb68b48-istio-envoy\") pod \"d7701de4-582e-442f-abc1-13e20bb68b48\" (UID: \"d7701de4-582e-442f-abc1-13e20bb68b48\") " Nov 03 01:37:34.795815 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:37:34.795739 4939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d7701de4-582e-442f-abc1-13e20bb68b48-istio-data" pod "d7701de4-582e-442f-abc1-13e20bb68b48" (UID: "d7701de4-582e-442f-abc1-13e20bb68b48"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 01:37:34.796365 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:37:34.796340 4939 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d7701de4-582e-442f-abc1-13e20bb68b48-credential-socket\") on node \"ip-10-0-140-107.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:37:34.796479 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:37:34.796371 4939 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d7701de4-582e-442f-abc1-13e20bb68b48-istiod-ca-cert\") on node \"ip-10-0-140-107.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:37:34.796479 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:37:34.796387 4939 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d7701de4-582e-442f-abc1-13e20bb68b48-workload-certs\") on node \"ip-10-0-140-107.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:37:34.796479 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:37:34.796402 4939 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d7701de4-582e-442f-abc1-13e20bb68b48-istio-data\") on node \"ip-10-0-140-107.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:37:34.796479 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:37:34.796416 4939 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d7701de4-582e-442f-abc1-13e20bb68b48-workload-socket\") on node \"ip-10-0-140-107.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:37:34.797169 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:37:34.797136 4939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/d7701de4-582e-442f-abc1-13e20bb68b48-istio-podinfo" pod "d7701de4-582e-442f-abc1-13e20bb68b48" (UID: "d7701de4-582e-442f-abc1-13e20bb68b48"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 01:37:34.797412 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:37:34.797395 4939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d7701de4-582e-442f-abc1-13e20bb68b48-kube-api-access-vnth5" pod "d7701de4-582e-442f-abc1-13e20bb68b48" (UID: "d7701de4-582e-442f-abc1-13e20bb68b48"). InnerVolumeSpecName "kube-api-access-vnth5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 01:37:34.797536 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:37:34.797519 4939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d7701de4-582e-442f-abc1-13e20bb68b48-istio-token" pod "d7701de4-582e-442f-abc1-13e20bb68b48" (UID: "d7701de4-582e-442f-abc1-13e20bb68b48"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 01:37:34.797605 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:37:34.797573 4939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d7701de4-582e-442f-abc1-13e20bb68b48-istio-envoy" pod "d7701de4-582e-442f-abc1-13e20bb68b48" (UID: "d7701de4-582e-442f-abc1-13e20bb68b48"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 01:37:34.897780 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:37:34.897740 4939 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d7701de4-582e-442f-abc1-13e20bb68b48-istio-envoy\") on node \"ip-10-0-140-107.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:37:34.897780 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:37:34.897780 4939 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d7701de4-582e-442f-abc1-13e20bb68b48-istio-podinfo\") on node \"ip-10-0-140-107.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:37:34.898032 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:37:34.897797 4939 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vnth5\" (UniqueName: \"kubernetes.io/projected/d7701de4-582e-442f-abc1-13e20bb68b48-kube-api-access-vnth5\") on node \"ip-10-0-140-107.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:37:34.898032 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:37:34.897814 4939 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d7701de4-582e-442f-abc1-13e20bb68b48-istio-token\") on node \"ip-10-0-140-107.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:37:35.118179 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:37:35.118145 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:37:35.321126 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:37:35.321071 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:37:35.589981 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:37:35.582543 4939 generic.go:356] "Generic (PLEG): container finished" podID="d7701de4-582e-442f-abc1-13e20bb68b48" containerID="cbedf046d78219b00d277e77d21175532544301e6147a9198d528c309dc3e1af" exitCode=0 Nov 03 01:37:35.589981 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:37:35.582792 4939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w0-020af9-openshift-default-d77df45d5-mpgn5" event={"ID":"d7701de4-582e-442f-abc1-13e20bb68b48","Type":"ContainerDied","Data":"cbedf046d78219b00d277e77d21175532544301e6147a9198d528c309dc3e1af"} Nov 03 01:37:35.589981 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:37:35.582846 4939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w0-020af9-openshift-default-d77df45d5-mpgn5" event={"ID":"d7701de4-582e-442f-abc1-13e20bb68b48","Type":"ContainerDied","Data":"0bc1bc040fddcd83d0ca53a57f427d7dd98b795ded8669fbf8bfaa73f94b3838"} Nov 03 01:37:35.589981 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:37:35.582879 4939 scope.go:122] "RemoveContainer" containerID="cbedf046d78219b00d277e77d21175532544301e6147a9198d528c309dc3e1af" Nov 03 01:37:35.589981 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:37:35.583075 4939 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w0-020af9-openshift-default-d77df45d5-mpgn5" Nov 03 01:37:35.628823 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:37:35.628799 4939 scope.go:122] "RemoveContainer" containerID="fd71a31300e5566a1081fb5c193940d6e11b98826ebca49ff97ee875fc10533b" Nov 03 01:37:35.662848 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:37:35.662722 4939 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w0-020af9-openshift-default-d77df45d5-mpgn5"] Nov 03 01:37:35.695452 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:37:35.695413 4939 scope.go:122] "RemoveContainer" containerID="cbedf046d78219b00d277e77d21175532544301e6147a9198d528c309dc3e1af" Nov 03 01:37:35.696173 ip-10-0-140-107 kubenswrapper[4939]: E1103 01:37:35.696147 4939 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cbedf046d78219b00d277e77d21175532544301e6147a9198d528c309dc3e1af\": container with ID starting with cbedf046d78219b00d277e77d21175532544301e6147a9198d528c309dc3e1af not found: ID does not exist" containerID="cbedf046d78219b00d277e77d21175532544301e6147a9198d528c309dc3e1af" Nov 03 01:37:35.696352 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:37:35.696326 4939 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cbedf046d78219b00d277e77d21175532544301e6147a9198d528c309dc3e1af"} err="failed to get container status \"cbedf046d78219b00d277e77d21175532544301e6147a9198d528c309dc3e1af\": rpc error: code = NotFound desc = could not find container \"cbedf046d78219b00d277e77d21175532544301e6147a9198d528c309dc3e1af\": container with ID starting with cbedf046d78219b00d277e77d21175532544301e6147a9198d528c309dc3e1af not found: ID does not exist" Nov 03 01:37:35.696467 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:37:35.696455 4939 scope.go:122] "RemoveContainer" containerID="fd71a31300e5566a1081fb5c193940d6e11b98826ebca49ff97ee875fc10533b" Nov 03 01:37:35.696923 ip-10-0-140-107 kubenswrapper[4939]: E1103 01:37:35.696885 4939 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fd71a31300e5566a1081fb5c193940d6e11b98826ebca49ff97ee875fc10533b\": container with ID starting with fd71a31300e5566a1081fb5c193940d6e11b98826ebca49ff97ee875fc10533b not found: ID does not exist" containerID="fd71a31300e5566a1081fb5c193940d6e11b98826ebca49ff97ee875fc10533b" Nov 03 01:37:35.697077 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:37:35.697051 4939 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fd71a31300e5566a1081fb5c193940d6e11b98826ebca49ff97ee875fc10533b"} err="failed to get container status \"fd71a31300e5566a1081fb5c193940d6e11b98826ebca49ff97ee875fc10533b\": rpc error: code = NotFound desc = could not find container \"fd71a31300e5566a1081fb5c193940d6e11b98826ebca49ff97ee875fc10533b\": container with ID starting with fd71a31300e5566a1081fb5c193940d6e11b98826ebca49ff97ee875fc10533b not found: ID does not exist" Nov 03 01:37:35.737409 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:37:35.737322 4939 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w0-020af9-openshift-default-d77df45d5-mpgn5"] Nov 03 01:37:35.875113 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:37:35.874891 4939 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d7701de4-582e-442f-abc1-13e20bb68b48" path="/var/lib/kubelet/pods/d7701de4-582e-442f-abc1-13e20bb68b48/volumes" Nov 03 01:37:36.291094 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:37:36.291004 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:37:37.058633 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:37:37.058594 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:37:37.610434 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:37:37.610398 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:37:37.626048 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:37:37.626022 4939 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-w1-2729f2/rate-test-model-2729f2-kserve-6d6745bb5c-dm56b"] Nov 03 01:37:37.626934 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:37:37.626888 4939 cpu_manager.go:414] "containerMap: removing container" podUID="d28c9737-e9a3-4497-8b53-a47062c1a2f1" containerName="istio-proxy" Nov 03 01:37:37.626934 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:37:37.626924 4939 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d28c9737-e9a3-4497-8b53-a47062c1a2f1" containerName="istio-proxy" Nov 03 01:37:37.627145 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:37:37.626948 4939 cpu_manager.go:414] "containerMap: removing container" podUID="949acf43-3c7b-4074-b2f1-c323e596c786" containerName="payload-processing" Nov 03 01:37:37.627145 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:37:37.626956 4939 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="949acf43-3c7b-4074-b2f1-c323e596c786" containerName="payload-processing" Nov 03 01:37:37.627145 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:37:37.626973 4939 cpu_manager.go:414] "containerMap: removing container" podUID="d7701de4-582e-442f-abc1-13e20bb68b48" containerName="istio-proxy" Nov 03 01:37:37.627145 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:37:37.626980 4939 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d7701de4-582e-442f-abc1-13e20bb68b48" containerName="istio-proxy" Nov 03 01:37:37.627145 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:37:37.626994 4939 cpu_manager.go:414] "containerMap: removing container" podUID="b7d0152f-07a6-429b-8c20-e29139cf1d23" containerName="payload-pre-processing" Nov 03 01:37:37.627145 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:37:37.627001 4939 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b7d0152f-07a6-429b-8c20-e29139cf1d23" containerName="payload-pre-processing" Nov 03 01:37:37.627145 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:37:37.627012 4939 cpu_manager.go:414] "containerMap: removing container" podUID="d7701de4-582e-442f-abc1-13e20bb68b48" containerName="istio-proxy" Nov 03 01:37:37.627145 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:37:37.627019 4939 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d7701de4-582e-442f-abc1-13e20bb68b48" containerName="istio-proxy" Nov 03 01:37:37.627145 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:37:37.627048 4939 cpu_manager.go:414] "containerMap: removing container" podUID="d28c9737-e9a3-4497-8b53-a47062c1a2f1" containerName="istio-proxy" Nov 03 01:37:37.627145 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:37:37.627056 4939 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d28c9737-e9a3-4497-8b53-a47062c1a2f1" containerName="istio-proxy" Nov 03 01:37:37.627692 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:37:37.627291 4939 memory_manager.go:365] "RemoveStaleState removing state" podUID="d7701de4-582e-442f-abc1-13e20bb68b48" containerName="istio-proxy" Nov 03 01:37:37.627692 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:37:37.627308 4939 memory_manager.go:365] "RemoveStaleState removing state" podUID="b7d0152f-07a6-429b-8c20-e29139cf1d23" containerName="payload-pre-processing" Nov 03 01:37:37.627692 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:37:37.627319 4939 memory_manager.go:365] "RemoveStaleState removing state" podUID="d7701de4-582e-442f-abc1-13e20bb68b48" containerName="istio-proxy" Nov 03 01:37:37.627692 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:37:37.627332 4939 memory_manager.go:365] "RemoveStaleState removing state" podUID="d28c9737-e9a3-4497-8b53-a47062c1a2f1" containerName="istio-proxy" Nov 03 01:37:37.627692 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:37:37.627352 4939 memory_manager.go:365] "RemoveStaleState removing state" podUID="d28c9737-e9a3-4497-8b53-a47062c1a2f1" containerName="istio-proxy" Nov 03 01:37:37.627692 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:37:37.627361 4939 memory_manager.go:365] "RemoveStaleState removing state" podUID="949acf43-3c7b-4074-b2f1-c323e596c786" containerName="payload-processing" Nov 03 01:37:37.636461 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:37:37.636441 4939 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w1-2729f2/rate-test-model-2729f2-kserve-6d6745bb5c-dm56b" Nov 03 01:37:37.642313 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:37:37.642284 4939 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w1-2729f2/rate-test-model-2729f2-kserve-6d6745bb5c-dm56b"] Nov 03 01:37:37.643105 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:37:37.643082 4939 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w1-2729f2\"/\"default-dockercfg-lhn8w\"" Nov 03 01:37:37.643240 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:37:37.643131 4939 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w1-2729f2\"/\"kube-root-ca.crt\"" Nov 03 01:37:37.643240 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:37:37.643171 4939 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w1-2729f2\"/\"openshift-service-ca.crt\"" Nov 03 01:37:37.643240 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:37:37.643177 4939 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w1-2729f2\"/\"rate-test-model-2729f2-kserve-self-signed-certs\"" Nov 03 01:37:37.860479 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:37:37.860405 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9z42c\" (UniqueName: \"kubernetes.io/projected/5b48a10a-63a5-440b-b08d-0e3190a77997-kube-api-access-9z42c\") pod \"rate-test-model-2729f2-kserve-6d6745bb5c-dm56b\" (UID: \"5b48a10a-63a5-440b-b08d-0e3190a77997\") " pod="ai-tenant-e2e-shared-a-w1-2729f2/rate-test-model-2729f2-kserve-6d6745bb5c-dm56b" Nov 03 01:37:37.860653 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:37:37.860557 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5b48a10a-63a5-440b-b08d-0e3190a77997-home\") pod \"rate-test-model-2729f2-kserve-6d6745bb5c-dm56b\" (UID: \"5b48a10a-63a5-440b-b08d-0e3190a77997\") " pod="ai-tenant-e2e-shared-a-w1-2729f2/rate-test-model-2729f2-kserve-6d6745bb5c-dm56b" Nov 03 01:37:37.860710 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:37:37.860680 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5b48a10a-63a5-440b-b08d-0e3190a77997-tmp-dir\") pod \"rate-test-model-2729f2-kserve-6d6745bb5c-dm56b\" (UID: \"5b48a10a-63a5-440b-b08d-0e3190a77997\") " pod="ai-tenant-e2e-shared-a-w1-2729f2/rate-test-model-2729f2-kserve-6d6745bb5c-dm56b" Nov 03 01:37:37.860867 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:37:37.860842 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5b48a10a-63a5-440b-b08d-0e3190a77997-model-cache\") pod \"rate-test-model-2729f2-kserve-6d6745bb5c-dm56b\" (UID: \"5b48a10a-63a5-440b-b08d-0e3190a77997\") " pod="ai-tenant-e2e-shared-a-w1-2729f2/rate-test-model-2729f2-kserve-6d6745bb5c-dm56b" Nov 03 01:37:37.861053 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:37:37.861030 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5b48a10a-63a5-440b-b08d-0e3190a77997-dshm\") pod \"rate-test-model-2729f2-kserve-6d6745bb5c-dm56b\" (UID: \"5b48a10a-63a5-440b-b08d-0e3190a77997\") " pod="ai-tenant-e2e-shared-a-w1-2729f2/rate-test-model-2729f2-kserve-6d6745bb5c-dm56b" Nov 03 01:37:37.861150 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:37:37.861136 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5b48a10a-63a5-440b-b08d-0e3190a77997-tls-certs\") pod \"rate-test-model-2729f2-kserve-6d6745bb5c-dm56b\" (UID: \"5b48a10a-63a5-440b-b08d-0e3190a77997\") " pod="ai-tenant-e2e-shared-a-w1-2729f2/rate-test-model-2729f2-kserve-6d6745bb5c-dm56b" Nov 03 01:37:37.922766 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:37:37.922733 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:37:37.974987 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:37:37.971512 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5b48a10a-63a5-440b-b08d-0e3190a77997-tmp-dir\") pod \"rate-test-model-2729f2-kserve-6d6745bb5c-dm56b\" (UID: \"5b48a10a-63a5-440b-b08d-0e3190a77997\") " pod="ai-tenant-e2e-shared-a-w1-2729f2/rate-test-model-2729f2-kserve-6d6745bb5c-dm56b" Nov 03 01:37:37.974987 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:37:37.971816 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5b48a10a-63a5-440b-b08d-0e3190a77997-model-cache\") pod \"rate-test-model-2729f2-kserve-6d6745bb5c-dm56b\" (UID: \"5b48a10a-63a5-440b-b08d-0e3190a77997\") " pod="ai-tenant-e2e-shared-a-w1-2729f2/rate-test-model-2729f2-kserve-6d6745bb5c-dm56b" Nov 03 01:37:37.974987 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:37:37.971988 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5b48a10a-63a5-440b-b08d-0e3190a77997-dshm\") pod \"rate-test-model-2729f2-kserve-6d6745bb5c-dm56b\" (UID: \"5b48a10a-63a5-440b-b08d-0e3190a77997\") " pod="ai-tenant-e2e-shared-a-w1-2729f2/rate-test-model-2729f2-kserve-6d6745bb5c-dm56b" Nov 03 01:37:37.974987 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:37:37.972084 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5b48a10a-63a5-440b-b08d-0e3190a77997-tls-certs\") pod \"rate-test-model-2729f2-kserve-6d6745bb5c-dm56b\" (UID: \"5b48a10a-63a5-440b-b08d-0e3190a77997\") " pod="ai-tenant-e2e-shared-a-w1-2729f2/rate-test-model-2729f2-kserve-6d6745bb5c-dm56b" Nov 03 01:37:37.974987 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:37:37.972141 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9z42c\" (UniqueName: \"kubernetes.io/projected/5b48a10a-63a5-440b-b08d-0e3190a77997-kube-api-access-9z42c\") pod \"rate-test-model-2729f2-kserve-6d6745bb5c-dm56b\" (UID: \"5b48a10a-63a5-440b-b08d-0e3190a77997\") " pod="ai-tenant-e2e-shared-a-w1-2729f2/rate-test-model-2729f2-kserve-6d6745bb5c-dm56b" Nov 03 01:37:37.974987 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:37:37.972235 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5b48a10a-63a5-440b-b08d-0e3190a77997-home\") pod \"rate-test-model-2729f2-kserve-6d6745bb5c-dm56b\" (UID: \"5b48a10a-63a5-440b-b08d-0e3190a77997\") " pod="ai-tenant-e2e-shared-a-w1-2729f2/rate-test-model-2729f2-kserve-6d6745bb5c-dm56b" Nov 03 01:37:37.980192 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:37:37.980110 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5b48a10a-63a5-440b-b08d-0e3190a77997-model-cache\") pod \"rate-test-model-2729f2-kserve-6d6745bb5c-dm56b\" (UID: \"5b48a10a-63a5-440b-b08d-0e3190a77997\") " pod="ai-tenant-e2e-shared-a-w1-2729f2/rate-test-model-2729f2-kserve-6d6745bb5c-dm56b" Nov 03 01:37:37.980654 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:37:37.980608 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5b48a10a-63a5-440b-b08d-0e3190a77997-tls-certs\") pod \"rate-test-model-2729f2-kserve-6d6745bb5c-dm56b\" (UID: \"5b48a10a-63a5-440b-b08d-0e3190a77997\") " pod="ai-tenant-e2e-shared-a-w1-2729f2/rate-test-model-2729f2-kserve-6d6745bb5c-dm56b" Nov 03 01:37:37.980959 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:37:37.980939 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5b48a10a-63a5-440b-b08d-0e3190a77997-tmp-dir\") pod \"rate-test-model-2729f2-kserve-6d6745bb5c-dm56b\" (UID: \"5b48a10a-63a5-440b-b08d-0e3190a77997\") " pod="ai-tenant-e2e-shared-a-w1-2729f2/rate-test-model-2729f2-kserve-6d6745bb5c-dm56b" Nov 03 01:37:37.981183 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:37:37.981158 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5b48a10a-63a5-440b-b08d-0e3190a77997-home\") pod \"rate-test-model-2729f2-kserve-6d6745bb5c-dm56b\" (UID: \"5b48a10a-63a5-440b-b08d-0e3190a77997\") " pod="ai-tenant-e2e-shared-a-w1-2729f2/rate-test-model-2729f2-kserve-6d6745bb5c-dm56b" Nov 03 01:37:37.987399 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:37:37.987376 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5b48a10a-63a5-440b-b08d-0e3190a77997-dshm\") pod \"rate-test-model-2729f2-kserve-6d6745bb5c-dm56b\" (UID: \"5b48a10a-63a5-440b-b08d-0e3190a77997\") " pod="ai-tenant-e2e-shared-a-w1-2729f2/rate-test-model-2729f2-kserve-6d6745bb5c-dm56b" Nov 03 01:37:37.992774 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:37:37.992753 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9z42c\" (UniqueName: \"kubernetes.io/projected/5b48a10a-63a5-440b-b08d-0e3190a77997-kube-api-access-9z42c\") pod \"rate-test-model-2729f2-kserve-6d6745bb5c-dm56b\" (UID: \"5b48a10a-63a5-440b-b08d-0e3190a77997\") " pod="ai-tenant-e2e-shared-a-w1-2729f2/rate-test-model-2729f2-kserve-6d6745bb5c-dm56b" Nov 03 01:37:38.251871 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:37:38.251791 4939 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w1-2729f2/rate-test-model-2729f2-kserve-6d6745bb5c-dm56b" Nov 03 01:37:38.533493 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:37:38.533463 4939 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w1-2729f2/rate-test-model-2729f2-kserve-6d6745bb5c-dm56b"] Nov 03 01:37:38.534394 ip-10-0-140-107 kubenswrapper[4939]: W1103 01:37:38.534366 4939 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod5b48a10a_63a5_440b_b08d_0e3190a77997.slice/crio-cbe4f516623fe32fdfeabf0ffda3addf8b8135c9ca9bdfb309016fd6656d6c5f WatchSource:0}: Error finding container cbe4f516623fe32fdfeabf0ffda3addf8b8135c9ca9bdfb309016fd6656d6c5f: Status 404 returned error can't find the container with id cbe4f516623fe32fdfeabf0ffda3addf8b8135c9ca9bdfb309016fd6656d6c5f Nov 03 01:37:38.607989 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:37:38.607957 4939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w1-2729f2/rate-test-model-2729f2-kserve-6d6745bb5c-dm56b" event={"ID":"5b48a10a-63a5-440b-b08d-0e3190a77997","Type":"ContainerStarted","Data":"cbe4f516623fe32fdfeabf0ffda3addf8b8135c9ca9bdfb309016fd6656d6c5f"} Nov 03 01:37:38.658783 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:37:38.658748 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:37:39.382824 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:37:39.382794 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:37:39.618121 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:37:39.618086 4939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w1-2729f2/rate-test-model-2729f2-kserve-6d6745bb5c-dm56b" event={"ID":"5b48a10a-63a5-440b-b08d-0e3190a77997","Type":"ContainerStarted","Data":"ae1a7f2c9c04b99985d987f515b6cce073c14ad1637969a50c79e97285d8ab45"} Nov 03 01:37:39.638101 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:37:39.638008 4939 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-w1-2729f2/rate-test-model-2729f2-kserve-6d6745bb5c-dm56b" podStartSLOduration=2.637993368 podStartE2EDuration="2.637993368s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 01:37:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 01:37:39.633990784 +0000 UTC m=+3407.898325729" watchObservedRunningTime="2026-11-03 01:37:39.637993368 +0000 UTC m=+3407.902328345" Nov 03 01:37:39.948367 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:37:39.948285 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:37:40.099169 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:37:40.099137 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:37:40.251974 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:37:40.251864 4939 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-w1-2729f2/rate-test-model-2729f2-kserve-6d6745bb5c-dm56b" Nov 03 01:37:40.944096 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:37:40.937527 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:37:41.580078 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:37:41.580045 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:37:42.300437 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:37:42.300401 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:37:43.211751 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:37:43.211720 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:37:43.984026 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:37:43.983991 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:37:44.695656 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:37:44.695622 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:37:44.896634 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:37:44.896600 4939 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-2f9a3a/distinct-w0-2f9a3a-kserve-555565cfcf-jtcr9"] Nov 03 01:37:44.913026 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:37:44.912997 4939 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-2f9a3a/simulator-w0-2f9a3a-kserve-9596bbb8f-2trgz"] Nov 03 01:37:45.012475 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:37:45.012384 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:37:45.391144 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:37:45.391108 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:37:45.597927 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:37:45.597881 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:37:46.095678 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:37:46.095640 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:37:46.842453 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:37:46.842415 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:37:47.615792 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:37:47.615758 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:37:48.033400 ip-10-0-140-107 kubenswrapper[4939]: E1103 01:37:48.033317 4939 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-sthjh" containerName="extractor" Nov 03 01:37:48.252520 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:37:48.252469 4939 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-w1-2729f2/rate-test-model-2729f2-kserve-6d6745bb5c-dm56b" Nov 03 01:37:48.265083 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:37:48.265060 4939 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-w1-2729f2/rate-test-model-2729f2-kserve-6d6745bb5c-dm56b" Nov 03 01:37:48.265876 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:37:48.265853 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:37:48.729944 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:37:48.729881 4939 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-w1-2729f2/rate-test-model-2729f2-kserve-6d6745bb5c-dm56b" Nov 03 01:37:48.892376 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:37:48.892347 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:37:49.596469 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:37:49.596437 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:37:50.307942 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:37:50.307878 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:37:51.079685 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:37:51.079646 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:37:51.696693 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:37:51.696660 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:37:52.294539 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:37:52.294500 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:37:52.431942 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:37:52.429422 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:37:52.693953 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:37:52.693884 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:37:53.072846 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:37:53.072765 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:37:53.706573 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:37:53.706540 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:37:54.395157 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:37:54.395122 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:37:55.089238 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:37:55.089182 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:37:55.856465 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:37:55.856434 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:37:56.497487 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:37:56.497451 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:37:57.292109 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:37:57.292077 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:37:58.150011 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:37:58.149975 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:37:58.362786 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:37:58.362747 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:37:58.970451 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:37:58.970393 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:37:59.114147 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:37:59.114112 4939 scope.go:122] "RemoveContainer" containerID="16d8bdb1497555572a0251253d1caf45892687b5a545b6f7024a88b2c49ddaf0" Nov 03 01:37:59.491222 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:37:59.491181 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:37:59.783617 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:37:59.783536 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:37:59.920994 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:37:59.920957 4939 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w0-2f9a3a/distinct-w0-2f9a3a-kserve-555565cfcf-jtcr9" podUID="ac0ece06-8718-463f-915f-5315988d9711" containerName="main" containerID="cri-o://80286a14e895c6364341c44e7045505b4d8a73753b16649945598afd3730fd5e" gracePeriod=45 Nov 03 01:37:59.943475 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:37:59.943432 4939 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w0-2f9a3a/simulator-w0-2f9a3a-kserve-9596bbb8f-2trgz" podUID="207e2207-fb6c-4076-95df-72e95671c261" containerName="main" containerID="cri-o://dfef807da95ad9c22322d8175eec57c184708ee6486d8d5363ac97e094c8deb4" gracePeriod=45 Nov 03 01:37:59.966706 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:37:59.966672 4939 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-2f9a3a/simulator-w0-2f9a3a-kserve-9596bbb8f-2trgz" podUID="207e2207-fb6c-4076-95df-72e95671c261" containerName="main" probeResult="failure" output="Get \"https://10.132.0.61:8000/ready\": dial tcp 10.132.0.61:8000: connect: connection refused" Nov 03 01:37:59.995184 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:37:59.995151 4939 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-2f9a3a/distinct-w0-2f9a3a-kserve-555565cfcf-jtcr9" podUID="ac0ece06-8718-463f-915f-5315988d9711" containerName="main" probeResult="failure" output="Get \"https://10.132.0.76:8000/ready\": dial tcp 10.132.0.76:8000: connect: connection refused" Nov 03 01:38:00.225636 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:38:00.225611 4939 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-2f9a3a/distinct-w0-2f9a3a-kserve-555565cfcf-jtcr9" Nov 03 01:38:00.264426 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:38:00.264388 4939 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-2f9a3a/simulator-w0-2f9a3a-kserve-9596bbb8f-2trgz" Nov 03 01:38:00.328591 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:38:00.328493 4939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ac0ece06-8718-463f-915f-5315988d9711-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ac0ece06-8718-463f-915f-5315988d9711-model-cache\") pod \"ac0ece06-8718-463f-915f-5315988d9711\" (UID: \"ac0ece06-8718-463f-915f-5315988d9711\") " Nov 03 01:38:00.328591 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:38:00.328563 4939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ac0ece06-8718-463f-915f-5315988d9711-tls-certs\" (UniqueName: \"kubernetes.io/secret/ac0ece06-8718-463f-915f-5315988d9711-tls-certs\") pod \"ac0ece06-8718-463f-915f-5315988d9711\" (UID: \"ac0ece06-8718-463f-915f-5315988d9711\") " Nov 03 01:38:00.328843 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:38:00.328633 4939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/207e2207-fb6c-4076-95df-72e95671c261-home\" (UniqueName: \"kubernetes.io/empty-dir/207e2207-fb6c-4076-95df-72e95671c261-home\") pod \"207e2207-fb6c-4076-95df-72e95671c261\" (UID: \"207e2207-fb6c-4076-95df-72e95671c261\") " Nov 03 01:38:00.328843 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:38:00.328691 4939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ac0ece06-8718-463f-915f-5315988d9711-dshm\" (UniqueName: \"kubernetes.io/empty-dir/ac0ece06-8718-463f-915f-5315988d9711-dshm\") pod \"ac0ece06-8718-463f-915f-5315988d9711\" (UID: \"ac0ece06-8718-463f-915f-5315988d9711\") " Nov 03 01:38:00.328843 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:38:00.328725 4939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/207e2207-fb6c-4076-95df-72e95671c261-tls-certs\" (UniqueName: \"kubernetes.io/secret/207e2207-fb6c-4076-95df-72e95671c261-tls-certs\") pod \"207e2207-fb6c-4076-95df-72e95671c261\" (UID: \"207e2207-fb6c-4076-95df-72e95671c261\") " Nov 03 01:38:00.328843 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:38:00.328788 4939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/207e2207-fb6c-4076-95df-72e95671c261-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/207e2207-fb6c-4076-95df-72e95671c261-tmp-dir\") pod \"207e2207-fb6c-4076-95df-72e95671c261\" (UID: \"207e2207-fb6c-4076-95df-72e95671c261\") " Nov 03 01:38:00.328843 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:38:00.328836 4939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ac0ece06-8718-463f-915f-5315988d9711-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ac0ece06-8718-463f-915f-5315988d9711-tmp-dir\") pod \"ac0ece06-8718-463f-915f-5315988d9711\" (UID: \"ac0ece06-8718-463f-915f-5315988d9711\") " Nov 03 01:38:00.329161 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:38:00.328947 4939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/207e2207-fb6c-4076-95df-72e95671c261-dshm\" (UniqueName: \"kubernetes.io/empty-dir/207e2207-fb6c-4076-95df-72e95671c261-dshm\") pod \"207e2207-fb6c-4076-95df-72e95671c261\" (UID: \"207e2207-fb6c-4076-95df-72e95671c261\") " Nov 03 01:38:00.329161 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:38:00.329001 4939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/207e2207-fb6c-4076-95df-72e95671c261-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/207e2207-fb6c-4076-95df-72e95671c261-model-cache\") pod \"207e2207-fb6c-4076-95df-72e95671c261\" (UID: \"207e2207-fb6c-4076-95df-72e95671c261\") " Nov 03 01:38:00.329161 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:38:00.329059 4939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ac0ece06-8718-463f-915f-5315988d9711-home\" (UniqueName: \"kubernetes.io/empty-dir/ac0ece06-8718-463f-915f-5315988d9711-home\") pod \"ac0ece06-8718-463f-915f-5315988d9711\" (UID: \"ac0ece06-8718-463f-915f-5315988d9711\") " Nov 03 01:38:00.329161 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:38:00.329091 4939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ac0ece06-8718-463f-915f-5315988d9711-kube-api-access-28rcq\" (UniqueName: \"kubernetes.io/projected/ac0ece06-8718-463f-915f-5315988d9711-kube-api-access-28rcq\") pod \"ac0ece06-8718-463f-915f-5315988d9711\" (UID: \"ac0ece06-8718-463f-915f-5315988d9711\") " Nov 03 01:38:00.329161 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:38:00.329125 4939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/207e2207-fb6c-4076-95df-72e95671c261-kube-api-access-4qvwx\" (UniqueName: \"kubernetes.io/projected/207e2207-fb6c-4076-95df-72e95671c261-kube-api-access-4qvwx\") pod \"207e2207-fb6c-4076-95df-72e95671c261\" (UID: \"207e2207-fb6c-4076-95df-72e95671c261\") " Nov 03 01:38:00.330039 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:38:00.329709 4939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/207e2207-fb6c-4076-95df-72e95671c261-model-cache" pod "207e2207-fb6c-4076-95df-72e95671c261" (UID: "207e2207-fb6c-4076-95df-72e95671c261"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 01:38:00.330039 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:38:00.329724 4939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/207e2207-fb6c-4076-95df-72e95671c261-tmp-dir" pod "207e2207-fb6c-4076-95df-72e95671c261" (UID: "207e2207-fb6c-4076-95df-72e95671c261"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 01:38:00.330039 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:38:00.329972 4939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ac0ece06-8718-463f-915f-5315988d9711-model-cache" pod "ac0ece06-8718-463f-915f-5315988d9711" (UID: "ac0ece06-8718-463f-915f-5315988d9711"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 01:38:00.330327 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:38:00.330144 4939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ac0ece06-8718-463f-915f-5315988d9711-home" pod "ac0ece06-8718-463f-915f-5315988d9711" (UID: "ac0ece06-8718-463f-915f-5315988d9711"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 01:38:00.330448 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:38:00.330394 4939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/207e2207-fb6c-4076-95df-72e95671c261-home" pod "207e2207-fb6c-4076-95df-72e95671c261" (UID: "207e2207-fb6c-4076-95df-72e95671c261"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 01:38:00.330448 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:38:00.330400 4939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ac0ece06-8718-463f-915f-5315988d9711-tmp-dir" pod "ac0ece06-8718-463f-915f-5315988d9711" (UID: "ac0ece06-8718-463f-915f-5315988d9711"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 01:38:00.333185 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:38:00.333154 4939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/207e2207-fb6c-4076-95df-72e95671c261-tls-certs" pod "207e2207-fb6c-4076-95df-72e95671c261" (UID: "207e2207-fb6c-4076-95df-72e95671c261"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 01:38:00.333286 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:38:00.333189 4939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ac0ece06-8718-463f-915f-5315988d9711-tls-certs" pod "ac0ece06-8718-463f-915f-5315988d9711" (UID: "ac0ece06-8718-463f-915f-5315988d9711"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 01:38:00.333370 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:38:00.333350 4939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/207e2207-fb6c-4076-95df-72e95671c261-dshm" pod "207e2207-fb6c-4076-95df-72e95671c261" (UID: "207e2207-fb6c-4076-95df-72e95671c261"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 01:38:00.333416 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:38:00.333361 4939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ac0ece06-8718-463f-915f-5315988d9711-kube-api-access-28rcq" pod "ac0ece06-8718-463f-915f-5315988d9711" (UID: "ac0ece06-8718-463f-915f-5315988d9711"). InnerVolumeSpecName "kube-api-access-28rcq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 01:38:00.333513 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:38:00.333494 4939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/207e2207-fb6c-4076-95df-72e95671c261-kube-api-access-4qvwx" pod "207e2207-fb6c-4076-95df-72e95671c261" (UID: "207e2207-fb6c-4076-95df-72e95671c261"). InnerVolumeSpecName "kube-api-access-4qvwx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 01:38:00.335165 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:38:00.335143 4939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ac0ece06-8718-463f-915f-5315988d9711-dshm" pod "ac0ece06-8718-463f-915f-5315988d9711" (UID: "ac0ece06-8718-463f-915f-5315988d9711"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 01:38:00.430878 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:38:00.430841 4939 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ac0ece06-8718-463f-915f-5315988d9711-dshm\") on node \"ip-10-0-140-107.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:38:00.430878 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:38:00.430876 4939 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/207e2207-fb6c-4076-95df-72e95671c261-tls-certs\") on node \"ip-10-0-140-107.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:38:00.431147 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:38:00.430891 4939 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/207e2207-fb6c-4076-95df-72e95671c261-tmp-dir\") on node \"ip-10-0-140-107.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:38:00.431147 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:38:00.430903 4939 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ac0ece06-8718-463f-915f-5315988d9711-tmp-dir\") on node \"ip-10-0-140-107.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:38:00.431147 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:38:00.430951 4939 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/207e2207-fb6c-4076-95df-72e95671c261-dshm\") on node \"ip-10-0-140-107.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:38:00.431147 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:38:00.430963 4939 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/207e2207-fb6c-4076-95df-72e95671c261-model-cache\") on node \"ip-10-0-140-107.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:38:00.431147 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:38:00.430975 4939 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ac0ece06-8718-463f-915f-5315988d9711-home\") on node \"ip-10-0-140-107.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:38:00.431147 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:38:00.430991 4939 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-28rcq\" (UniqueName: \"kubernetes.io/projected/ac0ece06-8718-463f-915f-5315988d9711-kube-api-access-28rcq\") on node \"ip-10-0-140-107.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:38:00.431147 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:38:00.431004 4939 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4qvwx\" (UniqueName: \"kubernetes.io/projected/207e2207-fb6c-4076-95df-72e95671c261-kube-api-access-4qvwx\") on node \"ip-10-0-140-107.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:38:00.431147 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:38:00.431018 4939 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ac0ece06-8718-463f-915f-5315988d9711-model-cache\") on node \"ip-10-0-140-107.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:38:00.431147 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:38:00.431032 4939 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ac0ece06-8718-463f-915f-5315988d9711-tls-certs\") on node \"ip-10-0-140-107.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:38:00.431147 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:38:00.431044 4939 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/207e2207-fb6c-4076-95df-72e95671c261-home\") on node \"ip-10-0-140-107.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:38:00.510010 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:38:00.509978 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:38:00.839149 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:38:00.839109 4939 generic.go:356] "Generic (PLEG): container finished" podID="ac0ece06-8718-463f-915f-5315988d9711" containerID="80286a14e895c6364341c44e7045505b4d8a73753b16649945598afd3730fd5e" exitCode=0 Nov 03 01:38:00.839391 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:38:00.839375 4939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-2f9a3a/distinct-w0-2f9a3a-kserve-555565cfcf-jtcr9" event={"ID":"ac0ece06-8718-463f-915f-5315988d9711","Type":"ContainerDied","Data":"80286a14e895c6364341c44e7045505b4d8a73753b16649945598afd3730fd5e"} Nov 03 01:38:00.839445 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:38:00.839408 4939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-2f9a3a/distinct-w0-2f9a3a-kserve-555565cfcf-jtcr9" event={"ID":"ac0ece06-8718-463f-915f-5315988d9711","Type":"ContainerDied","Data":"3835b2b377d056f2885d14046786441888c8c501770596213edbfc7f6f4c1ad5"} Nov 03 01:38:00.839445 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:38:00.839424 4939 scope.go:122] "RemoveContainer" containerID="80286a14e895c6364341c44e7045505b4d8a73753b16649945598afd3730fd5e" Nov 03 01:38:00.839552 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:38:00.839542 4939 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-2f9a3a/distinct-w0-2f9a3a-kserve-555565cfcf-jtcr9" Nov 03 01:38:00.841386 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:38:00.841357 4939 generic.go:356] "Generic (PLEG): container finished" podID="207e2207-fb6c-4076-95df-72e95671c261" containerID="dfef807da95ad9c22322d8175eec57c184708ee6486d8d5363ac97e094c8deb4" exitCode=0 Nov 03 01:38:00.841515 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:38:00.841410 4939 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-2f9a3a/simulator-w0-2f9a3a-kserve-9596bbb8f-2trgz" Nov 03 01:38:00.841515 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:38:00.841436 4939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-2f9a3a/simulator-w0-2f9a3a-kserve-9596bbb8f-2trgz" event={"ID":"207e2207-fb6c-4076-95df-72e95671c261","Type":"ContainerDied","Data":"dfef807da95ad9c22322d8175eec57c184708ee6486d8d5363ac97e094c8deb4"} Nov 03 01:38:00.841515 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:38:00.841470 4939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-2f9a3a/simulator-w0-2f9a3a-kserve-9596bbb8f-2trgz" event={"ID":"207e2207-fb6c-4076-95df-72e95671c261","Type":"ContainerDied","Data":"d5ac9e84b2bffd4fe57f914fd389bbda4c03c22e4a04644567e20f8221feaa93"} Nov 03 01:38:00.854745 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:38:00.854720 4939 scope.go:122] "RemoveContainer" containerID="80286a14e895c6364341c44e7045505b4d8a73753b16649945598afd3730fd5e" Nov 03 01:38:00.855363 ip-10-0-140-107 kubenswrapper[4939]: E1103 01:38:00.855248 4939 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"80286a14e895c6364341c44e7045505b4d8a73753b16649945598afd3730fd5e\": container with ID starting with 80286a14e895c6364341c44e7045505b4d8a73753b16649945598afd3730fd5e not found: ID does not exist" containerID="80286a14e895c6364341c44e7045505b4d8a73753b16649945598afd3730fd5e" Nov 03 01:38:00.855363 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:38:00.855297 4939 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"80286a14e895c6364341c44e7045505b4d8a73753b16649945598afd3730fd5e"} err="failed to get container status \"80286a14e895c6364341c44e7045505b4d8a73753b16649945598afd3730fd5e\": rpc error: code = NotFound desc = could not find container \"80286a14e895c6364341c44e7045505b4d8a73753b16649945598afd3730fd5e\": container with ID starting with 80286a14e895c6364341c44e7045505b4d8a73753b16649945598afd3730fd5e not found: ID does not exist" Nov 03 01:38:00.855363 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:38:00.855319 4939 scope.go:122] "RemoveContainer" containerID="dfef807da95ad9c22322d8175eec57c184708ee6486d8d5363ac97e094c8deb4" Nov 03 01:38:00.865207 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:38:00.865184 4939 scope.go:122] "RemoveContainer" containerID="dfef807da95ad9c22322d8175eec57c184708ee6486d8d5363ac97e094c8deb4" Nov 03 01:38:00.865538 ip-10-0-140-107 kubenswrapper[4939]: E1103 01:38:00.865498 4939 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"dfef807da95ad9c22322d8175eec57c184708ee6486d8d5363ac97e094c8deb4\": container with ID starting with dfef807da95ad9c22322d8175eec57c184708ee6486d8d5363ac97e094c8deb4 not found: ID does not exist" containerID="dfef807da95ad9c22322d8175eec57c184708ee6486d8d5363ac97e094c8deb4" Nov 03 01:38:00.865637 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:38:00.865537 4939 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"dfef807da95ad9c22322d8175eec57c184708ee6486d8d5363ac97e094c8deb4"} err="failed to get container status \"dfef807da95ad9c22322d8175eec57c184708ee6486d8d5363ac97e094c8deb4\": rpc error: code = NotFound desc = could not find container \"dfef807da95ad9c22322d8175eec57c184708ee6486d8d5363ac97e094c8deb4\": container with ID starting with dfef807da95ad9c22322d8175eec57c184708ee6486d8d5363ac97e094c8deb4 not found: ID does not exist" Nov 03 01:38:00.870119 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:38:00.870096 4939 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-2f9a3a/distinct-w0-2f9a3a-kserve-555565cfcf-jtcr9"] Nov 03 01:38:00.875431 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:38:00.875408 4939 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w0-2f9a3a/distinct-w0-2f9a3a-kserve-555565cfcf-jtcr9"] Nov 03 01:38:00.892413 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:38:00.892372 4939 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-2f9a3a/simulator-w0-2f9a3a-kserve-9596bbb8f-2trgz"] Nov 03 01:38:00.898385 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:38:00.898359 4939 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w0-2f9a3a/simulator-w0-2f9a3a-kserve-9596bbb8f-2trgz"] Nov 03 01:38:01.185522 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:38:01.185434 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:38:01.869874 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:38:01.869838 4939 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="207e2207-fb6c-4076-95df-72e95671c261" path="/var/lib/kubelet/pods/207e2207-fb6c-4076-95df-72e95671c261/volumes" Nov 03 01:38:01.870358 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:38:01.870342 4939 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ac0ece06-8718-463f-915f-5315988d9711" path="/var/lib/kubelet/pods/ac0ece06-8718-463f-915f-5315988d9711/volumes" Nov 03 01:38:01.872656 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:38:01.872632 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:38:02.508666 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:38:02.508633 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:38:03.164984 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:38:03.164945 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:38:03.880134 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:38:03.880103 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:38:04.682047 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:38:04.682012 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:38:05.209029 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:38:05.208995 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:38:05.890822 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:38:05.890766 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:38:06.043465 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:38:06.043436 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:38:06.514943 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:38:06.514891 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:38:07.264595 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:38:07.264546 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:38:07.905772 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:38:07.905739 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:38:08.654888 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:38:08.650994 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:38:09.386222 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:38:09.386184 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:38:09.986994 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:38:09.986952 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:38:10.708582 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:38:10.708540 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:38:11.687637 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:38:11.687604 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:38:12.314956 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:38:12.314904 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:38:13.064831 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:38:13.064796 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:38:13.191246 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:38:13.191213 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:38:13.798221 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:38:13.798180 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:38:14.381006 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:38:14.380968 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:38:14.976488 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:38:14.976458 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:38:15.620066 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:38:15.620026 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:38:16.307045 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:38:16.307009 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:38:17.004210 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:38:17.004170 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:38:17.695811 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:38:17.695768 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:38:18.037480 ip-10-0-140-107 kubenswrapper[4939]: E1103 01:38:18.037385 4939 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-sthjh" containerName="extractor" Nov 03 01:38:18.324529 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:38:18.324432 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:38:19.067554 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:38:19.067522 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:38:19.701983 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:38:19.701942 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:38:20.377759 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:38:20.377723 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:38:21.059459 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:38:21.059411 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:38:21.812138 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:38:21.812098 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:38:22.726697 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:38:22.726641 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:38:23.121629 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:38:23.121598 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:38:23.224150 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:38:23.224112 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:38:23.399457 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:38:23.399374 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:38:23.593508 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:38:23.593471 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:38:24.100156 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:38:24.100112 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:38:24.821510 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:38:24.821477 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:38:25.411183 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:38:25.411149 4939 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w0-2f9a3a-7b485fd99-hjcmm"] Nov 03 01:38:25.411458 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:38:25.411412 4939 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-2f9a3a-7b485fd99-hjcmm" podUID="5240e2be-d3c2-45ef-8165-aec738684748" containerName="maas-api" containerID="cri-o://b8846f80d56673691d0d98e1a01abf8a1bd755ec0ffe0b7f95cfa9d40f91724b" gracePeriod=30 Nov 03 01:38:25.479516 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:38:25.479488 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:38:25.786434 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:38:25.785361 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:38:25.890595 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:38:25.890550 4939 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-2f9a3a-7b485fd99-hjcmm" Nov 03 01:38:25.991117 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:38:25.991080 4939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5240e2be-d3c2-45ef-8165-aec738684748-kube-api-access-zl5rf\" (UniqueName: \"kubernetes.io/projected/5240e2be-d3c2-45ef-8165-aec738684748-kube-api-access-zl5rf\") pod \"5240e2be-d3c2-45ef-8165-aec738684748\" (UID: \"5240e2be-d3c2-45ef-8165-aec738684748\") " Nov 03 01:38:25.991303 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:38:25.991153 4939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5240e2be-d3c2-45ef-8165-aec738684748-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5240e2be-d3c2-45ef-8165-aec738684748-openshift-service-ca\") pod \"5240e2be-d3c2-45ef-8165-aec738684748\" (UID: \"5240e2be-d3c2-45ef-8165-aec738684748\") " Nov 03 01:38:25.991303 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:38:25.991223 4939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5240e2be-d3c2-45ef-8165-aec738684748-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5240e2be-d3c2-45ef-8165-aec738684748-maas-api-tls\") pod \"5240e2be-d3c2-45ef-8165-aec738684748\" (UID: \"5240e2be-d3c2-45ef-8165-aec738684748\") " Nov 03 01:38:25.991461 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:38:25.991444 4939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5240e2be-d3c2-45ef-8165-aec738684748-metrics-certs\" (UniqueName: \"kubernetes.io/secret/5240e2be-d3c2-45ef-8165-aec738684748-metrics-certs\") pod \"5240e2be-d3c2-45ef-8165-aec738684748\" (UID: \"5240e2be-d3c2-45ef-8165-aec738684748\") " Nov 03 01:38:25.991536 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:38:25.991508 4939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5240e2be-d3c2-45ef-8165-aec738684748-openshift-service-ca" pod "5240e2be-d3c2-45ef-8165-aec738684748" (UID: "5240e2be-d3c2-45ef-8165-aec738684748"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 01:38:25.992085 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:38:25.992052 4939 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5240e2be-d3c2-45ef-8165-aec738684748-openshift-service-ca\") on node \"ip-10-0-140-107.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:38:25.993368 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:38:25.993347 4939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5240e2be-d3c2-45ef-8165-aec738684748-kube-api-access-zl5rf" pod "5240e2be-d3c2-45ef-8165-aec738684748" (UID: "5240e2be-d3c2-45ef-8165-aec738684748"). InnerVolumeSpecName "kube-api-access-zl5rf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 01:38:25.993468 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:38:25.993371 4939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5240e2be-d3c2-45ef-8165-aec738684748-maas-api-tls" pod "5240e2be-d3c2-45ef-8165-aec738684748" (UID: "5240e2be-d3c2-45ef-8165-aec738684748"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 01:38:25.993596 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:38:25.993582 4939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5240e2be-d3c2-45ef-8165-aec738684748-metrics-certs" pod "5240e2be-d3c2-45ef-8165-aec738684748" (UID: "5240e2be-d3c2-45ef-8165-aec738684748"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 01:38:26.060621 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:38:26.060538 4939 generic.go:356] "Generic (PLEG): container finished" podID="5240e2be-d3c2-45ef-8165-aec738684748" containerID="b8846f80d56673691d0d98e1a01abf8a1bd755ec0ffe0b7f95cfa9d40f91724b" exitCode=0 Nov 03 01:38:26.060621 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:38:26.060585 4939 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-2f9a3a-7b485fd99-hjcmm" Nov 03 01:38:26.060816 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:38:26.060624 4939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-2f9a3a-7b485fd99-hjcmm" event={"ID":"5240e2be-d3c2-45ef-8165-aec738684748","Type":"ContainerDied","Data":"b8846f80d56673691d0d98e1a01abf8a1bd755ec0ffe0b7f95cfa9d40f91724b"} Nov 03 01:38:26.060816 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:38:26.060685 4939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-2f9a3a-7b485fd99-hjcmm" event={"ID":"5240e2be-d3c2-45ef-8165-aec738684748","Type":"ContainerDied","Data":"67aca44a2d6b34ef859f70bbe52c36c2344a99802ebba096023b894fdb3fd2d4"} Nov 03 01:38:26.060816 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:38:26.060713 4939 scope.go:122] "RemoveContainer" containerID="b8846f80d56673691d0d98e1a01abf8a1bd755ec0ffe0b7f95cfa9d40f91724b" Nov 03 01:38:26.073426 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:38:26.073408 4939 scope.go:122] "RemoveContainer" containerID="b8846f80d56673691d0d98e1a01abf8a1bd755ec0ffe0b7f95cfa9d40f91724b" Nov 03 01:38:26.073714 ip-10-0-140-107 kubenswrapper[4939]: E1103 01:38:26.073694 4939 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b8846f80d56673691d0d98e1a01abf8a1bd755ec0ffe0b7f95cfa9d40f91724b\": container with ID starting with b8846f80d56673691d0d98e1a01abf8a1bd755ec0ffe0b7f95cfa9d40f91724b not found: ID does not exist" containerID="b8846f80d56673691d0d98e1a01abf8a1bd755ec0ffe0b7f95cfa9d40f91724b" Nov 03 01:38:26.073764 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:38:26.073725 4939 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b8846f80d56673691d0d98e1a01abf8a1bd755ec0ffe0b7f95cfa9d40f91724b"} err="failed to get container status \"b8846f80d56673691d0d98e1a01abf8a1bd755ec0ffe0b7f95cfa9d40f91724b\": rpc error: code = NotFound desc = could not find container \"b8846f80d56673691d0d98e1a01abf8a1bd755ec0ffe0b7f95cfa9d40f91724b\": container with ID starting with b8846f80d56673691d0d98e1a01abf8a1bd755ec0ffe0b7f95cfa9d40f91724b not found: ID does not exist" Nov 03 01:38:26.087795 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:38:26.087758 4939 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w0-2f9a3a-7b485fd99-hjcmm"] Nov 03 01:38:26.091323 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:38:26.091303 4939 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w0-2f9a3a-7b485fd99-hjcmm"] Nov 03 01:38:26.093889 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:38:26.093857 4939 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zl5rf\" (UniqueName: \"kubernetes.io/projected/5240e2be-d3c2-45ef-8165-aec738684748-kube-api-access-zl5rf\") on node \"ip-10-0-140-107.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:38:26.093993 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:38:26.093893 4939 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5240e2be-d3c2-45ef-8165-aec738684748-maas-api-tls\") on node \"ip-10-0-140-107.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:38:26.093993 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:38:26.093904 4939 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5240e2be-d3c2-45ef-8165-aec738684748-metrics-certs\") on node \"ip-10-0-140-107.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:38:26.421507 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:38:26.421445 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:38:27.075023 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:38:27.074987 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:38:27.808564 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:38:27.808522 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:38:27.874312 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:38:27.874278 4939 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5240e2be-d3c2-45ef-8165-aec738684748" path="/var/lib/kubelet/pods/5240e2be-d3c2-45ef-8165-aec738684748/volumes" Nov 03 01:38:28.085210 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:38:28.085179 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:38:28.314919 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:38:28.314881 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:38:28.619581 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:38:28.619546 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:38:28.901540 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:38:28.901444 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:38:29.024935 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:38:29.016466 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:38:29.779779 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:38:29.779745 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:38:29.901759 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:38:29.901682 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:38:30.492649 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:38:30.492604 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:38:31.209072 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:38:31.209018 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:38:32.011868 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:38:32.011833 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:38:32.779170 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:38:32.779133 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:38:33.180014 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:38:33.179980 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:38:33.284164 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:38:33.284131 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:38:33.493623 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:38:33.493517 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:38:34.213898 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:38:34.213854 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:38:34.318028 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:38:34.317950 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:38:34.897674 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:38:34.897610 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:38:35.576302 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:38:35.576257 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:38:36.202876 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:38:36.202845 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:38:36.858538 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:38:36.858500 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:38:37.476249 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:38:37.476207 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:38:38.109288 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:38:38.109252 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:38:38.768664 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:38:38.768629 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:38:39.379952 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:38:39.379847 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:38:39.988373 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:38:39.988340 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:38:40.592204 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:38:40.592173 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:38:41.278183 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:38:41.278145 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:38:41.979111 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:38:41.979076 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:38:42.651451 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:38:42.651413 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:38:43.409500 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:38:43.409468 4939 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-4pw7v"] Nov 03 01:38:43.420350 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:38:43.420316 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:38:44.289255 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:38:44.289220 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:38:45.312933 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:38:45.299527 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:38:45.702816 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:38:45.702780 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:38:45.939176 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:38:45.939145 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:38:46.158484 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:38:46.158446 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:38:46.966814 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:38:46.966780 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:38:47.579620 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:38:47.579585 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:38:48.034064 ip-10-0-140-107 kubenswrapper[4939]: E1103 01:38:48.034023 4939 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-sthjh" containerName="extractor" Nov 03 01:38:48.207924 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:38:48.207882 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:38:48.984572 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:38:48.984535 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:38:49.691471 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:38:49.691441 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:38:50.278937 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:38:50.278891 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:38:50.959622 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:38:50.959585 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:38:51.605791 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:38:51.605755 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:38:52.282824 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:38:52.282787 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:38:52.878467 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:38:52.878384 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:38:53.470620 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:38:53.470584 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:38:54.106589 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:38:54.106558 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:38:54.811897 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:38:54.811851 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:38:55.592552 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:38:55.592514 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:38:56.201019 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:38:56.200986 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:38:56.866636 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:38:56.866597 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:38:57.491419 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:38:57.491387 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:38:58.085253 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:38:58.085214 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:38:58.697199 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:38:58.697167 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:38:59.386451 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:38:59.386410 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:38:59.986285 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:38:59.986254 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:39:00.749915 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:39:00.749856 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:39:01.389706 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:39:01.389673 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:39:02.068273 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:39:02.068235 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:39:02.693059 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:39:02.693026 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:39:03.306748 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:39:03.306713 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:39:03.957656 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:39:03.957584 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:39:04.687287 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:39:04.687238 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:39:05.287993 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:39:05.287963 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:39:05.898350 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:39:05.898317 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:39:06.564415 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:39:06.564382 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:39:07.178969 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:39:07.178930 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:39:07.896400 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:39:07.896349 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:39:08.494627 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:39:08.494596 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:39:09.118654 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:39:09.118619 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:39:09.850407 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:39:09.850365 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:39:10.466031 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:39:10.465997 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:39:11.099901 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:39:11.099864 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:39:11.805899 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:39:11.805864 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:39:12.523221 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:39:12.523186 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:39:13.459328 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:39:13.459290 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:39:14.103887 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:39:14.103847 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:39:14.780454 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:39:14.780424 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:39:15.380015 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:39:15.379982 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:39:16.009945 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:39:16.009880 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:39:16.764644 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:39:16.764611 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:39:17.356343 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:39:17.356310 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:39:18.062177 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:39:18.062140 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:39:18.075495 ip-10-0-140-107 kubenswrapper[4939]: E1103 01:39:18.075458 4939 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-sthjh" containerName="extractor" Nov 03 01:39:18.690276 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:39:18.690242 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:39:19.319835 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:39:19.319799 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:39:19.913750 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:39:19.913713 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:39:20.578343 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:39:20.578311 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:39:21.178786 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:39:21.178730 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:39:21.781867 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:39:21.781832 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:39:22.591254 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:39:22.591206 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:39:23.403178 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:39:23.403147 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:39:24.065662 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:39:24.065628 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:39:24.677442 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:39:24.677408 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:39:25.306862 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:39:25.306826 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:39:25.893780 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:39:25.893749 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:39:26.470466 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:39:26.470433 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:39:27.195142 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:39:27.195106 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:39:27.403788 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:39:27.403751 4939 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-4pw7v"] Nov 03 01:39:27.901299 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:39:27.901256 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:39:28.725312 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:39:28.725264 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:39:29.429154 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:39:29.428313 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:39:30.187938 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:39:30.187854 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:39:30.523719 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:39:30.523626 4939 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w2-531447/sub-test-model-531447-kserve-65cffbb7c-h6b59"] Nov 03 01:39:30.593358 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:39:30.593324 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:39:31.025216 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:39:31.025177 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:39:31.694950 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:39:31.694868 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:39:32.348900 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:39:32.348860 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:39:33.199967 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:39:33.199901 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:39:33.990756 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:39:33.990724 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:39:34.675353 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:39:34.675304 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:39:35.304769 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:39:35.304732 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:39:36.056759 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:39:36.056720 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:39:36.622319 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:39:36.622286 4939 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w2-ef434a/sub-test-model-ef434a-kserve-659ddb8595-zr7c2"] Nov 03 01:39:36.702902 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:39:36.702866 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:39:36.822402 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:39:36.822366 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:39:37.185743 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:39:37.185699 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:39:37.395363 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:39:37.395326 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:39:38.052569 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:39:38.052536 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:39:38.696886 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:39:38.696850 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:39:39.391978 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:39:39.391900 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:39:40.179188 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:39:40.179159 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:39:40.779669 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:39:40.779634 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:39:41.645923 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:39:41.645871 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:39:42.408997 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:39:42.408960 4939 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-027c19-596c6c9695-wm55p"] Nov 03 01:39:42.422653 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:39:42.422619 4939 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-027c19-67b548955-rltd2"] Nov 03 01:39:42.428112 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:39:42.428082 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:39:42.534855 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:39:42.534822 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:39:43.292043 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:39:43.292008 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:39:43.398503 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:39:43.398469 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:39:43.976186 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:39:43.976147 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:39:44.598187 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:39:44.598119 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:39:45.276418 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:39:45.276386 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:39:45.553941 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:39:45.553816 4939 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-w2-531447/sub-test-model-531447-kserve-65cffbb7c-h6b59" podUID="de222447-f690-4b85-bdee-b419f81767eb" containerName="main" containerID="cri-o://4e0a1360f81ad320881890154d63a557f4a5c96ef7afc03eb7d8c2141d13d420" gracePeriod=45 Nov 03 01:39:45.715783 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:39:45.715748 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:39:45.853923 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:39:45.853888 4939 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w2-531447/sub-test-model-531447-kserve-65cffbb7c-h6b59" Nov 03 01:39:45.884726 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:39:45.884692 4939 generic.go:356] "Generic (PLEG): container finished" podID="de222447-f690-4b85-bdee-b419f81767eb" containerID="4e0a1360f81ad320881890154d63a557f4a5c96ef7afc03eb7d8c2141d13d420" exitCode=0 Nov 03 01:39:45.885059 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:39:45.885033 4939 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w2-531447/sub-test-model-531447-kserve-65cffbb7c-h6b59" Nov 03 01:39:45.885330 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:39:45.885053 4939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w2-531447/sub-test-model-531447-kserve-65cffbb7c-h6b59" event={"ID":"de222447-f690-4b85-bdee-b419f81767eb","Type":"ContainerDied","Data":"4e0a1360f81ad320881890154d63a557f4a5c96ef7afc03eb7d8c2141d13d420"} Nov 03 01:39:45.885826 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:39:45.885356 4939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w2-531447/sub-test-model-531447-kserve-65cffbb7c-h6b59" event={"ID":"de222447-f690-4b85-bdee-b419f81767eb","Type":"ContainerDied","Data":"e2a6aeecc5071b6ef4960bb7c5eee1d850e7e9597b0a1e743bea8135d49833a6"} Nov 03 01:39:45.885826 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:39:45.885390 4939 scope.go:122] "RemoveContainer" containerID="4e0a1360f81ad320881890154d63a557f4a5c96ef7afc03eb7d8c2141d13d420" Nov 03 01:39:45.901161 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:39:45.901140 4939 scope.go:122] "RemoveContainer" containerID="4e0a1360f81ad320881890154d63a557f4a5c96ef7afc03eb7d8c2141d13d420" Nov 03 01:39:45.901528 ip-10-0-140-107 kubenswrapper[4939]: E1103 01:39:45.901496 4939 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4e0a1360f81ad320881890154d63a557f4a5c96ef7afc03eb7d8c2141d13d420\": container with ID starting with 4e0a1360f81ad320881890154d63a557f4a5c96ef7afc03eb7d8c2141d13d420 not found: ID does not exist" containerID="4e0a1360f81ad320881890154d63a557f4a5c96ef7afc03eb7d8c2141d13d420" Nov 03 01:39:45.901662 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:39:45.901538 4939 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4e0a1360f81ad320881890154d63a557f4a5c96ef7afc03eb7d8c2141d13d420"} err="failed to get container status \"4e0a1360f81ad320881890154d63a557f4a5c96ef7afc03eb7d8c2141d13d420\": rpc error: code = NotFound desc = could not find container \"4e0a1360f81ad320881890154d63a557f4a5c96ef7afc03eb7d8c2141d13d420\": container with ID starting with 4e0a1360f81ad320881890154d63a557f4a5c96ef7afc03eb7d8c2141d13d420 not found: ID does not exist" Nov 03 01:39:45.932292 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:39:45.932256 4939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/de222447-f690-4b85-bdee-b419f81767eb-tls-certs\" (UniqueName: \"kubernetes.io/secret/de222447-f690-4b85-bdee-b419f81767eb-tls-certs\") pod \"de222447-f690-4b85-bdee-b419f81767eb\" (UID: \"de222447-f690-4b85-bdee-b419f81767eb\") " Nov 03 01:39:45.932487 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:39:45.932443 4939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/de222447-f690-4b85-bdee-b419f81767eb-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/de222447-f690-4b85-bdee-b419f81767eb-tmp-dir\") pod \"de222447-f690-4b85-bdee-b419f81767eb\" (UID: \"de222447-f690-4b85-bdee-b419f81767eb\") " Nov 03 01:39:45.932576 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:39:45.932519 4939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/de222447-f690-4b85-bdee-b419f81767eb-kube-api-access-ssj4g\" (UniqueName: \"kubernetes.io/projected/de222447-f690-4b85-bdee-b419f81767eb-kube-api-access-ssj4g\") pod \"de222447-f690-4b85-bdee-b419f81767eb\" (UID: \"de222447-f690-4b85-bdee-b419f81767eb\") " Nov 03 01:39:45.932576 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:39:45.932548 4939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/de222447-f690-4b85-bdee-b419f81767eb-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/de222447-f690-4b85-bdee-b419f81767eb-model-cache\") pod \"de222447-f690-4b85-bdee-b419f81767eb\" (UID: \"de222447-f690-4b85-bdee-b419f81767eb\") " Nov 03 01:39:45.932818 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:39:45.932797 4939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/de222447-f690-4b85-bdee-b419f81767eb-dshm\" (UniqueName: \"kubernetes.io/empty-dir/de222447-f690-4b85-bdee-b419f81767eb-dshm\") pod \"de222447-f690-4b85-bdee-b419f81767eb\" (UID: \"de222447-f690-4b85-bdee-b419f81767eb\") " Nov 03 01:39:45.932897 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:39:45.932850 4939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/de222447-f690-4b85-bdee-b419f81767eb-tmp-dir" pod "de222447-f690-4b85-bdee-b419f81767eb" (UID: "de222447-f690-4b85-bdee-b419f81767eb"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 01:39:45.932897 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:39:45.932869 4939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/de222447-f690-4b85-bdee-b419f81767eb-home\" (UniqueName: \"kubernetes.io/empty-dir/de222447-f690-4b85-bdee-b419f81767eb-home\") pod \"de222447-f690-4b85-bdee-b419f81767eb\" (UID: \"de222447-f690-4b85-bdee-b419f81767eb\") " Nov 03 01:39:45.933148 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:39:45.933125 4939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/de222447-f690-4b85-bdee-b419f81767eb-home" pod "de222447-f690-4b85-bdee-b419f81767eb" (UID: "de222447-f690-4b85-bdee-b419f81767eb"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 01:39:45.933557 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:39:45.933532 4939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/de222447-f690-4b85-bdee-b419f81767eb-model-cache" pod "de222447-f690-4b85-bdee-b419f81767eb" (UID: "de222447-f690-4b85-bdee-b419f81767eb"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 01:39:45.933857 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:39:45.933836 4939 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/de222447-f690-4b85-bdee-b419f81767eb-home\") on node \"ip-10-0-140-107.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:39:45.934018 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:39:45.933860 4939 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/de222447-f690-4b85-bdee-b419f81767eb-tmp-dir\") on node \"ip-10-0-140-107.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:39:45.934018 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:39:45.933875 4939 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/de222447-f690-4b85-bdee-b419f81767eb-model-cache\") on node \"ip-10-0-140-107.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:39:45.935214 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:39:45.935188 4939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/de222447-f690-4b85-bdee-b419f81767eb-tls-certs" pod "de222447-f690-4b85-bdee-b419f81767eb" (UID: "de222447-f690-4b85-bdee-b419f81767eb"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 01:39:45.935830 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:39:45.935801 4939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/de222447-f690-4b85-bdee-b419f81767eb-kube-api-access-ssj4g" pod "de222447-f690-4b85-bdee-b419f81767eb" (UID: "de222447-f690-4b85-bdee-b419f81767eb"). InnerVolumeSpecName "kube-api-access-ssj4g". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 01:39:45.936254 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:39:45.936228 4939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/de222447-f690-4b85-bdee-b419f81767eb-dshm" pod "de222447-f690-4b85-bdee-b419f81767eb" (UID: "de222447-f690-4b85-bdee-b419f81767eb"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 01:39:46.000583 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:39:46.000516 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:39:46.042816 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:39:46.042724 4939 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/de222447-f690-4b85-bdee-b419f81767eb-dshm\") on node \"ip-10-0-140-107.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:39:46.042816 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:39:46.042815 4939 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/de222447-f690-4b85-bdee-b419f81767eb-tls-certs\") on node \"ip-10-0-140-107.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:39:46.043110 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:39:46.042868 4939 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ssj4g\" (UniqueName: \"kubernetes.io/projected/de222447-f690-4b85-bdee-b419f81767eb-kube-api-access-ssj4g\") on node \"ip-10-0-140-107.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:39:46.210706 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:39:46.210673 4939 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w2-531447/sub-test-model-531447-kserve-65cffbb7c-h6b59"] Nov 03 01:39:46.217268 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:39:46.217236 4939 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-w2-531447/sub-test-model-531447-kserve-65cffbb7c-h6b59"] Nov 03 01:39:47.142475 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:39:47.140440 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:39:47.292726 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:39:47.292683 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:39:47.483144 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:39:47.482990 4939 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-027c19-67b548955-rltd2" podUID="ab5ffea1-5d04-47c2-9d4d-d720550bc30f" containerName="payload-pre-processing" containerID="cri-o://5f1ab4affd67f9c19aeb9973a274a941d79ca3d8585bd1f61f141a9875346ae5" gracePeriod=25 Nov 03 01:39:47.496933 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:39:47.483540 4939 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-extmod-a-027c19-596c6c9695-wm55p" podUID="ce37c54e-196b-4861-8b0d-e2d4351eed24" containerName="payload-processing" containerID="cri-o://89db74dd8260ac3a9c06c43a81706ad1b6010460e1db156d2a53ccad4650368c" gracePeriod=25 Nov 03 01:39:47.503038 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:39:47.503006 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:39:47.896294 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:39:47.893721 4939 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="de222447-f690-4b85-bdee-b419f81767eb" path="/var/lib/kubelet/pods/de222447-f690-4b85-bdee-b419f81767eb/volumes" Nov 03 01:39:48.041170 ip-10-0-140-107 kubenswrapper[4939]: E1103 01:39:48.041126 4939 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-sthjh" containerName="extractor" Nov 03 01:39:48.261376 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:39:48.261301 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:39:48.978763 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:39:48.978728 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:39:49.586812 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:39:49.586780 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:39:49.774951 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:39:49.774351 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:39:49.917988 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:39:49.913333 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:39:50.250215 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:39:50.250139 4939 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7037-98d1a5"] Nov 03 01:39:50.251004 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:39:50.250984 4939 cpu_manager.go:414] "containerMap: removing container" podUID="207e2207-fb6c-4076-95df-72e95671c261" containerName="main" Nov 03 01:39:50.251083 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:39:50.251005 4939 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="207e2207-fb6c-4076-95df-72e95671c261" containerName="main" Nov 03 01:39:50.251083 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:39:50.251032 4939 cpu_manager.go:414] "containerMap: removing container" podUID="de222447-f690-4b85-bdee-b419f81767eb" containerName="main" Nov 03 01:39:50.251083 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:39:50.251037 4939 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="de222447-f690-4b85-bdee-b419f81767eb" containerName="main" Nov 03 01:39:50.251083 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:39:50.251049 4939 cpu_manager.go:414] "containerMap: removing container" podUID="5240e2be-d3c2-45ef-8165-aec738684748" containerName="maas-api" Nov 03 01:39:50.251083 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:39:50.251054 4939 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5240e2be-d3c2-45ef-8165-aec738684748" containerName="maas-api" Nov 03 01:39:50.251083 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:39:50.251071 4939 cpu_manager.go:414] "containerMap: removing container" podUID="ac0ece06-8718-463f-915f-5315988d9711" containerName="main" Nov 03 01:39:50.251083 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:39:50.251076 4939 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ac0ece06-8718-463f-915f-5315988d9711" containerName="main" Nov 03 01:39:50.251311 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:39:50.251166 4939 memory_manager.go:365] "RemoveStaleState removing state" podUID="de222447-f690-4b85-bdee-b419f81767eb" containerName="main" Nov 03 01:39:50.251311 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:39:50.251177 4939 memory_manager.go:365] "RemoveStaleState removing state" podUID="5240e2be-d3c2-45ef-8165-aec738684748" containerName="maas-api" Nov 03 01:39:50.251311 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:39:50.251188 4939 memory_manager.go:365] "RemoveStaleState removing state" podUID="ac0ece06-8718-463f-915f-5315988d9711" containerName="main" Nov 03 01:39:50.251311 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:39:50.251199 4939 memory_manager.go:365] "RemoveStaleState removing state" podUID="207e2207-fb6c-4076-95df-72e95671c261" containerName="main" Nov 03 01:39:50.256206 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:39:50.256189 4939 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7037-98d1a5" Nov 03 01:39:50.260279 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:39:50.260254 4939 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-7l8r6\"" Nov 03 01:39:50.261921 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:39:50.261882 4939 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7037-98d1a5"] Nov 03 01:39:50.346330 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:39:50.346288 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hbjnc\" (UniqueName: \"kubernetes.io/projected/9af5e272-6d22-4fcb-bc96-0574e84a6c4f-kube-api-access-hbjnc\") pod \"test-curl-7037-98d1a5\" (UID: \"9af5e272-6d22-4fcb-bc96-0574e84a6c4f\") " pod="opendatahub/test-curl-7037-98d1a5" Nov 03 01:39:50.448397 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:39:50.448354 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hbjnc\" (UniqueName: \"kubernetes.io/projected/9af5e272-6d22-4fcb-bc96-0574e84a6c4f-kube-api-access-hbjnc\") pod \"test-curl-7037-98d1a5\" (UID: \"9af5e272-6d22-4fcb-bc96-0574e84a6c4f\") " pod="opendatahub/test-curl-7037-98d1a5" Nov 03 01:39:50.456822 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:39:50.456797 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hbjnc\" (UniqueName: \"kubernetes.io/projected/9af5e272-6d22-4fcb-bc96-0574e84a6c4f-kube-api-access-hbjnc\") pod \"test-curl-7037-98d1a5\" (UID: \"9af5e272-6d22-4fcb-bc96-0574e84a6c4f\") " pod="opendatahub/test-curl-7037-98d1a5" Nov 03 01:39:50.572818 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:39:50.572740 4939 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7037-98d1a5" Nov 03 01:39:50.660141 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:39:50.659394 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:39:50.883341 ip-10-0-140-107 kubenswrapper[4939]: W1103 01:39:50.883312 4939 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod9af5e272_6d22_4fcb_bc96_0574e84a6c4f.slice/crio-cdcf51da1b1c3499113aa74d20ca4c54ede208a3f997136ced6197f458e6586d WatchSource:0}: Error finding container cdcf51da1b1c3499113aa74d20ca4c54ede208a3f997136ced6197f458e6586d: Status 404 returned error can't find the container with id cdcf51da1b1c3499113aa74d20ca4c54ede208a3f997136ced6197f458e6586d Nov 03 01:39:50.884458 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:39:50.884428 4939 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7037-98d1a5"] Nov 03 01:39:50.992699 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:39:50.992664 4939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7037-98d1a5" event={"ID":"9af5e272-6d22-4fcb-bc96-0574e84a6c4f","Type":"ContainerStarted","Data":"80ac084a5ce9bd3c0dc634a50e09491d8bbc3d87e14a454fac2500b8f84c7e1f"} Nov 03 01:39:50.992699 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:39:50.992705 4939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7037-98d1a5" event={"ID":"9af5e272-6d22-4fcb-bc96-0574e84a6c4f","Type":"ContainerStarted","Data":"cdcf51da1b1c3499113aa74d20ca4c54ede208a3f997136ced6197f458e6586d"} Nov 03 01:39:51.010384 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:39:51.010332 4939 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/test-curl-7037-98d1a5" podStartSLOduration=1.010317589 podStartE2EDuration="1.010317589s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 01:39:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 01:39:51.006697098 +0000 UTC m=+3539.271032046" watchObservedRunningTime="2026-11-03 01:39:51.010317589 +0000 UTC m=+3539.274652531" Nov 03 01:39:51.281829 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:39:51.281735 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:39:51.651387 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:39:51.651340 4939 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-w2-ef434a/sub-test-model-ef434a-kserve-659ddb8595-zr7c2" podUID="5f7f5188-d311-486b-bb78-583a436145c5" containerName="main" containerID="cri-o://bdd2ff5643812c0947899c193a7d22052a6cc38247cf5cbb3acc63b98c0907d1" gracePeriod=45 Nov 03 01:39:51.676155 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:39:51.676125 4939 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7037-98d1a5"] Nov 03 01:39:51.680264 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:39:51.680236 4939 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7037-98d1a5"] Nov 03 01:39:51.852475 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:39:51.846115 4939 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7037-1655dd"] Nov 03 01:39:51.852475 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:39:51.847375 4939 cpu_manager.go:414] "containerMap: removing container" podUID="9af5e272-6d22-4fcb-bc96-0574e84a6c4f" containerName="test-curl-7037-98d1a5" Nov 03 01:39:51.852475 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:39:51.847398 4939 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9af5e272-6d22-4fcb-bc96-0574e84a6c4f" containerName="test-curl-7037-98d1a5" Nov 03 01:39:51.852475 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:39:51.847586 4939 memory_manager.go:365] "RemoveStaleState removing state" podUID="9af5e272-6d22-4fcb-bc96-0574e84a6c4f" containerName="test-curl-7037-98d1a5" Nov 03 01:39:51.852475 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:39:51.852174 4939 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7037-1655dd" Nov 03 01:39:51.854302 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:39:51.853514 4939 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7037-1655dd"] Nov 03 01:39:51.883947 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:39:51.883891 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:39:51.990186 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:39:51.990148 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jqbg6\" (UniqueName: \"kubernetes.io/projected/12a62cc4-d79a-495f-80c5-a2449e754e1b-kube-api-access-jqbg6\") pod \"test-curl-7037-1655dd\" (UID: \"12a62cc4-d79a-495f-80c5-a2449e754e1b\") " pod="opendatahub/test-curl-7037-1655dd" Nov 03 01:39:52.003446 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:39:52.003416 4939 generic.go:356] "Generic (PLEG): container finished" podID="5f7f5188-d311-486b-bb78-583a436145c5" containerID="bdd2ff5643812c0947899c193a7d22052a6cc38247cf5cbb3acc63b98c0907d1" exitCode=0 Nov 03 01:39:52.003602 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:39:52.003485 4939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w2-ef434a/sub-test-model-ef434a-kserve-659ddb8595-zr7c2" event={"ID":"5f7f5188-d311-486b-bb78-583a436145c5","Type":"ContainerDied","Data":"bdd2ff5643812c0947899c193a7d22052a6cc38247cf5cbb3acc63b98c0907d1"} Nov 03 01:39:52.003602 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:39:52.003523 4939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w2-ef434a/sub-test-model-ef434a-kserve-659ddb8595-zr7c2" event={"ID":"5f7f5188-d311-486b-bb78-583a436145c5","Type":"ContainerDied","Data":"e5d69550164c1b4d6d0fe38a095ad984d71b1813445a98f3b0def53b22052fb0"} Nov 03 01:39:52.003602 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:39:52.003538 4939 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e5d69550164c1b4d6d0fe38a095ad984d71b1813445a98f3b0def53b22052fb0" Nov 03 01:39:52.006210 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:39:52.006189 4939 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w2-ef434a/sub-test-model-ef434a-kserve-659ddb8595-zr7c2" Nov 03 01:39:52.091990 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:39:52.091955 4939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5f7f5188-d311-486b-bb78-583a436145c5-home\" (UniqueName: \"kubernetes.io/empty-dir/5f7f5188-d311-486b-bb78-583a436145c5-home\") pod \"5f7f5188-d311-486b-bb78-583a436145c5\" (UID: \"5f7f5188-d311-486b-bb78-583a436145c5\") " Nov 03 01:39:52.092187 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:39:52.092042 4939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5f7f5188-d311-486b-bb78-583a436145c5-kube-api-access-4s57v\" (UniqueName: \"kubernetes.io/projected/5f7f5188-d311-486b-bb78-583a436145c5-kube-api-access-4s57v\") pod \"5f7f5188-d311-486b-bb78-583a436145c5\" (UID: \"5f7f5188-d311-486b-bb78-583a436145c5\") " Nov 03 01:39:52.092187 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:39:52.092114 4939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5f7f5188-d311-486b-bb78-583a436145c5-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5f7f5188-d311-486b-bb78-583a436145c5-tmp-dir\") pod \"5f7f5188-d311-486b-bb78-583a436145c5\" (UID: \"5f7f5188-d311-486b-bb78-583a436145c5\") " Nov 03 01:39:52.092339 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:39:52.092204 4939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5f7f5188-d311-486b-bb78-583a436145c5-tls-certs\" (UniqueName: \"kubernetes.io/secret/5f7f5188-d311-486b-bb78-583a436145c5-tls-certs\") pod \"5f7f5188-d311-486b-bb78-583a436145c5\" (UID: \"5f7f5188-d311-486b-bb78-583a436145c5\") " Nov 03 01:39:52.092339 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:39:52.092250 4939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5f7f5188-d311-486b-bb78-583a436145c5-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5f7f5188-d311-486b-bb78-583a436145c5-model-cache\") pod \"5f7f5188-d311-486b-bb78-583a436145c5\" (UID: \"5f7f5188-d311-486b-bb78-583a436145c5\") " Nov 03 01:39:52.092339 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:39:52.092271 4939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5f7f5188-d311-486b-bb78-583a436145c5-home" pod "5f7f5188-d311-486b-bb78-583a436145c5" (UID: "5f7f5188-d311-486b-bb78-583a436145c5"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 01:39:52.092339 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:39:52.092309 4939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5f7f5188-d311-486b-bb78-583a436145c5-tmp-dir" pod "5f7f5188-d311-486b-bb78-583a436145c5" (UID: "5f7f5188-d311-486b-bb78-583a436145c5"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 01:39:52.092556 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:39:52.092419 4939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5f7f5188-d311-486b-bb78-583a436145c5-dshm\" (UniqueName: \"kubernetes.io/empty-dir/5f7f5188-d311-486b-bb78-583a436145c5-dshm\") pod \"5f7f5188-d311-486b-bb78-583a436145c5\" (UID: \"5f7f5188-d311-486b-bb78-583a436145c5\") " Nov 03 01:39:52.092608 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:39:52.092577 4939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5f7f5188-d311-486b-bb78-583a436145c5-model-cache" pod "5f7f5188-d311-486b-bb78-583a436145c5" (UID: "5f7f5188-d311-486b-bb78-583a436145c5"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 01:39:52.092782 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:39:52.092760 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jqbg6\" (UniqueName: \"kubernetes.io/projected/12a62cc4-d79a-495f-80c5-a2449e754e1b-kube-api-access-jqbg6\") pod \"test-curl-7037-1655dd\" (UID: \"12a62cc4-d79a-495f-80c5-a2449e754e1b\") " pod="opendatahub/test-curl-7037-1655dd" Nov 03 01:39:52.093137 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:39:52.093118 4939 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5f7f5188-d311-486b-bb78-583a436145c5-model-cache\") on node \"ip-10-0-140-107.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:39:52.093137 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:39:52.093134 4939 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5f7f5188-d311-486b-bb78-583a436145c5-home\") on node \"ip-10-0-140-107.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:39:52.093322 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:39:52.093144 4939 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5f7f5188-d311-486b-bb78-583a436145c5-tmp-dir\") on node \"ip-10-0-140-107.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:39:52.094604 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:39:52.094579 4939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5f7f5188-d311-486b-bb78-583a436145c5-tls-certs" pod "5f7f5188-d311-486b-bb78-583a436145c5" (UID: "5f7f5188-d311-486b-bb78-583a436145c5"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 01:39:52.094696 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:39:52.094644 4939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5f7f5188-d311-486b-bb78-583a436145c5-dshm" pod "5f7f5188-d311-486b-bb78-583a436145c5" (UID: "5f7f5188-d311-486b-bb78-583a436145c5"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 01:39:52.094696 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:39:52.094668 4939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5f7f5188-d311-486b-bb78-583a436145c5-kube-api-access-4s57v" pod "5f7f5188-d311-486b-bb78-583a436145c5" (UID: "5f7f5188-d311-486b-bb78-583a436145c5"). InnerVolumeSpecName "kube-api-access-4s57v". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 01:39:52.100611 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:39:52.100588 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jqbg6\" (UniqueName: \"kubernetes.io/projected/12a62cc4-d79a-495f-80c5-a2449e754e1b-kube-api-access-jqbg6\") pod \"test-curl-7037-1655dd\" (UID: \"12a62cc4-d79a-495f-80c5-a2449e754e1b\") " pod="opendatahub/test-curl-7037-1655dd" Nov 03 01:39:52.181602 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:39:52.181567 4939 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7037-1655dd" Nov 03 01:39:52.194445 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:39:52.194421 4939 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5f7f5188-d311-486b-bb78-583a436145c5-tls-certs\") on node \"ip-10-0-140-107.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:39:52.194525 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:39:52.194448 4939 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5f7f5188-d311-486b-bb78-583a436145c5-dshm\") on node \"ip-10-0-140-107.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:39:52.194525 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:39:52.194458 4939 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4s57v\" (UniqueName: \"kubernetes.io/projected/5f7f5188-d311-486b-bb78-583a436145c5-kube-api-access-4s57v\") on node \"ip-10-0-140-107.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:39:52.459679 ip-10-0-140-107 kubenswrapper[4939]: W1103 01:39:52.459651 4939 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod12a62cc4_d79a_495f_80c5_a2449e754e1b.slice/crio-2ebd061fe79f08eb4dda3735aa499ef97ff5cd7e6a93d000406248616123b298 WatchSource:0}: Error finding container 2ebd061fe79f08eb4dda3735aa499ef97ff5cd7e6a93d000406248616123b298: Status 404 returned error can't find the container with id 2ebd061fe79f08eb4dda3735aa499ef97ff5cd7e6a93d000406248616123b298 Nov 03 01:39:52.459783 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:39:52.459702 4939 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7037-1655dd"] Nov 03 01:39:52.554439 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:39:52.553885 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:39:53.012354 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:39:53.012299 4939 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w2-ef434a/sub-test-model-ef434a-kserve-659ddb8595-zr7c2" Nov 03 01:39:53.012354 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:39:53.012312 4939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7037-1655dd" event={"ID":"12a62cc4-d79a-495f-80c5-a2449e754e1b","Type":"ContainerStarted","Data":"c27560451a6fae82bd80da730c54590b4f81ff85b6f5ba7b4d25303fd92c3471"} Nov 03 01:39:53.012963 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:39:53.012361 4939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7037-1655dd" event={"ID":"12a62cc4-d79a-495f-80c5-a2449e754e1b","Type":"ContainerStarted","Data":"2ebd061fe79f08eb4dda3735aa499ef97ff5cd7e6a93d000406248616123b298"} Nov 03 01:39:53.012963 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:39:53.012728 4939 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/test-curl-7037-98d1a5" podUID="9af5e272-6d22-4fcb-bc96-0574e84a6c4f" containerName="test-curl-7037-98d1a5" containerID="cri-o://80ac084a5ce9bd3c0dc634a50e09491d8bbc3d87e14a454fac2500b8f84c7e1f" gracePeriod=2 Nov 03 01:39:53.032449 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:39:53.032407 4939 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/test-curl-7037-1655dd" podStartSLOduration=2.032396602 podStartE2EDuration="2.032396602s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 01:39:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 01:39:53.031148137 +0000 UTC m=+3541.295483084" watchObservedRunningTime="2026-11-03 01:39:53.032396602 +0000 UTC m=+3541.296731543" Nov 03 01:39:53.045733 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:39:53.045710 4939 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w2-ef434a/sub-test-model-ef434a-kserve-659ddb8595-zr7c2"] Nov 03 01:39:53.050779 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:39:53.050758 4939 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-w2-ef434a/sub-test-model-ef434a-kserve-659ddb8595-zr7c2"] Nov 03 01:39:53.172226 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:39:53.172196 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:39:53.686077 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:39:53.686046 4939 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7037-1655dd"] Nov 03 01:39:53.691294 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:39:53.691262 4939 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7037-1655dd"] Nov 03 01:39:53.835602 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:39:53.835159 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:39:53.871400 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:39:53.871357 4939 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5f7f5188-d311-486b-bb78-583a436145c5" path="/var/lib/kubelet/pods/5f7f5188-d311-486b-bb78-583a436145c5/volumes" Nov 03 01:39:54.378636 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:39:54.378604 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:39:55.000574 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:39:55.000533 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:39:55.071791 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:39:55.071176 4939 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/test-curl-7037-1655dd" podUID="12a62cc4-d79a-495f-80c5-a2449e754e1b" containerName="test-curl-7037-1655dd" containerID="cri-o://c27560451a6fae82bd80da730c54590b4f81ff85b6f5ba7b4d25303fd92c3471" gracePeriod=2 Nov 03 01:39:55.260312 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:39:55.260240 4939 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7037-98d1a5" Nov 03 01:39:55.360819 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:39:55.360789 4939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9af5e272-6d22-4fcb-bc96-0574e84a6c4f-kube-api-access-hbjnc\" (UniqueName: \"kubernetes.io/projected/9af5e272-6d22-4fcb-bc96-0574e84a6c4f-kube-api-access-hbjnc\") pod \"9af5e272-6d22-4fcb-bc96-0574e84a6c4f\" (UID: \"9af5e272-6d22-4fcb-bc96-0574e84a6c4f\") " Nov 03 01:39:55.363101 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:39:55.363072 4939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9af5e272-6d22-4fcb-bc96-0574e84a6c4f-kube-api-access-hbjnc" pod "9af5e272-6d22-4fcb-bc96-0574e84a6c4f" (UID: "9af5e272-6d22-4fcb-bc96-0574e84a6c4f"). InnerVolumeSpecName "kube-api-access-hbjnc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 01:39:55.462519 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:39:55.462488 4939 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hbjnc\" (UniqueName: \"kubernetes.io/projected/9af5e272-6d22-4fcb-bc96-0574e84a6c4f-kube-api-access-hbjnc\") on node \"ip-10-0-140-107.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:39:55.662509 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:39:55.662464 4939 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-027c19-67b548955-rltd2" podUID="ab5ffea1-5d04-47c2-9d4d-d720550bc30f" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 03 01:39:55.662696 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:39:55.662465 4939 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-extmod-a-027c19-596c6c9695-wm55p" podUID="ce37c54e-196b-4861-8b0d-e2d4351eed24" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 03 01:39:55.676848 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:39:55.676820 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:39:55.870310 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:39:55.870272 4939 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9af5e272-6d22-4fcb-bc96-0574e84a6c4f" path="/var/lib/kubelet/pods/9af5e272-6d22-4fcb-bc96-0574e84a6c4f/volumes" Nov 03 01:39:55.880978 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:39:55.880951 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:39:56.007991 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:39:56.007900 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:39:56.079183 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:39:56.079149 4939 generic.go:356] "Generic (PLEG): container finished" podID="9af5e272-6d22-4fcb-bc96-0574e84a6c4f" containerID="80ac084a5ce9bd3c0dc634a50e09491d8bbc3d87e14a454fac2500b8f84c7e1f" exitCode=137 Nov 03 01:39:56.079393 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:39:56.079192 4939 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7037-98d1a5" Nov 03 01:39:56.079393 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:39:56.079250 4939 scope.go:122] "RemoveContainer" containerID="80ac084a5ce9bd3c0dc634a50e09491d8bbc3d87e14a454fac2500b8f84c7e1f" Nov 03 01:39:56.092471 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:39:56.092446 4939 scope.go:122] "RemoveContainer" containerID="80ac084a5ce9bd3c0dc634a50e09491d8bbc3d87e14a454fac2500b8f84c7e1f" Nov 03 01:39:56.092811 ip-10-0-140-107 kubenswrapper[4939]: E1103 01:39:56.092781 4939 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"80ac084a5ce9bd3c0dc634a50e09491d8bbc3d87e14a454fac2500b8f84c7e1f\": container with ID starting with 80ac084a5ce9bd3c0dc634a50e09491d8bbc3d87e14a454fac2500b8f84c7e1f not found: ID does not exist" containerID="80ac084a5ce9bd3c0dc634a50e09491d8bbc3d87e14a454fac2500b8f84c7e1f" Nov 03 01:39:56.092895 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:39:56.092823 4939 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"80ac084a5ce9bd3c0dc634a50e09491d8bbc3d87e14a454fac2500b8f84c7e1f"} err="failed to get container status \"80ac084a5ce9bd3c0dc634a50e09491d8bbc3d87e14a454fac2500b8f84c7e1f\": rpc error: code = NotFound desc = could not find container \"80ac084a5ce9bd3c0dc634a50e09491d8bbc3d87e14a454fac2500b8f84c7e1f\": container with ID starting with 80ac084a5ce9bd3c0dc634a50e09491d8bbc3d87e14a454fac2500b8f84c7e1f not found: ID does not exist" Nov 03 01:39:56.490631 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:39:56.490593 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:39:56.612777 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:39:56.612734 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:39:56.764980 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:39:56.762896 4939 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-a-027c19-openshift-default-58d5bc7645-zjnf8"] Nov 03 01:39:56.764980 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:39:56.763157 4939 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-extmod-a-027c19-openshift-default-58d5bc7645-zjnf8" podUID="f09e7c2c-246e-4c96-a8ca-147e8ca83560" containerName="istio-proxy" containerID="cri-o://42a909aaa61a1c703977779fe62c035ddf2c41e0d466ec8d14c4edffdf0f4ee5" gracePeriod=30 Nov 03 01:39:56.820652 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:39:56.820625 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:39:57.283497 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:39:57.283467 4939 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7037-1655dd" Nov 03 01:39:57.378753 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:39:57.378719 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:39:57.422932 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:39:57.422477 4939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/12a62cc4-d79a-495f-80c5-a2449e754e1b-kube-api-access-jqbg6\" (UniqueName: \"kubernetes.io/projected/12a62cc4-d79a-495f-80c5-a2449e754e1b-kube-api-access-jqbg6\") pod \"12a62cc4-d79a-495f-80c5-a2449e754e1b\" (UID: \"12a62cc4-d79a-495f-80c5-a2449e754e1b\") " Nov 03 01:39:57.433339 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:39:57.433301 4939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/12a62cc4-d79a-495f-80c5-a2449e754e1b-kube-api-access-jqbg6" pod "12a62cc4-d79a-495f-80c5-a2449e754e1b" (UID: "12a62cc4-d79a-495f-80c5-a2449e754e1b"). InnerVolumeSpecName "kube-api-access-jqbg6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 01:39:57.525087 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:39:57.525053 4939 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jqbg6\" (UniqueName: \"kubernetes.io/projected/12a62cc4-d79a-495f-80c5-a2449e754e1b-kube-api-access-jqbg6\") on node \"ip-10-0-140-107.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:39:57.543723 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:39:57.543690 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:39:57.869865 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:39:57.869832 4939 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="12a62cc4-d79a-495f-80c5-a2449e754e1b" path="/var/lib/kubelet/pods/12a62cc4-d79a-495f-80c5-a2449e754e1b/volumes" Nov 03 01:39:57.981585 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:39:57.981548 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:39:58.095825 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:39:58.095781 4939 generic.go:356] "Generic (PLEG): container finished" podID="12a62cc4-d79a-495f-80c5-a2449e754e1b" containerID="c27560451a6fae82bd80da730c54590b4f81ff85b6f5ba7b4d25303fd92c3471" exitCode=137 Nov 03 01:39:58.095825 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:39:58.095824 4939 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7037-1655dd" Nov 03 01:39:58.096111 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:39:58.095877 4939 scope.go:122] "RemoveContainer" containerID="c27560451a6fae82bd80da730c54590b4f81ff85b6f5ba7b4d25303fd92c3471" Nov 03 01:39:58.105835 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:39:58.105815 4939 scope.go:122] "RemoveContainer" containerID="c27560451a6fae82bd80da730c54590b4f81ff85b6f5ba7b4d25303fd92c3471" Nov 03 01:39:58.106133 ip-10-0-140-107 kubenswrapper[4939]: E1103 01:39:58.106112 4939 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c27560451a6fae82bd80da730c54590b4f81ff85b6f5ba7b4d25303fd92c3471\": container with ID starting with c27560451a6fae82bd80da730c54590b4f81ff85b6f5ba7b4d25303fd92c3471 not found: ID does not exist" containerID="c27560451a6fae82bd80da730c54590b4f81ff85b6f5ba7b4d25303fd92c3471" Nov 03 01:39:58.106212 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:39:58.106144 4939 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c27560451a6fae82bd80da730c54590b4f81ff85b6f5ba7b4d25303fd92c3471"} err="failed to get container status \"c27560451a6fae82bd80da730c54590b4f81ff85b6f5ba7b4d25303fd92c3471\": rpc error: code = NotFound desc = could not find container \"c27560451a6fae82bd80da730c54590b4f81ff85b6f5ba7b4d25303fd92c3471\": container with ID starting with c27560451a6fae82bd80da730c54590b4f81ff85b6f5ba7b4d25303fd92c3471 not found: ID does not exist" Nov 03 01:39:58.571814 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:39:58.571777 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:39:59.255762 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:39:59.255730 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:39:59.895839 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:39:59.895796 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:40:00.008732 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:40:00.008699 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:40:00.129436 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:40:00.129404 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:40:00.588619 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:40:00.588580 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:40:01.269592 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:40:01.269559 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:40:01.905252 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:40:01.905215 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:40:02.109516 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:40:02.109492 4939 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-027c19-openshift-default-58d5bc7645-zjnf8" Nov 03 01:40:02.129007 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:40:02.128977 4939 generic.go:356] "Generic (PLEG): container finished" podID="f09e7c2c-246e-4c96-a8ca-147e8ca83560" containerID="42a909aaa61a1c703977779fe62c035ddf2c41e0d466ec8d14c4edffdf0f4ee5" exitCode=0 Nov 03 01:40:02.129146 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:40:02.129017 4939 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-027c19-openshift-default-58d5bc7645-zjnf8" Nov 03 01:40:02.129146 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:40:02.129061 4939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-027c19-openshift-default-58d5bc7645-zjnf8" event={"ID":"f09e7c2c-246e-4c96-a8ca-147e8ca83560","Type":"ContainerDied","Data":"42a909aaa61a1c703977779fe62c035ddf2c41e0d466ec8d14c4edffdf0f4ee5"} Nov 03 01:40:02.129146 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:40:02.129117 4939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-027c19-openshift-default-58d5bc7645-zjnf8" event={"ID":"f09e7c2c-246e-4c96-a8ca-147e8ca83560","Type":"ContainerDied","Data":"1e215616e471783f85fa757cd1a5a4a3db0d8141baf07cd643ddced3a53ddd2f"} Nov 03 01:40:02.129275 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:40:02.129145 4939 scope.go:122] "RemoveContainer" containerID="42a909aaa61a1c703977779fe62c035ddf2c41e0d466ec8d14c4edffdf0f4ee5" Nov 03 01:40:02.142575 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:40:02.142555 4939 scope.go:122] "RemoveContainer" containerID="a547b507af135685227fd7790e98b40a7d9718e0f67e158871a6c8acb5162fcc" Nov 03 01:40:02.156850 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:40:02.156823 4939 scope.go:122] "RemoveContainer" containerID="42a909aaa61a1c703977779fe62c035ddf2c41e0d466ec8d14c4edffdf0f4ee5" Nov 03 01:40:02.157223 ip-10-0-140-107 kubenswrapper[4939]: E1103 01:40:02.157197 4939 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"42a909aaa61a1c703977779fe62c035ddf2c41e0d466ec8d14c4edffdf0f4ee5\": container with ID starting with 42a909aaa61a1c703977779fe62c035ddf2c41e0d466ec8d14c4edffdf0f4ee5 not found: ID does not exist" containerID="42a909aaa61a1c703977779fe62c035ddf2c41e0d466ec8d14c4edffdf0f4ee5" Nov 03 01:40:02.157351 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:40:02.157230 4939 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"42a909aaa61a1c703977779fe62c035ddf2c41e0d466ec8d14c4edffdf0f4ee5"} err="failed to get container status \"42a909aaa61a1c703977779fe62c035ddf2c41e0d466ec8d14c4edffdf0f4ee5\": rpc error: code = NotFound desc = could not find container \"42a909aaa61a1c703977779fe62c035ddf2c41e0d466ec8d14c4edffdf0f4ee5\": container with ID starting with 42a909aaa61a1c703977779fe62c035ddf2c41e0d466ec8d14c4edffdf0f4ee5 not found: ID does not exist" Nov 03 01:40:02.157351 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:40:02.157250 4939 scope.go:122] "RemoveContainer" containerID="a547b507af135685227fd7790e98b40a7d9718e0f67e158871a6c8acb5162fcc" Nov 03 01:40:02.157601 ip-10-0-140-107 kubenswrapper[4939]: E1103 01:40:02.157577 4939 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a547b507af135685227fd7790e98b40a7d9718e0f67e158871a6c8acb5162fcc\": container with ID starting with a547b507af135685227fd7790e98b40a7d9718e0f67e158871a6c8acb5162fcc not found: ID does not exist" containerID="a547b507af135685227fd7790e98b40a7d9718e0f67e158871a6c8acb5162fcc" Nov 03 01:40:02.157672 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:40:02.157605 4939 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a547b507af135685227fd7790e98b40a7d9718e0f67e158871a6c8acb5162fcc"} err="failed to get container status \"a547b507af135685227fd7790e98b40a7d9718e0f67e158871a6c8acb5162fcc\": rpc error: code = NotFound desc = could not find container \"a547b507af135685227fd7790e98b40a7d9718e0f67e158871a6c8acb5162fcc\": container with ID starting with a547b507af135685227fd7790e98b40a7d9718e0f67e158871a6c8acb5162fcc not found: ID does not exist" Nov 03 01:40:02.201157 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:40:02.201128 4939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f09e7c2c-246e-4c96-a8ca-147e8ca83560-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f09e7c2c-246e-4c96-a8ca-147e8ca83560-workload-certs\") pod \"f09e7c2c-246e-4c96-a8ca-147e8ca83560\" (UID: \"f09e7c2c-246e-4c96-a8ca-147e8ca83560\") " Nov 03 01:40:02.201297 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:40:02.201192 4939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f09e7c2c-246e-4c96-a8ca-147e8ca83560-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f09e7c2c-246e-4c96-a8ca-147e8ca83560-workload-socket\") pod \"f09e7c2c-246e-4c96-a8ca-147e8ca83560\" (UID: \"f09e7c2c-246e-4c96-a8ca-147e8ca83560\") " Nov 03 01:40:02.201297 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:40:02.201242 4939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f09e7c2c-246e-4c96-a8ca-147e8ca83560-istio-token\" (UniqueName: \"kubernetes.io/projected/f09e7c2c-246e-4c96-a8ca-147e8ca83560-istio-token\") pod \"f09e7c2c-246e-4c96-a8ca-147e8ca83560\" (UID: \"f09e7c2c-246e-4c96-a8ca-147e8ca83560\") " Nov 03 01:40:02.201383 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:40:02.201325 4939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/f09e7c2c-246e-4c96-a8ca-147e8ca83560-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f09e7c2c-246e-4c96-a8ca-147e8ca83560-istio-podinfo\") pod \"f09e7c2c-246e-4c96-a8ca-147e8ca83560\" (UID: \"f09e7c2c-246e-4c96-a8ca-147e8ca83560\") " Nov 03 01:40:02.201383 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:40:02.201359 4939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f09e7c2c-246e-4c96-a8ca-147e8ca83560-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f09e7c2c-246e-4c96-a8ca-147e8ca83560-istiod-ca-cert\") pod \"f09e7c2c-246e-4c96-a8ca-147e8ca83560\" (UID: \"f09e7c2c-246e-4c96-a8ca-147e8ca83560\") " Nov 03 01:40:02.201484 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:40:02.201395 4939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f09e7c2c-246e-4c96-a8ca-147e8ca83560-workload-certs" pod "f09e7c2c-246e-4c96-a8ca-147e8ca83560" (UID: "f09e7c2c-246e-4c96-a8ca-147e8ca83560"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 01:40:02.201553 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:40:02.201476 4939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f09e7c2c-246e-4c96-a8ca-147e8ca83560-workload-socket" pod "f09e7c2c-246e-4c96-a8ca-147e8ca83560" (UID: "f09e7c2c-246e-4c96-a8ca-147e8ca83560"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 01:40:02.201553 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:40:02.201488 4939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f09e7c2c-246e-4c96-a8ca-147e8ca83560-kube-api-access-dtzkd\" (UniqueName: \"kubernetes.io/projected/f09e7c2c-246e-4c96-a8ca-147e8ca83560-kube-api-access-dtzkd\") pod \"f09e7c2c-246e-4c96-a8ca-147e8ca83560\" (UID: \"f09e7c2c-246e-4c96-a8ca-147e8ca83560\") " Nov 03 01:40:02.201553 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:40:02.201545 4939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f09e7c2c-246e-4c96-a8ca-147e8ca83560-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f09e7c2c-246e-4c96-a8ca-147e8ca83560-credential-socket\") pod \"f09e7c2c-246e-4c96-a8ca-147e8ca83560\" (UID: \"f09e7c2c-246e-4c96-a8ca-147e8ca83560\") " Nov 03 01:40:02.201732 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:40:02.201610 4939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f09e7c2c-246e-4c96-a8ca-147e8ca83560-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f09e7c2c-246e-4c96-a8ca-147e8ca83560-istio-envoy\") pod \"f09e7c2c-246e-4c96-a8ca-147e8ca83560\" (UID: \"f09e7c2c-246e-4c96-a8ca-147e8ca83560\") " Nov 03 01:40:02.201732 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:40:02.201634 4939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f09e7c2c-246e-4c96-a8ca-147e8ca83560-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f09e7c2c-246e-4c96-a8ca-147e8ca83560-istio-data\") pod \"f09e7c2c-246e-4c96-a8ca-147e8ca83560\" (UID: \"f09e7c2c-246e-4c96-a8ca-147e8ca83560\") " Nov 03 01:40:02.201948 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:40:02.201923 4939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f09e7c2c-246e-4c96-a8ca-147e8ca83560-istiod-ca-cert" pod "f09e7c2c-246e-4c96-a8ca-147e8ca83560" (UID: "f09e7c2c-246e-4c96-a8ca-147e8ca83560"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 01:40:02.202168 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:40:02.202143 4939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f09e7c2c-246e-4c96-a8ca-147e8ca83560-istio-data" pod "f09e7c2c-246e-4c96-a8ca-147e8ca83560" (UID: "f09e7c2c-246e-4c96-a8ca-147e8ca83560"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 01:40:02.202330 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:40:02.201940 4939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f09e7c2c-246e-4c96-a8ca-147e8ca83560-credential-socket" pod "f09e7c2c-246e-4c96-a8ca-147e8ca83560" (UID: "f09e7c2c-246e-4c96-a8ca-147e8ca83560"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 01:40:02.202330 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:40:02.202266 4939 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f09e7c2c-246e-4c96-a8ca-147e8ca83560-istiod-ca-cert\") on node \"ip-10-0-140-107.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:40:02.202487 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:40:02.202350 4939 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f09e7c2c-246e-4c96-a8ca-147e8ca83560-workload-certs\") on node \"ip-10-0-140-107.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:40:02.202487 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:40:02.202370 4939 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f09e7c2c-246e-4c96-a8ca-147e8ca83560-workload-socket\") on node \"ip-10-0-140-107.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:40:02.203652 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:40:02.203622 4939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/f09e7c2c-246e-4c96-a8ca-147e8ca83560-istio-podinfo" pod "f09e7c2c-246e-4c96-a8ca-147e8ca83560" (UID: "f09e7c2c-246e-4c96-a8ca-147e8ca83560"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 01:40:02.203652 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:40:02.203646 4939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f09e7c2c-246e-4c96-a8ca-147e8ca83560-istio-token" pod "f09e7c2c-246e-4c96-a8ca-147e8ca83560" (UID: "f09e7c2c-246e-4c96-a8ca-147e8ca83560"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 01:40:02.203774 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:40:02.203680 4939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f09e7c2c-246e-4c96-a8ca-147e8ca83560-kube-api-access-dtzkd" pod "f09e7c2c-246e-4c96-a8ca-147e8ca83560" (UID: "f09e7c2c-246e-4c96-a8ca-147e8ca83560"). InnerVolumeSpecName "kube-api-access-dtzkd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 01:40:02.204224 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:40:02.204204 4939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f09e7c2c-246e-4c96-a8ca-147e8ca83560-istio-envoy" pod "f09e7c2c-246e-4c96-a8ca-147e8ca83560" (UID: "f09e7c2c-246e-4c96-a8ca-147e8ca83560"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 01:40:02.303768 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:40:02.303729 4939 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f09e7c2c-246e-4c96-a8ca-147e8ca83560-credential-socket\") on node \"ip-10-0-140-107.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:40:02.303768 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:40:02.303764 4939 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f09e7c2c-246e-4c96-a8ca-147e8ca83560-istio-envoy\") on node \"ip-10-0-140-107.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:40:02.304256 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:40:02.303776 4939 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f09e7c2c-246e-4c96-a8ca-147e8ca83560-istio-data\") on node \"ip-10-0-140-107.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:40:02.304256 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:40:02.303790 4939 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f09e7c2c-246e-4c96-a8ca-147e8ca83560-istio-token\") on node \"ip-10-0-140-107.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:40:02.304256 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:40:02.303802 4939 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f09e7c2c-246e-4c96-a8ca-147e8ca83560-istio-podinfo\") on node \"ip-10-0-140-107.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:40:02.304256 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:40:02.303817 4939 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dtzkd\" (UniqueName: \"kubernetes.io/projected/f09e7c2c-246e-4c96-a8ca-147e8ca83560-kube-api-access-dtzkd\") on node \"ip-10-0-140-107.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:40:02.457014 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:40:02.456977 4939 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-a-027c19-openshift-default-58d5bc7645-zjnf8"] Nov 03 01:40:02.464850 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:40:02.464823 4939 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-extmod-a-027c19-openshift-default-58d5bc7645-zjnf8"] Nov 03 01:40:02.586488 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:40:02.586454 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:40:03.125366 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:40:03.125138 4939 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-b-3c5983-7fbf586cf5-92rtm"] Nov 03 01:40:03.139415 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:40:03.139383 4939 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-3c5983-564977cd99-qs5fk"] Nov 03 01:40:03.227479 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:40:03.227446 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:40:03.344193 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:40:03.344156 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:40:03.475079 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:40:03.474976 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:40:03.882771 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:40:03.882734 4939 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f09e7c2c-246e-4c96-a8ca-147e8ca83560" path="/var/lib/kubelet/pods/f09e7c2c-246e-4c96-a8ca-147e8ca83560/volumes" Nov 03 01:40:04.077370 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:40:04.077336 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:40:04.183862 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:40:04.183765 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:40:04.853930 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:40:04.841375 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:40:05.441043 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:40:05.441008 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:40:05.662027 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:40:05.661987 4939 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-extmod-a-027c19-596c6c9695-wm55p" podUID="ce37c54e-196b-4861-8b0d-e2d4351eed24" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 03 01:40:05.662228 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:40:05.661988 4939 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-027c19-67b548955-rltd2" podUID="ab5ffea1-5d04-47c2-9d4d-d720550bc30f" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 03 01:40:06.104595 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:40:06.104555 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:40:06.887533 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:40:06.887499 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:40:07.472635 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:40:07.472586 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:40:07.764748 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:40:07.764715 4939 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-a-027c19-596c6c9695-wm55p" Nov 03 01:40:07.863968 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:40:07.863941 4939 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-027c19-67b548955-rltd2" Nov 03 01:40:07.937080 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:40:07.937041 4939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ce37c54e-196b-4861-8b0d-e2d4351eed24-kube-api-access-cckw9\" (UniqueName: \"kubernetes.io/projected/ce37c54e-196b-4861-8b0d-e2d4351eed24-kube-api-access-cckw9\") pod \"ce37c54e-196b-4861-8b0d-e2d4351eed24\" (UID: \"ce37c54e-196b-4861-8b0d-e2d4351eed24\") " Nov 03 01:40:07.937451 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:40:07.937428 4939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ce37c54e-196b-4861-8b0d-e2d4351eed24-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ce37c54e-196b-4861-8b0d-e2d4351eed24-plugins-config-volume\") pod \"ce37c54e-196b-4861-8b0d-e2d4351eed24\" (UID: \"ce37c54e-196b-4861-8b0d-e2d4351eed24\") " Nov 03 01:40:07.938137 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:40:07.938106 4939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ce37c54e-196b-4861-8b0d-e2d4351eed24-plugins-config-volume" pod "ce37c54e-196b-4861-8b0d-e2d4351eed24" (UID: "ce37c54e-196b-4861-8b0d-e2d4351eed24"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 01:40:07.940309 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:40:07.940290 4939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ce37c54e-196b-4861-8b0d-e2d4351eed24-kube-api-access-cckw9" pod "ce37c54e-196b-4861-8b0d-e2d4351eed24" (UID: "ce37c54e-196b-4861-8b0d-e2d4351eed24"). InnerVolumeSpecName "kube-api-access-cckw9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 01:40:08.039505 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:40:08.039426 4939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ab5ffea1-5d04-47c2-9d4d-d720550bc30f-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ab5ffea1-5d04-47c2-9d4d-d720550bc30f-plugins-config-volume\") pod \"ab5ffea1-5d04-47c2-9d4d-d720550bc30f\" (UID: \"ab5ffea1-5d04-47c2-9d4d-d720550bc30f\") " Nov 03 01:40:08.039669 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:40:08.039580 4939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ab5ffea1-5d04-47c2-9d4d-d720550bc30f-kube-api-access-cwwzd\" (UniqueName: \"kubernetes.io/projected/ab5ffea1-5d04-47c2-9d4d-d720550bc30f-kube-api-access-cwwzd\") pod \"ab5ffea1-5d04-47c2-9d4d-d720550bc30f\" (UID: \"ab5ffea1-5d04-47c2-9d4d-d720550bc30f\") " Nov 03 01:40:08.039821 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:40:08.039796 4939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ab5ffea1-5d04-47c2-9d4d-d720550bc30f-plugins-config-volume" pod "ab5ffea1-5d04-47c2-9d4d-d720550bc30f" (UID: "ab5ffea1-5d04-47c2-9d4d-d720550bc30f"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 01:40:08.040354 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:40:08.040330 4939 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ce37c54e-196b-4861-8b0d-e2d4351eed24-plugins-config-volume\") on node \"ip-10-0-140-107.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:40:08.040476 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:40:08.040359 4939 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ab5ffea1-5d04-47c2-9d4d-d720550bc30f-plugins-config-volume\") on node \"ip-10-0-140-107.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:40:08.040476 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:40:08.040376 4939 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cckw9\" (UniqueName: \"kubernetes.io/projected/ce37c54e-196b-4861-8b0d-e2d4351eed24-kube-api-access-cckw9\") on node \"ip-10-0-140-107.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:40:08.042008 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:40:08.041982 4939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ab5ffea1-5d04-47c2-9d4d-d720550bc30f-kube-api-access-cwwzd" pod "ab5ffea1-5d04-47c2-9d4d-d720550bc30f" (UID: "ab5ffea1-5d04-47c2-9d4d-d720550bc30f"). InnerVolumeSpecName "kube-api-access-cwwzd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 01:40:08.126087 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:40:08.126049 4939 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-extmod-b-3c5983-7fbf586cf5-92rtm" podUID="16b64414-6144-4743-bc96-91dffcd4a6b7" containerName="payload-processing" containerID="cri-o://177c06e36a95a1b726d3dc4d6ca587944b9f13bdb65004220d6bc5f91c9c44d6" gracePeriod=25 Nov 03 01:40:08.139640 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:40:08.139607 4939 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-3c5983-564977cd99-qs5fk" podUID="3137d8da-32e8-47fd-87a6-f08aebe2a384" containerName="payload-pre-processing" containerID="cri-o://aae7b6322990462714bb68ba10cf5baf2dc655cea6d5819f6d0a0a81bad6fed4" gracePeriod=25 Nov 03 01:40:08.142114 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:40:08.142089 4939 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cwwzd\" (UniqueName: \"kubernetes.io/projected/ab5ffea1-5d04-47c2-9d4d-d720550bc30f-kube-api-access-cwwzd\") on node \"ip-10-0-140-107.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:40:08.145627 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:40:08.145606 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:40:08.212708 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:40:08.212491 4939 generic.go:356] "Generic (PLEG): container finished" podID="ce37c54e-196b-4861-8b0d-e2d4351eed24" containerID="89db74dd8260ac3a9c06c43a81706ad1b6010460e1db156d2a53ccad4650368c" exitCode=0 Nov 03 01:40:08.212708 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:40:08.212548 4939 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-a-027c19-596c6c9695-wm55p" Nov 03 01:40:08.212708 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:40:08.212545 4939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-a-027c19-596c6c9695-wm55p" event={"ID":"ce37c54e-196b-4861-8b0d-e2d4351eed24","Type":"ContainerDied","Data":"89db74dd8260ac3a9c06c43a81706ad1b6010460e1db156d2a53ccad4650368c"} Nov 03 01:40:08.212708 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:40:08.212688 4939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-a-027c19-596c6c9695-wm55p" event={"ID":"ce37c54e-196b-4861-8b0d-e2d4351eed24","Type":"ContainerDied","Data":"d180dbd0cfea2d54af90aa6744d70bb4c7eb445d847462d21ebebe567b66b143"} Nov 03 01:40:08.213158 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:40:08.212722 4939 scope.go:122] "RemoveContainer" containerID="89db74dd8260ac3a9c06c43a81706ad1b6010460e1db156d2a53ccad4650368c" Nov 03 01:40:08.214551 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:40:08.214527 4939 generic.go:356] "Generic (PLEG): container finished" podID="ab5ffea1-5d04-47c2-9d4d-d720550bc30f" containerID="5f1ab4affd67f9c19aeb9973a274a941d79ca3d8585bd1f61f141a9875346ae5" exitCode=0 Nov 03 01:40:08.214704 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:40:08.214558 4939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-027c19-67b548955-rltd2" event={"ID":"ab5ffea1-5d04-47c2-9d4d-d720550bc30f","Type":"ContainerDied","Data":"5f1ab4affd67f9c19aeb9973a274a941d79ca3d8585bd1f61f141a9875346ae5"} Nov 03 01:40:08.214704 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:40:08.214574 4939 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-027c19-67b548955-rltd2" Nov 03 01:40:08.214704 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:40:08.214612 4939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-027c19-67b548955-rltd2" event={"ID":"ab5ffea1-5d04-47c2-9d4d-d720550bc30f","Type":"ContainerDied","Data":"83d0ef44df2da4f704f3b2a785e33921707cf24b178fbfd056f09e13ec1b933e"} Nov 03 01:40:08.233754 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:40:08.233727 4939 scope.go:122] "RemoveContainer" containerID="89db74dd8260ac3a9c06c43a81706ad1b6010460e1db156d2a53ccad4650368c" Nov 03 01:40:08.234149 ip-10-0-140-107 kubenswrapper[4939]: E1103 01:40:08.234126 4939 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"89db74dd8260ac3a9c06c43a81706ad1b6010460e1db156d2a53ccad4650368c\": container with ID starting with 89db74dd8260ac3a9c06c43a81706ad1b6010460e1db156d2a53ccad4650368c not found: ID does not exist" containerID="89db74dd8260ac3a9c06c43a81706ad1b6010460e1db156d2a53ccad4650368c" Nov 03 01:40:08.234283 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:40:08.234161 4939 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"89db74dd8260ac3a9c06c43a81706ad1b6010460e1db156d2a53ccad4650368c"} err="failed to get container status \"89db74dd8260ac3a9c06c43a81706ad1b6010460e1db156d2a53ccad4650368c\": rpc error: code = NotFound desc = could not find container \"89db74dd8260ac3a9c06c43a81706ad1b6010460e1db156d2a53ccad4650368c\": container with ID starting with 89db74dd8260ac3a9c06c43a81706ad1b6010460e1db156d2a53ccad4650368c not found: ID does not exist" Nov 03 01:40:08.234283 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:40:08.234184 4939 scope.go:122] "RemoveContainer" containerID="5f1ab4affd67f9c19aeb9973a274a941d79ca3d8585bd1f61f141a9875346ae5" Nov 03 01:40:08.246015 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:40:08.245934 4939 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-027c19-67b548955-rltd2"] Nov 03 01:40:08.246113 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:40:08.246091 4939 scope.go:122] "RemoveContainer" containerID="5f1ab4affd67f9c19aeb9973a274a941d79ca3d8585bd1f61f141a9875346ae5" Nov 03 01:40:08.246391 ip-10-0-140-107 kubenswrapper[4939]: E1103 01:40:08.246370 4939 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5f1ab4affd67f9c19aeb9973a274a941d79ca3d8585bd1f61f141a9875346ae5\": container with ID starting with 5f1ab4affd67f9c19aeb9973a274a941d79ca3d8585bd1f61f141a9875346ae5 not found: ID does not exist" containerID="5f1ab4affd67f9c19aeb9973a274a941d79ca3d8585bd1f61f141a9875346ae5" Nov 03 01:40:08.246464 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:40:08.246402 4939 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5f1ab4affd67f9c19aeb9973a274a941d79ca3d8585bd1f61f141a9875346ae5"} err="failed to get container status \"5f1ab4affd67f9c19aeb9973a274a941d79ca3d8585bd1f61f141a9875346ae5\": rpc error: code = NotFound desc = could not find container \"5f1ab4affd67f9c19aeb9973a274a941d79ca3d8585bd1f61f141a9875346ae5\": container with ID starting with 5f1ab4affd67f9c19aeb9973a274a941d79ca3d8585bd1f61f141a9875346ae5 not found: ID does not exist" Nov 03 01:40:08.249532 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:40:08.249510 4939 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-027c19-67b548955-rltd2"] Nov 03 01:40:08.257811 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:40:08.257786 4939 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-027c19-596c6c9695-wm55p"] Nov 03 01:40:08.261334 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:40:08.261316 4939 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-027c19-596c6c9695-wm55p"] Nov 03 01:40:08.439425 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:40:08.439395 4939 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-3c5983-564977cd99-qs5fk" podUID="3137d8da-32e8-47fd-87a6-f08aebe2a384" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 03 01:40:08.904051 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:40:08.904018 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:40:09.590660 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:40:09.590628 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:40:09.870884 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:40:09.870807 4939 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ab5ffea1-5d04-47c2-9d4d-d720550bc30f" path="/var/lib/kubelet/pods/ab5ffea1-5d04-47c2-9d4d-d720550bc30f/volumes" Nov 03 01:40:09.871291 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:40:09.871273 4939 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ce37c54e-196b-4861-8b0d-e2d4351eed24" path="/var/lib/kubelet/pods/ce37c54e-196b-4861-8b0d-e2d4351eed24/volumes" Nov 03 01:40:09.887512 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:40:09.887482 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:40:10.132173 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:40:10.132092 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:40:10.504415 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:40:10.504318 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:40:11.111975 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:40:11.111942 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:40:11.710283 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:40:11.710248 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:40:12.298145 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:40:12.298106 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:40:12.978152 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:40:12.978118 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:40:13.588644 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:40:13.588608 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:40:14.172420 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:40:14.172385 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:40:14.776126 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:40:14.776062 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:40:15.482380 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:40:15.482341 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:40:16.183508 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:40:16.183474 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:40:16.306891 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:40:16.306854 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:40:16.422794 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:40:16.422764 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:40:16.820871 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:40:16.820834 4939 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-4pw7v"] Nov 03 01:40:16.882520 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:40:16.882487 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:40:17.098946 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:40:17.098902 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:40:17.187967 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:40:17.187931 4939 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-b-3c5983-openshift-default-86c5bc855c-2hx8m"] Nov 03 01:40:17.188451 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:40:17.188417 4939 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-extmod-b-3c5983-openshift-default-86c5bc855c-2hx8m" podUID="78a38f10-6762-489e-813a-ac1cb91f5844" containerName="istio-proxy" containerID="cri-o://d25cdbfedd2c4039d8e796f62d8dd7d44fa46d1f7ff0d34c9badaa9eb7301a3d" gracePeriod=30 Nov 03 01:40:17.227127 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:40:17.227085 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:40:17.414101 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:40:17.411028 4939 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-extmod-b-3c5983-7fbf586cf5-92rtm" podUID="16b64414-6144-4743-bc96-91dffcd4a6b7" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 03 01:40:18.058149 ip-10-0-140-107 kubenswrapper[4939]: E1103 01:40:18.058112 4939 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-sthjh" containerName="extractor" Nov 03 01:40:18.059178 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:40:18.059147 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:40:18.072456 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:40:18.072414 4939 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w3-717217-openshift-default-578c69554c-mwwv6"] Nov 03 01:40:18.073776 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:40:18.073750 4939 cpu_manager.go:414] "containerMap: removing container" podUID="ab5ffea1-5d04-47c2-9d4d-d720550bc30f" containerName="payload-pre-processing" Nov 03 01:40:18.073927 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:40:18.073778 4939 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ab5ffea1-5d04-47c2-9d4d-d720550bc30f" containerName="payload-pre-processing" Nov 03 01:40:18.073927 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:40:18.073804 4939 cpu_manager.go:414] "containerMap: removing container" podUID="f09e7c2c-246e-4c96-a8ca-147e8ca83560" containerName="istio-proxy" Nov 03 01:40:18.073927 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:40:18.073813 4939 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f09e7c2c-246e-4c96-a8ca-147e8ca83560" containerName="istio-proxy" Nov 03 01:40:18.073927 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:40:18.073833 4939 cpu_manager.go:414] "containerMap: removing container" podUID="ce37c54e-196b-4861-8b0d-e2d4351eed24" containerName="payload-processing" Nov 03 01:40:18.073927 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:40:18.073841 4939 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ce37c54e-196b-4861-8b0d-e2d4351eed24" containerName="payload-processing" Nov 03 01:40:18.073927 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:40:18.073862 4939 cpu_manager.go:414] "containerMap: removing container" podUID="5f7f5188-d311-486b-bb78-583a436145c5" containerName="main" Nov 03 01:40:18.073927 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:40:18.073871 4939 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5f7f5188-d311-486b-bb78-583a436145c5" containerName="main" Nov 03 01:40:18.073927 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:40:18.073882 4939 cpu_manager.go:414] "containerMap: removing container" podUID="f09e7c2c-246e-4c96-a8ca-147e8ca83560" containerName="istio-proxy" Nov 03 01:40:18.073927 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:40:18.073891 4939 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f09e7c2c-246e-4c96-a8ca-147e8ca83560" containerName="istio-proxy" Nov 03 01:40:18.073927 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:40:18.073931 4939 cpu_manager.go:414] "containerMap: removing container" podUID="12a62cc4-d79a-495f-80c5-a2449e754e1b" containerName="test-curl-7037-1655dd" Nov 03 01:40:18.074418 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:40:18.073939 4939 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="12a62cc4-d79a-495f-80c5-a2449e754e1b" containerName="test-curl-7037-1655dd" Nov 03 01:40:18.074418 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:40:18.074132 4939 memory_manager.go:365] "RemoveStaleState removing state" podUID="5f7f5188-d311-486b-bb78-583a436145c5" containerName="main" Nov 03 01:40:18.074418 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:40:18.074151 4939 memory_manager.go:365] "RemoveStaleState removing state" podUID="f09e7c2c-246e-4c96-a8ca-147e8ca83560" containerName="istio-proxy" Nov 03 01:40:18.074418 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:40:18.074168 4939 memory_manager.go:365] "RemoveStaleState removing state" podUID="12a62cc4-d79a-495f-80c5-a2449e754e1b" containerName="test-curl-7037-1655dd" Nov 03 01:40:18.074418 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:40:18.074190 4939 memory_manager.go:365] "RemoveStaleState removing state" podUID="ab5ffea1-5d04-47c2-9d4d-d720550bc30f" containerName="payload-pre-processing" Nov 03 01:40:18.074418 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:40:18.074203 4939 memory_manager.go:365] "RemoveStaleState removing state" podUID="ce37c54e-196b-4861-8b0d-e2d4351eed24" containerName="payload-processing" Nov 03 01:40:18.078557 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:40:18.078536 4939 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w3-717217-openshift-default-578c69554c-mwwv6" Nov 03 01:40:18.081286 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:40:18.081269 4939 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-717217-openshift-default-578c69554c-mwwv6"] Nov 03 01:40:18.085373 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:40:18.085346 4939 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-w3-717217-openshift-default-dockercfg-4kr2v\"" Nov 03 01:40:18.143556 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:40:18.143491 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d0933771-7032-4cff-aafe-2f7211ebb549-istio-data\") pod \"e2e-shared-a-w3-717217-openshift-default-578c69554c-mwwv6\" (UID: \"d0933771-7032-4cff-aafe-2f7211ebb549\") " pod="openshift-ingress/e2e-shared-a-w3-717217-openshift-default-578c69554c-mwwv6" Nov 03 01:40:18.143758 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:40:18.143609 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rdlmd\" (UniqueName: \"kubernetes.io/projected/d0933771-7032-4cff-aafe-2f7211ebb549-kube-api-access-rdlmd\") pod \"e2e-shared-a-w3-717217-openshift-default-578c69554c-mwwv6\" (UID: \"d0933771-7032-4cff-aafe-2f7211ebb549\") " pod="openshift-ingress/e2e-shared-a-w3-717217-openshift-default-578c69554c-mwwv6" Nov 03 01:40:18.144062 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:40:18.143830 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d0933771-7032-4cff-aafe-2f7211ebb549-workload-socket\") pod \"e2e-shared-a-w3-717217-openshift-default-578c69554c-mwwv6\" (UID: \"d0933771-7032-4cff-aafe-2f7211ebb549\") " pod="openshift-ingress/e2e-shared-a-w3-717217-openshift-default-578c69554c-mwwv6" Nov 03 01:40:18.144062 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:40:18.143857 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d0933771-7032-4cff-aafe-2f7211ebb549-istiod-ca-cert\") pod \"e2e-shared-a-w3-717217-openshift-default-578c69554c-mwwv6\" (UID: \"d0933771-7032-4cff-aafe-2f7211ebb549\") " pod="openshift-ingress/e2e-shared-a-w3-717217-openshift-default-578c69554c-mwwv6" Nov 03 01:40:18.144062 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:40:18.143887 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d0933771-7032-4cff-aafe-2f7211ebb549-istio-token\") pod \"e2e-shared-a-w3-717217-openshift-default-578c69554c-mwwv6\" (UID: \"d0933771-7032-4cff-aafe-2f7211ebb549\") " pod="openshift-ingress/e2e-shared-a-w3-717217-openshift-default-578c69554c-mwwv6" Nov 03 01:40:18.144062 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:40:18.143991 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d0933771-7032-4cff-aafe-2f7211ebb549-istio-podinfo\") pod \"e2e-shared-a-w3-717217-openshift-default-578c69554c-mwwv6\" (UID: \"d0933771-7032-4cff-aafe-2f7211ebb549\") " pod="openshift-ingress/e2e-shared-a-w3-717217-openshift-default-578c69554c-mwwv6" Nov 03 01:40:18.144062 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:40:18.144019 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d0933771-7032-4cff-aafe-2f7211ebb549-istio-envoy\") pod \"e2e-shared-a-w3-717217-openshift-default-578c69554c-mwwv6\" (UID: \"d0933771-7032-4cff-aafe-2f7211ebb549\") " pod="openshift-ingress/e2e-shared-a-w3-717217-openshift-default-578c69554c-mwwv6" Nov 03 01:40:18.144062 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:40:18.144041 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d0933771-7032-4cff-aafe-2f7211ebb549-workload-certs\") pod \"e2e-shared-a-w3-717217-openshift-default-578c69554c-mwwv6\" (UID: \"d0933771-7032-4cff-aafe-2f7211ebb549\") " pod="openshift-ingress/e2e-shared-a-w3-717217-openshift-default-578c69554c-mwwv6" Nov 03 01:40:18.144062 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:40:18.144069 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d0933771-7032-4cff-aafe-2f7211ebb549-credential-socket\") pod \"e2e-shared-a-w3-717217-openshift-default-578c69554c-mwwv6\" (UID: \"d0933771-7032-4cff-aafe-2f7211ebb549\") " pod="openshift-ingress/e2e-shared-a-w3-717217-openshift-default-578c69554c-mwwv6" Nov 03 01:40:18.182046 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:40:18.181989 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:40:18.245838 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:40:18.245800 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d0933771-7032-4cff-aafe-2f7211ebb549-istio-data\") pod \"e2e-shared-a-w3-717217-openshift-default-578c69554c-mwwv6\" (UID: \"d0933771-7032-4cff-aafe-2f7211ebb549\") " pod="openshift-ingress/e2e-shared-a-w3-717217-openshift-default-578c69554c-mwwv6" Nov 03 01:40:18.246056 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:40:18.245885 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rdlmd\" (UniqueName: \"kubernetes.io/projected/d0933771-7032-4cff-aafe-2f7211ebb549-kube-api-access-rdlmd\") pod \"e2e-shared-a-w3-717217-openshift-default-578c69554c-mwwv6\" (UID: \"d0933771-7032-4cff-aafe-2f7211ebb549\") " pod="openshift-ingress/e2e-shared-a-w3-717217-openshift-default-578c69554c-mwwv6" Nov 03 01:40:18.246110 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:40:18.246074 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d0933771-7032-4cff-aafe-2f7211ebb549-workload-socket\") pod \"e2e-shared-a-w3-717217-openshift-default-578c69554c-mwwv6\" (UID: \"d0933771-7032-4cff-aafe-2f7211ebb549\") " pod="openshift-ingress/e2e-shared-a-w3-717217-openshift-default-578c69554c-mwwv6" Nov 03 01:40:18.246110 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:40:18.246097 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d0933771-7032-4cff-aafe-2f7211ebb549-istiod-ca-cert\") pod \"e2e-shared-a-w3-717217-openshift-default-578c69554c-mwwv6\" (UID: \"d0933771-7032-4cff-aafe-2f7211ebb549\") " pod="openshift-ingress/e2e-shared-a-w3-717217-openshift-default-578c69554c-mwwv6" Nov 03 01:40:18.246196 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:40:18.246121 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d0933771-7032-4cff-aafe-2f7211ebb549-istio-token\") pod \"e2e-shared-a-w3-717217-openshift-default-578c69554c-mwwv6\" (UID: \"d0933771-7032-4cff-aafe-2f7211ebb549\") " pod="openshift-ingress/e2e-shared-a-w3-717217-openshift-default-578c69554c-mwwv6" Nov 03 01:40:18.246251 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:40:18.246198 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d0933771-7032-4cff-aafe-2f7211ebb549-istio-data\") pod \"e2e-shared-a-w3-717217-openshift-default-578c69554c-mwwv6\" (UID: \"d0933771-7032-4cff-aafe-2f7211ebb549\") " pod="openshift-ingress/e2e-shared-a-w3-717217-openshift-default-578c69554c-mwwv6" Nov 03 01:40:18.246251 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:40:18.246217 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d0933771-7032-4cff-aafe-2f7211ebb549-istio-podinfo\") pod \"e2e-shared-a-w3-717217-openshift-default-578c69554c-mwwv6\" (UID: \"d0933771-7032-4cff-aafe-2f7211ebb549\") " pod="openshift-ingress/e2e-shared-a-w3-717217-openshift-default-578c69554c-mwwv6" Nov 03 01:40:18.246363 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:40:18.246268 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d0933771-7032-4cff-aafe-2f7211ebb549-istio-envoy\") pod \"e2e-shared-a-w3-717217-openshift-default-578c69554c-mwwv6\" (UID: \"d0933771-7032-4cff-aafe-2f7211ebb549\") " pod="openshift-ingress/e2e-shared-a-w3-717217-openshift-default-578c69554c-mwwv6" Nov 03 01:40:18.246363 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:40:18.246299 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d0933771-7032-4cff-aafe-2f7211ebb549-workload-certs\") pod \"e2e-shared-a-w3-717217-openshift-default-578c69554c-mwwv6\" (UID: \"d0933771-7032-4cff-aafe-2f7211ebb549\") " pod="openshift-ingress/e2e-shared-a-w3-717217-openshift-default-578c69554c-mwwv6" Nov 03 01:40:18.246363 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:40:18.246331 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d0933771-7032-4cff-aafe-2f7211ebb549-credential-socket\") pod \"e2e-shared-a-w3-717217-openshift-default-578c69554c-mwwv6\" (UID: \"d0933771-7032-4cff-aafe-2f7211ebb549\") " pod="openshift-ingress/e2e-shared-a-w3-717217-openshift-default-578c69554c-mwwv6" Nov 03 01:40:18.246543 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:40:18.246497 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d0933771-7032-4cff-aafe-2f7211ebb549-workload-socket\") pod \"e2e-shared-a-w3-717217-openshift-default-578c69554c-mwwv6\" (UID: \"d0933771-7032-4cff-aafe-2f7211ebb549\") " pod="openshift-ingress/e2e-shared-a-w3-717217-openshift-default-578c69554c-mwwv6" Nov 03 01:40:18.246673 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:40:18.246648 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d0933771-7032-4cff-aafe-2f7211ebb549-workload-certs\") pod \"e2e-shared-a-w3-717217-openshift-default-578c69554c-mwwv6\" (UID: \"d0933771-7032-4cff-aafe-2f7211ebb549\") " pod="openshift-ingress/e2e-shared-a-w3-717217-openshift-default-578c69554c-mwwv6" Nov 03 01:40:18.246748 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:40:18.246730 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d0933771-7032-4cff-aafe-2f7211ebb549-credential-socket\") pod \"e2e-shared-a-w3-717217-openshift-default-578c69554c-mwwv6\" (UID: \"d0933771-7032-4cff-aafe-2f7211ebb549\") " pod="openshift-ingress/e2e-shared-a-w3-717217-openshift-default-578c69554c-mwwv6" Nov 03 01:40:18.246863 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:40:18.246844 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d0933771-7032-4cff-aafe-2f7211ebb549-istiod-ca-cert\") pod \"e2e-shared-a-w3-717217-openshift-default-578c69554c-mwwv6\" (UID: \"d0933771-7032-4cff-aafe-2f7211ebb549\") " pod="openshift-ingress/e2e-shared-a-w3-717217-openshift-default-578c69554c-mwwv6" Nov 03 01:40:18.248643 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:40:18.248614 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d0933771-7032-4cff-aafe-2f7211ebb549-istio-envoy\") pod \"e2e-shared-a-w3-717217-openshift-default-578c69554c-mwwv6\" (UID: \"d0933771-7032-4cff-aafe-2f7211ebb549\") " pod="openshift-ingress/e2e-shared-a-w3-717217-openshift-default-578c69554c-mwwv6" Nov 03 01:40:18.248788 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:40:18.248773 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d0933771-7032-4cff-aafe-2f7211ebb549-istio-podinfo\") pod \"e2e-shared-a-w3-717217-openshift-default-578c69554c-mwwv6\" (UID: \"d0933771-7032-4cff-aafe-2f7211ebb549\") " pod="openshift-ingress/e2e-shared-a-w3-717217-openshift-default-578c69554c-mwwv6" Nov 03 01:40:18.253544 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:40:18.253521 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d0933771-7032-4cff-aafe-2f7211ebb549-istio-token\") pod \"e2e-shared-a-w3-717217-openshift-default-578c69554c-mwwv6\" (UID: \"d0933771-7032-4cff-aafe-2f7211ebb549\") " pod="openshift-ingress/e2e-shared-a-w3-717217-openshift-default-578c69554c-mwwv6" Nov 03 01:40:18.253867 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:40:18.253849 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rdlmd\" (UniqueName: \"kubernetes.io/projected/d0933771-7032-4cff-aafe-2f7211ebb549-kube-api-access-rdlmd\") pod \"e2e-shared-a-w3-717217-openshift-default-578c69554c-mwwv6\" (UID: \"d0933771-7032-4cff-aafe-2f7211ebb549\") " pod="openshift-ingress/e2e-shared-a-w3-717217-openshift-default-578c69554c-mwwv6" Nov 03 01:40:18.426087 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:40:18.426050 4939 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w3-717217-openshift-default-578c69554c-mwwv6" Nov 03 01:40:18.439946 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:40:18.439877 4939 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-3c5983-564977cd99-qs5fk" podUID="3137d8da-32e8-47fd-87a6-f08aebe2a384" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 03 01:40:18.715577 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:40:18.715542 4939 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-717217-openshift-default-578c69554c-mwwv6"] Nov 03 01:40:18.715827 ip-10-0-140-107 kubenswrapper[4939]: W1103 01:40:18.715803 4939 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd0933771_7032_4cff_aafe_2f7211ebb549.slice/crio-b3864d9992430be907ca4918e5ec90b3fc2c8800bb2bc26816bc92f5d4d845e7 WatchSource:0}: Error finding container b3864d9992430be907ca4918e5ec90b3fc2c8800bb2bc26816bc92f5d4d845e7: Status 404 returned error can't find the container with id b3864d9992430be907ca4918e5ec90b3fc2c8800bb2bc26816bc92f5d4d845e7 Nov 03 01:40:18.718863 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:40:18.718816 4939 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 01:40:18.718995 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:40:18.718921 4939 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 01:40:18.718995 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:40:18.718962 4939 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 01:40:18.888734 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:40:18.888691 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:40:19.303166 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:40:19.303123 4939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-717217-openshift-default-578c69554c-mwwv6" event={"ID":"d0933771-7032-4cff-aafe-2f7211ebb549","Type":"ContainerStarted","Data":"9827abc03252ecfd590c15fdd611c447bd1281bceaf0d372ef235336b8d21aa5"} Nov 03 01:40:19.303570 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:40:19.303181 4939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-717217-openshift-default-578c69554c-mwwv6" event={"ID":"d0933771-7032-4cff-aafe-2f7211ebb549","Type":"ContainerStarted","Data":"b3864d9992430be907ca4918e5ec90b3fc2c8800bb2bc26816bc92f5d4d845e7"} Nov 03 01:40:19.328736 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:40:19.328687 4939 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w3-717217-openshift-default-578c69554c-mwwv6" podStartSLOduration=1.328674181 podStartE2EDuration="1.328674181s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 01:40:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 01:40:19.322838931 +0000 UTC m=+3567.587173876" watchObservedRunningTime="2026-11-03 01:40:19.328674181 +0000 UTC m=+3567.593009200" Nov 03 01:40:19.427101 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:40:19.427017 4939 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w3-717217-openshift-default-578c69554c-mwwv6" Nov 03 01:40:19.429104 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:40:19.429064 4939 patch_prober.go:29] interesting pod/e2e-shared-a-w3-717217-openshift-default-578c69554c-mwwv6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.85:15021/healthz/ready\": dial tcp 10.132.1.85:15021: connect: connection refused" start-of-body= Nov 03 01:40:19.429259 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:40:19.429129 4939 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-717217-openshift-default-578c69554c-mwwv6" podUID="d0933771-7032-4cff-aafe-2f7211ebb549" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.85:15021/healthz/ready\": dial tcp 10.132.1.85:15021: connect: connection refused" Nov 03 01:40:19.514521 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:40:19.514485 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:40:19.771176 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:40:19.771082 4939 patch_prober.go:29] interesting pod/e2e-extmod-b-3c5983-openshift-default-86c5bc855c-2hx8m container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 03 01:40:19.771445 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:40:19.771408 4939 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-extmod-b-3c5983-openshift-default-86c5bc855c-2hx8m" podUID="78a38f10-6762-489e-813a-ac1cb91f5844" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 03 01:40:19.841319 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:40:19.841276 4939 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w4-6c0ec0-openshift-default-79c7584bfc-kxj54"] Nov 03 01:40:19.842138 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:40:19.842119 4939 memory_manager.go:365] "RemoveStaleState removing state" podUID="f09e7c2c-246e-4c96-a8ca-147e8ca83560" containerName="istio-proxy" Nov 03 01:40:19.845843 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:40:19.845826 4939 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w4-6c0ec0-openshift-default-79c7584bfc-kxj54" Nov 03 01:40:19.863293 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:40:19.863259 4939 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-w4-6c0ec0-openshift-default-dockercfg-dgkf8\"" Nov 03 01:40:19.897927 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:40:19.897884 4939 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w4-6c0ec0-openshift-default-79c7584bfc-kxj54"] Nov 03 01:40:19.984727 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:40:19.984684 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d1ad622d-53f3-48ca-bc88-b7ab4ae452fe-workload-socket\") pod \"e2e-shared-a-w4-6c0ec0-openshift-default-79c7584bfc-kxj54\" (UID: \"d1ad622d-53f3-48ca-bc88-b7ab4ae452fe\") " pod="openshift-ingress/e2e-shared-a-w4-6c0ec0-openshift-default-79c7584bfc-kxj54" Nov 03 01:40:19.984991 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:40:19.984964 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d1ad622d-53f3-48ca-bc88-b7ab4ae452fe-istiod-ca-cert\") pod \"e2e-shared-a-w4-6c0ec0-openshift-default-79c7584bfc-kxj54\" (UID: \"d1ad622d-53f3-48ca-bc88-b7ab4ae452fe\") " pod="openshift-ingress/e2e-shared-a-w4-6c0ec0-openshift-default-79c7584bfc-kxj54" Nov 03 01:40:19.985097 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:40:19.985020 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ltpnq\" (UniqueName: \"kubernetes.io/projected/d1ad622d-53f3-48ca-bc88-b7ab4ae452fe-kube-api-access-ltpnq\") pod \"e2e-shared-a-w4-6c0ec0-openshift-default-79c7584bfc-kxj54\" (UID: \"d1ad622d-53f3-48ca-bc88-b7ab4ae452fe\") " pod="openshift-ingress/e2e-shared-a-w4-6c0ec0-openshift-default-79c7584bfc-kxj54" Nov 03 01:40:19.985171 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:40:19.985087 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d1ad622d-53f3-48ca-bc88-b7ab4ae452fe-istio-data\") pod \"e2e-shared-a-w4-6c0ec0-openshift-default-79c7584bfc-kxj54\" (UID: \"d1ad622d-53f3-48ca-bc88-b7ab4ae452fe\") " pod="openshift-ingress/e2e-shared-a-w4-6c0ec0-openshift-default-79c7584bfc-kxj54" Nov 03 01:40:19.985275 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:40:19.985256 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d1ad622d-53f3-48ca-bc88-b7ab4ae452fe-istio-podinfo\") pod \"e2e-shared-a-w4-6c0ec0-openshift-default-79c7584bfc-kxj54\" (UID: \"d1ad622d-53f3-48ca-bc88-b7ab4ae452fe\") " pod="openshift-ingress/e2e-shared-a-w4-6c0ec0-openshift-default-79c7584bfc-kxj54" Nov 03 01:40:19.985418 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:40:19.985396 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d1ad622d-53f3-48ca-bc88-b7ab4ae452fe-workload-certs\") pod \"e2e-shared-a-w4-6c0ec0-openshift-default-79c7584bfc-kxj54\" (UID: \"d1ad622d-53f3-48ca-bc88-b7ab4ae452fe\") " pod="openshift-ingress/e2e-shared-a-w4-6c0ec0-openshift-default-79c7584bfc-kxj54" Nov 03 01:40:19.985548 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:40:19.985530 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d1ad622d-53f3-48ca-bc88-b7ab4ae452fe-istio-envoy\") pod \"e2e-shared-a-w4-6c0ec0-openshift-default-79c7584bfc-kxj54\" (UID: \"d1ad622d-53f3-48ca-bc88-b7ab4ae452fe\") " pod="openshift-ingress/e2e-shared-a-w4-6c0ec0-openshift-default-79c7584bfc-kxj54" Nov 03 01:40:19.985598 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:40:19.985571 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d1ad622d-53f3-48ca-bc88-b7ab4ae452fe-credential-socket\") pod \"e2e-shared-a-w4-6c0ec0-openshift-default-79c7584bfc-kxj54\" (UID: \"d1ad622d-53f3-48ca-bc88-b7ab4ae452fe\") " pod="openshift-ingress/e2e-shared-a-w4-6c0ec0-openshift-default-79c7584bfc-kxj54" Nov 03 01:40:19.985647 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:40:19.985600 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d1ad622d-53f3-48ca-bc88-b7ab4ae452fe-istio-token\") pod \"e2e-shared-a-w4-6c0ec0-openshift-default-79c7584bfc-kxj54\" (UID: \"d1ad622d-53f3-48ca-bc88-b7ab4ae452fe\") " pod="openshift-ingress/e2e-shared-a-w4-6c0ec0-openshift-default-79c7584bfc-kxj54" Nov 03 01:40:20.086959 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:40:20.086885 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d1ad622d-53f3-48ca-bc88-b7ab4ae452fe-workload-socket\") pod \"e2e-shared-a-w4-6c0ec0-openshift-default-79c7584bfc-kxj54\" (UID: \"d1ad622d-53f3-48ca-bc88-b7ab4ae452fe\") " pod="openshift-ingress/e2e-shared-a-w4-6c0ec0-openshift-default-79c7584bfc-kxj54" Nov 03 01:40:20.087168 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:40:20.087020 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d1ad622d-53f3-48ca-bc88-b7ab4ae452fe-istiod-ca-cert\") pod \"e2e-shared-a-w4-6c0ec0-openshift-default-79c7584bfc-kxj54\" (UID: \"d1ad622d-53f3-48ca-bc88-b7ab4ae452fe\") " pod="openshift-ingress/e2e-shared-a-w4-6c0ec0-openshift-default-79c7584bfc-kxj54" Nov 03 01:40:20.087168 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:40:20.087047 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ltpnq\" (UniqueName: \"kubernetes.io/projected/d1ad622d-53f3-48ca-bc88-b7ab4ae452fe-kube-api-access-ltpnq\") pod \"e2e-shared-a-w4-6c0ec0-openshift-default-79c7584bfc-kxj54\" (UID: \"d1ad622d-53f3-48ca-bc88-b7ab4ae452fe\") " pod="openshift-ingress/e2e-shared-a-w4-6c0ec0-openshift-default-79c7584bfc-kxj54" Nov 03 01:40:20.087168 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:40:20.087080 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d1ad622d-53f3-48ca-bc88-b7ab4ae452fe-istio-data\") pod \"e2e-shared-a-w4-6c0ec0-openshift-default-79c7584bfc-kxj54\" (UID: \"d1ad622d-53f3-48ca-bc88-b7ab4ae452fe\") " pod="openshift-ingress/e2e-shared-a-w4-6c0ec0-openshift-default-79c7584bfc-kxj54" Nov 03 01:40:20.087168 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:40:20.087156 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d1ad622d-53f3-48ca-bc88-b7ab4ae452fe-istio-podinfo\") pod \"e2e-shared-a-w4-6c0ec0-openshift-default-79c7584bfc-kxj54\" (UID: \"d1ad622d-53f3-48ca-bc88-b7ab4ae452fe\") " pod="openshift-ingress/e2e-shared-a-w4-6c0ec0-openshift-default-79c7584bfc-kxj54" Nov 03 01:40:20.087583 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:40:20.087262 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d1ad622d-53f3-48ca-bc88-b7ab4ae452fe-workload-certs\") pod \"e2e-shared-a-w4-6c0ec0-openshift-default-79c7584bfc-kxj54\" (UID: \"d1ad622d-53f3-48ca-bc88-b7ab4ae452fe\") " pod="openshift-ingress/e2e-shared-a-w4-6c0ec0-openshift-default-79c7584bfc-kxj54" Nov 03 01:40:20.087583 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:40:20.087355 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d1ad622d-53f3-48ca-bc88-b7ab4ae452fe-workload-socket\") pod \"e2e-shared-a-w4-6c0ec0-openshift-default-79c7584bfc-kxj54\" (UID: \"d1ad622d-53f3-48ca-bc88-b7ab4ae452fe\") " pod="openshift-ingress/e2e-shared-a-w4-6c0ec0-openshift-default-79c7584bfc-kxj54" Nov 03 01:40:20.087583 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:40:20.087372 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d1ad622d-53f3-48ca-bc88-b7ab4ae452fe-istio-envoy\") pod \"e2e-shared-a-w4-6c0ec0-openshift-default-79c7584bfc-kxj54\" (UID: \"d1ad622d-53f3-48ca-bc88-b7ab4ae452fe\") " pod="openshift-ingress/e2e-shared-a-w4-6c0ec0-openshift-default-79c7584bfc-kxj54" Nov 03 01:40:20.087583 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:40:20.087477 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d1ad622d-53f3-48ca-bc88-b7ab4ae452fe-credential-socket\") pod \"e2e-shared-a-w4-6c0ec0-openshift-default-79c7584bfc-kxj54\" (UID: \"d1ad622d-53f3-48ca-bc88-b7ab4ae452fe\") " pod="openshift-ingress/e2e-shared-a-w4-6c0ec0-openshift-default-79c7584bfc-kxj54" Nov 03 01:40:20.087583 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:40:20.087526 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d1ad622d-53f3-48ca-bc88-b7ab4ae452fe-istio-token\") pod \"e2e-shared-a-w4-6c0ec0-openshift-default-79c7584bfc-kxj54\" (UID: \"d1ad622d-53f3-48ca-bc88-b7ab4ae452fe\") " pod="openshift-ingress/e2e-shared-a-w4-6c0ec0-openshift-default-79c7584bfc-kxj54" Nov 03 01:40:20.087879 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:40:20.087770 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d1ad622d-53f3-48ca-bc88-b7ab4ae452fe-workload-certs\") pod \"e2e-shared-a-w4-6c0ec0-openshift-default-79c7584bfc-kxj54\" (UID: \"d1ad622d-53f3-48ca-bc88-b7ab4ae452fe\") " pod="openshift-ingress/e2e-shared-a-w4-6c0ec0-openshift-default-79c7584bfc-kxj54" Nov 03 01:40:20.087879 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:40:20.087812 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d1ad622d-53f3-48ca-bc88-b7ab4ae452fe-credential-socket\") pod \"e2e-shared-a-w4-6c0ec0-openshift-default-79c7584bfc-kxj54\" (UID: \"d1ad622d-53f3-48ca-bc88-b7ab4ae452fe\") " pod="openshift-ingress/e2e-shared-a-w4-6c0ec0-openshift-default-79c7584bfc-kxj54" Nov 03 01:40:20.088029 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:40:20.087950 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d1ad622d-53f3-48ca-bc88-b7ab4ae452fe-istio-data\") pod \"e2e-shared-a-w4-6c0ec0-openshift-default-79c7584bfc-kxj54\" (UID: \"d1ad622d-53f3-48ca-bc88-b7ab4ae452fe\") " pod="openshift-ingress/e2e-shared-a-w4-6c0ec0-openshift-default-79c7584bfc-kxj54" Nov 03 01:40:20.088094 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:40:20.088027 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d1ad622d-53f3-48ca-bc88-b7ab4ae452fe-istiod-ca-cert\") pod \"e2e-shared-a-w4-6c0ec0-openshift-default-79c7584bfc-kxj54\" (UID: \"d1ad622d-53f3-48ca-bc88-b7ab4ae452fe\") " pod="openshift-ingress/e2e-shared-a-w4-6c0ec0-openshift-default-79c7584bfc-kxj54" Nov 03 01:40:20.089614 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:40:20.089583 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d1ad622d-53f3-48ca-bc88-b7ab4ae452fe-istio-envoy\") pod \"e2e-shared-a-w4-6c0ec0-openshift-default-79c7584bfc-kxj54\" (UID: \"d1ad622d-53f3-48ca-bc88-b7ab4ae452fe\") " pod="openshift-ingress/e2e-shared-a-w4-6c0ec0-openshift-default-79c7584bfc-kxj54" Nov 03 01:40:20.089944 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:40:20.089893 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d1ad622d-53f3-48ca-bc88-b7ab4ae452fe-istio-podinfo\") pod \"e2e-shared-a-w4-6c0ec0-openshift-default-79c7584bfc-kxj54\" (UID: \"d1ad622d-53f3-48ca-bc88-b7ab4ae452fe\") " pod="openshift-ingress/e2e-shared-a-w4-6c0ec0-openshift-default-79c7584bfc-kxj54" Nov 03 01:40:20.095390 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:40:20.095366 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d1ad622d-53f3-48ca-bc88-b7ab4ae452fe-istio-token\") pod \"e2e-shared-a-w4-6c0ec0-openshift-default-79c7584bfc-kxj54\" (UID: \"d1ad622d-53f3-48ca-bc88-b7ab4ae452fe\") " pod="openshift-ingress/e2e-shared-a-w4-6c0ec0-openshift-default-79c7584bfc-kxj54" Nov 03 01:40:20.095537 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:40:20.095516 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ltpnq\" (UniqueName: \"kubernetes.io/projected/d1ad622d-53f3-48ca-bc88-b7ab4ae452fe-kube-api-access-ltpnq\") pod \"e2e-shared-a-w4-6c0ec0-openshift-default-79c7584bfc-kxj54\" (UID: \"d1ad622d-53f3-48ca-bc88-b7ab4ae452fe\") " pod="openshift-ingress/e2e-shared-a-w4-6c0ec0-openshift-default-79c7584bfc-kxj54" Nov 03 01:40:20.194361 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:40:20.194322 4939 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w4-6c0ec0-openshift-default-79c7584bfc-kxj54" Nov 03 01:40:20.427285 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:40:20.427198 4939 patch_prober.go:29] interesting pod/e2e-shared-a-w3-717217-openshift-default-578c69554c-mwwv6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.85:15021/healthz/ready\": dial tcp 10.132.1.85:15021: connect: connection refused" start-of-body= Nov 03 01:40:20.427285 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:40:20.427258 4939 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-717217-openshift-default-578c69554c-mwwv6" podUID="d0933771-7032-4cff-aafe-2f7211ebb549" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.85:15021/healthz/ready\": dial tcp 10.132.1.85:15021: connect: connection refused" Nov 03 01:40:20.475546 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:40:20.475513 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:40:20.518004 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:40:20.517967 4939 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w4-6c0ec0-openshift-default-79c7584bfc-kxj54"] Nov 03 01:40:20.582157 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:40:20.582105 4939 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 01:40:20.582314 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:40:20.582213 4939 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 01:40:20.582314 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:40:20.582249 4939 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 01:40:21.183013 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:40:21.182978 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:40:21.323858 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:40:21.323822 4939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w4-6c0ec0-openshift-default-79c7584bfc-kxj54" event={"ID":"d1ad622d-53f3-48ca-bc88-b7ab4ae452fe","Type":"ContainerStarted","Data":"f060af1ea2ac13a918c74b9aa73b49a77c6dc196c94c4c14ae9c408564eac77b"} Nov 03 01:40:21.324054 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:40:21.323885 4939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w4-6c0ec0-openshift-default-79c7584bfc-kxj54" event={"ID":"d1ad622d-53f3-48ca-bc88-b7ab4ae452fe","Type":"ContainerStarted","Data":"06a27d0a7508d1223685658daa66ab3fd6b16c8c7bd625ae3f686bba11e39472"} Nov 03 01:40:21.363934 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:40:21.354470 4939 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w4-6c0ec0-openshift-default-79c7584bfc-kxj54" podStartSLOduration=2.354450431 podStartE2EDuration="2.354450431s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 01:40:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 01:40:21.345082412 +0000 UTC m=+3569.609417366" watchObservedRunningTime="2026-11-03 01:40:21.354450431 +0000 UTC m=+3569.618785377" Nov 03 01:40:21.426830 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:40:21.426798 4939 patch_prober.go:29] interesting pod/e2e-shared-a-w3-717217-openshift-default-578c69554c-mwwv6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.85:15021/healthz/ready\": dial tcp 10.132.1.85:15021: connect: connection refused" start-of-body= Nov 03 01:40:21.427022 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:40:21.426852 4939 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-717217-openshift-default-578c69554c-mwwv6" podUID="d0933771-7032-4cff-aafe-2f7211ebb549" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.85:15021/healthz/ready\": dial tcp 10.132.1.85:15021: connect: connection refused" Nov 03 01:40:22.017331 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:40:22.017295 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:40:22.200742 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:40:22.200703 4939 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w4-6c0ec0-openshift-default-79c7584bfc-kxj54" Nov 03 01:40:22.203595 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:40:22.203563 4939 patch_prober.go:29] interesting pod/e2e-shared-a-w4-6c0ec0-openshift-default-79c7584bfc-kxj54 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.86:15021/healthz/ready\": dial tcp 10.132.1.86:15021: connect: connection refused" start-of-body= Nov 03 01:40:22.204274 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:40:22.203888 4939 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-6c0ec0-openshift-default-79c7584bfc-kxj54" podUID="d1ad622d-53f3-48ca-bc88-b7ab4ae452fe" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.86:15021/healthz/ready\": dial tcp 10.132.1.86:15021: connect: connection refused" Nov 03 01:40:22.361396 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:40:22.361357 4939 generic.go:356] "Generic (PLEG): container finished" podID="78a38f10-6762-489e-813a-ac1cb91f5844" containerID="d25cdbfedd2c4039d8e796f62d8dd7d44fa46d1f7ff0d34c9badaa9eb7301a3d" exitCode=0 Nov 03 01:40:22.368951 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:40:22.362763 4939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-3c5983-openshift-default-86c5bc855c-2hx8m" event={"ID":"78a38f10-6762-489e-813a-ac1cb91f5844","Type":"ContainerDied","Data":"d25cdbfedd2c4039d8e796f62d8dd7d44fa46d1f7ff0d34c9badaa9eb7301a3d"} Nov 03 01:40:22.369360 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:40:22.369341 4939 scope.go:122] "RemoveContainer" containerID="2379ea793c99a3d43b7adbe002e932cb7452b8a0597b970f11c7f367a61f7b14" Nov 03 01:40:22.426627 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:40:22.426593 4939 patch_prober.go:29] interesting pod/e2e-shared-a-w3-717217-openshift-default-578c69554c-mwwv6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.85:15021/healthz/ready\": dial tcp 10.132.1.85:15021: connect: connection refused" start-of-body= Nov 03 01:40:22.426806 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:40:22.426658 4939 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-717217-openshift-default-578c69554c-mwwv6" podUID="d0933771-7032-4cff-aafe-2f7211ebb549" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.85:15021/healthz/ready\": dial tcp 10.132.1.85:15021: connect: connection refused" Nov 03 01:40:22.645282 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:40:22.645252 4939 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-3c5983-openshift-default-86c5bc855c-2hx8m" Nov 03 01:40:22.817448 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:40:22.817402 4939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/78a38f10-6762-489e-813a-ac1cb91f5844-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/78a38f10-6762-489e-813a-ac1cb91f5844-istiod-ca-cert\") pod \"78a38f10-6762-489e-813a-ac1cb91f5844\" (UID: \"78a38f10-6762-489e-813a-ac1cb91f5844\") " Nov 03 01:40:22.817640 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:40:22.817544 4939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/78a38f10-6762-489e-813a-ac1cb91f5844-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/78a38f10-6762-489e-813a-ac1cb91f5844-credential-socket\") pod \"78a38f10-6762-489e-813a-ac1cb91f5844\" (UID: \"78a38f10-6762-489e-813a-ac1cb91f5844\") " Nov 03 01:40:22.817640 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:40:22.817593 4939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/78a38f10-6762-489e-813a-ac1cb91f5844-kube-api-access-zmx8s\" (UniqueName: \"kubernetes.io/projected/78a38f10-6762-489e-813a-ac1cb91f5844-kube-api-access-zmx8s\") pod \"78a38f10-6762-489e-813a-ac1cb91f5844\" (UID: \"78a38f10-6762-489e-813a-ac1cb91f5844\") " Nov 03 01:40:22.817719 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:40:22.817639 4939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/78a38f10-6762-489e-813a-ac1cb91f5844-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/78a38f10-6762-489e-813a-ac1cb91f5844-istio-podinfo\") pod \"78a38f10-6762-489e-813a-ac1cb91f5844\" (UID: \"78a38f10-6762-489e-813a-ac1cb91f5844\") " Nov 03 01:40:22.817719 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:40:22.817682 4939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/78a38f10-6762-489e-813a-ac1cb91f5844-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/78a38f10-6762-489e-813a-ac1cb91f5844-istio-envoy\") pod \"78a38f10-6762-489e-813a-ac1cb91f5844\" (UID: \"78a38f10-6762-489e-813a-ac1cb91f5844\") " Nov 03 01:40:22.817833 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:40:22.817774 4939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/78a38f10-6762-489e-813a-ac1cb91f5844-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/78a38f10-6762-489e-813a-ac1cb91f5844-workload-certs\") pod \"78a38f10-6762-489e-813a-ac1cb91f5844\" (UID: \"78a38f10-6762-489e-813a-ac1cb91f5844\") " Nov 03 01:40:22.817833 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:40:22.817815 4939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/78a38f10-6762-489e-813a-ac1cb91f5844-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/78a38f10-6762-489e-813a-ac1cb91f5844-workload-socket\") pod \"78a38f10-6762-489e-813a-ac1cb91f5844\" (UID: \"78a38f10-6762-489e-813a-ac1cb91f5844\") " Nov 03 01:40:22.817920 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:40:22.817868 4939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/78a38f10-6762-489e-813a-ac1cb91f5844-istio-token\" (UniqueName: \"kubernetes.io/projected/78a38f10-6762-489e-813a-ac1cb91f5844-istio-token\") pod \"78a38f10-6762-489e-813a-ac1cb91f5844\" (UID: \"78a38f10-6762-489e-813a-ac1cb91f5844\") " Nov 03 01:40:22.817964 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:40:22.817927 4939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/78a38f10-6762-489e-813a-ac1cb91f5844-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/78a38f10-6762-489e-813a-ac1cb91f5844-istio-data\") pod \"78a38f10-6762-489e-813a-ac1cb91f5844\" (UID: \"78a38f10-6762-489e-813a-ac1cb91f5844\") " Nov 03 01:40:22.818001 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:40:22.817950 4939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/78a38f10-6762-489e-813a-ac1cb91f5844-credential-socket" pod "78a38f10-6762-489e-813a-ac1cb91f5844" (UID: "78a38f10-6762-489e-813a-ac1cb91f5844"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 01:40:22.818136 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:40:22.818115 4939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/78a38f10-6762-489e-813a-ac1cb91f5844-workload-socket" pod "78a38f10-6762-489e-813a-ac1cb91f5844" (UID: "78a38f10-6762-489e-813a-ac1cb91f5844"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 01:40:22.818231 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:40:22.818204 4939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/78a38f10-6762-489e-813a-ac1cb91f5844-istiod-ca-cert" pod "78a38f10-6762-489e-813a-ac1cb91f5844" (UID: "78a38f10-6762-489e-813a-ac1cb91f5844"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 01:40:22.818479 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:40:22.818448 4939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/78a38f10-6762-489e-813a-ac1cb91f5844-istio-data" pod "78a38f10-6762-489e-813a-ac1cb91f5844" (UID: "78a38f10-6762-489e-813a-ac1cb91f5844"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 01:40:22.818572 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:40:22.818511 4939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/78a38f10-6762-489e-813a-ac1cb91f5844-workload-certs" pod "78a38f10-6762-489e-813a-ac1cb91f5844" (UID: "78a38f10-6762-489e-813a-ac1cb91f5844"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 01:40:22.818710 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:40:22.818687 4939 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/78a38f10-6762-489e-813a-ac1cb91f5844-istiod-ca-cert\") on node \"ip-10-0-140-107.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:40:22.818795 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:40:22.818714 4939 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/78a38f10-6762-489e-813a-ac1cb91f5844-credential-socket\") on node \"ip-10-0-140-107.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:40:22.818795 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:40:22.818724 4939 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/78a38f10-6762-489e-813a-ac1cb91f5844-workload-certs\") on node \"ip-10-0-140-107.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:40:22.818795 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:40:22.818733 4939 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/78a38f10-6762-489e-813a-ac1cb91f5844-workload-socket\") on node \"ip-10-0-140-107.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:40:22.818795 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:40:22.818741 4939 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/78a38f10-6762-489e-813a-ac1cb91f5844-istio-data\") on node \"ip-10-0-140-107.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:40:22.820692 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:40:22.820663 4939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/78a38f10-6762-489e-813a-ac1cb91f5844-istio-podinfo" pod "78a38f10-6762-489e-813a-ac1cb91f5844" (UID: "78a38f10-6762-489e-813a-ac1cb91f5844"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 01:40:22.820839 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:40:22.820817 4939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/78a38f10-6762-489e-813a-ac1cb91f5844-istio-envoy" pod "78a38f10-6762-489e-813a-ac1cb91f5844" (UID: "78a38f10-6762-489e-813a-ac1cb91f5844"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 01:40:22.821659 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:40:22.821638 4939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/78a38f10-6762-489e-813a-ac1cb91f5844-istio-token" pod "78a38f10-6762-489e-813a-ac1cb91f5844" (UID: "78a38f10-6762-489e-813a-ac1cb91f5844"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 01:40:22.821809 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:40:22.821784 4939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/78a38f10-6762-489e-813a-ac1cb91f5844-kube-api-access-zmx8s" pod "78a38f10-6762-489e-813a-ac1cb91f5844" (UID: "78a38f10-6762-489e-813a-ac1cb91f5844"). InnerVolumeSpecName "kube-api-access-zmx8s". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 01:40:22.899063 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:40:22.898772 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:40:22.924931 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:40:22.923009 4939 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zmx8s\" (UniqueName: \"kubernetes.io/projected/78a38f10-6762-489e-813a-ac1cb91f5844-kube-api-access-zmx8s\") on node \"ip-10-0-140-107.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:40:22.924931 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:40:22.923050 4939 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/78a38f10-6762-489e-813a-ac1cb91f5844-istio-podinfo\") on node \"ip-10-0-140-107.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:40:22.924931 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:40:22.923067 4939 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/78a38f10-6762-489e-813a-ac1cb91f5844-istio-envoy\") on node \"ip-10-0-140-107.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:40:22.924931 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:40:22.923080 4939 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/78a38f10-6762-489e-813a-ac1cb91f5844-istio-token\") on node \"ip-10-0-140-107.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:40:23.200985 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:40:23.196310 4939 patch_prober.go:29] interesting pod/e2e-shared-a-w4-6c0ec0-openshift-default-79c7584bfc-kxj54 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.86:15021/healthz/ready\": dial tcp 10.132.1.86:15021: connect: connection refused" start-of-body= Nov 03 01:40:23.200985 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:40:23.196381 4939 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-6c0ec0-openshift-default-79c7584bfc-kxj54" podUID="d1ad622d-53f3-48ca-bc88-b7ab4ae452fe" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.86:15021/healthz/ready\": dial tcp 10.132.1.86:15021: connect: connection refused" Nov 03 01:40:23.406838 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:40:23.406804 4939 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-3c5983-openshift-default-86c5bc855c-2hx8m" Nov 03 01:40:23.407135 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:40:23.407086 4939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-3c5983-openshift-default-86c5bc855c-2hx8m" event={"ID":"78a38f10-6762-489e-813a-ac1cb91f5844","Type":"ContainerDied","Data":"1ffede5791fcbca7c79ae803e5dea9dedf6b5f5bf9989978b6f7fcdfdd1a95e7"} Nov 03 01:40:23.408317 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:40:23.408294 4939 scope.go:122] "RemoveContainer" containerID="d25cdbfedd2c4039d8e796f62d8dd7d44fa46d1f7ff0d34c9badaa9eb7301a3d" Nov 03 01:40:23.426589 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:40:23.426565 4939 patch_prober.go:29] interesting pod/e2e-shared-a-w3-717217-openshift-default-578c69554c-mwwv6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.85:15021/healthz/ready\": dial tcp 10.132.1.85:15021: connect: connection refused" start-of-body= Nov 03 01:40:23.426695 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:40:23.426613 4939 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-717217-openshift-default-578c69554c-mwwv6" podUID="d0933771-7032-4cff-aafe-2f7211ebb549" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.85:15021/healthz/ready\": dial tcp 10.132.1.85:15021: connect: connection refused" Nov 03 01:40:23.438120 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:40:23.438098 4939 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-b-3c5983-openshift-default-86c5bc855c-2hx8m"] Nov 03 01:40:23.470460 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:40:23.470367 4939 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-extmod-b-3c5983-openshift-default-86c5bc855c-2hx8m"] Nov 03 01:40:23.676933 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:40:23.676016 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:40:23.906374 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:40:23.906338 4939 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="78a38f10-6762-489e-813a-ac1cb91f5844" path="/var/lib/kubelet/pods/78a38f10-6762-489e-813a-ac1cb91f5844/volumes" Nov 03 01:40:24.198274 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:40:24.196290 4939 patch_prober.go:29] interesting pod/e2e-shared-a-w4-6c0ec0-openshift-default-79c7584bfc-kxj54 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.86:15021/healthz/ready\": dial tcp 10.132.1.86:15021: connect: connection refused" start-of-body= Nov 03 01:40:24.198274 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:40:24.196349 4939 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-6c0ec0-openshift-default-79c7584bfc-kxj54" podUID="d1ad622d-53f3-48ca-bc88-b7ab4ae452fe" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.86:15021/healthz/ready\": dial tcp 10.132.1.86:15021: connect: connection refused" Nov 03 01:40:24.434956 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:40:24.432100 4939 patch_prober.go:29] interesting pod/e2e-shared-a-w3-717217-openshift-default-578c69554c-mwwv6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.85:15021/healthz/ready\": dial tcp 10.132.1.85:15021: connect: connection refused" start-of-body= Nov 03 01:40:24.434956 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:40:24.432182 4939 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-717217-openshift-default-578c69554c-mwwv6" podUID="d0933771-7032-4cff-aafe-2f7211ebb549" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.85:15021/healthz/ready\": dial tcp 10.132.1.85:15021: connect: connection refused" Nov 03 01:40:24.436417 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:40:24.436153 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:40:24.598951 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:40:24.597441 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:40:24.943212 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:40:24.942783 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:40:25.194706 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:40:25.194622 4939 patch_prober.go:29] interesting pod/e2e-shared-a-w4-6c0ec0-openshift-default-79c7584bfc-kxj54 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.86:15021/healthz/ready\": dial tcp 10.132.1.86:15021: connect: connection refused" start-of-body= Nov 03 01:40:25.194706 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:40:25.194674 4939 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-6c0ec0-openshift-default-79c7584bfc-kxj54" podUID="d1ad622d-53f3-48ca-bc88-b7ab4ae452fe" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.86:15021/healthz/ready\": dial tcp 10.132.1.86:15021: connect: connection refused" Nov 03 01:40:25.505771 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:40:25.505694 4939 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w3-717217-openshift-default-578c69554c-mwwv6" Nov 03 01:40:25.506407 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:40:25.506361 4939 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w3-717217-openshift-default-578c69554c-mwwv6" Nov 03 01:40:25.507492 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:40:25.507469 4939 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w3-717217-openshift-default-578c69554c-mwwv6" Nov 03 01:40:25.587993 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:40:25.587958 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:40:26.194947 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:40:26.194877 4939 patch_prober.go:29] interesting pod/e2e-shared-a-w4-6c0ec0-openshift-default-79c7584bfc-kxj54 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.86:15021/healthz/ready\": dial tcp 10.132.1.86:15021: connect: connection refused" start-of-body= Nov 03 01:40:26.195131 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:40:26.194971 4939 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-6c0ec0-openshift-default-79c7584bfc-kxj54" podUID="d1ad622d-53f3-48ca-bc88-b7ab4ae452fe" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.86:15021/healthz/ready\": dial tcp 10.132.1.86:15021: connect: connection refused" Nov 03 01:40:26.229540 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:40:26.229507 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:40:26.957652 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:40:26.957346 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:40:27.194873 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:40:27.194827 4939 patch_prober.go:29] interesting pod/e2e-shared-a-w4-6c0ec0-openshift-default-79c7584bfc-kxj54 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.86:15021/healthz/ready\": dial tcp 10.132.1.86:15021: connect: connection refused" start-of-body= Nov 03 01:40:27.195088 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:40:27.194925 4939 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-6c0ec0-openshift-default-79c7584bfc-kxj54" podUID="d1ad622d-53f3-48ca-bc88-b7ab4ae452fe" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.86:15021/healthz/ready\": dial tcp 10.132.1.86:15021: connect: connection refused" Nov 03 01:40:27.409144 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:40:27.409099 4939 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-extmod-b-3c5983-7fbf586cf5-92rtm" podUID="16b64414-6144-4743-bc96-91dffcd4a6b7" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 03 01:40:27.781903 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:40:27.781815 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:40:28.199276 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:40:28.199244 4939 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w4-6c0ec0-openshift-default-79c7584bfc-kxj54" Nov 03 01:40:28.199746 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:40:28.199653 4939 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w4-6c0ec0-openshift-default-79c7584bfc-kxj54" Nov 03 01:40:28.200313 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:40:28.200292 4939 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w4-6c0ec0-openshift-default-79c7584bfc-kxj54" Nov 03 01:40:28.475630 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:40:28.475601 4939 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-3c5983-564977cd99-qs5fk" Nov 03 01:40:28.497055 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:40:28.496997 4939 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-3c5983-564977cd99-qs5fk" Nov 03 01:40:28.497055 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:40:28.497002 4939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-3c5983-564977cd99-qs5fk" event={"ID":"3137d8da-32e8-47fd-87a6-f08aebe2a384","Type":"ContainerDied","Data":"aae7b6322990462714bb68ba10cf5baf2dc655cea6d5819f6d0a0a81bad6fed4"} Nov 03 01:40:28.497329 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:40:28.497081 4939 scope.go:122] "RemoveContainer" containerID="aae7b6322990462714bb68ba10cf5baf2dc655cea6d5819f6d0a0a81bad6fed4" Nov 03 01:40:28.497838 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:40:28.497576 4939 generic.go:356] "Generic (PLEG): container finished" podID="3137d8da-32e8-47fd-87a6-f08aebe2a384" containerID="aae7b6322990462714bb68ba10cf5baf2dc655cea6d5819f6d0a0a81bad6fed4" exitCode=0 Nov 03 01:40:28.497838 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:40:28.497738 4939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-3c5983-564977cd99-qs5fk" event={"ID":"3137d8da-32e8-47fd-87a6-f08aebe2a384","Type":"ContainerDied","Data":"e001b6889046d2fb5bbc3068b42115b7e1d69afe8c8cbe137943670683419f24"} Nov 03 01:40:28.500391 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:40:28.500364 4939 generic.go:356] "Generic (PLEG): container finished" podID="16b64414-6144-4743-bc96-91dffcd4a6b7" containerID="177c06e36a95a1b726d3dc4d6ca587944b9f13bdb65004220d6bc5f91c9c44d6" exitCode=0 Nov 03 01:40:28.501444 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:40:28.501416 4939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-b-3c5983-7fbf586cf5-92rtm" event={"ID":"16b64414-6144-4743-bc96-91dffcd4a6b7","Type":"ContainerDied","Data":"177c06e36a95a1b726d3dc4d6ca587944b9f13bdb65004220d6bc5f91c9c44d6"} Nov 03 01:40:28.523198 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:40:28.523170 4939 scope.go:122] "RemoveContainer" containerID="aae7b6322990462714bb68ba10cf5baf2dc655cea6d5819f6d0a0a81bad6fed4" Nov 03 01:40:28.523492 ip-10-0-140-107 kubenswrapper[4939]: E1103 01:40:28.523467 4939 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"aae7b6322990462714bb68ba10cf5baf2dc655cea6d5819f6d0a0a81bad6fed4\": container with ID starting with aae7b6322990462714bb68ba10cf5baf2dc655cea6d5819f6d0a0a81bad6fed4 not found: ID does not exist" containerID="aae7b6322990462714bb68ba10cf5baf2dc655cea6d5819f6d0a0a81bad6fed4" Nov 03 01:40:28.523597 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:40:28.523505 4939 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"aae7b6322990462714bb68ba10cf5baf2dc655cea6d5819f6d0a0a81bad6fed4"} err="failed to get container status \"aae7b6322990462714bb68ba10cf5baf2dc655cea6d5819f6d0a0a81bad6fed4\": rpc error: code = NotFound desc = could not find container \"aae7b6322990462714bb68ba10cf5baf2dc655cea6d5819f6d0a0a81bad6fed4\": container with ID starting with aae7b6322990462714bb68ba10cf5baf2dc655cea6d5819f6d0a0a81bad6fed4 not found: ID does not exist" Nov 03 01:40:28.524385 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:40:28.524362 4939 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-b-3c5983-7fbf586cf5-92rtm" Nov 03 01:40:28.558534 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:40:28.558501 4939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/16b64414-6144-4743-bc96-91dffcd4a6b7-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/16b64414-6144-4743-bc96-91dffcd4a6b7-plugins-config-volume\") pod \"16b64414-6144-4743-bc96-91dffcd4a6b7\" (UID: \"16b64414-6144-4743-bc96-91dffcd4a6b7\") " Nov 03 01:40:28.558732 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:40:28.558574 4939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3137d8da-32e8-47fd-87a6-f08aebe2a384-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3137d8da-32e8-47fd-87a6-f08aebe2a384-plugins-config-volume\") pod \"3137d8da-32e8-47fd-87a6-f08aebe2a384\" (UID: \"3137d8da-32e8-47fd-87a6-f08aebe2a384\") " Nov 03 01:40:28.558822 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:40:28.558745 4939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3137d8da-32e8-47fd-87a6-f08aebe2a384-kube-api-access-w9kx9\" (UniqueName: \"kubernetes.io/projected/3137d8da-32e8-47fd-87a6-f08aebe2a384-kube-api-access-w9kx9\") pod \"3137d8da-32e8-47fd-87a6-f08aebe2a384\" (UID: \"3137d8da-32e8-47fd-87a6-f08aebe2a384\") " Nov 03 01:40:28.559024 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:40:28.558899 4939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/16b64414-6144-4743-bc96-91dffcd4a6b7-kube-api-access-r2qwl\" (UniqueName: \"kubernetes.io/projected/16b64414-6144-4743-bc96-91dffcd4a6b7-kube-api-access-r2qwl\") pod \"16b64414-6144-4743-bc96-91dffcd4a6b7\" (UID: \"16b64414-6144-4743-bc96-91dffcd4a6b7\") " Nov 03 01:40:28.559457 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:40:28.559408 4939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/16b64414-6144-4743-bc96-91dffcd4a6b7-plugins-config-volume" pod "16b64414-6144-4743-bc96-91dffcd4a6b7" (UID: "16b64414-6144-4743-bc96-91dffcd4a6b7"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 01:40:28.559457 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:40:28.559410 4939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3137d8da-32e8-47fd-87a6-f08aebe2a384-plugins-config-volume" pod "3137d8da-32e8-47fd-87a6-f08aebe2a384" (UID: "3137d8da-32e8-47fd-87a6-f08aebe2a384"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 01:40:28.560080 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:40:28.560054 4939 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/16b64414-6144-4743-bc96-91dffcd4a6b7-plugins-config-volume\") on node \"ip-10-0-140-107.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:40:28.560196 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:40:28.560084 4939 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3137d8da-32e8-47fd-87a6-f08aebe2a384-plugins-config-volume\") on node \"ip-10-0-140-107.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:40:28.561808 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:40:28.561779 4939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3137d8da-32e8-47fd-87a6-f08aebe2a384-kube-api-access-w9kx9" pod "3137d8da-32e8-47fd-87a6-f08aebe2a384" (UID: "3137d8da-32e8-47fd-87a6-f08aebe2a384"). InnerVolumeSpecName "kube-api-access-w9kx9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 01:40:28.562638 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:40:28.562607 4939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/16b64414-6144-4743-bc96-91dffcd4a6b7-kube-api-access-r2qwl" pod "16b64414-6144-4743-bc96-91dffcd4a6b7" (UID: "16b64414-6144-4743-bc96-91dffcd4a6b7"). InnerVolumeSpecName "kube-api-access-r2qwl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 01:40:28.602116 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:40:28.602091 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:40:28.680274 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:40:28.680184 4939 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-w9kx9\" (UniqueName: \"kubernetes.io/projected/3137d8da-32e8-47fd-87a6-f08aebe2a384-kube-api-access-w9kx9\") on node \"ip-10-0-140-107.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:40:28.680554 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:40:28.680531 4939 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-r2qwl\" (UniqueName: \"kubernetes.io/projected/16b64414-6144-4743-bc96-91dffcd4a6b7-kube-api-access-r2qwl\") on node \"ip-10-0-140-107.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:40:28.802584 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:40:28.802543 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:40:28.822548 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:40:28.822517 4939 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-3c5983-564977cd99-qs5fk"] Nov 03 01:40:28.828583 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:40:28.828558 4939 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-3c5983-564977cd99-qs5fk"] Nov 03 01:40:29.212798 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:40:29.212736 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:40:29.439207 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:40:29.439105 4939 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-3c5983-564977cd99-qs5fk" podUID="3137d8da-32e8-47fd-87a6-f08aebe2a384" containerName="payload-pre-processing" probeResult="failure" output="timeout: failed to connect service \"10.132.0.216:50052\" within 1s: context deadline exceeded" Nov 03 01:40:29.439207 ip-10-0-140-107 kubenswrapper[4939]: W1103 01:40:29.439200 4939 logging.go:55] [core] [Channel #19704 SubChannel #19705]grpc: addrConn.createTransport failed to connect to {Addr: "10.132.0.216:50052", ServerName: "10.132.0.216:50052", BalancerAttributes: {"<%!p(pickfirstleaf.managedByPickfirstKeyType={})>": "<%!p(bool=true)>" }}. Err: connection error: desc = "transport: Error while dialing: dial tcp 10.132.0.216:50052: operation was canceled" Nov 03 01:40:29.511585 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:40:29.511496 4939 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-b-3c5983-7fbf586cf5-92rtm" Nov 03 01:40:29.511585 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:40:29.511525 4939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-b-3c5983-7fbf586cf5-92rtm" event={"ID":"16b64414-6144-4743-bc96-91dffcd4a6b7","Type":"ContainerDied","Data":"e671955e66bd811cf8e13de3486a2c54213b910412a59dc7b8507d4cc7ff39d8"} Nov 03 01:40:29.511585 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:40:29.511584 4939 scope.go:122] "RemoveContainer" containerID="177c06e36a95a1b726d3dc4d6ca587944b9f13bdb65004220d6bc5f91c9c44d6" Nov 03 01:40:29.540836 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:40:29.540803 4939 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-b-3c5983-7fbf586cf5-92rtm"] Nov 03 01:40:29.544812 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:40:29.544786 4939 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-b-3c5983-7fbf586cf5-92rtm"] Nov 03 01:40:29.873653 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:40:29.873618 4939 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="16b64414-6144-4743-bc96-91dffcd4a6b7" path="/var/lib/kubelet/pods/16b64414-6144-4743-bc96-91dffcd4a6b7/volumes" Nov 03 01:40:29.874169 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:40:29.874148 4939 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3137d8da-32e8-47fd-87a6-f08aebe2a384" path="/var/lib/kubelet/pods/3137d8da-32e8-47fd-87a6-f08aebe2a384/volumes" Nov 03 01:40:29.914257 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:40:29.914220 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:40:30.551999 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:40:30.551970 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:40:31.059730 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:40:31.059696 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:40:31.169962 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:40:31.169899 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:40:31.808342 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:40:31.808308 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:40:32.521625 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:40:32.521586 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:40:33.093923 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:40:33.093880 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:40:33.773047 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:40:33.773015 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:40:34.371065 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:40:34.371019 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:40:34.905659 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:40:34.905620 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:40:35.284322 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:40:35.284236 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:40:35.979634 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:40:35.979596 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:40:36.114719 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:40:36.114680 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:40:36.674090 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:40:36.674056 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:40:37.307593 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:40:37.307555 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:40:37.891686 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:40:37.891648 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:40:38.013897 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:40:38.013857 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:40:38.122160 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:40:38.122127 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:40:38.369421 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:40:38.369386 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:40:38.704644 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:40:38.704043 4939 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-531447-openshift-default-7995d46767-jpz2z"] Nov 03 01:40:38.704644 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:40:38.704279 4939 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w2-531447-openshift-default-7995d46767-jpz2z" podUID="b7603676-9654-4c7f-ba1c-8a6a9c7e7382" containerName="istio-proxy" containerID="cri-o://663e1f52536f8a8802570c3e93687487dac156a592342f44a73a689467d24874" gracePeriod=30 Nov 03 01:40:38.718111 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:40:38.716275 4939 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-531447-openshift-default-7995d46767-jcnlj"] Nov 03 01:40:38.718111 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:40:38.716607 4939 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w2-531447-openshift-default-7995d46767-jcnlj" podUID="aa4094a2-6349-4637-9630-40a4ee74ecce" containerName="istio-proxy" containerID="cri-o://2920ddcaec0d00e9f2de1c4222fe18cead32be1c73ee861d487bb3f6fc8daddb" gracePeriod=30 Nov 03 01:40:38.743535 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:40:38.743499 4939 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-531447-openshift-default-7995d46767-gfccz"] Nov 03 01:40:38.743760 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:40:38.743738 4939 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w2-531447-openshift-default-7995d46767-gfccz" podUID="fb776e00-2114-4f9c-86b1-bb790c3d64ca" containerName="istio-proxy" containerID="cri-o://c3b5e51c1b87ee07927c892b6fdd6c9e9b924b2402f13910c7bea18ce7e58644" gracePeriod=30 Nov 03 01:40:38.769491 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:40:38.769455 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:40:39.118737 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:40:39.118703 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:40:39.417411 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:40:39.417334 4939 patch_prober.go:29] interesting pod/e2e-shared-a-w2-531447-openshift-default-7995d46767-gfccz container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 03 01:40:39.417411 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:40:39.417400 4939 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-a-w2-531447-openshift-default-7995d46767-gfccz" podUID="fb776e00-2114-4f9c-86b1-bb790c3d64ca" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 03 01:40:39.513758 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:40:39.513727 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:40:39.801420 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:40:39.801334 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:40:40.199164 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:40:40.199125 4939 patch_prober.go:29] interesting pod/e2e-shared-a-w2-531447-openshift-default-7995d46767-jcnlj container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 03 01:40:40.199353 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:40:40.199186 4939 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-a-w2-531447-openshift-default-7995d46767-jcnlj" podUID="aa4094a2-6349-4637-9630-40a4ee74ecce" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 03 01:40:40.262792 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:40:40.262757 4939 patch_prober.go:29] interesting pod/e2e-shared-a-w2-531447-openshift-default-7995d46767-jpz2z container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 03 01:40:40.262972 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:40:40.262825 4939 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-a-w2-531447-openshift-default-7995d46767-jpz2z" podUID="b7603676-9654-4c7f-ba1c-8a6a9c7e7382" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 03 01:40:40.493100 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:40:40.493018 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:40:41.199011 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:40:41.198978 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:40:41.326082 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:40:41.326040 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:40:41.675401 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:40:41.675363 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:40:41.985799 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:40:41.985673 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:40:42.744891 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:40:42.744859 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:40:43.204362 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:40:43.204332 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:40:43.550017 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:40:43.549885 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:40:44.052825 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:40:44.052777 4939 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w2-531447-openshift-default-7995d46767-jcnlj" Nov 03 01:40:44.099066 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:40:44.099032 4939 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w2-531447-openshift-default-7995d46767-gfccz" Nov 03 01:40:44.151978 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:40:44.150587 4939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fb776e00-2114-4f9c-86b1-bb790c3d64ca-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fb776e00-2114-4f9c-86b1-bb790c3d64ca-workload-socket\") pod \"fb776e00-2114-4f9c-86b1-bb790c3d64ca\" (UID: \"fb776e00-2114-4f9c-86b1-bb790c3d64ca\") " Nov 03 01:40:44.151978 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:40:44.150623 4939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/aa4094a2-6349-4637-9630-40a4ee74ecce-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/aa4094a2-6349-4637-9630-40a4ee74ecce-istio-data\") pod \"aa4094a2-6349-4637-9630-40a4ee74ecce\" (UID: \"aa4094a2-6349-4637-9630-40a4ee74ecce\") " Nov 03 01:40:44.151978 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:40:44.150655 4939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fb776e00-2114-4f9c-86b1-bb790c3d64ca-kube-api-access-dqzqz\" (UniqueName: \"kubernetes.io/projected/fb776e00-2114-4f9c-86b1-bb790c3d64ca-kube-api-access-dqzqz\") pod \"fb776e00-2114-4f9c-86b1-bb790c3d64ca\" (UID: \"fb776e00-2114-4f9c-86b1-bb790c3d64ca\") " Nov 03 01:40:44.151978 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:40:44.150675 4939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fb776e00-2114-4f9c-86b1-bb790c3d64ca-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fb776e00-2114-4f9c-86b1-bb790c3d64ca-workload-certs\") pod \"fb776e00-2114-4f9c-86b1-bb790c3d64ca\" (UID: \"fb776e00-2114-4f9c-86b1-bb790c3d64ca\") " Nov 03 01:40:44.151978 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:40:44.150694 4939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/aa4094a2-6349-4637-9630-40a4ee74ecce-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/aa4094a2-6349-4637-9630-40a4ee74ecce-istio-podinfo\") pod \"aa4094a2-6349-4637-9630-40a4ee74ecce\" (UID: \"aa4094a2-6349-4637-9630-40a4ee74ecce\") " Nov 03 01:40:44.151978 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:40:44.150736 4939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/aa4094a2-6349-4637-9630-40a4ee74ecce-kube-api-access-kp9kt\" (UniqueName: \"kubernetes.io/projected/aa4094a2-6349-4637-9630-40a4ee74ecce-kube-api-access-kp9kt\") pod \"aa4094a2-6349-4637-9630-40a4ee74ecce\" (UID: \"aa4094a2-6349-4637-9630-40a4ee74ecce\") " Nov 03 01:40:44.151978 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:40:44.150753 4939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/aa4094a2-6349-4637-9630-40a4ee74ecce-istio-token\" (UniqueName: \"kubernetes.io/projected/aa4094a2-6349-4637-9630-40a4ee74ecce-istio-token\") pod \"aa4094a2-6349-4637-9630-40a4ee74ecce\" (UID: \"aa4094a2-6349-4637-9630-40a4ee74ecce\") " Nov 03 01:40:44.151978 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:40:44.150782 4939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fb776e00-2114-4f9c-86b1-bb790c3d64ca-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fb776e00-2114-4f9c-86b1-bb790c3d64ca-istio-envoy\") pod \"fb776e00-2114-4f9c-86b1-bb790c3d64ca\" (UID: \"fb776e00-2114-4f9c-86b1-bb790c3d64ca\") " Nov 03 01:40:44.151978 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:40:44.150813 4939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fb776e00-2114-4f9c-86b1-bb790c3d64ca-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fb776e00-2114-4f9c-86b1-bb790c3d64ca-istiod-ca-cert\") pod \"fb776e00-2114-4f9c-86b1-bb790c3d64ca\" (UID: \"fb776e00-2114-4f9c-86b1-bb790c3d64ca\") " Nov 03 01:40:44.151978 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:40:44.150859 4939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/aa4094a2-6349-4637-9630-40a4ee74ecce-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/aa4094a2-6349-4637-9630-40a4ee74ecce-workload-socket\") pod \"aa4094a2-6349-4637-9630-40a4ee74ecce\" (UID: \"aa4094a2-6349-4637-9630-40a4ee74ecce\") " Nov 03 01:40:44.151978 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:40:44.150899 4939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/aa4094a2-6349-4637-9630-40a4ee74ecce-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/aa4094a2-6349-4637-9630-40a4ee74ecce-istio-envoy\") pod \"aa4094a2-6349-4637-9630-40a4ee74ecce\" (UID: \"aa4094a2-6349-4637-9630-40a4ee74ecce\") " Nov 03 01:40:44.151978 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:40:44.150949 4939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fb776e00-2114-4f9c-86b1-bb790c3d64ca-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fb776e00-2114-4f9c-86b1-bb790c3d64ca-istio-data\") pod \"fb776e00-2114-4f9c-86b1-bb790c3d64ca\" (UID: \"fb776e00-2114-4f9c-86b1-bb790c3d64ca\") " Nov 03 01:40:44.151978 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:40:44.151017 4939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/aa4094a2-6349-4637-9630-40a4ee74ecce-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/aa4094a2-6349-4637-9630-40a4ee74ecce-credential-socket\") pod \"aa4094a2-6349-4637-9630-40a4ee74ecce\" (UID: \"aa4094a2-6349-4637-9630-40a4ee74ecce\") " Nov 03 01:40:44.151978 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:40:44.151043 4939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fb776e00-2114-4f9c-86b1-bb790c3d64ca-istio-token\" (UniqueName: \"kubernetes.io/projected/fb776e00-2114-4f9c-86b1-bb790c3d64ca-istio-token\") pod \"fb776e00-2114-4f9c-86b1-bb790c3d64ca\" (UID: \"fb776e00-2114-4f9c-86b1-bb790c3d64ca\") " Nov 03 01:40:44.151978 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:40:44.151123 4939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/aa4094a2-6349-4637-9630-40a4ee74ecce-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/aa4094a2-6349-4637-9630-40a4ee74ecce-istiod-ca-cert\") pod \"aa4094a2-6349-4637-9630-40a4ee74ecce\" (UID: \"aa4094a2-6349-4637-9630-40a4ee74ecce\") " Nov 03 01:40:44.151978 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:40:44.151154 4939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/fb776e00-2114-4f9c-86b1-bb790c3d64ca-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fb776e00-2114-4f9c-86b1-bb790c3d64ca-istio-podinfo\") pod \"fb776e00-2114-4f9c-86b1-bb790c3d64ca\" (UID: \"fb776e00-2114-4f9c-86b1-bb790c3d64ca\") " Nov 03 01:40:44.151978 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:40:44.151181 4939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/aa4094a2-6349-4637-9630-40a4ee74ecce-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/aa4094a2-6349-4637-9630-40a4ee74ecce-workload-certs\") pod \"aa4094a2-6349-4637-9630-40a4ee74ecce\" (UID: \"aa4094a2-6349-4637-9630-40a4ee74ecce\") " Nov 03 01:40:44.151978 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:40:44.151241 4939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fb776e00-2114-4f9c-86b1-bb790c3d64ca-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fb776e00-2114-4f9c-86b1-bb790c3d64ca-credential-socket\") pod \"fb776e00-2114-4f9c-86b1-bb790c3d64ca\" (UID: \"fb776e00-2114-4f9c-86b1-bb790c3d64ca\") " Nov 03 01:40:44.153703 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:40:44.151995 4939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fb776e00-2114-4f9c-86b1-bb790c3d64ca-credential-socket" pod "fb776e00-2114-4f9c-86b1-bb790c3d64ca" (UID: "fb776e00-2114-4f9c-86b1-bb790c3d64ca"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 01:40:44.153703 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:40:44.152245 4939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fb776e00-2114-4f9c-86b1-bb790c3d64ca-workload-socket" pod "fb776e00-2114-4f9c-86b1-bb790c3d64ca" (UID: "fb776e00-2114-4f9c-86b1-bb790c3d64ca"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 01:40:44.153703 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:40:44.152444 4939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/aa4094a2-6349-4637-9630-40a4ee74ecce-istio-data" pod "aa4094a2-6349-4637-9630-40a4ee74ecce" (UID: "aa4094a2-6349-4637-9630-40a4ee74ecce"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 01:40:44.153703 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:40:44.152894 4939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/aa4094a2-6349-4637-9630-40a4ee74ecce-workload-socket" pod "aa4094a2-6349-4637-9630-40a4ee74ecce" (UID: "aa4094a2-6349-4637-9630-40a4ee74ecce"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 01:40:44.153703 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:40:44.153127 4939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fb776e00-2114-4f9c-86b1-bb790c3d64ca-workload-certs" pod "fb776e00-2114-4f9c-86b1-bb790c3d64ca" (UID: "fb776e00-2114-4f9c-86b1-bb790c3d64ca"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 01:40:44.153703 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:40:44.153260 4939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fb776e00-2114-4f9c-86b1-bb790c3d64ca-istio-data" pod "fb776e00-2114-4f9c-86b1-bb790c3d64ca" (UID: "fb776e00-2114-4f9c-86b1-bb790c3d64ca"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 01:40:44.153703 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:40:44.153368 4939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/aa4094a2-6349-4637-9630-40a4ee74ecce-istiod-ca-cert" pod "aa4094a2-6349-4637-9630-40a4ee74ecce" (UID: "aa4094a2-6349-4637-9630-40a4ee74ecce"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 01:40:44.154739 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:40:44.154527 4939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/aa4094a2-6349-4637-9630-40a4ee74ecce-credential-socket" pod "aa4094a2-6349-4637-9630-40a4ee74ecce" (UID: "aa4094a2-6349-4637-9630-40a4ee74ecce"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 01:40:44.155115 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:40:44.154927 4939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fb776e00-2114-4f9c-86b1-bb790c3d64ca-kube-api-access-dqzqz" pod "fb776e00-2114-4f9c-86b1-bb790c3d64ca" (UID: "fb776e00-2114-4f9c-86b1-bb790c3d64ca"). InnerVolumeSpecName "kube-api-access-dqzqz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 01:40:44.155358 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:40:44.155263 4939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fb776e00-2114-4f9c-86b1-bb790c3d64ca-istiod-ca-cert" pod "fb776e00-2114-4f9c-86b1-bb790c3d64ca" (UID: "fb776e00-2114-4f9c-86b1-bb790c3d64ca"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 01:40:44.155358 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:40:44.155324 4939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/aa4094a2-6349-4637-9630-40a4ee74ecce-workload-certs" pod "aa4094a2-6349-4637-9630-40a4ee74ecce" (UID: "aa4094a2-6349-4637-9630-40a4ee74ecce"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 01:40:44.155694 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:40:44.155667 4939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/aa4094a2-6349-4637-9630-40a4ee74ecce-istio-podinfo" pod "aa4094a2-6349-4637-9630-40a4ee74ecce" (UID: "aa4094a2-6349-4637-9630-40a4ee74ecce"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 01:40:44.158089 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:40:44.158045 4939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/aa4094a2-6349-4637-9630-40a4ee74ecce-istio-token" pod "aa4094a2-6349-4637-9630-40a4ee74ecce" (UID: "aa4094a2-6349-4637-9630-40a4ee74ecce"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 01:40:44.159809 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:40:44.159747 4939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fb776e00-2114-4f9c-86b1-bb790c3d64ca-istio-envoy" pod "fb776e00-2114-4f9c-86b1-bb790c3d64ca" (UID: "fb776e00-2114-4f9c-86b1-bb790c3d64ca"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 01:40:44.159809 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:40:44.159777 4939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/aa4094a2-6349-4637-9630-40a4ee74ecce-kube-api-access-kp9kt" pod "aa4094a2-6349-4637-9630-40a4ee74ecce" (UID: "aa4094a2-6349-4637-9630-40a4ee74ecce"). InnerVolumeSpecName "kube-api-access-kp9kt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 01:40:44.160059 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:40:44.160035 4939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/fb776e00-2114-4f9c-86b1-bb790c3d64ca-istio-podinfo" pod "fb776e00-2114-4f9c-86b1-bb790c3d64ca" (UID: "fb776e00-2114-4f9c-86b1-bb790c3d64ca"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 01:40:44.160707 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:40:44.160688 4939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fb776e00-2114-4f9c-86b1-bb790c3d64ca-istio-token" pod "fb776e00-2114-4f9c-86b1-bb790c3d64ca" (UID: "fb776e00-2114-4f9c-86b1-bb790c3d64ca"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 01:40:44.161147 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:40:44.161126 4939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/aa4094a2-6349-4637-9630-40a4ee74ecce-istio-envoy" pod "aa4094a2-6349-4637-9630-40a4ee74ecce" (UID: "aa4094a2-6349-4637-9630-40a4ee74ecce"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 01:40:44.254007 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:40:44.253959 4939 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fb776e00-2114-4f9c-86b1-bb790c3d64ca-workload-socket\") on node \"ip-10-0-140-107.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:40:44.254007 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:40:44.254003 4939 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/aa4094a2-6349-4637-9630-40a4ee74ecce-istio-data\") on node \"ip-10-0-140-107.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:40:44.254280 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:40:44.254020 4939 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dqzqz\" (UniqueName: \"kubernetes.io/projected/fb776e00-2114-4f9c-86b1-bb790c3d64ca-kube-api-access-dqzqz\") on node \"ip-10-0-140-107.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:40:44.254280 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:40:44.254036 4939 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fb776e00-2114-4f9c-86b1-bb790c3d64ca-workload-certs\") on node \"ip-10-0-140-107.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:40:44.254280 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:40:44.254050 4939 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/aa4094a2-6349-4637-9630-40a4ee74ecce-istio-podinfo\") on node \"ip-10-0-140-107.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:40:44.254280 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:40:44.254064 4939 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kp9kt\" (UniqueName: \"kubernetes.io/projected/aa4094a2-6349-4637-9630-40a4ee74ecce-kube-api-access-kp9kt\") on node \"ip-10-0-140-107.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:40:44.254280 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:40:44.254079 4939 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/aa4094a2-6349-4637-9630-40a4ee74ecce-istio-token\") on node \"ip-10-0-140-107.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:40:44.254280 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:40:44.254094 4939 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fb776e00-2114-4f9c-86b1-bb790c3d64ca-istio-envoy\") on node \"ip-10-0-140-107.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:40:44.254280 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:40:44.254108 4939 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fb776e00-2114-4f9c-86b1-bb790c3d64ca-istiod-ca-cert\") on node \"ip-10-0-140-107.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:40:44.254280 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:40:44.254124 4939 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/aa4094a2-6349-4637-9630-40a4ee74ecce-workload-socket\") on node \"ip-10-0-140-107.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:40:44.254280 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:40:44.254138 4939 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/aa4094a2-6349-4637-9630-40a4ee74ecce-istio-envoy\") on node \"ip-10-0-140-107.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:40:44.254280 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:40:44.254152 4939 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fb776e00-2114-4f9c-86b1-bb790c3d64ca-istio-data\") on node \"ip-10-0-140-107.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:40:44.254280 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:40:44.254166 4939 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/aa4094a2-6349-4637-9630-40a4ee74ecce-credential-socket\") on node \"ip-10-0-140-107.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:40:44.254280 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:40:44.254180 4939 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fb776e00-2114-4f9c-86b1-bb790c3d64ca-istio-token\") on node \"ip-10-0-140-107.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:40:44.254280 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:40:44.254194 4939 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/aa4094a2-6349-4637-9630-40a4ee74ecce-istiod-ca-cert\") on node \"ip-10-0-140-107.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:40:44.254280 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:40:44.254208 4939 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fb776e00-2114-4f9c-86b1-bb790c3d64ca-istio-podinfo\") on node \"ip-10-0-140-107.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:40:44.254280 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:40:44.254222 4939 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/aa4094a2-6349-4637-9630-40a4ee74ecce-workload-certs\") on node \"ip-10-0-140-107.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:40:44.254280 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:40:44.254236 4939 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fb776e00-2114-4f9c-86b1-bb790c3d64ca-credential-socket\") on node \"ip-10-0-140-107.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:40:44.296146 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:40:44.293791 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:40:44.406379 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:40:44.406316 4939 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w2-531447-openshift-default-7995d46767-jpz2z" Nov 03 01:40:44.468808 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:40:44.468772 4939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b7603676-9654-4c7f-ba1c-8a6a9c7e7382-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b7603676-9654-4c7f-ba1c-8a6a9c7e7382-workload-certs\") pod \"b7603676-9654-4c7f-ba1c-8a6a9c7e7382\" (UID: \"b7603676-9654-4c7f-ba1c-8a6a9c7e7382\") " Nov 03 01:40:44.469021 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:40:44.468819 4939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b7603676-9654-4c7f-ba1c-8a6a9c7e7382-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b7603676-9654-4c7f-ba1c-8a6a9c7e7382-workload-socket\") pod \"b7603676-9654-4c7f-ba1c-8a6a9c7e7382\" (UID: \"b7603676-9654-4c7f-ba1c-8a6a9c7e7382\") " Nov 03 01:40:44.469021 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:40:44.468868 4939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b7603676-9654-4c7f-ba1c-8a6a9c7e7382-istio-token\" (UniqueName: \"kubernetes.io/projected/b7603676-9654-4c7f-ba1c-8a6a9c7e7382-istio-token\") pod \"b7603676-9654-4c7f-ba1c-8a6a9c7e7382\" (UID: \"b7603676-9654-4c7f-ba1c-8a6a9c7e7382\") " Nov 03 01:40:44.469021 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:40:44.468899 4939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b7603676-9654-4c7f-ba1c-8a6a9c7e7382-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b7603676-9654-4c7f-ba1c-8a6a9c7e7382-istiod-ca-cert\") pod \"b7603676-9654-4c7f-ba1c-8a6a9c7e7382\" (UID: \"b7603676-9654-4c7f-ba1c-8a6a9c7e7382\") " Nov 03 01:40:44.469206 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:40:44.469024 4939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b7603676-9654-4c7f-ba1c-8a6a9c7e7382-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b7603676-9654-4c7f-ba1c-8a6a9c7e7382-credential-socket\") pod \"b7603676-9654-4c7f-ba1c-8a6a9c7e7382\" (UID: \"b7603676-9654-4c7f-ba1c-8a6a9c7e7382\") " Nov 03 01:40:44.469206 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:40:44.469089 4939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b7603676-9654-4c7f-ba1c-8a6a9c7e7382-workload-certs" pod "b7603676-9654-4c7f-ba1c-8a6a9c7e7382" (UID: "b7603676-9654-4c7f-ba1c-8a6a9c7e7382"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 01:40:44.469206 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:40:44.469128 4939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b7603676-9654-4c7f-ba1c-8a6a9c7e7382-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b7603676-9654-4c7f-ba1c-8a6a9c7e7382-istio-envoy\") pod \"b7603676-9654-4c7f-ba1c-8a6a9c7e7382\" (UID: \"b7603676-9654-4c7f-ba1c-8a6a9c7e7382\") " Nov 03 01:40:44.469374 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:40:44.469198 4939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b7603676-9654-4c7f-ba1c-8a6a9c7e7382-workload-socket" pod "b7603676-9654-4c7f-ba1c-8a6a9c7e7382" (UID: "b7603676-9654-4c7f-ba1c-8a6a9c7e7382"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 01:40:44.469374 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:40:44.469220 4939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/b7603676-9654-4c7f-ba1c-8a6a9c7e7382-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b7603676-9654-4c7f-ba1c-8a6a9c7e7382-istio-podinfo\") pod \"b7603676-9654-4c7f-ba1c-8a6a9c7e7382\" (UID: \"b7603676-9654-4c7f-ba1c-8a6a9c7e7382\") " Nov 03 01:40:44.469374 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:40:44.469317 4939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b7603676-9654-4c7f-ba1c-8a6a9c7e7382-kube-api-access-7z8rk\" (UniqueName: \"kubernetes.io/projected/b7603676-9654-4c7f-ba1c-8a6a9c7e7382-kube-api-access-7z8rk\") pod \"b7603676-9654-4c7f-ba1c-8a6a9c7e7382\" (UID: \"b7603676-9654-4c7f-ba1c-8a6a9c7e7382\") " Nov 03 01:40:44.469544 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:40:44.469405 4939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b7603676-9654-4c7f-ba1c-8a6a9c7e7382-istiod-ca-cert" pod "b7603676-9654-4c7f-ba1c-8a6a9c7e7382" (UID: "b7603676-9654-4c7f-ba1c-8a6a9c7e7382"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 01:40:44.469544 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:40:44.469435 4939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b7603676-9654-4c7f-ba1c-8a6a9c7e7382-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b7603676-9654-4c7f-ba1c-8a6a9c7e7382-istio-data\") pod \"b7603676-9654-4c7f-ba1c-8a6a9c7e7382\" (UID: \"b7603676-9654-4c7f-ba1c-8a6a9c7e7382\") " Nov 03 01:40:44.469737 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:40:44.469711 4939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b7603676-9654-4c7f-ba1c-8a6a9c7e7382-credential-socket" pod "b7603676-9654-4c7f-ba1c-8a6a9c7e7382" (UID: "b7603676-9654-4c7f-ba1c-8a6a9c7e7382"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 01:40:44.469807 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:40:44.469750 4939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b7603676-9654-4c7f-ba1c-8a6a9c7e7382-istio-data" pod "b7603676-9654-4c7f-ba1c-8a6a9c7e7382" (UID: "b7603676-9654-4c7f-ba1c-8a6a9c7e7382"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 01:40:44.470214 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:40:44.470184 4939 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b7603676-9654-4c7f-ba1c-8a6a9c7e7382-istio-data\") on node \"ip-10-0-140-107.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:40:44.470347 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:40:44.470216 4939 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b7603676-9654-4c7f-ba1c-8a6a9c7e7382-workload-certs\") on node \"ip-10-0-140-107.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:40:44.470347 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:40:44.470234 4939 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b7603676-9654-4c7f-ba1c-8a6a9c7e7382-workload-socket\") on node \"ip-10-0-140-107.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:40:44.470347 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:40:44.470249 4939 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b7603676-9654-4c7f-ba1c-8a6a9c7e7382-istiod-ca-cert\") on node \"ip-10-0-140-107.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:40:44.470347 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:40:44.470264 4939 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b7603676-9654-4c7f-ba1c-8a6a9c7e7382-credential-socket\") on node \"ip-10-0-140-107.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:40:44.471983 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:40:44.471950 4939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/b7603676-9654-4c7f-ba1c-8a6a9c7e7382-istio-podinfo" pod "b7603676-9654-4c7f-ba1c-8a6a9c7e7382" (UID: "b7603676-9654-4c7f-ba1c-8a6a9c7e7382"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 01:40:44.472081 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:40:44.472061 4939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b7603676-9654-4c7f-ba1c-8a6a9c7e7382-kube-api-access-7z8rk" pod "b7603676-9654-4c7f-ba1c-8a6a9c7e7382" (UID: "b7603676-9654-4c7f-ba1c-8a6a9c7e7382"). InnerVolumeSpecName "kube-api-access-7z8rk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 01:40:44.472127 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:40:44.472101 4939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b7603676-9654-4c7f-ba1c-8a6a9c7e7382-istio-token" pod "b7603676-9654-4c7f-ba1c-8a6a9c7e7382" (UID: "b7603676-9654-4c7f-ba1c-8a6a9c7e7382"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 01:40:44.473200 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:40:44.473175 4939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b7603676-9654-4c7f-ba1c-8a6a9c7e7382-istio-envoy" pod "b7603676-9654-4c7f-ba1c-8a6a9c7e7382" (UID: "b7603676-9654-4c7f-ba1c-8a6a9c7e7382"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 01:40:44.572010 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:40:44.571975 4939 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b7603676-9654-4c7f-ba1c-8a6a9c7e7382-istio-podinfo\") on node \"ip-10-0-140-107.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:40:44.572010 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:40:44.572006 4939 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7z8rk\" (UniqueName: \"kubernetes.io/projected/b7603676-9654-4c7f-ba1c-8a6a9c7e7382-kube-api-access-7z8rk\") on node \"ip-10-0-140-107.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:40:44.572226 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:40:44.572024 4939 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b7603676-9654-4c7f-ba1c-8a6a9c7e7382-istio-token\") on node \"ip-10-0-140-107.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:40:44.572226 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:40:44.572037 4939 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b7603676-9654-4c7f-ba1c-8a6a9c7e7382-istio-envoy\") on node \"ip-10-0-140-107.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:40:44.652698 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:40:44.652662 4939 generic.go:356] "Generic (PLEG): container finished" podID="aa4094a2-6349-4637-9630-40a4ee74ecce" containerID="2920ddcaec0d00e9f2de1c4222fe18cead32be1c73ee861d487bb3f6fc8daddb" exitCode=0 Nov 03 01:40:44.652897 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:40:44.652716 4939 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w2-531447-openshift-default-7995d46767-jcnlj" Nov 03 01:40:44.652897 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:40:44.652747 4939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-531447-openshift-default-7995d46767-jcnlj" event={"ID":"aa4094a2-6349-4637-9630-40a4ee74ecce","Type":"ContainerDied","Data":"2920ddcaec0d00e9f2de1c4222fe18cead32be1c73ee861d487bb3f6fc8daddb"} Nov 03 01:40:44.652897 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:40:44.652814 4939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-531447-openshift-default-7995d46767-jcnlj" event={"ID":"aa4094a2-6349-4637-9630-40a4ee74ecce","Type":"ContainerDied","Data":"39dcda2883b0b8c244b2046274b295e5db4fb1bfd9d169e480653a5679dfeab2"} Nov 03 01:40:44.652897 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:40:44.652844 4939 scope.go:122] "RemoveContainer" containerID="2920ddcaec0d00e9f2de1c4222fe18cead32be1c73ee861d487bb3f6fc8daddb" Nov 03 01:40:44.654388 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:40:44.654362 4939 generic.go:356] "Generic (PLEG): container finished" podID="b7603676-9654-4c7f-ba1c-8a6a9c7e7382" containerID="663e1f52536f8a8802570c3e93687487dac156a592342f44a73a689467d24874" exitCode=0 Nov 03 01:40:44.654514 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:40:44.654421 4939 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w2-531447-openshift-default-7995d46767-jpz2z" Nov 03 01:40:44.654514 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:40:44.654419 4939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-531447-openshift-default-7995d46767-jpz2z" event={"ID":"b7603676-9654-4c7f-ba1c-8a6a9c7e7382","Type":"ContainerDied","Data":"663e1f52536f8a8802570c3e93687487dac156a592342f44a73a689467d24874"} Nov 03 01:40:44.654514 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:40:44.654479 4939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-531447-openshift-default-7995d46767-jpz2z" event={"ID":"b7603676-9654-4c7f-ba1c-8a6a9c7e7382","Type":"ContainerDied","Data":"7f0397dc14202abf8f2e0962fdab8b7cc159e373b5955c0215adde1dbd895873"} Nov 03 01:40:44.656229 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:40:44.656195 4939 generic.go:356] "Generic (PLEG): container finished" podID="fb776e00-2114-4f9c-86b1-bb790c3d64ca" containerID="c3b5e51c1b87ee07927c892b6fdd6c9e9b924b2402f13910c7bea18ce7e58644" exitCode=0 Nov 03 01:40:44.656339 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:40:44.656265 4939 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w2-531447-openshift-default-7995d46767-gfccz" Nov 03 01:40:44.656475 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:40:44.656425 4939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-531447-openshift-default-7995d46767-gfccz" event={"ID":"fb776e00-2114-4f9c-86b1-bb790c3d64ca","Type":"ContainerDied","Data":"c3b5e51c1b87ee07927c892b6fdd6c9e9b924b2402f13910c7bea18ce7e58644"} Nov 03 01:40:44.656475 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:40:44.656469 4939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-531447-openshift-default-7995d46767-gfccz" event={"ID":"fb776e00-2114-4f9c-86b1-bb790c3d64ca","Type":"ContainerDied","Data":"8eabfa514ab17e69492abb6a64c4066d7e6dba422e4376e70981a0e8ea50973f"} Nov 03 01:40:44.665584 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:40:44.665567 4939 scope.go:122] "RemoveContainer" containerID="2920ddcaec0d00e9f2de1c4222fe18cead32be1c73ee861d487bb3f6fc8daddb" Nov 03 01:40:44.666035 ip-10-0-140-107 kubenswrapper[4939]: E1103 01:40:44.665864 4939 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2920ddcaec0d00e9f2de1c4222fe18cead32be1c73ee861d487bb3f6fc8daddb\": container with ID starting with 2920ddcaec0d00e9f2de1c4222fe18cead32be1c73ee861d487bb3f6fc8daddb not found: ID does not exist" containerID="2920ddcaec0d00e9f2de1c4222fe18cead32be1c73ee861d487bb3f6fc8daddb" Nov 03 01:40:44.666035 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:40:44.665885 4939 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2920ddcaec0d00e9f2de1c4222fe18cead32be1c73ee861d487bb3f6fc8daddb"} err="failed to get container status \"2920ddcaec0d00e9f2de1c4222fe18cead32be1c73ee861d487bb3f6fc8daddb\": rpc error: code = NotFound desc = could not find container \"2920ddcaec0d00e9f2de1c4222fe18cead32be1c73ee861d487bb3f6fc8daddb\": container with ID starting with 2920ddcaec0d00e9f2de1c4222fe18cead32be1c73ee861d487bb3f6fc8daddb not found: ID does not exist" Nov 03 01:40:44.666035 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:40:44.665901 4939 scope.go:122] "RemoveContainer" containerID="663e1f52536f8a8802570c3e93687487dac156a592342f44a73a689467d24874" Nov 03 01:40:44.676728 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:40:44.676711 4939 scope.go:122] "RemoveContainer" containerID="663e1f52536f8a8802570c3e93687487dac156a592342f44a73a689467d24874" Nov 03 01:40:44.676979 ip-10-0-140-107 kubenswrapper[4939]: E1103 01:40:44.676961 4939 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"663e1f52536f8a8802570c3e93687487dac156a592342f44a73a689467d24874\": container with ID starting with 663e1f52536f8a8802570c3e93687487dac156a592342f44a73a689467d24874 not found: ID does not exist" containerID="663e1f52536f8a8802570c3e93687487dac156a592342f44a73a689467d24874" Nov 03 01:40:44.677038 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:40:44.676983 4939 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"663e1f52536f8a8802570c3e93687487dac156a592342f44a73a689467d24874"} err="failed to get container status \"663e1f52536f8a8802570c3e93687487dac156a592342f44a73a689467d24874\": rpc error: code = NotFound desc = could not find container \"663e1f52536f8a8802570c3e93687487dac156a592342f44a73a689467d24874\": container with ID starting with 663e1f52536f8a8802570c3e93687487dac156a592342f44a73a689467d24874 not found: ID does not exist" Nov 03 01:40:44.677038 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:40:44.677000 4939 scope.go:122] "RemoveContainer" containerID="c3b5e51c1b87ee07927c892b6fdd6c9e9b924b2402f13910c7bea18ce7e58644" Nov 03 01:40:44.687375 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:40:44.687312 4939 scope.go:122] "RemoveContainer" containerID="c3b5e51c1b87ee07927c892b6fdd6c9e9b924b2402f13910c7bea18ce7e58644" Nov 03 01:40:44.687709 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:40:44.687689 4939 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-531447-openshift-default-7995d46767-jcnlj"] Nov 03 01:40:44.687812 ip-10-0-140-107 kubenswrapper[4939]: E1103 01:40:44.687712 4939 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c3b5e51c1b87ee07927c892b6fdd6c9e9b924b2402f13910c7bea18ce7e58644\": container with ID starting with c3b5e51c1b87ee07927c892b6fdd6c9e9b924b2402f13910c7bea18ce7e58644 not found: ID does not exist" containerID="c3b5e51c1b87ee07927c892b6fdd6c9e9b924b2402f13910c7bea18ce7e58644" Nov 03 01:40:44.687812 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:40:44.687740 4939 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c3b5e51c1b87ee07927c892b6fdd6c9e9b924b2402f13910c7bea18ce7e58644"} err="failed to get container status \"c3b5e51c1b87ee07927c892b6fdd6c9e9b924b2402f13910c7bea18ce7e58644\": rpc error: code = NotFound desc = could not find container \"c3b5e51c1b87ee07927c892b6fdd6c9e9b924b2402f13910c7bea18ce7e58644\": container with ID starting with c3b5e51c1b87ee07927c892b6fdd6c9e9b924b2402f13910c7bea18ce7e58644 not found: ID does not exist" Nov 03 01:40:44.691224 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:40:44.691200 4939 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-531447-openshift-default-7995d46767-jcnlj"] Nov 03 01:40:44.700904 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:40:44.700882 4939 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-531447-openshift-default-7995d46767-jpz2z"] Nov 03 01:40:44.707542 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:40:44.707519 4939 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-531447-openshift-default-7995d46767-jpz2z"] Nov 03 01:40:44.717853 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:40:44.717835 4939 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-531447-openshift-default-7995d46767-gfccz"] Nov 03 01:40:44.721809 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:40:44.721787 4939 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-531447-openshift-default-7995d46767-gfccz"] Nov 03 01:40:45.006412 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:40:45.006311 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:40:45.305344 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:40:45.305257 4939 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w2-ef434a-7b64fdc54f-9zxp5"] Nov 03 01:40:45.313364 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:40:45.313332 4939 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-ef434a-845d4b6b99-nqmzg"] Nov 03 01:40:45.391729 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:40:45.391693 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:40:45.504245 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:40:45.504206 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:40:45.869589 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:40:45.869555 4939 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="aa4094a2-6349-4637-9630-40a4ee74ecce" path="/var/lib/kubelet/pods/aa4094a2-6349-4637-9630-40a4ee74ecce/volumes" Nov 03 01:40:45.870236 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:40:45.870212 4939 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b7603676-9654-4c7f-ba1c-8a6a9c7e7382" path="/var/lib/kubelet/pods/b7603676-9654-4c7f-ba1c-8a6a9c7e7382/volumes" Nov 03 01:40:45.870787 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:40:45.870768 4939 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fb776e00-2114-4f9c-86b1-bb790c3d64ca" path="/var/lib/kubelet/pods/fb776e00-2114-4f9c-86b1-bb790c3d64ca/volumes" Nov 03 01:40:45.955342 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:40:45.955310 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:40:46.077814 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:40:46.077778 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:40:47.093146 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:40:47.093097 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:40:47.993772 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:40:47.993737 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:40:48.054627 ip-10-0-140-107 kubenswrapper[4939]: E1103 01:40:48.054582 4939 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-sthjh" containerName="extractor" Nov 03 01:40:48.730894 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:40:48.730857 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:40:48.833412 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:40:48.833376 4939 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-7c759a-7745bc7499-bmfl4"] Nov 03 01:40:48.845657 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:40:48.845629 4939 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-7c759a-66695c8d79-7ws2z"] Nov 03 01:40:48.923551 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:40:48.923520 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:40:49.041876 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:40:49.041794 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:40:49.288163 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:40:49.288133 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:40:49.508835 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:40:49.508776 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:40:49.923309 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:40:49.923270 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:40:50.029959 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:40:50.029924 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:40:50.306903 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:40:50.306817 4939 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-ef434a-7b64fdc54f-9zxp5" podUID="7c2434b1-8471-41fb-b20d-118aa8f1afc4" containerName="payload-processing" containerID="cri-o://c047ced06b4bc5d1024afd3d2f4221a8d559d5ccc61b216e9f8ed31aba73d6b2" gracePeriod=25 Nov 03 01:40:50.314389 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:40:50.314357 4939 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-ef434a-845d4b6b99-nqmzg" podUID="9a90ca63-d97d-4695-ad39-eaaa58204ecc" containerName="payload-pre-processing" containerID="cri-o://11f7251c363756a37e10e790158301d48898dab3f8155045c17c54b7e3e22883" gracePeriod=25 Nov 03 01:40:50.443448 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:40:50.443407 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:40:51.054721 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:40:51.054688 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:40:51.421068 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:40:51.421037 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:40:51.989196 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:40:51.989145 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:40:52.184860 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:40:52.184832 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:40:52.861104 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:40:52.861066 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:40:53.079533 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:40:53.079499 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:40:53.175355 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:40:53.175280 4939 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-n42s6_65d134ac-8a25-4c67-b6c7-cd752a098daf/console-operator/2.log" Nov 03 01:40:53.175355 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:40:53.175287 4939 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-n42s6_65d134ac-8a25-4c67-b6c7-cd752a098daf/console-operator/2.log" Nov 03 01:40:53.742113 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:40:53.742082 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:40:53.834267 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:40:53.834218 4939 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-extmod-a-7c759a-7745bc7499-bmfl4" podUID="19d08863-aa23-47f6-98cd-7908001a4458" containerName="payload-processing" containerID="cri-o://2c1ef432bd467355b686710995c8a2911b651dcbd6ac353b5209684b8930d156" gracePeriod=25 Nov 03 01:40:53.846581 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:40:53.846551 4939 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-7c759a-66695c8d79-7ws2z" podUID="653b839f-b60a-466c-8b49-24aacea6dd75" containerName="payload-pre-processing" containerID="cri-o://8f175f99fa639f041e070a9428562d3a812c348ca9357f735b0959496103542a" gracePeriod=25 Nov 03 01:40:54.349048 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:40:54.349000 4939 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-7c759a-66695c8d79-7ws2z" podUID="653b839f-b60a-466c-8b49-24aacea6dd75" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 03 01:40:54.368817 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:40:54.368784 4939 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-extmod-a-7c759a-7745bc7499-bmfl4" podUID="19d08863-aa23-47f6-98cd-7908001a4458" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 03 01:40:54.504412 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:40:54.504363 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:40:55.305577 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:40:55.305524 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:40:55.831719 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:40:55.831685 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:40:55.917957 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:40:55.917896 4939 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-d25759-5587dc5df9-tl5d6"] Nov 03 01:40:55.977704 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:40:55.977610 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:40:56.098257 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:40:56.098172 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:40:56.410778 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:40:56.410703 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:40:56.534359 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:40:56.534325 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:40:56.754100 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:40:56.754027 4939 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-a-7c759a-openshift-default-56576554df-wkcvp"] Nov 03 01:40:56.754295 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:40:56.754273 4939 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-extmod-a-7c759a-openshift-default-56576554df-wkcvp" podUID="6ea24c7b-6d13-4661-bdff-cd52792cb6b6" containerName="istio-proxy" containerID="cri-o://225a0fd49c51b5f986149881dd6aa294b9a1e96c6ebda525835578dd224f0a1e" gracePeriod=30 Nov 03 01:40:56.902935 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:40:56.900422 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:40:57.030774 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:40:57.030675 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:40:57.348666 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:40:57.348627 4939 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-ef434a-845d4b6b99-nqmzg" podUID="9a90ca63-d97d-4695-ad39-eaaa58204ecc" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 03 01:40:57.348666 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:40:57.348649 4939 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-ef434a-7b64fdc54f-9zxp5" podUID="7c2434b1-8471-41fb-b20d-118aa8f1afc4" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 03 01:40:57.382069 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:40:57.382038 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:40:57.659507 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:40:57.659429 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:40:58.086624 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:40:58.086591 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:40:58.603591 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:40:58.603555 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:40:58.715499 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:40:58.715467 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:40:59.077625 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:40:59.077529 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:40:59.389719 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:40:59.389685 4939 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-ef434a-openshift-default-67cd7c96dc-q5p6d"] Nov 03 01:40:59.389976 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:40:59.389948 4939 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w2-ef434a-openshift-default-67cd7c96dc-q5p6d" podUID="285b736b-2a9f-40cd-96c4-3d7193d3eb8d" containerName="istio-proxy" containerID="cri-o://e7842b2bdf2d7feb7daf09483668d416de00149c08408519bd1e496b3a83bd46" gracePeriod=30 Nov 03 01:40:59.587486 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:40:59.587450 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:40:59.715549 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:40:59.715462 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:40:59.981151 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:40:59.981075 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:41:00.281213 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:41:00.281139 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:41:00.573517 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:41:00.573441 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:41:00.918993 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:41:00.918956 4939 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-extmod-a-d25759-5587dc5df9-tl5d6" podUID="f2d0a2aa-d02a-4153-b8e8-997c8d1120fe" containerName="payload-processing" containerID="cri-o://e51084d1ae4e0dd5dfdb55fd149c245c6d3f834a9ed6e1ecde4ee57467423824" gracePeriod=25 Nov 03 01:41:01.213605 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:41:01.213580 4939 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-a-d25759-5587dc5df9-tl5d6" Nov 03 01:41:01.301166 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:41:01.301133 4939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f2d0a2aa-d02a-4153-b8e8-997c8d1120fe-kube-api-access-ttxsg\" (UniqueName: \"kubernetes.io/projected/f2d0a2aa-d02a-4153-b8e8-997c8d1120fe-kube-api-access-ttxsg\") pod \"f2d0a2aa-d02a-4153-b8e8-997c8d1120fe\" (UID: \"f2d0a2aa-d02a-4153-b8e8-997c8d1120fe\") " Nov 03 01:41:01.301352 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:41:01.301240 4939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f2d0a2aa-d02a-4153-b8e8-997c8d1120fe-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f2d0a2aa-d02a-4153-b8e8-997c8d1120fe-plugins-config-volume\") pod \"f2d0a2aa-d02a-4153-b8e8-997c8d1120fe\" (UID: \"f2d0a2aa-d02a-4153-b8e8-997c8d1120fe\") " Nov 03 01:41:01.301613 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:41:01.301585 4939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f2d0a2aa-d02a-4153-b8e8-997c8d1120fe-plugins-config-volume" pod "f2d0a2aa-d02a-4153-b8e8-997c8d1120fe" (UID: "f2d0a2aa-d02a-4153-b8e8-997c8d1120fe"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 01:41:01.301888 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:41:01.301872 4939 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f2d0a2aa-d02a-4153-b8e8-997c8d1120fe-plugins-config-volume\") on node \"ip-10-0-140-107.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:41:01.303413 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:41:01.303382 4939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f2d0a2aa-d02a-4153-b8e8-997c8d1120fe-kube-api-access-ttxsg" pod "f2d0a2aa-d02a-4153-b8e8-997c8d1120fe" (UID: "f2d0a2aa-d02a-4153-b8e8-997c8d1120fe"). InnerVolumeSpecName "kube-api-access-ttxsg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 01:41:01.321568 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:41:01.321541 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:41:01.403360 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:41:01.403320 4939 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ttxsg\" (UniqueName: \"kubernetes.io/projected/f2d0a2aa-d02a-4153-b8e8-997c8d1120fe-kube-api-access-ttxsg\") on node \"ip-10-0-140-107.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:41:01.961473 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:41:01.961430 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:41:02.027188 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:41:02.020584 4939 generic.go:356] "Generic (PLEG): container finished" podID="f2d0a2aa-d02a-4153-b8e8-997c8d1120fe" containerID="e51084d1ae4e0dd5dfdb55fd149c245c6d3f834a9ed6e1ecde4ee57467423824" exitCode=0 Nov 03 01:41:02.027188 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:41:02.020659 4939 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-a-d25759-5587dc5df9-tl5d6" Nov 03 01:41:02.027188 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:41:02.020743 4939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-a-d25759-5587dc5df9-tl5d6" event={"ID":"f2d0a2aa-d02a-4153-b8e8-997c8d1120fe","Type":"ContainerDied","Data":"e51084d1ae4e0dd5dfdb55fd149c245c6d3f834a9ed6e1ecde4ee57467423824"} Nov 03 01:41:02.027188 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:41:02.020795 4939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-a-d25759-5587dc5df9-tl5d6" event={"ID":"f2d0a2aa-d02a-4153-b8e8-997c8d1120fe","Type":"ContainerDied","Data":"b0ec7a02c5d1501dad844dfecaf6d2e2d88d4983fad2ea417e90748ff4ac4a58"} Nov 03 01:41:02.027188 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:41:02.020825 4939 scope.go:122] "RemoveContainer" containerID="e51084d1ae4e0dd5dfdb55fd149c245c6d3f834a9ed6e1ecde4ee57467423824" Nov 03 01:41:02.027188 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:41:02.024273 4939 generic.go:356] "Generic (PLEG): container finished" podID="6ea24c7b-6d13-4661-bdff-cd52792cb6b6" containerID="225a0fd49c51b5f986149881dd6aa294b9a1e96c6ebda525835578dd224f0a1e" exitCode=0 Nov 03 01:41:02.027188 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:41:02.024390 4939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-7c759a-openshift-default-56576554df-wkcvp" event={"ID":"6ea24c7b-6d13-4661-bdff-cd52792cb6b6","Type":"ContainerDied","Data":"225a0fd49c51b5f986149881dd6aa294b9a1e96c6ebda525835578dd224f0a1e"} Nov 03 01:41:02.038827 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:41:02.038801 4939 scope.go:122] "RemoveContainer" containerID="e51084d1ae4e0dd5dfdb55fd149c245c6d3f834a9ed6e1ecde4ee57467423824" Nov 03 01:41:02.039231 ip-10-0-140-107 kubenswrapper[4939]: E1103 01:41:02.039199 4939 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e51084d1ae4e0dd5dfdb55fd149c245c6d3f834a9ed6e1ecde4ee57467423824\": container with ID starting with e51084d1ae4e0dd5dfdb55fd149c245c6d3f834a9ed6e1ecde4ee57467423824 not found: ID does not exist" containerID="e51084d1ae4e0dd5dfdb55fd149c245c6d3f834a9ed6e1ecde4ee57467423824" Nov 03 01:41:02.039362 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:41:02.039241 4939 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e51084d1ae4e0dd5dfdb55fd149c245c6d3f834a9ed6e1ecde4ee57467423824"} err="failed to get container status \"e51084d1ae4e0dd5dfdb55fd149c245c6d3f834a9ed6e1ecde4ee57467423824\": rpc error: code = NotFound desc = could not find container \"e51084d1ae4e0dd5dfdb55fd149c245c6d3f834a9ed6e1ecde4ee57467423824\": container with ID starting with e51084d1ae4e0dd5dfdb55fd149c245c6d3f834a9ed6e1ecde4ee57467423824 not found: ID does not exist" Nov 03 01:41:02.039362 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:41:02.039264 4939 scope.go:122] "RemoveContainer" containerID="a45f53d3c2fe34da76be639f2a986c7317499e4dc3d560d8ca33ca00bdf94750" Nov 03 01:41:02.043167 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:41:02.043136 4939 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-d25759-5587dc5df9-tl5d6"] Nov 03 01:41:02.052243 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:41:02.052224 4939 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-d25759-5587dc5df9-tl5d6"] Nov 03 01:41:02.115185 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:41:02.115158 4939 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-7c759a-openshift-default-56576554df-wkcvp" Nov 03 01:41:02.220974 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:41:02.220897 4939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6ea24c7b-6d13-4661-bdff-cd52792cb6b6-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6ea24c7b-6d13-4661-bdff-cd52792cb6b6-istiod-ca-cert\") pod \"6ea24c7b-6d13-4661-bdff-cd52792cb6b6\" (UID: \"6ea24c7b-6d13-4661-bdff-cd52792cb6b6\") " Nov 03 01:41:02.221156 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:41:02.221035 4939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6ea24c7b-6d13-4661-bdff-cd52792cb6b6-istio-token\" (UniqueName: \"kubernetes.io/projected/6ea24c7b-6d13-4661-bdff-cd52792cb6b6-istio-token\") pod \"6ea24c7b-6d13-4661-bdff-cd52792cb6b6\" (UID: \"6ea24c7b-6d13-4661-bdff-cd52792cb6b6\") " Nov 03 01:41:02.221156 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:41:02.221129 4939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/6ea24c7b-6d13-4661-bdff-cd52792cb6b6-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6ea24c7b-6d13-4661-bdff-cd52792cb6b6-istio-podinfo\") pod \"6ea24c7b-6d13-4661-bdff-cd52792cb6b6\" (UID: \"6ea24c7b-6d13-4661-bdff-cd52792cb6b6\") " Nov 03 01:41:02.221253 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:41:02.221209 4939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6ea24c7b-6d13-4661-bdff-cd52792cb6b6-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6ea24c7b-6d13-4661-bdff-cd52792cb6b6-workload-socket\") pod \"6ea24c7b-6d13-4661-bdff-cd52792cb6b6\" (UID: \"6ea24c7b-6d13-4661-bdff-cd52792cb6b6\") " Nov 03 01:41:02.221313 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:41:02.221259 4939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6ea24c7b-6d13-4661-bdff-cd52792cb6b6-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6ea24c7b-6d13-4661-bdff-cd52792cb6b6-istio-envoy\") pod \"6ea24c7b-6d13-4661-bdff-cd52792cb6b6\" (UID: \"6ea24c7b-6d13-4661-bdff-cd52792cb6b6\") " Nov 03 01:41:02.221377 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:41:02.221328 4939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6ea24c7b-6d13-4661-bdff-cd52792cb6b6-istiod-ca-cert" pod "6ea24c7b-6d13-4661-bdff-cd52792cb6b6" (UID: "6ea24c7b-6d13-4661-bdff-cd52792cb6b6"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 01:41:02.221448 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:41:02.221395 4939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6ea24c7b-6d13-4661-bdff-cd52792cb6b6-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6ea24c7b-6d13-4661-bdff-cd52792cb6b6-workload-certs\") pod \"6ea24c7b-6d13-4661-bdff-cd52792cb6b6\" (UID: \"6ea24c7b-6d13-4661-bdff-cd52792cb6b6\") " Nov 03 01:41:02.221448 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:41:02.221432 4939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6ea24c7b-6d13-4661-bdff-cd52792cb6b6-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6ea24c7b-6d13-4661-bdff-cd52792cb6b6-credential-socket\") pod \"6ea24c7b-6d13-4661-bdff-cd52792cb6b6\" (UID: \"6ea24c7b-6d13-4661-bdff-cd52792cb6b6\") " Nov 03 01:41:02.221568 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:41:02.221490 4939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6ea24c7b-6d13-4661-bdff-cd52792cb6b6-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6ea24c7b-6d13-4661-bdff-cd52792cb6b6-istio-data\") pod \"6ea24c7b-6d13-4661-bdff-cd52792cb6b6\" (UID: \"6ea24c7b-6d13-4661-bdff-cd52792cb6b6\") " Nov 03 01:41:02.221568 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:41:02.221526 4939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6ea24c7b-6d13-4661-bdff-cd52792cb6b6-kube-api-access-4kmml\" (UniqueName: \"kubernetes.io/projected/6ea24c7b-6d13-4661-bdff-cd52792cb6b6-kube-api-access-4kmml\") pod \"6ea24c7b-6d13-4661-bdff-cd52792cb6b6\" (UID: \"6ea24c7b-6d13-4661-bdff-cd52792cb6b6\") " Nov 03 01:41:02.221568 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:41:02.221541 4939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6ea24c7b-6d13-4661-bdff-cd52792cb6b6-workload-socket" pod "6ea24c7b-6d13-4661-bdff-cd52792cb6b6" (UID: "6ea24c7b-6d13-4661-bdff-cd52792cb6b6"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 01:41:02.221739 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:41:02.221665 4939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6ea24c7b-6d13-4661-bdff-cd52792cb6b6-workload-certs" pod "6ea24c7b-6d13-4661-bdff-cd52792cb6b6" (UID: "6ea24c7b-6d13-4661-bdff-cd52792cb6b6"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 01:41:02.221806 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:41:02.221748 4939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6ea24c7b-6d13-4661-bdff-cd52792cb6b6-credential-socket" pod "6ea24c7b-6d13-4661-bdff-cd52792cb6b6" (UID: "6ea24c7b-6d13-4661-bdff-cd52792cb6b6"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 01:41:02.221806 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:41:02.221777 4939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6ea24c7b-6d13-4661-bdff-cd52792cb6b6-istio-data" pod "6ea24c7b-6d13-4661-bdff-cd52792cb6b6" (UID: "6ea24c7b-6d13-4661-bdff-cd52792cb6b6"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 01:41:02.222264 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:41:02.222242 4939 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6ea24c7b-6d13-4661-bdff-cd52792cb6b6-workload-socket\") on node \"ip-10-0-140-107.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:41:02.222379 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:41:02.222266 4939 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6ea24c7b-6d13-4661-bdff-cd52792cb6b6-workload-certs\") on node \"ip-10-0-140-107.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:41:02.222379 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:41:02.222281 4939 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6ea24c7b-6d13-4661-bdff-cd52792cb6b6-credential-socket\") on node \"ip-10-0-140-107.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:41:02.222379 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:41:02.222293 4939 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6ea24c7b-6d13-4661-bdff-cd52792cb6b6-istio-data\") on node \"ip-10-0-140-107.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:41:02.222379 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:41:02.222306 4939 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6ea24c7b-6d13-4661-bdff-cd52792cb6b6-istiod-ca-cert\") on node \"ip-10-0-140-107.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:41:02.223522 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:41:02.223493 4939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6ea24c7b-6d13-4661-bdff-cd52792cb6b6-istio-envoy" pod "6ea24c7b-6d13-4661-bdff-cd52792cb6b6" (UID: "6ea24c7b-6d13-4661-bdff-cd52792cb6b6"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 01:41:02.223880 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:41:02.223858 4939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6ea24c7b-6d13-4661-bdff-cd52792cb6b6-kube-api-access-4kmml" pod "6ea24c7b-6d13-4661-bdff-cd52792cb6b6" (UID: "6ea24c7b-6d13-4661-bdff-cd52792cb6b6"). InnerVolumeSpecName "kube-api-access-4kmml". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 01:41:02.223987 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:41:02.223887 4939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/6ea24c7b-6d13-4661-bdff-cd52792cb6b6-istio-podinfo" pod "6ea24c7b-6d13-4661-bdff-cd52792cb6b6" (UID: "6ea24c7b-6d13-4661-bdff-cd52792cb6b6"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 01:41:02.223987 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:41:02.223894 4939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6ea24c7b-6d13-4661-bdff-cd52792cb6b6-istio-token" pod "6ea24c7b-6d13-4661-bdff-cd52792cb6b6" (UID: "6ea24c7b-6d13-4661-bdff-cd52792cb6b6"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 01:41:02.323853 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:41:02.323784 4939 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6ea24c7b-6d13-4661-bdff-cd52792cb6b6-istio-podinfo\") on node \"ip-10-0-140-107.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:41:02.323853 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:41:02.323814 4939 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6ea24c7b-6d13-4661-bdff-cd52792cb6b6-istio-envoy\") on node \"ip-10-0-140-107.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:41:02.323853 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:41:02.323824 4939 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4kmml\" (UniqueName: \"kubernetes.io/projected/6ea24c7b-6d13-4661-bdff-cd52792cb6b6-kube-api-access-4kmml\") on node \"ip-10-0-140-107.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:41:02.323853 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:41:02.323834 4939 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6ea24c7b-6d13-4661-bdff-cd52792cb6b6-istio-token\") on node \"ip-10-0-140-107.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:41:02.710049 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:41:02.710015 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:41:03.018937 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:41:03.018845 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:41:03.033430 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:41:03.033405 4939 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-7c759a-openshift-default-56576554df-wkcvp" Nov 03 01:41:03.033628 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:41:03.033436 4939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-7c759a-openshift-default-56576554df-wkcvp" event={"ID":"6ea24c7b-6d13-4661-bdff-cd52792cb6b6","Type":"ContainerDied","Data":"b18df9e0980f634a78d89b2482a2f52611e2e5d34f6b940894f8d2367455c994"} Nov 03 01:41:03.033628 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:41:03.033512 4939 scope.go:122] "RemoveContainer" containerID="225a0fd49c51b5f986149881dd6aa294b9a1e96c6ebda525835578dd224f0a1e" Nov 03 01:41:03.059705 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:41:03.059677 4939 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-a-7c759a-openshift-default-56576554df-wkcvp"] Nov 03 01:41:03.063279 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:41:03.063256 4939 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-extmod-a-7c759a-openshift-default-56576554df-wkcvp"] Nov 03 01:41:03.133649 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:41:03.131052 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:41:03.357489 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:41:03.357455 4939 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-b-57079b-5f4958f857-v8cxd"] Nov 03 01:41:03.381413 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:41:03.381382 4939 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-57079b-666f95c947-dv7mr"] Nov 03 01:41:03.470887 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:41:03.470855 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:41:03.582768 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:41:03.582733 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:41:03.824021 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:41:03.823886 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:41:03.869949 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:41:03.869876 4939 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6ea24c7b-6d13-4661-bdff-cd52792cb6b6" path="/var/lib/kubelet/pods/6ea24c7b-6d13-4661-bdff-cd52792cb6b6/volumes" Nov 03 01:41:03.870469 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:41:03.870445 4939 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f2d0a2aa-d02a-4153-b8e8-997c8d1120fe" path="/var/lib/kubelet/pods/f2d0a2aa-d02a-4153-b8e8-997c8d1120fe/volumes" Nov 03 01:41:03.953233 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:41:03.953198 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:41:04.200625 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:41:04.200590 4939 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-a-d25759-openshift-default-d47fd9785-7gzj2"] Nov 03 01:41:04.201181 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:41:04.200860 4939 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-extmod-a-d25759-openshift-default-d47fd9785-7gzj2" podUID="ba585f4b-4916-4d03-bf1e-454c029d46ad" containerName="istio-proxy" containerID="cri-o://4dff96a063f74b8b786efbb9331372ae589a636a6bf3dadbf90aa7da47d8e532" gracePeriod=30 Nov 03 01:41:04.253417 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:41:04.253387 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:41:04.353968 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:41:04.352119 4939 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-7c759a-66695c8d79-7ws2z" podUID="653b839f-b60a-466c-8b49-24aacea6dd75" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 03 01:41:04.383015 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:41:04.382969 4939 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-extmod-a-7c759a-7745bc7499-bmfl4" podUID="19d08863-aa23-47f6-98cd-7908001a4458" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 03 01:41:04.681702 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:41:04.681669 4939 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w2-ef434a-openshift-default-67cd7c96dc-q5p6d" Nov 03 01:41:04.696558 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:41:04.696513 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:41:04.774868 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:41:04.774785 4939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/285b736b-2a9f-40cd-96c4-3d7193d3eb8d-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/285b736b-2a9f-40cd-96c4-3d7193d3eb8d-workload-certs\") pod \"285b736b-2a9f-40cd-96c4-3d7193d3eb8d\" (UID: \"285b736b-2a9f-40cd-96c4-3d7193d3eb8d\") " Nov 03 01:41:04.774868 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:41:04.774836 4939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/285b736b-2a9f-40cd-96c4-3d7193d3eb8d-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/285b736b-2a9f-40cd-96c4-3d7193d3eb8d-credential-socket\") pod \"285b736b-2a9f-40cd-96c4-3d7193d3eb8d\" (UID: \"285b736b-2a9f-40cd-96c4-3d7193d3eb8d\") " Nov 03 01:41:04.775174 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:41:04.774882 4939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/285b736b-2a9f-40cd-96c4-3d7193d3eb8d-istio-token\" (UniqueName: \"kubernetes.io/projected/285b736b-2a9f-40cd-96c4-3d7193d3eb8d-istio-token\") pod \"285b736b-2a9f-40cd-96c4-3d7193d3eb8d\" (UID: \"285b736b-2a9f-40cd-96c4-3d7193d3eb8d\") " Nov 03 01:41:04.775174 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:41:04.774978 4939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/285b736b-2a9f-40cd-96c4-3d7193d3eb8d-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/285b736b-2a9f-40cd-96c4-3d7193d3eb8d-istio-data\") pod \"285b736b-2a9f-40cd-96c4-3d7193d3eb8d\" (UID: \"285b736b-2a9f-40cd-96c4-3d7193d3eb8d\") " Nov 03 01:41:04.775174 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:41:04.775044 4939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/285b736b-2a9f-40cd-96c4-3d7193d3eb8d-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/285b736b-2a9f-40cd-96c4-3d7193d3eb8d-workload-socket\") pod \"285b736b-2a9f-40cd-96c4-3d7193d3eb8d\" (UID: \"285b736b-2a9f-40cd-96c4-3d7193d3eb8d\") " Nov 03 01:41:04.775174 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:41:04.775075 4939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/285b736b-2a9f-40cd-96c4-3d7193d3eb8d-workload-certs" pod "285b736b-2a9f-40cd-96c4-3d7193d3eb8d" (UID: "285b736b-2a9f-40cd-96c4-3d7193d3eb8d"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 01:41:04.775174 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:41:04.775099 4939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/285b736b-2a9f-40cd-96c4-3d7193d3eb8d-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/285b736b-2a9f-40cd-96c4-3d7193d3eb8d-istio-envoy\") pod \"285b736b-2a9f-40cd-96c4-3d7193d3eb8d\" (UID: \"285b736b-2a9f-40cd-96c4-3d7193d3eb8d\") " Nov 03 01:41:04.775174 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:41:04.775170 4939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/285b736b-2a9f-40cd-96c4-3d7193d3eb8d-kube-api-access-wgsfl\" (UniqueName: \"kubernetes.io/projected/285b736b-2a9f-40cd-96c4-3d7193d3eb8d-kube-api-access-wgsfl\") pod \"285b736b-2a9f-40cd-96c4-3d7193d3eb8d\" (UID: \"285b736b-2a9f-40cd-96c4-3d7193d3eb8d\") " Nov 03 01:41:04.775544 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:41:04.775203 4939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/285b736b-2a9f-40cd-96c4-3d7193d3eb8d-credential-socket" pod "285b736b-2a9f-40cd-96c4-3d7193d3eb8d" (UID: "285b736b-2a9f-40cd-96c4-3d7193d3eb8d"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 01:41:04.775544 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:41:04.775260 4939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/285b736b-2a9f-40cd-96c4-3d7193d3eb8d-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/285b736b-2a9f-40cd-96c4-3d7193d3eb8d-istio-podinfo\") pod \"285b736b-2a9f-40cd-96c4-3d7193d3eb8d\" (UID: \"285b736b-2a9f-40cd-96c4-3d7193d3eb8d\") " Nov 03 01:41:04.775544 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:41:04.775308 4939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/285b736b-2a9f-40cd-96c4-3d7193d3eb8d-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/285b736b-2a9f-40cd-96c4-3d7193d3eb8d-istiod-ca-cert\") pod \"285b736b-2a9f-40cd-96c4-3d7193d3eb8d\" (UID: \"285b736b-2a9f-40cd-96c4-3d7193d3eb8d\") " Nov 03 01:41:04.775544 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:41:04.775321 4939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/285b736b-2a9f-40cd-96c4-3d7193d3eb8d-workload-socket" pod "285b736b-2a9f-40cd-96c4-3d7193d3eb8d" (UID: "285b736b-2a9f-40cd-96c4-3d7193d3eb8d"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 01:41:04.775544 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:41:04.775335 4939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/285b736b-2a9f-40cd-96c4-3d7193d3eb8d-istio-data" pod "285b736b-2a9f-40cd-96c4-3d7193d3eb8d" (UID: "285b736b-2a9f-40cd-96c4-3d7193d3eb8d"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 01:41:04.775809 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:41:04.775713 4939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/285b736b-2a9f-40cd-96c4-3d7193d3eb8d-istiod-ca-cert" pod "285b736b-2a9f-40cd-96c4-3d7193d3eb8d" (UID: "285b736b-2a9f-40cd-96c4-3d7193d3eb8d"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 01:41:04.775887 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:41:04.775860 4939 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/285b736b-2a9f-40cd-96c4-3d7193d3eb8d-workload-socket\") on node \"ip-10-0-140-107.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:41:04.775968 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:41:04.775886 4939 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/285b736b-2a9f-40cd-96c4-3d7193d3eb8d-istiod-ca-cert\") on node \"ip-10-0-140-107.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:41:04.775968 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:41:04.775902 4939 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/285b736b-2a9f-40cd-96c4-3d7193d3eb8d-workload-certs\") on node \"ip-10-0-140-107.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:41:04.775968 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:41:04.775964 4939 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/285b736b-2a9f-40cd-96c4-3d7193d3eb8d-credential-socket\") on node \"ip-10-0-140-107.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:41:04.776101 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:41:04.775979 4939 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/285b736b-2a9f-40cd-96c4-3d7193d3eb8d-istio-data\") on node \"ip-10-0-140-107.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:41:04.777371 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:41:04.777345 4939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/285b736b-2a9f-40cd-96c4-3d7193d3eb8d-istio-envoy" pod "285b736b-2a9f-40cd-96c4-3d7193d3eb8d" (UID: "285b736b-2a9f-40cd-96c4-3d7193d3eb8d"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 01:41:04.777548 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:41:04.777529 4939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/285b736b-2a9f-40cd-96c4-3d7193d3eb8d-kube-api-access-wgsfl" pod "285b736b-2a9f-40cd-96c4-3d7193d3eb8d" (UID: "285b736b-2a9f-40cd-96c4-3d7193d3eb8d"). InnerVolumeSpecName "kube-api-access-wgsfl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 01:41:04.777669 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:41:04.777652 4939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/285b736b-2a9f-40cd-96c4-3d7193d3eb8d-istio-token" pod "285b736b-2a9f-40cd-96c4-3d7193d3eb8d" (UID: "285b736b-2a9f-40cd-96c4-3d7193d3eb8d"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 01:41:04.777727 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:41:04.777713 4939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/285b736b-2a9f-40cd-96c4-3d7193d3eb8d-istio-podinfo" pod "285b736b-2a9f-40cd-96c4-3d7193d3eb8d" (UID: "285b736b-2a9f-40cd-96c4-3d7193d3eb8d"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 01:41:04.876717 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:41:04.876689 4939 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/285b736b-2a9f-40cd-96c4-3d7193d3eb8d-istio-envoy\") on node \"ip-10-0-140-107.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:41:04.876717 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:41:04.876717 4939 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wgsfl\" (UniqueName: \"kubernetes.io/projected/285b736b-2a9f-40cd-96c4-3d7193d3eb8d-kube-api-access-wgsfl\") on node \"ip-10-0-140-107.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:41:04.876860 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:41:04.876728 4939 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/285b736b-2a9f-40cd-96c4-3d7193d3eb8d-istio-podinfo\") on node \"ip-10-0-140-107.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:41:04.876860 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:41:04.876738 4939 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/285b736b-2a9f-40cd-96c4-3d7193d3eb8d-istio-token\") on node \"ip-10-0-140-107.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:41:04.912547 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:41:04.912517 4939 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w2-f7680d/distinct-2-w2-f7680d-kserve-7d7c4564bf-cbv2c"] Nov 03 01:41:04.915596 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:41:04.915562 4939 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w2-f7680d/distinct-w2-f7680d-kserve-756f874b98-qnpv8"] Nov 03 01:41:04.941294 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:41:04.941255 4939 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w2-f7680d/premium-w2-f7680d-kserve-76565b8cf9-hqbbt"] Nov 03 01:41:05.056951 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:41:05.056596 4939 generic.go:356] "Generic (PLEG): container finished" podID="285b736b-2a9f-40cd-96c4-3d7193d3eb8d" containerID="e7842b2bdf2d7feb7daf09483668d416de00149c08408519bd1e496b3a83bd46" exitCode=0 Nov 03 01:41:05.056951 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:41:05.056658 4939 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w2-ef434a-openshift-default-67cd7c96dc-q5p6d" Nov 03 01:41:05.056951 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:41:05.056695 4939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-ef434a-openshift-default-67cd7c96dc-q5p6d" event={"ID":"285b736b-2a9f-40cd-96c4-3d7193d3eb8d","Type":"ContainerDied","Data":"e7842b2bdf2d7feb7daf09483668d416de00149c08408519bd1e496b3a83bd46"} Nov 03 01:41:05.056951 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:41:05.056742 4939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-ef434a-openshift-default-67cd7c96dc-q5p6d" event={"ID":"285b736b-2a9f-40cd-96c4-3d7193d3eb8d","Type":"ContainerDied","Data":"438cf23a4e7648ba51c3aaa84fe347aeb05d38750e2624151d0cbe607de85850"} Nov 03 01:41:05.056951 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:41:05.056772 4939 scope.go:122] "RemoveContainer" containerID="e7842b2bdf2d7feb7daf09483668d416de00149c08408519bd1e496b3a83bd46" Nov 03 01:41:05.071471 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:41:05.071450 4939 scope.go:122] "RemoveContainer" containerID="a01d3cc1289dcec7066b49be8e8f85d54d5805285a2912e982dc947054666749" Nov 03 01:41:05.081540 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:41:05.081518 4939 scope.go:122] "RemoveContainer" containerID="e7842b2bdf2d7feb7daf09483668d416de00149c08408519bd1e496b3a83bd46" Nov 03 01:41:05.081815 ip-10-0-140-107 kubenswrapper[4939]: E1103 01:41:05.081794 4939 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e7842b2bdf2d7feb7daf09483668d416de00149c08408519bd1e496b3a83bd46\": container with ID starting with e7842b2bdf2d7feb7daf09483668d416de00149c08408519bd1e496b3a83bd46 not found: ID does not exist" containerID="e7842b2bdf2d7feb7daf09483668d416de00149c08408519bd1e496b3a83bd46" Nov 03 01:41:05.081877 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:41:05.081827 4939 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e7842b2bdf2d7feb7daf09483668d416de00149c08408519bd1e496b3a83bd46"} err="failed to get container status \"e7842b2bdf2d7feb7daf09483668d416de00149c08408519bd1e496b3a83bd46\": rpc error: code = NotFound desc = could not find container \"e7842b2bdf2d7feb7daf09483668d416de00149c08408519bd1e496b3a83bd46\": container with ID starting with e7842b2bdf2d7feb7daf09483668d416de00149c08408519bd1e496b3a83bd46 not found: ID does not exist" Nov 03 01:41:05.081877 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:41:05.081844 4939 scope.go:122] "RemoveContainer" containerID="a01d3cc1289dcec7066b49be8e8f85d54d5805285a2912e982dc947054666749" Nov 03 01:41:05.082170 ip-10-0-140-107 kubenswrapper[4939]: E1103 01:41:05.082150 4939 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a01d3cc1289dcec7066b49be8e8f85d54d5805285a2912e982dc947054666749\": container with ID starting with a01d3cc1289dcec7066b49be8e8f85d54d5805285a2912e982dc947054666749 not found: ID does not exist" containerID="a01d3cc1289dcec7066b49be8e8f85d54d5805285a2912e982dc947054666749" Nov 03 01:41:05.082214 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:41:05.082176 4939 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a01d3cc1289dcec7066b49be8e8f85d54d5805285a2912e982dc947054666749"} err="failed to get container status \"a01d3cc1289dcec7066b49be8e8f85d54d5805285a2912e982dc947054666749\": rpc error: code = NotFound desc = could not find container \"a01d3cc1289dcec7066b49be8e8f85d54d5805285a2912e982dc947054666749\": container with ID starting with a01d3cc1289dcec7066b49be8e8f85d54d5805285a2912e982dc947054666749 not found: ID does not exist" Nov 03 01:41:05.093207 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:41:05.093180 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:41:05.097351 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:41:05.097329 4939 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-ef434a-openshift-default-67cd7c96dc-q5p6d"] Nov 03 01:41:05.112070 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:41:05.112039 4939 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-ef434a-openshift-default-67cd7c96dc-q5p6d"] Nov 03 01:41:05.264127 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:41:05.264096 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:41:05.385009 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:41:05.384979 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:41:05.521874 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:41:05.521825 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:41:05.641991 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:41:05.640746 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:41:05.869576 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:41:05.869539 4939 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="285b736b-2a9f-40cd-96c4-3d7193d3eb8d" path="/var/lib/kubelet/pods/285b736b-2a9f-40cd-96c4-3d7193d3eb8d/volumes" Nov 03 01:41:06.275540 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:41:06.275507 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:41:06.952727 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:41:06.952695 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:41:07.348152 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:41:07.348114 4939 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-ef434a-7b64fdc54f-9zxp5" podUID="7c2434b1-8471-41fb-b20d-118aa8f1afc4" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 03 01:41:07.348604 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:41:07.348147 4939 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-ef434a-845d4b6b99-nqmzg" podUID="9a90ca63-d97d-4695-ad39-eaaa58204ecc" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 03 01:41:07.593476 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:41:07.593441 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:41:08.105826 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:41:08.105795 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:41:08.357822 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:41:08.357722 4939 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-extmod-b-57079b-5f4958f857-v8cxd" podUID="5d25771d-8931-4840-86be-9ebaf92fafca" containerName="payload-processing" containerID="cri-o://efe834ec176f778ad2632569bbf778eb71e02e0117cf2037200131ba3869f8fc" gracePeriod=25 Nov 03 01:41:08.381683 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:41:08.381643 4939 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-57079b-666f95c947-dv7mr" podUID="a79df098-23e4-4971-9263-3f52b7d25b0b" containerName="payload-pre-processing" containerID="cri-o://7ba5aaa0f3337d62777e32bc996b60a0573018aafc9e436e464208aeb016bc29" gracePeriod=25 Nov 03 01:41:08.432614 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:41:08.432585 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:41:09.010585 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:41:09.010550 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:41:09.498132 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:41:09.498100 4939 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-d25759-openshift-default-d47fd9785-7gzj2" Nov 03 01:41:09.657152 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:41:09.657113 4939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ba585f4b-4916-4d03-bf1e-454c029d46ad-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ba585f4b-4916-4d03-bf1e-454c029d46ad-workload-socket\") pod \"ba585f4b-4916-4d03-bf1e-454c029d46ad\" (UID: \"ba585f4b-4916-4d03-bf1e-454c029d46ad\") " Nov 03 01:41:09.657349 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:41:09.657230 4939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ba585f4b-4916-4d03-bf1e-454c029d46ad-kube-api-access-rstkw\" (UniqueName: \"kubernetes.io/projected/ba585f4b-4916-4d03-bf1e-454c029d46ad-kube-api-access-rstkw\") pod \"ba585f4b-4916-4d03-bf1e-454c029d46ad\" (UID: \"ba585f4b-4916-4d03-bf1e-454c029d46ad\") " Nov 03 01:41:09.657349 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:41:09.657280 4939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ba585f4b-4916-4d03-bf1e-454c029d46ad-istio-token\" (UniqueName: \"kubernetes.io/projected/ba585f4b-4916-4d03-bf1e-454c029d46ad-istio-token\") pod \"ba585f4b-4916-4d03-bf1e-454c029d46ad\" (UID: \"ba585f4b-4916-4d03-bf1e-454c029d46ad\") " Nov 03 01:41:09.657349 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:41:09.657320 4939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/ba585f4b-4916-4d03-bf1e-454c029d46ad-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ba585f4b-4916-4d03-bf1e-454c029d46ad-istio-podinfo\") pod \"ba585f4b-4916-4d03-bf1e-454c029d46ad\" (UID: \"ba585f4b-4916-4d03-bf1e-454c029d46ad\") " Nov 03 01:41:09.657545 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:41:09.657362 4939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ba585f4b-4916-4d03-bf1e-454c029d46ad-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ba585f4b-4916-4d03-bf1e-454c029d46ad-workload-certs\") pod \"ba585f4b-4916-4d03-bf1e-454c029d46ad\" (UID: \"ba585f4b-4916-4d03-bf1e-454c029d46ad\") " Nov 03 01:41:09.657545 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:41:09.657421 4939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ba585f4b-4916-4d03-bf1e-454c029d46ad-workload-socket" pod "ba585f4b-4916-4d03-bf1e-454c029d46ad" (UID: "ba585f4b-4916-4d03-bf1e-454c029d46ad"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 01:41:09.657545 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:41:09.657427 4939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ba585f4b-4916-4d03-bf1e-454c029d46ad-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ba585f4b-4916-4d03-bf1e-454c029d46ad-istiod-ca-cert\") pod \"ba585f4b-4916-4d03-bf1e-454c029d46ad\" (UID: \"ba585f4b-4916-4d03-bf1e-454c029d46ad\") " Nov 03 01:41:09.657545 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:41:09.657534 4939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ba585f4b-4916-4d03-bf1e-454c029d46ad-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ba585f4b-4916-4d03-bf1e-454c029d46ad-istio-envoy\") pod \"ba585f4b-4916-4d03-bf1e-454c029d46ad\" (UID: \"ba585f4b-4916-4d03-bf1e-454c029d46ad\") " Nov 03 01:41:09.657793 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:41:09.657695 4939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ba585f4b-4916-4d03-bf1e-454c029d46ad-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ba585f4b-4916-4d03-bf1e-454c029d46ad-credential-socket\") pod \"ba585f4b-4916-4d03-bf1e-454c029d46ad\" (UID: \"ba585f4b-4916-4d03-bf1e-454c029d46ad\") " Nov 03 01:41:09.657793 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:41:09.657744 4939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ba585f4b-4916-4d03-bf1e-454c029d46ad-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ba585f4b-4916-4d03-bf1e-454c029d46ad-istio-data\") pod \"ba585f4b-4916-4d03-bf1e-454c029d46ad\" (UID: \"ba585f4b-4916-4d03-bf1e-454c029d46ad\") " Nov 03 01:41:09.657933 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:41:09.657749 4939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ba585f4b-4916-4d03-bf1e-454c029d46ad-istiod-ca-cert" pod "ba585f4b-4916-4d03-bf1e-454c029d46ad" (UID: "ba585f4b-4916-4d03-bf1e-454c029d46ad"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 01:41:09.657933 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:41:09.657823 4939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ba585f4b-4916-4d03-bf1e-454c029d46ad-workload-certs" pod "ba585f4b-4916-4d03-bf1e-454c029d46ad" (UID: "ba585f4b-4916-4d03-bf1e-454c029d46ad"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 01:41:09.658069 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:41:09.658025 4939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ba585f4b-4916-4d03-bf1e-454c029d46ad-credential-socket" pod "ba585f4b-4916-4d03-bf1e-454c029d46ad" (UID: "ba585f4b-4916-4d03-bf1e-454c029d46ad"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 01:41:09.658197 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:41:09.658174 4939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ba585f4b-4916-4d03-bf1e-454c029d46ad-istio-data" pod "ba585f4b-4916-4d03-bf1e-454c029d46ad" (UID: "ba585f4b-4916-4d03-bf1e-454c029d46ad"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 01:41:09.658457 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:41:09.658433 4939 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ba585f4b-4916-4d03-bf1e-454c029d46ad-credential-socket\") on node \"ip-10-0-140-107.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:41:09.658562 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:41:09.658464 4939 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ba585f4b-4916-4d03-bf1e-454c029d46ad-istio-data\") on node \"ip-10-0-140-107.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:41:09.658562 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:41:09.658480 4939 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ba585f4b-4916-4d03-bf1e-454c029d46ad-workload-socket\") on node \"ip-10-0-140-107.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:41:09.658562 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:41:09.658495 4939 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ba585f4b-4916-4d03-bf1e-454c029d46ad-workload-certs\") on node \"ip-10-0-140-107.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:41:09.658562 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:41:09.658510 4939 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ba585f4b-4916-4d03-bf1e-454c029d46ad-istiod-ca-cert\") on node \"ip-10-0-140-107.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:41:09.659685 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:41:09.659660 4939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/ba585f4b-4916-4d03-bf1e-454c029d46ad-istio-podinfo" pod "ba585f4b-4916-4d03-bf1e-454c029d46ad" (UID: "ba585f4b-4916-4d03-bf1e-454c029d46ad"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 01:41:09.660015 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:41:09.659880 4939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ba585f4b-4916-4d03-bf1e-454c029d46ad-kube-api-access-rstkw" pod "ba585f4b-4916-4d03-bf1e-454c029d46ad" (UID: "ba585f4b-4916-4d03-bf1e-454c029d46ad"). InnerVolumeSpecName "kube-api-access-rstkw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 01:41:09.660015 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:41:09.659979 4939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ba585f4b-4916-4d03-bf1e-454c029d46ad-istio-envoy" pod "ba585f4b-4916-4d03-bf1e-454c029d46ad" (UID: "ba585f4b-4916-4d03-bf1e-454c029d46ad"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 01:41:09.661069 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:41:09.661045 4939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ba585f4b-4916-4d03-bf1e-454c029d46ad-istio-token" pod "ba585f4b-4916-4d03-bf1e-454c029d46ad" (UID: "ba585f4b-4916-4d03-bf1e-454c029d46ad"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 01:41:09.686713 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:41:09.686678 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:41:09.760203 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:41:09.760168 4939 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rstkw\" (UniqueName: \"kubernetes.io/projected/ba585f4b-4916-4d03-bf1e-454c029d46ad-kube-api-access-rstkw\") on node \"ip-10-0-140-107.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:41:09.760203 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:41:09.760198 4939 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ba585f4b-4916-4d03-bf1e-454c029d46ad-istio-token\") on node \"ip-10-0-140-107.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:41:09.760203 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:41:09.760208 4939 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ba585f4b-4916-4d03-bf1e-454c029d46ad-istio-podinfo\") on node \"ip-10-0-140-107.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:41:09.760448 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:41:09.760218 4939 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ba585f4b-4916-4d03-bf1e-454c029d46ad-istio-envoy\") on node \"ip-10-0-140-107.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:41:10.103543 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:41:10.103504 4939 generic.go:356] "Generic (PLEG): container finished" podID="ba585f4b-4916-4d03-bf1e-454c029d46ad" containerID="4dff96a063f74b8b786efbb9331372ae589a636a6bf3dadbf90aa7da47d8e532" exitCode=0 Nov 03 01:41:10.103767 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:41:10.103587 4939 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-d25759-openshift-default-d47fd9785-7gzj2" Nov 03 01:41:10.103767 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:41:10.103599 4939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-d25759-openshift-default-d47fd9785-7gzj2" event={"ID":"ba585f4b-4916-4d03-bf1e-454c029d46ad","Type":"ContainerDied","Data":"4dff96a063f74b8b786efbb9331372ae589a636a6bf3dadbf90aa7da47d8e532"} Nov 03 01:41:10.103767 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:41:10.103683 4939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-d25759-openshift-default-d47fd9785-7gzj2" event={"ID":"ba585f4b-4916-4d03-bf1e-454c029d46ad","Type":"ContainerDied","Data":"c746ff69fb39792575d80fb88e665bfa7bf1e612cdfb668ce9b2670dbc84a992"} Nov 03 01:41:10.103767 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:41:10.103708 4939 scope.go:122] "RemoveContainer" containerID="4dff96a063f74b8b786efbb9331372ae589a636a6bf3dadbf90aa7da47d8e532" Nov 03 01:41:10.115366 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:41:10.115345 4939 scope.go:122] "RemoveContainer" containerID="4dff96a063f74b8b786efbb9331372ae589a636a6bf3dadbf90aa7da47d8e532" Nov 03 01:41:10.115750 ip-10-0-140-107 kubenswrapper[4939]: E1103 01:41:10.115677 4939 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4dff96a063f74b8b786efbb9331372ae589a636a6bf3dadbf90aa7da47d8e532\": container with ID starting with 4dff96a063f74b8b786efbb9331372ae589a636a6bf3dadbf90aa7da47d8e532 not found: ID does not exist" containerID="4dff96a063f74b8b786efbb9331372ae589a636a6bf3dadbf90aa7da47d8e532" Nov 03 01:41:10.115750 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:41:10.115715 4939 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4dff96a063f74b8b786efbb9331372ae589a636a6bf3dadbf90aa7da47d8e532"} err="failed to get container status \"4dff96a063f74b8b786efbb9331372ae589a636a6bf3dadbf90aa7da47d8e532\": rpc error: code = NotFound desc = could not find container \"4dff96a063f74b8b786efbb9331372ae589a636a6bf3dadbf90aa7da47d8e532\": container with ID starting with 4dff96a063f74b8b786efbb9331372ae589a636a6bf3dadbf90aa7da47d8e532 not found: ID does not exist" Nov 03 01:41:10.125706 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:41:10.125681 4939 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-a-d25759-openshift-default-d47fd9785-7gzj2"] Nov 03 01:41:10.130482 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:41:10.130458 4939 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-extmod-a-d25759-openshift-default-d47fd9785-7gzj2"] Nov 03 01:41:10.324882 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:41:10.324841 4939 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-extmod-b-57079b-5f4958f857-v8cxd" podUID="5d25771d-8931-4840-86be-9ebaf92fafca" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 03 01:41:10.422826 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:41:10.422793 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:41:10.609678 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:41:10.609645 4939 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-ef434a-7b64fdc54f-9zxp5" Nov 03 01:41:10.643427 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:41:10.643377 4939 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-ef434a-845d4b6b99-nqmzg" Nov 03 01:41:10.673016 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:41:10.672984 4939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9a90ca63-d97d-4695-ad39-eaaa58204ecc-kube-api-access-5zhnm\" (UniqueName: \"kubernetes.io/projected/9a90ca63-d97d-4695-ad39-eaaa58204ecc-kube-api-access-5zhnm\") pod \"9a90ca63-d97d-4695-ad39-eaaa58204ecc\" (UID: \"9a90ca63-d97d-4695-ad39-eaaa58204ecc\") " Nov 03 01:41:10.673208 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:41:10.673073 4939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9a90ca63-d97d-4695-ad39-eaaa58204ecc-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9a90ca63-d97d-4695-ad39-eaaa58204ecc-plugins-config-volume\") pod \"9a90ca63-d97d-4695-ad39-eaaa58204ecc\" (UID: \"9a90ca63-d97d-4695-ad39-eaaa58204ecc\") " Nov 03 01:41:10.673208 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:41:10.673134 4939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7c2434b1-8471-41fb-b20d-118aa8f1afc4-kube-api-access-gtst8\" (UniqueName: \"kubernetes.io/projected/7c2434b1-8471-41fb-b20d-118aa8f1afc4-kube-api-access-gtst8\") pod \"7c2434b1-8471-41fb-b20d-118aa8f1afc4\" (UID: \"7c2434b1-8471-41fb-b20d-118aa8f1afc4\") " Nov 03 01:41:10.673355 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:41:10.673281 4939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7c2434b1-8471-41fb-b20d-118aa8f1afc4-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7c2434b1-8471-41fb-b20d-118aa8f1afc4-plugins-config-volume\") pod \"7c2434b1-8471-41fb-b20d-118aa8f1afc4\" (UID: \"7c2434b1-8471-41fb-b20d-118aa8f1afc4\") " Nov 03 01:41:10.673841 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:41:10.673810 4939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9a90ca63-d97d-4695-ad39-eaaa58204ecc-plugins-config-volume" pod "9a90ca63-d97d-4695-ad39-eaaa58204ecc" (UID: "9a90ca63-d97d-4695-ad39-eaaa58204ecc"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 01:41:10.674251 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:41:10.674220 4939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7c2434b1-8471-41fb-b20d-118aa8f1afc4-plugins-config-volume" pod "7c2434b1-8471-41fb-b20d-118aa8f1afc4" (UID: "7c2434b1-8471-41fb-b20d-118aa8f1afc4"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 01:41:10.676203 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:41:10.676176 4939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9a90ca63-d97d-4695-ad39-eaaa58204ecc-kube-api-access-5zhnm" pod "9a90ca63-d97d-4695-ad39-eaaa58204ecc" (UID: "9a90ca63-d97d-4695-ad39-eaaa58204ecc"). InnerVolumeSpecName "kube-api-access-5zhnm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 01:41:10.677035 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:41:10.677008 4939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7c2434b1-8471-41fb-b20d-118aa8f1afc4-kube-api-access-gtst8" pod "7c2434b1-8471-41fb-b20d-118aa8f1afc4" (UID: "7c2434b1-8471-41fb-b20d-118aa8f1afc4"). InnerVolumeSpecName "kube-api-access-gtst8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 01:41:10.775049 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:41:10.775015 4939 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7c2434b1-8471-41fb-b20d-118aa8f1afc4-plugins-config-volume\") on node \"ip-10-0-140-107.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:41:10.775049 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:41:10.775046 4939 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5zhnm\" (UniqueName: \"kubernetes.io/projected/9a90ca63-d97d-4695-ad39-eaaa58204ecc-kube-api-access-5zhnm\") on node \"ip-10-0-140-107.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:41:10.775279 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:41:10.775059 4939 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9a90ca63-d97d-4695-ad39-eaaa58204ecc-plugins-config-volume\") on node \"ip-10-0-140-107.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:41:10.775279 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:41:10.775071 4939 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gtst8\" (UniqueName: \"kubernetes.io/projected/7c2434b1-8471-41fb-b20d-118aa8f1afc4-kube-api-access-gtst8\") on node \"ip-10-0-140-107.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:41:11.082216 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:41:11.082186 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:41:11.114574 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:41:11.114540 4939 generic.go:356] "Generic (PLEG): container finished" podID="7c2434b1-8471-41fb-b20d-118aa8f1afc4" containerID="c047ced06b4bc5d1024afd3d2f4221a8d559d5ccc61b216e9f8ed31aba73d6b2" exitCode=0 Nov 03 01:41:11.114754 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:41:11.114603 4939 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-ef434a-7b64fdc54f-9zxp5" Nov 03 01:41:11.114754 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:41:11.114624 4939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-ef434a-7b64fdc54f-9zxp5" event={"ID":"7c2434b1-8471-41fb-b20d-118aa8f1afc4","Type":"ContainerDied","Data":"c047ced06b4bc5d1024afd3d2f4221a8d559d5ccc61b216e9f8ed31aba73d6b2"} Nov 03 01:41:11.114754 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:41:11.114675 4939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-ef434a-7b64fdc54f-9zxp5" event={"ID":"7c2434b1-8471-41fb-b20d-118aa8f1afc4","Type":"ContainerDied","Data":"4b6e30d8b7fb1dd70f5240dce880798681ef77caa960260919612a772f461738"} Nov 03 01:41:11.114754 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:41:11.114707 4939 scope.go:122] "RemoveContainer" containerID="c047ced06b4bc5d1024afd3d2f4221a8d559d5ccc61b216e9f8ed31aba73d6b2" Nov 03 01:41:11.116482 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:41:11.116454 4939 generic.go:356] "Generic (PLEG): container finished" podID="9a90ca63-d97d-4695-ad39-eaaa58204ecc" containerID="11f7251c363756a37e10e790158301d48898dab3f8155045c17c54b7e3e22883" exitCode=0 Nov 03 01:41:11.116645 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:41:11.116489 4939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-ef434a-845d4b6b99-nqmzg" event={"ID":"9a90ca63-d97d-4695-ad39-eaaa58204ecc","Type":"ContainerDied","Data":"11f7251c363756a37e10e790158301d48898dab3f8155045c17c54b7e3e22883"} Nov 03 01:41:11.116645 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:41:11.116520 4939 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-ef434a-845d4b6b99-nqmzg" Nov 03 01:41:11.116645 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:41:11.116542 4939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-ef434a-845d4b6b99-nqmzg" event={"ID":"9a90ca63-d97d-4695-ad39-eaaa58204ecc","Type":"ContainerDied","Data":"dc9a9782097b2362a92d87e984d3044451120d02f86b6117c342441f028c4560"} Nov 03 01:41:11.126531 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:41:11.126409 4939 scope.go:122] "RemoveContainer" containerID="c047ced06b4bc5d1024afd3d2f4221a8d559d5ccc61b216e9f8ed31aba73d6b2" Nov 03 01:41:11.126746 ip-10-0-140-107 kubenswrapper[4939]: E1103 01:41:11.126721 4939 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c047ced06b4bc5d1024afd3d2f4221a8d559d5ccc61b216e9f8ed31aba73d6b2\": container with ID starting with c047ced06b4bc5d1024afd3d2f4221a8d559d5ccc61b216e9f8ed31aba73d6b2 not found: ID does not exist" containerID="c047ced06b4bc5d1024afd3d2f4221a8d559d5ccc61b216e9f8ed31aba73d6b2" Nov 03 01:41:11.126828 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:41:11.126754 4939 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c047ced06b4bc5d1024afd3d2f4221a8d559d5ccc61b216e9f8ed31aba73d6b2"} err="failed to get container status \"c047ced06b4bc5d1024afd3d2f4221a8d559d5ccc61b216e9f8ed31aba73d6b2\": rpc error: code = NotFound desc = could not find container \"c047ced06b4bc5d1024afd3d2f4221a8d559d5ccc61b216e9f8ed31aba73d6b2\": container with ID starting with c047ced06b4bc5d1024afd3d2f4221a8d559d5ccc61b216e9f8ed31aba73d6b2 not found: ID does not exist" Nov 03 01:41:11.126828 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:41:11.126775 4939 scope.go:122] "RemoveContainer" containerID="11f7251c363756a37e10e790158301d48898dab3f8155045c17c54b7e3e22883" Nov 03 01:41:11.136471 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:41:11.136451 4939 scope.go:122] "RemoveContainer" containerID="11f7251c363756a37e10e790158301d48898dab3f8155045c17c54b7e3e22883" Nov 03 01:41:11.136744 ip-10-0-140-107 kubenswrapper[4939]: E1103 01:41:11.136725 4939 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"11f7251c363756a37e10e790158301d48898dab3f8155045c17c54b7e3e22883\": container with ID starting with 11f7251c363756a37e10e790158301d48898dab3f8155045c17c54b7e3e22883 not found: ID does not exist" containerID="11f7251c363756a37e10e790158301d48898dab3f8155045c17c54b7e3e22883" Nov 03 01:41:11.136817 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:41:11.136753 4939 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"11f7251c363756a37e10e790158301d48898dab3f8155045c17c54b7e3e22883"} err="failed to get container status \"11f7251c363756a37e10e790158301d48898dab3f8155045c17c54b7e3e22883\": rpc error: code = NotFound desc = could not find container \"11f7251c363756a37e10e790158301d48898dab3f8155045c17c54b7e3e22883\": container with ID starting with 11f7251c363756a37e10e790158301d48898dab3f8155045c17c54b7e3e22883 not found: ID does not exist" Nov 03 01:41:11.144344 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:41:11.144323 4939 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w2-ef434a-7b64fdc54f-9zxp5"] Nov 03 01:41:11.153767 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:41:11.153744 4939 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w2-ef434a-7b64fdc54f-9zxp5"] Nov 03 01:41:11.165490 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:41:11.165464 4939 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-ef434a-845d4b6b99-nqmzg"] Nov 03 01:41:11.170225 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:41:11.170204 4939 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-ef434a-845d4b6b99-nqmzg"] Nov 03 01:41:11.207886 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:41:11.207858 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:41:11.355077 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:41:11.339133 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:41:11.502245 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:41:11.502204 4939 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-57079b-666f95c947-dv7mr" podUID="a79df098-23e4-4971-9263-3f52b7d25b0b" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 03 01:41:11.870243 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:41:11.870207 4939 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7c2434b1-8471-41fb-b20d-118aa8f1afc4" path="/var/lib/kubelet/pods/7c2434b1-8471-41fb-b20d-118aa8f1afc4/volumes" Nov 03 01:41:11.870844 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:41:11.870822 4939 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9a90ca63-d97d-4695-ad39-eaaa58204ecc" path="/var/lib/kubelet/pods/9a90ca63-d97d-4695-ad39-eaaa58204ecc/volumes" Nov 03 01:41:11.871372 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:41:11.871351 4939 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ba585f4b-4916-4d03-bf1e-454c029d46ad" path="/var/lib/kubelet/pods/ba585f4b-4916-4d03-bf1e-454c029d46ad/volumes" Nov 03 01:41:12.167468 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:41:12.167335 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:41:12.296083 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:41:12.296039 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:41:12.435427 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:41:12.435319 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:41:12.662525 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:41:12.662491 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:41:12.974506 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:41:12.974463 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:41:13.102117 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:41:13.102066 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:41:13.224921 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:41:13.224824 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:41:13.417817 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:41:13.417776 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:41:13.743737 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:41:13.743706 4939 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-b-57079b-openshift-default-5c84d8878f-drtdm"] Nov 03 01:41:13.743961 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:41:13.743899 4939 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-extmod-b-57079b-openshift-default-5c84d8878f-drtdm" podUID="a68659b3-912d-4df9-a162-fa6d5f949d2a" containerName="istio-proxy" containerID="cri-o://339bc7b99b219889b6a9d290a2a36b76ab9c5eb0d5ad09fc64724f2574ac1ff3" gracePeriod=30 Nov 03 01:41:13.855989 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:41:13.850560 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:41:14.149225 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:41:14.149192 4939 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-a-7c759a-7745bc7499-bmfl4" Nov 03 01:41:14.158376 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:41:14.158342 4939 generic.go:356] "Generic (PLEG): container finished" podID="19d08863-aa23-47f6-98cd-7908001a4458" containerID="2c1ef432bd467355b686710995c8a2911b651dcbd6ac353b5209684b8930d156" exitCode=0 Nov 03 01:41:14.158577 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:41:14.158387 4939 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-a-7c759a-7745bc7499-bmfl4" Nov 03 01:41:14.159218 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:41:14.158592 4939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-a-7c759a-7745bc7499-bmfl4" event={"ID":"19d08863-aa23-47f6-98cd-7908001a4458","Type":"ContainerDied","Data":"2c1ef432bd467355b686710995c8a2911b651dcbd6ac353b5209684b8930d156"} Nov 03 01:41:14.159218 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:41:14.158632 4939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-a-7c759a-7745bc7499-bmfl4" event={"ID":"19d08863-aa23-47f6-98cd-7908001a4458","Type":"ContainerDied","Data":"02266c02dd4a303d8ef0f525c10a22b08ac887816e26212d0b702127e8351525"} Nov 03 01:41:14.159218 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:41:14.158661 4939 scope.go:122] "RemoveContainer" containerID="2c1ef432bd467355b686710995c8a2911b651dcbd6ac353b5209684b8930d156" Nov 03 01:41:14.161224 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:41:14.161201 4939 generic.go:356] "Generic (PLEG): container finished" podID="653b839f-b60a-466c-8b49-24aacea6dd75" containerID="8f175f99fa639f041e070a9428562d3a812c348ca9357f735b0959496103542a" exitCode=0 Nov 03 01:41:14.161359 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:41:14.161291 4939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-7c759a-66695c8d79-7ws2z" event={"ID":"653b839f-b60a-466c-8b49-24aacea6dd75","Type":"ContainerDied","Data":"8f175f99fa639f041e070a9428562d3a812c348ca9357f735b0959496103542a"} Nov 03 01:41:14.191029 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:41:14.176115 4939 scope.go:122] "RemoveContainer" containerID="2c1ef432bd467355b686710995c8a2911b651dcbd6ac353b5209684b8930d156" Nov 03 01:41:14.191029 ip-10-0-140-107 kubenswrapper[4939]: E1103 01:41:14.183258 4939 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2c1ef432bd467355b686710995c8a2911b651dcbd6ac353b5209684b8930d156\": container with ID starting with 2c1ef432bd467355b686710995c8a2911b651dcbd6ac353b5209684b8930d156 not found: ID does not exist" containerID="2c1ef432bd467355b686710995c8a2911b651dcbd6ac353b5209684b8930d156" Nov 03 01:41:14.191029 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:41:14.183301 4939 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2c1ef432bd467355b686710995c8a2911b651dcbd6ac353b5209684b8930d156"} err="failed to get container status \"2c1ef432bd467355b686710995c8a2911b651dcbd6ac353b5209684b8930d156\": rpc error: code = NotFound desc = could not find container \"2c1ef432bd467355b686710995c8a2911b651dcbd6ac353b5209684b8930d156\": container with ID starting with 2c1ef432bd467355b686710995c8a2911b651dcbd6ac353b5209684b8930d156 not found: ID does not exist" Nov 03 01:41:14.191983 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:41:14.191816 4939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/19d08863-aa23-47f6-98cd-7908001a4458-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/19d08863-aa23-47f6-98cd-7908001a4458-plugins-config-volume\") pod \"19d08863-aa23-47f6-98cd-7908001a4458\" (UID: \"19d08863-aa23-47f6-98cd-7908001a4458\") " Nov 03 01:41:14.191983 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:41:14.191891 4939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/19d08863-aa23-47f6-98cd-7908001a4458-kube-api-access-7dfb2\" (UniqueName: \"kubernetes.io/projected/19d08863-aa23-47f6-98cd-7908001a4458-kube-api-access-7dfb2\") pod \"19d08863-aa23-47f6-98cd-7908001a4458\" (UID: \"19d08863-aa23-47f6-98cd-7908001a4458\") " Nov 03 01:41:14.192311 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:41:14.192283 4939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/19d08863-aa23-47f6-98cd-7908001a4458-plugins-config-volume" pod "19d08863-aa23-47f6-98cd-7908001a4458" (UID: "19d08863-aa23-47f6-98cd-7908001a4458"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 01:41:14.193536 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:41:14.193512 4939 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/19d08863-aa23-47f6-98cd-7908001a4458-plugins-config-volume\") on node \"ip-10-0-140-107.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:41:14.195725 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:41:14.195703 4939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/19d08863-aa23-47f6-98cd-7908001a4458-kube-api-access-7dfb2" pod "19d08863-aa23-47f6-98cd-7908001a4458" (UID: "19d08863-aa23-47f6-98cd-7908001a4458"). InnerVolumeSpecName "kube-api-access-7dfb2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 01:41:14.208327 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:41:14.208303 4939 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-7c759a-66695c8d79-7ws2z" Nov 03 01:41:14.293998 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:41:14.293891 4939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/653b839f-b60a-466c-8b49-24aacea6dd75-kube-api-access-zdnkv\" (UniqueName: \"kubernetes.io/projected/653b839f-b60a-466c-8b49-24aacea6dd75-kube-api-access-zdnkv\") pod \"653b839f-b60a-466c-8b49-24aacea6dd75\" (UID: \"653b839f-b60a-466c-8b49-24aacea6dd75\") " Nov 03 01:41:14.294203 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:41:14.294101 4939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/653b839f-b60a-466c-8b49-24aacea6dd75-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/653b839f-b60a-466c-8b49-24aacea6dd75-plugins-config-volume\") pod \"653b839f-b60a-466c-8b49-24aacea6dd75\" (UID: \"653b839f-b60a-466c-8b49-24aacea6dd75\") " Nov 03 01:41:14.294459 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:41:14.294444 4939 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7dfb2\" (UniqueName: \"kubernetes.io/projected/19d08863-aa23-47f6-98cd-7908001a4458-kube-api-access-7dfb2\") on node \"ip-10-0-140-107.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:41:14.294528 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:41:14.294464 4939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/653b839f-b60a-466c-8b49-24aacea6dd75-plugins-config-volume" pod "653b839f-b60a-466c-8b49-24aacea6dd75" (UID: "653b839f-b60a-466c-8b49-24aacea6dd75"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 01:41:14.296021 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:41:14.295994 4939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/653b839f-b60a-466c-8b49-24aacea6dd75-kube-api-access-zdnkv" pod "653b839f-b60a-466c-8b49-24aacea6dd75" (UID: "653b839f-b60a-466c-8b49-24aacea6dd75"). InnerVolumeSpecName "kube-api-access-zdnkv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 01:41:14.373940 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:41:14.373878 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:41:14.412930 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:41:14.402503 4939 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/653b839f-b60a-466c-8b49-24aacea6dd75-plugins-config-volume\") on node \"ip-10-0-140-107.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:41:14.412930 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:41:14.402551 4939 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zdnkv\" (UniqueName: \"kubernetes.io/projected/653b839f-b60a-466c-8b49-24aacea6dd75-kube-api-access-zdnkv\") on node \"ip-10-0-140-107.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:41:14.483719 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:41:14.483680 4939 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-7c759a-7745bc7499-bmfl4"] Nov 03 01:41:14.486502 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:41:14.486473 4939 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-7c759a-7745bc7499-bmfl4"] Nov 03 01:41:14.538279 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:41:14.538246 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:41:15.159187 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:41:15.159109 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:41:15.168783 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:41:15.168755 4939 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-7c759a-66695c8d79-7ws2z" Nov 03 01:41:15.168958 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:41:15.168785 4939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-7c759a-66695c8d79-7ws2z" event={"ID":"653b839f-b60a-466c-8b49-24aacea6dd75","Type":"ContainerDied","Data":"28179f533661c0f7690d0da0523c93e4ead2f7c58226dc9901cc26c606e1d504"} Nov 03 01:41:15.168958 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:41:15.168851 4939 scope.go:122] "RemoveContainer" containerID="8f175f99fa639f041e070a9428562d3a812c348ca9357f735b0959496103542a" Nov 03 01:41:15.212558 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:41:15.212524 4939 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-7c759a-66695c8d79-7ws2z"] Nov 03 01:41:15.221452 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:41:15.221425 4939 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-7c759a-66695c8d79-7ws2z"] Nov 03 01:41:15.779315 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:41:15.779262 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:41:15.868768 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:41:15.868727 4939 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="19d08863-aa23-47f6-98cd-7908001a4458" path="/var/lib/kubelet/pods/19d08863-aa23-47f6-98cd-7908001a4458/volumes" Nov 03 01:41:15.869315 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:41:15.869288 4939 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="653b839f-b60a-466c-8b49-24aacea6dd75" path="/var/lib/kubelet/pods/653b839f-b60a-466c-8b49-24aacea6dd75/volumes" Nov 03 01:41:16.500486 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:41:16.500444 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:41:16.917739 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:41:16.917705 4939 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-4pw7v"] Nov 03 01:41:17.199125 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:41:17.199038 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:41:17.920862 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:41:17.920827 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:41:18.035685 ip-10-0-140-107 kubenswrapper[4939]: E1103 01:41:18.035646 4939 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-sthjh" containerName="extractor" Nov 03 01:41:18.250098 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:41:18.250018 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:41:18.865063 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:41:18.865028 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:41:18.998852 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:41:18.998748 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:41:19.079956 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:41:19.079926 4939 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-57079b-openshift-default-5c84d8878f-drtdm" Nov 03 01:41:19.201217 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:41:19.201160 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:41:19.233570 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:41:19.233526 4939 generic.go:356] "Generic (PLEG): container finished" podID="a68659b3-912d-4df9-a162-fa6d5f949d2a" containerID="339bc7b99b219889b6a9d290a2a36b76ab9c5eb0d5ad09fc64724f2574ac1ff3" exitCode=0 Nov 03 01:41:19.233776 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:41:19.233597 4939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-57079b-openshift-default-5c84d8878f-drtdm" event={"ID":"a68659b3-912d-4df9-a162-fa6d5f949d2a","Type":"ContainerDied","Data":"339bc7b99b219889b6a9d290a2a36b76ab9c5eb0d5ad09fc64724f2574ac1ff3"} Nov 03 01:41:19.233776 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:41:19.233675 4939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-57079b-openshift-default-5c84d8878f-drtdm" event={"ID":"a68659b3-912d-4df9-a162-fa6d5f949d2a","Type":"ContainerDied","Data":"ec1d9252517606456a366046a57c04a9b47dcc942a716dbd772dfbd9ad502835"} Nov 03 01:41:19.233776 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:41:19.233703 4939 scope.go:122] "RemoveContainer" containerID="339bc7b99b219889b6a9d290a2a36b76ab9c5eb0d5ad09fc64724f2574ac1ff3" Nov 03 01:41:19.234009 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:41:19.233863 4939 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-57079b-openshift-default-5c84d8878f-drtdm" Nov 03 01:41:19.244355 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:41:19.244332 4939 scope.go:122] "RemoveContainer" containerID="a097be29f3ff470550b4dbcb5148e73d2e6ca303080ba91af7a7b5ff2045d80d" Nov 03 01:41:19.255786 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:41:19.255761 4939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a68659b3-912d-4df9-a162-fa6d5f949d2a-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a68659b3-912d-4df9-a162-fa6d5f949d2a-istio-envoy\") pod \"a68659b3-912d-4df9-a162-fa6d5f949d2a\" (UID: \"a68659b3-912d-4df9-a162-fa6d5f949d2a\") " Nov 03 01:41:19.255954 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:41:19.255802 4939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/a68659b3-912d-4df9-a162-fa6d5f949d2a-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a68659b3-912d-4df9-a162-fa6d5f949d2a-istio-podinfo\") pod \"a68659b3-912d-4df9-a162-fa6d5f949d2a\" (UID: \"a68659b3-912d-4df9-a162-fa6d5f949d2a\") " Nov 03 01:41:19.255954 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:41:19.255851 4939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a68659b3-912d-4df9-a162-fa6d5f949d2a-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a68659b3-912d-4df9-a162-fa6d5f949d2a-workload-certs\") pod \"a68659b3-912d-4df9-a162-fa6d5f949d2a\" (UID: \"a68659b3-912d-4df9-a162-fa6d5f949d2a\") " Nov 03 01:41:19.256120 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:41:19.255999 4939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a68659b3-912d-4df9-a162-fa6d5f949d2a-istio-token\" (UniqueName: \"kubernetes.io/projected/a68659b3-912d-4df9-a162-fa6d5f949d2a-istio-token\") pod \"a68659b3-912d-4df9-a162-fa6d5f949d2a\" (UID: \"a68659b3-912d-4df9-a162-fa6d5f949d2a\") " Nov 03 01:41:19.256184 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:41:19.256141 4939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a68659b3-912d-4df9-a162-fa6d5f949d2a-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a68659b3-912d-4df9-a162-fa6d5f949d2a-credential-socket\") pod \"a68659b3-912d-4df9-a162-fa6d5f949d2a\" (UID: \"a68659b3-912d-4df9-a162-fa6d5f949d2a\") " Nov 03 01:41:19.256245 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:41:19.256188 4939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a68659b3-912d-4df9-a162-fa6d5f949d2a-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a68659b3-912d-4df9-a162-fa6d5f949d2a-istio-data\") pod \"a68659b3-912d-4df9-a162-fa6d5f949d2a\" (UID: \"a68659b3-912d-4df9-a162-fa6d5f949d2a\") " Nov 03 01:41:19.256245 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:41:19.256229 4939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a68659b3-912d-4df9-a162-fa6d5f949d2a-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a68659b3-912d-4df9-a162-fa6d5f949d2a-istiod-ca-cert\") pod \"a68659b3-912d-4df9-a162-fa6d5f949d2a\" (UID: \"a68659b3-912d-4df9-a162-fa6d5f949d2a\") " Nov 03 01:41:19.256355 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:41:19.256304 4939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a68659b3-912d-4df9-a162-fa6d5f949d2a-kube-api-access-dwwc2\" (UniqueName: \"kubernetes.io/projected/a68659b3-912d-4df9-a162-fa6d5f949d2a-kube-api-access-dwwc2\") pod \"a68659b3-912d-4df9-a162-fa6d5f949d2a\" (UID: \"a68659b3-912d-4df9-a162-fa6d5f949d2a\") " Nov 03 01:41:19.256355 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:41:19.256343 4939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a68659b3-912d-4df9-a162-fa6d5f949d2a-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a68659b3-912d-4df9-a162-fa6d5f949d2a-workload-socket\") pod \"a68659b3-912d-4df9-a162-fa6d5f949d2a\" (UID: \"a68659b3-912d-4df9-a162-fa6d5f949d2a\") " Nov 03 01:41:19.256865 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:41:19.256823 4939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a68659b3-912d-4df9-a162-fa6d5f949d2a-credential-socket" pod "a68659b3-912d-4df9-a162-fa6d5f949d2a" (UID: "a68659b3-912d-4df9-a162-fa6d5f949d2a"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 01:41:19.257049 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:41:19.256990 4939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a68659b3-912d-4df9-a162-fa6d5f949d2a-istiod-ca-cert" pod "a68659b3-912d-4df9-a162-fa6d5f949d2a" (UID: "a68659b3-912d-4df9-a162-fa6d5f949d2a"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 01:41:19.257216 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:41:19.257191 4939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a68659b3-912d-4df9-a162-fa6d5f949d2a-workload-socket" pod "a68659b3-912d-4df9-a162-fa6d5f949d2a" (UID: "a68659b3-912d-4df9-a162-fa6d5f949d2a"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 01:41:19.257312 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:41:19.257269 4939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a68659b3-912d-4df9-a162-fa6d5f949d2a-istio-data" pod "a68659b3-912d-4df9-a162-fa6d5f949d2a" (UID: "a68659b3-912d-4df9-a162-fa6d5f949d2a"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 01:41:19.257768 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:41:19.257746 4939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a68659b3-912d-4df9-a162-fa6d5f949d2a-workload-certs" pod "a68659b3-912d-4df9-a162-fa6d5f949d2a" (UID: "a68659b3-912d-4df9-a162-fa6d5f949d2a"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 01:41:19.258741 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:41:19.258714 4939 scope.go:122] "RemoveContainer" containerID="339bc7b99b219889b6a9d290a2a36b76ab9c5eb0d5ad09fc64724f2574ac1ff3" Nov 03 01:41:19.259174 ip-10-0-140-107 kubenswrapper[4939]: E1103 01:41:19.259144 4939 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"339bc7b99b219889b6a9d290a2a36b76ab9c5eb0d5ad09fc64724f2574ac1ff3\": container with ID starting with 339bc7b99b219889b6a9d290a2a36b76ab9c5eb0d5ad09fc64724f2574ac1ff3 not found: ID does not exist" containerID="339bc7b99b219889b6a9d290a2a36b76ab9c5eb0d5ad09fc64724f2574ac1ff3" Nov 03 01:41:19.259288 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:41:19.259186 4939 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"339bc7b99b219889b6a9d290a2a36b76ab9c5eb0d5ad09fc64724f2574ac1ff3"} err="failed to get container status \"339bc7b99b219889b6a9d290a2a36b76ab9c5eb0d5ad09fc64724f2574ac1ff3\": rpc error: code = NotFound desc = could not find container \"339bc7b99b219889b6a9d290a2a36b76ab9c5eb0d5ad09fc64724f2574ac1ff3\": container with ID starting with 339bc7b99b219889b6a9d290a2a36b76ab9c5eb0d5ad09fc64724f2574ac1ff3 not found: ID does not exist" Nov 03 01:41:19.259288 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:41:19.259213 4939 scope.go:122] "RemoveContainer" containerID="a097be29f3ff470550b4dbcb5148e73d2e6ca303080ba91af7a7b5ff2045d80d" Nov 03 01:41:19.259620 ip-10-0-140-107 kubenswrapper[4939]: E1103 01:41:19.259590 4939 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a097be29f3ff470550b4dbcb5148e73d2e6ca303080ba91af7a7b5ff2045d80d\": container with ID starting with a097be29f3ff470550b4dbcb5148e73d2e6ca303080ba91af7a7b5ff2045d80d not found: ID does not exist" containerID="a097be29f3ff470550b4dbcb5148e73d2e6ca303080ba91af7a7b5ff2045d80d" Nov 03 01:41:19.259698 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:41:19.259630 4939 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a097be29f3ff470550b4dbcb5148e73d2e6ca303080ba91af7a7b5ff2045d80d"} err="failed to get container status \"a097be29f3ff470550b4dbcb5148e73d2e6ca303080ba91af7a7b5ff2045d80d\": rpc error: code = NotFound desc = could not find container \"a097be29f3ff470550b4dbcb5148e73d2e6ca303080ba91af7a7b5ff2045d80d\": container with ID starting with a097be29f3ff470550b4dbcb5148e73d2e6ca303080ba91af7a7b5ff2045d80d not found: ID does not exist" Nov 03 01:41:19.260466 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:41:19.260431 4939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/a68659b3-912d-4df9-a162-fa6d5f949d2a-istio-podinfo" pod "a68659b3-912d-4df9-a162-fa6d5f949d2a" (UID: "a68659b3-912d-4df9-a162-fa6d5f949d2a"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 01:41:19.260794 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:41:19.260741 4939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a68659b3-912d-4df9-a162-fa6d5f949d2a-istio-token" pod "a68659b3-912d-4df9-a162-fa6d5f949d2a" (UID: "a68659b3-912d-4df9-a162-fa6d5f949d2a"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 01:41:19.261820 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:41:19.261790 4939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a68659b3-912d-4df9-a162-fa6d5f949d2a-istio-envoy" pod "a68659b3-912d-4df9-a162-fa6d5f949d2a" (UID: "a68659b3-912d-4df9-a162-fa6d5f949d2a"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 01:41:19.263427 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:41:19.263400 4939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a68659b3-912d-4df9-a162-fa6d5f949d2a-kube-api-access-dwwc2" pod "a68659b3-912d-4df9-a162-fa6d5f949d2a" (UID: "a68659b3-912d-4df9-a162-fa6d5f949d2a"). InnerVolumeSpecName "kube-api-access-dwwc2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 01:41:19.355644 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:41:19.355601 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:41:19.358570 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:41:19.358532 4939 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a68659b3-912d-4df9-a162-fa6d5f949d2a-credential-socket\") on node \"ip-10-0-140-107.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:41:19.358570 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:41:19.358569 4939 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a68659b3-912d-4df9-a162-fa6d5f949d2a-istio-data\") on node \"ip-10-0-140-107.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:41:19.358812 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:41:19.358584 4939 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a68659b3-912d-4df9-a162-fa6d5f949d2a-istiod-ca-cert\") on node \"ip-10-0-140-107.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:41:19.358812 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:41:19.358598 4939 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dwwc2\" (UniqueName: \"kubernetes.io/projected/a68659b3-912d-4df9-a162-fa6d5f949d2a-kube-api-access-dwwc2\") on node \"ip-10-0-140-107.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:41:19.358812 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:41:19.358613 4939 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a68659b3-912d-4df9-a162-fa6d5f949d2a-workload-socket\") on node \"ip-10-0-140-107.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:41:19.358812 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:41:19.358627 4939 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a68659b3-912d-4df9-a162-fa6d5f949d2a-istio-envoy\") on node \"ip-10-0-140-107.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:41:19.358812 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:41:19.358642 4939 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a68659b3-912d-4df9-a162-fa6d5f949d2a-istio-podinfo\") on node \"ip-10-0-140-107.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:41:19.358812 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:41:19.358655 4939 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a68659b3-912d-4df9-a162-fa6d5f949d2a-workload-certs\") on node \"ip-10-0-140-107.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:41:19.358812 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:41:19.358668 4939 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a68659b3-912d-4df9-a162-fa6d5f949d2a-istio-token\") on node \"ip-10-0-140-107.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:41:19.566020 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:41:19.565983 4939 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-b-57079b-openshift-default-5c84d8878f-drtdm"] Nov 03 01:41:19.568964 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:41:19.568932 4939 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-extmod-b-57079b-openshift-default-5c84d8878f-drtdm"] Nov 03 01:41:19.700984 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:41:19.700870 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:41:19.871492 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:41:19.871452 4939 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a68659b3-912d-4df9-a162-fa6d5f949d2a" path="/var/lib/kubelet/pods/a68659b3-912d-4df9-a162-fa6d5f949d2a/volumes" Nov 03 01:41:19.940787 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:41:19.940740 4939 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w2-f7680d/distinct-w2-f7680d-kserve-756f874b98-qnpv8" podUID="836753fc-f9ef-4f61-b3fe-e5b92b7ee27b" containerName="main" containerID="cri-o://d334fd3c1f721eecec1716833695512720ecab73858e5add00260032237f4f93" gracePeriod=45 Nov 03 01:41:19.946101 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:41:19.945696 4939 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w2-f7680d/distinct-2-w2-f7680d-kserve-7d7c4564bf-cbv2c" podUID="2de18086-761b-4773-b9fc-15ddc4a220fc" containerName="main" containerID="cri-o://e47719852a6e462cb0947c01e78ec27e6a6dfa8d40a4f01271c3a12a16113800" gracePeriod=45 Nov 03 01:41:19.965019 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:41:19.964885 4939 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w2-f7680d/premium-w2-f7680d-kserve-76565b8cf9-hqbbt" podUID="4712e708-18a2-4a13-a12c-a5669397b2c3" containerName="main" containerID="cri-o://031486de41eb4ed29f41d79f4243ab5a68a82bb031b3d1042ca44cd3d215aa04" gracePeriod=45 Nov 03 01:41:20.076850 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:41:20.076814 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:41:20.244386 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:41:20.244294 4939 generic.go:356] "Generic (PLEG): container finished" podID="2de18086-761b-4773-b9fc-15ddc4a220fc" containerID="e47719852a6e462cb0947c01e78ec27e6a6dfa8d40a4f01271c3a12a16113800" exitCode=0 Nov 03 01:41:20.244627 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:41:20.244459 4939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-f7680d/distinct-2-w2-f7680d-kserve-7d7c4564bf-cbv2c" event={"ID":"2de18086-761b-4773-b9fc-15ddc4a220fc","Type":"ContainerDied","Data":"e47719852a6e462cb0947c01e78ec27e6a6dfa8d40a4f01271c3a12a16113800"} Nov 03 01:41:20.247869 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:41:20.247844 4939 generic.go:356] "Generic (PLEG): container finished" podID="4712e708-18a2-4a13-a12c-a5669397b2c3" containerID="031486de41eb4ed29f41d79f4243ab5a68a82bb031b3d1042ca44cd3d215aa04" exitCode=0 Nov 03 01:41:20.248035 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:41:20.247881 4939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-f7680d/premium-w2-f7680d-kserve-76565b8cf9-hqbbt" event={"ID":"4712e708-18a2-4a13-a12c-a5669397b2c3","Type":"ContainerDied","Data":"031486de41eb4ed29f41d79f4243ab5a68a82bb031b3d1042ca44cd3d215aa04"} Nov 03 01:41:20.249879 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:41:20.249854 4939 generic.go:356] "Generic (PLEG): container finished" podID="836753fc-f9ef-4f61-b3fe-e5b92b7ee27b" containerID="d334fd3c1f721eecec1716833695512720ecab73858e5add00260032237f4f93" exitCode=0 Nov 03 01:41:20.250027 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:41:20.249885 4939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-f7680d/distinct-w2-f7680d-kserve-756f874b98-qnpv8" event={"ID":"836753fc-f9ef-4f61-b3fe-e5b92b7ee27b","Type":"ContainerDied","Data":"d334fd3c1f721eecec1716833695512720ecab73858e5add00260032237f4f93"} Nov 03 01:41:20.285051 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:41:20.285025 4939 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-f7680d/distinct-2-w2-f7680d-kserve-7d7c4564bf-cbv2c" Nov 03 01:41:20.310429 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:41:20.309939 4939 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-f7680d/premium-w2-f7680d-kserve-76565b8cf9-hqbbt" Nov 03 01:41:20.349494 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:41:20.349451 4939 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-extmod-b-57079b-5f4958f857-v8cxd" podUID="5d25771d-8931-4840-86be-9ebaf92fafca" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 03 01:41:20.359442 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:41:20.359404 4939 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-f7680d/distinct-w2-f7680d-kserve-756f874b98-qnpv8" Nov 03 01:41:20.377840 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:41:20.377802 4939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2de18086-761b-4773-b9fc-15ddc4a220fc-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2de18086-761b-4773-b9fc-15ddc4a220fc-model-cache\") pod \"2de18086-761b-4773-b9fc-15ddc4a220fc\" (UID: \"2de18086-761b-4773-b9fc-15ddc4a220fc\") " Nov 03 01:41:20.378057 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:41:20.377854 4939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2de18086-761b-4773-b9fc-15ddc4a220fc-home\" (UniqueName: \"kubernetes.io/empty-dir/2de18086-761b-4773-b9fc-15ddc4a220fc-home\") pod \"2de18086-761b-4773-b9fc-15ddc4a220fc\" (UID: \"2de18086-761b-4773-b9fc-15ddc4a220fc\") " Nov 03 01:41:20.378180 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:41:20.378078 4939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2de18086-761b-4773-b9fc-15ddc4a220fc-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2de18086-761b-4773-b9fc-15ddc4a220fc-tmp-dir\") pod \"2de18086-761b-4773-b9fc-15ddc4a220fc\" (UID: \"2de18086-761b-4773-b9fc-15ddc4a220fc\") " Nov 03 01:41:20.378180 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:41:20.378124 4939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2de18086-761b-4773-b9fc-15ddc4a220fc-kube-api-access-zjb7h\" (UniqueName: \"kubernetes.io/projected/2de18086-761b-4773-b9fc-15ddc4a220fc-kube-api-access-zjb7h\") pod \"2de18086-761b-4773-b9fc-15ddc4a220fc\" (UID: \"2de18086-761b-4773-b9fc-15ddc4a220fc\") " Nov 03 01:41:20.378180 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:41:20.378167 4939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2de18086-761b-4773-b9fc-15ddc4a220fc-tls-certs\" (UniqueName: \"kubernetes.io/secret/2de18086-761b-4773-b9fc-15ddc4a220fc-tls-certs\") pod \"2de18086-761b-4773-b9fc-15ddc4a220fc\" (UID: \"2de18086-761b-4773-b9fc-15ddc4a220fc\") " Nov 03 01:41:20.378465 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:41:20.378444 4939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2de18086-761b-4773-b9fc-15ddc4a220fc-dshm\" (UniqueName: \"kubernetes.io/empty-dir/2de18086-761b-4773-b9fc-15ddc4a220fc-dshm\") pod \"2de18086-761b-4773-b9fc-15ddc4a220fc\" (UID: \"2de18086-761b-4773-b9fc-15ddc4a220fc\") " Nov 03 01:41:20.384440 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:41:20.381787 4939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2de18086-761b-4773-b9fc-15ddc4a220fc-home" pod "2de18086-761b-4773-b9fc-15ddc4a220fc" (UID: "2de18086-761b-4773-b9fc-15ddc4a220fc"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 01:41:20.384440 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:41:20.381809 4939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2de18086-761b-4773-b9fc-15ddc4a220fc-tmp-dir" pod "2de18086-761b-4773-b9fc-15ddc4a220fc" (UID: "2de18086-761b-4773-b9fc-15ddc4a220fc"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 01:41:20.384440 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:41:20.382362 4939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2de18086-761b-4773-b9fc-15ddc4a220fc-model-cache" pod "2de18086-761b-4773-b9fc-15ddc4a220fc" (UID: "2de18086-761b-4773-b9fc-15ddc4a220fc"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 01:41:20.386350 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:41:20.386275 4939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2de18086-761b-4773-b9fc-15ddc4a220fc-dshm" pod "2de18086-761b-4773-b9fc-15ddc4a220fc" (UID: "2de18086-761b-4773-b9fc-15ddc4a220fc"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 01:41:20.388991 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:41:20.388963 4939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2de18086-761b-4773-b9fc-15ddc4a220fc-tls-certs" pod "2de18086-761b-4773-b9fc-15ddc4a220fc" (UID: "2de18086-761b-4773-b9fc-15ddc4a220fc"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 01:41:20.397531 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:41:20.397506 4939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2de18086-761b-4773-b9fc-15ddc4a220fc-kube-api-access-zjb7h" pod "2de18086-761b-4773-b9fc-15ddc4a220fc" (UID: "2de18086-761b-4773-b9fc-15ddc4a220fc"). InnerVolumeSpecName "kube-api-access-zjb7h". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 01:41:20.423833 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:41:20.423774 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:41:20.479992 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:41:20.479880 4939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4712e708-18a2-4a13-a12c-a5669397b2c3-kube-api-access-c6lhc\" (UniqueName: \"kubernetes.io/projected/4712e708-18a2-4a13-a12c-a5669397b2c3-kube-api-access-c6lhc\") pod \"4712e708-18a2-4a13-a12c-a5669397b2c3\" (UID: \"4712e708-18a2-4a13-a12c-a5669397b2c3\") " Nov 03 01:41:20.479992 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:41:20.479965 4939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/836753fc-f9ef-4f61-b3fe-e5b92b7ee27b-tls-certs\" (UniqueName: \"kubernetes.io/secret/836753fc-f9ef-4f61-b3fe-e5b92b7ee27b-tls-certs\") pod \"836753fc-f9ef-4f61-b3fe-e5b92b7ee27b\" (UID: \"836753fc-f9ef-4f61-b3fe-e5b92b7ee27b\") " Nov 03 01:41:20.480247 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:41:20.480002 4939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/836753fc-f9ef-4f61-b3fe-e5b92b7ee27b-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/836753fc-f9ef-4f61-b3fe-e5b92b7ee27b-model-cache\") pod \"836753fc-f9ef-4f61-b3fe-e5b92b7ee27b\" (UID: \"836753fc-f9ef-4f61-b3fe-e5b92b7ee27b\") " Nov 03 01:41:20.480247 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:41:20.480026 4939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/836753fc-f9ef-4f61-b3fe-e5b92b7ee27b-kube-api-access-jcp5z\" (UniqueName: \"kubernetes.io/projected/836753fc-f9ef-4f61-b3fe-e5b92b7ee27b-kube-api-access-jcp5z\") pod \"836753fc-f9ef-4f61-b3fe-e5b92b7ee27b\" (UID: \"836753fc-f9ef-4f61-b3fe-e5b92b7ee27b\") " Nov 03 01:41:20.480247 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:41:20.480062 4939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4712e708-18a2-4a13-a12c-a5669397b2c3-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4712e708-18a2-4a13-a12c-a5669397b2c3-model-cache\") pod \"4712e708-18a2-4a13-a12c-a5669397b2c3\" (UID: \"4712e708-18a2-4a13-a12c-a5669397b2c3\") " Nov 03 01:41:20.480247 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:41:20.480085 4939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/836753fc-f9ef-4f61-b3fe-e5b92b7ee27b-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/836753fc-f9ef-4f61-b3fe-e5b92b7ee27b-tmp-dir\") pod \"836753fc-f9ef-4f61-b3fe-e5b92b7ee27b\" (UID: \"836753fc-f9ef-4f61-b3fe-e5b92b7ee27b\") " Nov 03 01:41:20.480247 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:41:20.480109 4939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4712e708-18a2-4a13-a12c-a5669397b2c3-dshm\" (UniqueName: \"kubernetes.io/empty-dir/4712e708-18a2-4a13-a12c-a5669397b2c3-dshm\") pod \"4712e708-18a2-4a13-a12c-a5669397b2c3\" (UID: \"4712e708-18a2-4a13-a12c-a5669397b2c3\") " Nov 03 01:41:20.480578 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:41:20.480267 4939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4712e708-18a2-4a13-a12c-a5669397b2c3-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4712e708-18a2-4a13-a12c-a5669397b2c3-tmp-dir\") pod \"4712e708-18a2-4a13-a12c-a5669397b2c3\" (UID: \"4712e708-18a2-4a13-a12c-a5669397b2c3\") " Nov 03 01:41:20.480578 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:41:20.480327 4939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4712e708-18a2-4a13-a12c-a5669397b2c3-home\" (UniqueName: \"kubernetes.io/empty-dir/4712e708-18a2-4a13-a12c-a5669397b2c3-home\") pod \"4712e708-18a2-4a13-a12c-a5669397b2c3\" (UID: \"4712e708-18a2-4a13-a12c-a5669397b2c3\") " Nov 03 01:41:20.480578 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:41:20.480364 4939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/836753fc-f9ef-4f61-b3fe-e5b92b7ee27b-home\" (UniqueName: \"kubernetes.io/empty-dir/836753fc-f9ef-4f61-b3fe-e5b92b7ee27b-home\") pod \"836753fc-f9ef-4f61-b3fe-e5b92b7ee27b\" (UID: \"836753fc-f9ef-4f61-b3fe-e5b92b7ee27b\") " Nov 03 01:41:20.480578 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:41:20.480402 4939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/836753fc-f9ef-4f61-b3fe-e5b92b7ee27b-dshm\" (UniqueName: \"kubernetes.io/empty-dir/836753fc-f9ef-4f61-b3fe-e5b92b7ee27b-dshm\") pod \"836753fc-f9ef-4f61-b3fe-e5b92b7ee27b\" (UID: \"836753fc-f9ef-4f61-b3fe-e5b92b7ee27b\") " Nov 03 01:41:20.480578 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:41:20.480440 4939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4712e708-18a2-4a13-a12c-a5669397b2c3-tls-certs\" (UniqueName: \"kubernetes.io/secret/4712e708-18a2-4a13-a12c-a5669397b2c3-tls-certs\") pod \"4712e708-18a2-4a13-a12c-a5669397b2c3\" (UID: \"4712e708-18a2-4a13-a12c-a5669397b2c3\") " Nov 03 01:41:20.480890 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:41:20.480610 4939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/836753fc-f9ef-4f61-b3fe-e5b92b7ee27b-tmp-dir" pod "836753fc-f9ef-4f61-b3fe-e5b92b7ee27b" (UID: "836753fc-f9ef-4f61-b3fe-e5b92b7ee27b"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 01:41:20.481342 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:41:20.481045 4939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4712e708-18a2-4a13-a12c-a5669397b2c3-tmp-dir" pod "4712e708-18a2-4a13-a12c-a5669397b2c3" (UID: "4712e708-18a2-4a13-a12c-a5669397b2c3"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 01:41:20.481342 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:41:20.481073 4939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/836753fc-f9ef-4f61-b3fe-e5b92b7ee27b-model-cache" pod "836753fc-f9ef-4f61-b3fe-e5b92b7ee27b" (UID: "836753fc-f9ef-4f61-b3fe-e5b92b7ee27b"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 01:41:20.481728 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:41:20.481658 4939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/836753fc-f9ef-4f61-b3fe-e5b92b7ee27b-home" pod "836753fc-f9ef-4f61-b3fe-e5b92b7ee27b" (UID: "836753fc-f9ef-4f61-b3fe-e5b92b7ee27b"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 01:41:20.481840 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:41:20.481751 4939 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/836753fc-f9ef-4f61-b3fe-e5b92b7ee27b-model-cache\") on node \"ip-10-0-140-107.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:41:20.481840 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:41:20.481775 4939 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/836753fc-f9ef-4f61-b3fe-e5b92b7ee27b-tmp-dir\") on node \"ip-10-0-140-107.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:41:20.481840 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:41:20.481791 4939 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2de18086-761b-4773-b9fc-15ddc4a220fc-dshm\") on node \"ip-10-0-140-107.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:41:20.481840 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:41:20.481805 4939 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2de18086-761b-4773-b9fc-15ddc4a220fc-model-cache\") on node \"ip-10-0-140-107.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:41:20.481840 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:41:20.481820 4939 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2de18086-761b-4773-b9fc-15ddc4a220fc-home\") on node \"ip-10-0-140-107.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:41:20.482893 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:41:20.481848 4939 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4712e708-18a2-4a13-a12c-a5669397b2c3-tmp-dir\") on node \"ip-10-0-140-107.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:41:20.482893 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:41:20.481863 4939 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2de18086-761b-4773-b9fc-15ddc4a220fc-tmp-dir\") on node \"ip-10-0-140-107.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:41:20.482893 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:41:20.481879 4939 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zjb7h\" (UniqueName: \"kubernetes.io/projected/2de18086-761b-4773-b9fc-15ddc4a220fc-kube-api-access-zjb7h\") on node \"ip-10-0-140-107.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:41:20.482893 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:41:20.482444 4939 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2de18086-761b-4773-b9fc-15ddc4a220fc-tls-certs\") on node \"ip-10-0-140-107.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:41:20.482893 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:41:20.482483 4939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4712e708-18a2-4a13-a12c-a5669397b2c3-home" pod "4712e708-18a2-4a13-a12c-a5669397b2c3" (UID: "4712e708-18a2-4a13-a12c-a5669397b2c3"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 01:41:20.482893 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:41:20.482862 4939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4712e708-18a2-4a13-a12c-a5669397b2c3-kube-api-access-c6lhc" pod "4712e708-18a2-4a13-a12c-a5669397b2c3" (UID: "4712e708-18a2-4a13-a12c-a5669397b2c3"). InnerVolumeSpecName "kube-api-access-c6lhc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 01:41:20.483374 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:41:20.483342 4939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/836753fc-f9ef-4f61-b3fe-e5b92b7ee27b-kube-api-access-jcp5z" pod "836753fc-f9ef-4f61-b3fe-e5b92b7ee27b" (UID: "836753fc-f9ef-4f61-b3fe-e5b92b7ee27b"). InnerVolumeSpecName "kube-api-access-jcp5z". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 01:41:20.483524 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:41:20.483500 4939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/836753fc-f9ef-4f61-b3fe-e5b92b7ee27b-tls-certs" pod "836753fc-f9ef-4f61-b3fe-e5b92b7ee27b" (UID: "836753fc-f9ef-4f61-b3fe-e5b92b7ee27b"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 01:41:20.484007 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:41:20.483975 4939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4712e708-18a2-4a13-a12c-a5669397b2c3-tls-certs" pod "4712e708-18a2-4a13-a12c-a5669397b2c3" (UID: "4712e708-18a2-4a13-a12c-a5669397b2c3"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 01:41:20.484593 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:41:20.484566 4939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4712e708-18a2-4a13-a12c-a5669397b2c3-dshm" pod "4712e708-18a2-4a13-a12c-a5669397b2c3" (UID: "4712e708-18a2-4a13-a12c-a5669397b2c3"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 01:41:20.491081 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:41:20.491044 4939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/836753fc-f9ef-4f61-b3fe-e5b92b7ee27b-dshm" pod "836753fc-f9ef-4f61-b3fe-e5b92b7ee27b" (UID: "836753fc-f9ef-4f61-b3fe-e5b92b7ee27b"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 01:41:20.492121 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:41:20.492091 4939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4712e708-18a2-4a13-a12c-a5669397b2c3-model-cache" pod "4712e708-18a2-4a13-a12c-a5669397b2c3" (UID: "4712e708-18a2-4a13-a12c-a5669397b2c3"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 01:41:20.500548 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:41:20.500521 4939 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w4-6c0ec0-7474bf5fc5-dhqxc"] Nov 03 01:41:20.514962 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:41:20.514928 4939 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w4-6c0ec0-65c5d46767-hsxz4"] Nov 03 01:41:20.583616 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:41:20.583582 4939 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4712e708-18a2-4a13-a12c-a5669397b2c3-home\") on node \"ip-10-0-140-107.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:41:20.583616 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:41:20.583613 4939 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/836753fc-f9ef-4f61-b3fe-e5b92b7ee27b-home\") on node \"ip-10-0-140-107.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:41:20.583843 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:41:20.583635 4939 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/836753fc-f9ef-4f61-b3fe-e5b92b7ee27b-dshm\") on node \"ip-10-0-140-107.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:41:20.583843 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:41:20.583646 4939 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4712e708-18a2-4a13-a12c-a5669397b2c3-tls-certs\") on node \"ip-10-0-140-107.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:41:20.583843 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:41:20.583661 4939 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-c6lhc\" (UniqueName: \"kubernetes.io/projected/4712e708-18a2-4a13-a12c-a5669397b2c3-kube-api-access-c6lhc\") on node \"ip-10-0-140-107.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:41:20.583843 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:41:20.583675 4939 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/836753fc-f9ef-4f61-b3fe-e5b92b7ee27b-tls-certs\") on node \"ip-10-0-140-107.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:41:20.583843 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:41:20.583688 4939 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jcp5z\" (UniqueName: \"kubernetes.io/projected/836753fc-f9ef-4f61-b3fe-e5b92b7ee27b-kube-api-access-jcp5z\") on node \"ip-10-0-140-107.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:41:20.583843 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:41:20.583704 4939 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4712e708-18a2-4a13-a12c-a5669397b2c3-model-cache\") on node \"ip-10-0-140-107.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:41:20.583843 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:41:20.583720 4939 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4712e708-18a2-4a13-a12c-a5669397b2c3-dshm\") on node \"ip-10-0-140-107.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:41:20.600583 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:41:20.600547 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:41:20.711873 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:41:20.711838 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:41:21.184346 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:41:21.184313 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:41:21.259994 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:41:21.259861 4939 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-f7680d/premium-w2-f7680d-kserve-76565b8cf9-hqbbt" Nov 03 01:41:21.259994 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:41:21.259891 4939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-f7680d/premium-w2-f7680d-kserve-76565b8cf9-hqbbt" event={"ID":"4712e708-18a2-4a13-a12c-a5669397b2c3","Type":"ContainerDied","Data":"6789d8930e178842740e13fd1a143c9f06fb93885382f253e83b404486842a22"} Nov 03 01:41:21.259994 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:41:21.259952 4939 scope.go:122] "RemoveContainer" containerID="031486de41eb4ed29f41d79f4243ab5a68a82bb031b3d1042ca44cd3d215aa04" Nov 03 01:41:21.261724 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:41:21.261693 4939 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-f7680d/distinct-w2-f7680d-kserve-756f874b98-qnpv8" Nov 03 01:41:21.261724 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:41:21.261702 4939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-f7680d/distinct-w2-f7680d-kserve-756f874b98-qnpv8" event={"ID":"836753fc-f9ef-4f61-b3fe-e5b92b7ee27b","Type":"ContainerDied","Data":"8b3d5b6db611466c6dc5c2d2e82dc2be9814cac4ab327a523754b0dc99f2e1dd"} Nov 03 01:41:21.263279 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:41:21.263253 4939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-f7680d/distinct-2-w2-f7680d-kserve-7d7c4564bf-cbv2c" event={"ID":"2de18086-761b-4773-b9fc-15ddc4a220fc","Type":"ContainerDied","Data":"298e21e863aa9fdf46bfe9391a1719c66f790b930026380395fca1218f96e34d"} Nov 03 01:41:21.263485 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:41:21.263290 4939 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-f7680d/distinct-2-w2-f7680d-kserve-7d7c4564bf-cbv2c" Nov 03 01:41:21.272347 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:41:21.272328 4939 scope.go:122] "RemoveContainer" containerID="d334fd3c1f721eecec1716833695512720ecab73858e5add00260032237f4f93" Nov 03 01:41:21.281869 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:41:21.281847 4939 scope.go:122] "RemoveContainer" containerID="e47719852a6e462cb0947c01e78ec27e6a6dfa8d40a4f01271c3a12a16113800" Nov 03 01:41:21.290269 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:41:21.290247 4939 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w2-f7680d/premium-w2-f7680d-kserve-76565b8cf9-hqbbt"] Nov 03 01:41:21.296168 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:41:21.296143 4939 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w2-f7680d/premium-w2-f7680d-kserve-76565b8cf9-hqbbt"] Nov 03 01:41:21.308058 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:41:21.308035 4939 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w2-f7680d/distinct-2-w2-f7680d-kserve-7d7c4564bf-cbv2c"] Nov 03 01:41:21.310517 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:41:21.310494 4939 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w2-f7680d/distinct-2-w2-f7680d-kserve-7d7c4564bf-cbv2c"] Nov 03 01:41:21.322955 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:41:21.322927 4939 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w2-f7680d/distinct-w2-f7680d-kserve-756f874b98-qnpv8"] Nov 03 01:41:21.326953 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:41:21.326928 4939 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w2-f7680d/distinct-w2-f7680d-kserve-756f874b98-qnpv8"] Nov 03 01:41:21.348096 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:41:21.348052 4939 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-f7680d/distinct-w2-f7680d-kserve-756f874b98-qnpv8" podUID="836753fc-f9ef-4f61-b3fe-e5b92b7ee27b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.70:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 01:41:21.453313 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:41:21.453283 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:41:21.501475 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:41:21.501438 4939 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-57079b-666f95c947-dv7mr" podUID="a79df098-23e4-4971-9263-3f52b7d25b0b" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 03 01:41:21.869819 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:41:21.869783 4939 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2de18086-761b-4773-b9fc-15ddc4a220fc" path="/var/lib/kubelet/pods/2de18086-761b-4773-b9fc-15ddc4a220fc/volumes" Nov 03 01:41:21.870342 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:41:21.870321 4939 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4712e708-18a2-4a13-a12c-a5669397b2c3" path="/var/lib/kubelet/pods/4712e708-18a2-4a13-a12c-a5669397b2c3/volumes" Nov 03 01:41:21.870762 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:41:21.870749 4939 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="836753fc-f9ef-4f61-b3fe-e5b92b7ee27b" path="/var/lib/kubelet/pods/836753fc-f9ef-4f61-b3fe-e5b92b7ee27b/volumes" Nov 03 01:41:21.895110 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:41:21.895083 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:41:22.414840 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:41:22.414782 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:41:22.765175 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:41:22.765077 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:41:23.500446 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:41:23.500410 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:41:24.188927 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:41:24.188885 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:41:24.780893 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:41:24.780860 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:41:25.452373 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:41:25.452342 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:41:25.500975 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:41:25.500934 4939 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-6c0ec0-7474bf5fc5-dhqxc" podUID="c6de1910-0365-4f48-99f4-b32aebd747bb" containerName="payload-processing" containerID="cri-o://94803b27fbf936d8d29f1d086acc91666547bca0730a0e5ec7c0362f7887771a" gracePeriod=25 Nov 03 01:41:25.515386 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:41:25.515349 4939 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-6c0ec0-65c5d46767-hsxz4" podUID="61b34e6e-2690-4080-bced-3a577c665f12" containerName="payload-pre-processing" containerID="cri-o://4953379acfa82f023c872f91888a0f003a2e7fc293fc9ccc0020117a5946d212" gracePeriod=25 Nov 03 01:41:25.813363 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:41:25.813248 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:41:25.947858 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:41:25.947803 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:41:26.504994 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:41:26.504929 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:41:26.943934 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:41:26.943892 4939 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-6c0ec0-fbd7fd95-chmfz"] Nov 03 01:41:26.944652 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:41:26.944618 4939 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-6c0ec0-fbd7fd95-chmfz" podUID="4bcdc6fd-f0da-46e3-bd48-14b1179bb3f1" containerName="maas-api" containerID="cri-o://50bf3e38ce34a7c755e5054daa66036ac27246669c747a22296823c86da51b3e" gracePeriod=30 Nov 03 01:41:27.007483 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:41:27.007451 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:41:27.219003 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:41:27.218501 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:41:27.281196 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:41:27.280861 4939 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-6c0ec0-7474bf5fc5-dhqxc" podUID="c6de1910-0365-4f48-99f4-b32aebd747bb" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 03 01:41:27.310263 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:41:27.310229 4939 generic.go:356] "Generic (PLEG): container finished" podID="4bcdc6fd-f0da-46e3-bd48-14b1179bb3f1" containerID="50bf3e38ce34a7c755e5054daa66036ac27246669c747a22296823c86da51b3e" exitCode=0 Nov 03 01:41:27.310395 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:41:27.310297 4939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-6c0ec0-fbd7fd95-chmfz" event={"ID":"4bcdc6fd-f0da-46e3-bd48-14b1179bb3f1","Type":"ContainerDied","Data":"50bf3e38ce34a7c755e5054daa66036ac27246669c747a22296823c86da51b3e"} Nov 03 01:41:27.329807 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:41:27.329781 4939 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-6c0ec0-fbd7fd95-chmfz" Nov 03 01:41:27.403219 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:41:27.403186 4939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4bcdc6fd-f0da-46e3-bd48-14b1179bb3f1-metrics-certs\" (UniqueName: \"kubernetes.io/secret/4bcdc6fd-f0da-46e3-bd48-14b1179bb3f1-metrics-certs\") pod \"4bcdc6fd-f0da-46e3-bd48-14b1179bb3f1\" (UID: \"4bcdc6fd-f0da-46e3-bd48-14b1179bb3f1\") " Nov 03 01:41:27.403395 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:41:27.403228 4939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4bcdc6fd-f0da-46e3-bd48-14b1179bb3f1-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4bcdc6fd-f0da-46e3-bd48-14b1179bb3f1-openshift-service-ca\") pod \"4bcdc6fd-f0da-46e3-bd48-14b1179bb3f1\" (UID: \"4bcdc6fd-f0da-46e3-bd48-14b1179bb3f1\") " Nov 03 01:41:27.403395 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:41:27.403295 4939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4bcdc6fd-f0da-46e3-bd48-14b1179bb3f1-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4bcdc6fd-f0da-46e3-bd48-14b1179bb3f1-maas-api-tls\") pod \"4bcdc6fd-f0da-46e3-bd48-14b1179bb3f1\" (UID: \"4bcdc6fd-f0da-46e3-bd48-14b1179bb3f1\") " Nov 03 01:41:27.403561 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:41:27.403453 4939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4bcdc6fd-f0da-46e3-bd48-14b1179bb3f1-kube-api-access-k22mq\" (UniqueName: \"kubernetes.io/projected/4bcdc6fd-f0da-46e3-bd48-14b1179bb3f1-kube-api-access-k22mq\") pod \"4bcdc6fd-f0da-46e3-bd48-14b1179bb3f1\" (UID: \"4bcdc6fd-f0da-46e3-bd48-14b1179bb3f1\") " Nov 03 01:41:27.403716 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:41:27.403679 4939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4bcdc6fd-f0da-46e3-bd48-14b1179bb3f1-openshift-service-ca" pod "4bcdc6fd-f0da-46e3-bd48-14b1179bb3f1" (UID: "4bcdc6fd-f0da-46e3-bd48-14b1179bb3f1"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 01:41:27.404051 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:41:27.404032 4939 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4bcdc6fd-f0da-46e3-bd48-14b1179bb3f1-openshift-service-ca\") on node \"ip-10-0-140-107.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:41:27.405675 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:41:27.405651 4939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4bcdc6fd-f0da-46e3-bd48-14b1179bb3f1-kube-api-access-k22mq" pod "4bcdc6fd-f0da-46e3-bd48-14b1179bb3f1" (UID: "4bcdc6fd-f0da-46e3-bd48-14b1179bb3f1"). InnerVolumeSpecName "kube-api-access-k22mq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 01:41:27.405978 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:41:27.405957 4939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4bcdc6fd-f0da-46e3-bd48-14b1179bb3f1-maas-api-tls" pod "4bcdc6fd-f0da-46e3-bd48-14b1179bb3f1" (UID: "4bcdc6fd-f0da-46e3-bd48-14b1179bb3f1"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 01:41:27.406080 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:41:27.405998 4939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4bcdc6fd-f0da-46e3-bd48-14b1179bb3f1-metrics-certs" pod "4bcdc6fd-f0da-46e3-bd48-14b1179bb3f1" (UID: "4bcdc6fd-f0da-46e3-bd48-14b1179bb3f1"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 01:41:27.505954 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:41:27.505856 4939 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k22mq\" (UniqueName: \"kubernetes.io/projected/4bcdc6fd-f0da-46e3-bd48-14b1179bb3f1-kube-api-access-k22mq\") on node \"ip-10-0-140-107.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:41:27.505954 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:41:27.505888 4939 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4bcdc6fd-f0da-46e3-bd48-14b1179bb3f1-metrics-certs\") on node \"ip-10-0-140-107.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:41:27.505954 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:41:27.505897 4939 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4bcdc6fd-f0da-46e3-bd48-14b1179bb3f1-maas-api-tls\") on node \"ip-10-0-140-107.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:41:27.512200 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:41:27.512172 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:41:27.811863 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:41:27.811766 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:41:27.921336 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:41:27.921304 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:41:28.037596 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:41:28.037562 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:41:28.139660 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:41:28.139592 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:41:28.277769 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:41:28.277724 4939 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-6c0ec0-65c5d46767-hsxz4" podUID="61b34e6e-2690-4080-bced-3a577c665f12" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 03 01:41:28.320539 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:41:28.320476 4939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-6c0ec0-fbd7fd95-chmfz" event={"ID":"4bcdc6fd-f0da-46e3-bd48-14b1179bb3f1","Type":"ContainerDied","Data":"e27acbe07b46892600b9c3515e3df96927a20de1f12ff69da30c657f8a3fcec8"} Nov 03 01:41:28.320744 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:41:28.320584 4939 scope.go:122] "RemoveContainer" containerID="50bf3e38ce34a7c755e5054daa66036ac27246669c747a22296823c86da51b3e" Nov 03 01:41:28.320827 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:41:28.320762 4939 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-6c0ec0-fbd7fd95-chmfz" Nov 03 01:41:28.342500 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:41:28.342463 4939 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-6c0ec0-fbd7fd95-chmfz"] Nov 03 01:41:28.345302 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:41:28.345277 4939 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-6c0ec0-fbd7fd95-chmfz"] Nov 03 01:41:28.380665 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:41:28.380631 4939 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-717217-54c569b974-sxbg4"] Nov 03 01:41:28.380900 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:41:28.380874 4939 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-717217-54c569b974-sxbg4" podUID="01da212a-c344-4e25-9f26-ef2224e32b46" containerName="maas-api" containerID="cri-o://3b873459106f64fde040b9ea0ae6399541cb945be8d76a3cc4103114a497ee9c" gracePeriod=30 Nov 03 01:41:28.459889 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:41:28.459855 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:41:28.578525 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:41:28.578385 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:41:28.782700 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:41:28.782665 4939 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-b-57079b-5f4958f857-v8cxd" Nov 03 01:41:28.853972 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:41:28.853831 4939 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w4-6c0ec0-openshift-default-79c7584bfc-kxj54"] Nov 03 01:41:28.854150 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:41:28.854074 4939 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w4-6c0ec0-openshift-default-79c7584bfc-kxj54" podUID="d1ad622d-53f3-48ca-bc88-b7ab4ae452fe" containerName="istio-proxy" containerID="cri-o://f060af1ea2ac13a918c74b9aa73b49a77c6dc196c94c4c14ae9c408564eac77b" gracePeriod=30 Nov 03 01:41:28.885112 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:41:28.885085 4939 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-57079b-666f95c947-dv7mr" Nov 03 01:41:28.887712 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:41:28.887687 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:41:28.891079 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:41:28.891057 4939 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-717217-54c569b974-sxbg4" Nov 03 01:41:28.971221 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:41:28.971191 4939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5d25771d-8931-4840-86be-9ebaf92fafca-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5d25771d-8931-4840-86be-9ebaf92fafca-plugins-config-volume\") pod \"5d25771d-8931-4840-86be-9ebaf92fafca\" (UID: \"5d25771d-8931-4840-86be-9ebaf92fafca\") " Nov 03 01:41:28.971383 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:41:28.971342 4939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5d25771d-8931-4840-86be-9ebaf92fafca-kube-api-access-qzwpl\" (UniqueName: \"kubernetes.io/projected/5d25771d-8931-4840-86be-9ebaf92fafca-kube-api-access-qzwpl\") pod \"5d25771d-8931-4840-86be-9ebaf92fafca\" (UID: \"5d25771d-8931-4840-86be-9ebaf92fafca\") " Nov 03 01:41:28.971561 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:41:28.971539 4939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5d25771d-8931-4840-86be-9ebaf92fafca-plugins-config-volume" pod "5d25771d-8931-4840-86be-9ebaf92fafca" (UID: "5d25771d-8931-4840-86be-9ebaf92fafca"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 01:41:28.972093 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:41:28.972067 4939 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5d25771d-8931-4840-86be-9ebaf92fafca-plugins-config-volume\") on node \"ip-10-0-140-107.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:41:28.973685 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:41:28.973662 4939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5d25771d-8931-4840-86be-9ebaf92fafca-kube-api-access-qzwpl" pod "5d25771d-8931-4840-86be-9ebaf92fafca" (UID: "5d25771d-8931-4840-86be-9ebaf92fafca"). InnerVolumeSpecName "kube-api-access-qzwpl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 01:41:29.010664 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:41:29.010601 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:41:29.076931 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:41:29.076585 4939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/01da212a-c344-4e25-9f26-ef2224e32b46-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/01da212a-c344-4e25-9f26-ef2224e32b46-openshift-service-ca\") pod \"01da212a-c344-4e25-9f26-ef2224e32b46\" (UID: \"01da212a-c344-4e25-9f26-ef2224e32b46\") " Nov 03 01:41:29.076931 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:41:29.076672 4939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/01da212a-c344-4e25-9f26-ef2224e32b46-kube-api-access-7rwmk\" (UniqueName: \"kubernetes.io/projected/01da212a-c344-4e25-9f26-ef2224e32b46-kube-api-access-7rwmk\") pod \"01da212a-c344-4e25-9f26-ef2224e32b46\" (UID: \"01da212a-c344-4e25-9f26-ef2224e32b46\") " Nov 03 01:41:29.076931 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:41:29.076825 4939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/01da212a-c344-4e25-9f26-ef2224e32b46-metrics-certs\" (UniqueName: \"kubernetes.io/secret/01da212a-c344-4e25-9f26-ef2224e32b46-metrics-certs\") pod \"01da212a-c344-4e25-9f26-ef2224e32b46\" (UID: \"01da212a-c344-4e25-9f26-ef2224e32b46\") " Nov 03 01:41:29.076931 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:41:29.076882 4939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a79df098-23e4-4971-9263-3f52b7d25b0b-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a79df098-23e4-4971-9263-3f52b7d25b0b-plugins-config-volume\") pod \"a79df098-23e4-4971-9263-3f52b7d25b0b\" (UID: \"a79df098-23e4-4971-9263-3f52b7d25b0b\") " Nov 03 01:41:29.076931 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:41:29.076936 4939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/01da212a-c344-4e25-9f26-ef2224e32b46-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/01da212a-c344-4e25-9f26-ef2224e32b46-maas-api-tls\") pod \"01da212a-c344-4e25-9f26-ef2224e32b46\" (UID: \"01da212a-c344-4e25-9f26-ef2224e32b46\") " Nov 03 01:41:29.077661 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:41:29.077048 4939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a79df098-23e4-4971-9263-3f52b7d25b0b-kube-api-access-29ptd\" (UniqueName: \"kubernetes.io/projected/a79df098-23e4-4971-9263-3f52b7d25b0b-kube-api-access-29ptd\") pod \"a79df098-23e4-4971-9263-3f52b7d25b0b\" (UID: \"a79df098-23e4-4971-9263-3f52b7d25b0b\") " Nov 03 01:41:29.077661 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:41:29.077575 4939 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qzwpl\" (UniqueName: \"kubernetes.io/projected/5d25771d-8931-4840-86be-9ebaf92fafca-kube-api-access-qzwpl\") on node \"ip-10-0-140-107.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:41:29.085192 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:41:29.085161 4939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/01da212a-c344-4e25-9f26-ef2224e32b46-openshift-service-ca" pod "01da212a-c344-4e25-9f26-ef2224e32b46" (UID: "01da212a-c344-4e25-9f26-ef2224e32b46"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 01:41:29.090524 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:41:29.090488 4939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a79df098-23e4-4971-9263-3f52b7d25b0b-kube-api-access-29ptd" pod "a79df098-23e4-4971-9263-3f52b7d25b0b" (UID: "a79df098-23e4-4971-9263-3f52b7d25b0b"). InnerVolumeSpecName "kube-api-access-29ptd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 01:41:29.091248 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:41:29.091135 4939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/01da212a-c344-4e25-9f26-ef2224e32b46-kube-api-access-7rwmk" pod "01da212a-c344-4e25-9f26-ef2224e32b46" (UID: "01da212a-c344-4e25-9f26-ef2224e32b46"). InnerVolumeSpecName "kube-api-access-7rwmk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 01:41:29.091474 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:41:29.091445 4939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a79df098-23e4-4971-9263-3f52b7d25b0b-plugins-config-volume" pod "a79df098-23e4-4971-9263-3f52b7d25b0b" (UID: "a79df098-23e4-4971-9263-3f52b7d25b0b"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 01:41:29.098090 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:41:29.098063 4939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/01da212a-c344-4e25-9f26-ef2224e32b46-metrics-certs" pod "01da212a-c344-4e25-9f26-ef2224e32b46" (UID: "01da212a-c344-4e25-9f26-ef2224e32b46"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 01:41:29.098241 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:41:29.098126 4939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/01da212a-c344-4e25-9f26-ef2224e32b46-maas-api-tls" pod "01da212a-c344-4e25-9f26-ef2224e32b46" (UID: "01da212a-c344-4e25-9f26-ef2224e32b46"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 01:41:29.178452 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:41:29.178419 4939 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/01da212a-c344-4e25-9f26-ef2224e32b46-metrics-certs\") on node \"ip-10-0-140-107.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:41:29.178452 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:41:29.178450 4939 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a79df098-23e4-4971-9263-3f52b7d25b0b-plugins-config-volume\") on node \"ip-10-0-140-107.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:41:29.178675 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:41:29.178461 4939 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/01da212a-c344-4e25-9f26-ef2224e32b46-maas-api-tls\") on node \"ip-10-0-140-107.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:41:29.178675 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:41:29.178470 4939 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-29ptd\" (UniqueName: \"kubernetes.io/projected/a79df098-23e4-4971-9263-3f52b7d25b0b-kube-api-access-29ptd\") on node \"ip-10-0-140-107.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:41:29.178675 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:41:29.178481 4939 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/01da212a-c344-4e25-9f26-ef2224e32b46-openshift-service-ca\") on node \"ip-10-0-140-107.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:41:29.178675 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:41:29.178489 4939 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7rwmk\" (UniqueName: \"kubernetes.io/projected/01da212a-c344-4e25-9f26-ef2224e32b46-kube-api-access-7rwmk\") on node \"ip-10-0-140-107.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:41:29.328132 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:41:29.328034 4939 generic.go:356] "Generic (PLEG): container finished" podID="01da212a-c344-4e25-9f26-ef2224e32b46" containerID="3b873459106f64fde040b9ea0ae6399541cb945be8d76a3cc4103114a497ee9c" exitCode=0 Nov 03 01:41:29.328132 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:41:29.328079 4939 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-717217-54c569b974-sxbg4" Nov 03 01:41:29.328132 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:41:29.328114 4939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-717217-54c569b974-sxbg4" event={"ID":"01da212a-c344-4e25-9f26-ef2224e32b46","Type":"ContainerDied","Data":"3b873459106f64fde040b9ea0ae6399541cb945be8d76a3cc4103114a497ee9c"} Nov 03 01:41:29.328436 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:41:29.328173 4939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-717217-54c569b974-sxbg4" event={"ID":"01da212a-c344-4e25-9f26-ef2224e32b46","Type":"ContainerDied","Data":"18b46cbc88d7a3e05c2e104e437ac2b344c6b6d5845e221a4cc6247657ca2937"} Nov 03 01:41:29.328436 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:41:29.328188 4939 scope.go:122] "RemoveContainer" containerID="3b873459106f64fde040b9ea0ae6399541cb945be8d76a3cc4103114a497ee9c" Nov 03 01:41:29.329979 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:41:29.329690 4939 generic.go:356] "Generic (PLEG): container finished" podID="5d25771d-8931-4840-86be-9ebaf92fafca" containerID="efe834ec176f778ad2632569bbf778eb71e02e0117cf2037200131ba3869f8fc" exitCode=0 Nov 03 01:41:29.329979 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:41:29.329722 4939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-b-57079b-5f4958f857-v8cxd" event={"ID":"5d25771d-8931-4840-86be-9ebaf92fafca","Type":"ContainerDied","Data":"efe834ec176f778ad2632569bbf778eb71e02e0117cf2037200131ba3869f8fc"} Nov 03 01:41:29.329979 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:41:29.329752 4939 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-b-57079b-5f4958f857-v8cxd" Nov 03 01:41:29.329979 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:41:29.329785 4939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-b-57079b-5f4958f857-v8cxd" event={"ID":"5d25771d-8931-4840-86be-9ebaf92fafca","Type":"ContainerDied","Data":"5101ed9255aef93f6e336f9779a488084a365d3549eec894f950371700ef407b"} Nov 03 01:41:29.332565 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:41:29.332539 4939 generic.go:356] "Generic (PLEG): container finished" podID="a79df098-23e4-4971-9263-3f52b7d25b0b" containerID="7ba5aaa0f3337d62777e32bc996b60a0573018aafc9e436e464208aeb016bc29" exitCode=0 Nov 03 01:41:29.332694 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:41:29.332589 4939 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-57079b-666f95c947-dv7mr" Nov 03 01:41:29.332694 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:41:29.332586 4939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-57079b-666f95c947-dv7mr" event={"ID":"a79df098-23e4-4971-9263-3f52b7d25b0b","Type":"ContainerDied","Data":"7ba5aaa0f3337d62777e32bc996b60a0573018aafc9e436e464208aeb016bc29"} Nov 03 01:41:29.332805 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:41:29.332711 4939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-57079b-666f95c947-dv7mr" event={"ID":"a79df098-23e4-4971-9263-3f52b7d25b0b","Type":"ContainerDied","Data":"1a5d340aead8d2aad8850271e122f1d1e8ffce58e7b4e21bea5299e64e183d5c"} Nov 03 01:41:29.341735 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:41:29.341716 4939 scope.go:122] "RemoveContainer" containerID="3b873459106f64fde040b9ea0ae6399541cb945be8d76a3cc4103114a497ee9c" Nov 03 01:41:29.342068 ip-10-0-140-107 kubenswrapper[4939]: E1103 01:41:29.342047 4939 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3b873459106f64fde040b9ea0ae6399541cb945be8d76a3cc4103114a497ee9c\": container with ID starting with 3b873459106f64fde040b9ea0ae6399541cb945be8d76a3cc4103114a497ee9c not found: ID does not exist" containerID="3b873459106f64fde040b9ea0ae6399541cb945be8d76a3cc4103114a497ee9c" Nov 03 01:41:29.342161 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:41:29.342073 4939 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3b873459106f64fde040b9ea0ae6399541cb945be8d76a3cc4103114a497ee9c"} err="failed to get container status \"3b873459106f64fde040b9ea0ae6399541cb945be8d76a3cc4103114a497ee9c\": rpc error: code = NotFound desc = could not find container \"3b873459106f64fde040b9ea0ae6399541cb945be8d76a3cc4103114a497ee9c\": container with ID starting with 3b873459106f64fde040b9ea0ae6399541cb945be8d76a3cc4103114a497ee9c not found: ID does not exist" Nov 03 01:41:29.342161 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:41:29.342090 4939 scope.go:122] "RemoveContainer" containerID="efe834ec176f778ad2632569bbf778eb71e02e0117cf2037200131ba3869f8fc" Nov 03 01:41:29.351453 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:41:29.351431 4939 scope.go:122] "RemoveContainer" containerID="efe834ec176f778ad2632569bbf778eb71e02e0117cf2037200131ba3869f8fc" Nov 03 01:41:29.351724 ip-10-0-140-107 kubenswrapper[4939]: E1103 01:41:29.351703 4939 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"efe834ec176f778ad2632569bbf778eb71e02e0117cf2037200131ba3869f8fc\": container with ID starting with efe834ec176f778ad2632569bbf778eb71e02e0117cf2037200131ba3869f8fc not found: ID does not exist" containerID="efe834ec176f778ad2632569bbf778eb71e02e0117cf2037200131ba3869f8fc" Nov 03 01:41:29.351793 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:41:29.351731 4939 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"efe834ec176f778ad2632569bbf778eb71e02e0117cf2037200131ba3869f8fc"} err="failed to get container status \"efe834ec176f778ad2632569bbf778eb71e02e0117cf2037200131ba3869f8fc\": rpc error: code = NotFound desc = could not find container \"efe834ec176f778ad2632569bbf778eb71e02e0117cf2037200131ba3869f8fc\": container with ID starting with efe834ec176f778ad2632569bbf778eb71e02e0117cf2037200131ba3869f8fc not found: ID does not exist" Nov 03 01:41:29.351793 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:41:29.351748 4939 scope.go:122] "RemoveContainer" containerID="7ba5aaa0f3337d62777e32bc996b60a0573018aafc9e436e464208aeb016bc29" Nov 03 01:41:29.355611 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:41:29.355588 4939 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-717217-54c569b974-sxbg4"] Nov 03 01:41:29.358251 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:41:29.358227 4939 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-717217-54c569b974-sxbg4"] Nov 03 01:41:29.360945 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:41:29.360929 4939 scope.go:122] "RemoveContainer" containerID="7ba5aaa0f3337d62777e32bc996b60a0573018aafc9e436e464208aeb016bc29" Nov 03 01:41:29.361231 ip-10-0-140-107 kubenswrapper[4939]: E1103 01:41:29.361209 4939 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7ba5aaa0f3337d62777e32bc996b60a0573018aafc9e436e464208aeb016bc29\": container with ID starting with 7ba5aaa0f3337d62777e32bc996b60a0573018aafc9e436e464208aeb016bc29 not found: ID does not exist" containerID="7ba5aaa0f3337d62777e32bc996b60a0573018aafc9e436e464208aeb016bc29" Nov 03 01:41:29.361283 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:41:29.361235 4939 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7ba5aaa0f3337d62777e32bc996b60a0573018aafc9e436e464208aeb016bc29"} err="failed to get container status \"7ba5aaa0f3337d62777e32bc996b60a0573018aafc9e436e464208aeb016bc29\": rpc error: code = NotFound desc = could not find container \"7ba5aaa0f3337d62777e32bc996b60a0573018aafc9e436e464208aeb016bc29\": container with ID starting with 7ba5aaa0f3337d62777e32bc996b60a0573018aafc9e436e464208aeb016bc29 not found: ID does not exist" Nov 03 01:41:29.367038 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:41:29.367016 4939 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-57079b-666f95c947-dv7mr"] Nov 03 01:41:29.370621 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:41:29.370600 4939 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-57079b-666f95c947-dv7mr"] Nov 03 01:41:29.382070 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:41:29.382048 4939 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-b-57079b-5f4958f857-v8cxd"] Nov 03 01:41:29.384241 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:41:29.384220 4939 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-b-57079b-5f4958f857-v8cxd"] Nov 03 01:41:29.695174 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:41:29.695140 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:41:29.820784 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:41:29.819894 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:41:29.868208 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:41:29.868173 4939 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="01da212a-c344-4e25-9f26-ef2224e32b46" path="/var/lib/kubelet/pods/01da212a-c344-4e25-9f26-ef2224e32b46/volumes" Nov 03 01:41:29.868664 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:41:29.868650 4939 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4bcdc6fd-f0da-46e3-bd48-14b1179bb3f1" path="/var/lib/kubelet/pods/4bcdc6fd-f0da-46e3-bd48-14b1179bb3f1/volumes" Nov 03 01:41:29.869046 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:41:29.869033 4939 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5d25771d-8931-4840-86be-9ebaf92fafca" path="/var/lib/kubelet/pods/5d25771d-8931-4840-86be-9ebaf92fafca/volumes" Nov 03 01:41:29.869740 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:41:29.869723 4939 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a79df098-23e4-4971-9263-3f52b7d25b0b" path="/var/lib/kubelet/pods/a79df098-23e4-4971-9263-3f52b7d25b0b/volumes" Nov 03 01:41:30.494824 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:41:30.494785 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:41:31.097586 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:41:31.097541 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:41:31.781025 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:41:31.780988 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:41:32.379629 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:41:32.379583 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:41:33.045389 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:41:33.045313 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:41:33.256145 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:41:33.256113 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:41:33.403728 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:41:33.403681 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:41:33.707064 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:41:33.706975 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:41:33.994195 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:41:33.994148 4939 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-717217-openshift-default-578c69554c-mwwv6"] Nov 03 01:41:33.994953 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:41:33.994407 4939 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w3-717217-openshift-default-578c69554c-mwwv6" podUID="d0933771-7032-4cff-aafe-2f7211ebb549" containerName="istio-proxy" containerID="cri-o://9827abc03252ecfd590c15fdd611c447bd1281bceaf0d372ef235336b8d21aa5" gracePeriod=30 Nov 03 01:41:33.998933 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:41:33.997877 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:41:34.164405 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:41:34.164372 4939 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w4-6c0ec0-openshift-default-79c7584bfc-kxj54" Nov 03 01:41:34.204003 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:41:34.203959 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:41:34.250168 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:41:34.250083 4939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/d1ad622d-53f3-48ca-bc88-b7ab4ae452fe-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d1ad622d-53f3-48ca-bc88-b7ab4ae452fe-istio-podinfo\") pod \"d1ad622d-53f3-48ca-bc88-b7ab4ae452fe\" (UID: \"d1ad622d-53f3-48ca-bc88-b7ab4ae452fe\") " Nov 03 01:41:34.250168 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:41:34.250149 4939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d1ad622d-53f3-48ca-bc88-b7ab4ae452fe-istio-token\" (UniqueName: \"kubernetes.io/projected/d1ad622d-53f3-48ca-bc88-b7ab4ae452fe-istio-token\") pod \"d1ad622d-53f3-48ca-bc88-b7ab4ae452fe\" (UID: \"d1ad622d-53f3-48ca-bc88-b7ab4ae452fe\") " Nov 03 01:41:34.250434 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:41:34.250183 4939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d1ad622d-53f3-48ca-bc88-b7ab4ae452fe-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d1ad622d-53f3-48ca-bc88-b7ab4ae452fe-istio-envoy\") pod \"d1ad622d-53f3-48ca-bc88-b7ab4ae452fe\" (UID: \"d1ad622d-53f3-48ca-bc88-b7ab4ae452fe\") " Nov 03 01:41:34.250434 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:41:34.250201 4939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d1ad622d-53f3-48ca-bc88-b7ab4ae452fe-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d1ad622d-53f3-48ca-bc88-b7ab4ae452fe-workload-certs\") pod \"d1ad622d-53f3-48ca-bc88-b7ab4ae452fe\" (UID: \"d1ad622d-53f3-48ca-bc88-b7ab4ae452fe\") " Nov 03 01:41:34.250434 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:41:34.250231 4939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d1ad622d-53f3-48ca-bc88-b7ab4ae452fe-kube-api-access-ltpnq\" (UniqueName: \"kubernetes.io/projected/d1ad622d-53f3-48ca-bc88-b7ab4ae452fe-kube-api-access-ltpnq\") pod \"d1ad622d-53f3-48ca-bc88-b7ab4ae452fe\" (UID: \"d1ad622d-53f3-48ca-bc88-b7ab4ae452fe\") " Nov 03 01:41:34.250434 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:41:34.250262 4939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d1ad622d-53f3-48ca-bc88-b7ab4ae452fe-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d1ad622d-53f3-48ca-bc88-b7ab4ae452fe-credential-socket\") pod \"d1ad622d-53f3-48ca-bc88-b7ab4ae452fe\" (UID: \"d1ad622d-53f3-48ca-bc88-b7ab4ae452fe\") " Nov 03 01:41:34.250434 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:41:34.250315 4939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d1ad622d-53f3-48ca-bc88-b7ab4ae452fe-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d1ad622d-53f3-48ca-bc88-b7ab4ae452fe-workload-socket\") pod \"d1ad622d-53f3-48ca-bc88-b7ab4ae452fe\" (UID: \"d1ad622d-53f3-48ca-bc88-b7ab4ae452fe\") " Nov 03 01:41:34.250434 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:41:34.250405 4939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d1ad622d-53f3-48ca-bc88-b7ab4ae452fe-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d1ad622d-53f3-48ca-bc88-b7ab4ae452fe-istio-data\") pod \"d1ad622d-53f3-48ca-bc88-b7ab4ae452fe\" (UID: \"d1ad622d-53f3-48ca-bc88-b7ab4ae452fe\") " Nov 03 01:41:34.250758 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:41:34.250499 4939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d1ad622d-53f3-48ca-bc88-b7ab4ae452fe-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d1ad622d-53f3-48ca-bc88-b7ab4ae452fe-istiod-ca-cert\") pod \"d1ad622d-53f3-48ca-bc88-b7ab4ae452fe\" (UID: \"d1ad622d-53f3-48ca-bc88-b7ab4ae452fe\") " Nov 03 01:41:34.250758 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:41:34.250587 4939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d1ad622d-53f3-48ca-bc88-b7ab4ae452fe-workload-certs" pod "d1ad622d-53f3-48ca-bc88-b7ab4ae452fe" (UID: "d1ad622d-53f3-48ca-bc88-b7ab4ae452fe"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 01:41:34.250885 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:41:34.250781 4939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d1ad622d-53f3-48ca-bc88-b7ab4ae452fe-credential-socket" pod "d1ad622d-53f3-48ca-bc88-b7ab4ae452fe" (UID: "d1ad622d-53f3-48ca-bc88-b7ab4ae452fe"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 01:41:34.251300 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:41:34.250995 4939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d1ad622d-53f3-48ca-bc88-b7ab4ae452fe-workload-socket" pod "d1ad622d-53f3-48ca-bc88-b7ab4ae452fe" (UID: "d1ad622d-53f3-48ca-bc88-b7ab4ae452fe"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 01:41:34.251300 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:41:34.251077 4939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d1ad622d-53f3-48ca-bc88-b7ab4ae452fe-istiod-ca-cert" pod "d1ad622d-53f3-48ca-bc88-b7ab4ae452fe" (UID: "d1ad622d-53f3-48ca-bc88-b7ab4ae452fe"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 01:41:34.251300 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:41:34.251114 4939 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d1ad622d-53f3-48ca-bc88-b7ab4ae452fe-workload-certs\") on node \"ip-10-0-140-107.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:41:34.251300 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:41:34.251128 4939 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d1ad622d-53f3-48ca-bc88-b7ab4ae452fe-credential-socket\") on node \"ip-10-0-140-107.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:41:34.251300 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:41:34.251138 4939 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d1ad622d-53f3-48ca-bc88-b7ab4ae452fe-workload-socket\") on node \"ip-10-0-140-107.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:41:34.251300 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:41:34.250931 4939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d1ad622d-53f3-48ca-bc88-b7ab4ae452fe-istio-data" pod "d1ad622d-53f3-48ca-bc88-b7ab4ae452fe" (UID: "d1ad622d-53f3-48ca-bc88-b7ab4ae452fe"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 01:41:34.253114 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:41:34.253091 4939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/d1ad622d-53f3-48ca-bc88-b7ab4ae452fe-istio-podinfo" pod "d1ad622d-53f3-48ca-bc88-b7ab4ae452fe" (UID: "d1ad622d-53f3-48ca-bc88-b7ab4ae452fe"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 01:41:34.253114 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:41:34.253102 4939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d1ad622d-53f3-48ca-bc88-b7ab4ae452fe-istio-envoy" pod "d1ad622d-53f3-48ca-bc88-b7ab4ae452fe" (UID: "d1ad622d-53f3-48ca-bc88-b7ab4ae452fe"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 01:41:34.253313 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:41:34.253296 4939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d1ad622d-53f3-48ca-bc88-b7ab4ae452fe-kube-api-access-ltpnq" pod "d1ad622d-53f3-48ca-bc88-b7ab4ae452fe" (UID: "d1ad622d-53f3-48ca-bc88-b7ab4ae452fe"). InnerVolumeSpecName "kube-api-access-ltpnq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 01:41:34.253404 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:41:34.253384 4939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d1ad622d-53f3-48ca-bc88-b7ab4ae452fe-istio-token" pod "d1ad622d-53f3-48ca-bc88-b7ab4ae452fe" (UID: "d1ad622d-53f3-48ca-bc88-b7ab4ae452fe"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 01:41:34.352270 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:41:34.352240 4939 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d1ad622d-53f3-48ca-bc88-b7ab4ae452fe-istiod-ca-cert\") on node \"ip-10-0-140-107.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:41:34.352270 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:41:34.352268 4939 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d1ad622d-53f3-48ca-bc88-b7ab4ae452fe-istio-podinfo\") on node \"ip-10-0-140-107.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:41:34.352459 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:41:34.352277 4939 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d1ad622d-53f3-48ca-bc88-b7ab4ae452fe-istio-token\") on node \"ip-10-0-140-107.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:41:34.352459 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:41:34.352287 4939 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d1ad622d-53f3-48ca-bc88-b7ab4ae452fe-istio-envoy\") on node \"ip-10-0-140-107.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:41:34.352459 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:41:34.352298 4939 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ltpnq\" (UniqueName: \"kubernetes.io/projected/d1ad622d-53f3-48ca-bc88-b7ab4ae452fe-kube-api-access-ltpnq\") on node \"ip-10-0-140-107.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:41:34.352459 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:41:34.352307 4939 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d1ad622d-53f3-48ca-bc88-b7ab4ae452fe-istio-data\") on node \"ip-10-0-140-107.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:41:34.367601 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:41:34.367567 4939 generic.go:356] "Generic (PLEG): container finished" podID="d1ad622d-53f3-48ca-bc88-b7ab4ae452fe" containerID="f060af1ea2ac13a918c74b9aa73b49a77c6dc196c94c4c14ae9c408564eac77b" exitCode=0 Nov 03 01:41:34.367754 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:41:34.367618 4939 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w4-6c0ec0-openshift-default-79c7584bfc-kxj54" Nov 03 01:41:34.367754 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:41:34.367649 4939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w4-6c0ec0-openshift-default-79c7584bfc-kxj54" event={"ID":"d1ad622d-53f3-48ca-bc88-b7ab4ae452fe","Type":"ContainerDied","Data":"f060af1ea2ac13a918c74b9aa73b49a77c6dc196c94c4c14ae9c408564eac77b"} Nov 03 01:41:34.367754 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:41:34.367705 4939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w4-6c0ec0-openshift-default-79c7584bfc-kxj54" event={"ID":"d1ad622d-53f3-48ca-bc88-b7ab4ae452fe","Type":"ContainerDied","Data":"06a27d0a7508d1223685658daa66ab3fd6b16c8c7bd625ae3f686bba11e39472"} Nov 03 01:41:34.367754 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:41:34.367727 4939 scope.go:122] "RemoveContainer" containerID="f060af1ea2ac13a918c74b9aa73b49a77c6dc196c94c4c14ae9c408564eac77b" Nov 03 01:41:34.379147 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:41:34.379126 4939 scope.go:122] "RemoveContainer" containerID="f060af1ea2ac13a918c74b9aa73b49a77c6dc196c94c4c14ae9c408564eac77b" Nov 03 01:41:34.379437 ip-10-0-140-107 kubenswrapper[4939]: E1103 01:41:34.379417 4939 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f060af1ea2ac13a918c74b9aa73b49a77c6dc196c94c4c14ae9c408564eac77b\": container with ID starting with f060af1ea2ac13a918c74b9aa73b49a77c6dc196c94c4c14ae9c408564eac77b not found: ID does not exist" containerID="f060af1ea2ac13a918c74b9aa73b49a77c6dc196c94c4c14ae9c408564eac77b" Nov 03 01:41:34.379528 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:41:34.379446 4939 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f060af1ea2ac13a918c74b9aa73b49a77c6dc196c94c4c14ae9c408564eac77b"} err="failed to get container status \"f060af1ea2ac13a918c74b9aa73b49a77c6dc196c94c4c14ae9c408564eac77b\": rpc error: code = NotFound desc = could not find container \"f060af1ea2ac13a918c74b9aa73b49a77c6dc196c94c4c14ae9c408564eac77b\": container with ID starting with f060af1ea2ac13a918c74b9aa73b49a77c6dc196c94c4c14ae9c408564eac77b not found: ID does not exist" Nov 03 01:41:34.392297 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:41:34.392275 4939 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w4-6c0ec0-openshift-default-79c7584bfc-kxj54"] Nov 03 01:41:34.396694 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:41:34.396667 4939 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w4-6c0ec0-openshift-default-79c7584bfc-kxj54"] Nov 03 01:41:34.816938 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:41:34.816888 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:41:34.946399 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:41:34.946369 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:41:35.585745 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:41:35.585713 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:41:35.869049 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:41:35.868963 4939 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d1ad622d-53f3-48ca-bc88-b7ab4ae452fe" path="/var/lib/kubelet/pods/d1ad622d-53f3-48ca-bc88-b7ab4ae452fe/volumes" Nov 03 01:41:36.178976 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:41:36.178870 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:41:36.883081 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:41:36.883041 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:41:36.966154 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:41:36.966012 4939 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w4-4a697f-79fb8d7d9-gr8cv"] Nov 03 01:41:36.975250 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:41:36.975221 4939 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w4-4a697f-94d8d8fc9-kzr29"] Nov 03 01:41:37.036073 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:41:37.036021 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:41:37.147514 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:41:37.147401 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:41:37.272920 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:41:37.272840 4939 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-6c0ec0-7474bf5fc5-dhqxc" podUID="c6de1910-0365-4f48-99f4-b32aebd747bb" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 03 01:41:37.676406 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:41:37.676372 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:41:38.183512 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:41:38.183470 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:41:38.278192 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:41:38.278123 4939 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-6c0ec0-65c5d46767-hsxz4" podUID="61b34e6e-2690-4080-bced-3a577c665f12" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 03 01:41:38.384179 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:41:38.384139 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:41:39.027218 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:41:39.027178 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:41:39.318536 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:41:39.318506 4939 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w3-717217-openshift-default-578c69554c-mwwv6" Nov 03 01:41:39.402187 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:41:39.402154 4939 generic.go:356] "Generic (PLEG): container finished" podID="d0933771-7032-4cff-aafe-2f7211ebb549" containerID="9827abc03252ecfd590c15fdd611c447bd1281bceaf0d372ef235336b8d21aa5" exitCode=0 Nov 03 01:41:39.402361 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:41:39.402212 4939 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w3-717217-openshift-default-578c69554c-mwwv6" Nov 03 01:41:39.402361 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:41:39.402237 4939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-717217-openshift-default-578c69554c-mwwv6" event={"ID":"d0933771-7032-4cff-aafe-2f7211ebb549","Type":"ContainerDied","Data":"9827abc03252ecfd590c15fdd611c447bd1281bceaf0d372ef235336b8d21aa5"} Nov 03 01:41:39.402361 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:41:39.402290 4939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-717217-openshift-default-578c69554c-mwwv6" event={"ID":"d0933771-7032-4cff-aafe-2f7211ebb549","Type":"ContainerDied","Data":"b3864d9992430be907ca4918e5ec90b3fc2c8800bb2bc26816bc92f5d4d845e7"} Nov 03 01:41:39.402361 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:41:39.402324 4939 scope.go:122] "RemoveContainer" containerID="9827abc03252ecfd590c15fdd611c447bd1281bceaf0d372ef235336b8d21aa5" Nov 03 01:41:39.412182 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:41:39.412162 4939 scope.go:122] "RemoveContainer" containerID="9827abc03252ecfd590c15fdd611c447bd1281bceaf0d372ef235336b8d21aa5" Nov 03 01:41:39.412443 ip-10-0-140-107 kubenswrapper[4939]: E1103 01:41:39.412422 4939 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9827abc03252ecfd590c15fdd611c447bd1281bceaf0d372ef235336b8d21aa5\": container with ID starting with 9827abc03252ecfd590c15fdd611c447bd1281bceaf0d372ef235336b8d21aa5 not found: ID does not exist" containerID="9827abc03252ecfd590c15fdd611c447bd1281bceaf0d372ef235336b8d21aa5" Nov 03 01:41:39.412529 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:41:39.412453 4939 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9827abc03252ecfd590c15fdd611c447bd1281bceaf0d372ef235336b8d21aa5"} err="failed to get container status \"9827abc03252ecfd590c15fdd611c447bd1281bceaf0d372ef235336b8d21aa5\": rpc error: code = NotFound desc = could not find container \"9827abc03252ecfd590c15fdd611c447bd1281bceaf0d372ef235336b8d21aa5\": container with ID starting with 9827abc03252ecfd590c15fdd611c447bd1281bceaf0d372ef235336b8d21aa5 not found: ID does not exist" Nov 03 01:41:39.438478 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:41:39.438448 4939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/d0933771-7032-4cff-aafe-2f7211ebb549-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d0933771-7032-4cff-aafe-2f7211ebb549-istio-podinfo\") pod \"d0933771-7032-4cff-aafe-2f7211ebb549\" (UID: \"d0933771-7032-4cff-aafe-2f7211ebb549\") " Nov 03 01:41:39.438617 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:41:39.438564 4939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d0933771-7032-4cff-aafe-2f7211ebb549-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d0933771-7032-4cff-aafe-2f7211ebb549-workload-certs\") pod \"d0933771-7032-4cff-aafe-2f7211ebb549\" (UID: \"d0933771-7032-4cff-aafe-2f7211ebb549\") " Nov 03 01:41:39.438694 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:41:39.438631 4939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d0933771-7032-4cff-aafe-2f7211ebb549-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d0933771-7032-4cff-aafe-2f7211ebb549-istiod-ca-cert\") pod \"d0933771-7032-4cff-aafe-2f7211ebb549\" (UID: \"d0933771-7032-4cff-aafe-2f7211ebb549\") " Nov 03 01:41:39.438694 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:41:39.438675 4939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d0933771-7032-4cff-aafe-2f7211ebb549-kube-api-access-rdlmd\" (UniqueName: \"kubernetes.io/projected/d0933771-7032-4cff-aafe-2f7211ebb549-kube-api-access-rdlmd\") pod \"d0933771-7032-4cff-aafe-2f7211ebb549\" (UID: \"d0933771-7032-4cff-aafe-2f7211ebb549\") " Nov 03 01:41:39.438821 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:41:39.438776 4939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d0933771-7032-4cff-aafe-2f7211ebb549-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d0933771-7032-4cff-aafe-2f7211ebb549-credential-socket\") pod \"d0933771-7032-4cff-aafe-2f7211ebb549\" (UID: \"d0933771-7032-4cff-aafe-2f7211ebb549\") " Nov 03 01:41:39.438821 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:41:39.438810 4939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d0933771-7032-4cff-aafe-2f7211ebb549-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d0933771-7032-4cff-aafe-2f7211ebb549-istio-data\") pod \"d0933771-7032-4cff-aafe-2f7211ebb549\" (UID: \"d0933771-7032-4cff-aafe-2f7211ebb549\") " Nov 03 01:41:39.438943 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:41:39.438826 4939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d0933771-7032-4cff-aafe-2f7211ebb549-workload-certs" pod "d0933771-7032-4cff-aafe-2f7211ebb549" (UID: "d0933771-7032-4cff-aafe-2f7211ebb549"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 01:41:39.439022 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:41:39.438995 4939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d0933771-7032-4cff-aafe-2f7211ebb549-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d0933771-7032-4cff-aafe-2f7211ebb549-istio-envoy\") pod \"d0933771-7032-4cff-aafe-2f7211ebb549\" (UID: \"d0933771-7032-4cff-aafe-2f7211ebb549\") " Nov 03 01:41:39.439099 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:41:39.439069 4939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d0933771-7032-4cff-aafe-2f7211ebb549-istio-token\" (UniqueName: \"kubernetes.io/projected/d0933771-7032-4cff-aafe-2f7211ebb549-istio-token\") pod \"d0933771-7032-4cff-aafe-2f7211ebb549\" (UID: \"d0933771-7032-4cff-aafe-2f7211ebb549\") " Nov 03 01:41:39.439166 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:41:39.439111 4939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d0933771-7032-4cff-aafe-2f7211ebb549-credential-socket" pod "d0933771-7032-4cff-aafe-2f7211ebb549" (UID: "d0933771-7032-4cff-aafe-2f7211ebb549"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 01:41:39.439166 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:41:39.439119 4939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d0933771-7032-4cff-aafe-2f7211ebb549-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d0933771-7032-4cff-aafe-2f7211ebb549-workload-socket\") pod \"d0933771-7032-4cff-aafe-2f7211ebb549\" (UID: \"d0933771-7032-4cff-aafe-2f7211ebb549\") " Nov 03 01:41:39.439269 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:41:39.439163 4939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d0933771-7032-4cff-aafe-2f7211ebb549-istio-data" pod "d0933771-7032-4cff-aafe-2f7211ebb549" (UID: "d0933771-7032-4cff-aafe-2f7211ebb549"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 01:41:39.439328 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:41:39.439297 4939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d0933771-7032-4cff-aafe-2f7211ebb549-workload-socket" pod "d0933771-7032-4cff-aafe-2f7211ebb549" (UID: "d0933771-7032-4cff-aafe-2f7211ebb549"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 01:41:39.439479 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:41:39.439455 4939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d0933771-7032-4cff-aafe-2f7211ebb549-istiod-ca-cert" pod "d0933771-7032-4cff-aafe-2f7211ebb549" (UID: "d0933771-7032-4cff-aafe-2f7211ebb549"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 01:41:39.439741 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:41:39.439721 4939 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d0933771-7032-4cff-aafe-2f7211ebb549-workload-socket\") on node \"ip-10-0-140-107.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:41:39.439803 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:41:39.439750 4939 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d0933771-7032-4cff-aafe-2f7211ebb549-workload-certs\") on node \"ip-10-0-140-107.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:41:39.439803 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:41:39.439767 4939 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d0933771-7032-4cff-aafe-2f7211ebb549-istiod-ca-cert\") on node \"ip-10-0-140-107.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:41:39.439803 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:41:39.439782 4939 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d0933771-7032-4cff-aafe-2f7211ebb549-credential-socket\") on node \"ip-10-0-140-107.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:41:39.439803 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:41:39.439796 4939 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d0933771-7032-4cff-aafe-2f7211ebb549-istio-data\") on node \"ip-10-0-140-107.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:41:39.440827 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:41:39.440806 4939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/d0933771-7032-4cff-aafe-2f7211ebb549-istio-podinfo" pod "d0933771-7032-4cff-aafe-2f7211ebb549" (UID: "d0933771-7032-4cff-aafe-2f7211ebb549"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 01:41:39.441132 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:41:39.441115 4939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d0933771-7032-4cff-aafe-2f7211ebb549-kube-api-access-rdlmd" pod "d0933771-7032-4cff-aafe-2f7211ebb549" (UID: "d0933771-7032-4cff-aafe-2f7211ebb549"). InnerVolumeSpecName "kube-api-access-rdlmd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 01:41:39.441505 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:41:39.441486 4939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d0933771-7032-4cff-aafe-2f7211ebb549-istio-envoy" pod "d0933771-7032-4cff-aafe-2f7211ebb549" (UID: "d0933771-7032-4cff-aafe-2f7211ebb549"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 01:41:39.441505 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:41:39.441496 4939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d0933771-7032-4cff-aafe-2f7211ebb549-istio-token" pod "d0933771-7032-4cff-aafe-2f7211ebb549" (UID: "d0933771-7032-4cff-aafe-2f7211ebb549"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 01:41:39.553996 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:41:39.551862 4939 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d0933771-7032-4cff-aafe-2f7211ebb549-istio-envoy\") on node \"ip-10-0-140-107.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:41:39.553996 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:41:39.551923 4939 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d0933771-7032-4cff-aafe-2f7211ebb549-istio-token\") on node \"ip-10-0-140-107.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:41:39.553996 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:41:39.551942 4939 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d0933771-7032-4cff-aafe-2f7211ebb549-istio-podinfo\") on node \"ip-10-0-140-107.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:41:39.553996 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:41:39.551958 4939 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rdlmd\" (UniqueName: \"kubernetes.io/projected/d0933771-7032-4cff-aafe-2f7211ebb549-kube-api-access-rdlmd\") on node \"ip-10-0-140-107.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:41:39.732042 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:41:39.732008 4939 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-717217-openshift-default-578c69554c-mwwv6"] Nov 03 01:41:39.735747 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:41:39.735719 4939 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-717217-openshift-default-578c69554c-mwwv6"] Nov 03 01:41:39.742185 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:41:39.742159 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:41:39.874679 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:41:39.874638 4939 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d0933771-7032-4cff-aafe-2f7211ebb549" path="/var/lib/kubelet/pods/d0933771-7032-4cff-aafe-2f7211ebb549/volumes" Nov 03 01:41:40.130606 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:41:40.130524 4939 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w3-556c14-566c7cc65c-lcbl4"] Nov 03 01:41:40.140350 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:41:40.140311 4939 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w3-556c14-669d5fb59f-cjtpf"] Nov 03 01:41:40.225165 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:41:40.225132 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:41:40.338601 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:41:40.338566 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:41:40.796806 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:41:40.796776 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:41:41.323500 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:41:41.323468 4939 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w1-2729f2/rate-test-model-2729f2-kserve-6d6745bb5c-dm56b"] Nov 03 01:41:41.388340 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:41:41.388304 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:41:41.801147 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:41:41.801111 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:41:41.966975 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:41:41.966898 4939 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-4a697f-79fb8d7d9-gr8cv" podUID="ef6478d3-bd62-45a7-8acc-1055c24dc53b" containerName="payload-processing" containerID="cri-o://4fcbd3d36db1cb376909b018910f367575ae43d65ae89c447ec2b35bc985516d" gracePeriod=25 Nov 03 01:41:41.976501 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:41:41.976467 4939 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-4a697f-94d8d8fc9-kzr29" podUID="ef71e635-99c1-4f1c-99b0-014934ba2f76" containerName="payload-pre-processing" containerID="cri-o://b6bb749d739c14292a547641ed6d373e8323f531c716126beccd5591b304334d" gracePeriod=25 Nov 03 01:41:41.994335 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:41:41.994297 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:41:42.499652 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:41:42.499614 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:41:43.027174 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:41:43.027127 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:41:43.189140 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:41:43.189105 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:41:43.392270 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:41:43.392235 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:41:43.532799 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:41:43.532728 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:41:43.708220 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:41:43.708121 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:41:43.944275 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:41:43.944235 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:41:44.197893 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:41:44.197853 4939 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w4-4a697f-openshift-default-666db9f8f5-qn8tj"] Nov 03 01:41:44.198129 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:41:44.198100 4939 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w4-4a697f-openshift-default-666db9f8f5-qn8tj" podUID="5b0cabc6-4960-4dc1-8317-3faebf25b30c" containerName="istio-proxy" containerID="cri-o://58e1616a5833fada05ba97f4eb82d69ef9c4f95be968f1d1f96f1849f94d30cd" gracePeriod=30 Nov 03 01:41:44.200052 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:41:44.200007 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:41:44.321182 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:41:44.321098 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:41:44.754903 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:41:44.754864 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:41:44.957007 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:41:44.956963 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:41:45.131636 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:41:45.131597 4939 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-556c14-566c7cc65c-lcbl4" podUID="23f02c28-7f9e-40d8-990c-aea9e85f1bf6" containerName="payload-processing" containerID="cri-o://82722e009994f48eae235ec601132b70f0dee007f538c43936ef0e5e48c7ff34" gracePeriod=25 Nov 03 01:41:45.141148 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:41:45.141116 4939 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-556c14-669d5fb59f-cjtpf" podUID="192c8a1d-60d2-4928-9579-0231138280bb" containerName="payload-pre-processing" containerID="cri-o://f0face4ca56ac8b0d09a7aa6ea1c56ac5622db1ed5a8238106e45cc3a42061c2" gracePeriod=25 Nov 03 01:41:45.617107 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:41:45.617075 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:41:45.858927 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:41:45.856552 4939 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-6c0ec0-7474bf5fc5-dhqxc" Nov 03 01:41:45.908772 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:41:45.908746 4939 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-6c0ec0-65c5d46767-hsxz4" Nov 03 01:41:45.967614 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:41:45.967587 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:41:45.988382 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:41:45.988349 4939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c6de1910-0365-4f48-99f4-b32aebd747bb-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c6de1910-0365-4f48-99f4-b32aebd747bb-plugins-config-volume\") pod \"c6de1910-0365-4f48-99f4-b32aebd747bb\" (UID: \"c6de1910-0365-4f48-99f4-b32aebd747bb\") " Nov 03 01:41:45.988534 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:41:45.988402 4939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c6de1910-0365-4f48-99f4-b32aebd747bb-kube-api-access-vzz2v\" (UniqueName: \"kubernetes.io/projected/c6de1910-0365-4f48-99f4-b32aebd747bb-kube-api-access-vzz2v\") pod \"c6de1910-0365-4f48-99f4-b32aebd747bb\" (UID: \"c6de1910-0365-4f48-99f4-b32aebd747bb\") " Nov 03 01:41:45.988774 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:41:45.988745 4939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c6de1910-0365-4f48-99f4-b32aebd747bb-plugins-config-volume" pod "c6de1910-0365-4f48-99f4-b32aebd747bb" (UID: "c6de1910-0365-4f48-99f4-b32aebd747bb"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 01:41:45.989184 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:41:45.989158 4939 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c6de1910-0365-4f48-99f4-b32aebd747bb-plugins-config-volume\") on node \"ip-10-0-140-107.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:41:45.990821 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:41:45.990783 4939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c6de1910-0365-4f48-99f4-b32aebd747bb-kube-api-access-vzz2v" pod "c6de1910-0365-4f48-99f4-b32aebd747bb" (UID: "c6de1910-0365-4f48-99f4-b32aebd747bb"). InnerVolumeSpecName "kube-api-access-vzz2v". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 01:41:46.078716 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:41:46.078621 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:41:46.090534 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:41:46.090506 4939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/61b34e6e-2690-4080-bced-3a577c665f12-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/61b34e6e-2690-4080-bced-3a577c665f12-plugins-config-volume\") pod \"61b34e6e-2690-4080-bced-3a577c665f12\" (UID: \"61b34e6e-2690-4080-bced-3a577c665f12\") " Nov 03 01:41:46.090684 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:41:46.090590 4939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/61b34e6e-2690-4080-bced-3a577c665f12-kube-api-access-52w9v\" (UniqueName: \"kubernetes.io/projected/61b34e6e-2690-4080-bced-3a577c665f12-kube-api-access-52w9v\") pod \"61b34e6e-2690-4080-bced-3a577c665f12\" (UID: \"61b34e6e-2690-4080-bced-3a577c665f12\") " Nov 03 01:41:46.090904 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:41:46.090874 4939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/61b34e6e-2690-4080-bced-3a577c665f12-plugins-config-volume" pod "61b34e6e-2690-4080-bced-3a577c665f12" (UID: "61b34e6e-2690-4080-bced-3a577c665f12"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 01:41:46.091062 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:41:46.090965 4939 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vzz2v\" (UniqueName: \"kubernetes.io/projected/c6de1910-0365-4f48-99f4-b32aebd747bb-kube-api-access-vzz2v\") on node \"ip-10-0-140-107.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:41:46.092825 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:41:46.092804 4939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/61b34e6e-2690-4080-bced-3a577c665f12-kube-api-access-52w9v" pod "61b34e6e-2690-4080-bced-3a577c665f12" (UID: "61b34e6e-2690-4080-bced-3a577c665f12"). InnerVolumeSpecName "kube-api-access-52w9v". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 01:41:46.190804 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:41:46.190764 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:41:46.194557 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:41:46.194506 4939 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-52w9v\" (UniqueName: \"kubernetes.io/projected/61b34e6e-2690-4080-bced-3a577c665f12-kube-api-access-52w9v\") on node \"ip-10-0-140-107.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:41:46.194557 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:41:46.194549 4939 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/61b34e6e-2690-4080-bced-3a577c665f12-plugins-config-volume\") on node \"ip-10-0-140-107.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:41:46.318366 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:41:46.318279 4939 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-f7680d-openshift-default-c4488d5c9-qstvv"] Nov 03 01:41:46.318570 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:41:46.318546 4939 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-f7680d-openshift-default-c4488d5c9-qstvv" podUID="ccf63a28-a2d8-4686-8555-968e54429210" containerName="istio-proxy" containerID="cri-o://5607cb9bdc1566797246d0558e8a8a2611dd45b7a214459c511d4a8025ac6e3b" gracePeriod=30 Nov 03 01:41:46.373448 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:41:46.373418 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:41:46.486970 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:41:46.486934 4939 generic.go:356] "Generic (PLEG): container finished" podID="c6de1910-0365-4f48-99f4-b32aebd747bb" containerID="94803b27fbf936d8d29f1d086acc91666547bca0730a0e5ec7c0362f7887771a" exitCode=0 Nov 03 01:41:46.487165 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:41:46.486980 4939 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-6c0ec0-7474bf5fc5-dhqxc" Nov 03 01:41:46.487165 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:41:46.487016 4939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-6c0ec0-7474bf5fc5-dhqxc" event={"ID":"c6de1910-0365-4f48-99f4-b32aebd747bb","Type":"ContainerDied","Data":"94803b27fbf936d8d29f1d086acc91666547bca0730a0e5ec7c0362f7887771a"} Nov 03 01:41:46.487165 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:41:46.487068 4939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-6c0ec0-7474bf5fc5-dhqxc" event={"ID":"c6de1910-0365-4f48-99f4-b32aebd747bb","Type":"ContainerDied","Data":"3b50a4be96bcd5030b24947c77fb5fdde836f9081bdf85694bd4d4b3b5cbb742"} Nov 03 01:41:46.487165 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:41:46.487092 4939 scope.go:122] "RemoveContainer" containerID="94803b27fbf936d8d29f1d086acc91666547bca0730a0e5ec7c0362f7887771a" Nov 03 01:41:46.488802 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:41:46.488764 4939 generic.go:356] "Generic (PLEG): container finished" podID="61b34e6e-2690-4080-bced-3a577c665f12" containerID="4953379acfa82f023c872f91888a0f003a2e7fc293fc9ccc0020117a5946d212" exitCode=0 Nov 03 01:41:46.488944 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:41:46.488810 4939 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-6c0ec0-65c5d46767-hsxz4" Nov 03 01:41:46.488944 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:41:46.488845 4939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-6c0ec0-65c5d46767-hsxz4" event={"ID":"61b34e6e-2690-4080-bced-3a577c665f12","Type":"ContainerDied","Data":"4953379acfa82f023c872f91888a0f003a2e7fc293fc9ccc0020117a5946d212"} Nov 03 01:41:46.488944 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:41:46.488899 4939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-6c0ec0-65c5d46767-hsxz4" event={"ID":"61b34e6e-2690-4080-bced-3a577c665f12","Type":"ContainerDied","Data":"ca7d9f5a7656de2ff40d1feab185e4bc13acc8e369a86c0f77888afa7ddd40a3"} Nov 03 01:41:46.500210 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:41:46.500096 4939 scope.go:122] "RemoveContainer" containerID="94803b27fbf936d8d29f1d086acc91666547bca0730a0e5ec7c0362f7887771a" Nov 03 01:41:46.500473 ip-10-0-140-107 kubenswrapper[4939]: E1103 01:41:46.500446 4939 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"94803b27fbf936d8d29f1d086acc91666547bca0730a0e5ec7c0362f7887771a\": container with ID starting with 94803b27fbf936d8d29f1d086acc91666547bca0730a0e5ec7c0362f7887771a not found: ID does not exist" containerID="94803b27fbf936d8d29f1d086acc91666547bca0730a0e5ec7c0362f7887771a" Nov 03 01:41:46.500550 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:41:46.500484 4939 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"94803b27fbf936d8d29f1d086acc91666547bca0730a0e5ec7c0362f7887771a"} err="failed to get container status \"94803b27fbf936d8d29f1d086acc91666547bca0730a0e5ec7c0362f7887771a\": rpc error: code = NotFound desc = could not find container \"94803b27fbf936d8d29f1d086acc91666547bca0730a0e5ec7c0362f7887771a\": container with ID starting with 94803b27fbf936d8d29f1d086acc91666547bca0730a0e5ec7c0362f7887771a not found: ID does not exist" Nov 03 01:41:46.500550 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:41:46.500516 4939 scope.go:122] "RemoveContainer" containerID="4953379acfa82f023c872f91888a0f003a2e7fc293fc9ccc0020117a5946d212" Nov 03 01:41:46.510437 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:41:46.510415 4939 scope.go:122] "RemoveContainer" containerID="4953379acfa82f023c872f91888a0f003a2e7fc293fc9ccc0020117a5946d212" Nov 03 01:41:46.510863 ip-10-0-140-107 kubenswrapper[4939]: E1103 01:41:46.510834 4939 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4953379acfa82f023c872f91888a0f003a2e7fc293fc9ccc0020117a5946d212\": container with ID starting with 4953379acfa82f023c872f91888a0f003a2e7fc293fc9ccc0020117a5946d212 not found: ID does not exist" containerID="4953379acfa82f023c872f91888a0f003a2e7fc293fc9ccc0020117a5946d212" Nov 03 01:41:46.511014 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:41:46.510871 4939 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4953379acfa82f023c872f91888a0f003a2e7fc293fc9ccc0020117a5946d212"} err="failed to get container status \"4953379acfa82f023c872f91888a0f003a2e7fc293fc9ccc0020117a5946d212\": rpc error: code = NotFound desc = could not find container \"4953379acfa82f023c872f91888a0f003a2e7fc293fc9ccc0020117a5946d212\": container with ID starting with 4953379acfa82f023c872f91888a0f003a2e7fc293fc9ccc0020117a5946d212 not found: ID does not exist" Nov 03 01:41:46.518760 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:41:46.518734 4939 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w4-6c0ec0-65c5d46767-hsxz4"] Nov 03 01:41:46.525991 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:41:46.525969 4939 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w4-6c0ec0-65c5d46767-hsxz4"] Nov 03 01:41:46.535126 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:41:46.535101 4939 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w4-6c0ec0-7474bf5fc5-dhqxc"] Nov 03 01:41:46.538082 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:41:46.538060 4939 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w4-6c0ec0-7474bf5fc5-dhqxc"] Nov 03 01:41:46.588151 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:41:46.588109 4939 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-4a697f-94d8d8fc9-kzr29" podUID="ef71e635-99c1-4f1c-99b0-014934ba2f76" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 03 01:41:46.594675 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:41:46.594642 4939 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-4a697f-79fb8d7d9-gr8cv" podUID="ef6478d3-bd62-45a7-8acc-1055c24dc53b" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 03 01:41:46.682686 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:41:46.682606 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:41:46.996980 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:41:46.996890 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:41:47.246269 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:41:47.246234 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:41:47.798326 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:41:47.798286 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:41:47.868391 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:41:47.868358 4939 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="61b34e6e-2690-4080-bced-3a577c665f12" path="/var/lib/kubelet/pods/61b34e6e-2690-4080-bced-3a577c665f12/volumes" Nov 03 01:41:47.868779 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:41:47.868765 4939 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c6de1910-0365-4f48-99f4-b32aebd747bb" path="/var/lib/kubelet/pods/c6de1910-0365-4f48-99f4-b32aebd747bb/volumes" Nov 03 01:41:48.032995 ip-10-0-140-107 kubenswrapper[4939]: E1103 01:41:48.032953 4939 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-sthjh" containerName="extractor" Nov 03 01:41:48.527192 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:41:48.527149 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:41:49.011736 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:41:49.011680 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:41:49.149065 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:41:49.149028 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:41:49.266432 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:41:49.266323 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:41:49.471059 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:41:49.471024 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:41:49.473152 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:41:49.473124 4939 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w4-4a697f-openshift-default-666db9f8f5-qn8tj" Nov 03 01:41:49.513030 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:41:49.512992 4939 generic.go:356] "Generic (PLEG): container finished" podID="5b0cabc6-4960-4dc1-8317-3faebf25b30c" containerID="58e1616a5833fada05ba97f4eb82d69ef9c4f95be968f1d1f96f1849f94d30cd" exitCode=0 Nov 03 01:41:49.513030 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:41:49.513036 4939 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w4-4a697f-openshift-default-666db9f8f5-qn8tj" Nov 03 01:41:49.513248 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:41:49.513081 4939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w4-4a697f-openshift-default-666db9f8f5-qn8tj" event={"ID":"5b0cabc6-4960-4dc1-8317-3faebf25b30c","Type":"ContainerDied","Data":"58e1616a5833fada05ba97f4eb82d69ef9c4f95be968f1d1f96f1849f94d30cd"} Nov 03 01:41:49.513248 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:41:49.513144 4939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w4-4a697f-openshift-default-666db9f8f5-qn8tj" event={"ID":"5b0cabc6-4960-4dc1-8317-3faebf25b30c","Type":"ContainerDied","Data":"137d8b8a9ddc5a288b3c4eb85967de7675b8cc390343656575bf6dd655888030"} Nov 03 01:41:49.513248 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:41:49.513175 4939 scope.go:122] "RemoveContainer" containerID="58e1616a5833fada05ba97f4eb82d69ef9c4f95be968f1d1f96f1849f94d30cd" Nov 03 01:41:49.524064 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:41:49.524046 4939 scope.go:122] "RemoveContainer" containerID="2f9dce65eb234b7acf638877b722bc4efa6182ce79db38c29344f4c05eee7784" Nov 03 01:41:49.534653 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:41:49.534631 4939 scope.go:122] "RemoveContainer" containerID="58e1616a5833fada05ba97f4eb82d69ef9c4f95be968f1d1f96f1849f94d30cd" Nov 03 01:41:49.535065 ip-10-0-140-107 kubenswrapper[4939]: E1103 01:41:49.535011 4939 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"58e1616a5833fada05ba97f4eb82d69ef9c4f95be968f1d1f96f1849f94d30cd\": container with ID starting with 58e1616a5833fada05ba97f4eb82d69ef9c4f95be968f1d1f96f1849f94d30cd not found: ID does not exist" containerID="58e1616a5833fada05ba97f4eb82d69ef9c4f95be968f1d1f96f1849f94d30cd" Nov 03 01:41:49.535211 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:41:49.535055 4939 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"58e1616a5833fada05ba97f4eb82d69ef9c4f95be968f1d1f96f1849f94d30cd"} err="failed to get container status \"58e1616a5833fada05ba97f4eb82d69ef9c4f95be968f1d1f96f1849f94d30cd\": rpc error: code = NotFound desc = could not find container \"58e1616a5833fada05ba97f4eb82d69ef9c4f95be968f1d1f96f1849f94d30cd\": container with ID starting with 58e1616a5833fada05ba97f4eb82d69ef9c4f95be968f1d1f96f1849f94d30cd not found: ID does not exist" Nov 03 01:41:49.535211 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:41:49.535079 4939 scope.go:122] "RemoveContainer" containerID="2f9dce65eb234b7acf638877b722bc4efa6182ce79db38c29344f4c05eee7784" Nov 03 01:41:49.535431 ip-10-0-140-107 kubenswrapper[4939]: E1103 01:41:49.535408 4939 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2f9dce65eb234b7acf638877b722bc4efa6182ce79db38c29344f4c05eee7784\": container with ID starting with 2f9dce65eb234b7acf638877b722bc4efa6182ce79db38c29344f4c05eee7784 not found: ID does not exist" containerID="2f9dce65eb234b7acf638877b722bc4efa6182ce79db38c29344f4c05eee7784" Nov 03 01:41:49.535497 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:41:49.535443 4939 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2f9dce65eb234b7acf638877b722bc4efa6182ce79db38c29344f4c05eee7784"} err="failed to get container status \"2f9dce65eb234b7acf638877b722bc4efa6182ce79db38c29344f4c05eee7784\": rpc error: code = NotFound desc = could not find container \"2f9dce65eb234b7acf638877b722bc4efa6182ce79db38c29344f4c05eee7784\": container with ID starting with 2f9dce65eb234b7acf638877b722bc4efa6182ce79db38c29344f4c05eee7784 not found: ID does not exist" Nov 03 01:41:49.540694 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:41:49.540670 4939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5b0cabc6-4960-4dc1-8317-3faebf25b30c-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5b0cabc6-4960-4dc1-8317-3faebf25b30c-istio-data\") pod \"5b0cabc6-4960-4dc1-8317-3faebf25b30c\" (UID: \"5b0cabc6-4960-4dc1-8317-3faebf25b30c\") " Nov 03 01:41:49.540805 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:41:49.540751 4939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5b0cabc6-4960-4dc1-8317-3faebf25b30c-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5b0cabc6-4960-4dc1-8317-3faebf25b30c-istio-envoy\") pod \"5b0cabc6-4960-4dc1-8317-3faebf25b30c\" (UID: \"5b0cabc6-4960-4dc1-8317-3faebf25b30c\") " Nov 03 01:41:49.540878 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:41:49.540827 4939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5b0cabc6-4960-4dc1-8317-3faebf25b30c-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5b0cabc6-4960-4dc1-8317-3faebf25b30c-workload-certs\") pod \"5b0cabc6-4960-4dc1-8317-3faebf25b30c\" (UID: \"5b0cabc6-4960-4dc1-8317-3faebf25b30c\") " Nov 03 01:41:49.540972 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:41:49.540942 4939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5b0cabc6-4960-4dc1-8317-3faebf25b30c-istio-data" pod "5b0cabc6-4960-4dc1-8317-3faebf25b30c" (UID: "5b0cabc6-4960-4dc1-8317-3faebf25b30c"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 01:41:49.541099 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:41:49.540988 4939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5b0cabc6-4960-4dc1-8317-3faebf25b30c-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5b0cabc6-4960-4dc1-8317-3faebf25b30c-istiod-ca-cert\") pod \"5b0cabc6-4960-4dc1-8317-3faebf25b30c\" (UID: \"5b0cabc6-4960-4dc1-8317-3faebf25b30c\") " Nov 03 01:41:49.541099 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:41:49.541025 4939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5b0cabc6-4960-4dc1-8317-3faebf25b30c-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5b0cabc6-4960-4dc1-8317-3faebf25b30c-workload-socket\") pod \"5b0cabc6-4960-4dc1-8317-3faebf25b30c\" (UID: \"5b0cabc6-4960-4dc1-8317-3faebf25b30c\") " Nov 03 01:41:49.541099 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:41:49.541071 4939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5b0cabc6-4960-4dc1-8317-3faebf25b30c-kube-api-access-29zgn\" (UniqueName: \"kubernetes.io/projected/5b0cabc6-4960-4dc1-8317-3faebf25b30c-kube-api-access-29zgn\") pod \"5b0cabc6-4960-4dc1-8317-3faebf25b30c\" (UID: \"5b0cabc6-4960-4dc1-8317-3faebf25b30c\") " Nov 03 01:41:49.541276 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:41:49.541163 4939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/5b0cabc6-4960-4dc1-8317-3faebf25b30c-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5b0cabc6-4960-4dc1-8317-3faebf25b30c-istio-podinfo\") pod \"5b0cabc6-4960-4dc1-8317-3faebf25b30c\" (UID: \"5b0cabc6-4960-4dc1-8317-3faebf25b30c\") " Nov 03 01:41:49.541276 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:41:49.541197 4939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5b0cabc6-4960-4dc1-8317-3faebf25b30c-istio-token\" (UniqueName: \"kubernetes.io/projected/5b0cabc6-4960-4dc1-8317-3faebf25b30c-istio-token\") pod \"5b0cabc6-4960-4dc1-8317-3faebf25b30c\" (UID: \"5b0cabc6-4960-4dc1-8317-3faebf25b30c\") " Nov 03 01:41:49.541276 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:41:49.541203 4939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5b0cabc6-4960-4dc1-8317-3faebf25b30c-workload-certs" pod "5b0cabc6-4960-4dc1-8317-3faebf25b30c" (UID: "5b0cabc6-4960-4dc1-8317-3faebf25b30c"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 01:41:49.541276 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:41:49.541263 4939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5b0cabc6-4960-4dc1-8317-3faebf25b30c-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5b0cabc6-4960-4dc1-8317-3faebf25b30c-credential-socket\") pod \"5b0cabc6-4960-4dc1-8317-3faebf25b30c\" (UID: \"5b0cabc6-4960-4dc1-8317-3faebf25b30c\") " Nov 03 01:41:49.541639 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:41:49.541283 4939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5b0cabc6-4960-4dc1-8317-3faebf25b30c-workload-socket" pod "5b0cabc6-4960-4dc1-8317-3faebf25b30c" (UID: "5b0cabc6-4960-4dc1-8317-3faebf25b30c"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 01:41:49.541639 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:41:49.541325 4939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5b0cabc6-4960-4dc1-8317-3faebf25b30c-istiod-ca-cert" pod "5b0cabc6-4960-4dc1-8317-3faebf25b30c" (UID: "5b0cabc6-4960-4dc1-8317-3faebf25b30c"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 01:41:49.541762 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:41:49.541652 4939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5b0cabc6-4960-4dc1-8317-3faebf25b30c-credential-socket" pod "5b0cabc6-4960-4dc1-8317-3faebf25b30c" (UID: "5b0cabc6-4960-4dc1-8317-3faebf25b30c"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 01:41:49.541866 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:41:49.541843 4939 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5b0cabc6-4960-4dc1-8317-3faebf25b30c-credential-socket\") on node \"ip-10-0-140-107.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:41:49.541956 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:41:49.541872 4939 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5b0cabc6-4960-4dc1-8317-3faebf25b30c-istio-data\") on node \"ip-10-0-140-107.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:41:49.541956 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:41:49.541888 4939 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5b0cabc6-4960-4dc1-8317-3faebf25b30c-workload-certs\") on node \"ip-10-0-140-107.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:41:49.541956 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:41:49.541902 4939 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5b0cabc6-4960-4dc1-8317-3faebf25b30c-istiod-ca-cert\") on node \"ip-10-0-140-107.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:41:49.541956 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:41:49.541938 4939 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5b0cabc6-4960-4dc1-8317-3faebf25b30c-workload-socket\") on node \"ip-10-0-140-107.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:41:49.543606 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:41:49.543581 4939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5b0cabc6-4960-4dc1-8317-3faebf25b30c-istio-envoy" pod "5b0cabc6-4960-4dc1-8317-3faebf25b30c" (UID: "5b0cabc6-4960-4dc1-8317-3faebf25b30c"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 01:41:49.543719 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:41:49.543585 4939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5b0cabc6-4960-4dc1-8317-3faebf25b30c-istio-token" pod "5b0cabc6-4960-4dc1-8317-3faebf25b30c" (UID: "5b0cabc6-4960-4dc1-8317-3faebf25b30c"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 01:41:49.543804 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:41:49.543758 4939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/5b0cabc6-4960-4dc1-8317-3faebf25b30c-istio-podinfo" pod "5b0cabc6-4960-4dc1-8317-3faebf25b30c" (UID: "5b0cabc6-4960-4dc1-8317-3faebf25b30c"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 01:41:49.543960 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:41:49.543889 4939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5b0cabc6-4960-4dc1-8317-3faebf25b30c-kube-api-access-29zgn" pod "5b0cabc6-4960-4dc1-8317-3faebf25b30c" (UID: "5b0cabc6-4960-4dc1-8317-3faebf25b30c"). InnerVolumeSpecName "kube-api-access-29zgn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 01:41:49.642853 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:41:49.642823 4939 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5b0cabc6-4960-4dc1-8317-3faebf25b30c-istio-envoy\") on node \"ip-10-0-140-107.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:41:49.642853 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:41:49.642850 4939 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-29zgn\" (UniqueName: \"kubernetes.io/projected/5b0cabc6-4960-4dc1-8317-3faebf25b30c-kube-api-access-29zgn\") on node \"ip-10-0-140-107.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:41:49.643072 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:41:49.642863 4939 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5b0cabc6-4960-4dc1-8317-3faebf25b30c-istio-podinfo\") on node \"ip-10-0-140-107.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:41:49.643072 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:41:49.642877 4939 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5b0cabc6-4960-4dc1-8317-3faebf25b30c-istio-token\") on node \"ip-10-0-140-107.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:41:49.774675 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:41:49.774586 4939 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-556c14-openshift-default-754b5bc799-dt7h4"] Nov 03 01:41:49.774863 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:41:49.774834 4939 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w3-556c14-openshift-default-754b5bc799-dt7h4" podUID="fdeb6f74-c52a-4497-be8e-516b4244aefe" containerName="istio-proxy" containerID="cri-o://fa857baf4fd7a0bb80fc8c8b74ea6d8fb9998f9c7498de24814258378cbf5930" gracePeriod=30 Nov 03 01:41:49.827871 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:41:49.827845 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:41:49.836654 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:41:49.836627 4939 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w4-4a697f-openshift-default-666db9f8f5-qn8tj"] Nov 03 01:41:49.840867 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:41:49.840843 4939 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w4-4a697f-openshift-default-666db9f8f5-qn8tj"] Nov 03 01:41:49.875363 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:41:49.869742 4939 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5b0cabc6-4960-4dc1-8317-3faebf25b30c" path="/var/lib/kubelet/pods/5b0cabc6-4960-4dc1-8317-3faebf25b30c/volumes" Nov 03 01:41:49.949933 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:41:49.949887 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:41:50.266857 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:41:50.266816 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:41:50.381633 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:41:50.381601 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:41:50.799095 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:41:50.799055 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:41:50.923585 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:41:50.923487 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:41:51.500967 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:41:51.500931 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:41:51.533308 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:41:51.533273 4939 generic.go:356] "Generic (PLEG): container finished" podID="ccf63a28-a2d8-4686-8555-968e54429210" containerID="5607cb9bdc1566797246d0558e8a8a2611dd45b7a214459c511d4a8025ac6e3b" exitCode=0 Nov 03 01:41:51.533494 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:41:51.533349 4939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-f7680d-openshift-default-c4488d5c9-qstvv" event={"ID":"ccf63a28-a2d8-4686-8555-968e54429210","Type":"ContainerDied","Data":"5607cb9bdc1566797246d0558e8a8a2611dd45b7a214459c511d4a8025ac6e3b"} Nov 03 01:41:51.596517 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:41:51.596449 4939 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-f7680d-openshift-default-c4488d5c9-qstvv" Nov 03 01:41:51.611430 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:41:51.611401 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:41:51.668460 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:41:51.668427 4939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ccf63a28-a2d8-4686-8555-968e54429210-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ccf63a28-a2d8-4686-8555-968e54429210-istiod-ca-cert\") pod \"ccf63a28-a2d8-4686-8555-968e54429210\" (UID: \"ccf63a28-a2d8-4686-8555-968e54429210\") " Nov 03 01:41:51.668671 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:41:51.668499 4939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ccf63a28-a2d8-4686-8555-968e54429210-istio-token\" (UniqueName: \"kubernetes.io/projected/ccf63a28-a2d8-4686-8555-968e54429210-istio-token\") pod \"ccf63a28-a2d8-4686-8555-968e54429210\" (UID: \"ccf63a28-a2d8-4686-8555-968e54429210\") " Nov 03 01:41:51.668671 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:41:51.668545 4939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/ccf63a28-a2d8-4686-8555-968e54429210-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ccf63a28-a2d8-4686-8555-968e54429210-istio-podinfo\") pod \"ccf63a28-a2d8-4686-8555-968e54429210\" (UID: \"ccf63a28-a2d8-4686-8555-968e54429210\") " Nov 03 01:41:51.668671 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:41:51.668605 4939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ccf63a28-a2d8-4686-8555-968e54429210-kube-api-access-2nvmc\" (UniqueName: \"kubernetes.io/projected/ccf63a28-a2d8-4686-8555-968e54429210-kube-api-access-2nvmc\") pod \"ccf63a28-a2d8-4686-8555-968e54429210\" (UID: \"ccf63a28-a2d8-4686-8555-968e54429210\") " Nov 03 01:41:51.668671 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:41:51.668652 4939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ccf63a28-a2d8-4686-8555-968e54429210-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ccf63a28-a2d8-4686-8555-968e54429210-credential-socket\") pod \"ccf63a28-a2d8-4686-8555-968e54429210\" (UID: \"ccf63a28-a2d8-4686-8555-968e54429210\") " Nov 03 01:41:51.668938 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:41:51.668689 4939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ccf63a28-a2d8-4686-8555-968e54429210-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ccf63a28-a2d8-4686-8555-968e54429210-workload-certs\") pod \"ccf63a28-a2d8-4686-8555-968e54429210\" (UID: \"ccf63a28-a2d8-4686-8555-968e54429210\") " Nov 03 01:41:51.668938 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:41:51.668794 4939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ccf63a28-a2d8-4686-8555-968e54429210-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ccf63a28-a2d8-4686-8555-968e54429210-istio-data\") pod \"ccf63a28-a2d8-4686-8555-968e54429210\" (UID: \"ccf63a28-a2d8-4686-8555-968e54429210\") " Nov 03 01:41:51.668938 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:41:51.668870 4939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ccf63a28-a2d8-4686-8555-968e54429210-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ccf63a28-a2d8-4686-8555-968e54429210-workload-socket\") pod \"ccf63a28-a2d8-4686-8555-968e54429210\" (UID: \"ccf63a28-a2d8-4686-8555-968e54429210\") " Nov 03 01:41:51.669127 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:41:51.668963 4939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ccf63a28-a2d8-4686-8555-968e54429210-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ccf63a28-a2d8-4686-8555-968e54429210-istio-envoy\") pod \"ccf63a28-a2d8-4686-8555-968e54429210\" (UID: \"ccf63a28-a2d8-4686-8555-968e54429210\") " Nov 03 01:41:51.670011 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:41:51.669980 4939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ccf63a28-a2d8-4686-8555-968e54429210-credential-socket" pod "ccf63a28-a2d8-4686-8555-968e54429210" (UID: "ccf63a28-a2d8-4686-8555-968e54429210"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 01:41:51.670460 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:41:51.670021 4939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ccf63a28-a2d8-4686-8555-968e54429210-workload-certs" pod "ccf63a28-a2d8-4686-8555-968e54429210" (UID: "ccf63a28-a2d8-4686-8555-968e54429210"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 01:41:51.670738 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:41:51.670403 4939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ccf63a28-a2d8-4686-8555-968e54429210-workload-socket" pod "ccf63a28-a2d8-4686-8555-968e54429210" (UID: "ccf63a28-a2d8-4686-8555-968e54429210"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 01:41:51.670861 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:41:51.670460 4939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ccf63a28-a2d8-4686-8555-968e54429210-istiod-ca-cert" pod "ccf63a28-a2d8-4686-8555-968e54429210" (UID: "ccf63a28-a2d8-4686-8555-968e54429210"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 01:41:51.670978 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:41:51.670933 4939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ccf63a28-a2d8-4686-8555-968e54429210-istio-data" pod "ccf63a28-a2d8-4686-8555-968e54429210" (UID: "ccf63a28-a2d8-4686-8555-968e54429210"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 01:41:51.672239 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:41:51.672211 4939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ccf63a28-a2d8-4686-8555-968e54429210-istio-envoy" pod "ccf63a28-a2d8-4686-8555-968e54429210" (UID: "ccf63a28-a2d8-4686-8555-968e54429210"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 01:41:51.672361 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:41:51.672317 4939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ccf63a28-a2d8-4686-8555-968e54429210-kube-api-access-2nvmc" pod "ccf63a28-a2d8-4686-8555-968e54429210" (UID: "ccf63a28-a2d8-4686-8555-968e54429210"). InnerVolumeSpecName "kube-api-access-2nvmc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 01:41:51.672728 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:41:51.672707 4939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/ccf63a28-a2d8-4686-8555-968e54429210-istio-podinfo" pod "ccf63a28-a2d8-4686-8555-968e54429210" (UID: "ccf63a28-a2d8-4686-8555-968e54429210"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 01:41:51.673584 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:41:51.673231 4939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ccf63a28-a2d8-4686-8555-968e54429210-istio-token" pod "ccf63a28-a2d8-4686-8555-968e54429210" (UID: "ccf63a28-a2d8-4686-8555-968e54429210"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 01:41:51.730607 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:41:51.730536 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:41:51.771022 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:41:51.770986 4939 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ccf63a28-a2d8-4686-8555-968e54429210-istio-podinfo\") on node \"ip-10-0-140-107.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:41:51.771022 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:41:51.771018 4939 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2nvmc\" (UniqueName: \"kubernetes.io/projected/ccf63a28-a2d8-4686-8555-968e54429210-kube-api-access-2nvmc\") on node \"ip-10-0-140-107.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:41:51.771022 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:41:51.771030 4939 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ccf63a28-a2d8-4686-8555-968e54429210-credential-socket\") on node \"ip-10-0-140-107.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:41:51.771270 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:41:51.771039 4939 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ccf63a28-a2d8-4686-8555-968e54429210-workload-certs\") on node \"ip-10-0-140-107.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:41:51.771270 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:41:51.771047 4939 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ccf63a28-a2d8-4686-8555-968e54429210-istio-data\") on node \"ip-10-0-140-107.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:41:51.771270 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:41:51.771056 4939 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ccf63a28-a2d8-4686-8555-968e54429210-workload-socket\") on node \"ip-10-0-140-107.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:41:51.771270 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:41:51.771064 4939 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ccf63a28-a2d8-4686-8555-968e54429210-istio-envoy\") on node \"ip-10-0-140-107.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:41:51.771270 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:41:51.771072 4939 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ccf63a28-a2d8-4686-8555-968e54429210-istiod-ca-cert\") on node \"ip-10-0-140-107.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:41:51.771270 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:41:51.771080 4939 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ccf63a28-a2d8-4686-8555-968e54429210-istio-token\") on node \"ip-10-0-140-107.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:41:52.394594 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:41:52.394555 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:41:52.541111 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:41:52.541078 4939 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-f7680d-openshift-default-c4488d5c9-qstvv" Nov 03 01:41:52.541521 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:41:52.541089 4939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-f7680d-openshift-default-c4488d5c9-qstvv" event={"ID":"ccf63a28-a2d8-4686-8555-968e54429210","Type":"ContainerDied","Data":"0ff751b36761ef753b4d27625c2cffe978c6f5af2a54a83e4b72ef7c76c9df97"} Nov 03 01:41:52.541521 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:41:52.541214 4939 scope.go:122] "RemoveContainer" containerID="5607cb9bdc1566797246d0558e8a8a2611dd45b7a214459c511d4a8025ac6e3b" Nov 03 01:41:52.560154 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:41:52.560125 4939 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-f7680d-openshift-default-c4488d5c9-qstvv"] Nov 03 01:41:52.562233 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:41:52.562212 4939 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-f7680d-openshift-default-c4488d5c9-qstvv"] Nov 03 01:41:53.011310 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:41:53.011278 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:41:53.170423 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:41:53.170388 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:41:53.287584 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:41:53.287480 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:41:53.403760 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:41:53.403725 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:41:53.655783 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:41:53.655744 4939 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-556c14-669d5fb59f-cjtpf" podUID="192c8a1d-60d2-4928-9579-0231138280bb" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 03 01:41:53.691885 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:41:53.691855 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:41:53.868615 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:41:53.868582 4939 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ccf63a28-a2d8-4686-8555-968e54429210" path="/var/lib/kubelet/pods/ccf63a28-a2d8-4686-8555-968e54429210/volumes" Nov 03 01:41:54.426375 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:41:54.426335 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:41:54.663196 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:41:54.663159 4939 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-556c14-566c7cc65c-lcbl4" podUID="23f02c28-7f9e-40d8-990c-aea9e85f1bf6" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 03 01:41:55.013564 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:41:55.013530 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:41:55.075760 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:41:55.075727 4939 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w3-556c14-openshift-default-754b5bc799-dt7h4" Nov 03 01:41:55.217720 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:41:55.217687 4939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fdeb6f74-c52a-4497-be8e-516b4244aefe-kube-api-access-l9l2j\" (UniqueName: \"kubernetes.io/projected/fdeb6f74-c52a-4497-be8e-516b4244aefe-kube-api-access-l9l2j\") pod \"fdeb6f74-c52a-4497-be8e-516b4244aefe\" (UID: \"fdeb6f74-c52a-4497-be8e-516b4244aefe\") " Nov 03 01:41:55.217947 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:41:55.217759 4939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fdeb6f74-c52a-4497-be8e-516b4244aefe-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fdeb6f74-c52a-4497-be8e-516b4244aefe-credential-socket\") pod \"fdeb6f74-c52a-4497-be8e-516b4244aefe\" (UID: \"fdeb6f74-c52a-4497-be8e-516b4244aefe\") " Nov 03 01:41:55.217947 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:41:55.217857 4939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fdeb6f74-c52a-4497-be8e-516b4244aefe-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fdeb6f74-c52a-4497-be8e-516b4244aefe-istiod-ca-cert\") pod \"fdeb6f74-c52a-4497-be8e-516b4244aefe\" (UID: \"fdeb6f74-c52a-4497-be8e-516b4244aefe\") " Nov 03 01:41:55.217947 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:41:55.217890 4939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/fdeb6f74-c52a-4497-be8e-516b4244aefe-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fdeb6f74-c52a-4497-be8e-516b4244aefe-istio-podinfo\") pod \"fdeb6f74-c52a-4497-be8e-516b4244aefe\" (UID: \"fdeb6f74-c52a-4497-be8e-516b4244aefe\") " Nov 03 01:41:55.218146 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:41:55.217956 4939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fdeb6f74-c52a-4497-be8e-516b4244aefe-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fdeb6f74-c52a-4497-be8e-516b4244aefe-workload-certs\") pod \"fdeb6f74-c52a-4497-be8e-516b4244aefe\" (UID: \"fdeb6f74-c52a-4497-be8e-516b4244aefe\") " Nov 03 01:41:55.218146 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:41:55.218004 4939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fdeb6f74-c52a-4497-be8e-516b4244aefe-istio-token\" (UniqueName: \"kubernetes.io/projected/fdeb6f74-c52a-4497-be8e-516b4244aefe-istio-token\") pod \"fdeb6f74-c52a-4497-be8e-516b4244aefe\" (UID: \"fdeb6f74-c52a-4497-be8e-516b4244aefe\") " Nov 03 01:41:55.218146 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:41:55.218096 4939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fdeb6f74-c52a-4497-be8e-516b4244aefe-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fdeb6f74-c52a-4497-be8e-516b4244aefe-istio-envoy\") pod \"fdeb6f74-c52a-4497-be8e-516b4244aefe\" (UID: \"fdeb6f74-c52a-4497-be8e-516b4244aefe\") " Nov 03 01:41:55.218146 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:41:55.218128 4939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fdeb6f74-c52a-4497-be8e-516b4244aefe-credential-socket" pod "fdeb6f74-c52a-4497-be8e-516b4244aefe" (UID: "fdeb6f74-c52a-4497-be8e-516b4244aefe"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 01:41:55.218393 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:41:55.218210 4939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fdeb6f74-c52a-4497-be8e-516b4244aefe-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fdeb6f74-c52a-4497-be8e-516b4244aefe-workload-socket\") pod \"fdeb6f74-c52a-4497-be8e-516b4244aefe\" (UID: \"fdeb6f74-c52a-4497-be8e-516b4244aefe\") " Nov 03 01:41:55.218393 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:41:55.218248 4939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fdeb6f74-c52a-4497-be8e-516b4244aefe-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fdeb6f74-c52a-4497-be8e-516b4244aefe-istio-data\") pod \"fdeb6f74-c52a-4497-be8e-516b4244aefe\" (UID: \"fdeb6f74-c52a-4497-be8e-516b4244aefe\") " Nov 03 01:41:55.218393 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:41:55.218266 4939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fdeb6f74-c52a-4497-be8e-516b4244aefe-workload-certs" pod "fdeb6f74-c52a-4497-be8e-516b4244aefe" (UID: "fdeb6f74-c52a-4497-be8e-516b4244aefe"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 01:41:55.218393 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:41:55.218278 4939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fdeb6f74-c52a-4497-be8e-516b4244aefe-istiod-ca-cert" pod "fdeb6f74-c52a-4497-be8e-516b4244aefe" (UID: "fdeb6f74-c52a-4497-be8e-516b4244aefe"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 01:41:55.218639 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:41:55.218551 4939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fdeb6f74-c52a-4497-be8e-516b4244aefe-istio-data" pod "fdeb6f74-c52a-4497-be8e-516b4244aefe" (UID: "fdeb6f74-c52a-4497-be8e-516b4244aefe"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 01:41:55.218639 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:41:55.218570 4939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fdeb6f74-c52a-4497-be8e-516b4244aefe-workload-socket" pod "fdeb6f74-c52a-4497-be8e-516b4244aefe" (UID: "fdeb6f74-c52a-4497-be8e-516b4244aefe"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 01:41:55.218753 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:41:55.218719 4939 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fdeb6f74-c52a-4497-be8e-516b4244aefe-istiod-ca-cert\") on node \"ip-10-0-140-107.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:41:55.218753 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:41:55.218741 4939 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fdeb6f74-c52a-4497-be8e-516b4244aefe-workload-certs\") on node \"ip-10-0-140-107.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:41:55.218873 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:41:55.218758 4939 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fdeb6f74-c52a-4497-be8e-516b4244aefe-workload-socket\") on node \"ip-10-0-140-107.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:41:55.218873 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:41:55.218772 4939 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fdeb6f74-c52a-4497-be8e-516b4244aefe-istio-data\") on node \"ip-10-0-140-107.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:41:55.218873 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:41:55.218786 4939 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fdeb6f74-c52a-4497-be8e-516b4244aefe-credential-socket\") on node \"ip-10-0-140-107.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:41:55.220393 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:41:55.220366 4939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fdeb6f74-c52a-4497-be8e-516b4244aefe-kube-api-access-l9l2j" pod "fdeb6f74-c52a-4497-be8e-516b4244aefe" (UID: "fdeb6f74-c52a-4497-be8e-516b4244aefe"). InnerVolumeSpecName "kube-api-access-l9l2j". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 01:41:55.220538 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:41:55.220427 4939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fdeb6f74-c52a-4497-be8e-516b4244aefe-istio-token" pod "fdeb6f74-c52a-4497-be8e-516b4244aefe" (UID: "fdeb6f74-c52a-4497-be8e-516b4244aefe"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 01:41:55.220600 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:41:55.220541 4939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/fdeb6f74-c52a-4497-be8e-516b4244aefe-istio-podinfo" pod "fdeb6f74-c52a-4497-be8e-516b4244aefe" (UID: "fdeb6f74-c52a-4497-be8e-516b4244aefe"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 01:41:55.220856 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:41:55.220832 4939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fdeb6f74-c52a-4497-be8e-516b4244aefe-istio-envoy" pod "fdeb6f74-c52a-4497-be8e-516b4244aefe" (UID: "fdeb6f74-c52a-4497-be8e-516b4244aefe"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 01:41:55.233106 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:41:55.233081 4939 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w3-83755a/distinct-2-w3-83755a-kserve-6cf66b995d-2l47b"] Nov 03 01:41:55.319767 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:41:55.319692 4939 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fdeb6f74-c52a-4497-be8e-516b4244aefe-istio-envoy\") on node \"ip-10-0-140-107.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:41:55.319767 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:41:55.319720 4939 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-l9l2j\" (UniqueName: \"kubernetes.io/projected/fdeb6f74-c52a-4497-be8e-516b4244aefe-kube-api-access-l9l2j\") on node \"ip-10-0-140-107.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:41:55.319767 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:41:55.319733 4939 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fdeb6f74-c52a-4497-be8e-516b4244aefe-istio-podinfo\") on node \"ip-10-0-140-107.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:41:55.319767 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:41:55.319742 4939 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fdeb6f74-c52a-4497-be8e-516b4244aefe-istio-token\") on node \"ip-10-0-140-107.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:41:55.371282 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:41:55.371233 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:41:55.562684 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:41:55.562645 4939 generic.go:356] "Generic (PLEG): container finished" podID="fdeb6f74-c52a-4497-be8e-516b4244aefe" containerID="fa857baf4fd7a0bb80fc8c8b74ea6d8fb9998f9c7498de24814258378cbf5930" exitCode=0 Nov 03 01:41:55.562891 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:41:55.562693 4939 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w3-556c14-openshift-default-754b5bc799-dt7h4" Nov 03 01:41:55.562891 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:41:55.562727 4939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-556c14-openshift-default-754b5bc799-dt7h4" event={"ID":"fdeb6f74-c52a-4497-be8e-516b4244aefe","Type":"ContainerDied","Data":"fa857baf4fd7a0bb80fc8c8b74ea6d8fb9998f9c7498de24814258378cbf5930"} Nov 03 01:41:55.562891 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:41:55.562783 4939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-556c14-openshift-default-754b5bc799-dt7h4" event={"ID":"fdeb6f74-c52a-4497-be8e-516b4244aefe","Type":"ContainerDied","Data":"a9786ae06ff4ee4af5655728e22c71b793b96e1077ba038e387f54d43822d8e4"} Nov 03 01:41:55.562891 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:41:55.562805 4939 scope.go:122] "RemoveContainer" containerID="fa857baf4fd7a0bb80fc8c8b74ea6d8fb9998f9c7498de24814258378cbf5930" Nov 03 01:41:55.577489 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:41:55.577469 4939 scope.go:122] "RemoveContainer" containerID="fa857baf4fd7a0bb80fc8c8b74ea6d8fb9998f9c7498de24814258378cbf5930" Nov 03 01:41:55.577752 ip-10-0-140-107 kubenswrapper[4939]: E1103 01:41:55.577731 4939 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fa857baf4fd7a0bb80fc8c8b74ea6d8fb9998f9c7498de24814258378cbf5930\": container with ID starting with fa857baf4fd7a0bb80fc8c8b74ea6d8fb9998f9c7498de24814258378cbf5930 not found: ID does not exist" containerID="fa857baf4fd7a0bb80fc8c8b74ea6d8fb9998f9c7498de24814258378cbf5930" Nov 03 01:41:55.577808 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:41:55.577760 4939 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fa857baf4fd7a0bb80fc8c8b74ea6d8fb9998f9c7498de24814258378cbf5930"} err="failed to get container status \"fa857baf4fd7a0bb80fc8c8b74ea6d8fb9998f9c7498de24814258378cbf5930\": rpc error: code = NotFound desc = could not find container \"fa857baf4fd7a0bb80fc8c8b74ea6d8fb9998f9c7498de24814258378cbf5930\": container with ID starting with fa857baf4fd7a0bb80fc8c8b74ea6d8fb9998f9c7498de24814258378cbf5930 not found: ID does not exist" Nov 03 01:41:55.589161 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:41:55.589137 4939 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-556c14-openshift-default-754b5bc799-dt7h4"] Nov 03 01:41:55.592946 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:41:55.592919 4939 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-556c14-openshift-default-754b5bc799-dt7h4"] Nov 03 01:41:55.715200 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:41:55.715167 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:41:55.867460 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:41:55.867430 4939 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fdeb6f74-c52a-4497-be8e-516b4244aefe" path="/var/lib/kubelet/pods/fdeb6f74-c52a-4497-be8e-516b4244aefe/volumes" Nov 03 01:41:56.236004 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:41:56.235926 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:41:56.361422 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:41:56.358010 4939 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-w1-2729f2/rate-test-model-2729f2-kserve-6d6745bb5c-dm56b" podUID="5b48a10a-63a5-440b-b08d-0e3190a77997" containerName="main" containerID="cri-o://ae1a7f2c9c04b99985d987f515b6cce073c14ad1637969a50c79e97285d8ab45" gracePeriod=45 Nov 03 01:41:56.515275 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:41:56.515186 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:41:56.574950 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:41:56.573150 4939 generic.go:356] "Generic (PLEG): container finished" podID="5b48a10a-63a5-440b-b08d-0e3190a77997" containerID="ae1a7f2c9c04b99985d987f515b6cce073c14ad1637969a50c79e97285d8ab45" exitCode=0 Nov 03 01:41:56.574950 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:41:56.573314 4939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w1-2729f2/rate-test-model-2729f2-kserve-6d6745bb5c-dm56b" event={"ID":"5b48a10a-63a5-440b-b08d-0e3190a77997","Type":"ContainerDied","Data":"ae1a7f2c9c04b99985d987f515b6cce073c14ad1637969a50c79e97285d8ab45"} Nov 03 01:41:56.588399 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:41:56.588360 4939 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-4a697f-94d8d8fc9-kzr29" podUID="ef71e635-99c1-4f1c-99b0-014934ba2f76" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 03 01:41:56.593315 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:41:56.593286 4939 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-4a697f-79fb8d7d9-gr8cv" podUID="ef6478d3-bd62-45a7-8acc-1055c24dc53b" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 03 01:41:56.643477 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:41:56.643450 4939 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w1-2729f2/rate-test-model-2729f2-kserve-6d6745bb5c-dm56b" Nov 03 01:41:56.737676 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:41:56.737639 4939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5b48a10a-63a5-440b-b08d-0e3190a77997-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5b48a10a-63a5-440b-b08d-0e3190a77997-tmp-dir\") pod \"5b48a10a-63a5-440b-b08d-0e3190a77997\" (UID: \"5b48a10a-63a5-440b-b08d-0e3190a77997\") " Nov 03 01:41:56.738209 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:41:56.737697 4939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5b48a10a-63a5-440b-b08d-0e3190a77997-kube-api-access-9z42c\" (UniqueName: \"kubernetes.io/projected/5b48a10a-63a5-440b-b08d-0e3190a77997-kube-api-access-9z42c\") pod \"5b48a10a-63a5-440b-b08d-0e3190a77997\" (UID: \"5b48a10a-63a5-440b-b08d-0e3190a77997\") " Nov 03 01:41:56.738209 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:41:56.737716 4939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5b48a10a-63a5-440b-b08d-0e3190a77997-tls-certs\" (UniqueName: \"kubernetes.io/secret/5b48a10a-63a5-440b-b08d-0e3190a77997-tls-certs\") pod \"5b48a10a-63a5-440b-b08d-0e3190a77997\" (UID: \"5b48a10a-63a5-440b-b08d-0e3190a77997\") " Nov 03 01:41:56.738209 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:41:56.737768 4939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5b48a10a-63a5-440b-b08d-0e3190a77997-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5b48a10a-63a5-440b-b08d-0e3190a77997-model-cache\") pod \"5b48a10a-63a5-440b-b08d-0e3190a77997\" (UID: \"5b48a10a-63a5-440b-b08d-0e3190a77997\") " Nov 03 01:41:56.738209 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:41:56.737831 4939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5b48a10a-63a5-440b-b08d-0e3190a77997-dshm\" (UniqueName: \"kubernetes.io/empty-dir/5b48a10a-63a5-440b-b08d-0e3190a77997-dshm\") pod \"5b48a10a-63a5-440b-b08d-0e3190a77997\" (UID: \"5b48a10a-63a5-440b-b08d-0e3190a77997\") " Nov 03 01:41:56.738209 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:41:56.737890 4939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5b48a10a-63a5-440b-b08d-0e3190a77997-home\" (UniqueName: \"kubernetes.io/empty-dir/5b48a10a-63a5-440b-b08d-0e3190a77997-home\") pod \"5b48a10a-63a5-440b-b08d-0e3190a77997\" (UID: \"5b48a10a-63a5-440b-b08d-0e3190a77997\") " Nov 03 01:41:56.738209 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:41:56.738038 4939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5b48a10a-63a5-440b-b08d-0e3190a77997-tmp-dir" pod "5b48a10a-63a5-440b-b08d-0e3190a77997" (UID: "5b48a10a-63a5-440b-b08d-0e3190a77997"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 01:41:56.738209 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:41:56.738141 4939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5b48a10a-63a5-440b-b08d-0e3190a77997-model-cache" pod "5b48a10a-63a5-440b-b08d-0e3190a77997" (UID: "5b48a10a-63a5-440b-b08d-0e3190a77997"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 01:41:56.738209 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:41:56.738191 4939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5b48a10a-63a5-440b-b08d-0e3190a77997-home" pod "5b48a10a-63a5-440b-b08d-0e3190a77997" (UID: "5b48a10a-63a5-440b-b08d-0e3190a77997"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 01:41:56.738539 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:41:56.738363 4939 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5b48a10a-63a5-440b-b08d-0e3190a77997-home\") on node \"ip-10-0-140-107.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:41:56.738539 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:41:56.738402 4939 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5b48a10a-63a5-440b-b08d-0e3190a77997-tmp-dir\") on node \"ip-10-0-140-107.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:41:56.738539 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:41:56.738413 4939 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5b48a10a-63a5-440b-b08d-0e3190a77997-model-cache\") on node \"ip-10-0-140-107.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:41:56.739992 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:41:56.739964 4939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5b48a10a-63a5-440b-b08d-0e3190a77997-kube-api-access-9z42c" pod "5b48a10a-63a5-440b-b08d-0e3190a77997" (UID: "5b48a10a-63a5-440b-b08d-0e3190a77997"). InnerVolumeSpecName "kube-api-access-9z42c". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 01:41:56.740413 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:41:56.740388 4939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5b48a10a-63a5-440b-b08d-0e3190a77997-dshm" pod "5b48a10a-63a5-440b-b08d-0e3190a77997" (UID: "5b48a10a-63a5-440b-b08d-0e3190a77997"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 01:41:56.740532 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:41:56.740451 4939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5b48a10a-63a5-440b-b08d-0e3190a77997-tls-certs" pod "5b48a10a-63a5-440b-b08d-0e3190a77997" (UID: "5b48a10a-63a5-440b-b08d-0e3190a77997"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 01:41:56.839237 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:41:56.839205 4939 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5b48a10a-63a5-440b-b08d-0e3190a77997-dshm\") on node \"ip-10-0-140-107.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:41:56.839237 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:41:56.839235 4939 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9z42c\" (UniqueName: \"kubernetes.io/projected/5b48a10a-63a5-440b-b08d-0e3190a77997-kube-api-access-9z42c\") on node \"ip-10-0-140-107.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:41:56.839429 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:41:56.839247 4939 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5b48a10a-63a5-440b-b08d-0e3190a77997-tls-certs\") on node \"ip-10-0-140-107.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:41:57.195460 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:41:57.195380 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:41:57.303231 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:41:57.303170 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:41:57.506091 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:41:57.506008 4939 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-4pw7v"] Nov 03 01:41:57.582256 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:41:57.582228 4939 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w1-2729f2/rate-test-model-2729f2-kserve-6d6745bb5c-dm56b" Nov 03 01:41:57.582449 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:41:57.582263 4939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w1-2729f2/rate-test-model-2729f2-kserve-6d6745bb5c-dm56b" event={"ID":"5b48a10a-63a5-440b-b08d-0e3190a77997","Type":"ContainerDied","Data":"cbe4f516623fe32fdfeabf0ffda3addf8b8135c9ca9bdfb309016fd6656d6c5f"} Nov 03 01:41:57.582449 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:41:57.582313 4939 scope.go:122] "RemoveContainer" containerID="ae1a7f2c9c04b99985d987f515b6cce073c14ad1637969a50c79e97285d8ab45" Nov 03 01:41:57.604776 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:41:57.604746 4939 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w1-2729f2/rate-test-model-2729f2-kserve-6d6745bb5c-dm56b"] Nov 03 01:41:57.607225 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:41:57.607191 4939 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-w1-2729f2/rate-test-model-2729f2-kserve-6d6745bb5c-dm56b"] Nov 03 01:41:57.800642 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:41:57.800543 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:41:57.868561 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:41:57.868527 4939 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5b48a10a-63a5-440b-b08d-0e3190a77997" path="/var/lib/kubelet/pods/5b48a10a-63a5-440b-b08d-0e3190a77997/volumes" Nov 03 01:41:58.471780 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:41:58.471749 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:41:59.046520 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:41:59.046485 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:41:59.167239 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:41:59.167202 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:41:59.283334 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:41:59.283302 4939 scope.go:122] "RemoveContainer" containerID="bdd2ff5643812c0947899c193a7d22052a6cc38247cf5cbb3acc63b98c0907d1" Nov 03 01:41:59.668340 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:41:59.668311 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:41:59.905180 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:41:59.905143 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:42:00.131487 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:42:00.131452 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:42:00.743321 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:42:00.743286 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:42:00.851443 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:42:00.851407 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:42:01.032332 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:42:01.032245 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:42:01.383732 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:42:01.383699 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:42:01.976896 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:42:01.976865 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:42:02.276452 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:42:02.276421 4939 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-4a697f-79fb8d7d9-gr8cv" Nov 03 01:42:02.279840 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:42:02.279811 4939 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-4a697f-94d8d8fc9-kzr29" Nov 03 01:42:02.310141 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:42:02.310106 4939 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w4-650be0-86d95657b7-vm62r"] Nov 03 01:42:02.322691 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:42:02.322657 4939 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w4-650be0-58fc9b6455-9jl7f"] Nov 03 01:42:02.406929 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:42:02.406880 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:42:02.410106 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:42:02.410078 4939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ef71e635-99c1-4f1c-99b0-014934ba2f76-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ef71e635-99c1-4f1c-99b0-014934ba2f76-plugins-config-volume\") pod \"ef71e635-99c1-4f1c-99b0-014934ba2f76\" (UID: \"ef71e635-99c1-4f1c-99b0-014934ba2f76\") " Nov 03 01:42:02.410260 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:42:02.410119 4939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ef71e635-99c1-4f1c-99b0-014934ba2f76-kube-api-access-t8s9l\" (UniqueName: \"kubernetes.io/projected/ef71e635-99c1-4f1c-99b0-014934ba2f76-kube-api-access-t8s9l\") pod \"ef71e635-99c1-4f1c-99b0-014934ba2f76\" (UID: \"ef71e635-99c1-4f1c-99b0-014934ba2f76\") " Nov 03 01:42:02.410260 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:42:02.410185 4939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ef6478d3-bd62-45a7-8acc-1055c24dc53b-kube-api-access-hrtqs\" (UniqueName: \"kubernetes.io/projected/ef6478d3-bd62-45a7-8acc-1055c24dc53b-kube-api-access-hrtqs\") pod \"ef6478d3-bd62-45a7-8acc-1055c24dc53b\" (UID: \"ef6478d3-bd62-45a7-8acc-1055c24dc53b\") " Nov 03 01:42:02.410260 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:42:02.410230 4939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ef6478d3-bd62-45a7-8acc-1055c24dc53b-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ef6478d3-bd62-45a7-8acc-1055c24dc53b-plugins-config-volume\") pod \"ef6478d3-bd62-45a7-8acc-1055c24dc53b\" (UID: \"ef6478d3-bd62-45a7-8acc-1055c24dc53b\") " Nov 03 01:42:02.410571 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:42:02.410526 4939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ef71e635-99c1-4f1c-99b0-014934ba2f76-plugins-config-volume" pod "ef71e635-99c1-4f1c-99b0-014934ba2f76" (UID: "ef71e635-99c1-4f1c-99b0-014934ba2f76"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 01:42:02.410867 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:42:02.410838 4939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ef6478d3-bd62-45a7-8acc-1055c24dc53b-plugins-config-volume" pod "ef6478d3-bd62-45a7-8acc-1055c24dc53b" (UID: "ef6478d3-bd62-45a7-8acc-1055c24dc53b"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 01:42:02.411031 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:42:02.410927 4939 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ef71e635-99c1-4f1c-99b0-014934ba2f76-plugins-config-volume\") on node \"ip-10-0-140-107.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:42:02.417146 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:42:02.417120 4939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ef6478d3-bd62-45a7-8acc-1055c24dc53b-kube-api-access-hrtqs" pod "ef6478d3-bd62-45a7-8acc-1055c24dc53b" (UID: "ef6478d3-bd62-45a7-8acc-1055c24dc53b"). InnerVolumeSpecName "kube-api-access-hrtqs". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 01:42:02.418782 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:42:02.418759 4939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ef71e635-99c1-4f1c-99b0-014934ba2f76-kube-api-access-t8s9l" pod "ef71e635-99c1-4f1c-99b0-014934ba2f76" (UID: "ef71e635-99c1-4f1c-99b0-014934ba2f76"). InnerVolumeSpecName "kube-api-access-t8s9l". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 01:42:02.512448 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:42:02.512362 4939 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-t8s9l\" (UniqueName: \"kubernetes.io/projected/ef71e635-99c1-4f1c-99b0-014934ba2f76-kube-api-access-t8s9l\") on node \"ip-10-0-140-107.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:42:02.512448 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:42:02.512397 4939 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hrtqs\" (UniqueName: \"kubernetes.io/projected/ef6478d3-bd62-45a7-8acc-1055c24dc53b-kube-api-access-hrtqs\") on node \"ip-10-0-140-107.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:42:02.512448 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:42:02.512407 4939 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ef6478d3-bd62-45a7-8acc-1055c24dc53b-plugins-config-volume\") on node \"ip-10-0-140-107.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:42:02.525039 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:42:02.525005 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:42:02.612534 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:42:02.612480 4939 generic.go:356] "Generic (PLEG): container finished" podID="ef71e635-99c1-4f1c-99b0-014934ba2f76" containerID="b6bb749d739c14292a547641ed6d373e8323f531c716126beccd5591b304334d" exitCode=0 Nov 03 01:42:02.612722 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:42:02.612530 4939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-4a697f-94d8d8fc9-kzr29" event={"ID":"ef71e635-99c1-4f1c-99b0-014934ba2f76","Type":"ContainerDied","Data":"b6bb749d739c14292a547641ed6d373e8323f531c716126beccd5591b304334d"} Nov 03 01:42:02.612722 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:42:02.612550 4939 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-4a697f-94d8d8fc9-kzr29" Nov 03 01:42:02.612722 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:42:02.612635 4939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-4a697f-94d8d8fc9-kzr29" event={"ID":"ef71e635-99c1-4f1c-99b0-014934ba2f76","Type":"ContainerDied","Data":"c6e3b07ee331461205ce146f0224539ed704b267c2ab2c2e28efdce3bb775e82"} Nov 03 01:42:02.612722 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:42:02.612668 4939 scope.go:122] "RemoveContainer" containerID="b6bb749d739c14292a547641ed6d373e8323f531c716126beccd5591b304334d" Nov 03 01:42:02.614277 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:42:02.614252 4939 generic.go:356] "Generic (PLEG): container finished" podID="ef6478d3-bd62-45a7-8acc-1055c24dc53b" containerID="4fcbd3d36db1cb376909b018910f367575ae43d65ae89c447ec2b35bc985516d" exitCode=0 Nov 03 01:42:02.614415 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:42:02.614307 4939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-4a697f-79fb8d7d9-gr8cv" event={"ID":"ef6478d3-bd62-45a7-8acc-1055c24dc53b","Type":"ContainerDied","Data":"4fcbd3d36db1cb376909b018910f367575ae43d65ae89c447ec2b35bc985516d"} Nov 03 01:42:02.614415 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:42:02.614342 4939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-4a697f-79fb8d7d9-gr8cv" event={"ID":"ef6478d3-bd62-45a7-8acc-1055c24dc53b","Type":"ContainerDied","Data":"a7dd1eeb8839df04e75fa07242029ec7dfca0cfaa3efaaaf034314d3b1980c80"} Nov 03 01:42:02.614415 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:42:02.614312 4939 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-4a697f-79fb8d7d9-gr8cv" Nov 03 01:42:02.625624 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:42:02.625603 4939 scope.go:122] "RemoveContainer" containerID="b6bb749d739c14292a547641ed6d373e8323f531c716126beccd5591b304334d" Nov 03 01:42:02.625941 ip-10-0-140-107 kubenswrapper[4939]: E1103 01:42:02.625882 4939 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b6bb749d739c14292a547641ed6d373e8323f531c716126beccd5591b304334d\": container with ID starting with b6bb749d739c14292a547641ed6d373e8323f531c716126beccd5591b304334d not found: ID does not exist" containerID="b6bb749d739c14292a547641ed6d373e8323f531c716126beccd5591b304334d" Nov 03 01:42:02.626055 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:42:02.625952 4939 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b6bb749d739c14292a547641ed6d373e8323f531c716126beccd5591b304334d"} err="failed to get container status \"b6bb749d739c14292a547641ed6d373e8323f531c716126beccd5591b304334d\": rpc error: code = NotFound desc = could not find container \"b6bb749d739c14292a547641ed6d373e8323f531c716126beccd5591b304334d\": container with ID starting with b6bb749d739c14292a547641ed6d373e8323f531c716126beccd5591b304334d not found: ID does not exist" Nov 03 01:42:02.626055 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:42:02.625973 4939 scope.go:122] "RemoveContainer" containerID="4fcbd3d36db1cb376909b018910f367575ae43d65ae89c447ec2b35bc985516d" Nov 03 01:42:02.639380 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:42:02.639361 4939 scope.go:122] "RemoveContainer" containerID="4fcbd3d36db1cb376909b018910f367575ae43d65ae89c447ec2b35bc985516d" Nov 03 01:42:02.639651 ip-10-0-140-107 kubenswrapper[4939]: E1103 01:42:02.639630 4939 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4fcbd3d36db1cb376909b018910f367575ae43d65ae89c447ec2b35bc985516d\": container with ID starting with 4fcbd3d36db1cb376909b018910f367575ae43d65ae89c447ec2b35bc985516d not found: ID does not exist" containerID="4fcbd3d36db1cb376909b018910f367575ae43d65ae89c447ec2b35bc985516d" Nov 03 01:42:02.639691 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:42:02.639660 4939 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4fcbd3d36db1cb376909b018910f367575ae43d65ae89c447ec2b35bc985516d"} err="failed to get container status \"4fcbd3d36db1cb376909b018910f367575ae43d65ae89c447ec2b35bc985516d\": rpc error: code = NotFound desc = could not find container \"4fcbd3d36db1cb376909b018910f367575ae43d65ae89c447ec2b35bc985516d\": container with ID starting with 4fcbd3d36db1cb376909b018910f367575ae43d65ae89c447ec2b35bc985516d not found: ID does not exist" Nov 03 01:42:02.643412 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:42:02.643388 4939 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w4-4a697f-94d8d8fc9-kzr29"] Nov 03 01:42:02.647313 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:42:02.647286 4939 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w4-4a697f-94d8d8fc9-kzr29"] Nov 03 01:42:02.661968 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:42:02.660530 4939 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w4-4a697f-79fb8d7d9-gr8cv"] Nov 03 01:42:02.662715 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:42:02.662696 4939 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w4-4a697f-79fb8d7d9-gr8cv"] Nov 03 01:42:02.873213 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:42:02.873179 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:42:03.084041 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:42:03.084005 4939 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w1-2729f2-openshift-default-9fb97b4fc-6jhb8"] Nov 03 01:42:03.084710 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:42:03.084687 4939 cpu_manager.go:414] "containerMap: removing container" podUID="f2d0a2aa-d02a-4153-b8e8-997c8d1120fe" containerName="payload-processing" Nov 03 01:42:03.084710 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:42:03.084711 4939 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f2d0a2aa-d02a-4153-b8e8-997c8d1120fe" containerName="payload-processing" Nov 03 01:42:03.084844 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:42:03.084726 4939 cpu_manager.go:414] "containerMap: removing container" podUID="ef6478d3-bd62-45a7-8acc-1055c24dc53b" containerName="payload-processing" Nov 03 01:42:03.084844 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:42:03.084732 4939 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ef6478d3-bd62-45a7-8acc-1055c24dc53b" containerName="payload-processing" Nov 03 01:42:03.084844 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:42:03.084743 4939 cpu_manager.go:414] "containerMap: removing container" podUID="16b64414-6144-4743-bc96-91dffcd4a6b7" containerName="payload-processing" Nov 03 01:42:03.084844 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:42:03.084749 4939 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="16b64414-6144-4743-bc96-91dffcd4a6b7" containerName="payload-processing" Nov 03 01:42:03.084844 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:42:03.084760 4939 cpu_manager.go:414] "containerMap: removing container" podUID="a79df098-23e4-4971-9263-3f52b7d25b0b" containerName="payload-pre-processing" Nov 03 01:42:03.084844 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:42:03.084765 4939 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a79df098-23e4-4971-9263-3f52b7d25b0b" containerName="payload-pre-processing" Nov 03 01:42:03.084844 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:42:03.084776 4939 cpu_manager.go:414] "containerMap: removing container" podUID="836753fc-f9ef-4f61-b3fe-e5b92b7ee27b" containerName="main" Nov 03 01:42:03.084844 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:42:03.084781 4939 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="836753fc-f9ef-4f61-b3fe-e5b92b7ee27b" containerName="main" Nov 03 01:42:03.084844 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:42:03.084791 4939 cpu_manager.go:414] "containerMap: removing container" podUID="4712e708-18a2-4a13-a12c-a5669397b2c3" containerName="main" Nov 03 01:42:03.084844 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:42:03.084795 4939 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4712e708-18a2-4a13-a12c-a5669397b2c3" containerName="main" Nov 03 01:42:03.084844 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:42:03.084801 4939 cpu_manager.go:414] "containerMap: removing container" podUID="3137d8da-32e8-47fd-87a6-f08aebe2a384" containerName="payload-pre-processing" Nov 03 01:42:03.084844 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:42:03.084806 4939 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3137d8da-32e8-47fd-87a6-f08aebe2a384" containerName="payload-pre-processing" Nov 03 01:42:03.084844 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:42:03.084816 4939 cpu_manager.go:414] "containerMap: removing container" podUID="6ea24c7b-6d13-4661-bdff-cd52792cb6b6" containerName="istio-proxy" Nov 03 01:42:03.084844 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:42:03.084821 4939 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6ea24c7b-6d13-4661-bdff-cd52792cb6b6" containerName="istio-proxy" Nov 03 01:42:03.084844 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:42:03.084829 4939 cpu_manager.go:414] "containerMap: removing container" podUID="285b736b-2a9f-40cd-96c4-3d7193d3eb8d" containerName="istio-proxy" Nov 03 01:42:03.084844 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:42:03.084833 4939 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="285b736b-2a9f-40cd-96c4-3d7193d3eb8d" containerName="istio-proxy" Nov 03 01:42:03.084844 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:42:03.084843 4939 cpu_manager.go:414] "containerMap: removing container" podUID="a68659b3-912d-4df9-a162-fa6d5f949d2a" containerName="istio-proxy" Nov 03 01:42:03.084844 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:42:03.084848 4939 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a68659b3-912d-4df9-a162-fa6d5f949d2a" containerName="istio-proxy" Nov 03 01:42:03.085427 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:42:03.084855 4939 cpu_manager.go:414] "containerMap: removing container" podUID="78a38f10-6762-489e-813a-ac1cb91f5844" containerName="istio-proxy" Nov 03 01:42:03.085427 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:42:03.084860 4939 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="78a38f10-6762-489e-813a-ac1cb91f5844" containerName="istio-proxy" Nov 03 01:42:03.085427 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:42:03.084866 4939 cpu_manager.go:414] "containerMap: removing container" podUID="78a38f10-6762-489e-813a-ac1cb91f5844" containerName="istio-proxy" Nov 03 01:42:03.085427 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:42:03.084870 4939 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="78a38f10-6762-489e-813a-ac1cb91f5844" containerName="istio-proxy" Nov 03 01:42:03.085427 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:42:03.084879 4939 cpu_manager.go:414] "containerMap: removing container" podUID="c6de1910-0365-4f48-99f4-b32aebd747bb" containerName="payload-processing" Nov 03 01:42:03.085427 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:42:03.084885 4939 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c6de1910-0365-4f48-99f4-b32aebd747bb" containerName="payload-processing" Nov 03 01:42:03.085427 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:42:03.084893 4939 cpu_manager.go:414] "containerMap: removing container" podUID="6ea24c7b-6d13-4661-bdff-cd52792cb6b6" containerName="istio-proxy" Nov 03 01:42:03.085427 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:42:03.084898 4939 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6ea24c7b-6d13-4661-bdff-cd52792cb6b6" containerName="istio-proxy" Nov 03 01:42:03.085427 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:42:03.084904 4939 cpu_manager.go:414] "containerMap: removing container" podUID="fb776e00-2114-4f9c-86b1-bb790c3d64ca" containerName="istio-proxy" Nov 03 01:42:03.085427 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:42:03.084922 4939 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fb776e00-2114-4f9c-86b1-bb790c3d64ca" containerName="istio-proxy" Nov 03 01:42:03.085427 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:42:03.084928 4939 cpu_manager.go:414] "containerMap: removing container" podUID="5b0cabc6-4960-4dc1-8317-3faebf25b30c" containerName="istio-proxy" Nov 03 01:42:03.085427 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:42:03.084933 4939 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5b0cabc6-4960-4dc1-8317-3faebf25b30c" containerName="istio-proxy" Nov 03 01:42:03.085427 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:42:03.084941 4939 cpu_manager.go:414] "containerMap: removing container" podUID="5b48a10a-63a5-440b-b08d-0e3190a77997" containerName="main" Nov 03 01:42:03.085427 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:42:03.084945 4939 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5b48a10a-63a5-440b-b08d-0e3190a77997" containerName="main" Nov 03 01:42:03.085427 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:42:03.084952 4939 cpu_manager.go:414] "containerMap: removing container" podUID="2de18086-761b-4773-b9fc-15ddc4a220fc" containerName="main" Nov 03 01:42:03.085427 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:42:03.084957 4939 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2de18086-761b-4773-b9fc-15ddc4a220fc" containerName="main" Nov 03 01:42:03.085427 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:42:03.084963 4939 cpu_manager.go:414] "containerMap: removing container" podUID="61b34e6e-2690-4080-bced-3a577c665f12" containerName="payload-pre-processing" Nov 03 01:42:03.085427 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:42:03.084967 4939 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="61b34e6e-2690-4080-bced-3a577c665f12" containerName="payload-pre-processing" Nov 03 01:42:03.085427 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:42:03.084974 4939 cpu_manager.go:414] "containerMap: removing container" podUID="7c2434b1-8471-41fb-b20d-118aa8f1afc4" containerName="payload-processing" Nov 03 01:42:03.085427 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:42:03.084979 4939 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7c2434b1-8471-41fb-b20d-118aa8f1afc4" containerName="payload-processing" Nov 03 01:42:03.085427 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:42:03.084985 4939 cpu_manager.go:414] "containerMap: removing container" podUID="9a90ca63-d97d-4695-ad39-eaaa58204ecc" containerName="payload-pre-processing" Nov 03 01:42:03.085427 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:42:03.084989 4939 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9a90ca63-d97d-4695-ad39-eaaa58204ecc" containerName="payload-pre-processing" Nov 03 01:42:03.085427 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:42:03.084997 4939 cpu_manager.go:414] "containerMap: removing container" podUID="285b736b-2a9f-40cd-96c4-3d7193d3eb8d" containerName="istio-proxy" Nov 03 01:42:03.085427 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:42:03.085001 4939 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="285b736b-2a9f-40cd-96c4-3d7193d3eb8d" containerName="istio-proxy" Nov 03 01:42:03.085427 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:42:03.085006 4939 cpu_manager.go:414] "containerMap: removing container" podUID="d0933771-7032-4cff-aafe-2f7211ebb549" containerName="istio-proxy" Nov 03 01:42:03.085427 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:42:03.085010 4939 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d0933771-7032-4cff-aafe-2f7211ebb549" containerName="istio-proxy" Nov 03 01:42:03.085427 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:42:03.085019 4939 cpu_manager.go:414] "containerMap: removing container" podUID="ef71e635-99c1-4f1c-99b0-014934ba2f76" containerName="payload-pre-processing" Nov 03 01:42:03.085427 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:42:03.085024 4939 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ef71e635-99c1-4f1c-99b0-014934ba2f76" containerName="payload-pre-processing" Nov 03 01:42:03.085427 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:42:03.085030 4939 cpu_manager.go:414] "containerMap: removing container" podUID="653b839f-b60a-466c-8b49-24aacea6dd75" containerName="payload-pre-processing" Nov 03 01:42:03.085427 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:42:03.085035 4939 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="653b839f-b60a-466c-8b49-24aacea6dd75" containerName="payload-pre-processing" Nov 03 01:42:03.085427 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:42:03.085046 4939 cpu_manager.go:414] "containerMap: removing container" podUID="fdeb6f74-c52a-4497-be8e-516b4244aefe" containerName="istio-proxy" Nov 03 01:42:03.085427 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:42:03.085050 4939 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fdeb6f74-c52a-4497-be8e-516b4244aefe" containerName="istio-proxy" Nov 03 01:42:03.085427 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:42:03.085058 4939 cpu_manager.go:414] "containerMap: removing container" podUID="a68659b3-912d-4df9-a162-fa6d5f949d2a" containerName="istio-proxy" Nov 03 01:42:03.085427 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:42:03.085062 4939 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a68659b3-912d-4df9-a162-fa6d5f949d2a" containerName="istio-proxy" Nov 03 01:42:03.085427 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:42:03.085071 4939 cpu_manager.go:414] "containerMap: removing container" podUID="01da212a-c344-4e25-9f26-ef2224e32b46" containerName="maas-api" Nov 03 01:42:03.085427 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:42:03.085076 4939 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="01da212a-c344-4e25-9f26-ef2224e32b46" containerName="maas-api" Nov 03 01:42:03.085427 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:42:03.085082 4939 cpu_manager.go:414] "containerMap: removing container" podUID="5d25771d-8931-4840-86be-9ebaf92fafca" containerName="payload-processing" Nov 03 01:42:03.085427 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:42:03.085086 4939 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5d25771d-8931-4840-86be-9ebaf92fafca" containerName="payload-processing" Nov 03 01:42:03.085427 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:42:03.085094 4939 cpu_manager.go:414] "containerMap: removing container" podUID="ccf63a28-a2d8-4686-8555-968e54429210" containerName="istio-proxy" Nov 03 01:42:03.085427 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:42:03.085098 4939 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ccf63a28-a2d8-4686-8555-968e54429210" containerName="istio-proxy" Nov 03 01:42:03.085427 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:42:03.085106 4939 cpu_manager.go:414] "containerMap: removing container" podUID="4bcdc6fd-f0da-46e3-bd48-14b1179bb3f1" containerName="maas-api" Nov 03 01:42:03.085427 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:42:03.085110 4939 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4bcdc6fd-f0da-46e3-bd48-14b1179bb3f1" containerName="maas-api" Nov 03 01:42:03.085427 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:42:03.085117 4939 cpu_manager.go:414] "containerMap: removing container" podUID="b7603676-9654-4c7f-ba1c-8a6a9c7e7382" containerName="istio-proxy" Nov 03 01:42:03.085427 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:42:03.085122 4939 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b7603676-9654-4c7f-ba1c-8a6a9c7e7382" containerName="istio-proxy" Nov 03 01:42:03.085427 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:42:03.085128 4939 cpu_manager.go:414] "containerMap: removing container" podUID="ba585f4b-4916-4d03-bf1e-454c029d46ad" containerName="istio-proxy" Nov 03 01:42:03.085427 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:42:03.085132 4939 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ba585f4b-4916-4d03-bf1e-454c029d46ad" containerName="istio-proxy" Nov 03 01:42:03.085427 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:42:03.085140 4939 cpu_manager.go:414] "containerMap: removing container" podUID="aa4094a2-6349-4637-9630-40a4ee74ecce" containerName="istio-proxy" Nov 03 01:42:03.085427 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:42:03.085145 4939 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="aa4094a2-6349-4637-9630-40a4ee74ecce" containerName="istio-proxy" Nov 03 01:42:03.085427 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:42:03.085151 4939 cpu_manager.go:414] "containerMap: removing container" podUID="d1ad622d-53f3-48ca-bc88-b7ab4ae452fe" containerName="istio-proxy" Nov 03 01:42:03.085427 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:42:03.085156 4939 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d1ad622d-53f3-48ca-bc88-b7ab4ae452fe" containerName="istio-proxy" Nov 03 01:42:03.085427 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:42:03.085165 4939 cpu_manager.go:414] "containerMap: removing container" podUID="19d08863-aa23-47f6-98cd-7908001a4458" containerName="payload-processing" Nov 03 01:42:03.085427 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:42:03.085169 4939 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="19d08863-aa23-47f6-98cd-7908001a4458" containerName="payload-processing" Nov 03 01:42:03.085427 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:42:03.085240 4939 memory_manager.go:365] "RemoveStaleState removing state" podUID="ef6478d3-bd62-45a7-8acc-1055c24dc53b" containerName="payload-processing" Nov 03 01:42:03.085427 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:42:03.085247 4939 memory_manager.go:365] "RemoveStaleState removing state" podUID="16b64414-6144-4743-bc96-91dffcd4a6b7" containerName="payload-processing" Nov 03 01:42:03.085427 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:42:03.085255 4939 memory_manager.go:365] "RemoveStaleState removing state" podUID="a79df098-23e4-4971-9263-3f52b7d25b0b" containerName="payload-pre-processing" Nov 03 01:42:03.085427 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:42:03.085263 4939 memory_manager.go:365] "RemoveStaleState removing state" podUID="836753fc-f9ef-4f61-b3fe-e5b92b7ee27b" containerName="main" Nov 03 01:42:03.085427 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:42:03.085268 4939 memory_manager.go:365] "RemoveStaleState removing state" podUID="4712e708-18a2-4a13-a12c-a5669397b2c3" containerName="main" Nov 03 01:42:03.085427 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:42:03.085273 4939 memory_manager.go:365] "RemoveStaleState removing state" podUID="3137d8da-32e8-47fd-87a6-f08aebe2a384" containerName="payload-pre-processing" Nov 03 01:42:03.085427 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:42:03.085280 4939 memory_manager.go:365] "RemoveStaleState removing state" podUID="6ea24c7b-6d13-4661-bdff-cd52792cb6b6" containerName="istio-proxy" Nov 03 01:42:03.085427 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:42:03.085286 4939 memory_manager.go:365] "RemoveStaleState removing state" podUID="285b736b-2a9f-40cd-96c4-3d7193d3eb8d" containerName="istio-proxy" Nov 03 01:42:03.085427 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:42:03.085291 4939 memory_manager.go:365] "RemoveStaleState removing state" podUID="a68659b3-912d-4df9-a162-fa6d5f949d2a" containerName="istio-proxy" Nov 03 01:42:03.085427 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:42:03.085296 4939 memory_manager.go:365] "RemoveStaleState removing state" podUID="78a38f10-6762-489e-813a-ac1cb91f5844" containerName="istio-proxy" Nov 03 01:42:03.085427 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:42:03.085302 4939 memory_manager.go:365] "RemoveStaleState removing state" podUID="78a38f10-6762-489e-813a-ac1cb91f5844" containerName="istio-proxy" Nov 03 01:42:03.085427 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:42:03.085308 4939 memory_manager.go:365] "RemoveStaleState removing state" podUID="c6de1910-0365-4f48-99f4-b32aebd747bb" containerName="payload-processing" Nov 03 01:42:03.085427 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:42:03.085313 4939 memory_manager.go:365] "RemoveStaleState removing state" podUID="6ea24c7b-6d13-4661-bdff-cd52792cb6b6" containerName="istio-proxy" Nov 03 01:42:03.085427 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:42:03.085318 4939 memory_manager.go:365] "RemoveStaleState removing state" podUID="fb776e00-2114-4f9c-86b1-bb790c3d64ca" containerName="istio-proxy" Nov 03 01:42:03.085427 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:42:03.085324 4939 memory_manager.go:365] "RemoveStaleState removing state" podUID="5b0cabc6-4960-4dc1-8317-3faebf25b30c" containerName="istio-proxy" Nov 03 01:42:03.085427 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:42:03.085328 4939 memory_manager.go:365] "RemoveStaleState removing state" podUID="5b48a10a-63a5-440b-b08d-0e3190a77997" containerName="main" Nov 03 01:42:03.085427 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:42:03.085333 4939 memory_manager.go:365] "RemoveStaleState removing state" podUID="2de18086-761b-4773-b9fc-15ddc4a220fc" containerName="main" Nov 03 01:42:03.085427 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:42:03.085338 4939 memory_manager.go:365] "RemoveStaleState removing state" podUID="61b34e6e-2690-4080-bced-3a577c665f12" containerName="payload-pre-processing" Nov 03 01:42:03.085427 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:42:03.085343 4939 memory_manager.go:365] "RemoveStaleState removing state" podUID="7c2434b1-8471-41fb-b20d-118aa8f1afc4" containerName="payload-processing" Nov 03 01:42:03.085427 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:42:03.085349 4939 memory_manager.go:365] "RemoveStaleState removing state" podUID="9a90ca63-d97d-4695-ad39-eaaa58204ecc" containerName="payload-pre-processing" Nov 03 01:42:03.085427 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:42:03.085354 4939 memory_manager.go:365] "RemoveStaleState removing state" podUID="d0933771-7032-4cff-aafe-2f7211ebb549" containerName="istio-proxy" Nov 03 01:42:03.085427 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:42:03.085359 4939 memory_manager.go:365] "RemoveStaleState removing state" podUID="ef71e635-99c1-4f1c-99b0-014934ba2f76" containerName="payload-pre-processing" Nov 03 01:42:03.085427 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:42:03.085364 4939 memory_manager.go:365] "RemoveStaleState removing state" podUID="653b839f-b60a-466c-8b49-24aacea6dd75" containerName="payload-pre-processing" Nov 03 01:42:03.085427 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:42:03.085370 4939 memory_manager.go:365] "RemoveStaleState removing state" podUID="fdeb6f74-c52a-4497-be8e-516b4244aefe" containerName="istio-proxy" Nov 03 01:42:03.085427 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:42:03.085375 4939 memory_manager.go:365] "RemoveStaleState removing state" podUID="a68659b3-912d-4df9-a162-fa6d5f949d2a" containerName="istio-proxy" Nov 03 01:42:03.085427 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:42:03.085380 4939 memory_manager.go:365] "RemoveStaleState removing state" podUID="01da212a-c344-4e25-9f26-ef2224e32b46" containerName="maas-api" Nov 03 01:42:03.085427 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:42:03.085386 4939 memory_manager.go:365] "RemoveStaleState removing state" podUID="5d25771d-8931-4840-86be-9ebaf92fafca" containerName="payload-processing" Nov 03 01:42:03.085427 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:42:03.085392 4939 memory_manager.go:365] "RemoveStaleState removing state" podUID="ccf63a28-a2d8-4686-8555-968e54429210" containerName="istio-proxy" Nov 03 01:42:03.085427 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:42:03.085396 4939 memory_manager.go:365] "RemoveStaleState removing state" podUID="4bcdc6fd-f0da-46e3-bd48-14b1179bb3f1" containerName="maas-api" Nov 03 01:42:03.085427 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:42:03.085403 4939 memory_manager.go:365] "RemoveStaleState removing state" podUID="5b0cabc6-4960-4dc1-8317-3faebf25b30c" containerName="istio-proxy" Nov 03 01:42:03.085427 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:42:03.085408 4939 memory_manager.go:365] "RemoveStaleState removing state" podUID="b7603676-9654-4c7f-ba1c-8a6a9c7e7382" containerName="istio-proxy" Nov 03 01:42:03.085427 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:42:03.085413 4939 memory_manager.go:365] "RemoveStaleState removing state" podUID="ba585f4b-4916-4d03-bf1e-454c029d46ad" containerName="istio-proxy" Nov 03 01:42:03.085427 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:42:03.085419 4939 memory_manager.go:365] "RemoveStaleState removing state" podUID="aa4094a2-6349-4637-9630-40a4ee74ecce" containerName="istio-proxy" Nov 03 01:42:03.085427 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:42:03.085424 4939 memory_manager.go:365] "RemoveStaleState removing state" podUID="d1ad622d-53f3-48ca-bc88-b7ab4ae452fe" containerName="istio-proxy" Nov 03 01:42:03.085427 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:42:03.085430 4939 memory_manager.go:365] "RemoveStaleState removing state" podUID="19d08863-aa23-47f6-98cd-7908001a4458" containerName="payload-processing" Nov 03 01:42:03.085427 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:42:03.085435 4939 memory_manager.go:365] "RemoveStaleState removing state" podUID="f2d0a2aa-d02a-4153-b8e8-997c8d1120fe" containerName="payload-processing" Nov 03 01:42:03.090251 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:42:03.090232 4939 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w1-2729f2-openshift-default-9fb97b4fc-6jhb8" Nov 03 01:42:03.098547 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:42:03.098106 4939 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-w1-2729f2-openshift-default-dockercfg-9nk2m\"" Nov 03 01:42:03.098898 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:42:03.098874 4939 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w1-2729f2-openshift-default-9fb97b4fc-6jhb8"] Nov 03 01:42:03.221680 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:42:03.221596 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/61c402c5-4efa-4821-b51e-da74f4d9eff7-istio-podinfo\") pod \"e2e-shared-a-w1-2729f2-openshift-default-9fb97b4fc-6jhb8\" (UID: \"61c402c5-4efa-4821-b51e-da74f4d9eff7\") " pod="openshift-ingress/e2e-shared-a-w1-2729f2-openshift-default-9fb97b4fc-6jhb8" Nov 03 01:42:03.221680 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:42:03.221644 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/61c402c5-4efa-4821-b51e-da74f4d9eff7-workload-socket\") pod \"e2e-shared-a-w1-2729f2-openshift-default-9fb97b4fc-6jhb8\" (UID: \"61c402c5-4efa-4821-b51e-da74f4d9eff7\") " pod="openshift-ingress/e2e-shared-a-w1-2729f2-openshift-default-9fb97b4fc-6jhb8" Nov 03 01:42:03.221860 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:42:03.221743 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/61c402c5-4efa-4821-b51e-da74f4d9eff7-istio-token\") pod \"e2e-shared-a-w1-2729f2-openshift-default-9fb97b4fc-6jhb8\" (UID: \"61c402c5-4efa-4821-b51e-da74f4d9eff7\") " pod="openshift-ingress/e2e-shared-a-w1-2729f2-openshift-default-9fb97b4fc-6jhb8" Nov 03 01:42:03.221860 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:42:03.221774 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/61c402c5-4efa-4821-b51e-da74f4d9eff7-credential-socket\") pod \"e2e-shared-a-w1-2729f2-openshift-default-9fb97b4fc-6jhb8\" (UID: \"61c402c5-4efa-4821-b51e-da74f4d9eff7\") " pod="openshift-ingress/e2e-shared-a-w1-2729f2-openshift-default-9fb97b4fc-6jhb8" Nov 03 01:42:03.221860 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:42:03.221818 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/61c402c5-4efa-4821-b51e-da74f4d9eff7-istiod-ca-cert\") pod \"e2e-shared-a-w1-2729f2-openshift-default-9fb97b4fc-6jhb8\" (UID: \"61c402c5-4efa-4821-b51e-da74f4d9eff7\") " pod="openshift-ingress/e2e-shared-a-w1-2729f2-openshift-default-9fb97b4fc-6jhb8" Nov 03 01:42:03.222042 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:42:03.221960 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qqhj4\" (UniqueName: \"kubernetes.io/projected/61c402c5-4efa-4821-b51e-da74f4d9eff7-kube-api-access-qqhj4\") pod \"e2e-shared-a-w1-2729f2-openshift-default-9fb97b4fc-6jhb8\" (UID: \"61c402c5-4efa-4821-b51e-da74f4d9eff7\") " pod="openshift-ingress/e2e-shared-a-w1-2729f2-openshift-default-9fb97b4fc-6jhb8" Nov 03 01:42:03.222042 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:42:03.221993 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/61c402c5-4efa-4821-b51e-da74f4d9eff7-workload-certs\") pod \"e2e-shared-a-w1-2729f2-openshift-default-9fb97b4fc-6jhb8\" (UID: \"61c402c5-4efa-4821-b51e-da74f4d9eff7\") " pod="openshift-ingress/e2e-shared-a-w1-2729f2-openshift-default-9fb97b4fc-6jhb8" Nov 03 01:42:03.222111 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:42:03.222081 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/61c402c5-4efa-4821-b51e-da74f4d9eff7-istio-data\") pod \"e2e-shared-a-w1-2729f2-openshift-default-9fb97b4fc-6jhb8\" (UID: \"61c402c5-4efa-4821-b51e-da74f4d9eff7\") " pod="openshift-ingress/e2e-shared-a-w1-2729f2-openshift-default-9fb97b4fc-6jhb8" Nov 03 01:42:03.222173 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:42:03.222160 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/61c402c5-4efa-4821-b51e-da74f4d9eff7-istio-envoy\") pod \"e2e-shared-a-w1-2729f2-openshift-default-9fb97b4fc-6jhb8\" (UID: \"61c402c5-4efa-4821-b51e-da74f4d9eff7\") " pod="openshift-ingress/e2e-shared-a-w1-2729f2-openshift-default-9fb97b4fc-6jhb8" Nov 03 01:42:03.323451 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:42:03.323411 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/61c402c5-4efa-4821-b51e-da74f4d9eff7-istio-token\") pod \"e2e-shared-a-w1-2729f2-openshift-default-9fb97b4fc-6jhb8\" (UID: \"61c402c5-4efa-4821-b51e-da74f4d9eff7\") " pod="openshift-ingress/e2e-shared-a-w1-2729f2-openshift-default-9fb97b4fc-6jhb8" Nov 03 01:42:03.323451 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:42:03.323457 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/61c402c5-4efa-4821-b51e-da74f4d9eff7-credential-socket\") pod \"e2e-shared-a-w1-2729f2-openshift-default-9fb97b4fc-6jhb8\" (UID: \"61c402c5-4efa-4821-b51e-da74f4d9eff7\") " pod="openshift-ingress/e2e-shared-a-w1-2729f2-openshift-default-9fb97b4fc-6jhb8" Nov 03 01:42:03.323671 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:42:03.323505 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/61c402c5-4efa-4821-b51e-da74f4d9eff7-istiod-ca-cert\") pod \"e2e-shared-a-w1-2729f2-openshift-default-9fb97b4fc-6jhb8\" (UID: \"61c402c5-4efa-4821-b51e-da74f4d9eff7\") " pod="openshift-ingress/e2e-shared-a-w1-2729f2-openshift-default-9fb97b4fc-6jhb8" Nov 03 01:42:03.323671 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:42:03.323551 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qqhj4\" (UniqueName: \"kubernetes.io/projected/61c402c5-4efa-4821-b51e-da74f4d9eff7-kube-api-access-qqhj4\") pod \"e2e-shared-a-w1-2729f2-openshift-default-9fb97b4fc-6jhb8\" (UID: \"61c402c5-4efa-4821-b51e-da74f4d9eff7\") " pod="openshift-ingress/e2e-shared-a-w1-2729f2-openshift-default-9fb97b4fc-6jhb8" Nov 03 01:42:03.323756 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:42:03.323674 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/61c402c5-4efa-4821-b51e-da74f4d9eff7-workload-certs\") pod \"e2e-shared-a-w1-2729f2-openshift-default-9fb97b4fc-6jhb8\" (UID: \"61c402c5-4efa-4821-b51e-da74f4d9eff7\") " pod="openshift-ingress/e2e-shared-a-w1-2729f2-openshift-default-9fb97b4fc-6jhb8" Nov 03 01:42:03.323796 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:42:03.323747 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/61c402c5-4efa-4821-b51e-da74f4d9eff7-istio-data\") pod \"e2e-shared-a-w1-2729f2-openshift-default-9fb97b4fc-6jhb8\" (UID: \"61c402c5-4efa-4821-b51e-da74f4d9eff7\") " pod="openshift-ingress/e2e-shared-a-w1-2729f2-openshift-default-9fb97b4fc-6jhb8" Nov 03 01:42:03.323889 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:42:03.323870 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/61c402c5-4efa-4821-b51e-da74f4d9eff7-istio-envoy\") pod \"e2e-shared-a-w1-2729f2-openshift-default-9fb97b4fc-6jhb8\" (UID: \"61c402c5-4efa-4821-b51e-da74f4d9eff7\") " pod="openshift-ingress/e2e-shared-a-w1-2729f2-openshift-default-9fb97b4fc-6jhb8" Nov 03 01:42:03.324065 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:42:03.323959 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/61c402c5-4efa-4821-b51e-da74f4d9eff7-istio-podinfo\") pod \"e2e-shared-a-w1-2729f2-openshift-default-9fb97b4fc-6jhb8\" (UID: \"61c402c5-4efa-4821-b51e-da74f4d9eff7\") " pod="openshift-ingress/e2e-shared-a-w1-2729f2-openshift-default-9fb97b4fc-6jhb8" Nov 03 01:42:03.324065 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:42:03.324001 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/61c402c5-4efa-4821-b51e-da74f4d9eff7-workload-socket\") pod \"e2e-shared-a-w1-2729f2-openshift-default-9fb97b4fc-6jhb8\" (UID: \"61c402c5-4efa-4821-b51e-da74f4d9eff7\") " pod="openshift-ingress/e2e-shared-a-w1-2729f2-openshift-default-9fb97b4fc-6jhb8" Nov 03 01:42:03.324224 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:42:03.324164 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/61c402c5-4efa-4821-b51e-da74f4d9eff7-workload-certs\") pod \"e2e-shared-a-w1-2729f2-openshift-default-9fb97b4fc-6jhb8\" (UID: \"61c402c5-4efa-4821-b51e-da74f4d9eff7\") " pod="openshift-ingress/e2e-shared-a-w1-2729f2-openshift-default-9fb97b4fc-6jhb8" Nov 03 01:42:03.324317 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:42:03.324294 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/61c402c5-4efa-4821-b51e-da74f4d9eff7-credential-socket\") pod \"e2e-shared-a-w1-2729f2-openshift-default-9fb97b4fc-6jhb8\" (UID: \"61c402c5-4efa-4821-b51e-da74f4d9eff7\") " pod="openshift-ingress/e2e-shared-a-w1-2729f2-openshift-default-9fb97b4fc-6jhb8" Nov 03 01:42:03.324366 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:42:03.324340 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/61c402c5-4efa-4821-b51e-da74f4d9eff7-istiod-ca-cert\") pod \"e2e-shared-a-w1-2729f2-openshift-default-9fb97b4fc-6jhb8\" (UID: \"61c402c5-4efa-4821-b51e-da74f4d9eff7\") " pod="openshift-ingress/e2e-shared-a-w1-2729f2-openshift-default-9fb97b4fc-6jhb8" Nov 03 01:42:03.324573 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:42:03.324552 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/61c402c5-4efa-4821-b51e-da74f4d9eff7-workload-socket\") pod \"e2e-shared-a-w1-2729f2-openshift-default-9fb97b4fc-6jhb8\" (UID: \"61c402c5-4efa-4821-b51e-da74f4d9eff7\") " pod="openshift-ingress/e2e-shared-a-w1-2729f2-openshift-default-9fb97b4fc-6jhb8" Nov 03 01:42:03.324676 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:42:03.324653 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/61c402c5-4efa-4821-b51e-da74f4d9eff7-istio-data\") pod \"e2e-shared-a-w1-2729f2-openshift-default-9fb97b4fc-6jhb8\" (UID: \"61c402c5-4efa-4821-b51e-da74f4d9eff7\") " pod="openshift-ingress/e2e-shared-a-w1-2729f2-openshift-default-9fb97b4fc-6jhb8" Nov 03 01:42:03.326473 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:42:03.326441 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/61c402c5-4efa-4821-b51e-da74f4d9eff7-istio-envoy\") pod \"e2e-shared-a-w1-2729f2-openshift-default-9fb97b4fc-6jhb8\" (UID: \"61c402c5-4efa-4821-b51e-da74f4d9eff7\") " pod="openshift-ingress/e2e-shared-a-w1-2729f2-openshift-default-9fb97b4fc-6jhb8" Nov 03 01:42:03.326689 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:42:03.326671 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/61c402c5-4efa-4821-b51e-da74f4d9eff7-istio-podinfo\") pod \"e2e-shared-a-w1-2729f2-openshift-default-9fb97b4fc-6jhb8\" (UID: \"61c402c5-4efa-4821-b51e-da74f4d9eff7\") " pod="openshift-ingress/e2e-shared-a-w1-2729f2-openshift-default-9fb97b4fc-6jhb8" Nov 03 01:42:03.330987 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:42:03.330960 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qqhj4\" (UniqueName: \"kubernetes.io/projected/61c402c5-4efa-4821-b51e-da74f4d9eff7-kube-api-access-qqhj4\") pod \"e2e-shared-a-w1-2729f2-openshift-default-9fb97b4fc-6jhb8\" (UID: \"61c402c5-4efa-4821-b51e-da74f4d9eff7\") " pod="openshift-ingress/e2e-shared-a-w1-2729f2-openshift-default-9fb97b4fc-6jhb8" Nov 03 01:42:03.331284 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:42:03.331265 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/61c402c5-4efa-4821-b51e-da74f4d9eff7-istio-token\") pod \"e2e-shared-a-w1-2729f2-openshift-default-9fb97b4fc-6jhb8\" (UID: \"61c402c5-4efa-4821-b51e-da74f4d9eff7\") " pod="openshift-ingress/e2e-shared-a-w1-2729f2-openshift-default-9fb97b4fc-6jhb8" Nov 03 01:42:03.338230 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:42:03.338210 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:42:03.405048 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:42:03.405016 4939 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w1-2729f2-openshift-default-9fb97b4fc-6jhb8" Nov 03 01:42:03.606874 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:42:03.606842 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:42:03.655797 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:42:03.655759 4939 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-556c14-669d5fb59f-cjtpf" podUID="192c8a1d-60d2-4928-9579-0231138280bb" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 03 01:42:03.688986 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:42:03.688960 4939 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w1-2729f2-openshift-default-9fb97b4fc-6jhb8"] Nov 03 01:42:03.691217 ip-10-0-140-107 kubenswrapper[4939]: W1103 01:42:03.691191 4939 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod61c402c5_4efa_4821_b51e_da74f4d9eff7.slice/crio-6e47f1b506e288518698d95572ed6bb934e02c5482862931a8ec32306cd70f91 WatchSource:0}: Error finding container 6e47f1b506e288518698d95572ed6bb934e02c5482862931a8ec32306cd70f91: Status 404 returned error can't find the container with id 6e47f1b506e288518698d95572ed6bb934e02c5482862931a8ec32306cd70f91 Nov 03 01:42:03.693641 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:42:03.693607 4939 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 01:42:03.693715 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:42:03.693676 4939 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 01:42:03.693715 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:42:03.693708 4939 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 01:42:03.870129 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:42:03.870045 4939 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ef6478d3-bd62-45a7-8acc-1055c24dc53b" path="/var/lib/kubelet/pods/ef6478d3-bd62-45a7-8acc-1055c24dc53b/volumes" Nov 03 01:42:03.870545 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:42:03.870523 4939 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ef71e635-99c1-4f1c-99b0-014934ba2f76" path="/var/lib/kubelet/pods/ef71e635-99c1-4f1c-99b0-014934ba2f76/volumes" Nov 03 01:42:04.389975 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:42:04.389939 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:42:04.628370 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:42:04.628325 4939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w1-2729f2-openshift-default-9fb97b4fc-6jhb8" event={"ID":"61c402c5-4efa-4821-b51e-da74f4d9eff7","Type":"ContainerStarted","Data":"26b79eddeeac2d4fc4ade04398e94fa28e6ddcc8d46013647c68074fbdad8d12"} Nov 03 01:42:04.628796 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:42:04.628396 4939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w1-2729f2-openshift-default-9fb97b4fc-6jhb8" event={"ID":"61c402c5-4efa-4821-b51e-da74f4d9eff7","Type":"ContainerStarted","Data":"6e47f1b506e288518698d95572ed6bb934e02c5482862931a8ec32306cd70f91"} Nov 03 01:42:04.661981 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:42:04.658794 4939 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w1-2729f2-openshift-default-9fb97b4fc-6jhb8" podStartSLOduration=1.6587738490000001 podStartE2EDuration="1.658773849s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 01:42:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 01:42:04.651526439 +0000 UTC m=+3672.915861386" watchObservedRunningTime="2026-11-03 01:42:04.658773849 +0000 UTC m=+3672.923108795" Nov 03 01:42:04.675334 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:42:04.674897 4939 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-556c14-566c7cc65c-lcbl4" podUID="23f02c28-7f9e-40d8-990c-aea9e85f1bf6" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 03 01:42:05.121402 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:42:05.121367 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:42:05.406379 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:42:05.405764 4939 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w1-2729f2-openshift-default-9fb97b4fc-6jhb8" Nov 03 01:42:05.408043 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:42:05.407534 4939 patch_prober.go:29] interesting pod/e2e-shared-a-w1-2729f2-openshift-default-9fb97b4fc-6jhb8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.87:15021/healthz/ready\": dial tcp 10.132.1.87:15021: connect: connection refused" start-of-body= Nov 03 01:42:05.408043 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:42:05.407596 4939 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w1-2729f2-openshift-default-9fb97b4fc-6jhb8" podUID="61c402c5-4efa-4821-b51e-da74f4d9eff7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.87:15021/healthz/ready\": dial tcp 10.132.1.87:15021: connect: connection refused" Nov 03 01:42:05.493951 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:42:05.493624 4939 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-556c14-566c7cc65c-lcbl4" Nov 03 01:42:05.498200 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:42:05.498166 4939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/23f02c28-7f9e-40d8-990c-aea9e85f1bf6-kube-api-access-v8np4\" (UniqueName: \"kubernetes.io/projected/23f02c28-7f9e-40d8-990c-aea9e85f1bf6-kube-api-access-v8np4\") pod \"23f02c28-7f9e-40d8-990c-aea9e85f1bf6\" (UID: \"23f02c28-7f9e-40d8-990c-aea9e85f1bf6\") " Nov 03 01:42:05.498362 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:42:05.498284 4939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/23f02c28-7f9e-40d8-990c-aea9e85f1bf6-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/23f02c28-7f9e-40d8-990c-aea9e85f1bf6-plugins-config-volume\") pod \"23f02c28-7f9e-40d8-990c-aea9e85f1bf6\" (UID: \"23f02c28-7f9e-40d8-990c-aea9e85f1bf6\") " Nov 03 01:42:05.499055 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:42:05.499028 4939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/23f02c28-7f9e-40d8-990c-aea9e85f1bf6-plugins-config-volume" pod "23f02c28-7f9e-40d8-990c-aea9e85f1bf6" (UID: "23f02c28-7f9e-40d8-990c-aea9e85f1bf6"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 01:42:05.500780 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:42:05.500749 4939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/23f02c28-7f9e-40d8-990c-aea9e85f1bf6-kube-api-access-v8np4" pod "23f02c28-7f9e-40d8-990c-aea9e85f1bf6" (UID: "23f02c28-7f9e-40d8-990c-aea9e85f1bf6"). InnerVolumeSpecName "kube-api-access-v8np4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 01:42:05.533067 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:42:05.533041 4939 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-556c14-669d5fb59f-cjtpf" Nov 03 01:42:05.599820 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:42:05.599781 4939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/192c8a1d-60d2-4928-9579-0231138280bb-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/192c8a1d-60d2-4928-9579-0231138280bb-plugins-config-volume\") pod \"192c8a1d-60d2-4928-9579-0231138280bb\" (UID: \"192c8a1d-60d2-4928-9579-0231138280bb\") " Nov 03 01:42:05.600040 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:42:05.599857 4939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/192c8a1d-60d2-4928-9579-0231138280bb-kube-api-access-ps7qn\" (UniqueName: \"kubernetes.io/projected/192c8a1d-60d2-4928-9579-0231138280bb-kube-api-access-ps7qn\") pod \"192c8a1d-60d2-4928-9579-0231138280bb\" (UID: \"192c8a1d-60d2-4928-9579-0231138280bb\") " Nov 03 01:42:05.600234 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:42:05.600209 4939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/192c8a1d-60d2-4928-9579-0231138280bb-plugins-config-volume" pod "192c8a1d-60d2-4928-9579-0231138280bb" (UID: "192c8a1d-60d2-4928-9579-0231138280bb"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 01:42:05.600354 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:42:05.600332 4939 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/23f02c28-7f9e-40d8-990c-aea9e85f1bf6-plugins-config-volume\") on node \"ip-10-0-140-107.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:42:05.600427 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:42:05.600360 4939 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/192c8a1d-60d2-4928-9579-0231138280bb-plugins-config-volume\") on node \"ip-10-0-140-107.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:42:05.600427 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:42:05.600375 4939 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v8np4\" (UniqueName: \"kubernetes.io/projected/23f02c28-7f9e-40d8-990c-aea9e85f1bf6-kube-api-access-v8np4\") on node \"ip-10-0-140-107.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:42:05.602213 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:42:05.602191 4939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/192c8a1d-60d2-4928-9579-0231138280bb-kube-api-access-ps7qn" pod "192c8a1d-60d2-4928-9579-0231138280bb" (UID: "192c8a1d-60d2-4928-9579-0231138280bb"). InnerVolumeSpecName "kube-api-access-ps7qn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 01:42:05.636260 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:42:05.636092 4939 generic.go:356] "Generic (PLEG): container finished" podID="23f02c28-7f9e-40d8-990c-aea9e85f1bf6" containerID="82722e009994f48eae235ec601132b70f0dee007f538c43936ef0e5e48c7ff34" exitCode=0 Nov 03 01:42:05.636260 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:42:05.636135 4939 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-556c14-566c7cc65c-lcbl4" Nov 03 01:42:05.636260 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:42:05.636170 4939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-556c14-566c7cc65c-lcbl4" event={"ID":"23f02c28-7f9e-40d8-990c-aea9e85f1bf6","Type":"ContainerDied","Data":"82722e009994f48eae235ec601132b70f0dee007f538c43936ef0e5e48c7ff34"} Nov 03 01:42:05.636260 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:42:05.636223 4939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-556c14-566c7cc65c-lcbl4" event={"ID":"23f02c28-7f9e-40d8-990c-aea9e85f1bf6","Type":"ContainerDied","Data":"44e45593589bba64cda93b2acbc4214851055f7fa19585c418bd10786111836b"} Nov 03 01:42:05.636260 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:42:05.636245 4939 scope.go:122] "RemoveContainer" containerID="82722e009994f48eae235ec601132b70f0dee007f538c43936ef0e5e48c7ff34" Nov 03 01:42:05.638408 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:42:05.638384 4939 generic.go:356] "Generic (PLEG): container finished" podID="192c8a1d-60d2-4928-9579-0231138280bb" containerID="f0face4ca56ac8b0d09a7aa6ea1c56ac5622db1ed5a8238106e45cc3a42061c2" exitCode=0 Nov 03 01:42:05.638504 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:42:05.638451 4939 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-556c14-669d5fb59f-cjtpf" Nov 03 01:42:05.638504 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:42:05.638481 4939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-556c14-669d5fb59f-cjtpf" event={"ID":"192c8a1d-60d2-4928-9579-0231138280bb","Type":"ContainerDied","Data":"f0face4ca56ac8b0d09a7aa6ea1c56ac5622db1ed5a8238106e45cc3a42061c2"} Nov 03 01:42:05.638640 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:42:05.638530 4939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-556c14-669d5fb59f-cjtpf" event={"ID":"192c8a1d-60d2-4928-9579-0231138280bb","Type":"ContainerDied","Data":"e0ab39b6a66829e96c05ad51f999acdb2212ad90364da3e376ea8f9d381d3b34"} Nov 03 01:42:05.677146 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:42:05.675188 4939 scope.go:122] "RemoveContainer" containerID="82722e009994f48eae235ec601132b70f0dee007f538c43936ef0e5e48c7ff34" Nov 03 01:42:05.677146 ip-10-0-140-107 kubenswrapper[4939]: E1103 01:42:05.675626 4939 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"82722e009994f48eae235ec601132b70f0dee007f538c43936ef0e5e48c7ff34\": container with ID starting with 82722e009994f48eae235ec601132b70f0dee007f538c43936ef0e5e48c7ff34 not found: ID does not exist" containerID="82722e009994f48eae235ec601132b70f0dee007f538c43936ef0e5e48c7ff34" Nov 03 01:42:05.677146 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:42:05.675669 4939 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"82722e009994f48eae235ec601132b70f0dee007f538c43936ef0e5e48c7ff34"} err="failed to get container status \"82722e009994f48eae235ec601132b70f0dee007f538c43936ef0e5e48c7ff34\": rpc error: code = NotFound desc = could not find container \"82722e009994f48eae235ec601132b70f0dee007f538c43936ef0e5e48c7ff34\": container with ID starting with 82722e009994f48eae235ec601132b70f0dee007f538c43936ef0e5e48c7ff34 not found: ID does not exist" Nov 03 01:42:05.677146 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:42:05.675696 4939 scope.go:122] "RemoveContainer" containerID="f0face4ca56ac8b0d09a7aa6ea1c56ac5622db1ed5a8238106e45cc3a42061c2" Nov 03 01:42:05.688145 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:42:05.688118 4939 scope.go:122] "RemoveContainer" containerID="f0face4ca56ac8b0d09a7aa6ea1c56ac5622db1ed5a8238106e45cc3a42061c2" Nov 03 01:42:05.689536 ip-10-0-140-107 kubenswrapper[4939]: E1103 01:42:05.689452 4939 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f0face4ca56ac8b0d09a7aa6ea1c56ac5622db1ed5a8238106e45cc3a42061c2\": container with ID starting with f0face4ca56ac8b0d09a7aa6ea1c56ac5622db1ed5a8238106e45cc3a42061c2 not found: ID does not exist" containerID="f0face4ca56ac8b0d09a7aa6ea1c56ac5622db1ed5a8238106e45cc3a42061c2" Nov 03 01:42:05.689536 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:42:05.689494 4939 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f0face4ca56ac8b0d09a7aa6ea1c56ac5622db1ed5a8238106e45cc3a42061c2"} err="failed to get container status \"f0face4ca56ac8b0d09a7aa6ea1c56ac5622db1ed5a8238106e45cc3a42061c2\": rpc error: code = NotFound desc = could not find container \"f0face4ca56ac8b0d09a7aa6ea1c56ac5622db1ed5a8238106e45cc3a42061c2\": container with ID starting with f0face4ca56ac8b0d09a7aa6ea1c56ac5622db1ed5a8238106e45cc3a42061c2 not found: ID does not exist" Nov 03 01:42:05.694216 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:42:05.694179 4939 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w3-556c14-566c7cc65c-lcbl4"] Nov 03 01:42:05.696780 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:42:05.696753 4939 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w3-556c14-566c7cc65c-lcbl4"] Nov 03 01:42:05.701498 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:42:05.701475 4939 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ps7qn\" (UniqueName: \"kubernetes.io/projected/192c8a1d-60d2-4928-9579-0231138280bb-kube-api-access-ps7qn\") on node \"ip-10-0-140-107.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:42:05.704994 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:42:05.704971 4939 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w3-556c14-669d5fb59f-cjtpf"] Nov 03 01:42:05.707214 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:42:05.707190 4939 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w3-556c14-669d5fb59f-cjtpf"] Nov 03 01:42:05.878656 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:42:05.878623 4939 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="192c8a1d-60d2-4928-9579-0231138280bb" path="/var/lib/kubelet/pods/192c8a1d-60d2-4928-9579-0231138280bb/volumes" Nov 03 01:42:05.879092 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:42:05.879076 4939 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="23f02c28-7f9e-40d8-990c-aea9e85f1bf6" path="/var/lib/kubelet/pods/23f02c28-7f9e-40d8-990c-aea9e85f1bf6/volumes" Nov 03 01:42:05.917821 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:42:05.917788 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:42:06.417831 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:42:06.414609 4939 patch_prober.go:29] interesting pod/e2e-shared-a-w1-2729f2-openshift-default-9fb97b4fc-6jhb8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.87:15021/healthz/ready\": dial tcp 10.132.1.87:15021: connect: connection refused" start-of-body= Nov 03 01:42:06.417831 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:42:06.414677 4939 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w1-2729f2-openshift-default-9fb97b4fc-6jhb8" podUID="61c402c5-4efa-4821-b51e-da74f4d9eff7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.87:15021/healthz/ready\": dial tcp 10.132.1.87:15021: connect: connection refused" Nov 03 01:42:06.646935 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:42:06.641242 4939 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-b-af8244-75d9474dd7-pr6cd"] Nov 03 01:42:06.721091 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:42:06.721003 4939 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-af8244-84f67c568c-m9kgf"] Nov 03 01:42:06.778158 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:42:06.762148 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:42:06.879787 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:42:06.879750 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:42:06.997034 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:42:06.996090 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:42:07.311209 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:42:07.311122 4939 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-worker-w4-650be0-86d95657b7-vm62r" podUID="58875207-cd7a-4da9-aee7-f538e7f44963" containerName="payload-processing" containerID="cri-o://368faaefacd433a21be99f2ad8a3867bc4ce409dac664ba52af1d6fd56e3ce4c" gracePeriod=25 Nov 03 01:42:07.324587 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:42:07.324541 4939 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-650be0-58fc9b6455-9jl7f" podUID="c72f9b11-1218-4c3f-be7c-6541feeb1ed2" containerName="payload-pre-processing" containerID="cri-o://4438a1bb491548b813040ebf4eeb229485ebbe25ba7dfe0c4027433520ddd46e" gracePeriod=25 Nov 03 01:42:07.405881 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:42:07.405498 4939 patch_prober.go:29] interesting pod/e2e-shared-a-w1-2729f2-openshift-default-9fb97b4fc-6jhb8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.87:15021/healthz/ready\": dial tcp 10.132.1.87:15021: connect: connection refused" start-of-body= Nov 03 01:42:07.405881 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:42:07.405550 4939 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w1-2729f2-openshift-default-9fb97b4fc-6jhb8" podUID="61c402c5-4efa-4821-b51e-da74f4d9eff7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.87:15021/healthz/ready\": dial tcp 10.132.1.87:15021: connect: connection refused" Nov 03 01:42:07.830367 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:42:07.830327 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:42:08.126950 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:42:08.123570 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:42:08.269145 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:42:08.269106 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:42:08.390350 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:42:08.390262 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:42:08.406200 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:42:08.406152 4939 patch_prober.go:29] interesting pod/e2e-shared-a-w1-2729f2-openshift-default-9fb97b4fc-6jhb8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.87:15021/healthz/ready\": dial tcp 10.132.1.87:15021: connect: connection refused" start-of-body= Nov 03 01:42:08.406390 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:42:08.406233 4939 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w1-2729f2-openshift-default-9fb97b4fc-6jhb8" podUID="61c402c5-4efa-4821-b51e-da74f4d9eff7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.87:15021/healthz/ready\": dial tcp 10.132.1.87:15021: connect: connection refused" Nov 03 01:42:08.500706 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:42:08.500667 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:42:09.096943 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:42:09.096887 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:42:09.405597 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:42:09.405510 4939 patch_prober.go:29] interesting pod/e2e-shared-a-w1-2729f2-openshift-default-9fb97b4fc-6jhb8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.87:15021/healthz/ready\": dial tcp 10.132.1.87:15021: connect: connection refused" start-of-body= Nov 03 01:42:09.405597 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:42:09.405564 4939 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w1-2729f2-openshift-default-9fb97b4fc-6jhb8" podUID="61c402c5-4efa-4821-b51e-da74f4d9eff7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.87:15021/healthz/ready\": dial tcp 10.132.1.87:15021: connect: connection refused" Nov 03 01:42:09.407812 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:42:09.407781 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:42:10.100954 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:42:10.100918 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:42:10.254362 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:42:10.254312 4939 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w3-83755a/distinct-2-w3-83755a-kserve-6cf66b995d-2l47b" podUID="c3b05e88-5d5b-4d03-b28d-832f22e3c868" containerName="main" containerID="cri-o://bdbe879f0d2fc0f251c3802a5e1e8b6dab2a240c43fe6064b6c071717ef02875" gracePeriod=45 Nov 03 01:42:10.410308 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:42:10.410275 4939 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w1-2729f2-openshift-default-9fb97b4fc-6jhb8" Nov 03 01:42:10.410696 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:42:10.410673 4939 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w1-2729f2-openshift-default-9fb97b4fc-6jhb8" Nov 03 01:42:10.411534 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:42:10.411516 4939 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w1-2729f2-openshift-default-9fb97b4fc-6jhb8" Nov 03 01:42:10.533160 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:42:10.533122 4939 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-83755a/distinct-2-w3-83755a-kserve-6cf66b995d-2l47b" Nov 03 01:42:10.609415 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:42:10.609380 4939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c3b05e88-5d5b-4d03-b28d-832f22e3c868-tls-certs\" (UniqueName: \"kubernetes.io/secret/c3b05e88-5d5b-4d03-b28d-832f22e3c868-tls-certs\") pod \"c3b05e88-5d5b-4d03-b28d-832f22e3c868\" (UID: \"c3b05e88-5d5b-4d03-b28d-832f22e3c868\") " Nov 03 01:42:10.609685 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:42:10.609439 4939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c3b05e88-5d5b-4d03-b28d-832f22e3c868-home\" (UniqueName: \"kubernetes.io/empty-dir/c3b05e88-5d5b-4d03-b28d-832f22e3c868-home\") pod \"c3b05e88-5d5b-4d03-b28d-832f22e3c868\" (UID: \"c3b05e88-5d5b-4d03-b28d-832f22e3c868\") " Nov 03 01:42:10.609685 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:42:10.609581 4939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c3b05e88-5d5b-4d03-b28d-832f22e3c868-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c3b05e88-5d5b-4d03-b28d-832f22e3c868-model-cache\") pod \"c3b05e88-5d5b-4d03-b28d-832f22e3c868\" (UID: \"c3b05e88-5d5b-4d03-b28d-832f22e3c868\") " Nov 03 01:42:10.609685 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:42:10.609649 4939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c3b05e88-5d5b-4d03-b28d-832f22e3c868-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c3b05e88-5d5b-4d03-b28d-832f22e3c868-tmp-dir\") pod \"c3b05e88-5d5b-4d03-b28d-832f22e3c868\" (UID: \"c3b05e88-5d5b-4d03-b28d-832f22e3c868\") " Nov 03 01:42:10.609877 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:42:10.609688 4939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c3b05e88-5d5b-4d03-b28d-832f22e3c868-kube-api-access-2s8z2\" (UniqueName: \"kubernetes.io/projected/c3b05e88-5d5b-4d03-b28d-832f22e3c868-kube-api-access-2s8z2\") pod \"c3b05e88-5d5b-4d03-b28d-832f22e3c868\" (UID: \"c3b05e88-5d5b-4d03-b28d-832f22e3c868\") " Nov 03 01:42:10.609877 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:42:10.609761 4939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c3b05e88-5d5b-4d03-b28d-832f22e3c868-dshm\" (UniqueName: \"kubernetes.io/empty-dir/c3b05e88-5d5b-4d03-b28d-832f22e3c868-dshm\") pod \"c3b05e88-5d5b-4d03-b28d-832f22e3c868\" (UID: \"c3b05e88-5d5b-4d03-b28d-832f22e3c868\") " Nov 03 01:42:10.610035 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:42:10.609870 4939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c3b05e88-5d5b-4d03-b28d-832f22e3c868-home" pod "c3b05e88-5d5b-4d03-b28d-832f22e3c868" (UID: "c3b05e88-5d5b-4d03-b28d-832f22e3c868"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 01:42:10.610122 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:42:10.610095 4939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c3b05e88-5d5b-4d03-b28d-832f22e3c868-model-cache" pod "c3b05e88-5d5b-4d03-b28d-832f22e3c868" (UID: "c3b05e88-5d5b-4d03-b28d-832f22e3c868"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 01:42:10.610255 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:42:10.610187 4939 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c3b05e88-5d5b-4d03-b28d-832f22e3c868-home\") on node \"ip-10-0-140-107.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:42:10.610598 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:42:10.610573 4939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c3b05e88-5d5b-4d03-b28d-832f22e3c868-tmp-dir" pod "c3b05e88-5d5b-4d03-b28d-832f22e3c868" (UID: "c3b05e88-5d5b-4d03-b28d-832f22e3c868"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 01:42:10.612557 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:42:10.612522 4939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c3b05e88-5d5b-4d03-b28d-832f22e3c868-kube-api-access-2s8z2" pod "c3b05e88-5d5b-4d03-b28d-832f22e3c868" (UID: "c3b05e88-5d5b-4d03-b28d-832f22e3c868"). InnerVolumeSpecName "kube-api-access-2s8z2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 01:42:10.612686 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:42:10.612597 4939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c3b05e88-5d5b-4d03-b28d-832f22e3c868-tls-certs" pod "c3b05e88-5d5b-4d03-b28d-832f22e3c868" (UID: "c3b05e88-5d5b-4d03-b28d-832f22e3c868"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 01:42:10.612686 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:42:10.612605 4939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c3b05e88-5d5b-4d03-b28d-832f22e3c868-dshm" pod "c3b05e88-5d5b-4d03-b28d-832f22e3c868" (UID: "c3b05e88-5d5b-4d03-b28d-832f22e3c868"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 01:42:10.697691 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:42:10.697656 4939 generic.go:356] "Generic (PLEG): container finished" podID="c3b05e88-5d5b-4d03-b28d-832f22e3c868" containerID="bdbe879f0d2fc0f251c3802a5e1e8b6dab2a240c43fe6064b6c071717ef02875" exitCode=0 Nov 03 01:42:10.697902 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:42:10.697709 4939 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-83755a/distinct-2-w3-83755a-kserve-6cf66b995d-2l47b" Nov 03 01:42:10.697902 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:42:10.697740 4939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-83755a/distinct-2-w3-83755a-kserve-6cf66b995d-2l47b" event={"ID":"c3b05e88-5d5b-4d03-b28d-832f22e3c868","Type":"ContainerDied","Data":"bdbe879f0d2fc0f251c3802a5e1e8b6dab2a240c43fe6064b6c071717ef02875"} Nov 03 01:42:10.697902 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:42:10.697788 4939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-83755a/distinct-2-w3-83755a-kserve-6cf66b995d-2l47b" event={"ID":"c3b05e88-5d5b-4d03-b28d-832f22e3c868","Type":"ContainerDied","Data":"d8d6c22ddb44f0dbd29527555b4ca12f953f6492cd1a9ce2ab50fe7e33f1ad03"} Nov 03 01:42:10.697902 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:42:10.697806 4939 scope.go:122] "RemoveContainer" containerID="bdbe879f0d2fc0f251c3802a5e1e8b6dab2a240c43fe6064b6c071717ef02875" Nov 03 01:42:10.708234 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:42:10.708217 4939 scope.go:122] "RemoveContainer" containerID="bdbe879f0d2fc0f251c3802a5e1e8b6dab2a240c43fe6064b6c071717ef02875" Nov 03 01:42:10.708563 ip-10-0-140-107 kubenswrapper[4939]: E1103 01:42:10.708544 4939 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bdbe879f0d2fc0f251c3802a5e1e8b6dab2a240c43fe6064b6c071717ef02875\": container with ID starting with bdbe879f0d2fc0f251c3802a5e1e8b6dab2a240c43fe6064b6c071717ef02875 not found: ID does not exist" containerID="bdbe879f0d2fc0f251c3802a5e1e8b6dab2a240c43fe6064b6c071717ef02875" Nov 03 01:42:10.708614 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:42:10.708571 4939 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bdbe879f0d2fc0f251c3802a5e1e8b6dab2a240c43fe6064b6c071717ef02875"} err="failed to get container status \"bdbe879f0d2fc0f251c3802a5e1e8b6dab2a240c43fe6064b6c071717ef02875\": rpc error: code = NotFound desc = could not find container \"bdbe879f0d2fc0f251c3802a5e1e8b6dab2a240c43fe6064b6c071717ef02875\": container with ID starting with bdbe879f0d2fc0f251c3802a5e1e8b6dab2a240c43fe6064b6c071717ef02875 not found: ID does not exist" Nov 03 01:42:10.711328 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:42:10.711300 4939 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c3b05e88-5d5b-4d03-b28d-832f22e3c868-model-cache\") on node \"ip-10-0-140-107.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:42:10.711472 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:42:10.711400 4939 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c3b05e88-5d5b-4d03-b28d-832f22e3c868-tmp-dir\") on node \"ip-10-0-140-107.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:42:10.711472 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:42:10.711415 4939 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2s8z2\" (UniqueName: \"kubernetes.io/projected/c3b05e88-5d5b-4d03-b28d-832f22e3c868-kube-api-access-2s8z2\") on node \"ip-10-0-140-107.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:42:10.711472 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:42:10.711427 4939 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c3b05e88-5d5b-4d03-b28d-832f22e3c868-dshm\") on node \"ip-10-0-140-107.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:42:10.711472 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:42:10.711440 4939 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c3b05e88-5d5b-4d03-b28d-832f22e3c868-tls-certs\") on node \"ip-10-0-140-107.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:42:10.721256 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:42:10.721234 4939 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w3-83755a/distinct-2-w3-83755a-kserve-6cf66b995d-2l47b"] Nov 03 01:42:10.724825 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:42:10.724802 4939 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w3-83755a/distinct-2-w3-83755a-kserve-6cf66b995d-2l47b"] Nov 03 01:42:10.798070 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:42:10.798039 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:42:11.477306 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:42:11.477272 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:42:11.641534 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:42:11.641495 4939 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-extmod-b-af8244-75d9474dd7-pr6cd" podUID="d1f73b7c-084a-44b7-a3bd-c5042532a732" containerName="payload-processing" containerID="cri-o://4eccef524b9821502c808736a0d11ae8bd8a6add2167c878b11568b82991675f" gracePeriod=25 Nov 03 01:42:11.721604 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:42:11.721568 4939 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-af8244-84f67c568c-m9kgf" podUID="1f0db3da-dcf7-459b-994e-a7081e6f0958" containerName="payload-pre-processing" containerID="cri-o://22f578bc64e5a78ccb195588fcf509b901b156969faf20a928fc0d92ff8e06fb" gracePeriod=25 Nov 03 01:42:11.868413 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:42:11.868377 4939 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c3b05e88-5d5b-4d03-b28d-832f22e3c868" path="/var/lib/kubelet/pods/c3b05e88-5d5b-4d03-b28d-832f22e3c868/volumes" Nov 03 01:42:12.199724 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:42:12.199641 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:42:12.887466 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:42:12.887431 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:42:13.581223 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:42:13.581192 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:42:13.679427 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:42:13.679381 4939 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w4-650be0-86d95657b7-vm62r" podUID="58875207-cd7a-4da9-aee7-f538e7f44963" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 03 01:42:13.679604 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:42:13.679443 4939 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-650be0-58fc9b6455-9jl7f" podUID="c72f9b11-1218-4c3f-be7c-6541feeb1ed2" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 03 01:42:14.356271 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:42:14.356233 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:42:14.469117 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:42:14.469086 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:42:14.713960 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:42:14.713874 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:42:14.828879 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:42:14.828844 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:42:15.086576 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:42:15.086546 4939 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-b-af8244-openshift-default-594d5d565c-4f5nx"] Nov 03 01:42:15.086784 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:42:15.086758 4939 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-extmod-b-af8244-openshift-default-594d5d565c-4f5nx" podUID="efde608b-c334-4d17-a0aa-cad98a95a999" containerName="istio-proxy" containerID="cri-o://e0519bcdbf5cb34499c8b3aff5536f3c55998f633bc17fcd25b7cb0f97269e8c" gracePeriod=30 Nov 03 01:42:15.145751 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:42:15.145717 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:42:15.299674 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:42:15.299641 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:42:15.515076 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:42:15.515000 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:42:15.901127 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:42:15.901088 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:42:16.012485 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:42:16.012361 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:42:16.585108 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:42:16.585069 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:42:17.110734 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:42:17.110693 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:42:17.396567 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:42:17.396480 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:42:17.681981 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:42:17.681866 4939 patch_prober.go:29] interesting pod/e2e-extmod-b-af8244-openshift-default-594d5d565c-4f5nx container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 03 01:42:17.681981 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:42:17.681962 4939 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-extmod-b-af8244-openshift-default-594d5d565c-4f5nx" podUID="efde608b-c334-4d17-a0aa-cad98a95a999" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 03 01:42:17.981670 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:42:17.981572 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:42:18.042245 ip-10-0-140-107 kubenswrapper[4939]: E1103 01:42:18.042206 4939 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-sthjh" containerName="extractor" Nov 03 01:42:18.111186 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:42:18.111146 4939 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-4pw7v"] Nov 03 01:42:18.344394 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:42:18.344357 4939 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-af8244-84f67c568c-m9kgf" podUID="1f0db3da-dcf7-459b-994e-a7081e6f0958" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 03 01:42:18.814400 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:42:18.814306 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:42:19.378809 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:42:19.378768 4939 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-extmod-b-af8244-75d9474dd7-pr6cd" podUID="d1f73b7c-084a-44b7-a3bd-c5042532a732" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 03 01:42:19.596069 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:42:19.596037 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:42:20.358678 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:42:20.358617 4939 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-af8244-openshift-default-594d5d565c-4f5nx" Nov 03 01:42:20.406072 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:42:20.406036 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:42:20.424708 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:42:20.424672 4939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/efde608b-c334-4d17-a0aa-cad98a95a999-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/efde608b-c334-4d17-a0aa-cad98a95a999-istio-data\") pod \"efde608b-c334-4d17-a0aa-cad98a95a999\" (UID: \"efde608b-c334-4d17-a0aa-cad98a95a999\") " Nov 03 01:42:20.424865 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:42:20.424738 4939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/efde608b-c334-4d17-a0aa-cad98a95a999-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/efde608b-c334-4d17-a0aa-cad98a95a999-istiod-ca-cert\") pod \"efde608b-c334-4d17-a0aa-cad98a95a999\" (UID: \"efde608b-c334-4d17-a0aa-cad98a95a999\") " Nov 03 01:42:20.424865 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:42:20.424797 4939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/efde608b-c334-4d17-a0aa-cad98a95a999-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/efde608b-c334-4d17-a0aa-cad98a95a999-workload-socket\") pod \"efde608b-c334-4d17-a0aa-cad98a95a999\" (UID: \"efde608b-c334-4d17-a0aa-cad98a95a999\") " Nov 03 01:42:20.424865 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:42:20.424850 4939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/efde608b-c334-4d17-a0aa-cad98a95a999-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/efde608b-c334-4d17-a0aa-cad98a95a999-istio-podinfo\") pod \"efde608b-c334-4d17-a0aa-cad98a95a999\" (UID: \"efde608b-c334-4d17-a0aa-cad98a95a999\") " Nov 03 01:42:20.425060 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:42:20.424930 4939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/efde608b-c334-4d17-a0aa-cad98a95a999-kube-api-access-fn4f9\" (UniqueName: \"kubernetes.io/projected/efde608b-c334-4d17-a0aa-cad98a95a999-kube-api-access-fn4f9\") pod \"efde608b-c334-4d17-a0aa-cad98a95a999\" (UID: \"efde608b-c334-4d17-a0aa-cad98a95a999\") " Nov 03 01:42:20.425060 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:42:20.425007 4939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/efde608b-c334-4d17-a0aa-cad98a95a999-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/efde608b-c334-4d17-a0aa-cad98a95a999-credential-socket\") pod \"efde608b-c334-4d17-a0aa-cad98a95a999\" (UID: \"efde608b-c334-4d17-a0aa-cad98a95a999\") " Nov 03 01:42:20.425060 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:42:20.425038 4939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/efde608b-c334-4d17-a0aa-cad98a95a999-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/efde608b-c334-4d17-a0aa-cad98a95a999-istio-envoy\") pod \"efde608b-c334-4d17-a0aa-cad98a95a999\" (UID: \"efde608b-c334-4d17-a0aa-cad98a95a999\") " Nov 03 01:42:20.425225 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:42:20.425088 4939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/efde608b-c334-4d17-a0aa-cad98a95a999-istio-token\" (UniqueName: \"kubernetes.io/projected/efde608b-c334-4d17-a0aa-cad98a95a999-istio-token\") pod \"efde608b-c334-4d17-a0aa-cad98a95a999\" (UID: \"efde608b-c334-4d17-a0aa-cad98a95a999\") " Nov 03 01:42:20.425225 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:42:20.425123 4939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/efde608b-c334-4d17-a0aa-cad98a95a999-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/efde608b-c334-4d17-a0aa-cad98a95a999-workload-certs\") pod \"efde608b-c334-4d17-a0aa-cad98a95a999\" (UID: \"efde608b-c334-4d17-a0aa-cad98a95a999\") " Nov 03 01:42:20.425225 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:42:20.425168 4939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/efde608b-c334-4d17-a0aa-cad98a95a999-istio-data" pod "efde608b-c334-4d17-a0aa-cad98a95a999" (UID: "efde608b-c334-4d17-a0aa-cad98a95a999"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 01:42:20.431679 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:42:20.425694 4939 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/efde608b-c334-4d17-a0aa-cad98a95a999-istio-data\") on node \"ip-10-0-140-107.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:42:20.431902 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:42:20.431812 4939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/efde608b-c334-4d17-a0aa-cad98a95a999-workload-certs" pod "efde608b-c334-4d17-a0aa-cad98a95a999" (UID: "efde608b-c334-4d17-a0aa-cad98a95a999"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 01:42:20.432287 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:42:20.432262 4939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/efde608b-c334-4d17-a0aa-cad98a95a999-credential-socket" pod "efde608b-c334-4d17-a0aa-cad98a95a999" (UID: "efde608b-c334-4d17-a0aa-cad98a95a999"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 01:42:20.433471 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:42:20.433439 4939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/efde608b-c334-4d17-a0aa-cad98a95a999-workload-socket" pod "efde608b-c334-4d17-a0aa-cad98a95a999" (UID: "efde608b-c334-4d17-a0aa-cad98a95a999"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 01:42:20.434435 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:42:20.434396 4939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/efde608b-c334-4d17-a0aa-cad98a95a999-istiod-ca-cert" pod "efde608b-c334-4d17-a0aa-cad98a95a999" (UID: "efde608b-c334-4d17-a0aa-cad98a95a999"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 01:42:20.435888 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:42:20.435841 4939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/efde608b-c334-4d17-a0aa-cad98a95a999-istio-envoy" pod "efde608b-c334-4d17-a0aa-cad98a95a999" (UID: "efde608b-c334-4d17-a0aa-cad98a95a999"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 01:42:20.435888 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:42:20.435845 4939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/efde608b-c334-4d17-a0aa-cad98a95a999-istio-token" pod "efde608b-c334-4d17-a0aa-cad98a95a999" (UID: "efde608b-c334-4d17-a0aa-cad98a95a999"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 01:42:20.436724 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:42:20.436685 4939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/efde608b-c334-4d17-a0aa-cad98a95a999-kube-api-access-fn4f9" pod "efde608b-c334-4d17-a0aa-cad98a95a999" (UID: "efde608b-c334-4d17-a0aa-cad98a95a999"). InnerVolumeSpecName "kube-api-access-fn4f9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 01:42:20.443877 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:42:20.443738 4939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/efde608b-c334-4d17-a0aa-cad98a95a999-istio-podinfo" pod "efde608b-c334-4d17-a0aa-cad98a95a999" (UID: "efde608b-c334-4d17-a0aa-cad98a95a999"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 01:42:20.526836 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:42:20.526804 4939 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/efde608b-c334-4d17-a0aa-cad98a95a999-istiod-ca-cert\") on node \"ip-10-0-140-107.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:42:20.526836 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:42:20.526836 4939 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/efde608b-c334-4d17-a0aa-cad98a95a999-workload-socket\") on node \"ip-10-0-140-107.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:42:20.527031 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:42:20.526845 4939 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/efde608b-c334-4d17-a0aa-cad98a95a999-istio-podinfo\") on node \"ip-10-0-140-107.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:42:20.527031 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:42:20.526855 4939 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fn4f9\" (UniqueName: \"kubernetes.io/projected/efde608b-c334-4d17-a0aa-cad98a95a999-kube-api-access-fn4f9\") on node \"ip-10-0-140-107.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:42:20.527031 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:42:20.526864 4939 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/efde608b-c334-4d17-a0aa-cad98a95a999-credential-socket\") on node \"ip-10-0-140-107.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:42:20.527031 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:42:20.526873 4939 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/efde608b-c334-4d17-a0aa-cad98a95a999-istio-envoy\") on node \"ip-10-0-140-107.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:42:20.527031 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:42:20.526882 4939 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/efde608b-c334-4d17-a0aa-cad98a95a999-istio-token\") on node \"ip-10-0-140-107.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:42:20.527031 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:42:20.526891 4939 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/efde608b-c334-4d17-a0aa-cad98a95a999-workload-certs\") on node \"ip-10-0-140-107.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:42:20.757045 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:42:20.756948 4939 generic.go:356] "Generic (PLEG): container finished" podID="efde608b-c334-4d17-a0aa-cad98a95a999" containerID="e0519bcdbf5cb34499c8b3aff5536f3c55998f633bc17fcd25b7cb0f97269e8c" exitCode=0 Nov 03 01:42:20.757045 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:42:20.757003 4939 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-af8244-openshift-default-594d5d565c-4f5nx" Nov 03 01:42:20.757318 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:42:20.757031 4939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-af8244-openshift-default-594d5d565c-4f5nx" event={"ID":"efde608b-c334-4d17-a0aa-cad98a95a999","Type":"ContainerDied","Data":"e0519bcdbf5cb34499c8b3aff5536f3c55998f633bc17fcd25b7cb0f97269e8c"} Nov 03 01:42:20.757318 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:42:20.757132 4939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-af8244-openshift-default-594d5d565c-4f5nx" event={"ID":"efde608b-c334-4d17-a0aa-cad98a95a999","Type":"ContainerDied","Data":"2b68b0c8553e06e22edc362642aa3f346dd654dd1a1452e7424336802726c284"} Nov 03 01:42:20.757318 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:42:20.757166 4939 scope.go:122] "RemoveContainer" containerID="e0519bcdbf5cb34499c8b3aff5536f3c55998f633bc17fcd25b7cb0f97269e8c" Nov 03 01:42:20.767739 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:42:20.767723 4939 scope.go:122] "RemoveContainer" containerID="be600a011fd4e99c6ae1e3f5111b289f859d31f27a7639165b30f43a4bbe3776" Nov 03 01:42:20.777079 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:42:20.777058 4939 scope.go:122] "RemoveContainer" containerID="e0519bcdbf5cb34499c8b3aff5536f3c55998f633bc17fcd25b7cb0f97269e8c" Nov 03 01:42:20.777356 ip-10-0-140-107 kubenswrapper[4939]: E1103 01:42:20.777333 4939 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e0519bcdbf5cb34499c8b3aff5536f3c55998f633bc17fcd25b7cb0f97269e8c\": container with ID starting with e0519bcdbf5cb34499c8b3aff5536f3c55998f633bc17fcd25b7cb0f97269e8c not found: ID does not exist" containerID="e0519bcdbf5cb34499c8b3aff5536f3c55998f633bc17fcd25b7cb0f97269e8c" Nov 03 01:42:20.777411 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:42:20.777364 4939 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e0519bcdbf5cb34499c8b3aff5536f3c55998f633bc17fcd25b7cb0f97269e8c"} err="failed to get container status \"e0519bcdbf5cb34499c8b3aff5536f3c55998f633bc17fcd25b7cb0f97269e8c\": rpc error: code = NotFound desc = could not find container \"e0519bcdbf5cb34499c8b3aff5536f3c55998f633bc17fcd25b7cb0f97269e8c\": container with ID starting with e0519bcdbf5cb34499c8b3aff5536f3c55998f633bc17fcd25b7cb0f97269e8c not found: ID does not exist" Nov 03 01:42:20.777411 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:42:20.777381 4939 scope.go:122] "RemoveContainer" containerID="be600a011fd4e99c6ae1e3f5111b289f859d31f27a7639165b30f43a4bbe3776" Nov 03 01:42:20.777613 ip-10-0-140-107 kubenswrapper[4939]: E1103 01:42:20.777595 4939 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"be600a011fd4e99c6ae1e3f5111b289f859d31f27a7639165b30f43a4bbe3776\": container with ID starting with be600a011fd4e99c6ae1e3f5111b289f859d31f27a7639165b30f43a4bbe3776 not found: ID does not exist" containerID="be600a011fd4e99c6ae1e3f5111b289f859d31f27a7639165b30f43a4bbe3776" Nov 03 01:42:20.777661 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:42:20.777622 4939 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"be600a011fd4e99c6ae1e3f5111b289f859d31f27a7639165b30f43a4bbe3776"} err="failed to get container status \"be600a011fd4e99c6ae1e3f5111b289f859d31f27a7639165b30f43a4bbe3776\": rpc error: code = NotFound desc = could not find container \"be600a011fd4e99c6ae1e3f5111b289f859d31f27a7639165b30f43a4bbe3776\": container with ID starting with be600a011fd4e99c6ae1e3f5111b289f859d31f27a7639165b30f43a4bbe3776 not found: ID does not exist" Nov 03 01:42:20.780366 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:42:20.780345 4939 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-b-af8244-openshift-default-594d5d565c-4f5nx"] Nov 03 01:42:20.784305 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:42:20.784284 4939 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-extmod-b-af8244-openshift-default-594d5d565c-4f5nx"] Nov 03 01:42:21.195027 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:42:21.194994 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:42:21.395133 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:42:21.395091 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:42:21.867880 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:42:21.867846 4939 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="efde608b-c334-4d17-a0aa-cad98a95a999" path="/var/lib/kubelet/pods/efde608b-c334-4d17-a0aa-cad98a95a999/volumes" Nov 03 01:42:22.025988 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:42:22.025957 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:42:22.087610 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:42:22.087580 4939 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w1-2729f2-565847c555-wqvj7"] Nov 03 01:42:22.097014 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:42:22.096985 4939 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w1-2729f2-558f777dc-9gclz"] Nov 03 01:42:22.186196 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:42:22.186102 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:42:22.298720 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:42:22.298687 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:42:22.431946 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:42:22.431509 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:42:22.638098 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:42:22.638046 4939 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-2729f2-6f5b74c6cf-h9g6h"] Nov 03 01:42:22.638328 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:42:22.638301 4939 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-2729f2-6f5b74c6cf-h9g6h" podUID="6ec3abd1-a454-4860-8c1b-d70c26b14b3a" containerName="maas-api" containerID="cri-o://eff241f8fefc5e2ba6484d635be2a30a46e1089a49c2b897e7b35f8fa90a0b04" gracePeriod=30 Nov 03 01:42:22.770619 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:42:22.770587 4939 generic.go:356] "Generic (PLEG): container finished" podID="6ec3abd1-a454-4860-8c1b-d70c26b14b3a" containerID="eff241f8fefc5e2ba6484d635be2a30a46e1089a49c2b897e7b35f8fa90a0b04" exitCode=0 Nov 03 01:42:22.770777 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:42:22.770656 4939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-2729f2-6f5b74c6cf-h9g6h" event={"ID":"6ec3abd1-a454-4860-8c1b-d70c26b14b3a","Type":"ContainerDied","Data":"eff241f8fefc5e2ba6484d635be2a30a46e1089a49c2b897e7b35f8fa90a0b04"} Nov 03 01:42:22.816463 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:42:22.816420 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:42:22.856946 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:42:22.856898 4939 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w3-83755a-84f6d788fc-bgk7t"] Nov 03 01:42:22.857134 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:42:22.857101 4939 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-83755a-84f6d788fc-bgk7t" podUID="68597924-b259-47a2-b006-0d94dcb6315f" containerName="maas-api" containerID="cri-o://c47b18d53a2f3cfa9a35572473600e99d0a508ff19b12d7209d16df6fced9a52" gracePeriod=30 Nov 03 01:42:22.958832 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:42:22.958788 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:42:22.987134 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:42:22.987109 4939 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-2729f2-6f5b74c6cf-h9g6h" Nov 03 01:42:23.066222 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:42:23.066182 4939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6ec3abd1-a454-4860-8c1b-d70c26b14b3a-metrics-certs\" (UniqueName: \"kubernetes.io/secret/6ec3abd1-a454-4860-8c1b-d70c26b14b3a-metrics-certs\") pod \"6ec3abd1-a454-4860-8c1b-d70c26b14b3a\" (UID: \"6ec3abd1-a454-4860-8c1b-d70c26b14b3a\") " Nov 03 01:42:23.066456 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:42:23.066294 4939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6ec3abd1-a454-4860-8c1b-d70c26b14b3a-kube-api-access-hfbfw\" (UniqueName: \"kubernetes.io/projected/6ec3abd1-a454-4860-8c1b-d70c26b14b3a-kube-api-access-hfbfw\") pod \"6ec3abd1-a454-4860-8c1b-d70c26b14b3a\" (UID: \"6ec3abd1-a454-4860-8c1b-d70c26b14b3a\") " Nov 03 01:42:23.066456 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:42:23.066341 4939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6ec3abd1-a454-4860-8c1b-d70c26b14b3a-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6ec3abd1-a454-4860-8c1b-d70c26b14b3a-maas-api-tls\") pod \"6ec3abd1-a454-4860-8c1b-d70c26b14b3a\" (UID: \"6ec3abd1-a454-4860-8c1b-d70c26b14b3a\") " Nov 03 01:42:23.066456 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:42:23.066381 4939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6ec3abd1-a454-4860-8c1b-d70c26b14b3a-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6ec3abd1-a454-4860-8c1b-d70c26b14b3a-openshift-service-ca\") pod \"6ec3abd1-a454-4860-8c1b-d70c26b14b3a\" (UID: \"6ec3abd1-a454-4860-8c1b-d70c26b14b3a\") " Nov 03 01:42:23.068437 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:42:23.068388 4939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6ec3abd1-a454-4860-8c1b-d70c26b14b3a-openshift-service-ca" pod "6ec3abd1-a454-4860-8c1b-d70c26b14b3a" (UID: "6ec3abd1-a454-4860-8c1b-d70c26b14b3a"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 01:42:23.069528 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:42:23.069488 4939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6ec3abd1-a454-4860-8c1b-d70c26b14b3a-kube-api-access-hfbfw" pod "6ec3abd1-a454-4860-8c1b-d70c26b14b3a" (UID: "6ec3abd1-a454-4860-8c1b-d70c26b14b3a"). InnerVolumeSpecName "kube-api-access-hfbfw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 01:42:23.069790 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:42:23.069751 4939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6ec3abd1-a454-4860-8c1b-d70c26b14b3a-metrics-certs" pod "6ec3abd1-a454-4860-8c1b-d70c26b14b3a" (UID: "6ec3abd1-a454-4860-8c1b-d70c26b14b3a"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 01:42:23.070810 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:42:23.070782 4939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6ec3abd1-a454-4860-8c1b-d70c26b14b3a-maas-api-tls" pod "6ec3abd1-a454-4860-8c1b-d70c26b14b3a" (UID: "6ec3abd1-a454-4860-8c1b-d70c26b14b3a"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 01:42:23.123521 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:42:23.123494 4939 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-83755a-84f6d788fc-bgk7t" Nov 03 01:42:23.169107 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:42:23.169008 4939 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6ec3abd1-a454-4860-8c1b-d70c26b14b3a-metrics-certs\") on node \"ip-10-0-140-107.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:42:23.169107 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:42:23.169050 4939 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hfbfw\" (UniqueName: \"kubernetes.io/projected/6ec3abd1-a454-4860-8c1b-d70c26b14b3a-kube-api-access-hfbfw\") on node \"ip-10-0-140-107.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:42:23.169107 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:42:23.169068 4939 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6ec3abd1-a454-4860-8c1b-d70c26b14b3a-maas-api-tls\") on node \"ip-10-0-140-107.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:42:23.169107 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:42:23.169080 4939 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6ec3abd1-a454-4860-8c1b-d70c26b14b3a-openshift-service-ca\") on node \"ip-10-0-140-107.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:42:23.176447 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:42:23.176361 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:42:23.270258 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:42:23.270167 4939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/68597924-b259-47a2-b006-0d94dcb6315f-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/68597924-b259-47a2-b006-0d94dcb6315f-maas-api-tls\") pod \"68597924-b259-47a2-b006-0d94dcb6315f\" (UID: \"68597924-b259-47a2-b006-0d94dcb6315f\") " Nov 03 01:42:23.270424 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:42:23.270357 4939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/68597924-b259-47a2-b006-0d94dcb6315f-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/68597924-b259-47a2-b006-0d94dcb6315f-openshift-service-ca\") pod \"68597924-b259-47a2-b006-0d94dcb6315f\" (UID: \"68597924-b259-47a2-b006-0d94dcb6315f\") " Nov 03 01:42:23.270424 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:42:23.270382 4939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/68597924-b259-47a2-b006-0d94dcb6315f-kube-api-access-z5jrq\" (UniqueName: \"kubernetes.io/projected/68597924-b259-47a2-b006-0d94dcb6315f-kube-api-access-z5jrq\") pod \"68597924-b259-47a2-b006-0d94dcb6315f\" (UID: \"68597924-b259-47a2-b006-0d94dcb6315f\") " Nov 03 01:42:23.270424 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:42:23.270403 4939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/68597924-b259-47a2-b006-0d94dcb6315f-metrics-certs\" (UniqueName: \"kubernetes.io/secret/68597924-b259-47a2-b006-0d94dcb6315f-metrics-certs\") pod \"68597924-b259-47a2-b006-0d94dcb6315f\" (UID: \"68597924-b259-47a2-b006-0d94dcb6315f\") " Nov 03 01:42:23.270781 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:42:23.270736 4939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/68597924-b259-47a2-b006-0d94dcb6315f-openshift-service-ca" pod "68597924-b259-47a2-b006-0d94dcb6315f" (UID: "68597924-b259-47a2-b006-0d94dcb6315f"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 01:42:23.270938 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:42:23.270827 4939 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/68597924-b259-47a2-b006-0d94dcb6315f-openshift-service-ca\") on node \"ip-10-0-140-107.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:42:23.272454 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:42:23.272425 4939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/68597924-b259-47a2-b006-0d94dcb6315f-maas-api-tls" pod "68597924-b259-47a2-b006-0d94dcb6315f" (UID: "68597924-b259-47a2-b006-0d94dcb6315f"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 01:42:23.272563 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:42:23.272472 4939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/68597924-b259-47a2-b006-0d94dcb6315f-kube-api-access-z5jrq" pod "68597924-b259-47a2-b006-0d94dcb6315f" (UID: "68597924-b259-47a2-b006-0d94dcb6315f"). InnerVolumeSpecName "kube-api-access-z5jrq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 01:42:23.272563 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:42:23.272533 4939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/68597924-b259-47a2-b006-0d94dcb6315f-metrics-certs" pod "68597924-b259-47a2-b006-0d94dcb6315f" (UID: "68597924-b259-47a2-b006-0d94dcb6315f"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 01:42:23.307404 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:42:23.307376 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:42:23.372387 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:42:23.372349 4939 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-z5jrq\" (UniqueName: \"kubernetes.io/projected/68597924-b259-47a2-b006-0d94dcb6315f-kube-api-access-z5jrq\") on node \"ip-10-0-140-107.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:42:23.372387 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:42:23.372383 4939 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/68597924-b259-47a2-b006-0d94dcb6315f-metrics-certs\") on node \"ip-10-0-140-107.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:42:23.372387 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:42:23.372393 4939 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/68597924-b259-47a2-b006-0d94dcb6315f-maas-api-tls\") on node \"ip-10-0-140-107.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:42:23.680071 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:42:23.680032 4939 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-650be0-58fc9b6455-9jl7f" podUID="c72f9b11-1218-4c3f-be7c-6541feeb1ed2" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 03 01:42:23.680444 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:42:23.680089 4939 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w4-650be0-86d95657b7-vm62r" podUID="58875207-cd7a-4da9-aee7-f538e7f44963" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 03 01:42:23.779261 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:42:23.779225 4939 generic.go:356] "Generic (PLEG): container finished" podID="68597924-b259-47a2-b006-0d94dcb6315f" containerID="c47b18d53a2f3cfa9a35572473600e99d0a508ff19b12d7209d16df6fced9a52" exitCode=0 Nov 03 01:42:23.779454 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:42:23.779324 4939 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-83755a-84f6d788fc-bgk7t" Nov 03 01:42:23.779454 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:42:23.779356 4939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-83755a-84f6d788fc-bgk7t" event={"ID":"68597924-b259-47a2-b006-0d94dcb6315f","Type":"ContainerDied","Data":"c47b18d53a2f3cfa9a35572473600e99d0a508ff19b12d7209d16df6fced9a52"} Nov 03 01:42:23.779454 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:42:23.779414 4939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-83755a-84f6d788fc-bgk7t" event={"ID":"68597924-b259-47a2-b006-0d94dcb6315f","Type":"ContainerDied","Data":"4bd4619b5544726517f2ead4e9ca162d1d45e189c2eee69c8786a4dfea8f599a"} Nov 03 01:42:23.779454 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:42:23.779443 4939 scope.go:122] "RemoveContainer" containerID="c47b18d53a2f3cfa9a35572473600e99d0a508ff19b12d7209d16df6fced9a52" Nov 03 01:42:23.781462 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:42:23.781434 4939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-2729f2-6f5b74c6cf-h9g6h" event={"ID":"6ec3abd1-a454-4860-8c1b-d70c26b14b3a","Type":"ContainerDied","Data":"06bfe3e3f1661f92afcdbe649013a7c13afee3cf46ce83ac5a921a2cf129c5e4"} Nov 03 01:42:23.781603 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:42:23.781445 4939 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-2729f2-6f5b74c6cf-h9g6h" Nov 03 01:42:23.791861 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:42:23.791838 4939 scope.go:122] "RemoveContainer" containerID="c47b18d53a2f3cfa9a35572473600e99d0a508ff19b12d7209d16df6fced9a52" Nov 03 01:42:23.793132 ip-10-0-140-107 kubenswrapper[4939]: E1103 01:42:23.793095 4939 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c47b18d53a2f3cfa9a35572473600e99d0a508ff19b12d7209d16df6fced9a52\": container with ID starting with c47b18d53a2f3cfa9a35572473600e99d0a508ff19b12d7209d16df6fced9a52 not found: ID does not exist" containerID="c47b18d53a2f3cfa9a35572473600e99d0a508ff19b12d7209d16df6fced9a52" Nov 03 01:42:23.793279 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:42:23.793181 4939 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c47b18d53a2f3cfa9a35572473600e99d0a508ff19b12d7209d16df6fced9a52"} err="failed to get container status \"c47b18d53a2f3cfa9a35572473600e99d0a508ff19b12d7209d16df6fced9a52\": rpc error: code = NotFound desc = could not find container \"c47b18d53a2f3cfa9a35572473600e99d0a508ff19b12d7209d16df6fced9a52\": container with ID starting with c47b18d53a2f3cfa9a35572473600e99d0a508ff19b12d7209d16df6fced9a52 not found: ID does not exist" Nov 03 01:42:23.793279 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:42:23.793206 4939 scope.go:122] "RemoveContainer" containerID="eff241f8fefc5e2ba6484d635be2a30a46e1089a49c2b897e7b35f8fa90a0b04" Nov 03 01:42:23.813091 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:42:23.813064 4939 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-2729f2-6f5b74c6cf-h9g6h"] Nov 03 01:42:23.816676 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:42:23.816651 4939 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-2729f2-6f5b74c6cf-h9g6h"] Nov 03 01:42:23.824704 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:42:23.824681 4939 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w3-83755a-84f6d788fc-bgk7t"] Nov 03 01:42:23.827979 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:42:23.827955 4939 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w3-83755a-84f6d788fc-bgk7t"] Nov 03 01:42:23.869918 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:42:23.869879 4939 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="68597924-b259-47a2-b006-0d94dcb6315f" path="/var/lib/kubelet/pods/68597924-b259-47a2-b006-0d94dcb6315f/volumes" Nov 03 01:42:23.870463 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:42:23.870436 4939 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6ec3abd1-a454-4860-8c1b-d70c26b14b3a" path="/var/lib/kubelet/pods/6ec3abd1-a454-4860-8c1b-d70c26b14b3a/volumes" Nov 03 01:42:24.010005 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:42:24.009894 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:42:24.653850 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:42:24.653809 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:42:25.290036 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:42:25.290004 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:42:25.890903 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:42:25.890873 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:42:26.590219 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:42:26.590173 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:42:27.088376 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:42:27.088335 4939 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-2729f2-565847c555-wqvj7" podUID="89acf0f1-ef07-4c99-afd3-de3d06335488" containerName="payload-processing" containerID="cri-o://a29f4ada5ae4f3bf33d23981cc60dd8a9fb34c3ed134e3c6cad41f2cfea2c7cd" gracePeriod=25 Nov 03 01:42:27.097615 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:42:27.097584 4939 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-2729f2-558f777dc-9gclz" podUID="c1e683ec-d1fc-4273-9384-1ffcc06a832a" containerName="payload-pre-processing" containerID="cri-o://076f1302b3c0a56f35ca8f05d46e5d588e15f1312c7e774aefbb2da2db6c2c2c" gracePeriod=25 Nov 03 01:42:27.247042 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:42:27.247010 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:42:27.272232 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:42:27.272189 4939 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-2729f2-565847c555-wqvj7" podUID="89acf0f1-ef07-4c99-afd3-de3d06335488" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 03 01:42:27.272403 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:42:27.272228 4939 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-2729f2-558f777dc-9gclz" podUID="c1e683ec-d1fc-4273-9384-1ffcc06a832a" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 03 01:42:27.357875 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:42:27.357848 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:42:27.477947 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:42:27.476512 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:42:27.610096 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:42:27.610034 4939 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w4-650be0-86d95657b7-vm62r" Nov 03 01:42:27.654205 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:42:27.654182 4939 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-650be0-58fc9b6455-9jl7f" Nov 03 01:42:27.722860 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:42:27.722827 4939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/58875207-cd7a-4da9-aee7-f538e7f44963-kube-api-access-qnlh8\" (UniqueName: \"kubernetes.io/projected/58875207-cd7a-4da9-aee7-f538e7f44963-kube-api-access-qnlh8\") pod \"58875207-cd7a-4da9-aee7-f538e7f44963\" (UID: \"58875207-cd7a-4da9-aee7-f538e7f44963\") " Nov 03 01:42:27.723067 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:42:27.722933 4939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/58875207-cd7a-4da9-aee7-f538e7f44963-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/58875207-cd7a-4da9-aee7-f538e7f44963-plugins-config-volume\") pod \"58875207-cd7a-4da9-aee7-f538e7f44963\" (UID: \"58875207-cd7a-4da9-aee7-f538e7f44963\") " Nov 03 01:42:27.723342 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:42:27.723313 4939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/58875207-cd7a-4da9-aee7-f538e7f44963-plugins-config-volume" pod "58875207-cd7a-4da9-aee7-f538e7f44963" (UID: "58875207-cd7a-4da9-aee7-f538e7f44963"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 01:42:27.725045 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:42:27.725021 4939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/58875207-cd7a-4da9-aee7-f538e7f44963-kube-api-access-qnlh8" pod "58875207-cd7a-4da9-aee7-f538e7f44963" (UID: "58875207-cd7a-4da9-aee7-f538e7f44963"). InnerVolumeSpecName "kube-api-access-qnlh8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 01:42:27.807262 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:42:27.807228 4939 generic.go:356] "Generic (PLEG): container finished" podID="c72f9b11-1218-4c3f-be7c-6541feeb1ed2" containerID="4438a1bb491548b813040ebf4eeb229485ebbe25ba7dfe0c4027433520ddd46e" exitCode=0 Nov 03 01:42:27.807450 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:42:27.807273 4939 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-650be0-58fc9b6455-9jl7f" Nov 03 01:42:27.807450 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:42:27.807304 4939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-650be0-58fc9b6455-9jl7f" event={"ID":"c72f9b11-1218-4c3f-be7c-6541feeb1ed2","Type":"ContainerDied","Data":"4438a1bb491548b813040ebf4eeb229485ebbe25ba7dfe0c4027433520ddd46e"} Nov 03 01:42:27.807450 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:42:27.807351 4939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-650be0-58fc9b6455-9jl7f" event={"ID":"c72f9b11-1218-4c3f-be7c-6541feeb1ed2","Type":"ContainerDied","Data":"f464f01b23f98abff545fc1f7bf400e86eb8bf6b808deb40742362525b3e0004"} Nov 03 01:42:27.807450 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:42:27.807374 4939 scope.go:122] "RemoveContainer" containerID="4438a1bb491548b813040ebf4eeb229485ebbe25ba7dfe0c4027433520ddd46e" Nov 03 01:42:27.808663 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:42:27.808643 4939 generic.go:356] "Generic (PLEG): container finished" podID="58875207-cd7a-4da9-aee7-f538e7f44963" containerID="368faaefacd433a21be99f2ad8a3867bc4ce409dac664ba52af1d6fd56e3ce4c" exitCode=0 Nov 03 01:42:27.808769 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:42:27.808689 4939 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w4-650be0-86d95657b7-vm62r" Nov 03 01:42:27.808769 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:42:27.808692 4939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w4-650be0-86d95657b7-vm62r" event={"ID":"58875207-cd7a-4da9-aee7-f538e7f44963","Type":"ContainerDied","Data":"368faaefacd433a21be99f2ad8a3867bc4ce409dac664ba52af1d6fd56e3ce4c"} Nov 03 01:42:27.808904 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:42:27.808805 4939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w4-650be0-86d95657b7-vm62r" event={"ID":"58875207-cd7a-4da9-aee7-f538e7f44963","Type":"ContainerDied","Data":"4ad2081b3d72b46f9fea2179cfd4fd66089cc26c3954c47f2cabb7959843af3d"} Nov 03 01:42:27.818121 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:42:27.818102 4939 scope.go:122] "RemoveContainer" containerID="4438a1bb491548b813040ebf4eeb229485ebbe25ba7dfe0c4027433520ddd46e" Nov 03 01:42:27.818421 ip-10-0-140-107 kubenswrapper[4939]: E1103 01:42:27.818401 4939 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4438a1bb491548b813040ebf4eeb229485ebbe25ba7dfe0c4027433520ddd46e\": container with ID starting with 4438a1bb491548b813040ebf4eeb229485ebbe25ba7dfe0c4027433520ddd46e not found: ID does not exist" containerID="4438a1bb491548b813040ebf4eeb229485ebbe25ba7dfe0c4027433520ddd46e" Nov 03 01:42:27.818514 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:42:27.818432 4939 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4438a1bb491548b813040ebf4eeb229485ebbe25ba7dfe0c4027433520ddd46e"} err="failed to get container status \"4438a1bb491548b813040ebf4eeb229485ebbe25ba7dfe0c4027433520ddd46e\": rpc error: code = NotFound desc = could not find container \"4438a1bb491548b813040ebf4eeb229485ebbe25ba7dfe0c4027433520ddd46e\": container with ID starting with 4438a1bb491548b813040ebf4eeb229485ebbe25ba7dfe0c4027433520ddd46e not found: ID does not exist" Nov 03 01:42:27.818514 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:42:27.818455 4939 scope.go:122] "RemoveContainer" containerID="368faaefacd433a21be99f2ad8a3867bc4ce409dac664ba52af1d6fd56e3ce4c" Nov 03 01:42:27.824244 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:42:27.824220 4939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c72f9b11-1218-4c3f-be7c-6541feeb1ed2-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c72f9b11-1218-4c3f-be7c-6541feeb1ed2-plugins-config-volume\") pod \"c72f9b11-1218-4c3f-be7c-6541feeb1ed2\" (UID: \"c72f9b11-1218-4c3f-be7c-6541feeb1ed2\") " Nov 03 01:42:27.824326 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:42:27.824291 4939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c72f9b11-1218-4c3f-be7c-6541feeb1ed2-kube-api-access-ppcrv\" (UniqueName: \"kubernetes.io/projected/c72f9b11-1218-4c3f-be7c-6541feeb1ed2-kube-api-access-ppcrv\") pod \"c72f9b11-1218-4c3f-be7c-6541feeb1ed2\" (UID: \"c72f9b11-1218-4c3f-be7c-6541feeb1ed2\") " Nov 03 01:42:27.824692 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:42:27.824665 4939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c72f9b11-1218-4c3f-be7c-6541feeb1ed2-plugins-config-volume" pod "c72f9b11-1218-4c3f-be7c-6541feeb1ed2" (UID: "c72f9b11-1218-4c3f-be7c-6541feeb1ed2"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 01:42:27.824796 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:42:27.824672 4939 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qnlh8\" (UniqueName: \"kubernetes.io/projected/58875207-cd7a-4da9-aee7-f538e7f44963-kube-api-access-qnlh8\") on node \"ip-10-0-140-107.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:42:27.824796 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:42:27.824720 4939 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/58875207-cd7a-4da9-aee7-f538e7f44963-plugins-config-volume\") on node \"ip-10-0-140-107.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:42:27.826424 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:42:27.826403 4939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c72f9b11-1218-4c3f-be7c-6541feeb1ed2-kube-api-access-ppcrv" pod "c72f9b11-1218-4c3f-be7c-6541feeb1ed2" (UID: "c72f9b11-1218-4c3f-be7c-6541feeb1ed2"). InnerVolumeSpecName "kube-api-access-ppcrv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 01:42:27.827821 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:42:27.827807 4939 scope.go:122] "RemoveContainer" containerID="368faaefacd433a21be99f2ad8a3867bc4ce409dac664ba52af1d6fd56e3ce4c" Nov 03 01:42:27.828124 ip-10-0-140-107 kubenswrapper[4939]: E1103 01:42:27.828100 4939 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"368faaefacd433a21be99f2ad8a3867bc4ce409dac664ba52af1d6fd56e3ce4c\": container with ID starting with 368faaefacd433a21be99f2ad8a3867bc4ce409dac664ba52af1d6fd56e3ce4c not found: ID does not exist" containerID="368faaefacd433a21be99f2ad8a3867bc4ce409dac664ba52af1d6fd56e3ce4c" Nov 03 01:42:27.828195 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:42:27.828133 4939 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"368faaefacd433a21be99f2ad8a3867bc4ce409dac664ba52af1d6fd56e3ce4c"} err="failed to get container status \"368faaefacd433a21be99f2ad8a3867bc4ce409dac664ba52af1d6fd56e3ce4c\": rpc error: code = NotFound desc = could not find container \"368faaefacd433a21be99f2ad8a3867bc4ce409dac664ba52af1d6fd56e3ce4c\": container with ID starting with 368faaefacd433a21be99f2ad8a3867bc4ce409dac664ba52af1d6fd56e3ce4c not found: ID does not exist" Nov 03 01:42:27.848956 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:42:27.848935 4939 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w4-650be0-86d95657b7-vm62r"] Nov 03 01:42:27.867664 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:42:27.867614 4939 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w4-650be0-86d95657b7-vm62r"] Nov 03 01:42:27.925566 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:42:27.925528 4939 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c72f9b11-1218-4c3f-be7c-6541feeb1ed2-plugins-config-volume\") on node \"ip-10-0-140-107.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:42:27.925566 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:42:27.925566 4939 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ppcrv\" (UniqueName: \"kubernetes.io/projected/c72f9b11-1218-4c3f-be7c-6541feeb1ed2-kube-api-access-ppcrv\") on node \"ip-10-0-140-107.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:42:28.123537 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:42:28.123456 4939 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w4-650be0-58fc9b6455-9jl7f"] Nov 03 01:42:28.125185 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:42:28.125160 4939 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w4-650be0-58fc9b6455-9jl7f"] Nov 03 01:42:28.204736 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:42:28.204704 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:42:28.315135 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:42:28.315097 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:42:28.344077 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:42:28.344045 4939 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-af8244-84f67c568c-m9kgf" podUID="1f0db3da-dcf7-459b-994e-a7081e6f0958" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 03 01:42:28.447809 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:42:28.447730 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:42:28.997540 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:42:28.997503 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:42:29.378698 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:42:29.378660 4939 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-extmod-b-af8244-75d9474dd7-pr6cd" podUID="d1f73b7c-084a-44b7-a3bd-c5042532a732" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 03 01:42:29.394388 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:42:29.394358 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:42:29.574180 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:42:29.574138 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:42:29.666251 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:42:29.666161 4939 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-650be0-openshift-default-5c7c5fb7d9-84qlq"] Nov 03 01:42:29.666717 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:42:29.666684 4939 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-650be0-openshift-default-5c7c5fb7d9-84qlq" podUID="d2dabe94-37c9-4a3f-bf9a-15e4f53e05d0" containerName="istio-proxy" containerID="cri-o://463608114ded4c17d374f75c240e2b2f1556c617d103054903616e64d1a01bc6" gracePeriod=30 Nov 03 01:42:29.679810 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:42:29.679747 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:42:29.783583 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:42:29.783549 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:42:29.868903 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:42:29.868866 4939 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="58875207-cd7a-4da9-aee7-f538e7f44963" path="/var/lib/kubelet/pods/58875207-cd7a-4da9-aee7-f538e7f44963/volumes" Nov 03 01:42:29.869295 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:42:29.869279 4939 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c72f9b11-1218-4c3f-be7c-6541feeb1ed2" path="/var/lib/kubelet/pods/c72f9b11-1218-4c3f-be7c-6541feeb1ed2/volumes" Nov 03 01:42:29.948981 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:42:29.948869 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:42:30.111555 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:42:30.111523 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:42:30.391429 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:42:30.391382 4939 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w1-2729f2-openshift-default-9fb97b4fc-6jhb8"] Nov 03 01:42:30.391680 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:42:30.391647 4939 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w1-2729f2-openshift-default-9fb97b4fc-6jhb8" podUID="61c402c5-4efa-4821-b51e-da74f4d9eff7" containerName="istio-proxy" containerID="cri-o://26b79eddeeac2d4fc4ade04398e94fa28e6ddcc8d46013647c68074fbdad8d12" gracePeriod=30 Nov 03 01:42:30.453821 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:42:30.453790 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:42:30.766213 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:42:30.766123 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:42:31.259016 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:42:31.258985 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:42:31.602868 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:42:31.602830 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:42:31.725758 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:42:31.725725 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:42:31.847373 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:42:31.847172 4939 generic.go:356] "Generic (PLEG): container finished" podID="d1f73b7c-084a-44b7-a3bd-c5042532a732" containerID="4eccef524b9821502c808736a0d11ae8bd8a6add2167c878b11568b82991675f" exitCode=0 Nov 03 01:42:31.847373 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:42:31.847307 4939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-b-af8244-75d9474dd7-pr6cd" event={"ID":"d1f73b7c-084a-44b7-a3bd-c5042532a732","Type":"ContainerDied","Data":"4eccef524b9821502c808736a0d11ae8bd8a6add2167c878b11568b82991675f"} Nov 03 01:42:31.849269 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:42:31.849243 4939 generic.go:356] "Generic (PLEG): container finished" podID="1f0db3da-dcf7-459b-994e-a7081e6f0958" containerID="22f578bc64e5a78ccb195588fcf509b901b156969faf20a928fc0d92ff8e06fb" exitCode=0 Nov 03 01:42:31.849390 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:42:31.849300 4939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-af8244-84f67c568c-m9kgf" event={"ID":"1f0db3da-dcf7-459b-994e-a7081e6f0958","Type":"ContainerDied","Data":"22f578bc64e5a78ccb195588fcf509b901b156969faf20a928fc0d92ff8e06fb"} Nov 03 01:42:31.933946 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:42:31.932618 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:42:32.033220 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:42:32.033191 4939 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-b-af8244-75d9474dd7-pr6cd" Nov 03 01:42:32.054533 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:42:32.054506 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:42:32.066781 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:42:32.066757 4939 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-af8244-84f67c568c-m9kgf" Nov 03 01:42:32.107184 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:42:32.106426 4939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d1f73b7c-084a-44b7-a3bd-c5042532a732-kube-api-access-jfltb\" (UniqueName: \"kubernetes.io/projected/d1f73b7c-084a-44b7-a3bd-c5042532a732-kube-api-access-jfltb\") pod \"d1f73b7c-084a-44b7-a3bd-c5042532a732\" (UID: \"d1f73b7c-084a-44b7-a3bd-c5042532a732\") " Nov 03 01:42:32.107184 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:42:32.106512 4939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1f0db3da-dcf7-459b-994e-a7081e6f0958-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1f0db3da-dcf7-459b-994e-a7081e6f0958-plugins-config-volume\") pod \"1f0db3da-dcf7-459b-994e-a7081e6f0958\" (UID: \"1f0db3da-dcf7-459b-994e-a7081e6f0958\") " Nov 03 01:42:32.107184 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:42:32.106559 4939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1f0db3da-dcf7-459b-994e-a7081e6f0958-kube-api-access-xb8cj\" (UniqueName: \"kubernetes.io/projected/1f0db3da-dcf7-459b-994e-a7081e6f0958-kube-api-access-xb8cj\") pod \"1f0db3da-dcf7-459b-994e-a7081e6f0958\" (UID: \"1f0db3da-dcf7-459b-994e-a7081e6f0958\") " Nov 03 01:42:32.107184 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:42:32.106646 4939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d1f73b7c-084a-44b7-a3bd-c5042532a732-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d1f73b7c-084a-44b7-a3bd-c5042532a732-plugins-config-volume\") pod \"d1f73b7c-084a-44b7-a3bd-c5042532a732\" (UID: \"d1f73b7c-084a-44b7-a3bd-c5042532a732\") " Nov 03 01:42:32.107554 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:42:32.107501 4939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d1f73b7c-084a-44b7-a3bd-c5042532a732-plugins-config-volume" pod "d1f73b7c-084a-44b7-a3bd-c5042532a732" (UID: "d1f73b7c-084a-44b7-a3bd-c5042532a732"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 01:42:32.111952 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:42:32.111902 4939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1f0db3da-dcf7-459b-994e-a7081e6f0958-kube-api-access-xb8cj" pod "1f0db3da-dcf7-459b-994e-a7081e6f0958" (UID: "1f0db3da-dcf7-459b-994e-a7081e6f0958"). InnerVolumeSpecName "kube-api-access-xb8cj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 01:42:32.113556 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:42:32.113139 4939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d1f73b7c-084a-44b7-a3bd-c5042532a732-kube-api-access-jfltb" pod "d1f73b7c-084a-44b7-a3bd-c5042532a732" (UID: "d1f73b7c-084a-44b7-a3bd-c5042532a732"). InnerVolumeSpecName "kube-api-access-jfltb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 01:42:32.144179 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:42:32.144087 4939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1f0db3da-dcf7-459b-994e-a7081e6f0958-plugins-config-volume" pod "1f0db3da-dcf7-459b-994e-a7081e6f0958" (UID: "1f0db3da-dcf7-459b-994e-a7081e6f0958"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 01:42:32.208559 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:42:32.208525 4939 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jfltb\" (UniqueName: \"kubernetes.io/projected/d1f73b7c-084a-44b7-a3bd-c5042532a732-kube-api-access-jfltb\") on node \"ip-10-0-140-107.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:42:32.208559 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:42:32.208556 4939 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1f0db3da-dcf7-459b-994e-a7081e6f0958-plugins-config-volume\") on node \"ip-10-0-140-107.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:42:32.208559 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:42:32.208567 4939 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xb8cj\" (UniqueName: \"kubernetes.io/projected/1f0db3da-dcf7-459b-994e-a7081e6f0958-kube-api-access-xb8cj\") on node \"ip-10-0-140-107.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:42:32.208802 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:42:32.208576 4939 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d1f73b7c-084a-44b7-a3bd-c5042532a732-plugins-config-volume\") on node \"ip-10-0-140-107.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:42:32.316120 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:42:32.316079 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:42:32.516558 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:42:32.516470 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:42:32.856299 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:42:32.856268 4939 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-b-af8244-75d9474dd7-pr6cd" Nov 03 01:42:32.856723 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:42:32.856306 4939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-b-af8244-75d9474dd7-pr6cd" event={"ID":"d1f73b7c-084a-44b7-a3bd-c5042532a732","Type":"ContainerDied","Data":"87e3e43af5dcf61e99c104b70b58c3918bb7cad0b95c9257e96b9793b280bc89"} Nov 03 01:42:32.856723 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:42:32.856372 4939 scope.go:122] "RemoveContainer" containerID="4eccef524b9821502c808736a0d11ae8bd8a6add2167c878b11568b82991675f" Nov 03 01:42:32.857805 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:42:32.857785 4939 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-af8244-84f67c568c-m9kgf" Nov 03 01:42:32.857888 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:42:32.857785 4939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-af8244-84f67c568c-m9kgf" event={"ID":"1f0db3da-dcf7-459b-994e-a7081e6f0958","Type":"ContainerDied","Data":"59ae3b21edab922dd4ad5111e7c776e38e208f20099ea2868c3e243e2e7197bc"} Nov 03 01:42:32.867677 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:42:32.867629 4939 scope.go:122] "RemoveContainer" containerID="22f578bc64e5a78ccb195588fcf509b901b156969faf20a928fc0d92ff8e06fb" Nov 03 01:42:32.879515 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:42:32.879491 4939 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-b-af8244-75d9474dd7-pr6cd"] Nov 03 01:42:32.883571 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:42:32.883546 4939 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-b-af8244-75d9474dd7-pr6cd"] Nov 03 01:42:32.892623 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:42:32.892600 4939 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-af8244-84f67c568c-m9kgf"] Nov 03 01:42:32.896554 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:42:32.896532 4939 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-af8244-84f67c568c-m9kgf"] Nov 03 01:42:33.214137 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:42:33.214044 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:42:33.679832 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:42:33.679789 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:42:33.874903 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:42:33.874859 4939 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1f0db3da-dcf7-459b-994e-a7081e6f0958" path="/var/lib/kubelet/pods/1f0db3da-dcf7-459b-994e-a7081e6f0958/volumes" Nov 03 01:42:33.875377 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:42:33.875358 4939 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d1f73b7c-084a-44b7-a3bd-c5042532a732" path="/var/lib/kubelet/pods/d1f73b7c-084a-44b7-a3bd-c5042532a732/volumes" Nov 03 01:42:33.988414 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:42:33.988337 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:42:34.637483 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:42:34.637436 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:42:34.816241 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:42:34.816208 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:42:34.878856 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:42:34.878816 4939 generic.go:356] "Generic (PLEG): container finished" podID="d2dabe94-37c9-4a3f-bf9a-15e4f53e05d0" containerID="463608114ded4c17d374f75c240e2b2f1556c617d103054903616e64d1a01bc6" exitCode=0 Nov 03 01:42:34.879053 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:42:34.878876 4939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-650be0-openshift-default-5c7c5fb7d9-84qlq" event={"ID":"d2dabe94-37c9-4a3f-bf9a-15e4f53e05d0","Type":"ContainerDied","Data":"463608114ded4c17d374f75c240e2b2f1556c617d103054903616e64d1a01bc6"} Nov 03 01:42:34.940758 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:42:34.940684 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:42:35.130891 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:42:35.130859 4939 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-83755a-openshift-default-c86f9565c-x9v2k"] Nov 03 01:42:35.131318 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:42:35.131303 4939 cpu_manager.go:414] "containerMap: removing container" podUID="23f02c28-7f9e-40d8-990c-aea9e85f1bf6" containerName="payload-processing" Nov 03 01:42:35.131370 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:42:35.131319 4939 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="23f02c28-7f9e-40d8-990c-aea9e85f1bf6" containerName="payload-processing" Nov 03 01:42:35.131370 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:42:35.131331 4939 cpu_manager.go:414] "containerMap: removing container" podUID="192c8a1d-60d2-4928-9579-0231138280bb" containerName="payload-pre-processing" Nov 03 01:42:35.131370 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:42:35.131336 4939 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="192c8a1d-60d2-4928-9579-0231138280bb" containerName="payload-pre-processing" Nov 03 01:42:35.131370 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:42:35.131352 4939 cpu_manager.go:414] "containerMap: removing container" podUID="efde608b-c334-4d17-a0aa-cad98a95a999" containerName="istio-proxy" Nov 03 01:42:35.131370 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:42:35.131358 4939 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="efde608b-c334-4d17-a0aa-cad98a95a999" containerName="istio-proxy" Nov 03 01:42:35.131370 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:42:35.131366 4939 cpu_manager.go:414] "containerMap: removing container" podUID="efde608b-c334-4d17-a0aa-cad98a95a999" containerName="istio-proxy" Nov 03 01:42:35.131370 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:42:35.131370 4939 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="efde608b-c334-4d17-a0aa-cad98a95a999" containerName="istio-proxy" Nov 03 01:42:35.131587 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:42:35.131385 4939 cpu_manager.go:414] "containerMap: removing container" podUID="d1f73b7c-084a-44b7-a3bd-c5042532a732" containerName="payload-processing" Nov 03 01:42:35.131587 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:42:35.131391 4939 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d1f73b7c-084a-44b7-a3bd-c5042532a732" containerName="payload-processing" Nov 03 01:42:35.131587 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:42:35.131399 4939 cpu_manager.go:414] "containerMap: removing container" podUID="c72f9b11-1218-4c3f-be7c-6541feeb1ed2" containerName="payload-pre-processing" Nov 03 01:42:35.131587 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:42:35.131404 4939 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c72f9b11-1218-4c3f-be7c-6541feeb1ed2" containerName="payload-pre-processing" Nov 03 01:42:35.131587 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:42:35.131412 4939 cpu_manager.go:414] "containerMap: removing container" podUID="c3b05e88-5d5b-4d03-b28d-832f22e3c868" containerName="main" Nov 03 01:42:35.131587 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:42:35.131417 4939 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c3b05e88-5d5b-4d03-b28d-832f22e3c868" containerName="main" Nov 03 01:42:35.131587 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:42:35.131425 4939 cpu_manager.go:414] "containerMap: removing container" podUID="5b0cabc6-4960-4dc1-8317-3faebf25b30c" containerName="istio-proxy" Nov 03 01:42:35.131587 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:42:35.131429 4939 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5b0cabc6-4960-4dc1-8317-3faebf25b30c" containerName="istio-proxy" Nov 03 01:42:35.131587 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:42:35.131442 4939 cpu_manager.go:414] "containerMap: removing container" podUID="68597924-b259-47a2-b006-0d94dcb6315f" containerName="maas-api" Nov 03 01:42:35.131587 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:42:35.131447 4939 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="68597924-b259-47a2-b006-0d94dcb6315f" containerName="maas-api" Nov 03 01:42:35.131587 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:42:35.131453 4939 cpu_manager.go:414] "containerMap: removing container" podUID="1f0db3da-dcf7-459b-994e-a7081e6f0958" containerName="payload-pre-processing" Nov 03 01:42:35.131587 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:42:35.131458 4939 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1f0db3da-dcf7-459b-994e-a7081e6f0958" containerName="payload-pre-processing" Nov 03 01:42:35.131587 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:42:35.131468 4939 cpu_manager.go:414] "containerMap: removing container" podUID="58875207-cd7a-4da9-aee7-f538e7f44963" containerName="payload-processing" Nov 03 01:42:35.131587 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:42:35.131472 4939 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="58875207-cd7a-4da9-aee7-f538e7f44963" containerName="payload-processing" Nov 03 01:42:35.131587 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:42:35.131479 4939 cpu_manager.go:414] "containerMap: removing container" podUID="6ec3abd1-a454-4860-8c1b-d70c26b14b3a" containerName="maas-api" Nov 03 01:42:35.131587 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:42:35.131485 4939 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6ec3abd1-a454-4860-8c1b-d70c26b14b3a" containerName="maas-api" Nov 03 01:42:35.131587 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:42:35.131545 4939 memory_manager.go:365] "RemoveStaleState removing state" podUID="68597924-b259-47a2-b006-0d94dcb6315f" containerName="maas-api" Nov 03 01:42:35.131587 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:42:35.131554 4939 memory_manager.go:365] "RemoveStaleState removing state" podUID="1f0db3da-dcf7-459b-994e-a7081e6f0958" containerName="payload-pre-processing" Nov 03 01:42:35.131587 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:42:35.131561 4939 memory_manager.go:365] "RemoveStaleState removing state" podUID="58875207-cd7a-4da9-aee7-f538e7f44963" containerName="payload-processing" Nov 03 01:42:35.131587 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:42:35.131566 4939 memory_manager.go:365] "RemoveStaleState removing state" podUID="6ec3abd1-a454-4860-8c1b-d70c26b14b3a" containerName="maas-api" Nov 03 01:42:35.131587 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:42:35.131573 4939 memory_manager.go:365] "RemoveStaleState removing state" podUID="23f02c28-7f9e-40d8-990c-aea9e85f1bf6" containerName="payload-processing" Nov 03 01:42:35.131587 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:42:35.131579 4939 memory_manager.go:365] "RemoveStaleState removing state" podUID="192c8a1d-60d2-4928-9579-0231138280bb" containerName="payload-pre-processing" Nov 03 01:42:35.132303 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:42:35.131603 4939 memory_manager.go:365] "RemoveStaleState removing state" podUID="efde608b-c334-4d17-a0aa-cad98a95a999" containerName="istio-proxy" Nov 03 01:42:35.132303 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:42:35.131610 4939 memory_manager.go:365] "RemoveStaleState removing state" podUID="efde608b-c334-4d17-a0aa-cad98a95a999" containerName="istio-proxy" Nov 03 01:42:35.132303 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:42:35.131617 4939 memory_manager.go:365] "RemoveStaleState removing state" podUID="d1f73b7c-084a-44b7-a3bd-c5042532a732" containerName="payload-processing" Nov 03 01:42:35.132303 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:42:35.131622 4939 memory_manager.go:365] "RemoveStaleState removing state" podUID="285b736b-2a9f-40cd-96c4-3d7193d3eb8d" containerName="istio-proxy" Nov 03 01:42:35.132303 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:42:35.131628 4939 memory_manager.go:365] "RemoveStaleState removing state" podUID="c72f9b11-1218-4c3f-be7c-6541feeb1ed2" containerName="payload-pre-processing" Nov 03 01:42:35.132303 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:42:35.131634 4939 memory_manager.go:365] "RemoveStaleState removing state" podUID="c3b05e88-5d5b-4d03-b28d-832f22e3c868" containerName="main" Nov 03 01:42:35.134847 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:42:35.134826 4939 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-83755a-openshift-default-c86f9565c-x9v2k" Nov 03 01:42:35.145937 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:42:35.145897 4939 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-83755a-openshift-default-c86f9565c-vm5n9"] Nov 03 01:42:35.152721 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:42:35.152698 4939 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-83755a-openshift-default-c86f9565c-x9v2k"] Nov 03 01:42:35.152876 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:42:35.152794 4939 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-83755a-openshift-default-c86f9565c-vm5n9" Nov 03 01:42:35.160433 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:42:35.160394 4939 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w3-83755a-openshift-default-dockercfg-qhtrs\"" Nov 03 01:42:35.168647 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:42:35.168569 4939 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-83755a-openshift-default-c86f9565c-vm5n9"] Nov 03 01:42:35.242146 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:42:35.242116 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4bcec813-d864-450b-b606-d97e7c1a1948-workload-socket\") pod \"e2e-worker-w3-83755a-openshift-default-c86f9565c-x9v2k\" (UID: \"4bcec813-d864-450b-b606-d97e7c1a1948\") " pod="openshift-ingress/e2e-worker-w3-83755a-openshift-default-c86f9565c-x9v2k" Nov 03 01:42:35.242241 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:42:35.242167 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bdedc3dc-7ab3-4c41-84ea-e1443fd30898-istio-data\") pod \"e2e-worker-w3-83755a-openshift-default-c86f9565c-vm5n9\" (UID: \"bdedc3dc-7ab3-4c41-84ea-e1443fd30898\") " pod="openshift-ingress/e2e-worker-w3-83755a-openshift-default-c86f9565c-vm5n9" Nov 03 01:42:35.242302 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:42:35.242259 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ggmpl\" (UniqueName: \"kubernetes.io/projected/bdedc3dc-7ab3-4c41-84ea-e1443fd30898-kube-api-access-ggmpl\") pod \"e2e-worker-w3-83755a-openshift-default-c86f9565c-vm5n9\" (UID: \"bdedc3dc-7ab3-4c41-84ea-e1443fd30898\") " pod="openshift-ingress/e2e-worker-w3-83755a-openshift-default-c86f9565c-vm5n9" Nov 03 01:42:35.242355 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:42:35.242332 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bdedc3dc-7ab3-4c41-84ea-e1443fd30898-workload-certs\") pod \"e2e-worker-w3-83755a-openshift-default-c86f9565c-vm5n9\" (UID: \"bdedc3dc-7ab3-4c41-84ea-e1443fd30898\") " pod="openshift-ingress/e2e-worker-w3-83755a-openshift-default-c86f9565c-vm5n9" Nov 03 01:42:35.242410 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:42:35.242368 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bdedc3dc-7ab3-4c41-84ea-e1443fd30898-istio-token\") pod \"e2e-worker-w3-83755a-openshift-default-c86f9565c-vm5n9\" (UID: \"bdedc3dc-7ab3-4c41-84ea-e1443fd30898\") " pod="openshift-ingress/e2e-worker-w3-83755a-openshift-default-c86f9565c-vm5n9" Nov 03 01:42:35.242410 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:42:35.242397 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kwdqt\" (UniqueName: \"kubernetes.io/projected/4bcec813-d864-450b-b606-d97e7c1a1948-kube-api-access-kwdqt\") pod \"e2e-worker-w3-83755a-openshift-default-c86f9565c-x9v2k\" (UID: \"4bcec813-d864-450b-b606-d97e7c1a1948\") " pod="openshift-ingress/e2e-worker-w3-83755a-openshift-default-c86f9565c-x9v2k" Nov 03 01:42:35.242516 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:42:35.242437 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bdedc3dc-7ab3-4c41-84ea-e1443fd30898-istio-podinfo\") pod \"e2e-worker-w3-83755a-openshift-default-c86f9565c-vm5n9\" (UID: \"bdedc3dc-7ab3-4c41-84ea-e1443fd30898\") " pod="openshift-ingress/e2e-worker-w3-83755a-openshift-default-c86f9565c-vm5n9" Nov 03 01:42:35.242516 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:42:35.242499 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bdedc3dc-7ab3-4c41-84ea-e1443fd30898-workload-socket\") pod \"e2e-worker-w3-83755a-openshift-default-c86f9565c-vm5n9\" (UID: \"bdedc3dc-7ab3-4c41-84ea-e1443fd30898\") " pod="openshift-ingress/e2e-worker-w3-83755a-openshift-default-c86f9565c-vm5n9" Nov 03 01:42:35.242617 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:42:35.242538 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4bcec813-d864-450b-b606-d97e7c1a1948-credential-socket\") pod \"e2e-worker-w3-83755a-openshift-default-c86f9565c-x9v2k\" (UID: \"4bcec813-d864-450b-b606-d97e7c1a1948\") " pod="openshift-ingress/e2e-worker-w3-83755a-openshift-default-c86f9565c-x9v2k" Nov 03 01:42:35.242617 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:42:35.242564 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4bcec813-d864-450b-b606-d97e7c1a1948-istio-envoy\") pod \"e2e-worker-w3-83755a-openshift-default-c86f9565c-x9v2k\" (UID: \"4bcec813-d864-450b-b606-d97e7c1a1948\") " pod="openshift-ingress/e2e-worker-w3-83755a-openshift-default-c86f9565c-x9v2k" Nov 03 01:42:35.242617 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:42:35.242609 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4bcec813-d864-450b-b606-d97e7c1a1948-istio-data\") pod \"e2e-worker-w3-83755a-openshift-default-c86f9565c-x9v2k\" (UID: \"4bcec813-d864-450b-b606-d97e7c1a1948\") " pod="openshift-ingress/e2e-worker-w3-83755a-openshift-default-c86f9565c-x9v2k" Nov 03 01:42:35.242780 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:42:35.242664 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4bcec813-d864-450b-b606-d97e7c1a1948-istio-token\") pod \"e2e-worker-w3-83755a-openshift-default-c86f9565c-x9v2k\" (UID: \"4bcec813-d864-450b-b606-d97e7c1a1948\") " pod="openshift-ingress/e2e-worker-w3-83755a-openshift-default-c86f9565c-x9v2k" Nov 03 01:42:35.242780 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:42:35.242699 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4bcec813-d864-450b-b606-d97e7c1a1948-istiod-ca-cert\") pod \"e2e-worker-w3-83755a-openshift-default-c86f9565c-x9v2k\" (UID: \"4bcec813-d864-450b-b606-d97e7c1a1948\") " pod="openshift-ingress/e2e-worker-w3-83755a-openshift-default-c86f9565c-x9v2k" Nov 03 01:42:35.242780 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:42:35.242726 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bdedc3dc-7ab3-4c41-84ea-e1443fd30898-istiod-ca-cert\") pod \"e2e-worker-w3-83755a-openshift-default-c86f9565c-vm5n9\" (UID: \"bdedc3dc-7ab3-4c41-84ea-e1443fd30898\") " pod="openshift-ingress/e2e-worker-w3-83755a-openshift-default-c86f9565c-vm5n9" Nov 03 01:42:35.242780 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:42:35.242753 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4bcec813-d864-450b-b606-d97e7c1a1948-istio-podinfo\") pod \"e2e-worker-w3-83755a-openshift-default-c86f9565c-x9v2k\" (UID: \"4bcec813-d864-450b-b606-d97e7c1a1948\") " pod="openshift-ingress/e2e-worker-w3-83755a-openshift-default-c86f9565c-x9v2k" Nov 03 01:42:35.243015 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:42:35.242796 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4bcec813-d864-450b-b606-d97e7c1a1948-workload-certs\") pod \"e2e-worker-w3-83755a-openshift-default-c86f9565c-x9v2k\" (UID: \"4bcec813-d864-450b-b606-d97e7c1a1948\") " pod="openshift-ingress/e2e-worker-w3-83755a-openshift-default-c86f9565c-x9v2k" Nov 03 01:42:35.243015 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:42:35.242830 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bdedc3dc-7ab3-4c41-84ea-e1443fd30898-istio-envoy\") pod \"e2e-worker-w3-83755a-openshift-default-c86f9565c-vm5n9\" (UID: \"bdedc3dc-7ab3-4c41-84ea-e1443fd30898\") " pod="openshift-ingress/e2e-worker-w3-83755a-openshift-default-c86f9565c-vm5n9" Nov 03 01:42:35.243015 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:42:35.242892 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bdedc3dc-7ab3-4c41-84ea-e1443fd30898-credential-socket\") pod \"e2e-worker-w3-83755a-openshift-default-c86f9565c-vm5n9\" (UID: \"bdedc3dc-7ab3-4c41-84ea-e1443fd30898\") " pod="openshift-ingress/e2e-worker-w3-83755a-openshift-default-c86f9565c-vm5n9" Nov 03 01:42:35.336055 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:42:35.336029 4939 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-650be0-openshift-default-5c7c5fb7d9-84qlq" Nov 03 01:42:35.343672 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:42:35.343640 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bdedc3dc-7ab3-4c41-84ea-e1443fd30898-istio-token\") pod \"e2e-worker-w3-83755a-openshift-default-c86f9565c-vm5n9\" (UID: \"bdedc3dc-7ab3-4c41-84ea-e1443fd30898\") " pod="openshift-ingress/e2e-worker-w3-83755a-openshift-default-c86f9565c-vm5n9" Nov 03 01:42:35.343827 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:42:35.343674 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kwdqt\" (UniqueName: \"kubernetes.io/projected/4bcec813-d864-450b-b606-d97e7c1a1948-kube-api-access-kwdqt\") pod \"e2e-worker-w3-83755a-openshift-default-c86f9565c-x9v2k\" (UID: \"4bcec813-d864-450b-b606-d97e7c1a1948\") " pod="openshift-ingress/e2e-worker-w3-83755a-openshift-default-c86f9565c-x9v2k" Nov 03 01:42:35.343827 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:42:35.343697 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bdedc3dc-7ab3-4c41-84ea-e1443fd30898-istio-podinfo\") pod \"e2e-worker-w3-83755a-openshift-default-c86f9565c-vm5n9\" (UID: \"bdedc3dc-7ab3-4c41-84ea-e1443fd30898\") " pod="openshift-ingress/e2e-worker-w3-83755a-openshift-default-c86f9565c-vm5n9" Nov 03 01:42:35.343827 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:42:35.343715 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bdedc3dc-7ab3-4c41-84ea-e1443fd30898-workload-socket\") pod \"e2e-worker-w3-83755a-openshift-default-c86f9565c-vm5n9\" (UID: \"bdedc3dc-7ab3-4c41-84ea-e1443fd30898\") " pod="openshift-ingress/e2e-worker-w3-83755a-openshift-default-c86f9565c-vm5n9" Nov 03 01:42:35.344080 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:42:35.343879 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4bcec813-d864-450b-b606-d97e7c1a1948-credential-socket\") pod \"e2e-worker-w3-83755a-openshift-default-c86f9565c-x9v2k\" (UID: \"4bcec813-d864-450b-b606-d97e7c1a1948\") " pod="openshift-ingress/e2e-worker-w3-83755a-openshift-default-c86f9565c-x9v2k" Nov 03 01:42:35.344175 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:42:35.344152 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bdedc3dc-7ab3-4c41-84ea-e1443fd30898-workload-socket\") pod \"e2e-worker-w3-83755a-openshift-default-c86f9565c-vm5n9\" (UID: \"bdedc3dc-7ab3-4c41-84ea-e1443fd30898\") " pod="openshift-ingress/e2e-worker-w3-83755a-openshift-default-c86f9565c-vm5n9" Nov 03 01:42:35.344270 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:42:35.344204 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4bcec813-d864-450b-b606-d97e7c1a1948-istio-envoy\") pod \"e2e-worker-w3-83755a-openshift-default-c86f9565c-x9v2k\" (UID: \"4bcec813-d864-450b-b606-d97e7c1a1948\") " pod="openshift-ingress/e2e-worker-w3-83755a-openshift-default-c86f9565c-x9v2k" Nov 03 01:42:35.344270 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:42:35.344243 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4bcec813-d864-450b-b606-d97e7c1a1948-istio-data\") pod \"e2e-worker-w3-83755a-openshift-default-c86f9565c-x9v2k\" (UID: \"4bcec813-d864-450b-b606-d97e7c1a1948\") " pod="openshift-ingress/e2e-worker-w3-83755a-openshift-default-c86f9565c-x9v2k" Nov 03 01:42:35.344370 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:42:35.344296 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4bcec813-d864-450b-b606-d97e7c1a1948-istio-token\") pod \"e2e-worker-w3-83755a-openshift-default-c86f9565c-x9v2k\" (UID: \"4bcec813-d864-450b-b606-d97e7c1a1948\") " pod="openshift-ingress/e2e-worker-w3-83755a-openshift-default-c86f9565c-x9v2k" Nov 03 01:42:35.344370 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:42:35.344323 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4bcec813-d864-450b-b606-d97e7c1a1948-istiod-ca-cert\") pod \"e2e-worker-w3-83755a-openshift-default-c86f9565c-x9v2k\" (UID: \"4bcec813-d864-450b-b606-d97e7c1a1948\") " pod="openshift-ingress/e2e-worker-w3-83755a-openshift-default-c86f9565c-x9v2k" Nov 03 01:42:35.344370 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:42:35.344349 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bdedc3dc-7ab3-4c41-84ea-e1443fd30898-istiod-ca-cert\") pod \"e2e-worker-w3-83755a-openshift-default-c86f9565c-vm5n9\" (UID: \"bdedc3dc-7ab3-4c41-84ea-e1443fd30898\") " pod="openshift-ingress/e2e-worker-w3-83755a-openshift-default-c86f9565c-vm5n9" Nov 03 01:42:35.344550 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:42:35.344374 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4bcec813-d864-450b-b606-d97e7c1a1948-istio-podinfo\") pod \"e2e-worker-w3-83755a-openshift-default-c86f9565c-x9v2k\" (UID: \"4bcec813-d864-450b-b606-d97e7c1a1948\") " pod="openshift-ingress/e2e-worker-w3-83755a-openshift-default-c86f9565c-x9v2k" Nov 03 01:42:35.344550 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:42:35.344417 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4bcec813-d864-450b-b606-d97e7c1a1948-workload-certs\") pod \"e2e-worker-w3-83755a-openshift-default-c86f9565c-x9v2k\" (UID: \"4bcec813-d864-450b-b606-d97e7c1a1948\") " pod="openshift-ingress/e2e-worker-w3-83755a-openshift-default-c86f9565c-x9v2k" Nov 03 01:42:35.344550 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:42:35.344452 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bdedc3dc-7ab3-4c41-84ea-e1443fd30898-istio-envoy\") pod \"e2e-worker-w3-83755a-openshift-default-c86f9565c-vm5n9\" (UID: \"bdedc3dc-7ab3-4c41-84ea-e1443fd30898\") " pod="openshift-ingress/e2e-worker-w3-83755a-openshift-default-c86f9565c-vm5n9" Nov 03 01:42:35.344550 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:42:35.344347 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4bcec813-d864-450b-b606-d97e7c1a1948-credential-socket\") pod \"e2e-worker-w3-83755a-openshift-default-c86f9565c-x9v2k\" (UID: \"4bcec813-d864-450b-b606-d97e7c1a1948\") " pod="openshift-ingress/e2e-worker-w3-83755a-openshift-default-c86f9565c-x9v2k" Nov 03 01:42:35.344550 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:42:35.344525 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bdedc3dc-7ab3-4c41-84ea-e1443fd30898-credential-socket\") pod \"e2e-worker-w3-83755a-openshift-default-c86f9565c-vm5n9\" (UID: \"bdedc3dc-7ab3-4c41-84ea-e1443fd30898\") " pod="openshift-ingress/e2e-worker-w3-83755a-openshift-default-c86f9565c-vm5n9" Nov 03 01:42:35.344852 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:42:35.344610 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4bcec813-d864-450b-b606-d97e7c1a1948-istio-data\") pod \"e2e-worker-w3-83755a-openshift-default-c86f9565c-x9v2k\" (UID: \"4bcec813-d864-450b-b606-d97e7c1a1948\") " pod="openshift-ingress/e2e-worker-w3-83755a-openshift-default-c86f9565c-x9v2k" Nov 03 01:42:35.344852 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:42:35.344616 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4bcec813-d864-450b-b606-d97e7c1a1948-workload-socket\") pod \"e2e-worker-w3-83755a-openshift-default-c86f9565c-x9v2k\" (UID: \"4bcec813-d864-450b-b606-d97e7c1a1948\") " pod="openshift-ingress/e2e-worker-w3-83755a-openshift-default-c86f9565c-x9v2k" Nov 03 01:42:35.344852 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:42:35.344670 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bdedc3dc-7ab3-4c41-84ea-e1443fd30898-istio-data\") pod \"e2e-worker-w3-83755a-openshift-default-c86f9565c-vm5n9\" (UID: \"bdedc3dc-7ab3-4c41-84ea-e1443fd30898\") " pod="openshift-ingress/e2e-worker-w3-83755a-openshift-default-c86f9565c-vm5n9" Nov 03 01:42:35.344852 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:42:35.344702 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ggmpl\" (UniqueName: \"kubernetes.io/projected/bdedc3dc-7ab3-4c41-84ea-e1443fd30898-kube-api-access-ggmpl\") pod \"e2e-worker-w3-83755a-openshift-default-c86f9565c-vm5n9\" (UID: \"bdedc3dc-7ab3-4c41-84ea-e1443fd30898\") " pod="openshift-ingress/e2e-worker-w3-83755a-openshift-default-c86f9565c-vm5n9" Nov 03 01:42:35.344852 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:42:35.344764 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bdedc3dc-7ab3-4c41-84ea-e1443fd30898-workload-certs\") pod \"e2e-worker-w3-83755a-openshift-default-c86f9565c-vm5n9\" (UID: \"bdedc3dc-7ab3-4c41-84ea-e1443fd30898\") " pod="openshift-ingress/e2e-worker-w3-83755a-openshift-default-c86f9565c-vm5n9" Nov 03 01:42:35.344852 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:42:35.344824 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4bcec813-d864-450b-b606-d97e7c1a1948-workload-certs\") pod \"e2e-worker-w3-83755a-openshift-default-c86f9565c-x9v2k\" (UID: \"4bcec813-d864-450b-b606-d97e7c1a1948\") " pod="openshift-ingress/e2e-worker-w3-83755a-openshift-default-c86f9565c-x9v2k" Nov 03 01:42:35.345254 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:42:35.344897 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4bcec813-d864-450b-b606-d97e7c1a1948-workload-socket\") pod \"e2e-worker-w3-83755a-openshift-default-c86f9565c-x9v2k\" (UID: \"4bcec813-d864-450b-b606-d97e7c1a1948\") " pod="openshift-ingress/e2e-worker-w3-83755a-openshift-default-c86f9565c-x9v2k" Nov 03 01:42:35.345254 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:42:35.345021 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4bcec813-d864-450b-b606-d97e7c1a1948-istiod-ca-cert\") pod \"e2e-worker-w3-83755a-openshift-default-c86f9565c-x9v2k\" (UID: \"4bcec813-d864-450b-b606-d97e7c1a1948\") " pod="openshift-ingress/e2e-worker-w3-83755a-openshift-default-c86f9565c-x9v2k" Nov 03 01:42:35.345254 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:42:35.345106 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bdedc3dc-7ab3-4c41-84ea-e1443fd30898-workload-certs\") pod \"e2e-worker-w3-83755a-openshift-default-c86f9565c-vm5n9\" (UID: \"bdedc3dc-7ab3-4c41-84ea-e1443fd30898\") " pod="openshift-ingress/e2e-worker-w3-83755a-openshift-default-c86f9565c-vm5n9" Nov 03 01:42:35.345254 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:42:35.345143 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bdedc3dc-7ab3-4c41-84ea-e1443fd30898-istio-data\") pod \"e2e-worker-w3-83755a-openshift-default-c86f9565c-vm5n9\" (UID: \"bdedc3dc-7ab3-4c41-84ea-e1443fd30898\") " pod="openshift-ingress/e2e-worker-w3-83755a-openshift-default-c86f9565c-vm5n9" Nov 03 01:42:35.345254 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:42:35.345197 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bdedc3dc-7ab3-4c41-84ea-e1443fd30898-credential-socket\") pod \"e2e-worker-w3-83755a-openshift-default-c86f9565c-vm5n9\" (UID: \"bdedc3dc-7ab3-4c41-84ea-e1443fd30898\") " pod="openshift-ingress/e2e-worker-w3-83755a-openshift-default-c86f9565c-vm5n9" Nov 03 01:42:35.345865 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:42:35.345842 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bdedc3dc-7ab3-4c41-84ea-e1443fd30898-istiod-ca-cert\") pod \"e2e-worker-w3-83755a-openshift-default-c86f9565c-vm5n9\" (UID: \"bdedc3dc-7ab3-4c41-84ea-e1443fd30898\") " pod="openshift-ingress/e2e-worker-w3-83755a-openshift-default-c86f9565c-vm5n9" Nov 03 01:42:35.346590 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:42:35.346571 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4bcec813-d864-450b-b606-d97e7c1a1948-istio-envoy\") pod \"e2e-worker-w3-83755a-openshift-default-c86f9565c-x9v2k\" (UID: \"4bcec813-d864-450b-b606-d97e7c1a1948\") " pod="openshift-ingress/e2e-worker-w3-83755a-openshift-default-c86f9565c-x9v2k" Nov 03 01:42:35.346757 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:42:35.346743 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bdedc3dc-7ab3-4c41-84ea-e1443fd30898-istio-envoy\") pod \"e2e-worker-w3-83755a-openshift-default-c86f9565c-vm5n9\" (UID: \"bdedc3dc-7ab3-4c41-84ea-e1443fd30898\") " pod="openshift-ingress/e2e-worker-w3-83755a-openshift-default-c86f9565c-vm5n9" Nov 03 01:42:35.346938 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:42:35.346894 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bdedc3dc-7ab3-4c41-84ea-e1443fd30898-istio-podinfo\") pod \"e2e-worker-w3-83755a-openshift-default-c86f9565c-vm5n9\" (UID: \"bdedc3dc-7ab3-4c41-84ea-e1443fd30898\") " pod="openshift-ingress/e2e-worker-w3-83755a-openshift-default-c86f9565c-vm5n9" Nov 03 01:42:35.347464 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:42:35.347444 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4bcec813-d864-450b-b606-d97e7c1a1948-istio-podinfo\") pod \"e2e-worker-w3-83755a-openshift-default-c86f9565c-x9v2k\" (UID: \"4bcec813-d864-450b-b606-d97e7c1a1948\") " pod="openshift-ingress/e2e-worker-w3-83755a-openshift-default-c86f9565c-x9v2k" Nov 03 01:42:35.355299 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:42:35.355270 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bdedc3dc-7ab3-4c41-84ea-e1443fd30898-istio-token\") pod \"e2e-worker-w3-83755a-openshift-default-c86f9565c-vm5n9\" (UID: \"bdedc3dc-7ab3-4c41-84ea-e1443fd30898\") " pod="openshift-ingress/e2e-worker-w3-83755a-openshift-default-c86f9565c-vm5n9" Nov 03 01:42:35.355299 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:42:35.355281 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kwdqt\" (UniqueName: \"kubernetes.io/projected/4bcec813-d864-450b-b606-d97e7c1a1948-kube-api-access-kwdqt\") pod \"e2e-worker-w3-83755a-openshift-default-c86f9565c-x9v2k\" (UID: \"4bcec813-d864-450b-b606-d97e7c1a1948\") " pod="openshift-ingress/e2e-worker-w3-83755a-openshift-default-c86f9565c-x9v2k" Nov 03 01:42:35.357167 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:42:35.357141 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4bcec813-d864-450b-b606-d97e7c1a1948-istio-token\") pod \"e2e-worker-w3-83755a-openshift-default-c86f9565c-x9v2k\" (UID: \"4bcec813-d864-450b-b606-d97e7c1a1948\") " pod="openshift-ingress/e2e-worker-w3-83755a-openshift-default-c86f9565c-x9v2k" Nov 03 01:42:35.357430 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:42:35.357406 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ggmpl\" (UniqueName: \"kubernetes.io/projected/bdedc3dc-7ab3-4c41-84ea-e1443fd30898-kube-api-access-ggmpl\") pod \"e2e-worker-w3-83755a-openshift-default-c86f9565c-vm5n9\" (UID: \"bdedc3dc-7ab3-4c41-84ea-e1443fd30898\") " pod="openshift-ingress/e2e-worker-w3-83755a-openshift-default-c86f9565c-vm5n9" Nov 03 01:42:35.378138 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:42:35.378104 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:42:35.445152 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:42:35.445125 4939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d2dabe94-37c9-4a3f-bf9a-15e4f53e05d0-kube-api-access-5tj7h\" (UniqueName: \"kubernetes.io/projected/d2dabe94-37c9-4a3f-bf9a-15e4f53e05d0-kube-api-access-5tj7h\") pod \"d2dabe94-37c9-4a3f-bf9a-15e4f53e05d0\" (UID: \"d2dabe94-37c9-4a3f-bf9a-15e4f53e05d0\") " Nov 03 01:42:35.445264 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:42:35.445173 4939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d2dabe94-37c9-4a3f-bf9a-15e4f53e05d0-istio-token\" (UniqueName: \"kubernetes.io/projected/d2dabe94-37c9-4a3f-bf9a-15e4f53e05d0-istio-token\") pod \"d2dabe94-37c9-4a3f-bf9a-15e4f53e05d0\" (UID: \"d2dabe94-37c9-4a3f-bf9a-15e4f53e05d0\") " Nov 03 01:42:35.445369 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:42:35.445348 4939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d2dabe94-37c9-4a3f-bf9a-15e4f53e05d0-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d2dabe94-37c9-4a3f-bf9a-15e4f53e05d0-istio-data\") pod \"d2dabe94-37c9-4a3f-bf9a-15e4f53e05d0\" (UID: \"d2dabe94-37c9-4a3f-bf9a-15e4f53e05d0\") " Nov 03 01:42:35.445423 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:42:35.445401 4939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/d2dabe94-37c9-4a3f-bf9a-15e4f53e05d0-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d2dabe94-37c9-4a3f-bf9a-15e4f53e05d0-istio-podinfo\") pod \"d2dabe94-37c9-4a3f-bf9a-15e4f53e05d0\" (UID: \"d2dabe94-37c9-4a3f-bf9a-15e4f53e05d0\") " Nov 03 01:42:35.445471 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:42:35.445453 4939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d2dabe94-37c9-4a3f-bf9a-15e4f53e05d0-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d2dabe94-37c9-4a3f-bf9a-15e4f53e05d0-istiod-ca-cert\") pod \"d2dabe94-37c9-4a3f-bf9a-15e4f53e05d0\" (UID: \"d2dabe94-37c9-4a3f-bf9a-15e4f53e05d0\") " Nov 03 01:42:35.445535 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:42:35.445482 4939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d2dabe94-37c9-4a3f-bf9a-15e4f53e05d0-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d2dabe94-37c9-4a3f-bf9a-15e4f53e05d0-workload-socket\") pod \"d2dabe94-37c9-4a3f-bf9a-15e4f53e05d0\" (UID: \"d2dabe94-37c9-4a3f-bf9a-15e4f53e05d0\") " Nov 03 01:42:35.445535 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:42:35.445524 4939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d2dabe94-37c9-4a3f-bf9a-15e4f53e05d0-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d2dabe94-37c9-4a3f-bf9a-15e4f53e05d0-workload-certs\") pod \"d2dabe94-37c9-4a3f-bf9a-15e4f53e05d0\" (UID: \"d2dabe94-37c9-4a3f-bf9a-15e4f53e05d0\") " Nov 03 01:42:35.445662 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:42:35.445539 4939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d2dabe94-37c9-4a3f-bf9a-15e4f53e05d0-istio-data" pod "d2dabe94-37c9-4a3f-bf9a-15e4f53e05d0" (UID: "d2dabe94-37c9-4a3f-bf9a-15e4f53e05d0"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 01:42:35.445662 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:42:35.445578 4939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d2dabe94-37c9-4a3f-bf9a-15e4f53e05d0-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d2dabe94-37c9-4a3f-bf9a-15e4f53e05d0-credential-socket\") pod \"d2dabe94-37c9-4a3f-bf9a-15e4f53e05d0\" (UID: \"d2dabe94-37c9-4a3f-bf9a-15e4f53e05d0\") " Nov 03 01:42:35.445770 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:42:35.445679 4939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d2dabe94-37c9-4a3f-bf9a-15e4f53e05d0-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d2dabe94-37c9-4a3f-bf9a-15e4f53e05d0-istio-envoy\") pod \"d2dabe94-37c9-4a3f-bf9a-15e4f53e05d0\" (UID: \"d2dabe94-37c9-4a3f-bf9a-15e4f53e05d0\") " Nov 03 01:42:35.445770 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:42:35.445755 4939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d2dabe94-37c9-4a3f-bf9a-15e4f53e05d0-workload-socket" pod "d2dabe94-37c9-4a3f-bf9a-15e4f53e05d0" (UID: "d2dabe94-37c9-4a3f-bf9a-15e4f53e05d0"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 01:42:35.445902 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:42:35.445824 4939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d2dabe94-37c9-4a3f-bf9a-15e4f53e05d0-workload-certs" pod "d2dabe94-37c9-4a3f-bf9a-15e4f53e05d0" (UID: "d2dabe94-37c9-4a3f-bf9a-15e4f53e05d0"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 01:42:35.445902 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:42:35.445886 4939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d2dabe94-37c9-4a3f-bf9a-15e4f53e05d0-credential-socket" pod "d2dabe94-37c9-4a3f-bf9a-15e4f53e05d0" (UID: "d2dabe94-37c9-4a3f-bf9a-15e4f53e05d0"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 01:42:35.446096 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:42:35.445976 4939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d2dabe94-37c9-4a3f-bf9a-15e4f53e05d0-istiod-ca-cert" pod "d2dabe94-37c9-4a3f-bf9a-15e4f53e05d0" (UID: "d2dabe94-37c9-4a3f-bf9a-15e4f53e05d0"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 01:42:35.446208 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:42:35.446186 4939 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d2dabe94-37c9-4a3f-bf9a-15e4f53e05d0-istio-data\") on node \"ip-10-0-140-107.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:42:35.446274 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:42:35.446211 4939 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d2dabe94-37c9-4a3f-bf9a-15e4f53e05d0-istiod-ca-cert\") on node \"ip-10-0-140-107.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:42:35.446274 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:42:35.446228 4939 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d2dabe94-37c9-4a3f-bf9a-15e4f53e05d0-workload-socket\") on node \"ip-10-0-140-107.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:42:35.446274 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:42:35.446243 4939 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d2dabe94-37c9-4a3f-bf9a-15e4f53e05d0-workload-certs\") on node \"ip-10-0-140-107.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:42:35.446274 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:42:35.446258 4939 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d2dabe94-37c9-4a3f-bf9a-15e4f53e05d0-credential-socket\") on node \"ip-10-0-140-107.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:42:35.447966 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:42:35.447943 4939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d2dabe94-37c9-4a3f-bf9a-15e4f53e05d0-istio-envoy" pod "d2dabe94-37c9-4a3f-bf9a-15e4f53e05d0" (UID: "d2dabe94-37c9-4a3f-bf9a-15e4f53e05d0"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 01:42:35.447966 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:42:35.447956 4939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d2dabe94-37c9-4a3f-bf9a-15e4f53e05d0-kube-api-access-5tj7h" pod "d2dabe94-37c9-4a3f-bf9a-15e4f53e05d0" (UID: "d2dabe94-37c9-4a3f-bf9a-15e4f53e05d0"). InnerVolumeSpecName "kube-api-access-5tj7h". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 01:42:35.448150 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:42:35.448074 4939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d2dabe94-37c9-4a3f-bf9a-15e4f53e05d0-istio-token" pod "d2dabe94-37c9-4a3f-bf9a-15e4f53e05d0" (UID: "d2dabe94-37c9-4a3f-bf9a-15e4f53e05d0"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 01:42:35.448345 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:42:35.448322 4939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/d2dabe94-37c9-4a3f-bf9a-15e4f53e05d0-istio-podinfo" pod "d2dabe94-37c9-4a3f-bf9a-15e4f53e05d0" (UID: "d2dabe94-37c9-4a3f-bf9a-15e4f53e05d0"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 01:42:35.458208 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:42:35.458182 4939 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-83755a-openshift-default-c86f9565c-x9v2k" Nov 03 01:42:35.474301 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:42:35.474274 4939 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-83755a-openshift-default-c86f9565c-vm5n9" Nov 03 01:42:35.547388 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:42:35.547356 4939 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d2dabe94-37c9-4a3f-bf9a-15e4f53e05d0-istio-token\") on node \"ip-10-0-140-107.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:42:35.547516 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:42:35.547395 4939 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d2dabe94-37c9-4a3f-bf9a-15e4f53e05d0-istio-podinfo\") on node \"ip-10-0-140-107.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:42:35.547516 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:42:35.547415 4939 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d2dabe94-37c9-4a3f-bf9a-15e4f53e05d0-istio-envoy\") on node \"ip-10-0-140-107.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:42:35.547516 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:42:35.547433 4939 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5tj7h\" (UniqueName: \"kubernetes.io/projected/d2dabe94-37c9-4a3f-bf9a-15e4f53e05d0-kube-api-access-5tj7h\") on node \"ip-10-0-140-107.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:42:35.625052 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:42:35.624958 4939 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-83755a-openshift-default-c86f9565c-x9v2k"] Nov 03 01:42:35.626076 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:42:35.626045 4939 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-83755a-openshift-default-c86f9565c-vm5n9"] Nov 03 01:42:35.742971 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:42:35.741582 4939 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w1-2729f2-openshift-default-9fb97b4fc-6jhb8" Nov 03 01:42:35.755903 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:42:35.755867 4939 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-83755a-openshift-default-c86f9565c-x9v2k"] Nov 03 01:42:35.770090 ip-10-0-140-107 kubenswrapper[4939]: W1103 01:42:35.769272 4939 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4bcec813_d864_450b_b606_d97e7c1a1948.slice/crio-90688c930cc663dae862979a4759d2bbfcf71aab1def82e949454b406d5df34b WatchSource:0}: Error finding container 90688c930cc663dae862979a4759d2bbfcf71aab1def82e949454b406d5df34b: Status 404 returned error can't find the container with id 90688c930cc663dae862979a4759d2bbfcf71aab1def82e949454b406d5df34b Nov 03 01:42:35.774348 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:42:35.774303 4939 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 01:42:35.774472 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:42:35.774418 4939 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 01:42:35.774472 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:42:35.774452 4939 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 01:42:35.777045 ip-10-0-140-107 kubenswrapper[4939]: W1103 01:42:35.777013 4939 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbdedc3dc_7ab3_4c41_84ea_e1443fd30898.slice/crio-c66675a73f2caad6b3fb8ea671a735edbbbe2ac24f961f71cd5030d0d34c5d43 WatchSource:0}: Error finding container c66675a73f2caad6b3fb8ea671a735edbbbe2ac24f961f71cd5030d0d34c5d43: Status 404 returned error can't find the container with id c66675a73f2caad6b3fb8ea671a735edbbbe2ac24f961f71cd5030d0d34c5d43 Nov 03 01:42:35.783885 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:42:35.783749 4939 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-83755a-openshift-default-c86f9565c-vm5n9"] Nov 03 01:42:35.785230 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:42:35.785196 4939 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 01:42:35.785428 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:42:35.785404 4939 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 01:42:35.785547 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:42:35.785532 4939 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 01:42:35.850947 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:42:35.850889 4939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/61c402c5-4efa-4821-b51e-da74f4d9eff7-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/61c402c5-4efa-4821-b51e-da74f4d9eff7-workload-socket\") pod \"61c402c5-4efa-4821-b51e-da74f4d9eff7\" (UID: \"61c402c5-4efa-4821-b51e-da74f4d9eff7\") " Nov 03 01:42:35.851490 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:42:35.850975 4939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/61c402c5-4efa-4821-b51e-da74f4d9eff7-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/61c402c5-4efa-4821-b51e-da74f4d9eff7-istio-data\") pod \"61c402c5-4efa-4821-b51e-da74f4d9eff7\" (UID: \"61c402c5-4efa-4821-b51e-da74f4d9eff7\") " Nov 03 01:42:35.851490 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:42:35.851005 4939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/61c402c5-4efa-4821-b51e-da74f4d9eff7-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/61c402c5-4efa-4821-b51e-da74f4d9eff7-istio-envoy\") pod \"61c402c5-4efa-4821-b51e-da74f4d9eff7\" (UID: \"61c402c5-4efa-4821-b51e-da74f4d9eff7\") " Nov 03 01:42:35.851490 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:42:35.851031 4939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/61c402c5-4efa-4821-b51e-da74f4d9eff7-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/61c402c5-4efa-4821-b51e-da74f4d9eff7-istiod-ca-cert\") pod \"61c402c5-4efa-4821-b51e-da74f4d9eff7\" (UID: \"61c402c5-4efa-4821-b51e-da74f4d9eff7\") " Nov 03 01:42:35.851490 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:42:35.851083 4939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/61c402c5-4efa-4821-b51e-da74f4d9eff7-kube-api-access-qqhj4\" (UniqueName: \"kubernetes.io/projected/61c402c5-4efa-4821-b51e-da74f4d9eff7-kube-api-access-qqhj4\") pod \"61c402c5-4efa-4821-b51e-da74f4d9eff7\" (UID: \"61c402c5-4efa-4821-b51e-da74f4d9eff7\") " Nov 03 01:42:35.851490 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:42:35.851122 4939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/61c402c5-4efa-4821-b51e-da74f4d9eff7-workload-socket" pod "61c402c5-4efa-4821-b51e-da74f4d9eff7" (UID: "61c402c5-4efa-4821-b51e-da74f4d9eff7"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 01:42:35.851490 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:42:35.851213 4939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/61c402c5-4efa-4821-b51e-da74f4d9eff7-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/61c402c5-4efa-4821-b51e-da74f4d9eff7-istio-podinfo\") pod \"61c402c5-4efa-4821-b51e-da74f4d9eff7\" (UID: \"61c402c5-4efa-4821-b51e-da74f4d9eff7\") " Nov 03 01:42:35.851490 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:42:35.851278 4939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/61c402c5-4efa-4821-b51e-da74f4d9eff7-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/61c402c5-4efa-4821-b51e-da74f4d9eff7-workload-certs\") pod \"61c402c5-4efa-4821-b51e-da74f4d9eff7\" (UID: \"61c402c5-4efa-4821-b51e-da74f4d9eff7\") " Nov 03 01:42:35.851490 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:42:35.851292 4939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/61c402c5-4efa-4821-b51e-da74f4d9eff7-istio-data" pod "61c402c5-4efa-4821-b51e-da74f4d9eff7" (UID: "61c402c5-4efa-4821-b51e-da74f4d9eff7"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 01:42:35.851490 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:42:35.851318 4939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/61c402c5-4efa-4821-b51e-da74f4d9eff7-istio-token\" (UniqueName: \"kubernetes.io/projected/61c402c5-4efa-4821-b51e-da74f4d9eff7-istio-token\") pod \"61c402c5-4efa-4821-b51e-da74f4d9eff7\" (UID: \"61c402c5-4efa-4821-b51e-da74f4d9eff7\") " Nov 03 01:42:35.851490 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:42:35.851401 4939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/61c402c5-4efa-4821-b51e-da74f4d9eff7-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/61c402c5-4efa-4821-b51e-da74f4d9eff7-credential-socket\") pod \"61c402c5-4efa-4821-b51e-da74f4d9eff7\" (UID: \"61c402c5-4efa-4821-b51e-da74f4d9eff7\") " Nov 03 01:42:35.852285 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:42:35.851541 4939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/61c402c5-4efa-4821-b51e-da74f4d9eff7-workload-certs" pod "61c402c5-4efa-4821-b51e-da74f4d9eff7" (UID: "61c402c5-4efa-4821-b51e-da74f4d9eff7"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 01:42:35.852285 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:42:35.851595 4939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/61c402c5-4efa-4821-b51e-da74f4d9eff7-istiod-ca-cert" pod "61c402c5-4efa-4821-b51e-da74f4d9eff7" (UID: "61c402c5-4efa-4821-b51e-da74f4d9eff7"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 01:42:35.852285 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:42:35.851812 4939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/61c402c5-4efa-4821-b51e-da74f4d9eff7-credential-socket" pod "61c402c5-4efa-4821-b51e-da74f4d9eff7" (UID: "61c402c5-4efa-4821-b51e-da74f4d9eff7"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 01:42:35.852285 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:42:35.851976 4939 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/61c402c5-4efa-4821-b51e-da74f4d9eff7-credential-socket\") on node \"ip-10-0-140-107.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:42:35.852285 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:42:35.851996 4939 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/61c402c5-4efa-4821-b51e-da74f4d9eff7-workload-socket\") on node \"ip-10-0-140-107.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:42:35.852285 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:42:35.852013 4939 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/61c402c5-4efa-4821-b51e-da74f4d9eff7-istio-data\") on node \"ip-10-0-140-107.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:42:35.852285 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:42:35.852027 4939 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/61c402c5-4efa-4821-b51e-da74f4d9eff7-istiod-ca-cert\") on node \"ip-10-0-140-107.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:42:35.852285 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:42:35.852043 4939 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/61c402c5-4efa-4821-b51e-da74f4d9eff7-workload-certs\") on node \"ip-10-0-140-107.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:42:35.855147 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:42:35.855122 4939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/61c402c5-4efa-4821-b51e-da74f4d9eff7-kube-api-access-qqhj4" pod "61c402c5-4efa-4821-b51e-da74f4d9eff7" (UID: "61c402c5-4efa-4821-b51e-da74f4d9eff7"). InnerVolumeSpecName "kube-api-access-qqhj4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 01:42:35.855383 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:42:35.855364 4939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/61c402c5-4efa-4821-b51e-da74f4d9eff7-istio-token" pod "61c402c5-4efa-4821-b51e-da74f4d9eff7" (UID: "61c402c5-4efa-4821-b51e-da74f4d9eff7"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 01:42:35.855587 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:42:35.855571 4939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/61c402c5-4efa-4821-b51e-da74f4d9eff7-istio-envoy" pod "61c402c5-4efa-4821-b51e-da74f4d9eff7" (UID: "61c402c5-4efa-4821-b51e-da74f4d9eff7"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 01:42:35.855658 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:42:35.855586 4939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/61c402c5-4efa-4821-b51e-da74f4d9eff7-istio-podinfo" pod "61c402c5-4efa-4821-b51e-da74f4d9eff7" (UID: "61c402c5-4efa-4821-b51e-da74f4d9eff7"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 01:42:35.887478 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:42:35.887376 4939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-83755a-openshift-default-c86f9565c-x9v2k" event={"ID":"4bcec813-d864-450b-b606-d97e7c1a1948","Type":"ContainerStarted","Data":"481bd4b694627888a206f229ddb618ac62b2aaa0f58179ed105f18b8079b8456"} Nov 03 01:42:35.887478 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:42:35.887417 4939 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-83755a-openshift-default-c86f9565c-x9v2k" podUID="4bcec813-d864-450b-b606-d97e7c1a1948" containerName="istio-proxy" containerID="cri-o://481bd4b694627888a206f229ddb618ac62b2aaa0f58179ed105f18b8079b8456" gracePeriod=30 Nov 03 01:42:35.887478 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:42:35.887433 4939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-83755a-openshift-default-c86f9565c-x9v2k" event={"ID":"4bcec813-d864-450b-b606-d97e7c1a1948","Type":"ContainerStarted","Data":"90688c930cc663dae862979a4759d2bbfcf71aab1def82e949454b406d5df34b"} Nov 03 01:42:35.891274 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:42:35.891241 4939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-83755a-openshift-default-c86f9565c-vm5n9" event={"ID":"bdedc3dc-7ab3-4c41-84ea-e1443fd30898","Type":"ContainerStarted","Data":"5fac3ac8ceffd3f03776f98efad40a11602b60a9f0c1ab836c3ba9793d118330"} Nov 03 01:42:35.891366 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:42:35.891304 4939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-83755a-openshift-default-c86f9565c-vm5n9" event={"ID":"bdedc3dc-7ab3-4c41-84ea-e1443fd30898","Type":"ContainerStarted","Data":"c66675a73f2caad6b3fb8ea671a735edbbbe2ac24f961f71cd5030d0d34c5d43"} Nov 03 01:42:35.891476 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:42:35.891444 4939 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-83755a-openshift-default-c86f9565c-vm5n9" podUID="bdedc3dc-7ab3-4c41-84ea-e1443fd30898" containerName="istio-proxy" containerID="cri-o://5fac3ac8ceffd3f03776f98efad40a11602b60a9f0c1ab836c3ba9793d118330" gracePeriod=30 Nov 03 01:42:35.893887 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:42:35.893808 4939 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-650be0-openshift-default-5c7c5fb7d9-84qlq" Nov 03 01:42:35.893887 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:42:35.893842 4939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-650be0-openshift-default-5c7c5fb7d9-84qlq" event={"ID":"d2dabe94-37c9-4a3f-bf9a-15e4f53e05d0","Type":"ContainerDied","Data":"59229abf1249ba042d9b842f2814fd31ff087502d8a464bb514a33b0b86121e4"} Nov 03 01:42:35.894058 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:42:35.893899 4939 scope.go:122] "RemoveContainer" containerID="463608114ded4c17d374f75c240e2b2f1556c617d103054903616e64d1a01bc6" Nov 03 01:42:35.896073 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:42:35.896050 4939 generic.go:356] "Generic (PLEG): container finished" podID="61c402c5-4efa-4821-b51e-da74f4d9eff7" containerID="26b79eddeeac2d4fc4ade04398e94fa28e6ddcc8d46013647c68074fbdad8d12" exitCode=0 Nov 03 01:42:35.896180 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:42:35.896091 4939 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w1-2729f2-openshift-default-9fb97b4fc-6jhb8" Nov 03 01:42:35.896180 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:42:35.896127 4939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w1-2729f2-openshift-default-9fb97b4fc-6jhb8" event={"ID":"61c402c5-4efa-4821-b51e-da74f4d9eff7","Type":"ContainerDied","Data":"26b79eddeeac2d4fc4ade04398e94fa28e6ddcc8d46013647c68074fbdad8d12"} Nov 03 01:42:35.896367 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:42:35.896180 4939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w1-2729f2-openshift-default-9fb97b4fc-6jhb8" event={"ID":"61c402c5-4efa-4821-b51e-da74f4d9eff7","Type":"ContainerDied","Data":"6e47f1b506e288518698d95572ed6bb934e02c5482862931a8ec32306cd70f91"} Nov 03 01:42:35.904180 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:42:35.904161 4939 scope.go:122] "RemoveContainer" containerID="26b79eddeeac2d4fc4ade04398e94fa28e6ddcc8d46013647c68074fbdad8d12" Nov 03 01:42:35.907518 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:42:35.907477 4939 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-83755a-openshift-default-c86f9565c-x9v2k" podStartSLOduration=0.907465975 podStartE2EDuration="907.465975ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 01:42:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 01:42:35.906041994 +0000 UTC m=+3704.170376939" watchObservedRunningTime="2026-11-03 01:42:35.907465975 +0000 UTC m=+3704.171800920" Nov 03 01:42:35.920098 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:42:35.920077 4939 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w1-2729f2-openshift-default-9fb97b4fc-6jhb8"] Nov 03 01:42:35.923728 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:42:35.923706 4939 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w1-2729f2-openshift-default-9fb97b4fc-6jhb8"] Nov 03 01:42:35.934039 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:42:35.934016 4939 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-650be0-openshift-default-5c7c5fb7d9-84qlq"] Nov 03 01:42:35.938152 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:42:35.938129 4939 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-650be0-openshift-default-5c7c5fb7d9-84qlq"] Nov 03 01:42:35.953634 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:42:35.953604 4939 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/61c402c5-4efa-4821-b51e-da74f4d9eff7-istio-envoy\") on node \"ip-10-0-140-107.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:42:35.953740 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:42:35.953636 4939 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qqhj4\" (UniqueName: \"kubernetes.io/projected/61c402c5-4efa-4821-b51e-da74f4d9eff7-kube-api-access-qqhj4\") on node \"ip-10-0-140-107.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:42:35.953740 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:42:35.953652 4939 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/61c402c5-4efa-4821-b51e-da74f4d9eff7-istio-podinfo\") on node \"ip-10-0-140-107.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:42:35.953740 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:42:35.953680 4939 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/61c402c5-4efa-4821-b51e-da74f4d9eff7-istio-token\") on node \"ip-10-0-140-107.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:42:35.955061 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:42:35.955016 4939 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-83755a-openshift-default-c86f9565c-vm5n9" podStartSLOduration=0.95500028 podStartE2EDuration="955.00028ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 01:42:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 01:42:35.953481262 +0000 UTC m=+3704.217816206" watchObservedRunningTime="2026-11-03 01:42:35.95500028 +0000 UTC m=+3704.219335227" Nov 03 01:42:35.999404 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:42:35.999376 4939 scope.go:122] "RemoveContainer" containerID="26b79eddeeac2d4fc4ade04398e94fa28e6ddcc8d46013647c68074fbdad8d12" Nov 03 01:42:35.999759 ip-10-0-140-107 kubenswrapper[4939]: E1103 01:42:35.999738 4939 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"26b79eddeeac2d4fc4ade04398e94fa28e6ddcc8d46013647c68074fbdad8d12\": container with ID starting with 26b79eddeeac2d4fc4ade04398e94fa28e6ddcc8d46013647c68074fbdad8d12 not found: ID does not exist" containerID="26b79eddeeac2d4fc4ade04398e94fa28e6ddcc8d46013647c68074fbdad8d12" Nov 03 01:42:35.999814 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:42:35.999769 4939 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"26b79eddeeac2d4fc4ade04398e94fa28e6ddcc8d46013647c68074fbdad8d12"} err="failed to get container status \"26b79eddeeac2d4fc4ade04398e94fa28e6ddcc8d46013647c68074fbdad8d12\": rpc error: code = NotFound desc = could not find container \"26b79eddeeac2d4fc4ade04398e94fa28e6ddcc8d46013647c68074fbdad8d12\": container with ID starting with 26b79eddeeac2d4fc4ade04398e94fa28e6ddcc8d46013647c68074fbdad8d12 not found: ID does not exist" Nov 03 01:42:36.134594 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:42:36.134564 4939 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-83755a-openshift-default-c86f9565c-x9v2k" Nov 03 01:42:36.156051 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:42:36.155967 4939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4bcec813-d864-450b-b606-d97e7c1a1948-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4bcec813-d864-450b-b606-d97e7c1a1948-credential-socket\") pod \"4bcec813-d864-450b-b606-d97e7c1a1948\" (UID: \"4bcec813-d864-450b-b606-d97e7c1a1948\") " Nov 03 01:42:36.156248 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:42:36.156218 4939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4bcec813-d864-450b-b606-d97e7c1a1948-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4bcec813-d864-450b-b606-d97e7c1a1948-istio-data\") pod \"4bcec813-d864-450b-b606-d97e7c1a1948\" (UID: \"4bcec813-d864-450b-b606-d97e7c1a1948\") " Nov 03 01:42:36.156422 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:42:36.156257 4939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4bcec813-d864-450b-b606-d97e7c1a1948-credential-socket" pod "4bcec813-d864-450b-b606-d97e7c1a1948" (UID: "4bcec813-d864-450b-b606-d97e7c1a1948"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 01:42:36.156422 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:42:36.156344 4939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4bcec813-d864-450b-b606-d97e7c1a1948-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4bcec813-d864-450b-b606-d97e7c1a1948-workload-certs\") pod \"4bcec813-d864-450b-b606-d97e7c1a1948\" (UID: \"4bcec813-d864-450b-b606-d97e7c1a1948\") " Nov 03 01:42:36.156422 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:42:36.156387 4939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4bcec813-d864-450b-b606-d97e7c1a1948-kube-api-access-kwdqt\" (UniqueName: \"kubernetes.io/projected/4bcec813-d864-450b-b606-d97e7c1a1948-kube-api-access-kwdqt\") pod \"4bcec813-d864-450b-b606-d97e7c1a1948\" (UID: \"4bcec813-d864-450b-b606-d97e7c1a1948\") " Nov 03 01:42:36.156716 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:42:36.156423 4939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4bcec813-d864-450b-b606-d97e7c1a1948-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4bcec813-d864-450b-b606-d97e7c1a1948-workload-socket\") pod \"4bcec813-d864-450b-b606-d97e7c1a1948\" (UID: \"4bcec813-d864-450b-b606-d97e7c1a1948\") " Nov 03 01:42:36.156716 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:42:36.156463 4939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4bcec813-d864-450b-b606-d97e7c1a1948-istio-token\" (UniqueName: \"kubernetes.io/projected/4bcec813-d864-450b-b606-d97e7c1a1948-istio-token\") pod \"4bcec813-d864-450b-b606-d97e7c1a1948\" (UID: \"4bcec813-d864-450b-b606-d97e7c1a1948\") " Nov 03 01:42:36.156716 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:42:36.156522 4939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4bcec813-d864-450b-b606-d97e7c1a1948-istio-data" pod "4bcec813-d864-450b-b606-d97e7c1a1948" (UID: "4bcec813-d864-450b-b606-d97e7c1a1948"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 01:42:36.156716 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:42:36.156545 4939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/4bcec813-d864-450b-b606-d97e7c1a1948-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4bcec813-d864-450b-b606-d97e7c1a1948-istio-podinfo\") pod \"4bcec813-d864-450b-b606-d97e7c1a1948\" (UID: \"4bcec813-d864-450b-b606-d97e7c1a1948\") " Nov 03 01:42:36.156716 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:42:36.156591 4939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4bcec813-d864-450b-b606-d97e7c1a1948-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4bcec813-d864-450b-b606-d97e7c1a1948-istiod-ca-cert\") pod \"4bcec813-d864-450b-b606-d97e7c1a1948\" (UID: \"4bcec813-d864-450b-b606-d97e7c1a1948\") " Nov 03 01:42:36.156716 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:42:36.156666 4939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4bcec813-d864-450b-b606-d97e7c1a1948-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4bcec813-d864-450b-b606-d97e7c1a1948-istio-envoy\") pod \"4bcec813-d864-450b-b606-d97e7c1a1948\" (UID: \"4bcec813-d864-450b-b606-d97e7c1a1948\") " Nov 03 01:42:36.157306 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:42:36.157091 4939 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4bcec813-d864-450b-b606-d97e7c1a1948-credential-socket\") on node \"ip-10-0-140-107.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:42:36.157306 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:42:36.157113 4939 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4bcec813-d864-450b-b606-d97e7c1a1948-istio-data\") on node \"ip-10-0-140-107.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:42:36.157814 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:42:36.157788 4939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4bcec813-d864-450b-b606-d97e7c1a1948-workload-socket" pod "4bcec813-d864-450b-b606-d97e7c1a1948" (UID: "4bcec813-d864-450b-b606-d97e7c1a1948"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 01:42:36.157993 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:42:36.157873 4939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4bcec813-d864-450b-b606-d97e7c1a1948-istiod-ca-cert" pod "4bcec813-d864-450b-b606-d97e7c1a1948" (UID: "4bcec813-d864-450b-b606-d97e7c1a1948"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 01:42:36.157993 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:42:36.157932 4939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4bcec813-d864-450b-b606-d97e7c1a1948-workload-certs" pod "4bcec813-d864-450b-b606-d97e7c1a1948" (UID: "4bcec813-d864-450b-b606-d97e7c1a1948"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 01:42:36.159820 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:42:36.159786 4939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4bcec813-d864-450b-b606-d97e7c1a1948-istio-token" pod "4bcec813-d864-450b-b606-d97e7c1a1948" (UID: "4bcec813-d864-450b-b606-d97e7c1a1948"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 01:42:36.160226 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:42:36.160201 4939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4bcec813-d864-450b-b606-d97e7c1a1948-kube-api-access-kwdqt" pod "4bcec813-d864-450b-b606-d97e7c1a1948" (UID: "4bcec813-d864-450b-b606-d97e7c1a1948"). InnerVolumeSpecName "kube-api-access-kwdqt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 01:42:36.160226 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:42:36.160213 4939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4bcec813-d864-450b-b606-d97e7c1a1948-istio-envoy" pod "4bcec813-d864-450b-b606-d97e7c1a1948" (UID: "4bcec813-d864-450b-b606-d97e7c1a1948"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 01:42:36.162432 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:42:36.162411 4939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/4bcec813-d864-450b-b606-d97e7c1a1948-istio-podinfo" pod "4bcec813-d864-450b-b606-d97e7c1a1948" (UID: "4bcec813-d864-450b-b606-d97e7c1a1948"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 01:42:36.172053 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:42:36.172032 4939 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-83755a-openshift-default-c86f9565c-vm5n9" Nov 03 01:42:36.258147 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:42:36.258116 4939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bdedc3dc-7ab3-4c41-84ea-e1443fd30898-istio-token\" (UniqueName: \"kubernetes.io/projected/bdedc3dc-7ab3-4c41-84ea-e1443fd30898-istio-token\") pod \"bdedc3dc-7ab3-4c41-84ea-e1443fd30898\" (UID: \"bdedc3dc-7ab3-4c41-84ea-e1443fd30898\") " Nov 03 01:42:36.258315 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:42:36.258180 4939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bdedc3dc-7ab3-4c41-84ea-e1443fd30898-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bdedc3dc-7ab3-4c41-84ea-e1443fd30898-workload-certs\") pod \"bdedc3dc-7ab3-4c41-84ea-e1443fd30898\" (UID: \"bdedc3dc-7ab3-4c41-84ea-e1443fd30898\") " Nov 03 01:42:36.258315 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:42:36.258215 4939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bdedc3dc-7ab3-4c41-84ea-e1443fd30898-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bdedc3dc-7ab3-4c41-84ea-e1443fd30898-credential-socket\") pod \"bdedc3dc-7ab3-4c41-84ea-e1443fd30898\" (UID: \"bdedc3dc-7ab3-4c41-84ea-e1443fd30898\") " Nov 03 01:42:36.258315 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:42:36.258283 4939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bdedc3dc-7ab3-4c41-84ea-e1443fd30898-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bdedc3dc-7ab3-4c41-84ea-e1443fd30898-workload-socket\") pod \"bdedc3dc-7ab3-4c41-84ea-e1443fd30898\" (UID: \"bdedc3dc-7ab3-4c41-84ea-e1443fd30898\") " Nov 03 01:42:36.258495 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:42:36.258317 4939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bdedc3dc-7ab3-4c41-84ea-e1443fd30898-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bdedc3dc-7ab3-4c41-84ea-e1443fd30898-istiod-ca-cert\") pod \"bdedc3dc-7ab3-4c41-84ea-e1443fd30898\" (UID: \"bdedc3dc-7ab3-4c41-84ea-e1443fd30898\") " Nov 03 01:42:36.258495 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:42:36.258379 4939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/bdedc3dc-7ab3-4c41-84ea-e1443fd30898-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bdedc3dc-7ab3-4c41-84ea-e1443fd30898-istio-podinfo\") pod \"bdedc3dc-7ab3-4c41-84ea-e1443fd30898\" (UID: \"bdedc3dc-7ab3-4c41-84ea-e1443fd30898\") " Nov 03 01:42:36.258495 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:42:36.258410 4939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bdedc3dc-7ab3-4c41-84ea-e1443fd30898-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bdedc3dc-7ab3-4c41-84ea-e1443fd30898-istio-data\") pod \"bdedc3dc-7ab3-4c41-84ea-e1443fd30898\" (UID: \"bdedc3dc-7ab3-4c41-84ea-e1443fd30898\") " Nov 03 01:42:36.258665 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:42:36.258500 4939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bdedc3dc-7ab3-4c41-84ea-e1443fd30898-kube-api-access-ggmpl\" (UniqueName: \"kubernetes.io/projected/bdedc3dc-7ab3-4c41-84ea-e1443fd30898-kube-api-access-ggmpl\") pod \"bdedc3dc-7ab3-4c41-84ea-e1443fd30898\" (UID: \"bdedc3dc-7ab3-4c41-84ea-e1443fd30898\") " Nov 03 01:42:36.258665 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:42:36.258507 4939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bdedc3dc-7ab3-4c41-84ea-e1443fd30898-workload-certs" pod "bdedc3dc-7ab3-4c41-84ea-e1443fd30898" (UID: "bdedc3dc-7ab3-4c41-84ea-e1443fd30898"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 01:42:36.258665 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:42:36.258549 4939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bdedc3dc-7ab3-4c41-84ea-e1443fd30898-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bdedc3dc-7ab3-4c41-84ea-e1443fd30898-istio-envoy\") pod \"bdedc3dc-7ab3-4c41-84ea-e1443fd30898\" (UID: \"bdedc3dc-7ab3-4c41-84ea-e1443fd30898\") " Nov 03 01:42:36.258665 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:42:36.258575 4939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bdedc3dc-7ab3-4c41-84ea-e1443fd30898-credential-socket" pod "bdedc3dc-7ab3-4c41-84ea-e1443fd30898" (UID: "bdedc3dc-7ab3-4c41-84ea-e1443fd30898"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 01:42:36.258894 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:42:36.258713 4939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bdedc3dc-7ab3-4c41-84ea-e1443fd30898-istio-data" pod "bdedc3dc-7ab3-4c41-84ea-e1443fd30898" (UID: "bdedc3dc-7ab3-4c41-84ea-e1443fd30898"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 01:42:36.258894 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:42:36.258754 4939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bdedc3dc-7ab3-4c41-84ea-e1443fd30898-istiod-ca-cert" pod "bdedc3dc-7ab3-4c41-84ea-e1443fd30898" (UID: "bdedc3dc-7ab3-4c41-84ea-e1443fd30898"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 01:42:36.259025 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:42:36.259009 4939 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kwdqt\" (UniqueName: \"kubernetes.io/projected/4bcec813-d864-450b-b606-d97e7c1a1948-kube-api-access-kwdqt\") on node \"ip-10-0-140-107.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:42:36.259091 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:42:36.259034 4939 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4bcec813-d864-450b-b606-d97e7c1a1948-workload-socket\") on node \"ip-10-0-140-107.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:42:36.259091 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:42:36.259051 4939 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4bcec813-d864-450b-b606-d97e7c1a1948-istio-token\") on node \"ip-10-0-140-107.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:42:36.259091 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:42:36.259068 4939 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bdedc3dc-7ab3-4c41-84ea-e1443fd30898-workload-certs\") on node \"ip-10-0-140-107.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:42:36.259091 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:42:36.259084 4939 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4bcec813-d864-450b-b606-d97e7c1a1948-istio-podinfo\") on node \"ip-10-0-140-107.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:42:36.259303 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:42:36.259099 4939 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bdedc3dc-7ab3-4c41-84ea-e1443fd30898-credential-socket\") on node \"ip-10-0-140-107.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:42:36.259303 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:42:36.259115 4939 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4bcec813-d864-450b-b606-d97e7c1a1948-istiod-ca-cert\") on node \"ip-10-0-140-107.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:42:36.259303 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:42:36.259157 4939 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4bcec813-d864-450b-b606-d97e7c1a1948-istio-envoy\") on node \"ip-10-0-140-107.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:42:36.259303 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:42:36.259171 4939 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bdedc3dc-7ab3-4c41-84ea-e1443fd30898-istiod-ca-cert\") on node \"ip-10-0-140-107.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:42:36.259303 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:42:36.259185 4939 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bdedc3dc-7ab3-4c41-84ea-e1443fd30898-istio-data\") on node \"ip-10-0-140-107.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:42:36.259303 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:42:36.259192 4939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bdedc3dc-7ab3-4c41-84ea-e1443fd30898-workload-socket" pod "bdedc3dc-7ab3-4c41-84ea-e1443fd30898" (UID: "bdedc3dc-7ab3-4c41-84ea-e1443fd30898"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 01:42:36.259303 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:42:36.259198 4939 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4bcec813-d864-450b-b606-d97e7c1a1948-workload-certs\") on node \"ip-10-0-140-107.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:42:36.260551 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:42:36.260529 4939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/bdedc3dc-7ab3-4c41-84ea-e1443fd30898-istio-podinfo" pod "bdedc3dc-7ab3-4c41-84ea-e1443fd30898" (UID: "bdedc3dc-7ab3-4c41-84ea-e1443fd30898"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 01:42:36.260672 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:42:36.260595 4939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bdedc3dc-7ab3-4c41-84ea-e1443fd30898-kube-api-access-ggmpl" pod "bdedc3dc-7ab3-4c41-84ea-e1443fd30898" (UID: "bdedc3dc-7ab3-4c41-84ea-e1443fd30898"). InnerVolumeSpecName "kube-api-access-ggmpl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 01:42:36.260793 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:42:36.260780 4939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bdedc3dc-7ab3-4c41-84ea-e1443fd30898-istio-token" pod "bdedc3dc-7ab3-4c41-84ea-e1443fd30898" (UID: "bdedc3dc-7ab3-4c41-84ea-e1443fd30898"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 01:42:36.260876 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:42:36.260862 4939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bdedc3dc-7ab3-4c41-84ea-e1443fd30898-istio-envoy" pod "bdedc3dc-7ab3-4c41-84ea-e1443fd30898" (UID: "bdedc3dc-7ab3-4c41-84ea-e1443fd30898"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 01:42:36.360187 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:42:36.360156 4939 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bdedc3dc-7ab3-4c41-84ea-e1443fd30898-workload-socket\") on node \"ip-10-0-140-107.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:42:36.360187 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:42:36.360185 4939 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bdedc3dc-7ab3-4c41-84ea-e1443fd30898-istio-podinfo\") on node \"ip-10-0-140-107.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:42:36.360377 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:42:36.360200 4939 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ggmpl\" (UniqueName: \"kubernetes.io/projected/bdedc3dc-7ab3-4c41-84ea-e1443fd30898-kube-api-access-ggmpl\") on node \"ip-10-0-140-107.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:42:36.360377 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:42:36.360213 4939 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bdedc3dc-7ab3-4c41-84ea-e1443fd30898-istio-envoy\") on node \"ip-10-0-140-107.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:42:36.360377 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:42:36.360227 4939 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bdedc3dc-7ab3-4c41-84ea-e1443fd30898-istio-token\") on node \"ip-10-0-140-107.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:42:36.492667 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:42:36.492629 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:42:36.594536 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:42:36.594498 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:42:36.701692 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:42:36.701660 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:42:36.902523 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:42:36.902483 4939 generic.go:356] "Generic (PLEG): container finished" podID="4bcec813-d864-450b-b606-d97e7c1a1948" containerID="481bd4b694627888a206f229ddb618ac62b2aaa0f58179ed105f18b8079b8456" exitCode=2 Nov 03 01:42:36.902523 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:42:36.902518 4939 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-83755a-openshift-default-c86f9565c-x9v2k" Nov 03 01:42:36.903513 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:42:36.902558 4939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-83755a-openshift-default-c86f9565c-x9v2k" event={"ID":"4bcec813-d864-450b-b606-d97e7c1a1948","Type":"ContainerDied","Data":"481bd4b694627888a206f229ddb618ac62b2aaa0f58179ed105f18b8079b8456"} Nov 03 01:42:36.903513 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:42:36.902610 4939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-83755a-openshift-default-c86f9565c-x9v2k" event={"ID":"4bcec813-d864-450b-b606-d97e7c1a1948","Type":"ContainerDied","Data":"90688c930cc663dae862979a4759d2bbfcf71aab1def82e949454b406d5df34b"} Nov 03 01:42:36.903513 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:42:36.902640 4939 scope.go:122] "RemoveContainer" containerID="481bd4b694627888a206f229ddb618ac62b2aaa0f58179ed105f18b8079b8456" Nov 03 01:42:36.904230 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:42:36.904204 4939 generic.go:356] "Generic (PLEG): container finished" podID="bdedc3dc-7ab3-4c41-84ea-e1443fd30898" containerID="5fac3ac8ceffd3f03776f98efad40a11602b60a9f0c1ab836c3ba9793d118330" exitCode=2 Nov 03 01:42:36.904377 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:42:36.904358 4939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-83755a-openshift-default-c86f9565c-vm5n9" event={"ID":"bdedc3dc-7ab3-4c41-84ea-e1443fd30898","Type":"ContainerDied","Data":"5fac3ac8ceffd3f03776f98efad40a11602b60a9f0c1ab836c3ba9793d118330"} Nov 03 01:42:36.904433 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:42:36.904408 4939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-83755a-openshift-default-c86f9565c-vm5n9" event={"ID":"bdedc3dc-7ab3-4c41-84ea-e1443fd30898","Type":"ContainerDied","Data":"c66675a73f2caad6b3fb8ea671a735edbbbe2ac24f961f71cd5030d0d34c5d43"} Nov 03 01:42:36.904496 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:42:36.904477 4939 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-83755a-openshift-default-c86f9565c-vm5n9" Nov 03 01:42:36.912438 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:42:36.912422 4939 scope.go:122] "RemoveContainer" containerID="481bd4b694627888a206f229ddb618ac62b2aaa0f58179ed105f18b8079b8456" Nov 03 01:42:36.913018 ip-10-0-140-107 kubenswrapper[4939]: E1103 01:42:36.912989 4939 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"481bd4b694627888a206f229ddb618ac62b2aaa0f58179ed105f18b8079b8456\": container with ID starting with 481bd4b694627888a206f229ddb618ac62b2aaa0f58179ed105f18b8079b8456 not found: ID does not exist" containerID="481bd4b694627888a206f229ddb618ac62b2aaa0f58179ed105f18b8079b8456" Nov 03 01:42:36.913159 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:42:36.913028 4939 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"481bd4b694627888a206f229ddb618ac62b2aaa0f58179ed105f18b8079b8456"} err="failed to get container status \"481bd4b694627888a206f229ddb618ac62b2aaa0f58179ed105f18b8079b8456\": rpc error: code = NotFound desc = could not find container \"481bd4b694627888a206f229ddb618ac62b2aaa0f58179ed105f18b8079b8456\": container with ID starting with 481bd4b694627888a206f229ddb618ac62b2aaa0f58179ed105f18b8079b8456 not found: ID does not exist" Nov 03 01:42:36.913159 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:42:36.913053 4939 scope.go:122] "RemoveContainer" containerID="5fac3ac8ceffd3f03776f98efad40a11602b60a9f0c1ab836c3ba9793d118330" Nov 03 01:42:36.922524 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:42:36.922508 4939 scope.go:122] "RemoveContainer" containerID="5fac3ac8ceffd3f03776f98efad40a11602b60a9f0c1ab836c3ba9793d118330" Nov 03 01:42:36.922784 ip-10-0-140-107 kubenswrapper[4939]: E1103 01:42:36.922765 4939 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5fac3ac8ceffd3f03776f98efad40a11602b60a9f0c1ab836c3ba9793d118330\": container with ID starting with 5fac3ac8ceffd3f03776f98efad40a11602b60a9f0c1ab836c3ba9793d118330 not found: ID does not exist" containerID="5fac3ac8ceffd3f03776f98efad40a11602b60a9f0c1ab836c3ba9793d118330" Nov 03 01:42:36.922864 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:42:36.922796 4939 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5fac3ac8ceffd3f03776f98efad40a11602b60a9f0c1ab836c3ba9793d118330"} err="failed to get container status \"5fac3ac8ceffd3f03776f98efad40a11602b60a9f0c1ab836c3ba9793d118330\": rpc error: code = NotFound desc = could not find container \"5fac3ac8ceffd3f03776f98efad40a11602b60a9f0c1ab836c3ba9793d118330\": container with ID starting with 5fac3ac8ceffd3f03776f98efad40a11602b60a9f0c1ab836c3ba9793d118330 not found: ID does not exist" Nov 03 01:42:36.930332 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:42:36.930309 4939 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-83755a-openshift-default-c86f9565c-vm5n9"] Nov 03 01:42:36.934055 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:42:36.934032 4939 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-83755a-openshift-default-c86f9565c-vm5n9"] Nov 03 01:42:36.947432 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:42:36.947377 4939 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-83755a-openshift-default-c86f9565c-x9v2k"] Nov 03 01:42:36.951258 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:42:36.951238 4939 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-83755a-openshift-default-c86f9565c-x9v2k"] Nov 03 01:42:37.272354 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:42:37.272250 4939 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-2729f2-558f777dc-9gclz" podUID="c1e683ec-d1fc-4273-9384-1ffcc06a832a" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 03 01:42:37.272354 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:42:37.272318 4939 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-2729f2-565847c555-wqvj7" podUID="89acf0f1-ef07-4c99-afd3-de3d06335488" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 03 01:42:37.397862 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:42:37.397823 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:42:37.515923 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:42:37.515876 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:42:37.868417 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:42:37.868382 4939 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4bcec813-d864-450b-b606-d97e7c1a1948" path="/var/lib/kubelet/pods/4bcec813-d864-450b-b606-d97e7c1a1948/volumes" Nov 03 01:42:37.868872 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:42:37.868857 4939 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="61c402c5-4efa-4821-b51e-da74f4d9eff7" path="/var/lib/kubelet/pods/61c402c5-4efa-4821-b51e-da74f4d9eff7/volumes" Nov 03 01:42:37.869312 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:42:37.869299 4939 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bdedc3dc-7ab3-4c41-84ea-e1443fd30898" path="/var/lib/kubelet/pods/bdedc3dc-7ab3-4c41-84ea-e1443fd30898/volumes" Nov 03 01:42:37.870044 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:42:37.870029 4939 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d2dabe94-37c9-4a3f-bf9a-15e4f53e05d0" path="/var/lib/kubelet/pods/d2dabe94-37c9-4a3f-bf9a-15e4f53e05d0/volumes" Nov 03 01:42:37.993445 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:42:37.993410 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:42:38.504103 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:42:38.504066 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:42:38.619295 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:42:38.619260 4939 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-4pw7v"] Nov 03 01:42:38.780249 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:42:38.780173 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:42:39.496711 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:42:39.496675 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:42:39.757279 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:42:39.757197 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:42:39.880615 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:42:39.880585 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:42:40.199419 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:42:40.199388 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:42:40.788196 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:42:40.788144 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:42:41.401045 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:42:41.401011 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:42:42.127862 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:42:42.127830 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:42:42.802729 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:42:42.802693 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:42:43.553787 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:42:43.553737 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:42:43.988805 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:42:43.988770 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:42:44.107288 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:42:44.107250 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:42:44.292154 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:42:44.292077 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:42:44.470708 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:42:44.470678 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:42:44.739316 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:42:44.739262 4939 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w1-4af1f7-openshift-default-745ff98565-ktw4m"] Nov 03 01:42:44.739894 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:42:44.739550 4939 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w1-4af1f7-openshift-default-745ff98565-ktw4m" podUID="d2965442-bcbd-4959-b60a-08074c5aead5" containerName="istio-proxy" containerID="cri-o://5d291651f437ed1fb8d1fa3e61977d39dd0987d4be3ae72a2a5424d02e9e4256" gracePeriod=30 Nov 03 01:42:44.795530 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:42:44.795498 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:42:45.172292 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:42:45.172253 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:42:45.293632 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:42:45.293596 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:42:45.543226 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:42:45.543148 4939 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-14ca4876-fvwl9"] Nov 03 01:42:45.543580 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:42:45.543567 4939 cpu_manager.go:414] "containerMap: removing container" podUID="61c402c5-4efa-4821-b51e-da74f4d9eff7" containerName="istio-proxy" Nov 03 01:42:45.543626 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:42:45.543582 4939 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="61c402c5-4efa-4821-b51e-da74f4d9eff7" containerName="istio-proxy" Nov 03 01:42:45.543626 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:42:45.543600 4939 cpu_manager.go:414] "containerMap: removing container" podUID="4bcec813-d864-450b-b606-d97e7c1a1948" containerName="istio-proxy" Nov 03 01:42:45.543626 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:42:45.543605 4939 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4bcec813-d864-450b-b606-d97e7c1a1948" containerName="istio-proxy" Nov 03 01:42:45.543733 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:42:45.543631 4939 cpu_manager.go:414] "containerMap: removing container" podUID="bdedc3dc-7ab3-4c41-84ea-e1443fd30898" containerName="istio-proxy" Nov 03 01:42:45.543733 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:42:45.543636 4939 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bdedc3dc-7ab3-4c41-84ea-e1443fd30898" containerName="istio-proxy" Nov 03 01:42:45.543733 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:42:45.543648 4939 cpu_manager.go:414] "containerMap: removing container" podUID="d2dabe94-37c9-4a3f-bf9a-15e4f53e05d0" containerName="istio-proxy" Nov 03 01:42:45.543733 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:42:45.543653 4939 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d2dabe94-37c9-4a3f-bf9a-15e4f53e05d0" containerName="istio-proxy" Nov 03 01:42:45.543733 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:42:45.543715 4939 memory_manager.go:365] "RemoveStaleState removing state" podUID="bdedc3dc-7ab3-4c41-84ea-e1443fd30898" containerName="istio-proxy" Nov 03 01:42:45.543733 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:42:45.543726 4939 memory_manager.go:365] "RemoveStaleState removing state" podUID="d2dabe94-37c9-4a3f-bf9a-15e4f53e05d0" containerName="istio-proxy" Nov 03 01:42:45.543733 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:42:45.543734 4939 memory_manager.go:365] "RemoveStaleState removing state" podUID="61c402c5-4efa-4821-b51e-da74f4d9eff7" containerName="istio-proxy" Nov 03 01:42:45.543979 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:42:45.543741 4939 memory_manager.go:365] "RemoveStaleState removing state" podUID="4bcec813-d864-450b-b606-d97e7c1a1948" containerName="istio-proxy" Nov 03 01:42:45.548596 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:42:45.548572 4939 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-14ca4876-fvwl9" Nov 03 01:42:45.555916 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:42:45.555882 4939 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 01:42:45.556235 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:42:45.556120 4939 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 03 01:42:45.556235 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:42:45.556167 4939 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 03 01:42:45.556235 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:42:45.556206 4939 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-6q4xp\"" Nov 03 01:42:45.558647 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:42:45.558621 4939 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-14ca4876-fvwl9"] Nov 03 01:42:45.660014 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:42:45.659980 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/26577b0e-2eca-45e7-84cc-8aab89dcd232-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-14ca4876-fvwl9\" (UID: \"26577b0e-2eca-45e7-84cc-8aab89dcd232\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-14ca4876-fvwl9" Nov 03 01:42:45.660208 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:42:45.660030 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/26577b0e-2eca-45e7-84cc-8aab89dcd232-maas-api-xks-ca\") pod \"maas-api-revoke-sub-14ca4876-fvwl9\" (UID: \"26577b0e-2eca-45e7-84cc-8aab89dcd232\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-14ca4876-fvwl9" Nov 03 01:42:45.660208 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:42:45.660115 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lmc2q\" (UniqueName: \"kubernetes.io/projected/26577b0e-2eca-45e7-84cc-8aab89dcd232-kube-api-access-lmc2q\") pod \"maas-api-revoke-sub-14ca4876-fvwl9\" (UID: \"26577b0e-2eca-45e7-84cc-8aab89dcd232\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-14ca4876-fvwl9" Nov 03 01:42:45.761082 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:42:45.761042 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/26577b0e-2eca-45e7-84cc-8aab89dcd232-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-14ca4876-fvwl9\" (UID: \"26577b0e-2eca-45e7-84cc-8aab89dcd232\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-14ca4876-fvwl9" Nov 03 01:42:45.761502 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:42:45.761095 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/26577b0e-2eca-45e7-84cc-8aab89dcd232-maas-api-xks-ca\") pod \"maas-api-revoke-sub-14ca4876-fvwl9\" (UID: \"26577b0e-2eca-45e7-84cc-8aab89dcd232\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-14ca4876-fvwl9" Nov 03 01:42:45.761502 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:42:45.761136 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lmc2q\" (UniqueName: \"kubernetes.io/projected/26577b0e-2eca-45e7-84cc-8aab89dcd232-kube-api-access-lmc2q\") pod \"maas-api-revoke-sub-14ca4876-fvwl9\" (UID: \"26577b0e-2eca-45e7-84cc-8aab89dcd232\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-14ca4876-fvwl9" Nov 03 01:42:45.761714 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:42:45.761689 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/26577b0e-2eca-45e7-84cc-8aab89dcd232-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-14ca4876-fvwl9\" (UID: \"26577b0e-2eca-45e7-84cc-8aab89dcd232\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-14ca4876-fvwl9" Nov 03 01:42:45.763691 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:42:45.763664 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/26577b0e-2eca-45e7-84cc-8aab89dcd232-maas-api-xks-ca\") pod \"maas-api-revoke-sub-14ca4876-fvwl9\" (UID: \"26577b0e-2eca-45e7-84cc-8aab89dcd232\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-14ca4876-fvwl9" Nov 03 01:42:45.769735 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:42:45.769708 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lmc2q\" (UniqueName: \"kubernetes.io/projected/26577b0e-2eca-45e7-84cc-8aab89dcd232-kube-api-access-lmc2q\") pod \"maas-api-revoke-sub-14ca4876-fvwl9\" (UID: \"26577b0e-2eca-45e7-84cc-8aab89dcd232\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-14ca4876-fvwl9" Nov 03 01:42:45.863151 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:42:45.863116 4939 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-14ca4876-fvwl9" Nov 03 01:42:45.890312 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:42:45.890283 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:42:46.049328 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:42:46.049293 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:42:46.159488 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:42:46.159460 4939 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-14ca4876-fvwl9"] Nov 03 01:42:46.162040 ip-10-0-140-107 kubenswrapper[4939]: W1103 01:42:46.162004 4939 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod26577b0e_2eca_45e7_84cc_8aab89dcd232.slice/crio-09d5cccfa0d5f0581cba957e9dccd04fdeff287e65c0b30a4659bdd1ac73106e WatchSource:0}: Error finding container 09d5cccfa0d5f0581cba957e9dccd04fdeff287e65c0b30a4659bdd1ac73106e: Status 404 returned error can't find the container with id 09d5cccfa0d5f0581cba957e9dccd04fdeff287e65c0b30a4659bdd1ac73106e Nov 03 01:42:46.480110 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:42:46.480079 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:42:46.807157 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:42:46.807081 4939 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-4pw7v"] Nov 03 01:42:46.977277 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:42:46.977244 4939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-14ca4876-fvwl9" event={"ID":"26577b0e-2eca-45e7-84cc-8aab89dcd232","Type":"ContainerStarted","Data":"9b2e377fcbe44a1a579d909f66bc27739eea889e2f89654501b2d41fad4d2c4c"} Nov 03 01:42:46.977464 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:42:46.977299 4939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-14ca4876-fvwl9" event={"ID":"26577b0e-2eca-45e7-84cc-8aab89dcd232","Type":"ContainerStarted","Data":"09d5cccfa0d5f0581cba957e9dccd04fdeff287e65c0b30a4659bdd1ac73106e"} Nov 03 01:42:46.994772 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:42:46.994716 4939 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-14ca4876-fvwl9" podStartSLOduration=1.99470262 podStartE2EDuration="1.99470262s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 01:42:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 01:42:46.992599147 +0000 UTC m=+3715.256934091" watchObservedRunningTime="2026-11-03 01:42:46.99470262 +0000 UTC m=+3715.259037565" Nov 03 01:42:47.204656 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:42:47.204597 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:42:47.272941 ip-10-0-140-107 kubenswrapper[4939]: W1103 01:42:47.271020 4939 logging.go:55] [core] [Channel #20267 SubChannel #20269]grpc: addrConn.createTransport failed to connect to {Addr: "10.132.0.168:50052", ServerName: "10.132.0.168:50052", BalancerAttributes: {"<%!p(pickfirstleaf.managedByPickfirstKeyType={})>": "<%!p(bool=true)>" }}. Err: connection error: desc = "transport: Error while dialing: dial tcp 10.132.0.168:50052: connect: connection refused" Nov 03 01:42:47.378423 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:42:47.378396 4939 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-2729f2-565847c555-wqvj7" Nov 03 01:42:47.381482 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:42:47.381460 4939 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-2729f2-558f777dc-9gclz" Nov 03 01:42:47.480052 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:42:47.480021 4939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c1e683ec-d1fc-4273-9384-1ffcc06a832a-kube-api-access-npp6d\" (UniqueName: \"kubernetes.io/projected/c1e683ec-d1fc-4273-9384-1ffcc06a832a-kube-api-access-npp6d\") pod \"c1e683ec-d1fc-4273-9384-1ffcc06a832a\" (UID: \"c1e683ec-d1fc-4273-9384-1ffcc06a832a\") " Nov 03 01:42:47.480271 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:42:47.480121 4939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/89acf0f1-ef07-4c99-afd3-de3d06335488-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/89acf0f1-ef07-4c99-afd3-de3d06335488-plugins-config-volume\") pod \"89acf0f1-ef07-4c99-afd3-de3d06335488\" (UID: \"89acf0f1-ef07-4c99-afd3-de3d06335488\") " Nov 03 01:42:47.480271 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:42:47.480171 4939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/89acf0f1-ef07-4c99-afd3-de3d06335488-kube-api-access-tkkxj\" (UniqueName: \"kubernetes.io/projected/89acf0f1-ef07-4c99-afd3-de3d06335488-kube-api-access-tkkxj\") pod \"89acf0f1-ef07-4c99-afd3-de3d06335488\" (UID: \"89acf0f1-ef07-4c99-afd3-de3d06335488\") " Nov 03 01:42:47.480271 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:42:47.480203 4939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c1e683ec-d1fc-4273-9384-1ffcc06a832a-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c1e683ec-d1fc-4273-9384-1ffcc06a832a-plugins-config-volume\") pod \"c1e683ec-d1fc-4273-9384-1ffcc06a832a\" (UID: \"c1e683ec-d1fc-4273-9384-1ffcc06a832a\") " Nov 03 01:42:47.480662 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:42:47.480632 4939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/89acf0f1-ef07-4c99-afd3-de3d06335488-plugins-config-volume" pod "89acf0f1-ef07-4c99-afd3-de3d06335488" (UID: "89acf0f1-ef07-4c99-afd3-de3d06335488"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 01:42:47.480777 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:42:47.480648 4939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c1e683ec-d1fc-4273-9384-1ffcc06a832a-plugins-config-volume" pod "c1e683ec-d1fc-4273-9384-1ffcc06a832a" (UID: "c1e683ec-d1fc-4273-9384-1ffcc06a832a"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 01:42:47.482254 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:42:47.482236 4939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c1e683ec-d1fc-4273-9384-1ffcc06a832a-kube-api-access-npp6d" pod "c1e683ec-d1fc-4273-9384-1ffcc06a832a" (UID: "c1e683ec-d1fc-4273-9384-1ffcc06a832a"). InnerVolumeSpecName "kube-api-access-npp6d". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 01:42:47.482381 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:42:47.482359 4939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/89acf0f1-ef07-4c99-afd3-de3d06335488-kube-api-access-tkkxj" pod "89acf0f1-ef07-4c99-afd3-de3d06335488" (UID: "89acf0f1-ef07-4c99-afd3-de3d06335488"). InnerVolumeSpecName "kube-api-access-tkkxj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 01:42:47.581032 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:42:47.580997 4939 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/89acf0f1-ef07-4c99-afd3-de3d06335488-plugins-config-volume\") on node \"ip-10-0-140-107.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:42:47.581032 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:42:47.581025 4939 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tkkxj\" (UniqueName: \"kubernetes.io/projected/89acf0f1-ef07-4c99-afd3-de3d06335488-kube-api-access-tkkxj\") on node \"ip-10-0-140-107.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:42:47.581032 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:42:47.581035 4939 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c1e683ec-d1fc-4273-9384-1ffcc06a832a-plugins-config-volume\") on node \"ip-10-0-140-107.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:42:47.581277 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:42:47.581046 4939 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-npp6d\" (UniqueName: \"kubernetes.io/projected/c1e683ec-d1fc-4273-9384-1ffcc06a832a-kube-api-access-npp6d\") on node \"ip-10-0-140-107.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:42:47.794557 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:42:47.794519 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:42:47.983970 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:42:47.983931 4939 generic.go:356] "Generic (PLEG): container finished" podID="c1e683ec-d1fc-4273-9384-1ffcc06a832a" containerID="076f1302b3c0a56f35ca8f05d46e5d588e15f1312c7e774aefbb2da2db6c2c2c" exitCode=0 Nov 03 01:42:47.984453 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:42:47.983975 4939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-2729f2-558f777dc-9gclz" event={"ID":"c1e683ec-d1fc-4273-9384-1ffcc06a832a","Type":"ContainerDied","Data":"076f1302b3c0a56f35ca8f05d46e5d588e15f1312c7e774aefbb2da2db6c2c2c"} Nov 03 01:42:47.984453 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:42:47.984004 4939 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-2729f2-558f777dc-9gclz" Nov 03 01:42:47.984453 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:42:47.984040 4939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-2729f2-558f777dc-9gclz" event={"ID":"c1e683ec-d1fc-4273-9384-1ffcc06a832a","Type":"ContainerDied","Data":"ba1e8e787ac227c00d92b2e496fa6c7eef28d21491e470fca67d71fb7682a460"} Nov 03 01:42:47.984453 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:42:47.984070 4939 scope.go:122] "RemoveContainer" containerID="076f1302b3c0a56f35ca8f05d46e5d588e15f1312c7e774aefbb2da2db6c2c2c" Nov 03 01:42:47.985718 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:42:47.985698 4939 generic.go:356] "Generic (PLEG): container finished" podID="89acf0f1-ef07-4c99-afd3-de3d06335488" containerID="a29f4ada5ae4f3bf33d23981cc60dd8a9fb34c3ed134e3c6cad41f2cfea2c7cd" exitCode=0 Nov 03 01:42:47.985815 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:42:47.985775 4939 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-2729f2-565847c555-wqvj7" Nov 03 01:42:47.985815 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:42:47.985781 4939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-2729f2-565847c555-wqvj7" event={"ID":"89acf0f1-ef07-4c99-afd3-de3d06335488","Type":"ContainerDied","Data":"a29f4ada5ae4f3bf33d23981cc60dd8a9fb34c3ed134e3c6cad41f2cfea2c7cd"} Nov 03 01:42:47.985944 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:42:47.985835 4939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-2729f2-565847c555-wqvj7" event={"ID":"89acf0f1-ef07-4c99-afd3-de3d06335488","Type":"ContainerDied","Data":"a236efb3b54615897784863c0defdd5c318fa8c499abd6632fcde6f948fdda95"} Nov 03 01:42:47.987866 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:42:47.987837 4939 generic.go:356] "Generic (PLEG): container finished" podID="26577b0e-2eca-45e7-84cc-8aab89dcd232" containerID="9b2e377fcbe44a1a579d909f66bc27739eea889e2f89654501b2d41fad4d2c4c" exitCode=0 Nov 03 01:42:47.988043 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:42:47.987901 4939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-14ca4876-fvwl9" event={"ID":"26577b0e-2eca-45e7-84cc-8aab89dcd232","Type":"ContainerDied","Data":"9b2e377fcbe44a1a579d909f66bc27739eea889e2f89654501b2d41fad4d2c4c"} Nov 03 01:42:47.994112 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:42:47.994095 4939 scope.go:122] "RemoveContainer" containerID="076f1302b3c0a56f35ca8f05d46e5d588e15f1312c7e774aefbb2da2db6c2c2c" Nov 03 01:42:47.994445 ip-10-0-140-107 kubenswrapper[4939]: E1103 01:42:47.994421 4939 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"076f1302b3c0a56f35ca8f05d46e5d588e15f1312c7e774aefbb2da2db6c2c2c\": container with ID starting with 076f1302b3c0a56f35ca8f05d46e5d588e15f1312c7e774aefbb2da2db6c2c2c not found: ID does not exist" containerID="076f1302b3c0a56f35ca8f05d46e5d588e15f1312c7e774aefbb2da2db6c2c2c" Nov 03 01:42:47.994545 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:42:47.994452 4939 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"076f1302b3c0a56f35ca8f05d46e5d588e15f1312c7e774aefbb2da2db6c2c2c"} err="failed to get container status \"076f1302b3c0a56f35ca8f05d46e5d588e15f1312c7e774aefbb2da2db6c2c2c\": rpc error: code = NotFound desc = could not find container \"076f1302b3c0a56f35ca8f05d46e5d588e15f1312c7e774aefbb2da2db6c2c2c\": container with ID starting with 076f1302b3c0a56f35ca8f05d46e5d588e15f1312c7e774aefbb2da2db6c2c2c not found: ID does not exist" Nov 03 01:42:47.994545 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:42:47.994467 4939 scope.go:122] "RemoveContainer" containerID="a29f4ada5ae4f3bf33d23981cc60dd8a9fb34c3ed134e3c6cad41f2cfea2c7cd" Nov 03 01:42:48.002253 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:42:48.002236 4939 scope.go:122] "RemoveContainer" containerID="a29f4ada5ae4f3bf33d23981cc60dd8a9fb34c3ed134e3c6cad41f2cfea2c7cd" Nov 03 01:42:48.002518 ip-10-0-140-107 kubenswrapper[4939]: E1103 01:42:48.002496 4939 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a29f4ada5ae4f3bf33d23981cc60dd8a9fb34c3ed134e3c6cad41f2cfea2c7cd\": container with ID starting with a29f4ada5ae4f3bf33d23981cc60dd8a9fb34c3ed134e3c6cad41f2cfea2c7cd not found: ID does not exist" containerID="a29f4ada5ae4f3bf33d23981cc60dd8a9fb34c3ed134e3c6cad41f2cfea2c7cd" Nov 03 01:42:48.002566 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:42:48.002525 4939 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a29f4ada5ae4f3bf33d23981cc60dd8a9fb34c3ed134e3c6cad41f2cfea2c7cd"} err="failed to get container status \"a29f4ada5ae4f3bf33d23981cc60dd8a9fb34c3ed134e3c6cad41f2cfea2c7cd\": rpc error: code = NotFound desc = could not find container \"a29f4ada5ae4f3bf33d23981cc60dd8a9fb34c3ed134e3c6cad41f2cfea2c7cd\": container with ID starting with a29f4ada5ae4f3bf33d23981cc60dd8a9fb34c3ed134e3c6cad41f2cfea2c7cd not found: ID does not exist" Nov 03 01:42:48.021065 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:42:48.020982 4939 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w1-2729f2-565847c555-wqvj7"] Nov 03 01:42:48.027446 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:42:48.027417 4939 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w1-2729f2-565847c555-wqvj7"] Nov 03 01:42:48.034723 ip-10-0-140-107 kubenswrapper[4939]: E1103 01:42:48.034684 4939 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-sthjh" containerName="extractor" Nov 03 01:42:48.038451 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:42:48.038424 4939 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w1-2729f2-558f777dc-9gclz"] Nov 03 01:42:48.041443 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:42:48.041418 4939 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w1-2729f2-558f777dc-9gclz"] Nov 03 01:42:48.271335 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:42:48.271230 4939 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-2729f2-565847c555-wqvj7" podUID="89acf0f1-ef07-4c99-afd3-de3d06335488" containerName="payload-processing" probeResult="failure" output="timeout: failed to connect service \"10.132.0.169:50052\" within 1s: context deadline exceeded" Nov 03 01:42:48.271514 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:42:48.271355 4939 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-2729f2-558f777dc-9gclz" podUID="c1e683ec-d1fc-4273-9384-1ffcc06a832a" containerName="payload-pre-processing" probeResult="failure" output="timeout: failed to connect service \"10.132.0.168:50052\" within 1s: context deadline exceeded" Nov 03 01:42:48.271514 ip-10-0-140-107 kubenswrapper[4939]: W1103 01:42:48.271403 4939 logging.go:55] [core] [Channel #20267 SubChannel #20269]grpc: addrConn.createTransport failed to connect to {Addr: "10.132.0.168:50052", ServerName: "10.132.0.168:50052", BalancerAttributes: {"<%!p(pickfirstleaf.managedByPickfirstKeyType={})>": "<%!p(bool=true)>" }}. Err: connection error: desc = "transport: Error while dialing: dial tcp 10.132.0.168:50052: operation was canceled" Nov 03 01:42:48.271514 ip-10-0-140-107 kubenswrapper[4939]: W1103 01:42:48.271403 4939 logging.go:55] [core] [Channel #20268 SubChannel #20270]grpc: addrConn.createTransport failed to connect to {Addr: "10.132.0.169:50052", ServerName: "10.132.0.169:50052", BalancerAttributes: {"<%!p(pickfirstleaf.managedByPickfirstKeyType={})>": "<%!p(bool=true)>" }}. Err: connection error: desc = "transport: Error while dialing: dial tcp 10.132.0.169:50052: operation was canceled" Nov 03 01:42:48.401800 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:42:48.401764 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:42:49.090748 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:42:49.090711 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:42:49.143894 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:42:49.143867 4939 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-14ca4876-fvwl9" Nov 03 01:42:49.296342 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:42:49.296258 4939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/26577b0e-2eca-45e7-84cc-8aab89dcd232-kube-api-access-lmc2q\" (UniqueName: \"kubernetes.io/projected/26577b0e-2eca-45e7-84cc-8aab89dcd232-kube-api-access-lmc2q\") pod \"26577b0e-2eca-45e7-84cc-8aab89dcd232\" (UID: \"26577b0e-2eca-45e7-84cc-8aab89dcd232\") " Nov 03 01:42:49.296342 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:42:49.296308 4939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/26577b0e-2eca-45e7-84cc-8aab89dcd232-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/26577b0e-2eca-45e7-84cc-8aab89dcd232-maas-api-xks-ca\") pod \"26577b0e-2eca-45e7-84cc-8aab89dcd232\" (UID: \"26577b0e-2eca-45e7-84cc-8aab89dcd232\") " Nov 03 01:42:49.296545 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:42:49.296449 4939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/26577b0e-2eca-45e7-84cc-8aab89dcd232-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/26577b0e-2eca-45e7-84cc-8aab89dcd232-maas-api-openshift-ca\") pod \"26577b0e-2eca-45e7-84cc-8aab89dcd232\" (UID: \"26577b0e-2eca-45e7-84cc-8aab89dcd232\") " Nov 03 01:42:49.296845 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:42:49.296818 4939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/26577b0e-2eca-45e7-84cc-8aab89dcd232-maas-api-openshift-ca" pod "26577b0e-2eca-45e7-84cc-8aab89dcd232" (UID: "26577b0e-2eca-45e7-84cc-8aab89dcd232"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 01:42:49.298392 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:42:49.298364 4939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/26577b0e-2eca-45e7-84cc-8aab89dcd232-maas-api-xks-ca" pod "26577b0e-2eca-45e7-84cc-8aab89dcd232" (UID: "26577b0e-2eca-45e7-84cc-8aab89dcd232"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 01:42:49.298520 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:42:49.298502 4939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/26577b0e-2eca-45e7-84cc-8aab89dcd232-kube-api-access-lmc2q" pod "26577b0e-2eca-45e7-84cc-8aab89dcd232" (UID: "26577b0e-2eca-45e7-84cc-8aab89dcd232"). InnerVolumeSpecName "kube-api-access-lmc2q". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 01:42:49.397886 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:42:49.397852 4939 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lmc2q\" (UniqueName: \"kubernetes.io/projected/26577b0e-2eca-45e7-84cc-8aab89dcd232-kube-api-access-lmc2q\") on node \"ip-10-0-140-107.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:42:49.397886 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:42:49.397883 4939 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/26577b0e-2eca-45e7-84cc-8aab89dcd232-maas-api-xks-ca\") on node \"ip-10-0-140-107.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:42:49.398121 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:42:49.397894 4939 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/26577b0e-2eca-45e7-84cc-8aab89dcd232-maas-api-openshift-ca\") on node \"ip-10-0-140-107.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:42:49.694020 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:42:49.693984 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:42:49.866686 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:42:49.866654 4939 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="89acf0f1-ef07-4c99-afd3-de3d06335488" path="/var/lib/kubelet/pods/89acf0f1-ef07-4c99-afd3-de3d06335488/volumes" Nov 03 01:42:49.867115 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:42:49.867100 4939 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c1e683ec-d1fc-4273-9384-1ffcc06a832a" path="/var/lib/kubelet/pods/c1e683ec-d1fc-4273-9384-1ffcc06a832a/volumes" Nov 03 01:42:49.968696 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:42:49.968668 4939 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w1-4af1f7-openshift-default-745ff98565-ktw4m" Nov 03 01:42:50.003471 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:42:50.003429 4939 generic.go:356] "Generic (PLEG): container finished" podID="d2965442-bcbd-4959-b60a-08074c5aead5" containerID="5d291651f437ed1fb8d1fa3e61977d39dd0987d4be3ae72a2a5424d02e9e4256" exitCode=0 Nov 03 01:42:50.003684 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:42:50.003482 4939 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w1-4af1f7-openshift-default-745ff98565-ktw4m" Nov 03 01:42:50.003684 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:42:50.003512 4939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w1-4af1f7-openshift-default-745ff98565-ktw4m" event={"ID":"d2965442-bcbd-4959-b60a-08074c5aead5","Type":"ContainerDied","Data":"5d291651f437ed1fb8d1fa3e61977d39dd0987d4be3ae72a2a5424d02e9e4256"} Nov 03 01:42:50.003684 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:42:50.003568 4939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w1-4af1f7-openshift-default-745ff98565-ktw4m" event={"ID":"d2965442-bcbd-4959-b60a-08074c5aead5","Type":"ContainerDied","Data":"86f0f2a7c74a736a7de851e69d10082bea516054b535c56a2218344ac8767568"} Nov 03 01:42:50.003684 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:42:50.003595 4939 scope.go:122] "RemoveContainer" containerID="5d291651f437ed1fb8d1fa3e61977d39dd0987d4be3ae72a2a5424d02e9e4256" Nov 03 01:42:50.005828 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:42:50.005804 4939 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-14ca4876-fvwl9" Nov 03 01:42:50.005828 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:42:50.005800 4939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-14ca4876-fvwl9" event={"ID":"26577b0e-2eca-45e7-84cc-8aab89dcd232","Type":"ContainerDied","Data":"09d5cccfa0d5f0581cba957e9dccd04fdeff287e65c0b30a4659bdd1ac73106e"} Nov 03 01:42:50.006037 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:42:50.005866 4939 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="09d5cccfa0d5f0581cba957e9dccd04fdeff287e65c0b30a4659bdd1ac73106e" Nov 03 01:42:50.014398 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:42:50.014380 4939 scope.go:122] "RemoveContainer" containerID="5d291651f437ed1fb8d1fa3e61977d39dd0987d4be3ae72a2a5424d02e9e4256" Nov 03 01:42:50.014731 ip-10-0-140-107 kubenswrapper[4939]: E1103 01:42:50.014706 4939 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5d291651f437ed1fb8d1fa3e61977d39dd0987d4be3ae72a2a5424d02e9e4256\": container with ID starting with 5d291651f437ed1fb8d1fa3e61977d39dd0987d4be3ae72a2a5424d02e9e4256 not found: ID does not exist" containerID="5d291651f437ed1fb8d1fa3e61977d39dd0987d4be3ae72a2a5424d02e9e4256" Nov 03 01:42:50.014842 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:42:50.014736 4939 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5d291651f437ed1fb8d1fa3e61977d39dd0987d4be3ae72a2a5424d02e9e4256"} err="failed to get container status \"5d291651f437ed1fb8d1fa3e61977d39dd0987d4be3ae72a2a5424d02e9e4256\": rpc error: code = NotFound desc = could not find container \"5d291651f437ed1fb8d1fa3e61977d39dd0987d4be3ae72a2a5424d02e9e4256\": container with ID starting with 5d291651f437ed1fb8d1fa3e61977d39dd0987d4be3ae72a2a5424d02e9e4256 not found: ID does not exist" Nov 03 01:42:50.105556 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:42:50.105522 4939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d2965442-bcbd-4959-b60a-08074c5aead5-istio-token\" (UniqueName: \"kubernetes.io/projected/d2965442-bcbd-4959-b60a-08074c5aead5-istio-token\") pod \"d2965442-bcbd-4959-b60a-08074c5aead5\" (UID: \"d2965442-bcbd-4959-b60a-08074c5aead5\") " Nov 03 01:42:50.106084 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:42:50.105583 4939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d2965442-bcbd-4959-b60a-08074c5aead5-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d2965442-bcbd-4959-b60a-08074c5aead5-credential-socket\") pod \"d2965442-bcbd-4959-b60a-08074c5aead5\" (UID: \"d2965442-bcbd-4959-b60a-08074c5aead5\") " Nov 03 01:42:50.106084 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:42:50.105612 4939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d2965442-bcbd-4959-b60a-08074c5aead5-kube-api-access-xgds7\" (UniqueName: \"kubernetes.io/projected/d2965442-bcbd-4959-b60a-08074c5aead5-kube-api-access-xgds7\") pod \"d2965442-bcbd-4959-b60a-08074c5aead5\" (UID: \"d2965442-bcbd-4959-b60a-08074c5aead5\") " Nov 03 01:42:50.106084 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:42:50.105641 4939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/d2965442-bcbd-4959-b60a-08074c5aead5-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d2965442-bcbd-4959-b60a-08074c5aead5-istio-podinfo\") pod \"d2965442-bcbd-4959-b60a-08074c5aead5\" (UID: \"d2965442-bcbd-4959-b60a-08074c5aead5\") " Nov 03 01:42:50.106084 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:42:50.105690 4939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d2965442-bcbd-4959-b60a-08074c5aead5-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d2965442-bcbd-4959-b60a-08074c5aead5-workload-socket\") pod \"d2965442-bcbd-4959-b60a-08074c5aead5\" (UID: \"d2965442-bcbd-4959-b60a-08074c5aead5\") " Nov 03 01:42:50.106084 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:42:50.105755 4939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d2965442-bcbd-4959-b60a-08074c5aead5-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d2965442-bcbd-4959-b60a-08074c5aead5-workload-certs\") pod \"d2965442-bcbd-4959-b60a-08074c5aead5\" (UID: \"d2965442-bcbd-4959-b60a-08074c5aead5\") " Nov 03 01:42:50.106084 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:42:50.105829 4939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d2965442-bcbd-4959-b60a-08074c5aead5-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d2965442-bcbd-4959-b60a-08074c5aead5-istio-envoy\") pod \"d2965442-bcbd-4959-b60a-08074c5aead5\" (UID: \"d2965442-bcbd-4959-b60a-08074c5aead5\") " Nov 03 01:42:50.106084 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:42:50.105869 4939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d2965442-bcbd-4959-b60a-08074c5aead5-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d2965442-bcbd-4959-b60a-08074c5aead5-istio-data\") pod \"d2965442-bcbd-4959-b60a-08074c5aead5\" (UID: \"d2965442-bcbd-4959-b60a-08074c5aead5\") " Nov 03 01:42:50.106084 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:42:50.105879 4939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d2965442-bcbd-4959-b60a-08074c5aead5-credential-socket" pod "d2965442-bcbd-4959-b60a-08074c5aead5" (UID: "d2965442-bcbd-4959-b60a-08074c5aead5"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 01:42:50.106084 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:42:50.105921 4939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d2965442-bcbd-4959-b60a-08074c5aead5-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d2965442-bcbd-4959-b60a-08074c5aead5-istiod-ca-cert\") pod \"d2965442-bcbd-4959-b60a-08074c5aead5\" (UID: \"d2965442-bcbd-4959-b60a-08074c5aead5\") " Nov 03 01:42:50.106084 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:42:50.106060 4939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d2965442-bcbd-4959-b60a-08074c5aead5-workload-socket" pod "d2965442-bcbd-4959-b60a-08074c5aead5" (UID: "d2965442-bcbd-4959-b60a-08074c5aead5"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 01:42:50.106725 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:42:50.106106 4939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d2965442-bcbd-4959-b60a-08074c5aead5-workload-certs" pod "d2965442-bcbd-4959-b60a-08074c5aead5" (UID: "d2965442-bcbd-4959-b60a-08074c5aead5"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 01:42:50.106725 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:42:50.106294 4939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d2965442-bcbd-4959-b60a-08074c5aead5-istio-data" pod "d2965442-bcbd-4959-b60a-08074c5aead5" (UID: "d2965442-bcbd-4959-b60a-08074c5aead5"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 01:42:50.106725 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:42:50.106367 4939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d2965442-bcbd-4959-b60a-08074c5aead5-istiod-ca-cert" pod "d2965442-bcbd-4959-b60a-08074c5aead5" (UID: "d2965442-bcbd-4959-b60a-08074c5aead5"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 01:42:50.106725 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:42:50.106450 4939 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d2965442-bcbd-4959-b60a-08074c5aead5-istio-data\") on node \"ip-10-0-140-107.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:42:50.106725 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:42:50.106472 4939 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d2965442-bcbd-4959-b60a-08074c5aead5-credential-socket\") on node \"ip-10-0-140-107.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:42:50.106725 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:42:50.106487 4939 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d2965442-bcbd-4959-b60a-08074c5aead5-workload-socket\") on node \"ip-10-0-140-107.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:42:50.106725 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:42:50.106502 4939 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d2965442-bcbd-4959-b60a-08074c5aead5-workload-certs\") on node \"ip-10-0-140-107.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:42:50.108138 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:42:50.108112 4939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d2965442-bcbd-4959-b60a-08074c5aead5-istio-envoy" pod "d2965442-bcbd-4959-b60a-08074c5aead5" (UID: "d2965442-bcbd-4959-b60a-08074c5aead5"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 01:42:50.108138 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:42:50.108132 4939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d2965442-bcbd-4959-b60a-08074c5aead5-kube-api-access-xgds7" pod "d2965442-bcbd-4959-b60a-08074c5aead5" (UID: "d2965442-bcbd-4959-b60a-08074c5aead5"). InnerVolumeSpecName "kube-api-access-xgds7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 01:42:50.108327 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:42:50.108308 4939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/d2965442-bcbd-4959-b60a-08074c5aead5-istio-podinfo" pod "d2965442-bcbd-4959-b60a-08074c5aead5" (UID: "d2965442-bcbd-4959-b60a-08074c5aead5"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 01:42:50.108389 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:42:50.108341 4939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d2965442-bcbd-4959-b60a-08074c5aead5-istio-token" pod "d2965442-bcbd-4959-b60a-08074c5aead5" (UID: "d2965442-bcbd-4959-b60a-08074c5aead5"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 01:42:50.141659 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:42:50.141631 4939 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w1-88bbf7/distinct-2-w1-88bbf7-kserve-5457cdcdbd-l6tfd"] Nov 03 01:42:50.162370 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:42:50.162338 4939 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w1-88bbf7/simulator-w1-88bbf7-kserve-5fbb669f8f-42hgd"] Nov 03 01:42:50.207792 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:42:50.207757 4939 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d2965442-bcbd-4959-b60a-08074c5aead5-istio-token\") on node \"ip-10-0-140-107.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:42:50.207792 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:42:50.207788 4939 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xgds7\" (UniqueName: \"kubernetes.io/projected/d2965442-bcbd-4959-b60a-08074c5aead5-kube-api-access-xgds7\") on node \"ip-10-0-140-107.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:42:50.207792 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:42:50.207799 4939 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d2965442-bcbd-4959-b60a-08074c5aead5-istio-podinfo\") on node \"ip-10-0-140-107.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:42:50.208047 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:42:50.207809 4939 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d2965442-bcbd-4959-b60a-08074c5aead5-istio-envoy\") on node \"ip-10-0-140-107.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:42:50.208047 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:42:50.207817 4939 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d2965442-bcbd-4959-b60a-08074c5aead5-istiod-ca-cert\") on node \"ip-10-0-140-107.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:42:50.245560 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:42:50.245499 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:42:50.328361 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:42:50.328327 4939 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w1-4af1f7-openshift-default-745ff98565-ktw4m"] Nov 03 01:42:50.340577 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:42:50.340543 4939 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w1-4af1f7-openshift-default-745ff98565-ktw4m"] Nov 03 01:42:50.358468 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:42:50.358436 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:42:50.617861 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:42:50.617817 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:42:50.977768 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:42:50.977685 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:42:51.599171 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:42:51.599129 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:42:51.867589 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:42:51.867510 4939 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d2965442-bcbd-4959-b60a-08074c5aead5" path="/var/lib/kubelet/pods/d2965442-bcbd-4959-b60a-08074c5aead5/volumes" Nov 03 01:42:52.283232 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:42:52.283154 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:42:52.610736 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:42:52.610694 4939 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-4pw7v"] Nov 03 01:42:52.939964 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:42:52.939854 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:42:53.047923 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:42:53.047867 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:42:53.166497 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:42:53.166460 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:42:53.563874 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:42:53.563841 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:42:54.201189 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:42:54.201152 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:42:54.891512 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:42:54.891479 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:42:55.513544 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:42:55.513501 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:42:56.115028 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:42:56.114992 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:42:56.413325 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:42:56.413241 4939 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cace40c1-s2475"] Nov 03 01:42:56.413644 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:42:56.413632 4939 cpu_manager.go:414] "containerMap: removing container" podUID="c1e683ec-d1fc-4273-9384-1ffcc06a832a" containerName="payload-pre-processing" Nov 03 01:42:56.413695 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:42:56.413646 4939 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c1e683ec-d1fc-4273-9384-1ffcc06a832a" containerName="payload-pre-processing" Nov 03 01:42:56.413695 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:42:56.413662 4939 cpu_manager.go:414] "containerMap: removing container" podUID="d2965442-bcbd-4959-b60a-08074c5aead5" containerName="istio-proxy" Nov 03 01:42:56.413695 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:42:56.413668 4939 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d2965442-bcbd-4959-b60a-08074c5aead5" containerName="istio-proxy" Nov 03 01:42:56.413695 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:42:56.413685 4939 cpu_manager.go:414] "containerMap: removing container" podUID="26577b0e-2eca-45e7-84cc-8aab89dcd232" containerName="revoke-keys" Nov 03 01:42:56.413695 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:42:56.413690 4939 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="26577b0e-2eca-45e7-84cc-8aab89dcd232" containerName="revoke-keys" Nov 03 01:42:56.413861 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:42:56.413698 4939 cpu_manager.go:414] "containerMap: removing container" podUID="89acf0f1-ef07-4c99-afd3-de3d06335488" containerName="payload-processing" Nov 03 01:42:56.413861 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:42:56.413703 4939 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="89acf0f1-ef07-4c99-afd3-de3d06335488" containerName="payload-processing" Nov 03 01:42:56.413861 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:42:56.413764 4939 memory_manager.go:365] "RemoveStaleState removing state" podUID="c1e683ec-d1fc-4273-9384-1ffcc06a832a" containerName="payload-pre-processing" Nov 03 01:42:56.413861 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:42:56.413774 4939 memory_manager.go:365] "RemoveStaleState removing state" podUID="d2965442-bcbd-4959-b60a-08074c5aead5" containerName="istio-proxy" Nov 03 01:42:56.413861 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:42:56.413784 4939 memory_manager.go:365] "RemoveStaleState removing state" podUID="26577b0e-2eca-45e7-84cc-8aab89dcd232" containerName="revoke-keys" Nov 03 01:42:56.413861 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:42:56.413790 4939 memory_manager.go:365] "RemoveStaleState removing state" podUID="89acf0f1-ef07-4c99-afd3-de3d06335488" containerName="payload-processing" Nov 03 01:42:56.418320 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:42:56.418295 4939 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cace40c1-s2475" Nov 03 01:42:56.426658 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:42:56.426068 4939 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cace40c1-s2475"] Nov 03 01:42:56.426658 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:42:56.426246 4939 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 03 01:42:56.426658 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:42:56.426315 4939 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 01:42:56.427117 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:42:56.427094 4939 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-6q4xp\"" Nov 03 01:42:56.427117 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:42:56.427112 4939 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 03 01:42:56.570969 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:42:56.570933 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x5dch\" (UniqueName: \"kubernetes.io/projected/7f05b5e8-35b2-4f88-b6f2-ff4b645a34c5-kube-api-access-x5dch\") pod \"maas-api-revoke-sub-cace40c1-s2475\" (UID: \"7f05b5e8-35b2-4f88-b6f2-ff4b645a34c5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cace40c1-s2475" Nov 03 01:42:56.571357 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:42:56.570979 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7f05b5e8-35b2-4f88-b6f2-ff4b645a34c5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-cace40c1-s2475\" (UID: \"7f05b5e8-35b2-4f88-b6f2-ff4b645a34c5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cace40c1-s2475" Nov 03 01:42:56.571357 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:42:56.571082 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7f05b5e8-35b2-4f88-b6f2-ff4b645a34c5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-cace40c1-s2475\" (UID: \"7f05b5e8-35b2-4f88-b6f2-ff4b645a34c5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cace40c1-s2475" Nov 03 01:42:56.672570 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:42:56.672445 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x5dch\" (UniqueName: \"kubernetes.io/projected/7f05b5e8-35b2-4f88-b6f2-ff4b645a34c5-kube-api-access-x5dch\") pod \"maas-api-revoke-sub-cace40c1-s2475\" (UID: \"7f05b5e8-35b2-4f88-b6f2-ff4b645a34c5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cace40c1-s2475" Nov 03 01:42:56.672570 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:42:56.672514 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7f05b5e8-35b2-4f88-b6f2-ff4b645a34c5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-cace40c1-s2475\" (UID: \"7f05b5e8-35b2-4f88-b6f2-ff4b645a34c5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cace40c1-s2475" Nov 03 01:42:56.672570 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:42:56.672556 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7f05b5e8-35b2-4f88-b6f2-ff4b645a34c5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-cace40c1-s2475\" (UID: \"7f05b5e8-35b2-4f88-b6f2-ff4b645a34c5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cace40c1-s2475" Nov 03 01:42:56.673336 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:42:56.673305 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7f05b5e8-35b2-4f88-b6f2-ff4b645a34c5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-cace40c1-s2475\" (UID: \"7f05b5e8-35b2-4f88-b6f2-ff4b645a34c5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cace40c1-s2475" Nov 03 01:42:56.675349 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:42:56.675325 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7f05b5e8-35b2-4f88-b6f2-ff4b645a34c5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-cace40c1-s2475\" (UID: \"7f05b5e8-35b2-4f88-b6f2-ff4b645a34c5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cace40c1-s2475" Nov 03 01:42:56.688555 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:42:56.688512 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-x5dch\" (UniqueName: \"kubernetes.io/projected/7f05b5e8-35b2-4f88-b6f2-ff4b645a34c5-kube-api-access-x5dch\") pod \"maas-api-revoke-sub-cace40c1-s2475\" (UID: \"7f05b5e8-35b2-4f88-b6f2-ff4b645a34c5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cace40c1-s2475" Nov 03 01:42:56.733926 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:42:56.733872 4939 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cace40c1-s2475" Nov 03 01:42:56.772851 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:42:56.772822 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:42:56.881549 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:42:56.881510 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:42:57.013038 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:42:57.013013 4939 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cace40c1-s2475"] Nov 03 01:42:57.015420 ip-10-0-140-107 kubenswrapper[4939]: W1103 01:42:57.015388 4939 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7f05b5e8_35b2_4f88_b6f2_ff4b645a34c5.slice/crio-c66d50f65ff0d4d7d69f301782d563ec39589f173777f314aa11fdd9c7295e2e WatchSource:0}: Error finding container c66d50f65ff0d4d7d69f301782d563ec39589f173777f314aa11fdd9c7295e2e: Status 404 returned error can't find the container with id c66d50f65ff0d4d7d69f301782d563ec39589f173777f314aa11fdd9c7295e2e Nov 03 01:42:57.051964 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:42:57.051937 4939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cace40c1-s2475" event={"ID":"7f05b5e8-35b2-4f88-b6f2-ff4b645a34c5","Type":"ContainerStarted","Data":"c66d50f65ff0d4d7d69f301782d563ec39589f173777f314aa11fdd9c7295e2e"} Nov 03 01:42:57.101513 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:42:57.101483 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:42:58.059066 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:42:58.058975 4939 generic.go:356] "Generic (PLEG): container finished" podID="7f05b5e8-35b2-4f88-b6f2-ff4b645a34c5" containerID="140682075db05de362fbc3d45d65471e1f469ca7281b2567ed682bc6dac64f4d" exitCode=0 Nov 03 01:42:58.059066 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:42:58.059041 4939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cace40c1-s2475" event={"ID":"7f05b5e8-35b2-4f88-b6f2-ff4b645a34c5","Type":"ContainerDied","Data":"140682075db05de362fbc3d45d65471e1f469ca7281b2567ed682bc6dac64f4d"} Nov 03 01:42:58.135733 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:42:58.135701 4939 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-88bbf7-openshift-default-574f47fff7-6qqxz"] Nov 03 01:42:58.139192 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:42:58.139172 4939 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-88bbf7-openshift-default-574f47fff7-6qqxz" Nov 03 01:42:58.143038 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:42:58.143016 4939 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w1-88bbf7-openshift-default-dockercfg-h6n25\"" Nov 03 01:42:58.150294 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:42:58.150272 4939 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-88bbf7-openshift-default-574f47fff7-6qqxz"] Nov 03 01:42:58.289137 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:42:58.289093 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3ceb505f-63e4-427e-aad1-d40ca8062b2b-istiod-ca-cert\") pod \"e2e-worker-w1-88bbf7-openshift-default-574f47fff7-6qqxz\" (UID: \"3ceb505f-63e4-427e-aad1-d40ca8062b2b\") " pod="openshift-ingress/e2e-worker-w1-88bbf7-openshift-default-574f47fff7-6qqxz" Nov 03 01:42:58.289137 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:42:58.289143 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3ceb505f-63e4-427e-aad1-d40ca8062b2b-workload-socket\") pod \"e2e-worker-w1-88bbf7-openshift-default-574f47fff7-6qqxz\" (UID: \"3ceb505f-63e4-427e-aad1-d40ca8062b2b\") " pod="openshift-ingress/e2e-worker-w1-88bbf7-openshift-default-574f47fff7-6qqxz" Nov 03 01:42:58.289400 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:42:58.289229 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3ceb505f-63e4-427e-aad1-d40ca8062b2b-credential-socket\") pod \"e2e-worker-w1-88bbf7-openshift-default-574f47fff7-6qqxz\" (UID: \"3ceb505f-63e4-427e-aad1-d40ca8062b2b\") " pod="openshift-ingress/e2e-worker-w1-88bbf7-openshift-default-574f47fff7-6qqxz" Nov 03 01:42:58.289400 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:42:58.289274 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3ceb505f-63e4-427e-aad1-d40ca8062b2b-istio-token\") pod \"e2e-worker-w1-88bbf7-openshift-default-574f47fff7-6qqxz\" (UID: \"3ceb505f-63e4-427e-aad1-d40ca8062b2b\") " pod="openshift-ingress/e2e-worker-w1-88bbf7-openshift-default-574f47fff7-6qqxz" Nov 03 01:42:58.289400 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:42:58.289377 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3ceb505f-63e4-427e-aad1-d40ca8062b2b-istio-data\") pod \"e2e-worker-w1-88bbf7-openshift-default-574f47fff7-6qqxz\" (UID: \"3ceb505f-63e4-427e-aad1-d40ca8062b2b\") " pod="openshift-ingress/e2e-worker-w1-88bbf7-openshift-default-574f47fff7-6qqxz" Nov 03 01:42:58.289570 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:42:58.289436 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3ceb505f-63e4-427e-aad1-d40ca8062b2b-istio-envoy\") pod \"e2e-worker-w1-88bbf7-openshift-default-574f47fff7-6qqxz\" (UID: \"3ceb505f-63e4-427e-aad1-d40ca8062b2b\") " pod="openshift-ingress/e2e-worker-w1-88bbf7-openshift-default-574f47fff7-6qqxz" Nov 03 01:42:58.289570 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:42:58.289467 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3ceb505f-63e4-427e-aad1-d40ca8062b2b-workload-certs\") pod \"e2e-worker-w1-88bbf7-openshift-default-574f47fff7-6qqxz\" (UID: \"3ceb505f-63e4-427e-aad1-d40ca8062b2b\") " pod="openshift-ingress/e2e-worker-w1-88bbf7-openshift-default-574f47fff7-6qqxz" Nov 03 01:42:58.289570 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:42:58.289514 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zclss\" (UniqueName: \"kubernetes.io/projected/3ceb505f-63e4-427e-aad1-d40ca8062b2b-kube-api-access-zclss\") pod \"e2e-worker-w1-88bbf7-openshift-default-574f47fff7-6qqxz\" (UID: \"3ceb505f-63e4-427e-aad1-d40ca8062b2b\") " pod="openshift-ingress/e2e-worker-w1-88bbf7-openshift-default-574f47fff7-6qqxz" Nov 03 01:42:58.289697 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:42:58.289632 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3ceb505f-63e4-427e-aad1-d40ca8062b2b-istio-podinfo\") pod \"e2e-worker-w1-88bbf7-openshift-default-574f47fff7-6qqxz\" (UID: \"3ceb505f-63e4-427e-aad1-d40ca8062b2b\") " pod="openshift-ingress/e2e-worker-w1-88bbf7-openshift-default-574f47fff7-6qqxz" Nov 03 01:42:58.390474 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:42:58.390437 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3ceb505f-63e4-427e-aad1-d40ca8062b2b-istio-podinfo\") pod \"e2e-worker-w1-88bbf7-openshift-default-574f47fff7-6qqxz\" (UID: \"3ceb505f-63e4-427e-aad1-d40ca8062b2b\") " pod="openshift-ingress/e2e-worker-w1-88bbf7-openshift-default-574f47fff7-6qqxz" Nov 03 01:42:58.390638 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:42:58.390507 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3ceb505f-63e4-427e-aad1-d40ca8062b2b-istiod-ca-cert\") pod \"e2e-worker-w1-88bbf7-openshift-default-574f47fff7-6qqxz\" (UID: \"3ceb505f-63e4-427e-aad1-d40ca8062b2b\") " pod="openshift-ingress/e2e-worker-w1-88bbf7-openshift-default-574f47fff7-6qqxz" Nov 03 01:42:58.390638 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:42:58.390536 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3ceb505f-63e4-427e-aad1-d40ca8062b2b-workload-socket\") pod \"e2e-worker-w1-88bbf7-openshift-default-574f47fff7-6qqxz\" (UID: \"3ceb505f-63e4-427e-aad1-d40ca8062b2b\") " pod="openshift-ingress/e2e-worker-w1-88bbf7-openshift-default-574f47fff7-6qqxz" Nov 03 01:42:58.390638 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:42:58.390556 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3ceb505f-63e4-427e-aad1-d40ca8062b2b-credential-socket\") pod \"e2e-worker-w1-88bbf7-openshift-default-574f47fff7-6qqxz\" (UID: \"3ceb505f-63e4-427e-aad1-d40ca8062b2b\") " pod="openshift-ingress/e2e-worker-w1-88bbf7-openshift-default-574f47fff7-6qqxz" Nov 03 01:42:58.390638 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:42:58.390577 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3ceb505f-63e4-427e-aad1-d40ca8062b2b-istio-token\") pod \"e2e-worker-w1-88bbf7-openshift-default-574f47fff7-6qqxz\" (UID: \"3ceb505f-63e4-427e-aad1-d40ca8062b2b\") " pod="openshift-ingress/e2e-worker-w1-88bbf7-openshift-default-574f47fff7-6qqxz" Nov 03 01:42:58.390638 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:42:58.390614 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3ceb505f-63e4-427e-aad1-d40ca8062b2b-istio-data\") pod \"e2e-worker-w1-88bbf7-openshift-default-574f47fff7-6qqxz\" (UID: \"3ceb505f-63e4-427e-aad1-d40ca8062b2b\") " pod="openshift-ingress/e2e-worker-w1-88bbf7-openshift-default-574f47fff7-6qqxz" Nov 03 01:42:58.390964 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:42:58.390663 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3ceb505f-63e4-427e-aad1-d40ca8062b2b-istio-envoy\") pod \"e2e-worker-w1-88bbf7-openshift-default-574f47fff7-6qqxz\" (UID: \"3ceb505f-63e4-427e-aad1-d40ca8062b2b\") " pod="openshift-ingress/e2e-worker-w1-88bbf7-openshift-default-574f47fff7-6qqxz" Nov 03 01:42:58.390964 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:42:58.390691 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3ceb505f-63e4-427e-aad1-d40ca8062b2b-workload-certs\") pod \"e2e-worker-w1-88bbf7-openshift-default-574f47fff7-6qqxz\" (UID: \"3ceb505f-63e4-427e-aad1-d40ca8062b2b\") " pod="openshift-ingress/e2e-worker-w1-88bbf7-openshift-default-574f47fff7-6qqxz" Nov 03 01:42:58.390964 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:42:58.390715 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zclss\" (UniqueName: \"kubernetes.io/projected/3ceb505f-63e4-427e-aad1-d40ca8062b2b-kube-api-access-zclss\") pod \"e2e-worker-w1-88bbf7-openshift-default-574f47fff7-6qqxz\" (UID: \"3ceb505f-63e4-427e-aad1-d40ca8062b2b\") " pod="openshift-ingress/e2e-worker-w1-88bbf7-openshift-default-574f47fff7-6qqxz" Nov 03 01:42:58.391143 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:42:58.391085 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3ceb505f-63e4-427e-aad1-d40ca8062b2b-workload-socket\") pod \"e2e-worker-w1-88bbf7-openshift-default-574f47fff7-6qqxz\" (UID: \"3ceb505f-63e4-427e-aad1-d40ca8062b2b\") " pod="openshift-ingress/e2e-worker-w1-88bbf7-openshift-default-574f47fff7-6qqxz" Nov 03 01:42:58.391143 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:42:58.391126 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3ceb505f-63e4-427e-aad1-d40ca8062b2b-credential-socket\") pod \"e2e-worker-w1-88bbf7-openshift-default-574f47fff7-6qqxz\" (UID: \"3ceb505f-63e4-427e-aad1-d40ca8062b2b\") " pod="openshift-ingress/e2e-worker-w1-88bbf7-openshift-default-574f47fff7-6qqxz" Nov 03 01:42:58.391439 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:42:58.391412 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3ceb505f-63e4-427e-aad1-d40ca8062b2b-istio-data\") pod \"e2e-worker-w1-88bbf7-openshift-default-574f47fff7-6qqxz\" (UID: \"3ceb505f-63e4-427e-aad1-d40ca8062b2b\") " pod="openshift-ingress/e2e-worker-w1-88bbf7-openshift-default-574f47fff7-6qqxz" Nov 03 01:42:58.391552 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:42:58.391474 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3ceb505f-63e4-427e-aad1-d40ca8062b2b-workload-certs\") pod \"e2e-worker-w1-88bbf7-openshift-default-574f47fff7-6qqxz\" (UID: \"3ceb505f-63e4-427e-aad1-d40ca8062b2b\") " pod="openshift-ingress/e2e-worker-w1-88bbf7-openshift-default-574f47fff7-6qqxz" Nov 03 01:42:58.391742 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:42:58.391722 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3ceb505f-63e4-427e-aad1-d40ca8062b2b-istiod-ca-cert\") pod \"e2e-worker-w1-88bbf7-openshift-default-574f47fff7-6qqxz\" (UID: \"3ceb505f-63e4-427e-aad1-d40ca8062b2b\") " pod="openshift-ingress/e2e-worker-w1-88bbf7-openshift-default-574f47fff7-6qqxz" Nov 03 01:42:58.393278 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:42:58.393246 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3ceb505f-63e4-427e-aad1-d40ca8062b2b-istio-envoy\") pod \"e2e-worker-w1-88bbf7-openshift-default-574f47fff7-6qqxz\" (UID: \"3ceb505f-63e4-427e-aad1-d40ca8062b2b\") " pod="openshift-ingress/e2e-worker-w1-88bbf7-openshift-default-574f47fff7-6qqxz" Nov 03 01:42:58.393380 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:42:58.393301 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3ceb505f-63e4-427e-aad1-d40ca8062b2b-istio-podinfo\") pod \"e2e-worker-w1-88bbf7-openshift-default-574f47fff7-6qqxz\" (UID: \"3ceb505f-63e4-427e-aad1-d40ca8062b2b\") " pod="openshift-ingress/e2e-worker-w1-88bbf7-openshift-default-574f47fff7-6qqxz" Nov 03 01:42:58.399121 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:42:58.399096 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3ceb505f-63e4-427e-aad1-d40ca8062b2b-istio-token\") pod \"e2e-worker-w1-88bbf7-openshift-default-574f47fff7-6qqxz\" (UID: \"3ceb505f-63e4-427e-aad1-d40ca8062b2b\") " pod="openshift-ingress/e2e-worker-w1-88bbf7-openshift-default-574f47fff7-6qqxz" Nov 03 01:42:58.399377 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:42:58.399359 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zclss\" (UniqueName: \"kubernetes.io/projected/3ceb505f-63e4-427e-aad1-d40ca8062b2b-kube-api-access-zclss\") pod \"e2e-worker-w1-88bbf7-openshift-default-574f47fff7-6qqxz\" (UID: \"3ceb505f-63e4-427e-aad1-d40ca8062b2b\") " pod="openshift-ingress/e2e-worker-w1-88bbf7-openshift-default-574f47fff7-6qqxz" Nov 03 01:42:58.453311 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:42:58.453280 4939 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-88bbf7-openshift-default-574f47fff7-6qqxz" Nov 03 01:42:58.730872 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:42:58.730848 4939 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-88bbf7-openshift-default-574f47fff7-6qqxz"] Nov 03 01:42:58.733059 ip-10-0-140-107 kubenswrapper[4939]: W1103 01:42:58.733028 4939 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3ceb505f_63e4_427e_aad1_d40ca8062b2b.slice/crio-d5425de7d92cd3d3a0d5a5973f5ba2e875f66a12ca25a25c3fc05ae301056f73 WatchSource:0}: Error finding container d5425de7d92cd3d3a0d5a5973f5ba2e875f66a12ca25a25c3fc05ae301056f73: Status 404 returned error can't find the container with id d5425de7d92cd3d3a0d5a5973f5ba2e875f66a12ca25a25c3fc05ae301056f73 Nov 03 01:42:58.735283 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:42:58.735244 4939 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 01:42:58.735348 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:42:58.735309 4939 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 01:42:58.735348 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:42:58.735340 4939 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 01:42:59.065563 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:42:59.065472 4939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-88bbf7-openshift-default-574f47fff7-6qqxz" event={"ID":"3ceb505f-63e4-427e-aad1-d40ca8062b2b","Type":"ContainerStarted","Data":"1360b1db3879eb318e46aad725095dba682ba897cb83beae80a8a8a488c63b87"} Nov 03 01:42:59.065563 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:42:59.065535 4939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-88bbf7-openshift-default-574f47fff7-6qqxz" event={"ID":"3ceb505f-63e4-427e-aad1-d40ca8062b2b","Type":"ContainerStarted","Data":"d5425de7d92cd3d3a0d5a5973f5ba2e875f66a12ca25a25c3fc05ae301056f73"} Nov 03 01:42:59.225295 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:42:59.225268 4939 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cace40c1-s2475" Nov 03 01:42:59.242856 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:42:59.242803 4939 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-88bbf7-openshift-default-574f47fff7-6qqxz" podStartSLOduration=1.242786272 podStartE2EDuration="1.242786272s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 01:42:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 01:42:59.088310681 +0000 UTC m=+3727.352645627" watchObservedRunningTime="2026-11-03 01:42:59.242786272 +0000 UTC m=+3727.507121221" Nov 03 01:42:59.405353 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:42:59.405315 4939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7f05b5e8-35b2-4f88-b6f2-ff4b645a34c5-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7f05b5e8-35b2-4f88-b6f2-ff4b645a34c5-maas-api-openshift-ca\") pod \"7f05b5e8-35b2-4f88-b6f2-ff4b645a34c5\" (UID: \"7f05b5e8-35b2-4f88-b6f2-ff4b645a34c5\") " Nov 03 01:42:59.405587 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:42:59.405395 4939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7f05b5e8-35b2-4f88-b6f2-ff4b645a34c5-kube-api-access-x5dch\" (UniqueName: \"kubernetes.io/projected/7f05b5e8-35b2-4f88-b6f2-ff4b645a34c5-kube-api-access-x5dch\") pod \"7f05b5e8-35b2-4f88-b6f2-ff4b645a34c5\" (UID: \"7f05b5e8-35b2-4f88-b6f2-ff4b645a34c5\") " Nov 03 01:42:59.405587 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:42:59.405435 4939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7f05b5e8-35b2-4f88-b6f2-ff4b645a34c5-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7f05b5e8-35b2-4f88-b6f2-ff4b645a34c5-maas-api-xks-ca\") pod \"7f05b5e8-35b2-4f88-b6f2-ff4b645a34c5\" (UID: \"7f05b5e8-35b2-4f88-b6f2-ff4b645a34c5\") " Nov 03 01:42:59.405738 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:42:59.405700 4939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7f05b5e8-35b2-4f88-b6f2-ff4b645a34c5-maas-api-openshift-ca" pod "7f05b5e8-35b2-4f88-b6f2-ff4b645a34c5" (UID: "7f05b5e8-35b2-4f88-b6f2-ff4b645a34c5"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 01:42:59.407537 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:42:59.407512 4939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7f05b5e8-35b2-4f88-b6f2-ff4b645a34c5-maas-api-xks-ca" pod "7f05b5e8-35b2-4f88-b6f2-ff4b645a34c5" (UID: "7f05b5e8-35b2-4f88-b6f2-ff4b645a34c5"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 01:42:59.407623 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:42:59.407545 4939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7f05b5e8-35b2-4f88-b6f2-ff4b645a34c5-kube-api-access-x5dch" pod "7f05b5e8-35b2-4f88-b6f2-ff4b645a34c5" (UID: "7f05b5e8-35b2-4f88-b6f2-ff4b645a34c5"). InnerVolumeSpecName "kube-api-access-x5dch". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 01:42:59.466932 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:42:59.454273 4939 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-88bbf7-openshift-default-574f47fff7-6qqxz" Nov 03 01:42:59.472472 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:42:59.472441 4939 patch_prober.go:29] interesting pod/e2e-worker-w1-88bbf7-openshift-default-574f47fff7-6qqxz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.92:15021/healthz/ready\": dial tcp 10.132.1.92:15021: connect: connection refused" start-of-body= Nov 03 01:42:59.472592 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:42:59.472509 4939 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-88bbf7-openshift-default-574f47fff7-6qqxz" podUID="3ceb505f-63e4-427e-aad1-d40ca8062b2b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.92:15021/healthz/ready\": dial tcp 10.132.1.92:15021: connect: connection refused" Nov 03 01:42:59.506810 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:42:59.506787 4939 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7f05b5e8-35b2-4f88-b6f2-ff4b645a34c5-maas-api-openshift-ca\") on node \"ip-10-0-140-107.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:42:59.506884 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:42:59.506817 4939 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-x5dch\" (UniqueName: \"kubernetes.io/projected/7f05b5e8-35b2-4f88-b6f2-ff4b645a34c5-kube-api-access-x5dch\") on node \"ip-10-0-140-107.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:42:59.506884 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:42:59.506832 4939 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7f05b5e8-35b2-4f88-b6f2-ff4b645a34c5-maas-api-xks-ca\") on node \"ip-10-0-140-107.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:43:00.075780 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:43:00.075694 4939 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cace40c1-s2475" Nov 03 01:43:00.075780 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:43:00.075731 4939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cace40c1-s2475" event={"ID":"7f05b5e8-35b2-4f88-b6f2-ff4b645a34c5","Type":"ContainerDied","Data":"c66d50f65ff0d4d7d69f301782d563ec39589f173777f314aa11fdd9c7295e2e"} Nov 03 01:43:00.076282 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:43:00.075814 4939 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c66d50f65ff0d4d7d69f301782d563ec39589f173777f314aa11fdd9c7295e2e" Nov 03 01:43:00.464942 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:43:00.460373 4939 patch_prober.go:29] interesting pod/e2e-worker-w1-88bbf7-openshift-default-574f47fff7-6qqxz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.92:15021/healthz/ready\": dial tcp 10.132.1.92:15021: connect: connection refused" start-of-body= Nov 03 01:43:00.464942 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:43:00.460455 4939 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-88bbf7-openshift-default-574f47fff7-6qqxz" podUID="3ceb505f-63e4-427e-aad1-d40ca8062b2b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.92:15021/healthz/ready\": dial tcp 10.132.1.92:15021: connect: connection refused" Nov 03 01:43:01.467932 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:43:01.461340 4939 patch_prober.go:29] interesting pod/e2e-worker-w1-88bbf7-openshift-default-574f47fff7-6qqxz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.92:15021/healthz/ready\": dial tcp 10.132.1.92:15021: connect: connection refused" start-of-body= Nov 03 01:43:01.467932 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:43:01.461417 4939 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-88bbf7-openshift-default-574f47fff7-6qqxz" podUID="3ceb505f-63e4-427e-aad1-d40ca8062b2b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.92:15021/healthz/ready\": dial tcp 10.132.1.92:15021: connect: connection refused" Nov 03 01:43:02.030595 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:43:02.030553 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:43:02.465932 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:43:02.460276 4939 patch_prober.go:29] interesting pod/e2e-worker-w1-88bbf7-openshift-default-574f47fff7-6qqxz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.92:15021/healthz/ready\": dial tcp 10.132.1.92:15021: connect: connection refused" start-of-body= Nov 03 01:43:02.465932 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:43:02.460340 4939 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-88bbf7-openshift-default-574f47fff7-6qqxz" podUID="3ceb505f-63e4-427e-aad1-d40ca8062b2b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.92:15021/healthz/ready\": dial tcp 10.132.1.92:15021: connect: connection refused" Nov 03 01:43:02.872281 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:43:02.872243 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:43:03.455295 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:43:03.455257 4939 patch_prober.go:29] interesting pod/e2e-worker-w1-88bbf7-openshift-default-574f47fff7-6qqxz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.92:15021/healthz/ready\": dial tcp 10.132.1.92:15021: connect: connection refused" start-of-body= Nov 03 01:43:03.455611 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:43:03.455577 4939 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-88bbf7-openshift-default-574f47fff7-6qqxz" podUID="3ceb505f-63e4-427e-aad1-d40ca8062b2b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.92:15021/healthz/ready\": dial tcp 10.132.1.92:15021: connect: connection refused" Nov 03 01:43:04.467934 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:43:04.464305 4939 patch_prober.go:29] interesting pod/e2e-worker-w1-88bbf7-openshift-default-574f47fff7-6qqxz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.92:15021/healthz/ready\": dial tcp 10.132.1.92:15021: connect: connection refused" start-of-body= Nov 03 01:43:04.467934 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:43:04.464383 4939 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-88bbf7-openshift-default-574f47fff7-6qqxz" podUID="3ceb505f-63e4-427e-aad1-d40ca8062b2b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.92:15021/healthz/ready\": dial tcp 10.132.1.92:15021: connect: connection refused" Nov 03 01:43:05.163067 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:43:05.163023 4939 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w1-88bbf7/distinct-2-w1-88bbf7-kserve-5457cdcdbd-l6tfd" podUID="a397931b-faa9-4c2e-8290-491350546663" containerName="main" containerID="cri-o://4d89519b7718515ab659a365ce964abdde58e902233a6a73f99bb5a317f2eb19" gracePeriod=45 Nov 03 01:43:05.187343 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:43:05.187305 4939 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w1-88bbf7/simulator-w1-88bbf7-kserve-5fbb669f8f-42hgd" podUID="83b38883-2e7a-444d-aa68-52be4a25ae66" containerName="main" containerID="cri-o://f6ef9c5f02772524826ae140c107b0209f1ee2cb412948574b7ba2a235ab0860" gracePeriod=45 Nov 03 01:43:05.444714 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:43:05.444689 4939 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-88bbf7/distinct-2-w1-88bbf7-kserve-5457cdcdbd-l6tfd" Nov 03 01:43:05.469175 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:43:05.469140 4939 patch_prober.go:29] interesting pod/e2e-worker-w1-88bbf7-openshift-default-574f47fff7-6qqxz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.92:15021/healthz/ready\": dial tcp 10.132.1.92:15021: connect: connection refused" start-of-body= Nov 03 01:43:05.469616 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:43:05.469204 4939 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-88bbf7-openshift-default-574f47fff7-6qqxz" podUID="3ceb505f-63e4-427e-aad1-d40ca8062b2b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.92:15021/healthz/ready\": dial tcp 10.132.1.92:15021: connect: connection refused" Nov 03 01:43:05.486404 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:43:05.485639 4939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a397931b-faa9-4c2e-8290-491350546663-kube-api-access-vnpdj\" (UniqueName: \"kubernetes.io/projected/a397931b-faa9-4c2e-8290-491350546663-kube-api-access-vnpdj\") pod \"a397931b-faa9-4c2e-8290-491350546663\" (UID: \"a397931b-faa9-4c2e-8290-491350546663\") " Nov 03 01:43:05.486404 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:43:05.485681 4939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a397931b-faa9-4c2e-8290-491350546663-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a397931b-faa9-4c2e-8290-491350546663-model-cache\") pod \"a397931b-faa9-4c2e-8290-491350546663\" (UID: \"a397931b-faa9-4c2e-8290-491350546663\") " Nov 03 01:43:05.486404 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:43:05.485739 4939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a397931b-faa9-4c2e-8290-491350546663-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a397931b-faa9-4c2e-8290-491350546663-tmp-dir\") pod \"a397931b-faa9-4c2e-8290-491350546663\" (UID: \"a397931b-faa9-4c2e-8290-491350546663\") " Nov 03 01:43:05.486404 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:43:05.485781 4939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a397931b-faa9-4c2e-8290-491350546663-home\" (UniqueName: \"kubernetes.io/empty-dir/a397931b-faa9-4c2e-8290-491350546663-home\") pod \"a397931b-faa9-4c2e-8290-491350546663\" (UID: \"a397931b-faa9-4c2e-8290-491350546663\") " Nov 03 01:43:05.486404 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:43:05.485840 4939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a397931b-faa9-4c2e-8290-491350546663-tls-certs\" (UniqueName: \"kubernetes.io/secret/a397931b-faa9-4c2e-8290-491350546663-tls-certs\") pod \"a397931b-faa9-4c2e-8290-491350546663\" (UID: \"a397931b-faa9-4c2e-8290-491350546663\") " Nov 03 01:43:05.486404 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:43:05.485963 4939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a397931b-faa9-4c2e-8290-491350546663-dshm\" (UniqueName: \"kubernetes.io/empty-dir/a397931b-faa9-4c2e-8290-491350546663-dshm\") pod \"a397931b-faa9-4c2e-8290-491350546663\" (UID: \"a397931b-faa9-4c2e-8290-491350546663\") " Nov 03 01:43:05.487030 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:43:05.486997 4939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a397931b-faa9-4c2e-8290-491350546663-tmp-dir" pod "a397931b-faa9-4c2e-8290-491350546663" (UID: "a397931b-faa9-4c2e-8290-491350546663"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 01:43:05.487137 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:43:05.487097 4939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a397931b-faa9-4c2e-8290-491350546663-home" pod "a397931b-faa9-4c2e-8290-491350546663" (UID: "a397931b-faa9-4c2e-8290-491350546663"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 01:43:05.487207 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:43:05.487149 4939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a397931b-faa9-4c2e-8290-491350546663-model-cache" pod "a397931b-faa9-4c2e-8290-491350546663" (UID: "a397931b-faa9-4c2e-8290-491350546663"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 01:43:05.490550 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:43:05.490459 4939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a397931b-faa9-4c2e-8290-491350546663-tls-certs" pod "a397931b-faa9-4c2e-8290-491350546663" (UID: "a397931b-faa9-4c2e-8290-491350546663"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 01:43:05.491777 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:43:05.491748 4939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a397931b-faa9-4c2e-8290-491350546663-dshm" pod "a397931b-faa9-4c2e-8290-491350546663" (UID: "a397931b-faa9-4c2e-8290-491350546663"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 01:43:05.494684 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:43:05.494658 4939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a397931b-faa9-4c2e-8290-491350546663-kube-api-access-vnpdj" pod "a397931b-faa9-4c2e-8290-491350546663" (UID: "a397931b-faa9-4c2e-8290-491350546663"). InnerVolumeSpecName "kube-api-access-vnpdj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 01:43:05.511170 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:43:05.511148 4939 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-88bbf7/simulator-w1-88bbf7-kserve-5fbb669f8f-42hgd" Nov 03 01:43:05.586728 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:43:05.586695 4939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/83b38883-2e7a-444d-aa68-52be4a25ae66-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/83b38883-2e7a-444d-aa68-52be4a25ae66-tmp-dir\") pod \"83b38883-2e7a-444d-aa68-52be4a25ae66\" (UID: \"83b38883-2e7a-444d-aa68-52be4a25ae66\") " Nov 03 01:43:05.586953 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:43:05.586766 4939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/83b38883-2e7a-444d-aa68-52be4a25ae66-dshm\" (UniqueName: \"kubernetes.io/empty-dir/83b38883-2e7a-444d-aa68-52be4a25ae66-dshm\") pod \"83b38883-2e7a-444d-aa68-52be4a25ae66\" (UID: \"83b38883-2e7a-444d-aa68-52be4a25ae66\") " Nov 03 01:43:05.586953 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:43:05.586830 4939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/83b38883-2e7a-444d-aa68-52be4a25ae66-tls-certs\" (UniqueName: \"kubernetes.io/secret/83b38883-2e7a-444d-aa68-52be4a25ae66-tls-certs\") pod \"83b38883-2e7a-444d-aa68-52be4a25ae66\" (UID: \"83b38883-2e7a-444d-aa68-52be4a25ae66\") " Nov 03 01:43:05.586953 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:43:05.586936 4939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/83b38883-2e7a-444d-aa68-52be4a25ae66-kube-api-access-9qxj6\" (UniqueName: \"kubernetes.io/projected/83b38883-2e7a-444d-aa68-52be4a25ae66-kube-api-access-9qxj6\") pod \"83b38883-2e7a-444d-aa68-52be4a25ae66\" (UID: \"83b38883-2e7a-444d-aa68-52be4a25ae66\") " Nov 03 01:43:05.587165 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:43:05.586973 4939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/83b38883-2e7a-444d-aa68-52be4a25ae66-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/83b38883-2e7a-444d-aa68-52be4a25ae66-model-cache\") pod \"83b38883-2e7a-444d-aa68-52be4a25ae66\" (UID: \"83b38883-2e7a-444d-aa68-52be4a25ae66\") " Nov 03 01:43:05.587165 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:43:05.587032 4939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/83b38883-2e7a-444d-aa68-52be4a25ae66-home\" (UniqueName: \"kubernetes.io/empty-dir/83b38883-2e7a-444d-aa68-52be4a25ae66-home\") pod \"83b38883-2e7a-444d-aa68-52be4a25ae66\" (UID: \"83b38883-2e7a-444d-aa68-52be4a25ae66\") " Nov 03 01:43:05.587165 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:43:05.587045 4939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/83b38883-2e7a-444d-aa68-52be4a25ae66-tmp-dir" pod "83b38883-2e7a-444d-aa68-52be4a25ae66" (UID: "83b38883-2e7a-444d-aa68-52be4a25ae66"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 01:43:05.587340 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:43:05.587305 4939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/83b38883-2e7a-444d-aa68-52be4a25ae66-home" pod "83b38883-2e7a-444d-aa68-52be4a25ae66" (UID: "83b38883-2e7a-444d-aa68-52be4a25ae66"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 01:43:05.587507 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:43:05.587480 4939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/83b38883-2e7a-444d-aa68-52be4a25ae66-model-cache" pod "83b38883-2e7a-444d-aa68-52be4a25ae66" (UID: "83b38883-2e7a-444d-aa68-52be4a25ae66"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 01:43:05.587618 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:43:05.587538 4939 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/83b38883-2e7a-444d-aa68-52be4a25ae66-home\") on node \"ip-10-0-140-107.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:43:05.587618 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:43:05.587559 4939 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a397931b-faa9-4c2e-8290-491350546663-home\") on node \"ip-10-0-140-107.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:43:05.587618 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:43:05.587575 4939 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a397931b-faa9-4c2e-8290-491350546663-tls-certs\") on node \"ip-10-0-140-107.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:43:05.587618 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:43:05.587592 4939 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/83b38883-2e7a-444d-aa68-52be4a25ae66-tmp-dir\") on node \"ip-10-0-140-107.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:43:05.587618 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:43:05.587606 4939 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a397931b-faa9-4c2e-8290-491350546663-dshm\") on node \"ip-10-0-140-107.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:43:05.587618 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:43:05.587620 4939 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vnpdj\" (UniqueName: \"kubernetes.io/projected/a397931b-faa9-4c2e-8290-491350546663-kube-api-access-vnpdj\") on node \"ip-10-0-140-107.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:43:05.588003 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:43:05.587635 4939 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a397931b-faa9-4c2e-8290-491350546663-model-cache\") on node \"ip-10-0-140-107.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:43:05.588003 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:43:05.587649 4939 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a397931b-faa9-4c2e-8290-491350546663-tmp-dir\") on node \"ip-10-0-140-107.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:43:05.589549 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:43:05.589518 4939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/83b38883-2e7a-444d-aa68-52be4a25ae66-dshm" pod "83b38883-2e7a-444d-aa68-52be4a25ae66" (UID: "83b38883-2e7a-444d-aa68-52be4a25ae66"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 01:43:05.589658 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:43:05.589555 4939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/83b38883-2e7a-444d-aa68-52be4a25ae66-kube-api-access-9qxj6" pod "83b38883-2e7a-444d-aa68-52be4a25ae66" (UID: "83b38883-2e7a-444d-aa68-52be4a25ae66"). InnerVolumeSpecName "kube-api-access-9qxj6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 01:43:05.589944 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:43:05.589896 4939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/83b38883-2e7a-444d-aa68-52be4a25ae66-tls-certs" pod "83b38883-2e7a-444d-aa68-52be4a25ae66" (UID: "83b38883-2e7a-444d-aa68-52be4a25ae66"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 01:43:05.688401 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:43:05.688318 4939 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/83b38883-2e7a-444d-aa68-52be4a25ae66-tls-certs\") on node \"ip-10-0-140-107.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:43:05.688401 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:43:05.688349 4939 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9qxj6\" (UniqueName: \"kubernetes.io/projected/83b38883-2e7a-444d-aa68-52be4a25ae66-kube-api-access-9qxj6\") on node \"ip-10-0-140-107.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:43:05.688401 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:43:05.688360 4939 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/83b38883-2e7a-444d-aa68-52be4a25ae66-model-cache\") on node \"ip-10-0-140-107.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:43:05.688401 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:43:05.688369 4939 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/83b38883-2e7a-444d-aa68-52be4a25ae66-dshm\") on node \"ip-10-0-140-107.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:43:06.104051 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:43:06.104013 4939 generic.go:356] "Generic (PLEG): container finished" podID="a397931b-faa9-4c2e-8290-491350546663" containerID="4d89519b7718515ab659a365ce964abdde58e902233a6a73f99bb5a317f2eb19" exitCode=0 Nov 03 01:43:06.104241 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:43:06.104057 4939 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-88bbf7/distinct-2-w1-88bbf7-kserve-5457cdcdbd-l6tfd" Nov 03 01:43:06.104241 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:43:06.104094 4939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-88bbf7/distinct-2-w1-88bbf7-kserve-5457cdcdbd-l6tfd" event={"ID":"a397931b-faa9-4c2e-8290-491350546663","Type":"ContainerDied","Data":"4d89519b7718515ab659a365ce964abdde58e902233a6a73f99bb5a317f2eb19"} Nov 03 01:43:06.104241 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:43:06.104138 4939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-88bbf7/distinct-2-w1-88bbf7-kserve-5457cdcdbd-l6tfd" event={"ID":"a397931b-faa9-4c2e-8290-491350546663","Type":"ContainerDied","Data":"04c5c3048d85531cb0f5ce68c456ca67a406d641e5cd4d43f6f3dad0173ed240"} Nov 03 01:43:06.104241 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:43:06.104159 4939 scope.go:122] "RemoveContainer" containerID="4d89519b7718515ab659a365ce964abdde58e902233a6a73f99bb5a317f2eb19" Nov 03 01:43:06.105666 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:43:06.105640 4939 generic.go:356] "Generic (PLEG): container finished" podID="83b38883-2e7a-444d-aa68-52be4a25ae66" containerID="f6ef9c5f02772524826ae140c107b0209f1ee2cb412948574b7ba2a235ab0860" exitCode=0 Nov 03 01:43:06.105797 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:43:06.105679 4939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-88bbf7/simulator-w1-88bbf7-kserve-5fbb669f8f-42hgd" event={"ID":"83b38883-2e7a-444d-aa68-52be4a25ae66","Type":"ContainerDied","Data":"f6ef9c5f02772524826ae140c107b0209f1ee2cb412948574b7ba2a235ab0860"} Nov 03 01:43:06.105797 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:43:06.105710 4939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-88bbf7/simulator-w1-88bbf7-kserve-5fbb669f8f-42hgd" event={"ID":"83b38883-2e7a-444d-aa68-52be4a25ae66","Type":"ContainerDied","Data":"c12115237becab031644b60fce23c7384f55c7176eb70bc9a37d3bd6bd484b1e"} Nov 03 01:43:06.105797 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:43:06.105767 4939 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-88bbf7/simulator-w1-88bbf7-kserve-5fbb669f8f-42hgd" Nov 03 01:43:06.113702 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:43:06.113686 4939 scope.go:122] "RemoveContainer" containerID="4d89519b7718515ab659a365ce964abdde58e902233a6a73f99bb5a317f2eb19" Nov 03 01:43:06.113988 ip-10-0-140-107 kubenswrapper[4939]: E1103 01:43:06.113967 4939 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4d89519b7718515ab659a365ce964abdde58e902233a6a73f99bb5a317f2eb19\": container with ID starting with 4d89519b7718515ab659a365ce964abdde58e902233a6a73f99bb5a317f2eb19 not found: ID does not exist" containerID="4d89519b7718515ab659a365ce964abdde58e902233a6a73f99bb5a317f2eb19" Nov 03 01:43:06.114080 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:43:06.113994 4939 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4d89519b7718515ab659a365ce964abdde58e902233a6a73f99bb5a317f2eb19"} err="failed to get container status \"4d89519b7718515ab659a365ce964abdde58e902233a6a73f99bb5a317f2eb19\": rpc error: code = NotFound desc = could not find container \"4d89519b7718515ab659a365ce964abdde58e902233a6a73f99bb5a317f2eb19\": container with ID starting with 4d89519b7718515ab659a365ce964abdde58e902233a6a73f99bb5a317f2eb19 not found: ID does not exist" Nov 03 01:43:06.114080 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:43:06.114012 4939 scope.go:122] "RemoveContainer" containerID="f6ef9c5f02772524826ae140c107b0209f1ee2cb412948574b7ba2a235ab0860" Nov 03 01:43:06.121669 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:43:06.121648 4939 scope.go:122] "RemoveContainer" containerID="f6ef9c5f02772524826ae140c107b0209f1ee2cb412948574b7ba2a235ab0860" Nov 03 01:43:06.122032 ip-10-0-140-107 kubenswrapper[4939]: E1103 01:43:06.121995 4939 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f6ef9c5f02772524826ae140c107b0209f1ee2cb412948574b7ba2a235ab0860\": container with ID starting with f6ef9c5f02772524826ae140c107b0209f1ee2cb412948574b7ba2a235ab0860 not found: ID does not exist" containerID="f6ef9c5f02772524826ae140c107b0209f1ee2cb412948574b7ba2a235ab0860" Nov 03 01:43:06.122166 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:43:06.122037 4939 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f6ef9c5f02772524826ae140c107b0209f1ee2cb412948574b7ba2a235ab0860"} err="failed to get container status \"f6ef9c5f02772524826ae140c107b0209f1ee2cb412948574b7ba2a235ab0860\": rpc error: code = NotFound desc = could not find container \"f6ef9c5f02772524826ae140c107b0209f1ee2cb412948574b7ba2a235ab0860\": container with ID starting with f6ef9c5f02772524826ae140c107b0209f1ee2cb412948574b7ba2a235ab0860 not found: ID does not exist" Nov 03 01:43:06.123529 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:43:06.123511 4939 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w1-88bbf7/simulator-w1-88bbf7-kserve-5fbb669f8f-42hgd"] Nov 03 01:43:06.127001 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:43:06.126978 4939 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w1-88bbf7/simulator-w1-88bbf7-kserve-5fbb669f8f-42hgd"] Nov 03 01:43:06.137672 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:43:06.137652 4939 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w1-88bbf7/distinct-2-w1-88bbf7-kserve-5457cdcdbd-l6tfd"] Nov 03 01:43:06.142545 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:43:06.142525 4939 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w1-88bbf7/distinct-2-w1-88bbf7-kserve-5457cdcdbd-l6tfd"] Nov 03 01:43:06.457555 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:43:06.457472 4939 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-88bbf7-openshift-default-574f47fff7-6qqxz" Nov 03 01:43:06.457788 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:43:06.457761 4939 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-88bbf7-openshift-default-574f47fff7-6qqxz" Nov 03 01:43:06.458382 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:43:06.458357 4939 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-88bbf7-openshift-default-574f47fff7-6qqxz" Nov 03 01:43:07.599144 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:43:07.599108 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:43:07.866488 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:43:07.866411 4939 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="83b38883-2e7a-444d-aa68-52be4a25ae66" path="/var/lib/kubelet/pods/83b38883-2e7a-444d-aa68-52be4a25ae66/volumes" Nov 03 01:43:07.866838 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:43:07.866825 4939 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a397931b-faa9-4c2e-8290-491350546663" path="/var/lib/kubelet/pods/a397931b-faa9-4c2e-8290-491350546663/volumes" Nov 03 01:43:08.292200 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:43:08.292107 4939 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-4pw7v"] Nov 03 01:43:08.767843 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:43:08.767774 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:43:08.879709 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:43:08.879670 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:43:13.168685 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:43:13.168650 4939 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-88bbf7-openshift-default-574f47fff7-x96kz"] Nov 03 01:43:13.169346 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:43:13.169329 4939 cpu_manager.go:414] "containerMap: removing container" podUID="7f05b5e8-35b2-4f88-b6f2-ff4b645a34c5" containerName="revoke-keys" Nov 03 01:43:13.169346 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:43:13.169347 4939 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7f05b5e8-35b2-4f88-b6f2-ff4b645a34c5" containerName="revoke-keys" Nov 03 01:43:13.169494 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:43:13.169363 4939 cpu_manager.go:414] "containerMap: removing container" podUID="83b38883-2e7a-444d-aa68-52be4a25ae66" containerName="main" Nov 03 01:43:13.169494 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:43:13.169370 4939 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="83b38883-2e7a-444d-aa68-52be4a25ae66" containerName="main" Nov 03 01:43:13.169494 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:43:13.169390 4939 cpu_manager.go:414] "containerMap: removing container" podUID="a397931b-faa9-4c2e-8290-491350546663" containerName="main" Nov 03 01:43:13.169494 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:43:13.169397 4939 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a397931b-faa9-4c2e-8290-491350546663" containerName="main" Nov 03 01:43:13.169656 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:43:13.169502 4939 memory_manager.go:365] "RemoveStaleState removing state" podUID="7f05b5e8-35b2-4f88-b6f2-ff4b645a34c5" containerName="revoke-keys" Nov 03 01:43:13.169656 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:43:13.169518 4939 memory_manager.go:365] "RemoveStaleState removing state" podUID="83b38883-2e7a-444d-aa68-52be4a25ae66" containerName="main" Nov 03 01:43:13.169656 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:43:13.169529 4939 memory_manager.go:365] "RemoveStaleState removing state" podUID="a397931b-faa9-4c2e-8290-491350546663" containerName="main" Nov 03 01:43:13.174643 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:43:13.174613 4939 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-88bbf7-openshift-default-574f47fff7-x96kz" Nov 03 01:43:13.182438 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:43:13.182415 4939 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-88bbf7-openshift-default-574f47fff7-x96kz"] Nov 03 01:43:13.260982 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:43:13.260937 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0baee2a2-f093-4adc-ab4b-8541f498abf3-istio-podinfo\") pod \"e2e-worker-w1-88bbf7-openshift-default-574f47fff7-x96kz\" (UID: \"0baee2a2-f093-4adc-ab4b-8541f498abf3\") " pod="openshift-ingress/e2e-worker-w1-88bbf7-openshift-default-574f47fff7-x96kz" Nov 03 01:43:13.260982 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:43:13.260985 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0baee2a2-f093-4adc-ab4b-8541f498abf3-istiod-ca-cert\") pod \"e2e-worker-w1-88bbf7-openshift-default-574f47fff7-x96kz\" (UID: \"0baee2a2-f093-4adc-ab4b-8541f498abf3\") " pod="openshift-ingress/e2e-worker-w1-88bbf7-openshift-default-574f47fff7-x96kz" Nov 03 01:43:13.261200 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:43:13.261085 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0baee2a2-f093-4adc-ab4b-8541f498abf3-workload-certs\") pod \"e2e-worker-w1-88bbf7-openshift-default-574f47fff7-x96kz\" (UID: \"0baee2a2-f093-4adc-ab4b-8541f498abf3\") " pod="openshift-ingress/e2e-worker-w1-88bbf7-openshift-default-574f47fff7-x96kz" Nov 03 01:43:13.261200 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:43:13.261127 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0baee2a2-f093-4adc-ab4b-8541f498abf3-credential-socket\") pod \"e2e-worker-w1-88bbf7-openshift-default-574f47fff7-x96kz\" (UID: \"0baee2a2-f093-4adc-ab4b-8541f498abf3\") " pod="openshift-ingress/e2e-worker-w1-88bbf7-openshift-default-574f47fff7-x96kz" Nov 03 01:43:13.261279 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:43:13.261196 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mrggf\" (UniqueName: \"kubernetes.io/projected/0baee2a2-f093-4adc-ab4b-8541f498abf3-kube-api-access-mrggf\") pod \"e2e-worker-w1-88bbf7-openshift-default-574f47fff7-x96kz\" (UID: \"0baee2a2-f093-4adc-ab4b-8541f498abf3\") " pod="openshift-ingress/e2e-worker-w1-88bbf7-openshift-default-574f47fff7-x96kz" Nov 03 01:43:13.261279 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:43:13.261226 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0baee2a2-f093-4adc-ab4b-8541f498abf3-istio-token\") pod \"e2e-worker-w1-88bbf7-openshift-default-574f47fff7-x96kz\" (UID: \"0baee2a2-f093-4adc-ab4b-8541f498abf3\") " pod="openshift-ingress/e2e-worker-w1-88bbf7-openshift-default-574f47fff7-x96kz" Nov 03 01:43:13.261351 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:43:13.261317 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0baee2a2-f093-4adc-ab4b-8541f498abf3-workload-socket\") pod \"e2e-worker-w1-88bbf7-openshift-default-574f47fff7-x96kz\" (UID: \"0baee2a2-f093-4adc-ab4b-8541f498abf3\") " pod="openshift-ingress/e2e-worker-w1-88bbf7-openshift-default-574f47fff7-x96kz" Nov 03 01:43:13.261351 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:43:13.261337 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0baee2a2-f093-4adc-ab4b-8541f498abf3-istio-data\") pod \"e2e-worker-w1-88bbf7-openshift-default-574f47fff7-x96kz\" (UID: \"0baee2a2-f093-4adc-ab4b-8541f498abf3\") " pod="openshift-ingress/e2e-worker-w1-88bbf7-openshift-default-574f47fff7-x96kz" Nov 03 01:43:13.261428 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:43:13.261394 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0baee2a2-f093-4adc-ab4b-8541f498abf3-istio-envoy\") pod \"e2e-worker-w1-88bbf7-openshift-default-574f47fff7-x96kz\" (UID: \"0baee2a2-f093-4adc-ab4b-8541f498abf3\") " pod="openshift-ingress/e2e-worker-w1-88bbf7-openshift-default-574f47fff7-x96kz" Nov 03 01:43:13.362104 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:43:13.362072 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0baee2a2-f093-4adc-ab4b-8541f498abf3-istio-podinfo\") pod \"e2e-worker-w1-88bbf7-openshift-default-574f47fff7-x96kz\" (UID: \"0baee2a2-f093-4adc-ab4b-8541f498abf3\") " pod="openshift-ingress/e2e-worker-w1-88bbf7-openshift-default-574f47fff7-x96kz" Nov 03 01:43:13.362104 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:43:13.362103 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0baee2a2-f093-4adc-ab4b-8541f498abf3-istiod-ca-cert\") pod \"e2e-worker-w1-88bbf7-openshift-default-574f47fff7-x96kz\" (UID: \"0baee2a2-f093-4adc-ab4b-8541f498abf3\") " pod="openshift-ingress/e2e-worker-w1-88bbf7-openshift-default-574f47fff7-x96kz" Nov 03 01:43:13.362366 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:43:13.362133 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0baee2a2-f093-4adc-ab4b-8541f498abf3-workload-certs\") pod \"e2e-worker-w1-88bbf7-openshift-default-574f47fff7-x96kz\" (UID: \"0baee2a2-f093-4adc-ab4b-8541f498abf3\") " pod="openshift-ingress/e2e-worker-w1-88bbf7-openshift-default-574f47fff7-x96kz" Nov 03 01:43:13.362366 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:43:13.362155 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0baee2a2-f093-4adc-ab4b-8541f498abf3-credential-socket\") pod \"e2e-worker-w1-88bbf7-openshift-default-574f47fff7-x96kz\" (UID: \"0baee2a2-f093-4adc-ab4b-8541f498abf3\") " pod="openshift-ingress/e2e-worker-w1-88bbf7-openshift-default-574f47fff7-x96kz" Nov 03 01:43:13.362366 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:43:13.362188 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mrggf\" (UniqueName: \"kubernetes.io/projected/0baee2a2-f093-4adc-ab4b-8541f498abf3-kube-api-access-mrggf\") pod \"e2e-worker-w1-88bbf7-openshift-default-574f47fff7-x96kz\" (UID: \"0baee2a2-f093-4adc-ab4b-8541f498abf3\") " pod="openshift-ingress/e2e-worker-w1-88bbf7-openshift-default-574f47fff7-x96kz" Nov 03 01:43:13.362366 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:43:13.362207 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0baee2a2-f093-4adc-ab4b-8541f498abf3-istio-token\") pod \"e2e-worker-w1-88bbf7-openshift-default-574f47fff7-x96kz\" (UID: \"0baee2a2-f093-4adc-ab4b-8541f498abf3\") " pod="openshift-ingress/e2e-worker-w1-88bbf7-openshift-default-574f47fff7-x96kz" Nov 03 01:43:13.362366 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:43:13.362268 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0baee2a2-f093-4adc-ab4b-8541f498abf3-workload-socket\") pod \"e2e-worker-w1-88bbf7-openshift-default-574f47fff7-x96kz\" (UID: \"0baee2a2-f093-4adc-ab4b-8541f498abf3\") " pod="openshift-ingress/e2e-worker-w1-88bbf7-openshift-default-574f47fff7-x96kz" Nov 03 01:43:13.362366 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:43:13.362295 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0baee2a2-f093-4adc-ab4b-8541f498abf3-istio-data\") pod \"e2e-worker-w1-88bbf7-openshift-default-574f47fff7-x96kz\" (UID: \"0baee2a2-f093-4adc-ab4b-8541f498abf3\") " pod="openshift-ingress/e2e-worker-w1-88bbf7-openshift-default-574f47fff7-x96kz" Nov 03 01:43:13.362366 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:43:13.362325 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0baee2a2-f093-4adc-ab4b-8541f498abf3-istio-envoy\") pod \"e2e-worker-w1-88bbf7-openshift-default-574f47fff7-x96kz\" (UID: \"0baee2a2-f093-4adc-ab4b-8541f498abf3\") " pod="openshift-ingress/e2e-worker-w1-88bbf7-openshift-default-574f47fff7-x96kz" Nov 03 01:43:13.362794 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:43:13.362611 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0baee2a2-f093-4adc-ab4b-8541f498abf3-workload-certs\") pod \"e2e-worker-w1-88bbf7-openshift-default-574f47fff7-x96kz\" (UID: \"0baee2a2-f093-4adc-ab4b-8541f498abf3\") " pod="openshift-ingress/e2e-worker-w1-88bbf7-openshift-default-574f47fff7-x96kz" Nov 03 01:43:13.362794 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:43:13.362665 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0baee2a2-f093-4adc-ab4b-8541f498abf3-credential-socket\") pod \"e2e-worker-w1-88bbf7-openshift-default-574f47fff7-x96kz\" (UID: \"0baee2a2-f093-4adc-ab4b-8541f498abf3\") " pod="openshift-ingress/e2e-worker-w1-88bbf7-openshift-default-574f47fff7-x96kz" Nov 03 01:43:13.362794 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:43:13.362779 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0baee2a2-f093-4adc-ab4b-8541f498abf3-workload-socket\") pod \"e2e-worker-w1-88bbf7-openshift-default-574f47fff7-x96kz\" (UID: \"0baee2a2-f093-4adc-ab4b-8541f498abf3\") " pod="openshift-ingress/e2e-worker-w1-88bbf7-openshift-default-574f47fff7-x96kz" Nov 03 01:43:13.362990 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:43:13.362886 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0baee2a2-f093-4adc-ab4b-8541f498abf3-istiod-ca-cert\") pod \"e2e-worker-w1-88bbf7-openshift-default-574f47fff7-x96kz\" (UID: \"0baee2a2-f093-4adc-ab4b-8541f498abf3\") " pod="openshift-ingress/e2e-worker-w1-88bbf7-openshift-default-574f47fff7-x96kz" Nov 03 01:43:13.362990 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:43:13.362922 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0baee2a2-f093-4adc-ab4b-8541f498abf3-istio-data\") pod \"e2e-worker-w1-88bbf7-openshift-default-574f47fff7-x96kz\" (UID: \"0baee2a2-f093-4adc-ab4b-8541f498abf3\") " pod="openshift-ingress/e2e-worker-w1-88bbf7-openshift-default-574f47fff7-x96kz" Nov 03 01:43:13.364636 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:43:13.364614 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0baee2a2-f093-4adc-ab4b-8541f498abf3-istio-podinfo\") pod \"e2e-worker-w1-88bbf7-openshift-default-574f47fff7-x96kz\" (UID: \"0baee2a2-f093-4adc-ab4b-8541f498abf3\") " pod="openshift-ingress/e2e-worker-w1-88bbf7-openshift-default-574f47fff7-x96kz" Nov 03 01:43:13.364636 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:43:13.364625 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0baee2a2-f093-4adc-ab4b-8541f498abf3-istio-envoy\") pod \"e2e-worker-w1-88bbf7-openshift-default-574f47fff7-x96kz\" (UID: \"0baee2a2-f093-4adc-ab4b-8541f498abf3\") " pod="openshift-ingress/e2e-worker-w1-88bbf7-openshift-default-574f47fff7-x96kz" Nov 03 01:43:13.376559 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:43:13.376527 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mrggf\" (UniqueName: \"kubernetes.io/projected/0baee2a2-f093-4adc-ab4b-8541f498abf3-kube-api-access-mrggf\") pod \"e2e-worker-w1-88bbf7-openshift-default-574f47fff7-x96kz\" (UID: \"0baee2a2-f093-4adc-ab4b-8541f498abf3\") " pod="openshift-ingress/e2e-worker-w1-88bbf7-openshift-default-574f47fff7-x96kz" Nov 03 01:43:13.381109 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:43:13.381087 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0baee2a2-f093-4adc-ab4b-8541f498abf3-istio-token\") pod \"e2e-worker-w1-88bbf7-openshift-default-574f47fff7-x96kz\" (UID: \"0baee2a2-f093-4adc-ab4b-8541f498abf3\") " pod="openshift-ingress/e2e-worker-w1-88bbf7-openshift-default-574f47fff7-x96kz" Nov 03 01:43:13.484611 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:43:13.484537 4939 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-4pw7v"] Nov 03 01:43:13.489024 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:43:13.489004 4939 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-88bbf7-openshift-default-574f47fff7-x96kz" Nov 03 01:43:13.765801 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:43:13.765774 4939 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-88bbf7-openshift-default-574f47fff7-x96kz"] Nov 03 01:43:13.768265 ip-10-0-140-107 kubenswrapper[4939]: W1103 01:43:13.768241 4939 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0baee2a2_f093_4adc_ab4b_8541f498abf3.slice/crio-73951df182b7bf8f16ab31f7282bf473a958257c388ef6a3bff351ad04b34ff1 WatchSource:0}: Error finding container 73951df182b7bf8f16ab31f7282bf473a958257c388ef6a3bff351ad04b34ff1: Status 404 returned error can't find the container with id 73951df182b7bf8f16ab31f7282bf473a958257c388ef6a3bff351ad04b34ff1 Nov 03 01:43:13.770433 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:43:13.770396 4939 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 01:43:13.770554 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:43:13.770480 4939 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 01:43:13.770554 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:43:13.770527 4939 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 01:43:14.153931 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:43:14.151975 4939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-88bbf7-openshift-default-574f47fff7-x96kz" event={"ID":"0baee2a2-f093-4adc-ab4b-8541f498abf3","Type":"ContainerStarted","Data":"62b015a9f97f8415af0b1dc95239d03051ff0e6e4c1fcdda37a2b4b0d510945f"} Nov 03 01:43:14.153931 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:43:14.152059 4939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-88bbf7-openshift-default-574f47fff7-x96kz" event={"ID":"0baee2a2-f093-4adc-ab4b-8541f498abf3","Type":"ContainerStarted","Data":"73951df182b7bf8f16ab31f7282bf473a958257c388ef6a3bff351ad04b34ff1"} Nov 03 01:43:14.179522 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:43:14.179469 4939 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-88bbf7-openshift-default-574f47fff7-x96kz" podStartSLOduration=1.17945348 podStartE2EDuration="1.17945348s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 01:43:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 01:43:14.17615377 +0000 UTC m=+3742.440488715" watchObservedRunningTime="2026-11-03 01:43:14.17945348 +0000 UTC m=+3742.443788424" Nov 03 01:43:14.490119 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:43:14.490030 4939 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-88bbf7-openshift-default-574f47fff7-x96kz" Nov 03 01:43:14.491519 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:43:14.491490 4939 patch_prober.go:29] interesting pod/e2e-worker-w1-88bbf7-openshift-default-574f47fff7-x96kz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.93:15021/healthz/ready\": dial tcp 10.132.1.93:15021: connect: connection refused" start-of-body= Nov 03 01:43:14.491664 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:43:14.491545 4939 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-88bbf7-openshift-default-574f47fff7-x96kz" podUID="0baee2a2-f093-4adc-ab4b-8541f498abf3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.93:15021/healthz/ready\": dial tcp 10.132.1.93:15021: connect: connection refused" Nov 03 01:43:15.489647 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:43:15.489610 4939 patch_prober.go:29] interesting pod/e2e-worker-w1-88bbf7-openshift-default-574f47fff7-x96kz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.93:15021/healthz/ready\": dial tcp 10.132.1.93:15021: connect: connection refused" start-of-body= Nov 03 01:43:15.490099 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:43:15.489665 4939 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-88bbf7-openshift-default-574f47fff7-x96kz" podUID="0baee2a2-f093-4adc-ab4b-8541f498abf3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.93:15021/healthz/ready\": dial tcp 10.132.1.93:15021: connect: connection refused" Nov 03 01:43:16.489544 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:43:16.489507 4939 patch_prober.go:29] interesting pod/e2e-worker-w1-88bbf7-openshift-default-574f47fff7-x96kz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.93:15021/healthz/ready\": dial tcp 10.132.1.93:15021: connect: connection refused" start-of-body= Nov 03 01:43:16.489742 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:43:16.489563 4939 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-88bbf7-openshift-default-574f47fff7-x96kz" podUID="0baee2a2-f093-4adc-ab4b-8541f498abf3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.93:15021/healthz/ready\": dial tcp 10.132.1.93:15021: connect: connection refused" Nov 03 01:43:17.489368 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:43:17.489331 4939 patch_prober.go:29] interesting pod/e2e-worker-w1-88bbf7-openshift-default-574f47fff7-x96kz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.93:15021/healthz/ready\": dial tcp 10.132.1.93:15021: connect: connection refused" start-of-body= Nov 03 01:43:17.489595 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:43:17.489388 4939 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-88bbf7-openshift-default-574f47fff7-x96kz" podUID="0baee2a2-f093-4adc-ab4b-8541f498abf3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.93:15021/healthz/ready\": dial tcp 10.132.1.93:15021: connect: connection refused" Nov 03 01:43:18.031228 ip-10-0-140-107 kubenswrapper[4939]: E1103 01:43:18.031188 4939 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-sthjh" containerName="extractor" Nov 03 01:43:18.489405 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:43:18.489370 4939 patch_prober.go:29] interesting pod/e2e-worker-w1-88bbf7-openshift-default-574f47fff7-x96kz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.93:15021/healthz/ready\": dial tcp 10.132.1.93:15021: connect: connection refused" start-of-body= Nov 03 01:43:18.489608 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:43:18.489425 4939 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-88bbf7-openshift-default-574f47fff7-x96kz" podUID="0baee2a2-f093-4adc-ab4b-8541f498abf3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.93:15021/healthz/ready\": dial tcp 10.132.1.93:15021: connect: connection refused" Nov 03 01:43:19.490158 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:43:19.490115 4939 patch_prober.go:29] interesting pod/e2e-worker-w1-88bbf7-openshift-default-574f47fff7-x96kz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.93:15021/healthz/ready\": dial tcp 10.132.1.93:15021: connect: connection refused" start-of-body= Nov 03 01:43:19.490665 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:43:19.490177 4939 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-88bbf7-openshift-default-574f47fff7-x96kz" podUID="0baee2a2-f093-4adc-ab4b-8541f498abf3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.93:15021/healthz/ready\": dial tcp 10.132.1.93:15021: connect: connection refused" Nov 03 01:43:20.494191 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:43:20.494159 4939 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-88bbf7-openshift-default-574f47fff7-x96kz" Nov 03 01:43:20.494595 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:43:20.494429 4939 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-88bbf7-openshift-default-574f47fff7-x96kz" Nov 03 01:43:20.495111 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:43:20.495091 4939 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-88bbf7-openshift-default-574f47fff7-x96kz" Nov 03 01:43:26.383234 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:43:26.383188 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:43:26.500745 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:43:26.500707 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:43:27.948635 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:43:27.948601 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:43:28.842689 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:43:28.842653 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:43:29.254695 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:43:29.254600 4939 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-88bbf7-openshift-default-574f47fff7-x96kz"] Nov 03 01:43:29.255219 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:43:29.254742 4939 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-88bbf7-openshift-default-574f47fff7-6qqxz"] Nov 03 01:43:29.255219 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:43:29.255053 4939 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-88bbf7-openshift-default-574f47fff7-6qqxz" podUID="3ceb505f-63e4-427e-aad1-d40ca8062b2b" containerName="istio-proxy" containerID="cri-o://1360b1db3879eb318e46aad725095dba682ba897cb83beae80a8a8a488c63b87" gracePeriod=30 Nov 03 01:43:29.256194 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:43:29.256133 4939 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-88bbf7-openshift-default-574f47fff7-x96kz" podUID="0baee2a2-f093-4adc-ab4b-8541f498abf3" containerName="istio-proxy" containerID="cri-o://62b015a9f97f8415af0b1dc95239d03051ff0e6e4c1fcdda37a2b4b0d510945f" gracePeriod=30 Nov 03 01:43:29.306475 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:43:29.306445 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:43:31.791605 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:43:31.791575 4939 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-7c9b87d574-4pw7v_e1b47e77-a4ab-4d61-ace8-deeea43ea944/limitador/0.log" Nov 03 01:43:33.083320 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:43:33.083277 4939 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_ai-gateway-controller-75bbf6968c-fz65x_f690a187-3701-4c28-a425-e5e1f447b650/manager/0.log" Nov 03 01:43:33.357301 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:43:33.357215 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:43:33.474517 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:43:33.474455 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:43:34.223239 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:43:34.223199 4939 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-7b85f9594f-9gw68_071dd59f-2a89-48c6-aea0-2a552f5bae08/manager/0.log" Nov 03 01:43:34.536516 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:43:34.536428 4939 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-88bbf7-openshift-default-574f47fff7-x96kz" Nov 03 01:43:34.558655 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:43:34.558632 4939 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-88bbf7-openshift-default-574f47fff7-6qqxz" Nov 03 01:43:34.685970 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:43:34.685934 4939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0baee2a2-f093-4adc-ab4b-8541f498abf3-kube-api-access-mrggf\" (UniqueName: \"kubernetes.io/projected/0baee2a2-f093-4adc-ab4b-8541f498abf3-kube-api-access-mrggf\") pod \"0baee2a2-f093-4adc-ab4b-8541f498abf3\" (UID: \"0baee2a2-f093-4adc-ab4b-8541f498abf3\") " Nov 03 01:43:34.686161 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:43:34.686009 4939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/0baee2a2-f093-4adc-ab4b-8541f498abf3-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0baee2a2-f093-4adc-ab4b-8541f498abf3-istio-podinfo\") pod \"0baee2a2-f093-4adc-ab4b-8541f498abf3\" (UID: \"0baee2a2-f093-4adc-ab4b-8541f498abf3\") " Nov 03 01:43:34.686161 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:43:34.686040 4939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3ceb505f-63e4-427e-aad1-d40ca8062b2b-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3ceb505f-63e4-427e-aad1-d40ca8062b2b-istio-data\") pod \"3ceb505f-63e4-427e-aad1-d40ca8062b2b\" (UID: \"3ceb505f-63e4-427e-aad1-d40ca8062b2b\") " Nov 03 01:43:34.686161 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:43:34.686073 4939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0baee2a2-f093-4adc-ab4b-8541f498abf3-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0baee2a2-f093-4adc-ab4b-8541f498abf3-istio-envoy\") pod \"0baee2a2-f093-4adc-ab4b-8541f498abf3\" (UID: \"0baee2a2-f093-4adc-ab4b-8541f498abf3\") " Nov 03 01:43:34.686161 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:43:34.686094 4939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3ceb505f-63e4-427e-aad1-d40ca8062b2b-istio-token\" (UniqueName: \"kubernetes.io/projected/3ceb505f-63e4-427e-aad1-d40ca8062b2b-istio-token\") pod \"3ceb505f-63e4-427e-aad1-d40ca8062b2b\" (UID: \"3ceb505f-63e4-427e-aad1-d40ca8062b2b\") " Nov 03 01:43:34.686161 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:43:34.686135 4939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0baee2a2-f093-4adc-ab4b-8541f498abf3-istio-token\" (UniqueName: \"kubernetes.io/projected/0baee2a2-f093-4adc-ab4b-8541f498abf3-istio-token\") pod \"0baee2a2-f093-4adc-ab4b-8541f498abf3\" (UID: \"0baee2a2-f093-4adc-ab4b-8541f498abf3\") " Nov 03 01:43:34.686485 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:43:34.686191 4939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3ceb505f-63e4-427e-aad1-d40ca8062b2b-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3ceb505f-63e4-427e-aad1-d40ca8062b2b-istiod-ca-cert\") pod \"3ceb505f-63e4-427e-aad1-d40ca8062b2b\" (UID: \"3ceb505f-63e4-427e-aad1-d40ca8062b2b\") " Nov 03 01:43:34.686485 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:43:34.686242 4939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0baee2a2-f093-4adc-ab4b-8541f498abf3-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0baee2a2-f093-4adc-ab4b-8541f498abf3-workload-certs\") pod \"0baee2a2-f093-4adc-ab4b-8541f498abf3\" (UID: \"0baee2a2-f093-4adc-ab4b-8541f498abf3\") " Nov 03 01:43:34.686485 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:43:34.686269 4939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3ceb505f-63e4-427e-aad1-d40ca8062b2b-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3ceb505f-63e4-427e-aad1-d40ca8062b2b-workload-certs\") pod \"3ceb505f-63e4-427e-aad1-d40ca8062b2b\" (UID: \"3ceb505f-63e4-427e-aad1-d40ca8062b2b\") " Nov 03 01:43:34.686485 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:43:34.686293 4939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/3ceb505f-63e4-427e-aad1-d40ca8062b2b-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3ceb505f-63e4-427e-aad1-d40ca8062b2b-istio-podinfo\") pod \"3ceb505f-63e4-427e-aad1-d40ca8062b2b\" (UID: \"3ceb505f-63e4-427e-aad1-d40ca8062b2b\") " Nov 03 01:43:34.686485 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:43:34.686323 4939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0baee2a2-f093-4adc-ab4b-8541f498abf3-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0baee2a2-f093-4adc-ab4b-8541f498abf3-istiod-ca-cert\") pod \"0baee2a2-f093-4adc-ab4b-8541f498abf3\" (UID: \"0baee2a2-f093-4adc-ab4b-8541f498abf3\") " Nov 03 01:43:34.686485 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:43:34.686354 4939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0baee2a2-f093-4adc-ab4b-8541f498abf3-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0baee2a2-f093-4adc-ab4b-8541f498abf3-credential-socket\") pod \"0baee2a2-f093-4adc-ab4b-8541f498abf3\" (UID: \"0baee2a2-f093-4adc-ab4b-8541f498abf3\") " Nov 03 01:43:34.686485 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:43:34.686386 4939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3ceb505f-63e4-427e-aad1-d40ca8062b2b-kube-api-access-zclss\" (UniqueName: \"kubernetes.io/projected/3ceb505f-63e4-427e-aad1-d40ca8062b2b-kube-api-access-zclss\") pod \"3ceb505f-63e4-427e-aad1-d40ca8062b2b\" (UID: \"3ceb505f-63e4-427e-aad1-d40ca8062b2b\") " Nov 03 01:43:34.686485 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:43:34.686462 4939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0baee2a2-f093-4adc-ab4b-8541f498abf3-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0baee2a2-f093-4adc-ab4b-8541f498abf3-istio-data\") pod \"0baee2a2-f093-4adc-ab4b-8541f498abf3\" (UID: \"0baee2a2-f093-4adc-ab4b-8541f498abf3\") " Nov 03 01:43:34.687083 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:43:34.686517 4939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3ceb505f-63e4-427e-aad1-d40ca8062b2b-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3ceb505f-63e4-427e-aad1-d40ca8062b2b-istio-envoy\") pod \"3ceb505f-63e4-427e-aad1-d40ca8062b2b\" (UID: \"3ceb505f-63e4-427e-aad1-d40ca8062b2b\") " Nov 03 01:43:34.687083 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:43:34.686527 4939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3ceb505f-63e4-427e-aad1-d40ca8062b2b-istio-data" pod "3ceb505f-63e4-427e-aad1-d40ca8062b2b" (UID: "3ceb505f-63e4-427e-aad1-d40ca8062b2b"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 01:43:34.687083 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:43:34.686556 4939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3ceb505f-63e4-427e-aad1-d40ca8062b2b-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3ceb505f-63e4-427e-aad1-d40ca8062b2b-workload-socket\") pod \"3ceb505f-63e4-427e-aad1-d40ca8062b2b\" (UID: \"3ceb505f-63e4-427e-aad1-d40ca8062b2b\") " Nov 03 01:43:34.687083 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:43:34.686600 4939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3ceb505f-63e4-427e-aad1-d40ca8062b2b-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3ceb505f-63e4-427e-aad1-d40ca8062b2b-credential-socket\") pod \"3ceb505f-63e4-427e-aad1-d40ca8062b2b\" (UID: \"3ceb505f-63e4-427e-aad1-d40ca8062b2b\") " Nov 03 01:43:34.687083 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:43:34.686630 4939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0baee2a2-f093-4adc-ab4b-8541f498abf3-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0baee2a2-f093-4adc-ab4b-8541f498abf3-workload-socket\") pod \"0baee2a2-f093-4adc-ab4b-8541f498abf3\" (UID: \"0baee2a2-f093-4adc-ab4b-8541f498abf3\") " Nov 03 01:43:34.687083 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:43:34.686752 4939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0baee2a2-f093-4adc-ab4b-8541f498abf3-workload-certs" pod "0baee2a2-f093-4adc-ab4b-8541f498abf3" (UID: "0baee2a2-f093-4adc-ab4b-8541f498abf3"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 01:43:34.687083 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:43:34.687038 4939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0baee2a2-f093-4adc-ab4b-8541f498abf3-workload-socket" pod "0baee2a2-f093-4adc-ab4b-8541f498abf3" (UID: "0baee2a2-f093-4adc-ab4b-8541f498abf3"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 01:43:34.687508 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:43:34.687117 4939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3ceb505f-63e4-427e-aad1-d40ca8062b2b-istiod-ca-cert" pod "3ceb505f-63e4-427e-aad1-d40ca8062b2b" (UID: "3ceb505f-63e4-427e-aad1-d40ca8062b2b"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 01:43:34.687508 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:43:34.687247 4939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3ceb505f-63e4-427e-aad1-d40ca8062b2b-workload-socket" pod "3ceb505f-63e4-427e-aad1-d40ca8062b2b" (UID: "3ceb505f-63e4-427e-aad1-d40ca8062b2b"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 01:43:34.687508 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:43:34.687422 4939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3ceb505f-63e4-427e-aad1-d40ca8062b2b-workload-certs" pod "3ceb505f-63e4-427e-aad1-d40ca8062b2b" (UID: "3ceb505f-63e4-427e-aad1-d40ca8062b2b"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 01:43:34.687508 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:43:34.687440 4939 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3ceb505f-63e4-427e-aad1-d40ca8062b2b-istio-data\") on node \"ip-10-0-140-107.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:43:34.687508 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:43:34.687447 4939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3ceb505f-63e4-427e-aad1-d40ca8062b2b-credential-socket" pod "3ceb505f-63e4-427e-aad1-d40ca8062b2b" (UID: "3ceb505f-63e4-427e-aad1-d40ca8062b2b"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 01:43:34.687508 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:43:34.687464 4939 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3ceb505f-63e4-427e-aad1-d40ca8062b2b-istiod-ca-cert\") on node \"ip-10-0-140-107.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:43:34.687508 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:43:34.687480 4939 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0baee2a2-f093-4adc-ab4b-8541f498abf3-workload-certs\") on node \"ip-10-0-140-107.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:43:34.687508 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:43:34.687495 4939 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3ceb505f-63e4-427e-aad1-d40ca8062b2b-workload-socket\") on node \"ip-10-0-140-107.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:43:34.687508 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:43:34.687510 4939 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0baee2a2-f093-4adc-ab4b-8541f498abf3-workload-socket\") on node \"ip-10-0-140-107.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:43:34.688069 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:43:34.687683 4939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0baee2a2-f093-4adc-ab4b-8541f498abf3-credential-socket" pod "0baee2a2-f093-4adc-ab4b-8541f498abf3" (UID: "0baee2a2-f093-4adc-ab4b-8541f498abf3"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 01:43:34.688069 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:43:34.687702 4939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0baee2a2-f093-4adc-ab4b-8541f498abf3-istio-data" pod "0baee2a2-f093-4adc-ab4b-8541f498abf3" (UID: "0baee2a2-f093-4adc-ab4b-8541f498abf3"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 01:43:34.688069 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:43:34.687713 4939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0baee2a2-f093-4adc-ab4b-8541f498abf3-istiod-ca-cert" pod "0baee2a2-f093-4adc-ab4b-8541f498abf3" (UID: "0baee2a2-f093-4adc-ab4b-8541f498abf3"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 01:43:34.688621 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:43:34.688594 4939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0baee2a2-f093-4adc-ab4b-8541f498abf3-kube-api-access-mrggf" pod "0baee2a2-f093-4adc-ab4b-8541f498abf3" (UID: "0baee2a2-f093-4adc-ab4b-8541f498abf3"). InnerVolumeSpecName "kube-api-access-mrggf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 01:43:34.688777 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:43:34.688755 4939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3ceb505f-63e4-427e-aad1-d40ca8062b2b-istio-token" pod "3ceb505f-63e4-427e-aad1-d40ca8062b2b" (UID: "3ceb505f-63e4-427e-aad1-d40ca8062b2b"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 01:43:34.689221 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:43:34.689201 4939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/0baee2a2-f093-4adc-ab4b-8541f498abf3-istio-podinfo" pod "0baee2a2-f093-4adc-ab4b-8541f498abf3" (UID: "0baee2a2-f093-4adc-ab4b-8541f498abf3"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 01:43:34.689451 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:43:34.689432 4939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0baee2a2-f093-4adc-ab4b-8541f498abf3-istio-envoy" pod "0baee2a2-f093-4adc-ab4b-8541f498abf3" (UID: "0baee2a2-f093-4adc-ab4b-8541f498abf3"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 01:43:34.689973 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:43:34.689946 4939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3ceb505f-63e4-427e-aad1-d40ca8062b2b-kube-api-access-zclss" pod "3ceb505f-63e4-427e-aad1-d40ca8062b2b" (UID: "3ceb505f-63e4-427e-aad1-d40ca8062b2b"). InnerVolumeSpecName "kube-api-access-zclss". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 01:43:34.690212 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:43:34.690189 4939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0baee2a2-f093-4adc-ab4b-8541f498abf3-istio-token" pod "0baee2a2-f093-4adc-ab4b-8541f498abf3" (UID: "0baee2a2-f093-4adc-ab4b-8541f498abf3"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 01:43:34.690351 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:43:34.690334 4939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3ceb505f-63e4-427e-aad1-d40ca8062b2b-istio-envoy" pod "3ceb505f-63e4-427e-aad1-d40ca8062b2b" (UID: "3ceb505f-63e4-427e-aad1-d40ca8062b2b"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 01:43:34.690473 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:43:34.690455 4939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/3ceb505f-63e4-427e-aad1-d40ca8062b2b-istio-podinfo" pod "3ceb505f-63e4-427e-aad1-d40ca8062b2b" (UID: "3ceb505f-63e4-427e-aad1-d40ca8062b2b"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 01:43:34.788344 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:43:34.788241 4939 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3ceb505f-63e4-427e-aad1-d40ca8062b2b-credential-socket\") on node \"ip-10-0-140-107.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:43:34.788344 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:43:34.788293 4939 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mrggf\" (UniqueName: \"kubernetes.io/projected/0baee2a2-f093-4adc-ab4b-8541f498abf3-kube-api-access-mrggf\") on node \"ip-10-0-140-107.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:43:34.788344 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:43:34.788305 4939 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0baee2a2-f093-4adc-ab4b-8541f498abf3-istio-podinfo\") on node \"ip-10-0-140-107.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:43:34.788344 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:43:34.788316 4939 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0baee2a2-f093-4adc-ab4b-8541f498abf3-istio-envoy\") on node \"ip-10-0-140-107.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:43:34.788344 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:43:34.788325 4939 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3ceb505f-63e4-427e-aad1-d40ca8062b2b-istio-token\") on node \"ip-10-0-140-107.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:43:34.788344 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:43:34.788333 4939 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0baee2a2-f093-4adc-ab4b-8541f498abf3-istio-token\") on node \"ip-10-0-140-107.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:43:34.788344 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:43:34.788342 4939 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3ceb505f-63e4-427e-aad1-d40ca8062b2b-workload-certs\") on node \"ip-10-0-140-107.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:43:34.788344 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:43:34.788351 4939 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3ceb505f-63e4-427e-aad1-d40ca8062b2b-istio-podinfo\") on node \"ip-10-0-140-107.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:43:34.788773 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:43:34.788359 4939 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0baee2a2-f093-4adc-ab4b-8541f498abf3-istiod-ca-cert\") on node \"ip-10-0-140-107.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:43:34.788773 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:43:34.788382 4939 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0baee2a2-f093-4adc-ab4b-8541f498abf3-credential-socket\") on node \"ip-10-0-140-107.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:43:34.788773 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:43:34.788391 4939 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zclss\" (UniqueName: \"kubernetes.io/projected/3ceb505f-63e4-427e-aad1-d40ca8062b2b-kube-api-access-zclss\") on node \"ip-10-0-140-107.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:43:34.788773 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:43:34.788399 4939 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0baee2a2-f093-4adc-ab4b-8541f498abf3-istio-data\") on node \"ip-10-0-140-107.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:43:34.788773 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:43:34.788408 4939 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3ceb505f-63e4-427e-aad1-d40ca8062b2b-istio-envoy\") on node \"ip-10-0-140-107.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:43:35.270567 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:43:35.270533 4939 generic.go:356] "Generic (PLEG): container finished" podID="3ceb505f-63e4-427e-aad1-d40ca8062b2b" containerID="1360b1db3879eb318e46aad725095dba682ba897cb83beae80a8a8a488c63b87" exitCode=0 Nov 03 01:43:35.271012 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:43:35.270662 4939 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-88bbf7-openshift-default-574f47fff7-6qqxz" Nov 03 01:43:35.271012 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:43:35.270971 4939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-88bbf7-openshift-default-574f47fff7-6qqxz" event={"ID":"3ceb505f-63e4-427e-aad1-d40ca8062b2b","Type":"ContainerDied","Data":"1360b1db3879eb318e46aad725095dba682ba897cb83beae80a8a8a488c63b87"} Nov 03 01:43:35.271147 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:43:35.271036 4939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-88bbf7-openshift-default-574f47fff7-6qqxz" event={"ID":"3ceb505f-63e4-427e-aad1-d40ca8062b2b","Type":"ContainerDied","Data":"d5425de7d92cd3d3a0d5a5973f5ba2e875f66a12ca25a25c3fc05ae301056f73"} Nov 03 01:43:35.271147 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:43:35.271063 4939 scope.go:122] "RemoveContainer" containerID="1360b1db3879eb318e46aad725095dba682ba897cb83beae80a8a8a488c63b87" Nov 03 01:43:35.272313 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:43:35.272292 4939 generic.go:356] "Generic (PLEG): container finished" podID="0baee2a2-f093-4adc-ab4b-8541f498abf3" containerID="62b015a9f97f8415af0b1dc95239d03051ff0e6e4c1fcdda37a2b4b0d510945f" exitCode=0 Nov 03 01:43:35.272440 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:43:35.272333 4939 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-88bbf7-openshift-default-574f47fff7-x96kz" Nov 03 01:43:35.272440 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:43:35.272378 4939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-88bbf7-openshift-default-574f47fff7-x96kz" event={"ID":"0baee2a2-f093-4adc-ab4b-8541f498abf3","Type":"ContainerDied","Data":"62b015a9f97f8415af0b1dc95239d03051ff0e6e4c1fcdda37a2b4b0d510945f"} Nov 03 01:43:35.272440 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:43:35.272429 4939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-88bbf7-openshift-default-574f47fff7-x96kz" event={"ID":"0baee2a2-f093-4adc-ab4b-8541f498abf3","Type":"ContainerDied","Data":"73951df182b7bf8f16ab31f7282bf473a958257c388ef6a3bff351ad04b34ff1"} Nov 03 01:43:35.280097 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:43:35.280077 4939 scope.go:122] "RemoveContainer" containerID="1360b1db3879eb318e46aad725095dba682ba897cb83beae80a8a8a488c63b87" Nov 03 01:43:35.280351 ip-10-0-140-107 kubenswrapper[4939]: E1103 01:43:35.280334 4939 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1360b1db3879eb318e46aad725095dba682ba897cb83beae80a8a8a488c63b87\": container with ID starting with 1360b1db3879eb318e46aad725095dba682ba897cb83beae80a8a8a488c63b87 not found: ID does not exist" containerID="1360b1db3879eb318e46aad725095dba682ba897cb83beae80a8a8a488c63b87" Nov 03 01:43:35.280416 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:43:35.280360 4939 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1360b1db3879eb318e46aad725095dba682ba897cb83beae80a8a8a488c63b87"} err="failed to get container status \"1360b1db3879eb318e46aad725095dba682ba897cb83beae80a8a8a488c63b87\": rpc error: code = NotFound desc = could not find container \"1360b1db3879eb318e46aad725095dba682ba897cb83beae80a8a8a488c63b87\": container with ID starting with 1360b1db3879eb318e46aad725095dba682ba897cb83beae80a8a8a488c63b87 not found: ID does not exist" Nov 03 01:43:35.280416 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:43:35.280378 4939 scope.go:122] "RemoveContainer" containerID="62b015a9f97f8415af0b1dc95239d03051ff0e6e4c1fcdda37a2b4b0d510945f" Nov 03 01:43:35.295113 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:43:35.295093 4939 scope.go:122] "RemoveContainer" containerID="62b015a9f97f8415af0b1dc95239d03051ff0e6e4c1fcdda37a2b4b0d510945f" Nov 03 01:43:35.295376 ip-10-0-140-107 kubenswrapper[4939]: E1103 01:43:35.295356 4939 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"62b015a9f97f8415af0b1dc95239d03051ff0e6e4c1fcdda37a2b4b0d510945f\": container with ID starting with 62b015a9f97f8415af0b1dc95239d03051ff0e6e4c1fcdda37a2b4b0d510945f not found: ID does not exist" containerID="62b015a9f97f8415af0b1dc95239d03051ff0e6e4c1fcdda37a2b4b0d510945f" Nov 03 01:43:35.295439 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:43:35.295383 4939 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"62b015a9f97f8415af0b1dc95239d03051ff0e6e4c1fcdda37a2b4b0d510945f"} err="failed to get container status \"62b015a9f97f8415af0b1dc95239d03051ff0e6e4c1fcdda37a2b4b0d510945f\": rpc error: code = NotFound desc = could not find container \"62b015a9f97f8415af0b1dc95239d03051ff0e6e4c1fcdda37a2b4b0d510945f\": container with ID starting with 62b015a9f97f8415af0b1dc95239d03051ff0e6e4c1fcdda37a2b4b0d510945f not found: ID does not exist" Nov 03 01:43:35.307648 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:43:35.307624 4939 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-88bbf7-openshift-default-574f47fff7-6qqxz"] Nov 03 01:43:35.310658 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:43:35.310634 4939 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-88bbf7-openshift-default-574f47fff7-6qqxz"] Nov 03 01:43:35.321620 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:43:35.321599 4939 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-88bbf7-openshift-default-574f47fff7-x96kz"] Nov 03 01:43:35.325150 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:43:35.325128 4939 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-88bbf7-openshift-default-574f47fff7-x96kz"] Nov 03 01:43:35.867119 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:43:35.867085 4939 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0baee2a2-f093-4adc-ab4b-8541f498abf3" path="/var/lib/kubelet/pods/0baee2a2-f093-4adc-ab4b-8541f498abf3/volumes" Nov 03 01:43:35.867566 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:43:35.867552 4939 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3ceb505f-63e4-427e-aad1-d40ca8062b2b" path="/var/lib/kubelet/pods/3ceb505f-63e4-427e-aad1-d40ca8062b2b/volumes" Nov 03 01:43:39.201656 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:43:39.201614 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:43:39.570674 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:43:39.570561 4939 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ait-a932d7e1-openshift-default-994d78cb5-qwmlw"] Nov 03 01:43:39.571372 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:43:39.571343 4939 cpu_manager.go:414] "containerMap: removing container" podUID="3ceb505f-63e4-427e-aad1-d40ca8062b2b" containerName="istio-proxy" Nov 03 01:43:39.571453 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:43:39.571378 4939 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3ceb505f-63e4-427e-aad1-d40ca8062b2b" containerName="istio-proxy" Nov 03 01:43:39.571453 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:43:39.571423 4939 cpu_manager.go:414] "containerMap: removing container" podUID="0baee2a2-f093-4adc-ab4b-8541f498abf3" containerName="istio-proxy" Nov 03 01:43:39.571453 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:43:39.571432 4939 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0baee2a2-f093-4adc-ab4b-8541f498abf3" containerName="istio-proxy" Nov 03 01:43:39.571584 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:43:39.571548 4939 memory_manager.go:365] "RemoveStaleState removing state" podUID="3ceb505f-63e4-427e-aad1-d40ca8062b2b" containerName="istio-proxy" Nov 03 01:43:39.571584 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:43:39.571569 4939 memory_manager.go:365] "RemoveStaleState removing state" podUID="0baee2a2-f093-4adc-ab4b-8541f498abf3" containerName="istio-proxy" Nov 03 01:43:39.576883 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:43:39.576859 4939 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-a932d7e1-openshift-default-994d78cb5-qwmlw" Nov 03 01:43:39.583019 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:43:39.582987 4939 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ait-a932d7e1-openshift-default-dockercfg-42pjb\"" Nov 03 01:43:39.584210 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:43:39.584184 4939 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-a932d7e1-openshift-default-994d78cb5-qwmlw"] Nov 03 01:43:39.643596 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:43:39.643565 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:43:39.738184 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:43:39.738144 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1624410a-c346-403b-bb74-9f18b5f5aada-workload-certs\") pod \"e2e-ait-a932d7e1-openshift-default-994d78cb5-qwmlw\" (UID: \"1624410a-c346-403b-bb74-9f18b5f5aada\") " pod="openshift-ingress/e2e-ait-a932d7e1-openshift-default-994d78cb5-qwmlw" Nov 03 01:43:39.738417 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:43:39.738200 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g2hp9\" (UniqueName: \"kubernetes.io/projected/1624410a-c346-403b-bb74-9f18b5f5aada-kube-api-access-g2hp9\") pod \"e2e-ait-a932d7e1-openshift-default-994d78cb5-qwmlw\" (UID: \"1624410a-c346-403b-bb74-9f18b5f5aada\") " pod="openshift-ingress/e2e-ait-a932d7e1-openshift-default-994d78cb5-qwmlw" Nov 03 01:43:39.738417 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:43:39.738308 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1624410a-c346-403b-bb74-9f18b5f5aada-istio-envoy\") pod \"e2e-ait-a932d7e1-openshift-default-994d78cb5-qwmlw\" (UID: \"1624410a-c346-403b-bb74-9f18b5f5aada\") " pod="openshift-ingress/e2e-ait-a932d7e1-openshift-default-994d78cb5-qwmlw" Nov 03 01:43:39.738417 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:43:39.738346 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1624410a-c346-403b-bb74-9f18b5f5aada-istio-data\") pod \"e2e-ait-a932d7e1-openshift-default-994d78cb5-qwmlw\" (UID: \"1624410a-c346-403b-bb74-9f18b5f5aada\") " pod="openshift-ingress/e2e-ait-a932d7e1-openshift-default-994d78cb5-qwmlw" Nov 03 01:43:39.738417 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:43:39.738388 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1624410a-c346-403b-bb74-9f18b5f5aada-istiod-ca-cert\") pod \"e2e-ait-a932d7e1-openshift-default-994d78cb5-qwmlw\" (UID: \"1624410a-c346-403b-bb74-9f18b5f5aada\") " pod="openshift-ingress/e2e-ait-a932d7e1-openshift-default-994d78cb5-qwmlw" Nov 03 01:43:39.738726 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:43:39.738429 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1624410a-c346-403b-bb74-9f18b5f5aada-istio-podinfo\") pod \"e2e-ait-a932d7e1-openshift-default-994d78cb5-qwmlw\" (UID: \"1624410a-c346-403b-bb74-9f18b5f5aada\") " pod="openshift-ingress/e2e-ait-a932d7e1-openshift-default-994d78cb5-qwmlw" Nov 03 01:43:39.738726 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:43:39.738508 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1624410a-c346-403b-bb74-9f18b5f5aada-workload-socket\") pod \"e2e-ait-a932d7e1-openshift-default-994d78cb5-qwmlw\" (UID: \"1624410a-c346-403b-bb74-9f18b5f5aada\") " pod="openshift-ingress/e2e-ait-a932d7e1-openshift-default-994d78cb5-qwmlw" Nov 03 01:43:39.738726 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:43:39.738686 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1624410a-c346-403b-bb74-9f18b5f5aada-credential-socket\") pod \"e2e-ait-a932d7e1-openshift-default-994d78cb5-qwmlw\" (UID: \"1624410a-c346-403b-bb74-9f18b5f5aada\") " pod="openshift-ingress/e2e-ait-a932d7e1-openshift-default-994d78cb5-qwmlw" Nov 03 01:43:39.738997 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:43:39.738813 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1624410a-c346-403b-bb74-9f18b5f5aada-istio-token\") pod \"e2e-ait-a932d7e1-openshift-default-994d78cb5-qwmlw\" (UID: \"1624410a-c346-403b-bb74-9f18b5f5aada\") " pod="openshift-ingress/e2e-ait-a932d7e1-openshift-default-994d78cb5-qwmlw" Nov 03 01:43:39.782222 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:43:39.782181 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:43:39.840214 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:43:39.840181 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1624410a-c346-403b-bb74-9f18b5f5aada-istio-token\") pod \"e2e-ait-a932d7e1-openshift-default-994d78cb5-qwmlw\" (UID: \"1624410a-c346-403b-bb74-9f18b5f5aada\") " pod="openshift-ingress/e2e-ait-a932d7e1-openshift-default-994d78cb5-qwmlw" Nov 03 01:43:39.840414 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:43:39.840250 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1624410a-c346-403b-bb74-9f18b5f5aada-workload-certs\") pod \"e2e-ait-a932d7e1-openshift-default-994d78cb5-qwmlw\" (UID: \"1624410a-c346-403b-bb74-9f18b5f5aada\") " pod="openshift-ingress/e2e-ait-a932d7e1-openshift-default-994d78cb5-qwmlw" Nov 03 01:43:39.840414 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:43:39.840284 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g2hp9\" (UniqueName: \"kubernetes.io/projected/1624410a-c346-403b-bb74-9f18b5f5aada-kube-api-access-g2hp9\") pod \"e2e-ait-a932d7e1-openshift-default-994d78cb5-qwmlw\" (UID: \"1624410a-c346-403b-bb74-9f18b5f5aada\") " pod="openshift-ingress/e2e-ait-a932d7e1-openshift-default-994d78cb5-qwmlw" Nov 03 01:43:39.840414 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:43:39.840302 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1624410a-c346-403b-bb74-9f18b5f5aada-istio-envoy\") pod \"e2e-ait-a932d7e1-openshift-default-994d78cb5-qwmlw\" (UID: \"1624410a-c346-403b-bb74-9f18b5f5aada\") " pod="openshift-ingress/e2e-ait-a932d7e1-openshift-default-994d78cb5-qwmlw" Nov 03 01:43:39.840414 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:43:39.840316 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1624410a-c346-403b-bb74-9f18b5f5aada-istio-data\") pod \"e2e-ait-a932d7e1-openshift-default-994d78cb5-qwmlw\" (UID: \"1624410a-c346-403b-bb74-9f18b5f5aada\") " pod="openshift-ingress/e2e-ait-a932d7e1-openshift-default-994d78cb5-qwmlw" Nov 03 01:43:39.840414 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:43:39.840355 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1624410a-c346-403b-bb74-9f18b5f5aada-istiod-ca-cert\") pod \"e2e-ait-a932d7e1-openshift-default-994d78cb5-qwmlw\" (UID: \"1624410a-c346-403b-bb74-9f18b5f5aada\") " pod="openshift-ingress/e2e-ait-a932d7e1-openshift-default-994d78cb5-qwmlw" Nov 03 01:43:39.840414 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:43:39.840396 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1624410a-c346-403b-bb74-9f18b5f5aada-istio-podinfo\") pod \"e2e-ait-a932d7e1-openshift-default-994d78cb5-qwmlw\" (UID: \"1624410a-c346-403b-bb74-9f18b5f5aada\") " pod="openshift-ingress/e2e-ait-a932d7e1-openshift-default-994d78cb5-qwmlw" Nov 03 01:43:39.840788 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:43:39.840467 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1624410a-c346-403b-bb74-9f18b5f5aada-workload-socket\") pod \"e2e-ait-a932d7e1-openshift-default-994d78cb5-qwmlw\" (UID: \"1624410a-c346-403b-bb74-9f18b5f5aada\") " pod="openshift-ingress/e2e-ait-a932d7e1-openshift-default-994d78cb5-qwmlw" Nov 03 01:43:39.840788 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:43:39.840520 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1624410a-c346-403b-bb74-9f18b5f5aada-credential-socket\") pod \"e2e-ait-a932d7e1-openshift-default-994d78cb5-qwmlw\" (UID: \"1624410a-c346-403b-bb74-9f18b5f5aada\") " pod="openshift-ingress/e2e-ait-a932d7e1-openshift-default-994d78cb5-qwmlw" Nov 03 01:43:39.840899 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:43:39.840784 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1624410a-c346-403b-bb74-9f18b5f5aada-workload-certs\") pod \"e2e-ait-a932d7e1-openshift-default-994d78cb5-qwmlw\" (UID: \"1624410a-c346-403b-bb74-9f18b5f5aada\") " pod="openshift-ingress/e2e-ait-a932d7e1-openshift-default-994d78cb5-qwmlw" Nov 03 01:43:39.840899 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:43:39.840867 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1624410a-c346-403b-bb74-9f18b5f5aada-credential-socket\") pod \"e2e-ait-a932d7e1-openshift-default-994d78cb5-qwmlw\" (UID: \"1624410a-c346-403b-bb74-9f18b5f5aada\") " pod="openshift-ingress/e2e-ait-a932d7e1-openshift-default-994d78cb5-qwmlw" Nov 03 01:43:39.841023 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:43:39.840977 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1624410a-c346-403b-bb74-9f18b5f5aada-workload-socket\") pod \"e2e-ait-a932d7e1-openshift-default-994d78cb5-qwmlw\" (UID: \"1624410a-c346-403b-bb74-9f18b5f5aada\") " pod="openshift-ingress/e2e-ait-a932d7e1-openshift-default-994d78cb5-qwmlw" Nov 03 01:43:39.841065 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:43:39.841024 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1624410a-c346-403b-bb74-9f18b5f5aada-istio-data\") pod \"e2e-ait-a932d7e1-openshift-default-994d78cb5-qwmlw\" (UID: \"1624410a-c346-403b-bb74-9f18b5f5aada\") " pod="openshift-ingress/e2e-ait-a932d7e1-openshift-default-994d78cb5-qwmlw" Nov 03 01:43:39.841196 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:43:39.841170 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1624410a-c346-403b-bb74-9f18b5f5aada-istiod-ca-cert\") pod \"e2e-ait-a932d7e1-openshift-default-994d78cb5-qwmlw\" (UID: \"1624410a-c346-403b-bb74-9f18b5f5aada\") " pod="openshift-ingress/e2e-ait-a932d7e1-openshift-default-994d78cb5-qwmlw" Nov 03 01:43:39.842800 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:43:39.842780 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1624410a-c346-403b-bb74-9f18b5f5aada-istio-envoy\") pod \"e2e-ait-a932d7e1-openshift-default-994d78cb5-qwmlw\" (UID: \"1624410a-c346-403b-bb74-9f18b5f5aada\") " pod="openshift-ingress/e2e-ait-a932d7e1-openshift-default-994d78cb5-qwmlw" Nov 03 01:43:39.843144 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:43:39.843127 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1624410a-c346-403b-bb74-9f18b5f5aada-istio-podinfo\") pod \"e2e-ait-a932d7e1-openshift-default-994d78cb5-qwmlw\" (UID: \"1624410a-c346-403b-bb74-9f18b5f5aada\") " pod="openshift-ingress/e2e-ait-a932d7e1-openshift-default-994d78cb5-qwmlw" Nov 03 01:43:39.848262 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:43:39.848234 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-g2hp9\" (UniqueName: \"kubernetes.io/projected/1624410a-c346-403b-bb74-9f18b5f5aada-kube-api-access-g2hp9\") pod \"e2e-ait-a932d7e1-openshift-default-994d78cb5-qwmlw\" (UID: \"1624410a-c346-403b-bb74-9f18b5f5aada\") " pod="openshift-ingress/e2e-ait-a932d7e1-openshift-default-994d78cb5-qwmlw" Nov 03 01:43:39.848349 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:43:39.848302 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1624410a-c346-403b-bb74-9f18b5f5aada-istio-token\") pod \"e2e-ait-a932d7e1-openshift-default-994d78cb5-qwmlw\" (UID: \"1624410a-c346-403b-bb74-9f18b5f5aada\") " pod="openshift-ingress/e2e-ait-a932d7e1-openshift-default-994d78cb5-qwmlw" Nov 03 01:43:39.894565 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:43:39.894531 4939 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-a932d7e1-openshift-default-994d78cb5-qwmlw" Nov 03 01:43:40.173588 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:43:40.173506 4939 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-a932d7e1-openshift-default-994d78cb5-qwmlw"] Nov 03 01:43:40.175779 ip-10-0-140-107 kubenswrapper[4939]: W1103 01:43:40.175753 4939 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1624410a_c346_403b_bb74_9f18b5f5aada.slice/crio-db34ee0a52b57789aa91da0c7c1f28670075e9276b0ffa6427402e6593270949 WatchSource:0}: Error finding container db34ee0a52b57789aa91da0c7c1f28670075e9276b0ffa6427402e6593270949: Status 404 returned error can't find the container with id db34ee0a52b57789aa91da0c7c1f28670075e9276b0ffa6427402e6593270949 Nov 03 01:43:40.177769 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:43:40.177736 4939 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 01:43:40.177848 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:43:40.177803 4939 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 01:43:40.177848 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:43:40.177843 4939 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 01:43:40.302460 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:43:40.302421 4939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-a932d7e1-openshift-default-994d78cb5-qwmlw" event={"ID":"1624410a-c346-403b-bb74-9f18b5f5aada","Type":"ContainerStarted","Data":"c1c38528a4010f07925cf47b3dc19fccca68aa360ef58796e56d187c8302060f"} Nov 03 01:43:40.303003 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:43:40.302484 4939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-a932d7e1-openshift-default-994d78cb5-qwmlw" event={"ID":"1624410a-c346-403b-bb74-9f18b5f5aada","Type":"ContainerStarted","Data":"db34ee0a52b57789aa91da0c7c1f28670075e9276b0ffa6427402e6593270949"} Nov 03 01:43:40.324736 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:43:40.324670 4939 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ait-a932d7e1-openshift-default-994d78cb5-qwmlw" podStartSLOduration=1.3246520849999999 podStartE2EDuration="1.324652085s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 01:43:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 01:43:40.323607947 +0000 UTC m=+3768.587942926" watchObservedRunningTime="2026-11-03 01:43:40.324652085 +0000 UTC m=+3768.588987032" Nov 03 01:43:40.894886 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:43:40.894852 4939 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-a932d7e1-openshift-default-994d78cb5-qwmlw" Nov 03 01:43:41.773282 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:43:41.773246 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:43:41.885645 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:43:41.885594 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:43:41.900333 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:43:41.900300 4939 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-a932d7e1-openshift-default-994d78cb5-qwmlw" Nov 03 01:43:42.312259 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:43:42.312227 4939 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-a932d7e1-openshift-default-994d78cb5-qwmlw" Nov 03 01:43:42.313404 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:43:42.313380 4939 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-a932d7e1-openshift-default-994d78cb5-qwmlw" Nov 03 01:43:42.953787 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:43:42.953751 4939 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-a932d7e1-6dbcfd9bd5-bhgq4"] Nov 03 01:43:42.958010 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:43:42.957987 4939 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-a932d7e1-6dbcfd9bd5-bhgq4" Nov 03 01:43:42.962510 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:43:42.962482 4939 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-a932d7e1-6dbcfd9bd5-bhgq4"] Nov 03 01:43:42.966147 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:43:42.966122 4939 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ait-a932d7e1-serving-cert\"" Nov 03 01:43:42.967756 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:43:42.967733 4939 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-nfcxl\"" Nov 03 01:43:42.968598 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:43:42.968011 4939 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 03 01:43:42.968598 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:43:42.968097 4939 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-metrics-tls\"" Nov 03 01:43:42.969345 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:43:42.969325 4939 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 03 01:43:42.972135 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:43:42.972106 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:43:42.972573 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:43:42.972556 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4ff16b4f-3938-4184-bc01-603126df8ee6-metrics-certs\") pod \"maas-api-e2e-ait-a932d7e1-6dbcfd9bd5-bhgq4\" (UID: \"4ff16b4f-3938-4184-bc01-603126df8ee6\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-a932d7e1-6dbcfd9bd5-bhgq4" Nov 03 01:43:42.972635 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:43:42.972585 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4ff16b4f-3938-4184-bc01-603126df8ee6-openshift-service-ca\") pod \"maas-api-e2e-ait-a932d7e1-6dbcfd9bd5-bhgq4\" (UID: \"4ff16b4f-3938-4184-bc01-603126df8ee6\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-a932d7e1-6dbcfd9bd5-bhgq4" Nov 03 01:43:42.972676 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:43:42.972639 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zlqqv\" (UniqueName: \"kubernetes.io/projected/4ff16b4f-3938-4184-bc01-603126df8ee6-kube-api-access-zlqqv\") pod \"maas-api-e2e-ait-a932d7e1-6dbcfd9bd5-bhgq4\" (UID: \"4ff16b4f-3938-4184-bc01-603126df8ee6\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-a932d7e1-6dbcfd9bd5-bhgq4" Nov 03 01:43:42.972719 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:43:42.972700 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4ff16b4f-3938-4184-bc01-603126df8ee6-maas-api-tls\") pod \"maas-api-e2e-ait-a932d7e1-6dbcfd9bd5-bhgq4\" (UID: \"4ff16b4f-3938-4184-bc01-603126df8ee6\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-a932d7e1-6dbcfd9bd5-bhgq4" Nov 03 01:43:42.981491 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:43:42.981464 4939 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-a932d7e1-5dff9b7b87-wf7p6"] Nov 03 01:43:42.986225 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:43:42.986196 4939 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-a932d7e1-5dff9b7b87-wf7p6" Nov 03 01:43:42.991591 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:43:42.991565 4939 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-a932d7e1-5c58fb7f-7xzkk"] Nov 03 01:43:42.996679 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:43:42.996656 4939 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ait-a932d7e1-dockercfg-l66s8\"" Nov 03 01:43:42.999201 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:43:42.999178 4939 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-a932d7e1-5dff9b7b87-wf7p6"] Nov 03 01:43:42.999314 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:43:42.999256 4939 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-a932d7e1-5c58fb7f-7xzkk" Nov 03 01:43:43.004608 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:43:43.004585 4939 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-a932d7e1-5c58fb7f-7xzkk"] Nov 03 01:43:43.066775 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:43:43.066746 4939 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ait-a932d7e1\"" Nov 03 01:43:43.073480 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:43:43.073450 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4ff16b4f-3938-4184-bc01-603126df8ee6-openshift-service-ca\") pod \"maas-api-e2e-ait-a932d7e1-6dbcfd9bd5-bhgq4\" (UID: \"4ff16b4f-3938-4184-bc01-603126df8ee6\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-a932d7e1-6dbcfd9bd5-bhgq4" Nov 03 01:43:43.073611 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:43:43.073501 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f2d38bf5-57c3-46f2-8c10-5b3468bbd3b9-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-a932d7e1-5c58fb7f-7xzkk\" (UID: \"f2d38bf5-57c3-46f2-8c10-5b3468bbd3b9\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-a932d7e1-5c58fb7f-7xzkk" Nov 03 01:43:43.073611 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:43:43.073571 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zlqqv\" (UniqueName: \"kubernetes.io/projected/4ff16b4f-3938-4184-bc01-603126df8ee6-kube-api-access-zlqqv\") pod \"maas-api-e2e-ait-a932d7e1-6dbcfd9bd5-bhgq4\" (UID: \"4ff16b4f-3938-4184-bc01-603126df8ee6\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-a932d7e1-6dbcfd9bd5-bhgq4" Nov 03 01:43:43.073762 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:43:43.073695 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m8hqd\" (UniqueName: \"kubernetes.io/projected/f2d38bf5-57c3-46f2-8c10-5b3468bbd3b9-kube-api-access-m8hqd\") pod \"payload-pre-processing-e2e-ait-a932d7e1-5c58fb7f-7xzkk\" (UID: \"f2d38bf5-57c3-46f2-8c10-5b3468bbd3b9\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-a932d7e1-5c58fb7f-7xzkk" Nov 03 01:43:43.073762 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:43:43.073738 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/267546ba-79f3-4c43-8c27-b446fcf18a10-plugins-config-volume\") pod \"payload-processing-e2e-ait-a932d7e1-5dff9b7b87-wf7p6\" (UID: \"267546ba-79f3-4c43-8c27-b446fcf18a10\") " pod="openshift-ingress/payload-processing-e2e-ait-a932d7e1-5dff9b7b87-wf7p6" Nov 03 01:43:43.073894 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:43:43.073873 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4ff16b4f-3938-4184-bc01-603126df8ee6-maas-api-tls\") pod \"maas-api-e2e-ait-a932d7e1-6dbcfd9bd5-bhgq4\" (UID: \"4ff16b4f-3938-4184-bc01-603126df8ee6\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-a932d7e1-6dbcfd9bd5-bhgq4" Nov 03 01:43:43.074005 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:43:43.073988 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4ff16b4f-3938-4184-bc01-603126df8ee6-metrics-certs\") pod \"maas-api-e2e-ait-a932d7e1-6dbcfd9bd5-bhgq4\" (UID: \"4ff16b4f-3938-4184-bc01-603126df8ee6\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-a932d7e1-6dbcfd9bd5-bhgq4" Nov 03 01:43:43.074056 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:43:43.074024 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7jdlg\" (UniqueName: \"kubernetes.io/projected/267546ba-79f3-4c43-8c27-b446fcf18a10-kube-api-access-7jdlg\") pod \"payload-processing-e2e-ait-a932d7e1-5dff9b7b87-wf7p6\" (UID: \"267546ba-79f3-4c43-8c27-b446fcf18a10\") " pod="openshift-ingress/payload-processing-e2e-ait-a932d7e1-5dff9b7b87-wf7p6" Nov 03 01:43:43.074257 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:43:43.074240 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4ff16b4f-3938-4184-bc01-603126df8ee6-openshift-service-ca\") pod \"maas-api-e2e-ait-a932d7e1-6dbcfd9bd5-bhgq4\" (UID: \"4ff16b4f-3938-4184-bc01-603126df8ee6\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-a932d7e1-6dbcfd9bd5-bhgq4" Nov 03 01:43:43.076541 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:43:43.076512 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4ff16b4f-3938-4184-bc01-603126df8ee6-maas-api-tls\") pod \"maas-api-e2e-ait-a932d7e1-6dbcfd9bd5-bhgq4\" (UID: \"4ff16b4f-3938-4184-bc01-603126df8ee6\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-a932d7e1-6dbcfd9bd5-bhgq4" Nov 03 01:43:43.076634 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:43:43.076512 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4ff16b4f-3938-4184-bc01-603126df8ee6-metrics-certs\") pod \"maas-api-e2e-ait-a932d7e1-6dbcfd9bd5-bhgq4\" (UID: \"4ff16b4f-3938-4184-bc01-603126df8ee6\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-a932d7e1-6dbcfd9bd5-bhgq4" Nov 03 01:43:43.086389 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:43:43.086365 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zlqqv\" (UniqueName: \"kubernetes.io/projected/4ff16b4f-3938-4184-bc01-603126df8ee6-kube-api-access-zlqqv\") pod \"maas-api-e2e-ait-a932d7e1-6dbcfd9bd5-bhgq4\" (UID: \"4ff16b4f-3938-4184-bc01-603126df8ee6\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-a932d7e1-6dbcfd9bd5-bhgq4" Nov 03 01:43:43.106573 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:43:43.106544 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:43:43.174585 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:43:43.174536 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m8hqd\" (UniqueName: \"kubernetes.io/projected/f2d38bf5-57c3-46f2-8c10-5b3468bbd3b9-kube-api-access-m8hqd\") pod \"payload-pre-processing-e2e-ait-a932d7e1-5c58fb7f-7xzkk\" (UID: \"f2d38bf5-57c3-46f2-8c10-5b3468bbd3b9\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-a932d7e1-5c58fb7f-7xzkk" Nov 03 01:43:43.174800 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:43:43.174593 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/267546ba-79f3-4c43-8c27-b446fcf18a10-plugins-config-volume\") pod \"payload-processing-e2e-ait-a932d7e1-5dff9b7b87-wf7p6\" (UID: \"267546ba-79f3-4c43-8c27-b446fcf18a10\") " pod="openshift-ingress/payload-processing-e2e-ait-a932d7e1-5dff9b7b87-wf7p6" Nov 03 01:43:43.174800 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:43:43.174683 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7jdlg\" (UniqueName: \"kubernetes.io/projected/267546ba-79f3-4c43-8c27-b446fcf18a10-kube-api-access-7jdlg\") pod \"payload-processing-e2e-ait-a932d7e1-5dff9b7b87-wf7p6\" (UID: \"267546ba-79f3-4c43-8c27-b446fcf18a10\") " pod="openshift-ingress/payload-processing-e2e-ait-a932d7e1-5dff9b7b87-wf7p6" Nov 03 01:43:43.174800 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:43:43.174730 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f2d38bf5-57c3-46f2-8c10-5b3468bbd3b9-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-a932d7e1-5c58fb7f-7xzkk\" (UID: \"f2d38bf5-57c3-46f2-8c10-5b3468bbd3b9\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-a932d7e1-5c58fb7f-7xzkk" Nov 03 01:43:43.175588 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:43:43.175563 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/267546ba-79f3-4c43-8c27-b446fcf18a10-plugins-config-volume\") pod \"payload-processing-e2e-ait-a932d7e1-5dff9b7b87-wf7p6\" (UID: \"267546ba-79f3-4c43-8c27-b446fcf18a10\") " pod="openshift-ingress/payload-processing-e2e-ait-a932d7e1-5dff9b7b87-wf7p6" Nov 03 01:43:43.175800 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:43:43.175780 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f2d38bf5-57c3-46f2-8c10-5b3468bbd3b9-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-a932d7e1-5c58fb7f-7xzkk\" (UID: \"f2d38bf5-57c3-46f2-8c10-5b3468bbd3b9\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-a932d7e1-5c58fb7f-7xzkk" Nov 03 01:43:43.189731 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:43:43.189698 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-m8hqd\" (UniqueName: \"kubernetes.io/projected/f2d38bf5-57c3-46f2-8c10-5b3468bbd3b9-kube-api-access-m8hqd\") pod \"payload-pre-processing-e2e-ait-a932d7e1-5c58fb7f-7xzkk\" (UID: \"f2d38bf5-57c3-46f2-8c10-5b3468bbd3b9\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-a932d7e1-5c58fb7f-7xzkk" Nov 03 01:43:43.195204 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:43:43.195175 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7jdlg\" (UniqueName: \"kubernetes.io/projected/267546ba-79f3-4c43-8c27-b446fcf18a10-kube-api-access-7jdlg\") pod \"payload-processing-e2e-ait-a932d7e1-5dff9b7b87-wf7p6\" (UID: \"267546ba-79f3-4c43-8c27-b446fcf18a10\") " pod="openshift-ingress/payload-processing-e2e-ait-a932d7e1-5dff9b7b87-wf7p6" Nov 03 01:43:43.210273 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:43:43.210169 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:43:43.277119 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:43:43.277077 4939 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-a932d7e1-6dbcfd9bd5-bhgq4" Nov 03 01:43:43.307417 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:43:43.307387 4939 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-a932d7e1-5dff9b7b87-wf7p6" Nov 03 01:43:43.318277 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:43:43.318246 4939 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-a932d7e1-5c58fb7f-7xzkk" Nov 03 01:43:43.553046 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:43:43.553021 4939 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-a932d7e1-6dbcfd9bd5-bhgq4"] Nov 03 01:43:43.555962 ip-10-0-140-107 kubenswrapper[4939]: W1103 01:43:43.555902 4939 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4ff16b4f_3938_4184_bc01_603126df8ee6.slice/crio-6654777cac17071df5b4dbfae08c9f6640c3941d8b16fea40a9518d10df629a6 WatchSource:0}: Error finding container 6654777cac17071df5b4dbfae08c9f6640c3941d8b16fea40a9518d10df629a6: Status 404 returned error can't find the container with id 6654777cac17071df5b4dbfae08c9f6640c3941d8b16fea40a9518d10df629a6 Nov 03 01:43:43.557470 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:43:43.557448 4939 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Nov 03 01:43:43.609002 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:43:43.598000 4939 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-a932d7e1-5dff9b7b87-wf7p6"] Nov 03 01:43:43.631557 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:43:43.631525 4939 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-a932d7e1-5c58fb7f-7xzkk"] Nov 03 01:43:43.631834 ip-10-0-140-107 kubenswrapper[4939]: W1103 01:43:43.631805 4939 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf2d38bf5_57c3_46f2_8c10_5b3468bbd3b9.slice/crio-09cfba02a1f3d5850a4809cb066685ad3b0c7fc936c9b9791061757e78b762d6 WatchSource:0}: Error finding container 09cfba02a1f3d5850a4809cb066685ad3b0c7fc936c9b9791061757e78b762d6: Status 404 returned error can't find the container with id 09cfba02a1f3d5850a4809cb066685ad3b0c7fc936c9b9791061757e78b762d6 Nov 03 01:43:43.726965 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:43:43.726928 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:43:44.324449 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:43:44.324410 4939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-a932d7e1-6dbcfd9bd5-bhgq4" event={"ID":"4ff16b4f-3938-4184-bc01-603126df8ee6","Type":"ContainerStarted","Data":"6654777cac17071df5b4dbfae08c9f6640c3941d8b16fea40a9518d10df629a6"} Nov 03 01:43:44.326092 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:43:44.326062 4939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-a932d7e1-5c58fb7f-7xzkk" event={"ID":"f2d38bf5-57c3-46f2-8c10-5b3468bbd3b9","Type":"ContainerStarted","Data":"02bd6b09b351ebea5b63c22d498a1c7a2a2ce2df7b40a2a0cf8ffec9ff7e8545"} Nov 03 01:43:44.326227 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:43:44.326111 4939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-a932d7e1-5c58fb7f-7xzkk" event={"ID":"f2d38bf5-57c3-46f2-8c10-5b3468bbd3b9","Type":"ContainerStarted","Data":"09cfba02a1f3d5850a4809cb066685ad3b0c7fc936c9b9791061757e78b762d6"} Nov 03 01:43:44.326227 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:43:44.326138 4939 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-a932d7e1-5c58fb7f-7xzkk" Nov 03 01:43:44.327823 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:43:44.327799 4939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-a932d7e1-5dff9b7b87-wf7p6" event={"ID":"267546ba-79f3-4c43-8c27-b446fcf18a10","Type":"ContainerStarted","Data":"29faba7e4d3558f237bef031c15d92a5fe0b603777ec3ef4e4a4bee7191b92db"} Nov 03 01:43:44.327978 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:43:44.327839 4939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-a932d7e1-5dff9b7b87-wf7p6" event={"ID":"267546ba-79f3-4c43-8c27-b446fcf18a10","Type":"ContainerStarted","Data":"fe1feb23b1ff285bedbe57b6bd82d6109d9d20f1c02f27cd7a51abe445c1b094"} Nov 03 01:43:44.327978 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:43:44.327880 4939 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ait-a932d7e1-5dff9b7b87-wf7p6" Nov 03 01:43:44.345002 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:43:44.344964 4939 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ait-a932d7e1-5c58fb7f-7xzkk" podStartSLOduration=2.344950618 podStartE2EDuration="2.344950618s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 01:43:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 01:43:44.343084283 +0000 UTC m=+3772.607419229" watchObservedRunningTime="2026-11-03 01:43:44.344950618 +0000 UTC m=+3772.609285554" Nov 03 01:43:44.358488 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:43:44.358442 4939 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ait-a932d7e1-5dff9b7b87-wf7p6" podStartSLOduration=2.358427379 podStartE2EDuration="2.358427379s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 01:43:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 01:43:44.357989303 +0000 UTC m=+3772.622324251" watchObservedRunningTime="2026-11-03 01:43:44.358427379 +0000 UTC m=+3772.622762321" Nov 03 01:43:44.385076 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:43:44.385052 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:43:45.102450 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:43:45.102419 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:43:45.183298 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:43:45.183264 4939 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-4pw7v"] Nov 03 01:43:45.336655 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:43:45.336058 4939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-a932d7e1-6dbcfd9bd5-bhgq4" event={"ID":"4ff16b4f-3938-4184-bc01-603126df8ee6","Type":"ContainerStarted","Data":"cbaf6ee391f5c7a649e6ced0983a9efe2e976dac26f0618f78c32a0caaed9b76"} Nov 03 01:43:45.336655 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:43:45.336597 4939 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-a932d7e1-6dbcfd9bd5-bhgq4" Nov 03 01:43:45.353250 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:43:45.353163 4939 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ait-a932d7e1-6dbcfd9bd5-bhgq4" podStartSLOduration=2.585205266 podStartE2EDuration="3.353150442s" totalImagesPullingTime="767.945176ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 01:43:42 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 01:43:45.352002248 +0000 UTC m=+3773.616337194" watchObservedRunningTime="2026-11-03 01:43:45.353150442 +0000 UTC m=+3773.617485388" Nov 03 01:43:45.892866 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:43:45.892835 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:43:46.076066 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:43:46.076030 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:43:46.585709 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:43:46.585673 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:43:47.318573 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:43:47.318519 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:43:48.030527 ip-10-0-140-107 kubenswrapper[4939]: E1103 01:43:48.030488 4939 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-sthjh" containerName="extractor" Nov 03 01:43:48.109533 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:43:48.109499 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:43:48.808626 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:43:48.808594 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:43:49.508516 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:43:49.508473 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:43:50.195762 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:43:50.195725 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:43:50.900495 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:43:50.900454 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:43:51.598416 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:43:51.598382 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:43:52.193755 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:43:52.193722 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:43:52.355586 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:43:52.355547 4939 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-a932d7e1-6dbcfd9bd5-bhgq4" Nov 03 01:43:52.888704 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:43:52.888670 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:43:53.639543 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:43:53.639497 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:43:54.391433 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:43:54.391397 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:43:54.998346 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:43:54.998315 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:43:55.338759 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:43:55.338730 4939 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-a932d7e1-5c58fb7f-7xzkk" Nov 03 01:43:55.339051 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:43:55.339030 4939 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ait-a932d7e1-5dff9b7b87-wf7p6" Nov 03 01:43:55.426720 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:43:55.426689 4939 generic.go:356] "Generic (PLEG): container finished" podID="1624410a-c346-403b-bb74-9f18b5f5aada" containerID="c1c38528a4010f07925cf47b3dc19fccca68aa360ef58796e56d187c8302060f" exitCode=0 Nov 03 01:43:55.426948 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:43:55.426765 4939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-a932d7e1-openshift-default-994d78cb5-qwmlw" event={"ID":"1624410a-c346-403b-bb74-9f18b5f5aada","Type":"ContainerDied","Data":"c1c38528a4010f07925cf47b3dc19fccca68aa360ef58796e56d187c8302060f"} Nov 03 01:43:55.427280 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:43:55.427251 4939 scope.go:122] "RemoveContainer" containerID="c1c38528a4010f07925cf47b3dc19fccca68aa360ef58796e56d187c8302060f" Nov 03 01:43:55.428590 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:43:55.428557 4939 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 01:43:55.428714 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:43:55.428645 4939 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 01:43:55.428821 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:43:55.428714 4939 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 01:43:55.775196 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:43:55.775109 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:43:56.405506 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:43:56.405471 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:43:56.437353 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:43:56.437270 4939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-a932d7e1-openshift-default-994d78cb5-qwmlw" event={"ID":"1624410a-c346-403b-bb74-9f18b5f5aada","Type":"ContainerStarted","Data":"20fdb9add42b4d555b8d85a55eabf4704a7c0cbb4b07b25e9211b69b414e8b5e"} Nov 03 01:43:56.895366 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:43:56.895331 4939 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-a932d7e1-openshift-default-994d78cb5-qwmlw" Nov 03 01:43:56.896609 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:43:56.896570 4939 patch_prober.go:29] interesting pod/e2e-ait-a932d7e1-openshift-default-994d78cb5-qwmlw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.94:15021/healthz/ready\": dial tcp 10.132.1.94:15021: connect: connection refused" start-of-body= Nov 03 01:43:56.896762 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:43:56.896630 4939 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-a932d7e1-openshift-default-994d78cb5-qwmlw" podUID="1624410a-c346-403b-bb74-9f18b5f5aada" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.94:15021/healthz/ready\": dial tcp 10.132.1.94:15021: connect: connection refused" Nov 03 01:43:57.114121 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:43:57.114086 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:43:57.312421 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:43:57.312336 4939 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-a932d7e1-openshift-default-994d78cb5-qwmlw" Nov 03 01:43:57.803778 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:43:57.803744 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:43:57.895835 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:43:57.895784 4939 patch_prober.go:29] interesting pod/e2e-ait-a932d7e1-openshift-default-994d78cb5-qwmlw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.94:15021/healthz/ready\": dial tcp 10.132.1.94:15021: connect: connection refused" start-of-body= Nov 03 01:43:57.896077 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:43:57.895853 4939 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-a932d7e1-openshift-default-994d78cb5-qwmlw" podUID="1624410a-c346-403b-bb74-9f18b5f5aada" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.94:15021/healthz/ready\": dial tcp 10.132.1.94:15021: connect: connection refused" Nov 03 01:43:58.613738 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:43:58.613704 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:43:58.895399 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:43:58.895312 4939 patch_prober.go:29] interesting pod/e2e-ait-a932d7e1-openshift-default-994d78cb5-qwmlw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.94:15021/healthz/ready\": dial tcp 10.132.1.94:15021: connect: connection refused" start-of-body= Nov 03 01:43:58.895399 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:43:58.895381 4939 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-a932d7e1-openshift-default-994d78cb5-qwmlw" podUID="1624410a-c346-403b-bb74-9f18b5f5aada" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.94:15021/healthz/ready\": dial tcp 10.132.1.94:15021: connect: connection refused" Nov 03 01:43:59.317759 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:43:59.317650 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:43:59.895584 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:43:59.895546 4939 patch_prober.go:29] interesting pod/e2e-ait-a932d7e1-openshift-default-994d78cb5-qwmlw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.94:15021/healthz/ready\": dial tcp 10.132.1.94:15021: connect: connection refused" start-of-body= Nov 03 01:43:59.896034 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:43:59.895600 4939 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-a932d7e1-openshift-default-994d78cb5-qwmlw" podUID="1624410a-c346-403b-bb74-9f18b5f5aada" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.94:15021/healthz/ready\": dial tcp 10.132.1.94:15021: connect: connection refused" Nov 03 01:44:00.016565 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:44:00.016521 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:44:00.702155 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:44:00.702115 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:44:00.895699 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:44:00.895659 4939 patch_prober.go:29] interesting pod/e2e-ait-a932d7e1-openshift-default-994d78cb5-qwmlw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.94:15021/healthz/ready\": dial tcp 10.132.1.94:15021: connect: connection refused" start-of-body= Nov 03 01:44:00.896150 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:44:00.895711 4939 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-a932d7e1-openshift-default-994d78cb5-qwmlw" podUID="1624410a-c346-403b-bb74-9f18b5f5aada" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.94:15021/healthz/ready\": dial tcp 10.132.1.94:15021: connect: connection refused" Nov 03 01:44:01.386386 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:44:01.386332 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:44:01.899112 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:44:01.899079 4939 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-a932d7e1-openshift-default-994d78cb5-qwmlw" Nov 03 01:44:01.899922 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:44:01.899887 4939 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-a932d7e1-openshift-default-994d78cb5-qwmlw" Nov 03 01:44:01.987953 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:44:01.987896 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:44:02.735063 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:44:02.735026 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:44:03.288514 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:44:03.288470 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:44:03.901089 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:44:03.901058 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:44:04.498309 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:44:04.498277 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:44:05.127649 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:44:05.127616 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:44:05.796245 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:44:05.796204 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:44:06.401609 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:44:06.401572 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:44:07.001227 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:44:07.001182 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:44:07.602315 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:44:07.602266 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:44:07.869024 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:44:07.868947 4939 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-a932d7e1-5dff9b7b87-wf7p6"] Nov 03 01:44:07.880372 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:44:07.880341 4939 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-a932d7e1-5c58fb7f-7xzkk"] Nov 03 01:44:07.963243 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:44:07.963206 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:44:08.076435 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:44:08.076399 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:44:08.178169 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:44:08.178062 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:44:08.288147 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:44:08.288098 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:44:08.805288 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:44:08.805256 4939 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-a932d7e1-6dbcfd9bd5-bhgq4"] Nov 03 01:44:08.805553 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:44:08.805526 4939 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ait-a932d7e1-6dbcfd9bd5-bhgq4" podUID="4ff16b4f-3938-4184-bc01-603126df8ee6" containerName="maas-api" containerID="cri-o://cbaf6ee391f5c7a649e6ced0983a9efe2e976dac26f0618f78c32a0caaed9b76" gracePeriod=30 Nov 03 01:44:08.878159 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:44:08.878130 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:44:09.007470 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:44:09.007370 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:44:09.088230 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:44:09.087958 4939 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-a932d7e1-6dbcfd9bd5-bhgq4" Nov 03 01:44:09.102698 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:44:09.102661 4939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4ff16b4f-3938-4184-bc01-603126df8ee6-metrics-certs\" (UniqueName: \"kubernetes.io/secret/4ff16b4f-3938-4184-bc01-603126df8ee6-metrics-certs\") pod \"4ff16b4f-3938-4184-bc01-603126df8ee6\" (UID: \"4ff16b4f-3938-4184-bc01-603126df8ee6\") " Nov 03 01:44:09.102900 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:44:09.102709 4939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4ff16b4f-3938-4184-bc01-603126df8ee6-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4ff16b4f-3938-4184-bc01-603126df8ee6-maas-api-tls\") pod \"4ff16b4f-3938-4184-bc01-603126df8ee6\" (UID: \"4ff16b4f-3938-4184-bc01-603126df8ee6\") " Nov 03 01:44:09.102900 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:44:09.102786 4939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4ff16b4f-3938-4184-bc01-603126df8ee6-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4ff16b4f-3938-4184-bc01-603126df8ee6-openshift-service-ca\") pod \"4ff16b4f-3938-4184-bc01-603126df8ee6\" (UID: \"4ff16b4f-3938-4184-bc01-603126df8ee6\") " Nov 03 01:44:09.102900 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:44:09.102823 4939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4ff16b4f-3938-4184-bc01-603126df8ee6-kube-api-access-zlqqv\" (UniqueName: \"kubernetes.io/projected/4ff16b4f-3938-4184-bc01-603126df8ee6-kube-api-access-zlqqv\") pod \"4ff16b4f-3938-4184-bc01-603126df8ee6\" (UID: \"4ff16b4f-3938-4184-bc01-603126df8ee6\") " Nov 03 01:44:09.103312 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:44:09.103277 4939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4ff16b4f-3938-4184-bc01-603126df8ee6-openshift-service-ca" pod "4ff16b4f-3938-4184-bc01-603126df8ee6" (UID: "4ff16b4f-3938-4184-bc01-603126df8ee6"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 01:44:09.105084 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:44:09.105046 4939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4ff16b4f-3938-4184-bc01-603126df8ee6-metrics-certs" pod "4ff16b4f-3938-4184-bc01-603126df8ee6" (UID: "4ff16b4f-3938-4184-bc01-603126df8ee6"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 01:44:09.105456 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:44:09.105429 4939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4ff16b4f-3938-4184-bc01-603126df8ee6-kube-api-access-zlqqv" pod "4ff16b4f-3938-4184-bc01-603126df8ee6" (UID: "4ff16b4f-3938-4184-bc01-603126df8ee6"). InnerVolumeSpecName "kube-api-access-zlqqv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 01:44:09.105548 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:44:09.105448 4939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4ff16b4f-3938-4184-bc01-603126df8ee6-maas-api-tls" pod "4ff16b4f-3938-4184-bc01-603126df8ee6" (UID: "4ff16b4f-3938-4184-bc01-603126df8ee6"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 01:44:09.150385 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:44:09.150360 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:44:09.204112 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:44:09.204084 4939 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4ff16b4f-3938-4184-bc01-603126df8ee6-openshift-service-ca\") on node \"ip-10-0-140-107.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:44:09.204112 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:44:09.204110 4939 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zlqqv\" (UniqueName: \"kubernetes.io/projected/4ff16b4f-3938-4184-bc01-603126df8ee6-kube-api-access-zlqqv\") on node \"ip-10-0-140-107.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:44:09.204283 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:44:09.204125 4939 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4ff16b4f-3938-4184-bc01-603126df8ee6-metrics-certs\") on node \"ip-10-0-140-107.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:44:09.204283 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:44:09.204138 4939 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4ff16b4f-3938-4184-bc01-603126df8ee6-maas-api-tls\") on node \"ip-10-0-140-107.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:44:09.524078 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:44:09.524039 4939 generic.go:356] "Generic (PLEG): container finished" podID="4ff16b4f-3938-4184-bc01-603126df8ee6" containerID="cbaf6ee391f5c7a649e6ced0983a9efe2e976dac26f0618f78c32a0caaed9b76" exitCode=0 Nov 03 01:44:09.524258 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:44:09.524103 4939 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-a932d7e1-6dbcfd9bd5-bhgq4" Nov 03 01:44:09.524258 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:44:09.524118 4939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-a932d7e1-6dbcfd9bd5-bhgq4" event={"ID":"4ff16b4f-3938-4184-bc01-603126df8ee6","Type":"ContainerDied","Data":"cbaf6ee391f5c7a649e6ced0983a9efe2e976dac26f0618f78c32a0caaed9b76"} Nov 03 01:44:09.524258 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:44:09.524175 4939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-a932d7e1-6dbcfd9bd5-bhgq4" event={"ID":"4ff16b4f-3938-4184-bc01-603126df8ee6","Type":"ContainerDied","Data":"6654777cac17071df5b4dbfae08c9f6640c3941d8b16fea40a9518d10df629a6"} Nov 03 01:44:09.524258 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:44:09.524202 4939 scope.go:122] "RemoveContainer" containerID="cbaf6ee391f5c7a649e6ced0983a9efe2e976dac26f0618f78c32a0caaed9b76" Nov 03 01:44:09.534547 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:44:09.534528 4939 scope.go:122] "RemoveContainer" containerID="cbaf6ee391f5c7a649e6ced0983a9efe2e976dac26f0618f78c32a0caaed9b76" Nov 03 01:44:09.534831 ip-10-0-140-107 kubenswrapper[4939]: E1103 01:44:09.534810 4939 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cbaf6ee391f5c7a649e6ced0983a9efe2e976dac26f0618f78c32a0caaed9b76\": container with ID starting with cbaf6ee391f5c7a649e6ced0983a9efe2e976dac26f0618f78c32a0caaed9b76 not found: ID does not exist" containerID="cbaf6ee391f5c7a649e6ced0983a9efe2e976dac26f0618f78c32a0caaed9b76" Nov 03 01:44:09.534878 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:44:09.534843 4939 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cbaf6ee391f5c7a649e6ced0983a9efe2e976dac26f0618f78c32a0caaed9b76"} err="failed to get container status \"cbaf6ee391f5c7a649e6ced0983a9efe2e976dac26f0618f78c32a0caaed9b76\": rpc error: code = NotFound desc = could not find container \"cbaf6ee391f5c7a649e6ced0983a9efe2e976dac26f0618f78c32a0caaed9b76\": container with ID starting with cbaf6ee391f5c7a649e6ced0983a9efe2e976dac26f0618f78c32a0caaed9b76 not found: ID does not exist" Nov 03 01:44:09.546876 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:44:09.546825 4939 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-a932d7e1-6dbcfd9bd5-bhgq4"] Nov 03 01:44:09.548148 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:44:09.548125 4939 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-a932d7e1-6dbcfd9bd5-bhgq4"] Nov 03 01:44:09.867816 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:44:09.867777 4939 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4ff16b4f-3938-4184-bc01-603126df8ee6" path="/var/lib/kubelet/pods/4ff16b4f-3938-4184-bc01-603126df8ee6/volumes" Nov 03 01:44:10.295627 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:44:10.295548 4939 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-4pw7v"] Nov 03 01:44:12.869937 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:44:12.869881 4939 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ait-a932d7e1-5dff9b7b87-wf7p6" podUID="267546ba-79f3-4c43-8c27-b446fcf18a10" containerName="payload-processing" containerID="cri-o://29faba7e4d3558f237bef031c15d92a5fe0b603777ec3ef4e4a4bee7191b92db" gracePeriod=25 Nov 03 01:44:12.880860 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:44:12.880810 4939 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ait-a932d7e1-5c58fb7f-7xzkk" podUID="f2d38bf5-57c3-46f2-8c10-5b3468bbd3b9" containerName="payload-pre-processing" containerID="cri-o://02bd6b09b351ebea5b63c22d498a1c7a2a2ce2df7b40a2a0cf8ffec9ff7e8545" gracePeriod=25 Nov 03 01:44:13.125245 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:44:13.125187 4939 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-a932d7e1-5dff9b7b87-wf7p6" Nov 03 01:44:13.141101 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:44:13.141076 4939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/267546ba-79f3-4c43-8c27-b446fcf18a10-kube-api-access-7jdlg\" (UniqueName: \"kubernetes.io/projected/267546ba-79f3-4c43-8c27-b446fcf18a10-kube-api-access-7jdlg\") pod \"267546ba-79f3-4c43-8c27-b446fcf18a10\" (UID: \"267546ba-79f3-4c43-8c27-b446fcf18a10\") " Nov 03 01:44:13.141299 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:44:13.141222 4939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/267546ba-79f3-4c43-8c27-b446fcf18a10-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/267546ba-79f3-4c43-8c27-b446fcf18a10-plugins-config-volume\") pod \"267546ba-79f3-4c43-8c27-b446fcf18a10\" (UID: \"267546ba-79f3-4c43-8c27-b446fcf18a10\") " Nov 03 01:44:13.141698 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:44:13.141670 4939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/267546ba-79f3-4c43-8c27-b446fcf18a10-plugins-config-volume" pod "267546ba-79f3-4c43-8c27-b446fcf18a10" (UID: "267546ba-79f3-4c43-8c27-b446fcf18a10"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 01:44:13.142016 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:44:13.141989 4939 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/267546ba-79f3-4c43-8c27-b446fcf18a10-plugins-config-volume\") on node \"ip-10-0-140-107.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:44:13.144619 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:44:13.144592 4939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/267546ba-79f3-4c43-8c27-b446fcf18a10-kube-api-access-7jdlg" pod "267546ba-79f3-4c43-8c27-b446fcf18a10" (UID: "267546ba-79f3-4c43-8c27-b446fcf18a10"). InnerVolumeSpecName "kube-api-access-7jdlg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 01:44:13.156002 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:44:13.155980 4939 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-a932d7e1-5c58fb7f-7xzkk" Nov 03 01:44:13.242688 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:44:13.242652 4939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f2d38bf5-57c3-46f2-8c10-5b3468bbd3b9-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f2d38bf5-57c3-46f2-8c10-5b3468bbd3b9-plugins-config-volume\") pod \"f2d38bf5-57c3-46f2-8c10-5b3468bbd3b9\" (UID: \"f2d38bf5-57c3-46f2-8c10-5b3468bbd3b9\") " Nov 03 01:44:13.242866 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:44:13.242745 4939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f2d38bf5-57c3-46f2-8c10-5b3468bbd3b9-kube-api-access-m8hqd\" (UniqueName: \"kubernetes.io/projected/f2d38bf5-57c3-46f2-8c10-5b3468bbd3b9-kube-api-access-m8hqd\") pod \"f2d38bf5-57c3-46f2-8c10-5b3468bbd3b9\" (UID: \"f2d38bf5-57c3-46f2-8c10-5b3468bbd3b9\") " Nov 03 01:44:13.242964 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:44:13.242950 4939 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7jdlg\" (UniqueName: \"kubernetes.io/projected/267546ba-79f3-4c43-8c27-b446fcf18a10-kube-api-access-7jdlg\") on node \"ip-10-0-140-107.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:44:13.243159 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:44:13.243124 4939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f2d38bf5-57c3-46f2-8c10-5b3468bbd3b9-plugins-config-volume" pod "f2d38bf5-57c3-46f2-8c10-5b3468bbd3b9" (UID: "f2d38bf5-57c3-46f2-8c10-5b3468bbd3b9"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 01:44:13.244861 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:44:13.244841 4939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f2d38bf5-57c3-46f2-8c10-5b3468bbd3b9-kube-api-access-m8hqd" pod "f2d38bf5-57c3-46f2-8c10-5b3468bbd3b9" (UID: "f2d38bf5-57c3-46f2-8c10-5b3468bbd3b9"). InnerVolumeSpecName "kube-api-access-m8hqd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 01:44:13.343759 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:44:13.343717 4939 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m8hqd\" (UniqueName: \"kubernetes.io/projected/f2d38bf5-57c3-46f2-8c10-5b3468bbd3b9-kube-api-access-m8hqd\") on node \"ip-10-0-140-107.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:44:13.343759 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:44:13.343754 4939 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f2d38bf5-57c3-46f2-8c10-5b3468bbd3b9-plugins-config-volume\") on node \"ip-10-0-140-107.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:44:13.545329 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:44:13.545234 4939 generic.go:356] "Generic (PLEG): container finished" podID="f2d38bf5-57c3-46f2-8c10-5b3468bbd3b9" containerID="02bd6b09b351ebea5b63c22d498a1c7a2a2ce2df7b40a2a0cf8ffec9ff7e8545" exitCode=0 Nov 03 01:44:13.545329 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:44:13.545291 4939 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-a932d7e1-5c58fb7f-7xzkk" Nov 03 01:44:13.545546 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:44:13.545319 4939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-a932d7e1-5c58fb7f-7xzkk" event={"ID":"f2d38bf5-57c3-46f2-8c10-5b3468bbd3b9","Type":"ContainerDied","Data":"02bd6b09b351ebea5b63c22d498a1c7a2a2ce2df7b40a2a0cf8ffec9ff7e8545"} Nov 03 01:44:13.545546 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:44:13.545398 4939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-a932d7e1-5c58fb7f-7xzkk" event={"ID":"f2d38bf5-57c3-46f2-8c10-5b3468bbd3b9","Type":"ContainerDied","Data":"09cfba02a1f3d5850a4809cb066685ad3b0c7fc936c9b9791061757e78b762d6"} Nov 03 01:44:13.545546 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:44:13.545420 4939 scope.go:122] "RemoveContainer" containerID="02bd6b09b351ebea5b63c22d498a1c7a2a2ce2df7b40a2a0cf8ffec9ff7e8545" Nov 03 01:44:13.546766 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:44:13.546746 4939 generic.go:356] "Generic (PLEG): container finished" podID="267546ba-79f3-4c43-8c27-b446fcf18a10" containerID="29faba7e4d3558f237bef031c15d92a5fe0b603777ec3ef4e4a4bee7191b92db" exitCode=0 Nov 03 01:44:13.546849 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:44:13.546779 4939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-a932d7e1-5dff9b7b87-wf7p6" event={"ID":"267546ba-79f3-4c43-8c27-b446fcf18a10","Type":"ContainerDied","Data":"29faba7e4d3558f237bef031c15d92a5fe0b603777ec3ef4e4a4bee7191b92db"} Nov 03 01:44:13.546849 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:44:13.546816 4939 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-a932d7e1-5dff9b7b87-wf7p6" Nov 03 01:44:13.546969 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:44:13.546823 4939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-a932d7e1-5dff9b7b87-wf7p6" event={"ID":"267546ba-79f3-4c43-8c27-b446fcf18a10","Type":"ContainerDied","Data":"fe1feb23b1ff285bedbe57b6bd82d6109d9d20f1c02f27cd7a51abe445c1b094"} Nov 03 01:44:13.554224 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:44:13.554204 4939 scope.go:122] "RemoveContainer" containerID="02bd6b09b351ebea5b63c22d498a1c7a2a2ce2df7b40a2a0cf8ffec9ff7e8545" Nov 03 01:44:13.554484 ip-10-0-140-107 kubenswrapper[4939]: E1103 01:44:13.554465 4939 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"02bd6b09b351ebea5b63c22d498a1c7a2a2ce2df7b40a2a0cf8ffec9ff7e8545\": container with ID starting with 02bd6b09b351ebea5b63c22d498a1c7a2a2ce2df7b40a2a0cf8ffec9ff7e8545 not found: ID does not exist" containerID="02bd6b09b351ebea5b63c22d498a1c7a2a2ce2df7b40a2a0cf8ffec9ff7e8545" Nov 03 01:44:13.554543 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:44:13.554493 4939 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"02bd6b09b351ebea5b63c22d498a1c7a2a2ce2df7b40a2a0cf8ffec9ff7e8545"} err="failed to get container status \"02bd6b09b351ebea5b63c22d498a1c7a2a2ce2df7b40a2a0cf8ffec9ff7e8545\": rpc error: code = NotFound desc = could not find container \"02bd6b09b351ebea5b63c22d498a1c7a2a2ce2df7b40a2a0cf8ffec9ff7e8545\": container with ID starting with 02bd6b09b351ebea5b63c22d498a1c7a2a2ce2df7b40a2a0cf8ffec9ff7e8545 not found: ID does not exist" Nov 03 01:44:13.554543 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:44:13.554509 4939 scope.go:122] "RemoveContainer" containerID="29faba7e4d3558f237bef031c15d92a5fe0b603777ec3ef4e4a4bee7191b92db" Nov 03 01:44:13.563310 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:44:13.563283 4939 scope.go:122] "RemoveContainer" containerID="29faba7e4d3558f237bef031c15d92a5fe0b603777ec3ef4e4a4bee7191b92db" Nov 03 01:44:13.563532 ip-10-0-140-107 kubenswrapper[4939]: E1103 01:44:13.563513 4939 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"29faba7e4d3558f237bef031c15d92a5fe0b603777ec3ef4e4a4bee7191b92db\": container with ID starting with 29faba7e4d3558f237bef031c15d92a5fe0b603777ec3ef4e4a4bee7191b92db not found: ID does not exist" containerID="29faba7e4d3558f237bef031c15d92a5fe0b603777ec3ef4e4a4bee7191b92db" Nov 03 01:44:13.563591 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:44:13.563541 4939 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"29faba7e4d3558f237bef031c15d92a5fe0b603777ec3ef4e4a4bee7191b92db"} err="failed to get container status \"29faba7e4d3558f237bef031c15d92a5fe0b603777ec3ef4e4a4bee7191b92db\": rpc error: code = NotFound desc = could not find container \"29faba7e4d3558f237bef031c15d92a5fe0b603777ec3ef4e4a4bee7191b92db\": container with ID starting with 29faba7e4d3558f237bef031c15d92a5fe0b603777ec3ef4e4a4bee7191b92db not found: ID does not exist" Nov 03 01:44:13.571496 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:44:13.571474 4939 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-a932d7e1-5c58fb7f-7xzkk"] Nov 03 01:44:13.576516 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:44:13.576495 4939 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-a932d7e1-5c58fb7f-7xzkk"] Nov 03 01:44:13.586315 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:44:13.586280 4939 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-a932d7e1-5dff9b7b87-wf7p6"] Nov 03 01:44:13.590027 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:44:13.590008 4939 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-a932d7e1-5dff9b7b87-wf7p6"] Nov 03 01:44:13.867356 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:44:13.867320 4939 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="267546ba-79f3-4c43-8c27-b446fcf18a10" path="/var/lib/kubelet/pods/267546ba-79f3-4c43-8c27-b446fcf18a10/volumes" Nov 03 01:44:13.867722 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:44:13.867708 4939 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f2d38bf5-57c3-46f2-8c10-5b3468bbd3b9" path="/var/lib/kubelet/pods/f2d38bf5-57c3-46f2-8c10-5b3468bbd3b9/volumes" Nov 03 01:44:14.129745 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:44:14.129641 4939 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-a932d7e1-openshift-default-994d78cb5-qwmlw"] Nov 03 01:44:14.130360 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:44:14.129885 4939 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ait-a932d7e1-openshift-default-994d78cb5-qwmlw" podUID="1624410a-c346-403b-bb74-9f18b5f5aada" containerName="istio-proxy" containerID="cri-o://20fdb9add42b4d555b8d85a55eabf4704a7c0cbb4b07b25e9211b69b414e8b5e" gracePeriod=30 Nov 03 01:44:14.190263 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:44:14.190231 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:44:15.783975 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:44:15.783939 4939 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-7b85f9594f-9gw68"] Nov 03 01:44:15.784462 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:44:15.784133 4939 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/maas-controller-7b85f9594f-9gw68" podUID="071dd59f-2a89-48c6-aea0-2a552f5bae08" containerName="manager" containerID="cri-o://aa982ab5e129d0ec9dbefb32ca09899961f0b1c2328c2dd029a141f3912d1c69" gracePeriod=30 Nov 03 01:44:16.039922 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:44:16.039849 4939 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-7b85f9594f-9gw68" Nov 03 01:44:16.069470 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:44:16.069442 4939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/071dd59f-2a89-48c6-aea0-2a552f5bae08-kube-api-access-cgzlc\" (UniqueName: \"kubernetes.io/projected/071dd59f-2a89-48c6-aea0-2a552f5bae08-kube-api-access-cgzlc\") pod \"071dd59f-2a89-48c6-aea0-2a552f5bae08\" (UID: \"071dd59f-2a89-48c6-aea0-2a552f5bae08\") " Nov 03 01:44:16.069658 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:44:16.069514 4939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/071dd59f-2a89-48c6-aea0-2a552f5bae08-metrics-certs\" (UniqueName: \"kubernetes.io/secret/071dd59f-2a89-48c6-aea0-2a552f5bae08-metrics-certs\") pod \"071dd59f-2a89-48c6-aea0-2a552f5bae08\" (UID: \"071dd59f-2a89-48c6-aea0-2a552f5bae08\") " Nov 03 01:44:16.069658 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:44:16.069564 4939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/071dd59f-2a89-48c6-aea0-2a552f5bae08-webhook-cert\" (UniqueName: \"kubernetes.io/secret/071dd59f-2a89-48c6-aea0-2a552f5bae08-webhook-cert\") pod \"071dd59f-2a89-48c6-aea0-2a552f5bae08\" (UID: \"071dd59f-2a89-48c6-aea0-2a552f5bae08\") " Nov 03 01:44:16.071800 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:44:16.071752 4939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/071dd59f-2a89-48c6-aea0-2a552f5bae08-webhook-cert" pod "071dd59f-2a89-48c6-aea0-2a552f5bae08" (UID: "071dd59f-2a89-48c6-aea0-2a552f5bae08"). InnerVolumeSpecName "webhook-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 01:44:16.071963 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:44:16.071833 4939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/071dd59f-2a89-48c6-aea0-2a552f5bae08-kube-api-access-cgzlc" pod "071dd59f-2a89-48c6-aea0-2a552f5bae08" (UID: "071dd59f-2a89-48c6-aea0-2a552f5bae08"). InnerVolumeSpecName "kube-api-access-cgzlc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 01:44:16.071963 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:44:16.071856 4939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/071dd59f-2a89-48c6-aea0-2a552f5bae08-metrics-certs" pod "071dd59f-2a89-48c6-aea0-2a552f5bae08" (UID: "071dd59f-2a89-48c6-aea0-2a552f5bae08"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 01:44:16.171256 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:44:16.171223 4939 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/071dd59f-2a89-48c6-aea0-2a552f5bae08-metrics-certs\") on node \"ip-10-0-140-107.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:44:16.171256 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:44:16.171252 4939 reconciler_common.go:299] "Volume detached for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/071dd59f-2a89-48c6-aea0-2a552f5bae08-webhook-cert\") on node \"ip-10-0-140-107.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:44:16.171256 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:44:16.171263 4939 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cgzlc\" (UniqueName: \"kubernetes.io/projected/071dd59f-2a89-48c6-aea0-2a552f5bae08-kube-api-access-cgzlc\") on node \"ip-10-0-140-107.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:44:16.564453 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:44:16.564417 4939 generic.go:356] "Generic (PLEG): container finished" podID="071dd59f-2a89-48c6-aea0-2a552f5bae08" containerID="aa982ab5e129d0ec9dbefb32ca09899961f0b1c2328c2dd029a141f3912d1c69" exitCode=0 Nov 03 01:44:16.564630 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:44:16.564464 4939 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-7b85f9594f-9gw68" Nov 03 01:44:16.564630 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:44:16.564508 4939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-7b85f9594f-9gw68" event={"ID":"071dd59f-2a89-48c6-aea0-2a552f5bae08","Type":"ContainerDied","Data":"aa982ab5e129d0ec9dbefb32ca09899961f0b1c2328c2dd029a141f3912d1c69"} Nov 03 01:44:16.564630 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:44:16.564565 4939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-7b85f9594f-9gw68" event={"ID":"071dd59f-2a89-48c6-aea0-2a552f5bae08","Type":"ContainerDied","Data":"a50cc92868a148c2492344807eba85ef2e3a675a2c8f721b6b220f945ab5e343"} Nov 03 01:44:16.564630 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:44:16.564588 4939 scope.go:122] "RemoveContainer" containerID="aa982ab5e129d0ec9dbefb32ca09899961f0b1c2328c2dd029a141f3912d1c69" Nov 03 01:44:16.576826 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:44:16.576806 4939 scope.go:122] "RemoveContainer" containerID="aa982ab5e129d0ec9dbefb32ca09899961f0b1c2328c2dd029a141f3912d1c69" Nov 03 01:44:16.577126 ip-10-0-140-107 kubenswrapper[4939]: E1103 01:44:16.577106 4939 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"aa982ab5e129d0ec9dbefb32ca09899961f0b1c2328c2dd029a141f3912d1c69\": container with ID starting with aa982ab5e129d0ec9dbefb32ca09899961f0b1c2328c2dd029a141f3912d1c69 not found: ID does not exist" containerID="aa982ab5e129d0ec9dbefb32ca09899961f0b1c2328c2dd029a141f3912d1c69" Nov 03 01:44:16.577178 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:44:16.577136 4939 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"aa982ab5e129d0ec9dbefb32ca09899961f0b1c2328c2dd029a141f3912d1c69"} err="failed to get container status \"aa982ab5e129d0ec9dbefb32ca09899961f0b1c2328c2dd029a141f3912d1c69\": rpc error: code = NotFound desc = could not find container \"aa982ab5e129d0ec9dbefb32ca09899961f0b1c2328c2dd029a141f3912d1c69\": container with ID starting with aa982ab5e129d0ec9dbefb32ca09899961f0b1c2328c2dd029a141f3912d1c69 not found: ID does not exist" Nov 03 01:44:16.587813 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:44:16.587790 4939 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-7b85f9594f-9gw68"] Nov 03 01:44:16.590189 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:44:16.590168 4939 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-controller-7b85f9594f-9gw68"] Nov 03 01:44:16.902855 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:44:16.902820 4939 patch_prober.go:29] interesting pod/e2e-ait-a932d7e1-openshift-default-994d78cb5-qwmlw container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=upstream connect error or disconnect/reset before headers. reset reason: remote connection failure, transport failure reason: delayed connect error: Connection refused Nov 03 01:44:16.903371 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:44:16.902878 4939 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ait-a932d7e1-openshift-default-994d78cb5-qwmlw" podUID="1624410a-c346-403b-bb74-9f18b5f5aada" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 03 01:44:17.392867 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:44:17.392829 4939 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-7b85f9594f-vwbsl"] Nov 03 01:44:17.393494 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:44:17.393471 4939 cpu_manager.go:414] "containerMap: removing container" podUID="267546ba-79f3-4c43-8c27-b446fcf18a10" containerName="payload-processing" Nov 03 01:44:17.393560 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:44:17.393497 4939 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="267546ba-79f3-4c43-8c27-b446fcf18a10" containerName="payload-processing" Nov 03 01:44:17.393560 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:44:17.393542 4939 cpu_manager.go:414] "containerMap: removing container" podUID="f2d38bf5-57c3-46f2-8c10-5b3468bbd3b9" containerName="payload-pre-processing" Nov 03 01:44:17.393560 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:44:17.393552 4939 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f2d38bf5-57c3-46f2-8c10-5b3468bbd3b9" containerName="payload-pre-processing" Nov 03 01:44:17.393674 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:44:17.393584 4939 cpu_manager.go:414] "containerMap: removing container" podUID="071dd59f-2a89-48c6-aea0-2a552f5bae08" containerName="manager" Nov 03 01:44:17.393674 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:44:17.393593 4939 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="071dd59f-2a89-48c6-aea0-2a552f5bae08" containerName="manager" Nov 03 01:44:17.393674 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:44:17.393610 4939 cpu_manager.go:414] "containerMap: removing container" podUID="4ff16b4f-3938-4184-bc01-603126df8ee6" containerName="maas-api" Nov 03 01:44:17.393674 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:44:17.393618 4939 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4ff16b4f-3938-4184-bc01-603126df8ee6" containerName="maas-api" Nov 03 01:44:17.393811 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:44:17.393715 4939 memory_manager.go:365] "RemoveStaleState removing state" podUID="f2d38bf5-57c3-46f2-8c10-5b3468bbd3b9" containerName="payload-pre-processing" Nov 03 01:44:17.393811 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:44:17.393731 4939 memory_manager.go:365] "RemoveStaleState removing state" podUID="071dd59f-2a89-48c6-aea0-2a552f5bae08" containerName="manager" Nov 03 01:44:17.393811 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:44:17.393742 4939 memory_manager.go:365] "RemoveStaleState removing state" podUID="4ff16b4f-3938-4184-bc01-603126df8ee6" containerName="maas-api" Nov 03 01:44:17.393811 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:44:17.393756 4939 memory_manager.go:365] "RemoveStaleState removing state" podUID="267546ba-79f3-4c43-8c27-b446fcf18a10" containerName="payload-processing" Nov 03 01:44:17.398800 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:44:17.398777 4939 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-7b85f9594f-vwbsl" Nov 03 01:44:17.402969 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:44:17.402872 4939 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"maas-parameters\"" Nov 03 01:44:17.405085 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:44:17.405057 4939 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-webhook-cert\"" Nov 03 01:44:17.405216 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:44:17.405169 4939 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-dockercfg-2xnh6\"" Nov 03 01:44:17.405285 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:44:17.405222 4939 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-metrics-tls\"" Nov 03 01:44:17.406694 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:44:17.406670 4939 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-7b85f9594f-vwbsl"] Nov 03 01:44:17.482567 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:44:17.482526 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/4da21d0c-6130-49ec-8042-97eed34d06e0-webhook-cert\") pod \"maas-controller-7b85f9594f-vwbsl\" (UID: \"4da21d0c-6130-49ec-8042-97eed34d06e0\") " pod="opendatahub/maas-controller-7b85f9594f-vwbsl" Nov 03 01:44:17.482748 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:44:17.482711 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4da21d0c-6130-49ec-8042-97eed34d06e0-metrics-certs\") pod \"maas-controller-7b85f9594f-vwbsl\" (UID: \"4da21d0c-6130-49ec-8042-97eed34d06e0\") " pod="opendatahub/maas-controller-7b85f9594f-vwbsl" Nov 03 01:44:17.482877 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:44:17.482850 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cf7kl\" (UniqueName: \"kubernetes.io/projected/4da21d0c-6130-49ec-8042-97eed34d06e0-kube-api-access-cf7kl\") pod \"maas-controller-7b85f9594f-vwbsl\" (UID: \"4da21d0c-6130-49ec-8042-97eed34d06e0\") " pod="opendatahub/maas-controller-7b85f9594f-vwbsl" Nov 03 01:44:17.583512 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:44:17.583471 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cf7kl\" (UniqueName: \"kubernetes.io/projected/4da21d0c-6130-49ec-8042-97eed34d06e0-kube-api-access-cf7kl\") pod \"maas-controller-7b85f9594f-vwbsl\" (UID: \"4da21d0c-6130-49ec-8042-97eed34d06e0\") " pod="opendatahub/maas-controller-7b85f9594f-vwbsl" Nov 03 01:44:17.589118 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:44:17.589088 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/4da21d0c-6130-49ec-8042-97eed34d06e0-webhook-cert\") pod \"maas-controller-7b85f9594f-vwbsl\" (UID: \"4da21d0c-6130-49ec-8042-97eed34d06e0\") " pod="opendatahub/maas-controller-7b85f9594f-vwbsl" Nov 03 01:44:17.589430 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:44:17.589410 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4da21d0c-6130-49ec-8042-97eed34d06e0-metrics-certs\") pod \"maas-controller-7b85f9594f-vwbsl\" (UID: \"4da21d0c-6130-49ec-8042-97eed34d06e0\") " pod="opendatahub/maas-controller-7b85f9594f-vwbsl" Nov 03 01:44:17.600974 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:44:17.600952 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4da21d0c-6130-49ec-8042-97eed34d06e0-metrics-certs\") pod \"maas-controller-7b85f9594f-vwbsl\" (UID: \"4da21d0c-6130-49ec-8042-97eed34d06e0\") " pod="opendatahub/maas-controller-7b85f9594f-vwbsl" Nov 03 01:44:17.601629 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:44:17.601609 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/4da21d0c-6130-49ec-8042-97eed34d06e0-webhook-cert\") pod \"maas-controller-7b85f9594f-vwbsl\" (UID: \"4da21d0c-6130-49ec-8042-97eed34d06e0\") " pod="opendatahub/maas-controller-7b85f9594f-vwbsl" Nov 03 01:44:17.625644 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:44:17.625618 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cf7kl\" (UniqueName: \"kubernetes.io/projected/4da21d0c-6130-49ec-8042-97eed34d06e0-kube-api-access-cf7kl\") pod \"maas-controller-7b85f9594f-vwbsl\" (UID: \"4da21d0c-6130-49ec-8042-97eed34d06e0\") " pod="opendatahub/maas-controller-7b85f9594f-vwbsl" Nov 03 01:44:17.715373 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:44:17.714931 4939 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-7b85f9594f-vwbsl" Nov 03 01:44:17.941228 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:44:17.941164 4939 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="071dd59f-2a89-48c6-aea0-2a552f5bae08" path="/var/lib/kubelet/pods/071dd59f-2a89-48c6-aea0-2a552f5bae08/volumes" Nov 03 01:44:18.076987 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:44:18.060035 4939 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-7b85f9594f-vwbsl"] Nov 03 01:44:18.127522 ip-10-0-140-107 kubenswrapper[4939]: E1103 01:44:18.127438 4939 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-sthjh" containerName="extractor" Nov 03 01:44:18.620115 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:44:18.620059 4939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-7b85f9594f-vwbsl" event={"ID":"4da21d0c-6130-49ec-8042-97eed34d06e0","Type":"ContainerStarted","Data":"826e363c4d56a1ac2a5e927eebffcee95aee6254bb2c8d5706ffa0d37a9f4e00"} Nov 03 01:44:19.410763 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:44:19.410739 4939 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-a932d7e1-openshift-default-994d78cb5-qwmlw" Nov 03 01:44:19.544318 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:44:19.544231 4939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1624410a-c346-403b-bb74-9f18b5f5aada-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1624410a-c346-403b-bb74-9f18b5f5aada-credential-socket\") pod \"1624410a-c346-403b-bb74-9f18b5f5aada\" (UID: \"1624410a-c346-403b-bb74-9f18b5f5aada\") " Nov 03 01:44:19.544479 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:44:19.544332 4939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1624410a-c346-403b-bb74-9f18b5f5aada-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1624410a-c346-403b-bb74-9f18b5f5aada-istio-data\") pod \"1624410a-c346-403b-bb74-9f18b5f5aada\" (UID: \"1624410a-c346-403b-bb74-9f18b5f5aada\") " Nov 03 01:44:19.544479 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:44:19.544367 4939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/1624410a-c346-403b-bb74-9f18b5f5aada-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1624410a-c346-403b-bb74-9f18b5f5aada-istio-podinfo\") pod \"1624410a-c346-403b-bb74-9f18b5f5aada\" (UID: \"1624410a-c346-403b-bb74-9f18b5f5aada\") " Nov 03 01:44:19.544479 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:44:19.544409 4939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1624410a-c346-403b-bb74-9f18b5f5aada-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1624410a-c346-403b-bb74-9f18b5f5aada-istiod-ca-cert\") pod \"1624410a-c346-403b-bb74-9f18b5f5aada\" (UID: \"1624410a-c346-403b-bb74-9f18b5f5aada\") " Nov 03 01:44:19.544479 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:44:19.544446 4939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1624410a-c346-403b-bb74-9f18b5f5aada-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1624410a-c346-403b-bb74-9f18b5f5aada-workload-socket\") pod \"1624410a-c346-403b-bb74-9f18b5f5aada\" (UID: \"1624410a-c346-403b-bb74-9f18b5f5aada\") " Nov 03 01:44:19.544729 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:44:19.544485 4939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1624410a-c346-403b-bb74-9f18b5f5aada-kube-api-access-g2hp9\" (UniqueName: \"kubernetes.io/projected/1624410a-c346-403b-bb74-9f18b5f5aada-kube-api-access-g2hp9\") pod \"1624410a-c346-403b-bb74-9f18b5f5aada\" (UID: \"1624410a-c346-403b-bb74-9f18b5f5aada\") " Nov 03 01:44:19.544729 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:44:19.544548 4939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1624410a-c346-403b-bb74-9f18b5f5aada-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1624410a-c346-403b-bb74-9f18b5f5aada-istio-envoy\") pod \"1624410a-c346-403b-bb74-9f18b5f5aada\" (UID: \"1624410a-c346-403b-bb74-9f18b5f5aada\") " Nov 03 01:44:19.544729 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:44:19.544553 4939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1624410a-c346-403b-bb74-9f18b5f5aada-credential-socket" pod "1624410a-c346-403b-bb74-9f18b5f5aada" (UID: "1624410a-c346-403b-bb74-9f18b5f5aada"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 01:44:19.544729 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:44:19.544584 4939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1624410a-c346-403b-bb74-9f18b5f5aada-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1624410a-c346-403b-bb74-9f18b5f5aada-workload-certs\") pod \"1624410a-c346-403b-bb74-9f18b5f5aada\" (UID: \"1624410a-c346-403b-bb74-9f18b5f5aada\") " Nov 03 01:44:19.544729 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:44:19.544636 4939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1624410a-c346-403b-bb74-9f18b5f5aada-istio-data" pod "1624410a-c346-403b-bb74-9f18b5f5aada" (UID: "1624410a-c346-403b-bb74-9f18b5f5aada"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 01:44:19.544729 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:44:19.544662 4939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1624410a-c346-403b-bb74-9f18b5f5aada-istio-token\" (UniqueName: \"kubernetes.io/projected/1624410a-c346-403b-bb74-9f18b5f5aada-istio-token\") pod \"1624410a-c346-403b-bb74-9f18b5f5aada\" (UID: \"1624410a-c346-403b-bb74-9f18b5f5aada\") " Nov 03 01:44:19.545123 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:44:19.544826 4939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1624410a-c346-403b-bb74-9f18b5f5aada-workload-socket" pod "1624410a-c346-403b-bb74-9f18b5f5aada" (UID: "1624410a-c346-403b-bb74-9f18b5f5aada"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 01:44:19.545123 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:44:19.544894 4939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1624410a-c346-403b-bb74-9f18b5f5aada-workload-certs" pod "1624410a-c346-403b-bb74-9f18b5f5aada" (UID: "1624410a-c346-403b-bb74-9f18b5f5aada"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 01:44:19.545123 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:44:19.544952 4939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1624410a-c346-403b-bb74-9f18b5f5aada-istiod-ca-cert" pod "1624410a-c346-403b-bb74-9f18b5f5aada" (UID: "1624410a-c346-403b-bb74-9f18b5f5aada"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 01:44:19.545123 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:44:19.545059 4939 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1624410a-c346-403b-bb74-9f18b5f5aada-credential-socket\") on node \"ip-10-0-140-107.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:44:19.545123 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:44:19.545080 4939 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1624410a-c346-403b-bb74-9f18b5f5aada-istio-data\") on node \"ip-10-0-140-107.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:44:19.545123 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:44:19.545093 4939 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1624410a-c346-403b-bb74-9f18b5f5aada-istiod-ca-cert\") on node \"ip-10-0-140-107.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:44:19.545123 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:44:19.545107 4939 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1624410a-c346-403b-bb74-9f18b5f5aada-workload-socket\") on node \"ip-10-0-140-107.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:44:19.545123 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:44:19.545122 4939 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1624410a-c346-403b-bb74-9f18b5f5aada-workload-certs\") on node \"ip-10-0-140-107.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:44:19.546655 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:44:19.546631 4939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/1624410a-c346-403b-bb74-9f18b5f5aada-istio-podinfo" pod "1624410a-c346-403b-bb74-9f18b5f5aada" (UID: "1624410a-c346-403b-bb74-9f18b5f5aada"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 01:44:19.546799 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:44:19.546779 4939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1624410a-c346-403b-bb74-9f18b5f5aada-kube-api-access-g2hp9" pod "1624410a-c346-403b-bb74-9f18b5f5aada" (UID: "1624410a-c346-403b-bb74-9f18b5f5aada"). InnerVolumeSpecName "kube-api-access-g2hp9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 01:44:19.547070 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:44:19.547053 4939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1624410a-c346-403b-bb74-9f18b5f5aada-istio-envoy" pod "1624410a-c346-403b-bb74-9f18b5f5aada" (UID: "1624410a-c346-403b-bb74-9f18b5f5aada"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 01:44:19.547188 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:44:19.547169 4939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1624410a-c346-403b-bb74-9f18b5f5aada-istio-token" pod "1624410a-c346-403b-bb74-9f18b5f5aada" (UID: "1624410a-c346-403b-bb74-9f18b5f5aada"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 01:44:19.627000 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:44:19.626964 4939 generic.go:356] "Generic (PLEG): container finished" podID="1624410a-c346-403b-bb74-9f18b5f5aada" containerID="20fdb9add42b4d555b8d85a55eabf4704a7c0cbb4b07b25e9211b69b414e8b5e" exitCode=0 Nov 03 01:44:19.627153 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:44:19.627030 4939 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-a932d7e1-openshift-default-994d78cb5-qwmlw" Nov 03 01:44:19.627153 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:44:19.627049 4939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-a932d7e1-openshift-default-994d78cb5-qwmlw" event={"ID":"1624410a-c346-403b-bb74-9f18b5f5aada","Type":"ContainerDied","Data":"20fdb9add42b4d555b8d85a55eabf4704a7c0cbb4b07b25e9211b69b414e8b5e"} Nov 03 01:44:19.627153 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:44:19.627102 4939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-a932d7e1-openshift-default-994d78cb5-qwmlw" event={"ID":"1624410a-c346-403b-bb74-9f18b5f5aada","Type":"ContainerDied","Data":"db34ee0a52b57789aa91da0c7c1f28670075e9276b0ffa6427402e6593270949"} Nov 03 01:44:19.627153 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:44:19.627128 4939 scope.go:122] "RemoveContainer" containerID="20fdb9add42b4d555b8d85a55eabf4704a7c0cbb4b07b25e9211b69b414e8b5e" Nov 03 01:44:19.628852 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:44:19.628827 4939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-7b85f9594f-vwbsl" event={"ID":"4da21d0c-6130-49ec-8042-97eed34d06e0","Type":"ContainerStarted","Data":"7cba2617f729abfa5529b22dc4a9d78f3789eefe7799e5278a33a1359c479149"} Nov 03 01:44:19.636445 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:44:19.636427 4939 scope.go:122] "RemoveContainer" containerID="c1c38528a4010f07925cf47b3dc19fccca68aa360ef58796e56d187c8302060f" Nov 03 01:44:19.645354 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:44:19.645333 4939 scope.go:122] "RemoveContainer" containerID="20fdb9add42b4d555b8d85a55eabf4704a7c0cbb4b07b25e9211b69b414e8b5e" Nov 03 01:44:19.645642 ip-10-0-140-107 kubenswrapper[4939]: E1103 01:44:19.645619 4939 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"20fdb9add42b4d555b8d85a55eabf4704a7c0cbb4b07b25e9211b69b414e8b5e\": container with ID starting with 20fdb9add42b4d555b8d85a55eabf4704a7c0cbb4b07b25e9211b69b414e8b5e not found: ID does not exist" containerID="20fdb9add42b4d555b8d85a55eabf4704a7c0cbb4b07b25e9211b69b414e8b5e" Nov 03 01:44:19.645707 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:44:19.645650 4939 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"20fdb9add42b4d555b8d85a55eabf4704a7c0cbb4b07b25e9211b69b414e8b5e"} err="failed to get container status \"20fdb9add42b4d555b8d85a55eabf4704a7c0cbb4b07b25e9211b69b414e8b5e\": rpc error: code = NotFound desc = could not find container \"20fdb9add42b4d555b8d85a55eabf4704a7c0cbb4b07b25e9211b69b414e8b5e\": container with ID starting with 20fdb9add42b4d555b8d85a55eabf4704a7c0cbb4b07b25e9211b69b414e8b5e not found: ID does not exist" Nov 03 01:44:19.645707 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:44:19.645666 4939 scope.go:122] "RemoveContainer" containerID="c1c38528a4010f07925cf47b3dc19fccca68aa360ef58796e56d187c8302060f" Nov 03 01:44:19.645846 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:44:19.645792 4939 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1624410a-c346-403b-bb74-9f18b5f5aada-istio-token\") on node \"ip-10-0-140-107.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:44:19.645846 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:44:19.645815 4939 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1624410a-c346-403b-bb74-9f18b5f5aada-istio-podinfo\") on node \"ip-10-0-140-107.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:44:19.645846 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:44:19.645828 4939 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-g2hp9\" (UniqueName: \"kubernetes.io/projected/1624410a-c346-403b-bb74-9f18b5f5aada-kube-api-access-g2hp9\") on node \"ip-10-0-140-107.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:44:19.645846 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:44:19.645838 4939 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1624410a-c346-403b-bb74-9f18b5f5aada-istio-envoy\") on node \"ip-10-0-140-107.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:44:19.646103 ip-10-0-140-107 kubenswrapper[4939]: E1103 01:44:19.646018 4939 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c1c38528a4010f07925cf47b3dc19fccca68aa360ef58796e56d187c8302060f\": container with ID starting with c1c38528a4010f07925cf47b3dc19fccca68aa360ef58796e56d187c8302060f not found: ID does not exist" containerID="c1c38528a4010f07925cf47b3dc19fccca68aa360ef58796e56d187c8302060f" Nov 03 01:44:19.646103 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:44:19.646040 4939 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c1c38528a4010f07925cf47b3dc19fccca68aa360ef58796e56d187c8302060f"} err="failed to get container status \"c1c38528a4010f07925cf47b3dc19fccca68aa360ef58796e56d187c8302060f\": rpc error: code = NotFound desc = could not find container \"c1c38528a4010f07925cf47b3dc19fccca68aa360ef58796e56d187c8302060f\": container with ID starting with c1c38528a4010f07925cf47b3dc19fccca68aa360ef58796e56d187c8302060f not found: ID does not exist" Nov 03 01:44:19.648547 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:44:19.648498 4939 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/maas-controller-7b85f9594f-vwbsl" podStartSLOduration=1.7652810429999999 podStartE2EDuration="2.648483244s" totalImagesPullingTime="883.202201ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 01:44:17 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 01:44:19.646949277 +0000 UTC m=+3807.911284219" watchObservedRunningTime="2026-11-03 01:44:19.648483244 +0000 UTC m=+3807.912818187" Nov 03 01:44:19.660583 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:44:19.660558 4939 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-a932d7e1-openshift-default-994d78cb5-qwmlw"] Nov 03 01:44:19.664362 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:44:19.664340 4939 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ait-a932d7e1-openshift-default-994d78cb5-qwmlw"] Nov 03 01:44:19.867814 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:44:19.867784 4939 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1624410a-c346-403b-bb74-9f18b5f5aada" path="/var/lib/kubelet/pods/1624410a-c346-403b-bb74-9f18b5f5aada/volumes" Nov 03 01:44:20.634612 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:44:20.634578 4939 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-7b85f9594f-vwbsl" Nov 03 01:44:20.676616 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:44:20.676584 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:44:20.786869 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:44:20.786835 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:44:31.641409 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:44:31.641376 4939 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-7b85f9594f-vwbsl" Nov 03 01:44:31.732342 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:44:31.732301 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:44:31.862164 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:44:31.862128 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:44:38.009702 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:44:38.009607 4939 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-8fv7k"] Nov 03 01:44:38.010335 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:44:38.010231 4939 cpu_manager.go:414] "containerMap: removing container" podUID="1624410a-c346-403b-bb74-9f18b5f5aada" containerName="istio-proxy" Nov 03 01:44:38.010335 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:44:38.010254 4939 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1624410a-c346-403b-bb74-9f18b5f5aada" containerName="istio-proxy" Nov 03 01:44:38.010335 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:44:38.010303 4939 cpu_manager.go:414] "containerMap: removing container" podUID="1624410a-c346-403b-bb74-9f18b5f5aada" containerName="istio-proxy" Nov 03 01:44:38.010335 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:44:38.010311 4939 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1624410a-c346-403b-bb74-9f18b5f5aada" containerName="istio-proxy" Nov 03 01:44:38.010587 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:44:38.010395 4939 memory_manager.go:365] "RemoveStaleState removing state" podUID="1624410a-c346-403b-bb74-9f18b5f5aada" containerName="istio-proxy" Nov 03 01:44:38.010587 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:44:38.010413 4939 memory_manager.go:365] "RemoveStaleState removing state" podUID="1624410a-c346-403b-bb74-9f18b5f5aada" containerName="istio-proxy" Nov 03 01:44:38.013092 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:44:38.013070 4939 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-8fv7k" Nov 03 01:44:38.019366 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:44:38.019331 4939 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 03 01:44:38.019524 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:44:38.019363 4939 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-swfcr\"" Nov 03 01:44:38.019524 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:44:38.019422 4939 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 03 01:44:38.020435 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:44:38.020009 4939 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 01:44:38.021683 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:44:38.021663 4939 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-8fv7k"] Nov 03 01:44:38.113293 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:44:38.113258 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3655425e-192d-4875-b633-19ca4e7309de-maas-api-xks-ca\") pod \"maas-api-revoke-sub-85237e45-8fv7k\" (UID: \"3655425e-192d-4875-b633-19ca4e7309de\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-8fv7k" Nov 03 01:44:38.113501 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:44:38.113307 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s8vrw\" (UniqueName: \"kubernetes.io/projected/3655425e-192d-4875-b633-19ca4e7309de-kube-api-access-s8vrw\") pod \"maas-api-revoke-sub-85237e45-8fv7k\" (UID: \"3655425e-192d-4875-b633-19ca4e7309de\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-8fv7k" Nov 03 01:44:38.113501 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:44:38.113429 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3655425e-192d-4875-b633-19ca4e7309de-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-85237e45-8fv7k\" (UID: \"3655425e-192d-4875-b633-19ca4e7309de\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-8fv7k" Nov 03 01:44:38.214899 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:44:38.214862 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3655425e-192d-4875-b633-19ca4e7309de-maas-api-xks-ca\") pod \"maas-api-revoke-sub-85237e45-8fv7k\" (UID: \"3655425e-192d-4875-b633-19ca4e7309de\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-8fv7k" Nov 03 01:44:38.215116 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:44:38.214945 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s8vrw\" (UniqueName: \"kubernetes.io/projected/3655425e-192d-4875-b633-19ca4e7309de-kube-api-access-s8vrw\") pod \"maas-api-revoke-sub-85237e45-8fv7k\" (UID: \"3655425e-192d-4875-b633-19ca4e7309de\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-8fv7k" Nov 03 01:44:38.215116 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:44:38.214988 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3655425e-192d-4875-b633-19ca4e7309de-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-85237e45-8fv7k\" (UID: \"3655425e-192d-4875-b633-19ca4e7309de\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-8fv7k" Nov 03 01:44:38.215730 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:44:38.215702 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3655425e-192d-4875-b633-19ca4e7309de-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-85237e45-8fv7k\" (UID: \"3655425e-192d-4875-b633-19ca4e7309de\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-8fv7k" Nov 03 01:44:38.217357 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:44:38.217333 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3655425e-192d-4875-b633-19ca4e7309de-maas-api-xks-ca\") pod \"maas-api-revoke-sub-85237e45-8fv7k\" (UID: \"3655425e-192d-4875-b633-19ca4e7309de\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-8fv7k" Nov 03 01:44:38.222819 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:44:38.222799 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-s8vrw\" (UniqueName: \"kubernetes.io/projected/3655425e-192d-4875-b633-19ca4e7309de-kube-api-access-s8vrw\") pod \"maas-api-revoke-sub-85237e45-8fv7k\" (UID: \"3655425e-192d-4875-b633-19ca4e7309de\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-8fv7k" Nov 03 01:44:38.326725 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:44:38.326639 4939 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-8fv7k" Nov 03 01:44:38.596727 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:44:38.596504 4939 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-8fv7k"] Nov 03 01:44:38.598987 ip-10-0-140-107 kubenswrapper[4939]: W1103 01:44:38.598960 4939 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3655425e_192d_4875_b633_19ca4e7309de.slice/crio-8d71fad56f09b5051f938dbc5713e9466ec83edd57012e7129e4475568a67268 WatchSource:0}: Error finding container 8d71fad56f09b5051f938dbc5713e9466ec83edd57012e7129e4475568a67268: Status 404 returned error can't find the container with id 8d71fad56f09b5051f938dbc5713e9466ec83edd57012e7129e4475568a67268 Nov 03 01:44:38.721193 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:44:38.721157 4939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-8fv7k" event={"ID":"3655425e-192d-4875-b633-19ca4e7309de","Type":"ContainerStarted","Data":"8d71fad56f09b5051f938dbc5713e9466ec83edd57012e7129e4475568a67268"} Nov 03 01:44:39.727545 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:44:39.727507 4939 generic.go:356] "Generic (PLEG): container finished" podID="3655425e-192d-4875-b633-19ca4e7309de" containerID="5cd2815b53a556d0054cbda9ca75ff3c5d36dd4b81b23f0f53151a4fb36a7a9a" exitCode=0 Nov 03 01:44:39.727980 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:44:39.727593 4939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-8fv7k" event={"ID":"3655425e-192d-4875-b633-19ca4e7309de","Type":"ContainerDied","Data":"5cd2815b53a556d0054cbda9ca75ff3c5d36dd4b81b23f0f53151a4fb36a7a9a"} Nov 03 01:44:40.869415 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:44:40.869390 4939 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-8fv7k" Nov 03 01:44:40.939640 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:44:40.939599 4939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3655425e-192d-4875-b633-19ca4e7309de-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3655425e-192d-4875-b633-19ca4e7309de-maas-api-xks-ca\") pod \"3655425e-192d-4875-b633-19ca4e7309de\" (UID: \"3655425e-192d-4875-b633-19ca4e7309de\") " Nov 03 01:44:40.939810 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:44:40.939675 4939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3655425e-192d-4875-b633-19ca4e7309de-kube-api-access-s8vrw\" (UniqueName: \"kubernetes.io/projected/3655425e-192d-4875-b633-19ca4e7309de-kube-api-access-s8vrw\") pod \"3655425e-192d-4875-b633-19ca4e7309de\" (UID: \"3655425e-192d-4875-b633-19ca4e7309de\") " Nov 03 01:44:40.939810 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:44:40.939710 4939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3655425e-192d-4875-b633-19ca4e7309de-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3655425e-192d-4875-b633-19ca4e7309de-maas-api-openshift-ca\") pod \"3655425e-192d-4875-b633-19ca4e7309de\" (UID: \"3655425e-192d-4875-b633-19ca4e7309de\") " Nov 03 01:44:40.940153 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:44:40.940125 4939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3655425e-192d-4875-b633-19ca4e7309de-maas-api-openshift-ca" pod "3655425e-192d-4875-b633-19ca4e7309de" (UID: "3655425e-192d-4875-b633-19ca4e7309de"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 01:44:40.941817 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:44:40.941797 4939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3655425e-192d-4875-b633-19ca4e7309de-maas-api-xks-ca" pod "3655425e-192d-4875-b633-19ca4e7309de" (UID: "3655425e-192d-4875-b633-19ca4e7309de"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 01:44:40.941901 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:44:40.941869 4939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3655425e-192d-4875-b633-19ca4e7309de-kube-api-access-s8vrw" pod "3655425e-192d-4875-b633-19ca4e7309de" (UID: "3655425e-192d-4875-b633-19ca4e7309de"). InnerVolumeSpecName "kube-api-access-s8vrw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 01:44:41.041057 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:44:41.040984 4939 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3655425e-192d-4875-b633-19ca4e7309de-maas-api-xks-ca\") on node \"ip-10-0-140-107.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:44:41.041057 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:44:41.041021 4939 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-s8vrw\" (UniqueName: \"kubernetes.io/projected/3655425e-192d-4875-b633-19ca4e7309de-kube-api-access-s8vrw\") on node \"ip-10-0-140-107.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:44:41.041057 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:44:41.041032 4939 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3655425e-192d-4875-b633-19ca4e7309de-maas-api-openshift-ca\") on node \"ip-10-0-140-107.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:44:41.737294 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:44:41.737260 4939 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-8fv7k" Nov 03 01:44:41.737487 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:44:41.737263 4939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-8fv7k" event={"ID":"3655425e-192d-4875-b633-19ca4e7309de","Type":"ContainerDied","Data":"8d71fad56f09b5051f938dbc5713e9466ec83edd57012e7129e4475568a67268"} Nov 03 01:44:41.737487 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:44:41.737389 4939 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="8d71fad56f09b5051f938dbc5713e9466ec83edd57012e7129e4475568a67268" Nov 03 01:44:46.128330 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:44:46.128291 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:44:48.030246 ip-10-0-140-107 kubenswrapper[4939]: E1103 01:44:48.030206 4939 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-sthjh" containerName="extractor" Nov 03 01:44:48.999101 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:44:48.999063 4939 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-7kxj4"] Nov 03 01:44:48.999566 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:44:48.999549 4939 cpu_manager.go:414] "containerMap: removing container" podUID="3655425e-192d-4875-b633-19ca4e7309de" containerName="revoke-keys" Nov 03 01:44:48.999629 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:44:48.999570 4939 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3655425e-192d-4875-b633-19ca4e7309de" containerName="revoke-keys" Nov 03 01:44:48.999671 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:44:48.999658 4939 memory_manager.go:365] "RemoveStaleState removing state" podUID="3655425e-192d-4875-b633-19ca4e7309de" containerName="revoke-keys" Nov 03 01:44:49.003038 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:44:49.003017 4939 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-7kxj4" Nov 03 01:44:49.015160 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:44:49.015133 4939 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-7kxj4"] Nov 03 01:44:49.113711 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:44:49.113676 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8acf90f6-6fff-4a69-b67d-c3ba365c6103-workload-socket\") pod \"maas-default-gateway-openshift-default-596dc9545f-7kxj4\" (UID: \"8acf90f6-6fff-4a69-b67d-c3ba365c6103\") " pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-7kxj4" Nov 03 01:44:49.113711 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:44:49.113714 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8acf90f6-6fff-4a69-b67d-c3ba365c6103-istio-podinfo\") pod \"maas-default-gateway-openshift-default-596dc9545f-7kxj4\" (UID: \"8acf90f6-6fff-4a69-b67d-c3ba365c6103\") " pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-7kxj4" Nov 03 01:44:49.114265 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:44:49.113736 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8acf90f6-6fff-4a69-b67d-c3ba365c6103-istio-token\") pod \"maas-default-gateway-openshift-default-596dc9545f-7kxj4\" (UID: \"8acf90f6-6fff-4a69-b67d-c3ba365c6103\") " pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-7kxj4" Nov 03 01:44:49.114265 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:44:49.113798 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8acf90f6-6fff-4a69-b67d-c3ba365c6103-istio-data\") pod \"maas-default-gateway-openshift-default-596dc9545f-7kxj4\" (UID: \"8acf90f6-6fff-4a69-b67d-c3ba365c6103\") " pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-7kxj4" Nov 03 01:44:49.114265 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:44:49.113903 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8acf90f6-6fff-4a69-b67d-c3ba365c6103-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-596dc9545f-7kxj4\" (UID: \"8acf90f6-6fff-4a69-b67d-c3ba365c6103\") " pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-7kxj4" Nov 03 01:44:49.114265 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:44:49.113983 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8acf90f6-6fff-4a69-b67d-c3ba365c6103-credential-socket\") pod \"maas-default-gateway-openshift-default-596dc9545f-7kxj4\" (UID: \"8acf90f6-6fff-4a69-b67d-c3ba365c6103\") " pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-7kxj4" Nov 03 01:44:49.114265 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:44:49.114010 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jl7pv\" (UniqueName: \"kubernetes.io/projected/8acf90f6-6fff-4a69-b67d-c3ba365c6103-kube-api-access-jl7pv\") pod \"maas-default-gateway-openshift-default-596dc9545f-7kxj4\" (UID: \"8acf90f6-6fff-4a69-b67d-c3ba365c6103\") " pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-7kxj4" Nov 03 01:44:49.114265 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:44:49.114135 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8acf90f6-6fff-4a69-b67d-c3ba365c6103-istio-envoy\") pod \"maas-default-gateway-openshift-default-596dc9545f-7kxj4\" (UID: \"8acf90f6-6fff-4a69-b67d-c3ba365c6103\") " pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-7kxj4" Nov 03 01:44:49.114265 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:44:49.114177 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8acf90f6-6fff-4a69-b67d-c3ba365c6103-workload-certs\") pod \"maas-default-gateway-openshift-default-596dc9545f-7kxj4\" (UID: \"8acf90f6-6fff-4a69-b67d-c3ba365c6103\") " pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-7kxj4" Nov 03 01:44:49.215225 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:44:49.215189 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8acf90f6-6fff-4a69-b67d-c3ba365c6103-credential-socket\") pod \"maas-default-gateway-openshift-default-596dc9545f-7kxj4\" (UID: \"8acf90f6-6fff-4a69-b67d-c3ba365c6103\") " pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-7kxj4" Nov 03 01:44:49.215407 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:44:49.215236 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jl7pv\" (UniqueName: \"kubernetes.io/projected/8acf90f6-6fff-4a69-b67d-c3ba365c6103-kube-api-access-jl7pv\") pod \"maas-default-gateway-openshift-default-596dc9545f-7kxj4\" (UID: \"8acf90f6-6fff-4a69-b67d-c3ba365c6103\") " pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-7kxj4" Nov 03 01:44:49.215407 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:44:49.215293 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8acf90f6-6fff-4a69-b67d-c3ba365c6103-istio-envoy\") pod \"maas-default-gateway-openshift-default-596dc9545f-7kxj4\" (UID: \"8acf90f6-6fff-4a69-b67d-c3ba365c6103\") " pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-7kxj4" Nov 03 01:44:49.215407 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:44:49.215324 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8acf90f6-6fff-4a69-b67d-c3ba365c6103-workload-certs\") pod \"maas-default-gateway-openshift-default-596dc9545f-7kxj4\" (UID: \"8acf90f6-6fff-4a69-b67d-c3ba365c6103\") " pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-7kxj4" Nov 03 01:44:49.215407 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:44:49.215356 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8acf90f6-6fff-4a69-b67d-c3ba365c6103-workload-socket\") pod \"maas-default-gateway-openshift-default-596dc9545f-7kxj4\" (UID: \"8acf90f6-6fff-4a69-b67d-c3ba365c6103\") " pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-7kxj4" Nov 03 01:44:49.215407 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:44:49.215381 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8acf90f6-6fff-4a69-b67d-c3ba365c6103-istio-podinfo\") pod \"maas-default-gateway-openshift-default-596dc9545f-7kxj4\" (UID: \"8acf90f6-6fff-4a69-b67d-c3ba365c6103\") " pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-7kxj4" Nov 03 01:44:49.215711 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:44:49.215408 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8acf90f6-6fff-4a69-b67d-c3ba365c6103-istio-token\") pod \"maas-default-gateway-openshift-default-596dc9545f-7kxj4\" (UID: \"8acf90f6-6fff-4a69-b67d-c3ba365c6103\") " pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-7kxj4" Nov 03 01:44:49.215711 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:44:49.215434 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8acf90f6-6fff-4a69-b67d-c3ba365c6103-istio-data\") pod \"maas-default-gateway-openshift-default-596dc9545f-7kxj4\" (UID: \"8acf90f6-6fff-4a69-b67d-c3ba365c6103\") " pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-7kxj4" Nov 03 01:44:49.215711 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:44:49.215509 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8acf90f6-6fff-4a69-b67d-c3ba365c6103-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-596dc9545f-7kxj4\" (UID: \"8acf90f6-6fff-4a69-b67d-c3ba365c6103\") " pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-7kxj4" Nov 03 01:44:49.215711 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:44:49.215634 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8acf90f6-6fff-4a69-b67d-c3ba365c6103-credential-socket\") pod \"maas-default-gateway-openshift-default-596dc9545f-7kxj4\" (UID: \"8acf90f6-6fff-4a69-b67d-c3ba365c6103\") " pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-7kxj4" Nov 03 01:44:49.216013 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:44:49.215787 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8acf90f6-6fff-4a69-b67d-c3ba365c6103-workload-certs\") pod \"maas-default-gateway-openshift-default-596dc9545f-7kxj4\" (UID: \"8acf90f6-6fff-4a69-b67d-c3ba365c6103\") " pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-7kxj4" Nov 03 01:44:49.216013 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:44:49.215863 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8acf90f6-6fff-4a69-b67d-c3ba365c6103-istio-data\") pod \"maas-default-gateway-openshift-default-596dc9545f-7kxj4\" (UID: \"8acf90f6-6fff-4a69-b67d-c3ba365c6103\") " pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-7kxj4" Nov 03 01:44:49.216013 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:44:49.215887 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8acf90f6-6fff-4a69-b67d-c3ba365c6103-workload-socket\") pod \"maas-default-gateway-openshift-default-596dc9545f-7kxj4\" (UID: \"8acf90f6-6fff-4a69-b67d-c3ba365c6103\") " pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-7kxj4" Nov 03 01:44:49.216308 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:44:49.216286 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8acf90f6-6fff-4a69-b67d-c3ba365c6103-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-596dc9545f-7kxj4\" (UID: \"8acf90f6-6fff-4a69-b67d-c3ba365c6103\") " pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-7kxj4" Nov 03 01:44:49.217729 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:44:49.217699 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8acf90f6-6fff-4a69-b67d-c3ba365c6103-istio-envoy\") pod \"maas-default-gateway-openshift-default-596dc9545f-7kxj4\" (UID: \"8acf90f6-6fff-4a69-b67d-c3ba365c6103\") " pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-7kxj4" Nov 03 01:44:49.217829 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:44:49.217798 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8acf90f6-6fff-4a69-b67d-c3ba365c6103-istio-podinfo\") pod \"maas-default-gateway-openshift-default-596dc9545f-7kxj4\" (UID: \"8acf90f6-6fff-4a69-b67d-c3ba365c6103\") " pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-7kxj4" Nov 03 01:44:49.223607 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:44:49.223583 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8acf90f6-6fff-4a69-b67d-c3ba365c6103-istio-token\") pod \"maas-default-gateway-openshift-default-596dc9545f-7kxj4\" (UID: \"8acf90f6-6fff-4a69-b67d-c3ba365c6103\") " pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-7kxj4" Nov 03 01:44:49.223891 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:44:49.223873 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jl7pv\" (UniqueName: \"kubernetes.io/projected/8acf90f6-6fff-4a69-b67d-c3ba365c6103-kube-api-access-jl7pv\") pod \"maas-default-gateway-openshift-default-596dc9545f-7kxj4\" (UID: \"8acf90f6-6fff-4a69-b67d-c3ba365c6103\") " pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-7kxj4" Nov 03 01:44:49.317197 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:44:49.317095 4939 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-7kxj4" Nov 03 01:44:49.474567 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:44:49.474531 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:44:49.584699 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:44:49.584669 4939 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-7b85f9594f-vwbsl"] Nov 03 01:44:49.584935 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:44:49.584896 4939 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/maas-controller-7b85f9594f-vwbsl" podUID="4da21d0c-6130-49ec-8042-97eed34d06e0" containerName="manager" containerID="cri-o://7cba2617f729abfa5529b22dc4a9d78f3789eefe7799e5278a33a1359c479149" gracePeriod=30 Nov 03 01:44:49.599928 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:44:49.599885 4939 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-7kxj4"] Nov 03 01:44:49.601924 ip-10-0-140-107 kubenswrapper[4939]: W1103 01:44:49.601889 4939 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8acf90f6_6fff_4a69_b67d_c3ba365c6103.slice/crio-7e8f1a4d3f898a249f2f27bfeb6359ff353d978ab49b0ff89d086a347de80c70 WatchSource:0}: Error finding container 7e8f1a4d3f898a249f2f27bfeb6359ff353d978ab49b0ff89d086a347de80c70: Status 404 returned error can't find the container with id 7e8f1a4d3f898a249f2f27bfeb6359ff353d978ab49b0ff89d086a347de80c70 Nov 03 01:44:49.690577 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:44:49.690523 4939 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 01:44:49.690698 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:44:49.690606 4939 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 01:44:49.690698 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:44:49.690636 4939 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 01:44:49.779031 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:44:49.777989 4939 generic.go:356] "Generic (PLEG): container finished" podID="4da21d0c-6130-49ec-8042-97eed34d06e0" containerID="7cba2617f729abfa5529b22dc4a9d78f3789eefe7799e5278a33a1359c479149" exitCode=0 Nov 03 01:44:49.779031 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:44:49.778140 4939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-7b85f9594f-vwbsl" event={"ID":"4da21d0c-6130-49ec-8042-97eed34d06e0","Type":"ContainerDied","Data":"7cba2617f729abfa5529b22dc4a9d78f3789eefe7799e5278a33a1359c479149"} Nov 03 01:44:49.783652 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:44:49.781325 4939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-7kxj4" event={"ID":"8acf90f6-6fff-4a69-b67d-c3ba365c6103","Type":"ContainerStarted","Data":"debf1c6bb3a1dba1c1280706ec175f3eb788199e0f349474d7ebebe0bc5eb4e7"} Nov 03 01:44:49.783652 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:44:49.781401 4939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-7kxj4" event={"ID":"8acf90f6-6fff-4a69-b67d-c3ba365c6103","Type":"ContainerStarted","Data":"7e8f1a4d3f898a249f2f27bfeb6359ff353d978ab49b0ff89d086a347de80c70"} Nov 03 01:44:49.801351 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:44:49.801296 4939 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-7kxj4" podStartSLOduration=1.8012783190000001 podStartE2EDuration="1.801278319s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 01:44:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 01:44:49.799285109 +0000 UTC m=+3838.063620065" watchObservedRunningTime="2026-11-03 01:44:49.801278319 +0000 UTC m=+3838.065613262" Nov 03 01:44:49.825694 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:44:49.825667 4939 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-7b85f9594f-vwbsl" Nov 03 01:44:49.921878 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:44:49.921789 4939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4da21d0c-6130-49ec-8042-97eed34d06e0-webhook-cert\" (UniqueName: \"kubernetes.io/secret/4da21d0c-6130-49ec-8042-97eed34d06e0-webhook-cert\") pod \"4da21d0c-6130-49ec-8042-97eed34d06e0\" (UID: \"4da21d0c-6130-49ec-8042-97eed34d06e0\") " Nov 03 01:44:49.921878 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:44:49.921867 4939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4da21d0c-6130-49ec-8042-97eed34d06e0-kube-api-access-cf7kl\" (UniqueName: \"kubernetes.io/projected/4da21d0c-6130-49ec-8042-97eed34d06e0-kube-api-access-cf7kl\") pod \"4da21d0c-6130-49ec-8042-97eed34d06e0\" (UID: \"4da21d0c-6130-49ec-8042-97eed34d06e0\") " Nov 03 01:44:49.922152 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:44:49.921919 4939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4da21d0c-6130-49ec-8042-97eed34d06e0-metrics-certs\" (UniqueName: \"kubernetes.io/secret/4da21d0c-6130-49ec-8042-97eed34d06e0-metrics-certs\") pod \"4da21d0c-6130-49ec-8042-97eed34d06e0\" (UID: \"4da21d0c-6130-49ec-8042-97eed34d06e0\") " Nov 03 01:44:49.924261 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:44:49.924229 4939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4da21d0c-6130-49ec-8042-97eed34d06e0-kube-api-access-cf7kl" pod "4da21d0c-6130-49ec-8042-97eed34d06e0" (UID: "4da21d0c-6130-49ec-8042-97eed34d06e0"). InnerVolumeSpecName "kube-api-access-cf7kl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 01:44:49.924407 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:44:49.924283 4939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4da21d0c-6130-49ec-8042-97eed34d06e0-webhook-cert" pod "4da21d0c-6130-49ec-8042-97eed34d06e0" (UID: "4da21d0c-6130-49ec-8042-97eed34d06e0"). InnerVolumeSpecName "webhook-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 01:44:49.924531 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:44:49.924510 4939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4da21d0c-6130-49ec-8042-97eed34d06e0-metrics-certs" pod "4da21d0c-6130-49ec-8042-97eed34d06e0" (UID: "4da21d0c-6130-49ec-8042-97eed34d06e0"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 01:44:50.023245 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:44:50.023211 4939 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cf7kl\" (UniqueName: \"kubernetes.io/projected/4da21d0c-6130-49ec-8042-97eed34d06e0-kube-api-access-cf7kl\") on node \"ip-10-0-140-107.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:44:50.023245 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:44:50.023242 4939 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4da21d0c-6130-49ec-8042-97eed34d06e0-metrics-certs\") on node \"ip-10-0-140-107.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:44:50.023245 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:44:50.023253 4939 reconciler_common.go:299] "Volume detached for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/4da21d0c-6130-49ec-8042-97eed34d06e0-webhook-cert\") on node \"ip-10-0-140-107.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:44:50.317328 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:44:50.317288 4939 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-7kxj4" Nov 03 01:44:50.318803 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:44:50.318773 4939 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-596dc9545f-7kxj4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.100:15021/healthz/ready\": dial tcp 10.132.1.100:15021: connect: connection refused" start-of-body= Nov 03 01:44:50.318945 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:44:50.318835 4939 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-7kxj4" podUID="8acf90f6-6fff-4a69-b67d-c3ba365c6103" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.100:15021/healthz/ready\": dial tcp 10.132.1.100:15021: connect: connection refused" Nov 03 01:44:50.788520 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:44:50.788486 4939 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-7b85f9594f-vwbsl" Nov 03 01:44:50.788520 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:44:50.788507 4939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-7b85f9594f-vwbsl" event={"ID":"4da21d0c-6130-49ec-8042-97eed34d06e0","Type":"ContainerDied","Data":"826e363c4d56a1ac2a5e927eebffcee95aee6254bb2c8d5706ffa0d37a9f4e00"} Nov 03 01:44:50.788787 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:44:50.788570 4939 scope.go:122] "RemoveContainer" containerID="7cba2617f729abfa5529b22dc4a9d78f3789eefe7799e5278a33a1359c479149" Nov 03 01:44:50.810260 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:44:50.810236 4939 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-7b85f9594f-vwbsl"] Nov 03 01:44:50.814254 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:44:50.814225 4939 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-controller-7b85f9594f-vwbsl"] Nov 03 01:44:51.317900 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:44:51.317861 4939 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-596dc9545f-7kxj4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.100:15021/healthz/ready\": dial tcp 10.132.1.100:15021: connect: connection refused" start-of-body= Nov 03 01:44:51.318318 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:44:51.317949 4939 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-7kxj4" podUID="8acf90f6-6fff-4a69-b67d-c3ba365c6103" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.100:15021/healthz/ready\": dial tcp 10.132.1.100:15021: connect: connection refused" Nov 03 01:44:51.873613 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:44:51.873578 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:44:51.877415 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:44:51.877383 4939 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4da21d0c-6130-49ec-8042-97eed34d06e0" path="/var/lib/kubelet/pods/4da21d0c-6130-49ec-8042-97eed34d06e0/volumes" Nov 03 01:44:51.987377 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:44:51.987341 4939 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-7b85f9594f-xg8nb"] Nov 03 01:44:51.987774 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:44:51.987760 4939 cpu_manager.go:414] "containerMap: removing container" podUID="4da21d0c-6130-49ec-8042-97eed34d06e0" containerName="manager" Nov 03 01:44:51.987824 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:44:51.987776 4939 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4da21d0c-6130-49ec-8042-97eed34d06e0" containerName="manager" Nov 03 01:44:51.987864 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:44:51.987849 4939 memory_manager.go:365] "RemoveStaleState removing state" podUID="4da21d0c-6130-49ec-8042-97eed34d06e0" containerName="manager" Nov 03 01:44:51.992335 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:44:51.992316 4939 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-7b85f9594f-xg8nb" Nov 03 01:44:51.998546 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:44:51.998487 4939 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"maas-parameters\"" Nov 03 01:44:52.000638 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:44:52.000615 4939 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-webhook-cert\"" Nov 03 01:44:52.000768 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:44:52.000714 4939 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-dockercfg-2xnh6\"" Nov 03 01:44:52.000940 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:44:52.000878 4939 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-metrics-tls\"" Nov 03 01:44:52.002152 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:44:52.002133 4939 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-7b85f9594f-xg8nb"] Nov 03 01:44:52.143260 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:44:52.143182 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/1be824b5-4745-41c9-856c-1dc0c4b531d4-webhook-cert\") pod \"maas-controller-7b85f9594f-xg8nb\" (UID: \"1be824b5-4745-41c9-856c-1dc0c4b531d4\") " pod="opendatahub/maas-controller-7b85f9594f-xg8nb" Nov 03 01:44:52.143260 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:44:52.143232 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fvfpq\" (UniqueName: \"kubernetes.io/projected/1be824b5-4745-41c9-856c-1dc0c4b531d4-kube-api-access-fvfpq\") pod \"maas-controller-7b85f9594f-xg8nb\" (UID: \"1be824b5-4745-41c9-856c-1dc0c4b531d4\") " pod="opendatahub/maas-controller-7b85f9594f-xg8nb" Nov 03 01:44:52.143460 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:44:52.143354 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1be824b5-4745-41c9-856c-1dc0c4b531d4-metrics-certs\") pod \"maas-controller-7b85f9594f-xg8nb\" (UID: \"1be824b5-4745-41c9-856c-1dc0c4b531d4\") " pod="opendatahub/maas-controller-7b85f9594f-xg8nb" Nov 03 01:44:52.244110 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:44:52.244073 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/1be824b5-4745-41c9-856c-1dc0c4b531d4-webhook-cert\") pod \"maas-controller-7b85f9594f-xg8nb\" (UID: \"1be824b5-4745-41c9-856c-1dc0c4b531d4\") " pod="opendatahub/maas-controller-7b85f9594f-xg8nb" Nov 03 01:44:52.244299 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:44:52.244118 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fvfpq\" (UniqueName: \"kubernetes.io/projected/1be824b5-4745-41c9-856c-1dc0c4b531d4-kube-api-access-fvfpq\") pod \"maas-controller-7b85f9594f-xg8nb\" (UID: \"1be824b5-4745-41c9-856c-1dc0c4b531d4\") " pod="opendatahub/maas-controller-7b85f9594f-xg8nb" Nov 03 01:44:52.244299 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:44:52.244178 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1be824b5-4745-41c9-856c-1dc0c4b531d4-metrics-certs\") pod \"maas-controller-7b85f9594f-xg8nb\" (UID: \"1be824b5-4745-41c9-856c-1dc0c4b531d4\") " pod="opendatahub/maas-controller-7b85f9594f-xg8nb" Nov 03 01:44:52.254934 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:44:52.250222 4939 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-webhook-cert\"" Nov 03 01:44:52.254934 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:44:52.250342 4939 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-metrics-tls\"" Nov 03 01:44:52.266936 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:44:52.265283 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1be824b5-4745-41c9-856c-1dc0c4b531d4-metrics-certs\") pod \"maas-controller-7b85f9594f-xg8nb\" (UID: \"1be824b5-4745-41c9-856c-1dc0c4b531d4\") " pod="opendatahub/maas-controller-7b85f9594f-xg8nb" Nov 03 01:44:52.270927 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:44:52.269415 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/1be824b5-4745-41c9-856c-1dc0c4b531d4-webhook-cert\") pod \"maas-controller-7b85f9594f-xg8nb\" (UID: \"1be824b5-4745-41c9-856c-1dc0c4b531d4\") " pod="opendatahub/maas-controller-7b85f9594f-xg8nb" Nov 03 01:44:52.273671 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:44:52.273624 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fvfpq\" (UniqueName: \"kubernetes.io/projected/1be824b5-4745-41c9-856c-1dc0c4b531d4-kube-api-access-fvfpq\") pod \"maas-controller-7b85f9594f-xg8nb\" (UID: \"1be824b5-4745-41c9-856c-1dc0c4b531d4\") " pod="opendatahub/maas-controller-7b85f9594f-xg8nb" Nov 03 01:44:52.309755 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:44:52.309736 4939 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-dockercfg-2xnh6\"" Nov 03 01:44:52.317942 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:44:52.317920 4939 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-7b85f9594f-xg8nb" Nov 03 01:44:52.318269 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:44:52.318198 4939 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-596dc9545f-7kxj4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.100:15021/healthz/ready\": dial tcp 10.132.1.100:15021: connect: connection refused" start-of-body= Nov 03 01:44:52.318330 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:44:52.318261 4939 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-7kxj4" podUID="8acf90f6-6fff-4a69-b67d-c3ba365c6103" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.100:15021/healthz/ready\": dial tcp 10.132.1.100:15021: connect: connection refused" Nov 03 01:44:52.608897 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:44:52.608873 4939 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-7b85f9594f-xg8nb"] Nov 03 01:44:52.611215 ip-10-0-140-107 kubenswrapper[4939]: W1103 01:44:52.611180 4939 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1be824b5_4745_41c9_856c_1dc0c4b531d4.slice/crio-83049db4e66d90da25226d811f7dc634a8ba01c51a512df6ebba37afd98b2d86 WatchSource:0}: Error finding container 83049db4e66d90da25226d811f7dc634a8ba01c51a512df6ebba37afd98b2d86: Status 404 returned error can't find the container with id 83049db4e66d90da25226d811f7dc634a8ba01c51a512df6ebba37afd98b2d86 Nov 03 01:44:52.799687 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:44:52.799650 4939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-7b85f9594f-xg8nb" event={"ID":"1be824b5-4745-41c9-856c-1dc0c4b531d4","Type":"ContainerStarted","Data":"83049db4e66d90da25226d811f7dc634a8ba01c51a512df6ebba37afd98b2d86"} Nov 03 01:44:53.102993 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:44:53.102962 4939 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"maas-parameters\"" Nov 03 01:44:53.317452 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:44:53.317415 4939 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-596dc9545f-7kxj4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.100:15021/healthz/ready\": dial tcp 10.132.1.100:15021: connect: connection refused" start-of-body= Nov 03 01:44:53.317622 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:44:53.317466 4939 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-7kxj4" podUID="8acf90f6-6fff-4a69-b67d-c3ba365c6103" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.100:15021/healthz/ready\": dial tcp 10.132.1.100:15021: connect: connection refused" Nov 03 01:44:53.806772 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:44:53.806738 4939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-7b85f9594f-xg8nb" event={"ID":"1be824b5-4745-41c9-856c-1dc0c4b531d4","Type":"ContainerStarted","Data":"c4c256cddcf21a4b7dedf70f10bc9c74af7ab961594439fab7ec94a0b73a3224"} Nov 03 01:44:53.807423 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:44:53.807406 4939 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-7b85f9594f-xg8nb" Nov 03 01:44:53.827713 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:44:53.827601 4939 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/maas-controller-7b85f9594f-xg8nb" podStartSLOduration=2.340269981 podStartE2EDuration="2.827585921s" totalImagesPullingTime="487.31594ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 01:44:51 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 01:44:53.826865165 +0000 UTC m=+3842.091200109" watchObservedRunningTime="2026-11-03 01:44:53.827585921 +0000 UTC m=+3842.091920867" Nov 03 01:44:54.318162 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:44:54.318124 4939 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-596dc9545f-7kxj4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.100:15021/healthz/ready\": dial tcp 10.132.1.100:15021: connect: connection refused" start-of-body= Nov 03 01:44:54.318393 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:44:54.318187 4939 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-7kxj4" podUID="8acf90f6-6fff-4a69-b67d-c3ba365c6103" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.100:15021/healthz/ready\": dial tcp 10.132.1.100:15021: connect: connection refused" Nov 03 01:44:55.318012 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:44:55.317974 4939 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-596dc9545f-7kxj4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.100:15021/healthz/ready\": dial tcp 10.132.1.100:15021: connect: connection refused" start-of-body= Nov 03 01:44:55.318432 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:44:55.318034 4939 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-7kxj4" podUID="8acf90f6-6fff-4a69-b67d-c3ba365c6103" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.100:15021/healthz/ready\": dial tcp 10.132.1.100:15021: connect: connection refused" Nov 03 01:44:56.318039 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:44:56.317991 4939 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-596dc9545f-7kxj4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.100:15021/healthz/ready\": dial tcp 10.132.1.100:15021: connect: connection refused" start-of-body= Nov 03 01:44:56.318450 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:44:56.318066 4939 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-7kxj4" podUID="8acf90f6-6fff-4a69-b67d-c3ba365c6103" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.100:15021/healthz/ready\": dial tcp 10.132.1.100:15021: connect: connection refused" Nov 03 01:44:57.878825 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:44:57.878788 4939 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-7kxj4" Nov 03 01:44:57.879695 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:44:57.879673 4939 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-7kxj4" Nov 03 01:44:57.880711 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:44:57.880691 4939 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-7kxj4" Nov 03 01:45:00.134627 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:45:00.134594 4939 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29894505-gr9dh"] Nov 03 01:45:00.139696 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:45:00.139677 4939 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894505-gr9dh" Nov 03 01:45:00.141156 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:45:00.141131 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k4qgz\" (UniqueName: \"kubernetes.io/projected/6c35a563-8547-465f-a9d9-1a45e5c68d09-kube-api-access-k4qgz\") pod \"maas-api-key-cleanup-29894505-gr9dh\" (UID: \"6c35a563-8547-465f-a9d9-1a45e5c68d09\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894505-gr9dh" Nov 03 01:45:00.141310 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:45:00.141290 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6c35a563-8547-465f-a9d9-1a45e5c68d09-maas-api-service-ca\") pod \"maas-api-key-cleanup-29894505-gr9dh\" (UID: \"6c35a563-8547-465f-a9d9-1a45e5c68d09\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894505-gr9dh" Nov 03 01:45:00.146267 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:45:00.146236 4939 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29894505-gr9dh"] Nov 03 01:45:00.146549 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:45:00.146523 4939 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 03 01:45:00.146549 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:45:00.146539 4939 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-swfcr\"" Nov 03 01:45:00.146754 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:45:00.146610 4939 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 03 01:45:00.241929 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:45:00.241876 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k4qgz\" (UniqueName: \"kubernetes.io/projected/6c35a563-8547-465f-a9d9-1a45e5c68d09-kube-api-access-k4qgz\") pod \"maas-api-key-cleanup-29894505-gr9dh\" (UID: \"6c35a563-8547-465f-a9d9-1a45e5c68d09\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894505-gr9dh" Nov 03 01:45:00.242114 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:45:00.241997 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6c35a563-8547-465f-a9d9-1a45e5c68d09-maas-api-service-ca\") pod \"maas-api-key-cleanup-29894505-gr9dh\" (UID: \"6c35a563-8547-465f-a9d9-1a45e5c68d09\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894505-gr9dh" Nov 03 01:45:00.242623 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:45:00.242601 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6c35a563-8547-465f-a9d9-1a45e5c68d09-maas-api-service-ca\") pod \"maas-api-key-cleanup-29894505-gr9dh\" (UID: \"6c35a563-8547-465f-a9d9-1a45e5c68d09\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894505-gr9dh" Nov 03 01:45:00.250607 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:45:00.250577 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-k4qgz\" (UniqueName: \"kubernetes.io/projected/6c35a563-8547-465f-a9d9-1a45e5c68d09-kube-api-access-k4qgz\") pod \"maas-api-key-cleanup-29894505-gr9dh\" (UID: \"6c35a563-8547-465f-a9d9-1a45e5c68d09\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894505-gr9dh" Nov 03 01:45:00.457358 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:45:00.457274 4939 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894505-gr9dh" Nov 03 01:45:00.730104 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:45:00.730079 4939 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29894505-gr9dh"] Nov 03 01:45:00.731919 ip-10-0-140-107 kubenswrapper[4939]: W1103 01:45:00.731882 4939 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6c35a563_8547_465f_a9d9_1a45e5c68d09.slice/crio-8b2c2e932bb181724c1c103b429f713d7df68da254e22c0df72ad3763d364fb7 WatchSource:0}: Error finding container 8b2c2e932bb181724c1c103b429f713d7df68da254e22c0df72ad3763d364fb7: Status 404 returned error can't find the container with id 8b2c2e932bb181724c1c103b429f713d7df68da254e22c0df72ad3763d364fb7 Nov 03 01:45:00.845285 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:45:00.845249 4939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894505-gr9dh" event={"ID":"6c35a563-8547-465f-a9d9-1a45e5c68d09","Type":"ContainerStarted","Data":"8b2c2e932bb181724c1c103b429f713d7df68da254e22c0df72ad3763d364fb7"} Nov 03 01:45:01.851533 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:45:01.851497 4939 generic.go:356] "Generic (PLEG): container finished" podID="6c35a563-8547-465f-a9d9-1a45e5c68d09" containerID="c1fb9677183685174046450147af118a3cef2fcec6ba44049de7280507743452" exitCode=60 Nov 03 01:45:01.852073 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:45:01.851545 4939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894505-gr9dh" event={"ID":"6c35a563-8547-465f-a9d9-1a45e5c68d09","Type":"ContainerDied","Data":"c1fb9677183685174046450147af118a3cef2fcec6ba44049de7280507743452"} Nov 03 01:45:01.852073 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:45:01.851893 4939 scope.go:122] "RemoveContainer" containerID="c1fb9677183685174046450147af118a3cef2fcec6ba44049de7280507743452" Nov 03 01:45:02.857727 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:45:02.857691 4939 generic.go:356] "Generic (PLEG): container finished" podID="6c35a563-8547-465f-a9d9-1a45e5c68d09" containerID="06b3c4aa18d507ff938fbeed9d70e9491b0bed6691f6d952e1ff7713e0c9b331" exitCode=60 Nov 03 01:45:02.858298 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:45:02.857773 4939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894505-gr9dh" event={"ID":"6c35a563-8547-465f-a9d9-1a45e5c68d09","Type":"ContainerDied","Data":"06b3c4aa18d507ff938fbeed9d70e9491b0bed6691f6d952e1ff7713e0c9b331"} Nov 03 01:45:02.858298 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:45:02.857835 4939 scope.go:122] "RemoveContainer" containerID="c1fb9677183685174046450147af118a3cef2fcec6ba44049de7280507743452" Nov 03 01:45:02.858298 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:45:02.858155 4939 scope.go:122] "RemoveContainer" containerID="06b3c4aa18d507ff938fbeed9d70e9491b0bed6691f6d952e1ff7713e0c9b331" Nov 03 01:45:02.858494 ip-10-0-140-107 kubenswrapper[4939]: E1103 01:45:02.858447 4939 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29894505-gr9dh_odh-ai-gateway-infra(6c35a563-8547-465f-a9d9-1a45e5c68d09)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894505-gr9dh" podUID="6c35a563-8547-465f-a9d9-1a45e5c68d09" Nov 03 01:45:03.864423 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:45:03.864392 4939 scope.go:122] "RemoveContainer" containerID="06b3c4aa18d507ff938fbeed9d70e9491b0bed6691f6d952e1ff7713e0c9b331" Nov 03 01:45:03.864835 ip-10-0-140-107 kubenswrapper[4939]: E1103 01:45:03.864765 4939 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29894505-gr9dh_odh-ai-gateway-infra(6c35a563-8547-465f-a9d9-1a45e5c68d09)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894505-gr9dh" podUID="6c35a563-8547-465f-a9d9-1a45e5c68d09" Nov 03 01:45:05.819250 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:45:05.819219 4939 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-7b85f9594f-xg8nb" Nov 03 01:45:13.913672 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:45:13.913640 4939 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-4pw7v"] Nov 03 01:45:14.861156 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:45:14.861119 4939 scope.go:122] "RemoveContainer" containerID="06b3c4aa18d507ff938fbeed9d70e9491b0bed6691f6d952e1ff7713e0c9b331" Nov 03 01:45:16.057153 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:45:16.057109 4939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894505-gr9dh" event={"ID":"6c35a563-8547-465f-a9d9-1a45e5c68d09","Type":"ContainerStarted","Data":"6be58f1b6fa43d68e7bfcceff720cdefe07784029f904ca5761be6d98e6ff67c"} Nov 03 01:45:16.104930 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:45:16.103931 4939 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894505-gr9dh" podStartSLOduration=16.103894453 podStartE2EDuration="16.103894453s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 01:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 01:45:16.09808536 +0000 UTC m=+3864.362420306" watchObservedRunningTime="2026-11-03 01:45:16.103894453 +0000 UTC m=+3864.368229403" Nov 03 01:45:17.064699 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:45:17.064661 4939 generic.go:356] "Generic (PLEG): container finished" podID="6c35a563-8547-465f-a9d9-1a45e5c68d09" containerID="6be58f1b6fa43d68e7bfcceff720cdefe07784029f904ca5761be6d98e6ff67c" exitCode=60 Nov 03 01:45:17.065237 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:45:17.064735 4939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894505-gr9dh" event={"ID":"6c35a563-8547-465f-a9d9-1a45e5c68d09","Type":"ContainerDied","Data":"6be58f1b6fa43d68e7bfcceff720cdefe07784029f904ca5761be6d98e6ff67c"} Nov 03 01:45:17.065237 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:45:17.064818 4939 scope.go:122] "RemoveContainer" containerID="06b3c4aa18d507ff938fbeed9d70e9491b0bed6691f6d952e1ff7713e0c9b331" Nov 03 01:45:17.065237 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:45:17.065197 4939 scope.go:122] "RemoveContainer" containerID="6be58f1b6fa43d68e7bfcceff720cdefe07784029f904ca5761be6d98e6ff67c" Nov 03 01:45:17.065476 ip-10-0-140-107 kubenswrapper[4939]: E1103 01:45:17.065453 4939 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-29894505-gr9dh_odh-ai-gateway-infra(6c35a563-8547-465f-a9d9-1a45e5c68d09)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894505-gr9dh" podUID="6c35a563-8547-465f-a9d9-1a45e5c68d09" Nov 03 01:45:17.094520 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:45:17.094495 4939 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29894505-gr9dh"] Nov 03 01:45:18.029980 ip-10-0-140-107 kubenswrapper[4939]: E1103 01:45:18.029941 4939 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-sthjh" containerName="extractor" Nov 03 01:45:18.161730 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:45:18.161684 4939 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f904bff9-hgm8v"] Nov 03 01:45:18.167678 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:45:18.167647 4939 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f904bff9-hgm8v" Nov 03 01:45:18.171807 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:45:18.171261 4939 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 01:45:18.175345 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:45:18.175322 4939 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f904bff9-hgm8v"] Nov 03 01:45:18.208863 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:45:18.208834 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9c7f9e2a-3774-4cf5-8997-6fbf6d8c614b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f904bff9-hgm8v\" (UID: \"9c7f9e2a-3774-4cf5-8997-6fbf6d8c614b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f904bff9-hgm8v" Nov 03 01:45:18.208995 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:45:18.208878 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9c7f9e2a-3774-4cf5-8997-6fbf6d8c614b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f904bff9-hgm8v\" (UID: \"9c7f9e2a-3774-4cf5-8997-6fbf6d8c614b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f904bff9-hgm8v" Nov 03 01:45:18.208995 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:45:18.208922 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zkpqn\" (UniqueName: \"kubernetes.io/projected/9c7f9e2a-3774-4cf5-8997-6fbf6d8c614b-kube-api-access-zkpqn\") pod \"maas-api-revoke-sub-f904bff9-hgm8v\" (UID: \"9c7f9e2a-3774-4cf5-8997-6fbf6d8c614b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f904bff9-hgm8v" Nov 03 01:45:18.223770 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:45:18.223750 4939 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894505-gr9dh" Nov 03 01:45:18.309883 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:45:18.309794 4939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6c35a563-8547-465f-a9d9-1a45e5c68d09-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6c35a563-8547-465f-a9d9-1a45e5c68d09-maas-api-service-ca\") pod \"6c35a563-8547-465f-a9d9-1a45e5c68d09\" (UID: \"6c35a563-8547-465f-a9d9-1a45e5c68d09\") " Nov 03 01:45:18.310109 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:45:18.310010 4939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6c35a563-8547-465f-a9d9-1a45e5c68d09-kube-api-access-k4qgz\" (UniqueName: \"kubernetes.io/projected/6c35a563-8547-465f-a9d9-1a45e5c68d09-kube-api-access-k4qgz\") pod \"6c35a563-8547-465f-a9d9-1a45e5c68d09\" (UID: \"6c35a563-8547-465f-a9d9-1a45e5c68d09\") " Nov 03 01:45:18.310251 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:45:18.310225 4939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6c35a563-8547-465f-a9d9-1a45e5c68d09-maas-api-service-ca" pod "6c35a563-8547-465f-a9d9-1a45e5c68d09" (UID: "6c35a563-8547-465f-a9d9-1a45e5c68d09"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 01:45:18.310350 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:45:18.310327 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9c7f9e2a-3774-4cf5-8997-6fbf6d8c614b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f904bff9-hgm8v\" (UID: \"9c7f9e2a-3774-4cf5-8997-6fbf6d8c614b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f904bff9-hgm8v" Nov 03 01:45:18.310420 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:45:18.310390 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9c7f9e2a-3774-4cf5-8997-6fbf6d8c614b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f904bff9-hgm8v\" (UID: \"9c7f9e2a-3774-4cf5-8997-6fbf6d8c614b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f904bff9-hgm8v" Nov 03 01:45:18.310481 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:45:18.310446 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zkpqn\" (UniqueName: \"kubernetes.io/projected/9c7f9e2a-3774-4cf5-8997-6fbf6d8c614b-kube-api-access-zkpqn\") pod \"maas-api-revoke-sub-f904bff9-hgm8v\" (UID: \"9c7f9e2a-3774-4cf5-8997-6fbf6d8c614b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f904bff9-hgm8v" Nov 03 01:45:18.310559 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:45:18.310537 4939 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6c35a563-8547-465f-a9d9-1a45e5c68d09-maas-api-service-ca\") on node \"ip-10-0-140-107.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:45:18.311045 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:45:18.311021 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9c7f9e2a-3774-4cf5-8997-6fbf6d8c614b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f904bff9-hgm8v\" (UID: \"9c7f9e2a-3774-4cf5-8997-6fbf6d8c614b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f904bff9-hgm8v" Nov 03 01:45:18.312213 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:45:18.312188 4939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6c35a563-8547-465f-a9d9-1a45e5c68d09-kube-api-access-k4qgz" pod "6c35a563-8547-465f-a9d9-1a45e5c68d09" (UID: "6c35a563-8547-465f-a9d9-1a45e5c68d09"). InnerVolumeSpecName "kube-api-access-k4qgz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 01:45:18.312772 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:45:18.312752 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9c7f9e2a-3774-4cf5-8997-6fbf6d8c614b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f904bff9-hgm8v\" (UID: \"9c7f9e2a-3774-4cf5-8997-6fbf6d8c614b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f904bff9-hgm8v" Nov 03 01:45:18.319566 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:45:18.319545 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zkpqn\" (UniqueName: \"kubernetes.io/projected/9c7f9e2a-3774-4cf5-8997-6fbf6d8c614b-kube-api-access-zkpqn\") pod \"maas-api-revoke-sub-f904bff9-hgm8v\" (UID: \"9c7f9e2a-3774-4cf5-8997-6fbf6d8c614b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f904bff9-hgm8v" Nov 03 01:45:18.411385 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:45:18.411325 4939 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k4qgz\" (UniqueName: \"kubernetes.io/projected/6c35a563-8547-465f-a9d9-1a45e5c68d09-kube-api-access-k4qgz\") on node \"ip-10-0-140-107.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:45:18.487264 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:45:18.487220 4939 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f904bff9-hgm8v" Nov 03 01:45:18.756606 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:45:18.756579 4939 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f904bff9-hgm8v"] Nov 03 01:45:18.758402 ip-10-0-140-107 kubenswrapper[4939]: W1103 01:45:18.758375 4939 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9c7f9e2a_3774_4cf5_8997_6fbf6d8c614b.slice/crio-9e1dc57b63a78b628d3a07cdee01adfc3526c0b411ecbf85760ce7e5941316e8 WatchSource:0}: Error finding container 9e1dc57b63a78b628d3a07cdee01adfc3526c0b411ecbf85760ce7e5941316e8: Status 404 returned error can't find the container with id 9e1dc57b63a78b628d3a07cdee01adfc3526c0b411ecbf85760ce7e5941316e8 Nov 03 01:45:19.078302 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:45:19.078209 4939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f904bff9-hgm8v" event={"ID":"9c7f9e2a-3774-4cf5-8997-6fbf6d8c614b","Type":"ContainerStarted","Data":"1ffc4d6a697b8762c02de05e54a73cc06d03b92cdc7003e36a809e83827fc20f"} Nov 03 01:45:19.078302 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:45:19.078267 4939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f904bff9-hgm8v" event={"ID":"9c7f9e2a-3774-4cf5-8997-6fbf6d8c614b","Type":"ContainerStarted","Data":"9e1dc57b63a78b628d3a07cdee01adfc3526c0b411ecbf85760ce7e5941316e8"} Nov 03 01:45:19.079577 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:45:19.079553 4939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894505-gr9dh" event={"ID":"6c35a563-8547-465f-a9d9-1a45e5c68d09","Type":"ContainerDied","Data":"8b2c2e932bb181724c1c103b429f713d7df68da254e22c0df72ad3763d364fb7"} Nov 03 01:45:19.079669 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:45:19.079577 4939 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894505-gr9dh" Nov 03 01:45:19.079669 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:45:19.079604 4939 scope.go:122] "RemoveContainer" containerID="6be58f1b6fa43d68e7bfcceff720cdefe07784029f904ca5761be6d98e6ff67c" Nov 03 01:45:19.097527 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:45:19.097480 4939 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f904bff9-hgm8v" podStartSLOduration=1.097467563 podStartE2EDuration="1.097467563s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 01:45:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 01:45:19.095889635 +0000 UTC m=+3867.360224579" watchObservedRunningTime="2026-11-03 01:45:19.097467563 +0000 UTC m=+3867.361802507" Nov 03 01:45:19.108482 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:45:19.108452 4939 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29894505-gr9dh"] Nov 03 01:45:19.114253 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:45:19.114232 4939 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29894505-gr9dh"] Nov 03 01:45:19.866386 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:45:19.866355 4939 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6c35a563-8547-465f-a9d9-1a45e5c68d09" path="/var/lib/kubelet/pods/6c35a563-8547-465f-a9d9-1a45e5c68d09/volumes" Nov 03 01:45:20.087509 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:45:20.087477 4939 generic.go:356] "Generic (PLEG): container finished" podID="9c7f9e2a-3774-4cf5-8997-6fbf6d8c614b" containerID="1ffc4d6a697b8762c02de05e54a73cc06d03b92cdc7003e36a809e83827fc20f" exitCode=0 Nov 03 01:45:20.087689 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:45:20.087550 4939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f904bff9-hgm8v" event={"ID":"9c7f9e2a-3774-4cf5-8997-6fbf6d8c614b","Type":"ContainerDied","Data":"1ffc4d6a697b8762c02de05e54a73cc06d03b92cdc7003e36a809e83827fc20f"} Nov 03 01:45:21.223985 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:45:21.223955 4939 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f904bff9-hgm8v" Nov 03 01:45:21.341003 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:45:21.340966 4939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9c7f9e2a-3774-4cf5-8997-6fbf6d8c614b-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9c7f9e2a-3774-4cf5-8997-6fbf6d8c614b-maas-api-xks-ca\") pod \"9c7f9e2a-3774-4cf5-8997-6fbf6d8c614b\" (UID: \"9c7f9e2a-3774-4cf5-8997-6fbf6d8c614b\") " Nov 03 01:45:21.341187 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:45:21.341072 4939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9c7f9e2a-3774-4cf5-8997-6fbf6d8c614b-kube-api-access-zkpqn\" (UniqueName: \"kubernetes.io/projected/9c7f9e2a-3774-4cf5-8997-6fbf6d8c614b-kube-api-access-zkpqn\") pod \"9c7f9e2a-3774-4cf5-8997-6fbf6d8c614b\" (UID: \"9c7f9e2a-3774-4cf5-8997-6fbf6d8c614b\") " Nov 03 01:45:21.341187 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:45:21.341179 4939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9c7f9e2a-3774-4cf5-8997-6fbf6d8c614b-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9c7f9e2a-3774-4cf5-8997-6fbf6d8c614b-maas-api-openshift-ca\") pod \"9c7f9e2a-3774-4cf5-8997-6fbf6d8c614b\" (UID: \"9c7f9e2a-3774-4cf5-8997-6fbf6d8c614b\") " Nov 03 01:45:21.341569 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:45:21.341539 4939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9c7f9e2a-3774-4cf5-8997-6fbf6d8c614b-maas-api-openshift-ca" pod "9c7f9e2a-3774-4cf5-8997-6fbf6d8c614b" (UID: "9c7f9e2a-3774-4cf5-8997-6fbf6d8c614b"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 01:45:21.343254 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:45:21.343197 4939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9c7f9e2a-3774-4cf5-8997-6fbf6d8c614b-maas-api-xks-ca" pod "9c7f9e2a-3774-4cf5-8997-6fbf6d8c614b" (UID: "9c7f9e2a-3774-4cf5-8997-6fbf6d8c614b"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 01:45:21.343344 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:45:21.343276 4939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9c7f9e2a-3774-4cf5-8997-6fbf6d8c614b-kube-api-access-zkpqn" pod "9c7f9e2a-3774-4cf5-8997-6fbf6d8c614b" (UID: "9c7f9e2a-3774-4cf5-8997-6fbf6d8c614b"). InnerVolumeSpecName "kube-api-access-zkpqn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 01:45:21.442343 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:45:21.442304 4939 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9c7f9e2a-3774-4cf5-8997-6fbf6d8c614b-maas-api-openshift-ca\") on node \"ip-10-0-140-107.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:45:21.442343 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:45:21.442336 4939 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9c7f9e2a-3774-4cf5-8997-6fbf6d8c614b-maas-api-xks-ca\") on node \"ip-10-0-140-107.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:45:21.442343 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:45:21.442347 4939 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zkpqn\" (UniqueName: \"kubernetes.io/projected/9c7f9e2a-3774-4cf5-8997-6fbf6d8c614b-kube-api-access-zkpqn\") on node \"ip-10-0-140-107.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:45:22.097718 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:45:22.097683 4939 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f904bff9-hgm8v" Nov 03 01:45:22.097718 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:45:22.097699 4939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f904bff9-hgm8v" event={"ID":"9c7f9e2a-3774-4cf5-8997-6fbf6d8c614b","Type":"ContainerDied","Data":"9e1dc57b63a78b628d3a07cdee01adfc3526c0b411ecbf85760ce7e5941316e8"} Nov 03 01:45:22.097959 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:45:22.097746 4939 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="9e1dc57b63a78b628d3a07cdee01adfc3526c0b411ecbf85760ce7e5941316e8" Nov 03 01:45:28.539965 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:45:28.539930 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:45:28.987170 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:45:28.987122 4939 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-4pw7v"] Nov 03 01:45:32.961941 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:45:32.961357 4939 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-4pw7v"] Nov 03 01:45:36.606112 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:45:36.606075 4939 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-4pw7v"] Nov 03 01:45:40.814719 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:45:40.814676 4939 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-zr995"] Nov 03 01:45:40.815195 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:45:40.815154 4939 cpu_manager.go:414] "containerMap: removing container" podUID="6c35a563-8547-465f-a9d9-1a45e5c68d09" containerName="cleanup" Nov 03 01:45:40.815195 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:45:40.815169 4939 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6c35a563-8547-465f-a9d9-1a45e5c68d09" containerName="cleanup" Nov 03 01:45:40.815195 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:45:40.815180 4939 cpu_manager.go:414] "containerMap: removing container" podUID="6c35a563-8547-465f-a9d9-1a45e5c68d09" containerName="cleanup" Nov 03 01:45:40.815195 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:45:40.815185 4939 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6c35a563-8547-465f-a9d9-1a45e5c68d09" containerName="cleanup" Nov 03 01:45:40.815195 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:45:40.815194 4939 cpu_manager.go:414] "containerMap: removing container" podUID="6c35a563-8547-465f-a9d9-1a45e5c68d09" containerName="cleanup" Nov 03 01:45:40.815195 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:45:40.815199 4939 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6c35a563-8547-465f-a9d9-1a45e5c68d09" containerName="cleanup" Nov 03 01:45:40.815422 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:45:40.815217 4939 cpu_manager.go:414] "containerMap: removing container" podUID="9c7f9e2a-3774-4cf5-8997-6fbf6d8c614b" containerName="revoke-keys" Nov 03 01:45:40.815422 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:45:40.815223 4939 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9c7f9e2a-3774-4cf5-8997-6fbf6d8c614b" containerName="revoke-keys" Nov 03 01:45:40.815422 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:45:40.815306 4939 memory_manager.go:365] "RemoveStaleState removing state" podUID="6c35a563-8547-465f-a9d9-1a45e5c68d09" containerName="cleanup" Nov 03 01:45:40.815422 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:45:40.815313 4939 memory_manager.go:365] "RemoveStaleState removing state" podUID="6c35a563-8547-465f-a9d9-1a45e5c68d09" containerName="cleanup" Nov 03 01:45:40.815422 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:45:40.815320 4939 memory_manager.go:365] "RemoveStaleState removing state" podUID="9c7f9e2a-3774-4cf5-8997-6fbf6d8c614b" containerName="revoke-keys" Nov 03 01:45:40.818579 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:45:40.818558 4939 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-zr995" Nov 03 01:45:40.823938 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:45:40.823890 4939 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 03 01:45:40.824247 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:45:40.824222 4939 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 01:45:40.824512 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:45:40.824484 4939 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 03 01:45:40.824512 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:45:40.824340 4939 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-swfcr\"" Nov 03 01:45:40.826656 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:45:40.826555 4939 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-zr995"] Nov 03 01:45:40.909305 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:45:40.909271 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8c61f48c-10e8-40c2-9105-375e8f7b1318-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f03629db-zr995\" (UID: \"8c61f48c-10e8-40c2-9105-375e8f7b1318\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-zr995" Nov 03 01:45:40.909485 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:45:40.909330 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t8mp9\" (UniqueName: \"kubernetes.io/projected/8c61f48c-10e8-40c2-9105-375e8f7b1318-kube-api-access-t8mp9\") pod \"maas-api-revoke-sub-f03629db-zr995\" (UID: \"8c61f48c-10e8-40c2-9105-375e8f7b1318\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-zr995" Nov 03 01:45:40.909485 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:45:40.909449 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8c61f48c-10e8-40c2-9105-375e8f7b1318-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f03629db-zr995\" (UID: \"8c61f48c-10e8-40c2-9105-375e8f7b1318\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-zr995" Nov 03 01:45:41.010441 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:45:41.010401 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8c61f48c-10e8-40c2-9105-375e8f7b1318-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f03629db-zr995\" (UID: \"8c61f48c-10e8-40c2-9105-375e8f7b1318\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-zr995" Nov 03 01:45:41.010613 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:45:41.010473 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t8mp9\" (UniqueName: \"kubernetes.io/projected/8c61f48c-10e8-40c2-9105-375e8f7b1318-kube-api-access-t8mp9\") pod \"maas-api-revoke-sub-f03629db-zr995\" (UID: \"8c61f48c-10e8-40c2-9105-375e8f7b1318\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-zr995" Nov 03 01:45:41.010613 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:45:41.010543 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8c61f48c-10e8-40c2-9105-375e8f7b1318-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f03629db-zr995\" (UID: \"8c61f48c-10e8-40c2-9105-375e8f7b1318\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-zr995" Nov 03 01:45:41.011217 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:45:41.011192 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8c61f48c-10e8-40c2-9105-375e8f7b1318-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f03629db-zr995\" (UID: \"8c61f48c-10e8-40c2-9105-375e8f7b1318\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-zr995" Nov 03 01:45:41.012982 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:45:41.012963 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8c61f48c-10e8-40c2-9105-375e8f7b1318-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f03629db-zr995\" (UID: \"8c61f48c-10e8-40c2-9105-375e8f7b1318\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-zr995" Nov 03 01:45:41.021709 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:45:41.021684 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-t8mp9\" (UniqueName: \"kubernetes.io/projected/8c61f48c-10e8-40c2-9105-375e8f7b1318-kube-api-access-t8mp9\") pod \"maas-api-revoke-sub-f03629db-zr995\" (UID: \"8c61f48c-10e8-40c2-9105-375e8f7b1318\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-zr995" Nov 03 01:45:41.133307 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:45:41.133268 4939 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-zr995" Nov 03 01:45:41.408369 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:45:41.408345 4939 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-zr995"] Nov 03 01:45:41.410301 ip-10-0-140-107 kubenswrapper[4939]: W1103 01:45:41.410267 4939 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8c61f48c_10e8_40c2_9105_375e8f7b1318.slice/crio-50be78ce29cdf0f99609e0aa1fc61e837c5ef0ad7085f15ebda3409a55e3b93f WatchSource:0}: Error finding container 50be78ce29cdf0f99609e0aa1fc61e837c5ef0ad7085f15ebda3409a55e3b93f: Status 404 returned error can't find the container with id 50be78ce29cdf0f99609e0aa1fc61e837c5ef0ad7085f15ebda3409a55e3b93f Nov 03 01:45:42.231536 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:45:42.231501 4939 generic.go:356] "Generic (PLEG): container finished" podID="8c61f48c-10e8-40c2-9105-375e8f7b1318" containerID="da8e79899a06ed26b0203c46c71c2d369e7e005781c0076c214faccfb16e2526" exitCode=0 Nov 03 01:45:42.231986 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:45:42.231592 4939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-zr995" event={"ID":"8c61f48c-10e8-40c2-9105-375e8f7b1318","Type":"ContainerDied","Data":"da8e79899a06ed26b0203c46c71c2d369e7e005781c0076c214faccfb16e2526"} Nov 03 01:45:42.231986 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:45:42.231660 4939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-zr995" event={"ID":"8c61f48c-10e8-40c2-9105-375e8f7b1318","Type":"ContainerStarted","Data":"50be78ce29cdf0f99609e0aa1fc61e837c5ef0ad7085f15ebda3409a55e3b93f"} Nov 03 01:45:42.252565 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:45:42.252531 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:45:42.360554 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:45:42.360525 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:45:43.368502 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:45:43.368470 4939 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-zr995" Nov 03 01:45:43.536970 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:45:43.536860 4939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8c61f48c-10e8-40c2-9105-375e8f7b1318-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8c61f48c-10e8-40c2-9105-375e8f7b1318-maas-api-openshift-ca\") pod \"8c61f48c-10e8-40c2-9105-375e8f7b1318\" (UID: \"8c61f48c-10e8-40c2-9105-375e8f7b1318\") " Nov 03 01:45:43.537146 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:45:43.537036 4939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8c61f48c-10e8-40c2-9105-375e8f7b1318-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8c61f48c-10e8-40c2-9105-375e8f7b1318-maas-api-xks-ca\") pod \"8c61f48c-10e8-40c2-9105-375e8f7b1318\" (UID: \"8c61f48c-10e8-40c2-9105-375e8f7b1318\") " Nov 03 01:45:43.537146 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:45:43.537080 4939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8c61f48c-10e8-40c2-9105-375e8f7b1318-kube-api-access-t8mp9\" (UniqueName: \"kubernetes.io/projected/8c61f48c-10e8-40c2-9105-375e8f7b1318-kube-api-access-t8mp9\") pod \"8c61f48c-10e8-40c2-9105-375e8f7b1318\" (UID: \"8c61f48c-10e8-40c2-9105-375e8f7b1318\") " Nov 03 01:45:43.537288 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:45:43.537250 4939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8c61f48c-10e8-40c2-9105-375e8f7b1318-maas-api-openshift-ca" pod "8c61f48c-10e8-40c2-9105-375e8f7b1318" (UID: "8c61f48c-10e8-40c2-9105-375e8f7b1318"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 01:45:43.537402 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:45:43.537386 4939 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8c61f48c-10e8-40c2-9105-375e8f7b1318-maas-api-openshift-ca\") on node \"ip-10-0-140-107.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:45:43.539229 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:45:43.539205 4939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8c61f48c-10e8-40c2-9105-375e8f7b1318-maas-api-xks-ca" pod "8c61f48c-10e8-40c2-9105-375e8f7b1318" (UID: "8c61f48c-10e8-40c2-9105-375e8f7b1318"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 01:45:43.539372 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:45:43.539352 4939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8c61f48c-10e8-40c2-9105-375e8f7b1318-kube-api-access-t8mp9" pod "8c61f48c-10e8-40c2-9105-375e8f7b1318" (UID: "8c61f48c-10e8-40c2-9105-375e8f7b1318"). InnerVolumeSpecName "kube-api-access-t8mp9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 01:45:43.638459 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:45:43.638423 4939 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8c61f48c-10e8-40c2-9105-375e8f7b1318-maas-api-xks-ca\") on node \"ip-10-0-140-107.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:45:43.638459 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:45:43.638454 4939 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-t8mp9\" (UniqueName: \"kubernetes.io/projected/8c61f48c-10e8-40c2-9105-375e8f7b1318-kube-api-access-t8mp9\") on node \"ip-10-0-140-107.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:45:44.243206 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:45:44.243116 4939 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-zr995" Nov 03 01:45:44.243206 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:45:44.243150 4939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-zr995" event={"ID":"8c61f48c-10e8-40c2-9105-375e8f7b1318","Type":"ContainerDied","Data":"50be78ce29cdf0f99609e0aa1fc61e837c5ef0ad7085f15ebda3409a55e3b93f"} Nov 03 01:45:44.243206 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:45:44.243201 4939 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="50be78ce29cdf0f99609e0aa1fc61e837c5ef0ad7085f15ebda3409a55e3b93f" Nov 03 01:45:46.169388 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:45:46.169352 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:45:48.034778 ip-10-0-140-107 kubenswrapper[4939]: E1103 01:45:48.034741 4939 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-sthjh" containerName="extractor" Nov 03 01:45:51.627041 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:45:51.627003 4939 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-4pw7v"] Nov 03 01:45:53.205896 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:45:53.205865 4939 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-n42s6_65d134ac-8a25-4c67-b6c7-cd752a098daf/console-operator/2.log" Nov 03 01:45:53.208426 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:45:53.208391 4939 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-n42s6_65d134ac-8a25-4c67-b6c7-cd752a098daf/console-operator/2.log" Nov 03 01:45:55.304933 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:45:55.276305 4939 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-4pw7v"] Nov 03 01:45:56.973158 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:45:56.973123 4939 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-mj52m"] Nov 03 01:45:56.973574 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:45:56.973560 4939 cpu_manager.go:414] "containerMap: removing container" podUID="8c61f48c-10e8-40c2-9105-375e8f7b1318" containerName="revoke-keys" Nov 03 01:45:56.973622 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:45:56.973574 4939 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8c61f48c-10e8-40c2-9105-375e8f7b1318" containerName="revoke-keys" Nov 03 01:45:56.973669 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:45:56.973633 4939 memory_manager.go:365] "RemoveStaleState removing state" podUID="8c61f48c-10e8-40c2-9105-375e8f7b1318" containerName="revoke-keys" Nov 03 01:45:56.973669 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:45:56.973641 4939 memory_manager.go:365] "RemoveStaleState removing state" podUID="6c35a563-8547-465f-a9d9-1a45e5c68d09" containerName="cleanup" Nov 03 01:45:56.978103 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:45:56.978076 4939 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-mj52m" Nov 03 01:45:56.983408 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:45:56.983203 4939 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 03 01:45:56.983408 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:45:56.983306 4939 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-swfcr\"" Nov 03 01:45:56.983408 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:45:56.983341 4939 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 01:45:56.983408 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:45:56.983379 4939 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 03 01:45:56.984788 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:45:56.984745 4939 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-mj52m"] Nov 03 01:45:57.039691 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:45:57.039656 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4e17db07-898d-45cb-a49a-66ff396d9964-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0165d7d2-mj52m\" (UID: \"4e17db07-898d-45cb-a49a-66ff396d9964\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-mj52m" Nov 03 01:45:57.039891 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:45:57.039805 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4e17db07-898d-45cb-a49a-66ff396d9964-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0165d7d2-mj52m\" (UID: \"4e17db07-898d-45cb-a49a-66ff396d9964\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-mj52m" Nov 03 01:45:57.039891 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:45:57.039872 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xd665\" (UniqueName: \"kubernetes.io/projected/4e17db07-898d-45cb-a49a-66ff396d9964-kube-api-access-xd665\") pod \"maas-api-revoke-sub-0165d7d2-mj52m\" (UID: \"4e17db07-898d-45cb-a49a-66ff396d9964\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-mj52m" Nov 03 01:45:57.140664 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:45:57.140627 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4e17db07-898d-45cb-a49a-66ff396d9964-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0165d7d2-mj52m\" (UID: \"4e17db07-898d-45cb-a49a-66ff396d9964\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-mj52m" Nov 03 01:45:57.140857 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:45:57.140680 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xd665\" (UniqueName: \"kubernetes.io/projected/4e17db07-898d-45cb-a49a-66ff396d9964-kube-api-access-xd665\") pod \"maas-api-revoke-sub-0165d7d2-mj52m\" (UID: \"4e17db07-898d-45cb-a49a-66ff396d9964\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-mj52m" Nov 03 01:45:57.140857 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:45:57.140743 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4e17db07-898d-45cb-a49a-66ff396d9964-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0165d7d2-mj52m\" (UID: \"4e17db07-898d-45cb-a49a-66ff396d9964\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-mj52m" Nov 03 01:45:57.141435 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:45:57.141404 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4e17db07-898d-45cb-a49a-66ff396d9964-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0165d7d2-mj52m\" (UID: \"4e17db07-898d-45cb-a49a-66ff396d9964\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-mj52m" Nov 03 01:45:57.142984 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:45:57.142957 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4e17db07-898d-45cb-a49a-66ff396d9964-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0165d7d2-mj52m\" (UID: \"4e17db07-898d-45cb-a49a-66ff396d9964\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-mj52m" Nov 03 01:45:57.148540 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:45:57.148517 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xd665\" (UniqueName: \"kubernetes.io/projected/4e17db07-898d-45cb-a49a-66ff396d9964-kube-api-access-xd665\") pod \"maas-api-revoke-sub-0165d7d2-mj52m\" (UID: \"4e17db07-898d-45cb-a49a-66ff396d9964\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-mj52m" Nov 03 01:45:57.296267 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:45:57.296197 4939 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-mj52m" Nov 03 01:45:57.568139 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:45:57.568110 4939 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-mj52m"] Nov 03 01:45:57.569635 ip-10-0-140-107 kubenswrapper[4939]: W1103 01:45:57.569610 4939 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4e17db07_898d_45cb_a49a_66ff396d9964.slice/crio-dd1987b8dc92502cdd14680dbc75fbdddaf2b0287f629eccee8e08da6598e4b6 WatchSource:0}: Error finding container dd1987b8dc92502cdd14680dbc75fbdddaf2b0287f629eccee8e08da6598e4b6: Status 404 returned error can't find the container with id dd1987b8dc92502cdd14680dbc75fbdddaf2b0287f629eccee8e08da6598e4b6 Nov 03 01:45:58.362765 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:45:58.362726 4939 generic.go:356] "Generic (PLEG): container finished" podID="4e17db07-898d-45cb-a49a-66ff396d9964" containerID="cfee89ea1230f87f6286bbdd7ee7fdeaf762a0e9b9603a799389748e5ec7871b" exitCode=0 Nov 03 01:45:58.363395 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:45:58.362803 4939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-mj52m" event={"ID":"4e17db07-898d-45cb-a49a-66ff396d9964","Type":"ContainerDied","Data":"cfee89ea1230f87f6286bbdd7ee7fdeaf762a0e9b9603a799389748e5ec7871b"} Nov 03 01:45:58.363395 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:45:58.362864 4939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-mj52m" event={"ID":"4e17db07-898d-45cb-a49a-66ff396d9964","Type":"ContainerStarted","Data":"dd1987b8dc92502cdd14680dbc75fbdddaf2b0287f629eccee8e08da6598e4b6"} Nov 03 01:45:58.486694 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:45:58.486663 4939 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-4pw7v"] Nov 03 01:45:59.505240 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:45:59.505216 4939 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-mj52m" Nov 03 01:45:59.564082 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:45:59.564055 4939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4e17db07-898d-45cb-a49a-66ff396d9964-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4e17db07-898d-45cb-a49a-66ff396d9964-maas-api-openshift-ca\") pod \"4e17db07-898d-45cb-a49a-66ff396d9964\" (UID: \"4e17db07-898d-45cb-a49a-66ff396d9964\") " Nov 03 01:45:59.564240 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:45:59.564091 4939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4e17db07-898d-45cb-a49a-66ff396d9964-kube-api-access-xd665\" (UniqueName: \"kubernetes.io/projected/4e17db07-898d-45cb-a49a-66ff396d9964-kube-api-access-xd665\") pod \"4e17db07-898d-45cb-a49a-66ff396d9964\" (UID: \"4e17db07-898d-45cb-a49a-66ff396d9964\") " Nov 03 01:45:59.564240 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:45:59.564195 4939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4e17db07-898d-45cb-a49a-66ff396d9964-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4e17db07-898d-45cb-a49a-66ff396d9964-maas-api-xks-ca\") pod \"4e17db07-898d-45cb-a49a-66ff396d9964\" (UID: \"4e17db07-898d-45cb-a49a-66ff396d9964\") " Nov 03 01:45:59.564495 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:45:59.564458 4939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4e17db07-898d-45cb-a49a-66ff396d9964-maas-api-openshift-ca" pod "4e17db07-898d-45cb-a49a-66ff396d9964" (UID: "4e17db07-898d-45cb-a49a-66ff396d9964"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 01:45:59.566273 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:45:59.566243 4939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4e17db07-898d-45cb-a49a-66ff396d9964-maas-api-xks-ca" pod "4e17db07-898d-45cb-a49a-66ff396d9964" (UID: "4e17db07-898d-45cb-a49a-66ff396d9964"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 01:45:59.566406 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:45:59.566386 4939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4e17db07-898d-45cb-a49a-66ff396d9964-kube-api-access-xd665" pod "4e17db07-898d-45cb-a49a-66ff396d9964" (UID: "4e17db07-898d-45cb-a49a-66ff396d9964"). InnerVolumeSpecName "kube-api-access-xd665". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 01:45:59.665644 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:45:59.665582 4939 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4e17db07-898d-45cb-a49a-66ff396d9964-maas-api-openshift-ca\") on node \"ip-10-0-140-107.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:45:59.665644 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:45:59.665612 4939 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xd665\" (UniqueName: \"kubernetes.io/projected/4e17db07-898d-45cb-a49a-66ff396d9964-kube-api-access-xd665\") on node \"ip-10-0-140-107.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:45:59.665644 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:45:59.665622 4939 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4e17db07-898d-45cb-a49a-66ff396d9964-maas-api-xks-ca\") on node \"ip-10-0-140-107.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:46:00.373874 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:46:00.373840 4939 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-mj52m" Nov 03 01:46:00.374069 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:46:00.373885 4939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-mj52m" event={"ID":"4e17db07-898d-45cb-a49a-66ff396d9964","Type":"ContainerDied","Data":"dd1987b8dc92502cdd14680dbc75fbdddaf2b0287f629eccee8e08da6598e4b6"} Nov 03 01:46:00.374069 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:46:00.373960 4939 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="dd1987b8dc92502cdd14680dbc75fbdddaf2b0287f629eccee8e08da6598e4b6" Nov 03 01:46:07.672482 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:46:07.672442 4939 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-tsx87"] Nov 03 01:46:07.672877 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:46:07.672856 4939 cpu_manager.go:414] "containerMap: removing container" podUID="4e17db07-898d-45cb-a49a-66ff396d9964" containerName="revoke-keys" Nov 03 01:46:07.672877 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:46:07.672868 4939 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4e17db07-898d-45cb-a49a-66ff396d9964" containerName="revoke-keys" Nov 03 01:46:07.672994 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:46:07.672971 4939 memory_manager.go:365] "RemoveStaleState removing state" podUID="4e17db07-898d-45cb-a49a-66ff396d9964" containerName="revoke-keys" Nov 03 01:46:07.677191 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:46:07.677167 4939 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-tsx87" Nov 03 01:46:07.681433 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:46:07.681376 4939 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 03 01:46:07.682535 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:46:07.682510 4939 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 03 01:46:07.685072 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:46:07.685050 4939 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-tsx87"] Nov 03 01:46:07.714163 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:46:07.714121 4939 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 01:46:07.714346 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:46:07.714186 4939 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-swfcr\"" Nov 03 01:46:07.742788 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:46:07.742760 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/929fd880-1454-43b6-ab02-f8d47fb0c483-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6b19338f-tsx87\" (UID: \"929fd880-1454-43b6-ab02-f8d47fb0c483\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-tsx87" Nov 03 01:46:07.742895 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:46:07.742795 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/929fd880-1454-43b6-ab02-f8d47fb0c483-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6b19338f-tsx87\" (UID: \"929fd880-1454-43b6-ab02-f8d47fb0c483\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-tsx87" Nov 03 01:46:07.742895 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:46:07.742883 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qgpds\" (UniqueName: \"kubernetes.io/projected/929fd880-1454-43b6-ab02-f8d47fb0c483-kube-api-access-qgpds\") pod \"maas-api-revoke-sub-6b19338f-tsx87\" (UID: \"929fd880-1454-43b6-ab02-f8d47fb0c483\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-tsx87" Nov 03 01:46:07.844311 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:46:07.844272 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/929fd880-1454-43b6-ab02-f8d47fb0c483-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6b19338f-tsx87\" (UID: \"929fd880-1454-43b6-ab02-f8d47fb0c483\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-tsx87" Nov 03 01:46:07.844311 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:46:07.844315 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/929fd880-1454-43b6-ab02-f8d47fb0c483-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6b19338f-tsx87\" (UID: \"929fd880-1454-43b6-ab02-f8d47fb0c483\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-tsx87" Nov 03 01:46:07.844566 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:46:07.844352 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qgpds\" (UniqueName: \"kubernetes.io/projected/929fd880-1454-43b6-ab02-f8d47fb0c483-kube-api-access-qgpds\") pod \"maas-api-revoke-sub-6b19338f-tsx87\" (UID: \"929fd880-1454-43b6-ab02-f8d47fb0c483\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-tsx87" Nov 03 01:46:07.845122 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:46:07.845095 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/929fd880-1454-43b6-ab02-f8d47fb0c483-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6b19338f-tsx87\" (UID: \"929fd880-1454-43b6-ab02-f8d47fb0c483\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-tsx87" Nov 03 01:46:07.846816 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:46:07.846787 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/929fd880-1454-43b6-ab02-f8d47fb0c483-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6b19338f-tsx87\" (UID: \"929fd880-1454-43b6-ab02-f8d47fb0c483\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-tsx87" Nov 03 01:46:07.852134 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:46:07.852112 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qgpds\" (UniqueName: \"kubernetes.io/projected/929fd880-1454-43b6-ab02-f8d47fb0c483-kube-api-access-qgpds\") pod \"maas-api-revoke-sub-6b19338f-tsx87\" (UID: \"929fd880-1454-43b6-ab02-f8d47fb0c483\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-tsx87" Nov 03 01:46:07.994335 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:46:07.994250 4939 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-tsx87" Nov 03 01:46:08.265981 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:46:08.265949 4939 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-tsx87"] Nov 03 01:46:08.268003 ip-10-0-140-107 kubenswrapper[4939]: W1103 01:46:08.267975 4939 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod929fd880_1454_43b6_ab02_f8d47fb0c483.slice/crio-b169577665f1ca0d941e13ce9da63c738859f46ff87e9e221beab4a64d93891d WatchSource:0}: Error finding container b169577665f1ca0d941e13ce9da63c738859f46ff87e9e221beab4a64d93891d: Status 404 returned error can't find the container with id b169577665f1ca0d941e13ce9da63c738859f46ff87e9e221beab4a64d93891d Nov 03 01:46:08.416441 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:46:08.416401 4939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-tsx87" event={"ID":"929fd880-1454-43b6-ab02-f8d47fb0c483","Type":"ContainerStarted","Data":"b169577665f1ca0d941e13ce9da63c738859f46ff87e9e221beab4a64d93891d"} Nov 03 01:46:09.422673 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:46:09.422632 4939 generic.go:356] "Generic (PLEG): container finished" podID="929fd880-1454-43b6-ab02-f8d47fb0c483" containerID="30db0d587ed90afcda1a386abc7be6d156738119a3526132b6b58c5dd0b4ad9c" exitCode=0 Nov 03 01:46:09.423085 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:46:09.422712 4939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-tsx87" event={"ID":"929fd880-1454-43b6-ab02-f8d47fb0c483","Type":"ContainerDied","Data":"30db0d587ed90afcda1a386abc7be6d156738119a3526132b6b58c5dd0b4ad9c"} Nov 03 01:46:10.564952 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:46:10.564928 4939 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-tsx87" Nov 03 01:46:10.673150 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:46:10.673118 4939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/929fd880-1454-43b6-ab02-f8d47fb0c483-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/929fd880-1454-43b6-ab02-f8d47fb0c483-maas-api-openshift-ca\") pod \"929fd880-1454-43b6-ab02-f8d47fb0c483\" (UID: \"929fd880-1454-43b6-ab02-f8d47fb0c483\") " Nov 03 01:46:10.673324 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:46:10.673259 4939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/929fd880-1454-43b6-ab02-f8d47fb0c483-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/929fd880-1454-43b6-ab02-f8d47fb0c483-maas-api-xks-ca\") pod \"929fd880-1454-43b6-ab02-f8d47fb0c483\" (UID: \"929fd880-1454-43b6-ab02-f8d47fb0c483\") " Nov 03 01:46:10.673375 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:46:10.673328 4939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/929fd880-1454-43b6-ab02-f8d47fb0c483-kube-api-access-qgpds\" (UniqueName: \"kubernetes.io/projected/929fd880-1454-43b6-ab02-f8d47fb0c483-kube-api-access-qgpds\") pod \"929fd880-1454-43b6-ab02-f8d47fb0c483\" (UID: \"929fd880-1454-43b6-ab02-f8d47fb0c483\") " Nov 03 01:46:10.673500 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:46:10.673474 4939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/929fd880-1454-43b6-ab02-f8d47fb0c483-maas-api-openshift-ca" pod "929fd880-1454-43b6-ab02-f8d47fb0c483" (UID: "929fd880-1454-43b6-ab02-f8d47fb0c483"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 01:46:10.673692 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:46:10.673670 4939 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/929fd880-1454-43b6-ab02-f8d47fb0c483-maas-api-openshift-ca\") on node \"ip-10-0-140-107.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:46:10.675549 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:46:10.675517 4939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/929fd880-1454-43b6-ab02-f8d47fb0c483-kube-api-access-qgpds" pod "929fd880-1454-43b6-ab02-f8d47fb0c483" (UID: "929fd880-1454-43b6-ab02-f8d47fb0c483"). InnerVolumeSpecName "kube-api-access-qgpds". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 01:46:10.675549 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:46:10.675539 4939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/929fd880-1454-43b6-ab02-f8d47fb0c483-maas-api-xks-ca" pod "929fd880-1454-43b6-ab02-f8d47fb0c483" (UID: "929fd880-1454-43b6-ab02-f8d47fb0c483"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 01:46:10.774306 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:46:10.774272 4939 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/929fd880-1454-43b6-ab02-f8d47fb0c483-maas-api-xks-ca\") on node \"ip-10-0-140-107.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:46:10.774306 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:46:10.774302 4939 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qgpds\" (UniqueName: \"kubernetes.io/projected/929fd880-1454-43b6-ab02-f8d47fb0c483-kube-api-access-qgpds\") on node \"ip-10-0-140-107.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:46:11.434360 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:46:11.434329 4939 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-tsx87" Nov 03 01:46:11.434360 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:46:11.434355 4939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-tsx87" event={"ID":"929fd880-1454-43b6-ab02-f8d47fb0c483","Type":"ContainerDied","Data":"b169577665f1ca0d941e13ce9da63c738859f46ff87e9e221beab4a64d93891d"} Nov 03 01:46:11.434606 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:46:11.434397 4939 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b169577665f1ca0d941e13ce9da63c738859f46ff87e9e221beab4a64d93891d" Nov 03 01:46:13.846205 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:46:13.846169 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:46:13.950154 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:46:13.950120 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:46:14.858877 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:46:14.858843 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:46:18.030084 ip-10-0-140-107 kubenswrapper[4939]: E1103 01:46:18.030039 4939 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-sthjh" containerName="extractor" Nov 03 01:46:18.603669 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:46:18.603632 4939 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-4pw7v"] Nov 03 01:46:19.036038 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:46:19.033408 4939 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/e2e-prio-pool-55b7e383-kserve-7968bccd9f-5w5g8"] Nov 03 01:46:19.133070 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:46:19.132183 4939 cpu_manager.go:414] "containerMap: removing container" podUID="929fd880-1454-43b6-ab02-f8d47fb0c483" containerName="revoke-keys" Nov 03 01:46:19.133070 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:46:19.132223 4939 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="929fd880-1454-43b6-ab02-f8d47fb0c483" containerName="revoke-keys" Nov 03 01:46:19.133070 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:46:19.132373 4939 memory_manager.go:365] "RemoveStaleState removing state" podUID="929fd880-1454-43b6-ab02-f8d47fb0c483" containerName="revoke-keys" Nov 03 01:46:19.141870 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:46:19.141509 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:46:19.155943 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:46:19.155520 4939 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-prio-pool-55b7e383-kserve-7968bccd9f-5w5g8"] Nov 03 01:46:19.155943 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:46:19.155640 4939 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-prio-pool-55b7e383-kserve-7968bccd9f-5w5g8" Nov 03 01:46:19.166419 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:46:19.166386 4939 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-prio-pool-55b7e383-kserve-self-signed-certs\"" Nov 03 01:46:19.290686 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:46:19.290577 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b0a9de9f-853a-4520-ad72-ad5078b906f9-home\") pod \"e2e-prio-pool-55b7e383-kserve-7968bccd9f-5w5g8\" (UID: \"b0a9de9f-853a-4520-ad72-ad5078b906f9\") " pod="llm/e2e-prio-pool-55b7e383-kserve-7968bccd9f-5w5g8" Nov 03 01:46:19.291042 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:46:19.291016 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k5tkn\" (UniqueName: \"kubernetes.io/projected/b0a9de9f-853a-4520-ad72-ad5078b906f9-kube-api-access-k5tkn\") pod \"e2e-prio-pool-55b7e383-kserve-7968bccd9f-5w5g8\" (UID: \"b0a9de9f-853a-4520-ad72-ad5078b906f9\") " pod="llm/e2e-prio-pool-55b7e383-kserve-7968bccd9f-5w5g8" Nov 03 01:46:19.291199 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:46:19.291181 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b0a9de9f-853a-4520-ad72-ad5078b906f9-tmp-dir\") pod \"e2e-prio-pool-55b7e383-kserve-7968bccd9f-5w5g8\" (UID: \"b0a9de9f-853a-4520-ad72-ad5078b906f9\") " pod="llm/e2e-prio-pool-55b7e383-kserve-7968bccd9f-5w5g8" Nov 03 01:46:19.291309 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:46:19.291294 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b0a9de9f-853a-4520-ad72-ad5078b906f9-model-cache\") pod \"e2e-prio-pool-55b7e383-kserve-7968bccd9f-5w5g8\" (UID: \"b0a9de9f-853a-4520-ad72-ad5078b906f9\") " pod="llm/e2e-prio-pool-55b7e383-kserve-7968bccd9f-5w5g8" Nov 03 01:46:19.291428 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:46:19.291414 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b0a9de9f-853a-4520-ad72-ad5078b906f9-dshm\") pod \"e2e-prio-pool-55b7e383-kserve-7968bccd9f-5w5g8\" (UID: \"b0a9de9f-853a-4520-ad72-ad5078b906f9\") " pod="llm/e2e-prio-pool-55b7e383-kserve-7968bccd9f-5w5g8" Nov 03 01:46:19.291563 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:46:19.291550 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b0a9de9f-853a-4520-ad72-ad5078b906f9-tls-certs\") pod \"e2e-prio-pool-55b7e383-kserve-7968bccd9f-5w5g8\" (UID: \"b0a9de9f-853a-4520-ad72-ad5078b906f9\") " pod="llm/e2e-prio-pool-55b7e383-kserve-7968bccd9f-5w5g8" Nov 03 01:46:19.351617 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:46:19.351584 4939 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/e2e-prio-pool-55b7e383-kserve-router-scheduler-57bb946f54-zt76n"] Nov 03 01:46:19.555993 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:46:19.555531 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b0a9de9f-853a-4520-ad72-ad5078b906f9-home\") pod \"e2e-prio-pool-55b7e383-kserve-7968bccd9f-5w5g8\" (UID: \"b0a9de9f-853a-4520-ad72-ad5078b906f9\") " pod="llm/e2e-prio-pool-55b7e383-kserve-7968bccd9f-5w5g8" Nov 03 01:46:19.555993 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:46:19.555610 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k5tkn\" (UniqueName: \"kubernetes.io/projected/b0a9de9f-853a-4520-ad72-ad5078b906f9-kube-api-access-k5tkn\") pod \"e2e-prio-pool-55b7e383-kserve-7968bccd9f-5w5g8\" (UID: \"b0a9de9f-853a-4520-ad72-ad5078b906f9\") " pod="llm/e2e-prio-pool-55b7e383-kserve-7968bccd9f-5w5g8" Nov 03 01:46:19.555993 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:46:19.555655 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b0a9de9f-853a-4520-ad72-ad5078b906f9-tmp-dir\") pod \"e2e-prio-pool-55b7e383-kserve-7968bccd9f-5w5g8\" (UID: \"b0a9de9f-853a-4520-ad72-ad5078b906f9\") " pod="llm/e2e-prio-pool-55b7e383-kserve-7968bccd9f-5w5g8" Nov 03 01:46:19.555993 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:46:19.555690 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b0a9de9f-853a-4520-ad72-ad5078b906f9-model-cache\") pod \"e2e-prio-pool-55b7e383-kserve-7968bccd9f-5w5g8\" (UID: \"b0a9de9f-853a-4520-ad72-ad5078b906f9\") " pod="llm/e2e-prio-pool-55b7e383-kserve-7968bccd9f-5w5g8" Nov 03 01:46:19.555993 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:46:19.555732 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b0a9de9f-853a-4520-ad72-ad5078b906f9-dshm\") pod \"e2e-prio-pool-55b7e383-kserve-7968bccd9f-5w5g8\" (UID: \"b0a9de9f-853a-4520-ad72-ad5078b906f9\") " pod="llm/e2e-prio-pool-55b7e383-kserve-7968bccd9f-5w5g8" Nov 03 01:46:19.555993 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:46:19.555786 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b0a9de9f-853a-4520-ad72-ad5078b906f9-tls-certs\") pod \"e2e-prio-pool-55b7e383-kserve-7968bccd9f-5w5g8\" (UID: \"b0a9de9f-853a-4520-ad72-ad5078b906f9\") " pod="llm/e2e-prio-pool-55b7e383-kserve-7968bccd9f-5w5g8" Nov 03 01:46:19.575617 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:46:19.575574 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b0a9de9f-853a-4520-ad72-ad5078b906f9-tmp-dir\") pod \"e2e-prio-pool-55b7e383-kserve-7968bccd9f-5w5g8\" (UID: \"b0a9de9f-853a-4520-ad72-ad5078b906f9\") " pod="llm/e2e-prio-pool-55b7e383-kserve-7968bccd9f-5w5g8" Nov 03 01:46:19.576246 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:46:19.576219 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b0a9de9f-853a-4520-ad72-ad5078b906f9-home\") pod \"e2e-prio-pool-55b7e383-kserve-7968bccd9f-5w5g8\" (UID: \"b0a9de9f-853a-4520-ad72-ad5078b906f9\") " pod="llm/e2e-prio-pool-55b7e383-kserve-7968bccd9f-5w5g8" Nov 03 01:46:19.577097 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:46:19.577074 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b0a9de9f-853a-4520-ad72-ad5078b906f9-model-cache\") pod \"e2e-prio-pool-55b7e383-kserve-7968bccd9f-5w5g8\" (UID: \"b0a9de9f-853a-4520-ad72-ad5078b906f9\") " pod="llm/e2e-prio-pool-55b7e383-kserve-7968bccd9f-5w5g8" Nov 03 01:46:19.662931 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:46:19.656173 4939 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-prio-pool-55b7e383-kserve-router-scheduler-57bb946f54-zt76n"] Nov 03 01:46:19.663638 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:46:19.663607 4939 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-prio-pool-55b7e383-kserve-router-scheduler-57bb946f54-zt76n" Nov 03 01:46:19.687708 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:46:19.674421 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b0a9de9f-853a-4520-ad72-ad5078b906f9-dshm\") pod \"e2e-prio-pool-55b7e383-kserve-7968bccd9f-5w5g8\" (UID: \"b0a9de9f-853a-4520-ad72-ad5078b906f9\") " pod="llm/e2e-prio-pool-55b7e383-kserve-7968bccd9f-5w5g8" Nov 03 01:46:19.693388 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:46:19.693220 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:46:19.715932 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:46:19.695013 4939 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-prio-pool-55b7e383-epp-sa-dockercfg-4dg6w\"" Nov 03 01:46:19.809050 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:46:19.790957 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ae5bd350-1fd2-4e49-b19f-c2dbf502d74c-tls-certs\") pod \"e2e-prio-pool-55b7e383-kserve-router-scheduler-57bb946f54-zt76n\" (UID: \"ae5bd350-1fd2-4e49-b19f-c2dbf502d74c\") " pod="llm/e2e-prio-pool-55b7e383-kserve-router-scheduler-57bb946f54-zt76n" Nov 03 01:46:19.809406 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:46:19.806596 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b0a9de9f-853a-4520-ad72-ad5078b906f9-tls-certs\") pod \"e2e-prio-pool-55b7e383-kserve-7968bccd9f-5w5g8\" (UID: \"b0a9de9f-853a-4520-ad72-ad5078b906f9\") " pod="llm/e2e-prio-pool-55b7e383-kserve-7968bccd9f-5w5g8" Nov 03 01:46:19.813027 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:46:19.812984 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m8ss2\" (UniqueName: \"kubernetes.io/projected/ae5bd350-1fd2-4e49-b19f-c2dbf502d74c-kube-api-access-m8ss2\") pod \"e2e-prio-pool-55b7e383-kserve-router-scheduler-57bb946f54-zt76n\" (UID: \"ae5bd350-1fd2-4e49-b19f-c2dbf502d74c\") " pod="llm/e2e-prio-pool-55b7e383-kserve-router-scheduler-57bb946f54-zt76n" Nov 03 01:46:19.813670 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:46:19.813647 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-k5tkn\" (UniqueName: \"kubernetes.io/projected/b0a9de9f-853a-4520-ad72-ad5078b906f9-kube-api-access-k5tkn\") pod \"e2e-prio-pool-55b7e383-kserve-7968bccd9f-5w5g8\" (UID: \"b0a9de9f-853a-4520-ad72-ad5078b906f9\") " pod="llm/e2e-prio-pool-55b7e383-kserve-7968bccd9f-5w5g8" Nov 03 01:46:19.856030 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:46:19.855982 4939 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-prio-pool-55b7e383-kserve-7968bccd9f-5w5g8" Nov 03 01:46:19.917962 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:46:19.914074 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ae5bd350-1fd2-4e49-b19f-c2dbf502d74c-tls-certs\") pod \"e2e-prio-pool-55b7e383-kserve-router-scheduler-57bb946f54-zt76n\" (UID: \"ae5bd350-1fd2-4e49-b19f-c2dbf502d74c\") " pod="llm/e2e-prio-pool-55b7e383-kserve-router-scheduler-57bb946f54-zt76n" Nov 03 01:46:19.917962 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:46:19.914151 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m8ss2\" (UniqueName: \"kubernetes.io/projected/ae5bd350-1fd2-4e49-b19f-c2dbf502d74c-kube-api-access-m8ss2\") pod \"e2e-prio-pool-55b7e383-kserve-router-scheduler-57bb946f54-zt76n\" (UID: \"ae5bd350-1fd2-4e49-b19f-c2dbf502d74c\") " pod="llm/e2e-prio-pool-55b7e383-kserve-router-scheduler-57bb946f54-zt76n" Nov 03 01:46:19.975571 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:46:19.975534 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ae5bd350-1fd2-4e49-b19f-c2dbf502d74c-tls-certs\") pod \"e2e-prio-pool-55b7e383-kserve-router-scheduler-57bb946f54-zt76n\" (UID: \"ae5bd350-1fd2-4e49-b19f-c2dbf502d74c\") " pod="llm/e2e-prio-pool-55b7e383-kserve-router-scheduler-57bb946f54-zt76n" Nov 03 01:46:20.142724 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:46:20.134770 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-m8ss2\" (UniqueName: \"kubernetes.io/projected/ae5bd350-1fd2-4e49-b19f-c2dbf502d74c-kube-api-access-m8ss2\") pod \"e2e-prio-pool-55b7e383-kserve-router-scheduler-57bb946f54-zt76n\" (UID: \"ae5bd350-1fd2-4e49-b19f-c2dbf502d74c\") " pod="llm/e2e-prio-pool-55b7e383-kserve-router-scheduler-57bb946f54-zt76n" Nov 03 01:46:20.179443 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:46:20.179114 4939 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-prio-pool-55b7e383-kserve-router-scheduler-57bb946f54-zt76n" Nov 03 01:46:20.878291 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:46:20.877561 4939 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-prio-pool-55b7e383-kserve-7968bccd9f-5w5g8"] Nov 03 01:46:21.072173 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:46:21.068743 4939 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-prio-pool-55b7e383-kserve-router-scheduler-57bb946f54-zt76n"] Nov 03 01:46:21.738444 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:46:21.738395 4939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-prio-pool-55b7e383-kserve-router-scheduler-57bb946f54-zt76n" event={"ID":"ae5bd350-1fd2-4e49-b19f-c2dbf502d74c","Type":"ContainerStarted","Data":"e661a41fbdb4dd2f11593b03ac62214f04f3392c4b15f0b8e665031a4c3e099e"} Nov 03 01:46:21.740519 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:46:21.740470 4939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-prio-pool-55b7e383-kserve-7968bccd9f-5w5g8" event={"ID":"b0a9de9f-853a-4520-ad72-ad5078b906f9","Type":"ContainerStarted","Data":"2230326c159275849b2b87f745186483cd45c63c8605740124f5909c850ba640"} Nov 03 01:46:21.740668 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:46:21.740526 4939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-prio-pool-55b7e383-kserve-7968bccd9f-5w5g8" event={"ID":"b0a9de9f-853a-4520-ad72-ad5078b906f9","Type":"ContainerStarted","Data":"fd71bf99afeb2d84ad9957bd9702c10ca805ee94055ffe1928af1f9b0873fc6c"} Nov 03 01:46:21.758347 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:46:21.758288 4939 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/e2e-prio-pool-55b7e383-kserve-7968bccd9f-5w5g8" podStartSLOduration=2.758272068 podStartE2EDuration="2.758272068s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 01:46:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 01:46:21.757007979 +0000 UTC m=+3930.021342922" watchObservedRunningTime="2026-11-03 01:46:21.758272068 +0000 UTC m=+3930.022607013" Nov 03 01:46:21.857281 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:46:21.857236 4939 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-prio-pool-55b7e383-kserve-7968bccd9f-5w5g8" Nov 03 01:46:22.040435 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:46:22.040344 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:46:24.793262 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:46:24.793213 4939 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-4pw7v"] Nov 03 01:46:25.792563 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:46:25.792515 4939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-prio-pool-55b7e383-kserve-router-scheduler-57bb946f54-zt76n" event={"ID":"ae5bd350-1fd2-4e49-b19f-c2dbf502d74c","Type":"ContainerStarted","Data":"f4eb13a358dcb33f4a2a4ae45b7653db60b733ed96c1848e39d23ff8e0d79012"} Nov 03 01:46:25.792788 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:46:25.792698 4939 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-prio-pool-55b7e383-kserve-router-scheduler-57bb946f54-zt76n" Nov 03 01:46:25.809581 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:46:25.809528 4939 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/e2e-prio-pool-55b7e383-kserve-router-scheduler-57bb946f54-zt76n" podStartSLOduration=2.767931256 podStartE2EDuration="6.809511118s" totalImagesPullingTime="4.041579862s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 01:46:19 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 01:46:25.808522998 +0000 UTC m=+3934.072857942" watchObservedRunningTime="2026-11-03 01:46:25.809511118 +0000 UTC m=+3934.073846060" Nov 03 01:46:29.857927 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:46:29.857885 4939 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-prio-pool-55b7e383-kserve-7968bccd9f-5w5g8" Nov 03 01:46:29.871438 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:46:29.871411 4939 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-prio-pool-55b7e383-kserve-7968bccd9f-5w5g8" Nov 03 01:46:30.828102 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:46:30.828072 4939 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-prio-pool-55b7e383-kserve-7968bccd9f-5w5g8" Nov 03 01:46:46.213100 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:46:46.213067 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:46:48.029814 ip-10-0-140-107 kubenswrapper[4939]: E1103 01:46:48.029779 4939 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-sthjh" containerName="extractor" Nov 03 01:46:56.799077 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:46:56.799040 4939 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-prio-pool-55b7e383-kserve-router-scheduler-57bb946f54-zt76n" Nov 03 01:47:02.325283 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:47:02.325244 4939 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-4pw7v"] Nov 03 01:47:16.269562 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:47:16.269528 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:47:18.030608 ip-10-0-140-107 kubenswrapper[4939]: E1103 01:47:18.030567 4939 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-sthjh" containerName="extractor" Nov 03 01:47:20.824623 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:47:20.824587 4939 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-4pw7v"] Nov 03 01:47:21.568154 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:47:21.568112 4939 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["llm/e2e-prio-pool-55b7e383-kserve-router-scheduler-57bb946f54-zt76n"] Nov 03 01:47:21.573466 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:47:21.573432 4939 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["llm/e2e-prio-pool-55b7e383-kserve-7968bccd9f-5w5g8"] Nov 03 01:47:21.621377 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:47:21.621344 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:47:21.735892 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:47:21.735824 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:47:25.393116 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:47:25.393079 4939 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-4pw7v"] Nov 03 01:47:26.817081 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:47:26.817039 4939 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/e2e-prio-pool-55b7e383-kserve-router-scheduler-57bb946f54-zt76n" podUID="ae5bd350-1fd2-4e49-b19f-c2dbf502d74c" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 03 01:47:36.591742 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:47:36.591622 4939 kuberuntime_container.go:898] "Killing container with a grace period" pod="llm/e2e-prio-pool-55b7e383-kserve-router-scheduler-57bb946f54-zt76n" podUID="ae5bd350-1fd2-4e49-b19f-c2dbf502d74c" containerName="main" containerID="cri-o://f4eb13a358dcb33f4a2a4ae45b7653db60b733ed96c1848e39d23ff8e0d79012" gracePeriod=45 Nov 03 01:47:36.597856 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:47:36.597803 4939 kuberuntime_container.go:898] "Killing container with a grace period" pod="llm/e2e-prio-pool-55b7e383-kserve-7968bccd9f-5w5g8" podUID="b0a9de9f-853a-4520-ad72-ad5078b906f9" containerName="main" containerID="cri-o://2230326c159275849b2b87f745186483cd45c63c8605740124f5909c850ba640" gracePeriod=45 Nov 03 01:47:36.799589 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:47:36.799546 4939 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/e2e-prio-pool-55b7e383-kserve-router-scheduler-57bb946f54-zt76n" podUID="ae5bd350-1fd2-4e49-b19f-c2dbf502d74c" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 03 01:47:36.854954 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:47:36.854887 4939 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-prio-pool-55b7e383-kserve-7968bccd9f-5w5g8" Nov 03 01:47:36.912597 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:47:36.912568 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:47:36.986379 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:47:36.986345 4939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b0a9de9f-853a-4520-ad72-ad5078b906f9-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b0a9de9f-853a-4520-ad72-ad5078b906f9-model-cache\") pod \"b0a9de9f-853a-4520-ad72-ad5078b906f9\" (UID: \"b0a9de9f-853a-4520-ad72-ad5078b906f9\") " Nov 03 01:47:36.986548 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:47:36.986401 4939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b0a9de9f-853a-4520-ad72-ad5078b906f9-kube-api-access-k5tkn\" (UniqueName: \"kubernetes.io/projected/b0a9de9f-853a-4520-ad72-ad5078b906f9-kube-api-access-k5tkn\") pod \"b0a9de9f-853a-4520-ad72-ad5078b906f9\" (UID: \"b0a9de9f-853a-4520-ad72-ad5078b906f9\") " Nov 03 01:47:36.986548 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:47:36.986499 4939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b0a9de9f-853a-4520-ad72-ad5078b906f9-dshm\" (UniqueName: \"kubernetes.io/empty-dir/b0a9de9f-853a-4520-ad72-ad5078b906f9-dshm\") pod \"b0a9de9f-853a-4520-ad72-ad5078b906f9\" (UID: \"b0a9de9f-853a-4520-ad72-ad5078b906f9\") " Nov 03 01:47:36.986681 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:47:36.986556 4939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b0a9de9f-853a-4520-ad72-ad5078b906f9-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b0a9de9f-853a-4520-ad72-ad5078b906f9-tmp-dir\") pod \"b0a9de9f-853a-4520-ad72-ad5078b906f9\" (UID: \"b0a9de9f-853a-4520-ad72-ad5078b906f9\") " Nov 03 01:47:36.986681 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:47:36.986597 4939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b0a9de9f-853a-4520-ad72-ad5078b906f9-home\" (UniqueName: \"kubernetes.io/empty-dir/b0a9de9f-853a-4520-ad72-ad5078b906f9-home\") pod \"b0a9de9f-853a-4520-ad72-ad5078b906f9\" (UID: \"b0a9de9f-853a-4520-ad72-ad5078b906f9\") " Nov 03 01:47:36.986681 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:47:36.986605 4939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b0a9de9f-853a-4520-ad72-ad5078b906f9-model-cache" pod "b0a9de9f-853a-4520-ad72-ad5078b906f9" (UID: "b0a9de9f-853a-4520-ad72-ad5078b906f9"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 01:47:36.986870 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:47:36.986689 4939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b0a9de9f-853a-4520-ad72-ad5078b906f9-tls-certs\" (UniqueName: \"kubernetes.io/secret/b0a9de9f-853a-4520-ad72-ad5078b906f9-tls-certs\") pod \"b0a9de9f-853a-4520-ad72-ad5078b906f9\" (UID: \"b0a9de9f-853a-4520-ad72-ad5078b906f9\") " Nov 03 01:47:36.986870 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:47:36.986846 4939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b0a9de9f-853a-4520-ad72-ad5078b906f9-tmp-dir" pod "b0a9de9f-853a-4520-ad72-ad5078b906f9" (UID: "b0a9de9f-853a-4520-ad72-ad5078b906f9"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 01:47:36.986870 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:47:36.986857 4939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b0a9de9f-853a-4520-ad72-ad5078b906f9-home" pod "b0a9de9f-853a-4520-ad72-ad5078b906f9" (UID: "b0a9de9f-853a-4520-ad72-ad5078b906f9"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 01:47:36.987151 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:47:36.987129 4939 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b0a9de9f-853a-4520-ad72-ad5078b906f9-tmp-dir\") on node \"ip-10-0-140-107.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:47:36.987151 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:47:36.987152 4939 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b0a9de9f-853a-4520-ad72-ad5078b906f9-home\") on node \"ip-10-0-140-107.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:47:36.987296 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:47:36.987161 4939 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b0a9de9f-853a-4520-ad72-ad5078b906f9-model-cache\") on node \"ip-10-0-140-107.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:47:36.988715 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:47:36.988682 4939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b0a9de9f-853a-4520-ad72-ad5078b906f9-dshm" pod "b0a9de9f-853a-4520-ad72-ad5078b906f9" (UID: "b0a9de9f-853a-4520-ad72-ad5078b906f9"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 01:47:36.988715 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:47:36.988700 4939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b0a9de9f-853a-4520-ad72-ad5078b906f9-tls-certs" pod "b0a9de9f-853a-4520-ad72-ad5078b906f9" (UID: "b0a9de9f-853a-4520-ad72-ad5078b906f9"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 01:47:36.988715 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:47:36.988706 4939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b0a9de9f-853a-4520-ad72-ad5078b906f9-kube-api-access-k5tkn" pod "b0a9de9f-853a-4520-ad72-ad5078b906f9" (UID: "b0a9de9f-853a-4520-ad72-ad5078b906f9"). InnerVolumeSpecName "kube-api-access-k5tkn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 01:47:37.088408 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:47:37.088375 4939 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b0a9de9f-853a-4520-ad72-ad5078b906f9-tls-certs\") on node \"ip-10-0-140-107.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:47:37.088408 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:47:37.088405 4939 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k5tkn\" (UniqueName: \"kubernetes.io/projected/b0a9de9f-853a-4520-ad72-ad5078b906f9-kube-api-access-k5tkn\") on node \"ip-10-0-140-107.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:47:37.088586 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:47:37.088422 4939 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b0a9de9f-853a-4520-ad72-ad5078b906f9-dshm\") on node \"ip-10-0-140-107.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:47:37.262521 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:47:37.262434 4939 generic.go:356] "Generic (PLEG): container finished" podID="b0a9de9f-853a-4520-ad72-ad5078b906f9" containerID="2230326c159275849b2b87f745186483cd45c63c8605740124f5909c850ba640" exitCode=0 Nov 03 01:47:37.262521 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:47:37.262481 4939 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-prio-pool-55b7e383-kserve-7968bccd9f-5w5g8" Nov 03 01:47:37.262765 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:47:37.262521 4939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-prio-pool-55b7e383-kserve-7968bccd9f-5w5g8" event={"ID":"b0a9de9f-853a-4520-ad72-ad5078b906f9","Type":"ContainerDied","Data":"2230326c159275849b2b87f745186483cd45c63c8605740124f5909c850ba640"} Nov 03 01:47:37.262765 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:47:37.262570 4939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-prio-pool-55b7e383-kserve-7968bccd9f-5w5g8" event={"ID":"b0a9de9f-853a-4520-ad72-ad5078b906f9","Type":"ContainerDied","Data":"fd71bf99afeb2d84ad9957bd9702c10ca805ee94055ffe1928af1f9b0873fc6c"} Nov 03 01:47:37.262765 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:47:37.262594 4939 scope.go:122] "RemoveContainer" containerID="2230326c159275849b2b87f745186483cd45c63c8605740124f5909c850ba640" Nov 03 01:47:37.273606 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:47:37.273587 4939 scope.go:122] "RemoveContainer" containerID="2230326c159275849b2b87f745186483cd45c63c8605740124f5909c850ba640" Nov 03 01:47:37.273974 ip-10-0-140-107 kubenswrapper[4939]: E1103 01:47:37.273950 4939 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2230326c159275849b2b87f745186483cd45c63c8605740124f5909c850ba640\": container with ID starting with 2230326c159275849b2b87f745186483cd45c63c8605740124f5909c850ba640 not found: ID does not exist" containerID="2230326c159275849b2b87f745186483cd45c63c8605740124f5909c850ba640" Nov 03 01:47:37.274082 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:47:37.273980 4939 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2230326c159275849b2b87f745186483cd45c63c8605740124f5909c850ba640"} err="failed to get container status \"2230326c159275849b2b87f745186483cd45c63c8605740124f5909c850ba640\": rpc error: code = NotFound desc = could not find container \"2230326c159275849b2b87f745186483cd45c63c8605740124f5909c850ba640\": container with ID starting with 2230326c159275849b2b87f745186483cd45c63c8605740124f5909c850ba640 not found: ID does not exist" Nov 03 01:47:37.299086 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:47:37.299054 4939 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["llm/e2e-prio-pool-55b7e383-kserve-7968bccd9f-5w5g8"] Nov 03 01:47:37.301013 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:47:37.300988 4939 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["llm/e2e-prio-pool-55b7e383-kserve-7968bccd9f-5w5g8"] Nov 03 01:47:37.818209 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:47:37.818169 4939 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/e2e-prio-pool-55b7e383-kserve-7968bccd9f-5w5g8" podUID="b0a9de9f-853a-4520-ad72-ad5078b906f9" containerName="main" probeResult="failure" output="Get \"https://10.132.1.107:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 01:47:37.866620 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:47:37.866582 4939 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b0a9de9f-853a-4520-ad72-ad5078b906f9" path="/var/lib/kubelet/pods/b0a9de9f-853a-4520-ad72-ad5078b906f9/volumes" Nov 03 01:47:38.413632 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:47:38.413571 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:47:41.205192 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:47:41.205155 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:47:41.384868 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:47:41.384831 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:47:42.405716 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:47:42.405672 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:47:42.525680 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:47:42.525648 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:47:42.654402 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:47:42.654366 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:47:43.305048 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:47:43.305011 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:47:44.095355 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:47:44.095322 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:47:44.305529 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:47:44.305494 4939 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-4pw7v"] Nov 03 01:47:44.783126 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:47:44.783094 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:47:45.410662 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:47:45.410633 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:47:46.075035 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:47:46.075004 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:47:46.259071 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:47:46.259038 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:47:46.697228 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:47:46.697196 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:47:46.799799 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:47:46.799760 4939 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/e2e-prio-pool-55b7e383-kserve-router-scheduler-57bb946f54-zt76n" podUID="ae5bd350-1fd2-4e49-b19f-c2dbf502d74c" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 03 01:47:46.800019 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:47:46.799884 4939 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-prio-pool-55b7e383-kserve-router-scheduler-57bb946f54-zt76n" Nov 03 01:47:47.380811 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:47:47.380778 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:47:47.984109 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:47:47.984075 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:47:48.047951 ip-10-0-140-107 kubenswrapper[4939]: E1103 01:47:48.047893 4939 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-sthjh" containerName="extractor" Nov 03 01:47:48.587061 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:47:48.587023 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:47:49.286600 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:47:49.286568 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:47:49.893604 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:47:49.893572 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:47:50.019997 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:47:50.019959 4939 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-14ca4876-fvwl9"] Nov 03 01:47:50.022193 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:47:50.022164 4939 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-14ca4876-fvwl9"] Nov 03 01:47:50.495555 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:47:50.495512 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:47:51.358746 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:47:51.358713 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:47:51.867219 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:47:51.867188 4939 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="26577b0e-2eca-45e7-84cc-8aab89dcd232" path="/var/lib/kubelet/pods/26577b0e-2eca-45e7-84cc-8aab89dcd232/volumes" Nov 03 01:47:51.996327 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:47:51.996290 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:47:52.584382 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:47:52.584344 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:47:53.199580 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:47:53.199547 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:47:53.791937 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:47:53.791889 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:47:54.382804 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:47:54.382770 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:47:54.988173 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:47:54.988140 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:47:55.592229 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:47:55.592190 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:47:56.191111 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:47:56.191075 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:47:56.298982 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:47:56.298933 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:47:56.786455 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:47:56.786422 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:47:56.799693 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:47:56.799651 4939 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/e2e-prio-pool-55b7e383-kserve-router-scheduler-57bb946f54-zt76n" podUID="ae5bd350-1fd2-4e49-b19f-c2dbf502d74c" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 03 01:47:56.906029 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:47:56.905989 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:47:57.015087 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:47:57.015028 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:47:57.792217 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:47:57.792130 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:47:58.394719 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:47:58.394685 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:47:59.096500 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:47:59.096461 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:47:59.694641 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:47:59.694588 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:48:00.019522 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:48:00.019421 4939 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cace40c1-s2475"] Nov 03 01:48:00.022012 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:48:00.021984 4939 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cace40c1-s2475"] Nov 03 01:48:00.294551 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:48:00.294475 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:48:00.614806 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:48:00.614773 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:48:00.998802 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:48:00.998717 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:48:01.690261 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:48:01.690220 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:48:01.867775 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:48:01.867742 4939 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7f05b5e8-35b2-4f88-b6f2-ff4b645a34c5" path="/var/lib/kubelet/pods/7f05b5e8-35b2-4f88-b6f2-ff4b645a34c5/volumes" Nov 03 01:48:02.409122 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:48:02.409087 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:48:02.414224 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:48:02.414188 4939 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-4pw7v"] Nov 03 01:48:03.194094 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:48:03.194060 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:48:03.891458 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:48:03.891423 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:48:04.503635 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:48:04.503601 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:48:05.206014 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:48:05.205979 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:48:05.797474 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:48:05.797439 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:48:06.689681 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:48:06.689645 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:48:06.862856 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:48:06.862770 4939 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-prio-pool-55b7e383-kserve-router-scheduler-57bb946f54-zt76n" Nov 03 01:48:06.990386 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:48:06.990352 4939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ae5bd350-1fd2-4e49-b19f-c2dbf502d74c-kube-api-access-m8ss2\" (UniqueName: \"kubernetes.io/projected/ae5bd350-1fd2-4e49-b19f-c2dbf502d74c-kube-api-access-m8ss2\") pod \"ae5bd350-1fd2-4e49-b19f-c2dbf502d74c\" (UID: \"ae5bd350-1fd2-4e49-b19f-c2dbf502d74c\") " Nov 03 01:48:06.990386 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:48:06.990390 4939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ae5bd350-1fd2-4e49-b19f-c2dbf502d74c-tls-certs\" (UniqueName: \"kubernetes.io/secret/ae5bd350-1fd2-4e49-b19f-c2dbf502d74c-tls-certs\") pod \"ae5bd350-1fd2-4e49-b19f-c2dbf502d74c\" (UID: \"ae5bd350-1fd2-4e49-b19f-c2dbf502d74c\") " Nov 03 01:48:06.992570 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:48:06.992537 4939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ae5bd350-1fd2-4e49-b19f-c2dbf502d74c-kube-api-access-m8ss2" pod "ae5bd350-1fd2-4e49-b19f-c2dbf502d74c" (UID: "ae5bd350-1fd2-4e49-b19f-c2dbf502d74c"). InnerVolumeSpecName "kube-api-access-m8ss2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 01:48:06.992694 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:48:06.992667 4939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ae5bd350-1fd2-4e49-b19f-c2dbf502d74c-tls-certs" pod "ae5bd350-1fd2-4e49-b19f-c2dbf502d74c" (UID: "ae5bd350-1fd2-4e49-b19f-c2dbf502d74c"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 01:48:07.092102 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:48:07.092062 4939 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m8ss2\" (UniqueName: \"kubernetes.io/projected/ae5bd350-1fd2-4e49-b19f-c2dbf502d74c-kube-api-access-m8ss2\") on node \"ip-10-0-140-107.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:48:07.092102 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:48:07.092096 4939 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ae5bd350-1fd2-4e49-b19f-c2dbf502d74c-tls-certs\") on node \"ip-10-0-140-107.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:48:07.424636 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:48:07.424595 4939 generic.go:356] "Generic (PLEG): container finished" podID="ae5bd350-1fd2-4e49-b19f-c2dbf502d74c" containerID="f4eb13a358dcb33f4a2a4ae45b7653db60b733ed96c1848e39d23ff8e0d79012" exitCode=0 Nov 03 01:48:07.425011 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:48:07.424983 4939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-prio-pool-55b7e383-kserve-router-scheduler-57bb946f54-zt76n" event={"ID":"ae5bd350-1fd2-4e49-b19f-c2dbf502d74c","Type":"ContainerDied","Data":"f4eb13a358dcb33f4a2a4ae45b7653db60b733ed96c1848e39d23ff8e0d79012"} Nov 03 01:48:07.425182 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:48:07.425152 4939 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-prio-pool-55b7e383-kserve-router-scheduler-57bb946f54-zt76n" Nov 03 01:48:07.425324 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:48:07.425221 4939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-prio-pool-55b7e383-kserve-router-scheduler-57bb946f54-zt76n" event={"ID":"ae5bd350-1fd2-4e49-b19f-c2dbf502d74c","Type":"ContainerDied","Data":"e661a41fbdb4dd2f11593b03ac62214f04f3392c4b15f0b8e665031a4c3e099e"} Nov 03 01:48:07.425324 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:48:07.425272 4939 scope.go:122] "RemoveContainer" containerID="f4eb13a358dcb33f4a2a4ae45b7653db60b733ed96c1848e39d23ff8e0d79012" Nov 03 01:48:07.435597 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:48:07.435580 4939 scope.go:122] "RemoveContainer" containerID="f4eb13a358dcb33f4a2a4ae45b7653db60b733ed96c1848e39d23ff8e0d79012" Nov 03 01:48:07.435867 ip-10-0-140-107 kubenswrapper[4939]: E1103 01:48:07.435846 4939 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f4eb13a358dcb33f4a2a4ae45b7653db60b733ed96c1848e39d23ff8e0d79012\": container with ID starting with f4eb13a358dcb33f4a2a4ae45b7653db60b733ed96c1848e39d23ff8e0d79012 not found: ID does not exist" containerID="f4eb13a358dcb33f4a2a4ae45b7653db60b733ed96c1848e39d23ff8e0d79012" Nov 03 01:48:07.435965 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:48:07.435874 4939 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f4eb13a358dcb33f4a2a4ae45b7653db60b733ed96c1848e39d23ff8e0d79012"} err="failed to get container status \"f4eb13a358dcb33f4a2a4ae45b7653db60b733ed96c1848e39d23ff8e0d79012\": rpc error: code = NotFound desc = could not find container \"f4eb13a358dcb33f4a2a4ae45b7653db60b733ed96c1848e39d23ff8e0d79012\": container with ID starting with f4eb13a358dcb33f4a2a4ae45b7653db60b733ed96c1848e39d23ff8e0d79012 not found: ID does not exist" Nov 03 01:48:07.449527 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:48:07.449503 4939 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["llm/e2e-prio-pool-55b7e383-kserve-router-scheduler-57bb946f54-zt76n"] Nov 03 01:48:07.453026 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:48:07.453004 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:48:07.453633 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:48:07.453613 4939 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["llm/e2e-prio-pool-55b7e383-kserve-router-scheduler-57bb946f54-zt76n"] Nov 03 01:48:07.798531 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:48:07.798421 4939 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/e2e-prio-pool-55b7e383-kserve-router-scheduler-57bb946f54-zt76n" podUID="ae5bd350-1fd2-4e49-b19f-c2dbf502d74c" containerName="main" probeResult="failure" output="timeout: failed to connect service \"10.132.1.108:9003\" within 1s: context deadline exceeded" Nov 03 01:48:07.798803 ip-10-0-140-107 kubenswrapper[4939]: W1103 01:48:07.798565 4939 logging.go:55] [core] [Channel #20514 SubChannel #20515]grpc: addrConn.createTransport failed to connect to {Addr: "10.132.1.108:9003", ServerName: "10.132.1.108:9003", BalancerAttributes: {"<%!p(pickfirstleaf.managedByPickfirstKeyType={})>": "<%!p(bool=true)>" }}. Err: connection error: desc = "transport: Error while dialing: dial tcp 10.132.1.108:9003: operation was canceled" Nov 03 01:48:07.877642 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:48:07.877600 4939 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ae5bd350-1fd2-4e49-b19f-c2dbf502d74c" path="/var/lib/kubelet/pods/ae5bd350-1fd2-4e49-b19f-c2dbf502d74c/volumes" Nov 03 01:48:08.086166 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:48:08.086132 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:48:08.850305 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:48:08.850271 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:48:09.497752 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:48:09.497716 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:48:10.104704 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:48:10.104670 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:48:10.793698 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:48:10.793667 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:48:11.605494 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:48:11.605462 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:48:12.297445 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:48:12.297405 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:48:12.897368 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:48:12.897328 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:48:13.594315 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:48:13.594281 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:48:14.200878 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:48:14.200843 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:48:14.790664 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:48:14.790633 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:48:15.398344 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:48:15.398311 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:48:15.993807 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:48:15.993772 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:48:16.697370 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:48:16.697334 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:48:17.284363 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:48:17.284322 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:48:17.880638 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:48:17.880604 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:48:18.030366 ip-10-0-140-107 kubenswrapper[4939]: E1103 01:48:18.030329 4939 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-sthjh" containerName="extractor" Nov 03 01:48:18.490178 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:48:18.490142 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:48:19.188343 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:48:19.188313 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:48:19.833459 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:48:19.833404 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:48:20.404122 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:48:20.404092 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:48:21.102720 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:48:21.102678 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:48:21.605864 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:48:21.605831 4939 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-4pw7v"] Nov 03 01:48:21.701077 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:48:21.701046 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:48:22.391113 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:48:22.391077 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:48:22.991518 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:48:22.991484 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:48:23.665929 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:48:23.665867 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:48:24.278078 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:48:24.278035 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:48:24.905804 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:48:24.905768 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:48:25.482353 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:48:25.482319 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:48:26.200673 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:48:26.200632 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:48:26.799747 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:48:26.799710 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:48:27.396289 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:48:27.396254 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:48:27.998362 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:48:27.998294 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:48:28.652809 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:48:28.652764 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:48:29.286669 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:48:29.286637 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:48:29.889510 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:48:29.889479 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:48:30.481755 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:48:30.481721 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:48:31.085181 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:48:31.085143 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:48:31.684558 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:48:31.684522 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:48:32.401267 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:48:32.401228 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:48:32.897107 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:48:32.897075 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:48:33.409036 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:48:33.409004 4939 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-89dc6dff-2ddf6"] Nov 03 01:48:33.409436 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:48:33.409393 4939 cpu_manager.go:414] "containerMap: removing container" podUID="b0a9de9f-853a-4520-ad72-ad5078b906f9" containerName="main" Nov 03 01:48:33.409436 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:48:33.409404 4939 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b0a9de9f-853a-4520-ad72-ad5078b906f9" containerName="main" Nov 03 01:48:33.409518 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:48:33.409442 4939 cpu_manager.go:414] "containerMap: removing container" podUID="ae5bd350-1fd2-4e49-b19f-c2dbf502d74c" containerName="main" Nov 03 01:48:33.409518 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:48:33.409448 4939 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ae5bd350-1fd2-4e49-b19f-c2dbf502d74c" containerName="main" Nov 03 01:48:33.409586 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:48:33.409522 4939 memory_manager.go:365] "RemoveStaleState removing state" podUID="ae5bd350-1fd2-4e49-b19f-c2dbf502d74c" containerName="main" Nov 03 01:48:33.409586 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:48:33.409532 4939 memory_manager.go:365] "RemoveStaleState removing state" podUID="b0a9de9f-853a-4520-ad72-ad5078b906f9" containerName="main" Nov 03 01:48:33.412544 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:48:33.412526 4939 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-89dc6dff-2ddf6" Nov 03 01:48:33.420240 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:48:33.419984 4939 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-89dc6dff-2ddf6"] Nov 03 01:48:33.420240 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:48:33.420224 4939 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 03 01:48:33.420365 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:48:33.420250 4939 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 03 01:48:33.420365 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:48:33.420312 4939 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 01:48:33.420438 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:48:33.420411 4939 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-swfcr\"" Nov 03 01:48:33.569999 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:48:33.569966 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1d4b3cc0-b712-4e81-b29c-e456c60518d3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-89dc6dff-2ddf6\" (UID: \"1d4b3cc0-b712-4e81-b29c-e456c60518d3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-89dc6dff-2ddf6" Nov 03 01:48:33.570215 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:48:33.570095 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1d4b3cc0-b712-4e81-b29c-e456c60518d3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-89dc6dff-2ddf6\" (UID: \"1d4b3cc0-b712-4e81-b29c-e456c60518d3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-89dc6dff-2ddf6" Nov 03 01:48:33.570215 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:48:33.570164 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q6k68\" (UniqueName: \"kubernetes.io/projected/1d4b3cc0-b712-4e81-b29c-e456c60518d3-kube-api-access-q6k68\") pod \"maas-api-revoke-sub-89dc6dff-2ddf6\" (UID: \"1d4b3cc0-b712-4e81-b29c-e456c60518d3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-89dc6dff-2ddf6" Nov 03 01:48:33.576601 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:48:33.576576 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:48:33.671694 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:48:33.671602 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q6k68\" (UniqueName: \"kubernetes.io/projected/1d4b3cc0-b712-4e81-b29c-e456c60518d3-kube-api-access-q6k68\") pod \"maas-api-revoke-sub-89dc6dff-2ddf6\" (UID: \"1d4b3cc0-b712-4e81-b29c-e456c60518d3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-89dc6dff-2ddf6" Nov 03 01:48:33.671900 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:48:33.671695 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1d4b3cc0-b712-4e81-b29c-e456c60518d3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-89dc6dff-2ddf6\" (UID: \"1d4b3cc0-b712-4e81-b29c-e456c60518d3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-89dc6dff-2ddf6" Nov 03 01:48:33.671900 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:48:33.671808 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1d4b3cc0-b712-4e81-b29c-e456c60518d3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-89dc6dff-2ddf6\" (UID: \"1d4b3cc0-b712-4e81-b29c-e456c60518d3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-89dc6dff-2ddf6" Nov 03 01:48:33.672446 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:48:33.672425 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1d4b3cc0-b712-4e81-b29c-e456c60518d3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-89dc6dff-2ddf6\" (UID: \"1d4b3cc0-b712-4e81-b29c-e456c60518d3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-89dc6dff-2ddf6" Nov 03 01:48:33.674243 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:48:33.674221 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1d4b3cc0-b712-4e81-b29c-e456c60518d3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-89dc6dff-2ddf6\" (UID: \"1d4b3cc0-b712-4e81-b29c-e456c60518d3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-89dc6dff-2ddf6" Nov 03 01:48:33.679229 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:48:33.679205 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-q6k68\" (UniqueName: \"kubernetes.io/projected/1d4b3cc0-b712-4e81-b29c-e456c60518d3-kube-api-access-q6k68\") pod \"maas-api-revoke-sub-89dc6dff-2ddf6\" (UID: \"1d4b3cc0-b712-4e81-b29c-e456c60518d3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-89dc6dff-2ddf6" Nov 03 01:48:33.727259 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:48:33.727238 4939 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-89dc6dff-2ddf6" Nov 03 01:48:34.004299 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:48:34.004274 4939 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-89dc6dff-2ddf6"] Nov 03 01:48:34.006955 ip-10-0-140-107 kubenswrapper[4939]: W1103 01:48:34.006926 4939 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1d4b3cc0_b712_4e81_b29c_e456c60518d3.slice/crio-c11c51db42a0f3c5b3c03871853576441be3bbe66687b5a9f938612e09f13957 WatchSource:0}: Error finding container c11c51db42a0f3c5b3c03871853576441be3bbe66687b5a9f938612e09f13957: Status 404 returned error can't find the container with id c11c51db42a0f3c5b3c03871853576441be3bbe66687b5a9f938612e09f13957 Nov 03 01:48:34.203487 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:48:34.203435 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:48:34.570627 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:48:34.570535 4939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-89dc6dff-2ddf6" event={"ID":"1d4b3cc0-b712-4e81-b29c-e456c60518d3","Type":"ContainerStarted","Data":"836888c608e62eb5fb5332f98b84116f883673564b9c72ee2d6d73438dc404e6"} Nov 03 01:48:34.570627 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:48:34.570587 4939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-89dc6dff-2ddf6" event={"ID":"1d4b3cc0-b712-4e81-b29c-e456c60518d3","Type":"ContainerStarted","Data":"c11c51db42a0f3c5b3c03871853576441be3bbe66687b5a9f938612e09f13957"} Nov 03 01:48:34.589274 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:48:34.589226 4939 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-89dc6dff-2ddf6" podStartSLOduration=1.589213057 podStartE2EDuration="1.589213057s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 01:48:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 01:48:34.586968448 +0000 UTC m=+4062.851303394" watchObservedRunningTime="2026-11-03 01:48:34.589213057 +0000 UTC m=+4062.853548001" Nov 03 01:48:34.897209 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:48:34.897171 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:48:35.516493 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:48:35.516458 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:48:35.577397 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:48:35.577360 4939 generic.go:356] "Generic (PLEG): container finished" podID="1d4b3cc0-b712-4e81-b29c-e456c60518d3" containerID="836888c608e62eb5fb5332f98b84116f883673564b9c72ee2d6d73438dc404e6" exitCode=0 Nov 03 01:48:35.577838 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:48:35.577446 4939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-89dc6dff-2ddf6" event={"ID":"1d4b3cc0-b712-4e81-b29c-e456c60518d3","Type":"ContainerDied","Data":"836888c608e62eb5fb5332f98b84116f883673564b9c72ee2d6d73438dc404e6"} Nov 03 01:48:36.179092 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:48:36.179053 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:48:36.721629 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:48:36.721605 4939 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-89dc6dff-2ddf6" Nov 03 01:48:36.786732 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:48:36.786703 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:48:36.904377 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:48:36.904296 4939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1d4b3cc0-b712-4e81-b29c-e456c60518d3-kube-api-access-q6k68\" (UniqueName: \"kubernetes.io/projected/1d4b3cc0-b712-4e81-b29c-e456c60518d3-kube-api-access-q6k68\") pod \"1d4b3cc0-b712-4e81-b29c-e456c60518d3\" (UID: \"1d4b3cc0-b712-4e81-b29c-e456c60518d3\") " Nov 03 01:48:36.904562 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:48:36.904375 4939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1d4b3cc0-b712-4e81-b29c-e456c60518d3-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1d4b3cc0-b712-4e81-b29c-e456c60518d3-maas-api-xks-ca\") pod \"1d4b3cc0-b712-4e81-b29c-e456c60518d3\" (UID: \"1d4b3cc0-b712-4e81-b29c-e456c60518d3\") " Nov 03 01:48:36.904562 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:48:36.904478 4939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1d4b3cc0-b712-4e81-b29c-e456c60518d3-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1d4b3cc0-b712-4e81-b29c-e456c60518d3-maas-api-openshift-ca\") pod \"1d4b3cc0-b712-4e81-b29c-e456c60518d3\" (UID: \"1d4b3cc0-b712-4e81-b29c-e456c60518d3\") " Nov 03 01:48:36.904936 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:48:36.904874 4939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1d4b3cc0-b712-4e81-b29c-e456c60518d3-maas-api-openshift-ca" pod "1d4b3cc0-b712-4e81-b29c-e456c60518d3" (UID: "1d4b3cc0-b712-4e81-b29c-e456c60518d3"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 01:48:36.906525 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:48:36.906503 4939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1d4b3cc0-b712-4e81-b29c-e456c60518d3-maas-api-xks-ca" pod "1d4b3cc0-b712-4e81-b29c-e456c60518d3" (UID: "1d4b3cc0-b712-4e81-b29c-e456c60518d3"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 01:48:36.906681 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:48:36.906656 4939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1d4b3cc0-b712-4e81-b29c-e456c60518d3-kube-api-access-q6k68" pod "1d4b3cc0-b712-4e81-b29c-e456c60518d3" (UID: "1d4b3cc0-b712-4e81-b29c-e456c60518d3"). InnerVolumeSpecName "kube-api-access-q6k68". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 01:48:37.005825 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:48:37.005794 4939 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1d4b3cc0-b712-4e81-b29c-e456c60518d3-maas-api-openshift-ca\") on node \"ip-10-0-140-107.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:48:37.005825 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:48:37.005821 4939 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-q6k68\" (UniqueName: \"kubernetes.io/projected/1d4b3cc0-b712-4e81-b29c-e456c60518d3-kube-api-access-q6k68\") on node \"ip-10-0-140-107.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:48:37.006130 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:48:37.005832 4939 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1d4b3cc0-b712-4e81-b29c-e456c60518d3-maas-api-xks-ca\") on node \"ip-10-0-140-107.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:48:37.389972 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:48:37.389931 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:48:37.587661 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:48:37.587631 4939 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-89dc6dff-2ddf6" Nov 03 01:48:37.587849 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:48:37.587665 4939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-89dc6dff-2ddf6" event={"ID":"1d4b3cc0-b712-4e81-b29c-e456c60518d3","Type":"ContainerDied","Data":"c11c51db42a0f3c5b3c03871853576441be3bbe66687b5a9f938612e09f13957"} Nov 03 01:48:37.587849 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:48:37.587713 4939 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c11c51db42a0f3c5b3c03871853576441be3bbe66687b5a9f938612e09f13957" Nov 03 01:48:38.087738 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:48:38.087697 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:48:38.682882 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:48:38.682848 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:48:39.301428 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:48:39.301394 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:48:39.887716 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:48:39.887682 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:48:40.590656 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:48:40.590618 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:48:41.186434 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:48:41.186392 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:48:41.803516 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:48:41.803480 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:48:42.476753 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:48:42.476717 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:48:43.107273 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:48:43.107230 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:48:43.810150 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:48:43.810111 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:48:44.125180 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:48:44.125131 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:48:44.404685 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:48:44.404601 4939 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-4pw7v"] Nov 03 01:48:44.498531 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:48:44.498495 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:48:44.620781 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:48:44.620747 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:48:45.201983 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:48:45.201947 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:48:45.804328 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:48:45.804287 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:48:46.292537 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:48:46.292478 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:48:46.411123 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:48:46.411089 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:48:47.109419 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:48:47.109386 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:48:47.689075 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:48:47.689037 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:48:48.030265 ip-10-0-140-107 kubenswrapper[4939]: E1103 01:48:48.030184 4939 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-sthjh" containerName="extractor" Nov 03 01:48:48.294851 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:48:48.294771 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:48:48.882222 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:48:48.882188 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:48:49.154043 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:48:49.153947 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:48:49.492090 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:48:49.492002 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:48:50.081323 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:48:50.081286 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:48:50.192635 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:48:50.192599 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:48:50.680821 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:48:50.680787 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:48:51.715123 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:48:51.715087 4939 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-4pw7v"] Nov 03 01:48:56.309087 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:48:56.309049 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:48:56.428643 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:48:56.428609 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:48:59.609785 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:48:59.609749 4939 scope.go:122] "RemoveContainer" containerID="9b2e377fcbe44a1a579d909f66bc27739eea889e2f89654501b2d41fad4d2c4c" Nov 03 01:48:59.618710 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:48:59.618689 4939 scope.go:122] "RemoveContainer" containerID="140682075db05de362fbc3d45d65471e1f469ca7281b2567ed682bc6dac64f4d" Nov 03 01:49:18.030043 ip-10-0-140-107 kubenswrapper[4939]: E1103 01:49:18.030004 4939 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-sthjh" containerName="extractor" Nov 03 01:49:20.491829 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:49:20.491784 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:49:20.619832 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:49:20.619791 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:49:21.652518 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:49:21.652480 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:49:21.675067 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:49:21.675039 4939 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-515c8f-6bb9bf56f4-9l2jh"] Nov 03 01:49:21.675599 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:49:21.675572 4939 cpu_manager.go:414] "containerMap: removing container" podUID="1d4b3cc0-b712-4e81-b29c-e456c60518d3" containerName="revoke-keys" Nov 03 01:49:21.675767 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:49:21.675745 4939 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1d4b3cc0-b712-4e81-b29c-e456c60518d3" containerName="revoke-keys" Nov 03 01:49:21.675890 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:49:21.675878 4939 memory_manager.go:365] "RemoveStaleState removing state" podUID="1d4b3cc0-b712-4e81-b29c-e456c60518d3" containerName="revoke-keys" Nov 03 01:49:21.680416 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:49:21.680395 4939 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-515c8f-6bb9bf56f4-9l2jh" Nov 03 01:49:21.684790 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:49:21.684764 4939 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-515c8f-78658vth74"] Nov 03 01:49:21.690162 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:49:21.690140 4939 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-515c8f-6bb9bf56f4-9l2jh"] Nov 03 01:49:21.690285 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:49:21.690239 4939 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-515c8f-78658vth74" Nov 03 01:49:21.691323 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:49:21.691301 4939 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-metrics-tls\"" Nov 03 01:49:21.691660 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:49:21.691640 4939 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 03 01:49:21.691660 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:49:21.691655 4939 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-nfcxl\"" Nov 03 01:49:21.691798 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:49:21.691681 4939 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 03 01:49:21.691798 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:49:21.691732 4939 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-worker-single-main-515c8f-serving-cert\"" Nov 03 01:49:21.700032 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:49:21.700007 4939 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-515c8f-78658vth74"] Nov 03 01:49:21.756326 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:49:21.756298 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ac7998dc-9420-4207-95fc-cf5bb9ddc27b-openshift-service-ca\") pod \"maas-api-e2e-worker-single-main-515c8f-6bb9bf56f4-9l2jh\" (UID: \"ac7998dc-9420-4207-95fc-cf5bb9ddc27b\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-515c8f-6bb9bf56f4-9l2jh" Nov 03 01:49:21.756495 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:49:21.756339 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5fkln\" (UniqueName: \"kubernetes.io/projected/ac7998dc-9420-4207-95fc-cf5bb9ddc27b-kube-api-access-5fkln\") pod \"maas-api-e2e-worker-single-main-515c8f-6bb9bf56f4-9l2jh\" (UID: \"ac7998dc-9420-4207-95fc-cf5bb9ddc27b\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-515c8f-6bb9bf56f4-9l2jh" Nov 03 01:49:21.756495 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:49:21.756373 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ac7998dc-9420-4207-95fc-cf5bb9ddc27b-maas-api-tls\") pod \"maas-api-e2e-worker-single-main-515c8f-6bb9bf56f4-9l2jh\" (UID: \"ac7998dc-9420-4207-95fc-cf5bb9ddc27b\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-515c8f-6bb9bf56f4-9l2jh" Nov 03 01:49:21.756495 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:49:21.756468 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ac7998dc-9420-4207-95fc-cf5bb9ddc27b-metrics-certs\") pod \"maas-api-e2e-worker-single-main-515c8f-6bb9bf56f4-9l2jh\" (UID: \"ac7998dc-9420-4207-95fc-cf5bb9ddc27b\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-515c8f-6bb9bf56f4-9l2jh" Nov 03 01:49:21.756694 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:49:21.756534 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/472ad6a5-f51e-4c14-b2d7-bed131e23705-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-single-main-515c8f-78658vth74\" (UID: \"472ad6a5-f51e-4c14-b2d7-bed131e23705\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-515c8f-78658vth74" Nov 03 01:49:21.756694 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:49:21.756577 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p27jt\" (UniqueName: \"kubernetes.io/projected/472ad6a5-f51e-4c14-b2d7-bed131e23705-kube-api-access-p27jt\") pod \"payload-pre-processing-e2e-worker-single-main-515c8f-78658vth74\" (UID: \"472ad6a5-f51e-4c14-b2d7-bed131e23705\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-515c8f-78658vth74" Nov 03 01:49:21.779703 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:49:21.779649 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:49:21.790288 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:49:21.790257 4939 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-single-m-7c3b72c5-dockercfg-9s2fr\"" Nov 03 01:49:21.790869 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:49:21.790841 4939 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-single-main-515c8f\"" Nov 03 01:49:21.857587 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:49:21.857550 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ac7998dc-9420-4207-95fc-cf5bb9ddc27b-openshift-service-ca\") pod \"maas-api-e2e-worker-single-main-515c8f-6bb9bf56f4-9l2jh\" (UID: \"ac7998dc-9420-4207-95fc-cf5bb9ddc27b\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-515c8f-6bb9bf56f4-9l2jh" Nov 03 01:49:21.857799 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:49:21.857596 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5fkln\" (UniqueName: \"kubernetes.io/projected/ac7998dc-9420-4207-95fc-cf5bb9ddc27b-kube-api-access-5fkln\") pod \"maas-api-e2e-worker-single-main-515c8f-6bb9bf56f4-9l2jh\" (UID: \"ac7998dc-9420-4207-95fc-cf5bb9ddc27b\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-515c8f-6bb9bf56f4-9l2jh" Nov 03 01:49:21.857799 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:49:21.857640 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ac7998dc-9420-4207-95fc-cf5bb9ddc27b-maas-api-tls\") pod \"maas-api-e2e-worker-single-main-515c8f-6bb9bf56f4-9l2jh\" (UID: \"ac7998dc-9420-4207-95fc-cf5bb9ddc27b\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-515c8f-6bb9bf56f4-9l2jh" Nov 03 01:49:21.857799 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:49:21.857666 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ac7998dc-9420-4207-95fc-cf5bb9ddc27b-metrics-certs\") pod \"maas-api-e2e-worker-single-main-515c8f-6bb9bf56f4-9l2jh\" (UID: \"ac7998dc-9420-4207-95fc-cf5bb9ddc27b\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-515c8f-6bb9bf56f4-9l2jh" Nov 03 01:49:21.857799 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:49:21.857696 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/472ad6a5-f51e-4c14-b2d7-bed131e23705-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-single-main-515c8f-78658vth74\" (UID: \"472ad6a5-f51e-4c14-b2d7-bed131e23705\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-515c8f-78658vth74" Nov 03 01:49:21.857799 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:49:21.857722 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p27jt\" (UniqueName: \"kubernetes.io/projected/472ad6a5-f51e-4c14-b2d7-bed131e23705-kube-api-access-p27jt\") pod \"payload-pre-processing-e2e-worker-single-main-515c8f-78658vth74\" (UID: \"472ad6a5-f51e-4c14-b2d7-bed131e23705\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-515c8f-78658vth74" Nov 03 01:49:21.858467 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:49:21.858407 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ac7998dc-9420-4207-95fc-cf5bb9ddc27b-openshift-service-ca\") pod \"maas-api-e2e-worker-single-main-515c8f-6bb9bf56f4-9l2jh\" (UID: \"ac7998dc-9420-4207-95fc-cf5bb9ddc27b\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-515c8f-6bb9bf56f4-9l2jh" Nov 03 01:49:21.858624 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:49:21.858535 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/472ad6a5-f51e-4c14-b2d7-bed131e23705-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-single-main-515c8f-78658vth74\" (UID: \"472ad6a5-f51e-4c14-b2d7-bed131e23705\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-515c8f-78658vth74" Nov 03 01:49:21.860271 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:49:21.860243 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ac7998dc-9420-4207-95fc-cf5bb9ddc27b-metrics-certs\") pod \"maas-api-e2e-worker-single-main-515c8f-6bb9bf56f4-9l2jh\" (UID: \"ac7998dc-9420-4207-95fc-cf5bb9ddc27b\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-515c8f-6bb9bf56f4-9l2jh" Nov 03 01:49:21.860377 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:49:21.860285 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ac7998dc-9420-4207-95fc-cf5bb9ddc27b-maas-api-tls\") pod \"maas-api-e2e-worker-single-main-515c8f-6bb9bf56f4-9l2jh\" (UID: \"ac7998dc-9420-4207-95fc-cf5bb9ddc27b\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-515c8f-6bb9bf56f4-9l2jh" Nov 03 01:49:21.867459 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:49:21.867429 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p27jt\" (UniqueName: \"kubernetes.io/projected/472ad6a5-f51e-4c14-b2d7-bed131e23705-kube-api-access-p27jt\") pod \"payload-pre-processing-e2e-worker-single-main-515c8f-78658vth74\" (UID: \"472ad6a5-f51e-4c14-b2d7-bed131e23705\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-515c8f-78658vth74" Nov 03 01:49:21.867740 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:49:21.867717 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5fkln\" (UniqueName: \"kubernetes.io/projected/ac7998dc-9420-4207-95fc-cf5bb9ddc27b-kube-api-access-5fkln\") pod \"maas-api-e2e-worker-single-main-515c8f-6bb9bf56f4-9l2jh\" (UID: \"ac7998dc-9420-4207-95fc-cf5bb9ddc27b\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-515c8f-6bb9bf56f4-9l2jh" Nov 03 01:49:21.997042 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:49:21.996961 4939 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-515c8f-6bb9bf56f4-9l2jh" Nov 03 01:49:22.009239 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:49:22.009214 4939 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-515c8f-78658vth74" Nov 03 01:49:22.279477 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:49:22.279370 4939 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-515c8f-6bb9bf56f4-9l2jh"] Nov 03 01:49:22.286899 ip-10-0-140-107 kubenswrapper[4939]: W1103 01:49:22.286870 4939 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podac7998dc_9420_4207_95fc_cf5bb9ddc27b.slice/crio-583369071ebb81afd55d4cfcd68eb252d118bdc5ffa000e57e11514adf3eeb30 WatchSource:0}: Error finding container 583369071ebb81afd55d4cfcd68eb252d118bdc5ffa000e57e11514adf3eeb30: Status 404 returned error can't find the container with id 583369071ebb81afd55d4cfcd68eb252d118bdc5ffa000e57e11514adf3eeb30 Nov 03 01:49:22.288531 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:49:22.288501 4939 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Nov 03 01:49:22.304392 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:49:22.304364 4939 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-515c8f-78658vth74"] Nov 03 01:49:22.304580 ip-10-0-140-107 kubenswrapper[4939]: W1103 01:49:22.304558 4939 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod472ad6a5_f51e_4c14_b2d7_bed131e23705.slice/crio-d8044dcf4395a3ac3443edeaeaff4392d232013f0b45372e478e59238b0c4492 WatchSource:0}: Error finding container d8044dcf4395a3ac3443edeaeaff4392d232013f0b45372e478e59238b0c4492: Status 404 returned error can't find the container with id d8044dcf4395a3ac3443edeaeaff4392d232013f0b45372e478e59238b0c4492 Nov 03 01:49:22.410383 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:49:22.410339 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:49:22.808555 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:49:22.808512 4939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-515c8f-6bb9bf56f4-9l2jh" event={"ID":"ac7998dc-9420-4207-95fc-cf5bb9ddc27b","Type":"ContainerStarted","Data":"583369071ebb81afd55d4cfcd68eb252d118bdc5ffa000e57e11514adf3eeb30"} Nov 03 01:49:22.810109 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:49:22.810077 4939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-515c8f-78658vth74" event={"ID":"472ad6a5-f51e-4c14-b2d7-bed131e23705","Type":"ContainerStarted","Data":"1d02137d50516d70d9207ed54bb546f822d8f39093b02767a2e52d4cd0e2a702"} Nov 03 01:49:22.810260 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:49:22.810124 4939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-515c8f-78658vth74" event={"ID":"472ad6a5-f51e-4c14-b2d7-bed131e23705","Type":"ContainerStarted","Data":"d8044dcf4395a3ac3443edeaeaff4392d232013f0b45372e478e59238b0c4492"} Nov 03 01:49:22.810260 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:49:22.810161 4939 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-515c8f-78658vth74" Nov 03 01:49:22.827256 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:49:22.827211 4939 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-515c8f-78658vth74" podStartSLOduration=1.827199986 podStartE2EDuration="1.827199986s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 01:49:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 01:49:22.82559281 +0000 UTC m=+4111.089927755" watchObservedRunningTime="2026-11-03 01:49:22.827199986 +0000 UTC m=+4111.091534931" Nov 03 01:49:23.107271 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:49:23.107242 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:49:23.775806 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:49:23.775768 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:49:23.824008 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:49:23.823190 4939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-515c8f-6bb9bf56f4-9l2jh" event={"ID":"ac7998dc-9420-4207-95fc-cf5bb9ddc27b","Type":"ContainerStarted","Data":"aac9f9b316f88e33e41fae97b0aa674fbfe02260329ea79604d1a2d125a289ad"} Nov 03 01:49:23.824008 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:49:23.823258 4939 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-515c8f-6bb9bf56f4-9l2jh" Nov 03 01:49:23.842326 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:49:23.842277 4939 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-515c8f-6bb9bf56f4-9l2jh" podStartSLOduration=2.264365238 podStartE2EDuration="2.842262958s" totalImagesPullingTime="577.89772ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 01:49:21 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 01:49:23.839705813 +0000 UTC m=+4112.104040773" watchObservedRunningTime="2026-11-03 01:49:23.842262958 +0000 UTC m=+4112.106597903" Nov 03 01:49:24.393447 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:49:24.393411 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:49:25.061614 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:49:25.061581 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:49:25.230189 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:49:25.230152 4939 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-4pw7v"] Nov 03 01:49:25.783766 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:49:25.783720 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:49:26.015414 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:49:26.015385 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:49:26.447893 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:49:26.447831 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:49:26.931372 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:49:26.931339 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:49:27.149502 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:49:27.149465 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:49:27.970315 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:49:27.970285 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:49:28.596054 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:49:28.596016 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:49:29.196358 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:49:29.196319 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:49:29.426750 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:49:29.426712 4939 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-4pw7v"] Nov 03 01:49:29.801356 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:49:29.801318 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:49:29.847145 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:49:29.847113 4939 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-515c8f-6bb9bf56f4-9l2jh" Nov 03 01:49:30.498711 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:49:30.498677 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:49:31.097784 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:49:31.097750 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:49:31.693044 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:49:31.693012 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:49:32.287756 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:49:32.287724 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:49:32.472208 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:49:32.472170 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:49:32.587330 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:49:32.587294 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:49:32.958482 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:49:32.958399 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:49:33.685262 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:49:33.685229 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:49:33.825571 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:49:33.825542 4939 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-515c8f-78658vth74" Nov 03 01:49:34.308993 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:49:34.308956 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:49:34.985496 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:49:34.985460 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:49:35.608334 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:49:35.608300 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:49:36.282309 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:49:36.282278 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:49:36.890944 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:49:36.890889 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:49:37.488474 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:49:37.488438 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:49:38.090035 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:49:38.090002 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:49:38.692436 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:49:38.692399 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:49:39.301058 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:49:39.301022 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:49:39.906298 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:49:39.906263 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:49:40.364551 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:49:40.364517 4939 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-515c8f-78658vth74"] Nov 03 01:49:40.453447 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:49:40.453395 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:49:40.571560 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:49:40.571526 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:49:40.880312 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:49:40.880261 4939 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-515c8f-6bb9bf56f4-9l2jh"] Nov 03 01:49:40.880589 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:49:40.880558 4939 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-515c8f-6bb9bf56f4-9l2jh" podUID="ac7998dc-9420-4207-95fc-cf5bb9ddc27b" containerName="maas-api" containerID="cri-o://aac9f9b316f88e33e41fae97b0aa674fbfe02260329ea79604d1a2d125a289ad" gracePeriod=30 Nov 03 01:49:40.948545 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:49:40.948499 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:49:41.020387 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:49:41.020351 4939 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-8fv7k"] Nov 03 01:49:41.026228 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:49:41.026199 4939 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-8fv7k"] Nov 03 01:49:41.067206 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:49:41.067124 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:49:41.174764 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:49:41.174739 4939 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-515c8f-6bb9bf56f4-9l2jh" Nov 03 01:49:41.200521 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:49:41.200475 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:49:41.214673 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:49:41.214640 4939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ac7998dc-9420-4207-95fc-cf5bb9ddc27b-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ac7998dc-9420-4207-95fc-cf5bb9ddc27b-maas-api-tls\") pod \"ac7998dc-9420-4207-95fc-cf5bb9ddc27b\" (UID: \"ac7998dc-9420-4207-95fc-cf5bb9ddc27b\") " Nov 03 01:49:41.214862 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:49:41.214705 4939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ac7998dc-9420-4207-95fc-cf5bb9ddc27b-kube-api-access-5fkln\" (UniqueName: \"kubernetes.io/projected/ac7998dc-9420-4207-95fc-cf5bb9ddc27b-kube-api-access-5fkln\") pod \"ac7998dc-9420-4207-95fc-cf5bb9ddc27b\" (UID: \"ac7998dc-9420-4207-95fc-cf5bb9ddc27b\") " Nov 03 01:49:41.214862 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:49:41.214821 4939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ac7998dc-9420-4207-95fc-cf5bb9ddc27b-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ac7998dc-9420-4207-95fc-cf5bb9ddc27b-openshift-service-ca\") pod \"ac7998dc-9420-4207-95fc-cf5bb9ddc27b\" (UID: \"ac7998dc-9420-4207-95fc-cf5bb9ddc27b\") " Nov 03 01:49:41.215251 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:49:41.215220 4939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ac7998dc-9420-4207-95fc-cf5bb9ddc27b-openshift-service-ca" pod "ac7998dc-9420-4207-95fc-cf5bb9ddc27b" (UID: "ac7998dc-9420-4207-95fc-cf5bb9ddc27b"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 01:49:41.217074 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:49:41.217036 4939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ac7998dc-9420-4207-95fc-cf5bb9ddc27b-kube-api-access-5fkln" pod "ac7998dc-9420-4207-95fc-cf5bb9ddc27b" (UID: "ac7998dc-9420-4207-95fc-cf5bb9ddc27b"). InnerVolumeSpecName "kube-api-access-5fkln". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 01:49:41.217217 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:49:41.217139 4939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ac7998dc-9420-4207-95fc-cf5bb9ddc27b-maas-api-tls" pod "ac7998dc-9420-4207-95fc-cf5bb9ddc27b" (UID: "ac7998dc-9420-4207-95fc-cf5bb9ddc27b"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 01:49:41.316159 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:49:41.316123 4939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ac7998dc-9420-4207-95fc-cf5bb9ddc27b-metrics-certs\" (UniqueName: \"kubernetes.io/secret/ac7998dc-9420-4207-95fc-cf5bb9ddc27b-metrics-certs\") pod \"ac7998dc-9420-4207-95fc-cf5bb9ddc27b\" (UID: \"ac7998dc-9420-4207-95fc-cf5bb9ddc27b\") " Nov 03 01:49:41.316393 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:49:41.316380 4939 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ac7998dc-9420-4207-95fc-cf5bb9ddc27b-openshift-service-ca\") on node \"ip-10-0-140-107.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:49:41.316446 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:49:41.316396 4939 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ac7998dc-9420-4207-95fc-cf5bb9ddc27b-maas-api-tls\") on node \"ip-10-0-140-107.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:49:41.316446 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:49:41.316406 4939 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5fkln\" (UniqueName: \"kubernetes.io/projected/ac7998dc-9420-4207-95fc-cf5bb9ddc27b-kube-api-access-5fkln\") on node \"ip-10-0-140-107.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:49:41.318313 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:49:41.318281 4939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ac7998dc-9420-4207-95fc-cf5bb9ddc27b-metrics-certs" pod "ac7998dc-9420-4207-95fc-cf5bb9ddc27b" (UID: "ac7998dc-9420-4207-95fc-cf5bb9ddc27b"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 01:49:41.417882 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:49:41.417798 4939 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ac7998dc-9420-4207-95fc-cf5bb9ddc27b-metrics-certs\") on node \"ip-10-0-140-107.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:49:41.597821 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:49:41.597784 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:49:41.869317 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:49:41.869277 4939 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3655425e-192d-4875-b633-19ca4e7309de" path="/var/lib/kubelet/pods/3655425e-192d-4875-b633-19ca4e7309de/volumes" Nov 03 01:49:41.934316 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:49:41.934277 4939 generic.go:356] "Generic (PLEG): container finished" podID="ac7998dc-9420-4207-95fc-cf5bb9ddc27b" containerID="aac9f9b316f88e33e41fae97b0aa674fbfe02260329ea79604d1a2d125a289ad" exitCode=0 Nov 03 01:49:41.934516 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:49:41.934327 4939 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-515c8f-6bb9bf56f4-9l2jh" Nov 03 01:49:41.934516 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:49:41.934359 4939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-515c8f-6bb9bf56f4-9l2jh" event={"ID":"ac7998dc-9420-4207-95fc-cf5bb9ddc27b","Type":"ContainerDied","Data":"aac9f9b316f88e33e41fae97b0aa674fbfe02260329ea79604d1a2d125a289ad"} Nov 03 01:49:41.934516 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:49:41.934419 4939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-515c8f-6bb9bf56f4-9l2jh" event={"ID":"ac7998dc-9420-4207-95fc-cf5bb9ddc27b","Type":"ContainerDied","Data":"583369071ebb81afd55d4cfcd68eb252d118bdc5ffa000e57e11514adf3eeb30"} Nov 03 01:49:41.934516 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:49:41.934446 4939 scope.go:122] "RemoveContainer" containerID="aac9f9b316f88e33e41fae97b0aa674fbfe02260329ea79604d1a2d125a289ad" Nov 03 01:49:41.944103 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:49:41.944082 4939 scope.go:122] "RemoveContainer" containerID="aac9f9b316f88e33e41fae97b0aa674fbfe02260329ea79604d1a2d125a289ad" Nov 03 01:49:41.944370 ip-10-0-140-107 kubenswrapper[4939]: E1103 01:49:41.944343 4939 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"aac9f9b316f88e33e41fae97b0aa674fbfe02260329ea79604d1a2d125a289ad\": container with ID starting with aac9f9b316f88e33e41fae97b0aa674fbfe02260329ea79604d1a2d125a289ad not found: ID does not exist" containerID="aac9f9b316f88e33e41fae97b0aa674fbfe02260329ea79604d1a2d125a289ad" Nov 03 01:49:41.944455 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:49:41.944383 4939 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"aac9f9b316f88e33e41fae97b0aa674fbfe02260329ea79604d1a2d125a289ad"} err="failed to get container status \"aac9f9b316f88e33e41fae97b0aa674fbfe02260329ea79604d1a2d125a289ad\": rpc error: code = NotFound desc = could not find container \"aac9f9b316f88e33e41fae97b0aa674fbfe02260329ea79604d1a2d125a289ad\": container with ID starting with aac9f9b316f88e33e41fae97b0aa674fbfe02260329ea79604d1a2d125a289ad not found: ID does not exist" Nov 03 01:49:41.951812 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:49:41.951790 4939 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-515c8f-6bb9bf56f4-9l2jh"] Nov 03 01:49:41.954438 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:49:41.954413 4939 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-515c8f-6bb9bf56f4-9l2jh"] Nov 03 01:49:42.916925 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:49:42.916880 4939 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-4pw7v"] Nov 03 01:49:43.867339 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:49:43.867306 4939 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ac7998dc-9420-4207-95fc-cf5bb9ddc27b" path="/var/lib/kubelet/pods/ac7998dc-9420-4207-95fc-cf5bb9ddc27b/volumes" Nov 03 01:49:45.365211 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:49:45.365149 4939 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-515c8f-78658vth74" podUID="472ad6a5-f51e-4c14-b2d7-bed131e23705" containerName="payload-pre-processing" containerID="cri-o://1d02137d50516d70d9207ed54bb546f822d8f39093b02767a2e52d4cd0e2a702" gracePeriod=25 Nov 03 01:49:45.614681 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:49:45.614658 4939 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-515c8f-78658vth74" Nov 03 01:49:45.661073 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:49:45.660986 4939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/472ad6a5-f51e-4c14-b2d7-bed131e23705-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/472ad6a5-f51e-4c14-b2d7-bed131e23705-plugins-config-volume\") pod \"472ad6a5-f51e-4c14-b2d7-bed131e23705\" (UID: \"472ad6a5-f51e-4c14-b2d7-bed131e23705\") " Nov 03 01:49:45.661073 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:49:45.661027 4939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/472ad6a5-f51e-4c14-b2d7-bed131e23705-kube-api-access-p27jt\" (UniqueName: \"kubernetes.io/projected/472ad6a5-f51e-4c14-b2d7-bed131e23705-kube-api-access-p27jt\") pod \"472ad6a5-f51e-4c14-b2d7-bed131e23705\" (UID: \"472ad6a5-f51e-4c14-b2d7-bed131e23705\") " Nov 03 01:49:45.661472 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:49:45.661445 4939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/472ad6a5-f51e-4c14-b2d7-bed131e23705-plugins-config-volume" pod "472ad6a5-f51e-4c14-b2d7-bed131e23705" (UID: "472ad6a5-f51e-4c14-b2d7-bed131e23705"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 01:49:45.663232 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:49:45.663207 4939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/472ad6a5-f51e-4c14-b2d7-bed131e23705-kube-api-access-p27jt" pod "472ad6a5-f51e-4c14-b2d7-bed131e23705" (UID: "472ad6a5-f51e-4c14-b2d7-bed131e23705"). InnerVolumeSpecName "kube-api-access-p27jt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 01:49:45.762182 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:49:45.762143 4939 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/472ad6a5-f51e-4c14-b2d7-bed131e23705-plugins-config-volume\") on node \"ip-10-0-140-107.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:49:45.762182 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:49:45.762178 4939 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p27jt\" (UniqueName: \"kubernetes.io/projected/472ad6a5-f51e-4c14-b2d7-bed131e23705-kube-api-access-p27jt\") on node \"ip-10-0-140-107.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:49:45.957038 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:49:45.956960 4939 generic.go:356] "Generic (PLEG): container finished" podID="472ad6a5-f51e-4c14-b2d7-bed131e23705" containerID="1d02137d50516d70d9207ed54bb546f822d8f39093b02767a2e52d4cd0e2a702" exitCode=0 Nov 03 01:49:45.957038 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:49:45.957008 4939 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-515c8f-78658vth74" Nov 03 01:49:45.957237 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:49:45.957044 4939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-515c8f-78658vth74" event={"ID":"472ad6a5-f51e-4c14-b2d7-bed131e23705","Type":"ContainerDied","Data":"1d02137d50516d70d9207ed54bb546f822d8f39093b02767a2e52d4cd0e2a702"} Nov 03 01:49:45.957237 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:49:45.957094 4939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-515c8f-78658vth74" event={"ID":"472ad6a5-f51e-4c14-b2d7-bed131e23705","Type":"ContainerDied","Data":"d8044dcf4395a3ac3443edeaeaff4392d232013f0b45372e478e59238b0c4492"} Nov 03 01:49:45.957237 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:49:45.957115 4939 scope.go:122] "RemoveContainer" containerID="1d02137d50516d70d9207ed54bb546f822d8f39093b02767a2e52d4cd0e2a702" Nov 03 01:49:45.965953 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:49:45.965936 4939 scope.go:122] "RemoveContainer" containerID="1d02137d50516d70d9207ed54bb546f822d8f39093b02767a2e52d4cd0e2a702" Nov 03 01:49:45.966218 ip-10-0-140-107 kubenswrapper[4939]: E1103 01:49:45.966199 4939 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1d02137d50516d70d9207ed54bb546f822d8f39093b02767a2e52d4cd0e2a702\": container with ID starting with 1d02137d50516d70d9207ed54bb546f822d8f39093b02767a2e52d4cd0e2a702 not found: ID does not exist" containerID="1d02137d50516d70d9207ed54bb546f822d8f39093b02767a2e52d4cd0e2a702" Nov 03 01:49:45.966289 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:49:45.966230 4939 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1d02137d50516d70d9207ed54bb546f822d8f39093b02767a2e52d4cd0e2a702"} err="failed to get container status \"1d02137d50516d70d9207ed54bb546f822d8f39093b02767a2e52d4cd0e2a702\": rpc error: code = NotFound desc = could not find container \"1d02137d50516d70d9207ed54bb546f822d8f39093b02767a2e52d4cd0e2a702\": container with ID starting with 1d02137d50516d70d9207ed54bb546f822d8f39093b02767a2e52d4cd0e2a702 not found: ID does not exist" Nov 03 01:49:45.974271 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:49:45.974246 4939 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-515c8f-78658vth74"] Nov 03 01:49:45.979985 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:49:45.979964 4939 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-515c8f-78658vth74"] Nov 03 01:49:46.332067 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:49:46.332030 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:49:47.867101 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:49:47.867056 4939 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="472ad6a5-f51e-4c14-b2d7-bed131e23705" path="/var/lib/kubelet/pods/472ad6a5-f51e-4c14-b2d7-bed131e23705/volumes" Nov 03 01:49:48.028957 ip-10-0-140-107 kubenswrapper[4939]: E1103 01:49:48.028921 4939 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-sthjh" containerName="extractor" Nov 03 01:49:48.777336 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:49:48.777300 4939 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/e2e-same-id-a-077b6796-kserve-55c58b6bcc-6rs9p"] Nov 03 01:49:48.777723 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:49:48.777709 4939 cpu_manager.go:414] "containerMap: removing container" podUID="472ad6a5-f51e-4c14-b2d7-bed131e23705" containerName="payload-pre-processing" Nov 03 01:49:48.777772 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:49:48.777724 4939 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="472ad6a5-f51e-4c14-b2d7-bed131e23705" containerName="payload-pre-processing" Nov 03 01:49:48.777772 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:49:48.777738 4939 cpu_manager.go:414] "containerMap: removing container" podUID="ac7998dc-9420-4207-95fc-cf5bb9ddc27b" containerName="maas-api" Nov 03 01:49:48.777772 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:49:48.777744 4939 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ac7998dc-9420-4207-95fc-cf5bb9ddc27b" containerName="maas-api" Nov 03 01:49:48.777876 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:49:48.777820 4939 memory_manager.go:365] "RemoveStaleState removing state" podUID="472ad6a5-f51e-4c14-b2d7-bed131e23705" containerName="payload-pre-processing" Nov 03 01:49:48.777876 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:49:48.777828 4939 memory_manager.go:365] "RemoveStaleState removing state" podUID="ac7998dc-9420-4207-95fc-cf5bb9ddc27b" containerName="maas-api" Nov 03 01:49:48.783994 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:49:48.783969 4939 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-077b6796-kserve-55c58b6bcc-6rs9p" Nov 03 01:49:48.790539 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:49:48.789695 4939 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-a-077b6796-kserve-55c58b6bcc-6rs9p"] Nov 03 01:49:48.791800 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:49:48.791768 4939 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-same-id-a-077b6796-kserve-self-signed-certs\"" Nov 03 01:49:48.846158 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:49:48.846131 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:49:48.892021 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:49:48.891989 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hmr2l\" (UniqueName: \"kubernetes.io/projected/56d83725-cf8a-44e5-8af7-1e7a0eb4107f-kube-api-access-hmr2l\") pod \"e2e-same-id-a-077b6796-kserve-55c58b6bcc-6rs9p\" (UID: \"56d83725-cf8a-44e5-8af7-1e7a0eb4107f\") " pod="llm/e2e-same-id-a-077b6796-kserve-55c58b6bcc-6rs9p" Nov 03 01:49:48.892403 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:49:48.892031 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/56d83725-cf8a-44e5-8af7-1e7a0eb4107f-home\") pod \"e2e-same-id-a-077b6796-kserve-55c58b6bcc-6rs9p\" (UID: \"56d83725-cf8a-44e5-8af7-1e7a0eb4107f\") " pod="llm/e2e-same-id-a-077b6796-kserve-55c58b6bcc-6rs9p" Nov 03 01:49:48.892403 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:49:48.892060 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/56d83725-cf8a-44e5-8af7-1e7a0eb4107f-dshm\") pod \"e2e-same-id-a-077b6796-kserve-55c58b6bcc-6rs9p\" (UID: \"56d83725-cf8a-44e5-8af7-1e7a0eb4107f\") " pod="llm/e2e-same-id-a-077b6796-kserve-55c58b6bcc-6rs9p" Nov 03 01:49:48.892403 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:49:48.892160 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/56d83725-cf8a-44e5-8af7-1e7a0eb4107f-tmp-dir\") pod \"e2e-same-id-a-077b6796-kserve-55c58b6bcc-6rs9p\" (UID: \"56d83725-cf8a-44e5-8af7-1e7a0eb4107f\") " pod="llm/e2e-same-id-a-077b6796-kserve-55c58b6bcc-6rs9p" Nov 03 01:49:48.892403 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:49:48.892200 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/56d83725-cf8a-44e5-8af7-1e7a0eb4107f-tls-certs\") pod \"e2e-same-id-a-077b6796-kserve-55c58b6bcc-6rs9p\" (UID: \"56d83725-cf8a-44e5-8af7-1e7a0eb4107f\") " pod="llm/e2e-same-id-a-077b6796-kserve-55c58b6bcc-6rs9p" Nov 03 01:49:48.892403 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:49:48.892227 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/56d83725-cf8a-44e5-8af7-1e7a0eb4107f-model-cache\") pod \"e2e-same-id-a-077b6796-kserve-55c58b6bcc-6rs9p\" (UID: \"56d83725-cf8a-44e5-8af7-1e7a0eb4107f\") " pod="llm/e2e-same-id-a-077b6796-kserve-55c58b6bcc-6rs9p" Nov 03 01:49:48.993452 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:49:48.993409 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/56d83725-cf8a-44e5-8af7-1e7a0eb4107f-dshm\") pod \"e2e-same-id-a-077b6796-kserve-55c58b6bcc-6rs9p\" (UID: \"56d83725-cf8a-44e5-8af7-1e7a0eb4107f\") " pod="llm/e2e-same-id-a-077b6796-kserve-55c58b6bcc-6rs9p" Nov 03 01:49:48.993779 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:49:48.993483 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/56d83725-cf8a-44e5-8af7-1e7a0eb4107f-tmp-dir\") pod \"e2e-same-id-a-077b6796-kserve-55c58b6bcc-6rs9p\" (UID: \"56d83725-cf8a-44e5-8af7-1e7a0eb4107f\") " pod="llm/e2e-same-id-a-077b6796-kserve-55c58b6bcc-6rs9p" Nov 03 01:49:48.993779 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:49:48.993535 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/56d83725-cf8a-44e5-8af7-1e7a0eb4107f-tls-certs\") pod \"e2e-same-id-a-077b6796-kserve-55c58b6bcc-6rs9p\" (UID: \"56d83725-cf8a-44e5-8af7-1e7a0eb4107f\") " pod="llm/e2e-same-id-a-077b6796-kserve-55c58b6bcc-6rs9p" Nov 03 01:49:48.993779 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:49:48.993595 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/56d83725-cf8a-44e5-8af7-1e7a0eb4107f-model-cache\") pod \"e2e-same-id-a-077b6796-kserve-55c58b6bcc-6rs9p\" (UID: \"56d83725-cf8a-44e5-8af7-1e7a0eb4107f\") " pod="llm/e2e-same-id-a-077b6796-kserve-55c58b6bcc-6rs9p" Nov 03 01:49:48.993779 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:49:48.993678 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hmr2l\" (UniqueName: \"kubernetes.io/projected/56d83725-cf8a-44e5-8af7-1e7a0eb4107f-kube-api-access-hmr2l\") pod \"e2e-same-id-a-077b6796-kserve-55c58b6bcc-6rs9p\" (UID: \"56d83725-cf8a-44e5-8af7-1e7a0eb4107f\") " pod="llm/e2e-same-id-a-077b6796-kserve-55c58b6bcc-6rs9p" Nov 03 01:49:48.993779 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:49:48.993728 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/56d83725-cf8a-44e5-8af7-1e7a0eb4107f-home\") pod \"e2e-same-id-a-077b6796-kserve-55c58b6bcc-6rs9p\" (UID: \"56d83725-cf8a-44e5-8af7-1e7a0eb4107f\") " pod="llm/e2e-same-id-a-077b6796-kserve-55c58b6bcc-6rs9p" Nov 03 01:49:48.994151 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:49:48.993943 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/56d83725-cf8a-44e5-8af7-1e7a0eb4107f-tmp-dir\") pod \"e2e-same-id-a-077b6796-kserve-55c58b6bcc-6rs9p\" (UID: \"56d83725-cf8a-44e5-8af7-1e7a0eb4107f\") " pod="llm/e2e-same-id-a-077b6796-kserve-55c58b6bcc-6rs9p" Nov 03 01:49:48.994151 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:49:48.994008 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/56d83725-cf8a-44e5-8af7-1e7a0eb4107f-model-cache\") pod \"e2e-same-id-a-077b6796-kserve-55c58b6bcc-6rs9p\" (UID: \"56d83725-cf8a-44e5-8af7-1e7a0eb4107f\") " pod="llm/e2e-same-id-a-077b6796-kserve-55c58b6bcc-6rs9p" Nov 03 01:49:48.994151 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:49:48.994085 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/56d83725-cf8a-44e5-8af7-1e7a0eb4107f-home\") pod \"e2e-same-id-a-077b6796-kserve-55c58b6bcc-6rs9p\" (UID: \"56d83725-cf8a-44e5-8af7-1e7a0eb4107f\") " pod="llm/e2e-same-id-a-077b6796-kserve-55c58b6bcc-6rs9p" Nov 03 01:49:48.995973 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:49:48.995949 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/56d83725-cf8a-44e5-8af7-1e7a0eb4107f-dshm\") pod \"e2e-same-id-a-077b6796-kserve-55c58b6bcc-6rs9p\" (UID: \"56d83725-cf8a-44e5-8af7-1e7a0eb4107f\") " pod="llm/e2e-same-id-a-077b6796-kserve-55c58b6bcc-6rs9p" Nov 03 01:49:48.996105 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:49:48.996089 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/56d83725-cf8a-44e5-8af7-1e7a0eb4107f-tls-certs\") pod \"e2e-same-id-a-077b6796-kserve-55c58b6bcc-6rs9p\" (UID: \"56d83725-cf8a-44e5-8af7-1e7a0eb4107f\") " pod="llm/e2e-same-id-a-077b6796-kserve-55c58b6bcc-6rs9p" Nov 03 01:49:49.003279 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:49:49.003256 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hmr2l\" (UniqueName: \"kubernetes.io/projected/56d83725-cf8a-44e5-8af7-1e7a0eb4107f-kube-api-access-hmr2l\") pod \"e2e-same-id-a-077b6796-kserve-55c58b6bcc-6rs9p\" (UID: \"56d83725-cf8a-44e5-8af7-1e7a0eb4107f\") " pod="llm/e2e-same-id-a-077b6796-kserve-55c58b6bcc-6rs9p" Nov 03 01:49:49.112506 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:49:49.112464 4939 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-077b6796-kserve-55c58b6bcc-6rs9p" Nov 03 01:49:49.384300 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:49:49.384270 4939 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-a-077b6796-kserve-55c58b6bcc-6rs9p"] Nov 03 01:49:49.386146 ip-10-0-140-107 kubenswrapper[4939]: W1103 01:49:49.386116 4939 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod56d83725_cf8a_44e5_8af7_1e7a0eb4107f.slice/crio-8dc0bc1cba95ff1b568f600ade22a955a8bba5646a89e36b9698c7563405899d WatchSource:0}: Error finding container 8dc0bc1cba95ff1b568f600ade22a955a8bba5646a89e36b9698c7563405899d: Status 404 returned error can't find the container with id 8dc0bc1cba95ff1b568f600ade22a955a8bba5646a89e36b9698c7563405899d Nov 03 01:49:49.978543 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:49:49.978507 4939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-077b6796-kserve-55c58b6bcc-6rs9p" event={"ID":"56d83725-cf8a-44e5-8af7-1e7a0eb4107f","Type":"ContainerStarted","Data":"d73fd6ed6b0ac43a3a1706f7bba56e99d434acf898c2534cb01e9a2ef74779c5"} Nov 03 01:49:49.978970 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:49:49.978555 4939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-077b6796-kserve-55c58b6bcc-6rs9p" event={"ID":"56d83725-cf8a-44e5-8af7-1e7a0eb4107f","Type":"ContainerStarted","Data":"8dc0bc1cba95ff1b568f600ade22a955a8bba5646a89e36b9698c7563405899d"} Nov 03 01:49:49.995246 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:49:49.995199 4939 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/e2e-same-id-a-077b6796-kserve-55c58b6bcc-6rs9p" podStartSLOduration=1.995185886 podStartE2EDuration="1.995185886s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 01:49:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 01:49:49.994147099 +0000 UTC m=+4138.258482045" watchObservedRunningTime="2026-11-03 01:49:49.995185886 +0000 UTC m=+4138.259520830" Nov 03 01:49:50.115261 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:49:50.115140 4939 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-same-id-a-077b6796-kserve-55c58b6bcc-6rs9p" Nov 03 01:49:50.199688 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:49:50.199649 4939 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-4pw7v"] Nov 03 01:49:59.113503 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:49:59.113466 4939 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-same-id-a-077b6796-kserve-55c58b6bcc-6rs9p" Nov 03 01:49:59.125740 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:49:59.125713 4939 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-same-id-a-077b6796-kserve-55c58b6bcc-6rs9p" Nov 03 01:50:00.069264 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:50:00.069185 4939 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-same-id-a-077b6796-kserve-55c58b6bcc-6rs9p" Nov 03 01:50:01.982649 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:50:01.982613 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:50:03.319493 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:50:03.319459 4939 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-4pw7v"] Nov 03 01:50:14.789223 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:50:14.789167 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:50:18.030575 ip-10-0-140-107 kubenswrapper[4939]: E1103 01:50:18.030537 4939 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-sthjh" containerName="extractor" Nov 03 01:50:19.540945 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:50:19.540895 4939 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-4pw7v"] Nov 03 01:50:22.024373 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:50:22.024329 4939 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f904bff9-hgm8v"] Nov 03 01:50:22.044931 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:50:22.032270 4939 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f904bff9-hgm8v"] Nov 03 01:50:23.867706 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:50:23.867668 4939 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9c7f9e2a-3774-4cf5-8997-6fbf6d8c614b" path="/var/lib/kubelet/pods/9c7f9e2a-3774-4cf5-8997-6fbf6d8c614b/volumes" Nov 03 01:50:26.403717 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:50:26.403681 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:50:26.503508 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:50:26.503478 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:50:29.941677 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:50:29.941634 4939 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-a-077b6796-kserve-55c58b6bcc-6rs9p"] Nov 03 01:50:29.988613 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:50:29.988560 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:50:31.282351 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:50:31.282317 4939 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-4pw7v"] Nov 03 01:50:31.536065 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:50:31.535968 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:50:36.902987 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:50:36.901887 4939 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-4pw7v"] Nov 03 01:50:44.029347 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:50:44.029314 4939 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-zr995"] Nov 03 01:50:44.032515 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:50:44.032485 4939 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-zr995"] Nov 03 01:50:44.962980 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:50:44.962921 4939 kuberuntime_container.go:898] "Killing container with a grace period" pod="llm/e2e-same-id-a-077b6796-kserve-55c58b6bcc-6rs9p" podUID="56d83725-cf8a-44e5-8af7-1e7a0eb4107f" containerName="main" containerID="cri-o://d73fd6ed6b0ac43a3a1706f7bba56e99d434acf898c2534cb01e9a2ef74779c5" gracePeriod=45 Nov 03 01:50:45.058531 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:50:45.058495 4939 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/e2e-same-id-a-077b6796-kserve-55c58b6bcc-6rs9p" podUID="56d83725-cf8a-44e5-8af7-1e7a0eb4107f" containerName="main" probeResult="failure" output="Get \"https://10.132.1.112:8000/ready\": dial tcp 10.132.1.112:8000: connect: connection refused" Nov 03 01:50:45.208883 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:50:45.208857 4939 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-077b6796-kserve-55c58b6bcc-6rs9p" Nov 03 01:50:45.329555 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:50:45.329478 4939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/56d83725-cf8a-44e5-8af7-1e7a0eb4107f-dshm\" (UniqueName: \"kubernetes.io/empty-dir/56d83725-cf8a-44e5-8af7-1e7a0eb4107f-dshm\") pod \"56d83725-cf8a-44e5-8af7-1e7a0eb4107f\" (UID: \"56d83725-cf8a-44e5-8af7-1e7a0eb4107f\") " Nov 03 01:50:45.329555 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:50:45.329554 4939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/56d83725-cf8a-44e5-8af7-1e7a0eb4107f-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/56d83725-cf8a-44e5-8af7-1e7a0eb4107f-model-cache\") pod \"56d83725-cf8a-44e5-8af7-1e7a0eb4107f\" (UID: \"56d83725-cf8a-44e5-8af7-1e7a0eb4107f\") " Nov 03 01:50:45.329790 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:50:45.329627 4939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/56d83725-cf8a-44e5-8af7-1e7a0eb4107f-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/56d83725-cf8a-44e5-8af7-1e7a0eb4107f-tmp-dir\") pod \"56d83725-cf8a-44e5-8af7-1e7a0eb4107f\" (UID: \"56d83725-cf8a-44e5-8af7-1e7a0eb4107f\") " Nov 03 01:50:45.329790 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:50:45.329670 4939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/56d83725-cf8a-44e5-8af7-1e7a0eb4107f-home\" (UniqueName: \"kubernetes.io/empty-dir/56d83725-cf8a-44e5-8af7-1e7a0eb4107f-home\") pod \"56d83725-cf8a-44e5-8af7-1e7a0eb4107f\" (UID: \"56d83725-cf8a-44e5-8af7-1e7a0eb4107f\") " Nov 03 01:50:45.329790 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:50:45.329690 4939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/56d83725-cf8a-44e5-8af7-1e7a0eb4107f-tls-certs\" (UniqueName: \"kubernetes.io/secret/56d83725-cf8a-44e5-8af7-1e7a0eb4107f-tls-certs\") pod \"56d83725-cf8a-44e5-8af7-1e7a0eb4107f\" (UID: \"56d83725-cf8a-44e5-8af7-1e7a0eb4107f\") " Nov 03 01:50:45.329790 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:50:45.329733 4939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/56d83725-cf8a-44e5-8af7-1e7a0eb4107f-kube-api-access-hmr2l\" (UniqueName: \"kubernetes.io/projected/56d83725-cf8a-44e5-8af7-1e7a0eb4107f-kube-api-access-hmr2l\") pod \"56d83725-cf8a-44e5-8af7-1e7a0eb4107f\" (UID: \"56d83725-cf8a-44e5-8af7-1e7a0eb4107f\") " Nov 03 01:50:45.330067 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:50:45.329844 4939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/56d83725-cf8a-44e5-8af7-1e7a0eb4107f-model-cache" pod "56d83725-cf8a-44e5-8af7-1e7a0eb4107f" (UID: "56d83725-cf8a-44e5-8af7-1e7a0eb4107f"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 01:50:45.330067 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:50:45.329961 4939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/56d83725-cf8a-44e5-8af7-1e7a0eb4107f-home" pod "56d83725-cf8a-44e5-8af7-1e7a0eb4107f" (UID: "56d83725-cf8a-44e5-8af7-1e7a0eb4107f"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 01:50:45.330067 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:50:45.329982 4939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/56d83725-cf8a-44e5-8af7-1e7a0eb4107f-tmp-dir" pod "56d83725-cf8a-44e5-8af7-1e7a0eb4107f" (UID: "56d83725-cf8a-44e5-8af7-1e7a0eb4107f"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 01:50:45.330219 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:50:45.330101 4939 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/56d83725-cf8a-44e5-8af7-1e7a0eb4107f-model-cache\") on node \"ip-10-0-140-107.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:50:45.330219 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:50:45.330121 4939 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/56d83725-cf8a-44e5-8af7-1e7a0eb4107f-tmp-dir\") on node \"ip-10-0-140-107.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:50:45.330219 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:50:45.330136 4939 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/56d83725-cf8a-44e5-8af7-1e7a0eb4107f-home\") on node \"ip-10-0-140-107.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:50:45.331765 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:50:45.331739 4939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/56d83725-cf8a-44e5-8af7-1e7a0eb4107f-dshm" pod "56d83725-cf8a-44e5-8af7-1e7a0eb4107f" (UID: "56d83725-cf8a-44e5-8af7-1e7a0eb4107f"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 01:50:45.331901 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:50:45.331854 4939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/56d83725-cf8a-44e5-8af7-1e7a0eb4107f-tls-certs" pod "56d83725-cf8a-44e5-8af7-1e7a0eb4107f" (UID: "56d83725-cf8a-44e5-8af7-1e7a0eb4107f"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 01:50:45.331901 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:50:45.331856 4939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/56d83725-cf8a-44e5-8af7-1e7a0eb4107f-kube-api-access-hmr2l" pod "56d83725-cf8a-44e5-8af7-1e7a0eb4107f" (UID: "56d83725-cf8a-44e5-8af7-1e7a0eb4107f"). InnerVolumeSpecName "kube-api-access-hmr2l". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 01:50:45.431353 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:50:45.431325 4939 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/56d83725-cf8a-44e5-8af7-1e7a0eb4107f-tls-certs\") on node \"ip-10-0-140-107.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:50:45.431353 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:50:45.431351 4939 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hmr2l\" (UniqueName: \"kubernetes.io/projected/56d83725-cf8a-44e5-8af7-1e7a0eb4107f-kube-api-access-hmr2l\") on node \"ip-10-0-140-107.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:50:45.431526 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:50:45.431362 4939 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/56d83725-cf8a-44e5-8af7-1e7a0eb4107f-dshm\") on node \"ip-10-0-140-107.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:50:45.452789 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:50:45.452764 4939 generic.go:356] "Generic (PLEG): container finished" podID="56d83725-cf8a-44e5-8af7-1e7a0eb4107f" containerID="d73fd6ed6b0ac43a3a1706f7bba56e99d434acf898c2534cb01e9a2ef74779c5" exitCode=0 Nov 03 01:50:45.452968 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:50:45.452826 4939 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-077b6796-kserve-55c58b6bcc-6rs9p" Nov 03 01:50:45.452968 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:50:45.452843 4939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-077b6796-kserve-55c58b6bcc-6rs9p" event={"ID":"56d83725-cf8a-44e5-8af7-1e7a0eb4107f","Type":"ContainerDied","Data":"d73fd6ed6b0ac43a3a1706f7bba56e99d434acf898c2534cb01e9a2ef74779c5"} Nov 03 01:50:45.452968 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:50:45.452874 4939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-077b6796-kserve-55c58b6bcc-6rs9p" event={"ID":"56d83725-cf8a-44e5-8af7-1e7a0eb4107f","Type":"ContainerDied","Data":"8dc0bc1cba95ff1b568f600ade22a955a8bba5646a89e36b9698c7563405899d"} Nov 03 01:50:45.452968 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:50:45.452891 4939 scope.go:122] "RemoveContainer" containerID="d73fd6ed6b0ac43a3a1706f7bba56e99d434acf898c2534cb01e9a2ef74779c5" Nov 03 01:50:45.462444 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:50:45.462428 4939 scope.go:122] "RemoveContainer" containerID="d73fd6ed6b0ac43a3a1706f7bba56e99d434acf898c2534cb01e9a2ef74779c5" Nov 03 01:50:45.462713 ip-10-0-140-107 kubenswrapper[4939]: E1103 01:50:45.462694 4939 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d73fd6ed6b0ac43a3a1706f7bba56e99d434acf898c2534cb01e9a2ef74779c5\": container with ID starting with d73fd6ed6b0ac43a3a1706f7bba56e99d434acf898c2534cb01e9a2ef74779c5 not found: ID does not exist" containerID="d73fd6ed6b0ac43a3a1706f7bba56e99d434acf898c2534cb01e9a2ef74779c5" Nov 03 01:50:45.462768 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:50:45.462723 4939 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d73fd6ed6b0ac43a3a1706f7bba56e99d434acf898c2534cb01e9a2ef74779c5"} err="failed to get container status \"d73fd6ed6b0ac43a3a1706f7bba56e99d434acf898c2534cb01e9a2ef74779c5\": rpc error: code = NotFound desc = could not find container \"d73fd6ed6b0ac43a3a1706f7bba56e99d434acf898c2534cb01e9a2ef74779c5\": container with ID starting with d73fd6ed6b0ac43a3a1706f7bba56e99d434acf898c2534cb01e9a2ef74779c5 not found: ID does not exist" Nov 03 01:50:45.475325 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:50:45.475304 4939 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-a-077b6796-kserve-55c58b6bcc-6rs9p"] Nov 03 01:50:45.479078 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:50:45.479048 4939 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["llm/e2e-same-id-a-077b6796-kserve-55c58b6bcc-6rs9p"] Nov 03 01:50:45.714090 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:50:45.714051 4939 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-4pw7v"] Nov 03 01:50:45.866917 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:50:45.866875 4939 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="56d83725-cf8a-44e5-8af7-1e7a0eb4107f" path="/var/lib/kubelet/pods/56d83725-cf8a-44e5-8af7-1e7a0eb4107f/volumes" Nov 03 01:50:45.867321 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:50:45.867307 4939 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8c61f48c-10e8-40c2-9105-375e8f7b1318" path="/var/lib/kubelet/pods/8c61f48c-10e8-40c2-9105-375e8f7b1318/volumes" Nov 03 01:50:46.531932 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:50:46.531306 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:50:47.662098 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:50:47.662046 4939 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-596dc9545f-vm5w5 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.1.20:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 01:50:47.663194 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:50:47.662779 4939 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-vm5w5" podUID="9a335557-68f1-44ab-b673-f9e0a18be917" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.20:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 01:50:48.092591 ip-10-0-140-107 kubenswrapper[4939]: E1103 01:50:48.092543 4939 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-sthjh" containerName="extractor" Nov 03 01:50:49.640766 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:50:49.640716 4939 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-4pw7v"] Nov 03 01:50:53.236619 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:50:53.236589 4939 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-n42s6_65d134ac-8a25-4c67-b6c7-cd752a098daf/console-operator/2.log" Nov 03 01:50:53.239971 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:50:53.239947 4939 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-n42s6_65d134ac-8a25-4c67-b6c7-cd752a098daf/console-operator/2.log" Nov 03 01:50:59.669786 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:50:59.669748 4939 scope.go:122] "RemoveContainer" containerID="5cd2815b53a556d0054cbda9ca75ff3c5d36dd4b81b23f0f53151a4fb36a7a9a" Nov 03 01:51:00.022515 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:51:00.022428 4939 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-mj52m"] Nov 03 01:51:00.026059 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:51:00.026029 4939 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-mj52m"] Nov 03 01:51:01.868113 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:51:01.868060 4939 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4e17db07-898d-45cb-a49a-66ff396d9964" path="/var/lib/kubelet/pods/4e17db07-898d-45cb-a49a-66ff396d9964/volumes" Nov 03 01:51:02.517261 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:51:02.517225 4939 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-4pw7v"] Nov 03 01:51:11.026687 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:51:11.026646 4939 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-tsx87"] Nov 03 01:51:11.029321 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:51:11.029293 4939 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-tsx87"] Nov 03 01:51:11.866920 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:51:11.866884 4939 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="929fd880-1454-43b6-ab02-f8d47fb0c483" path="/var/lib/kubelet/pods/929fd880-1454-43b6-ab02-f8d47fb0c483/volumes" Nov 03 01:51:17.334194 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:51:17.334157 4939 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-7b98bb678c-6q4gn"] Nov 03 01:51:18.030121 ip-10-0-140-107 kubenswrapper[4939]: E1103 01:51:18.030081 4939 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-sthjh" containerName="extractor" Nov 03 01:51:22.334615 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:51:22.334543 4939 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-7b98bb678c-6q4gn" podUID="6dde687f-2bc9-4f1f-9dac-9b7557f0327e" containerName="payload-pre-processing" containerID="cri-o://bd29709ad3952ca1b1796127296c134e3de3dfbf4aaa56ca17f8c8c5843788b8" gracePeriod=25 Nov 03 01:51:22.579193 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:51:22.579166 4939 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-7b98bb678c-6q4gn" Nov 03 01:51:22.658419 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:51:22.658323 4939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6dde687f-2bc9-4f1f-9dac-9b7557f0327e-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6dde687f-2bc9-4f1f-9dac-9b7557f0327e-plugins-config-volume\") pod \"6dde687f-2bc9-4f1f-9dac-9b7557f0327e\" (UID: \"6dde687f-2bc9-4f1f-9dac-9b7557f0327e\") " Nov 03 01:51:22.658419 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:51:22.658367 4939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6dde687f-2bc9-4f1f-9dac-9b7557f0327e-kube-api-access-8l8g4\" (UniqueName: \"kubernetes.io/projected/6dde687f-2bc9-4f1f-9dac-9b7557f0327e-kube-api-access-8l8g4\") pod \"6dde687f-2bc9-4f1f-9dac-9b7557f0327e\" (UID: \"6dde687f-2bc9-4f1f-9dac-9b7557f0327e\") " Nov 03 01:51:22.658790 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:51:22.658766 4939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6dde687f-2bc9-4f1f-9dac-9b7557f0327e-plugins-config-volume" pod "6dde687f-2bc9-4f1f-9dac-9b7557f0327e" (UID: "6dde687f-2bc9-4f1f-9dac-9b7557f0327e"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 01:51:22.660509 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:51:22.660487 4939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6dde687f-2bc9-4f1f-9dac-9b7557f0327e-kube-api-access-8l8g4" pod "6dde687f-2bc9-4f1f-9dac-9b7557f0327e" (UID: "6dde687f-2bc9-4f1f-9dac-9b7557f0327e"). InnerVolumeSpecName "kube-api-access-8l8g4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 01:51:22.726230 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:51:22.726189 4939 generic.go:356] "Generic (PLEG): container finished" podID="6dde687f-2bc9-4f1f-9dac-9b7557f0327e" containerID="bd29709ad3952ca1b1796127296c134e3de3dfbf4aaa56ca17f8c8c5843788b8" exitCode=0 Nov 03 01:51:22.726436 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:51:22.726240 4939 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-7b98bb678c-6q4gn" Nov 03 01:51:22.726436 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:51:22.726267 4939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-7b98bb678c-6q4gn" event={"ID":"6dde687f-2bc9-4f1f-9dac-9b7557f0327e","Type":"ContainerDied","Data":"bd29709ad3952ca1b1796127296c134e3de3dfbf4aaa56ca17f8c8c5843788b8"} Nov 03 01:51:22.726436 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:51:22.726320 4939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-7b98bb678c-6q4gn" event={"ID":"6dde687f-2bc9-4f1f-9dac-9b7557f0327e","Type":"ContainerDied","Data":"26ec763c6b27b76bf345e225d6abcbd02f5686849482a97da4c4527a6fb4411d"} Nov 03 01:51:22.726436 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:51:22.726341 4939 scope.go:122] "RemoveContainer" containerID="bd29709ad3952ca1b1796127296c134e3de3dfbf4aaa56ca17f8c8c5843788b8" Nov 03 01:51:22.735206 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:51:22.735188 4939 scope.go:122] "RemoveContainer" containerID="bd29709ad3952ca1b1796127296c134e3de3dfbf4aaa56ca17f8c8c5843788b8" Nov 03 01:51:22.735499 ip-10-0-140-107 kubenswrapper[4939]: E1103 01:51:22.735479 4939 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bd29709ad3952ca1b1796127296c134e3de3dfbf4aaa56ca17f8c8c5843788b8\": container with ID starting with bd29709ad3952ca1b1796127296c134e3de3dfbf4aaa56ca17f8c8c5843788b8 not found: ID does not exist" containerID="bd29709ad3952ca1b1796127296c134e3de3dfbf4aaa56ca17f8c8c5843788b8" Nov 03 01:51:22.735554 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:51:22.735507 4939 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bd29709ad3952ca1b1796127296c134e3de3dfbf4aaa56ca17f8c8c5843788b8"} err="failed to get container status \"bd29709ad3952ca1b1796127296c134e3de3dfbf4aaa56ca17f8c8c5843788b8\": rpc error: code = NotFound desc = could not find container \"bd29709ad3952ca1b1796127296c134e3de3dfbf4aaa56ca17f8c8c5843788b8\": container with ID starting with bd29709ad3952ca1b1796127296c134e3de3dfbf4aaa56ca17f8c8c5843788b8 not found: ID does not exist" Nov 03 01:51:22.748369 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:51:22.748338 4939 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-7b98bb678c-6q4gn"] Nov 03 01:51:22.749810 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:51:22.749790 4939 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-7b98bb678c-6q4gn"] Nov 03 01:51:22.760116 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:51:22.760094 4939 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6dde687f-2bc9-4f1f-9dac-9b7557f0327e-plugins-config-volume\") on node \"ip-10-0-140-107.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:51:22.760197 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:51:22.760116 4939 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8l8g4\" (UniqueName: \"kubernetes.io/projected/6dde687f-2bc9-4f1f-9dac-9b7557f0327e-kube-api-access-8l8g4\") on node \"ip-10-0-140-107.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:51:23.866373 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:51:23.866340 4939 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6dde687f-2bc9-4f1f-9dac-9b7557f0327e" path="/var/lib/kubelet/pods/6dde687f-2bc9-4f1f-9dac-9b7557f0327e/volumes" Nov 03 01:51:31.405819 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:51:31.405772 4939 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-55fbbbf9dc-qvj7w"] Nov 03 01:51:33.296470 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:51:33.296432 4939 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-754bb5cbcf-77zpm"] Nov 03 01:51:33.296936 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:51:33.296831 4939 cpu_manager.go:414] "containerMap: removing container" podUID="56d83725-cf8a-44e5-8af7-1e7a0eb4107f" containerName="main" Nov 03 01:51:33.296936 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:51:33.296843 4939 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="56d83725-cf8a-44e5-8af7-1e7a0eb4107f" containerName="main" Nov 03 01:51:33.296936 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:51:33.296873 4939 cpu_manager.go:414] "containerMap: removing container" podUID="6dde687f-2bc9-4f1f-9dac-9b7557f0327e" containerName="payload-pre-processing" Nov 03 01:51:33.296936 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:51:33.296878 4939 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6dde687f-2bc9-4f1f-9dac-9b7557f0327e" containerName="payload-pre-processing" Nov 03 01:51:33.297105 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:51:33.296960 4939 memory_manager.go:365] "RemoveStaleState removing state" podUID="6dde687f-2bc9-4f1f-9dac-9b7557f0327e" containerName="payload-pre-processing" Nov 03 01:51:33.297105 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:51:33.296974 4939 memory_manager.go:365] "RemoveStaleState removing state" podUID="56d83725-cf8a-44e5-8af7-1e7a0eb4107f" containerName="main" Nov 03 01:51:33.300105 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:51:33.300083 4939 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-754bb5cbcf-77zpm" Nov 03 01:51:33.308618 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:51:33.308587 4939 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-754bb5cbcf-77zpm"] Nov 03 01:51:33.366674 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:51:33.366644 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8d256a28-b126-4673-bd1e-63aaef592334-plugins-config-volume\") pod \"payload-processing-754bb5cbcf-77zpm\" (UID: \"8d256a28-b126-4673-bd1e-63aaef592334\") " pod="openshift-ingress/payload-processing-754bb5cbcf-77zpm" Nov 03 01:51:33.366816 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:51:33.366681 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mc8x8\" (UniqueName: \"kubernetes.io/projected/8d256a28-b126-4673-bd1e-63aaef592334-kube-api-access-mc8x8\") pod \"payload-processing-754bb5cbcf-77zpm\" (UID: \"8d256a28-b126-4673-bd1e-63aaef592334\") " pod="openshift-ingress/payload-processing-754bb5cbcf-77zpm" Nov 03 01:51:33.463376 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:51:33.463345 4939 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-59b8dd6b64-jcv9p"] Nov 03 01:51:33.467033 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:51:33.467007 4939 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-59b8dd6b64-jcv9p" Nov 03 01:51:33.467376 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:51:33.467352 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8d256a28-b126-4673-bd1e-63aaef592334-plugins-config-volume\") pod \"payload-processing-754bb5cbcf-77zpm\" (UID: \"8d256a28-b126-4673-bd1e-63aaef592334\") " pod="openshift-ingress/payload-processing-754bb5cbcf-77zpm" Nov 03 01:51:33.467441 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:51:33.467395 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mc8x8\" (UniqueName: \"kubernetes.io/projected/8d256a28-b126-4673-bd1e-63aaef592334-kube-api-access-mc8x8\") pod \"payload-processing-754bb5cbcf-77zpm\" (UID: \"8d256a28-b126-4673-bd1e-63aaef592334\") " pod="openshift-ingress/payload-processing-754bb5cbcf-77zpm" Nov 03 01:51:33.468011 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:51:33.467994 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8d256a28-b126-4673-bd1e-63aaef592334-plugins-config-volume\") pod \"payload-processing-754bb5cbcf-77zpm\" (UID: \"8d256a28-b126-4673-bd1e-63aaef592334\") " pod="openshift-ingress/payload-processing-754bb5cbcf-77zpm" Nov 03 01:51:33.475105 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:51:33.475078 4939 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-59b8dd6b64-jcv9p"] Nov 03 01:51:33.480246 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:51:33.480219 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mc8x8\" (UniqueName: \"kubernetes.io/projected/8d256a28-b126-4673-bd1e-63aaef592334-kube-api-access-mc8x8\") pod \"payload-processing-754bb5cbcf-77zpm\" (UID: \"8d256a28-b126-4673-bd1e-63aaef592334\") " pod="openshift-ingress/payload-processing-754bb5cbcf-77zpm" Nov 03 01:51:33.568431 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:51:33.568356 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kbsgh\" (UniqueName: \"kubernetes.io/projected/dfb7b62c-1cf4-4bf0-b5d3-df882d0078d4-kube-api-access-kbsgh\") pod \"payload-pre-processing-59b8dd6b64-jcv9p\" (UID: \"dfb7b62c-1cf4-4bf0-b5d3-df882d0078d4\") " pod="openshift-ingress/payload-pre-processing-59b8dd6b64-jcv9p" Nov 03 01:51:33.568550 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:51:33.568493 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dfb7b62c-1cf4-4bf0-b5d3-df882d0078d4-plugins-config-volume\") pod \"payload-pre-processing-59b8dd6b64-jcv9p\" (UID: \"dfb7b62c-1cf4-4bf0-b5d3-df882d0078d4\") " pod="openshift-ingress/payload-pre-processing-59b8dd6b64-jcv9p" Nov 03 01:51:33.614217 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:51:33.614189 4939 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-754bb5cbcf-77zpm" Nov 03 01:51:33.676493 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:51:33.675470 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kbsgh\" (UniqueName: \"kubernetes.io/projected/dfb7b62c-1cf4-4bf0-b5d3-df882d0078d4-kube-api-access-kbsgh\") pod \"payload-pre-processing-59b8dd6b64-jcv9p\" (UID: \"dfb7b62c-1cf4-4bf0-b5d3-df882d0078d4\") " pod="openshift-ingress/payload-pre-processing-59b8dd6b64-jcv9p" Nov 03 01:51:33.676493 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:51:33.675579 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dfb7b62c-1cf4-4bf0-b5d3-df882d0078d4-plugins-config-volume\") pod \"payload-pre-processing-59b8dd6b64-jcv9p\" (UID: \"dfb7b62c-1cf4-4bf0-b5d3-df882d0078d4\") " pod="openshift-ingress/payload-pre-processing-59b8dd6b64-jcv9p" Nov 03 01:51:33.676493 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:51:33.676413 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dfb7b62c-1cf4-4bf0-b5d3-df882d0078d4-plugins-config-volume\") pod \"payload-pre-processing-59b8dd6b64-jcv9p\" (UID: \"dfb7b62c-1cf4-4bf0-b5d3-df882d0078d4\") " pod="openshift-ingress/payload-pre-processing-59b8dd6b64-jcv9p" Nov 03 01:51:33.684631 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:51:33.684608 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kbsgh\" (UniqueName: \"kubernetes.io/projected/dfb7b62c-1cf4-4bf0-b5d3-df882d0078d4-kube-api-access-kbsgh\") pod \"payload-pre-processing-59b8dd6b64-jcv9p\" (UID: \"dfb7b62c-1cf4-4bf0-b5d3-df882d0078d4\") " pod="openshift-ingress/payload-pre-processing-59b8dd6b64-jcv9p" Nov 03 01:51:33.781091 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:51:33.781063 4939 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-59b8dd6b64-jcv9p" Nov 03 01:51:33.886995 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:51:33.886962 4939 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-754bb5cbcf-77zpm"] Nov 03 01:51:33.889997 ip-10-0-140-107 kubenswrapper[4939]: W1103 01:51:33.889968 4939 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8d256a28_b126_4673_bd1e_63aaef592334.slice/crio-3bec89554180a7910c4a6e27516cfd4b2e7f9c9bd545ff7515ae66217955776e WatchSource:0}: Error finding container 3bec89554180a7910c4a6e27516cfd4b2e7f9c9bd545ff7515ae66217955776e: Status 404 returned error can't find the container with id 3bec89554180a7910c4a6e27516cfd4b2e7f9c9bd545ff7515ae66217955776e Nov 03 01:51:34.053490 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:51:34.053464 4939 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-59b8dd6b64-jcv9p"] Nov 03 01:51:34.055872 ip-10-0-140-107 kubenswrapper[4939]: W1103 01:51:34.055841 4939 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddfb7b62c_1cf4_4bf0_b5d3_df882d0078d4.slice/crio-e0494bd34a2ae9ab1f1156b9833d20c6f2923a2fdc378e41ed81d1e81cde08a4 WatchSource:0}: Error finding container e0494bd34a2ae9ab1f1156b9833d20c6f2923a2fdc378e41ed81d1e81cde08a4: Status 404 returned error can't find the container with id e0494bd34a2ae9ab1f1156b9833d20c6f2923a2fdc378e41ed81d1e81cde08a4 Nov 03 01:51:34.786424 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:51:34.786382 4939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-59b8dd6b64-jcv9p" event={"ID":"dfb7b62c-1cf4-4bf0-b5d3-df882d0078d4","Type":"ContainerStarted","Data":"e650af15b340bedd0648875319e19db62c1172ec74e08e12a9d8f6aae8b81833"} Nov 03 01:51:34.786925 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:51:34.786434 4939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-59b8dd6b64-jcv9p" event={"ID":"dfb7b62c-1cf4-4bf0-b5d3-df882d0078d4","Type":"ContainerStarted","Data":"e0494bd34a2ae9ab1f1156b9833d20c6f2923a2fdc378e41ed81d1e81cde08a4"} Nov 03 01:51:34.786925 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:51:34.786478 4939 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-59b8dd6b64-jcv9p" Nov 03 01:51:34.788027 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:51:34.788003 4939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-754bb5cbcf-77zpm" event={"ID":"8d256a28-b126-4673-bd1e-63aaef592334","Type":"ContainerStarted","Data":"28d7382010e7d06b5c5c6f6507c48e90b627a45a1276338e342a426feba785b6"} Nov 03 01:51:34.788101 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:51:34.788053 4939 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-754bb5cbcf-77zpm" Nov 03 01:51:34.788101 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:51:34.788070 4939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-754bb5cbcf-77zpm" event={"ID":"8d256a28-b126-4673-bd1e-63aaef592334","Type":"ContainerStarted","Data":"3bec89554180a7910c4a6e27516cfd4b2e7f9c9bd545ff7515ae66217955776e"} Nov 03 01:51:34.804191 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:51:34.804141 4939 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-59b8dd6b64-jcv9p" podStartSLOduration=1.804125432 podStartE2EDuration="1.804125432s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 01:51:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 01:51:34.803037571 +0000 UTC m=+4243.067372520" watchObservedRunningTime="2026-11-03 01:51:34.804125432 +0000 UTC m=+4243.068460381" Nov 03 01:51:34.820654 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:51:34.820608 4939 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-754bb5cbcf-77zpm" podStartSLOduration=1.820591618 podStartE2EDuration="1.820591618s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 01:51:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 01:51:34.818042649 +0000 UTC m=+4243.082377595" watchObservedRunningTime="2026-11-03 01:51:34.820591618 +0000 UTC m=+4243.084926568" Nov 03 01:51:36.406837 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:51:36.406778 4939 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-55fbbbf9dc-qvj7w" podUID="7cab38b8-5158-4678-a429-350af5e2f72f" containerName="payload-processing" containerID="cri-o://616536dff201b76ae71b9d490fc4aee724e32dd83b0d45b81bfcdd86208c24db" gracePeriod=25 Nov 03 01:51:36.659234 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:51:36.659176 4939 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-55fbbbf9dc-qvj7w" Nov 03 01:51:36.799215 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:51:36.799177 4939 generic.go:356] "Generic (PLEG): container finished" podID="7cab38b8-5158-4678-a429-350af5e2f72f" containerID="616536dff201b76ae71b9d490fc4aee724e32dd83b0d45b81bfcdd86208c24db" exitCode=0 Nov 03 01:51:36.799403 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:51:36.799224 4939 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-55fbbbf9dc-qvj7w" Nov 03 01:51:36.799403 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:51:36.799259 4939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-55fbbbf9dc-qvj7w" event={"ID":"7cab38b8-5158-4678-a429-350af5e2f72f","Type":"ContainerDied","Data":"616536dff201b76ae71b9d490fc4aee724e32dd83b0d45b81bfcdd86208c24db"} Nov 03 01:51:36.799403 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:51:36.799308 4939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-55fbbbf9dc-qvj7w" event={"ID":"7cab38b8-5158-4678-a429-350af5e2f72f","Type":"ContainerDied","Data":"fe1d1261ba4a7fac93fa31f0a0a52007d46dd64d6ec353e3f71b4b074e57eb5f"} Nov 03 01:51:36.799403 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:51:36.799328 4939 scope.go:122] "RemoveContainer" containerID="616536dff201b76ae71b9d490fc4aee724e32dd83b0d45b81bfcdd86208c24db" Nov 03 01:51:36.807291 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:51:36.807267 4939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7cab38b8-5158-4678-a429-350af5e2f72f-kube-api-access-lpcts\" (UniqueName: \"kubernetes.io/projected/7cab38b8-5158-4678-a429-350af5e2f72f-kube-api-access-lpcts\") pod \"7cab38b8-5158-4678-a429-350af5e2f72f\" (UID: \"7cab38b8-5158-4678-a429-350af5e2f72f\") " Nov 03 01:51:36.807419 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:51:36.807364 4939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7cab38b8-5158-4678-a429-350af5e2f72f-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7cab38b8-5158-4678-a429-350af5e2f72f-plugins-config-volume\") pod \"7cab38b8-5158-4678-a429-350af5e2f72f\" (UID: \"7cab38b8-5158-4678-a429-350af5e2f72f\") " Nov 03 01:51:36.807747 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:51:36.807725 4939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7cab38b8-5158-4678-a429-350af5e2f72f-plugins-config-volume" pod "7cab38b8-5158-4678-a429-350af5e2f72f" (UID: "7cab38b8-5158-4678-a429-350af5e2f72f"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 01:51:36.808253 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:51:36.808230 4939 scope.go:122] "RemoveContainer" containerID="616536dff201b76ae71b9d490fc4aee724e32dd83b0d45b81bfcdd86208c24db" Nov 03 01:51:36.808560 ip-10-0-140-107 kubenswrapper[4939]: E1103 01:51:36.808541 4939 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"616536dff201b76ae71b9d490fc4aee724e32dd83b0d45b81bfcdd86208c24db\": container with ID starting with 616536dff201b76ae71b9d490fc4aee724e32dd83b0d45b81bfcdd86208c24db not found: ID does not exist" containerID="616536dff201b76ae71b9d490fc4aee724e32dd83b0d45b81bfcdd86208c24db" Nov 03 01:51:36.808629 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:51:36.808573 4939 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"616536dff201b76ae71b9d490fc4aee724e32dd83b0d45b81bfcdd86208c24db"} err="failed to get container status \"616536dff201b76ae71b9d490fc4aee724e32dd83b0d45b81bfcdd86208c24db\": rpc error: code = NotFound desc = could not find container \"616536dff201b76ae71b9d490fc4aee724e32dd83b0d45b81bfcdd86208c24db\": container with ID starting with 616536dff201b76ae71b9d490fc4aee724e32dd83b0d45b81bfcdd86208c24db not found: ID does not exist" Nov 03 01:51:36.809424 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:51:36.809405 4939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7cab38b8-5158-4678-a429-350af5e2f72f-kube-api-access-lpcts" pod "7cab38b8-5158-4678-a429-350af5e2f72f" (UID: "7cab38b8-5158-4678-a429-350af5e2f72f"). InnerVolumeSpecName "kube-api-access-lpcts". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 01:51:36.908714 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:51:36.908690 4939 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lpcts\" (UniqueName: \"kubernetes.io/projected/7cab38b8-5158-4678-a429-350af5e2f72f-kube-api-access-lpcts\") on node \"ip-10-0-140-107.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:51:36.908714 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:51:36.908714 4939 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7cab38b8-5158-4678-a429-350af5e2f72f-plugins-config-volume\") on node \"ip-10-0-140-107.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:51:37.121022 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:51:37.120994 4939 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-55fbbbf9dc-qvj7w"] Nov 03 01:51:37.123494 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:51:37.123468 4939 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-55fbbbf9dc-qvj7w"] Nov 03 01:51:37.865942 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:51:37.865887 4939 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7cab38b8-5158-4678-a429-350af5e2f72f" path="/var/lib/kubelet/pods/7cab38b8-5158-4678-a429-350af5e2f72f/volumes" Nov 03 01:51:45.796741 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:51:45.796707 4939 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-59b8dd6b64-jcv9p" Nov 03 01:51:45.797157 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:51:45.796781 4939 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-754bb5cbcf-77zpm" Nov 03 01:51:46.583187 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:51:46.583122 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:51:48.030562 ip-10-0-140-107 kubenswrapper[4939]: E1103 01:51:48.030527 4939 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-sthjh" containerName="extractor" Nov 03 01:51:48.085435 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:51:48.085403 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:51:48.196642 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:51:48.196607 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:51:48.548570 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:51:48.548539 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:51:48.644552 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:51:48.644500 4939 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["models-as-a-service/payload-processing-external-model-5bcd7997-m85fl"] Nov 03 01:51:48.645144 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:51:48.645122 4939 cpu_manager.go:414] "containerMap: removing container" podUID="7cab38b8-5158-4678-a429-350af5e2f72f" containerName="payload-processing" Nov 03 01:51:48.645222 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:51:48.645148 4939 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7cab38b8-5158-4678-a429-350af5e2f72f" containerName="payload-processing" Nov 03 01:51:48.645296 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:51:48.645284 4939 memory_manager.go:365] "RemoveStaleState removing state" podUID="7cab38b8-5158-4678-a429-350af5e2f72f" containerName="payload-processing" Nov 03 01:51:48.649608 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:51:48.649590 4939 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-5bcd7997-m85fl" Nov 03 01:51:48.655485 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:51:48.655460 4939 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"models-as-a-service\"/\"kube-root-ca.crt\"" Nov 03 01:51:48.655625 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:51:48.655513 4939 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"models-as-a-service\"/\"e2e-rotation-external-model-api-key\"" Nov 03 01:51:48.655688 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:51:48.655626 4939 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"models-as-a-service\"/\"payload-processing-external-model-dockercfg-x76ws\"" Nov 03 01:51:48.656339 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:51:48.656308 4939 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"models-as-a-service\"/\"openshift-service-ca.crt\"" Nov 03 01:51:48.656460 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:51:48.656389 4939 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["models-as-a-service/payload-processing-external-model-5bcd7997-m85fl"] Nov 03 01:51:48.656460 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:51:48.656403 4939 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"models-as-a-service\"/\"routing-overlay\"" Nov 03 01:51:48.656606 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:51:48.656448 4939 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"models-as-a-service\"/\"payload-processing-external-model-plugins\"" Nov 03 01:51:48.722805 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:51:48.722765 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/d3a5d4f8-4397-40d2-842e-16ee92fa1063-routing-overlay-volume\") pod \"payload-processing-external-model-5bcd7997-m85fl\" (UID: \"d3a5d4f8-4397-40d2-842e-16ee92fa1063\") " pod="models-as-a-service/payload-processing-external-model-5bcd7997-m85fl" Nov 03 01:51:48.723052 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:51:48.722826 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/d3a5d4f8-4397-40d2-842e-16ee92fa1063-provider-credentials\") pod \"payload-processing-external-model-5bcd7997-m85fl\" (UID: \"d3a5d4f8-4397-40d2-842e-16ee92fa1063\") " pod="models-as-a-service/payload-processing-external-model-5bcd7997-m85fl" Nov 03 01:51:48.723052 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:51:48.722983 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k5f2b\" (UniqueName: \"kubernetes.io/projected/d3a5d4f8-4397-40d2-842e-16ee92fa1063-kube-api-access-k5f2b\") pod \"payload-processing-external-model-5bcd7997-m85fl\" (UID: \"d3a5d4f8-4397-40d2-842e-16ee92fa1063\") " pod="models-as-a-service/payload-processing-external-model-5bcd7997-m85fl" Nov 03 01:51:48.723198 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:51:48.723079 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d3a5d4f8-4397-40d2-842e-16ee92fa1063-plugins-config-volume\") pod \"payload-processing-external-model-5bcd7997-m85fl\" (UID: \"d3a5d4f8-4397-40d2-842e-16ee92fa1063\") " pod="models-as-a-service/payload-processing-external-model-5bcd7997-m85fl" Nov 03 01:51:48.824646 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:51:48.824545 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d3a5d4f8-4397-40d2-842e-16ee92fa1063-plugins-config-volume\") pod \"payload-processing-external-model-5bcd7997-m85fl\" (UID: \"d3a5d4f8-4397-40d2-842e-16ee92fa1063\") " pod="models-as-a-service/payload-processing-external-model-5bcd7997-m85fl" Nov 03 01:51:48.824948 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:51:48.824928 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/d3a5d4f8-4397-40d2-842e-16ee92fa1063-routing-overlay-volume\") pod \"payload-processing-external-model-5bcd7997-m85fl\" (UID: \"d3a5d4f8-4397-40d2-842e-16ee92fa1063\") " pod="models-as-a-service/payload-processing-external-model-5bcd7997-m85fl" Nov 03 01:51:48.825096 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:51:48.825082 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/d3a5d4f8-4397-40d2-842e-16ee92fa1063-provider-credentials\") pod \"payload-processing-external-model-5bcd7997-m85fl\" (UID: \"d3a5d4f8-4397-40d2-842e-16ee92fa1063\") " pod="models-as-a-service/payload-processing-external-model-5bcd7997-m85fl" Nov 03 01:51:48.825230 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:51:48.825217 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k5f2b\" (UniqueName: \"kubernetes.io/projected/d3a5d4f8-4397-40d2-842e-16ee92fa1063-kube-api-access-k5f2b\") pod \"payload-processing-external-model-5bcd7997-m85fl\" (UID: \"d3a5d4f8-4397-40d2-842e-16ee92fa1063\") " pod="models-as-a-service/payload-processing-external-model-5bcd7997-m85fl" Nov 03 01:51:48.825559 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:51:48.825527 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d3a5d4f8-4397-40d2-842e-16ee92fa1063-plugins-config-volume\") pod \"payload-processing-external-model-5bcd7997-m85fl\" (UID: \"d3a5d4f8-4397-40d2-842e-16ee92fa1063\") " pod="models-as-a-service/payload-processing-external-model-5bcd7997-m85fl" Nov 03 01:51:48.826192 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:51:48.826168 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/d3a5d4f8-4397-40d2-842e-16ee92fa1063-routing-overlay-volume\") pod \"payload-processing-external-model-5bcd7997-m85fl\" (UID: \"d3a5d4f8-4397-40d2-842e-16ee92fa1063\") " pod="models-as-a-service/payload-processing-external-model-5bcd7997-m85fl" Nov 03 01:51:48.829236 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:51:48.828616 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/d3a5d4f8-4397-40d2-842e-16ee92fa1063-provider-credentials\") pod \"payload-processing-external-model-5bcd7997-m85fl\" (UID: \"d3a5d4f8-4397-40d2-842e-16ee92fa1063\") " pod="models-as-a-service/payload-processing-external-model-5bcd7997-m85fl" Nov 03 01:51:48.834866 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:51:48.834836 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-k5f2b\" (UniqueName: \"kubernetes.io/projected/d3a5d4f8-4397-40d2-842e-16ee92fa1063-kube-api-access-k5f2b\") pod \"payload-processing-external-model-5bcd7997-m85fl\" (UID: \"d3a5d4f8-4397-40d2-842e-16ee92fa1063\") " pod="models-as-a-service/payload-processing-external-model-5bcd7997-m85fl" Nov 03 01:51:48.965949 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:51:48.965916 4939 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-5bcd7997-m85fl" Nov 03 01:51:49.240370 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:51:49.240331 4939 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["models-as-a-service/payload-processing-external-model-5bcd7997-m85fl"] Nov 03 01:51:49.243691 ip-10-0-140-107 kubenswrapper[4939]: W1103 01:51:49.243649 4939 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd3a5d4f8_4397_40d2_842e_16ee92fa1063.slice/crio-8454883f6d5f09db569f2fc94fe6d99e88c718b7f2487a67633c3238e11ce96b WatchSource:0}: Error finding container 8454883f6d5f09db569f2fc94fe6d99e88c718b7f2487a67633c3238e11ce96b: Status 404 returned error can't find the container with id 8454883f6d5f09db569f2fc94fe6d99e88c718b7f2487a67633c3238e11ce96b Nov 03 01:51:49.605793 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:51:49.605760 4939 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-4pw7v"] Nov 03 01:51:49.867485 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:51:49.867410 4939 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="models-as-a-service/payload-processing-external-model-5bcd7997-m85fl" Nov 03 01:51:49.867485 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:51:49.867463 4939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-5bcd7997-m85fl" event={"ID":"d3a5d4f8-4397-40d2-842e-16ee92fa1063","Type":"ContainerStarted","Data":"331775ffa654343bcf410779833b06e19d14912e9479412f4d6e984d1021a9da"} Nov 03 01:51:49.867485 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:51:49.867484 4939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-5bcd7997-m85fl" event={"ID":"d3a5d4f8-4397-40d2-842e-16ee92fa1063","Type":"ContainerStarted","Data":"8454883f6d5f09db569f2fc94fe6d99e88c718b7f2487a67633c3238e11ce96b"} Nov 03 01:51:49.881809 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:51:49.881745 4939 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="models-as-a-service/payload-processing-external-model-5bcd7997-m85fl" podStartSLOduration=1.881724932 podStartE2EDuration="1.881724932s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 01:51:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 01:51:49.88022909 +0000 UTC m=+4258.144564032" watchObservedRunningTime="2026-11-03 01:51:49.881724932 +0000 UTC m=+4258.146059880" Nov 03 01:51:59.693726 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:51:59.693693 4939 scope.go:122] "RemoveContainer" containerID="cfee89ea1230f87f6286bbdd7ee7fdeaf762a0e9b9603a799389748e5ec7871b" Nov 03 01:51:59.704868 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:51:59.704841 4939 scope.go:122] "RemoveContainer" containerID="da8e79899a06ed26b0203c46c71c2d369e7e005781c0076c214faccfb16e2526" Nov 03 01:51:59.712844 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:51:59.712820 4939 scope.go:122] "RemoveContainer" containerID="1ffc4d6a697b8762c02de05e54a73cc06d03b92cdc7003e36a809e83827fc20f" Nov 03 01:52:00.875805 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:52:00.875774 4939 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="models-as-a-service/payload-processing-external-model-5bcd7997-m85fl" Nov 03 01:52:02.057078 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:52:02.057041 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:52:18.030044 ip-10-0-140-107 kubenswrapper[4939]: E1103 01:52:18.030006 4939 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-sthjh" containerName="extractor" Nov 03 01:52:26.431286 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:52:26.431244 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:52:26.541216 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:52:26.541181 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:52:46.624456 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:52:46.624404 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:52:48.029932 ip-10-0-140-107 kubenswrapper[4939]: E1103 01:52:48.029883 4939 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-sthjh" containerName="extractor" Nov 03 01:52:59.744633 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:52:59.744597 4939 scope.go:122] "RemoveContainer" containerID="30db0d587ed90afcda1a386abc7be6d156738119a3526132b6b58c5dd0b4ad9c" Nov 03 01:53:18.030402 ip-10-0-140-107 kubenswrapper[4939]: E1103 01:53:18.030356 4939 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-sthjh" containerName="extractor" Nov 03 01:53:27.995448 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:53:27.995407 4939 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["models-as-a-service/payload-processing-external-model-5bcd7997-m85fl"] Nov 03 01:53:28.017335 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:53:28.017293 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:53:28.148752 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:53:28.148710 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:53:28.349787 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:53:28.349742 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:53:28.405239 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:53:28.405199 4939 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-4pw7v"] Nov 03 01:53:32.999990 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:53:32.998974 4939 kuberuntime_container.go:898] "Killing container with a grace period" pod="models-as-a-service/payload-processing-external-model-5bcd7997-m85fl" podUID="d3a5d4f8-4397-40d2-842e-16ee92fa1063" containerName="payload-processing" containerID="cri-o://331775ffa654343bcf410779833b06e19d14912e9479412f4d6e984d1021a9da" gracePeriod=25 Nov 03 01:53:33.430929 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:53:33.430704 4939 generic.go:356] "Generic (PLEG): container finished" podID="d3a5d4f8-4397-40d2-842e-16ee92fa1063" containerID="331775ffa654343bcf410779833b06e19d14912e9479412f4d6e984d1021a9da" exitCode=0 Nov 03 01:53:33.430929 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:53:33.430847 4939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-5bcd7997-m85fl" event={"ID":"d3a5d4f8-4397-40d2-842e-16ee92fa1063","Type":"ContainerDied","Data":"331775ffa654343bcf410779833b06e19d14912e9479412f4d6e984d1021a9da"} Nov 03 01:53:33.596918 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:53:33.596880 4939 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-5bcd7997-m85fl" Nov 03 01:53:33.637226 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:53:33.637189 4939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d3a5d4f8-4397-40d2-842e-16ee92fa1063-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d3a5d4f8-4397-40d2-842e-16ee92fa1063-plugins-config-volume\") pod \"d3a5d4f8-4397-40d2-842e-16ee92fa1063\" (UID: \"d3a5d4f8-4397-40d2-842e-16ee92fa1063\") " Nov 03 01:53:33.637435 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:53:33.637371 4939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d3a5d4f8-4397-40d2-842e-16ee92fa1063-kube-api-access-k5f2b\" (UniqueName: \"kubernetes.io/projected/d3a5d4f8-4397-40d2-842e-16ee92fa1063-kube-api-access-k5f2b\") pod \"d3a5d4f8-4397-40d2-842e-16ee92fa1063\" (UID: \"d3a5d4f8-4397-40d2-842e-16ee92fa1063\") " Nov 03 01:53:33.637435 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:53:33.637431 4939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d3a5d4f8-4397-40d2-842e-16ee92fa1063-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/d3a5d4f8-4397-40d2-842e-16ee92fa1063-routing-overlay-volume\") pod \"d3a5d4f8-4397-40d2-842e-16ee92fa1063\" (UID: \"d3a5d4f8-4397-40d2-842e-16ee92fa1063\") " Nov 03 01:53:33.637588 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:53:33.637470 4939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d3a5d4f8-4397-40d2-842e-16ee92fa1063-provider-credentials\" (UniqueName: \"kubernetes.io/projected/d3a5d4f8-4397-40d2-842e-16ee92fa1063-provider-credentials\") pod \"d3a5d4f8-4397-40d2-842e-16ee92fa1063\" (UID: \"d3a5d4f8-4397-40d2-842e-16ee92fa1063\") " Nov 03 01:53:33.637700 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:53:33.637672 4939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d3a5d4f8-4397-40d2-842e-16ee92fa1063-plugins-config-volume" pod "d3a5d4f8-4397-40d2-842e-16ee92fa1063" (UID: "d3a5d4f8-4397-40d2-842e-16ee92fa1063"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 01:53:33.637854 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:53:33.637835 4939 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d3a5d4f8-4397-40d2-842e-16ee92fa1063-plugins-config-volume\") on node \"ip-10-0-140-107.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:53:33.638242 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:53:33.638213 4939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d3a5d4f8-4397-40d2-842e-16ee92fa1063-routing-overlay-volume" pod "d3a5d4f8-4397-40d2-842e-16ee92fa1063" (UID: "d3a5d4f8-4397-40d2-842e-16ee92fa1063"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 01:53:33.639532 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:53:33.639502 4939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d3a5d4f8-4397-40d2-842e-16ee92fa1063-kube-api-access-k5f2b" pod "d3a5d4f8-4397-40d2-842e-16ee92fa1063" (UID: "d3a5d4f8-4397-40d2-842e-16ee92fa1063"). InnerVolumeSpecName "kube-api-access-k5f2b". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 01:53:33.639813 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:53:33.639792 4939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d3a5d4f8-4397-40d2-842e-16ee92fa1063-provider-credentials" pod "d3a5d4f8-4397-40d2-842e-16ee92fa1063" (UID: "d3a5d4f8-4397-40d2-842e-16ee92fa1063"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 01:53:33.689720 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:53:33.689619 4939 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-4pw7v"] Nov 03 01:53:33.739269 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:53:33.739218 4939 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k5f2b\" (UniqueName: \"kubernetes.io/projected/d3a5d4f8-4397-40d2-842e-16ee92fa1063-kube-api-access-k5f2b\") on node \"ip-10-0-140-107.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:53:33.739269 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:53:33.739251 4939 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/d3a5d4f8-4397-40d2-842e-16ee92fa1063-routing-overlay-volume\") on node \"ip-10-0-140-107.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:53:33.739269 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:53:33.739266 4939 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/d3a5d4f8-4397-40d2-842e-16ee92fa1063-provider-credentials\") on node \"ip-10-0-140-107.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:53:34.437794 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:53:34.437763 4939 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-5bcd7997-m85fl" Nov 03 01:53:34.438295 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:53:34.437799 4939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-5bcd7997-m85fl" event={"ID":"d3a5d4f8-4397-40d2-842e-16ee92fa1063","Type":"ContainerDied","Data":"8454883f6d5f09db569f2fc94fe6d99e88c718b7f2487a67633c3238e11ce96b"} Nov 03 01:53:34.438295 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:53:34.437864 4939 scope.go:122] "RemoveContainer" containerID="331775ffa654343bcf410779833b06e19d14912e9479412f4d6e984d1021a9da" Nov 03 01:53:34.455562 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:53:34.455529 4939 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["models-as-a-service/payload-processing-external-model-5bcd7997-m85fl"] Nov 03 01:53:34.458555 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:53:34.458525 4939 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["models-as-a-service/payload-processing-external-model-5bcd7997-m85fl"] Nov 03 01:53:35.878461 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:53:35.878419 4939 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d3a5d4f8-4397-40d2-842e-16ee92fa1063" path="/var/lib/kubelet/pods/d3a5d4f8-4397-40d2-842e-16ee92fa1063/volumes" Nov 03 01:53:37.020765 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:53:37.020725 4939 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-89dc6dff-2ddf6"] Nov 03 01:53:37.032795 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:53:37.032764 4939 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-89dc6dff-2ddf6"] Nov 03 01:53:37.518391 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:53:37.518355 4939 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-4pw7v"] Nov 03 01:53:37.866792 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:53:37.866759 4939 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1d4b3cc0-b712-4e81-b29c-e456c60518d3" path="/var/lib/kubelet/pods/1d4b3cc0-b712-4e81-b29c-e456c60518d3/volumes" Nov 03 01:53:46.400875 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:53:46.400836 4939 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-4pw7v"] Nov 03 01:53:46.671833 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:53:46.671737 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:53:48.030148 ip-10-0-140-107 kubenswrapper[4939]: E1103 01:53:48.030113 4939 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-sthjh" containerName="extractor" Nov 03 01:53:50.922737 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:53:50.922699 4939 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-4pw7v"] Nov 03 01:54:02.827402 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:54:02.827362 4939 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-4pw7v"] Nov 03 01:54:15.500044 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:54:15.500010 4939 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-4pw7v"] Nov 03 01:54:16.977932 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:54:16.977844 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:54:18.318193 ip-10-0-140-107 kubenswrapper[4939]: E1103 01:54:18.318140 4939 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-sthjh" containerName="extractor" Nov 03 01:54:39.981137 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:54:39.981058 4939 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-4pw7v"] Nov 03 01:54:42.434018 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:54:42.433013 4939 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-596dc9545f-9jvbb container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.41:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 01:54:42.434018 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:54:42.433082 4939 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-9jvbb" podUID="512d26b5-8ad9-4cca-8687-cd06d31d8369" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.41:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 01:54:46.719108 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:54:46.719071 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:54:48.029605 ip-10-0-140-107 kubenswrapper[4939]: E1103 01:54:48.029567 4939 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-sthjh" containerName="extractor" Nov 03 01:54:50.187877 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:54:50.187840 4939 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-4pw7v"] Nov 03 01:54:54.018925 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:54:54.018866 4939 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-4pw7v"] Nov 03 01:54:59.774842 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:54:59.774807 4939 scope.go:122] "RemoveContainer" containerID="836888c608e62eb5fb5332f98b84116f883673564b9c72ee2d6d73438dc404e6" Nov 03 01:55:10.883900 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:55:10.883815 4939 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-4pw7v"] Nov 03 01:55:14.555931 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:55:14.544806 4939 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-4pw7v"] Nov 03 01:55:15.518932 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:55:15.518003 4939 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-596dc9545f-v5p6s container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.69:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 01:55:15.518932 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:55:15.518106 4939 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-v5p6s" podUID="e6aa4af1-9a04-4116-9d8a-1d8ef5b04958" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.69:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 01:55:18.029858 ip-10-0-140-107 kubenswrapper[4939]: E1103 01:55:18.029813 4939 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-sthjh" containerName="extractor" Nov 03 01:55:27.013475 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:55:27.013439 4939 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-4pw7v"] Nov 03 01:55:32.027943 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:55:32.018128 4939 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-4pw7v"] Nov 03 01:55:34.368085 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:55:34.368049 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:55:34.621666 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:55:34.621582 4939 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-4pw7v"] Nov 03 01:55:43.298668 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:55:43.298627 4939 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-4pw7v"] Nov 03 01:55:46.759528 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:55:46.759483 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:55:48.029377 ip-10-0-140-107 kubenswrapper[4939]: E1103 01:55:48.029335 4939 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-sthjh" containerName="extractor" Nov 03 01:55:53.265318 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:55:53.265281 4939 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-n42s6_65d134ac-8a25-4c67-b6c7-cd752a098daf/console-operator/2.log" Nov 03 01:55:53.269334 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:55:53.269306 4939 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-n42s6_65d134ac-8a25-4c67-b6c7-cd752a098daf/console-operator/2.log" Nov 03 01:55:57.198945 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:55:57.198890 4939 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-4pw7v"] Nov 03 01:56:07.799060 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:56:07.799025 4939 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-4pw7v"] Nov 03 01:56:11.840931 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:56:11.840476 4939 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-4pw7v"] Nov 03 01:56:18.030352 ip-10-0-140-107 kubenswrapper[4939]: E1103 01:56:18.030313 4939 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-sthjh" containerName="extractor" Nov 03 01:56:24.918591 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:56:24.918550 4939 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-4pw7v"] Nov 03 01:56:28.535362 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:56:28.535318 4939 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-4pw7v"] Nov 03 01:56:31.304244 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:56:31.304208 4939 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-4pw7v"] Nov 03 01:56:46.797094 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:56:46.797019 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:56:48.029874 ip-10-0-140-107 kubenswrapper[4939]: E1103 01:56:48.029837 4939 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-sthjh" containerName="extractor" Nov 03 01:56:49.801644 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:56:49.801599 4939 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-4pw7v"] Nov 03 01:57:00.017991 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:57:00.017948 4939 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-4pw7v"] Nov 03 01:57:18.029454 ip-10-0-140-107 kubenswrapper[4939]: E1103 01:57:18.029412 4939 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-sthjh" containerName="extractor" Nov 03 01:57:25.311806 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:57:25.311769 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:57:25.651721 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:57:25.651683 4939 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-single-main-7fde87-openshift-default-5f8558df65m5nbx"] Nov 03 01:57:25.652211 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:57:25.652191 4939 cpu_manager.go:414] "containerMap: removing container" podUID="d3a5d4f8-4397-40d2-842e-16ee92fa1063" containerName="payload-processing" Nov 03 01:57:25.652319 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:57:25.652213 4939 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d3a5d4f8-4397-40d2-842e-16ee92fa1063" containerName="payload-processing" Nov 03 01:57:25.652381 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:57:25.652355 4939 memory_manager.go:365] "RemoveStaleState removing state" podUID="d3a5d4f8-4397-40d2-842e-16ee92fa1063" containerName="payload-processing" Nov 03 01:57:25.655620 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:57:25.655592 4939 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-single-main-7fde87-openshift-default-5f8558df65m5nbx" Nov 03 01:57:25.669667 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:57:25.669634 4939 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-single-main-7fde87-openshift-default-5f8558df65m5nbx"] Nov 03 01:57:25.669831 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:57:25.669752 4939 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-single-main-7fde87-openshift-default-dockercfg-nd4ps\"" Nov 03 01:57:25.749389 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:57:25.749350 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:57:25.756436 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:57:25.756407 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d7a9f51d-0fa4-4e3d-b431-d9712ce8bad0-istio-podinfo\") pod \"e2e-worker-single-main-7fde87-openshift-default-5f8558df65m5nbx\" (UID: \"d7a9f51d-0fa4-4e3d-b431-d9712ce8bad0\") " pod="openshift-ingress/e2e-worker-single-main-7fde87-openshift-default-5f8558df65m5nbx" Nov 03 01:57:25.756549 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:57:25.756470 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d7a9f51d-0fa4-4e3d-b431-d9712ce8bad0-istiod-ca-cert\") pod \"e2e-worker-single-main-7fde87-openshift-default-5f8558df65m5nbx\" (UID: \"d7a9f51d-0fa4-4e3d-b431-d9712ce8bad0\") " pod="openshift-ingress/e2e-worker-single-main-7fde87-openshift-default-5f8558df65m5nbx" Nov 03 01:57:25.756610 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:57:25.756586 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d7a9f51d-0fa4-4e3d-b431-d9712ce8bad0-workload-socket\") pod \"e2e-worker-single-main-7fde87-openshift-default-5f8558df65m5nbx\" (UID: \"d7a9f51d-0fa4-4e3d-b431-d9712ce8bad0\") " pod="openshift-ingress/e2e-worker-single-main-7fde87-openshift-default-5f8558df65m5nbx" Nov 03 01:57:25.756649 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:57:25.756618 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d7a9f51d-0fa4-4e3d-b431-d9712ce8bad0-workload-certs\") pod \"e2e-worker-single-main-7fde87-openshift-default-5f8558df65m5nbx\" (UID: \"d7a9f51d-0fa4-4e3d-b431-d9712ce8bad0\") " pod="openshift-ingress/e2e-worker-single-main-7fde87-openshift-default-5f8558df65m5nbx" Nov 03 01:57:25.756686 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:57:25.756645 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d7a9f51d-0fa4-4e3d-b431-d9712ce8bad0-istio-data\") pod \"e2e-worker-single-main-7fde87-openshift-default-5f8558df65m5nbx\" (UID: \"d7a9f51d-0fa4-4e3d-b431-d9712ce8bad0\") " pod="openshift-ingress/e2e-worker-single-main-7fde87-openshift-default-5f8558df65m5nbx" Nov 03 01:57:25.756770 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:57:25.756752 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d7a9f51d-0fa4-4e3d-b431-d9712ce8bad0-istio-token\") pod \"e2e-worker-single-main-7fde87-openshift-default-5f8558df65m5nbx\" (UID: \"d7a9f51d-0fa4-4e3d-b431-d9712ce8bad0\") " pod="openshift-ingress/e2e-worker-single-main-7fde87-openshift-default-5f8558df65m5nbx" Nov 03 01:57:25.756819 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:57:25.756779 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d7a9f51d-0fa4-4e3d-b431-d9712ce8bad0-istio-envoy\") pod \"e2e-worker-single-main-7fde87-openshift-default-5f8558df65m5nbx\" (UID: \"d7a9f51d-0fa4-4e3d-b431-d9712ce8bad0\") " pod="openshift-ingress/e2e-worker-single-main-7fde87-openshift-default-5f8558df65m5nbx" Nov 03 01:57:25.756856 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:57:25.756846 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d7a9f51d-0fa4-4e3d-b431-d9712ce8bad0-credential-socket\") pod \"e2e-worker-single-main-7fde87-openshift-default-5f8558df65m5nbx\" (UID: \"d7a9f51d-0fa4-4e3d-b431-d9712ce8bad0\") " pod="openshift-ingress/e2e-worker-single-main-7fde87-openshift-default-5f8558df65m5nbx" Nov 03 01:57:25.756892 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:57:25.756879 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-npr5c\" (UniqueName: \"kubernetes.io/projected/d7a9f51d-0fa4-4e3d-b431-d9712ce8bad0-kube-api-access-npr5c\") pod \"e2e-worker-single-main-7fde87-openshift-default-5f8558df65m5nbx\" (UID: \"d7a9f51d-0fa4-4e3d-b431-d9712ce8bad0\") " pod="openshift-ingress/e2e-worker-single-main-7fde87-openshift-default-5f8558df65m5nbx" Nov 03 01:57:25.858139 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:57:25.858098 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d7a9f51d-0fa4-4e3d-b431-d9712ce8bad0-istio-podinfo\") pod \"e2e-worker-single-main-7fde87-openshift-default-5f8558df65m5nbx\" (UID: \"d7a9f51d-0fa4-4e3d-b431-d9712ce8bad0\") " pod="openshift-ingress/e2e-worker-single-main-7fde87-openshift-default-5f8558df65m5nbx" Nov 03 01:57:25.858387 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:57:25.858177 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d7a9f51d-0fa4-4e3d-b431-d9712ce8bad0-istiod-ca-cert\") pod \"e2e-worker-single-main-7fde87-openshift-default-5f8558df65m5nbx\" (UID: \"d7a9f51d-0fa4-4e3d-b431-d9712ce8bad0\") " pod="openshift-ingress/e2e-worker-single-main-7fde87-openshift-default-5f8558df65m5nbx" Nov 03 01:57:25.858387 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:57:25.858222 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d7a9f51d-0fa4-4e3d-b431-d9712ce8bad0-workload-socket\") pod \"e2e-worker-single-main-7fde87-openshift-default-5f8558df65m5nbx\" (UID: \"d7a9f51d-0fa4-4e3d-b431-d9712ce8bad0\") " pod="openshift-ingress/e2e-worker-single-main-7fde87-openshift-default-5f8558df65m5nbx" Nov 03 01:57:25.858387 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:57:25.858249 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d7a9f51d-0fa4-4e3d-b431-d9712ce8bad0-workload-certs\") pod \"e2e-worker-single-main-7fde87-openshift-default-5f8558df65m5nbx\" (UID: \"d7a9f51d-0fa4-4e3d-b431-d9712ce8bad0\") " pod="openshift-ingress/e2e-worker-single-main-7fde87-openshift-default-5f8558df65m5nbx" Nov 03 01:57:25.858387 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:57:25.858289 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d7a9f51d-0fa4-4e3d-b431-d9712ce8bad0-istio-data\") pod \"e2e-worker-single-main-7fde87-openshift-default-5f8558df65m5nbx\" (UID: \"d7a9f51d-0fa4-4e3d-b431-d9712ce8bad0\") " pod="openshift-ingress/e2e-worker-single-main-7fde87-openshift-default-5f8558df65m5nbx" Nov 03 01:57:25.858630 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:57:25.858429 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d7a9f51d-0fa4-4e3d-b431-d9712ce8bad0-istio-token\") pod \"e2e-worker-single-main-7fde87-openshift-default-5f8558df65m5nbx\" (UID: \"d7a9f51d-0fa4-4e3d-b431-d9712ce8bad0\") " pod="openshift-ingress/e2e-worker-single-main-7fde87-openshift-default-5f8558df65m5nbx" Nov 03 01:57:25.858630 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:57:25.858466 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d7a9f51d-0fa4-4e3d-b431-d9712ce8bad0-istio-envoy\") pod \"e2e-worker-single-main-7fde87-openshift-default-5f8558df65m5nbx\" (UID: \"d7a9f51d-0fa4-4e3d-b431-d9712ce8bad0\") " pod="openshift-ingress/e2e-worker-single-main-7fde87-openshift-default-5f8558df65m5nbx" Nov 03 01:57:25.858630 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:57:25.858513 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d7a9f51d-0fa4-4e3d-b431-d9712ce8bad0-credential-socket\") pod \"e2e-worker-single-main-7fde87-openshift-default-5f8558df65m5nbx\" (UID: \"d7a9f51d-0fa4-4e3d-b431-d9712ce8bad0\") " pod="openshift-ingress/e2e-worker-single-main-7fde87-openshift-default-5f8558df65m5nbx" Nov 03 01:57:25.858630 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:57:25.858563 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-npr5c\" (UniqueName: \"kubernetes.io/projected/d7a9f51d-0fa4-4e3d-b431-d9712ce8bad0-kube-api-access-npr5c\") pod \"e2e-worker-single-main-7fde87-openshift-default-5f8558df65m5nbx\" (UID: \"d7a9f51d-0fa4-4e3d-b431-d9712ce8bad0\") " pod="openshift-ingress/e2e-worker-single-main-7fde87-openshift-default-5f8558df65m5nbx" Nov 03 01:57:25.858875 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:57:25.858841 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:57:25.859224 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:57:25.859200 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d7a9f51d-0fa4-4e3d-b431-d9712ce8bad0-workload-certs\") pod \"e2e-worker-single-main-7fde87-openshift-default-5f8558df65m5nbx\" (UID: \"d7a9f51d-0fa4-4e3d-b431-d9712ce8bad0\") " pod="openshift-ingress/e2e-worker-single-main-7fde87-openshift-default-5f8558df65m5nbx" Nov 03 01:57:25.859380 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:57:25.859255 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d7a9f51d-0fa4-4e3d-b431-d9712ce8bad0-istio-data\") pod \"e2e-worker-single-main-7fde87-openshift-default-5f8558df65m5nbx\" (UID: \"d7a9f51d-0fa4-4e3d-b431-d9712ce8bad0\") " pod="openshift-ingress/e2e-worker-single-main-7fde87-openshift-default-5f8558df65m5nbx" Nov 03 01:57:25.859380 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:57:25.859268 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d7a9f51d-0fa4-4e3d-b431-d9712ce8bad0-istiod-ca-cert\") pod \"e2e-worker-single-main-7fde87-openshift-default-5f8558df65m5nbx\" (UID: \"d7a9f51d-0fa4-4e3d-b431-d9712ce8bad0\") " pod="openshift-ingress/e2e-worker-single-main-7fde87-openshift-default-5f8558df65m5nbx" Nov 03 01:57:25.859380 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:57:25.859312 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d7a9f51d-0fa4-4e3d-b431-d9712ce8bad0-workload-socket\") pod \"e2e-worker-single-main-7fde87-openshift-default-5f8558df65m5nbx\" (UID: \"d7a9f51d-0fa4-4e3d-b431-d9712ce8bad0\") " pod="openshift-ingress/e2e-worker-single-main-7fde87-openshift-default-5f8558df65m5nbx" Nov 03 01:57:25.859380 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:57:25.859329 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d7a9f51d-0fa4-4e3d-b431-d9712ce8bad0-credential-socket\") pod \"e2e-worker-single-main-7fde87-openshift-default-5f8558df65m5nbx\" (UID: \"d7a9f51d-0fa4-4e3d-b431-d9712ce8bad0\") " pod="openshift-ingress/e2e-worker-single-main-7fde87-openshift-default-5f8558df65m5nbx" Nov 03 01:57:25.861135 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:57:25.861113 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d7a9f51d-0fa4-4e3d-b431-d9712ce8bad0-istio-envoy\") pod \"e2e-worker-single-main-7fde87-openshift-default-5f8558df65m5nbx\" (UID: \"d7a9f51d-0fa4-4e3d-b431-d9712ce8bad0\") " pod="openshift-ingress/e2e-worker-single-main-7fde87-openshift-default-5f8558df65m5nbx" Nov 03 01:57:25.861266 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:57:25.861144 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d7a9f51d-0fa4-4e3d-b431-d9712ce8bad0-istio-podinfo\") pod \"e2e-worker-single-main-7fde87-openshift-default-5f8558df65m5nbx\" (UID: \"d7a9f51d-0fa4-4e3d-b431-d9712ce8bad0\") " pod="openshift-ingress/e2e-worker-single-main-7fde87-openshift-default-5f8558df65m5nbx" Nov 03 01:57:25.867046 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:57:25.867025 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-npr5c\" (UniqueName: \"kubernetes.io/projected/d7a9f51d-0fa4-4e3d-b431-d9712ce8bad0-kube-api-access-npr5c\") pod \"e2e-worker-single-main-7fde87-openshift-default-5f8558df65m5nbx\" (UID: \"d7a9f51d-0fa4-4e3d-b431-d9712ce8bad0\") " pod="openshift-ingress/e2e-worker-single-main-7fde87-openshift-default-5f8558df65m5nbx" Nov 03 01:57:25.867301 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:57:25.867281 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d7a9f51d-0fa4-4e3d-b431-d9712ce8bad0-istio-token\") pod \"e2e-worker-single-main-7fde87-openshift-default-5f8558df65m5nbx\" (UID: \"d7a9f51d-0fa4-4e3d-b431-d9712ce8bad0\") " pod="openshift-ingress/e2e-worker-single-main-7fde87-openshift-default-5f8558df65m5nbx" Nov 03 01:57:25.971713 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:57:25.971631 4939 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-single-main-7fde87-openshift-default-5f8558df65m5nbx" Nov 03 01:57:26.249539 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:57:26.249508 4939 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-single-main-7fde87-openshift-default-5f8558df65m5nbx"] Nov 03 01:57:26.251445 ip-10-0-140-107 kubenswrapper[4939]: W1103 01:57:26.251414 4939 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd7a9f51d_0fa4_4e3d_b431_d9712ce8bad0.slice/crio-1d9772d7165e12f6738e51dea97f362fec462690fc11905b5b85e42a9a68de27 WatchSource:0}: Error finding container 1d9772d7165e12f6738e51dea97f362fec462690fc11905b5b85e42a9a68de27: Status 404 returned error can't find the container with id 1d9772d7165e12f6738e51dea97f362fec462690fc11905b5b85e42a9a68de27 Nov 03 01:57:26.253579 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:57:26.253540 4939 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 01:57:26.253694 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:57:26.253626 4939 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 01:57:26.253694 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:57:26.253671 4939 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 01:57:26.991185 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:57:26.991147 4939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-7fde87-openshift-default-5f8558df65m5nbx" event={"ID":"d7a9f51d-0fa4-4e3d-b431-d9712ce8bad0","Type":"ContainerStarted","Data":"a01a0aa2fc01e4697197f293390e85686f18395c27ab30299df3e590f2416dae"} Nov 03 01:57:26.991672 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:57:26.991200 4939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-7fde87-openshift-default-5f8558df65m5nbx" event={"ID":"d7a9f51d-0fa4-4e3d-b431-d9712ce8bad0","Type":"ContainerStarted","Data":"1d9772d7165e12f6738e51dea97f362fec462690fc11905b5b85e42a9a68de27"} Nov 03 01:57:27.015748 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:57:27.015690 4939 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-single-main-7fde87-openshift-default-5f8558df65m5nbx" podStartSLOduration=2.015675078 podStartE2EDuration="2.015675078s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 01:57:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 01:57:27.013844649 +0000 UTC m=+4595.278179609" watchObservedRunningTime="2026-11-03 01:57:27.015675078 +0000 UTC m=+4595.280010023" Nov 03 01:57:27.972829 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:57:27.972798 4939 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-single-main-7fde87-openshift-default-5f8558df65m5nbx" Nov 03 01:57:27.977641 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:57:27.977615 4939 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-single-main-7fde87-openshift-default-5f8558df65m5nbx" Nov 03 01:57:27.996286 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:57:27.996256 4939 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-single-main-7fde87-openshift-default-5f8558df65m5nbx" Nov 03 01:57:27.997148 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:57:27.997129 4939 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-single-main-7fde87-openshift-default-5f8558df65m5nbx" Nov 03 01:57:29.194099 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:57:29.194053 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:57:29.308858 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:57:29.308811 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:57:30.396079 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:57:30.396048 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:57:30.456501 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:57:30.456464 4939 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-7fde87-7b7c5wmrlg"] Nov 03 01:57:30.460318 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:57:30.460297 4939 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-7fde87-7b7c5wmrlg" Nov 03 01:57:30.469612 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:57:30.469586 4939 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-single-main-7fde87\"" Nov 03 01:57:30.471417 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:57:30.471396 4939 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-single-m-bf3a2bea-dockercfg-d2pfg\"" Nov 03 01:57:30.473837 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:57:30.473814 4939 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-7fde87-7b7c5wmrlg"] Nov 03 01:57:30.547262 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:57:30.544640 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:57:30.609893 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:57:30.609835 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fr6rk\" (UniqueName: \"kubernetes.io/projected/fee4ebdb-2ebb-4918-ab3f-f9fbc049880d-kube-api-access-fr6rk\") pod \"payload-pre-processing-e2e-worker-single-main-7fde87-7b7c5wmrlg\" (UID: \"fee4ebdb-2ebb-4918-ab3f-f9fbc049880d\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-7fde87-7b7c5wmrlg" Nov 03 01:57:30.610143 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:57:30.610005 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fee4ebdb-2ebb-4918-ab3f-f9fbc049880d-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-single-main-7fde87-7b7c5wmrlg\" (UID: \"fee4ebdb-2ebb-4918-ab3f-f9fbc049880d\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-7fde87-7b7c5wmrlg" Nov 03 01:57:30.711333 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:57:30.711242 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fr6rk\" (UniqueName: \"kubernetes.io/projected/fee4ebdb-2ebb-4918-ab3f-f9fbc049880d-kube-api-access-fr6rk\") pod \"payload-pre-processing-e2e-worker-single-main-7fde87-7b7c5wmrlg\" (UID: \"fee4ebdb-2ebb-4918-ab3f-f9fbc049880d\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-7fde87-7b7c5wmrlg" Nov 03 01:57:30.711559 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:57:30.711406 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fee4ebdb-2ebb-4918-ab3f-f9fbc049880d-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-single-main-7fde87-7b7c5wmrlg\" (UID: \"fee4ebdb-2ebb-4918-ab3f-f9fbc049880d\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-7fde87-7b7c5wmrlg" Nov 03 01:57:30.712185 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:57:30.712161 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fee4ebdb-2ebb-4918-ab3f-f9fbc049880d-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-single-main-7fde87-7b7c5wmrlg\" (UID: \"fee4ebdb-2ebb-4918-ab3f-f9fbc049880d\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-7fde87-7b7c5wmrlg" Nov 03 01:57:30.719763 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:57:30.719735 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fr6rk\" (UniqueName: \"kubernetes.io/projected/fee4ebdb-2ebb-4918-ab3f-f9fbc049880d-kube-api-access-fr6rk\") pod \"payload-pre-processing-e2e-worker-single-main-7fde87-7b7c5wmrlg\" (UID: \"fee4ebdb-2ebb-4918-ab3f-f9fbc049880d\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-7fde87-7b7c5wmrlg" Nov 03 01:57:30.774793 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:57:30.774759 4939 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-7fde87-7b7c5wmrlg" Nov 03 01:57:31.054813 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:57:31.054786 4939 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-7fde87-7b7c5wmrlg"] Nov 03 01:57:31.056979 ip-10-0-140-107 kubenswrapper[4939]: W1103 01:57:31.056950 4939 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfee4ebdb_2ebb_4918_ab3f_f9fbc049880d.slice/crio-3cd5192ff9e5ebc92d058f001a8e24792a1bd1ebf1f904c10353d25cce2e6414 WatchSource:0}: Error finding container 3cd5192ff9e5ebc92d058f001a8e24792a1bd1ebf1f904c10353d25cce2e6414: Status 404 returned error can't find the container with id 3cd5192ff9e5ebc92d058f001a8e24792a1bd1ebf1f904c10353d25cce2e6414 Nov 03 01:57:31.410167 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:57:31.410135 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:57:32.016984 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:57:32.016944 4939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-7fde87-7b7c5wmrlg" event={"ID":"fee4ebdb-2ebb-4918-ab3f-f9fbc049880d","Type":"ContainerStarted","Data":"21fb07471c0b2097957e4de8acf7946e250bb1579e36a3d538a40a2d6836ad93"} Nov 03 01:57:32.017185 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:57:32.017015 4939 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-7fde87-7b7c5wmrlg" Nov 03 01:57:32.017185 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:57:32.017043 4939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-7fde87-7b7c5wmrlg" event={"ID":"fee4ebdb-2ebb-4918-ab3f-f9fbc049880d","Type":"ContainerStarted","Data":"3cd5192ff9e5ebc92d058f001a8e24792a1bd1ebf1f904c10353d25cce2e6414"} Nov 03 01:57:32.038545 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:57:32.038485 4939 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-7fde87-7b7c5wmrlg" podStartSLOduration=2.038471936 podStartE2EDuration="2.038471936s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 01:57:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 01:57:32.036331334 +0000 UTC m=+4600.300666280" watchObservedRunningTime="2026-11-03 01:57:32.038471936 +0000 UTC m=+4600.302806878" Nov 03 01:57:32.128903 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:57:32.128872 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:57:32.794575 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:57:32.794540 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:57:32.928556 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:57:32.928510 4939 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-4pw7v"] Nov 03 01:57:33.634751 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:57:33.634717 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:57:34.305790 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:57:34.305752 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:57:35.021015 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:57:35.020978 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:57:35.776628 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:57:35.776588 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:57:36.524734 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:57:36.524695 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:57:37.217344 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:57:37.217312 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:57:37.975819 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:57:37.975779 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:57:38.720937 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:57:38.720887 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:57:39.591976 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:57:39.591918 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:57:40.302668 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:57:40.302631 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:57:40.956527 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:57:40.956490 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:57:41.674982 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:57:41.674944 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:57:42.074196 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:57:42.074162 4939 generic.go:356] "Generic (PLEG): container finished" podID="d7a9f51d-0fa4-4e3d-b431-d9712ce8bad0" containerID="a01a0aa2fc01e4697197f293390e85686f18395c27ab30299df3e590f2416dae" exitCode=0 Nov 03 01:57:42.074400 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:57:42.074229 4939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-7fde87-openshift-default-5f8558df65m5nbx" event={"ID":"d7a9f51d-0fa4-4e3d-b431-d9712ce8bad0","Type":"ContainerDied","Data":"a01a0aa2fc01e4697197f293390e85686f18395c27ab30299df3e590f2416dae"} Nov 03 01:57:42.074662 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:57:42.074636 4939 scope.go:122] "RemoveContainer" containerID="a01a0aa2fc01e4697197f293390e85686f18395c27ab30299df3e590f2416dae" Nov 03 01:57:42.075865 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:57:42.075835 4939 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 01:57:42.075953 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:57:42.075904 4939 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 01:57:42.076009 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:57:42.075951 4939 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 01:57:42.496485 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:57:42.496449 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:57:42.997245 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:57:42.997198 4939 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-single-main-7fde87-openshift-default-5f8558df65m5nbx" Nov 03 01:57:43.031312 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:57:43.031284 4939 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-7fde87-7b7c5wmrlg" Nov 03 01:57:43.097557 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:57:43.097508 4939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-7fde87-openshift-default-5f8558df65m5nbx" event={"ID":"d7a9f51d-0fa4-4e3d-b431-d9712ce8bad0","Type":"ContainerStarted","Data":"8b98e8c5cae2374068da28d8fe5e716ca825f1a93455b56f3261515f9be8864b"} Nov 03 01:57:43.244362 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:57:43.244329 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:57:43.972544 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:57:43.972496 4939 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-single-main-7fde87-openshift-default-5f8558df65m5nbx" Nov 03 01:57:43.973151 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:57:43.973126 4939 patch_prober.go:29] interesting pod/e2e-worker-single-main-7fde87-openshift-default-5f8558df65m5nbx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.116:15021/healthz/ready\": dial tcp 10.132.1.116:15021: connect: connection refused" start-of-body= Nov 03 01:57:43.973273 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:57:43.973194 4939 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-single-main-7fde87-openshift-default-5f8558df65m5nbx" podUID="d7a9f51d-0fa4-4e3d-b431-d9712ce8bad0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.116:15021/healthz/ready\": dial tcp 10.132.1.116:15021: connect: connection refused" Nov 03 01:57:43.999624 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:57:43.999600 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:57:44.227153 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:57:44.227068 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:57:44.816348 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:57:44.816313 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:57:44.938500 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:57:44.938463 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:57:44.972519 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:57:44.972480 4939 patch_prober.go:29] interesting pod/e2e-worker-single-main-7fde87-openshift-default-5f8558df65m5nbx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.116:15021/healthz/ready\": dial tcp 10.132.1.116:15021: connect: connection refused" start-of-body= Nov 03 01:57:44.972716 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:57:44.972582 4939 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-single-main-7fde87-openshift-default-5f8558df65m5nbx" podUID="d7a9f51d-0fa4-4e3d-b431-d9712ce8bad0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.116:15021/healthz/ready\": dial tcp 10.132.1.116:15021: connect: connection refused" Nov 03 01:57:45.058508 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:57:45.058473 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:57:45.657667 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:57:45.657614 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:57:45.972507 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:57:45.972401 4939 patch_prober.go:29] interesting pod/e2e-worker-single-main-7fde87-openshift-default-5f8558df65m5nbx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.116:15021/healthz/ready\": dial tcp 10.132.1.116:15021: connect: connection refused" start-of-body= Nov 03 01:57:45.972507 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:57:45.972478 4939 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-single-main-7fde87-openshift-default-5f8558df65m5nbx" podUID="d7a9f51d-0fa4-4e3d-b431-d9712ce8bad0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.116:15021/healthz/ready\": dial tcp 10.132.1.116:15021: connect: connection refused" Nov 03 01:57:46.317471 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:57:46.317376 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:57:46.472183 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:57:46.472147 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:57:46.839994 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:57:46.839944 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:57:46.973665 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:57:46.973362 4939 patch_prober.go:29] interesting pod/e2e-worker-single-main-7fde87-openshift-default-5f8558df65m5nbx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.116:15021/healthz/ready\": dial tcp 10.132.1.116:15021: connect: connection refused" start-of-body= Nov 03 01:57:46.973665 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:57:46.973448 4939 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-single-main-7fde87-openshift-default-5f8558df65m5nbx" podUID="d7a9f51d-0fa4-4e3d-b431-d9712ce8bad0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.116:15021/healthz/ready\": dial tcp 10.132.1.116:15021: connect: connection refused" Nov 03 01:57:47.072307 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:57:47.072274 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:57:47.820738 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:57:47.820703 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:57:47.972453 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:57:47.972412 4939 patch_prober.go:29] interesting pod/e2e-worker-single-main-7fde87-openshift-default-5f8558df65m5nbx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.116:15021/healthz/ready\": dial tcp 10.132.1.116:15021: connect: connection refused" start-of-body= Nov 03 01:57:47.972651 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:57:47.972479 4939 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-single-main-7fde87-openshift-default-5f8558df65m5nbx" podUID="d7a9f51d-0fa4-4e3d-b431-d9712ce8bad0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.116:15021/healthz/ready\": dial tcp 10.132.1.116:15021: connect: connection refused" Nov 03 01:57:48.006323 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:57:48.006287 4939 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-4pw7v"] Nov 03 01:57:48.048463 ip-10-0-140-107 kubenswrapper[4939]: E1103 01:57:48.048419 4939 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-sthjh" containerName="extractor" Nov 03 01:57:48.976947 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:57:48.976902 4939 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-single-main-7fde87-openshift-default-5f8558df65m5nbx" Nov 03 01:57:48.977373 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:57:48.977327 4939 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-single-main-7fde87-openshift-default-5f8558df65m5nbx" Nov 03 01:57:48.977998 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:57:48.977969 4939 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-single-main-7fde87-openshift-default-5f8558df65m5nbx" Nov 03 01:57:49.020031 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:57:49.020007 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:57:49.381608 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:57:49.381576 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:57:50.000131 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:57:50.000097 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:57:50.740116 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:57:50.740056 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:57:51.315356 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:57:51.315323 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:57:52.002268 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:57:52.002230 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:57:52.688232 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:57:52.688195 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:57:53.350688 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:57:53.350658 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:57:54.026941 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:57:54.026506 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:57:54.693777 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:57:54.693743 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:57:55.407859 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:57:55.407791 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:57:56.079499 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:57:56.079463 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:57:56.685434 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:57:56.685397 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:57:57.356201 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:57:57.356164 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:57:58.006100 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:57:58.006066 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:57:58.667178 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:57:58.667139 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:57:59.404076 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:57:59.404039 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:58:00.098365 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:58:00.098328 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:58:00.717240 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:58:00.717199 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:58:01.382771 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:58:01.382727 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:58:02.089730 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:58:02.089698 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:58:02.699423 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:58:02.699386 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:58:03.468473 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:58:03.468422 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:58:03.598256 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:58:03.598217 4939 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-4pw7v"] Nov 03 01:58:04.290345 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:58:04.290304 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:58:04.992967 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:58:04.992886 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:58:05.682974 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:58:05.682932 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:58:06.392272 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:58:06.392236 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:58:07.037326 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:58:07.037294 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:58:07.688143 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:58:07.688103 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:58:08.390317 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:58:08.390277 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:58:09.017707 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:58:09.017670 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:58:09.687406 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:58:09.687371 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:58:10.304418 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:58:10.304363 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:58:10.997065 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:58:10.997033 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:58:11.797334 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:58:11.797297 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:58:12.490156 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:58:12.490108 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:58:13.111117 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:58:13.111084 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:58:13.801687 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:58:13.801640 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:58:14.514295 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:58:14.514247 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:58:15.191328 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:58:15.191295 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:58:15.882385 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:58:15.882351 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:58:15.992074 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:58:15.992037 4939 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6876bd20-qn6mv"] Nov 03 01:58:15.995829 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:58:15.995811 4939 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6876bd20-qn6mv" Nov 03 01:58:16.003740 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:58:16.003711 4939 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 03 01:58:16.004177 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:58:16.004159 4939 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 01:58:16.004343 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:58:16.004226 4939 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 03 01:58:16.004343 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:58:16.004333 4939 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-swfcr\"" Nov 03 01:58:16.007758 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:58:16.007727 4939 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6876bd20-qn6mv"] Nov 03 01:58:16.037984 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:58:16.037959 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fv4pv\" (UniqueName: \"kubernetes.io/projected/d6b552a0-b9c4-42cc-a0b4-cee73867e470-kube-api-access-fv4pv\") pod \"maas-api-revoke-sub-6876bd20-qn6mv\" (UID: \"d6b552a0-b9c4-42cc-a0b4-cee73867e470\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6876bd20-qn6mv" Nov 03 01:58:16.038096 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:58:16.038008 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d6b552a0-b9c4-42cc-a0b4-cee73867e470-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6876bd20-qn6mv\" (UID: \"d6b552a0-b9c4-42cc-a0b4-cee73867e470\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6876bd20-qn6mv" Nov 03 01:58:16.038184 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:58:16.038157 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d6b552a0-b9c4-42cc-a0b4-cee73867e470-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6876bd20-qn6mv\" (UID: \"d6b552a0-b9c4-42cc-a0b4-cee73867e470\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6876bd20-qn6mv" Nov 03 01:58:16.139063 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:58:16.138990 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d6b552a0-b9c4-42cc-a0b4-cee73867e470-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6876bd20-qn6mv\" (UID: \"d6b552a0-b9c4-42cc-a0b4-cee73867e470\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6876bd20-qn6mv" Nov 03 01:58:16.139189 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:58:16.139089 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fv4pv\" (UniqueName: \"kubernetes.io/projected/d6b552a0-b9c4-42cc-a0b4-cee73867e470-kube-api-access-fv4pv\") pod \"maas-api-revoke-sub-6876bd20-qn6mv\" (UID: \"d6b552a0-b9c4-42cc-a0b4-cee73867e470\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6876bd20-qn6mv" Nov 03 01:58:16.139189 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:58:16.139135 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d6b552a0-b9c4-42cc-a0b4-cee73867e470-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6876bd20-qn6mv\" (UID: \"d6b552a0-b9c4-42cc-a0b4-cee73867e470\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6876bd20-qn6mv" Nov 03 01:58:16.139849 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:58:16.139823 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d6b552a0-b9c4-42cc-a0b4-cee73867e470-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6876bd20-qn6mv\" (UID: \"d6b552a0-b9c4-42cc-a0b4-cee73867e470\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6876bd20-qn6mv" Nov 03 01:58:16.141588 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:58:16.141562 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d6b552a0-b9c4-42cc-a0b4-cee73867e470-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6876bd20-qn6mv\" (UID: \"d6b552a0-b9c4-42cc-a0b4-cee73867e470\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6876bd20-qn6mv" Nov 03 01:58:16.148119 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:58:16.148098 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fv4pv\" (UniqueName: \"kubernetes.io/projected/d6b552a0-b9c4-42cc-a0b4-cee73867e470-kube-api-access-fv4pv\") pod \"maas-api-revoke-sub-6876bd20-qn6mv\" (UID: \"d6b552a0-b9c4-42cc-a0b4-cee73867e470\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6876bd20-qn6mv" Nov 03 01:58:16.316579 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:58:16.316542 4939 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6876bd20-qn6mv" Nov 03 01:58:16.593089 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:58:16.593064 4939 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6876bd20-qn6mv"] Nov 03 01:58:16.595388 ip-10-0-140-107 kubenswrapper[4939]: W1103 01:58:16.595345 4939 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd6b552a0_b9c4_42cc_a0b4_cee73867e470.slice/crio-9488a8b0f0a2a3438b7a9eb74d716db7795c5fbe0970bd6d767a16965465d090 WatchSource:0}: Error finding container 9488a8b0f0a2a3438b7a9eb74d716db7795c5fbe0970bd6d767a16965465d090: Status 404 returned error can't find the container with id 9488a8b0f0a2a3438b7a9eb74d716db7795c5fbe0970bd6d767a16965465d090 Nov 03 01:58:16.620995 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:58:16.620970 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:58:17.269738 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:58:17.269645 4939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6876bd20-qn6mv" event={"ID":"d6b552a0-b9c4-42cc-a0b4-cee73867e470","Type":"ContainerStarted","Data":"150a640998bd75d7365b66716a268353c08ac534fcf2c8c414498f72e5188d00"} Nov 03 01:58:17.269738 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:58:17.269707 4939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6876bd20-qn6mv" event={"ID":"d6b552a0-b9c4-42cc-a0b4-cee73867e470","Type":"ContainerStarted","Data":"9488a8b0f0a2a3438b7a9eb74d716db7795c5fbe0970bd6d767a16965465d090"} Nov 03 01:58:17.286946 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:58:17.286886 4939 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6876bd20-qn6mv" podStartSLOduration=2.286873432 podStartE2EDuration="2.286873432s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 01:58:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 01:58:17.284515398 +0000 UTC m=+4645.548850342" watchObservedRunningTime="2026-11-03 01:58:17.286873432 +0000 UTC m=+4645.551208376" Nov 03 01:58:17.334667 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:58:17.334629 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:58:17.985400 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:58:17.985364 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:58:18.047742 ip-10-0-140-107 kubenswrapper[4939]: E1103 01:58:18.047705 4939 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-sthjh" containerName="extractor" Nov 03 01:58:18.276028 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:58:18.275935 4939 generic.go:356] "Generic (PLEG): container finished" podID="d6b552a0-b9c4-42cc-a0b4-cee73867e470" containerID="150a640998bd75d7365b66716a268353c08ac534fcf2c8c414498f72e5188d00" exitCode=0 Nov 03 01:58:18.276028 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:58:18.276013 4939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6876bd20-qn6mv" event={"ID":"d6b552a0-b9c4-42cc-a0b4-cee73867e470","Type":"ContainerDied","Data":"150a640998bd75d7365b66716a268353c08ac534fcf2c8c414498f72e5188d00"} Nov 03 01:58:18.710389 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:58:18.710340 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:58:19.406205 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:58:19.406178 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:58:19.443147 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:58:19.442829 4939 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6876bd20-qn6mv" Nov 03 01:58:19.478443 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:58:19.478408 4939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d6b552a0-b9c4-42cc-a0b4-cee73867e470-kube-api-access-fv4pv\" (UniqueName: \"kubernetes.io/projected/d6b552a0-b9c4-42cc-a0b4-cee73867e470-kube-api-access-fv4pv\") pod \"d6b552a0-b9c4-42cc-a0b4-cee73867e470\" (UID: \"d6b552a0-b9c4-42cc-a0b4-cee73867e470\") " Nov 03 01:58:19.478628 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:58:19.478496 4939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d6b552a0-b9c4-42cc-a0b4-cee73867e470-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d6b552a0-b9c4-42cc-a0b4-cee73867e470-maas-api-xks-ca\") pod \"d6b552a0-b9c4-42cc-a0b4-cee73867e470\" (UID: \"d6b552a0-b9c4-42cc-a0b4-cee73867e470\") " Nov 03 01:58:19.478682 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:58:19.478636 4939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d6b552a0-b9c4-42cc-a0b4-cee73867e470-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d6b552a0-b9c4-42cc-a0b4-cee73867e470-maas-api-openshift-ca\") pod \"d6b552a0-b9c4-42cc-a0b4-cee73867e470\" (UID: \"d6b552a0-b9c4-42cc-a0b4-cee73867e470\") " Nov 03 01:58:19.479017 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:58:19.478992 4939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d6b552a0-b9c4-42cc-a0b4-cee73867e470-maas-api-openshift-ca" pod "d6b552a0-b9c4-42cc-a0b4-cee73867e470" (UID: "d6b552a0-b9c4-42cc-a0b4-cee73867e470"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 01:58:19.480705 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:58:19.480673 4939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d6b552a0-b9c4-42cc-a0b4-cee73867e470-maas-api-xks-ca" pod "d6b552a0-b9c4-42cc-a0b4-cee73867e470" (UID: "d6b552a0-b9c4-42cc-a0b4-cee73867e470"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 01:58:19.480811 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:58:19.480746 4939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d6b552a0-b9c4-42cc-a0b4-cee73867e470-kube-api-access-fv4pv" pod "d6b552a0-b9c4-42cc-a0b4-cee73867e470" (UID: "d6b552a0-b9c4-42cc-a0b4-cee73867e470"). InnerVolumeSpecName "kube-api-access-fv4pv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 01:58:19.580228 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:58:19.580143 4939 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d6b552a0-b9c4-42cc-a0b4-cee73867e470-maas-api-openshift-ca\") on node \"ip-10-0-140-107.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:58:19.580228 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:58:19.580175 4939 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fv4pv\" (UniqueName: \"kubernetes.io/projected/d6b552a0-b9c4-42cc-a0b4-cee73867e470-kube-api-access-fv4pv\") on node \"ip-10-0-140-107.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:58:19.580228 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:58:19.580187 4939 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d6b552a0-b9c4-42cc-a0b4-cee73867e470-maas-api-xks-ca\") on node \"ip-10-0-140-107.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:58:20.098438 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:58:20.098403 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:58:20.286976 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:58:20.286888 4939 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6876bd20-qn6mv" Nov 03 01:58:20.286976 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:58:20.286939 4939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6876bd20-qn6mv" event={"ID":"d6b552a0-b9c4-42cc-a0b4-cee73867e470","Type":"ContainerDied","Data":"9488a8b0f0a2a3438b7a9eb74d716db7795c5fbe0970bd6d767a16965465d090"} Nov 03 01:58:20.287171 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:58:20.286988 4939 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="9488a8b0f0a2a3438b7a9eb74d716db7795c5fbe0970bd6d767a16965465d090" Nov 03 01:58:20.769766 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:58:20.769732 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:58:21.390033 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:58:21.389997 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:58:21.992744 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:58:21.992695 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:58:22.685499 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:58:22.685465 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:58:23.129312 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:58:23.129275 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:58:23.238035 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:58:23.238006 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:58:23.341970 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:58:23.341897 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:58:24.023722 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:58:24.023675 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:58:24.609743 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:58:24.609689 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:58:25.385747 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:58:25.385710 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:58:26.107162 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:58:26.107129 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:58:26.772122 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:58:26.772085 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:58:26.915667 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:58:26.915633 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:58:27.272693 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:58:27.272659 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:58:27.372316 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:58:27.372283 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:58:27.728078 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:58:27.728046 4939 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-4pw7v"] Nov 03 01:58:28.075330 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:58:28.075245 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:58:28.783006 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:58:28.782970 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:58:29.570631 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:58:29.570593 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:58:30.199256 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:58:30.199218 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:58:30.907832 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:58:30.907795 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:58:31.497773 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:58:31.497734 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:58:31.870180 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:58:31.869192 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:58:32.196681 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:58:32.196598 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:58:32.353866 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:58:32.353828 4939 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-7fde87-7b7c5wmrlg"] Nov 03 01:58:32.441314 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:58:32.441279 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:58:32.548514 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:58:32.548434 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:58:32.995655 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:58:32.995617 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:58:33.624118 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:58:33.624086 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:58:33.796054 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:58:33.796019 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:58:35.193884 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:58:35.193845 4939 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-4pw7v"] Nov 03 01:58:37.354280 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:58:37.354235 4939 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-7fde87-7b7c5wmrlg" podUID="fee4ebdb-2ebb-4918-ab3f-f9fbc049880d" containerName="payload-pre-processing" containerID="cri-o://21fb07471c0b2097957e4de8acf7946e250bb1579e36a3d538a40a2d6836ad93" gracePeriod=25 Nov 03 01:58:39.435983 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:58:39.435947 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:58:39.549058 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:58:39.549025 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:58:43.029588 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:58:43.029550 4939 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-7fde87-7b7c5wmrlg" podUID="fee4ebdb-2ebb-4918-ab3f-f9fbc049880d" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 03 01:58:46.874252 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:58:46.874213 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:58:48.029487 ip-10-0-140-107 kubenswrapper[4939]: E1103 01:58:48.029446 4939 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-sthjh" containerName="extractor" Nov 03 01:58:53.029741 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:58:53.029700 4939 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-7fde87-7b7c5wmrlg" podUID="fee4ebdb-2ebb-4918-ab3f-f9fbc049880d" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 03 01:58:57.604991 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:58:57.604968 4939 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-7fde87-7b7c5wmrlg" Nov 03 01:58:57.682535 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:58:57.682460 4939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fee4ebdb-2ebb-4918-ab3f-f9fbc049880d-kube-api-access-fr6rk\" (UniqueName: \"kubernetes.io/projected/fee4ebdb-2ebb-4918-ab3f-f9fbc049880d-kube-api-access-fr6rk\") pod \"fee4ebdb-2ebb-4918-ab3f-f9fbc049880d\" (UID: \"fee4ebdb-2ebb-4918-ab3f-f9fbc049880d\") " Nov 03 01:58:57.682669 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:58:57.682613 4939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fee4ebdb-2ebb-4918-ab3f-f9fbc049880d-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fee4ebdb-2ebb-4918-ab3f-f9fbc049880d-plugins-config-volume\") pod \"fee4ebdb-2ebb-4918-ab3f-f9fbc049880d\" (UID: \"fee4ebdb-2ebb-4918-ab3f-f9fbc049880d\") " Nov 03 01:58:57.682986 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:58:57.682964 4939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fee4ebdb-2ebb-4918-ab3f-f9fbc049880d-plugins-config-volume" pod "fee4ebdb-2ebb-4918-ab3f-f9fbc049880d" (UID: "fee4ebdb-2ebb-4918-ab3f-f9fbc049880d"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 01:58:57.684557 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:58:57.684529 4939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fee4ebdb-2ebb-4918-ab3f-f9fbc049880d-kube-api-access-fr6rk" pod "fee4ebdb-2ebb-4918-ab3f-f9fbc049880d" (UID: "fee4ebdb-2ebb-4918-ab3f-f9fbc049880d"). InnerVolumeSpecName "kube-api-access-fr6rk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 01:58:57.783541 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:58:57.783513 4939 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fee4ebdb-2ebb-4918-ab3f-f9fbc049880d-plugins-config-volume\") on node \"ip-10-0-140-107.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:58:57.783541 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:58:57.783537 4939 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fr6rk\" (UniqueName: \"kubernetes.io/projected/fee4ebdb-2ebb-4918-ab3f-f9fbc049880d-kube-api-access-fr6rk\") on node \"ip-10-0-140-107.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:58:58.470531 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:58:58.470493 4939 generic.go:356] "Generic (PLEG): container finished" podID="fee4ebdb-2ebb-4918-ab3f-f9fbc049880d" containerID="21fb07471c0b2097957e4de8acf7946e250bb1579e36a3d538a40a2d6836ad93" exitCode=0 Nov 03 01:58:58.470718 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:58:58.470542 4939 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-7fde87-7b7c5wmrlg" Nov 03 01:58:58.470718 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:58:58.470573 4939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-7fde87-7b7c5wmrlg" event={"ID":"fee4ebdb-2ebb-4918-ab3f-f9fbc049880d","Type":"ContainerDied","Data":"21fb07471c0b2097957e4de8acf7946e250bb1579e36a3d538a40a2d6836ad93"} Nov 03 01:58:58.470718 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:58:58.470626 4939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-7fde87-7b7c5wmrlg" event={"ID":"fee4ebdb-2ebb-4918-ab3f-f9fbc049880d","Type":"ContainerDied","Data":"3cd5192ff9e5ebc92d058f001a8e24792a1bd1ebf1f904c10353d25cce2e6414"} Nov 03 01:58:58.470718 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:58:58.470647 4939 scope.go:122] "RemoveContainer" containerID="21fb07471c0b2097957e4de8acf7946e250bb1579e36a3d538a40a2d6836ad93" Nov 03 01:58:58.479641 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:58:58.479439 4939 scope.go:122] "RemoveContainer" containerID="21fb07471c0b2097957e4de8acf7946e250bb1579e36a3d538a40a2d6836ad93" Nov 03 01:58:58.479729 ip-10-0-140-107 kubenswrapper[4939]: E1103 01:58:58.479712 4939 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"21fb07471c0b2097957e4de8acf7946e250bb1579e36a3d538a40a2d6836ad93\": container with ID starting with 21fb07471c0b2097957e4de8acf7946e250bb1579e36a3d538a40a2d6836ad93 not found: ID does not exist" containerID="21fb07471c0b2097957e4de8acf7946e250bb1579e36a3d538a40a2d6836ad93" Nov 03 01:58:58.479773 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:58:58.479737 4939 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"21fb07471c0b2097957e4de8acf7946e250bb1579e36a3d538a40a2d6836ad93"} err="failed to get container status \"21fb07471c0b2097957e4de8acf7946e250bb1579e36a3d538a40a2d6836ad93\": rpc error: code = NotFound desc = could not find container \"21fb07471c0b2097957e4de8acf7946e250bb1579e36a3d538a40a2d6836ad93\": container with ID starting with 21fb07471c0b2097957e4de8acf7946e250bb1579e36a3d538a40a2d6836ad93 not found: ID does not exist" Nov 03 01:58:58.487074 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:58:58.487049 4939 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-7fde87-7b7c5wmrlg"] Nov 03 01:58:58.493064 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:58:58.493042 4939 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-7fde87-7b7c5wmrlg"] Nov 03 01:58:59.866318 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:58:59.866288 4939 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fee4ebdb-2ebb-4918-ab3f-f9fbc049880d" path="/var/lib/kubelet/pods/fee4ebdb-2ebb-4918-ab3f-f9fbc049880d/volumes" Nov 03 01:59:03.794067 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:59:03.794031 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:59:03.928339 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:59:03.928303 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:59:04.956429 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:59:04.956396 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:59:04.990012 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:59:04.989975 4939 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-7fde87-7b7c5lgrsh"] Nov 03 01:59:04.990617 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:59:04.990594 4939 cpu_manager.go:414] "containerMap: removing container" podUID="fee4ebdb-2ebb-4918-ab3f-f9fbc049880d" containerName="payload-pre-processing" Nov 03 01:59:04.990690 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:59:04.990619 4939 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fee4ebdb-2ebb-4918-ab3f-f9fbc049880d" containerName="payload-pre-processing" Nov 03 01:59:04.990690 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:59:04.990645 4939 cpu_manager.go:414] "containerMap: removing container" podUID="d6b552a0-b9c4-42cc-a0b4-cee73867e470" containerName="revoke-keys" Nov 03 01:59:04.990690 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:59:04.990654 4939 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d6b552a0-b9c4-42cc-a0b4-cee73867e470" containerName="revoke-keys" Nov 03 01:59:04.990820 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:59:04.990809 4939 memory_manager.go:365] "RemoveStaleState removing state" podUID="fee4ebdb-2ebb-4918-ab3f-f9fbc049880d" containerName="payload-pre-processing" Nov 03 01:59:04.990868 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:59:04.990823 4939 memory_manager.go:365] "RemoveStaleState removing state" podUID="d6b552a0-b9c4-42cc-a0b4-cee73867e470" containerName="revoke-keys" Nov 03 01:59:04.994361 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:59:04.994337 4939 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-7fde87-7b7c5lgrsh" Nov 03 01:59:05.003330 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:59:05.003304 4939 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-single-m-bf3a2bea-dockercfg-lcvml\"" Nov 03 01:59:05.006636 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:59:05.006607 4939 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-7fde87-7b7c5lgrsh"] Nov 03 01:59:05.056310 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:59:05.056279 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/751d7569-667f-4fdf-aba3-c842afc12464-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-single-main-7fde87-7b7c5lgrsh\" (UID: \"751d7569-667f-4fdf-aba3-c842afc12464\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-7fde87-7b7c5lgrsh" Nov 03 01:59:05.056477 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:59:05.056458 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gfxkz\" (UniqueName: \"kubernetes.io/projected/751d7569-667f-4fdf-aba3-c842afc12464-kube-api-access-gfxkz\") pod \"payload-pre-processing-e2e-worker-single-main-7fde87-7b7c5lgrsh\" (UID: \"751d7569-667f-4fdf-aba3-c842afc12464\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-7fde87-7b7c5lgrsh" Nov 03 01:59:05.079014 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:59:05.078989 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:59:05.084757 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:59:05.084734 4939 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-single-main-7fde87\"" Nov 03 01:59:05.157689 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:59:05.157648 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gfxkz\" (UniqueName: \"kubernetes.io/projected/751d7569-667f-4fdf-aba3-c842afc12464-kube-api-access-gfxkz\") pod \"payload-pre-processing-e2e-worker-single-main-7fde87-7b7c5lgrsh\" (UID: \"751d7569-667f-4fdf-aba3-c842afc12464\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-7fde87-7b7c5lgrsh" Nov 03 01:59:05.157900 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:59:05.157727 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/751d7569-667f-4fdf-aba3-c842afc12464-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-single-main-7fde87-7b7c5lgrsh\" (UID: \"751d7569-667f-4fdf-aba3-c842afc12464\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-7fde87-7b7c5lgrsh" Nov 03 01:59:05.158481 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:59:05.158459 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/751d7569-667f-4fdf-aba3-c842afc12464-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-single-main-7fde87-7b7c5lgrsh\" (UID: \"751d7569-667f-4fdf-aba3-c842afc12464\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-7fde87-7b7c5lgrsh" Nov 03 01:59:05.166463 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:59:05.166437 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gfxkz\" (UniqueName: \"kubernetes.io/projected/751d7569-667f-4fdf-aba3-c842afc12464-kube-api-access-gfxkz\") pod \"payload-pre-processing-e2e-worker-single-main-7fde87-7b7c5lgrsh\" (UID: \"751d7569-667f-4fdf-aba3-c842afc12464\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-7fde87-7b7c5lgrsh" Nov 03 01:59:05.238607 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:59:05.238524 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:59:05.315431 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:59:05.315396 4939 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-7fde87-7b7c5lgrsh" Nov 03 01:59:05.601678 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:59:05.601651 4939 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-7fde87-7b7c5lgrsh"] Nov 03 01:59:05.604302 ip-10-0-140-107 kubenswrapper[4939]: W1103 01:59:05.604272 4939 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod751d7569_667f_4fdf_aba3_c842afc12464.slice/crio-ca180f7c7e31db9886bd5cb62544d0fdfff028a97a8eb5b071de69afde951d65 WatchSource:0}: Error finding container ca180f7c7e31db9886bd5cb62544d0fdfff028a97a8eb5b071de69afde951d65: Status 404 returned error can't find the container with id ca180f7c7e31db9886bd5cb62544d0fdfff028a97a8eb5b071de69afde951d65 Nov 03 01:59:05.823880 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:59:05.823829 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:59:06.495669 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:59:06.495630 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:59:06.511224 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:59:06.511190 4939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-7fde87-7b7c5lgrsh" event={"ID":"751d7569-667f-4fdf-aba3-c842afc12464","Type":"ContainerStarted","Data":"339c7f7262b1f8abca22ba0c4ebe16da30ea976dad8e2cba5debbef9cd88f5f8"} Nov 03 01:59:06.511403 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:59:06.511248 4939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-7fde87-7b7c5lgrsh" event={"ID":"751d7569-667f-4fdf-aba3-c842afc12464","Type":"ContainerStarted","Data":"ca180f7c7e31db9886bd5cb62544d0fdfff028a97a8eb5b071de69afde951d65"} Nov 03 01:59:06.511469 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:59:06.511412 4939 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-7fde87-7b7c5lgrsh" Nov 03 01:59:06.541047 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:59:06.540980 4939 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-7fde87-7b7c5lgrsh" podStartSLOduration=2.540962488 podStartE2EDuration="2.540962488s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 01:59:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 01:59:06.539817352 +0000 UTC m=+4694.804152293" watchObservedRunningTime="2026-11-03 01:59:06.540962488 +0000 UTC m=+4694.805297435" Nov 03 01:59:07.114864 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:59:07.114823 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:59:07.231118 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:59:07.231085 4939 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-4pw7v"] Nov 03 01:59:08.009710 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:59:08.009669 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:59:08.218299 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:59:08.218242 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:59:08.353495 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:59:08.353435 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:59:08.779007 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:59:08.778901 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:59:09.131073 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:59:09.131021 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:59:09.637959 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:59:09.636229 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:59:10.242811 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:59:10.242776 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:59:10.986874 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:59:10.986831 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:59:11.589752 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:59:11.589716 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:59:11.819736 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:59:11.819703 4939 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-4pw7v"] Nov 03 01:59:12.208294 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:59:12.208259 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:59:12.655479 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:59:12.655436 4939 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-7fde87-gkn9m"] Nov 03 01:59:12.659372 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:59:12.659350 4939 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-7fde87-gkn9m" Nov 03 01:59:12.665325 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:59:12.665293 4939 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 01:59:12.665532 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:59:12.665507 4939 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 03 01:59:12.665692 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:59:12.665530 4939 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 03 01:59:12.665692 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:59:12.665601 4939 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-swfcr\"" Nov 03 01:59:12.667988 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:59:12.667965 4939 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-7fde87-gkn9m"] Nov 03 01:59:12.747838 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:59:12.747786 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9zfbt\" (UniqueName: \"kubernetes.io/projected/14a22989-9cad-47ee-9626-7c3b0600f76c-kube-api-access-9zfbt\") pod \"maas-api-revoke-keys-e2e-worker-single-main-7fde87-gkn9m\" (UID: \"14a22989-9cad-47ee-9626-7c3b0600f76c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-7fde87-gkn9m" Nov 03 01:59:12.748092 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:59:12.747862 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/14a22989-9cad-47ee-9626-7c3b0600f76c-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-7fde87-gkn9m\" (UID: \"14a22989-9cad-47ee-9626-7c3b0600f76c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-7fde87-gkn9m" Nov 03 01:59:12.748092 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:59:12.747939 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/14a22989-9cad-47ee-9626-7c3b0600f76c-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-7fde87-gkn9m\" (UID: \"14a22989-9cad-47ee-9626-7c3b0600f76c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-7fde87-gkn9m" Nov 03 01:59:12.848613 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:59:12.848576 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9zfbt\" (UniqueName: \"kubernetes.io/projected/14a22989-9cad-47ee-9626-7c3b0600f76c-kube-api-access-9zfbt\") pod \"maas-api-revoke-keys-e2e-worker-single-main-7fde87-gkn9m\" (UID: \"14a22989-9cad-47ee-9626-7c3b0600f76c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-7fde87-gkn9m" Nov 03 01:59:12.848816 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:59:12.848628 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/14a22989-9cad-47ee-9626-7c3b0600f76c-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-7fde87-gkn9m\" (UID: \"14a22989-9cad-47ee-9626-7c3b0600f76c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-7fde87-gkn9m" Nov 03 01:59:12.848816 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:59:12.848668 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/14a22989-9cad-47ee-9626-7c3b0600f76c-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-7fde87-gkn9m\" (UID: \"14a22989-9cad-47ee-9626-7c3b0600f76c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-7fde87-gkn9m" Nov 03 01:59:12.849470 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:59:12.849445 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/14a22989-9cad-47ee-9626-7c3b0600f76c-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-7fde87-gkn9m\" (UID: \"14a22989-9cad-47ee-9626-7c3b0600f76c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-7fde87-gkn9m" Nov 03 01:59:12.851218 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:59:12.851198 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/14a22989-9cad-47ee-9626-7c3b0600f76c-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-7fde87-gkn9m\" (UID: \"14a22989-9cad-47ee-9626-7c3b0600f76c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-7fde87-gkn9m" Nov 03 01:59:12.856961 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:59:12.856934 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9zfbt\" (UniqueName: \"kubernetes.io/projected/14a22989-9cad-47ee-9626-7c3b0600f76c-kube-api-access-9zfbt\") pod \"maas-api-revoke-keys-e2e-worker-single-main-7fde87-gkn9m\" (UID: \"14a22989-9cad-47ee-9626-7c3b0600f76c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-7fde87-gkn9m" Nov 03 01:59:12.892285 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:59:12.892258 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:59:12.974865 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:59:12.974790 4939 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-7fde87-gkn9m" Nov 03 01:59:13.246277 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:59:13.246252 4939 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-7fde87-gkn9m"] Nov 03 01:59:13.248246 ip-10-0-140-107 kubenswrapper[4939]: W1103 01:59:13.248215 4939 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod14a22989_9cad_47ee_9626_7c3b0600f76c.slice/crio-42f599af9213984d12f66ae4bf24a5acf9ee879ca04a995abde9fbdfbdbb2ff4 WatchSource:0}: Error finding container 42f599af9213984d12f66ae4bf24a5acf9ee879ca04a995abde9fbdfbdbb2ff4: Status 404 returned error can't find the container with id 42f599af9213984d12f66ae4bf24a5acf9ee879ca04a995abde9fbdfbdbb2ff4 Nov 03 01:59:13.550461 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:59:13.550428 4939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-7fde87-gkn9m" event={"ID":"14a22989-9cad-47ee-9626-7c3b0600f76c","Type":"ContainerStarted","Data":"42f599af9213984d12f66ae4bf24a5acf9ee879ca04a995abde9fbdfbdbb2ff4"} Nov 03 01:59:13.576008 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:59:13.575901 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:59:14.197863 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:59:14.197833 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:59:14.556469 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:59:14.556428 4939 generic.go:356] "Generic (PLEG): container finished" podID="14a22989-9cad-47ee-9626-7c3b0600f76c" containerID="182b49b73a53963e8921f9a4d3a502954e39c240ecaa331162eb652a08c5f378" exitCode=0 Nov 03 01:59:14.556666 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:59:14.556516 4939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-7fde87-gkn9m" event={"ID":"14a22989-9cad-47ee-9626-7c3b0600f76c","Type":"ContainerDied","Data":"182b49b73a53963e8921f9a4d3a502954e39c240ecaa331162eb652a08c5f378"} Nov 03 01:59:14.807144 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:59:14.807060 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:59:14.921593 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:59:14.921493 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:59:15.645234 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:59:15.645188 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:59:15.734499 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:59:15.734471 4939 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-7fde87-gkn9m" Nov 03 01:59:15.882773 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:59:15.882741 4939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/14a22989-9cad-47ee-9626-7c3b0600f76c-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/14a22989-9cad-47ee-9626-7c3b0600f76c-maas-api-openshift-ca\") pod \"14a22989-9cad-47ee-9626-7c3b0600f76c\" (UID: \"14a22989-9cad-47ee-9626-7c3b0600f76c\") " Nov 03 01:59:15.882977 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:59:15.882799 4939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/14a22989-9cad-47ee-9626-7c3b0600f76c-kube-api-access-9zfbt\" (UniqueName: \"kubernetes.io/projected/14a22989-9cad-47ee-9626-7c3b0600f76c-kube-api-access-9zfbt\") pod \"14a22989-9cad-47ee-9626-7c3b0600f76c\" (UID: \"14a22989-9cad-47ee-9626-7c3b0600f76c\") " Nov 03 01:59:15.882977 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:59:15.882961 4939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/14a22989-9cad-47ee-9626-7c3b0600f76c-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/14a22989-9cad-47ee-9626-7c3b0600f76c-maas-api-xks-ca\") pod \"14a22989-9cad-47ee-9626-7c3b0600f76c\" (UID: \"14a22989-9cad-47ee-9626-7c3b0600f76c\") " Nov 03 01:59:15.883399 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:59:15.883183 4939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/14a22989-9cad-47ee-9626-7c3b0600f76c-maas-api-openshift-ca" pod "14a22989-9cad-47ee-9626-7c3b0600f76c" (UID: "14a22989-9cad-47ee-9626-7c3b0600f76c"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 01:59:15.883399 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:59:15.883299 4939 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/14a22989-9cad-47ee-9626-7c3b0600f76c-maas-api-openshift-ca\") on node \"ip-10-0-140-107.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:59:15.884967 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:59:15.884942 4939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/14a22989-9cad-47ee-9626-7c3b0600f76c-maas-api-xks-ca" pod "14a22989-9cad-47ee-9626-7c3b0600f76c" (UID: "14a22989-9cad-47ee-9626-7c3b0600f76c"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 01:59:15.885126 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:59:15.885104 4939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/14a22989-9cad-47ee-9626-7c3b0600f76c-kube-api-access-9zfbt" pod "14a22989-9cad-47ee-9626-7c3b0600f76c" (UID: "14a22989-9cad-47ee-9626-7c3b0600f76c"). InnerVolumeSpecName "kube-api-access-9zfbt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 01:59:15.983958 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:59:15.983903 4939 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/14a22989-9cad-47ee-9626-7c3b0600f76c-maas-api-xks-ca\") on node \"ip-10-0-140-107.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:59:15.983958 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:59:15.983955 4939 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9zfbt\" (UniqueName: \"kubernetes.io/projected/14a22989-9cad-47ee-9626-7c3b0600f76c-kube-api-access-9zfbt\") on node \"ip-10-0-140-107.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:59:16.276010 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:59:16.275903 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:59:16.567113 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:59:16.567029 4939 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-7fde87-gkn9m" Nov 03 01:59:16.567113 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:59:16.567052 4939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-7fde87-gkn9m" event={"ID":"14a22989-9cad-47ee-9626-7c3b0600f76c","Type":"ContainerDied","Data":"42f599af9213984d12f66ae4bf24a5acf9ee879ca04a995abde9fbdfbdbb2ff4"} Nov 03 01:59:16.567314 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:59:16.567114 4939 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="42f599af9213984d12f66ae4bf24a5acf9ee879ca04a995abde9fbdfbdbb2ff4" Nov 03 01:59:16.999216 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:59:16.999184 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:59:17.519678 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:59:17.519649 4939 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-7fde87-7b7c5lgrsh" Nov 03 01:59:17.601533 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:59:17.601501 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:59:18.030351 ip-10-0-140-107 kubenswrapper[4939]: E1103 01:59:18.030313 4939 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-sthjh" containerName="extractor" Nov 03 01:59:18.274059 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:59:18.274021 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:59:18.895874 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:59:18.895841 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:59:19.512440 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:59:19.512411 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:59:20.180231 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:59:20.180194 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:59:20.811671 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:59:20.811632 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:59:21.481456 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:59:21.481423 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:59:22.091418 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:59:22.091384 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:59:22.693111 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:59:22.693068 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:59:22.783954 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:59:22.783902 4939 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-7fde87-7b7c5lgrsh"] Nov 03 01:59:22.874137 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:59:22.874086 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:59:22.999891 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:59:22.999782 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:59:23.392721 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:59:23.392679 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:59:23.916645 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:59:23.916604 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:59:24.050072 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:59:24.050036 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:59:24.123983 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:59:24.123947 4939 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-7fde87-gkn9m"] Nov 03 01:59:24.127209 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:59:24.127174 4939 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-7fde87-gkn9m"] Nov 03 01:59:24.156118 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:59:24.156084 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:59:24.540590 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:59:24.540516 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:59:24.559073 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:59:24.559038 4939 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-single-main-7fde87-openshift-default-5f8558df65m5nbx"] Nov 03 01:59:24.559278 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:59:24.559258 4939 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-single-main-7fde87-openshift-default-5f8558df65m5nbx" podUID="d7a9f51d-0fa4-4e3d-b431-d9712ce8bad0" containerName="istio-proxy" containerID="cri-o://8b98e8c5cae2374068da28d8fe5e716ca825f1a93455b56f3261515f9be8864b" gracePeriod=30 Nov 03 01:59:25.868323 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:59:25.868291 4939 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="14a22989-9cad-47ee-9626-7c3b0600f76c" path="/var/lib/kubelet/pods/14a22989-9cad-47ee-9626-7c3b0600f76c/volumes" Nov 03 01:59:26.020237 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:59:26.020202 4939 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-4pw7v"] Nov 03 01:59:27.785283 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:59:27.785187 4939 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-7fde87-7b7c5lgrsh" podUID="751d7569-667f-4fdf-aba3-c842afc12464" containerName="payload-pre-processing" containerID="cri-o://339c7f7262b1f8abca22ba0c4ebe16da30ea976dad8e2cba5debbef9cd88f5f8" gracePeriod=25 Nov 03 01:59:27.981724 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:59:27.981692 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:59:28.051933 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:59:28.051882 4939 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-7fde87-7b7c5lgrsh" Nov 03 01:59:28.131836 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:59:28.131799 4939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/751d7569-667f-4fdf-aba3-c842afc12464-kube-api-access-gfxkz\" (UniqueName: \"kubernetes.io/projected/751d7569-667f-4fdf-aba3-c842afc12464-kube-api-access-gfxkz\") pod \"751d7569-667f-4fdf-aba3-c842afc12464\" (UID: \"751d7569-667f-4fdf-aba3-c842afc12464\") " Nov 03 01:59:28.132084 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:59:28.131858 4939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/751d7569-667f-4fdf-aba3-c842afc12464-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/751d7569-667f-4fdf-aba3-c842afc12464-plugins-config-volume\") pod \"751d7569-667f-4fdf-aba3-c842afc12464\" (UID: \"751d7569-667f-4fdf-aba3-c842afc12464\") " Nov 03 01:59:28.132299 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:59:28.132273 4939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/751d7569-667f-4fdf-aba3-c842afc12464-plugins-config-volume" pod "751d7569-667f-4fdf-aba3-c842afc12464" (UID: "751d7569-667f-4fdf-aba3-c842afc12464"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 01:59:28.134098 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:59:28.134075 4939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/751d7569-667f-4fdf-aba3-c842afc12464-kube-api-access-gfxkz" pod "751d7569-667f-4fdf-aba3-c842afc12464" (UID: "751d7569-667f-4fdf-aba3-c842afc12464"). InnerVolumeSpecName "kube-api-access-gfxkz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 01:59:28.232795 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:59:28.232756 4939 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gfxkz\" (UniqueName: \"kubernetes.io/projected/751d7569-667f-4fdf-aba3-c842afc12464-kube-api-access-gfxkz\") on node \"ip-10-0-140-107.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:59:28.232795 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:59:28.232789 4939 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/751d7569-667f-4fdf-aba3-c842afc12464-plugins-config-volume\") on node \"ip-10-0-140-107.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:59:28.635211 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:59:28.635174 4939 generic.go:356] "Generic (PLEG): container finished" podID="751d7569-667f-4fdf-aba3-c842afc12464" containerID="339c7f7262b1f8abca22ba0c4ebe16da30ea976dad8e2cba5debbef9cd88f5f8" exitCode=0 Nov 03 01:59:28.635435 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:59:28.635236 4939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-7fde87-7b7c5lgrsh" event={"ID":"751d7569-667f-4fdf-aba3-c842afc12464","Type":"ContainerDied","Data":"339c7f7262b1f8abca22ba0c4ebe16da30ea976dad8e2cba5debbef9cd88f5f8"} Nov 03 01:59:28.635435 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:59:28.635253 4939 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-7fde87-7b7c5lgrsh" Nov 03 01:59:28.635435 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:59:28.635303 4939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-7fde87-7b7c5lgrsh" event={"ID":"751d7569-667f-4fdf-aba3-c842afc12464","Type":"ContainerDied","Data":"ca180f7c7e31db9886bd5cb62544d0fdfff028a97a8eb5b071de69afde951d65"} Nov 03 01:59:28.635435 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:59:28.635328 4939 scope.go:122] "RemoveContainer" containerID="339c7f7262b1f8abca22ba0c4ebe16da30ea976dad8e2cba5debbef9cd88f5f8" Nov 03 01:59:28.645510 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:59:28.645489 4939 scope.go:122] "RemoveContainer" containerID="339c7f7262b1f8abca22ba0c4ebe16da30ea976dad8e2cba5debbef9cd88f5f8" Nov 03 01:59:28.645776 ip-10-0-140-107 kubenswrapper[4939]: E1103 01:59:28.645759 4939 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"339c7f7262b1f8abca22ba0c4ebe16da30ea976dad8e2cba5debbef9cd88f5f8\": container with ID starting with 339c7f7262b1f8abca22ba0c4ebe16da30ea976dad8e2cba5debbef9cd88f5f8 not found: ID does not exist" containerID="339c7f7262b1f8abca22ba0c4ebe16da30ea976dad8e2cba5debbef9cd88f5f8" Nov 03 01:59:28.645829 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:59:28.645784 4939 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"339c7f7262b1f8abca22ba0c4ebe16da30ea976dad8e2cba5debbef9cd88f5f8"} err="failed to get container status \"339c7f7262b1f8abca22ba0c4ebe16da30ea976dad8e2cba5debbef9cd88f5f8\": rpc error: code = NotFound desc = could not find container \"339c7f7262b1f8abca22ba0c4ebe16da30ea976dad8e2cba5debbef9cd88f5f8\": container with ID starting with 339c7f7262b1f8abca22ba0c4ebe16da30ea976dad8e2cba5debbef9cd88f5f8 not found: ID does not exist" Nov 03 01:59:28.657551 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:59:28.657522 4939 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-7fde87-7b7c5lgrsh"] Nov 03 01:59:28.661333 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:59:28.661311 4939 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-7fde87-7b7c5lgrsh"] Nov 03 01:59:29.824308 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:59:29.824247 4939 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-single-main-7fde87-openshift-default-5f8558df65m5nbx" Nov 03 01:59:29.869281 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:59:29.869253 4939 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="751d7569-667f-4fdf-aba3-c842afc12464" path="/var/lib/kubelet/pods/751d7569-667f-4fdf-aba3-c842afc12464/volumes" Nov 03 01:59:29.950813 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:59:29.950780 4939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d7a9f51d-0fa4-4e3d-b431-d9712ce8bad0-istio-token\" (UniqueName: \"kubernetes.io/projected/d7a9f51d-0fa4-4e3d-b431-d9712ce8bad0-istio-token\") pod \"d7a9f51d-0fa4-4e3d-b431-d9712ce8bad0\" (UID: \"d7a9f51d-0fa4-4e3d-b431-d9712ce8bad0\") " Nov 03 01:59:29.951037 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:59:29.950829 4939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d7a9f51d-0fa4-4e3d-b431-d9712ce8bad0-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d7a9f51d-0fa4-4e3d-b431-d9712ce8bad0-istio-envoy\") pod \"d7a9f51d-0fa4-4e3d-b431-d9712ce8bad0\" (UID: \"d7a9f51d-0fa4-4e3d-b431-d9712ce8bad0\") " Nov 03 01:59:29.951037 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:59:29.950868 4939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d7a9f51d-0fa4-4e3d-b431-d9712ce8bad0-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d7a9f51d-0fa4-4e3d-b431-d9712ce8bad0-workload-socket\") pod \"d7a9f51d-0fa4-4e3d-b431-d9712ce8bad0\" (UID: \"d7a9f51d-0fa4-4e3d-b431-d9712ce8bad0\") " Nov 03 01:59:29.951037 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:59:29.950988 4939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d7a9f51d-0fa4-4e3d-b431-d9712ce8bad0-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d7a9f51d-0fa4-4e3d-b431-d9712ce8bad0-istiod-ca-cert\") pod \"d7a9f51d-0fa4-4e3d-b431-d9712ce8bad0\" (UID: \"d7a9f51d-0fa4-4e3d-b431-d9712ce8bad0\") " Nov 03 01:59:29.951248 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:59:29.951071 4939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/d7a9f51d-0fa4-4e3d-b431-d9712ce8bad0-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d7a9f51d-0fa4-4e3d-b431-d9712ce8bad0-istio-podinfo\") pod \"d7a9f51d-0fa4-4e3d-b431-d9712ce8bad0\" (UID: \"d7a9f51d-0fa4-4e3d-b431-d9712ce8bad0\") " Nov 03 01:59:29.951248 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:59:29.951183 4939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d7a9f51d-0fa4-4e3d-b431-d9712ce8bad0-kube-api-access-npr5c\" (UniqueName: \"kubernetes.io/projected/d7a9f51d-0fa4-4e3d-b431-d9712ce8bad0-kube-api-access-npr5c\") pod \"d7a9f51d-0fa4-4e3d-b431-d9712ce8bad0\" (UID: \"d7a9f51d-0fa4-4e3d-b431-d9712ce8bad0\") " Nov 03 01:59:29.951248 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:59:29.951186 4939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d7a9f51d-0fa4-4e3d-b431-d9712ce8bad0-workload-socket" pod "d7a9f51d-0fa4-4e3d-b431-d9712ce8bad0" (UID: "d7a9f51d-0fa4-4e3d-b431-d9712ce8bad0"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 01:59:29.951433 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:59:29.951258 4939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d7a9f51d-0fa4-4e3d-b431-d9712ce8bad0-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d7a9f51d-0fa4-4e3d-b431-d9712ce8bad0-istio-data\") pod \"d7a9f51d-0fa4-4e3d-b431-d9712ce8bad0\" (UID: \"d7a9f51d-0fa4-4e3d-b431-d9712ce8bad0\") " Nov 03 01:59:29.951433 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:59:29.951350 4939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d7a9f51d-0fa4-4e3d-b431-d9712ce8bad0-istiod-ca-cert" pod "d7a9f51d-0fa4-4e3d-b431-d9712ce8bad0" (UID: "d7a9f51d-0fa4-4e3d-b431-d9712ce8bad0"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 01:59:29.951549 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:59:29.951502 4939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d7a9f51d-0fa4-4e3d-b431-d9712ce8bad0-istio-data" pod "d7a9f51d-0fa4-4e3d-b431-d9712ce8bad0" (UID: "d7a9f51d-0fa4-4e3d-b431-d9712ce8bad0"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 01:59:29.951657 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:59:29.951638 4939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d7a9f51d-0fa4-4e3d-b431-d9712ce8bad0-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d7a9f51d-0fa4-4e3d-b431-d9712ce8bad0-workload-certs\") pod \"d7a9f51d-0fa4-4e3d-b431-d9712ce8bad0\" (UID: \"d7a9f51d-0fa4-4e3d-b431-d9712ce8bad0\") " Nov 03 01:59:29.951944 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:59:29.951885 4939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d7a9f51d-0fa4-4e3d-b431-d9712ce8bad0-workload-certs" pod "d7a9f51d-0fa4-4e3d-b431-d9712ce8bad0" (UID: "d7a9f51d-0fa4-4e3d-b431-d9712ce8bad0"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 01:59:29.952089 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:59:29.952046 4939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d7a9f51d-0fa4-4e3d-b431-d9712ce8bad0-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d7a9f51d-0fa4-4e3d-b431-d9712ce8bad0-credential-socket\") pod \"d7a9f51d-0fa4-4e3d-b431-d9712ce8bad0\" (UID: \"d7a9f51d-0fa4-4e3d-b431-d9712ce8bad0\") " Nov 03 01:59:29.952326 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:59:29.952297 4939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d7a9f51d-0fa4-4e3d-b431-d9712ce8bad0-credential-socket" pod "d7a9f51d-0fa4-4e3d-b431-d9712ce8bad0" (UID: "d7a9f51d-0fa4-4e3d-b431-d9712ce8bad0"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 01:59:29.953706 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:59:29.953677 4939 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d7a9f51d-0fa4-4e3d-b431-d9712ce8bad0-workload-socket\") on node \"ip-10-0-140-107.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:59:29.953858 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:59:29.953707 4939 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d7a9f51d-0fa4-4e3d-b431-d9712ce8bad0-istiod-ca-cert\") on node \"ip-10-0-140-107.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:59:29.953858 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:59:29.953723 4939 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d7a9f51d-0fa4-4e3d-b431-d9712ce8bad0-istio-data\") on node \"ip-10-0-140-107.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:59:29.953858 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:59:29.953740 4939 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d7a9f51d-0fa4-4e3d-b431-d9712ce8bad0-workload-certs\") on node \"ip-10-0-140-107.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:59:29.953858 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:59:29.953756 4939 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d7a9f51d-0fa4-4e3d-b431-d9712ce8bad0-credential-socket\") on node \"ip-10-0-140-107.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:59:29.954225 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:59:29.953928 4939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d7a9f51d-0fa4-4e3d-b431-d9712ce8bad0-kube-api-access-npr5c" pod "d7a9f51d-0fa4-4e3d-b431-d9712ce8bad0" (UID: "d7a9f51d-0fa4-4e3d-b431-d9712ce8bad0"). InnerVolumeSpecName "kube-api-access-npr5c". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 01:59:29.954520 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:59:29.954490 4939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d7a9f51d-0fa4-4e3d-b431-d9712ce8bad0-istio-token" pod "d7a9f51d-0fa4-4e3d-b431-d9712ce8bad0" (UID: "d7a9f51d-0fa4-4e3d-b431-d9712ce8bad0"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 01:59:29.954631 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:59:29.954580 4939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d7a9f51d-0fa4-4e3d-b431-d9712ce8bad0-istio-envoy" pod "d7a9f51d-0fa4-4e3d-b431-d9712ce8bad0" (UID: "d7a9f51d-0fa4-4e3d-b431-d9712ce8bad0"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 01:59:29.954631 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:59:29.954597 4939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/d7a9f51d-0fa4-4e3d-b431-d9712ce8bad0-istio-podinfo" pod "d7a9f51d-0fa4-4e3d-b431-d9712ce8bad0" (UID: "d7a9f51d-0fa4-4e3d-b431-d9712ce8bad0"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 01:59:30.054734 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:59:30.054691 4939 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d7a9f51d-0fa4-4e3d-b431-d9712ce8bad0-istio-token\") on node \"ip-10-0-140-107.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:59:30.054734 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:59:30.054727 4939 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d7a9f51d-0fa4-4e3d-b431-d9712ce8bad0-istio-envoy\") on node \"ip-10-0-140-107.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:59:30.054734 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:59:30.054739 4939 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d7a9f51d-0fa4-4e3d-b431-d9712ce8bad0-istio-podinfo\") on node \"ip-10-0-140-107.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:59:30.055041 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:59:30.054750 4939 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-npr5c\" (UniqueName: \"kubernetes.io/projected/d7a9f51d-0fa4-4e3d-b431-d9712ce8bad0-kube-api-access-npr5c\") on node \"ip-10-0-140-107.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 01:59:30.648297 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:59:30.648261 4939 generic.go:356] "Generic (PLEG): container finished" podID="d7a9f51d-0fa4-4e3d-b431-d9712ce8bad0" containerID="8b98e8c5cae2374068da28d8fe5e716ca825f1a93455b56f3261515f9be8864b" exitCode=0 Nov 03 01:59:30.648496 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:59:30.648317 4939 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-single-main-7fde87-openshift-default-5f8558df65m5nbx" Nov 03 01:59:30.648496 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:59:30.648348 4939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-7fde87-openshift-default-5f8558df65m5nbx" event={"ID":"d7a9f51d-0fa4-4e3d-b431-d9712ce8bad0","Type":"ContainerDied","Data":"8b98e8c5cae2374068da28d8fe5e716ca825f1a93455b56f3261515f9be8864b"} Nov 03 01:59:30.648496 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:59:30.648416 4939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-7fde87-openshift-default-5f8558df65m5nbx" event={"ID":"d7a9f51d-0fa4-4e3d-b431-d9712ce8bad0","Type":"ContainerDied","Data":"1d9772d7165e12f6738e51dea97f362fec462690fc11905b5b85e42a9a68de27"} Nov 03 01:59:30.648496 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:59:30.648445 4939 scope.go:122] "RemoveContainer" containerID="8b98e8c5cae2374068da28d8fe5e716ca825f1a93455b56f3261515f9be8864b" Nov 03 01:59:30.659330 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:59:30.659311 4939 scope.go:122] "RemoveContainer" containerID="a01a0aa2fc01e4697197f293390e85686f18395c27ab30299df3e590f2416dae" Nov 03 01:59:30.667568 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:59:30.667547 4939 scope.go:122] "RemoveContainer" containerID="8b98e8c5cae2374068da28d8fe5e716ca825f1a93455b56f3261515f9be8864b" Nov 03 01:59:30.667831 ip-10-0-140-107 kubenswrapper[4939]: E1103 01:59:30.667813 4939 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8b98e8c5cae2374068da28d8fe5e716ca825f1a93455b56f3261515f9be8864b\": container with ID starting with 8b98e8c5cae2374068da28d8fe5e716ca825f1a93455b56f3261515f9be8864b not found: ID does not exist" containerID="8b98e8c5cae2374068da28d8fe5e716ca825f1a93455b56f3261515f9be8864b" Nov 03 01:59:30.667873 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:59:30.667842 4939 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8b98e8c5cae2374068da28d8fe5e716ca825f1a93455b56f3261515f9be8864b"} err="failed to get container status \"8b98e8c5cae2374068da28d8fe5e716ca825f1a93455b56f3261515f9be8864b\": rpc error: code = NotFound desc = could not find container \"8b98e8c5cae2374068da28d8fe5e716ca825f1a93455b56f3261515f9be8864b\": container with ID starting with 8b98e8c5cae2374068da28d8fe5e716ca825f1a93455b56f3261515f9be8864b not found: ID does not exist" Nov 03 01:59:30.667873 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:59:30.667860 4939 scope.go:122] "RemoveContainer" containerID="a01a0aa2fc01e4697197f293390e85686f18395c27ab30299df3e590f2416dae" Nov 03 01:59:30.668090 ip-10-0-140-107 kubenswrapper[4939]: E1103 01:59:30.668076 4939 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a01a0aa2fc01e4697197f293390e85686f18395c27ab30299df3e590f2416dae\": container with ID starting with a01a0aa2fc01e4697197f293390e85686f18395c27ab30299df3e590f2416dae not found: ID does not exist" containerID="a01a0aa2fc01e4697197f293390e85686f18395c27ab30299df3e590f2416dae" Nov 03 01:59:30.668135 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:59:30.668094 4939 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a01a0aa2fc01e4697197f293390e85686f18395c27ab30299df3e590f2416dae"} err="failed to get container status \"a01a0aa2fc01e4697197f293390e85686f18395c27ab30299df3e590f2416dae\": rpc error: code = NotFound desc = could not find container \"a01a0aa2fc01e4697197f293390e85686f18395c27ab30299df3e590f2416dae\": container with ID starting with a01a0aa2fc01e4697197f293390e85686f18395c27ab30299df3e590f2416dae not found: ID does not exist" Nov 03 01:59:30.671245 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:59:30.671222 4939 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-single-main-7fde87-openshift-default-5f8558df65m5nbx"] Nov 03 01:59:30.677386 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:59:30.677364 4939 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-single-main-7fde87-openshift-default-5f8558df65m5nbx"] Nov 03 01:59:31.866320 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:59:31.866286 4939 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d7a9f51d-0fa4-4e3d-b431-d9712ce8bad0" path="/var/lib/kubelet/pods/d7a9f51d-0fa4-4e3d-b431-d9712ce8bad0/volumes" Nov 03 01:59:36.520469 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:59:36.520370 4939 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-4pw7v"] Nov 03 01:59:40.403191 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:59:40.403154 4939 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-4pw7v"] Nov 03 01:59:46.939047 ip-10-0-140-107 kubenswrapper[4939]: I1103 01:59:46.939013 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 01:59:48.029879 ip-10-0-140-107 kubenswrapper[4939]: E1103 01:59:48.029839 4939 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-sthjh" containerName="extractor" Nov 03 02:00:18.030124 ip-10-0-140-107 kubenswrapper[4939]: E1103 02:00:18.030080 4939 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-sthjh" containerName="extractor" Nov 03 02:00:46.984871 ip-10-0-140-107 kubenswrapper[4939]: I1103 02:00:46.984834 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 02:00:48.029566 ip-10-0-140-107 kubenswrapper[4939]: E1103 02:00:48.029527 4939 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-sthjh" containerName="extractor" Nov 03 02:00:53.294006 ip-10-0-140-107 kubenswrapper[4939]: I1103 02:00:53.293974 4939 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-n42s6_65d134ac-8a25-4c67-b6c7-cd752a098daf/console-operator/2.log" Nov 03 02:00:53.298989 ip-10-0-140-107 kubenswrapper[4939]: I1103 02:00:53.298967 4939 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-n42s6_65d134ac-8a25-4c67-b6c7-cd752a098daf/console-operator/2.log" Nov 03 02:01:04.375960 ip-10-0-140-107 kubenswrapper[4939]: I1103 02:01:04.372718 4939 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-4pw7v"] Nov 03 02:01:17.385822 ip-10-0-140-107 kubenswrapper[4939]: I1103 02:01:17.385778 4939 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-4pw7v"] Nov 03 02:01:18.030586 ip-10-0-140-107 kubenswrapper[4939]: E1103 02:01:18.030548 4939 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-sthjh" containerName="extractor" Nov 03 02:01:20.702931 ip-10-0-140-107 kubenswrapper[4939]: I1103 02:01:20.702861 4939 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-4pw7v"] Nov 03 02:01:33.631300 ip-10-0-140-107 kubenswrapper[4939]: I1103 02:01:33.631268 4939 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-4pw7v"] Nov 03 02:01:34.724732 ip-10-0-140-107 kubenswrapper[4939]: I1103 02:01:34.724695 4939 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-55cc657fcb-54j7t"] Nov 03 02:01:34.725158 ip-10-0-140-107 kubenswrapper[4939]: I1103 02:01:34.725108 4939 cpu_manager.go:414] "containerMap: removing container" podUID="751d7569-667f-4fdf-aba3-c842afc12464" containerName="payload-pre-processing" Nov 03 02:01:34.725158 ip-10-0-140-107 kubenswrapper[4939]: I1103 02:01:34.725120 4939 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="751d7569-667f-4fdf-aba3-c842afc12464" containerName="payload-pre-processing" Nov 03 02:01:34.725158 ip-10-0-140-107 kubenswrapper[4939]: I1103 02:01:34.725138 4939 cpu_manager.go:414] "containerMap: removing container" podUID="d7a9f51d-0fa4-4e3d-b431-d9712ce8bad0" containerName="istio-proxy" Nov 03 02:01:34.725158 ip-10-0-140-107 kubenswrapper[4939]: I1103 02:01:34.725144 4939 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d7a9f51d-0fa4-4e3d-b431-d9712ce8bad0" containerName="istio-proxy" Nov 03 02:01:34.725318 ip-10-0-140-107 kubenswrapper[4939]: I1103 02:01:34.725170 4939 cpu_manager.go:414] "containerMap: removing container" podUID="14a22989-9cad-47ee-9626-7c3b0600f76c" containerName="revoke-keys" Nov 03 02:01:34.725318 ip-10-0-140-107 kubenswrapper[4939]: I1103 02:01:34.725176 4939 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="14a22989-9cad-47ee-9626-7c3b0600f76c" containerName="revoke-keys" Nov 03 02:01:34.725318 ip-10-0-140-107 kubenswrapper[4939]: I1103 02:01:34.725238 4939 memory_manager.go:365] "RemoveStaleState removing state" podUID="14a22989-9cad-47ee-9626-7c3b0600f76c" containerName="revoke-keys" Nov 03 02:01:34.725318 ip-10-0-140-107 kubenswrapper[4939]: I1103 02:01:34.725251 4939 memory_manager.go:365] "RemoveStaleState removing state" podUID="d7a9f51d-0fa4-4e3d-b431-d9712ce8bad0" containerName="istio-proxy" Nov 03 02:01:34.725318 ip-10-0-140-107 kubenswrapper[4939]: I1103 02:01:34.725259 4939 memory_manager.go:365] "RemoveStaleState removing state" podUID="751d7569-667f-4fdf-aba3-c842afc12464" containerName="payload-pre-processing" Nov 03 02:01:34.725318 ip-10-0-140-107 kubenswrapper[4939]: I1103 02:01:34.725272 4939 memory_manager.go:365] "RemoveStaleState removing state" podUID="d7a9f51d-0fa4-4e3d-b431-d9712ce8bad0" containerName="istio-proxy" Nov 03 02:01:34.728240 ip-10-0-140-107 kubenswrapper[4939]: I1103 02:01:34.728224 4939 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-55cc657fcb-54j7t" Nov 03 02:01:34.734687 ip-10-0-140-107 kubenswrapper[4939]: I1103 02:01:34.734664 4939 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-metrics-tls\"" Nov 03 02:01:34.737678 ip-10-0-140-107 kubenswrapper[4939]: I1103 02:01:34.737546 4939 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-nfcxl\"" Nov 03 02:01:34.738144 ip-10-0-140-107 kubenswrapper[4939]: I1103 02:01:34.738122 4939 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-serving-cert\"" Nov 03 02:01:34.739767 ip-10-0-140-107 kubenswrapper[4939]: I1103 02:01:34.739725 4939 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 03 02:01:34.739941 ip-10-0-140-107 kubenswrapper[4939]: I1103 02:01:34.739766 4939 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 03 02:01:34.742268 ip-10-0-140-107 kubenswrapper[4939]: I1103 02:01:34.742231 4939 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-55cc657fcb-54j7t"] Nov 03 02:01:34.853552 ip-10-0-140-107 kubenswrapper[4939]: I1103 02:01:34.853512 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gsr96\" (UniqueName: \"kubernetes.io/projected/3a058653-d7b4-4040-a7fd-7df7ea9d6f23-kube-api-access-gsr96\") pod \"maas-api-55cc657fcb-54j7t\" (UID: \"3a058653-d7b4-4040-a7fd-7df7ea9d6f23\") " pod="odh-ai-gateway-infra/maas-api-55cc657fcb-54j7t" Nov 03 02:01:34.853738 ip-10-0-140-107 kubenswrapper[4939]: I1103 02:01:34.853586 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3a058653-d7b4-4040-a7fd-7df7ea9d6f23-metrics-certs\") pod \"maas-api-55cc657fcb-54j7t\" (UID: \"3a058653-d7b4-4040-a7fd-7df7ea9d6f23\") " pod="odh-ai-gateway-infra/maas-api-55cc657fcb-54j7t" Nov 03 02:01:34.853738 ip-10-0-140-107 kubenswrapper[4939]: I1103 02:01:34.853711 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3a058653-d7b4-4040-a7fd-7df7ea9d6f23-maas-api-tls\") pod \"maas-api-55cc657fcb-54j7t\" (UID: \"3a058653-d7b4-4040-a7fd-7df7ea9d6f23\") " pod="odh-ai-gateway-infra/maas-api-55cc657fcb-54j7t" Nov 03 02:01:34.853844 ip-10-0-140-107 kubenswrapper[4939]: I1103 02:01:34.853754 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3a058653-d7b4-4040-a7fd-7df7ea9d6f23-openshift-service-ca\") pod \"maas-api-55cc657fcb-54j7t\" (UID: \"3a058653-d7b4-4040-a7fd-7df7ea9d6f23\") " pod="odh-ai-gateway-infra/maas-api-55cc657fcb-54j7t" Nov 03 02:01:34.954342 ip-10-0-140-107 kubenswrapper[4939]: I1103 02:01:34.954308 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3a058653-d7b4-4040-a7fd-7df7ea9d6f23-maas-api-tls\") pod \"maas-api-55cc657fcb-54j7t\" (UID: \"3a058653-d7b4-4040-a7fd-7df7ea9d6f23\") " pod="odh-ai-gateway-infra/maas-api-55cc657fcb-54j7t" Nov 03 02:01:34.954556 ip-10-0-140-107 kubenswrapper[4939]: I1103 02:01:34.954360 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3a058653-d7b4-4040-a7fd-7df7ea9d6f23-openshift-service-ca\") pod \"maas-api-55cc657fcb-54j7t\" (UID: \"3a058653-d7b4-4040-a7fd-7df7ea9d6f23\") " pod="odh-ai-gateway-infra/maas-api-55cc657fcb-54j7t" Nov 03 02:01:34.954556 ip-10-0-140-107 kubenswrapper[4939]: I1103 02:01:34.954460 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gsr96\" (UniqueName: \"kubernetes.io/projected/3a058653-d7b4-4040-a7fd-7df7ea9d6f23-kube-api-access-gsr96\") pod \"maas-api-55cc657fcb-54j7t\" (UID: \"3a058653-d7b4-4040-a7fd-7df7ea9d6f23\") " pod="odh-ai-gateway-infra/maas-api-55cc657fcb-54j7t" Nov 03 02:01:34.954556 ip-10-0-140-107 kubenswrapper[4939]: I1103 02:01:34.954535 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3a058653-d7b4-4040-a7fd-7df7ea9d6f23-metrics-certs\") pod \"maas-api-55cc657fcb-54j7t\" (UID: \"3a058653-d7b4-4040-a7fd-7df7ea9d6f23\") " pod="odh-ai-gateway-infra/maas-api-55cc657fcb-54j7t" Nov 03 02:01:34.955529 ip-10-0-140-107 kubenswrapper[4939]: I1103 02:01:34.955496 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3a058653-d7b4-4040-a7fd-7df7ea9d6f23-openshift-service-ca\") pod \"maas-api-55cc657fcb-54j7t\" (UID: \"3a058653-d7b4-4040-a7fd-7df7ea9d6f23\") " pod="odh-ai-gateway-infra/maas-api-55cc657fcb-54j7t" Nov 03 02:01:34.957461 ip-10-0-140-107 kubenswrapper[4939]: I1103 02:01:34.957436 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3a058653-d7b4-4040-a7fd-7df7ea9d6f23-maas-api-tls\") pod \"maas-api-55cc657fcb-54j7t\" (UID: \"3a058653-d7b4-4040-a7fd-7df7ea9d6f23\") " pod="odh-ai-gateway-infra/maas-api-55cc657fcb-54j7t" Nov 03 02:01:34.957606 ip-10-0-140-107 kubenswrapper[4939]: I1103 02:01:34.957582 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3a058653-d7b4-4040-a7fd-7df7ea9d6f23-metrics-certs\") pod \"maas-api-55cc657fcb-54j7t\" (UID: \"3a058653-d7b4-4040-a7fd-7df7ea9d6f23\") " pod="odh-ai-gateway-infra/maas-api-55cc657fcb-54j7t" Nov 03 02:01:34.963738 ip-10-0-140-107 kubenswrapper[4939]: I1103 02:01:34.963713 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gsr96\" (UniqueName: \"kubernetes.io/projected/3a058653-d7b4-4040-a7fd-7df7ea9d6f23-kube-api-access-gsr96\") pod \"maas-api-55cc657fcb-54j7t\" (UID: \"3a058653-d7b4-4040-a7fd-7df7ea9d6f23\") " pod="odh-ai-gateway-infra/maas-api-55cc657fcb-54j7t" Nov 03 02:01:35.045586 ip-10-0-140-107 kubenswrapper[4939]: I1103 02:01:35.045513 4939 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-55cc657fcb-54j7t" Nov 03 02:01:35.318612 ip-10-0-140-107 kubenswrapper[4939]: I1103 02:01:35.318586 4939 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-55cc657fcb-54j7t"] Nov 03 02:01:35.321677 ip-10-0-140-107 kubenswrapper[4939]: W1103 02:01:35.321648 4939 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3a058653_d7b4_4040_a7fd_7df7ea9d6f23.slice/crio-2557c4deb8abba9ffbb70e186574dc53e50d43a1efdbc0c7f36d5ef046c1d542 WatchSource:0}: Error finding container 2557c4deb8abba9ffbb70e186574dc53e50d43a1efdbc0c7f36d5ef046c1d542: Status 404 returned error can't find the container with id 2557c4deb8abba9ffbb70e186574dc53e50d43a1efdbc0c7f36d5ef046c1d542 Nov 03 02:01:35.323430 ip-10-0-140-107 kubenswrapper[4939]: I1103 02:01:35.323409 4939 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Nov 03 02:01:35.342315 ip-10-0-140-107 kubenswrapper[4939]: I1103 02:01:35.342290 4939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-55cc657fcb-54j7t" event={"ID":"3a058653-d7b4-4040-a7fd-7df7ea9d6f23","Type":"ContainerStarted","Data":"2557c4deb8abba9ffbb70e186574dc53e50d43a1efdbc0c7f36d5ef046c1d542"} Nov 03 02:01:36.348034 ip-10-0-140-107 kubenswrapper[4939]: I1103 02:01:36.347995 4939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-55cc657fcb-54j7t" event={"ID":"3a058653-d7b4-4040-a7fd-7df7ea9d6f23","Type":"ContainerStarted","Data":"ee1403b4773c3303cf9197d9db553b8454f06246d72a8e29f1edcaf59ff30c94"} Nov 03 02:01:36.348434 ip-10-0-140-107 kubenswrapper[4939]: I1103 02:01:36.348083 4939 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-55cc657fcb-54j7t" Nov 03 02:01:36.369768 ip-10-0-140-107 kubenswrapper[4939]: I1103 02:01:36.369724 4939 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-55cc657fcb-54j7t" podStartSLOduration=1.805058397 podStartE2EDuration="2.369712878s" totalImagesPullingTime="564.654481ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 02:01:34 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 02:01:36.366808124 +0000 UTC m=+4844.631143070" watchObservedRunningTime="2026-11-03 02:01:36.369712878 +0000 UTC m=+4844.634047826" Nov 03 02:01:42.224163 ip-10-0-140-107 kubenswrapper[4939]: I1103 02:01:42.224120 4939 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-55cc657fcb-54j7t"] Nov 03 02:01:42.224684 ip-10-0-140-107 kubenswrapper[4939]: I1103 02:01:42.224385 4939 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-55cc657fcb-54j7t" podUID="3a058653-d7b4-4040-a7fd-7df7ea9d6f23" containerName="maas-api" containerID="cri-o://ee1403b4773c3303cf9197d9db553b8454f06246d72a8e29f1edcaf59ff30c94" gracePeriod=30 Nov 03 02:01:42.229588 ip-10-0-140-107 kubenswrapper[4939]: I1103 02:01:42.229564 4939 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-55cc657fcb-54j7t" Nov 03 02:01:42.265353 ip-10-0-140-107 kubenswrapper[4939]: I1103 02:01:42.265315 4939 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-7b97bffd64-sw5r8"] Nov 03 02:01:42.266055 ip-10-0-140-107 kubenswrapper[4939]: I1103 02:01:42.266026 4939 cpu_manager.go:414] "containerMap: removing container" podUID="d7a9f51d-0fa4-4e3d-b431-d9712ce8bad0" containerName="istio-proxy" Nov 03 02:01:42.266136 ip-10-0-140-107 kubenswrapper[4939]: I1103 02:01:42.266059 4939 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d7a9f51d-0fa4-4e3d-b431-d9712ce8bad0" containerName="istio-proxy" Nov 03 02:01:42.270458 ip-10-0-140-107 kubenswrapper[4939]: I1103 02:01:42.270433 4939 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-7b97bffd64-sw5r8" Nov 03 02:01:42.280463 ip-10-0-140-107 kubenswrapper[4939]: I1103 02:01:42.280413 4939 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-7b97bffd64-sw5r8"] Nov 03 02:01:42.376319 ip-10-0-140-107 kubenswrapper[4939]: I1103 02:01:42.376282 4939 generic.go:356] "Generic (PLEG): container finished" podID="3a058653-d7b4-4040-a7fd-7df7ea9d6f23" containerID="ee1403b4773c3303cf9197d9db553b8454f06246d72a8e29f1edcaf59ff30c94" exitCode=0 Nov 03 02:01:42.376498 ip-10-0-140-107 kubenswrapper[4939]: I1103 02:01:42.376352 4939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-55cc657fcb-54j7t" event={"ID":"3a058653-d7b4-4040-a7fd-7df7ea9d6f23","Type":"ContainerDied","Data":"ee1403b4773c3303cf9197d9db553b8454f06246d72a8e29f1edcaf59ff30c94"} Nov 03 02:01:42.427442 ip-10-0-140-107 kubenswrapper[4939]: I1103 02:01:42.427402 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b6183111-0ee0-4fc9-8a4c-9ae30021bead-metrics-certs\") pod \"maas-api-7b97bffd64-sw5r8\" (UID: \"b6183111-0ee0-4fc9-8a4c-9ae30021bead\") " pod="odh-ai-gateway-infra/maas-api-7b97bffd64-sw5r8" Nov 03 02:01:42.427654 ip-10-0-140-107 kubenswrapper[4939]: I1103 02:01:42.427455 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f469t\" (UniqueName: \"kubernetes.io/projected/b6183111-0ee0-4fc9-8a4c-9ae30021bead-kube-api-access-f469t\") pod \"maas-api-7b97bffd64-sw5r8\" (UID: \"b6183111-0ee0-4fc9-8a4c-9ae30021bead\") " pod="odh-ai-gateway-infra/maas-api-7b97bffd64-sw5r8" Nov 03 02:01:42.427654 ip-10-0-140-107 kubenswrapper[4939]: I1103 02:01:42.427531 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b6183111-0ee0-4fc9-8a4c-9ae30021bead-maas-api-tls\") pod \"maas-api-7b97bffd64-sw5r8\" (UID: \"b6183111-0ee0-4fc9-8a4c-9ae30021bead\") " pod="odh-ai-gateway-infra/maas-api-7b97bffd64-sw5r8" Nov 03 02:01:42.427654 ip-10-0-140-107 kubenswrapper[4939]: I1103 02:01:42.427584 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b6183111-0ee0-4fc9-8a4c-9ae30021bead-openshift-service-ca\") pod \"maas-api-7b97bffd64-sw5r8\" (UID: \"b6183111-0ee0-4fc9-8a4c-9ae30021bead\") " pod="odh-ai-gateway-infra/maas-api-7b97bffd64-sw5r8" Nov 03 02:01:42.474888 ip-10-0-140-107 kubenswrapper[4939]: I1103 02:01:42.474829 4939 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-55cc657fcb-54j7t" Nov 03 02:01:42.528644 ip-10-0-140-107 kubenswrapper[4939]: I1103 02:01:42.528612 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b6183111-0ee0-4fc9-8a4c-9ae30021bead-metrics-certs\") pod \"maas-api-7b97bffd64-sw5r8\" (UID: \"b6183111-0ee0-4fc9-8a4c-9ae30021bead\") " pod="odh-ai-gateway-infra/maas-api-7b97bffd64-sw5r8" Nov 03 02:01:42.528826 ip-10-0-140-107 kubenswrapper[4939]: I1103 02:01:42.528661 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f469t\" (UniqueName: \"kubernetes.io/projected/b6183111-0ee0-4fc9-8a4c-9ae30021bead-kube-api-access-f469t\") pod \"maas-api-7b97bffd64-sw5r8\" (UID: \"b6183111-0ee0-4fc9-8a4c-9ae30021bead\") " pod="odh-ai-gateway-infra/maas-api-7b97bffd64-sw5r8" Nov 03 02:01:42.528826 ip-10-0-140-107 kubenswrapper[4939]: I1103 02:01:42.528702 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b6183111-0ee0-4fc9-8a4c-9ae30021bead-maas-api-tls\") pod \"maas-api-7b97bffd64-sw5r8\" (UID: \"b6183111-0ee0-4fc9-8a4c-9ae30021bead\") " pod="odh-ai-gateway-infra/maas-api-7b97bffd64-sw5r8" Nov 03 02:01:42.528826 ip-10-0-140-107 kubenswrapper[4939]: I1103 02:01:42.528733 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b6183111-0ee0-4fc9-8a4c-9ae30021bead-openshift-service-ca\") pod \"maas-api-7b97bffd64-sw5r8\" (UID: \"b6183111-0ee0-4fc9-8a4c-9ae30021bead\") " pod="odh-ai-gateway-infra/maas-api-7b97bffd64-sw5r8" Nov 03 02:01:42.529447 ip-10-0-140-107 kubenswrapper[4939]: I1103 02:01:42.529416 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b6183111-0ee0-4fc9-8a4c-9ae30021bead-openshift-service-ca\") pod \"maas-api-7b97bffd64-sw5r8\" (UID: \"b6183111-0ee0-4fc9-8a4c-9ae30021bead\") " pod="odh-ai-gateway-infra/maas-api-7b97bffd64-sw5r8" Nov 03 02:01:42.531299 ip-10-0-140-107 kubenswrapper[4939]: I1103 02:01:42.531272 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b6183111-0ee0-4fc9-8a4c-9ae30021bead-maas-api-tls\") pod \"maas-api-7b97bffd64-sw5r8\" (UID: \"b6183111-0ee0-4fc9-8a4c-9ae30021bead\") " pod="odh-ai-gateway-infra/maas-api-7b97bffd64-sw5r8" Nov 03 02:01:42.531395 ip-10-0-140-107 kubenswrapper[4939]: I1103 02:01:42.531346 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b6183111-0ee0-4fc9-8a4c-9ae30021bead-metrics-certs\") pod \"maas-api-7b97bffd64-sw5r8\" (UID: \"b6183111-0ee0-4fc9-8a4c-9ae30021bead\") " pod="odh-ai-gateway-infra/maas-api-7b97bffd64-sw5r8" Nov 03 02:01:42.537971 ip-10-0-140-107 kubenswrapper[4939]: I1103 02:01:42.537937 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-f469t\" (UniqueName: \"kubernetes.io/projected/b6183111-0ee0-4fc9-8a4c-9ae30021bead-kube-api-access-f469t\") pod \"maas-api-7b97bffd64-sw5r8\" (UID: \"b6183111-0ee0-4fc9-8a4c-9ae30021bead\") " pod="odh-ai-gateway-infra/maas-api-7b97bffd64-sw5r8" Nov 03 02:01:42.608510 ip-10-0-140-107 kubenswrapper[4939]: I1103 02:01:42.608391 4939 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-7b97bffd64-sw5r8" Nov 03 02:01:42.629964 ip-10-0-140-107 kubenswrapper[4939]: I1103 02:01:42.629935 4939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3a058653-d7b4-4040-a7fd-7df7ea9d6f23-kube-api-access-gsr96\" (UniqueName: \"kubernetes.io/projected/3a058653-d7b4-4040-a7fd-7df7ea9d6f23-kube-api-access-gsr96\") pod \"3a058653-d7b4-4040-a7fd-7df7ea9d6f23\" (UID: \"3a058653-d7b4-4040-a7fd-7df7ea9d6f23\") " Nov 03 02:01:42.630098 ip-10-0-140-107 kubenswrapper[4939]: I1103 02:01:42.629992 4939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3a058653-d7b4-4040-a7fd-7df7ea9d6f23-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3a058653-d7b4-4040-a7fd-7df7ea9d6f23-maas-api-tls\") pod \"3a058653-d7b4-4040-a7fd-7df7ea9d6f23\" (UID: \"3a058653-d7b4-4040-a7fd-7df7ea9d6f23\") " Nov 03 02:01:42.630098 ip-10-0-140-107 kubenswrapper[4939]: I1103 02:01:42.630030 4939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3a058653-d7b4-4040-a7fd-7df7ea9d6f23-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3a058653-d7b4-4040-a7fd-7df7ea9d6f23-openshift-service-ca\") pod \"3a058653-d7b4-4040-a7fd-7df7ea9d6f23\" (UID: \"3a058653-d7b4-4040-a7fd-7df7ea9d6f23\") " Nov 03 02:01:42.630229 ip-10-0-140-107 kubenswrapper[4939]: I1103 02:01:42.630163 4939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3a058653-d7b4-4040-a7fd-7df7ea9d6f23-metrics-certs\" (UniqueName: \"kubernetes.io/secret/3a058653-d7b4-4040-a7fd-7df7ea9d6f23-metrics-certs\") pod \"3a058653-d7b4-4040-a7fd-7df7ea9d6f23\" (UID: \"3a058653-d7b4-4040-a7fd-7df7ea9d6f23\") " Nov 03 02:01:42.630467 ip-10-0-140-107 kubenswrapper[4939]: I1103 02:01:42.630428 4939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3a058653-d7b4-4040-a7fd-7df7ea9d6f23-openshift-service-ca" pod "3a058653-d7b4-4040-a7fd-7df7ea9d6f23" (UID: "3a058653-d7b4-4040-a7fd-7df7ea9d6f23"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 02:01:42.630601 ip-10-0-140-107 kubenswrapper[4939]: I1103 02:01:42.630571 4939 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3a058653-d7b4-4040-a7fd-7df7ea9d6f23-openshift-service-ca\") on node \"ip-10-0-140-107.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 02:01:42.632214 ip-10-0-140-107 kubenswrapper[4939]: I1103 02:01:42.632186 4939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3a058653-d7b4-4040-a7fd-7df7ea9d6f23-kube-api-access-gsr96" pod "3a058653-d7b4-4040-a7fd-7df7ea9d6f23" (UID: "3a058653-d7b4-4040-a7fd-7df7ea9d6f23"). InnerVolumeSpecName "kube-api-access-gsr96". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 02:01:42.632214 ip-10-0-140-107 kubenswrapper[4939]: I1103 02:01:42.632206 4939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3a058653-d7b4-4040-a7fd-7df7ea9d6f23-maas-api-tls" pod "3a058653-d7b4-4040-a7fd-7df7ea9d6f23" (UID: "3a058653-d7b4-4040-a7fd-7df7ea9d6f23"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 02:01:42.632401 ip-10-0-140-107 kubenswrapper[4939]: I1103 02:01:42.632224 4939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3a058653-d7b4-4040-a7fd-7df7ea9d6f23-metrics-certs" pod "3a058653-d7b4-4040-a7fd-7df7ea9d6f23" (UID: "3a058653-d7b4-4040-a7fd-7df7ea9d6f23"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 02:01:42.731864 ip-10-0-140-107 kubenswrapper[4939]: I1103 02:01:42.731782 4939 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3a058653-d7b4-4040-a7fd-7df7ea9d6f23-metrics-certs\") on node \"ip-10-0-140-107.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 02:01:42.731864 ip-10-0-140-107 kubenswrapper[4939]: I1103 02:01:42.731821 4939 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gsr96\" (UniqueName: \"kubernetes.io/projected/3a058653-d7b4-4040-a7fd-7df7ea9d6f23-kube-api-access-gsr96\") on node \"ip-10-0-140-107.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 02:01:42.731864 ip-10-0-140-107 kubenswrapper[4939]: I1103 02:01:42.731833 4939 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3a058653-d7b4-4040-a7fd-7df7ea9d6f23-maas-api-tls\") on node \"ip-10-0-140-107.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 02:01:42.884638 ip-10-0-140-107 kubenswrapper[4939]: I1103 02:01:42.884613 4939 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-7b97bffd64-sw5r8"] Nov 03 02:01:42.887727 ip-10-0-140-107 kubenswrapper[4939]: W1103 02:01:42.887529 4939 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb6183111_0ee0_4fc9_8a4c_9ae30021bead.slice/crio-78b1ccbab20b4834d99a994f134d41681dc08e93c0b0bf18c11dc769f6871966 WatchSource:0}: Error finding container 78b1ccbab20b4834d99a994f134d41681dc08e93c0b0bf18c11dc769f6871966: Status 404 returned error can't find the container with id 78b1ccbab20b4834d99a994f134d41681dc08e93c0b0bf18c11dc769f6871966 Nov 03 02:01:43.185153 ip-10-0-140-107 kubenswrapper[4939]: I1103 02:01:43.185102 4939 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-4pw7v"] Nov 03 02:01:43.382192 ip-10-0-140-107 kubenswrapper[4939]: I1103 02:01:43.382161 4939 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-55cc657fcb-54j7t" Nov 03 02:01:43.382683 ip-10-0-140-107 kubenswrapper[4939]: I1103 02:01:43.382198 4939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-55cc657fcb-54j7t" event={"ID":"3a058653-d7b4-4040-a7fd-7df7ea9d6f23","Type":"ContainerDied","Data":"2557c4deb8abba9ffbb70e186574dc53e50d43a1efdbc0c7f36d5ef046c1d542"} Nov 03 02:01:43.382683 ip-10-0-140-107 kubenswrapper[4939]: I1103 02:01:43.382275 4939 scope.go:122] "RemoveContainer" containerID="ee1403b4773c3303cf9197d9db553b8454f06246d72a8e29f1edcaf59ff30c94" Nov 03 02:01:43.383582 ip-10-0-140-107 kubenswrapper[4939]: I1103 02:01:43.383514 4939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-7b97bffd64-sw5r8" event={"ID":"b6183111-0ee0-4fc9-8a4c-9ae30021bead","Type":"ContainerStarted","Data":"78b1ccbab20b4834d99a994f134d41681dc08e93c0b0bf18c11dc769f6871966"} Nov 03 02:01:43.415318 ip-10-0-140-107 kubenswrapper[4939]: I1103 02:01:43.415291 4939 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-55cc657fcb-54j7t"] Nov 03 02:01:43.420686 ip-10-0-140-107 kubenswrapper[4939]: I1103 02:01:43.420663 4939 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-55cc657fcb-54j7t"] Nov 03 02:01:43.871054 ip-10-0-140-107 kubenswrapper[4939]: I1103 02:01:43.871004 4939 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3a058653-d7b4-4040-a7fd-7df7ea9d6f23" path="/var/lib/kubelet/pods/3a058653-d7b4-4040-a7fd-7df7ea9d6f23/volumes" Nov 03 02:01:44.418931 ip-10-0-140-107 kubenswrapper[4939]: I1103 02:01:44.414590 4939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-7b97bffd64-sw5r8" event={"ID":"b6183111-0ee0-4fc9-8a4c-9ae30021bead","Type":"ContainerStarted","Data":"320b239a0efe5ae1d56ae591e20db8c314dc16e75ce06b1d2cb81082e1a8de9b"} Nov 03 02:01:44.418931 ip-10-0-140-107 kubenswrapper[4939]: I1103 02:01:44.415583 4939 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-7b97bffd64-sw5r8" Nov 03 02:01:44.437860 ip-10-0-140-107 kubenswrapper[4939]: I1103 02:01:44.435738 4939 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-7b97bffd64-sw5r8" podStartSLOduration=1.9178509529999999 podStartE2EDuration="2.435719508s" totalImagesPullingTime="517.868555ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 02:01:42 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 02:01:44.434752527 +0000 UTC m=+4852.699087472" watchObservedRunningTime="2026-11-03 02:01:44.435719508 +0000 UTC m=+4852.700054455" Nov 03 02:01:45.511933 ip-10-0-140-107 kubenswrapper[4939]: I1103 02:01:45.509010 4939 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-596dc9545f-v5p6s container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.69:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 02:01:45.511933 ip-10-0-140-107 kubenswrapper[4939]: I1103 02:01:45.509075 4939 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-v5p6s" podUID="e6aa4af1-9a04-4116-9d8a-1d8ef5b04958" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.69:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 02:01:47.034231 ip-10-0-140-107 kubenswrapper[4939]: I1103 02:01:47.034197 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 02:01:48.043152 ip-10-0-140-107 kubenswrapper[4939]: E1103 02:01:48.043107 4939 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-sthjh" containerName="extractor" Nov 03 02:01:51.190423 ip-10-0-140-107 kubenswrapper[4939]: I1103 02:01:51.190385 4939 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9903c75e-8ck88"] Nov 03 02:01:51.190968 ip-10-0-140-107 kubenswrapper[4939]: I1103 02:01:51.190892 4939 cpu_manager.go:414] "containerMap: removing container" podUID="3a058653-d7b4-4040-a7fd-7df7ea9d6f23" containerName="maas-api" Nov 03 02:01:51.190968 ip-10-0-140-107 kubenswrapper[4939]: I1103 02:01:51.190922 4939 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3a058653-d7b4-4040-a7fd-7df7ea9d6f23" containerName="maas-api" Nov 03 02:01:51.191070 ip-10-0-140-107 kubenswrapper[4939]: I1103 02:01:51.191001 4939 memory_manager.go:365] "RemoveStaleState removing state" podUID="3a058653-d7b4-4040-a7fd-7df7ea9d6f23" containerName="maas-api" Nov 03 02:01:51.193959 ip-10-0-140-107 kubenswrapper[4939]: I1103 02:01:51.193941 4939 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9903c75e-8ck88" Nov 03 02:01:51.200842 ip-10-0-140-107 kubenswrapper[4939]: I1103 02:01:51.200436 4939 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 02:01:51.200842 ip-10-0-140-107 kubenswrapper[4939]: I1103 02:01:51.200507 4939 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-swfcr\"" Nov 03 02:01:51.202096 ip-10-0-140-107 kubenswrapper[4939]: I1103 02:01:51.202069 4939 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9903c75e-8ck88"] Nov 03 02:01:51.285490 ip-10-0-140-107 kubenswrapper[4939]: I1103 02:01:51.285451 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1d3f06f9-e048-456b-93df-7cd7c2d00f42-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9903c75e-8ck88\" (UID: \"1d3f06f9-e048-456b-93df-7cd7c2d00f42\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9903c75e-8ck88" Nov 03 02:01:51.285490 ip-10-0-140-107 kubenswrapper[4939]: I1103 02:01:51.285494 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rphlj\" (UniqueName: \"kubernetes.io/projected/1d3f06f9-e048-456b-93df-7cd7c2d00f42-kube-api-access-rphlj\") pod \"maas-api-revoke-sub-9903c75e-8ck88\" (UID: \"1d3f06f9-e048-456b-93df-7cd7c2d00f42\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9903c75e-8ck88" Nov 03 02:01:51.285740 ip-10-0-140-107 kubenswrapper[4939]: I1103 02:01:51.285526 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1d3f06f9-e048-456b-93df-7cd7c2d00f42-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9903c75e-8ck88\" (UID: \"1d3f06f9-e048-456b-93df-7cd7c2d00f42\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9903c75e-8ck88" Nov 03 02:01:51.386148 ip-10-0-140-107 kubenswrapper[4939]: I1103 02:01:51.386110 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1d3f06f9-e048-456b-93df-7cd7c2d00f42-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9903c75e-8ck88\" (UID: \"1d3f06f9-e048-456b-93df-7cd7c2d00f42\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9903c75e-8ck88" Nov 03 02:01:51.386148 ip-10-0-140-107 kubenswrapper[4939]: I1103 02:01:51.386149 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rphlj\" (UniqueName: \"kubernetes.io/projected/1d3f06f9-e048-456b-93df-7cd7c2d00f42-kube-api-access-rphlj\") pod \"maas-api-revoke-sub-9903c75e-8ck88\" (UID: \"1d3f06f9-e048-456b-93df-7cd7c2d00f42\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9903c75e-8ck88" Nov 03 02:01:51.386430 ip-10-0-140-107 kubenswrapper[4939]: I1103 02:01:51.386180 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1d3f06f9-e048-456b-93df-7cd7c2d00f42-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9903c75e-8ck88\" (UID: \"1d3f06f9-e048-456b-93df-7cd7c2d00f42\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9903c75e-8ck88" Nov 03 02:01:51.386859 ip-10-0-140-107 kubenswrapper[4939]: I1103 02:01:51.386837 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1d3f06f9-e048-456b-93df-7cd7c2d00f42-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9903c75e-8ck88\" (UID: \"1d3f06f9-e048-456b-93df-7cd7c2d00f42\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9903c75e-8ck88" Nov 03 02:01:51.388682 ip-10-0-140-107 kubenswrapper[4939]: I1103 02:01:51.388655 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1d3f06f9-e048-456b-93df-7cd7c2d00f42-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9903c75e-8ck88\" (UID: \"1d3f06f9-e048-456b-93df-7cd7c2d00f42\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9903c75e-8ck88" Nov 03 02:01:51.394594 ip-10-0-140-107 kubenswrapper[4939]: I1103 02:01:51.394566 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rphlj\" (UniqueName: \"kubernetes.io/projected/1d3f06f9-e048-456b-93df-7cd7c2d00f42-kube-api-access-rphlj\") pod \"maas-api-revoke-sub-9903c75e-8ck88\" (UID: \"1d3f06f9-e048-456b-93df-7cd7c2d00f42\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9903c75e-8ck88" Nov 03 02:01:51.440507 ip-10-0-140-107 kubenswrapper[4939]: I1103 02:01:51.440434 4939 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-7b97bffd64-sw5r8" Nov 03 02:01:51.508984 ip-10-0-140-107 kubenswrapper[4939]: I1103 02:01:51.508944 4939 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9903c75e-8ck88" Nov 03 02:01:51.789051 ip-10-0-140-107 kubenswrapper[4939]: I1103 02:01:51.789000 4939 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9903c75e-8ck88"] Nov 03 02:01:51.791499 ip-10-0-140-107 kubenswrapper[4939]: W1103 02:01:51.791471 4939 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1d3f06f9_e048_456b_93df_7cd7c2d00f42.slice/crio-ca1bb794da486469f4d5105941b434c813258a551b2094776f47f8ca888e781c WatchSource:0}: Error finding container ca1bb794da486469f4d5105941b434c813258a551b2094776f47f8ca888e781c: Status 404 returned error can't find the container with id ca1bb794da486469f4d5105941b434c813258a551b2094776f47f8ca888e781c Nov 03 02:01:52.495197 ip-10-0-140-107 kubenswrapper[4939]: I1103 02:01:52.495154 4939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9903c75e-8ck88" event={"ID":"1d3f06f9-e048-456b-93df-7cd7c2d00f42","Type":"ContainerStarted","Data":"117227a9f18d60e9fc8b40bca9f34ff7b30759f66fa12513dec76164b66a1324"} Nov 03 02:01:52.495626 ip-10-0-140-107 kubenswrapper[4939]: I1103 02:01:52.495243 4939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9903c75e-8ck88" event={"ID":"1d3f06f9-e048-456b-93df-7cd7c2d00f42","Type":"ContainerStarted","Data":"ca1bb794da486469f4d5105941b434c813258a551b2094776f47f8ca888e781c"} Nov 03 02:01:52.512659 ip-10-0-140-107 kubenswrapper[4939]: I1103 02:01:52.512605 4939 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9903c75e-8ck88" podStartSLOduration=1.512592884 podStartE2EDuration="1.512592884s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 02:01:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 02:01:52.510428001 +0000 UTC m=+4860.774762946" watchObservedRunningTime="2026-11-03 02:01:52.512592884 +0000 UTC m=+4860.776927828" Nov 03 02:01:53.501899 ip-10-0-140-107 kubenswrapper[4939]: I1103 02:01:53.501855 4939 generic.go:356] "Generic (PLEG): container finished" podID="1d3f06f9-e048-456b-93df-7cd7c2d00f42" containerID="117227a9f18d60e9fc8b40bca9f34ff7b30759f66fa12513dec76164b66a1324" exitCode=0 Nov 03 02:01:53.502361 ip-10-0-140-107 kubenswrapper[4939]: I1103 02:01:53.501948 4939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9903c75e-8ck88" event={"ID":"1d3f06f9-e048-456b-93df-7cd7c2d00f42","Type":"ContainerDied","Data":"117227a9f18d60e9fc8b40bca9f34ff7b30759f66fa12513dec76164b66a1324"} Nov 03 02:01:54.643504 ip-10-0-140-107 kubenswrapper[4939]: I1103 02:01:54.643480 4939 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9903c75e-8ck88" Nov 03 02:01:54.718418 ip-10-0-140-107 kubenswrapper[4939]: I1103 02:01:54.718382 4939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1d3f06f9-e048-456b-93df-7cd7c2d00f42-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1d3f06f9-e048-456b-93df-7cd7c2d00f42-maas-api-openshift-ca\") pod \"1d3f06f9-e048-456b-93df-7cd7c2d00f42\" (UID: \"1d3f06f9-e048-456b-93df-7cd7c2d00f42\") " Nov 03 02:01:54.718601 ip-10-0-140-107 kubenswrapper[4939]: I1103 02:01:54.718443 4939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1d3f06f9-e048-456b-93df-7cd7c2d00f42-kube-api-access-rphlj\" (UniqueName: \"kubernetes.io/projected/1d3f06f9-e048-456b-93df-7cd7c2d00f42-kube-api-access-rphlj\") pod \"1d3f06f9-e048-456b-93df-7cd7c2d00f42\" (UID: \"1d3f06f9-e048-456b-93df-7cd7c2d00f42\") " Nov 03 02:01:54.718601 ip-10-0-140-107 kubenswrapper[4939]: I1103 02:01:54.718495 4939 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1d3f06f9-e048-456b-93df-7cd7c2d00f42-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1d3f06f9-e048-456b-93df-7cd7c2d00f42-maas-api-xks-ca\") pod \"1d3f06f9-e048-456b-93df-7cd7c2d00f42\" (UID: \"1d3f06f9-e048-456b-93df-7cd7c2d00f42\") " Nov 03 02:01:54.718831 ip-10-0-140-107 kubenswrapper[4939]: I1103 02:01:54.718806 4939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1d3f06f9-e048-456b-93df-7cd7c2d00f42-maas-api-openshift-ca" pod "1d3f06f9-e048-456b-93df-7cd7c2d00f42" (UID: "1d3f06f9-e048-456b-93df-7cd7c2d00f42"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 02:01:54.718984 ip-10-0-140-107 kubenswrapper[4939]: I1103 02:01:54.718967 4939 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1d3f06f9-e048-456b-93df-7cd7c2d00f42-maas-api-openshift-ca\") on node \"ip-10-0-140-107.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 02:01:54.720708 ip-10-0-140-107 kubenswrapper[4939]: I1103 02:01:54.720686 4939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1d3f06f9-e048-456b-93df-7cd7c2d00f42-maas-api-xks-ca" pod "1d3f06f9-e048-456b-93df-7cd7c2d00f42" (UID: "1d3f06f9-e048-456b-93df-7cd7c2d00f42"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 02:01:54.720786 ip-10-0-140-107 kubenswrapper[4939]: I1103 02:01:54.720731 4939 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1d3f06f9-e048-456b-93df-7cd7c2d00f42-kube-api-access-rphlj" pod "1d3f06f9-e048-456b-93df-7cd7c2d00f42" (UID: "1d3f06f9-e048-456b-93df-7cd7c2d00f42"). InnerVolumeSpecName "kube-api-access-rphlj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 02:01:54.819770 ip-10-0-140-107 kubenswrapper[4939]: I1103 02:01:54.819692 4939 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rphlj\" (UniqueName: \"kubernetes.io/projected/1d3f06f9-e048-456b-93df-7cd7c2d00f42-kube-api-access-rphlj\") on node \"ip-10-0-140-107.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 02:01:54.819770 ip-10-0-140-107 kubenswrapper[4939]: I1103 02:01:54.819723 4939 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1d3f06f9-e048-456b-93df-7cd7c2d00f42-maas-api-xks-ca\") on node \"ip-10-0-140-107.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 02:01:55.512447 ip-10-0-140-107 kubenswrapper[4939]: I1103 02:01:55.512408 4939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9903c75e-8ck88" event={"ID":"1d3f06f9-e048-456b-93df-7cd7c2d00f42","Type":"ContainerDied","Data":"ca1bb794da486469f4d5105941b434c813258a551b2094776f47f8ca888e781c"} Nov 03 02:01:55.512447 ip-10-0-140-107 kubenswrapper[4939]: I1103 02:01:55.512443 4939 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9903c75e-8ck88" Nov 03 02:01:55.512700 ip-10-0-140-107 kubenswrapper[4939]: I1103 02:01:55.512461 4939 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ca1bb794da486469f4d5105941b434c813258a551b2094776f47f8ca888e781c" Nov 03 02:02:04.902980 ip-10-0-140-107 kubenswrapper[4939]: I1103 02:02:04.902938 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 02:02:12.801548 ip-10-0-140-107 kubenswrapper[4939]: I1103 02:02:12.801509 4939 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-4pw7v"] Nov 03 02:02:15.796567 ip-10-0-140-107 kubenswrapper[4939]: I1103 02:02:15.796531 4939 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-4pw7v"] Nov 03 02:02:18.147276 ip-10-0-140-107 kubenswrapper[4939]: E1103 02:02:18.147229 4939 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-sthjh" containerName="extractor" Nov 03 02:02:33.326601 ip-10-0-140-107 kubenswrapper[4939]: I1103 02:02:33.326513 4939 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-4pw7v"] Nov 03 02:02:38.411933 ip-10-0-140-107 kubenswrapper[4939]: I1103 02:02:38.411749 4939 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-4pw7v"] Nov 03 02:02:47.063474 ip-10-0-140-107 kubenswrapper[4939]: I1103 02:02:47.063430 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 02:02:48.029750 ip-10-0-140-107 kubenswrapper[4939]: E1103 02:02:48.029710 4939 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-sthjh" containerName="extractor" Nov 03 02:02:59.683053 ip-10-0-140-107 kubenswrapper[4939]: I1103 02:02:59.683015 4939 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-4pw7v"] Nov 03 02:03:09.923876 ip-10-0-140-107 kubenswrapper[4939]: I1103 02:03:09.923844 4939 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-4pw7v"] Nov 03 02:03:13.003221 ip-10-0-140-107 kubenswrapper[4939]: I1103 02:03:13.003180 4939 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-4pw7v"] Nov 03 02:03:15.511529 ip-10-0-140-107 kubenswrapper[4939]: I1103 02:03:15.511019 4939 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-596dc9545f-v5p6s container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.69:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 02:03:15.511529 ip-10-0-140-107 kubenswrapper[4939]: I1103 02:03:15.511101 4939 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-v5p6s" podUID="e6aa4af1-9a04-4116-9d8a-1d8ef5b04958" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.69:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 02:03:18.171206 ip-10-0-140-107 kubenswrapper[4939]: E1103 02:03:18.171144 4939 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-sthjh" containerName="extractor" Nov 03 02:03:19.900546 ip-10-0-140-107 kubenswrapper[4939]: I1103 02:03:19.900510 4939 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-4pw7v"] Nov 03 02:03:20.024670 ip-10-0-140-107 kubenswrapper[4939]: I1103 02:03:20.024639 4939 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6876bd20-qn6mv"] Nov 03 02:03:20.027809 ip-10-0-140-107 kubenswrapper[4939]: I1103 02:03:20.027781 4939 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6876bd20-qn6mv"] Nov 03 02:03:21.906622 ip-10-0-140-107 kubenswrapper[4939]: I1103 02:03:21.906579 4939 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d6b552a0-b9c4-42cc-a0b4-cee73867e470" path="/var/lib/kubelet/pods/d6b552a0-b9c4-42cc-a0b4-cee73867e470/volumes" Nov 03 02:03:36.489447 ip-10-0-140-107 kubenswrapper[4939]: I1103 02:03:36.489413 4939 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-4pw7v"] Nov 03 02:03:46.613658 ip-10-0-140-107 kubenswrapper[4939]: I1103 02:03:46.613623 4939 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-4pw7v"] Nov 03 02:03:47.140008 ip-10-0-140-107 kubenswrapper[4939]: I1103 02:03:47.139959 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 02:03:48.029424 ip-10-0-140-107 kubenswrapper[4939]: E1103 02:03:48.029390 4939 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-sthjh" containerName="extractor" Nov 03 02:03:50.164934 ip-10-0-140-107 kubenswrapper[4939]: I1103 02:03:50.161891 4939 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-4pw7v"] Nov 03 02:03:58.699811 ip-10-0-140-107 kubenswrapper[4939]: I1103 02:03:58.699760 4939 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-4pw7v"] Nov 03 02:04:14.308253 ip-10-0-140-107 kubenswrapper[4939]: I1103 02:04:14.308165 4939 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-4pw7v"] Nov 03 02:04:18.029807 ip-10-0-140-107 kubenswrapper[4939]: E1103 02:04:18.029762 4939 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-sthjh" containerName="extractor" Nov 03 02:04:47.291720 ip-10-0-140-107 kubenswrapper[4939]: I1103 02:04:47.291683 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 02:04:47.653157 ip-10-0-140-107 kubenswrapper[4939]: I1103 02:04:47.653081 4939 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-596dc9545f-vm5w5 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.1.20:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 02:04:47.653398 ip-10-0-140-107 kubenswrapper[4939]: I1103 02:04:47.653171 4939 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-vm5w5" podUID="9a335557-68f1-44ab-b673-f9e0a18be917" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.20:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 02:04:48.030332 ip-10-0-140-107 kubenswrapper[4939]: E1103 02:04:48.030236 4939 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-sthjh" containerName="extractor" Nov 03 02:04:57.448063 ip-10-0-140-107 kubenswrapper[4939]: I1103 02:04:57.448018 4939 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-596dc9545f-9jvbb container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.41:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 02:04:57.448725 ip-10-0-140-107 kubenswrapper[4939]: I1103 02:04:57.448680 4939 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-9jvbb" podUID="512d26b5-8ad9-4cca-8687-cd06d31d8369" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.41:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 02:04:59.876224 ip-10-0-140-107 kubenswrapper[4939]: I1103 02:04:59.876184 4939 scope.go:122] "RemoveContainer" containerID="150a640998bd75d7365b66716a268353c08ac534fcf2c8c414498f72e5188d00" Nov 03 02:05:17.680933 ip-10-0-140-107 kubenswrapper[4939]: I1103 02:05:17.673027 4939 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-596dc9545f-vm5w5 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.1.20:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 02:05:17.680933 ip-10-0-140-107 kubenswrapper[4939]: I1103 02:05:17.673125 4939 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-vm5w5" podUID="9a335557-68f1-44ab-b673-f9e0a18be917" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.20:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 02:05:18.196582 ip-10-0-140-107 kubenswrapper[4939]: E1103 02:05:18.196453 4939 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-sthjh" containerName="extractor" Nov 03 02:05:21.521922 ip-10-0-140-107 kubenswrapper[4939]: I1103 02:05:21.521887 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 02:05:21.640694 ip-10-0-140-107 kubenswrapper[4939]: I1103 02:05:21.640655 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 02:05:47.533681 ip-10-0-140-107 kubenswrapper[4939]: I1103 02:05:47.533477 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 02:05:48.339222 ip-10-0-140-107 kubenswrapper[4939]: E1103 02:05:48.339160 4939 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-sthjh" containerName="extractor" Nov 03 02:05:53.322748 ip-10-0-140-107 kubenswrapper[4939]: I1103 02:05:53.322716 4939 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-n42s6_65d134ac-8a25-4c67-b6c7-cd752a098daf/console-operator/2.log" Nov 03 02:05:53.328941 ip-10-0-140-107 kubenswrapper[4939]: I1103 02:05:53.328919 4939 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-n42s6_65d134ac-8a25-4c67-b6c7-cd752a098daf/console-operator/2.log" Nov 03 02:05:59.894885 ip-10-0-140-107 kubenswrapper[4939]: I1103 02:05:59.894850 4939 scope.go:122] "RemoveContainer" containerID="182b49b73a53963e8921f9a4d3a502954e39c240ecaa331162eb652a08c5f378" Nov 03 02:06:06.174090 ip-10-0-140-107 kubenswrapper[4939]: I1103 02:06:06.174048 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 02:06:06.190069 ip-10-0-140-107 kubenswrapper[4939]: I1103 02:06:06.190040 4939 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-4pw7v"] Nov 03 02:06:12.548667 ip-10-0-140-107 kubenswrapper[4939]: I1103 02:06:12.548629 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 02:06:18.030342 ip-10-0-140-107 kubenswrapper[4939]: E1103 02:06:18.030299 4939 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-sthjh" containerName="extractor" Nov 03 02:06:27.375647 ip-10-0-140-107 kubenswrapper[4939]: I1103 02:06:27.375612 4939 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_ai-gateway-controller-75bbf6968c-fz65x_f690a187-3701-4c28-a425-e5e1f447b650/manager/0.log" Nov 03 02:06:28.215882 ip-10-0-140-107 kubenswrapper[4939]: I1103 02:06:28.215831 4939 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-7b85f9594f-xg8nb_1be824b5-4745-41c9-856c-1dc0c4b531d4/manager/0.log" Nov 03 02:06:30.095050 ip-10-0-140-107 kubenswrapper[4939]: I1103 02:06:30.095019 4939 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-7b97bffd64-sw5r8_b6183111-0ee0-4fc9-8a4c-9ae30021bead/maas-api/0.log" Nov 03 02:06:32.470990 ip-10-0-140-107 kubenswrapper[4939]: I1103 02:06:32.470957 4939 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-9903c75e-8ck88_1d3f06f9-e048-456b-93df-7cd7c2d00f42/revoke-keys/0.log" Nov 03 02:06:36.587177 ip-10-0-140-107 kubenswrapper[4939]: I1103 02:06:36.587144 4939 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-7c9b87d574-4pw7v_e1b47e77-a4ab-4d61-ace8-deeea43ea944/limitador/0.log" Nov 03 02:06:38.040635 ip-10-0-140-107 kubenswrapper[4939]: I1103 02:06:38.040598 4939 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-7ddd9cfdc8-jkd9f_d9dfeb7b-ac37-456e-8239-43505d7f0ccb/discovery/0.log" Nov 03 02:06:38.419624 ip-10-0-140-107 kubenswrapper[4939]: I1103 02:06:38.419591 4939 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-594b68584c-km4bh_5acaee44-b6e2-4504-aebc-aa853d707830/kube-auth-proxy/0.log" Nov 03 02:06:38.789141 ip-10-0-140-107 kubenswrapper[4939]: I1103 02:06:38.789067 4939 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-596dc9545f-7kxj4_8acf90f6-6fff-4a69-b67d-c3ba365c6103/istio-proxy/0.log" Nov 03 02:06:38.978513 ip-10-0-140-107 kubenswrapper[4939]: I1103 02:06:38.978487 4939 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-596dc9545f-9jvbb_512d26b5-8ad9-4cca-8687-cd06d31d8369/istio-proxy/0.log" Nov 03 02:06:40.140138 ip-10-0-140-107 kubenswrapper[4939]: I1103 02:06:40.140104 4939 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-596dc9545f-v5p6s_e6aa4af1-9a04-4116-9d8a-1d8ef5b04958/istio-proxy/0.log" Nov 03 02:06:40.323347 ip-10-0-140-107 kubenswrapper[4939]: I1103 02:06:40.323311 4939 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-596dc9545f-vm5w5_9a335557-68f1-44ab-b673-f9e0a18be917/istio-proxy/0.log" Nov 03 02:06:40.677388 ip-10-0-140-107 kubenswrapper[4939]: I1103 02:06:40.677358 4939 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-59b8dd6b64-jcv9p_dfb7b62c-1cf4-4bf0-b5d3-df882d0078d4/payload-pre-processing/0.log" Nov 03 02:06:40.843600 ip-10-0-140-107 kubenswrapper[4939]: I1103 02:06:40.843570 4939 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-754bb5cbcf-77zpm_8d256a28-b126-4673-bd1e-63aaef592334/payload-processing/0.log" Nov 03 02:06:41.036141 ip-10-0-140-107 kubenswrapper[4939]: I1103 02:06:41.036061 4939 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-d7798ccf5-442kt_cf217a21-c1bd-48be-b203-f59743bceb33/router/0.log" Nov 03 02:06:42.011089 ip-10-0-140-107 kubenswrapper[4939]: I1103 02:06:42.011059 4939 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-distinct-2-simulated-kserve-cbd4f5586-kksqt_08eb2c0e-2bd2-4a9e-815e-da525dbd5f32/main/0.log" Nov 03 02:06:42.384622 ip-10-0-140-107 kubenswrapper[4939]: I1103 02:06:42.384591 4939 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-embedding-simulated-kserve-67bf594cd6-fp9vz_2e990a12-7eac-4d1a-84f1-05fe60a93761/main/0.log" Nov 03 02:06:42.770310 ip-10-0-140-107 kubenswrapper[4939]: I1103 02:06:42.770233 4939 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8dx487_33fe945b-76e3-4201-970d-235ff8f719a3/main/0.log" Nov 03 02:06:43.145486 ip-10-0-140-107 kubenswrapper[4939]: I1103 02:06:43.145456 4939 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_facebook-opt-125m-simulated-kserve-84b4ff656c-xggrt_9f0874ba-e3aa-4036-ad25-b2b97bf5ddc4/main/0.log" Nov 03 02:06:48.029604 ip-10-0-140-107 kubenswrapper[4939]: E1103 02:06:48.029563 4939 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-sthjh" containerName="extractor" Nov 03 02:06:49.535117 ip-10-0-140-107 kubenswrapper[4939]: I1103 02:06:49.535078 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 02:06:53.622246 ip-10-0-140-107 kubenswrapper[4939]: I1103 02:06:53.622208 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 02:06:53.747939 ip-10-0-140-107 kubenswrapper[4939]: I1103 02:06:53.747879 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 02:06:53.862677 ip-10-0-140-107 kubenswrapper[4939]: I1103 02:06:53.862643 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 02:06:55.021806 ip-10-0-140-107 kubenswrapper[4939]: I1103 02:06:55.021733 4939 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9903c75e-8ck88"] Nov 03 02:06:55.024164 ip-10-0-140-107 kubenswrapper[4939]: I1103 02:06:55.024141 4939 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9903c75e-8ck88"] Nov 03 02:06:55.866359 ip-10-0-140-107 kubenswrapper[4939]: I1103 02:06:55.866319 4939 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1d3f06f9-e048-456b-93df-7cd7c2d00f42" path="/var/lib/kubelet/pods/1d3f06f9-e048-456b-93df-7cd7c2d00f42/volumes" Nov 03 02:06:57.666604 ip-10-0-140-107 kubenswrapper[4939]: I1103 02:06:57.666576 4939 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-jv7cw_cabd402c-79c5-4699-9c85-d415d362b4d0/global-pull-secret-syncer/0.log" Nov 03 02:06:57.707423 ip-10-0-140-107 kubenswrapper[4939]: I1103 02:06:57.707396 4939 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kas-connection-checker-685895c7c9-ntqw6_b0bd4bd5-12ca-49df-a3d9-4243f4bd642e/connection-checker/0.log" Nov 03 02:06:57.734348 ip-10-0-140-107 kubenswrapper[4939]: I1103 02:06:57.734324 4939 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kas-connection-checker-685895c7c9-xst6j_29a64a08-67ef-4c82-8a82-8cd4a4675327/connection-checker/0.log" Nov 03 02:06:57.761562 ip-10-0-140-107 kubenswrapper[4939]: I1103 02:06:57.761537 4939 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kas-connection-checker-685895c7c9-zsd49_4237805a-d475-4bd7-865c-c6ef371c0e11/connection-checker/0.log" Nov 03 02:06:57.817621 ip-10-0-140-107 kubenswrapper[4939]: I1103 02:06:57.817591 4939 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" Nov 03 02:06:58.028963 ip-10-0-140-107 kubenswrapper[4939]: I1103 02:06:58.028866 4939 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-140-107.us-east-2.compute.internal_a730b11d5a720feea75c07e6e10d3b92/haproxy/0.log" Nov 03 02:07:01.202223 ip-10-0-140-107 kubenswrapper[4939]: I1103 02:07:01.202192 4939 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_ai-gateway-controller-75bbf6968c-fz65x_f690a187-3701-4c28-a425-e5e1f447b650/manager/0.log" Nov 03 02:07:02.293835 ip-10-0-140-107 kubenswrapper[4939]: I1103 02:07:02.293803 4939 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-7b85f9594f-xg8nb_1be824b5-4745-41c9-856c-1dc0c4b531d4/manager/0.log" Nov 03 02:07:04.319500 ip-10-0-140-107 kubenswrapper[4939]: I1103 02:07:04.319419 4939 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_ai-gateway-controller-75bbf6968c-fz65x_f690a187-3701-4c28-a425-e5e1f447b650/manager/0.log" Nov 03 02:07:05.481786 ip-10-0-140-107 kubenswrapper[4939]: I1103 02:07:05.481745 4939 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-7b85f9594f-xg8nb_1be824b5-4745-41c9-856c-1dc0c4b531d4/manager/0.log" Nov 03 02:07:05.947687 ip-10-0-140-107 kubenswrapper[4939]: I1103 02:07:05.947645 4939 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-7c9b87d574-4pw7v_e1b47e77-a4ab-4d61-ace8-deeea43ea944/limitador/0.log" Nov 03 02:07:07.762511 ip-10-0-140-107 kubenswrapper[4939]: I1103 02:07:07.762476 4939 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_ai-gateway-controller-75bbf6968c-fz65x_f690a187-3701-4c28-a425-e5e1f447b650/manager/0.log" Nov 03 02:07:08.082868 ip-10-0-140-107 kubenswrapper[4939]: I1103 02:07:08.082842 4939 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_cluster-monitoring-operator-884576c57-4bm4f_06973556-f51c-4d1c-ae55-fb77699bc5b3/cluster-monitoring-operator/0.log" Nov 03 02:07:08.485146 ip-10-0-140-107 kubenswrapper[4939]: I1103 02:07:08.485072 4939 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-rstd6_49e75d56-a78d-4c7f-88e3-c96e81dc026e/node-exporter/0.log" Nov 03 02:07:08.511151 ip-10-0-140-107 kubenswrapper[4939]: I1103 02:07:08.511119 4939 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-rstd6_49e75d56-a78d-4c7f-88e3-c96e81dc026e/kube-rbac-proxy/0.log" Nov 03 02:07:08.542203 ip-10-0-140-107 kubenswrapper[4939]: I1103 02:07:08.542178 4939 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-rstd6_49e75d56-a78d-4c7f-88e3-c96e81dc026e/init-textfile/0.log" Nov 03 02:07:08.945431 ip-10-0-140-107 kubenswrapper[4939]: I1103 02:07:08.945399 4939 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-7b85f9594f-xg8nb_1be824b5-4745-41c9-856c-1dc0c4b531d4/manager/0.log" Nov 03 02:07:08.981762 ip-10-0-140-107 kubenswrapper[4939]: I1103 02:07:08.981726 4939 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-must-gather-grd2d/perf-node-gather-daemonset-bkj9w"] Nov 03 02:07:08.982277 ip-10-0-140-107 kubenswrapper[4939]: I1103 02:07:08.982253 4939 cpu_manager.go:414] "containerMap: removing container" podUID="1d3f06f9-e048-456b-93df-7cd7c2d00f42" containerName="revoke-keys" Nov 03 02:07:08.982381 ip-10-0-140-107 kubenswrapper[4939]: I1103 02:07:08.982277 4939 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1d3f06f9-e048-456b-93df-7cd7c2d00f42" containerName="revoke-keys" Nov 03 02:07:08.982451 ip-10-0-140-107 kubenswrapper[4939]: I1103 02:07:08.982438 4939 memory_manager.go:365] "RemoveStaleState removing state" podUID="1d3f06f9-e048-456b-93df-7cd7c2d00f42" containerName="revoke-keys" Nov 03 02:07:08.985547 ip-10-0-140-107 kubenswrapper[4939]: I1103 02:07:08.985526 4939 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-grd2d/perf-node-gather-daemonset-bkj9w" Nov 03 02:07:08.995265 ip-10-0-140-107 kubenswrapper[4939]: I1103 02:07:08.995237 4939 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-grd2d/perf-node-gather-daemonset-bkj9w"] Nov 03 02:07:09.083875 ip-10-0-140-107 kubenswrapper[4939]: I1103 02:07:09.083848 4939 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-grd2d\"/\"default-dockercfg-wj6sf\"" Nov 03 02:07:09.086632 ip-10-0-140-107 kubenswrapper[4939]: I1103 02:07:09.086613 4939 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-grd2d\"/\"kube-root-ca.crt\"" Nov 03 02:07:09.086717 ip-10-0-140-107 kubenswrapper[4939]: I1103 02:07:09.086669 4939 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-grd2d\"/\"openshift-service-ca.crt\"" Nov 03 02:07:09.145089 ip-10-0-140-107 kubenswrapper[4939]: I1103 02:07:09.145057 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/0bfcb9f9-ab6d-4e86-bd78-88da24aee0d2-sys\") pod \"perf-node-gather-daemonset-bkj9w\" (UID: \"0bfcb9f9-ab6d-4e86-bd78-88da24aee0d2\") " pod="openshift-must-gather-grd2d/perf-node-gather-daemonset-bkj9w" Nov 03 02:07:09.145266 ip-10-0-140-107 kubenswrapper[4939]: I1103 02:07:09.145114 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tzvc4\" (UniqueName: \"kubernetes.io/projected/0bfcb9f9-ab6d-4e86-bd78-88da24aee0d2-kube-api-access-tzvc4\") pod \"perf-node-gather-daemonset-bkj9w\" (UID: \"0bfcb9f9-ab6d-4e86-bd78-88da24aee0d2\") " pod="openshift-must-gather-grd2d/perf-node-gather-daemonset-bkj9w" Nov 03 02:07:09.145266 ip-10-0-140-107 kubenswrapper[4939]: I1103 02:07:09.145140 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/0bfcb9f9-ab6d-4e86-bd78-88da24aee0d2-lib-modules\") pod \"perf-node-gather-daemonset-bkj9w\" (UID: \"0bfcb9f9-ab6d-4e86-bd78-88da24aee0d2\") " pod="openshift-must-gather-grd2d/perf-node-gather-daemonset-bkj9w" Nov 03 02:07:09.145399 ip-10-0-140-107 kubenswrapper[4939]: I1103 02:07:09.145284 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/0bfcb9f9-ab6d-4e86-bd78-88da24aee0d2-podres\") pod \"perf-node-gather-daemonset-bkj9w\" (UID: \"0bfcb9f9-ab6d-4e86-bd78-88da24aee0d2\") " pod="openshift-must-gather-grd2d/perf-node-gather-daemonset-bkj9w" Nov 03 02:07:09.145399 ip-10-0-140-107 kubenswrapper[4939]: I1103 02:07:09.145319 4939 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/0bfcb9f9-ab6d-4e86-bd78-88da24aee0d2-proc\") pod \"perf-node-gather-daemonset-bkj9w\" (UID: \"0bfcb9f9-ab6d-4e86-bd78-88da24aee0d2\") " pod="openshift-must-gather-grd2d/perf-node-gather-daemonset-bkj9w" Nov 03 02:07:09.246511 ip-10-0-140-107 kubenswrapper[4939]: I1103 02:07:09.246425 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tzvc4\" (UniqueName: \"kubernetes.io/projected/0bfcb9f9-ab6d-4e86-bd78-88da24aee0d2-kube-api-access-tzvc4\") pod \"perf-node-gather-daemonset-bkj9w\" (UID: \"0bfcb9f9-ab6d-4e86-bd78-88da24aee0d2\") " pod="openshift-must-gather-grd2d/perf-node-gather-daemonset-bkj9w" Nov 03 02:07:09.246511 ip-10-0-140-107 kubenswrapper[4939]: I1103 02:07:09.246468 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/0bfcb9f9-ab6d-4e86-bd78-88da24aee0d2-lib-modules\") pod \"perf-node-gather-daemonset-bkj9w\" (UID: \"0bfcb9f9-ab6d-4e86-bd78-88da24aee0d2\") " pod="openshift-must-gather-grd2d/perf-node-gather-daemonset-bkj9w" Nov 03 02:07:09.246740 ip-10-0-140-107 kubenswrapper[4939]: I1103 02:07:09.246537 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/0bfcb9f9-ab6d-4e86-bd78-88da24aee0d2-podres\") pod \"perf-node-gather-daemonset-bkj9w\" (UID: \"0bfcb9f9-ab6d-4e86-bd78-88da24aee0d2\") " pod="openshift-must-gather-grd2d/perf-node-gather-daemonset-bkj9w" Nov 03 02:07:09.246740 ip-10-0-140-107 kubenswrapper[4939]: I1103 02:07:09.246563 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/0bfcb9f9-ab6d-4e86-bd78-88da24aee0d2-proc\") pod \"perf-node-gather-daemonset-bkj9w\" (UID: \"0bfcb9f9-ab6d-4e86-bd78-88da24aee0d2\") " pod="openshift-must-gather-grd2d/perf-node-gather-daemonset-bkj9w" Nov 03 02:07:09.246740 ip-10-0-140-107 kubenswrapper[4939]: I1103 02:07:09.246602 4939 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/0bfcb9f9-ab6d-4e86-bd78-88da24aee0d2-sys\") pod \"perf-node-gather-daemonset-bkj9w\" (UID: \"0bfcb9f9-ab6d-4e86-bd78-88da24aee0d2\") " pod="openshift-must-gather-grd2d/perf-node-gather-daemonset-bkj9w" Nov 03 02:07:09.246740 ip-10-0-140-107 kubenswrapper[4939]: I1103 02:07:09.246641 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/0bfcb9f9-ab6d-4e86-bd78-88da24aee0d2-podres\") pod \"perf-node-gather-daemonset-bkj9w\" (UID: \"0bfcb9f9-ab6d-4e86-bd78-88da24aee0d2\") " pod="openshift-must-gather-grd2d/perf-node-gather-daemonset-bkj9w" Nov 03 02:07:09.246740 ip-10-0-140-107 kubenswrapper[4939]: I1103 02:07:09.246644 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/0bfcb9f9-ab6d-4e86-bd78-88da24aee0d2-lib-modules\") pod \"perf-node-gather-daemonset-bkj9w\" (UID: \"0bfcb9f9-ab6d-4e86-bd78-88da24aee0d2\") " pod="openshift-must-gather-grd2d/perf-node-gather-daemonset-bkj9w" Nov 03 02:07:09.246740 ip-10-0-140-107 kubenswrapper[4939]: I1103 02:07:09.246670 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/0bfcb9f9-ab6d-4e86-bd78-88da24aee0d2-sys\") pod \"perf-node-gather-daemonset-bkj9w\" (UID: \"0bfcb9f9-ab6d-4e86-bd78-88da24aee0d2\") " pod="openshift-must-gather-grd2d/perf-node-gather-daemonset-bkj9w" Nov 03 02:07:09.246740 ip-10-0-140-107 kubenswrapper[4939]: I1103 02:07:09.246709 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/0bfcb9f9-ab6d-4e86-bd78-88da24aee0d2-proc\") pod \"perf-node-gather-daemonset-bkj9w\" (UID: \"0bfcb9f9-ab6d-4e86-bd78-88da24aee0d2\") " pod="openshift-must-gather-grd2d/perf-node-gather-daemonset-bkj9w" Nov 03 02:07:09.255556 ip-10-0-140-107 kubenswrapper[4939]: I1103 02:07:09.255534 4939 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tzvc4\" (UniqueName: \"kubernetes.io/projected/0bfcb9f9-ab6d-4e86-bd78-88da24aee0d2-kube-api-access-tzvc4\") pod \"perf-node-gather-daemonset-bkj9w\" (UID: \"0bfcb9f9-ab6d-4e86-bd78-88da24aee0d2\") " pod="openshift-must-gather-grd2d/perf-node-gather-daemonset-bkj9w" Nov 03 02:07:09.299978 ip-10-0-140-107 kubenswrapper[4939]: I1103 02:07:09.299952 4939 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-grd2d/perf-node-gather-daemonset-bkj9w" Nov 03 02:07:09.574374 ip-10-0-140-107 kubenswrapper[4939]: I1103 02:07:09.574295 4939 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-grd2d/perf-node-gather-daemonset-bkj9w"] Nov 03 02:07:09.578099 ip-10-0-140-107 kubenswrapper[4939]: W1103 02:07:09.578075 4939 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod0bfcb9f9_ab6d_4e86_bd78_88da24aee0d2.slice/crio-da7614f87b4edb919a5a240ade05b0eb472ee3572335212303456e901d4f6693 WatchSource:0}: Error finding container da7614f87b4edb919a5a240ade05b0eb472ee3572335212303456e901d4f6693: Status 404 returned error can't find the container with id da7614f87b4edb919a5a240ade05b0eb472ee3572335212303456e901d4f6693 Nov 03 02:07:09.867603 ip-10-0-140-107 kubenswrapper[4939]: I1103 02:07:09.867569 4939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-grd2d/perf-node-gather-daemonset-bkj9w" event={"ID":"0bfcb9f9-ab6d-4e86-bd78-88da24aee0d2","Type":"ContainerStarted","Data":"fbe6c2a88fddcbac071840ef34c352cc0a131e546dc77ce97de19f4da4d675f1"} Nov 03 02:07:09.867776 ip-10-0-140-107 kubenswrapper[4939]: I1103 02:07:09.867629 4939 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-grd2d/perf-node-gather-daemonset-bkj9w" Nov 03 02:07:09.867776 ip-10-0-140-107 kubenswrapper[4939]: I1103 02:07:09.867646 4939 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-grd2d/perf-node-gather-daemonset-bkj9w" event={"ID":"0bfcb9f9-ab6d-4e86-bd78-88da24aee0d2","Type":"ContainerStarted","Data":"da7614f87b4edb919a5a240ade05b0eb472ee3572335212303456e901d4f6693"} Nov 03 02:07:09.885585 ip-10-0-140-107 kubenswrapper[4939]: I1103 02:07:09.885530 4939 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-must-gather-grd2d/perf-node-gather-daemonset-bkj9w" podStartSLOduration=1.885517391 podStartE2EDuration="1.885517391s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-03 02:07:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 02:07:09.88430109 +0000 UTC m=+5178.148636035" watchObservedRunningTime="2026-11-03 02:07:09.885517391 +0000 UTC m=+5178.149852372" Nov 03 02:07:10.685800 ip-10-0-140-107 kubenswrapper[4939]: I1103 02:07:10.685768 4939 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-network-console_networking-console-plugin-86c478bb88-z2qgj_bc513cba-d924-4719-a2ba-711529175ee0/networking-console-plugin/0.log" Nov 03 02:07:11.264527 ip-10-0-140-107 kubenswrapper[4939]: I1103 02:07:11.264491 4939 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-n42s6_65d134ac-8a25-4c67-b6c7-cd752a098daf/console-operator/2.log" Nov 03 02:07:11.270040 ip-10-0-140-107 kubenswrapper[4939]: I1103 02:07:11.270011 4939 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-n42s6_65d134ac-8a25-4c67-b6c7-cd752a098daf/console-operator/3.log" Nov 03 02:07:11.766901 ip-10-0-140-107 kubenswrapper[4939]: I1103 02:07:11.766869 4939 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-7ddd9cfdc8-jkd9f_d9dfeb7b-ac37-456e-8239-43505d7f0ccb/discovery/0.log" Nov 03 02:07:11.793715 ip-10-0-140-107 kubenswrapper[4939]: I1103 02:07:11.793689 4939 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-594b68584c-km4bh_5acaee44-b6e2-4504-aebc-aa853d707830/kube-auth-proxy/0.log" Nov 03 02:07:11.831039 ip-10-0-140-107 kubenswrapper[4939]: I1103 02:07:11.831010 4939 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-596dc9545f-7kxj4_8acf90f6-6fff-4a69-b67d-c3ba365c6103/istio-proxy/0.log" Nov 03 02:07:11.849139 ip-10-0-140-107 kubenswrapper[4939]: I1103 02:07:11.849111 4939 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-596dc9545f-9jvbb_512d26b5-8ad9-4cca-8687-cd06d31d8369/istio-proxy/0.log" Nov 03 02:07:11.963791 ip-10-0-140-107 kubenswrapper[4939]: I1103 02:07:11.963761 4939 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-596dc9545f-v5p6s_e6aa4af1-9a04-4116-9d8a-1d8ef5b04958/istio-proxy/0.log" Nov 03 02:07:11.980636 ip-10-0-140-107 kubenswrapper[4939]: I1103 02:07:11.980610 4939 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-596dc9545f-vm5w5_9a335557-68f1-44ab-b673-f9e0a18be917/istio-proxy/0.log" Nov 03 02:07:12.011206 ip-10-0-140-107 kubenswrapper[4939]: I1103 02:07:12.011175 4939 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-59b8dd6b64-jcv9p_dfb7b62c-1cf4-4bf0-b5d3-df882d0078d4/payload-pre-processing/0.log" Nov 03 02:07:12.026678 ip-10-0-140-107 kubenswrapper[4939]: I1103 02:07:12.026612 4939 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-754bb5cbcf-77zpm_8d256a28-b126-4673-bd1e-63aaef592334/payload-processing/0.log" Nov 03 02:07:12.051290 ip-10-0-140-107 kubenswrapper[4939]: I1103 02:07:12.051267 4939 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-d7798ccf5-442kt_cf217a21-c1bd-48be-b203-f59743bceb33/router/0.log" Nov 03 02:07:12.465775 ip-10-0-140-107 kubenswrapper[4939]: I1103 02:07:12.465733 4939 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-cluster-storage-operator_volume-data-source-validator-58c9f5f46d-w4xtw_56261beb-517f-4c88-b59a-42ab41e278fd/volume-data-source-validator/0.log" Nov 03 02:07:13.628293 ip-10-0-140-107 kubenswrapper[4939]: I1103 02:07:13.628259 4939 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-266ht_69b087f8-4a23-4ec8-808e-b61d2e16f67b/dns/0.log" Nov 03 02:07:13.650240 ip-10-0-140-107 kubenswrapper[4939]: I1103 02:07:13.650212 4939 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-266ht_69b087f8-4a23-4ec8-808e-b61d2e16f67b/kube-rbac-proxy/0.log" Nov 03 02:07:13.795731 ip-10-0-140-107 kubenswrapper[4939]: I1103 02:07:13.795696 4939 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-cjj2h_ad5bf442-729d-411c-96e4-2e5105ebadcf/dns-node-resolver/0.log" Nov 03 02:07:14.562058 ip-10-0-140-107 kubenswrapper[4939]: I1103 02:07:14.562027 4939 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-bbk2h_460779b2-3903-4b23-9b4d-e25142782a21/node-ca/0.log" Nov 03 02:07:15.879857 ip-10-0-140-107 kubenswrapper[4939]: I1103 02:07:15.879811 4939 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-grd2d/perf-node-gather-daemonset-bkj9w" Nov 03 02:07:16.048531 ip-10-0-140-107 kubenswrapper[4939]: I1103 02:07:16.048491 4939 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-7c9b87d574-4pw7v_e1b47e77-a4ab-4d61-ace8-deeea43ea944/limitador/0.log" Nov 03 02:07:16.215451 ip-10-0-140-107 kubenswrapper[4939]: I1103 02:07:16.215369 4939 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-7ddd9cfdc8-jkd9f_d9dfeb7b-ac37-456e-8239-43505d7f0ccb/discovery/0.log" Nov 03 02:07:16.695384 ip-10-0-140-107 kubenswrapper[4939]: I1103 02:07:16.695350 4939 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-594b68584c-km4bh_5acaee44-b6e2-4504-aebc-aa853d707830/kube-auth-proxy/0.log" Nov 03 02:07:16.764860 ip-10-0-140-107 kubenswrapper[4939]: I1103 02:07:16.764825 4939 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-596dc9545f-7kxj4_8acf90f6-6fff-4a69-b67d-c3ba365c6103/istio-proxy/0.log" Nov 03 02:07:16.795220 ip-10-0-140-107 kubenswrapper[4939]: I1103 02:07:16.795185 4939 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-596dc9545f-9jvbb_512d26b5-8ad9-4cca-8687-cd06d31d8369/istio-proxy/0.log" Nov 03 02:07:16.986534 ip-10-0-140-107 kubenswrapper[4939]: I1103 02:07:16.986452 4939 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-596dc9545f-v5p6s_e6aa4af1-9a04-4116-9d8a-1d8ef5b04958/istio-proxy/0.log" Nov 03 02:07:17.016822 ip-10-0-140-107 kubenswrapper[4939]: I1103 02:07:17.016795 4939 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-596dc9545f-vm5w5_9a335557-68f1-44ab-b673-f9e0a18be917/istio-proxy/0.log" Nov 03 02:07:17.078957 ip-10-0-140-107 kubenswrapper[4939]: I1103 02:07:17.078933 4939 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-59b8dd6b64-jcv9p_dfb7b62c-1cf4-4bf0-b5d3-df882d0078d4/payload-pre-processing/0.log" Nov 03 02:07:17.106075 ip-10-0-140-107 kubenswrapper[4939]: I1103 02:07:17.106049 4939 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-754bb5cbcf-77zpm_8d256a28-b126-4673-bd1e-63aaef592334/payload-processing/0.log" Nov 03 02:07:17.147877 ip-10-0-140-107 kubenswrapper[4939]: I1103 02:07:17.147848 4939 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-d7798ccf5-442kt_cf217a21-c1bd-48be-b203-f59743bceb33/router/0.log" Nov 03 02:07:18.030710 ip-10-0-140-107 kubenswrapper[4939]: E1103 02:07:18.030669 4939 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-sthjh" containerName="extractor" Nov 03 02:07:20.543810 ip-10-0-140-107 kubenswrapper[4939]: I1103 02:07:20.543779 4939 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-s9hbz_75481915-3cbe-475b-abe7-cb4e81c93427/serve-healthcheck-canary/0.log" Nov 03 02:07:21.184266 ip-10-0-140-107 kubenswrapper[4939]: I1103 02:07:21.184227 4939 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-operator-7c4986d449-xprcf_ca0f883c-c2b7-4bf6-a90b-ffdbeac161a8/insights-operator/1.log" Nov 03 02:07:21.187273 ip-10-0-140-107 kubenswrapper[4939]: I1103 02:07:21.187246 4939 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-operator-7c4986d449-xprcf_ca0f883c-c2b7-4bf6-a90b-ffdbeac161a8/insights-operator/2.log" Nov 03 02:07:21.306823 ip-10-0-140-107 kubenswrapper[4939]: I1103 02:07:21.306794 4939 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-sthjh_945da51a-32cb-4769-900d-1aeb266c7565/kube-rbac-proxy/0.log" Nov 03 02:07:21.331579 ip-10-0-140-107 kubenswrapper[4939]: I1103 02:07:21.331552 4939 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-sthjh_945da51a-32cb-4769-900d-1aeb266c7565/exporter/0.log" Nov 03 02:07:21.342425 ip-10-0-140-107 kubenswrapper[4939]: I1103 02:07:21.342402 4939 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-7ddd9cfdc8-jkd9f_d9dfeb7b-ac37-456e-8239-43505d7f0ccb/discovery/0.log" Nov 03 02:07:21.363451 ip-10-0-140-107 kubenswrapper[4939]: I1103 02:07:21.363427 4939 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-sthjh_945da51a-32cb-4769-900d-1aeb266c7565/extractor/0.log" Nov 03 02:07:21.372089 ip-10-0-140-107 kubenswrapper[4939]: I1103 02:07:21.372067 4939 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-594b68584c-km4bh_5acaee44-b6e2-4504-aebc-aa853d707830/kube-auth-proxy/0.log" Nov 03 02:07:21.411444 ip-10-0-140-107 kubenswrapper[4939]: I1103 02:07:21.411423 4939 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-596dc9545f-7kxj4_8acf90f6-6fff-4a69-b67d-c3ba365c6103/istio-proxy/0.log" Nov 03 02:07:21.429949 ip-10-0-140-107 kubenswrapper[4939]: I1103 02:07:21.429902 4939 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-596dc9545f-9jvbb_512d26b5-8ad9-4cca-8687-cd06d31d8369/istio-proxy/0.log" Nov 03 02:07:21.556725 ip-10-0-140-107 kubenswrapper[4939]: I1103 02:07:21.556655 4939 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-596dc9545f-v5p6s_e6aa4af1-9a04-4116-9d8a-1d8ef5b04958/istio-proxy/0.log" Nov 03 02:07:21.577310 ip-10-0-140-107 kubenswrapper[4939]: I1103 02:07:21.577279 4939 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-596dc9545f-vm5w5_9a335557-68f1-44ab-b673-f9e0a18be917/istio-proxy/0.log" Nov 03 02:07:21.612393 ip-10-0-140-107 kubenswrapper[4939]: I1103 02:07:21.612367 4939 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-59b8dd6b64-jcv9p_dfb7b62c-1cf4-4bf0-b5d3-df882d0078d4/payload-pre-processing/0.log" Nov 03 02:07:21.626773 ip-10-0-140-107 kubenswrapper[4939]: I1103 02:07:21.626752 4939 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-754bb5cbcf-77zpm_8d256a28-b126-4673-bd1e-63aaef592334/payload-processing/0.log" Nov 03 02:07:21.652590 ip-10-0-140-107 kubenswrapper[4939]: I1103 02:07:21.652562 4939 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-d7798ccf5-442kt_cf217a21-c1bd-48be-b203-f59743bceb33/router/0.log" Nov 03 02:07:24.048425 ip-10-0-140-107 kubenswrapper[4939]: I1103 02:07:24.048396 4939 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub-operator-system_opendatahub-operator-controller-manager-7cc9488979-jh6tz_f90f3499-7ad4-43f7-9b32-238b34c422c1/manager/0.log" Nov 03 02:07:24.074719 ip-10-0-140-107 kubenswrapper[4939]: I1103 02:07:24.074690 4939 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub-operator-system_opendatahub-operator-controller-manager-7cc9488979-jh6tz_f90f3499-7ad4-43f7-9b32-238b34c422c1/copy-manifests/0.log" Nov 03 02:07:24.096810 ip-10-0-140-107 kubenswrapper[4939]: I1103 02:07:24.096781 4939 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82x8c_c3ffbe53-7f23-4750-b95d-476e6cfc58a6/konnectivity-agent/0.log" limit=1024 Nov 03 02:07:26.160424 ip-10-0-140-107 kubenswrapper[4939]: I1103 02:07:26.160377 4939 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_ai-gateway-controller-75bbf6968c-fz65x_f690a187-3701-4c28-a425-e5e1f447b650/manager/0.log" Nov 03 02:07:26.395869 ip-10-0-140-107 kubenswrapper[4939]: I1103 02:07:26.395839 4939 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-7ddd9cfdc8-jkd9f_d9dfeb7b-ac37-456e-8239-43505d7f0ccb/discovery/0.log" Nov 03 02:07:26.460933 ip-10-0-140-107 kubenswrapper[4939]: I1103 02:07:26.460848 4939 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-594b68584c-km4bh_5acaee44-b6e2-4504-aebc-aa853d707830/kube-auth-proxy/0.log" Nov 03 02:07:26.506383 ip-10-0-140-107 kubenswrapper[4939]: I1103 02:07:26.506351 4939 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-596dc9545f-7kxj4_8acf90f6-6fff-4a69-b67d-c3ba365c6103/istio-proxy/0.log" Nov 03 02:07:26.528050 ip-10-0-140-107 kubenswrapper[4939]: I1103 02:07:26.528024 4939 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-596dc9545f-9jvbb_512d26b5-8ad9-4cca-8687-cd06d31d8369/istio-proxy/0.log"